[00:02] <wgrant> Hm, I didn't know enough data was exposed to create such a thing.
[00:04] <kb9vqf> Is the PPA build system open sourced as well?
[00:04] <wgrant> Yes.
[00:04] <wgrant> https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally
[00:04] <kb9vqf> Thanks!
[00:13] <jml> wgrant, that's 100% through the API
[00:13] <jml> wgrant, using anonymous access only.
[00:13] <jml> wgrant, although the numbers are wrong, for some reason that yet eludes me
[00:14] <wgrant> jml: Oh, using date_confirmed and its kin?
[00:14] <jml> yep
[00:15] <wgrant> kb9vqf: I've just tweaked that page a bit, trimming obsolete sections and that sort of thing.
[00:15] <kb9vqf> OK, thanks--I was still in the configuration section anyway ;-)
[00:18] <kb9vqf> wgrant: Any idea what might have caused this: http://pastebin.ubuntu.com/447456/
[00:18] <kb9vqf> All other pages seem to work fine, but the home page doesn't
[00:18] <kb9vqf> (https://launchpad.dev/)
[00:19] <wgrant> kb9vqf: You're using an empty DB without sample data?
[00:19] <kb9vqf> I trimmed out a lot of the sample data
[00:19] <kb9vqf> Guess I trimmed something I shouldn't have ;-)
[00:19] <wgrant> Try cronscripts/update-stats.py
[00:19] <wgrant> That may fix it, but it may not work unless you have at least one project.
[00:19] <kb9vqf> I was wondering if that would be the case
[00:19] <kb9vqf> Thanks for the cron hint as well
[00:22] <kb9vqf> One more thing...regarding changing all the instances of the word "Launchpad" to something else, would I be lucky enough for that to be controlled by a single file somewhere?
[00:22] <kb9vqf> Or will I have to resort to sed
[00:22] <wgrant> sed, probably.
[00:23] <wgrant> I use http://pastebin.ubuntu.com/447458/ and http://pastebin.ubuntu.com/447459/ on an empty DB (without any sample data at all).
[00:23] <wgrant> Creates most of the critical celebrities.
[00:23] <wgrant> Updates the statistics, that sort of thing.
[00:24] <wgrant> And the second populates Ubuntu.
[00:24]  * kb9vqf spent an entire day trying to get the celebrities in place
[00:24] <kb9vqf> I'll try the scripts as soon as Launchpad finishes rebuilding
[00:29] <ajmitch> wgrant: you have those scripts somewhere other than pastebin?
[00:29] <ajmitch> they look useful
[00:42] <wgrant> ajmitch: Not really.
[00:42] <wgrant> They just sit around in ~/launchpad/utils and occasionally get unbroken when I get fed up with how terrible the sample data is.
[00:47] <lifeless> jml: wow
[00:47] <lifeless> jml: lovely visualisation; can you do the bazaar project group too ?
[00:48] <jml> lifeless, I want to. You might be able to figure out how to do it yourself if you branch lp:~jml/+junk/convergence
[00:49] <lifeless> jml: I'm sure I can :) . Uhm this seems like a lptools thing, if you want to make a home for it
[00:49] <jml> lifeless, I'm deferring the problem of figuring out how to have couch views that I can give a parameter to.
[00:49] <jml> (probably the answer is something horrible)
[00:50] <lifeless> jml: this is using offlinelaunchpadstuff ?
[00:50] <jml> lifeless, no.
[00:50] <jml> I wanted to, and then I spent a bunch of time making the offlinelaunchpad stuff better in subtle, peripheral, important ways and then I got on with this.
[00:51] <wgrant> jml: Launchpad didn't collapse under the load of generating that graph?
[00:51] <lifeless> heh
[00:51] <lifeless> wgrant: no
[00:51] <jml> wgrant, not as far as I can tell.
[00:51] <lifeless> wgrant: we generate more load internally than the API servers can cause ;P
[00:52] <wgrant> What was it that was causing the DB issues?
[00:54] <lifeless> more load on master than desired, not enough on the slaves.
[00:54] <lifeless> which was due to replication lag being stale
[00:54] <wgrant> Ah.
[00:54] <lifeless> which was due to the replication lag monitor not running.
[00:55] <wgrant> Oops.
[00:55] <lifeless> which wasn't fixed immediately because its not, itself, monitored.
[00:55] <lifeless> Which is because we don't have a 'must have monitoring on day 1' rule.
[00:55] <lifeless> or something like htat.
[00:56] <lifeless> (such a rule might be more trouble than its worth)
[01:02] <jml> meh
[01:02] <lifeless> jml: ? I'm saying not everything is important enough to monitor, in my suggestion that a hard rule might be trouble.
[01:03] <jml> apart from the 162 bug tasks that are counted as "New" on that graph due to API exposure, I don't know why there are 500 bug tasks that have no "date_foo" field but are not actually "New"
[01:03] <jml> lifeless, the "meh" was in response to my internal angst :)
[01:03] <lifeless> ah :)
[01:03] <lifeless> by all means, mehangst away
[01:04] <wgrant> jml: Incomplete?
[01:04] <jml> wgrant, that's the 162
[01:04] <wgrant> Ah.
[01:11] <thumper> jml: hi
[01:11] <thumper> jml: are you skypeable?
[01:11] <jml> thumper, only for fun and social reasons :)
[01:11] <thumper> jml: I have something of strategic importance
[01:11] <thumper> heh
[01:11] <thumper> probably neither fun nor social
[01:12] <thumper> jml: are you on leave?
[01:12] <jml> thumper, yeah.
[01:12] <thumper> when are you back?
[01:12] <jml> thumper, thus dicking about with desktopcouch and graphs
[01:12] <jml> thumper, back Monday
[01:12] <thumper> ok
[01:12] <thumper> remind me Monday
[01:12] <jml> thumper, will do. look at the shiny. http://people.canonical.com/~jml/convergence/
[01:12] <thumper> I saw
[01:12] <thumper> lots of bugs
[01:13] <thumper> you didn't graph won't fix :)
[01:13] <jml> thumper, yeah I did!
[01:13] <thumper> really?
[01:13] <jml> thumper, it's the invisible bit at the bottom :)
[01:13] <wgrant> What about Opinion?
[01:13] <jml> thumper, or rather, wontfix are considered done for the purpose of this graph
[01:13] <wgrant> Or is that not actually in production yet.
[01:13] <thumper> jml: your key only shows five statuses
[01:14] <jml> thumper, it deliberately doesn't show non-open bugs.
[01:14] <thumper> It'd be interesting to see our rate of fixing ...
[01:14] <thumper> has it slowed down / sped up ...
[01:15] <thumper> jml: a first derivative graph of the bugs graph would also be interesting
[01:15] <jml> thumper, well, you can kind of see that by the "Fix committed" line at the bottom.
[01:15] <thumper> jml: at what rate are bugs being filed
[01:15] <jml> thumper, some of that stuff is on this (old) set of graphs: http://people.canonical.com/~jml/lp-bugs/
[01:17] <jml> thumper, I guess now that I've figured out the relevant technologies and don't have to do staging database queries to get this information, I can add those.
[01:18]  * thumper nods
[01:18]  * thumper goes back to wrapping bugs
[01:18] <jml> thumper, first derivative isn't useful with so many lines -- I think you only want "open" and "closed" then
[01:22]  * thumper stabs ORMs in the face
[01:22] <wgrant> Ridiculously inefficient SQL usage?
[01:23] <thumper> yes
[01:23] <thumper> and the hoop jumping to make it more efficient
[01:23] <thumper> part of what I need to talk to jml about
[01:23] <wgrant> I wish I could tell it that I wanted it to precalculate various attributes.
[01:23] <wgrant> Rather than having to do that manually, which makes everything about 10000000000x times uglier.
[01:24] <thumper> like what?
[01:25] <wgrant> For example, the publisher needs to retrieve lots of SourcePackagePublishingHistorys, their SourcePackageReleases, their SourcePackageReleaseFiles, and various other collections.
[01:26] <wgrant> If I want to do that in less than hundreds of thousands queries (quite literally), I have to formulate a query manually and can't use the objects' normal attributes.
[01:26] <lifeless> yes
[01:26] <lifeless> orms are evil
[01:27] <lifeless> db query interfaces - like storm /has/ - are very nice
[01:27] <wgrant> I should be able to tell Storm to precalculate SPPH.spr, SPR.files, and that sort of thing.
[01:27] <lifeless> but magic attributes - very hard to get the right balance
[01:27] <lifeless> wgrant: you can
[01:27] <wgrant> There is the prejoin implementation in the SQLObject wrapper.
[01:27] <lifeless> wgrant: there's a couple of hooks; not sure if the precise one you need is present
[01:27] <wgrant> But it doesn't do collections.
[01:27] <lifeless> wgrant: theres a 'when getting this from the DB do xyz' hook
[01:28] <lifeless> wgrant: I'm pretty sure
[01:32] <maxb> heh, did someone give launchpad/ppa a biased buildscore? :-)
[01:33] <lifeless> don't think so
[01:34] <maxb> I just uploaded stuff and it seems to have bypassed the entire queue :-)
[01:34] <lifeless> rotfl
[01:36] <wgrant> That is a little suspicious.
[01:37] <wgrant> I suppose we can delete most of the PPA now.
[01:37] <maxb> s/suspicious/convenient/ :-)
[01:37] <wgrant> Mm, although I guess we need to wait until 10.06 is out the door.
[01:38] <lifeless> is there a karma<->priority matchup ?
[01:40] <wgrant> lifeless: Only inasmuch as people with more karma are more likely to be able to hit up buildd admins :P
[01:41] <maxb> "Automatic retry of builds waiting on dependencies is disabled pending resolution of a performance problem" -- o rly?
[01:41] <wgrant> It's back now.
[01:41] <wgrant> Was announced on launchpadstatus, I thought.
[01:42] <maxb> Also, https://edge.launchpad.net/~lamont/+archive/psql8.3/+packages seems to have gone into an endless loop of depwait -> retry -> depwait -> retry ...
[01:42] <maxb> i.e. they're being erroneously retried
[01:43] <wgrant> If you ever catch a build log or dependencies value from any of them, please keep a note of it.
[01:44] <wgrant> There are some known bugs, but not any that should affect PPA builds.
[01:44] <maxb> the lpia build there is building again now
[01:44] <wgrant> Yeah.
[01:44] <wgrant> Oh, if it depwaits I guess it will finish soon.
[01:44] <wgrant> Forgot that detail.
[01:46] <wgrant> If buildd-manager ever wakes up, that is.
[01:47] <wgrant> I must fix it one day.
[01:48] <maxb> If there's a buildd-admin willing, can I get a rescore on at least one build in https://edge.launchpad.net/~maxb/+archive/launchpad/+packages ? I'd like to get a confirmed success before I sleep, so I can give them to stub tomorrow
[01:50] <thumper> ✁☹
[01:50] <thumper> ✁☹
[01:50] <thumper> ✁☹
[01:50] <thumper> ✁☹
[01:50] <thumper> ✁☹
[01:50] <wgrant> Ah, debhelper >=7.
[01:50] <thumper> grrrrraaaaahhhhhhhhhhhh!!!!!!!!!!!!!
[01:50] <wgrant> So it's probably the pocket-naïveté.
[01:51] <wgrant> thumper: What has Storm done now?
[01:51]  * jelmer waves to thumper and wgrant 
[01:51] <wgrant> Or Soyuz?
[01:51] <wgrant> Morning jelmer.
[01:51]  * jelmer ducks
[01:51] <thumper> I'm using lazr.delegate to wrap a bug so it doesn't hit the db for its tasks
[01:52] <thumper> but getBugTask(self, target) uses self.bug_tasks
[01:52] <thumper> so in order to get that method to the cached tasks,
[01:52] <thumper> I either have to copy the implementation
[01:52] <thumper> or violate one of our rules for imports
[01:52] <thumper> and import Bug into code.browser
[01:53] <thumper> to go Bug.getBugTask(my_wrapped_bug, target)
[01:53] <wgrant> So you're precalculating the collection?
[01:53] <thumper> yes
[01:53] <wgrant> Well.
[01:53] <thumper> in a single query to get the linked bugs and all their tasks
[01:53] <wgrant> One easy way out is to abuse the fact that Storm doesn't notice when you assign a list.
[01:54] <wgrant> So you clobber bug.bug_tasks with your precalculated list.
[01:54] <wgrant> Evil, yes.
[01:54] <wgrant> But so are the other methods.
[01:54] <thumper> eh?
[01:54] <thumper> that is evil
[01:54] <spm> maxb: did you get that rescored? I'll do it now?
[01:55] <maxb> spm: yes please
[01:55] <spm> ahh. slony. hell yes I'l rescore those.
[01:56]  * thumper goes for the lesser of three evils
[01:56] <maxb> :-)
[01:57] <spm> maxb: powa has been abused and used: https://edge.launchpad.net/~maxb/+archive/launchpad/+builds
[01:58] <maxb> neat, and they've been dispatched already :-)
[01:58] <spm> see. I am useful for *some* things. ;-)
[02:00] <maxb> Now I just need to talk lamont into using my builds instead of redoing it
[02:01] <wgrant> Doesn't Hardy have 8.3 already?
[02:02] <spm> the backport for hady-cat that we do? generally that's little more than trivial textual changes and getting a localised rebuild of same.
[02:08] <maxb> An 8.3 what?
[02:09] <maxb> These builds were about getting the most recent upstream of slony built for (hardy,lucid)*(8.3,8.4)
[02:09] <wgrant> Ah, right, Slony.
[02:09] <maxb> which it now has :-)
[02:56] <thumper> queries down from 294 to 93 using cached bug tasks
[02:56] <thumper> (for my test branch)
[03:08] <ajmitch> thumper: you managed to mangle it to use the right property?
[03:08] <thumper> no
[03:08] <thumper> well kinda
[03:08] <thumper> too much special code IMO
[03:09] <thumper> now I see most of the queries being in the menu link formatters :(
[03:09] <thumper> it should not be doing that :(
[03:10] <thumper> I thought that some old fix of salgado fixed that
[03:10]  * thumper wishes sinzui was around to explain
[03:11] <sinzui> Yes it should not be doing that
[03:11] <sinzui> I crafted a menu two weeks ago to avoid subscription checks. each menu call in the template was  a call to the db
[03:13] <sinzui> thumper, I think menus should be singletons or we use a utility that manages to menus for each object to ensure there is only for instance for each request
[03:13] <thumper> sinzui: ah, so it is known that it is somewhat fucked then?
[03:14] <sinzui> You and I know this
[03:14] <thumper> heh
[03:14] <thumper> ok, how to fix this in my branch?
[03:14] <thumper> ah
[03:14] <thumper> I know why it is screwed in this moment
[03:14] <sinzui> I was very tempted to risk toppling all menus to fix the performance issues working with milestones on pages
[03:15] <thumper> the context object is the unwrapped object
[03:15] <thumper> the view.context is the decorated object
[03:15] <thumper> I bet the pages are using context/menu rather than view/context/menu...
[03:15] <thumper> I'm caching the queries needed by the menu
[03:15] <thumper> in my wrapped object
[03:15]  * thumper checks the templates
[03:16] <sinzui> I think each template fragment is creating the menu on demand or for it's chunk, but that same menu for the context may be instantiated 30 times
[03:17] <thumper> right
[03:17] <sinzui> I think salgado had a hack to put the data in the request so that templates shared the menu, but I do not see evidence of that
[03:18] <thumper> sinzui: can I override the globs for the template?
[03:18] <thumper> sinzui: I want to set the context to be my wrapped context
[03:18] <thumper> rather than the unwrapped one
[03:18] <thumper> sinzui: do you know where the template render code is actually called?
[03:19] <sinzui> render() for the fmt:link?
[03:19] <thumper> I mean the code that sets the 'context' for the page template render
[03:19] <sinzui> ah
[03:20] <sinzui> That would be sidnei's chameleon engine
[03:20] <thumper> are we using chameleon?
[03:20] <sinzui> yes
[03:20]  * sidnei raises an eyebrow
[03:20] <thumper> where is the code?
[03:20] <sinzui> in an egg
[03:20] <thumper> :(
[03:21]  * sinzui saw it while purging py2.5 eggs today
[03:21] <thumper> surely we call it somewhere with a dictionary of locals right?
[03:21] <thumper> that is what I want to override
[03:21] <sidnei> sinzui, chameleon is not enabled, though the code is there
[03:21] <sinzui> :(
[03:21]  * thumper hunts in webapp for calling pagetemplates
[03:21] <thumper> sidnei: unless you can tell me where to look
[03:22] <sinzui> thumper, are these LaunchpadView views?
[03:22] <thumper> yep
[03:22] <sinzui> I believe we hacked __call__
[03:22] <sidnei> thumper, i might if sinzui doesn't beat me to it
[03:23] <thumper> :(
[03:23] <thumper> there seems to be much magic there
[03:24] <sidnei> there's a ton of magic involved yes. it left me scratching my had last time i looked at it.
[03:24] <sidnei> s/had/head
[03:24] <thumper> __call__ calls render
[03:24] <thumper> render calls self.template()
[03:24] <thumper> template is a property that returns self.index
[03:25] <thumper> self.index seems opaque
[03:25] <sinzui> thumper, during traversal, the view is construct. adapters from zcml marry the named view to a template and create the final view call, and it monekypatches __facet__ because using ILayer would have been too easy
[03:25] <sidnei> thumper, self.index is the template that gets registered by the zcml directive iirc
[03:26] <thumper> sinzui: do I have to read the entire publisher file?
[03:26] <sinzui> No, look for __facet__ I scream every time I see that monster
[03:27] <thumper> ENOTFOUND
[03:27]  * sinzui ponders
[03:28] <thumper> __launchpad_facet__
[03:29] <sidnei> sorry for not being more helpful, it's almost midnight here and im trying to wrap up a large refactoring
[03:30] <sinzui> thumper, yes, in metazcml
[03:32] <thumper> ✁☹
[03:36] <ajmitch> are there any tests around for scripts/ftpmaster-tools ?
[03:36] <lifeless> hah
[03:37] <ajmitch> I was afraid of that
[03:38] <thumper>  return mapply(ob, request.getPositionalArguments(), request)
[03:38] <thumper> publication.py line 436
[03:39] <ajmitch> on a related note, is launchpad.net running on lucid
[03:39] <sinzui> yes
[03:40] <sinzui> as of today, dev is on 2.6
[03:40] <sinzui> python 2.6
[03:41] <ajmitch> ok, I just saw some duplicated code in sync-source.py to remove then, I'm just changing some of the version matching in there :)
[03:41] <spm> sinzui: it is? oh *dev* is. right. prod is still running on hardy.
[03:42] <sinzui> all lpnet is hardy?
[03:42] <lifeless> ajmitch: different questions
[03:42] <lifeless> ajmitch: 'works on 2.6'
[03:42] <lifeless> ajmitch: != 'deployed on 2.6'
[03:42] <lifeless> ajmitch: we can't remove compat stuff until its deployed on 2.6
[03:42] <sinzui> We this will be a very exciting rollout. I am glad I am not the release manager for 10.06
[03:42] <spm> sinzui: I hope so, or else it's been upgraded without me knowing about it; which'd be fun :-)
[03:42] <ajmitch> lifeless: in this case, it's code duplicated from python-debian that is in lucid
[03:42] <ajmitch> rather than anything python 2.6-specific
[03:43] <lifeless> ajmitch: we're runniong on hardy
[03:43] <ajmitch> which is mostly what I was asking
[03:45] <lifeless> yeah, you asked the right question; but a little more clarity that it wasn't a proxy for python version would have helped ;)
[03:45] <ajmitch> I did clarify, just a little too late :)
[03:45] <spm> :-)
[03:46] <ajmitch> it was bug 520508, fwiw
[03:46] <mup> Bug #520508: Duplicate copy of split_gpg_and_payload() <soyuz-upload> <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/520508>
[03:49]  * thumper frowns
[03:50] <thumper> I can't find it!
[03:51] <thumper> damn it
[03:51] <thumper> the publication code calls request.getPositionalArgs
[03:51] <thumper> which defers to self._args of the request
[03:51] <thumper> which gets set with positional= kwarg
[03:51]  * thumper is frustrated
[03:52] <thumper> going back to changing context/menu to view/context/menu
[04:07] <thumper> for some reason my context is getting wrapped twice
[04:15] <thumper> \o/
[04:15] <thumper> down to 63 queries
[04:15]  * thumper looks for dupes
[04:22] <lifeless> nice
[04:24] <thumper> down from almost 300
[04:24] <thumper> still dupes though
[04:24] <thumper> and we have 20 queries before we even hit object traversal
[04:24] <thumper> I think with some effort we could drop that to under 1-
[04:24] <thumper> 10
[04:36] <wgrant> thumper: Ew, what are the pre-traversal queries?
[04:37] <wgrant> That's mildly insane.
[04:37] <thumper> :)
[04:37] <thumper> want me to pastebin it?
[04:38] <wgrant> Well, I have an exam in 20 minutes, so it's probably best that I don't look at it :P
[04:38] <thumper> http://pastebin.ubuntu.com/447557/
[04:38] <thumper> sorry
[04:38] <thumper> there goes the study
[04:38] <wgrant> Heh.
[04:38] <thumper> which exam?
[04:38] <wgrant> Logic Completeness and Incompleteness.
[04:39] <ajmitch> oh that sounds fun
[04:39] <wgrant> SELECT AccountPassword.account, AccountPassword.id, AccountPassword.password FROM AccountPassword WHERE AccountPassword.account = %s
[04:39] <wgrant> Why would it being doing that? It's not meant to know that passwords exist.
[04:46] <thumper> wgrant: go to your exam
[04:53] <thumper> ew
[04:53] <thumper> removeSecurityProxy(db.branch).__class__.displayname.__get__(db)
[04:53] <thumper> I think that passes the wrapped object (db) into the property getter as self
[04:54] <thumper> it is pretty revolting
[05:05] <stub> Hmm.... """sourcepackagerelease components "not null" site:bugs.launchpad.net """ in google finds me the bug I want on index pages, but not the bug itself.
[07:19] <madscientist159> wgrant: You still around?
[07:20] <madscientist159> wgrant: I tried to run your script to initialize the empty database, but all I got was this:
[07:20] <madscientist159> http://paste.ubuntu.com/447607/
[07:24] <kb9vqf> Anyone have an idea why I am getting that permission error?  I've tried as root and as the postgresql user...
[07:55] <kb9vqf> wgrant: I managed to get around the problem by temporarily granting all access to the karmacategory table and sequence, so my above comments are just an FYI
[07:57] <kb9vqf> Also FYI, the second script will notrun, complaining about "lp.registry.interfaces.distroseries" not existing
[08:07] <kb9vqf> Ideas for debugging this oops?
[08:07] <kb9vqf> v
[08:07] <kb9vqf> http://pastebin.ubuntu.com/447619/
[08:07] <kb9vqf> It happens only when trying to go to a project page such as https://launchpad.dev/launchpad
[08:08] <kb9vqf> Everything else seems to work great
[08:18] <adeuring> good morning
[08:20] <kb9vqf> wgrant: Here's a fixed version of the second script: http://pastebin.ubuntu.com/447622/
[09:20] <wgrant> kb9vqf: Ah, yeah, sorry, forgot about the karmacategory perm fix.
[09:20] <wgrant> And DistroSeriesStatus was recently moved and renamed to SeriesStatus, and apparently that version of the script hasn't been fixed yet.
[09:20] <wgrant> And it looks like you need to run rmy second script now, or all projects explode.
[09:20] <wgrant> That's pretty inconvenient.
[09:22] <kb9vqf> wgrant: Did you see my fixed version of the second script ;-)
[09:23] <wgrant> Aha.
[09:23] <kb9vqf> I figured out the projects exploding...my local installation is actually working quite well ATM
[09:23] <kb9vqf> Thanks to your scripts ;-)
[09:23] <kb9vqf> Still working on the PPA builders though
[09:23]  * kb9vqf crosses fingers
[09:24] <wgrant> What's wrong with them?
[09:24] <kb9vqf> Nothing yet
[09:24] <kb9vqf> Just don't have the software installed yet
[09:24] <wgrant> Ah.
[09:24] <kb9vqf> I fumble-fingered a move command earlier and deleted the past 12 hours of work
[09:25] <wgrant> Well, beware, I wrote those instructions too, so...
[09:25] <kb9vqf> :-)
[09:26] <wgrant> I might throw the scripts into a branch and push it up.
[09:27] <kb9vqf> Be aware that there is an issue (syntax error) in one of the Launchpad python files the first script calls...something to do with the distribution IIRC
[09:27] <kb9vqf> Sorry I don't have the filename handy, that was several hours ago
[09:27] <kb9vqf> There was a colon in the function declaration when it was really supposed to be a comma
[09:28] <wgrant> Hmm, odd. I guess I'll find out in a few minutes when I test the whole lot.
[09:28] <noodles775> kb9vqf: yes, a testfix branch was just landed. (The issue was the result of a bad-but-non-conflicting merge)
[09:28] <wgrant> However, I think the ubuntu.currentseries.displayname crash is a legitimate bug.
[09:28] <wgrant> Let's see...
[09:28] <kb9vqf> Well, it went away after the second script was modified and run
[09:29] <wgrant> Yeah.
[09:29] <wgrant> But I meant the second one to be optional; that's why it's separate.
[09:29] <kb9vqf> Oh, OK
[09:30] <wgrant> Ah, it's not a real bug that will show up in production :(
[09:38] <kb9vqf> How does the blog widget on the Launchpad home page operate?  Can I redirect it to my blog?
[09:38] <wgrant> It's... hardcoded.
[09:38] <kb9vqf> Oh
[09:38] <kb9vqf> How can I remove the entries that are there right now?
[09:39] <wgrant> grep around and find the template that has them.
[09:39] <kb9vqf> In the DB or in the files?
[09:39] <wgrant> The files.
[09:39] <wgrant> The entries themselves are hardcoded.
[09:39] <kb9vqf> Ohhhh
[09:39] <wgrant> Not the blog feed that it pulls from.
[09:40] <kb9vqf> Shall I say that seems rather...odd?
[09:40] <kb9vqf> ;-)
[09:40] <wgrant> Yes.
[09:53] <kb9vqf> Is there a list anywhere of how often each cron script should be run?
[09:53]  * kb9vqf doesn't want to waste system resources by running every minute if its not needed
[09:56] <wgrant> That's something they've not revealed.
[09:58] <kb9vqf> OK, trial and error it is then
[10:06] <kb9vqf> wgrant: Are there any instructions for setting up a build daemon on a remote machine?
[10:06] <kb9vqf> I have a rather large cluster here of identical machines I'd like to use
[10:08] <wgrant> kb9vqf: Just install launchpad-buildd on them and point LP at them.
[10:08] <wgrant> But watch out. lp-buildd is unauthenticated.
[10:08] <kb9vqf> I have a dedicated private network, so that's not a problem ;-)
[10:08] <kb9vqf> (Physically secured too)
[10:08] <kb9vqf> So I have to install the lp-branches/devel stuff on each build machine then?
[10:09] <wgrant> No. Just grab the launchpad-buildd .deb that is created.
[10:09] <kb9vqf> What about the chroot image?
[10:09] <kb9vqf> Oh..is that image system wide and uploaded to each builder on-demand?
[10:10] <wgrant> Right.
[10:10] <wgrant> The chroot is sent out for each build.
[10:10] <wgrant> (although it's cached on the slave)
[10:10] <kb9vqf> That makes a ton more sense
[10:10] <kb9vqf> Thanks!
[10:11] <wgrant> https://code.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch contains all the changes and fixed scripts for bootstrapping.
[10:11] <kb9vqf> Subscribed myself so I won't lose it :-)
[10:12] <kb9vqf> Has anyone else actuallydeployed Launchpad other than Canonical?
[10:12] <kb9vqf> Just out of curiosity
[10:12] <wgrant> I don't believe so.
[10:12] <kb9vqf> So I might be the first
[10:12] <kb9vqf> :-)
[10:12] <wgrant> Anyone who tried would have probably run into huge problems and asked here.
[10:12] <wgrant> Everyone who has asked here has given up.
[10:19] <kb9vqf> Hmmm....Librarian
[10:19] <kb9vqf> It doesn't seem to be started from "make run"
[10:19] <wgrant> noodles775: Have the getBuildRecords timeouts been fixed? I've only had a few emails from cron today, rather than the usual several dozen.
[10:19] <wgrant> It still times out sometimes, but not as much.
[10:19] <wgrant> kb9vqf: It probably only listens on localhost.
[10:20] <kb9vqf> Yes; I'm getting this though
[10:20] <kb9vqf> Librarian upload failed: [localhost:58090]: [Errno 111] Connection refused
[10:20] <wgrant> Hm.
[10:20] <wgrant> Is that from manage-chroot?
[10:20] <kb9vqf> Yup
[10:20] <wgrant> You've not touched the config?
[10:20] <kb9vqf> Nope
[10:20] <wgrant> make run definitely normally starts it.
[10:21] <kb9vqf> That's what I thought too; here's what is in the config file right now:
[10:21] <kb9vqf> [librarian]
[10:21] <kb9vqf> download_url: http://launchpad.dev:58080/
[10:21] <wgrant> Hm, that's not the default, is it?
[10:22] <kb9vqf> Yes it is
[10:22] <wgrant> Mm, so it is.
[10:22] <kb9vqf> At least that's what came from Bazaar
[10:22] <kb9vqf> Should it be localhost?
[10:22] <wgrant> Well, if you want to make really sure that it doesn't work from any of your other buildds :P
[10:23] <kb9vqf> No thanks, I'll pass :-P
[10:23] <kb9vqf> It doesn't seem to be running at all; there is nothing with "lib" in ps aux
[10:24] <wgrant> It's a twistd with librarian.tac in argv.
[10:24] <wgrant> So try killing and restarting make run.
[10:24] <kb9vqf> That's what I'm doing now
[10:24] <wgrant> So.
[10:25]  * kb9vqf is reminded of Windows..."When acting strangely, simply reboot!"
[10:25] <wgrant> I guess you need to override upload_host, download_host, restricted_upload_host and restricted_download_host in the librarian section in your config file.
[10:25] <wgrant> As well as making it somehow not listen on localhost only.
[10:25] <wgrant> Or stick a proxy in front of it.
[10:26] <kb9vqf> The reboot fixed it
[10:26] <kb9vqf> And I was thinking of doing a proxy, it's three lines in Apache ;-)
[10:26] <noodles775> wgrant: no, well not specifically. There has been some changes since yesterday that reduce the db load, so that might be the reason you're seeing less timeouts.
[10:27] <kb9vqf> Can I run  i386 buildd on a 64-bit installation of Ubuntu?
[10:27] <wgrant> kb9vqf: Yes.
[10:27] <kb9vqf> Good
[10:27] <wgrant> You'll need to tweak the arch in /etc/launchpad-buildd/default, though.
[10:27] <wgrant> It defaults to the system arch.
[10:27] <kb9vqf> Yeah, I noticed that
[10:27] <kb9vqf> Wasn't sure if it was a read-only attribute so to speak
[10:28] <wgrant> You can also drop additional config files alongside 'default' to run multiple lp-buildds on one host.
[10:28] <wgrant> Handy for handling multiple archs.
[10:28] <wgrant> They cannot, however, share a filecache.
[10:29] <kb9vqf> Cool
[10:29] <kb9vqf> Now I can put those 8 cores to good use
[10:43] <kb9vqf> Things not to do...try to run all of the cron scripts in parallel at once
[10:43] <kb9vqf> :-{
[10:48] <wgrant> Um, yeah, bad idea.
[10:52] <kb9vqf> wgrant: FYI, I got a "canonical.launchpad.utilities.celebrities.MissingCelebrityError: ubuntu-bugzilla" from one of those scripts
[10:52] <kb9vqf> You might want to add it to the DB building script
[10:57] <wgrant> kb9vqf: Note that the DB bootstrap script has lots of comments in the celebrity list for stuff that isn't created yet.
[10:57] <wgrant> You really don't want to be running scripts unless you know what they do.
[10:58] <kb9vqf> I glanced through it and verified the database; just didn't look real hard at which celebrities were being created
[10:58] <kb9vqf> Since that is a bit of a black box to me right now
[10:58] <wgrant> Right. But you should be working out what the cronscripts do before you run them.
[10:58] <kb9vqf> That is true
[11:00] <wgrant> Ewwwwwwwwwwwwwwwww.
[11:00] <kb9vqf> The main problem was launching them all at once; I will probably have to create groups of them (e.g. the PPA updating scripts could run every minute, etc)
[11:00] <wgrant> I'd never dared to look at why ubuntu-bugzilla was a celebrity.
[11:00] <wgrant> I... regret discovering why.
[11:01] <kb9vqf> Do I want to know?
[11:01] <wgrant> Well, once upon a time, Ubuntu used Bugzilla. Then all the bugs were imported into LP, with bugwatches pointing at the old Bugzilla bugs.
[11:01] <wgrant> Now, normally bugwatches are updated every day or so.
[11:02] <wgrant> But this doesn't make sense for Ubuntu Bugzilla, since it doesn't exist any more.
[11:02] <wgrant> So.... there's a hack to skip updating bugwatches if their tracker's name is the same as the ubuntu-bugzilla celebrity's.
[11:02] <kb9vqf> Ewwww
[11:04] <ajmitch> almost as ewww as the old ubuntu bugzilla was
[11:07] <wgrant> It wasn't *that* bad.
[11:25] <kb9vqf> If /var/tmp/poppy does not exist, should I just create it?
[11:25] <wgrant> Yes.
[11:25] <wgrant> I thought that was documented. Hmm.
[11:26] <wgrant> Yeah, the first process-upload.py call in the "Upload a source to the PPA" section will create /var/tmp/poppy.
[11:26] <wgrant> Or should, at least.
[11:27] <kb9vqf> It gave me an error until I created the directory manually
[11:27] <wgrant> Hrmph.
[11:28] <kb9vqf> One other question...how does Launchpad handle Email sending?  Can I just point it to my SMTP server?
[11:30] <wgrant> Pretty much. But you'll need to make sure any email addresses in the code or DB are really what you want, and aren't going to spam people.
[11:30] <kb9vqf> OK.  And the configuration file for that is schema-lazr.conf or something else?
[11:30] <wgrant> You shouldn't alter schema-lazr.conf itself.
[11:30] <wgrant> You should probably create a new config, perhaps based on the development one.
[11:30] <wgrant> Overriding more things.
[11:31] <kb9vqf> Ah, OK
[11:31] <wgrant> If you look really hard you might be able to find the old production configs and use bits of them as examples ;)
[11:40] <kb9vqf> I need to catch some sleep...the build farm looks like it is active and cron jobs are partially set up...I'll do the actual build test tomorrow
[11:40] <kb9vqf> Thanks for all your help so far!
[11:50] <deryck> wgrant, ping.  still around?
[11:53] <wgrant> deryck: Sure.
[12:03] <wgrant> Hm, translation template jobs are on now?
[12:07] <henninge> wgrant: you mean if they are working?
[12:07] <wgrant> I haven't seen dozens on the farm before.
[12:07] <wgrant> But most of the i386 builders are doing them at the moment.
[12:09] <henninge> wgrant: well, they are restricted to run only on i386
[12:09] <wgrant> Well, yes, but I hadn't seen any on production for weeks.
[12:10] <wgrant> Also, are they restricted to only happen every $INTERVAL?
[12:10] <wgrant> Or will they run for every rev?
[12:10] <henninge> AFAIK they run on every rev.
[12:10] <henninge> jtv might be surer about the details.
[12:10] <wgrant> erm.
[12:10] <jtv> hi wgrant
[12:11] <wgrant> Evening jtv.
[12:11]  * henninge goes to lunch
[12:12] <jtv> wgrant: we still need a lot more logging and tracking for these jobs.  :/  One thing we found some time ago was that there's been a change in gettext that makes our front-end vetting think a lot of pure-gettext branches are intltool-based.
[12:13] <wgrant> jtv: Yeah, well, we have the infrastructure to keep track of them after they're done now.
[12:13] <wgrant> Once that's done, it probably becomes easier to impose quotas, too.
[12:14] <wgrant> Since you can't actually tell at the moment if an attempt has been made lately...
[12:14] <jtv> It'd be nice to have some form of bundling.
[12:14] <wgrant> Hm?
[12:14] <jtv> Where we don't record separate jobs for every revision, but the need to re-process a given branch.
[12:14] <wgrant> Ah.
[12:15] <jtv> If you get a dozen revisions in quick succession, there shouldn't be a need to run a dozen jobs.
[12:15] <wgrant> Right.
[12:15] <wgrant> That seems easy enough to do, even now.
[12:16] <jtv> Perhaps, yes.
[12:17] <wgrant> Is the porting of SPRBs and TTBJs to the new infrastructure going to happen in the foreseeable future?
[12:19] <jtv> Frankly, I don't know.  We're focused on something else at the moment.
[13:00] <bigjools> wgrant, jtv: I expect buildd admins will insist so that those jobs show up in the history
[13:00] <jtv> bigjools: it'd definitely make sense to do, yes.  And we want to do it... question is when.
[13:01] <wgrant> Plus it means we can delete a whole lot of code.
[13:01] <wgrant> And simplify a whole lot of other code.
[13:01] <wgrant> And make logging suck less.
[13:07] <bigjools> win win win
[13:08] <bigjools> henninge has a decision to make ;)
[13:09] <henninge> bigjools: I would love to if I knew about what ...
[13:09] <bigjools> henninge: "porting of SPRBs and TTBJs to the new infrastructure"
[13:09] <bigjools> TTBJs in your case
[13:09] <henninge> also, I am currently listening to a Microsoft general manager talking about "Microsoft and Opennes" ...
[13:10] <bigjools> Microsoft has always been open
[13:10] <bigjools> ...to making more money
[16:32] <Ursinha> Chex, gary_poster, rockstar, bigjools, henning, sinzui, gmb: production meeting on #launchpad-meeting @ Freenode in 30 minutes
[16:32] <Chex> Ursinha: thanks for reminder
[16:32] <rockstar> Ursinha, correction: 28 minutes.
[16:32] <gary_poster> thanks
[16:33] <rockstar> :)
[16:33] <Ursinha> rockstar, thanks for that :)
[16:36] <gmb> Ursinha, Thanks.
[17:00] <Ursinha> Chex, gary_poster, rockstar, bigjools, henning, sinzui, gmb: production meeting on #launchpad-meeting @ Freenode now
[19:27] <kb9vqf> Any idea why Launchpad won't accept my key import?
[19:27] <kb9vqf> "Launchpad could not import your OpenPGP key"
[19:44] <kb9vqf> Any idea why Launchpad won't accept my key import?
[19:44] <kb9vqf> "Launchpad could not import your OpenPGP key"
[19:52] <kb9vqf> This is weird... "format '1.0' is not permitted in lucid."
[19:52] <kb9vqf> What does that mean?
[19:52] <kb9vqf> It came from a PPA upload rejection notice
[21:03] <kb9vqf> I have sucessfully uploaded a package to my PPA on my local Launchpad installation and it is marked as published, but for some reason it won't start building
[21:03] <kb9vqf> It's stuck on "Needs building"
[21:03] <kb9vqf> Any thoughts?
[21:12] <maxb> Have you read wgrant's documentation on making a local soyuz work?
[21:13] <kb9vqf> Yes, I followed it
[21:13] <kb9vqf> It seems like is almost working, the build machines show up in /+builds as idle, I added both the i386 and amd64 chroot images
[21:14] <kb9vqf> But the build itself does not fire off
[21:14] <kb9vqf> maxb: is there anywhere I should look for logging info?
[21:15] <maxb> Not sure, sorry.
[21:15] <kb9vqf> maxb: If I haven't signed the code of conduct could it do thise?
[21:15] <kb9vqf> this?
[21:16] <maxb> huh. I'd expect it to refuse the upload
[21:17] <kb9vqf> maxb: I'd like to try signing it and see what happens, but I get "Launchpad could not import your OpenPGP key" when I try to add my key
[21:17] <kb9vqf> Any thoughts on that?
[21:17] <maxb> I'm sure wgrant documented how to sign the CoC
[21:19] <kb9vqf> maxb: Well I think I know why it accepted my upload; I ran the upload acceptor with absolutely-anything
[21:20] <kb9vqf> But I don't know if the CoC is checked again afterwards before the package is built
[21:26] <kb9vqf> So what piece of Launchpad actually dispatches the builds?
[21:45] <mars> lifeless, ping, is there any chance you could approve https://code.edge.launchpad.net/~tseaver/subunit/add_distutils_support/+merge/26653 ?
[21:45] <mars> lifeless, s/approve/review/
[21:46] <lifeless> yup its on my todo
[21:46] <mars> lifeless, cool, thank you.
[21:49] <lifeless> is this work relasted for you? its currently in my personal pile
[21:58] <lifeless> mars: ^
[21:58] <mars> lifeless, relasted?
[21:59] <mars> ah, related
[21:59] <lifeless> related
[21:59] <mars> lifeless, yes, it is blocking my landing of https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting/+merge/27086
[21:59] <lifeless> how so ?
[22:00] <mars> if you read Tres' comment, you'll see that he is waiting for PyPI subunit.
[22:00] <mars> When that is available, then zope.testing's setup.py and buildout.cfg can be updated with conditional subunit[test] dependencies
[22:02] <mars> And then my patch will be sane: you will need subunit to run the zope.testing test suite
[22:02] <lifeless> ah
[22:03] <lifeless> there is a test tools bug thats related
[22:05] <mars> as Tres put it, then we can "ensure that the [subunit] features don't bitrot"
[22:06] <mars> gary_poster, ec2 land did not eat my branch.  It just took an hour to deliver my change from ec2 to PQM :/
[22:06] <mars> It fell through a timerift on the way over or something
[22:08] <gary_poster> mars, well, time rifts are interesting, and a lock of eating the branch is a good thing :-)
[22:08] <gary_poster> lack
[22:31] <kb9vqf> So what piece of Launchpad actually dispatches the builds?
[22:32] <lifeless> sidnei: around ?
[22:33] <lifeless> tres has said he is ok with https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting/+merge/27086 landing as long as you or someone else with subunit tests it; mars has said it may break other builds right now as the additional tests he adds aren't conditional.
[22:33] <lifeless> mars: you're going to fix that right ?
[22:33] <mars> lifeless, yes, I certainly will now
[22:51] <sidnei> lifeless, im about to leave. i can land it in the morning.
[22:52] <lifeless> sidnei: cool
[22:52] <lifeless> thanks
[23:09] <thumper> rockstar: call now?
[23:09] <rockstar> thumper, sure.
[23:10] <thumper> rockstar: and I suppose we have to mumble?
[23:10] <rockstar> thumper, yes.... :(
[23:16] <jml> lifeless: fwiw, that thing I did to make a pretty graph has grown a script that makes similarly pretty graphs given a csv file like YYYY-MM-DD,number,number,number,...
[23:16] <lifeless> nice
[23:17] <jml> I guess I should productize it, or something.
[23:19] <lifeless> meh
[23:19] <lifeless> tweet it
[23:19] <lifeless> your job is done
[23:25] <jml> hah
[23:25] <ajmitch> yay for documentation about testing soyuz
[23:26] <jml> at least now it has a README
[23:26] <jml> good bye
[23:28] <lifeless> fly well
[23:51] <wgrant> ajmitch: What about it?
[23:52] <ajmitch> wgrant: that I'm glad it's there, it was helpful
[23:53] <wgrant> Ah, you've actually tried it?
[23:53] <ajmitch> yes
[23:53] <ajmitch> at least the accepting source packages & publishing them, not the building
[23:54] <ajmitch> so that I could test out this change to sync-source.py:
[23:54] <ajmitch> http://bazaar.launchpad.net/~ajmitch/launchpad/fakesyncs/revision/10992
[23:55] <wgrant> Ah, right.