[00:00] <wgrant> StevenK: I can has blessing?
[00:00] <StevenK> thumper: https://code.launchpad.net/~wgrant/launchpad/bug-707741-addFile-master/+merge/47606 please
[00:01] <StevenK> wgrant: I was getting there :-)
[00:01] <wgrant> StevenK: Great, thanks.
[00:01] <wgrant> So many regressions :(
[00:02] <lifeless> leonardr: is there some reason we can't have a destructor parameter? That doesn't seem to alter the boolean nature
[00:05] <wgrant> lifeless: Thanks.
[00:09] <wgrant> lifeless: Bugs needs a kanban UI.
[00:11] <wgrant> ... well, this is amusing.
[00:11] <wgrant> getUnlandedSourceBranchRevisions has a reasonably complex query, and it is completely untested.
[00:12] <wgrant> I expect that sort of thing from Soyuz, not Code :/
[00:24] <lifeless> poolie: https://pastebin.canonical.com/42398/
[00:27] <lifeless> -> allergy injection
[00:29]  * henninge eods
[00:36] <wgrant> StevenK: Another regression fix for you, if you have a moment: https://code.launchpad.net/~wgrant/launchpad/bug-707808-unlanded-revisions/+merge/47617
[00:38] <StevenK> lifeless, thumper: https://code.launchpad.net/~wgrant/launchpad/bug-707808-unlanded-revisions/+merge/47617 if you please
[00:58] <thumper> StevenK: ack
[00:59]  * StevenK kicks the merging code
[01:04] <wgrant> thumper: https://code.launchpad.net/~deon-wurley/phpldapadmin/1.2.x <- are Remote branches without URLs legal?
[01:04] <thumper> wgrant: yes...
[01:04] <thumper> wgrant: but sad
[01:04] <wgrant> WTF
[01:04] <wgrant> But OK.
[01:04] <thumper> wgrant: not much point really
[01:04] <Ursinha> wgrant, hi
[01:05] <wgrant> Ursinha: Hi, 'sup?
[01:05] <Ursinha> wgrant, so, there are bunches of ApacheLogParserError
[01:05] <wgrant> Ah, those.
[01:05] <wgrant> All fixed, but not deployed yet.
[01:05] <Ursinha> like this: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1851PPA1051
[01:05] <wgrant> Since germanium isn't in nodowntime.
[01:05] <Ursinha> hmm, I see.
[01:06] <Ursinha> wgrant, thanks :)
[01:07] <wgrant> I hope we can germanium in nodowntime soon :(
[01:07] <wgrant> +put
[01:08] <wgrant> thumper: Thanks.
[01:08] <thumper> np
[01:08] <wgrant> All major regressions fixed, yay.
[01:09] <StevenK> Now deploy them?
[01:09] <wgrant> Maybe in 8 or so hours.
[01:11] <StevenK> Longer, devel will hit testfix
[01:14] <wgrant> Heh.
[01:14] <wgrant> Oh fuck, you're serious.
[01:15] <wgrant> Gnrwgwnfjew
[01:15] <thumper> why
[01:15] <thumper> ?
[01:15] <wgrant> That spurious librarian thing.
[01:15] <thumper> oh FFS
[01:15] <wgrant> We should kill the current build.
[01:15] <thumper> yes
[01:15] <thumper> +1
[01:15] <wgrant> spm: Can we easily kill a buildbot build?
[01:15] <spm> yah. which one.
[01:15] <StevenK> Where easy is er, not
[01:15] <wgrant> If not, could you do it the hard way? :P
[01:15] <wgrant> lucid_lp
[01:15] <spm> where kill == stop.
[01:15] <StevenK> spm: 556 on lucid_lp
[01:17] <wgrant> I'm off to lunch now; could someone force a build once it's done?
[01:18] <StevenK> wgrant: Aye
[01:18] <StevenK> spm: Tell me when it stops hurting
[01:18] <spm> so the lsave is completely ded. how do we clean this up so it won't run that revno again?
[01:19] <StevenK> The revno is fine
[01:19] <StevenK> The errors are spurious and we don't want to be in testfix
[01:19] <spm> oh I see. perhaps getting the spurious nature of the errors fixed would be a GoodIdea™? ;-)
[01:19] <StevenK> Here, have a Hudson
[01:19] <StevenK> That should help
[01:20] <spm> :-)
[01:20] <StevenK> spm: Do you need to restart the master to get the slave back?
[01:20] <spm> seems to be showing back to me. should be able to force and magic happen
[01:21] <StevenK> It's buildbot, it isn't close to magic
[01:21] <StevenK> spm: Done, thanks
[01:21] <spm> anti-magic?
[01:21] <StevenK> wgrant: Build forced
[02:28]  * thumper stabs the librarian
[02:28] <thumper> AttributeError: 'NoneType' object has no attribute 'add_section'
[02:28] <thumper> while trying to set up the librarian in tests
[02:28] <thumper> anyone got any ideas?
[02:40] <lifeless> poolie: ok I'm back
[02:40] <lifeless> thumper: that means the layer which supplies the config to edit hasn't been setup
[02:41] <thumper> lifeless: which I'm getting why?
[02:41] <thumper> it has only just started happening
[02:41] <thumper> after merging dev
[02:41] <thumper> also
[02:41] <thumper> bin/kill-test-services is borked
[02:41] <lifeless> theres a bug on that
[02:41] <lifeless> with an explanation; fix should be trivial
[02:42] <poolie> lifeless, i think your mail is fine
[02:42] <poolie> i replied too
[02:45] <lifeless> poolie: well, I've just sent my mail
[02:46] <poolie> interesting new reply from max
[02:46] <thumper> lifeless: my librarian layer won't setup
[02:47] <wgrant> thumper: Turn LP_PERSISTENT_TEST_SERVICES off.
[02:47] <wgrant> It doesn't work any more.
[02:47] <thumper> ah
[02:47] <thumper> poo
[02:47] <lifeless> iz bug
[02:47] <lifeless> I put work in to make it work
[02:47] <wgrant> It's been that way for a week or so now.
[02:47] <lifeless> but its not actually tested
[02:47] <wgrant> Ah.
[02:47] <wgrant> I see.
[02:47] <lifeless> I didn't *intend* to break it
[02:47] <lifeless> but I couldn't tell that I *had*
[02:47] <lifeless> also
[02:47] <lifeless> its useless for parallel testing
[02:48] <wgrant> I presumed you turned a blind eye to your breakage of it.
[02:48] <wgrant> It is, yes. But we have no parallel testing yet.
[02:48] <lifeless> so I'm not very interested in maintaining it
[02:48] <lifeless> wgrant: we have some, a decent subset of tests work fine in parallel now
[02:48] <wgrant> Indeed.
[02:48] <wgrant> It is close.
[02:48] <wgrant> The main problem now is AppServerLayer.
[02:48] <lifeless> indeed
[02:48] <wgrant> Plus some directories that need randomising.
[02:49] <lifeless> yes
[02:49] <maxb> lifeless: You binged?
[02:49] <wgrant> 2.7 is close enough to be done by a maintenance squad, and I think parallel testing is probably almost there too.
[02:50] <lifeless> maxb: python-subunit 0.0.6 in the lp ppa
[02:50] <lifeless> maxb: part of fixing pqm to mail on conflicts
[02:50] <lifeless> maxb: however it looks like the losas can grab from the bzr ppa ok
[02:50] <lifeless> wgrant: feature work -> feature queue
[02:50] <lifeless> wgrant: if you're looking for something to do, I see a single bug causing 16K oopses a day.
[02:51] <lifeless> wgrant: (hint)
[02:51] <wgrant> Heh, no, I'm not looking for stuff to do :(
[02:51] <wgrant> That's the codebrowse connection closed one?
[02:51] <StevenK> lifeless: No fair if it's loggerhead
[02:51] <lifeless> wgrant: seriously, yes its close, but I guarantee it will have a long tail
[02:51] <lifeless> StevenK: loggerhead probably has more test coverage the LP proper
[02:52] <lifeless> wgrant: and the long tail is what makes me want to give it to a feature squad: get it live in buildbot/hudson, working in tarmac, profile it and assess scaling, disk IO utilisation, make recommendations on a new server to run massively parallel tests
[02:52] <lifeless> wgrant: figure out the damn db leak bug
[02:53] <wgrant> lifeless: I think that figuring that out is probably best achieved by deleting layers and zope.testrunner.
[02:53] <lifeless> wgrant: I have handwavy plans for that
[02:53] <lifeless> wgrant: its also pretty shallow
[02:53] <lifeless> but, time.
[02:53] <wgrant> Indeed.
[02:53] <lifeless> wgrant: focus will get us places!
[02:53] <StevenK> lifeless: I'd prefer if codebrowse actually looked like the rest of LP
[02:53] <lifeless> StevenK: so would codebrowse :P
[02:54] <lifeless> StevenK: it is themeable
[02:54] <StevenK> But I can recall you telling me that's pointless in Dallas
[02:54] <lifeless> StevenK: that doesn't sound like me; I think I may have said tha tthat is the least of the issues
[02:54] <wgrant> StevenK: Everything is pointless in Dallas.
[02:55] <lifeless> StevenK: which != pointless, but I could well have been confusing the issue
[02:55] <poolie> i might try just running tip on a big tree
[02:55] <poolie> and see how it does
[02:56] <lifeless> huwshimi: hi
[02:57] <huwshimi> lifeless: Hey there
[02:57] <lifeless> huwshimi: I dunno if you've seen https://dev.launchpad.net/BugTriage yet ?
[02:58] <huwshimi> lifeless: Uh, no. Are you referring to the bug I just filed?
[03:00] <lifeless> huwshimi: yes I am :)
[03:00] <lifeless> huwshimi: have a read :)
[03:00] <huwshimi> lifeless: Yeah thanks. Reading it now.
[03:01] <lifeless> huwshimi: its useful context to fit in with the team; the specific thing that alerted me was creating a medium importance bug (we don't use medium) and then starting work on it (thus ignoring the critical and high bugs)
[03:01] <lifeless> huwshimi: I think your particular focus means that looking at all critical/high bugs is irrelevant to you
[03:02] <lifeless> huwshimi: -but- you probably want to be sorting the design related bugs by the impact/importance you think they have
[03:03] <lifeless> huwshimi: anyhow, no drama; we have a massive learning curve and I primarily wanted to let you know about this part of it :)
[03:03] <huwshimi> lifeless: Sure thanks for letting me know. I'm about to submit a bunch of bugs so I'm glad you told me now :)
[03:14] <huwshimi> lifeless: There was some discussion last week about triaging bugs ourselves. Was there a decision made about that? Am I better off not self triaging for now?
[03:18] <lifeless> huwshimi: self triage is great
[03:19] <lifeless> follow the guidelines in the wiki page
[03:19] <lifeless> beyond that, if there are disagreements, english is a wonderful tool for figuring em out ;)
[03:19] <huwshimi> lifeless: OK thanks for that :)
[03:31]  * thumper grrs
[04:41]  * thumper is slowly getting there
[04:52] <StevenK> lifeless: So I don't think I can do the heavy lifting with SQL. If the recipe has no builds, http://pastebin.ubuntu.com/558848/ returns nothing, and so no daily builds get dispatched
[04:52] <poolie> how do i, through the api, find all bugs assigned to a particular team?
[04:52] <poolie> s/team/person
[04:53] <lifeless> StevenK: looking
[04:53] <lifeless> StevenK: you probably want either a LEFT OUTER JOIN or a UNION, or a COALESCE
[04:53] <lifeless> StevenK: probably a left outer join
[04:54] <StevenK> lifeless: Storm doesn't have a LeftOuterJoin?
[04:54] <thumper> hmm...
[04:54] <thumper> how do I emit a tag in a page template of a particular type?
[04:55] <lifeless> StevenK: it does, LeftJoin IIRC
[04:55] <lifeless> we use it all over
[04:55] <thumper> view/tag is something like "h1", or "span"
[04:55] <thumper> and I want to get <h1 id='${view/id}'> in the pt based on view/tag
[04:55]  * thumper is open to suggestions 
[04:55]  * StevenK is open to patches, never having done joins through Storm before
[04:55] <thumper> I'm thinking I may need an attribute and structure
[04:56] <lifeless> StevenK: grep for LeftJoin
[04:56] <lifeless> StevenK: I'm in the middle of a deep n meaningful over loggerhead
[04:56] <lifeless> its breaking my brain to think SQL at the same time
[04:57] <StevenK> Mission accomplished
[04:57]  * StevenK hides
[05:19] <StevenK> lifeless: If you want to point what I'm doing wrong in http://pastebin.ubuntu.com/558856/ , that would be awesome. I can't find any usage like this in the tree
[05:22]  * thumper is almost done refactoring widgets
[05:25] <lifeless> StevenK: wtf
[05:26] <StevenK> Clearly then, the answer is "everything"
[05:26] <lifeless> StevenK: the pattern is
[05:27] <lifeless> (IIRC)    LeftJoin(lefttable, righttable, condition)
[05:27] <lifeless> e.g.
[05:28] <lifeless> SourcePackageRecipe LEFT OUTER JOIN SourcePackageRecipeBuild on SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id
[05:28] <lifeless> ->
[05:28] <lifeless> LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id)
[05:28] <lifeless> the result of that is a table itself
[05:28] <lifeless> so to nest you'd do
[05:29] <lifeless> LeftJoin(LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id), PackageBuild, PackageBuild.id ==SourcePackageRecipeBuild.package_build_id)
[05:29] <lifeless> but you don't need a nested left join
[05:29] <lifeless> you only need one outer join - at the point you're willing to have a NULL row
[05:29] <lifeless> so
[05:29] <lifeless> Join(LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id), PackageBuild, PackageBuild.id ==SourcePackageRecipeBuild.package_build_id)
[05:29] <lifeless> etc
[05:30] <lifeless> Join(Join(LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id), PackageBuild, PackageBuild.id ==SourcePackageRecipeBuild.package_build_id), BuildFarmJob, BuildFarmJob.id == PackageBuild.build_farm_job_id)
[05:30] <lifeless> is your using object
[05:30] <lifeless> then in the where clause you put
[05:30] <lifeless> Or(BuildFarmJob.id == None, BuildFarmJob.date_created > one_day_ago)
[05:35] <thumper> w00t w00t
[05:35] <lifeless> StevenK: does that make sense?
[05:35] <lifeless> StevenK: use LP_DEBUG_SQL to see the sql being emitted
[05:35] <lifeless> StevenK: and adjust until you have a query you're happy with
[05:36] <StevenK> Yes, it made sense
[05:36] <lifeless> ok
[05:36] <lifeless> did it help ?
[05:40] <StevenK> lifeless: Not really, it still doesn't return SPRecipes that haven't built
[05:41] <StevenK> I suspect SourcePackageRecipeBuild.id == None will help, since they are created for builds, which then creates the PackageBuild and BFJ
[05:43] <lifeless> StevenK: you probably want to bring back the sprb as well
[05:44] <lifeless> since that will tell you the when. Or perhaps not.
[06:03]  * thumper EODs
[06:08]  * huwshimi waves goodbye
[06:21] <lifeless> StevenK: need more help? I have cycles in ~ 10
[06:42] <StevenK> lifeless: Sorry, I was picking up Sarah, and I EOD'd 1.6 hours ago
[06:44] <lifeless> kk
[06:44] <lifeless> grab me tomorrow
[06:45] <lifeless> and we can nut it out
[07:12] <StevenK> lifeless: Was my plan
[09:19] <adeuring> good morning
[09:22] <mrevell> Guten morgen
[11:35] <gmb> Does anyone know what in the sweet hell this is about?:
[11:35] <gmb> An error occurred during a connection to launchpad.dev.
[11:35] <gmb> SSL received a record that exceeded the maximum permissible length.
[11:35] <gmb> (Error code: ssl_error_rx_record_too_long)
[11:42] <wgrant> gmb: This is on a fresh LP install?
[11:45] <gmb> wgrant: No, it's an old one. But I did have to re-run rocketfuel-setup recently to correct a couple of pebkacs.
[11:47] <Ursinha> good morning launchpad
[11:47]  * gmb re-does the apache config dance
[11:49] <wgrant> gmb: Odd. I normally only see that when it's a fresh install needing an Apache restart, or I've broken the vhost config somehow.
[11:49] <gmb> wgrant: Yeah. I think I've borked the vhost config in some myserious way (probably because I have a remote access setup and rocketfuel-setup fought with it).
[11:49] <wgrant> Ah.
[11:49] <gmb> I'm redoing it now.
[11:50] <maxb> I find the nicest way to manage a remote access thingy is to let rocketfuel-setup manage the local-launchpad config file, but a2dissite it, copy it, and modify the copy
[11:54] <gmb> maxb: Very wise. I shall do that henceforth.
[11:55] <gmb> Hoorah. It works.
[11:58] <gmb> /me lunches
[12:06] <deryck> Morning, all.
[12:08] <Ursinha> morning deryck
[12:20] <salgado> what's the best thing to use when writing a setup.py these days?  setuptools?  distribute?
[12:24] <jelmer> salgado: what sort of features do you need from it?
[12:24] <jelmer> distutils is widely available and seems to work pretty well for basic python modules
[12:29] <salgado> jelmer, our needs seem to be rather basic, so maybe distutils will be enough indeed
[12:31] <jml> salgado: I copy an existing setup.py :)
[12:45] <salgado> jml, that's always a good strategy
[13:01] <jml> can someone give me a dumb manager's summary of the codebrowse discussion on the mailing list?
[14:13] <salgado> benji, your last message to that mp I reviewed yesterday had only some quoted text from my previous message
[14:14] <benji> salgado: hmm, that's odd.  I just said "sorry about that, thanks for asking Curtis to look at it"
[14:15] <salgado> benji, looks like the issue I encountered yesterday; already reported a bug for it
[14:16] <benji> do you have the bug number at hand?
[14:18] <abentley> jelmer, ping
[14:18] <abentley> jelmer, unping
[14:18] <oyv> hi
[14:19] <oyv> i'm trying to setup launchpad on a virtual machine, but when i try make run i get an importerror (no module named loom.branch)
[14:19] <oyv> anybody got any hints?
[14:20] <oyv> http://pastebin.com/JBn59ekF
[14:22] <abentley> flacoste, now that domain expertise is spread across squads, maybe it would be good to have a list of domain experts, e.g. on the wiki?
[14:25] <oyv> (i can import the branch in python shells, so it should be installed and everything)
[14:28] <jcsackett> oyv: are you following the instructions on the wiki?
[14:29] <oyv> i use isntructions from here: https://dev.launchpad.net/Getting
[14:29] <oyv> (and the "running" page)
[14:30] <jcsackett> oyv, ok. what version of ubuntu are you running?
[14:32] <oyv> Ubuntu 10.04.1 LTS (lucid lynx)
[14:34] <jcsackett> oyv: i encountered this at one point; i am trying to remember how i fixed it.
[14:38] <jcsackett> oyv: i assume you are not actually using bzr loom? it shouldn't fail in that case, i'm just defining what's going on.
[14:39] <oyv> i don't think i'm using it, not really sure what it is ;)
[14:40] <jcsackett> it's a plugin for bzr, http://doc.bazaar.canonical.com/plugins/en/loom-plugin.html. many lp developers use it, but last i checked it was not a dependency.
[14:41] <jcsackett> and if it is, it should have been automatically installed when you hit the update step in the get/run instructions.
[14:41] <jml> jcsackett: it is a dep
[14:41] <jcsackett> jml: really? did that change? when i started on lp i was told it wasn't, b/c i was hitting this (or a similar) issue.
[14:42] <oyv> edb@launchpad:~/launchpad/lp-branches/devel$ bzr plugins

[14:42] <oyv> loom 2.1.0 Loom is a bzr plugin which adds new commands to manage a loom of patches.
[14:42]  * jcsackett supposes he could have been deasily misinformed.
[14:42] <jml> jcsackett: since 2008, I think. it's needed to process looms
[14:42] <oyv> doesn't that mean it's installed?
[14:43] <benji> either is fine with me; but more importantly it means I need to go pour my coffee right now
[14:43] <benji> pfft, wrong chan
[14:45] <jcsackett> jml: dig. clearly i was misinformed. :-P
[14:51] <jcsackett> oyv: sorry, i'm not sure what might be going on. it does seem that it is installed as a plugin.
[14:52] <jcsackett> i could be of more help if i had a functioning machine right now, but i'm rebuilding after hardware failure yesterday, so i can't explore much on my end. :-/
[14:53] <oyv> ok, thanks anyway :)
[14:55] <LPCIBot> Project devel build (396): FAILURE in 4 hr 38 min: https://hudson.wedontsleep.org/job/devel/396/
[14:55] <LPCIBot> Launchpad Patch Queue Manager: [r=lifeless,
[14:55] <LPCIBot> stevenk][bug=707741] Fix LibrarianClient.addFile to function under
[14:55] <LPCIBot> SlaveDatabasePolicy.
[14:55] <maxb> oyv: I don't suppose you get a traceback related to that ImportError?
[14:56] <oyv> http://pastebin.com/JBn59ekF
[14:56] <maxb> When you're running "make run", the copy of bzr-loom that is supposed to be being used is the one found via bzrplugins/loom/ in the Launchpad tree
[14:56] <salgado> benji, it's bug 708258
[14:56] <_mup_> Bug #708258: Failed to parse merge proposal comment sent via email <code-review> <email> <Launchpad itself:Triaged> < https://launchpad.net/bugs/708258 >
[14:56] <benji> salgado: thanks
[14:58] <oyv> hmm
[14:59]  * maxb sobs a bit as launchpad-database-setup tries to configure pg8.2
[14:59] <oyv> there's only one folder in the bzrplugin folder, lpserve
[14:59] <maxb> Your tree is broken then
[14:59] <oyv> damnit..
[14:59] <oyv> ;)
[14:59] <oyv> thanks
[14:59] <flacoste> abentley: that's what the wiki pages related to the Launchpad in 30 minutes presentation was meant to do
[15:00] <flacoste> abentley: https://dev.launchpad.net/Foundations/ComponentHelp
[15:00] <flacoste> i think Tim did something similar
[15:00] <flacoste> and so did Bugs
[15:01] <flacoste> not sure if Soyuz and Translations put their stuff in that format
[15:02] <abentley> flacoste, Ah, didn't think of looking there.
[15:02] <deryck> ah, adeuring, call time.  Sorry got distracted running tests and answering emails.
[15:02] <flacoste> abentley: it would probably make sense to collate all the content in one place, or at least make an easy-to-find index page to it
[15:02] <adeuring> deryck: ok, no problem
[15:02] <bigjools> goooooood morning
[15:02] <abentley> flacoste, +1
[15:07] <abentley> bigjools, can you confirm that we never send emails for successful binary builds?  Because there are some tests of Build.notify with BuildStatus.FULLYBUILT.
[15:09] <bigjools> abentley: I can confirm that
[15:09] <bigjools> that test sounds a bit bong
[15:09] <abentley> bigjools, great.
[15:11] <deryck> henninge, I need 5 minutes to wrap up some notes, and then we can chat.   if that works for you.
[15:11] <henninge> deryck: that's fine
[15:12] <abentley> bigjools, since recipe builds want to notify on success, I'm moving the differentiation into BinaryPackageBuild.notify/SourcePackageRecipeBuild.notify, and fixing the tests.
[15:12] <bigjools> abentley: you *want* them to notify on success or you're removing that?
[15:12] <abentley> bigjools, I *want* them to notify on success.
[15:13] <bigjools> ok
[15:13] <james_w> really?
[15:13] <bigjools> sounds odd to be
[15:13] <bigjools> me
[15:13] <james_w> will the coalesce?
[15:13] <james_w> they
[15:13] <abentley> james_w, yes.  How else do I know that a recipe build has happened?
[15:13] <bigjools> prepare for wrath if you do this!
[15:13] <abentley> bigjools, we've been doing this from the start.
[15:14] <bigjools> interesting
[15:14] <james_w> because they happen every day? because Launchpad is reliable and does what I ask?
[15:14] <bigjools> we really need a better email notification story
[15:15] <abentley> james_w, no, they only happen when the source changes.
[15:15] <abentley> james_w, my bzr recipe triggers once or twice a month.
[15:15] <james_w> are you /really/ going to send people 200 emails every day in the extreme case?
[15:15] <james_w> how is that useful?
[15:15] <abentley> james_w, it's useful so that people know when there's a new build.
[15:16] <james_w> but that information isn't very useful at all as you move towards heavy users
[15:16] <abentley> james_w, if you get it and you don't want it, you can filter it out.  If you don't get it and you want it, you have no option.
[15:16] <james_w> do you really want to discourage heavy use of the service using email volume
[15:17] <allenap> gmb: Do you have time to do a sanity check on https://code.launchpad.net/~allenap/launchpad/freedesktop-importance-flip-flop-bug-707478/+merge/47667 please?
[15:17] <gmb> allenap: Sure.
[15:17] <james_w> client-side filtering has been deemed to be not sufficient in the bugs case
[15:19] <abentley> james_w, client-side filtering is better than no mail.
[15:19] <james_w> are you sure your users would agree with that?
[15:19] <abentley> james_w, I am sure you can find users to disagree with anything.
[15:20] <james_w> well, let's all go shopping then
[15:22] <gmb> allenap: Approved.
[15:22] <abentley> james_w, I am just fixing a bug where the wrong kind of notification is sent, not increasing the number of notifications.
[15:23] <allenap> gmb: Thank you :)
[15:23] <james_w> ok, then you are absolved from any responsibility for the system
[15:26] <abentley> james_w, so I've been talking with jelmer about this, and we both agree that there's a lot of room for improvement in the build notification story.
[15:27] <abentley> james_w, jelmer would like to see a notification of successful binary builds, grouped so that you only get one email for multiple builds.
[15:28] <abentley> james_w, and then we would be able to omit the success emails for recipe builds.
[15:29] <bigjools> all emails that LP sends should be controllable from a single page
[15:29] <bigjools> per person, I mean
[15:30] <abentley> bigjools, I don't know what to think about that.  It would be a verrrry long page.
[15:30] <bigjools> potentially but not always
[15:31] <bigjools> it can be ajaxified but you know what I mean - the subject of being sent machine-generated email is very divisive
[15:35] <abentley> bigjools, generally, asking users to make choices is bad.  The fewer choices, the smoother something feels.  This makes me think that the need for such a page indicates a design problem.
[15:40] <henninge> allenap: thanks for the review!
[15:40] <bigjools> abentley: that's the Gnone way, which is completely disagree with
[15:40] <abentley> bigjools, I'm not disputing the actual need, but I do wonder whether we're not thinking big enough.
[15:40] <bigjools> s/is/I/
[15:41] <henninge> adeuring: did you see my review of your branch?
[15:41] <bigjools> anyway, I'm not bikeshedding over this
[15:42] <adeuring> henninge: yes, thanks. Actually, I think we should keep the assert() calls in setUp() because it is so esay to cause a mess there, and I am not sure if this would always result in test failures
[15:43] <henninge> adeuring: well, I think that is true for a lot of places in the code, though.
[15:44] <henninge> adeuring: but I don't really mind leaving them in there.
[15:44] <adeuring> henninge: OK; perhaps my concerns are related to my unfamiliarity with translations ;)
[15:45] <henninge> adeuring: ... I tried not to be so blunt ;-)
[15:45] <henninge> adeuring: I have done that before, added asserts to make sure I got things right but once it worked, I removed them.
[15:45] <adeuring> henninge: ;) but nevertheless, if we can screw the setup just by exchanging two factory calls, I think it is worth to check we don't do that...
[15:46] <henninge> adeuring: that's why I suggested leaving just the one in there.
[15:46] <henninge> and a comment
[15:47] <henninge> "# The order of creation is important here to make sure is_current_ubuntu is set to False.
[15:47] <henninge> self.assertFalse(message.is_current_ubuntu)"
[15:48] <henninge> adeuring: ^ like that, only use the right variable for 'message'.
[16:01] <allenap> gmb: Do you remember why sourceforge.net bug watches are disabled?
[16:02] <gmb> allenap: Because our super-duper sourceforge slurping screenscraping fantastical disaster rather relied on them never, ever changing their templates, and they did.
[16:02] <gmb> (I might have overstated how good that code actually was)
[16:09] <allenap> gmb: Oh yes, thanks. I've just had a look at their templates and they're very much simpler now. Might fix that bad boy.
[16:09] <gmb> allenap: Don't they offer an API now?
[16:10] <allenap> gmb: Do they? Awesome.
[16:10] <gmb> ISTR they do. Maybe I saw that on the ForgePlucker mailing list.
[16:14] <abentley> allenap, jcsackett could you please review https://code.launchpad.net/~abentley/launchpad/build-mail3/+merge/47679 ?  It's long, but only because of indentation fixes in a doctest.
[16:14] <jcsackett> abentley: sure.
[16:15] <abentley> jcsackett, thanks.
[16:15] <allenap> gmb: I can't find an API, but tell me if you happen upon it.
[16:16] <gmb> allenap: Yeah, I'm not able to find one either, disappointingly. Must have been a happy dream.
[16:16] <gmb> I need new dreams.
[16:16] <allenap> Hehe, you do :)
[16:30] <abentley> deryck, I feel like I lack the domain knowledge to dive into any of the tasks on the Kanban board.  Can you suggest one?
[16:30]  * deryck is looking
[16:31] <deryck> abentley, so I'm sure I lack the domain knowledge too :-)  However, the card for bug 696009 seems a  nice next step.....
[16:31] <_mup_> Bug #696009: Provide ITranslationMessage.shareIfPossible unit tests <tech-debt> <upstream-translations-sharing> <Launchpad itself:Triaged> < https://launchpad.net/bugs/696009 >
[16:32] <deryck> abentley, it's still in the area of test clean up.  so will broaden domain knowledge as cleanup happens.
[16:33] <abentley> deryck, okay, I'll tak that.
[16:33] <abentley> s/tak/take.
[16:33] <deryck> ok, cool
[16:35] <henninge> adeuring: am I to expect a new revision on your branch? Otherwise I'll start landing mine (which includes yours and abentley's)
[16:35] <henninge> actually, I'll make sure to merge the latest first
[16:38] <adeuring> henninge: well, I'd prefer to leave the tests as they are... so, no new revesion ;)
[16:38] <henninge> adeuring: ok, thanks. ;)
[16:46] <jcsackett> abentley: comments on your MP. feel free to answer there or here.
[16:46] <abentley> jcsackett, fire away.
[16:47] <jcsackett> abentley: as commented on the MP, i'm wondering about the use of self.builds in the from_build method.
[16:48] <jcsackett> mainly, in the first hit, can that get big enough to cause performance issues?
[16:48] <abentley> jcsackett, it is always one or zero hits.
[16:49] <jcsackett> abentley: so does the cache help? the cache is always gone at the end of the transaction, yes?
[16:50] <gmb> I broke the build; rolling it back now. Sorry folks.
[16:50] <abentley> jcsackett, the cache will not help with the cases we have at hand, because we only use it once.
[16:50] <jcsackett> so putting it in cached_property is forward looking?
[16:51] <abentley> jcsackett, but I felt that it made sense to cache it since one does not expect an attribute to be expensive.
[16:51] <gmb> Oh, nice. I specified --rollback for lp-land but it doesn't seem to have tagged the commit message thus.
[16:51]  * gmb files a bug.
[16:53] <abentley> jcsackett, you could say putting it in cached_property is forward-looking.
[16:53] <jcsackett> abentley: so do you see any problem with the use of self.builds being called in there? since no preloading will happen, can hitting self.builds get sufficiently expensive to be a worry?
[16:54] <jcsackett> i'm not saying it will. i'm not familiar with this part of the codebase, so i'm wondering.
[16:54] <abentley> jcsackett, I don't know what you're asking.  Are you wanting me to run an EXPLAIN on the query?
[16:55] <jcsackett> abentley: i'm just double checking performance concerns.
[16:55] <abentley> jcsackett, what kind of answer are you looking for?
[16:57] <jcsackett> abentley: builds is a SQLMultipleJoin; i'm wondering if when self.build gets called we may return a huge rowset in some cases.
[16:57] <abentley> jcsackett, as I said, it's always one or zero results.
[16:57] <jcsackett> abenltey, ah! i thought you meant from_build was called one or zero times.
[16:57] <abentley> jcsackett, there is a comment in the code saying it shouldn't be a multiple join.
[16:59] <jcsackett> abentley: yes, i see now.
[16:59] <jcsackett> apologies for the confusion.
[16:59] <abentley> jcsackett, cool
[17:01] <jcsackett> abentley: r=me. i need follow up, as a mentee :-P. bac, can you follow up on https://code.launchpad.net/~abentley/launchpad/build-mail3/+merge/47679?
[17:14] <abentley> jcsackett, I've added a comment per your suggestion.
[17:19] <jcsackett> abentley: thank you. :-)
[17:43] <deryck> hurrah!  Windmill re-enable branch finally made it through.
[18:26] <lifeless> jml: morning
[18:26] <lifeless> deryck: cool
[18:26] <jml> lifeless: hi
[18:27] <lifeless> jml: would you like a catchup call - we seem to be trading 1-liners this week
[18:27] <jml> lifeless: Yes, I'd like one. Only have 15mins though.
[18:28] <lifeless> skype?
[18:28] <jml> lifeless: sure.
[18:46] <thumper> flacoste: ping
[18:59] <bac> hi abentley
[19:00] <abentley> bac, hi.
[19:00] <bac> abentley: i'm trying to mentor jcsackett's review.  the diff in the MP is overwhelmed by lint changes.  i've tried to get a diff with just the non-lint changes but failed.  could you easily whip one up and paste it?
[19:02] <abentley> bac, not sure.  I don't think they were separate commits, or anything.
[19:04] <abentley> bac, how's this? http://pastebin.ubuntu.com/559153/
[19:05] <bac> abentley: 228 is much better than 1654!  :)  thanks!
[19:08] <aroman> hello, all! Is there any way of seeing how many people are using a PPA of mine/a package? Or even just seeing how many people have branched from bzr or downloaded a .tar.gz? Thanks!
[19:13] <lifeless> aroman: there is an LP API that can get you download statistics for a PPA
[19:14] <flacoste> hi thumper
[19:14] <aroman> lifeless: ah, but there's no sort of graphical frontend to see that information?
[19:14] <lifeless> not yet
[19:14] <thumper> flacoste: can you talk briefly about your review?
[19:14] <aroman> lifeless: ah, alright, well i'll check that out then. I assume it's python, right?
[19:14] <lifeless> flacoste: hi; can I get a small timeslice today ? (but not for ~30)
[19:14] <flacoste> thumper: sure, mumble?
[19:14] <thumper> ok
[19:14] <lifeless> aroman: its a RESTful api, but we have a python library you can use
[19:15] <flacoste> lifeless: how small?
[19:15] <lifeless> flacoste: 10 min ?
[19:15] <aroman> lifeless: excellent, thanks!
[19:15] <lifeless> flacoste: crossing t's dotting i's on the loggerhead thing
[19:27] <LPCIBot> Yippie, build fixed!
[19:27] <LPCIBot> Project devel build (397): FIXED in 4 hr 31 min: https://hudson.wedontsleep.org/job/devel/397/
[19:27] <LPCIBot> Launchpad Patch Queue Manager: [r=gary][ui=none][bug=704685] BugSubscription.bug_notification_level
[19:27] <LPCIBot> is now exposed in the devel webservice API.
[19:27] <gary_poster> yay gmb !
[19:28] <gary_poster> oh, boo
[19:28] <gary_poster> that's hudson
[19:28] <gary_poster> on buildbot, it is being rolled back :-/
[19:28] <lifeless> oh, why?
[19:29] <gary_poster> https://lpbuildbot.canonical.com/builders/lucid_lp/builds/561/steps/shell_6/logs/summary
[19:30] <lifeless> ah scaling test failed
[19:30] <lifeless> I'd like to make those more reliable and isolated
[19:30] <lifeless> they are just a little flaky atm
[19:30] <lifeless> (different standalone vs in a larger run)
[19:31] <gary_poster> yeah
[19:31] <lifeless> I think they are still a net win
[19:31] <lifeless> but it can be frustrating getting them going
[19:38] <lifeless> allenap: around?
[19:58] <jtv> bigjools: got a minute for a question about FTPArchiveHandler?
[19:58] <bigjools> jtv: yup
[19:59] <jtv> great
[19:59] <jtv> I was just wondering… it uses the LoopTuner all over the place already, but only for gc.
[19:59] <jtv> Was it originally meant to commit transactions as well?
[20:00] <bigjools> no, it's writing out files for apt-ftparchive to use
[20:01] <jtv> Okay, but it's not writing to the DB that I can see and it's a big suspect in the long-transaction crime mystery I'm pursuing.
[20:02] <jtv> So it'd seem that a few commits would be a relatively harmless way to get around the timeouts.
[20:03] <jtv> I was thinking perhaps I could force the main body of work on the slave store, and throw in a few commits.
[20:03]  * bigjools thinks
[20:03] <bigjools> that code is probably the bit I understand the least in soyuz
[20:03] <bigjools> it's never gone wrong since cprov left so I never had to look at it :)
[20:05] <bigjools> jtv: I'm a bit scared of using the slave store in case of replication delays
[20:06] <jtv> Scared is sensible.  But what's there to replicate?
[20:06] <bigjools> all the publishing data it relies on, which is fast-changing
[20:07] <jtv> I see.
[20:08] <bigjools> jtv: it would be a good start to work out if it really is keeping a transaction open
[20:08] <bigjools> if it's not writing to the db can we change the $mumble
[20:08] <jtv> It has to be.  It's not committing anywhere, and it's looping over lots of packages AFAICS.
[20:08] <jtv> The $mumble?
[20:09] <bigjools> yeah, the thing
[20:09] <jtv> Oh, the thing.
[20:09] <jtv> The database policy?
[20:09] <bigjools> isloation?
[20:09] <bigjools> isolation ,even
[20:09] <jtv> Isolation level?
[20:09] <bigjools> dunno if that would affect it
[20:09] <jtv> That would affect reads, not too much to do with writes.
[20:09] <bigjools> can we connection r/o to the master?
[20:09] <bigjools> sigh
[20:10] <bigjools> connect*
[20:10] <jtv> I don't think we can, no.
[20:10] <jtv> Something we could try though is run a test with a slave-only policy, and see what fails.
[20:11] <bigjools> ok
[20:11] <bigjools> I see why you suggested the slave now
[20:11] <allenap> lifeless: Hi.
[20:11] <allenap> lifeless: Is this about the BugMessage crack?
[20:11] <jtv> bigjools: Well it also helps scale-out and locking, of course.
[20:11] <lifeless> yes
[20:12] <lifeless> allenap: though I'm not sure its crack L)
[20:12] <jtv> bigjools: unless there's something that changes the database and then immediately runs the script, we should get about as consistent a view of the database from a slave as we do from the master—just slightly older.
[20:13] <allenap> I'm reading the code now to try and remind myself of the weirdnesses.
[20:13] <lifeless> allenap: so
[20:13] <lifeless> we decorate Message to make it an IndexedMessage but the actual relation is BugMessage which we don't expose at all
[20:13] <flacoste> thumper: i'm back
[20:14] <lifeless> thats beside the point though
[20:14] <jtv> bigjools: "the thing"—https://www.ubersoft.net/comic/hd/2011/01/next-time-try-index-cards
[20:14] <thumper> flacoste: want to continue?
[20:14] <flacoste> thumper: sure thing
[20:14] <lifeless> we can still decorate but get the indices from the db once they are cached
[20:15] <bigjools> jtv: the thing I am scared of is writing out inconsistent indices in the Ubuntu archive.  That would be fairly nasty.
[20:15] <bigjools> now while I am scared I am not sure how realistic the chances of that are
[20:16] <jtv> In that case, we could try moving to the slave and _not_ committing.
[20:17] <jtv> If that works out we'd still get a single huge transaction, but also a guarantee that it takes no write locks.
[20:17] <allenap> lifeless: Yeah, that sounds sane.
[20:17] <bigjools> jtv: well publish-distro does do commits
[20:17] <jtv> Yes, just not in its huge loops.
[20:17] <bigjools> after a-f runs
[20:17] <bigjools> it needs to mark stuff as publishe
[20:17] <bigjools> d
[20:18] <lifeless> allenap: the goal is to be able to do a range query on BugMessage and then get just the Messages we want
[20:18] <jtv> bigjools: I guess that happens all the way at the end?
[20:18] <lifeless> allenap: which will also let us move to ajax population of pages like bug 1
[20:18] <_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
[20:18] <allenap> lifeless: Judging by Bug._indexed_messages, that should work well.
[20:19] <lifeless> allenap: indeed; I wrote Bug._indexed_messages in october or so as part of optimising within the prior schema
[20:19] <allenap> lifeless: Like load-on-scroll?
[20:19] <allenap> lifeless: Yeah, it looked like one of yours :)
[20:19] <lifeless> allenap: hah! I'll take that as a complement.
[20:19] <lifeless> allenap: and yes, load on scroll
[20:19] <jtv> bigjools: if we did the actual work on the slave but the marking-as-published on the master, we'd risk marking a slightly newer version as published than we actually published.  Is that a problem?
[20:19] <allenap> Awesome.
[20:20] <lifeless> allenap: should be easy if we can get one message + adjacent actions pretty cheaply.
[20:20] <bigjools> jtv: there's 4 steps in the publisher: 1) write files to pool, 2) domination, 3) generate files for a-f and run it, 4) write release files
[20:20] <bigjools> jtv: big problem ,yes
[20:20] <allenap> lifeless: Adjacent actions?
[20:21] <jtv> bigjools: though strictly speaking, I would expect that that problem already exists to about the same extent
[20:21] <lifeless> allenap: look at a BugTask:+index page
[20:21] <lifeless> allenap: it shows action action message action message etc interleaved
[20:21] <lifeless> allenap: it pretends there is one sequence
[20:22] <bigjools> jtv: actually, the publishing record would only get written by  this process anyway at this point in its lifecycle
[20:22] <allenap> lifeless: Ah, yes, I wrote a lot of that ;) I was having an association fail on the word "actions".
[20:22] <allenap> Well, not a lot, some.
[20:22] <jtv> bigjools: assuming there's no overlapping runs, I guess it'd only be a problem if there were a few last changes while the script was running, and then nothing before the next run (so it'd think it was already published or something).
[20:23] <allenap> lifeless: Are you going to work on this, or are you softening me up to do it?
[20:23] <flacoste> lifeless: i'm free whenever you are
[20:23] <lifeless> allenap: I'm helping on the oops/performance aspect
[20:24] <lifeless> allenap: future stuff will be feature cards I suspect, unless you were to have a sudden fit of zomg I want to do this
[20:24] <lifeless> flacoste: voice?
[20:24] <flacoste> lifeless: sure
[20:24] <allenap> lifeless: Okay, so you'll get the index into the database, and the get-adjacent-actions and load-on-scroll stuff is up to others?
[20:25] <flacoste> lifeless: skype me
[20:25] <lifeless> allenap: yeah, I don't have long enough timeslices to do larger stuff
[20:26] <allenap> lifeless: That's cool, I just wanted to be sure what's expected of me.
[20:27] <lifeless> be excellent
[20:27] <lifeless> thats it
[20:28] <allenap> Heh :)
[20:30] <bigjools> jtv: so it marks everything published before a-f runs
[20:30] <bigjools> there's the long transaction
[20:30] <jtv> bigjools: it may help my understanding if I see what that entails… do you happen to know where that is done?
[20:31] <bigjools> jtv: look at lib/lp/archivepublisher/publishing.py
[20:31]  * jtv looks at lib/lp/archivepublisher/publishing.py
[20:31] <bigjools> and scripts/publish-distro.py which is what calls it
[20:32] <bigjools> C_doFTPArchive is for Ubuntu, C_writeIndexes is for PPAs
[20:32] <jtv> Ah, that helps
[20:32] <jtv> So the marking-as-published… what do I look for?
[20:32] <bigjools> ArchivePublisherBase.publish()
[20:33] <bigjools> follow it down to there from distroseries.publish()
[20:33] <jtv> ah cool
[20:33] <jtv> But I think the script commits soon after that happens anyway.
[20:33] <bigjools> the latter being called from A_publish()
[20:34] <jtv> The problem is in C_doFTPArchive
[20:34] <jtv> (I think)
[20:34] <bigjools> yes
[20:34] <bigjools> each stage is wrapped it try_and_commit()
[20:34] <bigjools> in*
[20:37] <jtv> ahh got it: setPublished
[20:38] <jtv> And we want to publish a state that's no older than that datepublished timestamp, right?
[20:40] <bigjools> jtv: so my concern is that we only see half of the records that were just set published if C_doFTPArchive is using a different store
[20:41] <bigjools> or maybe even none
[20:41] <bigjools> but that's extreme
[20:43] <jtv> I wonder if we could set "now minus replication lag" as the publication date.
[20:43] <bac> hi deryck, you still around?
[20:43] <bigjools> the date is not a concern
[20:44] <bigjools> it's the inconsistency
[20:45] <jtv> bigjools: would that change though if we otherwise ran everything on the slave store?
[20:45] <bigjools> because what can happen is that we write the pool file, miss it in a-f and then write the release file with it in
[20:45] <bigjools> that would be quite disastrous
[20:45] <jtv> What's a-f stand for by the way?
[20:45] <bigjools> Apt FTPArchive
[20:46] <bigjools> in that case we'd end up with checksums that are incorrect
[20:46] <jtv> You're saying the whole pool file might be lost?  Or an individual package that's in there?
[20:46] <bigjools> no
[20:47] <bigjools> a-f writes the indexes - with replication lag it could miss something
[20:47] <bigjools> then if that something replicates before we get to stage "D" where it writes the Release file, the Release file will be wrong compared to the index
[20:48] <bigjools> I think
[20:48] <jtv> Well if it's that complicated I probably can't afford to mess with it anyway.  :/
[20:48] <bigjools> the publisher is *the* most critical part of soyuz, if it goes wrong we can do a lot of damage
[20:49] <bigjools> which is why I am rather conservative in this area :)
[20:49] <jtv> If we could move the whole thing minus something small over to the slave, we'd have a consistent view (just a slightly older one) and no worries.  But if there's any sort of read-modify interaction it gets riskier.
[20:49] <bigjools> yeah
[20:50] <bigjools> we should go through it sometime and record all the interactions
[20:53] <jtv> bigjools: would it make sense to do a trial run with a slave-only policy and setPublished disabled?
[20:54] <bigjools> I dont think we have any tests that do everything at once like that, so we'd need to check the archive state manually
[20:55] <jtv> Meaning we'd probably miss something?
[20:57] <bigjools> possibly but if you point an apt client at the archive it would soon belly-ache
[20:58] <bigjools> we should get wgrant to comment on this
[21:04] <wallyworld_> thumper: StevenK: we having standup?
[21:05] <wallyworld_> thumper: i'm talking, jsut a sec
[21:05] <wallyworld_> thumper: you ok, my sound backend bad
[21:05] <wallyworld_> fixing
[21:05] <thumper> wallyworld_, StevenK: FYI https://code.launchpad.net/~thumper/launchpad/refactor-lazrjs-text-widgets/+merge/47634
[21:05] <StevenK> thumper: I can hear you too
[21:17] <wallyworld_> thumper:  https://code.edge.launchpad.net/~wallyworld/launchpad/recipe-find-related-branches/+merge/47367
[21:30] <huwshimi> Ugh, so my laptop has decided that when I'm logged in I should only be able to type numbers :(
[21:30] <StevenK> Yet, you're typing not numbers. :-)
[21:31] <huwshimi> StevenK: I am using my netbook to whinge about it
[21:31] <StevenK> Heh
[21:33] <wallyworld_> thumper: https://pastebin.canonical.com/42451/
[21:34] <wgrant> bigjools: Hi
[21:34] <bigjools> morning wgrant
[21:37] <wgrant> bigjools: This whole discussion makes me cry.
[21:39] <wgrant> Why do we want to move it to the slave?
[21:39] <bigjools> wgrant: no doubt
[21:39] <benji> huwshimi: numlock?
[21:39] <huwshimi> benji: Thanks, I just figured that out right then.
[21:39] <wgrant> We *could* do it, and things would remain consistent, but some things would show as published when they weren't.
[21:39] <StevenK> thumper: I've been trying!
[21:39] <bigjools> wgrant: jtv has been investigating but he wants to get rid of a long-running transaction
[21:39] <wgrant> And the latency would be two hours.
[21:40] <huwshimi> benji: They could have at least labelled it :)
[21:40] <bigjools> wgrant: yes that's exactly my concern in addition to incorrect indices vs release files
[21:41] <wgrant> bigjools: We may have extra stuff on disk, but we may also be removing stuff from disk later that is still referenced by the indices.
[21:41] <wgrant> So no, we cannot sensibly use the slave.
[21:41] <wgrant> However, yes, we can remove the long-running transactions by making the publisher not take 300 years.
[21:41] <wgrant> But that is a fair bit of work.
[21:41] <lifeless> we could add a ro connection to the master
[21:41] <bigjools> about 300 years
[21:42] <wgrant> lifeless: Is it only R/W transactions that are the problem?
[21:42] <lifeless> wgrant: no
[21:42] <wgrant> That's what I thought.
[21:42] <lifeless> wgrant: but they are a bigger problem than r/o transactions
[21:42] <lifeless> r/o transactions prevent index and row gc (mvcc chaff)
[21:43] <lifeless> r/w transactions cause related row exclusive locks
[21:43] <wgrant> Right.
[21:43] <lifeless> but its the actual *changes* in r/w transactions that matter
[21:43]  * bigjools gets tests working with a version column of type debversion \\o/
[21:43] <lifeless> length is just a (poor) proxy for predicting problems.
[21:43] <wgrant> There should be none. But it would be nice to verify that.
[21:43] <wgrant> bigjools: Yay!
[21:44] <bigjools> now, to optimise the queries
[21:44] <StevenK> thumper: So I use Mumble on the old laptop the audio is choppy and I can talk, if I use the new laptop the audio is excellent and I can't talk
[21:45] <thumper> heh
[21:45] <bigjools> wgrant: btw, the 2-builds-per-builder happened again today
[21:45] <wgrant> bigjools: Same builder?
[21:45] <bigjools> no,  after talking to lamont it turns out that it happens when he kills a long-running stuck build
[21:45] <wgrant> I noticed 6 or so builders earlier disabled with strange messages.
[21:46] <wgrant> But they were all happy once I re-enabled them.
[21:46] <bigjools> because we rely on aborting nonvirt builders
[21:46] <lifeless> flacoste: poolie: draft sent to you
[21:46] <bigjools> so I am going to change to code disable the builder it we see it aborting
[21:46] <wgrant> lifeless: Do you know what's going on with sinzui's mailing list QA?
[21:47] <wgrant> bigjools: Why?
[21:47] <bigjools> abort does not work on nonvirts
[21:47] <wgrant> ABORTING will drop to ABORTED, modulo a slave bug which doesn't properly kill sbuild because of a missing trap.
[21:47] <bigjools> exactly
[21:47] <bigjools> so until that's fixed...
[21:47] <lifeless> wgrant: yes
[21:47] <lifeless> wgrant: can I brain dump and have you chase?
[21:47] <wgrant> lifeless: That is what I was hoping for.
[21:48] <lifeless> wgrant: ok so lp prod configs was broken for qastaging mailman
[21:48] <lifeless> the qa port is 9097
[21:48] <wgrant> bigjools: Until that's fixed, buildd-manager will ignore the builder and there is a nagios check to tell lamont to fix it.
[21:48] <wgrant> Right.
[21:48] <lifeless> it said 8097
[21:48] <lifeless> there is a branch to fix that
[21:48] <wgrant> I believe that got reviewed and landed.
[21:48] <lifeless> so if it has
[21:48] <wgrant> It should be good?
[21:49] <lifeless> then it should be deployed and the mailman log should be calling into the api with the newer code
[21:49] <wgrant> It doesn't seem to be landed. I might fix that.
[21:49] <lifeless> remaining steps
[21:49] <wgrant> I've only run mailman locally a couple of times, so I'm not 100% on how exactly it works, but I guess I'll work it out.
[21:51] <wgrant> lifeless: qastaging should automatically update configs with its usual update, right?
[21:51] <lifeless> wgrant: yes
[21:51] <lifeless> wgrant: so, 1) get the right port out
[21:52] <lifeless> 2) make sure its querying members (via the logs)
[21:52] <lifeless> 3) profit
[21:52] <wgrant> Great, thanks.
[21:52] <wgrant> If all goes well, we can deploy in a couple of hours :)
[21:53] <StevenK> lifeless: So, the query Storm is creating is: SELECT SourcePackageRecipe.build_daily, SourcePackageRecipe.daily_build_archive, SourcePackageRecipe.date_created, SourcePackageRecipe.date_last_modified, SourcePackageRecipe.description, SourcePackageRecipe.id, SourcePackageRecipe.is_stale, SourcePackageRecipe.name, SourcePackageRecipe.owner, SourcePackageRecipe.registrant FROM SourcePackageRecipe LEFT JOIN SourcePackageRecipeBuild ON SourcePac
[21:53] <StevenK> Hm. That was a bit longer than it looked in the terminal
[21:54] <wgrant> bigjools: We haven't purged non-main indices for existing PPAs yet, have we?
[21:54] <lifeless> wgrant: new ppas won't have them at all
[21:54] <wgrant> I know.
[21:54] <wgrant> It broke a few things :)
[21:54] <wgrant> But they're all fixed now.
[21:55] <wgrant> But we were also going to manually remove the old ones.
[21:56] <flacoste> lifeless: replied
[21:57] <lifeless> thanks
[21:57] <lifeless> poolie: hi
[21:58] <bigjools> wgrant: no
[21:58] <bigjools> wgrant: go ahead and clean up, then we can re-enable the daily job
[22:40] <StevenK> lifeless: O hai. I have added a recipe with no builds and dropped the query and it returns 0 rows
[22:40] <lifeless> StevenK: you need to paste the query without getting cut off :)
[22:42] <StevenK> lifeless: http://pastebin.ubuntu.com/559270/ ; I've taken what LP_DEBUG_SQL gave me, replaced the %s's and dropped most of SourcePackageRecipe's columns from the SELECT
[22:43] <jtv> hi wgrant!
[22:43] <StevenK> If I drop the extra JOINs, it returns a row
[22:44] <wgrant> Morning jtv.
[22:45] <jtv> wgrant: bigjools & I were looking at the long transactions in the archive publisher earlier.
[22:45] <wgrant> jtv: So I saw.
[22:45] <jtv> You're everywhere.
[22:45] <wgrant> We cannot sanely move it onto a slave.
[22:45] <jtv> I was wondering whether we could move essentially the whole process over to a slave without risking inconsistencies.
[22:45] <wgrant> We need to make it take less insanely long.
[22:45] <jtv> Ah.
[22:46] <jtv> That answers that.
[22:46] <bigjools> jtv: wgrant made some more scary points that I'd not mentioned in addition to my scary ones
[22:46] <jtv> Oh good
[22:46] <jtv> that sounds like fun.
[22:46] <jtv> Making it take less insanely long looks somewhat doable, but I'll need a way to test.
[22:46] <jtv> It's FTPArchiveHandler.run that takes the bulk of the time, right?
[22:47] <wgrant> There are two things that take forever: file list generation, and a-f itself.
[22:47] <wgrant> The latter can be parallelised.
[22:47] <lifeless> StevenK: I'm fidddling
[22:47] <bigjools> wgrant: I'd like to see a-f and NMAF in a race
[22:47] <lifeless> StevenK: trivially doing left joins all over works, but that can get inefficient
[22:48] <jtv> wgrant: and the former looks like it may be a typical naïve-fetch-inside-loop pattern.
[22:48] <bigjools> I think it would be close
[22:48] <jtv> It's _almost_ a simple prejoin but there's an interaction with slicing to watch out for.
[22:48] <wgrant> bigjools: a-f is hundreds of times slower.
[22:48] <wgrant> Er.
[22:48] <wgrant> NMAF is.
[22:48] <StevenK> NMAF is *slower*?
[22:48] <wgrant> jtv: Possibly. But the queries themselves are bad.
[22:48] <wgrant> StevenK: Yes, it issues many many many more queries.
[22:49] <jtv> wgrant: the pattern, or some of the individual ones?  If you've got a list of known troublemakers, that'd help me see.
[22:50] <wgrant> jtv: I don't recall exactly. But I believe it's fairly efficient query-count-wise, but the queries are not quick.
[22:50] <wgrant> It's been more than a year since I seriously tried optimising this phase of the publisher.
[22:50] <wgrant> I got a *long* way, but it was sort of full of hacks.
[22:51] <wgrant> Down to 3-4 seconds for each index file, on NMAF, for 1.5x primary's size.
[22:51] <wgrant> cprov also optimised the a-f file list code a lot just before he left.
[22:51] <jtv> My may suspect based on following the arrows in the code was FTPArchiveHandler.publishFileLists.
[22:52] <jtv> That contains some loops over all source/binary packages, I believe.
[22:52] <wgrant> Yes, but that uses the views.
[22:52] <bigjools> which are evil
[22:52]  * jtv crosses himself
[22:52] <wgrant> It should not do any queries.
[22:52] <jtv> Luckily I made merit by visiting the temple: saw a working Difference Engine 2 yesterday.
[22:52]  * bigjools looks for garlic and silver
[22:52] <wgrant> Until right at the end when I do the disabled chekc.
[22:52] <wgrant> (which is only once per pocket-das)
[22:53] <jtv> What's a pocket-das?
[22:53] <wgrant> (PackagePublishingPocket, DistroArchSeries)
[22:53] <StevenK> hardy-i386-RELEASE
[22:53] <StevenK> For instance
[22:54] <wgrant> How many cores does cocoplum have?
[22:54] <bigjools> StevenK, wgrant: I am putting my debversion stuff on DF, please to not be touching for a bit
[22:54] <wgrant> bigjools: k
[22:54] <bigjools> unless you guys are using it in which case I can wait
[22:55] <StevenK> wgrant: 4
[22:55] <wgrant> :(
[22:55] <lifeless> StevenK: SourcePackageRecipeBuild JOIN PackageBuild ON PackageBuild.id = SourcePackageRecipeBuild.package_build JOIN BuildFarmJob ON BuildFarmJob.id = PackageBuild.build_farm_job right join SourcePackageRecipe  ON SourcePackageRecipeBuild.recipe = SourcePackageRecipe.id
[22:55] <lifeless> StevenK: put that in from 'FROM' ... 'WHERE'
[22:56] <jtv> wgrant: it looked to me as if, unless we mess with the code's structure which I'm reluctant to do, it should do a bunch of batched prefetch queries in each iteration of its loop-tuner callback.  If it's using views, maybe that's worth eliminating as well.
[22:56] <wgrant> jtv: Because it's using views, it's already entirely prefetched.
[22:56] <wgrant> All you can do is optimise the view.
[22:56] <jtv> Actually, prefetching may be what's slowing it down then.
[22:57] <wgrant> However, file list generation is only a couple of minutes.
[22:57] <wgrant> Running a-f takes 15 or so.
[22:57] <StevenK> lifeless: That looks good. How do I tell Storm that?
[22:57] <lifeless> RightJoin
[22:57] <StevenK> Instead of left?
[22:57] <lifeless> StevenK: works like LeftJoin in terms of function calls
[22:57] <lifeless> StevenK: yes; same parameters, same translation
[22:58] <lifeless> StevenK: play with it a little to get the invocation you need
[22:58] <lifeless> I'm just checking the query plan on staging
[22:58] <lifeless> 96ms
[22:59] <lifeless> so fast
[23:00] <lifeless> StevenK: you'll also want DISTINCT, because you only want one row per sourcepackagerecipe
[23:00] <lifeless> StevenK: as written, on staging, we can get
[23:00] <lifeless>  id |    name     |  owner
[23:00] <lifeless> ----+-------------+---------
[23:00] <lifeless>  15 | awn-testing | 1382524
[23:00] <lifeless> (3 rows)
[23:04] <StevenK> lifeless: I'm still distilling your query into Storm. *Then* I'll worry about distinct
[23:04] <lifeless> StevenK: I'd do it for you, but known how this works will help you
[23:06] <jtv> wgrant: drat.  So to speed the code up significantly we'd have to run parallel apt-ftparchive instances on separate file lists?
[23:07] <wgrant> jtv: Yes. We already have separate file lists, so it's fairly easy.
[23:07] <StevenK> lifeless: Storm has SELECT ... FROM SourcePackageRecipe; whereas your query has SELECT ... FROM SourcePackageRecipeBuild; is that pertinent?
[23:07] <jtv> wgrant: and then… just concatenate the outputs?
[23:08] <wgrant> jtv: No. Each file list results in one index.
[23:08] <wgrant> If you look at the log, you'll see it generates one index for (maverick-release, source, main), another for (maverick-release, i386, universe), etc.
[23:09] <wgrant> Each of those is big.
[23:09] <wgrant> And each of those can be done in parallel.
[23:09] <lifeless> StevenK: yes
[23:09] <lifeless> StevenK: or probably yes, show me thje FROM..WHERE bit ?
[23:10] <StevenK> lifeless: http://pastebin.ubuntu.com/559277/
[23:11] <jtv> wgrant: ah so parallel runs per architecture, plus one for source, and they should all overlap fairly well.
[23:11] <wgrant> jtv: Yes.
[23:11] <jtv> The hard part would probably managing the parallel processes.
[23:11] <wgrant> Yes. And that's not terribly hard.
[23:12] <wgrant> In the run I'm looking at, running a-f takes 16 minutes. File list generation takes 1.5.
[23:30] <StevenK> lifeless: I suspect you got distracted?
[23:37] <lifeless> StevenK: born distracted
[23:37] <StevenK> Duh
[23:42] <lifeless> StevenK: ok
[23:42] <lifeless> so that thing you pasted means
[23:43] <lifeless> 'select the things joined together where there are 0 or many sourcepackagerecipes'
[23:43] <lifeless> what you want is
[23:43] <lifeless> 'select source package recipes where there are 0 or many (things joined together)'
[23:43] <lifeless> right joins allow NULL on the left hand side
[23:43] <poolie> lifeless, spm, as a specific next step on bug 701329, can we get haproxy logs?
[23:43] <_mup_> Bug #701329: error: [Errno 32] Broken pipe <oops> <Launchpad itself:Triaged> <loggerhead:Triaged> < https://launchpad.net/bugs/701329 >
[23:44] <lifeless> that is, they include 'every row on the right hand side'
[23:44] <lifeless> poolie: can we not just catch the exception ?
[23:44] <poolie> or, try to titrate them to a level where they tell us about interesting errors without flooding the system
[23:44] <lifeless> poolie: what are you trying to determine
[23:44] <poolie> why haproxy closed the connection
[23:44] <poolie> and from which ip it came
[23:45] <lifeless> poolie: can you back it out a step higher
[23:45] <lifeless> I don't understand why this isn't a fairly direct code change
[23:47] <poolie> so, to close this particular bug, we can, say
[23:47] <poolie> just make it not oops when the connection is abruptly closed?
[23:47] <poolie> perhaps it should log a warning-type error
[23:47] <poolie> that seems pretty easy
[23:47] <lifeless> none of our other servers log on this
[23:47] <lifeless> *or*
[23:47] <poolie> it makes me wonder if we are papering over a problem
[23:47] <lifeless> none of our other servers are experiencing it
[23:47] <poolie> so i'd like to see a bit more data about why the connection is being dropped
[23:48] <poolie> haproxy is the obvious place to look for that
[23:48] <poolie> and, i think it is a bit weird to have a production service where logging is entirely disabled
[23:48] <poolie> (perhaps "it's weird" is not a reason to change it but i think it's reasonable to ask)
[23:48] <lifeless> I thought you were told that it was a volume problem ?
[23:49] <lifeless> haproxy being a bit of a firehose
[23:49] <lifeless> anyhow
[23:49] <lifeless> if you're working on this bug
[23:49] <lifeless> thats cool
[23:49] <poolie> lifeless, i'd guess that zope etc don't traceback on epipe, and i agree that in general loggerhead shouldn't
[23:49] <lifeless> uhm, what can I do to hepl.
[23:49] <lifeless> firstly, have you looked at some of the OOPSes?
[23:49] <poolie> yes
[23:49] <poolie> i'd like to turn logging >=WARNING
[23:49] <poolie> and see if that is in fact a firehose
[23:50] <poolie> if it is, that's a bit alarming, and i'd like to just see what it's logging
[23:50] <poolie> then we can turn it off again
[23:50] <poolie> is that ok?
[23:50] <lifeless> I've no objections in principle, though the losas generally make informed decisions about this sort of thing :)
[23:51] <poolie> spm, do you mind trying that?
[23:51] <spm> we start with "no" and negotiate from there.
[23:51] <poolie> sigh
[23:51] <spm> poolie: nope, worth a shot. :-)
[23:51] <poolie> thanks
[23:51]  * StevenK was waiting for "Depends. Do you have cake?"
[23:51] <poolie> the haproxy manual says you can set the log leevl
[23:52] <spm> StevenK: it's poolie. he lives closer than wgrant, and rides a motorbike ie Hells Angel in training. I've learnt to be nice to him.
[23:52] <lifeless> poolie: so
[23:52] <lifeless> poolie: looking at  https://lp-oops.canonical.com/oops.py/?oopsid=1836CBA1
[23:52] <lifeless> poolie: there are several things that make this hard to analyze
[23:53] <lifeless> poolie: if I was working on it, I would fix those first
[23:53] <lifeless> poolie: firstly there is no total time recorded
[23:53]  * StevenK suspects that poolie is shinier, due to lifeless completly context switching and goes afk for ten or so
[23:53] <lifeless> poolie: nor any timeline (which we could use to track xmlrpc / bzr call overhead if we wanted to)
[23:53] <poolie> i've learned to front-load anything that may need SA action
[23:53] <lifeless> StevenK: sorry, I thought I answered you?
[23:54] <lifeless> StevenK: @10:44
[23:54] <lifeless> poolie: if we had timing data, we could assess whether this is really a timeout or not
[23:55] <lifeless> poolie: in fact, we could add a thing to say 'if the time is > some N, convert to a timeout'
[23:55] <poolie> yup
[23:55] <lifeless> we probably need to expose an API call to ask for the timeout
[23:55] <poolie> but if haproxy is saying "request blah blah timeout, killed"
[23:55] <lifeless> e.g. 'what should my timeout be please?'
[23:55] <poolie> that seems easier
[23:56] <lifeless> poolie: I worry that we're going to need a lot of correlation to match up all 15K a day
[23:56] <lifeless> poolie: gathering data at source seems more useful (to me)
[23:56] <lifeless> poolie: anyhow, thats just what I'd do if I was working on it
[23:57] <lifeless> however
[23:57] <lifeless> note that haproxy probably won't tell us
[23:57] <lifeless> if this is coming in behind haproxy
[23:57] <lifeless> spm: how often does nagios query ?
[23:58] <spm> i think it's up to 3 times every 10 minutes
[23:58] <poolie> lifeless, i don't want to match up all of them, i just want to know if haproxy thinks its client is dropping the connection, or if it is dropping the connection itself
[23:58] <poolie> or if it's oblivious
[23:58] <poolie> it shouldn't be too hard to answer
[23:58] <lifeless> sure
[23:59] <spm> but probably easier to look in the logs and count
[23:59] <lifeless> I am open to many ways to figure out whats up
[23:59]  * thumper is busy writing real documentation