wgrantStevenK: I can has blessing?00:00
StevenKthumper: https://code.launchpad.net/~wgrant/launchpad/bug-707741-addFile-master/+merge/47606 please00:00
StevenKwgrant: I was getting there :-)00:01
wgrantStevenK: Great, thanks.00:01
wgrantSo many regressions :(00:01
lifelessleonardr: is there some reason we can't have a destructor parameter? That doesn't seem to alter the boolean nature00:02
wgrantlifeless: Thanks.00:05
wgrantlifeless: Bugs needs a kanban UI.00:09
wgrant... well, this is amusing.00:11
wgrantgetUnlandedSourceBranchRevisions has a reasonably complex query, and it is completely untested.00:11
wgrantI expect that sort of thing from Soyuz, not Code :/00:12
lifelesspoolie: https://pastebin.canonical.com/42398/00:24
lifeless-> allergy injection00:27
* henninge eods00:29
wgrantStevenK: Another regression fix for you, if you have a moment: https://code.launchpad.net/~wgrant/launchpad/bug-707808-unlanded-revisions/+merge/4761700:36
StevenKlifeless, thumper: https://code.launchpad.net/~wgrant/launchpad/bug-707808-unlanded-revisions/+merge/47617 if you please00:38
thumperStevenK: ack00:58
* StevenK kicks the merging code00:59
wgrantthumper: https://code.launchpad.net/~deon-wurley/phpldapadmin/1.2.x <- are Remote branches without URLs legal?01:04
thumperwgrant: yes...01:04
thumperwgrant: but sad01:04
wgrantBut OK.01:04
thumperwgrant: not much point really01:04
Ursinhawgrant, hi01:04
wgrantUrsinha: Hi, 'sup?01:05
Ursinhawgrant, so, there are bunches of ApacheLogParserError01:05
wgrantAh, those.01:05
wgrantAll fixed, but not deployed yet.01:05
Ursinhalike this: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1851PPA105101:05
wgrantSince germanium isn't in nodowntime.01:05
Ursinhahmm, I see.01:05
Ursinhawgrant, thanks :)01:06
wgrantI hope we can germanium in nodowntime soon :(01:07
wgrantthumper: Thanks.01:08
wgrantAll major regressions fixed, yay.01:08
StevenKNow deploy them?01:09
wgrantMaybe in 8 or so hours.01:09
StevenKLonger, devel will hit testfix01:11
wgrantOh fuck, you're serious.01:14
wgrantThat spurious librarian thing.01:15
thumperoh FFS01:15
wgrantWe should kill the current build.01:15
wgrantspm: Can we easily kill a buildbot build?01:15
spmyah. which one.01:15
StevenKWhere easy is er, not01:15
wgrantIf not, could you do it the hard way? :P01:15
spmwhere kill == stop.01:15
StevenKspm: 556 on lucid_lp01:15
wgrantI'm off to lunch now; could someone force a build once it's done?01:17
StevenKwgrant: Aye01:18
StevenKspm: Tell me when it stops hurting01:18
spmso the lsave is completely ded. how do we clean this up so it won't run that revno again?01:18
StevenKThe revno is fine01:19
StevenKThe errors are spurious and we don't want to be in testfix01:19
spmoh I see. perhaps getting the spurious nature of the errors fixed would be a GoodIdea™? ;-)01:19
StevenKHere, have a Hudson01:19
StevenKThat should help01:19
StevenKspm: Do you need to restart the master to get the slave back?01:20
spmseems to be showing back to me. should be able to force and magic happen01:20
StevenKIt's buildbot, it isn't close to magic01:21
StevenKspm: Done, thanks01:21
StevenKwgrant: Build forced01:21
* thumper stabs the librarian02:28
thumperAttributeError: 'NoneType' object has no attribute 'add_section'02:28
thumperwhile trying to set up the librarian in tests02:28
thumperanyone got any ideas?02:28
lifelesspoolie: ok I'm back02:40
lifelessthumper: that means the layer which supplies the config to edit hasn't been setup02:40
thumperlifeless: which I'm getting why?02:41
thumperit has only just started happening02:41
thumperafter merging dev02:41
thumperbin/kill-test-services is borked02:41
lifelesstheres a bug on that02:41
lifelesswith an explanation; fix should be trivial02:41
poolielifeless, i think your mail is fine02:42
pooliei replied too02:42
lifelesspoolie: well, I've just sent my mail02:45
poolieinteresting new reply from max02:46
thumperlifeless: my librarian layer won't setup02:46
wgrantthumper: Turn LP_PERSISTENT_TEST_SERVICES off.02:47
wgrantIt doesn't work any more.02:47
lifelessiz bug02:47
lifelessI put work in to make it work02:47
wgrantIt's been that way for a week or so now.02:47
lifelessbut its not actually tested02:47
wgrantI see.02:47
lifelessI didn't *intend* to break it02:47
lifelessbut I couldn't tell that I *had*02:47
lifelessits useless for parallel testing02:47
wgrantI presumed you turned a blind eye to your breakage of it.02:48
wgrantIt is, yes. But we have no parallel testing yet.02:48
lifelessso I'm not very interested in maintaining it02:48
lifelesswgrant: we have some, a decent subset of tests work fine in parallel now02:48
wgrantIt is close.02:48
wgrantThe main problem now is AppServerLayer.02:48
wgrantPlus some directories that need randomising.02:48
maxblifeless: You binged?02:49
wgrant2.7 is close enough to be done by a maintenance squad, and I think parallel testing is probably almost there too.02:49
lifelessmaxb: python-subunit 0.0.6 in the lp ppa02:50
lifelessmaxb: part of fixing pqm to mail on conflicts02:50
lifelessmaxb: however it looks like the losas can grab from the bzr ppa ok02:50
lifelesswgrant: feature work -> feature queue02:50
lifelesswgrant: if you're looking for something to do, I see a single bug causing 16K oopses a day.02:50
lifelesswgrant: (hint)02:51
wgrantHeh, no, I'm not looking for stuff to do :(02:51
wgrantThat's the codebrowse connection closed one?02:51
StevenKlifeless: No fair if it's loggerhead02:51
lifelesswgrant: seriously, yes its close, but I guarantee it will have a long tail02:51
lifelessStevenK: loggerhead probably has more test coverage the LP proper02:51
lifelesswgrant: and the long tail is what makes me want to give it to a feature squad: get it live in buildbot/hudson, working in tarmac, profile it and assess scaling, disk IO utilisation, make recommendations on a new server to run massively parallel tests02:52
lifelesswgrant: figure out the damn db leak bug02:52
wgrantlifeless: I think that figuring that out is probably best achieved by deleting layers and zope.testrunner.02:53
lifelesswgrant: I have handwavy plans for that02:53
lifelesswgrant: its also pretty shallow02:53
lifelessbut, time.02:53
lifelesswgrant: focus will get us places!02:53
StevenKlifeless: I'd prefer if codebrowse actually looked like the rest of LP02:53
lifelessStevenK: so would codebrowse :P02:53
lifelessStevenK: it is themeable02:54
StevenKBut I can recall you telling me that's pointless in Dallas02:54
lifelessStevenK: that doesn't sound like me; I think I may have said tha tthat is the least of the issues02:54
wgrantStevenK: Everything is pointless in Dallas.02:54
lifelessStevenK: which != pointless, but I could well have been confusing the issue02:55
pooliei might try just running tip on a big tree02:55
poolieand see how it does02:55
lifelesshuwshimi: hi02:56
huwshimilifeless: Hey there02:57
lifelesshuwshimi: I dunno if you've seen https://dev.launchpad.net/BugTriage yet ?02:57
huwshimilifeless: Uh, no. Are you referring to the bug I just filed?02:58
lifelesshuwshimi: yes I am :)03:00
lifelesshuwshimi: have a read :)03:00
huwshimilifeless: Yeah thanks. Reading it now.03:00
lifelesshuwshimi: its useful context to fit in with the team; the specific thing that alerted me was creating a medium importance bug (we don't use medium) and then starting work on it (thus ignoring the critical and high bugs)03:01
lifelesshuwshimi: I think your particular focus means that looking at all critical/high bugs is irrelevant to you03:01
lifelesshuwshimi: -but- you probably want to be sorting the design related bugs by the impact/importance you think they have03:02
lifelesshuwshimi: anyhow, no drama; we have a massive learning curve and I primarily wanted to let you know about this part of it :)03:03
huwshimilifeless: Sure thanks for letting me know. I'm about to submit a bunch of bugs so I'm glad you told me now :)03:03
huwshimilifeless: There was some discussion last week about triaging bugs ourselves. Was there a decision made about that? Am I better off not self triaging for now?03:14
lifelesshuwshimi: self triage is great03:18
lifelessfollow the guidelines in the wiki page03:19
lifelessbeyond that, if there are disagreements, english is a wonderful tool for figuring em out ;)03:19
huwshimilifeless: OK thanks for that :)03:19
* thumper grrs03:31
* thumper is slowly getting there04:41
StevenKlifeless: So I don't think I can do the heavy lifting with SQL. If the recipe has no builds, http://pastebin.ubuntu.com/558848/ returns nothing, and so no daily builds get dispatched04:52
pooliehow do i, through the api, find all bugs assigned to a particular team?04:52
lifelessStevenK: looking04:53
lifelessStevenK: you probably want either a LEFT OUTER JOIN or a UNION, or a COALESCE04:53
lifelessStevenK: probably a left outer join04:53
StevenKlifeless: Storm doesn't have a LeftOuterJoin?04:54
thumperhow do I emit a tag in a page template of a particular type?04:54
lifelessStevenK: it does, LeftJoin IIRC04:55
lifelesswe use it all over04:55
thumperview/tag is something like "h1", or "span"04:55
thumperand I want to get <h1 id='${view/id}'> in the pt based on view/tag04:55
* thumper is open to suggestions 04:55
* StevenK is open to patches, never having done joins through Storm before04:55
thumperI'm thinking I may need an attribute and structure04:55
lifelessStevenK: grep for LeftJoin04:56
lifelessStevenK: I'm in the middle of a deep n meaningful over loggerhead04:56
lifelessits breaking my brain to think SQL at the same time04:56
StevenKMission accomplished04:57
* StevenK hides04:57
StevenKlifeless: If you want to point what I'm doing wrong in http://pastebin.ubuntu.com/558856/ , that would be awesome. I can't find any usage like this in the tree05:19
* thumper is almost done refactoring widgets05:22
lifelessStevenK: wtf05:25
StevenKClearly then, the answer is "everything"05:26
lifelessStevenK: the pattern is05:26
lifeless(IIRC)    LeftJoin(lefttable, righttable, condition)05:27
lifelessSourcePackageRecipe LEFT OUTER JOIN SourcePackageRecipeBuild on SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id05:28
lifelessLeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id)05:28
lifelessthe result of that is a table itself05:28
lifelessso to nest you'd do05:28
lifelessLeftJoin(LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id), PackageBuild, PackageBuild.id ==SourcePackageRecipeBuild.package_build_id)05:29
lifelessbut you don't need a nested left join05:29
lifelessyou only need one outer join - at the point you're willing to have a NULL row05:29
lifelessJoin(LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id), PackageBuild, PackageBuild.id ==SourcePackageRecipeBuild.package_build_id)05:29
lifelessJoin(Join(LeftJoin(SourcePackageRecipe, SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == SourcePackageRecipe.id), PackageBuild, PackageBuild.id ==SourcePackageRecipeBuild.package_build_id), BuildFarmJob, BuildFarmJob.id == PackageBuild.build_farm_job_id)05:30
lifelessis your using object05:30
lifelessthen in the where clause you put05:30
lifelessOr(BuildFarmJob.id == None, BuildFarmJob.date_created > one_day_ago)05:30
thumperw00t w00t05:35
lifelessStevenK: does that make sense?05:35
lifelessStevenK: use LP_DEBUG_SQL to see the sql being emitted05:35
lifelessStevenK: and adjust until you have a query you're happy with05:35
StevenKYes, it made sense05:36
lifelessdid it help ?05:36
StevenKlifeless: Not really, it still doesn't return SPRecipes that haven't built05:40
StevenKI suspect SourcePackageRecipeBuild.id == None will help, since they are created for builds, which then creates the PackageBuild and BFJ05:41
lifelessStevenK: you probably want to bring back the sprb as well05:43
lifelesssince that will tell you the when. Or perhaps not.05:44
* thumper EODs06:03
* huwshimi waves goodbye06:08
lifelessStevenK: need more help? I have cycles in ~ 1006:21
StevenKlifeless: Sorry, I was picking up Sarah, and I EOD'd 1.6 hours ago06:42
lifelessgrab me tomorrow06:44
lifelessand we can nut it out06:45
StevenKlifeless: Was my plan07:12
=== almaisan-away is now known as al-maisan
adeuringgood morning09:19
mrevellGuten morgen09:22
=== al-maisan is now known as almaisan-away
=== allenap changed the topic of #launchpad-dev to: Launchpad development https:/​/​dev.launchpad.net/​ | PQM is open | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
=== salgado-afk is now known as salgado
gmbDoes anyone know what in the sweet hell this is about?:11:35
gmbAn error occurred during a connection to launchpad.dev.11:35
gmbSSL received a record that exceeded the maximum permissible length.11:35
gmb(Error code: ssl_error_rx_record_too_long)11:35
wgrantgmb: This is on a fresh LP install?11:42
gmbwgrant: No, it's an old one. But I did have to re-run rocketfuel-setup recently to correct a couple of pebkacs.11:45
Ursinhagood morning launchpad11:47
* gmb re-does the apache config dance11:47
wgrantgmb: Odd. I normally only see that when it's a fresh install needing an Apache restart, or I've broken the vhost config somehow.11:49
gmbwgrant: Yeah. I think I've borked the vhost config in some myserious way (probably because I have a remote access setup and rocketfuel-setup fought with it).11:49
gmbI'm redoing it now.11:49
maxbI find the nicest way to manage a remote access thingy is to let rocketfuel-setup manage the local-launchpad config file, but a2dissite it, copy it, and modify the copy11:50
=== Ursinha is now known as Ursinha-afk
gmbmaxb: Very wise. I shall do that henceforth.11:54
gmbHoorah. It works.11:55
gmb/me lunches11:58
deryckMorning, all.12:06
Ursinhamorning deryck12:08
=== mrevell is now known as mrevell-lunch
salgadowhat's the best thing to use when writing a setup.py these days?  setuptools?  distribute?12:20
jelmersalgado: what sort of features do you need from it?12:24
jelmerdistutils is widely available and seems to work pretty well for basic python modules12:24
salgadojelmer, our needs seem to be rather basic, so maybe distutils will be enough indeed12:29
jmlsalgado: I copy an existing setup.py :)12:31
salgadojml, that's always a good strategy12:45
jmlcan someone give me a dumb manager's summary of the codebrowse discussion on the mailing list?13:01
=== jcsackett changed the topic of #launchpad-dev to: Launchpad development https:/​/​dev.launchpad.net/​ | PQM is open | firefighting: - | On call reviewer: allenap, jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
=== mrevell-lunch is now known as mrevell
=== almaisan-away is now known as al-maisan
salgadobenji, your last message to that mp I reviewed yesterday had only some quoted text from my previous message14:13
benjisalgado: hmm, that's odd.  I just said "sorry about that, thanks for asking Curtis to look at it"14:14
salgadobenji, looks like the issue I encountered yesterday; already reported a bug for it14:15
benjido you have the bug number at hand?14:16
abentleyjelmer, ping14:18
abentleyjelmer, unping14:18
oyvi'm trying to setup launchpad on a virtual machine, but when i try make run i get an importerror (no module named loom.branch)14:19
oyvanybody got any hints?14:19
abentleyflacoste, now that domain expertise is spread across squads, maybe it would be good to have a list of domain experts, e.g. on the wiki?14:22
oyv(i can import the branch in python shells, so it should be installed and everything)14:25
jcsackettoyv: are you following the instructions on the wiki?14:28
oyvi use isntructions from here: https://dev.launchpad.net/Getting14:29
oyv(and the "running" page)14:29
jcsackettoyv, ok. what version of ubuntu are you running?14:30
oyvUbuntu 10.04.1 LTS (lucid lynx)14:32
jcsackettoyv: i encountered this at one point; i am trying to remember how i fixed it.14:34
jcsackettoyv: i assume you are not actually using bzr loom? it shouldn't fail in that case, i'm just defining what's going on.14:38
oyvi don't think i'm using it, not really sure what it is ;)14:39
jcsackettit's a plugin for bzr, http://doc.bazaar.canonical.com/plugins/en/loom-plugin.html. many lp developers use it, but last i checked it was not a dependency.14:40
jcsackettand if it is, it should have been automatically installed when you hit the update step in the get/run instructions.14:41
jmljcsackett: it is a dep14:41
jcsackettjml: really? did that change? when i started on lp i was told it wasn't, b/c i was hitting this (or a similar) issue.14:41
oyvedb@launchpad:~/launchpad/lp-branches/devel$ bzr plugins14:42
oyvloom 2.1.0 Loom is a bzr plugin which adds new commands to manage a loom of patches.14:42
* jcsackett supposes he could have been deasily misinformed.14:42
jmljcsackett: since 2008, I think. it's needed to process looms14:42
oyvdoesn't that mean it's installed?14:42
benjieither is fine with me; but more importantly it means I need to go pour my coffee right now14:43
benjipfft, wrong chan14:43
jcsackettjml: dig. clearly i was misinformed. :-P14:45
jcsackettoyv: sorry, i'm not sure what might be going on. it does seem that it is installed as a plugin.14:51
jcsacketti could be of more help if i had a functioning machine right now, but i'm rebuilding after hardware failure yesterday, so i can't explore much on my end. :-/14:52
=== al-maisan is now known as almaisan-away
oyvok, thanks anyway :)14:53
LPCIBotProject devel build (396): FAILURE in 4 hr 38 min: https://hudson.wedontsleep.org/job/devel/396/14:55
LPCIBotLaunchpad Patch Queue Manager: [r=lifeless,14:55
LPCIBotstevenk][bug=707741] Fix LibrarianClient.addFile to function under14:55
maxboyv: I don't suppose you get a traceback related to that ImportError?14:55
maxbWhen you're running "make run", the copy of bzr-loom that is supposed to be being used is the one found via bzrplugins/loom/ in the Launchpad tree14:56
salgadobenji, it's bug 70825814:56
_mup_Bug #708258: Failed to parse merge proposal comment sent via email <code-review> <email> <Launchpad itself:Triaged> < https://launchpad.net/bugs/708258 >14:56
benjisalgado: thanks14:56
=== Ursinha is now known as Ursinha-afk
* maxb sobs a bit as launchpad-database-setup tries to configure pg8.214:59
oyvthere's only one folder in the bzrplugin folder, lpserve14:59
maxbYour tree is broken then14:59
flacosteabentley: that's what the wiki pages related to the Launchpad in 30 minutes presentation was meant to do14:59
flacosteabentley: https://dev.launchpad.net/Foundations/ComponentHelp15:00
flacostei think Tim did something similar15:00
flacosteand so did Bugs15:00
flacostenot sure if Soyuz and Translations put their stuff in that format15:01
abentleyflacoste, Ah, didn't think of looking there.15:02
deryckah, adeuring, call time.  Sorry got distracted running tests and answering emails.15:02
flacosteabentley: it would probably make sense to collate all the content in one place, or at least make an easy-to-find index page to it15:02
adeuringderyck: ok, no problem15:02
bigjoolsgoooooood morning15:02
abentleyflacoste, +115:02
=== Ursinha-afk is now known as Ursinha
abentleybigjools, can you confirm that we never send emails for successful binary builds?  Because there are some tests of Build.notify with BuildStatus.FULLYBUILT.15:07
bigjoolsabentley: I can confirm that15:09
bigjoolsthat test sounds a bit bong15:09
abentleybigjools, great.15:09
deryckhenninge, I need 5 minutes to wrap up some notes, and then we can chat.   if that works for you.15:11
henningederyck: that's fine15:11
abentleybigjools, since recipe builds want to notify on success, I'm moving the differentiation into BinaryPackageBuild.notify/SourcePackageRecipeBuild.notify, and fixing the tests.15:12
bigjoolsabentley: you *want* them to notify on success or you're removing that?15:12
abentleybigjools, I *want* them to notify on success.15:12
bigjoolssounds odd to be15:13
james_wwill the coalesce?15:13
abentleyjames_w, yes.  How else do I know that a recipe build has happened?15:13
bigjoolsprepare for wrath if you do this!15:13
abentleybigjools, we've been doing this from the start.15:13
james_wbecause they happen every day? because Launchpad is reliable and does what I ask?15:14
bigjoolswe really need a better email notification story15:14
abentleyjames_w, no, they only happen when the source changes.15:15
abentleyjames_w, my bzr recipe triggers once or twice a month.15:15
james_ware you /really/ going to send people 200 emails every day in the extreme case?15:15
james_whow is that useful?15:15
=== salgado is now known as salgado-lunch
abentleyjames_w, it's useful so that people know when there's a new build.15:15
james_wbut that information isn't very useful at all as you move towards heavy users15:16
abentleyjames_w, if you get it and you don't want it, you can filter it out.  If you don't get it and you want it, you have no option.15:16
james_wdo you really want to discourage heavy use of the service using email volume15:16
allenapgmb: Do you have time to do a sanity check on https://code.launchpad.net/~allenap/launchpad/freedesktop-importance-flip-flop-bug-707478/+merge/47667 please?15:17
gmballenap: Sure.15:17
james_wclient-side filtering has been deemed to be not sufficient in the bugs case15:17
abentleyjames_w, client-side filtering is better than no mail.15:19
james_ware you sure your users would agree with that?15:19
abentleyjames_w, I am sure you can find users to disagree with anything.15:19
james_wwell, let's all go shopping then15:20
gmballenap: Approved.15:22
abentleyjames_w, I am just fixing a bug where the wrong kind of notification is sent, not increasing the number of notifications.15:22
allenapgmb: Thank you :)15:23
james_wok, then you are absolved from any responsibility for the system15:23
abentleyjames_w, so I've been talking with jelmer about this, and we both agree that there's a lot of room for improvement in the build notification story.15:26
abentleyjames_w, jelmer would like to see a notification of successful binary builds, grouped so that you only get one email for multiple builds.15:27
abentleyjames_w, and then we would be able to omit the success emails for recipe builds.15:28
bigjoolsall emails that LP sends should be controllable from a single page15:29
bigjoolsper person, I mean15:29
abentleybigjools, I don't know what to think about that.  It would be a verrrry long page.15:30
bigjoolspotentially but not always15:30
bigjoolsit can be ajaxified but you know what I mean - the subject of being sent machine-generated email is very divisive15:31
abentleybigjools, generally, asking users to make choices is bad.  The fewer choices, the smoother something feels.  This makes me think that the need for such a page indicates a design problem.15:35
henningeallenap: thanks for the review!15:40
bigjoolsabentley: that's the Gnone way, which is completely disagree with15:40
abentleybigjools, I'm not disputing the actual need, but I do wonder whether we're not thinking big enough.15:40
henningeadeuring: did you see my review of your branch?15:41
bigjoolsanyway, I'm not bikeshedding over this15:41
adeuringhenninge: yes, thanks. Actually, I think we should keep the assert() calls in setUp() because it is so esay to cause a mess there, and I am not sure if this would always result in test failures15:42
henningeadeuring: well, I think that is true for a lot of places in the code, though.15:43
henningeadeuring: but I don't really mind leaving them in there.15:44
adeuringhenninge: OK; perhaps my concerns are related to my unfamiliarity with translations ;)15:44
henningeadeuring: ... I tried not to be so blunt ;-)15:45
henningeadeuring: I have done that before, added asserts to make sure I got things right but once it worked, I removed them.15:45
adeuringhenninge: ;) but nevertheless, if we can screw the setup just by exchanging two factory calls, I think it is worth to check we don't do that...15:45
henningeadeuring: that's why I suggested leaving just the one in there.15:46
henningeand a comment15:46
henninge"# The order of creation is important here to make sure is_current_ubuntu is set to False.15:47
henningeadeuring: ^ like that, only use the right variable for 'message'.15:48
allenapgmb: Do you remember why sourceforge.net bug watches are disabled?16:01
gmballenap: Because our super-duper sourceforge slurping screenscraping fantastical disaster rather relied on them never, ever changing their templates, and they did.16:02
gmb(I might have overstated how good that code actually was)16:02
allenapgmb: Oh yes, thanks. I've just had a look at their templates and they're very much simpler now. Might fix that bad boy.16:09
gmballenap: Don't they offer an API now?16:09
allenapgmb: Do they? Awesome.16:10
gmbISTR they do. Maybe I saw that on the ForgePlucker mailing list.16:10
=== salgado-lunch is now known as salgado
abentleyallenap, jcsackett could you please review https://code.launchpad.net/~abentley/launchpad/build-mail3/+merge/47679 ?  It's long, but only because of indentation fixes in a doctest.16:14
jcsackettabentley: sure.16:14
abentleyjcsackett, thanks.16:15
allenapgmb: I can't find an API, but tell me if you happen upon it.16:15
gmballenap: Yeah, I'm not able to find one either, disappointingly. Must have been a happy dream.16:16
gmbI need new dreams.16:16
allenapHehe, you do :)16:16
=== almaisan-away is now known as al-maisan
abentleyderyck, I feel like I lack the domain knowledge to dive into any of the tasks on the Kanban board.  Can you suggest one?16:30
* deryck is looking16:30
deryckabentley, so I'm sure I lack the domain knowledge too :-)  However, the card for bug 696009 seems a  nice next step.....16:31
_mup_Bug #696009: Provide ITranslationMessage.shareIfPossible unit tests <tech-debt> <upstream-translations-sharing> <Launchpad itself:Triaged> < https://launchpad.net/bugs/696009 >16:31
deryckabentley, it's still in the area of test clean up.  so will broaden domain knowledge as cleanup happens.16:32
abentleyderyck, okay, I'll tak that.16:33
deryckok, cool16:33
=== deryck is now known as deryck[lunch]
henningeadeuring: am I to expect a new revision on your branch? Otherwise I'll start landing mine (which includes yours and abentley's)16:35
henningeactually, I'll make sure to merge the latest first16:35
adeuringhenninge: well, I'd prefer to leave the tests as they are... so, no new revesion ;)16:38
henningeadeuring: ok, thanks. ;)16:38
jcsackettabentley: comments on your MP. feel free to answer there or here.16:46
abentleyjcsackett, fire away.16:46
jcsackettabentley: as commented on the MP, i'm wondering about the use of self.builds in the from_build method.16:47
jcsackettmainly, in the first hit, can that get big enough to cause performance issues?16:48
abentleyjcsackett, it is always one or zero hits.16:48
jcsackettabentley: so does the cache help? the cache is always gone at the end of the transaction, yes?16:49
gmbI broke the build; rolling it back now. Sorry folks.16:50
abentleyjcsackett, the cache will not help with the cases we have at hand, because we only use it once.16:50
jcsackettso putting it in cached_property is forward looking?16:50
abentleyjcsackett, but I felt that it made sense to cache it since one does not expect an attribute to be expensive.16:51
=== al-maisan is now known as almaisan-away
gmbOh, nice. I specified --rollback for lp-land but it doesn't seem to have tagged the commit message thus.16:51
* gmb files a bug.16:51
abentleyjcsackett, you could say putting it in cached_property is forward-looking.16:53
jcsackettabentley: so do you see any problem with the use of self.builds being called in there? since no preloading will happen, can hitting self.builds get sufficiently expensive to be a worry?16:53
jcsacketti'm not saying it will. i'm not familiar with this part of the codebase, so i'm wondering.16:54
=== almaisan-away is now known as al-maisan
abentleyjcsackett, I don't know what you're asking.  Are you wanting me to run an EXPLAIN on the query?16:54
jcsackettabentley: i'm just double checking performance concerns.16:55
abentleyjcsackett, what kind of answer are you looking for?16:55
jcsackettabentley: builds is a SQLMultipleJoin; i'm wondering if when self.build gets called we may return a huge rowset in some cases.16:57
abentleyjcsackett, as I said, it's always one or zero results.16:57
=== al-maisan is now known as almaisan-away
jcsackettabenltey, ah! i thought you meant from_build was called one or zero times.16:57
abentleyjcsackett, there is a comment in the code saying it shouldn't be a multiple join.16:57
jcsackettabentley: yes, i see now.16:59
jcsackettapologies for the confusion.16:59
abentleyjcsackett, cool16:59
=== Ursinha is now known as Ursinha-lunch
jcsackettabentley: r=me. i need follow up, as a mentee :-P. bac, can you follow up on https://code.launchpad.net/~abentley/launchpad/build-mail3/+merge/47679?17:01
=== jcsackett changed the topic of #launchpad-dev to: Launchpad development https:/​/​dev.launchpad.net/​ | PQM is open | firefighting: - | On call reviewer: allenap, jcsackett* | https://code.launchpad.net/launchpad-project/+activereviews
=== matsubara is now known as matsubara-lunch
abentleyjcsackett, I've added a comment per your suggestion.17:14
jcsackettabentley: thank you. :-)17:19
=== deryck[lunch] is now known as deryck
=== beuno is now known as beuno-lunch
deryckhurrah!  Windmill re-enable branch finally made it through.17:43
=== matsubara-lunch is now known as matsubara
lifelessjml: morning18:26
lifelessderyck: cool18:26
jmllifeless: hi18:26
lifelessjml: would you like a catchup call - we seem to be trading 1-liners this week18:27
jmllifeless: Yes, I'd like one. Only have 15mins though.18:27
jmllifeless: sure.18:28
=== beuno-lunch is now known as beuno
thumperflacoste: ping18:46
bachi abentley18:59
abentleybac, hi.19:00
bacabentley: i'm trying to mentor jcsackett's review.  the diff in the MP is overwhelmed by lint changes.  i've tried to get a diff with just the non-lint changes but failed.  could you easily whip one up and paste it?19:00
abentleybac, not sure.  I don't think they were separate commits, or anything.19:02
abentleybac, how's this? http://pastebin.ubuntu.com/559153/19:04
bacabentley: 228 is much better than 1654!  :)  thanks!19:05
aromanhello, all! Is there any way of seeing how many people are using a PPA of mine/a package? Or even just seeing how many people have branched from bzr or downloaded a .tar.gz? Thanks!19:08
lifelessaroman: there is an LP API that can get you download statistics for a PPA19:13
flacostehi thumper19:14
aromanlifeless: ah, but there's no sort of graphical frontend to see that information?19:14
lifelessnot yet19:14
thumperflacoste: can you talk briefly about your review?19:14
aromanlifeless: ah, alright, well i'll check that out then. I assume it's python, right?19:14
lifelessflacoste: hi; can I get a small timeslice today ? (but not for ~30)19:14
flacostethumper: sure, mumble?19:14
lifelessaroman: its a RESTful api, but we have a python library you can use19:14
flacostelifeless: how small?19:15
lifelessflacoste: 10 min ?19:15
aromanlifeless: excellent, thanks!19:15
lifelessflacoste: crossing t's dotting i's on the loggerhead thing19:15
LPCIBotYippie, build fixed!19:27
LPCIBotProject devel build (397): FIXED in 4 hr 31 min: https://hudson.wedontsleep.org/job/devel/397/19:27
LPCIBotLaunchpad Patch Queue Manager: [r=gary][ui=none][bug=704685] BugSubscription.bug_notification_level19:27
LPCIBotis now exposed in the devel webservice API.19:27
gary_posteryay gmb !19:27
gary_posteroh, boo19:28
gary_posterthat's hudson19:28
gary_posteron buildbot, it is being rolled back :-/19:28
lifelessoh, why?19:28
lifelessah scaling test failed19:30
lifelessI'd like to make those more reliable and isolated19:30
lifelessthey are just a little flaky atm19:30
lifeless(different standalone vs in a larger run)19:30
lifelessI think they are still a net win19:31
lifelessbut it can be frustrating getting them going19:31
lifelessallenap: around?19:38
=== Ursinha-lunch is now known as Ursinha
jtvbigjools: got a minute for a question about FTPArchiveHandler?19:58
bigjoolsjtv: yup19:58
jtvI was just wondering… it uses the LoopTuner all over the place already, but only for gc.19:59
jtvWas it originally meant to commit transactions as well?19:59
bigjoolsno, it's writing out files for apt-ftparchive to use20:00
jtvOkay, but it's not writing to the DB that I can see and it's a big suspect in the long-transaction crime mystery I'm pursuing.20:01
jtvSo it'd seem that a few commits would be a relatively harmless way to get around the timeouts.20:02
jtvI was thinking perhaps I could force the main body of work on the slave store, and throw in a few commits.20:03
* bigjools thinks20:03
bigjoolsthat code is probably the bit I understand the least in soyuz20:03
bigjoolsit's never gone wrong since cprov left so I never had to look at it :)20:03
bigjoolsjtv: I'm a bit scared of using the slave store in case of replication delays20:05
jtvScared is sensible.  But what's there to replicate?20:06
bigjoolsall the publishing data it relies on, which is fast-changing20:06
jtvI see.20:07
bigjoolsjtv: it would be a good start to work out if it really is keeping a transaction open20:08
bigjoolsif it's not writing to the db can we change the $mumble20:08
jtvIt has to be.  It's not committing anywhere, and it's looping over lots of packages AFAICS.20:08
jtvThe $mumble?20:08
bigjoolsyeah, the thing20:09
jtvOh, the thing.20:09
jtvThe database policy?20:09
bigjoolsisolation ,even20:09
jtvIsolation level?20:09
bigjoolsdunno if that would affect it20:09
jtvThat would affect reads, not too much to do with writes.20:09
bigjoolscan we connection r/o to the master?20:09
jtvI don't think we can, no.20:10
jtvSomething we could try though is run a test with a slave-only policy, and see what fails.20:10
bigjoolsI see why you suggested the slave now20:11
allenaplifeless: Hi.20:11
allenaplifeless: Is this about the BugMessage crack?20:11
jtvbigjools: Well it also helps scale-out and locking, of course.20:11
lifelessallenap: though I'm not sure its crack L)20:12
jtvbigjools: unless there's something that changes the database and then immediately runs the script, we should get about as consistent a view of the database from a slave as we do from the master—just slightly older.20:12
allenapI'm reading the code now to try and remind myself of the weirdnesses.20:13
lifelessallenap: so20:13
lifelesswe decorate Message to make it an IndexedMessage but the actual relation is BugMessage which we don't expose at all20:13
flacostethumper: i'm back20:13
lifelessthats beside the point though20:14
jtvbigjools: "the thing"—https://www.ubersoft.net/comic/hd/2011/01/next-time-try-index-cards20:14
thumperflacoste: want to continue?20:14
flacostethumper: sure thing20:14
lifelesswe can still decorate but get the indices from the db once they are cached20:14
bigjoolsjtv: the thing I am scared of is writing out inconsistent indices in the Ubuntu archive.  That would be fairly nasty.20:15
bigjoolsnow while I am scared I am not sure how realistic the chances of that are20:15
jtvIn that case, we could try moving to the slave and _not_ committing.20:16
jtvIf that works out we'd still get a single huge transaction, but also a guarantee that it takes no write locks.20:17
allenaplifeless: Yeah, that sounds sane.20:17
bigjoolsjtv: well publish-distro does do commits20:17
jtvYes, just not in its huge loops.20:17
bigjoolsafter a-f runs20:17
bigjoolsit needs to mark stuff as publishe20:17
lifelessallenap: the goal is to be able to do a range query on BugMessage and then get just the Messages we want20:18
jtvbigjools: I guess that happens all the way at the end?20:18
lifelessallenap: which will also let us move to ajax population of pages like bug 120:18
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I20:18
allenaplifeless: Judging by Bug._indexed_messages, that should work well.20:18
lifelessallenap: indeed; I wrote Bug._indexed_messages in october or so as part of optimising within the prior schema20:19
allenaplifeless: Like load-on-scroll?20:19
allenaplifeless: Yeah, it looked like one of yours :)20:19
lifelessallenap: hah! I'll take that as a complement.20:19
lifelessallenap: and yes, load on scroll20:19
jtvbigjools: if we did the actual work on the slave but the marking-as-published on the master, we'd risk marking a slightly newer version as published than we actually published.  Is that a problem?20:19
lifelessallenap: should be easy if we can get one message + adjacent actions pretty cheaply.20:20
bigjoolsjtv: there's 4 steps in the publisher: 1) write files to pool, 2) domination, 3) generate files for a-f and run it, 4) write release files20:20
bigjoolsjtv: big problem ,yes20:20
allenaplifeless: Adjacent actions?20:20
jtvbigjools: though strictly speaking, I would expect that that problem already exists to about the same extent20:21
lifelessallenap: look at a BugTask:+index page20:21
lifelessallenap: it shows action action message action message etc interleaved20:21
lifelessallenap: it pretends there is one sequence20:21
bigjoolsjtv: actually, the publishing record would only get written by  this process anyway at this point in its lifecycle20:22
allenaplifeless: Ah, yes, I wrote a lot of that ;) I was having an association fail on the word "actions".20:22
allenapWell, not a lot, some.20:22
jtvbigjools: assuming there's no overlapping runs, I guess it'd only be a problem if there were a few last changes while the script was running, and then nothing before the next run (so it'd think it was already published or something).20:22
allenaplifeless: Are you going to work on this, or are you softening me up to do it?20:23
flacostelifeless: i'm free whenever you are20:23
lifelessallenap: I'm helping on the oops/performance aspect20:23
lifelessallenap: future stuff will be feature cards I suspect, unless you were to have a sudden fit of zomg I want to do this20:24
lifelessflacoste: voice?20:24
flacostelifeless: sure20:24
allenaplifeless: Okay, so you'll get the index into the database, and the get-adjacent-actions and load-on-scroll stuff is up to others?20:24
flacostelifeless: skype me20:25
lifelessallenap: yeah, I don't have long enough timeslices to do larger stuff20:25
allenaplifeless: That's cool, I just wanted to be sure what's expected of me.20:26
lifelessbe excellent20:27
lifelessthats it20:27
allenapHeh :)20:28
bigjoolsjtv: so it marks everything published before a-f runs20:30
bigjoolsthere's the long transaction20:30
jtvbigjools: it may help my understanding if I see what that entails… do you happen to know where that is done?20:30
bigjoolsjtv: look at lib/lp/archivepublisher/publishing.py20:31
* jtv looks at lib/lp/archivepublisher/publishing.py20:31
bigjoolsand scripts/publish-distro.py which is what calls it20:31
bigjoolsC_doFTPArchive is for Ubuntu, C_writeIndexes is for PPAs20:32
jtvAh, that helps20:32
jtvSo the marking-as-published… what do I look for?20:32
bigjoolsfollow it down to there from distroseries.publish()20:33
jtvah cool20:33
jtvBut I think the script commits soon after that happens anyway.20:33
bigjoolsthe latter being called from A_publish()20:33
jtvThe problem is in C_doFTPArchive20:34
jtv(I think)20:34
bigjoolseach stage is wrapped it try_and_commit()20:34
jtvahh got it: setPublished20:37
jtvAnd we want to publish a state that's no older than that datepublished timestamp, right?20:38
bigjoolsjtv: so my concern is that we only see half of the records that were just set published if C_doFTPArchive is using a different store20:40
bigjoolsor maybe even none20:41
bigjoolsbut that's extreme20:41
jtvI wonder if we could set "now minus replication lag" as the publication date.20:43
bachi deryck, you still around?20:43
bigjoolsthe date is not a concern20:43
bigjoolsit's the inconsistency20:44
jtvbigjools: would that change though if we otherwise ran everything on the slave store?20:45
bigjoolsbecause what can happen is that we write the pool file, miss it in a-f and then write the release file with it in20:45
bigjoolsthat would be quite disastrous20:45
jtvWhat's a-f stand for by the way?20:45
bigjoolsApt FTPArchive20:45
bigjoolsin that case we'd end up with checksums that are incorrect20:46
jtvYou're saying the whole pool file might be lost?  Or an individual package that's in there?20:46
bigjoolsa-f writes the indexes - with replication lag it could miss something20:47
bigjoolsthen if that something replicates before we get to stage "D" where it writes the Release file, the Release file will be wrong compared to the index20:47
bigjoolsI think20:48
jtvWell if it's that complicated I probably can't afford to mess with it anyway.  :/20:48
bigjoolsthe publisher is *the* most critical part of soyuz, if it goes wrong we can do a lot of damage20:48
bigjoolswhich is why I am rather conservative in this area :)20:49
jtvIf we could move the whole thing minus something small over to the slave, we'd have a consistent view (just a slightly older one) and no worries.  But if there's any sort of read-modify interaction it gets riskier.20:49
bigjoolswe should go through it sometime and record all the interactions20:50
jtvbigjools: would it make sense to do a trial run with a slave-only policy and setPublished disabled?20:53
bigjoolsI dont think we have any tests that do everything at once like that, so we'd need to check the archive state manually20:54
jtvMeaning we'd probably miss something?20:55
bigjoolspossibly but if you point an apt client at the archive it would soon belly-ache20:57
bigjoolswe should get wgrant to comment on this20:58
wallyworld_thumper: StevenK: we having standup?21:04
wallyworld_thumper: i'm talking, jsut a sec21:05
wallyworld_thumper: you ok, my sound backend bad21:05
thumperwallyworld_, StevenK: FYI https://code.launchpad.net/~thumper/launchpad/refactor-lazrjs-text-widgets/+merge/4763421:05
StevenKthumper: I can hear you too21:05
wallyworld_thumper:  https://code.edge.launchpad.net/~wallyworld/launchpad/recipe-find-related-branches/+merge/4736721:17
=== salgado is now known as salgado-afk
huwshimiUgh, so my laptop has decided that when I'm logged in I should only be able to type numbers :(21:30
StevenKYet, you're typing not numbers. :-)21:30
huwshimiStevenK: I am using my netbook to whinge about it21:31
wallyworld_thumper: https://pastebin.canonical.com/42451/21:33
wgrantbigjools: Hi21:34
bigjoolsmorning wgrant21:34
wgrantbigjools: This whole discussion makes me cry.21:37
wgrantWhy do we want to move it to the slave?21:39
bigjoolswgrant: no doubt21:39
benjihuwshimi: numlock?21:39
huwshimibenji: Thanks, I just figured that out right then.21:39
wgrantWe *could* do it, and things would remain consistent, but some things would show as published when they weren't.21:39
StevenKthumper: I've been trying!21:39
bigjoolswgrant: jtv has been investigating but he wants to get rid of a long-running transaction21:39
wgrantAnd the latency would be two hours.21:39
huwshimibenji: They could have at least labelled it :)21:40
bigjoolswgrant: yes that's exactly my concern in addition to incorrect indices vs release files21:40
wgrantbigjools: We may have extra stuff on disk, but we may also be removing stuff from disk later that is still referenced by the indices.21:41
wgrantSo no, we cannot sensibly use the slave.21:41
wgrantHowever, yes, we can remove the long-running transactions by making the publisher not take 300 years.21:41
wgrantBut that is a fair bit of work.21:41
lifelesswe could add a ro connection to the master21:41
bigjoolsabout 300 years21:41
wgrantlifeless: Is it only R/W transactions that are the problem?21:42
lifelesswgrant: no21:42
wgrantThat's what I thought.21:42
lifelesswgrant: but they are a bigger problem than r/o transactions21:42
lifelessr/o transactions prevent index and row gc (mvcc chaff)21:42
lifelessr/w transactions cause related row exclusive locks21:43
lifelessbut its the actual *changes* in r/w transactions that matter21:43
* bigjools gets tests working with a version column of type debversion \\o/21:43
lifelesslength is just a (poor) proxy for predicting problems.21:43
wgrantThere should be none. But it would be nice to verify that.21:43
wgrantbigjools: Yay!21:43
bigjoolsnow, to optimise the queries21:44
StevenKthumper: So I use Mumble on the old laptop the audio is choppy and I can talk, if I use the new laptop the audio is excellent and I can't talk21:44
bigjoolswgrant: btw, the 2-builds-per-builder happened again today21:45
wgrantbigjools: Same builder?21:45
bigjoolsno,  after talking to lamont it turns out that it happens when he kills a long-running stuck build21:45
wgrantI noticed 6 or so builders earlier disabled with strange messages.21:45
wgrantBut they were all happy once I re-enabled them.21:46
bigjoolsbecause we rely on aborting nonvirt builders21:46
lifelessflacoste: poolie: draft sent to you21:46
bigjoolsso I am going to change to code disable the builder it we see it aborting21:46
wgrantlifeless: Do you know what's going on with sinzui's mailing list QA?21:46
wgrantbigjools: Why?21:47
bigjoolsabort does not work on nonvirts21:47
wgrantABORTING will drop to ABORTED, modulo a slave bug which doesn't properly kill sbuild because of a missing trap.21:47
bigjoolsso until that's fixed...21:47
lifelesswgrant: yes21:47
lifelesswgrant: can I brain dump and have you chase?21:47
wgrantlifeless: That is what I was hoping for.21:47
lifelesswgrant: ok so lp prod configs was broken for qastaging mailman21:48
lifelessthe qa port is 909721:48
wgrantbigjools: Until that's fixed, buildd-manager will ignore the builder and there is a nagios check to tell lamont to fix it.21:48
lifelessit said 809721:48
lifelessthere is a branch to fix that21:48
wgrantI believe that got reviewed and landed.21:48
lifelessso if it has21:48
wgrantIt should be good?21:48
lifelessthen it should be deployed and the mailman log should be calling into the api with the newer code21:49
wgrantIt doesn't seem to be landed. I might fix that.21:49
lifelessremaining steps21:49
wgrantI've only run mailman locally a couple of times, so I'm not 100% on how exactly it works, but I guess I'll work it out.21:49
wgrantlifeless: qastaging should automatically update configs with its usual update, right?21:51
lifelesswgrant: yes21:51
lifelesswgrant: so, 1) get the right port out21:51
lifeless2) make sure its querying members (via the logs)21:52
lifeless3) profit21:52
wgrantGreat, thanks.21:52
wgrantIf all goes well, we can deploy in a couple of hours :)21:52
StevenKlifeless: So, the query Storm is creating is: SELECT SourcePackageRecipe.build_daily, SourcePackageRecipe.daily_build_archive, SourcePackageRecipe.date_created, SourcePackageRecipe.date_last_modified, SourcePackageRecipe.description, SourcePackageRecipe.id, SourcePackageRecipe.is_stale, SourcePackageRecipe.name, SourcePackageRecipe.owner, SourcePackageRecipe.registrant FROM SourcePackageRecipe LEFT JOIN SourcePackageRecipeBuild ON SourcePac21:53
StevenKHm. That was a bit longer than it looked in the terminal21:53
wgrantbigjools: We haven't purged non-main indices for existing PPAs yet, have we?21:54
lifelesswgrant: new ppas won't have them at all21:54
wgrantI know.21:54
wgrantIt broke a few things :)21:54
wgrantBut they're all fixed now.21:54
wgrantBut we were also going to manually remove the old ones.21:55
flacostelifeless: replied21:56
lifelesspoolie: hi21:57
bigjoolswgrant: no21:58
bigjoolswgrant: go ahead and clean up, then we can re-enable the daily job21:58
StevenKlifeless: O hai. I have added a recipe with no builds and dropped the query and it returns 0 rows22:40
lifelessStevenK: you need to paste the query without getting cut off :)22:40
StevenKlifeless: http://pastebin.ubuntu.com/559270/ ; I've taken what LP_DEBUG_SQL gave me, replaced the %s's and dropped most of SourcePackageRecipe's columns from the SELECT22:42
jtvhi wgrant!22:43
StevenKIf I drop the extra JOINs, it returns a row22:43
wgrantMorning jtv.22:44
jtvwgrant: bigjools & I were looking at the long transactions in the archive publisher earlier.22:45
wgrantjtv: So I saw.22:45
jtvYou're everywhere.22:45
wgrantWe cannot sanely move it onto a slave.22:45
jtvI was wondering whether we could move essentially the whole process over to a slave without risking inconsistencies.22:45
wgrantWe need to make it take less insanely long.22:45
jtvThat answers that.22:46
bigjoolsjtv: wgrant made some more scary points that I'd not mentioned in addition to my scary ones22:46
jtvOh good22:46
jtvthat sounds like fun.22:46
jtvMaking it take less insanely long looks somewhat doable, but I'll need a way to test.22:46
jtvIt's FTPArchiveHandler.run that takes the bulk of the time, right?22:46
wgrantThere are two things that take forever: file list generation, and a-f itself.22:47
wgrantThe latter can be parallelised.22:47
lifelessStevenK: I'm fidddling22:47
bigjoolswgrant: I'd like to see a-f and NMAF in a race22:47
lifelessStevenK: trivially doing left joins all over works, but that can get inefficient22:47
jtvwgrant: and the former looks like it may be a typical naïve-fetch-inside-loop pattern.22:48
bigjoolsI think it would be close22:48
jtvIt's _almost_ a simple prejoin but there's an interaction with slicing to watch out for.22:48
wgrantbigjools: a-f is hundreds of times slower.22:48
wgrantNMAF is.22:48
StevenKNMAF is *slower*?22:48
wgrantjtv: Possibly. But the queries themselves are bad.22:48
wgrantStevenK: Yes, it issues many many many more queries.22:48
jtvwgrant: the pattern, or some of the individual ones?  If you've got a list of known troublemakers, that'd help me see.22:49
wgrantjtv: I don't recall exactly. But I believe it's fairly efficient query-count-wise, but the queries are not quick.22:50
wgrantIt's been more than a year since I seriously tried optimising this phase of the publisher.22:50
wgrantI got a *long* way, but it was sort of full of hacks.22:50
wgrantDown to 3-4 seconds for each index file, on NMAF, for 1.5x primary's size.22:51
wgrantcprov also optimised the a-f file list code a lot just before he left.22:51
jtvMy may suspect based on following the arrows in the code was FTPArchiveHandler.publishFileLists.22:51
jtvThat contains some loops over all source/binary packages, I believe.22:52
wgrantYes, but that uses the views.22:52
bigjoolswhich are evil22:52
* jtv crosses himself22:52
wgrantIt should not do any queries.22:52
jtvLuckily I made merit by visiting the temple: saw a working Difference Engine 2 yesterday.22:52
* bigjools looks for garlic and silver22:52
wgrantUntil right at the end when I do the disabled chekc.22:52
wgrant(which is only once per pocket-das)22:52
jtvWhat's a pocket-das?22:53
wgrant(PackagePublishingPocket, DistroArchSeries)22:53
StevenKFor instance22:53
wgrantHow many cores does cocoplum have?22:54
bigjoolsStevenK, wgrant: I am putting my debversion stuff on DF, please to not be touching for a bit22:54
wgrantbigjools: k22:54
=== matsubara is now known as matsubara-afk
bigjoolsunless you guys are using it in which case I can wait22:54
StevenKwgrant: 422:55
lifelessStevenK: SourcePackageRecipeBuild JOIN PackageBuild ON PackageBuild.id = SourcePackageRecipeBuild.package_build JOIN BuildFarmJob ON BuildFarmJob.id = PackageBuild.build_farm_job right join SourcePackageRecipe  ON SourcePackageRecipeBuild.recipe = SourcePackageRecipe.id22:55
lifelessStevenK: put that in from 'FROM' ... 'WHERE'22:55
jtvwgrant: it looked to me as if, unless we mess with the code's structure which I'm reluctant to do, it should do a bunch of batched prefetch queries in each iteration of its loop-tuner callback.  If it's using views, maybe that's worth eliminating as well.22:56
wgrantjtv: Because it's using views, it's already entirely prefetched.22:56
wgrantAll you can do is optimise the view.22:56
jtvActually, prefetching may be what's slowing it down then.22:56
wgrantHowever, file list generation is only a couple of minutes.22:57
wgrantRunning a-f takes 15 or so.22:57
StevenKlifeless: That looks good. How do I tell Storm that?22:57
StevenKInstead of left?22:57
lifelessStevenK: works like LeftJoin in terms of function calls22:57
lifelessStevenK: yes; same parameters, same translation22:57
lifelessStevenK: play with it a little to get the invocation you need22:58
lifelessI'm just checking the query plan on staging22:58
lifelessso fast22:59
lifelessStevenK: you'll also want DISTINCT, because you only want one row per sourcepackagerecipe23:00
lifelessStevenK: as written, on staging, we can get23:00
lifeless id |    name     |  owner23:00
lifeless 15 | awn-testing | 138252423:00
lifeless(3 rows)23:00
StevenKlifeless: I'm still distilling your query into Storm. *Then* I'll worry about distinct23:04
lifelessStevenK: I'd do it for you, but known how this works will help you23:04
jtvwgrant: drat.  So to speed the code up significantly we'd have to run parallel apt-ftparchive instances on separate file lists?23:06
wgrantjtv: Yes. We already have separate file lists, so it's fairly easy.23:07
StevenKlifeless: Storm has SELECT ... FROM SourcePackageRecipe; whereas your query has SELECT ... FROM SourcePackageRecipeBuild; is that pertinent?23:07
jtvwgrant: and then… just concatenate the outputs?23:07
wgrantjtv: No. Each file list results in one index.23:08
wgrantIf you look at the log, you'll see it generates one index for (maverick-release, source, main), another for (maverick-release, i386, universe), etc.23:08
wgrantEach of those is big.23:09
wgrantAnd each of those can be done in parallel.23:09
lifelessStevenK: yes23:09
lifelessStevenK: or probably yes, show me thje FROM..WHERE bit ?23:09
StevenKlifeless: http://pastebin.ubuntu.com/559277/23:10
jtvwgrant: ah so parallel runs per architecture, plus one for source, and they should all overlap fairly well.23:11
wgrantjtv: Yes.23:11
jtvThe hard part would probably managing the parallel processes.23:11
wgrantYes. And that's not terribly hard.23:11
wgrantIn the run I'm looking at, running a-f takes 16 minutes. File list generation takes 1.5.23:12
StevenKlifeless: I suspect you got distracted?23:30
lifelessStevenK: born distracted23:37
lifelessStevenK: ok23:42
lifelessso that thing you pasted means23:42
lifeless'select the things joined together where there are 0 or many sourcepackagerecipes'23:43
lifelesswhat you want is23:43
lifeless'select source package recipes where there are 0 or many (things joined together)'23:43
lifelessright joins allow NULL on the left hand side23:43
poolielifeless, spm, as a specific next step on bug 701329, can we get haproxy logs?23:43
_mup_Bug #701329: error: [Errno 32] Broken pipe <oops> <Launchpad itself:Triaged> <loggerhead:Triaged> < https://launchpad.net/bugs/701329 >23:43
lifelessthat is, they include 'every row on the right hand side'23:44
lifelesspoolie: can we not just catch the exception ?23:44
poolieor, try to titrate them to a level where they tell us about interesting errors without flooding the system23:44
lifelesspoolie: what are you trying to determine23:44
pooliewhy haproxy closed the connection23:44
poolieand from which ip it came23:44
lifelesspoolie: can you back it out a step higher23:45
lifelessI don't understand why this isn't a fairly direct code change23:45
poolieso, to close this particular bug, we can, say23:47
pooliejust make it not oops when the connection is abruptly closed?23:47
poolieperhaps it should log a warning-type error23:47
pooliethat seems pretty easy23:47
lifelessnone of our other servers log on this23:47
poolieit makes me wonder if we are papering over a problem23:47
lifelessnone of our other servers are experiencing it23:47
poolieso i'd like to see a bit more data about why the connection is being dropped23:47
pooliehaproxy is the obvious place to look for that23:48
poolieand, i think it is a bit weird to have a production service where logging is entirely disabled23:48
poolie(perhaps "it's weird" is not a reason to change it but i think it's reasonable to ask)23:48
lifelessI thought you were told that it was a volume problem ?23:48
lifelesshaproxy being a bit of a firehose23:49
lifelessif you're working on this bug23:49
lifelessthats cool23:49
poolielifeless, i'd guess that zope etc don't traceback on epipe, and i agree that in general loggerhead shouldn't23:49
lifelessuhm, what can I do to hepl.23:49
lifelessfirstly, have you looked at some of the OOPSes?23:49
pooliei'd like to turn logging >=WARNING23:49
poolieand see if that is in fact a firehose23:49
poolieif it is, that's a bit alarming, and i'd like to just see what it's logging23:50
pooliethen we can turn it off again23:50
poolieis that ok?23:50
lifelessI've no objections in principle, though the losas generally make informed decisions about this sort of thing :)23:50
pooliespm, do you mind trying that?23:51
spmwe start with "no" and negotiate from there.23:51
spmpoolie: nope, worth a shot. :-)23:51
* StevenK was waiting for "Depends. Do you have cake?"23:51
pooliethe haproxy manual says you can set the log leevl23:51
spmStevenK: it's poolie. he lives closer than wgrant, and rides a motorbike ie Hells Angel in training. I've learnt to be nice to him.23:52
lifelesspoolie: so23:52
lifelesspoolie: looking at  https://lp-oops.canonical.com/oops.py/?oopsid=1836CBA123:52
lifelesspoolie: there are several things that make this hard to analyze23:52
lifelesspoolie: if I was working on it, I would fix those first23:53
lifelesspoolie: firstly there is no total time recorded23:53
* StevenK suspects that poolie is shinier, due to lifeless completly context switching and goes afk for ten or so23:53
lifelesspoolie: nor any timeline (which we could use to track xmlrpc / bzr call overhead if we wanted to)23:53
pooliei've learned to front-load anything that may need SA action23:53
lifelessStevenK: sorry, I thought I answered you?23:53
lifelessStevenK: @10:4423:54
lifelesspoolie: if we had timing data, we could assess whether this is really a timeout or not23:54
lifelesspoolie: in fact, we could add a thing to say 'if the time is > some N, convert to a timeout'23:55
lifelesswe probably need to expose an API call to ask for the timeout23:55
pooliebut if haproxy is saying "request blah blah timeout, killed"23:55
lifelesse.g. 'what should my timeout be please?'23:55
pooliethat seems easier23:55
lifelesspoolie: I worry that we're going to need a lot of correlation to match up all 15K a day23:56
lifelesspoolie: gathering data at source seems more useful (to me)23:56
lifelesspoolie: anyhow, thats just what I'd do if I was working on it23:56
lifelessnote that haproxy probably won't tell us23:57
lifelessif this is coming in behind haproxy23:57
lifelessspm: how often does nagios query ?23:57
spmi think it's up to 3 times every 10 minutes23:58
poolielifeless, i don't want to match up all of them, i just want to know if haproxy thinks its client is dropping the connection, or if it is dropping the connection itself23:58
poolieor if it's oblivious23:58
poolieit shouldn't be too hard to answer23:58
spmbut probably easier to look in the logs and count23:59
lifelessI am open to many ways to figure out whats up23:59
* thumper is busy writing real documentation23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!