#launchpad-dev 2009-10-26
<wgrant> lifeless: I see you added Intrepid to dev.lp.net/FAQ. LP is no longer supported on Intrepid, so it is tempting to remove it.
<lifeless> wgrant: go for it
<lifeless> wgrant: I was only making the bzr recommendation more complete
<wgrant> lifeless: I've replaced it with Hardy.
<lifeless> recommending 1.17 was nuts
<lifeless> :)
<thumper> btw, it is a public holiday in nz today :)
<lifeless> gstq?
<rockstar> mwhudson, around?
<al-maisan> Good morning
<MTecknology> so... now I see why that project I was going to work on got "fast-tracked"
<MTecknology> hopefully I can still help out :)
<wgrant> What got fasttracked?
<MTecknology> the upstream bug forwarding
<adeuring> good morning
<al-maisan> moin adeuring
<adeuring> hi al-maisan!
<mrevell> Morning!
<deryck> Morning, all.
<mrevell> deryck: Are you lot still on summer time?
<deryck> mrevell, yes, until this coming weekend, I believe.
<mrevell> deryck: Ah, that explains why you seem to be an hour early :)
<deryck> mrevell, :)
<deryck> I almost got up an hour earlier today, too.  That would have really messed with you.
<mrevell> :)
<BjornT_> deryck: gmb is away this week, right? could you please take a look at my post-merge review of r9757 that i sent to launchpad-dev? it needs some attention, since it might break the JS in some browsers.
<deryck> BjornT_, sure, I can look at that.
<deryck> BjornT_, yeah, I'll fix that today.
<wgrant> No bigjools today?
<noodles775> wgrant: he won't be available until next Monday.
<wgrant> noodles775: Ah, thanks.
<gary_poster> barry: morning. :-) should I send out the sprint announcement email?  Do you have anything to add?
<barry> gary_poster: i sent it out on saturday (i thought)
<gary_poster> barry, oh you did!  sorry, I was trying to keep an eye out for it but missed it.  thank you!
<barry> np!  please add anything i missed
<gary_poster> :-) looks good to me
<barry> thanks!  can't wait for it to land :)
 * mrevell --> rebooting
<EdwinGrubbs> bac: the solution I had for aligning things to the right conditionally is the table-actions css class. Its basic structure is <div><table/><ul class="table-actions"/></div>. If the table doesn't exist, the links are aligned to the left, otherwise to the right.
<EdwinGrubbs> bac: you can see an example of this for the "Create milestone" link on https://edge.launchpad.net/launchpad/2.1 and https://edge.launchpad.net/launchpad/trunk
<intellectronica> so i'm trying to hack on karmic, but i don't seem to have the module cProfile. does anyone know which package i need to install? also, what needs to be fixed for it to be installed as a dependency?
<BjornT_> intellectronica: cProfile.py is a symlink in lib/
<intellectronica> hmmmm ... interesting, so why am i missing it?
<BjornT_> intellectronica: where does the symlink point to? does that file exist?
<barry> bac: it's been 2 weeks since i did a 'make schema'.  does it ever end?  vacuuming steps are taking FOREVER
<leonardr> barry, i have a wsgi question
<leonardr> i've defined a class that implements a 3-argument __call__
<leonardr> then instantiated the class and passed the instance in to wsgi_intercept as a wsgi application
<leonardr> my __call__ is being triggered, but only one argument is being passed in (Self)
<leonardr> this goes against what i know about wsgi and other wsgi code i've written
<leonardr> do you know what might be wrong?
<barry> leonardr: off-hand i don't.  i'd be happy to look into it with you if you want
<leonardr> barry: i'll commit and push
<barry> leonardr: okay
<stub> barry: define FOREVER?
<leonardr> barry: lp:~leonardr/launchpadlib/wsgi-fake-launchpad
<leonardr> the wsgi class is defined in testing/helpers.py
<leonardr> set up in tests/test_browser.py
<barry> stub: an hour at least so far
<barry> leonardr: grabbing
<jml> hey guys
<stub> barry: That is very broken. I recommend rebuilding the database, as alternative methods of recovery will also likely take hours.
<jml> I've linked to a non-existing page https://dev.launchpad.net/UserStories
<barry> stub:  is rocketfuel-setup enough to do that?
<jml> I think it would be a good idea to edit that page to provide a general description of what user stories are and how we use them.
<jml> flacoste, ^^
<stub> barry: Doesn't look like it.
<flacoste> jml: ok, i can have a stab at it this afternoon
<barry> stub: okay.  i know there's a wiki page with the gory details.  will check that; thanks
<jml> flacoste, thanks.
<stub> barry: https://dev.launchpad.net/DatabaseSetup , step 4 is what you want I think
<barry> stub: awesome, thanks
<stub> barry: And the subsequent steps too
<deryck> leonardr, ping
<leonardr> deryck: hello
<deryck> leonardr, hi.  in the js client I've got lint warnings with for-in constructions, in lines unrelated to my changes.  Easy to fix, but wanted to check with you first.  See: http://pastebin.ubuntu.com/302098/
<deryck> leonardr, just wanted to make sure this wasn't intentionally left out, before I fixed it. :)
<leonardr> deryck, i don't see a problem with fixing that
<deryck> leonardr, cool, thanks.
<barry> leonardr: going on a c/c so it might be a little while
<leonardr> barry, ok
<bac> sinzui: i opened bug 461164 to track the +download issue.  i didn't include your comments on the design decision as it is outside the scope of this fix.
<mup> Bug #461164: Project +download page has extra vertical space <Launchpad Registry:New> <https://launchpad.net/bugs/461164>
<leonardr> barry, i think i'm misusing wsgi_intercept
<sinzui> bac: That is a prudent decision. Thanks
<leonardr> barry: yes, i've solved it
<barry> leonardr: great!  glad i could "help" ;)
<bac> sinzui: in https://bugs.edge.launchpad.net/launchpad-registry/+bug/461164 i don't understand the issue with breadcrumbs
<mup> Bug #461164: Project +download page has extra vertical space <Launchpad Registry:New> <https://launchpad.net/bugs/461164>
<sinzui> bac there is an xmlns on the ol. This implies that the template did not declare the namespace, so it was not stripped
 * sinzui cannot find what renders the breadcrumb <ol>
<bac> sinzui: every template i've looked at has that problem:  product-index and person-index
<bac> sinzui: check out https://edge.launchpad.net/~bac
<sinzui> yes, it is a universsal screwup
<jml> MTecknology, hi
<bac> sinzui: but the templates all start with xmlns="http://www.w3.org/1999/xhtml"
<sinzui> I do not think @@+hierarchy does
<MTecknology> jml: g'morning
<sinzui> actually, launchpad-hierarchy does, but something must be wrong since it is not handled like other pages
<jml> MTecknology, how's things?
<MTecknology> jml: good, you?
<jml> MTecknology, good :) wishing I were spending more time in emacs and less time in GMail :)
<MTecknology> emacs ... makes little children in nigeria sad
<jml> MTecknology, Uganda, I think you'll find.
<MTecknology> :P
<jml> MTecknology, how's the project going?
<MTecknology> which one?
<MTecknology> the bug forwarding?
<jml> MTecknology, well, the uni project in general
<MTecknology> so far they're still pending approval at the state
<sinzui> bac: I would not spend more than 15 more minutes fixing the breadcrumb namespace. It is a separate problem, and I do not see anything obvious in the template
<jml> MTecknology, ahh ok.
<MTecknology> jml: Since you guys are taking up that module, by part in it won't be so bad. I just need to keep up on what's going on and do my best to help out in a few places and make nice pretty reports
<MTecknology> not module, project
<MTecknology> I've been messing with modules for ~20hr so far this week
<barry> stub: something's still broken.  i believe i've done all the steps on DatabaseSetup.  'make schema' runs cleanly.  'make run' fails: http://paste.ubuntu.com/302138/
<stub> barry: Did you add the listed chunk to the *top* of pg_hba.conf, and did you 'pg_ctlcluster 8.3 main reload' ?
<MTecknology> jml: how busy you been?
<stub> (top == first uncommented line of course)
<barry> stub: um.  i did now. ;)  thanks!  it's working
<jml> MTecknology, more than usual :)
<MTecknology> jml: it's that gosh darned stock market i tellz u; it makes the world so so busy :P
<jml> MTecknology, heh
<MTecknology> apparently I don't get to slack off my senior year in college..
<jml> MTecknology, what is this world coming to!
<MTecknology> like, i know; right?
<MTecknology> I just need to learn how to set priorities and learn that sleep can't be the last or the one I mark 'when unavoidable' :P
<MTecknology> I say this while I chat online instead of paying attention in class
<MTecknology> food time - ttayl
<jml> MTecknology, ciao
<jml> g'night all
<mrevell> nytol
<MTecknology> ciao?
<mwhudson> good morning
<barry> sinzui: is this what you had in mind for bug 436503?  http://people.canonical.com/~barry/public.png http://people.canonical.com/~barry/private.png ?
<mup> Bug #436503: Team privacy portlet inconsistent with the rest <post-3-ui-cleanup> <trivial> <Launchpad Registry:Triaged by barry> <https://launchpad.net/bugs/436503>
<sinzui> Something is wrong
<sinzui> ah!
<sinzui> barry: we have two problems
<sinzui> barry: Your example shows something that we said we would remove months ago
<sinzui> barry: Can you provide an example or a Private team? We really need to drop private membership teams next week
<barry> sinzui: this bug is all about the ui
<sinzui> barry: I was expecting to see private in bold like we show private bugs, then I realised your example was not a private team
<sinzui> barry: yes, but your example is not a private team
<barry> sinzui: http://launchpad.dev/~myteam (well, it's a private membership team)
<sinzui> barry: private-membership should not exist we suck we said we would remove them in June. We double suck
<sinzui> barry: please show me an example of a TRUE private team
 * sinzui does not want to design for something that should not exist
<barry> sinzui: http://people.canonical.com/~barry/realprivate.png
<barry> sinzui: the ui changes are fairly simple and apply to all three PersonVisibility states
<sinzui> barry: the formatting is fixed, but that does not tell me it is private. I need to read the CSS to know what is happening. What does it take to say "Private team"
<barry> sinzui: do you still want it to also say "Viewable by team members" or delete that text and /just/ say "Private team"?
<sinzui> barry: that first part certainly is an understatement. Because the name is not knowable either
<sinzui> We do not say for private bugs: this bug is only viewable to subscribers
<sinzui> maybe we should do that though.
<barry> sinzui: i'd have to ask bac but i think that extra explanatory text for teams is there because no one really understand what a private membership team is.  ripping those out seems out of scope for this particular bug <wink>
<barry> sinzui: so i think we have a few ui choices here.
<sinzui> barry:  do you think we should state state <Public|Private> <object>. This <object> is visible to <subscribers|members|Illuminati>
<barry> sinzui: one possibility is that the text could just say "Public team" "Private team" "Private membership team" and have a help popup that contains that little explanatory text
<barry> sinzui: i'm not sure that it's always helpful to do that, no.  but on a help popup or balloon text, maybe so
<barry> sinzui: i would opt for keeping things simple on the page and elaborating in secondary text, e.g. (?) help
<sinzui> barry: I think the formatting that you are most concerned about is fine. I would prefer that the private team state is private, but I recognise that is a separate issue. r=me for the format change
<barry> sinzui: that's r=me for the screen shots i just posted? no additional change necessary?
<rockstar> Good morning mwhudson.
<sinzui> No. other changes. We need consistent messaging about what public/private and what that means to the users. That is a separate bug that affects all apps.
<barry> sinzui: +1
 * barry makes the tests pass
<sinzui> barry: ! destroy any story test that is checking markup.
<barry> sinzui: good idea
<sinzui> barry: Since we agreed to change the format, not the message, any test that  fails needs to made robust in case we change the formatting again
<barry> sinzui: note, we did slightly change the message.  for non-public teams it no longer prints the big fat <h2>
<barry> sinzui: but yes, i will make sure the tests are sane
<sinzui> barry: stories do not check markup.
<sinzui> barry: One of the goals of the 3.0 conversion was to remove any story that checks implementation details
<barry> sinzui: right.  well, we'll see what tests fail
<barry> sinzui: btw, i know what i want to work on next.  remember what i did for projects?  go to http://launchpad.net/~sinzui/+edit and scroll to the bottom of the page
<sinzui> barry: We need a bug. If you do that, can you also fix bug 113562, bug 273180, and bug 430680
<mup> Bug #113562: "Change Branding" is unclear for people <post-3-ui-cleanup> <story-logos> <ui> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/113562>
<mup> Bug #273180: 64x64 team branding image woes <post-3-ui-cleanup> <story-logos> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/273180>
<mup> Bug #430680: IPerson's +edithomepage is redundant <post-3-ui-cleanup> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/430680>
<barry> sinzui: +1
<sinzui> fab
<sinzui> barry:  the "woes" bug just requires that we explain where the images are used. We do not use an icon for a person today, but that may happen.
<thumper> morning
<rockstar> thumper, morning.
<rockstar> thumper, was there a holiday in NZ yesterday?
<thumper> yep
<thumper> labour day
<rockstar> thumper, ah.
<mwhudson> thumper: again
<rockstar> thumper, on the plus side, it didn't sound like we had a delay.  :)
<mwhudson> rockstar: i could hear you
<rockstar> thumper, shall I try and call?
<mwhudson> rockstar: but all thumper heard was plinky noises
<thumper> rockstar: yeah, or abentley
<thumper> rockstar: you try and see what you get
<thumper> abentley: still on for our normal chat?
<abentley> thumper: Sure.
<intellectronica> thumper: morning
<thumper> intellectronica: morning
<intellectronica> thumper: i hear you're finding difficulties getting up and running with integrating lazr-js widgets. i wanted to offer my help and more importantly, suggest that we work together on documenting the process for future generations
<thumper> intellectronica: on a call right now]
<wgrant> intellectronica: Can you land that branch for me?
<intellectronica> wgrant: oh, of course! i completely forgot that you need that landed by someone
<wgrant> intellectronica: Thanks/
<intellectronica> wgrant: what commit message should i use?
<intellectronica> wgrant: it will take a bit, since i need to run the test suite on it first, b.t.w
<wgrant> intellectronica: 'Add IArchive.debug_archive, pointing to the corresponding debug archive.', I suppose. Thanks.
<thumper> https://help.launchpad.net/HelpRotation
<thumper> mwhudson: you might want to choose a week before one is chosen for you :)
<mwhudson> thumper: done
<barry> bac: ping
<bac> hi barry
<barry> bac: hi.  are you ocr tomorrow?
<bac> barry: yes
<barry> bac: great!  i'll save this one for you.  it's easy; related to team visibility
<bac> barry: that's great.  get it to me early, though.
<MTecknology> Can we add a feature to LP so that after the bug report gets to be 100 lines long (where it breaks off and stops displaying all comments) when they add a comment thy will get this big massive page pop up that takes over their entire keyboard and screen and mouse and says "Are you sure this IS NOT a 'me too' post?" and then a description below and forces them to stare at it for 5min and then scroll through every comment says "no
<mwhudson> thumper: i notice a passing commit that makes Distribution.serieses a cachedproperty
<mwhudson> thumper: does this mean that there's a hack that can be deleted in branchlisting.py ?
#launchpad-dev 2009-10-27
<wgrant> Hm, nearly four hours since that ec2test probably started. Is PQM turned off, or does my branch just suck?
<spm> wgrant: it's stopped atm. not sure what/where/why yet...
<wgrant> spm: Ah, thanks. Is the queue visible somewhere
<wgrant> +?
<spm> https://pqm.launchpad.net/ - I belivee it's still openid locked tho
<mwhudson> it's not, but for no good reason
<wgrant> :(
<mwhudson> i guess we sometimes want to land security fix branches while they're private, but those should probably be cowboyed first anyway
<thumper> mwhudson: possibly
<thumper> mwhudson: probably even
<thumper> mwhudson: skype?
<mwhudson> thumper: sure
<thumper> https://devpad.canonical.com/~lpqateam/oops-summaries/lpnet-2009-10-14--2009-10-20.html
<mwhudson> stub: hello
<stub> hi
<mwhudson> stub: do you have any idea how to diagnose a timeout that is probably to do with locking?
<mwhudson> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1385XMLP15
 * stub has a look
<stub> mwhudson: I'd agree that you are probably attempting to update a locked row.
<stub> mwhudson: What do you need to diagnose? Do you need to confirm this, or track down what is holding the lock?
<mwhudson> stub: i guess there's no way to find out what locked the row after the fact
<stub> After the fact, yes. We would have to work out what needs to be logged to diagnose further, and add that to the OOPS reports, and wait for it to happen again.
<mwhudson> stub: i see
<mwhudson> stub: what/how can we log stuff?
<mwhudson> stub: is it possible to find out what has locked the row in an except TimeoutError: block or something?
<thumper> https://lp-oops.canonical.com/oops.py/?oopsid=1396EC76
<mwhudson> <stub> After the fact, yes. We would have to work out what needs to be logged to diagnose further, and add that to the OOPS reports, and wait for it to happen again.
<mwhudson> <mwhudson> stub: i see
<mwhudson> <mwhudson> stub: what/how can we log stuff?
<mwhudson> <mwhudson> stub: is it possible to find out what has locked the row in an except TimeoutError: block or something?
<stub> mwhudson: We should be able to get something that may be useful - a dump of the current locks if nothing else. We won't want it for all oopses, but I guess we could use a heuristic to decide when to log it.
<wgrant> What is the process for a DB patch these days?
<stub> Put it up for review, type 'db' to be reviewed by me and jml.
<stub> preferably a branch containing nothing but the database changes, before you spend time coding
<wgrant> Should I have the patch both in database/schema/pending/ and database/schema/ with 99 as the ID, as the README suggests?
<mwhudson> wgrant: just schema
<mwhudson> schema/pending is an appendix i think
<jamesh> these days pending/ is stuff that hasn't been merged yet
<wgrant> Ah, I see.
<wgrant> Thanks.
<thumper> https://code.edge.launchpad.net/~thumper/launchpad/auto-approve
<wgrant> I like the sound of that, I think (the branch is private)
<wgrant> Why is the title of the 403 page now 'Error: Launchpad system error'?
<wgrant> I think that changed from 'Error: Forbidden' around 3.0.
<thumper> wgrant: :)
<thumper> wgrant: I pasted that there just to frustrate you :)
<thumper> wgrant: it's public now
<stub> mwhudson: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/461661
<mup> Bug #461661: OOPS reports should log database locks <Launchpad Foundations:New> <https://launchpad.net/bugs/461661>
<mwhudson> stub: thanks
<wgrant> After three months, Ohloh just finished processing lp:launchpad/devel.
<mwhudson> wgrant: so you were bouncing on that page too eh?
<mwhudson> it's not clear that anything has yet happened that you can see
<thumper> wgrant: yes, but now I still don't see anything
<wgrant> It says the update has finished, but IIRC the report run is seperate from that.
<wgrant> Uhoh.
<wgrant> It just started updating again.
<thumper> https://www.ohloh.net/p/launchpad/analyses/latest
<thumper> there now
<wgrant> So it is.
<wgrant> Nice code volume graph.
<thumper> -4,430 lines of html
<wgrant> ... indeed.
<thumper> I love the 95% comment ratio for the 3 lines of perl
<jamesh> too bad ohloh doesn't show code:test ratio
<wgrant> That's odd.
<wgrant> There are several thousand lines of really foul Perl.
<mwhudson> stub: is #461661 something you're going to be able to work on at all soon?
<mup> Bug #461661: OOPS reports should log database locks <Launchpad Foundations:New> <https://launchpad.net/bugs/461661>
<stub> mwhudson: Depends on how gary prioritizes it
<mwhudson> stub: ok
<stub> We will need it for zero oops I imagine
<mwhudson> stub: indeed
<thumper> according to ohloh, I've changed 62,690 lines of Launchpad :)
<wgrant> I think that's all the aliases sorted out.
<wgrant> stub: Could you please review https://code.edge.launchpad.net/~wgrant/launchpad/bpr-ddeb_package-db/+merge/14008 at some point?
 * mwhudson eods
 * thumper too
<jamesh> spot the XSS bug on https://www.ohloh.net/p/launchpad/aliases
<wgrant> Indeed.
<wgrant> jamesh: We collided on the creation of two aliases. I wonder if it will die.
<jamesh> which one?
<wgrant> Sidnei and Cody.
<jamesh> it's a Ruby on Rails app.  They never have problems
<wgrant> Heh.
<wgrant> Launch Pad appears to be malcc, I think.
<wgrant> No idea about Launchpad Developers, though.
<jamesh> looks like commits from an external design company
<wgrant> Odd that there is no Cyrillic version of Danilo. I guess they must automatically merge on email address.
<wgrant> jamesh: Have you reported the XSS?
<jamesh> no
 * jamesh sends
<danilos> jtv, hi, it seems we've got more permissions issues on the bzr import, did you fix the last ones we've seen?
<jtv> danilos: think so, yes.  Is it in the oopses?
<jtv> (In call)
<mrevell> Morning!
<danilos> jtv, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1393RSBR1
<danilos> mrevell, morning
<jtv> danilos: thanks
<danilos> jtv, please also comment on https://bugs.edge.launchpad.net/rosetta/+bug/459836 if this wasn't in yet on 24th, and mark appropriately
<mup> Bug #459836: No templates are getting imported from my branch <Launchpad Translations:Triaged> <https://launchpad.net/bugs/459836>
<jtv> danilos: will do
<deryck> Morning, all.
<intellectronica> i need some help working with buildout. i've got a lazr.restful fix which i want to test with a local lp branch. how do i make my lp branch use the code in my restful branch, rather than the egg?
<intellectronica> allenap: maybe you know? ^^^
<allenap> intellectronica: I do know, but it's a bit ugly.
<allenap> intellectronica: As in, my solution is ugly.
<intellectronica> allenap: i didn't expect it to be beautiful
<allenap> intellectronica: In the lazr.restful branch I do python setup.py sdist.
<allenap> Then copy the tarball into my download-cache.
<allenap> Then make sure that all existing eggs have been removed, or that the version of the lazr.restful branch is distinct.
<intellectronica> wow, yes, that's pretty awful
<allenap> Then update setup.py in the lp branch to refer to the version of the lazr.restful distribution.
<allenap> intellectronica: That's not all. When you're done you have to remember to remove it all :)
<allenap> intellectronica: There must be a better way, but I don't know it.
<intellectronica> allenap: there must be. i thought buildout is supposed to make life easier for us, not harder
<allenap> Ah, the versions are in versions.cfg.
<intellectronica> BjornT_: maybe you know how to use a lazr.restful branch in development instead of an egg?
<allenap> intellectronica: There is the develop-eggs setting, which sounds like it should help, but I don't really understand what it does.
<intellectronica> oh
<allenap> intellectronica: Ah, the setting is "develop" in buildout.cfg. The docs say "This tells the buildout to install a development egg for our recipes. Any number of paths can be listed. The paths can be relative or absolute."
<allenap> intellectronica: "recipes" is a directory in the documentation, not a general reference to recipes.
<intellectronica> allenap: hmmmm ... i'm not sure i understand how i would use it
<intellectronica> allenap: what should i specify as the value of develop. is it the path to the external branch?
<allenap> intellectronica: It's already set as "develop = ." in buildout.cfg. I guess it might work to say "develop = . ${path-to-your-lazr.restful-branch}" (those settings might have to be on separate lines.
<intellectronica> allenap: yeah, i wonder how you separate paths in this case. maybe a semicolon?
<allenap> intellectronica: This is all guesswork though, I've not done this before.
<allenap> intellectronica: indented, on a new line.
<intellectronica> allenap: no, that doesn't seem to have any effect. maybe i need to somehow tell it to forget about the released eggs?
<allenap> intellectronica: Yeah, something like that. I'm going to play around with it now and see what I can discover.
<intellectronica> allenap: oh, it must have done something, because now in develop-eggs i have a new file, lazr.restful.egg-link
<intellectronica> which seems to point to the right place
<allenap> intellectronica: Cool, that sounds promising.
<allenap> intellectronica: I had to comment out the version specification in versions.cfg too.
<allenap> intellectronica: Then running bin/buildout used the "develop" version.
<allenap> intellectronica: If you confirm that works for you I'll add it to the wiki.
<intellectronica> allenap: well, i can confirm that _something_ happens. after doing that i get all sort of errors that seem unrelated. let me play with it a bit
<intellectronica> oh, leonardr, i bet you can help me!
<allenap> Good timing :)
<leonardr> intellectronica, what's going on?
<intellectronica> leonardr: i have a lazr.restful branch with a fix for a problem creating an etag for entries with non-unicode values. i want to test it with a development launchpad branch, but can't figure out how to do that with buildout
<intellectronica> leonardr: good morning, b.t.w :)
<leonardr> intellectronica:
<leonardr> 1. symlink your lazr.restful branch to the root of your launchpad branch
<leonardr> eg. [launchpad root]/lazr.restful
<leonardr> 2. add the name of the symlink to buildout.cfg's 'develop' section
<leonardr> for instance, i have a branch i use to test launchpadlib development
<leonardr> and its develop section looks like this
<leonardr> develop =
<leonardr>         .
<leonardr>         launchpadlib
<leonardr> 3. run bin/buildout
<intellectronica> leonardr: no, that doesn't seem to have worked. after doing that it still uses the released egg
<leonardr> intellectronica: another thing you can do is do a release of lazr.restful with setup.py sdist, copy that tarball into download-cache, and update versions.cfg to point to the new tarball
<mzz> that's what I did, and iirc it worked
<mzz> although at some point I had to update that release and that took some prodding
<leonardr> i don't know why the buildout.cfg doesn't work for you, you should ask gary about it
<intellectronica> leonardr: yes, i'll do that
<intellectronica> thanks a lot leonardr, allenap
<allenap> intellectronica: Cool.
 * henninge lunches
<jtv> allenap: I'm trying your ec2 workaround on jaunty (yes I do run karmic but on another machine, thank you) but for "ec2 land."  It's not finding lazr.uri though!  Any ideas?
<allenap> jtv: If it's available, install python-lazr-uri :-/
<jtv> no eggs?
<jtv> allenap: quod non  :-(
<allenap> jtv: You could muck around with eggs I suppose. Or just bzr branch lp:lazr.uri foo and add foo/src to PYTHONPATH.
<jtv> allenap: oh, that's not _too_ bad.  Awkward, but at least it doesn't leave forgotten crud lying around in places where it'll hurt me later.
<jtv> allenap: nope, not working either...  I did setup.py build, then used the build dir as part of my python path.  That leaves me with: type object 'Launchpad' has no attribute 'login_with'
<allenap> jtv: Okay, that's an issue with launchpadlib. You may have an out of date lazr.restfulclient or launchpadlib. Gah....
<allenap> jtv: The following:
<allenap> virtualenv --python python2.5 --no-site-packages gah
<allenap> gah/bin/easy_install pip
<allenap> gah/bin/pip install launchpadlib # Okay, easy_install would have done it, but I like pip.
 * jtv installs python-virtualenv
<allenap> gah/bin/python -c 'from devscripts.ec2test.entrypoint import main; main()' ...
<allenap> PYTHONPATH=lib gah/bin/python -c 'from devscripts.ec2test.entrypoint import main; main()' ...
<allenap> jtv: Gah! You might need to do gah/bin/pip install bzr too.
<jtv> allenap: wasn't done downloading yet anyway :)
<allenap> jtv: Looks like there are a few more steps: http://pastebin.ubuntu.com/302781/
<jtv> allenap: thanks!
<allenap> deryck: This might also help you: http://pastebin.ubuntu.com/302781/
<deryck> allenap, ok, thanks.  will try here in a sec
<sinzui> EdwinGrubbs: bac: stand-up in 2 minuted
<jtv> allenap: moving along with your recipe... now it wants bzrlib.plugin as well.
<allenap> jtv: Did you pip install bzr?
<jtv> allenap: no, I don't think so
<jtv> trying again...
<allenap> jtv: I had to also install bzr, paramiko and boto.
<jtv> allenap: now it just sits there doing nothing...
<jtv> allenap: oh, I did have bzr installed; it's in your pasted recipe
<jtv> so must have gotten the various paths and locations wrong
<allenap> jtv: Strange that it didn't find it.
<jtv> I probably did something stupid...  I set this up in a different place than inside the branch so I can reuse it.
<jtv> Interesting: it gets stuck in SSL read()
<jtv> And this time, a "no such file or directory"
<allenap> jtv: Argh! Can you paste the full error?
<jtv> Ah, but that's when I actually try to _use_ ec2.  I can get the help text out of it.
<jtv> allenap: http://paste.ubuntu.com/302812/
<allenap> jtv: Oh :( I know what that it. ec2 land actually calls utilities/ec2 test. That's not going to work. You'll need to just use ec2 land --dry-run then call ec2 test -s "..." by hand.
<allenap> jtv: Now, that won't work either. :-<
 * allenap despairs
<allenap> jtv: Sorry, you'll have to craft the commit message and just run ec2 test directly.
<allenap> jtv: Well, where "directly" means via the long, convoluted virtualenv approach.
<jtv> allenap: ec2 test won't work either, so I think I'll go with local tests & pqm-submit.
<allenap> jtv: Oh, is that the SSL hang you mentioned?>
<jtv> allenap: oh, with all the incantations.  Yes, I'll try that then
<jtv> That was probably just a hung connection that didn't happen the second time...
<jtv> unless it was the browser trying to ask me for authorizations
<jtv> allenap: ec2 test works this way!
<allenap> jtv: Hurrah :) I'm glad it didn't all turn out to be a waste of time :)
<jtv> allenap: since you seem to understand what's going on, worth a wiki page maybe?
<allenap> jtv: Yeah, coming up.
<sinzui> EdwinGrubbs: I have been looking into the bugtask__product__milestone__fk you saw on staging. There appears to be a bad order of events. The error implies that the milestone was not removed from the bugtask. The code shows we do.
<sinzui> EdwinGrubbs: I ponder two possible faults. 1) the view did not get all the bugtasks, or 2) the order of events played by Storm is different
<sinzui> danilos: ^ can you speculate as two how I go about ensuring the data is in the correct state before the milestone is deleted?
<sinzui> s/two/to/
<danilos> sinzui, reading up
<danilos> sinzui, you can probably use the same trick we are using for account merging (i.e. following all the references; I don't know exactly how it's done, but that's what I'd look up as an example); also, could this be related to nominations?
<sinzui> danilos: not exactly. https://dev.launchpad.net/RegistryTeam/RegistryTestPlans/3.1.10
<sinzui> danilos: The code does remove series targeted bugs. This issue relates to something that has been in production for 6 months.
<stub> It is possible to set foreign key references to ON DELETE NULL if we need to
<stub> That may be the right thing to do here (I don't know the details of the issue)
<sinzui> I'm going to delete a lot of things and see if the error is consistent.
<danilos> sinzui, I was going to suggest stub is a better person to ask, but then... :)
<sinzui> stub: I think the issue is the size of the delete. I can delete all the items in a small series, but a large one fails.
<stub> sinzui: what exactly are you deleting?
<sinzui> stub: https://staging.launchpad.net/launchpad-registry/2.2
<sinzui> stub: This is not a real world example. deletion is for mistakes
<sinzui> but non-the-less, I did not expect an oops
<stub> I was hoping for 'rows in the foo table' rather than a spec to read :)
<EdwinGrubbs> sinzui: I wonder if the problem is caused by the update not being flushed. Are you updating bugtask.milestone to NULL by setting an attribute on an object or an SQL statement.
<stub> So we are deleting a product series
<sinzui> stub: actually we are deleting the releases and milestones, everything else in the operation is an unlink/update
<sinzui> EdwinGrubbs: stub: the issue is milestone: I get the same error when I test just the milestone case: https://staging.launchpad.net/launchpad-foundations/+milestone/2.2.1/+delete
<sinzui> bugger!
<stub> If you have an OOPS, you should have a statement log which will tell you if edwin is right and a flush is needed (possible storm bug)
<intellectronica> gary_poster: hi, can i bother you for some help with buildout?
<sinzui> stub: EdwinGrubbs I think this is a code issue The milestone +index and milestone +delete list a different number of bugs. I think some bugs are being excluded from
<sinzui>     params = BugTaskSearchParams(milestone=target, user=None)
<gary_poster> intellectronica: hi, absolutely.  what's up
<EdwinGrubbs> sinzui: it probably uniques on the bug id.
<sinzui> ! stub: EdwinGrubbs private bugs are not included in the results
<intellectronica> gary_poster: i have a lazr.restful fix i want to test with a development lp branch, but i can't seem to make my lp branch use the lazr.restful branch instead of the egg
<stub> sinzui: We wouldn't have caught that if we used a cascading foreign key constraint
<sinzui> EdwinGrubbs: this has been broken for months then. I guess no one has ever tried to delete a milestone that had private bugs targeted to it (again this is not the really how we intended this feature to be used)
<gary_poster> intellectronica: ok, there are two ways to do it that should work.  one is to use develop egg, which is a way to use a branch; and the other is to make an sdist and test that way.  Let me see if I think the buildout.txt doc describes either or both of those in sufficient detail to be helpful in our conversation...
<sinzui> stub: EdwinGrubbs: thanks for you assistance, I can open a bug for this issue now.
<intellectronica> gary_poster: i think that using the branch itself is my preference. i tried doing that by adding the path to the develop setting of the buildout confing, but i must be getting something wrong. and yes, documentation will be great (if it's insufficient i'm happy to extend it once i get The Knowledge)
<gary_poster> intellectronica: :-) ok, cool.  yeah, there are no docs for that option.  thank you for the offer.  so, can you pastebin your launchpad diff and I'll look at it?  It sounds like you are on the right track.
<gary_poster> intellectronica: and did you run make, or (even better) bin/buildout after making the change to buildout.cfg?
<intellectronica> gary_poster: i did run bin/buildout
<gary_poster> cool
<intellectronica> gary_poster: http://pastebin.ubuntu.com/302859/ (note that lazr.restful is a symlink to the branch i want to use)
<gary_poster> intellectronica: I am of two minds as to whether I hope this fixes the problem ;-) but try putting those on two different lines?  e.g. http://pastebin.ubuntu.com/302862/ .  It's lame if that fixes it (an error message would have been nice, for instance), but that's what I would have expected to see.
<intellectronica> gary_poster: i've tried that too, and it didn't make any difference
<gary_poster> intellectronica: ok, fair enough.  so, I take it there are no errors when you run bin/buildout, or else you would have told me about them.  The only other thing I would expect to have to do is specify the new version of lazr.restful in versions.cfg.  I hoped that would raise an error too, but give that a try.  Am I right in guessing that your lazr.restful branch has a different version than the one specified in versions.cfg?
<intellectronica> gary_poster: no, that may be what i'm missing. how should i specify the new version of lazr.restful?
<gary_poster> intellectronica: in your lazr.restful branch, there should be a version.txt (or similar) in src/lazr/restful.  Make sure that is incremented.  in versions.cfg of launchpad, make sure that the version specified for lazr.restful matches that version.  (fwiw, this has been more obvious in the past; maybe I introduced a regression somewhere. :-/ )
<intellectronica> gary_poster: so just increment the value in version.txt to the next minor version, and adjust versions.cfg in my lp branch to match that?
<gary_poster> intellectronica: yes
<gary_poster> intellectronica: (fwiw, in lazr.restful, what you are *really* doing is increasing the revision in its setup.py.  We do it this way for this package)
<intellectronica> wow, it gets even more complicated. the version in versions.cfg is two minor revisions behind the current restful. does it mean that i have to branch from an earlier revision?
<gary_poster> intellectronica: is this for a CP?
<intellectronica> gary_poster: no, why?
<gary_poster> intellectronica: you would only need to branch from an earlier revision if you were making a CP (so that you are only going to change the minimal amount necessary for the change in production).  In this case, you are fine.  Update version.txt, change versions.cfg to match, and move on.
<gary_poster> intellectronica: (more complicated?  bah. :-) This sucks because I didn't document it, and because the errors are not helpful/non existent.  But So far we have "symlink the branch" "add the line to buildout.cfg" "make sure your branch has an updated version (as you would normally)" and "set the new version number in versions.cfg".  So, bah. :-) )
<gary_poster> (But agreed the docs and error sitation suck. :-( )
<intellectronica> gary_poster: tadaaa! :)
<gary_poster> ITR :-)
<gary_poster> yay!
<intellectronica> thanks muchly. any suggestions on where would be a good place to document it? wiki? text file?
<gary_poster> well, doc/buildout.txt would be the historical place.  There have been arguments about this between various People Who Will Not Be Nicked :-) .  I'd put it in doc/buildout.txt myself, unless you want to put it in the wiki, in which case you can JFDI or raise it in an email and see the fun :-)
<intellectronica> nah, i also prefer buildout.txt
<gary_poster> cool
<sinzui> EdwinGrubbs: Can you retest the delete series + structural subscription with https://staging.launchpad.net/launchpad-documentation/2.2 . I am subscribed to the series and the 2.2.1 milestone
<EdwinGrubbs> sinzui: ok, I'm deleting it now.
<EdwinGrubbs> sinzui: it deleted fine.
<sinzui> EdwinGrubbs: fab
<sinzui> I am still reporting the bug you discovered. I realise that this fix is not trivial. It is possible for a driver to target a private bug that the milestone owner or release manager cannot see
<henninge> Where can I find or activate a query log on launchpad.dev?
<henninge> So that I can see what sql queries where issued.
<henninge> danilos: ^ any idea?
<henninge> danilos: btw, I think I found the problem for the templates listing but I need to verify it.
<rockstar> abentley, replied to your review
<deryck> BjornT_, ping
<BjornT_> deryck: pong?
<sinzui> flacoste: I am ready for our call. I am idling my time refactoring a test
<intellectronica> so, how do i merge a lazr.restful branch now? just do it manually?
<intellectronica> also, once i do that, how do i make lp use the new revision?
<intellectronica> gary_poster: maybe you can help?
<intellectronica> sorry, i'm bugging you a lot today
<gary_poster> intellectronica: :-) np
<gary_poster> intellectronica: so, once you have a review, you merge it manually.  We want to do it in such a way that we get the usual [r=foo] commit messages for the trunk.  I have heard that there is a more elegant way to do this, and maybe you know it, but I just check out ``co`` the trunk, merge the reviewed branch, and commit with the desired commit message ([r=foo] blah blee bloo).
<gary_poster> For making LP use the new revision, we have docs (looking...)
<gary_poster> well, I don't talk about making a release, just incorporating it
<gary_poster> intellectronica: for making a public release, there is a wiki page.  See bottom ("Releases") section of https://dev.launchpad.net/HackingLazrLibraries.  I haven't tried it.  Ask leonardr for help (I am having lunch in < 5 min).  You will need privs.
<intellectronica> gary_poster: excellent, thanks
<gary_poster> intellectronica: just for our needs, you can do step 1 (python setup.py sdist) and then move the sdist to the download-cache/dist directory.  Then see the "Upgrade a Package" section of buldout.txt.  Honestly, you don't have to do the fancy ./bin/buildout command from step 4 in those instructions--just bin/buildout should be fine since you have already populated the download-cache
<gary_poster> intellectronica: also, note that the [TODO] section in 5 ("Test") has been done: see the -c oprion in ec2 test.
<gary_poster> intellectronica: OK, I have to run to lunch.  Will be back in a bit
<intellectronica> gary_poster: i'll be away by then, but will continue later/tomorrow. enjoy lunch
<mrevell> G'night all. I'm off to cook some spag and some bol.
<rockstar> abentley, hi
<abentley> rockstar: hi
<rockstar> abentley, did you see my reply to your review?
<abentley> I did.  I'm looking over it now.
<rockstar> abentley, great.  Can I send you a followup.
<rockstar> Er, a followup branch.
<abentley> rockstar: okay.
<abentley> Oh, a whole 'nother branch?
<rockstar> abentley, yeah, but it's really small.
<abentley> rockstar: alright.
<abentley> rockstar: I'm almost inclined to think it's a bug that create_branch_and_tree doesn't set target.branch_format and target.repository_format.
<abentley> rockstar: replied.
<thumper> morning
 * thumper has real coffee from fluid
<thumper> yum
<sinzui> thumper: abentley: I think code/browser/test_views could test everything on the DatabaseFunctionalLayer
<thumper> sinzui: prolly
<sinzui> thumper: I pushed two branches to lp:dev and create an MP. What do I run to scan the branches and create the diff?
<thumper> sinzui: make sync_branches
<thumper> sinzui: although that doesn't create the diff
<thumper> sinzui: abentley was talking about making more make targets to do these
<sinzui> well at least that saw my branches
<thumper> sinzui: right now I think you need to run the cron script manually
<sinzui> thumper: right I was doing that and am fine
<thumper> flacoste: are we talking today?
<flacoste> thumper: we should, i'm still on the phone with gary
<thumper> ok
<thumper> flacoste: afk for a few minutes
<thumper> rockstar: ping
<rockstar> thumper, pong
<flacoste> thumper: i hung up
<thumper> flacoste: ok, talk now?
<flacoste> thumper: yep
<EdwinGrubbs> sinzui: ping
<sinzui> Hi EdwinGrubbs
<EdwinGrubbs> sinzui: I'm wondering if I should change "See full publishing history" to just "Full Publishing History" to avoid wrapping. https://chinstrap.canonical.com/~egrubbs/publishing_history_link.png
<EdwinGrubbs> sinzui: Also, since I'm on that page already, I could add a javascript:confirm() to the delete button, but I don't know if that fits in with our general ajax style.
<sinzui> EdwinGrubbs: why add a confirm. Users have not reported a bug about that feature in the two years it has been available?
<EdwinGrubbs> sinzui: well, every delete button in a rails app asks you to confirm it, and it seems like a sane convention.
<sinzui> EdwinGrubbs: I see what you are seeing in the menu. We see it because we are modern men who understand that a windowed GUI and multi-tasking is practical. The design we have is for maximised windows though
<wgrant> sinzui: Previously it was a button.
<wgrant> Now it is not.
<wgrant> However, there are other 'green' delete links that have no text, so are not obviously green (the bug team unsubscription links, for example) , so it is not unprecedented.
<sinzui> EdwinGrubbs: we have many delete ops in Launchpad. I would expect a page that explains the consequences of my action.
<sinzui> EdwinGrubbs: I agree that confirmation is good, I do not agree with a js infobox. Linking is a part of our focus so we can fix this next release
<EdwinGrubbs> ok, I won't touch it.
<sinzui> EdwinGrubbs: I think the real problem with the action menu is not that we are expect to browser with window's maximised, but because we did not consider wrapping rules
<sinzui> EdwinGrubbs: We might want a hanging indent style so that all the link text does not appear under the icon
<sinzui> something like this may make the action menu pleasant: text-indent: -20px; padding-left -20px;
<sinzui> wgrant: if you delete a packaing link, is there information you expect to see on a confirmation screen that would make you change your mind? The history of versions in the series perhaps?
<wgrant> sinzui: I don't think there's anything useful besides the series and SP.
<sinzui> wgrant: thanks. I too could not think of anything more than what you can see on the page already
<al-maisan> hmm .. I am getting the following error when running "make":
<al-maisan> /usr/bin/python2.4: can't open file 'sourcecode/lazr-js/tools/build.py': [Errno 2] No such file or directory
<al-maisan> any idea how this can be mended?
<mwhudson> al-maisan: is your sourcecode directory up to date?
<al-maisan> BTW, this is a brand new db-devel branch and I did a rocketfuel-get beforehand
<wgrant> Did you link-external-sourcecode?
<mwhudson> maybe your update-sourcecode is too up to date :/
<al-maisan> aha..?
<al-maisan> wgrant: yes
<wgrant> :(
<al-maisan> I am missing a sourcecode/lazr-js link though..
<wgrant> That would do it...
<al-maisan> .. and I don't have it in ../../lp-sourcedeps/ either
<al-maisan> I guess I could pull it manually
<mwhudson> is it in utilities/sourcedeps.conf ?
 * al-maisan looks
<al-maisan> mwhudson: yes
<al-maisan> I fetched lazr-js manually
<mwhudson> al-maisan: i guess running ./utilities/update-sourcecode would work
<al-maisan> mwhudson: thanks for the tip!
<wgrant> But doesn't rocketfuel-get do that?
<mwhudson> yeah, i wonder how rocketfuel-get does this currently
<mwhudson> hmm odd
<wgrant> al-maisan: Is there an existing spec somewhere for Debian source format 3.0 support?
<al-maisan> wgrant: I am not aware of any ..
<al-maisan> wgrant: I'll look and let you know more tomorrow .. it's getting late here :P
<wgrant> 'cause as james_w says, it is getting pretty critical.
<wgrant> al-maisan: Yes, just a bit...
<al-maisan> wgrant: yup .. saw his email
<mwhudson> time for lunch, qa for my afternoon snack...
<thumper> :)
#launchpad-dev 2009-10-28
<BjornT_> mwhudson, al-maisan: sourcecode/lazr-js has been removed from devel. rocketfuel-get in devel will remove the lazr-js from sourcecode/, so if you use the to link the db-devel lazr-js branch to the devel one, it won't work.
<thumper> BjornT_: shouldn't you be sleeping?
<BjornT_> thumper: probably. but doing sports late in the evening usually causes me to fall asleep late.
<thumper> BjornT_: what sport?
<wgrant> BjornT_: You have an exaggerated definition of 'late'.
<BjornT_> thumper: floorball
<thumper> BjornT_: what is floorball?
<thumper> BjornT_: indoor football?
<BjornT_> thumper: more like indor http://en.wikipedia.org/wiki/Floorball
<BjornT_> more like indoor hockey, but without ice and skates
<thumper> BjornT_: so roller hockey for those that can't skate?
<BjornT_> thumper: yeah, something like that :)
<thumper> spm: ping
<al-maisan> BjornT_: thanks for the explanation re. sourcecode/lazr-js removal.
<thumper> mwhudson: do you know where the fix needs to go for https://bugs.edge.launchpad.net/launchpad-code/+bug/372506  ?
<mup> Bug #372506: Don't log OOPSes when people try to delete a directory <codehosting-ssh> <oops> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/372506>
<mwhudson> thumper: lp.codehosting.sftp probably
<thumper> mwhudson: want to take this one?
<mwhudson> catch the error in remove or unlink or whatever the method is and error nicely
<mwhudson> thumper: yeah, sure
<thumper> mwhudson: ta
 * mwhudson_ has probably stopped rebooting his modem for now
 * thumper whacks up a branch for review
<thumper> mwhudson: I don't expect you to do it at 5:30pm
<thumper> mwhudson: even though it is smallish
<mwhudson_> thumper: i don't see it
<thumper> mwhudson: I'm still typing the cover letter :)
<mwhudson> ah ok
<thumper> email sent, should flow through the system in 2 minutes
<thumper> mwhudson: it is there now
 * thumper EODs
<mwhudson> thumper: eurgh
<stub> has anyone elses gpg-agent crapped out today or yesterday?
<lifeless> nope
 * stub wonders how gpg-agent got uninstalled
<henninge> Hi everybody!
<henninge> Is there a way to see a log of sql queries issued by a request on launchpad.dev?
<henninge> Like the statement log I see in an OOPS report?
<henninge> stub: ^ would you know the answer?
<henninge> wgrant: ^ ?
<henninge> BjornT_: ^ ?
<spm_> henninge: there's a switch in postgres somewhere....
<henninge> Hi spm_, I was about to ping you next ... ;)
<lifeless> henninge: easiest way is to turn on verbose logging
<lifeless> in pg
<henninge> lifeless: how?
<spm_> henninge: http://archives.free.net.ph/message/20030820.033440.c8363e0d.en.html ?? maybe.
<wgrant> henninge: It's probably turned on for you.
<wgrant> henninge: It was only turned off in lp-database-setup last week.
<wgrant> henninge: It's not in the postgres log?
<wgrant> (log_statement='all' is the relevant postgres switch)
<lifeless> mm
<lifeless> https://code.edge.launchpad.net/~lifeless/drizzle/build
<lifeless> was pushed to 5 weeks back
<lifeless> mtaylor pulled from it
<lifeless> where did my content go??!
<wgrant> He pulled it alllll awaaaay.
<BjornT_> henninge: see lib/canonical/launchpad/pagetests/basics/user-requested-oops.txt
<henninge> BjornT_, lifeless, spm_: Thanks. Yes, it is logging for me. I just didn't think about the fact that postgres is not in the LP tree ...
<henninge> wgrant: ^
<lifeless> spm_: mwhudson: any ideas?
<spm_> henninge: LIES!!! EVERYTHING!!! is in the LP tree.
<lifeless> mtaylor: I see a repo there...
<wgrant> Win. First 3.0 source package in LP.
<spm_> lifeless: "This branch has not been pushed to yet." ?
<lifeless> spm_: mtaylor branched from it 5 weeks back
<lifeless> spm_: and had a copy on his disk, so it most definitely *was* pushed to - and mirrored
<spm_> impressive. that sounds like something's zotted it.
<lifeless> thats why I'm concerned
 * spm_ has a look at crowberry's disk.
<lifeless> mtaylor: I'm going to push up to build2, so I don't trash the evidence
<mtaylor> lifeless: k. works for me
<lifeless> mtaylor: a thought, perhaps I sent you a bundle instead?
<lifeless> drizzle is pushing up to build2 now, but will be a while - it can't stack.
<mtaylor> lifeless: hrm. perhaps that's what you did
<spm_> spm@crowberry:~$ bzr log --short -r -10..-1 /srv/bazaar.launchpad.net/push-branches/00/03/be/17
<spm_> bzr: ERROR: Not a branch: "/home/robertc/source/drizzle/autorundeps/".
<spm_> lifeless: ^^
<lifeless> spm_: ><
<lifeless> spm_: bzr info on that?
<mtaylor> lifeless: http://paste.ubuntu.com/279365/plain/
<lifeless> actually
<mtaylor> lifeless: that's what you gave me
<lifeless> ls .bzr
<lifeless> mtaylor: yeah I gave you a merge directive.
<mtaylor> yup
<lifeless> so false alarm methinks on the lp losing data
<mtaylor> k. well, I've got it now then
<spm_> lifeless: http://pastebin.ubuntu.com/303414/
 * mtaylor wags finger at lifeless for pastebinning a merge directive instead of pushing a branch ... but gets over it just as quickly
<lifeless> mtaylor: hey, drizzle isn't tiny ya know
<lifeless> spm_: cat .bzr/branch/format
<mtaylor> lifeless: wouldn't have been a problem if we had gotten around to --2a yet.
<mtaylor> sigh
<lifeless> mtaylor: hint. hint.
<lifeless> mtaylor: J F D I
<spm_> lifeless: Bazaar-NG Branch Reference Format 1
<mtaylor> lifeless: I sent the initial "o hai all ya'll upgrade" email
<lifeless> spm_: I pushed a checkout. \o/
<lifeless> mtaylor: you're mising the word 'now' ;)
<spm_> lifeless: I'm gathering that's officially a "Bad Thing(â¢)"
<lifeless> spm_: its a bug that I keep meaning to track down
<lifeless> its a bad thing, yes.
<lifeless> its like a symlik
<spm_> heh. sounds like you have extra incentive now then huh? :-P
<lifeless> spm_: nope, not really - 5 weeks ago occurnce != fresh
 * spm_ waves away unimportant minor facts
<adeuring> good morning
<stub> henninge: There is an environment variable you can set I think
<henninge> stub: thanks, I found the log.
<stub> henninge: LP_DEBUG_SQL and LP_DEBUG_SQL_EXTRA
<henninge> stub: that outputs to stdout?
<stub> henninge: Or stderr - its in lib/canonical/launchpad/webapp/adapter.py
<henninge> stderr is fine
<mrevell> Guten morgen!
<henninge> Guten Morgen mrevell!
<mrevell> hallo henninge :)
<jml> mrevell, henninge: Good morning!
<mrevell> yo jml
<henninge> Hallo jml!
<henninge> ;)
<maxb> I was trying to run tests on the python-migration branch last night, and they seemed broken :-/
<henninge> and a smile for Mr Re-VELL :)
<maxb> Does anyone know about the huge amounts of "warning: Not importing ... missing __init__" ?
<jml> maxb, I'm ensaddened.
<jml> maxb, yes, I do a little.
<jml> maxb, Python generates this error internally. It's a false warning.
<jml> maxb, it's caused by lazr.foo being a bit weird.
<maxb> Hmm. I thought I saw it trigger at least one "unclean stderr" -> fail
<jml> maxb, yes, that's a real problem.
<maxb> Life would be so much simpler if zope/lazr hadn't gone for namespace packages, when Python doesn't really support them :-/
<henninge> jtv!
<jtv> hi henninge!
<henninge> Hallo jtv! ;)
<jtv> henninge: working on message-sharing memory footprint... forgot I'm also supposed to OCR.
<henninge> jtv: working on the +templates page time out - almost done.
<jtv> henninge: cool... what are you doing to fix it?
<henninge> jtv: getting SourcePacakgeName in the same query as POTemplate for POTemplateSubSet
<jml> maxb, yes :)
<jtv> henninge: so a prejoin?
<henninge> jtv: actually, I have always meant to ask what exactly that is.
<henninge> a prejoin
<BjornT_> mthaddon: could you please disable edge updates? it seems like current trunk is a bit broken, so i want to make sure edge doesn't get updated until it's fixed.
<henninge> jtv: It's a join in "using", is that it?
<mthaddon> BjornT_: already disabled
<jtv> henninge: you add a join to your query, even though you're not actually using it right there.  But you're pre-seeding the cache with those objects so you don't end up fetching them one by one later.
<henninge> jtv: exactly
<jtv> henninge: in sqlobject we had a parameter with that exact name, though in Storm what you usually do is fetch tuples of objects.
<henninge> jtv: what made is difficult was that the POTemplateSubSet code did not distinguish bewtween the productseries and the distroseries case.
<maxb> I'm not in a position to make a branch and MP for it now, but it appears that the build is currently broken (for me, anyway), because lazr-js hasn't been removed from sourcecode/Makefile
<henninge> and this only applies to the latter, of course.
<jtv> henninge: that's no problem for a left join though!
<henninge> jtv: seriously?
<henninge> jtv: I could just left join SourcePackageName all the time and it doesn't cost me anything?
<jtv> henninge: seriously... if you add an outer join on POTemplate.sourcepackagename, then you'll just get a bunch of nulls instead of SourcePackageName rows in the case where it's null.
<henninge> jtv: no performance penalty?
<henninge> after all, I am trying to solve a performance problem
<jtv> henninge: I can't guarantee that it'll never cost you anything, but I'd say that it's _probably_ not an issue.
<BjornT_> mthaddon: could you add a comment saying that it shouldn't be enabled before bug 462502 is fixed?
<mup> Bug #462502: Javascript broken when devmode is turned off <Launchpad Foundations:In Progress by bjornt> <https://launchpad.net/bugs/462502>
<mthaddon> sure
<henninge> jtv: hm, that could make the code a lot simpler, of course.
<jtv> henninge: 'xacly
<jtv> henninge: I'd suggest that you ask a losa to run some explains to check this.
<henninge> jtv: woa, I remember stuart showing that at the epic. But that was about a year ago ...
<jtv> henninge: get a sample query that times out; do 2 "explain analyze" runs on it and record the second time; then compare to another "explain analyze" run that has the left join added.
<henninge> quite exactly, actually.
<henninge> jtv: ah, that is not the problem case
<jtv> what isn't?
<henninge> jtv: I'd be concerned about productseries +template pages
<henninge> but they are not timing out
<henninge> concerned, because they don't need the sourcepackagename
<henninge> jtv: but maybe that's the answer
<jtv> the sourcepackagename lookup is going to be quite fast in cases where the fk is always null, I suspect :)
<henninge> jtv: since product series pages don't get that long, there is no performance issue anyway.
<jtv> henninge: stub just pointed out the "precache" wrapper for result sets.  Lets you do something close to the classic prejoin, but without having to retrieve tuples that you only need one object out of.
<henninge> jtv: sounds interesting, I have some code going into exactly that.
<henninge> jtv: do you have an example?
<jtv> henninge: looking it up now...
<jtv> henninge: lib/lp/services/database/precache.py
<jtv> example is in the docstring.
<henninge> jtv: cheers
<jtv> np
<thumper> mwhudson: night all
<deryck> Morning, all.
<jtv> hi deryck
<henninge> stub, jtv: Are you sure that precache works as described in the doc string???
<jtv> henninge: see the XXX at the top of the file :)
<henninge> jtv: yup, timeout gone. https://translations.staging.launchpad.net/ubuntu/karmic/+templates
<jtv> henninge: \o/
<jtv> did you get the precache working for this?
<henninge> jtv: but that is not using precache yet. Actually, it is failing.
<jtv> :-(
<henninge> failing tests
<henninge> jtv: I got it to work for this page but I am currently running lp.translations tests on it and I already saw a failure.
<jtv> :/
<henninge> jtv: btw, ordering of the +templates page is next because when it gets that long, it is really useless the way it is now.
<henninge> jtv: POTemplateSubSet had an ordering (id,path) ...
<jtv> oh, how useful (!)
<henninge> jtv: I already got rid of the path part, os now it's only id.
<henninge> jtv: I think sourcepackagename.name, potemplate.name sounds sensible
<jtv> henninge: quite
<jtv> hmm... still seeing timeout on that staging page. :(
<henninge> jtv: I don't
<jtv> worked for me this time too
<jml> allenap, hi
<allenap> jml: Hi.
<jml> allenap, how's it going with the build eng stuff? love, joy and peace or fire, plague and the sword?
<allenap> jml: It's okay, but it's mostly task switching and wtfing :)
<jml> allenap, task switching :\
<henninge> jtv: I fixed precache, btw.
<jml> allenap, nothing would thrill me more than to have a chance to shoot the breeze about this stuff, so please feel free to call me if you want a teddy bear / rubber duck / cardboard cutout...
<henninge> jtv: it's in this patch: http://paste.ubuntu.com/303512/
<allenap> jml: That would be cool. I'm a bit afk and back today because t'other half is ill and there's no one else to look after the kids, but I should have time and my thoughts in order this afternoon.
<jtv> henninge: stub just asked if you were going to add a test too, and you did.  Good show!  :)
<jml> allenap, cool. that'd be great.
<henninge> jtv: no, the test was there already. I just extended it for the case that came up in our test suite.
<jtv> henninge: yes, just heard the same here.  :-)
<henninge> stub, jtv: I think I can remove that part of the XXX, then.
<henninge> jtv: btw, are you with stub?
<stub> +1
<jtv> henninge: no, we're just friends
<jtv> henninge: yes, we're working on the message-sharing migration footprint issue.
<jtv> stub: how's that doing btw?  :-)
<henninge> stub: Is the renaming of precache still an issue? That is the only thing left from the XXX (and the bug) that would need doing. If we won't rename it, I'll remove the XXX completely.
<jtv> I vote keep the name-changing part.  :-)
<stub> I expect we need a better name since I couldn't remember what it was called earlier. I can't think of a better name though.
<stub> Feel free to rename it if you want - there are only two or three callsites at the moment.
<henninge> so "prejoin" is not good?
<stub> I suspect that was what I intended to call it, but then a brain fade made me type 'precache'
<henninge> stub: I hope you are feeling better now ... ;)
<stub> $ make harness
<stub> >>> import gc
<stub> >>> len(gc.get_objects())
<stub> 461210
<stub> And a memory footprint of 480MB
<stub> Just for loading all our libraries and processing that zcml. Sheesh.
<jml> :(
<jml> stub, to quote _Anchorman_, that's not a small number.
<jml> oh wow
<jml> python 2.5 has datetime.strptime
<jml> O Happy Day!
<jml> \
<jtv> henninge: that's for cases such as KDE where translations get uploaded as part of one package, but need to be imported in another.
<jtv> I always forget the details of how it works.
<maxb> barry: Do you know what the last known state of the python-migration branch is? Because it doesn't work for me. (1) apport_python_hook fails the clean_modules test, (2) warning, not importing lazr, missing __init__.py breaks things looking for clean stderr
<sinzui> EdwinGrubbs: are you ready for the stand-up call?
<sinzui> maxb: barry is not online today
<EdwinGrubbs> sinzui: yes
<maxb> :-/
<maxb> ok
<maxb> gary_poster:  Do you know what the last known state of the python-migration branch is? Because it doesn't work for me. (1) apport_python_hook fails the clean_modules test, (2) warning, not importing lazr, missing __init__.py breaks things looking for clean stderr
<BjornT_> rockstar, abentley: can you confirm bug 462628 and bug 462632? if they are valid, they need to be dealt with now.
<mup> Bug #462628: Commenting on merge proposals is broken <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/462628>
<mup> Bug #462632: Edit commit message on a merge proposal is broken <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/462632>
<abentley> BjornT_: chasing
<abentley> BjornT_: I can confirm that inline commenting on merge proposals is broken.
<sinzui> reviewers meeting is 2 minutes in #launchpad-meeting
<deryck> abentley, BjornT_ -- that commenting is likely broken by my recent branch landing.  I need to get a lazr-js branch landed (if I can now) and both MP comments and commits need updating so the xhtml representation is utf-8 encoded.
<BjornT_> abentley: ok. i'd suggest that you talk to the losas and make sure edge doesn't get updated until this is resolved (and the fix is in stable)
<deryck> I commented on the encoding bit on thumper's MP for commit message branch.
<abentley> deryck: That looks like the cause of the commit message issue, but not the comment issue.
 * deryck looks closer at bug
<deryck> abentley, yeah, I can't tell much from the Windmill traceback, sorry.
<BjornT_> deryck: you can land your lazr-js branch. no you can even land the lazr-js branch first (well, in the future), so that lp won't break.
<abentley> losa: can we please disable edge updates-- we have a significant regression we don't want to show users.
<mthaddon> abentley: it's already been disabled
<abentley> mthaddon: Cool.
<deryck> BjornT_, right, I should have landed the lazr-js first, but thought with your work and edge updates being disabled I could beat the rush and get it landed.
<deryck> BjornT_, didn't think about having to read up on the new way of doing lazr-js branches ;)
<gary_poster> maxb: did you see https://dev.launchpad.net/PythonMigrationStatus and Barry's email?  per those documents, https://code.launchpad.net/~launchpad-dev/launchpad/ztk-2.5 is the primary one we care about now.  Moreover, there will be bit rot, because we can't land it until after the upcoming LP release (again, described in Barry's email).  Dealing with the bit rot and landing that will be my job then (i.e., in about 1.5 weeks)
<maxb> right, so I should essentially consider python-migration to be obsolete and "spliced out" of the merge flow?
<maxb> Unless there's non-zope stuff that's already landed into ztk-2.5 instead of python-migration already, my problems will still apply there
<maxb> Oh, so there are.
<maxb> Right
 * maxb marks python-migration as Abandoned, and updates wiki and retargets MP
<gary_poster> maxb: thank you
<maxb> Where has the release schedule gone these days?
<maxb> Releases/2009Calendar
<maxb> abentley: asdf to you too :-p
<abentley> Sorry about that.
<maxb> :-)
<maxb> grr. The Launchpad Moinmoin CSS has broken strikethrough support.
<maxb> gary_poster: Do you suppose you could pre-emptively add the ztk-2.5 new things to lp-source-dependencies at some point?
<gary_poster> maxb: my default inclination is to keep them out until I prepare the merge.  I'm probably pretty easy to convince otherwise though.  Why do you want to have them added early?  (One example of a winning argument that need no further explanation is "because I'm going to be keeping the bit-rot away and this would be convenient.")
<gary_poster> needs
<maxb> I figured I'd try my hand at resolving some of the "Bustimications" - that _might_ evolve into keeping pace with devel :-)
<gary_poster> ok fair enough.
<gary_poster> I'll do that today.
<al-maisan> Mould somebody please help me with create_initialized_view() usage? I am adding unit tests (http://pastebin.ubuntu.com/303650/, line 62-63) and the setUp() method fails with: http://pastebin.ubuntu.com/303651/
<al-maisan> *Could
<al-maisan> *sigh*
<gary_poster> maxb: rev 87 of lp-source-dependencies has new ztk deps
<allenap> jml: Are you around for a bit? If so, do you fancy having a chat?
<jml> allenap, I am. I'm on a call now though. I'll ping you when I'm off.
<allenap> jml: Cool.
<jml> allenap, ping!
<allenap> jml: pong
<allenap> jml: Or a mobile, if someone has sold you one yet. I have near infinite free credit for mobile-to-mobile calls.
<mrevell> I'm gonna head off before Xchat craps out again. Night all
<thumper> deryck: can I have a very quick call with you after the TL call?
<deryck> thumper, I have a quick chat with BjornT_ and then we can.
<thumper> deryck: or I could talk to you in 1.5h
<thumper> as it is breakfast time here :)
<deryck> thumper, I have to be EOD in 1.5 hour with family responsibilities today. anytime before 1.5 hour from now.
<thumper> :(
<thumper> deryck: how long is the BjornT_ likely to take?
<deryck> thumper, 5 mins, tops.  He has to go, too.
<thumper> ok
<thumper> I'll wait
<mwhudson> good morning
<BjornT_> deryck: so, what's up?
<thumper> mwhudson: early start?
<deryck> BjornT_, so I submitted a branch to lazr-js trunk via pqm.  Saw it playing there.  Then lunched, then it's finished in pqm....
<deryck> BjornT_, but no sign of it in lazr-js trunk, no email saying success or failure.
<deryck> BjornT_, where can I look to work out what happened?
<mwhudson> thumper: yeah, emma is off to a meeting today, left at 6:30
<thumper> :)
<thumper> deryck: I'd say chase a losa for the pqm logs
<BjornT_> deryck: yeah, what thumper said. you might be able to look at the logs yourself. they might be on devpad
<thumper> deryck: wild stab in the dark, but there was speed and connectivity issues at some stage yesterday
<thumper> deryck: perhaps the push failed?
<BjornT_> yeah, we have seen buildbot fail pulling a few times lately
<deryck> thumper, BjornT_, ok, I can try the submit again, and then chase a lose if no luck.  Or look for logs on devpad.  Thanks, guys.
<thumper> deryck: I'd look at the logs first
<deryck> thumper, ok
<deryck> thumper, shall we do a call now?
<thumper> deryck: it may be that your branch has landed on the pqm copy already
<thumper> deryck: yep
<deryck> BjornT_, thanks for the chat time!
<deryck> thumper, on skype?
<thumper> deryck: I'm trying to get skype to give me sound
<BjornT_> deryck: "different rich-root support"
<BjornT_> deryck: in patch.1256740771.log
<deryck> ah, suck.
<deryck> I know what I did.
<deryck> got a repo doesn't support 2a format, and couldn't tell, so I went one format back in the help text and forgot to ask :)
<thumper> we should upgrade lazr-js to 2a
<deryck> indeed we should
<mwhudson> i love the way all the launchpad developers answer "Whatâs more important? Principle or pragmatism?" by bashing the question
<intellectronica> yeah, i'm still waiting for someone to just make a simple choice, a one word answer
<intellectronica> of course, by not answering the questionaire i gave up the opportunity to do that myself :-/
<rockstar> mwhudson, :)
<flacoste> intellectronica: yeah, i should have answered 'Five pounds of flax'
<intellectronica> :)
<flacoste> kfogel: do you have a LWN subscribtion? did you look at http://lwn.net/Articles/359013?
<kfogel> flacoste: I don't have a subscription.  let me see
<kfogel> flacoste: oh, it's paywalled
<mwhudson> flacoste: i pasted a free link into my warthogs mail
<mwhudson> kfogel: ^^
<kfogel> mwhudson: thx
<jml> ISTR I answered "principle"
<jml> and then talked and talked and talked
<intellectronica> jml: that's encouraging
<jml> intellectronica, "Five pounds of flax" would have been a better answer.
<intellectronica> anyway, is it not five tons?
<mwhudson> jml: hello
<mwhudson> jml: i wanted to talk to you about something, i wonder what it was
<intellectronica> or are you just being pragmatic?
<mwhudson> jml: oh yes, maybe it was https://bugs.edge.launchpad.net/launchpad-code/+bug/450998
<mup> Bug #450998: branch puller can try to pull the same branch twice at once <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/450998>
<jml> mwhudson, I'd love to. I'm going to talk to lifeless first though.
<mwhudson> jml: ok
<mwhudson> it's not really week 2 still is it?
* mwhudson changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 3 of 3.1.10 | I am Zero OOPS and So Can You! http://is.gd/4fkLl | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<flacoste> intellectronica: : it is 5 tons!
<intellectronica> actually, according to google it's 3 pounds in the original zen story, and 5 tons in the principa discordia, so i guess that's a hybrid answer
<flacoste> right, i mixed it up :-)
<kfogel> mwhudson, flacoste: responded
<kfogel> mwhudson, flacoste: I mean "responded on warthogs", not publicly (and see warthogs response for why)
<flacoste> kfogel: note that this will not paywalled in one week
<kfogel> flacoste: yes, but by then it'll be old news :-)
<kfogel> flacoste: not completely joking -- one effect of paywalls is to fracture the audience and reduce feedback effects.
<flacoste> i never thought of it that way, but that's an interesting point
<intellectronica> are we in testfix mode?
<intellectronica> oh yes we are
<abentley> intellectronica: Just hit out of space on the builder.
<intellectronica> bÃ¶Ã¶Ã¶
<thumper> rockstar: don't see you on line
<thumper> rockstar: on skype that is
<rockstar> thumper, I didn't leave it up.  One sec.
<rockstar> thumper, you aren't picking up.
<rockstar> barry, are we having a meeting this week.
<abentley> rockstar: sinzui ran this morning's meeting because barry is unavailable.
<sinzui> oh
<sinzui> thanks for the reminder
<rockstar> sinzui, no problem.
<Ursinha> hi gary_poster :) do you have a moment to take a look in an oops, please?
<gary_poster> sure, Ursinha
<Ursinha> gary_poster, I have an oops, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1389H3247
<Ursinha> this one happened yesterday
<Ursinha> about 40 occurrences
<gary_poster> Ursinha: and is new?
<gary_poster> (there's some recent work on that page, possibly CP)
<Ursinha> looking for a bug, I found another bug that references https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1396L2872
<Ursinha> gary_poster, I just can't find the bug # again :/ still looking
<sinzui> Working in blueprints is a deep well of sadness.
<Ursinha> gary_poster, well, my question is if these two are related
<sinzui> I am writing so many basic tests for existing features so that I can 1 attribute
<gary_poster> Ursinha, I believe they are unrelated.  Let me look at one more thing.
<Ursinha> sure gary_poster
<Ursinha> thanks
<gary_poster> Ursinha: no they are unrelated except in the basic mechanism of the problem.  For the ResetPasswordView, next_url is an @property that does not have a setter defined, and yet _cancel in the superclass BaseTokenView tries to set next_url.  Boom.  That's a Foundations bug.  If you could create the bug, I'll assign it to myself and fix it tomorrow.  Now I'm looking at the other one.
<gary_poster> Ursinha: eh, without looking at the code, that's a bugs issue
<gary_poster> They are trying to do the same thing, but the only similarity is the kind of error in our code.
<Ursinha> gary_poster, I see.
<Ursinha> gary_poster, I found bug 54387, is it related to the resetpassword one?
<mup> Bug #54387: Race condition in reset password workflow leads to an oops. <oops> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/54387>
<gary_poster> Ursinha, no, the new OOPS is simpler and unrelated other than being in the same vague area of code
<Ursinha> gary_poster, all right, filing a bug..
<gary_poster> thank you
<jml> mwhudson, hi
<mwhudson> jml: hi
<jml> mwhudson, you wanted to talk about a bug?
<mwhudson> jml: only if it's not too late for you
<mwhudson> jml: it's the "pull a branch twice at once" bug
<jml> mwhudson, sure.
<Ursinha> gary_poster, bug 462923
<mup> Bug #462923: AttributeError in +resetpassword <oops> <Launchpad Foundations:New> <https://launchpad.net/bugs/462923>
<gary_poster> Ursinha: thank you!  triaged and assigned.
<Ursinha> gary_poster, thank you very very much
<gary_poster> :-)
<Ursinha> gary_poster, considering you'll fix soon, can I target that bug to the 3.1.10 milestone?
<gary_poster> :-) sure
<Ursinha> gary_poster, thank you :)
<gary_poster> Ursinha: I can only improve a few aspects of my managing habits at a time--milestones will have to come in a later revision of my software. ;-)
<Ursinha> gary_poster, hahahaha :)
<gary_poster> night all
<thumper> mwhudson: about that vocabulary branch...
<mwhudson> thumper: ah yes
<mwhudson> thumper: i didn't get a mail about your comment :/
<thumper> mwhudson: I bet it got filtered
<thumper> abentley: bug 462628, is this the one you have just fixed?
<mup> Bug #462628: Commenting on merge proposals is broken <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/462628>
<mwhudson> thumper: i guess i'll just settle for making you put in a new bug report and an XXX
<thumper> mwhudson: what for?
<thumper> mwhudson: why is it an issue there
<thumper> ?
<mwhudson> thumper: the problem is your fix is both too narrow and too broad
<thumper> mwhudson: so just right then
<mwhudson> too narrow: it won't help other callers of getByUrl, for example the merge proposal handling code
<mwhudson> too broad: it will mask bugs in getByUrl
<mwhudson> thumper: sadly, more than one axis
<thumper> mwhudson: I don't get the to narrow bit
<thumper> s/to/too
<mwhudson> thumper: it would be far better if getByUrl raised a predictable set of exceptions
<thumper> mwhudson: it is a predictable set, just about 10 different ones
<thumper> mwhudson: I could name them all
<thumper> mwhudson: I just didn't think it was worth it
<mwhudson> thumper: if you send in a merge proposal somehow that says "target_branch = lp:", it looks very much like the mail handler will oops
<thumper> mwhudson: and they don't have a common base class
<thumper> mwhudson: most likely
<mwhudson> thumper: maybe getByUrl itself should catch them and re raise?  there are basically three answers: 1) a branch 2) not found 3) wtf are you on about
<mwhudson> 200 OK, 404 Not Found, 400 Bad Request in http terms
<thumper> mwhudson: you are suggesting having getByUrl only raise one type of Bad Request error?
<mwhudson> thumper: yeah, i think so
<mwhudson> (it returns None rather than raising NotFoundError, I take it?)
<thumper> hmm...
<thumper> yeah
<thumper> so
<thumper> Branch if found, none if valid name but not found, and WTF if bad input
<mwhudson> i see it returns None when passed an invalid URI :/
<thumper> mwhudson: we can change that
<mwhudson> thumper: so perhaps to be consistent, getByLPPath errors should be masked too
<thumper> yeah...
<thumper> so much for a simple branch
<mwhudson> sorry :/
<mwhudson> thumper: i'd be happier about just sticking the try:except: around getByLPPath
<mwhudson> in getByUrl
<mwhudson> thumper: replied on the mp
<thumper> mwhudson: ok ta
<Ursinha> hey sinzui, do you know what's https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1396EA105 about?
<mwhudson> thumper: maybe i didn't get the email before because i hadn't claimed the review
<mwhudson> (i have now)
<thumper> mwhudson: yes, but the email would have been sent to the list
<mwhudson> thumper: i'm not subscribed to the list any more
<sinzui> Ursinha: intellectronica has a fix for that already approved to land
<thumper> mwhudson: ah ha, that'll be why
<Ursinha> sinzui, great, is there a bug for that?
<sinzui> Ursinha: the subscribe/unsubscribe link is fails for anonymous users
<sinzui> Ursinha:  bug #462742
<mup> Bug #462742: OOPS getting a project index page as an anonymous user <oops> <Launchpad Registry:In Progress by intellectronica> <https://launchpad.net/bugs/462742>
 * Ursinha stabs launchpad search
<thumper> mwhudson: so are you suggesting changing the except clause in branchlookup.py:231 to be Exception? m
<thumper> mwhudson: or a list of exceptions raisable ?
<mwhudson> thumper: list of exceptions raisable
<mwhudson> thumper: consider that part optional though
<thumper> I think a list is fine
<mwhudson> cool
<thumper> mwhudson: it is except (A, B, C): isn't it?
<mwhudson> thumper: yes
<Ursinha> thanks sinzui :)
 * thumper afk for lunch and class
<intellectronica> how come we're still in testfix mode? waterfall looks green
<elmo> wgrant: around?
<wgrant> elmo: I am.
<elmo> wgrant: how familiar are you with buildd-manager?
<wgrant> elmo: Far too...
<elmo> awesome
<elmo> it's SNAFU on production
<elmo> and I'm epically failing to debug it
<wgrant> Ew. What's it saying?
<wgrant> It can be really really really obscure when it fails.
<elmo> wgrant: it's not saying anything
<elmo> it's only schedulding builds to gold
<elmo> it's ignoring every other buildd
<wgrant> Huh.
<elmo> even though they're marked as active and ok
<wgrant> A month ago it was scheduling builds to everywhere *except* gold.
<wgrant> Let's see...
<elmo> http://paste.ubuntu.com/303891/ <-- log is just that
<wgrant> elmo: A restart doesn't help, I suppose?
<elmo> nope
<wgrant> Yup, I love that verbose logging.
<Ursinha> kfogel, hi :)
<wgrant> OK.
<wgrant> Might be best to flip the logging up higher.
 * wgrant tries to remember how.
<wgrant> 'cause all the buildds look fine...
<mwhudson> probably pass -v when you run it
<mwhudson> hm, maybe not
<wgrant> No.
<wgrant> It's hardcoded.
<wgrant> elmo: You'll have to hack lib/lp/builddmaster/manager.py, in _setupLogger. s/INFO/DEBUG/
<elmo> \o/
<wgrant> Something like that.
<wgrant> So, that should convince it to actually tell you what it's doing each cycle.
<wgrant> Hm.
<wgrant> elmo: Giving you anything useful?
<elmo> sorry, got distracted, trying now
<elmo> also, p.s. die twisted logging scheme, DIE DIE DIE DIE
<wgrant> Heh.
<mwhudson> yes
<mwhudson> all our (code's) twisted stuff logs via stdlib logging
<wgrant> So does buildd-manager.
<elmo> 2009-10-28 23:42:31+0000 [-] Considering rothera
<elmo> 2009-10-28 23:42:31+0000 [-] Builder is not available, ignored.
<elmo> OH REALLY
<elmo> how about, not
<wgrant> rothera unavailable. That's nice.
 * wgrant checks the code.
<wgrant> Ah.
<wgrant> That's fine.
<wgrant> A builder must be IDLE to be available.
<elmo> oh, bad example
<wgrant> Rather.
<elmo> it says it about all of them
<elmo> e.g.
<wgrant> How about you pastebin an entire cycle, if there're no private builds around.
<elmo> unfortunately there are
<elmo> 2009-10-28 23:45:07+0000 [-] Considering mercury
<elmo> 2009-10-28 23:45:07+0000 [-] Builder is not available, ignored.
<elmo> better example
<wgrant> Let's see...
<elmo> 2009-10-28 23:46:07+0000 [-] Checking zirconium
<elmo> 2009-10-28 23:46:07+0000 [-] Checking mercury
<elmo> 2009-10-28 23:46:07+0000 [-] Checking rhenium
<elmo> sorry, I should pastebin more
<elmo> anyway, that last part is interesting
<elmo> because it doesn't do:
<elmo> 2009-10-28 23:46:07+0000 [-] Adding DistroArchSeries ubuntu/hardy/amd64
<elmo> that
<elmo> which it does fgor gold
<wgrant> Argh, buildergroup. die die die
<wgrant> Those three 'Checking ' lines are contiguous?
<elmo> yeah
<elmo> AHA
<elmo> I bet the freaking slave died
<elmo> hmm
<wgrant> On all of them?
<wgrant> That seems unlikely.
<elmo> maybe not
<elmo> wgrant: oh, sorry, context
<elmo> wgrant: is that we fucked up the networking earlier and dropped the majority of the buildds off net for ~5 minutes
<elmo> during which time cesium wouldn't have been able to contact them
<wgrant> Oh.
<wgrant> But that should have just marked them un-OK.
<elmo> right, which we undid
<elmo> but it's certainly related
<wgrant> Yeah.
<elmo> as f.e. gold is in the DC that wasn't dropped off net
<wgrant> What if you bounce the lp-buildd on one of them?
<elmo> sure, trying
<wgrant> There's not too much state on the master side.
<elmo> ok, that didn't help
<wgrant> I can't think why slaves would have died from that, though...
<elmo> I bounced palmer, no luck
<wgrant> I was hoping only the virt builders would be dead...
<wgrant> Hmm hmm hmmmmmmm.
<wgrant> It looks like it is still actually scanning properly?
<wgrant> eg. rothera's build log is updating a bit.
<wgrant> Yeah, so it's just the dispatching...
<elmo> right
<elmo> the buildds that didn't drop off net are fine and building and uploading
<elmo> god I really badly needs a TAGS-ified LP tree
<wgrant> Ah, didn't know the uploading bit worked as well. So it's not really really sick.
<elmo> grep is sooooooooooooooo slow
<wgrant> elmo: This is why you need lots of RAM.
<mwhudson> elmo: make TAGS works
<mwhudson> i think
<wgrant> And to grep through just lib/lp/buildmaster and lib/lp/soyuz.
<elmo> mwhudson: yeah, but I'm looking at code on production ;-)
<mwhudson> elmo: i see :)
<elmo> and besides my team give me grief whenever I install a real editor on the servers
<wgrant> elmo: All the lines you pasted here so far look normal.
<elmo> wgrant: yeah
<wgrant> And the buildds are certainly OK in the DB.
<wgrant> elmo: There are no attempts to dispatch?
<elmo> wgrant: to the working buildds? yes
<wgrant> It will fail really strangely if, for example, the builders can't talk to librarian or ppa.launchpad.net
<elmo> see e.g. gold's history
#launchpad-dev 2009-10-29
<elmo> I still think the fact it's not running Add DistroArchSeres is key
<wgrant> I don't think so.
<wgrant> I think it only does that once.
<wgrant> Let's see...
<elmo> it seems to do it every time
<wgrant> Once each cycle (~5 seconds), and only for ubuntu/hardy/amd64?
<elmo> no, let me paste more of the log
<wgrant> Oh. BuilderSet calls addDistroArchSeries. I see.
<elmo> wgrant: http://paste.ubuntu.com/303910/
 * elmo cries a little at the mentions of hoary in there
<wgrant> Aha.
<wgrant> OK, this is useful.
<wgrant> WTF are BuilderGroup and BuilderSet separate arrrrgh
<wgrant> elmo: cesium can actually talk to the slaves, right?
<wgrant> What if you run an XML-RPC status() on them?
<wgrant> I can't see why else they would be unavailable!
<elmo> wgrant: checking
<wgrant> 'import xmlrpclib; xmlrpclib.ServerProxy("http://osmium.ppa:8221/rpc").status()' should do it.
<elmo> AAAAAAAAAAAAAAAAHA
<elmo> good call
<elmo> ['BuilderStatus.ABORTING', '1288257-2696293']
<elmo> wgrant: ^--
<wgrant> Aha.
<wgrant> Now.
<wgrant> Oh dear.
<wgrant> lp-buildd has a bug where it won't remove the old build directories.
<wgrant> So if you restart them, they might die again.
<wgrant> But we'll see.
<elmo> oh right
<elmo> and that just happened on a massive scale
<wgrant> Does bouncing osmium's lp-buildd do anything at all?
<elmo> awesome
<wgrant> Hmm.
<wgrant> I guess that the virtual builders should be OK, since they get reset each time.
<elmo> they only get reset for a new build
<wgrant> But any non-virt *might* need manual cleanup. You'll see soon.
<wgrant> I forget exact at which stage they get handed back.
<elmo> in this case we're not getting far enough to reset them
<elmo> but I can reset them by hand easy enough
<wgrant> That might be a good idea.
<wgrant> I guess you'll probably need to then re-OK them all, but that's easy enough.
<wgrant> Urgh, apologies for not thinking to check the status() first. I got distracted by other parts of the log fragments.
<wgrant> It might be an idea to grab the lp-buildd logs on a couple of failed builders to see why they got stuck in ABORTING.
<wgrant> Since they will of course vanish once you reset the guests...
<wgrant> I presume it's the residual build directory, but cannot be sure.
<wgrant> sbuild must have got really really wedged (it should have been SIGKILLed 10 seconds after the abort), or Twisted is being bad.
<elmo> wait what
<elmo> we knocked the buildds off line at 17:00
<elmo> palmer was building at 20:00
<wgrant> palmer is fine.
<wgrant> The queue is just empty.
<elmo> hmm
<wgrant> (if you look at the logs, it will tell you that there are no candidates for palmer)
<elmo> right
<elmo> I'm not sure it always was though
<elmo> or maybe I'm losing my mind
<elmo> the problem is i can't actually check on the virtual builders
<wgrant> Why not?
<elmo> we lock ourselves out of the guests for reasons of paranoia that are probably not sane
<wgrant> Ah.
<wgrant> Well.
<wgrant> We can at least get the logtail.
<elmo> ok, how about sparc
<elmo> that's certainly got shit to do :)
<elmo> wgrant: we can?
<elmo> interesting
<elmo> AFAICT, artigas was mid-build when cesium lost contact with it
<elmo> so I suspect this is readily reproducible
<wgrant> elmo: We can, through RPC. I would find the call, but I've had to move to another machine; it seems my WAP is unhappy.
<wgrant> Ah, no, we can't.
<wgrant> We can only get it while it's BUILDING :(
<wgrant> elmo: What became of that slave that you bounced before the issue was properly diagnosed?
<wgrant> Did it recover?
<elmo> that was palmer?
<elmo> I've reset all the virtual ones now
<elmo> artigas is still stuck and is:
<elmo> >>> import xmlrpclib; xmlrpclib.ServerProxy("http://artigas.buildd:8221/rpc").status()
<elmo> ['BuilderStatus.WAITING', 'BuildStatus.OK', '1312047-2740484',
<wgrant> Ahhh.
<elmo> I suspect bouncing it's slave will recover it
<wgrant> So palmer wasn't actually broken.
<wgrant> Hm.
<wgrant> That means it's waiting for the upload to happen.
<wgrant> What does buildd-manager's log say about it?
<elmo> whatever was in the pastebin I guess?  but checking
<wgrant> If you bounce it, it will have to build again, and you might have to manually clean up the build directory. I would have thought buildd-manager should have dealt with that one.
<wgrant> Did it recover, or did you kill it?
<elmo> artigas?  I bounced the slave
<wgrant> Aha.
<elmo> and am cleaning up byhand
<wgrant> Great.
<elmo> wgrant: thanks a lot for your help
<wgrant> elmo: np.
<wgrant> I can reproduce the artigas situation.
<wgrant> buildd-manager just sits there and ignores it.
 * mwhudson afk for a short while
<wgrant> elmo: OK, I think I see why artigas was special.
<elmo> oh?
<wgrant> elmo: There is code to rescue lost builders.
<wgrant> That is invoked if they report that they are currently building a (build ID, queue ID) pair that doesn't exist any more.
<wgrant> It is that code which aborts the build.
<wgrant> However...
<wgrant> there is a flaw!
<wgrant> It doesn't check that the build is still assigned to the builder.
<wgrant> So when the builder times out, it is unassigned from the buildqueue, and the build goes back to 'Needs Building'. But when the slave is scanned, the master sees that the referenced buildqueue still exists, so doesn't attempt a rescue.
<wgrant> For the PPA builders, that will resolve pretty quickly -- another builder will grab the build, complete it, and the buildqueue will disappear.
<wgrant> At that point, the builder will reference something that doesn't exist, and will be aborted.
<wgrant> But for sparc, there are only two buildds. I guess artigas's current build wasn't picked up by the other buildd, so it never got rescued.
<wgrant> But there are at least two bugs here, since the abort on at least some of the virt builders hung.
<wgrant> elmo: I have reproduced all pieces of the problem locally.
<wgrant> Including the ABORTING hang.
<elmo> wgrant: great
<wgrant> elmo: The pile of rescue bugs is growing...
<poolie> rockstar: or other, could you approve my list request for https://edge.launchpad.net/~ubuntu-distributed-development
<rockstar> poolie, now that's a question that could have gone to #launchpad - You're all turned around today.  :)
<poolie> i am :)
<poolie> this is not helped by pidgin showing this channel as "launchpad..." :)
<rockstar> poolie, :)
<rockstar> poolie, generally, anything Ubuntu related is passed to jorge
<poolie> ok
<poolie> and he has the (technical) right to approve it?
<thumper> poolie: normally it is because ubuntu lists are on ubuntu.com not launchpad.net
<poolie> mm
<poolie> is there a technical reason for that?
<thumper> that I don't know
<rockstar> poolie, no, I think it's so that he's aware of what's going on, and so that we don't have a bunch of ubuntu lists.
<poolie> ok, i'll mail him
<lifeless> hah
<lifeless> new list notice still points at launchpad.dev
<lifeless> poolie: did you undo the udd membership?
<poolie> i don't think so
<poolie> what do you mean?
<lifeless> well I got the notice that canonical-bazaar is in the team udd, and that we can enable list delivery
<lifeless> but I can't see the list on https://edge.launchpad.net/~lifeless/+editemails
<poolie> i presume it's because the list is in this half-created stnate
<lifeless> oh
<lifeless> have you pung spm to whack it on?
<poolie> read the scrollback
<poolie> mwhudson: did you get that mail about graphs?  do you think you could do it reasonably easily, not necessarily today
<mwhudson> poolie: yes, just writing up the last one now
<poolie> you legend
<lifeless> spm: ping; whats the mail list approval ul?
<lifeless> URL
<poolie> lifeless: i sent mail to jorge, i'll see what he thinks
<poolie> i don't want to force it today
<spm> lifeless: +mailinglists
<lifeless> poolie: hmm? if you've applied for the list, the CHR will probably just do it when they notice
<lifeless> oh, it has the word ubuntu, it may need the special treatment.
<lifeless> spm: thanks, but ignore that ;)
<poolie> no further action is needed til jorge replies
 * spm adds lifeless back to /ignore :-P
<poolie> :)
<lifeless> ooh quick, while hes not listening...
<spm> heh
<mwhudson> poolie: sent
<mwhudson> so where has inline editing of merge proposal status gone?
<thumper> mwhudson: it isn't there yet
<thumper> mwhudson: it is branch status that has been done
<thumper> mwhudson: it is on my todo list
<mwhudson> thumper: oh right
<poolie> mwhudson: could you add lpstats for failing imports by type?
<mwhudson> poolie: yes
 * mwhudson writes an RT
<thumper> mwhudson: want to add to that for one for succeeding imports by type?
<mwhudson> thumper: yeah
<mwhudson> suspended/invalid probably not all that interesting
<cody-somerville> Isn't there instructions somewhere on how to setup launchpad development instance without running rocketfuel-setup?
<thumper> cody-somerville: yes (ish)
<thumper> cody-somerville: I did it recently
<thumper> cody-somerville: as I don't use rocketfuel scripts
<thumper> cody-somerville: read the script and do the bits that make sense :)
<thumper> I don't use the 'blessed' directory or repo layout
 * thumper EODs for now
<wgrant> I see somebody renamed samarium to 'samariumd' fairly recently.
<wgrant> Somebody might want to fix that.
<elmo> we can't
<elmo> LP breaks when you try
<wgrant> elmo: Oh, it's always been like that? Hmm.
<elmo> wgrant: no it's recent
<elmo> we've no idea how it changed, but the web ui won't let you change it back
<rockstar> mwhudson, have you landed your branches I've reviewed.  pqm threw up on me.
<wgrant> Niiice.
 * wgrant tries to rename one locally.
<wgrant> Hm, WFM
<wgrant> elmo: How does it complain when you try?
<wgrant> (and are you going to sleep at some point?)
<mwhudson> rockstar: yes, i have a success mail from PQM
<mwhudson> rockstar: i just did ec2 land fwiw
<elmo> wgrant: it just times out, I think - I haven't had a chance to dig into the oops, I only noticed it myself earlier this evening
<wgrant> elmo: Urgh. That's not too nice.
<rockstar> mwhudson, hrm.
 * mwhudson EODs
<cody-somerville> wgrant, whats the url to your blog?
<adeuring> good morning
<henninge> Moin adeuring!
<adeuring> hi henninge!
<henninge> Hallo jtv!
<jtv> hi henninge!
<henninge> jtv: your memory footprint is about to land! cool!
<henninge> work
<jtv> henninge: yes... there's still a lot of buildup in the final phase, but the first phase has been tamed.
<wgrant> cody-somerville: I have none. Why?
<cody-somerville> wgrant, I thought you wrote a blog post on setting up soyuz.
<wgrant> cody-somerville: http://williamgrant.id.au/f/1/2009/running-soyuz.html
<wgrant> Should probably throw that on the wiki at some point.
<wgrant> lp-buildd is the troublesome bit, particularly on Karmic.
<jtv> henninge: I'm about to go for lunchâttyl!
<henninge> jtv: ok, enjoy it!
<wgrant> But even on pre-Karmic it requires some hacking if you grab official Ubuntu buildd chroots.
<wekt> will trade a bug triage.  you triage 1 mine, i'll try to triage yours.  (I have not triaged for Ubuntu yet) https://bugs.launchpad.net/launchpad/+bug/453804
<mup> Bug #453804: absolute font sizing and size smaller than default creates accessibility and usability difficulties <Launchpad itself:New> <https://launchpad.net/bugs/453804>
 * wgrant plots an evil PPA package which grabs stuff from the librarian, and uses a launchpadlib script to silently inject arbitrary untraceable binaries into the resultant packages.
<wgrant> I think that would work.
<noodles775> yeah, thanks wgrant.
<wgrant> noodles775: Hm?
<henninge> hey danilo_, danilos !
<noodles775> hm... is anyone else having trouble accessing dev.launchpad.net?
<wgrant> I couldn't last night, but that was massive packet loss in L3.
<wgrant> It's slow, but working.
<noodles775> ok, it's working again now... :)
<noodles775> yep, thanks wgrant.
<wgrant> al-maisan, noodles775: I suppose neither of you know too much about buildd-manager? I've fixed one of the pieces of this morning's build farm disaster, but I'm not sure about how to fix the other.
 * al-maisan looks at wgrant's emails again
<noodles775> wgrant: I've had to take a look at it recently and al-maisan is familiar with it too... so shoot (it'll be good for us) :)
<wgrant> Bug #463041 was easily fixed, but bug #463046 is rather less obvious.
<mup> Bug #463041: Lost builder detection is insufficiently aggressive <Soyuz:New> <https://launchpad.net/bugs/463041>
<mup> Bug #463046: Rescuing a BUILDING builder just makes things worse <Soyuz:New> <https://launchpad.net/bugs/463046>
<noodles775> wgrant: were you able to setup a test that shows the first issue that you mentioned in that bug?
<noodles775> (463041?)
<noodles775> ok.
<wgrant> noodles775: 463041 is fixed and tested in a branch here.
<noodles775> Great!
<wgrant> Landing that now would just make everything worse, however.
<noodles775> wgrant: have you got time for a call? (skype: absoludity)
<wgrant> noodles775: Sure. Let me just install Skype...
<noodles775> al-maisan: we can do a conf?
<al-maisan> noodles775: nice :)
<wekt> ekiga works well instead of skype as long as it gets ALSA directly.  pasuspender can be used for that
<wgrant> Hopefully Empathy will be usable for this sort of thing soonish.
<wekt> empathy uses all plugins through dbus.  I'd rather use pidgin and libpurple
<noodles775> I'm happy to use ekiga ... (although not sure of a conf line I can use).
<al-maisan> ekiga works here as well
<wekt> it has a conference room, but i have not tried it
<noodles775> ok, let's try that first then :)
<al-maisan> what's the sip address?
<al-maisan> 501@ekiga.net
<wekt> I'm there
<noodles775> I'm not... I can call other conf lines in ekiga, but that's just dropping for me :/
 * noodles775 registers at ekiga.
<wekt> is your account @ekiga.net?
<al-maisan> wekt: al-maisan
<wekt> al-maisan & I are chatting in the conference room
<wekt> i hear a third party.  with much humm
 * wgrant awards Ekiga the "even worse than Skype" award.
<al-maisan> nice award :)
<wekt> you might turn on silence detection.
<wgrant> (I have Skype working, but cannot get Ekiga to hear me :()
<wekt> i hear you
<al-maisan> we did hear somebody say "hello"
<wgrant> Ah.
<wgrant> I see.
<wgrant> Let's try that again.
<wekt> you might turn down your mic boost you were quite loud.
<wekt> & turn on silence detection
<wekt> I leave now
<al-maisan> bye wekt
<al-maisan> wgrant: I do hear you
<wgrant> al-maisan: The only thing I've heard from Ekiga is a very broken up introduction to the conference room :(
<wgrant> Skype works fine; shall we use that?
<al-maisan> hmm ..
<al-maisan> wgrant: yup
<wgrant> I'm william.a.grant.
<al-maisan> wgrant: what's your skype id? Mine is 'mhrnjad'.
<al-maisan> OK
<wgrant> I've added both you and noodles775.
<al-maisan> wgrant: thanks
 * wgrant has very little idea how to use Skype.
<al-maisan> noodles775 seems busy with a code review ..
<al-maisan> wgrant: let me call you
<elmo> wgrant: did you file any bugs about that drama last night btw?  (if not, I can, just don't want to duplicate)
<wgrant> elmo: I did.
<elmo> wgrant: neato; thanks
<wgrant> al-maisan: https://code.edge.launchpad.net/~wgrant/launchpad/rescue-robbed-builders
<wgrant> Well, that is a nice simple solution.
<wgrant> elmo: Just fixing and writing tests for the second master issue now. However, there still remains the issue of the ABORTING slave hang, which I don't know how best to resolve. I'm thinking that if lp-buildd can't be easily fixed, the master could be taught to just fire the reset trigger if it hasn't aborted after a minute or so.
<wgrant> Is there some script around to dispatch builds to manual builders?
<wgrant> I've occasionally seen it happen on production, and it would handy for testing sometimes.
<deryck> Morning, all.
<_thumper_> deryck: hey, sent you an email :)
<deryck> thumper, I saw.  Was going to reply here in a sec.
 * al-maisan needs a coffee
<maxb> I have an MP targeted at ~launchpad-dev//ztk-2.5 if someone has a moment: https://code.edge.launchpad.net/~maxb/launchpad/pymig-remove-lsprof/+merge/14091
<salgado> maxb, r=me
<allenap> jml: I have a branch that moves devscripts out of lib/. I've updated the Makefile to run the devscripts tests with zope.testrunner and trial, and I wondered which I should actually go for? See http://pastebin.ubuntu.com/303684/. Trial doesn't do subunit does it? zope.testrunner doesn't either, but there is (your) code to make it work already in the tree.
<jtv> BjornT_: You're at least one step ahead of me when it comes to submitting lp-production-configs branches.  Can you help me get to that point?
<jtv> salgado, you've got one of those branches in pqm... can you explain to me how to submit to lp-production-configs?
<jtv> I'm documenting this stuff, too.
<jtv> ah, it wasn't salgado, it was matsubara-afk
<salgado> jtv, I don't think I have any branches on PQM
<jtv> salgado: I was wrong, sorry.  I always knew I'd confuse you to someday
<jtv> you *two
<BjornT_> jtv: how far have you come so far?
<jtv> BjornT_: I'm trying to pqm-submit, but I don't know how to set up bzr for it.
<BjornT_> jtv: this is in my locations.conf: http://pastebin.ubuntu.com/304261/
<jtv> BjornT_: cool, thanks.  If I turn this into wiki documentation, would you mind casting a glance at it and seeing if it makes sense?
<BjornT_> jtv: sure. it's similar to all the other pqm-managed lp branches; it's just different product names and branches. there should be some documentation already...
<jtv> BjornT_: I searched all wikis for lp-production-configs without any luck
<BjornT_> jtv: well, i was thinking about pqm in general. i can't find anything either, except for some really old instructions
<jtv> BjornT_: cool, then we have a more or less clean slate.  :-)
<maxb> salgado: You did review:approve on my ztk-2.5 MP but not merge:approve - just oversight?
<salgado> maxb, yeah, I always forget that
<salgado> will do now
<maxb> I should try my hand at ajaxifying that
<BjornT_> EdwinGrubbs: ping?
<EdwinGrubbs> BjornT_: pong
<BjornT_> EdwinGrubbs: i think you had problems with windmill, in that asserting a property/element sometimes failed, after having waited for it, right?
<EdwinGrubbs> BjornT_: right
<BjornT_> EdwinGrubbs: so, i'm not sure why that is, i have to take a closer look. but i'm interested, why do you have to use an assert at all? why not do like this? http://pastebin.ubuntu.com/304288/
<EdwinGrubbs> BjornT_: basically, I don't believe theat waits.forElementProperty() works correctly if the assert after it fails. I would have removed it if I didn't start getting spurious errors.
<BjornT_> EdwinGrubbs: does it fail if you make sure the element isn't there?
<BjornT_> EdwinGrubbs: also, an occasional spurious success is better than an occsional spurious failure :)
<BjornT_> gary_poster: would you know anything about this error when running utilities/paste? http://pastebin.ubuntu.com/304293/
<BjornT_> gary_poster: the only thing before the _pythonpath import is whitespace and comments
<gary_poster> BjornT_: how are you starting paste?
<BjornT_> gary_poster: in that instance, by invoking /devel/launchpad/rocketfuel/utilities/paste
<BjornT_> gary_poster: hmm. i noticed that it's only there. it works in another branch, which should be just as up-to-date
<gary_poster> BjornT_: OK. Is ascii the usual Python encoding, and your system is mildly unusually specifying utf8?  If so, I would suggest modifying buildout-templates/_pythonpath.py.in to include encodings.utf_8 as hard-coded in the clean-modules list and committing with rs=gary.  There may be a better way to handle it, but that is a reasonable start IMO.
<EdwinGrubbs> BjornT_: yes, I believe it normally fails normally if I assert that the node doesn't have the style. I have not tested if there would be spurious successes with that. I'm not sure how that makes the test any better.
<BjornT_> EdwinGrubbs: because a spurious success means that we would miss catching a test failure in some rare instances; it would probably never happen. a spurious failure means that the we enter testfix mode, which is quite bad.
<maxb> LP auto-sets my MP to "Merged" when it is merged. Is it intentional that it doesn't also update the branch status to "Merged" ?
<BjornT_> EdwinGrubbs: we should be able to trust waiting for an element. my guess is that waiting for an element gets notified that the element gets added slightly before it actually gets added to the dom, which asserting it checks the dom directly, so you have a race condition there
<jtv> BjornT_: I seem to have pqm-submit working for my configs branch... would you care to review the documentation byproduct?  https://dev.launchpad.net/LoggingOopses
<jtv> BjornT_: if you don't see anything horribly wrong with it, I'll start linking to it and asking for more feedback from mailing lists.
<EdwinGrubbs> BjornT_: I assume what you are saying is that we catch that error and just ignore it. The waits.forElementProperty() also causes this spurious error, and that situation doesn't involve the timing of adding an element to the dom.
<EdwinGrubbs> BjornT_: why check the node if we are going to ignore the outcome?
<BjornT_> jtv: well, the documentaion looks good, but it seems misplaced. i would never have thought of looking for that documentation in a document called LoggingOopses
<BjornT_> EdwinGrubbs: s/element gets added/dom gets modified/
<BjornT_> EdwinGrubbs: i'm saying that waiting for an element/property probably works, and we should trust it. we don't have to assert that the wait worked everytime we wait for something. so we don't ignore the outcome. if the wait fails, the test fails.
<mars> Has anyone had a problem building ctypes on Karmic?
<mars> The egg build step complains about a missing Python.h file... missing the Python2.5 version maybe?
<BjornT_> mars: does the build fail? sometimes errors are produced for other pythons than 2.4, but they can be ignored.
<mars> BjornT_, yep, build fails: An error occured when trying to install ctypes 1.0.2.Look above this message for any errors thatwere output by easy_install.
<mars> make: *** [/home/mars/canonical/lp-branches/trunk/bin/py] Error 1
<jtv> BjornT_: the meat is to be about the code part, really.
<mars> might have to do a make clean
<BjornT_> mars: does it help if you install python-dev?
<mars> BjornT_, it is already installed for Python2.6.  Are we running on 2.6 yet?
<BjornT_> mars: is python2.4-dev installed?
<mars> BjornT_, nope
<mars> it must have been cleaned up during the upgrade
<EdwinGrubbs> BjornT_: I see what you are saying. I think I've just become pessimistic about windmill doing what I expect.
<BjornT_> mars: then you probably don't have launchpad-dependencies install either. the lp ppa does get disabled during upgrades
<mars> BjornT_, looking at the make output - looks like we are still on python2.4.  thanks for the pointer
<mars> yep, have to re-enable that next
<BjornT_> EdwinGrubbs: i'll certainly look into this, to see what is really going on, but until then i'm going to change the tests to be more robus
<EdwinGrubbs> ok
<maxb> I just pushed a branch: https://code.edge.launchpad.net/~maxb/launchpad/remove-stray-ref-to-sourcecode-lazr-js and it has a weird error
<maxb> No such file: u'/srv/bazaar.launchpad.net/mirrors/00/03/eb/bb/.bzr/repository/upload/28c77e113c8b6a60a9181c736b1c6ddf.pack': [Errno 2] No such file or directory: u'/srv/bazaar.launchpad.net/mirrors/00/03/eb/bb/.bzr/repository/upload/28c77e113c8b6a60a9181c736b1c6ddf.pack'
<mars> BjornT_, yep, the Karmic upgrade is the culprit.  I thought it was strange when the cleanup step told me Â¨You donÂ´t need all these developer tools any moreÂ¨
<Ursinha> a translations person is missing, so I think I'll be the double agent in the meeting
<Ursinha> stub, Chex, gary_poster, rockstar, bigjools, sinzui, allenap, Ursinha -> production meeting @ #launchpad-meeting in 18 mins
<allenap> Ursinha: Google Calendar says in 1h15m...
<Ursinha> allenap, google calendar is wrong :)
<sinzui> google calendar has difficulty with UTC
<allenap> Ursinha: Cool.
<Ursinha> allenap, unless UTC time changed...
 * sinzui only uses the shite...site under duress
<allenap> sinzui: I agree on both points.
<Ursinha> stub, Chex, gary_poster, rockstar, bigjools, sinzui, allenap, Ursinha -> production meeting @ #launchpad-meeting in 9 mins
<Ursinha> henninge, ha! :) are you joining the prod. meeting in 5 mins?
<henninge> Ursinha: I can "me" there, if you like ... ;)
<Ursinha> henninge, I can wear both hats, if you like
<Ursinha> I don't mind
<henninge> Ursinha: Please do that, there is no special information that I have
<henninge> at least, not that I can think of.
<Ursinha> henninge, except that I'll ask someone to triage https://bugs.edge.launchpad.net/rosetta/+bug/462891
<mup> Bug #462891: TraversalError on +export <oops> <Launchpad Translations:New> <https://launchpad.net/bugs/462891>
<Ursinha> :)
 * henninge looks at that
<Ursinha> thaaaanks henninge :)
<Ursinha> stub, Chex, gary_poster, rockstar, bigjools, sinzui, allenap, Ursinha -> production meeting @ #launchpad-meeting now :)
<maxb> I have a branch which has somehow broken - should I file a launchpad-code bug for someone to investigate? https://code.edge.launchpad.net/~maxb/launchpad/remove-stray-ref-to-sourcecode-lazr-js
<maxb> broken, as in, I pushed it, and it ended up in that stuck state
<Ursinha> rockstar, is there something maxb can do to dig the problem a bit more?
<rockstar> maxb, have you tried pushing again?
<maxb> rockstar: For my own use, I re-pushed the branch under a different name. I've left that one alone in case anyone wants to debug its state. If no one does, I shall simply delete it
<rockstar> maxb, yeah, I'd suggest just deleting it.  It looks like there was some issue on push.
 * sinzui plants face in plam
<sinzui> abentley: thank you for your bug report. launchpad.dev is hardcoded in a template
<abentley> sinzui: No problem.
<danilos> BjornT_, hi, still around?
<mrevell> Night all!
<rockstar> sinzui, jam is reporting many timeouts with https://edge.launchpad.net/bzr/+download
<mwhudson> good morning
 * mwhudson looks at oops-prune failure mails
<thumper> mwhudson: we should grab spm later
<thumper> oh and good morning
<mwhudson> thumper: good morning
<mwhudson> thumper: to run branch-distro.py ?
<mwhudson> thumper: or something else?
<thumper> mwhudson: yes to run branch-distro
<mwhudson> thumper: we could get mbarnett to start it now i guess?
<mwhudson> or at least do some prep
<thumper> mwhudson: what prep is needed?
<mwhudson> thumper: well getting a newer launchpad tree onto crowberry
<mwhudson> not much else i admit
<thumper> rockstar-afk: ping
<thumper> abentley: skype?
<thumper> sinzui: we are due to talk right?
<sinzui> I think so
<thumper> sinzui: when?
<thumper> sinzui: I think my local calendar is in local time
<sinzui> 40 minutes
<sinzui> if we are keeping UTC
<thumper> sinzui: can you do earlier, or shall we do it in 40?
<sinzui> I can do it earlier. We can talk now if you like
<thumper> sinzui: ok, lets do it now
<poolie> interesting responses re imports hey?
<poolie> but perhaps not reaching many people we don't already know
<rockstar-afk> thumper, pong
<awilkins> Offtopic but you guys are the most likely guys to know stuff about servers at Canonical I know of... anyone know if keyserver.ubuntu.com is being thrashed or just down?
<rockstar> awilkins, yeah, I don't think many people would know the direct answer to that.
<awilkins> I thikn just thrashed... it just served me a key
<awilkins> But it's very slow
<wgrant> Well, consider what day it is.
<awilkins> Indeed
 * rockstar awards wgrant a gold star
<wgrant> rockstar: What have I done now?
<rockstar> wgrant, pointing out that it's release day.
<wgrant> Ah.
<awilkins> Bah, my ISP still hasn't mirrored it
<wgrant> awilkins: Torrents!
<awilkins> Oh, I had the ISO before it became a release ISO
<awilkins> Three of them
<wgrant> awilkins: (also, #canonical-sysadmin is the right place)
<awilkins> Via torrents
<rockstar> wgrant, torrenting is illegal.
<wgrant> Yes, I know. Please don't announce it like that next time.
<wgrant> awilkins: Or we will kick you more aggressively.
<wgrant> rockstar: Naturally. Evil nasty Ubuntu pirates.
<awilkins> Hmm. I really don't see how uploading files to a world-visible webserver doesn't constitute an announcement in itself... but que sera
<wgrant> awilkins: Mirroring has to be done before the release is widely known, or mirroring becomes impossible.
 * thumper is looking for a nice unicode character to test with
<thumper> any suggestions?
<spm> thumper: â
<spm> cofee mug. how could you go wrong.
<thumper> spm: which character is that
<lifeless> â
<lifeless> obviosuly
<thumper> haha
<spm> braille is another useful to try
<thumper> 2615 apparently
<wgrant> thumper: You know you can use literals like u'\N{INTERROBANG}'?
<thumper> wgrant: and what does an interrobang look like?
<thumper> wgrant: and what is the name of the coffee cup?
<wgrant> thumper: It's a combo exclamation mark question mark.
<thumper> wgrant: I had vague recollections, thanks for reminding me
<thumper> abentley: still around?
<thumper> abentley: nm, fixed it
<thumper> mwhudson: had to fix a bug in attachment encoding to write the test for attachment viewing :-|
#launchpad-dev 2009-10-30
<mwhudson> thumper: heh
<thumper> mwhudson: want to review it?
<mwhudson> thumper: yeah alright
 * thumper sends
<thumper> mwhudson: sent, now just flowing through the ether
<mwhudson> thumper: ok
 * thumper heading afk for a while
<thumper> mwhudson: hmm, my bzr send caused an oops :(
<thumper> I'll look at that later
<thumper> mwhudson: did a manual propose
<mwhudson> ok
<mwhudson> thumper: i'm going to get this script running with spm and then break for lunch though
<spm> mwhudson: it may take a while; that's a huge patch for my itsy network connection.
<mwhudson> spm: 75k?
<spm> mwhudson: your irony detector is broken. ;-)
<mwhudson> i see :)
<mwhudson> it needs lunch too, it seems
<spm> that'd do it.
<wgrant> Uhoh.
<wgrant> Some bits of LP use string sorting on version numbers.
<wgrant> (DistributionSourcePackageView.active_series is the most obvious so far)
<mwhudson> i guess we've got one more release before that bites us on the arse?
<wgrant> mwhudson: No. Lucid exists now.
<wgrant> eg. see https://edge.launchpad.net/ubuntu/+source/dpkg
<wgrant> Just display glitches, fortunately.
<wgrant> But still had me confused that Lucid wasn't yet initialised.
<mwhudson> wgrant: oh yeah
 * Ursinha looks at spm
<spm> Ursinha: yo!
<Ursinha> :)
<Ursinha> spm, hey :) are you busy now?
<Ursinha> terribly-dying busy?
<spm> mind you - that's an impressive 15000km stare you have :-D
<Ursinha> hahaha
<spm> Ursinha: nothing that can't be put aside; what's up?
<Ursinha> spm, it's a simple question, that maybe has a simple answer :)
<Ursinha> spm, I see emails of a garbo-hourly script failing
<Ursinha> do you know why is it happening?
<spm> hahahahaa oh the irony.
<Ursinha> why? :)
<spm> I was just chasing that and emailed stub about 15 mins ago :-)
<Ursinha> oh, hehe
<spm> afaict, it is working; but isn't logging to scriptacivity.
<Ursinha> hmmm, I see
<spm> there's an error in the log file; that may be glitching the "success"; but locks are opened and removed; so it looks good.
<spm> checkwatches was hung - if that's the next question? :-)
<wgrant> Of course. It's checkwatches. It hasn't worked for ages.
<thumper> spm: how's our script going?
<spm> thumper: branch-distro ? you've not read mwh's email?
<mwhudson> thumper: it's not :(
<thumper> ah
<thumper> bum
<mwhudson> yes
<thumper> mwhudson: so... my review?
<mwhudson> thumper: sorry forgot, am on it now
<mwhudson> or would be if launchpad was behaving
<thumper> mwhudson: so... my oops with send oopsed due to a branch lock
<mwhudson> thumper: bzr lock or database lock?
<mwhudson> "!" either way mind
<thumper> bzr lock
<thumper> mwhudson: the scanner acquires a branch lock
<thumper> so...
<mwhudson> no it doesn't
 * thumper thinks
<thumper> ah
<mwhudson> or at least, it's only a read lock
<thumper> I'm pretty sure it does
<mwhudson> it access the branch over read only http
 * thumper looks
<mwhudson> if it can obtain an exclusive lock over that transport something very strange is going on! :)
<thumper> yep, lock-read
<thumper> so...
<thumper> how does that work then?
<mwhudson> a read lock just means that the bzr client assumes the branch won't change under it
<mwhudson> it caches revnos & probably some other things
<thumper> so...
<mwhudson> thumper: i would guess the lock was from the puller
<thumper> hmm...
<thumper> possibly
<thumper> I guess there isn't much else that tries to lock it
<lifeless> mwhudson: no, it doesn't assume it won't change
<lifeless> mwhudson: it presents a stable view of the data to the holder of the read lock
<mwhudson> er i guess that's more accurate yes
<lifeless> e.g. successive calls to last_revision() are the same within a read lock
<lifeless> old formats did mutually exclude writers
<lifeless> when we used os locks for reads of branch/repository
 * thumper goes to purchase many pizzas to feed a gaggle of girls
<mwhudson> thumper: i'll have your branch reviewed when you get back i hope
 * thumper is back
<mwhudson> thumper: ok i was wrong then
<thumper> mwhudson: :)
<thumper> I did look though
<mwhudson> thumper: the only concern i have really is what it will look like if the attachment isn't actually a diff
<thumper> mwhudson: the diff formatter looks at the first character
<thumper> to decide how to colour it
<mwhudson> thumper: oh ok
<thumper> but yes
<thumper> I did consider possibly doing text/plain different
<mwhudson> thumper: what would happen with say image/jpeg?
<mwhudson> do they get filtered somewhere else?
<thumper> but decided against it
<thumper> the diff formatter is pretty dumb
<thumper> most likely you'll get line numbers and everything plain
<thumper> for things that aren't actually diffs
<thumper> mwhudson: yes, not shown
<mwhudson> thumper: cool
<thumper> mwhudson: that is a different bug
<mwhudson> :)
<thumper> or feature
<mwhudson> not displaying random binaries as text is definitely a feature
<thumper> mwhudson: there is a filter on text/plain, text/x-diff, text/x-patch
<mwhudson> cool
 * mwhudson EOWs
<thumper> mwhudson: have a good weekend
<mwhudson> thumper: you too!
<jtv> stub: I think I've got my memory hog.  Essentially the same as in phase 2 of the algorithm.  Generalizing the same solution to re-apply it.  Care to review it in a few?
<jtv> stub: got my branch ready... would you be willing to (1) do a test run and (2) review it?
<jtv> stub: you already know a lot of what goes on in there, so you're probably the best reviewer at this point.
<stub> k
<stub> jtv: k
<stub> jtv: What was the branch again?
<stub> http://bazaar.launchpad.net/~jtv/launchpad/scale-message-sharing-migration/
<stub> jtv: I'll need to know the last unreviewed revision too
<jtv> hi henninge
<jtv> stub: I must be off... any news?
<henninge> Hi jtv, I somehow missed your hailing ...
<jtv> henninge: never mind, I'm not here
<henninge> jtv: oh right, that's why I didn't notice you, then ... ;)
<jtv> henninge: no, no word
<henninge> jtv: I'll be relocating now
<adeuring> good morning
<mrevell> Morning!
<henninge> Hello!
<henninge> Does anybody have an idea what this might be about? http://paste.ubuntu.com/304910/
<noodles775> henninge: usually it means that you're not currently logged in?
<henninge> noodles775: oh, that is true (that I am not logged in)
<noodles775> great.
<henninge> noodles775: I am doing this in a pagetest
<noodles775> henninge: yep, so pop a login('foo.bar@canonical.com') before it (or with relevant user).
<stub> henninge: Some code expects you to be logged in, such as something trying to security wrap an object.
<jml> good morning
<henninge> noodles775, stub: Thank you, that was it. I needed to logout, too, so that the rest of the test can run as before.
<noodles775> np.
<maxb> Is there a standard procedure for updating the version number of Launchpad in the Launchpad sourcecode documented anywhere?
<salgado> maxb, I don't think so.  do you just want to know how to do it or are you planning to create such doc if one doesn't exist?
<maxb> A bit of both. Mainly I think it sucks that some places still say 2.2.3
<salgado> really?  where does it say that?
<salgado> all places which mention the release should be using modules/canonical.launchpad.versioninfo/release
<maxb> salgado: setup.py
<salgado> and that call I pasted above would read the version number from /version.txt
<salgado> maxb, would it be possible to change canonical/launchpad/versioninfo.py to read the version number from setup.py?  that way we'd have that info in only one place
 * wgrant wouldn't mind another set of eyes on bug #54730.
<mup> Bug #54730: launchpad buildd abort does not work as expected <Launchpad Auto Build System:Triaged> <https://launchpad.net/bugs/54730>
<wgrant> Although I guess there are few left who know lp-buildd :(
<maxb> salgado: How about doing this in setup.py instead: ?
<maxb> import imp
<maxb> versioninfo_module = imp.load_source('setup_py_versioninfo',
<maxb>         'lib/canonical/launchpad/versioninfo.py')
<maxb> __version__ = versioninfo_module.release
<salgado> that adds one more level of indirection
<salgado> the actual version would go version.txt -> versioninfo -> setup.py
<salgado> if we move the version to setup.py, it'd go setup.py -> versioninfo.py
<maxb> The problem with putting it in setup.py is that you need to be able to load it without executing any of the setuptools gumph
<maxb> I guess we could move almost all of the current setup.py inside an if __name__ == '__main__'
<salgado> yeah, that'd work
<wgrant> How do you get at setup.py from versioninfo.py?
<maxb> Same way you get at bzr-version-info.py ?
<wgrant> Ah. I see.
<maxb> Though maybe rather than moving all of setup.py into an if-block, it would be neater to have setup.py and c.l.versioninfo both get the version from a oneliner lp_version_info.py ?
<henninge> Ursinha: what are you trying to achieve?
<Ursinha> henninge, good!
<Ursinha> henninge, I'm trying to add a Last Edited column to the +translations pages, like this one: https://translations.launchpad.dev/alsa-utils/trunk/+translations
<henninge> oops, no dev running here atm
<henninge> Ursinha: I think you are on the wrong track
<Ursinha> so I've added it to the template, and am trying to use the fmt:datetime do display the date the same way it's displayed in this one: https://translations.edge.launchpad.net/ubuntu/karmic/+lang/pt_BR/+index
<henninge> Ursinha: you are trying to format a datetime object
<Ursinha> henninge, hmm
<Ursinha> go ahead
<henninge> wait
<Ursinha> sure :)
<henninge> so, what goes behind the colon (:) is not what you are trying to format, but *how*
<henninge> let me look up what it's called again
<Ursinha> henninge, I know that
<Ursinha> :)
<Ursinha> henninge, the point is what do I have to do for it to use the fmt:datetime
<Ursinha> I couldn't find out what the other page does that I'm not doing
<Ursinha> I'm getting a traversal error
<henninge> Ursinha: sorry, I just saw that there is actually a formatter called datetime, didn't know that
<Ursinha> :)
<henninge> Ursinha: show me the code
<henninge> ;)
<Ursinha> henninge, sorry, sometimes I assume that you guys are translations code gods and know everything about it
<Ursinha> henninge, let me try something else here, a moment, please :)
<henninge> I have actually found out how to use a formatter from python and found that pretty cool
<henninge> ;-)
<Ursinha> henninge, mind to explain? :)
<henninge> I am just saying that I am exploring these things myself in varying depths
<Ursinha> henninge, oh, I see :)
<Ursinha> henninge, I think I found out the problem
<Ursinha> I have to have a Datetime object for it to let me use the formatter
<Ursinha> I may be wrong, but so far is my best guess
<henninge> Ursinha: I am pretty sure you need a Datetime object!
<henninge> ;-)
<henninge> Ursinha: what were you trying to use the formatter on?
<Ursinha> henninge, in what I thought that was a Datetime object, but realize it's not :)
<henninge> ah
<Ursinha> realizeD
<henninge> Ursinha: that's excusable ;-)
<Ursinha> henninge, hehe :)
<henninge> sinzui: ping
<sinzui> hi henninge
<henninge> hi sinzui
<henninge> sinzui: I am trying to triage bug 462891
<mup> Bug #462891: TraversalError on +export <oops> <Launchpad Translations:New> <https://launchpad.net/bugs/462891>
<henninge> sinzui: the oops happens when trying to access user/preferredemail/email
<sinzui> it is not visible
<sinzui> the app genuinely does not have permission to see it
<henninge> sinzui: does that happen when the user hides his/her email address?
<henninge> sinzui: although I tried that and it did not reproduce
<sinzui> henninge: Yes. In many mail/login usage, we use something link removeSecurityProxy(user.preferredemail).email
<sinzui> henninge: in this tales example. we simply do not have permission to show it. we should not
<henninge> sinzui: ok
<henninge> sinzui: is there an easy way to check in tales if we have that permission of not?
<sinzui> |nothing helps
<sinzui> it swallows exceptions
<henninge> sinzui: good idea, thanks
<sinzui> henninge: in tales you might also do
<sinzui>           <tal:visible-email
<sinzui>             condition="context/owner/preferredemail/required:launchpad.View">
<sinzui>             (<span tal:replace="context/owner/preferredemail/email" />)
<sinzui>           </tal:visible-email>
<henninge> sinzui: cool, thanks
<Ursinha> henninge, so, I think I'm missing something
<Ursinha> it seems the thing I'm returning is a datetime object
 * Ursinha just thought about make harness
<Ursinha> anyway, it seems the object is correct
<Ursinha> but I may be missing defining the property somewhere
<Ursinha> henninge, are you dying busy? :)
<henninge> Ursinha: sorry. Still trying to triage (reproduce) that traversal error
<henninge> Ursinha: define which property?
<Ursinha> the lastChangedDate that I want to add in that page
<henninge> Ursinha: can you push the branch and let me have a look at it?
<Ursinha> henninge, sure, but it's a mess, just a warning :)
 * henninge is used to messes
<BjornT_> mars, gary_poster: can any of you take a look at https://code.edge.launchpad.net/~bjornt/lazr-js/download-cache/+merge/14220? it's a quick fix, so that people won't have to know how to set up a download-cache
<gary_poster> BjornT_: on call, will check back
<mars> BjornT_, on a call for a bit
<mars> :)
<henninge> Ursinha: I cannot set the bug 462891 to triaged or set its importance. I have insufficient permissions for that.
<mup> Bug #462891: TraversalError on +export <oops> <Launchpad Translations:New> <https://launchpad.net/bugs/462891>
<henninge> Ursinha: ???
<Ursinha> henninge, how come?
<henninge> dunno
<henninge> The following errors were encountered:
<henninge>     * Only Bug Supervisors may change status to Triaged.
<Ursinha> henninge, try again
<sinzui> henninge: are you logged in
<Ursinha> eh eh
<henninge> sinzui: oh, I was fooled
<henninge> Ursinha: ^
<Ursinha> there should be something showing you're not logged in
<Ursinha> like a different color
<Ursinha> don't know
<Ursinha> henninge, was that the problem?
<henninge> I logged myself out in another tab
<henninge> but the bug page still looked like I was logged in ...
<Ursinha> henninge, oops :)
<Ursinha> henninge, happens to me sometimes, and my brain always fail to realize that could be this simple before panicking
<Ursinha> henninge, btw, thanks for triaging that bug
<henninge> Ursinha: np
<henninge> Ursinha: btw, where those the only two occurances of that particular OOPS? I have not been able to reproduce the bug and it would be interesting to know how often it occurs.
<henninge> s/where/were/
<Ursinha> henninge, we had about 40 a day since I've reported the bug
<henninge> oh!
<Ursinha> henninge, see here: https://devpad.canonical.com/~lpqateam/oops-summaries/lpnet-2009-10-29.html#exceptions
<Ursinha> for example
<henninge> Ursinha: oops.py should display the *account* name (which is unique), not the display name, which may appear gazillion times
<henninge> Ursinha: who maintains oops.py?
<Ursinha> henninge, matsubara-afk does
<henninge> Ursinha: is there a branch somewhere?
<Ursinha> henninge, yes, a moment, please
<mars> beuno, on a merge proposal, why is there a [Review] link and a [Claim Review] button?
<henninge> Ursinha: did you ever push that branch of yours?
<Ursinha> henninge, pushing, pushing
<Ursinha> :)
<beuno> mars, I wish I had a nice answer to that
<Ursinha> danilos is alive, I'll rip him them :P
<Ursinha> thanks henninge :)
<henninge> Ursinha: oh, that would be nice. I have guests tonight and wanted to cut it short. Is that ok with you?
<Ursinha> henninge, if it's fine with danilos than it's fine by me
<Ursinha> :)
<henninge> Ursinha: Oh, I thought you were already talking to him
<danilos> henninge, she was, I'll be happy to take on bug 462891 (we should always show that address, so removeSecurityProxy should do)
<mup> Bug #462891: TraversalError on +export <oops> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/462891>
<danilos> henninge, however, I am worried that this might happen when there's no preferred email as well
<henninge> danilos: I was not even trying to fix it, just to reproduce it
<danilos> henninge, as you can see, we are trying to traverse into None.email, so that's probably why it happens
<henninge> danilos, Ursinha: actually, I thought we were still talking about Ursinha's fmt:datetime problem ...
<danilos> henninge, heh, I am not sure about what we are talking about now :)
<Ursinha> danilos, I was asking henninge's help because I thought you weren't there, but about the bug I'm fixing, not the bug henninge is
<Ursinha> :)
<henninge> I am a bug?
<danilos> henninge, no, you just look like one
<rockstar> abentley, hi
<abentley> rockstar: Hi.
<rockstar> abentley, so, I'm trying to compare to jobs, using your implementation of __eq__ and it's complaining that context is a ForbiddenAttribute - is that by design?
<abentley> rockstar: No.
<rockstar> abentley, so it's okay that I add it to the interface to be accessible?
<abentley> rockstar: Doesn't forbidden mean it's not even on the underlying model object?
<rockstar> abentley, I don't think so.
<abentley> rockstar: I thought it was unauthorized if you don't have permission, forbidden if it doesn't exist.
 * rockstar checks
<rockstar> abentley, no, if if rSP an object, context is there.  In fact, the context should always be there in this case, since the context is the BranchJob
<danilos> sinzui, salgado-lunch, hey, just to confirm one thing: we can have logged in users into Launchpad with no preferred email set or no? (re bug 462891)
<mup> Bug #462891: TraversalError on +export <oops> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/462891>
<abentley> What is rSP?
<danilos> abentley, removeSecurityProxy?
<sinzui> danilos: I do not think that is possible.
<abentley> Okay, so expose it on the API.
<sinzui> danilos: I spoke to henninge about that
<rockstar> abentley, great, thanks.
<sinzui> danilos: User can set the email address to be hidden and we honour it
<danilos> sinzui, I know you did, but the thing is that we try to show email only to users themselves, and we should have privileges to do it
<danilos> sinzui, this page is rendered when user is requesting a download, and we display email address so they know where they'll get an email when the export is ready for download
<sinzui> danilos: have you tried the proven tales fraement to access a hidden email address
<danilos> sinzui, however, the traversal error tries to get email property on None, meaning that preferredemail is None, and not inaccessible
<sinzui> danilos: hmm
<danilos> sinzui, at least that's how I read it, maybe I need to look into preferredemail implementation
<sinzui> danilos: I still state that the login server should never let a user login without a preferred email address. I do not think anyone can login without creating a profile, and that step ensures a preferred email address.
<sinzui> danilos: do we know the user?
<danilos> sinzui, we've had a few bugs with person/account split; this is happening with more than one user, and we haven't seen this ever before
<sinzui> I see the user just came from login
<sinzui> I cannot believe we let that launchpad id exist
<danilos> sinzui, yeah, that was exactly my opinion when I saw the exclamation mark in the user ID, but that might be part of the displayname
<sinzui> danilos: i think we need to query the person table of staging to see if this a real user. I hope that id is person, not account
<sinzui> danilos: He has an email address https://edge.launchpad.net/~iqt
<danilos> sinzui, yeah, so I guess you were right about preferred email being returned as None when it's hidden (though, shouldn't it be hidden from us as well?)
<sinzui> danilos: You are special. You are a registry admin
<sinzui> danilos: That means your related software report is 100% crack
<danilos> sinzui, however, I see no person for this account
<danilos> sinzui, heh, right :)
<sinzui> danilos: This page does not who accounts. only person
<danilos> sinzui, I am looking at staging DB
<sinzui> danilos: there is only one page that can show an account and only a LOSA can see it
<sinzui> danilos: the account was created on the 27. maybe staging is old
<danilos> sinzui, staging DB seems outdated though
<danilos> sinzui, yep
<danilos> sinzui, just fyi, looking at the code, preferredemail has no restrictions by whether an address is hidden or not; and, all of these seem to have just logged in before that, and the account was created just a few minutes ago; I am guessing that launchpad_auth_master has some data that launchpad_main_master still doesn't
<henninge> danilos, Ursinha: Enjoy the weekend, I am off now!
<sinzui> danilos: thank makes sense. We had a number is similar oopses a few months ago where there as a 20 minute lag
<Ursinha> bye henninge :) you too
<henninge> danilos: you'd better get all well!
<danilos> henninge, cheers
<sinzui> Chex: ping
<Chex> sinzui: hello there
<sinzui> Chex: I want to test the product-release-finder on staging, but it just occurred to me that I do not know if the fix was pushed there this week
<Chex> sinzui: ok, was this a cowboy fix, or a revno release
<Chex> ?
 * sinzui loks for revno
<sinzui> no it did not arrive
<sinzui> Chex: do you know what is blocking staging from a code update?
<BjornT_> danilos: still there?
<BjornT_> danilos: those windmill errors are strange, i don't know what are causing them. you can merge lp:~bjornt/launchpad/bug-430702 to work around them
<Chex> sinzui: I am not sure, let me check
<danilos> BjornT_, ok, thanks
<danilos> BjornT_, also, thanks for looking into this, very much appreciated
<rockstar> abentley, what would it take to teach the scanner about bzr pipelines and the next and prev pipes?
<rockstar> abentley, (in the context of setting pre-req branches)
<abentley> rockstar: A database patch.
<abentley> rockstar: Why do you ask?
<sinzui> Chex: I think I am confused by the version of code on staging. I verified that other expected features were on staging, so...can you run cronscripts/product-release-finder.py on staging and paste/send the output?
<Chex> sinzui: jinx. I just finally verified that staging should be up-to-date within the past 24 hours or less
<Chex> sinzui: sure
<sinzui> fab
<rockstar> abentley, just curious, really.  I figure if we're bringing back prereq branches, it'd be really cool to teach the scanner about pipes soon after.
<abentley> rockstar: My plan is to teach pipelines about prereq branches first.
<rockstar> abentley, I aliased push to sync-pipeline for this very purpose a few weeks ago.
<rockstar> abentley, wait, that confuses me.  Pipelines knows which branch came before.
<abentley> Right, but right now, they can't propose a branch for merging.
<rockstar> abentley, ah, okay.  Yeah, we've talked about that.
<jml> leonardr, is the patch you just posted something of a pre-cursor to a trusted gtk workflow?
<abentley> So first pipelines is getting launchpad support.  Launchpad might get pipelines support later.
<leonardr> jml: yes, i spent this week landing the branch i mentioned to you a while ago
<jml> leonardr, cool.
<leonardr> i am supposed to talk to rickspencer3 about porting his gtk workflow to this engine but i can't find him. maybe he took the day off now that karmic is released
<rockstar> abentley, okay.  I've wanted to do something to the codehosting part of code, and mwhudson fixed the bug I was thinking I'd do, so I thought maybe I'd teach the scanner about pipelines.
<rockstar> abentley, just a thought in passing more than anything though.
<jml> leonardr, he's travelling back home
<leonardr> ah
<leonardr> i guess i'll talk to him on monday then
<abentley> rockstar: Well, the hardest part is getting a database patch through review.  After that, we can implement direct support for next_branch and prev_branch, or we can add the pipelines plugin to the codebase.
<jml> leonardr, trusted client for gtk would be very exciting.
<Chex> sinzui: ok, the script stopped output about 15 mins ago, should I kill it?
<sinzui> know. I think this can take some time since I expect it to be finding a lot of new downloads
<sinzui> s/know/no/
<sinzui> Chex: is it stuck on a specific url?
<Chex> sinzui: yes : 2009-10-30 17:39:03 INFO    Walking http://ftp.drupal.org
<Chex> 2009-10-30 17:39:03 INFO    Listing /files/projects/
<sinzui> that is big
<sinzui> Chex: Let this run. if this has not moved on in 30 minutes, I think we should kill it and restart
<intellectronica> have a lovely weekend and a great week four, everyone. see you in texas
<Chex> sinzui: oki doki.. still sitting there as of now.
<sinzui> :/
<mrevell> Night all. have great weekends!
<danilos> salgado, hi, do you know how closely are launchpad_auth_master and launchpad_main_master synced? are they the same DB or not?
<salgado> danilos, don't know
<danilos> salgado, but just to confirm, user/preferredemail should always be not None when accessed from code, right?
<salgado> if you mean LaunchBag.user, then yes
<flacoste> man, this django test runner sucks
<flacoste> gary_poster, barry: oopstools is testing a script using sys.executable
<flacoste> i'm buildoutifying the package, so that doesn't work
<flacoste> what is the best thing to do here
<flacoste> test through bin/the_buildoutified_script
<flacoste> or through bin/py
<flacoste> and what is the best way to get the path to bin/... from the test?
<barry> flacoste: bin/py bin/the_script probably.  iirc, we do something similar for mailman.  to get to bin i think we os.path.dirname from canonical.__file__
<gary_poster> flacoste: I prefer bin/the_buildoutified_script in general.  relying on bin/py is a bit of a hack in my mind, TBH.  It was convenient for us, but from scratch, it it is easy enough, I'd do the script.
<flacoste> gary_poster: i agree and that's what i started aiming for
<flacoste> now, what is the best way to get the path to bin?
<flacoste> should I assume tests are run from the buildout top-level directory
<flacoste> ?
<barry> gary_poster: did we use bin/py somescript because we would have had to hack mailman?
<flacoste> or should I use ../../../ from my test directory using the dirname(__file__) trick?
<barry> we really should have a utility that gives us the root of our tree, from which we can easily calculate bin or whatever
<gary_poster> flacoste: If you are working with something that has access to buildout variables, then you can use things like ${buildout:bin-directory} (IIRC; I would look for variables on buildout PyPI page)
<flacoste> gary_poster: that's just a regular doctest, so I don't think that's the case
<flacoste> unless we put those into a module
<gary_poster> flacoste: the z3c.recipe.filetemplate will let you do that.  Alternately, if you are doing a script, you can have code inserted (see how we do it in launchpad's buildout.cfg)
<flacoste> this is a django app
<flacoste> so it needs a settings file
<flacoste> which can contain a couple of global stuff
<flacoste> i think i'm going to generate that file using z3c.recipe.filetemplate
<gary_poster> you could generate that, sure.  I was talking about this: http://pypi.python.org/pypi/zc.recipe.egg#specifying-initialialization-code-and-arguments
<flacoste> and put a couple of well-known directory there
<gary_poster> that would work too, yeah
<flacoste> well, not really
<flacoste> because what i want is to find the bin path from the script
<flacoste> err, the test
<gary_poster> well, you could stuff info in someplace, like environ.
<flacoste> ah, you mean in the test wrapper for example
<flacoste> but that one is generated through djangorecipe
<flacoste> and it's a crappy test runner wrapper anyway
<gary_poster> :-) ok
<gary_poster> barry: "did we use bin/py somescript because we would have had to hack mailman?" yes, and because there was already precedent
<barry> gary_poster: cool
<flacoste> man, djangorecipe, what a "!&/)!*"
<flacoste> it doesn't put Django into the eggs
<flacoste> and it doesn't install django as an egg, but into build/parts
<flacoste> which means that scripts generated don't have it in their path
<flacoste> pff
<flacoste> extra_path to the rescue...
<flacoste> gary_poster: any idea why extra-paths isn't adding stuff to the generated script?
<flacoste> gary_poster: i think i know why
<gary_poster> flacoste: no.  It should.  (though, btw, on a a probably unreleated topic, I suggest that you do one of these three: (1) use a clean Python, (2) use a virtualenv clean Python (--no-site-packages), or (3) somehow use my zc.buildout and zc.recipe.egg and z3c.recipe.filetemplate internal releases.
<gary_poster> )
<flacoste> gary_poster: it was a confusion on my part
<flacoste> gary_poster: i had commented out the scripts stanza from the buildout parts
<flacoste> gary_poster: but the script was still generated through the interpreter section
<gary_poster> oic
<flacoste> gary_poster: which didn't have the extra-paths config
<gary_poster> you can combine the two fwiw
<flacoste> it's weird though that the interpreter section generates script
<flacoste> yeah
<flacoste> exactly
<flacoste> yeah! all tests are now passing!
<gary_poster> awesome :-)
<barry> sinzui: ping
<sinzui>  hi barry
<barry> sinzui: hi.  open up lib/canonical/launchpad/icing/style-3.0.css and search for registered
<sinzui> ouch
<barry> :)
<sinzui> so we still have a problem in that italics is wrong
<barry> sinzui: maybe it should have been "registering"?
<sinzui> oh, I see, it is correct when in an li
<sinzui> so the portlet is not putting the person's join time in an li
<barry> sinzui: maybe.  but if you look at https://launchpad.dev/firefox you'll see that the registration date is class="registering"
<barry> should would be nice if we had some more comments in our css :/
<sinzui> We certainly do no want to float the secondary information to the link
<sinzui> no, it would be nice if we took the time to rename our classes when we broaden their use
<sinzui> comments are needed when you write too many specific rules
<barry> well, perhaps, but the color and font-size seem more in line with registered
<sinzui> I want the text to look like other portlets. We have a markup/style problem because when that is next to other portlets it is clearly wrong. There is nothing special about it
<barry> the problem is that you really have no idea what ".registered" is used for, so you can't safely change it, so that leads to more fragmentation as you add more classes
<sinzui> barry: look at https://edge.launchpad.net/launchpad-registry
<sinzui> The grey normal text is what the user expects to see, he does not care how we do it. So when I saw italics, I knew something was wrong, my guess was certainly wrong about the cause
<sinzui> barry: can you spot the defect in this page by BTW?
<barry> sinzui: so, i agree that the italics is wrong, but /we/ definitely care how we fix it.  not much to say at the end of the cycle, but style-3.0.css is going to end up a mess this way
<sinzui> We can start cleaning it up by removing the old stylesheet rules
<barry> i don't understand why we have .registered and li . registered
<sinzui> one is italics and the other is not. The italics rule is from the contracted design. kiko later wanted italics removed, this one was not
<sinzui> .registering first used under the object's title. The design changed a lot in the 3.0 milestone.
<barry> sinzui: so, why wasn't font-style: italic; removed from .registered?
<kiko> ITALICS!!!
<sinzui> barry: dude get a grip. do I have to make you read the 3.0, and the post cleanup list?
<sinzui> barry: I'll be happy if you can triple your landings each month, but I do not want to break anyones spirt
<barry> sinzui: i am chilled out!  i'm trying to make sense of stuff in our tree that makes no sense
<sinzui> barry: label it tech-debt so that it makes the list of things we schedule work on.
<sinzui> except that we are not really scheduling tech-debt
<barry> sinzui: but we /are/ cleaning up ui, right?
<sinzui> yes, after oops
<sinzui> barry: our team closed about 20 ui issues this release, but we are a smaller team now, we still own 1/3 of all templates, and we are still expected to own blueprint and answers templates. The latter two completely missed their milestones
<barry> sinzui: cool.  this is definitely ui uncleanliness, but we need to avoid chaos too, or it's hopeless ;)
<sinzui> barry: We will never look as good as the other teams given that we cannot put enough people onto the issues
<flacoste> gary_poster: is there a list of buildout variables?
<gary_poster> flacoste: yes, in docs plus there is a command to see the defaults.  Looking...
<flacoste> gary_poster: i don't find it from the pypi page
<gary_poster> flacoste: not sure what you are looking for, but bin/buildout annotate will list default values
<flacoste> ok, thanks
<gary_poster> flacoste: also, http://pypi.python.org/pypi/zc.buildout#predefined-buildout-options
<Chex> sinzui: just a update, your script is still running, and creating output now
<sinzui> Chex: fab. I fixed a loop issue that prevented the script for download all the tarballs. I have a list of things I can check tonight if the script does not complete soon
<Chex> sinzui: ok, sounds good, just because us losas will be rolling up the sidewalk soon, wanted to see if you needed us to check status
<wgrant> How do week 4 landings work? I'll probably be wanting to land some new Soyuz enum values to make something easier to CP during 3.1.12 if Debian forces us to.
<lifeless> get RC approval
<lifeless> AIUI
<wgrant> sinzui: Can bug #464014 no make 3.1.10?
<mup> Bug #464014: DistributionSourcePackageView.active_series sorts versions as strings <Launchpad Registry:Triaged by edwin-grubbs> <https://launchpad.net/bugs/464014>
<wgrant> sinzui: It is going to become a problem very soon, and seems like an easy fix.
<sinzui> wgrant: no, we are lock down in a few minutes, the EC2 run time is 4 hours so once we crossed the threshold I push all the bugs back
<sinzui> wgrant: I am working in DSP's now and I feel the pain
<sinzui> Maybe I can convince someone that this is an RC able issue
<sinzui> wgrant: we will work have a branch ready next week, but I do not want to promise that it will be released
<wgrant> sinzui: cjwatson filed a dupe overnight.
<wgrant> Thanks.
<sinzui> wgrant: I think the fix is to use sorted_dotted_numbers
<sinzui> I assumed that was what we used to show the DSP
<wgrant> sinzui: Isn't the DS version a debversion?
<sinzui> either will do. We have be using the other because it handles cases of letters
<sinzui> not that LTS will ever appear in a version
 * wgrant checks how Distribution:+index does it.
<wgrant>         return sorted(ret, key=lambda a: Version(a.version), reverse=True)
<sinzui> Doom to fail
<wgrant> Why?
<sinzui> Doomed to fail
<sinzui> version is not a number
<wgrant> How does that matter?
<wgrant> debversion uses Debian ordering rules, which care about a lot more than just numbers.
<sinzui> sort is follows string rules, so 10 and 1 are much different
<sinzui> right, and debversion has a lot a tested/proven behaviour
<wgrant> Why would sort be using string rules on a Version object with a __cmp__?
<sinzui> The release/version is in point of fact a debversion, though the DB thinks it is something more loosely defined
<sinzui> Version is not an object
<wgrant> It is.
<wgrant> lp.archivepublisher.debversion.Version
<sinzui> I am mistaken then. In know the DB does not know that.
<wgrant> Right. The DB just enforces sane_version, which is something different. I wonder if the interface enforces debversion.
<sinzui> The IField does
<wgrant> Great.
<wgrant> Does Registry have sufficient engineering time this cycle to make that trivial change with a test?
<sinzui> I think so
<Ursinha> ahem.
<Ursinha> I hate writing tests.
#launchpad-dev 2009-10-31
 * wgrant boggles at dak_utils.py
<wgrant> Python. With semicolons.
<mwhudson> was a sysadmin involved in this code?
<wgrant> Given the filename, I imagine it has a rather strange history.
 * wgrant is going to have > 800 line RC branch :(
<lifeless> oooo
<wgrant> This is what happens when Debian sneaks v3 support in and starts uploading packages using it.
<maxb> When the codehosting tests fail complaining that bzr returned a non-zero exit status, do they log anything more informative anywhere?
<mwhudson> maxb: no, you have to hack the tests to find out what's going on :(
#launchpad-dev 2009-11-01
<maxb> Ah, that's better, SSHServerLayer tests are happier now I've uninstalled bzr-svn
 * maxb is starting to wonder if xx-resetpassword-of-sso-account.txt requires you to have canonical-identity-provider to pass :-/
<wgrant> maxb: Where does that live? I can't see it anywhere.
<maxb> ./lib/lp/registry/stories/foaf/xx-resetpassword-of-sso-account.txt
<wgrant> Oh.
<wgrant> Doesn't exist in db-devel.
<wgrant> WTF.
<wgrant> That seems like a somewhat concerning bug.
<wgrant> Oh, it's just brand new.
<wgrant> Nevermind.
<wgrant> But yes, that should live in c-i-p
<maxb> well, goody, that means ztk-2.5 is still alive and unbitrotten
<wgrant> Excellent.
<rockstar> sinzui, do you happen to be reviewing projects right now?
<sinzui> rockstar: I did a few hours ago
<rockstar> sinzui, ah, okay.  I just noticed the project count was dropping.
<rockstar> sinzui, I've been doing them in 20 project increments because it's so fucking tedious.
<sinzui> rockstar: okay. reviewed some commercial projects, those that has test in their descriptions,
<rockstar> sinzui, cool.  I'm trying to get that backlog out so that gmb doesn't have that much to do.
<sinzui> I was thinking of writing a script to send a license info email all "don't know" projects and mark them as reviewed
<sinzui> I see 1/3 of projects need a license and all that we can do is ask for one an mark it reviewed
<rockstar> sinzui, yeah, I was wondering if project review stuff was available through the API.  I'd be happy to script it out if it was.
<rockstar> Basically, does it have branding? Approve. Does it have a designated team as driver? Approve.  Does it have branches or bugs or translations? Approve.
<rockstar> That's what I've been doing.
<sinzui> I approved all those this morning
<sinzui> well all the obvious ones
<sinzui> After I disabled or approved all 'test' projects, I approved all projects that had code, or bugs, or was creates by someone with a lot of karma, or was created by a canonical employee
<sinzui> I cannot identify projects with branding or announcements :(
<mwhudson> why do i have 300 emails to look through :(
<sinzui> rockstar: I just tried a script that sent a variation of the license-info email to most the unreviewed undecided projects. 6 were skipped. I need to look at them to learn why
<sinzui> oh, I see. It was right to script them. They were dual/multi-license.
<thumper> mwhudson: when did you want to chat?
<thumper> mwhudson: I'm down to 80 odd emails from 300
<mwhudson> thumper: now ish is good i guess
<mwhudson> thumper: gimme a minute to refill my cup :-)
<thumper> mwhudson: ok
<thumper> mwhudson: I installed the gnome-desktop to try to debug some karmic issues
<thumper> mwhudson: and it installed pulse audio
<thumper> mwhudson: I need to work out how to kill it for kde
<mwhudson> thumper: aaron posted advice to warthogs on that i think
 * mwhudson wonders why all the staging cronscripts are barfing
<matgeek> Hi!
<matgeek> Just new to launchpad development. I am interested in working on modular backend plugins.  Git is really interesting to me, now that GNOME, the Linux kernel, and KDE are using it within their projects.  It is the DCVS that has buy-in with all the development communities.  Seamless integration is te name of the game.  It would be good to be able to work on a project with the VCS that the project uses, with out having to know about 
<lifeless> matgeek: your paragraph was too long and IRC cut it off at 'to know about'
<lifeless> matgeek: but have you seen bzr-git? it gives seamless git integration..
<matgeek> lifeless:  So I don't have to know any bzr commands to work with the launchpad web interface?  I can continue to use git as my development VCS without having to use bzr?
<lifeless> matgeek: other way around, you can use bzr without having to know any git commands :)
<thumper> matgeek: kinda, but Launchpad only supports bzr as a branch repository storage, and can mirror git repositories
<matgeek> lifeless:  That will not lead to buy-in from the development projects.  I use git myself on top of svn, and I don't want to have to know about bzr.  Is the launchpad - bzr backend based on a modular API?
<matgeek> lifeless: that is if I was from GNOME, project or working on kernel.
<matgeek> lifeless: hypothetical situation I know, but wanting to see if there is a case for making git another first class launchpad backend.
<lifeless> if you want to use git and push to launchpad, there is a git-bzr too, I believe
<matgeek> Mmmmm, I will have to explore further with my own Debian package, netscript-2.4
<lifeless> matgeek: the lp backend uses bzrlib, but bzrlib can talk to git, so in terms of /code/ it could be done. However we don't want the overhead of supporting many different VCS backends throughout the system
<lifeless> how to scale storage, ensure consistent backups, etc all are very simple at the moment
<matgeek> lifeless:  From my POV, it doesn't matter what backend launchpad uses, as long as it is seamless with the VCS a project is using.
<lifeless> I may not quite understand your criteria / rationale then
<matgeek> lifeless: OK, I am coming from the hypothetical POV of being a project developer interacting with launchpad.  It struck me that we want to get it to the point where everybody just wants to use it because it is so good.
<matgeek> lifeless:  Not knowing about bzr gets a big VCS elephant out of the room if you want a major project to actually start using launchpad itself.
<lifeless> matgeek: so, you don't need to use the VCS services of launchpad to use launchpad
<matgeek> lifeless: Yess, you just use the web frontend to point it at your project VCS of choice, the Web application uses bzr to get source trees etc, but the developers don't even have to type a bzr command, it is all transparent and just works!
<lifeless> matgeek: we have that today
<lifeless> https://help.launchpad.net/Code/Imports
<matgeek> lifeless: Ta! I will have a play around with my Debian package.  I am trying to come up with something useful for the launchpad project.  Where can I help?
<lifeless> thumper: ^
<lifeless> I'm not sure where the hotspots needing help are right now. thumper may.
<thumper> lifeless: on a call right now, so I can't focus here right now
<matgeek> thumper:  I will hang around till you have finished your call, if you want to chat with me.
 * thumper is off the call and reading scroll-back
<thumper> matgeek: what sort of help were you thinking about?
<thumper> matgeek: developing features? fixing bugs?
<thumper> matgeek: QA?
<matgeek> thumper: fixing a few bugs at first to get familiar with the code base, and then looking at some minor features before getting into the major stuff.
 * thumper nods
<thumper> ok
<thumper> matgeek: firstly get on the beta tester team if you aren't already
<thumper> matgeek: then get the code building locally
<thumper> matgeek: then look at bugs.edge.launchpad.net/launchpad-project
<thumper> matgeek: there should be some easy ones tagged trival or easy
<thumper> matgeek: the way we tend to work is:
<thumper> 1) identify what to fix
<thumper> 2) talk through the proposed fix with someone
<thumper> 3) fix it
<thumper> 4) submit the fix for merging, and go through the code review process
<matgeek> thumper:  Cool, I will need a day or two to get to the bug list, as I have to get some things done today.  Are people on this channel in the evening NZ time?
<thumper> matgeek: yes, where are you based?
<matgeek> Dunedin.
<thumper> matgeek: we should meet
<thumper> matgeek: I'm in dunners too
<thumper> matgeek: are you going to kiwipycon?
<matgeek> That would be good.  I am going to Earthlight this afternoon where I work on a Radius accounting project, using python.
<matgeek> When and where is kiwipycon? Could we meet today or tomorrow, or later this week?
<thumper> matgeek: you sent me an email the other week :)
<thumper> matgeek: if you hadn't heard, it is a little late as kiwipycon is this weekend in Christchurch, and sold out
<matgeek> Yes, I did email you.  I have applied for a couple of Ubuntu positions already.
<matgeek> thumper: About kiwipycon, a pity, but I can easily get accommodation up there with family.  It is only a bus ride or drive.  Are they full-house at the moment.
<thumper> I believe so.
<thumper> matgeek: you should also hang out in #nzpug
<thumper> matgeek: also get on the launchpad-dev and launchpad-users mailing lists
<matgeek> thumper: Will do.  Will subscribe tonight.
<thumper> ok
<thumper> I'm not usually on at NZ evening
<thumper> but the europeans are just starting
<thumper> and there are quite a few of them :)
<thumper> however,
<thumper> I do respond to email :) (eventually)
<jml> hello antipodes
<MTecknology> jml: hi
<MTecknology> if that includes me - long word
<jml> MTecknology, only if you are in Australia or New Zealand, normally
<MTecknology> jml: :(
<MTecknology> well... then I take back my hi
<mwhudson> jml: hello
<mwhudson> jml: a very euro-centric definition of antipodes!!
<jml> mwhudson, it's rather me-centric, actually
<RAOF> jml: Hello antipodes!
 * wgrant whinges a bit about the lack of APAC RMs and Soyuz people.
<thumper> wgrant: RM as in release manager?
<thumper> wgrant: you wanting a release-critical stamp?
<thumper> jml: hi
<jml> thumper, RAOF: hello
<thumper> I've finally booked my accomodation for kiwipycon
<wgrant> thumper: RM as in release manager, yes. It's not quite that simple -- I have ~1200 lines of branches that we need in the next couple of weeks for Debian compatibility. It would only need to be CPed onto a couple of the Soyuz machines, but there are safe enum and other DB changes that the appservers need as well.
<wgrant> The urgency of this only came to light two days ago, unfortunately.
<mwhudson> wgrant: well at least the RM is noodles so he should understand the issues more easily than some
<MTecknology> When is stu online? I want to ask him some questions.......
<thumper> MTecknology: it depends
<thumper> MTecknology: sometimes he is on in 3 to 4 hours, other times later
<MTecknology> thumper: ok - I was trying to figure out how he's managing the branches - but now that I'm looking at last revision times; it seems like there's little point in worrying about it
<wgrant> I think we are talking about different Stus here.
<MTecknology> thumper: heck - if they need the branches - then they can do bzr push and it'll all be happy
<MTecknology> stuart metcalfe
<wgrant> You mean Stuart Metcalfe.
<wgrant> thumper means stub.
<MTecknology> oh
<thumper> ah
<thumper> sorry
<thumper> misread stu as stub
<MTecknology> So when is this version of stu normally around?
<wgrant> His timezone suggests in about 9 hours.
<wgrant> But he's not around here.
<MTecknology> Where is he normally at?
<wgrant> I've never seen him on freenode, I don't think.
<wgrant> He lurks in the realms of ISD, I would imagine.
<MTecknology> oh - so I guess it's only when I ask him that he comes around
#launchpad-dev 2010-11-01
<wgrant> What's happening with vostok these days?
<LPCIBot> Project parallel-test build (12): STILL FAILING in 2 hr 11 min: https://hudson.wedontsleep.org/job/parallel-test/12/
<lifeless> mwhudson: hey
<lifeless> mwhudson: so in the person merge code
<lifeless> there is an XXX from you
<lifeless> about recipes and a function to write.
<lifeless> but no bug #
<mwhudson> ah oops
<lifeless> did you forget to put the bug # in, or is the bug mythical? It seems like an important before-live-todo, to me
<mwhudson> i don't remember filing a bug :/
<lifeless> perhaps you could ?
<mwhudson> this is about person merge not blowing up if you merge two accounts that have a recipe of the same name
<lifeless> I would assume so
<lifeless> also, I have sent a needs-to-be-digested mail to the list about code structure
<lifeless> I'm totally sure its impenetrable due to jetlag :)
<mwhudson> well, i'm pretty sure it would be to me :-)
<lifeless>     Hard / Soft  Page ID
<lifeless>      266 /    0  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>      162 /  260  BugTask:+index
<lifeless>      133 /  161  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>      121 /   43  Person:+commentedbugs
<lifeless>      107 /  282  Distribution:+bugs
<lifeless>      100 /    0  MailingListApplication:MailingListAPIView
<lifeless>       29 /   53  Archive:+packages
<lifeless>       26 / 1956  Archive:+index
<lifeless>       21 /   25  Milestone:+index
<lifeless>       15 /   85  POFile:+translate
<thumper> lifeless: did arsenic get new code?
<thumper> lifeless: MailingListApplication:MailingListAPIView lives there
<thumper> lifeless: and while I did only one method, it should have helped
<thumper> lifeless: see my note about getting the method used into the pageid for xmlrpc calls
<lifeless> no
<lifeless> its a SPOF at the moment
<lifeless> there is an RT to fix that
<lifeless> when thats done, it will be in the nodowntime set
<lifeless> thumper: I replied about the xmlrpc pageid
<lifeless> thumper: 'yes we can change it, there is a bug already I think
<lifeless> paraphrasing
<thumper> lifeless: did you? must of missed it
<lifeless> 11:58 < lifeless> whats arsenic
<lifeless> 11:59 < lifeless> yes, there is a bug already
<thumper> ah
<thumper> yes, see it now
 * thumper goes to make a coffee
<lifeless> wgrant: when are your exams done with ?
<wgrant> lifeless: 23rd.
<lifeless> of nov?
<wgrant> Ja.
<lifeless> kk
<lifeless> lots of prep time then ;)
<lifeless> gl
<wgrant> Thanks.
<thumper> wgrant: how many exams?
<wgrant> thumper: Just the three.
<thumper> not too bad then :)
<wgrant> Since the fourth subject was the project, which lacks an exam.
<wgrant> Indeed.
<wgrant> 10, 11, 23
<thumper> I remember my first year exams, Saturday am, Tuesday am, Tuesday pm, Thursday am, Thursday pm, Friday pm, then two weeks break before the last two
<wgrant> Weekend exams? Cruel.
<thumper> yeah
<spm> try having your 21st birthday in the middle of exam week. :-(
<thumper> spm: I got married the day before I turned 21 :)
<spm> thumper: so what I'm hearing, unhappy for pretty much the same reason??? :-P
<thumper> no comment
<spm> heh
<thumper> there is something wrong if the builds are taking 7 hours :(
<EdwinGrubbs> lifeless: sinzui mentioned that you did some research into the whole distro milestone +index timeout when there are lots of bugs. It seems that it is just spending a ton of time creating all the storm objects. Do you have any other insight?
 * thumper marvels at the amount of yak hair in blueprints
 * thumper has the clippers out
<lifeless> EdwinGrubbs: I setup eager loading
<lifeless> EdwinGrubbs: for both bugs and milestones
<lifeless> EdwinGrubbs: but I have only turned it on in one code path (to ensure I could actually deliver)
<lifeless> EdwinGrubbs: which pageid / bug are you looking at.
<EdwinGrubbs> lifeless: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1765ED1258
<EdwinGrubbs> bug 638924
<_mup_> Bug #638924: Milestone:+index timeouts with many bugs <pg83> <timeout> <Launchpad Registry:In Progress by edwin-grubbs> <https://launchpad.net/bugs/638924>
<wgrant> thumper: Was Blueprint touched significantly after its initial drop?
<thumper> wgrant: not much no
<thumper> wgrant: I now have a pipeline: ~/src/launchpad/blueprint-yak-shaving
<thumper> if anything was a test for not needing a review, this is
<wgrant> Heh.
<StevenK> thumper: So, I've done something surprising.
<StevenK> thumper: Robert Ancell was talking to me about specs not being exported over the API, so I made them be, in anger, on two plane flights.
<thumper> StevenK: you've moved to NZ?
<lifeless> thumper: its certainly a consideration
<lifeless> StevenK: have you moved the security checks?
<StevenK> lifeless: There's security checks?
<lifeless> StevenK: james_w has a branch that exports specs over the API; its blocked on that change.
<StevenK> Rargh, didn't know that
<thumper> StevenK: yeah, too many checks done in the browser code
<StevenK> Speaking of, I should charge my laptop
<thumper> I've started shaving blueprints...
<thumper> but it will take some time to see the shape of things
<thumper> under all that hair :)
<thumper> StevenK: but just think of the fun you had :)
<StevenK> Hah
<StevenK> It wasn't that bad, I'm just curious why I could only get close, but not all the way
<lifeless> StevenK: old LP code did security checks at the browser layer
<lifeless> StevenK: blueprints is old LP code
<StevenK> (I can grab a specifications object from lplib, and iterate over it, but it returns nothing)
<thumper> heh
<thumper> StevenK: it has an entry, but nothing exported
<thumper> StevenK: mildly shit
<StevenK> thumper, lifeless: To be perfectly honest, I did it for shits and giggles, while I was bored, so I'm happy enough to toss the work away.
<lifeless> StevenK: no worries
<lifeless> StevenK: I would love to see it happen
<StevenK> thumper: Oh, I fixed that. I can post a diff if you have tissues to stem the bleeding
<thumper> StevenK: I've more a problem with who sets what
<thumper> :)
<StevenK> I was also curious why I couldn't export Choice() elements where the vocab is in quotes, but can if it's a real object.
<StevenK> Right, now I get what james_w is blocked on.
<lifeless> Edwin-zzz: ah, yes
<lifeless> Edwin-zzz: so thats high python time
<thumper> StevenK: it'll have to do with the method trying to work out what is valid
<StevenK> lifeless: When did you get home, by the way?
<lifeless> Edwin-zzz: I'd get a profile trace of that on [qa]staging. Will mail you.
<StevenK> thumper: Hm, I hadn't considered that, but it would make sense
<StevenK> Starting to seriously consider a nanna-nap, and it's only 3:16pm
<lifeless> StevenK: 10:20am arrival
<StevenK> lifeless: At home, or the airport?
<lifeless> jtv: so, we're experimenting in testtools and a few other projects with
<lifeless> StevenK: airport, home @11
<StevenK> lifeless: Landed at 0825, home at 1140
<wgrant> StevenK: The empty collection is probably because of the lack of a launchpad.View adapter for ISpecification.
<lifeless> jtv: wishlist = fallback, medium = 'something we thought worked doesn't', critical = 'blocks project doing a release'
<jtv> lifeless: the awkward marriage between the "problem reporting" and "task tracking" faces of bug trackers has bothered me for a long time, so I'm all on boardâ¦ if you've been thinking about this, I'd be happy to introduce you to kirit for constructive discussion.
<lifeless> jtv: I had a great discussion with persia - theres a commons problem too.
<lifeless> bugs are global scope; work prioritisation is local (FSVO local) scope
<StevenK> wgrant: Ahh. However, I'm now not going to talk about it, since james_w has already done work on it.
<stub> spm: Do you know much about Puppet, and is it something we should be looking at for Launchpad configuration, scheduling or anything?
<lifeless> stub: I know a bit :) - no, no and no.
<lifeless> stub: the sysadmins will be using it to manage a growing set of services; for what we're doing/developing we should effectively ignore it, for now.
<stub> wondering if it was useful for readonly switch, or LPCONFIG type stuff.
<lifeless> it may be able to replace the deployment scripts eventually.
<lifeless> dunno about the readonly angle; offhand anyway.
<lifeless> perhaps setting a feature flag would be best for readonly.
<lifeless> it would propogate to the slave, be found there and be locked on until someone pokes more directly. [/wildarse thought from jetlagged person]
<stub> 'cept feature flags are in the database, and changing to read-only mode switches to a database that was broken out of replication before the switch was set ;)
<lifeless> stub: is that ordering changable ?
<stub> No - breaking the slave out of replication involves removing the replication triggers on every table and subsequent locking.
<jtv> And I guess it complicates the case where you go to read-only because you lose the master.
<stub> Anyway - we can't really ask the database if we are in read-only mode, because we need to know if we are in read-only mode to decide what database to talk to.
<lifeless> gotcha
<lifeless> the ini file you added is probably simplest for now
<StevenK> wgrant: And ohloh is still counting lines of source code, after 1 restart and, what, 3 weeks?
<spm> stub: not vast amounts no; but will be in Syd next week doing a course on same.
<wgrant> StevenK: At least it's on step 3 now!
<StevenK> But it has been for a week
<stub> I think we should stop using Zope's email delivery, instead storing emails in the database and having a cronscript or daemon that does the delivery. Pretty trivial, easier to improve later with things like delivery failure, and sidesteps the SMTP ZCML gumph.
<wgrant> StevenK: I think it's even slower than Launchpad git imports used to be.
<lifeless> stub: I'd like to know whats up before we reimplement exim/postfix - if you see what I mean.
<StevenK> stub: If you suggest writing EmailJob, I Will Find You
<stub> Nothing so complex :)
<stub> lifeless: At the moment appservers spool emails to the SMTP server during request. If we fix queued delivery, the appservers spool emails to the SMTP server in a separate thread. But why have the appservers do it at all?
<lifeless> stub: the expansion from intent->mails I'm delighted to move out of webapp request handling
<lifeless> stub: I don't really care if thats a Job or a thread.
<jtv> spm: just realized the humorous chance I missed Sydneyâ¦  to go around telling everyone how great their city and country are and, if this is just a local capital, how incredible Canberra must be then.
<lifeless> stub: mail sending is sometimes massively slower than expected, and we should fix that.
<lifeless> stub: at the moment the appservers spool *after* the commit *before the request ends*
<lifeless> which is a little odd :)
<stub> lifeless: Yes. In request delivery is slow, queued delivery is broken. We can fix queued delivery upstream, or move to a different model which I think is better in the long run.
<stub> I'm suggesting still allowing the appservers to assemble the actual emails, but stuff them into the database rather than do the delivery themselves.
<stub> Probably the same amount of work as fixing queued delivery.
<lifeless> stub: perf wise I'd expect it to be slower
<lifeless> stub: local mail spools /should/ be millisecond fast to accept things
<lifeless> anyhow
<lifeless> halt()
<wgrant> Anything sending more than a couple of emails from a request is probably a bug.
<stub> pickle vs. store.execute()
<StevenK> wgrant: Such as, say, updates to bugs?
<wgrant> StevenK: They don't.
<wgrant> StevenK: They use an external process.
<wgrant> StevenK: But still manage to be mortifyingly slow.
<StevenK> Pity, I was hoping to be ironic
<stub> wgrant: Agreed. But we can't control how long the SMTP server we are talking to will take before the email is handed off, unless we get a dedicated one we can turn off stuff like DNS checks etc.
<wgrant> stub: True.
<lifeless> stub: we have a dedicated SMTP server
<lifeless> stub: *per appserver*
<stub> We talking to localhost now? Ok.
<lifeless> yeah
<stub> thumper: Just reading meeting notes. Thanks for reducing the mailman query counts! Its a big burden on the db.
<spm> jtv: heh
<spm> jtv: it's .... different. big country town with the amenities (to some degree) of a city.
<jtv> You're saying it's a bureaucratic amusement park at the end of a fat fibre-optic cable?
<spm> close
<spm> "You're saying it's a bureaucratic amusement park at the end"
<jtv> It just became a whole lot less impressive.
<wgrant> It's Canberra. It is far less impressive than even your least impressive impression.
<spm> some of the more ... rigid bureaucratic side has changed, but it's still very much a town that revolves around federal govt. and that has implications right thru the entire place. unf, usually in painful ways.
<spm> flip side. I'm a 20min drive from the city centre; yet from our back deck can see paddocks with sheep/horses across the valley. We get Kangaroos < 50metres from our front door (I have pics!). That's pretty special.
<mwhudson> haven't we fixed this: http://launchpadlibrarian.net/58406579/vcs-imports-wordpress-trunk.log at least twice already?
<mwhudson> spm: ^^
<spm> ugh
<spm> actually - I wonder - I came across this doing some simple bzr ops last week - don't recall which server offhand.
<spm> mwhudson: which importd what that?
<spm> was* that
<mwhudson> spm: i'm a twenty minute _walk_ from wellington cbd and not much further from rural bliss than you :-p
<spm> :-)
<mwhudson> spm: well, https://code.launchpad.net/~vcs-imports/wordpress/trunk
<mwhudson> spm: ahh, seems there's more than one kind of failure :(
<mwhudson> spm: seems it's galapagos that's doing the nowhoami thing
<spm> ta
<lifeless> spm: I'm 20m from another entire city ;)
<thumper> stub: np, I might tackle the methods one by one, but I want the xmlrpc server less loaded
<mwhudson> https://launchpad.net/~mwhudson/+karma -> "Michael Hudson-Doyle's karma has expired. "Total karma: 20838"
<mwhudson> um?
<henninge> Does anybody have an idea how we can get out of testfix again?
<henninge> mwhudson: could that mean "Some of you karma has expired?"
<mwhudson> mayhap
<wgrant> It can also mean that the karma cache is being updated.
<wgrant> Sometimes a couple of the karma totals will vanish for a few minutes while that happens.
<wgrant> Or, in this case, all of them.
<mwhudson> could be
<wgrant> Everyone is similarly afflicted.
<wgrant> Which is rather suspicious indeed.
<jelmer> It looks like more windmill test failures.
 * jelmer waves to mwhudson, wgrant
<wgrant> Morning jelmer.
<mrevell> Hello
<henninge> Hi mrevell!
<mrevell> hello henninge
<henninge> mrevell: I just thought that you might not yet be aware of the fact that we moved the 10.11 release back by a week.
<wgrant> The guy in #launchpad looks like he could do with some commercial subscription help.
<henninge> mrevell: sorry for having kept you out of the loop here.
<mrevell> henninge, Thanks for letting me know now. So we're going for the 17th?
<henninge> wgrant: thanks, will talk to him.
<mrevell> wgrant, thanks
<mrevell> heh
<henninge> mrevell: yes, I just emailed the list.
<mrevell> Ah, I see your mail henninge
<henninge> mrevell: you go and talk ;)
<wgrant> What's this December "Bug Jam" thing? Does the following cycle not start until January?
<mrevell> henninge, I saw a discussion where I think the release was going to 00.00 UTC. Is that still the case?
<henninge> mrevell: no, I am proposing 10:00 UTC
<mrevell> wgrant, The idea was that rather than having a mini-cycle in December, we'd spend a couple of weeks doing nothing other than close bugs (not necessarily fix...)
<wgrant> Heh.
<wgrant> OK.
<henninge> mrevell: I had proposed 22 UTC for the 10th because I already knew that Tom would not be around.
<mrevell> wgrant, So, we could just have one week of bug jam and a full-sized December cycle
<mrevell> henninge, Ah, I see.
<henninge> mrevell: but with more LOSAs not there, we scratched the 10th completely and since Tom is there on the 17th, we can do it at 10:00
<mrevell> henninge, Are we pretty confident of the date and time? I mean, can I announce it?
<henninge> mrevell: it's not finalized yet, no
<henninge> oop, public channel ...
<henninge> ;-)
<henninge> but I don't expect any more problems.
<mrevell> heh, that's not a problem :)
<wgrant> jelmer: Hi.
<mrevell> Hey, I'm running make on a new branch on a clean Maverick install ... I'm getting a permission error when I run make (clean, schema and run so far)
<mrevell> Any ideas?
<jelmer> wgrant: Hi
<StevenK> mrevell: Can you pastebin the trace?
<wgrant> jelmer: I've addressed your comment on https://code.edge.launchpad.net/~wgrant/launchpad/destroy-distroseries-lucilleconfig/+merge/38648. Anything else?
<jelmer> wgrant: Yes, I need to land your branch. :-) I'll send it off to ec2 today.
<mrevell> StevenK, Thanks, Here it is: http://pastebin.ubuntu.com/523776/ .... that's just the last few lines ... do you need the full thing?
<wgrant> jelmer: Thanks!
<wgrant> mrevell: I normally fix that with "sudo rm -r /var/tmp/bazaar.launchpad.dev".
<wgrant> I think Apache creates it or something.
<mrevell> Oh, simple as that eh? Thanks wgrant, I'll give that a go.
<mrevell> whoop, that worked for make clean ... thanks wgrant and StevenK
<deryck> Morning, all.
 * lifeless yawns
<lifeless> hi jml
<jml> lifeless: hi
<lifeless> flacoste: jml: since we're all awake, perhaps we should continue our long running voice discussion
<jml> lifeless: on another call right now.
<lifeless> kk
<deryck> gmb or allenap, could one of you spare time for an easy review?
<allenap> deryck: Sure.
<deryck> allenap, thanks!  https://code.edge.launchpad.net/~deryck/launchpad/allow-reporter-fixed-released-unsetting-664096/+merge/39754
<jml> lifeless: maybe we could have a call later today
<lifeless> sure
<jml> sinzui: hello
<sinzui> hi jml
<jml> sinzui: when do you think we're supposed to have our call
<sinzui> next week -1 hour?
<jml> sinzui: I'm happy with that. Britain changed timezones during my flight home today
<sinzui> okay
<jml> sinzui: so I'm not sure when anything is
<sinzui> jml: you are my only meeting on Mondays. You can fit me in at your convenience
<jml> sinzui: cool.
<jml> sinzui: did you want to chat today?
<sinzui> sure
<sinzui> now?
<jml> yeah.
<lifeless> Ursula: hi
<lifeless> https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html is empty?
<lifeless> Ursula: bah, browser fail. ignore that question.
<lifeless> Ursula: however
<lifeless> Ursula: please do look - Revision 11801 is bad: possible blocker. is reported 4 or 5 times :)
<jcsackett> gary_poster: hey, an LP user has asked me for any info on this issue in #launchpad https://answers.launchpad.net/moin-openid/+question/110123; you were looking at it at one point, any notions about it or who i might point the user to for more info?
<lifeless>  Bug 664060 needs QA
<_mup_> Bug #664060: bug supervisor should be able to configure bugtracker for projects <qa-needstesting> <Launchpad Bugs:Fix Committed by brian-murray> <https://launchpad.net/bugs/664060>
<jml> lifeless: so, now that edge is almost gone, are you going to tackle the other subdomains next?
<lifeless> jml: thats a more complex question, and would benefit from specific discussion about it.
<lifeless> jml: the complexities for it are:
<lifeless>  - are they part of the UI/ease of use story?
<lifeless>  - impact on performance (more domains == more SSL handshakes, but also potentially more parallelism [though we don't do this :P so its a bit of a no-brainer today]
<lifeless>  - impact on vostok
<lifeless> jml: I have no immediate plans to do anything to/for/or about the bugs etc domains
<lifeless> jml: but perhaps you meant some other set of subdomains
<lifeless> https://bugs.launchpad.net/software-center-agent/+bug/627608 needs qa
<_mup_> Bug #627608: Got a 401 on a fresh purchase <qa-needstesting> <Software Center Agent:Fix Released> <Soyuz:Fix Committed by michael.nelson> <software-center (Ubuntu):Fix Released> <https://launchpad.net/bugs/627608>
<lifeless> jml: We did ~ 2200 commits to trunk in the last year.
<lifeless> jml: our velocity in terms of commits-per-month for the last two months is about the same as for the last 12.
<lifeless> jml: 20 hours per commit
<lifeless> per dev
<lifeless> (given a nominal 2000 hours/year work period)
<jml> lifeless: interesting
<jml> lifeless: because anecdotally I'd say committing to trunk is getting harder.
<lifeless> jml: there sure is a lot of friction we need to fix.
<jml> lifeless: fwiw, I meant the bugs etc subdomains.
<jml> lifeless: but I do realize it's a big question.
<lifeless> 11824-7230 = 4594 over the last two years
<lifeless> jml: I think the team size has grown a bit over two years?
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | Launchpad Development Channel | Week 1 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> lifeless: I'm not convinced it has. Anyway, it's easy enough to see how the number of distinct committers has changed
<lifeless> jml: indeed, I'm doing very crude stats just now - log | less :)
<jml> lifeless: well, easy-ish. bzr has a way to go before querying logs is as easy as SQL
<jml> lifeless: reckon it's worthwhile to file bugs on the remaining parallel testing work?
 * jml gone, back later
<lifeless> jml: not really; there is a parallel test job; looking at that will show whats broke
<lifeless> statik: shall we skip our call today? we caught up late last week ..
<statik> lifeless: that sounds good to me
<SpamapS> are there known bugs in reading archives of private mailing lists that a lp user is subscribed to?
<SpamapS> I'm unable to pass the openid "Continue" button.
<lifeless> ugh
<lifeless> http://python.mirocommunity.org/video/1186/djangocon-2009-django-is-obsol is quite interesting
<SpamapS> Is this a follow up to iamcal's "Why I Hate Django" ? ;)
<lifeless> no
<lifeless> its about evolution
<SpamapS> in a way, so was cal's talk
<lifeless> well, it might be then :P
<SpamapS> but, I'll refrain from commenting further until I've watched the presentation
<mars> lifeless, would be nice if there was a transcript there - it would be nice to just skim the facts of the argument
<SpamapS> hah.. onions spoil all kinds of apps
<SpamapS> Some might say "minimize window" is an onion.
<lifeless> SpamapS: you'll like the 'templates will finally die' line
<SpamapS> I stopped at about 15 minutes.. I get his point.. rethink.. multi-threaded.. etc. ;)
<lifeless> SpamapS: he switches themes
<SpamapS> lifeless: even livejournal figured out that you can do the various bits of the template asynchronously server-side using gearman.
<SpamapS> lifeless: oh very well.. starting back up.. ;)
<lifeless> lol
<lifeless> SpamapS: only if you're interested
<SpamapS> I'm very interested
<SpamapS> One thing I am starting to think about.. is why hadoop is the only one anybody mentions
<SpamapS> map/reduce isn't that hard!
<lifeless> he mentioned cassandra dinomite etc
<SpamapS> Yeah I mean the map/reduce frameworks.
<SpamapS> lifeless: so the last time I heard this argument, it was for xslt
<SpamapS> CSS is obviously a much simpler way to do presentation than xslt..
<lifeless> yeah
<lifeless> flacoste: ping
<lifeless> awol for a bit
<cr3> leonardr: hi there, is there a way to define a method in lazr that returns void? I have a write_operation method which seems to cause a subsequent GET on the server because launchpadlib seems to do self.resource.lp_refresh() if the return code is 200 and the method is post
<cr3> leonardr: I tried to redefine the method in question as a factory operation but then it expects the method to return the object created rather than void, which results in a ComponentLookupError for None
<leonardr> cr3: https://bugs.launchpad.net/launchpadlib/+bug/249950
<_mup_> Bug #249950: Stop refreshing after every POST named operation <launchpadlib :Triaged> <https://launchpad.net/bugs/249950>
<leonardr> no :|
<cr3> leonardr: thanks for the reference, good to know
<leonardr> cr3, hopefully it's just an annoyance?
<cr3> leonardr: a performance annoyance where each call results in two calls
<cr3> leonardr: a crappy workaround might be to convert the method from a write_operation to a read_operation, but I'm not sure how much data I could encode in the url
<leonardr> cr3: if it helps, i'm about to start on a big web service project and i could fix this problem as a side effect
<cr3> leonardr: in the meanwhile, how acceptable would it be for me to define my own decorator called export_void_operation which would derive export_write_operation and setStatus(201) in the overridden call method?
<leonardr> cr3: that's not good. you don't want to send 201 at all--you'd be lying
<leonardr> how bad a performance problem is this?
<leonardr> if it's absolutely awful i can spend some time pretty soon figuring out the best way to fix it
 * mars \o/ devel is Green once again!
<cr3> leonardr: it's not that awful, it's just that I know myself well enough that I'll probably forget to fix this later. I can take a note and wait, no worries
<leonardr> cr3: what we want is something like export_void_operation + sending a Cache-Control header or something that says 'you just made the object dirty, you need to refresh'
<cr3> leonardr: if I want to return an object which contains a reference to another object in the same call, can that be done?
<leonardr> cr3: you mean send both objects at one?
<leonardr> that's not possible
<leonardr> unless you can return a list of objects--that might work, but i  doubt it
<cr3> leonardr: yeah, both at once to spare the subsequent get on the referenced object
<leonardr> cr3: there's a general feature planned that will let a client tell the server it wants certain objects expanded, but it's not planned for a long time--i need to rationalize the web service design first
<cr3> leonardr: maybe it's a lesson for me to create even coarser grained objects
<leonardr> cr3: could be, if this is a really bad problem. would it help if this data was cached on the client side, or is even getting it once a big performance problem?
<lifeless> leonardr: whats the big web service project?
<leonardr> lifeless: refactoring the named operations into a few 'families' (search/filter operations and so on) so that each resource type in the web service doesn't have 5-20 miscellaneous and unstandardized operations dangling from it
<leonardr> make it easier to learn and use
<leonardr> i'm working with jml on it, but the desktop thing has intruded for several months
<jcsackett> lifeless: per the db-patch rules, i believe i need a review from you on this: https://code.edge.launchpad.net/~jcsackett/launchpad/migrate-official-bool-data-627631/+merge/39583
<lifeless> jcsackett: you need one from stub to assign the patch id, to land it
<jcsackett> lifeless: i know, but the docs say i need one from the architect as well. that would be you. :-)
<lifeless> yes
<lifeless> but not to land
<lifeless> the docs also say that :)
<jcsackett> dig. i was just pointing out the mp to you. my understanding is that stub is out, so i'll have to chase him down later.
<jcsackett> there's already a request for him on the MP as well.
<lifeless> there is a 'request a review' buttton on the mp - my preference is for you to use that for me too :). Thanks for letting me know.
<jcsackett> lifeless: i used that for you too, but it was while you were at uds, so this was sort of pushing it back into awareness. :-)
<wallyworld_> rockstar: abentley: thumper: standup?
<EdwinGrubbs> matsubara-afk, Ursula: ping
<wgrant> jelmer: Did you get around to EC2ing that branch?
<lifeless> sinzui: hey
<thumper> lifeless: so what is the process then for landing something without a review?
<thumper> lifeless: I have my very boring enum change for blueprints
<lifeless> https://dev.launchpad.net/PolicyAndProcess/OptionalReviews
<lifeless> Activities
<lifeless> Submit the branch to create an MP (our toolchains can look at this and it provides a location for a post landing review if the branch has that done to it). Self review with review type 'unreviewed'. Land via the normal landing process.
<lifeless> we may find this needs tuning
<lifeless> e.g. review type none/code
<lifeless> jml: have you crashed?
<wallyworld_> thumper: one of my branches got merged into db-devel rather than devel. i need to do stuff that depends on it. do i have any option other than to work on a db-devel branch?
<lifeless> wallyworld_: land it again.
<lifeless> wallyworld_: to the right place.
<thumper> wallyworld_: did you branch from db-devel?
<wallyworld_> ok
<lifeless> wallyworld_: just be sure that you don't suck all of db-devel in.
<wallyworld_> no i branched originally from devel
<thumper> wallyworld_: or more importantly, did you branch or merge from db-devel since the last rollout?
<thumper> wallyworld_: in which case, land it on devel as well
<lifeless> thumper: 'release' - not rollout ;)
<thumper> wallyworld_: bzr's merge detection will handle it
<wallyworld_> with that branch i have only merged into it from devel
<wallyworld_> not sure why it landed in db-devel
<lifeless> the default is bong.
<lifeless> mailing
<thumper> lifeless: the default is due to branch stacking
<thumper> lifeless: and that is all
<wallyworld_> thumper: i think we are still in textfix mode so i'll have to wait to land it again
<lifeless> thumper: its the series, we can change it right ?
<thumper> lifeless: if we had an easy way to say push and stack on db-devel when needed (that is tested and simple to use) then we could switch relatively easily
<lifeless> thumper: I realise we need to do a repo fetch
<lifeless> thumper: we can switch trivially
<thumper> lifeless: devel isn't stacked
<lifeless> thumper: right, but branches stacked on db-devel may need db-devel only revisions.
<thumper> lifeless: right
<lifeless> so its devel.fetch(db_devel), then change the default series.
<lifeless> 4 lines of python + a losa
<thumper> we don't need any code changes
<lifeless> right
<lifeless> just the fetch
<thumper> no
<thumper> not even the fetch
<thumper> branches are stacked on aboslute paths
<thumper> not shortcuts
<thumper> it is more the overhead of new db-devel branches
<thumper> but if we don't care...
 * thumper shrugs
<lifeless> thumper: devel branches are the common case
<lifeless> thumper: they pay massively today ;)
<lifeless> thumper: please do it
<lifeless> thumper: and let launchpad-dev know
 * thumper nods
<lifeless> (unless you think its wrong to change it)
<wgrant> Nothing depends on the lp:launchpad alias?
<lifeless> flacoste: ppr needs 'max', I think :)
<lifeless> losa ping
<spm> yo
<lifeless> https://pastebin.canonical.com/39248/
<lifeless> can you add that into
<lifeless> https://edge.launchpad.net/+feature-rules
<spm> sure. what does it all do? timeouts against specific pages?
<lifeless> yes
<lifeless> as per https://dev.launchpad.net/LEP/FeatureFlags
<spm> just never seen that style before
<lifeless> if you look at the lpnet timeout report
<lifeless> these are the current hard timeouts
<wgrant> lifeless: Exempting all the main timeouts and slashing the limit again?
<lifeless> I looked at the PPR to get a feel for how things are going
<lifeless> wgrant: raising slightly for stuff hurt by 8.4
<spm> haha
<wgrant> Bah.
<spm> lifeless: OOPS-1766ED2295
<lifeless> wgrant: will lower the timeout again once we've caught up with that
<lifeless> spm: rotlf
<lifeless> spm: that was near-instant, right ?
<spm> yah, purty much
<lifeless> spm: what was the traceback ?
<spm> Retry: duplicate key value violates unique constraint "feature_flag_unique_priority_per_flag"<br /> <br />
<lifeless> ah
<lifeless> I had 4 duplicated
<spm> yes. given the timeouts...
<lifeless> change the second ' 4 ' to ' 5 ' and so own down the page.
<lifeless> s/own/on/
<spm> hrm/ I think that's applied. it doesnt' exactly let you know that it has done anything.
<lifeless> please do file bugs on this interface
<spm> yeah, has.
<lifeless> its your primary knob for controlling LP in real time.
<wgrant> Hm,.
<wgrant> Do we really need such a massive freeze now that most stuff should be QA'd immediately?
<lifeless> its bare bones now, but it doesn't have to stay this way.
<wgrant> A week to QA db-devel changes seems excessive.
<lifeless> wgrant: the thing I proposed has a much smaller critical section
<lifeless> wgrant: we're still a week behind in QA right now.
<lifeless> wgrant: until we've caught up, we haven't caught up.
<wgrant> True.
<lifeless> spm: thanks, in 60 minutes we should see a drop to near-zero hard timeouts on lpstats
<wgrant> lifeless: Don't worry, Soyuz will fix that :)
<wgrant> sinzui: Why does DistributionMirror live in Registry?
<lifeless> aren't you meant to be stufying ?
<wgrant> Silence.
#launchpad-dev 2010-11-02
<lifeless> wgrant: https://bugs.edge.launchpad.net/soyuz/+bug/669717
<_mup_> Bug #669717: archive:+index timeout <dba> <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/669717>
<lifeless> wgrant: just to keep you on your toes ;)
<lifeless> holy f*ck:     5156  OOPS-1765XMLP299  MailingListApplication:MailingListAPIView
<lifeless> thats a query count
<thumper> lifeless: yeah
<thumper> lifeless: I think that is the one I fixed
 * thumper is a sad bunny
<thumper> it seems that the storm insert query has non-deterministic column ordering
<lifeless> s/insert //
<thumper> so two subsequent test runs with LP_DEBUG_SQL_EXTRA clash all the time
<thumper> lifeless: do you know a fix?
<lifeless> do you mean column or row ?
<lifeless> also, ECONTEXT
<lifeless> thumper: https://bugs.edge.launchpad.net/launchpad-registry/+bug/666580 - I've put the pageid in there - it helps me a lot.
<_mup_> Bug #666580: MailingListApplication:MailingListAPIView (getMessageDispositions ) mailing list xmlrpc api call makes excessive queries <mailing-lists> <qa-ok> <timeout> <Launchpad Registry:Fix Committed by thumper> <https://launchpad.net/bugs/666580>
<wgrant> lifeless: %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s to you too.
<wgrant> But wow.
<lifeless> thumper: did you mail matt?
<mars> lifeless, for shortening the release timeline, I was wondering if we could lock monday, QA tuesday, unlock wednesday, assuming staging updated ok.  Timeline goes from one week to three days.
<mars> if people finish their QA on Friday, then you can lock and unlock on monday
<lifeless> mars: I think there is not enough slack
<lifeless> mars: the critical issue is 'what if the db conversion is a problem' - it takes many hours to do a respin of that test
<mars> yeah, but if it went OK, and the QA is up-to-date, then you are looking at a much shorter release
<mars> "if the stars are all aligned just so" - thankfully there are only three to worry about :)
<lifeless> right
<lifeless> but we plan for how to handle failure
<lifeless> not how to handle success ;)
<lifeless> mars: I think we still need to start friday
<lifeless> but we could certainly unlock earlier
<lifeless> mars: I thought my proposal permitted that
<mars> absolutely, it does
<wgrant> lifeless: Can't you just freeze db-devel on Friday?
<mars> with that extra detail, "Must start Friday, can unlock ASAP", then you can rewrite the release docs
<wgrant> lifeless: devel doesn't have to be frozen to do DB restore testing.
<lifeless> wgrant: we don't need to freeze db-devel at all
<lifeless> wgrant: its not db restore testing
<lifeless> wgrant: its upgrade-script application
<wgrant> That's what I meant, sorry.
<lifeless> wgrant: which we can start doing on qastaging
<lifeless> (and should)
<wgrant> Hmm.
<wgrant> What's the point?
<lifeless> convergence
<mars> simplicity
<lifeless> ponies!
<mars> I would say 'speed', but ponies, well...
<lifeless> wgrant: qastaging is sitting there without any db patches, so we don't need to do a complete restore to test the upgrade
<lifeless> wgrant: which is /much/ faster
<wgrant> lifeless: True.
<lifeless> wgrant: and it updates its code every 30 minutes
<wgrant> But this still means we have a week where production doesn't update.
<lifeless> so if we have to fine tune the release, we can do so more rapidly.
<lifeless> wgrant: perhaps we should change buildbot to only merge QA-ok revisions from stable to db-devel.
<lifeless> wgrant: the week with prod not updating saddens me, but not as much as a month without updates
<wgrant> Right, but it doesn't mean we shouldn't try to think of a way to minimise that interval.
<lifeless> agreed
<wgrant> Particularly now that there are no CPs.
<wgrant> I don't see a huge benefit in doing the DB upgrade testing on qastaging rather than staging. And doing it on qastaging means we have to make devel undeployable several days earlier than it would be otherwise.
<lifeless> mmm
<lifeless> wgrant: I disagree
<wgrant> Oh?
<lifeless> wgrant: friday avo, sat, sun are all undeployable anyway
<wgrant> True.
<lifeless> I think if we get 3 clean deploys in a row without needing the extra time, we could move it up to monday.
<thumper> lifeless: yes I emailed matt
<lifeless> also if we have a completely clean development-dbstable report leading up to the report
<wgrant> I guess that's a good idea.
<StevenK> wgrant: You still have misgivings?
<lifeless> StevenK: I think he's looking for a larger win
<lifeless> which is good
<wgrant> StevenK: I'd like there to be as little freeze of landings and production deployments as possible. So I will have misgivings forever, unless we reduce both to zero :)
<wgrant> But this is better than what we have now, so it'll do.
<thumper> lifeless: I've got a problem with feature flags
<thumper> lifeless: it is breaking tests
<thumper> lifeless: and I think I know why
<lifeless> ok
<lifeless> 'sup?
<thumper> lifeless: there is code that checks the timeout values and uses a feature flag
<thumper> this causes a query, and prior to that a flush
<thumper> which can cause partially constructed object to be written to the db
<lifeless> say what?
<thumper> causing integrity constraint violations
<lifeless> that check is done before any domain code
<lifeless> its in publication
<thumper>  File "/home/tim/src/launchpad/incremental-diff-job/lib/lp/services/features/rulesource.py", line 90, in getAllRulesAsTuples
<thumper>     .find(FeatureFlag)
<thumper>   File "/home/tim/src/launchpad/devel/eggs/storm-0.18-py2.6-linux-x86_64.egg/storm/store.py", line 210, in find
<thumper>     self.flush()
<thumper> those are the bits causing test explosions
<thumper> perhaps for general publication
<thumper> but not for tests
<lifeless> so, there are several things we could do
<lifeless> firstly, outside of a publication stack or similar context, timeouts are meaningless
<lifeless> so we could remove that
<lifeless> secondly, the feature values are cached (including absence)
<lifeless> so evaluating it (e.g. by get_request_timeout()) will cache that outside your code
<thumper> how does that interact with the feature flag context manager for tests?
<lifeless> thirdly test requests have a Null scope provider - they could sensible /also/ install a Null rules provider
<lifeless> which wouldn't call into the db
<thumper> I don't grok that last oen
<thumper> one
<lifeless> see LaunchpadTestRequest.__init__
<thumper> lifeless: it has a NullFeatureController
<lifeless> thumper: ok
<lifeless> thumper: so in requests using LTR, no db access should happen at all.
<thumper> but it is
<lifeless> thumper: are you using maris' context manager?
<thumper> the code I'm looking at is yes
<thumper>         self.useContext(feature_flags())
<thumper>         set_feature_flag(...)
<thumper> although ...
<thumper> it is the set_feature_flag code that triggers the flush and the bomb...
 * thumper is now confused
<thumper> because the diff should be fully constructed there...
<lifeless> hmm, that would be nicer as a Fixture I think.
<lifeless> something for another time.
 * thumper relocates
<MTecknology> I'm gonna miss edge
<lifeless> why?
<MTecknology> lifeless: the thing about recipes that I spammed in the other channel
 * thumper is very confused
<thumper>         self.useContext(feature_flags())
<thumper>         set_feature_flag(u'code.incremental_diffs.enabled', u'enabled')
<thumper>         self.assertTrue(getFeatureFlag('code.incremental_diffs.enabled'))
<thumper> fails
<thumper> in a test
<thumper> but only if run after a different test
<MTecknology> thumper: probably has something to do with that "self." thing you have going on. Anytimg I rely on myself for something I crash.
<StevenK> Oh dear
<thumper> MTecknology: :-)
<lifeless> so this may be related to the cross-test thing gmb/deryck mentioned
<thumper> lifeless: almost certainly
<wallyworld_> lifeless: email to devel is on my todo list - i just wanted to get the mp set up first
<lifeless> wallyworld_: sure
<lifeless> wallyworld_: Belts and braces ;)
<wallyworld_> lifeless: save me looking it up, what's the losa mail list? i didn't know there was one
<thumper> lifeless: I'm pretty sure it is a cache invalidation problem :)
<lifeless> thumper: at what layer? we shouldn't have the same featurecontroller reference should we?
<lifeless> thumper: I'd guess at the request<->thread-local lifetimes being different.
<lifeless> thumper: perhaps a function to ensure when one is altered, the other is too
 * thumper is poking some more
<wallyworld_> lifeless: don't worry, found it
<lifeless> wallyworld_: oh sorry, yes.
<wallyworld_> lifeless: np. i was just letting you know. it wasn't mean to be snarky :-)
<lifeless> I know :)
<lifeless> man
<lifeless> I hope noone is pissed about all these bug closes :)
<MTecknology> lifeless: closed as in fixed- or closed as in won't fix?
<lifeless> both
<thumper> heh
<thumper> set_feature_flag has a store.flush
<thumper> no...
<thumper> set_feature_flag adds a feature to the db
<thumper> when the feature is being flushed
<thumper> it is doing a db query
<thumper> which queries the feature flags
<thumper> which sets the _rules cache
<thumper> which means the newly set feature isn't in the cache
<rockstar> thumper, yeah, deryck and I fought with that last week.
<rockstar> It caused much head scratching.
<thumper> I'm not yet sure why sometimes it hits this and other times it doesn't
 * thumper thinks it is a flush ordering problem
<thumper> I think it is easily solved though
<StevenK> transaction.commit() ?
<lifeless> no
<StevenK> thumper: No fair sending wallyworld_ my way with a 1,500 line MP
<StevenK> I know where you live ...
<thumper> StevenK: I didn't know it was that big
<thumper> sorry
<wallyworld_> StevenK: sorry :-( most of it is unit tests :-)
<wallyworld_> thumper: looks like devel is still in test fix mode? my pqm-submit was rejected again for that reason
<StevenK> wallyworld_: I'm working through it, but the current way to get MPs > 800 lines reviewed is to 1. Not, or 2. Bribe a reviewer
<StevenK> wallyworld_: Consult Ye Olde Buildbot
<StevenK> We shouldn't be
<wallyworld_> StevenK: what can thumper offer you by way of inducement? :-)
<StevenK> It isn't thumper's MP ... *cough* *hint*
<thumper> lifeless: http://pastebin.ubuntu.com/524222/
<wallyworld_> StevenK: yeah, but he was the one who *made* me throw you the hospital pass :-)
<lifeless> thumper: does that work for you?
<StevenK> It's all about choice
<wallyworld_> said it would be good for your soul :-)
<thumper> lifeless: yep
<lifeless> thumper: something smells here
<lifeless> thumper: I think features.per_thread.features is stale perhaps ?
<lifeless> thumper: if so, thats critical to fix.
<thumper> lifeless: yes... in that the features has cached the rules before we add one
<StevenK> wallyworld_: If thumper said that with a straight face, then I take the comment about his poker face back
<lifeless> thumper: if its cached the rules from another test, or even another *request*, then there is an isolation bug that this qualifies as a workaround for.
<wallyworld_> StevenK: i can offer you paul's eternal gratitutde as he *really* wants to get merge queues done before he takes his sabatical from the code team
<thumper> lifeless: more of a problem with tests than real life
<thumper> lifeless: let me plug in headphones
<StevenK> wallyworld_: I thought I already had that
<thumper> lifeless: then skype may help here
<lifeless> ok
<wallyworld_> StevenK: hmmm. i'm running out of possible bribes.
<StevenK> wallyworld_: Hah, serves you right :-P
<lifeless> thumper: so - two lines
<lifeless> da.set_permit_timeout_from_features(False)
<lifeless> ...
<lifeless> da.set_permit_timeout_from_features(True)
<lifeless> webapp/testing/helpers.py
<thumper> lifeless: http://pastebin.ubuntu.com/524230/
<lifeless> AdapterIsolator = FunctionFixture(set_request_started, lambda x:clear_request_started())
<lifeless> in setUp
<thumper> http://pastebin.ubuntu.com/524231/
<lifeless> self.useFixture(AdapterIsolator)
<lifeless> --- lib/lp/testing/__init__.py  2010-10-26 15:47:24 +0000
<lifeless> +++ lib/lp/testing/__init__.py  2010-11-02 03:26:01 +0000
<lifeless> @@ -495,6 +495,9 @@
<lifeless>          self.oopses = []
<lifeless>          self.useFixture(ZopeEventHandlerFixture(self._recordOops))
<lifeless>          self.addCleanup(self.attachOopses)
<lifeless> +        from canonical.launchpad.webapp import adapter
<lifeless> +        self.useFixture(fixtures.FunctionFixture(adapter.set_request_started,
<lifeless> +            lambda _:clear_request_started())
<lifeless> thumper: set_permit_timeout_from_features(False)
<lifeless> thats all
<lifeless> 11:33 < lifeless> https://dev.launchpad.net/PolicyAndProcess/OptionalReviews
<lifeless> 11:33 < lifeless> Activities
<lifeless> 11:33 < lifeless> Submit the branch to create an MP (our toolchains can look at this and it provides a location for a post landing review if the branch has that done to it). Self review with review type 'unreviewed'. Land via the normal
<lifeless>                   landing process.
<thumper> https://code.launchpad.net/~thumper/launchpad/fix-features/+merge/39819
<thumper> lifeless: bzr lp-land checks to make sure you don't review your own code :)
<thumper> lifeless: do we land it rs=?
<lifeless> thumper: change lp-land ?
<thumper> maybe it asked a slave
 * thumper checks
<StevenK> lp-land doesn't deal with rs=
<StevenK> Or MP-less lands
<thumper> the code seems to indicate it should be fine :(
<thumper> but something weird is happening
<thumper> me gives up and pqm-submits
<lifeless> thumper: please file a bug on foundations
<thumper> lifeless: on the thread local leaking?
<lifeless> on lp-land not accepting your MP
<lifeless> thumper: I bet its the review type though
<lifeless> thumper: try changing the type to nothing/code
<thumper> ah
<thumper> yeah, you are right
<thumper> which is fucked
 * thumper leaves to cook
<StevenK> At 5pm?
<thumper> StevenK: I have kids
<lifeless> StevenK: hey
<lifeless> you know soyuz stuff
<lifeless> help: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<StevenK> I will look after shopping
<thumper> heh
<lifeless> thanks
<LPCIBot> Project devel build (173): STILL FAILING in 3 hr 52 min: https://hudson.wedontsleep.org/job/devel/173/
<lifeless> wow Archive:EntryResource:getBuildSummariesForSourceIds is slow
<lifeless> still timing out at 20 seconds
<lifeless> 1/2 sec per SELECT * FROM ((SELECT BinaryPackageBuild.distro_arch_series, BinaryPackageBuild.id, BinaryPackageBuild.package_build,
<wgrant> lifeless: Do we have graphs of page performance vs time?
<wgrant> lifeless: ie. do we know if it's a pg 8.4 regression, or a BFJ refactor regression, or is it just generally crap?
<lifeless> its crap now
<lifeless> the api wasn't timing out before, so  8.4 regression
<lifeless> I think
<lifeless> no we don't have charts per-query
<lifeless> wgrant: yeah - https://bugs.edge.launchpad.net/soyuz/+bug/662523
<_mup_> Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/662523>
<jtv> Hi henninge
<henninge> Hey jtv! ;)
<henninge> Feeling better?
<jtv> Grr why don't we have checksums on the Ubuntu iso download page?
<thumper> interesting failure on launchpad prod branch in buildbot - http://pastebin.ubuntu.com/524300/
<wgrant> jtv: I suspect that you don't want to know.
<wgrant> (the design team probably said they were user-hostile, or something)
<wgrant> But they're easy enough to find on the mirrors.
<lifeless> night all
<wgrant> Night lifeless.
<jtv> wgrant: well I do want to bloody know.  I now have supposedly identical ISOs with different checksums and it should be easy to figure out which, if any, is right.
<jtv> Night lifeless
<wgrant> jtv: http://releases.ubuntu.com/10.10/MD5SUMS
<jtv> Thanks
<wgrant> Nice security.py speedup, btw.
<wgrant> It's, er, quite effective.
<jtv> Thanks.
<jtv> I didn't bother with the remaining largest time waster inside the script, since it only took 4 seconds.
<jtv> Script startup however does seem to take quite a while.
<wgrant> Most of the remaining 'make schema' time seems to come from build.
<jtv> So that's our next target.
<StevenK> The WADL isn't very fast either
<wgrant> Then lifeless can delete ZCML.
<wgrant> And we can build a tree in seconds!
<wgrant> StevenK: That's in build.
<wgrant> compile has slow buildout.
<wgrant> build has compile and WADL.
<StevenK> wgrant: I spent the trip from ORD to LAX exporting blueprints, I got to know very well how long it takes to generate.
<wgrant> StevenK: I tried to profile the WADL generator.
<wgrant> But it took too long.
<StevenK> Haha
 * StevenK attempts to come up with a clean joke, fails
<StevenK> Hmm. Apparently a horse race happened today
<wgrant> I've been deliberately avoiding finding out who won.
<StevenK> I didn't even realise until Sarah loaded smh.com.au while I was walking past
<wgrant> Hah.
<wgrant> It's hard not to realise down here; it's a public holiday.
<StevenK> Yeah, I knew that
<wgrant> Why a sporting event gets a public holiday I will never know.
<jpds> Is it the cricket?
<wgrant> Less boring than that.
<StevenK> Paint drying?
<wgrant> Indeed.
<nigelb> Something less boring than cricket? Interesting.
<jtv> I wouldn't go _that_ farâ¦
<wgrant> jtv: Do you suggest that something is more boring than cricket?
<LPCIBot> Project devel build (174): STILL FAILING in 3 hr 27 min: https://hudson.wedontsleep.org/job/devel/174/
<LPCIBot> Launchpad Patch Queue Manager: [r=jtv][ui=none][bug=667554] Don't send email for work in progress
<LPCIBot> merge proposals when requesting reviews or modifying the proposal.
<jtv> wgrant: read what nigelb said more closely!
<wgrant> Ah ha..
<jtv> My dad was dragged into a cricket match once.  Didn't even find it boring, to his surprise.
<jtv> A decade later he happened to open a cricket mag (at the dentist's or something) and guess what?  They were still talking about that famous exciting match.
<nigelb> lol
<nigelb> jtv: heh, you caught that :D
<jtv> nigelb: red-handed
<nigelb> heh
<nigelb> I watched IPL to any extent only for one season.  Lost interest.
<jtv> IPL = Initial Program Load?  Still talking about profiling?
<nigelb> No, was talking about criket :)
<jtv> What's IPL stand for?
<nigelb> Indian Premier League.  The 'big thing' in 20-20.  meh.
<jtv> nigelb: thank you for TLA #23662
<_mup_> Bug #23662: [network-admin] 'connection settings' should not be greyed out when not active (in network settings) <network-admin> <gnome-system-tools (Ubuntu):Invalid by desktop-bugs> <https://launchpad.net/bugs/23662>
<jtv> No mup, not that.
<jtv> http://xs4all.nl/~jtv/gtf/
<nigelb> jtv: lol
<jtv> nigelb: you can now carry the GTF Contributor Program (GCP) logo on your website or home page.
<nigelb> \o/
<nigelb> Well, this day did bring one achievement.
<jtv> Quite.
<StevenK> jtv: steven@liquified:~$ host hugeurl.wiggy.net
<StevenK> Host hugeurl.wiggy.net not found: 3(NXDOMAIN)
<StevenK> :-(
<nigelb> liquified, no wonder.
<jtv> I guess wichert must have shut it down.  It was about a decade ago.
<jtv> The idea was that tinyurl etc. are nice, but small URLs don't look impressive.
<jtv> So he created a URL stretcher.
<StevenK> Indeed
<StevenK> Yes, and Wichert is the kind of guy to do it. :-)
<jtv> One of the encoding schemes available was: look for three-letter combinations that are in the GTF, and expand them.
<jtv> Haven't seen him in ages.  You?
<StevenK> Neither
<jtv> :(
<StevenK> Not for 5 years or so
<jtv> One wonders how he is.
<StevenK> He fell into the black hole that previous DPLs get sucked into
<jtv> Does that apply to former colleagues and university friends?
<StevenK> jtv: Ah, you were at XS with him?
<jtv> No, cistron
<jtv> And the first FOSDEMâthen still called OSDEM.
<jtv> I arrived late, having escaped from a lady's bedroom window that morning in a different country.
<StevenK> Now there's something that sounds like an interesting story
<jtv> I think it was our mutual friend Ray who started the vicious and baseless rumour that I had escaped from a lady's bathroom window.
<jtv> If you ever hear that version, don't believe it!
<StevenK> Perhaps it was her ensuite window
<jtv> Her wha?
<StevenK> An ensuite is a small bathroom directly accessible from a bedroom
<jtv> Ah.
<jtv> No, it was definitely bedroom.
<jtv> But thanks for teaching me that word.
<StevenK> Heh
<nigelb> jtv: Wait, do we get to hear more of that exploit?
<jtv> nigelb: what do _you_ think?
<wgrant> We must.
<nigelb> We demand.
<jtv> Get me drunk and we'll talk.
<nigelb> dammit, if I had the money, I'd fly to Australia just to get you drunk ;)
<jtv> If you don't have the money to fly to Australia, what makes you think you have the money to get me drunk?
<nigelb> Good point.
<nigelb> I was planning on getting you drunk enough for you to hand me you purse and hards :p
<jtv> My secret is safe for now.
<nigelb> Good plan? ;D
<wgrant> Until January.
<nigelb> What's in Jan? Epic?
<wgrant> Ja.
<nigelb> Too bad its a closed event :(
<jtv> Face it.  You're not looking for an open event.  You're looking for an open bar.
<jtv> (See "money" above)
<nigelb> heh
<nigelb> True, that.
<nigelb> jtv: Love your homepage. "Nor do I speak for my employer; he is quite old enough to speak for himself."
<jtv> That's served me well for a fair number of employers.  :)
<jpds> What if they're a she?
<jtv> As now, I suppose, they are.
<jtv> I didn't know about that particular bit of the English language at the time of writing.  Will rectify.
<adeuring> good morning
<henninge> Everybody: land you branches! Testfix coming up again ... :-(
<StevenK> I've been seeing the same failures in Hudson
<mrevell> Hello
<deryck> Morning, all.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build (111): FIXED in 3 hr 55 min: https://hudson.wedontsleep.org/job/db-devel/111/
<deryck> gmb, hi.  Did you see thumper's email about test isolation failure and his fix to set_feature_flag?
<gmb> deryck: Yes; I've already merged my branch but I'll try splitting the tests up again and see if it works now.
<deryck> gmb, ok, cool.  I expect that will fix the problem we were seeing, too.  If using the testing helpers.  mars fixture would need to flush too.
<gmb> Right.
 * deryck feels like getting on to the kids.... "you need to flush every time!"
<LPCIBot> Project devel build (175): STILL FAILING in 3 hr 31 min: https://hudson.wedontsleep.org/job/devel/175/
<LPCIBot> * Launchpad Patch Queue Manager: [r=bac][ui=none][bug=586461, 634326,
<LPCIBot> 634646] Stop generating invalid memcache keys and allow more cache
<LPCIBot> sharing.
<LPCIBot> * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=668194] Fix IHasTranslationImports in the
<LPCIBot> API.
<LPCIBot> * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=668194] Split off some interfaces stuff
<LPCIBot> into separate files.
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][ui=none][no-qa] Move specification enums into
<LPCIBot> lp.blueprints.enums.
<gmb> Hurrah for ambiguous messages from bots.
<cjohnston> deryck: bug 483027 seems to be working now.. I was able to subscribe someone else and myself
<_mup_> Bug #483027: Display problem when adding subscribers to a bug report <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/483027>
<deryck> cjohnston, awesome.  Thanks for the confirmation.
<cjohnston> np
 * gmb just self-reviewed a branch, feels naughty.
<cjohnston> gotta do what you gotta do sometimes
<jml> mrevell: hello
<mrevell> hello juml
<mrevell> oh, jml
<jml> mrevell: :)
<mrevell> :)
<jml> mrevell: some guy at UDS came up to me and said, "How do you think of all those clever gadgets?"
<mrevell> Seriously? Excellent :)
<jml> yeah :)
<jml> mrevell: anyway, I was wondering if I could do anything to help with the user testing process discussion?
<mrevell> jml, Do you have time for a call this afternoon?
<jml> mrevell: yeah, I could have a call either after or before the standup
<mrevell> jml, After would be great. Thanks. I think we got pretty close to a general rule: if you need a LEP, you are likely to benefit from user testing.
<deryck> allenap, hi.  On my review yesterday, you suggested I use inTeam, but I don't follow why, since bug.owner can never be a team?
<allenap> deryck: It was a general comment that it's not always safe to just compare IDs, and inTeam() does the right thing. (I assume bug.owner is the reporter, not the assignee; if the latter then it can be a team.)
<deryck> allenap, right, it's the reporter.  So I'd prefer to keep it on ID, even if inTeam covers that case, just so it's clear.  Cool?
<allenap> deryck: Okay.
<deryck> allenap, ok, thanks.
<allenap> deryck: I didn't mean - I rarely mean - for my review comments to be prescriptive.
<deryck> allenap, oh, I didn't take it that way.  Just wanted to chat more to make sure I wasn't misunderstanding you. :)
<jml> I have a soyuz question
<jml> I'd like to open up "o" and "p" series for Ubuntu. I'm told that this ought to work, but everyone I've asked has sounded unsure
<jml> a) how could we test that this works? or
<jml> b) if we go ahead and do it, are we capable of easily undoing the mistake?
<jelmer> jml: I also think that it *should* work. You just want to have them in the database I assume, not allow uploads yet?
<jml> jelmer: exactly
<jelmer> jml: We should be able to test it on dogfood or staging I guess. It might be nice to check with bigjools or Colin that there isn't anything I'm overseeing, I haven't been involved in adding distroseries before so there might be some subtle bug I'm not aware of.
<jml> jelmer: bigjools has said much the same: "it should work, let's test on dogfood"
<jml> jelmer: and cjwatson, iirc, didn't have any reservations
<jelmer> jml: Ah, great. Just checking. :-)
<jml> jelmer: could you please test that on dogfood?
<jelmer> jml: Sure.
<jml> jelmer: thanks. it might be a good idea to also get someone platformish to try to break it. maybe cjwatson?
<allenap> deryck: I just realised that the bug importer could create a new bug with a team as owner.
<deryck> allenap, ah, good catch.  I'll add a test then with team ownership and fix up the code.
<allenap> deryck: The team would need to already exist in Launchpad and have an email address corresponding to an email address in the bug import XML.
<jelmer> jml: I'll check with him once I've got it set up.
<jml> jelmer: sweet. thanks.
<allenap> So, if I have a branch that I don't think needs review, how do I indicate that?
<allenap> Do I just approve the mp myself and land it?
<jml> allenap: that seems sensible
<allenap> jml: Okay, thanks :)
<jml> which makes me think of a thing
<jelmer> jml: btw, this would require knowing the distroseries codenames beforehand. would that possible?
<jml> jelmer: does it really? can we not rename them later?
<deryck> allenap, jml -- I thought there was something about [r=unreviewed] to track these.
<deryck> not sure our tools support that yet, obviously.
<jml> deryck: :9
<jml> (that was a frown fail)
<deryck> heh
<deryck> I thought it was a district 9 grimace.
 * allenap tried to do that with his mouth.
<jml> deryck: I don't know. is there a wiki page for the 'speriment?
<deryck> i think so....
 * deryck is looking
<jelmer> jml: I don't think we've ever tried that. I guess files in PPA's with the wrong names won't be an issue as we won't enable these distroseries yet, I wonder if there's other places where we have the distroseries name hardcoded.
<jml> allenap: btw, a thing that we do with testtools reviews is we have a 24hr timeout on review requests
<deryck> https://dev.launchpad.net/PolicyAndProcess/OptionalReviews
<allenap> jml: So, after 24h you can land without a review?
<jml> allenap: yeah
<jml> allenap: one thing I'll be doing for my own landings under this process is asking for a review and having a 5-10m timeout
<jml> deryck: ta
<allenap> jml: If I want to land sucky code I should submit on 24th or 31st December?
<jml> allenap: 24hr timeout on testtools for folk w/ commit access :)
<jml> allenap: if you don't have commit access, sucks to be you
<allenap> jml: Ah, and I've probably just made it harder to get commit access ;)
<jml> heh heh
<allenap> Ah, bin/ec2 land does not consider "unreviewed" reviews as reviews and won't land.
<jml> patch patch patch :)
<jelmer> jml: Can you think of any other things that have a distroseries name hardcoded at the moment? The packaging branches uses something based on the database id, right?
<jml> jelmer: hmm, yeah, but the stacking URL is hardcoded in the .bzr dir
<jml> jelmer: perhaps though we forbid setting the official branch for distroseries that are in FUTURE?
<jml> iirc the check is tied to "can you upload"?
<jelmer> jml: aren't they generally stacked on the project trunk though?
<jml> jelmer: packaging branches are stacked on lp:ubuntu/foo
<jelmer> jml: ah, ok
<jelmer> jml: yeah, forbidding the setting of official branches for non-enabled distroseries makes sense.
<jml> jelmer: but I'm not 100% sure that's the case. would need to test.
<jml> (or read the code)
<lifeless> garh
<allenap> jml: We could say that if you review your own mp then it's unreviewed, then we don't need to patch the tools, and there's no loss of information. Right now both bzr-pqm and lp need to be patched :-/
<jml> lifeless: your sleep cycle is still off?
<allenap> lifeless: ^ too.
<lifeless> jml: I just woke up.
<lifeless> jml: I feel a little tired but 'awake'.
<jml> allenap: yeah, that makes sense to me. I guess it would be nice to patch ./utilities/ec2 to accept self-reviews tagged 'unreviewed'
<lifeless> allenap: As long as I can reliably find the MP's that were self-reviewed, for the metrics angle.
<allenap> jml: Okay. I'll file a bug for that (which I might fix myself anyway).
<allenap> lifeless: Cool, I'm make sure we can do that. Is via the API enough?
<allenap> Actually, can merge proposals be searched for via the web UI anyway?
<allenap> Only by status it seems.
<lifeless> allenap: API is fine.
<allenap> Cool.
<LPCIBot> Project db-devel build (112): SUCCESS in 3 hr 31 min: https://hudson.wedontsleep.org/job/db-devel/112/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11820,
<LPCIBot> 11821, 11822, 11823, 11824 included.
<LPCIBot> Project devel build (176): STILL FAILING in 3 hr 29 min: https://hudson.wedontsleep.org/job/devel/176/
<LPCIBot> * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=656823] Various bits and pieces around
<LPCIBot> PersonSubscriptionsView.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml,
<LPCIBot> stevenk][ui=none][bug=666660] Lower log level for 'Translations ...
<LPCIBot> match n existing translations.' to INFO to avoid it being
<LPCIBot> turned into an OOPS.
<LPCIBot> * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=638920] Only try to show links to private
<LPCIBot> branches to authorized users ond on a product series'
<LPCIBot> translations page.
<LPCIBot> * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=664566,
<LPCIBot> 664569] BugNotificationLevel now has a more readable set of
<LPCIBot> descriptions for Bug:+subscribe. BugNotificationLevel.NOTHING
<LPCIBot> is no longer accepted when one is attempting to subscribe to a
<LPCIBot> bug through the web UI.
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][ui=none][no-qa] Explicitly flush the store when adding
<LPCIBot> feature flags, and stop our tests from checking features for timeouts.
<dobey> that is a noisy bot
<flacoste> lifeless: if you are bored, can you review https://code.launchpad.net/~flacoste/launchpad/ppr-constant-memory/+merge/39666 ?
<dobey> so with the API, it appears that branch.landing_targets doesn't contain all the proposals, if branch.status == 'Merged'; is that correct?
<dobey> gary_poster: ^^ you wanted to talk to me about reviews API anyway, right? :)
<lifeless> flacoste: bored. Hah! have you seen my job description :) - sit around bored ain't on it :P
<gary_poster> dobey: I'm afraid I don't know the answer to that. :-/
<flacoste> lifeless: well, it might help you sleep :-)
<dobey> gary_poster: hrmm, ok. i'm having some issues with landing branches with prerequisites becuase of it :(
<gary_poster> dobey: I'd be reading the code along with you.  Someone from the code team would probably be much more efficient help, if they are around.  (That said, if you really think a particular bit of code would be valuable for me to stare at, go ahead and send me there)
<lifeless> gary_poster: hi
<lifeless> gary_poster: got a few minutes to catch up? I particularly want to talk edge with you
<gary_poster> lifeless, sure
<gary_poster> now on Skype and mumble both
<dobey> gary_poster: i have no idea where in lp the code is. http://pastebin.ubuntu.com/524489/ is the code in tarmac that's checking landing_targets for prerequisite branches. and len(merges) seems to be 0 there, when the prerequisite branch is merged :(
<flacoste> lol, "trendy vs free"!
<flacoste> jml: should we list other internal lists tangentially related like canonical-tech or canonical-javascripters?
<flacoste> they are not strictly launchpad
<flacoste> otherwise, i don't see anything missing
<jml> flacoste: I guess we can list those in a separate section.
<lifeless> jml: flacoste: want to continue our calls ?
<flacoste> lifeless: going to lunch
<lifeless> gary_poster: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/670013
<_mup_> Bug #670013: preflight check for removing edge cluster <Launchpad Foundations:New> <https://launchpad.net/bugs/670013>
<gary_poster> ack lifeless, thanks
<jml> lifeless: I'm otp
<gary_poster> dobey, swamped, will ping you when I'm up for air
<lifeless> dobey: please file a question on launchpad-code
<lifeless> dobey: that will get to the right people
<dobey> ok
<deryck> lifeless, I'd like to change the hard_timeout rule for BugTask:+create-question to 25 seconds.  Any objections?
<lifeless> yes
<lifeless> past 20 seconds is likely to start permitting starvation of appservers via haproxy
<lifeless> sadly.
<lifeless> we have some data that mails are taking a long time to send.
<lifeless> adding more datagathering there would be a --very good-- idea
<lifeless> deryck: We know that that page goes past 30 seconds
<lifeless> deryck: and that will just get cut off by haproxy - there's no benefit taking it up that high.
<deryck> lifeless, no benefit past 30, for the case where we know we'll hit 30?
<lifeless> deryck: if the total time before the response hits haproxy is 30 seconds, the user gets a 'could not contact lp'
<lifeless> deryck: the hard_timeout only influences time from when the request servicing *starts* in LP
<lifeless> deryck: haproxy feeds a backlog of 4 requests per appserver (active threads=4, depth=8)
<lifeless> deryck: Taking the hard_timeout above 20 seconds means that < 10 seconds queue time (under load) is permitted, and the queue depth (under load) will be 4 on the server...
<lifeless> deryck: so I wouldn't take the hard_timeout much above 20 seconds
<lifeless> deryck: I set milestones to 22 seconds because the data suggest its capped there, more or less.
<lifeless> deryck: For +create-question, I don't think you'll let many more pages complete by adding 5 seconds - I think there is a big gap
<lifeless> between 'ok' and 'fucked'
<deryck> lifeless, ah, ok.  I follow you now.
<deryck> lifeless, the OOPS from micahg had 15.XX seconds.  But the graph seemed to indicate lots on the right side at 20, so was guessing really.
<lifeless> deryck: we need to debug the mail thing
<deryck> lifeless, right.  I also think we should do 20 then to see if that helps micahg temporariliy and get this work scheduled ASAP.  cool?
<lifeless> deryck: I realise its strictly foundations, but I'd like to encourage you to just go ahead and add instrumentation to find out where the time is going :)
<lifeless> deryck: cool
<deryck> lifeless, oh, I really don't think like that short of trying to assign bugs correctly :-)  I'm happy to add it.
<deryck> I'm extremely overloaded right now, though.  So it will be late this week or first of next before I can add it.  And want to get micahg going again if I can.
<sinzui> flacoste, mumble?
<flacoste> sinzui: yes sir
<jml> lifeless: would you be ok w/ me upgrading LP to use a build of testtools trunk?
<lifeless> +10000000
<jml> cool.
<jml> lifeless: I figure that upgrading fixtures & testtools separately will ease landing of my testtools-experiment branch
<lifeless> jml: I'm always happy with snapshots [of upstream] that make things better. Snapshots [not of upstream] need a /little/ more thought.
<jml> lifeless: makes sense.
<jml> normally I would do a release first, but I'm reluctant to release all of this deferred stuff until I've proven that it works for at least one project's test suite.
<lifeless> zigactly.
<lifeless> jml: Can we chat?
<lifeless> or rather. Speak.
<jml> lifeless: yes. gimme a sec to finish dumping state.
<mars> has anyone hit PQM with a testfix yet?
<mars> I'll assume no - I have a one-line change I can hit it with
<jml> lifeless: ok. ready.
<lifeless> skype doesn'tthink so
<lifeless> jml: you've dropped off?
<lifeless> flacoste: have discussed work queues with jml - EFUTURE, but broadly interesting
<flacoste> lifeless: ok
<flacoste> lifeless: btw, seems like gary and you see eye to eye on the value of regression tests :-)
<lifeless> :)
<gary_poster> :-)
<lifeless> bah
<lifeless> I'm going to have to do the mentoring patch on db-devel.
<lifeless> the magic cross-check stuff bites
<jml> g'night all.
<lifeless> night
<lifeless> jelmer: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html - we're blocked on QAing that
<lifeless> deryck: https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-pageids.html
 * deryck looks
<lifeless> deryck: click on '99% under time' and observe that the top (or near top) row is BugTask:+create-question, with a 99% completion time of 102 seconds.
 * deryck is waiting on the page
<lifeless> science bitches, it works!
<cr3> leonardr: hi there, I've been looking at the EntryResource class under the lazr.restful._resource package to find out how I could potentially add the attribute http_link, similar to the self_link, across all my objects.
<leonardr> cr3: are you trying to fix the bug about this? i thought rockstar already had something
<cr3> leonardr: I wasn't aware of any bug, might it be related to the EntryResource class is assumed in a few places, like EntryHTMLView for example?
<lifeless> I'm going to - shock, horror - write some code
<deryck> lifeless, so to simply state your assertion -- "changing the default_timeout will not help. We have to fix the page to help micahg."
<lifeless> deryck: yeah :)
<deryck> gotcha
<lifeless> deryck: I thought the evidence would be useful, and scary.
<deryck> lifeless, yes, it is useful.  Nothing scares me about lp anymore.  Scars me, yes.  Scares me, no.
<lifeless> \o/
<lifeless> you've passed the fear threshold
<deryck> or else I'm too dumb to be afraid.
<lifeless> rotfl
<lifeless> EdwinGrubbs: hey
<lifeless> EdwinGrubbs: did I answer your question sufficiently about that timeout bug the other day ? I put it in the bug discussion.
<SpamapS> I heard at UDS that stats for PPA downloads are coming soon? Anybody know if there's an open bug/blueprint for it that I can point others at?
<lifeless> SpamapS: http://www.google.com/search?sourceid=chrome&client=ubuntu&channel=cs&ie=UTF-8&q=ppa+stats
<lifeless> SpamapS: learn to love the search
<SpamapS> lifeless: i'd like to personally thank you for mot lmgtfy'ing me for that gross misappropriation of your time and brainpower. ;)
<EdwinGrubbs> lifeless: thanks for adding that comment. I'll try the eager loading, although I have doubts that it will help, since it looks like most of the time is spent creating storm objects. However, I've only looked at ++profile++show so far, and I need to check if the ++profile++log got copied over to devpad.
<lifeless> EdwinGrubbs: so the storm object creation concern is coupled to storm object volume
<lifeless> EdwinGrubbs: less queries can reduce that volume
<lifeless> EdwinGrubbs: how many objects are being created? 5K ? 10K ?
<lifeless> SpamapS: I'd never do that to you :).
<EdwinGrubbs> lifeless: I'm guessing 7,500 based on the number of _set_values calls. I just noticed that the ++profile++show says "Total(ms)". Isn't that in seconds?
<lifeless> EdwinGrubbs: yes
<lifeless> its bong from the 2.4 change to profiling
<lifeless> EAttentionToDetail.
<lifeless> EdwinGrubbs: for clarity, yes, its in seconds.
<lifeless> the ms clause is from the older profiler which did report in ms.
<james_w> jml, hi, I'm hearing rumours that a merge of the bug and blueprint code is planned, is that the case?
<bryceh> james_w, wow
<lifeless> james_w: jml has EOD'd.
<lifeless> james_w: but yes.
<lifeless> james_w: during 2011
<james_w> is there more information available anywhere?
<lifeless> dev.launchpad.net/IssueTracker is probably the best source today
<lifeless> I don't know if that captures jmls current thinking
<lifeless> but its certainly not going to be hugely far off
<cr3> have there been requests to get the web url for objects retrieved through the api?
<james_w> cr3, yes
<cr3> james_w: cool, I had a feeling there might be a reason why it wasn't currently available, either because there was no such request or because there might be a fundamental problem with providing this information
<cr3> and searching bugs for "api url" under the launchpad project, which returned nothing, only reinforced that feeling :)
<james_w> https://bugs.edge.launchpad.net/launchpadlib/+bug/316694
<_mup_> Bug #316694: Add web_link property to resources <launchpadlib :Triaged> <https://launchpad.net/bugs/316694>
<cr3> james_w: cool, now I know what to name it too!
<leonardr> cr3: you should definitely talk to rockstar, i know he did some work on this
<rockstar> leonardr, I did, but we got stuck on some weird test failures that made things more complicated.
<cr3> leonardr: has there been a consensus on whether this should be done client or server side? I found the thread in the bug fascinating and I have no position myself
<leonardr> cr3: i think server side
<cr3> rockstar: might there be a branch I could peek at?
<thumper> morning
<lifeless> note that canonical_url is a known performance problem.
<lifeless> it would be sad to make APIs substantially slower to do this; perhaps working on canonical_url would be a good first step.
<rockstar> cr3, there is indeed.  Lemme find it.
<rockstar> thumper, morning.  Might we have a chance to catch up?
<thumper> rockstar: we may
<thumper> rockstar: let me play with the mixer and mic
<rockstar> thumper, let me find this branch for cr3 real quick and then I'll jump on skype.
<rockstar> cr3,
<rockstar> <beuno> so I preemptively blacked it out
<rockstar> Argh...
<rockstar> Stupid middle click grabbing randomness...
<rockstar> cr3, https://code.launchpad.net/~rockstar/lazr.restful/web_link
<cr3> rockstar: cheers!
<lifeless> man, its so nice being able to just use lp directly.
<lifeless> \o/
<lifeless> james_w: did that help you ?
<allenap> Does anyone know why I have had to re-authorize my launchpad-branch-lander API key 2 (3 maybe?) times in the last couple of days?
<james_w> lifeless, the spec on blueprints and bugs?
<lifeless> james_w: the wiki page :)
<james_w> lifeless, as much as a spec from 2006 can, yes
<lifeless> kk
<lifeless> I'm happy to tell you more
<lifeless> or voice chat
<lifeless> it depends what you want to know
<james_w> I'm mainly interested in where it fits with other blueprint changes, such as the accepted stakeholder proposal for a better workload page
<lifeless> so the broad description is perhaps best said as 'combine the siloed 'blueprints' and 'bugs' into a single unified workflow with the ability to move from one focus to another as may make sense'
<lifeless> we're not going to throw away things folk use
<james_w> right, but at the same time, you probably don't want to be adding more features to blueprints until it is done
<james_w> and I have several requests for blueprint changes that I am being asked to bring to the stakeholders meeting
<lifeless> uhm
<lifeless> so adding new things in blueprints doesn't really make merging easier or harder.
<lifeless> the vast bulk of existing stuff already exists.
<rockstar> thumper, I wish mumble was happier in New Zealand.
<thumper> rockstar: actually
<lifeless> so I would say that there's no particular change to worry about vis a vis requested work
<thumper> rockstar: can we try that here?
<rockstar> thumper, sure, one sec.
<thumper> rockstar: I'm using a different provider
<lifeless> thumper: whom
<thumper> lifeless: WIC (I'm hotdesking at the centre for innovation)
<thumper> rockstar: mumble seems confused
<rockstar> thumper, looks like it.
 * thumper tries again
<thumper> rockstar: trying to tell mumble to use pulse and it shits itself
<rockstar> thumper, yeah, that's how it started to feel about my USB headset.
<rockstar> thumper, you should try blowing away your mumble config and starting over.
<thumper> rockstar: yeah, where is that kept?
<rockstar> thumper, no idea.  ~/.config/mumble?
 * thumper blew away ~/.config/Mumble
<thumper> mumble is working well here
<jelmer> lifeless: *nod*
<wgrant> jelmer: Evening.
<jelmer> wgrant: hey - your branch failed to merge. I've been meaning to look at resolving the conflict but other things have been interrupting me all day.
<wgrant> jelmer: Ah. I'll fix that.
<lifeless> jelmer: if you can simply assert that deploying bug 627608  to the nodowntime alias won't break anything, we can qa-ok the bug- it owuld be ok to deploy.
<_mup_> Bug #627608: Got a 401 on a fresh purchase <qa-needstesting> <Software Center Agent:Fix Released> <Soyuz:Fix Committed by michael.nelson> <software-center (Ubuntu):Fix Released> <https://launchpad.net/bugs/627608>
<lifeless> jelmer: but I can't tell if thats true or false
<wgrant> It's fine to go anywhere except germanium.
<wgrant> But it's not that hard to QA...
<lifeless> wgrant: pls help :)
<wgrant> Perhaps I should have added "for those with DF access right now"
<wallyworld> rockstar: abentley: thumper: standup?
<thumper> mumble is working 100% for me here
<thumper> wallyworld: we are doing it right now
<rockstar> wallyworld, mumble!
<thumper> wallyworld: on mumble
<lifeless> wgrant: its important to get the necessary discipline to qa on qastaging.
<lifeless> wgrant: how would one qa it
<wallyworld> i haven't got mumble installed
<rockstar> wallyworld, sudo apt-get install mumble
<wallyworld> rockstar: yep, doing it right now
<wgrant> lifeless: I see a few methods. 1) Add a sleep to the script to make the window for adding a new token long enough to actually do it. 2) Hack the finish time in the DB to make that window longer. 3) Hack the token creation time in the DB. or 4) Be really really quick and activate a subscription at just the right time.
<lifeless> wgrant: so, disable the script. Add lots of PPAs. Say 20. Get up to the last screen in adding another ppa. run the script and add the ppa
<wgrant> lifeless: A token is created when a user activates their subscription, not on PPA creation.
<wgrant> But yes.
<wallyworld> rockstar: so what server do i connect to? any?
<lifeless> wgrant: ah
<lifeless> wgrant: so those question are for verification
<lifeless> wgrant: I want validation - is it safe to deploy, not is the bug fixed.
<lifeless> wgrant: sounds like activating a private PPA token on qastaging will do.
<wgrant> lifeless: Ah, I didn't know that that was also unknown.
<rockstar> wallyworld, there's a wiki page. One sec.
<wgrant> lifeless: You'd need to do that then run the token generation script, which qastaging probably doesn't have configs for.
<lifeless> wgrant: ok, lets set it up
<lifeless> I wonder if I can make a p3a
<lifeless> actually, I can probably nuke and refresh a token
<lifeless> the reset passwor dbutton, right ?
<wgrant> That should do it.
 * wgrant checks.
<wgrant> Yeah.
<lifeless> ok
<wgrant> That'll work.
<lifeless> I've reset it
<lifeless> now, we check by looking in the htaccess, yeah?
<wgrant> Now we need to run cronscripts/generate-ppa-access.py, and watch what explodes.
<lifeless> mbarnett: ^
<lifeless> on qastaging
<wgrant> It has no interesting flags.
<mbarnett> so, a vanilla run of that with a bunch of -vvvvvvv s
<wgrant> But it will need config, I suspect.
<mbarnett> ?
<wgrant> Right.
<wgrant> As many -v s as you can muster!
<mbarnett> i can mimic how it runs on prod
<wgrant> How does it run on prod?
<lifeless> very slowly
<lifeless> boom-tish
<mbarnett> give me a sec to extract myself from this interview, will be literally one minute
<lifeless> mbarnett: no panic
<lifeless> mbarnett: thank you
<wallyworld> rockstar: connected now. what channel?
<rockstar> wallyworld, there's a code channel.
<mbarnett> cronscripts/generate-ppa-htaccess.py -vv
<mbarnett> is all we do on prod
<wallyworld> rockstar: i've added myself to that already
<lifeless> wgrant: so mbarnett runs that.
<lifeless> wgrant: then what
<wgrant> lifeless: Find out where the qastaging config puts private PPAs.
<wgrant> (personalpackagearchive.private_root is the config key)
<mbarnett> here is the run on qastaging: http://pastebin.ubuntu.com/524684/
<lifeless> whats the config key ?
<lifeless> wgrant: e.g. 'where do I look'
<wgrant> lifeless: I just said.
<lifeless> wgrant: sorry, link futzing
<wgrant> Yay, DB config fail.
<wgrant> Also, that librarian looks wrong.
<lifeless> mbarnett: did you run that on 'staging' or 'qastaging'
<wgrant> Half is qastaging, half is staging.
<wgrant> Huh.
<mbarnett> lifeless: i ran it out of qastaging
<lifeless> mbarnett: -odd-
<lifeless> mbarnett: so, we're missing a user in the qastaging db - the generatehtppaccess (spelling?) user
<lifeless> mbarnett: and it seemed to be speaking to the wrong librarian; could you confirm the production configs are @ rev 152?
<wgrant> lifeless: But it's also using the staging librarian and half of staging's zcml.
<lifeless> wgrant: possibly stale pyc files
<lifeless> wgrant: before we -panic-
<wgrant> Heh.
<lifeless> wgrant: possibly not, of course.
<mbarnett> lifeless: sorry, looking now
<mbarnett> too many things at once!  :)
<mbarnett> nope, 151
<lifeless> mbarnett: please pull 152 in, it fixes qastagin to use the right librarian. And we need to setup and configure that user.
<mbarnett> so, can't actually just pull there.  have to figure out the best way to get 152 there
<mbarnett> branched it locally, pushing it over now
<wgrant> jelmer: I pushed the conflict resolution. Could you try to land it again, please?
<mbarnett> haha, that was a lot of work for a 1 line change... i probably should have just taken a look at the diff between those two revnos
<lifeless> gary_poster: so, I've looked at one of those bugs
<lifeless> gary_poster: I'm going to stare a little harder and see if I can spot any potential bugs; I need to grab the python 2.6 code first though
<jelmer> lifeless: sure, I'm looking into a different generate-ppa-htaccess.py bug at the moment
<jelmer> wgrant: Sure
<mbarnett> lifeless: the configs are updated (with that 1 line change).  will just take a bit of figuring to get the database user configured properly
<lifeless> mbarnett: kk
<wgrant> jelmer: Thanks.
<gary_poster> thanks lifeless
<lifeless> gary_poster: no probs
<lifeless> gary_poster: it looks like an interpreter shutdown issue to me, IMBW
<gary_poster> lifeless: but as mwhudson said, that would happen after the LOSAs had run ``stop``--so I'd guess that what we are seeing is in phase two of the problem
<gary_poster> and we don't have visibility on phase 1
<gary_poster> so one debugging step is to ask the losas to run the gdb thread script on hung processes *before* they try calling stop
<gary_poster> will pass that to them on -ops...
<lifeless> gary_poster: well
<lifeless> that presumes that the problem exists before shutdown
<lifeless> gary_poster: if its not /hung/, the shutdown may be the problem.
<lifeless> remember that we're doing deploys more often
<gary_poster> in the bug description it is after a nagios check
<lifeless> right
<lifeless> the sequence I'm thinking is this
<gary_poster> we can verify that the nagios had been successful previously
<lifeless> we deploy 11808
<lifeless> some servers don't die properly
<lifeless> nagios goes 'hey'
<lifeless> we find rev 11793 still running
<lifeless> and wedged
<gary_poster> ok reasonable hypothesis
<lifeless> it was reported on the 1st
<lifeless> the day we deployed 11811
<lifeless> it has rev 11793 itself
<gary_poster> in your hypothesis, right?  We don't have evidence of that
<LPCIBot> Project devel build (177): STILL FAILING in 3 hr 59 min: https://hudson.wedontsleep.org/job/devel/177/
<gary_poster> if this hypothesis were correct, then I think the following would not be true:
<gary_poster> nagios saw all servers alive after a roll out
<lifeless> agreed
<lifeless> gary_poster: we have the deploy log
<lifeless> gary_poster: which isn't accruate to the minute sadly. Its on LPS.
<gary_poster> right
<gary_poster> do we have a nagios log?
<lifeless> spm: ^ :)
<mwhudson> interpreter teardown is a bucket of mess
<gary_poster> heh
<lifeless> mwhudson: its a shame we can't see thread 0 :)
<gary_poster> if nagios did show all servers up before this, then we're back to asking losas to try to gather thread info before they issue init.d stop
<lifeless> mwhudson: could it perhaps have exited already? whilst holding the HEAD_LOCK or something crazy like that ?
<mwhudson> i've semi-seriously advocated running appservers in vms and just terminating the vm when you want it to go away before...
<mwhudson> lifeless: there isn't a thread 0 usually?
<gary_poster> heh
<lifeless> mwhudson: I thought gdb was 0-indexed
<mwhudson> lifeless: as i said in the bug report, the rest of the traceback from thread 1 would be interesting
 * gary_poster thinks the question mark in mwhudson's statement was misleading :-P
<lifeless> mwhudson: I certainly can't see a thread with interpreter bootstrap present
<mwhudson> lifeless: pygdb's backtrace walks though c until you hit python and then only displays python
<lifeless> mwhudson: what is the 'stop script' you mean ?
<lifeless> mwhudson: can you fix that ?
<mwhudson> lifeless: i guess
<lifeless> mwhudson: pretty please with a We Can Fix This Then on top ?
<mwhudson> lp:pygdb is team-owned fwiw :-)
<mwhudson> but it's a bit write only code
<lifeless> mwhudson: I can context switch into it if needed, but I'd *prefer* to not bootstrap on that this week.
<gary_poster> according to flacoste this enters drop-everything mode after the next strike
<gary_poster> but not yet
<mwhudson> lifeless: the hard part is the heuristics
<gary_poster> I should run go do family things.  lifeless, should I pass the pygdb-before-init.d-stop request to the losas, and the nagios question, via RT, or may I leave it with you?
 * gary_poster needs to go.  Younger son is playing the paper-tube-o-phone
<gary_poster> thank you and good night
<mwhudson> ooh, i have one good heuristic actually
<spm> lifeless: heyo. just gimme a sec to get up to speed wrt context/backlog/handover
<mwhudson> huh, i want a multiset for about the first time ever
<spiv> mwhudson: collections.defaultdict(lambda: 0) ?
<mwhudson> spiv: well yeah + some sugar on that
<mwhudson> lifeless: r52 of lp:pygdb should be more useful in this case
<mwhudson> spiv: did you know that collections.defaultdict(int) is the same as that? :)
<spiv> mwhudson: I choose not to know that ;)
<spiv> I find it a bit weird for immutable, non-container objects to have constructors that make "empty" or "zero" instances of themselves.
<spiv> so list() and dict() seem reasonable, but int() and str() make me uncomfortable.
<spiv> And tuple()... I could go either way on that ;)
<spiv> mwhudson: how does pygdb relate to the shinier python-gdb stuff that's builtin in maverick?
<spiv> (I mean shinier relative to previous ubuntus, rather than implying it's shinier than than pygdb)
<mwhudson> spiv: it works on lucid i guess :-)
<mwhudson> i don't really know tbh, it's all a bit different
<mwhudson> in particular it drives gdb from outside
<lifeless> spm: we want to start using a newer pygdb to get backtraces from hung appserverfs.
<lifeless> spm: and, we think deploys trigger this, so we want to deliberately look for it after deploy
<spm> lifeless: why do you feel that deploys cause this?
<spm> (sure on the pygdb, np there)
<lifeless> spm: the most recent two show rev 11793 in their trace and come from the day we deployed 11811
<lifeless> spm: and the internals smell of interpreter shutdown.
<lifeless> so we think its all a dupe of the 'apservers not shutting down' bug
<spm> hmmm. one was actually dead for quite a long period... /me refreshes memory with times/dates
<lifeless> spm: that would fit quite well actually.
<lifeless> mwhudson: thank you
<spm> yes lpnet11, that was a dead on a monday (here), but istr had been dead for ~ 12-16 hours.
<lifeless> spm: hmmm
<lifeless> well the other requested thing
<spm> yeah - it doesn't quite fit.
<spm> sure. rev 52?
<lifeless> is to start getting a pygdb run *before* shutdown on deployments
<mwhudson> lifeless: np, it was easy when i actually switched my brain on a bit
<spm> Oooo kat
<spm> kay even
<lifeless> however
<lifeless> I think thats a problem
<lifeless> it will be a lot of data
<mwhudson> pygdb is fairly slow
<spm> mwhudson: it's overrated - brain switch on
<lifeless> that we won't look at
<lifeless> and its slow
<mwhudson> generating a core and pygdb-ing that will be less invasive
<lifeless> so I'd like to go with 'it looks like shutdown to us, and the new pygdb will tell us more'
<mwhudson> but will obviously use a lot more disk, if only temporarily
<spm> have you seen how big those core files are?? :-)
<lifeless> spm: hey, do you have the core from lpnet11 ?
<mwhudson> i guess dumping the core takes a while too
<spm> lifeless: probably... lemme check.
<lifeless> spm: if so please run rev52 against it
<lifeless> we should get more data immediately.
<spm> lifeless: I don't believe I did on yesterdays(??), but pretty sure I did on monday
<spm> bugger no. I didn't. most recent is 2010-10-18 03:42 lpnet12-2010-10-18.core.24795
<spm> I'm 8sure* I did take one this week tho... poking...
<lifeless> give it a shot
<lifeless> its more likely that we have one problem than two causing hangs.
<spm> hrm. might have been a codebounce hang that I dumped.
<spm> bother
<lifeless> lpnet12 should be useful to analyse
<lifeless> rev52 will show us the entry point to the app and hopefully where its at outside of python frames
<spm> nod
<spm> err - how do I run vs a core? backtrace <corefile> ?
<lifeless> mwhudson: ^
<mwhudson> spm: -c $core
 * spm won't point to --help giving a smash. cause that'd be rude. :-P
<mwhudson> spm: right
<spm> heh
<mwhudson> this was hardly written using principles of user interaction design
<spm> I can tell
<mwhudson> (or software engineering, come to that)
<lifeless> mwhudson: punching is a principle.
<mwhudson> lifeless: only one that was followed by accident in this case
<spm> lifeless: mwhudson: https://pastebin.canonical.com/39285/
<mwhudson> um
<mwhudson> that's not very useful, is python2.6-dbg installed on that machine?
<spm> ii  python2.6-dbg                               2.6.5-1ubuntu6
<spm> that dump is oldish - dunno if that messes things at all
<spm> 2-3 weeks old.
<mwhudson> shouldn't do
<spm> I can always gcore something now, and we can verify?
 * lifeless sings the edge is dead happy song
<spm> :-)
<mwhudson> i'm not sure there's much point
<spm> edge is dead, long live edge!
<mwhudson> the modification i made is fairly single-purpose: give more info when something hangs in a __del__ method
<lifeless> mwhudson: so I'd really quite like ot see the stack back to main()
<lifeless> mwhudson: the python lines are ok, but every interpreter hang previously I've solved purely on the C frames
<lifeless> mwhudson: if we have to *choose*, lets get the full C frame.
<lifeless> mwhudson: "'NoneType' object has no attribute 'exception'",), <traceback at remote 0x2b44aceb5128>), kw=0x0) from ../Python/ceval.c
<lifeless> mwhudson: \o/
<mwhudson> lifeless: if the c traceback is enough, thread apply all bt is all you need?
<lifeless> mwhudson: maybe just doing that in paralle.
<lifeless> spm: can you do thread apply all bt in that core too ?
<lifeless> spm: also, we need to start gathering the appserver log for the time that it hung
<lifeless> there may well be things like
<lifeless> 'unhandled exception in thread xxx' in stderr
<spm> lifeless: hrm. likely pebkac here. gdb <core> ; thread apply all bt<cr> ??
<lifeless> yes
<spm> hrm. "/home/launchpad/lpnet12-2010-10-18.core.24795": not in executable format: File format not recognised
<lifeless> gdb `which python` <core>
<spm> ah
<spm> https://pastebin.canonical.com/39286/
<spm> curious. it even looks the same, with the ??'s.
<lifeless> garh
<lifeless> thanks
<lifeless> warning: core file may not match specified executable file.
<mwhudson> lines like "#7  0x0000000000000010 in ?? ()" don't look very plausible to me
<mwhudson> is it possible that this core file predates the update to lucid, or something?
#launchpad-dev 2010-11-03
<lifeless> spm: https://bugs.launchpad.net/launchpad-foundations/+bug/669296 - added what we need there
<_mup_> Bug #669296: lpnet11 "critical timeout" to nagios, non responsive <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/669296>
<lifeless> mwhudson: 18th, no.
<lifeless> ok, popping this off stack
<lifeless> mbarnett: did you get that db user setup ?
<spm> mwhudson: hrm. no. the lucid u/g on this server was back in mid august
<spm> actually - doesn't appear to have had any interesting activity of note in quite some time - heh, the dbg was installed the day after that core file. probably directly as to why I asked for it.
<lifeless> is this a typical windmill failure:
<lifeless> WindmillTestClientException: {u'suite_name': u'Branch Subscription Ajax Load Test', u'result': False, u'starttime': u'2010-10-3T5:8:35.930Z', u'params': {u'xpath': u'//a[@class="sprite add subscribe-self js-action"]', u'validator': u'Subscribe yourself'}, u'debug': u'Looking up xpath //a[@class="sprite add subscribe-self js-action"], failed.', u'output': None, u'endtime': u'2010-10-3T5:8:35.932Z', u'method': u'asserts.assertText
<wgrant> lifeless: You got truncated.
<lifeless> WindmillTestClientException: {u'suite_name': u'Branch Subscription Ajax Load Test', u'result': False, u'starttime': u'2010-10-3T5:8:35.930Z', u'params': {u'xpath': u'//a[@class="sprite add subscribe-self js-action"]',
<lifeless>  u'validator': u'Subscribe yourself'}, u'debug': u'Looking up xpath //a[@class="sprite add subscribe-self js-action"], failed.', u'output': None, u'endtime': u'2010-10-3T5:8:35.932Z', u'method': u'asserts.assertText'}
<wgrant> Yeah, that's pretty normal.
<spiv> Heh, truncated by two bytes.
<wgrant> As in it looks almost like a legit failure.
<wgrant> Not like we've been seeing lately!
<lifeless> my mentoring branch failed in ec2
<lifeless> all the failures are windmill
<wgrant> Do they pass locally?
<wgrant> I haven't seen a spurious failure like that for months.
<lifeless> ok
<StevenK> https://hudson.wedontsleep.org/job/devel/lastFailedBuild/#showFailuresLink :-(
<lifeless> I'll try later, bootstraping groups first, fighting jetlag all the way.
<wgrant> StevenK: Huh.
<thumper> lifeless: why aren't we rolling code out to cesium?
<lifeless> thumper: https://rt.admin.canonical.com//Ticket/Display.html?id=42207
<thumper> lifeless: ta
 * thumper sighs
<lifeless> thumper: 'sup ?
<thumper> <ul><li>... gives no bullets
<persia> dump CSS and check again?  Then check render context?
<thumper> li { list-style:none outside none; } from the stylesheet
<persia> Doesn't list-style:none suppress the bullets?
<thumper> yep
<thumper> found it
<thumper> need <ul class="bulleted">
 * thumper relocates
<spiv> thumper: so when is bzr on codehosting going to be upgraded?  https://bugs.edge.launchpad.net/bzr/+bug/636930
<_mup_> Bug #636930: Upgrading a repository fails with 'Inter1and2Helper' object has no attribute 'source_repo' <bazaar> <sru> <upgrade> <verification-done> <Bazaar:Fix Released by spiv> <Bazaar 2.2:Fix Released> <Launchpad Bazaar Integration:Triaged> <bzr (Ubuntu):Fix Released> <bzr (Ubuntu Maverick):Fix Released> <https://launchpad.net/bugs/636930>
<lifeless> spiv: you could propose the patch
<lifeless> spiv: there aren't any api breaks in the point release are there?
<StevenK> thumper: Do you have a conflict of interest to mentor my review of wallyworld's branch, or shall I find another mentor just for that review?
<wallyworld> StevenK: perhaps thumper can explain why the diff messed up
<StevenK> I added a comment with a theory, but it's only that
<wgrant> StevenK, wallyworld: What's the problem with the diff? merge-queue-index was merged, but merge-queues-model is listed as the prereq.
<StevenK> Ah ha, perhaps there are two prereqs ...
<wgrant> merge-queues-model is (except for one rev) a prefix of merge-queue-index.
<spiv> lifeless: no API breaks, no
<spiv> lifeless: huh, but looking at NEWS I think I see a fix to another codehosting bug :)
<wallyworld> wgrant: the diff in the mp showed rockstar's changes as well as mine, even though i had listed his branch as a prereq
<wgrant> wallyworld: But you merged merge-queue-index, which contains more changes than just the merge-queues-model that you set as the prereq.
<wallyworld> oh crap. really?
<StevenK> Been there, done that.
<wallyworld> ffs
<wgrant> http://bazaar.launchpad.net/~wallyworld/launchpad/person-mergequeue-listview/revision/9917
<lifeless> spiv: so, would love it if you could just commit the tar to the dist cache , update versions.cfg and propose a merge ;)
<wallyworld> hmmm. thinking out loud - i did have all the necessary changes locally to build on for my stuff, so i must have merged the right thing at some point
<spiv> lifeless: that "just" there presupposes I have familiarity with "the dist cache" and "versions.cfg", not to mention an up-to-date launchpad dev environment.
<spiv> (FWIW the other bug is https://bugs.launchpad.net/launchpad-code/+bug/668176)
<lifeless> spiv: I have an up to date bzr dev environment :)
<StevenK> I think your guilt trip needs work
<wallyworld> wgrant: StevenK: yes, it appears my stupid fingers typed the wrong pre req branch when creating the mp :-(
<spiv> lifeless: well, if you "just" make the lp dev environment as easy to have as bzr's... ;)
<StevenK> wallyworld: Pity the only way to curently fix that is to recreate the MP
<lifeless> spiv: for this patch, it is ~ - branch twice,
<StevenK> spiv: rocketfuel-setup is easy ?
<wallyworld> StevenK: i can easily do that. if fact, i think i should so that the diff is fixed up
<wgrant> lifeless: branch $VERY_LARGE_THING.
<lifeless> spiv: I do realise that there is overhead
<lifeless> spiv: OTOH I think its sensible for bzr devs to have a (even dated) lp dev environment around, because so much of networking will interact with lp
<spiv> And I do have a dated lp dev environment around.
<StevenK> rf-get ; sleep 3h ?
<spiv> But I don't have the familiarity with the current dev processes and policies, for instance... anyway, I'm fairly sure the barriers here are already pretty well understood.
<lifeless> spiv: the review policy is unaltered [for you]; I'd be delighted to guide you through any hoops.
<lifeless> still, not to worry
 * lifeless goes back to perf work
<spiv> Well, let's put it this way: the way to get someone hooked on developing for your project is not to say "here do this menial task" :P
<lifeless> spiv: I was hoping it was 'here, you can get that thing you want done right now'
<lifeless> wgrant: yo
<lifeless> wgrant: from what I can see, there is archive-index.pt, archive-macros.pt and a vocabulary all using ArchiveSourcePublications
<lifeless> wgrant: archive-index.pt needs newer_source_distributions preloaded
<lifeless> the other two don't.
<lifeless> wgrant: does that seem plausible ?
<wgrant> lifeless: What about archive-packages?
<lifeless> wallyworld: do you want a voice call perhaps, as a pseudo pub?
<lifeless> wgrant: looking in a bit
<wallyworld> lifeless: if versions.cfg already lists bzr 2.2.0 (since mid Aug), what's to change to make codehosting use the correct bzr version?
<wallyworld> lifeless: we can have a voice call if you want
<lifeless> wgrant: archive-packages uses source-package-list macro
<lifeless> wallyworld: do you have skype?
<lifeless> wallyworld: 2.2.1 I think
<wgrant> lifeless: It doesn't need the prepopulation?
<lifeless> wgrant: it doesn't seem to reference newer_source_distributions
<lifeless> wgrant: so the tension is - more data, slower page.
<wgrant> lifeless: The macro probably does, though?
<wallyworld> yes. ian.m.booth at gmail.com is my skype id
<lifeless> wgrant: no, the macro doesn't
<lifeless> sad moment, removing ian c from my skype list ;(
<lifeless> s/;/:/
<wgrant> lifeless: greeping for newer_source_distributions shows nothing anywhere.
<wgrant> lifeless: What's the actual name?
<persia> wgrant, Going through my post-UDS notes, I'm supposed to ask you for details of the put-copyright-in-librarian script so StevenK can run it, and we're one step closer to native-source-sync.
<wgrant> persia, StevenK: http://pastebin.ubuntu.com/524798/ is the script. But it probably needs a DBLoopTuner thrown in.
<wgrant> And it might need some bitrot repair.
<StevenK> wgrant: Are you able to sort both of those?
<lifeless> newer_distroseries_version
<wgrant> lifeless: archive-packages uses source-package-list uses sourcepackagepublishinghistory-listing-archive-detailed uses newer_distroseries_version
<wgrant> StevenK: I probably shouldn't, but I will anyway.
 * wgrant fixes.
<LPCIBot> Project devel build (178): STILL FAILING in 3 hr 31 min: https://hudson.wedontsleep.org/job/devel/178/
<LPCIBot> Launchpad Patch Queue Manager: [r=allenap][ui=none][no-qa] Remove another with_statement.
 * wgrant stabs the sampledata in the face.
<lifeless> wallyworld: http://sweng.the-davies.net/Home/rustys-api-design-manifesto
<wgrant> stub: Hi.
<lifeless> wgrant: so
<lifeless> wgrant: do we think then that enough things use newer_distroseries_version that we should preload it unconditionally? it would make this patch rather simpler.
<wgrant> lifeless: What would preload it?
<wgrant> Only those two views should need it.
<wgrant> Mmm, I guess +copy-packages and +delete-packages have it too, but that's not really useful.
<wgrant> But where you would preload it unconditionally?
<lifeless> adapters
<lifeless> ArchiveSourcePublications:__iter__
<lifeless> this performance problem is a regression - someone added an attribute
<lifeless> the ArchiveSourcePublication adapter uses delegates()
<lifeless> so it passes through and we get - voila - death by queries.
<wgrant> Ah.
<wgrant> lifeless: ASP is only used by batched_sources, which is only used in those four views. So go ahead and make it unconditional.
<wgrant> StevenK: http://pastebin.ubuntu.com/524818/ seems to work.
<wgrant> StevenK: And should be relatively DB-friendly.
<persia> wgrant, Thanks a lot for both remembering what I meant regardless of what I said, and fixing that with so little notice.
<lifeless> brb
<stub> yo
<wgrant> stub: With the script I just linked, how would you prefer I handled transactions? One per chunk, or one per SPR?
<wgrant> Different TunableLoops seem to do it differently.
<stub> wgrant: TunableLoops should commit once per chunk
<wgrant> stub: That's what I thought. Thanks.
<stub> At least DBTunableLoops should. Let me know which ones if you find one that does differently.
<wgrant> I think it was one of the old Translations migration scripts.
<stub> Some of them existed before DBTunableLoop
<wgrant> Ah.
<StevenK> wgrant: I was planning on running it on dogfood, since persia said it only needed one run
<persia> The results of the run need to be saved appropriately, mind you.
<StevenK> Clearly
<wgrant> StevenK: It needs to be run on prod at some point... and running it on dogfood could well DoS it.
<wgrant> But perhaps you could run it for a chunk or two.
<persia> Since all recent uploads already have the changelogs in the right place, would it be possible to run against somewhere else and then stuff the results into prod, or is that just ugly, bad, and dangerous?
<wgrant> Mostly just pointless and possibly more than poor mawson would like to handle.
<persia> Ah.  pointlessness is to be avoided.
<StevenK> wgrant: So it needs to be checked in, or just run on loganberry?
<wgrant> StevenK: Just run on somewhere.
<wgrant> Are you going to try it on DF?
<LPCIBot> Project db-devel build (113): SUCCESS in 3 hr 55 min: https://hudson.wedontsleep.org/job/db-devel/113/
<lifeless>     Hard / Soft  Page ID
<lifeless>      279 /    8  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>      150 /   52  Person:+commentedbugs
<lifeless>       90 /    0  MailingListApplication:MailingListAPIView
<lifeless>       80 /  238  BugTask:+index
<lifeless>       78 /  251  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>       18 /   86  Archive:+packages
<lifeless>       16 /   16  Archive:+copy-packages
<lifeless>       14 /    8  ProjectGroup:+milestones
<lifeless>       12 /  270  Distribution:+bugs
<lifeless>       11 /   13  DistroSeries:+queue
<wgrant> lifeless: Those should mostly be 0 tomorrow, right?
<lifeless> well
<lifeless> we're seeing lower layer things now
<lifeless> s/layer/ frequency
<lifeless> and some - like Archive:EntryResource:getBuildSummariesForSourceIds just need fixing.
<lifeless> spm: reckon you could do the qastaging setup for the p3a thing we were discussing with mbarnett before
<rockstar> wallyworld, hi
<wallyworld> rockstar: yo
<rockstar> wallyworld, your recent branch deletes the branchmergequeue model.
<rockstar> Why?
<wallyworld> rockstar: the diff is lying. my local copy still has it and a bzr --preview merge still shows it
<wallyworld> what could have happened?
<rockstar> wallyworld, hm.
<rockstar> wallyworld, okay.
<wallyworld> i did modify it but not delete it
<wallyworld> did you see i made a new mp with the correct pre-req branch this time?
<rockstar> wallyworld, the other thing I'd say is that all links to branch merge queue pages need to be hidden behind the feature flag.
<wallyworld> i thought i had done that but will double check. i may well have missed something
<rockstar> wallyworld, the menu item Links should probably have the enabled flag hinge on the feature flag.
<wallyworld> or not even be rendered
<wallyworld> that was mmy intention but i may have got distracted and left that bit out
<rockstar> wallyworld, if they aren't enabled, they won't be rendered.
<wallyworld> cool. i should have remembered that from when i was in that bit of the code a few weeks ago
<wallyworld> i'm concerned that the diff is messed up though
<rockstar> wallyworld, yeah, you might have some wierd criss crossing going on.
<wallyworld> well, as i said a local bzr --preview merge .... looked ok and i double checked everything was pushed
<rockstar> wallyworld, yeah, criss cross merges do weird things to the preview diffs on Launchpad though.
<rockstar> Although usually they just claim conflicts.
<wallyworld> well, i can always grab another copy of that branch of lp and double check that file
<StevenK> wgrant: Sorry, was distracted. If you think it's useful to see if it works on dogfood, we can do that
<StevenK> lifeless: Do you disagree with the implementation in https://code.launchpad.net/~mars/launchpad/add-profiling-feature-flag/+merge/39179 ?
<lifeless> hmm?
<StevenK> lifeless: You made a comment, but I'm having trouble comparing the code and your comment
<lifeless> the code may have changed
<StevenK> Since I agree with your comment
<StevenK> It doesn't look to have changed since the 23rd
<lifeless> ah yes
<lifeless> I think the implementation is changing stuff it shouldn't
<wgrant> StevenK: I think that letting it go through a couple of hundred would be a good idea.
<lifeless> I gave the comment to provide a clear statement of what to aim for, IMNSHO
<wgrant> Shouldn't take long.
<StevenK> lifeless: Do you want to Needs Fixing it, then?
<lifeless> StevenK: be my guest
<lifeless> StevenK: sorry, I can if you want me to, but mars knows
<lifeless> StevenK: so there's not really any need to touch it at all, except to move it to 'WIP'
<StevenK> Okay, I've done that.
<lifeless> wgrant: http://pastebin.com/qgsKkCDR
<wgrant> lifeless: Yay.
<lifeless> qa-ok ?
<wgrant> lifeless: I'd reset the token and run the script it again just to be sure it works on an existing file.
<lifeless> spm: pls run again
<spm> lifeless: iz done
<lifeless> similar output
<lifeless> no crashie ?
<spm> similar, not identical, no crash
<spm> heh, similar in that no 'created' line, just the replaced.
<wgrant> Great.
<wgrant> qa-ok
<spm> wgrant: shouldn't you be studying or somthing?!??!! ;-)
<wgrant> Indeed, indeed.
<StevenK> Hah
<lifeless> StevenK: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<lifeless> qa please!
<lifeless> StevenK: or rather, what rev rolled it back ?
<lifeless> StevenK: I seem to recall that you didn't put the metadata in ?
<lifeless> spm: qastaging seems stale - it hasn't updated for a while
<LPCIBot> Project devel build (179): STILL FAILING in 3 hr 31 min: https://hudson.wedontsleep.org/job/devel/179/
<LPCIBot> Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=659171] Improve menu rendering performance
<spm> lifeless: yes. according to the log, about 2-3 days worth, which seems... odd. Wed Nov  3 05:48:02 UTC 2010 Current Revno: 11824, New Revno: 11824 - nothing to update
<lifeless> spm: this is driven by the thing on sodium
<wgrant> stable's out of date...
<lifeless> mmm, maybe. checking
<wgrant> r11824
<lifeless> ah yes
<lifeless> spm: never mind
<lifeless> buildbot hateness
<spm> np; fwiw, sodium side does seem to be working as expected.
<lifeless> StevenK: ping ;)
<wgrant> I think dogfood might have eaten him alive.
<wgrant> (he was attempting to run my script, and DF was resisting)
<StevenK> lifeless?
<lifeless> StevenK: rev 11820
<lifeless> StevenK: your testfix for it didn't include [rollback]
<lifeless> so the deploy script is unable to process it
<lifeless> StevenK: I need to know what happened
<StevenK> Can you give me the commit message?
<lifeless> dude!
<lifeless> https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<lifeless> or bzr log
<StevenK> lifeless: It was 5 days ago, and I was in Orlando!
<StevenK> And I'm trying not to swap out what I'm helping wgrant with
<lifeless> StevenK: yes, but you've got hte data locally :)
<StevenK> lifeless: If this was my testfix for devel, we spoke about it
<lifeless> StevenK: not in detail.
<StevenK> I did
<lifeless> StevenK: please look at the log and let me know
<StevenK> I changed the text scripts due to wgrant, and stupidly didn't run the branch through ec2
<StevenK> How much more detail do you need?
<lifeless> I need to know what revision fixes trunk.
<lifeless> which normally wouldbe marked with [rollback=11820] but you didn't put that in, so I have to guess.
<lifeless> I don't like guessing, so I'm asking you.
<StevenK> lifeless: 11823 fixes 11820
<lifeless> thank you
<lifeless> StevenK: the other thing is that
<lifeless> https://bugs.launchpad.net/soyuz/+bug/440652
<_mup_> Bug #440652: Allow creation of PPAs via API <api> <bad-commit-11820> <oem-services> <ppa> <qa-needstesting> <Soyuz:Fix Committed by stevenk> <https://launchpad.net/bugs/440652>
<StevenK> stevenk@mawson:~$ uptime
<StevenK>  06:09:59 up 24 days, 12:11,  2 users,  load average: 4.06, 3.77, 3.31
<StevenK> WGRANT!
<lifeless> needs qa
<lifeless> StevenK: so we're stalled until you do that.
<lifeless> or again, tell me how to.
<StevenK> lifeless: As something that's somewhat related, does lplib support qastaging? :-)
<lifeless> StevenK: yes, see uhm the token librarian bug
<lifeless> digging
<wgrant> StevenK: I blame mawson's puny RAM.
<wgrant> lifeless: Why should it have a rollback tag if it was a testfix, not a rollback?
<StevenK> Apparently 4GiB is puny
<lifeless> wgrant: because we don't have a rollforward tag yet
<lifeless> wgrant: the semantics are identical from a deploy perspective.
<StevenK> It's only 46KiB into swap, so its manging to keep everything in RAM so far
<wgrant> StevenK: But it'll be doing that by evicting the cache frequently :(
<StevenK> It's just utterly I/O constrained
<wgrant> => slow
<lifeless> StevenK: oh, it was on the list
<wgrant> DF-publisher-slow.
<lifeless> StevenK: look at the 'qa help needed' thread I started
<StevenK> s/publisher-//
<lifeless> kees describes the method
<wgrant> StevenK: Well, the publisher is the one bigjools always whinges about :P
<StevenK> wgrant: Can haz query that doesn't make mawson cry blood?
<wgrant> StevenK: Have you tried the second one I gave? What does a plain EXPLAIN (not ANALYZE) say?
<StevenK> Haha
<wgrant> That bad?
<StevenK> readline can't cope with start-of-line if the query is longer than the terminal length
<wgrant> Heh.
<spm> np; fwiw, sodium side does seem to be working as expected.rofl
<spm> rofl!!! gah.
<StevenK> wgrant: http://pastebin.ubuntu.com/524887/
<lifeless> StevenK: find the thread?
<StevenK> Still looking, trying to do three things at once
<StevenK> wgrant: Those row counts are fecking enormous
<wgrant> StevenK: They are, yes. (http://paste.ubuntu.com/524888/ has the same two queries with all the extra fields stripped out, so may be less abhorrent to read and handle)
<wgrant> I don't see how this could be any worse than the PPA expiration query.
<wgrant> Does the work on DF?
<wgrant> s/the/that/
<StevenK> wgrant: Sorry, got distracted, which do you want me to run?
<lifeless> StevenK: by deployment stuff?
 * lifeless injects a priority flag there
<StevenK> lifeless: Wha? ECONTEXT?
<lifeless> the qa
<wgrant> StevenK: Neither. Just throwing less insane versions out there in case anyone feels like looking.
 * StevenK tries to remember how to check what locales a machine supports
<wgrant> StevenK: C should be enough for everyone!
<persia> ãªã«ï¼
<spm> StevenK: "locale -a" ?
<StevenK> Ah, yes
<StevenK> Typical, mawson only has en_GB
<wgrant> persia: My Japanese is just good enough to understand that :P
<spm> 'what'?
<persia> It's not actually the correct response, in Japanese, but it translates better than the real way one expresses shocked surprise.
<spm> heh
<persia> (and, more importantly, fails to render in 'C')
<lifeless> StevenK: so, as soon as you get a cycle, lets do this qa
<lifeless> StevenK: its literally the most important thing we can do.
<StevenK> I'm working on it now
<lifeless> StevenK: \o/
<lifeless> StevenK: if I can help, I'm here.
 * StevenK tries to shake lifeless off his leg
<lifeless> oh man, terrible image
<StevenK> Haha
<StevenK> I was thinking that you have a bite like a pitbull
<lifeless> sure sure
<lifeless> I rewatched st trinians just before uds
<lifeless> *that* image was fresher.
<StevenK> I've not seen that
<StevenK> Note to self: Don't look up movie references that lifeless makes
<lifeless> lol?
<StevenK> I'd not heard of St. Trinians
<StevenK> But it did introduce me to the actual word 'spiv'
<lifeless> the new one or the old one ?
<StevenK> I'd not even heard of the books
<lifeless> thumper: looks like you didn't change the default series?
<spiv> Hah.
<spm> spiv is inferior tho. is only sp4, vs spm, which is sp1000. ergo. I win. QED.
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<spiv> spm: I like to think of it as a rank, not a score ;)
<spm> spiv: curious, was the delay there because you were laughing too hard, or more working on a suitable response to kick the pedestal out from underneath me (successfully, as it turns out) ?
<spiv> Merely because I wasn't watching IRC :)
<spm> pay that. :-)
<spm> StevenK: process-death-row on germs. thoughts here. Could that be nice'd (process pri as in) without being unduly "bad" or harmful in your view? or "Good Question" ?
<wgrant> p-d-r is misbehaving?
<wgrant> It's crap, but it should be crap on the DB, not germanium CPU, I would have thought.
<StevenK> spm: Niced up or down? :-)
<LPCIBot> Project db-devel build (114): SUCCESS in 3 hr 36 min: https://hudson.wedontsleep.org/job/db-devel/114/
<LPCIBot> Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=570506][no-qa] Add an explanitory note
<LPCIBot> about list formats.
<spm> niced to stop being so evil that I get apache timeout alerts
<spm> mind you, somewhat not helped by generate-ppa-htaccess atm, but that's a separate problem.
<spm> I've seen these alerts before, so it's not exactly "new".
<lifeless> we should have a dedicated script server soon
<StevenK> We already have one?
<lifeless> I asked charlie and james for one
<wgrant> spm: Oh, the fix for that... issue... was as braindead as I suspected it might be?
<spm> for soyuz?
<wgrant> s/fix/cowboy/
<lifeless> spm: for [qa]staging
<spm> wgrant: ah, right.
<StevenK> Note spm isn't talking about staging, but production
<spm> lifeless: this is live on ppa.lp.net
<StevenK> spm: Let me think on that
<spm> np
<spm> hrm. publishing/cron.daily-ppa could possibly do with some ionice'ing as well.
<spm> find over the entire ppa tree is never going to be friendly.
<wgrant> spm: If by "do with some ionice'ing" you mean "have some naivety beaten out of it", sure.
<spm> ha
<spm> yes, that too :-)
<StevenK> With a stick
<wgrant> Same with p-d-r.
<wgrant> And g-p-h.
<StevenK> We spoke about p-d-r at UDS
<StevenK> It's a HARD problem
<wgrant> Oh?
<wgrant> p-d-r can be done in two queries.
<spm> StevenK: so... put it on hold till... oh I dunno... middish december?
<StevenK> spm: Sure, if you double the disk space germanium has
<spm> easy! rm -rf /srv/ppa* ! SOLVED!
<StevenK> spm: p-d-r deletes crap which stops you lot coming after us with bats since the librarian is full
<wgrant> StevenK: What's making p-d-r so slow at the moment?
<wgrant> It's not doing the hard stuff.
<spm> seems pretty heaily cpu based atm
<wgrant> That's the dominator.
<StevenK> The sheer number of PPAs
<wgrant> StevenK: Right, it iterates over every PPA.
<wgrant> Rather than querying for those with condemned publications.
<StevenK> Right
<wgrant> == not terribly hard problem
<StevenK> wgrant: Do you think this is patchable?
<lifeless> bah
<lifeless> so those windmill tests
<lifeless> efail
<StevenK> I note they keep failing in Hudson too
<wgrant> StevenK: The query to reduce the set of candidate archives perfectly is non-trivial. But a slightly over-broad one is simple.
<wgrant> And would probably vastly improve performance.
<StevenK> wgrant: How much of a win are we talking?
<wgrant> StevenK: I don't know how it performs on production.
<StevenK> Slowly
<wgrant> But we can probably get it to only consider a few hundred archives each time.
<spm> I was going for badly, but slowly works.
<StevenK> wgrant: A few hundred rather than a few thousand?
<wgrant> StevenK: I think there's like 15000, but yeah.
<StevenK> A factor of 50 drop is teh awesome
<wgrant> If it's still really slow after that, we can look at optimising the remaining queries.
<spm> we should shard ppa.lp.net
<wgrant> But I think we should try the trivial fix first...
<lifeless> wgrant: you asked if the windmill tests were passing locally
<lifeless> wgrant: no
<lifeless> wgrant: but they don't on trunk either
<lifeless> wgrant: for me
<wgrant> lifeless: .... that's a bit awkward.
<StevenK> Oh, damn it, I think I know what's missing
<wgrant> Oh, trunk, not stable?
<lifeless> wgrant: the devel rev I branched from
<lifeless> wgrant: could you perhaps try
<wgrant> Which test?
<lifeless> lp.bugs.windmill.tests.test_bug_me_too.TestMeToo.test_me_too from my mentoring branch
<wgrant> That failed on Hudson.
<wgrant> So it sounds legit.
<wgrant> But running locally....
 * StevenK grumbles, since he broke dogfood
 * wgrant grumbles and prepares to go back to i386.
<lifeless> StevenK: so, how did the qa go
<StevenK> I'd like to test a hunch, but I need a dogfood for that
<lifeless> could a losa help you on qastaging instead ?
<StevenK> Can we do cowboys on qastaging?
<lifeless> yes
<lifeless> its our qa environment
<StevenK> Right, let me prepare a proper patch
<lifeless> it exists for figuring out if things are good or fucked.
<StevenK> With me, it's mostly the latter :-(
<wgrant> StevenK: Did you break Hudson too?
<wgrant> It just stopped responding to me.
<wgrant> Ah, alive again.
<wgrant> Or not.
<StevenK> It's working for me
<wgrant> Yeah, happy again.
<StevenK> It tends to not like loading our test results
<wgrant> Ah, indeed, that's what I'd just asked it to do.
<wgrant> lifeless: I don't know what buildbot thinks is good.
<wgrant> lifeless: But the culprit is r11825.
<wgrant> Reverting that fixes everything.
<lifeless> wgrant: thank you for digging
<lifeless> wgrant: please mail the list saying that windmill isn't random :)
<lifeless> I'll prep a rollback now
<lifeless> for clariry
<lifeless>   [r=thumper][ui=none][bug=667586] Improve title and error message of
<lifeless>         invalid branch links
<lifeless>  ?
<wgrant> Right, reverting that one fixed it (after a 'make build'; not sure if that is necessary).
<wgrant> Could you try?
<wgrant> wallyworld: Any ideas?
<wgrant> And reverting the reversion breaks it again.
<wgrant> Nothing obvious in the diff.
<wgrant> But it's YUI.
<wgrant> And I don't know YUI..
 * StevenK blinks
<wgrant> Hm?
<lifeless> wgrant: after make , it still fails for me
<thumper> lifeless: ?
<lifeless> thumper: the windmill failures look like they may be a bad commit
<wgrant> lifeless: You've made build?
<lifeless> wgrant: yes, repeating now
<thumper> lifeless: I guess roll it back and we'll take a look
<lifeless> thumper: trying to confirm
<lifeless> thumper: it fails both ways for me
 * wgrant is rereverting.
<lifeless> thumper: care to run lp.bugs.windmill.tests.test_bug_me_too.TestMeToo.test_me_too in devel
<thumper> sure
<lifeless> thumper: and then do a 'bzr merge -r 11825..11824 .' to back it out; make build, and run that test again ?
<wgrant> And it passes.
<lifeless> wgrant: yeah, I'd like one more data point cause its broke both ways for me
<wgrant> Certainly.
 * thumper is poking
<thumper> yep
<thumper> back it out
<thumper> I can see the error
<lifeless> doing so
<thumper> a.slice is not a function
<thumper> [Break on this error] var b = a.slice(), i = 0, n = -1, item = null;
<thumper> collection.js
<thumper> which was added in that branch
<wallyworld> thumper: just got back from dr appt - my branch break something?
<thumper> yep
<thumper> see comment just above
<thumper> breaks bug js
<thumper> and probably others
<thumper> I don't know why we are getting this
<thumper> but we are
<wallyworld> doesn't make sense to me either
<thumper> lets look at it tomorrow
<wallyworld> the only new thing done was importing collection.js to get the Array.unique() method
<thumper> right
<thumper> collection.js loading is causing problems
<StevenK> lifeless: My cowboy still doesn't fix it :-(
<wgrant> StevenK: What have you broken?
<StevenK> wgrant: A new method I added
<wgrant> createPPA?
<wallyworld> hmmm. i'll revert to the version of my js code that didn't rely on Array.unique()
<StevenK> Yeah
<wallyworld> a bit messy but doesn't need collection.js
<lifeless> StevenK: I don't care if it works :)
<lifeless> StevenK: I care if we can deploy.
<lifeless> StevenK: if we can deploy, qa-ok :)
<StevenK> It's incredibly strange. It's returning a dict of, well, me
<StevenK> ppa = me.createPPA(name='test2', displayname='Test 2')
<StevenK> That should return an IArchive
<wgrant> Let's see.
<lifeless> StevenK: sounds like its qa-ok
<StevenK> I still twitch when you tell me to do so
<StevenK> Penalty card for lying, etc
<lifeless> StevenK: we need to split out the different angles
<lifeless> StevenK: and fortunately, they already are:
<lifeless> set to in-progress
<lifeless> qa-ok
<StevenK> lifeless: I'd also refactored the browser code in that commit, but that works
<StevenK> It's the API call that doesn't
 * wgrant mauls the WADL generator.
<lifeless> StevenK: does it break anything that worked before
<StevenK> lifeless: Not that I can see
<lifeless> StevenK: so qa-ok it already ;)
<lifeless> thumper: separately, you were going to change the trunk series ?
<thumper> lifeless: I'll add it to my todo list
<lifeless> thanks!
<StevenK> lifeless: Commented and done
<lifeless> StevenK: thanks
<wgrant> StevenK: So, WTF.
<StevenK> lifeless: Re: your mail, does that mean Archive:+packages is now roughly 400 queries as opposed to 1100?
<lifeless> StevenK: probably 800
<StevenK> wgrant: Hm?
<lifeless> depends on how much work was hidden behind the thing I've grouped on
<wgrant> StevenK: That method.
<StevenK> ... is complete crack?
<lifeless> StevenK: https://pqm.launchpad.net/ - thats what a fix for a broken rev should look like
<wgrant> StevenK: Ahhhhh.
<wgrant> StevenK: launchpadlib bug.
<wgrant> StevenK: Makes sense if you think about it.
<wgrant> StevenK: lp.me.createPPA() results in a POST to /people/+me.
<wgrant> StevenK: /people/+me redirects to /~name16.
<wgrant> launchpadlib GETs /~name16.
<wgrant> Boom.
<lifeless> \o/
<StevenK> wgrant: However, even after calling that, iterating over me.ppas results in 2, not 3
<wgrant> StevenK: WFM :(
<wgrant> StevenK: You're not using a slave?
<StevenK> In the method? I use IMasterStore
<wgrant> I mean in .ppas.
<StevenK> wgrant: However, I'm perfectly willing to admit my code is wrong
<wgrant> I doubt it.
<StevenK> wgrant: I call .createPPA and then iterate over me.ppas
<wgrant> (both that you're willing to admit it, and that the code is wrong)
<StevenK> wgrant: I meant my QA script :-)
<wgrant> Ah.
<wgrant> paste paste
<StevenK> wgrant: And, keep laughing, I'm reloading
<StevenK> wgrant: http://pastebin.ubuntu.com/524936/
<wgrant> StevenK: You're still using launchpad.me...
<StevenK> Oh, I need to regrab it after .createPPA()?
<wgrant> No, *for* createPPA.
<StevenK> It's a person, isn't it?
<wgrant> It is.
<wgrant> But launchpad.me.createPPA() will post to something that redirects.
<wgrant> == bad idea.
<StevenK> Ah
<wgrant> HTTP is awesome.
<jml> morning all
<wgrant> Evening jml.
<lifeless> ola
<StevenK> wgrant: So I have a cowboy that adds IMasterStore, do you think that impacts?
<wgrant> StevenK: A write operation will always use the maste.r
<StevenK> wgrant: I mean this diff: http://pastebin.ubuntu.com/524926/
<wgrant> StevenK: Redundant.
<StevenK> wgrant: If you think that diff is pointless, then excellent
<StevenK> lifeless: It works, my qa script sucked
<wgrant> s/my qa script/launchpadlib/
<wgrant> s/launchpadlib/HTTP/
<StevenK> Haha
<wgrant> :( thumper doesn't appreciate Soyuz's acronyms :(
<StevenK> wgrant: Hm?
<wgrant> StevenK: In lifeless' MP.
<adeuring> good miorning
<jml> adeuring: good morning
<jml> wgrant: it's not clear to a relative newcomer how much complexity in soyuz is inherent and how much could be simplified.
<jml> wgrant: indeed, perhaps it's not clear to anyone :)
<wgrant> jml: Heh. True.
<stub> what is the recommended tool to view .subunit.gz files?
<jml> stub: testr, probably.
<jml> stub: there's also tribunal-subunit
<jml> stub: and the various subunit-*  commands
<stub> I was hoping for something that I could make run when I clicked on the attachment.
<jml> stub: tribunal-subunit
<mrevell> Good morning
<jml> mrevell: hello
<stub> jml: Is that hiding in a PPA somewhere?
<jml> stub: perhaps. I think it's overdue a release, tbh.
 * jml pokes around
<stub> As far as LP is concerned, it only exists as a branch
<jml> stub: yeah, that sounds about right.
<jml> stub: I'd suggest hassling poolie
<jml> or diy
<gmb> Is there any way that we can make PQM Not do this:
<gmb> 'Commit message [[rs=gmb][ui=none] The tests that needed to be combined because of\n\ttest isolation issues with feature flags have now been split up again.] does not match commit_re [(?# Strong suggestion: use ``bzr lp-land`` or ``ec2 land``.  Manual submission notes: your submission message needs [ui=...] AND at least one of [r=...] [rs=...] or [release-critical=...] AND at least one of [no-qa] [incr] [rollback=...] [bug=...].)(?is)(?=(\\s*\\[[
<gmb> ^\\]]+\\])*\\s*\\[(release-critical=[^\\]]+|rs?=[^\\]]+)\\])(?=(\\s*\\[[^\\]]+\\])*\\s*\\[ui=[^\\]]+\\])(?=(\\s*\\[[^\\]]+\\])*\\s*\\[(incr|no-qa|bugs?=\\d+(,\\s*\\d+)*|rollback=\\d+)\\])]'
<gmb> Cos that isn't helpful, at least when formatted that way.
<wgrant> That is a nice re.
<stub> Its like that to remind you to be thankful you are not a Perl programmer.
<jml> gmb: The only options that spring to mind are patching it or switching to a different tool
 * jml brb
<gmb> stub: Amen.
<stub> The regexp could get comments embedded in it perhaps... (?isx)
<gmb> jml: Ah, right.
 * gmb adds it to his list of things to try patching when he has a minute.
<wgrant> Or we could hope that Code finishes merge queues soon, and then switch to Tarmac :)
<stub> tribunal-subunit looks like what I'm after. 'setup.py install --user' worked fine for install
<jml> stub: cool.
<jml> stub: fwiw, I started making daily builds of a whole bunch of testing tools incl. tribunal. I'm still intending to actually make them work.
<jml> unfortunately my plane left on time :(
<jml> some may be interested in subscribing to https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/670289
<_mup_> Bug #670289: Laptop won't shut down with rabbitmq running <amd64> <apport-bug> <maverick> <rabbitmq-server (Ubuntu):New> <https://launchpad.net/bugs/670289>
<wgrant> Ah, so it's not just me.
<jml> wgrant: you might recall bigjools mentioning this on the list
<stub> Does the magic exist to only run previously failed tests (identified by a subunit stream of the previous run)
<stub> ?
<wgrant> That's what testr is for.
<jml> stub: 'testr run --failing'
<stub> Hmm.... tribunal-subunit identifies 89 failed tests, testr 91...
<stub> Need to generate a failing.list...
<wgrant> stub: 'testr load'
<stub> That doesn't seem useful for testr run --failing though - it seems to want a list of failed tests rather than extracting them from the testr database
<wgrant> Oh.
<wgrant> testr failing
<stub> "subunit-filter --no-passthrough < ~/Downloads/update-storm-r8330.subunit | subunit-ls" might be the ticket
<stub> Output from 'testr failing' seems to be doing something... no output but I've started swapping :)
<jml> 'testr failing --list' in trunk
<deryck> Morning, all.
<wgrant> jelmer: Did you end up ec2ing the branch? I heard nothing from it.
<jelmer> wgrant: Yes, but I didn't hear anything about it either.
<jelmer> wgrant: I'm working on a high priority thing at the moment, will have a look after that.
<jelmer> wgrant: Sorry it's taking this long.
<wgrant> Ah, yeah, forgot you were working on that.
<LPCIBot> Project devel build (180): STILL FAILING in 3 hr 55 min: https://hudson.wedontsleep.org/job/devel/180/
 * jml is being punched in the face by jetlag
<mrevell> deryck, Hi, is there a known problem with +manage-official-tags whereby the official tags list displays "{count}" beside some tags, rather than a number? I don't see a bug report, so will file one if it's new to you.
<deryck> mrevell, no, not known.  bdmurray did some work with allowing bug supervisors to edit official tags, but I don't think that would have caused that issue.
<mrevell> deryck, I'll report a bug
<deryck> mrevell, thanks!
<leonardr> mars, sorry to bug you about this, but do you have any idea what's going on with these ec2 test failures?
<leonardr> http://pastebin.ubuntu.com/525067/
<sinzui> leonardr, I have lost my mind. I added a new method to an interface that is a variation of an existing method. I cannot build my branch though: AssertionError: No IEntry adapter found for Interface (web service version: beta).
<sinzui> leonardr,  What do I need to register to solve the missing IEntry?
<leonardr> sinzui: let me see the diff?
<leonardr> i don't think you need to register anything. i think you prevented some generated code from being generated, or something like that
<jml> sinzui: perhaps one of the types used in the method sig is not exposed? alternatively, perhaps the circular import resolver isn't being used properly
<leonardr> yeah, i'm thinking along the same lines
<sinzui> leonardr, http://pastebin.ubuntu.com/525077/
<sinzui> jml: leonardr: there are no new objects. You can leave a team, but you cannot remove your team from a team. I added a method to do that ...
<leonardr> sinzui: is it possible that the type of ITeamMembership['team'] hasn't been defined at this point? that it's defined in circular_imports?
<jml> sinzui: I think you want to close paren after ITeamMembership['team']
<sinzui> leonardr, The 3 methods before this method are doing it. http://pastebin.ubuntu.com/525079/
<jml> sinzui: your parens are mismatched.
<sinzui> :(
<sinzui> thanks jml:
<jml> sinzui: np.
 * sinzui stabs editor
<jcsackett> sinzui: i find fire is really the best way to discipline your editor.
<mars> leonardr, looking
<mars> jml, future of the web sounds a bit like X windows :)
<mars> jml, and thank you for writing the synopsis.  I don't have the time to watch the video right now, and it nicely summarizes his argument
<mars> leonardr, I would guess your branch failures are because devel was broken yesterday.  My branch failed the same way.  Try merging from the latest devel and use ec2 land.
<leonardr> mars, ok
<mars> leonardr, actually, you should not have to update from devel, because of the way the ec2 merge works
<mars> I'm trying to pull a definitive set of failures
<leonardr> mars, kind of an embarrassing question, can you walk me through using ec2 land? (assuming there's anything special to it)? i'm still using pqm-submit manually because i never got over my fear of ec2 land when it was new
<mars> leonardr, just a minute
<mars> leonardr, no problem.  Does ec2 test work for you?
<leonardr> mars: yes, in general
<leonardr> (i assume you mean 'do you typically use it', and i do)
<mars> right
<mars> ok, good
<sinzui> leonardr, jml: sorry, I still get the same error after fixing the parens. http://pastebin.ubuntu.com/525096/
<jml> sinzui: what happens if you put call_with below operation_parameters?
 * sinzui tries
<mars> leonardr, ec2 land works just the same as ec2 test, except that it will check the merge proposal for the data it needs to commit
<leonardr> mars: so i set the commit message etc. there?
<mars> leonardr, to use ec2 land the merge proposal needs to be 'Approved'.  You can set the commit message on the MP, or using "ec2 land -s"
<sinzui> jml: same error
<leonardr> jml: there are lots of other cases where it's above @operation_paramters
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (181): FIXED in 3 hr 36 min: https://hudson.wedontsleep.org/job/devel/181/
<LPCIBot> * Launchpad Patch Queue Manager: [r=henninge][ui=none][bug=117460] Adds a help icon link beside the
<LPCIBot> "Also affects project" link on the bug page. Also adds a
<LPCIBot> modified version of Bryce's suggested explanatory text to the
<LPCIBot> +choose-affected-product page.
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][no-qa][incremental] unit tests for
<LPCIBot> BugTaskSet.search(): upstream status related filtering,
<LPCIBot> filtering by tags, by date_closed, by fulltext search,
<LPCIBot> by fast fullext search and by has_no_upstream_bugtask
<LPCIBot> * Launchpad Patch Queue Manager: [rs=gmb][ui=none][no-qa] The tests that needed to be combined because
<LPCIBot> of test isolation issues with feature flags have now been split
<LPCIBot> up again.
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][ui=none][no-qa][rollback=11825] Backout the cause of windmill failures.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][ui=none][bug=670081][no-qa] Delete the python component of the removed mentoring feature.
<jml> leonardr: not on my screen at the time :)
<lifeless> moin
 * mars is not accustomed to bot barf in the middle of conversations
<lifeless> jml: flacoste: Apologies for the TL meeting; taking Lynne to the airport.
<jml> lifeless: np
<flacoste> lifeless: no worries
<jml> StevenK: where do I file bugs against your IRC bot?
<lifeless> jml: launchpad itself.
<mars> leonardr, you do not need to add the [r=...] or any tags to the commit message.  ec2 land will do that for you
 * sinzui moved method higher in the interface
<lifeless> jml: its where we file things for bb after all; I'd assign to StevenK to get his attention.
<mars> leonardr, and you may want to set the QA status.  What kind of QA do you anticipate for this change?
<lifeless> oh yay, build fixed.
<lifeless> and, excellent. We have killed mentoring.
<mars> lifeless, ?
<leonardr> mars: the normal kind where i try a couple things out with the web browser
<lifeless> mars: the feature we removed from the UI is now gone from the code base. \o/.
<mars> leonardr, ok, ec2 land will assume 'qa-needstesting' on the linked bugs when you submit your change.  If there are no linked bugs, it will print a nice warning telling you how to set up QA properly.
<mars> lifeless, ah, awesome
<mars> leonardr, that's it.  Very simple.  You just set the commit message, consider your QA, and go.  It handles the rest.
<henninge> rockstar: ping
<jelmer> rockstar: hi
<henninge> ;-)
<shadeslayer> jam: any other news on my qtwebkit bug :D
<leonardr> mars, ok, thanks
<mars> leonardr, fwiw, the same flags are used for 'bzr lp-land'.  It will do the same as ec2 land, but without the ec2 run.  Since your failures look to be from a broken build, not your own work, you would be safe using that too.
<leonardr> mars: ok, two things that make me worry about using bzr lp-land here
<leonardr> 1. did i miss my chance to set a commit message on the merge proposal when i created the merge proposal? i don't see any way to add one now
<leonardr> 2. similarly, my mp is for merging into db-devel, not devel, because i made the same dumb mistake i always make
<leonardr> what do you think?
<leonardr> should i just try pqm-submit?
<m4n1sh> How do you control spam on Launchpad. Esp when some person/bot starts messing up with the bug reports?
<mars> leonardr, for the first one, you can edit the commit message on the MP at any time.
<m4n1sh> This is the person who is doing it right now
<m4n1sh> https://launchpad.net/~braulioareis
<mars> m4n1sh, sinzui can help you with that
<m4n1sh> these are the bu g reports he is fidding at http://paste.ubuntu.com/525113/
<m4n1sh> sinzui: ^^
<leonardr> oh, i see 'set commit message'
<leonardr> it's hard for me to use web apps because i subconsciously filter out a lot of things as 'advertising'
<sinzui> m4n1sh, ask a question so that the bugs team can investigate https://answers.launchpad.net/malone
<mars> leonardr, regarding the second - I don't actually know how to retarget a merge proposal like that.
<leonardr> mars: i usually have to delete and re-submit
<sinzui> m4n1sh, They can suspend a user and make arrangements to remove the spam from the comments
<leonardr> so this time, i'll just use pqm-submit as i usually do
<mars> leonardr, do you have to delete first?
<leonardr> but now i'm not afraid of ec2 land anymore
<mars> leonardr, or just resumit, and rs=mars
<leonardr> mars: let me try it
<leonardr> mars, it says "Another merge proposal will be created, with the same source, target and prerequisite branch (if any). "
<mars> leonardr, try hitting resubmit, send me the link to the MP
<leonardr> that's why i've been deleting and resubmitting, i think
<leonardr> but let's try it
<mars> leonardr, ah, hmm
<mars> abentley, ping
<leonardr> mars: yeah, it doesn't let me change that
<abentley> mars: pong
<mars> Hi abentley, leonardr and I have a merge proposal user question for you
<sinzui> m4n1sh, the user is not a spammer, he needs a talking too though about etiquette
<mars> abentley, leonardr accidentally set db-devel as his submission branch.  He now wants to merge his code to devel.  Unfortunately, the land commands read the original MP for instructions, so they will do the wrong thing.
<mars> abentley, resubmission does not work, because it does not let you change the merge target
<abentley> mars: he should use pqm-submit, then.
<mars> leonardr, ^ ah
<mars> abentley, ok, thank you
<abentley> mars, I have started a branch to allow changing the merge target with resubmit, but it's not ready yet.
<m4n1sh> sinzui: I just used the word spammer as he was changing random bug status without talking to anyone
<sinzui> m4n1sh, I suspect the user does not even know that he is generating emails
<m4n1sh> Or probably he was playing with Launchpad
<m4n1sh> sinzui: probably yes
<m4n1sh> atleast important bugs like https://bugs.launchpad.net/ubuntu/+source/libmodule-build-perl/+bug/661901 should be spared
<_mup_> Bug #661901: Needs re-promotion to Main <libmodule-build-perl (Ubuntu):New> <https://launchpad.net/bugs/661901>
<mars> abentley, nice, sounds like what I was hoping for
<lifeless> Ursinha: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html 404's?
<leonardr> mars: so, for real this time, using pqm-submit
<mars> leonardr, yes
<Ursinha> lifeless, https://devpad.canonical.com/~lpqateam/ 404's, losas are investigating
<lifeless> Ursinha: kk thanks
<leonardr> sinzui: sorry, coming back to you now
<sinzui> leonardr, np. I decided that I wont let this issue block me and set it aside. The answer many be more obvious after a break
<leonardr> sinzui: i suggest you put breakpoints in your lazr.restful declarations.py so that you can watch the named operation adapter being constructed
<sinzui> okay
<gary_poster> Anyone have any idea on what I should do with this? https://bugs.edge.launchpad.net/launchpad/+bug/670347
<gary_poster> For reference, Google translate gives the text as
<gary_poster> THANKS it youness stager of Fili from Morocco tmsri (mantenence and computer support and resalable I must pronder the operating system ebuntu plzzz
<gary_poster> application operating system ebuntu blzz
<gary_poster> Options in my head:
<gary_poster> - try assigning it to Ubuntu.  I feel kind of bad about doing that to Ubuntu because my other option is...
<gary_poster> - Mark it as Won't Fix and suggest that they use the questions application for Ubuntu
<gary_poster> and if they are serious they can do that, otherwise they will just disappear
<gary_poster> Actually, unless anyone stops me, I'm going to do #2
<leonardr> gary: fwiw, i'd give 3 to 2 odds they want ubuntu cds
<gary_poster> heh, maybe so leonardr .  I'll give that link too.
<lifeless> mrevell: how does one make a blog turn on up the home page? I want to broadly announce that edge is gone[ish]
<lifeless> deryck: after you fix text wrapping.... allowing reassign from launchpad to ubuntu would be THE MOST AWESOME CHANGE EVER
<jml> lifeless: can't wait for the real thing?
<lifeless> jml: coca cola?
<jml> lifeless: edge being fully gone
<deryck> lifeless, yeah, I agree.  Not sure we'll take that on anytime soon yet, though. :-)
<lifeless> -> airport
<lifeless> jml: users are confused now.
<jml> lifeless: Makes sense. I hadn't noticed.
<jml> mrevell: do we have a standard thing that we do for Launchpad jargon?
<lifeless> deryck: if we can make it a small change, would you do it ?
<deryck> lifeless, heck yeah
<deryck> so I assume it's not a small change.  I haven't looked in awhile at it
<mrevell> lifeless, Ah, that's superb. Are you on jetlag time at the moment? I was hoping to have a call with you about that sort of thing. For the front page, add the tag front-page to the post.
<abentley> rockstar: chat?
<mrevell> jml, Kinda. https://help,launchpad.net/Glossary is out of date and should probably go away but we could resurrect it.
<rockstar> abentley, sure.
<rockstar> abentley, mumble, eh?
<abentley> Sure.
<jml> mrevell: I meant in-app.
<jml> mrevell: but it is good to be reminded about the glossary :)
<mrevell> jml, I'm not sure what you mean. Do you have an example?
<jml> mrevell: e.g. one of the fields on the recipe creation page is "default distribution series"
<jml> mrevell: but what if I don't know what a distribution series is?
<rockstar> abentley, I'll be there in a sec.  Unpairing my headset and repairing seems to have confused mumble.
<abentley> rockstar: ah.
<mrevell> jml, Hmm. I can't think of anything particularly elegant that we can do with what we have now. A question mark icon beside the text linking to a help pop-up is what I'd opt for.
<jml> mrevell: ok, thanks.
<rockstar> abentley, you slowly faded off and now I can't hear you.
<abentley> rockstar: https://code.edge.launchpad.net/~rockstar/launchpad-code/mockups
 * jml off IRC for the evening
<jml> g'night all
<abentley> lifeless: Have you turned off edge redirection before turning on recipes for members of the beta team?
<lifeless> abentley: hi; yes edge redirection is off. The edge site still exists.
<lifeless> mrevell: hi
<mrevell> Hello lifeless
<lifeless> mrevell: I'm happy to do a call if you want
<mrevell> lifeless, I'm tied up for the remaining 30 mins of my day. Do you want to publish your post and then I'll email you if I have any questions. I want to mail beta testers, update the help wiki, etc, in response to the removal of edge.
<mrevell> I wasn't quite sure on the details but I reckon your post will clear that up.
<lifeless> mrevell: I'm not sure mailing beta users is a good idea.
<lifeless> mrevell: I agree that we should update the wiki and docs and so forth.
<mrevell> lifeless, No? I thought it would be a courtesy to explain the removal of the redirect.
<lifeless> mrevell: We change many things without directly contacting affected users.
<lifeless> What makes this change special enough to warrant a particular call-out ?
<LPCIBot> Project db-devel build (115): SUCCESS in 3 hr 55 min: https://hudson.wedontsleep.org/job/db-devel/115/
<mrevell> lifeless, True but we've always done things a little differently with the beta team and, when we moderated the team at least, we emailed each new member to explain that receiving infrequent email updates would be part of the deal.
<mrevell> Anyway, the blog and the wiki are my priorities so I'll wait on your blog post.
<lifeless> mrevell: so, I think many beta users were told 'join this group because XXX' rather than being really committed beta testers
<lifeless> mrevell: there are 2000 members.
<lifeless> mrevell: also there is a typo on the launchpad-beta-users team description
<mrevell> lifeless, I'm not sure that makes a difference but, certainly in the first two or three years of the team's existence, we advertised the team as being specifically for redirects to a beta version (initially for the new UI and then to get specific features that we emailed each team member about).
<mrevell> I'll fix the type.
<mrevell> heh
<mrevell> typo
<lifeless> lol
<lifeless> right, we advertised it for that
<lifeless> I think we should message this clearly
<lifeless> but the consistent feedback I've seen on email communications about our product is that less is more.
<lifeless> That is, that its better for us to make things smooth and painless, than to email.
<lifeless> YMMV
<mrevell> Yeah, no doubt, but I don't see how that ties up here. We have sent many emails to the beta team ... none in the past year, or so, admittedly. I recall no complaints. Either way, I don't want to get stuck in tangental conversation around this :) I'll drop you a mail/ping you when I've read your post ... thanks for writing that, btw.
<lifeless> mrevell: no probs
<lifeless> gary_poster: hi
<lifeless> gary_poster: how would you feel about making the deploy-latency graph high? I think its one of our key inner loops in a LEAN sense.
<lifeless> gary_poster: and thus worth measuring [and focusing on]
 * gary_poster tries to context switch
<lifeless> :) - sorry
<gary_poster> bug 668149
<_mup_> Bug #668149: graph deploy latency <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/668149>
<lifeless> yeah
<gary_poster> lifeless, as part of providing metrics for showing SMM, we will be gathering the amount of time it takes to go from an accepted MP to landing on stable (script is written now for review).  The amount of time going from an accepted MP to deployed is a similar measurement--and similar to what you have requested.
<gary_poster> Maris and Diogo have a lot of related measurements in mind for this kind of thing.
<gary_poster> I'm cool with saying "we'll work on some graphing of this stuff in December" and hammering out the details then.  We can even say that this equates to "High" :-)
<lifeless> gary_poster: what I'm specifically interested in is 'qaable to deployed'
<lifeless> gary_poster: which is deployed-on-qastaging through to deployed-on-prod
<gary_poster> yes, lifeless, by adding "when deployed" to our existing measurements, we would then have that.
<lifeless> ok
<lifeless> thank you
<gary_poster> np, will add summary to bug
<gary_poster> lifeless, re bug 670013, I currently believe that what Leonard decribed in comment 5 and I elaborated on in comment 6 is the best way forward identified so far.  It involves some dev work.  The only resource I have that might be available for this in Nov is stub, who would probably be very appropriate for it, and might have another idea on an approach.
<_mup_> Bug #670013: preflight check for removing edge cluster <Launchpad Foundations:Triaged by leonardr> <https://launchpad.net/bugs/670013>
<abentley> lifeless: You have effectively disabled recipes for people.  Do you plan to re-enable them?
<gary_poster> Didn't see your reply on the bug, lifeless; so, my comment # 7 is the elaboration.  That approach is not your client transfer, but sending the edge vhost to the production cluster, and having the application know how to handle it transparently.
<lifeless> abentley: how so?
 * gary_poster needs late lunch.  will be back in a while
<abentley> lifeless: they won't see it unless they specifically go to edge, which is a change in behaviour.  It even caught rockstar by surprise.
<lifeless> abentley: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/666538
<_mup_> Bug #666538: add team: feature scope selector <feature-flags> <Launchpad Foundations:Triaged by mbp> <https://launchpad.net/bugs/666538>
<lifeless> abentley: thats about top of my todo now, which will allow us to do gradual exposure on prod
<abentley> lifeless: I am glad that it's at the top, but I assumed it was a blocker for stopping edge redirection.
<lifeless> abentley: I assumed that folk already knew it only worked on edge
<lifeless> abentley: the bug is definitely a blocker for disabling or removing edge.. perhaps it should have been a blocker for removing the redirect
<abentley> lifeless: until now, I've always typed production urls and let it redirect me to edge.
<abentley> lifeless: for example, rockstar punched in a URL for one of his branches, then got worried because there was no "recipe" link.
<lifeless> abentley: I understand. This does speak positively towards consolidating on one UI with team (or other settings) controlled disclosure
<abentley> lifeless: I happened to navigate to an "edge" url, and then we argued about whether the link was gone for a minute until we figured it out.
<lifeless> abentley: we may have sequenced it wrongly. Rolling back will be tricky.
<lifeless> so, I'll whip up a team scope post haste.
<abentley> lifeless: Cool.  I wonder if we should post about this on the identica feed.
<lifeless> I'll do that
<abentley> lifeless: Great.  Thanks.
<abentley> lifeless: I don't understand "This does speak positively towards consolidating on one UI with team (or other settings) controlled disclosure".
<abentley> lifeless: you're talking about one UI with features optionally enabled depending on team (or other)?
<lifeless> yes
<abentley> lifeless: Yes, I think conditional feature flags are going to be very useful.  I can imagine more features being launched "in beta" like recipes.
<lifeless> gary_poster: I had replied on the bug before seeing your irc comment
<lifeless> gary_poster: we can stay on the bug, or irc, or both as you choose ;)
<lifeless> gary_poster: leonardr: I appreciate your rapid analysis of this.
<leonardr> np
<LPCIBot> Project devel build (182): SUCCESS in 3 hr 34 min: https://hudson.wedontsleep.org/job/devel/182/
<LPCIBot> * Launchpad Patch Queue Manager: [r=bac][ui=none][no-qa] Update cronscripts/expire-bugtasks to add
<LPCIBot> --ubuntu and --limit options,
<LPCIBot> which will allow batch runs of only a few Ubuntu bugtasks as we begin
<LPCIBot> to test run expiry.
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=657109] Do not snapshot PersonLocation.
<lifeless> StevenK: could you set the 'spam while failing or first success' rather than 'spam on completion' flag ?
<lifeless> flacoste: hey
<lifeless> flacoste: so, automatic releases. I really disagree. Vehemently
<flacoste> lifeless: that's a mistake in the notes
<flacoste> lifeless: what we talked about was actually not having to push the deployment from our side
<flacoste> lifeless: a kind of "manual" automatic deployement
<flacoste> lifeless: and the discussion actually needs to involve losa
<flacoste> i suggested something like having a schedule roll-out window where LOSA deploys regularly whatever deployment-stable says is ready for deployment
<lifeless> flacoste: I'd like to get devs more involved here rather than less.
<lifeless> flacoste: there will always be subtleties
<lifeless> flacoste: we should be deploying up to 10 revs a day tue-friday(am,nz-friday), and 30 on monday.
<flacoste> lifeless: care to justify the necessity of the dev / losa hands off here?
<flacoste> what are the subtleties that makes this necessary
<lifeless> flacoste: some angles:
<lifeless>  - 'devops' - its not a handoff, its a collaboration.
<lifeless>  - long term goal: realtime deployments, if something goes wrong the dev responsible is there to help.
<lifeless>  - accountability - its our job to ensure stuff gets deployed
<lifeless> flacoste: I realise we disagree here; I'd like to shelve this - and leave it shelved - until we've achieved the common elements:
<lifeless>  - high volume short duration low failure rate deploys
<lifeless>  - massive automation around the process [so that it is pushbutton for both dev+ops (its only pushbutton for ops today)]
<lifeless> flacoste: at which point we can do an assessment about the ongoing value of dev interaction there.
<lifeless> flacoste: right now, for instance, the last deploy, 3 back, and the next one all have bad metadata about broken revisions, so the process would stall if it was automated.
<lifeless> flacoste: by having dev accountability, theres no temptation to leave-it-to-the-robots
<flacoste> lifeless: i'm convinced about the value of on-demand deployments, i'm not advocating "leave-it-to-the-robot" roll-out anymore
<lifeless> flacoste: there may be a nuance I'm misunderstanding ;)
<flacoste> lifeless: what i'm not sure about is the value of why we need a different 'ask-a-losa-to-deploy' step that is different than the QA step which steps 'this-is-ready-to-deploy'
<flacoste> s/which steps/which says/
<flacoste> or actually says
<lifeless> flacoste: oh, right
<lifeless> flacoste: so, mthaddon doesn't want the losas overwhelmed with single-revision deployments
<lifeless> we need to iterate up to that.
<flacoste> right, i agree
<lifeless> devs can assess the importantance of a deploy. E.g. 'urgent' -> single rev is ok.
<lifeless> or routine - wait for a days worth.
<flacoste> that's why i'm proposing let's schedule one or two regular roll-out window, where losa deploys wathever 'new' is
<flacoste> so there is no request, but it's still run by a human
<flacoste> i agree that, it could be seen as less accountability from dev there
<flacoste> but it seems tenuous
<lifeless> flacoste: I suspect that Tom would rather a request, than be required to go look
<flacoste> i can also buy "assess the importance of a deploy"
<lifeless> flacoste: but I'm guessing there.
<flacoste> right, that's why the action was really 'Francis to discuss with LOSA and lifeless about this'
<flacoste> so I have your opinion, i'll discuss with IS in the next infrastructure meeting :-)
<lifeless> flacoste: ahh :)
<lifeless> flacoste: another angle is that I want to ratchet the frequency up as fast as we can sort toolchain issues.
<lifeless> flacoste: so I'd like to suggest that if we do make this a pull action from the losa side, that we do bug 668149 first
<_mup_> Bug #668149: graph deploy latency <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/668149>
<flacoste> lifeless: makes sense
<lifeless> its been so long since bb was happy
<lifeless> I'm worried we've landed *another* broken commit since 11825
<mars> lifeless, you can always run 'ec2 test -t', it will check devel for you
<lifeless> mars: the problem is bisecting, not how-to-do-it
<gary_poster> I thought https://launchpad.net/me/+reviewaccount would resolve.  Can someone remind me what the generic "go to your logged in account" URL is?
 * gary_poster tries help.launchpad.net
<gary_poster> it is /people/+me I think
<gary_poster> yup
<lifeless> OOPS-1768F2005 - thumper
<lifeless> https://code.launchpad.net/launchpad
<lifeless> looks like __traceback_info__: (<RevisionAuthor at 0x153770d0>, 'person') is triggering DB access
<lifeless> ok, back in a couple hours; time for my allergy injection
<leonardr> matsubara, do you have some time to talk about your use of launchpadlib? gary says you're having some problems that fit in with the work we're trying to do
<gary_poster> matsubara, I was thinking of the XXX you had where you wanted to get certain bugs/revisions/somethings in that stats script you were just working on
<gary_poster> and instead you had to iterate through all somethings
<matsubara> leonardr, gary_poster: right, it's the getMergeProposals() method which doesn't accept a date parameter
<gary_poster> I think it is bug 626680 or something similar
<_mup_> Bug #626680: iteration in LP API's is O(N^2) due to batching <api> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/626680>
 * gary_poster will get out of way now :-)
<leonardr> matsubara: so the problem is the lack of a specific filter on a given dataset, right?
<matsubara> leonardr, yep. it's similar to a bug recently fixed in the Bugs API. let me search that one
<matsubara> leonardr, https://bugs.launchpad.net/malone/+bug/327688
<_mup_> Bug #327688: It should be possible to search bugs given a range of date_created using the API <api> <qa-ok> <ubuntu-qa> <Launchpad Bugs:Fix Committed by brian-murray> <https://launchpad.net/bugs/327688>
<matsubara> leonardr, the script I recently worked on is using the getMergeProposals() method but it's not possible to filter the set by a date range or something. so I end up with all the set and have to filter in python by iteration over everything
<leonardr> matsubara: thanks
<leonardr> gary: although i agree with all the things you want to do on this project, i don't see think streamlining the web service will make this kind of performance improvement drop out. we'd still have to do work to make it possible to filter a set of merge proposals by a range for date_created (or whatever)
<leonardr> and then the question is, if our overriding priority is performance, why not just add that to the existing named operations?
<leonardr> i think the answer is "because that would make the web service incomprehensible and random, we need to have a plan"
<leonardr> and the streamlining gives us a plan
<gary_poster> exactly, leonardr
<leonardr> ok, then we're in agreement
<thumper> lifeless: that oops isn't found
<thumper> lifeless: what's it all about?
<jml> benji: *hugs*
<thumper> james_w: ping
<benji> :)
<james_w> hi thumper
<thumper> james_w: I have a couple of questions about your email re blueprints
<thumper> james_w: what do you mean by "read only view"
<thumper> ?
<james_w> thumper, can I edit e.g. the status of blueprints in the view
<thumper> do you think you should be able to?
<james_w> if it's a page for team leads to see workload, can they manage that workload from the same place?
 * benji is assuming bug 539070 is the source of aformentioned hugs.  If not, he may not want to know. ;)
<_mup_> Bug #539070: Unhelpful error on badly declared API export <Launchpad Foundations:In Progress by benji> <https://launchpad.net/bugs/539070>
<thumper> james_w: I think first cut it will be read only
<thumper> james_w: the blueprint code needs some more yak shaving to allow api exporting
<james_w> thumper, I do, but editing in list views is not a pervasive pattern in LP unfortunately, so I don't know how feasible it is
<thumper> james_w: which is what we'd really need for editing the status of the blueprints from there
<thumper> james_w: it is feasible (I think) if we have the api supported properly
<james_w> thumper, I'm intimately acquainted with that tak
<james_w> yak
<thumper> james_w: however I'm not entirely sure and the scalability of the widgets there
<thumper> james_w: I have started shaving the herd of yaks called blueprints
<james_w> thumper, you're working on the required model changes?
<james_w> (to export)
<thumper> james_w: I will be
<thumper> but this lep isn't about the api exporting
<thumper> that should just come from some of the yak shaving
<thumper> I think that we won't be able to filter by series/milestone if just looking on ~team page
<thumper> however that should be there when drilled down into pillar or series
<james_w> perhaps there could be a date filter at that level?
<james_w> based on the dates on the milestones etc?
<thumper> hmm...
<thumper> that seems like it is trying to be tricky
 * thumper wonders if we are going to rename Blueprints back to Specifications at some stage
 * thumper quietly ignores the plans to kill blueprints altogether
<james_w> it's just that I don't want to have to consult 5 pages and mash them together in my head to work out how much load is on my team in the next N months
<thumper> james_w: I think this idea is worth exploring, but we'd have to nut out how to filter effectively across all blueprints
<james_w> thumper, indeed, but I'm playing the role of a customer with demanding requirements :-)
<thumper> james_w: good, I'm happy with that
<thumper> james_w: an interesting question is how to ignore certain blueprints
<thumper> james_w: I'm sure flacoste doesn't care about my wikkid blueprints, even if they overlap in time
<thumper> james_w: I was wondering if project associations might solve part of that problem in a better way
<james_w> thumper, that's true
<thumper> james_w: but that still leaves you open to missing something
<thumper> as in not seeing a blueprint on a project you haven't currently marked as interesting
<thumper> sinzui: we now have a compelling reason for project associations
<sinzui> linaro had a pretty strong argument at UDS
<thumper> sinzui: do it!
<thumper> sinzui: I'd even be prepared to chip in
<sinzui> I will just before I delete projectgroups...and make everyone who working on converting projects to project groups weep at the wasted effort
<thumper> james_w: I'll be making some mockups for the pages anyway
<thumper> james_w: so we'll have something to discuss
<james_w> thumper, great, thanks for your work on this
<thumper> sinzui: we don't want to delete project groups, just change how they work
<thumper> sinzui: as you've said before they are for "control"
<sinzui> thumper, I think I want to. I think nested projects solve OEMs 500 skus. It also lets me have bugs and questions on the top level thing (/launchpad for example)
<thumper> ah...
<thumper> nested projects could be interesting...
<thumper> interesting in a way where queries are a PITA
<sinzui> I think making OEMs job easier will make Canonical more money and give me job security
<leonardr> matsubara, for the record, what kind of object are you calling getMergeProposals on?
<thumper> hah
<thumper> I just found something really useful in quassel
<thumper> if I double click in the chat monitor window, it changes me to that channel
<jcsackett> thumper: were you just trying to contact registry in mumble, or was that a mistake?
<thumper> jcsackett: I was checking that my mic and sound was working
<thumper> jcsackett: did it work?
<jcsackett> thumper: it did; i was in the other room and rushed back but you had already moved to code.
<sinzui> were you checking for heavy breathing?
<thumper> seems to be working now
<jcsackett> people always message me when i'm making coffee... :-P
<cr3> leonardr, rockstar: regarding the web_link that will be added toDataStructure in the lazr.restful EntryResource, I suppose, shouldn't it be named web_url to avoid using the _link suffix which has a special meaning?
<leonardr> cr3: i think it partakes of the same special meaning as _link (it's the web version of self_link). do you think it will actually cause problems?
<james_w> would accessing obj.web is launchpadlib cause it to attempt to get a resource from that url?
<cr3> leonardr: I don't quite see how web_url is special though, other than it being generated automatically, because _link attributes are used for resolving objects or collections thereof wherewas web_link (or web_url) is just a string with no such resolving
<leonardr> cr3: seems like GET would resolve it pretty well
<cr3> leonardr: touche, works for me then :)
<wallyworld> abentley: rockstar: so you guys will tell me if you need anything done?
<abentley> wallyworld: Sure, but probably rockstar or thumper will do more of that than me.
<thumper> wallyworld: I'll look over the bugs and we can chat later
<wallyworld> abentley: np. just wanted to make sure :-)
<wallyworld> ok
<rockstar> wallyworld, yeah, we can find some stuff.
<wallyworld> you only have 16 days left
<rockstar> wallyworld, me?
<wallyworld> no i mean you only have 16 days before thumper gets rid of me
<cr3> leonardr: I have to admit that one of my concerns was that naming the attribute web_link would require changes to lazr.restfulclient and even client.js, but that's not a real concern, more laziness :)
<leonardr> cr3: james_w has a point. we do need to see if it causes problems, and if so either change the client or change the name
<leonardr> i don't think we'd need to change client.js, but can you see what happens with lazr.restfulclient?
<cr3> leonardr: yes, and it took a moment for me to grasp what james_w meant by obj.web, where the _link suffix gets stripped and treated specially
<cr3> leonardr: in that respect, I don't think there's much value in GETing a web_link. as far as api goes, I think that attribute is most likely to be used as a string than a link to be gotten
<leonardr> cr3: i don't feel strongly about this. rockstar might have some wisdom from his attempt, and i'd also like benji to weigh in
 * benji reads
<cr3> leonardr: i don't feel strongly either, mostly sharing gut feeling. I'll be attempting to add web_* myself during my personal time, so I'll share my finding with rockstar if I find a reasonable solution
<leonardr> cr3: my gut feeling is siding towards you, but i think it would be nice if it were clear that self_link and web_link were two halves of the same coin
<benji> and it'd be nice if accessing .web generated an AttibuteError just as .doesnotexist would, but I don't think that's possible with the current mechanisms
<cr3> leonardr: if web_link == re.sub(r"/api/[^/]+/", "/", self_link), I would agree but I can't conceive it web_link being different from self_link in that respect
<leonardr> another possibility, which i'm not seriously suggesting, is that .web could actually get you the web page
<leonardr> in a mechanize Browser object or something
<leonardr> (again, this is just on a conceptual level,. *not* suggesting it, though it might be cool)
<cr3> leonardr: personally, I wouldn't be inclined to do that, but I wouldn't be surprised of people using that feature in ways I could not imagine :)
<leonardr> cr3, yeah, my point is, urls are urls, there's no fundamental reason why we can't have a tool that dereferences web service urls and non-web-service urls
<LPCIBot> Project devel build (183): SUCCESS in 3 hr 32 min: https://hudson.wedontsleep.org/job/devel/183/
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=654639,
<LPCIBot> 667883] Remove some Python 2.5 compatibility code. Fold edge database
<LPCIBot> stats into lpnet database stats.
<LPCIBot> * Launchpad Patch Queue Manager: [r=rockstar][ui=none][bug=667342] Bug filing form link will now fill
<LPCIBot> in summary and description for Trac external bug trackers.
<wgrant> Is buildbot happy after 11825's reversion?
<mars> checking
<mars> yes, lp db_lp are green
<wgrant> Excellent.
<wgrant> And Hudson is too.
<thumper> I find myself wanting bugs.launchpad.net/~person/project to go to a listing of bugs assigned to person for the project
<thumper> rockstar: ping
<persia> thumper, Could we also have /project/~person/ ?  That's how I always want to type it.
<thumper> persia: no
<thumper> at least I won't do that one
<persia> Heh, OK :)
<rockstar> thumper, pong
<thumper> rockstar: these bugs for recipes, are you doing them all in one branch?
<thumper> rockstar: if so please add one task to the kanban board for them
<rockstar> thumper, yes, because they are all small.
<rockstar> thumper, ah, yeah, kanban...
<thumper> :)
<rockstar> Huh. So, running ec2 land --attached doesn't actually land anything...
<rockstar> Also, it doesn't actually send any email.
#launchpad-dev 2010-11-04
<wallyworld> thumper: you have an answer for that bzr packaging question?
<maxb> bzr packaging question?
<wallyworld> maxb: +
<wallyworld> maxb: the version of bzr shipped with lp is 2.2.0
<thumper> sorry, just back from lunch
<maxb> hmm. probably time to get to 2.2.1
<wallyworld> we need to upgrade to either 2.2.1 or 2.2.2 and there's a timing issue that thumper needs to have input on
<maxb> Isn't there a 2a conversion bug fix we really really want in 2.2.1?
<thumper> wallyworld: lets get 2.2.1 in now
<wallyworld> yes. but there's also a spin wait issue fixed in 2.2.2
<thumper> we can get 2.2.2 when it is released
<wallyworld> ok 2.2.1 it is
<lifeless> back
<lifeless> thumper: the oops was new, it should have synced now
<lifeless> leonardr: A thought on web performance
<lifeless> leonardr: if its easy to use an API badly, people will use it badly.
<lifeless> leonardr: group based entry and exit signatures for high latency RPC is a way to make it easy to use an API efficiently and hard to use it poorly.
<leonardr> lifeless, can you spell out "group based entry and exit signatures for high latency RPC"? are you talking about setting up batch operations and callbacks?
<thumper> lifeless: oops still not matching
<lifeless> thumper: :(
<lifeless> leonardr: batch operations is one way of thinking about it
<lifeless> leonardr: less-than-one-roundtrip per object returned from lookups, a powerful query language, changes to multiple objects in <3 round trips - that sort of thing.
<lifeless> leonardr: but also simply moving away from 'an object is retrieved by reading a location' as an idiom.
<leonardr> lifeless: do locations now designate sets of objects, or something else?
<lifeless> leonardr: Perhaps API functions. I don't know - I was replying to your comment about streamlining being orthogonalish to making performance fast
<lifeless> leonardr: the key points I wanted to note are:
<lifeless>  - if its easy to use in a poor way, people will (and then blame us)
<lifeless>  - for high latency environments (e.g. SQL, let alone internet ops) operations on group with decomposition for single cases work better than operations on atoms with composition to talk about many things.
<lifeless> leonardr: I'd be delighted to help frame the problem, satisfying criteria, and brainstorm with you on what changes we should do.
<wallyworld> thumper: ignore that mp email - typo in branch name. will redo
<lifeless> leonardr: I don't want to suggest specific solutions [at least, not without really focusing on it - its nontrivial]
<thumper> wallyworld: ack
<leonardr> lifeless: sure. my main concern is that although "difficult to use poorly" is possible, it doesn't go very well with "easy to use on an absolute level"
<lifeless> leonardr: so, 'easy to use in a slow way' is different to 'easy to use in a fast way'
<leonardr> i can imagine systems like the one you're imagining, but i see users regarding them as further from their expected "web api" experience than what we have now
<lifeless> leonardr: the slow way case is something to avoid, because users are dissatisified and we pay more.
<leonardr> i'm certainly not arguing *for* anything being slow, but i think the price of making it really easy to get started may be that the thing you create when you get started is slow
<lifeless> leonardr: All - as far as I know without exception - of our current users complain about performance. And the usability angle seems, to me, to be tied into OAuth (which we need) as well as documentation (which is IMO poor compared to (for instance) the AWS web API docs)
<lifeless> leonardr: now, much of the performance issues may be request completion times, not latency - we don't really know [yet].
<leonardr> maybe by making the web service uniform we can "free up" some complexity that we can then use on a more complex, but faster, usage model
<lifeless> leonardr: and thats why I'm not personally planning on focusing on the API until we have a 1 second 99% completion time with no pages that violate that.
<lifeless> leonardr: thats an interesting angle
<leonardr> lifeless: that kind of bank shot is the only way i think that usability has much to do with performance right now
<leonardr> i think we're in a good position to massively improve usability without affecting performance at all, while making it easier for us to optimize specific cases
<leonardr> but as i told gary, i don't see any deep connection
<lifeless> when you talk usability, I think object serialisation, URL routing and documentation.
<lifeless> What are you thinking about when you talk usability ?
<leonardr> lifeless: same kind of thing
<leonardr> especially documentation
<lifeless> so, I guess I'd say that right now.
<lifeless> We should - based on jml's strategic goals - just focus on performance.
<lifeless> for the API
<lifeless> (riffing)
<leonardr> lifeless: that's legitimate, but my gut feeling is that we'll either be not changing the framework at all, or adding complexity
<lifeless> there are I think two broad aspects to API performance
<lifeless> *) implementation of specific calls
<lifeless> *) structure of the API
<lifeless> in the current structure the implementation of specific calls is multiplied by latency because we have a chatty model
<lifeless> so we can improve performance in two broad ways: address the structural issue, at which point specific call implementations will have a diminished but still significant impact (less calls), or we can polish the implementation of specific calls up, so that we can accurately assess the
<lifeless> overall balance between latency and service time
<leonardr> lifeless: my general belief is that the current model, chatty as it is, is the one that makes most sense to python programmers because it comes the closest to letting them pretend they're working with local python objects (this was our original goal)
<leonardr> obviously, pretending that the network boundary isn't there is going to cause problems
<lifeless> leonardr: We chatted about this on-list.
<lifeless> leonardr: I'm of the considered opinion that that goal is a problem.
<leonardr> that's fine, but there's a price to be paid: the alternate system will only be "easy to use well" in the sense that it's easier to use well than poorly
<lifeless> leonardr: our us of an ORM in launchpad with similar goals (make it look like regular POPO code) is a direct cause of the performance quagmire we're in.
<lifeless> leonardr: ec2's API has /massive/ adoption and it has paid that price.
<spiv> FWIW, I share the view that pretending that remote objects are like local objects isn't good, because they aren't in terms of performance, concurrent users, connection reliability, etc.
<lifeless> apples and oranges to be sure.
<leonardr> i'm fine with the direction this is going but i feel very uncomfortable putting the "easy" description anywhere near it
<lifeless> sure
<lifeless> strip that out, its pretty subjective.
<leonardr> ok
<lifeless> vision: the default way of using Launchpad APIs will be:
<lifeless>  - fast
<lifeless>  - correct
<lifeless>  - obvious
<spiv> I don't think lifeless meant "easy" in the sense of a DWIM API, just that the APIs should be high on the rusty scale.
<lifeless> right
<lifeless> 10 please
<leonardr> spiv: does not compute, is rusty a person or an adjective?
<spiv> http://sourcefrog.net/weblog/software/aesthetics/interface-levels.html http://ozlabs.org/~rusty/index.cgi/tech/2008-03-30.html http://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html
<spiv> (I like the first link's pithy one-page summary, but the second and third are from Rusty himself and go into a little more detail)
<leonardr> ok
<spiv> (And add a couple more levels?)
<leonardr> "getting it wrong" here meaning "taking a long time" or "requiring a lot of round trips"
<lifeless> three things I think
<lifeless> the two you specify
<lifeless> and thirdly doing something not obvious right - like squashing someone elses change to a bug without warning, adding a comment to the wrong bug, etc.
<leonardr> lifeless: if i may sum up my pov from a planning perspective, if we are going to go ahead and do a totally new web service, it's worth investigating a this-ish way to do it. but this morning i heard that our work wasn't to be coupled with the internal api work -- not that things can't change within a day
<lifeless> I know you've been extremely careful about that, I add it here so that when we talk with *other people* there is no confusion
<leonardr> ok
<lifeless> actually and a fourth thing
<leonardr> what we're talking about is a new web service, possibly with a new custom client, possibly not. if we leave the old web service intact, people will use it instead and get bad performance
<lifeless> the 'obvious' that I mentioned a page or so up.
<leonardr> yeah, i meant to click on that to expand
<lifeless> Even if its not like POPO, it should be obvious how to achieve something.
<lifeless> (plain old python objects)
<leonardr> lifeless: so, like the way that mapreduce is obvious?
<lifeless> leonardr: rotfl
<lifeless> uhm
 * leonardr semi-serious
<leonardr> we give you two simple concepts that may not be what you've encountered before, and the system is built from those, and it's incredibly scalable
<lifeless> here is a definition of obvious that I like: Once I've read the overview docs and played with a hello-world equivalent, I should be able to guess at the right place to look for the details on achieving something.
<lifeless> leonardr: yes, in that regard mapreduce very much meets the definition of obvious I'm giving.
<lifeless> leonardr: finding good axioms (like mapreduce) can be a great way to have a very clean very fast system
<leonardr> lifeless: in that case i'd like to reiterate that we may have to deliberately break backwards compatibility to stop tempting users with popos
<lifeless> leonardr: so, separation of concerns here
<lifeless> *) We need a revamped API that /can/ be fast, correct, obvious all at once
<lifeless> *) We need to transition users to this revamped API
<leonardr> ok
<lifeless> *) We need to keep supporting the older API in *some* fashion for *some time*
<lifeless> *) We need to keep meeting the web UI's ajax needs.
<lifeless>    (This may be a non-API problem. Or it might be something we do simultaneously. It would be nice to keep the current 2-birds 1-stone approach)
<lifeless> leonardr: on the in-process API stuff
<leonardr> lifeless: after what i've seen this year, if we have a specialized ajax service, i would not put it past our users to hack into it just so they can keep using pops
<leonardr> popos
<lifeless> leonardr: I wouldn't couple them together. The in-process stuff has a much lower churn cost and will involve experimentation. Lots of it.
<lifeless> leonardr: I think the revamping of APIs needs room to experiment and breath too
<lifeless> leonardr: but there's no reason why it should try the same things at the same time.
<lifeless> leonardr: if we decide later on that the things are just so darn similar, we can focus on harmonising them at that point.
<leonardr> lifeless: see if this interests you
<leonardr> let's stipulate that right now the web service is a big bushy mess
<lifeless> so stipulated
<leonardr> the thing i planned out last year, but haven't been able to tackle just yet, will comb its hair and make it look presentable, without actually getting rid of any of the hair
<leonardr> but, once the hair is in the appropriate place (and this is the place where the analogy falls apart), it may resemble something within spitting distance of a "sets and ways of operating on sets" type idea
<lifeless> ok
<lifeless> so I've no particular preconception about implementation or route-to-get-there.
<lifeless> improvements to what we have are still improvements.
<leonardr> at that point we may be able to go from 100,000 hairs all perfectly arranged to a single smooth piece of plastic (to bring back the analogy despite it still not working)
<leonardr> as i get back into the stuff i wrote last year, i will keep this alternate design in mind, and see whether we can really get there incrementally
<lifeless> awesome
<wallyworld> spiv: that bzr 2.2.1 upgrade has been approved and is currently being landed via ec2
<wallyworld> lifeless: with the reference to tempting people use popos etc  *by design*, given the impedance mismatch between layers and the distributed computing implications and the resulting performance/robustness issues etc, i'm happy we seem to be moving to a model where it's recognised that may no longer be the best approach :-)
 * thumper relocates
<wgrant> We have an SSO OOPS in #launchpad. Where should the user be sent?
 * mwhudson is tempted to say "coventry"
<wgrant> I wonder if they've revived the support link...
 * wgrant goes hunting.
<wgrant> Aha, it is back.
<spiv> wallyworld: yay, thanks!
<wallyworld> spiv: np. not sure how long it will be held up in ec2 :-)
<LPCIBot> Project devel build (184): SUCCESS in 3 hr 34 min: https://hudson.wedontsleep.org/job/devel/184/
<LPCIBot> * Launchpad Patch Queue Manager: [r=deryck][ui=none][bug=670352] Prevent {count} from showing up on
<LPCIBot> official bug tags management page when item.count is undefined.
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=henninge][bug=636000] Wraps configuration links in
<LPCIBot> the product involvement portlet with a collapsible that starts
<LPCIBot> collapsed when all configuration steps have been taken.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml,
<LPCIBot> mbp][ui=none][bug=645768][no-qa] Officially added a new test fixture
<LPCIBot> to help with the use of feature flags.
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=none][no-qa] Integrate the examples added to the
<LPCIBot> launchpadlib docs by https://launchpad.net/arsenal
<LPCIBot> * Launchpad Patch Queue Manager: [r=gmb][ui=none][bug=664571] It's now possible to update an existing
<LPCIBot> BugSubscription to have a new BugNotificationLevel.
<LPCIBot> * Launchpad Patch Queue Manager: [r=mars][ui=none][bug=532055] Fix grammatical errors that show up
<LPCIBot> when integrating a desktop with the web service.
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=664096] Allow the bug reporter to transition
<LPCIBot> away from Fix Released bug task status.
<lifeless> StevenK: yo
<wgrant> jelmer: Thanks.
<lifeless> wallyworld: indeed.
<cody-somerville> holy cow. how many times is the code page for projects going to change? lmao
<lifeless> wallyworld: I think you can write tasteful explicit popo stuff, but its often a bad idea ;)
<lifeless> cody-somerville: hmm ?
<cody-somerville> lifeless, the link to pending reviews for a project seems to have moved several times in the last few weeks on the code.launchpad.net/$PROJECT page.
<StevenK> lifeless: Hm?
<lifeless> StevenK: can you please change hudson to not publish subsequent successes
<StevenK> lifeless: I've changed it to 'failure and fixed'
<lifeless> thank you!
<thumper> cody-somerville: yeah... bit of a PITA IMO
<thumper> wallyworld: ping
<wallyworld> thumper: yo
<thumper> wallyworld: I vaguely remember you using a method in a test that got notification messages out of the response object
<thumper> wallyworld: do you remember it?
<wallyworld> makeQuestion
<wallyworld> um, no not that
<wallyworld> you mean for the branch alias stuff?
<thumper> yeah
<wallyworld> it's in test_launchpad, near the top from memory. i'll have a look
<wallyworld> thumper: lib/canonical/launchpad/browser/tests/test_launchpad.py, there's a _validateNotificationContext() method
<wallyworld> i think this is what you are after?
<thumper> wallyworld: ta, and yeah, I think so
 * thumper shaves a yak's butt
<wallyworld> thumper: careful of the dags. they're a lot bigger than on sheep
 * thumper ducks the dags
<lifeless> ewww
<lifeless> boom- lp.soyuz.windmill.tests.test_ppainlineedit.TestPPAInlineEditing
<StevenK> lifeless: And here I was about to tell you that db-devel is at 100% and devel is at 80% in Hudson
<wgrant> Hm, the QA bot just de-qa-ok'd a whole lot of bugs.
<lifeless> le sigh. Yes. We just moved to a new devpad
<lifeless> 100GB more disk and more memory.
<wgrant> And more flakiness?
<lifeless> nah
<lifeless> but we had some migration friction
 * StevenK smacks the QA tagger
<lifeless> wgrant: in fact, some of its changes are totally right
<lifeless> multiple landings linked to the same bug
<wgrant> Hmm.
<lifeless> whats the magic duck to wave to get 'current user' in lp code ?
<wgrant> getUtility(ILaunchBag).user or something like that.
<wgrant> But if you do that, you will be haunted forever.
 * lifeless prepares to be haunted forever
<lifeless> actually, its a participation lookup I need
<lifeless> interaction.get_current_principal
<wgrant> You could do that.
<wgrant> But that's used less than LaunchBag.
<lifeless> OTOH its correct.
<wgrant> LaunchBag is not?
<wgrant> It's ugly, but not incorrect TTBOMK
<lifeless> its redundant
<lifeless> that permits skew
<lifeless> skew leads to bugs
<lifeless> use the anger
<lifeless> wgrant: although arguably in this case its ok.
 * lifeless tosses and turns
<lifeless> StevenK: ping?
<StevenK> lifeless: Hm?
<lifeless> StevenK: spm was waiting to help you with qa
<lifeless> StevenK: you disappeared without saying anything
<lifeless> StevenK: so we were trying to get ahold o fyou
<wgrant> I wonder if PQM can get into testfix in the next half hour.
<wgrant> If it doesn't, I'll be having a branch land first time :(
<wgrant> Grar.
<wgrant> Indeed it did not land the first time.
<wgrant> It was submitted.
<wgrant> But nothing.
<lifeless> dinner time
<wgrant> jelmer: What did PQM whinge about this time?
<lifeless> does anyone know what sends mail for merge proposals ?
<lifeless> looks like merge-proposal-jobs
<wgrant> lifeless: I believe that's the case.
<wgrant> lifeless: Could you check the PQM history to see what happened to my branch?
<lifeless> nom nom nom
<wgrant> Clearly.
<lifeless> jml: https://code.launchpad.net/~lifeless/launchpad/recipes/+merge/40050
 * lifeless steps away
<mrevell> Howdy
<adeuring> good morning
<wgrant> jelmer: Ah, yay :(
<wgrant> Yet Hudson remains happy.
<deryck> Morning, all.
<mars> morning
<mars> StevenK, thanks for setting my MP to 'Work in Progress'.  I forgot you should do that for larger needs-fixing branches.
<jml> flacoste: hey
<flacoste> hi jml
<jml> flacoste: I was told recently that we don't have morphing dialog widgets yet
<flacoste> we don't
<jml> flacoste: I'm surprised
<flacoste> why?
<jml> flacoste: they were a very hot topic two years ago
<flacoste> they were
<jml> flacoste: why didn't we do them?
<flacoste> because we never reach a point that we had a workflow that needed them
<flacoste> we actually have now
<flacoste> we have a wizard widget
<flacoste> which is going to see some use
<flacoste> that widget should have the morphin behavior
<jml> we have a wizard widget?
<flacoste> if it doesn't have already
<flacoste> it's in lazr-js
<jml> who's planning on using that?
<flacoste> and deryck has a branch adding it to launchpad
<flacoste> rockstar had a use for it
<flacoste> and I think we are using it for the new subscription stuff
<jml> tbh, I don't care about the morphing part. I care about usability & consistency w/ the rest of Canonical web stuff.
<jml> flacoste: cool.
<jml> flacoste: thanks. that helps :)
<rockstar> jml, if only we could get a new lazr-js landed...
<flacoste> anything else in canonical does morhping yet?
<flacoste> rockstar: does your wizard widget do morphing?
<rockstar> flacoste, no.
<rockstar> flacoste, I was going to do that after the fact.
<rockstar> flacoste, but the fact that it's been pretty difficult just to get this lazr-js upgrade in really killed momentum there.
<jml> rockstar: I have every confidence in you
<jml> rockstar: you can do it.
<rockstar> jml, I know I CAN.  It's whether or not I have the time.  :)
<jml> rockstar: do you have anything more important to do than make recipes rock?
<rockstar> jml, I suspect my rotation into U1 will provide a bit more time for lazr-js hackery, since I'll be doing mostly js work anyway.
<flacoste> lunch time
<jml> danilo: where are we at with importing upstream translations?
<danilos> jml, importing them into LP projects works, we are finishing automatic import into Ubuntu from there
<jml> danilos: and once they are imported into Ubuntu, they'll be available as (specially marked?) suggestions for Ubuntu translation?
<danilos> jml, no, they are already available as suggestions, but we don't really want to set them up all because migrating is going to be harder later
<danilos> jml, they will be automatically used in Ubuntu
<jml> danilos: " migrating is going to be harder later"?
<danilos> jml, well, in order for us to scale, we're actually going to be using the same table rows for both upstream and Ubuntu translations (fwiw, more than 85% of them are identical)
<danilos> jml, so, if we import all upstreams now, we'd have to migrate when our current feature work gets rolled out
<danilos> jml, and doing any live migration is slow and painful, and even though we'll have to do some, we want it to be minimal
<jml> danilos: hmm. I think I understand. Mind if I try to say the same thing in my own words?
<danilos> jml, not at all :)
<jml> danilos: we can import translations into LP projects today;
<jml> danilos: we have WIP right now to automatically import those into Ubuntu, making those imported translations the actual translations for Ubuntu
<jml> danilos: we also need to do some data entry to set up imports for all of the interesting projects
<danilos> jml, that's right
<jml> danilos: but we aren't going to do that yet because we need to do some data migration first, in order to be even remotely scalable
<jml> is that it?
<danilos> jml, that final thing you said is the work that is ahead of us as well
<danilos> jml, well, one before final ("do some data entry")
<danilos> jml, as for the final thing, yes
<jml> danilos: I'm not sure what you mean by that. Do you mean there's also current programming work involved in doing it?
<danilos> jml, how about a quick call
<jml> danilos: ok :)
<danilos> jml, mumble has not really worked for me today, but let's try again :)
<lifeless> morning
<jml> lifeless: hi
<lifeless> oh hai
<jml> danilos: we have exports already, right?
<lifeless> jml: when are we talking
<lifeless> flacoste: ^
<jml> lifeless: in 55mins time
<danilos> jml, we have some exports targeted at upstreams, but it's not really what'd we call really a feature as part of seamless process if anyone wanted to use it
<jml> danilos: thanks.
<allenap> Does anyone remember how to deal with community contributions to lazr code?
<allenap> It's a long time since I've done it.
<jml> allenap: poke them in the eye, take their money and run for the hills?
<jml> allenap: hmm, maybe I'm thinking of something else.
<jml> allenap: what makes it different from dealing w/ community contributions to LP?
<allenap> jml: That'll do though :)
<allenap> jml: I haven't dealt with one of those in a while either.
<allenap> jml: I assume I need to check that they've signed a contributor agreement. Then I wondered what's the accepted way of landing to (in this case) lazr.restfulclient.
<jml> allenap: I think that's pretty standard across Canonical
 * lifeless hates on testfix
<lifeless> you know what I think would be most useful
<lifeless> an indicator that we're in testfix
<lifeless> or not
<jml> lifeless: bug 424060 (also the related bug 422964)
<_mup_> Bug #424060: Impossible to know whether we are in testfix <build-infrastructure> <feature> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/424060>
<_mup_> Bug #422964: testfix mode should stop the queue, rather than rejecting new items <build-infrastructure> <test-system> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/422964>
<lifeless> jml: ah
<deryck> ah, wifi again.
<abentley> wgrant: around?  I'd like to understand bug #645620 better.
<_mup_> Bug #645620: Deleting recipe leaves SourcePackageReleases with no traceability <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/645620>
<cr3> leonardr: if I want to extend the information exported by lazr.restful which would need to use the Request object, might there be a way to adapt Entry or ResourceEntry or somesuch so that when fields are extracted from my adapted object, then my own Entry could inject it's on context sensitive information?
<cr3> err, s/context sensitive/request sensitive/
<leonardr> cr3: i kind of think you want to create a new resource type?
<leonardr> can you spell out the request/response you want to do?
<cr3> leonardr: I'm not sure if I can just define a new resource type because I think I'd need to adapt it for another request interface as well, and all that
<leonardr> cr3: in that case i need a little clearer picture of what you're oding
<cr3> leonardr: for example, lets say I want to attach the version to objects returned by the restful interface, obj.version
<leonardr> cr3: this is the web service version, or something else?
<cr3> leonardr: yes, web service version
<cr3> leonardr: I realize it's redundant information because I'm making a request against a versionned url, but I'd like the remote object to carry that information for debugging purposes
<leonardr> cr3: ok, the best thing might be to subclass EntryResource and register your subclass wherever EntryResource is retrieved
<leonardr> benji might be able to put that better than i
<cr3> leonardr: if someone could figure how to make IJSONPublishable(obj) in _resource.py return anything other than EntryResource, that'd be awesome!
 * benji reads
<leonardr> cr3: couldn't you register your own IJSONPublishable adapter for your objects?
<cr3> leonardr: I tried and it didn't work, I'll try to remember why...
<benji> a custom IJSONPublishable adapter sounds to me like the way to go
<benji> ...which would probably be a subclass of the stock adapter
<cr3> leonardr: my understanding is that I need to adapt EntryResource to provide the IJSONPublisher interface, but that never gets resolved in ResourceJSONEncoder
<cr3> benji: class Foo: implements(IJSONPublishable); adapts(EntryResource); def toDataForJSON(self): import pdb; pdb.set_trace(); never gets called
<leonardr> cr3: sorry, benji's -fu in this matter is stronger than mine, and i have to work on other stuff
<benji> cr3: you have to register the adapter so the component machinery knows it exists
<cr3> benji: I also had something like this in my configure.zcml: <adapter factory="bar.Foo" />
<benji> cr3: don't you want something like adapts(EntryResourceWithVersion) instead of adapts(EntryResource)?
<cr3> benji: I meant EntryResource literally though, to override the one in _resource, not just for a specifically adapted class+version
 * cr3 had to lookup the latest lazr.restful code just in case "EntryResourceWithVersion" actually existed :)
<benji> :)
<benji> so you want the version on all objects then?
<benji> hmm, I'm surprised you didn't get an error on start-up, it sounds like you've created a conflicting registration (two adapters for the same set of interfaces)
<cr3> benji: I'd like to make a best effort, so if I could override EntryResource with my own toDataStructure or somesuch, I could try a few things
<benji> is there a branch I can see or can you pastebin the diff or something?
<cr3> benji: I worked on that yesterday evening and didn't get anywhere, so I deleted everything thinking I was going down the wrong path
<benji> it sounds like the right path to me, you want to replace one component with another (slightly different) one; that's what they're for :)
<cr3> benji: just to make sure I understand correctly though, I should be aiming for "return IJSONPublishable(obj).toDataForJSON()" in ResourceJSONEncoder to return my own IJSONPublishable implementation in order to augment the data normally returned by EntryResource.toDataForJSON, right?
<cr3> benji: err, "IJSONPublishable(obj)" should return my own IJSONPublishable..., the complete call should return the data :)
<benji> cr3: right (in other words "IJSONPublishable(obj) should return the result of your adapter (which is a class that /implements/ IJSONPublishable) which will be something that /provides/ IJSONPublishable (an instance of your class)"
<cr3> benji: awesome, if you happen to have an example somewhere, that might help. otherwise, since I seem to understand what's going on, I'll probably figure it out eventually :)
<cr3> benji: by the way, good work on the whole registration of adapters and interfaces in lazr.restful, pretty cool stuff!
<benji> cr3: I don't have an example at hand, but it sounds like your previous attemt was very close
<cr3> benji: I'll make sure to keep the code if I get blocked again :)
<benji> :)
<benji> Leonard deserves the credit for anything nice in lazr.restful and Jim Fulton gets credit for the Zope component framework
<leonardr> actually i bet flacoste deserves the credit for what cr3 is mentioning
<cr3> sounds like a few people making a project come together nicely, as far as I can see. I'm jealous :)
<jcsackett> has anyone seen a storm query replace a condition with FALSE?
<thumper> jcsackett: no
<wgrant> abentley: Hi.
<abentley> wgrant: doh.  EOD and I have company.
<wgrant> Heh, sorry.
<thumper> wgrant: hi
<thumper> wgrant: I'm about to head out to lunch, but I know a little about what abentley was going to talk to you about
<thumper> wgrant: bug 645620
<_mup_> Bug #645620: Deleting recipe leaves SourcePackageReleases with no traceability <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/645620>
<wgrant> thumper: What do you want to know about it?
<thumper> I think we were wondering how to create the traceability
<thumper> perhaps through making the recipe registrant the uploader?
<wgrant> Don't delete things.
<thumper> wgrant: not deleting things isn't really an option
<thumper> we should allow people to delete things
<persia> Deleting a semantic object doesn't necessarily require removal of the underlying datastore: one can have a "deleted" flag, or similar.
<wgrant> thumper: Normally we record the signing key used on the changes file.
<thumper> persia: sure
<wgrant> thumper: That is clearly not possible here.
<wgrant> So we agreed that the SPRB would be recorded.
<wgrant> And now you're nullifying that.
<thumper> hmm...
 * thumper has to run, but perhaps we could talk more after lunch
<wgrant> Sure.
<wgrant> jelmer: Hi. Could you try to land my branch before we get into testfix again?
<jelmer> wgrant: Yes.
<wgrant> jelmer: Thanks.
<wgrant> lifeless: Hi.
<lifeless> hi
<lifeless> 'sup ?
<wgrant> lifeless: Has cesium had anything CP'd lately?
<lifeless> I'm not sure; let me see
<wgrant> More precisely, do you know which rev of devel it is?
<lifeless> should be 11824
<lifeless> I'll check
<wgrant> Because the code involved in adare's issue has sort of been completely rewritten this cycle.
<lifeless> do we need to roll it back ?
<jelmer> wgrant: are you talking about the async builddmanager?
<wgrant> jelmer: Yes.
<wgrant> lifeless: Unlikely.
<lifeless> wgrant: confirmed
<lifeless> 11824
<wgrant> lifeless: Aha, thanks.
<wgrant> Now I have to try to make sense of this Twisted monster.
<wgrant> The deferreds really make the traceback rather useless :(
 * spm hands wgrant a lantern of codelight, flaming sword of gdb-ness and a pet grue - with which to hunt and slay the twistd monster.
<wgrant> spiv: Can I convince you to check for the existence of a directory on germanium?
<wgrant> Er.
<spm> probably not, but you can convince me?
<wgrant> spm: ^^
<wgrant> Stupid tabs.
<spm> :-)
<spm> which one?
<wgrant> spm: /srv/launchpad.net/ubuntu-archive/ubuntu-temp
<spm> 2010-11-04 23:25 /srv/launchpad.net/ubuntu-archive/ubuntu-temp/
<spm> verra recently updated going by that, if that helps
<wgrant> Yeah, as I suspected :(
<wgrant> Braindead lucilleconfig.
<wgrant> Makes my removal branch slightly more difficult.
<spm> rm -rf is hard?
<wgrant> spm: That's the primary archive's temp dir. PPAs happen to use it too, due to lucilleconfig being stupid (it was added 'temporarily' a little over six years ago). My removal branch changes the way that path is calculated, and it will end up somewhere different on germanium.
<wgrant> So yay, config changes.
<spm> ah. goo! I approve in general terms based on your description. sounds good! +1. etc etc. :-)
<spm> good!
<spm> mind you. $job-1. I was there for 6 years on a 4month temp contract. I can see how these temp things remain. :-D
<lifeless> spm: I was like that @ Micropay
<wgrant> Heh.
<LPCIBot> Project db-devel build (119): FAILURE in 3 hr 33 min: https://hudson.wedontsleep.org/job/db-devel/119/
#launchpad-dev 2010-11-05
<lifeless> thumper: ping
<lifeless> Bug 667554 is blocking deploys
<_mup_> Bug #667554: State change and reviewer request emails go out for work in progress <code-review> <email> <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by thumper> <https://launchpad.net/bugs/667554>
<lifeless> thumper: (This is the 'qa promptly...-please-' plea)
<lifeless> thumper: I would nag wallyworld, but he isn't on irc just now :)
<wgrant> lifeless: I think we may need to roll back cesium.
<wgrant> It's knocked out half the non-virt builders.
<StevenK> 'NoneType' object has no attribute 'processor'
<StevenK> Wheee
<wgrant> There's a Twisted callback being called after a failure, when it shouldn't.
<wgrant> And the builder's cached behaviour is incorrect at the that point.
<wgrant> A quick workaround may be to remove the behaviour cache.
<wgrant> But can we just rollback to whatever it was before?
<StevenK> lifeless: ^
<wgrant> I don't have the time nor the log access to debug this much further :/
<lifeless> ok
<lifeless> wgrant: 11811 be ok ?
<spm> soyuz-production-rev-9886 is more likely. that's a softlink change.
<wgrant> lifeless: No. 11808 is probably the culprit.
<wgrant> 11801 was the initial landing, it was rolled back in 11805, then brought back in 11808.
<wgrant> I'd go with spm's suggestion.
<lifeless> 11793 then
<spm> any other revno isn't so much a rollback, as a rollout
<lifeless> spm: ah, please do that.
<lifeless> spm: we should also take cesium back out of nodowntime
<spm> oki
<lifeless> spm: until this is addressed
<thumper> lifeless: where is the imap folder for the qastaging email box?
<lifeless> perhaps we need a 'lock' facility in the deployment manager
<lifeless> thumper: its the staging one
<thumper> lifeless: how do I tell the difference?
<lifeless> thumper: right now you don't. I've filed an RT asking for something
<thumper> lifeless: ok
<lifeless> thumper: OTOH there is almost no mail from staging/qastaging. So you can just empty the mbox
<lifeless> then do your thing
<thumper> lifeless: what bzr hosting do we have for qastaging?
<lifeless> thumper: EPARSE
<thumper> lifeless: like pushing branches to qastaging?
<lifeless> its all there
<lifeless> lp: won't work until the bzr bug I filed is fixed.
<thumper> ok
<lifeless> but the full paths will
<lifeless> I should probably do the bzr one myelf
<wgrant> spm: Once that's rolled back and b-m restarted, the disabled buildds need to be undisabled.
<spm> figures
<lifeless> spm: thanks
<spm> it's rolled back, fwiw. that was fairly easy. just making sure all teh required services are restarted
<lifeless> yeah, thats the idea :)
<lifeless> low TTR
<thumper> test_ppa_displayname_inline_edit  failing on db-devel builder
<lifeless> :(
<thumper> mwhudson: have you tried mumble recently?
<thumper> mwhudson: it has started working for me fine
<mwhudson> thumper: no
<thumper> mwhudson: in the end I deleted my ~/.config/Mumble dir and started again
<thumper> mwhudson: dropped the caching to 10ms
<thumper> mwhudson: and it is working fine now
<mwhudson> thumper: huh, cool
<mwhudson> i should try it again then
<thumper> I'm not entirely sure what else has changed, but I've been using it for standups this week
<thumper> and for talking with flacoste
 * StevenK peers at the build page. Where's my build log?
<wgrant> StevenK: Which build is borked?
<StevenK> wgrant: https://launchpad.net/ubuntu/+source/linux/2.6.37-2.10/+build/2032158 for example
<wgrant> StevenK: Looks like it was killed by the failure counter.
<wgrant> Retry.
<wgrant> Bonus points if you unkill all the builders.
<StevenK> I set it to OK and remove the failure message?
<wgrant> Indeedily.
<StevenK> Whee, adare promptly failed itself again
<wgrant> That's not unexpected.
<wgrant> ppc is sort of screwed at the moment.
<wgrant> At least it has a proper error this time.
<StevenK> I prefered the other error, it at least pointed to a code error in the buildd-manager :-)
<StevenK> allspice picked itself back up, though
<StevenK> wgrant: Have you filed a bug about this, or do I need to?
<wgrant> StevenK: I haven't. Do you have the traceback and related log entries?
<StevenK> Not to hand
<wgrant> http://paste.ubuntu.com/526017/
<StevenK> wgrant: bug 671242
<_mup_> Bug #671242: New buildd-manager disabling everything in sight <Soyuz:Confirmed> <https://launchpad.net/bugs/671242>
<persia> Good title!
<wgrant> StevenK: Thanks.
<wgrant> StevenK: is there a probe failure message immediately before each traceback?
<wgrant> I see there's one 9 seconds later, but that's probably from the next scan.
<StevenK> wgrant: After
<wgrant> :(
<StevenK> Traceback, and then the probe failure message immediately after
<wgrant> StevenK: Is there a "builder BLAH failure count: X" line before the traceback?
<wgrant> My theory is that assessFailureCounts knocks the build off the builder, then rescueIfLost fires in the same transaction, using the cached BFJB.
<StevenK> wgrant: That's attached to the failed probe messages, isn't it?
<StevenK> 2010-11-05 00:25:45+0000 [QueryProtocol,client] Builder lakoocha failed a probe, count: 4
<wgrant> StevenK: The 'failed a probe' message occurs when a failure is detected with no build assigned.
<wgrant> 'builder BLAH failure count: X, job 'BLAH' failure count: Y' is used when there is a build assigned.
<StevenK> wgrant: I can see that in the logs for rothera, but it's a timeout error, not the processor none traceback
<wgrant> Intriguing.
<wgrant> Anyway, I guess we'll have to wait until Monday evening.
<StevenK> wgrant: I wonder if there's an addErrBack for rescueIfLost
<wgrant> StevenK: rescueIfLost isn't the problem here.
<wgrant> Its caller is.
<wgrant> It is being called with a dirty transaction with a builder with an outdated BFJB cache.
<wgrant> There are enough commits and aborts around that that shouldn't happen.
<wgrant> But there must be a missed case.
<wgrant> But I am not exactly a Twisted expert.
<StevenK> Neither
<wgrant> Maybe I should acquire a book.
<thumper> StevenK: I hope there is lots of lovely logging for the new buildd manager
<wgrant> Not enough :(
<wgrant> But it's not as bad as it ws.
<thumper> well add some dear liza...
<thumper> has the buildd manager been rolled back?
<StevenK> wgrant: http://www.amazon.com/Twisted-Network-Programming-Essentials-Fettig/dp/0596100329/ref=sr_1_7?s=books&ie=UTF8&qid=1288923189&sr=1-7 ?
<StevenK> thumper: Yes
<thumper> StevenK: can I talk to you in about 30 minutes about PPAs?
<StevenK> thumper: Sure
<thumper> ta
<wgrant> thumper: As I see it, we have a bit of a conflict. Code likes to delete things, while the rest of Launchpad does not delete anything ever.
<thumper> wgrant: sure it does...
<wgrant> Oh?
<thumper> blueprints will be deletable soon
<thumper> :-)
<wgrant> I hope you mean the code.
<thumper> we have lots of garbo scripts that delete things
<thumper> and no, I don't mean the code
<wgrant> They delete records of inconsequential happenings.
<spm> hahaha
<wgrant> Not OMGCRITICAL stuff like who uploaded some malware to Ubuntu.
 * spm prints "<wgrant> They delete records of inconsequential happenings." to preserve for all time
<lifeless>   Hard / Soft  Page ID
<lifeless>      278 /    7  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>      207 /   55  Person:+commentedbugs
<lifeless>       94 /    0  MailingListApplication:MailingListAPIView
<lifeless>       90 /  375  BugTask:+index
<lifeless>       83 /  269  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>       34 / 2190  Archive:+index
<lifeless>       32 /  138  Archive:+packages
<lifeless>       19 /  323  Distribution:+bugs
<lifeless>       18 /   14  ProjectGroup:+milestones
<lifeless>       11 /   82  MaloneApplication:+bugs
<StevenK> lifeless: Are they timeouts?
<wgrant> Hm, we only hold three of the top ten, and two of those are probably the same issue. Not bad.
<thumper> StevenK: mumble?
<StevenK> thumper: Sec
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build (120): FIXED in 3 hr 9 min: https://hudson.wedontsleep.org/job/db-devel/120/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11862
<LPCIBot> included.
<StevenK> lifeless: Interestingly, the deployment page says bad, but the bug is marked 'qa-untestable'
<StevenK> thumper: Are you okay to mentor my review of https://code.edge.launchpad.net/~wallyworld/launchpad/person-mergequeue-listview/+merge/39933 or would you prefer someone els
<StevenK> s/els$/else?/
<thumper> I can take a look
<lifeless> StevenK: which bug
<lifeless> spm: can you please add a feature flag to qastaging for qa needs?
<lifeless> hard_timeout team:bzr 15 5000
<spm> lifeless: thatshould be applied
<lifeless> spm: can you drop that to 2000 ?
<spm> done
<lifeless> great
<lifeless> please remove the rule
<spm> done
<lifeless> thanks
<wgrant> Is PQM happy at the moment, and likely to remain so? It threw out https://code.edge.launchpad.net/~wgrant/launchpad/destroy-distroseries-lucilleconfig/+merge/38648 yesterday.
<lifeless> wgrant: we can't actually tell
<wgrant> ...
<lifeless> there is a bug
<wgrant> Hwhat?
<lifeless> the testfix state is not well exposed
<lifeless> last build of devel workd
<lifeless> last build of db-devel failed
<wgrant> Yay.
<lifeless> devel and db-devel are both building atm
<lifeless> WindmillTestClientException: {u'suite_name': u'lp.soyuz.windmill.tests.test_ppainlineedit', u'result': False, u'starttime': u'2010-10-5T6:12:5.320Z', u'params': {u'xpath': u"//h1[@id='displayname']/span[1]", u'validator': u"Celso's default PPA"}, u'debug': u"Text 'Celso's default PPA' was not found in the provided node.  Found instead: PPA for Celso Providelo", u'output': None, u'endtime': u'2010-10-5T6:12:5.321Z', u'method': u'
<lifeless>                   u'validator': u"Celso's default PPA"}, u'debug': u"Text 'Celso's default PPA' was not found in the provided node.  Found instead: PPA for Celso Providelo", u'output': None, u'endtime': u'2010-10-5T6:12:5.321Z',
<lifeless>                   u'method': u'asserts.assertText'}
<lifeless> looks like a genuine fail
<lifeless> StevenK: ^
<wgrant> Hudson's still happy, though :/
<wgrant> So not very legit.
<StevenK> That's ... odd
<SpamapS> lifeless: was it you that pointed out mod_pagespeed?
<lifeless> no
<lifeless> I was the one that went eeeeek omg nooo
<lifeless> [just on the probably complexity of the heuristics it must have]
<lifeless> let alone the potential to mess with our API implementation
<lifeless> SpamapS: ^
<SpamapS> so I filed an ITP w/ debian to package it..
<SpamapS> and have started looking into it
<SpamapS> 45 dependencies.. I think 3 of them are already packaged
<SpamapS> they build a .deb, but they don't use dpkg-buildpackage to do it
<lifeless> -lol-
<lifeless> I'm very unsurprised.
<lifeless> Google project yeah?
<SpamapS> the first thing it does, is sync all of its 3rd party dependencies from their upstream repositories at specific revisions.
<SpamapS> yes, Google seems to be against releasing and stablizing things. ;)
<lifeless> well
<SpamapS> I have to wonder, are we fighting the good fight, or holding back progress?
<lifeless> its a different definition of release ;)
<SpamapS> TODO: describe advanced compilation options such as compiling against system headers.
<lifeless> SpamapS: *blink*
<SpamapS> am I the only one who thinks thats *BASIC* compilation?
<lifeless> I hear ya
<SpamapS> I can see how the conversation went at Google that led to gyp and gclient (the things that make this "just work")
<SpamapS> "Its got 45 dependencies, it takes a long time to get them all onto one system."
<SpamapS> "I can solve that!"
<SpamapS> "Ok, solved. We can now compile all 45 dependencies with one command. Who wants to go play ping pong?"
<lifeless> they have a specific diagnostic story
<lifeless> if you look at go it becomes very clear
<wgrant> Well...
<lifeless> they -want- a single binary with no shared libraries
<wgrant> Have you seen Chromium?
<wgrant> It is the Java mentality cubed.
<SpamapS> Yes, this uses the chromium tools
<lifeless> the reason they want this is so that when there is a fault, they know all the deps precisely with no room for skew
<lifeless> and for complete isolation between concurrently installed things on the same machine
<wgrant> Yes, but it makes everyone else sick.
<lifeless> I don't completely buy the value of this - apport does a pretty tremendous job
<SpamapS> ridiculous.. they're including libjpeg, libpng, zlib
<SpamapS> things that have been on every system for a long, long time.
<lifeless> yes
<lifeless> its not because they have to
<lifeless> its because they want to
<lifeless> the numeric analysis is something like:
<wgrant> They *think* they want to.
<lifeless>  - if we get 10billion downloads
<lifeless>   - we'll save 5 hours diagnosis due to folk with zlib1e which had a buffer overflow
<lifeless> wgrant: they want to. No thinking involved.
<SpamapS> They're thinking in faster moving terms than most people actually move. Its valid for desktop software.. people see "there are updates" and they tend to update. Not so much for the server side.
<lifeless> no
<lifeless> its not about how fast things move
<lifeless> its about reproducability and support overhead
<SpamapS> Which is funny, thats why we do what we do. ;)
<lifeless> I got into this fairly deeply @ LCA last year
<SpamapS> But they're doing it by assembling a mini-distro around their product.
<lifeless> none of it attributable, of course
<SpamapS> which is actually totally awesome.. if you, the user, trust them to keep all those deps in top shape.
<SpamapS> lifeless: we didn't really solve this at UDS.. too many distractions.. its the same old problem.
<wgrant> SpamapS: Does the solution actually need discussion?
<wgrant> I don't think there's any option but to declare their strategy as being utterly crackful, ignore it, and port their build system to use real system libraries.
<SpamapS> wgrant: 45 is a drop in the bucket. Hudson has over 1000 java libraries that aren't packaged. This keeps coming up... upstreams don't build on top of distro's, they build on top of libraries.
<wgrant> SpamapS: But Hudson at least doesn't bundle modified copies of its libraries, right?
<SpamapS> modified, no, selected versions, yes absolutely.
<SpamapS> to me, these upstreams have done a lot of our work for us and we throw it away because we only want to have one version of everything to support.
<StevenK> Surely Hudson isn't 1,000? 200, 300, sure, but 1k?
<wgrant> Doing a Launchpad and specifying a known-good set of unpackaged versions is one thing.
<wgrant> But bundling libraries... uh... kill them.
<StevenK> Didn't dpkg used to statically link against zlib in the deep dark ages?
<StevenK> lifeless: Can you remind me of a discussion we had at UDS? You mentioned using btrfs on the Hudson build slaves -- that one I remember, but there was another suggestion
<SpamapS> StevenK: talk to JamesPage about it.. 1000 was the number of libs hudson needed to download from maven to build *after* it cross referenced with everything debian had packaged.
<lifeless> SpamapS: well I've argued (successfully I think) that we need N libraries concurrently for java and python
<lifeless> StevenK: errr I didn't suggest btrfs on the build slaves.
<lifeless> StevenK: I suggested an RT ticket asking for bigger images.
<lifeless> StevenK: our you can roll your own
<StevenK> lifeless: Actually, they're 20GiB, just the / is only 1.5GiB
<lifeless> StevenK: yes, but thats the partition you need bigger to install java etc, right ?
<StevenK> Currently it fits with about 50MiB to spare
<lifeless> EMEEP
<StevenK> Current devel slave has 120MiB, that will drop to roughly 50 over the course of the build
<StevenK> lifeless: Too close, and I should repack?
<lifeless> dunno about you but my sysadmin senses are screaming
 * StevenK ponders an EBS store for persistant branches
<StevenK> steven@hudson:/tmp$ ls -1 | wc -l
<StevenK> 3589
 * StevenK sobs
<StevenK> lp.soyuz.windmill.tests.test_ppainlineedit fails on db-devel again ... How odd
<wgrant> Huh.
<wgrant> I don't recall that failing within the last few months.
<StevenK> And has now failed twice in 2 builds, but not in devel
<wgrant> And not in Hudson.
<StevenK> Indeed
<StevenK> Sounds like a reason to switch to me
<StevenK> :-P
<bac> hi wgrant, you still around?
<wgrant> bac: Hi.
<bac> wgrant: you know much about the inner workings of merge proposals?
<wgrant> bac: Not a huge amount, but possibly enough.
<bac> wgrant: i've noticed rollback branches don't get their status updated to 'merged'
<bac> and they clutter up +activereviews until cleaned up
<wgrant> Do you have a current example?
<bac> https://code.launchpad.net/~bac/launchpad/rollback-11801/+merge/39422
<lifeless> bac: uhm
<bac> wgrant: unfortunately i spoiled it by updating the status manually
<lifeless> bac: that means one of two things
<lifeless> bac: it wasn't actually landed previous
<lifeless> bac: or the scanner glitched at the same time
<bac> https://code.launchpad.net/~mars/launchpad/rollback-11586-and-11588/+merge/36212
<lifeless> bac: launchpad itself doesn't know about the rollback stuff
<bac> lifeless: well, there is another
<wgrant> Hm.
<bac> i know the first actually landed
<bac> i haven't checked on the second
<wgrant> https://code.launchpad.net/~bac/launchpad/rollback-11801
<wgrant> It's being scanned.
<wgrant> Which is a little odd.
<bac> ah, so it is
<bac> for 10 days now
<StevenK> lifeless: Interestingly, the deployment page says bad for r11825, but the bug is marked 'qa-untestable'
<lifeless> see under scanner glitch
<bac> score one for lifeless
<wgrant> Indeed.
<lifeless> StevenK: https://bugs.edge.launchpad.net/qa-tagger/+bug/670792
<_mup_> Bug #670792: handling of bad-commit- tags needs improving <qa-tagger:Incomplete> <https://launchpad.net/bugs/670792>
<wgrant> These non-edge URLs are strange and foreign :(
<StevenK> Haha
<StevenK> Blame lifeless
<bac> wgrant: can i do anything to kick the scanner for that branch?
<lifeless> StevenK: its bad because of the bad-commit-REVNO tag (which is how we mark that 11825 was faulty)
<wgrant> bac: Not sure.
<wgrant> I don't really know how the scanner works these days.
<bac> wgrant: if it is a one-off i'm not concerned.  but i saw two that appeared to be in the same situation
<wgrant> That is indeed concerning.
<bac> lifeless: i see your comment on the MP
<bac> lifeless: i didn't just now land it, i did it the day of the problem
<bac> just updated the status of the MP today
<lifeless> grah
<lifeless> ok
<lifeless> I wouldn't do that
<bac> what would you do?
<lifeless> because the metadata will be faulty about the rev it was merged into
<lifeless> I would look to see why it hadn't updated and get that fixed ;)
<bac> mark the MP rejected?
<lifeless> no
<lifeless> so, when you look at the branch and its scanning
<lifeless> thats an issue
<lifeless> look into that
<bac> lifeless: i did not see the branch was scanning
<lifeless> when its fixed, lp will mark the mp appropriately.
<lifeless> bac: right, I know :). I'm saying what I would have done was to look around for clues.
<lifeless> one of the places I'd look would be the branch page.
<bac> lifeless: so would resetting the MP back to 'approved' be inappropriate now?
<lifeless> bac: if the branch is still scanning, setting it back to approved is probably best.
<lifeless> OTOH
<bac> and then trying to get that scanner process kicked...
<lifeless> the main thing I'm concerned about is figuring out what happened, ensuring there is a bug that when fixed will prevent it happening again.
<bac> lifeless: ok, i'll change the MP back to 'approved' and open a bug referencing that branch but not request anything be done about the stuck scanner.  thanks.
<wgrant> I'm not sure that it will be marked merged once it does scan, since the rev won't be new in devel. But I guess we'll see.
<bac> wgrant, lifeless: fwiw the other example i cited is a rollback branch that never landed so it was a bad example.
<lifeless> what pages does the recipe ui stuff turn up on ?
<wgrant> lifeless: Every branch page.
<lifeless> time to see if the flag really works
<wgrant> lifeless: There's a "Related source package recipes" section just below the "Branch merges" section.
<bac> ok
<bac> er
<adeuring> good morning
<bac> hi adeuring
<adeuring> hi bac!
<mrevell> Morning
<bac> hi mrevell
<mrevell> hey bac, welcome back, heh
<jml> hello
<lifeless> jml: your scopes rules are present in production; both patches are in stable and qa'd - we just need to qa up to them and they will go live.
<jml> lifeless: sweet.
<jml> lifeless: I guess I need to populate the team, delete the edge rule and do comms
<jml> anything else?
<lifeless> yes. Synced with the revs being deployed of course.
<lifeless> we probably want to do a beta program in-app too
<lifeless> a-la gmail labs
<jml> lifeless: "here are our beta features, sign up to them?"
<lifeless> yeah
<jml> lifeless: I think that would be an excellent thing.
<wgrant> Are we going to have a lot of features for which that would be relevant, though?
<lifeless> wgrant: we have 3 concurrent at the moment
<jml> wgrant: I'm guessing so.
<wgrant> Do we?
<lifeless> wgrant: recipes, merge queues, subscribe-to-search
<wgrant> Hm, true.
<lifeless> oh and derived-series ui
<jml> lifeless: https://code.edge.launchpad.net/~lifeless/launchpad/databasefixture/+merge/38694 â what's the actual state of that MP?
<lifeless> jml: in principle landable
<lifeless> it bounced with some glitches that I think I've fixed.
<lifeless> if you want to give it a spin, please do.
<jml> lifeless: ok. I probably won't today.
 * jml away for a bit
<deryck> Morning, all.
<jelmer> wgrant: Argh, it looks like your branch is cursed or something.
<wgrant> jelmer: Yeah, buildbot seems to have a failure that Hudson does not.
<wgrant> Still, this is just two failures so far. One of my previous batch took 10 attempts. So not that bad.
<mars> good morning
<jtv> hi mars
<mars> how is it possible that merely importing a TestCase subclass that defines 'layer = None' causes the Zope testrunner to barf?
<jml> mars: importing a TestCase makes it discoverabl
<jml> e
<mars> jml, that appears so.  Now I'm trying to figure out how other people actually manage to use the subclass without it blowing up on import
<mars> 'bin/py -m pdb bin/test lp.testing.tests' at least gives me a debugger
<jml> do we have a way to mass add people to a team?
<abentley> jml: why are you creating a new team for recipes instead of making them available to all members of the beta team as we have until now?
<benji> jml: I don't know if it exposes adding people to teams, but a launchpadlib script suggests itself to me
<jml> abentley: it seems a good way to try having a beta programme specific to a feature
<jml> benji: yeah, that would work. lots of roundtrips, but it would work & probably be faster than manual data entry.
<abentley> jml: I don't think we should take the feature away from people who already had the option of using it, even if they hadn't used it yet.
<abentley> jml: So please make the beta team a member of the recipe team :-)
<jml> the real spanner-in-the-works is that we don't yet have a way in-app of inviting people to join the beta
<mars> jml, that means we have a chance to do it well
<mars> jml, btw, turns out I had to put a 'del YUIUnitTestCase' at the end of the file, so the testrunner would not discover the class.
<jml> mars: is that what other modules do?
<abentley> jml: That is a serious deficiency.  People can join the team if they want, I suppose, but the requirement to do so will be new and surprising.
<jml> abentley: well, only surprising if they don't read the blog.
<abentley> jml: Who reads the blog?
<jml> abentley: I think Francis does
<jml> abentley: maybe some others.
<mars> jml, no, they define the test_suite() function in the module, and explicitly build the suite contents.  Same effect though - the module imports are not implicitly included
<jml> mars: how does YUTC blow up?
<mars> jml, deep in the zope testrunner, it trys to look up 'layer.__module__'.  YUTC sets that attribute as layers = None, so it raises a TypeError
<jml> mars: why not just delete that from YUTC?
<mars> jml, I assume it was put there for good purpose, perhaps as a subtle signal to the programmer that they need to define it
<mars> and I am writing a test for the class
<mars> which does not use the class in the standard way
<jml> mars: if it was put there for good purpose, then a test will fail if you delete it :)
<mars> interesting
<mars> well, worth a shot!
 * jml gets some thinking juice
<mars> jml, so this is fun - you can't import YUTC, because it uses layers.  But YUTC can't set layer=anything, because you can't import canonical.testing.layers into lp.testing (circular import maybe)
<jml> mars: that *is* fun.
<jml> but I have to go to a meeting
<mars> YUTC uses layers because it inherits from lp.testing.TestCaseWithFactory, which uses layers
<mars> jml, :)
<mars> and TestCaseWithFactory needs layer=FunctionalLayer, but again, it can't set it, because lp.testing can't import canonical.testing.layers.
<deryck> mars, hi.  Working on yui 3.2 upgrade in lp and have questions about something weird.  you have time to chat?
<mars> deryck, sure
<mars> deryck, how can I help?
<deryck> mars, so I see lots of 404s in console for css files.  I can't find what's trying to load them.  turning devmode off I see 404 for yui combo? urls pointing to yahoo's site....
<deryck> mars, I'm wondering if I have to explicitly turn of combo loading attempts?
<mars> deryck, well, for starters, we should not be going off-site for anything.  Is this in launchpad.dev?
<deryck> mars, yes
<deryck> mars, and only going off site when not in devmode.
<deryck> mars, I toggled out of devmode just to see if the 404s were devmode specific and maybe linked in a devmode block I couldn't find.
<mars> deryck, maybe you could hack the page template to show what the loader is doing - debug output
<deryck> ok
<mars> deryck, have you ever done that before?
<deryck> mars, yes, I have.  trying to recall how....
<mars> oh, interesting
<mars> deryck, reading the YUI Config object docs
<mars> you can set a custom filter: attribute
<mars> myFilter: {
<mars> 'searchExp': "-min\\.js",
<mars> 'replaceStr': "-debug.js"
<mars> }
<mars> which means you could do:
<mars> debugLoader: {
<mars> 'searchExp': "loader-.*js",
<mars> 'replaceStr': "loader-debug.js"}
<deryck> ah, ok.  That's not what I'm thinking of.  I thought you told me something similar before, though, about loader debugging.
<mars> deryck, in devmode, you can look at base-template-macros.pt and replace the loader.js line directly
<deryck> ok
<deryck> mars, found it.  http://pastebin.ubuntu.com/526338/
<mars> deryck, that was an easy fix
<deryck> mars, indeed.  and unrelated to my other issues, but it clears up the errors in my console better :-)
<deryck> mars, thanks for the help!
<mars> no problem
<mars> glad I could help
<jml> jelmer: hi
<jelmer> jml: hi
<jml> jelmer: what happens when you run ./bin/test -cvv test_builder?
<jml> sinzui: hello
<jml> sinzui: I saw that you removed a bunch of glob imports recently. If I remove one from c.l.database.__init__ manually, will that mess with your automated tools
<jelmer> jml: I get a test module import failure for lp.buildmaster.tests.test_builder, and it runs the test: lp.soyuz.browser.tests.test_builder_views.TestBuilderEditView.test_posting_form_doesnt_call_slave_xmlrpc
<jml> jelmer: cool, thanks. that's what I get too. I hate Python.
<jelmer> jml: I usually just run with -t  to avoid this sort of thing
<jml> jelmer: :(
<jelmer> jml: I agree.
<deryck> rockstar, ping
<rockstar> deryck, pong
<deryck> rockstar, hey, dude.  Getting close on lazr-js actually.  http://pastebin.ubuntu.com/526387/
<rockstar> deryck, is that working?  My sed of s/yui/yui3/ made things really broken.
<deryck> rockstar, yeah, so far.  notice it's template and lp js changes.  if one without the other, it would break.
<deryck> rockstar, I'm just reloading pages manually and making changes until it works.
<rockstar> deryck, I thought the picker had WAY more references to creating items called yui-* than that.  The picker's working now?
<deryck> rockstar, yes, I'm pretty sure.  I'm in a debugger session right now with the editor before I can confirm for sure.
<rockstar> deryck, cool. I was wondering if there'd be more picking and choosing of which s/yui/yui3/ stuff to change.
<deryck> rockstar, is there a yui lint tool of any sort?  To check a file for yui 3 vs 3.2 isms?
<rockstar> deryck, not that I know of.
<deryck> suck it
<rockstar> deryck, may I suggest that you file bugs liberally on the javascript oddities you find in there?  Anything that makes the upgrade hard.
<deryck> rockstar, sure.  just trying to get it done now.  but will certainly do it after it's good.
<rockstar> Takes notes then.  :)
<rockstar> deryck, I'm starting to wonder if there's a yak in our javascript anywhere.
<deryck> rockstar, we're not consistent in how we write it
<deryck> zope widgets, hand js, manual sniffing of classes
<rockstar> deryck, yeah, I'd like to write a guide on writing javascript for the dev wiki.  I need to get around to that soon.
<jml> jkakar: hello hello
<jkakar> jml: Hi!
<jml> jelmer: are there doctests that test the uploadpolicy?
<jml> jelmer: the answer is 'yes'.
<jelmer> jml: yes, and there are also some unit tests I believe.
<jelmer> heh
<deryck> rockstar, pickers do work for me.  But icons that launch overlays are now form buttons.  I guess I'm missing some CSS rules somewhere.
<rockstar> deryck, something with activators, I suspect.
<deryck> yeah
<deryck> or actually, it may be where we didn't use the activator
<rockstar> deryck, ah, okay, because I thought I had fixed that.
<jml> jelmer: what about ./bin/test -cvvt nascentupload-closing-bugs.txt?
<jelmer> jml: import failures in two mailman tests, but other than that it seems to work fine
<jml> jelmer: I get this: http://paste.ubuntu.com/526452/. Guess I'll make clean, schema etc.
<jelmer> hmm, interesting. is that current devel?
<deryck> rockstar, got the icons back.  needed yui3-skin-sam for the body class, not yui-skin-sam.
<rockstar> deryck, ugh.  That scares me a bit.  I suspect you'll want to check a large sampling of pages to make sure that doesn't break anything.
<deryck> rockstar, yeah, I am.  Doing all the manual checks first, then back to re-enabling windmill tests.
<jml> jelmer: stable.
<abentley> rockstar: where was that custom description field, again?
<rockstar> lp.services.fields I believe.
 * jelmer tries stable as well
<abentley> rockstar: thanks.
<deryck> rockstar, so i don't see anything but I think'll I'll use both classes, i.e. "yui3-skin-sam yui-skin-sam" just to cover us.  Sound reasonable?
<rockstar> deryck, if you don't see anything, then let's not even worry about it.
<rockstar> We could complicate things by using both classes.
<deryck> rockstar, ok
<jelmer> jml: works in stable too, no import errors
<jml> jelmer: ok, thanks.
<jml> jelmer: make clean; make schema seemed to sort it out for me
<jml> g'night all
<abentley> rockstar: chat?
<rockstar> abentley, sure!
<abentley> rockstar: hop in!
#launchpad-dev 2010-11-06
<wgrant> Hm, I see the Ohloh report finishes.
<wgrant> *finished
<wgrant> It has some interesting graphs.
<wgrant> In particular, our language pie chart (https://www.ohloh.net/p/launchpad/analyses/latest/languages.png?height=120&project=launchpad&width=120).
<StevenK> Ursinha-afk: I've noticed the deployment page lists rev11868 three times, probably due to the three bugs it's linked against, bug? :-)
<lifeless> StevenK: yes, filed.
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      271 /    9  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>      135 /   39  Person:+commentedbugs
<lifeless>      110 /  289  BugTask:+index
<lifeless>       82 /  268  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>       34 /    2  MailingListApplication:MailingListAPIView
<lifeless>       18 /   31  DistroSeries:+queue
<lifeless>       15 /  299  Distribution:+bugs
<lifeless>       12 /   50  Distribution:+addquestion
<lifeless>       11 /   33  Milestone:+index
<lifeless>       11 /    4  ProjectGroup:+milestones
<lifeless> lol  http://downforeveryoneorjustme.com/http://bugs.opencompositing.org/xmlrpc.cgi as a bug db
<lifeless> rockstar: are you around perhaps?
<lifeless> rockstar: I'm trying to make sense of the merge queues flag description on LEP/FeatureFlags
<lifeless> code.branch_merge_queue
<lifeless> 'disabled'
<lifeless> Will turn on the use of branch merge queues.
<lifeless> there's a sense mismatch between 'disabled' and 'turn on'
#launchpad-dev 2010-11-07
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      270 /   12  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>      151 /   43  Person:+commentedbugs
<lifeless>       75 /  269  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>       58 /  230  BugTask:+index
<lifeless>       10 /  265  Distribution:+bugs
<lifeless>        7 /   63  Archive:+packages
<lifeless>        7 /    7  ProjectGroup:+milestones
<lifeless>        6 /   62  MaloneApplication:+bugs
<lifeless>        6 /    9  Archive:+copy-packages
<lifeless>        5 /    1  BugTask:+editstatus-page
<thumper> lifeless: no mailing list ones :)
<lifeless> because we raised the timeout
<lifeless> hard_timeoutpageid:MailingListApplication:MailingListAPIView617000
<lifeless> https://launchpad.net/+feature-rules
<thumper> :(
<thumper> I thought my fix would have fixed that
<thumper> lifeless: it isn't in the top 10 statement counts any more
<lifeless> thats good
<thumper> lifeless: worth checking, but we should be able to get rid of that exception
<lifeless> it may be that we can remove ... yeah
<lifeless> see what the ppr shows
<thumper> check tomorrow
 * thumper agrees
<thumper> lifeless: I am a little upset though, I was hoping it whould have made some of the code import scheduler ones go away
<thumper> but not
<thumper> s/not/no
<lifeless> thumper: https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-pageids.html
<lifeless> thumper: it looks like its peaking at 10s now
<lifeless> thumper: so improved - yes.
<thumper> I'm guessing it is a different call that is peaking
<lifeless> thumper: I think the big thing is getting it off of the one machine.
<thumper> hence my desire for the method to be included in the pageid
<lifeless> thumper: about 0.01% of calls I'd say
<thumper> Failed to discover an OpenID server?
<lifeless> thumper: I agree, definitely doit
<lifeless> thumper: hmmm, I saw that the other day
<lifeless> I think SSO is having trouble of some sort
 * thumper feels a bid sad that he is on a work channel on Sunday afternoon :-|
 * thumper leaves
<lifeless> morning
<thumper> morning
<wallyworld> morning
<thumper> wallyworld: hi, I'm just moving, we can have a chat after that if you like
<thumper> wallyworld: or do you have kid stuff to do?
<wallyworld> thumper: i have school drop off in 30 minutes; that will tale 20 minutes or so; i can ping you when i'm back?
<thumper> yep
<thumper> ttyl
<lifeless> any soyuz folk around ?
<wgrant> lifeless: Hi.
<lifeless> hi
<lifeless> so I'm fixing the timeout
<lifeless> ArchivePublication injects itself into the code path for getBuilds
<lifeless> and this makes things slow.
<lifeless> there is a test that this works
<lifeless> but its being done 'to make things fast' - oh, the irony.
<wgrant> It makes things slow?
<wgrant> Nice.
<wgrant> How is it making things slow?
<lifeless> the API doesn't use ArchivePublication
<lifeless> https://bugs.launchpad.net/soyuz/+bug/662523
<_mup_> Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/662523>
<wgrant> Ah, the API timeout, I see.
<lifeless> so, I'm wondering whether, when the core is fast, if that delegate is still needed
<lifeless> or if I need to support the injection (hacky) because other layers will be impacted
<lifeless> I'm groupifying the code
<wgrant> Where's the injection?
<lifeless> gimme a minute to get the test passing and I'll push so you can see
<lifeless> lib/lp/soyuz/adapters/archivesourcepublication.py
<lifeless> getStatusSummaryForBuilds
<wgrant> Ahh, I see.
<wgrant> It should be fast enough if you rewrite it. It's only there to cache getBuilds and getUnpublishedBuilds.
<lifeless> Using saved push location: lp:~lifeless/launchpad/getBuildSummariesForSourceIds
<lifeless> rev 11878 should be there now
<thumper> wallyworld: you back?
<wallyworld> thumper: yep
<thumper> wallyworld: shall we mumble?
<wallyworld> already logged in
<lifeless> aahhhhhhhhhh
<wgrant> What have you fixed?
<lifeless> getBuildStatusSummariesForSourceIdsAndArchive calls getBuildsForSourceIds and getUnpublishedBuilds
<lifeless> getUnpublishedBuilds calls getBuildsForSourceIds
<lifeless> oh
<lifeless> and did I mention that we do a (potentially -huge-) set difference in python? :last line of getUnpublishedBuildsForSources
<wgrant> lifeless: That .difference is a Storm operation.
<wgrant> It's done in SQL.
<lifeless> ok, thats a relief.
<lifeless> But it totally sucks that you can't tel.
<lifeless> *tell*
<wgrant> I only know because I looked at the SQL and saw the EXCEPT.
<lifeless> so
<lifeless> we *still* end up doing the same thing twice
<lifeless> except more of its on the db server
<lifeless> I'll batch that up
<wgrant> Great.
<lifeless> and then call and end to this branch
<wgrant> I might be able to look at it after Thursday.
<lifeless> that would be sweet.
<lifeless> I think it will be approximately constant queries now
<lifeless> though I haven't reached the activation energy to write a test for that
<wgrant> Excellent.
<lifeless> I've just been refactoring as I read stuff
<wgrant> It's very easy to get carried away refactoring in Soyuz...
<lifeless> the core query is still going ot be slow (8.4 regression)
<lifeless> I'd really like to see:
<lifeless>  - the adapter nuked
<wgrant> Now that we're more allowed to cache in model objects, sure.
<lifeless>  - only one retrieval of the spph's (vs twice in my branch (or 3N in trunk))
<lifeless> wgrant: caching isn't needed
<lifeless> wgrant: you can cache, but its not what was needed
<wgrant> ArchiveSourcePackageRelease may still be needed.
<lifeless> sure
<lifeless> theres a lot of refactoring to do
<lifeless> many single object methods to nuke
<wgrant> I wonder if we should strongly encourage query count tests for new APIs and views.
<lifeless> certainly
<lifeless> if you're writing something new, a query count test at the start is an awesome idea.
<lifeless> https://code.launchpad.net/~lifeless/launchpad/getBuildSummariesForSourceIds/+merge/40298 if you're interested in this branch
<wgrant> I was just looking for it.
<wgrant> Any chance you could ec2 https://code.edge.launchpad.net/~wgrant/launchpad/destroy-distroseries-lucilleconfig/+merge/38648? PQM testfix-rejected it twice last week.
<lifeless> what flags
<lifeless> oh
<lifeless> we're in testfix right now
<lifeless> what are the odds
<wgrant> Awesome.
<lifeless> 38 mins to go
<lifeless> that should have the fix from sinzui
<wgrant> Ah, sinzui's fix still hasn't made it through?
<wgrant> Right.
<lifeless> so
<lifeless> what flags do you want?
<lifeless> --no-qa?
<wgrant> --no-qa was what was used last time.
<wgrant> So sounds good.
<lifeless> ok, kicked it off
<wgrant> Thanks.
<lifeless> de nada
<lifeless> jelmer: when you get up; https://bugs.launchpad.net/soyuz/+bug/669676 still needs qa, and its blocking further deployments.
<wgrant> It's easy enough to QA that on qastaging like we did last week.
<lifeless> a bit more involved
<lifeless> given we're looking for race conditions
<lifeless> anyhow, we'll be only 4 days back once we deploy up to it
<wgrant> Hm.
<wgrant> I think that has a bug.
<wgrant> Yeah, it looks like it has the same bug that r11812 fixed.
#launchpad-dev 2011-10-31
<lifeless> zopelessappserverlayer
<lifeless> wgrant: you're going to laugh at this
<lifeless> wgrant: in between the tears.
<wgrant> I hope so.
<lifeless> wgrant: ZASL runs up the codehosting server.
<lifeless> wgrant: that calls into private-xmlrpc
<lifeless> wgrant: thats raising oopses because of disconnected stores
<wgrant> ZASL itself shouldn't start codehosting...
<wgrant> Unless it's pretty different from AppServerLayer, which it shouldn't be since I made ZopelessLayer and FunctionalLayer almost equal.
<lifeless> wait, there is more
<lifeless> we also get a disconnectionerrror: terminating connection due to administrator command
<wgrant> Where are you running this?
<lifeless> wgrant: -> because the test db reset code pulls the db out from under the slave appserver
<wgrant> Locally?
<lifeless> wgrant: yes, totally reproducable
<wgrant> You need the new storm.
<lifeless> on lucid ?
<wgrant> Yes.
<wgrant> libpq5 8.4.9 breaks everything.
<wgrant> Destroyed buildbot on Friday.
<wgrant> ec2 isn't affected yet.
<lifeless> this is 8.4.3-1
<wgrant> New Storm is in devel now.
<wgrant> Ah.
<wgrant> Maybe your analysis is correct, then.
<lifeless> maybe :)
<lifeless> the reason it happens only when two tests are run
<lifeless> is that we have a brand new appserver if we only run one test
<wgrant> Yep.
<wgrant> Still, this looks similar to the libpq5 issue...
<wgrant> Exactly the same thing.
<wgrant> librarian was broken after the first test.
<lifeless> how so ?
<lifeless> I thought that was it not detecting the disconnect ?
<wgrant> Right.
<lifeless> this detects it fine
<wgrant> If this is a DisconnectionError, it's probably working.
<lifeless> its raised from storm.database line 376 _check_disconnect
<lifeless> yes, it is detecting it
<lifeless> we're just oopsing - correctly - because the db was yanked without telling the appserver
<lifeless> so I should say, rather than 'codehosting is started' the test using bzr transports and these are backing onto xmlrpc requests in some fashion
<lifeless> and bingo
<lifeless> the db is being dropped after the test, rather than the sequences reset
<lifeless> I'm now totally confident of the analysis, but wtf to do about it
<lifeless> I think I'll expect 1 or 9 oopses
<lifeless> along with an XXX about this
<lifeless> wgrant: as my victim^Wreviewer, how do you feel about this
<wgrant> lifeless: The DB will be dropped if it's dirty.
<wgrant> I think that's probably fine.
<wgrant> As long as it is really 1 or 9 :)
<lifeless> bug 884036
<_mup_> Bug #884036: db layer test resets interact badly with slave appserver instances <Launchpad itself:Triaged> < https://launchpad.net/bugs/884036 >
<lifeless> see if that convinces you
<lifeless> second oops added now
<wgrant> Sounds good.
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/cte-it-up/+merge/80767
<wgrant> 2.5s->1.6s on qastaging
<lifeless> wgrant: does this change +bugs as well? or just BugTask:* ?
<wgrant> lifeless: +bugs as well. Do you want that checked too?
<lifeless> uhm yes.
<lifeless> just cause, well, EPIC TIMEOUTS
<wgrant> It should in theory be a significant improvement. But let's see.
<lifeless> rev 14156 of my branch is up; Could you please incremental review anything you haven't, and I'll land.
<wgrant> Ermm.
<wgrant> This isn't landable...
<wgrant> There is no rabbit on prod, and the fallback is flaky at best.
<lifeless> with no rabbit it will behave as it did befre
<lifeless> *before*
<wgrant> And the moment we have rabbit, all OOPSes will disappear.
<wgrant> Also, BSON.
<lifeless> whats flaky about the fallback ?
<wgrant> If rabbit is up but amqp2disk hasn't set up its queue bindings, OOPSes go to nowhere.
<wgrant> And that's just what I've noticed.
<lifeless> thats how loosely coupled works.
<wgrant> Yes, but it's not how production should work.
<lifeless> Its not a defect, its expected. I'm hoping to get that side of it sorted later today.
<wgrant> eg. qastaging and staging have rabbit, no amqp2disk.
<lifeless> we can disable the config for them. Or get amqp2disk up and live.
<wgrant> Do that before landing.
<lifeless> sure
<lifeless> but whats flaky?
<lifeless> or was that code for 'new world order does not have the same guarantees' ?
<wgrant> It only works if rabbit is entirely unconnectable, and it's not exactly well-tested.
<lifeless> wgrant: its tested in oops-amqp; the tests there pull rabbit out from under it
<lifeless> wgrant: rabbit config isn't, and shouldn't, be the business of the oops producer
<wgrant> Sure.
<wgrant> But as your branch is now, as soon as rabbitmq is live on production then we lose all error reporting.
<lifeless> IFF the queue isn't setup
<wgrant> Which it probably won't be.
<lifeless> separate issue; we can toggle off the oops exchange very very easily.
<lifeless> (in prodconfigs)
<wgrant> lol prodconfigs easily
<lifeless> more easily than landing this
<lifeless> change, push, review, pqm-submit.
<wgrant> lifeless: Why is UnknownRemoteError now permitted in test_runner.py?
<lifeless> wgrant: because python OOM is nondeterministic
<lifeless> wgrant: I ran it in a loop and didn't get MemoryError every time
<wgrant> But that invalidates the test...
<wgrant> Also, isn't the setUpQueue call in sync() racy?
<lifeless> I can put it back, but I expect some buildbot failures as a result
<wgrant> I don't see why it needs a new queue.
<lifeless> wgrant: because auto_delete
<wgrant> They would be legitimate failures.
<wgrant> The current test is not flaky.
<wgrant> Ah.
<wgrant> Fair point.
<lifeless> wgrant: I bet it is
<wgrant> Still racy, but acceptable.
<wgrant> lifeless: It's never been a problem before.
<lifeless> wgrant: poolie ran into a race the other day, last happened 4 years ago.
<lifeless> wgrant: doesn't make it less of a race
<wgrant> Even so.
<lifeless> I can put it back.
<wgrant> It makes it an invalid test.
<lifeless> But I think the test lies today
<lifeless> the test isn't valid
<poolie> lifeless: two races
<wgrant> Better to have it failing and then disabled.
<wgrant> Than left entirely invalid.
<poolie> i guess it is the spring racing carnival in launchpad too
<wgrant> lifeless: Perhaps it's because you were running it i386?
<StevenK> wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it?
<lifeless> wgrant: yes, I am
<lifeless> wgrant: I have no particular interest in this; the runner code doesn't generate MemoryError synthetically, by checking for ulimit violations etc - it depends on the python runtime only ever raising MemoryError AFAICT (grepping for MemoryError ..)
<lifeless> wgrant: so pick a preferred resolution, and I'll do it.
<wgrant> lifeless: Your INTERACTIVE_TESTS change concerns me. Doesn't that mean we're just not going to see any oopses from the appserver?
<lifeless> wgrant: myself, I'm quite sure that we will eventually hit whatever point in CPython caused UnknownRemoteError
<wgrant> lifeless: Sure, but I'd prefer to have a flaky or disabled test, rather than one that doesn't actually test.
<lifeless> wgrant: no tests look for them, and its *only* yuixhr that uses this code path
<lifeless> wgrant: depends on the intent of the test; I think 'raises promptly' is sufficiently narrow.
<wgrant> Oh, it's in testapp, right.
<lifeless> wgrant: we don't depend on catching MemoryError anywhere (damn straight)
<lifeless> wgrant: so the -only- impact on broadening the exceptions is that the test suite lines up with the OOPS reports
<lifeless> wgrant: [if/when we see this in prod]
<wgrant> lifeless: Hm, UnknownRemoteError is probably txamqp...
<lifeless> wgrant: txamp do you mean ?
<wgrant> Yes.
<wgrant> I fail.
<wgrant> So, I'd prefer it if you reverted that change.
<wgrant> If it is a problem, we can unrevert it.
<lifeless> it is in Twisted AMP yes
<lifeless> sure, I will (it is a problem)
<wgrant> It's not a problem.
<wgrant> It may be on i386, but nobody's noticed that before :)
<lifeless> let me just thrash carob's pagecache
<wgrant> Oh?
<lifeless> launchpad.net-logs$ grep UnknownRemoteError . -r
<wgrant> Ah
<wgrant> That seems unwise...
<wgrant> It's like 150GB or more.
<wgrant> Or was it 250GB
<wgrant> I can't quite remember.
<lifeless> serial IO ftw
<wgrant> I still contend that carob's disks are actually a NAS in NZ.
<lifeless> I could grep faster if they were
<lifeless> anything else ?
 * StevenK prods wallyworld_.
<wallyworld_> ouch
<StevenK> [12:00] < StevenK> wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it?
<wallyworld_> sure.
<wgrant> lifeless: I think that's about it.
<lifeless> ok, so I'll push th revert to test_memory_hob_job, toss at ec2, and we'll see what we see
<wallyworld_> StevenK: '%s/+activate-ppa' % canonical_url(team) -> canonical_url(team, view_name='+activate-ppa')
<wallyworld_> other than that, looks good thanks
<lifeless> wgrant: can as approved vote?
<StevenK> wallyworld_: Thank you for the Approve.
<wallyworld_> np.
<wgrant> lifeless: Done.
<lifeless> thanks
<wgrant> lifeless: Erm.
<wgrant> What did I say about not landing this.
<wgrant> I approved mostly given that you were going to get at least (qa)staging fixed first :)
<wgrant> (and no, "It'll be done soon" is not valid)
<lifeless> wgrant: I'm working on that right now
<wgrant> If it's not done by the time ec2 finishes, please kill the ec2 instance.
<lifeless> that would be a waste
<wgrant> Let's not get into another 2-month cart-before-the-horse situation like we did with fastdowntime.
<lifeless> if its not ready I'll change prodconf to disable the key in staging
<lifeless> wgrant: you exaggerate there
<wgrant> No...
<lifeless> wgrant: I'm not about to block the deployment pipeline for any extended period
<lifeless> wgrant: I'm well aware of its impact, given I've done most of the analysis and discussion to demonstrate that
<wgrant> You are about to significantly hinder the crisis analysis process, though.
<lifeless> I'm about to change the exact recipe needed
<lifeless> grep still works, data can still be easily view
<lifeless> ed
<wgrant> No, grep doesn't still work.
<wgrant> I can't grep for ^Exception-Type: RequestExpired now.
<lifeless> yes it does, bson stores strings as length: prefix, so you can do exactly that.
<lifeless> just phrased -very slightly- differently.
<wgrant> I am close to rescinding my approval.
<lifeless> why?
<wgrant> 1) all (qa)staging OOPSes will be lost unless you sort things out in the next few hours.
<lifeless> 1) hystrionics, I've outlined 2 solid workarounds and taken responsibility for having them sorted.
<wgrant> 2) oops-tools is mostly useless, and this breaks shell pipelines.
<lifeless> 2) has had 3? more? weeks for folk to respond to the get-out-and-try aspect, and there are also recipes in that same thread to deal.
<wgrant> We already had three expressions of disapproval.
<wgrant> In addition to my initial one.
<lifeless> aaron asked why; stuart noted about readability with an anecdote from ISD who are doing noone-knows-what, and gary was -0.5 accepting my argument about greppability
<lifeless> I have been trying and playing around, and I'm satisfied I can do the analyses I was previously doing.
<lifeless> Have you experimented, or are you just concerned about the possible impact?
<wgrant> I know that my usual method for analysing performances crises now has to be rewritten in Python, or invoke Python thousands of times.
<lifeless> how do you know that ?
<wgrant> Because it relies on selecting on and extracting fields and parts of fields from OOPSes.
<lifeless> so, its an assumption. Thats ok, but its useful to be clear.
<lifeless> you are assuming, without trying, that grep and sed can't extract what you want from bson.
<lifeless> you may be right, you may not be.
<lifeless> I'm more worried about the url interactions than the change to bson; oops-tools has shocking understanding of urls-are-text
<lifeless> for instance our appserver generated urls are unicode
<wgrant> I'm not too concerned about how it interacts with oops-tools.
<lifeless> so we may end up double encoding for a day or two, which is tolerable but annoying
<wgrant> oops-tools isn't critical, and can be fixed.
<wgrant> lifeless: +bugs seems to be much faster on qastaging, with the CTE patch.
<wgrant> FTI Ubuntu searches are still timing out, but they were before too so I suspect the index is just cold.
<lifeless> ok.
<lifeless> uhm
<lifeless> just for insurance, lets FF this
<lifeless> its a particularly sensitive bit of code
<wgrant> k
<wgrant> It's all about to become a lot simpler, but until then...
<lifeless> I'm sure the FF will be overkill
<lifeless> but I'll feel a whole lot happier having it :)
<wgrant> Indeed.
<wgrant> test_bugtask_search is terrible :(
<wgrant> A slow base test case with dozens of derivatives.
<lifeless> yes
<wgrant> lifeless: It's now behind a flag, with a few strategic tests duplicated to test both paths. The new function is ugly as hell, but at least it should be safe :)
<lifeless> wgrant: thank you very much!
<wgrant> Bah, but now I have to ec2 it :(
<lifeless> what could possibly go wrong
<wgrant> The FF check will probably introduce a new query sometimes :(
<wgrant> Which could break some query limit tests.
<lifeless> most of them have some deliberate fat
<lifeless> with a generous backstop, but yes, it could
<wgrant> I've been meaning to do a test run with the matcher ensuring a lower bound as well.
<wgrant> Some of them are waaaaay high now.
<StevenK> Bah, I can't have my change in create_initialized_view, too many things can't be canonical_url()'d.
<poolie> what's a CTE?
<wgrant> poolie: Common Table Expression
<wgrant> Defined by the WITH clause.
<poolie> ok
<StevenK> wgrant: Is QA for r14209 on your list this afterrnoon?
<wgrant> StevenK: That's the userCanView change?
<StevenK> Yeah
<wgrant> It's semi-ok.
<StevenK> qa-ok-ish
<StevenK> ?
<wgrant> Liveable, but not ideal. I'm hoping that an improvement will land before someone tries to deploy.
<StevenK> We have a large amount of European QA before that rev.
<wgrant> Yes.
<lifeless> http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx is a good read
<poolie> can someone quickly review http://readthedocs.org/docs/judge/en/latest/ for me
<poolie> lifeless: that is interesting
<poolie> it would be interesting to study: how good are developers at predicting which factors drive quality on their projects
<lifeless> indeed
<poolie> what should i do now?
<lifeless> partay
<nigelb> heh
<nigelb> morning! :)
<poolie> hi nigelb
<nigelb> Hey poolie
<spm> o/ nigelb
<nigelb> ohai spm :)
<nigelb> Ugh, I now officially hate the expiring membership email.
<spm> yeah? ok then. INSERT into teams VALUES nibelb WHERE team has expiringing_membership;
<spm> spelling to be optional.
<StevenK> spm: Let me give you a cronjob that will reset nigelb's memberships to expire in seven days. Every day.
<spm> StevenK: approved.
<spm> this wouldn't be excessive abuse of power would it? :sincerity:
<nigelb> hah
<nigelb> StevenK, spm: Love you too :P
<spm> :-)
<lifeless> wgrant: you said txlongpoll was bust
<lifeless> wgrant: in that it didn't accept a queue name prefix, right ?
<wgrant> lifeless: Yes.
<lifeless> is bigjools aware of this? And that its a blocker to the deploy ?
<wgrant> I believe Red is aware.
<wgrant> However, AFAIK they are unaware that anybody is using rabbit except them.
<wgrant> So they are probably unaware that the fix is time-critical.
<lifeless> wgrant: there is no bug on https://bugs.launchpad.net/txlongpoll
<lifeless> wgrant: as you know the details, could you make one please?
<wgrant> Hmm, rvba said he was going to file one.
<wgrant> I left it with him, since it was well after midnight.
<lifeless> the rt claims that read is only on ^longpoll.*
<lifeless> so there is access control in place
<wgrant> That is a start.
<wgrant> Personally I'd prefer the old behaviour to be revived.
<lifeless> I'm not sure what that is; thus my encouragement for you to file a bug :)
<lifeless> hmm
<lifeless> my branch has been running from 0140 through to 0708
<lifeless> I suspect a stupidly overloaded host
<wgrant> lifeless: I just had a branch take 4:45
<wgrant> Which is longer than normal.
<lifeless> hmm
<lifeless> I suspect that that sync() in cleanUp is a problem
<lifeless> I have another trace here that took 6.5 hours
<lifeless> I'll land a branch immediately to remove the trailing sync()
<lifeless> [given I only added it while debugging that rosetta zopelessappserver test issue, I know its not needed know]
<wgrant> Ah.
<lifeless> its the only think I can think of that would add 100% test time
<lifeless> unless the rabbit layer reset is heinous, but I didn't change that, nor add it to more than a few dozen tests
<lifeless> wgrant: I suspect both the sync and the existence/presence of poor ec2 loading
<lifeless> wgrant: did we decide setting mandatory wouldn't work ?
<wgrant> lifeless: We can't necessarily receive the response.
<lifeless> wgrant: did we do an experiment ?
<wgrant> No.
 * StevenK has clearly forgotten how to drive subunit
<wgrant> But we'd need to asynchronously detect the basic.return, and then $something.
<wgrant> But I don't think amqplib exposes basic.returns.
<StevenK> I have a subunit stream and I want to see which tests failed or gave an error
<lifeless> StevenK: subunit-filter --no-skip --no-passthrough | subunit-ls
<lifeless> StevenK: or testr load < stream; testr failing
<lifeless> (or testr failing --list)
<lifeless> StevenK: or tribunal < stream
<StevenK> subunit-filter --no-success --no-passthrough --error --failure < lp.log | subunit-ls
<StevenK> That gives me a traceback
<lifeless> StevenK: you're making it hard for yourself :)
<lifeless> StevenK: 3 of those options are the defaults
<lifeless> StevenK: also whats the tb you get?
<StevenK> Broken pipe
<lifeless> subunit-filter is probably erroring
<StevenK> So one of subunit-filter or subunit-ls is misbehaving
<lifeless> try one of my recipes above
<lifeless> StevenK: ?
<StevenK> Sorry, got distracted. Your recipe works fine
<StevenK> "subunit-filter --no-success --no-passthrough --error --failure < lp.log 1>/dev/null" doesn't produce any output on stderr
<lifeless> its a legitimate set of options
<lifeless> I'm not sure why you had a tb
<lifeless> conceivably the pipe was closed with no output I guess
<lifeless> it works for me, fwiw
<StevenK> lifeless: http://pastebin.ubuntu.com/724060/
<lifeless> wgrant: so, you'll file a bug| get rvba to ?
<lifeless> StevenK: the first tb is the issue
<wgrant> lifeless: I'll ask rvba for the latest news.
<lifeless> StevenK: Can you reproduce with a current subunit ?
<lifeless> StevenK: (what does dpkg -l python-subunit say)
<StevenK> lifeless: 0.0.6-3
<lifeless> try 0.0.7
<StevenK> Such a pity, Mr Bond.
<nigelb> heh
<jelmer> lifeless: that reminds me, can you add a tag for 0.0.7 on lp:subunit?
<lifeless> jelmer: probably, there isn't one ?
<lifeless> jelmer: it'll be just tip
<lifeless> jelmer: :!bzr log -r -1
<lifeless> ------------------------------------------------------------
<lifeless> revno: 152
<lifeless> tags: 0.0.7
<jelmer> lifeless: "bzr tags -d lp:subunit" doesn't list a 0.0.7
<jelmer> lifeless: perhaps that tag is local?
<lifeless> jelmer: if so, I blame bzr
<lifeless> jelmer: hmm, diverged branches. funky.
<lifeless> yes, somethings up. I'm going to push --overwrite and then merge trunk back in
<lifeless> jelmer: there should be a tag now
<jelmer> lifeless: great, thanks!
<lifeless> useoops has landed \o/
<wgrant> Has the fix landed?
<bigjools> My next branch is going to be called The Eagle
<nigelb> Just to announce its landing?
<bigjools> correct
<nigelb> I'll call dibs on Airforce one and Canonical one ;)
<nigelb> Who's at UDS? francis, matt, dan..and?
<bigjools> that's about it IIRC
<bigjools> oh one of the US guys too
<adeuring> good morning
<lifeless> wgrant: its about to start ec2
<wgrant> nigelb: I thought it was Francie, Matt, Deryck and Curtis. Didn't know Dan was going too.
<lifeless> new kid, doing the rounds :)
<lifeless> bigjools: is rvba in today?
<bigjools> lifeless: doesn't look like it
<nigelb> bigjools / wgrant - Thanks :)
<bigjools> I'd quite like to be there again, was the best UDS hotel in ages
<nigelb> heh
<nigelb> I wish I could be there. I regret skipping this UDS>
<nigelb> bigjools: Will you be at the next one? :)
<nigelb> Too bad I missed you last UDS.
<bigjools> you walked right past me there :)
<nigelb> :(
<nigelb> That's the UDS where I asked Francis "Are you a launchpad guy?" :D
<bigjools> as for the next one, no idea, we don't decide until shortly before who's going
<nigelb> Ah!
<\sh> moins
<\sh> dear lp-devs how difficult will it be, to change someones launchpad id from X to Y ?
<nigelb> Its easy to do the change.
<nigelb> THe repurcusions might make some time to fix ^-^
<bigjools> \sh: the answer is "it depends"
<nigelb> And its not a launchpad problem. Its an SSO thing.
<wgrant> SSO actually steals that data from us.
<wgrant> It's a Launchpad thing.
<wgrant> \sh: Unless you have a PPA, you can just change it.
<\sh> bigjools, let's say, I would like to request to change my launchpad-id from shermann to sadig
<bigjools> \sh: then as wgrant says
<nigelb> wgrant: Meh. I think most of the time its our openid client which isn't implemented properly.
<\sh> wgrant, ok...my ppa is not a problem...so when I delete my ppa I can just change it?
<bigjools> if you have any PPAs then they need to be deleted
<bigjools> then yes
<wgrant> \sh: Yes. May have to wait up to 10 minutes for it to be completely deleted.
<lifeless> \sh: delete the ppa, wait for a publishing cycle, visit +edit and rename it
<lifeless> I think its +edit
<\sh> wgrant, lifeless , thx...I'm waiting now ;)
 * bigjools has a buildd-manager that will cancel builds \\o/
<StevenK> bigjools: Does it also impact on the production builders too :-P
<jelmer> bigjools: niiice!
<wgrant> bigjools: You know that's been possible for recipe builds for a year, right? :P
<bigjools> whatever
<lifeless> running out of memory != cancelling :)
<bigjools> lol
<nigelb> ZING.
<wgrant> They have a cancel button which sets them to SUPERSEDED. Which actually works pretty well.
<bigjools> yes but it doesn't rip the build off the builder
<jelmer> wgrant: that doesn't kill the in-progress build though, right?
<bigjools> we can set it to CANCELLED now
<wgrant> I believe it deletes the BQ too, which would kill the in-progress build.
<nigelb> So, the ones that are scheduled can be canceled?
<nigelb> Or is it the ones that are building that can be canceled.
<bigjools> interesting way of doing it
<bigjools> lifeless: yay for the OOPS changes!  Did you document what you said in the email in the dev wiki?
<lifeless> bigjools: all relevant docs were updated
<lifeless> bigjools: AFAIK
<lifeless> bigjools: however, there weren't many ;)
<bigjools> awesome
<lifeless> bigjools: https://dev.launchpad.net/QA/OopsToolsSetup is perhaps the least likely known by ye old average LP dev
<bigjools> well - the main thing is to note the stuff you wrote in the email, for posterity
<bigjools> I'm going on a documentation jihad in the near future
<lifeless> bigjools: I've given https://dev.launchpad.net/LoggingOopses a bit of a facelift too
<bigjools> lifeless: great!
<lifeless> ok, thats witching hour, time to EOD for a bit :)
<nigelb> aaah.
<nigelb> I now understand the outcome of comparing unicode incorrectly.
<nigelb> The unicode entry goes to the end of the list :(
<wgrant> Yep.
<nigelb> Nice that danilos is our unicode tester :D
<nigelb> Is there a better way to do that than what I did?
<wgrant> Sadly we are left with only rvba.
<wgrant> And his name is not very Unicode at all.
<nigelb> heh
<nigelb> Is there better way to sort unicode at the current point?
<nigelb> I remember jtv's argument at that time being more of future oriented.
<lifeless> need unicode local in the DB and appservers
<lifeless> presumably not /hard/
<lifeless> but needs doing + qa
<nigelb> ah
<nigelb> Nothing I an do I suppose?
<nigelb> *can
<cjwatson> bigjools: cancelled> ooh, awesome
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugtasks: 266
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, benji | Critical bugtasks: 266
<lifeless_> nigelb: you could do it
<lifeless_> nigelb: but indirectly; you'd need to file bugs / rt's to find out the exact current config, reproduce it, see what you want, test that, get confidence etc
<nigelb> ah
<bigjools> cjwatson: only for virtual builds right now though, don't get too excited
<bigjools> need to fix sbuild for non-virt
<cjwatson> bigjools: ah well, still a step forward
<bigjools> yes, 90% of work is done
<bigjools> s/is/will be/ :)
<bigjools> LinkedIn - just PO please
<cjwatson> Wow.  The publisher has only just got to around cron.germinate
<cjwatson> We're in serious danger of going past an hour here :-/
<bigjools> cjwatson: :(
<jcsackett> sinzui: free to mumble?
<sinzui> yes
<jcsackett> excellent.
<danhg> Hey everyone at UDS, I'm Dan, the new Usability & Communications Specialist at Launchpad. I'm going to be doing lots of usability testing today, so please ping me if you're around, and would like to get involved!
<nigelb> danhg: #ubuntu-uds is also a nice channel to advertise :)
<danhg> thanks nigelb, good plan!
<danhg> All week actually - so let me know when you're free guys!
<bigjools> cjwatson: yeah, it missed the run at :00
<bigjools> domination took 30m
<bigjools> I figured this would happen with the arch-all changes
<cjwatson> this is pretty bad - what can be done?
<cjwatson> don't get me wrong, I appreciate the new feature, but ...
<mrevell> jcsackett, yo
<jcsackett> mrevell: hello.
<mrevell> danilos, What's the rally project management session later all about?
<mrevell> jcsackett, Oh hey, we're just trying to find a good time to talk to you. When suits you?
<jcsackett> pretty much anytime today is fine. i imagine your schedule is a bit more bumpy than mine.
<jcsackett> mrevell ^
<jcsackett> mrevell: danhg has proposed now, which works for me.
<danhg> Cool
<mrevell> jcsackett, Oh hey, we're just trying to find a good time to talk to you. When suits you?
<jcsackett> mrevell: i think you meant perhaps to send another message, as that one's a repeat? :-P
<mrevell> jcsackett, I have no idea how that happened. Hah :)
<mrevell> jcsackett, I think danhg is talking to you about it.
<jcsackett> he is indeed.
<jml> awk: (FILENAME=lib/lp/contrib/javascript/yui3-gallery/gallery-accordion/gallery-accordion.js FNR=2916) fatal: cannot open file `lib/lp/contrib/javascript/mustache.js' for reading (No such file or directory)
<jml> make: *** [lib/canonical/launchpad/icing/build/launchpad.js] Error 2
<jml> just got that running 'make schema'
<bigjools> jml: update source?
<lifeless_> *yawn*, morning
<jml> hello, sorry, was away
<jml> really really want to get a working lp dev environ
<lifeless_> jml: utilities/update-sourcecode
<jml> lifeless_: done that.
<lifeless> jml: and still doesn't want to play?
<jml> lifeless: ahh, I know
<jml> forgot to run link-external-sourcecode
<jml> which appears to make the right symlink. /me makes schema again.
<jml> ... and it builds. thaks.
<lifeless> heh, de nada
<nigelb> Morning lifeless
<jml> actually, I should say what I want to do
<jml> I want to link to apps.ubuntu.com from Launchpad. Specifically getting package pages to link to relevant apps pages.
<lifeless> o/ nigelb
<nigelb> Well, I guess I should head to bed :)
<jml> I *think* I can get by with just a text transform, no queries.
<lifeless> \o/ test perf fix worked
<lifeless> Tests started at approximately 2011-10-31 08:45:30.145372
<lifeless> Source: bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/useoops r14214
<lifeless> Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ r14213
<lifeless> 16646 tests run in 4:33:15.575714, 0 failures, 0 errors
<lifeless> not as svelte as perhaps it was in the past, but not 7 hours either
<jml> Is there a fast way of consulting Contents.gz from Launchpad code? (Is there a version in the db?)
<jelmer> jml: the archivepublisher generates it and I'm pretty sure it doesn't store it in the database
<jelmer> jml: what are you trying to do?
<jml> jelmer: link to apps.ubuntu.com for a package from packages on Launchpad
<jml> jelmer: but apps.ubuntu.com has only apps, which are packages that have desktop files
<jelmer> jml: ah, and you need the section and the like?
<jml> jelmer: no, just to know whether or not it has a desktop file
<jml> e.g. https://apps.ubuntu.com/cat/applications/oneiric/evolution/ vs https://apps.ubuntu.com/cat/applications/oneiric/openssh/
<lifeless> jml: there is a version being populated in the DB
<jelmer> jml: do .desktop files necessarily indicate applications? IIUC it's possible to have .desktop files for non-application things
<lifeless> jml: StevenK knows the status of that
<jml> jelmer: that's how they make the distinction in Software Center, I'm told.
<lifeless> jml: My first thoughts about how to implement this are:
<lifeless>  - what about derived distros ?
<lifeless>  - and if software centre knows, why not consult with it (e.g. from client js, or even from the appserver) to see whether you can link to it safely
<jml> lifeless: I only want this for Ubuntu, not for derived distributions.
<james_w> software-center has some fudges to determine the list of apps I believe, some are ignored, others are added
<lifeless> jml: poking at contents because thats what SC uses seems like an abstraction violation to me
<jml> lifeless: yeah, that's another option. More implementation work than using a table LP already has, but maybe more feasible. Would probably have to add that API.
<james_w> I wonder if apps.ubuntu.com will grow the "technical items" thing too
<lifeless> jml: this is so folk can find out how to install the package or something?
<jml> It's hard enough writing code for what exists now. Dealing with possible futures is going to make this impossible.
<lifeless> jml: a different approach would be to embed the content you're interested in on the relevant LP pages. E.g. reviews.
<jelmer> jml: is software center using contents.gz directly though? I thought all that data was pre-generated in app-install-data
<lifeless> jml: In my head there are 2 services (LP and SC) and 2 UI's (LP and SC) here, so having LP's UI talk to SC's service for some stuff is totally fine.
<lifeless> jml: (if SC is fast enough)
<jml> lifeless: largely the motivation is to increase google juice of apps.ubuntu.com. I guess it could also help developers using LP to quickly get a sense of what their package looks like to end users.
<jml> lifeless: would look at reviews stuff in a later iteration.
<jml> jelmer: quite possibly.
<jml> lifeless: agree re abstraction violation. Doesn't feel a bad enough one to stop me though. Difficulty of implementation is more persuasive
<lifeless> jml: such violations create debt instantly
<lifeless> jml: I'd rather we not link at all than do it poorly
<jml> lifeless: right, but I don't even have a working prototype that I can improve at this point
<jml> jelmer: does that change anything?
<jelmer> jml: does what change anything? the fact the data is pregenerated?
<jml> jelmer: yeah
<jelmer> jml: I suspect there is more going on than just looking for .desktop files
<jelmer> jml: probably filtering out things that are not for applications
<lifeless> jml: I'm curious - why does apps. need google juice? I thought it was mainly a thin shell for its API used by the sc client on Ubuntu installs ?
<jelmer> but that's all speculation
<jml> jelmer: OK. It's definitely possible that I was misinformed.
<jml> jelmer: certainly using some kind of provided API would avoid that.
<jml> lifeless: It turns out that a lot of people don't actually look at the Software Center very much. People fresh from other platforms often just open a browser to search for software, rather than looking for Software Center (or an app store or what have you). When they do that search, we want them to find http://apps.ubuntu.com â the web directory for the users.
<jelmer> jml: it seems like a bad idea for software center to have to download a 20Mb file every time something in the archive changes
<jml> lifeless: (paragraph from an email I was drafting)
<lifeless> jml: interesting
<jml> lifeless: yeah.
<jml> lifeless: linking from LP is not the beginning & end of our approach, but it's something that would probably help.
<jml> jelmer: well, I get the very strong impression that mvo would prefer some debtags-based solution.
<jelmer> jml: debtags seems like a better approach for this kind of thing than inspecting Contents.gz
<jml> jelmer: I am thinking of volunteering to add debtags support. Sort of contingent on how hard it is to add a hyperlink to a web page though :)
<jml> hmm.
<jml> I guess a naive-but-restful API is to just GET the link and then see if it 404s
<jml> there's probably a way to do that without dl'ing the whole page too
<jml> wonder how late I can render the link and still get SEO
<jml> (also, battery about to fail)
<lifeless> jml: google execute js
<lifeless> jml: so I expect quite late
<lifeless> sinzui: hiya
<sinzui> hello
<lifeless> sinzui: I think we have a bug in team administration :)
<lifeless> sinzui: see #launchpad for context. Seeking your opinion.
<benji> sinzui: are you aware of a way I can postpone a user's team membership expiration date?  This is in reference to this question: https://answers.launchpad.net/launchpad/+question/177021
<lifeless> benji: needs a duckie
<benji> lifeless: come again?
<lifeless> ~admin ;)
<lifeless> the logo is, or was, a yellow rubber duck
<benji> lifeless: I think you're saying that I should be made a member of some administrative team, ~admin doesn't seem to be it though (https://launchpad.net/~admin).  If so, who can give me such membership?
<lifeless> benji: I was unclear; I mean you cannot; you have to get an admin to do it.
<benji> ah!  (where, I presume admin == LOSA)
<lifeless> yes
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<lifeless> \o/ instant OOPSes on qastaging.
<lifeless> booyah
<lifeless> flacoste: \o/ instant OOPSes on qastaging.
<lifeless> booyah
<flacoste> lifeless: awesome
<lifeless> try it, its spicy ;)
<beuno> lifeless, I like the word instant. Is it using rabbit or something like that?
<lifeless> beuno: yes
<beuno> lifeless, I WANTS
<beuno> so you've been talking to rye, right?
<lifeless> yes, he's production hardening the opened code base for your environment
<lifeless> there is a wsgi enhancement needed for your tx stack before you can widely migrate
<beuno> awesomeness
<lifeless> beuno: see b.l.n/python-oops-wsgi
<poolie> morning
<wallyworld_> jcsackett: r=me but i think the template still references the old public_or_private() method name
<jcsackett> wallyworld_: yes. yes it does. i'll fix that.
<wallyworld_> thanks :-)
<wallyworld_> jcsackett: i assume the stuff you will be sending through contains the necessary detail to understand what needs change wrt the searching and making it clearer when things are deleted etc?
<jcsackett> wallyworld_: indeed.
<wallyworld_> jcsackett: the fwd email is missing the original material
<jcsackett> wallyworld_: oh fantastic.
<wallyworld_> :-)
<wgrant> lifeless: https://pastebin.canonical.com/54976/ is what we used to fix the the BPBs.
<wgrant> A similar thing over SPRBs would be good.
<wgrant> (I'm not here today)
<lifeless> wgrant: ah, it is damage ?
<wgrant> lifeless: Yes, failure counting except buggy.
<lifeless> wgrant: qastaging. Rabbit. AMQP. OOPS. WIN WIN WIN
<wgrant> lifeless: It works?
<lifeless> hell yes
<wgrant> Excellent! Have we stopped rabbit to see that it falls back?
<lifeless> not yet
<lifeless> been doing uds stuff
<lifeless> wgrant: still, you should make qas oops and see the result.... its just nice
<wgrant> lifeless: I already have :)
<wgrant> It is indeed.
<wgrant> oops-tools needs some major hacking, but that can be done soon.
<wgrant> lifeless: Is https://pastebin.canonical.com/55154/ a yesterday-style query?
<wgrant> lifeless: ie. EXPLAIN ANALYZE is instant, execution is 300ms?
<lifeless> qaqs ok?
<wgrant> Yeah.
<lifeless> no limits  ?
<wgrant> None.
<wgrant> Should be 1 row.
<lifeless> 300ms
<lifeless> 323
<wgrant> And explain analyze?
<lifeless> 162
<lifeless> 162
<wgrant> It's 1ms on DF :/
<lifeless> 0.6ms to profile
<wgrant> lolnot
<wgrant> It's not even a very unpleasant plan at all.
<lifeless> ugh
<lifeless>   AND (TeamParticipation.team = BugSubscription.person)
<lifeless>   AND (Person.id = TeamParticipation.team)
<lifeless> wtf
<lifeless> oh, via TP, I guess.
<lifeless> not the join I expected.
<wgrant> StevenK: Can you QA or ignore jtv's notification refactor?
<lifeless> wgrant: in fact, that will give N rows per subscription
#launchpad-dev 2011-11-01
<StevenK> wgrant: I was going to do my own QA first.
<lifeless> wgrant: one per person in the team
<wgrant> lifeless: No.
<wgrant> lifeless: TeamParticipation.person is constrained to me. And Person is contrained to TeamParticipation.team
<wgrant> So it's finding all of my teams that have subscriptions
<lifeless> to bug 1234
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<wgrant> Right.
<lifeless> nested loop on TP though, which is a nuisance
<lifeless> if we're right
<lifeless> can teams have multiple subscriptions ?
<lifeless> ah it sthe order by
<lifeless> wgrant: will 8ms do ?
<wgrant> lifeless: Nah.
<wgrant> What'd you change?
<lifeless> couple of things
<lifeless> CTE
<wgrant> srsly?
<lifeless> plus narrower distinct by nested tables
<wgrant> For TP again?
<lifeless> https://pastebin.canonical.com/55155/
<lifeless> just checking on wildcherry
<lifeless> 28 and 11
<lifeless> checking old query
<lifeless> old query is 130ms on wildcherry consistently
<lifeless> 11ms new
<lifeless> consistently
<lifeless> wgrant: anyhow, context switching if you're finished with me ?
<wgrant> That indeed looks better. Odd.
<wgrant> Thanks.
<lifeless> https://pastebin.canonical.com/55156/
<lifeless> is the new plan
<lifeless> wgrant: I'm not sure the distinct is needed
<lifeless> wgrant: unless teams can have multiple subscriptions to a bug
<wgrant> I'm not sure.
<lifeless> wgrant: the reason for the CTE is the 105 teams all did table walks on TP
<wgrant> Hah
<lifeless> if you look at the original plan
<wgrant> But we have to guess at what's expensive. And I can't test fixes :(
<wgrant> Because
<wgrant>  Total runtime: 0.693 ms
<lifeless> who is reviewin today ?
<lifeless> StevenK: could you spare time to do a review?
<lifeless> https://code.launchpad.net/~lifeless/python-oops-datedir-repo/less-rsync/+merge/80861
<StevenK> lifeless: Can do -- I was noming, sorry
<lifeless> StevenK: food? zomg NOOOOOOO
<lifeless> [thanks]
<StevenK> lifeless: r=me
<lifeless> thanks
<lifeless> I have another in a sec :)
<StevenK> lifeless: And wgrant is OCR today, but there is some silly horse race on.
<lifeless> yeah, wiki was down when I asked
<lifeless> StevenK: https://code.launchpad.net/~lifeless/python-oops-tools/amqp-polish/+merge/80864 in a minute or two
<lifeless> StevenK: and thanks!
<lifeless> StevenK: diff is up
<StevenK> I've already read it, just thinking
<lifeless> I can do a test for the upper if you want
<lifeless> though its one of those things that isn't going to break once its done
<StevenK> That's not my concern
<lifeless> :)
<StevenK> My concern is if there is a neater way to do it
<lifeless> we could reindex the 27M oops ids in the db with a functional index on upper()
<StevenK> Ew
<StevenK> Approvinated
<lifeless> thanks!
<lifeless> have you had an instant-oops yet ?
<StevenK> I've seen the new ids
<lifeless> heh, close :)
<StevenK> Personally, I think lp-oops.c.c needs to be better before I stop ssh'ing into carob to see OOPSes
<lifeless> its pleasant to hack on now
<lifeless> 6s test suite
<lifeless> etc
<StevenK> lifeless: Can I take output from subunit-ls and say to bin/test please run those
<lifeless> StevenK: yes
<lifeless> thats what testr run --failing does
<lifeless> the machinery is:
<lifeless> subunit-ls > list-file
<lifeless> bin/test --load-list list-file
<StevenK> I wonder if bin/test will like /dev/stdin as a list file
<lifeless> StevenK: it might, but ewww ;)
<lifeless> StevenK: you know you're reinventing testr by hand, right (AFAICT)
 * StevenK tries it, just to make lifeless a little greener
<StevenK> Yeah, maybe I should just testr init, testr load, etc
<lifeless> ok, so thats deployed out; next thing, the workarounds for oops-amqp
<lifeless> almost need a fixed-amqplib
<lifeless> but I'm not quite there yet
<StevenK> Argh, model code updated by triggers
 * StevenK claws his eyes out
<lifeless> StevenK: bug heat ?
<StevenK> lifeless: Branch.unique_name
 * StevenK waits for his branch to stop WADLing
<lifeless> quack quack
 * StevenK grumbles.
<StevenK> I suspect this branch is impossible with FDT.
<StevenK> jtv: O hai, can you QA r14202?
<jtv> Hi StevenK
<jtv> I'll get to it in a moment
<wgrant> StevenK: Are you still trying to rename +junk without talking to mrevell and others?
<poolie> StevenK: i would like to be involved in planning what happens there
<wgrant> Renaming it is technically trivial.
<wgrant> Every other aspect of it is not trivial.
<lifeless> mostly trivial
<lifeless> depending on the new name +  backward compat desires
<poolie> i think we ought to at least think about the larger story of how projects get started and grow
<poolie> some things in junk are purely personal
<lifeless> poolie: I think that hinders the local improvement
<lifeless> poolie: of just having a better na,e
<poolie> others are nascent larger projects
<wgrant> A local improvement should be hindered.
<nigelb> Hey, can SSO grab from lp if a person is remote or not?
<wgrant> Because it is a large backwards-compatibility burden.
<nigelb> (for a sprint)
<lifeless> wgrant: only if it breaks +junk
<poolie> lifeless: thinking about it does not hinder it
<lifeless> wgrant: which isn't presumed
<wgrant> lifeless: We now have to support +junk forever.
<poolie> it's been like this for ~5 years
<wgrant> lifeless: Adding a new name means we have to support the new name forever too.
<lifeless> I shouldn't have said anything. I have work to do :)
<poolie> congratulations on getting realtime oopses done
<poolie> btw
<lifeless> thanks
<lifeless> StevenK: care to do https://code.launchpad.net/~lifeless/python-oops-amqp/0.0.4/+merge/80865 as well ?
<lifeless> StevenK: diff is there
<StevenK> wgrant: So its apparently pointless to do, then?
<StevenK> poolie: Involved how?
<poolie> StevenK: when lp changes things like this that affect bzr, i'd appreciate being told beforehand
<nigelb> +junk is going away?
<StevenK> It is not.
<lifeless> no
<nigelb> Or going to be called something "better"
<lifeless> but there is broad consensus that its a horrible name
<poolie> i'm heartily in favour of improving it and of looking for a small change that will improve it
<poolie> what are you planning to do?
<nigelb> heh, agreed.
<StevenK> poolie: I'm planning on changing the code to return +personal by default, but accepting both +junk and +personal.
<poolie> StevenK, we write documentation about launchpad and we answer a lot of questions about how to use it
<poolie> and we make design decisions that work with launchpad
<StevenK> However, I'm not even sure if it's landable
<poolie> so i think it's reasonable not to be surprised by changes there
<StevenK> Since there is a trigger involved.
<poolie> ok
<poolie> is this in an mp? i don't see any mail about it
<lifeless> StevenK: some care needed to check interactions with e.g. the apache rewrite map
<lifeless> StevenK: I think that uses that column
<StevenK> It's local only at this point.
<poolie> that sounds like a good step though
<poolie> a great step in fact
<lifeless> I think we should have a brief discussion with mrevell about the new url component
<lifeless> I'm in favour of something like 'p'
<lifeless> may as well make it not fugly
<StevenK> +p ?
<lifeless> no
<lifeless> p
<StevenK> We don't allow single character projects, do we?
<lifeless> exactly.
<StevenK> I'm not sure about p, TBH
<lifeless> p might be better reserved for project branches or something, but you get the idea
<lifeless> real-short-and-simple
<lifeless> I don't think + makes for a nice UI, never have.
<poolie> i think according to https://dev.launchpad.net/LaunchpadEnhancementProposalProcess you need to write a lep for it
<lifeless> StevenK: https://code.launchpad.net/~lifeless/python-oops-amqp/0.0.4/+merge/80865
<StevenK> So sinzui said on the call that +personal seems to be the prefered name, and that I can have a free rs=sinzui so he copes the flak and I don't. :-P
<StevenK> lifeless: Shall I just wait on IRC for your next MP?
<lifeless> StevenK: please
<lifeless> StevenK: that would be awesome
<StevenK> lifeless: Your timing the last two times has been impeccable
<lifeless> StevenK: blame your squadmate :)
<StevenK> I just swap my own work back in and I get pinged again.
<StevenK> I think I'll blame Melbourne.
<lifeless> StevenK: hey, I pinged this one 17 m back, but you were distracted :)
<StevenK> Oh, that one.
<StevenK> I already had it open, I was thinking.
<StevenK> lifeless: Approved, with a comment.
<poolie> :/
<StevenK> poolie?
<lifeless> poolie: I think you're asking for more from LP devs than they ask from bzr here, and I don't understand why.
<lifeless> poolie: both project cooperate, have a reasonable understanding of the other (at least at team mgmt levels) so they ask when they see impact, and don't when they don't.
<poolie> yeah my unhappiness is basically that they don't ask when they see an impact
<poolie> or perhaps they don't see an impact
<StevenK> poolie: How does the personal namespace impact on bzr?
<poolie> for instance, we have documentation that refers to +junk
<lifeless> So, imagine that we accept +personal and +junk, and advertise +personal. That obviously needs to be socialised amongst everyone doing support, and all our useres.
<lifeless> but its compatible; any docs won't be invalidated, they will keep working.
<poolie> yeah
<StevenK> Exactly
<poolie> i guess that seems a bit pedantic to me
<poolie> and it would be easy and more courteous to just say 'we're going to change this' in advance
<StevenK> poolie: Er, so I didn't even know if it was *possible* to change it until 4pm yesterday.
<StevenK> To be brutually honest, what I have is a play branch
<lifeless> would you tell LP if you added a new verb with no particularly surprising semantics in it, that was backwards compatible with old clients etc?
 * StevenK throws his Dell laptop out the window
<lifeless> StevenK: how will you play WoW ? Oh right ...
 * StevenK has removed that from his machines
<StevenK> Besides, I hated playing WoW with a trackpad
<poolie> lifeless: i guess that example means "that had no changes that are likely to impact lp"
<poolie> the difference is that changes to lp's ui/user model do impact us, because we document and support it
<poolie> StevenK: np, i don't really mind this case (and again, i'm happy to see it improved) it just seemed like a bit of a pattern
<lifeless> poolie: what data points are on this pattern? Last time we spoke about this feeling you were hard put to pin it down.
<lifeless> poolie: it *seemed*, as I remember it, to be mainly feeling that LP did stuff without talking about it : but when we compared notes team bzr wasn't communicating (in advance) any more fully
<poolie> i don't think it's about who's doing better or worse
<lifeless> poolie: I don't mean to frame this as what you do vs what lp does, though it seems I'm taking it that way
<poolie> but rather, which things are useful to communicate about
<poolie> if lp changes the namespace for branches that is the kind of thing i'd like to hear about
<StevenK> lifeless: Can we have a quick skype?
<lifeless> StevenK: sure
<poolie> if there's any thing we're changing that you wish you'd heard about earlier, let me know
<lifeless> poolie: I trust that (I track what you are doing enough, and that you make good decisions) to the extent that whatever falls through the cracks, falls through the cracks.
<poolie> yeah i normally do too
<poolie> i was surprised by this because i didn't see anything about it on a bug or mp
<nigelb> Hi, so +temp-meeting-export will not give a subscription information if someone is not registered for a sprint. Is there a chance I can remove that check?
<jtv> StevenK: I need to make dogfood or staging go through the motions of notifying about a package sync or uploadâ¦  I can't sync Ubuntu packages on dogfood any more; what would I need to do to elicit an email notification from an upload?
<StevenK> jtv: OTP
<jtv> ok
<lifeless> nigelb: I believe summit is the only user, so sure, though the data size can be pretty big on popular blueprints
<nigelb> lifeless: It has always been big. we hit issues with the 2 sprints thing this time.
<nigelb> Some BPs were only registered to LDS.
<nigelb> I'll change it after UDS, lest people murder me.
<lifeless> StevenK:         result = store.find(
<lifeless>             (Branch.id, Branch.unique_name),
<lifeless>             Branch.unique_name.is_in(prefixes),
<lifeless>             Branch.transitively_private == False).one()
<nigelb> Oh.
<nigelb> I just noticed rf-* is symblinked to the binary in devel.
<jtv> StevenK, wgrant: mind if I upgrade dogfood?
<poolie> when are builds deleted? like https://code.launchpad.net/~neon/+recipe/project-neon-kdesdk/+build/12437
<poolie> are they gc'd a while after they complete?
<wgrant> jtv: No.
<jtv> Thanks.
<wgrant> poolie: they're meant to never be deleted.
<wgrant> And I think I convinced code that deleting them was not going to work.
<poolie> so the user must have deleted them?
<wgrant> So IIRC they shouldn't ever be deleted.
<poolie> or the urls are just wrong
<wgrant> What linked to that?
<poolie> https://bugs.launchpad.net/launchpad-buildd/+bug/693524
<_mup_> Bug #693524: Daily builds fail because of insufficient memory <escalated> <linaro> <recipe> <Launchpad Auto Build System:Triaged> < https://launchpad.net/bugs/693524 >
<wgrant> Perhaps they renamed the recipe?
<wgrant> D:
<wgrant> SPRB.destroySelf exists.
 * wgrant goes stabbity stab.
<wgrant>         for release in releases:
<wgrant>             release.source_package_recipe_build = None
<wgrant> tergjkeriogjeroigteriogjioergioernjegkl
<wgrant> 23FTRWE89HGN A9PRGHERIUHGER
<wgrant> Yes that's right let's just erase the auditing information
<wgrant> WHY NOT
<poolie> :)
<poolie> wgrant: what would be the most practical way for me to get the versions of bzr and bzr-builder in to the build log file
<poolie> to make bzr-builder print them?
<wgrant> I'm not sure.
<wgrant> I would hesitate to suggest that a bzr-builder upgrade would be the easiest thing.
<poolie> meaning "it probably is, but only slightly" or "it's probably not?"
<wgrant> I would have said it was, until we ran into this recent trouble.
<wgrant> Now it's possibly easier and safer to fix lp-buildd.
<lifeless> ftr this is the bug - https://bugs.launchpad.net/launchpad/+bug/147407
<_mup_> Bug #147407: Junk sounds too harsh <lp-code> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/147407 >
<poolie> ftr my bit is https://bugs.launchpad.net/launchpad/+bug/884092
<_mup_> Bug #884092: source recipe build logs should mention versions used <Launchpad itself:Triaged> < https://launchpad.net/bugs/884092 >
<jtv> StevenK: not having much luck testing that revision.  Can't even upload a deb to dogfood.  :(  Since my branch doesn't change any functionality, I think I'll go with "qa-untestable."
<jtv> (The upload failure is something I've seen before â worth chasing down, but probably not worth holding up landings for)
<poolie> so does buildd/buildrecipe somehow get copied to buildrecipe.py, which seemsto be the name that isactually run?
<poolie> ok https://code.launchpad.net/~mbp/launchpad/884092-buildd-versions/+merge/80870
<poolie> stevenk, wrgant, how could i usefully locally test a change to buildrecipe?
<lifeless> poolie: there is a makefile in buildd, or a debian.rules - something
<poolie> yeah, even some documentation
<nigelb> wgrant: OCR-ing today?
<wgrant> nigelb: Public holiday.
<wgrant> So I'm mostly not here.
<nigelb> wgrant: HA, what's the holiday for? (its holiday for us too)
<wgrant> Melbourne Cup
<wgrant> Because horse races are awesome, apparently.
<nigelb> Horse race?
<nigelb> lol
<nigelb> Ours is slightly more reasonable. State formation anniversary thingie.
<nigelb> wgrant: Do you get holidays for cricket matches as well? ;)
<wgrant> Nope.
<nigelb> Sad.
<nigelb> That obviously meant more hlidays.
<nigelb> No more make lint?
 * nigelb moves to right folder.
<nigelb> Ugh, http://paste.ubuntu.com/725049/
<nigelb> Tried out +temp-meeting-export locally.
<nigelb> Ah, that was my fail :)
 * wallyworld_ off to see Cold Chisel :-)
<nigelb> Ew. This was done very badly.
<nigelb> For the meeting export, it gets all the sprint attendees and then gets people from it.
<nigelb> Could someone give some guidance on potential perfomance?
<nigelb> Currently, the meeting export fetches all the attendees for a summit and filters the people attending from that.
<lifeless> yup
<nigelb> so, the meeting export page is sort of fast.
<lifeless> this is an important characteristic
<nigelb> But the chnage I want kinda screws it up.
<nigelb> And I'm hesitant to do that :)
<lifeless> fair enough too :)
<nigelb> I wondering if we could add people to list as I find them
<nigelb> with the details I want
<lifeless> whats the big picture here
<nigelb> and build the list of people based on the people subscribed to the BPs in the sprint.
<nigelb> The big picture is we want all the people subscribed to a blueprint.
<lifeless> so the export should be 3 queries
<lifeless> -> sprint
<lifeless> -> blueprints
<lifeless> -> people
<lifeless> and then some glue
<nigelb> heh
<nigelb> Yeah
<nigelb> My idea is this
<nigelb> We do this currently
<nigelb>         people_by_id = dict((person.id, person) for person in
<nigelb>             getUtility(IPersonSet).getPrecachedPersonsFromIDs(attendee_set))
<nigelb> and then we look into that dict to find a person's ID and whether they are required.
<lifeless> and \o/ instant oopses on staging as well
<nigelb> heh
<nigelb> I'm running it locally :)
<rvba> lifeless: congrats!
<lifeless> rvba: thanks :)
<nigelb> lifeless: \o/ Nice :)
<lifeless> rvba: allenap: btw - you might like the last commit on oops-amqp - more errors amqplib throws up when rabbits are restsarted
 * rvba looks.
<nigelb> lifeless: Ideally, I'd like to look if a person exists in a dict I create as I go, if not lookup each person I find on a BP, and find their ID, name, required status and put into that dict.  I'll be quering more on launchpad, but only once person.  Is that too much of a perf drop?
<lifeless> nigelb: thats about as bad as it gets
<nigelb> I was afraid of that :)
<nigelb> Alternatively, can I look up subscribers to a BP instead?
<lifeless> nigelb: step back
<lifeless> nigelb: you need three queries:
<lifeless> sprint
<lifeless> blueprints
<lifeless> peple
<lifeless> and then glue
<lifeless> the glue is mostly done - jus tneeds shuffling
<nigelb> I get all the people in launchpad?
<lifeless> you can't work with the object model for doing queries though: the object model is intrinsically hostile to performance
<lifeless> nigelb: no, you get all the people relevant to the blueprints
<lifeless> one query.
<nigelb> I can do that?
 * nigelb didn't know.
<lifeless> well, you'll need to put the bits in the right order, but yes.
<lifeless> thats what the current code does (but it selects all attendees, not all subscribers-or-attendees)
<lifeless> [caveat, I haven' tlooked at that code in ~ 9 months, bzr annotate will tell you precisely when)
<nigelb> Is there code to do that or do you suggest I write it? :)
<lifeless> I don't know how much will need assembling and how much is pre canned
<lifeless> this is bread-and-butter to changing something in LP though, its not at all unordinary
<nigelb> Excellent. I'll look at the model and see if I can figure something out :)
<nigelb> I may need lots of help.
<lifeless> is this desirable? Won't the summit UI explode ?
<nigelb> No.
<nigelb> Summit was always doing this.
<nigelb> Showing all the subscribers to a BP.
<nigelb> It helps people hide the ones they aer not subscribed to.
<nigelb> And setup personal icals.
<lifeless> ok
<lifeless> so how is that data not present ?
<nigelb> all those break since some subscriptions don't get sycned thanks to two summits.
<lifeless> I don't understand the failure mode
<nigelb> We're using 2 launchpad sprints.
<nigelb> some Bps proposed to uds-p, some to lds
<nigelb> (Eventually, all blame goes to linaro :P)
<lifeless> but if you're using temp-meeting-export, that means you're not getting all subscriptions
<nigelb> we query both
<nigelb> so we get all subscriptions
<nigelb> But we don't get all the people
<nigelb> some people are registered for uds, some for lds.
<lifeless> I don't understand
<lifeless> neither export includes non-attendees
<lifeless> how are you getting 'all subscriberes to a BP'
<nigelb> Its from the export
<nigelb> I need to find out how james set this up.
<lifeless> so the export includes all subscribers irrespective of attendees ?
<nigelb> (at the summit end)
<nigelb> export does not.
<lifeless> then how is summit showing all subscribers ?
<nigelb> export includes only subscribers who are attendees.
<nigelb> Summit isn't showing all subscribers, which is what I'm digging for.
<lifeless> but you said 'summit was always doing this'
<nigelb> We always had one sprint.
<lifeless> 21:34 < lifeless> is this desirable? Won't the summit UI explode ?
<lifeless> 21:34 < nigelb> No.
<lifeless> 21:34 < nigelb> Summit was always doing this.
<lifeless> 21:34 < nigelb> Showing all the subscribers to a BP.
<nigelb> so, this was never a problem.
<lifeless> nigelb: one sprint doesn't mean showing all subscriberes
<lifeless> nigelb: the statements are not at all equivalent
<nigelb> lifeless: I'll rephrase.
<nigelb> Until now, we never noticed the impact.
<nigelb> Most Ubuntu folks were attendees.
<lifeless> right, and thats why I'm asking.
<lifeless> Won't the summit UI explode ?
<nigelb> explode in what sense?
<lifeless> 600 attendees
<nigelb> we don't show all attendees in one place.
<nigelb> But we show all subscribers to a BP.
<lifeless> bear with me
<nigelb> Yeah, I'm misunderstanding soething :)
<nigelb> Would skyping help?
<lifeless> ok, 597 subscribers across all bp;s
<lifeless> peaks is 46
<lifeless> thats probably tolerable
<lifeless> (on a single spec)
<nigelb> hah, that.
<nigelb> Yeah.
<nigelb> We do occasionally have UI explosion for big and famous sessions
<nigelb> But its only if you mouse over.
<nigelb> lifeless: You just made me realize I need to dig deeper into summit for this.
<nigelb> Because with both the meetings, we should get all the subscriberes, but we're not.
<lifeless> nigelb: consider the more permanent solution :)
<nigelb> lifeless: I like what's currently there for its performance :)
<nigelb> I probably only wold add a warning when subscribing that "You aren't markd as attending for this sprint" or osmething.
<allenap> lifeless: Yes, tip top, thanks.
<lifeless> allenap: also the current catch IOError in LP is too broad
<lifeless> allenap: (again, see oops-amqp for better handling of that)
<allenap> lifeless: rvba has been working on that, but is suffering badly from a Heisenbug around OOPSes. I suspect the big OOPS landing yesterday might help.
<rvba> allenap: I tried to use the new oops thing yesterday and land it again, but I've been suffering from the perf problem lifeless mentioned in his email and I killed it after 6 hours in ec2.
<lifeless> rvba: its fixed now :)
<rvba> lifeless: \o/.  I shall try to land it again then. Thanks for the heads up.
<lifeless> if it takes that long its your change :P
 * rvba tries to land it again.
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugtasks: 266
<bigjools> gmb: can I bother thee for a review please
<gmb> bigjools: Sure. I'll take a look as soon as I've done with rvba's.
<bigjools> thanks
<bigjools> gmb: heh, good point with the comments
<gmb> :)
<bigjools> I only even added those ones as an afterthought - never a good idea
<deryck> Morning, all.
<danhg> Hi Everyone, if you're a Launchpad user, please get in touch if you'd like to do some usability testing today.
<lifeless> danhg: I suggest trying in #ubuntu-uds
<danhg> thanks lifeless
<lifeless> danhg: folk here probably don't qualify for such tests :P
<danhg> I've posted there too
<lifeless> danhg: I don't see you in that channel ?
<danhg> I'm in it now
<danhg> I'm on Colloquy. It likes to mess around.
<lifeless> indeed, you are too
<rvba> lifeless: Your new oopses code fixed my Heisenbug.  Thank you for that!
<lifeless> \o/
<lifeless> how?
<rvba> I don't know (hence the Heisenbug), but I was able to land my code without a problem once I've refactored it to use the new oops code.
<rvba> I had this weird 'test hung' failure.
<rvba> That I could not reproduce locally.
<rvba> lifeless: If you're really interested, you have all the details about the Heisenbug in my email "Help with a hung test" to the mailing list.
<Beret> lifeless, thought you were going to bed? :)
<lifeless> I was
<lifeless> I am too :P
<lifeless> rvba: were you using getLastOops ?
<rvba> lifeless: yes.
<lifeless> then you were seeing an oops from the test before, in all probability
<lifeless> because it was a terrrrrible API (though probably not obviously so at the time it was written)
<lifeless> rvba: I'm glad its working for you
<lifeless> and with that, I really am halt()ing state
<rvba> Good night lifeless!
<rvba> gmb: I've got a tiny review for you if you're up for it ;). https://code.launchpad.net/~rvb/launchpad/combinator-bug-881144/+merge/80911
<gmb> rvba: I'm OTP and then am going to be grabbing a late lunch, but I'll come to it soon thereafter.
<rvba> gmb: sure. Thank you!
<deryck> abentley, should we do a late standup now, while I wait on staging update?
<abentley> deryck: sure.
<deryck> abentley, http://pastebin.ubuntu.com/725330/
<deryck> abentley, we will need to get a loading spinner of some sort working when changing the ordering.
<deryck> abentley, but I think we'll need to rethink our current spinner approach since a whole section of the page changes.
<abentley> deryck: We haven't implemented precaching yet :-)  Well, I guess we should add a spinner anyhow.
<deryck> abentley, yeah, just to be safe.
<deryck> mrevell, staging is working now.  https://bugs.staging.launchpad.net/launchpad/+bugs
<mrevell> Woooo! Thanks deryck
<deryck> mrevell, np.
<deryck> mrevell, some minor issues still, like needing a loading symbol to indicate we're getting results, but overall, the resorting is pretty darn cool.
<deryck> but after the initial load, it's super fast.
<mrevell> deryck, Cool :) What team do I need to be in the thing?
<mrevell> Sorry, that made little sense.
<deryck> mrevell, I think matsubara added the product team to the feature flag rule.
<mrevell> I'm not seeing the new listings so I'll hunt him down and ask him to pop me in the relevant team. Cheers.
<deryck> mrevell, cool.  cheers.
<abentley> deryck: very nice.
<deryck> thanks, abentley!
<jml> mrevell: in this ask ubuntu session, you'd be amazed at how much people are talking about 'reputation' (their karma equiv.) as a desirable commodity and a motivator for actions.
<gmb> rvba: I haven't had chance to review your branch yet, but I will do before I EoD.
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<rvba> gmb: Great, it's not urgent so if you don't have the time today that's fine, I'll get it reviewed tomorrow.
<gmb> rvba: Okay, thanks. I'll do my best though. Interviews are sucking up a lot of time for team Yellow this week.
<mrevell> hey huwshimi
<thumper> hi mrevell
<huwshimi> mrevell: Morning!
<thumper> mrevell: you in orlando?
<thumper> mrevell: is sinzui there with you?
<thumper> I'd like to talk privacy with him
<mrevell> thumper, sinzui is here, he arrived today
<mrevell> thumper, and "hello" :)
<mrevell> huwshimi, Hey, do you want to have a catch-up call?
<mrevell> thumper, I'm not sure where he is right now but I can let him know you want to talk.
<thumper> mrevell: that'd be cool
<huwshimi> mrevell: Sure
<thumper> mrevell: I'm back home now though, so it would have to be afternoon for him
<huwshimi> mrevell: 5 mins?
<mrevell> thumper, Okeydoke. I'll let him know to contact you when I see him.
<mrevell> huwshimi, Sure.
<lifeless> mrevell: allo
<mrevell> hello lifeless
<jelmer> hi launchpadders
<lifeless> mrevell: have you seen the new LessJunk LEP? I haven't read it yet but given that the implementation is pretty shallow it would be worth us working through the questions I expect it raises.
<huwshimi> mrevell: Ready when you are
<mrevell> lifeless, I've seen that it exists but not yet read it; it's been pretty full on today and yesterday. I'll take a look tonight or tomorrow morning and perhaps we can have a call tomorrow.
<mrevell> huwshimi, Cool. Is Skype okay?
<huwshimi> mrevell: yup
<lifeless> mrevell: sweet
<huwshimi> mrevell: I'm getting nothing
<mrevell> huwshimi, Oh, weird. It was ringing for me. I'll try again
<huwshimi> mrevell: It rang and I answered, but no sound
<huwshimi> mrevell: Oh, looks like skype crashed
<huwshimi> mrevell: Try again?
<lifeless> flacoste: dunno if you saw, but wgrant shaved a second off of ubuntu bug searches
<lifeless> flacoste: maybe more in fact - poor behaviour on the TeamParticipation table
<lifeless> flacoste: I going to ask stub to look into some pg innards relating to this
<flacoste> lifeless: i saw mention of a performance related feature flag earlier
<flacoste> didn't know the details
<flacoste> but that's excellent
<lifeless> its a bit worrying, but good we have a workaround - I suspect table and index bloat on the table
<lifeless> flacoste: also \o/ \o/ \o/ I'm so happy the major part of the oops arc is complete.
<flacoste> yes, this is also awesome news!
<lifeless> I know I mentioned it yesterday, but I'm still bouncy from it ;)
<poolie> flacoste, all
<flacoste> hi poolie
<poolie> hi huwshimi?
<poolie> lifeless, huw, i wonder if there should be some kind of systematic answer to whether to provide counts of objects from the thing that links to them
<poolie> like rvb's recent bug 827935
<_mup_> Bug #827935: Person:+branches timeouts <regression> <timeout> <Launchpad itself:In Progress by rvb> < https://launchpad.net/bugs/827935 >
<poolie> perhaps it's just simply: do it if it's useful but not if it's too expensive
<lifeless> I think it depends a lot on context
<lifeless> for instance, a dashboard telling you what things need doing it much more useful if you don't need to click through
<lifeless> the portlet showing high/critical/untriaged/my bugs for instance - the first three tell you things you need to action, the last one is more 'canned search' - if you see what I mean
<wgrant> lifeless: Is lp-oops missing its 500.html again?
<lifeless> wgrant: it hasn't been added ever
<lifeless> wgrant: its barfing on the index page?
<wgrant> Yes.
<wgrant> And all other pages.
<wgrant> Ah, no, some OOPSes work.
<lifeless> should wire it up with oops reporting
<wgrant> Heh
<lifeless> ./src/oopstools/settings.py:INDEX_TEMPLATE = "index-launchpad.html"
<poolie> lifeless: yeah i like the new/high/critical things
<StevenK> And then if the OOPS reporting causes an OOPS while reporting an OOPS, we reboot Launchpad.
<lifeless> wgrant: should be fixed
#launchpad-dev 2011-11-02
<poolie> hm
<poolie> if i change buildrecipe into an importable module so that i can test it...
<poolie> that seems to have a fair risk of breaking something during deployment
<lifeless> yes
<lifeless> its also an entirely separate project
<poolie> well
<poolie> i would like to add automatic tests for my changes
<lifeless> just something to be aware of
<poolie> i guess i could add a separate landing to just make it testable
<poolie> it seems like a high risk change so perhaps i'll just do without
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugtasks: 266
<wgrant> lifeless: Thanks.
<wgrant> Still extremely slow, but it works.
<wgrant> lifeless: What's the production rabbit status?
<wgrant> lifeless: And are we going to persist with rsync-based fallback, or have a disk2amqp daemon?
<wgrant> s/daemon/cronjob/, if you like.
<lifeless> wgrant: cron job
<lifeless> wgrant: plumbing written, just need a glue project to pull it all together (datedir repo on one hand, amqp on the other)
<lifeless> wgrant: see oops-datedir-repo 0.0.10
<wgrant> We should also ensure that the config name and tree base and script name are in the OOPSes, and then nuke all our OOPS configs :)
<wgrant> Ah, excellent!
<lifeless> prod rabbit is pending firewalls and tuolumne and nagios etc
<lifeless> I've an rt up for migration to this including moving to a shared local-oops dir for all servers
<wgrant> Great.
<lifeless> beuno is champing at the bit to get this for u1 ;)
<spm> i think the firewall's been done. was some chatter around that recently.
<beuno> YES
<lifeless> looks like it
<wgrant> We also need a better way to handle rabbit passwords.
<wgrant> Currently they're in the LP configs, which won't really do.
<lifeless> so yes, we're ready to receive prod oopses over rabbit when the rest of the bits come together
<wgrant> We probably need to read them from some other file.
<lifeless> whats the issue ?
<wgrant> Configs are widely accessible, and the rabbitmq instance is accessible from more than just production LP servers.
<lifeless> you mean 'someone could connect and either consume or inject inappropriate messages' ?
<wgrant> yes.
<lifeless> so, moving oops-tools to its own box will help there
<wgrant> Indeed.
<lifeless> also we need to make sure the lp rabbit user has narrow perms
<wgrant> But we should be treating our message queue like we treat our DB, I think.
<lifeless> I haven't checked on that
<wgrant> Good luck with that...
<wgrant> How do you propose to do that?
<wgrant> It basically needs to be able to write to everything.
<lifeless> not at all
<wgrant> And eventually read lots of things too, probably.
<lifeless> write to oopses, create and write to longpoll.*, read from nothing, admin nothing.
<lifeless> (today)
<lifeless> thats pretty narrow
<wgrant> Today.
<lifeless> and as it changes, we can change it.
<lifeless> bbs
<poolie> lifeless, i commented on https://code.launchpad.net/~mbp/launchpad/bug-width/+merge/80161
<poolie> wgrant: do you have any idea how buildrecipe gets installed, or where it gets renamed to buildrecipe.py?
<StevenK> It's probably contained in lp-buildd
<StevenK> So the build machinery of lp-buildd likely renames it
<poolie> and that's not in the lp tree?
<poolie> apparently not public ?
<StevenK> It is, lib/canonical/buildd
<wgrant> poolie: I don't think it actually does get renamed.
<wgrant> [sourcepackagerecipemanager]
<wgrant> buildrecipepath = /usr/share/launchpad-buildd/slavebin/buildrecipe
<poolie> StevenK: i don't see anything in there that renames it
<wgrant> argv[0] might be buildrecipe.py, but I think it's a lie.
<poolie> yes!
<poolie> just realized the same thing
<poolie> :/
<poolie> StevenK: hi
<poolie> you don't need to write a lep for my sake
<poolie> just a 'i'm planning to do X' on the bug would be ince in general though
<poolie> short review on https://code.launchpad.net/~mbp/launchpad/884997-rusage/+merge/80971please anyone?
<wgrant> lifeless: Did you change the uppercasing rule part-way through yesterday?
<wgrant> Some of the hashes are allcaps, some are not.
<lifeless> in the db ?
<lifeless> or as reported to users? the latter is hashlib variations
<wgrant> DB, I assume.
<wgrant> It's in the reports.
<lifeless> hmm, odd. Possibly dboopsloader needs a fix too.
<lifeless> I shall check.
<wgrant> Ah
<wgrant> Could be.
<lifeless> yeah, from_pathname needs a fix
<lifeless> cowboyed
<lifeless> === modified file 'src/oopstools/oops/models.py'
<lifeless> --- src/oopstools/oops/models.py        2011-10-31 02:07:24 +0000
<lifeless> +++ src/oopstools/oops/models.py        2011-11-02 02:02:03 +0000
<lifeless> @@ -476,6 +476,7 @@
<lifeless>          oopsid = parsed_oops.get('id')
<lifeless>          if oopsid is None:
<lifeless>              raise OopsReadError('Not a valid OOPS report')
<lifeless> +        parsed_oops['id'] = parsed_oops['id'].upper()
<wgrant> Thanks.
<wgrant> lifeless:        1 https%3A//launchpad.net/ubuntu/%2Bsource/oops/1.5.23.cvs-5.1/%2Bbuild/390896/%2Bindex (BinaryPackageBuild:+index)
<wgrant>         OOPS-48d0191b7ac2d3136e4bd47d385fceb7
<wgrant>        1 https://launchpad.net/ubuntu/+source/oops/1.5.23.cvs-5.1/+build/390896/+index (BinaryPackageBuild:+index)
<wgrant>         OOPS-2131AU23
<wgrant> New URLs are excessively URL-encoded.
<lifeless> I was afrait of that
<lifeless> I think I mentioned it
<lifeless> probably need an LP change - I suspect its emitting them as unicode
<poolie> is this pqm rejection lp's way of telling me you're only taking rc fixes?
<poolie> no
<wgrant> poolie: It's because we're in testfix.
<poolie> and is that advertised anywhere?
<StevenK> No, because PQM sucks.
<poolie> i thought people used to put it in the topic here or something
<poolie> anyhow, np, just glad it's not my branch
<wgrant> lifeless: Around?
<lifeless> yas
<wgrant> lifeless:   File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/src/oopstools/oops/models.py", line 309, in _get_oops_tuple
<wgrant>     for key, value in req_vars:
<wgrant> ValueError: need more than 1 value to unpack
<wgrant> Broken by a ['field.blob']
<wgrant> Without a value.
<wgrant> Assume ''?
<lifeless> ugh
<lifeless> also win
<lifeless> so that needs two bugs
<lifeless> a) yes we should handle that
<StevenK> I thought that was already done
<lifeless> and b) we shouldn't be generating that
<lifeless> also c) req_vars should be a dict
<StevenK> I remember reviewing something that did that
<lifeless> but there is a bug on that lready
<lifeless> StevenK: urls
<lifeless> StevenK: / encodings
<StevenK> Not everything has a URL
<StevenK> What do you do for script OOPS
<wgrant> lifeless: Are you fixing the immediate issue, or should I?
<lifeless> StevenK: I know
<lifeless> wgrant: please do
<wgrant> It even works.
<StevenK> OOPS summaries twice in one day?
<wgrant> Yes, the old ones were wrong.
<wgrant> And today's are of particular interest.
* jtv changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: StevenK, jtv | Critical bugtasks: 266
<wgrant> lifeless: Is there a branch for your cowboy?
<lifeless> not yet
<lifeless> if you're doing one, roll this in
<wgrant> Was about to ask you to do the same :)
<lifeless> I can later
<wgrant> Also tempting to upper() the oopsids that were imported badly.
<lifeless> it would make them accessible
<lifeless> the oops table is a bit big to do it without date constraints
<wgrant> I planned to restrict to the last few days.
<wgrant> All seems happy now.
<poolie> is lp out of testfix?
 * StevenK becks
 * StevenK grumbles at his keyboard
<wgrant> spm: Can you check if pigeonpea and pilinut need cleaning?
<spm> yup
<wgrant> lifeless: https://lpbuildbot.canonical.com/builders/lucid_lp/builds/1517/steps/shell_6/logs/summary looks possibly relevant to you
<spm> there's a bunch of oldish rabbits still running
<spm> only on pilinut tho
<spm> pigeonpea looks clean
<wgrant> Does pigeonpea have any old pid files in /var/tmp?
<lifeless> no old appserver processes ?
<spm> buildbot@pigeonpea:~$ ls /var/tmp/*.pid | wc -l
<spm> 111
<spm> maybe....
<spm> nothing running as BB, just the slave itself. no old processes running.
<spm> ^^ pigeonpea; all that applies to.
<spm> should I assume those pids should be rm'd?
<lifeless> pleas
<lifeless> e
<wgrant> Yup.
<spm> ugh. pilinut is just as dirty
<spm> any objections if I shutdown BB on pilinut and clear out the running rabbits
<StevenK> Do eet
<wgrant> No need to shut it down, but sure.
<wgrant> No build, so shutting it down is fine.
<spm> just to make it easy to clear out everything
<wgrant> What, kill all processes owned by the user?
<spm> yup
<spm> well. more making sure I don't clobber the BB process itself hard.
<StevenK> It deserves it, kill it anyway
<spm> huh. there's 3 pgbouncers in there as well
<spm> was
<spm> pilinut cleaned; pigeonpea cleaned
 * StevenK forces a devel builkd
<StevenK> s/k//
<wgrant> Thanks spm.
<spm> np
<StevenK> poolie: Your branch should be fine to land now
<poolie> thanks
 * StevenK sets lp-oops on fire.
<StevenK> Perhaps SSO is to blame.
<wgrant> StevenK: What's up?
<StevenK> Forgetting the query string when logging in.
<wgrant> That's apache-openid
<wgrant> We should work around it by not using the query string.
<wgrant> This isn't 1990 :)
<jtv> wgrant: kindly refrain from ridiculing as ancient history a year that I used to look forward to as the wondrous future.
<wgrant> Bah
<wgrant> It predates me by a year, therefore it is ancient history :)
<StevenK> Bwaha
<jtv> Insolent whippersnapper.
<jtv> This does illustrate why my quiz team lost that crucial point a few weeks back.
<jtv> The question: name the four horsemen of the Apocalypse.
<jtv> I shouted âRonnie, Paulâ etc. but whispered the real names to the teammate who was doing the writing.
<jtv> He dutifully, but not altogether correctly, noted down:
<jtv> Hunger
<jtv> War
<jtv> Dath
<jtv> *Death
<jtv> Insolence
<jtv> I took him to task about it afterwards, after we'd lost by 1 point.  His defense: âI'm British.  To us, Insolence *is* a horseman of the apocalypse!â
 * StevenK is trying to remember the four horsemen as according to Good Omens
<jtv> Hunger, War, Pestilence, Death, and Ronnie who left before they became famous.
<jtv> (Let's not give away the rest; there may be people here who haven't read it yet :)
<StevenK> No, it's Pollution, since Pestilence retired in 1963 following the discovery of pencillin.
<jtv> Damn you, Sir Alexander Fleming!
<jtv> (Why do I remember the name?  Because the anniversary of his death was a few weeks ago, on quiz night.  It pays to look these things up.)
<jtv> This just in: IPCC announces the floods in Thailand were caused by Global Warming, _not_ as a naÃ¯ve reader would deduct from looking at the actual numbers, from the reservoirs failing to run off excess water throughout the rain season.
<jtv> To understand these announcements better, replace âScienceâ with âGodâ and variants of âClimate Changeâ with âSatan.â
<nigelb> I'm apparently expiring from 3 teams almost at the same time :/
<nigelb> Yay launchpad spam
<jtv> Shouldn't have joined them all in the same week.  :)
<nigelb> Yeah, regrets.
<jtv> Funny really: apparently team memberships expire, but when we wonder if untended membership _requests_ should expire, everyone's up in arms!
<nigelb> heh.
<nigelb> I want to change the lies in the expiry warning email.
<nigelb> Or a way to say "Hey, I dont care, let me expire
<nigelb> "
<nigelb> If anyone else does it, I'll buy them a drink :)
<jtv> nigelb: I must warn you about one subtlety in Launchpad, and one in the real world.
<wgrant> jtv: Well, team memberships expire when the team admins want them too.
<wgrant> s/too/to/
<jtv> In Launchpad, where we say âperson,â that person can also be a team.
<jtv> And in the real world, champagne qualifies as a drink.
<jtv> wgrant: that explains, thanks.
<wgrant> We don't just expire them by default, so you can't use that as an argument for expiring requests :)
<wgrant> by default
<poolie> jtv, you're feisty today
<jtv> hi poolie!
* StevenK changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: jtv | Critical bugtasks: 266
<jtv> night StevenK
<nigelb> jtv: hah.
<jtv> nigelb: sooooâ¦ you just committed yourself to something that can be stretched to buying the entire team champagne.  On the bright side, that might make the search for a volunteer a little easier.  :)
 * nigelb whistles
<wgrant> Our tests are crap :(
<spm> wgrant: so. write a test to test the crapness of our tests. problem solved. /me cross arms. nods.
<wgrant> Also, bugtasks are the bane of my existence.
<nigelb> wgrant: I'd argue its not bugtasks, but launchpad itself :P
<wgrant> Nah, most Launchpad stuff is fixable without incredible migration costs.
<wgrant> The existence of bugtasks is not.
<poolie> ok, good night all
<wgrant> Night poolie.
<jtv> Reviewer needed for small & simple change!  https://code.launchpad.net/~jtv/launchpad/orderingcheck-unicode/+merge/80987
<rvba> jtv: I can do it ;)
<jtv> So you can!  Thanks.  :)
 * rvba likes Unicode stuff; yummy.
<stub> Spoken like a man with an umlaut
<wgrant> Evening stub.
<stub> Yo
<jtv> Hi there stub â did you go back home?
<jtv> Also, AFAIK RaphaÃ«l does not have an Umlaut; it's a different diacritical mark that happens to look the same (unless you're Hungarian I believe, in which case you use both for their respective purposes).
<jtv> Ah: diaeresis, that's the word.
<jtv> I think.
<rvba> True, Umlaut != Diaeresis.
<nigelb> All you pedantics :P
<jtv> An Umlaut transforms one vowel into another (a bit like the vowel changes in English, actually); this one separates two consecutive vowels, to show that they are in separate syllables.
<jtv> Nobody bothers with them now, but a few decades ago you'd still find the same mark in English: âcoÃ¶perateâ
<jtv> (To show that it's co-op-er-ate, not coop-er-ate)
<wgrant> stub: We ran into some slightly disturbing query speed issues yesterday. On qastaging, https://pastebin.canonical.com/55125/ takes <10ms to EXPLAIN ANALYZE, but 300-700ms to actually execute. On DF, it executes in <10ms too. It evaluates to [(1,)], so it's presumably not just terrible output formatting speed...
<jtv> wgrant: too late.  I chased him off.
<wgrant> Curses.
<jtv> (This suggests he probably didn't go back home.  Smart choice.)
<wgrant> Are you home?
<jtv> No.
<jtv> I'm making a trip there in a few hours, but don't want to sit around consuming scarce resources I'm not helping deliver.
<jtv> Oh, done already eh?  Thanks rvba
<rvba> np
<nigelb> jtv: scarce resources?
<jtv> Clean water, food, etc.
<jtv> Even in areas that aren't affected, shops have found it difficult to keep stocks up.
<nigelb> oh.
<bigjools> morning
<nigelb> Morning bigjools!
<adeuring> good morning
<jelmer> g'morning
<lifeless> 'lo
<lifeless>  4 / 1081  RootObject:+login
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<jelmer> hmm, this is odd. one of my pending builds has an estimated start and finish time in the past. is that possible?
<jtv> Any reviewers out there?  I need to leave soon but I have a largish branch up for review: https://code.launchpad.net/~jtv/launchpad/bug-884649-branch-1/+merge/80988
<bigjools> jtv: I'll take it
<jtv> Thanks!
<bigjools> jtv: why all the new tests?
<jtv> bigjools: so we won't need to iterate through all the possibilities in the integration tests.
<jtv> This is stuff that couldn't be tested in isolation before.
<bigjools> ok
<bigjools> jtv: approved
<jtv> Thanks.  That was fast!
<bigjools> can I bug someone for a review please? https://code.launchpad.net/~julian-edwards/launchpad/cancel-build-bug-173018-ui-part4/+merge/80997
<jtv> bigjools: least I can do.
<jtv> bigjools: I guess the âCancel buildâ link takes you to a confirmation form (at +cancel)?
<bigjools> jtv: correct
<jtv> I swear I don't remember a thing about menu links.  I thought a link wouldn't show up in menu.links if its "enabled" condition said it was disabled.
<bigjools> jtv: menu.links is just the list defined at the top of the class
<jtv> Ah
<jtv> bigjools: done
<bigjools> thanks
<jtv> I don't suppose python has a built-in equivalent to âlambda x: not xâ?
<jtv> Actually, no, that's not what I need.  What I need is partial application of operator.not_
<jtv> Or something.
<stub> __builtins__.not_ = lambda x: not x
<jtv> I was hoping to combine it with an attrgetter: filter(negate(attrgetter('my_bool_attr')), my_list)
<jtv> For some value of negater.
<benji> jtv: operator.not_
<jtv> The problem is combining it with the attrgetter.  :)
<benji> ah; lambda is your friend
<jtv> Or reduce maybe
<jtv> No
<jtv> So back to partial
<gary_poster> hey stub.  We are getting a lot of activity on pgkillactive.py.  This may be old news; I was out last week and am consumed by interviewing.  But the logs are telling us stuff like this:
<gary_poster> Killing lpnet127 (15802), 2011-11-01 11:20:17.341629+00:00, 2011-11-02 06:57:42.661591+00:00
<gary_poster> Locks found:
<gary_poster> relname, transactionid, mode, granted, current_query, query_start, age, procpid
<gary_poster> It's on lpnet 127, 129 and 130.
<gary_poster> Does that suggest an action item?  If so, has it already been taken?
<stub> gary_poster: I spent last week in a small village 100km from the Laos border and am now encircled by flood waters. I know less than you :-)
<gary_poster> stub, heh, ok
<gary_poster> are things getting better or worse, stub?
<gary_poster> for your physical environment I mean :-P
<stub> gary_poster: I'm still dry, and if it floods in my area I'll be fine upstairs. Official news is crap though so hard to follow.
<gary_poster> gotcha stub.  good luck.
<wgrant> gary_poster: That was several hours ago?
<wgrant> Yes.
<gary_poster> wgrant, yes
<wgrant> That was soybean wandering off into lala land.
<wgrant> It was powerstabbed and recovered.
<wgrant> We still don't really know what happened, but it wasn't an LP thing.
<gary_poster> ok thanks wgrant
<mrevell> Guten morgen alles.
<nigelb> Bonjour mrevell
<timrc> is the Launchpad web service going to get a 2.0 release for Precise?
<deryck> Morning, all.
<abentley> deryck: morning.
<abentley> timrc: I'm not aware of any plans to do another stable API.  However, the devel API is improving over time.
<timrc> abentley, okay... I wasn't sure what LTS would mean (if anything) to the web service
<abentley> timrc: you might want to ask lifeless for his thoughts.  He's usually around in 6-7 hours.  I have the impression that the devel service is much more popular than the 1.0 service, so stable versions may not be worth the extra effort they require.
<Ursinha> bug 879589
<Ursinha> where's the bot
 * nigelb pokes _mup_ 
<cjohnston> One of the blueprints for a session during UDS was deleted.. do any of the devs have the ability to figure out when it was deleted
<jtv> cjohnston: I'm not too familiar with blueprints (that'd be sinzui) but I'm not sure we normally delete blueprints at all.
<jtv> We have various statuses along the line of "obsolete" and "declined," and if that's what happened we may be able to learn more.
<cjohnston> its 404'ed
<cjohnston> we figured out the issue.. so its all good
<mwhudson> cjohnston: it's probably been renamed
<jtv> Ah.
<cjohnston> that could be too
<mwhudson> or retargeted
<cjohnston> would it be possible to create some sort of forward for a while when they are retargeted?
<mwhudson> if you don't mind changing launchpad, sure!
<cjohnston> lol
<nigelb> He already has patches
<nigelb> ;)
<cjohnston> ive already made changes
<nigelb> 6 landings I think?
<cjohnston> something like that
<jtv> And yet none of them fixed your problem?  Patch harder!
<jtv> Meanwhile, any reviewers for tiny tiny leetle branch?  https://code.launchpad.net/~jtv/launchpad/getBinariesForDomination-bulk/+merge/81020
<cjohnston> yup.. its a branch
<cjohnston> jtv / mwhudson.. for sprints (ie the current UDS) when a track lead is set as an approver.. do they get a notification and or a reminder about a  blueprint that needs to be approved
<mwhudson> cjohnston: don't know, sorry
<jtv> Same here.
<cjohnston> anyone else know by chance?
<nigelb> cjohnston: *cough* "Use the source, Like" *cough*
<cjohnston> too much work
<nigelb> Blueprints is probably the easiest to understand.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugtasks: 266
<allenap> Does anyone know why I'd get a NotABranch error from merge-proposal-jobs.py in qastaging? https://lp-oops.canonical.com/?oopsid=OOPS-d39c8d786a4c80ed1f17e9943d2f51ae
<allenap> s/NotABranch/NotBranchError/
<rvba> jtv: I've reviewed your tiny tiny branch ;)
<jtv> Thanks!
<jtv> allenap: not scanned yet?
<allenap> jtv: I pushed it yesterday, and revisions are appearing in the UI <https://code.qastaging.launchpad.net/~allenap/scriptify/test1>
<jtv> Oh, wait.  This is qastaging.
<jtv> Does it have its own codehosting server?
<allenap> jtv: It's running on gandwana as bzrsyncd.
<allenap> Which is the same host as used for staging.
<jtv> So much for that then.
<allenap> jtv: The error mentioned lp-internal:///~allenap/scriptify/test1... there's no mention of qastaging there, or is it, as it implied, a purely internal identifier.
<nigelb> danhg: Do you know if the launchpad session will have audio?
<allenap> $ bzr push lp://qastaging/~allenap/scriptify/test1 --use-existing-dir
<allenap> bzr: ERROR: Server sent an unexpected error: ('error', 'NotBranchError', 'Not a branch: "lp-83227408:///+branch-id/360039".')
<allenap> $ bzr push lp://qastaging/~allenap/scriptify/test1
<allenap> This transport does not update the working tree of: bzr+ssh://bazaar.qastaging.launchpad.net/~allenap/scriptify/test1/. See 'bzr help working-trees' for more information.
<allenap> Created new branch.
<allenap> $ bzr push lp://qastaging/~allenap/scriptify/test1
<allenap> No new revisions to push.
<allenap> Now I can see a .bzr directory in the right place (with hitchhiker).
<deryck> abentley, hey, shall we mumble a bit now?
<abentley> deryck: sure
<deryck> hi coffeedude
<rvba> jtv: I was thinking about BPPH.binarypackagename but I'm not sure that's helpful for you.
<jtv> rvba: not really â it's still one-by-one.
<jtv> But I'm not too worried about it; I think it'll be a secondary issue at worst.
<rvba> Okay.
<jtv> Oh wait, _now_ I get what you mean!
<coffeedude> hey deryck
<jtv> The column that's been recently added to the table.  Has that been fully initialized yet?
<rvba> jtv: exactly
<jtv> Has initialization completed yet?
<rvba> jtv: I think it has.
<jtv> If so then yes, it would be useful!
<jtv> Thanks for pointing that out.
<rvba> Glad to help ;)
<jtv> It means we can avoid loading the BPN anywhere.
<jtv> Maybe even the BPRâ¦ worth thinking about.
<cjohnston> jtv: fwiw, the answer is yes, they do get emails, but sometimes get lost in rules and what not
 * jtv wishes software could go back to being stupid and predictable, and leave the interesting stuff to us meatbags
<cjohnston> heh
<mrevell> nigelb, What's your Skype id?
<nigelb> mrevell: heh, I was kidding originally :)
<nigelb> mrevell: nigelbabu
<abentley> deryck: you changed the listing to be a wide listing, but to me it looks too wide now.  Too much space between bug title and bug heat, especially if maximized.  Can we revisit that?
<deryck> abentley, I'm just following the mockups.  I think we need to raise it with huw and mrevell if we want to change it.  FWIW, I like it wide. :)
<deryck> but then I don't have a huge monitor.
<deryck> also, it seems quite bare now, but when there are a lot of fields, it will fill up.  So there will always be some tension here.
<abentley> deryck: On my monitor, the space between the title and the heat is often as wide as the title itself.  That makes the listings hard to read, because there's not a strong visual connection.
<deryck> abentley, yeah, I can understand that.
<deryck> abentley, I wonder if heat should drop down to the second row and fall directly inline with everything else.
<deryck> or else directly to the right of the title.
<abentley> deryck: the mockups actually show the title's column being ~ the same width as the longest title, which was how I had it.
<deryck> abentley, all the mockups I see have it flush with the sidebar too.  so it's not clear which was meant -- i.e. follow with the title, or follow to the sidebar.
<deryck> mrevell, care to weigh in on the intent here ^^ ?
<deryck> abentley, what does the side bar do on a large display if the listings are narrow?
<deryck> if you recall.
<abentley> deryck: I don't recall.
<deryck> ok, np.
<deryck> abentley, are you wanting to fix it yourself in a branch your working on, or just raising the issue?
<abentley> deryck: just raising the issue.
<abentley> deryck: while it's still fresh.
<deryck> abentley, ok.  let's chat with mrevell and huw when they have more availability, and I'll play with some options in css to see what possibilities we have.  fair enough?
<abentley> deryck: sounds good.
<deryck> abentley, cool.  I added a Kanban misc card to remind me to follow up on it. :)
<allenap> jtv, rvba: Fwiw, the problem was that the *trunk* branch of the scriptify project was not pushed. The NotBranchError I got when pushing the test1 branch referred to an attempt to find the default stacking branch.
<jtv> ouch
<jtv> Bug.
<allenap> jtv: Indeed, but the good news is that long-polling for merge proposals works in staging and qastaging (restricted to ~launchpad for now).
<allenap> rvba: ^
<jtv> Nice!
<nigelb> allenap: <3
<nigelb> Will it be rolled to production for launchpad only at least? :)
<nigelb> Oh wait. ~launchpad. Not ~launchpad-dev :(
<allenap> nigelb: The LOSAs are going to figure out how to deploy the new service effectively (it's a bit of a cowboy right now) then, yes, it'll get rolled out :)
<nigelb> \o/
<allenap> nigelb: I think it'll be safe to make it available to ~launchpad-dev.
<nigelb> Yay!
<nigelb> mrevell: Thanks for skyping me in!
<mrevell> nigelb, Pleasure!
<abentley> deryck: css class "bugtitile" appears mispeled.
<deryck> heh
<deryck> I had to look a couple times to spot the typo.
<deryck> that's how I pronounce it in my accent. ;)
<deryck> tie-tile :)
<deryck> abentley, I'll fix in my branch.  Thanks!
<abentley> deryck: lol.  Cool.
<allenap> nigelb: Merge proposal long-poll stuff is enabled for ~launchpad-dev in staging and qastaging.
<nigelb> allenap: Thanks!
<allenap> nigelb: No worries. Please break it :)
<allenap> abentley: Up for a review? https://code.launchpad.net/~allenap/launchpad/participation-keyerror-bug-810114/+merge/81012
<abentley> allenap: sure.
<bigjools> allenap: It still says "reload to see changes", right?
<allenap> bigjools: Yes, it does.
<allenap> abentley: Thanks.
<bigjools> mrevell: is there an LP session on?
<abentley> allenap: module docstring is a placeholder.
<mrevell> bigjools, Not right now.
<mrevell> bigjools, We're doing a check-point, though.
<bigjools> cool
<allenap> abentley: Oops. I'll fix that, and all the functions too.
<abentley> allenap: yeah, the functions have comments rather than docstrings.
<deryck> abentley, so you think bugnumber CSS class should become bugid (or some such)?  Or do you think it doesn't matter for class names so much?
<abentley> deryck: I think we should standardize everything on bugnumber.
<deryck> abentley, ah, ok.  The class I use for sorting is sort-id because each of them get a "sort-X" class where X is the actual orderby phrase.
<deryck> abentley, so we can't actually change that to bugnumber.  but otherwise we can standardize.
<abentley> deryck: makes sense.
<deryck> ok, cool.
<bigjools> allenap: it isn't working for me on staging
<allenap> PEBKAC probably ;)
<allenap> bigjools: Whassup?
<bigjools> allenap: hoho!  yellow box still there, if I reload in a new tab, it's not there, but neither is the diff
<allenap> bigjools: Can you point me to the branch?
<bigjools> allenap: https://code.staging.launchpad.net/~julian-edwards/txlongpollfixture/FIRST/+merge/76676
<bigjools> I merged one that was already on staging
<abentley> allenap: transaction.abort in check_teamparticipation_consistency worries me a bit, because it doesn't seem to guarantee that any pending DB writes got written.
<allenap> abentley: There are no pending db writes, and there shouldn't be.
<gnuoy> allenap, Not a branch
<gnuoy> OOPS-21e6aba1bf48244072ebdc537396b722
<abentley> allenap: A function doesn't know who its caller is.
<abentley> allenap: I guess you can keep it if you document it.  Otherwise, I'd prefer a commit.
<allenap> abentley: Fair. Okay, I can do a store.rollback(), or even a store.close().
<allenap> abentley: Okay, commit is fine actually.
<abentley> allenap: cool.
<gnuoy> allenap, I think bigjools is hitting the same problem you were
<allenap> bigjools, gnuoy: It's because lp:txlongpollfixture is not a branch; branch content is not synced during a refresh. I've pushed it now, so try again.
<bigjools> ahhh
<allenap> bigjools: gnuoy and I spent a long time figuring out that little gem earlier.
<abentley> allenap: check_teamparticipation_consistency has a lot of inner functions.  Have you considered structuring it as a class?
<abentley> allenap: well, maybe "a lot" is a bit strong.  Some.
<allenap> abentley: Yes, but I think it would get longer and less readable, imo. I have considered splitting it into two - a slurping part and a checking part - but haven't done it yet. I am working on a follow-up so I might do it there.
<allenap> abentley: Also, the slurped data will use about ~200MB of memory so it's quite useful that this will be reclaimed as soon as the function returns.
<abentley> allenap: "len(out) > 0" might be clearer and faster as out != ""
<allenap> abentley: Okay.
<abentley> allenap: r=me with the docstring updates.
<allenap> abentley: Thanks!
<rvba> abentley: could you please have a look at this (tiny) MP? https://code.launchpad.net/~rvb/launchpad/branches-timeout-bug-827935-always-display/+merge/81022
<gnuoy> nigelb, bigjools, I need to quickly restart txlongpoll in staging and qastaging
<abentley> rvba: sure.
<rvba> ta
<bigjools> gnuoy: fine
<nigelb> I'm not testing yet.
<nigelb> UDS :)
<abentley> rvba: r=me
<rvba> abentley: Thank you!
<abentley> rvba: np
<gnuoy> nigelb, bigjools. thanks and done
<deryck> abentley, for your change you had to revert, you landed display-cleanup, reverted in another branch, and then fixed inside display-cleanup and landed that same branch again....
<deryck> abentley, is that right?
<abentley> deryck: that's right.
<deryck> abentley, the "fake merge" commit you had is needed to get in sync with devel after the revert?
<abentley> deryck: right.
<deryck> abentley, how exactly did you fake merge?
<abentley> deryck: bzr merge devel; bzr revert .; bzr commit
<deryck> ah ha.  right, that makes sense.
<deryck> abentley, thanks!
<abentley> "revert ." reverts all file changes, but keeps the merge marker.
<deryck> abentley, hey.  sorry to keep pestering you....
<abentley> deryck: no worries.  What's up?
<deryck> abentley, you have your object like this:  var navigator = Y.lp.bugs.buglisting.ListingNavigator.from_page();
<deryck> abentley, and I work from that when I create my OrderByBar.  I'm going to feature flag all of that, I think.
<deryck> abentley, there's no need for the navigator to be outside the flag, right?
<abentley> deryck: The only thing is testability.  It's easier to test how Navigator behaves when the flag is disabled than to test whether the page has no navigator when the flag is disabled.
<abentley> deryck: But I guess it's not that hard to test either way.
<deryck> ah
<deryck> abentley, so if I change it, tests will break?
<abentley> deryck: No, but you should delete the code they're testing and the tests themselves.
<abentley> deryck: test_render_no_client_listing, test_from_page_no_client
<deryck> abentley, I'm confused, sorry.  Why would I want to delete tests?  We don't want to lose that coverage, do we?
<abentley> deryck: We should have only one strategy for dealing with the feature flag.  So if you change the strategy, you should delete the old code.  And that would make the tests fail.
<deryck> abentley, ah, now I see what you mean.  Thanks.
<abentley> deryck: np.
<deryck> abentley, so this is the sort of thing, I'm thinking:  http://pastebin.ubuntu.com/726516/
<deryck> abentley, just putting our scripts behind a flag.
<deryck> abentley, and as far as I can tell has no impact on your current approach or testing, though I don't mind deleting the empty return and tests if you feel it's unneeded.
<deryck> oh, lunch now, sorry.  chat more when I'm back.
<deryck> abentley, did you see what I wrote in the scrollback?
<abentley> deryck: I did.  I do feel deleting the empty return and the check in render makes sense, because your change makes it dead code, and dead code should be deleted.
<deryck> abentley, fair enough.  I'll kill it then.
<abentley> deryck: cool.
<deryck> abentley, http://pastebin.ubuntu.com/726581/  cool?
<abentley> deryck: You've missed the check in render:     "if (! Y.Lang.isValue(this.target)){"
<deryck> abentley, updated then: http://pastebin.ubuntu.com/726584/  cool now?
<abentley> deryck: yes, thanks.
<deryck> abentley, np.
<abentley> Our linter has spelling errors: "To few newlines before selectors."
<jtv> abentley: it could be a toast.
<jtv> Here's to few newlines before selectors!  <ting> <slurp>
<abentley> jtv: Toasts are usually written as exclamations, so then it would be a grammar error :-)
<jtv> abentley: toasts must be considered in the context of the number, size, and alcohol content of preceding drinks.
<jtv> But I admit your interpretation has merit.
<abentley> deryck: could you please merge r14231 or later into custom-bug-listings and push?  My branch has conflicts that it's inheriting from custom-bug-listings.
<deryck> abentley, sure.  the css-polish branch?
<abentley> deryck: right.
<deryck> ok.
<deryck> abentley, done.  push includes the recent fixes we discussed in the scrollback too.
<abentley> deryck: cool, thanks.
<deryck> np
<abentley> deryck: test_update_from_model_caches is broken in custom-bug-listings-css-polish.  I belive this is because render no longer short-circuits when there's no target.
<deryck> abentley, ah, that sucks.  I think I broke the build then.  I could have sworn this test passed locally.
<deryck> abentley, so fix the test or put the short circuits back?
<abentley> deryck: Yep.  I'd fix the test, since there's only one, and we're already creating a custom object for it.
<deryck> abentley, ok, I'll work on a fix now.
<abentley> deryck: no rush.
<deryck> well, not until the build breaks :)
<deryck> abentley, this gets it working for me:  http://pastebin.ubuntu.com/726648/.   Look sane?
<abentley> deryck: yes.
<deryck> abentley, excellent.  I'll put up an MP.
<lifeless> wgrant: that field.blob - what OOPS id was it on
<lifeless> wgrant: I want to track the emitter down
<deryck> abentley, care to review that testfix?  https://code.launchpad.net/~deryck/launchpad/testfix-buglisting-js-test/+merge/81073
<abentley> deryck: r=me
<deryck> abentley, thanks!
<lifeless> abentley: could I please have a review of https://code.launchpad.net/~lifeless/python-oops-tools/amqp-polish/+merge/81075 ? diff coming in a minute
<abentley> lifeless: sure.
<lifeless> also, its so nice working on a code base with <20sec test runs :P
<lifeless> abentley: thanks!
<abentley> lifeless: to me, ValueError is vague enough that I'd rather check the length.
<lifeless> abentley: its pretty precise:
<lifeless> >>> foo, bar = 1
<lifeless> TypeError: 'int' object is not iterable
<lifeless> >>> foo, bar = [1]
<lifeless> ValueError: need more than 1 value to unpack
<lifeless> >>> foo, bar = [1,2,3]
<lifeless> ValueError: too many values to unpack
<lifeless> abentley: if its not iterable, you get TypeError
<lifeless> abentley: if it is but the wrong length you get ValueError
<lifeless> abentley: we haven't seen any corruption of the third form
<lifeless> abentley: and I plan to make this be an actual dict soon, which will eliminate the corruption entirely once the producers are upgraded
<abentley> lifeless: Yes, but that's not the only way ValueError is used.
<lifeless> abentley: thats true, but its the only way its used for the short expression given
<lifeless> abentley: we know that req_vars is a basic datatype because its just been deserialised
<lifeless> e.g. no special objects implementing their own __iter__
<abentley> lifeless: let's not go overboard about this.  I said I'd rather check the length, didn't mean that you had to.
<lifeless> abentley: sorry, didn't mean to get contentious
<lifeless> I was just about to say I'll change it if you want, was merely explaining why I did it the way I did
<abentley> lifeless: Is line 46 tested?
<lifeless> nope
<lifeless> its probably testable, though its a bit awkward
<lifeless> its in the factory-function that reads from disk files
<abentley> lifeless: I think since it fixes a bug, that we should test it.
<lifeless> if I change one of the sample oopses to have a lower case id
<lifeless> that will cause other test to fail if the line is not present
<lifeless> I think this is the simplest way to ensure its presence is needed
<lifeless> how does that sound?
<abentley> lifeless: that sounds fine to me.
<abentley> lifeless: So with testing, r=me.
<lifeless> thank you
<abentley> lifeless: np
<lifeless> ok, I get failures with the line removed
<lifeless> and not with it present
<lifeless> heh, spoke to soon
<lifeless> something else blew out :P
<lifeless> abentley: \o/ other oopses already exercised that failure mode. So, I'll have a slightly longer diff for you, showing the impact.
<lifeless> hah
<lifeless> abentley: I misanalyzed the cause of our issues around that line.
<lifeless> abentley: I'm going to drop it from the diff and do a different patch for that part of it
<abentley> lifeless: Okay.
<lifeless> src/oopstools/oops/views.py index() is the issue
<lifeless> it treats oopses starting with OOPS specially, which is why the u1 lowercase oopses work and the LP lowercase ones didn't.
<lifeless> abentley: I may end up including the line in the final fix, just want to decouple things to get more time to look at it.
<deryck> Later on, everyone.
<lifeless> abentley: https://code.launchpad.net/~lifeless/python-oops-tools/amqp-polish/+merge/81083 if you have time
<lifeless> bac: thanks!
<lifeless> bac: I was just about to start a branch to fix up my booboo
<bac> lifeless: actually, i haven't gotten too far with it.  if you'd like to add some thoughts to the bug that might be helpful.
<lifeless> bac: I just did
<bac> lifeless: i did see that the report created has no 'type', at least when created via the tests.
<lifeless> bac: tests for this need to be asynchronous tests
<bac> not have a type entry causes the filters to explode
<lifeless> bac: using the custom runner, and returning a deferred from the test to have that executed by twisted
<lifeless> bac: that may be a separate defect (but worth addressing too) - changing from
<lifeless> report['type'] to report.get('type', 'No exception type')
<lifeless> report['value'] to report.get('value', 'No exception value')
<lifeless> should do that
<bac> lifeless: yep, i'd done something similar
<lifeless> I think python-oops-twisted made that change; this is kindof code duplication, because its twisted code logging to to non-twisted logging module.
<lifeless> (also made that change)
<mrevell> hey huwshimi, got time to talk in the next 20 minutes? We need to try something new for the manage-disclosure mock-ups
<mrevell> huwshimi, Good morning, btw :)
<huwshimi> mrevell: Morning
<huwshimi> mrevell: 2 mins
<mrevell> huwshimi, I'll be popping offline but back shortly. No need to talk immediately.
<mrevell> huwshimi, But I'd love to hijack your day... :)
<huwshimi> mrevell: Now's fine with me
<mrevell> huwshimi, Oh, I'm going onto another call just now. I'll ping you in an hour or so.
<huwshimi> mrevell: Ah ok, no problems
<lifeless> flacoste: anytime..
<lifeless> abentley: actually, nevermind, there is a code hcange neeed; next reviewer up can cop it :)
<poolie> hi huwshimi, lifeless
<huwshimi> poolie: Morning
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<poolie> huwshimi: i replied about wrapping; we can talk about that later if you like
<huwshimi> poolie: Sure
<poolie> otp now
<huwshimi> wallyworld_: Do you know anything about adding a security view to the +manage-disclosure page?
<huwshimi> wallyworld_: Also, hello
<wallyworld_> huwshimi: hi. i'm not exactly sure what any security view needs
<wallyworld_> is this a user request?
<lifeless> can I have a review of https://code.launchpad.net/~lifeless/python-oops-tools/amqp-polish/+merge/81083 please? its simple.
<huwshimi> wallyworld_: No, from Matthew
<wallyworld_> huwshimi: the accessibility of project artifacts is basedon policies
<lifeless> huwshimi: what do you mean by security view?
<lifeless> :)
<wallyworld_> huwshimi:  security is orthogonal to that
<wgrant> I assume it's the policy stuff that I discussed on the call this morning.
<lifeless> huwshimi: as in, do you mean the red banner ?
<huwshimi> lifeless: I have no clue, that's what I'd trying to find out :)
<lifeless> wgrant: wallyworld_: I'm OCR today, so I need someone else :P
<wgrant> lifeless: Looking.l
<wallyworld_> huwshimi: i think we need to talk to matthew to clarify
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: lifeless | Critical bugtasks: 266
<lifeless> wgrant: thanks
<wallyworld_> lifeless: sorry, was going to look after conversation finished
<lifeless> wallyworld_: no worries
<huwshimi> wallyworld_: Yeah that's fine, he's just away for a bit and I think he was hoping for some mockups today
<lifeless> wgrant: my commit message and description are now stale.
<lifeless>  Fixing
<lifeless> Fixed.
<wallyworld_> huwshimi: i think there's confusion/disconnect about policy based access and security etc
<wallyworld_> huwshimi: so at some point i think we all need to get on a call to clarify
<huwshimi> wallyworld_: Ah ok
<wgrant> Might have to be late in our evening.
<wgrant> Unless we catch Matt while he's at UDS.
<wallyworld_> huwshimi: matthew is at uds? perhaps we can get together later today our time
<huwshimi> wallyworld_, wgrant: He's gone out for dinner and I think he will try and call when he gets back
<wgrant> Hopefully we can obtain a Curtis as well.
<wallyworld_> huwshimi: ok. let's try and have a call then
<wgrant> Otherwise I know mostly what's going on here.
<huwshimi> wallyworld_: Great
<wallyworld_> it would be good for curtis to be there so he can take what's discussed to folks at uds etc
<lifeless> wgrant: got a thumbnail sketch for me ?
<wgrant> lifeless: Of?
<lifeless> 12:30 < wgrant> I assume it's the policy stuff that I discussed on the call this morning.
<wgrant> lifeless: Pretty much what Curtis, you and I discussed a few weeks back.
<wgrant> That others seem to be largely unaware of.
<lifeless> wgrant: a little more context please
<wgrant> lifeless: The policy-based disclosure approach, to avoid all the hideous security/apport special casing. Private artifacts have an associated policy, with observers for that policy, optionally specific to an artifact.
<lifeless> righto
<lifeless> do you mean db backed policies ?
<wgrant> Yes.
<lifeless> k, thanks
<wgrant> Rather than the "security is magical lalalala" that was initially proposed.
<lifeless> great, so distraction over you can go back to my review ;)
<wgrant> Indeed.
<lifeless> also I may have made oops lookup a brazillion times faster in some cases
<wgrant> lifeless: + oopsids.add(with_oops.lower())
<wgrant> lifeless: Isn't that going to lower the OOPS- too?
<wgrant> Which is probably wrong.
<lifeless> well, depends on where you consider the abstraction barrier
<lifeless> but yes, I can see that being an issue.
<lifeless> will make it a .update(map(lambda x:'OOPS-' + x, oopsids)) instead
<wgrant> Thanks.
<poolie> jelmer: http://twitter.com/#!/mumak/status/131875802630471680
#launchpad-dev 2011-11-03
<jelmer> poolie: upstream-2.5.0~beta1+bzr6182
<StevenK> wgrant: Can I say <tal:block tal:condition="blah" replace="" /> ?
<wgrant> StevenK: replace="" sounds pretty useless, but yes.
<StevenK> wgrant: replace="foo", but okay, thanks.
 * StevenK needs a test first.
 * StevenK shakes fist at TDD.
<poolie> james_w, don't suppose you're here?
<poolie> ok https://bugs.launchpad.net/bzr-builder/+bug/885497
<poolie> wgrant: i think i will try to do that whole series of tweaks to buildd, including making it slightly testable, and then ask for that to be deployed
<poolie> since it is expensive, atomic, and independent of lp deployments it seems better to batch it
<wgrant> Yep.
<lifeless> poolie: one thing that would be brilliant is to remove it from the LP tree
<lifeless> poolie: this requires creating two small projects - one to hold the current common code, and one to hold the buildd production code.
<lifeless> you don't have to, but its in the same space you're looking at.
<poolie> ok, i'll consider that
<poolie> i'll see what  is common
<poolie> classic case of it hard to bootstrap out of being hard/scary to change
<lifeless> readyservice should be the entirety
<poolie> hahah
<poolie> another classic example of that:
<poolie> the bug we hit was fixed in june
<poolie> (885497) but we've been trying to deploy since then so it is not on the buildd today
<wgrant> Oh, so we're using a bzr-builder snapshot from months ago?
<poolie> we're using a snapshot from years ago
<poolie> (on prod today)
<wgrant> s/using/upgrading to/
<poolie> yeaoh
<jelmer> wgrant: it was the release I did back when I started trying to get a newer bzr-builder on the buildds
<wgrant> Right :(
<poolie> it's ok
<lifeless> these consistently make me sad - empty diffs : https://code.launchpad.net/~lifeless/python-oops-tools/amqp-polish/+merge/81083
<wgrant> lifeless: You pushed and merged around the same time?
<poolie> https://bugs.launchpad.net/launchpad/+bug/854520
<lifeless> wgrant: I pushed, clicked approved and tarmac ran
<wgrant> Right.
<wgrant> And MPJ is crap.
<wgrant> So it didn't run until the merge was done.
<jelmer> poolie: --allow-fallback-to-native
<poolie> ftr: ah ok so this is not totally fixed upstream, it just gives a cleaner error
<jelmer> poolie: actually, we shouldn't need a newer bzr-builder at all
<jelmer> just the --allow-fallback-to-native flag should be sufficient
<jelmer> the improved error message is only shown when that flag isn't present
<jelmer> not any newer than what we have in cat-proposed at the moment, that is
<poolie> how about the one that's in production now?
<jelmer> poolie: that doesn't have some of the new features required for bfbia and new debversion variables people have asked for
<jelmer> in particular, the one currently in production can create non-native packages
<wallyworld_> wgrant: StevenK: what's your recollection of the direction being taken for private team name visibility? are private team names still to be hidden when required?
<poolie> right but it will be safe to pass --allow-fallback to that as an interim step?
<poolie> i know it needs tobe updated, i'm just wondering if it needs to be serialized or synchronized with the buildd changes
<jelmer> poolie: older versions don't support --allow-fallback-to-native so will probably blow up if it is specified
<jelmer> poolie: we will need to update launchpad-buildd and bzr-builder at the same time
<poolie> ok
<StevenK> Hm. team.addMember() is returning a tuple that contains TeamMembershipStatus.INVITED. Do I have to do anything to that to make it show up?
<wgrant> StevenK: The invitation needs to be accepted.
<wgrant> Or possibly do it as an LP admin.
<StevenK> I'm doing it in a with celebery_logged_in block
<wgrant> Sounds like you need to do the invitation acceptance dance anyway, then.
 * StevenK was looking for an example of that, and came up empty-handed.
<wallyworld_> StevenK: you doing the addMember() in a test?
<StevenK> wallyworld_: Indeed.
<wallyworld_> StevenK: use makeTeam(members=[xxxx])
<wallyworld_> with name=xxxx as well of course
<StevenK> wallyworld_: Where do I send the cake?
<wallyworld_> give it to spm. he needs it more than i do :-)
<StevenK> Haha
<spm> rofl
<spm> dunno about "need", want, yes. need, not so much. :-)
<wallyworld_> well, need == want for some levels of want :-)
<StevenK> need == want for sufficently high levels of want
<wallyworld_> yes, that's more accurate
<wallyworld_> or, in javascript: need === want
<StevenK> https://launchpad.dev/~super doesn't even show Subteam of when not logged in :-(
<StevenK> Perhaps it needs to be restricted
<wgrant> Is super a subteam?
<wgrant> Sounds more like a superteam to me.
<StevenK> http://pastebin.ubuntu.com/726885/
<StevenK> Possibly badly named
<StevenK> Right, it does need to be restricted
<poolie> huwshimi: how about if we talk now about fonts and markup?
<huwshimi> poolie: Sure
<poolie> here, mumble, ..?
<huwshimi> poolie: Don't mind, whatever is easiest
<poolie> mumble!
<huwshimi> sure
<lifeless> u
<lifeless> can has review ? https://code.launchpad.net/~lifeless/launchpad/bug-885303/+merge/81106
<wgrant> lifeless: Want to trade? https://code.launchpad.net/~wgrant/launchpad/observer-db/+merge/81104
<wgrant> Not exactly equal, but I did do that 3kline one for you last week :P
<wgrant> lifeless: How about quickly hacking oops-tools to accept the ID in the path rather than query string, to work around apache-openid?
<lifeless> wgrant: I have some critical path stuff to finish, like the rsync replacing script
<lifeless> wgrant: remind me next week
<wgrant> Bah
<lifeless> I'm not sure quite whats involved, I'm sure it will be a little fiddly given my lack of know-how on django internals.
<lifeless> If I thought it was 5 minutes, I'd do it now.
<spm> wgrant: you did it wrong. <wgrant> lifeless: it's impossible to hack oops-tools to accept the ID in the path rather than query string, to work around apache-openid. so don't worry about it. we'll just mark a bug and ignore it. <== fixed
<wgrant> Heh
<lifeless> wgrant: your cover letter confuses me a little
<wgrant> lifeless: Good.
<lifeless> third paragraph
<lifeless> Permissions can be either policy-global or artifact-specific. In the first case APP.artifact is left unset,
<lifeless> letting the permission holder see all artifacts under the policy.
<wgrant> lifeless: What is confusing?
<wgrant> lifeless: Why'd you remove the "Check against false positives" test?
<lifeless> do you mean grants when you say permissions ?
<lifeless> what does policy-global mean
<lifeless> what is APP
<lifeless> who is the permission holder?
<wgrant> Well, they are grants, yes.
<wgrant> s/policy-global/policy-wide/
<wgrant> APP == AccessPolicyPermission
<wgrant> Permission holder == APP.person
<lifeless> how do you feel about s/Permission/Grant/
<wgrant> I was following an existing pattern (eg. ArchivePermission), but sure.
<lifeless> that one weirds me out too ;)
<lifeless> perhaps I've spent too long with SQL
<lifeless> wgrant: are you doing a pillar table still ?
<wgrant> lifeless: Possibly.
<wgrant> Not sure if it's worth it.
<huwshimi> poolie: https://bugs.launchpad.net/launchpad/+bug/30002
<wgrant> It's easy to work in if you want me to.
<lifeless> shrug :).
<lifeless> was just curious
<wgrant> lifeless: Oh, I see now why you removed the test.
<wgrant> lifeless: Should we mandate the hyphen?
<wgrant> Otherwise anyone talking about oopses is going to get linkified.
<wgrant> Which would be bad.
<lifeless> oops foo ?
<lifeless> I don't really see a use case for OOPS 123asd123
<wgrant> Right.
<lifeless> since we hyphenate it everywhere.
<wgrant> So mandate the hyphen.
<lifeless> WFM
<wgrant> Otherwise anybody mentioning "oops" in a sentence will get linkified.
<lifeless> also
<lifeless> OOPS - 12345
<lifeless> shouldn't link IMNSHO
<wgrant> Well, depends on word-wrapping.
<lifeless> ugh
<lifeless> sure
<lifeless> hates on that though.
<wgrant> Yes.
<wgrant> I'm reminded of the numbered lists with one item ending in "bug" case.
<poolie> lifeless: my discussion with huw is on hold but is mostly up to looking for a compromise width that is roughly (but no less than) 80 monospace cells
<lifeless> cool
<wgrant> lifeless: Thanks.
<lifeless> my branch has its new commit up.
<wgrant> lifeless: You want a note in the AccessPolicyArtifact comment about adding new types there?
<lifeless> wgrant: somewhere
<lifeless> wgrant: something that says 'this is how its meant to stick together'
<lifeless> wgrant: a memo for yourself in 3 years time
<wgrant> k
<huwshimi> wgrant, wallyworld_: Right, so with the security stuff, basically we need to show who has access to security bugs.
<huwshimi> wgrant, wallyworld_: I gather a team or a user can have access?
<wallyworld_> huwshimi: yes, a team or user or user via their team memberships
<wgrant> huwshimi: Security is going to stop being a special case.
<wgrant> huwshimi: Projects will have multiple access policies.
<wgrant> Each private bug or branch has a policy assigned.
<wgrant> Security bugs would be under the Security policy.
<wgrant> Non-security but private bugs could be under the Private policy.
<lifeless> wgrant: can you note to sinzui that https://help.launchpad.net/Bugs/PrivateBugs#Private_bugs implies everyone gets private-by-default bugs
<wgrant> Each policy has a different set of observers.
<wgrant> lifeless: Yeah, was going to bring that up next time we had a call.
<wgrant> Or throw the bug at him.
<wallyworld_> wgrant: yes, but security bugs still have the security checkbox ticked, so users still see that and wan to know who has access to such bugs
<wgrant> wallyworld_: Right, but the policies behave the same way.
<wgrant> It's no longer a separate role in the project.
<wgrant> To which only one person or team has access.
<huwshimi> wgrant, wallyworld_: So Matthew said this: "The manage disclosure page currently shows people who have access to private stuff but we realised today we also need some way of showing people who have access to security bugs. We treat security and private issues separately as they often have to be seen by different people."
<wallyworld_> so perhaps we need a view to show what artifacts can be seen under each policy
<wgrant> huwshimi: Right.
<wgrant> huwshimi: There will be two separate classes of private objects.
<huwshimi> wgrant: Is that no longer valid?
<wgrant> huwshimi: With the same set of controls.
<wgrant> huwshimi: It is mostly valid.
<lifeless> wgrant: btw, bugsummary will need an update for this too
<lifeless> wgrant: though I expect you knew that.
<wgrant> huwshimi: It will eventually be more than just private and security. It is expected that projects will be able to define their own policies in future.
<wgrant> lifeless: Yes, lots of stuff does :/
<wallyworld_> huwshimi: so if there were a listing where one could select the user and then select a policy and it would show what a user can see under that policy, would that help?
<huwshimi> So in the UI we will still need to distinguish between them
<wallyworld_> and one of the policies would be security
<wallyworld_> huwshimi: i mistyped a bit above. what i meant was choose a policy and then see who can access stuff
<huwshimi> wallyworld_: tight
<huwshimi> *right
<wgrant> lifeless: It's a bit sad that the first significant change to be done through fastdowntime is probably the biggest, most awkward model shift in Launchpad's production history.
<lifeless> wgrant: you're landing the soyuz un-refactoring finally?
<wgrant> Hah
<wallyworld_> huwshimi: so at the moment, we have a user centric listing ie what can a user see. sounds like we need another new page for policy centric listing
<lifeless> see what I did thar?
<wgrant> wallyworld_: I'm imagining something maybe sort of like the AJAX filtering on +localpackagediffs.
<huwshimi> wallyworld_: I think Matthew was talking about adding that to the filters
<wgrant> wallyworld_: So the main listing lists all permissions for the project.
<wallyworld_> yes
<lifeless> wgrant: going to approve my mp ?
<wallyworld_> the main overview page would be the union of all polocies
<wgrant> wallyworld_: With rows like "Some Team | Security (observer), Private (restricted observer)"
<wgrant> And then the AJAX filter widget on the second column.
<wgrant> Allowing you to select a subset of policies to care about.
<wgrant> Like the packagsets column on https://launchpad.net/ubuntu/precise/+localpackagediffs
<wgrant> lifeless: Once the diff's updated. Which it probably is by now.
 * wallyworld_ loks
<wallyworld_> looks
<huwshimi> wgrant: Can they have security and private?
<lifeless> there is no need for non-ajax rendering in this policy page: its modern browsers only stuff
<lifeless> and not googlebot visible
<wgrant> huwshimi: Hm?
<huwshimi> wgrant: Can a teams policy for viewing a private bug be because they can view private and security bugs
 * huwshimi may not be using the right terminology
<lifeless> huwshimi: the bug will be categorised as either private or security
<lifeless> huwshimi: thats a link to the policy
<wallyworld_> wgrant: that filter popup is yucky
<wallyworld_> there's no way to "select all" again
<wgrant> wallyworld_: Yes, but I think the concept's good.
<wallyworld_> +1
<wgrant> This is also not dissimilar to what Bugs is doing, I expect.
<lifeless> huwshimi: so they will be able to view a given bug (asset) because they have a grant in the matching policy
<wgrant> s/Bugs/bug-columns/
<wgrant> lifeless: approved, thanks.
<wallyworld_> lifeless: i wasn't planning on not doing ajax only :-)
<lifeless> wgrant: oh, I forgot one hting
<lifeless> wgrant: why not name column ?
<lifeless> wgrant: s/not/no/
<lifeless> wgrant: for identification in the API - url traversal etc
<wallyworld_> huwshimi: there will initially be only out-of-the-box polices, hard wired
<wallyworld_> huwshimi: and the meanings or what they allow to be seen will be well explained
<wallyworld_> huwshimi: do you have enough to take back to matthew?
<huwshimi> wallyworld_: I'm just uploading a mockup to see if what I'm doing makes sense
<wgrant> lifeless: It's not a very linkable thing, so I decided I didn't care much about pretty URL. Display name is a unique reference for use in the API. I can add a name if you want, but it's awkward since we don't autogenerate them.
<wallyworld_> huwshimi: i will likely not have enough time to change the current mockup to add in the policy stuff before the next planned testing
<lifeless> wgrant: we don't ?
<lifeless> wgrant: I mean, with 2 hard coded policies, they have fixed values
<huwshimi> wallyworld_: I think Matthew just wants something to show some people at UDS
<lifeless> wgrant: and private (name) Private (display name)
<wgrant> lifeless: True, and I guess we do have precedent for interactively autogenerating on ProductSet:+new
<wgrant> OK, will add.
<wallyworld_> i think we need a name FWIW
<huwshimi> wallyworld_ wgrant: http://people.canonical.com/~huwshimi/policy.png
<lifeless> wgrant: I see you decided on per-project rows rather than site-wide rows
<huwshimi> wallyworld_, wgrant: Does that make any sense?
<lifeless> wgrant: thats fine with me - what led you to that conclusion ?
<wgrant> huwshimi: Not exactly. It's possible for me to be an Observer for Private, and a Restricted Observer for Security.
<wallyworld_> huwshimi: we only want the policy column
<wallyworld_> the policy defines what the user can see
<wgrant> wallyworld_: Oh?
<wgrant> wallyworld_: There's still an observer/restrictedobserver concept, within each policy.
<wallyworld_> wgrant: yes
<wallyworld_> i was thinking it was part of the policy but yes, it is separate
<wgrant> lifeless: Nothing very strongly either way. It's possible that if we want to keep retargeting easy it'll be easier to have site-wide policies, but eeeh.
<wgrant> lifeless: But it's cheap enough to switch either way later.
<wgrant> Even with fastdowntime.;
<lifeless> wallyworld_: hi
<wallyworld_> wgrant: huwshimi: it will be difficult to represent the policy/permission concept concisely in a table row
<wallyworld_> o/
<lifeless> wallyworld_: your recent branches on filtering look broken to me - why are you changing browser code for model issues ?
<wgrant> wallyworld_: I don't see a better way than "Some Team | Private (observer), Security (restricted observer)"
<lifeless> wallyworld_: If I may suggest, this is code that doesn't need to be written at all, if you do the proposed partial disclosure of private teams, which I know sinzui has scheduled somewhere
<wgrant> Unless we assume that people have a reasonable number of policies and turn each into a column.
<huwshimi> wgrant: Oh, now I understand, you can have different permissions for different policies
<wallyworld_> lifeless: that's where the current filtering is done, i just added to the existing filter condition
<lifeless> wallyworld_: its a different sort of filtering
<wallyworld_> lifeless: i see you point about the partial p team disclosure. wonder why the bug was raised then
<lifeless> bugs :)
<wallyworld_> i raised the 2nd one
<wgrant> huwshimi: Right. eg. someone could have access to all of Ubuntu's Private bugs, and then report a Security bug. They need access to that Security bug, and all Private bugs, but not the rest of the Security bugs.
<wallyworld_> based on the first
<lifeless> kk
<wgrant> lifeless: I think I should still keep UNIQUE (pillar, display_name), as there's no reason the pillar owners can't keep their own display names unique?
<lifeless> anyhow, just a thought... your team will be fixing up the fallout one way or another :)
<lifeless> wgrant: two words: social attacks.
<wgrant> lifeless: If the project owner is launching a social attack, the project is in trouble anyway...
<wallyworld_> lifeless: i think the filtering is messed up then. "private and user_can_edit" also belongs in the model in that case
<lifeless> wgrant: not on themselves :P
<lifeless> wallyworld_: one of the first things I noted when I became TA is that we have too much code in our browser classes ;)
<wgrant> lifeless: So, should I keep the UNIQUE or not?
<wallyworld_> lifeless: agree 1000%
<lifeless> wgrant: I think the UNIQUE has to stay.
<wallyworld_> the privacy setting and event raisng stuff in the browser is total bollocks
<lifeless> wgrant: on name and separately on display name.
<wallyworld_> for example
<wgrant> lifeless: Right, that's what I had done. Thanks.
<wallyworld_> lifeless: i was merely taking a pragmatic view to tweak what was already there
<lifeless> wgrant: I'm just noting that editing is perhaps a problem (unless we show Private (private))
<lifeless> wallyworld_: you may not be aware of this, but bug subscription calculations are a big chunk of rendering a bug
<wgrant> lifeless: Given that the namespace is controlled by the project owner and display_name is UNIQUE, that should be unnecessary.
<lifeless> wallyworld_: so you want to be confident you're not triggering late evaluation when touching this stuff :)
<lifeless> wgrant: retargeting
<wgrant> lifeless, wallyworld_: I'm not sure if it's changed since stub's fixes last night, but yesterday the bug subscription queries made up like 80% of bug render time.
<huwshimi> wgrant, wallyworld_: Like this then? http://people.canonical.com/~huwshimi/policy2.png
<lifeless> wgrant: and in future linking probably
<wallyworld_> lifeless: i sort of knew. hence the desire not to mess with the existing stuff at a lower level also. i am almost certain that the permission check will not add any late eval but could do a test i suppose
<wgrant> lifeless: I'm not sure how that's relevant. You're only going to be seeing policies from one project. If you can't trust the owner of that project, you can't safely assign the bug there at all.
<wgrant> huwshimi: Indeed.
<lifeless> wallyworld_: anyhow, even if its moved to the model, it will still be wrong
<lifeless> wallyworld_: because we want to show the subscribers
<lifeless> wallyworld_: having hidden subscribers is a defect
<lifeless> wallyworld_: (consider someone replying to a bug notification email, who is a hidden subscriber...)
<huwshimi> wallyworld_: Does that mockup make sense to you too?
<wallyworld_> lifeless: sure, but at the moment it oopses
<wallyworld_> and until now, private team visibility was verboten
<lifeless> wallyworld_: its a soft oops added by StevenK to track places that need updating
<StevenK> In fact, I'm fixing one now.
<StevenK> Or attempting to, rargh!
<StevenK> Bloody tests.
<wallyworld_> lifeless: i didn't try the ui, just wrote the test, are you saying ui will render ok, just with "<hidden team>" as the display name?
<lifeless> wallyworld_: yes
<lifeless> wallyworld_: thats the status quo
<StevenK> *And* it will throw a MixedVisibility OOPS.
<StevenK> So fix it.
<lifeless> wallyworld_: the fix is to a) tell folk we're changing this and b) show the team display name, name and url.
<lifeless> StevenK: has the policy change been communicated yet ?
<StevenK> lifeless: I have no idea.
<lifeless> StevenK: just fix it isn't useful if we're not ready to change things
<wallyworld_> lifeless: so to do (b) we need to change the zope permissions, no?
<StevenK> Curtis filed a bug saying we shouldn't show private teams under Subteam of
<wallyworld_> and he filed the bug i did the mp for, same issue sort of
<lifeless> I think something has gotten confused then
<james_w> poolie, am now
<huwshimi> wallyworld_: Just wanted to check you were happy that the second mockup made sense before I spend to much time fixing up all the other mockups
<StevenK> wallyworld_: Then link lifeless the bug
<StevenK> Easy solution
<wallyworld_> huwshimi: sorry, just one sec
<lifeless> as that is diametrically opposed to the plan in e.g.
<lifeless> bug 855670
<huwshimi> wallyworld_: No problems :)
<poolie> hi james_w, i think it's all sorted
<lifeless> wallyworld_: and bug 405277
 * wallyworld_ looking
<james_w> good good
<james_w> how is everyone today?
<poolie> good
<poolie> are you at uds?
<poolie> i was going to show you https://bugs.launchpad.net/launchpad/+bug/885497
<lifeless> StevenK: what bug are you doing ?
<StevenK> lifeless: bug 884217
<wallyworld_> lifeless: so it seems there's a much larger issue at play here and when that is addressed, the bug i worked on will disappear
<lifeless> that larger issue is AIUI, in scope
<wallyworld_> huwshimi: wgrant: the 2nd mockup is better but i'm concerned about how it will scale for users with more than 1 or 2 policies/roles
<lifeless> StevenK: I've commented
<StevenK> I can't see Subteam of when I'm not logged in :-(
<wgrant> wallyworld_: Indeed.
<lifeless> StevenK: wallyworld_: so yes, there is a different picture here.
<lifeless> StevenK: wallyworld_: if curtis wants you to mask this then come back and change it, fine, but I would be very surprised by that.
<wallyworld_> lifeless: yes, it does appear in scope. but i'll need to talk to curtis before we charge off to do anythong
<wallyworld_> lifeless: i was wondering if those bugs were raised as a stop gap
<StevenK> Right, then I can't work on this either
<wallyworld_> since they are recent
<wallyworld_> and the original scope was done a while ago
<wallyworld_> huwshimi: wgrant: perhaps we should show the most permissive policy/role and have an ellipses to trigger a popup to see the rest?
<wgrant> ~wallyworld_, huwshimi: What if each policy is a column (possibly with the name rotated to vertical, if people end up having insane numbers)?
<wgrant> Possibly.
<wgrant> I guess people can filter if they want to see the more restricted policies.
<wallyworld_> yes
<wallyworld_> better to show worse case in the default view
<wgrant> Well.
<wgrant> That's the thing.
<wgrant> It's not necessarily the worst case.
<wallyworld_> but how to quantify that
<wgrant> Exactly.
<wgrant> For Ubuntu, the worst case is any permission on Security.
<james_w> poolie, I am
<huwshimi> wallyworld_ wgrant: How would the Driver, Owner etc. fit into those columns?
<wallyworld_> that the trouble with thinking out loud in a public forum
<wgrant> huwshimi: Ideally they don't.
<wgrant> huwshimi: Those roles aren't special.
<james_w> poolie, the synchronized update is a shame
<wgrant> huwshimi: Although in the default config they will give observer permissions.
<StevenK> lifeless: I was also looking at bug 618399, your analysis is out of date.
<wallyworld_> wgrant: huwshimi: the original mockups implied those roles were special
<poolie> james_w i guess i could make it look at the version number?
<poolie> or...
<StevenK> Where has mup gone?
<wgrant> wallyworld_: The original mockups are wrong :)
<poolie> maybe something else
<lifeless> StevenK: oh, the analysis of bug 618399 ?
<poolie> i don't have a good sense whether upgrading two packages is especially dangerous
<wgrant> huwshimi, wallyworld_: So, when I create a new project, the security contact will get an observer permission on the Security policy. But that can be changed.
<lifeless> StevenK: I had friction there ;)
<StevenK> lifeless: You did?
<wallyworld_> wgrant:  huwshimi: yes, just what i was going to type
<lifeless> StevenK: I thought you were saying the private team analysis was out of date :)
<StevenK> lifeless: No, which I mentioned a different bug number.
<huwshimi> wallyworld_ wgrant: so are they just treated like everyone else now? Do they need to be noted e.g. observer (Owner)
<wallyworld_> huwshimi: wgrant: and there's the display mode which shows implicit access via team membership, eg if someone is in the bug supervisor team
<wgrant> huwshimi: We could add that note if we want, but I don't think it's really critical.
<StevenK> lifeless: I strongly suspect there are a bunch of repeated queries, but the code you implicated in the bug has moved/been deleted.
<wgrant> wallyworld_: lifeless argues that that is silly.
<wgrant> wallyworld_: Because eg. most projects will have ~canonical, and that is huge.
<wgrant> wallyworld_: But I think it should still be done.
<wallyworld_> wgrant: i don't think it's silly
<wallyworld_> because you can see *why* someone has access
<wallyworld_> and correct it if needed
<huwshimi> wallyworld_ wgrant: So move to private/security columns then?
<lifeless> mmmm, I argue that a per-person collection for all of ~canonical does not meet user needs.
<lifeless> they need searches
<wgrant> It certainly needs a search.
<lifeless> e.g. 'show me folk with permission not in ~canonical'
<lifeless> -vastly- more useful.
<wallyworld_> lifeless: yes, but what about the case where a project owner says "tell me why fred can see my shit"
<wallyworld_> and you can say "that's because he is in team xxx"
<wallyworld_> "and team xxx has private/observer permission"
<lifeless> wallyworld_: thats useful too
<lifeless> wallyworld_: imagine finding fred by clicking next 15 times.
<lifeless> wallyworld_: and xxx likewise.
<lifeless> wallyworld_: *that* isn't useful.
 * StevenK notes lifeless has gotten distracted.
<lifeless> expanding out teams in the default view will *devalue* the view.
<wallyworld_> lifeless: no, but you search for fred and he comes up in the results and you can instantly see his indirect access
<lifeless> wallyworld_: sure, and I think thats desirable.
<wgrant> lifeless: In the default view, yets.
<wgrant> yes.
<wgrant> It certainly needs to default to showing only explicit grants.
<wallyworld_> lifeless: so i think we agree, we're just discussing different valid use cases, both of which will be supported
<lifeless> I think a displaymode that shows *everything* expanding teams isn't useful
<wallyworld_> lifeless: we don't have that
<lifeless> I have had the impression that that is currently intended.
<lifeless> from statements like '17:20 < wallyworld_> huwshimi: wgrant: and there's the display mode which shows implicit access via team membership, eg if someone is in the bug supervisor team
<wallyworld_> we have a list matching the search criteria which can optinally show if access is indirect or not and how
<wgrant> That has to show implicit access via team membership.;
<wallyworld_> lifeless: ^^^^
<wgrant> It's unlikely it will ever be used in an unfiltered form, however.
<lifeless> Perhaps I'm being odd about this
<lifeless> I'm sure you have a good understanding; I'm probably misunderstanding due to language things
<wallyworld_> lifeless: take my 2 statements above together and the other stuff i've just said that you agreed with. that's the intent of it, not an expanded team view
<lifeless> I just want to save you from implementing a very slow thing because you're bringing back hundreds of times the needed data within the DB
<poolie> james_w, oh, the other option would be to turn that option on by default for dailydeb
<wallyworld_> huwshimi: wgrant: the private/security columns question
<wallyworld_> not sure that will work well for everything
<wgrant> wallyworld_: Oh?
<james_w> poolie, yeah, I'm not sure if that's a good idea
<wallyworld_> perhaps it will. i guess those two categorisations are the only ones we will have
<james_w> poolie, I'd forgotten that this was both in packages
<poolie> builder and buildd?
<poolie> also they're hard to type together
<james_w> so upgrade should be fine
<james_w> heh, yeah
<lifeless> StevenK: still 80 spph lookups
<lifeless> StevenK: if you can repro locally, LP_SQL_DEBUG_EXTRA=1 should give you the code paths
<poolie> http://lpqateam.canonical.com/qa-reports/deployment-stable.html often seems to get confused about qa-ok vs untestable
<poolie> eg bug 884092
<wgrant> poolie: eg.?
<poolie> i guess it doesn't matter as long as it knows they're not blocking deployment
<StevenK> lifeless: Sure -- what I'm saying is I couldn't figure out which code path is implicated.
<huwshimi> wgrant wallyworld_: So with the columns approach it could be something more like this: http://people.canonical.com/~huwshimi/policy3.png
<poolie> 884092 is called 'ok' on the report but is untestable on the bug, and always has been
<wgrant> poolie: Ah, qa-untestable counts as qa-ok.
<poolie> but some others are called untestable?
<huwshimi> wgrant wallyworld_: But wallyworld_ might be right about it not scaling very well (if/when there are custom policies)
<wgrant> poolie: 'untestable' on that report means that there's no bug, or it's marked as [no-qa]
<poolie> oh, no-qa is called untestable
<poolie> ok
<StevenK> By rights, qa-untestable should be untestable too
<wgrant> StevenK: Yes.
<wallyworld_> huwshimi: wgrant: i think that look ok, but we want the edit icons next to each permission to trigger insitu popup editing
<StevenK> But the deployment report is ... crap
<wgrant> But lots of things should be different :P)
<poolie> not a big deal
<huwshimi> wallyworld_: To edit security/private individually?
<wgrant> wallyworld_, huwshimi It would be more scannable if the columns weren't so wide and text.
<wallyworld_> huwshimi: yes, on the current interactive demo, you can change the permission from observer -> restricted etc by clicking the cell
<wallyworld_> ie the edit icon in the cell
<wallyworld_> like for bug importance etc
<StevenK> lifeless: Did you determine the 80 SPPH lookups locally or from an OOPS?
<huwshimi> wgrant, wallyworld: So, sticking with the columns like this then (with a few improvements)?
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=2102F9#repeatedstatements
<wallyworld_> huwshimi: wgrant: i think for the next iteration that;s ok. get user feedback etc
<wgrant> wallyworld_, huwshimi: Yeah, looks reasonable.
<wgrant> I think it may be better to use less text, but I guess we'll see.
<huwshimi> wgrant: What do you mean by less text?
<wallyworld_> it's hard to think of what "less" text to use
<wgrant> huwshimi: Well, "Restricted observer" and "Observer" are pretty huge ways to represent a tri-state field.
<wgrant> Makes the table very unscannable.
<huwshimi> wgrant: Ah I seee
<wallyworld_> huwshimi: so when you talk to matthew, you can let him know the interactive demo has been tweak as per the initial feedback and we are looking to move forward also with the stuff discussed today
<wallyworld_> huwshimi: wgrant: we could use abbreviations like O and R and have popup help bubble?
<wgrant> wallyworld_: That was along the lines of what I was thinking. That would also allow us to fit in an awful lot of columns, but hopefully nobody will ever want to do that.
<wgrant> But you never know with OEM :)
<wallyworld_> wgrant: abbreviations are difficult because you need to discover what they mean, but seems like we do need it here
<wallyworld_> anyways, huwshimi can ask the users what they want :-)
<wgrant> These would pretty clearly make the page more readable, and the existing terms need description anyway.
<wallyworld_> yep
<wgrant> So, something to consider.
<huwshimi> wallyworld_: Well I think Matthew will be talking to people (I'm not at UDS)
<wallyworld_> oh, i thought you were for some reason
<wgrant> lifeless: Any objections to removing the old non-CTE visibility code?
<wgrant> lifeless: Doesn't seem to have caused any new timeouts.
<poolie> fruit salad! http://people.canonical.com/~jelmer/recipe-status/bzr.html
<wgrant> poolie: I hate to think how long that took to generate.
<poolie> or how many sql queries, when it should be about 1
<lifeless> wgrant: we may be able to revert back to it and it is a bit simpler (there was talk about a regression in 8.4.9)
<poolie> lifeless:  if you're orc and not flat out could you read
<poolie> https://code.launchpad.net/~mbp/testscenarios/module-scenarios/+merge/80287
<poolie> https://code.launchpad.net/~mbp/launchpad/678090-affected-count/+merge/81108
<poolie> neither is very big
<wgrant> lifeless: Well, it's all going to die soonish anyway.
<poolie> <lifeless> poolie: this requires creating two small projects - one to hold the current common code, and one to hold the buildd production code.
<poolie> do you have any opinion about what they should be called?
<poolie> lp-buildd and ...?
<lifeless> I haven't thought about it
<lifeless> but https://dev.launchpad.net/CreatingNewProjects
<lifeless> is our guidelines
<wgrant> Well.
<wgrant> Or you can just work out how to remove lp-buildd's dep on readyservice.
<wgrant> That's all it needs from LP nowadays, IIRC.
<lifeless> yes
<lifeless> thats used for the LP test suite
<lifeless> to detect when the test slave is ready to use
<wgrant> It can't just try to connect?
<lifeless> perhaps
<lifeless> I have no opinion
<poolie> https://bugs.launchpad.net/launchpad/+bug/800295
<poolie> bot is on leave today?
<nigelb> Its been like this since yesterday
<nigelb> (morning btw)
<poolie> hi nigelb
<nigelb> Hey poolie!
<nigelb> I realized UDS is hard even when remotely partcipating :)
<poolie> :)
<poolie> maybe i'll see you at the next one
<nigelb> Hopefully
<nigelb>  :)
<poolie> wgrant: just trying to connect sounds good to me
<poolie> it does seem to be barely used
<huwshimi> wallyworld_ wgrant: Does this look roughly ok now? http://people.canonical.com/~huwshimi/policy4.png
<wgrant> huwshimi: Hm, maybe the role could go in a separate column?
<wgrant> Since it's presently duplicated between the two, and bears no particular relevance to either one.
<wgrant> or it could just be parenthesised in the Person column.
<huwshimi> wgrant: Yeah I was going to suggest the latter
<wgrant> That looks reasonable. I hope we can do better than that with the search UI, but it otherwise looks good.
<huwshimi> wgrant: There's certainly room to improve, but Matthew just needs something by tonight
<wgrant> The edit icons should probably only be shown on hover... or there is an old more subtle variant that we could use.
<wgrant> Otherwise it's going to look *really* cluttered.
<wgrant> Right.
<wgrant> stub: Hi. Will you able to review https://code.launchpad.net/~wgrant/launchpad/observer-db/+merge/81104 todayish?
<stub> yer, already had a quick look
<wallyworld_> huwshimi: looks good
<wallyworld_> huwshimi: but i think we are missing the edit icons in a table column which take the user to the person/team listing page
<wallyworld_> huwshimi: like i did in the latest mockups
<stub> wgrant: How will the policies be created?
<stub> wgrant: As in the AccessPolicy records?
<wgrant> stub: Initially by model code on project creation, and a garbo job.
<huwshimi> wallyworld_: Oh, do you have a link?
<huwshimi> wallyworld_: Or are these the mockups you did ages ago?
<wgrant> stub: Once we're done with the migration, project owners will be able to create them themselves.
<wgrant> stub: (for example, Ubuntu has two additional polices to allow apport to work properly)
<wallyworld_> huwshimi: http://people.canonical.com/~ianb/disclosure/manage_disclosure_index.html
<huwshimi> wallyworld_: Ah thanks
<stub> Why did you go with a text field display_name rather than an enum?
<wallyworld_> huwshimi: same url as before, but i updated based on feedback from dan
<wallyworld_> huwshimi: i did a cc all on the email i sent but you must not have been on the list. i'll forward now
<wgrant> stub: Some projects need customisation, and it's not clear that an enum gives us significant value.
<wallyworld_> huwshimi: did you get dan's original email?
<huwshimi> wallyworld_: No I did not
<wallyworld_> i'll send that too then
<stub> wgrant: The theory is that enums involve less storage space and pick up typos at lint time.
<huwshimi> wallyworld_: This mockup seems to have some old stuff in it too (than the last mockups I sent)
<lifeless> stub: wgrant is adding a name column too FWIW. enum may be better - I have no fixed opinion
<wgrant> The column should be in the diff now.
<wgrant> name column, that is.
<stub> wgrant: Will every display_name have corresponding code changes in lp? And is it possible for a (product, display_name) policy for two different policies to act differently?
<wallyworld_> huwshimi: i only updated based on dan's feedback. which ones did you send?
<wgrant> stub: No, there will be no code changes for new policies.
<stub> I see no name field
<lifeless> welll
 * wgrant rechecks.
<lifeless> there may be business rules about who gets which policies
<wgrant> Well, it's probably a boolean of the form "you are allowed to create new policies"
<wgrant> Because you are Canonical or have paid us.
<wgrant> stub: Hm, what diff are you looking at?
<wgrant> I pushed the new column more than two hours ago, and requested your review afterwards.
<stub> I mean that if I create a record ('ubuntu', 'Lord of All') would there need to be code change to make that do something useful, or are people accessing this stuff via the apis or something external to Launchpad?
<wgrant> So even the email should have it.
<stub> https://code.launchpad.net/~wgrant/launchpad/observer-db/+merge/81102
<stub> oic. old mp
<wgrant> Yeah, 81104 is new.
<wgrant> 81102 should be dead...
<wgrant> (I merged devel, but didn't set it as a prereq the first time)
<wgrant> stub: It'll still do useful stuff. Eventually we'll have UI to choose a policy, and there will shortly be UI to customise who can access a policy.
<wgrant> The initial pass will retain the private/security flags in the UI, mapping them onto two well-known policy names.
<stub> Yes, I'm just trying to see if there is any point having this data in the database or if it should be hard coded in an enum (dropping name & display_name from the schema)
<wgrant> That will later be replaced/extended with a way to select an arbitrary policy.
<wgrant> For example, Ubuntu needs 'apport unprocessed' and 'apport processed' or similar policies.
<lifeless> well maybe
<lifeless> apport can just move off of lp entirely
<wgrant> In 5 years.
<lifeless> nah
<lifeless> no need to wait that long
<wgrant> It's been on LP for a bit over 5 years so far :)
<wgrant> Another 5 wouldn't surprise me.
<stub> At this point I can't see any reason to have two text columns and 4 indexes over one integer column and two indexes. I can see some disadvantages.
<wgrant> We'll need to talk to stakeholders about restricting them to a single set of artifacts.
<wgrant> But I guess it might work.
<stub> What is the restriction here?
<huwshimi> wallyworld_: The last mockups I sent were ages and ages ago
<wgrant> stub: I'm not adding an OEM Engineers value to an enum.
<poolie> https://code.launchpad.net/~mbp/launchpad/800295-buildd-split/+merge/81111
<poolie> probably prepares to split them
<wgrant> stub: If we restrict everything to an enum, when someone comes up with a new reason to split out some of their artifacts, they need a code change, and we need project-specific code in LP.
<stub> wgrant: How does this work with the proposed model?
<wgrant> stub: They click the "New access policy" link.
<lifeless> there is some undesigned stuff here
<wgrant> They enter details, then wander over to the relevant artifacts and click "Change access policy"
<stub> Right, and they select from an existing list or do they type in some names?
<lifeless> particularly around tightly coupled bugs (cross pillar)
<wgrant> Type in some names.
<wallyworld_> huwshimi: sorry i didn't use them this time. hopefully the latest work you've done plus the next round of testing will see us ready to start the proper coding work. i'd be interested in your thoughts on the emails i just sent too at some stage
<lifeless> and for now, we're forcing a specific set of policies
<wgrant> For the initial migration, yes.
<stub> ok. And then is there some science fiction to map this policy to a set of permissions?
<stub> through the web or via apis or something? Or does that require code changes?
<wgrant> Web UI.
<wgrant> Through the usual disclosure management web UI.
<stub> ok. So the current model makes sense to support that.
<wgrant> http://people.canonical.com/~huwshimi/policy4.png are the latest mockups.
<wgrant> Well, a subset of them.
<stub> How many rows do we expect in AccessPolicy btw? Wondering if the indexes should be partial or not.
<lifeless> wgrant: I thought that hadn't been decided on
<wgrant> lifeless: What hadn't?
<lifeless> wgrant: that we were implementing a saner way of doing what we do today, future proofing *future* design decisions
<huwshimi> wallyworld_: Sure, I need to EOD but I'll take a look tomorrow. I think I'll need to do a bit of a review of the UI after UDS is over and we have the feedback so I can make sure things are not being overlooked
<huwshimi> wgrant: Actually http://people.canonical.com/~huwshimi/policy5.png has the last change (moving the role)
<wgrant> lifeless: Well, we need at least two custom access policies, and I believe OEM will want more.
<lifeless> what two ?
<wgrant> lifeless: apport
<wgrant> And no, we're not moving apport off LP next week.
<lifeless> we don't
<wallyworld_> huwshimi: np. i didn't expect anything else today :-)
<wgrant> stub: Just over two per pillar, I suspect.
<wgrant> stub: They probably should be partial, yes.
<lifeless> wgrant: are you saying we *need* it because apport bugs are currently using the limited-subscriber-list hack ?
<stub> wgrant: Should AccessPolicyGrant.artifact be declared NOT NULL?
<wgrant> stub: No.
<wgrant> stub: There are policy-wide grants.
<wgrant> stub: Which are implemented by leaving artifact null.
<wgrant> lifeless: Yes.
<stub> wgrant: We will need to fix that UNIQUE then as it doesn't do what you think it does
<wgrant> stub: I have a second partial one afterwards.
<lifeless> wgrant: so, one crashdump policy should supply that
<wgrant> lifeless: WIth a few hundred thousand restricted observers.
<stub> wgrant: I think we need two partials here.
<wgrant> Sure.
<lifeless> wgrant: !cite
<wgrant> stub: UNIQUE (policy, person, artifact) should work where artifact is not null, right?
<wgrant> lifeless: Once apport has processed a bug and removed the core dump, it subscribes a restricted Ubuntu team.
<lifeless> wgrant: oh, for a second pass review? I thought it made them public straight away
<wgrant> stub: Then I have a UNIQUE (policy, person) WHERE artifact IS NULL afterwards.
<wgrant> lifeless: If they're dupes I think they get all the attachments obliterated, and then made public.
<stub> wgrant: I think what you have works, but the index has unnecessary rows (all the NULL artifact rows, and any queries with NULL artifact would be using the partial index)
<wgrant> lifeless: But normally they get the coredump removed, retraces attached, and ~crash-triagers-universe or similar subscribed.
<lifeless> we could do two tables
<lifeless> one full one restricted
<wgrant> stub: True.
<wgrant> Could.
<wgrant> But now I can just say AccessPolicyGrant.artifact IS NULL or AccessPolicyGrant.artifact = blah :)
<lifeless> OTOH preventing nuts data is easier with one table
<stub> wgrant: Can we swap the order of those indexes around, putting the person column first to keep people merge happiest?
<lifeless> e.g. if we want to say you can't be restricted *and* full
<stub> wgrant: if we are doing queries like that, fine.
<lifeless> wgrant: I don't think support apport is equivalent to supporting arbitrary custom policies
<wgrant> stub: That's the main query. Checking whether someone has access to the whole policy, or this particular artifact.
<lifeless> wgrant: I think its fixing 5 year old tech debt where apport used a poor mechanism to its advantage
<wgrant> stub: Ah, I did have a second index with person first, but dropped it somewhere along the line.
<wgrant> stub: Need policy first for some things, so I'll readd a second one.
<lifeless> wgrant: so we start with 4 policies, not 2
<stub> wgrant: yes, add a separate index for person then.
<lifeless> wgrant: that doesn't imply arbitrary custom policies, nor the UI to support that.
<wgrant> lifeless: Will need to check with stakeholders, particularly OEM, but perhaps.
<lifeless> wgrant: we haven't offered custom rules to OEM AFAIK
<lifeless> wgrant: but sure
<wgrant> lifeless: Yes, and we've already fucked them over once.
<lifeless> wgrant: you're referring to the bug pillar constraint?
<wgrant> Yes, particularly the mass-deletion.
<lifeless> hasn't been done yet has it ?
<wgrant> They're not at all clear on what we're doing.
<wgrant> And what constraints we're putting in place.
<lifeless> yeah, that has been largely resolved today, though I think you'll scream :)
<wgrant> But saying we only have two policies, we're restricting what they can do.
<wgrant> Oh?
<lifeless> the oem project is really the union of disclosure + bug deps
<wgrant> They had better not be back in scope.
<wgrant> That extends the project by months.
<lifeless> nope
<lifeless> just ignore them, disclosure doesn't need to be usable by oem until both projects are finished
<wgrant> Huh?
<wgrant> You know we're ripping out the old privacy mechanism, right?
<stub> wgrant: Why do we want the links on bug and branch to AccessPolicy, when the links are already there via AccessPolicyGrant and AccessPolicyArtifact?
<lifeless> the proposal is to leave them in place while you get all the rest of the work done, for oem under a ff
<wgrant> lifeless: Fuck. No.
<lifeless> more or less
<wgrant> Maybe if we were a sensible bugtracker.
<wgrant> But we're not.
<wgrant> All our projects are intertwined.
<wgrant> stub: That's one of the bits I'm less sure of. Firstly, there isn't an explicit grant for every artifact. I could put the policy on AccessPolicyArtifact, but then we have to join through an extra table for every private bug.
<stub> Yes, it seems denormalization. Wondering if it is necessary or if it is premature optimization.
<wgrant> It is a minor denormalisation, yes.
<stub> Branch and Bug just keep growing wider :-(
<wgrant> lifeless: Is this being seriously considered?
<wgrant> lifeless: It's probably faster to do bug dependencies.
<lifeless> wgrant: yes, seriously.
<lifeless> wgrant: I can talk on voice if you want, in ~15 minutes.
<wgrant> That's bordering on entirely ridiculous.
<wgrant> It would probably even be worth reducing ourselves to a single maintenance squad for a few weeks to get bug dependencies done early, rather than working on optimising all the hybrid queries and maintaining two UIs and models for years.
<wgrant> There'd be a lot of criticals from that.
<lifeless> not talking yeats
<lifeless> months at most
<wgrant> lolololol
<lifeless> bug deps are 2nd in queue
<lifeless> listings, testing, deps
<wgrant> Yes.
<lifeless> once deps are done, they get migrated, and disclosure can finish off at its leisure
<wgrant> Jump the queue, do deps after listings in parallel with disclosure.
<wgrant> Problem solved.
<lifeless> deps will be parallel with disclosure
<wgrant> But when?
<wgrant> We may be better off suspending the visibility work until it's done, if the alternative is to keep both around in parallel.
<lifeless> tell me something
<lifeless> if you migrate them to have rows in each project
<lifeless> and a flag saying this is ok for projects x y z
<lifeless> does that add significant overhead?
<lifeless> my understanding of the data model is that it doesn't
<lifeless> after all, we were going to do that originally but it was the ownership ramifications that were intractable
<wgrant> What does that mean?
<wgrant> Who has access?
<stub> What is the point of a bug/branch linked to an access policy without a access policy grant? Are there implicit grants?
<lifeless> union of
<wgrant> stub: Most grants are policy-wide.
<wgrant> stub: So they apply to all artifacts within the policy.
<wgrant> lifeless: That means the disclosure management views are a lie.
<lifeless> wgrant: yes, acceptable until all the pieces come together
<lifeless> wgrant: *for those projects*
<lifeless> wgrant: heck, it could oops for them, and it wouldm't matter - what they need involves disclosure + deps; until both are done they aren't able to use it anyhow
<wgrant> lifeless: So, this completely changes the model, because we now need multiple policies per bug. Or we need to entirely preserve the old mechanism, and reoptimise every query to be a hybrid and hope it's actually possible to make it performant.
<stub> Bah. I'm loath to extend bug and branch any more, but given the column is going to be used by pretty much all queries for visibility. Do we get to drop Bug/Branch.private with this?
<lifeless> stub: yes
<wgrant> stub: Yes, that will be dropped with legacy disclosure's destruction.
<wgrant> stub: Bug.security_related too, in all probability.
<wgrant> And Branch.transitively_private.
<lifeless> wgrant: I'm not sure transitively goes
<lifeless> wgrant: unless we don't permit private & ppublic branches in the same project ever
<stub> So I can understand the current model and am fine with it. I think we can make a lot of the indexes partial for a speed win in AccessPolicy & AccessPolicyArtifact
<wgrant> lifeless: We've not quite worked out how stacking works.
<stub> I suspect transitively_private will no longer make sense when the private flag goes. It is just a cache really.
<lifeless> wgrant: yes
<wgrant> As a flag it certainly will no longer make sense.
<lifeless> its a derived value though
<wgrant> It is, but saying that something is "just private" will no longer be possible.
<wgrant> There needs to be a policy.
<wgrant> I don't know what restrictions we'll have around stacking.
<wgrant> stub: Indeed, I should make lots of those partial.
<stub> Will Branch/Bug.access_policy become NOT NULL?
<wgrant> stub: I expect the schema to make privacy queries substantially faster, as it won't have to indirect through a linking table for most things.
<wgrant> stub: No -- NULL means public.
<stub> How does that work for hidden or private products? (do we have them?)
<wgrant> A reasonable point.
<wgrant> Hmm. I hate to think how that would perform.
<wgrant> Because that would require bug searches to look up policies for even public bugs...
<stub> Probably about the same as LEFT OUTER JOIN with the AccessPolicy or a UNION like we do with private stuff atm.
<stub> Anyway, it might end up nicer and/or faster if it becomes NOT NULL and public bugs are linked to a public AccessPolicy
<lifeless> we already check project.active
<wgrant> Hmm, so we do.
<wgrant> So maybe that would work.
<lifeless> if we want to denorm inactivity, we can do that by (for instance) creating a non-public policy for them
<lifeless> stub: a hidden or private product will have no public bugs
<lifeless> stub: by definition
<lifeless> stub: we don't [really] have them yet, but we will. Also private distros.
<stub> lifeless: Well, if a product could override what 'public' means you could have hidden bugtasks on a bug. But I'm not sure people want to go down that path.
<stub> But even without, the queries just might come out nicer so worth keeping in mind when hacking the code.
<stub> wgrant: Do you want to make the partial index changes and push?
<wgrant> stub: Do I also want to make AccessPolicyArtifact's indices partial?
<lifeless> wgrant: ok, do you want a voice chat about oem?
<stub> wgrant: Yes, I think so
<wgrant> lifeless: Sure.
<wgrant> I'm Skyped up.
<lifeless> try calling me
<lifeless> I think skype is running, can't tell(unity)
<lifeless> wgrant: ^
<wgrant> Calling
 * StevenK attempts to fix the qa-tagger
<StevenK> Two rollbacks, which I think overlap? Seriously? :-(
<wgrant> StevenK: That works normally.
<wgrant> What's the error?
<StevenK> It isn't crashing, but r14216 is marked as needstesting, I set it back to qa-bad bad-commit-14216
<StevenK> Revision 14216 can be deployed if 14223 is deployable: rolledback
<StevenK> There we go
 * StevenK fixes r14219 too
<wgrant> (for the record, my earlier objections to lifeless' crazy plan are rescinded now that he's clarified he doesn't actually suggest that we keep legacy disclosure around, just that we temporarily weaken new disclosure for some projects)
<StevenK> wgrant: Bleh.
<StevenK> wgrant: Now both bad commits are so marked, but r14229 is linked to the same bug.
<wgrant> StevenK: Mark that bug qa-ok or qa-untestable as appropriate.
<wgrant> The bad-commit-XXX will still taint the bad rev.
<StevenK> It was marked as needstesting 10 hours ago, I guess abentley has not done QA on it yet.
<wgrant> Leave it needstesting, then.
<StevenK> And I think deryck has done the same thing.
 * StevenK digs.
 * StevenK gives up on deploying before Monday.
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: lifeless, rvba | Critical bugtasks: 266
<StevenK> Right, two revisions need QA to get us to a deployable state.
<StevenK> rvba with r14225, and abentley with r14229
<StevenK> Well, deployable until r14231 at least.
<rvba> StevenK: r14225 is actually qa-ok (it's behind a ff anyway) but I forgot to mark it incr when I landed itâ¦
<rvba> so I marked it qa-ok first, and the I removed the qa tag altogether maybe I got the qa tagger confused, sorry about that.
<rvba> s/and the/and then/
<rvba> StevenK: should I re-add the qa-ok tag?
<StevenK> rvba: If you remove the tag altogether the qa tagger will assume it's not ok -- please add the tag if it is.
<rvba> StevenK: done.
<adeuring> good morning
<poolie> hi
<lifeless> allenap: so what was the rabbit/txlongpoll qa issue?
<lifeless> stub: fdt - how would you feel about us removing the long xact check
<lifeless> stub: (for readonly conns)
<lifeless> stub: or can we not tell..
<stub> lifeless: A readonly connection will still block the updates
<lifeless> stub: yes, thus killing it
<bigjools> lifeless: re. txlongpoll - it just started worked so we figured it was something you did to Rabbit
<bigjools> working*
<lifeless> ok great
<bigjools> need to get it in production now
<lifeless> i didn't see an update to the ticket about prod rabbit
<bigjools> but we need a deployment strategy
<lifeless> which IIRC was blocked pending txlongpoll on staging being happyt
<lifeless> bigjools: related question
<stub> lifeless: I mean if we let them stay, they will block the updates. ALTER TABLE and most DDL needs an ACCESS EXCLUSIVE lock, which conflicts with the ACCESS SHARE locks taken out by SELECT statements.
<poolie> bigjools, lifeless: so if we split out the buildds, i'm presuming we want the buildmaster tests to still be able to start it
<lifeless> stub: yes, I don't mean let them stay
<lifeless> stub: I mean kill them rather than aborting
<stub> oic.
<bigjools> poolie: sorry, I think I am missing context
<poolie> sorry
<poolie> i'm looking at https://bugs.launchpad.net/launchpad/+bug/800295
<lifeless> bigjools: does txlongpoll have amqp oopses configured ?
<poolie> but specifically, robert suggested trying to split out the buildd source from launchpad proper
<stub> We could, but it is a workaround the real problem that something screwy is happening that needs to be investigated.
<bigjools> lifeless: I believe so
<lifeless> bigjools: great
<lifeless> bigjools: I'd like to confirm that; how do I go about it ?
<bigjools> poolie: so yes I think some of the b-m tests use the buildd
<stub> lifeless: The theory is that nothing except a few well known system connections should have long running transactions open, and if this is not the case the system might be unstable and we should not proceed without investigation.
<bigjools> poolie: we just make it a source dependency in versions.cfg etc.
<lifeless> stub: so far it seems to be generating false positives
<bigjools> poolie: the buildd code should be a GPL2 project on its own
<lifeless> stub: things which yes we should fix but they are totally safe to interrupt
<bigjools> rvba: did txlongpoll get amqp oopses?
<poolie> right, and it should probably expose a python module that contains a test fixture etc?
<bigjools> poolie: right
<bigjools> poolie: in fact I suspect we need 2 projects, the buildd and a buildd-fixture
<stub> lifeless: Right. So we can either ignore everything and not do the check at all (possibly causing things to explode because something is screwing up) or keep things as they are
<bigjools> LP just depends on the latter
<poolie> ?
<poolie> won't it need the actual buildd, to run it?
<lifeless> stub: we put out a broad call for things that deal with nontransactional data
<lifeless> stub: and got back the whitelist
<bigjools> yes - dependency chain
<bigjools> it's what we did with txlongpollfixture
<stub> lifeless: yes. I'm more thinking of things like postgresql processes getting hung in odd states or thrashing trying to calculate multibillion row result sets and that sort of thing.
<poolie> oh i see, it only directly depends on the fixture, right
<poolie> though, if we just bring it in as an egg(?) or branch, it doesn't really need to be separately packaged
<bigjools> the thought of being able to type "make run_buildd" is appealing :)
<lifeless> stub: so we try a cancel backend; wait 5 seconds; check
<lifeless> stub: of the slow things first
<lifeless> ?
<stub> We also have the option of increasing the timeout.
<bigjools> poolie: if we want to pull it in on buildout it should be an egg really
<lifeless> stub: which timeout ?
<stub> lifeless: We could, yes. That starts causing outages before the real outage.
<stub> lifeless: The definition of a long running transaction
<lifeless> stub: of things with -long- transactions
<bigjools> we don't need to package it I guess
<lifeless> stub: and long is already defined longer than any webapp, right ?
<bigjools> but it is already packaged
<stub> lifeless: Yes. So it shouldn't affect anything interactive that isn't already messed up.
<poolie> what would be the steps, roughly, to 'make run_buildd'?
<lifeless> stub: which means we're not really causing an outage ;P
<stub> lifeless: Yes, we can define outage in such a way as to not have outages :-)
<lifeless> stub: \o/ at shifting goalposts
<stub> lifeless: The important things are backup processes, slon processes etc. and they are protected as we define this stuff as system users.
<lifeless> yes
<lifeless> stub: so my thrust is to reduce/eliminate failed fdt's where we miss the window and don't do the deploy
<stub> Increasing the long running transaction definition is the simplest thing that might fix the immediate problem, and I think it is even a command line option.
<lifeless> balls in your court :)
<stub> For killing long running transactions, we should be able to do the connections to the slaves. On the master, it would be all or nothing as I can't tell a read only connection from a read/write.
<lifeless> we had  in a row that failed IIRC
<lifeless> stub: can we not check held locks?
<stub> That might work, yes.
<stub> There are race conditions, but we already have that with connections that don't meet the 'long running' criteria but might cause problems when we take down pgbouncer.
<bigjools> poolie: I don't know precisely but we have done a couple of fixtures that start up with "make run" now, see rabbitfixture and txlongpollfixture
<bigjools> poolie: allenap did the work if you need details
<bigjools> poolie: although I suspect the buildd package will need some tweaking as it's currently designed to run in a chroot in its own account
<poolie> right, and some of it insists on being under xen
<poolie> do you test it locally at all?
<bigjools> poolie: not as such, I am talking about running it locally
<bigjools> yes - so you make a new "buildd" account and then chroot inside there
<lifeless> stub: there should be one lock on the virtual xaction and thats it, for a readonly op IIRC
 * bigjools OTP for 5m
<stub> lifeless: Right, I haven't looked closely yet but anything that doesn't have any sort of exclusive locks open would count I think.
<stub> lifeless: So I guess we can just ignore any read only transactions for the long running check and keep everything but that query the same.
<stub> masters and slaves.
<poolie> bigjools, could you read https://code.launchpad.net/~mbp/launchpad/800295-buildd-split/+merge/81111
<bigjools> poolie: OTP, will check shortly
<poolie> thanks
<bigjools> poolie: where is _hasDaemonStarted called from?
<poolie> bigjools, from the superclass TacTestSetup
<bigjools> poolie: that's code that's still in the main tree
<bigjools> poolie: I think we need to duplicate TacTestSetup in buildd
<poolie> oh probably
<poolie> if we want the split out buildd to be independently testable
<bigjools> indeed
<poolie> ... with tests that cover starting it this way
<bigjools> LP can depend on buildd but not the other way around
<poolie> and we probably do
<bigjools> thanks for looking at this though!
<poolie> i have ec2 seeing if it passes with these changes now
<jelmer> rvba: hi, do you have time for a review?
<rvba> jelmer: sure.
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba | Critical bugtasks: 266
<jelmer> rvba: great - the MP is at https://code.launchpad.net/~jelmer/launchpad/stacked-code-imports/+merge/81139
<deryck> Morning, everyone.
<jelmer> hey Deryck
<rvba> jelmer: okay, I confess I'm not familiar with this part of the code but it will be a good occasion to learn.  Besides this branch has sort of already been approved right?
<jelmer> rvba: yeah, benji reviewed it earlier
<rvba> jelmer: Okay, I'll have another look then.
<flacoste> bigjools: can you join the session going on in #ubuntu-uds-bonaire5 ?
<flacoste> bigjools: we are discussing gating uploads when the distro is in prerelease
<Ursinha> I wish I was there *sigh*
<jml> I'm creating a Launchpad project now
<jml> and setting up team structures is a bit dull
<jml> and easy to mess up
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba, jcsackett | Critical bugtasks: 266
<jml> Could I please get a team renamed? https://answers.launchpad.net/launchpad/+question/177395
<Beret> jml, I'm pretty sure that costs $19.95 plus S/H
<jml> Beret: Canonical can dock it from my salary then.
<Beret> hah
<rvba> Morning jcsackett!
<jcsackett> morning, rvba. :-)
<rvba> I'm afraid it's gonna be a crazy reviewing day.
<rvba> I'm on Jelmer's branch, then I'll review Ian's 1249 branch.
<rvba> After that I plan to kill myself ;).
<rvba> 1249 lines branch that is.
<jcsackett> rvba: 1249? that's not a typo? wow.
<jcsackett> rvba: i'm looking at your branch now.
<rvba> jcsackett: Thank you.
<jml> Is https://bugs.launchpad.net/launchpad/+bug/3945 different from https://bugs.launchpad.net/launchpad/+bug/57418, really?
<_mup_> Bug #3945: Support debtags in Launchpad for products and packages <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/3945 >
<_mup_> Bug #57418: Support debtags in Packages.gz <escalated> <feature> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/57418 >
<deryck> abentley, my css-polish branch is up to date now and pushed.
<abentley> deryck: thanks!
<deryck> np!
<jelmer> jml: it seems they're pretty similar. debtags in their original implementation were not in the Packages file, so it would be possible to do 3945 without 57418
<jml> jelmer: ah, ok.
<abentley> rvba, what do you mean by "unify the formatting here"?
<rvba> abentley: Well, it looks like sometimes the inner part is indented, sometimes not.
<abentley> rvba: okay.
<deryck> rvba or jcsackett, I have a CSS/html branch for review.  takers? :)
<abentley> rvba: better? http://pastebin.ubuntu.com/727255/
<rvba> deryck: We will add it to the queue :)
<rvba> abentley: I think so, don't you agree?
<deryck> rvba, ok, cool.  A busy day then, I take it. :)
<abentley> rvba: I waffle about indenting.
<jcsackett> deryck: a bit. i should be able to look at yours in a bit.
<rvba> deryck: busy day indeed.
<rvba> jcsackett: just for the fun of it, I invite you to take a look at the MP: https://code.launchpad.net/~wallyworld/launchpad/delete-bugtask-ui-ajax-878909/+merge/80779
<jelmer> rvba: thanks!
<rvba> np
<jcsackett> i will just say i'm glad you're taking that one, rvba. :-P
<rvba> hehe
<jcsackett> rvba: r=me on your branch.
<jcsackett> deryck, i'll look at yours now.
<deryck> jcsackett, ok thanks!
<rvba> jcsackett: thanks.  The nice part is that landing this branch will conflict with Ian's branch.
<rvba> I have my revenge!
 * jcsackett laughs.
<jcsackett> that's the problem with enormous branches -- the odds of someone stepping on your toes goes *way* up.
<rvba> True
<jcsackett> deryck: to make sure i understand, your css changes only effect the mustache template, right?
<deryck> jcsackett, correct
<jcsackett> awesome. r=me, then, deryck.
<deryck> jcsackett, rocking thanks!
<allenap> rvba, jcsackett: Please can you review my shortish branch? https://code.launchpad.net/~allenap/launchpad/participation-keyerror-bug-810114-2/+merge/81151
<jcsackett> allenap: sure, i'll jump that over the longer branch of jtv's i just grabbed.
<allenap> jcsackett: Hehe, thanks.
<deryck> abentley, my css-polish branch is approved and playing in ec2 now.
<abentley> deryck: excellent.
<jcsackett> allenap: i haven't seen the "slurp" function before, what's the deal?
<jcsackett> ... or is this defined elsewhere in the branch ...
<allenap> jcsackett: The latter :)
<abentley> deryck: I expect to have this branch up for review today.
<deryck> abentley, awesome.
 * deryck goes offline to switch work locations, back shortly
<jcsackett> allenap: r=me.
<allenap> jcsackett: Thanks!
<abentley> rvba or jcsackett: could you please review https://code.launchpad.net/~abentley/launchpad/client-bug-fields/+merge/81157 ?
<rvba> abentley: I'll take it.
<abentley> rvba: thanks.
<abentley> deryck: custom-bug-listings-css-polish fails lp.bugs.browser.tests.test_bugtask.TestBugTaskSearchListingView.test_mustache_rendering
<deryck> abentley, ah, dang.  I didn't run tests.  But ec2 will catch it.
<deryck> abentley, thanks for the heads up.  I'll get a fix shortly.
<abentley> deryck: np.
<abentley> deryck: let me know so I can merge the fix.
<deryck> abentley, will do
<adeuring> rvba, jcsackett: could you please  review this MP: https://code.launchpad.net/~adeuring/launchpad/banner-for-beta-features/+merge/81165
<jcsackett> adeuring: i'm doing one of jtv's right now, i can do yours right after.
<adeuring> jcsackett: trhanks!
<rvba> jcsackett: I told you today would be crazy ;)
<jcsackett> rvba: you weren't kidding. :-P
<jcsackett> adeuring: okay, i'm looking at yours now.
<adeuring> jcsackett: cool
<deryck> rvba or jcsackett, I have another branch for review if either of you is available.
<rvba> deryck: I'll take it once I'm finished with the one I'm working on right now.
<deryck> rvba, thanks!  It's here:  https://code.launchpad.net/~deryck/launchpad/orderby-bar-settings-slot/+merge/81148
<rvba> k
<deryck> abentley, test is fixed now and pushed up to lp in that branch.
<abentley> deryck: thanks.
<deryck> np
<rvba> abentley: I've got a question about this bit of code: http://paste.ubuntu.com/727380/
<abentley> rvba: sure.
<rvba> in the each loop, what kind of value have the variables value and key?
<abentley> rvba: value is a boolean, key is a string.
<rvba> abentley: field_visibility is something like {show_item: true} right?
<abentley> rvba: right.
<bigjools> any idea why I'd get a "Signature couldn't be verified: (7, 8, u'Bad signature')" when reviewing an MP over email?
<jelmer> bigjools: are you using thunderbird without MIME?
<bigjools> jelmer: no
<rvba> abentley: then I don't get why you do: delete batch.mustache_model.bugtasks[i][key];
<bigjools> the signature verifies locally (I am using kmail)
<rvba> abentley: could you please explain that a little bit?
<jelmer> bigjools: I'm finding that if I don't use MIME for GPG signatures in Thunderbird I get that error
<bigjools> jelmer: I have no idea if I can even config that
<abentley> rvba: that is to prevent "batch.mustache_model.bugtasks[i][key]" from shadowing "batch.mustache_model[key]"
<bigjools> jelmer: ah I found something - so "S/MIME" ought to work?
<abentley> rvba: mustache templates are supposed to have nested scopes, so that if key isn't found in current scope, it's looked up in the outer scopes.
<rvba> abentley: ah! that's what I was missing!
<rvba> Thanks!
<abentley> rvba: np.
<bigjools> jelmer: I've not used MIME before and it's worked, plus kmail bitches that it doesn't have the other side's keys if I try and force MIME :/
<jelmer> bigjools: that works for me, though it's obviously also a bug in launchpad if it only works with S/MIME (in some situations?)
<rvba> abentley: r=me
<abentley> rvba: thanks.
<adeuring> jcsackett: thanks for your review!
<rvba> deryck: I'm starting with your branch.
<jcsackett> adeuring: you're welcome, sorry i forgot to tell you i was done. :-P
<adeuring> np ;)
<deryck> rvba, thanks!
<rvba> deryck: r=me
<deryck> rvba, thanks!
<rvba> np
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugtasks: 266
<lifeless> moin
<bigjools> good night!
<abentley> deryck: looking at bug age, the examples seem to show a time formatting we haven't done before.  Or am I missing something?
<deryck> abentley, yeah, I thought that was new as well, but never asked about it.
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<lifeless> mrevell: I'm ready whenever
<mrevell> hey lifeless!
<timrc> cody-somerville,  #ubuntu-uds-Bonaire4
<lifeless> wow oops pruning is ... just wow
<poolie> lifeless, so my split-out-buildd thing has passed tests up to a certain level of completion
<poolie> the buildd tests still depend on the lp tree but there is no actually copied-around code
<timrc> I noticed that from a logged out perspective my ~canonical membership badge is not shown on my profile page.  Is this by design? Or, is it a bug? Or, am I missing how to show it?
<lifeless> timrc: canonical is a private team
<timrc> lifeless, so can someone logged in but not a member of Canonical see it?
<lifeless> no
<timrc> lifeless, okay, that's the information I needed
<wgrant> lifeless: Yeah, oops-prune is pretty special.
<wgrant> lifeless: I don't imagine wildcherry is very happy with it.
<StevenK> wgrant: I need to write an API script to copy from Ubuntu? I forget. :-(
<lifeless> so, I'm considering some options here
<lifeless> I need opinions :)
<lifeless> broadly, I'm thinking:
<lifeless>  - delta based - get changes across the tables for a date range
<wgrant> StevenK: You can copy from anywhere. Even create a new PPA and copy from an existing PPA into that.
<lifeless>  - use [for now] a local disk file to record last run time
<lifeless>  - add an API on project [product] that takes two dates - start and stop - and returns a set of oops ids
<lifeless> of course, this is probably impossible in the API
<lifeless> StevenK: wgrant: either of you familiar with api gotchas / benefits ?
<lifeless> if I just return a set([id, id2, ...]) from a method, will it work
<wgrant> That should work, yes.
<lifeless> or do I need an Interface OopsResult with an attribute oops_ids ?
<wgrant> But doing this through the lazr.restful API would indicate that you have a deathwish.
<lifeless> well
<lifeless> I'd like to let other folk running oops systems work with LP to keep interesting ones and prune the rest.
<lifeless> So there is very good reason to make this a public API
<wgrant> Yes, there is very good reason to do it.
<wgrant> But there's also very good reason to not.
<lifeless> ...
<wgrant> This sounds like it's going to be a fairly expensive method, and return a lot of data.
<lifeless> no
<wgrant> Neither of which the API is likely to be good at.
<wgrant> Ah, if it's partial, I guess.
<lifeless> there are very few oopses referenced in the system for any given short date range
<lifeless> the reason it takes 3 minutes to run is because we're searching everything
<lifeless> searching 1 day - one /(365*5) of the db - should be subsecond trivially.
<wgrant> Indeed.
<wgrant> So, yes, you can just return a sequence from a method, and it will work.
<lifeless> and if not, there are other similar uses so making it subsecond is desirable anyhow.
<wgrant> You can't declare the return type, so lazr.restfulclient won't do any special decoding.
<wgrant> But since it's just strings it should be fine.
<lifeless> the client script then needs to remember where it has pruned up to - thats the lower bound, and select a day more than (gc window) back, thats the upper bound.
<lifeless> so, for instance, if I default to 7 days, the first run would take the oldest datedir in the local datedir repo, and 7 days ago, and query.
<lifeless> which would suck, if we ran it on the old stuff on carob
<lifeless> but new stuff will be a few weeks.
<lifeless> max.
<poolie> lifeless, wgrant, what do you think of me landing https://code.launchpad.net/~mbp/launchpad/800295-buildd-split/+merge/81111 as an incremental step towards a buildd split out
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugtasks: 266
#launchpad-dev 2011-11-04
<StevenK> Ah ha. lifeless's comments are a little misleading.
<StevenK> BugNominationView._getListingItem isn't it at all, it's BugListingBatchNavigator._getListingItem
<poolie> StevenK, hey
<poolie> in pursuit of splitting out the buildd tree
<poolie> i'm thinking of moving the tests that rely on zopelessdblayer etc out of the buildd tree into lp
<poolie> so they will rely on a buildd but not be part of it
<StevenK> poolie: And making lp-buildd be pulled in via sourcecode or the download-cache?
<poolie> yep
<poolie> https://bugs.launchpad.net/launchpad/+bug/800295 fwiw
<_mup_> Bug #800295: buildd is unnecessarily coupled  to the launchpad tree through tac file and readyservice <tech-debt> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/800295 >
<poolie> welcome back, bot
<StevenK> Actually, it's not. It's NominatedBugListingBatchNavigator.
<wgrant> StevenK: Do you know if overlay distros work yet?
<wgrant> I think they probably function.
<wgrant> Enough for someone to kill archivepurpose 4 with a few hours of work.
<StevenK> Ha. Maybe they do.
<StevenK> wgrant: Should I put up a NDT for r14231?
<wgrant> Since I've got Julian convinced that we should kill archivepurpose 7...
<wgrant> StevenK: Was hoping you'd ask :)
<wgrant> And 4 is pretty clearly actually an overlay distro.
<StevenK> What's 7 again?
<wgrant> 7 is DEBUG
<wgrant> I successfully convinced him that it's a terrible idea.
<StevenK> And 4 is PARTNER
<wgrant> Yes.
<StevenK> They both need to die.
<wgrant> Yes.
<wgrant> And I think 4 can die pretty cheaply by being turned into https://launchpad.net/canonical-partner
<wgrant> And that lets us remove a tonnnnnne of awfulness.
<wgrant> all_distro_archives can fuck off.
<StevenK> Needs IS involvment to get it onto archive.c.c.
<StevenK> But certainly doable
<wgrant> We can remove all the Distribution and DistroSeries publication-related methods.
<wgrant> IT WILL BE BEAUTIFUL
<wgrant> Yes, it requires some IS involvement.
<wgrant> But I think it's very doable.
<wgrant> Although I bet it's signed with the Ubuntu archive key.
<wgrant> Just to make things difficult.
<StevenK> Oh, rofl
 * StevenK starts putting together a NDT
<wgrant> StevenK: Have you heard about my new solution to kill off 7?
<wgrant> I'm not sure if I've run it past you.
<wgrant> Need to run it past Colin and Adam and the like too, but I think it's sensible.
<StevenK> wgrant: Only in vague terms, I think.
<wgrant> StevenK: We publish them in separate $COMPONENT/debug indices, like $COMPONENT/debian-installer, and handle the split like we do with security and ports.
<wgrant> Eventually when Soyuz gets archive views, we can move the split to there... but until then we might as well utilise the existing mechanism.
<StevenK> Doesn't that make archive.u.c explode in size?
<wgrant> Remember that there's already the secret magic splitter that throws ports binaries and indices elsewhere.
<wgrant> And keeps security on security.u.c.
<wgrant> We repurpose that to create a fourth archive -- ddebs.ubuntu.com
<wgrant> Because this is really the same thing as the ports split.
<StevenK> Oh, right.
<wgrant> We need to keep some binaries off the mirrors.
<wgrant> It's divided by component instead of arch, but it's really the same thing.
<StevenK> With the appropriate magic-mirror hacking, that sounds like a good plan.
<StevenK> wgrant: And ddebs.u.c already exists on macquarie
<wgrant> Yes. The retracers will probably need to look at both for a while.
<wgrant> For values of "a while" that probably number in the several years.
<wgrant> But details, details.
<wgrant> apt and apt-ftparchive seem happy with custom slashed components.
<wgrant> Yay
<wgrant> It is practical.
<wgrant> And the Soyuz side is dead-easy.
<wgrant> So, we now have feasible solutions to both partner and debug that probably take less than two days each.
<wgrant> One is already escalated...
<StevenK> wgrant: Isn't there a better method to call to get upload perms for a user than distribution.main_archive.verifyUpload(person, name, component, distroseries) ?
<wgrant> Don't think so.
<wgrant> jml cleaned it up a couple of years back.
<wgrant> It would be nice if it was set-based, but it's not.
<StevenK> Since I think the lookup of the component is grabbing SPPH, and then verifyUpload does the same thing.
<wgrant> It's grabbing it to check the component?
<wgrant> It could be cleaned up further, but it's a deep hole.
<StevenK>                 component = self.distroseries.getSourcePackage(
<StevenK>                     name).latest_published_component
<StevenK>                 if distribution.main_archive.verifyUpload(
<StevenK>                     person, name, component, self.distroseries) is None:
<StevenK>                     return True
<wgrant> One I plan to follow eventually, but hopefully after abolishing archivepurpose (4, 7).
<wgrant> StevenK: You could possibly check how branches do it.;
<wgrant> launchpad.Edit for IBranch needs to do the same thing as IBugNomination.canApprove.
<wallyworld_> poolie: hi. i claimed your 678090-affected-count mp and then read the comments. are you still working on it?
<StevenK> wgrant: Digging around IBranch isn't turning up anything
<poolie> wallyworld_, not right at this moment but i will
<wgrant>     def checkAuthenticated(self, user):
<wgrant>         can_edit = (
<wgrant>             user.inTeam(self.obj.owner) or
<wgrant>             user_has_special_branch_access(user.person) or
<wgrant>             can_upload_linked_package(user, self.obj))
<wgrant>     for ssp in ssp_list:
<wgrant>         archive = ssp.sourcepackage.get_default_archive()
<wgrant>         if archive.canUploadSuiteSourcePackage(person_role.person, ssp):
<wgrant>             return True
<wallyworld_> poolie: ok. just wanted to see if i needed to review now or not. thanks
<poolie> do you have any comments beyond what robert and i said?
<wgrant> Which uses latest_published_component.
<wallyworld_> poolie: not sure. didn't get that far. i'll have a look
<StevenK> wgrant: Ah, where is that?
<wgrant> StevenK: c.l.security
<wallyworld_> poolie: so you can delete the view property 'other_users_affected_count' i think
<poolie> yep probably
<poolie> is my change to @cachedproperty ok?
<wallyworld_> poolie: other than that, looks like a nice change. i was going to also suggest the feature flag
<wallyworld_> poolie: i like it how you've been doing these polish 'paper cut' bugs of late
<poolie> thanks
<wallyworld_> poolie: i think the cachedproperty change is ok
<wallyworld_> i've run into issues before with cached permissions but i think this case is ok
<poolie> it's just cached for the duration of the page view, right?
<wallyworld_> yep. the rendering
<wallyworld_> poolie:  so, a cached property is good if a property is accessed more than once during a render. is this property used more than once?
<poolie> yes, the template puts it in twice
<poolie> for something to do with js fallbacks
<wallyworld_> ok
<poolie> which seems ugly but i didn't want to change it without fully understanding it
<poolie> and just caching it seemed fine
<wallyworld_> sure
<poolie> lifeless, do you have any opinion where TacTestSetup ought to live, so that it can be used by both lp and buildd tests?
<poolie> it is not enormous but not trivial
<wgrant> In Twisted :)
<poolie> it would seem like a waste to make a new package just for that
<poolie> :)
<poolie> perhaps it could go in testfixtures
<lifeless> poolie: you could create a txfixtures
<lifeless> avoiding packages just makes problems
<poolie> as a new package, depended upon by both?
<lifeless> yes
<lifeless> txfixtures with twisted specific Fixtures
<lifeless> it can depend on fixtures
<lifeless> lp and buildd can both use txfixtures
<poolie> k
<poolie> aside from that i think it's all ready to be cut out
<poolie> nb it doesn't actually depend on twisted
<poolie> so, in a sense it could go in plain fixtures
<poolie> well, you probably cannot run its owntests without twisted
<poolie> wallyworld_, while you're here can you read https://code.launchpad.net/~mbp/launchpad/800295-buildd-split/+merge/81111
 * wallyworld_ reads
<wallyworld_> poolie: looks ok to me, but i think the port to listen on needs to be paramaterised
<wgrant> lifeless: Do you know exactly what OEM needs? Just bugs with foo and hwe-foo tasks?
<wgrant> lifeless: I wonder if we can just have a temporary owning context.
<wgrant> lifeless: There's no real need to respect multiple policies.
<lifeless> I don't know precisely know; nor do I understand what you mean :)
<wgrant> lifeless: What if we say that OEM's multi-pillar bugs favour one of the targets, and use the access policy from that target?
<wgrant> Rather than respecting the union of policies from each target.
<lifeless> I have no objection to that
<lifeless> seems harder to do well
<lifeless> but thats your problem :)O
<wgrant> lifeless: Also, I forget some details of yesterday's conversation. IIRC you suggested just having the enum value on Bug/BugTask, without the pillar reference. But then we have to indirect through Product/Distribution/ProductSeries/DistroSeries to work out the policy.
<poolie> some of these osutils almost need their own package
<lifeless> for full observers
<lifeless> not for restricted
<wgrant> Sure.
<lifeless> which is a small set
<wgrant> Sure, but it makes queries far worse.
<lifeless> you've tested ?
<wgrant> I mean in terms of complexity.
<wgrant> Speed, I haven't tested yet.
<lifeless> the are a little more awkward to write, thats true.
<lifeless> might be a driver for having two separate tables
<wgrant> How?
 * lifeless speculates wildly
<wgrant> The problem is not that there's one table.
<lifeless> wgrant: its a lopsided table
<wgrant> THe problem is that we have to indirect through four other tables to get to the one table.
<lifeless> wgrant: task -> policyuse -> grants
<lifeless> gnuoy: one in between table
<lifeless> bah
<lifeless> wgrant: ^
<wgrant> lifeless: How do you get from task to policyuse unless you refer to policyuse on task?
<wgrant> You'd have to go through ProductSeries or DistroSeries.
<lifeless> task.product == policyuse.product
<lifeless> mmm, sure. ok - 2 tables
<lifeless> task -> [*series] -> policyuse
<wgrant> And we already have to denorm this slightly, on grant.
<wgrant> I'm not sure that one is avoidable.
<lifeless> grant knowing the asset ?
<wgrant> Right, an artifact-specific grant has to know the artifact, its target, and its policy.
<lifeless> target == user ?
<lifeless> 'person' ?
<wgrant> pillar
<lifeless> why does it need pillar? it can use policyuse.id
<lifeless> (person, artifact, policyuse)
<wgrant> Right, but policyuse requires target and policy.
<lifeless> yes
<wgrant> Which means we're denorming (target, policy) from bugtask onto grant.
<wgrant> It's indirect, but it's still denormed.
<lifeless> mmm
<lifeless> so we'd do that for reporting
<lifeless> ack, its a denorm
<lifeless> thats part of your schema anyhow though, right ?
<wgrant> Yes.
<lifeless> so no-change
<wgrant> The change is that you suggest I no longer denorm that on bugtask itself.
<wgrant> Which seems odd, when we're already denorming it elsewhere.
<wgrant> Further away.
<lifeless> well, denorms are done for a reason
<lifeless> whats the reason for it to be denormalised on bugtask ?
<lifeless> or rather
<lifeless> let me ask it differently
<lifeless> if bug is the native location for this
<lifeless> whats the best denorm to use on bugtask
<lifeless> so bug.policy = ENUM
<wgrant> Right, better question.
<lifeless> bugtask.whatshouldwehavehere.
<lifeless> brb, nature
<wgrant> I suspect policyuse. So setting Bug.access_policy calculates policyuse and throws it at BugTask and artifact.
<wgrant> Although then we can't optimise public bugs.
<wgrant> Because we'd have to look up the public policy. But that may not be a problem.
<poolie> ok, lp:txfixtures exists!
<wgrant> lifeless: We seem to have an OOPS issue.
<wgrant>  * 308 Exceptions
<wgrant> All the None/None OOPSes are missing.
 * wgrant checks devpad.
<wgrant> django.db.utils.DatabaseError: invalid byte sequence for encoding "UTF8": 0xf8
<wgrant> HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
<wgrant> yay
<wgrant>  raised with datedir: /srv/launchpad.net-logs/production/chaenomeles/2011-11-03 filename: OOPS-4e0b81ac6e642ee8099df77672c8be1a
<wgrant> fwiw
<wgrant> ['HTTP_USER_AGENT', '\xf8\x07p\x01']
<wgrant> That's a new one...
<poolie> M-x ^g p ^a
<poolie> help!
<poolie> wallyworld_, so what did you think of https://code.launchpad.net/~mbp/launchpad/800295-buildd-split/+merge/81111
<poolie> lifeless, i'm just going to copy some of the osutils functions in to txfixtures to stop this recursing too much
<wallyworld_> [11:17:10] <wallyworld_> poolie: looks ok to me, but i think the port to listen on needs to be paramaterised
<poolie> some kind of global osutils
<poolie> wallyworld_, where?
<lifeless> wgrant: win
<lifeless> wgrant: rye has a branch up that should help with this
<wgrant> lifeless: .decode('utf-8', 'replace')?
<lifeless> wgrant: not proposed for merging yet
<wallyworld_> poolie: _hasDaemonStarted().... return self._isPortListening('localhost', 8221)
<lifeless> well, the db schema is royally messed up. headers are bytes.
<wgrant> Not sure whether to hack code or OOPS now..
<poolie> wgrant, that particular one looks a lot like unix keyboard input not broken utf8
<wgrant> Yes.
<lifeless> wgrant: https://bugs.launchpad.net/python-oops-tools/+bug/884265
<_mup_> Bug #884265: req_vars header values may cause crash in dboopsloader <python-oops-tools:Triaged> < https://launchpad.net/bugs/884265 >
<poolie> of course in general escaping might be good
<wallyworld_> poolie: but i'm not very familiar with that stuff so i could be wrong
<poolie> wallyworld_, it's already hardcoded in the tac file...
<poolie> the coupling is not ideal but since we're apparently happy just counting on that port for these tests i can live with it
<lifeless> I think someone is going around attempting an exploit
<wallyworld_> poolie: so, my ignorance - i thought hasDaemonStarted() is a generic method used for different services
<wallyworld_> hence each would be on a different port
<lifeless> wgrant: https://code.launchpad.net/~rye/python-oops-tools/quote-req-vars
<poolie> i guess i could make a tac file in a tmpfile
<wgrant> lifeless: Intriguing.
<poolie> wallyworld_, the version that hardcodes the value is specific to tests that always use buildd-slave-test.conf which is always 8221
<wgrant> But I guess that works.
<wgrant> For now.
<wallyworld_> poolie: ok. thanks for clarifying
 * wgrant cowboys.
<poolie>  np
<poolie> thanks for the speedy review
<poolie> all good then?
<lifeless> wgrant: where did you see the error ?
<wgrant> lifeless: Ran make update_db manually.
<lifeless> poolie: lazr.utils might be a place you can put osutils things
<wgrant>         if isinstance(value, str):
<wgrant> Needed to add that.
<wgrant> Since we have floats.
<wgrant> Perhaps U1 does not.
<poolie> ok lunch, biab
<lifeless> poolie: though I have not checked the dependencies it would drag in
<lifeless> wgrant: well, they don't have native types at all yet (rfc822 still)
<poolie> i don't have a good bias towards lazr.* but perhaps it's only restfulclient
<lifeless> wgrant: I'm working on the IBodyProducer thing they need to migrate
<wgrant> lifeless: Ah, of course.
<wallyworld_> poolie: looks ok
<wgrant> lifeless: Thanks.
<wgrant> lifeless: Any thoughts on bugtask?
<lifeless> permit bug.accesspolicy to be NULL
<lifeless> I agree; trigger updates both denormed locations (grant + tasks)
<wgrant> Great.
<lifeless> wgrant: raises the question, perhaps the artifact table should denorm the policyuse, rather than the grants.
<lifeless> (Or does it already?
<lifeless> )
<wgrant> It doesn't already. It could. But currently the only method to manipulate artifacts is ensure(), which is nice.
<lifeless> so bug.accesspolicy being null implies task.policyuse is nullable, so public case is optimisable
<wgrant> Yep.
<wgrant> I'm not sure what to do with artifact-specific permissions when the artifact becomes public.
<lifeless> now, when the policy changes, there are less artifact rows to change than grant rows
<wgrant> Indeed.
<wgrant> And querying through artifact for reporting should be quickish... hopefully.
<lifeless> which, if you're hitting artifact anyway on reports
<lifeless> -> I would put the policyuse on artifact for restricted observers
<lifeless> -> and on grant for full observers.
<wgrant> Right.
<lifeless> yes, that breaks your combined schema. I'm feeling more and more sure it should be separate.
<wgrant> Possibly, yes.
<wgrant> It shouldn't make a significant performance difference.
<wgrant> To have to go through artifact.
<lifeless> Well, I think you are anyway, aren't you ?
<lifeless> or were you thinking to just report 'there are N total grants'
<wgrant> Pretty much that.
<lifeless> being able to report 'there are N private objects' and 'Y security ones' might be useful itself.
<wgrant> In order to be fast.
<wgrant> Yes.
<wgrant> True.
<lifeless> so, O(artifacts) < O(restricted grants) if there are 1 restricted grant per artifact. But there may not be.
<lifeless> so you could denorm to both.
<wgrant> I had considered putting policy on artifact, but it makes reporting slightly awkward.
<wgrant> But should be fine.
<lifeless> Hard to predict.
<wgrant> It really depends on the project.
<wgrant> Security will usually have one restricted observer.
<lifeless> the reporter?
<wgrant> Artifacts in private projects probably won't have any.
<wgrant> Right.
<lifeless> ok, so plan: bug.accesspolicy nullable; task.policyuse nullable; possibly add an artifact.policyuse; possibly drop grant.policyuse
<lifeless> last two subject to scaling testing by you
<wgrant> It's going to be awkward to get something representative, but I'll do what I can.
<lifeless> I feel your pain
<lifeless> you could do what I did with bugsummary
<wgrant> What's that?
<lifeless> conversion script into temp tables
<lifeless> ran it repeatedly on staging until happy
<wgrant> FSVO "you"
<lifeless> we should have a reasonable corpus already, particularly if you use private branches
<lifeless> wgrant: s/staging/dogfood/
<wgrant> Not very representative of performance, but maybe.
<lifeless> if its fast there...
<wgrant> Sometimes hundreds of times faster, sometimes infinitely slower :)
<wgrant> Heh
<wgrant> True
<lifeless> its not the resources that make us make things fast, its the constraints :)
<wgrant> Mmm, yeah, but there's so many bad queries and schema in LP that pretty much anything happening on mawson blows away the cache and kills everything.
<wgrant> I will turn everything off :)
<wgrant> lifeless: Ah, denorming like this also more easily allows us to have a master target for The OEM Conundrum.
<wgrant> Since the policyuse is not calculated from each task's target separately.
<lifeless> it seems to make that unneeded though
<lifeless> I suspect doing a master target may be a big fail
<lifeless> I wouldn't try it myself
<lifeless> (because what if hwe aren't allowed to see all of oem's bugs)
<wgrant> lifeless: Then they can have restricted observers on all the relevant artifacts, just like they do now.
<lifeless> true
<wgrant> Unless they're in the OEM bug supervisor team (which is subscribed automatically, and will be the observer in the new model), they are being subscribed manually.
<wgrant> How would you do it without a master task? Which target would it use to look up the policy?
<lifeless> would what
<wgrant> it == the trigger to push Bug.access_policy through to BugTask.policy_use
<wgrant> And to AccessPolicyArtifact.policy_use, or AccessPolicyGrant.policy_use.
<lifeless> wgrant: I was assuming it would run a small function (task, policy enum) on each task.
<wgrant> lifeless: One of those has to win, in order to be put into AccessPolicyArtifact or AccessPolicyGrant.
<lifeless> ah
<lifeless> ok, I see. Yes, go ahead.
<wgrant> I figure most of the cases are bugs with $foo and hwe-$foo tasks, in which case $foo should win. Not sure how we'll define that, but that can be clarified next week.
<wgrant> fuuuu
<wgrant>   File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/src/oopstools/oops/models.py", line 370, in _get_oops_tuple
<wgrant>     for (start, end, db_id, statement) in statements:
<wgrant> ValueError: need more than 3 values to unpack
 * wgrant enfixorates further.
<lifeless> EWTFISGOINGON
<lifeless> don't tell me bson is translating (foo, None) as (foo,)
<lifeless> that would be fuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
<wgrant> It would indeed.
<wgrant> I remember we ran into problems like this when I instrumented openid.
<wgrant> I assume that's what this is.
<wgrant> But I guess we'll find out in a sec.
<wgrant> Except we've had 2000ish openid timeouts since we moved, and they haven't been problematic.
<wgrant> [212, 212, 'sendmail'], [215, 215, 'sendmail'], [218, 218, 'sendmail']
<wgrant> etc.
<wgrant> sigh.
 * wgrant greps.
<lifeless> are they old oopses or something ?
<wgrant> No, this is OOPS-11a3f88d3d139e90764487de402825c4
<lifeless> can we make sure we have bugs on the producer side as well as whatever you do to fix the consumption ?
<wgrant> The fourth item should be the detail, shouldn't it?
<lifeless> yes
<lifeless> the statement
<lifeless> (start, end, db_id, statement)
<wgrant> For this issue I'll file a producer bug, sure.
<wgrant> For the user-agent one it probably didn't make sense.
<lifeless> the missing value one needs a bug too
<lifeless> user agent is crap data, I'm fine with sanitising that in oops-tools
<wgrant> The missing value one is quite possibly the same thing.
<lifeless> yes
<wgrant> Hmm, putting such an array through bson works fine.
<wgrant> with None and ''.
<wgrant> So not quite that trivial and awful, sadly.
<lifeless> headddeeeeesk
<wgrant> ?
<lifeless> just at the symptoms
<wgrant> Oh.
<wgrant> fwe9ofju*(OW#YU8947238942389234
<wgrant> \
<wgrant> >>> bson.loads(bson.dumps({'blah': (1, 2, object())}))
<wgrant> {u'blah': [1, 2]}
<wgrant> It's unrepresentable, so just omit it :D
<wgrant> lifeless: reasons for hating BSON++
<lifeless> wgrant: so we have unrepresentable objects in the dict ?
<lifeless> wgrant: I doubt json does better.
<lifeless> wgrant: or things we expect to stringify() ?
<lifeless> wgrant: anyhow, if we're doing that (or analagous) we're breaking the oops contract *anyhow*
<lifeless> >>> json.dumps({'blah': (1, 2, object())})
<lifeless> TypeError: <object object at 0x1731080> is not JSON serializable
<stub> Given this is error handling, a case can be made to sanitize the dictionary before serialization, converting unrepresentable stuff to repr(foo) and maybe logging a warning.
<wgrant> lifeless: JSON crashes, yes.
<wgrant> Which is better.
<wgrant> It tells you that you're being a moron, rather than nodding and ignoring you.
<wgrant> Ah.
<wgrant> In the sendmail case it's an email.header.Header.
<wgrant> Which stringifies fine.
<wgrant> But doesn't JSON.
<wgrant> HMmm.
<wgrant> Oh, but we've probably never used it with JSON.
<wgrant> Only RFC822.
<lifeless> where it worked by chance
<wgrant> Yep.
<lifeless> its against the oops contract
<wgrant> (Pdb) bson.loads(bson.dumps({'foo': object()}))
<wgrant> {}
<wgrant> yay
<wgrant> I guess that explains the field.blob thing too.
<wgrant> It'll be an uploaded file thingy.
<lifeless> yeah
<lifeless> reasonable to not include it :)
<wgrant> Yeah, but not to just exclude it.
<wgrant> Particularly since once it's a dict the key will just disappear.
<lifeless> yes
<lifeless> worth a bug filed ustream
<stub> Does the bson library provide hooks for serialization of custom objects?
<wgrant> Oh god bson uses tabs.
<wgrant> I guess the problem is encode_value()
<wgrant> Which has a huge long if/elif, and no catch-all.
<wgrant> lifeless: What happens if the OOPS serialiser raises an exception?
<lifeless> wgrant: it blows up the stack
<lifeless> wgrant: there isn't [yet] a fallback mechanism.
<wgrant> Probably ending up in the appserver error log.
<wgrant> So we can't very well just make this blow up.
<lifeless> wgrant: core oops + oops-twisted is a place to put one, and it probably makes sense to do so.
<lifeless> e.g. generate an oops saying 'could not generate this oops' with the dict keys and values coerced, nonrecursive, to strings, with try:excepts everywhere.
<wgrant> So, should I file LP bugs complaining about the sendmail and field.blob cases?
<lifeless> yes
<lifeless> field.blob might be oops-wsgi
<lifeless> dunno if it was a loggerhead or appserver oops
<wgrant> Don't see why loggerhead would have Zope field names and uploaded blobs :)
<lifeless> wallyworld_: yo
<lifeless> wallyworld_: OCR time :) - https://code.launchpad.net/~lifeless/python-oops-wsgi/0.0.6/+merge/81230
 * wallyworld_ looks
 * wallyworld_ wishes txlongpoll was done so he didn't have to wait for the mp diff
<lifeless> you'd still be waiting
<lifeless> you just wouldn't be refreshing
<wallyworld_> lifeless: at the last epic, the longpoll demo session made a point of showing that the diff generation was instant
<wgrant> wallyworld_: The page updates instantly once the diff is generated.
<wgrant> wallyworld_: The diff generation still takes a while, and is even still on a minutely cronjob.
<wallyworld_> that's a shame :-(
<wallyworld_> i hate cron jobs for this sort of stuff
<wallyworld_> should be event driven
<wgrant> Eventually we'll have an MQ-based jobrunner, I assume.
<wallyworld_> lifeless: dumb question given i'm not familiar with the the code i'm looking at - does it matter that the default tracker will call ensure_start_response twice?
<lifeless> wallyworld_: thats whats passed into it
<lifeless> wallyworld_: no it doesn't matter
<lifeless> wallyworld_: the code is structured to let that change without folk that write custom trackers having to know it changes
<lifeless> wallyworld_: on_first_bytes is called once only, and on_finish once only
<wallyworld_> lifeless: cool. it all seems ok but lacking any familiarity with the package, i'm not sure if i've missed anything
<lifeless> thats fine :)
<lifeless> unless you want to become a wsgi expert - you can cross reference with PEP 3333
<wallyworld_> to save stalling i'll +1 it :-)
<lifeless> <elvis>thankyouverramuch</elvis>
<stub> wgrant: https://code.launchpad.net/~wgrant/launchpad/observer-db/+merge/81104 is flagged as in progress, but the db patch seems done (including the partial index on AccessPolicyGrant which I thought was staying as it was). Want me to tick it yet?
<wgrant> stub: lifeless revealed some additional requirements last night. And convinced me to enum it.
<stub> ok :-)
<stub> Anyone know if there is a bug open for the slow bugsubscription,teamparticipation join we were discussing the other day?
<lifeless> more strictly, OEM revealed ...
<wgrant> lifeless: Well, I assume you found out earlier than you let on :)
<lifeless> yesterday early aco
<lifeless> *avo*
<wgrant> Oh.
<lifeless> ~EOD in florida
<stub> Where did the 'deployable revisions' reports go? By bookmarks are no longer valid
<StevenK> stub: I removed the private redirect yesterday
<StevenK> stub: Didn't you see my mail about the deployment reports becoming public, and that I would remove the redirect in one month?
<stub> StevenK: I probably saw it and ignored it :-)
<lifeless> StevenK: in such cases a follow up at the time can be very useful
<StevenK> Bad stub, no cookie.
<stub> I can just about cope with tomorrow. One months time? Ha!
<StevenK> stub: lpqateam.canonical.com/qa-reports
<spm> [15:58:40] <stub> StevenK: I probably saw it and it was washed away in the flood waters, while I desperately tried to recover it; but alas; twas not to be <== fixed
<StevenK> spm: Now do it in haiku?
<spm> I wouldn't like to keep stealing the limelight like that
<spm> I'm a shy retiring type doncha know
<lifeless> terrible liar
<StevenK> Who are you, and what have you done with the real spm?
<lifeless> spm 1, StevenK 0
<StevenK> Oh, pft.
<poolie> i'm ready to add txfixtures as a dependency
<poolie> are there any docs for this?
<poolie> also, i presume it will be an egg rather than a branch?
<lifeless> so, are intel 320 class SSD's good ?
<poolie> generally reckoned to be
<StevenK> This X201 has a 160, I think, and it's excellent.
<lifeless> I've hit the wall on my laptop
<lifeless> 130GB just isn't enough
<StevenK> I only have 160GB, but I also have a desktop and a fileserver
<lifeless> so do I but the pain of rotating stuff on and off had gotten to me
<StevenK> Sigh. Forgot to unmount NFS partition before moving networks.
<StevenK> wgrant: Can I get from ISPPH to ISourcePackage easily?
<StevenK> lifeless: What do you need to rotate on and off the laptop?
<wgrant> StevenK: Maybe.
<StevenK> wgrant: In a test/harness environment, so feel free to suggest evils like rSP()
<wgrant> StevenK: SPPH.meta_sourcepackage
<poolie> lifeless, oh they come up to 300gb now? or more?
<poolie> do i need a review to add the tar to the download-cache, or do i just add it?
<wgrant> Just add it.
<poolie> and then propose an addition to versions.conf
<StevenK> \
<StevenK> wgrant: I thought that returned a DSP?
<wgrant> StevenK: No.
<wgrant> You might be thinking of SP.distribution_sourcepackage
<StevenK> Bleh. Still doesn't make SPPH queries locally.
<wgrant> You're probably an admin.
<lifeless> poolie: yeah 360GB
<lifeless> poolie: faster too if you have  6GBps SATA
<StevenK> wgrant: No, I'm logged in as a user.
<StevenK> I already thought of that too.
<wgrant> It's not the archive owner?
<wgrant> Or the distribution or distroseries owner or driver?
<StevenK> Logged in as myself, as a member of ubuntu-team
<wgrant> That would do it.
<wgrant> Don't be a member of ubuntu-team.
<StevenK> Nice. ?batch=80 == 1500 queries
 * StevenK tries to work out how to make IBugNomination.canApprove() nicer.
<stub> lifeless: The Intel drives are what people are currently recommending for databases - at a decent price point, reliable but a little slower than some less reliable drives
<lifeless> stub: the 320 series are laptop form factor
<lifeless> stub: there are larger ones around I believe
<stub> lifeless: Intel 320's is what I'm seeing. Not sure if there are server versions too.
<stub> lifeless: http://postgresql.1045698.n5.nabble.com/Recommendations-for-SSDs-in-production-td4958689.html for a thread I saw today
<StevenK> Bleh. Brain fried.
<lifeless> hah up to 600GB now
<lifeless> http://ark.intel.com/products/56569/Intel-SSD-320-Series-%28600GB-2_5in-SATA-3Gbs-25nm-MLC%29
<poolie> do i have to do something special to get the egg hooked up by buildout
<poolie> after putting it in the download cache and in versions.cfg
<stub> Aparently there is a review of the Intel SSDs on Tom's Hardware today
<lifeless> the 710
<lifeless> http://www.tomshardware.com/reviews/ssd-710-enterprise-x25-e,3038.html
<poolie> when i try to import it (and do nothing else with it)
<lifeless> which looks to be that product http://ark.intel.com/products/56585/Intel-SSD-710-Series-%28300GB-2_5in-SATA-3Gbs-25nm-MLC%29
<poolie> i get an importerror kind of mixed together with a zopexmlconfigurationerror
<wallyworld_> poolie: did you do a make?
<poolie> yeah
<wallyworld_> you need to do that
<poolie> i hadn't mentioned it it setup.py, perhaps that was it
<poolie> yep got it
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<poolie> avagoodweekend wallyworld_
<wallyworld_> poolie: still here. just don't want any last minute reviews. i'm fighting doc tests :-(
<poolie> what's the oldest python version lp deploys on? 2.4?
<lifeless> 2.6
<poolie> great thanks
<poolie> so i can use relative imports to help the transition of buildd code?
<lifeless> maybe :) we has some trouble there recently
<poolie> i see https://bugs.launchpad.net/bugs/825485
<_mup_> Bug #825485: canonical.launchpad.scripts.oops imports broken on natty & oneiric <qa-untestable> <Launchpad itself:Fix Released by abentley> < https://launchpad.net/bugs/825485 >
<poolie> so i guess that's a no
<poolie> or, it's dangerous
<poolie> well, it's fine without
<wgrant> poolie, lifeless: LP deploys on 2.6. lp-buildd has no such guarantee.
<poolie> right and on hardy it may be earlier
<wgrant> Indeed, it's 2.5.
<wgrant> And lp-buildd is largely deployed on Hardy.
<poolie> it's only the tests
<poolie> but, it would be nice to run the tests there
<lifeless> wgrant: oh? we missed a platform
<lifeless> wgrant: is there a bug ?
<wgrant> lifeless: Huh?
<wgrant> lp-buildd has always has entirely unrelated deployment constraints.
<wgrant> Because it deploys on crap like hppa.
<poolie> https://code.launchpad.net/~mbp/launchpad/use-txfixtures/+merge/81243
<poolie> one step closer to deploying it separately
<wgrant> And has to build for 5 year old distroseries.
<wgrant> So can't run recent kernels.
<lifeless> wgrant: I thought it ran outside the build environment
<lifeless> wgrant: hppa I can understand :(
<wgrant> It runs outside the chroot.
<lifeless> right, so why does that constrain us
<wgrant> There were issues with building dapper on recent kernels, and we have to support hppa for another 18 months, which sticks them back on hardy.
<wgrant> lpia is similar.
<wgrant> Because those archs don't existing in >= lucid.
<wgrant> Our powerpc buildds also don't really like to boot on most releases.
<wgrant> sparc as well.
<wgrant> ia64/sparc can never come past lucid, hppa/lpia past hardy.
<wgrant> But we need to support ia64/sparc for another 3.5 years.
<poolie> ok
<poolie> i fixed a 2.5ism in txfixtures
<poolie> that's it for me for today
<adeuring> good morning
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring | Critical bugtasks: 266
<jelmer> hi adeuring, do you have time for a quick review?
<adeuring> jelmer: sure
<jelmer> adeuring: https://code.launchpad.net/~jelmer/launchpad/newer-bzr-git/+merge/81259
<jelmer> it's a single revision, the diff is also here: http://bazaar.launchpad.net/~jelmer/launchpad/newer-bzr-git/revision/14251
<adeuring> jelmer: r=me
<rvba> adeuring: Hi! Could you please have a look at this tiny optimization branch? https://code.launchpad.net/~rvb/launchpad/branches-timeout-bug-827935-3/+merge/81262
<bigjools> anyone else seeing MP preview diffs that just say "Empty" ?
<StevenK> bigjools: The MPJ ran after the branch was merged.
<nigelb> ah, if it was merged it can't find the diff?
<StevenK> It compares the branch against tip. If the tip has the contents merged, the diff is nil.
<nigelb> AHHH.
<bigjools> special
<StevenK> Just slow.
<bigjools> thanks
<nigelb> Actually, that means fast.
<bigjools> it means I pushed and lp-landed quickly
<StevenK> It may mean the MPJ ran slow.
<nigelb> bigjools managed to beat the cron.
<bigjools> rabbit is coming!
<StevenK> MPJ is horrible and needs to be beaten with a stick.
<nigelb> rabbit is the holder of the stick I hear.
<StevenK> Possibly over rabbit.
<nigelb> Wait, I thought this bit wwas done in Qastaging wwith rabbit!
<nigelb> I meant to test!
<bigjools> nigelb: no, I am talking about starting jobs with rabbit
<bigjools> different to the browser doing long-polling
<nigelb> bigjools: as opposed to a cron?
<bigjools> yes
<nigelb> Yeah, that'd be nice :)
<bigjools> we turn cronscripts into daemons
<nigelb> INSTANT GRATIFICATION
<nigelb> or at least more instant than currently.
<StevenK> Then we can remove the job system entirely.
<StevenK> AND IT WILL BE GLORIOUS
<nigelb> Some of the stuff discussed in the launchpad session at UDS was nice as well. Like the brainstorm for the new profile page.
<bigjools> StevenK: I doubt we can remove the job system for a while.  Although I'd love to replace it with celery etc.
<StevenK> :-(
<StevenK> The job system makes me very sad.
<bigjools> in what way?
<StevenK> The horrid duplication of effort
<bigjools> I only have one problem
<bigjools> it's NIH
<StevenK> bigjools: My main issue is the sixty-four million interfaces and model classes you have to add for a new job type and they're all subtly different
<StevenK> And require different cron jobs
<StevenK>  /wrists
<bigjools> yeah it's complex and over-engineered
<StevenK> bigjools: Preaching. Choir.
<bigjools> and on cue my random music selection starts playing Rage Against the Machine
<nigelb> bigjools: Excellent timing ;)
<bigjools> StevenK: I also get frustrated at the myriad ways to start a job off
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugtasks: 266
<jtv> Anyone mind if I run the archive publisher on dogfood?
<jtv> bigjools, StevenK, wgrant?
<wgrant> Not I.
<nigelb> WCPGW.
<StevenK> nigelb: It's dogfood. Don't ask that.
<nigelb> heh
<bigjools> jtv: fine
<bigjools> btw you can cancel virtual builds over the API now. You'll also be able to do it in the UI at the next rollout
 * StevenK QA'd it this morning
<bigjools> StevenK: it should not have been marked released though, because you qa-ed the wrong thing :)
<StevenK> I QA'd the UI only
<bigjools> ah ok - but it's not released
<bigjools> we need a permanent QA team in AsiaPac :)
<bigjools> right, food time, ttyl
<wgrant> Yay, ddebs built on prod.
<wgrant> Finally.
<bac> bigjools: would you revisit https://code.launchpad.net/~mbp/launchpad/800295-buildd-split/+merge/81111 and perhaps change your vote to 'approve' if you agree or make another comment?  horse is out of the barn.
<adeuring> rvba: sure, I'll look (sorry for the delay -- had lunch)
<rvba> adeuring: thanks!
<adeuring> rvba: the changes look good, but a suggestion nevertheless: What about making is_branch_count_zero a @cachedproperty? after all, its accessed more than once and it calls a method. (I hva no idea though how expensive the method is...)
<rvba> adeuring: well, the method is definitely no expensive: because self.branches().visible_branches_for_view and self.branch_count are @cacheproperty.
<rvba> s/no/not/
<adeuring> rvba: ah, ok, so let's leave the @property
<adeuring> rvba: another question: If branch_count needs to be evaluated in is_branchcount__zero: DO you expect the query in these cases to be faster?
<adeuring> ...if not, would it make sense to call resulstset.any() instead?
<maxb> Hm
<maxb> https://launchpad.net/builders --> Forbidden
<rvba> adeuring: I'm not sure I understand your suggestion, this branch does not bring any improvement to the cases where branch_count needs to be evaluated.
<maxb> I suppose a private object has leaked onto the page?
<wgrant> maxb: yeah, that happens when a recipe is building int a private archive
<wgrant> into
<rvba> adeuring: When the batch is empty that is.  But hopefully this wont happen in many cases because when the number of branches is huge, chances are that you have all the statuses represented and the only filtering possible is by status.
<wgrant> maxb: There's a bug about it.
<adeuring> rvba: right; i I just wondered if it would be hard to add that case too. But that's probably my bad habit to squeeze too much changes into one branch ;)
<adeuring> rbva: r=me then
<nigelb> g22
<rvba> adeuring: Thank you.  I know it's strange but for these requests .is_empty() is almost as expensive as .count().
<deryck> abentley, adeuring -- I'm on weekly checkpoint call for feature so will need to hold standup for a bit.
<rvba> I know that's utterly bizarre but that's what the profiling I've do says.
<deryck> abentley, adeuring -- I can ping when done.
<abentley> deryck: Okay.
<adeuring> rvba: well, SQL queries can be "strucluraaly slow" ;)
<adeuring> deryck: ok
<jelmer> thanks adeuring, bac
<danhg> #ubuntu-uds
<deryck> abentley, adeuring -- let's do standup at top of hour.  cool?
<adeuring> deryck: ok
<abentley> deryck: sure.
<rvba> adeuring: It's not a rewiew but I'd like your help for something if you have 2 minutesâ¦
<adeuring> rvba: sure
<rvba> adeuring: I'd like you to hit https://code.qastaging.launchpad.net/~ubuntu-branches 5-6 time and give me the oops links
<rvba> :)
<rvba> times* even
<rvba> the reason for this:
<rvba> I have landed 2 branches to try some improvements on this page. All is protected by a FF that is on only for me.
<adeuring> ok
<rvba> And since this page is heavily dependent on TeamParticipation I need someone in the same teams.
<rvba> to compare how my branches improve things
<adeuring> rvba: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-a22826044c7c82ef553e7d363ecb35d8 https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-06fabadb8b802665135e2f048d72e8ba https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-e13505381303340a3d138501c0f14c3a https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-51525de16620edd7a8c6c5ecfa64e39b https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-e7b00379ecad43e9a36798e84a87fb7a https://lp-oops.c
<rvba> adeuring: thank you very much.
<adeuring> welcome :)
<rvba> adeuring: sorry to bother you againâ¦ but did one of the requests succeed?  I'd like to make sure the FF has been properly setâ¦
<adeuring> rvba: no, all requests timed out
<rvba> adeuring: ok, thanks.
<gmb> danhg: You've marked bug 240067 as New... It's going to pop up when we come to process the to-Triage list, so I'm just wondering if there's any reason we should leave it as New.
<_mup_> Bug #240067: Launchpad projects need wikis <feature> <lp-foundations> <ubuntu-platform> <Launchpad itself:New> < https://launchpad.net/bugs/240067 >
<jelmer> danhg: did you really mean to change the status of bug 240067 back to new?
<_mup_> Bug #240067: Launchpad projects need wikis <feature> <lp-foundations> <ubuntu-platform> <Launchpad itself:Triaged> < https://launchpad.net/bugs/240067 >
<bigjools> jelmer: you scared him off
<rvba> adeuring: could you have a look at this tiny MP? https://code.launchpad.net/~rvb/launchpad/branches-timeout-bug-827935-4/+merge/81290
<adeuring> rvba: sure
<rvba> thx
<adeuring> rvba: looks good; I think always showing the link is fine. But again a question, just out of curiosity: Would it be possible to use a UNION ALL in the query? Tht might  be faster than the current query
<adeuring> rvba: r=me, btw
<rvba> adeuring: I tried a bunch of things with this query: the problem is not the OR in itself.
<adeuring> ok
<rvba> The problem is simply Branch.transitively_private = FALSE fetches 300k rows (in the bad case I'm working on)
<rvba> So we get a seq scan which is the right thing to do.
<rvba> Because 1/3 of the table is returned.
<rvba> But this is painfully slow.
<rvba> All of this simply to choose whether or not to display a linkâ¦
<rvba> You see what I mean :)
<adeuring> rvba: right...
<rvba> With the branch that you reviewed earlier, I expect a 50% gain.
<adeuring> rvba: sounds good!
<rvba> Indeed, but this is so sensible that I'll say "victory" when I see the numbers ;)
<rvba> adeuring: thanks for the review.
<adeuring> rvba: yeah, I know the feeling "what else might come next" with timeout issues ;)
<rvba> adeuring: exactly
<adeuring> rvba: but work on timeout bugs can be fiun nevertheless
<rvba> adeuring: It's rather interesting I think.  It feels a little bit like open heart surgery.
<adeuring> rvba: interesting comparison :) To me it felt sometimes like tinkering with a motorbike engine, when I was a youngster ;)  as a
<rvba> adeuring: ;)
<bigjools> jml: hello. I am just looking into the escalated debtags bug and wondered how much you've looked at that so far?
<gary_poster> danhg, are you around?  If so, may I assign this commercial feedback ticket to you?  https://support.one.ubuntu.com/Ticket/Display.html?id=7272
<nigelb> Is there a way for me to tell launchpadlib to use my local launchpad?
<nigelb> (besides hacking hosts file)
<gary_poster> yes, nigelb.  /me tries to remember how
<nigelb> yay!
<gary_poster> nigelb, use the service_root argument, and specify 'https://api.launchpad.dev/'
<gary_poster> or launchpadlib.uris.DEV_SERVICE_ROOT
<nigelb> gary_poster: Aha, thanks!
<gary_poster> welcome
<nigelb> I want to make it a setting in summit so I can move it out to local lp for testing :)
<deryck> abentley, adeuring -- I think by Monday or Tuesday, whenever we get the current branches all landed and deployed, we should turn on the new bug lists for ourselves.
<adeuring> deryck: yeah
<abentley> deryck: that includes the beta banner that lets us turn it off, right?
<deryck> should, yes.  depending on how well things go for adeuring.
<nigelb> buglists is ready?
<deryck> well, actually, I don't think adeuring will have this off switch in his first pass.
<deryck> nigelb, gettting very close.
<nigelb> deryck: Exciting! :)
<deryck> nigelb, we have two weeks left of polish and then we will turn them on for beta testers.
<adeuring> I'm still dealing with a test failure...
<nigelb> deryck: Yay! Looking forward :)
<deryck> adeuring, I looked at the MP for the python side today.  I imagine there is some test fallout :)
<deryck> adeuring, but I like the approach you've taken.
<adeuring> deryck: thanks!
<deryck> adeuring, I got it intellectually, but seeing it in code made it real for me.  And I like it.
<deryck> bac, I have a js branch for review if you have the time.
 * bigjools outta here, have a nice weekend all
<jml> the answer to bigjools's question is "not at all"
<abentley> bac: could you please review https://code.launchpad.net/~abentley/launchpad/new-bug-fields/+merge/81310 ?
<bac> deryck: done
<bac> abentley: sure
<deryck> bac, thanks!  I've struggled with the naming too.  Configurer was even an option at one point. ;)
<deryck> I thought "settings" might be more obvious, too.
 * deryck has to leave now.
<benji> does anyone know how to get a branch scan job to run in a dev environment?
<nigelb> I did that once.
<abentley> benji: make scan_branches
<nigelb> aha.
<abentley> benji: or just run cronscripts/scan_branches.py
<benji> abentley: right I know how to get the jobs to run, but it is creating the jobs to begin with that I don't see a sane way of doing
<abentley> benji: scheduling the jobs is done by codehosting.  Locking the branch (e.g. by pushing to it) will schedule a scan.
<benji> ok, let me see if I can do something with that (I have a branch that I want to get scanned, but it's foreign to the dev environment)
<abentley> benji: I don't understand how you could scan a branch that was not part of the dev environment.
<benji> abentley: I can't ;)  I want to get it into the environment so I can get a scan job created.
<abentley> benji: Okay, you need to push it, which will create a scan job, which will run when you do make scan_branches.
<benji> will someone with code import fu take a look at this question?  https://answers.launchpad.net/launchpad/+question/176710
<benji> I /think/ the answer is "no" but want to be sure.
<abentley> benji: the answer is "no", because you cannot move a branch from one project to another.
<benji> thanks abentley!  I'll reply thusly.
<jelmer> other than some extra resource usage on lp, deleteing and creating the branch again should have the same effect as moving
<abentley> bac: like this? http://pastebin.ubuntu.com/728537/
<bac> abentley: sure.  i just thought the test *after* it had been recalculated looked funny.  i'm sure it was correct but i had to think about it.
<abentley> bac: Yes, it had to be correct, (because x / 60 !=0 if x > 60) but it was confusing.
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<wgrant> benji, abentley: You *can* move a branch from one project to another, but the UI was removed because it was too hard to make it work for package branches as well. It's still perfectly doable through the API.
<jelmer> wgrant: oh, interesting
<jelmer> wgrant: moving between package branches and product branches would actually be a very nice use of such a UI
<wgrant> jelmer: Yes.
<wgrant> jelmer: Although I somewhat disagree with how package branches were implemented.
<wgrant> jelmer: Really they should probably be special distro-related branches on the product.
<wgrant> IMO
<wgrant> Rather than splitting all the branches for a codebase across $lots of places.
<jelmer> wgrant: agreed
<jelmer> wgrant: that still seems possible though, it mainly seems like a UI issue to me
<wgrant> It's going to be particularly awkward for derived distros, because they are going to need cross-pillar stacking or hundreds of gigabytes of extra storage.
<jelmer> we support multiple levels of stacking fine afaik
<wgrant> We do, but we rarely do it cross-pillar.
<wgrant> Because that crosses privilege boundaries, which is very riksy.
<jelmer> Hmm, I'm not sure I understand what you mean by cross-pilar in this context.. perhaps my understanding of that term is incorrect
<jelmer> do you mean between the different derived distros? Isn't there an implicit trust of the parent distro?
<wgrant> Well, say Linaro wants to use package branches.
<wgrant> Right.
<wgrant> I guess.
#launchpad-dev 2011-11-05
<lifeless> evening sinzui
<sinzui> yes it is
<lifeless> s/^/good /
<sinzui> I had higher hopes for it. I lost a lens to my glasses and I got lock out of my room in my bare feet (with plum painted nails)
<sinzui> I think I made Claire laugh though
<lifeless> lol
<lifeless> I shouldn't, but I am
<lifeless> I hope you find the lens
<sinzui> thanks
<lifeless> how was the party
<sinzui> I think it was nicer this time. I played a few games bzr+ex-launchpaders. I left after the music though. too loud
<lifeless> yeah, it can be
<lifeless> I keep meaning to take ear plugs
#launchpad-dev 2011-11-06
<thumper> morning
<nigelb> Morning!
<nigelb> thumper: Didn't realize till recently you knew Blair from Mozilla :)
<thumper> hi
<thumper> nigelb: and you know Blair too?
<nigelb> thumper: Yup!
<nigelb> I contribute to Mozilla as well :)
<thumper> ah
<thumper> Blair and I both go to the local CodeCraft meetings, and occasionally meet as an informal home-workers get together
<nigelb> Nice :)
<nigelb> We had fun bringing out a pirate locale for Firefox on Talk like a Pirate day
<thumper> heh
<lifeless> why is ~ubuntuqa tagging long-closed bug[s] ?
#launchpad-dev 2012-10-29
<wgrant> StevenK: Did you ec2 and pre-QA that twisted change?
<wgrant> (it's a bit awkward that we're upgrading to a new major Twisted release, but cannot QA buildd-manager)
<StevenK> wgrant: I did ec2 it. It was lightly-QA'd. IE, with a local dev instance, I could ssh to codehost
<wgrant> OK
<StevenK> If that doesn't ally your concerns, I understand. :-)
<lifeless> look at it this way
<lifeless> its UDS, noone needs builders this week
<wgrant> Hah
<StevenK> lifeless: Oh, so no gems about Twisted's upstream or anything? :-)
<lifeless> StevenK: twisted are pretty good at backward compat
<lifeless> StevenK: you probably don't even need to qa it </wink>
<wgrant> Indeed, so I'm not hugely concerned
<wgrant> But it's still something we should really be testing
<wallyworld_> StevenK: wgrant: do you know where the tests for the nascent uploads overrides are done? i can't see anywhere where the tests set up various scenarios, run the uploads, and check the overrides. but then again, i'm not sure where to look either
<StevenK> wallyworld_: lib/lp/archiveuploader
 * wallyworld_ looks
<StevenK> wgrant: Can I land https://code.launchpad.net/~stevenk/launchpad/db-destroy-potmsgset-potemplate/+merge/131294 ?
<wgrant> wallyworld_: A disturbingly large number of nascentupload tests are somewhat incidental results of specific prebuilt sample packages being uploaded
<wgrant> I'm not sure about this particular case; it may just be untested
<wgrant> There are also some archiveuploader-covering doctests in lib/lp/soyuz
<wgrant> StevenK: Sure
<wallyworld_> oh, will look around a bit. thanks.
<StevenK> I think the override policy bits are tested, but can't think of an end-to-end test
<wgrant> archiveuploader doesn't really use override policies today
<StevenK> Oh, it's the copier
<wallyworld_> i am looking for tests which set up various source packages etc and then do uploads to test the various scenarios
<StevenK> With the uploader being "Eh, later" IIRC
<wallyworld_> they mostly seem to be testing hard coded things
<StevenK> wallyworld_: Both wgrant and I dislike the archiveuploader tests
<wallyworld_> sadly, i made a change and ran all the tests with nascent in the test file/name and nothing failed
<StevenK> They're archaic, slow, mostly doctests and use vast amounts of sampledata.
<wallyworld_> yeah, so it seems
<wgrant> uploadprocessor tests a lot of things
<wgrant> You really have to run all the archiveuploader tests
<wgrant> + soyuz-set-of-uploads
<wgrant> and a couple of others
<wallyworld_> ok, will try thise
<StevenK> SSoU is horrid
<StevenK> wallyworld_: Have you killed yourself yet?
<wgrant> StevenK: wtf
<wgrant> StevenK: Bug #1013013 somehow got tagged as part of r16208
<_mup_> Bug #1013013: Timeout viewing my own translations <lp-translations> <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by stevenk> < https://launchpad.net/bugs/1013013 >
<wgrant> Oh
<wgrant> misread revno
<StevenK> wgrant: Reading is a good skill.
<wgrant> 16208 and 12088 are both translations changes and adjacent in my mailbox
<wgrant> Last digit matches, must be right :)
<StevenK> Haha
<StevenK> wgrant: Well, first and last. Who cares about the middle bit.
<wallyworld_> StevenK: no
<StevenK> wallyworld_: That's good to hear, at least.
<wallyworld_> just trying to see the best place to add some new tests. making progress
 * StevenK loads up OOPS-e8f60c07d46b84f6e939a46d4bc8966c and reaches for the rusty spoon
<wallyworld_> it does seem that things are scattered in a few places
<StevenK> wgrant: TranslationsPerson:+index on qas does not love me
<StevenK> The query that constantly shows up is ITranslationsPerson.suggestReviewableTranslationFiles() :-(
<wgrant> That's the one we didn't fix on Friday, right?
<StevenK> Indeed
<StevenK> Oh look, it finally worked
<StevenK> 8 page loads later
<wgrant> Is my query fast?
 * StevenK greps out an OOPS ID
<StevenK> wgrant: Based on the fact that suggest is called after get, I'm going to hazzard a 'yes', but confirming
<wgrant> StevenK: You should be able to use the inline query log
<wgrant> Just expand it, /WITH, look for MASSIVE BLOB OF PAIN
<StevenK> 11.434ms 	
<StevenK> SQL-main-slave: WITH recent_pofiles AS (SELECT POFile.id FROM POFileTranslator ...
<wgrant> Yay
<StevenK> 1023.917ms 	
<StevenK> SQL-main-slave: SELECT DISTINCT POFile.currentcount, POFile.date_changed,
<StevenK> That's suggest
<wgrant> Aya
<StevenK> Terrible query is still utterly terrible, news at 11
<StevenK> wgrant: I was thinking about it over the weekend ...
<StevenK> wgrant: Could we use the 4 CTEs of doom with an anti-join and limiting to like 50?
<StevenK> We only want to suggest a small number of projects, so limiting one of the CTEs to a small number of pofiles doesn't seem like a bad move
<wgrant> Do we only want to suggest a small number of projects?
<wgrant> We need a specific plan in mind for this sort of thing
<wgrant> For the previous one, it was that we only cared about recently translated pofiles.
<wgrant> Do we have such a shortcut here?
<StevenK> We have a no_older_than that might be specified
<StevenK> I'm re-checking what the view does, since Friday was a sleep-induced haze
<StevenK>         fetch = 5 * (list_length - len(overall))
<StevenK>         suggestions = self._suggestTargetsForReview(fetch)
<StevenK> And then,         pofiles = person.suggestReviewableTranslationFiles(
<StevenK>             no_older_than=self.history_horizon)[:max_fetch]
<StevenK> So a quicker fix might be tossing max_fetch down one level so the database can actually LIMIT
<wgrant> Do you have an explain analyze for the existing query?
<StevenK> But I like the CTEs better than the ten joins of utter doom, so I'd like to use them if we can
<wgrant> The CTEs we used for the previous query are somewhat irrelevant here, as they existed to coerce postgres into choosing a plan style that cannot work here
<StevenK> wgrant: No, and running queries on mawson is somewhat useless given it has no disk at all currently
<wgrant> It's also not completely indexed
<wgrant> Fortunately we have people with access to fast production databases :)
<StevenK> Let's try staging
<StevenK> Or prod, right
<wgrant> About 2/3 of the way through indexing the restore, then we will have DF back
<wgrant> The big indices are done
<StevenK> If you started it on Friday, then it's grown another day
<StevenK> wgrant: So we have an explain analyze now
<wgrant> StevenK: What's this query doing?
<StevenK> wgrant: It's virtually indentical to the get query, except it wants pofiles that the person hasn't contributed to
<wgrant> StevenK: Right, so you can probably see why the previous optimisation is no longer useful
<StevenK> Yes, lots and lots of pofiles
<StevenK> Which is why I suggested limiting the number of pofiles that the CTE returns
<wgrant> It wants the last 50. We can't really limit earlier than we do now, because then filtering my give us less than 50
<StevenK> Bleh
<StevenK> Blah, I can't check on DF due to the DB rebuild
<wgrant> So, we want pofiles that are enabled/preferred for translation, that the user has review privileges for, and that the user has not translated recently.
<StevenK> I was going to have webops repurpose a bunch of pofiles to me so I can see if qas will still load that page
<wgrant> StevenK: What are you wanting to check?
<StevenK> I wanted a user id with a 'average' number of pofiles
<wgrant> Well
<StevenK> So I could UPDATE pofile set ...
<wgrant> We don't really need that yet
<wgrant> We know the query performs terribly for even you now
<StevenK> True
<StevenK> So it's qa-ok, but it doesn't fix the bug
<wgrant> Right
<wgrant> We know from experiments last week that the fixed query performs well across a wide range of users, and we have good reason to believe it will behave for everyone.
<StevenK> Right
<StevenK> It's either quick or blistering fast if the person has no pofiles at all
<wgrant> Right
<wgrant> Because we're able to choose a small set to start with
<wgrant> That's the key to a fast query
<StevenK> It seems the kicker is 'that the user has no translated recently.'
<wgrant> Well, sort of
<wgrant> The kicker is the combination of that and the sort
<StevenK> Right
<StevenK> wgrant: So I guess the question is, how can we choose a small set?
<wgrant> StevenK: Not really
<wgrant> The better question is "is the result of this query useful?"
<StevenK> wgrant: You think suggesting things is not useful?
<wgrant> StevenK: Well, it's not really a binary thing, but a query is not always sufficiently useful to justify the effort expended in its calculation.
<StevenK> Right
<StevenK> We can't even tell if users actually do go "Oh, yeah, I should do some."
<StevenK> .. I can't even see the suggestions on Person:+translations
<wgrant> You probably don't have review privs in the sense that the query wants them
<StevenK> Ah ha
<StevenK> wgrant: So, should we kill it entirely?
<wgrant> If we can't thoroughly optimise it, deletion is one option
<StevenK> I can't think of how to optimise it, sadly.
<StevenK> steven@undermined:~/launchpad/lp-branches/remove-suggested-translations% bzr di | diffstat -s
<StevenK>  5 files changed, 8 insertions(+), 155 deletions(-)
<StevenK> wgrant, wallyworld_: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? Or shall I shake jtv until an opinion falls out?
<wallyworld_> StevenK: best to check with a SME i think
<wgrant> StevenK: I'd like to confirm with a rosettay person
<StevenK> wallyworld_: A what?
<wallyworld_> subject matter expert
<StevenK> wgrant: They're kinda thin on the ground. :-(
<lifeless> they should be online in a few minutes
<wgrant> StevenK: You need to talk to dpm anyway :)
<StevenK> nightly.sh might have actually run update-stats by now
<StevenK> Not sure when it's kicked off in the chain
<StevenK> jtv: Hai. Do you have one sec to cast your eye over https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? I'm not after a review at all, just your opinion on what the MP is doing.
<jtv> I probably do at that.  Hi.
<jtv> StevenK: best thing I can do right now is go over why it is the way it is.
<jtv> The purpose was to guide translators towards where their efforts can do the most good.
<StevenK> jtv: Right, and I understand. I just don't think it helps if the query that is trying to do that takes 4 seconds and causes continous timeouts. :-)
<jtv> Absolutely.  Just going over what it tries to achieve so as to help inform the decision.
<jtv> The obvious thing to do would be to steer translators to the highest-priority unfinished translations.  But that leads to duplicate effort, for several reasons:
<jtv>  - multiple translators will be looking at the same page at the same time.
<jtv>  - unreviewed translations pile up before reviewers can get to them.
<jtv>  - viewers are incorrectly concentrated on translations that may already be finished until the â100% completeâ statistics have filtered through.
<jtv> So we wanted a bit of random spread.
<StevenK> jtv: If you do have some suggestions on how to stop the query asking that question having to join through ten tables, that would help to.
<jtv> I don't think I'll have time to look into that, I'm afraid; but I'm sure there are more efficient ways of achieving something that's similar enough.
<jtv> We built that functionality from the UI down, and so we may have overspecified it slightly.
<StevenK> jtv: In which case, some clues about where to look would be nice.
<jtv> Hang on, I'll ask huwshimi
<StevenK> I'm just playing in the shallow end of translations. :-)
<jtv> StevenK: talking irl with Huw here
<jtv> StevenK: interestingly, it looks as if the suggestions are horribly broken.
<jtv> Huw gets a bunch of suggestions to review Low German translations.
<StevenK> Haha
<nigelb> haha, whaat
<jtv> He is not now, nor ever has been, associated with Low German
<jtv> Oh!
<jtv> He just discovered that he is a member of some translation teams for complicated and as-yet unexplored reasons.
<jtv> StevenK: Huw just asked a good question â is the query's performance very sensitive to the number of results?
<StevenK> Nope
<StevenK> It's terrible no matter who you're asking about
<jtv> Didn't think so.  :/
<jtv> Well it's particularly timeout-prone in my case.  :)
<StevenK> I bet
<jtv> StevenK: here's a solution we just discussed...
<jtv> We could stop filling out the list with random suggestions, but underneath the list, have a âtake me to a random project that needs my helpâ link.
<StevenK> jtv: Is there a way to fetch that quickly?
<jtv> StevenK: that means you never execute the query until the user requests it, but its functionality is still there _and_ it's explicit, not cryptically mixed in with more obvious functionality.
<StevenK> Well, moving the query from one page to another just means another page will timeout
<jtv> StevenK: Wouldn't need to be in the generated HTML â could link to a redirect.
<jtv> So at worst, the redirect would time out.  But there'd be a lot fewer hits on it.
<StevenK> jtv: Right, which means another page will timeout, and means I fix a critical bug by creating another.
<StevenK> IE, not on.
<jtv> Yes, but!
<jtv> It'd allow us to remove some of the selection criteria.
<jtv> As Huw just put it, take a slot-machine approach.
<jtv> If we have a link to a random project to work on, people may be a lot more acceptable of iffy guesses.
<jtv> For example, we won't need to exclude projects that the user has already been translating on.
<StevenK> Oh, so we just one random project/distro where rosetta_usage == ServiceUsage.LAUNCHPAD ?
<StevenK> Because that's fast
<StevenK> And if it isn't, it can indexed
<jtv> A bit more than that, but in principle, yes.
<StevenK> jtv: Do you have time to scribble all this down on the MP?
<jtv> Huw suggested we might make the query less sensitive to the user's identity.
<jtv> Should be able to, yes.
<jtv> Instead of checking translation permissions, we could just limit the suggestions to projects with sufficiently liberal permission models.
<StevenK> Right
<StevenK> Which is nice and easy to make performant.
<jtv> The core reason the query was so slow, really, was that we're pushing these suggestions in the user's face without explanation.
<jtv> The suggestions have to work hard to avoid including absolute nonsense.
<StevenK> jtv: So, we should be able to 1) Grab a number of projects that fit that, and then grab a random unreviewed POFile from those and job's done?
<jtv> StevenK: more or less â there's still the problem that you've got to look at multiple tables, several of which may provide a reason why a particular record is not a candidate.
<jtv> So you can't quite say âgive me one project, and then we'll look for a POFile there.â
<jtv> It's still not a trivial problem.
<jtv> But there's a lot of room for compromise once we make it clear that it's a random choice.
<jtv> Honesty has its benefits for both parties.  :)
<StevenK> Heh
<jtv> As Huw notes, that listing, hard as it tries not to present nonsense, already comes up with some very surprising stuff.
<StevenK> It will come up with at least 9 pofiles you have worked on, and then at least one suggestion
<jtv> Yeah.  Filling out that list with suggestions isn't really a very good thing to do.
<jtv> StevenK: it'd be fine with me to vary the length of that list.  It's tested behaviour.
<jtv> huwshimi, StevenK: shall I just summarize on the MP?
<StevenK> jtv: Please do.
<jtv> huwshimi also concurs
<jtv> (irl shortcuts irc but it can get confusing)
<jtv> StevenK: commented on the MP.
<abentley> czajkowski: I don't know.  jam?
<jam> abentley: I don't have czajkowski's question in my backlog, can you give context?
<rick_h_> jam:  czajkowski- does anyone know if we have an intro to bzr video anywhere done ?
<jam> rick_h_: http://wiki.bazaar.canonical.com/Documentation with a section on "Screencasts & Presentations" and an "Installing Bazaar introduction from EmmaJane."
<rick_h_> czajkowski: ^
<czajkowski> ahhh
<czajkowski> thanks you
<nigelb> Can a private team be a member of a public team?
<czajkowski> nigelb: why on earth would you want that!
<nigelb> czajkowski: I can think of a bunch of use cases, the most immediate one that I thought of was the possibility of having ~canonical a member of ~ubuntu-etherpad.
<czajkowski> nigelb: it's not need ed for canonical most are already on there and the rest join like everyone else.
<nigelb> czajkowski: I'm not saying it's needed. I was asking if it were possible on LP.
<czajkowski> sinzui: is there a way to check LP for Linkedin spam like we do for others like nude or prescription, save people creating questions https://dev.launchpad.net/MaintenanceRotationSchedule
<sinzui> czajkowski, just search for it
<sinzui> if it is an issue, we add it to the list of searches to check
<czajkowski> nods it is
<sinzui> czajkowski, I think this is it: https://launchpad.net/+search?field.text=%22LinkedIn+Corporation%22&field.actions.search=Search
<sinzui> yes, that shows a lot of comments I would hide
<czajkowski> sinzui: bah I was chaanging the old search and it wsnt returning that
<czajkowski> thanks
<czajkowski> this will make mpt happy
<sinzui> czajkowski, it will take 30-60 days for google to drop that match from its index
<czajkowski> ok good to know thanks curtis
<czajkowski> sinzui: at uds, so am working on answers and bugs, though the two current one stump me, only thing not able to do is keep an eye on irc
<sinzui> yes, there are are some hard questions at the moment
<sinzui> I am waiting for caffeine to rattle me some more
<czajkowski> sinzui: bit stumped on https://support.one.ubuntu.com/Ticket/Display.html?id=24331 can you point me in the rihght direction
<sinzui> The user does not understand that Lp requires a registered project
<sinzui> either he needs to register it, or he uses the one already registered
<czajkowski> ack
<sinzui> wow, this guy has been in Lp since 2005
<cjwatson> Oops, I broke buildbot.  Fixing.
<czajkowski> cjwatson: naughty
<sinzui> cjwatson, ping me if you want a review.
<cjwatson> sinzui: Could you just revert the broken rev for me for now, perhaps?  I do want to fix this, but it's going to require pretty extensive changes to this horrible doctest
<sinzui> :(
<cjwatson> sinzui: And I'm going to have difficulty finding enough time to fix in the remaining half-hour of UDS today
<sinzui> I will queue the revert
<cjwatson> So I'd rather unblock buildbot
<cjwatson> Thanks
<sinzui> I think I am the only developer today, and I am writing, not coding
<cjwatson> The whole of xx-copy-packages.txt is utterly built around synchronous copies
<cjwatson> It can probably be patched up by just inserting lots of job.run() kinds of things
<ricmm> hi there, got a question for some launchpad expert
<ricmm> I need to pull the binary debs for a given build in a private PPA
<ricmm> I'm logged in and I got credentials set up, but I cannot find a way to expose the binary file through lplib directly... only by getting the url with binaryFileUrls()
<ricmm> but that the latter I have to do it as a normal browser session, except this is a headless system
<ricmm> is there a way to maybe generate a one-time cookie so I can use urllib to pull the file? or is there a lplib-approved method?
<sinzui> ricmm, I don't know the answer to your question. I am sure there is no approved way. I think this approach looks viable: http://stackoverflow.com/questions/5082128/how-do-i-authenticate-a-urllib2-script-in-order-to-access-https-web-services-fro
<czajkowski> sinzui: dashing out again, mailed you re info I was unsure about
<czajkowski> if you have a chance can you look over it
<czajkowski> please
<sinzui> okay
<wallyworld_> wgrant: StevenK: sinzui: https://pastebin.canonical.com/77402/
<StevenK> wallyworld_, sinzui: https://www.youtube.com/watch?v=6TiXUF9xbTo ?
#launchpad-dev 2012-10-30
<StevenK> Purple buildbot
<wgrant> Exception?
<wgrant> Yeah
<wgrant> subunit corruption
<StevenK> wgrant, wallyworld_: http://instagram.com/p/RYnio1s6gf/
<wallyworld_> yeah, saw that. i don't like instagram
<wallyworld_> i can't see what all the fuss is about
<wallyworld_> why ruin perfectly good photos with crappy filters
<StevenK> wallyworld_: Yes, but you don't like anything that's social. And I'm getting off your lawn now.
<wallyworld_> how is ruining a good photo being social?
<wgrant> We may have a dogfood DB in about 20 minutes, finally...
<StevenK> Facebook bought out Instagram, which I thought was your issue.
<StevenK> However, I was talking about the content, not the platform hosting the photo. :-)
<wgrant> The issue with Instagram is that it's the worst idea ever
<wgrant> *and* it's "social"
<StevenK> Note to self: Do not share hurricane photos with workmates.
<wallyworld_> StevenK: sorry, i saw instagram and my radar went off
<wallyworld_> StevenK: did you see the one of the crane?
<StevenK> wallyworld_: Nope
<wallyworld_> StevenK: it was on this morning's news - a huge apartment building, still under construction - the crane is folded in half and dangling in the wind
<StevenK> http://www.smh.com.au/photogallery/environment/weather/hurricane-sandy-strikes-us-coast-20121030-28g5z.html?selectedImage=3
<StevenK> That one?
<wallyworld_> yeah
<StevenK> ... holy ...
<wallyworld_> the latest pictures have it swinging in the "breeze"
<wallyworld_> those apartments are selling for $90M
<enginespot> Hi everyone , I get this channel from here https://help.launchpad.net/
<enginespot> currently I have a question , I want to get people list of launchpadlib
<enginespot> but I only get 50 persons
<enginespot> so I do not know how to to
<wgrant> StevenK: DF is back, and timing out merrily
<wallyworld> wgrant: what's the difference between spr.upload_archive and spph.archive? are these related?
<StevenK> Loosely
<StevenK> spph.archive == Archive it is published in, spr.upload_archive == Archive it was first published in, so they can be different.
<wallyworld> right, ok. thanks
<wgrant> s/first published in/first uploaded to or seen in/
<wgrant> It wasn't necessarily ever published there.
<wallyworld> roo bad spph.archive is not denormalised ontop spr
<wallyworld> too
<wgrant> wallyworld: That doesn't make sense
<StevenK> It can't be.
<wgrant> Since an SPR can be published in an unbounded number of different archives
<StevenK> Or even different series in the same archive
<wallyworld> ok
<enginespot> when I use the method PersonSet.find()
<enginespot> it told me AttributeError: type object 'PersonSet' has no attribute 'find'
<StevenK> wallyworld: http://www.smh.com.au/photogallery/environment/weather/hurricane-sandy-strikes-us-coast-20121030-28g5z.html?selectedImage=1
<wallyworld> wgrant: these 2 queries should produce same result, but 2nd one faster. can you try on df for me when you have a moment? https://pastebin.canonical.com/77405/
<wallyworld> StevenK:  that's a lot of water
<wallyworld> enginespot: find() is exported, so perhaps you can paste your code for us to look at
<StevenK> wallyworld: http://www.smh.com.au/photogallery/environment/weather/hurricane-sandy-strikes-us-coast-20121030-28g5z.html?selectedImage=10 is pretty impressive too
<wallyworld> yes indeed
<enginespot> my code like the follows:
<enginespot> people=PersonSet.find()
<enginespot>     #    for project in pad.projects:
<enginespot>     #        print project.name
<enginespot>     people = pad.people
<enginespot>     i = 0
<enginespot>     j=0
<enginespot>     for person in people:
<enginespot>         i+=1
<enginespot>         ppas = person.ppas
<enginespot> #        r.set(person.name, 1)
<enginespot>         for ppa in ppas:
<enginespot>             j += 1
<enginespot>             r.set(j, ppa)
<enginespot> I want to get a person list
<enginespot> or launchpadlib can supply to me a pointer , I can read a person one by one
<StevenK> enginespot: You've been told by two different people that what you want isn't possible. Asking different people isn't going to change the answer.
<enginespot> yes , I find a different interface from launchpadlib
<wallyworld> StevenK: what was the strategy going to be to stop the mixed visibility error oopses? what we do now to show <hidden> seems reasonable? i can't recall what the end game was to be
<StevenK> I've can't recall off the top of my head, maybe wgrant can.
<wgrant> It's complicatedâ¢
<wgrant> Ignore for now
<wallyworld> wgrant: i made a mistake on the sql above, here's a fixed version https://pastebin.canonical.com/77410/
<wgrant> wallyworld: Ah, sorry, distracted by secure boot destroying the world
<wgrant> Will run in a sec
<wallyworld> no hurry
<wallyworld> i'm not sure if we need the filter on the spph subquery
<wgrant> wallyworld: So removing the persistent cache fixed the ec2 issue?
<wallyworld> wgrant: yeah, thanks. my mind was telling me it wasn't persistent when clearly it was
<wgrant> Great
<wallyworld> if you want to look at my soyuz mp, it's up.
<wgrant> I've glanced at it and opted to try to finish my current stuff before diving into it, unless you're in a hurry
<wgrant> wallyworld: Is this the +ppa-packages query?
<wallyworld> oh, no not at all. just mentioning it in case you hadn't noticed
<wallyworld> yes
<wgrant> Right, so you've changed the semantics slightly
<wgrant> But probably not in a way that anyone cares about
<wallyworld> i can't see why it was joining to spph when it didn't need to
<wgrant> And it's now much faster
<wgrant> It did technically need to, for the old definition of the package
<wallyworld> the semantics should be the same
<wgrant> s/package/page/
<wgrant> ... oh, I see
<wgrant> You're right
<wallyworld> since there's a clause that says spr.uploaded_archive = archive.id
<wgrant> So it joined Archive against SPR.upload_archive anyway?
<wallyworld> yes, so it seems
<wgrant> Right, so that SPPH join is completely unused?
<wallyworld> as far as i can tell
<wallyworld> but it's there to eliminate spr that haven't been published?
<wgrant> Ah, yes, of course
<wallyworld> so i did it as a subquery
<wallyworld> so the semantics should be the same
<wgrant> Right, I see the subquery now
<wgrant> Missed it because of the indentation
<wgrant> Um, well
<wgrant> It's slightly different
<wallyworld> it was quicker?
<wgrant> Heh
<wgrant> heh
<wgrant> heh
<wgrant> The subquery one is still running...
<wallyworld> oh :-(
<wallyworld> i would have thought postgres would have optimised that
<wgrant> I'm not sure why it didn't here.
<wgrant> But rewording it as an EXISTS is a bit cleaner and works fine
<wallyworld> i did it as an exists originally, but thought it too verbose
<wgrant> It's about 30% faster for me
<wallyworld> ok, will change it
<wgrant> using       AND EXISTS (SELECT 1 FROM sourcepackagepublishinghistory spph WHERE spph.sourcepackagerelease = sourcepackagerelease.id AND spph.archive = sourcepackagerelease.upload_archive)
<wallyworld> but is it faster than the original join?
<wgrant> Although the spph.archive constraint is new; it's not in the original SPPH-joining query
<wgrant> Right
<wgrant> 30% faster than the original query for me
<wallyworld> how much faster?
<wgrant> Trying with pathological cases now
<wallyworld> ok
<wgrant> 35ms vs 24ms
<wallyworld> the oops should it taking a lot longer
<wallyworld> showed
<wgrant> Well, I'm sort of the optimal case for the query
<wallyworld> try with fta
<wgrant> 14s -> 8s with fta
<lifeless> to slow :)
<wallyworld> well, it's still almost 50% improvement :-)
<wallyworld> and thats on DF
<wgrant> prod won't be significantly faster for this
<wallyworld> might be ok on prod
<wallyworld> i'm not sure hoe else to make it faster
<wgrant> Well
<lifeless> wallyworld: yes its a lot better, but prods cpu isn't /that/ much better than dogfoods; once you're in cache...
<wgrant> The general approach for this sort of problem is to not be stupid in the first place
<wgrant> Sadly we have history
<wgrant> So we can't really decide to not be stupid in the first place
<wgrant> But, in general, trying to calculate this sort of stuff out of the live tables is a terrible idea.
<wallyworld> agreed
<wgrant> We could perhaps redesign the page
<wgrant> Such that it returns recent PPA publications created by the user
<StevenK> Why do you need to chain through SPR anyway?
<wallyworld> i'm not sure of the requirements or history of what's required
<wallyworld> StevenK: not sure, that's how it was
<wgrant> It's not chaining through SPR
<wgrant> It's entirely based on SPR
<StevenK> Ew
<wgrant> It doesn't care about publications, beyond the "has this been published?" check to get around rejections
<wallyworld> with the CPU issue, that assumes the query will be in cache, but i would have thought this page would nearly always be loaded cold
<wgrant> The involved tables total only 10GB or so
<wgrant> And some of that is probably (hopefully) TOASTed
<StevenK> I wonder if SRF and batching will help
<wgrant> No
<wgrant> SRF helps only if we can calculate an ordered batch substantially more quickly than we can calculate the entire set
<wgrant> Which requires sensible schema design and indexing
<wallyworld> i may as well land this small change till we figure out what to do next
<wallyworld> since it helps as is
<wgrant> No
<wgrant> It probably doesn't help significantly, and it may regress other cases
<wgrant> It's liable to turn the subquery into a 4-6s seqscan if the wind blows the wrong way
<wallyworld> what would cause that to happen?
<wgrant> If it decides that there will be too many SPRs to efficiently use an index lookup
<wallyworld> but don't we batch so that we only ask for 75 or so at once?
<wgrant> We only ask for 75 at once, sure
<wallyworld> i guess it still needs to know the number
<wgrant> But it's an ordered set
<wgrant> And the order is sufficiently complex and layered that it can't be indexed
<wallyworld> yeah, sadly
<wgrant> So it has to calculate the entire set
<wgrant> Sort it
<wgrant> Then take the first 75
<wallyworld> are you sure it could become a seqscan?
<wgrant> Yes, I've seen it happen here with fta once
<wallyworld> and the join avoids that?
<wgrant> Somehow, yeah
<wgrant> The join has to be applied after the conditions
<wgrant> So for the join it doesn't have to optimally determine the condition order to work out numbers
<wallyworld> above you said for fta it went from 14->8
<wallyworld> and he is a bit of a corner case
<wgrant> And then it went to 16 or so a couple of times, as it chose a different plan for reasons which are unclear
<wallyworld> ah, bollocks, ok
<wallyworld> i could use a CTE perhaps
<wallyworld> to narrow down the spr records
<wallyworld> before checking for publiscation
<wallyworld> wgrant: is this any faster? https://pastebin.canonical.com/77411/   (when you have a moment)
<wgrant> I suspect not, but let's see
<wgrant> still going...
<wallyworld> :-(
<StevenK> wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/sensible-superseded-by/+merge/132009
<wallyworld> maybe i need to add an array column to spr - "archives_published_in"
<wallyworld> that would eliminate the join
<wgrant> That wouldn't help significantly.
<wgrant> It would eliminate the seqscan, but leave the big performance issue
<wallyworld> any other ideas?
<wgrant> I'm trying to use window functions to do it
<wallyworld> that don't involve a schema redesign
<wallyworld> StevenK: typing a blueprint name seems unfortunate.
<wallyworld> maybe a picker?
<StevenK> wallyworld: So you missed that part of the call when Curtis said don't write a picker?
<wallyworld> must have
<wallyworld> sometimes words cut out
<wallyworld> or whole sentences with curtis speaks
<wgrant> Hm
<wgrant> Curtis did say that :)
<wallyworld> i would hate to type a blueprint name
<StevenK> Copy and paste, hit continue
<StevenK> Move on
<wallyworld> guess so. seems very primative
<StevenK> wallyworld: Better or worse than a dropdown with 6200 items?
<wallyworld> that's why a search based solution like a picker is best
<wallyworld> not sure why it was rejected
<StevenK> Oh good god https://twitter.com/raywert/status/263102070989680640/photo/1/large
<wallyworld> StevenK: in validate, fetch the spec and put it in the data map. then do not fetch it again in the submit
<StevenK> wallyworld: I wasn't sure if I could do that.
<wallyworld> StevenK: yeah, the data dict is just passed arounf during the submit
<wallyworld> run the tests just to be sure your implementation is all good
<StevenK> I've not run all the blueprint tests, but xx-superseding{,-within-projects}.txt pass
<wallyworld> that should be enough to send to ec2 with
<StevenK> Pfft, I was going to play buildbot bingo
<StevenK> I already lost once today, though
<wallyworld> StevenK: also
<wallyworld> 192	+ if result is None:
<wallyworld> 193	+ return result
<wallyworld> rs.one() should be enough
<wallyworld> i think?
<StevenK> I thought that would die horribly?
<wallyworld> let me check the code
<wallyworld> yes, rs.one() works
<wallyworld> it returns None if rs is empty
<wgrant> It returns None, the sole value, or raises an exception
<wgrant> For 0, 1 and >1 results respectively
<StevenK> Yeah, I've changed it and checked it in iharness
<wallyworld> StevenK: r=me, gotta pick up kid from school
<wgrant> blah
<wgrant> PARTITION BY seems to always want to traverse the whole set
<wgrant> I guess it's not smart enough to realise that the inner and outer sorts match
<wgrant> eg. SELECT id, purpose, row_number() OVER (PARTITION BY purpose ORDER BY id) FROM archive ORDER BY id LIMIT 5; should just be able to walk down the id index until the fifth row
<wgrant> But it actually grabs the full table due to the PARTITION :/
<wgrant> Ah, it wants them sorted by purpose, id
<wgrant> Perhaps it doesn't want to have to remember the latest row_number for each purpose
<wgrant> So there's no efficient way to do the DISTINCT ON on the server
<wgrant> The quickest solution is probably to ask for a reasonable number and do the distinct in Python :/
<wallyworld> wgrant: so removing the distinct on from the query will make it fast?
<wgrant> wallyworld: Well, with an index on (creator, dateuploaded DESC, id)
<wgrant> You can then do a direct indexed query
<wgrant> If you ignore the DISTINCT ON bit
<wgrant> The page tries to only show the latest SPR for each (distroseries, archive, sourcepackagename)
<wallyworld> i guess i could iterate the result set till i have the requisite number of records
<wgrant> And that is the bit that is slow to implement in SQL
<StevenK> Bleh, sinzui didn't use rollback
<wallyworld> s/in SQL/in Postgres
<wgrant> We can do that using DISTINCT ON or a rank() with PARTITION BY, but both of those require that it be sorted by the partition/distinct first
<wgrant> s/in SQL/in SQL in postgres/, but yes
<wallyworld> so i assume without checking yet that we don't have the index
<wgrant> We don't yet, indeed
<wallyworld> would adding it make the distinct faster?
<wgrant> No
<wallyworld> ok, we can add it live too then i think
<wgrant> Indeed
 * StevenK tries to work out how to QA r16207
<wallyworld> oh well, thanks for experimenting with it
<wallyworld> too bad it doesn't want to play nice
<wgrant> StevenK: librarian, codehosting, regret that you didn't wait until we had builders
<wgrant> basically
<StevenK> wgrant: So push and pull a branch, but I'm not sure how to fiddle with the librarian
<wgrant> click link to librarian file
<wgrant> see if link to librarian file works
<wgrant> upload file to l[ibrarian
<wgrant> see if librarian file works
<StevenK> Bleh, can't push to lazr.restful on qas due to bzr: ERROR: Server sent an unexpected error: ('error', 'NotBranchError', 'Not a branch: "chroot-67793680:///+branch-id/43364/".')  which I guess is stacking
<wgrant> +junk is your friend
<StevenK> Excellent, all four look good
<enginespot> Hi everyone
<enginespot> how to get ppa from a project
<nigelb> enginespot: Projects don't have PPAs. People do.
<wallyworld> wgrant: do you think that without the distincts, it's best to do the subquery for the spph check or stick to a big join?
<wgrant> wallyworld: You'll need to do the subquery
<wallyworld> cool, thanks. that's what i've done
<wallyworld> just wanted to check
<wallyworld> i also need an index on maintainer
<wallyworld> since another query filters on that
<deryck> rick_h_, you actually on irc?
<rick_h_> deryck: rgr
<deryck> rick_h_, there's a hacking room all the way at the back of the hall.  right side of the hall as you go back.
<deryck> rick_h_, in case you need a place between sessions.
<rick_h_> deryck: cool thanks. hacking out in the main area from yesterday atm
<deryck> rick_h_, ok, might come say hello when I get coffee and chat through a couple things.
<adeuring> good morning
<rick_h_> morning adeuring
<deryck> morning, adeuring
<adeuring> hi ric, deryck
<StevenK> dpm: O HAI
<StevenK> dpm: Can you look at production in terms of raring translations?
<dpm> StevenK, hey people at UDS say hi, you were on the projector
<StevenK> Haha
<dpm> ajmitch says hi, and go to sleep :)
<StevenK> Pft, it's only 10pm
<StevenK> He can go to sleep.
<nigelb> lol
<dpm> StevenK, raring translations look good to me in production - only the contributors column has gone down in terms of number of contributors for each language comparing q to r - any ideas why that could have happened?
<StevenK> None, I'm afraid
<ajmitch> StevenK: I'm hardly going to go & sleep at noon
<StevenK> ajmitch: Pft. You fail at jetlag, then.
<dpm> StevenK, so the contributor numbers is the only issue I see, otherwise translations look good. Could you investigate what caused the decrease in number of contributors?
<jtv> sinzui, are you here?
<sinzui> I am
<jtv> Hi!  Long time no see.
<jtv> sinzui: Laura mentioned that you'd been busy opening translations, and were having some trouble with Blender's uploads.
<jtv> (As two separate issues)
<sinzui> blender was broken before we started opening
<jtv> The two things I wanted to ask are:
<jtv> (1) Need any urgent help with the blender issue?  I may even be able to meet up physically with them in the coming days or weeks.
<jtv> (2) Maybe we should try skipping the whole translations-copying step for S.
<sinzui> please help with blender, the users do not understand lp -- they are not configuring translations as I suggest and they do not believe that open source communities will hate them if they setup a separate bug tracker
<sinzui> translations for raring are in qa now. We might enable them in 12 hours
<sinzui> jtv, this is the first opening where there there no errors in the db or scripts for translations...
<sinzui> a non-event that we don't know how to document
<jtv> sinzui: that is fantastic â documentation should just show date done, steps taken & time spent.  The way we always hoped it would become.
<jtv> Can you give me a quick summary of what's wrong with the blender setup?
<sinzui> 1 they did not setup the right series.
<sinzui> 2. they did not setup the branch form the series
<sinzui> 3. yesterday, they had still not set the sync to import templates
 * jtv looks at blender's series
<sinzui> yes, we both can do it, but I think the suer should do it show that he knows ho to change it
<jtv> I agree.
<sinzui> and he has set pots and pos I see
<jtv> Ouch â a separate âtranslationsâ series
<sinzui> but still note mplates
<sinzui> still no templates
<sinzui> but I don't know when this change was made
<jtv> sinzui: this is https://launchpad.net/blender ?  I see a template.
<sinzui> jtv https://translations.launchpad.net/blender/2.6x/+templates has no templates, which is what he was trying to do
<jtv> Ah!
<sinzui> oh, the tree is scons an deep
<sinzui> is this intl tools?
<jtv> But there's nothing in the upload queue waiting for review.
<sinzui> I don't see any pots or pos
<jtv> I'm not finding it either.
<jtv> Maybe they want to get Launchpad to extract the pots.  But are we still running that?
<jtv> And it doesn't look as if strings are even marked for translationâ¦
<sinzui> jtv: http://wiki.blender.org/index.php/Dev:Doc/How_to/Translate_Blender
<jtv> Ah, I was getting the branch
<jtv> sinzui: not having much luck running their toolsâ¦  but it doesn't sound as if there's very much I can do to help.  :/
<sinzui> yuck
<sinzui> I think this is a case where we advise them to stick with the import of translation branch, and work closer with the upstream community
<jtv> Wait... these aren't the upstream people?
<sinzui> jcsackett, would you have time to review https://code.launchpad.net/~sinzui/launchpad/hide-question-comment/+merge/132177
<jcsackett> sinzui: sure.
<jcsackett> sinzui: r=me. thanks for the fix. :-)
<sinzui> jcsackett, how goes bug 164530
<_mup_> Bug #164530: User translations showing broken links <404> <lp-translations> <oops> <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/164530 >
<jcsackett> sinzui: it died in ec2 when i sent it out to land on thursday. i sent it out to ec2 again earlier today when i caught up with email and realized it had never landed and i had no ec2 results.
<sinzui> :(
<jcsackett> sinzui: it's been out for 3h45m. should get a (hopefully) ok result soon, so barring pqm issues it will be qa-able before i EoD.
 * jcsackett knocks on all available wood surfaces
<sinzui> jcsackett, baring that, I would run all lp.translations tests and submit if they all pass
<sinzui> jcsackett, what bugs are you looking at now?
<jcsackett> sinzui: bug 798954
<_mup_> Bug #798954: InvalidProductName: Invalid name for product: bookmark:galapagos.  <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/798954 >
<jcsackett> we discussed it the other day, i'm trying to replicate the error now.
<sinzui> hmm
<sinzui> jcsackett, Do we have any modern oopses
<sinzui> jcsackett, https://oops.canonical.com/oops/?oopsid=OOPS-1df2ed046d05000215e8d42933e44934
<jcsackett> sinzui: what method are you using to search the OOPS DB? you seem *much* faster at it than me.
<sinzui> jcsackett, I was already 2F logged in. I went to the lp production page, opened the latest report and search for InvalidProductName, got nothing, then url hacked to the day earlier, and repeated the search
<jcsackett> sinzui: ah.
<sinzui> chromium also remembers the search so I only type it once
<sinzui> I think I get these often when I forget how to push to lp://qastaging/ , but I don't see the oopses
<jcsackett> it's easy to reproduce manually on dev; i just had to spend some time getting codehosting working in my lxc.
<sinzui> Maybe I should never switch to lxc
<sinzui> this oops might be more interesting. https://oops.canonical.com/oops/?oopsid=OOPS-a3ef3a2868f310958e03998c18758fdb I don't think the user ever figured our how to push a branch
<jcsackett> sinzui: does look like they were having problems.
<sinzui> wgrant, https://bugs.launchpad.net/launchpad/+bug/408585
<_mup_> Bug #408585: choosing blueprint for branch is broken <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/408585 >
* StevenK changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~200
<StevenK> wgrant: http://pastebin.ubuntu.com/1319322/
<StevenK> wallyworld, sinzui: You two coming back?
<sinzui> I am not. got eat
<wallyworld> back where?
<wallyworld> sorry, i thought cal lhad finished
<StevenK> The server kicked everyone but wgrant off
#launchpad-dev 2012-10-31
<StevenK> wgrant: Shall I put a deployment together?
<wgrant> StevenK: I was just looking to see what our state was. Please do.
<wallyworld> wgrant: i've been reading some postgres tuning stuff and it seems it's best to create separate, single column indices to aid sorting, rather than a multi-column index. do you agree?
<wgrant> wallyworld: Where'd you read that?
<wgrant> It is nonsense :)
<wallyworld> i'd have to find the lin kagain
<wgrant> You can't sort using a composition of multiple indices -- that doesn't make sense
<wgrant> It has to be a single composite index
<StevenK> wallyworld: History -> Recently Closed Tabs ?
<StevenK> If using Firefox
<wallyworld> there's a lot of recently closed tabs
<wgrant> Indices are usually b-trees, and it's a bit difficult to aggregate b-trees usefully :)
<wallyworld> i can't recall which one
<wallyworld> in this case, we need to sort on distorseries, spn, archive, date, person
<wgrant> Why?
<wgrant> Which query's this?
<wallyworld> where person is creator or maintainor or neither
<wallyworld> because we need the grouping
<wgrant> Right, but think about how you'd follow indices to get what you need
<wallyworld> so the batch iteration can figure out the distinct sprs
<wgrant> You should quickly realise that that index is entirely inappropriate for it
<wgrant> We have a person, and want want the n most recent uploads
<wgrant> So the first key has to be the person
<wgrant> Following that must be the overall sort key
<wgrant> So we have (creator, dateuploaded DESC, id)
<wallyworld> sure, i didn;t say about that would be the index order
<wgrant> Ah
<wgrant> Why do we need to sort on distroseries, spn, archive?
<wallyworld> but if we need to sort on those columns, they would need to be in the index somewhere, no?
<wgrant> We only need that if we do the distinct in postgres
<wallyworld> for the frouping
<StevenK> wgrant: Tempted to nail bug 779367 shut
<_mup_> Bug #779367: spurious failure in test_builder.TestSlave <critical-analysis> <spurious-test-failure> <test-system> <Launchpad itself:Triaged> < https://launchpad.net/bugs/779367 >
<wallyworld> grouping
<wgrant> But we can't efficiently do the distinct in postgres
<wgrant> StevenK: Indeed
<wallyworld> wgrant: if we need to iterate the batch, we need the grouping
<wgrant> wallyworld: howso?
<wallyworld> so that all the 'like' records are together in the batch
<wgrant> Postgres requires that, but a Python implementation would not
<wgrant> We're doing it in Python precisely because postgres can't do it efficiently
<wgrant> If we sort by the grouping, then we have to traverse the whole set anyway
<wgrant> So the index is pointless
<wallyworld> so we could have a (archive, date) at the start, and an (archive, date) at the end of the query
<wallyworld> and we would not know they were for the same archive
<wallyworld> since they would be in different batches
<wgrant> Ah, indeed, it's batched, forgot that
<wallyworld> so sadly we need to group
<wgrant> To do the batching we will have to evaluate the whole thing in Python, either way
<wgrant> We cannot group at the DB level.
<wallyworld> so i wil have to iterate the entire result set
<wgrant> Which means that batches after the first will be inefficient, but there's not much we can do
<wgrant> You don't have to iterate the entire resultset
<wgrant> You only have to iterate until you find 75 that are distinct
<wallyworld> which could be everything
<wgrant> It could be, yes.
<wgrant> But it's not very likely
<wallyworld> also, the creator or maintainer is optional
<wgrant> How's it optional?
<wgrant> +ppa-packages has a person as context
<wgrant> Therefore it always has a creator
<wallyworld> sure, but the other ones don't
<wallyworld> there are 3
<wgrant> They're all pages under :Person
<wgrant> So they will always have as creator or maintainer
<wallyworld> but the same metyhod is used elsewhere
<wallyworld> with no creator or maintainer
<wgrant> It presumably has some other context, though
<wallyworld> but it will be slow then
<wgrant> In which other contexts is it used?
<wallyworld> getLatestMaintainedPackages
<wallyworld> it's also on the person page
<wallyworld> it sets uploader_only - false
<wallyworld> so no creator or maintainer is added to the query
<wgrant> But it's on Person, so it presumably has some kind of person filter
<wallyworld> not that i can see unless i missed something
<wgrant> Well, it would otherwise be global
<wgrant> Which is entirely nonsensical
<wgrant> It is called in three modes: maintainer, non-PPA uploader, and PPA uploader
<wallyworld> i just checked, i think there is an extra condition added in, but just not there
<StevenK> https://code.launchpad.net/~stevenk/launchpad/undisable-tests/+merge/132235
<wgrant> uploader_only=False and ppa_only=True would result in an unbounded set, but that's never used
<wallyworld> so it does seem it's always creator or maintainer in the query
<wgrant> Indeed
<wgrant> It doesn't really make sense for a method on Person to not constrain by person :)
<wallyworld> true
<wallyworld> so, what would a reasonable batch size be? 75?
<wgrant> That tends to be the default
<wallyworld> ok, will try that. we'll have to test with known pathological cases
<wgrant> Well, this will be exposed to a new variety of pathological cases
<wallyworld> yeah, sadly that may be the case
<wgrant> It may almost be worth denorming this, but the workaround for these pages shouldn't be too terrible for now
<wallyworld> let's see how this goes first
<StevenK> wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/undisable-tests/+merge/132235
<wallyworld> WCPGW
<wallyworld> is it worth looking at WHY they are fragile?
<wallyworld> and perhaps fixing a root cause issue?
<StevenK> Any data we have on these tests is >18 months old, we need to enable them to see what is going on now
<wallyworld> so maybe just an "ec2 test" run to see what's up?
<wallyworld> i fear that they may well pass once, geth through ok, and then fail tomorrow
<StevenK> I'll be tossing it at ec2 land once the MP is approved
<StevenK> wallyworld: And we can disable them if they prove to be doing their old tricks, but then we have current data
<wallyworld> guess so
<StevenK> What I am doing is a Curtis Approved Plan.
<wallyworld> a couple of ec2 test runs would also provide data
<wallyworld> ok
<StevenK> wallyworld: Wrong. buildbot is parallel, and ec2 is not.
<wallyworld> were the failures only parallel? i don't think so
<wallyworld> anyway, r=me
<StevenK> No, but they can be impacted by it
<wallyworld> maybe the stars will align and it will all be ok, let's see
<nigelb> Hi! I have an old lp tree lying around.
<nigelb> Will doing a bzr update and make run "Just Work"?
<nigelb> s/update/pull/g
 * nigelb does rf-get and hopes
<wallyworld> it should work, but you will need to do a make schema before you run
<StevenK> nigelb: rf-get will also grab new sourcedeps and such too
<nigelb> StevenK: Oh awesome.
<nigelb> wallyworld: Noted.
<nigelb> This is gonna take a while. it's at least an year old tree.
<wallyworld> wgrant: here's the mp for that related software work https://code.launchpad.net/~wallyworld/launchpad/ppa-packages-timeout-1071581/+merge/132236
<StevenK> lp.code.model.tests.test_sourcepackagerecipebuild.TestBuildNotifications.test_handleStatus_OK_successful_upload loses
<StevenK> Fails on ec2
<wallyworld> :-(
<nigelb> Apparently it's not very smooth.
<nigelb> I got hit with "ImportError: No module named convoy.meta"
<StevenK> Then your launchpad-dependencies is out of date
<nigelb> I thought rf-get would update it. no?
<nigelb> I'm on 0.119~lucid1
<StevenK> nigelb: rf-get won't update launchpad-dependencies, it's a system package, and rf-get does not have root.
<StevenK> nigelb: Is python-convoy installed?
<nigelb> StevenK: It wasn't. Started install.
<wallyworld> StevenK: archives have a checkArchivePermission method. the lp.View security adaptor checks this, and also checks if the user is subscribed for private archives, is there any reason why the subscription check is not done inside the checkArchivePermission method?
<wallyworld> ie irrespective of any other check permission checks, shouldn't subscribers always have access?
<StevenK> wallyworld: No, P3A subscribers are not allowed to see the archive
<StevenK> They can browse it, but not Archive:+packages
<StevenK> Which is why it's a little strange
<wallyworld> StevenK: so, i'm looking at person+index
<wallyworld> ok, Archive+packages only uses the checkArchivePermission check
<wallyworld> but not the p3a subscription one
<nigelb> woah
<nigelb> it worked I think.
<wallyworld> StevenK: Archive+packages uses lp.View permission, which also does the p3a subscriber check
<nigelb> hrm. did lp upgrade postgres versions?
<nigelb> I have this http://hastebin.com/qepumeveto.vhdl
<wallyworld> not sure, you need 9.1
<wallyworld> rf won't upgrade it
<wallyworld> but launchpad-database-dependencies should i think
<nigelb> if only rf told me.
<wallyworld> i never use rf
<nigelb> ah
<wallyworld> but rf should still work nonetheless
<nigelb> rf did most of the work.
<wallyworld> i haven't looked at the code so cannot really comment
<nigelb> The only things I failed doing was upgrading the dependency packages :)
<wallyworld> well, that's not too bad
<nigelb> Attempt #4.
<nigelb> Gah. Failed at the exact same point. I suspect I need to do some magic to start the right version of postgres. But now, I have to start the day job.
<StevenK> wallyworld: Actually, I might be wrong. We may allow Archive:+packages, but want to forbid subscribers from copying packages out using +copy-packages
<wallyworld> right, that makes more sense
<StevenK> nigelb: Do you have time to pastebin the error?
<nigelb> StevenK: I pastebin'd it earlier. http://hastebin.com/qepumeveto.vhdl
<StevenK> nigelb: You're still using 8.4?
<nigelb> StevenK: I installed 9.1, but I suspect it's not yet running. I dont want to break it just before I start work since I need PG for work.
<StevenK> nigelb: Ah, that could be a problem, since the LP setup scripts sort of destroy other databases
<nigelb> StevenK: I suspected as much :)
<stub> You want to use the LXC environment I think
<nigelb> I'm on lucid > <
<StevenK> Then you want to upgrade to Precise, and create an Lucid LXC for LP and another for $work? :-)
<nigelb> Yeah, that's the eventual plan.
<StevenK> lpsetup in the LP PPA will create a Lucid LXC for LP very easily
<nigelb> The trouble is upgrading.
<nigelb> My harddisk has no more space.
<nigelb> so I need to get a new one first before I upgrade.
<lifeless> nigelb: or delete crap :)
<nigelb> lifeless: I've deleted most of the crap I don't want.
<nigelb> Everything that's left, I do want :)
<StevenK> Oh bleh, everyone knows /usr isn't important
<StevenK> Or /lib
<nigelb> I deleted a ton of Lp branches.
<nigelb> I had all the ones I worked on lying around :)
<nigelb> I'm hoping to get a 1TB harddisk this month.
<nigelb> That should clear up significant amounts of space.
 * StevenK is still waiting for a du to scare nigelb with
<StevenK> steven@undermined:~/launchpad/lp-branches% du -sh .
<StevenK> 63G	.
<nigelb> o_O
<StevenK> Can we destroy delayed copies yet?
<StevenK> Hahaha
<StevenK> lp-oops is broken
<nigelb> so, are lp-oops erros logged with lp-oops-oops? ;)
<lifeless> nigelb: yes
<StevenK> The OOPS is useless
<nigelb> ha!
<StevenK> TemplateDoesNotExist: 500.html
<StevenK> So it's an ISE
<StevenK> Thanks for the content, you useless piece of crap, Django
<lifeless> thats fixed in trunk
<StevenK> I'm sure lp-oops is getting updated
<StevenK> </sarcasm>
<wgrant> wallyworld: The last index is not useful
<wgrant> Well, it would be useful if we wanted more than about 20% of the rows in the query, but we don't
<wallyworld> wgrant: the final query would use it, no?
<wallyworld> since we just order by date, id
<wallyworld> and on the individual pages, we do return all matching rows
<wallyworld> not just the top 5
<wgrant> wallyworld: "all matching rows" == 75
<wallyworld> huh?
<wallyworld> the individual pages load all matching records
<wallyworld> and then batch them
<wallyworld> i think you are referring to the other query to load the ids
<wgrant> We don't ever load all matching records unless we're on the last batch, do we?
<wallyworld> we do - the 3 individual pages for maintained, uploaded etc
<wgrant> They're in batches of 75
<wallyworld> a summary is shown on the related software overview page
<wallyworld> but the whole result set needs to be ordered, right?
<wallyworld> so that the batches can be calculated
<wgrant> But we only need to calculate the batches up to the current position
<wgrant> So if I'm on the first page, I only need to load until I have 75 records to show
<wgrant> I don't need to grab all 30000
<wallyworld> but wouldn't the load be done suing the index
<wallyworld> otherwise how would it efficently know what to load
<wallyworld> since we are odering by date
<wgrant> But we're filtering by creator/maintainer at that point
<wallyworld> so we only grab 75, but we need to know which 75
<wgrant> That's why we have the creator- and maintainer-prefixed indices
<wallyworld> no, not there we arent
<wallyworld> at that point, we are filtering by id in(....)
<wallyworld> we filter by creator etc to figure out which ids to include in the final result set
<wgrant> And that bit is only ever called with 75 items
<wgrant> Because the final set is known by then
<wgrant> We just have to load them
<wallyworld> only the ids
<wallyworld> and not the order
<wgrant> Sure
<wallyworld> so select * from sourcepackagerelease where id in (...) order by date desc
<wallyworld> why doesn't that need an index?
<wgrant> But how can the sort index be used for that?
<wgrant> It's not what needs an index.
<wgrant> It's a matter of what *can* be indexed
<wallyworld> an index on date can be used
<wgrant> How?
<wgrant> sourcepackagerelease has more than 1.5 million rows
<wallyworld> to efficiently communicate the order to traverse in
<wgrant> Indices don't work that way
<wgrant> In any database :)
<wgrant> An index on (dateuploaded DESC, id) is useful for sorting in some situations
<wallyworld> so you sating sorting doesn;t use indices?
<wgrant> But in this case we'd have to traverse 1.5 million index tuples to pull out 75 of them
<wgrant> Which is entirely not worth it
<wgrant> It would never be silly enough to use an index for that sort
<wallyworld> why? we would just traverse the first 75 in order
<wgrant> Huh, how?
<wgrant> We have IDs
<wallyworld> since the index is ordered by date
<wgrant> We know the IDs we want
<wgrant> How can we efficiently look that up in the index?
<wallyworld> well we may have 10000 ids
<wgrant> Even if we have 10000, the density is only <0.01 still
<wallyworld> i guess it could ignore the index and traverse all the ids
<wgrant> If we were to force it to use the sort index, it would have to traverse 1.5 million tuples to find 75
<wallyworld> why?
<wgrant> If it doesn't use the index, it can do 75 index lookups and sort in memory
<wallyworld> the index gices the order
<wgrant> Think about the physical layout of the index
<wgrant> The index allows us to find a tuple based on its dateuploaded *and* its id
<wgrant> Or to traverse tuples in (dateuploaded DESC, id) order
<wallyworld> which is what the query is asking for
<wallyworld> so it just has to walk the index
<wallyworld> 75 records at a time
<wgrant> huh?
<wgrant> We know the IDs of the 75 records that we need
<wallyworld> but not the date order
<wallyworld> so we have 100000 ids but don't know the order to present them in
<wgrant> Sure, but how does "75 records at a time" come into it?
<wallyworld> that's the batch size
<wgrant> We don't have 100000 IDs
<wgrant> We have 75
<wgrant> We partitioned early on, didn't we?
<wallyworld> the rs.countz() may be 100000
<wgrant> Because we read in IDs in date order, until we had 75 unique ones
<wallyworld> no, you are confusing the 2 queries
<wallyworld> we did the limit to 5 unique ones
<wallyworld> nbecause that's what the overview page shows
<wgrant> Then +ppa-packages shows batches of 75 unique ones
<wallyworld> but the individual pages show everything
<wallyworld> yes, out of say 100000
<wallyworld> so it has to know which 75 to load at a time
<wallyworld> so it has to have them ordered
<wgrant> The first query filters by creator or maintainer, and crawls back in time until it finds 75 unique ones
<wgrant> This is always filtering by person, so it can use the creator- or maintainer-prefixed index
<wallyworld> no it doesn't
<wgrant> Then, once it has those 75 IDs, it does a second query to load the SPRs
<wallyworld> it doesn't stop at 75
<wgrant> Oh?
<wallyworld> because the batching is done at the view level
<wgrant> 202	+ if max_results and len(ids) >= max_results:
<wallyworld> yes, max results is 5 and optional
<wallyworld> max results is used for the overview page sunnaries
<wallyworld> to show 5 from each list
<wgrant> Ah, well then this will probably always time out
<wallyworld> but the invidual pages don't use max_results
<wgrant> There's no point having the indexed sort if you're just going to load the entire set anyway
<wallyworld> only the ids
<wgrant> The whole problem with the query is that it ends up loading the whole set
<wallyworld> not the recordds, just the ids
<wallyworld> and then the outer query is batched
<wgrant> That's irrelevant; it's still considering ~100000 more tuples than is required
<wallyworld> so an indidual person can have 100000 ppas?
<wallyworld> because the ids are filtered
<wallyworld> to match the creator/maintainer etc
<wallyworld> only only those matching ones go to be used in the outer query
<wgrant> 100000 SPRs? Sure.
<wallyworld> well, the only alternative it to implement a custom batcher up in the view layer
<wallyworld> which can be done
<wallyworld> wgrant: is the current implementation worth trying? it won't be worse than what's there
<wgrant> wallyworld: It is strictly worse than what's there
<wgrant> We're no longer just asking postgres to consider every related row
<wgrant> We're asking it to return bits of them to Python
<wgrant> In terms of O(disk accesses) it's no worse, but there's an extra constant multiplier there :)
<wallyworld> ok, there's no choice then, but to move the code
<wallyworld> to a batcher
<wgrant> Right
<wallyworld> there will still be issues
<wgrant> Right, there is great potential for pathological cases
<wallyworld> if the batches are not traversed sequentually
<wgrant> Oh?
<wallyworld> if someobe url hacks
<wgrant> I'm not seeing the issue
<wallyworld> the relationship to records loaded to batches in not defined
<wgrant> (though ideally you'd use a derivative of SRF here, which doesn't support URL hacking anyway)
<wallyworld> so it would need to walk the records sequentuially
<wallyworld> instead of using offset and limit
<wallyworld> as one would do for a normal query
<wgrant> Right, that's the problem with an offset-based batcher
<wgrant> Although I guess due to deduping even a range-based duper would have to traverse the whole lot
<wallyworld> yep
<wgrant> Still, this will make the page suck less until we can denorm it properly
<wgrant> And the MP you have is strictly worse than the status quo
<wallyworld> and as someone clicks through the batches, stuff needs to be retained in memory
<wgrant> Not retained in memory as such
<wgrant> But subsequent batches will also have to calculate all previous batches
<wallyworld> or if it is retained, then notr really
<wgrant> How can we retain it?
<wallyworld> i session variable
<wgrant> Hah
<wallyworld> wgrant: so if someone is many batches into scrolling through the table, then each next click presents more and more work for the batcher
<wgrant> Sure
<wallyworld> with a large number of records, it surely will timeout
<wgrant> And that's something we probably have to live with until we fix the schema
<wgrant> It's relatively rare that someone will go deep enough that it will be a problem
<wgrant> And it's completely not worth hacking something into the session for this temporary fix
<wallyworld> i will have to handle Last properly though
<wallyworld> and Prev
<wgrant> They'll work fine as long as the user has a small number of uploads
<wallyworld> why don't i just say fuck it, there's no need to order by date the records on the individual pages
<wgrant> They have to be ordered somehow
<wgrant> Or batching will not work
<wgrant> It's not the order that's the problem; it's the distinct on
<wallyworld> yes, i'm just wonderong out loud if we can avoid that
<wgrant> If we dump the distinct on then the entire problem becomes trivial
<wallyworld> we need the distinct on to shw the latest 5 for the overview page
<wgrant> Why?
<wallyworld> just because
<wgrant> By the current definition of the latest 5 we need the distinct
<wallyworld> that's how it is put together
<wgrant> But by the current definition of +ppa-packages we need it there too
<wallyworld> propably, i ws just wondering out loud
<wallyworld> oh well, custom batcher it is i guess
<wallyworld> wgrant: so what's the "normal" number of ppas a person would have on that page?
<wgrant> s/ppas/packages/?
<wallyworld> yes
<wgrant> Let's see
<wallyworld> cutis has 38 i think
<wgrant> Curtis is at the low end :)
<wgrant> The most is probably about 250k
<wgrant> But that's exceptionally high
<wallyworld> so my implemention was premised on it only being a few 100 at most
<wallyworld> wow 250k!
<wgrant> Your implementation is basically what postgres chooses today, except slower.
<wgrant> The top 100 ranges from 2k to 220k
<wallyworld> ok
<wgrant> It may almost be worth just adding a new table maintained by garbo to make all these pages trivial
<StevenK> And Archive size?
<StevenK> Since I'm still sure you'll murder me for two columns on archive
<wgrant> Well
<wgrant> A similar approach is probably useful there
<wgrant> archive already has a tonne of related stats columns, and I haven't worked out whether they're worth it yet
<wgrant> They probably are
<wgrant> s/worth it/worth having in the main table/
<wallyworld> there's all sorts of stuff that can be denormed in the spr, spph area isn't there?
<wgrant> Those are the two big ones
<wallyworld> i should just finish this branch though
<StevenK> wgrant: It's clear we need a schema change. Even pulling the rows from LFC is 2.8seconds
<wgrant> StevenK: Certainly, we need a schema change
<wgrant> Calculating all this stuff live is insane
<wgrant> It's just not clear what the detail of that schema change is
<wgrant> For +ppa-packages and co it's much clearer and simpler
<StevenK> The two I can think of are Archive.{source,binary}_size or Archive.size as an array
<wgrant> It makes no sense as an array
<wallyworld> maybe i should just do the denorm then
<wgrant> wallyworld: Or decide to not do the DISTINCT ON
<wallyworld> would that matter? we would get dupes then
<wgrant> It would change the definition of the page
<wgrant> It's not clear whether it would be changing it in a problematic fashion
<wallyworld> from what i gather, we are not sure what the page is used for
<wallyworld> or who uses it
<wgrant> We know sort of
<wgrant> Anyway, the page can retain its current definition in a performant manner if we do the denorm, otherwise it probably cannot. It can be made performant without the denorm if we eliminate the distinctness
<wallyworld> and does that "sort of" indicate we couild remove the distinct on?
<wgrant> I don't think removing the distinct on would be a problem
<wgrant> It's probably more confusing than anything else
<wgrant> We've had at least one bug filed about that aspect of the behaviour, although I think it was invalided.
<wallyworld> we would need to ensure we display a distinguishing piece of data for each record
<wallyworld> let's discuss tomorrow
<wgrant> Indeed
<wgrant> The denorm is quite simple, if we opt to do it
<wgrant> Instantaneous updates are not mandatory, so we can just do it in garbo-frequently
<wallyworld> it would be nice if it also has line of sight for other fixes
<wallyworld> StevenK: i replied to your mp comment - do you agree?
<wgrant> Oh, crap, that MP
 * wgrant glances
<wallyworld> lol, you have too much on your plate
<wallyworld> ah, trick or treaters, better get some lollies for them
<wgrant> crazy qlders
<StevenK> I'd still prefer an override policy that backs onto UnknownOverridePolicy
<wgrant> Sure
<wgrant> That's the ideal solution
<wgrant> find_and_apply_overrides should be replaced by override policy calls
<wgrant> But Ian quite reasonably wants to be done before the heat death of the universe :)
<wallyworld> yes, but i thought we agreed that could wait
<StevenK> Like override policies are hard to right :-P
<StevenK> Sigh, write
<StevenK> Didn't I write all of the current ones?
<wgrant> I think so
<StevenK> wallyworld: Then add an XX
<StevenK> XXX
<StevenK> Obviously, typing is HARD
<wallyworld> ok, i'll take another look, maybe it can be done easily
<wgrant> This fix is nearly correct; let's not overcomplicate it now with it so close to done
<wgrant> This is a useful incremental improvement
<wgrant> Which doesn't break anything
<wallyworld> what did i miss?
<wgrant> You shouldn't be overriding the section, just the component
<wallyworld> is there an existing bug for the XXX i would add?
<StevenK> Not sure, but it would be Low
<wallyworld> ah ok, i saw somewhere else was overrding the secion
<wallyworld> why does that other place do the secion but not here?
<wgrant> wallyworld: We want to inherit the section of a previous publication, because for a particularly binary it doesn't usually change
<wallyworld> but not if we get the component from a source publication
<wgrant> eg. libfoo is in libs, libfoo-dev is in libdevel
<wgrant> It's a different dimension to component
<wgrant> component is archive policy, section is just package categorisation
<wallyworld> ok, i have no clue about the data model
<wallyworld> thanks
<wallyworld> i'll fix that bit
<wgrant> It makes sense to inherit the archive policy from the source, but not the package category (since the binaries usually specify a sensible section, and the section differs between binaries)
<wallyworld> ok, makes sense
<wgrant> The code structure is a bit terrible, but it was already awful so I can hardly object on that basis
<wallyworld> i didn't realise section was categorisation
<wallyworld> i tweaked an existing method
<wgrant> Cish libraries are in libs, X stuff is x11, Python libraries are in python, etc.
<wgrant> Right, but that method is called in an existing thing to handle various existing override cases
<wgrant> So it's slightly ugly
<wallyworld> yes, but it provided reusable code
<wallyworld> so i just hijacked it a bit
<wgrant> Right
<wallyworld> keeps loc count low :-)
 * wgrant needs to head out for a while now... hopefully StevenK can approve it, otherwise I can in a bit
<wallyworld> thanks again
<wallyworld> wgrant: going trick or treating :-)
<wgrant> Heh
<wgrant> Not since I lived in Canada :)
<wallyworld> it's getting bigger here
<wallyworld> i don't mind it cause it's pagan and celtic in orign
<StevenK> And it helps you're a Scrooge who hates Christmas :-)
<wallyworld> yep, that's me
<wallyworld> i hate anything to do with organised religion
<wallyworld> ewspecially since they stole Christmas from the pagans
<wallyworld> cause it was originally Saturnalia
<adeuring> good morning
<deryck> Morning, adeuring
<adeuring> hi deryck
<czajkowski> cjwatson: https://bugs.launchpad.net/launchpad/+bug/1073492  is this down to some of the work you've done recently ?
<_mup_> Bug #1073492: Sync changelog doesn't include all changelogs between release version and new Debian version <derivation> <Launchpad itself:New> < https://launchpad.net/bugs/1073492 >
<tumbleweed> czajkowski: that's not a launchpad bug
<tumbleweed> oh, it's not what I thought it was
<czajkowski> :)
 * tumbleweed is glad to see syncpackage getting it right, for a change  :)
<cjwatson> czajkowski: not sure, would probably need to spend more concentration than is typically available during a conference to figure that out
<cjwatson> czajkowski: I don't recall changing anything around there *intentionally*
<jcsackett> sinzui: can you review https://code.launchpad.net/~jcsackett/launchpad/invalid-product-names/+merge/132405
<sinzui> I will
<jcsackett> thanks.
<sinzui> jcsackett, r=me
<jcsackett> sinzui: awesome thanks.
<sinzui> jcsackett, you might want to look at bug 1055751 again
<_mup_> Bug #1055751: Permission denied: "Cannot create '+filediff' viewing diffs on +branch urls <403> <codebrowse> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1055751 >
<sinzui> I think the issue is in translatePath or BranchNamespace where we do not expand or lookup the aliased name
<jcsackett> sinzui: ok, happy to take another look at it.
<jcsackett> i've got some distance on it now and a bit more knowledge of the domain. :-P
<sinzui> jcsackett, We get the error when users are browsing the series alias rather than the 3-part/5-part names that the code you were looking at expects
<sinzui> eg lp:launchpad -> lp:~launchpad-pqm/launchpad/devel
<sinzui> jcsackett, I am not feeling well, and I have the added stress of children putting on Halloween costumes. I will send and email summarising my effort to be productive to the squad
<jcsackett> sinzui: dig.
<diptanuc`> Hi Guys
<diptanuc`> Just saw that you guys use Go to power services at canonical. Was interested to talk more about what you do with it.
<sinzui> diptanuc`, Launchpad doesn't use Go. JuJu and related cloud tools uses it.
<diptanuc`> sinzui: Oh i see, who is the right person to talk to?
<diptanuc`> Is there a channel where they hang out?
<lifeless> diptanuc`: gustavo niemeyer would be a good choice, in #juju
<czajkowski> cr3: ping, trying to work out your issue on https://answers.launchpad.net/launchpad/+question/212793
<cr3> czajkowski: thanks, I'm very curious to understand how to get checkbox-ihv to look like certify-web
<wgrant> cr3: Read the paragraph at the top of +sharing
<wgrant> On both projects
<czajkowski> cr3: I cant see the-ihv but as wgrant says , if you read that area on the +sharing it should become clearer.
<cr3> wgrant: yeah, they look different. should I have asked for a proprietary project instead of a private one?
<wgrant> cr3: You created an open source project
<cr3> wgrant: my mistake, it seems I should've created a proprietary one :(
<wgrant> You probably wanted to select Other/Proprietary as the license, which will enable to you use commercial features
<wgrant> You can change it on +edit
<czajkowski> cr3: is it actually properietary though
<czajkowski> or is it thats how you think it should be set up for sharing ?
<cr3> czajkowski: it's actually proprietary
<czajkowski> ok
<wgrant> Right, proprietary features tend to be easier to enable when you configure your project as not just another random open source project :)
<wgrant> cr3: Once you've set it to Other/Proprietary you'll get a 30 day trial commercial subscription, and you can set stuff on +sharing
<wgrant> You'll need to poke someone (or RT) to get the subscription extended
<czajkowski> aka me via commercial@lp.net
<wgrant> Ah yes, that's the address
<czajkowski> wgrant: care to explain https://answers.launchpad.net/launchpad/+question/212880  to me ?
<cr3> czajkowski, wgrant: thanks folks, I'll have another look into this tomorrow
<czajkowski> cr3: np
<wgrant> czajkowski: Looking
<czajkowski> wgrant: thank you
<wallyworld> StevenK: i've updated the code to add the XXX etc
<StevenK> wallyworld: I was expecting the XXX to be just above the if block you added
<wallyworld> StevenK: well, it's the whole method than needs to be refactored
<StevenK> Then why is the XXX half-way through it?
<wallyworld> it's in the doc at the top
<StevenK> Well, the hack is in processUnknownFile
#launchpad-dev 2012-11-01
<wallyworld> but the whole method can be refactored since it duplicates the override stuff
<wallyworld> not just for unknown files
<wallyworld> the policy can be got : override_policy = self.policy.archive.getOverridePolicy()
<wallyworld> and then the adaptor can be used
<StevenK> wallyworld: Approved
<wallyworld> thans
<StevenK> wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/send-host-header-during-getremotebug/+merge/132447
<wgrant> StevenK: The Host header doesn't work like that
<StevenK> wgrant: Yes it does? We need to instruct the remote site which one we're talking to.
<wgrant> +        return {'User-agent': LP_USER_AGENT, 'Host': self.baseurl}
<wgrant> I spy with my little eye a contradictory statement
<wgrant> (host != URL)
<StevenK> Bleh
<StevenK> wgrant: Changes pushing, thanks.
<StevenK> Diff updated.
<wgrant> Thanks
<wallyworld> wgrant: since you ran the queries on DF, if you get a moment at some point, could you look at this for me? https://code.launchpad.net/~wallyworld/launchpad/person-index-timeout-931771/+merge/132446
<wgrant> wallyworld: Sure
<wallyworld> thanks, no hurry
<StevenK> Blink
<StevenK> db-devel died
<wallyworld> what?
<StevenK> subunit corruption, looks like
<wallyworld> \o/
<wallyworld> wgrant: can tables be created live with no locks? i can't see why not
<wgrant> wallyworld: Only if they have no foreign keys
<wallyworld> it does, but the table will be empty
<wgrant> Adding a foreign key creates an internal trigger on both ends, which requires an access exclusive lock
<wallyworld> surely only for a micro second
<wgrant> Right, as with most ALTER TABLEs it's basically instantaneous
<wallyworld> but we still cannot create live?
<wgrant> Not without delaying things for maximum_txn_length * 2
<wallyworld> it would be no worse than may of our transactions
<wgrant> the ALTER TABLE will request an exclusive lock on the table
<wgrant> this will wait around until all existing transactions finish
<wgrant> the pending, ungranted lock will block any new transactions that want to use the table
<wgrant> Once all the existing transactions finish, the lock can be taken and then released
<wgrant> But we have lots of long transactions, so this could block everything for minutes or even longer
<StevenK> And given FDT is not really an issue
<wallyworld> ok, FDT it is
<StevenK> What new table?
<wallyworld> for recording the latest published sources
<wallyworld> source packages
<StevenK> BTF for source packages?
<wallyworld> BTF?
<StevenK> BugTaskFlat
<wallyworld> well, not really
<wallyworld> but it is a denorm of the data model for reporting
<wallyworld> it will be maintained by a garbo frequently job initially
<wallyworld> the business logic changes to consume it are trivial, and actually reduce LOC I think
<stub> wallyworld: You are missing foreign key definitions in that MP. Is that deliberate or an oversight?
<wallyworld> stub: deliberate
<wallyworld> it's only a reporting table
<wallyworld> so i didn't want to do anything to slow down performance
<wallyworld> AFAIK, BugTaskFlat has no FKs also
<wallyworld> bt i could be wrong
<stub> Yeah, but it is maintained entirely by triggers to enforce referential integrity
<wallyworld> i'm happy to add them if they are required
<wallyworld> do the triggers even check for missing references?
<stub> wallyworld: They are not required, but leaving them out is a potential foot gun
<wallyworld> eg with BugTaskFlat, what happens if a referrenfed object is deleted?
<wallyworld> i don't think that is caught
<stub> It should be
<wallyworld> should be or is?
<wallyworld> i'll add the FK references to the mp though
<stub> it should be, as that is what was supposed to happen when I reviewed it IIRC
<wallyworld> o
<stub> Otherwise we would be giving out incorrect results (the footgun)
<wallyworld> ok
<stub> So here for instance, missing FK to the person table could mean packages get lost when a person merge happens.
<wallyworld>  so i just need to update the table definition to say REFERENCES foo, nothing else?
<stub> Declaring them 'on delete cascade' may be useful, or may be a big problem
<stub> wallyworld: yes
<stub> oh, haven't finished looking at it yet :)
<wallyworld> ok
<stub> wallyworld: Yeah, all looks fine apart from the FK issue
<wallyworld> stub: what about the archive.purpose column?
<wallyworld> which is an enum
<wallyworld> do i need a fk there too?
<stub> wallyworld: If they are a problem, we can leave them out but we have to think hard about potential problems
<stub> wallyworld: no
<wallyworld> bah, archive_purpose
<wallyworld> just about to push changes
<wallyworld> thanks for looking
<stub> And now I read your statement about FKs in the MP :-/
<wallyworld> stub: changes pushed
<wgrant> stub, wallyworld: That table definitely needs foreign keys
<wallyworld> they are there now :-)
<wgrant> BugTaskFlat doesn't, since it's maintained entirely inline by triggers on FKed tables
<wallyworld> so i will need to update the person merge code when i do the devel work
<wgrant> And there's a one-to-one relationship between BugTask and BugTaskFlat
<wgrant> wallyworld: Possibly
<wgrant> wallyworld: Person merge will crash (and tests will fail) if you add a new person FK that's involved in a unique constraint
<stub> oh, archive_purpose needs to remain in sync with Archive.purpose?
<wallyworld> yes
<wgrant> Archive.purpose doesn't change
<wgrant> wallyworld: Should dateuploaded and publication be NOT NULL?
<wgrant> Also, that table name is far too generic
<wallyworld> wgrant: publication is a reference to SPPH
<wgrant> wallyworld: Sure
<wgrant> It still needs to be NOT NULL
<wallyworld> oh, i meant to put not null
<wgrant> It also can't be the PK
<wgrant> As it's mutable
<wallyworld> i thought primary keys were automatically not null
<wgrant> Well, it could be the PK, but Storm will kill you in your sleep if you try to change it
<wallyworld> ah, right. i was thinking the pib reference would not change
<wallyworld> publication i mean
<wallyworld> but it can of course
<wallyworld> i'd add a surrogate key
<wgrant> The natural key is (person, archive, distroseries)
<wallyworld> which person?
<wgrant> That's the question
<wallyworld> creator?
<wgrant> I'm not quite sure that this schema does what we need; we'll probably want a complete garbo+browser implementation reviewed before this landss
<wallyworld> i did the natural key as spn, archive, distroseries
<wgrant> Er, yeah, spn needs to be in there as well
<wgrant> But person obviously has to be too
<wallyworld> not obvious to me :-)
<wgrant> Hm?
<wallyworld> since i don;t know soyuz that well
<wgrant> Why not?
<wgrant> We care about the person's latest package
<wgrant> s
<wallyworld> i thought only one person would publish a spn, archive, distroseries
<stub> http://paste.ubuntu.com/1323048/ enforces the purpose can't get changed on archive
<wgrant> wallyworld: Only one person's publication will be current, sure
<wgrant> But this shows superseded ones too
<wallyworld> thanks stub, will add that
<wallyworld> ah, ok
<wallyworld> so creator i suppose is the person to use
<wgrant> Well
<stub> http://paste.ubuntu.com/1323050/ is probably better, or it will be a PITA to change the purpose of an archive
<wgrant> It probably needs to be both
<StevenK> stub: We never want to change the purpose of an archive.
<StevenK> It is by all accounts immutable
<wgrant> Right, archive purposes cannot change without manual SQL
<wgrant> I don't think one has ever changed
<wgrant> It doesn't make sense
<wallyworld> stub: is the preference to add constraints inline or use an alter table?
<wgrant> If one wants to change, we have bigger problems
<stub> So probably no need for the extra index and multi column FK constraint
<wgrant> This is a cache table that is cheap to regenerate and unproblematic if it breaks
<stub> wallyworld: technically it makes no difference. I find it all inline more readable myself.
<wallyworld> np, thanks
<wallyworld> stub: with the comments, there's a comments.sql - does that get rolled up at some point?
<StevenK> comments.sql no longer gets applied to production, either
<deryck> czajkowski, did the lp session get dropped today?
<stub> wallyworld: The comments should go in comments.sql sorry.
<czajkowski> deryck: it did, there was no other sign ups and didnt want to waste the lP dev time
<wallyworld> stub: ok, i saw some other patches contained them, so wasn;t sure
<deryck> czajkowski, ah, ok.  Thanks for putting those sessions together.
<czajkowski> no worires thanks for coming ;)
<stub> yeah, and your question made me question that.
<wallyworld> stub: i thought it might be like patches - at some point, a rollup is done if you know what i mean
<wallyworld> so add everything to the patch, incl comments
<stub> yeah, but we haven't written any code to do that rollup
 * stub thinks
<wallyworld> ok, will just add to comments.sql then
<stub> wallyworld: nah
<stub> wallyworld: leave them where they are. The rollup will happen
<wallyworld> ok
<stub> I should apply comments.sql to production and then nuke it so I don't confuse myself again
<wallyworld> wgrant: i'll add both creator and maintainer to the natural key for now and we can rethink before landing
<wgrant> The schema will likely need to evolve once we discover that the code cannot use it
<stub> reading scrollback, the natural key is complex enough to warrent just using an 'id' serial column
<wallyworld> yeah, adding
<wallyworld> i never use natural key as pk
<wallyworld> i only ever add a constraint to enforce it
<wallyworld> i mistakenly thought i could use the publication reference as the pk, thinking there was a 1-1 relationship
<wgrant> This table will never be referenced in an FK, so a natural key makes a reasonable amount of sense, but it's not going to run off the end of a 32-bit int any time soon, so it doesn't matter much either way
<wgrant> But we should avoid surrogate keys when we can :)
<wallyworld> i disagree :-)
<wgrant> Ah, so it was you who designed branchrevision :P
<wallyworld> surrogate keys form the basis of good db design
<wgrant> And oauthnonce
<wallyworld> not for join tables necessarily
<wallyworld> but for straight up entity tables, yes
<wgrant> Surrogate keys for things that need to be referenced by FKs, sure
<wgrant> This isn't an entity table
<wgrant> As such
<wallyworld> sure
<wallyworld> what would you prefer the table name to be?
<wgrant> Something involving person, sourcepackagerelease, and cache, probably.
<wgrant> LatestPublishedReleases doesn't mean anything
<wgrant> What do we do when we want a table showing the latest releases for a product, or binarypackagereleases, or...
<wallyworld> shouldn't the table name contain 'latest'?
<wgrant> Probably
<wallyworld> i thought all releases went into spph
<wallyworld> and spr
<wgrant> Source package releases go into SPR and SPPH
<wgrant> Binary package release don't
<wgrant> Product releases don't
<wallyworld> i didn't know there were any other kinds
<wgrant> Because they're not source package releases :)
<wallyworld> ok, i know that *now* :-)
<wallyworld> remember, i am a soyuz virgin
<wgrant> Product releases aren't Soyuz
<wgrant> They're Registry
<wgrant> :)
<wallyworld> well, clearly i have a lot to learn in this domain area
<wallyworld> stub: ignoring the tablename etc, is this correct for creating the surrogate id? https://pastebin.canonical.com/77517/
<wgrant> "id serial PRIMARY KEY"
<wgrant> is all you need
<wgrant> It will autocreate the sequence etc
<wgrant> wallyworld: ^^
<wallyworld> ok
<wallyworld> was the stuff i used just old code?
<wgrant> That's probably how they appear in dumps
<wgrant> serial maps to that
<wallyworld> ok
<wallyworld> i looked at our sql files though
<wgrant> But serial has been around forever. Doing it manually is not so much "old" as "positively ancient"
<wgrant> The base schema? That's pg_dump output
<wgrant> Not manually written
<wallyworld> not sure where i found it now, probably
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugs: ~200
<sinzui> jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/reimport-inactive-templates/+merge/132421
<jcsackett> sinzui: sure.
<jcsackett> sinzui: looks good, just want to make sure that the first check returning `entry` if true and others returnin None is intentional?
<sinzui> yes, because we do not want to tamper with it
<jcsackett> sinzui: ok, that's what i thought, but i figured i would double check.
<jcsackett> sinzui: r=me.
<sinzui> thank you
<czajkowski> cjwatson: people think you sleep! little do they know!
<sinzui> abentley, I think your PRF bug and my person merge bug can be fixed in the same branch
<abentley> sinzui: Mine's already landing via EC2.
<sinzui> excellent
<sinzui> I am moving SELECT to the script group so that no script needs basic access
<sinzui> I will resolve conflicts with your branch
<abentley> sinzui: thanks.
<jcsackett> sinzui: does the db buildbot failure prevent regular devel landings? pqm is yelling about my commit msg not satisfying the testfix requirement but it doesn't look like we should need one...
<sinzui> jcsackett, one failure breaks both. You need to land a fix correct builder or force a build for spurious
<jcsackett> sinzui: ah. for some reason i thought they were separate.
<sinzui> jcsackett, Do you have time to review a branch https://code.launchpad.net/~sinzui/launchpad/merge-db-permissions/+merge/132615
<jcsackett> sinzui: looking now.
<jcsackett> sinzui: r=me.
<sinzui> thank you
<abentley> jcsackett: Could you please review https://code.launchpad.net/~abentley/launchpad/fix-assignment-list/+merge/132621 ?
<jcsackett> abentley: sure.
<jcsackett> abentley: that was short. :-) r=me.
<abentley> jcsackett: Thanks.
<abentley> jcsackett: If only all critical bugs were so simple to fix :-)
<jcsackett> abentley: i hear that.
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~200
#launchpad-dev 2012-11-02
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/fetchpage-does-not-send-headers-always/+merge/132638
<wallyworld_> StevenK: i don't know the code - why can't those 2 classes be change to use getPage()?
<StevenK> I tried, and Trac broke
<wallyworld_> it just seems very hacky is all - fetchPage seems to expect a Request and nothing else
<wallyworld_> so it shouldn't be called with a string
<wallyworld_> and if it is then that's a mistake to be corrected upstream so to speak
<StevenK> It just tosses things into urlopen, and that will deal with either
<StevenK> It's just that we want to send headers
<StevenK> And for that, you need a Request
<wallyworld_> ok, makes more sense now
<wallyworld_> r=me
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-distribution-upstreamreport/+merge/132639
<StevenK> You two should enjoy this one, it's great.
<wgrant> There must be more
<wgrant> Look harder
<StevenK> Really now?
<wgrant> Also, we need to check with UE before this can go anywhere
<StevenK> wgrant: Tempted to close bug 899388
<_mup_> Bug #899388: Some translation statistics are not up to date <lp-translations> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/899388 >
<wgrant> StevenK: Nope
<StevenK> It's still broken?
<wgrant> The stats jobs don't do everything
<StevenK> wgrant: Ah, how do I read https://oops.canonical.com/oops/?oopsid=OOPS-13c0c8bb3601a44992977eaebb7c2d9e ? Since it's a high non-SQL time
<wgrant> StevenK: You could try getting a profile from qastaging
<wgrant> It's probably spending most of the time loading Storm objects
<wgrant> Enum columns, in particular
<StevenK> 3731696 function calls (3643838 primitive calls) in 28.437 CPU seconds
<StevenK> wgrant: enumcol calls don't look high to me
<StevenK>      2043    1.241    0.001    1.392    0.001 store.py:807(_add_to_alive)
<StevenK>    119112    1.237    0.000    2.744    0.000 {method 'set' of 'storm.variables.Variable' objects}
<StevenK>      2290    0.907    0.000    5.849    0.003 store.py:680(_load_object)
<StevenK> Looks like it's loading stacks of objects
<wgrant> StevenK: Found anything?
<StevenK> wgrant: Indeed, I found my lunch, and it was good.
<wgrant> You also found a way to break buildbot :P
<StevenK> Oh, for the love of ...
 * StevenK fixes
<wgrant> Now to pretend to be a builder...
<lifeless> wrong name
<StevenK> Crumbs
<StevenK> Last failure is test_rendered_query_counts_constant_with_team_memberships
<wgrant> :(
<StevenK> That didn't fail on buildbot, so I'm not sure why testr ran it
<StevenK> wgrant: Reply from Bryce on the bug
<wgrant> So I saw
<StevenK> I'm not sure how to read that reply.
<StevenK> It sounds like he's interested and used to graph the data, but doesn't answer my question on what we can do
<StevenK> wallyworld_: You'll get a kick out of this: http://www.theregister.co.uk/2012/11/01/samsung_case_apple_told_to_apologise_again/
<wallyworld_> lol
<wallyworld_> too bad the us courts are so stupid
<wallyworld_> i wonder how mych campaign money apple donates
<wallyworld_> that would have nothing to do with it i'm sure
<StevenK> wallyworld_: I'm just glad that 1. The UK court forced Apple to apologise. 2. That the UK judge is displeased with their first attempt and has told them they have 24 hours to fix it.
<wallyworld_> yeah, me too. but i despair that crapple lost the battle here but is winning the war
<sinzui> StevenK, Bryce and other upstream stewards want a chart of how many bugs in ubuntu packages are triaged, have an upstream task (a task that we presume is upstream), have a bug watch.
<sinzui> StevenK, each number comes from a bug search, so api could produce just the numbers you need.
<StevenK> sinzui: With no changes needed?
<sinzui> StevenK, , but if you are patient to get the page to load, you will see that the top 100 packages are someone's acid-fueled fantasy...pidgin, firefox 3.5, f-spot, banshee, ekiga...
<sinzui> StevenK, the page lists 6 x-related packages.
 * sinzui gets the example urls for an x package
<sinzui> StevenK, http://paste.ubuntu.com/1325544/
<StevenK> sinzui: So you think we provide enough tools for Bryce to graph without having to screenscrape and we're free to destroy Distribution:+upstreamreport and the horse it rode in on?
<sinzui> We know he cannot screenscrape. It took me 10 minutes to load the page
<sinzui> We should delete the page because we should not be working about ekiga bugs. There is not mention of bugs in zeitgeist which is what unity needs
<sinzui> Some one could write a better api script that does the same work for any package, then we let the community choose the packages instead of looking back at Hardy with fondness
<wgrant> If we were to try to fix it, we would likely end up having to rewrite it based on a full new set of denormalised data structures
<wgrant> Ubuntu has a few more bugs now than it did in 2008
<StevenK> Well, I'm happy we have the data and if it's up to Ubuntu to write an API script to pull it per package, then that's fine and we can destroy the page.
<sinzui> We can destroy the page. If some says they are using it for a package in Quantal, I will write the script on my Lunch
<StevenK> sinzui: https://code.launchpad.net/~stevenk/launchpad/destroy-distribution-upstreamreport/+merge/132639 is an MP I prepared earlier
<StevenK> wgrant said there must be more, but I'm not certain if he was trolling. :-)
<sinzui> r=me
 * sinzui find bed
<StevenK> sinzui: Night!
<StevenK> wgrant: Right, I think I'm undistracted enough.
<wallyworld_> wgrant: if you are interested, this currently seems to work nicely. the garbo job needs work to handle existing records in the cache table and to do bulk updates (it is just designed right now to run against a totally empty cache table to get data in)
<wallyworld_> https://code.launchpad.net/~wallyworld/launchpad/ppa-packages-timeout-1071581/+merge/132646
<wgrant> wallyworld_: How does it handle the case where I maintain a package that has been uploaded by two different people?
<wallyworld_> running it on dev produces the same results as previously as far as i can tell
<wgrant> Also, ew
<wallyworld_> there should be a row for each of those
<wgrant> Sure
<wgrant> That's the problem :)
<wallyworld_> isn't that the case now?
<wgrant> It's only meant to show on my +maintained-packages once, even if it was uploaded by multiple people
<wgrant> Today we filter by maintainer and distinct on (archive, distroseries, sourcepackagename)
<wallyworld_> oh ok, so just the latest shouyld show?
<wgrant> but the cache is distincted on (archive, distroseries, sourcepackagename, creator, maintainer)
<wallyworld_> sure, that's what i thought we needed
<wallyworld_> but i can change it
<wgrant> Also, the garbo job is very much non-wgrant compliant
<wallyworld_> actually, maybe it does correctly only have the one row, will have to check
<wallyworld_> why?
<wgrant> One does not simply concatenate arbitrary strings into SQL.
<wallyworld_> you mean for the job table update?
<wgrant> Yes
<wallyworld_> well, it's done elsewhere
<wallyworld_> and it's just for a simple, safe update
<wgrant> Where?
<StevenK> That is not an argument.
<wallyworld_> in many other garbo jobs
<wgrant> I'm going to find that place and delete it :)
<wallyworld_> have a look
<wallyworld_> there's lots
<StevenK> Cargo-culting existing bad practise is bad
<wallyworld_> it's not worth introducing a whole new class
<wgrant> Not seeing it...
<wallyworld_> just to update one row in a job table
<wgrant> Hmm?
<wgrant> Using store.execute and using string contatenation to create SQL are orthogonal concepts :)
<wgrant> store.execute is fine
<wgrant> String concatenation is not
<wallyworld_> see OpenIDConsumerAssociationPruner for example
<wallyworld_> i agree though, this was just quick and dirty to get something going
<wallyworld_> i could care less right now about the garbo job - it needs work as i said
<wallyworld_> i'm more concerned with the correctness of the table population
<wgrant> ("couldn't")
<wgrant> But everyone needs to not even jump to string concatenation as a first step, ever
<wallyworld_> i was using colloquial
<wgrant> , params=('foo', 'bar') is not that hard :)
<wallyworld_> sigh, it was a poc
<wgrant> LP PoCs have a habit of making it to production :(
<wallyworld_> not once i'm finnished with it
<wallyworld_> i just want to get the data in palce to see if the business logic works
<StevenK> wgrant: If you're done lambasting wallyworld_, could you help with ++profile++show?
<wallyworld_> and the important logic is all strormified
<wgrant> StevenK: What's the issue?
<StevenK> wallyworld_: You're fun to argue with. When you get frustrated you start typing like sinzui.
<wgrant> Heh
<wallyworld_> yes i do.
<StevenK> wgrant: Milestone:+index again, now I'm now undistracted.
<wgrant> StevenK: Sure, but what about the profile?
<wgrant> Like, what do you need help with?
<wgrant> If it involves killing dogfood I'm all for it
<StevenK> wgrant: I'm not sure how to use the profile to point out the terrible bits of code.
<wgrant> heh
<wgrant> type 'ubuntu-10' in awesomebar
<wgrant> first result is https://qastaging.launchpad.net/ubuntu/+milestone/ubuntu-10.04.2/++profile++show
<StevenK> Aside from 'all of it'
<wgrant> That was months ago
<StevenK> wgrant: Haha
<StevenK> I have https://qastaging.launchpad.net/ubuntu/+milestone/ubuntu-10.10/++profile++show/+index loaded
<wgrant> wallyworld_: So, the only obvious logic concern is the creator vs maintainer thingy
<wallyworld_> i just added in maintainer to distinct on
<wallyworld_> i think that wil fix it
<wgrant> Well
<wgrant> I'm not sure that a single table can satisfy both
<wallyworld_> not sure, i'll have to look into it some more
<wgrant> Given that the table is meant to satisfy the distinct itself, hmm
<wgrant> But it probably can't, because it has to filter on either creator or maintainer
<wallyworld_> maybe i need 2 tables then
<wgrant> StevenK: You managed to get the table to not time out, then?
<wallyworld_> or the record count will be ssmall enough that some live filtering can be done
<wgrant> wallyworld_: Quite likely. They end up pretty much identical and can be maintained by a live job
<wgrant> s/a live/the same/
<StevenK> wgrant: Yeah, 10.10 loaded with qas, but ++profile++show's title shows Error: Timeout
<wgrant> Hmm
<wgrant> So your initial assessment was pretty accurate
<StevenK> wgrant: That it's just loading 2,100 odd objects?
<wgrant> No, the "all of it" bit
<StevenK> Oh, right.
<StevenK> Yeah, the code looks pretty terrible
<wgrant> StevenK: http://people.canonical.com/~wgrant/output.png
<StevenK> What's that from?
<wgrant> gprof2dot
<wgrant> ubuntu-10.04.2, not ubuntu-10.10, but it's slow too
<StevenK> launchpad_dogfood=> SELECT productrelease, COUNT(libraryfile) FROM productreleasefile GROUP BY productrelease HAVING COUNT(libraryfile) > 0 ORDER BY COUNT(libraryfile) DESC LIMIT 1;
<StevenK>  productrelease | count
<StevenK> ----------------+-------
<StevenK>           32517 |   899
<wgrant> Yeah
<StevenK> I dispair at our DB
<wgrant> PRF is a bit crap
<StevenK> wgrant: It's pretty, but it's a mess of colour :-)
<StevenK> wgrant: So I do wonder if the mixin's _bugtasks method is too blame for most of the mess.
<wgrant> Well
<wgrant> That's probably just spending time loading lots of objects
<StevenK> Which is most of the problem, no?
<wgrant> Sure
<StevenK> We only want the title, importance, assignee and status. Can't we just populate a list straight from BTF and skip most of the garbage?
<wgrant> If you want to reimplement all of Launchpad, sure
<StevenK> wgrant: I meant for Milestone:+index specifically, how does your comment follow on?
<wgrant> StevenK: Milestone:+index is part of Launchpad, and relies on a lot of other parts of Launchpad, all of which rely on having these objects
<wgrant> We may be able to optimise the loading of those objects
<wgrant> The first step is probably to try to reproduce it locally.
<StevenK> wgrant: Well, what I was actually suggesting lobomitizing Milestone:+index to not have the bug objects which is pulling in a lot of baggage, and work from a list populated by BTF.
<wgrant> StevenK: The DB queries aren't the problem (and BTF doesn't have title)
<wgrant> Eliminating the bug objects would work, but it's a lot of work
<StevenK> Hmmm
<StevenK> wgrant: Did you attempt wallyworld_'s QA?
<wgrant> I've been at it on and off for the last 5 hours :/
<wgrant> DF is a little bit slow
<StevenK> A little? :-P
<wgrant> I've about to give in and qa-ok it half-way through
<StevenK> Might be a little late to deplay today, though
<wgrant> The bug is fixed, and I don't see how it could have broken the existing functionality
<wgrant> I can see my correctly defaulted binary in the queue, but acceptance times out...
 * wgrant concedes defeat
<czajkowski> morning
<deryck> Morning, all.
<czajkowski> deryck: ello
<rick_h_> morning deryck
<deryck> rick_h_, hey, man.
<czajkowski> rick_h_: what time is it where you are?
<rick_h_> czajkowski: 7am
<rick_h_> trying to get back on schedule
<czajkowski> wow
<deryck> czajkowski, rick_h_ never stops. :)
<czajkowski> finally choolate and nice fruit with ginger ale are on offer!
<czajkowski> *chocolate
<deryck> Sorry for all the email Orange Squad.  Trying to get cards on board categorized and prioritized.
<deryck> Assuming you get board email :)
<rick_h_> deryck: rgr yea noticed
<deryck> lunch time!
<abentley> deryck: I don't get board mail, so I'm happy.
<deryck> abentley, excellent then :)
<abentley> deryck: I plan to back-burner 1066905 in favour of 1074139, because the latter can lock users out of their projects, and it should be an easy fix.
<deryck> abentley, I trust your judgement.  can't look right now, in meeting.
<joey> Hi, what's the best way over the api to return a list of teams that match a wildcard?
<joey> e.g. ubuntu-*
<deryck> rick_h_: hey, there.
<rick_h_> deryck: howdy
<deryck> rick_h_: see my note on the timeout bug.  Did we change something with profile queries?  Or just now suddenly profile queries are slower for everyone?
<deryck> or timing out for everyone, rather
<sinzui> deryck, rick_h_, I am not seeing these profile page timeouts. I suspect I cannot because I am a CA...privacy security checks short circuit for me
<rick_h_> deryck: not sure, I see the one query that's taking 4s around blueprints
<deryck> sinzui: interesting
<rick_h_> so looking at where it's at and what it's trying to do
<sinzui> oh, but I still cannot see private blueprints/bugs/branches that I am not subscribed to, so that does not make sence
<deryck> rick_h_: yeah, that query is slow, no doubt.  But did we just add the filter?  or is maybe something else also changed recently to make matters worse?
<rick_h_> deryck: not sure, I've not tracked the bzr history of it atm. I can peek and try to see if there's some recent stuff that hit it
<rick_h_> was more just trying to understand and figure out where the ineffecient part is
<rick_h_> it's got somethjing like 5 or 6 subqueries in it atm ugh
<deryck> rick_h_: yeah, the 4 second query is awful.  fixing that will most likely fix the issue.
<deryck> rick_h_: it just seems strange that we *suddenly* started OOPSing every user page.  unless we just deployed this query, or something else changed around it.
<rick_h_> everything else seems pretty light
<rick_h_> deryck: ok, cool I didn't realize it was everyone hitting this and suddenly. I just assumed it was something for those with private blueprints started seeing recently
<deryck> rick_h_: no, you don't need private blueprints to cause the problem.  we're querying to make sure you can see your blueprints, whether you have private ones or not.
<rick_h_> right, my profile page loads fine
<deryck> hmmm
<deryck> rick_h_: not for me.
<rick_h_> deryck: but this is coming from abentley's new assigned_specs_in_progress
<rick_h_> so I bet that's why it's more recent.
<rick_h_> that was just something last week that was worked on/added
<rick_h_> while we were in CPH I believe
<deryck> rick_h_: ah, right, I see the rev now.  yeah, that makes more sense.
<deryck> rick_h_: so chat with abentley about it then.
<rick_h_> will do, thanks for forcing me to look at that closer and connect the dots.
<abentley> rick_h_: My profile page is zippy.
<deryck> rick_h_: np.  hi, abentley
<abentley> deryck: Hi.
<rick_h_> abentley: yea, mine seems to be ok as well, but we're getting a bunch of these
<deryck> so every profile page OOPes for me.  as does for other people in the bug.
<rick_h_> but I don't have many BP and none are private
<sinzui> We don't use blueprints, that is why we don't see them
<rick_h_> right, that seems to fit for me as well. and this is one UGLY generated sql in the oops
<abentley> deryck: Can I have an oops-id?
<rick_h_> https://oops.canonical.com/oops/?oopsid=OOPS-8844d78960633507626e1325e8cad359
<rick_h_> abentley: ^
<deryck> abentley: https://oops.canonical.com/oops/?oopsid=OOPS-1390ee2cc5227ea249e6dbdc7c1f79f0
<deryck> heh
<deryck> OOPS, twice :)
<rick_h_> you dare race me deryck :P
<deryck> never :)
<rick_h_> I'm back on a real keyboard in the office again bwuhahaha
<abentley> deryck: Are you free to compare query times?
<deryck> abentley: sure.
<abentley> Aw man, even here we don't have the substitutions.
<abentley> deryck: I can't see how to find hloeung's db id via the API.  Could you look that up, please?
<deryck> abentley: https://pastebin.canonical.com/77605/ (private paste, since it's data for a query.)
<deryck> abentley: I need to go down for dinner.  maybe sinzui could run queries for you?
<sinzui> I can
<abentley> sinzui: Thanks.
 * sinzui is stuck so needs a diversion
<deryck> awesome, thanks guy!
<deryck> guys even
<abentley> sinzui: Could you please time this against qastaging?  (I'm expecting it to run a long time.)
<abentley> sinzui: https://pastebin.canonical.com/77606/
<sinzui> oops, I ran on staging
<abentley> sinzui: Should be fine.
<sinzui> well both are subsecond to get 2 rowns
<sinzui> rows
<abentley> sinzui: That is not good, because it suggests I've done the substitutions wrong.
<abentley> sinzui: Or else the query is not consistently slow.
<sinzui> abentley, this is the query plan form qastaging: https://pastebin.canonical.com/77609/
<sinzui> abentley, surely there is more private blueprints in production than qastaging.
<sinzui> staging is 2 weeks old, so that would be better
<abentley> sinzui: You ran it against staging, though.
<sinzui> yes, but then I switched to qastaging...
<sinzui> oh, the query plans are different!
<abentley> sinzui: You said staging was subsecond.
<sinzui> this is staging's explain, and yes, subsecond for both: https://pastebin.canonical.com/77610/
<abentley> sinzui: I think we need to try it on prod.
<sinzui> we need webops for that
<abentley> rick_h_: Do we have a bug for these oopses?
<rick_h_> abentley: yes, https://bugs.launchpad.net/launchpad/+bug/1074239
<_mup_> Bug #1074239: Timeout error when trying to view any user profile page. <lp-blueprints> <private-projects> <timeout> <Launchpad itself:In Progress by rharding> < https://launchpad.net/bugs/1074239 >
<abentley> rick_h_: ty
<joey> well nobody liked my last question so how about a different one.. I need help optimizing a short piece of python code that's comparing LP object to LP object
<joey> anyone up for that?
<lifeless> joey: you know the rule... don't ask to ask ;)
<joey> yeah but I'm just too polite
<joey> google's not been very helpful... I think it's python thing...
<joey> lifeless: how's the new job?
<lifeless> joey: pretty good
<lifeless> joey: --big-- company
#launchpad-dev 2012-11-03
<smartboyhw> Any on call reviewer here?
#launchpad-dev 2013-10-28
<harishnavnit> okay , i finally logged into my lxc , now i'm about to run the rocketfuel setup inside the container , would it be better if i removed the previous lp-directories or let them remain as it is and run the rocketfuel setup again in a different direcotry ??
<wgrant> Leaving them as is is fine.
<wgrant> harishnavnit: No need to delete them.
<harishnavnit> okay , so i guess i should run the rocketfuel setup in a different directory then ?
<wgrant> It doesn't really matter where you run it.
<wgrant> It will always use ~/launchpad unless you alter the script to put it somewhere else.
<harishnavnit> yea , that is my confusion
<harishnavnit> i already have a launchpad folder in the ~/ directory
<harishnavnit> will it merge the contents ?
<wgrant> It will download any bits that are missing.
<harishnavnit> okay
<harishnavnit> just FYI , i had run the rocketfuel setup outside the lxc previously and I'm now going ahead with the setup inside a container
#launchpad-dev 2013-10-29
<StevenK> wgrant: So, microservices?
<wgrant> StevenK: Have you adequately stress-tested the implementation, including how it behaves around failed transactions and similar scenarios?
<StevenK> wgrant: You were attempting to break it, last I heard.
<StevenK> So I'm curious how that went.
<wgrant> I recall I couldn't get it to actually start.
<StevenK> How were you trying to start it?
<wgrant> I think even the tests failed.
<wgrant> I don't recall if you fixed that
<StevenK> Bleh, auditor tests fail on saucy due to django 1.5.4
<StevenK> ... python-django is 44MiB?
<wgrant> So remember what I said about framework choice.
<StevenK> I don't really want to rewrite auditor at this point :-)
<StevenK> wgrant: Keep in mind that auditor is an django app -- auditorfixture provides a handy wrapper to run it by hand
<StevenK> wgrant: http://pastebin.ubuntu.com/6322268/
<wgrant> StevenK: Looks like you fixed the bug.
<wgrant> StevenK: So, last time I had you make substantial changes, you added transaction support. How do failed transactions behave? What happens if the service is down? What is the HA story to prevent the service from going down except in network outages?
<wgrant> Has performance been tested? How will we handle upgrades?
<StevenK> wgrant: Failed transactions should be handled by the transaction manager, we only commit if the transaction status is COMMITTED
<StevenK> wgrant: I *think* we time out after 1 second if the service is down when trying to GET or POST.
<wgrant> "time out"
<wgrant> What does that mean?
<StevenK> The urlopen() calls in auditorclient have a default timeout of 1 second.
<wgrant> Sure.
<wgrant> But that doesn't tell us what happens when the service is down or there is a network fault.
<wgrant> That tells us that something will probably happen after approximately one second.
<StevenK> I think I tested this, and you get back no data on GET or POST
<StevenK> In terms of the HA story, I'm not sure.
<wgrant> Is it reasonable for it to simple return no data for failed reads, and throw away failed writes?
<StevenK> For the former, yes, probably not for the latter.
<StevenK> But I'm still trying to swap stuff in, since it's been months.
<wgrant> For the former it may be acceptable for the current scenario, I'm not sure. But it's certainly not acceptable for all scenarios.
<wgrant> I raised these issues last time too, and you ignored them :P
<StevenK> Performance was going to be looked at closely when I actually got this onto staging, since it won't be ready for an end-to-end test until this stuff lands.
<StevenK> Upgrades involves pushing a new dist tarball into the download-cache, and bumping versions.cfg of LP, auditorfixture and production-auditor.
<wgrant> DB upgrades?
<wgrant> Auditor is a well-isolated service; performance testing would want to be done outside the context of LP.
<StevenK> There is some support for south sprinkled in, but I've not had to change the schema yet.
<wgrant> Doing it on LP staging would be prohibitively slow; you'd need hundreds of thousands to millions of uploads.
<wgrant> Even if you do use south, how do we do DB upgrades?
<wgrant> Remember that Launchpad likes to read from and write to auditor.
<StevenK> Turn the feature flag off, and it will not any more.
<wgrant> But now I've lost data.
<wgrant> Now malicious archive admins have just approved a thousand rootkitted SRUs :(
<wgrant> And I can't tell who to fire :(
<StevenK> wgrant: Sure, and I answer "I don't know", and that's the answer I get if I ask if you're happy with the branch. :-P
<wgrant> Answers to these questions are essential for approval of the branch, because this is the first significant separate datastore we have.
<StevenK> Yes.
<StevenK> "Do it as a service,
<StevenK> Blah
<StevenK> "Do it as a microservice, it will be easy" and other lies lifeless has told.
<wgrant> It would be easy if we had existing microservice patterns, or even rules that they should follow.
<wgrant> But this is breaking new ground.
<wgrant> In the Launchpad world.
<lifeless> Sorry :)
<lifeless> But they aren't lies.
<StevenK> Is this where I give up horribly and put up a DB patch?
<lifeless> If I was still working on LP, I'd happily be helping you get through this transition.
<StevenK> Sure.
<StevenK> lifeless: Sure, and then you get to point at the ground that auditor has forged when someone writes another microservice, but there is no forging, just lots of roadblocks.
<lifeless> StevenK: wgrant: so, looking back at the discussion
<lifeless> StevenK: wgrant: it seems to me that the questions wgrant is asking are generic and a checklist for them around the design and client-usage-of microservices would be a good idea.
<wgrant> Certainly :)
<wgrant> The goal of the review process for the auditor integration is to become such a checklist.
<wgrant> That is, I believe, fundamentally more valuable than auditor itself.
<lifeless> so, my point is that listing them here is lossy.
<lifeless> How about wiki page in the services area
<lifeless> and a second wiki page with the concrete answers to the questions for auditor
<StevenK> I still don't know the answers.
<lifeless> where there answers may either answer it (at a design level) or say 'pending' or whatever if some evaluation hasn't been done
<lifeless> and then the two of you (plus LOSAs as appropriate) can collaborate on getting good answers and fillout out the questionnaire
#launchpad-dev 2013-10-30
<StevenK> wgrant: Oooh, when do I get to see this BFJO destruction?
<wgrant> StevenK: Where did you discover that?
<wgrant> Oh, dbpatches I guess.
<StevenK> Yeah
<lifeless> no shock and awe here :)
<xnox> Hm, my page https://launchpad.net/~xnox/+uploaded-packages is out of date.
<xnox> unless it doesn't show stuff which is in -proposed at the moment.
<wgrant> xnox: The script that updates that cached reporting data was backlogged. It's up to date now.
#launchpad-dev 2013-11-02
<lifeless> wgrant: does LP still have ~40 appservers ?
<wgrant> lifeless: 60
<lifeless> wgrant: thanks
<wgrant> Why?
<lifeless> wgrant: a) curious; b) openstack is going to roughly double one of it's sources of API traffic and while I didn't think that would be an issue..
<wgrant> Ah
<lifeless> reviewday; single threaded, will run for about 12 minutes
<wgrant> OpenStack-related traffic isn't particularly bad at the moment.
<lifeless> multiple times a day
<wgrant> So should be fine.
<lifeless> yeah, I figured it will be
<StevenK> I thought it was 20*4 == 80
#launchpad-dev 2013-11-03
<cjwatson>     """A utility of this interface used to create _things_."""
<cjwatson> helpful comment is helpful
<wgrant> cjwatson: :)
<wgrant> We try.
<wgrant> cjwatson: Oh, you're working on the master side of livefs things?
<wgrant> cjwatson: That's about to less messy, though I'm off this week.
<wgrant> about to become less messy, that is
<alexhenrie> hello, is there currently an on-call reviewer?
<wgrant> alexhenrie: It's a weekend, so not really. But I'm around.
<alexhenrie> well if you can help me, great, and if not, no big deal.
<alexhenrie> I just saw that my submission was marked "resubmit"
<alexhenrie> https://code.launchpad.net/~alexhenrie24/ubuntu/saucy/lubuntu-default-settings/fix-for-1232578
<alexhenrie> I didn't get an email, and I don't see any explanation
<alexhenrie> what am I supposed to do?
<wgrant> alexhenrie: Hm, I think you might be a bit confused. Reviewers here review branches of the code that runs Launchpad.net, not every branch on Launchpad. But I see a comment in that merge proposal explaining the Resubmit vote.
<wgrant> "Looks good to me, but can you please rebase this against lp:lubuntu-default-settings so that Lubuntu guys could take a look at this?"
<alexhenrie> I'm new to Ubuntu's code submission process. where did you find that comment?
<wgrant> At the URL you linked.
<alexhenrie> did you have to click something to see it?
<wgrant> No.
<elmo> alexhenrie: https://code.launchpad.net/~alexhenrie24/ubuntu/saucy/lubuntu-default-settings/fix-for-1232578/+merge/191047
<wgrant> Oh
<wgrant> I must have clicked through to the MP without thinking, sorry.
<elmo> alexhenrie: the 'ready for review link
<wgrant> alexhenrie: Anyway, #ubuntu-devel or a Lubuntu-related channel might be more relevant.
<alexhenrie> okay thanks
<alexhenrie> for now I want to concentrate on figuring out how to do what Dmitry asked
<alexhenrie> I found a tutotial on this before but I can't find it now
<wgrant> alexhenrie: bzr branch lp:lubuntu-default-settings; hack hack hack; bzr commit; bzr push lp:~alexhenrie24/lubuntu-default-settings/fix-for-1232578; then propose a merge using the web UI.
<alexhenrie> thank you :)
<alexhenrie> btw I found the page I was looking for: http://developer.ubuntu.com/packaging/html/fixing-a-bug.html
<alexhenrie> okay I have another easy question
<alexhenrie> was I supposed to have changed the revision from 0.35 to 0.35ubuntu1? http://bazaar.launchpad.net/~alexhenrie24/lubuntu-default-settings/fix-for-1232578/revision/263
<wgrant> alexhenrie: No, 0.35 is correct for that package. ubuntuX isn't appended to the version of some Ubuntu-native packages.
<wgrant> (the ubuntuX suffix exists to disambiguate Ubuntu and Debian package versions, but it's unlikely that lubuntu-default-settings will ever be in Debian)
<alexhenrie> okay so I need to edit my patch and resubmit
<alexhenrie> is there anything else obviously wrong with it?
<wgrant> alexhenrie: I'm no Lubuntu developer, but it looks sensible enough to me.
<alexhenrie> thanks. I was just looking for errors in the commit message or whatever.
<alexhenrie> I think I have it fixed now: https://code.launchpad.net/~alexhenrie24/lubuntu-default-settings/fix-for-1232578
<alexhenrie> thanks for the help!
<wgrant> np
#launchpad-dev 2014-10-27
<Neo31> hello, any help on this ticket pls (is that the right place to ask for help) ? https://answers.launchpad.net/launchpad/+question/256186
<wgrant> Neo31: Done.
#launchpad-dev 2014-10-28
<kb9vqf> Quick question--I have an older public Launchpad service here (circa 2009) that I am upgrading to the latest Launchpad code.  Does the database schema automatically upgrade or do I need to apply the schema patches individually?
<kb9vqf> For anyone reading the logs: after perusing the source more carefully I guess ./database/schema/upgrade.py will do what I want.  I'll try it tomorrow on the test box.
 * kb9vqf goes off to analyze more of the source
<daker> hi
<daker> there is an UI annoyin bug which cause apart of the the header to be unusable with right click :(
<daker> a UI*
#launchpad-dev 2014-10-29
<kb9vqf> I'm attempting to upgrade a Launchpad system (circa early 2010) to the latest code.  The system has been in production all this time; losing any data would not be acceptable.  On my test box I am having problems upgrading the database.
<kb9vqf> What is the best way to handle the schema changes?  The database upgrade process keeps dying, and it looks like somehow I need to apply the "new" 2009 schema
<kb9vqf> sorry, that should be 2209 schema above
<kb9vqf> The database upgrade paches refer to missing functions that are defined in the 2209 schema but nowhere else
<kb9vqf> I guess that over time some of the incremental patches were dropped from the source?  Any help would be appreciated here. :-)
<wgrant> kb9vqf: Hm, upgrading something that old without good knowledge of the datamodel would be rather challenging.
<wgrant> It's not just a matter of applying database patches in all cases.
<kb9vqf> Hmm, ok
<kb9vqf> We've only been using the PPA features
<kb9vqf> How were these upgrades performed on the production systems back whe?
<kb9vqf> *when?
 * kb9vqf has been having a hard time digging up any information outside of the source itself
<kb9vqf> I'm most interested in the missing functions and such--I'm guessing that the functions were added via SQL commands that are not part of the patchset, but don't know for sure
<wgrant> A combination of SQL patches and in some cases custom Python, and not all of the Python is in the tree. The 2209 baseline schema was introduced in 2011.
<wgrant> All DDL is in the patches.
<kb9vqf> OK
<wgrant> But the 2208 patches would have to be retrieved from bzr history.
<kb9vqf> They are in the archive directory fortunately
<wgrant> Oh, I thought I'd deleted them. Must have been saving them.
<kb9vqf> I've been able to apply most of the patches, but one in particular is failing that concerns me
<kb9vqf> patch-2209-00-1.sql
<kb9vqf> sets sourcepackagename NOT NULL constraint
<wgrant> Anyway, those will bring a database with no rows up to date. But there is much data migration that may be needed between some of the patches.
<wgrant> Right, that one, for example.
<kb9vqf> I take it custom python was used there?
<wgrant> Between the baseline and patch-2209-00-1.sql, an online data migration script was run to populate that column.
<kb9vqf> any chance of obtaining that script or do I have to write something?
<wgrant> It's probably possible to struggle through this, but it's probably more trouble than it's worth.
<kb9vqf> I'm running into obsolescence issues fast due to the old codebase relying on very old python libraries
<kb9vqf> so I need to do something soon, but at the same time we have years of work on the PPAs
<wgrant> Does that revno even run on lucid?
<kb9vqf> Yes
<kb9vqf> In fact I hacked it up to run on Debian Wheezy
<kb9vqf> to buy time :)
<wgrant> Heh
<kb9vqf> So far the only major problem I have run into (well, that I am aware of at the moment) is that sourcepackagename migration
<kb9vqf> Is there a better way to save the PPA information and just reload the database?
<wgrant> Isn't that the very first patch?
<wgrant> There are a lot of others further down the line which will be bigger problems.
<kb9vqf> Been applying 2208- patches
<kb9vqf> ok
<wgrant> Is it really that critical to preserve the PPA history?
<kb9vqf> Yeah, in this case it is...some of our software is still in use on the oldest PPAs and if there is a security patch we need to be able to rebuild
<kb9vqf> at the same time we can't sanely reupload the sources for all PPAs
<kb9vqf> the PPAs *are* the snapshot source repository in many cases
<kb9vqf> A related question: with the latest Launchpad is this kind of thing going to continue to to be the case where there is no way to sanely migrate to a new Launchpad version?
 * kb9vqf is envisioning a very ugly script to download source from the old Launchpad instance and rebuild on a new Launchpad instance
 * kb9vqf is also shuddering at the power bill for the tens of thousands of builds
<kb9vqf> wgrant: so if I take this one step at a time, first major problem is the sourcepackagename column.  is there any way to get the original migration script, and if not, what data is supposed to be in that field?
<wgrant> It's not fundamentally terribly difficult to make it very upgradable going forward, but there's been no motivation to do so in the past.
<kb9vqf> Am I still the only non-Canonical deployment?
<wgrant> That particular one is very simple: it's just a copy of SourcePackageRelease.sourcepackagename.
<wgrant> There have been a couple of others over the years, but none survived.
<kb9vqf> Heh.  This one needs to survive as we use it very heavily :)
<kb9vqf> And it's interesting to be the first and only :)
<kb9vqf> OK, I'm taking notes.  patch-2209-00-1.sql requires copies of SourcePackageRelease.sourcepackagename to be placed in sourcepackagename.  What other patches should I be concerned about?
<wgrant> kb9vqf: There are no bugs, blueprints, translations, etc. IIRC, are there?
<kb9vqf> no
<kb9vqf> Here is where most (but not all) of our work really is:
<kb9vqf> https://quickbuild.pearsoncomputing.net/~trinity
<kb9vqf> There are a few other PPAs, and we were going to deploy translations soon so I'm glad we had this chat first :)
<kb9vqf> If it makes any difference, the majority of the other work is at https://quickbuild.pearsoncomputing.net/~slavek-banko and https://quickbuild.pearsoncomputing.net/~kb9vqf
<kb9vqf> wgrant: hopefully the fact that we're only using PPAs (for now) makes this a bit easier?
<wgrant> The really awkward patches were around bugs, branches and blueprints. Translations changes were fairly managable.
<wgrant> Since the 2209 baseline, the only patches that should impact you apart from the 2209-00-1 are:
<wgrant> patch-2209-01-1.sql
<wgrant> patch-2209-40-*.sql (PackageUpload.searchable_*)
<wgrant> patch-2209-41-*.sql (BFJ refactor)
<wgrant> patch-2209-49-*.sql (ProcessorFamily refactor)
<wgrant> patch-2209-51-*.sql (BFJO refactor)
<wgrant> patch-2209-58-*.sql (LibraryFileContent.id widening)
<kb9vqf> Ah, ok.  No bugs, bzr is completely disabled, translations we want to get started with but currently we have no translations on Launchpad
<wgrant> The rest *should* apply cleanly, though there are also two postgres upgrades in the middle there.
<kb9vqf> wgrant: OK, thanks.  the two refactors look a bit scary though, any hints?
<wgrant> Of those, 2209-01-1.sql, 2209-40-*.sql and 2209-58-*.sql are pretty simple, the others I can dig up the prod migrations for.
<wgrant> And before 2209 you should be reasonably safe.
<wgrant> We only started doing the extra Python steps when we stopped doing 90 minute database upgrade outages, which was probably late 2010 or early 2011.
<kb9vqf> Yeah most of the pre-2209 stuff is just missing functions; I'm fixing up the migration patches for those as I sit here :)
<kb9vqf> wgrant: if you don't mind digging up the migration scripts I'd be very grateful
<kb9vqf> also if there is any way to register a "vote" for making Launchpad properly upgradable I'd like to see that...I don't think my instance is going anywhere anytime soon :)
<wgrant> You will want to check your LaunchpadDatabaseRevision table and make sure you apply any 2208-* patches that are missing.
<wgrant> Rather than dig up the incremental migration scripts, probably best to write some SQL that does each of the nasty sequences in one hit.
<wgrant> The postgres version upgrades may be more problematic. If you run into issues there, it may be worth hacking the database until it roughly resembles the new schema, then setting up a new instance on precise and postgres 9.3 and pg_restore'ing the data into the relevant tables.
<kb9vqf> wgrant: I'm actually on 9.3 now
<kb9vqf> pg_dump, g_restore from production to set up the staging box :)
<kb9vqf> so at this point I pretty much just need to know what to do to the data for those 5 patches
<kb9vqf> I can probably handle the SQL, but any advice you can give about what needs to actually happen is going to shave days/weeks of time off on this end (e.g. if I don't know what it does I get to study all of the bzr revisions around that timeframe...yuck ;-)
<wgrant> I'm not that cruel.
<kb9vqf> :)
<wgrant> The SQL will take me like 10 minutes, 'cause I wrote most of those patches.
<wgrant> I'll have a look at it tonight.
<kb9vqf> Actually you guys have been most supportive...I do appreciate it
<kb9vqf> Gives Canonical a high standing in my book
<kb9vqf> do you want my Email address?
<wgrant> It's just your nick @pearsoncomputing.net, isn't it?
<kb9vqf> correct
<kb9vqf> I'll work on getting the database up to a proper 2208 schema; you will handle the rest then or should I be writing some SQL for the first 2209 patch?
 * kb9vqf notes wgrant has a very good memory
<wgrant> It's only been like four years!
<kb9vqf> yeah :-P
<wgrant> Three of them are two lines of SQL which I could write in my sleep, the other three will take a few minutes.
<wgrant> If you can get to the 2209 baseline, my list should cover everything else.
<kb9vqf> ok, thanks again.  it helps to know what needs to be done to the database before coding (I'm still a bit in the dark there but I'll understand once I see the SQL)
<wgrant> And the patches from where you are now to 2209 should be OK. If not, they'll be similarly esy.
<kb9vqf> Yeah, I can hande the 2209 easily
<kb9vqf> *the conversion to 2209
<wgrant> Also, modern LP supports PPAs for multiple distros, so you can stop doing the awkward Debian-in-Ubuntu thing if you desire.
<kb9vqf> wgrant: that's one of the final straws that made me bite this upgrade bullet...that one along with the Any arch building on machines other than i386 :)
 * kb9vqf got a bit tired of watching half the build farm chew on l10n packages while the other half sat idle
<kb9vqf> so while you're here...we're looking at managing our translations with our Launchpad instance.  Problem is we use GIT not BZR; can Launchpad properly handle a GIT-based translation module at this point?
<kb9vqf> The features for translators on the Launchpad instance are compelling, but we need to stick with GIT for other reasons
<wgrant> LP can import git branches into bzr (as long as they don't use submodules or signed tags), but setting up all the codehosting infrastructure for an import of one project is probably excessive. I'd suggest just manually scripting the import.
<kb9vqf> Can they then export back to GIT?
<wgrant> No, that's not currently supported.
<wgrant> Though might be in a few months :)
<kb9vqf> Then I'll hold off--once that support is in I think we're good to go on the translation end
<kb9vqf> Most of the other open-source translation projects leave a lot to be desired
<kb9vqf> *translation infratructure projects
 * kb9vqf hopes there are no major database changes in the next 6 months so that he can upgrade his Launchpad instance more easily
<kb9vqf> wgrant: one more question: what exactly is the fatsam database anyway?  I'm assuming it's the binary data store but never could find any information on it
<wgrant> fatsam is the very old (like, 2004) name for the librarian.
<wgrant> The directory in the dev config never got renamed.
<wgrant> librarian is just a blob store. The filesystem structure is rather trivial.
<wgrant> For LibraryFileContent.id == 0xdeadbeef, the file is at /de/ad/be/ef
<kb9vqf> Yeah I saw the trivial structure (filelight FTW) but wanted confirmation
<wgrant> All the build logs, package files, etc. will be stored in that tree.
<kb9vqf> Doesn't it seem a bit dangerous to have "make schema" wipe it out?
<kb9vqf> Given that it's the heart of the librarian I would expect some guards on its destruction :-)
<wgrant> make schema hardcodes the path from the dev config.
<wgrant> Which is like /var/tmp/fatsam
<wgrant> It's in /var/tmp; it's fair game.
<kb9vqf> Ah, ok
 * kb9vqf goes to move fatsam out of /var/tmp on the upgraded system...
<wgrant> Heh
<kb9vqf> There's not much of a guide on how to run a properly installed Launchpad instance; I'm just glad the thing didn't blow up fatally in the past four years :)
 * kb9vqf also learned a lot
<kb9vqf> wgrant: Did you have a chance to assemble those upgrade scripts yet? :-)
 * kb9vqf has the database at 2208 on the test system; ready to proceed
<kb9vqf> wgrant: ping
<kb9vqf> wgrant: ping
<kb9vqf> wgrant: If you haven't written the update SQL yet you don't need to do anything for patch-2209-00-1.sql; I got somewhat impatient and banged out the 4 lines of SQL already. ;-)  I don't know what to do for the other patches so I'll leave them up to you.
<kb9vqf> wgrant: ping
#launchpad-dev 2014-10-30
<kb9vqf> wgrant: ping
 * kb9vqf found some of the custom upgrade python...sitting in the garbage collector....
<wgrant> kb9vqf: You're running an old version of garbo?
<wgrant> All the post-2209 bits should *probably* be there, if you can get them to run.
<kb9vqf> wgrant: hi there!
<kb9vqf> any chance of getting the SQL or should I just plow ahead?  I'm waiting at 2209-40 looking at the garbo code to proceed
<kb9vqf> (and yes I pulled an old rev from bzr for 2209-40)
<wgrant> Which garbo jobs have you run so far?
<wgrant> And what SQL have you run for 2209-00-1 and 2209-01-1?
<kb9vqf> wgrant: No garbo yet, you popped on just as I was setting up the run
<kb9vqf> Let me grab the SQL I ran...
<kb9vqf> (this is all on a test system so I can blow it away and start over if needed)
<kb9vqf> UPDATE sourcepackagepublishinghistory set sourcepackagename = (SELECT sourcepackagename FROM sourcepackagerelease WHERE sourcepackagerelease.id = sourcepac$ DELETE FROM sourcepackagepublishinghistory WHERE sourcepackagepublishinghistory.sourcepackagename IS NULL; UPDATE binarypackagepublishinghistory set binarypackagename = (SELECT binarypackagename FROM binarypackagerelease WHERE binarypackagerelease.id = binarypac$ DELETE FR
<kb9vqf> Hmm IRC ate that
<kb9vqf> let me try again....
<kb9vqf> UPDATE sourcepackagepublishinghistory set sourcepackagename = (SELECT sourcepackagename FROM sourcepackagerelease WHERE sourcepackagerelease.id = sourcepackagepublishinghistory.id);
<kb9vqf> DELETE FROM sourcepackagepublishinghistory WHERE sourcepackagepublishinghistory.sourcepackagename IS NULL;
<kb9vqf> UPDATE binarypackagepublishinghistory set binarypackagename = (SELECT binarypackagename FROM binarypackagerelease WHERE binarypackagerelease.id = binarypackagepublishinghistory.id);
<kb9vqf> DELETE FROM binarypackagepublishinghistory WHERE binarypackagepublishinghistory.binarypackagename IS NULL;
<kb9vqf> wgrant: ^^
<kb9vqf> The database is currently sitting at 2209-40-12
<kb9vqf> *2209-40-2
<wgrant> kb9vqf: Why are those DELETEs there?
<wgrant> Every SPPH has a BPR, so they should never delete anything.
<kb9vqf> wgrant: Over time some orphan rows crept in, probably during one of the numerous crashes
<wgrant> Oh, no, your query is totally wrong :)
<wgrant> UPDATE sourcepackagepublishinghistory set sourcepackagename = (SELECT sourcepackagename FROM sourcepackagerelease  WHERE sourcepackagerelease.id = sourcepackagepublishinghistory.id);
<kb9vqf> ok :)
<wgrant> SPR.id != SPPH.id
<wgrant> The condition should be 'WHERE sourcepackagerelease.id = sourcepackagepublishinghistory.sourcepackagerelease'
<wgrant> And similar with BPPH.
<kb9vqf> OK, well I did what I could with my limited knowledge of Launchpad :-P
<kb9vqf> Looks like I'll be reloading the test DB then
<kb9vqf> I can bring it back up to schema 2208 pretty quickly...having the proper SQL to execute would save lots of time ;-)
<kb9vqf> that is, time going forward applying 2209-00- etc.
<kb9vqf> wgrant: wait a minute, the SQL you posted is identical to mine
<kb9vqf> nvm
<kb9vqf> I can't read
<kb9vqf> :-P
 * kb9vqf is juggling two tasks ATM...be back in 10 mins when I can concentrate soeley on this
<kb9vqf> wgrant: I got to thinking...I don't need to dump/reload the DB on the test system; I'll worry about fixing up those two columns tomorrow.  How should we proceed on the rest of the migration?
<kb9vqf> wgrant: I'm back; if you want to work on this now give me a poke :-)
<wgrant> kb9vqf: http://paste.ubuntu.com/8741031/
<kb9vqf> Thanks!  That'll save a bunch of time
<kb9vqf> wgrant: I notice you set searchable* to empty.  Will this impact the system with regard to the old published packages?
 * kb9vqf is not sure what the searchable fields are used for
<wgrant> They're used for two things: https://launchpad.net/ubuntu/vivid/+queue (only for the primary and partner archives), and the getPackageUploads API call when using text search terms.
<wgrant> Neither of which are likely to affect you.
<kb9vqf> Right, OK.  I ask because I had the modified garbage collector ready to run, so if there's any advantage I'll just fire it off during the migration process
<wgrant> If you can get it to run, then by all means.
<wgrant> You should see no negative effects from either approach for this patch. This is the only one where I took a shortcut, as it was somewhat difficult to implement it properly in pure SQL.
<kb9vqf> Yeah I was looking at that and aborted real fast as soon as I saw the python
<kb9vqf> I think I can get it to run, so I'll use that approach
<kb9vqf> Thanks again; I'll apply these patches and should know sometime tomorrow if it all worked
<wgrant> :)
 * kb9vqf goes off to redo the sourcepackagename column
#launchpad-dev 2014-10-31
<kb9vqf> wgrant: Just wanted to let you know that (after some massaging and a couple of bug fixes) your patches worked great!  My test instance is on the latest source without any apparent data loss.  I was also able to get the search terms python code to work, so this should be a fairly clean upgrade during our next maintinance window.
<kb9vqf> Though I did notice that the OpenID code had changed enough to break compatibility with our old provider (sreg), so I spent the past day or so fixing and cleaning up a brand new provider based on the Prairie open source provider.  With a small patch to Launchpad to support the different URL format the new provider works great; if anyone wants to deploy launchpad and needs the code just ask. ;-)
<wgrant> kb9vqf: Excellent news!
#launchpad-dev 2015-10-27
<blr> would be nice to have a documentation url, similar to the wiki, homepage, screenshots and download urls.
#launchpad-dev 2015-10-28
<blr> morning
#launchpad-dev 2015-10-29
<bigjools> heh, someone went and made another LP http://www.breakingnewsnews.org/
<blr> bigjools: err...
<bigjools> and it's not using SSL
<wgrant> That's not another LP.
<wgrant> That's a sed over LP.
<cjwatson> bigjools: It's much weirder than that
<wgrant> It's the live site AFAICS.
<cjwatson> Yeah, that
<wgrant> Except s/Launchpad/Breakingnewsnews/
<wgrant> wtf
<bigjools> huh yeah I wondered why it had the same bugs
<wgrant> bigjools: How did you find it?
<bigjools> Appeared in a Google search
<bigjools> which is a little worrying
<wgrant> Oh.
<wgrant> This isn't you trolling us? :)
<wgrant> Look at the whois on that domain.
<bigjools> heh I know
<bigjools> not even I would go to that trouble to troll you
<wgrant> What on earth.
<blr> oh of course it's a Queenslander.
<bigjools> this isn't that nutjob we had to ban I wonder ...
<wgrant> My initial thought was amhnews, indeed.
<wgrant> The entire domain is aliased.
<blr> bigjools: is Nobby Beach a real place?
<wgrant> http://qastaging.breakingnewsnews.org/
<wgrant> http://blog.breakingnewsnews.org/general/launchpad-news-august-2015
<bigjools> rings a bell
<bigjools> wgrant: WTF!
<blr> only in Australia would you have a place called Nobby Beach.
<bigjools> There's a place in England called Bell End
<blr> oh dear
<wgrant> http://pqm.breakingnewsnews.org/ doesn't work :(
<bigjools> I used to live near a place called Minge Lane
<wgrant> I'm increasingly convinced it is amhnews, as it's so inexplicable and has news in the name.
<wgrant> amhnews is the master of things that make no sense at all but appear not to be trolling.
<wgrant> He's been gone for like a year now, though.
<bigjools> I am sure a redirect can be placed in the Canonical Apache configs
<blr> seems like the work of a rogue, and somewhat confused AI.
<bigjools> redirect to goatse? :)
<bigjools> Also if you're logging request headers, maybe check for others...
<cjwatson> It also inserts iframes into the pages it serves, and strips LP's JavaScript.
<cjwatson> (And does other reprocessing of the "pass through parser and back" kind)
<cjwatson> bigjools: blocking requested
<bigjools> eek
<bigjools> thanks cjwatson
<cjwatson> I think I've got about as much as I can out of logs
<bigjools> are you logging request urls?
<cjwatson> yes
<cjwatson> well, paths, not URLs
<bigjools> ah - I was thinking you can block where the requested site is not launchpad.net
<cjwatson> no, it's a proxy
<bigjools> oh, ok
<cjwatson> it's requesting launchpad.net as far as we're concerned
<bigjools> jeez, that's evil
<cjwatson> and then filtering (and caching) the results and returning them
<cjwatson> if it weren't requesting launchpad.net or similar then our Apache would say nope
<cjwatson> (not that I've actually checked but that seems highly likely)
<bigjools> the original request header is in the browser request header still?
<cjwatson> I don't see why they'd do that
<bigjools> unless the proxy is munging that too
<cjwatson> this is going to be more like a client/server pair than a legitimate proxy
<cjwatson> it will be making its own requests constructed based on the ones it receives
<bigjools> right
<bigjools> what on earth is someone's motivation for that I wonder
<cjwatson> my ticket title includes the string "incomprehensibly weird"
<bigjools> :)
<cjwatson> (I can certainly tell it's making its own requests to at least some extent, based on User-Agent)
#launchpad-dev 2016-10-31
<replaceafill> join #pyramid
#launchpad-dev 2016-11-01
<stub> Waiting for http://summit.ubuntu.com/uos-1611/meeting/22711/building-snaps-in-launchpad/ to appear, so I can pass it on to interested parties :)
<cjwatson> Yeah, I still need to write a skeleton presentation for that ...
#launchpad-dev 2017-10-30
<cjwatson> wgrant: I ran into an interesting issue when doing some casual Python 3 porting.  lib/lp/services/webapp/tests/test_servers.py:TestWebServiceRequestToBrowserRequest.test_unicode_path_info tests that Unicode is permitted in PATH_INFO.  PEP-3333 explicitly forbids this (or rather, it allows Unicode but only the bottom 256 codepoints, and anything else must be MIME-encoded), and zope.publisher ...
<cjwatson> ... 4.0.0 enforces this at least in some places.  Do you think we should adjust the test in this case?
<cjwatson> I can't think of a case where we'd actually need anything above U+00FF in PATH_INFO; surely practically everything there is a name or a fixed segment of some kind.
<wgrant> cjwatson: Filenames, mostly.
<cjwatson> Don't they get encoded?
<wgrant> You'd think so, but it's possible lazr.restful decodes them at some point.
<wgrant> Or zserver does
<cjwatson> I mean if they don't undergo quoting then we must have other bugs
<wgrant> It sounds like it's probably safe, but it needs checking
<cjwatson> Yeah
<cjwatson> It's conceivable we'd need to upgrade ZTK first, I suppose
<cjwatson> Also for bonus points py2 urllib.(un)quote and py3 urllib.parse.(un)quote don't behave the same way for non-trivial Unicode
<cjwatson> >>> unquote('%D7%90')
<cjwatson> '×'
<cjwatson> ^ py3
<cjwatson> >>> unquote('%D7%90')
<cjwatson> '\xd7\x90'
<cjwatson> ^ py2
<cjwatson> and the other way is a KeyError in py2
<cjwatson> so we need to make sure to consistently encode-then-quote and unquote-then-decode
#launchpad-dev 2017-10-31
<cjwatson> I suspect lazr.restfulclient may not be sufficiently careful, although that probably doesn't matter for this
<cjwatson> though actually that might be fine if it's only opening the filename and possibly stuffing it back into a header as a UTF-8-encoded bytestring
<tsimonq2> Hmph, so it seems I can no longer `make schema`...
<tsimonq2> While:
<tsimonq2>   Installing scripts.
<tsimonq2>   Getting distribution for 'pytz==2017.2'.
<tsimonq2> Error: Couldn't find a distribution for 'pytz==2017.2'.
<tsimonq2> And that's installed.
<tsimonq2> Oh. Duh. 2017.3 is installed. Let's manually edit that file...
<wgrant> tsimonq2: bzr up download-cache
<wgrant> LP doesn't use many system Python dependencies.
<wgrant> It gets them from download-cache
<tsimonq2> wgrant: Oh. Thanks.
<tsimonq2> wgrant: Did I overlook this in the docs somewhere or should it go in there? :)
<wgrant> tsimonq2: rocketfuel-get does it
<tsimonq2> Oh, gotcha.
<tsimonq2> cjwatson: Ok, so we agreed on doing the factory operation for bug 439470. So should someone be able to call CveSet via the API with a CVE number and it will automatically set it, should I give each (valuable) function its own thing to be able to access via the API, or should the database addition function be the only one that's publicly accessible?
<mup> Bug #439470: Cannot attach currently-unknown CVEs via linkCVE() <api> <lp-bugs> <platform-want> <Launchpad itself:In Progress by tsimonq2> <https://launchpad.net/bugs/439470>
<tsimonq2> cjwatson: I *could* be forgetting a little bit of our convo but I don't have the logs to grep any more and I don't remember when it was...
<cjwatson> tsimonq2: I'd probably be economical: the only new thing exposed on the API would be CveSet.new (as cves.new).
<cjwatson> tsimonq2: Webservice type declarations would get difficult if you tried to make e.g. linkCve take either an actual Cve object or a string.  If you wanted to do that I think you'd need to give linkCve an extra string-flavoured argument and require exactly one of cve or the new arg to be passed.
<cjwatson> tsimonq2: But I'd be inclined to do just the minimal functional thing as a first pass.
<tsimonq2> Ack cjwatson, thanks!
<cjwatson> wgrant_: codetree seems to work well; just needs copying into the relevant places and a shim in update-sourcecode to handle forcing HTTP (useful e.g. for buildbot).  Will poke as time permits.
<cjwatson> Parses our config file unchanged.
<wgrant> Handy.
