#launchpad-dev 2009-03-08
<bcurtiswx> Hi all, im a bug traiger for ubuntu.  I'm wondering how possible it would be to gather OS version and package version (depending on a package is provided in the bug report) to bug reports on submit.  this i guess would regard the manual submissions, as apport on ubuntu will report this automatically.
<ScottK> bcurtiswx: You want #launchpad.
<bcurtiswx> ScottK: ah, ok.. thought this was more of a future dev... apologies. thx
<ScottK> This is more after it's open sourced stuff should happen here.
<bcurtiswx> ScottK: gotcha.. thx
#launchpad-dev 2010-03-08
<thumper> mwhudson: I checked out the brothers hotel, it looks really nice
<thumper> mwhudson: nice rooms, wifi reaches and isn't charged for, no kids, breakfast included, only 15 rooms in the entire place
<thumper> mwhudson: quiet, nice views
<thumper> mwhudson: probably similar pricing to the southern cross
<mwhudson> thumper: sounds good to me, do they have room for us?
<thumper> mwhudson: yep
<thumper> mwhudson: I've got him to book them already
<thumper> mwhudson: I'm going to email flacoste and marianna with the details
<mwhudson> thumper: nice
<thumper> mwhudson: hopefully get sign off before the end of the week
<thumper> mwhudson: email sent
<thumper> mwhudson: fingers crossed for a quick approval
 * mwhudson zooms into town for a short while
<lifeless> zoom zoom zoom
<thumper> mwhudson: https://blueprints.edge.launchpad.net/launchpad-code/+spec/personal-branch-url-change
<thumper> mwhudson: https://blueprints.edge.launchpad.net/launchpad-code/+spec/code-import-watchdog, really deployment?
<wgrant> thumper: Bazaar follows 403s? Do you mean 301s/302s?
<mwhudson> that wasn't exactly a zoom :(
<mwhudson> thumper: yes
<mwhudson> oh, no
 * mwhudson misunderstood blueprint statuses, how did i manage that
<wgrant> Argh pagetitles.py.
<thumper> wgrant: probably
<thumper> wgrant: did you edit the page?
<wgrant> thumper: No.
<thumper> wgrant: I've edited the page now, thanks
<wgrant> Great.
 * wgrant points pointedly at the project reactivation request in #launchpad.
 * thumper goes to make dinner
<thumper> wgrant: thanks
<lifeless> re the project reactivation request
<lifeless> seems that folk misunderstand the 'registry' aspect of LP a lot
<lifeless> 'shutting it down' because its in  LP doesn't make a lot of sense.
<wgrant> LP misunderstands the Registry aspect of LP a lot.
<lifeless> :P
<lifeless> L:
<wgrant> It doesn't handle placeholder projects well.
<lifeless> ack. Have a tie fighter. or two. |-<>-| +-o-+
<wgrant> /Now/ I'm confused.
<lifeless> ;P
<poolie> hi
<poolie> mthaddon: in bug 534066 james_w says
<mup> Bug #534066: can't update bugtask importance via api <api> <Launchpad Bugs:New> <https://launchpad.net/bugs/534066>
<poolie> "OK, please ask a LOSA to check if all the edge appservers are on the same revision currently"
<poolie> spm^^
<poolie> could one of you do that please?
<mthaddon> poolie: they are, yes (the same as lpnet/production)
<poolie> k thanks
<poolie> mthaddon: separately, re my recent tuolumne merge, michael said
<poolie> the next step is setting up the datfiles (if they haven't been), graphs, and their associations.
<poolie> could you tell me how to do that?
<lifeless> poolie: your sql queries should be generating the datfiles; the db record for the datfiles can only by done by someone with web admin access
<poolie> yep, tom's helping me
<poolie> lifeless: in case you're interested, a slightly old version was accidentally merged that didn't record them properly
<adeuring> good morning
<wgrant> noodles775: Morning.
<noodles775> Hi wgrant!
<wgrant> noodles775: Can you please have a look at bug #534216 and maybe give some suggestions?
<noodles775> Suer.
<noodles775> Sure.
<noodles775> wgrant: pleasantly surprised that it's not a security bug :) So, +1 for the suggested breadcrumb urls, and regarding the second question,
<wgrant> noodles775: I decided that you didn't need any more of them in one week. :)
<noodles775> Why does it matter if two builds have the same breadcrumb (it's not a url),  and the last breadcrumb (the i386) won't be a link anyway.
<noodles775> heh.
<wgrant> Right, it's not really important.
<wgrant> And it's a rare case anyway.
<noodles775> Yep, so +1 from me :).
<wgrant> Great, thanks.
 * wgrant quickly implements.
<noodles775> wgrant: are you against there being a 'Packages for..." breadcrumb after the PPA title?
<noodles775> (ie. look at the breadcrumb when you traverse to +packages)
<wgrant> noodles775: Ah, yes, I thought about that earlier but it slipped my mind.
<noodles775> Great.
<wgrant> There should be, and probably a Builds breadcrumb as well. But that needs a custom hierarchy thingy.
<noodles775> Yeah, although the Builds breadcrumb is a nice-to-have-but-less-important IMO, as I'm assuming that most of the time people get to an individual build directly from the +packages page, rather than searching from +builds.
<wgrant> Right, probably.
<mrevell> Morning
<jtv> hi mrevell
<mrevell> hey there jtv
<jtv> henninge: I just updated RunningSoyuzLocally.  It refers to your buildd instructions (I much prefer the chroot, obviously!) and it strips out lots and lots of manual work.
<henninge> HowToUseSoyuzLocally
<jtv> Oh, sorry
<jtv> henninge: why all the trouble with the ntp IP address?  Doesn't pool.ntp.net work?
<henninge> jtv: I just couldn't remember if I can put other host names into /etc/hosts ...
<henninge> like: "pool.ntp.net  ntp.buildd"
<wgrant> henninge: Or just edit /etc/launchpad-buildd/default
<jtv> copy the ntp setup from the host?
<jtv> I mean, we do ship with a workable default I suppose
<noodles775> I've already got testopenid.dev in my /etc/hosts, but can no longer login on the devserver. Should I debug this or has it already been resolved?
<wgrant> noodles775: What happens when you try?
<noodles775> wgrant: http://pastebin.ubuntu.com/390927/
<wgrant> Um.
<wgrant> What?
<wgrant> That's not one I've seen before.
<noodles775> Yeah, me either. That's after logging out and trying to log back in on r10429
 * wgrant pulls.
<noodles775> ah, maybe I should update my source deps...
<noodles775> hrm, no that shoudl have been done by rf-get.
<wgrant> noodles775: Note that update-sourcecode is broken in lucid.
<wgrant> You need to change its interpret to the system Python.
<noodles775> ah thanks wgrant.
<wgrant> s/interpret/interpreter
<adiroiban> I have just updated lp/devel and the test from lib/lp/registry/stories/webservice/xx-project-registry.txt is failing http://paste.ubuntu.com/390930/
<wgrant> noodles775: Can I convince you to review the build breadcrumb fix along with another small old branch of mine to fix the titles and breadcrumbs of build listings and queue pages? Or should I use OCR?
<noodles775> wgrant: I'd love to, but would prefer the OCR right now as I'm a bit behind with a few other things.
<wgrant> noodles775: OK, no problem.
<wgrant> Thanks.
<noodles775> wgrant: btw, I assume you can login/out fine on r10429?
<wgrant> noodles775: Still updating/rebuilding.
<noodles775> wgrant: ah, thanks.
<noodles775> utilities/update-sourcecode appears to work fine here (on lucid).
<wgrant> Hm, odd.
<wgrant> It can't import bzrlib here, because it's running with 2.5 which is no longer in Lucid.
<noodles775> Mine's getting it from lp-sourcedeps/eggs/bzr-2.1.0-py2.5-linux-x86_64.egg/bzrlib/__init__.pyc
<henninge> wgrant, jtv: sorry, had a call
<henninge> wgrant: I guess, because I was trying out the buildd package, I didn't want to change any files in the package, rather change the environment to work with the package.
<noodles775> adiroiban: that's kinda weird given that the project entries are sorted by displayname just a few lines above.
<wgrant> henninge: Well, a config file is pretty close to the environment.
<doctormo> jml: https://blueprints.launchpad.net/launchpad/+spec/launchpad-desktop
<henninge> wgrant: yeah, I guess that's true.
<noodles775> adiroiban: ah, sorted by display name, not project name, so it looks like you could fix it by ensuring it's sorted on both?
<adiroiban> noodles775: ok. so this only happend on my computer?
<jtv> henninge: what you wrote is particularly useful to me because I wouldn't have taken the time to figure it out otherwise.  As I follow your trail I'll try to automate more; I'd love it if anyone could do all this.
<adiroiban> noodles775: ok. so this only happens on my computer?
<noodles775> adiroiban: the test passes here on r10429, but it looks spurious - sorting by both should ensure it will always work.
<adiroiban> noodles775: ok. I found it while working for bug 531261. Should I fix it in a new branch or together with this one?
<mup> Bug #531261: Move ISeriesMixin to lp.registry.interfaces.series <cleanup> <iseries> <tech-debt> <Launchpad Registry:In Progress by adiroiban> <https://launchpad.net/bugs/531261>
<jtv> henninge: would it make sense to make all the changes needed for a working setup in the chroot, then publish those changes as a package?
<wgrant> jtv: What changes are there, apart from the NTP host?
<noodles775> adiroiban: if your branch consistently fails with it, you'll need to fix it there (I thought you were testing your devel branch?). If it's a quick fix, I'd included there anyway.
<jtv> wgrant: whatever setup will be needed to check out a branch from the dev machine
<wgrant> jtv: Hmm.
<jtv> Can be done with ssh setup, but that requires your personal ssh key.  :/
<jtv> Another way is to copy the branch into the chroot's fs, but that has to be done from the outside.
<jtv> I guess it doesn't help for that.
<jtv> So never mind.  :)
<adiroiban> noodles775: well, I found it while working on that branch, but then I also tested with a fresh devel
<noodles775> adiroiban: Right.
<wgrant> bigjools: So, it turns out buildd-manager is really over-complicated...
<bigjools> wgrant: turns out?  We've known that for ever :)
<bigjools> if we had the time I'd re-write it
<bigjools> but given the pain last time it was re-written, it would be hard
<wgrant> bigjools: You know how it does that thing where it gets all the builders for each DAS, adds them to an internal strange DBNote thing, then eventually iterates of each builder in each DAS... that can be replaced with a 3-line for loop.
<bigjools> yes the dbnote stuff is utter crack
<bigjools> was part of the original code, from 5 years ago
 * bigjools -> otp
<jml> mrevell, hi
<mrevell> Hello Mr jml
<doctormo> jml: Are you are awake!
<wgrant> bigjools: Yeah, well, it's easy to get rid of. I was looking at what needed to be moved from buildmaster to soyuz and vice-versa, and realised that all of that code could die. A bit of refactoring later, it was replaced with three lines and everything was a few hundred lines shorter.
<bigjools> buildmaster should mostly die
<wgrant> Well, I got rid of all but master.py.
<bigjools> \o/
<wgrant> So only the twisted stuff lives outside the model.
 * jml interested in wgrant & bigjools conversation
<jml> doctormo, I am.
<doctormo> jml: See link to blueprint and bug it's linked to
<jtv> hey doctormo, à¸ªà¸à¸²à¸¢à¸à¸µà¸«à¸¡à¹
<wgrant> jml: Basically, buildd-manager scared me, so I deleted most of it.
<jml> wgrant, destructive rampage is sometimes an appropriate response to fear.
<doctormo> jtv: à¸à¹à¸à¸µ
<jtv> doctormo: glad to see you haven't lost it :)
<jtv> Where does the chroot go?  Arbitrary /tmp subdir?
<jtv> wgrant, henninge?
<wgrant> jtv: Doesn't matter.
<henninge> jtv: which chroot?
<henninge> pbuilder's goes into /var/cache/pbuilder/build/<buildno>
<jtv> ah, thanks
<wgrant> Oh, right, pbuilder.
<henninge> jtv: but you can specify it on the command line for pbuilder
<jtv> I'm asking because the only sane way I can think of for getting a branch in there is to copy it into the chroot from the outside.
<jtv> henninge: mind if I extend the instructions for that at some point, and pick a fixed location?
<wgrant> jtv: There is a Makefile target to construct demo branches.
<jtv> Makes it so much easier to write ready-to-go instructions.
<wgrant> jtv: Can't you use that, then get the slave to check it out normally?
<adiroiban> henninge: any idea why this test is passing on my devel computer, but alwasy failing on my testing computer http://paste.ubuntu.com/390946/ ?
<jtv> wgrant: I guess we could install the lp bzr plugin on the slave...
<jtv> wgrant: creating the branch itself isn't the issue, but getting it onto the simulated slave.
<wgrant> jtv: Your dev appserver must be able to generate direct HTTP URLs, or it could not resolve lp: URLs.
<henninge> jtv: do you want to copy stuff into the chroot? Or why do you need the location?
<wgrant> Therefore it can also pass HTTP URLs into the slave.
<persia> If you're talking about building code from arbitrary branches in chroots, why invoke pbuilder?  Isn't there already a sbuild inside Soyuz?
<henninge> adeuring: it's missing intltool
<jtv> wgrant: it generates a direct http url, but that won't work on a dev system will it?
<henninge> adiroiban: ^
<henninge> adeuring: moin ;-)
<wgrant> persia: This pbuilder is just to install the buildd code into on dev machines.
<wgrant> jtv: It should for dev, but not for tests.
<wgrant> In fact, yes, it does.
<wgrant> because recipe builds work.
<henninge> adiroiban: you need to install launchpad-developer-dependencies on the testing computer
<wgrant> So HTTP access works fine.
<jtv> wgrant, henninge: _running_ the buildd code on dev machines, right?
<wgrant> jtv: Right.
<adiroiban> henninge: thanks. I will do. I guess the package in installed, but not updated
<persia> wgrant: But why pbuilder?  That just adds wider support.  schroot can handle tarballs, etc. just fine.
<jtv> wgrant: can you show me an appropriate http url for dev access?  I tried before but couldn't make it work.
<wgrant> persia: Don't ask me! I use schroot.
<wgrant> jtv: http://bazaar.launchpad.net/~path/to/branch
<wgrant> Er.
<wgrant> s/net/dev/
<persia> Who suggested pbuilder for this?  I'd like to disabuse them :)
<jtv> hmm... that's what I thought didn't work earlier
<henninge> persia: me, I simply don't know about schroot ...
<henninge> wgrant: that's just the way I got it to run
<henninge> I am not saying that that is a canonical way.
<persia> henninge: So let me know when you have time for a /query about the benefits of only supporting one chroot management system in the buildd environment (including the dev environment).  Until then, I'm happy to answer questions about schroot or point at code to achieve what you seek.
<wgrant> persia: Note that Soyuz's sbuild doesn't use schroot.
<persia> wgrant: Yes, but not also using pbuilder makes the future alignment less painful :)
<wgrant> persia: Potentially, yes.
<henninge> persia: how do you build your chroot environment?
<henninge> or, were do you get it from?
<persia> `mk-sbuild lucid`
<henninge> s/were/where/
<henninge> oh, new in lucid?
<henninge> persia: ^ ?
<wgrant> mk-sbuild-lv has been around for a few releases, but I think mk-sbuild is reasonably new.
<persia> For non-LVM use cases, yes.
<persia> But the code ought to work in karmic, it just didn't get published then.
<jtv> wgrant: not having any luck with those http URLs...  maybe I need to re-do my setup or something
<wgrant> jtv: You've run make sync_branches?
<jtv> yes
<jtv> just says "not a branch" for every variant of the URL that I can think of
 * wgrant tries.
<jml> doctormo, that looks sane to me. I'll bounce it off gary & leonard
<jml> doctormo, you're in US east coast, right?
<doctormo> jml: souns good
<doctormo> jml: In Boston, MA, yes
<jml> doctormo, cool. gary & leonardr are in the same tz, so hopefully after I talk to them you can avoid relying on someone on the other side of the atlantic :)
<doctormo> jml: Agreed.
<wgrant> jtv: Do you have anything in /var/tmp/bazaar.launchpad.net except for push-branches?
<jelmer> jml: hi
<jml> jelmer, hi
<jelmer> jml: "ec2 land" is reliably losing email for me while "ec2 test" seems to mail back without problems, do you have any idea what the cause of that might be ?
<jtv> wgrant: a few things besides that... a log listing some bzr+ssh sessions, and directories config & mirrors
<jml> jelmer, no, sorry. I know that you aren't the only person to experience this behaviour though.
<jtv> wgrant: config/launchpad-lookup.txt, which is empty
<jelmer> jml: ok, I'll see if I can find something.
<jtv> wgrant: and mirrors/00/00/00/4d/.bzr
<wgrant> jtv: ps aux | grep rewrite
<jtv> wgrant: not running
<wgrant> I bet your branch-rewrite.py is failing due to that PATH error.
<jtv> (isn't aux bsd syntax?)
<jtv> what PATH error?
<wgrant> jtv: shhh I've been using that for 9 years years. old habits, etc.
<jtv> wgrant: just admit it, you picked it up while working on darwin
<wgrant> jtv: If you restart Apache, wait a few seconds, and check the error log you should see it.
<wgrant> jtv: I've not used OS X for more than half an hour, actually.
<jml> jelmer, thanks.
<jtv> wgrant: "oh, it was _that_ long ago?"  :)
<jml> jelmer, if you want to debug out loud, feel free to call me.
<jelmer> jml: ok - thanks
<jtv> wgrant: indeed, a beautiful traceback mentioning PATH
<wgrant> jtv: If you see the PATH KeyError, remove the WindmillTestClient import from lib/lp/testing/__init__.py in the branch mentioned in the traceback.
<wgrant> Then restart Apache, wait a few seconds, and try the branch again.
<jtv> wgrant: thanks... that's pretty horrible!
<wgrant> Yes.
<wgrant> I'm not sure if there's a bug.
<jtv> wgrant: something's clearing PATH... that does have the ring of a bug to it
<jtv> (And by the same token, how difficult is it to treat that as an empty string?)
<wgrant> Well, perhaps a better question is why WindmillTestClient is being imported there, and/or why it needs a PATH.
<jtv> wgrant: a less invasive fix perhaps is to use os.environ.get('PATH', '') instead of os.environ['PATH'].  Seems to work
<wgrant> jtv: Pfft.
<jtv> Still doesn't work though.  :(  Internal server error.  There's stuff in the apache error log, but nothing that looks like an error.
<wgrant> It's not trying to redirect to a non-running Loggerhead, is it?
<lifeless> jtv: for an ISE, check the loggerhead log file
<jtv> is that in the branch or somewhere in /var/log?
<wgrant> jtv: It works for me after I comment Windmill imports out of two files.
<jtv> wgrant: then I'll stop trying to be clever and just imitate.
<wgrant> jtv: Heh.
<deryck> Morning, all.
<wgrant> jtv: Try killing the windmill.bin.admin_lib import from lib/canonical/testing/layers.py too.
 * jtv looks for blunt instrument
<jtv> hi deryck
<jtv> wgrant: same result.  :(
<jtv> wgrant: yes, I did.
<jtv> (I assume you were about to ask if I restarted apache)
<jtv> wgrant: ah, I think I just edited in the wrong branch or something
<wgrant> I was, yes.
<wgrant> It's probably the last branch you ran 'make install' on, which is probably devel.
<jtv> "trunk" in my case, but yes.
<jtv> So I edited that, and still the same.  :(
<deryck> adeuring, I see you've take the doNotSnapshot-related OOPS card on the board...
<adeuring> deryck: yesÃ
<wgrant> Nothing in the error log a few seconds after restarting?
<adeuring> ...yes?
<wgrant> And branch-rewrite.py still not running?
<deryck> adeuring, just a pointer that the notes when you open the card list multiple bugs needing this.
<adeuring> deryck: yes, I've seen two bugs
<jtv> wgrant: branch-rewrite is running this time
<jtv> wgrant: maybe it's this...  ... waiting .Warning: DocumentRoot [/var/tmp/bazaar.launchpad.dev/static/] does not exist
<deryck> adeuring, great, thanks.
<wgrant> jtv: That's unlikely.
<wgrant> jtv: What happens when you browse to the branch in a browser?
<jtv> wgrant: with the http url?
<wgrant> jtv: And are you sure the branch exists in /var/tmp/bazaar.launchpad.dev/mirrors?
<wgrant> jtv: yeah.
<jtv> (btw after I created that dir & restarted apache, I got a _new_ complaint about /var/tmp/ppa not existing)
<deryck> adeuring, one may be a dupe of the other, but for some reason I kept the second open.  I think the OOPS looked slightly different.
<deryck> adeuring, if the second is a dupe, feel free to mark it so.
<jtv> wgrant: visiting the branch http URL in a browser I get 404
<adeuring> deryck: yes, I think it is worth to have a look at several properties, as sinzui notes in a comments on bug 505999
<mup> Bug #505999: Unsubscribe OOPS <oops> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/505999>
<adeuring> sigh, his comment is on bug 507642
<mup> Bug #507642: "Does this bug affect you?" link produces error <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/507642>
<deryck> adeuring, agreed.
<wgrant> jtv: You may find 'utilities/get-branch-info lp://dev/~path/to/branch' handy for working out paths.
<jtv> wgrant: that gives me an https url though
<jtv> (but thanks for the tip btw, nice to have)
<wgrant> jtv: That gives you an HTTPS path to the webapp, but it gives you filesystem paths to check.
<jtv> wgrant: ah yes, and indeed that's what I had in /var/tmp
<jtv> wgrant: seems I _can_ check out the branch from code.launchpad.dev!
<wgrant> jtv: But that should just redirect too bazaar.lp.d...
<jtv> wgrant: the http redirects to https, but for a local test that's fine
<jtv> actually, "https+urllib" not https
<jtv> and if the message is to be believed, still on code not bazaar
<wgrant> Huh.
<wgrant> That should be impossible.
<jtv> I've done something impossible already?  Great, then my working day is done.  :)
<jtv> oh, my chroot is ready
<wgrant> What was your chroot doing?
<jtv> setting up
<jtv> I had that going as a background task
<deryck> intellectronica, gmb -- everything should be landed for bug heat updating by now, right?
<intellectronica> deryck: for updating, yes
<intellectronica> is pqm open again? if so then i can also land my last branch for heat display
<deryck> intellectronica, so I marked a malone bug as affecting me... that should trigger a job for setHeat, which should recalculate max_bug_heat, too.  Is that right?
<intellectronica> deryck: correct
<deryck> intellectronica, hmmm, ok, so we may have a problem.  still all gray icons on the hots bugs list.
<wgrant> bigjools: Any objections to moving BuildStatus and BuildQueue to lp.buildmaster?
<deryck> intellectronica, and yes, pqm is open now.
<intellectronica> deryck: is that because setHeat is not being called, or because max_heat is not set?
<bigjools> wgrant: no it's fine
<deryck> intellectronica, max_bug_heat was not set, I assume.
<bigjools> consistency FTW
<wgrant> bigjools: I'm trying to get buildmaster's Soyuz dependencies under control, since at the moment it's more than a bit ugly.
<deryck> intellectronica, but if setHeat wasn't called, max_bug_heat wouldn't be set, right?  So it could be either.
<intellectronica> deryck: exactly
<bigjools> wgrant: I can imagine
<intellectronica> deryck: i'd bet on setHeat not called, because it would be hard to explain setHeat being called without recalculatMaxHeat being called at the end (without some error to account for that)
<deryck> intellectronica, right.
<deryck> intellectronica, let's check with gmb when he's available again and see if something obvious springs to his mind.
<intellectronica> sure
<jml> meh. code homepage timed out
<jml> memcache anyone?
<wgrant> Isn't it now just a matter of adding the magic attribute, since the TALES branch landed a day or two ago?
<jtv> wgrant: yup
<wgrant> Eeeexcellent.
<jtv> (and if the problem is querying for both private-but-visible-to-user and public branches in the same query, of picking the right caching mode)
<jtv> wgrant: how do I tell soyuz about the chroot tarball for an ubuntu release?
<jtv> the manage-chroot script?
<wgrant> I guess there aren't any caches active on edge, though, since the caching on the front page doesn't reduce the query count.
<wgrant> jtv: scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 -f something.tar.bz2 add
<jtv> wgrant: thanks
<jtv> wgrant: I don't think we're using these new caches anywhere yet
<wgrant> jtv: The front page now caches a few sections.
<jtv> wgrant: but the usual problem pattern isn't visible in query count; it's querying "where foo.private is false or is_visible_to(user, foo)"
<jtv> oh cool
<jtv> wgrant: oh wow, I think I've made a simulated slave accept a TranslationTemplatesBuildJob.
<wgrant> jtv: Checking out a branch and everything?
<jtv> wgrant: patience, patience!  I made it "accept" it.
<wgrant> Heh.
<jtv> wgrant: It went to Building and then Aborted, probably because of the things you already discovered with Henning
<wgrant> jtv: ... ABORTED? I hope it didn't end up there.
<wgrant> That should only happen when you call abort().
<jtv> wgrant: isn't that done for me when things fail horribly?
<wgrant> And even then, it's very unlikely to actually leave ABORTING, because it's buggy as hell.
<jtv> ah, the ntp setup was lost somewhere along the line
<wgrant> Odd.
<jtv> wgrant: got a whole lot of this documented... once we have it working and streamlined we can integrate it more.  https://dev.launchpad.net/Translations/BuildTemplatesOnLocalBuildFarm
<jtv> The page looks disappointingly short for the amount of BSW that went into it...
<jtv> Sorry, BST
<jtv> Blood, Sweat & Tears
<wgrant> jtv: What are you trying to do with buildd-queue-builder?
<wgrant> Whatever it is, you probably don't.
<jtv> wgrant: oh
<jtv> Then take that note as "I have to figure out what the right thing to do is, then do it."  :-)
<jtv> I'm trying to get soyuz to dispatch the job to the local slave.
<wgrant> buildd-manager will do that, and it's run by start-soyuz
<jtv> I suppose I'll also want to register the slave with soyuz somehow
<wgrant> Yeah, probably.
<wgrant> Does your code create the appropriate BuildQueue entry for your job?
<wgrant> Also, if you merge devel buildd-manager's logging will magically NO LONGER SUCK!
<wgrant> So you'll be able to see what it does.
<jtv> Yes, it creates a BuildQueue entry.
<wgrant> OK, so it should Just Work.
<jtv> No need to register the slave?
<wgrant> Not if it's running on localhost:8221.
<wgrant> But check out https://launchpad.dev/builders, anyway.
<wgrant> What is the state of the virtualized flag on your BuildQueue?
<jtv> oh!  that's probably True, innit?
<wgrant> True or None are what you want.
<jtv> I see a builder bob, and a disabled builder frog.
<wgrant> Right.
<wgrant> So...
<wgrant> Make bob virtualized.
<wgrant> By clicking the Virtualized checkbox, and entering anything at all into the PPA host box thingy.
<wgrant> And the build might start.
<jtv> BuildQueue.virtualized is null; there's also a "manual" flag there
<wgrant> (frog is on localhost:9221 and disabled, so buildd-manager won't be touching it)
<wgrant> Ignore the manual flag.
 * jtv tries
<jtv> ...and documents
<jtv> wgrant: leave Bob active?
<wgrant> jtv: If you disable it then you'll have no active builders, which is probably counterproductive.
<jtv> wgrant: Done.  I still see only the two builders.
<wgrant> jtv: Why would there be more?
<wgrant> You just adjusted bob to meet your needs.
<jtv> wgrant: oh, our local slave is going to _be_ Bob?
<wgrant> jtv: Yes.
<wgrant> bob is the primary local slave. frog is the secondary one that doesn't normally exist.
<jtv> Okay... but bob says it's building firefox-0.9 for hoary.
<wgrant> Well, that's pretty stupid of it.
<wgrant> Flip the OK checkbox off for 10 seconds.
<wgrant> That should fix that.
<jtv> You'd think that after a few years of attempted build, it'd get the hint.
<jtv> wgrant: bob is idle
<gmb> intellectronica, deryck: You rang m'luds?
<deryck> gmb, yeah, not sure bug heat is updating as it should...
<jtv> danilos, henninge: pieces of the puzzle are coming together on https://dev.launchpad.net/Translations/BuildTemplatesOnLocalBuildFarm
<wgrant> jtv: If you flip it back to OK, does it then start working?
<jtv> wgrant: that's after flipping it back to OK.
<danilos> jtv, woohooo, ready for me to try it out already? :)
<jtv> danilos: patience.
<deryck> gmb, I marked myself affected by a bug, expecting a job to be created, setHeat to be run, max_bug_heat to be updated...
<danilos> jtv, heh, ok, ok
<wgrant> jtv: Grrr. Merge devel, restart buildd-manager, and watch the logs.
<deryck> gmb, yet the flames icons are all gray still.
<gmb> deryck: Okay, have you checked with a LOSA to see if a job was created?
<deryck> gmb, no.  I can do that.  would it still be hanging around some 60 minutes later now to confirm?
<jtv> danilos: the page looks brief but plenty of effort went in (I probably removed more text from manual steps that I automated)
<gmb> deryck: I don't know. I'll walk through the process myself and have a LOSA query while I go.
<deryck> gmb, excellent.  thanks!
<jtv> wgrant: I think my bzr got swapped out...  Slow as molasses :)
<wgrant> jtv: Yeah, that happens :(
<wgrant> Particularly with eCryptFS.
<wgrant> And lots of Soyuz services.
<wgrant> And a slave VM or two.
<jtv> and criss-cross merges on devel
<wgrant> Urgh.
<jtv> stupid thing is, this should be an unmodified devel branch
<gmb> deryck: So, weirdly enough it looks like jobs aren't getting created. I just subscribed to bug 373683 but the oldest job in the DB is from the 5th. Lemme take a look at the source, see if I can figure out what's happening.
<wgrant> jtv: Any luck?
<deryck> gmb, thanks for looking into this
<gmb> np
<kfogel> deryck: I'd like to convert bug #345577 into a bidirectional "patch<->branch" equivalence bug.  Is that kind of expansion generally okay to do?
<mup> Bug #345577: Patches should be made into Bazaar branches <bug-branch-links> <code-review> <feature> <patch-tracking> <Launchpad Bazaar Integration:Triaged> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/345577>
<wgrant> noodles775: I'm glad we elected to not move BuildStatus when we created BuildBase during the sprint... just changing all the imports is a 1500 line diff.
<deryck> kfogel, yup
<kfogel> deryck: cool
<wgrant> jtv: Why do you run both upload2librarian and manage-chroot? The latter does the former.
<jml> kfogel, I think it's ok to do, but I'm curious -- why do you want to do it?
<kfogel> jml: because I feel that the equivalence is bidirectional, and this is the only bug in our database about it, so it should reflect that.
<kfogel> (rather than create a new bug for the other direction)
<jtv> wgrant: I just added the manage-chroot to an existing page...  but I think the URL is also needed further down on that page.
<jml> kfogel, what would that mean in terms of user-visible behaviour?
<kfogel> jml: oh, btw, when I see the "
<kfogel> Updating branch...
<kfogel> Launchpad is processing new changes to this branch which will be available in a few minutes. Reload to see the changes." message in the Launchpad web UI, that's irrelevant for ec2 / pqm, right?
<jml> kfogel, ummm.... I don't know, sorry.
<jml> kfogel, If they are reading the branch over SSH as someone who has write access, then yes it's irrelevant.
<kfogel> jml: what it would mean (I'm updating the desc to say this) is that also if someone links a branch to a bug, that it would be possible to get that branch in patch form, via the web UI without having to have bzr installed locally.
<intellectronica> kfogel: patches and branches aren't really equivalent, because patches lack the context for applying them. in practice, that makes them a lot less useful, and there's a lot less we can say about them with certainty.
<persia> Are chroots in Soyuz always .bz2 compressed?
<jml> kfogel, that sounds like a good thing, although maybe less desirable than the other way around
<kfogel> intellectronica: uhm.  Two things: one, I'm not completely sure that assertion is true in ways that are important here, but two, even if it were, the directionality in which it is important is the one already covered by the bug.  That is, the direction I'm adding (branch->patch) is the one where we *definitely* have enough information to make the transition.
<wgrant> persia: I think so.
<kfogel> jml: for an upstream dev it could be hugely important.  Now someone else makes a bzr branch against your code, and you can get it in a form you can actually use.
<persia> wgrant: Thanks.  I'll make that assumption in my code then.
<kfogel> (if you're not a bzr user)
<wgrant> persia: The buildd assumes that they are.
<intellectronica> kfogel: good point
<wgrant> persia: I really do not know why, but it bunzip2s the tarball before it untars it.
<kfogel> intellectronica: Also, I never thought I'd be standing here tonight, holding this little golden guy, and I'd like to thank all my teammates, and my mom and my dad, and especially Tom Berger for teaching me everything I know about JavaScript code with YUI.
<wgrant> Looks like they could also be uncompressed, but they never are in practice.
<kfogel> intellectronica: ooops, sorry, wires crossed -- was up too late last night.
<jml> kfogel, in that case, we'd probably want something a bit richer than patches for git & hg upstreams
<kfogel> jml: yes, that's part of the (blue-sky section of the) proposal :-).
<persia> wgrant: Autodetection of internal compression mechanism is relatively new (and may not have sunk into all developers heads)
<jml> kfogel, :)
<intellectronica> heh :)
<wgrant> persia: I guess this code is 5 years old, too.
<persia> I forget if that predates autodetection, but yeah.
<ricotz> Are there currently known problems with translation importing on launchpad?
<danilos> ricotz, not that I know of
<wgrant> jml: Will bug #534391 be fixed if an extra breadcrumb for the build is added, so the PPA breadcrumb becomes a link again?
<mup> Bug #534391: Build failure page for PPA has no link to PPA page <Soyuz:New> <https://launchpad.net/bugs/534391>
<danilos> ricotz, though, this is probably better asked in #launchpad (this channel is for development discussion)
<danilos> ricotz, I'd be happy to help diagnose the problem for you on #launchpad :)
<ricotz> danilos, ok
<jml> wgrant, I guess.
<wgrant> jml: Because I have a branch awaiting review that does just that.
<jml> wgrant, kaching!
<wgrant> Also, I've just noticed there is a link to the PPA on that page -- the top of the right column.
<jml> wgrant, oh I see.
<jml> wgrant, that's not a good place for a link, I feel.
<noodles775> wgrant: heh, sounds like the recent trivial fix I did to ensure the privacy of PPA's can't be switched once it's got publishings. 100line diff for the fix and 4.5k diff in total.
<wgrant> noodles775: Yeah, I saw that. Ouch...
<gmb> deryck: So, I'm going to go and write some tests to try and reproduce this; all of my testing so far suggests that subscribing or marking yourself as affected should create a job.
<deryck> gmb, ok.
<jml> sinzui, btw, I love the "Launchpad doesn't know which packages" thing on the package overview page! (just saw https://edge.launchpad.net/pino)
<jml> sinzui, I reckon the copy could do with a bit of tweaking, but otherwise it's very much what I hoped for.
<jelmer> jml: ooh, that is very neat indeed
<jml> sinzui, I guess it could also be more ajaxy too.
<wgrant> Can I tell that section to go away?
<wgrant> Lots of projects will probably never be packaged.
<sinzui> jml:We decided to postpone ajax for that feature until the core features were landed. Like its counter part on sourcepackages, the entire portlet could be an ajax call.
<jml> sinzui, that's a sound plan.
<aolmedo> hi, it's aolmedo from libresoft.es (spain)
<mrevell> Hi aolmedo
<allenap> abentley: I've just figured out the running-tests-in-a-subprocess issue you helped me with last week. The zope.testing TestResult object does layer setup and teardown if you can believe it, so replacing it with a non-Zope TestResult (i.e. subunit.TestProtocolClient) causes things to break a lot.
<allenap> abentley: I've worked around it using MultiTestResult for now.
<abentley> allenap, aha!  I'm glad you figured it out.
<bigjools> jml: dude, you don't look very hard to find links
<jml> bigjools, why should I have to look hard?
<bigjools> jml: well you don't :)
<bigjools> that's the point
<abentley> allenap, so why were you converting that test anyhow?
<jml> allenap, oh, I knew that. sorry I didn't actually think to help you with it.
<allenap> abentley: Because it was hanging when run in the same test run as one of my tests; both start the Twisted reactor.
<abentley> allenap, but as long as they both *stop* the twisted reactor, that shouldn't be a problem, right?
<allenap> jml: Hehe :) Oh well, I learnt a lot, and I don't have much hair left to tear out anyway :)
<jml> allenap, you might find https://code.edge.launchpad.net/~jml/zope.testing/subunit-output-formatter/+merge/19825 interesting
<allenap> abentley: They do, but for some reason they were not happy. I never figured that out. jml said that, in general, starting and stopping the reactor is not supported, so I just focused on running tests that need the reactor in a subprocess.
<abentley> jml, could you clarify?
<jml> abentley, starting the reactor after stopping it in the same process is not supported and expected to break. I don't know the details off-hand.
<abentley> jml, and this applies to any process that uses the reactor, not just Launchpad?
<jml> abentley, that's correct.
<jml> it's a deficiency of Twisted.
<henninge> jtv: you sounded like you were doing some investigating, too. Any useful information before your eod?
<jtv> henninge: I'm not eod, I'm still otp
 * henninge can't find a funny tla to respond .... ;)
<abentley> jml, I'm surprised that doesn't come up more often.  Back when we were talking about using Twisted in bzr, there was talk about converting async operations into synchronous ones, and I always assumed it would involve starting and stopping the reactor.
<jml> abentley, yes it would involve that.
<jml> abentley, trial gets around this by faking reactor start and stop in a really really horrible way that no one should ever use.
<jml> abentley, the Twisted community AIUI generally agrees that they should be restartable, but no one is interested in working on it right now.
<allenap> jml: Ah, so I could introduce a new layer that raises NotImplementedError in tearDown() for use with reactor tests?
<jml> allenap, maaaaaaaaaybe.
<abentley> allenap, thank you for correcting my oversight.
<allenap> abentley: Perhaps I should have told you why I was pursuing this earlier :)
<jml> allenap, I mean, I don't know what "reactor tests" means here.
<jml> allenap, we already have tests in the launchpad suite that subclass trial's TestCase and return Deferreds.
<allenap> jml: I mean anything that needs to start the reactor.
<jml> allenap, what sort of code do you have that calls reactor.start()?
<jml> .run
<jml> dammit
<james_w> one of my branches got somehow un-merged, how could that have happened?
<allenap> jml: Some tests for checkwatches; I want to <insert correct xunit term here> out lots of the actual networking stuff, but run the whole shebang up.
<jml> allenap, ok.
<abentley> jml, my test that runs a reactor is testing TwistedJobRunner, which itself starts and stops the reactor.
<jml> abentley, may I see the code?
<abentley> jml, of course.  lib/lp/services/job/runner.py
<jml> abentley, the idea of TwistedJobRunner is that it presents a synchronous interface?
<abentley> jml, yes.
<jml> abentley, ok.
<jml> abentley, could you test it by making 'reactor' a parameter to the constructor and providing a reactor that logs calls to key methods?
<jml> abentley, also, if I read the code aright, the reactor will spin forever if there's an exception raised in doConsumer
<jml> abentley, wait, I don't read it aright.
<abentley> jml, I don't think I care whether calls any particular methods on the reactor.
<jml> abentley, then what do you care about?
<abentley> jml, I care whether it runs the jobs and kills them if they take too long.
<jml> abentley, you can test that by calling runAll from a trial testcase if you make runAll return 'd'.
<jml> my DNS is playing up :(
<abentley> jml, you mean instead of running the reactor?
<jml> abentley, yes.
<jml> abentley, you'd also need to change terminated to not call reactor.stop itself.
<abentley> jml, I think I'd rather care less about the guts.
<abentley> And more about ensuring that it provides the same API as the synchronous runner.
<jml> abentley, in which case, subprocesses.
<abentley> jml, i.e. what allenap has done.
<jml> yeah.
<allenap> jml: Ah, the tearDown() idea doesn't work; Zope doesn't spawn a new process after each test case, just at the end.
<jml> allenap, testTearDown raising NotImplementedError then?
<jml> abentley, I think it would be useful to provide an interface that behaves more like regular Twisted code, but that's a separate issue.
<james_w> no-one has any suggestions?
<james_w> it could be that a bunch of fixes were just removed in the last week
<deryck> gah!  freakin' task age.
<deryck> james_w, is that what you're asking about?
<james_w> yeah
<james_w> I had a branch merged, and all was fine on edge again
<intellectronica> this is very very weird
<james_w> then some time in the last week (rollout?) the fix went away
<deryck> james_w, I suspect it was conflict resolution that caused this.  Someone likely being conservative, thinking it should be in there.
<deryck> unless we submitted to production-devel without going through devel?  intellectronica ?
<james_w> pretty sure it was devel
<intellectronica> no, this definitely was on devel
<deryck> oh, well.  I groan inwardly, but at this point we should just get it landed again. intellectronica, do you mind landing it and asking for a CP again?
<intellectronica> deryck: of course
<deryck> intellectronica, many thanks.  and james_w, thanks for the debugging help.
<james_w> https://code.edge.launchpad.net/~james-w/launchpad/kill-task_age/+merge/18418
<deryck> james_w, I'm assigning the bug to you again, just so you get the karma you're due. :-)
<james_w> heh
<deryck> I linked the old branch for reference, but I assume intellectronica will propose another version of the branch for re-submitting.
<deryck> intellectronica, in fact, rs=deryck on the same branch to devel.  then ask flacoste for the CP approval.
<jml> :(
<jml> internet is bad.
<allenap> jml: NotImplementedError in testTearDown() just breaks :( I will continue on the subunit+testtools path I think. That's working now; I just need to make it pretty.
<jml> ok.
<jml> allenap, fwiw, I'm working on a patch that'll give zope.testing native support for subunit.
<jml> allenap, it's 90% done, with only 90% to go.
<allenap> jml: Hehe :) That sounds cool :)
<jml> allenap, the second 90% includes actually reading lifeless's comments, responding to them and then navigating through whatever junk I need to in order to get it landed.
<james_w> deryck: yes, it appears to have been reverted in "Merging db-stable"
<intellectronica> james_w: i don't understand why. bzr should be able to recognize it.
 * kfogel is away: reboot
<intellectronica> unless some had a conflict involving these lines
<jtv> abentley: another thing I've been meaning to ask you... on my dev system, I can check out branches from http://code.launchpad.dev/ but not from http://bazaar.launchpad.dev/
<jtv> abentley: is that normal?  Because if so, we ought to fix composePublicURL for it
<deryck> james_w, thanks for checking into it further.  yeah, intellectronica, I recall adeuring and maybe someone else having to do manual conflict resolution once or twice the last week.
<abentley> jtv, no, it's not normal.
<mramirez> hi
<james_w> deryck: should we just delete task_age, I didn't realise it was created purely to be exported?
<deryck> james_w, yes, we should, especially since we have to re-land anyway.
<james_w> hmm
<james_w> doesn't seem to have been conflict resolution
<james_w> seems like bzr decided that the older code should win
<james_w> right
<james_w> the maze of branches meant that the merge from db-stable just has the export winning
<intellectronica> interesting
<intellectronica> james_w:  that's something we should write to the list about
<intellectronica> i'm happy to do that, but it looks like you might have more to say about the bzr side of things
<james_w> right
<james_w> except I've just confused myself more
<james_w> I was looking at the merge back, not at the devel->db-stable merge
<james_w> which is where the issue likely is
<allenap> jml, abentley: Here's my (working) solution to running individual tests in sub-processes (using subunit): http://paste.ubuntu.com/391187/
 * allenap will be back in 3 hours.
 * gary-lunch wants to know if https://bugs.edge.launchpad.net/launchpad-foundations/+bug/534399 is an annoyance or a critical problem.  He welcomes input, though responses may wait till after he returns from lunch.
<mup> Bug #534399: Windmill breaks branch-rewrite on dev systems <Launchpad Foundations:New> <https://launchpad.net/bugs/534399>
<gary-lunch> jtv: ^^^
<mrevell> night
<jtv> gary-lunch: doesn't affect production, but breaks manual testing.  So a major annoyance for a small group of developers.
<jml> g'night folks
<gary_poster> OK thanks jtv
<mwhudson> morning
<sinzui> Ursinha: why is you or your script changing the qa state of the work I QAed?
<Ursinha> sinzui: I put it to do a full run and as a default behavior it removes all qa tags if any and adds the qa-needstesting one
<Ursinha> sinzui: I've fixed all of them already
<Ursinha> sinzui: sorry for the noise, shouldn't happen again
<Ursinha> sinzui: just let the script add the qa-needstesting tag and it should be fine when you change to qa-ok
<sinzui> okay one pug has not passed QA. I see that it is still in needs testing so we will continue looking at a timeout oops that relates to it
<Ursinha> sinzui: you can add the qa-bad then and when the script detects a new commit with the fix it should change back to qa-needstesting again
<Ursinha> sinzui: I'm writing the docs and should mail the list soon explaining how it works
<sinzui> Well, the I will do that now
<Ursinha> sinzui: don't mark it as qa-needstesting, but as qa-bad, please :)
<Ursinha> sinzui: the script checks if the fix is already in staging or edge and then marks the bug as QAable, with the qa-needstesting tag
<sinzui> EdwinGrubbs: bug 512408
<mup> Bug #512408: Project page could suggest unlinked source packages <qa-bad> <Launchpad Registry:Fix Committed by bac> <https://launchpad.net/bugs/512408>
<Ursinha> thanks sinzui
<gary_poster> beuno: ping?
<beuno> gary_poster, pong
<gary_poster> hey beuno.  Do you know of any research that says what a minimally acceptable time to interact for a web page is, from a usability perspective?
<beuno> gary_poster, that's a big "it depends"
<beuno> is this load time?
<gary_poster> beuno: yeah, ish.  Time before you can use the page.
<beuno> gary_poster, off the top of my head, under 2 seconds
<beuno> I'll look up research once I finish bashing this javascript into complianhce
<gary_poster> beuno: awesome thank you that would be great
<wgrant> Can somebody please land lp:~wgrant/launchpad/fix-soyuz-build-and-queue-titles and lp:~wgrant/launchpad/buildqueue-to-buildmaster? They both passed EC2 three hours ago except for buildd-slavescanner.txt which was testfixed in devel overnight.
<beuno> gary_poster, http://www.useit.com/papers/responsetime.html
<beuno> is one of them
<gary_poster> beuno: ah, awesome, that sounds perfect. looking.
<beuno> gary_poster, the numbers change slightly depending on the audience
<beuno> and the task they're trying to achieve
<beuno> 2 seconds seemed to be acceptable for web apps that users wanted/needed to use, and where familiar with
<lifeless> beuno: ugh
<lifeless> 2 seconds is enough to switch, read an email, and return.
<beuno> lifeless, total time?  from click to render?
<beuno> very few websites give you 2 seconds
<wgrant> Ooh. It has improved. Only 6 seconds now to load a new bug with another bug page recently loaded, and 4 of those seconds were SQL time. That isn't actually too bad.
<wgrant> Can somebody please pqm-submit those above branches? I have another 5 or so dependent on them to get reviewed in the next day or two, so it would be nice to have these merged.
<mwhudson> jelmer: can you remember if there's a bug already about sharing revisions between related imports?
<wgrant> mwhudson: Do you guys notice when only one import machine needs a cert accepted?
<wgrant> Since I guess that won't often trigger suspension.
<mwhudson> wgrant: not automatically no :/
<mwhudson> galapagos got reinstalled a few months back and lost all it's certs
<mwhudson> -'
<wgrant> Ah. Yuck.
<wgrant> It doesn't know about the one for https://code.edge.launchpad.net/~vcs-imports/iptables/main.
<mwhudson> there are some crappy shell scripts somewhere to smear the certs around the various machines
<mwhudson> it would be really nice to fix this properly i guess, i just have no idea how
<mwhudson> (well, apart from turning all the branches into bzr-svn branches)
<wgrant> Get people to use non-crappy certs!
<intellectronica> wgrant: morning
<intellectronica> wgrant: i sent you some errors i got when running your branch through ec2, but i think they are actually unrelated errors that have since been fixed. if you merge the latest from devel i'll resubmit it.
<wgrant> intellectronica: Ah, you're still around. Great.
 * wgrant merges.
<intellectronica> wgrant: also, sorry, but eventually i didn't get to review your mega branch. it's been a busy day.
<wgrant> intellectronica: No problem, I'll find someone else to do it tonight.
<wgrant> intellectronica: OK, merged and pushed (r9083)
<wgrant> intellectronica: Can you also re-land lp:~wgrant/launchpad/fix-soyuz-build-and-queue-titles? henninge tried to do it last night, but failed the same way.
<intellectronica> wgrant: cool. resubmitted
<wgrant> intellectronica: Thanks.
<intellectronica> wgrant: sure
<jelmer> mwhudson: moin
<jelmer> mwhudson: afaik there is a bug about that (something like "code imports should use fallback repositories")
<mwhudson> ah yeah https://bugs.edge.launchpad.net/launchpad-code/+bug/485932
<mup> Bug #485932: should stack foreign branch imports <code-import> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/485932>
 * mwhudson ups importance
<wgrant> What, you don't want to have to spend a couple of months importing each Linux series?
<mwhudson> jelmer: it seems to me that fixing this bug would be much easier if we first fixed bug 375013
<mup> Bug #375013: Cannot commit directly to a stacked branch <commit> <stacking> <Bazaar:Triaged> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/375013>
<mwhudson> wgrant: yeah, a clearly silly suggestion!
<jelmer> mwhudson: wasn't that the bug that was just two hours worth of work?
<jelmer> wgrant: :-)
<mwhudson> jelmer: i don't know
<mwhudson> jelmer: i think i understand the bug well enough, but i certainly don't know how the apis that "allow a stacked repository to ask for more data" work
<lifeless> so ask
<lifeless> I hear bzr devs are accessible
<mwhudson> lifeless: well, i mainly mentioned it to see if jelmer agreed fixing it would help
<lifeless> mwhudson: do you want to know more about stacking/fallback repos?
<mwhudson> lifeless: not right now
<jelmer> mwhudson,lifeless: do we actually need to have that bug fixed before foreign branches can stack?
<lifeless> what bug
<lifeless> oh, 375013 - hell no
<mwhudson> jelmer: well, not necessarily, i can see various approaches that would work
<jelmer> so bug 375013 is not a prerequisite for bug 485932
<lifeless> thats /only/ relevant when doing WT.commit() and the WT.branch is itself stacked.
<mup> Bug #375013: Cannot commit directly to a stacked branch <commit> <stacking> <Bazaar:Triaged> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/375013>
<mup> Bug #485932: should stack foreign branch imports <code-import> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/485932>
<jelmer> lifeless: ok, good - thanks
<mwhudson> oh ok
<lifeless> e.g. in command line terms, you would have to be doing 'bzr checkout --lightweight <existing stacked branch url> foo; cd foo; bzr commit' for it to matter.
<lifeless> if however, you do that as part of your import workflow, then yes, you need to fix it.
<mwhudson> so we could create a branch, stack it on the existing kernel import, and then do bzr pull git://git.kernel.org/foo
<jelmer> mwhudson: right, so since you already do pull manually I guess the main change would be actually adding the stacking
<lifeless> jelmer: do you use commit builder ?
<lifeless> jelmer: or do you generate a stream ?
<jelmer> lifeless: no, insert_record_stream() and inventory delta's
<lifeless> then bug 375013 is totally irrelevant
<mup> Bug #375013: Cannot commit directly to a stacked branch <commit> <stacking> <Bazaar:Triaged> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/375013>
<lifeless> yes mup, we know. now STFU
<mwhudson> oh good
<mwhudson> jelmer: yeah, and it would make sense to preserve the stacking as the branch moves from import slave to central store to mirrored area
<mwhudson> but that's all pretty easy too i tihnk
<jelmer> mwhudson: yeah
<jelmer> mwhudson: how do the slaves work? do they have an NFS share that they pull into ? or do they fetch a copy of the branch, pull from the foreign repo and then push back
<mwhudson> jelmer: the latter
<mwhudson> there is some hystery here, i don't think we should take the existing setup as necessarily making perfect sense
<jelmer> ah, ok
<mwhudson> oh right, that was another reason i was thinking of stacking
<mwhudson> would it make sense for the slaves to do the equivalent of "bzr branch --stacked $central; bzr pull $foreign; bzr push $central" ?
<mwhudson> and it definitely makes sense to put the equivalent of a "--no-tree" in there
<jelmer> mwhudson: yeah, there's no need for trees
<jelmer> mwhudson: let me see what --stacked actually does, I'm not really familiar with anything other than the API :-)
<jelmer> mwhudson: no, you wouldn't be able to use --stacked
<jelmer> mwhudson: since you can't stack onto the source branch, only on another (bzr) branch
<jelmer> inter-format stacking does not (yet) work
<mwhudson> i think i was unclear
<mwhudson> this is for an update of an import
<mwhudson> $central is the location where the imported revisions live between import updates
<mwhudson> i guess the local network is fast, it probably doesn't matter too much
<mwhudson> otoh, branching outside a repo is cpu-intensive for 2a formats...
<jelmer> ahh, now I understand - yeah, that makes sense
<lifeless> mwhudson: shouldn't be that intensive
<wgrant> lifeless: It is, though :/
<lifeless> wgrant: mmm
<maxb> Alternatively, try 'bzr reconfigure --use-shared' in a launchpad branch when you mucked up and didn't branch into a repo in the first place.... you will cry :-(
<wgrant> lifeless: Also, do you know why non-empty LP pushes always transfer at least a megabyte, even for a tiny revision? Is seems to be "Inserting missing keys" for ages.
<lifeless> wgrant: new or existing branches ?
<wgrant> lifeless: Existing.
<lifeless> wgrant: we are a little lazy in that area at the moment - we don't try to diff the basis CHK pages, we just calculate the adjacent closure and send.
<lifeless> wgrant: please file a bug
<wgrant> lifeless: My local repo-less branch just started creating a working tree, 12 minutes after the fetch started.
<lifeless> wgrant: ok
<wgrant> That is slow, and it eats a core the whole time. That sounds intensive to me.
<wgrant> lifeless: Bug #534724
<mup> Bug #534724: Pushing an LP branch with an empty commit is slow and bandwidth-hungry <Bazaar:New> <https://launchpad.net/bugs/534724>
<wgrant> I would really like it if branch listings would show the review status rather than just the MP icon.
<wgrant> mwhudson/thumper: Can one of you please send https://code.edge.launchpad.net/~wgrant/launchpad/fix-build-breadcrumbs/+merge/20888 off to ec2test?
<mwhudson> wgrant: sure
<wgrant> mwhudson: Thanks.
<mwhudson> wgrant: it's gone headless now
<wgrant> mwhudson: Excellent.
#launchpad-dev 2010-03-09
 * mwhudson lunches
<thumper> how to I get emails into the stub mailer without transaction.commit() ?
<thumper> mwhudson: how do we force a remirror?
<thumper> mwhudson: lp:~kees/+junk/cpu-checker is fine in the hosted area, but screwed in mirrored
<mwhudson> thumper: launchpadlib?
<mwhudson> or lock and unlock
<mwhudson> (i have a dead simple plugin for the latter)
<thumper> mwhudson: we do pull --overwrite effectively don't we
<mwhudson> thumper: yep
<mwhudson> and blow it away if we can't open the branch
<thumper> mwhudson: the blowing it away bit seems to not work
<thumper> https://code.edge.launchpad.net/~kees/+junk/cpu-checker
<mwhudson> thumper: argl
<mwhudson> thumper: i see; you can open the branch in the mirrored side
<mwhudson> thumper: but not, say, run log on it
<thumper> mwhudson: yeah
<thumper> mwhudson: or get the ancestry
<thumper> mwhudson: any idea how that happened?
<thumper> I don't
<thumper> losa: ping
<mwhudson> thumper: nope
<wgrant> https://code.edge.launchpad.net/~launchpad-pqm/launchpad/devel
<wgrant> That says "Updating branch"
<wgrant> But I know that the revision was visible on LP 7 minutes ago.
<wgrant> (visible meaning "in the mirrored copy", I guess)
<thumper> wgrant: yes
<thumper> ...
<thumper> arse
<thumper> ENOLOSA
<wgrant> thumper: That seems an excessive amount of time to scan one new rev. That can't be normal, surely?
<thumper> wgrant: no it isn't normal
<thumper> wgrant: there is something wrong
<wgrant> Ah, good.
<thumper> wgrant: but the system is opaque
<thumper> jml: see, a job viewer would be really useful here
<mwhudson> launchpad/devel is up to date now
<wgrant> mwhudson: Yeah, it took a bit over 10 minutes to scan.
<thumper> wgrant: scanning is still a serial process for all branches in Launchpad
<wgrant> thumper: Ah, yes. Ew.
<thumper> wgrant: FWIW, if the email gets sent, the scanner has finished
<thumper> wgrant: it could have been replication lag you were seeing
<wgrant> thumper: The email wasn't sent until at least 10 minutes after it was pulled, so it wasn't repl lag.
<thumper> wgrant: well, there is a delay after the pull before the scan, and a delay after the scan to send the email
<thumper> wgrant: shouldn't be 10 minutes though
 * wgrant fixes community-contributions.py to work with db-devel.
<jml> thumper, I never argued that it was inherently useless.
<lifeless> jml: morning?
 * mwhudson winds up a not particularly productive day
<wgrant> mwhudson: Looks like that ec2 land evaporated.
<wgrant> Which is odd, since I've had 4 EC2 emails come through successfully in the past 24 hours. So maybe your mail config hates me.
<wgrant> kfogel: https://dev.launchpad.net/Contributions/Draft, produced by lp:~wgrant/launchpad/community-contributions-fixes, now with db-devel support and unbroken post-r9999 sorting.
<adeuring> good morning
<wgrant> henninge: Morning.
<henninge> Hi wgrant! ;)
<wgrant> henninge: So, intellectronica landed that first branch that you reviewed this morning after the testfix. Can you please try to land the second one that noodles UI-reviewed?
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/fix-build-breadcrumbs/+merge/20888, that is.
<noodles775> wgrant: oh, I saw that mwhudson had updated the commit message and assumed he was landing it?
<wgrant> noodles775: He tried, but APAC eC2 hates me and just avaporates.
<wgrant> European landings tend to work fine.
<noodles775> Ah.
<henninge> wgrant: np, I will do it.
<wgrant> But when the NZers try it, the instances almost always disappear :(
<wgrant> henninge: Thanks.
<mwhudson> wgrant: yep, instance gone :/
<wgrant> mwhudson: Yay.
<wgrant> noodles775: Thanks.
<noodles775> wgrant: np.
<mrevell> Guten morgen
<noodles775> Morgan mrevell.
<mrevell> :)
<deryck> Morning, all.
<allenap> adeuring: I'm having a look at your IPerson/IHasBugs issue.
<adeuring> allenap: thanks!
<deryck> gmb, intellectronica -- heat looks like it's finally updating as it should now.
<intellectronica> deryck: yes. loganberry was, as you suspected, behind. a re-roll fixed the problem.
<deryck> gmb, intellectronica -- can you two create a combined file of sql to update heat and max_bug_heat and get that run, to get us updated across the board, now that the jobs system will keep it up to date.
<intellectronica> i'm not sure we can do that with just sql, can we?
<gmb> intellectronica: We can update bug heat with SQL. max_bug_heat I don't know about.
<intellectronica> gmb: what do you think? can we do that with sql, or maybe a script that creates heat calculation jobs?
<intellectronica> gmb: we can calculate it in sql
<deryck> intellectronica, I think so.  that same queries in the model code for max_bug_heat can be fed into an update.
<gmb> intellectronica: In that case we should be fine. I'll dig out my bug heat queries.
<gmb> SQL for updating bug heat is here: http://pastebin.ubuntu.com/391693/
<intellectronica> gmb: the main problem i see with doing this with just sql is that it will result in queries that take too long to run
<gmb> intellectronica: Yeah, that's a risk.
<gmb> OTOH, if we create a CalculateBugHeatJob for each of 500,000+ bugs those jobs will take a long time, too.
<intellectronica> gmb: i wonder, since you've started working on a script to create jobs, if it wouldn't be easier to use that
<deryck> intellectronica, like gmb says jobs for every bug is too long, too.
<intellectronica> deryck: right, but it doesn't happen in a single transaction
<gmb> intellectronica: Well, it's not a very finessed script: http://pastebin.ubuntu.com/391695/
<deryck> intellectronica, hmmm, good point.
<intellectronica> :)
<intellectronica> simple i like
<deryck> hmmmm, I'm not liking the mass create bug jobs idea either.
<intellectronica> deryck: what are we actually trying to solve?
<gmb> deryck: We could use a looptuner. But that's liable to take a day or so to complete because it's quite conservative.
<intellectronica> do we want to calculate heat for all bugs, or just get max_heat calculated?
<deryck> intellectronica, I want both, heat for all bugs and max_bug_heat.  sort of a reset, since the update heat system hasn't been working completely until now.
<intellectronica> deryck, gmb: right. i'd say creating jobs (and letting them process over the day or so it might take) is probably the best approach, since it's safe and exercises the code we've already written (rather than emulate it with unrelated sql).
<gmb> intellectronica: In that case we should try the creation script on staging first to make sure that it doesn't hold the transaction open too long or anythign similarly annoying.
<deryck> intellectronica, gmb -- yeah, this is my only concern is that create 500,000+ jobs in one fell swoop has issues we don't yet know about.
<deryck> if we test on staging first, I'm fine to do it that way.
<gmb> deryck: Actually, an issue that occurs to me is that the processing script might try to do them all at once (it's generic code, so I've never subjected it to any kind of massive load test).
<intellectronica> let's try that, and if that doesn't work, batch it by target, or just in batches of bugs
<gmb> Once we've created the jobs on staging we should run cronscripts/calculate-bug-heat.py and see what happens.
<intellectronica> gmb: hmmm ... so maybe it's better to start with batching. create a batch of jobs, process them, repeat...
<gmb> intellectronica: Possibly. I think that the script will cope with that. The proof of the pudding is in the staging, however. :)
<intellectronica> heh
<allenap> adeuring: I can't figure out why it's not working, but I have a word-around: http://paste.ubuntu.com/391719/
<allenap> adeuring: I also noticed that the LaunchpadObjectFactory was doing some naughty things - unwrapping security proxies and never re-adding them - so I fixed that. You probably shouldn't apply that part of the patch; I'll file a bug about it.
<adeuring> allenap: thanks!
<allenap> adeuring: You're welcome :) Still a mystery though...
<adeuring> yeah...
<allenap> adeuring: salgado's solution is more elegant (bool). But that doesn't explain why person.all_bugtasks.limit is forbidden and product.all_bugtasks.limit isn't.
<adeuring> allenap: right, that's what I was asking me too ;)
<allenap> adeuring: Fwiw, I filed bug 535021 about the naked object problems in LPO.
<mup> Bug #535021: Some LaunchpadObjectFactory methods remove security proxies then inadvertently return naked objects <Launchpad itself:New> <https://launchpad.net/bugs/535021>
<adeuring> allenap: great, thanks
<adeuring> allenap: I'm wondering if that would also explain that the tests for product.has_bugtasks did not fail
<adeuring> argh... that's about _person_ objects....
<allenap> adeuring: Yeah, seems likely.
<allenap> adeuring: If it is the case, could you comment on that bug to say that it has caused confusion already?
<adeuring> sure
<salgado> adeuring, allenap, if it works for Product it's because for that class the return value of all_bugtasks is not security proxied
<adeuring> salgado: maybe ... but why?
<salgado> I see that Person.searchTasks() returns getUtility(IBugTaskSet).search(search_params, *args), so it is security proxied
<salgado> now looking for Product.searchTasks()
<salgado> HasBugsBase returns BugTaskSet().search(search_params)
<salgado> which is not security proxied
<salgado> that's why
<adeuring> ahhh, thanks!
<allenap> salgado: Of course! Somehow I missed that Person implements its own searchTasks().
<adeuring> so that's what should be fxided
<allenap> adeuring: Even so, probably still use the bool(result_set) or is_empty() fix too. It's supposedly more efficient than count(), though I don't know by how much.
<adeuring> allenap: right
<thekorn> mrevell, allenap, hi, I started writing a blog post about my experiences as a launchpad contributor, but I'm not done with it yet, I think I need another 2 days
<mrevell> thekorn, Hey, thanks for working on that. Whenever you're ready is great.
<allenap> thekorn: That's brilliant :)
<kfogel> wgrant: OMG, that rocks.  I will review today.  Want to make a merge proposal?
<intellectronica> jtv: i don't think this is a result of any of my branches, and i'm even tempted to just try and force a rebuild
<jelmer> allenap: You just filed a bug against launchpad itself rather than a particular component, was that intentional?
<intellectronica> but will look more closely
<jtv> intellectronica: noodles775 seems to concur... too late at night for me to look at the details but I agree.
<intellectronica> noodles775: so, force a rebuild?
<noodles775> intellectronica: yeah, I'm just running `bin/test -vvt testLibrarianWorking` at the moment on tip of devel, it's working so far without error.
<noodles775> yep, it just finished without error. So +1 for force a rebuild :)
<jtv> intellectronica: you want to press the button?
<intellectronica> jtv: pressed
<jtv> thanks!
<allenap> jelmer: Yes, kind of. A foundations responsibility perhaps, but one which anyone could fix.
<jml> allenap, that is that sad tune to which the song of foundations is sung.
<allenap> jml: :) Is there an easily identifiable subset of foundations bugs that could be tackled by us mortals? Would that simply be bugs tagged trivial?
<jml> allenap, good question! /me punts to gary_poster
<jelmer> allenap: ah, thanks
<gary_poster> jml, allenap, not really.  I suppose I could start doing that, though, if I knew what the distinction might be.  FWIW, bugs tagged as UI that don't involve CSS bugs are good candidates, but I think you are asking for the opposite way round (how do I tag, rather than how do I find)
<allenap> gary_poster: I don't really know what I'm thinking of, but it was brought about by bug 535021. It's something I'd be happy to fix, but it doesn't belong on the malone project where deryck or I might see it in the normal course of things, and schedule it. TBH, we're all busy enough that finding new sources of work is probably not going to fly far. Most of the time, work is feature driven rather than looking for bugs to fix.
<mup> Bug #535021: Some LaunchpadObjectFactory methods remove security proxies then inadvertently return naked objects <Launchpad Foundations:New> <https://launchpad.net/bugs/535021>
<gary_poster> allenap: yeah.  Non-foundations people take "foundations" bugs all the time.  Registry does this in particular.  I think it is fabulous.
<allenap> gary_poster: Cool. I'll try and join that merry band :) Want any malone bugs? ;)
<gary_poster> allenap: heh, uh, no. :-)
<mramirez>  might indicate where the configuration is positioned to Logear launchpad with ldap
<mramirez>  might indicate where the configuration is positioned to Logear launchpad with ldap
<mramirez> * [Chan
<didrocks> jml: I'm not as popular than you, my branch got rejected because of no description :)
<didrocks> jml: I'm adding some now
<adiroiban> danilos: do you know when the +translate UI rework will be done?
<danilos> adiroiban, no, though I'd like to raise priority because I consider it tech-debt and very important for anything we want to do
<adiroiban> in the Romanian team, we are trying to use Launchpad for our review process
<adiroiban> and bug 339507 is a real blocker
<mup> Bug #339507: translation page should show old translations <ui> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/339507>
<danilos> adiroiban, yeah, I understand; as I said, I won't mind you fixing that bug right now in the same manner, and we'll only have to be careful not to break it when we do the pofile:+translate rework
<adiroiban> well, there should be a next test for this bug and it should catch this kind of regressions
<danilos> adiroiban, well, with a thing like entire page/view rewrite, you don't really get the benefit of testing (with pagetests), since we'd be using entirely different views
<adiroiban> well, if there will be such a regression
<adiroiban> I will notice it :_)
<allenap> jelmer: I understand what you were asking now. I forgot that there were both launchpad and launchpad-foundations projects. Thank you for reassigning that bug.
<danilos> adiroiban, fwiw, I am having issues with ec2 land (if you are interested why your branch hasn't landed yet)
<adiroiban> danilos: ok. are they general issues, or directly caused by my branch?
<henninge> danilos, adiroiban: It took me a few attempts, too.
<danilos> adiroiban, general issues it seems
<henninge> danilos: have you done the sudo apt-get install python-zope.inferface thing?
<danilos> henninge, does it appear to you as if it doesn't go headless for me
<danilos> henninge, I think I did, I successfully ran a bunch of tests and ec2 test
<henninge> I had prlbmes landing branches, wgrant's lately.
<danilos> henninge, fwiw, there are some lp-dev-deps updates that I haven't installed, let me try that (though, I've had problems for more than a day now)
<leonardr> intellectronica, i'd like your opinion on bug 534066
<mup> Bug #534066: changing bugtask attributes via api gives "412 precondition failed" <api> <Launchpad Bugs:Triaged by james-w> <https://launchpad.net/bugs/534066>
<intellectronica> leonardr: sure, looking
<intellectronica> leonardr: that bug itself has a fix, and will be deployed soon, once it hits production-devel. but i agree this is a general problem we need a solution for
<leonardr> intellectronica: what's the fix, and can you tell me some other places where the problem exists?
<intellectronica> leonardr: so yes, task_age will be removed (it was removed already a month ago and only made a comeback because of a freak accident). but sooner or later we'll have a field which we can't remove and cause a similar problem
<leonardr> are there any such fields now? if not, i'm going to focus on other sources of 412 errors
<leonardr> also, my argument applies to any field that's important enough to go into the representation but not important enough to change the etag when it changes
<leonardr> you'll have no way of knowing whether your value for that field is accurate
<intellectronica> leonardr: none i can think of off the top of my head, but even task_age itself, while cheap to remove now, was introduced because a consumer wanted it.
<intellectronica> leonardr: well, it depends whether the field is writeable. i don't see why it's such a big problem to have read-only fields that are in the representation but aren't used for etag calculation.
<danilos> henninge, fwiw, updating all source deps made 'ec2 land' go headless for me
<leonardr> if the values of those fields never change, it's fine
<leonardr> like date_created
<danilos> henninge, s/source deps/lp deps/
<leonardr> if they change and the etag doesn't change, the client never finds out that the value changed
<henninge> danilos: yes, I basically had to do the same.
<intellectronica> leonardr: anyway, that's just something to keep in mind. i agree with you that it's best now to figure out all the other 412 errors and worry about adding this facility later.
<danilos> anyway, done for the day
<intellectronica> leonardr: right, but in the case of task_age this is something we could live with
<intellectronica> anyway, i didn't even think this through enough to have an opinion
<leonardr> intellectronica: for me "we could live with this data being arbitrarily out-of-date" -> "we don't need this data"
<leonardr> i have the first glimpses of a couple solutions, i'll write them up in the bug
<intellectronica> leonardr: that's not exactly true, but judging by this case that's an approximation of the truth. consumers can get this value by calculating locally.
<leonardr> intellectronica: without putting you on the spot, can you give me a hypothetical counterexample?
<intellectronica> leonardr: i can think of many read-only attributes where a change in their value is completely irrelevant to a patch you're submitting to the object, and where the only reason they don't usually cause a problem is that they don't tend to change as frequently as every second. consider the number of affected users for a bug, for example.
<intellectronica> or bug heat
<intellectronica> the likelihood of the values changing between a fetch and a patch operation is evidently not as high, but if it does, you won't be able to patch even though there's no real reason for you not to be able to, it's just a side effect of the way caching works
<leonardr> intellectronica: the problem is, if we changed it so those fields didn't affect the etag, clients would never know when the bug heat or affected users changed
<leonardr> or rather, they would only find out when something _else_ changed
<leonardr> it would be really nice if we could publish two etags, a GET etag and a PATCH etag
<intellectronica> leonardr: right, so maybe the problem isn't really how we do caching, but how we decide whether a patch is valid. we just don't have enough information because we don't know which fields changed.
<intellectronica> leonardr: you mean a read-only and a read/write tag? that sounds like a pretty good solution.
<intellectronica> is there a sensible way to publish this in as part of the protocol?
<leonardr> i've never heard of anyone doing that, and it's not part of the http standard
<intellectronica> but maybe as an extra header?
<intellectronica> this must be a problem many REST implementations must have
<leonardr> i may write a blog post about this and see if anyone has ideas
<leonardr> intellectronica: another way to do it is to check last-modified for PATCH and etag for GET
<leonardr> but we'd have to make sure that changes to those read-only fields don't affect last-modified
<intellectronica> leonardr: for last-modified you need to be synchronised with the server time somehow, but i guess that's not a serious problem
<leonardr> intellectronica: not really, last-modified is just a string you get from the server and send back, like etag
<intellectronica> heh, yeah, makes sense
<james_w> leonardr: did you see the mailing list discussion about this?
<leonardr> james_w: i know there was one a while back, i haven't gotten into it yet
<james_w> a while back being last week?
<leonardr> yeah
<james_w> right
<james_w> I would suggest you start there, it has some discussion on the questions you ask in the bug
<leonardr> james_w: ok, i'm going there now
<jml> thumper, ping
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/fix-build-breadcrumbs/+merge/20888 and https://code.edge.launchpad.net/~wgrant/launchpad/buildstatus-to-buildmaster/+merge/20892 failed a couple of tests last night, but I've fixed them -- can somebody ec2land them, please?
<wgrant> kfogel: Do you want to review the c-c.py branch?
<kfogel> wgrant: yes.  Can you just make it into a merge proposal and request a review from me?  (I can't do it right now, but I will later.)
<wgrant> kfogel: Done, thanks. You should have mail in a few seconds.
<kfogel> wgrant: I hate the way launchpad doesn't update the diff immediately, sigh :-).
<intellectronica> kfogel: is there a better way to not update the diff immediately? :)
<mwhudson> bizarre, pidgin seems to have got all crashy suddenly?
<NCommander> Is anyone actively working on implementing package sets (for archive reorg) in launchpad?
#launchpad-dev 2010-03-10
<wgrant> NCommander: They've been implemented for well over a year.
<wgrant> And permissions use them now.
<NCommander> wgrant: ah, I thought there were still bits that required implementation in Launchpad (I saw the IPackageset and friends, but not sure what the status was)
<wgrant> NCommander: There are a few bits and pieces that don't respect those permissions yet (bug nomination appproval and build retries come to mind), but most of it works and is in active use.
<NCommander> wgrant: they are?
 * NCommander didn't know this
<NCommander> wgrant: so its possible to generate Packages files with Sets: *blah* in them?
<wgrant> NCommander: That would not be too difficult.
<wgrant> Except for the difficulty inherent in the crazy apt-ftparchive integration.
<NCommander> wgrant: well, APT has to be patched regardless to understand sets
<NCommander> wgrant: is there a way to query sets currently in LP?
<wgrant> NCommander: Yes, through the API. Check out edit_acl.py in ubuntu-archive-tools.
<NCommander> wgrant: ah, I knew about those, just didn't realize those were IPackagesets in the backend
<NCommander> wgrant: I thought LP was changed to generating the Packages files directly off the  backend and obseleting apt-ftparchive
<wgrant> NCommander: Ha ha ha.
<wgrant> NCommander: It does for PPAs and partner.
<NCommander> oops
<wgrant> But primary and copy archives still use apt-ftparchive.
<NCommander> wgrant: fun
<lifeless> wgrant: performance?
<NCommander> lifeless: probably an issue with Contents-* files and other fun stuff
 * NCommander went down this road with dak
<wgrant> lifeless: Probably not, but it's unclear.
<wgrant> The main problem is with overrides.
<wgrant> NMAF doesn't really support them.
<wgrant> Contents files are another point, but they're generated somewhat separately.
<wgrant> (it would be nice if LP could generate them natively for PPAs as well, but a table listing all files would be HUGE.
<NCommander> Internet doesn't like me at the moment
<NCommander> <NCommander> wgrant: at the risk of being terrorifed, how do Tasks then end up in the packages file?
<elmo> overrides
<wgrant> Magical magical override files.
<wgrant> Not really generated by LP.
<wgrant> But by something that lives in the publisher cronscript.
<wgrant> elmo: I'm guessing no Soyuz people are around to fix that custom upload bug?
<elmo> wgrant: I guess not, sorry.  if you have a trivial patch, I'm willing to cowboy it on
<wgrant> elmo: See the branch on the bug. With tests and everything.
<elmo> oh, well, I didn't actually subscribe myself \o/
<wgrant> (cjwatson's fix was slightly wrong)
<elmo> aaaaa, LP code ui, you make me cry
<wgrant> What's it doing to you now?
<elmo> just going from that bug to your actual diff is requiring me to think
<elmo> and I'm not doing well at that right now
<NCommander> wgrant: I just looked at that crontab and ....
 * NCommander cries
<wgrant> NCommander: Yes, cron.publish-ftpmaster is scary.
<wgrant> elmo: It's easy if there's an MP, which there isn't yet.
<NCommander> wgrant: the changes look pretty straightforward, I just need to amend the extras sweep in cron.germinate (which probably will become to cron.extras) to dump Packagesets in-mass, and then let apt-ftparchive handle it
<wgrant> http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10460 and http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10461
<wgrant> Although the latter isn't important.
<wgrant> NCommander: Right.
<elmo> wgrant: yeah, I foujnd it eventually
<elmo> wgrant: now I'm applying my lack of thinking to the actual diff
<wgrant> elmo: An issue brought up in https://code.edge.launchpad.net/~julian-edwards/launchpad/symlink-vuln-bug-529710/+merge/20543 is what caused the bug.
<NCommander> wgrant: do we support a Supported: field ATM? (I see some references to it, but nothing in publisher)
<wgrant> NCommander: See maintenance-check.py, and whatever calls it.
<elmo> wgrant: I'm assuming this is just a context thing (i.e. I don't have enough in the diff), but I don't see how this code actually stops $tmpdir/../../../ other than having to guess $tmpdir?
<elmo> oh, duh
<elmo> realpath
<elmo> nevermind
<wgrant> Right.
<wgrant> realpath resolves symlinks and normpaths, eliminating that hole.
<wgrant> Once you've cowboyed it, process-accepted should successfully process those uploads on its next run.
<elmo> I'm happy enough to cowboy this, given it can only break customuploads
<wgrant> Right.
<lifeless> wgrant: all files is only about 2GB of data
<lifeless> wgrant: (for all files + all packages + per-binary file knowledge + relationships between packages)
<elmo> lifeless: err, not for PPAs
<lifeless> wgrant: 'conflictchecker' has this, and can query single package conflicts quickly
<wgrant> And not for multiple releases of each source package.
<lifeless> wgrant: conflict checker does precisely that
<wgrant> Hmm.
<lifeless> elmo: I don't have the data to estimate for PPAs; if there is interest I could page in conflict checkers schema and figure out what we need to know to estimate it for PPAs.
<lifeless> by per-binary I mean per-arch
<mwhudson> when you're talking about HUGE tables in launchpad, bear in mind we have TranslationMessage and BranchRevision :-)
<wgrant> mwhudson: That's true.
<mwhudson> though the last is totally silly
<wgrant> But BranchRevision is stupidly huge.
<wgrant> Yes.
<elmo> please tell me BranchRevision is not what I think it is
<lifeless> for added fun, conflictchecker was running on sqlite last I saw.
<mwhudson> elmo: lalalala fairies
<lifeless> elmo: its what you think it is.
<wgrant> elmo: Yes, every branch of Launchpad actually has 70000 BranchRevision rows linked to it.
<elmo> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
<mwhudson> i wonder if we should look at neo4j for this sort of thing
<wgrant> Hm, maybe.
<mwhudson> however right now i'm going to have lunch and water the garden
<lifeless> elmo: what machine is conflict-checker running on at the moment?
<elmo> lifeless: macaroodle or something
<elmo> lifeless: macaroni or macquarie
<elmo> probably macaroni
<wgrant> macaroni indeed.
<elmo> yeah, it's the source of constant 1 load on macaroni
<lifeless> hmm, 3GB now; probably need to ditch some old dev edition packages.
<elmo> wgrant: your patch is on production now
<wgrant> elmo: I guess we'll see if it works in 16 minutes.
<wgrant> Thanks.
<lifeless> wgrant: select count(*) from filepath;
<lifeless> 8319961
<wgrant> lifeless: Is that for all SPRs, or just current ones?
<lifeless> wgrant: thats all filenames seen in binary packages ever.
<lifeless> wgrant: (but it only scans the official archive)
<wgrant> Hm, not bad.
<lifeless> sqlite> select count(*) from binpackagefiles;
<lifeless> 28807743
<lifeless>  .schema binpackagefiles
<lifeless> CREATE TABLE binpackagefiles (package_id INTEGER, file_id INTEGER);
<lifeless> CREATE INDEX _bpf_fileid on binpackagefiles (file_id);
<lifeless> CREATE UNIQUE INDEX _unique_packagefile ON binpackagefiles (package_id, file_id);
<wgrant> I guess we could always purge the lists when we expire PPA sources and binaries, if it becomes a problem.
<wgrant> elmo: Did it work?
<elmo> wgrant: oh, I wasn't looking
<elmo> wgrant: didn't the previous upload get rejected?
<wgrant> elmo: Unlikely. It should have just sat in ACCEPTED.
<lifeless> wgrant: its really quite modest. You need to be careful to use indices, but thats about it.
<elmo> wgrant: ah
<wgrant> Actually, I should be able to check the queue pages...
<lifeless> if we had the blacklisting done, for handling hand-verified maintainer-scripts, we would probably keep indefinitely ahead of the archive.
<wgrant> elmo: They're all DONE now, so it probably worked.
<elmo> wgrant: sweet, thanks
<wgrant> Ah, yes, we have binary publications too. All is good.
<mwhudson> neo4j looks handy, apart from the j bit
<maxb> Ah. python2.5 has gone from lucid
<maxb> +copy-packages time
<wgrant> :(
<maxb> um. What does it mean when a single source package version has two source publications, one superseding the other?!
<maxb> oh, override change?
<wgrant> Normally, yes.
<maxb> oh this could be fun. So I need to pick the right one such that the package ends up in the PPA's main :-)
<maxb> ngh
<maxb> And I can't retry builds in the LP PPA
<maxb> Can someone in ~launchpad hit retry on these, please: https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553604 https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553605
<jelmer> maxb: done
<maxb> thanks
<maxb> Hrm.
<maxb> launchpad-developer-dependencies Depends spidermonkey-bin
<maxb> which has just been removed from lucid
<mwhudson> i wonder what that's for
 * maxb files it under the 'Someone thought it was a good idea' heading :-)
<maxb> Added anonymously in 0.35, October 2008
<mwhudson> sounds like something that could be deleted
<wgrant> maxb: PPA publications are overridden to main by a deeper hack, so it doesn't matter which you copy.
<thumper> :(
<thumper> spending lots of time reading code
<thumper> and not so much time writing
 * mwhudson is writing some very boring tests
<lifeless> \\ ?
<jtv> stub: I don't suppose fragment caching works within an arbitrary loop?
<stub> It does
<noodles775> wgrant: hi, shall I land your buildstatus-to-buildmaster branch, or have you already organised it since fixing the test?
<noodles775> I can't see it in the changelog, so I'll do so.
<jtv> stub: how does it know which iteration is which in the cache?
<stub> A counter is stored as a request annotation, and this is used to form part of the key.
<adeuring> good morning
<noodles775> Hi adeuring
<jtv> hi adeuring, hi noodles775
<adeuring> hi noodles775, jtv!
<noodles775> G'day jtv :)
<mrevell> Hallo
<wgrant> noodles775: I haven't organised it. Thanks.
<wgrant> noodles775: Can you please also land my fix-build-breadcrumbs? It failed a couple of tests overnight which I've fixed.
<noodles775> wgrant: sure.
<bigjools> stub: are you around much longer?  I'd like to talk to you about how to fix https://bugs.edge.launchpad.net/soyuz/+bug/517101
<mup> Bug #517101: update-pkgcache.py needs to be database friendly <Soyuz:Triaged> <https://launchpad.net/bugs/517101>
<stub> I'm free
<stub> Back later
<maxb> wgrant: for +copy-packages too? http://ppa.launchpad.net/maxb/ppa/ubuntu/pool/multiverse/s/sun-java5/ says otherwise :-)
<wgrant> maxb: That was ages ago.
 * wgrant checks the log.
<maxb> oh, ok. I admit to not having a more recent example
<persia> bigjools: So, would today be a good day to talk about Copyright stuff for native-source-sync?
<bigjools> persia: yes maybe in an hour?
<persia> bigjools: That sounds great.  Thanks.
<bigjools> great
<wgrant> persia: Changelog stuff?
<persia> wgrant: Yes, thanks for the correction.
<jtv> henninge: --force-depends works
<wgrant> Messy :(
<henninge> jtv: sure ;-)
<wgrant> maxb: It was fixed two days after those were published.
<bigjools> wgrant: can you propose a merge for your custom upload fix
<wgrant> bigjools: Sure.
<bigjools> thanks for fixing it, I'll get it landed
<bigjools> I was out yesterday
<wgrant> Thanks.
<wgrant> Unfortunately I last tested the original fix with a real upload before the os.sep change was added :(
<bigjools> :(
<bigjools> so I am setting up LP on a new laptop on lucid, and rocketfuel-setup runs aptitide which eventually says launchpad-developer-dependencies depends on spidermonkey-bin ... ?!
<wgrant> bigjools: Yeah, see the ML.
<wgrant> It was removed recently.
<bigjools> ok I'm well behind on email :(
<wgrant> bigjools: Should I specify you as the reviewer?
<bigjools> wgrant: yes
<persia> Is there any specific reason rocketfuel-setup runs aptitude?  I've heard the aptitude resolver isn't widely tested (and seen any number of bugs for stuff that didn't work with aptitude that did work with other resolvers)
<bigjools> I suspect whoever wrote that part of the script just happens to also prefer it
<bigjools> we should change it
<thumper> damn
<thumper> call in 6 hours
<thumper> best go to be
<thumper> d
<deryck> Morning
<wgrant> Do the Canonical webapp engineering teams not talk to each other?
<wgrant> I have found the same obvious hole in three Canonical webapps in the past two weeks.
<bigjools> wgrant: not as much as we should, we're setting up an architectural team to address that
<jtv> wgrant: not getting my Bob to pick up the jobs... are you sure I don't need to run anything?
<wgrant> jtv: As long as buildd-manager is working, it should be working. What do buildd-manager's logs say? What does an XML-RPC status() call on the builder say?
<jtv> wgrant: hmm... in /var/log/launchpad-buildd/default.log I see it shutting down after a SIGTERM, and nothing after that
<wgrant> jtv: Heh, you might want to start it.
<jtv> yes, I might
<jtv> wgrant: time to remove the -e I added to that helper script  :)
<wgrant> A good idea.
<jtv> still no joy though :(
<wgrant> What does buildd-manager say?
<jtv> wgrant: nothing in the log, nothing in the shell
<jtv> I tell a lie
<wgrant> jtv: Hmph. The builder is responding with IDLE?
<jtv> startup failed because there's "a twistd" already running
<wgrant> Hmm.
<jtv> yup, buildd manager was still running.
<jtv> I'll try killing that instance and restarting
<persia> bigjools: Let me know when you have time :)
<stub> How often will bug heat get refreshed do we think?
<bigjools> persia: right now
<persia> bigjools: Great.  So my current understanding is that there are 3-4 things involving changelogs that need to be sorted before we can turn on native-source-sync, and that you know which these are.  Is this true?  If so, could you expand?
<bigjools> persia: my memory is very hazy on this, I'll see if I wrote something down
<bigjools> ok so there's a few bugs
<bigjools> https://bugs.edge.launchpad.net/soyuz/+bug/55795
<jtv> wgrant: I have a "starting scanning cycle" in the buildd-manager log, but nothing else.  :)
<mup> Bug #55795: +changelog includes misleading information related to package versions and authors <soyuz-core> <soyuz-upload> <ui> <Soyuz:Triaged> <https://launchpad.net/bugs/55795>
<wgrant> jtv: The builder is still marked OK, and not manual?
<bigjools> https://bugs.edge.launchpad.net/soyuz/+bug/139162
<mup> Bug #139162: Store the pristine debian/changelog for each SourcePackageRelease <soyuz-core> <soyuz-upload> <Soyuz:Triaged> <https://launchpad.net/bugs/139162>
<wgrant> jtv: Can you see buildd-manager's polls in /var/log/launchpad-buildd/default?
<bigjools> https://bugs.edge.launchpad.net/soyuz/+bug/247456
<mup> Bug #247456: changelog listing are incomplete and/or undiscoverable <soyuz-core> <ui> <Soyuz:New> <https://launchpad.net/bugs/247456>
<bigjools> persia: basically it's horrible right now
<jtv> wgrant: the builder is OK and not manual
<wgrant> Also we need to be able to create something akin to a changes file.
<jtv> wgrant: it's not logging to there, it turns out; it's logging to /var/tmp/development-buildd-manager.log
<jtv> All I see there is "Starting scanning cycle."  Lots of times.
<wgrant> jtv: That's buildd-manager. I want launchpad-buildd.
<jtv> ahh
<persia> bigjools: There's a few other bugs surrounding that which I've seen, but yeah.  So do we actually need the changelog for anything, or could we just store it as requested in 139162 and present it as requested in 55795?
<jtv> wgrant: I've been looking at /var/log, but I guess that's on the slave.  :)
<bigjools> persia: 139162 is the first task
<jtv> wgrant: 2010-03-10 11:31:18+0000 [HTTPChannel,910,127.0.0.1] 127.0.0.1 - - [10/Mar/2010:11:31:18 +0000] "POST /rpc HTTP/1.0" 200 222 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
<jtv> wgrant: repeats
<wgrant> jtv: OK, so it is polling.
<wgrant> So maybe the candidate selection is not working properly.
<bigjools> persia: once we've done that the NSS becomes easier
<jtv> wgrant: I guess.  I was hoping to be able to stay out of that.  :)
<jtv> But oh well. :)
<wgrant> jtv: Check your postgres log for the query that buildd-manager is using.
<persia> bigjools: Right.  So we have code that does 139162 that we use for the changelog download in apitutude and update-manager, storing stuff on changelogs.ubuntu.com.
<wgrant> Run that, and work out why it isn't returning your job.
<wgrant> If it is, there is something more sinister going on.
<bigjools> persia: we also need to repackage existing records
<persia> bigjools: How do you mean "repackage"?
<jtv> wgrant: there's no extra-verbose logging in the script that I can enable?
<wgrant> bigjools: Can you answer jtv? You know the logging much better now.
<bigjools> persia: the existing changelogs need to be re-processed
<jtv> bigjools: I'm trying to figure out why the buildd isn't picking up my jobs.
<persia> bigjools: Is there any useful information there, or do we just need to extract changelogs from the existing packages?
<bigjools> jtv: got a b-m log?
<bigjools> persia: AIUI that's pretty much it
<jtv> bigjools: just says "Starting scanning cycle." all the time.
<bigjools> jtv: then your job is not eligible for dispatch (either a bug or genuinely)
<jtv> bigjools: so as wgrant just pointed out, sounds like candidate selection.
<persia> bigjools: And then 55795 becomes easy because we can just expose that blob?
<jtv> bigjools: right, so my question is: is there debug logging in the script that'll make this easier to pinpoint?
<bigjools> jtv: see the query it's issuing in the PG log
<bigjools> and go from there
<jtv> bigjools: that's what he said.  OK, I'll try that, thanks.
<bigjools> jtv: there's no debug logging that will help :(
<bigjools> it's the candidate selection query
<bigjools> persia: exactly
<wgrant> Which I know works generically now, but you may be missing a piece for your translations job.
<bigjools> persia: the way attribution is done is all messed up as well
<bigjools> yep
<persia> bigjools: That goes away if we use mvo's code for 139162
<bigjools> persia: it would need to work in LP, but that's the general idea yeah
<persia> bigjools: And then 247346 gets resolved trivially as well?
<bigjools> with any luck :)
<persia> Now, how ought this be implemented in LP?
<bigjools> jtv: paste the query from the pg log into a psql prompt and experiment
<jtv> bigjools: thanks...  for now I'm just checking it manually against the data I have.
<bigjools> persia: depends on what that code looks like, I've not analysed any of this
<jtv> bigjools: ahhh, got it: "(buildqueue.virtualized IS NULL OR 'f' IS TRUE)"
<jtv> I reviewed that code, so it can't be wrong.  :-)
<bigjools> the classic :)
<persia> bigjools: The code basically just knows the format of a .deb, and extracts the changelog.
<persia> bigjools: That logic can be embedded however is convenient.
<persia> We can do the same for a .dsc, but we need an unpack environment.
<bigjools> well it already does it for dsc, just not very well
<bigjools> the upload processor unpacks the source
<wgrant> We already have code to grab debian/copyright.
<jtv> bigjools: it does actually make sense, it's just that it reflects... IIRC whether the buildd manager is looking for arch-independent build candidates.
<wgrant> Which I believe was touched last week...
<bigjools> indeed
<persia> So really, that code should just spit debian/copyright into the DB?
<persia> And then we can expose it?
<bigjools> it does already
<bigjools> we just don't expose it
<persia> So what's missing in 139162 again?
<bigjools> https://dev.launchpad.net/Soyuz/Specs/RawSourceChangelog
<bigjools> ^^ that
<persia> I don't see the missing bit, if we're already stuffing full raw changelogs into the DB.
<bigjools> we're not, we're extracting bits of it
<persia> Or is it just the exposure and client changes?
<persia> Aha!
<bigjools> and then trying to re-attribute them
<wgrant> We store debian/copyright, not debian/changelog yet.
<bigjools> it all blows quite spectacularly
<bigjools> right, we store changelog_entry which is a partial changelog
 * persia slaps forehead
<persia> Right.  So we just need to pull debian/changelog *when* we pull debian/copyright.
<persia> Now, do we actually need any of the code that attempts to construct a changelog, or can we scrap that entirely?
<bigjools> I'm not sure yet, I suspect it could be scrapped but bits of it might get re-used in the presentation
<bigjools> stub: ok can we talk about pkg-cache now?
<stub> yup
<bigjools> stub: where does the problem lie, in the fact that it's reading too quick?  or udpating?
<persia> bigjools: Thanks!
<bigjools> or both?
<bigjools> persia: no worries!  I hope it's filled a gap
<stub> updating too quickly without regard to the system load
<bigjools> persia: and sorry it took me so long to get back to you
<bigjools> stub: so I could put a loop tuner at the top level where it reads package names in
<bigjools> which would also throttle writes
<stub> Ideally it should block when the system is busy. That logic is centrally encoded in DBLoopTuner at the moment.
<bigjools> yeah
<stub> I haven't looked at the existing structure of the script
<bigjools> stub: it's a wrapper around various bits
<bigjools> stub: it calls IDistribution.updateCompleteSourcePackageCache
<bigjools> I need to move that out of that model code and into soyuz somewhere
<bigjools> then I think I can just slap a loop tuner on there and profit
<stub> So updateCompleteSourcePackageCache sounds like a chunk that should be its own DBLoopTuner loop
<bigjools> exactly
<stub> You could install it into garbo-daily.py if you like - this fits in with its role.
 * stub looks at updateCompleteSourcePackageCache
<stub> Yer - doesn't look like much conversion to turn that loop in updateCompleteSourcePackageCache into a DBLoopTuner loop.
<jtv> al-maisan: wgrant, bigjools: looks like my job should have "virtualized" set to true in order to be selected by a non-virtualized builder
<wgrant> jtv: Didn't I convince you to set bob to virtualised?
<al-maisan> jtv: not setting it (i.e. a NULL) will be eqivalent to virtualized=True
<wgrant> You should set virtualized to either True or None in order for it to build on a virt builder.
<wgrant> False will let it build on non-virt.
<jtv> wgrant: I may easily have gotten confused between the setting on the builder and on the jobs.
<jtv> wgrant: ah, I see it: I skipped a step this time that I had documented
<wgrant> Ahh.
<jtv> sorry for the noise
<jtv> now I guess I wait for it to poll...
<wgrant> It should poll every five seconds.
<wgrant> bigjools: Thanks for landing that.
<wgrant> noodles775: I see that buildstatus-to-buildmaster should be landing RSN, but I have no EC2 email about the other one yet.
<bigjools> wgrant: welcome
<bigjools> is someone already fixing the spidermonkey-bin problem BTW?
<wgrant> bigjools: Do you have time to talk about PPA download stats?
<bigjools> wgrant: sure
<wgrant> so, bug #139855
<mup> Bug #139855: Display stats about PPA usage <feature> <Soyuz:Triaged> <https://launchpad.net/bugs/139855>
<wgrant> The only consensus seems to be that we want to store binary download counts like we do LFA downloads.
<wgrant> Indices probably want to be counted somehow too, but what to do with that is much less obvious.
<wgrant> I have a working implementation of binary counts.
<bigjools> I tihnk that doing the binary counts is a very good first step
<bigjools> do you want me to take a look at it?
<noodles775> wgrant: checking the ec2 instance...
<noodles775> wgrant: it seems to be finishing (it's not yet terminated, but I can no longer access it), so I'll wait a bit. If I've not seen anything by my EOD, i'll resend it.
<wgrant> noodles775: OK, odd...
<wgrant> At least the other one should be in PQM.
<noodles775> wgrant: it is (playing now).
<wgrant> bigjools: I think the backend branches (https://code.edge.launchpad.net/~wgrant/launchpad/parse-ppa-apache-logs is the final one) should be reasonably unobjectionable.
<wgrant> But https://code.edge.launchpad.net/~wgrant/launchpad/export-basic-binary-download-stats and its successor https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats might well be horribly slow.
<bigjools> wgrant: ok, file an MP and request me as a reviewer and I'll look at that one
<bigjools> can you paste a diff?  I'm lazy :)
<wgrant> bigjools: The combination is not quite oversized, but it has two logical prereqs which will make it smaller. Shall I propose them separately?
<bigjools> wgrant: I'm a little confused, you have three branches and the first one is not controversial, right?  but the other 2 might be?  so I was asking for diffs for those
<wgrant> bigjools: The first uncontroversial one is really three branches. You just want the two that might be? That's easy.
 * wgrant diffs.
<bigjools> wgrant: well if it's really big we should split it up and get more reviewers involved
<wgrant> bigjools: It's not really big. The first branch I mentioned comes it at around 800 lines -- it's just probably a bit easier if it's split into its three logical components.
<bigjools> ok
<wgrant> noodles775: Ah, that other run emailed a couple of minutes ago. Great.
<noodles775> Excellent.
<wgrant> bigjools: A few criss-cross merges and lots of conflicts later: http://pastebin.ubuntu.com/392515/ and http://pastebin.ubuntu.com/392521/
<bigjools> heh
<wgrant> The first just exports a single count per BPPH, the second gives detailed data.
<bigjools> wgrant: if it's expensive, don't make it a property
<bigjools> download_count that is
<wgrant> bigjools: Yeah, I guess not.
<leonardr> who should i speak to about the design of html pages for launchpad? is it still mpt?
<bigjools> it will get serialised on every fetch
<bigjools> leonardr: good point - it was beuno but he's moved to U1
<bigjools> he might still want to help :)
<wgrant> I'm not sure how expensive it will be. It could be a SUM across several hundred rows of a single-table join.
<bigjools> I want to avoid making properties that hide complex queries anyway
<bigjools> it's not really a property :)
<wgrant> This is true.
<asabil> hi all
<bigjools> wgrant: I think sum is pretty quick anyway
<bigjools> wgrant: but I wonder if we shouldn't just store that total to avoid doing it?
 * bigjools thinks about splitting up archive.txt
<wgrant> bigjools: Is archive.txt the 3000 line one?
<bigjools> yarp
<bigjools> it's useless as documentation
<bigjools> and lots of it can be a unit test
<wgrant> bigjools: We could, but there's no convenient table representing (archive, bpr)
<wgrant> Right.
<bigjools> presumably you've got a schema change hiding somewhere?
<wgrant> It's in the previous branch.
 * wgrant diffs.
<bigjools> ok
<bigjools> what is the .count counting then?
<bigjools> Sum(BinaryPackageReleaseDownloadCount.count)
<beuno> bigjools, I always want to help
<bigjools> if we're counting total downloads per archive we can add a column to Archive
<bigjools> beuno: yo da man
<wgrant> bigjools: Like LFDC, it maps (target (in this case (archive, bpr)), date, country) -> count
<jml> I'm sorry, I wasn't paying attention. Did I hear someone say they were adding download counts for PPAs to Launchpad?
<bigjools> heh
<wgrant> bigjools: http://pastebin.ubuntu.com/392534/ is the work before the two diffs I gave above.
<bigjools> wgrant: so we could use another column on IArchive for total download count and maintain it in the log scraper?
<jml> leonardr, there's no one person with responsibility for that any more. You consult the launchpad-dev list, basically.
<leonardr> jml: ok
<wgrant> jml: Has that been shown to work at all yet?
<jml> wgrant, asking launchpad-dev for advice?
<wgrant> bigjools: But counts per archive, per (archive, bpr) and per (archive, bpr, day) are useful.
<wgrant> We need a generic cache mechanism :(
<leonardr> well, i'm going to write something, and if launchpad folks don't like it they can lump^H^H^H^Hfix it
<wgrant> jml: For UI advice.
<jml> wgrant, how would you tell if it worked?
<bigjools> wgrant: yes I'm saying to do it instead of that new property
<bigjools> so you can avoid the Sum
<wgrant> jml: Well, given that LP now has no UI designers, and nobody from UX has replied to the RFC-UI emails on the list up to now, it seems like it probably won't.
<bigjools> wgrant: the other thing is that exposing database IDs on URL traversals makes me a bit sad
<wgrant> bigjools: But that's on BPPH, so it needs a download count cache on (archive, bpr), which is a tuple that the DB doesn't know about.
<wgrant> bigjools: Yes :(
<jml> wgrant, although I would highly value contributions from Canonical UX, I think that the broader launchpad developer community almost always has useful things to say about UI design.
<bigjools> can you traverse with the package name/version instead?
<wgrant> bigjools: Possibly. I suppose they should generally be unique.
<bigjools> wgrant: IArchive.getPackageDownloadTotal() - we can replace that with a new column on Archive
<wgrant> And any cases in which they are not are bugs.
<bigjools> unless I am misreading something?
<wgrant> bigjools: It takes a BPR as an argument.
<bigjools> yes I am misreading :)
<bigjools> so that's the total of all versions of that package in this archive
<wgrant> All versions? A BPR is one version.
<wgrant> All publications, yes.
<bigjools> all published versions is what I meant
<wgrant> Right.
<bigjools> looks great
<bigjools> just needs to be a method instead of a property on publishing
<wgrant> bigjools: Great, thanks. It is getting late, so I will fix that tomorrow. Shall I then request reviews from you for the three, or should I use OCR now you're obviously not too unhappy with the implementation?
<bigjools> wgrant: the traversal should not use the ID, other than that the approach looks great, thanks!
<wgrant> Oh, right, yeah.
<kfogel> adeuring: re bug #532022, I'm just starting to figure out where to add the criterion to suppress bugs whose status is "Fix Released".  Is this something that would go into those special DB queries you made for +patches?
<mup> Bug #532022: bugs with patches shows fix released bugs as well <Launchpad Bugs:In Progress by kfogel> <https://launchpad.net/bugs/532022>
<adeuring> kfogel: we call bugtarget.searchTasks() with a set of status values. I think it is enough to change this set of values, let me look...
<kfogel> adeuring: ah, heh -- that'd be easy.  But let me check: it *might* be the case that there's some big efficiency to be had by doing the filtering at a lower level, not sure.
<adeuring> kfogel: right, we call searchTasks(status=UNRESOLVED_PLUS_FIXRELEASED_BUGTASK_STATUSES, ...), so this value should be changed.
<kfogel> adeuring: okay, good.  In fact, I think I may have created that combination status explicitly for this, so if this is the only use of it, it can go away too.  I'll check.
<adeuring> kfogel: no, I don't think so. If the current query works, another one with a smaller set of status values should be fine
<adeuring> kfogel: yes, it can probably go again, if it's used only here..
<kfogel> adeuring: right, because the raw query that goes into the DB starts here (i.e., it's not like we're getting a bigger set of results back and then filtering down).
<adeuring> kfogel: right
<kfogel> adeuring: just talking out loud partly to make sure my understandings are right.  Thanks for reading :-).
<adeuring> kfogel: there are one more changes:
<adeuring> the number shown in the link to the +patches view
<kfogel> adeuring: oh, that's not connected to the searchTasks query?
<kfogel> adeuring: (I'd already forgotten how we implemented that number.)
<adeuring> and the test that ensures that this number euals the number of patches shown in the +patches view
<adeuring> s/euals/equals/
<kfogel> adeuring: thanks for the reminder -- will take a look.
<kfogel> adeuring: but anyway I can base my branch for this off of devel, not db-devel.
<adeuring> kfogel: sure, using devel is fine
<adeuring> kfogel:  lib/lp/bugs/browser/bugtask.py: BugsStatsMixin.bugs_with_patches_count
<adeuring> kfogel: and the test is: lib/lp/bugs/browser/tests/test_bugtarget_patches_view.py
<kfogel> adeuring: thank you.  It's starting to come back to me now...
<kfogel> adeuring: I have a tremendous ability to wipe my mind clean after finishing something.  I've often thought I could give lessons to trauma victims.
<adeuring> (see also bzr diff -r10453..10454
<adeuring> kfogel: sounds interesting ;) I could help there perhaps too... I had to look through "bzr log" to find the right revision, and then looked at the diff ;)
<kfogel> adeuring: yeah, but I actually wrote this code, like, a week ago IIRC :-]
<kfogel> wgrant: hey, I will review your branch today
<kfogel> wgrant: "today" for me means in the next eight hours (I'm in New York)
<mars> gary_poster, ping, trying to work around a buildout dependency.
<gary_poster> mars, what's up
<mars> gary_poster, I'm seeing 'mercurial' listed in the branch root's setup.py, and I do not see mercurial in versions.cfg.  Is that to be expected?
<mars> gary_poster, and bin/buildout is spitting errors at me because of mercurial for some odd reason.
<gary_poster> mars, no, doesn't sound right.
<mars> weird for a 3 line change...
<gary_poster> mars, investigating.  this is devel?
<mars> gary_poster, yes, fresh branch yesterday
<gary_poster> k
<mars> gary_poster, line 46, setup.py
<mars> looks like it is needed by code hosting.  Maybe a system package is supposed to install it?
<gary_poster> mars, typically still should have a version
<mars> ah
<gary_poster> mars, jumping around a bit, but trying to pull devel now.  In regards to your hypothesis, have you checked to see if launchpad-developer-dependencies are up to date for you?
<mars> gary_poster, chekcing
<mars> yes, they are up to date
<gary_poster> ok
<gary_poster> I'm now building Twisted 10...
<mars> gary_poster, here is what I am getting on the CLI when I run bin/buildout:
<mars> http://paste.ubuntu.com/392596/
<gary_poster> mars, ok, I just built fine...investigating setup.py and versions.cfg next, and will look at your pastebin
<gary_poster> mars, mercurial is in my devel's versions.cfg
<mars> line?
<gary_poster> mars, the error you are getting is exactly because it is not in your versions.cfg
<mars> hehe
<gary_poster> we have a setting to cause that
<gary_poster> mars, line 39
<gary_poster> mechanize = 0.1.11
<gary_poster> mercurial = 1.3.1
<gary_poster> mocker = 0.10.1
<mars> gary_poster, ok, thanks, I am missing that line.  More detective work then
<gary_poster> mars, ok cool
<deryck> Ursinha, ping
<Ursinha> deryck: hi
<deryck> Ursinha, hi.  Could the script that does qa tagging, which links the milestone if needed, also link the branch if it's not linked yet?
<deryck> Ursinha, not to give you more work even ;)
<Ursinha> deryck: hmmmmmmm I'll see what I can do :) I have to finish polishing the burndown chart, and have other task after that, but I put that in the queue, sure :)
<Ursinha> *I'll put
<deryck> Ursinha, np.  Thanks for considering it! :-)
<Ursinha> deryck: no problem :)
<mars> gary_poster, found the problem, thanks for the help.
<gary_poster> mars, great
<adiroiban> hi, any idea why I get this error on launchpad.dev when I try to login ? http://paste.ubuntu.com/392604/ ?
<mars> adiroiban, something to do with the new OpenID changes perhaps.  Maybe check the config variable listed on line 15 of your post?
<salgado> adiroiban, need to add testopenid.dev to /etc/hosts
<adiroiban> salgado: true. thanks!
<mars> salgado, if he re-runs rocketfuel-setup, will that happen automatically?
<salgado> mars, yes
<salgado> actually, maybe nt
<mars> salgado, so now we all have to do that to use launchpad.dev?
<salgado> not
<salgado> mars, yes
<salgado> adiroiban, mars, yes, running rocketfuel-setup should add it there
<mars> salgado, have we sent a mail to launchpad-dev about having to run rf-setup to add that host?
<mars> "Please re-run rocketfuel-setup" or something
<salgado> no, I didn't.  forgot about that
<salgado> mars, do you think it's still worth to do so?  I suspect most people have already found that out or they run rocketfuel-setup frequently and didn't notice it at all
<mars> salgado, depends on how long ago the change landed.  It could be that most people don't log out and in again.  And sending it can't hurt :)
<salgado> indeed.  although it's landed 2 weeks ago, it won't hurt to send an email
<mars> the more people that know about the problem and the answer, the fewer times you will have to answer it yourself :)
<didrocks> Launchpad screen refresh being very slow is a known issue? For instance, switching tabs in firefox, 2 to 4 seconds to refresh the page, without hitting F5; opening option bug in edit -> options takes 3 seconds to have it refreshed too
<didrocks> and this, only if I am on a LP page current tab
<didrocks> no issue with other sites in other tabs
<didrocks> apparently seb128 has the same issue too
<didrocks> I tried in edge/production server
<didrocks> and disabling greasemonkey too
<didrocks> (just to be sure, I'm not speaking about loading a page, just having something on a already loaded page)
<deryck> didrocks, I'm not familiar with this as a known issue... maybe mars knows something?
<didrocks> I would think the page content is not supported by either xulrunner or if there is an ajax things "on activate"
<mars> ?
<mars> didrocks, do you have a specific page you were trying this on?
<didrocks> mars: it's an all LP page for both seb128 and I (particularly bugs page). I can make a short screencast if you want to see the difference
<mars> didrocks, ok, if that is easiest :)
<mars> didrocks, do you mean, all LP pages on edge have this problem?
<didrocks> mars: no, LP edge and production server (non edge)
<didrocks> with firefox on lucid
<rockstar> leonardr, ping
<leonardr> rockstar, hi
<rockstar> leonardr, abentley and I were talking about bug 520446 this morning.
<mup> Bug #520446: OOPS creating duplicate merge proposal using API <api> <oops> <trivial> <Launchpad Bazaar Integration:In Progress by rockstar> <https://launchpad.net/bugs/520446>
<leonardr> ok
<rockstar> leonardr, basically, the fix is to catch the (more meaningful) exception and raise a ValueError.
<rockstar> leonardr, have you seen this kind of problem before?
<didrocks> mars: http://people.canonical.com/~didrocks/slowLP-1.ogv that should explain the issue :)
<mars> didrocks, looking
<leonardr> rockstar: so the problem is that a specific exception results in an oops when it should result in a 400 status code?
<leonardr> yes, that happens all the time
<leonardr> you can annotate the exception with @webservice_error(400) and it will be handled correctly if the error happens during field validation
<leonardr> if it happens in named operation invocation i don't think that will work
<mars> didrocks, so Chrome is fine
<didrocks> mars: right
<didrocks> (it's chromium, but the same base ^^)
<rockstar> leonardr, what do you mean annotate the exception?
<leonardr> rockstar, let me try to find an example
<mars> didrocks, could you try that tab-switching test with the gnome system monitor open?  To the Resources page, with the graphs on it
<didrocks> mars: sure
<leonardr> i have a feeling that this is used all over launchpad but maybe no one is using it
<mars> didrocks, should show a CPU spike, if there is one
<rockstar> leonardr, it seems like it should be used all over launchpad at least.  :)
<mars> didrocks, no need to record the CPU, just let me know if it spikes while the browser is hung
<didrocks> mars: right, one CPU is going to 100%
<didrocks> for the 3-4 seconds
<mars> hmm
<rockstar> leonardr, I found some examples.
<rockstar> lib/lp/hardwaredb/interfaces/hwdb.py
<mars> didrocks, thinking, who's the culprit?  The FF JS engine?  Or the FF rendering engine?
<leonardr> rockstar: yeah, try that and see if it works
<mars> didrocks, I don't think it is any JS on our page, but can't rule that out entirely either.
<didrocks> mars: I don't know enough Firefox to have a strong opinion on that. I can ask chriscoulson who is our new maintainer
<didrocks> mars: can I tell him to see that with you (tomorrow probably)
<didrocks> ?
<mars> didrocks, sure.  You say you have confirmed the behaviour with someone else using FF+Lucid
<didrocks> that's just pretty obvious in lucid and only affecting LP pages. So, there is something that FF doesn't like :)
<mars> ?
<didrocks> mars: right, I was thinking about a local profile issue first, but seb128 begins to have the same behavior
<mars> didrocks, ok, I can think of a test: I just need to find a relatively JS-free page first
<didrocks> mars: sure, going to get my dinner, you can ping me, bbiab
<mars> didrocks, ok
<rockstar> leonardr, oh man, MUCH better solution.  Thank you!
<leonardr> awesome
<didrocks> mars: so, chrisccoulson is our new firefox rockstar :)
<chrisccoulson> heh
<didrocks> mars: do you have any idea of tests I can do?
<didrocks> chrisccoulson: did you see the video?
<chrisccoulson> i did
<didrocks> chrisccoulson: you can see how my life is hard nowdays :-)
<mars> hi chrisccoulson
<chrisccoulson> hi
<mars> didrocks, looking around
<mars> chrisccoulson, did didrocks show you his screencast with the strange behaviour?
<chrisccoulson> mars - he did
<chrisccoulson> i've not noticed behaviour like that before though
<mars> chrisccoulson, I was thinking it may be the Firefox JS engine, the rendering/paint engine, or maybe something in our JavaScript itself that doesn't play nice with the new FF JS engine.
<mars> chrisccoulson, but I don't have the FF profiling tools at hand to narrow it down.  I'm trying to find a non-js page to test on.
<mars> would rule out the paint operation
<didrocks> mars: I mentionned also to chrisccoulson that I don't have accelerated GPU driver. That should emphasize, but as we can see, chromium plays a better role there
<mars> didrocks, try it with this page: https://code.launchpad.net/oops
<mars> that is the 404 page: quite minimal, and no javascript
<mars> bit of a dumb test, but better than nothing to start I guess
<didrocks> mars: no issue with this 404 page
<mars> didrocks, and https://code.launchpad.net/  ?
<didrocks> its fast to say "you screw something" :)
<didrocks> mars: I have a delay
<didrocks> not as long as on a bug page
<mars> interesting
<didrocks> but still, 1.5s I would say
<mars> didrocks, are you on a 64bit system?
<didrocks> mars: no
<didrocks> mars: google reader and gmail are quite fast
<didrocks> (for the js part)
<mars> chrisccoulson, ^ fun FF bug for you - our automated integration testing tool using FF goes nuts and eats 100% CPU if our uncompressed JS exceeds 512Kb, but only on  64 bit systems.  Not a problem here though :)
<chrisccoulson> sorry, i just tried using venkman to profile the javascript
<chrisccoulson> i need to figure out how to use it really though ;)
<mars> didrocks, ok, so it is *not* the invisible iframe we use for the help system.  That is evidence against a DOM issue.
<mars> hmm
<mars> Google PageSpeed could tease out the issue.  It hooks right in the guts of firefox
<mars> chrisccoulson, I take it you see the same behaviour on your system?
<chrisccoulson> mars - i don't
<chrisccoulson> which is a bit of a pain ;)
<mars> ...
<didrocks> chrisccoulson: can you try to switch to a 2D driver?
<didrocks> maybe that'll be more obvious
<mars> didrocks, do you see this behaviour when you run in -safemode (I think that's the switch?)
<didrocks> on seb128's laptop is something like 2s
<chrisccoulson> didrocks - i could do, but that might have to wait until later
<didrocks> trying
<mwhudson> morning
<mars> morning mwhudson
<mars> didrocks, ah: "firefox -safe-mode"  http://support.mozilla.com/en-US/kb/Safe+Mode
<mars> forgot the dash
<didrocks> mars: a little bit better, only 1.5 - 2s now. What's strange is that the only extension I have is the ubuntu one (ubufox)
<mars> chrisccoulson, didrocks, since I don't have access to a proper FF profiler (though I have read about them), it maybe be possible to use the PageSpeed extension to narrow the problem down to JS or the browser: http://code.google.com/speed/page-speed/docs/using.html#activities
<chrisccoulson> yeah, i just installed that
<chrisccoulson> it looks pretty neat
<mars> didrocks, do you see the delay/hang when you first open a page?
<mars> didrocks, or just when visiting a tab?
<didrocks> mars: as it's loading at the same time, it's hard to tell
<didrocks> mars: but in the video you can see that when firefox has something to show (like the option dialog box), there is the same hang
<mars> didrocks, the same hang?
<kfogel> I have never seen this happen before: I did 'bzr branch devel 532022-patches-view-omit-fixreleased' to create a working branch, and cd'd into the new branch and did 'utilities/link-external-sourcecode ../devel'.  But for whatever reason, my new branch does not have a bin/ directory at the top level!
 * mars re-reads the last word
<didrocks> mars: the same delay before showing the dialog box
<mars> didrocks, ok.
<mars> didrocks, chrisccoulson, if you see the same delay when showing a dialog, then I suspect two things
<salgado> kfogel, need to 'make build' -- that's automatically done by rocketfuel-branch
<kfogel> salgado: aaaah.  thanks
<mars> didrocks, chrisccoulson: 1) The browser paint and render code is messing up in a bad way
<mars> didrocks, chrisccoulson: 2) There is a focus/blur event that our JavaScript is processing and choking on
<didrocks> chrisccoulson, mars: showing a dialog only over a LP page, I must precise
<mars> I don't know enough about the Firefox 3.6 architecture to say if the JS execution thread has been decoupled from the application threads: to say if it is possible for JS to block page paints
<mars> didrocks, it is only on tab focus/blur, not window focus/blur?
<didrocks> mars: you mean, other windows?
<didrocks> mars: if I, for instance, move a gnome-terminal over it, I get the same issue
<mars> didrocks, yes.  If you have the tab open, click on another window, click back on Firefox
<mars> chrisccoulson, ^ !
<didrocks> seems that FF has some issue to just redraw the LP page. It's just strange that it only happens with LP and not on other "big page"
<mars> didrocks, so it chugs if you move another window of the Firefox painted area
<didrocks> mars: exactly
<mars> ok, so that means it is probably not the JavaScript events
<mars> It is the Firefox rendering engine trying to paint the screen
<mars> didrocks, there is a way to test that hypothesis
<mars> didrocks, save the page to a local temp folder, then open the local page and try the tests again
<didrocks> mars: ok, doing this
<mars> didrocks, if it is the page structure, or something in the page, then it will probably show up when the page is local
<didrocks> mars: yes! same issue with the local page :)
<mars> didrocks, to be extra sure, we can break the javascript after you have saved it to disk.  No JS + chugging means it is the page structure or content
<didrocks> oh, good idea
<mars> didrocks, you can shut of JS in Firefox itself, can't you?
<mars> (wish I'd thought of that a few minutes ago)
<mars> didrocks, after shutting off JS, try the test.  If it still chugs, then shut off images, and please try it again.
<mars> should narrow the bug down to the content, or the page structure itself.
<didrocks> mwhudson: yeah, still the bug with JS unactivated
<didrocks> now images :)
<didrocks> mars: not showing images fix it
<mars> ok, so it might be painting images from Launchpad that is messed up
<thumper> morning
<didrocks> morning thumper
<didrocks> mars: right
<mars> morning thumper
<didrocks> (I still have the LP logo even when selecting "not showing images"), but the others are gone
<mars> So, so so so....
<mars> we could try cherry-picking just the images from the page, put them on an empty HTML page, see if the problem persists
<mars> shouldn't take long
<chrisccoulson> sorry, i disappeared for a bit there
<mars> didrocks, what was the URL you have been testing with?
<didrocks> mars: I cached locally https://bugs.edge.launchpad.net/launchpad-registry/+bug/357235
<mup> Bug #357235: A user's ssh keys are not currently available throug the APIs <api> <feature> <quickly> <Launchpad Registry:Triaged by didrocks> <https://launchpad.net/bugs/357235>
<mars> chrisccoulson, np.  We narrowed it down to the Launchpad page images.  The bug is present irregardless of JS being on or off.  But deaactivating images made the bug go away.
<chrisccoulson> ah, ok
<didrocks> I would never thought it was image related :)
<chrisccoulson> there's not many images is there?
<mars> well, lets see
<chrisccoulson> oh, i notice all the images when i disable them ;)
<mars> Firebug Net tab has that nice "images only" filter
<didrocks> chrisccoulson: that's weird, there is one that obviously FF doesn't like :)
<mars> chrisccoulson, after finding that 64bit 512Kb JS limit bug.... And some of the DOM bugs I've seen...
<thumper> sinzui, bac: ping
<thumper> re bug 246594
<mup> Bug #246594: Registering a code import from a series is clunky. <confusing-ui> <Launchpad Bazaar Integration:In Progress by bac> <https://launchpad.net/bugs/246594>
<mars> oh.  hmmm
<mars> thumper, they are sprinting this week, may not be around
<mars> IIRC
<thumper> mars: ah
<thumper> I'll comment on the bug
<leonardr> sinzui, can you help me with xslt-fu?
<mars> leonardr, see previous comment to thumper :)
<leonardr> ah, thanks mars
<mars> didrocks, chrisccoulson, 26 images on that bugs page according to Firebug.
<mars> didrocks, chrisccoulson, give me a bit, I can make a temporary page with just those images.  *might* have the paint bug.
<didrocks> mars: sure, thanks. I saw too that there is not that much image (just reinstalled firebug, that was a whileâ¦ ;))
<mars> didrocks, you need to use Control-F5 in case the browser cache was being used, to get the complete image count.
<mars> ok, this test is shaky, but better than nothing.  Off to code
<didrocks> mars: right, that's what I did when I test :)
<didrocks> thanks mars :)
<mars> didrocks, is seb128 online?
<didrocks> mars: I guess he is away for a couple of horus
<didrocks> hours*
<mars> yes, and he is PST, so he'll be around :)
<didrocks> mars: hum, he is in France too, so, even he's going to bed late, I don't think so :)
<didrocks> mars: in any case, you can still ping me and I will do the test tomorrow morning
<mars> didrocks, ?  His whois info says otherwise: seb128, right?
<didrocks> (my irc is always opened)
<didrocks> mars: he maybe connects to an PST located IRC server, but I can ensure you he is French and live in France :)
<mars> must like using US/west servers
<didrocks> https://launchpad.net/~seb128
<mars> ah
<didrocks> I just guess I made the issue more noticeable in not having 3D driver currently installed and so making FF software rendered painting suffers :)
<mars> yeah, that was quite the hiccup though.  2-3 seconds.  And why oh why did it have to be Launchpad, of all pages? :D
<mars> hmmmm]
<mars> didrocks, I wonder if it is the transparent PNGs doing it?
<mars> we'll see
<mars> back to code
<didrocks> mars: hum, maybe  :)
<didrocks> ok
<mars> didrocks, please try the test using the following page: http://people.canonical.com/~mars/lp-images-test.html
<didrocks> mars: no latency
<mars> phooey.
<didrocks> mars: I guess you expected some, right?
<mars> didrocks, I'm not surprised.  It was the only test I could think of at the moment :)
<mars> so we have a launchpad page that, when shown with images and HTML, causes the browser paint to mess up
<mars> shut off the images, everything is fine
<mars> shut off the content, everything is fine
<mars> so images+content produced the bug
<mars> brutal.  That is hard to debug.
<didrocks> right, I tried again and even with the online version, that's definitely reproduceable
<didrocks> definitively
<mars> didrocks, ok, we could try to rule out alpha using the offline version.  Could you please upload your offline version to p.c.c?  I'll hack on it.
<thumper> abentley: AssertionError: Wrong min time to next available builder (47099 > 300)
<thumper> abentley: that's what I get
<mwhudson> 47099 is quite a lot bigger than 300
<abentley> That's not the same one I get.
<didrocks> mars: sure, one sec
<abentley> I get 120 != 30
 * thumper makes schema
<didrocks> mars: FYI, if I keep the images on and I unactivate the css, no bug
<didrocks> mars: so, definitively images + layout
<mars> ok
<mars> that is good to know
<didrocks> mars: http://people.canonical.com/~didrocks/357235.html
<mars> didrocks, perfect, thanks.  So I can try cutting out the transparent pngs.  That is something at least.
<mars> Unless there is some other way to look at the problem.
<didrocks> that can be a good idea at least to ensure it's not layout + transparent pngs
<mars> maybe not the transparent PNGs.  They aren't visible on the default page.
<mars> oh, wait, at least one is, in the footer.
<didrocks> oh, the little logos have white bg? I hadn't a look TBH
<mwhudson> thumper, abentley, rockstar: http://pastebin.ubuntu.com/392769/
<thumper> http://pastebin.ubuntu.com/392770/
<mwhudson> https://bugs.edge.launchpad.net/soyuz/+bug/525329
<mup> Bug #525329: spurious failure in test_min_time_to_next_builder <qa-needstesting> <spurious-test-failure> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/525329>
<mwhudson> thumper, abentley, rockstar: do you all want to approve this mp then? https://code.edge.launchpad.net/~mwhudson/launchpad/fix-test_min_time_to_next_builder-non-utc-bug-525329/+merge/21082
<mars> didrocks, this file has heavily mangled CSS, does it show the bug?  http://people.canonical.com/~mars/lp-images-test/1/357235.html
<mars> didrocks, this is a copy of what you sent me.  It absolutely should have the bug: http://people.canonical.com/~mars/lp-images-test/0/357235.html
<thumper> mwhudson: we could ask mars to approve it too and go for the patch with the most reviews :)
<mars> thumper, I must have missed where the 'utc' variable came from
<didrocks> mars: yeah! the first one has no bug. The second one has it
<mars> didrocks, ok.  Doesn't help much, except that almost every image is broken in /1/
<mars> didrocks, so again, it is an image+layout problem.  Narrowing it down is going to be a really big pain though.  It will take time.
<thumper> mars: the utc comes from pytz at the top of the file: from pytz import utc
<didrocks> mars: right, it's really not obvious, but at least, the issue is narrowed much more than an hour ago :)
<mars> mwhudson, nice change.  Easier to read than before.
 * mwhudson pqm-submits
<thumper> mwhudson: don't forget to add mars to the r= list :)
<mars> :D
<mwhudson> thumper: i used ec2 land --dry-run to make the list, never fear
<thumper> :)
<rockstar> I assume there's no reviewers meeting today...
<mwhudson> rockstar: bac cancelled it
<rockstar> mwhudson, yeah, I figured.  I actually had something bring up too.
<mwhudson> rockstar: gosh
<mars> didrocks, I think I am going to call it a night.  Do you think you could file a bug about this problem?  Be sure to attach the screencast.
<didrocks> mars: sure, just a bug against LP itself so?
<mars> didrocks, for now, until we narrow it down.
<didrocks> mars: I'll link to your file too
<mars> didrocks, htanks
<didrocks> mars: doing that tomorrow morning as well, it's getting late there :)
<didrocks> thanks a lot for your debugging too :)
<mwhudson> wgrant: congrats on getting your name into a debian-security mail :-)
<wgrant> Ah, so that's out now.
<wgrant> I assume that production is patched?
<NCommander> does anyone know who would be the person to talk to about extending packagesets to be able to work with binary packages in addition to source packages?
<wgrant> NCommander: Distro people.
<NCommander> wgrant: care to be more specific?
<wgrant> NCommander: Core relevant members of the distro team.
<lifeless> NCommander: start with cjwatson or persia
<lifeless> as a hint
<wgrant> Something like that, yes.
<NCommander> lifeless: I've been talking with persia, but I need to poke someones brain about the current implementation in launchpad because I'm not quite sure I have a full grasp of its current status and/or limitations.
<wgrant> NCommander: Extending it to binary packages is relatively easy.
<wgrant> If such functionality is desired.
<wgrant> Which it may be, to perhaps obsolete cron.germinate's Task extra-overrides.
<NCommander> wgrant: well, talking with persia, to fully replace the current OGRE mode, we need per-binary package set selections
<NCommander> wgrant: since we do have source packages who binaries are split between main/universe
<wgrant> Ah, yes, true.
<wgrant> So, get cjwatson/TB to work out what needs doing, and ask for it.
<wgrant> Extending the current implementation to binary packages is not difficult.
<NCommander> wgrant: right, but to make a proposal, I need to know where we current stand w.r.t. to backend capability :-)
 * NCommander has had some interest in getting involved in soyuz development as a hobby for some time now and has quite a lot of interest in assisting with archive reorg
<NCommander> wgrant: correct me if I'm wrong, but the basic way to getting packagesets implemented into LP would be extending IPackageset to have binary methods in addition to the current source methods. The other issue I see is packagesets don't seem to work against series; its an all or nothing things looking at IPackageset
<EdwinGrubbs> wgrant: ping
<wgrant> EdwinGrubbs: Hi.
<wgrant> NCommander: I thought packagesets were series-specific now
 * wgrant checks.
<wgrant> NCommander: packageset.distroseries
<NCommander> wgrant: oops
<NCommander> wgrant: so that removes that concern. Then all thats left is a way to assiocate packagesets with a BinaryPackageRelease record so sets can be added like sections/component overrides can
<wgrant> NCommander: History is not currently stored for source<->packageset links. This needs to be discussed
<NCommander> wgrant: history?
<wgrant> NCommander: At the moment, you can see the full component/section/priority history for a source or binary package.
<wgrant> You can do no such thing with package set memberships.
<wgrant> This could be confusing when it starts being used for ogreing.
<wgrant> And package set membership should probably be between a Packageset and a BinaryPackageName, not a BinaryPackageRelease.
<NCommander> wgrant: how is history recorded for those?
 * thumper fires off an ec2 land before closing laptop
<wgrant> NCommander: By keeping multiple (Binary|Source)PackagePublishingHistories.
<wgrant> NCommander: They are the things that link an archive-agnostic BPR to an archive, in a particular component, section and (for binaries) priority.
<wgrant> When an override is changed, a new [SB]PPH is created, and the old one is marked Superseded.
<wgrant> So you can see the full history.
<NCommander> wgrant: so we need a SourcePackagesetPublishingHistory(Public), and records to be created for each change
<wgrant> NCommander: I don't know. I don't think this has been discussed before.
<NCommander> and when binary packagesets materialize, BinaryPagesetPublishingHistory
<wgrant> This also makes it just about impossible to change package sets in post-Release pockts.
<EdwinGrubbs> wgrant: hi. you were mentioned as somebody that might have insight into the benefits of crowdsourcing the linking of projects to upstream bugtrackers and linking the development focus series to an imported upstream branch.
<NCommander> wgrant:?
<wgrant> EdwinGrubbs: Yes. At the moment Ubuntu bug triagers can't do any of that, so are crippled and have to go and talk to ~registry or $ARBITRARY_PROJECT_REGISTRANT in order to streamline the upstream integration workflow with projects that do not maintain an LP-hosted presence.
<NCommander> wgrant: oh, because packagesets have no concepts of components, right?
<wgrant> NCommander: No concepts of *pockets*.
<wgrant> They don't need to know about components.
<NCommander> s/components/pockets/g :-)
<wgrant> But at the moment I can technically promote something in -updates, for example.
<wgrant> This is potentially useful, although it needs discussion.
<NCommander> wgrant: its been done before
<wgrant> It would be unfortunate if this capability was lost.
<NCommander> (promotion in a pocket)
<wgrant> Right.
 * NCommander remembers we had a KDE bug that needed that done, and we had to test on dogfood to make sure Soyuz won't blow up on publishing
<wgrant> Then again, somebody has also done a promotion in the Release pocket after a release, which ends up a little odd...
<wgrant> Yep.
<NCommander> wgrant: LP prevents promotion in Release after release
<wgrant> NCommander: Heh, well, it should.
<wgrant> But it doesn't.
<wgrant> The new publication will just sit there forever.
<wgrant> (because the publisher won't touch new records in Release)
<NCommander> wgrant: it doesn't?
<NCommander> ugh
<wgrant> I need to fix that.
 * wgrant -> uni
<NCommander> wgrant: cya later
<mwhudson> gosh lunchtime already
<mwhudson> where is the day going
#launchpad-dev 2010-03-11
<mwhudson> thumper: woo, new code import mail with actual information in it \o/
<jelmer> mwhudson: it looks like bzr-git will switch to a new caching format as of the next version
<mwhudson> jelmer: does that mean we'll need to do something to update the old caches?
<jelmer> mwhudson: better performant, but I guess it'll also mean there's some changes that have to happen in lp-code to cope with new filenames, etc
<mwhudson> or will regenerating them work?
<mwhudson> yeah, guess so
<mwhudson> should be easy though
<mwhudson> jelmer: is the new format going to be pack based?
<jelmer> regenerating them should work, if it doesn't that's a bug in bzr-git :-)
<jelmer> mwhudson: not pack based, but based on bzr indexes (there's no content so no need for packs)
<mwhudson> ah, ok
<mwhudson> will it be a single file still?
<jelmer> no, it's a directory with multiple files
<jelmer> though if you wanted to you could trigger a repack so it ends up being just a single file (and a format indicator) in that directory
<mwhudson> mmf, that sounds boring
<mwhudson> probably tarring it up is easier
<mwhudson> jelmer: will it be .bzr/repository/git-cache or something like that?
<jelmer> mwhudson: yeah, .bzr/repository/git
<jelmer> mwhudson: of course, Bazaar should really have hooks for fetch that allow us to copy this data along with the branch..
<mwhudson> jelmer: have you seen how AMAZINGLY SLOW the kernel import is going? https://code.edge.launchpad.net/~kiko/linux/2.6.31
<jelmer> mwhudson: yeah
<lifeless> jelmer: yes, we should.
<jelmer> lifeless: is this the moment where you suggest I submit a patch ? ;-)
<lifeless> jelmer: I wouldn't insult your intelligence that way :)
<lifeless> actually this is not trivial to do well
<lifeless> and doing it badly will make push and pull suck arse for everyone
<jelmer> mwhudson: it'll be a bit faster with the new cache
<mwhudson> jelmer: i *think* we have tests that will fail when bzr-git changes here
<jelmer> mwhudson: also, there's some Google folks working on improving the performance of Dulwich, that'll help a bit
<mwhudson> jelmer: it will also be a bit faster when we get that new machine online...
<mwhudson> jelmer: heh, interesting
<jelmer> lifeless: what are you thinking might make it problematic ? Figuring out what revisions to fetch?
<mwhudson> do you know why it's so slow for the kernel?
<mwhudson> just because it's a big tree?
<lifeless> jelmer: not triggering VFS on every pull
<lifeless> jelmer: and integrating with streaming
<lifeless> how much to fetch and so on is also interesting
<lifeless> but I was figuring plugins will need their own protocol for negotiating that shit
<jelmer> lifeless: yeah - how hard is it to add custom commands to the smart server protocol?
<jelmer> hopefully using the bzr btree indexes will also make it a bit easier to just fetch those keys that we need - it'd be trickier than tdb
<wgrant> mwhudson: It seemed to go from 3-5 hours per run to > 12 a couple of days ago.
<jelmer> wgrant: it depends on the machine
<jelmer> galapagos in particular is very slow
<mwhudson> wgrant: galapagos seems particularly slow
<mwhudson> i think elmo said one of the machines was one of canonical's very first machines :)
<lifeless> jelmer: custom commands are easy; making it all hook in well not so much - we don't want plugin-count extra round trips either
<wgrant> mwhudson: Ah, yeah, but neumayer isn't fast either by the look of things.
<wgrant> Yeah, galapagos is from the original naming scheme, so it makes sense that it's old...
<mwhudson> oh right, yeah
<jelmer> lifeless: so even a single extra roundtrip if there's no data would be a problem? I guess in that case we really need something like what's specified in the 'branch baggage' spec to make it perform well
<mwhudson> neumayer and russkaya are both antartic bases
<wgrant> Yep.
<mwhudson> there's a new machine ready and waiting apparently
<lifeless> jelmer: something-mumble-mumble
<wgrant> Penguins, Antarctic bases, elements, fruits... is that it?
<lifeless> jelmer: imagine 10 plugins doing this
<mwhudson> waiting for our sysadmin to recover from dental surgery :/
<wgrant> mwhudson: Yeah :(
<lifeless> jelmer: if that was a rtt per just to probe, we'd spend 3 more seconds on an empty pull
<lifeless> for me <-> London
<mwhudson> wgrant: yeah, still on friut afaik
<mwhudson> jelmer: in other news
<mwhudson> jelmer: bzr-hg's tests are appalling
<jelmer> mwhudson: don't say I didn't warn you :-P
<mwhudson> this is true
<jelmer> mwhudson, thumper: oh, btw, is it possible to upgrade lp:django ? www.isgitbetterthanx.com uses it to show that bzr is slow, and since it uses 0.92 it's indeed very slow
<mwhudson> jelmer: with a losa to help, yes :/
<mwhudson> jelmer: ask a question?
<jelmer> against launchpad-code and assign to losas?
<mwhudson> yeah
<mwhudson> i'll post some more details
<lifeless> huh
<lifeless> can't a branch expert upgrade it on the web ui?
<mwhudson> not for an import branch, no
<EdwinGrubbs> wgrant, jelmer: Can you look at this proposal if you have time? https://dev.launchpad.net/Registry/InvolvementPortletRefactor
<wgrant> EdwinGrubbs: Looking.
<EdwinGrubbs> sorry, I have to leave now
<jelmer> EdwinGrubbs: Sure, I'll have a look (might be tomorrow morning CET though)
<mwhudson> grar
<mwhudson> calling mercurial.commands.commit() in a test is asking me to choose an editor
<jelmer> mwhudson: I think you can specify a custom ui object to prevent that
<jelmer> mwhudson: we have one that wraps a bzr ui factory
<mwhudson> yeah, seems likely
<mwhudson> oh right
<mwhudson> woo, i think my tests are ok now
<mwhudson> now why doesn't stop_revision to InterBranch.fetch() work...
<thumper> and now to attempt to focus on work...
<mwhudson> jelmer: still there?
<jelmer> mwhudson: yeah
<mwhudson> jelmer: i'm not sure where to do the limit of importing in bzr-hg
<mwhudson> it doesn't seem to produce a nice toposorted list of the hg revisions to import anywhere
<mwhudson> unless a mercurial.changegroup.chunkiter() iterates in toposorted order?
<jelmer> yeah, that's guaranteed to be toposorted
<jelmer> although it would be nice to not fetch those revisions in the first place
<thumper> jelmer: any movement on the bzr-hg bug that is causing 80% of them to fail?
<mwhudson> jelmer: is that going to be possible?
<mwhudson> given a head, you'll have to walk back to null to find which ones to import first, won't you?
<poolie> thumper: is a 'revision feed' a feed of all revisions for a particular project
<mwhudson> poolie: or person
<thumper> or team
<thumper> or project group
<thumper> poolie: but yes
<lifeless> not quite
<thumper> ok
<thumper> latest revisions
<lifeless> there is a feed of new braches
<thumper> in the last 30 days
<lifeless> and per branch a feed of revisions
<lifeless> I filed a bug asking for project feeds the day they went into beta
<thumper> lifeless: and a revision feed for a project for over a year
<thumper> lifeless: and I did it
<jelmer> mwhudson: I think that might be possible by doing the right thing in findmissing()
<lifeless> thumper: ah! For some reason I thought it hadn't been done.
<lifeless> thumper: sorry!
<thumper> lifeless: it has been so long I've forgotten when I did it
<jelmer> thumper: not yet, I need to spend some quality time with a particular branch that's failing to figure out what's going wrong
<jelmer> thumper: and preferably write some tests that demonstrate the problem, that bit is probably the hardest
<thumper> jelmer: ok
 * mwhudson is completely confused
<mwhudson> making a method which is called in one place which doesn't look at the return value return something causes tests to fail
<mwhudson> ooh
 * mwhudson slaps himself
<mwhudson> jelmer: findmissing() is fairly opaque to me :/
 * mwhudson stares at it some more
<thumper> mwhudson: what are you doing?
<mwhudson> thumper: incremental imports for bzr-hg
<thumper> ah
 * thumper is still doing code review comment emails...
<jtv> wgrant: bob seems to think it's building for me
<wgrant> jtv: I saw your bug reports that implied that, yeah.
<wgrant> Is it actually working?
<jtv> no
<wgrant> :(
<wgrant> What does the slave think it's doing?
<jtv> that's what I wanted to ask you: is there anything more to look at slave-side than the launchpad-buildd.log?
<wgrant> jtv: No. That's all there is.
<jtv> and I just zapped that for my next attempt :)
<wgrant> But it should give you lots of information.
<jtv> I can go through the setup pretty quickly now though
<jtv> There wasn't much in it last time
<wgrant> Why are you reinstalling each time?
<jtv> Am I?
<jtv> I mean, I'm reinstalling every time I reinstall, obviously, but...
<wgrant> Why do you have to go through the setup again?
 * NCommander wonders if anyone can explain why we keep copyright file contents in the database and not in librarian
<wgrant> NCommander: I asked that myself.
<NCommander> wgrant: was there an answer?
<wgrant> NCommander: No.
<jtv> wgrant: various reasons.  So far I haven't had to re-do the slave setup at all.
<jtv> I have had to reboot a few times though.
<jtv> wgrant: in many cases I had changes I _could_ just patch through without re-doing anything, but didn't want to because I'm trying to write up something reproducible.  It's easy to mess that up without knowing it.
<mwhudson> i don't like the mercurial code very much :/
<wgrant> jtv: Ah, right.
<jtv> wgrant: for example, between henning's work and mine we had a case where for him "dpkg -i <package> ; apt-get -f install" worked, and for me the other way around worked, because the --force-depends was missing from the dpkg command line.
<jtv> We'd like one order that works for everyone, but differences in our setup hid the problem in different ways
<wgrant> The dpkg, apt-get order should always work.
<wgrant> If it doesn't, give me examples because I don't believe you :P
<jtv> Note the bit where I said --force-depends was missing...
<jtv> if you do dpkg -i first, well, launchpad-buildd won't install because the dependencies aren't there
<jtv> if you do apt-get -f first, I don't think there's necessarily anything there that'll make it install all dependencies
<jtv> for the package you have yet to install
<jtv> ...or maybe other stuff was broken in my case, which is the sort of thing you get when you don't re-do from scratch from time to time, which was my original point :-)
<wgrant> dpkg -i should keep the package installed but unconfigured.
<wgrant> Then apt-get -f install will satisfy the dependencies and finish installation.
<jtv> dpkg -i leaves it unconfigured?
<wgrant> If the dependencies are not satisfied, yeah.
<jtv> Anyway, when you do dpkg -i first, it refuses service because the dependencies aren't there, doesn't it?
<jtv> The --force-depends forces it to break things, and then the apt-get -f can fix the deliberate breakage
<wgrant> It probably looks like it refuses, but it actually lets it go half way.
<jtv> ah, like so
<jtv> wgrant: meanwhile, it seems as if my job has stopped cold somewhere below the radar :(
<jtv> The buildd-manager log says it's dispatching.  The slave log says nothing.
<jtv> Maybe the previous instance of the slave, which I killed a little late, grabbed it.
<wgrant> jtv: Is buildd-manager repeatedly dispatching it?
<jtv> no, just once
<wgrant> And can you see buildd-manager currently polling in the slave log?
<jtv> The launchpad-buildd log on the slave ended with "daemon ready."
<jtv> So I think it went to the wrong buildd instance... I'm retrying.
<jtv> wgrant: marking Bob as not-OK is not stopping its ongoing job this time...
<jtv> Does changing the OK setting POST to the slave on xmlrpc?
<wgrant> jtv: Only buildd-manager can talk XML-RPC to the slave.
<wgrant> Marking it not-OK will stop scanning.
<wgrant> It will not touch the slave, but it should deassign the job in the UI.
<wgrant> And when you mark the builder OK again, buildd-manager should declare the builder lost (since it's building an unassigned job), and abort the builder.
<wgrant> The lost builder detection may not work in this case, however.
<jtv> That sounds plausible
<jtv> what would break it?
<wgrant> It being crap.
 * wgrant finds the bug.
<jtv> wgrant: ahhhh, _that_ explains _everything_!  :-)
<wgrant> Bug #463041
<mup> Bug #463041: Lost builder detection is insufficiently aggressive <buildd-manager> <Soyuz:Triaged> <https://launchpad.net/bugs/463041>
<wgrant> Is there a way to make a @stepthrough consume multiple segments?
<wgrant> I want a @stepthrough method to take multiple arguments, without having to create intermediate objects.
 * thumper stabs his tests
<mwhudson> wgrant: i think you need to write your own navigation (?) to do that
<mwhudson> there's something more general than @stepthrough, anyway
<wgrant> The closest thing I can think of is that BranchNamespace thingy.
<mwhudson> yeah
 * jtv has to be off for a bit
<mwhudson> lp.registry.browser.person.BranchTraversalMixin
<wgrant> mwhudson: Ah, thanks.
<thumper> gah
<thumper> grrr
<thumper> testing jobs is a PITA
<wgrant> It seems sort of evil, but it might work.
<jtv> amen to that
 * mwhudson EODs
 * thumper waves at mwhudson
 * thumper sprinkes some removeSecurityProxy around
<mwhudson> heh heh
<poolie> thumper: did robert ask you yesterday what happens if you set a bmp to state 'queued'?
<poolie> does the world end? does the ui cope?
<thumper> poolie: he did
<thumper> poolie: I suggested that he shouldn't
<thumper> poolie: weird shit may happen
<thumper> it might work
<thumper> it might not
<lifeless> poolie: thumper also said that staging would probably be a good enough indication
<thumper> it should
<lifeless> of what weird shit would happen
<thumper> it'll either work or not
<thumper> staging should be able to tell you
 * persia wonders of !ohmy is supposed to apply in launchpad channels
<wgrant> It is often violated here, but it seems unproblematic.
<lifeless> !ohmy?
<wgrant> !ohmy
<wgrant> No bot :(
<lifeless> so
<lifeless> what doth it mean?
<wgrant> 15:55:19 <ubottu> Please remember that all Ubuntu IRC channels share the same attitude of providing friendly and polite interaction with all users of all ages and cultures. Basically, this means no  foul language and no abuse towards others.
<lifeless> ah
<thumper> persia: NZ has a slightly more relaxed attitude to swearing
<thumper> persia: there are several prime time adds that use "bugger" and "bloody idiot"
<thumper> sorry if I offended anyone
<persia> thumper: I'm sure.  I don't claim that ohmy applies here, just asking.  Mostly because of e.g. http://irclogs.ubuntu.com/2010/03/11/#launchpad-dev.txt
<thumper> 'twas not my intention
 * persia certainly isn't offended, but is so used to typing !ohmy in other channels, that the message had to be erased before the query
<thumper> which bit?
<lifeless> persia: I don't see the thing that triggered you to say that
<poolie> lifeless: i think you meant to say "wtf does that mean?" :)
<poolie> the s-word
 * thumper EODs
<thumper> later people
<poolie> cheerio
<poolie> thump em and thump em again
<lifeless> oh, wow.
<persia> heh
<poolie> persia: i think we share that attitude but i don't think tim's expression trips it
<poolie> this may just show i need sensitivity training
<wgrant> In some channels it does.
<lifeless> as colloquial for 'things' 'my shit', 'weird shit' etc are pretty ingrained. 'your shit' would be an aggressive use and would trip my bad-censor
<lifeless> I blame the US for popular culture and tv shots.
<lifeless> s/shots/shows/
<persia> poolie: Personally speaking, I'm open to lots of expressions.  That said, I've seen long-winded arguments about usage of terms, and self-admitted 8-year-old people in some development channels.
 * persia agrees with lifeless
<lifeless> ok, EODing
<poolie> bye
<NCommander> where can I get a log of the local keyserver? I'm getting a 500 when I try to create a soyuz user
 * persia has a sense that the query is dropping into the nebulous dark span that lies between NZ and EU timezones
<NCommander> nm, got it by killed twistd and restating everything
 * wgrant appears.
<wgrant> NCommander: It all ended up working?
<NCommander> wgrant: when applied a heavy enough wrench, although the soyuz script didn't add my GPG key to the ppa-user, and I'm getting issues with rejection based on "Can not build for architecture: any"
<wgrant> NCommander: You specified the right email address?
<wgrant> The "Cannot build for architecture" thing normally means that your DASes either aren't enabled for virtualisation, or do not have chroots.
<NCommander> wgrant: yes, and the key posted to the internal keyserver
<NCommander> wgrant: it might have gotten confused though, I have my keys on a GPG smartcard
<wgrant> Hmm.
<NCommander> which makes gpg --list-keys look a bit different
<wgrant> Ah.
<NCommander> wgrant: second BTW, for pygpgme, there's a patch in the archive that fixes it
<wgrant> jtv: Did you have any more luck with the slave?
<wgrant> NCommander: We don't use the archive version :(
<jtv> wgrant: no...
<NCommander> wgrant: right, but that patch can just be applied to the version in the tree
<wgrant> jtv: You didn't try, or it didn't work?
<NCommander> wgrant: I would do it, but I know diddly about pushing to launchpad-pqm or doing a review on an external component
<jtv> wgrant: I didn't try, and waiting didn't help either :)
<jtv> (helping out with something urgent here)
<wgrant> Ah.
<NCommander> wgrant: can you help me get ready to push my first changes to LP? I have a database change that needs to be made, and I'm kinda confused on how I edit the schema
<wgrant> NCommander: Sure. What's the change?
<NCommander> wgrant: first pat of implemented raw source changelogs
<NCommander> wgrant: the part I wrote causes new packages to have the changelog stripped out and put in the database
<NCommander> on upload
<wgrant> NCommander: ie. new column SPR.changelog -> LFA.id?
<NCommander> wgrant: no. just SPR.changelog
<NCommander> per bigjools and sadbfl's comments
<wgrant> But changelogs get unsmall.
<persia> THey don't "get" unsmall, they are by default.  Still : https://bugs.edge.launchpad.net/soyuz/+bug/139028/comments/1
<mup> Bug #139028: SPR.changelog should be renamed to 'changelog_entry' <soyuz-core> <trivial> <Soyuz:Fix Released by julian-edwards> <https://launchpad.net/bugs/139028>
<wgrant> Hmm.
<wgrant> Has the DBA been consulted?
<lifeless> pep8 names are  preferred
<lifeless> if thats your concern
<wgrant> lifeless: I'm more wondering about the storing large strings in the DB thing.
<NCommander> wgrant: well, worse case, I rewrite a bit of code, and some comments :-)
<wgrant> stub: Do you have an opinion on this?
<NCommander> wgrant: while we wait, maybe you can explain how do I integrate a schema change? I've already gotten the bit htat I need db-devel
<wgrant> NCommander: Ah, right. I tried to find the wiki page, but then U1 ate my bandwidth.
<wgrant> I've killed it, so maybe it will work better this time.
<wgrant> I believe that https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess is the latest.
<NCommander> wgrant: cool, so now I need to learn how to interact better with bzr
 * NCommander reads on bzr loom
<wgrant> NCommander: I prefer pipeline these days.
<NCommander> wgrant: pipeline?
 * NCommander admits he perfers git when dealing with non-trivial archives
<wgrant> NCommander: https://launchpad.net/bzr-pipeline
<wgrant> NCommander: Why?
<NCommander> wgrant: git rebase is my best friend with dealing with local trees before submisison
<wgrant> NCommander: Ew.
<wgrant> History destruction is revolting.
<wgrant> (but bzr can rebase too)
<NCommander> wgrant: having all of ones changes ontop of the history is very handy
<NCommander> But we digress
 * NCommander decides not to bother with looms or pipeline with this change
<adeuring> good morning
 * thumper prefers pipelines too
<stub> Whoops.
<lifeless> thumper: different use cases
<lifeless> thumper: hey, how is your bzr using project going
<thumper> lifeless: slowly
<thumper> lifeless: how to I create a preview tree for a particular revision?
<thumper> lifeless: I want to commit to it
<thumper> lifeless: and either pull or merge into the main branch
<stub> wgrant, NCommander: (from 1 hour 15 mins ago) It depends :-) changelogs are not that big. If we want to search on them, they certainly can go in the database. If we don't, we are juggling the convenience of stuffing them in the database vs. sticking them in the librarian. I want to keep binary blobs out of the database, so if the text is in an unknown encoding it should definitely get stuffed in the Librarian.
<lifeless> thumper: a preview tree is the result of a merge done into a revision tree AIUI
<lifeless> thumper: so I'd look at the pipeline pump code
<thumper> lifeless: ok
<thumper> lifeless:  a revision tree is tree based on a revision?
<lifeless> you can commit from the transform I believe
<thumper> lifeless: can you commit to a revision tree?
<lifeless> thumper: a revision tree - b.basis_tree(), or b.repository.revision_tree(arbitrary_revid)
<lifeless> thumper: no
<lifeless> revision trees are immutable
<lifeless> 'MutableTree.commit()' commits a new tree to a branch.
<NCommander> stub: we already have copyright in the database and the encoding is guessed (although I realize that kinda is hit or miss). sabdfl made a comment that we should stuff it in the DB already as well
<lifeless> thumper: give my phone a ring; just going for a walk and voice will sort this quicker
<NCommander> stub: plus I want to retroactively add all the past changelogs, which is going to be about 5 GiB when done
<stub> NCommander: What do we do with them once they are stuffed in the database? Are we going to display them inline in the web UI or similar?
<NCommander> stub: ideally
<stub> If we are guessing encoding, is there a use case for keeping an unadulterated copy in the Librarian (eg. are they GPG signed)
<NCommander> stub: the file ripped out of the source package, the original can always be retrieved there
<NCommander> stub: right now, the code just exists to put them in the database. I need to talk to bigjools and persia on how we want to change +changelogs and the SRP page specifically.
<stub> Yup. Also, do we want to store the entire changelog (a potential DOS perhaps), or truncate it to some sane limit.
<wgrant> We pretty much need the whole thing.
 * NCommander thinks wgrant can explain better than I can
<wgrant> Since we need to be able to produce an Ubuntu->Debian changelog delta.
<stub> Basic procedure for altering the schema is to create a database/schema/patch-2207-99-0.sql file containing a sequence of CREATE TABLE, ALTER TABLE etc. DDL commands. The first line and last line are dead chickens you can steal from one of the other patches. When requesting a review, request 'db' reviews from me (stub) and bjorn (bjornt).
<stub> So if I upload a sourcepackage to my PPA that when uncompressed contains a 60GB changelog.... ?
<wgrant> Right, we're screwed.
<thumper> lifeless: sorry, called away
<thumper> lifeless: I'll talk to you tomorrow about it
<stub> I suspect we do want some limit. Just pick an arbitrary number we won't hit except in case of a) major screwup or b) attack.
<jtv> wgrant: meanwhile, on my simulated buildd, I've deleted the job that was in the way but my own is just not getting picked up.  :(  Still sits there w status 0
<NCommander> er
 * NCommander coughs
<wgrant> jtv: What does the query do?
<jtv> wgrant: the candidate selection query?
<noodles775> jtv: put a break point in the candidate selection and follow it through.
<jtv> Last time I tried that, it worked.  Maybe it's failing for some new reason.
<noodles775> Aha.
<stub> NCommander: Of course, the 2GB limit on a text column is the arbitrary limit if you don't set one, so it probably doesn't matter.
<NCommander> stub: ugh :-/.
<noodles775> jtv: related to bug 536797
<mup> Bug #536797: Builder pages break for job without build <wellington> <Soyuz:New> <https://launchpad.net/bugs/536797>
<jtv> noodles775: no, it's before it gets to that
<noodles775> jtv: I think it's straight forward to ensure that the builder index works for your jobs, but the harder part is how we present the history.
<NCommander> stub: thats really nasty :-/
<noodles775> jtv: (no, I have a question about it ;)).
<jtv> ah :)
<jtv> (al-maisan, what's supposed to happen right after this query?  SELECT SUM(BuildQueue.estimated_duration) FROM Archive, Build, buildpackagejob, BuildQueue, DistroArchSeries, Processor WHERE buildpackagejob.job = BuildQueue.job AND buildpackagejob.build = Build.id AND Build.distroarchseries = DistroArchSeries.id AND Build.archive = Archive.id AND DistroArchSeries.processorfamily = Processor.family AND Build.buildstate = 0 AND Archive.ena
<jtv>  )
<noodles775> jtv: Currently the builder histor relies on the builds (as they are the only permanent record that something happened on the builder), but I'm guessing we'll need to modify it to instead rely on BuildFarmJob records, and not delete those immediately. Let me know what you think (you too wgrant) and I'll update the bug.
<jtv> noodles775: I think I see what you mean.  Yes, getting this right is going to be an interesting question; it also came up in the sprint.  My immediate worry was getting it not oopsing though.  :-)
<al-maisan> jtv: where's that query from?
<wgrant> noodles775: I think we'll have to present each Job, with adapters for each type.
<jtv> al-maisan: the buildd manager
<jtv> I agree with wgrant
<wgrant> noodles775: Hmm, I guess TTBJs don't actually have a builder once the BuildQueue dies, do they?
<jtv> (I have it not oopsing for now)
<noodles775> wgrant: nope, we'd need to defer the deletion of those, perhaps truncating the history to 1000 BuildFarmJobs or something.
<wgrant> I think we have too many tables.
<noodles775> +1
<wgrant> noodles775: Why truncate them?
<noodles775> wgrant: at the moment we delete the BFJ, Job, QueueItem etc. once the (soyuz) build completes.
<wgrant> noodles775: I think just the BuildQueue is deleted.
<wgrant> The other three (Build, BPJ, Job) should survive.
<wgrant> I hope.
 * wgrant checks.
<noodles775> Yes, the Build is permanent of course, but I'd thought the others were deleted, but haven't checked. Doing so now too.
<jtv> wgrant: there's some propagation of deletions in the python though
<wgrant> noodles775: BuildBase appears to kill the buildqueue.
<wgrant> But not the others.
<lifeless> thumper: de nada
<wgrant> Oh.
<wgrant> But the BuildQueue kills the rest.
<wgrant> Damn.
<noodles775> Yep.
<wgrant> I think we should just keep them all.
<al-maisan> FWIW: we also discussed briefly *not* deleting the BuildQueue records upon job completion and putting in place a purge mechanism that deletes them after some configurable period of time (e.g. a month)
<wgrant> We are going to have at least three five rows in other tables from each build for eternity anyway.
<wgrant> So three extra very tiny rows for each really shouldn't make much difference...
<NCommander> stub: so no issue with changelog in database aside from a sanity check on size?
<wgrant> Er, s/three // in that first sentence.
<wgrant> al-maisan: IIRC we argued this during the sprint, and nobody brought up a compelling reason to purge them.
<al-maisan> wgrant: data volume?
<wgrant> al-maisan: It's trivial compared to the other stuff we store for each build.
<noodles775> jtv: so you've got a branch that ensures the builder-index doesn't oops for jobs without builds?
<wgrant> (the build, build log, changes file, upload entries...)
<jtv> noodles775: yes
<jtv> not pretty, but...
<wgrant> jtv: Does it just not link them if they have no build attribute?
<stub> NCommander: Should be fine. (Assuming this is SourcepackageRelease) We probably want to split out the bulk data from SourcepackageRelease such as copyright, dsc_* etc. into a seperate table as it is getting pretty fat. Fine in theory, but because we use an ORM we generally load all columns even when we don't need them.
<wgrant> al-maisan: Plus, if we keep some of the others than we can trim Build.
<al-maisan> wgrant: I am inclined to agree .. however: we currently operate on a BuildQueue table with a very *small* number of rows and if that were to change we may see some performance issues
<NCommander> stub: ugh :-/. So maybe make a sourcepackagerelease_metadata in the nearish future?
<jtv> wgrant: close.  It names the job type and says there's no build for it; and it shows the log regardless of who you are.
<stub> NCommander: Yes, but with a name that doesn't suck :)
<bigjools> morning guys
<NCommander> stub: I'm from the platform team, we have a bad history of sucky names. (i.e., bundles, petals, leafs, etc. :-))
<wgrant> Morning bigjools.
<noodles775> Morning bigjools
<al-maisan> Good morning bigjools
<wgrant> NCommander: Heh, I remember that discussion.
<stub> NCommander: It might not be a problem in reality - it depends if we have pages that load a large number of SourcePackageRelease records
<NCommander> stub: suggestions welcome
<NCommander> stub: no, its usually just loaded one at a time except for +copy-packages (I think)
<stub> SourcePackageReleaseData sucks less in my opinion
<wgrant> NCommander: It's a lot more than that, sadly.
<wgrant> NCommander: Anything that displays a version number is loading an SPR to get it.
<NCommander> wgrant: ugh. Maybe I'll make mychangeset grow so we only need one or two hits against db-devel
<NCommander> wgrant: ow.
<stub> But that can be future anyway - mention it in the review request and open a bug if you can be bothered.
<lifeless> hey
 * NCommander remember he could make the database timeout by access the archive copy-package page 
<lifeless> is there a 'create a project' API script yet ?
<NCommander> Doing the near-equivelent ofSELECT * FROM SourcereleasePackages didn't make the database happy :-)
<wgrant> bigjools: Do you have an opinion on preserving the buildqueue, job and specificjob for completed builds?
<bigjools> NCommander: that happens because it does a lot of checks
<bigjools> wgrant: strongly against preserving
<wgrant> :(
<wgrant> kfogel: Hm, that single "Merging db-stable" is interesting.
<wgrant> Not sure why that is.
<bigjools> wgrant: why has this come up?
<wgrant> bigjools: There was a little bit of a discussion about how to present a generalised history.
<mrevell> Morning
<bigjools> those table rows are cruft once the job is done.  We should find a better way of keeping history.
<bigjools> and I really, really like having a single table that contains all the waiting jobs
<wgrant> bigjools: OK, so maybe BuildQueue could be purged. But the Job seems useful.
<wgrant> (in the current model, all record of a translations job is destroyed once the build completes)
<bigjools> doesn't it have build records?
<wgrant> I don't believe so.
<wgrant> jtv: ^^?
 * jtv reads
<jtv> nope
<jtv> No builds.
<bigjools> it should do!
<jtv> Would they be the same kind of build records?  Do build records need to go into buildfarm?
 * jtv wonders if we should have a buildfarm team
<bigjools> please no more teams :)
<bigjools> so noodles775 and I discussed last night that we would rename Build -> BinaryBuild and then a new Build table would have common fields from all build types
<wgrant> kfogel: Ah! They were merged in db-devel r8326, which is the rev before the one c-c.py listed as being the first merge.
<lifeless> wgrant: pop quiz, connecting lplib to staging
<bigjools> having persistent data hanging around in a set of tables that represent a queue is crack IMO
<jtv> bigjools: and those will stay around "forever"?
<bigjools> jtv: yes
<noodles775> bigjools: but that's only really suitable for SPRecipeBuilds and BinaryBuilds... not for the TranslationTemplatesBuildJob tyep.
<noodles775> I thought.
<wgrant> lifeless: Launchpadlib.login_anonymously('your app', 'staging')
<wgrant> bigjools: Does Job represent a queue?
 * noodles775 reads the back-scroll.
<bigjools> noodles775: if that's the case then we should take a look
<lifeless> wgrant: is there a constant like EDGE_SERVICE_ROOT ?
<bigjools> wgrant: no, but buildqueue does :)
<wgrant> bigjools: right.
<jtv> well, all 1:1 anyway in the cases we care about...
<wgrant> lifeless: I think they're deprecated by the simple strings. But yes, STAGING_SERVICE_ROOT should exist.
<jtv> bigjools, noodles775: anyway the new Build could be just an id to hold things together and still be useful.
<bigjools> jtv: id, builder are the 2 main things
<noodles775> OK, so if there was a build for TranslationTemplatesBuildJob's then that would enable us to generalise for all three build farm job types.
<lifeless> wgrant: is there any convention for environment variables to control this?
<jtv> bigjools: I'm sure there's stuff like dates that would otherwise perish with the more transient job/buildqueue rows
<noodles775> But I think we need to think through this carefully.
<bigjools> yes very carefully
<jtv> why?  hack the thing up & roll it out I say!
<lifeless> wgrant: like LAUNCHPAD_API=staging
<jtv> phone
<noodles775> lol
<wgrant> lifeless: Not sure.
 * lifeless makes one up
<bigjools> jtv: date is also another good one :)
 * noodles775 updates the relevant bugs trying to summarise.
<wgrant> noodles775, bigjools: Why Build rather than Job?
<jtv-otp> wgrant: lets us denormalize to our little hearts' content
<jtv-otp> s
<wgrant> All build farm jobs might not always be Builds.
<al-maisan> the actual build duration is another piece of data  for the generic `Build` table
<bigjools> build farm jobs might not always be Jobs
<bigjools> no wait :)
<bigjools> Jobs might not always be build farm jobs, is what I meant to say
<wgrant> And indeed they are not. True.
<wgrant> So.. we need BuildFarmJob?
<bigjools> or... Build :)
<bigjools> which is the same thing
<bigjools> but it could be called BuildFarmJob
<wgrant> I thought you were averse to the idea of calling something Build ever again.
<bigjools> I'm not precious about it
<wgrant> I think we need a good list of all the columns on all the tables, and what we need to keep.
<wgrant> Is there such a thing around?
<bigjools> I don't even remember what I did yesterday so if I said that then great :)
<bigjools> such a thing is the db schema. no?
<wgrant> The second point is a subjectively processed version of the DB schema.
<bigjools> if we could factor stuff out, that'd be great
<bigjools> yeah it needs analysis
<lifeless> wgrant: thanks
<noodles785> Not that I know of, I'll try to create something like http://people.canonical.com/~michaeln/bfb/sourcepackagerecipe.png for the build/queue related tables today, if it's helpful.
<bigjools> noodles775: if it's easy, yes please
<lifeless> now thats fairly cool
<lifeless> oauth tokens for edge work on staging
<bigjools> noodles775, wgrant, jtv-otp: Job on its own is too generic for a build farm job.  I'd even be happy to ditch our use of Job and migrate to BuildFarmJob completely.
<wgrant> bigjools: Yeah, quite possibly.
<lifeless> call it task
<lifeless> that way you can really confuse people
<bigjools> I've never been really happy with using Job
<jtv-otp> bigjools: that whole subsystem needs a cleanup... not sure I'd enjoy yet more duplication
<bigjools> I felt press-ganged into using it
<wgrant> It never seemed like a really good idea to use it.
<wgrant> We do not use much of it.
<bigjools> I continually fail to see the duplication between Job and build farm jobs
<bigjools> someone wanted the buildd manager to become more like a job runner
<bigjools> it's not going to happen
<jtv> bigjools, wgrant: I'm not sure the Job runner uses most of what you see in the schema there either.
<jtv> My impression is, Job failed to become all it was planned to be, with failover mechanisms that never materialized.  Okay, I'm not trying to force anyone to stick with Job, but it'd be nice to have something that everyone could use and be happier for it.  Message queue plus log table maybe?
<jtv> bigjools: unrelated note... what happens to a build slave when its chroot falls out of date?
<jtv> Would it be noticed at all?  Would jobs still be dispatched to it?
 * persia has seen the results of such dispatches
<persia> Dunno if it still can happen
<jtv> wgrant, could this be related to my buildd stuckage?  lib/canonical/twistedsupport/processmonitor.py:254: twisted.internet.error.PotentialZombieWarning: spawnProcess called, but the SIGCHLD handler is not installed. This probably means you have not yet called reactor.run, or called reactor.run(installSignalHandler=0). You will probably never see this process finish, and it may become a zombie process.
<jtv> persia: what did those results look like?  I'm talking about my local system, so weeeird things can happen :)
<persia> jtv: Generally FTBFS because the build-deps couldn't be installed.
<jtv> persia: hmm... so sounds like it wouldn't affect me.
<persia> jtv: chroot-out-of-date shouldn't do anything particularly odd, other than make the build take longer.
<jtv> persia: because of the apt-get update?  I'm skipping those in this case, re-using the same slave in the same state, so I guess for my case even that wouldn't apply.
<bigjools> jtv: sorry was otp
<jtv> it happens
<jtv> to all of us with hones
<persia> jtv: There's two bits.  The update, and the build-dep install.  But yeah, if neither of those breaks because of old-chroot, you shouldn't care.
<jtv> phones
<bigjools> jtv: the job runner is being improved all the time, but I see it as orthogonal to the build farm
<bigjools> jtv: as for chroots, what do you mean by out of date?
<jtv> persia: ok, thanks; that means I'd better look in other directions for the explanation of my troubles.  Thanks!
<jtv> bigjools: I've got a test setup as per https://dev.launchpad.net/Translations/BuildTemplatesOnLocalBuildFarm (which I'm writing), but this time around my jobs fail to dispatch.
<jtv> When I restarted everything just now, I got a timeout from an attempt to talk to the Librarian though.
<jtv> Maybe it's just the order in which I did things.
<bigjools> jtv: quite possibly
<bigjools> jtv: the builder does an apt-get dist-upgrade as part of setting up the chroot, BTW
<jtv> bigjools: as a prelude to dispatch, I guess?
<bigjools> yes
<jtv> Or in the actual slave-side FSM?
<bigjools> well the slave does it as part of kicking off the build
<jtv> ok, so that's not working I guess until I get my dispatch working
<bigjools> the chroot comes from the librarian so I guess not :)
<jtv> I'll try registering my chroot earlier on in the process
<jtv> okay, my librarian is down.  that'll do it.
<wgrant> jtv: The chroot stuff is all done as part of the build process.
<wgrant> As long as you've run ensurepresent with that SHA1 before, the librarian doesn't matter.
<wgrant> (as long as you're not calling ensurepresent again -- are you dispatching manually now, or with buildd-manager?)
<jtv> buildd-manager
<lifeless> wgrant: you might like  merge_sorted in bzrlib.tsort
<jtv> wgrant: I'm using manage-chroot; where would ensurepresent be called from?
<wgrant> jtv: ensurepresent is a buildd XML-RPC. It's called from buildd-manager.
<wgrant> lifeless: For what?
<jtv> wgrant: so would that fail right there if the chroot on the slave were out of date _and_ the librarian is down?
<lifeless> determining the revision thats shallowest
<lifeless> and/or which ones should show up in a log-like situation
<wgrant> jtv: "out of date" meaning you've uploaded a new one with manage-chroot.py?
<wgrant> lifeless: Hmm. Possibly, yes
<jtv> wgrant: I wouldn't swear to it never having happened since I created this slave image
<jtv> So I could just create a new slave image, but it takes a while
<wgrant> Or start the librarian.
<jtv> Fixing the librarian seems like a better idea.
<jtv> Exactly.  :-)
<jtv> I've started poppy... what else?
<jtv> start_librarian?
<wgrant> make run
<wgrant> Or that.
<wgrant> poppy isn't important for you.
<wgrant> Unless you're uploading source packages with dput.
<jtv> No, no package uploads for this
<jtv> I tried make run_codehosting
<jtv> ...which should have started the librarian.  ook.
 * jtv looks for pidfiles
<lifeless> wgrant: how do you say 'me' in LP apis ?
<wgrant> lifeless: lp.me
<lifeless> so team.owner = self.launchpad.me?
<wgrant> lifeless: Yes.
 * lifeless tries
<lifeless> hmm, its not .owner
<wgrant> Ah, yeah, team_owner
<lifeless> odd
<lifeless> oh well
<lifeless> IPerson claims owner, in the apidoc
<lifeless> and team claims to extend person
<wgrant> team has team_owner_link.
<wgrant> Where do you see IPerson's owner on +apidoc?
<lifeless> person ... owner 'Link to a person'
<lifeless> oh, method declaration
 * lifeless whines about apidoc again
<lifeless> I don't think I've seen a more confusing api documentation layout, ever.
<lifeless> wgrant: any idea what 'team_owner_link: Constraint not satisfied.' actually means?
<wgrant> lifeless: You're probably trying to unset it on a team, set it on a person, or set it to a Private Membership Team.
<lifeless> ah, no
<lifeless> I was trying to something I was sure worked, but apparently not.
<lifeless> which is to set it's owner to itself.
<wgrant> What was it?
<wgrant> Ah.
<wgrant> No.
<lifeless> Which means you can't have a trivial anarchy.
<wgrant> Correct.
<wgrant> There has to be someone at the root.
<lifeless> can you do A->B->A ?
<wgrant> Memberships can't work like that, so ownerships probably can't. But you could try.
<lifeless> meh, its ok
<lifeless> I just want to automate all the manual 'good practice' stuff I do on new projects.
<lifeless> if I was imagining that I did something, thats ok :)
<deryck> Morning, all.
<lifeless> how do you say 'project uses launchpad for bug tracking' via API's ?
<lifeless> likewise 'code is published in bzr branches' etc
<wgrant> How do I identify a binarypackagerelease's archtag?
<wgrant> Do I have to go through the build's DAS?
<deryck> lifeless, you want to set this via the API or just get at the info?
<lifeless> set it
<deryck> lifeless, there is official_malone on product, distribution, etc. but it's not exported in the API.
<lifeless> deryck: exactly :)
<deryck> :-)
<lifeless> deryck: in case you're wondering, I hit my DRY threshold for making projects on launchpad.
<lifeless> I already copy n paste 90% of the docs and settings do that.
<lifeless> s/do/doing/
<deryck> ah, ok.
<lifeless> while I can think of interesting things to do by reading these settings
<lifeless> I really do want to set them
<lifeless> do we have staging codehosting?
<wgrant> lifeless: Yes.
<wgrant> lifeless: The branch content isn't copied from prod, but you can push branches up.
<lifeless> bah
<lifeless> lp: doesn't seem to support it yet
<wgrant> lifeless: lp://staging/~blah/blah/blah
<lifeless> oh, handled in the xmlrpc server side?
<wgrant> That is all magic to me.
<wgrant> I haven't bothered to look.
<wgrant> bigjools: What should the BPRDC URL? The easiest appears to be something like +binaryhits/foo_1.0_i386.deb/2010-03-11/AU
<lifeless> wgrant: you know far too much about our systems :)
<lifeless> \o/
<lifeless> check out staging.launchpad.net/duh9
<wgrant> lifeless: Maybe.
<lifeless> hmm, maintainer isn't set yet
<wgrant> That was set up automatically?
<lifeless> yes
<lifeless> http://pastebin.org/109632
<lifeless> is there a better url than self_link ?
<wgrant> lifeless: web_link, but that doesn't exist yet.
<lifeless> is there a bug for that yet ?
<wgrant> Bug #316694
<mup> Bug #316694: Add web_link property to resources <launchpadlib :Triaged> <https://launchpad.net/bugs/316694>
<lifeless> ok, duh11 has the right maintainer
 * lifeless commits and proposes for merge
<lifeless> wgrant: lp:~lifeless/lptools/lp-project/ if you are interested.
 * lifeless now uses it to make his project
<lifeless> deryck: you might be interested in that script too
 * deryck looks
<lifeless> did we end up turning on auto-list-approval?
<lifeless> (the answer is yes)
<bigjools> wgrant: +binaryhits/foo/1.0/20100311/AU would be nicer
<lifeless> deryck: what do you think ?
<wgrant> bigjools: Yes, that's sort of what I have now, but the arch is a bit awkward.
<wgrant> bigjools: Should I just use the archtag of the build DAS? That will be a bit odd for arch-indep?
<bigjools> hmmm
<bigjools> it has precedent though
<wgrant> Where?
<deryck> lifeless, its nice, definitely.  I'll use it myself. :-)
<lifeless> deryck: :) high complement - thanks
<didrocks> mars: if you are interested to track the bug we discussed yesterday: bug #537298
<mup> Bug #537298: Launchpad make firefox rendering pages very slow with lucid version <Launchpad itself:New> <https://launchpad.net/bugs/537298>
<bigjools> wgrant: sorry, phone again. Meh, I thought it was used for DASSP[R] but those are linked to publications of course
<wgrant> bigjools: Yes :(
<bigjools> wgrant: I can't see any great harm just using the arch-indep though?
<wgrant> bigjools: ie. i386 instead?
<bigjools> yes
<wgrant> I think it is probably ambiguous, although it shouldn't be a problem in practice.
<wgrant> (think foo_1.0_i386.deb, then foo_1.0_all.deb appears in another upload.
 * jtv got his job to dispatch again, but has no clue why it fails
<wgrant> jtv: What does the slave log say this time?
<jtv> wgrant: it gets stuck...  I snipped copies at the bottom of the wiki page: https://dev.launchpad.net/Translations/BuildTemplatesOnLocalBuildFarm
<jtv> On the bright side, the librarian was working and it looks as if the slave tried to get the tarball
<wgrant> jtv: Does the fetch complete?
<jtv> I don't know.
<wgrant> sha1sum /home/buildd/filecache-default/f1f10b8402ed686aaf0307676c76f07b45af2a09
<jtv> f1f10b8402ed686aaf0307676c76f07b45af2a09  /home/buildd/filecache-default/f1f10b8402ed686aaf0307676c76f07b45af2a09
<jtv> (hey, we're a remote terminal session)
<jtv> $
<wgrant> So it worked.
<jtv> ...yup, that's the sha of my tarball.
<wgrant> jtv: Can you flip buildd-manager into debug mode?
<wgrant> I'm not sure if it's any more useful now, but it's worth a try.
<lifeless> gnight
<wgrant> Night lifeless.
<jtv> g'night lifeless
<jtv> wgrant: how do I do that?
<wgrant> jtv: s/logging.INFO/logging.DEBUG/ in lib/lp/buildmaster/manager.py, IIRC.
<jtv> ah
 * jtv edits
<jtv> now that would be a nice thing for the config...
<jtv> and now I guess I restart the daemon to make it take effect?
<wgrant> Yes.
<jtv> seems to be running again, and talking to bob
<jtv> 010-03-11 19:42:52+0700 [-] Dispatching: <bob:http://localhost:8221/>
<jtv> 2010-03-11 19:42:52+0700 [-] <bob:http://localhost:8221/> -> ensurepresent((u'f1f10b8402ed686aaf0307676c76f07b45af2a09', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', ''))
<jtv> 2010-03-11 19:42:52+0700 [QueryWithTimeoutProtocol,client] <bob:http://localhost:8221/> response for "ensurepresent": [True, 'Cache']
<jtv> And the slave got another Twisted/XMLRPClib post
<jtv> but now once again everything seems to sit still
<jtv> Bob is listed as doing "Job of type TranslationTemplatesBuildJob with no associated Build."  Which is what I taught it to say.
<jtv> My job has status 3, so that explains why there's no progress; but why no more scanning?
<wgrant> What is status 3?
<jtv> failure iirc
<jtv> yup
<jtv> it had that before I restarted the buildd, so why did it re-dispatch?
<jtv> maybe the best thing is to delete the job, restart the daemons, and retry
<wgrant> It does what it wants.
<wgrant> Probably, yeah.
<jtv> I can confidently confirm that debug output has been enabled...
<jtv> it's trying to add a bunch of distroarchseries every 5 seconds
<jtv> loudly
<jtv> well, here goes nothing!
<jtv> ensurepresent
<jtv> xmlrpclib post on slave
<jtv> silence
<wgrant> jtv: Yeah, the DAS-driven approach is pretty stupid.
<jtv> job is marked as started.
<jtv> damage assessment strike?
<wgrant> DistroArchSeries.
<jtv> oh, that would work too
<wgrant> Fortunately I trialled a branch to remove it, and it worked.
<wgrant> So... what's the final POST?
<jtv> 2010-03-11 12:54:33+0000 [HTTPChannel,104,127.0.0.1] 127.0.0.1 - - [11/Mar/2010:12:54:32 +0000] "POST /rpc HTTP/1.0" 200 212 "-" "Twisted/XMLRPClib"
<jtv> is what the slave says
<wgrant> Well, how useful.
<wgrant> What if you call status() on it manually?
<stub> DistroSeries.getQueueItems tries and fails to return rows in a particular order. This is failing under PG 8.4. Should I just work around the bug by fixing the test? The method is flagged as depricated.
<jtv> wgrant: what, "make harness," retrieve the slave, and invoke it on that?
<wgrant> jtv: import xmlrpclib; xmlrpclib.ServerProxy('http://localhost:8221/rpc').status()
<jtv> ah like that... IDLE
<wgrant> Hm.
<wgrant> And it's marked OK in the UI?
<jtv> but the job still has status 1
<jtv> I think that's the part where Build came in...  :/
<wgrant> jtv: Is the slave code reasonably similar to what's in trunk now?
<jtv> wgrant: no, I merged henninge's branch; see the top of the page under "patches"
<wgrant> jtv: I don't see a henning branch there
<henninge> wgrant: it's henningE ;-)
<henninge> Although my name *is* Henning
<jtv> wgrant: hang on, I'll add that right away
<wgrant> henninge: Right, I just didn't want to ping you unnecessarily.
<henninge> wgrant: thanks ;-)
 * henninge has an alarm on henning, too
<wgrant> :(
 * henninge lunches
<jtv> wgrant: lp:~launchpad/bug-509557-invoke-pottery
<wgrant> jtv: s@~@~henninge/@?
<jtv> wgrant: yes
 * wgrant looks.
<wgrant> jtv: I'd be trying to work out what the XML-RPC call was.
<wgrant> But I should also be sleeping.
<jtv> wgrant: both of those make sense
<jtv> I guess I could just try the state machine.
<jtv> oh! the slave just showed some activity again
<wgrant> So it was just hanging for a while?
<wgrant> Maybe you're not using the state machine properly.
<jtv> this is with henning's branch that just re-uses the debian builder
<wgrant> Hm, OK.
<wgrant> Anyway, ping me when you're giving up for the day if you can't get it working, and I'll have a poke around myself in a few hours.
<jtv> wgrant: I'm being kicked out of here right now, in fact
<jtv> wgrant: but you mentioned sleep.  :)
<wgrant> Yeah, I should. Uni is forcing me to sleep at reasonable hours for now.
<wgrant> Night
<jtv> good night!
<jtv> damn those fascists
<NCommander> When building a test case, how do I properly catch something raised so the test passes?
 * NCommander kills his laptop
<allenap> NCommander: Have a look at TestCase.assertRaises()
<NCommander> allenap: thanks, although I don't seem to be having much luck with it
<allenap> NCommander: How are you invoking it?
<NCommander> allenap: the wrong way it seems :-)
<NCommander>         self.FailUnlessRaises(UploadError, self.findCopyright, self.dsc_file,â©                              self.tmpdir, mock_logger_quiet)
<NCommander> (the raise error comes up in a function that findCopyright calls, but changing the arguments to that also fails)
<allenap> NCommander: Can you paste against LP (I assume this is LP) so I can replicate?
<allenap> s/paste/paste a diff/
<NCommander> allenap: its a set of changes with a database mod, not a small set :-/
<allenap> NCommander: I'm kinda here to help, so it's fine :) If you'd rather plug away at it, I don't mind either :)
<NCommander> allenap: let me post to launchpad
<NCommander> allenap: its based off db-devel versus devel, but given my internet connection ATM, this will take awhile
<NCommander> allenap: lp:~mcasadevall/launchpad/raw_source_changelog
<NCommander> I think I pushed everything properly w.r.t. to database changes
<allenap> NCommander: Okay, I'm getting that now.
<allenap> NCommander: Okay, I have a fix for you I think.
<NCommander> allenap: I assume you know this, but you can run the test code that breaks with ./bin/test -t lp.archiveuploader.tests.test_dscfile
<NCommander> :-)
<allenap> NCommander: The new function findFile() that you've added raises UploadError, whereas errors are yielded elsewhere.
<allenap> NCommander: I can get the test to pass with http://paste.ubuntu.com/393289/, but I think the callers of findFile need to change.
<NCommander> allenap: well, I refactored the code this way so I don't have to duplicate code between findChangelog/findCopyright
<allenap> NCommander: Yes. The way errors are handled - by yielding them - is interesting but a bit weird, so the refactoring in your branch doesn't quite work. I've nearly got a suggestion for you...
<allenap> NCommander: http://paste.ubuntu.com/393291/
<allenap> NCommander: Callers of findFile() must catch UploadError and yield it to their callers.
<NCommander> allenap: that's ugly, but :-/
<allenap> NCommander: Yes it is :)
<NCommander> allenap: I'm kinda concerned that won't make it through a review :-/
<NCommander> allenap: *sigh* NameError: global name 'errors' is not defined
<NCommander> allenap: (on a second note, how can I land a change for one of the sourcedeps; I have a fix that gets LP going again on lucid)
<NCommander> allenap: wait, that a fail to save error :-)
<allenap> NCommander: It's been a while since I did that... First, get an LP dev to add your sourcedep to lp:~launchpad/lp-source-dependencies/trunk, then propose merging an LP branch to update versions.cfg.
<allenap> NCommander: In findFile(), why do you do os.path.exists()? Does globbing not demonstrate that the name exists?
<NCommander> allenap: that may just be a case of code being copied; I refactored old code into findFile, I didn't write it
<allenap> NCommander: Ah, okay, it might be safest to leave it then.
<bigjools> allenap, NCommander: errors are yielded instead of raised so that we can collect as many as possible to return to the uploader
<allenap> bigjools: Yeah, the problem NCommander's facing is that it's difficult for helper functions to raise errors; the caller must add boilerplate to catch the error and yield it. It works, it's just ugly.
<NCommander> bigjools: ah, that explains it
<bigjools> allenap: what's the context here
<bigjools> ?
<NCommander> bigjools: I just refactored findChangelog to call a new findFiles
<NCommander> which is also called by findCopyright :-)
<NCommander> as I didn't want to duplicate code logic
<bigjools> where is findChangelog?
<allenap> NCommander: Here's another approach: http://paste.ubuntu.com/393298/
<NCommander> bigjools: in code not yet merged ;-)
<bigjools> ah :)
<NCommander> allenap: I need to add another sanity check to findFile to prevent a possible DOS, so I like the former approach. Thanks for your help
<allenap> NCommander: Okay, you're welcome :)
<NCommander> bigjools: this is the code that implement raw_source_changelog in soyuz (I'm not sure I grok zope UI yet, but at least the mind numbling part is in)
<bigjools> NCommander: ah interesting
 * bigjools will need to review that ;)
<NCommander> bigjools: branch up on LP already
<NCommander>  lp:~mcasadevall/launchpad/raw_source_changelog
<bigjools> NCommander: so are you fixing changesfile.py?
<NCommander> bigjools: not yet. That requires a proper changelog parser unfortunately
<bigjools> aye
<bigjools> NCommander: so you're just storing the raw changelog?
<NCommander> bigjools: ATM, yes. We ran it by stub this morning
 * bigjools looks at the branch instead of postulating
 * NCommander hears bigjools screaming for some reason
<bigjools> NCommander: be careful with those yield error texts
<bigjools> they are copied verbatim into emails
<bigjools> so "Failed to copy changelog file" isn't that useful
<NCommander> bigjools: that came from Failed to copy copyright file :-)
<bigjools> that sucks too :)
<bigjools> open(fullpath).read().strip() should probably be broken down into different stages and a separate error for each
<NCommander> ugh
 * NCommander is getting the feeling that since I'm now the most recent to touch this file, I get to fix and make it publish
<bigjools> NCommander: if we didn't do drive-by fixes they'd never get done :/
<bigjools> so we like to encourage it :)
<NCommander> bigjools: this doesn't seem like the right way to maintain critical infrastructure ...
<bigjools> why's that?
<persia> Shhhh !
 * NCommander wonders who persia is shhhhing
<bigjools> it seems a great way to me, but perhaps we're thinking about different things
<bigjools> NCommander: anyway your change looks great from what I can see so far
<NCommander> bigjools: I'm probably going to run into issues when it gets to the point of bending the UI to my will
<bigjools> NCommander: yeah probably, but we can help you!  Get this one done first though.
<NCommander> bigjools: I also have no clue how to write the migration tool when we get there
<bigjools> NCommander: yeah that will be tough
<NCommander> bigjools: there's code in dak that can be recycled :-)
<bigjools> we need something to parse the changelogs and poke the database
<bigjools> it will be an interesting script
 * NCommander rewrote a good chunk of dak pre-Canonical days, including support to generate dists/* from the database
<bigjools> I don't think it's too hard though
<bigjools> tough but not impossible :)
<NCommander> bigjools: its more just going one by one through sourcereleasepackage, finding its files, downloading the package from librarian, sucking out a changelog, and updating the database
<NCommander> Its going to be bloody time consuming
<bigjools> well we only need to scan published sources
<NCommander> bigjools: we don't want everything?
<bigjools> NCommander: what's the point?
<bigjools> if there's a good reason it's fine of course
<NCommander> ^- persia - please provide a point if there is one
<bigjools> :)
 * persia reads backscroll
 * bigjools going otp
<persia> Hrm.  I think I have to defer to mvo on that.
<persia> The point would depend on where/how the comparisons are happening for the update-manager display.
<persia> If it's happening between two pulls from the server, we need everything.  If it's happening between the server and the client, we only need urrently published stuff.
<persia> c
<deryck> adeuring, , is bug 505999 really a dupe?  It was marked fixed so I assume you passed the bug number to the commit message?
<mup> Bug #505999: Unsubscribe OOPS <oops> <qa-needstesting> <Launchpad Bugs:Fix Committed by adeuring> <https://launchpad.net/bugs/505999>
<adeuring> deryck: yes, this really looks like a dupe.
<adeuring> the OOPS data is quite similar
<deryck> adeuring, ok, thanks.
<deryck> adeuring, did you QA the doNotSnapshot-related OOPS bugs?
<adeuring> deryck: yes, I tried for example to (un)subscripbe from bug one
<deryck> adeuring, can you update all the related bugs to change to the qa-ok tag then?
<adeuring> deryck: sure
<deryck> thanks!
<deryck> adeuring, sorry, many questions about this. :-)  Bug #518254 was fixed, too?
<adeuring> deryck: probably.... but I need to check this
<deryck> adeuring, ok, just claim it and mark it fixed if so.  if not, ping me to follow up about it.
<adeuring> ok
<leonardr> flacoste, i'm having trouble iterating over a shell glob in the launchpad makefile. can you take a look?
<leonardr> http://pastebin.ubuntu.com/393326/
<leonardr> that same code works in a shell script
<mcasadevall> allenap: I have no luck with exceptions :-/
<allenap> mcasadevall: Hehe :) Can I help?
<mcasadevall>         errors = findFile(self.tmpdir, self.changelog_file, mock_logger_quiet)â©        self.failUnlessRaises(UploadError, list, errors)                      â©
<mcasadevall> allenap: I'm trying to trigger an upload error in findFiles that I added for files being too big
<allenap> mcasadevall: Sorry, I have to go out for a minute; I'll be back in <10m.
<NCommander> np
<leonardr> flacoste: i changed $(WADL_FILE_GLOB) to $(wildcard WADL_FILE_GLOB), but that's expanding to the empty string for some reason
<jelmer> leonardr: you'd want $(wildcard $(WADL_FILE_GLOB)) probably?
<leonardr> jelmer: madness!
<persia> No, just make
<leonardr> jelmer, thanks, that works
<allenap> NCommander: Do you have a diff of your latest changes?
<NCommander> allenap: see PM
<NCommander> bigjools: so, I think I'm at the point where I need to run the local test suite to make sure this all is working sanely
<NCommander> bigjools: how do I do that? :-)
<bigjools> NCommander: best thing is to type:
<bigjools> bin/test -cvv -m soyuz -m archiveuploader -m archivepublisher
<bigjools> and go get coffee for 30m
 * NCommander lets it run
<bigjools> oh -m buildmaster too
<NCommander> Test-module import failures:â©â©Module: lp.soyuz.browser.tests.test_archive_admin_viewâ©
<NCommander> uh oh
<NCommander> What did I break
<bigjools> happens here too
<bigjools> don't worry
<bigjools> jelmer was going to look into that
<jelmer> I'll be mailing the list about it before EOD
<NCommander> bigjools: w.r.t. to findFile(), allenap is concerned thats not going to pass review. If thats the case, I rather rewrite it now. Care to weigh in? (or any other soyuz person)
<bigjools> what is the concern, exactly?
<NCommander> bigjools: it kinda handles errors funny
<NCommander> bigjools: it uses raise to raise an error, then catches them one level up and yeilds them so I don't have a lot of code duplication across changelog and copyright file findings stuff
<bigjools> NCommander: I can't think of a better way of doing it
<allenap> NCommander: I don't see the exception catching code in the branch...?
<allenap> NCommander: If you catch UploadError in findCopyright() and findChangelog() and yield it then it's fine. The test we just worked on would not work though, and would need to be changed; that's why I assumed you weren't doing that.
<NCommander> allenap:
<NCommander> try:â©        copyright_file = findFile(source_dir, 'debian/copyright', logger);â©    except UploadError, error:â©        yield errorâ©
<NCommander> hrm
<NCommander> that came out a bit flatter than I expected
<NCommander> unless I'm misunderstanding the code patch you wrote
<allenap> NCommander: I only got "try:" then "hrm" there; maybe you hit some flood protection.
<NCommander> oh
<NCommander> http://paste.ubuntu.com/393377/
<allenap> NCommander: Cool :)
<leonardr> gary: i'm going to create a .pt file for the apidoc index file. should that go into lib/canonical/launchpad/templates?
<gary_poster> leonardr: mm.  I'd be tempted to put it closer to the machinery that you are building.  It is a not a template that the application should use itself, except in this rare case, yes?
<leonardr> gary: right
<leonardr> should it go in utilities or utilities/templates?
<gary_poster> leonardr: in favor of ./lib/canonical/launchpad/utilities
<leonardr> ok, cool
<leonardr> but, that's not really anywhere near create-lp-wadl.py, which is in /utilities
<gary_poster> leonardr: oh
<gary_poster> leonardr: I thought it would be over there.
<leonardr> gary: i think it's in /utilities because it's only called by the makefile?
<gary_poster> leonardr: right.  since this template needs to run within lp, it should be in the lp tree.  if there's not an obvious place for it, but it in the standard template location you mentioned first
<gary_poster> s/but/put/
<leonardr> ok
<NCommander> zeca failed to start in the test suite :-/
<bigjools> NCommander: yeah it's very problematic sometimes
<NCommander> bigjools: so five tests failed, I'm looking over the failures now. Is the full log automatically saved somewhere?
<bigjools> NCommander: no :/
<NCommander> ugh
 * NCommander accidently cleared his screen
<bigjools> NCommander: it sounds spurious to me
<NCommander> now I need to wait an hour for the other failures to repop up
<bigjools> do you remember the test names?>
 * bigjools keeps meaning to make bin/test and make check write a log
<NCommander> bigjools: nops :-/
<NCommander> I have one as a doc test, although I'm unsure how to rerun it individually
<NCommander> lib/lp/soyuz/tests/../doc/distroseriesqueue-translations.txt
 * kfogel is away: switching consoles for a bit, back soon
<kfogel> wgrant: I was thinking, now that the community-contributions.py script is counting Canonical devs (who aren't on the launchpad team) too, maybe it no longer makes sense to have these hardcode start revs in the file?
<NCommander> Stupid question but are there any know broken tests in soyuz, I have a test thats failing that doesn't seem be related to my changes, and it fails in devel and in my branch
<thumper> morning
<thumper> NCommander: which tests are breaking for you?
<NCommander> thumper: FTPconfig
<NCommander>    test_generateConfig (lp.archivepublisher.tests.test_ftparchive.TestFTPArchive)â©
<thumper> NCommander: I'll check mine on devel
<NCommander> thumper: it fails consistantly on my branch, and on stock devel
<NCommander> thumper: I'd also like to start the review process for my changeset into db-devel
<thumper> NCommander: propose your branch for merging
<thumper> NCommander: that's all you have to do to start the process
<thumper> TestFTPArchive tests pass for me locally with up to date devel
<NCommander> thumper: are you onlucid?
<thumper> NCommander: no, karmic still
 * NCommander is suspecting he's having some unique failures because of it
<NCommander> thumper: thats probably why
<NCommander> thumper: who should I put down for reviewer
<thumper> NCommander: just leave it as launchpad
<thumper> NCommander: it should have that already
<thumper> NCommander: although are you wanting to land on devel or db-devel?
<NCommander> thumper: I'm just reading on the wiki about who to set for reviewer and cover letter
<NCommander> thumper: db-devel
<thumper> NCommander: ok, lp:launchpad is the right branch then
<NCommander> thumper: I'm running make lint as recommended by the cover page instructiosn, and I'm getting a bunch of can't import messages
<NCommander>     29: [F0401] Unable to import 'zope.component'
<thumper> NCommander: have you linked the external sourcecode in your branch?
<NCommander> thumper: yes
<thumper> hmm...
 * thumper confesses to not running make lint in a while
 * NCommander has run make run and make schema without issues
<thumper> NCommander: I'd say ignore those errors/warnings
<wgrant> NCommander: generateConfig is the only test broken on lucid.
<wgrant> kfogel: Hm, possibly.
<NCommander> wgrant: make lint seems broke too :-/
<wgrant> NCommander: What does it do?
<NCommander> wgrant: runs pylint, the submission directions say your supposed to run it and attach the output to any merge cover letter
<wgrant> NCommander: I mean, why is it broken?
<wgrant> It works for me.
<NCommander> wgrant:     17: [F0401] Unable to import 'zope.schema'â©
<NCommander> wgrant: and other such messages
<wgrant> That's normal.
<wgrant> Ignore external missing imports like that; they're in eggs.
<wgrant> Internal imports should work fine, though.
<NCommander> merge proposed
<wgrant> kfogel: I suspect we'll probably need to extend the lists, then, but let's just see what happens.
 * wgrant runs.
<NCommander> thumper: so I take it now I just wait?
<thumper> NCommander: yep or go into #lauchpad-reviews and bug the on call reviewer
<NCommander> thumper: don't DB changes need some special reviewing as well?
<wgrant> Request 'db' reviews from stub and BjornT.
<NCommander> wgrant: before or after the technical review?
<wgrant> NCommander: I don't believe it matters.
<thumper> NCommander: same time
<lifeless> jml: morning
<jml> lifeless, hi
<lifeless> I'd love it if I could understand the progress-to-hard issue. I feel like I just don't get the crux of it
<lifeless> s/to/too/
<jml> I don't want to write a thing that displays a progress bar while dumping subunit output
<kfogel> jml: for changes to the community-contributions script (by wgrant) it's okay if I just review and land them, right?
<jml> kfogel, yes.
<lifeless> jml: ok. So you're saying that it is possible, but you don't want to do it. I was saying that its pointless (you can get the bar from subunit in real time anyway).
<jml> lifeless, it's also pointless
<lifeless> jml: ok, so I think we're agreed at this level. The reason I suggested saying pointless in the docstring, was so that other people examining it won't think you're discarding an important feature.
<jml> lifeless, ok.
<jml>     # subunit output is designed for computers, so displaying a progress bar
<jml>     # isn't helpful.
<jml> also, we're on the wrong channel :)
<lifeless> jml: well, its a launchpad patch.
<lifeless> jml: so I think this is the right channel :)
<jelmer> You should crosspost.
<lifeless> hah. Now perhaps finally thats a reason to install bip
<jml> lifeless, uhh, no, it's a patch to zope.testing.
<lifeless> oh, my bad
<lifeless> I thought it was going to go into the lp tree for some reason.
<lifeless> jml: should I join #zope? I think we're essentially done on the review, right?
<kfogel> wgrant: one teeny little thing left for the community-contributions-fixes branch:
<wgrant> kfogel: Sure.
<kfogel> wgrant: the line "lr.merge_depth < ec.seen_revs[lr.rev.revision_id][0].merge_depth):" and the one below it go longer than 78 columns.  https://dev.launchpad.net/PythonStyleGuide#Whitespace%20and%20Wrapping
<jml> lifeless, I think we're done, yeah
<jml> lifeless, #zope3-dev is the appropriate channel AIUI. Also, you're CCd on the email to the zope mailing list.
<wgrant> kfogel: Really? Argh.
 * wgrant fixes.
<kfogel> wgrant: if you fix up, then we have a clean set of commits by you.
<kfogel> wgrant: (I could fix it, but I hate to taint the history with my shameful name...)
<lifeless> jml: sweet.
<wgrant> Heh.
<wgrant> kfogel: Pushing.
<kfogel> wgrant: thanks.  I will land.
<wgrant> kfogel: Thanks.
<jml> now all I need to do is trick gary_poster or barry or another person w/ commit privs to land it for me :)
<lifeless> jml: I'm a little nervous about the embedded protocol, because I kindof expect the chunked stuff to change - martin is working on a patch to use a ascii coding rather than 8bit
<lifeless> when I say working on, I mean he wants it and I keep saying patches accepted.
<lifeless> I think barry makes a good victim^Wassistance
<wgrant> kfogel: The current version of /Draft was generated from r1 of both branches, with an extended name map.
<kfogel> wgrant: oh, you filled in the name map with those earlier devs?  you rock again
 * kfogel visits
<kfogel> wgrant: looks good to me
<kfogel> wgrant: are all your changes pushed to the lp branch now?
<wgrant> kfogel: No, I haven't committed these yet.
<wgrant> Should I remove the start_revno support?
<kfogel> wgrant: I think it's kind of pointless now.  Do you concur?
<wgrant> Indeed. If we end up needing it, we have a VCS...
 * wgrant removes.
<wgrant> kfogel: r10469 pushing.
<kfogel> wgrant: thank you
<wgrant> np
<lifeless> sidnei: you are da man
<lifeless> bah echannel
<kfogel> wgrant: got your changes, looking now; expect to land today.
<wgrant> kfogel: Great.
<lifeless> oh jml - https://code.edge.launchpad.net/~lifeless/lptools/lp-project/+merge/21124 - may interest you; I'm going to do a further variation for bzr plugins.
 * kfogel is away: machine fan cleanup; back later
<wgrant> Hum.
<wgrant> I have twisted.web.xmlrpc.Proxy.callRemote hanging when one of its arguments is a dict containing a None value.
<shen-long> ok ... wow ...
<shen-long> trying to https://dev.launchpad.net/Running/RemoteAccess
<shen-long> getting a 503 error
<shen-long> followed directions to a T, not using two ip's though, just one public
<shen-long> this is on a fresh ubuntu 9.10
<shen-long> ImportError: No module named _pythonpath in apache error log is the only thing I can see
<shen-long> from scripts/branch-rewrite.py
<shen-long> everything else loops fine
<shen-long> wrong version of python
<shen-long> thanks for talking me through it #launchpad-dev ;p
#launchpad-dev 2010-03-12
<mwhudson> hm
<mwhudson> what to do for the last ~2 hours of work on a friday afternoon
<lifeless> mwhudson: pawn?
<lifeless> mwhudson: land that bzr-git change I did for you
<mwhudson> oh right yeah
<lifeless> I need a teddy bear
<lifeless> for a bzr patch.
<mwhudson> lifeless: i can try
<lifeless> I want to make commitfromnews also set --fixed
<lifeless> this means a hook to run after the commit message is set by the user
<lifeless> rather than at generation time
<mwhudson> that makes sense i guess
<mwhudson> one could imagine something similar involving debian/changelog
<lifeless> right
<lifeless> in fact commitfromnews could look at debian/changelog if wanted
<lifeless> or bzr-builddeb can supply the changelog entries as a template
<lifeless> $whatever
<lifeless> so the questions are
<mwhudson> i think it already does
<lifeless> should the message be editable at this point (by the hook)
<lifeless> e.g.
<lifeless> it could look for a
<lifeless> ----bugs----
<lifeless> lp:1234
<lifeless> deb:456
<lifeless> --endbugs--
<lifeless> region, strip that from the commit and turn it into properties
<lifeless> commitfromnews and others could put this in the template
<lifeless> and a user can then correct it if desired
<lifeless> or should the message be immutable
<mwhudson> you could put that --bugs-- stuff after the "-- the part of the message after this marker will be ignored --" i gues
<lifeless> mwhudson: sadly no
<mwhudson> although that might lead to people not reading it
<lifeless> mwhudson: because it gets stripped off early
<mwhudson> oh right, i guess the hook probably wouldn't see it then?
<mwhudson> heh
<lifeless> 'tada' :P
<lifeless> I think the constraints are:
<lifeless>  - humans can correct the heuristics
<lifeless>  - not -too- ugly if the plugin to do bug handling is missing/disabled for some reason
<lifeless> any feedback
<mwhudson> lifeless: another constraint might be: allowing a way for a gui to be more tasteful in handling this
<lifeless> so the msg template does have a commit object passed in
<lifeless> I was thinking to have all the plugins that find bugs tell the commit object
<lifeless> and the callback in builtins.py (this is really ugly)
<lifeless> add the visibilty stuff
<lifeless> so a gui could do better
<lifeless> there is so much code in builtins.py that doesn't belong there.
<mwhudson> yes
<mwhudson> lifeless: i think what you first suggested sounds good enough to be worth trying to use for a while
<lifeless> argh, lenovo please please please provide more visibilty
<mwhudson> lifeless: the part that's unsettling is making the commit-message-as-edited into some semi structured thing
 * lifeless tries a summary:
<lifeless> a) CLI case
<lifeless> 1) message templates provide a template and also call into the commit to say 'possibly fixes X'
<lifeless> 2) CLI UI layer says 'template += list of bugs fixed'
<lifeless> 3) CLI UI layer strips out list of bugs fixed and adds them as really fixed to the commit
<lifeless> done
<lifeless> b) GUI cas
<lifeless> 1) same
<lifeless> 2) GUI shows additional widget; easy.
<lifeless> CLI with commit -m - what happens here?
<lifeless> template doesn't run, no prompt, fail.
<lifeless> take 3)
<lifeless> rather than message templates use a Commit hook to scan for bug status, which can look at:
<lifeless>  - the diff
<lifeless>  - commit message
<lifeless>  - $whatever
<lifeless> so the CLI case calls this hook after getting the message template from the user
<lifeless> adds it to the template
<lifeless> then parses the result back out again
<lifeless> cli with -m, shows the commit editor when the detected candidate bug list != supplied bug list from the command line
<lifeless> and doesn't show the commit message itself
<lifeless> so 'commit message' here means either 'template or final commit message depending on whether the user has supplied it or not'
<lifeless> mwhudson: so sounds plausible?
 * mwhudson rubs eyes, reads again
<mwhudson> lifeless: yeah, i think so
<lifeless> this seems too complex
<lifeless> take 4
<lifeless> what I specifically want is 'bzr commit' -> editor -> adds --fixes for me
<lifeless> with, also specific, the ability to correct mistakes
<lifeless> the easiest thing is to not do this when -m is supplied
<mwhudson> i think not worrying over much about what happens whem -m is supplied makes sense
<lifeless> that binds it to the UI
<lifeless> which as a heuristic, is appropriate.
<thumper> lifeless: nudge re: code snippet
<lifeless> oh right
<lifeless> uhm committing patches right
 * mwhudson leaves the cafe
<mwhudson> probably back on line for a bit later
<lifeless> http://pastebin.org/110384
<lifeless> thumper: &
<thumper> lifeless: ta
<lifeless> my fault if it works, yours if it fails
<thumper> lifeless: does bzrlib have any context managers yet for handling locks?
<lifeless> there was a branch
<lifeless> dunno if its landed
<lifeless> we write to 2.4 still though
<lifeless> so we can't use or even really test them.
<thumper> lifeless: I was considering targetting 2.6+ for my work so I can use them :)
<lifeless> thumper: note that the approach I use is a bit fugly
<lifeless> you could check Merge3Merger to see how the regular merge code detects conflicts
<lifeless> but the class I use is the one you should use
<thumper> lifeless: thanks
 * thumper EODs
<wgrant> jtv: Morning.
<jtv> wgrant: afternoon!
<wgrant> That too.
<wgrant> So, I had a look at the hanging problem this morning.
<wgrant> twisted.web.xmlrpc.Proxy.callRemote is hanging when you attempt to run build()
<wgrant> There are two bugs here: 1) It hangs when you give it the extra_args dict with None as a value. I don't know why this is. 2) You are using IBranch.url, which is None, when you in fact mean IBranch.composePublicURL().
<wgrant> Fixing the latter makes it dispatch.
<wgrant> But it gets aborted on the next scan cycle, since TTBB doesn't have a build, and doesn't override verifySlaveBuildID.
<wgrant> Once you override that properly, it gets dispatched and finishes fine, although the slave seems to just return filenames rather than a tarball right now.
<jtv> wgrant: my apologies for my silence; I was looking at a different workspace
<jtv> Where are we using IBranch.url?
<jtv> I specifically wrote IBranch.composePublicURL for this... it'd be an embarrassing oversight if I then neglected to use it in creating the job parameters.
<wgrant> jtv: TTBJ.dispatchBuildToSlave
<wgrant> Er, TTBB, I guess.
<jtv> OMG
<jtv> and through sheer coincidence it comes out as a http URL that does work for the slave?  weird
<wgrant> .url comes out as None, which causes the hang.
<wgrant> .composePublicURL comes out an an HTTP URL which works perfectlyt.
<jtv> But I was seeing the proper URL in my BranchJob!?
<jtv> Yes, that's why I wrote the damn thing.  This is galling.  My compliments on finding it!
<wgrant> The URL isn't contained in the brnachjob...
<wgrant> It just references a branch ID.
<jtv> the json_data is {'branch_url': 'http:....'}
<wgrant> jtv: I appear to have two branch jobs for each scan. One has a URL, and one has from_revision_id/force_translations_upload.
<jtv> ah
<wgrant> Is the one with the URL the translations job?
<wgrant> I haven't actually checked the type enum...
<jtv> they're both translations jobs, of different kinds :-)
<jtv> IIRC RosettaUploadJob has a type 3 or 4.  Mine has type 6.
<wgrant> Ahh.
<wgrant> So the BranchJob row has a URL, but dispatchBuildToSlave then inaccurately recalculates it?
<jtv> I guess!
<jtv> I think dispatchBuildToSlave should just use BranchJob.json_data
<jtv> Well, I think it's called metadata or something
<wgrant> I don't see why you precalculate the URL.
<jtv> One very stupid practical reason is that it makes it a lot easier to see what's going on in psql!
<wgrant> shh.
<jtv> Well, when I'm not second-guessing it and replacing perfectly good json data with the wrong dict.
<wgrant> Hmm.
<wgrant> Henning's latest branch (lp:~henninge/launchpad/bug-507680-upload-templates) seems like it's attempting to do the upload from the slave.
<wgrant> I don't think that's going to work.
<jtv> No, that wouldn't
<jtv> You're talking about the upload to the translations import queue?
<wgrant> That's what the comments suggest.
 * jtv looks
<jtv> wgrant: I'm looking at the version I merged into my local branch, but don't see it there
<jtv> wgrant: oh, you're looking at the branch for bug 507680?
<mup> Bug #507680: Upload buildfarm-generated templates <wellington> <Launchpad Translations:In Progress by henninge> <https://launchpad.net/bugs/507680>
<jtv> I'll add a note to the bug
<wgrant> jtv: Yeah, that one.
<jtv> Wow, thanks for spotting that.  It's obscene how much we've come to rely on you.
<wgrant> Heh.
<jtv> hi henninge!
<henninge> Hi jtv! ;)
<henninge> jtv: thanks for the comment.
<jtv> henninge: ah, you saw that :)
<henninge> I had already gotten the feeling that I had something wrong in my thinking therer.
<wgrant> It looks like you're very close.
<jtv> another thing: don't call tarfiles "zipfile," or confusion and the possible collapse of civilization will ensue.
<wgrant> Dispatching all works, so you just have to get the tarball out of the slave now.
<jtv> (there's a builtin module called zipfile, and we use it elsewhere in our code)
<jtv> Okay, I may be exaggerating a tiny bit
<henninge> jtv: talking to me?
<henninge> ;)
<jtv> henninge: yes.  Okay, I may be understating the "tiny bit" of my exaggeration.
<henninge> yes, right.
<henninge> jtv: how do I get the tarball from the slave to the master?
<jtv> grawr, I hate how my mouse pointer moves because my hands are near the touchpad while I'm typing
<wgrant> jtv: Karmic and above should disable that by default.
 * henninge was going to look at other builder's code but might as well ask ... ;)
<wgrant> Check System->Preferences->Mouse->Touchpad, the "Disable touchpad while typing" checkbox.
<jtv> wgrant: this is karmic, but I think I disabled it by hand
<jtv> yes, it'd be nice if that actually worked, wouldn't it  :)
<wgrant> henninge: I see you have code to produce a tarball. You'll need to add that into the returned filemap somehow.
<jtv> henninge: see my comment... that part is already done
<wgrant> henninge: The other slaves do that by looking at a changes file.
<jtv> ah, right, you do need to put it in filemaps
<wgrant> Once it's in there, it's trivial to get to it from the master code.
<jtv> henninge: IIRC I hard-coded a path for the tarball
<jtv> ah yes, "translation-templates.tar.gz"
<jtv> feel free to change that; it's just a placeholder
<wgrant> See gatherResults in the Debian and SourcePackageRecipeBuildManagers.
<wgrant> I suspect that you'll need to copy your file into the filecache, and then call addWaitingFile.
<henninge> wgrant: yes, saw that codenow . I'll look deeper into it to find out exactly.
<henninge> jtv: I saved the world! ;)
<henninge> i.e. renamed zipfile to tarball
<jtv> millions sigh in relief
<jtv> \o/  \o/  \o/
<al-maisan> henninge: nice way to end the week :)
<jtv> lol
<al-maisan> saving the world
<wgrant> Heh.
<wgrant> Morning al-maisan.
<henninge> al-maisan: yes, feels so good
<jtv> okay, who's going to put al-maisan on the quotes page?
<henninge> Moin al-maisan!
<al-maisan> Good morning henninge, jtv and wgrant :)
<jtv> :)
<henninge> Oh yes, good afternoon wgrant!
<jtv> wgrant: in the slave, iterate_CLEANUP calls buildComplete regardless of success, but it turns out buildComplete expects to be coming from a BUILDING state.
<wgrant> jtv: Why would it be building but not BUILDING?
<wgrant> CLEANUP is still called during BUILDING.
<jtv> wgrant: this may be a cleanup after failure
<wgrant> At which point alreadyfailed should be True, but the status should still be BUILDING, IIRC.
<jtv> (as in my case it probably is)
<jtv> unless the failure happened in a different state...
<wgrant> In what other state would it fail?
<wgrant> You shouldn't be setting builderstatus yourself at all, I don't think.
<jtv> pretty much anything at this stage
 * henninge gets breakfast
<wgrant> Oh, are you talking about the case where iterate_CLEANUP throws it to BUILDERFAIL, then calls buildComplete?
<wgrant> Wait what.
 * jtv looks again
<wgrant> Oh, right.
<wgrant> Are you aware of the difference between builderstatus and buildstatus?
<wgrant> It's very confusing if you forget that they are separate.
<wgrant> iterate_CLEANUP will set the buildstatus to OK or BUILDERFAIL.
<jtv> wait, you're talking about build status the database field?  I don't see a connection
<wgrant> But the builderstatus should remain BUILDING until buildComplete sets it to WAITING.
<wgrant> No, the slave property.
<jtv> now _that_ is confusing :)
<wgrant> Yes!
<wgrant> So the slave returns something like (BuilderStatus.WAITING, BuildStatus.OK)
<wgrant> Or (BuilderStatus.WAITING, BuildStatus.BUILDERFAIL)
<jtv> ok
<wgrant> Is the initial non-BUILDING thing you brought up actually appearing in practice?
<jtv> So BuilderStatus.BUILDING should start as soon as the builder accepts a job?
<jtv> wgrant: yes, that's why I'm bringing it up.
<wgrant> I think so, yes.
<wgrant> IDLE -> BUILDING -> WAITING -> IDLE -> ... is the normal path.
<wgrant> Any of the other states are probably buggy and won't work properly.
<wgrant> (although the translations job handles ABORTING much much better than binary builds)
<wgrant> jtv: So, got any more info on the odd failure?
<jtv> wgrant: yup!  tiny innocuous little line: 2010-03-12 13:38:33+0700 [-] Builder 'bob' rescued from 'bf-test-14': 'Malformed build ID'
<jtv> And the FSM goes through INIT-UNPACK-CLEANUP
<wgrant> jtv: Ahaha, so it was ABORTING after all.
<jtv> And to some extent, recovering which is also cool.
<wgrant> Yeah, that doesn't work for binary builds.
<wgrant> But yours isn't horrible so it might work better.
<jtv> Oh, I missed today's Hacker Minute
<wgrant> Hm?
<jtv> 13:37
<wgrant> Ah.
<jtv> Then again, last night I finally attended the fortnightly hacker get-together again.  Finally saw James Clark again, which was so much fun after all those years.
<jtv> And of course one Catalan hacker who somehow knows I used to live in his town, and we find a common acquaintance.
<jtv> This world is insane.  Run by someone with a sense of humour.
<wgrant> Haha.
<jtv> Still, that flight to Wellington takes the cake.
<wgrant> I was thinking of that, yes.
<wgrant> Quite incredible.
<jtv> Funny... it looks as if I'm expected to generate an id consisting of "buildid-buildqueueid" where "buildid" is ignored.
<wgrant> Not ignored.
<wgrant> It's compared down the bottom.
<wgrant> But I wrote that method, and I expect you to override it.
<wgrant> Since you don't have a build.
<wgrant> (it used to be prettier, but then we got more job types)
<jtv> override verifySlaveBuildID?  Right ho.
<wgrant> Right.
<wgrant> I just said 'pass' in my version, but you can probably do slightly better than that.
<jtv> Makes it a bit dual-purpose though, so maybe I should break that method in twainâ"verify that the id makes sense for this job type" and "verify that this BuildQueue object exists."
<jtv> The latter is generic.
<wgrant> It's entirely up to the BuildFarmJobBehaviour to decide the slave build ID string.
<wgrant> So it might not contain the buildqueue ID, but it probably does.
<jtv> True, but a helper seems useful (also means you don't need to look up the utility)
<wgrant> Indeed.
<wgrant> Also, I see some workarounds in the slave for parts of the interface that don't quite make sense for you.
<wgrant> Remember that you can refactor it too...
<jtv> otp with hardcore Windows dev
<jtv> "you didn't hear this from me, but I'm thinking of switching to Ubuntu Server"
<wgrant> Haha.
<wgrant> Good to know.
<jtv> "I didn't realize that Ubuntu had a Server edition"
<jtv> Ouch, he really wants to do it all by mouse.
<noodles775> Good day :)
<jtv> hi noodles775
<noodles775> Hi jtv
<wgrant> Morning noodles775.
<noodles775> Hi William.
<noodles775> jtv: thanks for the explanation about your objections to the with statement.
<jtv> noodles775: I didn't _want_ to write a novel, but given that it wasn't yet completely clear...  :-)
<noodles775> jtv: no, it was exactly what I was interested to know.
<jtv> "This Ubuntu website is amazing!  Very easy, it's listing all the advantages and features and technologies... fucking wow!"
<jtv> He's considering jumping ship from Windows 7, mainly because of licensing cost
<wgrant> Wow.
<jtv> noodles775: you came in in the middle of this...  I'm otp with a hardcore windows dev
<noodles775> Ah, I was wondering what the context was.
<noodles775> jtv: and regarding the benefit of switching back afterwards (back on the context manager), I was thinking whether it could be something re-usable for other tests, where in the setup code you need to switch db users to set something up and then switch back.
<noodles775> wgrant: if you've time, can you add your thoughts/disagreements/clarifications to bug 536700 (I added a png of the current db tables with some notes).
<mup> Bug #536700: Present other build types in a PPA context <wellington> <Soyuz:Triaged> <https://launchpad.net/bugs/536700>
<noodles775> And feel free to simply edit the description where necessary of course.
<jtv> noodles775: might be, yes, but a proper switch back may involve a commit; plus switching back and forth in a test is probably an indicator that you're doing too much in one test.  So I don't particularly feel like doing work to encourage it.  :)
<wgrant> noodles775: Is the source for that image around somewhere.
<wgrant> ?
<wgrant> I want to shuffle it around a bit.
<noodles775> wgrant: one tick, I'll upload the .dot file that I used.
<noodles775> The only way I could produce it was to use postgres_autodoc to generate the source for *all* tables, and then edit the file removing all the other tables/relns...
<wgrant> Ah :(
<noodles775> wgrant: http://people.canonical.com/~michaeln/tmp/launchpad_dev.dot
<noodles775> I then ran that through `dot -Tpng -o build-related.png launchpad_dev.dot`
<wgrant> I wonder why it is laying it out so badly.
<noodles775> Yeah, I used dia once before for the recipe tables, which allows you to lay it out of course, but it was a pain as it crashed every few minutes, hence trying dot this time.
<noodles775> (although it might be better on Lucid, that was on Karmic).
<wgrant> Odd. Dia has been just about unchanged for many years, and is reasonably reliable for me.
 * wgrant tries.
<wgrant> (and dot normally does a better job than it has this time)
<noodles775> wgrant: I think it was more to do with the size of the file... it may have been that once I manually deleted all the other tables it was stable, I can't remember.
<wgrant> Ah.
<wgrant> Yeah, I guess it could be pretty huge otherwise.
<noodles775> I generated a png of the complete schema just for fun... it was 22mb, and looked very scary ;)
<wgrant> Heh.
<wgrant> noodles775: Dia gets very very slow with the entire scheme, but it hasn't crashed yet.
<noodles775> Great.
<adeuring> good morning
<stub> wgrant: Unfortunately, dot is pretty bad at laying out stuff like ORM diagrams. Its designed for other stuff.
<wgrant> stub: So it seems :(
<wgrant> kfogel: Oops. Pushing removal now.
<kfogel> wgrant: ah, great.  I'm still up; I'll pqm-submit it afterwards.
<wgrant> kfogel: Thanks.
<kfogel> wgrant: you've tested your latest at /Draft, right?
<wgrant> kfogel: Except for the comment removal, I believe so. let me try again, just in case.
<kfogel> wgrant: I believe this fixes bug 436957; do you concur?
<mup> Bug #436957: Contributions script gets confused with db-stable=>devel merge <Launchpad Foundations:Triaged by kfogel> <https://launchpad.net/bugs/436957>
<wgrant> kfogel: Ah, so there is a bug!
<wgrant> And I even filed it.
<wgrant> yes, it does fix it.
<kfogel> wgrant: ok, listing in PQM msg
<kfogel> wgrant: can you link your branch to the bug please/
<kfogel> ?
<wgrant> kfogel: Done.
<kfogel> wgrant: arrrgh.  how to make pqm submit your branch instead of ~kfogel/launchpad/community-contributions-fixes (which doesn't exist).
<kfogel> sigh
 * kfogel growls at bzr pqm-submit
<wgrant> It does exist, though.
<wgrant> Or at least yours popped up too when i searched for my branch.
<kfogel> wgrant: well, I could push to my own area and then land that, but it seems so silly.
<kfogel> I want to land *yours*.
<kfogel> wgrant: hunh.  THe "--public-location" option seems to have done it.  I wouldn't have expected that to work.
<kfogel> bzr pqm-submit -m "[r=kfogel][ui=none][bug=436957] In the community-contributions.py script, snarf contributors from both devel and db-devel branches, and fix a sorting bug." --public-location=lp:~wgrant/launchpad/community-contributions-fixes/
<kfogel> wgrant: okay, it's in the Now Playing queue.  I'm off to bed.  Thank you SO much for fixing these long-standing problems!
<wgrant> Thanks for the review and landing!
<wgrant> Night.
<jtv> wgrant, noodles775: I'd like to make some Build and BuildQueue objects for my tests... do you happen to have any handy makeBuild[Queue] functions lying around that I could move into the factory?
<wgrant> jtv: See SoyuzTestPublisher.
<noodles775> jtv: yeah, the SoyuzTestPublisher is what we usually use.
<wgrant> Also see noodles' blog post.
<noodles775> it's in lib/lp/soyuz/tests/test_publishing
<jtv> cool, thanks
<wgrant> stub: I wondered about indices. I guess we'll see.
<wgrant> Thanks.
<mrevell_> Morning
<deryck> Morning, all.
<jml> morning
<jml> intellectronica, it turns out that renaming attributes is difficult.
<jml> intellectronica, thank heavens for our test suite
<intellectronica> jml: right. feel free to rename them only in the api if it turns out too difficult to change them internally
<jml> intellectronica, it's been easy enough, I just missed a couple of places.
<intellectronica> cool
<rockstar> Has anyone done anything about the buildbot failure?
<gary_poster> rockstar: apparently not.  I'll own the db-devel failure, since it is the librarian.  Hopefully someone else can step up for the devel failure in malone.
<rockstar> gary_poster, I'm working on the devel one right now.
<gary_poster> rockstar: awesome thanks
<rockstar> henninge, the buildbot failure email came to you 2 hours ago. Why did you not at least reply to the list with the causation?
<deryck> adeuring, intellectronica, gmb, allenap -- is someone looking into our failure gary_poster mentions above? ^^
<deryck> oh, I see rockstar says he is.
<deryck> rockstar, thanks!
<henninge> rockstar: sorry, I was going to but got distracted ...
 * henninge hides in shame
<rockstar> deryck, can I get you to review the fix?
<deryck> rockstar, sure.
<gary_poster> flacoste: I know zero about the librarian, and we have a seemingly spurious failure on db-devel because of it (https://lpbuildbot.canonical.com/builders/db_lp/builds/591/steps/shell_7/logs/summary).  Who knows the librarian right now?  You and stub?  ...I think I'm going to force db-devel right now but should not let this lie.
<flacoste> gary_poster: do we have /var/tmp/librarian.log
<gary_poster> flacoste: no
<flacoste> gary_poster: there isn't much to do beside restarting, without that file we have no idea what went wrong
<flacoste> gary_poster: is that a first?
<gary_poster> flacoste: to my knowledge, yes
<gary_poster> something to do in the abstract: contemplate buildbot slaves sending files if there are failues.  But that's wasted work, since we are moving the slaves to the DC
<adeuring> deryck: no, I haven't looked yet
<deryck> adeuring, no need now, rockstar is on it.  sorry for the mass ping.
<rockstar> deryck, https://code.edge.launchpad.net/~rockstar/launchpad/testfix/+merge/21243
 * rockstar waits for diff
<rockstar> deryck, diff is there.
<deryck> rockstar, r=me.  Thanks!
 * deryck curses at page tests
<jml> good news!
<adiroiban> x
<jml> subunit output is now in zope.testing on trunk
<jml> I can delete my horrible hack once they release.
<rockstar> gary_poster, so, the devel testfix is in and should make buildbot green there again.
<gary_poster> rockstar: Awesome, thanks.  I forced db-devel, so we should now be out of testfix
<intellectronica> mrevell: we shouldn't use bug.reporter@gmail.com in our text if we don't own this address. we should use example.com. we can change that in another branch, though.
<mrevell> intellectronica, Good point! I'll change it in this branch; no harm doing that.
<intellectronica> mrevell: lovely, thanks
<gmb> allenap: Does this look sane? I went with BugWatchActivity rather than *History; it seemed to lead to nicer column names. http://pastebin.ubuntu.com/394114/
<allenap> gmb: So, success = (error_type IS NULL)? Perhaps a column for the OOPS number?
<allenap> gmb: Might need an index on bug_watch, but I guess that's up to stub.
<gmb> allenap: Yes, that's how I see us working out success. OOPS number is a good idea. Isn't bug_watch indexed by default, being an FK?
<allenap> gmb: Maybe it is indexed.
<gmb> allenap: Ah, no, I'm misreading the output of \d.
<gmb> allenap: It probably should be indexed; I'll add one.
<Breaking_Pitt> I have tried to putt the launchpad opent to all the computer in my network but I have get the following message
<Breaking_Pitt>  The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
<Breaking_Pitt> following this info https://dev.launchpad.net/Running/RemoteAccess
<Breaking_Pitt> some advice about what can cause this?
<bjf> if launchpad is telling me "This bug report was marked for expiration 164 days ago." then why hasn't it "expired", what does "expire" mean?
<jml> bjf, "expire" means "automatically set to Invalid", I think.
<bjf> jml, ok, but nothing seems to be "automatically" setting the bugs to Invalid
<deryck> bjf, jml -- yes, expired means set to invalid.  This is currently disabled, so the messages are incorrect.
<deryck> we're doing work soon (likely this month) to renable it with an EXPIRED status.
<bjf> if I were so inclined, can I hack on launchpad on lucid (the web page says Hardy, Jaunty and Karmic)?
<maxb> bjf: it's less well supported
<maxb> which is to say, yes, but expect things to break
<maxb> much like lucid itself, I guess :-)
<maxb> https://dev.launchpad.net/Running/Lucid
<jml> also, the Canonical Launchpad folk are switching to lucid on the beta, so it'll probably get better quickly
<deryck> Later on, everyone.... have nice weekend.
<sinzui> gary_poster: ping
<gary_poster> sinzui: pong.
<sinzui> gary_poster: bug 538207 may require a bandaid or a team of people to fix. Or we can rollback one of salgado's landings
<mup> Bug #538207: Opps calling view/isRedirectInhibited from non-launchpadview <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/538207>
<gary_poster> sinzui: I was just looking at that one actually.
<gary_poster> sinzui: so lp.registry.browser.teammembership.TeamMembershipEditView does not inherit from LaunchpadView?  I was just about to go there in my investigation of what was going on
<sinzui> I favor fixing the views (I am certain I can fix the registry ones), but I think we have views that are defined in ZCML only, so they will never have the needed attribute
<gary_poster> sinzui: you mean without a view class.  Right.
<sinzui> gary_poster: yes, that is what I mean
<gary_poster> sinzui: what do you think of this instead:
<sinzui>  ââgary_poster: we would also have to fix this bug too bug 433074
<mup> Bug #433074: Remove launchpad-addform.pt and launchpad-editform.pt <tech-debt> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/433074>
<gary_poster> move isRedirectInhibited to the request
<gary_poster> change that template to say request/isRedirectInhibited
<gary_poster> and move on
<sinzui> gary_poster: ah...You are thinking much more clearly than myself
<sinzui> We do get to fix a lot of tech-debt if we refuse to inline the python
<gary_poster> sinzui: ok, seems like you like that idea.  I don't know how critical this is.  What do you think?  Is this a drop everything and do it now, or do it early next week?  Without your adjustment, I will schedule for Monday
<sinzui> This is not critical yet, collin disable the redirect to complete his task
<gary_poster> true
<gary_poster> ok thank you sinzui.  I'll follow up on bug
<sinzui> We should fix it next week. I was concern that if we did want everything to descend from LPVIew, that we need to get several people involved
<gary_poster> understood, cool
<wgrant> gary_poster: Bug #529348 hasn't been fixed yet, has it? (re. #launchpad)
<gary_poster> wgrant: it is waiting for an RC.  I
<gary_poster> will land the branch on devel when it goes into production
<gary_poster> (so as not to call attention to it)
<wgrant> Hm, OK, so there is something else going on.
<wgrant> Yep.
<gary_poster> hm, not on #launchpad, looking
<wgrant> gary_poster: c_korn is getting an Unexpected Form Data error on what appears to be all LP forms.
<gary_poster> that's more than a little bit weird, yes, wgrant
<wgrant> I don't think UFDs return an OOPS number, though they do give a traceback if you're in ~launchpad. :(
<gary_poster> ah, wgrant. :-(
<gary_poster> sinzui: I unintentionally stomped on you for https://bugs.edge.launchpad.net/launchpad-foundations/+bug/538236
<mup> Bug #538236: UnexpectedFormData submitting answers comment <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/538236>
<sinzui> stomp away
<gary_poster> sinzui, I was going to change this bug to say that we don't give adequate feedback for UnexpectedFormErrors.  I agree that user was missing REFERER.  However, there's no way for user or us to know that.  You agree?
<sinzui> yes. I agree
<gary_poster> cool thanks
<wgrant> gary_poster: Simple solution being the adapter?
<peitschie> hi everyone :)
<gary_poster> wgrant: yes, the adapter to the AssertionError, as opposed to the adapter to something like a PublicAssertionError
<wgrant> Hm, OK.
<peitschie> hey... i was wondering if anyone knows who is best to talk to about getting a patch into the python launchpad integration source?
<wgrant> peitschie: python-launchpad-integration, or python-launchpadlib?
<peitschie> wgrant: the first :)
<wgrant> peitschie: I believe that's in Ubuntu's domain, not Launchpad's.
<peitschie> wgrant: ahh.  you wouldn't happen to have the room for them handy by any chance would you?  Thanks by the way :)
<wgrant> peitschie: #ubuntu-bugs might be able to help.
#launchpad-dev 2010-03-13
<peitschie> wgrant: many thanks again :)
<wgrant> np
<wgrant> Hmmm.
<wgrant> I just got a really foul OOPS that I both shouldn't have got, and is broken.
<wgrant> There is a self-closing <script> in the header, so the page is just about empty.
<wgrant> (edge)
<jtv> hi wgrant, thanks for saving us from that dpkg path traversal vulnerability :)
<wgrant> jtv: Morning.
<jtv> wgrant: I've been up all night playing with representing different kinds of buildfarmjobs in the UI
<jtv> My current experiment goes something like this: delegate representing the current job to the buildqueue, which in turn delegates to specific_job.
<jtv> I introduce two new interfaces: IBuildFarmBuildJob ("an IBuildFarmJob that also refers to a Build") and IBuildFarmBranchJob ("an IBuildFarmJob that's also an IBranchJob").
<jtv> These have their own templates to render them on the builder pages.
<jtv> There's a fallback for "just an IBuildFarmJob without a Branch or a Build," but currently I don't think anything would use that.
<jtv> So everything is based on a hierarchy of interfaces, with TAL at various levels of refinement.
<jtv> Now, the builder-index TAL only has the "there is a current job" case and the "there is no current job" case.
<jtv> If there is a job, the rendering mostly happens in a BuildQueue view (the permissions check moves in there as well so the TAL only needs to deal with a single condition).
<jtv> The TAL for the BuildQueue then incorporates the polymorphic rendering of the BranchJob.
<jtv> Sorry, the BuildFarmJob
<wgrant> Right, this is sort of what I was thinking.
<wgrant> But then people started throwing around ideas about redesigning the model.
<jtv> I think know how you feel: "yes that has to happen, but given the choice between discussing that some more and taking the first useful step..."  :)
<wgrant> Heh.
<wgrant> Yes.
<jtv> Funny thing is, from where I'm standing, the whole where-do-we-keep-history thing seems to be entirely independent of this.
<lifeless> bzr
<lifeless> kthankssolveitonce
<jtv> lifeless: ??
<lifeless> jtv: 'where to store history' :)
<jtv> ah, that part I was emphatically not planning to solve anytime soon  :)
<wgrant> jml: Is having an open team as launchpad-web's owner a really great idea? It lets everyone in the world change the security contact, for example.
<jtv> wgrant: I've been staring myself blind at how the buildd master keeps track of what build belongs to a buildfarmjob... does it simply keep the buildfarmjobbehavior in memory and just forget about the whole thing if it ever gets restarted before the slave is finished?
<wgrant> jtv: No, it can be restarted fine.
<wgrant> jtv: Isn't a Build a BuildFarmJob?
<wgrant> Builder.current_build_behavior looks at Builder.current_job and magically creates the behaviour from that.
<jtv> Build is not a BuildFarmJob, but in some of the base buildfarm code it's tacitly understood that a buildfarmjob _has_ a build.
<wgrant> Oh, is an IBuildFarmJob that sits between Job and (SourcePackageRecipe)Build?
 * wgrant could just check the source.
<jtv> Yes, pretty much...  generally the Job is the magic magnet that brings all the various parts of a slave job together.
<jtv> Note that a SourcePackageRecipeBuild is not a Build.
<wgrant> Right, it's a BuildBase.
<wgrant> jtv: Don't the relevant implementations of IBuildFarmJob have a 'build' attribute?
<jtv> True, they do.
 * jtv needs pen and paper
<wgrant> I think we just needed a bigger whiteboard at the sprint.
<jtv> With this being BuildMaster's dirty little secret, it gets hard to keep track
 * wgrant blames that for all design flaws.
<jtv> Nice one
<wgrant> It's nothing to do with buildd-manager.
<jtv> Isn't that the process that drives this though?
<wgrant> It is. But all the stuff that deals with jobs like that is all deep in the model.
<wgrant> buildd-manager knows only about BuildQueues and Builders.
<jtv> Right.  So what I'm trying to figure out is if we can do away with the Build.id in the buildfarmjob names.
<jtv> And instead, have a single implementation for generating/verifying all those names, which contains just (1) the BuildQueue.id and (2) a cookie.
<wgrant> Names meaning the slave build ID?
<jtv> Yes
<jtv> (confusingly referred to as "name" in IBuildFarmJob but "slave build ID" in IBuildFarmJobBehavior)
<wgrant> I wasn't aware of the 'name' name.
 * wgrant checks.
<jtv> It's subtle: what I mean is it's the only sensible implementation for getName
<jtv> ...though it's actually up to the behavior class to compose the slave build id in that way or some other way.
 * wgrant isn't sure why it's getName() rather than just name.
<wgrant> That makes me a little suspicious.
<jtv> Yeah.  Right now, the conventional structure for the id is <Build.id>-<BuildQueue.id>
<wgrant> (I added getTitle following that convention; I probably shouldn't have)
<wgrant> Right.
<jtv> But recipe builds use SourcePackageRecipeBuild.id instead of Build.id
<wgrant> The content of it doesn't matter. As long as it's reasonably unique.
<jtv> It's not even used to find back the BuildQueue?
<wgrant> We use a method on BuildQueueSet for that.
<wgrant> It's deep in SOMEWHERE.
<jtv> Oh, that's getByJob's job isn't it
<wgrant> I have branches that clean buildd-manager up and make it less completely ununderstandable.
<wgrant> BuilddMaster.scanActiveBuilders is all that should need to retrieve a BuildQueue within the manager.
<jtv> I can only cheer for you.  Given all this, let's stop pretending there's any structure to the slave build id at all, beyond what's needed to guarantee uniqueness.
<wgrant> Except for the thing that finds a new job to dispatch.
<wgrant> Right.
<wgrant> I mean, it could probably just be the BQ ID.
<wgrant> Although it would be nice to be safe.
<jtv> It should include that, but shouldn't it also be resistant against a compromised slave guessing the id of another ongoing slave job?
<jtv> Which is why I was suggesting <BuildQueue.id>-<cookie>
<jtv> Because you obviously can't trust a slave who doesn't have his cookie.
<wgrant> jtv: I thought so when looking at the code on the first night -- I got very scared.
<wgrant> But the master doesn't trust that.
<wgrant> it only uses it to check against the DB record and potentially rescue.
<wgrant> You can't use it to hijack a build.
<jtv> So what does the master trust?
<wgrant> The DB.
<jtv> BuildQueue.builder?
<jtv> Builder.currentjob?
<wgrant> Right.
<jtv> I'm sort of disoriented... is there any reason why we have this id at all now?
<wgrant> We need something to check if the builder needs to be rescued. A hijacked builder may be able to subvert that detection, but it's nice to be able to handle the non-malicious case.
<jtv> The thing about malicious cases is that they have a way of disguising themselves as whatever non-malicious cases you try to support.
<jtv> :)
<wgrant> The worst they can do is prevent the builder from being aborted, thus hanging it, or cause inappropriate abortion, in which case they get aborted.
<wgrant> Also, they used to be able to break a log message or two, but that might have been fixed in the refactor.
<jtv> You're saying we've been very thoroughly checking for a complex, unlikely error condition that we've got no idea what to do with if we ever spotted it.
<jtv> This has all the hallmarks of government.
<wgrant> No, it has happened twice today.
<wgrant> It happens when builders drop of the network for a few minutes, for example.
<jtv> I don't mean the rescue, I mean mismatched ids!
<wgrant> The mismatched string is useful. Detecting a mismatch between the build and buildqueue IDs has probably never been useful, right.
<jtv> waitwaitwhat
<jtv> "The mismatched string" is useful?
<wgrant> There are two cases: the slave thinks it's building something, but the master thinks it's building something else.
<wgrant> The string mismatch will be noticed by the master, and it will solve everything.
<wgrant> The other case is where the slave thinks it's building something, but the master thinks it isn't building anything, and the something doesn't exist in the DB.
<jtv> Ah, ok, so matching the slave build id to the buildqueue id is useful, but further sanity-checking on the slave build id is not.
<wgrant> Let me think.
<wgrant> Probably not, no.
<wgrant> So, for the case where the master thinks it isn't building anything, it should probably just abort straight away.
<wgrant> When the master does think it's building something, the slave just needs to possess some value that can be linked back to the buildqueue.
<jtv> Even if we're not sure of the hazards, would you agree that we can do a more thorough job of the check with less work if we just use <buildqueue>-<cookie>?
<wgrant> We can't reliably detect compromised builders, so we should not try.
<wgrant> Right.
<wgrant> Even just <buildqueue> is probably fine, but who knows...
<jtv> Right.  If we're not 100% sure, we can add the cookie with very little workâand still scratch all this &$%*@# complexity
<wgrant> Well, it's not *that* complex.
<wgrant> I mean, if I'm malicious I can easily steal the cookie from another build log.
<jtv> I just had a recipe-build test fail because it passed a SourcePackageRecipeBuild.id instead of a Build.id to BuildFarmJobBehaviorBase.verifySlaveBuildID.
<wgrant> Hm? That should be fine.
<wgrant> As long as the correct behaviour class was selected.
<jtv> The correct behaviour class didn't have its own verifySlaveBuildID
<wgrant> It doesn't need it.
<wgrant> Doesn't SourcePackageRecipeBuildFarmJob have a .build?
<jtv> Well, I was simplifying verifySlaveBuildID and a slightly more thorough check turned out to be simpler.
<wgrant> What was the change?
<jtv> Yes, it does, but it's a buildbase not a build.
<wgrant> Both Build and SourcePackageRecipeBuild have an 'id' attribute, and they are the only concretions of BuildBase.
<jtv> IIRC the original checked the build id it got from the slave build id to self.buildfarmjob.build.id
<wgrant> Right.
<wgrant> That was done in order to be generic.
<jtv> I see.
<wgrant> Ugly, yes.
<jtv> I think part of the problem there is that I genericized the lookups of Build.id, SourcePackageBuild.id, and BuildQueue.id to use a single helper function.
<jtv> Which I still think makes sense.
<jtv> But being so generic, of course it does some sanity checks against comparing ids from different classes.
<jtv> Except obviously I can't just go "are these two of the same class?"
<jtv> So instead, the helper takes the expected class as an argument and does a zope "isinstance" check.
<wgrant> Ahh.
<jtv> And no matter which way we turn it, that work seems pointlessâthe real link is made by the buildqueue id (which by rights should come first in the slave build id) and the added "specialized extra id stuff" in there is just going to make accidental mismatches more likely and any stopping power for attacks less likely.
<wgrant> Right.
<jtv> abentley should be asleep :)
<wgrant> remember this dates from the beggining of time.
<wgrant> abentley: He pinged out a couple of minutes ago, so he probably is.
<jtv> wgrant: you seem to be overestimating my age, which is not generally considered the safe side to err on in conversation.
<jtv> I do not "remember" the beginning of time, thank you very much.
<jtv> I'll ask barry though.
<wgrant> Heh.
<jtv> So here's what I suggest, and can propose on the mailing list:
<jtv>  * We have a single "concoct an id for this buildfarmjob" method.  It produces <BuildQueue.id>-<pseudo-unique string>
<jtv>  * Nobody fucking messes with this.  Laziness is encouraged in the implementation classes.
<jtv>  * A single verification method is enough.
<wgrant> Hopefully Soyuz will agree that the pseudo-unique string can be dropped.
<wgrant> But yes, please propose.
<jtv>  * The entire, ready-made id gets passed to dispatchBuildToSlave.
<jtv>  * ...which probably doesn't have much left besides this to justify specialized implementations, but I digress.
<wgrant> dispatchBuildToSlave has to cache files and calculate arguments and all that.
<jtv>  * Everybody agrees to bury the whole issue and forget the episode ever happened.
<jtv> Yeah, but these are separate jobs.  I'm not convinced we need a single "do all the household stuff you need here" method.
<wgrant> Right.
<wgrant> It should all change if everything goes properly async, anyway.
<wgrant> RecordingSlave's trickery makes me sad.
<wgrant> But it was the quickest way to make bits of it async.
<jtv> BTW you may think it odd that I keep coming back to the cookie...  you probably have a much better understanding of whether it would be really needed; for me it's an insurance policy.  "If double-checking the slave build id is necessary, this does it better.  If it's not necessary, this is still less work than what we have now.  And future changes are more likely to strengthen these arguments than to weaken them."
<jtv> -> no need to argue over the colour of the inside of the bikeshed.  :-)
<wgrant> Yep.
<jtv> I'd much rather add that piece of code than get stuck in that discussion while maintaining the current code!
<wgrant> But cookie probably implies DB change.
<wgrant> And DB changes make me sad.
<jtv> ah, there's the catch: is this id actually stored anywhere
<wgrant> No.
<wgrant> It is calculated by the master, and stored by the slave during the build.
<wgrant> At no point should it be possessed by the master outside dispatchBuildToSlave and rescueBuilderIfLost.
<jtv> So that does make the case against the cookie stronger, unless we can find some generic, reproducible, and less-than-predictable stuff to hash.
<jtv> Or include in some way.
<wgrant> Let's just ask Soyuz, I think.
<jtv> Yes.
<jtv> (FWIW the exact Job.date_created would IMHO probably be a better "cookie" than anything we have now, though still not particularly good)
<wgrant> Ah, yes, that is a good idea actually.
<jtv> Certainly better than using floating-point primary keys, as a friend's customer did.
<wgrant> Since it's the particular build of the BQ.
<wgrant> Haha.
<jtv> (BTW I got a call from the hardcore windows guy who's trying out Ubuntu Server...  to my utter amazement he's wildly enthusiastic despite the teething problems you'd expect)
<jtv> Okay, I'll write this up for the ML
<wgrant> Great! On both counts.
<jtv> wgrant: message is on the way
 * wgrant reads.
<wgrant> jtv: Build.id and BuildBase.id are the same thing.
<wgrant> Build inherits BuildBase.
<jtv> There isn't really a BuildBase.id though, is there?  I meant "the id of some other BuildBase-derived class."
<wgrant> True.
<jtv> Which only goes to illustrate how messy the whole thing is, I suppose
<jtv> wgrant: it's been about a 32-hour working day, so I think I'll go enjoy the tail of weekend now.  See you next week I hope!
<wgrant> jtv-afk: Wow! Yes, go and have a weekend.
<wgrant> See you.
<jtv-afk> :)
#launchpad-dev 2010-03-14
<dhillon-v10> hi all :) okay so I am trying to fix this little bug so I branched out from devel and made my changes in this new branch, now when I do make schema to see how the change in this new "launchpad" look it gives me errors: http://pastebin.com/2e9r9DeH why is this happening
<dhillon-v10> nvm I got it working, it was an indentation issue :)
<maxb> Aaaaand I've fixed the lucid spidermonkey-bin issue..... just in time for postgresql-contrib-8.3 to become uninstallable :-)
<wgrant> maxb: At least things should quieten down soon.
<wgrant> And the likelihood of a Python 2.6 or PostgreSQL 8.4 removal in the near future is... low.
<wgrant> So you might have almost won. :/
 * maxb dputs a no-change rebuild of postgresql-8.3 to hopefully fix that.
<mwhudson> good morning
<thumper> morning
<maxb> Um. I just ran the full testsuite it caused 5 "ppa-user joined ubuntu-team" emails to be sent to root@localhost.
<wgrant> That would be soyuz-sampledata-setup
<wgrant> But I didn't know it was tested.
<wgrant> Ah, yes, lib/lp/soyuz/doc/sampledata-setup.txt
<mwhudson> firefox is very crashy for me today
<mwhudson> for example, gmail makes it segfault
<mwhudson> i guess my profile is screwed somehow
<lifeless> \o/
<lifeless> on lucid ?
<mwhudson> no
<mwhudson> karmic
<mwhudson> i wonder how to install the debug symbols
<lifeless> there is a specific archive
<lifeless> apport recan can grab em for you
<wgrant> ddebs.ubuntu.com
<mwhudson> a specific *archive* ? wow
<mwhudson> bah, the firefox debug syms are not up to date?
<mwhudson> oh, probably need -updates
<mwhudson> wgrant: do i have to do something magic to get gdb to see them?
<wgrant> mwhudson: You shouldn't need to.
<mwhudson> !!
<mwhudson> even with a new profile it crashes
<wgrant> Ouch.
<wgrant> No new system-wide plugins?
<mwhudson> maybe
<mwhudson> it crashes in safe mode though
<wgrant> Hm, the text on +upgrade is oddly absent.
<wgrant> It should probably tell you that you should be really sure you want to do this, that it will break people, and which format it will upgrade to.
#launchpad-dev 2011-03-07
<wallyworld> yes, but we're getting back to the topic i raised on the dev list - i disagree with our current workflow because it encourages only superficial qa just get get stuff deployed rather than testing for defects / verification
<lifeless> wallyworld: uhm, thats the point though.
<wallyworld> not imho :-)
<lifeless> wallyworld: accept *some* risk, in return for velocity.
<lifeless> wallyworld: higher velocity gives faster TTR
<lifeless> wallyworld: and smaller deltas per deploy
<wallyworld> sure, but i think we go too far and rework is the enemy of velocity
<lifeless> wallyworld: and that gives a gaster TTD
<lifeless> wallyworld: we are doing far less rework than in the old system
<wgrant> Anyone want to review https://code.launchpad.net/~wgrant/launchpad/bug-728836/+merge/52355?
<lifeless> wgrant: I think it may conflict with my fix
<lifeless> the distroseries attribute in particular
<wallyworld> now, if there were metrics to show the number of bugs that had to be reopened/reworked, that would be interesting
<wgrant> lifeless: Which fix?
<lifeless> wallyworld: I don't see how the ok to deploy process affects rework in this case
<lifeless> wallyworld: 723560 or whatever it is
<lifeless> wgrant: bug 727560
<_mup_> Bug #727560: Archive:EntryResource:getPublishedSources <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727560 >
<wgrant> Ah, right.
<lifeless> wallyworld: the process of closing bugs when the first deploy against them is done is easily tweaked and doesn't affect the deploy discussion at all.
<lifeless> wallyworld: I appreciate that we need to improve the way these two processes interact, but its also -extremly- important we don't conflate the way we improve one with the way we improve the other.
<StevenK> I think the qatagger needs a patch to give a list of bugs that can be closed by the rev we can deploy to.
<wallyworld> lifeless: if affects rework because it encourages only superficial qa and not a "proper" check that the issue supposed to be fixed really is fully fixed. and so increases the chance that the defect will have to be revisited to fix it "properly"
<lifeless> wallyworld: once a commit is in trunk, it *must* be deployable or rolledback in a matter of houres
<StevenK> Makes it easier to cut-n-paste into LPS
<wgrant> lifeless: It appears not to conflict.
<lifeless> StevenK: there is a bug open, and when Ursinha rolls out my tweak to the templates it wil lshow Bug:1234 rather than Bug 1234 in the report
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<lifeless> wgrant: cool
<StevenK> Haha, I love that bug
<StevenK> Since it STILL is
<StevenK> lifeless: I meant a line at the top that has "If we deploy, the following bugs can be closed: 58585 58568 58922 66856 ..."
<wgrant> StevenK: That bug's fix is the cause of it being so terrible.
<lifeless> StevenK: yes there is a bug open for that
<StevenK> gina needs a good hard rewrite
<lifeless> wgrant: does get_archive do queries ?
<wgrant> lifeless: I made Archive.debug_archive a cachedproperty to prevent that.
<lifeless> where does bpr come from ?
<lifeless> 278	+        candidates = (And(
<lifeless> 279	+                BinaryPackagePublishingHistory.archiveID ==
<lifeless> 280	+                    get_archive(archive, bpr).id,
<lifeless> oh, I see
<lifeless> 8 line generator.
<lifeless> wgrant: not the clearest
<wgrant> Bah.
<wgrant> Don't blame me for generators being backwards :P
<lifeless> forward ones are known as 'for loops'
<wgrant> Maybe.
<wgrant> It wasn't initially quite so big.
<wgrant> The archive used to be in the main query.
 * StevenK notes SMM is now out of date, given 45 second PQM is real
<StevenK> And I think we could go one better and hook tarmac into Jenkins with a paramaterized build
<wgrant> Heh.
<wgrant> lifeless: Thanks.
<wgrant> benji: Not around?
<lifeless> wgrant: what do you need ?
<wgrant> lifeless: Unfreezing.
<lifeless> wgrant: we don't need benji, we need a losa
<lifeless> wgrant: IIRC the docs were updated on friday
<lifeless> wgrant: so it should be a simple point, ask a losa, and mail benji & the list
<wgrant> I haven't read the new ones.
<lifeless> wgrant: might be an idea ;)
<wgrant> Better to ignore the docs and see what works ;)
<lifeless> wgrant: other folk will look to the docs for reference
<lifeless> wgrant: so keeping them in sync is important
<lifeless> (or delete if not needed)
 * thumper is close to pounding the table in frustration...
<wgrant> lifeless: Where are these docs?
<wgrant> thumper: If you need to pound a table, please choose BranchRevision.
<thumper> haha
<wgrant> If you can break a few indices off it would be great.
<thumper> wgrant: we have a solution, but no time to implement
<wgrant> It is like 130GB!
<thumper> wgrant: jam and abentley have a workable solution that'll fix loggerhead speed too
<thumper> wgrant: they just need the OK to spend some time doing it
<wgrant> Ah.
<thumper> wgrant: using bzr-historydb
<wgrant> Will that work for LP too?
<thumper> wgrant: it'll kill the table completely
<thumper> wgrant: as far as we are aware
<lifeless> wth is getQuery for
<wgrant> I thought that was only planned for loggerhead.
<wgrant> That's great news.
<thumper> wgrant: quite a bit of analysis went into it
<thumper> lifeless: on what?
<lifeless> thumper: grep for '\<getQuery\>'
<thumper> heh
<thumper> kill it
<wgrant> It's not needed by the Zope vocab stuff?
<thumper> ah... maybe
<thumper> check the Interface
<wgrant> I'm grepping.
<wgrant> Doesn't seem to be.
<lifeless> Yeah, I had grepped there already
 * thumper stabs some tech-debt in the face
<wgrant> lifeless: https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadRollout says that we can only unfreeze once the stable rev is determined.
<wgrant> By the RM.
<wgrant> This is why I don't like reading documentation.
<wgrant> Because I means I have to knowingly subvert processes instead of just being ignorant of them.
<lifeless> wgrant: === this stuff below is old and somewhat invalid with buildbot ===
<wgrant> lifeless: Not there... near the bottom of the first section of "Preparatory Tasks"
<wgrant> "Once all QA is done (including any RC branches that were landed after the db-stable -> devel merge), the RM will let the LOSAs know the revno of stable that we will use as the release branch. Once this release branch is determined, a LOSA can put devel back into normal mode on pqm and prep the stable branch for rollout"
<wallyworld> anyone know if the teamparticipation table is designed reflect valid team memberships taking account of expiry dates, membership status etc? ie can i just query that one teamparticipation table to see if a person validly belongs to a team?
<wgrant> wallyworld: Yes, use TeamParticipation.
<wallyworld> wgrant: thanks
<wgrant> wallyworld: It is unreliable in one case: some code (eg. inTeam) considers the team owner to be an implicit member.
<wgrant> But we consider that a bug, and anything that uses TP directly (lots of stuff does) does not see that.
<wallyworld> wgrant: ok. so add a teamparticipation clause to a query is considered ok i assume
<wallyworld> adding
<wgrant> Yup.
<wallyworld> cool
<wgrant> That's part of the point of it.
<wgrant> Nice and easy.
<wgrant> And quick.
<wallyworld> sure is. so that means there must be code to maintain that table based on expring memberships etc
<lifeless> wgrant: fixed
<lifeless> thanks doko
<wgrant> What about doko?
<lifeless> mbf on dh_python
<wgrant> Aha.
<lifeless> so my inbox just exploded
 * wgrant finds food and works out whether to drive +copy-packages down further or attack checkwatches.
<lifeless> wgrant: 59 /    0  LanguageSet:CollectionResource:#languages
<lifeless> wgrant: has noone attacking it right now
<wgrant> lifeless: Maybe, but I want to get +copy-packages down low enough that I can stick a test over it.
<wgrant> I basically just need to preload source files now.
<StevenK> Didn't we add a CopyPackagesJob or so?
<wgrant> There is the package clone job which is used for copy archives.
<wgrant> But anyone pushing normal package copying out into a job is insane and likes causing QISEs.
<wgrant> (see delayed copies, for example)
<StevenK> Yes, you said earlier delayed copies don't need to exist?
<wgrant> One reason for the existence (reuploading files to the public librarian) is wrong.
<wgrant> Another reason (slowness) will be wrong once this lands.
<wgrant> And the other (correctness when copying into the primary archive) is easily fixed by moving the overriding and emailing stuff into the normal copy logic.
<lifeless> StevenK: before we move things into backend jobs we should make them efficient.
<wgrant> lifeless: It is a dupe, but I cannot remember the summary of the original :/
<lifeless> nor I
<StevenK> wgrant: Lets kill delayed copies, then? :-)
<wgrant> StevenK: When I have a moment I am going to investigate how much work that is.
<wgrant> Custom uploads were another reason for delayed copies, but they were never actually implemented.
<wgrant> And any future implementation should be done by making custom uploads not INSANE.
<StevenK> Right, I have my test written, now comes the lovely "test dies with a permission error, edit security.cfg, make schema", rinse and repeat
<wallyworld> StevenK: i did that over the w/e. not much fun :-(
<mwhudson> you don't need to run the whole make schema do you?
<mwhudson> just python database/schema/security.py or something
<StevenK> I investigated last time I had this issue, and the only thing that helped was the full monty of make schema
<mwhudson> enjoy then
<lifeless> you need to run it against the template test database
<lifeless> but yes, you don't need full schema build
<StevenK> It was only 3 perms, so the pain was minimal
<StevenK> Oh, sigh, every SPPH for Debian is PENDING.
<StevenK> *That'll* help.
<wgrant> StevenK: We're going to need to solve that eventually.
<wgrant> I have considered getting the dominator to run over it.
<StevenK> And after it takes 4 days?
<wgrant> But we also need to handle deletions somehow.
 * thumper grabs a bite to eat, then gets the girls
<StevenK> wgrant: Just trying to write a better query to see how much of Debian's SPR.changelog are None
<wgrant> StevenK: You should always use IN (1, 2), whenever you are querying for publications.
<wgrant> Querying only for 2 is always wrong, unless you are the dominator.
<StevenK> Apparently, 12k. I think my query is wrong, anyway
<wgrant> StevenK: Maybe just SELECT COUNT(*) FROM sourcepackagerelease WHERE changelog IS NULL and upload_archive=3;
<wgrant> Won't only give you current stuff.
<wgrant> But meh.
<StevenK> wgrant: Can't we craft the query like the populator does and ignore stuff with expired SPRFs?
<wgrant> StevenK: You could. But nothing in Debian should be expired.
<wgrant> Because our GC is sort of awful awful awful awful.
<StevenK> I guess the timing of a test can't really be taken as anything?
<wgrant> What do you mean?
<StevenK> From devel: Total: 16 tests, 0 failures, 0 errors in 1 minutes 4.196 seconds.
<lifeless> what about it
<StevenK> My branch: Total: 17 tests, 0 failures, 0 errors in 1 minutes 5.195 seconds.
<lifeless> sounds like your branch adds a 1 second test
<StevenK> Clearly, my question is that number "reliable"
<lifeless> repeat it 5 times
<lifeless> take the lowest time for devel and the lowest time for your branch
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews | PQM: open RM: benji
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> LEAN
<wgrant> Heh.
<lifeless>   Hard / Soft  Page ID
<lifeless>      130 /  223  BugTask:+index
<lifeless>      109 /    0  LanguageSet:CollectionResource:#languages
<lifeless> wgrant: ^ let me encourage you to stay flexible and responsive
<lifeless> wgrant: we will get back to copy packages
<wgrant> Indeed.
 * wgrant finds a timeout.
<wgrant> Hmm. Or I could fix two checkwatches OOPSes and get us below 1000 exceptions by the end of the week.
<wgrant> Tempting, tempting.
<wgrant> But #languages it is.
<wgrant> WTF, why is it querying Karma?
 * wgrant deletes #languages as being stupid.
<StevenK> Thereby fixing the timeout?
<wgrant> Yes.
<StevenK> ... maybe people use it?
<wgrant> It fixes OOPSes *and* decreases WTF/m.
<StevenK> wgrant: Sigh, SELECT count(spr.id) from sourcepackagerelease as spr JOIN sourcepackagereleasefile as sprf ON sprf.sourcepackagerelease = spr.id JOIN libraryfilealias AS lfa ON sprf.libraryfile = lfa.id WHERE lfa.content is NULL and spr.upload_archive = 3; => 0
<wgrant> Who cares if peopel use it :P
<StevenK> wgrant: The users, I guess.
<wgrant> Hmm.
<wgrant> I think I will unexport Language.translators_count, and maybe turn it into a method.
<StevenK> Maybe people use that too? :-)
<wgrant> No widely distributed scripts use it, so I do not care much about the users.
<wgrant> They can adapt.
<StevenK> What are you, a kernel developer?
<wgrant> Probably.
<lifeless> wgrant: is it in 1.0 ?
<lifeless> wgrant: was it in 1.0 at release ?
<wgrant> lifeless: It was added in March 2010. So it was probably just after 1.0 was declared.
<StevenK> lifeless: Anyway, aren't you supposed to drop the lowest, the highest and take the average of the remaining?
<lifeless> StevenK: huh?
<StevenK> lifeless: Rather than run the tests 5 times, and just take the lowest number
<wgrant> So, I could preload, or I could temporarily break a couple of scripts that might not exist and make requests a second or so faster.
<wgrant> jtv: Heh, I was just looking at that.
<lifeless> StevenK: we assume that a noisy environment can't make your test faster
<lifeless> StevenK: and thus everything that makes it slower is noise
<wgrant> jtv: We could optimise the query, or we could reexport the attribute as a method, breaking all of the very few scripts that use it.
<lifeless> wow
<lifeless> wgrant: I would default to optimising in the first instance
<lifeless> wgrant: its easier (than the coordination to remove something in 1.0)
<wgrant> lifeless: We need to coordinate the removal of most of 1.0 soon.
<StevenK> Total: 16 tests, 0 failures, 0 errors in 56.713 seconds.
<StevenK> Total: 17 tests, 0 failures, 0 errors in 1 minutes 2.982 seconds.
<lifeless> wgrant: is that hyperbole or do you know something I don't ?
<wgrant> lifeless: We have committed to supporting 1.0 for another 4 years. That is completely stupid.
<StevenK> 6 seconds with the two best times, 2 seconds with the two worst times.
<wgrant> We need to rescind that commitment.
<lifeless> wgrant: I agree that is has a high cost; its why I'm encouraging devs to only put stuff in devel, and am sure that for 1.1 (or $label) we will only want to move some stuff into supported.
<lifeless> so
<lifeless> I have an idea
<lifeless> why don't we write a caching layer for milestones
<lifeless> and use it for the hidden edit forms but not for the visible bugtask rows.
<jtv> Hi wgrantâwhatever works, I guess.  :)  I'm not sure that "it's a method now" will console many people thoughâ¦
<jtv> wgrant: plus, there's already a bug for timeouts onâ¦ getAllLanguages IIRC.  So if you want a method that times out rather than a property that times out, well, you've already got one.
<jtv> Though I do think what you say has merit in itself!
<wgrant> jtv: Hmm? getAllLanguages is exactly the same problem.
<wgrant> It gets lots of Languages, and serialising the Language calculates the translators_count.
<jtv> Yes, I figured, but ironically the bug page didn't load.
<jtv> That definitely shouldn't be a property, noâ¦
<wgrant> It is always going to be expensive.
<wgrant> It doesn't have to be so expensive.
<lifeless> so
<wgrant> But it's still going to be expensive.
<wgrant> The number of reasonable uses for that property by an API script is 0.
<jtv> I was assuming that the API request specifically was deliberately getting contributor counts for all languages.
<lifeless> I really suggest a) optimise, b) if we want to change start the discussion to change it
<wgrant> Hmm.
<wgrant> Maybe I can reexport it on devel, and preload on 1.0.
<lifeless> I need a reviewer
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-724033/+merge/52358
<lifeless> wgrant: yes. Which means starting with the preload, and then (probably) file a bug to do something about it in a while in devel.
<lifeless> :P
<wgrant> Right.
<lifeless> hmm, POFile:+translate is still showing up
<jtv> Yes, it's difficult and IIRC there was a rewrite that we never completed.
<jtv> lifeless: looking at your branchâ¦
<wallyworld> jtv: wanna review my fix for 407260 when you are free? https://code.launchpad.net/~wallyworld/launchpad/branch-picker-team-branches/+merge/52356
<jtv> wallyworld: I'm just reviewing Robert's branch
<wallyworld> jtv: no hurry at all. i just thought you may be interested :-)
<jtv> I am, I amâ¦ just not free to look at it just yet.  :)
<lifeless> wgrant: DistroSeries:EntryResource:getBuildRecords - improved by your branch ?
<wgrant> lifeless: no.
<wallyworld> thumper: do you have any examples of making a ws api call on one of our domain objects within javascript?
 * thumper looks
<lifeless> jtv: fixing the silly docstring
<thumper> wallyworld: there is something in lp.code.tests.test_branch_webservice
<wallyworld> thumper: thanks.
<jtv> lifeless: speaking of docstrings, it might be nice to have one on CachedMilestoneFactory._load.  I have a feeling there's more going on there than I understand.
<wallyworld> thumper: that's all python. i was after javascript :-)
<lifeless> jtv: what do you think is going on ?
<wallyworld> thumper:  i know how to do it from python :-)
<thumper> wallyworld: ah...
<thumper> wallyworld: what do you want to do?
<lifeless> jtv: ah, its buggy. Fixed.
<jtv> lifeless: the fact that it's based on self.contexts suggests that it's meant to be called only once, but if it were meant to, I get the impression it would've been a one-liner.
<wallyworld> thumper: i want to invoke the recently exported pending_builds api on a recipe object from within javascript
<lifeless> jtv: pushing the just once guard now.
<jtv> The what
<jtv> ?
<jtv> Ahhhh, _that_ is what the comment meantâ¦
<lifeless> jtv: uhm its not a one liner because our vocabularies are based on /a context/ not on /contexts/
<jtv> â¦it meant you only want to call this once?
<jtv> It was very, very unclear and helped confuse me.
<lifeless> jtv: its called many times
<wallyworld> thumper: so i'm on the recipe index page and there'll be a recipe web_link i can use
<lifeless> jtv: I want the heavy lifting to take place just once, and all at once
<lifeless> jtv: we currently have some bugs where we construct *300* milestone vocabularies.
<thumper> wallyworld: you want to do something like... :
<thumper> lp = Y.lp.client.Launchpad();
<lifeless> jtv: http://bazaar.launchpad.net/~lifeless/launchpad/bug-724033/revision/12538
<thumper> lp.get(some_path, config)l;
<thumper> and that'll be the recipe
<thumper> then call the method on that object
<thumper>  var client =  new Y.lp.client.Launchpad();
<wallyworld> thumper: oh ok. looks reasonable. i'll give it a go thanks.
<thumper> wallyworld: grep for "client.get" and you'll find some stuff
<wallyworld> thumper: yeah, just grepping for ".get(" right now in fact :-)
<lifeless> looks like I get POFile:+translate again
<wallyworld> thumper: i've got enough to go on now but there really don't seem to be very many places at all we use the ws api from our js. unless i only had a boy look :-)
<thumper> wallyworld: correct, we don't
<wallyworld> thumper: is that something considered bad or alternatively we just haven't gotten around to it yet? i would have thought it would be the "right thing to do" from a soa perspective
<thumper> wallyworld: we don't make many api calls over javascript to code objects
<wallyworld> thumper: in that case would you prefer that i got the pending builds for a recipe another way? the ws api seems correct though. otherwise i'll need to hack zcml and add zope stuff
<thumper> wallyworld: what are you doing?
<wallyworld> thumper: may be easier via mumble?
<thumper> wallyworld: yep,
<thumper> me hooks up
<thumper>         client.asserts.assertProperty(
<thumper>             jquery=u'("div#edit-lifecycle_status span.value")',
<thumper>             validator=u'className|branchstatusEXPERIMENTAL')
<jtv> ahh I'm back.  lifeless, the last I got from you was "I want the heavy lifting to take place just once, and all at once."  It makes sense to me now that you've fixed it: the code I read initially re-created all the MilestoneVocabs on every invocation.
<jtv> (Unless I'm going completely dyslexic, which I also wouldn't discount as a possibility :)
<jtv> wallyworld: posted a comment on your branchâ¦  hard-hitting questions about your approach.  :)
<wallyworld> jtv: thanks. otp. will look soon
<jtv> lifeless: to explain what I meant with _load being essentially a one-liner: I was thinking if the work is done just once, then ISTM it would amount to self.vocabularies = dict(target, MilestoneVocabularyTarget(target) for target in map(MilestoneVocabulary.getMilestoneTarget, set(self.contexts)))
<jtv> Is that correct or am I still misunderstanding something?
<lifeless> jtv: you're missing that it instantiates the vocaularies
<lifeless> jtv: not the targes
<lifeless> targets
<lifeless> milestone_vocabulary = MilestoneVocabulary(target)
<jtv> I thought MilestoneVocabularyTarget(target) instantiated a vocabulary?
 * thumper afk for a while, back later tonight
<lifeless> jtv: a product and a product series of that product share a set of milestones
<lifeless> jtv: there is no MilestoneVocabularyTarget
<jtv> Sorry, I mis-typed that.
<jtv> Scratch the Target.
<lifeless> jtv: so its a two pass filter: one to map context->targets which is the actual things that we need milestones on
<lifeless> jtv: and a second to construct the vocabularies for each target
<lifeless> jtv: a later branch will make the second pass a single call to do optimised sql and generate all the vocabularies out of a minimal set of lookups
<jtv> lifeless: but without the typo, and modulo icky external references to self.vocabularies, you're saying _load does not amount to "self.vocabularies = dict(target, MilestoneVocabulary(target) for target in map(MilestoneVocabulary.getMilestoneTarget, set(self.contexts)))"?
<lifeless> jtv: it won't once the next stage is done
<jtv> The set() needs to be outside the map() perhaps?
<lifeless> jtv: because it won't loop at that point
<jtv> Ah, that'll be moved inside some batched method?
<lifeless> yes
<lifeless> this is prep work to do an eager load optimisation on bugs that have 150 different products / distro combinations
<lifeless> it will reduce the query counts we have today substantially I hope, and make it easier to do the next stage
<jtv> Then maybe you'd also like to move the self.cached_milestone_source.contexts.add() outside the loop that calls _getTableRowView?  The loop iterates over a list anyway.
<jtv> Whaa I need to be afk for a moment
<lifeless> jtv: I wanted to keep it adjacent to the code that hands out the milestone_source
<lifeless> jtv: that way they can't get out of sync
<jtv> back
<lifeless> jtv: so its layering: someone calling _addTableRow or whatever its called *has* to make sure the context is added to the contexts set or the assertion will trip
<lifeless> jtv: the easiest way to ensure that is to have addTableRow do it
<jtv> lifeless: sure, makes sense
<jtv> _getTableRow
<jtv> View
<lifeless> right
<lifeless> I'm in POFile:+translate now, so going from memory
<jtv> (Also, I just noticed it's called in another place as well)
<jtv> (Still in the same loop, but nevertheless)
<jtv> lifeless: if you have a bug open for the future task of batching this, it'd be nice to make that TODO comment a proper XXX so that all you have to do later is grep for the bug number.  Or someone else doing the same job, if it comes to that.
<lifeless> jtv: its the same bug
<jtv> Oh so you're doing that right after this branch?
<lifeless> probably
<lifeless> depends on the impact, but I suspect we'll still have timeouts on this once this lands
<lifeless> BugTask:+index is our #1 timeout
<jtv> lifeless: I'd prefer a note to check up on that (and file a bug if appropriateâwhether it's timing out or "still a bit slow") over a TODO comment in the source TBH.
<jtv> (Since you're going to Q/A this anyway, I'm assuming that your eyes will pass over the relevant data anyway)
<lifeless> jtv: I think that thats waste TBH: the data will tell us
<lifeless> I can delete teh comment if you like, but thats also waste - we're probably going to be back here in two days doing more.
<lifeless> the comment will tell someone else guided here by profiling an option I've considered; a bug would not be usefully triaged outside of the data that we have driving maintenance
<jtv> I don't think the comment will be very helpful to someone else reading it.  It's little more than a reference to something you have in your head that they'll have to reconstruct by reading the same data, or dig up who wrote it so they can ask.
<jtv> If you can make it clearer, then great.
<jtv> I take that back; it's clearer now.
<jtv> Only other thing is testâit'd be good to know that calls for the same target will return the same vocab.
<lifeless> I'm really not inclined to - because:
<lifeless>  - I've refactored, vs introducing a new layer: whatever tests there are for this either already ensure that, or were judged unnecessary in the past
<lifeless>  - the key thing isn't whether the same instance is returned, its whether the query count is flat
<lifeless>  - and we can't test the key thing yet because the page is so darn horrible.
<jtv> lifeless: so there is a test to ensure that the query count is flat?
<lifeless> jtv: no, because its not
<lifeless> jtv: 675  OOPS-1891H475   NullBugTask:+index for instance
<lifeless> wgrant: want to mentor https://code.launchpad.net/~stevenk/launchpad/populate-spr-changelogs/+merge/52363?
<lifeless> sorry, be mentored on
<wgrant> lifeless: I probably shouldn't, since it's an adaptation of my code. But I could.
<lifeless> wgrant: I'll still give it a full review
<wgrant> k
<lifeless> jtv: can I ask a question about pofile:+translate
<jtv> go ahead
<lifeless> in lib/lp/translations/browser/translationmessage.py
<jtv> oh God
<lifeless> line 1287 says if self.form_is_writeable:
<lifeless> explaining we don't nee dsuggestions if the user cannot do translations
<lifeless> but
<lifeless> after the else clause of that if block
<StevenK> lifeless, wgrant: I'm still attempting to get LP to generate a diff
<lifeless> jtv: there is another if block starting with if self.sec_lang is None:
<wgrant> StevenK: Push again. If that doesn't work, push -2 then push again.
<wgrant> I think I know what's going on, but can't check because ENOLOSA.
<lifeless> jtv: and it *also* asks for suggestions, but is not guarded by the form_is_writable check
<lifeless> jtv: I am thinking that that is a bug, but is it ?
<jtv> lifeless: arguablyâ¦ the secondary language serves two very different use cases.
<StevenK> Unable to obtain lock  held by stevenk@bazaar.launchpad.net
<StevenK> at crowberry [process #14934], acquired 52 minutes, 15 seconds ago.
<_mup_> Bug #14934: [warty] mozilla-browser: JS can access any mozilla memory <mozilla (Ubuntu):Fix Released by thombot> <mozilla (Debian):Fix Released> < https://launchpad.net/bugs/14934 >
<StevenK> Whee!
<lifeless> jtv: e.g. alt_submissions should be [] if form_is_writable is False
<jtv> One is "this language is so similar, I'd like suggestions from it so I can just click them into the translation I'm working on."
<jtv> The other is "me speak english no good, me like read translations other language as well"
<lifeless> jtv: ok, so its not a bug
<jtv> No
<lifeless> so, I'll make it more complex there to keep the behaviour
<jtv> Sorry to hear it, but I think it's for the best.
<lifeless> jtv: its about 80% more efficient to query for the alt language with the primary language than to query seperately
 * jtv whistles
<lifeless> 350ms query * two, or one 355ms query
<jtv> There's something that could be done about that, but not now:
<jtv> re-design with the two use-cases in mind as separate goals.
<jtv> The second use case doesn't actually need the suggestions if there's an accepted translation.
<jtv> lifeless: there's something we did in the pre-sharing model that we threw out ("for the time being") to keep things simple.
<jtv> Just fetch all relevant TranslationMessages, and sort out their roles python-side.
<lifeless> jtv: yeah, thats what I'm bringing back in here
<lifeless> jtv: my patch last week took a solid chunk of time off
<lifeless> but not enough
<lifeless> so this is the next step
<jtv> There's really no point IMO in having separate calls "get me the current translation," "get me the current translation for the other side, and by the way is it the same one," "get me the suggestions on this side," and even "get me the suggestions for other, similar messages."
<lifeless> jtv: the query plan for this is very expensive - it expects tonnes of IO
<jtv> You're telling meâ¦ cost me years of my life
<lifeless> but just think about all the sql you know now.
<jtv> Done.
<jtv> Why?
<lifeless> a joke; suggesting that getting this fast had driven some sql knowledge
<jtv> Not reallyâit drove a lot of optimization knowledge, but optimization goals have shifted a lot since the page formed.
<lifeless> jtv: it was humour, not a serious statement
<jtv> Forgive meâI am not well
<lifeless> jtv: :( jetlag?
<jtv> Hang onâfriend in what sounds like serious trouble
<wgrant> Multiple stores make me sad.
<huwshimi> Has anyone here tried to set up Launchpad on Natty and had a problem installing python2.6-minimal?
<wgrant> That should still work... what's the error?
<huwshimi> wgrant: http://paste.ubuntu.com/576805/
<wgrant> lifeless: ^^ you are probably keeping better track of modern Python transitions than I.
<lifeless> fun
<StevenK> bzr-lpreview-body!!
<lifeless> huwshimi: what does dpkg -S /usr/lib/python2.6/site-packages show
<StevenK> I suspect huwshimi's output to be the same
<huwshimi> yeah the same
<huwshimi> bzr-lpreview-body: /usr/lib/python2.6/site-packages
<lifeless> in which case, remove that package
<lifeless> then install python2.6-minimal
<StevenK> But fixing bzr-lpreview-body doesn't enter into it?
<lifeless> sure it does
<lifeless> two separate things though
<lifeless> unblock, then root cause
<huwshimi> lifeless: That wants to remove python2.6 and some other python stuff, is this going to kill me?
<wgrant> huwshimi: Use dpkg -r bzr-lpreview-body
<StevenK> As root or with sudo
<wgrant> apt-get remove will try to resolve the dependency mess, dpkg -r won't.
<huwshimi> StevenK: thanks :P
<StevenK> lifeless, wgrant: Diff for https://code.launchpad.net/~stevenk/launchpad/populate-spr-changelogs/+merge/52363 looks fine now
<wgrant> StevenK: OK, will look in a sec.
<huwshimi> Thanks guys, all working now
<lifeless> jtv: so where are we at on this review?
<lifeless> jtv: if you're sick, I can get a different reviewer.
<wgrant> StevenK: Commented.
<StevenK> wgrant: Rargh! (my fault, but your problem now)
<wgrant> Heh.
<wgrant> I presume I had a good reason for doing it.
<wgrant> But I don't recall.
<wgrant> It was my first tunableloop, so I may have been wrong.
<StevenK> I'm not sure how to record to skip a given SPR, though
<wgrant> Neither am I.
<wgrant> Which is why i still hold the opinion that having this in garbo is crack.
 * StevenK waves his hands furiously
<StevenK> You say garbo is crack, lifeless says this is the way
<StevenK> Rock, hard place!
<wgrant> Yes, but I am right :P
<StevenK> lifeless: ^
<jtv> lifeless: back... I think my friend must have run out of battery so I can complete the review.  Not that much more I can say to him except as a way to let him distract himself from his problems.  :(
<jtv> So I can complete that review.
<lifeless> jtv: cool, he is ok ?
<lifeless> wgrant: whats the story
<lifeless> wgrant: and what does skipping an spr have to do with running it in the garbo
<wgrant> lifeless: SPRs may fail to extract or be otherwise unimportable.
<wgrant> lifeless: The job will start from the first SPR without expired files and without a changelog.
<wgrant> So say we have 1000 SPRs that fail to import.
<wgrant> The job will process those 1000 at the start of every run, then run out of time.
<jtv> lifeless: No, he's not OK but he's out of immediate danger.  Part of the work every time this happens is to listen to him explain how he's not OK, and then the other part is the uphill struggle to convince him that he's not OK and there must be things he can do about it.
<lifeless> wgrant: this seems orthogonal to the execution framework
<wgrant> lifeless: It was designed to be started once.
<wgrant> Garbo starts it hundreds or thousands of times.
<lifeless> wgrant: thats a high risk design
<wgrant> Oh?
<lifeless> its extremely likely that anything running for more than a few hours will be interrupted
<wgrant> Sure.
<lifeless> stash the failed sprs in memcache and move on
<wgrant> Well, not extremely likely.
<wgrant> Back when this was written we were deploying once a week at most.
<wgrant> The script was meant to work and be run just a couple of times, not to be the most robust, efficient creature in the universe.
<lifeless> there is a pretty big spectrum in the middle
<wgrant> I am curious as to why you say anything running for more than a few hours will be interrupted, though.
<wgrant> We have some regular scripts that run for much longer than that.
<lifeless> sources of interrupts;
<lifeless>  - deploys
<lifeless>  - long running transaction killing
<lifeless>  - machine maintenance
<wgrant> Deploys: OK.
<lifeless>  - db deploys [forced interrupt]
<wgrant> Long running transactions: there are none.
<lifeless>  - 'kill X its making things slow'
<wgrant> Sure, and it can easily handle being killed once a day or so.
<wgrant> But once every 10 minutes?
<lifeless> its a 1 second query to find the work to do
<lifeless> why are we expecting failurs?
<wgrant> Because some don't have changelogs, some don't unpack any more, and I forget other reasons.
<lifeless> what will be the impact o fthat on derived distros
<wgrant> Most of them are old, and those that are not old will just not have a base version.
<lifeless> are you still doing it in increasing id order?
<lifeless> just stash the last done id in memcache; 3 line patch
<wgrant> Sounds reasonable.
<StevenK> lifeless: Uh, I don't know how to do that
<wgrant> There's a utility.
<StevenK> Which, sadly, is as useful as saying "There's this website, it's called Google."
<wgrant> Well, a memcache utility probably starts with IMemcache.
 * wgrant greps.
<wgrant> IMemcacheClient.
<stub> client = getUtility(IMemcacheClient)
<stub> result = client.set(key, value)
<stub> if not result: raise RuntimeError('Memcached access denied or memcached servers down')
<stub> value = client.get(key)
<StevenK> wgrant: To answer one of your points, I'm using the master store since everything else in garbo is.
<StevenK> stub: That's excellent, thanks.
<jtv> lifeless: with apologies for the delay (my friend found a phone charger) I just approved your branch.
<jtv> wallyworld: I initially typed my comment for your MP into the wrong window, so that it now defaces an unrelated bug.  Hopefully it's also on your MP now.
<jtv> And now, food.
<lifeless> jtv-eat: thanks!
<lifeless> stub: https://code.launchpad.net/~lifeless/launchpad/bug-730391/+merge/52370 - I can have review?
<stub> k
<jtv> StevenK: something I find myself in doubt about nowâ¦  I've done the code to create a bunch of DSDs in Needs Attention state, but do I need to create DSDJs for those _as well_?
<lifeless> wgrant: could you do me a favour?
<lifeless> wgrant: with timeout and oops bugs, always put the oops report key - the page id - in the title.
<lifeless> wgrant: e.g. bug 728836
<_mup_> Bug #728836: copyBinariesTo query count scales by number of binaries <oops> <timeout> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/728836 >
<lifeless> looks like a dupe of bug 575450
<_mup_> Bug #575450: Archive:+copy-packages nearly unusable due to timeouts <lp-soyuz> <ppa> <timeout> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/575450 >
<wgrant> lifeless: But which one? that has three :(
<wgrant> lifeless: It is part of 575450
<lifeless> wgrant: list em all
<wgrant> But 575450 is big.
<wgrant> So I wanted a separate one.
<wallyworld> jtv: i was picking up the kid from school before but got your comments and am about to push up changes
<jtv> cool
<stub> lifeless: r=stub with minor bits
<wgrant> Could someone review https://code.launchpad.net/~wgrant/launchpad/bug-728174/+merge/52374?
<lifeless> wgrant: so, on bugs - I think its fine to do new bugs
<wgrant> lifeless: I see that you tend to close the main one when the first fix is done.
<lifeless> wgrant: I'd just like it to be a little easier for ursula and diogo and others that don't have the entire callgraph internalised, to see that there are related things.
<lifeless> wgrant: What I do - yes. Uhm, I generally do that because the data is changed by the first fix.
<lifeless> wgrant: like, for bugtask:+index it will be 7 or 8 or more distinct fixes before we're flat
<lifeless> these are different to use case bugs where there is a specific use case with lots of rationale etc
<wgrant> True.
<wgrant> But I treat them differently if a user has filed and subscribed to the timeout.
<wgrant> Because the bug is not just for us.
<wgrant> It is for them too.
<lifeless> wgrant: I'm happy either way
<lifeless> like i say, what I want is better connections between what folk see in the oops reports and the open bugs
<lifeless> so that its easier to avoid duplicates and overlappying work
<lifeless> wgrant: why are you using ISlaveStore ?
<wgrant> lifeless: getAllLanguages already did.
<lifeless> so that leads to two Person objects in memory, for instaance.
<wgrant> Indeed.
<lifeless> don't destabilise things, but generally ISlaveStore should never be used. Ditto IMasterStore
<wgrant> We often use ISlaveStore...
<lifeless> we've had traumatic timeouts because of that.
<wgrant> Oh?
<lifeless> yeah
<lifeless> eager load from one store, get object from the other, dereference.
<wgrant> Ah.
<lifeless> we should, I think, change things to only ever have one store connected.
<wgrant> That sounds reasonable.
<wgrant> I shall change this to use the default store, as there is no clear benefit, you object, and it makes tests easier.
<lifeless> wgrant: there is a risk something else will barf at you from ec2. So its up to you whether you change this or not.
<wgrant> lifeless: If it does then I will kill it :)
<lifeless> :)
<lifeless> ok, two timeout fixes playing today, and a third landed in the morning.
<lifeless> not too bad.
<lifeless> stub: I'd like a brief voice chat about fast db deploys; got some time?
<wgrant> I want Python 2.7 :(
<StevenK> Why?
<wgrant> Set literals, among other things.
<StevenK> You can't import it from __future__?
<wgrant> No, they are a new backport in 2.7.
<jtv> wallyworld: approved with notes
<lifeless> jtv: could you comment on and triage bug 729520 - i'm out of my knowledge zone
<_mup_> Bug #729520: Translations changed in Ubuntu should be marked as such, not as current <Launchpad itself:New> < https://launchpad.net/bugs/729520 >
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/690356
<_mup_> Bug #690356: The "xubuntu" packageset is missing in the lp.packagesets collection (LP API) <bugjam2010> <lp-soyuz> <packagesets> <qa-needstesting> <Launchpad itself:Fix Committed by wgrant> < https://launchpad.net/bugs/690356 >
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: Sorry, was at dinner, qa'd it, noticed you'd already tagged it, then saw your ping.
<wgrant> Sort of the wrong order.
<lifeless> ;)
<stub> lifeless: back
<lifeless> stub: hey, cool
<lifeless> stub: now good ?
<stub> lifeless: irc or skype if you want to put up with the power tool noise
<lifeless> I heard you say hello
<stub> lifeless: no sound from you - checking audio here
<lifeless> kk
<wallyworld> jtv: thanks for the review and tip about assertContentEqual (which i didn't know about). i'll make the necessary tweaks
<jam> hi wgrant
<jam> and lifeless
<jtv> lifeless: henninge needs to know about that bug.
<wgrant> Morning jam.
<wgrant> jam: We may be LOSAless after the deployment :/
 * henninge pumps up the volume to hear pings
<henninge> jtv: Hi! which bug?
<adeuring> good morning
<jtv> henninge: hi!  bug 729520
<_mup_> Bug #729520: Translations changed in Ubuntu should be marked as such, not as current <Launchpad itself:New> < https://launchpad.net/bugs/729520 >
<henninge> Moin abel
<jtv> and good morning adeuring :)
<jam> wgrant: well that sucks. I was hoping to hear better news. I'm a bit surprised that they trust general rollouts without a LOSA available.
<jam> Is it just that they make sure to have one *during* the deploy? And assume they will catch things then?
<jml> lifeless: hello
<wgrant> jam: There will be one during the deploy.
<wgrant> jam: And hopefully spm will be well again by then :)
<jam> wgrant: I know you have to have one *during*. My point is why there isn't one *right after*, in case things go wrong. They just trust that the only problems will be the deployment itself?
<lifeless> jml: hello
<lifeless> jml: skype?
<jml> lifeless: sure. just call me.
<lifeless> jml: says you re offline
<jml> lifeless: lies
<jml> actually
<jml> no
<jml> lifeless: it's having trouble signing in
<jtv> henninge: read the bug?
<henninge> jtv: yes, the upstream project has a template ...
<henninge> so that behavior sound correct
<lifeless> jml: should I call your phone or mobile then?
<jtv> henninge: Ahhhâ¦ I hadn't realized there was upstream translation!
<jml> lifeless: ok.
<henninge> jtv: but it's a testcase I created two years ago. Disabled it now.
<jtv> Oh!
<henninge> jtv: and I just realized I that that won't help because the code does not check if a template is deactivated ...
<jtv> Oops.
<henninge> jtv: you actually pointed me to that but I forgot to implement it.
 * henninge goes to move template out of the way completely.
<jtv> henninge: are you sure though?  Maybe you used one of those methods that ignore deactivated templates.
<henninge> jtv: no, it's a storm query. pretty obvious
<jtv> oic
<henninge> translations/utilities/translationsharinginfo.py
<jtv> henninge: I'm on a feature rotation now, but robert needed someone to comment on & triage that bug.
<henninge> jtv: I will leave a comment.
<jtv> Great, thanks!  I'm very relieved that this is not a big design problem.
<henninge> jtv: what was that obsolete series called again, that we moved template like that to?
<jtv> obsolete-junk?
<henninge> "too many matches" ...
<henninge> jtv: but that's it
<jtv> The pyromaniac in me whispers "there's no such thing as too many matches when it comes to obsolete junk"â¦
<henninge> ;)
<lifeless> jml: https://launchpad.net/ubuntu/+source/apport
<jml> lifeless: lp:principia
<jml> lifeless: sound debugging
<lifeless> can you hear me?
<lifeless> if so, kick twice
<jam>  /kick /kick
<wgrant> jam: Will you be able to look at bug #726985 soonish, or should someone else? It'll be in the top 10 exceptions on Wed.
<_mup_> Bug #726985: codebrowse OOPSes with GeneratorExit when connection closed early <Launchpad itself:Triaged> < https://launchpad.net/bugs/726985 >
<jam> wgrant: if that's a request, sure
<jam> wgrant: you create it by doing a HEAD on a 404 page, right?
<jam> do we know how they are getting OOPS in production?
<wgrant> jam: It seems to always be large files in production.
<wgrant> Hmm. Always binary URLs, actually.
<wgrant> Which shouldn't be rendering...
<wgrant> Oh, it's /download, so that's OK.
<jam> wgrant: so basically, people who are ^Cing binary downloads in the middle of streaming are causing this OOPS
<wgrant> jam: I suspect so.
<jam> no big deal, just trying to understand
<jam> easy to reproduce a couple of ways, was hoping for something concrete
<jam> 90% of loggerhead doesn't stream
<jam> it directly calls start_response().write()
<jam> rather than 'yield' the content
<wgrant> Ah, that makes more sense now.
<dpm> hi henninge_, the unity-2d guys are implementing i18n and using LP for translations. Unity 2D is written in Qt but using a conversion of the Qt format to gettext for translations, so they can be done in LP. A question has come up though, and it is whether qt-format-plurals are supported in LP. I think they should be, as they are supported in gettext itself, but I just want to confirm. They look like this:
<dpm> http://paste.ubuntu.com/576943/
<LPCIBot> Project db-devel build #424: FAILURE in 5 hr 16 min: https://hudson.wedontsleep.org/job/db-devel/424/
<henninge> dpm: I take it that that is how it is exported to gettext.
<dpm> henninge, yeah
<henninge> dpm: because it looks just like gettext
<dpm> exactly, the only "difference" is that normally you don't get the %n in translations coming from most gettext packages
<dpm> I know we support this in #kde-format IIRC, but I want to double check we support it for qt-format-plural as well, so that I can give the Unity guys a proper answer
<henninge> dpm: the %n has nothing to do with plurals.
<henninge> that's c-format
<henninge> dpm: so to be correct gettext, the flags schould contain the "c-format" flag.
<deryck> Morning, all.
<dpm> henninge, hm, no, I don't think that's c-format, hence the question. c-format strings contain specifiers like %d or %s, but not %n, I think
<dpm> that's why they use a separate format specifier in gettext
<henninge> dpm: argh, got me!
<dpm> :-)
<henninge> yes, you are right
<henninge> dpm: But that does not matter very much for the plurals display in launchpad.
<henninge> s/very much/at all/
<dpm> henninge, yes, that's right, so I guess I should rephrase the basic question as whether qt-format is supported (nevermind about qt-plural-format)
<henninge> dpm: we use the standard gettext library to check the format of such strings. I think the only problem could be that bad translation strings would not be detected.
<henninge> "bad" meaning a mismatch in the identifiers.
<dpm> henninge, ok, gotcha, thanks
<henninge> .. if the gettext library does not know qt-format.
<dpm> it does, it does, it does: http://www.gnu.org/software/gettext/manual/gettext.html#qt_002dplural_002dformat
<dpm> err, copy paste fail, too many "it does" there
 * jml -> lunch
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge, benji | https://code.launchpad.net/launchpad-project/+activereviews
<gmb> Hi henninge, benji: Can I get a review for https://code.launchpad.net/~gmb/launchpad/make-sub-link-nothing-aware-bug-721410/+merge/52409 please?
<henninge> gmb: otp atm, lunch after that, stand-up after that. I can look at in 90 min.
<benji> gmb: I'll take it unless henninge is overcome with the need to do so.
<benji> looks like I'm the lucky winner
<gmb> benji: Cool, thanks.
<benji> gmb: review done
<gmb> benji: Thanks
<benji> np
<gmb> benji: Aha, clever irony in review comments; I like it.
<benji> :)
<sinzui> jcsackett: ping
<jcsackett> hello sinzui.
<sinzui> Will you be watching #launchpad this morning?
<jcsackett> sinzui: i will; did my setting of channel info not take?
<jcsackett> i see that it did not.
<sinzui> jcsackett: maybe topics are knackered for me again
<jcsackett> sinzui: no, it looks like i didn't properly save the change to info.
<deryck> henninge-lunch, wanna chat for a couple minutes now?
<henninge> deryck: sure
<henninge> deryck: https://launchpad.dev/netapplet/+configure-translations
<sinzui> leonardr: do you have time to review https://code.launchpad.net/~sinzui/lazr.restful/json-not-xhtml-0/+merge/52141
<leonardr> sinzui, sure
<bigjools> heh, thanks benji :)
<bigjools> didn't get around to asking for the review, but you got to it anyway, fantastic
<leonardr> sinzui: and maybe you'll have some insights into https://code.launchpad.net/~leonardr/launchpad/bug-271029/+merge/52423? otherwise i'll give it to ocr
<benji> I'm like the genetically engineered kid's immune system in that episode of Star Trek: TNG, I don't wait for the microbes to come to me, I go out and get them. ;)
<bigjools> heh
<leonardr> sinzui, i don't understand the change you made to test_webservice.py. did you just disable the test?
<leonardr> i think you should change it so it tests the actual behavior
<leonardr> unless this is something we plan to bring back soon
<leonardr> apart from that, it looks right
<sinzui> leonardr: I did disable it. if you recall I suggested to change the name to start with disabled instead of test. I think I mis understood you I thought you said to remove test_ only
<leonardr> sinzui: no, i meant to change the behavior of the test. unless we're going to go back to the old behavior someday, the test should be changed or removed
<sinzui> I will change the behaviour
<jml> sinzui: did you want to have a call today?
<sinzui> jml: I have nothing to talk about today
<jml> sinzui: good good :)
<sinzui> :)
<leonardr> sinzui, do you want to take a look at my branch or shall i give it to henninge/benji?
<sinzui> leonardr: I can look at your branch now
<leonardr> great, i think you will have good opinions on the topic
<leonardr> argh, my push is still happening? what's the deal?
<leonardr> well, i'll tell you when it finishes. it seems almost done
<sinzui> leonardr: I updated the test based on the subsequent test in the module: https://code.launchpad.net/~sinzui/lazr.restful/json-not-xhtml-0/+merge/52141
<leonardr> ok
<leonardr> i seem to be pushing the entire launchpad tree for some reason
<leonardr> sinzui: r=me
<sinzui> thank you
<dobey> leonardr: ping
<leonardr> dobey, hi
<dobey> leonardr: hey, am having some trouble with launchpadlib 1.9.7
<dobey> 2011-03-07 10:30:07 ERROR    An error occurred trying to merge lp:ubuntuone-storage-protocol: 'NoneType' object has no attribute 'landing_candidates'
<dobey> leonardr: ^^ tarmac keeps failing with that now. but it works fine with 1.6.2
<leonardr> dobey: can you find the launchpadlib code in tarmac and paste it to me?
<dobey> so the aprt that's failing is doing "for entry in lp_branch.landing_candidates"
<leonardr> dobey: how is lp_branch set?
<dobey> leonardr: lp_branch = self.launchpad.branches.getByUrl(url=branch_url)
<leonardr> also, my generic advice: run 'import httplib2; httplib2.debuglevel=1' at the beginning of the code and paste me the http requests made
<leonardr> and where does branch_url come from?
<leonardr> if the versions are mismatched, that'll probably fail
<dobey> the config. is "lp:foo"
<leonardr> ok, no version there
<leonardr> show me the http requests being made. then add a "version='devel'" argument to the launchpad constructor and see if it works. then try the same with version="beta"
<leonardr> sinzui: https://code.launchpad.net/~leonardr/launchpad/bug-271029/+merge/52423 is ready to review
<leonardr> sinzui: i took out the part i really wanted your opinion on, so i think i can give it to henninge or benji
<leonardr> let me know if you'd like that
<dobey> leonardr: oh hrmm
<dobey> leonardr: it seems that launchpadlib 1.9.7 is trying to connect to staging by default
<dobey> Host: api.staging.launchpad.net
<leonardr> dobey: i believe that's true of older versions as well
<leonardr> but, one thing i mentioned in the upgrade email is that all clients need to start explicitly specifying their host
<leonardr> to avoid exactly this problem
<dobey> we WERE doing that already
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: benji | https://code.launchpad.net/launchpad-project/+activereviews
<dobey> leonardr: we're using LPNET_SERVICE_ROOT which is https://api.launchpad.net/
<leonardr> hm, so you are passing in production and it's being ignored
<gary_poster> henninge, benji: I'd approaciate a review of https://code.launchpad.net/~gary/launchpad/bug723999-2a/+merge/52427 when one of you has some time available.  Fair warning: I'm afraid I may be prolific in my review requests today.  I certainly intend to be, at least.
<leonardr> how is the Launchpad object being created?
<leonardr> dobey -^
<gary_poster> oh, well, nm on henninge :-P :-)
<leonardr> is it using login_with? get_token_and_login?
<dobey> Launchpad(credentials, SERVICE_ROOT, self.config.CACHE_HOME)
<benji> gary_poster: I'll take it.
 * henninge looks up prolific
<henninge> gary_poster: sorry
<gary_poster> np henninge! :-)
<dobey> leonardr: it uses get_token_and_login() only if there are are no existing credentials
<dobey> leonardr: looks like perhaps the API was broken for Launchpad() instantiation
<gary_poster> thank you benji.  to state the obvious, I know you are busy, so if you don't get to all the branches I throw your way today, of course I will understand.
<benji> sure
<dobey> ugh
<sinzui> leonardr: lp thinks that you more than just removed the all the changes
<leonardr> dammit!
<dobey> leonardr: so it's because new non-kwargs were added to the init, and tarmac was relying on position for 2 things that were kwargs. :-/
<abentley> leonardr: lifeless has asked me to ensure that the stuff I export is only available on the devel version.  Is there a way to prevent webservice_entries from being exported in older versions?
<abentley> leonardr: And for exported attributes, does this mean I need to declare that they are not exported in every other version?
<gary_poster> benji, a large but hopefully easy branch (it just moves tests from one file to another): https://code.launchpad.net/~gary/launchpad/bug723999-2b/+merge/52429
<benji> gary_poster: k
<leonardr> dobey: yes, i'm pretty sure i talked to the tarmac developer about this
<leonardr> tarmac should not be storing its own credentials anymore
<dobey> rockstar: ^^ ?
<dobey> leonardr: the credentials storing isn't the problem afaict. it's the API breakage. :-/
<leonardr> dobey: the api breakage happens to apps that try to store their own credentials and instantiate Launchpad objects from them
<leonardr> if i had known about this, i wouldn't have changed the constructor arguments, but now that it's broken i'd rather fix the apps, since they need to change anyway
<rockstar> leonardr, dobey and I are "the tarmac developer."  :)
<rockstar> leonardr, we stored our own credentials because launchpadlib was storing them in a place that kees said was a bad place...
<leonardr> rockstar: ah, tarmac isn't in ubuntu. that's why i didn't contact you
<rockstar> leonardr, yeah, I missed the boat in Natty.
<leonardr> rockstar: ok, the new system is kees-approved, so you can use it without worry
<rockstar> leonardr, ack.
<rockstar> dobey, I still have leonardr's email with the changes we need to make.
<dobey> what does the new system do?
<leonardr> abentley: no, there's no way to prevent an entry from being exported. i need to add that
<abentley> leonardr: I see.
<leonardr> for exported attributes, you need to set exported=False by default, and pass in a version setting, something like [('devel', dict(exported=True)]
<leonardr> i also need to add syntactic sugar to make that easier
<leonardr> basically, the original design was that almost everything would be available in earlier versions unless it caused a huge problem
<leonardr> and the new design is that nothing is available in earlier versions unless it was originally present there
<abentley> leonardr: Hmm.  From the docs, I thought that exported=False set the first version, not a default.
<leonardr> abentley: it's also a default, because later versions inherit from the first version
<leonardr> so exported=False in beta means exported=False in 1.0, and it would mean exported=False in devel except for the override
<abentley> leonardr, aha!
<leonardr> dobey: the new system stores credentials in the gnome keyring/kde wallet
<dobey> leonardr: what if there is no keyring?
<leonardr> dobey: then it's stored encrypted on disk
<dobey> ok
<leonardr> dobey: there is something of an unchallenged assumption that IF you are in a gui environment you have some kind of keyring available
<leonardr> since decrypting the encrypted file requires that you type in a passphrase into the current terminal
<dobey> leonardr: tarmac is not meant to be run generally in a gui environment. it is a service run by cron.
<leonardr> dobey: in that case, you want to use the 'service run by cron' mechanism
<leonardr> that's in the email as well
<dobey> leonardr: so any user interaction is not acceptable for it
<leonardr> there will be none
<dobey> ok, i don't have the email
<leonardr> you will pass in the location of an unencrypted credentials file, and launchpadlib will read/write it
<leonardr> so you can probably use the same file you use today
<leonardr> i'll forward oyu the email
<dobey> and we will still need to support both methods in tarmac
<dobey> i think
<leonardr> dobey: what do you mean by both methods? when there are existing credentials and when there aren't?
<dobey> leonardr: i mean "the current way tarmac works"
<dobey> leonardr: as we often run it on lucid also
<leonardr> dobey: if you switch to passing in the unencrypted credentials file it should work on both lucid and natty
<leonardr> sinzui: diff is pasted to https://code.launchpad.net/~leonardr/launchpad/bug-271029/+merge/52423
<leonardr> dobey: if credentials_file doesn't work for you, let me know
<benji> gary_poster: both of your branches are reviewed
<gary_poster> awesome, thanks benji.
<gary_poster> more coming ;-)
<dobey> leonardr: ok
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #425: FIXED in 4 hr 50 min: https://hudson.wedontsleep.org/job/db-devel/425/
<sinzui> thanks leonardr
<sinzui> leonardr: r=me, take a look at bug 579602 before you land. I think you fixed it
<_mup_> Bug #579602: Trying to getBranches of a source package using the API oopses with ForbiddenAttribute <api> <lp-code> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/579602 >
<bigjools> sinzui: we're still doing ui reviews, right?
<sinzui> bigjools: voluntarily.
<bigjools> ok, thanks.
<bigjools> who's a valid reviewer for that nowadays? :)
<sinzui> bigjools: um, I think it is still me and henninge. I have been passing UI reviews to huwshimi
<sinzui> bigjools: you can always get two engineers to agree that the UI is good
<bigjools> ah I will get rvba to push his review to huwshimi
<leonardr> sinzui: i think i stopped the oops, but the bug isn't that there's an oops--i think that code should work
<sinzui> leonardr: okay
<LPCIBot> Project devel build #512: FAILURE in 5 hr 4 min: https://hudson.wedontsleep.org/job/devel/512/
<LPCIBot> Project windmill build #21: FAILURE in 1 hr 4 min: https://hudson.wedontsleep.org/job/windmill/21/
<LPCIBot> Project db-devel build #426: FAILURE in 43 min: https://hudson.wedontsleep.org/job/db-devel/426/
<gary_poster> benji, https://code.launchpad.net/~gary/launchpad/bug723999-2c/+merge/52448 is ready for you (except for the diff, which will presumably arrive anon), but I'm going to go get some lunch.
<benji> k
<jcsackett> bigjools: are you (or can you point me to someone) to talk to in regards to https://answers.launchpad.net/launchpad/+question/148005 ?
<dobey> rockstar: https://code.launchpad.net/~dobey/tarmac/login-with/+merge/52451
<leonardr> i need some help from someone who knows launchpad bugs. preparing a pastebin...
<leonardr> http://pastebin.ubuntu.com/577101/
<leonardr> i changed createTask so that it calls valid_upstreamtask to validate, and that's breaking some test setup
<leonardr> is the test setup technically invalid, or did i make a bad change?
<leonardr> sinzui, maybe you know, i think you helped me with this in the first place
 * sinzui looks
<leonardr> sinzui: i changed the code so that the invalid bugtask was created first, and got the same error
<leonardr> it seems okay to create another bugtask for a product if the earlier one was marked invalid?
<sinzui> leonardr: I suspect you added a validate method, but our ORM needs an exception to set things up. So we need to exit a method early or return True for validate
<sinzui> This happened to me recently. What field was I changing?
<sinzui> ah description fields cannot contain only white-space
<leonardr> sinzui: i added a _call_ to a validate method within createTask
<leonardr> i'm not sure what you mean by "exit a method early..."
<sinzui> leonardr: when working on the StrippableText field that lots of things descend from, I broken the initialisation of a lot of objects because the ORM create empty objects, then sets the data. The field should only validate or normalise data when when values are set (not inited). I used a check for current state and the new value to be certain I should proceed. I know there are fields that set a marker in init to show nothi
<sinzui> ng is set.
<leonardr> sinzui: i'm way out of my depth here, but it seems that this validation _should_ be run upon creation
<leonardr> it seems like the test is doing something by manipulating the orm that should not be possible
<leonardr> if the rules are to be applied consistently
<sinzui> leonardr: if you mean creation means all the data is loaded, then yes
<sinzui> storm/sqlobject creates an empty object, then puts data in the fields, and they fire in sequence...before all the data is there to validate an invariant state
<sinzui> leonardr: can you pastbin your change?
<sinzui> pastebun
<sinzui> I give up
<leonardr> sinzui: sure
<abentley> sinzui: Pastbin was a rejected name for The History Channel :-)
<leonardr> here's te whole thing: http://pastebin.ubuntu.com/577120/
<leonardr> the relevant code is on line 166
<leonardr> i think the problem is simpler than you're saying
<leonardr> i started calling a certain method from the create method
<sinzui> They rejects The Nazi Channel, but decided to run Nazi only programming anyway
<leonardr> this method says "you can't have two bugtasks for the same bug for the same product, no exceptions"
<leonardr> so the test that creates two bugtasks for the same bug for the same product fails, even though one of the bugtasks is invalid
<sinzui> The International History channel's real name is Alien Encounters Channel
<sinzui> leonardr: well, I would expect a failure in that case. The DB constraint does not look at status, it only looks at bug, distro, dseries, sourcepackagename, product, and pseries. Maybe that test is totally bugus
<sinzui> excellent, I worked totally bogus into a sentence.
<leonardr> sinzui: here's just the test that fails: http://pastebin.ubuntu.com/577123/
<sinzui> The example i see is distro and then a distro + source package
<leonardr> you think that deserves to fail?
<leonardr> here's the original, not modified by me:
<leonardr> http://pastebin.ubuntu.com/577124/
<sinzui> leonardr: these bug tasks are different
<leonardr> sinzui: am i calling valid_upstreamtask with the wrong argument?
<leonardr>  target = product or productseries or distribution or distroseries
<leonardr> should i be going in the other direction?
<leonardr> target = distroseries or distribution or productseries or product
<sinzui> leonardr: but I can imagine that if validate is called after the distro is set, but before the spn is set, you will get the error
<leonardr> sinzui: i don't see what this has to do with validate at all
<leonardr> the exception is raised by a method that i call explicitly
<leonardr> i think we are talking past each other
<sinzui> I have not seen the lines where you changed to cause the error yet
<sinzui> where do you call valid_upstreamtask, and what is the context lines before and after it
<leonardr>         if not milestone:
<leonardr>             milestone = None
<leonardr>         # Raise a WidgetError if this product bugtask already exists.
<leonardr>         target = distroseries or distribution or productseries or product
<leonardr>         valid_upstreamtask(bug, target)
<leonardr>         if not bug.private and bug.security_related:
<leonardr> the BugTask object is not created until much further down in the method
<leonardr> here's the entire method with plus signs by the code i added
<leonardr> http://pastebin.ubuntu.com/577131/
<lifeless> jml: I forgot two things; paralel test lep & high frequency db deploys that I wanted to speak with you about
<sinzui> what no, spn? that definitely will cause this error. The second example in the test is not getting the sourcepackagename=ubuntu_evolution.sourcepackagename that was passed
<leonardr> what does spn stand for?
<sinzui> source package name
<sinzui> A bug in a distro can affect dozens of packages, so will be represented by dozens of bug tasks
<sinzui> all ubuntu, may be all the same distro series, but each has a unique source package name
<leonardr> sinzui: the test has changed back and forth quite a bit since i started talking to you. let me paste the original version and ask you to walk me through it
<leonardr> http://pastebin.ubuntu.com/577132/
<leonardr> so, you are saying that the call to createTask(sourcepackagename=ubuntu_evolution.sourcepackagename)
<leonardr> at some point the source package name is lost
<sinzui> leonardr: I still see sourcepackagename=ubuntu_evolution.sourcepackagename in all examples of the test, but the fragment of code does
<sinzui>     target = distroseries or distribution or productseries or product
<sinzui>     target = distroseries or distribution or productseries or product
<leonardr> so you're saying that sourcepackagename is also an acceptable target
<sinzui> clearly target is a composite maybe we need it to be tuples
<leonardr> sinzui: it's not clear to me at all. are you saying that the 'target' might be some combination of, eg. sourcepackagename and distroseries?
<sinzui> (ubuntu, ), (ubuntu, natty), (ubuntu, natty, evolution), (ubuntu, evolution) (evolution-project) are all valid combinations for bugtasks, but I see target is just one items by order of precedence
<sinzui> ^ leonardr that is what the database sees for the unique constraint
<leonardr> sinzui: valid_upstreamtask takes a 'product' and invokes searchTasks() on the product
<leonardr> i'll paste the code
<leonardr> http://pastebin.ubuntu.com/577133/
<leonardr> it sounds like this code is too specific to be called from createTask
<sinzui> I suspect I know what is a mis before even looking
 * sinzui looks
<sinzui> yes, we are checking one of many conditions
<sinzui> leonardr: I think the issue many years ago was that the four types were assign targets to have enough information to know everything, but that is not true for an SPN. It is just a name, we do not know what distro it is, and we do not know if the distro publishes it
<sinzui> leonardr: we can start making sens of this by fixing the signature: valid_upstreamtask(bug, product) => valid_upstreamtask(bug, bug_target)
<leonardr> ok, i understood that
<sinzui> yuck. we cannot search an SPN's tasks
<leonardr> sinzui: do you think it would be possible now to use the website to create multiple bugtasks for a single spn?
<leonardr> or would that finally be caught at the database level?
<sinzui> leonardr: I think empty() means return early. If this is a distro or distro series, we need to see if any of the bug tasks have the same sourcepackagename as the spn passed
<sinzui> leonardr: we will get a DB ConstraintViolationError after the call without a stack trace. We do need to verify the spn is unique
 * sinzui looks for SPN bug target
<sinzui> leonardr: the signature is a real issue. I *think* we can get past the issue by getting a DSP or SP for the D or DS respectively. Include that in the target = sp or distroseries or distro line
<sinzui> So while I *do* think we need to fix the signature to say bug_target, the real fix is to get the correct target in the callsite
 * sinzui hacks
<sinzui> leonardr: I think this is what createBugTask needs to do: http://pastebin.ubuntu.com/577145/
<leonardr> sinzui: ok, i think i have something similar, i'll check
<sinzui> I wonder if we have an adapter
<leonardr> yeah, i have something equivalent
<leonardr> i'll paste what i have
<leonardr> sinzui: http://pastebin.ubuntu.com/577149/
<sinzui> leonardr: +1
<leonardr> that makes _that_ test work, let me see about the others
<leonardr> looking good...
<leonardr> ok, now down to 2 test failures...
<gary_poster> benji, getStructuralSubscriptionsForPerson was not suppsed to be in this branch.  Sorry for the confusion.  It goes in the next (and hopefully last) branch
<lifeless> matsubara_: ping
<gary_poster> I'm going to remove ans push
<lifeless> Ursinha: ping (the Bug:xyz patch didn't seem to be live on qatagger last night)
<benji> gary_poster: ok; just about done reviewing that branch
<gary_poster> cool thanks benji
<leonardr> sinzui, i wrote this code, but i don't remember what it means anymore. can you sanity check it for me?
<leonardr> http://pastebin.ubuntu.com/577158/
<benji> gary_poster: review done, including a solution to your lint mystery
<leonardr> it's not raising an exception where it should
<gary_poster> sweet, thanks benji!
<leonardr> but i don't know if the 'conjoined_bugtask' actually is a conjoined bugtask, because i don't know what a conjoined bugtask is
<sinzui> leonardr: conjoined_bugtask is a temporary state where a task's series (distro or product) happens to be the development series
<lifeless> its a series specific task that also has a project scope task
<lifeless> so there are two related tasks and its shown as one in the UI
<lifeless> its a significant chunk of complexity :(
<sinzui> leonardr: This is very esoteric knowledge. For ubuntu, a task is conjoined only for 6 months
<sinzui> leonardr: since you use product.development_focus, I expect this test to work if there is also a bug for the product
<leonardr> it doesn't looks like there is
<leonardr> conjoined_master is None
<deryck> abentley, thanks for sending that packaging permissions email.
<sinzui> leonardr: I think this subtle change will work: target=bugproduct.development_focus
<sinzui> target=bug.product.development_focus
<abentley> deryck: np
<sinzui> I think the factory created a product for the bug, we want to reuse it
<leonardr> ahg
<sinzui> leonardr: I think you can make the the reviewer of your branch. I think I can now skip the cover letter and just read the code
<leonardr> ok, we just have some failures in patches-view.txt...
<lifeless> deryck: had you seen http://www.pivotaltracker.com/features
<deryck> lifeless, no, I haven't seen that.  looking now.
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: benji | https://code.launchpad.net/launchpad-project/+activereviews
<deryck> lifeless, ah, interesting.  it's not what I thought on first glance.  It's basically a scrum board, right?
<lifeless> looks like
<lifeless> bug cast as a bugtracker replacement (which they effectively are)
<lifeless> I -love- the infinite resolution on ordering
 * lifeless wants this for bug priority.
<lifeless> drag n drop, done.
<deryck> yeah, that's nice.
<lifeless> abentley was suggesting something like this for lp way back
<deryck> but their model is a set of cards.  not bugs.
<lifeless> deryck: right, it is a different take on the whole problem
<jcsackett> my last job briefly used pivotal. it was a nice tool.
<deryck> lifeless, right.
<deryck> yeah, it looks like a nice tool.
<leonardr> sinzui: ok, help me out one more time
<deryck> jcsackett, did you use it as a bug tracker?
<leonardr> patches-view contains this code:
<leonardr>     >>> make_bugtask(
<leonardr>     ...     bug=bug_a, target='hoary', target_is_distroseries_name=True)
<jcsackett> deryck: eh, sort of. more of an issue tracker, i suppose?
<leonardr> that looks up the 'hoary' distro series and calls factory.makeBugTask
<jcsackett> if you're doing things in the agile card mode, it's less about bugs and more about "user stories."
<deryck> jcsackett, yeah, it looks like project tracking, more than bug tracking to me.
<leonardr> but factory.makeBugTask doesn't like being passed a distro series
<leonardr>         if IDistroSeries.providedBy(target):
<leonardr>             # We can't have a series task without a distribution task.
<leonardr>             self.makeBugTask(bug, target.distribution)
<jcsackett> deryck: yes.
<leonardr> so instead of creating a bugtask for 'hoary', we create one for 'ubuntu', and we trigger the validation error
<deryck> like what we use Kanban for.
<lifeless> thats the conjoined master bit
<jcsackett> deryck: similarly; though we're sort of using bugs as project tracking as well.
<deryck> jcsackett, oh we definitely are.
<leonardr> lifeless: i think it's supposed to add the conjoined master and then call addTask *again* to add the slave
<leonardr> the problem is that adding a conjoined master when there's already a bugtask for 'ubuntu' now raises an exception
<lifeless> what did it do before (and what bug are you working on , to give me context)
<leonardr> lifeless: the context is my fix to bug 106338: https://code.launchpad.net/~leonardr/launchpad/bug-106338
<_mup_> Bug #106338: Editing a bug targeted to a release crashes if you directly edit the untargeted task  <api> <lp-bugs> <oops> <Launchpad itself:Triaged by leonardr> < https://launchpad.net/bugs/106338 >
<leonardr> i added to createTask a call to valid_upstreamtask(), so that validation code previously called only in the view would be called everywhere
<sinzui> leonardr: I think the issue here is the factory is wrong. It could be smart enough to do the right thing. It onyl has the features that engineers needs at the time
<leonardr> sinzui: ok, that's what i was thinking too
<leonardr> so we should see if the master already exists?
<sinzui> Lots of factory methods were creating objects with impossible series. I fixed them
<sinzui> leonardr: yes, the factory should check and make a decision.
<lifeless> I concur
<leonardr> ok, help me out. can i adapt the code from valid_upstreamtask to see if there's already a bug targeted to a given target, or is there a simpler way?
<leonardr> i'll write some cheap code and you can tell me how to fix it
<lifeless> leonardr: 'try: ... except:'
<lifeless> (though you'll want an actual type in the except clause)
<sinzui> I do not this the try except will work because the exception will be a DB IntegrityError contstaint violation. We need to look before we leap with the ORM
<leonardr> my cheap code goes through the existing tasks
<lifeless> sinzui: oh :)
<lifeless> sinzui: ok
<lifeless> leonardr: there is a getBugTask(target) method on bug.
<leonardr> let's try it...
<jcsackett> leonardr: can i bug you for a sec regarding 404s on lplib?
<leonardr> jcsackett, sure
<sinzui> benji: do you have time to review a small change to address a project review nuance: https://code.launchpad.net/~sinzui/launchpad/hide-license-info/+merge/52467
<benji> sinzui: sure
<jcsackett> leonardr: so i am looking at bug 701246
<_mup_> Bug #701246: missing bug dereference generates OOPS <oops> <Launchpad itself:Triaged by jcsackett> < https://launchpad.net/bugs/701246 >
<jcsackett> it looks as though the problem is just that a NotFound that gets raised in lp doesn't go back to lplib.
<jcsackett> i have tried https://pastebin.canonical.com/44375/ as a fix.
<jcsackett> while that works, leonardr, i'm unconvinced its the best (or correct) solution.
<jcsackett> leonardr: that change is in _handle_next_obj, to save you a trip into the code tree.
<leonardr> jcsackett: ok, give me am inute to finish what i'm doing
<jcsackett> leonardr: sure.
<lifeless> jcsackett: +1 AIUI
<jcsackett> lifeless: cool. my major concern is i thought we should a) avoid expose() and b) i'm frightened of the publisher. :-P
<leonardr> sinzui, lifeless: going to paste my code and look at jcsackett's code. for some reason, valid_upstreamtask is raising an exception for 'hoary' even though bug.getBugTask(hoary) is returning None
<benji> sinzui: review done (https://code.launchpad.net/~sinzui/launchpad/hide-license-info/+merge/52467)
<sinzui> thanks benji
<benji> np
<leonardr> lifeless, sinzui: http://pastebin.ubuntu.com/577173/ looks good to me, don't know why it sometimes fails as i mentioned above
<leonardr> jcsackett: at the very least, you need to do expose(NotFound(self.context, name), 404)
<leonardr> otherwise you will start giving out 400 errors
<jcsackett> leonardr: ok.
<leonardr> i think it would make more sense to publish NotFound as 404 _in general_
<jcsackett> i sort of assumed it was, to be honest. :-P
<sinzui> leonardr: I think the code is creating confusion between master and slave. the distro is the slave, the series is the master.
<leonardr> jcsackett: maybe it's the fact that the exception is raised during traversal
<leonardr> you say that the code you have works? what behavior do you see?
<sinzui> We really need to rethink the value for conjoined relationships. They are ephemeral, limited to projects that develop with leading series, and the code is very expensive to work with
<jcsackett> leonardr: with make run_all going, if i connect to my dev instance via lplib, and run through the code shown in the bug, i get a 404 http error, and no oops is generated.
<leonardr> jcsackett: we do have special views registered for NotFound that should cause them to yield 404 errors, but i bet that code doesn't apply to exceptions raised during traversal
<leonardr> that's weird
<leonardr> if you put a weird error code like expose(..., 409), does that get passed through?
<sinzui> leonardr: the if ISourcePackage.providedBy block talks about series but we can have bugs in SPs without a series
<sinzui> maybe the whole paste represents an attempt to solve the case where the target implements ISeries
<jcsackett> leonardr: let me check.
<leonardr> sinzui: i was trying to solve the case where it tries to create a prerequisite target *of some kind*
<leonardr> there were three such cases
<leonardr> are you saying these cases are different, that in some cases the target is the master and in some cases the target is the slave?
<leonardr> i can change the variable names so i'm not talking about 'master_target'. i don't think i changed the actual meaning of the code
<leonardr> sinzui: see http://pastebin.ubuntu.com/577178/. the code is the same, except i check for a bug before creating one
<sinzui> leonardr: okay. I agree with what you are doing. Lp does require a PrimaryContext level bug before creating a series bug.
<leonardr> sinzui: i am in a situation where bug.getBugTask(bug_target) returns None, but valid_upstreamtask raises an exception
<leonardr> because bug_target.searchTasks(BugTaskSearchParams(user, bug=bug)) finds something
<leonardr> so i think bug.getBugTask(bug_target) does not tell the whole story
<sinzui> leonardr: by "sometimes fails" do you mean not reliably repeatable? maybe we need something like IStore(prerequisite).flush()
<lifeless> leonardr: getBugTask is trivial
<lifeless> leonardr: but note that it uses cached data.
<lifeless> leonardr: you may need to try: del get_propery_cache(bug).bugtasks; except AttributeError: pass
<leonardr> sinzui: i mean it fails when i don't think it should
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project windmill build #22: FIXED in 1 hr 12 min: https://hudson.wedontsleep.org/job/windmill/22/
<lifeless> leonardr: at the point where bug tasks are added
<leonardr> lifeless:trying that now
<lifeless> leonardr: bottom of Bug.addTask would be the place to add it
<lifeless> leonardr: a more complex version would be to add the new one to the tasks IFF the tasks attribute is already populated
<jcsackett> leonardr: actually, expose is telling me it only accepts on argument, which prevents doing expose(...,409). something trivial i've missed?
<jcsackett> s/on argument/one argument/
<leonardr> jcsackett: ah, it's because i haven't landed my branch that fixes this, because of these test failures
<leonardr> i got ahead of myself
<leonardr> but, if expose() is working at all, you should be getting 400 errors, not 404
<lifeless> leonardr: normal browsers will still see a 404, right ?
<leonardr> lifeless: browsers on the website will see a 404. if you hit the web service with a browser it will go through the lazr.restful publisher and you'll see what a non-browser client would see
<lifeless> leonardr: ok, and we map 404 to 400 for some reason here?
<leonardr> lifeless: no, i'm saying that jcsackett's change shouldn't be doing what he's seeing
<lifeless> ah cool
<leonardr> if expose() was working at all, it would be giving him a 400 error, not a 404
<lifeless> leonardr: would it be reasonable for NotFound to map to 404 in the webservice by default?
<leonardr> still getting the failures in patches-view. again, the problem is that attempting to create a bugtask for Hoary gives an error "there's already a bugtask for Ubuntu"
<leonardr> lifeless: yes, and that is what happens normally. i think the problem we're seeing is that the exception is happening in traversal, and lazr.restful doesn't catch those
<lifeless> ah
<lifeless> yes, that makes sense
<leonardr> lifeless: the end of createTask already contains 'del get_property_cache(bug).bugtasks'
<lifeless> leonardr: ok, well thats not it then :)
<jcsackett> leonardr: so, basically, what i'm seeing isn't possible?
<thumper> leonardr: mumble?
<leonardr> jcsackett: i don't understand how you're seeing it. i wouldn't say it's impossible, especialyl if you can turn it on and off easily
<jcsackett> leonardr: so, either way a 404 is returned; it's just that with expose, no OOPS is created.
<leonardr> jcsackett: oh, i see how that could happen
<gary_poster> benji, one last one, then the marathon is done.  Maybe you can fit it in, maybe not. :-)  https://code.launchpad.net/~gary/launchpad/bug723999-2d/+merge/52478
<benji> gary_poster: good timing, I was just about to resort to real work. ;p
<gary_poster> :-)
<jcsackett> leonardr: and basically killing the oops is the thing. given that context, this seem better?
<leonardr> jcsackett: my gut feeling is that your bug will simply go away once my branch lands
<jcsackett> leonardr: really?
<leonardr> yeah
<jcsackett> that will be twice that you make my work irrelevant, but i'm totally happy with that. :-)
<jcsackett> i'm going to move on to something else and revisit this post your branch landing.
<leonardr> ok, it'll probably be tomorrow
<jcsackett> leonardr: cool. can you link me the branch (mostly for my curiosity)?
<leonardr> jcsackett: https://code.launchpad.net/~leonardr/launchpad/bug-106338
<jcsackett> thanks
<kfogel> From sabdfl's latest blog post: "Nothing hugs quite like dholbach, though, and he's no hairy ape."  I'm sorry, but I just had to repeat that.
<gary_poster> :-)
<kfogel> We now return you to your regularly scheduled, uh, programming :-).
<lifeless> kfogel: hey, how goes it
<gary_poster> thanks for saying hi kfogel
<kfogel> hey lifeless -- okay.  Busy, but good.  You?
<kfogel> gary_poster: miss y'all!
<gary_poster> you too :-)
<lifeless> kfogel: insanely busy, very well - lp is going great guns, and lynne is 14 weeks now ;)
<kfogel> lifeless: congrats!!!
<kfogel> lifeless: and re that other thing, launchpad, I've been watching the blog and been seeing the progress with pleasure.
<lifeless> kfogel: thanks
<lifeless> wow
<lifeless> I think we're spending 12 seconds /rendering/ 150 bugtask rows.
<lifeless> https://bugs.qastaging.launchpad.net/++profile++show/ubuntu/+source/afflib/+bug/230350
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<leonardr> sinzui: ok, i'm in pdb right at the point where the error is about to happen
<leonardr> i'm about to call self.makeBugTask(bug, a)
<leonardr> a is the Distribution 'ubuntu', and the target from which i derived a is the DistroSeries 'hoary'
<lifeless> gary_poster: chameleon templates; can they mix n patch with the tal engine ones?
<leonardr> bug.getBugTask(a) is None
<leonardr> but, if i call makeBugTask, i'll get an error that there's already a task for 'ubuntu'
<gary_poster> lifeless, once we switch to chameleon, everything uses the chameleon engine, including current TAL stuff.
<lifeless> gary_poster: sure; I'm looking at bug 724033
<_mup_> Bug #724033: BugTask:+index timeouts - distributions and milestones being late evaluation loaded - repeatedly - on bug 230350 <qa-ok> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/724033 >
<gary_poster> and it is supposed to help with rendering nasties like the one you are mentioning
<lifeless> gary_poster: I've gotten the sql component down to 2 seconds
<gary_poster> 12334: suck
<lifeless> gary_poster: I'm wondering if I could migrate just the bug task row & edit forms - which are the bits repeated - to chameleon, assuming further investigation doesn't show it to be something simple like the view initialization being slow
<gary_poster> OIC
<mtaylor> lifeless: hey - whatever happened to those merge queue things rockstar was working on?
<lifeless> mtaylor: deferred
<mtaylor> lifeless: k. so it's not just that I missed them somewhere
<lifeless> mtaylor: not at all; we had too much work in progress. May do them when we overhaul our landing process
<gary_poster> lifeless, you could.  The only way I can think of doing it right now is having the view render the chamelon bits as a string, inserted into the TAL as structure
<gary_poster> does that make sense?
<lifeless> yes
<gary_poster> cool
<lifeless> thats how the rows are done already, more or less
<gary_poster> oh, ok, cool
<lifeless> <div tal:replace="structure context/bug/@@+bugtasks-and-nominations-table" />
<gary_poster> that would be a great test to see of chameleon actually buys us what we expect then, particularly if it is apples to apples
<leonardr> sinzui: i'm starting to think there's a problem with vaild_upstreamtask
<gary_poster> yeah
<leonardr> check this out
<leonardr> (Pdb) [bug.title for bug in bug_target.searchTasks(params)]
<leonardr> [u'Bug #16 in evolution (Ubuntu): "bug_a title"']
<_mup_> Bug #16: "Swedish" and "Swedish (Sweden)" should be the same language <lp-translations> <Launchpad itself:Fix Released by daf> < https://launchpad.net/bugs/16 >
<lifeless> and then inside it
<lifeless> <tal:bugtask-or-nomination  repeat="task_or_nom_view view/getBugTaskAndNominationViews"><tal:block replace="structure task_or_nom_view" />
<gary_poster> ew
<gary_poster> that may be a problem inherently
<lifeless> so it ends up calling __call__() on the view objects its manually created.
<gary_poster> though chameleon may make that better too, dunno
<lifeless> gary_poster: whats the problem you perceive there?
<gary_poster> well, it's a hunch, but I'd be interested to see if doing structure calls and inserting the output of another view within a tight loop is just bad.
<lifeless> your hunch is that inserting structure is slow ?
<mtaylor> lifeless: well, fwiw, I just wrote a hacky launchpadlib script to make a page showing me which of our brances are in which of our fake-queue-branches ... hopefully it's not causing too much activity :)
<gary_poster> I would expect this, from worst to best:
<lifeless> mtaylor: heh
<lifeless> mtaylor: patch LP!
<gary_poster> insert structure of another view, which itself uses a template
<gary_poster> use a macro
<mtaylor> lifeless: I do not believe you would accept a patch that hardcodes lp:drizzle/build in to it
<gary_poster> have code done locally
<lifeless> mtaylor: seriously, 50% of our page hits are API now. (and not 'API to make a web page be nice'. Pure launchpadlib style API)
<wallyworld> lifeless: code.launchpad.net/+daily-builds - 67 queries/external actions issued in 2.36 seconds \o/
<gary_poster> so, lifeless, "structure" itself is fine.  My hunch o' nastiness is that the inner loop, calling a view for each loop, is bad.
<lifeless> gary_poster: I'm going to generate some hundreds of rows locally and do some elimination tests to narrow this down. Thanks for the tips - helps me understand this area.
<gary_poster> np
<mtaylor> lifeless: oh, well in this case, I doubt the launchpadlib hits are gonna kill you (I grab a list of approved brancehs) ... but the "do a checkout of each branch and run run bzr missing a few times to see if it's contained within another branch" might be annoying
<gary_poster> I'll look forward to reading the results lifeless!
<leonardr> sinzui: we don't have a bugtask for ubuntu, so bug.getTask(ubuntu) returns nothing. but we do have a bugtask for evolution in ubuntu, so the searchTasks thing fails
<lifeless> gary_poster: now I have to write em up :)
<gary_poster> lifeless: lol :-P
<lifeless> leonardr: what searchTasks thing are you referring to ?
<leonardr> lifeless: the one run in valid_upstreamtask
<leonardr> user = getUtility(ILaunchBag).user
<leonardr>     params = BugTaskSearchParams(user, bug=bug)
<leonardr>     if not bug_target.searchTasks(params).is_empty():
<lifeless> thats buggy
<sinzui> leonardr: Ubuntu and evolution in ubuntu are not the same, one being a distro and the other being a package. I do not think any query should confuse the two
<lifeless> it will find tasks on related series.
<leonardr> ok, now we're getting somewhere
<lifeless> I think the validator is broken
<lifeless> -or-
<lifeless> its not the right thing to use to fix the oops
<lifeless> if you have a product with a series A and a series B
<lifeless> this validator will raise an error if the product, or series A, or series B have a bugtask.
<lifeless> but you are working on distros
<lifeless> and distros have another layer in there - the source package
<lifeless> you can have a distro task or a distroseries task, or a distro sourcepackage task, or a distroseries sourcepackage task
<lifeless> leonardr: what is calling valid_upstreamtask ?
<leonardr> lifeless: my code causes it to be called from createTask()
<leonardr> formerly it was only called from view code. looking up the view code now
<lifeless> leonardr: you need to call validate_distrotask when dealing with distro contexts
<lifeless> or validate_new_distrotask etc
<lifeless> I don't know offhand which you'll need for createTask
<sinzui> wtf: can we have one method that differs to the hidden methods?
<lifeless> I need to pop out to the shops
<sinzui> There must be duplicate code in three validation methods
<lifeless> sinzui: tonnes from the look of it
<lifeless> I'm sure you guys will get to the bottom of it soon. BBIAB
<lifeless> wallyworld: nice
* benji changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> Morning all.
<mtaylor> am I dumb - there doesn't seem to be an api member on a merge_proposal to get the web url to see that prop
<mtaylor> ?
<jcsackett> morning, wgrant.
<lifeless> oh grah
<lifeless> IOError: [Errno 2] No such file or directory: '/home/pqm/pqm-workdir/home/trunk/launchpad/production-configs/lpnetSLAVE-lazr.conf'
<leonardr> mtaylor: there should be one
<lifeless> lp-production-config landings are bust
<lifeless> testConfig (canonical.config.tests.test_config.../production-configs/lpnet-template/launchpad-lazr.conf)
 * lifeless goes really
<wallyworld> lifeless: the business queries - the ones to get the data - there's only 4 or so of those. there's an awful many of these: SELECT ValidPersonCache.id FROM ValidPersonCache WHERE ValidPersonCache.id = xxx
<leonardr> mtaylor: it should be web_link. what version of the web service are you using?
<wgrant> lifeless: How did it land in the first place? :/
<mtaylor> leonardr: LPNET_SERVICE_ROOT?
<mtaylor> leonardr: (not sure how to answer your question ...)
<wgrant> mtaylor: Your cached WADL is probably out of date.
<mtaylor> hrm. how do I update my locally cached WADL?
<leonardr> wgrant: that's possible, but it should have been updated by now. it's only cached for a week
<wgrant> I just blow away the whole launchpadlib cache, but there is probably a better way.
<leonardr> mtaylor: show me how you call Launchpad.login_with()
<wgrant> Hmm. We were seeing it happening three or four days ago.
<mtaylor> launchpad = Launchpad.login_with('Branch Merge Status', LPNET_SERVICE_ROOT, cachedir)
<leonardr> mtaylor: add version="devel" and see if that works
 * wallyworld has to go and drop tax stuff off at the accountant
<mtaylor> leonardr: that just sort of has it hanging there now
<leonardr> mtaylor: that's odd. put this code before the login_with call and send me the output
<leonardr> import httplib2
<leonardr> httplib2.debuglevel = 1
<mtaylor> leonardr: well, of course, it didn't hang that time
<mtaylor> leonardr: oh - actually, I suck - one more try
<leonardr> sinzui: argh, changing the validation code is creating _more_ test failures and taking me further into realms of which i know nothing
<leonardr> to start with, can you sanity-check this?
<leonardr> http://pastebin.ubuntu.com/577211/
<mtaylor> leonardr: it's sitting there doing this:
<mtaylor> http://paste.drizzle.org/show/427/
<mtaylor> and now it works
<leonardr> it hung for a long time and then worked, or you tried again and it worked?
<sinzui> leonardr: That block looks good to me
<mtaylor> leonardr: tried again - it hung for a while and then worked
<leonardr> mtaylor: and is web_link accessible?
<leonardr> sinzui: that's giving me a new test failure (which might be legit):
<leonardr>     LaunchpadValidationError: u'Package a52dec not published in Ubuntu'
<leonardr> except later on there's a reference to ubuntu/+source/a52dec
<mtaylor> leonardr: yup. all is now happy and sunny
<mtaylor> leonardr: thanks!
<leonardr> ok, great. basically, that's a new feature that's not in version 1.0 of the web service
<wgrant> leonardr: It's in the 1.0 WADL.
<leonardr> hmmm
<leonardr> mtaylor: did you end up wiping your launchpadlib directory?
<mtaylor> leonardr: I did not - just doing the version='devel' made it happier
<leonardr> well, devel would have a _different_ wadl file, that you hadn't retrieved before, so that could have the same effect
<SpamapS> lifeless: ping.. trying to wrap my head around adding a new distro to launchpad that is not for built packages...
<sinzui> huwshimi: will you be mumbling today?
<huwshimi> sinzui: Yes, just getting my other laptop, it doesn't seem to want to connect in Natty
<huwshimi> sinzui: two minutes
<sinzui> leonardr: I got confused for a moment. That error really could be legitimate. When I started fixing packaging link data 18 months ago, I discovered that 80% of you tests were working with invalid data...no surprising that the real data created by the feature was rubbish
 * sinzui looks for test helper to create a published spr
<leonardr> sinzui: i'm pushing my changes
<leonardr> lp:~leonardr/launchpad/bug-106338
<sinzui> leonardr: I am in a meeting. I am also looking for a trick that makes a valid package
<leonardr> ok
<leonardr> sinzui, i may come back to this tomorrow, because i don't understand this stuff even when my brain is fresh
<sinzui> leonardr: I think that is sensible. This is really messed up and timeoff will help us forget what does not work
<leonardr> ok
<sinzui> leonardr: the failing tests probably need something like this: self.factory.makeSourcePackagePublishingHistory(
<sinzui>             sourcepackagename=self.sourcepackagename, distroseries=self.hoary)
<wgrant> sinzui: All those style fixes have almost let you catch up :(
<sinzui> wgrant: There were some bugs I fixed by accident too.
<sinzui> but you are the winner
<lifeless> wgrant: mthaddon probably manual landed
<lifeless> SpamapS: hi
<wgrant> lifeless: Yeah, probably :(
<SpamapS> lifeless: I understand you had a chat w/ jml about a distro for ensemble formulae
<lifeless> SpamapS: creating a distro is a privileged task; please wait for the ensemble list discussion and LEP before doing it
<SpamapS> lifeless: right, I'm not going to do it.. I am trying to understand the challenges.
<lifeless> SpamapS: ah, do you mean challenges in using or creating? :)
<SpamapS> lifeless: well I don't understand the suggestion that there's no bug tracking if you don't have a built package. How do non-built distros work given that restriction?
<lifeless> SpamapS: they don't
<lifeless> SpamapS: try and file a bug on openssh in baltic.
<lifeless> *baltix*
<SpamapS> Oh.
<lifeless> I dare you
<SpamapS> whats the use of distros then? ;)
 * SpamapS is tempted now
<lifeless> SpamapS: distros are all about building and shipping packages
<SpamapS> Yeah , be that the case in reality.. in my head, they're about tracking collections of software. :-P
<lifeless> SpamapS: so we don't permit bugs filed on package names we know about unless the package is in the distro
<lifeless> SpamapS: this stops folk filing bugs on e.g. cassandra before its actually in Ubuntu
<SpamapS> that makes perfect sense. :)
<lifeless> and this matters because all our metadata driving policy - like packagesets etc - depends on source package / binary packages existing.
<SpamapS> So the real issue is there's no way, other than building, to create a package.
<lifeless> *I* don't know how deep the rabbit hole goes
<SpamapS> The way I saw it working was a mapping of a set of directories into source package names...
<lifeless> SpamapS: you can map source packages into source package names
<SpamapS> But I'm starting to wonder if, for the beginning of the project, we shouldn't just use a flat bzr and let people use tags to distinguish the formula name.
<lifeless> SpamapS: that would be a -lot- simpler.
<lifeless> SpamapS: jml and I had trouble inferring /requirements/ vs /things that using a distro would let you do/
<SpamapS> Yeah I think people got excited about reusing the distro code (myself included)
<lifeless> In some ways it makes a lot of sense
<lifeless> in other ways we really have two quite different things wedged into one in lp's distro bug support
<SpamapS> If the project takes off... we hope to have similar numbers of formulae as we have packages in Ubuntu so it would be very hard to track bugs on such a large body of work.
<lifeless> one things is 'lightweight components in a bug tracker' and the other is 'policy driven by uploads of stuff'
<lifeless> SpamapS: really? 20K forumulas?
<SpamapS> a formula for anything that opens a network port
<lifeless> SpamapS: thats more like 1K
<SpamapS> so, maybe 2000
<lifeless> we can start with one structure and migrate
<lifeless> like, it would be great to have lightweight components for e.g. bzr
<lifeless> something more than tags and less than separate projects
<SpamapS> Yeah.. I think thats probably going to be the thing I recommend in the list discussion.
<SpamapS> Tight coupling, once again, proves to make things less usable...
<lifeless> indeed
<lifeless> I'm a big fan of proving a core facility and letting folk do something cool with it
<lifeless> now, on the vcs side
<lifeless> jml said we need branch per (formula, release) tuple
<lifeless> which the (packagename, series) namespace in distro branches supplies
<lifeless> SpamapS: oh, on bugs, you /can/ use product series as a component like thing. same place in the UI
<lifeless> but the concept of nominations vs tasks would make it a little ugly
<SpamapS> lifeless: the series will be the series of formulae tho.. I'd like to keep that nice and clean.
<lifeless> indeed
<lifeless> so back to vcs
<lifeless> do you need that tuple now, or really just (formula, 'sid') - e.g. you only need trunks for the formulas?
<SpamapS> lifeless: I think having one bzr branch for all 2000 formulas would be a little painful. But you just mentioned in a recent discussion that bzr can branch the sub-dirs just fine, right?
<lifeless> SpamapS: you can work on just a subdir yes.
<lifeless> all the history comes, but honestly, 2K small projects is tiny bzr scale wise
<SpamapS> so if its  /principia/sid/formulas/databases/mysql  .. I can bzr branch lp:principia/sid/formulas/databases/mysql and interact with it, yes?
<SpamapS> Oh but I still get the whole history for /principia/sid .. :(
<SpamapS> A minor concession.. but one that our target market will hate.. because they're all git/ruby fanbois. ;)
<lifeless> SpamapS: yes, but I think you're overthinking the size here.
<lifeless> when you do this in git you get all the history too
<SpamapS> Right its just faster. ;)
 * wgrant frowns at Lenovo.
<SpamapS> which I *hate* as an argument for any vcs..
<SpamapS> but some people think it matters. ;)
<lifeless> SpamapS: mmm, not as such per various benchmarks we've done.
<lifeless> SpamapS: bzr network fetch is pretty good these days
<wgrant> 2a fetch is really slow even locally...
<SpamapS> lifeless: Yeah, I'm going to again say that its a very small issue.. the main thing is to start getting those merge proposals cranking.. not achieve intergalactic domination overnight. :)
<lifeless> SpamapS: anyhow, i have not particular preference for what we do
<lifeless> I'm not trying to guide towards or away from distros
<lifeless> but - they are what they are.
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/730987
<_mup_> Bug #730987: cannot merge to lp-production-configs <Launchpad itself:Triaged> < https://launchpad.net/bugs/730987 >
<wgrant> lifeless: Will look soon. Currently trying to figure out whether my T400's battery is being malicious or is actually dead.
<wgrant> It is 100% charged and still at around 60% design capacity, but this morning refuses to deliver any power.
<lifeless> wgrant: freeze it?
<wgrant> The battery monitor tool says it has failed due to normal wear.
<wgrant> I call BS.
<wgrant> Maybe I'll leave it all off and unplugged for a few hours and see what happens.
<wgrant> I have never had a Li-ion battery do this before, so I tend to think it is being malicious.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #513: FIXED in 5 hr 17 min: https://hudson.wedontsleep.org/job/devel/513/
<lifeless> wallyworld: you need to gall PersonSet's getPrecachedPersonsFromIDs() method and listify the result, or something like that
<lifeless> wgrant: thanks!
<wgrant> lifeless: The obvious fix is to add a list of excluded configs to the test... but maybe a marker file in the config dir would be better.
#launchpad-dev 2011-03-08
<lifeless> wgrant: blacklist +1
<wgrant> lifeless: But that means having the list of exclusions in a separate tree.
<lifeless> wgrant: we've needed one exclusion in 6 years
<lifeless> wgrant: I think we can deal
<wgrant> True.
<wgrant> Pushing.
<wgrant> Ah, allenap beat me to a bulk loading helper.
 * lifeless isn't sure its the right way
<wgrant> I'm not sure it is either.
<lifeless> we want, for instance, to eager load visibility rules etc
<lifeless> so I think it needs per-type handling rather than a separate helper
<lifeless> however, allenap has written something that will be usable for somethings
<wgrant> Right, that's what I was thinking. It is a pattern that we already use in several places.
<wgrant> And this helper can do nice stuff like not issuing a query if everything is already preloaded.
<lifeless> it introspects the storm cache?
<wgrant> Since that is going to become a common case, I think.
<wgrant> *can do*
<wgrant> Not does now.
<lifeless> ah
<lifeless> sure
<thumper> arse
<thumper> leonardr: I don't suppose you are still around?
<thumper> leonardr: I have a lazr-restful problem
<leonardr> thumper: i'm here
<thumper> ah, awesome
<thumper> leonardr: let me pastebin something...
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/exclude-lpnet-template/+merge/52497
<wgrant> Thanks.
<wgrant> lifeless: You're landing lamont's thing and the staging fix?
<lifeless> thats how I found this, yes.
<lifeless> I'm not sure what branch the check checks out
<wgrant> Checking.
<thumper> leonardr: http://pastebin.ubuntu.com/577252/
<wgrant> ./launchpad                             bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable
<wgrant> fuuuu
<thumper> leonardr: getting TypeError: Only a read-only field can have a mutator method.
<wgrant> lifeless: But you can fix that in the branch :P
<lifeless> wgrant: meh, will wait for bb
<thumper> leonardr: wanting the accessor and mutator only in "devel"
<lifeless> stable is not unreasonable
<thumper> leonardr: any ideas?
<leonardr> reading
<leonardr> thumper: have you tried putting @operation_for_version first?
<thumper> leonardr: first as in at the top, or bottom?
<leonardr> at the bottom
 * thumper tries
<thumper> nope, that doesn't fix it
<leonardr> right now the mutator is exported in beta nad the field is not exported at all in beta
<leonardr> same error?
<thumper> yep
<leonardr> thumper: the field itself needs to be read-only. it can't just be exported as read-only
<thumper> oh?
<leonardr> see the mutator_for method in lazr.restful declarations.py
<thumper> really?
<leonardr> that's what it says
<leonardr>         if not self.field.readonly:
<leonardr>             raise TypeError("Only a read-only field can have a mutator "
<leonardr>                             "method.")
<thumper> hmm...
<thumper> we should fix that
<thumper> maybe...
<wgrant> Why can't you make the field read-only? It only affects forms and the webservice.
<lifeless> sinzui: are you going to land the new lazr restful egg?
<thumper> wgrant: because the forms edit it
<thumper> wgrant: I'd have to override the field...
<wgrant> :(
<thumper> wgrant: and it changes the permissions
<thumper> wgrant: if the field is readonly, the permissions don't allow setting
<wgrant> It doesn't affect security proxies.
<leonardr> lifeless: i think it would be okay to let mutator_for succeed if the field is never *published* as read-write in any version for which it has a mutator
<wgrant> Hmm, unless maybe it affects set_schema, but not much uses that.
 * leonardr has to go
<thumper> wgrant: I think it does...
<thumper> leonardr: ok
<wgrant> thumper: Anyway, shouldn't the form be using the mutator anyway?
<thumper> wgrant: it means more hoop jumping to extract the field from the data dictionary, getting the delta object first, raising the object modified event manually
<thumper> man I wish our form infrastructure used the API :)
 * thumper has a plan
<thumper> :(
<thumper> ohh... tentative success
<thumper> :( now getting  You tried to modify a read-only attribute. in the 400 response
<wgrant> checkwatches OOPSes and bye-bye-nullbugtask today, I think.
<lifeless> wgrant: but its perf tuesday!
<wgrant> lifeless: Killing NullBugTask gets rid of some timeouts!
<wgrant> But OK.
<wgrant> Maybe getBuildRecords.
<lifeless> \o/
<wgrant> I also need to attack +copy-packages again.. the copying is now fast, but the rest of the page is not :/
<lifeless> wgrant: its a never ending story
<leonardr> thumper: it sounds like the lookup of the mutator operation didn't succeed. you could try putting a breakpoint in generate_entry_adapters() (lazr.restful declarations.py)
<leonardr> it should be creating a PropertyWithMutator for the field in the appropriate version
<thumper> i'll try that
 * thumper sighs
<thumper> every freaking entry
<thumper> ok
<thumper> lots of continues, and found the right interface
<lifeless> huwshimi: Bug 730535
<_mup_> Bug #730535: Enter key ignored the first time in bug tag editing field <bugtag> <javascript> <Launchpad itself:Triaged> < https://launchpad.net/bugs/730535 >
<lifeless> huwshimi: is that a regression from your changes?
<huwshimi> lifeless: it looks like the same issue. I believe my changes won't be rolled out until lazr-js is updated in LP.
<lifeless> huwshimi: ah! then your bug should not be closed
<lifeless> huwshimi: and this new bug should be a dupe.
<huwshimi> lifeless: yeah
<huwshimi> lifeless: I think it was marked as closed during a deployment or something
 * thumper sighs
<lifeless> huwshimi: do you know how to move forward on this?
<huwshimi> lifeless: Nope :)
<lifeless> ok
<lifeless> first things first, lets get the bugs sorted out
<lifeless> find your old one
<lifeless> set to in progress
<lifeless> dupe the new onto the old
<lifeless> the old one should have two tasks, one on lazr-js and one on launchpad,t he lazr-js one should be fix released if your change is merged into it.
<lifeless> I'm moderately sure deryck finished upgrading us to the latest lazr-js recently so your fix should be easy to get to.
<wgrant> Ah, sorry, that was probably my fault. I thought I set it back, though :/
<huwshimi> lifeless: The old one (bug #580404) doe not have a lazr-js task, I assume I can just add that?
<_mup_> Bug #580404: pressing enter in tags field doesn't save them <ajax> <javascript> <lp-bugs> <qa-needstesting> <Launchpad itself:In Progress by huwshimi> < https://launchpad.net/bugs/580404 >
<lifeless> huwshimi: yes
<lifeless> huwshimi: you made the change in lazr-js right ?
<huwshimi> lifeless: ugh, sorry I'm getting random lockups with my keyboard and mouse (but I can see that the OS is still working)
<huwshimi> lifeless: Yes the changes were made to lazr-js
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #427: FIXED in 5 hr 14 min: https://hudson.wedontsleep.org/job/db-devel/427/
<huwshimi> lifeless: ok, I've done all that. What next?
<lifeless> next, you need to figure out whats new in lazr-js leading up to the commit where your patch landed, so you can decide how much qa you want to do
<lifeless> and/or whether you want to land everything or just up to your patch
<lifeless> use bzr and or bazaar.launchpad.net to look at the history and decide
<lifeless> pick the rev of lazr-js you want to update to
<lifeless> and then we go to the actually-doing-it step
<huwshimi> lifeless: ok thanks will take a look
<huwshimi> lifeless: How do I know what hasn't been landed?
<huwshimi> lifeless: Or rather released
<lifeless> look at versions.cfg, see the version there, look at the branch for lazr-js, walk back until the release
<huwshimi> lifeless: I'm not entirely sure what you mean by that, there are a lot of version numbers in that file and none seem to refer to lazr-js? I may be completely mis-understanding you
<lifeless> lazr-js = 1.6DEV-r202
<wgrant> OOPS reports during release week are depressing.
<lifeless> wgrant: because deploys stop ?
<lifeless> https://lpstats.canonical.com/graphs/AppServerRequestLpnet
<lifeless> 14:51 < huwshimi> lifeless: I'm not entirely sure what you mean by that, there are a lot of version numbers in that file and none seem to refer to lazr-js? I may be completely mis-understanding you
<lifeless> 14:51 < lifeless> lazr-js = 1.6DEV-r202
<huwshimi> lifeless: Sorry crashed again
<wgrant> lifeless: Yes.
<huwshimi> lifeless: Ah right, you're referring to the versings.cfg in Launchpad, not the one it lazr-js :)
<huwshimi> lifeless: ok, so there is one other commit since the last release
<wgrant>  * 1954 Exceptions
<huwshimi> lifeless: The other commit does not have an attached bug, is it still possible for me to be able to qa it somehow?
<lifeless> huwshimi: of course
<lifeless> bugs don't permit qa, they are just about recording stuff
<lifeless> huwshimi: make an assessment of the other change
<huwshimi> lifeless: It's a module that's been updated (looks like a file has been replaced with a whole new version).
<huwshimi> ok next time this crashes I'm giving up and going back to my old laptop
<lifeless> huwshimi: what is it?
<huwshimi> lifeless: the laptop?
<lifeless> huwshimi: so you need to decide if its safe to include in lp as-is
<lifeless> yes the laptop
<huwshimi> it's a 2011 Macbook Pro.
<Ursinha> lifeless, and now?
<Ursinha> I forgot to change a link
<Ursinha> sorry
<lifeless> Ursinha: brilliant
<huwshimi> ok that time it was Natty crashing. Laptop, I'm giving you one more chance
<huwshimi> lifeless: from a quick grep I can't find any references to that lazr-js module.
<lifeless> where ? ;)
<huwshimi> lifeless: in the launchpad codebase
<huwshimi> lifeless: I could be wrong though :)
<lifeless> huwshimi: so if we're not using galery-form, its a no brainer
<lifeless> huwshimi: I think its on the wiki somewhere, but what you need to do next is make a new sdist, put it in the lp download cache (copy it in, bzr add, bzr commit), update versions.cfg in lp and propose that as a branch to land in devel
<huwshimi> lifeless: Where do we define which javascript files we include?
 * thumper leaps through a few more hoops
<lifeless> huwshimi: I don't know
<huwshimi> lifeless: So I'm wrong about us not including that file
<lifeless> huwshimi: in which case, check that lp will still work with the new version :)
<huwshimi> lifeless: And how do I do that? :)
<lifeless> I'm sure you'll think of something
<lifeless> e.g. how did you make sure the bug tag editing worked
<huwshimi> lifeless: Well I'm trying to figure out if we actually use this module at all
<lifeless> wgrant: do you remember where we set allowed attributes for built in types?
<lifeless> thumper: ^
<thumper> for built in types?
<thumper> like what
<thumper> ?
<lifeless> dict
<lifeless> for instance
<thumper> lifeless: hmm...
<lifeless> in this case, defaultdict
<thumper> no
<lifeless> doesn't have __getitem__ permitted
<thumper> grep is your friend :)
<lifeless> and making an interface would be batshit
<thumper> it'll be in a configure.zcml
<huwshimi> Does anyone here know how we handle which javascript files are included in Launchpad? I've found this, but it doesn't seem to be correct: https://dev.launchpad.net/JavaScriptBuildSystem
<thumper> huwshimi: what do you want to know?
<thumper> I've learned some things
<lifeless> lp_sitecustomize.py
<huwshimi> thumper: There is a javascript file (as part of lazr-js) that is included in our launchpad.js, but I can't see where we actually include that to be built.
<thumper> jsbuild is full of magic
<thumper> and a PITA
<thumper> make jsbuild calls out to a couple of utility scripts to determine dependancies
<huwshimi> thumper: do you remember which scripts they are?
<thumper> huwshimi: look in the Makefile
<thumper> for the jsbuild: target
<thumper> they are in backquotes
<wgrant> The file list is hardcoded now, isn't it?
<wgrant> In utilities/yui-deps.py
<thumper> wgrant: does that include the lazr-js bits?
<thumper> huwshimi: are you doing something in lazr-js?
<thumper> huwshimi: I have a request
<wgrant> thumper: Ah, no, just all of YUI :(
<huwshimi> thumper: I'm trying to build lazr-js for Launchpad and trying to determine if we are using a module that was updated to a new version by someone else.
<huwshimi> thumper: What's the request
<huwshimi> ?
<thumper> huwshimi: somewhere in the picker code, it is adding a &thinsp;
<thumper> huwshimi: it needs to die, and CSS used instead
<huwshimi> thumper: I don't think I want to change anything else right now or else my brain will melt, but feel free to file a bug or something and I'll try and get to it :P
<thumper> :)
<thumper> ok
<huwshimi> thumper: if you landed a branch quickly I could probably include this in the build.
<thumper> huwshimi: ETOOMUCHTODO
<huwshimi> thumper: Fair enough :)
<thumper> huwshimi: I've already been on a diversion
<thumper> anyone: https://code.launchpad.net/~thumper/launchpad/choice-widget/+merge/52357
<huwshimi> thumper: Those scripts don't seem to mention lazr-js stuff
<huwshimi> thumper: or am I missing something?
<thumper> 	lib/canonical/launchpad/icing/lazr/build/lazr.js
<thumper> huwshimi: that is the last line of the jsbuild command in the Makefile
<thumper> huwshimi: I'm guessing that you'll find that is a symlink to the location where lazr-js is built
<wgrant> What's the setuptools sdist option to add a version suffix?
<thumper> using the jsbuild_lazr make target
<thumper> wgrant: I don't remember but I need it too
<thumper> wgrant: what are you packaging?
<wgrant> thumper: Updating pydkim.
 * wgrant greps logs.
<thumper> not in my history sorry
<wgrant> Because it's not in --help :(
<wgrant> -b
<huwshimi> thumper: Ah right, so we do just include the whole of lazr-js and don't control which files we include.
<thumper> huwshimi: maybe...
<huwshimi> thumper: Thanks for that. Now I can get on with my investigating.
<wgrant> Except that the egg_info command doesn't work here :/
<wgrant> It's like setuptools isn't installed...
<huwshimi>  lifeless: so I think after all that we include that file but don't use it, but I can't be sure
<StevenK> Does anyone want to fix the failed stable into db-devel merge?
<wgrant> Not again :(
<wgrant> Fixing.
<StevenK> wgrant: Thanks
<wgrant> This is changing one that I already resolved on Saturday :(
<StevenK> wgrant: Can you have another look at https://code.launchpad.net/~stevenk/launchpad/populate-spr-changelogs/+merge/52363 ? I've addressed all of your concerns, bar IMasterStore, since everything else done by garbo uses it.
<wgrant> StevenK: You renamed getCandidateSPRIDs, but it still returns SPR IDs...
<wgrant> Also, why are we still restricting to Debian?
<StevenK> 1. Our sampledata sucks. 2. I don't think there is any point doing it over more.
<wgrant> Apart from that it looks good.
<wgrant> What about all the Ubuntu publications from more than a year ago?
<StevenK> wgrant: Er? It returns a ResultSet of SPRs ordered by id?
<wgrant> No, it returns a ResultSet of SPR.id ordered by id.
<StevenK> Bah, fixed
<StevenK> wgrant: I'm happy to say upload_archiveID IN (ubuntu.main_archive.id, debian.main_archive.id)
<wgrant> StevenK: Why say that at all?
<StevenK> Running it over sampledata which the tests is horrid
<StevenK> Er. Which the tests do
<wgrant> It's easy to clean the sampledata up.
<StevenK> I'd rather just delete it :-)
<wgrant> Right, but deleting it in the tests is hard.
<huwshimi> lifeless: What should I do with this then? I don't want to release these changes if they'll break things...
<lifeless> huwshimi: will they break things?
<huwshimi> lifeless: Well that's the problem, I don't know. I can't see where we're using this module, but that doesn't mean we're not using it. I could just be looking for the wrong things.
<StevenK> wgrant: Personally, I'd prefer to leave it for Debian. For the derived case, we aren't going to care about the older Ubuntus
<wgrant> StevenK: There is stuff in Natty that has no changelogs...
<lifeless> huwshimi: check the wiki for info, failing that ask for help: mail the dev list asking for 'how to find out where a lazy-js module is used by lp'
<StevenK> Really?
<huwshimi> lifeless: OK thanks, will do.
<wgrant> StevenK: Anything that was uploaded before we started importing changelogs not-very-long-ago.
<StevenK> wgrant: The DB on DF for maverick said zero had a null changelog ...
<wgrant> StevenK: With upload_distroseries=natty? Sure.
<wgrant> But not much stuff in natty was uploaded to natty.
<StevenK> The natty DB on DF is very little
<wgrant> Even moreso there.
<lifeless> StevenK: the point is the things uploaded to maverick that haven't been changed in natty
<StevenK> wgrant: So, how do you propose I have the tests ignore the sampledata?
<wgrant> StevenK: Expire all LFAs at the start of the test.
<wgrant> The brutal approach.
<StevenK> Allow me to say "Ew"
<wgrant> It is better than sampledata.
 * StevenK tries to work out how to do so
<wgrant> We may eventually want two template DBs, one without crap.
<wgrant> Since we won't be able to drop the crapful one for a long time :(
<StevenK> wgrant: I can't think of an elegant way to expire all the LFAs
<wgrant> StevenK: Store.find(LibraryFileAlias).set(content=None)?
<StevenK> wgrant: Having it return SPRs directly complains about the GROUP BY
<wgrant> StevenK: Group by SourcePackageRelease instead.
<wgrant> Hmm.
<wgrant> That might be why I was grabbing IDs... grouping by all the fields may have proved slower, although that seems odd.
<wgrant> Since it should be able to optimise to a pkey grouping.
<lifeless> shoulda coulda woulda
<lifeless> the way to do that is to use the id based query as an inner query
<lifeless> like e.g. _getExternalMessages does
<jtv> StevenK: can you think of any reason why gina would initialize the references in SPPHs it creates using the ids for distroseries, SPR, etc. instead of the actual model objects?
<StevenK> Nope
<StevenK> gina is .... special
 * StevenK grumbles
<lifeless> jtv: code fragment ?
<StevenK> Now it dies due to a variable used before assignment :-(
<StevenK> And if I change its indent level, it loops until killed
<jtv> lifeless:
<jtv>         entry = SourcePackagePublishingHistory(
<jtv>             distroseries=self.distroseries.id,
<jtv>             sourcepackagerelease=sourcepackagerelease.id,
<jtv>             status=PackagePublishingStatus.PENDING,
<jtv>             component=component.id,
<jtv>             section=section.id,
<jtv>             datecreated=UTC_NOW,
<jtv>             datepublished=UTC_NOW,
<jtv>             pocket=self.pocket,
<jtv>             archive=archive
<jtv>             )
<jtv> StevenK: is gina special enough to warrant that?
<wgrant> I have a branch which fixes that.
<lifeless> jtv: I would change it to distroseriesID=self.distroseries.id etc
<StevenK> jtv: Er, things like that are the REASON why gina is special
<lifeless> we shouldn't overuse Reference objects, they are fragile
<jtv> Okay, but is there any justification?
<wgrant> For manually instantiating it, rather than using PublishingSet.newSourcePublication?
<jtv> No, for using the ids instead of the model objects.
<StevenK> I suspect SPPH support was bolted on.
<jtv> Because I'm trying to make it use newSourcePublication.
<wgrant> https://code.launchpad.net/~wgrant/launchpad/unify-publication-creation
<wgrant> One remaining issue: it doesn't set datepublished.
<StevenK> Which is fine
<StevenK> We don't want it to
<jtv> It shouldn't even do that in the first place.
<wgrant> Perhaps so, but inconsistent.
<jtv> Shame that branch sat around for half a yearâit's a change I need for my current work.
<wgrant> jtv: I didn't want to get into the gina argument at that point.
<jtv> But all in all we don't really know whether there's any particular reason to use ids there?
<wgrant> There isn't one.
<wgrant> If you see something odd in gina, it's because gina is crap, not because it needs to be that way.
<jtv> That's nice to hear.
<jtv> Brings me to the next question: newSourcePackagingPublication creates a new DSP if one does not existâwhich is breaking gina tests.
<wgrant> I added a permission in that branch.
<wgrant> I think it's the right thing to do.
<jtv> I was wondering if maybe it was just laziness of the test, and there should always be a DSP
<wgrant> gina predates DSP.
<wgrant> There's no problem with creating one.
<StevenK> wgrant: Why didn't you want to get into a gina argument?
<wgrant> StevenK: Because nobody knows whether we want to stop setting datepublished or start creating them Published or burn gina to death.
<lifeless> whats gina used for?
<StevenK> Julian said yesterday we want to stop setting datepublished
<wgrant> OK, how does _'s string interpolation work?
<StevenK> lifeless: It imports Debian
<wgrant> lifeless: Importing Debianish archives into the DB and librarian.
<jtv> And so the datepublished=UTC_NOW is completely bogus
<wgrant> jtv: It is a lie, and not one with considerable utility.
<wgrant> Kill it.
<jtv> I had already done that.
<StevenK> wgrant: http://pastebin.ubuntu.com/577330/ makes me sad
<wgrant> StevenK: Oh?
<wgrant> It makes me happy.
<StevenK> http://pastebin.ubuntu.com/577331/
<wgrant> StevenK: Which line does it fail on?
<StevenK> self.start_at = spr.id + 1
<wgrant> Ah, that's not a new issue.
<wgrant> It'll happen whenever there are no SPRs to process.
<StevenK> :-(
<lifeless> 200 tasks -> 10 second renders :(
<wgrant> lifeless: But it at least renders now?
<lifeless> thats on my laptop
<wgrant> Ah.
<lifeless> which is an i7
<wgrant> Ow.
<lifeless> 50ms per bugtask hmm
<wgrant> lifeless: Is wampee running with the second new set of appservers?
<lifeless> wgrant: yes
<wgrant> Excellent.
<jtv> wgrant, want to review my branch?  https://code.launchpad.net/~jtv/launchpad/bug-730460-use-spph-factory/+merge/52515
<wgrant> jtv: Gladly.
<jtv> Thanks.
<jtv> wgrant: I split off the factory part to free me up for the real work, so there's not much there.
<wgrant> lifeless: Could you mentor that?
<jtv> I'll go grab some food.
<wgrant> lifeless: What would you recommend as a Python testrunner these days? I normally use nose, but you probably have better ideas.
<lifeless> wgrant: testr with subunit.run as the core
<wgrant> lifeless: subunit.run doesn't do test discovery, does it?
<lifeless> wgrant: it does if you have python 2.7 or the discovery module installed
<wgrant> Ah, handy!
<wgrant> Thanks.
<adeuring> good morning
<StevenK> wgrant: So, that script doesn't love me.
<wgrant> StevenK: Oh?
<StevenK> FeatureError: Single aggregates aren't supported after a  GROUP BY clause
<wgrant> bigjools: What do you think about replacing cron.daily-ppa with DiskPool improvements?
<StevenK> wgrant: I think we're going to have to shift back to ids ...
<lifeless> wgrant: see the oops-repository .testr.conf for inspiration
<wgrant> StevenK: Yeah, probably.
<lifeless> or
<lifeless> wrap the result as I suggested
<StevenK> wgrant: Will you rescind your issue with it if I do that?
<lifeless> see _getExternalthingy in translations for an example
<wgrant> StevenK: Maybe.
<wgrant> lifeless: No lp:oops-repository?
<lifeless> wgrant: https://code.launchpad.net/oops-repository
<bigjools> wgrant: sounds reasonable as long as it doesn't slow down the publisher
<lifeless> wgrant: I haven't setup a 'trunk' yet, nor all the metadata
<bigjools> figure out where empty dirs are left
<wgrant> bigjools: The file removal method will just check if the directory it's removing from is empty.
<wgrant> So it will slow p-d-r insignificantly.
<bigjools> poifekt
<bigjools> lifeless/stub: can you guys look at my MP with schema changes please?
<lifeless> bigjools: I'll look in the morning - brain is hurting ;)
<lifeless> bigjools: I've just enough energy left to send a perf tuesday mail :)
<bigjools> lifeless: it is the morning :)  ok, no prob
<lifeless> morning schmorning
<StevenK> wgrant: :-(
<jml> hmm
<stub> When do we merge db-devel -> devel? I've got a futzed merge proposal up (it will land on devel, but contains stuff currently only in db-devel)
<stub> bigjools: I'll look now
<bigjools> c=heers
<bigjools> cheers, even
<stub> bigjools: you targetted to devel, not db-devel
<bigjools> FFS, I didn't, but bzr send did :/
<bigjools> grar
<lifeless> stub: anytime prior to the pqm freeze
<lifeless> stub: so anytime in the next 4 weeks
<stub> bigjools: point me to an updated mp when you have one. its r=stub, patch-2208-52-0.sql
<stub> I think that is anytime prior to reopening db-devel for landings, innit?
<bigjools> stub: ta
<lifeless> stub: db-devel is open
<lifeless> stub: we had the freeze for this month already
<stub> There are changes on that branch that landed before the freeze that are not in devel
<lifeless> stub: we merged everything from db-stable
<stub> hmm
<lifeless> night y'all
<bigjools> nn lifeless
<stub> I can't see a db-devel or db-stable merge since feb
<bigjools> stub: when it scans: https://code.launchpad.net/~julian-edwards/launchpad/publisher-config-db-schema/+merge/52530
<stub> oic it
<stub> nfi how my stuff missed going across - it landed on Friday
<stub> I even got poked to qa it
<stub> Mine can wait, but if an earlier revision got merged we might be missing something important.
<wgrant> stub: We merged the rev right before yours.
<wgrant> stub: Yours was un-QA'd at the time.
<stub> ok
<wgrant> We were already very much out of LOSA time, so we didn't want to push it.
<wgrant> We could almost sensibly merge again, though...
<stub> there is no particular reason to land it this cycle.
<wgrant> There is.
<wgrant> The spam from the old one is irritating!
<jam> can somebody land https://code.launchpad.net/~jameinel/launchpad/use_loggerhead_trunk/+merge/52062
<jam> At least, lifeless said on IRC, "Just land the branch" because loggerhead's pqm didn't actually run the test suite
<jam> so we didn't bother setting it up again yet
<jml> jam: sure thing.
<jml> jam: will let you know when the EC2 instance has detached.
<jam> jml: thanks
<jml> hmm. looking at the diff, I'll skip EC2
<jml> ffs.
<jml> lp-land requires a local checkout
<jam> jml: it is pretty simple :)
<jml> jam: it's in PQM now.
<jam> jml: thanks
<bigjools> this whole Storm Unicode vs RawStr thing is driving me potty
<jml> what about it?
<bigjools> I can't throw str at unicode and expect it to work
<jml> jam: "81 queries/external actions issued in 10.73 seconds" -- I want to debug your problem instead of answering email :\
<bigjools> if I try and use RawStr and str() everywhere, somehow unicode gets in....
<jml> bigjools: the first thing you said is basically the whole point. A str doesn't have any encoding information, so it can't be used to represent human-readable text.
<bigjools> but it can ...
<jam> jml: this is http://pad.lv/OOPS-1893C1040 ?
<jml> no, it's just bytes. a_str.decode('utf8') is something that might represent human-readable text
<jml> jam: yes.
<bigjools> I would not mind if this was consistent everywhere, but it's not
<jml> bigjools: can you give an example of where it's not?
<jml> (Python is lousy at this, but Storm is pretty good)
<jml> le sigh
<bigjools> jml: I defined a text column in PG, the storm model was RawStr, and I poked in some data from a form with an explicit str() case.  When Storm reads it back, it blows up with a unicode error.
<bigjools> I'm reverting it all back to unicode now
<jam> benji: since you approved https://code.launchpad.net/~jameinel/launchpad/suppress-generator-exit-726985/+merge/52414
<jam> can you ec2land it?
<jam> Or does it need a second review because you didn't mark it overall approved.
<jam> jml: I'd be happy for you to fix it :0
<jam> It isn't a page I go to often
<jml> jam: I'll be lucky to have caught up on my email by the end of the day.
<jam> so not something that is blocking me. But a bit surprising to have it take 14s to load a code overview
<benji> jam: it doesn't need more approval unless you think it does
<jam> benji: fine with me. I don't think it is very controversial
<jam> wgrant might want to look at it. Since he asked me to write it. but it was up there for him yesterday :)
<benji> I guess can land it for you if you need.
<jam> benji: well, if you think it needs ec2land, I would appreciate it. Since I'm not an lp dev, I don't have an ec2 account, etc.
<benji> ah, gotcha; sure I'll do it for you
<wgrant> jam: Looks great.
<wgrant> benji: Thanks.
<jam> wgrant: what time is it there?
<wgrant> jam: Not quite midnight.
<jam> ah, k
<jam> wasn't sure if it was late enough to be considered "early" yet
<wgrant> Heh, no.
<gary_poster> losas, is there a way for me to check how long the current staging update has taken without bothering you?
<benji> gary_poster: I think devpad.canonical.com:/srv/launchpad.net-logs/staging/sourcherry/2011-03-08-staging_restore.log holds the answer, but I'm not entirely sure how to read it
<gary_poster> benji, cool, thanks, looking
<jcsackett> sinzui: if a project is marked as inactive, that still doesn't free up the name for another project to use, does it?
<gary_poster> benji, losas, AFAICT staging started Tue Mar 8 04:14:09 UTC 2011 and ran without errors, and yet 10 hours later we still have no staging. :-( (I synced the logs just to make sure I wasn't missing anything)
<gary_poster> benji, just sharing with you in case you were curious :-)
<Chex_> gary_poster: let me take a look
<gary_poster> thank you Chex_
<benji> I am.
<Chex_> gary_poster: ok, appears we had a postgres/slony failure, getting you a pastebin
<gary_poster> thanks Chex_.  Did I just miss it in that log benji mentioned above, or was it somewhere else?
<Chex_> gary_poster: yes, it fools me often as well, the slony error happens before the end of the process, see here for where it was in the logfile: https://pastebin.canonical.com/44394/
 * Chex_ looks at his nick, and frowns
<gary_poster> ah!  ok, sorry Chex_, will be more careful next time
<gary_poster> I guess I'll go look at the server log, because I have no idea how to remiedaiate that so far
<gary_poster> jcsackett, I think you were talking to an absent sinzui before, and he is now present
 * leonardr also wishes to talk to a present sinzui
<jcsackett> ah, thanks gary_poster.
<gary_poster> :-)
<jcsackett> sinzui: if a project is marked as inactive, that still doesn't free up the name for another project to use, does it?
<sinzui> I did not know I was offline.
<sinzui> Not at all
<jcsackett> sinzui: clearly, neither did i. :-P
<sinzui> We rename them on request, or we give the old project to the user
<jcsackett> ah, okay. i am looking at https://answers.launchpad.net/launchpad/+question/148314
<jcsackett> global does look like it's not doing anything. but as project-reviewing-overlord, perhaps you would like to weigh in, sinzui?
<sinzui> just rename the old project to global-old
<gary_poster> stub, you around by chance?
<gary_poster> I'd like to know if there's any remediation you can suggest for https://pastebin.canonical.com/44394/ which is causing staging to be down
<sinzui> jcsackett:  this issue relates to bug 106501
<_mup_> Bug #106501: Automatically warn about, then delete, unused projects <chr> <lp-registry> <projects> <Launchpad itself:Triaged> < https://launchpad.net/bugs/106501 >
<jcsackett> sinzui: i concur.
<jcsackett> that process would be nice.
<jcsackett> so, you think we can offer to make the person the head of global, sinzui?
<sinzui> jcsackett: just rename the old one.
<jcsackett> sinzui: cool.
<sinzui> jcsackett: we really do not think about it. we just get the trash out of the way
<jcsackett> sinzui: sounds good.
<leonardr> sinzui, when will you have time to revisit my sampledata problem?
<sinzui> leonardr: about 45 minutes?
<leonardr> sinzui, cool
<rvba> sinzui: Hi ... I've been tasked with bug 727632 and I wonder if you could give me hints on how you think it should look in the UI as I'm still fairly new to the UI (or the code for that matter). I just sent you an email with a few details and a mockup.
<_mup_> Bug #727632: distro and distro series pages do not specify their owner <derivation> <distributions> <series> <Launchpad itself:Triaged by rvb> < https://launchpad.net/bugs/727632 >
<sinzui> Do they have an owner?
<sinzui> No they do not
<sinzui> distros and series have registrants.
<sinzui> The owner is always the project maintainer
<sinzui> rvba you can cargo cult the "registering" slot from a lp/registry/templates/product-index.pt into the distro and series pages. The tales  need to pull the series/owner field, but the wording will be "Registered by"
<rvba> sinzui: isn't this what's currently displayed already?
<sinzui> rvba: oh. I think soo
<sinzui> so
<sinzui> rvba, is the real issue that we do not have this data in the schema
<rvba> I confess I'm not sure if the issue is only in the UI or in the data. I though it was only a UI bug.
<sinzui> rvba, poolie is confused in this bug
<sinzui> we do not have "owners" of these objects, They must be like projects and project series
 * sinzui looks at the schema
<sinzui> rvba. These pages look like the project and series pages. They are correct! I think the issue may be that the schema does not provide the the information, so it is pulled from a surprising field.
<rvba> sinzui: maybe I'll just correct the wording "Registered" instead of "registered" and ask poolie more details about this then ...?
<sinzui> rvba distro is missing the registrant field! we are showing the maintainer as the registrant. This is a lie. *All* distros and distros series were registered by a losa, most by mthaddon actually
<rvba> sinzui: ok I get it. So the information is wrong then.
<sinzui> rvba: distro has a owner field, but it is actually the registrant. The owner is always the distro maintainer (owner).
<sinzui> oops
<sinzui> disrgard that
<sinzui> *distroseries* has a owner field, but it is actually the registrant. The owner is always the distro maintainer (owner).
<rvba> so if we want to display the owner field I just have to pull that info from the corresponding distro (for series)
<sinzui> rvba: so this bug is about 1, the distro does not know who the registrant is. 2, distroseries stores the registrant in the owner field.
<sinzui> rvba: Many object store the registrant in the owner field because the PrimaryContext (project or distro) always owns them
 * sinzui looks for bug
 * rvba looks into the datastructure
<sinzui> excellent. I find duplicate bugs
<sinzui> rvba: This issue overlaps with bug 153333, bug 188402, bug 207532
<_mup_> Bug #153333: Clean up owner and registrant attributes <feature> <lp-registry> <registry> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/153333 >
<_mup_> Bug #188402: Include registrant on people page <feature> <lp-blueprints> <Launchpad itself:Triaged> < https://launchpad.net/bugs/188402 >
<_mup_> Bug #207532: Product, Project and Series don't have read-only registrant <lp-registry> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/207532 >
<rvba> sinzui: this is much more involved than I anticipated :-)
<sinzui> rvba: you do not need to fix everything. those bugs provide context by what we mean by registrant and owner in the schema
<sinzui> rvba: I updated your bug about what is wrong with distro and distroseries
<rvba> sinzui: all right, I'll focus on fixing this problem for series and distro series
<rvba> but still, since registrant and owner are two different things (?) the UI will have to evolve to present the 2 informations right?
<sinzui> There is one caveat. both distro and distroseries implement IHasOwner, which means they must have an owner field, but in the case of distro we mean maintain, and in the case of distroseries, we mean registrant
<sinzui> We either change the interface / implementation of these objects, or we continue to ignore them.
<rvba> sinzui: don't you think that changing the interface / implem should be part of the larger refactoring you were just talking about
<sinzui> rvba: we could remove the registrant field from distros so we stop lying. We could stop there. As pointed out in the other bugs. registrant should be immutable. Since distros series do not have a direct owner, it seemed okay in the past to reuse the field for registrant. but owner is intended to be mutable. It is not possible BTW to change a series owner. We removed the field from edit forms
<rvba> sinzui: thanks for the clarification
<rvba> sinzui: I guess I'll fix the interface for now since we (red team) are in feature mode.
<sinzui> rvba: this could get tricky. set a limit of a few hours for each task and if you do not make progress on the task, shout for help. you are in 6 years of muddled code and It is easy to loose scope
<rvba> if I understand correctly to fix it properly we would have to create proper interfaces like IHasRegistrant and refactor quite some code ... plus the data migration.
<deryck> gmb, ping.  you got a sec?
<sinzui> rvba: I think that increases scope. I am sure you do not want to retrofit that into all the objects.
<rvba> sinzui: you're saying I should focus on distributions and distroseries?
<sinzui> rvba: maybe we want to make add registrant to distroseries and make the property always return the owner. We can export owner as registrant.
<rvba> sinzui: so no data migration if I understand you suggestion
<rvba> s/you/your/g
<sinzui> rvba: yes the issue is about making the ui clear. You need to add a registrant field to distro. that is a schema..ui change. I do not think we can migrate the data, it is all lies. We usually state the object was registered by ~registy in this case.
<sinzui> rvba: I think these are two bugs, because you need two branches to fix the different needs of each object and page
<gmb> deryck: Sure.
<deryck> gmb, so I retriaged bug 605923 at leonardr's request.  It's the missing archived debian bug bugwatch issue again.
<_mup_> Bug #605923: bug watches for archived debian bug reports appear not to exist <bugwatch> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/605923 >
<gmb> Ugh
<deryck> gmb, I was wondering if you could take a look at the bug and just brain dump some info there, for anyone wanting to look into fixing it?
<gmb> deryck: Sure. I'll take a look at it in a little bit.
<deryck> gmb, many thanks!
<gmb> np
<rvba> sinzui: so you want to add a registrant field to the distro but still populate the owner field?
<sinzui> rvba: add registrant to distro, the migration script will set the value to ~regisrty
<rvba> sinzui: I'm not sure I understand the delimitation of the "2 bugs" properly
<sinzui> rvba: most distros were registered by kiko, most distroseries where registered by mthaddon. both cases they were admins, they were not involved in the project and no one should contact them about them
<sinzui> rvba: 1. distro need a registrant field. the page needs to use the registrant field in the registering slot. the migration script will set all existing distro registrants to ~registry
<sinzui> rvba: 2. distroseries registrant field is the owner, we have choices: a) rename the field field to registrant, b) add a registrant property that always returns the owner, update the interface to export the owner as registrant.
<rvba> sinzui: all right ... this seems to be good bug to deep further into the code
<rvba> sinzui: unless you think you need to give me more information I think I'll dive in and come back with question when I'll be stuck
<sinzui> rvba: happy hacking.
<rvba> sinzui: s/question/questions/g
<rvba> sinzui: thx a lot
<deryck> henninge, looking at your MP.... should TranslationSharingDetailsMixin also raise NotImplementedError for is_sharing and can_edit_sharing_details?
<deryck> henninge, so the API is clear?
<henninge> deryck: Yes, I think that makes sense. I thought about it, too, but I don't know why I discarded the thought.
<deryck> henninge, ok, cool.
<deryck> henninge, ok, looks good to me otherwise.  I'm not sure I see the value in both the page test and the browser unit test.  they seem very similar....
<deryck> henninge, but I guess there is already too much there and it needs updating.
<henninge> deryck: The browser unit test never checks the actual URL, but the page test does.
<deryck> henninge, what value is there in the URL?  Just that hitting the URL produces the page we expect?
<henninge> The link is still dead at this point, so it's just showing that it is pointing to the right place.
<deryck> henninge, sure, that's fine.  Seems like a lot of page test for this.  but no worries.  it's your call.
<sinzui> hi leonardr: I think I have my head clear and can help. I recall we were looking a test failures that I thought were really because the test was not setup right; sample data is wrong
<deryck> henninge, so r=me, with the tiny change mentioned above.
<leonardr> sinzui: right
<henninge> deryck: thanks a lot ;-)
<deryck> np
<leonardr> sinzui: my code is at lp:~leonardr/launchpad/bug-106338
<leonardr> the failure is in patches-view.txt
<sinzui> leonardr: I saw many tests fail last year when we added code that checked that the package was legit. I fixed them by ensuring they were published in Ubuntu:
<sinzui> self.factory.makeSourcePackagePublishingHistory(
<sinzui>     sourcepackagename='alsa-utils', distroseries=distroseries)
<leonardr> i think that could solve the 'a52dec is not in ubuntu' error
<leonardr> do you think it makes sense to go through patches-view and get it to stop using sampledata?
<sinzui> this test sucks
<sinzui> I want to delete the test and declare victory
<leonardr> the whole thing? do we have coverage elsewhere?
<sinzui> I doubt there is proper test coverage
<sinzui> where is the failure in this test?
 * sinzui runs
<sinzui> it
<leonardr> sinzui: there are some problems creating the data in the first place, and then there are problems where the new validation code i added says there are problems with which bugs are conjoined to other bugs
<sinzui> this test could use the factory instead
<sinzui> leonardr: is evolution and a52dec the problems? I think there are missing SPPH in hoary in sample data. We can use the factory to make them genuine ubuntu packages
<leonardr> ok, i'm going to write some code that might or might not work
<sinzui> leonardr: pause
<leonardr> k
<sinzui> I just got this built. I will play the test and try a trick with the factory and transaction.commit() to put the packages in the right state. Feel free to look at something else or get some tea
<leonardr> ok, i'm working on another project as well
<sinzui> excellent, someone solved this in a story before
<sinzui> leonardr: sorry. I had make schema. I think I have this fixed. Looks like I fixed the same problem in a bug story last year
<leonardr> that's great
<m4n1sh> in https://launchpad.net/+apidoc/1.0.html#person the field hide_email_addresses
<m4n1sh> "hide_email_addresses"
<m4n1sh> what should it be set?
<m4n1sh> true/false?\
<leonardr> m4n1sh: yes
<m4n1sh> leonardr: can it be reflected in the WADL?
<m4n1sh> WADL file
<sinzui> unity is a memory hog
<leonardr> yes, we could set type="xsd:boolean" as we have for dates
<sinzui> and I think it leaks like a sieve.
<m4n1sh> sinzui: it has a great ALPHA plastered all over
<m4n1sh> leonardr: under which project should I file a bug for this?
<m4n1sh> foundations?
<leonardr> m4n1sh: put it in lazr.restful
<m4n1sh> thanks
<sinzui> leonardr: I need to reboot to reclaim 1.5G of memory to run the test suite: This is what I did and It think it will fix the issue: http://pastebin.ubuntu.com/577529/
<leonardr> sinzui, thanks
<leonardr> sinzui: ok, that got rid of the sampledata problem. i still have the problem that the validation is preventing bugtasks from being created
<leonardr> make_bugtask(bug=bug_a, target='hoary', target_is_distroseries_name=True)
<leonardr> =>
<leonardr> LaunchpadValidationError: This bug is already on Ubuntu. Please specify an affected package in which the bug has not yet been reported.
<sinzui> leonardr: are we certain that these bugs are unique? They may not given this is a story test
<leonardr> sinzui: well, the bugs are created within the test
<leonardr> so any bug tasks for that bug only exist within the test
<sinzui> yes, but since the test does note setup isolation from the blocks in each narrative, it is easy to step on something
<leonardr> sinzui: looking at the calls to make_bugtask, for bug_a we have a task on the 'trunk product series of patchy,
<leonardr> the evolution source package (of what?)
<leonardr> the a52dec source package (of what?)
<leonardr> and then we try to create one on hoary
<leonardr> could the source-package ones be interfering with the attempt to create a generic hoary bugtask?
<sinzui> leonardr: possibly. If the validator looked at distroseries before source package
<sinzui> leonardr: the calls to create SPPH were in hoary. I assume that we are working hoary in the test because it is the current series in sample data
<leonardr> sinzui: here's the validate code
<leonardr>         target = None
<leonardr>         if distribution is not None:
<leonardr>             validate_new_distrotask(bug, distribution, sourcepackagename)
<leonardr>         elif sourcepackagename is not None:
<leonardr>             target = distroseries.getSourcePackage(sourcepackagename)
<leonardr>         else:
<leonardr>             target = product or productseries
<leonardr>         if target is not None:
<leonardr>             valid_upstreamtask(bug, target)
<leonardr> that was the best i could figure out when to call validate_new_distrotask/validate_distrotask/valid_upstreamtasks
<leonardr> you think source package name should take precedence over distribution?
<sinzui> yes I do
<sinzui> I think the rules are package before series before primary context
<leonardr> ok, trying that now
<leonardr> sinzui: 'series before primary content' -> product or productseries should come before distribution?
<sinzui> I think it is distro before project. project and distribution are primary context...that is to say they are pillars, they are the first item we traverse
<leonardr> sinzui: i gotta tell you that everything you say on this topic goes off on a 45 degree angle from me understanding it
<sinzui> leonardr: bugtasks and conjoined bugs, packages and publishing history are arcane stuff. If the test were good, this would not be a morass.
<sinzui> leonardr: I write something that I think is the correct test, then delete everything that contradicts it.
<leonardr> sinzui: but i have no idea how this works. i'm just trying to make an error give a 400 status code when it happens
<leonardr> let's see what happens now...
<sinzui> leonardr: :( this is why these bugs are so old. fixing it means making our applicate make sense
<leonardr> sinzui: revising the check to my understanding of the precedence doesn't affect the error
<leonardr> i'll paste
<sinzui> hmm, well maybe we do not need the sourcepackage check. does validate_new_distrotask already do that?
<leonardr> sinzui: i think it does. can we call validate_new_distrotask on a distro series or can it only be a distribution?
<leonardr> i think it can only be a distribution
<leonardr> so, if we're passed in a distro series, we still need the sourcepackage check
<sinzui> okay
<leonardr> sinzui: http://pastebin.ubuntu.com/577553/
<leonardr> i've spelled out the precedence in such explicit detail that i now understand it, and i'm still getting the error
<leonardr> does that code look right?
<sinzui> leonardr: It looks right, but it makes me weep.
<sinzui> validate_new_distrotask is called twice I know a distroseries is never upstream, yet we validate it with upstream. I think something stinks below your method. I would not ask you to fix it just yet given the difficulties we are having in the branch too
<sinzui> ^ leonardr if your branch starts passing tests now, lets report a bug about this insanity and let it block the value your branch already provides
<leonardr> sinzui: tests are still not passing
<leonardr> validate_new_distrotask is raising the exception
<sinzui> leonardr: doh, I do *not* want to block your branch from landing. I prefer to report a bug about the issue and say we are done
<leonardr> am i calling it when i should not be?
<sinzui> leonardr: maybe.
<leonardr> sinzui: ok, i understand what's happening
<leonardr> this code prohibits you from adding a bugtask to the 'ubuntu' distribution if there is already a bugtask in ubuntu's 'evolution' sourcepackage
<leonardr> is that the correct behavior?
<sinzui> I am thinking
<sinzui> leonardr: I want to say no, but if I were more involved in Ubuntu, I think I would say this is sensible. A bug is Ubuntu is generic. the bug must really be in a package. We know many users cannot guess the correct package. We move the bug to the package during triage...
<sinzui> but I still believe I could have a task that is definitely not in the package specified, and I do not know where it is
 * sinzui plays on staging
<leonardr> sinzui: i do not want this branch to change the behavior
<sinzui> staging I hate you. May a pox befall your first born
<leonardr> no, the first born is production
<sinzui> leonardr: the validator is correct. We do not want a generic bug when we have a bug that is specific. The test needs to be buried alive, then dug back up and shot
<m4n1sh> done filing the bug https://bugs.launchpad.net/lazr.restful/+bug/731518
<_mup_> Bug #731518: Set hide_email_addresses in person of type xsd:boolean <lazr.restful:New> < https://launchpad.net/bugs/731518 >
<m4n1sh> anyone can have a look at it
 * sinzui looks at what could be deleted
<sinzui> leonardr: If I find that one template/view is governing all the layout in this test, I will start chopping
<leonardr> sinzui: ok, let me push my revisions
<sinzui> leonardr: there is one view and one template. This test is repetitive nonsense. It works on bugtarget. We do not care about each bugtarget, neither does the user according the the bad narrative in the so-called story. We can delete from the second header to the end of the test. "Patches View by Distro" onward is redundant with product.
<leonardr> m4n1sh, the quickest way to get that into lazr.restful would be if you gave me a branch. you just need to add binary support to WadlFieldAPI.type, and copy some tests from what we do to test xsd:datetime
<sinzui> leonardr: the story should be about "a bug target, such as a project"
<leonardr> ok...
<m4n1sh> leonardr: will do in meanwhile when I get time
<leonardr> sinzui: ok, do i have r=you on the changes i've already pushed + destroying the rest of that test?
 * sinzui looks one more time
 * leonardr has now pushed everything
<sinzui> leonardr: r=me with the fixing of the two issue I think I see in the diff: http://pastebin.ubuntu.com/577568/
<sinzui> I seem to have traded a memory leak for a runaway proc. ubuntuone ate 66% of my battery in one hour.
 * sinzui restarts to kill procs
<leonardr> ok, bombs away
<jml> gary_poster, benji: help requested re https://code.launchpad.net/~jml/launchpad/what-is-in-the-web-ui/+merge/52594 (any other zope experts or interested punters welcome too)
 * jml is off for the evening
<jml> g'night all. happy hacking.
<bac> hi sinzui
<lifeless> lets play spot the deploy: http://webnumbr.com/launchpad-critical-bugs
<sinzui> hi bac: sorry. I was dealing with a run away proc. you have my attention now
<bac> hi sinzui.  np
<bac> i was just going to ask you some yui test questions.
<bac> i'm trying to get some debugging output.  Y.log() disappears.  i tried console.log() too but it doesn't output to the browser console.  any tips
<sinzui> um. no. are you saying the out never arrives, or that something clears it?
<bac> expected to see it in the browser console but see nothing
<bac> sinzui: oh, wait, i do see it in the console div, amongst the other output.  i was looking in the error console
<sinzui> oh, I was nervous asking if you were sure you were looking at the log
<thumper> sinzui: got a minute to mumble?
<bac> sinzui: deryck pointed out that commenting out the console in the test_.js dumps everything into the console and is interogatable.
<sinzui> thumper: I can talk now
<lifeless> sinzui: https://code.launchpad.net/~sinzui/lazr.restful/json-not-xhtml-0 looks like it isn't actually landed
<sinzui> lifeless: I do not know what is up with that. I do not see it, but I think I pushed it.
 * sinzui tries again
<lifeless> sinzui: I looked at https://code.launchpad.net/~lazr-developers/lazr.restful/trunk
<sinzui> so did I ;(
<lifeless> wgrant: remerging db-devel would be cutting it -real- fine
<thumper> leonardr: mumble?
<leonardr> thumper, yes
<sinzui> lifeless: sorry, I think I had pushed to the wrong location :(. The fix is in rev 178, version 0.17.4
<lifeless> sinzui: thats cool; are you going to do the lazr.restful update in lp itself?
<sinzui> leonardr: can you look at the revs in https://code.launchpad.net/~lazr-developers/lazr.restful/trunk ? I am having a panic attack. I think something you landed today is missing because of my rememerge
<leonardr> sinzui: it's possible, i'll look
<sinzui> lifeless: please wait for leonardr to be sure I did not loose something
<lifeless> sinzui: of course
<jcsackett> lifeless: can you take a look at the reply to the mp of mine that stub added you to? and if you have any further concerns holding it back from being landed, let me know?
<lifeless> sinzui: I'm hoping we can get the oem bug through qa for the deploy on thursday
<lifeless> jcsackett: if its a db patch, the requirement is stubs approval - I just audit so I'm in the loop on db schema changes
<lifeless> jcsackett: per the db schema change policy
<jcsackett> lifeless: dig, i just saw a pending review for you that i didn't add. :-P
<lifeless> jcsackett: so, you *should* have added it :) but you don't need my vote
<jcsackett> right on. :-)
<jcsackett> lifeless: did i read an earlier comment correct, that we may be remerging db-devel before the rollout?
<lifeless> no
<jcsackett> oh good.
<lifeless> wgrant was expressing enthusiasm for getting more fixes in
<jcsackett> lifeless: dig. i had some concern there.
<leonardr> sinzui: i think we may have lost your *own* changes
<sinzui> leonardr: I really think I lost that last revision. if you have a copy of lazr-restful from you push today. I think you want to repush, maybe with overwrite. bzr cdiff -r176..-1 | less -r only shows my changes
<sinzui> leonardr: but...I though I had pushed my changes yesterday, lifeless reported they were not there. So I remerged and pushed. So regardless of what happened. I want to be sure both out changes are in the branch
<wallyworld> leonardr: https://code.launchpad.net/~wallyworld/launchpad/recipe-request-builds-existing/+merge/52389
<leonardr> sinzui: i'm on revision 178 and my change to _resource.py is present
<sinzui> leonardr: raise e?
<leonardr> yeah
<leonardr> raise e within the else clause
<sinzui> okay I see that, We lost your commit message though
<sinzui> see the bottom of the page: https://code.launchpad.net/~lazr-developers/lazr.restful/trunk
<sinzui> leonardr: I would be happy if you could push your copy to be trunk, then restart my merge so that the changelog is correct
<bac> sinzui: have time to talk on mumble briefly?
<leonardr> sinzui: i'm not sure how to do that. 'bzr push lp:lazr.restful' says no new revisions to push
<sinzui> oh
<bac> brb
<sinzui> leonardr: does push --overwrite do nothing?
<leonardr> let's see
<sinzui> leonardr: bzr log -n 0 -r 177..178 shows my remerge swallowed out commit message. So I do not think we lost code, I just messed up the log of histrory
<leonardr> i agree
<leonardr> --overwrite also does nothing
<sinzui> :( I suck
<sinzui> leonardr: thanks for your time. I will try a reverse merge to undo this
<leonardr> do you want my original commit message?
<leonardr> wallyworld: are we talking about 2 extra web service calls, or 2n extra calls?
 * wallyworld thinks for a second
<wallyworld> leonardr: akin to 2n - 1 per archive and 1 per distroseries
<leonardr> i think the current approach is fine
<lifeless> sinzui: you don't suck, the trunk isn't setup correctly
<wallyworld> leonardr: cool. it just seems kludgy
<bac> sinzui: when you have time i'd like to ask you about how yui unit tests find the local JS resources.
<lifeless> it needs the setting in bzr for no history changes to be set
<leonardr> it is, but kludgy is better than super-inefficient, and we don't have a good alternative right now
<wallyworld> leonardr: it would be nice to somehow be able to specify what attributes we wanted to return for elements in a collection_link
<wallyworld> when the link is fetched
<leonardr> wallyworld: yeah, that's part of expand-and-filter
<wallyworld> leonardr: is that something currently in progress?
<leonardr> wallyworld: no, i planned to work on it but then the teams were rearranged. it needs to be scheduled as a large new feature
<wallyworld> leonardr: imho it a pretty important feature as we move towards more xhr support in lp, so it would be good to see it on the radar :-)
<lifeless> wallyworld: leonardr: can I but in and ask about the context ?
<lifeless> s/but/butt/
<wallyworld> lifeless: i'm retrieveing a collection of build records via the ws api from javascript
<wallyworld> lifeless: by getting a collection link
<wallyworld> lifeless: the records contain links to other objects
<wallyworld> but i need attributes on those objects and the returned collection elements just have the links
<lifeless> wallyworld: planning to, or are ?
<wallyworld> lifeless: i have a mp for review
<lifeless> I mean, is this in trunk already ?
<lifeless> wallyworld: so, this sets of red flags for me.
<wallyworld> me to but there's no infrastructure to support what i need yet
<lifeless> what are the alternatives ?
<wallyworld> see leonardr's comment about expand and filter support
<wallyworld> lifeless: i have an alternative - and am using it in the mp
<lifeless> sorry, I wasn't clear
<lifeless> what are the alternatives to crippling our appservers when users hit this page
 * lifeless adds hyperbole to help the discussion
<wallyworld> lifeless: but we don't cripple the app servers - i'm not making N+1 calls - the mp uses another way of doing it
<leonardr> lifeless: wallyworld is using url hacking to make 1 request instead of 2n
<lifeless> ok, whew.
<lifeless> thats brilliant
<leonardr> i told him that was the right thing to do
<wallyworld> lifeless: give me some credit - i'm not that stupid :-)
<sinzui> leonardr: lifeless the history is back in the correct order. My remerge did not increment version.txt so my log fix really was wanted
<lifeless> wallyworld: well, I read '10:41 < wallyworld> lifeless: i have a mp for review' as meaning the 2n approach
<wallyworld> lifeless: i was just after a +1 from leonardron the kludgy approach used
<lifeless> wallyworld: leonardr: thanks for humouring me, totally agree we should use the kludge
<wallyworld> lifeless: fair enough :-)
<leonardr> cool
<lifeless> sinzui: great, so - are you going to do the lazr.restful upgrade for lp ?
<wallyworld> lifeless: yeah, but we will run into more cases requiring it as we expand out xhr support, so I +1 on leonardr implementing expand and filter support into lazr restful :-)
<wallyworld> if we ever get the time to do such things :-)
<leonardr> sinzui, lifeless: i have a branch in ec2 land that will upgrade launchpad to 0.17.3, but you probably want 0.17.4
<sinzui> lifeless: I do not what what i do for that. cut a release, push it to pypi, increment egg version in buildhout?
<sinzui> leonardr: which branch? I will cargo cult
<leonardr> sinzui: the one you were looking at earlier
<leonardr> https://code.launchpad.net/~leonardr/launchpad/bug-106338
<sinzui> ah.
<leonardr> sinzui: here's my 'release script': http://pastebin.ubuntu.com/577617/
<sinzui> excellent. thank you!
<lifeless> sinzui: cut a release; add to download-cache and commit, then update versions.cfg in lp and merge that to devel.
<lifeless> gary_poster: hi
<gary_poster> lifeless, hi
<lifeless> hi
<lifeless> this bug I filed
<gary_poster> bug 731099?
<_mup_> Bug #731099: structural subscriptions have poor queries on bugs with many bugtasks <story-better-bug-notification> <Launchpad itself:Triaged> < https://launchpad.net/bugs/731099 >
<lifeless> yeah
<lifeless> at the risk of being annoying, I was wondering if you had the time to chat - just here - about it interactively
<gary_poster> subscription portlet shouldn't use the same code path anymore.  I haven't checked.  That's a bug IMO if it does
<gary_poster> sure
<lifeless> so what I did locally was add 200 bugtasks to a single bug
<wallyworld> leonardr: you ok to approve the mp so i can push it through ec2? thanks :-)
<lifeless> the main bug html rendered without the query I saw - it was the async population of the subscribers portlet that did the big query
<gary_poster> ah-ha
<gary_poster> if you get me that url I'll look at it lifeless.  essentially, here's where I am:
<lifeless>  /bugs/1/+bug-portlet-subscribers-content
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> is what my local log shows
<gary_poster> 1) the portlet should not use that "let's send things" query.  There's a much simpler approach that gives a reasonable approximation.
<gary_poster> Further, Jono and I have tentatively agreed that the subscribers info should probably be deleted because it is no longer accurate enough or interpretable-by-humans enough to be of value, but that's a but of a by the way.
<gary_poster> 2) the best way to get rid of the doubling in query size is to add with support to Storm (which I'd love to get to, but is not currently a priority).  What we can do now without losing functionality is to  make an intermediate query so we don't have to duplicate things.  As I said in the bug report, I'm not convinced it is the right balance of things, but it's not an inner loop, so it probably won't make that big of a
<gary_poster> that was probably truncated :-/
<gary_poster> As I said in the bug report, I'm not convinced it is the right balance of things, but it's not an inner loop, so it probably won't make that big of a diff either way
<lifeless> gary_poster: it looked to me eyeballing the query that the inner clauses are able to be optimised out - but that may be my eyes glazing over at hand evaluating OR vs AND precedence
<gary_poster> There are two possibilities in your eyeballing
<gary_poster> that I see
<lifeless> gary_poster: specifically, what I saw was (project-or-product and (status-of-filter) OR (project-or-product and (status-of-filter)
<gary_poster> 1) what you said
<lifeless> gary_poster: and I thought it might be factorable to (status-of-filter and (project-or-product OR project-or-product OR ..)
<gary_poster> 2) I missed something
<gary_poster> heh, I was going to go on for a while so I went for summaries ;-)
<lifeless> :)
<gary_poster> lifeless, is there a pertinent bit in the stuff you filed that's an example?  Let's see if I can explain it or not
<lifeless> context -
<lifeless> L
<lifeless> EFT JOIN BugSubscriptionFilterTag ON BugSubscriptionFilterTag.filter = BugSubscriptionFilter.id WHERE StructuralSubscription.id IN (1)
<lifeless> repeating bits -
<lifeless> AND ((StructuralSubscription.product = 4 OR StructuralSubscription.project = 4) AND (BugSubscriptionFilt
<lifeless> erImportance.importance = 20 OR BugSubscriptionFilterImportance.importance IS NULL) AND (BugSubscriptionFilterStatus.status = 10 OR BugSubscriptionFilterStatus.status IS NULL)
<lifeless> OR
<lifeless> (StructuralSubscription.product = 130) AND (BugSubscriptionF
<lifeless> ilterImportance.importance = 5 OR BugSubscriptionFilterImportance.importance IS NULL) AND (BugSubscriptionFilterStatus.status = 10 OR BugSubscriptionFilterStatus.status IS NULL)
<lifeless> (repeats with only the product clause changing in each repeeat
<gary_poster> lifeless, that's because the individual bugtasks have those values, individually
<lifeless> ok
<lifeless> so we could group this into one complex thing per (importance,status) tuple
<lifeless> which would be about 25 clauses on pathological bugs
<lifeless> gary_poster: I think we should be data driven about whether this matters
<lifeless> gary_poster: the /bugs/1/+bug-portlet-subscribers-content url on a machine with the filters will let us gather that data
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<gary_poster> lifeless, right; I thought about the optimization and rejected it.  I didn't expect anything like 200 bugtasks.  bug 1 is like that, I take it?
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> gary_poster: 1 is small -
<lifeless>       1 |    21
<lifeless>  230350 |   159
<gary_poster> my goodness
<gary_poster> why, out of curiosity?
<lifeless> when I get the main page for bug 230350 rendering, I can tell you
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<gary_poster> :-P
<lifeless> It looks like a very meta bug
<lifeless> packages were missing a field, and it got batch filed on all the packages with that issue.
<lifeless> we could have bugs with thousands of tasks of this nature
<gary_poster> wow, blech
<gary_poster> isn't that indicative of a bug in whatever is using LP?
<gary_poster> in any case...
<lifeless> it *is* what multiple tasks are intended for.
<lifeless> its just a scale that us devs don't inuit
<lifeless> *intuit*
<gary_poster> IOW, it is reasonable to expect one bug to have action items in 150+ projects?
<gary_poster> This is pure curiosity
<gary_poster> I'm trying to assemble my action item summary in a separate "thread"
<gary_poster> there are three action items I can take
<lifeless> I believe 159 is our largest bug at the moment
<gary_poster> Right, my question is not whether they exist, but whether they actually serve a useful purpose
<wgrant> FWIW, performance was fine at the time the bug was filed.
<wgrant> But subscription noise stopped us from using them.
<lifeless> but this is also self limiting at the moment - the bug page scales at 50ms per task *after* 5 patches to improve it, so falls over well before larger split-outs
<leonardr> wallyworld: sorry, didn't realize you wanted me to review the whole branch
<gary_poster> 1) if you file a bug about /bugs/1/+bug-portlet-subscribers-content tickling this code path and assign it to me, I'll promise to see if I can address it quickly.  If I can't address it quickly, I'll push it off till bug rotation.
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> gary_poster: I'll retitle the current bug, because it should be more clear anyway
<wallyworld> leonardr: sorry. you don't have to. i can get the ocr to do it no problems.
<leonardr> wallyworld: that would be better, i'm eod
<wallyworld> leonardr: will do.
<gary_poster> 2) if there's an indication that this code path is causing timeouts, I can try to do the optimization you describe.  Even though I'm fairly skeptical of its utility, I've been wrong plenty before.
<lifeless> gary_poster: I don't know that there are timeouts on this yet, https://bugs.qastaging.launchpad.net/bugs/230350/+bug-portlet-subscribers-content renders ok
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<lifeless> gary_poster: and staging is down as you noted
<gary_poster> 3) I continue to want to reduce the duplication of the SQL size as I was describing in that bug, but I want to do that with the WITH statement.  It is an easy experiment to see if an intermediate query helps in this case, though, so that's another thing we can try.
<gary_poster> If you want me to prepare a branch with that change for you to test with (in the 200 bugtask page), I could do so in a min of half hour and a max hopefully of 2 hours.
<gary_poster> Meanwhile, I have to run.  Please ping me with any bugs you create
<gary_poster> bye everyone
<lifeless> gary_poster: ciao
<lifeless> gary_poster: I've retitled the bug, I think high pri is appropriate and we should see how it behaves on that bug's subscriber portlet on staging when its back
<lifeless> gary_poster: no point polishing something that is ok for now
<sinzui> lifeless: leonardr: My bad day continues. I seem to have destroy my lp tree in an errant branching of launchapdlib
<sinzui> please bear/bare/bair/bar with me as I look for where I placed my brain
<lifeless> sinzui: ><
<jcsackett> sinzui: i do not appear to have sound. give me a moment.
<huwshimi> sinzui: I appear to be getting very intermittent sound from everyone, restarting mumble
<jcsackett> yay! mumble issues for everyone.
<huwshimi> sinzui: ugh this is useless
<sinzui> huwshimi: I called an end to the stand up. I hope we can talk tomorrow and the monkey god of software is kind
<huwshimi> sinzui: Ah ok, I'm not sure what went wrong. It might have been network issues or something
<wgrant> huwshimi: jcsackett was having similar issues (although it eventually came mostly good), so I suspect it's not your fault.
<jcsackett> sinzui: i found the test in question.
<jcsackett> wgrant: huwshimi: i find mumble is frequently having issues.
<sinzui> jcsackett: does it look sane and simple. I think I was sane that day I wrote it
<jcsackett> sinzui: it looks fairly direct, yes.
<sinzui> huwshimi: wgrant: yes sound is very dodgy now. At least the new sound-indicator tells me I have not in/out.
<sinzui> Oh, but it does not tell me when mumble + pulse will throw a wobbly and take my CPU with it
<wgrant> sinzui: The U1 crasher is fixed now, btw.
<wallyworld> thumper: leonard has eod. he's fine with the approach since we don't have expand and filter implemented in ws yet. can you +1 it?  https://code.launchpad.net/~wallyworld/launchpad/recipe-request-builds-existing/+merge/52389
<sinzui> fab. I was thinking of not taking updates for few days
<lifeless> meh, bug text wrapping breaks my brain
<lifeless> wgrant: hey
<lifeless>                ->  Index Scan using securesourcepackagepublishinghistory_sourcepackagerelease_idx on sourcepackagepublishinghistory  (cost=0.00..0.50 rows=1 width=4) (actual time=0.011..0.011 rows=0 loops=1276160)
<lifeless>                      Index Cond: (sourcepackagepublishinghistory.sourcepackagerelease = sourcepackagerelease.id)
<lifeless>                      Filter: ((sourcepackagepublishinghistory.archive = ANY ('{1,534}'::integer[])) AND (sourcepackagepublishinghistory.distroseries = 106) AND (sourcepackagepublishinghistory.component = 3) AND (sourcepackagepublishinghistory.status = 2))
<lifeless> whats that filter checking for in english
<wgrant> lifeless: Published sourcepackagereleases in universe, natty(?), primary and partner.
<lifeless> hmm, I think this is the issue though
<lifeless>                      ->  Index Scan Backward using bugtask_importance_idx on bugtask  (cost=0.00..116380.60 rows=110413 width=276) (actual time=9.909..2039.202 rows=3788 loops=1)
<lifeless>                            Filter: ((distribution = 1) AND (status = 10))
<lifeless>                      ->  Index Scan using sourcepackagerelease_sourcepackagename_idx on sourcepackagerelease  (cost=0.00..4.77 rows=21 width=8) (actual time=0.508..34.643 rows=337 loops=3788)
<lifeless>                            Index Cond: (sourcepackagerelease.sourcepackagename = bugtask.sourcepackagename)
<wgrant> What's this from?
<wgrant> Oh.
<wgrant> Component bug searches?
<lifeless> https://bugs.launchpad.net/launchpad/+bug/731679
<_mup_> Bug #731679: distribution:+bugs timeout <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/731679 >
<lifeless> I'm trying to figure out the intent, yes
<lifeless> I think its 'new bugs only'
<lifeless> but https://bugs.launchpad.net/ubuntu/+bugs?orderby=-importance&search=Search&field.status=NEW is snappy
<wgrant> Which OOPS is that from?
<lifeless> wgrant: the first one
<wgrant> I'd say it was from DistroSeries:+bugs...
<wgrant> oh, I see.
<wgrant> Of course.
<wgrant> A component filter should be all that is necessary. Have you tried that?
<wgrant> Yeah, that works.
<wgrant> Where "works" is "times out"
<wgrant> https://bugs.launchpad.net/ubuntu/+bugs?field.component=3
<lifeless> https://bugs.launchpad.net/ubuntu/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status:list=NEW&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_supervisor=&field.bug_commenter=&field.subscriber=&field.component=1&field.component-empty-marker=1&field.tag=&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_m
#launchpad-dev 2011-03-09
<lifeless> bah
<lifeless> I selected a component of main
<lifeless> status new
<lifeless> 5.53 seconds
<wgrant> main is tiny.
<wgrant> Doesn't count.
<LPCIBot> Project devel build #518: FAILURE in 4 hr 48 min: https://hudson.wedontsleep.org/job/devel/518/
<lifeless> ok
<lifeless> universe works
<lifeless> and its the - guess what - spph join thats the problem
<wallyworld> sinzui: if you are still online, can you tell me how to construct a PrettyOverlay but with a different BOUNDING_TEMPLATE. I'm not sure how to override the default one
<sinzui> oh.
<sinzui> That just went over my head. I think I got a nose bleed thinking about it
<wallyworld> sorry :-)
<wallyworld> i want to replace the close (x) with a pair of (tick) (cross) buttons
<wallyworld> since the user needs to be able to poke some widgets on the overlay and then either accept or cancel
<wallyworld> i could get the close div after construction and insert a new node
<wallyworld> that would do it
<lifeless> wgrant: whats silly is that we get back no packages
<wgrant> getBuildRecords is a mess :(
<wgrant> lifeless: ... oh?
<lifeless> wgrant: see actual time=0.011..0.011 rows=0 loops=1276160)
<lifeless> perhaps thats just misleading
<lifeless> we do get 76 bugs
<wgrant> Ursinha: Hi.
<Ursinha> hi
<Ursinha> it's carnaval, so this might be an automated response :P
<wgrant> haha, OK.
<Ursinha> wgrant, what's going on?
<thumper> wallyworld: I'll take a look
<wallyworld> thumper: ta
<wgrant> Ursinha: I was just wondering what sends the daily error reports these days. launchpad-errors-yesterday in lp:oops-tools seems to be out of date.
 * Ursinha checks lpqateam's crontab
<lifeless> Ursinha: is matsubara_ around ?
<Ursinha> lifeless, unlikely, 9pm of a carnaval's tuesday night
<lifeless> heh, ok :)
<Ursinha> wgrant, so, it's a script called report (heh)
<Ursinha> generated automatically by oops-tools
<Ursinha> wgrant, why?
<Ursinha> lifeless, do you need something specific?
<lifeless> Ursinha: I have a oops tool merge proposal, hadn't heard anything in a few days
<wgrant> Ursinha: I want to change a prefix.
<wgrant> Was hoping to change it there too.
<Ursinha> wgrant, hm, prefixes now are changed using an admin django page
<wgrant> Ah, handy.
<Ursinha> prefixes are loaded dynamically from oops-tools database
<Ursinha> I'm sure there's a wiki page explaining that
 * Ursinha looks
<lifeless> there is
<lifeless> matsubara mailed it to me :)
<Ursinha> wgrant, https://dev.launchpad.net/Foundations/QA/OopsToolsSetup
<wgrant> Ursinha: Thanks.
<wgrant> Who has access to add a new prefix to a report?
<lifeless> ok, got that down to 800ms.
<lifeless> nice
<wgrant> Nice.
<wgrant> What did you change?
<wgrant> Hm, build breakage.
<wgrant> Same one us hudson.
<wgrant> Intriguing.
<wgrant> Because that's a spurious failure.
<wgrant> Must be time-based.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/731679/comments/3
<_mup_> Bug #731679: distribution:+bugs timeout searching for bugs by component <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/731679 >
<wgrant> I've seen this happen before, but dismissed it as coincidence :/
<lifeless> time to dig into storm, I just wish it was more pleasant
<wgrant> lifeless: Can't you easily word that as a subselect?
<lifeless> wgrant: you'd think so
<wgrant> Does it break the plan?
<thumper> sinzui: still around?
<sinzui> I am
<thumper> I'm just encoding that video :)
<sinzui> thumper: I was just about to have dinner. I will return in about an hour.
<thumper> sinzui: ack
<wgrant> wtf
<lifeless> ???
<wgrant> This test failure.
<wgrant> It is real.
<wgrant> But completely unrelated.
<wgrant> 12555 must have changed test order somehow.
<wgrant> 'bin/test -m lp.services.job.tests.test_runner -1cvvt TestTwistedJobRunner.test_timeout' reproduces the failure even before then. Dropping the -m works fine. So it needs some test module loaded before it will work.
<wgrant> Hmm, ampoule can't import _pythonpath.
<lifeless> mtaylor: does mysql 5.1 support 'with foo as (...) select ... where my.id in (select thing from foo)' ?
<wgrant> Gnrgh.
<lifeless> ?
<wgrant> This test. A subprocess fails if lp.testing imports anything from canonical.testing.layers.
<wgrant> Must be a circular import, I guess, but Twisted is good at swallowing details.
<lifeless> eugh
<lifeless> \o/ 176 /    0  LanguageSet:CollectionResource:#languages
<lifeless> 58 /  155  BugTask:+index
<mtaylor> lifeless: I have never seen 'with foo as (...)' - but I could of course have just missed it
<lifeless> mtaylor: thanks, I will write this as a postgresql only tested feature then :)
<mtaylor> lifeless: my pleasure!
<wgrant> Excellent, the top 50 exceptions now encompass all with multiple occurrences.
<lifeless> https://bugs.launchpad.net/storm/+bug/731739
<_mup_> Bug #731739: with foo as (select...) SELECT ... not supported <Storm:New> < https://launchpad.net/bugs/731739 >
<lifeless> now to figure out how to do a snapshot of storm
<wgrant> You've done it already?
<lifeless> yes
<lifeless> sufficient for our needs anyhow
<lifeless> https://code.launchpad.net/~lifeless/storm/with/+merge/52630
<StevenK> lifeless: You linked to the Postgres 9.0 docs, it's supported earlier?
* StevenK changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> StevenK: It's new in 8.4.
<StevenK> That's probably okay, then.
<lifeless> StevenK: its supported wherever lp is supported
<thumper> I seem to be getting  lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout failing every time on EC2
<lifeless> wgrant is working on that
<wgrant> thumper: Yes, I'm looking at that.
<thumper> others getting this?
<thumper> wgrant: awesome, thanks
<wgrant> About to give up and just work around it, though.
<StevenK> Jenkins is to
<wgrant> Ampoule is almost more evil than I am willing to deal with.
<thumper> wgrant: want to talk through it?
<wgrant> thumper: That test breaks with an unknown subprocess failure if lib/lp/testing/__init__.py imports anything from canonical.testing.layers (r12555 introduces one such import).
<wgrant> thumper: The test also fails when run on its own with -m lp.services.job.tests.test_runner, even before r12555.
<thumper> hmm...
<wgrant> But with a slightly different failure.
<thumper> it was failing intermittantly
<wgrant> It always has, yes.
<thumper> and there is a bug for that
<wgrant> There is.
<wgrant> Bug #505913
<_mup_> Bug #505913: TestTwistedJobRunner.test_timeout fails intermittently <lp-foundations> <spurious-test-failure> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/505913 >
<thumper> how are you working around it?
<wgrant> Removing the import.
<wgrant> Running with -m causes ampoule to fail to import _pythonpath. If I stop it from attempting to use _pythonpath, we get the same error that r12555 introduces.
<wgrant> Any useful diagnostic information is swalled by Twisted, though.
<wgrant> +ow
<thumper> hmm...
<wgrant> It also still happens if I stop the tests from running in a subprocess.
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<wallyworld> thumper: it seems one is not allowed to modify a CollectionField attribute via the web service? i get a "You tried to modify a collection attribute" error
<wgrant> That's right.
<wgrant> You might want a List instead.
<wgrant> eg. IBug['tags']
<wallyworld> wgrant: thanks. will have a look
<wgrant> wallyworld: Modifying CollectionFields is hard... what does it mean to remove an object from a resultset?
<thumper> wallyworld: hmm
<thumper> wallyworld: I'm sure leonardr suggested to me that we'd need add and remove methods to post to.
<thumper> wallyworld: or perhaps an update?
<thumper> wallyworld: you're one of the first attempting this
<wallyworld> wgrant: i thought a collectionfield attribute were like lists
<thumper> wallyworld: so you get to blaze the way
<wallyworld> thumper: i'm replacing the entire field's value so i may as well change it to a List
<wgrant> wallyworld: Note that it changes the representation significantly; it becomes inline instead of a link.
<wgrant> This has performance implications.
<wallyworld> wgrant: i'm not familiar with the terminology in this context - inline vs link
<thumper> wallyworld: I think you want an update method for the API
<wallyworld> thumper: so leave it as CollectionField and use an update method? if List fields are mutable, given the size of the list is small (<20), does it make sense just to go with that for this case?
<wgrant> Is this for SourcePackageRecipe.distroseries?
<thumper> wgrant: it is for recipe build series
<wgrant> Right.
<wgrant> List is probably not appropriate there.
<wgrant> I would have (add|remove)Series named ops, I think.
<wallyworld> hmm. seems like overkill given the size of the collection and the fact that we really just replace one version with another
<wallyworld> by version i should say value
<wallyworld> add/remove is typically more suited to larger collection sizes
<thumper> I think updateSeries should be sufficient
<StevenK>     Store.find(LibraryFileAlias).set(content=None)
<StevenK> TypeError: unbound method find() must be called with Store instance as first argument (got SQLObjectMeta instance instead)
 * StevenK grumbles more
<thumper> StevenK: Store.of(something).find
<wgrant> StevenK: IStore(LibraryFileAlias)
<wgrant> Or that.
<StevenK> wgrant: I've reverted to getCandidates returns ids
<wgrant> wallyworld: Hmm, a collection can have custom named operations.
<wgrant> Possibly not collection fields, though... hmmm.
<wallyworld> wgrant: i'm grepping the code as we speak to figure out how to do it :-)
<wallyworld> i'll do  a named_post
<wallyworld> SPR.updateseries() or something
<wgrant> wallyworld: Right, that's easy enough, but from looking at lazr.restful it seems like it might be possible to do something on the collection itself.
<wgrant> But updateSeries() is probably simpler.
<wallyworld> i'll have a poke around. i'm generally against fine grained collection updates
<wallyworld> especially when the semantics of this is a complete replacement
<wallyworld> of the collection's values
<wgrant> Right.
<wallyworld> where it would make sense for example is adding/removing a single person from a team
<lifeless> wgrant: not only done, done and working.
<wgrant> lifeless: Excellent!
<lifeless> I can has reviewer? https://code.launchpad.net/~lifeless/launchpad/bug-731679/+merge/52631
 * StevenK looks
<StevenK> No reviewer for you. Come back when you have a diff
<StevenK> :-P
<lifeless> don't blame me
<StevenK> lifeless: Does it require the storm changes, or are those seperate?
<lifeless> requires em
<lifeless> I've a snapshot ready to commit to the downloadcache
<lifeless> StevenK: it has a diff
<StevenK> Yes, I'm reading it
<lifeless> thanks!
<StevenK> lifeless: orig_store, and then you call orig_store.with_(with_clause) when you'd already set store = store.with_(with_clause)?
<wgrant> That version number looks like a lie.
<lifeless> StevenK: different with_clause
<StevenK> And yes, I'd prefer that the storm changes get approved and landed first
<lifeless> StevenK: note the for loop
<lifeless> StevenK: not going to happen
<lifeless> StevenK: storm is basically stalled.
<lifeless> StevenK: I've 0 intention of letting us stall just because its stalled.
<StevenK> Orsum
<lifeless> feel free to review my storm changes though
<StevenK> thumper: https://code.launchpad.net/~lifeless/launchpad/bug-731679/+merge/52631 when you have a tick
<lifeless> but its an odd coding style there
<StevenK> lifeless: I glanced at them, but I'm not comfortable reviewing
 * thumper has two ticks
<lifeless> thumper: thanks
<thumper> lifeless: what was the concern addressed on irc?
<lifeless> thumper: which concern?
<wgrant> I still think that version number is wrong.
<thumper> lifeless: the one that StevenK says in the code review
<StevenK> thumper: I'd prefer the changes to Storm landed
<StevenK> [14:28] < lifeless> StevenK: storm is basically stalled.
<StevenK> Ergo ...
<wgrant> I don't care if they land, but that 0.18.0.99 is deceptive.
<lifeless> wgrant: pick a version, I'll roll with it
<lifeless> wgrant: note that it does string algebra
<lifeless> wgrant: so 0.18.0.0r14 will be stupidly painful to make
<wgrant> lifeless: We normally use a suffix like -lp-r1234
<lifeless> wgrant: -how-
<wgrant> lifeless: How are you building it?
<lifeless> wgrant: like I say, storms setup.py / __init__ do algebra on a constant
<lifeless> wgrant: setup.py sdist
<StevenK> Store.of(LibraryFileAlias).find().set(content=None)
<wgrant> All our previous custom Storms have had a good suffix.
<StevenK> That doesn't work either, am I doing it wrong?
<wgrant> StevenK: find(LibraryFileALias)
<wgrant> lifeless: What about 'setup.py egg_info -b-lp-r1234 sdist'?
<lifeless> wgrant: that worked, I'll use it
<StevenK> wgrant: Still ends up with AttributeError: 'NoneType' object has no attribute 'find'
<lifeless> StevenK: IStore(LibraryFileAlias)...
<wgrant> Ah, didn't bother to look further than the find()
<lifeless> StevenK: Store.of() only applies to instances that have either come from a store or been added to a store.
<wgrant> Store.of(someobject)
<wgrant> IStore(someclass)
<StevenK> Ahh
<lifeless> StevenK: this was the only thing wrong with your example I think
<lifeless> StevenK: also, when something doesn't work, show how :)
<lifeless> thumper: thanks
<StevenK> IStore(LibraryFileAlias).find(LibraryFileAlias).set(content=None) results in:
<StevenK> CompileError: Don't know how to compile type Reference of <storm.references.Reference object at 0x6a30450>
<lifeless> StevenK: don't use reference columns
<lifeless> contentID
<wgrant> Huh, that should work.
 * wgrant blames the SQLObject layer.
<StevenK> Trying contentID
<lifeless> reference columns are basically broken in storm and upstream are not receptive
<lifeless> wgrant: no, its not sqlobject layer
 * wgrant gives up.
<StevenK> wgrant: Oh?
<wgrant> The ampoule thing.
 * wgrant is testfixing.
<StevenK> Rargh, how can setting content == NULL for every LFA *still* have SPRs being processed!
<lifeless> clearly something doesn't do what you expect
<StevenK> I'm tempted to blindly ignore 'has no DSC' and continue
<lifeless> wgrant: is 681770 a dupe?
<wgrant> lifeless: Already done.
<lifeless> cool
<StevenK> wgrant: Any thoughts?
<wgrant> StevenK: Can you push up your latest changes?
<StevenK> wgrant: Done.
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/testfix/+merge/52632
<wgrant> Looking at yours.
<StevenK> Same
<StevenK> thumper: https://code.launchpad.net/~wgrant/launchpad/testfix/+merge/52632
 * StevenK nails jtv to the Internet
<wgrant> lifeless: Thanks.
<StevenK> It probably won't help, though.
<jtv> StevenK: no, but at least you got some fun out of it
<lifeless> wth is this exposed  TemporaryStorageManager:CollectionResource:#temporary_blobs
<wgrant> Possibly only so apport can upload to it.
<wgrant> I can't think why an API client would want to read the collection.
<wgrant> StevenK: Is it only test_hourly_script failing for you?
<StevenK> wgrant: Yes.
<wgrant> StevenK: Are you killing the LFAs there?
<wgrant> Ah, yes.
<lifeless> wgrant: and yet, they are
<wgrant> lifeless: Grep logs and strangle them?
<wgrant> StevenK: I wonder if LaunchpadScriptLayer might not use the right DB.
<lifeless> wgrant: Revision 12550 can not be deployed: needstesting
<lifeless> Preload BinaryPackageNames before listing successful copies' displaynames.
<wgrant> lifeless: Fixed.
<StevenK> wgrant: I wonder if using IMasterStore would make any difference. I suspect not.
<lifeless> sigh
<lifeless> overengineering
<lifeless> https://code.launchpad.net/~gmb/launchpad/processapportblobjob-api-bug-513191/+merge/20209
<wgrant> StevenK: I doubt it.
<wgrant> lifeless: Ah, of course. That doesn't need a listable collection, though.
<wgrant> StevenK: Check the name of the DB that each uses.
<lifeless> wgrant: precisely.
<StevenK> wgrant: How can I do that?
<lifeless> StevenK: wgrant: tests that use IMasterStore need to use it consistently or they will break
<lifeless> you can't use IStore in any helper :)
<lifeless> ditto ISlaveStore
<wgrant> lifeless: This one runs a script.
<lifeless> wgrant: I also particularly love the 'last 50' statement which is a total lie.
<wgrant> lifeless: Ah, it's not actually a security hole, because the blobs don't have launchpad.View.
<wgrant> So the list is empty.
<lifeless> wgrant: but are filtered in the appserver
<wgrant> lifeless: Sure, so it's slow. But my main concern is that we aren't leaking them.
<lifeless> wgrant: sure, thats a concern too
<lifeless> just noting
<lifeless> this is basically utter crack
<wgrant> What happens if you remove the collection_default_content?
<lifeless> wgrant: look at the end of lib/canonical/launchpad/database/temporaryblobstorage.py
<wgrant> Or does export_webservice_collection require it?
<lifeless> wgrant: needs it AFAIK
<lifeless> I've a faster query
<wgrant> lifeless: Don't use it.
<wgrant> Return an empty list instead.
<mwhudson> wgrant: wow, r12557 is >:(
<wgrant> mwhudson: YES.
<mwhudson> did you annotate for great justice?
<wgrant> Which bit?
<mwhudson> who added the flamingly incorrect comment
<wgrant> The circular import one?
<wgrant> That was added in the problematic revision.
<mwhudson> ah ok
<mwhudson> anyway, i have a bbq to get to, better than being frustrated by software
<lifeless> wgrant: you've checked you don't see the blobs
<wgrant> So, I spent a couple of hours trying to work out what was up, but I only ended up finding out that buildout, Twisted and Ampoule all need to DIE.
<lifeless> wgrant: right ?
<wgrant> lifeless: Yes.
<lifeless> wgrant: but - how does the 'is blob X processed' work then ?
<wgrant> lifeless: lazr.restful always checks for launchpad.View.
<wgrant> Regardless of what the security adapters say.
<wgrant> Anyone can view a blob, if they know its UUID.
<lifeless> wgrant: but does't htat also check for View ?
<wgrant>     <allow
<wgrant>       interface="canonical.launchpad.interfaces.temporaryblobstorage.ITemporaryBlobStorage" />
<wgrant> So, no.
<lifeless> glue the dots together gfor me
<wgrant> lazr.restful always checks for launchpad.View. But the security declarations can make anything publicly available.
<wgrant> lazr.restful checks for launchpad.View before it shows an object in a collection representation, that is.
<wgrant> For returning a single entry resoruce it will respect the security proxies.
<lifeless> ah
<lifeless> so collections are spethial
<wgrant> Yeth.
 * lifeless considers lp-landing this
 * wgrant defers getBuildRecords until tomorrow, and instead fixes a couple of quick checkwatches OOPSes, since even checkwatches is likely to incite less violence than getBuildRecords and test_timeout.
<wgrant> lifeless: What have you changed?
<lifeless> wgrant: lp:~lifeless/launchpad/bug-731736
<LPCIBot> Project devel build #519: STILL FAILING in 4 hr 49 min: https://hudson.wedontsleep.org/job/devel/519/
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-731736/+merge/52634
<wgrant> lifeless: As long as nothing else uses that property, and all the blob webservice tests pass. doit.
<lifeless> wgrant: I've changed one test which psases and the default collection method which there are no other references to outside of the webservice definition and implementation
<wgrant> Right.
<wgrant> Just checked the diff in LH
<lifeless> and there are no other hits for the webservice collection name either
<wgrant> You have my codestar.
<lifeless> heh
<lifeless> I was just self reviewing :) - thanks though
<wgrant> That works too.
<lifeless> sounds like bsg (codestar)
<wgrant> That was the feel I was going for.
<lifeless> so top timoeuts 1, 2, 3, all have patches pending deploy
<wgrant> And 1 and 3 are actually fixed.
<wgrant> 2 maybe not.
<lifeless> time to look at 4 again
<lifeless> bah
<lifeless> no oops in the report
<StevenK> wgrant: Grabbing the DB name in garbo-hourly is launchpad-main-master
<lifeless> hah
<lifeless> they missed a status
<lifeless> field.status: [u'NEW', u'INCOMPLETE_WITH_RESPONSE', u'INCOMPLETE_WITHOUT_RESPONSE', u'INVALID', u'WONTFIX', u'EXPIRED', u'CONFIRMED', u'TRIAGED', u'INPROGRESS', u'FIXCOMMITTED', u'FIXRELEASED']
<wgrant> StevenK: The postgres DB name is the interesting one. That's the store name.
<wgrant> lifeless: Just opinion missing?
<StevenK> wgrant: Did you miss my question of how to actually get it, then?
<lifeless> wgrant: yeah
<lifeless> wgrant: this is a /bugs/ search.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/716774
<_mup_> Bug #716774: Timeout on MaloneApplication:+bugs <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/716774 >
<wgrant> StevenK: Well, you could check the postgres log.
<wgrant> Rather than adding debug prints everywhere.
<StevenK> /var/log/postgresql/postgresql-8.4-main.log is fairly useless
<wgrant> Ah, you don't have statement logging on?
<StevenK> Doubtful
<wgrant> log_statement='all' in postgresql.conf
<wgrant> It's very handy.
<StevenK> wgrant: If that logs into the main log, I can't see it
<wgrant> /var/log/postgresql/postgresql-8.4-main.log
<StevenK> wgrant: Doesn't seem to work for me. :-(
<wgrant> StevenK: You put it at the bottom of /etc/postgresql/8.4/main/postgresql.conf, and it doesn't have any other log_statement lines?
<StevenK> Rargh, there's two!
<wgrant> Heh.
<StevenK> Now to distill the log
<StevenK> Looks to use the correct DB name
<wgrant> Both of them?
<StevenK> wgrant: They all use launchpad_ftest_23325
<thumper> StevenK: sorry was at guitar lesson, thanks lifeless for approving
<wgrant> :(
<wgrant> Can you see the LFA UPDATES?
<StevenK> Yes
<lifeless> and you commit ?
<StevenK> Yes
<wgrant> There is a transaction.commit()
<wgrant> But can you see it in the log?
<wgrant> Some layers might not have the transaction manager configured properly.
<wgrant> But it still fails on LaunchpadZopelessLayer :/
<StevenK> [2011-03-09 16:19:28 EST] launchpad_main@launchpad_ftest_23325 LOG:  statement: UPDATE LibraryFileAlias SET content=NULL
<StevenK> [2011-03-09 16:19:28 EST] launchpad_main@launchpad_ftest_23325 LOG:  statement: COMMIT
<wgrant> That seems pretty definitive.
<StevenK> The next bits are all garbo_hourly, so that's the script running
<wgrant> Oh.
<wgrant> Ha ha ha.
<wgrant> Ha ha ha ha ha.
<wgrant> I bet those SPRs have *no files at all*.
<wgrant> Fuck sampledata, seriously.
<StevenK> wgrant: 0 rows for id 17, for instance, so agreed
<wgrant> Looks like I need a sharper desk still.
<StevenK> I wonder if we can fix the query to ignore SPRs with no rows in SPRF
<wgrant> Difficult. Just set changelog=1 for all of the SPRs.
<wgrant> Much easier.
 * StevenK tries that.
 * StevenK attempts to not put "# <wgrant> Fuck sampledata, seriously." as the comment
<StevenK> wgrant: Thanks, changed pushed.
<wgrant> Great.
<StevenK> wgrant: Now approve it, damn it. :-(
<wgrant> lifeless: Could you mentor https://code.launchpad.net/~stevenk/launchpad/populate-spr-changelogs/+merge/52363?
<StevenK> wgrant: Sure, let me fix the comment.
<wgrant> StevenK: Thanks.
<StevenK> wgrant: http://pastebin.ubuntu.com/577696/
<wgrant> StevenK: Say something about why we can't expire the LFAs.
<wgrant> That's the really confusing bit.
<StevenK> wgrant: http://pastebin.ubuntu.com/577697/
<wgrant> StevenK: +1
<StevenK> wgrant: Changes pushed, thanks.
<huwshimi> Night people.
<wgrant> Night huwshimi.
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<jam> I thought there was a way to do an ssh tunnel to get direct psql access to the staging db. I was hoping to find the instructions to see if I could do something similar for loganberry's tuolumne db. Anyone have a link?
<jam> my google fu is weak (and it may be private docs)
<wgrant> jam: TLs can access the staging DB from devpad.
<wgrant> But I doubt !IS has access to loganberry.
<wgrant> Given that it's a prod machine.
<adeuring> good morning
<mrevell> Hallo
<henninge> mrevell: Hallo
<henninge> Moin adeuring ;)
<stub> jam: It is just standard port forwarding a local port to port 5432 on the target machine. Connect to it locally with 'psql --port=55432 prod_db'
<jam> stub: right. I thought so, but I wanted to check that you have to have a valid ssh login on the target machine.
<jam> And I was curious why local psql is better than just running it on the target machine
<jam> since you have a valid login already
<jam> more-interactive typing ?
<stub> jam: Yes, you need an account on the target machine or machine that can get past ipfilter to port 5432 on the target machine
<lifeless> + ident or similar auth permissions
<stub> jam: Using local psql makes it interactive. Typing into remote shells from the other side of the planet sucks big time.
<wgrant> jam is right next to the DC :(
<stub> So no benefit whatsoever :)
<jam> stub: well, it confirms what I thought, thanks
<jam> i agree that when you have 200+ms latency, typing is rather cumbersome
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #520: FIXED in 4 hr 50 min: https://hudson.wedontsleep.org/job/devel/520/
* jtv changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: jtv | https://code.launchpad.net/launchpad-project/+activereviews
<bigjools> ye
<bigjools> gads
<bigjools> 4h 50m
<StevenK> 20 minutes of that is check-out
<StevenK> Which is just *orsum*
<jml> good morning Launchpadders.
<lifeless> jml: I forgot two things to chat about the other day
<lifeless> jml: do you have 15?
<jml> lifeless: not immediately. I'll need a few minutes.
<lifeless> jml: is that few ~=5 or few ~=30 ?
<jml> ~10
<lifeless> ok, cool, I'll hang for that
<rvba> lifeless: Hi ... I'm trying to ec2 test on a very small branch of mine and I'm getting an error in a test bzr annotate told me you wrote.Julian thinks it might be spurious (and it certainly looks like it when I look at the test) but advised me to ask you anyway. Would you take a look:
<rvba> The whole log file http://dl.free.fr/pMlVoHAN6
<rvba> The only place where it seems there is a failure https://pastebin.canonical.com/44450/
<wgrant> That one is know to be spurious.
<wgrant> If it was the only failure, the branch is fine.
<wgrant> rvba: ^^
<wgrant> It shows up rarely enough that it's difficult to debug :/
<rvba> wgrant: ok
<rvba> I'll launch ec2 land (again) then
<rvba> thx
<wgrant> No point running it through ec2 again.
<wgrant> Just 'bzr lp-land' it.
<wgrant> That submits it directly to PQM.
<rvba> ok
<lifeless> gnight all
<bigjools> lifeless: review? :)
<gmb> Well, that's a new one: "TypeError: Result of expression 'protonode' [null] is not an object."
 * gmb goes grepping
<bigjools> rvba: did you lp-land your branch ok?
<rvba> bigjools: I relaunched the test suite to make sure I'll receive and email all right
<rvba> s/and/an/g
<bigjools> rvba: fair enough
<rvba> bigjools: I'm working on 727632. Reading though the schema change docs.
<bigjools> rvba: great
<maxb> Hmm, odd. Launchpad appears to no longer be notifying me of recipe build failures
<maxb> Or rather, it's not notifying me about recipe build failures due to depwait
<henninge> adeuring: http://people.canonical.com/~henninge/mockups/
<henninge> adeuring: see also my mail
 * henninge lunces
<henninge> lunches, even
<heninge-lunch> Hi deryck!
<deryck> Morning, heninge-lunch
<heninge-lunch> deryck: http://people.canonical.com/~henninge/mockups/
<heninge-lunch> deryck: see also my mail
<deryck> heninge-lunch, ok, looking at it all now.  Thanks!
<jml> deryck: is there an ACL for Fix Released like there is for wontfix etc?
<deryck> jml, yes.  Only bug supervisor, or the original reporter, IIRC.
<maxb> I have a pending application to ~loggerhead-team, who should I talk to to get a yay or nay?
<jml> deryck: what does it take to set it up
<jml> deryck: or is it enabled by default?
<deryck> jml, I think defining a bug supervisor is enough.  I'd have to look at code to remember for sure. ;)
<jml> deryck: that's fine.
<jml> deryck: thanks very much.
<deryck> jml, so I just looked to be sure, and it's just a check in canTransitionToStatus.  So nothing required.
<deryck> wanted to remember clearly anyway :-)
<jml> deryck: not even defining a bug supervisor?
<deryck> jml, no, not required.
<jam> I'm trying to qa bug #726985 but I'm unable to get any requests to succeed at "http://bazaar.qastaging.launchpad.net/"
<_mup_> Bug #726985: codebrowse OOPSes with GeneratorExit when connection closed early <qa-needstesting> <Launchpad itself:Fix Committed by jameinel> < https://launchpad.net/bugs/726985 >
<jam> When I go to: http://bazaar.qastaging.launchpad.net/~jameinel/+junk/foo/changes
<jam> It just issues a permanent redirect, *to the same URL*
<deryck> henninge, adeuring -- let's spend the standup talking about these mockups and come to some conclusion about them.
<adeuring> sure
<jam> jelmer: I tried to approve your change, but you deleted the merge proposal
<jelmer> jam: It was an uncontroversial one-liner so I just submitted it.
<jam> jelmer: well, maybe I'll just take back my approval then :)
<jelmer> jam: heh
<jelmer> jam: Sorry for the noise :)
<henninge> deryck: great idea
<wgrant> jam: qastaging codebrowse was never set up properly.
<jam> wgrant: good enough for me, I guess
<jam> I'm not really sure how to qa the "doesn't oops anymore" anyway
<jam> can we just set the bug to qa-ok?
<wgrant> jam: I QA'd the last one by poking around from some internal hosts, but for this I don't think it's worth it.
<wgrant> qa-untestable
<jam> fine by me
<wgrant> jtv: Any luck with QAing the DSD script?
<jtv> wgrant: I filed an RT.  I gather the losas are rather busy.
<wgrant> jtv: Can we qa-untestable it, given that it presumably won't be run unless you ask for it?
<jtv> wgrant: yes, come to think of it that does make sense now
<jtv> Done.
<wgrant> Thanks.
<deryck> adeuring, henninge -- let's delay the standup by 15 minutes, to give abentley time to see the mockups when he comes on.
<adeuring> ok
<allenap> Running `make JSFLAGS="--filetype=debug"` seems to invoke the correct bin/jsbuild command line, but it still uses "min" files. Does anyone know how I get it to DTRT?
<deryck> allenap, it should work for lp js code.  but utilities/yui-deps.py hard codes the min version.
<deryck> so yui code is still minified, but lp code is not.
<deryck> allenap, also, I've been doing filetype=raw, too.  not sure if that matters.
<deryck> and you can save time doing "jsbuild", i.e. make jsbuild JSFLAGS='--filetype=raw'
<allenap> deryck: Does this end up in lib/canonical/launchpad/icing/build/launchpad.js?
<deryck> allenap, yes
<allenap> deryck: It's still using the min files. I must be doing something stupid.
<deryck> allenap, make clean_js ?  maybe it's being dumb and doing nothing?
<bigjools> rvba: congratulations, your branch landed
<deryck> henninge, we can standup when you're ready.
<rvba> bigjools: well, one of them
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<rvba> bigjools: the other one triggered an error sending the email
<rvba> https://pastebin.canonical.com/44466/
<rvba> the strange thing is that the email conf should be the same for this branch and the one which landed
<bigjools> no idea then :/
<bigjools> use bzr lp-land for now
<rvba> bigjools: yes
<allenap> deryck: It doesn't seem to want to play.
<allenap> Any other ideas?
<bigjools> benji: yo
<benji> bigjools: yo
<bigjools> benji: the MP of mine that you approved the other day was targeted to devel instead of db-devel, can you do me a favour and tick off the newer MP please? https://code.launchpad.net/~julian-edwards/launchpad/publisher-config-db-schema/+merge/52530
<benji> sure
<bigjools> benji: I added some code that you might want to look at, there were no security declarations in the old branch :)
<benji> k
<benji> bigjools: done
<bigjools> benji: cheers
<bigjools> benji: good catch with the zcml
<allenap> deryck: Ah, I've found the culprit. utilities/lp-deps.py force minifies the lp code too. Btw, thank you for your help earlier.
<deryck> allenap, cool.  sorry on call now.  I really thought it used to work for me.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<deryck> abentley, hey.  How about bottom of the hour for a chat?  roughly 12-13 minutes from now?
<abentley> deryck: sure.
<deryck> abentley, ok, cool.  meet you in mumble then.
<deryck> jml, I'd like to book 15 minutes with you tomorrow, if possible.
<deryck> jml, for some voice chat time about how/when to turn of this translations story for your review.
<bigjools> jml: thanks for triaging my bug, I didn't mean to be lazy but I am a forgetful bastard
<jml> bigjools: that's OK. I actually thought there was a dupe (there wasn't), and have been using this as an opportunity to make sure all the ec2test bugs are tagged up
<jml> found a couple that have already been fixed
<jml> inspired to go on a bug fixing rampage later on
<jml> in my copious free time
<jml> speaking of which...
<jml> deryck: that'd be great
<jml> deryck: just put something on my calendar
<deryck> jml, ok, great.  thanks!
<bigjools> jml: what is this free time of which you speak?
<bigjools> is it somehow associated with the weird bright glowing ball I see in the sky sometimes?
<jml> bigjools: I think the medical term is insomnia
<jml> (or, more honestly, being naughty and doing something I want to do instead of something I should be doing)
<abentley> deryck: http://pastebin.ubuntu.com/577884/
<abentley> deryck: http://pastebin.ubuntu.com/577886/
<rvba> sinzui: I've started to work on bug 727632. I've worked on the first part of the bug we talked about yesterday: fixing distro's owner usage.
<_mup_> Bug #727632: distro and distro series pages do not specify their owner <derivation> <distributions> <series> <Launchpad itself:In Progress by rvb> < https://launchpad.net/bugs/727632 >
<rvba> sinzui: This is still in the works (no tests yet etc.) but I'd be happy if you could quickly review my branch and give me feedback (lp:~rvb/launchpad/distro-add-registrant).
<sinzui> rvba: okay. I will now
<deryck> abentley, so here's a bit on why we do namespacing the way we do:  https://dev.launchpad.net/JavaScriptReviewNotes#Naming%20Javascript%20Modules%20and%20Namespaces
<deryck> abentley, I think the "do this so you only have to change it in one place" comment is the best reason.  some of the rest seems suspect to me.
<deryck> abentley, YUI's way is to create the namespace and then address it directly:  http://developer.yahoo.com/yui/3/yui/#yuiadd
<abentley> deryck: are modules and namespaces orthagonal?
<deryck> abentley, by that you mean that what I name it in the add constructor has nothing to do with initialziing the namespace in said module?
<abentley> deryck: yes.  (except that we have a convention connecting them)
<deryck> abentley, yes.  you could have a single module that creates two different namespaces, for example.
<abentley> deryck: the UI docs you point at, the module is named 'mymodules-mod1' and the namespace is "Y.mynamespace".
<abentley> s/UI/YUI
<deryck> abentley, right.  we only it do it by convention, as you noted.
<leonardr> abentley, i have an exciting branch for your review: https://code.launchpad.net/~leonardr/lazr.restful/must-specify-version/+merge/52705
<abentley> deryck: okay, that explains why we specify the same string twice, then.
<abentley> leonardr: ack
<rvba> sinzui: thx for the review. I'll go on with my branch then and fix the -almost- same problem for distro series.
<sinzui> okay
<rvba> sinzui: one question though. When I 'make lint' I've got a warning about the fact that database/sampledata/lintdata.sql differs from database/sampledata/current.sql ... and indeed I had to modify current.sql ...
<rvba> the warning print also a few shell commands ... that will take me to erase my modifications if I follow them :-)
<sinzui> rvba: we do not edit those files for schema changes, though I realsied you did
<rvba> sinzui: oh I get it, they are supposed to be migrated I guess ...
<sinzui> rvba: after you` make schema`, run `make newsampledata`, then rename the new files to the old files
<rvba> sinzui: ok I get it. thx
<sinzui> rvba: I am certain you will see more than your changes. Lp engineers rarely reconcile their schema changes to sampledata
<rvba> sinzui: indeed, I saw a small difference.
<abentley> leonardr: does zope.Cleanup do something that testtools.TestCase.addCleanup can't do?
<abentley> leonardr: as_of is a nice improvement over current syntax, but did you consider allowing the default version to be 'devel'?
<leonardr> abentley: we'd still need as_of (since the existing syntax is ugly), it just wouldn't be required
<abentley> leonardr: agreed.
<leonardr> i'm pretty sure we explored that possibility and rejected it, but i don't remember the reasoning
<leonardr> i will say that with this implementation it's a lot easier to patch our existing code
<leonardr> than with an implementation in which our existing code will change meaning (rather than fail) if we forget to patch something
<abentley> leonardr: True, but we could generate before-and-after wadls to ensure the interface stayed the same.
<leonardr> abentley: i think there's not a whole lot of difference between these implementations, and this is the one lifeless prefers, so i'm going with it
<abentley> leonardr: okay.  Just seems like extra typing for no benefit.
<abentley> leonardr: Now, about zope.Cleanup.  Does that do something similar to testtools.addCleanup?
<leonardr> abentley: there's no net decrease in typing. if the default is devel, you're going to have to do the typing when you decide to cut a new release
<leonardr> i don't know what zope.Cleanup does. i'll check
<abentley> leonardr: Oh, so that was from a contributor?
<leonardr> abentley: i didn't write that code, if that's what you're saying. i just refactored it
<leonardr> and i remember what the problem was. zope.Cleanup takes care of the current interaction
<abentley> leonardr: There is a net increase in typing, because first you set export_as='devel', and then afterward you change export_as to '1.1'.
<lifeless> gmb: hi - qa plox - 12551
<leonardr> fair enough. i'd say that's worth it because you can grep for 'devel', see everything that hasn't been changed, and make a decision based on that
<leonardr> if there is something exported with no version specified, there's no way to know whether it's in devel or whether someone just made a mistake
<abentley> leonardr: okay.  That takes care of the real issues.
<gmb> lifeless: done
<jcsackett> i'm trying to use bzr pipeline, and i think i might have screwed something up. when i try to test a particular pipe via ec2 i get a NotABranch error.
<jcsackett> anyone have any thoughts?
<abentley> leonardr: you have some extra blank lines at 679 and 669.  It would also be nice if you documented every test.
<abentley> leonardr: But r=me.
<leonardr> ok
<leonardr> sinzui: i am getting totally random test failures when i try to land the branch we worked on yesterday
<leonardr> InternalError: current transaction is aborted, commands ignored until end of transaction block
<leonardr> this has happened twice, so i doubt it's spurious
<leonardr> is it possible that adding the validation for duplicate bugtasks could cause that error?
<leonardr> oh, and i'm getting the error locally, so it's not an ec2 problem
<leonardr> sinzui: http://pastebin.ubuntu.com/577933/
<leonardr> that's the source of the problem, but i don't know why it wouldn't have permission
<sinzui> leonardr: this looks like a db issue something related to security.cfg
<leonardr> is process_email running as a different database user?
<sinzui> leonardr: yes it is
<sinzui> leonardr: we just need to find the user and add the tables to the section.
<sinzui> I think we only need insert
<leonardr> i think we only need select, right?
<sinzui> oh, yes, select. you are right.
<leonardr> processmail user?
<sinzui> I think so
<leonardr> sinzui: the obvious thing doesn't help: http://pastebin.ubuntu.com/577935/
<leonardr> any ideas?
<leonardr> perhaps that's the wrong user??
<lifeless> abentley: trying to get as complete a deploy tonight as possible - can you qa Revision 12555 ?
<sinzui> leonardr: You had the identical error after making schema?
<abentley> lifeless: in the middle of doing so.
<lifeless> abentley: cool!
<lifeless> henninge: ping
<henninge> lifeless: pong
<henninge> ;)
<lifeless> henninge: if you're still around, and can qa https://bugs.launchpad.net/launchpad/+bug/705176 we should be able to include it in the deploy tonight
<_mup_> Bug #705176: Improve sharing information on translation pages <qa-needstesting> <upstream-translations-sharing> <Launchpad itself:Fix Committed by henninge> < https://launchpad.net/bugs/705176 >
<henninge> lifeless: I was just looking at it. something might be wrong but it is behind a feature flag anyway. so qa-ok
<lifeless> henninge: nice! thanks
<leonardr> ah, make schema
<leonardr> the missing ingredient?
<sinzui> leonardr: yes, after after the schema is setup, security.cfg is parsed and the users are setup
 * jml heading back home & thus offline
<leonardr> sinzui: unfortunately, yes, make schema doesn't stop the error
<lifeless> matsubara: hi
<lifeless> matsubara: you've probably got a tonne of mail
<lifeless> matsubara: but I've got a oops-tools branch you may not have seen
<lifeless> matsubara: which I'd love a hand with - test suite doesn't want to play for me
<leonardr> oh, wait, it's slightly different! binarypackage*name*
<leonardr> great, i just have to make schema over and over until i run out of errors. sinzui, is there a better way?
<lifeless> what is going wrong?
<leonardr> lifeless: i have a branch that adds some error checking to bugtask creation. the problem is that this requires bugtask creation to access database tables it formerly did not access
<leonardr> so when it happens, eg., over email, the database user doesn't have permission to run the checks
<sinzui> leonardr: I have not learned of a better way
<lifeless> ah checkwatches
<lifeless> security.py can update that
<lifeless> you need to point it at the ftest *template* db
<matsubara> lifeless, yep, saw that one. I'll review it today and investigate the test suite failure
<lifeless> matsubara: thank you!
<lifeless> matsubara: if you can take it over, that would be awesome.
<leonardr> lifeless: i'm intrigued. this is a substitute for 'make schema'? what's the name of the ftest template db?
<lifeless> launchpad_ftest_template
<lifeless> make schema builds that db
<lifeless> and as part of building it invokes security.py pointed at that template
<leonardr> that should speed up the process considerably, thanks
<lifeless> I don't know the magic runes to make that happen later, but it should be amenable to a little bit of spelunking in database/
<leonardr> yes, this is making it go faster
<lifeless> it might be nice to have a make target to do this to the ftest template, as a convenience for other developers
<lifeless> gary_poster: my first patch to lp using 'with' has test failures, I'll be working through those today.
<jam> anyone else running LP in a VM rather than raw on their system?
<jam> I managed to get it all set up
<jam> but realized I'd really like to access it from the container
<jam> so have it serve in the VM's public address
<jam> not just on the loopback
<lifeless> gary_poster: one of them is in the subscription code - http://pastebin.com/bHJgS6DG - wondering why that is an exact check rather than a LessThan
<lifeless> jam: I am
<abentley> lifeless: I see a broken link on qastaging that I don't see on production.
<lifeless> jam: https://dev.launchpad.net/Running/VirtualMachine
<lifeless> abentley: on what page?
<abentley> lifeless: the "Set up sharing" on https://translations.qastaging.launchpad.net/bzrtools/trunk
<abentley> lifeless: this was not introduced by my code.  So far my code is qa-ok.
<jam> lifeless: right, I got that part working
<lifeless> abentley: I can't see +sharing-details on production either
<jam> I just want to be able to access it from the host
<jam> rather than eg having to run Firefox in the VM
<abentley> lifeless: Yes, but the link isn't present on production.
<lifeless> abentley: ok
<lifeless> abentley: is it a feature flag hidden thing, or will everyone see it ?
<abentley> lifeless: I believe everyone will see it.  It looks like it was introduced in r12559
<lifeless> abentley: translations.sharing_information.enabled is set on qastaging
<lifeless> abentley: in the template, I see 'tal:condition="features/translations.sharing_information.enabled'
<abentley> lifeless: Okay.  I probably misunderstood henninge.
<lifeless> abentley: so it won't show up on prod at all until that flag is enabled
<lifeless> s/it/the link that is broken will be hidden/
<lifeless> jam: right, I thought I put notes on that on the wiki page
<jam> lifeless: there is a small link to an email thread, perhaps in there
<lifeless> apparently not
<lifeless> jam: so, copy the new stuff from /etc/hosts to your host os
<maxb> I have a pending application for ~loggerhead-team, who should I talk to to get a yay or nay?
<abentley> lifeless: qa-ok.
<leonardr> sinzui: good news is i've got the tests running again. bad news is we have more tests that try to create bad bugtasks
<sinzui> we suck
<lifeless> abentley: cool, thank you
<sinzui> leonardr: I favour deleting any test that possibly duplicates what is done in a lower level test. This is an opportunity to make the test suite faster
<leonardr> sinzui: i like this plan but i don't trust my ability to quickly determine that one test duplicates another. let me get the big list of failures up and we can go through it
<lifeless> maxb: what do you need that for ?
<sinzui> leonardr: that is a good plan
<lifeless> maxb: or better phrased: what are you trying to do that you can't today, that ~loggerhead-team membership will let you do
<maxb> I would like to help out with getting some of the changes moved to the experimental branch re-landed. One of the convenient ways to do that would be to re-open merge proposals for stuff that has been un-landed.
<lifeless> maxb: you don't need any special privilege to do that
<maxb> I can't edit the status of merge proposals for other peoples landings on lp:loggerhead - being in ~loggerhead-team would enable this
<lifeless> maxb: oh, I see what you're thinking. That won't do the right thing *anyway*
<maxb> it wouldn't?
<leonardr> sinzui: http://pastebin.ubuntu.com/577955/ is the full list, i'll trim it down to just the initial errors
<lifeless> I don't think so
<maxb> Doesn't being in the team owning the target branch of a MP entitle you to edit it?
<lifeless> maxb: I think it would be better to make a branch containing a specific thing merged out of experimental - probably oldest-first is easy (can avoid cherrypicks)
<lifeless> maxb: the problem is the source branch: unless the mp is /perfect already/ it will need further changes
<maxb> Right.... I'd hope to convince the original submitter to do them, then :-)
<lifeless> maxb: and you can't push to e.g. mkanat's branches
<leonardr> sinzui: http://pastebin.ubuntu.com/577957/
<lifeless> maxb: so a number of the changes were done on contract to Canonical
<leonardr> not too bad, only 4 failures, and one of them is clearly my fault (somehow)
<sinzui> I have got the files open
<lifeless> maxb: and the contract is over. Its up to 'us' - folk interested in those changes - to pick them up and polish/revisit/land
<leonardr> sinzui: i'm looking at the first one. would you take a look at the other 3?
<sinzui> leonardr: 30-nominate-bug-for-distrorelease needs history like I added to another test. maybe...
<leonardr> i'll push my changes
<maxb> alright - not all the MPs I was thinking of were mkanat's, but I could opt for emailing people asking them to reopen their own MPs
<lifeless> maxb: Personally, I would just make a new one
<lifeless> maxb: we took on the burden of re-reviewing and assessing and fixing those changes when we pivoted the trunks arounds
<leonardr> sinzui: lp:~leonardr/launchpad/bug-106338 is up to date
<lifeless> maxb: I don't think its all that fair to push back to folk whom we have told their change is done with.
<maxb> ok
<lifeless> maxb: if you need bug triage permissions, or are going to get involved in code review & qa, I'd be delighted to talk loggerhead-team membership
<lifeless> it is however the project committers team, so I think the usual 'get involved, be involved, ok heres the membership' process should apply
<maxb> alright, I'll re-raise at an appropriate time
<sinzui> leonardr: I think I have 4 fixed. 3 fails trying to show it will fail. The test is bad, but we might have a case where the use can oops the UI
<leonardr> no luck on 1 so far
<lifeless> sinzui: is there any way on e.g. https://launchpad.net/~loggerhead-team/+editproposedmembers to discuss with the applier why they are applying ?
<sinzui> No
<sinzui> That is a long requested feature
<lifeless> sinzui: is there a bug complaining about same?
<jam> lifeless: https://dev.launchpad.net/Running/RemoteAccess looks like it has what I was looking for
<sinzui> lifeless: bug 4976
<_mup_> Bug #4976: When trying to join a team, user should be able to give comment <feature> <lp-registry> <registry> <Launchpad itself:Triaged> < https://launchpad.net/bugs/4976 >
<jam> I added it to the page
<lifeless> sinzui: sigh, thats like 3 difference features rolled into one :(
<lifeless> sinzui: so many things from users are like that
<sinzui> leonardr: I been to build a pristine tree to test it will be a moment: this is what I have done: http://pastebin.ubuntu.com/577973/
<leonardr> ok
<leonardr> sinzui: the problem with #1 is that the productseries bugtask created by the factory is not conjoined to the bugtask for that productseries' product
<leonardr> does that make sense? i'm not sure why it does happen
<sinzui> leonardr: it does if the productseries is not the product's development_focus
<leonardr> but it is
<leonardr> conjoined_bugtask = self.factory.makeBugTask(bug=bug, target=product.development_focus)
<leonardr> getConjoinedMaster has the right bugtask in its shortlist but for some reason decides that's not it...
<leonardr> sinzui: the bugtask (whose target is a ProductSeries) does not have either IDistroBugTask or IUpstreamBugTask
<leonardr> ie. doesn't provide
<leonardr> should it be providing IDistroBugTask? that doesn't seem right
<leonardr> but that's the only way it can have a conjoined master
<sinzui> leonardr: we are testing the IUpstreamBugTask.providedBy condition
<leonardr> ok, i get that now. any idea why this task doesn't provide that?
<sinzui> leonardr: it doesn't
<leonardr> i guess because its .product is none? it's got a productseries but no .product?
<sinzui> product==upstream there is even a bug report to rename IUpstreamBugTask to IProductBugTask
<sinzui> what!
<sinzui> that is F'd up
<leonardr> ok, probably a problem with createBugTask
<leonardr> or, maybe in the factory
<sinzui> is it I would have thought it was makeBugTask() which I know was making invalid series last year
<lifeless> thats normal
<lifeless>  count
<lifeless> -------
<lifeless>   7703
<lifeless> (1 row
<gary_poster> lifeless, pastebin you gave me doesn't work anymore.  can look if you like
<lifeless> sinzui: launchpad_qastaging=> select count(*) from bugtask where product is null and productseries is not null;
<lifeless> -> 7703
<lifeless> sinzui: product is accessed via productseries.product
<sinzui> leonardr: last year the factory was creating products without series, which is invalid. I fixed that. I think we just found the reverse is true
<leonardr> sinzui: the product and series both _exist_
<leonardr> but, the bug task is created with a series and no product, and nobody says 'wait, that's wrong'
<lifeless> leonardr: thats how it should be
<lifeless> gary_poster: http://pastebin.com/21LHXw6D
<sinzui> lifeless: I think I need to put my hands over my ears and hum very loudly. I do not want to hear about how messed up production data is. It took me 4 months to clean up milestones without a series
<lifeless> sinzui: this is not messed up
<lifeless> sinzui: its /correct/
<lifeless> WHEN product IS NOT NULL THEN productseries IS NULL AND distribution IS NULL AND distroseries IS NULL AND sourcepackagename IS NULL
<lifeless> sinzui: I think you have an invalid assumption about how it should be, and you're wrong.
<sinzui> How do I access a product series that has no product?
<leonardr> sinzui: the product series has a product
<lifeless> sinzui: thats not what leonardr is saying.
<leonardr> the *bug task* has a .productseries but does not have a .product
<lifeless> leonardr: thats normal.
<leonardr> and the check for conjoined master only looks at .product
<lifeless> leonardr: and thats also normal
<lifeless> leonardr: the conjoined master is the *nonseries* task
<sinzui> lifeless: we are looking at this
<sinzui>         product = self.factory.makeProduct()
<sinzui>         bug = self.factory.makeBug(product=product)
<sinzui>         conjoined_bugtask = self.factory.makeBugTask(
<sinzui>             bug=bug, target=product.development_focus)
<sinzui> ^ I think there should be a bugtask with .product and a bugtask with .productseries
<gary_poster> lifeless no idea on the face of it.  AFAIK I haven't/yellow hasn't touched that particular bit.  I encountered something somewhat like that before and investigated to discover that the difference in query count was in fact indicative of a problem, but that's certainly not the ideal way to show it. ...I'm sorry I can't offer more.
<lifeless> sinzui: no, thats not possible.
<lifeless> gary_poster: no worries, thanks for peeking
<lifeless> sinzui: bugtask links to only one of (product, productseries, ...)
<leonardr> lifeless: my only goal in this is to create something that launchpad will treat as a conjoined bugtask, so that i can test its behavior. what do you recommend?
<lifeless> upstream or distro
<sinzui> lifeless that are 2 bug tasks. how would you setup a conjoined condition in a test. the are not examples in factory as far as I can see
<fjlacoste> lifeless: catch-up call?
<lifeless> flacoste: yes, but give me 2 minutes to unblock sinzui and leonardr
<lifeless> leonardr: upstream or distro ?
<sinzui> lifeless: leonardr, I am mistaken, there is a test and it does what we are doing in the broken test
<lifeless> task = factory.makeBugTask()
<lifeless> conjoined = factory.makeBugTask(bug=task.bug, target=factory.makeProductSeries(product=task.product)
<lifeless> sinzui: ^ those two lines will make a conjoined master for upstream
<lifeless> erm, not quite
<lifeless> conjoined = factory.makeBugTask(bug=task.bug, target=task.product.devel_focus)
<lifeless> sinzui: ^ that will do it
<sinzui> I am looking at a test that does this:
<sinzui>     self.factory.makeBugTask(
<sinzui>                 bug=bug, target=self.product.development_focus)
<lifeless> following the definition of 'a conjoined master is a series task on the development focus of a project'
<lifeless> what we should do is drop product bugs altogether.
<sinzui> which identical to: self.factory.makeBugTask(
<sinzui>             bug=bug, target=product.development_focus)
<lifeless> sinzui: if your product is the same product
<sinzui> yes, we make the bug with the same product, I pasted the lines
<sinzui>         product = self.factory.makeProduct()
<sinzui>         bug = self.factory.makeBug(product=product)
<sinzui>         conjoined_bugtask = self.factory.makeBugTask(
<sinzui>             bug=bug, target=product.development_focus)
<lifeless> cool
<sinzui> we could
<sinzui>         product = self.factory.makeProduct()
<sinzui>         bug = self.factory.makeBug()
<sinzui>         conjoined_bugtask = self.factory.makeBugTask(
<sinzui>             bug=bug, target=product.development_focus)
<sinzui> ^ disregard that
<lifeless> so, the quest is
<sinzui> lifeless: but the issue is that the test fails
<lifeless> what is the default_bugtask
<lifeless> I think you should not call makeBug
<lifeless> but call makeBugTask twice, because you *do care* about the target.
<lifeless> that shouldn't affect the validity though
<lifeless> sinzui: have you looked at what ends up in the db ?
<lifeless> something I do in this sort of situation is put a import pdb;pdb.set_trace() call at the point of failure, and then go sniffing
<lifeless> sinzui: I'll chat to flacoste and come back to help you in a bit
<sinzui> that was what I was thinking. leonardr. we might need a IStore(conjoined_bugtas).flush()
<leonardr> sinzui: in createBugTask?
<sinzui> after the test created conjoined_bugtask, we can explicitly flush the state to the db for the query that is test a few lines later
<leonardr> i see
<leonardr> sinzui: does it change your suggestion to know that conjoined_bugtask.conjoined_master is None immediately after it's created?
<sinzui> no, that is a sql query
<leonardr> ok
<leonardr> i disagree--i don't think it gets to the sql, but i'll try it
<leonardr> sinzui: after the flush, conjoined_bugtask.conjoined_master is still None and I can still set that bugtask's status to Invalid without raising an exception
<sinzui> leonardr: your line of code does not make sense. I think you are confused again. the series task is the master, the product/distro is the slave so  conjoined_bugtask.conjoined_master should always be None
<sinzui> conjoined_bugtask.conjoined_slave == product
<leonardr> so i am editing the wrong task?
<sinzui> maybe product==slave, series==master
<sinzui> conjoined bugs are like X servers. The relationship is reversed. It is like the odd fact that seahorse males give birth, not femals
<leonardr> yes, that was the problem
<leonardr> man
<sinzui> leonardr: I just confirmed that 30-nominate-bug-for-distrorelease.txt is fixed by my change. I need to rethink 10-bug-requestdistrofix.txt.
<leonardr> ok, we're getting there
<sinzui> leonardr: I had to restore this line in sourcepackagerecipe.py to get the tests to run
<sinzui> from lp.registry.interfaces.pocket import PackagePublishingPocket
<lifeless> flacoste: grab devpad:~mthaddon/banana.html
<sinzui> leonardr: #3 is a duplicate of bugtask editing tests. I will remove the redundant parts and hope for a pass
<lifeless> flacoste:     WHEN productseries IS NOT NULL THEN distribution IS NULL AND distroseries IS NULL AND sourcepackagename IS NULL
<leonardr> sinzui: ok, my side is committed and pushed
<lifeless> flacoste: https://lpstats.canonical.com/graphs/AppServerRequestLpnet/
<sinzui> leonardr: did you apply my changes to 30-... from http://pastebin.ubuntu.com/577973/
<leonardr> sinzui: no, doing that now
<lifeless> flacoste: http://www.mongodb.org/display/DOCS/Replication
<leonardr> sinzui: for some reason my bin/test is not finding 30-nominate-bug
<leonardr> but, i have applied your change
<sinzui> leonardr: the test runner does not let you run tests our of sequence
<sinzui> leonardr: you need to test the whole directory
<sinzui> we were smoking a lot of crack in 2005 and 2006
<leonardr> ah, i see. i was wondering how that test was run earlier, but earlier i'd run the whole directory
<leonardr> ok, your test fixes it
<mwhudson> i wonder how hard it would be to remove that sequential test feature
<mwhudson> probably not hard, just very annoying
<sinzui> leonardr: my effort to fix 10-... broke 20-... so I am fixing that too :(
<leonardr> ok, don't be discouraged
<jcsackett> sinzui: skipping our 1to1, based on how busy you seem to be?
<sinzui> jcsackett: yes, we can try in an hour
<jcsackett> sinzui: sure.
<jcsackett> lifeless: any chance you would be available to talk a bit about bug 716786?
<_mup_> Bug #716786: Product:+filebug-show-similar timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/716786 >
<lifeless> sure
<lifeless> in a little bit
<jcsackett> cool.
<leonardr> sinzui: argh, looking at the failure message more closely i see that i neglected failures in unit tests
<sinzui> :(
<sinzui> leonardr: my branch that increments lazr.restful to 0.17.4 passed, but was bounced because we are in rc mode. I think my branch will land before yours. maybe you should say your branch requests the same version before you submit it again
<leonardr> sinzui: makes sense
<sinzui> leonardr: I got rid of the duplicate parts in the 10-* test. There is still an error. I believe that bugalsoaffect.BugTaskCreationStep is not validating before creating the bugtask! We need to fix this in this branch
<leonardr> sinzui: ok, give me a pastebin and i'll try to figure out what you mean
<sinzui> leonardr: I mutated by test, but you can see it in your own paste. #2 and #3 are the same error. The view assumes the selected target is sane: http://pastebin.ubuntu.com/577957/
<leonardr> sinzui: what would it mean to validate before creating the bugtask? we would call validate_new_distrotask/valid_upstreamtask and catch the exception?
<leonardr> what would happen if createBugtask just threw a WidgetError? (that's what should happe nnow)
<thumper> anyone know what happened to the builders?
<thumper> they both seem to have failed mid test
<thumper> with what looks like a missing database
<sinzui> leonardr: I think we are too far into the view to raise an error. The field value (a distro in the test I am reading) is invalid. I do not think we should have entered the method. I expect something to set fieldError()
<leonardr> sinzui: in that case... we have a lot of error checking code in createTask (determining the order of precedence). should we create a method that takes similar arguments to createTask and makes the check? then the view can catch the error and set fieldError
<sinzui> oh, leonardr: I think DistroBugTaskCreationStep.validateStep() is at fault
<leonardr> yeah, something like that. except that code doesn't know about the precedence order we decided on? is that hte problem?
<lifeless> jcsackett: hi
<jcsackett> lifeless: hi.
<jcsackett> i think there may be a conflict in times to talk, unless sinzui is still too busy for 1-to-1, or is good with doing it tomorrow.
<sinzui> jcsackett: we can talk tomorrow
 * sinzui is free all dat
<sinzui> day
<jcsackett> okay, lifeless, no conflict after all. :-)
<thumper> leonardr: mumble ping
<lifeless> jcsackett: I can wait for you to talk with sinzui
<leonardr> k
<jcsackett> lifeless: sinzui and i can chat tomorrow--it's not a pressing thing, just  a regular checkin.
<lifeless> ok
<lifeless> so how can I help you
<sinzui> lifeless: leonardr and I want to land this branch and never speak of it again
 * jcsackett wants them to land the branch as well.
<jcsackett> so, lifeless, i'm looking at bug 716786
<_mup_> Bug #716786: Product:+filebug-show-similar timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/716786 >
<jcsackett> the one about timeouts with insane fti strings.
<lifeless> right
<leonardr> jcsackett: actually you're in good shape, you just need the new version of lazr.restful, which will be installed anyway when _sinzui's_ branch lands
<jcsackett> leonardr: oh, cool.
<leonardr> my problem is that i pulled on a thread that turned out to be the tail of a dragon
<jcsackett> lifeless: my naive thought, based on a comment by stub on a related bug, is that maybe we just need to kick out any super long bug titles?
<jcsackett> but a) i'm not sure that's best and b) i'm not sure what the cap should be if it is.
<lifeless> jcsackett: why does it do this search at all ?
<jcsackett> show similar is using an fti search to find any bugs similar to the one you might be reporting.
<jcsackett> so you can say "oh, yeah, that's the issue" rather than filing a new bug.
<sinzui> leonardr: I am uncertain what happened in the two examples. I see DistroBugTaskCreationStep.validateStep() and it does validate_new_distrotask. either we skipped that line, or multistep() never called validateStep().
<leonardr> sinzui: on call, 1 min
<lifeless> jcsackett: oh, its not actually on the bug filing
<lifeless> its in the prepatory workflow ?
<jcsackett> no, though bug 722787, which you filed against +filebug, is the from the same thing.
<_mup_> Bug #722787: Product:+filebug timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722787 >
<lifeless> so title=%5Barmel%5D%20mawk%20testsuite%20fails%20(-fif-conversion2%20problem) is the sesarch string
<lifeless> so title=%5Barmel%5D%20mawk%20testsuite%20fails%20(-fif-conversion2%20problem) is the sesarch string
<lifeless> jcsackett: that sample query runs in 800ms on qastaging
<lifeless> jcsackett: this may not need code changes
<jcsackett> lifeless: i lost a bit there; accidentally closed the channel.
<lifeless> jcsackett: I repasted
<jcsackett> ah.
<jcsackett> so, the problem in fti is that whatever you search on, gets redone N ways, where N is number of tokens in your search string left over.
<lifeless> yes, I put that together as a compromise
<lifeless> we used to do some very expensive preprocessing to filter out common terms
 * jcsackett nods.
<jcsackett> so, you think this may not need code changes?
<jcsackett> is our full text index just not up to snuff?
 * jcsackett confesses fair amount of ignorance on tsearch2 and the like.
<lifeless> jcsackett: I think its db layer, stub bounced it as crazy, but I want to talk through it with him
<jcsackett> fair. i'll mosey on to something else then. :-)
<lifeless> we may need to do a context-term document frequency index
<lifeless> to permit getting back what we had
<lifeless> -or-
<lifeless> we need to move ahead on the lucene thing
<jcsackett> lucene is pretty hot. we used it at my last job.
<jcsackett> we were using mysql full text prior, so *huge* improvement.
<sinzui> leonardr: I am still stepping through. We are validating, and I can see that we did a sane validation, so I now looking at the base class to see it if is creating the bugtask without the sourcepackage (which is what was validated)
<leonardr> k
<lifeless> jcsackett: may I humbly suggest bug 727023
<_mup_> Bug #727023: Cve:+index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727023 >
<lifeless> jcsackett: 20 timeouts a day
<lifeless> jcsackett: needs eager loading
<jcsackett> lifeless: sure, i'll take a look at that.
<leonardr> sinzui: off the phone
<thumper> sinzui: I'm getting google service problems
<thumper> RuntimeError: Socket poll time exceeded.
<thumper> starting the google service layer
<thumper> WTF?
 * thumper tries to rebuild
<thumper> yeah, rebuild fixed it
<sinzui> thumper: is it because you branch still says it changed something in lp.services.googlesearch
<thumper> sinzui: no, this is the other branch :)
<sinzui> thumper: you may need to do bin/buildout if you merged the change from from canonical or the rename from .search to .googlesearch
<thumper> sinzui: yeah, the make fixed that
<sinzui> leonardr: the data passed to validateStep and main_action is different. We validated spn=alsa-utils, but the action is working with spn=mozilla-firefox
 * sinzui now wonders if there is a very old and deep bug in the code
<leonardr> sinzui: remind me which test file you are in?
<sinzui> xx-duplicate-bugwatches.txt, the first test
<sinzui> leonardr: we are on mozilla-firefox, but we are adding alsa-utils
<sinzui> We validate alsa-utils, but the main_action thinks we working with mozilla-firefox
<leonardr> sinzui: ok, and which class has the main_action?
<sinzui> leonardr: It was DistroBugTaskCreationStep.main_action. I am stepping though it again and I see everything right so far
<lifeless> sinzui: leonardr: would you like me to look as well? if so whats the branch
<leonardr> lifeless: knock yourself out. lp:~leonardr/launchpad/bug-106338
<leonardr> i'll forward you the failure message
<leonardr> we've fixed all but one error in the pagetests, so it's just the unit test failures after shis
<lifeless> leonardr: if its under control I'll leave it with you guys; if you need a hand I'll pull the branch and have a look
<leonardr> sinzui: how much longer can you spend on this?
<sinzui> leonardr: we get a LaunchpadValidationError creating a task for debian alsa-utils. This causes the view to attempt to recover (maybe by backing up a step) and it rebuilds the data from the existing context.
<leonardr> sinzui: so instead we should set a field error and let it go?
<sinzui> leonardr: so I need to see know why the call to validate passed, but the call to create failed
<leonardr> sinzui: probably because the call to validate was much simpler than the equivalent validation code in create
<leonardr> remember the complicated order-of-operations code i wrote yesterday?
<leonardr> that calls validate_new_distrotask in two places and calls valid_upstreamtask if neither of those was triggered?
<leonardr> i bet the validate() only calls valid_upstreamtask
<leonardr> well, we only call validate_new_distrotask
<leonardr> however, a casual glance shows that in this case, createTask should be running the same code as validateStep()
<StevenK> lifeless: I was having trouble working out the bug you said was in the temp directory handling, but first I wanted to get a pointer or two on fixtures, since I've not used them before.
<lifeless> StevenK: from fixtures import TempDir
<lifeless> with TempDir() as dir:
<lifeless>     cwd = dir.path
<lifeless>    ...
<lifeless> StevenK: fixtures implement the context manager protocol
<StevenK> lifeless: So, just replace the outside try: with that, and it will create the directory and remove it when it falls out of scope?
<lifeless> StevenK: the bug is that you make a directory and may not clean it up because you don't have a try: finally:
<lifeless> StevenK: yes.
<StevenK> lifeless: Er, I do have a try: finally:. I might not in the test, but I do in the code
<lifeless> StevenK: whats the mp again ?
<leonardr> sinzui: let me throw this out there. the problem is that validate_new_distrotask is called twice, once for the source package and once for the distribution
<leonardr> it should only be called for the source package
<leonardr> that's what i'm seeing, anyway
<StevenK> lifeless: https://code.launchpad.net/~stevenk/launchpad/populate-spr-changelogs/+merge/52363
<wgrant> Do WIP MPs send mail still?
<lifeless> StevenK: ok, so your test is definitely buggy
<lifeless> StevenK: and your try:finally: is ok  but would be improved by using the fixture I think
<StevenK> lifeless: Agreed. I'll look at them, thanks for the pointer and the help.
<lifeless> StevenK: anytime
<sinzui> leonardr: sorry, unity went belly up. validate_new_distrotas()k and bugtask.createTask() disagree about debian/alsa-utils
<thumper> wallyworld: can you update your cards on the kanban board?
<leonardr> sinzui: i got it
<leonardr> the _reason_ it was trying again with firefox was that createTask was checking alsa-utils and then debia
<leonardr> it shouldn't have been checking debian
<leonardr> sinzui: see if this makes sense to you: http://pastebin.ubuntu.com/578052/
 * leonardr hopefully checks to see if that fixed other bugs, even though it's unlikely
<sinzui> leonardr: I think that may do the trick. I am in the condition that your paste is trying to stop
<leonardr> yeah, not really fixing other bugs, oh well
<sinzui> leonardr: does that fix xx-duplicate-bugwatches?
<leonardr> sinzui: yes
<leonardr> but not, apparently, anything else
<sinzui> leonardr: yes. I think the first chunk I remove from the the 10.* test was needed. I will try your changes with mine
<leonardr> ok, then we can spend a little time on the unit test failures
<wallyworld> thumper: will do
<leonardr> i think these test failures need a call to factory.makeDistroSeries
<leonardr> sinzui: ok, i know what's going on for one of these tests, but i'm not sure what to do
 * sinzui listens
<leonardr> the test is calling self.factory.makeBugTask(sourcepackage)
<leonardr> this code runs in makeBugTask
<leonardr>         if ISourcePackage.providedBy(target):
<leonardr>             # We can't have a series task without a distribution task.
<leonardr>             prerequisite_target = target.distribution.getSourcePackage(
<leonardr>                 target.sourcepackagename)
<leonardr> that gives us a DistributionSourcePackage
<leonardr> then we create a task for that DistributionSourcePackage, and that raises an error
<leonardr> LaunchpadValidationError: u'Package generic-string927887 not published in Distribution927888'
<sinzui> leonardr: I reverted my changes to 10-*. bug-also-affects passed with your change. So I think the doctests are now addressed.
<leonardr> how can we get that error when prerequisite_target *is* the package generic-string... as published in Distribution...?
<leonardr> that's why i said maybe we need to call makeSourcePackagePublishingHistory
<leonardr> but i'm not sure how, or if that will really help
<sinzui> leonardr: makeSourcePackagePublishingHistory() will fix that
<leonardr> ok, 1) how do i call it, 2) what if there already is a publishing history?
<sinzui> makeSourcePackagePublishingHistory(series=series, sourcepackagename=dsp.sourcepackagename)
<leonardr> what is 'series' here?
<leonardr> target?
<sinzui> leonardr: a distroseries possibly the development_focus of the distro
<sinzui> leonardr: since sample data has bugger all for SPPH, there is little chance we can collide. since SPPH is a record of changes, adding publishing records jut means a new version was released
<wgrant> thumper: Do we send email for WIP MPs still?
<thumper> wgrant: not on creation
<lifeless> wgrant: not when creaeted WIP IIRC. possibly not on changes either.
<wgrant> Great, thanks.
<thumper> wgrant: there are a few nitty bits around commenting on WIP
<thumper> if you change the description or commit message, it doesn't send email
<lifeless> jcsackett: why is bug 731841 critical?
<_mup_> Bug #731841: rosetta-admins team cannot change project translation settings <Launchpad itself:Triaged> < https://launchpad.net/bugs/731841 >
<thumper> if you request reviews, it "shouldn't", I'm not sure if it odes
<jcsackett> lifeless: i misread it earlier, thought they were getting an OOPS, not just a "you're not allowed here"
<lifeless> fixing
<leonardr> ok, that got rid of a lot of the errors
<jcsackett> dig.
<leonardr> sinzui: http://pastebin.ubuntu.com/578065/
<sinzui> leonardr: that looks good
<leonardr> ok, on to the next one
<leonardr> we have this code:
<leonardr>                bug = self.factory.makeBug(
<leonardr>                     distribution=self.searchtarget.distribution)
<leonardr>                 bugtask = self.factory.makeBugTask(
<leonardr>                     bug=bug, target=self.searchtarget)
<leonardr> that gives this error, i think correctly:
<leonardr> This bug is already open on Distribution360034 with no package specified. You should fill in a package name for the existing bug.
<sinzui> leonardr: I think the test is bad.
<sinzui> we should provide a package or makeup a new distro
<leonardr> self.searchtarget is a DistributionSourcePackage
<leonardr> it has a .sourcepackagename
<leonardr> so maybe it's not a bad test?
<leonardr> sinzui: that error is given by validate_new_distrotask(bug, distribution, sourcepackagename) when sourcepackagename is not None
<leonardr> i would only expect it when sourcepackagename is None
<leonardr> ARGH
<leonardr> now i understand the error message
<leonardr> it's saying "you shouldn't be filing this bug"
<thumper> leonardr: hi, how to I stop exporting an attribute in devel?
<leonardr> thumper: pass in ('devel', dict(exported=False)) right after the field object
<thumper> leonardr: no need
<leonardr> k
<thumper> leonardr: it is only exported in devel
<thumper> not in the others
<thumper> so I can just kill it
<leonardr> yup
<leonardr> sinzui: so, yes, i think the test is wrong. what would be a better test?
<sinzui> leonardr: yes, the test is wrong, not the code
<leonardr> i don't think "provide a package or make up a new distro" makes sense here, but i could be wrong
<sinzui> we can create a new dsp
<leonardr> sinzui: for context, this is test_bugtask_search.py line 327
<wgrant> sinzui: I hear you... but I'm muted, and Unity is not showing Mumble's notification area icon.
<wgrant> sinzui: Any hints?
<leonardr> sinzui, i am eod and then some. i've pushed my changes. did i give you the list of unit tests that failed?
<sinzui> leonardr: no you did not. I have few hours left in my day
<leonardr> ok, let me give you the master list and you can chip away at it. i think there are only 2 or 3 distinct bugs, despite the huge number of failure
<leonardr> s
<leonardr> sinzui: http://pastebin.ubuntu.com/578071/
<lifeless> ugh
<lifeless> did something break date formatting recently?
<lifeless> ----------------------------------------------------------------------
<lifeless> File "lib/lp/registry/tests/../stories/webservice/xx-project-registry.txt", line 932, in xx-project-registry.txt
<lifeless> Failed example:
<lifeless>     print milestone['date_targeted']
<lifeless> Differences (ndiff with -expected +actual):
<lifeless>     - 2005-06-06T00:00:00
<lifeless>     + 2005-06-06
<lifeless> StevenK: wgrant: you guys watch CI like hawks - seen anything like that recently ?
<StevenK> lifeless: I can't recall anything
<StevenK> r12561 on Jenkins had no failures.
<wgrant> lifeless: I've never seen that before.
<wgrant> That field was changed a few months ago.
<lifeless>     for milestone in firefox.milestones:
<lifeless>         print '%-7s %s' % (milestone.name, milestone.dateexpected)
<lifeless> Differences (ndiff with -expected +actual):
<lifeless>     - 1.0     2056-10-16 18:31:44.293448
<lifeless>     + 1.0     2056-10-16
<lifeless>       1.0-rc1 2018-10-01
<wgrant> Maybe lazr.restful has changed?
<leonardr> the changes to it haven't landed recently, and there weren't any changes like that...
<wgrant> lifeless: Where are you seeing this failure?
<bac> hi sinzui, quick JS test question?
<sinzui> I hope I know the answer
<lifeless> wgrant: my storm branch
<lifeless> wgrant: my with branch, I mean
<wgrant> Maybe Storm has started returning proper dates...
<lifeless> oh that would be just typical
<wgrant> 386. Merged datetime-in-datevariable
<wgrant> Last rev on lp:storm
<lifeless> yeah
<lifeless> I'll generate a storm tarball without it
<wgrant> Please also push up a branch with it backed out...
<lifeless> yes
<lifeless> thats what I'm doing
<wgrant> Great.
<lifeless> after I confirm it, of course
<lifeless> I've also got this weird thing with subscription tests
<lifeless> one less query than expected
<wgrant> Hah
<wgrant> Reduce the number of queries issued when retrieving an object with
<wgrant> Store.get() for which we have an existing invalidated Python object
<wgrant> available.
<StevenK> And that's a problem?
<wgrant> Oh, that is awesome.
<wgrant> Merge it merge it merge it.
<lifeless> StevenK: it is when the tests are written with an exact match
<StevenK> It certainly got wgrant excited
<wgrant> It's going to make query tests not suck.
<lifeless> yeah
<StevenK> lifeless: http://pastebin.ubuntu.com/578091/
<lifeless> with TempDir as tmp_dir:
<lifeless> should be
<lifeless> with TempDir() as tmp_dir:
<lifeless> and
<lifeless> +                    tmp_dir, 'extracted')],
<lifeless> should be
<lifeless> +                    tmp_dir.path, 'extracted')],
<lifeless> ditto on its other uses
<lifeless> StevenK: the fixture isn't a path itself, its an object with the path on .path
<StevenK> Right
<StevenK> lifeless: Then http://pastebin.ubuntu.com/578092/ should be good
<lifeless> looks good
<lifeless> tests will tell you
<StevenK> Sure, I just wanted to hear your thoughts
<lifeless> sure :) just saying my built in interpreter is an imperfect oracle :)
<lifeless> as for my branch
<lifeless> down to just a checkwatches fail
<lifeless> wgrant:
<lifeless>     - ...SELECT 1 FROM BugTracker WHERE BugTracker.id = ...
<lifeless>     - ...Transaction completed, status: Active...
 * StevenK waits for the car alarm to SHUT UP
<lifeless> wgrant: i don't see that subselect
<lifeless> and the last line of the timeline is
<lifeless>     + 00016-00016@SQL-nostore Transaction completed, status: Committed
<StevenK> RARGH, need to run make schema before tests, HULK SMASH
<lifeless> wgrant: for future reference https://code.launchpad.net/~lifeless/storm/with-without-datetime
<lifeless> wgrant: lp:~lifeless/launchpad/bug-731679 is all go except this checkwatches change now.
<lifeless> wgrant: I'm nagging you cause the test looks like one you did :)
<wgrant> lifeless: Back, sorry.
<wgrant> lifeless: Do you have a full paste of that error?
<lifeless> can do
<wgrant> I think just dropping the SELECT 1 statement is probably best, but I'll check.
<lifeless> http://pastebin.com/MV3yQfrt
<wgrant> StevenK: Does loganberry have a memcache set up?
<wgrant> It might not.
<wgrant> Check the configs.
<wgrant> lifeless: Replace the "...SELECT 1 [...]" line with "...SELECT BugTracker...", I guess. That test will hopefully die next week, but we might as well keep it mildly useful until then.
<lifeless> what happened to the SELECT 1
<wgrant> The Storm revision eliminates that.
<lifeless> righto
<wgrant> On an invalidated object it used to do a SELECT 1 to check its existence, then select its data when it was used.
<wgrant> Now it just grabs the data in the first query.
<wgrant> On a related matter, what is your opinion on LessThan vs Equals for query test?
<wgrant> +s
<wgrant> I think Equals is probably better, as it ensures we are never regressing.
<lifeless> depends on the test
<lifeless> so
<lifeless> it would be sad to have to change 5K tests when we fix traversal to be better
<wgrant> True.
<lifeless> ok, pqmed
<lifeless> wgrant: I think things that are really isolated and specific should be specific
<wgrant> Right.
<lifeless> wgrant: things that are not really isolated and depend on the behaviour of many subcomponets should be less sensitive
<lifeless> wgrant: while still providing a solid backstop
<StevenK> wgrant: Can you recall if there a bug for older SPRs not having .changelog?
<wgrant> StevenK: I don't think so, but I don't know for sure.
<StevenK> Meh, filing one
#launchpad-dev 2011-03-10
<lifeless> I wonder if my with patch will be past buildbot in time for the deploy
 * lifeless has a cunning plan to add as many things as possible
<wgrant> lifeless: Will we have an mthaddon in time?
<lifeless> also a good question
<wgrant> I was assuming not, otherwise I would have been landing stuff more aggressively today.
<lifeless> staging takes 20 minutes
<lifeless> if he shows up at ~8 as normal
<lifeless> then yes
<lifeless> otherwise no
<wgrant> Hmm, the docs say the tree should be prepped 2.5 hours before :)
<lifeless> yes
<lifeless> and we have such a tree
<wgrant> But I guess you could try to get another pushed out, and just use the old one if it's not there in time.
<lifeless> as long as we don't bork it we're fine
<lifeless> the 2.5 hours - if you are reading the losa docs - is time for 2 staging attempts + discussion
<wgrant> Oh.
<wgrant> staging as in staging, not staging, right. I was wondering why you were talking about staging, and how you'd managed to get it to update in 20 minutes.
 * wgrant lunches.
<StevenK> But it isn't even midday yet?
<StevenK> Another day, another firefox update.
<thumper> StevenK: it is past midday
<thumper> well past :)
<StevenK> thumper: Timezone fail :-)
<wgrant> StevenK: Shh.
<StevenK> wgrant: Do you have dinner at 4pm, too?
<wgrant> StevenK: No :(
<wgrant> thumper: Is the EnumChoiceWidget suitable for the bugtask table too?
<thumper> wgrant: it should be, perhaps with a little tweaking
<thumper> wgrant: in the same way the InlineEditPickerWidget should be for selecting a person
<wgrant> thumper: Why does one have InlineEdit and the other not?
<thumper> wgrant: because I never got around to renaming it, and that is what it was originally called
<wgrant> Ah, good. Was hoping I wasn't missing some difference.
<thumper> nah...
<thumper> I'm just fixing a few tests on my blueprint-magic branch
<thumper> which widgetizes the blueprint page as a proof of concept
<thumper> or I should say
<wgrant> Excellent.
<thumper> another proof of use
<StevenK> Awww. Here I was hoping it removed blueprints.
<thumper> I like blueprints
<StevenK> Blueprints are annoying
<thumper> personally I think merging blueprints and bugs is wrong
 * thumper runs blueprint-magic through ec2
<wgrant> lifeless: what benefit does colocation provide us?
<lifeless> wgrant: we need to support the protocol: more metadata, streaming fetch of N branches at onces, multiple heads etc
<lifeless> wgrant: + [possibly] get rid of stacking and massively simplify things
<wgrant> I guess.
<StevenK> Get rid of stacking?
<lifeless> wgrant: think about ideal loom behaviour
<lifeless> wgrant: pushing 200 vim patches == pain; pushing 1 collection of branches == nice
<lifeless> StevenK: stacking is the source of many bugs and slowdowns in bzr
<StevenK> I like it for LP development
<lifeless> StevenK: you like the performance
<lifeless> StevenK: if it was faster than it is now, would you really whine?
<StevenK> lifeless: TBH, with stacking I don't mind bzr push performance, and I'm happy about the disk space win for crowberry. If it was faster without losing the win for crowberry, that would be awesome.
<lifeless> it has the potential to be smaller
<lifeless> we don't delete branches
<lifeless> and the minimum size for a stacked branch is the size of one inventory - which doesn't compress well
<lifeless> if all those branches were combined, the incremental overhead per branch could be a lot lower
<lifeless> the question is whether the baseline overhead would be more or less
<StevenK> But we can't do that today, right?
<lifeless> no
<lifeless> its a nontrivial discussion
<lifeless> and we have other fish to fry
<StevenK> Right.
<wgrant> lifeless: Why does it need a full inventory? Because it starts a new compression group?
<wgrant> (my knowledge of 2a and above is sorely lacking)
<StevenK> And above?
<StevenK> There's another format after 2a?
<wgrant> development-subtree, for one. But it's not exactly very different.
<lifeless> wgrant: it has to be able to generate a delta
<lifeless> wgrant: for anything in it
<StevenK> Sigh. Minutes after I say I'm happy with push performance, I'm stuck waiting for it.
<wgrant> lifeless: And it can't use just a delta on top of the stacked-on CHK tree?
<wgrant> I should probably read how CHK actually works :)
<lifeless> wgrant: fetch operations are single repo always
<lifeless> wgrant: consider: client A, servers B and C with a firewall between B and C
<lifeless> wgrant: if sftp to B and C worked but bzr+ssh didn't it would be unpleasant
<lifeless> wgrant: so the way we did it is to say that a repository must:
<lifeless>  - for any rev R it has:
<lifeless>    - be able to return the content of the texts in R [in a repo specific format - e.g. fulltext, delta against some ancestor, whatever]
<lifeless>    - be able to describe the content of R as a delta against the immediate ancestors of R on all sides
<lifeless>  - on pushes the server says "I am missing the parents of revisions X,Y,Z"
<wgrant> Oh, right.
<lifeless> we could, in theory, have a partial CHK tree for a given rev
<lifeless> so far we haven't implemented that
<lifeless> hmm, new timeout
<lifeless> SourcePackage:+index
<wgrant> What's the bad query?
<lifeless> dunno yet
<wgrant> Ah, there.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/derive-common-ancestor/+merge/52796 -- given it's our work, I'm not asking for a reviewer, but look it over?
<wgrant> lifeless: Ouch, 9s in 3 repeated queries.
<wgrant> Rather one triplicated query.
<wgrant> It seems to be exactly the same query.
<wgrant> (looking at https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1894B1679, queries 20, 40 and 41)
<lifeless> and there is AND SourcePackagePublishingHistory.status IN (2)
<lifeless> on q 19
<wgrant> So there is.
<lifeless> so maybe 4 calls
<wgrant> I would normally say that two of them are probably TAL guarding the display of the third... but these are full queries.
<wgrant> Might try to get tracebacks from DF.
<wgrant> Or go TAL-diving, but that is more tedious.
<lifeless> I'm doing some houseworky stuff atm
<lifeless> but if you wanted to identify the call sites, that would be awesome
<wgrant> Sure, just trying not to collide with you.
<StevenK> We can't get full tracebacks from qas?
<wgrant> StevenK: I have a patch to give OOPSes a traceback for every query,.
<wgrant> But it makes them unparseable, so it's not really usable on qas.
<wgrant> So I pretend that Julian doesn't exist and use it on mawson.
<StevenK> Haha
 * StevenK ponders ringing Subaru
<wgrant> Oh?
<StevenK> They've had my car for 5 and a half hours now. Surely they're done servicing it.
<wgrant> :(
<StevenK> wgrant: Can haz opinion on MP? Or is it on your list?
<wgrant> StevenK: Looking.
<wgrant> mawson will take about forever to update.
<StevenK> Just one or two eons
<wgrant> +                'derived': dervied_changelog,
<wgrant> typo
<StevenK> Sigh
<StevenK> All instances fixed, thanks.
<wgrant> You want to factory out the madness into something like get_ancestry(SPR)
<wgrant> Otherwise that looks good.
<StevenK> I do? I figured _updateBaseVersion() was self-contained enough.
<wgrant> You're duplicating the set(Changelog(spr.changelog.read()).versions)
<wgrant> Also, what does debian.changelog do if the changelog is unparsable?
<StevenK> Returns an empty list
<StevenK> Which is fine by me
<wgrant> It's not going to raise exceptions in any case?
<StevenK> wgrant: I'm happy to write a test for that case.
<wgrant> That would be handy.
<StevenK> wgrant: get_ancestry in DSD or SPR?
<wgrant> StevenK: SPR is big enough already.
<wgrant> Keep it in DSD until we need it elsewhere, I think.
<StevenK> wgrant: http://pastebin.ubuntu.com/578173/
<wgrant> StevenK: Looks reasonable.
<StevenK> No manual entry for subunit-stats
<StevenK> Subunit has to be the most poorly documented set of scripts ever
<lifeless> StevenK: I give you perl
<lifeless> StevenK: seriously, subunit-stats --help.
<StevenK> help2man, kthxbye
<lifeless> patches appreciated kthxdeal
<wgrant> lifeless: So, I've found the sources of the queries.
<wgrant> lifeless: They are very fast on DF when caches are hot.
<wgrant> Well, not very fast, but <200ms.
<StevenK> But very fast on DF is still 2 seconds.
<wgrant> lifeless: SourcePackage.summary and SourcePackage.published_by_pocket. A couple of calls to each.
<wgrant> Can you get a plan from a staging?
<lifeless> sure
<wgrant> lifeless: Thanks for the review.
<wgrant> I hope next week that the OOPS counts will be low enough that I can sensibly go through and tear out the old OOPS reporting stuff.
<wgrant> And then clean up the exception handling.
<lifeless> I'm using query 11 from https://lp-oops.canonical.com/oops.py/?oopsid=1894B1679#statementlog
<wgrant> Now that we know (hopefully) everywhere it's needed.
<lifeless> note that its not a hold/cold issue because its consistnetly slow in the oops
<wgrant> Indeed, I noticed that.
<lifeless> sadly, tis fast on qas
<wgrant> Hmmmmmmm.
<lifeless> want me to check staging ?
<wgrant> Worth a try, I guess :/
<lifeless> oh but
<lifeless> there is also the deserialiation overhead
<lifeless> same results on staging
<wgrant> :(
<lifeless> no, not that
<lifeless> 72 rows
<lifeless> tagged it dba
<wgrant> Thanks.
<lifeless> we need to start capturing db hostnames
<lifeless> still, we should fix
<lifeless> no need to do 3 lookps
<StevenK> Huzzah, I have my car back
<lifeless> stub: hi
<stub> yo
<stub> lifeless: https://dev.launchpad.net/Database/LivePatching
<lifeless> stub: I saw - looking good
<lifeless> stub: I am drafting a 'ReliabileDBDeployments' LEP too
<lifeless> stub: which will frame whatever work we need to invest in this
<stub> Ok. Do you want to incorporate what I put together?
<wgrant> I was very pleased to see LivePatching this morning.
<lifeless> stub: I think the are complementary - the L-P page is how and implementation strategies
<lifeless> stub: the LEP will be what, goals, constraints, requirements
<lifeless> s/the/they/
<stub> Ok. I'll update that wiki document if I think of anything new or get feedback then.
<lifeless> excellent
<lifeless> stub: we have some queries running slow on prod slaves, but fast on qastaging/staging
<lifeless> stub: two so far that I know of : the duplicate bug detection FTI queries (the ones you said we can't do realtime) and the one in https://bugs.launchpad.net/launchpad/+bug/732398
<_mup_> Bug #732398: SourcePackage:+index timeout <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/732398 >
<stub> wgrant: So I'm worried the extra overhead (sometimes needing 3x as many db patches, extra code to support 'old' and 'new' schemas) could deter devs. You disagree and think you would make use of the process?
<wgrant> stub: We can push things out more quickly and without hideous amounts of downtime.
<stub> Sounds like we need to pull some RAM out of prod....
<stub> ;)
<wgrant> Why wouldn't people make use of it, even if it slightly more cumbersome?:
<wgrant> (I note that the page doesn't define what a light patch is, though.
<lifeless> stub: if you could get an explain analyze on all three db's for the query in https://bugs.launchpad.net/launchpad/+bug/732398/comments/1 - that would be awesome
<_mup_> Bug #732398: SourcePackage:+index timeout <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/732398 >
<stub> wgrant: I'm just a born devil's advocate. There is more overhead in this process, and I'm interested in if the extra overhead will overcome the desire to get stuff rolled out 'now' rather than 'next cycle'
<lifeless> bah, brb
<lifeless> am I back?
<stub> lifeless: your back
<lifeless> cool
<wgrant> You're not gone.
<lifeless> so - can has analyze ?
<lifeless> stub: and then, I'd like to talk fti briefly, possibly voice, possibly here
<wgrant> Interestingly enough, as soon as I said "you're not gone", freenode lagged for 30s.
<lifeless> stub: (short story, I want to know what I'm missing on the query - it's plan and behaviour on staging seem totally fine)
<lifeless> wgrant: had to bounce wifi to stop openid trashing all my open tabs when I restarted chromium
<wgrant> Hah.
<lifeless> and I had to restart chromium becuase it had forgotten about a popup window which was permanently stuck inthe foreground
<cody-somerville> Software sucks :(
<lifeless> (not a browser window, right mouse context window)
<stub> Cold, that query ran in 500ms or less on all prod servers
<lifeless> stub: argh
<lifeless> stub: so, we have a diagnostic challenge
<wgrant> Because it took 3s hot.
<stub> lifeless: Somehow get more information about locks
<lifeless> wgrant: lets be precise
<lifeless> wgrant: our timelime which records query serialisation, queuing, deserialistion and upcasting to objects and any time given to another worker thread, showed 3 seconds.
<wgrant> True.
<stub> Interestingly, the fastest one (launchpad_prod_1) had a slightly different plan
<stub> Sorry - launchpad_prod_2
<lifeless> perhaps we should capture the plan for any query over 1 second
<lifeless> into the timeline
<lifeless> there is already a per thread timeline bug
<lifeless> bug 243554
<_mup_> Bug #243554: oops report should record information about the running environment <lp-foundations> <oops-infrastructure> <Launchpad itself:Triaged> <OOPS Tools:Triaged> < https://launchpad.net/bugs/243554 >
 * lifeless retitles
<lifeless> stub: so the same query is run three times in that page
<stub> In this case, I think the plan is a red herring and just an artifact of different statistics. The costs of the different parts of the plans are close enough to identical.
<lifeless> stub: https://launchpad.net/ubuntu/lucid/+source/chromium-browser/+index - and its timing out now
<lifeless> OOPS-1895M373
<lifeless> trigging an lpnet sync
<stub> Just reran the previous query - slowest was 318ms
<wgrant> Is that with or without status IN (2)?
<lifeless> stub: would lock contention explain the same query being slow 3 times in a row ?
<stub> no
<stub> Well... maybe.
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1894B1679#repeatedstatements
<lifeless> 4th row
<lifeless> 3 calls to it, average time 2911ms
<stub> If it is a slow process like the publisher it could be locking rows in the same set returned by the slow query in different transactions
<lifeless> ok, so lets see the times for these oopses
<stub> Oh... 3 queries in one transaction, no - not lock contention
<lifeless> they are spread over the day
<stub> locks shouldn't be blocking selects anyway.
<lifeless> time of day for the oopses: 1937 1420 1937 0846 1201 1335 1258 1056
<lifeless> wgrant: what time range does the publisher run in ?
<wgrant> lifeless: Primary archive? Normally 03-40
<lifeless> ok, not that then
<wgrant> But it should release locks a good 10-15 minutes before it finishes.
<stub> High replication load possibly - look for corresponding lag spikes
<lifeless> all the oopses have exactly the same pattern
<lifeless> stub: where is the replication lag graph ?
 * stub is looking for it
<lifeless> stub: and do we have one right now ?
<lifeless> https://launchpad.net/ubuntu/lucid/+source/chromium-browser/+index is the page I hit to generate an oops
<stub> Not lagged atm.
<lifeless> stub: then thats likely not it, cause its still timing out on prod :>
<stub> Graph is here anyway: https://lpstats.canonical.com/graphs/ProductionDBReplicationLag/
<wgrant> Seems to time out on the master too.
<wgrant> OOPS-1895ED372
<stub> So the same query with different parameters from the bug is still not having any problems.
<stub> Anyone have the actual query currently timing out handy yet?
<lifeless> stub: the one I linked is the one timing out
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1895M373
<lifeless> http://pastebin.com/f8Vdi5p0
<lifeless> 3.7 seconds in prod
<stub> When I run it, slowest 536ms.
 * stub checks the pastebin matches
<stub> yup
<lifeless> ok thats strange
<lifeless> stub: try this
<lifeless> stub: run it, *not explain*, and press 'end'
<lifeless> make sure you have \timing on
<stub> The query is returning 5k rows and building 10k objects - would this be appserver time hidden from our metrics?
<lifeless> hidden from an explain analyze anyhow
<stub> That will give crap results atm.... I'll open some local psql shells.
<stub> erm... remote shells
<stub> Ooh... perhaps there are some silly large text fields in there?
<wgrant> SPR is pretty fat.
<wgrant> Because of SPR.copyright.
<wgrant> There shouldn't be that many rows, though :/
<wgrant> I'd expect a few dozen at most.
<lifeless> count(*) says 72 rows for me
<lifeless> doing select count(*) from (SE...) as _tmp;
<lifeless> stub: how are you measuring the 5K ?
<stub> Sorry - I was looking at the estimate, not the actual returned count.
<wgrant> Ahh
<wgrant> I was scared that our kernel developers were even more insane than I thought.
<lifeless> wgrant: can you put the functions into the bug ?
<wgrant> lifeless: SUre.
<lifeless> so
<lifeless> theory is the analyse is not showing us the cost of a file processing of the rows in some fashion
<stub> It certainly is taking a lot longer to get the results to the client than it is to get the query plan.
<lifeless> yeah
<lifeless> one mystery solved
<lifeless> another item for the db performance tips page
<stub> Just seeing bug queries causing large temporary files in the logs
<wgrant> Oh hah.
<wgrant> I guess linux might have an enormous copyright file.
<wgrant> As well as a few uploads.
<wgrant> We should really do away with that column.
<wgrant> stub: Can you see how big a column is across an entire table?
<stub> I'm about to do that.
<stub> max(length(foo)) stuff
<wgrant> It could well be a few times the size of the rest of the table.
<wgrant> As a bonus, the data is not used by anything yet.
<lifeless> I selected into a temp table
<lifeless> \dt+ foo2
<lifeless>                     List of relations
<lifeless>    Schema   | Name | Type  | Owner | Size  | Description
<lifeless> ------------+------+-------+-------+-------+-------------
<lifeless>  pg_temp_30 | foo2 | table | ro    | 96 kB |
<lifeless> its only 96kB apparently
<wgrant> Did you select all the SPR changelogs into a temp table?
<wgrant> Or is that the result of the problematic select?
<lifeless> thats the result of the problematic select
<wgrant> I find that difficult to believe.
<wgrant> But I guess it's possible.
<lifeless> with id as id2, component as c2 etc etc
<lifeless> launchpad_qastaging=> select count(*) from foo2;
<lifeless>  count
<lifeless> -------
<lifeless>     72
<stub> 1.1 mb is the largest text field in the current slow query
<stub> (the one from the last oops)
 * stub checks the entire table.
<lifeless> stub: whats the sum of that field ?
<wgrant> DF is being a bit slow at summing all the lengths.
<stub> 81MB...
<wgrant> Ouch. Which SPR is that?
<lifeless> stub: whats the length() function return ?
<stub> That is the sum of copywrite... so multiple
<lifeless> I got noddy small values
<wgrant> Hah.
<wgrant> 2.5GB of linux copyright files on DF.
<lifeless> but the text is big
<stub> sum of the length? 84466050
<lifeless> no
<stub> 81MB
<lifeless> I mean hte function
<lifeless> select max(length(changelog_entry)) from foo2;
<lifeless>  max
<lifeless> ------
<lifeless>  3418
<stub> 1.1MB
<lifeless> what does the 3418 mean ?
<lifeless> is that pages? sectors?
<stub> Its bytes
<wgrant> changelog_entry?
<stub> Sorry - characters
<wgrant> You mean changelog?
<lifeless> wgrant: changelog is an int
<wgrant> changelog_entry is different.
<wgrant> Er.
<wgrant> copyright, not changelog
<stub> So UTF-8 might theoretically be 4x as many bytes?
<wgrant> changelog_entry is always going to be small; it's only the latest.
<stub> I've checked all text fields on the table. The problem is copyright as you suspected.
<lifeless> select max(length(copyright)) from foo2;
<lifeless>    max
<lifeless> ---------
<lifeless>  1126214
<lifeless> (1 row)
<lifeless> ok, confusion sorted
<stub> http://paste.ubuntu.com/578202/
<lifeless> right
<lifeless> dropping copyright fixes it
<lifeless> what do we have this in there for ?
<wgrant> Nothing at all.
<wgrant> It's populated, but not used.
<lifeless> drop the column definition from launchpad ?
<wgrant> We could possibly drop it from the class immediately, and just set it on upload, and then migrate it out of the DB later.
<lifeless> that will stop storm querying it
<wgrant> Right.
<stub> So yeah, that column needs to be split into a separate table. Code only fix might be to remove that field from the main Storm class, and have a separate Storm class with the extra column and use that only where necessary.
<wgrant> stub: s/table/librarian/, I think.
<lifeless> query can show all the rows in 300ms with that field removed.
<stub> wgrant: If there is no need for it to be in the DB, sure.
<lifeless> wgrant: want to do this one?
<stub> wgrant: If it is being shown inline on the page, I guess that is a performance call (pull it from the librarian and render it will be slower than from the db... unless it is ajax, and then search engines won't see it)
<wgrant> lifeless: I'll drop the column from the Storm definition now.
<lifeless> wgrant: cool
<lifeless> stub: its not used at all
<lifeless> stub: premature optimisation years ago
<wgrant> stub: It may plausibly be displayed in the page.
<stub> k
<wgrant> But it isn't yet.
<wgrant> And if someone wants to, they can damn well pull it from the librarian.
<lifeless> iframe embedding ftw
<wgrant> And if someone wants to display multiple on a single page, they are probably wrong.
<wgrant> So we don't have to consider that case.
<stub> So asuming ascii text, and Python decoding it into Unicode, that is 324MB of RAM needed before it even gets past psycopg2. This sort of thing will certainly be driving our memory footprint.
<lifeless> or they can just hand out librarian urls
<wgrant> I wonder if the uploader can update SPR. I guess I'm about to find out.
<stub> Given Python won't clean up, and that is per thread.
<lifeless> stub: ok, next one up is fti
<wgrant> I've been concerned about SPR.copyright for well over a year now, but never had the data to confirm that it was a problem.
<lifeless> wgrant: data is wonderful isn't it
<stub> I prefer a bucket of sand
<stub> ;)
<stub> nah... nah... fti... can't hear you... nah... nah
<jtv> stub: I just put up a security.py cleanup for your review that also gets it to run under 3 seconds.
<stub> So the query I looked at the other day had 32 lines of multiple boolean operations going on. I'm surprised it doesn't timeout just generating the plan for that, let alone get around to running the query
<stub> jtv: Ta.
<lifeless> stub: the plan is simple :)
<lifeless> stub: its an 800ms query on qastaging
<jtv> stub: Just testing if you really couldn't hear anyone.  :)  https://code.launchpad.net/~jtv/launchpad/faster-security/+merge/52804
<lifeless> stub: http://pastebin.com/7DuF11Z8
 * stub finds the bug 726175
<_mup_> Bug #726175: DistributionSourcePackage:+filebug Timeout trying to file bug due to FTI timeout <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/726175 >
<stub> And 3.6 seconds on production
<lifeless> stub: doing the same show-the-results test?
<stub> By accident, yes....
<lifeless> :)
 * stub waits for his terminal to return to normal
<lifeless> so 3.6 is << 15
<stub> So width isn't the problem here - no insane targetnamecaches or statusexplanation
<lifeless> stub: right, and it performs tolerably interactively
<stub> 65% of the index is data, which is a little bloated but not bad (launchpad_prod_2 here)
<lifeless> using package git not installed failed to install upgrade ErrorMessage: subprocess installed post-installation script returned error exit status 1 as a search term the search completes right now on prod
<lifeless> at /ubuntu/+filebug
<lifeless> huwshimi: hey
<lifeless> huwshimi: I have a challenge for you
<lifeless> huwshimi: when an ajax/api request is made, I'd like to put the time in the top right - you know where - into a journal of items
<stub> The limit isn't helping at all - all the rows are going to be pulled to calculate the rank so things can be ordered.
<lifeless> sure
<lifeless> it would be nice to set a cap on the relevance but I don't htink our tsearch setup is ready for that
<huwshimi> lifeless: so you'd like to figure out how to get the time for the roundtrip?
<lifeless> huwshimi: and glue it all together :)
<lifeless> huwshimi: its not urgent
<lifeless> huwshimi: but its the next step in visibility to devs of perofrmance
<stub> http://paste.ubuntu.com/578204/ is the plan I'm looking at btw.
<lifeless> stub: yeah, thats what i see too
<lifeless> stub: except mine is a little faster
<lifeless>  Sort  (cost=41.08..41.09 rows=1 width=650) (actual time=554.699..554.699 rows=1 loops=1)
<stub> lifeless: So wtf is the query doing for the first 1.4 seconds?
<stub> The fti stuff starts 1412 ms in.
<lifeless> stub: search me
<stub> Oh.... remember I mentioned those big temporary tables in the pg logs?
<lifeless> stub: I think you may have *meant* to, but I don't remember a discussion
<lifeless> stub: or you did tell me and I've lost it ;)
<wgrant> He mentioned them.
<huwshimi> lifeless: I'm not sure about YUI, but with other ajax frameworks you can add hooks to every ajax event. And then it's just a matter of recording the time the ajax event fires and then comparing it to the time when the ajax event completes. I'll take a look some time
<stub> (13:06:15) stub: Just seeing bug queries causing large temporary files in the logs
<wgrant> Gnargh gina.
<lifeless> stub: ah thaks
<StevenK> wgrant: ?
<lifeless> stub: you think this is it ?
<huwshimi> lifeless: The one thing I've learnt though is that everything is an order of magnitude harder with YUI than other frameworks :)
<lifeless> huwshimi: what framework should we be using?
<wgrant> StevenK: I'm removing SPR.copyright.
<jtv> stub: I may be missing the point since I just jumped in, but all it's got at the point where the 1.4 seconds are lost is 1 tuple
<jtv> (See all the way at the bottom of the plan)
<huwshimi> lifeless: Do you really want to go into that? :)
<StevenK> wgrant: Why? I think some people want it.
<lifeless> huwshimi: sure, why not?
<wgrant> StevenK: I'm removing it from the class for now. It will be moved into the librarian later.
<lifeless> stub: launchpad_qastaging=> SELECT BugTask. ... assignee |  bug   | bugwatch | date_assigned | date_closed | date_confirmed | date_fix_committed | date_fix_released | date_incomplete | date_inprogress | date_left_closed | date_left_new | date_triaged |        datecreated         | distribution | distroseries |   id   | importance | milestone |  owner  | product | productseries | sourcepackagename | status | statusexplanation | t
<lifeless> ----------+--------+----------+---------------+-------------+----------------+--------------------+-------------------+-----------------+-----------------+------------------+---------------+--------------+----------------------------+--------------+--------------+--------+------------+-----------+---------+---------+---------------+-------------------+--------+-------------------+-----------------
<lifeless>           | 715778 |          |               |             |                |                    |                   |                 |                 |                  |               |              | 2011-02-09 14:06:41.463232 |              |              | 836597 |          5 |           | 3439478 |   24742 |               |                   |     10 |                   | Linaro GCC
<lifeless> (1 row)
<lifeless> Time: 861.816 ms
<lifeless> jtv: thats a relative offset, not absolute
<lifeless> jtv: because its in the nested loop
<lifeless> jtv: so its at 1413.054 + 0.040 and takes 0.042
<jtv> lifeless: it's in the same nested loop where the other part starts at 1.4 seconds
<jtv> Don't they both count from the start of the loop?
<lifeless> no
<huwshimi> lifeless: OK, I think there are a bunch of issues with YUI. Its development is really slow and can not keep up with the pace of other frameworks. With the way things are going at Yahoo I also worry that there will get to a point where it may stop being developed by Yahoo at all.
<lifeless> or at least, I've only been able to make sense from plans if I assume the answer is no
<huwshimi> lifeless: The community around YUI is tiny and I don't think a serious community effort would be made to keep it going (I could be wrong and by the community taking it over it might help drastically)
 * jtv retouches his fading formerly-yellow note saying "read actual documentation"
<lifeless> huwshimi: so thats a bunch of risks
<huwshimi> lifeless: All of this means that there are a lot of half baked features in YUI.
<lifeless> not really an argument for where to aim *at*
<stub> jtv: From the plan, it seems we start looking at the fti index at 1.4 seconds in. Before that, the only thing that was done was an index scan on bugtask_product__bug__key that completed less than 1ms in.
<huwshimi> lifeless: Some things people seem to really like. Like the testing framework
<stub> (looking at the start times in actual time=)
<stub> I can't see it being a temporary file issue since the plan is reporting memory sorting.
<huwshimi> lifeless: But for many things it seems that you write more code and spend more time getting around YUI's drawbacks than you should.
<jtv> stub: soâ¦ some weird combinatorial behaviour in the planner for those booleans?  Try removing one and see if the time halves.  :)
<huwshimi> lifeless: Also, I'm not a YUI expert, from discussions with sinzui, he has very similar feelings about a lot of this and he might be a better person to talk to.
<stub> It might be spending 1.4seconds working out wtf that obscene boolean calculates down to...
<stub> I can time that...
<huwshimi> lifeless: Another side of YUI having such a small community is that the plugin ecosystem is tiny.
<lifeless> stub: that last line is a per-found-bug inner loop
<lifeless> stub: it can /only/ execute after the fti starts spitting out rows
<huwshimi> lifeless: We would save a lot of development time if we didn't have to reinvent the wheel for a lot of stuff.
<lifeless> huwshimi: so, there is a slightly larger discussion to have if we decide that a different framework would be better
<huwshimi> lifeless: And most of the plugins that do exist were developed a number of years ago and are not maintained or don't work with new version of YUI
<lifeless> huwshimi: but I want to be clear that it is a discussion we can have if you want
<huwshimi> lifeless: I would be *very* open to that discussion.
<lifeless> huwshimi: then I suggest you start it up :) - talk to sinzui, talk to rockstar
<lifeless> huwshimi: take it to the canonical-rhinos list if those two folk are agreeable
<lifeless> feel free to cc me
<huwshimi> lifeless: my concern is that we have a lot of existing YUI dependant code
<lifeless> huwshimi: we had a lot of slow code 6 months ago
<huwshimi> lifeless: Haha
<lifeless> huwshimi: but we're down to 2.7seconds for our 99th percentile request time
<lifeless> huwshimi: so big things can be don
<lifeless> e
<lifeless> huwshimi: and we've much less yui code than slow code
<LPCIBot> Project windmill build #32: FAILURE in 1 hr 8 min: https://hudson.wedontsleep.org/job/windmill/32/
<wgrant> Heh. It hasn't failed in more than a week, and it fails while we are discussing its demise...
<wgrant> Although it looks like it could be a real failure.
<huwshimi> lifeless: Is there a particular reason you're bringing this up (I'm just kind of surprised that you've raised the topic)?
<lifeless> huwshimi: you whinged
<huwshimi> lifeless: Haha ok
<lifeless> huwshimi: my job can be summarised as:
<lifeless>  - figure out what makes delivering on our goals hard for our devs
<lifeless>  - and arrange for it to be fixed
<lifeless> stub: to make sure we are looking at the same thing
<lifeless> http://paste.ubuntu.com/578212/
<huwshimi> lifeless: I've very aware that just because I have opinions on (or prefer) things it doesn't mean that I'm right. I wouldn't want to duplicate a bunch of effort in rewriting/training etc. without it being worth it.
<lifeless> huwshimi: indeed, so thats part of the discussion that you need to have
<huwshimi> lifeless: As we're moving to a more javascript heavy version of the site this is probably a good time to discuss it
<stub> lifeless: So the difference between the qastaging query plan and the production query plan is the qastaging query plan starts doing the fti index scan 230ms in, and the production query plan starts doing the fti index scan 1412ms in. That seems to account for most of the difference.
<lifeless> stub: ok
<lifeless> stub: how do we do we figure out the 15000seconds case ?
<stub> So on #postgres, seems unanimous you see this when PG is waiting for disk
<lifeless> stub: -> #postgres then
<wgrant> Hah.
<wgrant> The test suite breaks if you have debian-keyring installed.
<lifeless> \o/
<wgrant> Because dpkg-source then has keys to verify some of the packages in the gina test archive.
<wgrant> It by default allows an unverifiable signature, but refuses to unpack a bad one.
<lifeless> win
<lifeless> \o/ the with query works
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/221938
<_mup_> Bug #221938: Email interface crashes when an attachment file name contains a slash <email> <lp-bugs> <oops> <qa-needstesting> <Launchpad itself:Fix Committed by thumper> < https://launchpad.net/bugs/221938 >
<wgrant> lifeless: Currently fixing Windmill breakage (real bug, but it won't affect any production data).
<lifeless> bah, no wally
* mthaddon changed the topic of #launchpad-dev to: Launchpad down/read-only from 09:00-10:30UTC for a code update | https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> :(
<wgrant> ec2 mail fails if your default bzr email address isn't @canonical.com.
<StevenK> My instances mail me at my debian.org address just fine
<wgrant> You're probably not using the canonical.com SMTP server, though.
<StevenK> I am, and my from address is @ubuntu.com
<StevenK> So it's fine
<wgrant> Hmm.
<wgrant> It stopped working for me today, and I changed my default email address last night...
<StevenK> wgrant: Happy to share headers if it will help.
<wgrant> StevenK: bazaar.conf would be handy.
<StevenK> Hmm, when did smtp.c.c start being youngberry ...
<stub> debian.org might be special cased given the background of our admins...
<StevenK> wgrant: https://pastebin.canonical.com/44501/ is the relevant bit
<wgrant> StevenK: Your default bzr email address looks a lot like ubuntu.com
<StevenK> Which is what I said my From address was ...
<wgrant> Oh, mail you *at* your debian.org address.
<wgrant> Right.
<wgrant> Fail.
<StevenK> wgrant: You can't talk via smtp.c.c if your From address isn't canonical.com or ubuntu.com. If you want to use something else as your From address, use a different SMTP server.
<wgrant> StevenK: Sure. But my email address is @canonical.com for my branches... ec2 must not use locations.conf, or it uses some other path.
<StevenK> I didn't think your bazaar config was copied to the instances ...
<wgrant> It does some evil stuff.
<wgrant> Not copying the file directly, but some of its values.
<lifeless> wgrant: what are you fighting
<wgrant> lifeless: Hm?
<lifeless> js
<lifeless> whining
<lifeless> whats up
<wgrant> Bug #732442
<_mup_> Bug #732442: disable_existing_builds compares series name to display name <recipe> <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/732442 >
<wgrant> Given up for now, will ask wallyworld tomorrow.
<lifeless> stub: hey, so the analyze might have helped ?
<lifeless> wgrant: oh right
<lifeless> wgrant: ;(
<wgrant> I say it's qa-ok, although that is sort of cheating.
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/unuse-spr-copyright/+merge/52808 may interest you.
<lifeless> stub: how often do we do full backups ?
<stub> lifeless: daily dumps
<stub> lifeless: No, analyze didn't change anything.
<stub> lifeless: So waiting a while, I still see the initial query with a high startup time and a query immediately after a lower startup time. So I think we must be seeing the effects of shuffling data between disk cache and the pg shared memory area.
<lifeless> stub: thats plausible
<stub> lifeless: Currently, the shared memory area is set to 5GB on all the production boxes, which is the high side of best practice and when people start seeing degraded performance.
<lifeless> stub: can we change this during this downtime ?
<lifeless> stub: ah
<lifeless> stub: how can we test to see whether we would suffer
<stub> lifeless: I'm considering bumping it up to 7GB on one of the slaves. Even though we will be going 'too high' in most peoples opinions, we do have an unusual load so best practice might not apply here.
<stub> We have run with 8GB before, no particular ill effects, so I don't think it will hurt. And we can then check out differences in performance between the two slaves.
<lifeless> +1
<stub> But still, even 8GB will be lower than the hotset of data.
<lifeless> whats it set to on staging?
<stub> probably about 3...
 * stub checks
<stub> Of course, it could be counter intuitive and lowering the value might work better ;)
<wgrant> Hah
<stub> 2GB on staging
<adeuring> good morning
<wgrant> Hm, no.
<wgrant> stupid ec2.
<jam> morning all
<wgrant> Morning jam.
<jam> man, living in a country where you don't speak the language causes all sorts of web confusion
<jam> every site wants to default me to Dutch
<jam> stupid geoip :)
<jam> maybe google is just on crack. Because I manage to get to the "Settings" page, set my language as English, looks ok
<jam> go back to account settings, and everything is back in dutch
<bigjools> yeah that's annoying when travelling to sprints
<jam> signing out and back in again seemed to help in the end
<jam> but yeha
<jam> yeah
<jam> wgrant: are you watching the upgrade? I'll be happy to start testing/monitoring once things are up
<wgrant> jam: I'm watching.
 * wgrant opens the graph.
<jam> wgrant: looks like bzr-sftp still isn't wanting to shut down cleanly. getting the "cannot shutdown reactor that isn't running"
<jam> Maybe that is the code that wants to always cleanly shut down, waiting for the last connection to exit
<wgrant> jam: There were 5 connections remaining when it was forcibly killed.
<jam> wgrant: k, where do you get this info? Maybe I'm in the wrong channels?
<wgrant> jam: #launchpad-ops is where it happens, and you seem to be there.
<wgrant> But I checked the connection count myself.
<jam> how did you check it?
<jam> Yeah, I'm there, but I haven't been following as much as I should :)
<wgrant> jam: Certainly machines can see bazaar.launchpad.net:8022.
<jam> wgrant: "certain" machines ?
<wgrant> Um, yes, that.
<jam> wgrant: devpad doesn't appear to be one of them
<jam> wgrant: do you know the rsync module for the crowberry-sftp-log subdir?
<wgrant> jam: I did... let me check.
<wgrant> Could be sftp-logs
<wgrant> logs-sftp
<jam> crowberry::sftp-logs/ => unknown module
<wgrant> 20:39:26 < wgrant> logs-sftp
<jam> yeah, just found that myself
<wgrant> Great.
<henninge> Hi adeuring!
<adeuring> hi henninge
<wgrant> jam: Looking OK?
<henninge> Did you change anything on the branch since you last pushed it?
<henninge> adeuring: ^
<jam> wgrant: so far, haven't gotten to the end. the sftp service started slightly before the forking service, and was already getting connection requsetts
<adeuring> henninge: I just merged devel yesterday evening. nothing else yet
<henninge> yes, I saw that
<wgrant> jam: We do have slight ordering issues in both directions :(
<henninge> adeuring: You used "sharing status" throughout while I had been talking about "sharing details".
<jam> wgrant: as long as both are up when we consider the site 'live' I'm not really worried :)
<wgrant> Response times are still fine from here.
<henninge> adeuring: any particular reason for that naming?
<wgrant> No graphs yet.
<adeuring> henninge: no particular reason. I'll change it to details
<henninge> adeuring: I can do that.
<adeuring> henninge: ok
<henninge> adeuring: I finished the dummy template.
<adeuring> henninge: sounds great
<henninge> adeuring: you will have to add the conditions and such from the view.
<adeuring> henninge: ok
<henninge> adeuring: I will push that when I am done with the renaming.
* allenap changed the topic of #launchpad-dev to: Launchpad down/read-only from 09:00-10:30UTC for a code update | https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
 * allenap assumes abentley is no longer reviewing.
<StevenK> In a slightly related topic, is Firefox history editable?
<StevenK> (I have a whole bunch of edge URLs there that need to die. Slowly.)
<StevenK> allenap: https://code.launchpad.net/~stevenk/launchpad/derive-common-ancestor/+merge/52796 (and thanks!)
<allenap> StevenK: Got it.
<StevenK> allenap: afk for dinner, if you have questions queue them up here and I'll answer them when I can.
<allenap> StevenK: Cool.
<jam> wgrant: is it just that you're machine is allowed through the firewall to :8022?
<wgrant> jam: I guess it must be. I presumed carob could too, but I guess not.
<jam> wgrant: well 'w3m http://bazaar.launchpad.net:8022' didn't do much for me
<jam> could be a proxy issue
<wgrant> Oh, it's certainly not going to work externally :)
<jam> no, from carob
<jam> but I can do wget just  fine
<jam> so good enough, I guess
<jam> 134 conn
<jam> still says "unavailable" though, which is odd
<jam> I though spiv had a fix for that
<jam> wgrant: I'm a bit surprised about IP addresses that have 10+ connections active
<jam> (wget | sort)
<lifeless> jam: spiv couldn't reproduce it
<jam> just jumped to 246 conns...
<jam> wgrant: but I'm not seeing any failures, yet
<jml> https://code.launchpad.net/~jml/launchpad/what-is-in-the-web-ui/+merge/52594 up for review
<jam> note that conns includes ones that aren't authenticated, IIRC
<wgrant> jam: 246? That's not good.
<jam> not many access failures in the log, though
<jam> wgrant: 259
<jam> but no failures in the other logs
<henninge> adeuring: I will have to wait for the roll-out to finish before I can push my changes. :(
<adeuring> henninge: no problem
<wgrant> henninge: Hm? It should all be back now.
<henninge> oh, already?
<henninge> cool
<wgrant> henninge: Codehosting has been back for like 25 minutes.
<wgrant> jam: How are your connection times? Up to 8s here ;/
<henninge> oh, I just didn't try. thanks wgrant
<henninge> adeuring: pushed
<adeuring> henninge: I'll look
<henninge> adeuring: "pull" is the right term ;-)
* mthaddon changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
<jam> wgrant: https://code.launchpad.net/~jameinel/lp-production-configs/disable-forking/+merge/52818
<wgrant> jam: LOSAs review those.
<jam> wgrant: https://lpstats.canonical.com/graphs/CodehostingPerformance/20110310/20110311/nocache/
<jam> the 900s spike is going to kill the graph for a long time to come... :(
<wgrant> Heh, yes.
<StevenK> allenap: Thank you for the review!
<allenap> StevenK: You're welcome :)
<StevenK> allenap: How did you find the trailing whitespace?
<allenap> StevenK: I load the diff into my editor, and I've set it to show trailing whitespace as big red blocks. I can't miss it :)
<StevenK> allenap: Is your editor vim?
<allenap> StevenK: The other one.
<StevenK> Heh
<jam> I'm trying to set up lp on a new virtual host, but it is giving me failures during "make-lp-user".
<jam> https://pastebin.canonical.com/44511/
<jam> any ideas?
<wgrant> Ah,.
<wgrant> Well then.
<wgrant> Lucid?
<jam> yes
<wgrant> Did you use rocketfuel-setup?
<jam> yep
<jam> and then tweaked after for being in a VM
<jam> but not much changed there
<wgrant> You ran launchpad-database-setup?
<wgrant> make schema should have failed without it, but it's possible you did enough manually to unbreak it.
<wgrant> There should be "Launchpad configuration" bits at the end of postgresql.conf.
<leonardr> my internet connection is up and down rightn ow
<jam> wgrant: running it directly
<jam> I did very little manually
<jam> but I didn't see a Launchpad configuration bit at the end.
<jam> and I did have earlier problems connecting to postgres
<wgrant> jam: Try running it again, I guess.
<wgrant> It is meant to change the default search path.
<jam> yeah, running it now, but have to wait for "make schema" to finish
<jam> wgrant: looks like that did it. Thanks!
<wgrant> jam: Great.
<wgrant> 2011-03-10 09:40:34+0000 [SSHChannel session (0) on SSHService ssh-connection on ProtocolWrapper,36,80.11.180.42] Forking returned pid: 18870, path: /tmp/lp-forking-service-child-kwWa4B
<wgrant> echan, but yeah.
<jam> wgrant: that was the first death?
<wgrant> jam: It's the first one that's still alive at the end.
<jam> wgrant: if you grep around there, you can see what user it was
<wgrant> Indeed, but that was less than a minute after the service started...
<jam> wgrant: yay, even have codehosting serving on 5022. Though I wonder if there is a way to do that without hacking the source code.
<jam> so I don't have to worry about accidentally committing that
<wgrant> jam: You could possibly create a config overlay and run it with that.
<jam> wgrant: wouldn't integrate well with 'make run_codehosting' I imagine
<wgrant> jam: 'LPCONFIG=mycustomconfig make run_codehosting' should work.
<deryck> Morning, all.
<wgrant> leonardr: Hi.
<leonardr> wgrant, hey
<wgrant> leonardr: What is the recommended way to use a keyringed launchpadlib from a cron job?
<leonardr> wgrant: you need to store the credential in an unencrypted file, and pass it in as credentials_file
<leonardr> see section 5 of https://lists.launchpad.net/launchpad-users/msg06239.html
<wgrant> :(
<wgrant> OK.
<LPCIBot> Project windmill build #33: STILL FAILING in 1 hr 10 min: https://hudson.wedontsleep.org/job/windmill/33/
<vila> Hey all, what the update frequency for the downloads stats ?
<vila> as in https://launchpad.net/bzr/+download for example
<bac> hi mrevell
<mrevell> hi bac
<jam> anyone else here use Empathy? I'm trying to use it, since it integrates with the OS (and is the suggested default)
<jam> but it only really shows about 10-20 lines of actual content per page
<jam> with all the extra formatting
<jam> Is there a way to make all the bubbles smaller?
<deryck> henninge, adeuring -- ping for standup
<wgrant> bigjools: Hi.
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap, jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
<jml> allenap: thanks for the review. I had posted some updates as you were doing it (to docs, mostly) and have now fixed the zcml problem
<allenap> jml: Cool, I'll have a look.
<allenap> jml: Did you consider parsing the zcml directly?
<jml> allenap: to what end?
<jml> allenap: I mean, yes I did, but then I thought that all I'd be doing is constructing objects much like the adapters that Zope provides in the first place.
<allenap> jml: Because it might end up being less hacky... I haven't given it much thought, but wondered if you had.
<jml> allenap: it might fix some of the hackiness in format_page_adapter, but not much else. It would come at a cost of parsing ZCML myself
<allenap> jml: I guess the only thing you might avoid is filtering through the mro to find the real view.
<jml> allenap: yeah, exactly.
<jml> hmm.
<allenap> jml: Yeah, and processing includes. Okay. I don't think the mro thing is too bad actually.
<jml> another approach would be overriding the browser:page handler
<jml> but I'm not sure that would be less hacky, since our custom handler is specified in zcml also
<jml> so I'd have to find that ZCML and somehow exclude it.
<jml> also, Launchpad has 462 different types of page.
<deryck> henninge, I'm wondering about this "deactivate translation imports on a per package basis" card.  Is there a bug for this?
<leonardr> allenap or jcsackett, could you take a look at https://code.launchpad.net/~leonardr/lazr.restful/operation-must-be-versioned/+merge/52858?
<jtv> And I have one that's particularly relevant to allenap: https://code.launchpad.net/~jtv/launchpad/bug-730460-job-class/+merge/52857
<jcsackett> allenap: i'll take leonardr, you take jtv?
<allenap> Sounds good.
<jcsackett> leonardr: this the one that's been causing you and sinzui so much pain?
<henninge> deryck: hm ...
<henninge> oh!
<henninge> deryck: no, there is not.
<henninge> deryck: It came up in a discussion and is something that still needs to be done before the feature can really go live.
<deryck> henninge, why do we need per-package disabling?
<henninge> deryck: but it should be quite simple
<henninge> we just need to find the place in the code where it needs to be done.
<deryck> henninge, can't you already disable by changing the template name?
<henninge> deryck: once a sourcepackage has translation sharing set up, imports from package uploads must stop.
<henninge> deryck: it should only import the template from then on.
<henninge> deryck: I don't know what you mean about changing thetemplate name.
<henninge> deryck: I will file a bug and explain there.
<deryck> henninge, ah!  I get you now.  I misunderstood.
<deryck> henninge, yes, please file a bug and tag it with the story.
<jml> wgrant: I thought you said PQM was way faster now.
<henninge> deryck: bug 732612
<deryck> henninge, thanks!
<henninge> deryck: updated card
<deryck> henninge, thanks, again! :-)
<leonardr> jcsackett: no, i'm only working a half day today so i'm not even touching that one
<jcsackett> leonardr: sounds wise. :-)
<leonardr> oops, forgot to push the latest version
<leonardr> it's pushed now
<deryck> abentley, bug 719521 is fix released, yes?
<jcsackett> leonardr: i gather this branch is part of helping us not randomly bork the 1.0 version of the webservice?
<leonardr> jcsackett: exactly
<abentley> deryck: I don't think so.
<deryck> abentley, no?  The card is in done-done.
<abentley> deryck: I can move it back if you'd like :-)
<deryck> abentley, heh. well, I don't know. :-)  What's left to do?
<abentley> deryck: the code is in place, but the configs need updating and the cron script needs to be configured before unlinking will actually split translations.
<deryck> abentley, ok, the cron script is the same for linking, right?
<abentley> deryck: right.
<jcsackett> leonardr: r=me.
<deryck> abentley, ok, so I will close the bug as fix released.  Since the coding is done.  We have a card for the configs.  and the cronscript is an RT away.
<leonardr> jcsackett: just in time!
<jcsackett> :0()
<abentley> deryck: from an end-user perspective, no fix is released, so we could get pushback.
<jcsackett> replate that with ":-)"
<abentley> deryck: but of course it's your call.
<deryck> abentley, thanks, and I can live with that risk.  also, bug 696009 is fix released, too?
<jtv> I suppose jml was thinking lÃ¥Ã¼nchpÃ¤d
<abentley> deryck: yes.
<deryck> abentley, great, many thanks for the chat.
<jml> jtv: huh?
<abentley> deryck: np
<jtv> jml: wasn't that you?  About sticking umlauts on the name?
<jml> jtv: oh, right.
<jtv> "It's not bad, for a two-umlaut site"
<jml> :D
<jtv> LÃ¥Ã¼Ã±ÄhpÃ¤Ä
<deryck> abentley, sorry, one more.  bug 706005 is released?
<abentley> jtv: lauÃ±Ã§Ä§pad?
<jtv> Funny how most of the diacritics I can think to stick on there are actually more or less appropriateâ¦  the first "a" really is like "Ã¥," the "n" really sounds like "Ã±," and so on.
<jtv> abentley: what's the "h-like" letter?
<abentley> deryck: yes, but not deployed, like 719521.
<jtv> Ah, forgot oneâ¦ ÅÃ¥Ã¼Ã±ÄÄ§pÃ¤Ä
<deryck> abentley, gotcha. thanks.
<abentley> jtv: No idea, I've just seen it in names.
<jtv> You clearly hang out at cooler clubs than I do.
<abentley> deryck: should I be using YUI attributes or normal Javascript object properties by default?
<jtv> allenap: I'm going to have to call it a dayâ¦ can we continue the review offline?
<jtv> And by "we" I mean you.
<allenap> jtv: Sure, no worries. Have a good evening :)
<jtv> Thanks. :)
<deryck> abentley, YUI attrs.  Since it makes it obvious when you're doing get and set on an attr.
<deryck> and not silently adding something that didn't previously exist.
<abentley> deryck: coming from python, the risk of silently adding something that didn't previously exist doesn't seem very serious.
<deryck> abentley, yeah, maybe it's not in javascript, too.  but behavior is not always clear.  do I check for undefined or null if I want to be sure, for example?  Using YUI attrs offers a consistent API, among other benefits.
<deryck> henninge, adeuring -- I filed bug 732633 about your work, and made it team assigned.  please link branches there.
<adeuring> ok
<jml> wgrant: never mind, I'm behind an lp-production-configs branch.
<deryck> gah.  can't save two cards with the same bug id again.  I thought this was fixed.
<deryck> adeuring, because of bug ^^ your card lacks the bug link.  sorry.
<adeuring> 732633
<deryck> abentley, and I made bug 732639 for the js work if you want to link branches there.
<abentley> deryck: okay.
<bigjools> WOA, stop adding tests to Lanchpad.  "13373 tests run..."  :-)
<rvba> sinzui: when we were talking about modifying distroseries (to clean up this registrant/owner) thing, you said I had to options: migrate the field owner to registrant or add a registrant field that would return the content of the owner field.
<rvba> sinzui: At first I thought the most clean way to do this was to migrate the field but after looking at the code I'm not so sure because this object would then differ from most of the others ... any thought on this?
<sinzui> rvba: firstly, distroseries and productseries, being series should be the same in this case, I suspect milestones and releases should be the same. 1, the real owner is always the project/distro owner. The creator is the registrant. and has no power. So all four object use owner when we mean registrant
<sinzui> ^ well the 'firstly' and '1' never got answered in that sentence
<rvba> sinzui: should I be waiting to a secondly or a 2. then :-) ?
<sinzui> sorry, did I miss a message? I had display issues so I left the channel for a moment
<sinzui> rvba I think the distroseries is is larger than you started with. You are looking at a group of objects  that could have different owners in 2005, but, by 2007, the project/distro became the owner in the permissions rules, so we resused the owner field as the registrant
<rvba> sinzui: so I guess your advice is to fix the interface then
<rvba> creating a registrant field returning the content of owner
<rvba> sinzui: or do you think I should engage in refactoring the schemas to create a proper registrant field
<sinzui> rvba: storm does not require that the field name be the same as the db column. Several objects differ. The simply fix might be to rename the owner => registrant in the interface and model for the objects, and ensure that column=owner
 * sinzui looks for example
<rvba> sinzui: I get it, seems like a very simple fix
<sinzui> yes. I might have done it years ago if I had this conversation.
<rvba> once this is done, it's really nothing to migrate the data properly and rename the field in the database for consistency
<rvba> well, I guess :-)
<sinzui> great, the models look good they all specify dbname="owner". rvba: I think you can change the interface and models attribute names owner => registrant of distroseries, productseries, and productrelease
<abentley> henninge: do you expect the sharing details page to control translation permissions or translation group?
<rvba> sinzui: all right ... so I think it's most clean to migrate the db column as well
<sinzui> rvba: So you will need to update all the callsites and templates, but since this information is not very important, you will not need to change many.
<henninge> abentley: we need to set the permission to "Closed" if the project is unmaintained.
<sinzui> rvba: you certainly can change the column name
<rvba> sinzui: ok, that is pretty much what I did for the distributions so I think I'll manage thx
<abentley> henninge: how would we do that from the sharing details page?
<rvba> sinzui: you'll get to be the reviewer on this though :)
<sinzui> okay
<henninge> abentley: actually, thinking about it, I am not sure that we need to do it through ajax.
<rvba> sinzui:thanks a lot
<henninge> abentley: it's just that projects are created with "OPEN" permissions by default and that needs to be changed if the project is not using Launchpad.
<sinzui> I am happy to help
<abentley> henninge: so if a project is unmaintained, then it's owned by Registry Admins, and the logged-in user probably can't change the setting.
<henninge> abentley: so it's "if product.translation_usage != LAUNCHPAD: permission =CLOSED"
<henninge> true.
<henninge> so it's probably something that we need to change in the product creation itself.
<sinzui> has leonardr been about? I really hope not
<abentley> henninge: oh dear.  You add the card :-P
<henninge> sinzui: did anybody ever answer your question about reviewing translation imports?
<henninge> abentley: gee, thanks ;)
<sinzui> abentley: henninge about the previous conversation. Users often ask for the project back or ask me to change settings once they realise that giving the project to ~registry excludes them from configuring it...
<henninge> maybe we can do it in a way that it is owned by the creating user until all is done?
<sinzui> abentley: henninge: I proposed they we let anyone or trusted users be permitted to configure unconfigured projects or those owned by ~registry. The conversation went into audit logs and new showing permissions in the UI. So that was way out of scope. I stopped working on the bug
<jam> well, I at least managed to track down the cause of one of my critical failures today (bug #732481). Always funny when releasing code X triggers a bug in code Y.
<jam> anyway, EOD for now. Maybe I'll be back on to finish it later tonight.
<sinzui> henninge: as to my question, no, it was not answered. I am trying to answer the translations questions, but I am very slow at providing answers
<jam> mthaddon: if you are still around, can you post what you did to make HAPROXY use a GET request instead of a HEAD request, we probably want to revert it once we get bug #732481 fixed.
<henninge> sinzui: ok, let me see if I can shed some light on that
<abentley> henninge: If we did that, it would cover the case where the upstream project is new.
<henninge> abentley: yes, that is the one I am mostly thinking of. In the other case they'd need the help of the project's owner.
<abentley> henninge: which might be ~registry.
<henninge> oh
<sinzui> abentley: henninge: we did want project registration to be guided, take the user through all the configuration screens. Such a process should make the ~registry the owner at the end
<henninge> abentley: in that case we'd have to make the current user the owner temporarily.
<henninge> but how to know when "temporarily" ends?
<abentley> henninge: I don't know.
<abentley> henninge: we should be able to write a script to ensure all the ~registry-owned projects already have the right settings.
<bigjools> gary_poster: if a tarball in dists is not mentioned in versions.cfg, I can completely remove it right?  CherryPy has a zip file but is not in versions.cfg.
<gary_poster> bigjools, that *should* be true, but fwiw, versions.cfg is about enforcing the versions, not about enforcing their presence or absence.  One sec, there is an easy way to doublecheck...
<gary_poster> bigjools, there may be a nicer way, but ./bin/buildout -vvv | grep 'Getting required' will get you a list of everything that buildout wants.  you could grep the same output for CherryPy if you like
<bigjools> excellent, thanks
<gary_poster> and fwiw, no, we don't use it
<gary_poster> (having just tasted my own medicine)
<bigjools> :)
<bigjools> I was gonna leave 2 versions of everything "just in case", but bugger it, I am going to be ruthless.
<bigjools> bzr is used for a reason
<gary_poster> :-) being ruthless will require being careful, but I admire the idea
<gary_poster> and now, just after a rollout, is a good time to do it
<bigjools> as long as both db-devel and trunk build locally, I'm happy
<bigjools> exactly!
<bigjools> gary_poster: here's a weird one - Jinja 2.2 is required in versions.cfg but it doesn't exist
<bigjools> Jinja2 even
<bigjools> oh bugger
<bigjools> yes it does :)
<gary_poster> bigjools, as I said, versions.cfg is just about enforcing version numbers.  presence or absence is a dependency tree based on setup.py declarations.  So it sounds like you can remove it from versions.cfg.  buildout will complain if it doesn
<gary_poster> oh well there you go :-)
<bigjools> 2.4.1 and 2.5.5 are there though, I wonder why they're not used
<gary_poster> someone was unduly optimistic, I'm guessing :-)
<bigjools> very likekly :)
<bigjools> and other words that are spelled right
<gary_poster> heh
<sinzui> henninge: should ~rosetta-admins really own translation teams? I see from many teams listed on the +subscribe page that I own https://launchpad.net/gedit-plugins/+subscribe
<sinzui> henninge: When I see odd teams like this, they are often because someone made ~registry own a team, which is a no-no. So I either delete the team or make someone else an owner.
<henninge> sinzui: no, I don't think we should be the owner.
<henninge> they should be owned by someone from that team
<sinzui> okay, I will put that on my todo list
<sinzui> thank your
<sinzui> thank you!
<jml> who has filed the most bugs that are still open? http://paste.ubuntu.com/578421/ (yeah, I'm easily distractable today)
<sinzui> before I look, I think it is mpt
<henninge> sinzui: you win
<LPCIBot> Project windmill build #34: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/34/
<rvba> sinzui: """mars 08 16:46:20 <sinzui> It is not possible BTW to change a series owner. We removed the field from edit forms""" ... but the /ubuntu/hoary/+reassign is still there (and tested) ... is there something I don't understand or is this wrong?
<bigjools> jcsackett, the difference is that the upload processor is the lp_queue user and the ftp server lp_upload
<maxb> So basically we are questioning what exactly getUtility(IGPGHandler).getVerifiedSignatureResilient() verifies
<jcsackett> but you earlier had the gpg conf copied from one to the other, right? so our simplest solution is hosed? :-P
<bigjools> jcsackett: lp_upload had no gpg.conf at all
<bigjools> we copied lp_queue's
<bigjools> maxb: yes
<jcsackett> okay, i don't see anything obvious in getVerififiedSignature. there's a fair bit of cruft in there XXX notice-wise, but nothing obviously weird.
<bigjools> in lib/lp/archiveuploader/dscfile.py look for its usage in there - it's no different to the ftp server
 * bigjools scratches head
<adeuring> abentley: how can I figure out if a transtion sharing job is pending for a source package?
<bigjools> sinzui: can you think of anything that would make IGPGHandler.getVerifiedSignatureResilient return an error in one account but not another?
<maxb> bigjools: It couldn't possibly be simply a difference in where stderr points in the two processes, could it?
<bigjools> maxb: maybe, why would that have an affect?
<maxb> The only problem here is an unnecessary warning, right?
<maxb> Could that just be harmlessly disappearing into an ignored logfile in the upload processor?
<abentley> adeuring: You'll need to write a new query.
<adeuring> abentley: ok... so, other question: where are these jobs created? (I need a straing point to look around ;)
<adeuring> ...starting point..
<bigjools> maxb: the stderr goes to the log in both, AFAIK
<abentley> adeuring: Are you sure you don't want me to point out similar queries instead?
<adeuring> abentley: well, that would be fine too :)
<maxb> hrm. So I tried running a verify locally in an interactive python interpreter and got none of that output
<bigjools> maxb: the pastebin output is from dput though
<abentley> adeuring: see registry.model.packagingjob.PackagingJob.iterReady() and translations.model.translationpackagingjob.TranslationPackagingJob.iterReady for inspiration.
<bigjools> on the server side, it needs to see getVerifiedSignatureResilient throwing exceptions to return errors down the ftp session
<adeuring> abentley: thanks
<maxb> *blink*
<maxb> bigjools: Oh, I've completely misunderstood the issue :-)
<bigjools> maxb: yeah :)
<abentley> adeuring: But instead of checking whether the jobs are ready_jobs, you want to check whether their status is "pending".
<adeuring> ok
<abentley> adeuring: Actually, it's JobStatus.WAITING
<adeuring> thanks
<maxb> bigjools: gah, ok. So we need to trap whatever the exception is that is supposedly bubbling out of PoppyFileWriter.close() and breaking things?
<bigjools> maxb: yup.
<abentley> adeuring: you probably also want to look for running jobs.
<maxb> And that's not being logged anywhere useful at all currently?
<adeuring> right
<bigjools> trying to get to the logs right now, they're a bit out of date...
<sinzui> jcsackett: bigjools: the keyserver issue I was looking at was that that keys from keyserver.ubuntu.com were taking 24 hours to get to canonical's keyserver
<jcsackett> dig. not related.
<bigjools> yeah
<sinzui> rvba, mars was confused at the time. remember that the owner field on series has no power, users think it does so they want to change it. It is really the registrant. Some users think the registrant has power, but it does not. We should never let users edit the historical record of who registered it.
<rvba> sinzui: I'm sorry "mars = march" in french, this was a quote from what you told me yesterday.
<sinzui> rvba: just in case you ask, the driver or a series is called a release manager. That user can edit the series and can create milestones. The project owner delegates power over the series to the RM so that the user/team can accomplish their task
<rvba> sinzui: so for the distroseries, productseries, and productrelease it should not be possible to reassign them right?
<sinzui> correct
<rvba> sinzui: all right ... I'll have a few tests to refactor then ;-) ...
<rvba> I meant I'll have _quite_ a few tests to refactor
<sinzui> rvba: do we have tests that show changing the owner of a series?
<rvba> sinzui: ./lib/lp/registry/stories/distroseries/xx-reassign-distroseries.txt
 * sinzui looks
<sinzui> rvba: that ancient test can be deleted
<rvba> sinzui: so I figured if series don't have an owner ;-)
<sinzui> rvba: a story test is a user acceptance test. The narrative is written from the user perspective and show a simple path to accomplish a task. That test does not state who the user use, what is task is, show how he accomplishes it, and how he knows he is done. When you see stories like this, or testing error conditions, there is a good chance you can delete instead of refactor
<rvba> ok
<jml> allenap: could you please take a look at https://code.launchpad.net/~jml/launchpad/reported-by-me-121646/+merge/51148
<jml> allenap: would appreciate someone familiar w/ bugs taking a look at it.
<rvba> sinzui: I'll have to go someplace now ... but I think I'm ok to continue, I've made the structural changes (interface, model, sql patch) and then I correct things bit by bit as tests fails. Good to know that I can delete tests also ;-).
<sinzui> Have a good evening
<rvba> thanks a lot for your support. I think I'll have something for you to review when you log in tomorrow.
<bigjools> sinzui: hi, did you get my email about the ftp server bug?
<sinzui> bigjools: I did and I subscribed to the bug
<bigjools> sinzui: great, not sure out of you and tim who's best suited to fix it
<sinzui> we will talk about it in a few hours
<bigjools> something is arsed up with the logging which causes gpg verification to fail
<bigjools> jml might be able to help
<jml> what
<jml> I don't help, I *strategize*
<lifeless> thats very strategic of you
<jml> lifeless: hi
<lifeless> ola
<jml> there are a bunch of critical bugs for loggerhead that are fix committed
<jml> what has to happen to get them fix released?
<lifeless> I'm not sure
<jml> hmm.
<lifeless> we could do what we do with lp
<lifeless> and say 'once lp has deployed it, we're done'
<lifeless> encourage the packagers of loggerhead to package trunk
<lifeless> or we could do what we do with e.g. lazr.restful
<lifeless> and cut releases regularly
<jml> lifeless: we have a regular release for lazr.restful?
<lifeless> jml: we make a release when we fix something
<jml> oh right.
<Ronnie> if someone files a private bug to a project where im administrator of, why cant i view the bug?
<lifeless> Ronnie: are you the bug supervisor as well ?
<Ronnie> lifeless: how can i see that?
<lifeless> its on the front page for the project IIRC
<Ronnie> lifeless: the project-group where im admin of, is maintainer and driver
<lifeless> Ronnie: whats the project group and project in question
<Ronnie> https://staging.launchpad.net/ubuntu-nl-artwork
<jcsackett> sinzui: i think you and i got irritated by pqm at the same time.
<Ronnie> and bug: https://api.staging.launchpad.net/1.0/bugs/728920
<Ronnie> the bug is created with a script were testing
<sinzui> jcsackett: have you submitted a fix
<lifeless> Ronnie: https://bugs.staging.launchpad.net/ubuntu-nl-artwork
<jcsackett> sinzui: not yet submitted, but https://code.launchpad.net/~jcsackett/launchpad/resolve-conflicts has no text conflicts and the conflicted test file passes.
<lifeless> Ronnie: see the bug supervisor is a team, not you
<jcsackett> i'm not 100% certain it's a good resolution, as i wasn't entirley certain what some of the changes were doing.
<lifeless> Ronnie: have you added yourself to the team ?
<Ronnie> https://staging.launchpad.net/~ubuntu-nl-artwork/+members (name=Ronnie)
<sinzui> jcsackett: if db-devel does not conflict with stable, someone post a fix without says so
<jml> oh
<Ronnie> im also the owner of the team
<jml> I'm fixing that too
<lifeless> Ronnie: I know it shows you as an admin, but https://staging.launchpad.net/~ronnie.vd.c/+participation
<jcsackett> sinzui: no, there were conflicts when i merged stable into a checkout of db-devel.
<jml> jcsackett: I'll look
<jcsackett> jml: you mean pqm conflicts?
<lifeless> Ronnie: you may not actually be in the team
<jml> jcsackett: yeah. I'll take a look at your branch.
<lifeless> Ronnie: can other folk in that team see the bug ?
<jcsackett> jml: sinzui: okay. it's the garbo file i'm not sure i cleaned up right, but again, test_garbo passes.
<jml> I got distracted while make schema was running
<jml> jcsackett: yeah. I'll check to see if your fixes match mine
<jcsackett> jml: cool.
<jml> jcsackett: did you just union the changes?
<jcsackett> jml: yeah, i just whacked all the conflict gunk.
<jcsackett> most naive fix possible, but tests pass, so...
<jml> jcsackett: ok. that's what I did. And, as you say, the tests pass.
<sinzui> jcsackett: sorry for misunderstanding. I am struggling to concentrate this week. Too many issues in my head
<jml> jcsackett: go ahead and land it, r=jml
<jcsackett> jml: dig.
<jcsackett> jml: skip ec2?
<jml> jcsackett: I would.
<jml> jcsackett: thanks for resolving it, those mails are a pain.
<sinzui> jcsackett: yes, submit to pqm
<sinzui> okay, I will now fix leonardr's bugtask branch
<lifeless> http://webnumbr.com/.join(launchpad-oops-bugs.all,launchpad-timeout-bugs.all,launchpad-critical-bugs.all)
<sinzui> \o/ no critical bugs
 * sinzui opens the champaign.
<lifeless> sinzui: heh, not yet :)
<Ronnie> lifeless: another member of that team cant access the bug
<lifeless> ok
<lifeless> then its probably your script thats at fault
<lifeless> the rules for visibility on private bugs are:
<lifeless>  - if you are subscribed, you can see it
<lifeless>  - you aren't, you can't.
<lifeless> the default for private bugs filed through the web ui is to subcribe the security team or failing that the bug maintenance team
<Ronnie> new_bug = launchpad.bugs.createBug(title=mail['Subject'], description=message, private=True, target=forum_project, tags=tags)
<Ronnie> lifeless: thats the line that creates the bug
<sinzui> lifeless: Ronnie: has the bug supervisor changed? Lp really does not let you change it once set. Changing the bug supervisor will not change the permission on existing bugs. eg, the old supervisor still has access, and the new one does not
<sinzui> This scenario is a common way users shoot themselves in the foot
<Ronnie> sinzui: i didnt change the team settings for months now
<Ronnie> and were testing the script today
<lifeless> so first step
<lifeless> I'l file a bug for you manually
<lifeless> see if you can see it
<lifeless> bug 728921
<Ronnie> lifeless: nope, private
<Ronnie> ow wait, i used the api link, moment
<Ronnie> lifeless: i can see the bug you made
<lifeless> great
<lifeless> so its the api made bug not having subscribers
<sinzui> really! surely such an issue would have been reported years ago
 * sinzui looks
<sinzui> Ronnie: lifeless bug 398846 says it is low. I think that is insane since you can make it private at the same time
<sinzui> lifeless: High at least, do you think it is critical
<sinzui> https://bugs.launchpad.net/launchpad/+bug/398846
<sinzui> I see I glanced at the problem from a permission's sperpective
<Ronnie> is there some way to create the bug non private, add a bug supervisor and then make the bug private?
<lifeless> Ronnie: the usercode running the script can subscribe the security team
<lifeless> I've commented on the bug
<lifeless> bug.subscribe(person=yoursecurityteam)
<lifeless> you can probably get the security team from the api
<Ronnie> lifeless: ill try
<allenap> jcsackett: Do you need a mentored review for jml's branch from earlier?
<jcsackett> allenap: no, i'm a graduate. :-)
<allenap> jcsackett: Woohoo :)
<jml> allenap: it's already in EC2, but I would appreciate a quick glance if you've got the time
<allenap> jml: Sure.
<jml> allenap: mostly because I'm not familiar with searchTasks and maybe there are booby traps.
<lifeless> jml: jcsackett: before you land that patch
<lifeless> are you aware you are going to cause timeouts ?
<allenap> lifeless: The beauty of that portlet is that few people will notice ;)
<lifeless> allenap: /everyone/ will notice
<jml> lifeless: no, I'm not.
<lifeless> allenap: and we should care deeply about it
<jml> lifeless: why is it going to cause timeouts?
<lifeless> jml: because 'reported by me' is an expensive query that times out regularly on Person:+bugs
<allenap> lifeless: I'm being silly. But in this case, the portlet with links is rendered in the initial request, and the stats are calculated and slotted into place after the fact. So, only the people who read the numbers will notice. But, with my serious hat on, yes, we should care.
<lifeless> allenap: folk will notice, and think less of LP; folk will notice and ask in #launchpad; we'll have added another bug to the critical pile we're trying to reduce.
<jml> lifeless: the search never times out
<jml> lifeless: at least, not for me.
<lifeless> bug 421901
<lifeless> jml: you're adding the overhead of the search to the overhead of the bug counts portlet
<jml> lifeless: it's a different query
<Ronnie> lifeless: that worked:    new_bug.subscribe(person=project.owner_link)
<jcsackett> \o/ pqm success.
<lifeless> Ronnie: cool
<Ronnie> thx all
<lifeless> jml: reported by in ubuntu is up to 4 seconds of overhead.
<jml> well that was a huge waste of time
<lifeless> jml: you've added 14 bugs in ubuntu, so for you the query will be fast
<lifeless> at the other send of the scale, keybuk has filed nearly 2000 bugs in ubuntu
<lifeless> jml: I may be wrong
<lifeless> jml: we will need to qa carefully though - particularly by taking over canonical-scott on qastaging and making sure that the portlet doesn't time out
<lifeless>  bug 711071
<lifeless> is the existing bug about the portlet timing out
<jml> too late. I've cancelled the branch. I've got too much on at the moment to go through that for what was an opportunistic bug fix.
<lifeless> jml: I understand
<lifeless> jml: to help you assess things in future - if you add work to a page, assume its nontrivial.
<lifeless> jml: that includes adding links to person objects we don't already show, aggregate and non aggregate stats, tables, and branding
<lifeless> jcsackett: ^ when reviewing - you need to also think about the performance impact of a change : not /really really deeply/ - but just : 'has performance been considered? If not, what could go wrong?'
<jcsackett> lifeless: yeah, i saw searchTasks and thought about it, but didn't think that was an expensive query.
<jcsackett> i'll remember next time.
<lifeless> jcsackett: 4 seconds worst case isn't /terrible/
<lifeless> jcsackett: but the portlet in the distro context is 10 seconds already.
<allenap> jml: Is it worth leaving the link in, without the stats?
<lifeless> 10+4 ...
 * jcsackett nods.
<jcsackett> yeah, that's hitting our timeout threshold.
<jcsackett> not to mention sort of sucking. :-P
<lifeless> its a shame that we have things like the portlet which are already so slow
<lifeless> we're making progress though
<jml> allenap: for me, it would be an incremental improvement over having to type my nick in on the advanced search page every time I want to do it
<allenap> lifeless: The non-personal bug counts could be cached quite readily, and refreshed out of band.
<jml> allenap: but visually, it would look like a bug
<jml> allenap: like we forgot to add the count.
<lifeless> allenap: indeed, as long as we inject them rather than doing on-miss
<lifeless> allenap: -but- we can make it massively faster just using aggregate queries
<lifeless> allenap: see what I did for series + tag counts
<allenap> jml: Yeah, that's a good point.
<lifeless> allenap: we should make the miss case faster before going out of band: its more efficient for when we do need out of band / denormalisation in the future
<jml> lifeless: when we lower the timeout, do we automatically add exceptions for the pages that are still over it?
<lifeless> jml: if we have things go completely out of whack
<lifeless> jml: we don't default to do doing that
<jml> lifeless: hmm ok.
<jml> lifeless: the kernel team remonstrated with me about how our timeout lowering is blocking them from doing critical things
<lifeless> possibly https://bugs.launchpad.net/launchpad/+bug/732398 ?
<lifeless> which wgrant is landing a branch for today
<jml> lifeless: maybe. they are mostly from scripts they run, I gather.
<lifeless> if they have a particular thing that has /stopped/ working they shuld pop into #launchpad and we'll see what we can do
<jml> lifeless: they feel quite strongly that it is wrong to stop a fraction of users from doing their work so we can lower our timeouts.
<lifeless> that isn't the intention, and if they come talk to us when a particular thing stops working, we'll look and see whats up
<lifeless> yesterdays oops report shows 531 timeouts
<lifeless> and 6.3Million non-monitoring web pages served
<lifeless> thats 0.009% failure rate
<jml> lifeless: that includes API calls?
<lifeless> yes
<jml> lifeless: hmm. so either they are very unlucky or were speaking from stale data.
<lifeless> the page ids with # in them
<lifeless> https://bugs.launchpad.net/launchpad-project/+bugs?field.tag=timeout
<lifeless> are api calls
<lifeless> jml: they need to come communicate when the problem is happening. its /cheap/ for us to workaround *if* its a simple 'this is on the edge'
<lifeless> jml: in the past things have been taking 30+ seconds to complete.
<jml> lifeless: OK. I'll pass that on.
<lifeless> jml: which -regretfully- there is no sensible way for us to work around
<Ronnie> lifeless: another question, which characters are allowed for tags?
<jml> lifeless: they are working on a two week cadence now
<lifeless> jml: I hold the failure rate below 0.05%
<jml> lifeless: if they do come to us for help and you aren't around, is it clear what we should do?
<lifeless> Ronnie: lowercase
<Ronnie> lowercase and dash
<lifeless> jml: the CHR should evaluate the problem and decide what to do
<lifeless> jml: I don't think there is a cut and dried precanned answer
<lifeless> jml: in previous discussions with e.g. andy, they have not considered the costs of us leaving the timeout high
<jml> lifeless: ok. so that's a "no", at least for the moment.
<lifeless> jml: I have pretty huge overlap with the bulk of the kernel team - them being most us based AFAIK
<jml> lifeless: yeah. it's understandable. having the timeout high is a cost borne by the commons
<lifeless> jml: and by then - slower limits = choppier queuing = more latency on their own requests
<lifeless> s/then/them/
<jml> lifeless: yeah, tragedy of the commons.
<lifeless> jml: I've said in previous mails to the list that I'm happy for any dev to request a timeout exception - and for losas to Just Do Them
<lifeless> jml: all I ask is that it be <= 20 seconds, and there must be a critical timeout bug associated with it.
<jml> lifeless: ok, thanks.
<lifeless> I'm happy for losas to add them off their own bat as well, of course
<jml> lifeless: fwiw, apw reported that the site feels faster in the morning.
<lifeless> its very much a 'look after the site' kindof thing - exactly what ops should be doing
<lifeless> jml: naturally, we deployed nearly a weeks worth of timeout fixes
<lifeless> jml: its the downtick on http://webnumbr.com/.join(launchpad-oops-bugs.all,launchpad-timeout-bugs.all,launchpad-critical-bugs.all)
<jml> lifeless: not quite what I meant. During mornings in general.
<lifeless> oh
<jml> lifeless: before the US awakes.
<lifeless> thats probably in dc queuing
<jml> lifeless: that was my guess.
<lifeless> we are out of the terrible phase there
<lifeless> but still have a ways to go
<jml> (insofar as anecdata needs explanations)
<lifeless> we're bringing up 50 more appserver instances
<lifeless> probably not next week, as we're still a losa short
<lifeless> maybe even two
<jcsackett> jml: i officially love the word "anecdata"
<jml> jcsackett: it's one of my favourites.
<lifeless> thumper: Revision 12563 can not be deployed: needstesting
<thumper> lifeless: which one is that?
<lifeless> Strip any path information off email attachments when storing in the librarian
<thumper> lifeless: I thought I marked that as untestable
<thumper> https://bugs.launchpad.net/launchpad/+bugs?field.tag=qa-needstesting doesn't show it
<lifeless> thumper: ah, just recently
 * lifeless refreshes the report
<thumper> lifeless: well... in the last half hour
<lifeless> thumper: yeah, I'm jus tseeing if we can fix things for oem
<thumper> 9:17 according to the email
<sinzui> thumper: abentley: I am looking at https://answers.launchpad.net/launchpad/+question/148614 . I have seen this before. I think the svn repo is corrupt and we cannot complete the import. I do not see this kind of problem described in https://help.launchpad.net/VcsImportRequests
<thumper> sinzui: oh god
<thumper> sinzui: get them to request another import
<thumper> sinzui: as this one uses CSCVS
<thumper> sinzui: bzr-svn is much more robust
<thumper> that'll most likely fix it
<sinzui> oh, that's right. I was surpised to see cscvs but did not think getting a new one was the right thing to do
<sinzui> noted
<sinzui> thanks thumper
<wgrant> lifeless: Does garbo-hourly run regularly on qas?
<lifeless> wgrant: we did start it running, I don't know if its still running
<lifeless> wgrant: check with losa
<abentley> sinzui: you can simulate getting a new one using bzr-svn on your local machine, if you like.
<sinzui> abentley: good to know
<wgrant> lifeless: Can you do a quick 'SELECT COUNT(*) FROM sourcepackagerelease WHERE changelog IS NULL' on qas, so we can check?
<lifeless> not exactly quick.
<wgrant> Really?
<lifeless> 586610
<wgrant> Thanks.
<wgrant> This seems like it deserves an incident-level response...
<LPCIBot> Project windmill build #35: STILL FAILING in 1 hr 14 min: https://hudson.wedontsleep.org/job/windmill/35/
<wgrant> Particularly since we can't fix it today.
<wgrant> Because ELOSA.
<lifeless> wgrant: I agree
<wgrant> This should have been a drop-everything situation 8 hours ago :/
<wgrant> I realise that we have no EU maintenance teams, but...
<lifeless> benji: I filled out the deploy report
<benji> thanks!
<jml> huwshimi: hi
<huwshimi> jml: Hey there. Sorry I got distracted :(
<jml> huwshimi: np
<jml> huwshimi: skype?
<huwshimi> jml: yes
<jml> huwshimi: http://pastebin.ubuntu.com/578349/
<LPCIBot> Project windmill build #36: STILL FAILING in 48 min: https://hudson.wedontsleep.org/job/windmill/36/
<wgrant> wallyworld: Hi.
<wallyworld> hello
<wgrant> wallyworld: Bug #732442
<wgrant> That's the Windmill failure.
<wgrant> It's a real bug, but doesn't affect production data.
<wgrant> wallyworld: I looked at it last night, but it's non-trivial.
<wallyworld> wgrant: thanks. that was on my todo list to follow up. i'll fix it.
<wallyworld> good that the production issue is fixed
<thumper> wallyworld: hi, mumble?
<wallyworld> thumper: ok
<wallyworld> thumper: https://code.launchpad.net/~wallyworld/launchpad/inline-recipe-distro-series-edit
<wallyworld> thumper: https://code.launchpad.net/~wallyworld/launchpad/inline-recipe-distro-series-edit/+merge/52940
<wallyworld> thumper: http://people.canonical.com/~ianb/distroseries-checkbox.png
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On  call reviewer: allenap |  https://code.launchpad.net/launchpad-project/+activereviews
<wallyworld> thumper: https://code.launchpad.net/~wallyworld/launchpad/inline-multicheckbox-widget/+merge/52943
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wallyworld> thumper: http://people.canonical.com/~ianb/distroseries-popup.png
<lifeless> wallyworld: does it need a popup?
<lifeless> wallyworld: why not just edit in-place?
<wallyworld> lifeless: because you need to show all the choices
<wallyworld> lifeless: the html only shows the selected ones
<thumper> lifeless: just a design decision
<lifeless> wallyworld: hmm, I guess I'm thinking down the track
<thumper> lifeless: we aren't using this pattern anywhere else
<lifeless> wallyworld: e.g. youtubes 'save this video' ui
<thumper> lifeless: if we decide it is worth a change, we can look at it then
<lifeless> thumper: of course
<thumper> lifeless: it should be implemented as a widget, so easy to fix
<lifeless> benji: I'm volunteering to be the next 'rm'
<elmo> -fr
<lifeless> elmo: yes, thats my plan
<lifeless> gary_poster: with_ is deployed and working.
<lifeless> gary_poster: we're likely to change the api to make storm upstream happy, but changing a few callsites will be easy enough
<thumper> lunch is calling
#launchpad-dev 2011-03-11
<lifeless> -> pet store and stuff
<lifeless> gary-lunch: look at rev 12564 in devel to see how to do a with query with my patched storm
<wgrant> I tried to create a packageupload(archive, distroseries, status) index on DF. It works for queries like archive=534, distroseries=X, status=Y. But when archive=1 it refuses to use the index. I guess it thinks its stats know better.
<lifeless> do you have a sample query ?
<wgrant> EXPLAIN ANALYZE SELECT PackageUpload.archive, PackageUpload.changesfile, PackageUpload.date_created, PackageUpload.distroseries, PackageUpload.id, PackageUpload.pocket, PackageUpload.signing_key, PackageUpload.status FROM PackageUpload WHERE packageupload.distroseries = 103 AND packageupload.archive = 534 AND packageupload.status IN (0) ORDER BY PackageUpload.id DESC LIMIT 31 OFFSET 0;
<wgrant> That uses the index.
<wgrant> And is fast.
<wgrant> s/534/1/, no index, slow.
<lifeless> gotta run, bbiab
<wgrant> Oh. Now that is interesting.
<wgrant> It uses the index if distroseries=102
<wgrant> But not 103.
<mwhudson> is it the usual thing where it thinks it's going to be looking at most of the table anyway?
<wgrant> I presume so.
<wgrant> 103 does show up in pg_stats, so that may be it. But other archives that show up there don't see the same issue, so I might have to check out pg_statistics to see what it's doing.
<wgrant> Or harass stub.
<gary-lunch> cool thanks lifeless
 * gary-lunch wonders how he became gary-lunch
<StevenK> gary-lunch: It's a bit late for lunch, isn't it?
<gary_poster> yes indeed
<StevenK> Almost lunch time here, in fact.
<wgrant> Ah.
<wgrant> It will happily use packageupload(archive, distroseries, status, id).
<wgrant> I guess it must have decided that archive=1, distroseries=103 meant the final sort would be too expensive.
<wgrant> Despite the fact that there are actually 0 rows.
<lifeless> wgrant:  Limit  (cost=0.00..954.05 rows=31 width=36) (actual time=263.988..31328.809 rows=1 loops=1)
<lifeless>    ->  Index Scan Backward using distroreleasequeue_pkey on packageupload  (cost=0.00..93404.35 rows=3035 width=36) (actual time=263.985..31328.804 rows=1 loops=1)
<lifeless>          Filter: ((distroseries = 103) AND (archive = 1) AND (status = 0))
<lifeless>  Total runtime: 31328.871 ms
<lifeless> (4 rows)
<lifeless> 1.5 seconds hot
<lifeless> wgrant: thats qastaging
<wgrant> lifeless: Right, that's about what I see on mawson.
<wgrant> Except about 25% faster.
<wgrant> I'll talk to stub about unbreaking the indices tonight.
<wgrant> Since the current ones were designed before archives were introduced...
<lifeless> you're adding an index to match the critera more closely
<lifeless> whats the explain analyze of the query that (isn't using) the new index look like
<lifeless> wgrant: can you drop the order by ?
<lifeless> Time: 37.729 ms
<wgrant> lifeless: The current indices are not reasonable.
<wgrant> We need the order by.
<lifeless> wgrant: why?
<wgrant> packageupload(distroseries, status) does not make sense any more.
<wgrant> Since you never query PUs by series outside an archive.
<lifeless> wgrant: sure, I'm not debating that, just looking at whats going on
<lifeless> wgrant: I meant 'why do we need the order by'
<wgrant> lifeless: So we don't show them in arbitrary order on DistroSeries:+queue
<lifeless> wgrant: why would that matter ?
<wgrant> Because that would make batching sort of bad.
<lifeless> wgrant:  SELECT PackageUpload.archive, PackageUpload.changesfile, PackageUpload.date_created, PackageUpload.distroseries, PackageUpload.id, PackageUpload.pocket, PackageUpload.signing_key, PackageUpload.status FROM
<lifeless>                 PackageUpload WHERE packageupload.distroseries = 103 AND packageupload.archive = 1 AND packageupload.status IN (0) ORDER BY packageupload.distroseries, packageupload.status, PackageUpload.date_created desc LIMIT 31 OFFSET 0;
<wgrant> lifeless: That is cheating.
<lifeless> wgrant: so is life
<wgrant> There's no index on PackageUpload.date_created, so it has no choice.
<wgrant> It will be slow if it returns lots of results.
<lifeless> wgrant: right, so we would not want to do that for every query
<lifeless> wgrant: but the queue isn't unlimited
<lifeless> wgrant: and lots could be a few thousand without perf worries
<lifeless> wgrant: I agree we need to improve the indices
<wgrant> lifeless: https://launchpad.net/ubuntu/lucid/+queue?queue_state=3&queue_text=
<wgrant> 130000
<lifeless> wgrant: whats 3
<wgrant> lifeless: Done.
<lifeless> we should delete them
<wgrant> You could argue that.
<lifeless> s/c/sh/
<wgrant> It requires reworking lots of Soyuz. We should probably do that, but that is a *huge* *huge* thing.
<wgrant> We need to develop a strategy.
<LPCIBot> Project windmill build #37: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/37/
<wgrant> On how we handle history like this.
<wgrant> And apply it everywhere.
<StevenK> Windmill, you make me very sad.
<wgrant> StevenK: It's right, though.
<lifeless> wgrant: so, does the queue *post* need new indices to be fixed?
<lifeless> wgrant: or would my hack give breathing space?
<wgrant> lifeless: I personally think that some of this data is useful. The table is small, and it's extremely fast to query with the right indices.
<wgrant> lifeless: I don't think that hack is useful.
<wgrant> It would work.
<lifeless> ok
<wgrant> But an index is less of a hack, and works everywhere.
<lifeless> 455 Time Outs
<lifeless> nice
<lifeless> bugger
<lifeless>       23 /   22  POFile:+translate
<lifeless> 20 /    0  LanguageSet:CollectionResource:#languages
<wgrant> Wow.
<lifeless> ah
<wgrant> Bugger?
<lifeless> wgrant: things we had fixed
<wgrant> They weren't deployed yet, were they?
<lifeless> but yeah
<lifeless> 9 hours of them
<wgrant> They are now.
<lifeless> tomorrow will give better results again
<wgrant> Yep.
<wgrant> This is why I want 12-hourly ones :(
<lifeless> wgrant: I want a sliding window
<wgrant> Right.
<lifeless> plus per-deploy
<wgrant> But that needs oops-repository.
<wgrant> (I suspect)
<lifeless> well, can be done in the current code, just a matter of effort
<wgrant> Right.
<lifeless> I'm going to look into hbase as an alternative to cassandra
<wgrant> Oh?
<lifeless> need to send a raft of emails around
<wgrant> I haven't heard much about hbase...
<wgrant> But it's Apache too, right?
<lifeless> that aspect is fine
<lifeless> the issue with cassandra is lack of range queries
<wgrant> :(
<wgrant> lifeless: Do we expect daemons to survive DB restarts? I know eg. librarian doesn't.
<lifeless> wgrant: yes
<lifeless> wgrant: exact implementation to be figured out later
<wgrant> So that's a no?
<lifeless> today, restarting the db may wed daemons. If so thats a bug.
<wgrant> I'm tempted to leave poppy broken in that situation, given that most other daemons will have the same issue.
<wgrant> Can you see any problem with that?
<wgrant> It's not going to try to hold transactions open forever any more, so it shouldn't get killed :)
<lifeless> if you know of a specific defect, please make sure there is a high bug about it
<wgrant> I think it's a general defect in all daemons...
<lifeless> wgrant: I wouldn't assume that
<wgrant> Anyway, lunch, double poppy cowboy fixing, and then +copy-packages UI optimisations.
<wgrant> Because the actual copy operation is now fast :(
<lifeless> \o/
<StevenK> wgrant: That timeout was due to UI?
<LPCIBot> Project windmill build #38: STILL FAILING in 46 min: https://hudson.wedontsleep.org/job/windmill/38/
<wgrant> StevenK: Yes ;(
<wgrant> At least I think so.
<wgrant> It's a bit hard to tell exactly what's what.
<wgrant> Because the UI stuff generates a lot of queries.
<wgrant> It's also not copying binaries, which is a case that is less optimised. But it doesn't seem to get to the actual copies.
<wgrant> Erm.
<wgrant> "I get a timeout error when trying to copy a few (5+) packages from a ppa"
<wgrant> But field.selected_sources has at least 15...
<wgrant> I suppose that is technically still 5+...
<lifeless> :>
<lifeless> wow
<lifeless> Aggregate  (cost=7238627.72..7238627.73
<wgrant> Which query's that?
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1895B1870
<wgrant> Ah.
<lifeless> partitioning the query into two unioned on each fti column gets
<lifeless>  Aggregate  (cost=291.55..291.56 rows=1 width=0)
<lifeless> I've been dreading this
<lifeless> but its time to bite the bullet and do it
<lifeless> I'll do my administrivia stuff monday
<wgrant> Heh
<lifeless> I wonder
<lifeless> maybe I'll just ditch the bugtask fti column
<wgrant> Does that just index targetnamecache?
<lifeless> 16 more hits
<lifeless> for synaptics touchpad
<lifeless> targetnamecache and statusexplanation
<lifeless> -wtf-
<wgrant> I thought statusexplanation was actually dead now.
<wgrant> Apparently not :/
<lifeless> COMMENT ON COLUMN BugTask.statusexplanation IS 'A place to store bug task specific information as free text';
<wgrant> You know what it is, right?
<lifeless> no idea
<wgrant> You know how the old task edit form had a comment field?
<wgrant> That adds a comment and sets statusexplanation.
<wgrant> Those changes are also logged in bugactivity.
<lifeless> lib/lp/bugs/scripts/bugzilla.py uses it
<wgrant> If that's what I think it is, it will die.
<wgrant> I wonder if it still works.
<lifeless> the task edit form still exists
<lifeless> browser/bugtask.py has
<wgrant> Yes, but it doesn't need to set statusexplanation.
<lifeless> if comment_on_change:
<lifeless> bugtask.statusexplanation = comment_on_change
<lifeless>  select count(*) from bugtask where statusexplanation is null or statusexplanation='';
<lifeless>  count
<lifeless> --------
<lifeless>  531321
<lifeless> select count(*) from bugtask where statusexplanation is not null;
<lifeless>  count
<lifeless> --------
<lifeless>  480008
<lifeless> bah
<wgrant> Hmmm?
<lifeless> select count(*) from bugtask where statusexplanation is not null and statusexplanation != '';
<lifeless>  count
<lifeless> --------
<lifeless> so 300K set, 530K unset
<lifeless>  300198
<wgrant> Right. Kill it.
<wgrant> It has been thoroughly deprecated for more than 5 years.
<wgrant> We have these wonderful things called comments to store that information now :)
<lifeless> bug/88545
<lifeless> bug 88545
<lifeless> no mup?
<thumper> statusexplanation?
<lifeless> a discarded idea
<StevenK> Is it actually used in the codebase at all?
<lifeless> grep ftw
<wgrant> The only place I've seen it exposed is BugTask:+activity
<StevenK> lifeless: Isn't there another clean up you wanted to do with BugComment and Bug<name escapes me currently>
<lifeless> jcsackett is on that
<lifeless> or at least on nearby stuff
<wgrant> BugMessage.visible was moved this morning.
<StevenK> Ah yes BugComment versus BugMessage
<lifeless> there is a completely redundant python class constructed from bugmessage+message
<lifeless> if we had separation of concerns - a persistence layer - it would make sense
<lifeless> but its purely a Data Object at the moment, valueless.
<StevenK> libwebkitgtk-1.0.0-dbg DIALCF!
<wgrant> StevenK: Why?
<StevenK> 240MiB debug packages make me cranky.
<wgrant> Hah.
<StevenK> And it's Webkit, which I have other issues with.
<lifeless> like, it existing?
<StevenK> lifeless: Is that your issue with it?
<lifeless> no
<lifeless> bundled libraries
<lifeless> bugs
<lifeless> footprint
<StevenK> That's one of my issues -- my major gripe of it is just how much crap is in it
<wgrant> I prefer WebKit to Gecko... at least once it has been deGoogled.
<lifeless> its a whole desktop
<wgrant> ie. had the hundred external libraries excised.
<lifeless> what could be wrong
<StevenK> Which gives us wonderful things like the aforementioned 240MiB debug packages.
<StevenK> wgrant: Oh, Gecko is no better
<StevenK> From the same people who gave us building UIs from XML
<StevenK> Oh crap, now I need to review my lunch.
<wgrant> Um, most GTK apps now build UIs from XML :)
<lifeless> wgrant: exactly
<StevenK> Didn't XUL start that madness?
<wgrant> It is unclear that it is madness.
<lifeless> wgrant: you saw https://code.launchpad.net/~lifeless/storm/with-without-datetime right ?
<StevenK> Anyway, yay for feeping creaturism
<wgrant> lifeless: I never looked at the code.
<lifeless> wgrant: thats fine
<lifeless> whats the script that gives committer frequencies in lp
<wgrant> What do you mean? Number of commits by person for a particular branch?
<lifeless> within a timeframe, or per month, or whatever.
<lifeless> yes
<lifeless> I can has review? https://code.launchpad.net/~lifeless/launchpad/bug-716774/+merge/52966
<wgrant> I don't know of anything beyond bzr-stats...
<lifeless> wgrant: does PoppyFileWriter's validateGPG affect sftp uploads?
<wgrant> lifeless: No.
<wgrant> It's only used in FTP for now.
<wgrant> (WTF, I know, but convenient)
<lifeless> thanks
<wgrant> I guess I should start that IR at some point.
<lifeless> or ask me to do it
<lifeless> but don't sit on it
<wgrant> I'm sure you have better things to do. I'll do it.
<StevenK> wgrant: Julian says he couldn't work out how to do it for SFTP uploads. Which made me sad. :_(
<wgrant> :(
<StevenK> I'm hoping poppy-{s,}ftp now get a good hard refactoring
<StevenK> $DEITY knows they deserve it.
<wgrant> A renaming wouldn't go astray either.
<StevenK> Oh, and death to zope ftp would be nice, too.
<wgrant> lifeless: An advertised feature is no longer working. Not Critical?
<lifeless> wgrant: we've never had it live
<lifeless> wgrant: we tried and it went boom
<wgrant> It was working fine for like an hour :P
<lifeless> wgrant: that doesn't count
<wgrant> :(
<lifeless> julian blogging about it was great enthusiasm, but we probably want to wait for things to be stable before announcing
<wgrant> Indeed.
<lifeless> wgrant: point is, if we stopped work on the improvement right now, we haven't actually gone back on anything folk could have started depending on.
<lifeless> that said, I think we should push forward and get it going
<lifeless> it will make a huge difference to usability.
<lifeless> and that will mean happier and less confused users
<lifeless> lower support overhead and easier training for new ubuntu devs
<wgrant> thumper: Hi.
<thumper> wgrant: hi
<wgrant> thumper: Could you mentor https://code.launchpad.net/~lifeless/launchpad/bug-716774/+merge/52966?
 * thumper looks
<wgrant> Anyone want to review https://code.launchpad.net/~wgrant/launchpad/poppy-transaction-unbreakage/+merge/52967?
<StevenK> wgrant: Done. thumper ^
<thumper> OMG
<thumper> my recursive query worked
<thumper> http://pastebin.ubuntu.com/578642/
<StevenK> According to the postgres docs it's iteration, not recursion :_)
<thumper> does postgres have stored procs?
<StevenK> From memory, yes
<thumper> hmm
<StevenK> wgrant: Wait. Are you sure you ran bzr add?
<thumper> doing the blueprint dependencies in the db has to be better than doing X queries
<lifeless> thumper: yes
<lifeless> thumper: you can do that with the storm we have in trunk
<thumper> lifeless: how?
<lifeless> store.with(SQL('RECURSIVE dependencies(id, name, depth) AS (
<lifeless>         SELECT s.id, s.name, 1
<lifeless>         FROM specification s
<lifeless>         where s.id = 20428
<lifeless>       UNION ALL
<lifeless>         SELECT s.id, s.name, d.depth + 1
<lifeless>         FROM specificationdependency sd, dependencies d, specification s
<lifeless>         WHERE sd.specification = d.id
<lifeless>           AND s.id = sd.dependency
<lifeless> )')
<lifeless> bah
<lifeless> store.with_
<lifeless> so store.with_(clause).find(Specification)
<lifeless> or whatever
<lifeless> probably
<lifeless> so store.with_(clause).find(Specification, SQL('Specification.id in (select id from dependencies))')
<lifeless> thumper: care to review https://code.launchpad.net/~lifeless/launchpad/bug-716774/+merge/52966 ?
<thumper> lifeless: I have
<lifeless> oh sweet
<lifeless> thanks
<lifeless> sorry I hadn't noticed
 * lifeless ec2inates
<StevenK> Which just sounds ... wrong.
<wgrant> StevenK: What did I miss?
<StevenK> wgrant: It looks to me that you moved all the imports but not the actual method itself?
<wgrant> 211	=== renamed file 'lib/lp/bugs/externalbugtracker/isolation.py' => 'lib/lp/services/database/isolation.py'
<wgrant> 212	=== renamed file 'lib/lp/bugs/externalbugtracker/tests/test_isolation.py' => 'lib/lp/services/database/tests/test_isolation.py'
<StevenK> Ah ha. Objection withdrawn, the MP is still approved.
<thumper> I'm actually pretty stoked about that blueprint dependency query
<thumper> now to squeeze it into the code somehow
<wgrant> Let's hope we don't have to revert to pg 8.3 :)
<StevenK> Haha
<thumper> wgrant: recursive queries were added in 8.3
<thumper> at least I thought so
<StevenK> WITH RECURSIVE was added in 8.4
<wgrant> I thought WITH was added in 8.4.
<thumper> ah...
<wgrant> Right.
<thumper> I wonder what the likelyhood of that is then...
<wgrant> 0
<thumper> \o/
<StevenK> thumper: Can haz mentor? :-)
<wgrant> thumper: Could you mentor https://code.launchpad.net/~wgrant/launchpad/poppy-transaction-unbreakage/+merge/52967?
<StevenK> Haha
 * thumper looks
<wgrant> Thanks.
<thumper> 'tis past wine o'clock on a Friday
<thumper> you're lucky :0
<lifeless> we're not reverting
<lifeless> not to 8.3
<lifeless> that goose has flown
<StevenK> Can we revert to 9.0? That sounds like FUN.
<thumper> done
<wgrant> 9.0 already? Well done.
<StevenK> Damn it, I just missed a devel build on buildbot
<StevenK> ... by about 30 seconds. :-(
<wgrant> We can't roll out until Tuesday, either :(
<lifeless> tuesday?
<wgrant> ACT public holiday on Monday, and no mthaddon next week.
<StevenK> Haha, awesome
 * thumper wines a little
 * thumper AFK
<StevenK> s/w/wh/
 * thumper pokes StevenK in the eye
<StevenK> Ow!
<wgrant> :(
<wgrant> Does BugTask.status serve any useful purpose?
<lifeless> uhm
<lifeless> its where we store triaged etc
<wgrant> Opinion/Invalid/Won'tFix aren't usefully separated, New is implicit, Confirmed is implied by affects, In Progress is effectively implied by assignee, Fix Committed and Fix Released are implied by MP status.
<wgrant> So we can replace them all with "valid" and "incomplete" flags.
<cody-somerville> or customizable bug statuses. \o/
<StevenK> wgrant: Right, which works for *us*, but may not work for other projects.
<wgrant> StevenK: They work for anyone that uses MPs.
<wgrant> Unless they are abusing statuses, in which case they are wrong :)
<cody-somerville> no
<cody-somerville> its not 'wrong'. its the system speaking to you.
<StevenK> cody-somerville: What possible need do you have for customizing bug statuses?
<cody-somerville> Oh, OEM Services has tons since we abuse bugs for other things.
<huwshimi> wgrant: Take a look at the statuses proposed on this page: https://dev.launchpad.net/IssueTracker
<wgrant> huwshimi: Mmm. Better, but still not optimal.
<cody-somerville> ugh. not better IMHO.
<StevenK> What, two statuses? Fixed and Unfixed? :-P
<wgrant> cody-somerville: Why?
<wgrant> The only significant change is merging Fix Committed and Fix Released.
<StevenK> I think cody-somerville wants *more* statuses, not less.
<lifeless> so workflow around bugs is a fascinating topic
<cody-somerville> indeed
<StevenK> OH FER. libwebkitgtk-1.0-0-dbg isn't enough, libwebkitgtk-3.0-0-dbg exists too
<wgrant> StevenK: Why are you dealing with them?
<StevenK> Local mirror sync
<wgrant> Ah.
<wgrant> :(
<wgrant> DF doesn't have memcached installed.
<cody-somerville> but for what its worth... I think the current set of bug statuses are alright. I'm much more interested and would get much more value  (professionally and personally) from things like build-from-branch and derivative distributions. :)
<StevenK> cody-somerville: You know recipe builds are "done", right?
<wgrant> Recipes != BFB
<StevenK> Oh, right.
<StevenK> That saddens me.
<StevenK> lol. OO.o is in NBS
<cody-somerville> cause although tweaking the bug statuses might make LP better to some degree, it isn't going to make it more competitive against say github or OBS.
<cody-somerville> Anyhow, I should try and get some sleep. :) *waves*
 * wgrant murders get(Build)StatusSummar(y|ies)(For(Builds|Source(Publication|IdsAndArchive)))
<StevenK> wgrant: Seriously?
<LPCIBot> Project windmill build #39: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/39/
<wgrant> StevenK: Well, bits of them at least.
<StevenK> Aww
<wgrant> SourceForge's new platform is interesting.
<wgrant> Rather GitHubish.
<wgrant> And far less awful than their old one, at first glance.
<huwshimi> wgrant: They've just open sourced their backend too
<wgrant> Yeah, that's the new one.
<wgrant> Currently grabbing it...
<StevenK> Haha, of course you are.
<wallyworld> wgrant: i can't seem to run any windmill tests locally (to fix the breakage). is there a config item i need to tweak do you know?
<wgrant> wallyworld: --layer=WindmillLayer
<wgrant> It defaults to --layer=!(MailmanLayer|WindmillLayer)
<wallyworld> wgrant: thanks
<wallyworld> was that what you changed to stop the tests by default?
<wgrant> Yeah. It was previously just !MailmanLayer
<wgrant> (see buildout-templates/bin/test.in)
<wgrant> wallyworld: I considered just changing the vocab keys to series.name, but I'm not sure how much the form machinery will like that.
<wgrant> Evening stub.
<wallyworld> wgrant: you mean to fix the test? i'm still considering how to fix it. shouldn't be too hard
<wgrant> wallyworld: The test isn't broken.
<wgrant> wallyworld: The code is.
<stub> yo
<wallyworld> wgrant: yeah, can see that :-(
<wgrant> stub: mawson's garbo-hourly is a bit unhappy. Do I have to manually apply session.sql?
<lifeless> stub: how hard is it to update fti ?
<stub> Parts of launchpad_session.sql
<stub> wgrant: What bit is it barfing on?
<wgrant>  -> http://librarian.dogfood.launchpad.net/57720639/nqvma824KMUojGISQIkXotVXlmD.txt (function cursor_fetch(unknown, integer) does not exist
<stub> lifeless: Just edit the data structure at the top of fti.py - it handles creating all the triggers, columns, indexes etc. automatically
<lifeless> stub: can we update it live?
<stub> lifeless: No. Adding or changing the triggers requires a table lock.
<lifeless> k
<lifeless> was thinking of sneaking up on https://code.launchpad.net/bugs/88545
<stub> wgrant: So from database/schema/launchpad_session.sql you want CREATE OR REPLACE FUNCTION cursor_fetch and the GRANT EXECUTE ON FUNCTION cursor_fetch
<stub> (I'd applied this manually to staging and production systems before the code landed but forgot about mawson)
<stub> wgrant: That needs to be applied to the session database on mawson of course
<wgrant> Right.
<wgrant> Thanks.
<wgrant> Much happier, thanks.
<wgrant> stub: I also have some concerns about PackageUpload's indices.
<stub> wgrant: ?
<wgrant> stub: See the last comment in bug #276950
<wgrant> I don't think packageupload(distroseries, status) has been very useful in the last ~4 years.
<stub> wgrant: It seems to be used, but for what I can't tell: http://paste.ubuntu.com/578697/
<stub> Looks like we haven't deleted any GPG keys for a while though
<wgrant> stub: I guess it might try to use that if there's no order, even if the archive is constrained.
<wgrant> It avoids (archive, distroseries, status) for some values if it has to order by ID.
<wgrant> But will use it for others.
<StevenK> stub: Does that mean the signing_key index is unused?
<stub> StevenK: It means no lookups are done using signingkey. That index exists to support deleting a row from the gpgkey table (which needs to locate any referencing rows).
<StevenK> stub: Ah right, which also explains your deletion of GPG keys comment.
<StevenK> I'm pondering a new key, my current one is 7.5 years old
<stub> wgrant: Do you have a problem with the table, such as really slow restore times, disk space utilization, or slow insert times?
<lifeless> slow queries
<wgrant> stub: Just slow queries.
<wgrant> I think the existing index is pointless.
<wgrant> And should be replaced.
<wgrant> There's an example query in the bug I referenced.
<stub> It might make sense to replace it.
<stub> poke mup
<wgrant> It's been dead most of the day.
<wgrant> https://bugs.launchpad.net/launchpad/+bug/276950
<stub> wgrant: Your suggestion sounds sane. I wouldn't normally make such a specific index without knowing for sure it will be used, but you have already done that research.
<wgrant> (archive, distroseries, status) queries are done extensively.
<stub> We don't have to drop the existing index though - if it becomes unused I'll pick it up next time I do a scan. It will slow down inserts, but won't slow down selects.
<stub> Right. I don't think they used to, because we didn't used to have ppas etc.
<wgrant> Exactly.
<wgrant> I think this index is just a few years out of date.
<stub> Back from when the table was called DistroReleaseQueue it seems
<stub> (maybe I can rename primary key indexes without pain now in 8.4?)
<lifeless> stub: does postgresql have a log-slow-queries facility ?
<wgrant> (I initially tried without id, but the slow view's queries sort by id, and in some conditions it reverts back to the plain id index)
<stub> lifeless: yes
<lifeless> stub: can we turn that on, set to 15000ms, and get some reports around it ? Ideally we'd tie it back to the request/script
<lifeless> stub: I'm thinking with all the improvements we've done, we're at the point where starting to track slow queries isn't just insane
<wgrant> lifeless: You clearly haven't seen the scripts lately.
<lifeless> wgrant: I'm very worried about script performance
<stub> log_min_duration_statement. We should enable it for limited users though, perhaps just one appserver, to keep the logs sane.
<lifeless> stub: by sane, do you mean 'not overwhelming' ?
<stub> Yes
<lifeless> stub: rather than do it for limited users, lets set a higher minimum
<lifeless> stub: that way we don't have to keep updating it as more users are added
<stub> lifeless: So it will be full of garbo queries then.
<stub> We expect some longer queries for the batch jobs - that is why they are batch jobs.
<lifeless> stub: I want the backend tasks to be fast too
<lifeless> stub: for a few reasons - latency, total overhead, mvcc garbage overhead
<lifeless> stub: so I'm just as interested in garbo, rosetta etc as I am in appservers
<stub> So you won't see any appserver queries then, because I'd need to set the timeout to something nuts like 10seconds.
<lifeless> stub: I was suggesting an initial setting of 15 seconds
<lifeless> stub: we see 15 second queries on appservers at the moment
<lifeless> well, 13 now - cause they hit the timeout
<stub> I'll turn it on at 15s and see how our logs survive.
<lifeless> and expire on the server - but we still have a couple of overrides
<lifeless> cool
<stub> We already get the statements killed by timeout btw. They show up as errors in the logs.
<lifeless> cool
<stub> parsing the logs sucks though. The format is bogus.
<lifeless> the oops reporting system should be telling us about all of the expired ones
<lifeless> and i see this pattern a lot:
<lifeless> a few cheap queries
<lifeless> a mammoth query
<lifeless> a few cheap queries
<lifeless> an expired query
<stub> pgfouine might make a reasonable report even if we don't feed it full logs.
<wgrant> lifeless: Bugzilla integration? You mean the one-off script that imported Ubuntu Bugzilla?
<lifeless> wgrant: poppy could run in autocommit mode perhaps
<lifeless> wgrant: grep for the field, you tell me
<stub> I'm not seeing any log spam at 15s
<wgrant> Wait 'til the publisher runs :)
<stub> <update-pkg-cache@launchpad_prod_3:31495> 2011-03-11 07:22:14 UTC LOG:  duration: 88923.523 ms  statement: SELECT COUNT(*) FROM (SELECT DISTINCT BinaryPackageName.id, BinaryPackageName.name FROM BinaryPackageName, BinaryPackagePublishingHistory, BinaryPackageRelease, DistroArchSeries WHERE
<stub>                     BinaryPackagePublishingHistory.binarypackagerelease =
<stub>                         BinaryPackageRelease.id AND
<stub>                     BinaryPackageRelease.binarypackagename =
<stub>                         BinaryPackageName.id AND
<stub>                     BinaryPackagePublishingHistory.status IN (1, 2) AND
<stub>                     BinaryPackagePublishingHistory.pocket = 0 AND
<stub>                     BinaryPackagePublishingHistory.distroarchseries =
<stub>                         DistroArchSeries.id AND
<stub>                     DistroArchSeries.distroseries = 103 AND
<stub>                     BinaryPackagePublishingHistory.archive IN (1, 534)
<stub>                      AND (1=1)) AS "_tmp"
<stub> So its working.
<stub> And hopefully the publisher doesn't have any queries that evil ;)
<wgrant> Well, some of its queries take 10 minutes on mawson... they're not quite that bad on prod, though.
<lifeless> stub: I'd leave it for a day
<lifeless> review on monday
<stub> I'm about to poke the GSAs to twiddle logrotate to guard against this taking us down.
<lifeless> stub: awesome, thanks
<adeuring> good morning
<henninge> Hi adeuring!
<adeuring> moin henninge
<henninge> adeuring: I am almost done with the check list.
<adeuring> great
* adeuring changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: adeuring | https://code.launchpad.net/launchpad-project/+activereviews
<henninge> adeuring: pulled, merged, pushed
<adeuring> cool
<henninge> And it looks really cool, too.
<henninge> and works well for most items because the views I am linking to use "ReturnToReferrerMixin", so I mostly get back to the details page.
<henninge> only the last one doesn't
<henninge> adeuring: I have to run an errand, be back later.
<jam> lifeless: can you ec2land https://code.launchpad.net/~jameinel/launchpad/loggerhead-732481/+merge/52902
<jam> or maybe adeuring ^^
<adeuring> jam: yes, I'll land it.
<jam> adeuring: thanks.
<LPCIBot> Project windmill build #40: STILL FAILING in 1 hr 8 min: https://hudson.wedontsleep.org/job/windmill/40/
<lifeless> jam: something that would be nice: a patch to the loggerhead Makefile so that 'make check' runs the test suite with the loggerhead tree from .
<deryck> Morning, all.
<allenap> deryck: Good morning :)
<allenap> deryck: I have a branch that does some work on Javascript building. As an expert in this area, would you have time to review it?
<deryck> allenap, sure.
<allenap> deryck: https://code.launchpad.net/~allenap/launchpad/alt-javascript-build/+merge/53006
<allenap> Thank you :)
<deryck> np
<deryck> looking now....
<jtv> Anyone here really familiar with feature flags?  I just can't figure out what the proper way to handle an "enabled" setting is.  Should I set true/false, or on/off?  Should I evaluate its value as boolean, or compare to my chosen "on" string?  How do I clear the flag in a test, and how in production?
<jtv> The existing code seems divided on the subject.
<wgrant> jtv: I standardised it a few weeks back...
<wgrant> It should be pretty standard, except for memcache.
<wgrant> if getFeatureFlag('foo'):
<jtv> And then what are appropriate values for that flag?
<wgrant> It's not optimal ('' = False, everything else True), but it's simple and easy to use in TAL.
<wgrant> We agreed that it was the least evil solution for now.
<jtv> Can we set a feature flag to the empty string>
<jtv> ?
<wgrant> Yes.
<deryck> allenap, looks great, thanks for doing this!  r=me
<allenap> deryck: Woohoo :) Thank you.
<deryck> np :-)
<jtv> wgrant: oh well, it seems to satisfy my tests.  Thanks.
<jtv> And then I guess I can use any string whatsoever to indicate "on": yes, true, enable, on, or even off, disable, die, nothanks, samovar, â¦
<wgrant> Yes :(
<jtv> Is the data still entered as a tab-separated text field?  Because that gives extra juice to the whole "blank" thing.
<wgrant> The separator can be any whitespace.
<allenap> Is it me, or is Launchpad very slow right now?
<bigjools> allenap: I don't know, are you slow?
<allenap> bigjools: Fairly slow, yes.
<wgrant> It's slow, but not slower than usual AFAICT.
<allenap> Okay, thanks. It's not doing anything at all for me now.
<bigjools> might be a bad appserver
<allenap> bigjools: Sounds about right... it's okay again now.
<jtv> wgrant: the feature flags documentation still uses "off" as an example.
<wgrant> jtv: Where?
<wgrant> jtv: The docs on /+feature-info should be up to date.
<jtv> wgrant: https://dev.launchpad.net/FeatureFlags
<wgrant> jtv: Fixed.
<jtv> great, thanks
<jtv> And the "on|off" bit as well, I see.
<jtv> adeuring: got a review for you!  https://code.launchpad.net/~jtv/launchpad/bug-733132/+merge/53009
<adeuring> jtv: I'll look
<jtv> dankeschÃ¶n
<adeuring> jtv: the branch looks good. only one detail: FEATURE_FLAG as a variable name does not tell what the feature is about. Could you rename it to something like ENABLE_DERIVED_SERIES_JOBS ?
<jtv> adeuring: I'm in two minds about thatâ¦ doesn't the module it's in do enough to disambiguate it?
<jtv> I could import it into the test as distroseriesdifferencejob.FEATURE_FLAG
<adeuring> jtv: well, we could have two feature flags in this module
<jtv> That way the name is always disambiguated.
<jtv> Ah I see what you mean.
<adeuring> I prefer names which tell what is done over how things are done
<jtv> Okay, then how about ENABLING_FEATURE_FLAG?
<jtv> Because it is important to my mind that it is the feature flag, not e.g. a boolean that controls the feature.
<jtv> If the test refers to it as distroseriesdifferencejob.ENABLING_FEATURE_FLAG, I think that'd be about as clear as it could be.
<adeuring> jtv: what about FEATURE_FLAG_ENABLE_DERIVED_JOBS?
<jtv> Gets a bit long, and basically for the purpose of repeating the module name.
<adeuring> without say what is enabled, it looks to me still like a label on an electrial switch just saying "SWITCH" ;)
<jtv> Well that would make sense in the context of a little circuit board when you know what the board is for, just not what outside part should be attached to the particular piece of wire you're looking at.
<adeuring> jtv, ok, so... ENABLE_MODULE? or FLAG_ENALBE_MODULE?
<jtv> It would have to say "flag," but I don't see why it should say "module."
<adeuring> well, I still think that its good to indicate what is enabled
<jtv> Yes, but I'm relying on the "distroseriesdifferencejob" to imply that it's DistroSeriesDifferenceJob that's being enabled.
<jtv> I don't think it'll confuse anyone, especially since it's not even being exportedâit's just the test that refers to it from the outside.
<adeuring> jtv: ok, for the test distroseriesdifferencejob.WHAETEVER is more or less fine, but _inside_ the module something like FEATURE_FLAG_ENABLE_MODULE would still make more sense to me
<jtv> OK, I can't say I like it much but you have the advantage of a fresh outside look.  I'll use that.
<adeuring> jtv: thanks :)
<adeuring> so, r=me
<jtv> Thanks
<bigjools> gary_poster: another quick buildout question - if we have sourcecode deps in the dists tree but not mentioned in versions.cfg, are they used somewhere else or are they safe to remove?
<gary_poster> maybe used somewhere else bigjools.  sadly, sourcecode is a separate mechanism
<gary_poster> (and on call :-) )
<bigjools> gary_poster: darn :/  ok thanks
<allenap> adeuring: Hi Abel. Could you review a tiny branch for me please? https://code.launchpad.net/~allenap/launchpad/fix-librarian-startup-warnings/+merge/53018
<adeuring> allenap: sure, I'll look
<allenap> Thanks.
<adeuring> allenap: r=me
<allenap> adeuring: Cheers.
<wgrant> le
<wgrant> sinzui: Hi.
<sinzui> hi wgrant
<wgrant> sinzui: You upgraded lazr.restful to 0.17.4 a couple of days ago, right?
<sinzui> I did
<wgrant> 0.17.2 has a regression in exception handling.
<wgrant> See #launchpad.
<sinzui> was that fixed in 17.3 by leonardr
<wgrant> No.
<wgrant> It's still broken in 0.17.4 (which is on prod now)
<wgrant> OOPS-1896STAGING128
<sinzui> Well I am not going to fix that regression by restoring another regression
<sinzui> I will need to talk to leonardr
<wgrant> Indeed, that's what I was going to suggest.
<wgrant> I haven't filed a bug, since it is 1am.
<wgrant> Can you poke leonard about this when he appears?
<sinzui> My change was to revert the xhtml-in-json because it was being injected into pages without being escaped
<wgrant> Right.
<wgrant> It just pulled in the exception handling changes too.
<sinzui> 0.17.3 does something interesting with exception handling. It fixed some broken tests I saw in 0.17.2
<wgrant> Ahhh.
<wgrant> I think I see what's going on, maybe.
<sinzui> Reverting that change breaks the test suite :)
<wgrant> Hmm, no, I don't see it.
<sinzui> Which was also why I was not going to land my work...I hand never seen my tests run successfully.
<wgrant> But the 0.17.2 diff looks like a hack :/
<wgrant> It also looks like it should be using .response_code, not .status...
<LPCIBot> Project windmill build #41: STILL FAILING in 1 hr 8 min: https://hudson.wedontsleep.org/job/windmill/41/
<sinzui> wgrant: I recall "status" reappeared in response output when I added this version to Lp. I assumed it was because of my change, not leonardr's
<wgrant> Anyway, leonardr will save the world.
<sinzui> wgrant: are you certain 0.17.2 was live? the expose(exception, code) signature was very new. like late last week
<wgrant> sinzui: We rolled out your change (with 0.17.4) this morning, AFAIK.
<sinzui> I had to update my deps to resolve the sugnature change
<sinzui> right, and 0.17.3 was never released. But was 0.17.2 really the version in production. 0.17.1 looks like the version I have last week
<wgrant> Hmm? It broke in 0.17.2, and is still broken in 0.17.4
<wgrant> Timeouts OOPS like that staging OOPS I gave above.
<wgrant> 0.17.4 was introduced in r12565, and r12568 is now on production.
<abentley> deryck: bzr+ssh://bazaar.launchpad.net/~abentley/launchpad/js-translation
<wgrant> Anyway, night.
<abentley> deryck: http://pastebin.ubuntu.com/578842/
<deryck> abentley, ci = new CheckItem({identifier: 'id1'})
<jml> how do I search for bugs that don't have certain tags?
<james_w> jml, https://bugs.launchpad.net/launchpad/+bug/81575
<_mup_> Bug #81575: no way to search for absence of a tag <bugtag> <lp-bugs> <oem-services> <ubuntu-qa> <Launchpad itself:Fix Released by allenap> < https://launchpad.net/bugs/81575 >
<jml> james_w: ta
<jcsackett> is it bad that i'm excited a db patch i landed took under 10 minutes?
 * jcsackett figures it probably is.
<sinzui> Not bad
<sinzui> will your patch break everything?
 * jml trying out new battery.
<jcsackett> sinzui: no, my patch fixes messages.
<jcsackett> sinzui: it was the visibility db patch.
<jcsackett> sinzui: real question is how long it takes on the weekend super-rebuild.
<rvba> jcsackett: quick question: how did you know how long your patch took to be applied?
<rvba> sinzui: thanks for your review ... I think I misunderstood the word 'resubmit' and did put a red mark on my own proposal ;-).
<jcsackett> does staging have different timeout thresholds from production?
<sinzui> rvba: I'll follow up the review when the diff completes its update
<deryck> henninge, still around?
<henninge> deryck: yes, for the moment
<deryck> henninge, need just a quick ack that this looks sane for displaying "not translated" -- http://people.canonical.com/~deryck/in-upstream-msg.png
<deryck> henninge, in your opinion.  not asking for a formal ui review ;)
<henninge> that looks good, although we use "(not translated yet)" in other places  AFAIR
<deryck> henninge, ok, I'll match that.  thanks.
<bac> you still around adeuring?
* bac changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: adeuring, bac | https://code.launchpad.net/launchpad-project/+activereviews
<adeuring> bac: yes
<sinzui> rvba: r=me for code and tests
<bac> adeuring: sorry it took me so long to show up today
<adeuring> well, for an hour or so
<adeuring> bac:  no problem :)
<bac> adeuring: i'll take wallyworld's branch
<adeuring> bac: cool, thanks
<adeuring> bac; I must admit that I simply cducked out from looking at the loggerhead stuff -- I have no real clue about it...
<bac> adeuring: you're the best judge of what you can do a good job reviewing
<rvba> sinzui: nice. thx for the careful review. It's precious to me because I can find many layers of different ways to do the same things in the code.
<sinzui> rvba: yes. We are very bad to removing code/test styles that we decide were bad
<rvba> sinzui: I'll have to correct my own changes from last week.
<deryck> adeuring, you still open for reviews?
<adeuring> derycksure
<deryck> adeuring, https://code.launchpad.net/~deryck/launchpad/upstream-empty-display-707825/+merge/53050  thanks!
<deryck> adeuring, I was thinking of taking lunch now, too, but if you want me to hang around for interactive chatter, I can.
<adeuring> deryck: na, I don't want you to become unnecessarily hungry ;)
<deryck> heh, thanks!
<adeuring> deryck[lunch]: r=me
<LPCIBot> Project windmill build #42: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/42/
* adeuring changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer:  bac | https://code.launchpad.net/launchpad-project/+activereviews
<abentley> deryck: I've written a stub html file to use until henninge and abel's work is ready, but it says "YUI id not defined": http://pastebin.ubuntu.com/578968/
<abentley> s/id/is
 * deryck looks
<deryck> abentley, paths are wrong in the script links.  Did you copy from the test file?  If so, you're now up one "../" level from before.
<abentley> deryck: ah.  Yes, I did.  mutter.
<allenap> maxb, jam: Did you get reject messages from launchpad-bugs-owner or launchpad-reviews-owner?
<maxb> oh dear, I deleted them without paying attention. Let me oscillate the status of a MP to send some more
<allenap> Thanks.
<maxb> Received: from loganberry.canonical.com (localhost [127.0.0.1]) for <launchpad-bugs@lists.ubuntu.com>; Fri, 11 Mar 2011 20:19:54 +0000 (UTC)
<maxb> X-Launchpad-Message-Rationale: Reviewer @loggerhead-team
<maxb> allenap: ^
<allenap> maxb: Cool, thanks.
<sinzui> I worry for the 4 kittens in my house. These old bug views are evil. I sense the Monkey god will seek vengeance :(
<allenap> maxb: I've deactivated the membership of ~launchpad from ~loggerhead-reviewers. I'll reply to the thread on launchpad-dev too. I think it's happened because ~launchpad was automatically made a member of ~loggerhead-reviewers when I set it as the owner.
<maxb> allenap: The  X-Launchpad-Message-Rationale: Reviewer @loggerhead-team  makes me doubt this will fix the issue?
<allenap> maxb: Was that on a new mp? Perhaps the branch reviewer is copied at mp creation.
<maxb> oh, I see your point.
<lifeless> matsubara: hi
<matsubara> lifeless, hi
<lifeless> matsubara: how do I land https://code.launchpad.net/~lifeless/oops-tools/limits/+merge/52348 ?
<lifeless> matsubara: or were you going to check it passes tests etc? [it may now]
<matsubara> lifeless, I checked. all tests pass. to land it just set a commit message and set the MP to approved. tarmac will take care of running the test suite and merging on trunk
<matsubara> I'll deploy afterwards
<lifeless> doing now
<matsubara> cool. thanks
<lifeless> done
<matsubara> lifeless, looks like the postgresql instance in the tarmac chroot is not working
<lifeless> matsubara: why the change to vcs-imports/registry ?
<matsubara> lifeless, what change?
<lifeless> matsubara: you removed ~registry from ~vcs-imports
<matsubara> lifeless, did I? I deleted ~launchpad-chr but that shouldn't be related
<lifeless> forwarded to you
<matsubara> lifeless, don't know what happened there, but I suspect it was because I deleted ~launchpad-chr today
<lifeless> might be worth checking that maxb etc can still admin imports
<maxb> I believe I was/am a direct member of ~vcs-imports
<maxb> yes, I still seem to have the access
<lifeless> kk
<LPCIBot> Project windmill build #43: STILL FAILING in 1 hr 12 min: https://hudson.wedontsleep.org/job/windmill/43/
#launchpad-dev 2011-03-12
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project windmill build #44: FIXED in 52 min: https://hudson.wedontsleep.org/job/windmill/44/
<wgrant> Erm.
<wgrant> StevenK: It failed to detect the testrunner breakage :(
<StevenK> Whee, subunit bug
<lifeless> woo
<lifeless>  * 367 Time Outs
<wgrant> Er?
<wgrant> Oh, I read the wrong one this morning :(
<wgrant> I saw 1225 and was pretty disappointed.
<wgrant> We had less than 2000 hard OOPSes/timeouts. Not bad.
<wgrant> And about 400 exceptions are fixed by stuff in the deploy queue.
<lifeless> we're getting there
<StevenK> lifeless: Are you happy with the teams progress?
<lifeless> StevenK: I think we're making fantastic progress on maintenance
<lifeless> StevenK: recipes are looking awesome, but I think the transition off them has been a little fuzzy
<lifeless> StevenK: message shared/bug subscriptions seem to be coming along well
<lifeless> StevenK: and derived distros has only had one week now, so no opinion
<LPCIBot> Project windmill build #45: FAILURE in 1 hr 10 min: https://hudson.wedontsleep.org/job/windmill/45/
<StevenK> lifeless: the transition off recipes has been fuzzy? What do you mean?
 * StevenK wonders if he can fetch an SPR by ID using the API
<wgrant> StevenK: SPRs aren't exposed, so no.
<StevenK> :-(
<wgrant> You're wanting to check the changelogs?
<StevenK> Just one or two, yes
<wgrant> I ran it for a while on DF and it seemed sane there.
<wgrant> so I've already qa-ok'd it.
<StevenK> So on qastaging it looks to be doing around 500 in 15 minutes.
<StevenK> And we have what, 600,000 that are unset?
<wgrant> I don't recall the number, but that seems plausible.
<StevenK> Which means 1200 runs to finish
<wgrant> On overloaded asuka.
<StevenK> Yes, we should silence garbo-hourly on Tuesday
<StevenK> I was thinking when it starts running on production.
<StevenK> 1200/23 is 52, so, almost 2 months?
<StevenK> Am I doing that right?
<wgrant> Yes.
<wgrant> But !asuka should be faster.
<wgrant> At least a bit.
<wgrant> But we're still only running at 25% of what I had initially planned, so it's going to take a while.
<StevenK> I don't restrict the archive?
<wgrant> It is only running for 15 minutes out of every 60.
<StevenK> Ah well.
<StevenK> And garbo-hourly loses an hour to garbo-nightly doesn't?
<StevenK> Er, doesn't it
<wgrant> I'm not sure.
<StevenK> LFA is exposed, isn't it?
<wgrant> No.
<wgrant> Some files are exposed, but not LFA itself.
<wgrant> Why?
<StevenK> Can I trivially go from LFA ID to URL?
<wgrant> If you know the filename.
<StevenK> They're all "changelog"
<wgrant> http://staging.launchpadlibrarian.net/$ID/changelog
<wgrant> Er, qastaging, but yeah.
<StevenK> lifeless: When you care, "SELECT id, changelog from SourcePackageRelease WHERE id IN (63623, 63624, 63625);" on qastaging
<wgrant> StevenK: You could also guess.
<wgrant> Add an attachment to get an idea of where the IDs are, then search around there.
<StevenK> That doesn't sound nearly as much fun as bugging lifeless.
<wgrant> It is more fun when it leads to security holes :)
 * StevenK ponders when to start on the Poulet SautÃ© Au Vinaigre
<StevenK> It sounds better in French
<StevenK> "Chicken in Vinegar" doesn't sound very appealing.
<StevenK> wgrant: Current plan is to average six runs on production and work out length from there.
<wgrant> StevenK: You might also consider a graph.
<wgrant> I believe they're pretty easy to add.
<lifeless> StevenK:   id   | changelog
<lifeless> -------+-----------
<lifeless>  63623 |  64295224
<lifeless>  63624 |  64295225
<lifeless>  63625 |  64295226
<lifeless> StevenK: lfa isn't exposed because lfa is just a join
<lifeless> files don't have standalone existence, you need a containing object
<lifeless> like spr, bug, etc
<lifeless> it would be like exposing message
<wgrant> lifeless: Uh...
<wgrant> https://api.launchpad.net/+apidoc/devel.html#message
<lifeless> wgrant: bug
<StevenK> Right, those 3 look good
<StevenK> lifeless: And thank you
<lifeless> de nada
<lifeless> StevenK: do my answers on team progress vs happiness make sense?
<StevenK> [17:43] < StevenK> lifeless: the transition off recipes has been fuzzy? What do you mean?
<lifeless> its dragging on
<lifeless> at bit
<lifeless> s/at/a/
<StevenK> lifeless: I think that might be a little unfair, since it is the first transition since the move to squads.
<wgrant> I think it's inevitable.
<wgrant> For a feature like that.
<StevenK> I think transitions need to handled with an iron fist -- finish off anything you have in progress, then pick up criticals
<wgrant> That requires completion to be declared at a reasonable point.
<wgrant> And that's difficult.
<wgrant> I think recipes can be considered delivered, but not done.
<lifeless> StevenK: I didn't say anyone had done anything wrong
<lifeless> StevenK: I think its reasonable to assess how we do dispassionately-  haven't dug into causes yet
<jml> as far as I'm concerned, recipes aren't delivered as long as they have the beta flag up
<wgrant> Well, that's a matter of LOSA.
<StevenK> jml: Isn't that waiting for you to review it again?
<LPCIBot> Project windmill build #46: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/46/
<m4n1sh> so in Launchpad API's bug representation "owner_link" is used for "Reported by"
<m4n1sh> or is it represented by some other field. Cant find it
<lifeless> owner_link, yes
#launchpad-dev 2011-03-13
<lifeless> wgrant: why does deleting a ppa supersede its publications ?
<huwshimi> The monday morning channel silence...
<huwshimi> Which makes me fear that:
<huwshimi> a) I'm not connected, or
<huwshimi> b) it's not Monday
<lifeless> its a public holiday in vic/sa/tas
<huwshimi> lifeless: Ah of course
<lifeless> thumper and I are here
<jelmer> huwshimi: also, it's not Monday :-P
<lifeless> jelmer: yes it is :)
<thumper> lifeless: what about queensland?
<lifeless> thumper: not a holiday
<lifeless> thumper: AFAIK
<lifeless> oh, and act
 * mwhudson waves
 * thumper waves at mwhudson
<wgrant> lifeless: Possibly to make it easier to expire, although it fails to let them expire anyway.
<wgrant> lifeless: If you want to make it fast, just grab all the binary and source pubs separately and delete them.
<wgrant> Rather than letting the sources find and delete their binaries.
<lifeless> wgrant: I'm wondering why we bother
<wgrant> Or make it batch-call getBuiltBinaries, since that's reasonably efficient when not called hundreds of times.
<lifeless> wgrant: isn't just flagging it deleted sufficient ?
<wgrant> lifeless: That won't let the files expire.
<wgrant> But as it stands they won't expire anyway.
<wgrant> Due to another bug.
<lifeless> wgrant: so the point is to free uplibrarian space
<lifeless> ?
<wgrant> I believe so.
<lifeless> wgrant: anyhow, the reason I ask is we have a timeout with 9 second spent in a single update query marking the rows expired
<wgrant> Hmm, we must be down about 5 appservers by now.
<lifeless> 435 rows
<wgrant> lifeless: Yeah, I saw that.
<lifeless> wgrant: you think more are dying again ?
<wgrant> lifeless: Appservers hang... it's a normal expected thing :/
<lifeless> wgrant: not entirely
<lifeless> wgrant: we had a bunch exit
<wgrant> lpnet1/2/6/10 are dead at the moment.
<lifeless> wgrant: what tells you that ?
<wgrant> lifeless: They had connections killed over the weekend.
<lifeless> wgrant: elmo started them up again around 5am your time
<lifeless> wgrant: see #is backscroll
<wgrant> Ah, handy.
<lifeless> wgrant: you're on leave today?
<wgrant> lifeless: Indeed.
<lifeless> kk
<lifeless> can I interest you in qa of Revision 12581 can not be deployed: needstesting
<wgrant> Is that poppy?
<lifeless> StevenK: 12580 needs qa too
<lifeless> wgrant: yes
<lifeless> we get those fixed and us losa can deploy a fix for search performance
<wgrant> I already mostly QA'd that before it landed for just this reason, so I'll quickly retest now.
<lifeless> looks like we're still doing memcache on bugtask:+index
<lifeless> :(
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1898L1962#statementlog is full of get/sets
<wgrant> lifeless: NullBugTask:+indec
<wgrant> s/c/x/
<wgrant> Not BugTask:+index.
<lifeless> argh
<lifeless> righto
<wgrant> lifeless: You can qa-ok the DSD thing if you want. The altered method is not called yet.
#launchpad-dev 2012-03-05
<wgrant> lifeless: Foreign keys on trigger-maintained denorm tables -- any opinions?
<wgrant> Seems like they're basically just going to slow things down.
<StevenK> wgrant: https://devpad.canonical.com/~stevenk/query.log.gz
<wgrant> waitwhat
<wgrant> accesspolicy_pkey should be a serial
<wgrant> It can't conflict, unless you're not dirtying the DB properly.
<StevenK> I don't get it either. :-/
<lifeless> wgrant: stub wants them
<lifeless> wgrant: I'm not strongly enough opinionated to debate either way yet.
<lifeless> wgrant: intel SSD+1
<wgrant> lifeless: Well, we can't do them for array columns...
<StevenK> lifeless: http://pastebin.ubuntu.com/869213/
<wgrant> The enterprise ID probably can't include the instance like that.
<wgrant> Since they're clones of production.
<wgrant> But that's rather awkward.
<lifeless> StevenK: might want a qualifier added - e.g. 'object_to_enterpriseid(bug, 'comments')
<StevenK> wgrant: I was following lifeless' mail on the subject
<StevenK> lifeless: Why?
<lifeless> StevenK: I sketched out some thoughts on the txlongpoll discussion last weekend or so
<lifeless> StevenK: can always add it later
<lifeless> StevenK: it looks fine to me in all other regards
<StevenK> I don't see the point. We're converting an object to a representation of one, *not* a method/property on an object.
<StevenK> lifeless: I have hit a roadblock for enterpriseid_to_object(), in that I can't seem to get it return the object
<lifeless> StevenK: there is a relatively low cost of change here if we're careful, so I'm fine leaving it out.
<lifeless> StevenK: it wasn't in my initial emails after all :)
<StevenK> IE, going from 'Person' to Person()
<lifeless> StevenK: sys.modules['Person'] isn't what you want
<lifeless> StevenK: you want sys.modules['lp.registry.model.person'].Person
<StevenK> Right, which is fine for Person, but not PackageUpload
<lifeless> StevenK: I think you'll probably need something a bit more directly connected to the LP code layout than what you have
<lifeless> e.g. it may not be possible to JustSupport everything.
<lifeless> also remember that type needs to be unique
<lifeless> and your current implementation would be broken if e.g. lp.services.email grew a Person class
<lifeless> so, I suggest having an explicit map of supported types to strings, and strings to factories
<lifeless> e.g. known_types = [
<lifeless>     (Person, 'Person'),
<lifeless>     ]
<lifeless> types = dict(known_types)
<lifeless> factories = dict((label, klass) for klass, label in known_types)
<lifeless> # adjust for taste
<StevenK> wgrant: I've pushed my branch to lp:~stevenk/launchpad/accesspolicy-garbo if you can peer at it.
<wgrant> StevenK: Looking
<wgrant> Ah
<wgrant> There's the problem.
<wgrant> test_hourly_script doesn't dirty its DB
<wgrant> test_daily_script does
<wgrant> StevenK: Try fixing that.
<StevenK> AH HA
<StevenK> About time :-(
<lifeless> wgrant: 'psycopg2.ProgrammingError: syntax error at or near "."
<lifeless> LINE 3: ALTER TABLE BugJob RENAME TO todrop.BugJob;
<lifeless> '
<lifeless> wgrant: I think your advice is wrong :)
<StevenK> Haha
<wgrant> lifeless: That wasn't my command.
<wgrant> ALTER SCHEMA todrop
<StevenK> SET SCHEMA todrop
<lifeless> wgrant: you cleverly preserved it :>
<wgrant> Er, that
<wgrant> lifeless: I didn't really read the first sentence.
<wgrant> It was there already :)
<StevenK> wallyworld: Are you up for reviewing a branch of mine?
<wallyworld> ok
<StevenK> wallyworld: https://code.launchpad.net/~stevenk/launchpad/accesspolicy-garbo/+merge/95834
<wallyworld> StevenK: i think you mean embardgoessecurity, not proprietary
<StevenK> wgrant: ^
<wgrant> The wallyworld is correct.
<wallyworld> proprietary is only for products with commercial subscriptions
<wallyworld> StevenK: i'd like the tests to check the data as well as just the count
<wallyworld> StevenK: because i'm stupid, why did you need to make a call to DatabaseLayer.force_dirty_database()  ?
<StevenK> wallyworld: Because test_hourly_script is broken
<wallyworld> of course :-)
<StevenK> test_daily_script did it, but test_hourly_script did not.
<wallyworld> and existing tests didn;t need this?
<StevenK> And it was causing test failures in unrealted tests
<wallyworld> ok
<wallyworld> thanks for explaining
<wgrant> It depended on ordering.
<wgrant> Depending on what the next test did, it would have worked even with the dirty DB
<wgrant> eg. if it was just deleting stuff
<wallyworld> yuk
<wgrant> Or using a non-serial primary key.
<StevenK> wallyworld: I don't see the point of testing the APs themselves in the test.
<wgrant> I would.
<wgrant> And it's like 2 lines.
<wallyworld> StevenK: it's not testing the api, but that you have told the garbo scrip tto insert the correct data
<wallyworld> ah, can't read, i thought you said api
<wallyworld> but my point stands :-)
<StevenK> wallyworld: Anything else?
<wallyworld> StevenK: no, looks nice otherwise
<lifeless> wgrant: could I ask a favour? ec2 land https://code.launchpad.net/~lifeless/launchpad/bugjob/+merge/95837 ?
<wgrant> lifeless: Self-reviewing a DB patch? Dubious.
<wgrant> But OK :)
 * lifeless shrugs
<lifeless> I don't require stub to block on me for his schema changes.
<StevenK> wallyworld: http://pastebin.ubuntu.com/869352/
<wallyworld> StevenK: i'd use assertContentEqual to ignore sorting differences and you don't need the .count() check if you are conparing the actual values
<StevenK> wallyworld: Good point.
<StevenK> wallyworld: http://pastebin.ubuntu.com/869357/
<wallyworld> StevenK: i had a thought. will the logic fail if we add a policy to a pillar by hand? i think it will
<wallyworld> it will think the pillar is already processed
<wallyworld> but there may not be the 2 required policy types
<StevenK> Not for the test data, but I can't see us doing so until the garbo jobs are done
<StevenK> Distribution I can see being done during the first run
<wallyworld> yes :-)
<StevenK> Product will take a little longer
<wallyworld> yes, i'm concerned though since the fflag will be on for select users
<wallyworld> and they can use the gui
<wallyworld> perhaps we delay the fflag being on till garbo kob done
<StevenK> Right
<lifeless> .
<StevenK> lifeless: Your DSL sucks.
<StevenK> wallyworld: I've commited and pushed those changes.
 * wallyworld taps fingers
<StevenK> wallyworld: Diff is updated
<wallyworld> yes
<StevenK> I was just pondering filing a bug or [no-qa]
<wallyworld> StevenK: r=me. to not cut corners, you should probably do qa on it
<StevenK> wallyworld, wgrant: QA-Landing looks awesomely full -- is it the truth?
<wallyworld> StevenK: yes, i have a 4 pipe landing
<wallyworld> but 2 yui tests are failing on ec2
<wallyworld> so am trying to fix
<StevenK> Ah yes, QA-Landing really has been wallyworlded.
<lifeless> mmm
 * lifeless forgets when implicit db user switches occur
<lifeless> wgrant: sanity check me - we can drop the calculate-bug-heat db user and permissions
<StevenK> Why does https://launchpad.net/oops-tools say "This project is currently inactive." ?
<lifeless> because it is
<lifeless> you want python-oops-tools
<huwshimi> wgrant: On the +sharing page when you are selecting a team to share with, why are there options to grant access to public and public security... am I missing something? It seems like you don't need to grant access for those things.
<huwshimi> wallyworld: ^
<lifeless> huwshimi: for private projects
<lifeless> huwshimi: or projects with private-by-default artifacts
<huwshimi> lifeless: I feel like I'm missing something. So "Public" for a private project is private?
<huwshimi> I thought this was the whole point of the policies
<huwshimi> to have this stuff make sense :)
<lifeless> oh I see, you're saying that any artifact covered by the public policy really should be public
<lifeless> I'd need to page a bit more in to answer that; so I'll let wallyworld / wgrant / StevenK do so ;)
<huwshimi> :)
 * StevenK is neck deep in Django
 * StevenK glares at lifeless 
<huwshimi> oop, gotta go
<nigelb> StevenK: should we send someone after you? :)
<czajkowski> aloha
<adeuring> good  morning
<StevenK> Errr, why did lifeless land a DB patch to devel?
<czajkowski> StevenK: do you actually sleep ?
<stub> Whoopsies
<StevenK> czajkowski: Sometimes.
<micahg> is it even sleep time in StevenK's TZ?
<czajkowski> jtv: any reason why on the RT we get mails *to* rosetta@launhcpad.net  on a daily basis?
<jtv> czajkowski: we stopped using that address a while back (it generated more pain than gain), so I'd guess there's a "legacy" forwarding to RT.
<czajkowski> jtv: aye on a daily basis at least 3-4 rts are created
<jtv> So you may want to figure out why the email gets there.  For instance, users may be replying to outgoing mail that accidentally still uses rosetta@.
<czajkowski> jtv: MAILER-DAEMON" <noreply@launchpad.net>
<wgrant> lifeless: Yes, they can go.
<wgrant> lifeless: AFAIK I removed them from the codebase a while back.
<wgrant> lifeless: But they need to be manually dropped from prod.
<wgrant> (the bug heat users)
<salgado> mrevell, hey there.  sorry to keep nagging, but I was wondering if you'll have some time to work with us on that work-items help text today?  we would like to land it together with the new UI, if possible :)
<czajkowski> salgado: morning
<salgado> hi czajkowski!
<mrevell> salgado, Email incoming! :)
<salgado> mrevell, perfect timing, thanks a bunch!
<mrevell> My pleasure :) Let me know if you need to talk about it. I'll be afk for 40 mins or so.
<salgado> mrevell, there was a typo and the list of states was incomplete. I guess you won't mind me fixing that :)
<salgado> mrevell, and https://help.launchpad.net/WorkItems doesn't exist yet
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugtasks: 4*10
<salgado> benji, I've put a trivial one up for review (https://code.launchpad.net/~salgado/launchpad/workitems-widget-help-popup/+merge/95894), if you have a couple minutes :)
<benji> salgado: sure
<salgado> benji, fwiw, mrevell gave me the text you see there and I've already told him the wiki page linked to from there doesn't exist yet. I suppose he plans to steal the one from wiki.u.c
<benji> salgado: ok
<mrevell> salgado, benji: Hey, back from lunch. I'll create the wiki page this afternoon.
<mrevell> salgado, Thanks for making the corrections :)
<benji> salgado: review done
<salgado> thanks benji. do you know how I'd go about using CSS to format the <pre> block there?
<benji> salgado: only with inline styles, but I suspect there's a CSS class already in existance for that
<salgado> benji, ah, that'd be nice. I'll see if I can find it
<salgado> benji, can't seem to find anything that would do it. is it ok if I just inline a margin-top: there?
<benji> salgado: you'd better ask someone more font-endy than me, we generally frown on inline style, but I don't know where the line is drawn, precisely
<salgado> benji, ok, will do. anyone you'd recommend?
<salgado> (don't really know who to ask for that these days)
<salgado> deryck, maybe? ;)
<deryck> salgado, hi. what's up?
<benji> salgado: Huw (huwshimi) would be a good start, but mrevell probably knows too
<salgado> deryck, hi there.  we're discussing how to avoid a blank line in a <pre> block: https://code.launchpad.net/~salgado/launchpad/workitems-widget-help-popup/+merge/95894
<mrevell> salgado, Yeah, it's best if you ask for a review from huwshimi.
<salgado> and benji asked me to check with somebody more frontend-y than him, right when you joined
<mrevell> salgado, He'll help with the CSS side of things.
<salgado> deryck, looks like you don't need to worry, though; I'll bug huw :)
<deryck> salgado, ah, ok. cool :)
<salgado> mrevell, cool, will do that
<mrevell> salgado, Huw's in Tasmania, so he won't be around for a few hours yet.
<deryck> adeuring, abentley -- I'm coming, just taking a bit for the hangout to start for me.
<mabac> rick_h_, could you have another look at the review since I fixed the info message for the whiteboard? I used your javascript
<deryck> mabac, rick_h_ is unavailable until wed. maybe jcsackett could take it, since he's the mentor of rick_h_
 * jcsackett perks up, goes to take a look
<mabac> deryck, ok thanks
<mabac> jcsackett, great thanks!
<jcsackett> mabac: oh wow, this branch has had a lot happen since i last looked at it.
<mabac> jcsackett, oh I hoped it wasn't too bad. it should just be changing the order of the text fields and then the js for displaying an info message when editing the whiteboard.
<jcsackett> mabac: no, it's not bad. just a case of me sorting out all the activity so i know what i'm looking at. :-P
<mabac> jcsackett, hehe ok. note that two revisions where backed out entirely
<mabac> jcsackett, 14855 and 14856 are reverted in 14857
<sinzui> benji, do you have time to review my branches listed on https://code.launchpad.net/launchpad/+activereviews
<benji> sinzui: sure
<czajkowski> sinzui: morning I think this one is a you question, as I remember I can't change a projects licience. https://answers.launchpad.net/launchpad/+question/189488
<sinzui> czajkowski, we do not own the project.
<sinzui> czajkowski, reactivate the project, then explain to the user that he can use the Change details link to update the license
<czajkowski> sinzui: thank you.
<rick_h_> jcsackett: I'm in/out (snack time for the boy) let me know if you need anything from me. Sorry it ran over until I was out
<mrevell> salgado, https://help.launchpad.net/WorkItems now redirects to a draft page that I'll flesh out and make live when we're ready.
<salgado> mrevell, cool!
<czajkowski> deryck: allenap what is the max apport attachment that can be uploaded to a bug ?
<deryck> czajkowski, hmmm, not sure actually.  I can poke at code to see if allenap doesn't know off the top of his head.
<deryck> rick_h_, geez, man, we're going to have to start calling you lifeless2
<czajkowski> deryck: asking as someone reported a bug at the weekend, but just wondering was things crashing as lotta folks submitting things due to UGJ
<czajkowski> deryck: https://bugs.launchpad.net/launchpad/+bug/945629
<_mup_> Bug #945629: Launchpad chokes on tiny package, deeper throat required? <Launchpad itself:New> < https://launchpad.net/bugs/945629 >
<deryck> czajkowski, ah, this is a known bug.
<deryck> czajkowski, it's not attachment limits, am looking for the bug.
<abentley> adeuring: are you up for a chat?
<benji> sinzui: I just finished your two reviews.
<allenap> czajkowski: Comments (or at least bug descriptions) have a limit of 50,000 characters. I don't know of a limit for attachments, so... what deryck said :)
<deryck> czajkowski, this is bug 194558.  I'll dupe against it.
<_mup_> Bug #194558: Project file upload timeout (and often do not OOPS) <arm> <escalated> <linaro> <Launchpad itself:Triaged> < https://launchpad.net/bugs/194558 >
<czajkowski> allenap: deryck thank you
<sinzui> thank you benji
<benji> my pleasure
<deryck> allenap, yeah, I guess we have no limits for how many attachments.
<adeuring> abentley: sure. mumbleÃ
<adeuring> =
<abentley> adeuring: sure.
<rick_h_> deryck: heh, love my job and all that right? Can't be too far from the laptop.
<deryck> rick_h_, heh, fair enough. Just don't love it so much that you hate it in 6 months from burnout. ;)
<benji> sinzui: Gary points out that my code in the reviews isn't syntacticly correct, but I bet you'll get the gist
<sinzui> benji, thank. Your point is correct and I am playing with it now
<abentley> adeuring: celery does indeed implement soft timeouts via a signal handler: celery/concurrency/processes/pool.py:188
<adeuring> abentley: cool.
<abentley> adeuring: However, I was wrong about soft timeouts being configurable from the apply_async invocation.  They are only configurable on a task-type basis.  We may need RunFastJob, RunSlowJob, RunReallySlowJob tasks.
<adeuring> abentley: ok, sounds good
<sinzui> jcsackett, do you have a few minutes to discuss a bug I am working on http://pastebin.ubuntu.com/870095/
<jcsackett> sinzui: sure, i'm free.
<sinzui> jcsackett, fab. mumble or hangout
<jcsackett> sinzui: i can do either. hangouts worked well last time.
<sinzui> okay hangout.
<sinzui> jcsackett, https://plus.google.com/hangouts/c264c08394aabf490ee4492d4bc30adeb7741e97?authuser=0&hl=en#
<adeuring> abentley: lp:~adeuring/launchpad/lazr.jobrunner-oops
<abentley> adeuring: thanks!
<adeuring> abentley: I think we should add a class BaseJob to runjob.py / jobrunner.py, just to document which methods and properties are required
<abentley> adeuring: That makes sense.  We could extract it from FakeJob
<adeuring> abentley: yes, that's what I basically meant ;)
<abentley> adeuring: I've put some changes in trunk, so it's a good idea to merge trunk.
<adeuring> abentley: right
<abentley> adeuring: It looks interesting-- so we just build up an oops report and publish.
<adeuring> abentley: basically, yes. We still need to properly configure the OOPS reporting -- but that should not be done in lazr.jobrunner
<adeuring> (where "configure" means to pull in things like including a timeline, for exmaple)
<abentley> adeuring: On the launchpad side, can we re-use the existing oops configuration?
<adeuring> abentley: I think we need to refactor the ErrorReportingUtility
<abentley> adeuring: Oh, I see.
<adeuring> abentley: but generally my dea is that we can use the same, or at least a similar, oops_config in ErrorReportingUtility and in lazr.jobrunner
<abentley> adeuring: Sounds good.
<sinzui> benji, do you have time to review https://code.launchpad.net/~sinzui/launchpad/commercial-project-picker/+merge/95970
<matsubara> hi there, I'm trying to run the workitems branch but can't run it locally. I keep getting this error:  https://pastebin.canonical.com/61627/ when I make schema. Anyone have ideas how to fix that?
<salgado> matsubara, if nobody has any ideas you might want to run ./utilities/launchpad-database-setup again
<matsubara> salgado, I did that already
<salgado> hmm
<salgado> are you running with postgresql8.4 or 9.1?
<matsubara> salgado, I think I have both installed. this last time I ran lp-db-deps it ran against the 9.1 db
<matsubara> do you have a debversion in your tree somewhere?
<salgado> lib/lp/archivepublisher/debversion.py
<matsubara> right, same here
<matsubara> psql:launchpad-2209-00-0.sql:1230: ERROR:  could not access file "$libdir/debversion": No such file or directory
<matsubara> this is the error I'm getting
<salgado> but that's not it
<matsubara> what's the libdir variable?
<salgado>  /usr/lib/postgresql/8.4/lib/debversion.so
<salgado> that's the one
<salgado> you seem to lack
<salgado> I'd say get rid of 9.1 and try with 8.4. that works for me, on oneiric
<salgado> there's a postgresql-8.4-debversion package, btw
<salgado> but launchpad-database-dependencies depends on it
<matsubara> salgado, right, for 8.4 I had the debversion file, but not for 9.1
<matsubara> salgado, I stopped 9.1 and left only 8.4 running and am trying again
<matsubara> (I'm precise btw)
<matsubara> (I'm on precise btw)
<salgado> good luck, then :)
<matsubara> :-)
<salgado> I tried to set it up on precise about a month ago and gave up; ended up setting up an Oneiric container
<matsubara> salgado, it's applying the db patches now, so I think it's fixed. thanks salgado
<sinzui> salgado, I believe 9.1 can be used. I think we are switching to it soon
<matsubara>   File "/home/matsubara/devel/canonical/lp-sourcedeps/eggs/txlongpollfixture-0.1.3-py2.7.egg/txlongpollfixture/server.py", line 131, in _start
<matsubara>     raise Exception("Timeout waiting for txlongpoll to start.")
<matsubara> Exception: Timeout waiting for txlongpoll to start.
<matsubara> now I got that error when I make run
<matsubara> and it looks like it's running: 25320 pts/9    Sl     0:02 txlongpoll: accepting connections on 22435
<matsubara> ok, trying again worked.
<abentley> sinzui: to support the move of ec2 scripts from lp:launchpad to lp-dev-tools, I am thinking of copying the lastest bzr-pqm packages from the bzr daily PPA to the launchpad PPA.  Does that make sense?
<sinzui> abentley, could be. I run my own hacked version though since pqm-submit does not work on precise
<lifeless> abentley: that would be easier for folk w/out the bzr ppa
<sinzui> It would be nice to get the fix when someone actually applies my patch
<sinzui> Or discovered why I needed to write such am atrocious patch to make pqm-submit worl
<abentley> sinzui: Where is your patch?
<sinzui> https://bugs.launchpad.net/bzr-pqm/+bug/922741
<abentley> lifeless: Yes, that was my thinking.  I assume not everyone is on the dailies.
<sinzui> abentley, ^
<_mup_> Bug #922741: AttributeError: 'BzrBranch7' object has no attribute 'get_config_stack' <amd64> <apport-bug> <patch> <precise> <running-unity> <Bazaar PQM Plugin:New> < https://launchpad.net/bugs/922741 >
<abentley> sinzui: Just going by the title, that may have been fixed in r86
<sinzui> well. I think I can test that in a few minutes. I am about submit a branch
<abentley> sinzui: which bzr are you running?
<poolie> o/
<abentley> poolie: \o
<sinzui> abentley, 1.4.0~bzr83-1
 * sinzui looks at his own branch
<abentley> sinzui: which bzr, not which bzr-pqm?
<sinzui> oh duh
<abentley> though from the traceback, I'd guess 2.5
<sinzui> 2.5.0-1ubuntu1
<abentley> sinzui: This is wacky, because in 2.5, BzrBranch7 does provide get_config_stack.
<sinzui> abentley, wacky indeed
<sinzui> I really don't think I should have needed that patch
<sinzui> abentley, I pulled tip (bzr-pqm) and submitted.
<sinzui> Look like it did its part correctly
<abentley> sinzui: I'm glad of that.
<sinzui> abentley, and It was a success.
<sinzui> I think I can mark my patch invalid now
<abentley> sinzui: I don't really get what happened there, though I'll mention ec2 is using the egg version of bzr, not the system version.
<sinzui> abentley, given the oddness of my patch (wallyworld ran it too), I think there were version inconsistencies for about 6 weeks
<abentley> sinzui: It appears that the egg version of bzr did not support get_config_stack-- it landed in r6157.
<abentley> sinzui: So we could chalk this up to launchpad running a beta bzr.
<sinzui> fab
<sinzui> I will tell my team
<abentley> sinzui: I don't understand how a direct pqm-submit would be broken, unless you ran "bin/bzr".
<abentley> sinzui: Or were also running a similar beta version.
<sinzui> abentley, ec2 <land|test> and pqm-submit were all broken for me at the time I reported the bug
<abentley> sinzui: Are you certain ec2 test was broken?  It shouldn't be using pqm-submit at all.
<sinzui> abentley, yes it was. I had to use a different computer for a week
<sinzui> until I decided to add the hack
<abentley> sinzui: Okay.  I guess you were using "ec2 test -s" or similar?
<sinzui> yes, that is right
<abentley> sinzui: forgive my confusion about ec2 test.  I don't use it, so I didn't realize it can land branches.
<salgado> huwshimi, hi there. I've proposed a trivial branch (https://code.launchpad.net/~salgado/launchpad/workitems-widget-help-popup/+merge/95894) and my reviewer had some CSS-related questions so I was pointed at you :)
<salgado> it'd be great if you could have a look and tell me how I should proceed there :)
<huwshimi> salgado: Sure! Let me take a look
<huwshimi> salgado: Would you like me to comment here or on the mp?
<salgado> huwshimi, either way is fine with me
<huwshimi> salgado: I just replied on the mp
<salgado> oh, of course!
<salgado> thanks huwshimi!
<huwshimi> salgado: Hopefully that will give you proper spacing, but if not we can look at some CSS
<salgado> yep, confirming that now
<mwhudson> salgado: you should steal my greasemonkey work items editor at some point :)
<salgado> mwhudson, we considered that but in the name of simplicity we decided to use a textarea for now
<mwhudson> fair enough
<mwhudson> it will be trivial to adapt the greasemonkey to that i assume
<mwhudson> although the parsing/unparsing/parsing thing will be a bit strange
<mwhudson> salgado: you are exposing the work items on the api i assume?
<salgado> mwhudson, yes, for now only as a text field, using the same format as before. so you just have to change .whiteboard to .workitems_text, hopefully
<mwhudson> salgado: oh, the api representation is as text as well?
<salgado> mwhudson, yep
<mwhudson> ok
<mwhudson> salgado: how does this help status.l.o then?  you just know that it's formatted correctly?
<salgado> mwhudson, yes, and we catch/fix errors when users enter them so they won't have to be nagged by status to go back and fix it later.  our actual goal is not really to help status though -- we want to implement work item reports in LP: https://dev.launchpad.net/Projects/WorkItems
<mwhudson> ah ok
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<StevenK> lifeless: So why did your DB patch branch hit devel?
<lifeless> hahhaa bad defaults.
<lifeless> fortunately it will do no harm
<StevenK> What? We had db-devel as the default for YEARS
<StevenK> And it caused no end of issues
<sinzui> StevenK, regardless of future sharing and past legacy rules, user should be permitted to see public bugs. https://bugs.launchpad.net/launchpad/+bug/872870
<_mup_> Bug #872870: Public bug cannot be viewed when linked to a private branch <403> <branches> <bugs> <disclosure> <privacy> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/872870 >
<sinzui> wallyworld_, your can pull https://code.launchpad.net/html5-browser I tested the env using
<sinzui> HTML5BROWSER_USE_PYGTK='true' ./test.py
<sinzui> wallyworld_, see https://code.launchpad.net/~sinzui/+recipe/html5-browser-daily
<bigjools> morning
 * StevenK 's world crashes down
<sinzui> wallyworld_, have you run this yet
<sinzui> apt-add-repository ppa:sinzui/ppa
<wallyworld_> sinzui: sure have
<wallyworld_> just waiting for build now :-)
<lifeless> wgrant: do youi know of a way to say 'fuk it everything will need [SELECT|UPDATE|DELETE] on table Y' ?
<wgrant> lifeless: public
<wgrant> lifeless: But please don't.
<wgrant> THe only things that really belong there are...
<wgrant> hmm
<wgrant> SELECT on featureflag
<wgrant> And nothing else.
<wgrant> Ah, that's there already.
<wgrant> And pillarname
<wallyworld_> huwshimi: hi, sorry i missed you yesterday - was buying food for dinner. with your question, the +sharing page is wip. those options are being fixed and other stuff also being sorted out
<huwshimi> wallyworld: No problems
<huwshimi> wallyworld: So will the public options still be there?
<wallyworld> huwshimi: nope. only embargoes security and userdata, and for commercial projects, proprietrary
<huwshimi> wallyworld: OK great, thanks for that!
<wallyworld> huwshimi:  np. i have a branch that i'm about to land with those changes, plus there are now checkboxes so you can select more than one
<huwshimi> wallyworld: Ah great
<wgrant> wallyworld: The new packages should be published.
<wallyworld> wgrant: thanks, that was fast
<lifeless> wgrant: mail sending
<wgrant> wallyworld: Queue-jumping is handy :)
<wallyworld> indeed
<wgrant> lifeless: I await it in fear.
<lifeless>  /win goto mbarnett
<lifeless> bah
<wgrant> That makes me even more concerned.
<mbarnett> lifeless: that was more like a summon!
<lifeless> mbarnett: My fingers apologise to you.
<wgrant> lifeless: I see no mail.
<wgrant> Did it go via forster?
<wgrant> lifeless: you suck
<wgrant> 2012-03-05 23:50:25 WARNING Bad object name 'public.bugjob'
<lifeless> wgrant: you suck too. But enough of the compliments.
<lifeless> wgrant: you're talking about bugjob landing on the wrong branch? Has it caused a buildbot failure ?
<wgrant> lifeless: No, but you forgot to remove DB permissions.
<wgrant> So security.py whinges.
<wgrant> The mislanding won't cause any problems.
<lifeless> yeah
<wgrant> It just makes you the first violator of your policy :)
<StevenK> Haha
<lifeless> wgrant: does anything need calculate-bug-heat, talking of security.cfg ?
<wgrant> lifeless: No.
<wgrant> I apparently forgot to remove it from security.cfg
<wgrant> In my cleanup a few months back.
<lifeless> (and from prod)
<wgrant> And then it needs to be manually dropped from prod
<wgrant> yeah
<wgrant> And cleaned up from pgpass etc
<lifeless> I will leave that to you, enough WIP as it is
<wgrant> I may demolish it along with bugjob permissions in a few seconds.
<wgrant> lifeless: Oh, by "mail sending" you meant you were talking about permissions for sending mail, rather than that you were sending a mail about the permissions?
<lifeless> right
<wgrant> That explains why I see no email.
<lifeless> and notifications more generally.
<wgrant> I shall stop looking for the problem with forster.
<lifeless> wgrant: hah! indeed.
<bigjools> poolie: sat here chuckling at your tweet
<wgrant> Is that a replacement for "lol"?
<bigjools> lolling is not chuckling
<bigjools> lol is for when people want you to think they liked something your said but are not really laughing at all
<bigjools> you* said, even
<wgrant> True, true.
#launchpad-dev 2012-03-06
<jelmer> argh, still a surprising amount of people registering imports of http://code.launchpad.net/ URLs
<bigjools> damn, I need to procrastinate but I can't be bothered
<wgrant> jelmer: Yeah, I'm not quite sure how we have so many terrible users :(
<wgrant> bigjools: Enjoying the slow Internet access, at least? :)
<StevenK> Haha
<bigjools> wgrant: it's actually quite quick
<bigjools> better b/w than I had back at my house in the UK
<wgrant> Bandwidth-wise, sure :)
<bigjools> latency not so good but I'm not that fussed
<bigjools> LP is noticably slower to start loading pages
<StevenK> Yes
<jelmer> wgrant: I think our UI (at least this bit of it) is worse than our users :)
<lifeless> wgrant: I think we should blame our users, en masse, last, not first, when there is a repeated issue.
<lifeless> its much more likely that we are confusing them, than that they all arrive pre-confused.
<wgrant> lifeless: Mmm, seeing some of the questions that come through...
<wgrant> We have some preeeeetty moronic users.
<lifeless> I have no doubt that there are some which get excruciatingly confused; so far down the learning curve its hard to show them that they have stuff to learn
<lifeless> but still
<wgrant> Now, the UI around all this abominable.
<StevenK> Note how many recipes are against lp:launchpad/stable
<wgrant> But some of the mistakes are so incomprehensible that it can only be explained by users being braindead.
<lifeless> wgrant: I think you underestimate the confusion causable by poor UI
<wgrant> Does anyone know what one does about whoopsie crash reports?
<wgrant> It does the WER thing.
<lifeless> chat to ev
<wgrant> Of just submitting and then disappearing.
<lifeless> its the new shiny
<wgrant> So I can't track the progress of the X crash that I see hourly.
<wgrant> https://lpstats.canonical.com/graphs/WildcherryCPU/20111106/20120307/ is pretty
<wgrant> In 3 months wildcherry should have no load at all :)
<lifeless> hah
<lifeless> its dropped 15%
<lifeless> which is substantial
<wgrant> Another 7% will go once slony dies.
<wgrant> Now, with a few fixes to the store selection code, we can sensibly have downtime on wildcherry without hardware upgrades :)
<wgrant> Which means that once slony is gone we will never have any reason to have downtime ever again.
<wgrant> (we might just have to pause write requests for a few seconds here and there)
<lifeless> wgrant: and reads...
<wgrant> Nah. We can do a rolling upgrade on the DB servers to avoid that.
<wgrant> Pause replication, pause writes, switch to slave-only, upgrade master, switch to master-only, unpause writes, reattach slaves, wait for slaves to sync, switch to normal.
<wgrant> s/reattach slaves/unpause replication/
<lifeless> wgrant: we've had this talk before
<lifeless> wgrant: I know you're convinced; I'm not
<lifeless> wgrant: I'd like to make sure that we don't count on something that isn't (yet) known to work-or-not work : decide at the last possible moment
<wgrant> I'm hoping one day I'll catch you over-tired and you'll be convinced too.
<lifeless> we can talk on skype now if you want
<lifeless> I'm certainly over-tired ATM
<wgrant> Nah.
<wgrant> This is still a while off :)
<wgrant> Heh
<lifeless> the fundamental issue I see is that we don't have a hard timeout on scripts yet, don't have the hard timeout <=5 seconds everywhere, and without both of those things, downtime will be faster overall.
<wgrant> Also, pg9.1 has some pretty nice lock reductions which let fkey modifications etc only take out exclusive read locks.
<wgrant> Er
<wgrant> exclusive *write* locks
<wgrant> So reads can continue while you're adding constraints.
<wgrant> Which is interesting.
<wgrant> Sutre, there are those issues.
<wgrant> This is a future thing :)
<wallyworld> StevenK: discovered the ff issue - not a real issue. there's a "legit" try/catch in the yui code to handle new vs legacy querySelector behaviour and it seems firebug had "break on exception" on (i didn't turn it on). you turn it off using the toolbar button in the console panel
<StevenK> Right
<wallyworld> the other thing is that sadly even with the new html5 browser behaviour to use lucid's webkit the yui tests still pass locally
<StevenK> Run ec2 with -p?
<wallyworld> -p does?
<wgrant> Or ec2 demo, ssh in, and test manually?
<wallyworld> might try that. will ec2 demo do all the setup that ec2 test does?
<StevenK> -p is --postmortem
<wgrant> Pretty much
<wallyworld> with -p i have to wait for failure, right? whereas with ec2 demo i can get straight in and try stuff
<StevenK> Right
<wgrant> wallyworld: Want me to try them locally?
<wallyworld> with the html5 browser, i had to move and egg from lp-source out of the way so it would use the system version
<wgrant> I think I have YUITestLayer working now.
<wgrant> In my container.
<wallyworld> wgrant: that would be nice if you could
<wallyworld> wgrant: you would need to pull pillar-access-service-infrastructure,  more-accesspolicy-service, more-accesspolicy-service-2, more-accesspolicy-service-3, more-accesspolicy-service-4
<StevenK> Or just more-accesspolicy-service-4?
<wallyworld> and then bin/test -vvct test_observertable
<wallyworld> it's a pipe, so the previous ones are needed
<wgrant> s/are/are not/
<StevenK> Not if you've used bzr pump
<wallyworld> which i have
<wallyworld> didn't realise that
<wgrant> Pipelines aren't magical.
<wgrant> They're just a series of branches with a command to merge between them, basically.
<wallyworld> right, i didn't appreciate/forgot that pump did the merge
<StevenK> It does mulitple if you pump from the first pipe :-)
<wallyworld> yeah. and i've had several conflicts to resolve doing that the past week or so
<StevenK> Welcome to pipelines
<wgrant> Ah
<wgrant> The tests run better when the JS has been built, it turns out.
<wallyworld> indeed
<wgrant> There we go, errors.
<wallyworld> that is a new thing due to the combo loader work
<wallyworld> so wonder why it errors for you and not me
<wallyworld> given we are both using the lucid webkit etc
<wgrant> This is on lucid
<wgrant> You're not using lucid webkit
<wallyworld> wonder why your lxc seup worked then
<wgrant> You're using a possibly old version of webkit that's in precise, but might not actually be old.
<wallyworld> wgrant: did you run "sudo lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username" ?
<wgrant> Different name and different config file, but yes.
<wgrant> lxc.network.type = veth
<wgrant> lxc.network.link = lxcbr0
<wgrant> lxc.network.flags = up
<wgrant> Was the config I used
<wgrant> Note that the bridge device should be lxcbr0 in precise, rather than virbr0 in earlier releases.
<wallyworld> i used virbr0
<wgrant> Should still work, if you have a virbr0
<wallyworld> but my issue what the container wouldn't start
<wgrant> But it's not created by default when you install lxc
<wgrant> Oh?
<wallyworld> it said could not mount rootfs
<wallyworld> and then other error messages from there
<wgrant> Tried disabling the apparmor profile?
<wgrant> sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disable/; sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start
<wallyworld> i think i tried that. will try again
<wallyworld> well fuck me, that worked
<wallyworld> i will go add that to the wiki
<wallyworld> thanks
<wallyworld> wonder why apparmor doesn't like lxc
<wgrant> Bug #925024
<_mup_> Bug #925024: apparmor makes it impossible to install postgresql-common on Precise <apport-collected> <bot-stop-nagging> <kernel-request-3.2.0-13.22> <kernel-request-3.2.0-14.23> <precise> <rls-p-tracking> <running-unity> <linux (Ubuntu):Confirmed> <lxc (Ubuntu):Confirmed> < https://launchpad.net/bugs/925024 >
<wgrant> It seems to be a kernel bug.
<wgrant> AFAICT the rejected operations should be permitted.
<wgrant> So possibly cgroups and LSM don't like each other very much.
<wgrant> Again.
<wgrant> wallyworld: Note that for html5-browser to work in the container it'll need an X server.
<wgrant> So either ssh in with -X, or use xvfb-run
<wallyworld> will try that, thanks
<wallyworld> wgrant: do you use a separate source tree for lp? otherwise you need to make each time you switch using lxc to metal
<wallyworld> due to python 2.6 vs 2.7 etc
<wgrant> wallyworld: I don't run on metal.
<wallyworld> ah, ok
<wgrant> Interestingly, the rest of the workarounds on /Running/LXC seem to be unnecessary on precise.
<wgrant> All I needed was to apt-get install lxc, disable the apparmor profile, lxc-create, and rocketfuel-setup
<wallyworld> i find it too much of a pita to start/login to lxc so i stick to metal
<wgrant> I have stuff in ~/bin to handle that.
<wgrant> Just tiny shell scripts
<wallyworld> still extra typing
<wgrant> lp-start, lp-sh, lp-stop
<wallyworld> i need to run rocketfuel with --no-workspace too
<wgrant> lp-sh handles sshing in with -A -X, and changing to the right dir
<wgrant> And lp-bounce restarts the appserver in the current directory.
<wgrant> Reasonably smooth.
<wallyworld> yes, doesn't wound too bad. you also need to hack postgres config too i think
<wallyworld> if you run postgres outside the container
<wgrant> They should be unrelated.
<wgrant> Oh, you mean run the client outside the container?
<wgrant> You could, but I just do it through SSH
<wallyworld> you need to alter the config to allow "external" connections. well i did last time i tried lxc
<wallyworld> ah yes, client outside container i think was it
<lifeless> the main thing for me is editing hsots all the time
<lifeless> drives batty me
<wallyworld> my view is that using lxc just isn't worth the hassle, expecially when lp dev is painless on metal
<wgrant> lifeless: Why do you have to edit it frequently?
<wallyworld> i'm only using it to try and reproduce a ec2 test failure
<wgrant> My dnsmasq tends to be good about reassigning the address like once every 6 months.
<lifeless> wgrant: so my primary lxc is on my desktop (which has 8 cores and 16GB ram); it doesn't keep the same IP (which is being allocated from my LAN dhcp server)
<wgrant> Ah
<wgrant> lifeless: It has a static MAC...
<lifeless> I haven't tracked down how it gets a 'hwaddr' for its 'ethernet'
<lifeless> wgrant: it should, but ..
<wgrant> $ grep hwaddr /var/lib/lxc/lplucid/config
<wgrant> lxc.network.hwaddr= 00:16:3e:24:3f:3e
<lifeless>  grep hwaddr /var/lib/lxc/lucid-test-lp/config
<lifeless> robertc@lifeless-64:~$
<wgrant> loloneiric?
<lifeless> natty
<lifeless> I think
<lifeless> anyhoo
<wgrant> Yeah
<wgrant> precise is muuuuch nicer
<lifeless> how does that interact with ephemeral
<wgrant> No workarounds required.
<wgrant> Good question.;
<lifeless> (I'm on precise)
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/more-bug-heat-murder/+merge/96033
<poolie> bigjools, thanks :), 'instant virgin' or 'comic sans'?
<poolie> oh dev tips :)
<wallyworld> wgrant: can you recall the command to recompile things like svn libs in sourcecode since my lxc is 32bit and host is 64bit and stuff won't run atm
<lifeless> wallyworld: make clean; make
<wgrant> wallyworld: make -C sourcecode clean all
<poolie> sudo make shorter
<wgrant> s/all/build/, maybe
<wallyworld> lifeless: did make clean; make, will try what wgrant said
<wallyworld> ah -C sourcecode does delete all the .so files
<StevenK> wgrant: What's the ppad user?
<wallyworld> wgrant: faaaaark. now bin/test -vvct test_observertable gives me a core dump
<lifeless> quick reminder
<lifeless> does the job runner switch db user?
<lifeless> or do they all run as one megaar user ?
<wgrant> StevenK: Ancient, from before ArchiveRework
<wgrant> lifeless: It switches sometimes.
<wgrant> wallyworld_: You're running with -X?
<wgrant> wallyworld_: It'll segfault or abort if there's no X server.
<wallyworld_> wgrant: ah ffs. forgot that. thanks
<wallyworld_> wgrant: i was recalling how previously we had issues running a 32 bit lxc container on a 64 bit host and there were seg faults
<wgrant> wallyworld_: That's what I'm doing.
<wgrant> Always have done.
<wallyworld_> there were segfaults mid last year
<wallyworld_> and curtis had to add changes to html5 browser
<wgrant> Oh, for html5-browser
<wallyworld_> strangely, some of the yui tests passed without X
<wgrant> I've never really used it.
<wallyworld_> wgrant: that's what's used to drive the yui tests
<wgrant> I know.
<wgrant> I hadn't run them before today.
<wallyworld_> ah ok
 * wallyworld_ sees a pommy git ^H^H^H^H^H bigjools arriving at his house to use the printer
<wgrant> Heh
<nigelb> haha
<wgrant> lifeless: I suspect this branch will fail ec2.
<wgrant> lifeless: Because person merge will still want to talk to the table.
<wgrant> It should really have landed with the patch, to db-devel.
<wgrant> But we'll see.
<lifeless> should be able to drop bugjob tonight no ?
<wgrant> Might *just* be in db-stable in time.
<wgrant> Or did it make it through before the devel failure?
<lifeless> enoidea
<wgrant> It's been in db-stable for about 20 minutes.
<wgrant> Er
<wgrant> s/db-//
<wgrant> So it will be in db-stable in about 6.5 hours
<wgrant> So it's probably going to be tomorrow.
<wgrant> Unless we violate the usually procedures and do the db-deploy from stable
<lifeless> wgrant: whats the new shiny for db user switching ?
<lifeless> StevenK: have you landed your enterprise id helper yet ?
<wallyworld_> I'd just like to say that bigjools is awesome
 * wallyworld_ decides to locks his screen next time he makes that pommy git a coffee
<wgrant> lifeless: see lib/lp/testing/dbuser.py
<wgrant> with dbuser('foo')
<wgrant> Or with lp_dbuser()
<wgrant> Or switch_dbuser('foo') if you hate life and context managers.
<wgrant> lifeless: Unless you want to do it in production code, in which case you're approaching things the wrong way.
<lifeless> nah, just writing smoke tests
<lifeless> and oot of practice
<StevenK> lifeless: Playing in ec2 last I checked
<StevenK> add-enterpriseid => devel                 [OK]       (up for 4:31:32) i-dbc91abf
<StevenK> So, 30 minutes ago? :-P
<StevenK> lifeless: Just landed. r14907
<sinzui> wallyworld_, have you made progress? Is there a firefox defect in Lp's javascript?
<wallyworld_> sinzui: html-browser still worked fine on precise with the older webkit, but i did get lxc working - needed to disable apparmor
<StevenK> sinzui: Based on my investigation, public bugs with private branches work fine, and that bug can be closed.
<wallyworld_> sinzui: the issue is that a Y.one() call is returning null when it shouldn't - i have printed the innerHTML and the requested node is there
<sinzui> StevenK, really? okay
<StevenK> sinzui: It worked for me on .dev, if you have a few minutes we can try it on prod
<wallyworld_> sinzui: the one issue i can see is that there is an <img blah blah> with no closing tag within the <span> that is being selected and not returned
<sinzui> wallyworld_, are events involved...is the node not there *yet*, but is byt the time print happens
<sinzui> StevenK, sure
<sinzui> wallyworld_, !
<wallyworld_> sinzui: but the mustache template uses "<img blah />" and we do that same thing in lots of other places. using "<img blah></img>" makes no difference
<StevenK> sinzui: Do you have a private branch that you can see and I can't?
<sinzui> wallyworld_, invalid HTML does indeed lead to unpredictable DOM traversal
<sinzui> wallyworld_, "<img blah></img> really is invalid and a *HTML* parser will throw a wobbly
<wallyworld_> sinzui: weird thing - using "<img blah></img>" and logging innerHTML still prints "<img blah>" with no closing tag
<sinzui> and <img ...> will also throw the *HTML* parser for a wobbly if the page claims it is xhtml
<wallyworld_> so it should be <img blah /> ?
<sinzui> wallyworld_, do not assume that the thing printing is the thing interpreting
<wallyworld_> what's the correct syntax for img ?
<sinzui> that is how lots of fucked up markup is introduced
<sinzui> wallyworld_, we state we serve xhtml, the tag is a self-closing empty tag
<wallyworld_> sinzui: like "<img blah />"? that's what's in the mustache template
<sinzui> wallyworld_, yes
<wallyworld_> but print innerHTML is leaving out the closing tag
<wallyworld_> which i was thinking was the issue
<wallyworld_> but is not
<sinzui> wallyworld_, yes because that method is 1, not in any standards
<sinzui> and 2 it predated xhtml by 3 years
<wallyworld_> so it still makes no sense why the Y.one() is failing on lucid
<wgrant> wallyworld_: What is the precise text of the template?
<wallyworld_> wgrant: https://pastebin.canonical.com/61669/
<wgrant> Is that valid?
<wgrant> An img with no src?
<sinzui> wgrant, I think you have a point
<wgrant> You might mean <a href="#" class="editicon sprite edit">&nbsp;</a> or so
<wgrant> I'm not sure what the usual pattern is.
<wgrant> Since there's like 4 different ways to do it in the codebase already.
<wgrant> Let's throw in a couple more :)
<wallyworld_> wgrant: sinzui: got it working. used Y.one("[id=xxxx]") syntax
<wallyworld_> instead of "#xxx"
<wgrant> Still, not valid HTML
<sinzui> wgrant, close enough. We use hidden text for TestBrowser, which will never see this anyway
<bigjools> wallyworld_: I smell of Baldrick
<wgrant>   src         %URI;          #REQUIRED -- URI of image to embed --
<wgrant> QED
<wallyworld_> bigjools: too much info
<wallyworld_> wgrant: here there is no src, it's a sprite
<sinzui> wallyworld_, I think that means an invalid id was generated. does it have dots in it
<wgrant> wallyworld_: Then it's not an img
<bigjools> wallyworld_: enjoying the pommy weather?
<sinzui> or was it a duplicate...which is also invalid
<wallyworld_> sinzui: nope, otherwise i would not haves used #
<wallyworld_> not a dupe either
<wallyworld_> bigjools: no
<bigjools> wallyworld_: bwahaha :)
<bigjools> wallyworld_: anyway strictly speaking, you're the pohm
<wgrant> http://www.w3.org/TR/xhtml1/dtds.html#dtdentry_xhtml1-strict.dtd_img agrees
<wgrant> So both XHTML1(.1) and HTML4.01 hate you
<wallyworld_> wgrant: i see where i fucked up - i don't need the img
<wallyworld_> i just needed to add class = "sprite edit" to the anchor
<wgrant> Yeah, and probably an nbsp or similar inside it
<wgrant> Otherwise it can end up zero-width
<sinzui> wallyworld_, correct. we use sprites on other elements to avoid the img element
<wgrant> Which isn't awesome for displaying bgimages.
<sinzui> we do not want to tempt the browser to download
<wallyworld_> sinzui: so webkit on precise must be more forgiving
<wgrant> We've had problems like that before, yeah.
<wgrant> Some old Firefoxes don't like self-closing <div>s, for example.
<sinzui> wallyworld_, yes. the newer lib emulates the behaviours of gecko and trident better
<wallyworld_> sinzui: i started out using img with src='@@/edit' or whatever and when i decided to use sprites instead messed up the conversion
<sinzui> yes, a div is not allowed to be empty. the same is true for <p>
<wgrant> Ah
<wgrant> Easily fixed :)
<wallyworld_> sinzui: there should be a strict xhtml or fail mode on the newer libs
<sinzui> wallyworld_, that is from 2009. by 2010 we were supposed to stop using @@/ urls because they waste resources
<wallyworld_> sinzui: yeah, i copied from elsewhere and realised sprites were preferred and messed up the fix
<sinzui> wallyworld_, lots of old code still uses the spectacles/testicles style
<sinzui> wallyworld_, IE 9 actually has a instruction to tell it not to fallback. ...do xhtml or html5 or die trying
<wallyworld_> sinzui: i'm still perplexed at the behaviour - Y.one('#xxx') fails, Y.one('[id=xxx]') works, and in both cases, the offending <img> is well inside the containing element being selected
<wgrant> You're invoking undefined behaviour.
<wgrant> Of course it's perplexing :)
<wallyworld_> sinzui: so no webkit library strict xhtml switch ?
<sinzui> wallyworld_, The fact is some many people never learn the standards, that all the engines are written in a try/except pattern. Try to do the doc type. on exception slow down, choose to re-write the dom to be valid, or fallback to quirksmode
<wgrant> wallyworld_: Serving as application/xhtml+xml will cause Gecko and WebKit to parse it as XML
<wgrant> IE9 too
<wgrant> But we can't do it, because Microsoft are fucktards and didn't support it until IE9
<wallyworld_> sinzui: also, i hacked in hide_console_messages=False to see the logged output
<wallyworld_> i think that's what you were going to put back in as a cmd line switch?
<wallyworld_> wgrant: i was meaning xhtml for running the tests
<sinzui> wallyworld_, I don't see anything wrong with id="P1-permission"
<sinzui> Maybe we need to look at the whole file to see if something interfered with the block under test
<wallyworld_> sinzui: me either. but since the <span> contains that invalid <img> it must bark
<wallyworld_> barf
<sinzui> wallyworld_, this test is from your 3rd branch right?
<wallyworld_> sinzui: yes
<wgrant> -4
<wallyworld_> -3
<wallyworld_> or -4
<wallyworld_> introduced in -3
<wgrant> Ah
<wgrant> The test results were from -4
<wallyworld_> wgrant: that's the one i was landing
<wallyworld_> wgrant:  sinzui: but fixing the <img> still fails when i used Y.one('#xxx')
<wallyworld_> s/fixing/deleting
<wgrant> :(
<wgrant> While it didn't make much sense that that would cause the problem, I had hoped the layout engine was retaliating to discourage people from writing blatantly invalid markup.
<wallyworld_> but Y.one('[id=xxx]') works
<sinzui> wallyworld_,  Can you convince the console to print everything in the body?
<wallyworld_> sinzui: yes, believe so. let me do that
<wgrant> the fuck
<wgrant> Line 47 of base-layout.pt
 * wgrant deletes
<wallyworld_> sinzui: ah, ffs. i know why i left the img in there. lazr ChoiceSource requires an editcon img node
<wallyworld_> i will have to look at the ChoiceSource src code to see if there's a way around that
<sinzui> see, we wrote a lib that requires us to screw with things
<wallyworld_> sinzui: so rather than go back to using <img src='@@/edit'/> i'll fix ChoiceSource or figure out how to call it differently
<sinzui> wgrant, stab away
<wallyworld_> i may just be able to pass in the anchor
<wgrant> wallyworld_: Then we can fix the inline bug modifying stuff to not be terrible :)
<wgrant> Currently the edit icon loads late
<wgrant> Because it's not a sprite
<wallyworld_> yeah :-(
<wallyworld_> well, what a frustrating day or so
<sinzui> wallyworld_, the image is secondary to the id-lookup-fail
<wallyworld_> sinzui: ok, will get print the body, just a sec
<wallyworld_> sinzui: https://pastebin.canonical.com/61671/
<wallyworld_> let's see if it looks valid
<sinzui> wallyworld_, looking at the source _observer_policy_template may be creating duplicate ids if there is more than one user
<wallyworld_> sinzui: you may be correct there - this is in the li elements
<wallyworld_> sinzui: and indeed, in the test, data for 2 users is rendered
<sinzui> yep
<sinzui> the ids are reused
<sinzui> they MUST fail since they are invalid
<wallyworld_> sinzui: scary that it passes on precise
<sinzui> wallyworld_, I declare shenanigans. I see some of the test already looks up id as a normal attr. I think only the first id in the tree is returned
<sinzui> or in the case of a tree fragement, only the child matches
<wallyworld_> sinzui: all but one of those test lookups do lookup unique ids
<wallyworld_> so should be changed to #
<sinzui> I see this is common practice in out code too because zope is putting dots in our ids :(
<wallyworld_> yes :-(
<sinzui> wallyworld_, we do want to it be #, but we need to add the user to the id's string as was done with remove
<wallyworld_> sinzui: the other workaround used is to Y.DOM.xxx
<wallyworld_> yes, i need to fix the template
<wallyworld_> sinzui: my point about it being scary that the tests pass on precise still stands though. we should see if something can be done
<sinzui> wallyworld_, I am looking at http://webkitgtk.org/reference/index.html right now
<wallyworld_> ok ;-). and i'm figuring out how to adjust my use of partials to get the required data available for the template to use
<sinzui> I do not see a strict flag to ensure the test dies
<wallyworld_> that is unfortunate
<sinzui> wallyworld_, !
<wallyworld_> ?
<sinzui> wallyworld_, your test html does not claim to be xhtml strict
<sinzui> wallyworld_, set this as the first two declarations
<sinzui> <?xml version="1.0" encoding="UTF-8"?>
<sinzui> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<wallyworld_> in the test harness html?
<sinzui> change transitional to strict
<sinzui> yes
<sinzui> That will change how any browser written in the last 5 years interprets the content
<wgrant> We should really be polyglot (X)HTML5 with a doctype of <!DOCTYPE html>
<wallyworld_> sinzui: test_observertable.html has:
<wallyworld_> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
<wallyworld_>   "http://www.w3.org/TR/html4/strict.dtd">
<wgrant> lolwut?
<wgrant> Launchpad has *never* been HTML 4.01
<wgrant> It's had a doctype of XHTML 1.0 Transitional from the very start.
<StevenK> % bzr grep Transitional | grep -c 4.01
<StevenK> 6
<sinzui> wgrant, I was thinking about that last week. I suspect there will be some oddness that we need to test to ensure we do not gt stung. such as our ids
<StevenK> lib/lp/registry/help/sharing.html:<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
 * wgrant blames wallyworld and jcsackett for living in the 1990s :)
<sinzui> wgrant, We tried html5 in the 2.0 design. IE hated it. we know know it is trivial to make it accept it
<wgrant> sinzui: Well, AIUI IE6 is now dead to us.
<wallyworld_> sinzui: given test_observertable.html appears correct, i'm not sure what else i should be changing
<wgrant> IE7 and IE8 still only have HTML4.01 parsers, with no HTML5 or XHTML, but they should cope better.
<sinzui> Ie 7 and 8 need a js initiate dom check to reenable css after it encounters html 5
<StevenK> wgrant: I didn't think we could declare IE6 support dead?
<wgrant> IE6 is more than 10 years old.
<wgrant> Even if there's a good reason to support it, there's no good reason to support it.
<sinzui> Like Jimmy Hoffa and Elvis, we do not need to declare them dead. They are dead, and we will ignore them
<wgrant> Now, Microsoft continues to fuck the world over by preventing IE9 from running on Windows XP. But at least declaring IE6 and potentially IE7 dead gives us *some* freedom.
<sinzui> regardless, the js-dom-creation hack works in IE 6 and restores CSS after the html5 element is encountered
<sinzui> or at least say, you must try another OS, how about Ubuntu
<StevenK> sinzui: Are you undistracted now? :-)
<sinzui> yes
<StevenK> sinzui: Do you have a private branch on prod that you can see, but I can't?
<sinzui> maybe
<sinzui> StevenK, what bug do you want me to link it too?
<StevenK> sinzui: https://bugs.launchpad.net/moblin-multimedia/+bug/221429 ?
<_mup_> Bug #221429: moblin media fails to launch <intel> <acton:Invalid> <Moblin Applets:Invalid> <Moblin Multimedia:In Progress by prajwal-linux> <Ubuntu Mobile Edition:Fix Released> < https://launchpad.net/bugs/221429 >
<sinzui> linked
<sinzui> StevenK, can you still see the bug?
<StevenK> sinzui: Looks good to me. No Linked branches.
<StevenK> sinzui: Can you see the branch linked?
<sinzui> fab. This tested private team and private branch
<sinzui> I removed the link
<StevenK> Oh, wait, it's talking about a different page
<sinzui> is this a doh moment
<sinzui> ?
<StevenK> No, I'm wrong, the user says he can't see a bug page linked from +assigned-bugs
<StevenK> So it is IBugTask:+index that is implicated.
<sinzui> StevenK, that is still the bug page
<StevenK> sinzui: I'll comment and mark it fix released, do you want me to just archive the card?
<sinzui> yes. We did time on it
<StevenK> sinzui: Plan to tackle 933759 for the rest of the day
<StevenK> Sigh. bug 933759
<_mup_> Bug #933759: Added information_visibility_policy to bug and branch <branches> <bugs> <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/933759 >
<StevenK> I just wrote 2 garbo jobs, what's one/two more
<sinzui> Have fun storming the castle. I plan to take a Cadbury chocolate bar for the rest of my day
<StevenK> I suppose it requires a DB patch first
<StevenK> wallyworld_: Has your enum landed? If so, what is it called?
<StevenK> Oh, duh, it's AccessPolicyType
<StevenK> Does that mean bug 933759 is out of date and the column on bug and branch should be named differently?
<_mup_> Bug #933759: Added information_visibility_policy to bug and branch <branches> <bugs> <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/933759 >
<lifeless> StevenK: cool
<StevenK> wgrant: Am I right? I should be adding an accesspolicy column to bug and branch?
<StevenK> lifeless: Can you untestable or something r14902?
<wgrant> StevenK: Not accesspolicy
<wgrant> Potentially accesspolicytype
<StevenK> wgrant: A shorter form of what sinzui suggested is information_visibility, but the enum is AccessPolicyType.
<StevenK> Perhaps visibility_policy
<wgrant> It is complicated.
<wgrant> Because there are public bits in AccessPolicyType
<wgrant> Which I had not envisioned.
<huwshimi> wgrant, wallyworld_: In a private project isn't everything Proprietary data?
<wgrant> huwshimi: It should be, yes.
<huwshimi> wgrant: Do you have the option for anything else?
<wgrant> I'd hope not.
<wgrant> Potentially you could argue that userdata and embargoedsecurity could exist in a private project.
<wgrant> But I don't believe we've been asked to do that.
<huwshimi> wgrant: and some public projects can have some Proprietary data?
<wgrant> All three can exist in a public project with optional proprietary bugs.
<huwshimi> wgrant: Gotcha
<huwshimi> wgrant: Thanks for that
<wallyworld_> wgrant: I am adding a mthod to IAccessPolicyGrantSource....
<wallyworld_>     def delete(grants):
<wallyworld_>         """Delete the specified grants.
<wallyworld_>         :param grants: a collection of (`IAccessPolicy`, grantee `IPerson`)
<wallyworld_>             pairs.
<wallyworld_>         """
<wallyworld_> that ok with you?
<wallyworld_> i need it for the +sharing page
<wallyworld_> wgrant: also, have you considered using named tuples for the various parameters and return types?
<wgrant> wallyworld_: s/delete/revoke/
<wallyworld_> ok
<wgrant> wallyworld_: I avoided namedtuples because they're not very JSON-friendly.
<wallyworld_> i thought there would be support for them
<wallyworld_> seems like an expected thing to have
<wallyworld_> but maybe not
<wgrant> Why would you expect that?
<wgrant> They're a native feature in just about 0 languages.
<wgrant> They're an awkward hybrid of a map and a list.
<wgrant> Which has very little reason to exist.
<wallyworld_> they are implemented under the covers like a normal tuple
<wgrant> Sure.
<wallyworld_> so i thought the generated implementation might have something
<wgrant> But you can't represent a namedtuple in JSON.
<wallyworld_> since it would be such a common requirement
<wgrant> Heh, common?
<wgrant> There's no sensible way to represent a namedtuple in JSON, so it's unlikely that people would do it...
<wallyworld_> but you can represrnt a dict in json which a named tuple can be mapped to i think
<wgrant> No.
<wgrant> namedtuples have order
<wgrant> You could map a namedtuple to a dict.
<wgrant> But you can't round-trip through a sensible JSON representation.
<wallyworld_> sure, but the order could be encoded in the json
<wgrant> How?
<wgrant> Make JSON map definitions order-sensitive?
<wallyworld_> as a list of names
<wallyworld_> for example
<wallyworld_> it would just need serial/deserialisation on the named tuple generation
<wallyworld_> s/generattion/generated class
<wgrant> You can't deserialise.
<wgrant> Because you don't know what namedtuple to deserialise to.
<wallyworld_> you'd encode that as well
<wallyworld_> i used to do this sort of thing in java all the time
<wgrant> Adding namedtuples to JSON would provide very minimal benefit for large additional complexity.
<wallyworld_> surely pythin can support similar pluggable serialisation mechanisms
<wgrant> Sure.
<wgrant> But this isn't Enterprise Java XML SOAP Beans EE2 7.5
<wallyworld_> anyways, it's more of a pub discussion :-)
<wgrant> This is JSON
<wgrant> It's simple.
<bigjools> awesome - bugs.kde.org changed its design and now the crash reporting assistant fails to work...!
<wallyworld_> it can be done simply in java, without any "enterprise" stuff, it's supported by the langiage
<wgrant> wallyworld_: It's not supported by the format.
<wallyworld_> bigjools: what, bugs in kde? surely not?
<bigjools> wallyworld_: shocking I know
<wgrant> wallyworld_: And I would damn well hope it's not supported by the language.
<wgrant> By parts of the (potentially standard) libraries, perhaps.
<wallyworld_> wgrant: what's so bad about pluggable serialisation?
<wgrant> Nothing, but serialisation doesn't belong in the core language :)
<wallyworld_> the language sh/could support mechanisms to add it
<wallyworld_> easily
<wallyworld_> via libraries available at runtime
<wgrant> Anyway, there's no reason to serialise namedtuples to JSON.
<wallyworld_> don't agree, but doesn't matter
<wallyworld_> wgrant: another question - why in the impl for IAccessPolicyGrantSouuce call AccessPolicyGrant?
<wallyworld_> and not AccessPolicyGrantSource?
<wgrant> eparse
<wallyworld_> the implementing class
<wgrant> Oh, s/in/is/?
<wallyworld_> is
<wallyworld_> sorry, tying
<wallyworld_> typing
<wallyworld_> arrgh
<wgrant> It's one of the two major conventions we have.
<wgrant> Either you have Foo implementing IFoo, and FooSet implement IFooSet
<wgrant> Or you have Foo implementing IFoo, and class methods implementing IFooSource
<wgrant> That was used a bit more in code
<wgrant> s/code/Code/
<wgrant> Well, I guess technically Foo implements IFoo, and provides IFooSource.
<wallyworld_> hmm. ok. i find that latter convention "interesting"
<wgrant> Most of Launchpad is "interesting"
<wallyworld_> lol
 * wallyworld_ pauses new AccessPolicyGrant method implementation - off to have dinner with bigjools. he is even older today
<bigjools> wallyworld_: where's my birthday pizza you aussie git?
<wallyworld_> i am leaving to pick it up now you pommy whinger
<StevenK> Haha
 * bigjools hears aussies whinging
<wgrant> Ha ha hgwefwhui
<wgrant> So it turns out that some of our pagetests only work because our HTML is invalid, preventing some directives from being parsed.
<wgrant> eg. zope.testbrowser follows http-equiv refresh
<wgrant> But currently they have an illegal <noscript> element before them.
<wgrant> So mechanize doesn't see them.
<bigjools> heh
<adeuring> good morning
 * StevenK stabs kanban
<czajkowski> Good morning
<wgrant> jelmer: https://code.launchpad.net/~elementary-os/+recipe/elementary-os-qt4-x11-patch dies with a bzr MemoryError... do you want a bug?
<wgrant> Using 800MB so far in my test.
<jelmer> wgrant: yes, please
<jelmer> wgrant: any idea what's so particularly big about the branches in that recipe?
<wgrant> jelmer: Qt :)
<wgrant> Wow
<wgrant> 5.4GB so far in bzr's progress meter
<wgrant> over HTTP
<wgrant> 431MB on disk so far
<jelmer> urgh
<wgrant> Up to 1GB resident
<jelmer> that is one ginormous branch
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Firefighting: - | Critical bugtasks: 4*10
<StevenK> gmb: Ah ha, so evil BT managed to fix their crap.
<gmb> StevenK, Wasn't their fault, as it happens; some electronics needed replacing at the exchange. I guess it's the telecoms equivalent of a NIC burning out.
<gmb> They fixed it pretty quickly.
<gmb> Oh, wow. I've been given a Â£0.01 refund on the Game of Thrones DVD set. Thanks Amazon, I'll try not to spend it all at once...
<StevenK> gmb: Telstra here don't tend to admit that here. "Oh, it's affecting ... some other people. Which may or may not be geographically close to you."
<gmb> StevenK, Ah, we're a small enough community for them not to be able to get away with that. Also, there's an engineer living in the village, so breakages tend to affect him, too.
<StevenK> Heh
<StevenK> The last time I can remember Telstra pulling that crap was when one of their conduits wasn't properly closed in. So within about five minutes of it raining, bang, DSL drops.
<gmb> Nice
<StevenK> With some bullying on my part I got the nice girl to admit that it was impacting ~ 200 people which is why it got fixed in 3 hours instead of 3 days.
<StevenK> salgado! Why is workitems run from frequently? :-(
<salgado> StevenK, because it's low-overhead
<salgado> is it causing problems?
<StevenK> It shouldn't be frequently, there is absolutely no need for it
<salgado> StevenK, it's important to have it run frequently because then it will kick in as soon as we switch it on in production, thus reducing the time window where we have the new UI but not all work items migrated
<StevenK> salgado: Utterly pointless. You are not going to lose much by using hourly, and even so, you can make sure the workitems are migrated before switching the UI on.
<salgado> StevenK, I don't want to do the migration before the UI is live, for obvious reasons
<StevenK> salgado: Frequently is used for things that will cause production issues if it lags, like OpenID gardening and so on. I am distinctly uncomfortable with it being in frequently
<salgado> StevenK, that's not what the docstring says
<StevenK> That was the entire reason it was added by stub.
<StevenK> But, whatever, I'm tired of arguring.
<jelmer> wgrant: did you end up filing a bug for that qt branch?
<stub> salgado, StevenK: We have started moving stuff with low setup costs into frequently because the load gets distributed better (a small bit of work every 5 minutes vs. a larger chunk of work every hour). But things are trivial to shuffle between daily/hourly/frequently if one of them gets overfull or 'low overhead' turns out to be not so low.
<deryck> Morning, all.
<deryck> abentley, adeuring -- let's hold standup for 30 minutes past normal time.  so 15:00 UTC.
<adeuring> deryck: ok
<abentley> deryck: okay.
<deryck> abentley, adeuring -- I have sick kids again. :(  And their aunt is coming for them at normal standup time.
<abentley> deryck: That's too bad.
<deryck> yeah, it's like we're on a flu/cold loop at my house.
<sinzui> abentley, I restored my patch the bzr-pqm because ec2 land failed: https://bugs.launchpad.net/bzr-pqm/+bug/922741
<_mup_> Bug #922741: AttributeError: 'BzrBranch7' object has no attribute 'get_config_stack' <amd64> <apport-bug> <patch> <precise> <running-unity> <Bazaar PQM Plugin:New> < https://launchpad.net/bugs/922741 >
<sinzui> abentley, do you want me to retarget the bug to lp-dev-utils?
<abentley> sinzui: did it fail locally?
<sinzui> abentley, I don't understand
<abentley> sinzui: Yesterday you tried it on your local machine and said it was fine.  I don't understand what's changed.
<sinzui> abentley, I used bzr pqm-submit yesterday
<sinzui> today I used ec2 land
<abentley> sinzui: did you use the lp-dev-tools version of ec2 land?
<sinzui> no I  used the one in the Lp tree
<abentley> sinzui: I would be very surprised if the lp-dev-tools version was affected.
<sinzui> I will revert and do it again
<abentley> sinzui: because the problem with the launchpad-tree version is that it's using a beta version of 2.5.  The lp-dev-tools version will use your system bzr, and any non-beta bzr should work with it.
<sinzui> abentley, I see you are correct
<sinzui> I am now marking that bug invalid
<abentley> sinzui: Or maybe affects: launchpad.
<sinzui> Didn't you commit to removing the ec2 from Lp's tree
<abentley> sinzui: Yes, but it may not be for a day or so, because I want to get a suitable version of bzr-pqm in the PPA.
 * sinzui nods
<abentley> sinzui: My plan to copy the package has been thwarted because only the Precise package built successfully.
<sinzui> abentley, good enough for me
<sinzui> abentley, Ubuntu is in beta...Lp engineers are supposed to be on Precise now
<deryck> grrr, my home internet provider hates me some days.
<abentley> sinzui: true.  Maybe that's good enough.
<sinzui> abentley, stevenk deleted packages for unsupported series. I imagine he is already eyeing oneiric
 * deryck is waiting on the new laptop to upgrade
<sinzui> what are you getting? Are you joining the SSD party
<abentley> sinzui: I believe deryck's getting a System76 with an SSD.
<salgado> danhg, hey there.  was wondering if you'd have some time to chat about the results of the user testing with those two new LP pages?
<czajkowski> system 76 do nice machines alright
<deryck> I am indeed getting system 76.  16 Gb RAM, Super hot graphics card, and 480 Gb SSD.
<czajkowski> nice
<al-maisan> "480 Gb SSD" -- wow!
 * al-maisan envies deryck 
<deryck> :)
<sinzui> I was very disappointed with System76's quality and support. They would not admit there was a defect in the cdrom drive because it is revealed their source. I had to borrow someone's windows installer to install the free firmware fix that they did not want to admit existed. The display died 4 months later. Though they fixed it, the trackpad became unreliable 6 months after that.
<deryck> ouch
<deryck> I've heard stuff like that, and heard others who love them.
<deryck> I'll let you all know what I think soon :)
<deryck> adeuring, abentley -- standup in 2?
<adeuring> ack
<abentley> deryck: sure.
<deryck> g+ is taking a bit, sorry
<salgado> mrevell, hi there.  did you get my email about the notification about work-items that can't be migrated automatically?
<salgado> (on launchpad-dev@)
<czajkowski> sinzui: aye one reason I didnt buy one, lving in the UK wasnt sure of the level of customer care
<czajkowski> sinzui: the last 2 licence review projects that I had waiting in there I was leaving to ask you stuff about them. one was eclisep but owned by us I think, and the other you'd left comments on the white board
<sinzui> czajkowski,  I approved both
<czajkowski> ok as I didnt know what the comments on the white board were referring to
<sinzui> the epl is legitimate. There was a branch but the user did not set it as the focus of development
<czajkowski> hence me leaving them, only noticed they were gone when I refreshed the page
<sinzui> czajkowski, the other project was warning in the past the NC was not open and free, their project is fine now that they picked a license from the know good licenses.
<czajkowski> nods it's hard for me to understand the comments as I dont know what is being referenced
<czajkowski> aso hard for you to know I'm doing the licence review either as I leave al my tabs open and review them all durin the day
<czajkowski> as going by the rotation maintence schedule
<sinzui> I panic when I see 5 tabs open
<czajkowski> sinzui: everyone has their own working method, I've a desktop for lp work and irc and mail, another desktop for browsing
<czajkowski> everying has it's place
<mrevell> salgado, How would you send the notification email to people's whose blueprints haven't migrated? I don't understand because in your email you say that it wouldn't work if you sent the notification from the script.
<salgado> mrevell, a separate, throw-away, script to send notifications to the owner/assignee of a list of BPs
<mrevell> salgado, Ah, I see.
<czajkowski> does that mean I'm going to get a ton of notifactions about bluerprints which are 4 cycles ago ?
<mrevell> czajkowski, I'm guessing only if you're the owner. IS that right salgado?
<salgado> only if you're the owner/assignee and the work items there can't be parsed
<czajkowski> not if you're subscribed to them
<czajkowski> ugh gonna create a filter now I'm gonna get a lotta mail
<salgado> and even then we might decide to completely ignore BPs older than, say, 6 months
<mrevell> czajkowski, Can you help salgado gauge how old a blueprint people in the Ubuntu community care about?
<mrevell> salgado, Wouldn't it be better to email only the blueprint owner, rather than the subscribers as well?
<salgado> mrevell, I don't have plans to email subscribers. I only considered owner/assignee
<czajkowski> hmm owner would make sense, but not all blueprints may have an assingee can go and find out
<salgado> if there's no assignee then just the owner
<salgado> I mean, if a BP has no assignee we notify only the owner
<czajkowski> salgado: and the driver?
<salgado> sure, we can notify the driver as well
<mrevell> Sounds good to me salgado
<czajkowski> cool
<salgado> mrevell, cool. I'll send you the notification text for review if that's ok with you?
<czajkowski> salgado: 15:48 < cjohnston> not all BPs use cycles... would be my only concern
<czajkowski> 15:48 < jussi> czajkowski: what cjohnston said.
<mrevell> salgado, Do you mind sending it to danhg?
<salgado> mrevell, not at all
<mrevell> salgado, I'd love to look too but it's his area now.
<danhg> Sending what?
<danhg> Sorry, I've not been following the thread.
<salgado> mrevell, btw, what about the announcement?  should I pester danhg about it as well?
<czajkowski> salgado: those were comments fro folks who use blueprints reguarly each cycle.
<salgado> czajkowski, right, and that means we may want to notify people about all WIs that can't be parsed and not just for the recent ones?
<mrevell> danhg, A notification to tell people that we tried to migrate their work items from the whiteboard on their blueprint to the work items box, but it didn't work. If you read up you'll see the conversation.
<mrevell> salgado, As for the announcement, yeah, I think danhg will handle that and I'll make sure he's up to speed.
<salgado> mrevell, cool!  any idea when we can have that?  the code is done and we're just waiting for the announcement/notification-script to get it deployed :)
<salgado> danhg, I guess you might be in a better position to answer that ;)
<czajkowski> salgado: nods
<mrevell> salgado, danhg and I have a catch-up now so I'll see how Dan's schedule is looking. It may be that it's quicker for me knock one out after that call.
<czajkowski> salgado: possibly again the lists and contacts I suggested before to be sent to as well
<mrevell> By "knock one out", I mean "carefully craft".
<salgado> czajkowski, I don't think sending 200+ messages to those folks/lists will help.  the whole point of sending to the owner/assignee/driver is to email only the relevant people
<salgado> the people who will know how to fix the WIs so that we can parse them
<czajkowski> hmm ok
<sinzui> I agree with salgado. only the assigned people can fix the data, and the data is not doing what they thought it was
<sinzui> salgado, I had two rounds of emails when hardening bugs and teams. We set a cut off date and fixed the remaining data to meet our deadlines
<salgado> sinzui, cool.  btw, what's the best way to write this one-of script to get a list of BPs and send a fixed notification to their owners/assignees/drivers?  can I do that as a script to be cowboyed into a LP tree and ran from there?
<sinzui> salgado, I pulled email addresses from staging and deduped them. I then wrote a script that sent the email though canonical.
 * sinzui looks for example script and data
<salgado> sinzui, from your own email address?
<sinzui> yes
<salgado> yeah, that's probably easier
<sinzui> salgado, I have a script that send an email to people found in a psql report. I did /massage/ the report to ensure no one got 2 emails. I think I had to research two email address for teams
<sinzui> I will sed you the script, with the example sql and output that was used to test and send the email
<salgado> sinzui, great, thanks a bunch!
<lifeless> sinzui: â¥ my hero
 * sinzui wonders what he has done
<lifeless> partial fix for person merge of proposed memberships
<sinzui> ah.
<lifeless> I say partial, because pg8.4 will let races exist still
<sinzui> I am writing the SQL to fix the remaining production data now.
<lifeless> but in practice we can probably ignore it, as long as once the merge is scheduled attempts to propose are rejected
<lifeless> do we guard new proposals (in either direction) against being done after a merge is requested ?
<sinzui> lifeless, we do not guard against new proposals. Merge does not need to know about it since the work is not calculated until the moment the merge happens. Since the merge process opens a cursor and starts updating and deleting, I assume it has a wrote lock preventing anyone from proposing a membership in the seconds needed to complete the merge
<sinzui> The team being merged has a warning on its page explaining the scheduled merge.
<sinzui> Someone with a stale browser could propose a merge the instant after a team is merged. It will error, just like it can happen now in cases like bug subscription
<lifeless> sinzui: inserting a new row into another table is protected in 9.1, not in 8.4
<lifeless> sinzui: here is a sequence that I believe will work in 8.4
<lifeless> A: begin
<lifeless> A: read person row
<lifeless> B: begin
<lifeless> B: read person row, insert proposed membership
<lifeless> A: write to person row (blocks on B, blocks new inserts of related rows)
<lifeless> B: commit
<lifeless> A: read proposed memberships - does not see the one from B
<lifeless> A: commit
<lifeless> sinzui: so I think we need to guard against new proposals after the merge is scheduled
<lifeless> sinzui: to be fully safe
<sinzui> why
<sinzui> lifeless, why. There is no db conflict.
<lifeless> sinzui: exactly
<lifeless> sinzui: if A is the merge job, and B is a stale browser POST request, we end up with bad data again.
<sinzui> lifeless, I do not write jobs that presupose work. The pending merge will still remove the pending membersip
<sinzui> membership
<lifeless> sinzui: no, because A *is* the merge
<sinzui> Then Lp should not let any user use any team that is in a pending merge job. This has nothing to do with pending memberships
<sinzui> This condition exists to day with merge proposals, drivers, maintainers, supervisors, contacts, subscriptions
<lifeless> sinzui: +1
<sinzui> No user has ever reported there is an issue with one user scheduling a merge and another user putting the same team in a new relationship
<lifeless> perhaps I'm paranoid.
<lifeless> I'm not asking you to do the work now or anything.
<lifeless> I am however sure that this race exists.
<sinzui> I can show you sql that might make you cry. It shows everything I know of that is superfluous  data left behind by merge and deactivation. I might even consider some of it a security risk, but I think sharing policies fix that
<lifeless> As long as we have a bug identifying this stuff, I won't cry. I might sob a little.
<sinzui> There are no comprehensive bugs describing that merged/deactivated users continue to have artefact subscriptions.
<lifeless> sinzui: may I impose on you to create one ?
<sinzui> We have about 12,000 rows of data we search work with to send emails, but we abort because the user/team is gnoe
<lifeless> sinzui: where is grackle at? Schema designed I know, but is it code-started? no-code-yet? ...
<sinzui> There is code. The client lib is 50% ready and the browser code is 90% there
<sinzui> No work on the server to show yet
<sinzui> I only work on it on Fridays
<lifeless> thats cool
<lifeless> I want to let -tech know about the use of cassandra
<sinzui> Actually, I failed to work on it last Friday. I owe it an day of labour?
<lifeless> don't fear the grackle
<abentley> deryck: There's no OCR.  Could you please review https://code.launchpad.net/~abentley/launchpad/upgrade-all-2/+merge/96247 ?
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<deryck> abentley, sure
<abentley> deryck: thanks.
<abentley> deryck: btw, how does one claim ec2 expenses?  I've never done it before.
<deryck> abentley, file expenses through canonical admin.  Send in copy of receipt to the email we normally send in receipts.
<deryck> abentley, and the charges are listed as "misc" on the expense report.
<deryck> abentley, and r=me on the proposal.
<abentley> deryck: thanks.  I will ec2 land it, of course :-)
<deryck> abentley, awesome :)
<wgrant> lifeless: Are you referring to predicate locking?
<lifeless> wgrant: SSIS or whatever it is
<wgrant> lifeless: Right, with predicate locking.
<wgrant> lifeless: To use that effectively we'd have to garboify the thing.
<wgrant> (which we possibly should anyway, but it's complicated)
<lifeless> wgrant: you're talking person merge?
<lifeless> wgrant: I thought we already did that in a Job
<james_w> hi wgrant, are you on the stakeholders mailing list?
<lifeless> james_w: which one
<james_w> LP\
<lifeless> I think he is,yes.
<wgrant> lifeless: Um, by garbo I mean looptune
<wgrant> lifeless: Currently it's in a single transaction, I think.
<lifeless> wgrant: I don't see the issue; as long as we do merge > Hard-timeout after any started transactions, and we reject use of to-be-merged persons, we're safe.
<wgrant> lifeless: It could take minutes.
<wgrant> lifeless: Keeping a serialised transaction open like that sounds dangerous.
<lifeless> wgrant: ehhh
<lifeless> wgrant: I think there is a massive comms fail happening here
<lifeless> and planet earth is blue
<wgrant> Oh?
<lifeless> I didn't suggest minutes, nor keeping a serialised xaction open the whole time
<lifeless> I didn't imply it either :)
<StevenK> lifeless: Can you untestable or something r14902? Please?
<wgrant> lifeless: Using SSI in something like the current implementation implies it
<lifeless> wgrant: I am not proposing SSI, I was saying in the absence of it ...
<lifeless> StevenK: Naturally. I was waiting for it to hit staging so it can be QA'd.
<james_w> wgrant, did you see the discussions around creating commercial PPAs?
<StevenK> lifeless: Bah, this whole devel versus db-devel thing.
<lifeless> so, staging is running
<lifeless> and has the right revno
<lifeless> checking logs
<wgrant> james_w: Yeah. IIRC we basically want to make commercial-ppa-uploaders a celebrity or add an extra field on distribution, which allows setting of commercial.
<wgrant> I suspect the latter.
<lifeless> 0.3 of a second
<lifeless> StevenK: having looked in backlog I see you pinged me last night; if it was on staging already you could have qa'd it - no need to block on me.
<StevenK> lifeless: I was implying that you should practise what you preach in terms of QA. :-P
<lifeless> StevenK: I preach that we should qa each others stuff
<wgrant> james_w: Are you looking at implementing it?
<lifeless> StevenK: and I qa other folks stuff, so  :-P
<james_w> wgrant, do we also want to make ~commercial-ppa-uploaders a private team, so that we don't have to worry about also being able to make archives private?
<james_w> (the team loaded for me today, so I was able to see that it is currently public)
<james_w> or maybe cleaning up Launchpad.CommericalAdmin would be a maintainance win that would offset the increased lines of code
<james_w> wgrant, it's a blocker to the work we want to do next, so yeah
<wgrant> james_w: Mmmm. That's a hard one.
<sinzui> wgrant, this appears to generate sane ids http://pypi.python.org/pypi/z3c.form
<wgrant> Anyone feel like reviewing https://code.launchpad.net/~wgrant/launchpad/html-wtf/+merge/96265?
#launchpad-dev 2012-03-07
<StevenK> I'm looking at it.
<StevenK> I'm vaguely disgusted by lines 70-84 of the diff
<StevenK> Oh, you're just moving code.
<wgrant> Yeah, moving body elements into the body :)
<StevenK> &lt;/script&gt;
<StevenK> Excuse me while I review my breakfast.
<wgrant> Thankyou Internet Explorer :)
<StevenK> wgrant: r=me
<wgrant> StevenK: Thanks.
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Firefighting: - | Critical bugtasks: 4*10
<StevenK> wallyworld__: I've handled PUBLICSECURITY -> UNEMBARGOEDSECURITY in my branch.
<wallyworld__> StevenK: aewsome, thank you :-)
<wallyworld__> StevenK: there's a card for that i think
<StevenK> 510 line branch
<StevenK> wallyworld__, wgrant: Either of you want to review https://code.launchpad.net/~stevenk/launchpad/move-to-informationtype/+merge/96271 ?
<wallyworld__> i can
<wgrant> StevenK: Why did you use sponge?
<wgrant> sed -i...
<wallyworld__> StevenK: btw, a branch of mine should be out of ec2 rsn and wil need to be merged into yours
<StevenK> Arse
<StevenK> wallyworld__: When is your branch due out of ec2?
<wallyworld__> StevenK: i ec2 landed it around 7am
<StevenK> My time or yours?
<wallyworld__> not should be done very soon i hope
<wallyworld__> s/not/so
<wallyworld__> my time of course :-)
<StevenK> wallyworld__: Run 'bin/ec2 ls' ?
<wallyworld__> up for 3:54
<StevenK> Right, it should land in the next 15 or so
<StevenK> wallyworld__: I'll hold off on the branch you approved, then.
<wallyworld__> StevenK: thanks, otherwise there will be conflicts
<StevenK> Wait for yours, merge, fix conflicts and re-check.
<wallyworld__> yes, plus rename new uses of AccessPolicyType
<wallyworld__> in tests etc
<StevenK> Right
<wallyworld__> there will only be a few, if any, can't remember
<StevenK> I was going to grep for it again
<wallyworld__> ok
<StevenK> I can share the one-liner I used, if you wish
<wallyworld__> ok
<wallyworld__> i don't use sed that much
<StevenK> wallyworld__: for f in $(bzr grep -l AccessPolicyType) ; do sed -e 's/AccessPolicy\(Type\)/Information\1/g' < $f | sponge $f ; done
<wallyworld__> i love a got bit of regex
<wgrant> bzr grep -l AccessPolicyType | xargs sed -i -e ''s/AccessPolicyType/InformationType/g'
<wallyworld__> good
<StevenK> Bah, xargs
<wallyworld__> i like xargs better too
<wallyworld__> what does sponge do?
<StevenK> wallyworld__: sponge soaks up all standard input and then writes it to a file
<StevenK> wallyworld__: If you do 'sed < file > file' what happens is the shell opens file for writing with truncate, so when sed gets a hold of it, it's empty
<wallyworld__> oh, interesting
<StevenK> wallyworld__: http://pastebin.ubuntu.com/872355/
<wallyworld__> right
<wallyworld__> StevenK: merge done
 * StevenK updates his devel with fear and trepidation
<wallyworld__> StevenK: if you get yours into ec2 within say 30 minutes or a bit longer, you'll make the next bb run
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bug-869631/+merge/96280
<StevenK> wgrant: Excellent work, r=me
<wgrant> Thanks.
<StevenK> wallyworld__, wgrant: Either of you want to review https://code.launchpad.net/~stevenk/launchpad/drop-garbo-accesspolicy/+merge/96285 ?
<wgrant> StevenK: Weren't there additional product and distribution grants to remove security.cfg?
<wgrant> This should probably just be a straight antimerge of the rev.
<StevenK> It was
<StevenK> wgrant: bzr di -r 14900..14901 doesn't show the distribution/product addition
<wgrant> StevenK: Hm, I thought it was in the original MP. Maybe it isn't any more.
<wgrant> Or it was added with the workitem migrator
<wgrant> That's more likely.
<StevenK> I remember adding it
<StevenK> wgrant: Removed them manually
<wgrant> StevenK: You'll probably break the workitem migrator.
<StevenK> "Good" ?
<wgrant> Heh
<wgrant> Yes, r14893 (the workitem migrator) added those two
<wgrant> Which is why they're not in your diff.
<StevenK> Ah
 * StevenK wonders if bzr uncommit and push --overwrite will cause issues
<wgrant> Will work fine.
<StevenK> wgrant: Since we've just spent 15 minutes talking about it, do you want to rubber stamp it?
<wgrant> If X stops crashing.
<StevenK> Unlikely.
<wgrant> At least with -radeon it doesn't hang the kernel.
<StevenK> I suggest NVidia.
<wgrant> Is approved.
<StevenK> wgrant: Thank you.
<wgrant> Nah, these crashes are unrelated to the drive.
<wgrant> r
<StevenK> I can see it conflicting horribly with my branch that is in ec2 now, but meh
<StevenK> wgrant: Can you explain the trigger stuff you mentioned on the call this morning?
<wgrant> StevenK: You may not need a trigger, but it's possibly easiest.
<wgrant> StevenK: We need to migrate from the two flags to the enum.
<wgrant> There are multiple triggers and bits of application code that use the flags.
<wgrant> The easiest way to migrate may be to map flag changes to enum changes in the DB.
<StevenK> So I add a bit to bug_mirror_legacy_access ?
<nigelb> Where is bigjools? Apartment hunting?
<bigjools> sat here!
<StevenK> He's teasing me in another channel, at the moment.
<wgrant> StevenK: You could potentially work it into that.
<bigjools> I was straight-batting your troll back
<wgrant> StevenK: It's not clear how it fits into bugtaskflat.
<wgrant> StevenK: (which isn't landed yet, partly for this reason)
<StevenK> Mmmm
<StevenK> So we can ignore the trigger thing, and do it in code+garbo
<StevenK> Which allows bugtaskflat in
<wgrant> Our data access layer is sufficiently chaotic that that is normally impractical.
<wgrant> But for this limited case you may be able to do it.
<wgrant> Since the two flags already have custom behaviour.
<StevenK> Right.
<wgrant> bugtaskflat may continue to have the private flag, I suspect.
<wgrant> Calculated from whether the type is private or not.
<wgrant> But it may also want to have a copy of the type.
<StevenK> I suspect it would do better with just a copy of the type
<StevenK> Then Bug.private and Bug.security_related can just *die*
<StevenK> And good riddance
<wgrant> Right, those two certainly die.
<wgrant> But for performance and sense it may be best for BugTaskFlat.private to still exist.
<nigelb> bigjools: HA. I did not se that reply until now :)
<bigjools> nigelb: :)
<StevenK> wgrant: I will ignore triggers for now and deal with updating information_type in my model branch
<wgrant> StevenK: Sounds reasonable.
<wgrant> I guess it's only the appservers that really matter here.
<StevenK> ENOSTUB though
<StevenK> lifeless: Are you happy to review this DB patch, or do I get to wait until tomorrow afternoon?
<wgrant> lifeless: Also, want to request a deploy of your DB patch?
<lifeless> StevenK: stub should be around soon, I may have a look if you toss me a url
<lifeless> wgrant: good idea! want to toss it up?
<StevenK> lifeless: stub will not be around. It's a public holidays in .th today.
<wgrant> lifeless: Sure
<lifeless> wgrant: thanks
<lifeless> StevenK: ah, so what is the url?
<StevenK> lifeless: Still sorting out the MP, give me a few.
<StevenK> lifeless: https://code.launchpad.net/~stevenk/launchpad/db-add-information_type/+merge/96290
<lifeless> we should probably deprecate comments
<lifeless> do them in the patches
<StevenK> lifeless: I think that's probably orthogonal :-)
<lifeless> anyhow, you tried to use a default, which you can't.
<wgrant> Well
<wgrant> You can add a default afterwards
<wgrant> But you can't set a default on create
<wgrant> Should be able to do it in the same patch, no?
<lifeless> alter table add column foo integer default 1 -> rewrites the heap.
<lifeless> alter table add column foo integer; alter table alter column foo set default 1; -> ok.
<lifeless> (not-quite-sql)
<wgrant> Right.
<wgrant> That's what I Meant.
<lifeless> StevenK had the former
<wgrant> Also, comments don't take locks, so we can keep comments.sql if we want
<lifeless> wgrant: meh.
<lifeless> wgrant: one system to rule them all
<StevenK> Just thinking about it, the default is dangerous, so I'll remove it
<wgrant> Hmmm, I think xx-builder-page.txt might get into an infinite redirect loop with the broken <noscript>...
<wgrant> StevenK: We already default private to false.
<wgrant> IIRC
<wgrant>  private                | boolean                     | not null default false
<StevenK> wgrant: Right, but DB patch gets deployed with information_type default to 1, and someone files a private bug. It's now private = true and information_type = 1
<wgrant> Sure, we need to run garbo and then verify.
<lifeless> I prefer no default
<lifeless> for that exact reason
<StevenK> I agree
<wgrant> This is going to be a common thing with migrations, but sure.
<StevenK> I'm happy to add a default when there is code to deal with the above case.
<StevenK> lifeless: Diff updated
<lifeless> wgrant: it is, I think the defaults should be added late or never
<lifeless> wgrant: -> because the code to sync them with other things can only be added after the columns
<lifeless> wgrant: -> and the way to do a write-through-check e.g. garbo migrator, is to look for unset values
<wgrant> lifeless: In some cases.
<wgrant> Sometimes that's not possible.
<lifeless> do you think there is a better sane-default-position ?
<wgrant> I guess looking for unset values works for this case, so we can do that.
<lifeless> so, if you think we should recommend something else, lets do that
<StevenK> Was my plan
<lifeless> I totally ack that there may be different cases out there
<lifeless> and am v. happy for folk to do whatever is needed; I'd like the common case to be what we encourage
<lifeless> which, AFAICT, is add without default and use null to detect unset [excluding new FK's]
<StevenK> lifeless: Can haz another look?
<StevenK> wgrant: Have you seen http://pastebin.ubuntu.com/872533/ before?
<wgrant> StevenK: It happens one in a couple of hundred runs.
<wgrant> Maybe less
<wgrant> But it's not unheardof.
<StevenK> I'll ignore it then
<StevenK> My move-to-informationtype branch has tripped over it.
<wgrant> Debugging hanging doctests makes me sad.
<wallyworld__> StevenK: how far away is your enum branch?
<wallyworld__> ah, bb. are you going to land it manually?
<StevenK> It just failed ec2.
<wallyworld__> ok. i have a new branch that will conflict with it. i'll wait till yours in merged before landing mine
<StevenK> And we're in testfix
<StevenK> For the love of ...
 * StevenK marks it testfix because buildbot is so crap
<wallyworld__> tsk tsk
<StevenK> If I would be allowed to actually move us to Jenkins, we wouldn't have to do this crap
<wallyworld__> yep :-(
<wgrant> StevenK: You must never mark a non-testfix branch as testfix
<wgrant> Force the buildbot build.
<StevenK> And then wait twelve hours instead of six
<StevenK> wallyworld__: r14911, and I'm a bad man.
<wgrant> In either case you'll be asleep.
<wallyworld__> StevenK: thanks. and yes, you are very naughty and must be punished
<StevenK> I should look at writing a fixture for convoy for use in buildbot and ec2. That sounds fairly punishing.
<wgrant> Well
<wgrant> It probably wants to be a layer.
<wgrant> Using a fixture.
<bigjools> there's a thing for that
<adeuring> good morning
<rick_h_> morning
<czajkowski> rick_h_: good day to you sir
<rick_h_> gah, let the email catch up begin!
<deryck> adeuring, ping for standup.
<adeuring> deryck: whoops, coming..
<abentley> deryck: I think my --dry-run output looks right, but could you check? https://pastebin.canonical.com/61826/
<deryck> abentley, looking....
<deryck> abentley, yeah, looks fine to me.
<abentley> deryck: cool, thanks.
<salgado> mrevell, hey there. did you get a chance to work on that announcement?
<abentley> deryck: My mail issue appears to be because ec2 doesn't copy my smtp credentials to the instance.  But Canonical smtp requires credentials, so I don't understand how this works for anyone.
<deryck> abentley, hmmm, yeah, I'm not sure either.
<deryck> abentley, maybe gary_poster has an idea about how the mailing works?
<abentley> gary_poster: ping :-)
<gary_poster> abentley, :-) I'm reading backlog
<gary_poster> um
<gary_poster> that's a long time ago...
<gary_poster> thinking
<rick_h_> abentley: the only thing I recall is getting my smtp creds into my bazaar.conf on default
<rick_h_> my only other issues were gpg related, not email side
<abentley> rick_h_: Are they in bazaar.conf or authentication.conf?
<rick_h_> bazaar.conf
<rick_h_> abentley: I don't have any email settings in authentication.conf
<abentley> rick_h_: mine is in authentication.conf, so maybe that's why ec2 doesn't see it.
<rick_h_> yea, I've got smtp_username/server/password in bazaar.conf and I've never had email issues with ec2
<sinzui> rick_h_, I too had to add smtp_* which also requires me to set my address to the one provide by canonical. I then had to visit location.conf to put my email address back for non-canonical projects
<abentley> sinzui: Well, you don't technically *have* to use a smtp.canonical.com to send mail from a canonical.com address :-)
<gary_poster> abentley, confirmed that my mail configuration is in bazaar.conf.
<sinzui> abentley, are you running your own smtp?
<gary_poster> I think some docs (used to?) advise this
<sinzui> gary_poster, yes, the new launchpader setup and rocketfuel setup pages both recommended it
<gary_poster> yeah
<czajkowski> ok hands up who understand private bugs as I'm a bit confused (yes I know it happens a lot)
<abentley> gary_poster: I believe authentication.conf is the preferred method within the Bazaar community, so that's what I've used.  It would be nice to support everything Bazaar does.
<abentley> gary_poster: I'm a bit lost trying to figure out how the credentials are transmitted to the test instance.
<czajkowski> trying to work out if a person should see a private bug if they are part of a project or not
<abentley> sinzui: yes, I am running my own smtp.
<gary_poster> abentley, this code has changed far, far from what I wrote.  I would look for what is copying over bazaar.conf and have it also copy over authentication.conf, perhaps
<gary_poster> czajkowski, I don't but people on sinzui's squad will, and deryck and gmb probably will, among others (whee, ping fest, hi everybody!)
<gmb> Wat?
 * gmb backscrolls
<czajkowski> gary_poster: good day :)
<gary_poster> :-)
 * deryck looks up ???
<sinzui> czajkowski, no, not today
<gmb> czajkowski: sinzui and squad are your folks for that .
<gary_poster> gmb, are you willing to admit knowledge of private bugs?
<sinzui> Only subscribed users/teams/bots can access a private bug.
<czajkowski> gmb: wondering does a get to see private bugs if they are in a project that the private bug has been reported on
<abentley> gary_poster: cool.  I found it.
<gmb> gary_poster, Not any more, unless sinzui hasn't been a-changing all the rules, in which case those were some very strong drugs I took.
<gary_poster> deryck and gmb, sorry, was just thinking bug thoughts
<gary_poster> heh
<czajkowski> https://answers.launchpad.net/launchpad/+question/188450
<czajkowski> is why I am asking
<czajkowski> sinzui: *hugs*
 * gary_poster runs away
<gmb> Why I can't open a private bug that affect PlayOnLinux project?
<gmb> Thanks in advance,
<gmb> Aymeric.
<deryck> heh.  I shall ignore you all.
<gmb> WTF?
<gmb> OIC.
<gmb> Sorry. C+P spam
<sinzui> czajkowski, For 5 months, we allows the project maintainer to access all private bugs, but we are now transitioning to sharing. In two weeks, we might restore project-level access to private bugs
<gmb> czajkowski: Ah, I filed a bug about private things returning an OOPS with their 404, which I think is wrong. There may have been some discussion about this.
 * czajkowski hands gmb a very large coffee
<gmb> You saw what the nespresso machine did to me, let's not go down that road again.
<sinzui> gmb: lifeless dupe that bug after we investigated and found that the bug was targeted to a deactivated project
<gmb> Ah.
<czajkowski> sinzui: gotcha so right now, unless they are the project maintainer they cannot see the private bugs
<gmb> That's a merry can of worms.
<sinzui> czajkowski, no
<sinzui> The user must be subscribed to the bug
<sinzui> no exceptions this week
 * gmb goes back to futzing around with packaging, which is marginally less fun than poking a knitting needle into one's eye to see what happens.
<sinzui> czajkowski, commercial projects *had* an exception for 5 months
<czajkowski> sinzui: gotcha but it's not a commerical project.
<sinzui> okay. then subscription was the only way to get access to a private bug or branch for that matter
<stokachu> does the oauth_tokens expire on staging?
<sinzui> stokachu, they are no different from production...but staging's database it replaced almost every Saturday
<stokachu> sinzui: ok
<mrevell> salgado, flacoste: I'm in the Product channel on Mumble.
<mrevell> salgado, flacoste: https://dev.launchpad.net/Projects/WorkItems/
<flacoste> salgado: http://bazaar.launchpad.net/~canonical-launchpad-branches/lp-dev-utils/trunk/view/head:/spam.py
<stokachu> what api call(s) should i be using to grab a list of bugs (public/private) that are assigned to me?
<stokachu> is al lthat done through collections link?
<stokachu> do i just query a collection of bugs and find their owner link?
<sinzui> stokachu, this is an example in Python: http://pastebin.ubuntu.com/873230/
<sinzui> The documentation is https://launchpad.net/+apidoc/devel.html#has_bugs
<stokachu> ok thanks ill look at that
<stokachu> ah makes sense, sinzui thanks :)
<abentley> sinzui: Now that ec2test is standalone, it requires the python-tz package to be installed as a deb.  Can we add it to launchpad-developer-dependencies?
<sinzui> abentley, we can. I thought it was indirectly in the deps though. I have always had the package without the need to install it
<stokachu> sinzui: im doing this in php and has_bugs mentioned underlying entry, where does this entry get called from?
<abentley> sinzui: I did have to install it.
<abentley> sinzui: But I don't have launchpad-developer-dependencies installed right now, because it says rabbitmq-management is broken.
<sinzui> stokachu, Launchpadlib maps the entity data from Lp to the WADL definition. It essentially instantiates a generic object and adds the properties and methods.
<sinzui> abentley, that is true...I have pinned my rabbit libs so that I can keep the meta packages installed
<sinzui> wgrant, has some experience packaging rabbitmq for Lp. I do not know why we cannot use what Ubuntu provides
<stokachu> sinzui: ok but i can't pull objects through api interface over php
<sinzui> stokachu, LPAPI does not provide objects. It send a json data about the object. One of the json properties maps to type defined in the WADL.
<stokachu> ok
<stokachu> sinzui: so that brings me back to calling the api for all bugs and breaking it down through there?
<stokachu> i see the resource_type_link in the properties
<stokachu> So my entry point looks like it would be api/bugs, grab the bug_tasks_collection_link, and there i can capture the assignee link
<sinzui> stokachu, resource_type_link points to the WADL that states what you can do with the resource. The resource has a self link. posting data to that self link will return another entity or maybe return a collection of entities.
<sinzui> stokachu, you might be able to search all bugs, but I do not think you should. There will be a million bugs in a few months. Your requests will timeout out or if your script DOSes Lp, it will be blocked
<stokachu> yea i wasn't planning on searching all bugs
<stokachu> but im still a little confused on what entry point to use.. if i pull a person collection there are no resources there that describe what bugs i may be assigned to
<stokachu> pulling a bug collection pulls all bugs and thats not what we want
<stokachu> and in order to get a bug tasks collection you need to know the bug id
<stokachu> looks like launchpad lib gives you the ability to searchTasks which meets a certain criteria
<stokachu> the top level bugs collection doesn't give me the ability to define criteria
<stokachu> sinzui: in your example you do a lp.people['registry']
<stokachu> what is registry?
<sinzui> ~registry is a team I am a member of.
<sinzui> Teams and users hare about 75% of the same methods and properties
<stokachu> ahh.. maybe i should think in terms of teams rather than a single person
<stokachu> so when you return lp.people['registry'] what is providing you with searchTasks
<stokachu> ah team..
<stokachu> sinzui: now i see...
<stokachu> team->searchTasks
<stokachu> ok ok.. sinzui think i understand now, ill leave you alone :)
<sinzui> stokachu, Lp uses person to mean team or user. They are often interchangable. We use "user" or "team" only when we are doing something specifically to the type. Teams have members, users have ssh keys for example. the API webdoc also confuses users and teams. Both are Person and you can do most things operations without needing to check which type you got back from Lp
<stokachu> sinzui: yea the whole 'user' 'team' thing was giving me a runaround
<stokachu> like in the sub_teams_collection_link for person collection it says lists 'subteams of this team'
<stokachu> so that is veryyyy confusing
<stokachu> for clarity once i retrieve my person collection i can do api.launchpad.net/~adam-stokes?ws.op=searchTasks?
<sinzui> stokachu, I think you need to post to an API version. Maybe api.launchpad.net/devel/~adam-stokes?ws.op=searchTasks
<stokachu> sinzui: this is my url https://api.staging.launchpad.net/1.0/~adam-stokes?oauth_consumer_key=drupallp&oauth_nonce=517b21443bea00d082c9961b40c79bb6&oauth_signature=%26mZQnG1jG85KLcn1j1p6Q4SZf5c4CcpWzJbcrvW2GX2xS2gqnmZCm6lSw7Q10gfc5m1zG9zr8QZMTp6rp&oauth_signature_method=PLAINTEXT&oauth_timestamp=1331145573&oauth_token=6Srv0p37zxM4k1fMqBrr&oauth_version=1.0&ws.op=searchTasks
<stokachu> it looks like its a GET request from the documentation
<stokachu> crap
<stokachu> i thought i scrubbed the tokens
<stokachu> lmao
<stokachu> ill revoke it in a few
<stokachu> sinzui: so with the above url it just returns false
<stokachu> i didnt see any other requirements for this call either
<sinzui> stokachu, I think the oauth cannot be in the URL. The method you are testing can be used anonymously (only public data is returned): eg, you can see me at https://api.launchpad.net/devel/~sinzui?ws.op=searchTasks
<sinzui> I have a lot of bugs
<stokachu> ok, what about accessing private bugs?
<sinzui> Your request must have a cookie issued by oauth
<sinzui> You can see my assigned bugs with this query direct request https://api.launchpad.net/devel/~sinzui?ws.op=searchTasks&assignee=https://api.launchpad.net/devel/~sinzui
 * sinzui has no private bugs assigned
<stokachu> sinzui: so in my headers i have the authorization set
<stokachu> ok so.. i think i see, my get requests shouldnt be putting in my oauth data because its in the authorization header
<stokachu> lemme try something
<sinzui> stokachu, I think the process LPLib process is start engine, call login, check credentials+host to get session cookie info, do oauth to get session cookie info, make request with cookie info and go post/get to url
<stokachu> yea i think thats what im doing wrong..only keep the authorization in the header and just make the requests
<salgado> jcsackett, around?  would you mind ec2-landing https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-widget/+merge/94790?  the announcement is in the works so we can expect it to be ready by the time this reaches staging and is QAed
<salgado> sinzui, maybe you can do that for me? (^)
<sinzui> salgado, I will do that now
<salgado> sinzui, yay, thank you!
<StevenK> lifeless: Can *please* haz another look at my DB patch?
<sinzui> StevenK, wgrant: can either of you review this and +1  it http://pastebin.ubuntu.com/873679/
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<sinzui> wallyworld__, http://bazaar.launchpad.net/~sinzui/essence-contact/trunk/view/head:/essence/test_contact.py
 * wgrant is still shocked by seeing bigjools commenting on bugs at odd hours
#launchpad-dev 2012-03-08
<lifeless> StevenK: I -really- prefer stub to get a lookin with the narrow exception of him being away for an extended period; he is back today.
<StevenK> lifeless: I would have really prefered to already have it landed, but you disappeared yesterday afternoon. Fine, I will wait for stub.
<lifeless> StevenK: I EOD'd at 1700 here :)
<lifeless> StevenK: and cynthia has a cold, so it really was EOD
<lifeless> StevenK: in any event, I would have said the same thing yesterday
<lifeless> so bigjools, tell me about messaging servers
<StevenK> Do we have some report that reports if any patch in db-stable/devel isn't applied to prod?
<lifeless> no, but we have a bug asking for one
<StevenK> wgrant: Product done. Would about Distribution?
<wgrant> StevenK: Yes
<StevenK> Bah, okay.
<wgrant> You expected otherwise?
<StevenK> I didn't, actually.
<StevenK> wgrant, wallyworld__: https://code.launchpad.net/~stevenk/launchpad/product-distribution-accesspolicy/+merge/96499
 * wallyworld__ has a look
<wallyworld__> StevenK: you probably also need to create a proprietary policy when a product has a commercial subscription created for it
<StevenK> Hmmm
<StevenK> wallyworld__: So I'm not sure if products go from non-commercial to commercial, or if they are created as commercial
<wallyworld__> StevenK: i *think* they are created as non commercial and then the subscription is added by an admin
<wallyworld__> ?
 * StevenK can't see what method on IProduct is called for that.
<wallyworld__> i don't think it's on IProduct
<wallyworld__> StevenK: didn't you add a factory method to create a subscription?
<StevenK> That just creates a CommercialSubscription
<StevenK> We don't have CommercialSubscriptionS{et,ource}, just it.
<wallyworld__> i think redeemSubscriptionVoucher does it perhaps?
<wallyworld__> which is on IProductPublic
<StevenK> Have you managed to read that function without your eyes bleeding?
<wallyworld__> not yet
<wallyworld__> yes, it looks like that is the one
<StevenK> Right
<StevenK> The garbo job doesn't handle that process either
<wallyworld__> not sure what to do when the subscription expires though. do we remove the policy?
<StevenK> Right, that's the tricky question.
<wallyworld__> wgrant: ? ^^^^
<wgrant> wallyworld__: We don't know how that will work yet.
<wgrant> wallyworld__: This is just for the initial migration.
<wallyworld__> ok, just checking :-)
<wgrant> At what stage we create the policies once the New World is complete, we don't know.
<bigjools> lifeless: helleau
<lifeless> bigjools: hellauo
<bigjools> like the accent
<bigjools> what do you want to know then?
<lifeless> were they useful, what interface did they have, would you use one again, and whatever you think I should know about them
<wallyworld__> wgrant: so, for StevenK's branch, do we want to add a proprietary policy when a voucher is redeemed?
<wgrant> wallyworld__: No, proprietary doesn't exist yet.
<wgrant> wallyworld__: We'll redefine at what point any policies are created at all later on.
<wgrant> The first step is to get bugs onto the new model, which just needs these two created at all times.
<wallyworld__> ok. the current work i've done in the ui shows proprietary as an option in the disclosure picker for commercial projects
<wgrant> This is just a dirt cheap way to achieve that.
<lifeless> bigjools: can go voice if that would be easier ;)
<bigjools> lifeless: can do
<bigjools> lifeless: I'll make it quick - I am starving :)
<lifeless> skype/hangout?
<wallyworld__> StevenK: r=me
<bigjools> lifeless: I'll skype you
<bigjools> lifeless: well I would do if you were on
<bigjools> ah skype is being crap
<lifeless> I despair about my ADSL
<bigjools> I despair about random desktop lockups
<StevenK> lifeless: In your case, the A is not Asynchronous, but Abominable.
<bigjools> lifeless: frozen
<bigjools> lifeless: for f....
<bigjools> lifeless: reliable internets in NZ then?
<lifeless> will be if you answer, switching to tablet's skype rather than faildsl
<bigjools> lifeless: not geting your call
<bigjools> tried to call you and you didn't answer either :)
<wgrant> StevenK: Any test failures from that branch yet?
<wgrant> StevenK: I'd expect some in test_bug_mirror_legacy_access_triggers
<bigjools> wallyworld__: are you at home?
<wallyworld__> yeeees?
 * wallyworld__ is nervous now
<bigjools> wallyworld__: eeeexcellent, can I pop round in a bit and print and scan sumting pleez :)
<wallyworld__> it will cost you
<bigjools> wine?
<bigjools> I can pay in wine
<wallyworld__> we'll discuss it
<bigjools> ooerr missus
 * bigjools out
<stokachu> could someone check out my headers and see if im missing anything when doing an authenticated call to me?ws.op=searchTasks? http://pastebin.ubuntu.com/873995/
<stokachu> it times out, but doing an anonymous call works
<stokachu> i dont have OAuth realm="https://api.launchpad.net/" but that is optional according to the oauth specs
<wallyworld__> wgrant: StevenK: what would i need to do to get a review of a branch which merely renames stuff, but the line count is high
<lifeless> timeout -> do you get an OOPS in the response headers ?
<StevenK> wallyworld__: Beg.
<stokachu> lifeless: yea
<lifeless> ... which is ?
 * wallyworld__ is on his knees
<wallyworld__> https://code.launchpad.net/~wallyworld/launchpad/fix-sharing-terminology-948083/+merge/96506
<stokachu> lifeless: one sec :)
<stokachu> lifeless: apparently a code update is in progress at this very moment
<stokachu> im not sure how long those take
<lifeless> stokachu: you're testing on staging ?
<stokachu> lifeless: yea
<lifeless> try qastaging
<stokachu> ok
<lifeless> it should be up
<stokachu> lifeless: OOPS-7f4c62461a1ee30fcb05df7f03a8a3ef
<stokachu> lifeless: fwiw i just tried on production and it worked
<wgrant> Heh, GitHub seems to have lost all their CSS
<stokachu> they had ssh problems earlier
<wgrant> stokachu: Bug searches except in small projects will tend to time out on staging.
<StevenK> wgrant: So I have an attribute that requires view to see and edit to change. Do I need a decorator for that?
<wgrant> And qastaging.
<wgrant> StevenK: That applies to most attributes in Launchpad.
<wgrant> StevenK: It goes on IBugView
<wgrant> We don't usually use interfaces to specify edit permissions.
<stokachu> wgrant: ok, is this not the correct approach i should take then?
<StevenK> wgrant: Okay, so the IBugEdit I'm making is pointless?
<wgrant> StevenK: Although in some simple cases (eg. if launchpad.Edit grants edit privileges on everything that launchpad.View can see) you can use interfaces.
<wgrant> StevenK: IBugEdit would conventionally be things that you require launchpad.Edit to see.
<wgrant> stokachu: I don't know... what are you trying to do? :)
<StevenK> wgrant: There's a bunch of methods you need .Edit to call
<stokachu> wgrant: im just trying to get my profile and see bugs (public/private) im assigned to
<wgrant> StevenK: Then they belong on IBugEdit
<stokachu> so far ive just been investigating searchTasks
<wgrant> Also, it should be illegal to have two people talking with the same first two letters.
<stokachu> then ill append like &assignee='adam-stokes' ?
<wgrant> One of you needs to go awawy :)
<stokachu> LOL
<stokachu> it is getting late i could go play some dawn of war
<wgrant> stokachu: So, if you just want assigned bugs, specify assignee=, yeah
<StevenK> Let's get stub in here too!
<StevenK> Then we can have three!
<wgrant> stokachu: By default it will do things like search by bugs you've commented on.
<wgrant> stokachu: Which can be very very slow.
<stokachu> wgrant: ah ok
<lifeless> stokachu: so yeah, thats a simple code/performance issue for us
<stokachu> so with assignee can it be just my name? like searchTasks&assignee=adam-stokes
<lifeless> stokachu: however
<lifeless> User: Anonymous User
<lifeless> DB id: Anonymous
<wgrant> stokachu: Probably needs to be assignee=/~admin-stokes
<lifeless> -> it hasn't logged you in
<wgrant> Er
<wgrant> s/admin/adam/
<stokachu> wgrant: ok :)
<stokachu> lifeless: ah i didn't get new credentials on qastaging
<stokachu> can never have to many staging servers i guess :)
<lifeless> stokachu: so you probably should if you want confirmation that the auth works ;)
<stokachu> lifeless: haha ok ill re-auth, in your opinion is it better to test against qastaging or staging?
<stokachu> i dont want to anger the gods by messing with production
<lifeless> you can use either
<stokachu> ok cool.. thanks for everyones help :D
<StevenK> wallyworld__: I think this branch I'm in the middle of will turn out bigger.
<wgrant> wallyworld__: addPillarSharee -> sharePillarInformation?
<wgrant> I've already tried to make nouns out of these; it doesn't work well :)
<wallyworld__> wgrant: that sounds reasonable
<wallyworld__> i have bigjools here at the moment
<StevenK> wallyworld__: Why did you remove 2011 from the copyright in sharingservice?
<wallyworld__> because i only started that file in 2012
<StevenK> But it's a renamed file :-)
<wallyworld__> the original file was started in 2012 also
<StevenK> Not according to the copyright :-)
<wallyworld__> it was just a drive-by
<wallyworld__> the copyright was originally wrong
<wallyworld__> cut and paste
<wallyworld__> from elsewhere
<StevenK> Fine
<StevenK> I was mostly trolling
<wallyworld__> :-)
<StevenK> Hmmmm.
<StevenK> A bunch of methods that are in IBugEdit are mutators for properties in IBugView.
<StevenK> I might split them all out into IBugPrivacy
<wgrant> confused
<wgrant> StevenK: What?
<StevenK> wgrant: So private and security_related are defined in IBugView, but there are few methods in IBugEdit that opperate on them
<StevenK> Same for duplicateof, which is in IBugView, but markAsDuplicate is in IBugEdit and marked as a mutator of it.
<wgrant> StevenK: Right, that's normal.
<wgrant> Or does lazr.restful really not like cross-interface mutator_fors?
<wgrant> It should work...
<wgrant> As long as you define them in the right order/
<StevenK> wgrant: pyflakes is complaining
<wgrant> Oh?
<StevenK> lib/lp/bugs/interfaces/bug.py:840: undefined name 'private'
<StevenK> lib/lp/bugs/interfaces/bug.py:841: undefined name 'private'
<StevenK> And so on
<wgrant> Well, yes, that's not going to work.
<wgrant> IBugView['private']
<wgrant> This is Python, not magic :)
<nigelb> haha
<StevenK> But, it should be!
<wallyworld__> bigjools has left the building
<StevenK> Right, fixed.\
<StevenK> So now pyflakes is happy.
 * StevenK tries 'make'
<StevenK>     @mutator_for(IBugView['private'])
<StevenK>   File "/home/steven/launchpad/lp-sourcedeps/eggs/zope.interface-3.5.2-py2.6-linux-x86_64.egg/zope/interface/interface.py", line 552, in getDescriptionFor
<StevenK>     raise KeyError(name)
<StevenK> KeyError: 'private'
<wgrant> IBugPublic['private']
<StevenK> Oh damn it, I should have realised that myself.
<bigjools> wallyworld__: back now and I can't stop thinking about your special offer
<StevenK> Oh, bwahaha
 * StevenK tries VERY hard to not quotefile that comment
<StevenK> wgrant: http://pastebin.ubuntu.com/874044/ Zope hates me
<wallyworld__> bigjools: i've been thinking about you too
<bigjools> and other things beginning with w
<stokachu> watermelon!
<wgrant> StevenK: One of your interfaces probably inherits another one.
<wgrant> StevenK: That is wrong.
<wgrant> And bad and evil.
<wgrant> And won't even work.
<StevenK> I don't think
<StevenK> so
<StevenK> wgrant: http://pastebin.ubuntu.com/874045/
<wgrant> Perhaps the wrong interface is inherited IPrivacy?
<wgrant> StevenK: Recall that one of those interfaces redefines bits of IPrivacy
<wgrant> So declaring access on both won't work.
<StevenK> Right, and linked_branches is probably in the exactly same boat
<wgrant> I would assume so.
<StevenK> wgrant: Working, but I'm not sure about IBugPublic inheriting from IPrivacy.
<wgrant> It's fine if it's OK.
<stokachu> wgrant: are you or anyone else able to see private bugs assigned to me specifically?
<stokachu> i keep pulling up 0 bugs
<wgrant> stokachu: How are you filtering to private bugs?
<stokachu> 'request' => 'GET /1.0/~adam-stokes?ws.op=searchTasks&assignee=https%3A%2F%2Fapi.launchpad.net%2F1.0%2F%7Eadam-stokes
<stokachu> thats my urlencoded request being made, after ive retrieved my oauth token
<stokachu> http://pastebin.ubuntu.com/874054/ heres my full response request
<wgrant> stokachu: https://launchpad.net/api/1.0/~adam-stokes?ws.op=searchTasks&assignee=https%3A%2F%2Flaunchpad.net%2Fapi%2F1.0%2F~adam-stokes shows a private bug for me (it uses normal webapp cookies, so you can use your browser rather than an OAuth client)
<wgrant> So I suspect you're not authing properly.
<wgrant> stokachu: Alternatively, are you sure you authorized your token for private data?
<stokachu> hmm I have a header defined with my Authorization credentials
<stokachu> i used the change anything button
<wgrant> That should be fine.
<wgrant> What happens if you just "GET /1.0/~" with appropriate auth headers?
<wgrant> Hm
<wgrant> I'd expect the Authorization header to start with 'OAuth ', I think.
<stokachu> so OAuth realm is not optional?
<StevenK> wgrant: I'm getting a bunch of ForbiddenAttribute failures, including one that is in the ZCML, but not the interface.
<stokachu> http://pastebin.ubuntu.com/874061/ here is the response to a /~
<wgrant> stokachu: That's a response to /~adam-stokes
<wgrant> stokachu: Which?
<wgrant> Bah
<wgrant> StevenK: Which?
<StevenK> Oh, hahaha. maybeConfirmBugtasks is listed in the ZCML, not in the interface, but is defined in the model.
<wgrant> stokachu: 'OAuth realm' isn't an atom. 'OAuth' is the authorization scheme, realm is a field like oauth_nonce
<StevenK> wgrant: ForbiddenAttribute for things like name, tags, date_last_updated
<wgrant> You may be able to omit the realm field
<wgrant> StevenK: On what operation?
<StevenK>     current_bug.date_last_updated = now
<StevenK> ForbiddenAttribute: ('date_last_updated', <Bug at 0xcefb390>)
<wgrant> StevenK: Right, you need to define set_schema or set_attributes
<stokachu> wgrant: http://pastebin.ubuntu.com/874063/ :(
<wgrant> stokachu: That's more like it
<StevenK> wgrant: I should keep the enormous list of set_attributes for lp.Edit?
<wgrant> stokachu: Prepend 'OAuth ' to your Authorization value
<wgrant> StevenK: Probably, yes.
<wgrant> StevenK: Note that some of them are obsolete.
<wgrant> Particularly on BugTask
<wgrant> Attributes that don't exist any more.
<StevenK> I'll try setSchema IBugView
<wgrant> No.
<wgrant> Unless all those attributes were formerly settable.
<wgrant> But I suspect they were not.
<StevenK> You spoil all of my fun
<wgrant> See Branch for a reasonable example of how to do things.
<StevenK> But okay, grabbing the list
<StevenK> I think it's only hits{,timestamp}
<StevenK> Oh, community{score,timestamp}
<wgrant> See what I mean?
<StevenK> And a few others
<StevenK> Right, so there's 9 left
<StevenK> I think that's a reasonable list
<wgrant> Most should have setters.
<wgrant> So not too many should need naming explicitly.
<stokachu> SUCCESS!
<StevenK> Hm, this branch isn't too bad. It's just going to be hell to get reviewed.
<wgrant> stokachu: It works better when you specify an auth scheme? :)
<stokachu> wgrant: lol sooo much better
<StevenK> wgrant: My branch that makes ProductSet add AccessPolicy fails with missing db perms. :-(
<wgrant> StevenK: I am unsurprised.
<wgrant> StevenK: Grant INSERT on AP to anything with INSERT on Distribution or Product
<wgrant> Problem solved.
<StevenK> That's disgusting.
<StevenK> But yeah.
<StevenK> Looks like just write and updateremoteproduct
<StevenK> Oh, bah
<StevenK> Forgot to add maybeConfirmBugtasks to IBugSomething
<StevenK> wgrant: I think I've got it working.
<StevenK> Which is kind of awesome, considering what I did to IBug.
<wgrant> :)
<StevenK> Sadly, the branch is 1215 lines, but I don't think that can really be helped given just how big IBug is.
<wgrant> Indeed.
<wgrant> I won't kill you.
<StevenK> Oh, are you offering to review it, then? :-)
<wgrant> Depends on when it's proposed :)
<wgrant> I'll be reading the diff anyway...
<StevenK> Passed tests:   1159
<StevenK> Failed tests:      2
<StevenK> 1 failed test due to the fact that the stream is still being written to, and another which I've fixed.
<StevenK> I think that's enough to prove the changes are okay enough to toss at ec2.
<wgrant> Sounds reasonable.
<StevenK> I'm not in danger of the instance terminating after 20 minutes because make failed, or getting the dreaded message that it tried to send a mail and failed because the message was so damn large.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/force-ibug-into-line/+merge/96513
<StevenK>         linux_source_bug_dupe.syncUpdate()
<StevenK>     ForbiddenAttribute: ('syncUpdate', <Bug at 0xfb78dd0>)
<StevenK> I can't see syncUpdate() in either the model or interface, but the ZCML allows it.
<wgrant> StevenK: Done
<wgrant> syncUpdate is probably an SQLObject method
<wgrant> Anything using it is probably wrong
<wgrant> Delete the call and see if the test still works.
<StevenK> There's a bunch of them
<wgrant> Does the test explain why they're there?
<StevenK> I've not checked, just noticed they were failing
<wgrant> No
<wgrant> So, delete them.
<wgrant> Flushes are largely implicit now.
<StevenK> wgrant: maybeConfirmBugtasks is in the model, and was in the ZCML, so by rights it should be on the interface?
<wgrant> StevenK: You're probably not going to be confirming tasks due to an edit if you can't edit the bug.
<StevenK> Hmmm, it's only called by the bug model itself
<wgrant> Not by BugTask?
<wgrant> No
<wgrant> So kill it from the interface.
<cody-somerville> Are there any plans to rename the arm restricted arch family to armel?
<wgrant> It's unprecedented manual SQL with minimal benefit.
<wgrant> Doable, but there are no plans.
<cody-somerville> How much trouble would it be to do? With it being 'arm' currently, I have to special case the arch in our automated project setup code.
<StevenK> wgrant: personIs{DirectSubscriber,AlsoNotifiedSubscriber,SubscribedToDuplicate} were also declared under IBugSet, their docstrings meantioned IBug and they were in IBug's ZCML.
<wgrant> bigjools: Around?
<bigjools> yes
<cody-somerville> wgrant, or alternatively, would it be possible to introduce armel and just leave arm there as legacy?
<wgrant> cody-somerville: No
<wgrant> bigjools: UPDATE processorfamily SET name='armel' WHERE name='arm'?
<wgrant> (see above)
 * bigjools tries to think what that will break
<wgrant> The DAS archtags are already all armel.
<bigjools> yeah
<wgrant> The only people it can break are commercial admins.
<wgrant> And I don't care about them :)
<bigjools> pfff
<bigjools> wth can see proc family?
<StevenK> But PFs should not change. :-(
<bigjools> y'know
<bigjools> this should not change, as StevenK says
<wgrant> Why not?
<wgrant> It's wrong, and nothing except the UI and API use the anem.
<wgrant> name
<bigjools> why is there not an armhf?
<cody-somerville> There is an armhf.
<wgrant> arm is armel
<wgrant> armhf is armhf
<bigjools> oh ok - feh I was looking at dev data not prod
<wgrant>   6 | arm     | ARM Processors                        | The ARM family of processors, primarily used in embedded applications. | t
<wgrant>   9 | armhf   | ARM hard float processors             | ARM processors with an FPU                                             | t
<StevenK>   File "/home/steven/launchpad/lp-branches/force-ibug-into-line/lib/lp/bugs/model/bug.py", line 2071, in _markAsDuplicate
<StevenK>     duplicate_of.maybeConfirmBugtasks()
<StevenK> ForbiddenAttribute: ('maybeConfirmBugtasks', <Bug at 0x159d31d0>)
<StevenK> wgrant: ^
<StevenK> That is what prompted me to add it to the interface.
<wgrant> StevenK: Ah, of course. It's called from the other bug.
<cody-somerville> Also, does anyone know why https://api.launchpad.net/1.0/unr/+milestone/unr-9.10 is returning a HasBug object instead of Milestone object?
<bigjools> wgrant: ok do it
<wgrant> cody-somerville: launchpadlib is thpecial
<StevenK> wgrant: Okay, I'll re-add it to the interface.
<cody-somerville> wgrant, lol. say again?
<wgrant> cody-somerville: unr is a project group
<wgrant> Project groups are evil.
<wgrant> Don't do it :)
<StevenK> wgrant: So I drop IBugAdmin, where should heat_last_updated and setHeat() go?
<wgrant> StevenK: Well, I was going to say to just drop them from the interface. But setHeat seems to be entirely unused.
<wgrant> Except by tests.
<StevenK> And heat_last_updated is used by garbo
<wgrant> Yes.
<bigjools> what download limits does internode have?
<wgrant> But not through the interface.
<wgrant> bigjools: About 10 different ones.
<StevenK> wgrant: Right, so I should remove the setHeat tests and then IBugAdmin can just die?
<wgrant> Ranging from probably 10GB to a terabyte.
<bigjools> oh well, hope I don't blow this place's limit
<StevenK> You can probably ring up and ask
<wgrant> StevenK: Assuming the tests aren't doing anything interesting, yep.
<StevenK> wgrant: It seems to just be doc/bug-heat.txt
<StevenK> wgrant: So I can drop those sections of the doctest
<wgrant> StevenK: Sounds reasonable.
<wgrant> cody-somerville: Should be renamed.
<cody-somerville> wgrant, Okay, thanks.
<bigjools> wgrant: is there a broadband comparison/guide site that lists all providers instead of just the crappy ones?
<StevenK> Haha
<wgrant> bc.whirlpool.net.au
<wgrant> Although I can't talk to it...
<wgrant> Whirlpool down. That's amusing.
<bigjools> broadbandguide is a joke
<wgrant> Looks like it's been down for about 15 minutes.
<nigelb> bigjools: I heard broadband down there is a joke anyway ;)
<wgrant> How odd.
<bigjools> nigelb: an ISP walked into a bar ...
<nigelb> hehe
<StevenK> wgrant: Now where people complain about NBN?
<wgrant> Coalition headquarters?
<StevenK> Haha
<StevenK> Which is even more pointless than complaining about it on Whirlpool.
<bigjools> odev setup, why u wear out my ssd
<StevenK> wgrant: I've had to remove a bunch of bug-heat.txt
<StevenK> Ah crap. I need IHasLinkedBranches on IBugView due to linked_branches, but that causes a test failure because linkBranch and unlinkBranch require Edit
<wgrant> StevenK: Probably best to have IHasLinkedBranches inherited by IBug directly, and name the attributes in configure.zcml
<wgrant> bigjools: Whirlpool is back, btw
<bigjools> wgrant: yeah looking at it
<bigjools> wgrant: it's a bit confusing for a newb
<bigjools> I dunno what half the terms are
<StevenK> bigjools: Did you search?
<bigjools> StevenK: for what?
<StevenK> bigjools: From bc.whirlpool.net.au there's a "or your can search for ..." which is a link
<StevenK> wgrant: http://pastebin.ubuntu.com/874142/
<StevenK> wgrant: Oh, this is not helped that IBugView has its own definition of linked_branches
<bigjools> is it worth doing the phone via internode?
<StevenK> wgrant: And IBugView['linked_branches'] is subtly different
<bigjools> StevenK: don't see that link
<wallyworld__> internode is expensive
<StevenK> bigjools: Oh, bc.whirlpool -> QLD, search
<bigjools> StevenK: ta
<bigjools> wallyworld__: you're a stuck record :)
<wallyworld__> but it's true
<wallyworld__> it's your money i guess
<bigjools> wallyworld__: I'm comparing all on offer before I decide
<StevenK> Just note that TPG couldn't run a reliable network if their lives depended on it.
<wallyworld__> np. i'm just trying to stir up StevenK
<bigjools> wallyworld__: +1
<wallyworld__> StevenK: tpg has been 10000% reliable for me
<wallyworld__> not one issue
<bigjools> only 10000?
<wallyworld__> fast and unlimited
<bigjools> what d/l speeds?
<StevenK> And yet you ping timeout at least 4 times a day and your IRC nick keeps growing underscores.
<nigelb> pwned.
<bigjools> hah
<bigjools> 10000 is not enough of a superlative, let's say 1000000%
<wallyworld__> bigjools: i get about 9000Mbps
<wallyworld__> StevenK: i don't timoeout
<wallyworld__> i disconnect when i change rooms etc
<wallyworld__> nothing to do with tpg
<wallyworld__> sometimes my wireless router craps out also
<wallyworld__> tpg is innocent
<bigjools> they are popular on whirlpool
<wallyworld__> for good reason
<StevenK> Yes, they're cheap and crap.
<wallyworld__> why do you say crap?
<wallyworld__> evidence?
<bigjools> StevenK: be specific?
<StevenK> wgrant: Right, the only thing I can't figure out how to solve with my current horrible solution is the @accessor_for() for getVisibileLinkedBranches()
<wallyworld__> i know several people who use them
<wallyworld__> and they are all very happy
<StevenK> I have had a few friends here who got random dropouts, slow speeds, and the support would follow a script
<bigjools> I miss my UK ISP
<wgrant> It depends on region.
<wallyworld__> i reckon 9000Mbps is ok given i'm a fair way from the exchange
<wgrant> They're massively oversubscribed on most Melbourne nodes.
<wallyworld__> and have a rats nest of cabling to the socket
<wgrant> People struggling to get 1Mbps
<wgrant> Don't know how it is in QLD
<wgrant> StevenK: What about it?
<wgrant> StevenK: This should be no different from before.
<StevenK> wgrant: http://pastebin.ubuntu.com/874160/
<StevenK> wgrant: I had to remove linked_branches from IBugView, since IHasLinkedBranches also defines it.
<wgrant> StevenK: Just move it into IBug
<StevenK> I did
<wgrant> What's with that insane "Please Zope" thing, then?
<StevenK> wgrant: Because with linked_branches in IBugView and IHasLinkedBranches under IBug, Zope de-pramed its toys with the protectName traceback
<wgrant> StevenK: That's why I said to move it to IBug
<StevenK> Oh, move everything, right.
<StevenK> And getVisibleLinkedBranches ?
<wgrant> Everything that conflicts.
<StevenK> Right, that looks ... acceptable.
<StevenK> Certainly a fuckload better than it was.
<StevenK> Hey wow, due to ripping out most of bug-heat.txt, I think I can remove dangerousGetAllBugs()
<wgrant> Did you only rip out the relevant bits?
<StevenK> wgrant: http://pastebin.ubuntu.com/874168/
<StevenK> It uses setHeat with gay abandon
<wgrant> StevenK: And the award for test that is rendered useless for no good reason goes to...
<StevenK> Oh, so I can just remove the whole thing then? :-)
<wgrant> I suggested that you rip it out, not remove every block that uses setHeat and then mangle the expected output of every other block to fit :)
<StevenK> wgrant: Oh, you wanted me to delete the entire test?
<wgrant> No
<StevenK> I thought so. :-(
<wgrant> Delete the *relevant* bits of the test.
<wgrant> Not delete parts of it, and render the rest useless.
<StevenK> Well, setHeat is still on the model, so I can just rSP call it
<wgrant> Where that is unavoidable, yes.
<wgrant> In most cases you don't need to use setHeat, however.
<StevenK> I don't see how
<StevenK> Given my reading of the test
<wgrant> -First, we'll set the heat of all bugs so that none of them are out of
<wgrant> -date.
<wgrant> That doesn't not require setHeat
<StevenK> Oh, IStore(Bug).set(heat=0) ?
<StevenK> I *think*
<StevenK> It's been a while
<wgrant> Read the test.
<wgrant> It cares about the timestamp.
<wgrant> Not the heat value.
<wgrant> (Your solution is almost valid, but it's probably not what the test wants to test.)
<StevenK> Right, so I should set all heat_last_updated = now
<wgrant> Most likely.
 * StevenK tries to work out how to do that with Storm
<StevenK> And then dangerousGetAllBugs() can die too
<wgrant> IStore(Bug).find(Bug).set(heat_last_updated=UTC_NOW)
<StevenK> stub: Hai, can you look at https://code.launchpad.net/~stevenk/launchpad/db-add-information_type/+merge/96290 ?
<stub> k
<stub> StevenK: Is it possible to describe what an InformationType is?
<stub> Its a very 'long cat is long' type comment description at the moment :-P
<stub> 'Enum describing what type of information is stored and used to determine how the access policy is applied', or something like that if that isn't quite riht
<stub> I do see that it will not be changing frequently, so I guess adding yet another column to Bug and Branch is fine.
<wgrant> It will change infrequently and we're about to stop doing searches over Bug anyway :)
<wgrant> It replaces the private and security_related flags.
<StevenK> wgrant: Do you agree with stub's comment for the two columns?
<wgrant> I haven't looked at the MP.
 * wgrant looks.
<StevenK> I meant the SQL comment above
<StevenK> 'Enum describing what type of information is stored and used to determine how the access policy is applied'
<wgrant> There's a better one in the MP
<StevenK> Right, I'll update the comment
<StevenK> stub: Thank you for the review.
<stub> np
<wgrant> stub: FYI, https://code.launchpad.net/~wgrant/launchpad/bugtaskflat-db/+merge/96520 will be happening once StevenK's stuff is finalised and migrated on prod.
<wgrant> Be afraid.
<nigelb> I feel like a lot of conversations in this channel should be preserved in a QDB.
<stub> Think I've seen most of that already.
<wgrant> Yeah
<wgrant> stub: We can't set user-specific connection limits in pgbouncer, can we?
<wgrant> stub: That was garbo's problem.
<stub> Bug #949740
<_mup_> Bug #949740: No connection timeout makes transaction timeout unreliable <Storm:New> < https://launchpad.net/bugs/949740 >
<wgrant> Ah, interesting.
 * stub looks into user-specific connection limits
<wgrant> Although there's no timeout in garbo.
<wgrant> stub: I couldn't see any way to do it.
<wgrant> But you may have more luck.
<stub> We can, but we need to define it as a separate database (so connection to session_prod_garbo rather than session_prod)
<stub> But we still need nagios checks that the pools are not full, so I don't think we gain much except complexity.
<stub> Well... maybe protect against processes with potentially unlimited numbers of connections like the Librarian or django servers
<lifeless> stub: hey
<stub> ho
<lifeless> stub: so, cynthia is having a terror night, going to pass on catching up if thats ok
<stub> No probs. Want to make a time for tomorrow?
<lifeless> stub: lets give it a shot :)
<lifeless> my 4pm would be great
<stub> You're at +13 now?
<lifeless> stub: https://bugs.launchpad.net/launchpad/+bug/948786 - I spoke with Tom; if we automate master-slave cutovers we can allow safer upgrades of pg etc rather than squashing it into the schema change window
<_mup_> Bug #948786: add an option to full-update.py to not restart pgbouncer when finished <canonical-losa-lp> <Launchpad itself:Incomplete> < https://launchpad.net/bugs/948786 >
<lifeless> stub: yeah
<lifeless> (+13)
<stub> Ok. In bed by 1:30am :)
<lifeless> *snort*
<lifeless> ok, /wave then see you tomorrow
<stub> lifeless: Switching master/slave is pretty heavyweight, and if all you are doing is upgrading PG to a new point release way overkill.
<stub> lifeless: Also, I certainly would want to defer any automated switchover to PG 9.1 + new replication
<wgrant> Switchover with WAL streaming involves a bit of rsync.
<wgrant> But I trust it to be a bit more reliable.
<stub> lifeless: But it might just be a big catch-22, as switching master/slave roles requires a bit of downtime
<wgrant> A neutered version of readonly mode would work extraordinarily well here :)
<stub> (at least until we are able to use transaction pooling across the board, at which point we could pause db access, wait for all transactions to complete or kill them, switch roles, reconfigure pgbouncer, reenable access)
<wgrant> stub: Can't we tell eg. pgbouncer to hold new connections, wait a few seconds for old ones to clear, then do the switch?
<wgrant> That sounds nearer-term than letting us switch to transaction mode.
<stub> (although I just opened a bug that would cause requests to OOPS if their connection attempts timed out)
<StevenK> wgrant: Er, so, I think we should allow date_last_updated
<StevenK> wgrant: steven@liquified:~/launchpad/lp-branches/force-ibug-into-line% grep Forbidden lp.log | cut -d\' -f2 | sort | uniq -c | sort -k1 -rn | head -n 1
<StevenK>     300 date_last_updated
<wgrant> StevenK: What broke?
<StevenK> LOTS
<wgrant> eunhelpful
<stub> wgrant: Thats what I wrote. But it requires transaction pooling mode, as pgbouncer only notices when a transaction completes in that mode. It won't interrupt transactions in session mode as that might be interrupting cross-transaction resources like cursors or temporary tables.
<StevenK> wgrant: Not sure how to get a list of failing tests from a subunit stream.
<wgrant> stub: Bah, I meant s/pgbouncer/haproxy/
<wgrant> stub: To avoid that issue
<wgrant> Still requires that we turn evil scripts off.
<stub> Yes. And appservers, as they don't reopen connections each transaction
<stub> oh nm.
<wgrant> Make haproxy hang, wait $TIMEOUT, stop pgbouncer, switch, reconfig pgbouncer, start pgbouncer, unhang haproxy
<wgrant> (hopefully timeout will be <5s by then :))
<stub> you can replace haproxy with pgbouncer there
<wgrant> Can we?
<stub> oic.
<wgrant> Maybe if we taught the appserver that connection start hangs were to be expected.
<wgrant> But ideally we'd somehow just hang master connections.
<wgrant> Since slave connections aren't a problem.
<adeuring> good morning
<stub> wgrant: They are, which is why this is rather pointless for the most part. When we do a point release, we do it everywhere.
<stub> pg point release upgrade
<StevenK> wgrant: Failing tests: http://pastebin.ubuntu.com/874240/
<wgrant> stub: But we don't have to do them all at the same time.
<stub> No. If we want, we could do it three times with more downtime.
<wgrant> Assuming we rework our store management to be HA
<wgrant> Rather than failxploding
<wgrant> stub: LP should pick an available slave or the master if no slave is available.
<wgrant> Then we can murder slaves at will.
<wgrant> And nobody will notice.
<wgrant> StevenK: Tracebacks?
<wgrant> s/s//
<stub> With a suitably complex system, we might be able to catch all the states. But a system that complex will have more overall downtime because it is more complex.
<StevenK> wgrant: I can copy the log to carob/something else if you like.
<wgrant> StevenK: No. How widespread are the calls?
<StevenK> wgrant: It's 2.3MiB uncompressed.
<StevenK> wgrant: Like the count above, there are 300 calls to set date_last_updated that fail.
<wgrant> StevenK: There are 300 tests that fail.
<wgrant> StevenK: They won't all be distinct calls.
<StevenK> No, there are 165 tests that fail, 300 calls.
<wgrant> s/calls/callsites/
<StevenK> A fair whack of them are in doctests.
<StevenK> Where they are calling it directly.
<lifeless> StevenK: cat stream | subunit-filter --no-skips | subunit-ls
<wgrant> StevenK: Setting it directly!?
<wgrant> stub: How is basic client-side DB failover complex? :)
<StevenK> wgrant: Yes, setting date_last_updated directly.
<StevenK>   File "/home/steven/launchpad/lp-branches/force-ibug-into-line/lib/lp/bugs/subs
<StevenK> cribers/buglastupdated.py", line 31, in update_bug_date_last_updated
<StevenK>     current_bug.date_last_updated = now
<StevenK> That's another callsite.
<StevenK> ForbiddenAttribute: ('date_last_updated', <Bug at 0xf0b63d0>)
<lifeless> StevenK: or load it into testr
<wgrant> StevenK: Oh, directly except not.
 * lifeless goes again (someone rang our house phone... at 10pm!)
<wgrant> StevenK: That function is probably the source of 99% of the exceptions.
<StevenK> It is not
<StevenK> From unittests, sure. doctests are fiddling with date_last_updated directly
<wgrant> $ bzr grep '\.date_last_updated =' | grep -v tests | grep -v doc | grep -v translations | grep -v stories | grep -v answers | wc -l
<wgrant> 1
<stub> wgrant: scripting haproxy + pgbouncer was what I was talking about.
<wgrant> (and that one is update_bug_date_last_updated)
<StevenK> wgrant: Which is probably that callsite then
<StevenK> wgrant: Right, but I can't rSP that :-(
<wgrant> StevenK: Why not?
<stub> wgrant: client-side db failover? yer, but that is for unexpected slave outages. For expected outages, we just do the redirections in pgbouncer.
<wgrant> subscribers do it occasionally
<StevenK> I don't like using rSP in non-test code
<wgrant> StevenK: Sometimes it's justified. When there's a single callsite that's outside the model class only because someone thought subscribers were awesome, I think it's justified.
<wgrant> stub: Why do them in pgbouncer? ISlaveStore can pick the first available slave. IMasterStore can pick the master, or display a "come back in 10 seconds" page if it's not available.
<stub> wgrant: Because then the clients need to know where all the slaves actually *are* so it can make that choice. And because one use case is for reshaping the cluster, that means pushing configuration changes out and having them take effect.
<wgrant> stub: Well, yeah, would need some means for determining the master dynamically. Not entirely sure how that would work. It may be by pgbouncer, but it would be nice to not have to reconfig pgbouncer in a reshape situation.
<stub> We have to reconfigure something. pgbouncer is easy because it is running on the same host as we are run these admin scripts from.
<wgrant> At minimum we have to reconfigure the databases to have a new master, so the ideal situation is that the appservers get that from postgres.
<wgrant> But that seems like it could be hilariously awful.
<stub> Yer. complex, leading to downtime because this sort of thing always breaks unless you invest hugely in all the testing.
<wgrant> Anyway, I mainly suggested the sensible client-side slave fault tolerance as a solution to the 3x pg upgrade issue.
<wgrant> If we can take a slave down without anybody noticing, things become easier.
<stub> Except the other half of the connections that need a master :)
<wgrant> It means that we no longer need to SSH across 3 hosts during downtime.
<wgrant> And it paves the way to full nodowntime :)
<stub> We can take a slave down with nobody noticing with pgbouncer. We just redirect launchpad_prod_slave_1 to a different database and deal with the existing sessions.
<stub> And that is easy to automate, and even easier if we are running in transaction mode and have server connections returned to the pool after every transaction.
<wgrant> That handles scheduled downtime, indeed.
<wgrant> And admittedly the DB servers have been rather more reliable than the rest.
<wgrant> But it seems sort of odd that we have this nice replicated DB setup, yet if any one of the three nodes goes down, LP goes down with it.
<stub> It takes a lot of work and resources to do automated failover properly. It is pretty rare to be able to justify those sorts of resources, and because of this more often than not you end up causing downtime than stopping it.
<stub> redundant slaves is the easier problem, and can be handled by the connection pool
<stub> (in pgbouncer's case, it involves putting haproxy between pgbouncer and the databases)
<wgrant> Perhaps.
<wgrant> But we want things like SSO to get their noses out of our DB, which really requires us to have good availability.
<wgrant> Read availability, that is.
<wallyworld__> wgrant: can you recall the lp way to disable anchor? add "disabled" to the class? or something like that?
<wgrant> wallyworld__: This isn't client-side rendered?
<wallyworld__> wgrant: yes, but i need to dynamically disable a link depending on what the user clicks
<wgrant> wallyworld__: The bug "Also affects" links use private-disallow, so perhaps there's not a real convention.
<wgrant> wallyworld__: Or do you mean something other than hiding?
<wallyworld__> wgrant: i want the "Submit" link to be there but greyed out until the select at least one info type
<wgrant> Do we have any existing things like that?
<wgrant> All I know of is disabled buttons.
<wallyworld__> ah, the auto bug/branch linking code disables links
<wallyworld__> i'll look at that
<wgrant> Mm
<wgrant> Sort of.
<wgrant> They're not really disabled.
<wallyworld__> link.addClass('invalid-link')
<wgrant> Just greyed out and making odd popups.
<wallyworld__> clicking is disabled and they are grey
<wallyworld__> perfect :-)
<wallyworld__> well, not sure about clicking, but i;ll handle that in my code. the css class works
<wgrant> Is this going into the picker infrastructure, rather than something specific to disclosure?
<wgrant> We already have about 4 submit patterns in pickers.
<wgrant> Would be nice to not introduce more.
<dpm> hi all, I've got a couple of launchpadlib/API questions: given a project name, is it possible to find out if that project has any distro packages in Ubuntu?
<dpm> I've read the api docs, but I couldn't figure that one out
<dpm> hi all, would anyone have an aswer to the earlier question re: the Launchpad API?
<dpm> <dpm> hi all, I've got a couple of launchpadlib/API questions: given a project name, is it possible to find out if that project has any distro packages in Ubuntu?
<dpm> I've read the api docs, but I couldn't figure that one out
<jelmer> hi dpm
<dpm> hey jelmer :)
<jelmer> dpm: I'm pretty sure the reverse is possible (ubuntu pkg -> project), not sure about project -> ubuntu pkg
 * dpm looks at the reverse in the API docs, which might be a workaround
<jelmer> dpm: lp.distributions["ubuntu"].getSourcePackage(name="bzr").upstream_product
<dpm> hm, at a quick glance I cannot see how the reverse is possible from https://api.launchpad.net/devel.html#source_package
<dpm> ah, cool, thanks jelmer :)
<dpm> I was looking for project instead of product
<jelmer> product is the old name for project
<salgado> sinzui, around?  did you get a failure email from that ec2-land you did for me yesterday?
<benji> I'm getting an error submitting a branch using either pqm-submit or ec2 land, has anyone seen this one: https://pastebin.canonical.com/61922/
<benji> heh, my AWS newsletter includes a new instance type that might work for our buildbot stuff: m1.medium.  It's half the cost of the m1.large we're using now.
<jelmer> benji: your bzr is too old for your version of bzr-pqm
<benji> jelmer: how does one go about rectifying that?
<jelmer> benji: install a newer version of bzr (are you on precise yet?)
<benji> jelmer: lets see... the VM I'm using at the moment is oneiric
<deryck> Morning, all.
<rick_h_> morning
<rick_h_> deryck: on the new hardware this morning?
<deryck> rick_h_, indeed
<deryck> rick_h_, feels nice :)
<rick_h_> as shiny as hoped?
<rick_h_> hehe
<deryck> yeah, definitely.  very nice laptop/
<flacoste> deryck: convoy has no release?
<flacoste> deryck: Daviey is packaging it for precise for the maas project
<czajkowski> deryck: how has your team managed all the questions/issues re private bugs in the last few weeks ?
<czajkowski> deryck: can you see private bugs or do you like me see oops ?
<deryck> flacoste, I don't think it has a release.  rick_h_ do you know for sure?
<deryck> czajkowski, no, I can't see them either. Usually, you just have to ask follow up questions to the person to figure things out.
<rick_h_> flacoste: no release? it's in a daily build setup by StevenK and we use that
<czajkowski> deryck: am fairly stuck on https://answers.launchpad.net/launchpad/+question/188450
<rick_h_> flacoste: it's not on pypi, but I need to bug the guys about that one
<flacoste> rick_h_: it's not on lp either (i mean there are no releases on lp either)
<flacoste> rick_h_: how's pycon?
<deryck> pasting for bot link:  OOPS-dd4660fd4f3a19b43f9d6649dc99eda
<rick_h_> flacoste: there's a recipe to build everyone uses (landscape has one, etc) but yea, no releases done
<rick_h_> flacoste: leaving for pycon this afternoon, so hopefully awesome! :)
<deryck> oh the bot doesn't giving me a link here.  hmmm
<deryck> what's the oop site URL?  new lappie and all. ;)
<czajkowski> deryck: https://lp-oops.canonical.com/openid/+login
<deryck> czajkowski, so I would have thought it would have redirected to the correct URL then said unauthorized to you and I.
<deryck> czajkowski, maybe this is different now with purple squad's new work.
<deryck> sinzui, can you see bug 924475?
<salgado> sinzui, I guess you didn't see my question earlier?  did you get a failure email from that ec2-land you did for me yesterday?
<sinzui> I do indeed have a failure
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugtasks: 4*10
<sinzui> salgado, did you get the email. I read it 10 minutes before you pinged me, not I do not see and it is not in my trash?
<salgado> sinzui, no, I didn't get it
<sinzui> :(
 * sinzui keeps looking
<rick_h_> jcsackett: ping, can you peek at this when you get a sec please? https://code.launchpad.net/~rharding/launchpad/fix_cal2/+merge/96584
<jcsackett> rick_h_: r=me.
<rick_h_> jcsackett: ty
<jcsackett> aren't you supposed to be pyconning? :-P
<rick_h_> I leave this afternoon
<mhall119> I have an API question
<mhall119> if I have a Branch object from launchpadlib, is there a way to get a list of all the branches that have been previously merged into it?
<sinzui> salgado, I found the email, well in a manner of speaking. I can see the email and will forward it to you. I do not know where thunder(shite)bird moved it too)
<sinzui> salgado, one failure. Looks easy to fix
<salgado> sinzui, heh, that's fine.  thanks a lot!
<czajkowski> salgado: mails now being sent and post done
<salgado> czajkowski, thanks!  the blog post will be held until everything is live, right?
<czajkowski> salgado: the blog post is just a copy of the mail annoucment you sent me
<czajkowski> is thre another more detailed post ?
<salgado> czajkowski, mrevell said in the email that he'd be writing the blog post as well.  but maybe the idea is to have two posts: one with the warning and another one when the changes are live?
<czajkowski> salgado: thats how I read it
<mhall119> lifeless: ping
<mhall119> wgrant: ping
<czajkowski> mhall119: wrong times for both of them
<mhall119> hmmm, who's awake at this time of day?
 * mhall119 will try the on-call reviewer
<mhall119> jcsackett: ping
<mhall119> oh sweet, LP support for work items!
<mhall119> czajkowski: \o/
<mhall119> has that been deployed yet?
<czajkowski> nope not yet
<jcsackett> mhall119: pong.
<mhall119> jcsackett: if I have a Branch object from launchpadlib, is there a way to get a  list of all the branches that have been previously merged into it?
<jcsackett> mhall119: not sure. one sec.
 * jcsackett takes a look at API docs
<mhall119> the API will get me branches currently in MPs, but I don't know about past-merged MPs
<mhall119> and I guess it wouldn't be able to give me branches merged manually, without an MP
<rick_h_> mhall119: I'd expect that to be the job of bzr history more than LP api
<mhall119> rick_h_: yeah, me too, it's just that I'm already using launchpadlib to generate stats about MPs
<mhall119> and trudging through bzr branch history is likely to be orders of magnitide sloper
<mhall119> slower
<jcsackett> mhall119: you can get merged MPs from the API. so you could construct a script that gets all the MPs for the branch with a date_merged, and get the branches from that.
<rick_h_> mhall119: yea, I guess need to split bzr merge vs MP records in LP.
<jcsackett> mhall119: but it's likely to be convoluted.
<mhall119> ok, thanks, that's all I needed to know
<jcsackett> and yes, it'll only get you branches that were merged with an MP. if someone just merged in a branch on their own that's not going to show up.
<jcsackett> sinzui: i see there is a pending request for ui review from you on a loggerhead change. https://code.launchpad.net/~cruzjbishop/loggerhead/ui-changes/+merge/95090
<jcsackett> did you actually want to weigh in on that, or shall i just move it along? it's a switch of browser-specific rounded corners to the generalized version.
<sinzui> I don't see why I need to review, but...
<jcsackett> sinzui: you don't.
<jcsackett> i'm asking if you want to.
<sinzui> I think we can drop -webkit-border-radius -moz-border-radius because firefox and webkit switched to the official property about 3 years ago
<jcsackett> i don't think it's necessary, and was prepping to just move it along.
<jcsackett> sinzui: dig. i'll move this into approved and get it merged into loggerhead.
<mabac> jcsackett, could you bear having another look at https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-widget/+merge/94790 for me? We found just another little bit we needed to add.
<sinzui> jcsackett, FTW.., why do we have border-radius at 4px, 5px, and this example of px. There can be only one
<jcsackett> sinzui: FTW? or WTF?
<sinzui> I think 5px is the standard
<sinzui> Consider -2 WTF, consider -1 FTW
<sinzui> I think the CSS in the review should use 5px for the radius. I think Lp's own CSS also need fixing and we might be able to delete a few rules
<rick_h_> jcsackett: sorry, added sinzui because I wasn't sure on how only some corners were rounded and if that was something that should be ui reviewed/standardized
<rick_h_> jcsackett: and sinzui was listed in the wiki for ui reviews :)
<sinzui> rick_h_, that is ancient history...but you were right to get me to look at the oddness
<jcsackett> sinzui: ok, i'll leave a comment to the effect of plz change to 5px.
<jcsackett> or you can, if you like.
 * sinzui could land an rs branch to just fix all the references tomorrow
<jcsackett> sinzui: it'd be two branches, right? one for LP and one for loggerhead.
<sinzui> jcsackett, yes. The logger head fix can be made in the branch that is being reviewed
<jcsackett> sinzui: dig. just wanted to make sure i was on the same page as you.
<sinzui> I feel like I am flipping though a book at the moment. I hate editing moin
<abentley> adeuring: chat?
<sinzui> When I am lord god-emperor of the Earth, the Universe, and Canada, Moin syntax will be 100 papre-cut offense.
<jcsackett> sinzui: if you are creating new pages, allenap created a method to use rst.
<jcsackett> or you could update the entire page you're editing and use his method. :-P
<sinzui> second offenders will get lemon juice on the cuts
<allenap> sinzui: +1
<sinzui> jcsackett, i am editing information is a large table
<abentley> jcsackett: I just wish rst table headers had consistent meaning.
<jcsackett> mabac: this branch is getting sort of unwieldy. are you merging in branches approved in other MPs as well?
<jcsackett> abentley: +1
<mabac> jcsackett, sorry about that. yes we thought that the help-popup belonged in this branch for landing
<mabac> jcsackett, and we really believed this branch was done.
<mabac> jcsackett, which is not a (good) excuse for messing it up
<jcsackett> mabac: i'll take a look at these later revisions. but an ongoing branch for weeks is a recipe for problems. :-)
<mabac> jcsackett, I promise that I've learned a lot in the process? :D
<jcsackett> mabac: :-P
<mabac> jcsackett, but yes, I'll keep it neater next time
<jcsackett> mabac: i'm seeing changes to the Makefile in those revisions? it looks like you've got a weird diff going on with rick_h_'s recent js_watch work ...
<jcsackett> rick_h_: did your js_watch stuff ever land? i don't see it in devel, but i think i'm seeing it in mabac's diff.
<mabac> jcsackett, I can see that we merged devel just recently. perhaps something more changed since then. I'll take a look
<jcsackett> mabac: i've sorted it. i was looking at changes in the recent revisions, which includes when rick_h_'s stuff was introduced; there's an update from devel in those revisions i missed.
<mabac> jcsackett, ok good
<jcsackett> mabac: b/c of that, though, i can't get a diff of just the things you've added that need to be looked at. can you generate a diff of just the new stuff and throw it up on pastebin or something?
<jcsackett> otherwise i'm going to have to look at the whole diff again, which i can do today but not anytime soon.
<rick_h_> jcsackett: yes, the js watch stuff landed
<mabac> jcsackett, is it better if I merge from devel and push again?
<jcsackett> mabac: no, that won't sort it. :-)
<mabac> jcsackett, ok diff coming right up
<jcsackett> mabac: if you don't have time to do that, it's ok. i can review this today, just not now.
<mabac> jcsackett, http://paste.ubuntu.com/874794/
<mabac> jcsackett, thanks
<popey> hullo.
<czajkowski> popey: boo
<popey> I saw czajkowski's announce about blueprints and wondered if it would adversely affect http://status.ubuntu.com/ (given that site directly plucks work-item text in specially formatted ways from blueprints) or whether you've all thought about that and I should stop worrying âº
<czajkowski> salgado: ^^^
<jcsackett> mabac: perfect, exactly what i needed.
<jcsackett> reviewing now.
<salgado> popey, we planned it so that for users there are no visible changes (i.e. they will continue to enter the work items in a text field using the exact same format as before) and the tools will keep working (there was a trivial change to launchpad-work-items-tracker which I did to make it work against the new API and it was accepted yesterday)
<jcsackett> mabac: new revisions look fine. thanks.
<popey> salgado: groovy
<mabac> jcsackett, great thanks for reviewing it again :)
<mabac> jcsackett, will you make another go at landing it too?
<adeuring> abensure. mumble (sorry for the delay...)
<jcsackett> mabac: sure.
<salgado> jcsackett, actually, let's wait a bit.  see discussion on #launchpad-ops :)
<jcsackett> salgado: got it. :-P
<lifeless> morning
<czajkowski> lifeless: good day
<lifeless> hi czajkowski
<sinzui> jcsackett, do you have time to review this long branch: https://code.launchpad.net/~sinzui/launchpad/remove-register-branch/+merge/96649
<jcsackett> sinzui: sure, i can review it. it's mostly big b/c you're ripping stuff out, right?
<sinzui> Yes -800+ lines of unused views and tests
<jcsackett> sinzui: cool. i'll be happy to review it. these things needed killing for awhile.
<jcsackett> sinzui: r=me.
<jcsackett> and now i need more coffee. :-P
<sinzui> thank you
<sinzui> lifeless, or deryck can I have your +1 to try this script on staging, then production: http://pastebin.ubuntu.com/873679/
<lifeless> sinzui: the statement timeout is too high; needs to be 2000 to be sure we won't cause timeouts
<sinzui> okay
<czajkowski> if someone had some spare time could they answer https://answers.launchpad.net/launchpad/+question/190103  so I can book mark it as well for me to learn.
<czajkowski> thank you
<lifeless> sinzui: other than that +1 - or perhaps we should drop the rows?
<sinzui> lifeless,  we do not as yet (merge doesn't and the merged team continues to exist as a historical record if the team leaves before being merged). I am pondering deletion though; only someone with db access can see the data
<StevenK> lifeless: 2000? Er, what? I thought the policy was 2500.
<lifeless> pretty sure it was a round 2 seconds
<StevenK> For manual SQL? Look it up, it's 2500.
<lifeless> whats the wiki pge
<StevenK> No fair if you edit the wiki page to claim you're right.
<StevenK> "The write lock holding portion of the script must complete in 2.5 seconds to avoid causing timeouts in appserver requests."
<StevenK> https://wiki.canonical.com/InformationInfrastructure/OSA/RequestLogging/LP/SQL
<lifeless> StevenK: I considered it, but it wouldn't be fair.
<lifeless> however, I probably should drop it down to 1s
<StevenK> In either case, I'm right. :-)
<lifeless> twitch
<StevenK> wgrant: http://pastebin.ubuntu.com/875283/
<StevenK> wgrant: http://pastebin.ubuntu.com/875299/
<jelmer> sinzui: YOU ROCK
<jelmer> sinzui: thanks for lp:~sinzui/launchpad/remove-register-branch
<wgrant> I said you'd be interested :)
<jelmer> :)
<bigjools> jelmer: does any of these errors look familiar for recipe builds? https://bugs.launchpad.net/launchpad-buildd/+bug/940157
<_mup_> Bug #940157: PPA builders have corrupted filesystem <ppa> <Launchpad Auto Build System:Triaged> < https://launchpad.net/bugs/940157 >
<jelmer> bigjools: I don't recall having seen those for any recipe builds
<bigjools> jelmer: ok. Very weird then.
<wgrant> bigjools: Given that no other package has ever seen it, it seems unlikely that it's not a package bug.
<bigjools> wgrant: quite possibly yes
<bigjools> but I want some opinions from more people first
<wgrant> Everyone agrees that there's no reason to believe it's not a package bug.
<bigjools> who is everyone?
<wgrant> infinity, me
<wgrant> ie. most of the relevant people who weren't away when this came up originally.
<bigjools> I agree too, but I still want to ask around
<elmo> poolie: ping
<elmo> how rude
<elmo> bigjools: I agree too btw
<elmo> someone even check the first accused build; there's nothing wrong with the FS
<elmo> I'm not quite sure why gustavo doesn't add more output, -v to rsync or cp, strace or whatever
<wgrant> And palmer, vernadsky and rothera aren't exactly untested machines.
<elmo> wgrant: I've been meaning to ask - can receipe builds install from PPAs?
<wgrant> Perhaps Go is really so amazing that it exposes bugs that no other piece of software in the world does.
<wgrant> elmo: Yes.
<elmo> wgrant: so, umm
<elmo> wgrant: why are they running non-virtualized?
<elmo> isn't that a gaping security hole?
<wgrant> elmo: It's a non-virt PPA, I assume.
<elmo> ah, hmm
<wgrant> Possibly because ARM, but I'm not quite sure.
<elmo> ok, so normal receipe builds will only happen on a PPA?
<wgrant> Yes.
<elmo> cool, I'll stop freaking out quite so much
<wgrant> We're not entirely stupid.
<wgrant> I hope.
<bigjools> speak for yourself. wheeeeeeee
<bigjools> elmo: furry muff, ta
#launchpad-dev 2012-03-09
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Firefighting: - | Critical bugtasks: 4*10
<wgrant> lifeless: Around?
<lifeless> yup
<wgrant> lifeless: I'd like to discuss bugsummary-ng at some point.
<wgrant> Making it compatible with disclosure, and also making it performant.
<wgrant> And fixable.
<lifeless> voice?
<wgrant> Anything else would be likely to result in the heat death of the universe.
<lifeless> skynet?
<wgrant> Give me a sec to get it running, reinstalled on Monday.
<lifeless> 1
<wgrant> fuuuu
<StevenK> wallyworld: O hai
<wallyworld> yello
<StevenK> wallyworld: I've put two new photos in the river directory taken this morning which show the roughly 'normal' river level
<wallyworld> remind me of the dir name?
<StevenK> wallyworld: http://wedontsleep.org/~steven/river/
<StevenK> wallyworld: The two 03-09 photos are the normal leve
<wallyworld> big difference
<StevenK> *level
<StevenK> Oh yeah
<wallyworld> did you go for a swim?
<StevenK> Haha, no
<StevenK> It was quite impressive given how fast the water was flowing when it was up that high
<wallyworld> indeed
<StevenK> IntegrityError: duplicate key value violates unique constraint "accesspolicy__product__type__key"
<StevenK> :-(
<wgrant> StevenK: Only in test_bug_mirror_legacy_access?
<wgrant> Possibly in wallyworld's tests as well, I guess.
<StevenK> It was about sixteen tests
<StevenK> I've just merged devel and am running product tests manually to see if wallyworld has made life harder for me.
<wallyworld> who me?
<StevenK> lp.bugs.tests.test_bug_mirror_access_triggers.TestBugMirrorAccessTriggers.test_productseries_task is one failure
<wgrant> That's one of mine.
<wgrant> Easy to fix :)
<StevenK> Ah yes
 * StevenK hugs alt-q in zsh
 * StevenK fixes wgrant's test with +1/-21
<StevenK> lp.registry.tests.test_accesspolicy.TestAccessPolicySource.test_create
<StevenK> lp.registry.tests.test_accesspolicy.TestAccessPolicySource.test_find
<StevenK> Those two :-(
<wgrant> Hmmm, they're going to be difficult now.
<wgrant> You may want to fix them to manually create a Product
<wgrant> Rather than use ProductSet via the factory
<wgrant> Ah
<wgrant> Or just use PROPRIETARY instead
 * StevenK kicks TestAccessPolicySource.test_findByPillar hard
<StevenK> I can't use a tuple to describe that types that already exist, since they're returned as APs
<StevenK> And I don't want to just call findByPillar since that defeats the entire point of the test.
<poolie> hi bigjools
<wgrant> StevenK: Why can't you use a tuple?
<wgrant> StevenK: I use tuples to match them throughout that test file.
<StevenK> wgrant: It didn't work. I'll keep digging after lunch.
<wgrant> It's not magic. It works.
<lifeless> so I whinged to telecom yesterday
<lifeless> my adsl is being line checked overnight
<lifeless> guess what hasn't happened today
<wgrant> It seemed to be being a bit bad this morning.
<wgrant> But maybe it was just Skype.
<wgrant> Rather stuttery.
<lifeless> yah
<lifeless> but wrkoing
<wgrant> True
 * wgrant narrows eyes
<wgrant>   [r=deryck][bug=922741] Remove ec2 scripts from launchpad tree
<wgrant> added: lib/lp/codehosting/scripts/tests/test_upgrade_all_branches.py
<wgrant> wallyworld_: Feel like reviewing https://code.launchpad.net/~wgrant/launchpad/bug-950502/+merge/96701?
<lifeless> stub: good mornink
<stub> morning
<lifeless> wgrant: hah, dirty patch?
<lifeless> stub: whenever you are caffeinated etc
<stub> Now is fine. Skype?
<lifeless> yupyup
<wgrant> lifeless: Hm?
<lifeless> wgrant: the ec2 + upgrade all thing
<wgrant> lifeless: The ec2 removal was a prereq for the upgrade all thing.
<lifeless> how so ?
<wgrant> But they were landed together with a rather misleading message.
<wgrant> ec2 depended on system bzrlib plugins
<wgrant> upgradeall reworks how we load plugins, which ends up shadowing the system ones.
 * wgrant hacks SSO to pieces.
<StevenK> wgrant: http://pastebin.ubuntu.com/875545/
<wgrant> StevenK: Or just change makeAccessPolicy to use a non-default type.
<wgrant> And you're neutering the other tests somewhat, as you're no longer testing various types.
<StevenK> Well, sure, but the APs *are* there
<StevenK> So if I can use them, it's a good idea
<wgrant> The fist hunk is also wrong.
<wgrant> find() returns a list, not a policy
<StevenK> Right, that fixed test_updateDistroSharee
<wgrant> And the other one is pretty obvious.
<StevenK> Iterate over getUtility and return a tuple
<wallyworld_> wgrant: i was out to lunch with bigjools. i'll look now
<wgrant> Is there something going on there we should know about? :)
<StevenK> Haha
<StevenK> bigjools was saying that he was still thinking about wallyworld_ after getting home.
<StevenK> Maybe I should ring Belinda.
<wallyworld_> she was watching
<StevenK> Only watching? :-P
<bigjools> rubbing
 * StevenK puts himself up as OCR, so he can review his lunch.
<wallyworld_> wgrant: i think you should add the new flag to flags.py in your branch
<wgrant> wallyworld_: Mmm, I deliberately chose not to, but I guess I might as well.
<wallyworld_> wgrant: well, if there's a good reason not to
<wgrant> I'm lazy and it's a short-lived flag? :)
<StevenK> wgrant: Did you want me to not remove InformationType.USERDATA from test_create and test_find ?
<wgrant> StevenK: That was there to test that it actually filtered properly by type, not just by pillar.
<StevenK> I can add UNEMBARGOEDSECURITY instead if you wish
<wallyworld_> wgrant: for clarity,and given it's a few lines of cut and paste, you should add it i think
<wgrant> wallyworld_: Done
<wallyworld_> thanks :-)
<wgrant> lifeless: Do you know if Orange plans to remove the old bug listings eventually?
<wallyworld_> wgrant: r=me. thanks
<wgrant> wallyworld_: Thank you.
<lifeless> wgrant: I believe its on deryck's plate, as is removing the in-browser results cache
<lifeless> wgrant: remind me tuessday if you want me to ping him about it
 * StevenK stabs buildbot a few times with a rusty knife.
<wgrant> lifeless: It's fairly large and annoying tech-debt, so it'd be nice if it went away.
<StevenK> Because having a CI system that's not INSANE is too hard.
<lifeless> wgrant: remind me tuessday if you want me to ping him about it
<wgrant> k
<wgrant> Yay, xmlrpc-private 99% down from 1.7s to 0.88s in two weeks
<wgrant> Next stop 0.3s.
<StevenK> If I can't deploy this DB patch tonight due to buildbot losing it's mind, I'm going to be very very cranky.
<mwhudson> wgrant: what's the min and median for xmlrpc-private?  if you have that?
<wgrant> mwhudson: I don't think we have those. But there's mean.
<wgrant> https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-categories.html and https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-pageids.html
<wgrant> Interesting
<wgrant> That says the 99% was 0.74
<wgrant> But lpstats says 0.88
<wgrant> Maybe lpstats lags more than a day behind.
<StevenK> Oh damn it
<wgrant> (mean 0.14)
<mwhudson> mean of 0.14, not bad
<StevenK> Updating devel has made bin/ec2 disappear
<wgrant> StevenK: bzr branch lp:lp-dev-utils somewhere
<wgrant> somewhere/ec2 land
<StevenK> Well, duh
<nigelb> StevenK: it would have been more fun if it lost rf-get
<wgrant> mwhudson: CodeImportSchedulerAPI has a 99% of 0.68
<wgrant> mwhudson: Which is a worry
<wgrant> Considering what it does.
<StevenK> No, rf-get is copied to /usr/local/bin by rf-setup
<StevenK> So you can't lose it on purpose
<nigelb> AH
<mwhudson> wgrant: well, it used to time out a lot ...
<wgrant> mwhudson: Sure, but there's no justification for it to take more than 20ms.
<mwhudson> or was that the other bit
<mwhudson> wgrant: heh well, can you get any xml-rpc method to take less than 100ms?
<wgrant> There's 60-70ms of overhead
<wgrant> Which I'm going to tackle soon
<mwhudson> my impression (from a few years ago) was that the maximum performance from xml-rpc methods was not very good
<wgrant> But <100ms is doable at present, yes.
<mwhudson> ah ok
<StevenK> In fact, Aaron broke it
<StevenK> bin/ec2 still exists
<wgrant> StevenK: It doesn't still exist.
<wgrant> Only in your existing branch.
<wgrant> Because buildout doesn't delete things.
<mwhudson> 70ms of overhead is still pretty rubbish of course, i hope you can make inroads there
<wgrant> mwhudson: That's the plan.
<StevenK> Right
<mwhudson> ironically the twisted-based authserver had a way better base line
<wgrant> mwhudson: AuthServerAPIView is 0.32/0.05
<wgrant> So 50ms is doable.
<wgrant> Which is interesting.
<wgrant> But the queries in there should be roughly 1-3ms.
<wgrant> So there's still a fair bit of overhead.
<mwhudson> wgrant: i wonder if codeimportschedulerapi rollbacks relatively frequently
<wgrant> I really want to throw Zope out a window and use Pyramid instead.
<StevenK> That would be a fun branch to review.
<wgrant> mwhudson: It's possible.
<wgrant> mwhudson: But given the way it works, I wouldn't expect it to conflict with the scanner much.
<wgrant> And not much else should be locking rows for long very often.
<mwhudson> wgrant: well
<mwhudson> wgrant: there are 4 machines that call it, right?
<wgrant> Surely not.
<wgrant> Hmm.
<wgrant> */1 cronjob, I guess...
<wgrant> plausible
<mwhudson> right
<StevenK> wgrant: make clean && make and bin/ec2 still exists
<mwhudson> it could also be missing an index i suppose
<wgrant> StevenK: Does make clean actually remove stuff from bin/?
<StevenK> steven@liquified:~/launchpad/lp-branches/devel% ls -lh bin/ec2
<StevenK> -rwxr-xr-x 1 steven steven 616 2012-03-09 15:23 bin/ec2
<StevenK> steven@liquified:~/launchpad/lp-branches/devel% make clean >/dev/null
<StevenK> steven@liquified:~/launchpad/lp-branches/devel% ls -lh bin/ec2
<StevenK> ls: cannot access bin/ec2: No such file or directory
<wgrant> mwhudson: I guess that's actually not implausible, given how small the table is.
<mwhudson> ah right, codeimportjob is kept small
<mwhudson> i guess there is some query you can run to find out if it's being seqscanned
<StevenK> wgrant: So, yes, it does.
<mwhudson> oy oy oy sqlobject methods
<lifeless> (wifi fail)
<StevenK> Sure sure
<lifeless> StevenK: deploying a db patch on friday - keen :)
<wgrant> StevenK: Lies
<lifeless> mwhudson: the min and median are on the PPR
<wgrant> Ah, median is, but I can't see min...
<mwhudson> lifeless: oh right, i see median now, don't see min though
<lifeless> connection reset from SSO. Hmmm.
<StevenK> wgrant: It does!
<wgrant> StevenK: I'd say your devel is out of date.
<StevenK> No, Aaron didn't fix setup.py
<wgrant> Possibly not, but I did.
<StevenK> steven@liquified:~/launchpad/lp-branches/devel% bzr revno
<StevenK> 14926
<wgrant> Revision: 14927
<wgrant> Commit Message: [r=wgrant][no-qa] Remove ec2 leftovers.
<StevenK> I bet you fixed while I was putting together a branch to do the same thing, you cheater.
<wgrant> Busted.
<StevenK> However, lp-dev-utils/ec2 is broken
<wgrant> sudo apt-get install python-tz python-boto
<wgrant> bzr branch lp:bzr-pqm ~/.bzrlib/plugins/pqm
<wgrant> s/bzrlib/bazaar/
<StevenK> Nice to know it just works out of the box
<StevenK> </sarcasm>
<wgrant> At least it's not SSO or Launchpad :)
 * StevenK removes his local branch that destroys the ec2 leftovers and glares at wgrant.
<wgrant> :)
<wgrant> Like I'm going to let you get the LOC credit.
<StevenK> I was about to commit and push!
<lifeless> oh look
<lifeless> *that* was ADSL
<lifeless> fail
 * StevenK brands lifeless' connection as Abominable Digitial Subscriber Line.
<lifeless> mwhudson: https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-categories.html
<jamesh> lifeless: fyi, I put out a new release of pygpgme last night.  The main new feature is Python 3 compatibility, which I doubt you need yet but thought you might want to know
<lifeless> jamesh: thanks for the headsup
<lifeless> wgrant: LOC for what?
<jamesh> lifeless: while the package should still support older Python versions, the test suite probably requires 2.6 now though
<lifeless> wgrant: LOC for what?
<wgrant> lifeless: I beat StevenK to deleting some ec2 remains.
<lifeless> wgrant: ah,c ool
<lifeless> I thought that was address to me, for some reason
<wgrant> SSO's dependency system might be even worse than Launchpad's, I think.
<lifeless> !cite
<wgrant> It's very good at branching the same stuff from LP every time you ask it to update deps.
<StevenK> Hah
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<czajkowski> morning
<wgrant> lifeless: What do you think about pages like https://launchpad.net/ubuntu/+ppas? The slow stats can all reasonably be publicly cached for a day, or we could just remove them.
<bigjools> hey poolie, did you want to talk about that bug?
<lifeless> wgrant: so, I think the page should be just a search widget, without that ridiculous 'show me empty ppas'
<lifeless> wgrant: the stats aren't (IMNSHO) interesting, nor are the lateset uploads to any ppa
<lifeless> activity -might- be interesting, but probably best as a ranking component for search results
<lifeless> the supported series list probably wants to be something like
<lifeless> 'PPAs support <policy statement>. The current list of Ubuntu versions that can be targeted in a PPA build is <compact list>'
<wgrant> lifeless: Latest uploads might be vaguely interesting, and are quick to calculate.
<adeuring> gd mrning
<lifeless> wgrant: 'might' isn't a good reason for clutter
<wgrant> lifeless: Lies.
<stub> You don't need a reason for clutter.
* adeuring changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: adeuring | Firefighting: - | Critical bugtasks: 4*10
<czajkowski> salgado: mrevell any idea on when the roll out will now happen for the blueprints
<mrevell> czajkowski, The migration of Ubuntu work items will take place, most likely, after the 12.04 release, at a time to suit the Ubuntu Engineering team and wider Ubuntu community. So, not sure of an exact date.
<czajkowski> will it happen before UDS ?
<czajkowski> as bleprint creation will start once 12.04 is released for next release and for uds sessions
<mrevell> czajkowski, I think that's a decision for the Ubuntu community. We'll fit with whatever schedule works for them. It's possible that the work items feature could go live before we migrate the existing Ubuntu blueprints to the new format.
<czajkowski> nods ok
<czajkowski> thanks
<deryck> Morning, all.
<czajkowski> deryck: morning get your machine all up and running
<deryck> czajkowski, hey. unfortunately, still busted launchpad-developer-dependencies on precise for me.
<deryck> will be following up to my email shortly.
<czajkowski> :(
<wallyworld_> sinzui: with the sprite issue, if you goto a bug page eg https://bugs.launchpad.net/launchpad/+bug/950562 you can see examples in the bug title and security portal
<_mup_> Bug #950562: Sharing service needs to support 'Some' <disclosure> <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/950562 >
<wallyworld_> i'll revert the css though, but i think there's an underlying css issue we have
<sinzui> wallyworld_, I see gecko has issues, webkit does not!
<wallyworld_> ffs
<sinzui> wallyworld_, We will look at a proper fix next week. I agree this needs fixing and I have some ideas
<wallyworld_> ideally we need to alter our css to address these x-browser issues ?
<sinzui> we often need to add a reset rule to before we apply our rule, or we look for a proprietary selector/property that compensates for the mis-aligned background
<wallyworld_> ok
<wallyworld_> maybe one day there quirks will go away
<wallyworld_> these
<jcsackett> wallyworld_: i think a nontrivial part of the web economy thrives on the quirks. don't bet on them ever vanishing. what you call a pain in the ass, someone else calls "competitive advantage". :-P
<wallyworld_> lol
<wallyworld_> sadly true
<jcsackett> yup. it's only funny cause it's heartbreaking. :-P
<wallyworld_> yeah, i wanted the sharing page to look real nice like
<wallyworld_> and truncated sprites messed it up :-(
<jcsackett> the sprites things is so irritating.
<sinzui> Looks like firefox does not believe line-height is inclusive with the space allocated to the padding. this looks like a violation of the box model
<jcsackett> sinzui: able to chat at say 10?
<wallyworld_> stupid firefox
<sinzui> yes, but it need to get more coffee. This cup killed a passing hnat
<sinzui> gnat
<sinzui> wallyworld_, keep you sprite hack!
<wallyworld_> really?
<sinzui> I agree it fixes firefox. It does change the webkit layout. subtly, but I think you need to look hard to see the spacing change.
<wallyworld_> i could make it 2 or 3 pixels instead of 4
<wallyworld_> just enough to fix the issue
 * sinzui checks
<wallyworld_> i tried 3 and i think it worked ok
<sinzui> keep 4
<wallyworld_> will do. i'll fix the other issues in the morning and send off the ec2.
<sinzui> I I think the actual space should be 6 (18 line-height - 12 font-size = 6px)
<wallyworld_> to
<wallyworld_> ok, so change it to 6
 * sinzui checks
<sinzui> wallyworld_, yes, use 6px. webkit has already allocated that same space as a part of the box allocated to the padding
<wallyworld_> ok, will do. thanks for the help. i'm pleased this will fix other pages as well as sharing
<sinzui> wallyworld_, The edit icon next to a header moved slightly in webkit (different line-height rules), but there is no change in normal budy text
<sinzui> This fixes the bug page and the project front page
<sinzui> PS, it is after midnight for you. It is Saturday. have you been drinking?
<wallyworld_> cool. i'll also address the other anchor stuff you mentioned
<wallyworld_> no, sadly not.
<wallyworld_> i checked email to see if you had +1ed my mp
<wallyworld_> cause i want to get everything landed
<sinzui> I am doing that now. I am commenting about my reverse decision
<wallyworld_> sadly, we can't turn on the fflag since edits will mess up stuff
<wallyworld_> and the sharing page is kinda useless if you can't edit stuff
<sinzui> I think pointing people to staging and qastaging is good enough for the curious
<wallyworld_> sounds ok to me
<wallyworld_> deryck: in the dark recesses of my mind, i think i had to manually install postgresql-8.4-debversion to allow the lp database packages to be installed
<wallyworld_> but that issue may have been fixed since then
<wallyworld_> that's the only thing i can contribute to your problem
<deryck> wallyworld_, yeah, I did try that. Just tried again now and get: postgresql-8.4-debversion : Depends: postgresql-8.4 but it is not installable
<deryck> wallyworld_, and hey, btw. :) How's everything going? :)
<sinzui> deryck, you need to pull the packages from oneiric
<sinzui> They Lp is running on obsolete PG
<deryck> sinzui, ah.  and how do I do that? Sorry to be so dumb about this.
<sinzui> jcsackett, I am now ready
<sinzui> sorry for the dealy
<sinzui> I download them from Lp. I think you need for.
 * sinzui looks
<jcsackett> sinzui: no delay at all. i asked about 10, it's now 10. :-)
<czajkowski> sinzui: the licience review of a project which had a made up licience I take it you denied it ?
<deryck> sinzui, ah, I get you now. I can chase down the packages myself now. Thanks.
<sinzui> deryck, See the binary packages listed on https://launchpad.net/ubuntu/oneiric/+source/postgresql-8.4
<sinzui> We want postgresql-plpython-8.4
<sinzui> postgresql-client-8.4
<sinzui> postgresql-contrib-8.4
<sinzui> postgresql-doc-8.4
<sinzui> postgresql-plpython-8.4
<jcsackett> sinzui: hangout, mumble, skype?
<sinzui> hangout
<jcsackett> sinzui: cool. starting one now.
<deryck> sinzui, many thanks!
<stub> Aren't all those packages in the PPA?
<stub> oh.. thinking of the lucid backports
<jcsackett> sinzui: you vanished! :-P
<sinzui> I did.
<sinzui> Phone rang
<jcsackett> dig.
<jcsackett> i think we were done though, so all is well. thanks. :-)
<sinzui> fab
<gary_poster> deryck, btw, re your emails about LP on precise, you can develop LP with lxc on precise, and then you are using lucid packages. can be nice.  that's what I do.  You don't pollute your main system that much either--getting rid of LP would mean getting rid of the LP directories in your home dir and getting rid of the associated LXC instance(s).  We can help if you'd like.  There's a wiki page about doing it manually,
<gary_poster> and a script to give some automation.  I'm guessing you are past the point that this would have been helpful, but I still wanted to mention it
<sinzui> gary_poster, your remark implies you do not know about .rocketfuel-env.sh http://pastebin.ubuntu.com/876388/
<sinzui> I have never had my home directory polluted
<deryck> gary_poster, ah, that actually might be nice.  I'm *more-or-less* working. :)  But if I could get into the more category than the less, might be worth it. :)
<deryck> gary_poster, can you point me at the wiki page?
<gary_poster> sinzui, I was vaguely familiar with that, but I don't mind having stuff in my home directory.  The pollution I mind is having everything else installed on my system (postgres being taken over, tons of packages installed, and so on)
<gary_poster> but to each his own in this regard I'd say :-)
<gary_poster> and thank you
<gary_poster> deryck, https://dev.launchpad.net/Running/LXC
<gary_poster> deryck, we use automated scripts now, so we haven't touched this in a while, but I think it will still be good
<deryck> gary_poster, awesome. I'll have a look now.
<gary_poster> cool
<sinzui> does that have wgrants and wallyworld_'s updates? the instructions did not work for precise last week
 * sinzui looks
<sinzui> well I has Ian's and I think that means he got it running
<sinzui> deryck, keep in mind Lp will be running Precises in production in a few months. Lp engineers we need enough engineers running precise to be sure we can switch over without dedicating a team to make it happen in the chain
<deryck> sinzui, I was actually just thinking about that.
<gary_poster> sinzui, deryck, you can also create a precise container
<gary_poster> lxc is still nice
<sinzui> gary_poster, yes. I agree that keeping out ridiculous Lp deps in a box is best.
<sinzui> deryck, You are doing a fresh install aren't you. My upgrade to precise was easy...given that I have learned to pin packages to ensure dist-upgrade does not remove my income.
<deryck> sinzui, not really a fresh install.  My laptop came preinstalled with Oneiric, then I got LP running, then I upgraded to Precise....
<deryck> sinzui, however, I didn't pin packages, so it removed all of lp, which is what got me in this shape.
<deryck> well, all of lp deps.
<sinzui> deryck, right update removed all PPA from dep checks to ensure the update goes well. You have to say no to all removes. You can them pin the one you need see issues with on reboot
<sinzui> I have pgsql and rabbitmq pinned
 * sinzui high fives StevenK for demonstrating that Lp can give any project a complimentary 30 day commercial subscription
#launchpad-dev 2012-03-11
<lifeless> grah, sick. :(
<czajkowski> lifeless: not a good way to be
<wallyworld_> StevenK: if you do qa we could do a deployment of many revisions :-)
<StevenK> wallyworld_: Still trying to work out *how* to QA it.
<wallyworld_> hah
<czajkowski> lifeless: is this a feature that lp could look at implementing, https://bugs.launchpad.net/python-oops-wsgi/+bug/949013
<_mup_> Bug #949013: Can't access oops-id from django <python-oops-wsgi:New> < https://launchpad.net/bugs/949013 >
#launchpad-dev 2013-03-04
<StevenK> wgrant: I've gotten it from 88 queries for 50 private BMPs to 40.
<wgrant> StevenK: What're the rest?
<StevenK> Trying to determine, since the test insists on printing a code traceback for each query
<StevenK> wgrant: They look plausible
<StevenK> wgrant: I can pastebin them if you wish
<wgrant> Oh, that's for the whole page?
<wgrant> That's not bad
<StevenK> Yeah
<StevenK> Except it turns out it only shows 7 BMPs per batch
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/product-merges-preload/+merge/151410
<wgrant> StevenK: Done
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/notify-when-spec-renamed/+merge/151419
<StevenK> noodles775: O HAI. Are you able to QA your changes on qastaging at some stage today?
<noodles775> Hi StevenK - I thought I put qa notes on the bug?
 * noodles775 checks.
<noodles775> StevenK: yep, check if the notes linked in the description are ok: https://bugs.launchpad.net/developer-portal/+bug/1088527
<_mup_> Bug #1088527: Needlessly asks for exact package name, file size, uploaded version <ca-escalated> <qa-needstesting> <Developer registration portal:In Progress by michael.nelson> <Launchpad itself:Fix Committed by michael.nelson> < https://launchpad.net/bugs/1088527 >
#launchpad-dev 2013-03-05
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/notify-when-spec-renamed-redux/+merge/151678
<wgrant> StevenK: k
<StevenK> wgrant: Too late
<StevenK> self-reviewed and landed
<wgrant> Ah, so you did
<StevenK> Looking for another critical
<StevenK> Open for suggestions
<StevenK> wgrant: It's essential? How?
<StevenK> NBS doesn't apply
<wgrant> StevenK: Why not?
<wgrant> The UI only exposes source deletion
<wgrant> But sometimes you need to delete binaries
<wgrant> Because they're left over even though their source is long gone
<wgrant> So you need to be able to delete an already dead source, if it has live binaries
<StevenK> wgrant: Sure. But IArchive.getSourcesForDeletion will only include sources if and only if it has a published binary.
<StevenK> If we can destroy that check it might actually be performant, because then it will only require SPPH and SPN
<wgrant>         clauses.append("""
<wgrant>            (%s OR SourcePackagePublishingHistory.status IN %s)
<wgrant>         """ % (has_published_binaries_clause,
<wgrant>                quote(source_deletable_states)))
<wgrant> OR
<StevenK> We still pull in SPR
<StevenK> Always
<wgrant> Sure
<StevenK> And the EXPLAIN ALANYZE for the query makes me cry blood.
<wgrant> How's that relevant?
<wgrant> Sure
<wgrant> The plan is bad
<StevenK> Modulo spelling
<wgrant> That doesn't mean the idea of the query is
<StevenK> wgrant: So it has to stay? :-(
<wgrant> We need a way to delete NBS binaries
<wgrant> The way we do that today is by permitting deletion of a source if it's published, or if it has published binaries.
<StevenK> wgrant: Then we're stuck with SPR. :-(
<wgrant> StevenK: SPR isn't the bad bit
<wgrant> But yes
<StevenK> wgrant: Oh? The EXISTS?
<wgrant> Well, think about what the query has to do
<wgrant> I haven't explained it
<wgrant> But it is fairly obvious what the issue is
<StevenK> FROM SourcePackageName,
<StevenK>      SourcePackageRelease,
<StevenK> ...
<StevenK> LEFT JOIN SourcePackageRelease AS "_prejoin1" ON SourcePackagePublishingHistory.sourcepackagerelease = "_prejoin1".id
<StevenK> I'm not sure if Postgres enjoys that much
<StevenK> wgrant: Which I seem to be missing.
<wgrant> StevenK: Consider what the query is doing
<wgrant> The root condition is very expensive
<StevenK> SourcePackagePublishingHistory.archive = 24522 ?
<wgrant> It's asking for every SPPH in this archive that is published, or has published binaries
<wgrant> Now, finding the published ones is relatively cheap
<wgrant> It's somewhere around O(number of published sources), which is the dataset we actually care about
<wgrant> So that's fine
<wgrant> Finding the published binaries for a source is also *reasonably* inexpensive
<wgrant> But the problem is that we have to find the published binaries for not just every source
<wgrant> Er
<wgrant> Not just a source
<wgrant> And not just the published sources
<StevenK> But every source
<wgrant> But every source every in the history of the archive
<wgrant> So it's O(entire source history)
<wgrant> For a fairly slow operation
<wgrant> Can you think of any potential optimisations?
<wgrant> There might be one we could sort of do.
<StevenK> CTE of published sources won't help, because we want to include deleted sources
<wgrant> Right
<wgrant> Finding published sources is easy
<wgrant> Finding sources with published binaries is not, because we have to join through several tables and filter from both ends.
<StevenK> Right
<StevenK> wgrant: Can we check status first and only drop to EXISTS if we need to?
<wgrant> StevenK: That won't help in most cases
<wgrant> StevenK: You'll still have to do the published binaries check for every source that is not published
<StevenK> And we probably want to restrict the EXISTS to BPB.source_package_release = SPPH.spr
<wgrant> Which will usually vastly dominate the number of published sources for which you can skip the check
<wgrant> Why?
<wgrant>             SourcePackagePublishingHistory.sourcepackagerelease =
<wgrant>                 SourcePackageRelease.id AND
<wgrant>                 BinaryPackageBuild.source_package_release =
<wgrant>                     SourcePackageRelease.id)
<wgrant> That's already done
<StevenK> Even in the subquery?
<wgrant> SourcePackageRelease isn't rebound in the subquery
<wgrant> That's why it works at all
<wgrant> Rather than just selecting every publication in the archive ever
<StevenK> I'm also unsure why this query bothers to link against SPR if it's in the FROM spec
<StevenK> s/link/JOIN/
<StevenK> wgrant: So the subquery can't be restricted?
<wgrant> HMm?
<wgrant> It doesn't specifically JOIN it
<wgrant> Restricted how?
<StevenK> LEFT JOIN SourcePackageRelease AS "_prejoin1" ON SourcePackagePublishingHistory.sourcepackagerelease = "_prejoin1".id
<wgrant> Ah, a prejoin
<StevenK> And the code has preJoins = ['sourcepackagerelease']
<wgrant> That's unrelated
<wgrant> Ignore it
<StevenK> wgrant: That we can't restrict the EXISTS subquery against the SPR in the outer query
<wgrant> It is
<wgrant> Otherwise it would return every published binary in archive, rather than just binaries for the relevant source
<wgrant> aka. completely broken
<StevenK> Then I'm not sure :-(
<wgrant> Without completely redesigning the UI around handling NBS binaries, the only potential optimisation I can think of is to restrict to sources with a null scheduleddeletiondate.
<wgrant> Because we don't condemn sources while their binaries are alive
<wgrant> For GPL compliance
<wgrant> That *may* be enough to fix it in most cases.
<wgrant> (for now)
<StevenK> wgrant: In the outer query?
<wgrant> StevenK: Probably
<wgrant> For the other case it doesn't really matter
<wgrant> since if we have a Published source that's condemned then we have a big problem
<StevenK> wgrant: That runs quickly on qas, but I'm not sure if its still hot enough
<wgrant> You'll also want to find the most pathological archive
<wgrant> And see if it still works
<wgrant> Well, most pathological PPA
<StevenK> 32 seconds
<StevenK> For ~ubuntu-langpack's PPA.
<StevenK> wgrant: So I guess the SourcePackagePublishingHistory.scheduleddeletiondate IS NULL doesn't help
<wgrant> Have you checked the plan?
<StevenK> wgrant: http://pastebin.ubuntu.com/5587192/
<wgrant>                            ->  Index Scan using securesourcepackagepublishinghistory__archive__status__idx on sourcepackagepublishinghistory  (cost=0.00..69803.45 rows=51405 width=135) (actual time=0.129..2090.566 rows=3060 loops=1)
<wgrant>                                  Index Cond: (archive = 32)
<wgrant>                                  Filter: (scheduleddeletiondate IS NULL)
<StevenK> Ah
<StevenK> wgrant: So we need another index?
<StevenK> I thought index scans with filters were still cheap-ish
<wgrant> StevenK: Think about what we're trying to do.
<wgrant> "Cheap-ish" depending on the selectivity of the filter
<StevenK> Right
<mpt> https://launchpad.net/~78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwfnr8jn6ln0m3jxne2k9x1ohg85w3jabxlrqbgszpjpwcmvkb-launchpad-a811i2i3ytqlsztthjth0svbccw8inm65tmkqp9sarr553jq53in4xm1m8wn3o4rlwaer06ogwvqwv9mrqoku2x334n7di44o65qze67n1wneepm is a good test for Launchpad page layouts...
<czajkowski> mpt: lots of them around
<StevenK> 475
<StevenK> I'm tempted to re-seed them via their e-mail address so they get renamed to an account name that isn't completly and utterly bonkers.
<RoelV> dear launchpad devs! I have a question
<RoelV> we currently have a problem where we would like an additional "importance" status
<RoelV> (Percona)
<RoelV> so that we can register "QA blocking bugs" - so a status would be "QA blocking"
<RoelV> or even simply "QA request"
<RoelV> we would not like to use tags for this
<RoelV> is there any way to do this - for example by using some API, patch LP code, ... ?
<czajkowski> RoelV: http://blog.launchpad.net/bug-tracking/new-bugs-status-opinion
<RoelV> czajkowski, thanks! so iow "it ain't going to happen?"
<RoelV> czajkowski, any other "best solutions/practices" you can think off?
<czajkowski> besides dropping a mail to the list no afraid not
<czajkowski> but the blog post sums it up tbh
<RoelV> czajkowski, yep thanks
<lifeless> wgrant: StevenK: rabbitfixture is missing a LICENSE file
<wgrant> RoelV: Why not just use tags?
#launchpad-dev 2013-03-06
<RoelV> hi launchpad devs
<RoelV> is this branch stuck
<RoelV> https://code.launchpad.net/~roel11/percona-server/bug-1144059-5.6
<cjwatson> RoelV: Yes, the branch scanner seems to have OOPSed while scanning it.  The simplest workaround is probably to delete the branch and re-push
<RoelV> cjohnston, thanks
<RoelV> cjohnston, in progress :)
#launchpad-dev 2013-03-07
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/hide-build-portlet-archive/+merge/151864
<wgrant> StevenK: Have you confirmed that there are no similar cases?
<StevenK> On Archive:+index? Yeah
<wgrant> StevenK: Great
<wgrant> While you're on the PPA pages...
<wgrant> Bug #1149544
<_mup_> Bug #1149544: PPA +packages timeout when too many package copy errors are shown <Launchpad itself:New> < https://launchpad.net/bugs/1149544 >
<wgrant> 15s or so of non-SQL time
<wgrant> Rendering PCJ errors
<wgrant> Seems a bit odd
<StevenK> wgrant: I have a second branch on archive pages already
<StevenK> wgrant: I'll look at that bug after I clear my plate. That MP is first on my list.
<wgrant> StevenK: I'm not sure it's worth having display_latest_updates, rather than just using context/required:launchpad.View in the template
<StevenK> wgrant: Oh, I didn't know about that pattern.
 * StevenK tries to swap back in the changes for that branch
<StevenK> wgrant: http://pastebin.ubuntu.com/5591925/
<StevenK> wgrant: Any other issues?
<wgrant> StevenK: Seems fine otherwise
<StevenK> Strange that r16519 turns up as an unmerged revision
<wgrant> The scan of the last landing probably failed
<StevenK> Indeed
 * StevenK stabs the branch scanner more
<wgrant> Was that the spec notification fix?
<StevenK> Yeah
<StevenK> name => Name
<StevenK> wgrant: MP updated
<wgrant> StevenK: Done
<wgrant> Thanks
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/preload-subscriptions/+merge/152073
<wgrant> StevenK: We do not want to link to Archive:+index
<wgrant> As it's completely useless to the subscriber until they've activated their sub to get a token
<StevenK> wgrant: Gar.
<StevenK> wgrant: So I have to cache a negative permission
<wgrant> StevenK: An easy change
<wgrant> However
<wgrant> You'd need to actually do the view check as well
<wgrant> Or you can just fix +archivesubscriptions so it never links :)
<StevenK> That sounds good
<StevenK> It uses subscription/archive/displayname and subscription/archive/fmt:reference
<StevenK> So the PPAFormatterAPI is doing the linking
<wgrant> Are you sure that it actually does link?
<wgrant> I don't get links
<wgrant> And I can see every PPA
<StevenK> It looks up View to see if it should link
<wgrant> Are you sure?
<wgrant> Because it doesn't link even though I do have View
<StevenK> The code that's on prod is odd
<StevenK> It checks if you're a subscriber if it's private
<StevenK> Oh
<StevenK> It doesn't link
<StevenK> But it should still check permission so it doesn't leak
<wgrant> Ah, right
<wgrant> So, that bit could be adjusted to SubscriberView
<StevenK> Right
<StevenK> wgrant: http://pastebin.ubuntu.com/5592006/
<wgrant> StevenK: Sounds reasonable.
<StevenK> wgrant: Can haz +1?
<wgrant> StevenK: Done
<StevenK> wgrant: Current plan is to deploy after r16521 is QA'd.
<StevenK> wgrant: One failure in tales.txt
<StevenK> wgrant: Trying to fmt:link a disabled ppa for anonymous results in <BLANKLINE> now, rather than <span>displayname</span>
<wgrant> StevenK: Hm, I guess you'll need to reintroduce the private special case
<StevenK> 280 PCJs makes me sad
<StevenK> wgrant: I wonder if we want to limit the view method to only return 5 and show somehow that there are X others
<wgrant> StevenK: Quite possibly, yes
<wgrant> But it's also worth investigating why they take so long to render without queries
<StevenK> Because TAL hates life?
<StevenK> wgrant: The view method that drags them out of the DB is quite good and preloads everything
<wgrant> Sure, but what sort of data do they have?
<wgrant> Can't be that huge
<StevenK>                   tal:attributes="class python: 'failed' if job.status.title=='Failed' else 'non-failed'"
<StevenK> That can't be helping
<wgrant> Why not?
<StevenK> Let's see. It pulls out job.status, job.package_name, job.package_version, job.source_archive, job.requestor, job.target_distroseries and job.error_message
<StevenK> wgrant: OOPS-7ed453c05c38c1f860dcf3c63e47eb6c
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/preload-subscriptions-redux/+merge/152302
<wgrant> StevenK: k
<cjwatson> wgrant: existing_publishing_status> what would a legal method name be?
 * cjwatson digs through style guides
<cjwatson> Oh, whoops, you mean existingPublishingStatus?
<wgrant> cjwatson: Technically it should be a verb too, but there is precedent for just mixedCasing it, yes
<cjwatson> OK.  In an earlier version it was just a class attribute, hence the misnaming
<cjwatson> Sod it, I"
<cjwatson> 'll be lazy and prepend "get".  There's precedent in the same file
<wgrant> Yup
<cjwatson> Now, will bzr-pqm work on this new machine ...
<cjwatson> Yes.  Excellent.
#launchpad-dev 2013-03-08
<mpt> czajkowski, if I hit "Submit Bug Report" on this Launchpad bug, do I have to give those Easter eggs back?
<mpt> oops, too late
<czajkowski> mpt: I can be nice you know no more eggs for you mister!
<mpt> It's a regression, but a trivial one
<czajkowski> nods
<czajkowski> all bugs are trivial if yuo have enough people working on them
<czajkowski> :)
<czajkowski> we have two
<jelmer> just two bugs? must not be hard then
<czajkowski> jelmer: \o/
<czajkowski> jelmer: ging to give a talk tomorrow, have lots of foods, hping people turn up
<czajkowski> 67 people signed up but you know not everyone turns up :/
<czajkowski> so a little worried
 * jelmer will be there :)
<czajkowski> \o/
<jelmer> czajkowski: what's your usual experience with the ratio of people who sign up vs how many actually come?
<czajkowski> litle over half
<czajkowski> and that'd be good as the first one I've run in the UK tbh
<czajkowski> loads of space to hack on on stuff and I do have people already wanting to give talks which is good
<czajkowski> and xnox is also coming along :D
<cjwatson> wgrant: I needed to upgrade mawson for QA, and there was a fairly substantial diff there including a merge of lp:~wgrant/launchpad/renovate-deathrow.  I saved the diff in ~launchpad/diff-20130308 and reverted it.
<wgrant> cjwatson: Oops, thanks.
 * czajkowski peers at wgrant 
<czajkowski> why are you here at this hour
<czajkowski> nothing is broken
#launchpad-dev 2013-03-09
<smartboyhw> Finding some bugs to play with (fixing) ca n I do Bug 1137716?
<_mup_> Bug #1137716: Code page suggests using invalid bzr flag --use-existing <bzr> <lp-code> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1137716 >
<cjwatson> smartboyhw: Generally I think you can just go ahead, at least for simple things like that that don't involve major reengineering
<smartboyhw> cjwatson, :)
<smartboyhw> cjwatson, done and merge proposal proposed.
<smartboyhw> How come the merge proposal went timeout!?!!?!??!
<cjwatson> I don't know.  Do you have an OOPS id?
<czajkowski> smartboyhw: whats the oops id ?
<czajkowski> \o/
<smartboyhw> czajkowski, cjwatson anyway I think it works now:P
<cjwatson> LP's own code tends to hit pathological cases in the code component, unfortunately
<cjwatson> (Lots of revisions, lots of merges)
<smartboyhw> .....
<StevenK> smartboyhw: I'd prefer a lot less brevity in the MP description.
<smartboyhw> StevenK, brevity = ?
<cjwatson> After pushing an LP branch it's often necessary to delete it after a while and push again
<cjwatson> (But not in this case - I mean if it fails to scan the branch)
<cjwatson> smartboyhw: Please follow https://dev.launchpad.net/PatchSubmission
<cjwatson> brevity = shortness
<smartboyhw> cjwatson, OK sorry
<smartboyhw> Is this Lint thing strange?
<smartboyhw> Missing ./download-cache.
<smartboyhw> Developers: please run utilities/link-external-sourcecode.
<smartboyhw> make: *** [download-cache] Error 1
<smartboyhw> smartboyhw@smartboyhw-Compaq-Presario-CQ41-Notebook-PC:~/launchpad$ ./utilities/link-external-sourcecode
<smartboyhw> Usage: link-external-sourcecode [options] [parent]
<cjwatson> You need to run it in the branch you're changing
<cjwatson> And it will help if your branches are in the conventional layout
<cjwatson> You ought to have a clean devel branch locally, and do your work in a separate local branch of that
<cjwatson> Rather than e.g. just branching lp:launchpad directly and then pushing the result back
<cjwatson> https://dev.launchpad.net/Running
#launchpad-dev 2015-03-02
<wgrant> It's too easy for a factory bug or change to invalidate the test otherwise.
<thomi> right
<thomi> wgrant: new diff ready...
<thomi> I guess I got the zcml stuff correct? I wasn't really sure about that...
<wgrant> Yep, that's right.
<thomi> is securedutility something launchpad specific? I couldn't find it in http://docs.zope.org/zope.component/zcml.html
<wgrant> Yes, that's one of ours.
<wgrant> See SecuredUtilityDirective in lp.services.webapp.metazcml
<wgrant> But in general you shouldn't have to care about the details.
<thomi> ok.. blindly copy + paste. check :D
<blr> wgrant: regarding the ugly list comprehension, were you suggesting it would be better to return a dict of dicts, rather than the list of ref dicts?
<wgrant> blr: Right, a list of dicts with a unique "name" field feels like it should be a dict.
<wgrant> (though that becomes more questionable if we ever batch that endpoint, that'll be an API break for other reasons anyway)
<blr> wgrant: yep, that would be better. Also, prepending 'refs/' will still be necessary in the view, but you're entirely right that it shouldn't in store.
<wgrant> Right, the view presumably keys on refs/, so it doesn't get that part of the path.
<blr> wgrant: well, when we need to address paging we can revisit it
<wgrant> Yep.
<blr> thanks
<wgrant> thomi: Approved with a couple of trivial fixes.
 * thomi looks
<wgrant> thomi: Can you file an RT ticket to get your OpenPGP key added to the PQM keyring, so you can land things yourself?
<wgrant> Mail it to launchpad@ and I'll approve it.
<thomi> wgrant: I file the rt by emailing launchpad@rt.canonical.com ?
<wgrant> yep
<thomi> wgrant: filed
<wgrant> Got it, thanks.
<thomi> wgrant: OK, I made those changes. Should to top-approve that MP, or would you like to look at it again?
<thomi> I guess the PQM keyring won't get updated today
<wgrant> thomi: Top-approval doesn't do anything special for us today, as we don't use Tarmac. But I tend to use it as if we did.
<wgrant> Looks fine to me. I'll land it unless you have objections to your own branch.
<thomi> no objections here :D
<wgrant> Good, I'd be worried.
<thomi> heh
<thomi> wgrant: I have another MP for you... the docs for zope_isinstance are rather lacking, so I'm not convinced that it's this easy... but the function is reasonably well tested, and the tests all pass, so... : https://code.launchpad.net/~thomir/launchpad/devel-fix-import-violations-buildfarmjob/+merge/251397
<wgrant> thomi: zope_isinstance is just an isinstance that looks through security proxies.
<wgrant> Do the tests pass? I think that change is incorrect.
<wgrant> The structure here is a little bit confusing.
<thomi> wgrant: the tests pass
<wgrant> Odd
<thomi> I'm running lp.soyuz.browser.tests.test_builder_views.TestgetSpecificJobs
<thomi> wgrant: I'm not sure what the word 'specific' means in that docstring
<wgrant> thomi: What about test_builder_views as a whole?
<wgrant> thomi: There are four types of build farm jobs. IBinaryPackageBuild, ISourcePackageRecipeBuild, ITranslationTemplatesBuild and ILiveFSBuild are all derivatives of IBuildFarmJob.
<wgrant> But, crucially, they are *specific* jobs.
<wgrant> They each have their own table named after them, and the data is primarily stored there.
<thomi> wgrant: I'm running those tests now
<wgrant> But each specific job also references a non-specific row in the BuildFarmJob table, which is used for builder histories etc. that show all job types.
<thomi> I'm still not sure what specific means in that context.
<thomi> wgrant: those tests all pass as well :D
<blr> wgrant: just as a wee experiment, all of the refs in the linux git repo, serialized are 9.6k
<wgrant> BuildFarmJob is effectively an abstract base class.
<wgrant> Except it happens to actually exist for DB reasons.
<wgrant> thomi: So, the BuildFarmJob class is this non-specific DB table, which needs to be resolved to a concrete specific job for display. But IBuildFarmJob is implemented by both specific and non-specific classes, so your check changes the behaviour.
<wgrant> I was hoping a test would fail to prove that, but to preserve the behaviour you can check against IBuildFarmJobDB instead.
<wgrant> Only BuildFarmJob itself implements IBuildFarmJobDB.
<wgrant> (BuildFarmJob should probably be renamed to BuildFarmJobDB, but historical reasons)
<wgrant> Specific jobs have a non-specific BuildFarmJob.
<wgrant> Specific and non-specific jobs implement IBuildFarmJob.
<wgrant> Only non-specific jobs implement IBuildFarmJobDB.
<thomi> so... that check should be 'IBuildFarmJob.providedBy(job) and not IBuildFarmJobDB.providedBy(job)' ?
<wgrant> zope_isinstance(job, BuildFarmJob) is only true for a non-specific job.
<wgrant> So I think it can be replaced with just "IBuildFarmJobDB.provideBy(job)"
<thomi> oh, right.
<thomi> hmmmm, I'll add a test for that failure case as well
<wgrant> This code and schema has been through three or four serious redesigns over the years, so the tests may well be dodgy.
<thomi> yeah... some of these could be better named
 * thomi gets out his yak razor
<wgrant> But basically the method is designed to take an arbitrary combination of non-specific and specific jobs, and turn them all into specific jobs.
<wgrant> Obviously for a specific job that transformation is a no-op.
<wgrant> It may be that the tests work because the non-specific and specific IDs match, which will happen if you only use one kind of job.
<wgrant> blr: We'll need to handle non-ASCII refs soon, but that can be a subsequent iteration.
<blr> wgrant: ugh, right.
<blr> I think #git exists primarily to slag off other dvcses, sadly.
<wgrant> Well, what else are they going to talk about? How good git's UI is? :)
<blr> wgrant: noooo... don't do it.
<xnox> they are quite helpful with git intrisics support.
<wgrant> Evening xnox.
<thomi> wgrant: if you have a moment: https://code.launchpad.net/~thomir/launchpad/devel-fix-import-violations-buildfarmjob/+merge/251397
<wgrant> thomi: Done.
<thomi> wgrant: I was tempted to rewrite that function, as I think i could be a lot clearer, but decided not to.
<thomi> thanks... down to 5 violations now :D
<wgrant> thomi: You should feel free to rewrite things when it makes sense to. A lot of this code is old and things have evolved around it.
<thomi> yeah... it would have been an aesthetic change only, and the tests aren't the best, so I figured it was best to leave it alone :D
<wgrant> Indeed.
<wgrant> thomi: Oh, I was about to land it for you, but blahdeblah has added you to the keyring.
<thomi> oh!
<wgrant> thomi: 'bzr lp-land --no-qa' in the branch directory should do what you want.
<wgrant> Run it from your host, not inside the container, for keyring access.
<thomi> oh, well that's easy then
<wgrant> You may need to grab bzr-pqm from the PPA, though.
<wgrant> And you'll need SMTP server config.
<thomi> the container has access to gpg keyring too...
<thomi> oh.. smtp thing will be "fun"
<wgrant> thomi: It needs launchpadlib keys too.
<wgrant> Which are in your gnome-keyring.
<wgrant> thomi: https://pastebin.canonical.com/126638/ has a superset of the relevant settings from my bzr configs.
<thomi> how come no bzr-pqm package in utopic, but exists in saucy?
<wgrant> thomi: Because bzr and LP are probably the only two projects in the world that still use PQM.
<thomi> oh ok
<wgrant> The saucy package should work, though.
<thomi> the package in ppa:launchpad/ppa should as well I guess
<wgrant> Ah yes, I copied that up, didn't I.
 * wgrant lunches.
<wgrant> thomi: Looks like you got lp-land working?
<thomi> wgrant: maybe? The branch is still 'approved'
<wgrant> (also, you don't actually need to top-approve at all; lp-land will be happy as long as it has valid reviews)
<wgrant> thomi: Heh, check your email.
<thomi> oh, right
<wgrant> You set from Merged to Approved, I guess you raced with the scanner.
<thomi> I was about 10 seconds too early :D
<thomi> ahh
<thomi> wgrant: I wanted to ask you about the final few import violations
<thomi> they're (almost) all where we're using load_related for (I think) database efficiency
<wgrant> Yup, that's what I thought they'd mostly be.
<wgrant> That's usually done with a DecoratedResultSet using pre_iter_hook.
<wgrant> Many methods in the model take an eager_load option, or a number of need_WHATEVER options to enable preloading of certain related objects.
<wgrant> Or some other sets have a preloadForWhatever method.
<thomi> I'm not sure what the fix is for that - just move the code into a SomethingSet ?
<thomi> for example, lp.soyuz.browser.archivesubscription line 334 (or thereabouts)
<wgrant> thomi: I'd fix getBySubscriberWithActiveToken. Depending on how many places it's used, you might just be able to push the load_related down into there without needing to make it optional.
<wgrant> Loading a handful of archives is relatively cheap, so it's not terrible to do it where it's not necessary.
<wgrant> But other types of preloading definitely want to be up to the caller to enable.
<thomi> hmmm
<wgrant> Do you understand what all the preloading code is for?
<wgrant> It's to prevent the ORM from issuing a bazillion queries when accessing objects related to sets of other objects.
<wgrant> Each ArchiveSubscription obviously has a related Archive, and naive code looping through some subscriptions would issue a separate query for each subscription's Archive.
<wgrant> This is not good for performance.
<wgrant> load_related collects the IDs and does a single query for them all, to get them into Storm's cache cheaply.
<thomi> somehwat unrelated, but presumably it'd only load the archive details when you accessed that part of the subscription object, right?
<wgrant> Right, code like "subscription.archive" pulls it in.
<wgrant> But it checks in Storm's cache (keyed on (class, primary key)) before making the query, which is why load_related prevents the extra query.
<thomi> ok.. I think I just need to get my head around how this code gets moved around
<wgrant> You can probably just literally move the load_related call into getBySubscriberWithActiveToken, except that security proxies may ruin your day.
<thomi> so getBySubscriberWithActiveToken would include the load_related call and throw away the result? I mean, it'd still be in the storm cache
<thomi> oh, except we use the result from load_related, but I guess if it's in the cache already we can look it up again and it should be cheap?
<wgrant> thomi: afk, but yes. once it's loaded the view can just loop through the subscriptions and get the archive out
<thomi> awesome. There are some other things I can change in here as well.
<wgrant> yep, clean away
<thomi> my brain is melting
<wgrant> :(
<thomi> ugh. given that it's EOD for me, and I turn into a CI-team pumpkin tomorrow, I might back away sloooowly from this one :D
<wgrant> thomi: heh, no worries. you got a couple of good branches landed today :)
<thomi> yeah
<thomi> wgrant: Does http://lpqateam.canonical.com/qa-reports/deployment-stable.html limit the number of grey revisions? I expected to see two of mine at the bottom of the list, but I only see one...
<StevenK> qastaging may not have updated
<StevenK> Check what rev is running on qas
<thomi> StevenK: doh - thanks, that should have been obvious :-/
<StevenK> thomi: devpad has a log of what qas is doing, but I suspect WADL'ing
<thomi> devpad.canonical.com gives me an empty directory listing :D
<StevenK> thomi: Via ssh
<thomi> oh, gotchya
<StevenK> Somewhere under /srv
<thomi> anyway, it's EOD for me, I'll care about it tomorrow if I have to :D
<blr> wgrant_: api-refs branch ready for re-review when you have a moment.
<cjwatson> wgrant: Oh, um, could you have a look at https://code.launchpad.net/~cjwatson/turnip/pastescript/+merge/251303 ?  I just realised that I merged https://code.launchpad.net/~cjwatson/charms/trusty/turnip/api-server/+merge/251305 too early.
<cjwatson> sigh Mondays
<wgrant> cjwatson: You should totally have self-reviewed that, but done.
<cjwatson> Fair enough
<cjwatson> Ta
<sil2100> Hey!
<sil2100> I'm experiencing some strange problems, I don't seem to be able to subscribe the 'landing-team-trackers' team to bug LP: #1425737
<mup> Bug #1425737: Wizard freezes on blank screen after language <vivid> <vivid-stab-candidate> <libqofono (Ubuntu):New> <ofono (Ubuntu):New> <unity8 (Ubuntu):New for mterry> <https://launchpad.net/bugs/1425737>
<sil2100> ...aaaand nevermind!
<sil2100> Scratch  that, it seems to have been some temprorary issue
<sil2100> I was worried since refreshing didn't help and I tried like 5 times
<sil2100> But now it's working again ;)
<thomi> morning
<cjwatson> Hi
<cjwatson> I was wondering if we could have our team meeting today again this week, but neglected to ask people in my morning ...
 * cjwatson grumbles at complex and not-entirely-predictable personal life conflicts but that's evening meetings for you
<thomi> cjwatson: well I'm officially a CI person this week, so don't mind me :D
<cjwatson> Aha
<cjwatson> Splitter
<thomi> and I doubt blr is up yet
<thomi> yeah
<cjwatson> I've spent half the day fighting (successfully, eventually) with juju, and the other half writing LP git xmlrpc tests
<thomi> "xmlrpc" - that's a technology I haven't heard in a long time :D
<thomi> ...reminds me of automating chyron devices
<cjwatson> Yeah, well, we don't use it for talking from LP to turnip, but when talking from turnip to LP it's a lot easier than inventing a new endpoint stack
<thomi> ahh, right
<cjwatson> (Because we already have a private XML-RPC port that's used by several privileged services)
<wgrant> cjwatson, blr: Today works for me.
<blr> morning wgrant :)
<blr> missing context, but I'm guessing that was our HO today rather than tomorrow?
<wgrant> blr: Right, Colin suggested it.
<wgrant> Though he may be gone now.
<blr> sure, I'm easy
<cjwatson> Here
<wgrant> Aha, good morning.
<cjwatson> Was just getting a bit of relaxing reading in beforehand
<blr> cjwatson: oh?
<cjwatson> "Spin State", Chris Moriarty
#launchpad-dev 2015-03-03
<blr> cjwatson: that looks fun. Last sci-fi novel I read was Delany's Dhalgren, which was rather bizarre.
<cjwatson> reading the plot summary, reminds me of a cross between Michael Moorcock and Philip K. Dick ...
<blr> fond of both of those writers, so imagine I might like it in that case.
<cjwatson> Dhalgren, I mean
<blr> oh right
<cjwatson> my wife remarks that Moorcock is best read at the speed he wrote it
<cjwatson> I'm fairly sure Dick is best read with the same drug levels
<cjwatson> still have never really understood the last couple of chapters of "The Game-Players of Titan" :)
<cjwatson> Moriarty is hard SF, perhaps closest to Egan of things I've read at all recently although much more character-focused
<blr> I'm not certain I can compare Dhalgren to anything I've read, maybe Kafka's The Castle
<wgrant> Maybe I need to start reading sci-fi.
<blr> cjwatson: hah right, he was fully cracked by the time he was writing Valis, which is possibly his most interesting consequently
<blr> that was the point in his life when he was certain god was beaming a stream of gnostic data directly into his head.
<cjwatson> That sounds appropriately Dickian
<cjwatson> wgrant: I'm sure we can come up with a decent list of recommendations :)
<wgrant> cjwatson: Thta would be lovely. I haven't read much in a good many years :/
<wgrant> cjwatson: Ah, you can fortunately avoid that currval. Will mention it when I do the full review.
<cjwatson> wgrant: Interesting, OK.  (And a bit longer than my earlier guess suggested, sorry!)
<wgrant> cjwatson: Hey, it's more than half tests. Can't complain about that.
<blr> there are some weirdly awkward things in pygit2, create_commit returns an oid sha rather than a commit object.
<wgrant> blr: Do you have a reasonable set of repo construction helpers now?
<blr> wgrant: just about, although nothing overly sophisticated, just what the tests need atm.
<blr> thinking about how it could be refactored to a library with a sensible interface
<wgrant> Yup, that's the way to go.
<wgrant> Possibly, yeah. We'll see how it works when we try to integrate it into Launchpad.
<blr> wgrant: I need a side-project atm at any rate :)
<wgrant> blr: We've neeeeearly got enough of the LP and API sides to start working on slices of features across the stack. And of course feel free to distract yourself with random LP bugfixes.
#launchpad-dev 2015-03-04
<blr> wgrant: the lack of that feature unfortunately means the diff api test is less than ideal. not entirely sure what to do about that
<blr> wgrant: here if you have a moment to look https://code.launchpad.net/~blr/turnip/api-diff
<wgrant> blr: The lack of which feature?
<blr> patch parsing
<blr> sorry split-channel context :P
<wgrant> Oh, #ci, right.
<wgrant> Just catching up.
<blr> wgrant: test is a bit braindead (test_repo_compare_commits() in http://bazaar.launchpad.net/~blr/turnip/api-diff/revision/96 test_api.py) without patch parsing
<blr> on a related note, have since moved some of that pygit2 ugliness to test_helpers property in a later revision.
<wgrant> blr: We can write some better tests if we construct a known tree and check for relevant +/- lines etc.
<blr> wgrant: right, that would be better, but will still need to regex the json body.
<blr> wgrant: just trying to think about how to achieve that within the context of http://bazaar.launchpad.net/~blr/turnip/api-diff/view/head:/turnip/api/tests/test_helpers.py
<blr> wgrant: afaict, I can't _apply_ a patch with pygit2 either.
<wgrant> blr: I don't believe there's a way to do that today, no.
<wgrant> Fortunately we don't need to right now :)
<wgrant> blr: It probably makes sense to have a test helper that constructs a tree given a dict of {path: content}, allowing us to easily create commits that look as we desire.
<wgrant> That would make the diff test quite simple.
<blr> that makes sense... is what is there currently reasonably sane?
<wgrant> blr: Looks generally sensible to me!
<blr> wgrant: great, thanks
<wgrant> blr: Any idea why the turnipcake charm doesn't seem to expose any ports? I can see them defined in the service config, just like in turnip, but juju status doesn't show them.
<wgrant> Oh, it would help if I added the relation so the services framework actually started it, wouldn't it.
<lifeless> wgrant: test helpers for file trees - bzr has one already
<lifeless> wgrant: if you can't reuse that, you could add one to fixtures easily enough, I'd be happy to accept a patch extending the existing stuff
<wgrant> lifeless: "tree" as in the git object, not an actual tree.
<lifeless> wgrant: sure; probably one of those in jelmers git protocol impl :)
<wgrant> Indeed.
<wgrant> cjwatson: Hrm, did you run the test suite after your turnip changes to introduce authenticated-uid?
<cjwatson> wgrant: I'd hope so, but it was a while back so I don't specifically remember.  And I guess not given that you're asking the question ...
<wgrant> cjwatson: Heh, yeah, it was all very broken.
<wgrant> Fixed now, and I've adjusted setup.py so "setup.py test" runs the full suite now.
<wgrant> I'd been using "python -m unittest discover turnip", didn't realise "setup.py test" had been made to nearly-work.
<cjwatson> wgrant: Thanks, sorry about that!
<wgrant> cjwatson: np, the fixes were easy. Just a bit confusing that the world was screwed when I was jujuing.
<wgrant> cjwatson: With that review out of the way, is anything left blocking you?
<cjwatson> wgrant: Nope, thanks.
<blr_> wgrant: did you get the turnipcake ports open?
<blr_> lifeless: presumably you were referring to something in dulwich? Writing test helpers wrapping pygit2 atm to make git repo/object fixtures easier. On a tangential note, a lot of things I'm trying to achieve would be much easier in bzrlib!
<lifeless> blr_: yes
<blr> I'll have a look through the dulwich source for inspiration, thanks
<blr> wgrant: my diff api MP has unmerged changes from api-ref (ready for re-review), perhaps just consider both in the new MP and I can delete the other?
<wgrant> blr: Ah, yeah, that probably makes sense.
<wgrant> blr: Sorry, got distracted by juju yesterday...
<blr> no worries - did you get turnipcake up in the end?
<wgrant> blr: Yep, found a few issues with both charm and code on the way, but I know my way around enough to fix the charms now.
<wgrant> The services framework stuff seems to make them muuuuch nicer.
<wgrant> (I created a whole lot of Asana tasks for most of the issues I found, as I blundered through)
<blr> wgrant: yes, I removed lots of boilerplate once I switched to the services framework.
<blr> wgrant: if turnip.endpoint was in /srv/envs it should have been available in the environment which will override the default set in the pyramid config
<blr> or am I missing something
<blr> turnip.endpoint in turnipcake.ini
<wgrant> blr: It didn't seem to be overriding it. I couldn't see code that read it.
<blr> wgrant: that's pyramid's default behaviour
<wgrant> orly
<blr> http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/narr/environment.html
<blr> which is pretty sensible!
<blr> +1 pyramid
<blr> it will translate MY_ENV_VAR to my.env.var as well
<blr> a concern if it isn't actually working though :/
<wgrant> blr: I don't think that works in general.
<wgrant> It would be rather unsafe if it did.
<wgrant> Yeah, the pyramid.config.settings does this sort of thing:
<wgrant>         config_debug_auth = self.get('pyramid.debug_authorization',
<wgrant>                                      config_debug_auth)
<wgrant>         eff_debug_auth = asbool(eget('PYRAMID_DEBUG_AUTHORIZATION',
<wgrant>                                      config_debug_auth))
<blr> wgrant: ah I see, it is limited to a specific set of config values
<blr> welp, I can see why that wouldn't be working in that case, apologies.
<blr> might be worth reassessing the state of amulet, to have some basic integration tests in place for these relations
<wgrant> blr: Easy enough for me to fix, I suspect.
<wgrant> I still don't know quite where amulet fits with mojo.
<blr> no, nor am I...
<wgrant> blr: (I'm glad it's broken like this; Pyramid allowing any option to be overridden by a non-prefixed envvar would have made me switch away from it.)
<blr> wgrant: you're right, that would be a little terrifying.
#launchpad-dev 2015-03-05
<wgrant> blr: Could you please have a look at https://code.launchpad.net/~wgrant/charms/trusty/turnip/single-dir/+merge/251873?
<wgrant> Oh, you already have. I must have blackholed all charm mail.
<blr> even without other services running on the instance, that seems better.
<wgrant> Yeah
<wgrant> My intent is that I should be able to deploy eg. both on one unit.
<wgrant> I'm going to add a new relation this morning.
<wgrant> To automate the turnip <-> turnipcake setup entirely.
<wgrant> Currently you have to deploy both, relate them, then set the virt endpoint on the turnip unit, when that should be fetched from the relation by default.
<wgrant> So it'll be deploy, deploy, add-relation, oh look you have a git server.
#launchpad-dev 2015-03-06
<infinity> wgrant: MP for you for 20-30s of low-hanging publisher fruit.
<infinity> Mmm, fruit.
<wgrant> infinity: What is it?
<wgrant> Huh, does that really take non-trivial time?
<infinity> wgrant: According to the logs, yeah.
<infinity> wgrant: Even if it was trivial, it's the right thing to do, based on how security_only is designed to be barebones "just enough to publish Packages".
<wgrant> Indeed.
<infinity> wgrant: Doubly-annoying, since it ran even if security_only was otherwise a no-op.
<infinity> wgrant: Admittedly, given what it does, the part where it doesn't happen in under a second is also a bit disconcerting, but I didn't look into that.
#launchpad-dev 2016-03-07
<jtv> cjwatson, jelmer: I've been trying to replace my broken branch and branch import, but I think I've only succeeded in making a new branch import for the same branch.  Doesn't help that I'm not the owner of the branch â it belongs to VCSImports.
#launchpad-dev 2016-03-08
<xnox> (Error ID: OOPS-5c8ad2224a4853254a137a30308ccba6)
#launchpad-dev 2016-03-09
<mwhudson> so um
<mwhudson> have we thought about adding https support for bazaar.launchpad.net at all?
<xnox> mwhudson, i thought we do have it.... please elaborate?
<mwhudson> xnox: not for accessing bzr branches over https we don't
<mwhudson> i think?
<mwhudson> there is for codebrowse
<xnox> ah
<cjwatson> mwhudson: the chances of any development that substantial happening on bzr codehosting are relatively slim
<mwhudson> cjwatson: it's mostly ops isn't it? but yeah
<mwhudson> cjwatson: this is what prompted it https://groups.google.com/d/msg/golang-nuts/bY5qSPjBUCk/FkkAujU2AQAJ
<mwhudson> it's slightly embarrassing
<cjwatson> sure - we do have it on git.launchpad.net
<wgrant> mwhudson: It's complicated due to domain arrangements and security.
<mwhudson> wgrant: bleh ok
<mwhudson> i don't really see why but i'm sure the details are horrible :-)
<wgrant> mwhudson: Users holding SFTP access to a subdomain of a webapp is a Very Bad Ideaâ¢.
<wgrant> We are saved today only by the Secure bit on our cookies.
<mwhudson> oh
<mwhudson> i guess we could turn off sftp but i guess the smart server provides broadly equivalent abilities?
<wgrant> Correct.
<wgrant> VFS access can't readily be eliminated.
<wgrant> It is possible to fix at the web server level, but the security considerations are complicated and we certainly don't have time for that now.
<cjwatson> Good excuse to encourage Go projects hosted on Launchpad to switch to git.launchpad.net on general principles.
<cjwatson> (which I realise is a little unhelpful, but aligned with general goals ...)
<mwhudson> can't decide whether to reply and say that or just ignore it and hope it goes away
#launchpad-dev 2016-03-10
<xnox> cjwatson, but does $ go get launchpad.net/foo --> works with git?
<xnox> or does one need $ go get git.launchpad.net/foo for that?
<xnox> mwhudson, ^
<mwhudson> xnox: former works
<mwhudson> with go 1.5+
<xnox> cool.
#launchpad-dev 2017-03-06
<cjwatson> wgrant: Considering buildinfo storage.  Current tools may include a buildinfo file with each upload.  But PackageUpload is probably too recondite a place to store this, and I think it should probably go on SourcePackageRelease or BinaryPackageBuild as appropriate.  Does that make sense?
<cjwatson> The plausible alternative would be to just ignore buildinfo files in uploads, but I'd rather keep them even if we do only minimal processing at the moment.
<wgrant> cjwatson: Seems to belong to SPR and BPB to me. PU doesn't make much sense.
#launchpad-dev 2017-03-09
<stub> https://code.launchpad.net/~stub/launchpad/botbake/+merge/319445
#launchpad-dev 2020-03-02
<pappacena> Can anyone help on how to create a MP for the turnip deps? It seems that there is no project for that, only a branch of a repo (~canonical-launchpad-branches/turnip/+git/dependencies)... and when I try to open a MP, I get an error saying  "lp:~pappacena/+git/turnip-dependencies is not mergeable into this repository."
<cjwatson> pappacena: So that's not in the same target
<cjwatson> pappacena: ~canonical-launchpad-branches/turnip/+git/dependencies is a repository in the turnip project
<cjwatson> pappacena: In order to make it match for MP creation purposes, you want lp:~pappacena/turnip/+git/dependencies (the exact name of the last segment doesn't matter, but it has to be in ~.../turnip/+git/... rather than just ~.../+git/...)
<cjwatson> (Roll on a fork button)
<pappacena> ahhhmm... ok, got it!
<pappacena> Let me try! Thanks, cjwatson!
<pappacena> tomwardill, you might have the same problem in the future... I think your turnip deps repo in Launchpad is also not with the "/turnip" part :-D
<cjwatson> pappacena: You need to re-propose https://code.launchpad.net/~pappacena/turnip/+git/turnip-dependencies/+merge/380119 with the target repository set to ~canonical-launchpad-branches/turnip/+git/dependencies (either use "Resubmit proposal", or just delete the MP and start over)
<cjwatson> It's a merge into lp:turnip at the moment
<cjwatson> The defaults aren't right when the target is a non-default repo like this
<pappacena> makes sense... I was trying to find why the diff was showing more changes then I actually did...
<pappacena> thanks, cjwatson
<cjwatson> A couple of py3 MPs extracted from my queue: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/380096, https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/380128
<pappacena> I'll take a look
<cjwatson> Thanks
<cjwatson> I should probably sleep :)
<pappacena> Indeed, you should!
<cjwatson> Silly 8:30 starts
<pappacena> Well, anyway you should not be working... it's late there...
<cjwatson> Fair ...
#launchpad-dev 2020-03-03
<SpecialK|Canon> pappacena++
<tomwardill> pappacena: are you looking at the buildbot failure too?
<pappacena> I'm trying, but I cannot make it fail...
<tomwardill> pappacena: same here, ilasc is wondering if it's to do with the test parallelisation
<pappacena> uhm... it's a possibility...
<wgrant> Colin said there was something weird like it being a TrialTestCase -- not obviously wrong, but something relatively uncommon that may well be relevant.
<cjwatson> Right, possibly just changing it to be like the equivalent in the snapbuildbehaviour tests will make buildbot not fail.
<cjwatson> Even if we can't reproduce it locally, which is unfortunate but well.
<cjwatson> TrialTestCase is needed for some of the other mixin-based tests there, but not (AFAICT) for that one.
<cjwatson> Given that plausible theory it may not be worth spending lots of time trying to reproduce it.
<ilasc> tomwardill successfully did that and passed local tests
<pappacena> I agree with cjwatson. It might be better to just change it, push and see if that works on buildbot...
<ilasc> that = moving to equivalent of snapbuildbehaviour
<tomwardill> yeah, I think that's where we've got to
<tomwardill> MP incoming
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380154 cjwatson, pappacena
<pappacena> I'll check
<leni> Hi, is someone familiar with the API ?
<wgrant> Nobody!
<pappacena> hahaha! Nobody is familiar with the API, but we can all be confused together... do you have any specific question about it? :-D
<leni> I was not expecting that ahah
<leni> Well I'm trying to understand the way /projects behaves
<leni> It looks like the first projects returned are the last created but for some it's not true
<cjwatson> You shouldn't depend on the ordering
<cjwatson> Unless you're looking at .latest(), where it's documented
<leni> So there's no indexing ?
<cjwatson> What are you trying to do?
<leni> We're working on implementing launchpad in Software Heritage, so for the tests it would have been great if there was a reliable indexing
<leni> ordering
<leni> *
<cjwatson> https://launchpad.net/+apidoc/devel.html#projects-latest documents an ordering
<cjwatson> You just can't rely on any particular ordering for the default collection on /projects
<cjwatson> But I don't think you need to use the default collection, from the sounds of things
<leni> So it would be ws.latest=true ?
<cjwatson> ws.op=latest
<cjwatson> I think we need to step back and talk more about your requirements though - projects won't get you everything
<leni> Ok thanks, that's helpful
<leni> For now we're only working on listing projects
<leni> Is there more source code hosted besides projects ?
<cjwatson> Yes, for example all source code in distributions
<cjwatson> Like everything in Ubuntu
<cjwatson> (But I'm in a key management session right now, so this isn't quite the best time)
<leni> Well that would be great to add that as well
<leni> I imagine you're here quite often, I could come by another day to discuss this more if you'd like
<cjwatson> Yep, in principle this is a good week for it since LP folks happen to be physically together, just busy right now
<leni> Perfect thank you !
<leni> What timezone are you in ?
<tomwardill> leni: we're currently in CET
#launchpad-dev 2020-03-04
<cjwatson> tomwardill: I think you can land https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380046 now - the DROP NOT NULL DB patch is in
<tomwardill> cjwatson: ace, thanks
<tomwardill> landing
<cjwatson> tomwardill: And then I guess the constraint change, but we'll need to get this all the way to production first
<tomwardill> yeah
<cjwatson> tomwardill: But would be good to prepare the branch
<tomwardill> ah yeah, forgot it didn't exist anymore
<tomwardill> will do that this morning
<tomwardill> cjwatson: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380200
<cjwatson> tomwardill: I'd rather see this without the code change mixed in
<cjwatson> tomwardill: Will wait until your code change hits db-devel I guess, and then the diff should simplify itself
<tomwardill> ah, sorry, forgot about that merge and based it off the wrong tree
<tomwardill> I can rebase it off db-devel, one sec
<cjwatson> ok
<tomwardill> cjwatson: done
#launchpad-dev 2020-03-05
<tomwardill> cjwatson: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380273
<wgrant> https://code.launchpad.net/~wgrant/launchpad/+git/launchpad/+merge/380297
<cjwatson> wgrant: r=me
<leni> Hi could we discuss your API ?
<cjwatson> Ah yes, sorry, been busy today
<cjwatson> So I'm thinking we could have something like what ddeb-retriever uses to get a feed of changes to debug symbol packages, only for bzr branches and git repositories
<wgrant> Indeed, I think that makes a lot of sense
<cjwatson> Something like lp.git_repositories.getAllRepositories(modified_since_date=...)
<leni> That would be great
<cjwatson> leni: What's the client code for this?  Are you in a position to use the Python launchpadlib package, or is the client in something other than Python?
<cjwatson> If you aren't using launchpadlib (or at least lazr.restfulclient) you'll need to implement iterating over batched collections yourself
<cjwatson> Which isn't too hard, just needs a bit of care
<leni> Is it possible to also have an option to list them by creation date ?
<cjwatson> Can you explain why?
<leni> We're using python so we can use launchpadlib
<leni> Because it would be easier to make with an indexing value
<cjwatson> Indexing value?
<cjwatson> Creation doesn't seem a very valuable thing to consider; a repository might well be created entirely blank
<leni> Like a creation date or a uid
<leni> We take everything even blank repos
<cjwatson> Right, but you surely want to know when the repository stops being blank
<cjwatson> Modification seems much more interesting than creation (creation is just a specific kind of modification)
<cjwatson> So you just need an identifier for the repository?
<leni> Modification is already known as the content is updated by pulling at a certain interval
<cjwatson> Hm, we also need to think a bit about exactly how things work if a repository is modified while somebody is iterating over a date-ordered collection of repositories
<cjwatson> We would much rather you only poll when we tell you that the repository has changed, if possible
<cjwatson> That's why I'm suggesting giving you a feed ordered by modification date - so that you can pull only the ones that have changed
<cjwatson> s/poll/pull
<cjwatson> Should be much more efficient for both of us
<leni> I'm looking into how that could work
<cjwatson> Iterating over ~1000000 public bzr branches and ~17000 public git repositories even though they mostly haven't changed would be pretty inefficient
<leni> Of course
<leni> So apparently it's not yet doable to only pull on new changes but that's something they're willing to add in the future
<leni> But it would still be interesting for the initial listing as we get all the repos and not just the projects
<cjwatson> This might be acceptable for git repositories because there are fewer of them, but I think I'm not very willing to do the bzr side until you're only pulling ones we tell you are changed
<leni> For now there's no bzr loader so it's only on the git side
<leni> We'd happy to get that for now
<cjwatson> If we did ordering by modification date, that ought to still get you everything you need, you might just have to deduplicate slightly
<cjwatson> But I'm looking into some technical details of ordering
<cjwatson> What do you plan to do when repositories are renamed?
<leni> It would create a new one
<leni> By modification date as you stated if there's a modification while iterating it would skip that one no ?
<cjwatson> So you can probably just use repository.unique_name as an identifier
<cjwatson> We need to solve that anyway, so I'm thinking about i
<cjwatson> t
<leni> If we have that unique_name that's great
<cjwatson> https://launchpad.net/+apidoc/devel.html#git_repository
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380300 - make the default target on a git MP page more friendly
<cjwatson> r=me
<leni> And that unique_name cannot be changed ?
<cjwatson> It's unique at a given point in time, but is mutable
<cjwatson> But that should be OK since you said that if a repository is renamed it would create a new one
<cjwatson> We could also expose the immutable ID.  We normally prefer not to, but it's possible
<leni> Yes it would not be a problem
<leni> When you call a function like that does it send back everything or is it paginated ?
<cjwatson> You get an initial batch of 75 and it's paginated
<cjwatson> https://code.launchpad.net/~cjwatson/lazr.restful/range-factory/+merge/355966 would be needed I think
<cjwatson> leni: Is the code here going to be open (or at least in a position where we can review it)?
<leni> Yes of course
<cjwatson> We have a hacky option that will require a bit more care on your side; but if we can review the code it ought to be possible to make it safe
<leni> https://forge.softwareheritage.org/ is where it will be reviewed
<cjwatson> (It's quite a bit easier on our side)
<leni> In what way would it be hacky ?
<wgrant> The LP API's pagination system doesn't quite support the ordering that we need for this to be completely safe. We're devising a solution which will effectively emulate pagination by making a bunch of different getRepository requests.
<tomwardill> cjwatson: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380302 I did an oops and diddn't get it switched back to 'Needs Review' in time, so have a follow on MP
<cjwatson> r=me
<leni> So if you think that will work we're ok with it
#launchpad-dev 2020-03-06
<tomwardill> buildbot has had 2 fails on db-devel with a rabbitmq startup timeout. I've restarted
<cjwatson> Yeah, that's unfortunately a common one.  I've bumped the timeout once or twice; I think when this happens it may be stuck rather than slow.  Not completely sure though
<cjwatson> It'd be good to figure out how to get it to dump the server log when that happens (which might end up being a change in rabbitfixture rather than in LP, I'm not sure
<cjwatson> )
<tomwardill> I don't think there's been a reoccurence of the OCI failure though
<cjwatson> Indeed, haven't seen one, nice
<cjwatson> TrialTestCase delenda est
<cjwatson> OCI webhooks almost working locally; just need to fix a couple of tests
<tomwardill> oooh, nice
<cjwatson> tomwardill: Updated https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+ref/db-oci-push-rule, I think
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/380353   attempt at helping users to help us
<SpecialK|Canon> Nice
<wgrant> Ah, good idea.
<wgrant> I wish we had metrics to see whether it actually works!
<wgrant> Ah, https://lpstats.canonical.com/graphs/CHRFrustrationLevel
<cjwatson> haha
<SpecialK|Canon> ...that's Django with DEBUG=True!
<SpecialK|Canon> :s
<wgrant> Yup
<cjwatson> Fortunately also behind fairly restrictive OpenID.  But yes
<cjwatson> ilasc: https://dev.launchpad.net/Running/LXD
<cjwatson> http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1094/steps/shell_9/logs/summary
<cjwatson> There's a librarian OOPS in there; I wonder if that's relevant
<cjwatson> (Trying to talk to Swift, which it probably shouldn't be in this context)
<cjwatson> I think that the fact that we're past the assertion that the build status is UPLOADING rules out an ordinary race
#launchpad-dev 2020-03-07
<tomwardill> got a confusion. +login does a request to the testopenid server, which is only on SSL. We have a self signed cert, how does that actually succeed? Asking because my 39A build is throwing SSL validation errors and I'm confused as to how it ever works in a development env.
<cjwatson> tomwardill: See lib/lp/services/webapp/openid.py, which passes an extra certificate file to urlopen if the test OpenID provider is in use.  I suspect you'll need to adjust the certificate path there.
<tomwardill> oh, interesting
<tomwardill> ta :)
<cjwatson> (I might move that to a different place at some point for other reasons; the set_default_openid_fetcher function, anyway)
<tomwardill> it lives!
<cjwatson> \o/
<tomwardill> and just in time for flight boarding, look at that
 * tomwardill -> FRA -> MAN -> SHF
<cjwatson> gute Reise
