#launchpad-dev 2010-03-22
 * mwhudson lunches
 * NCommander is amazed at how long the LP test suite takes to run
<mwhudson> NCommander: welcome to our world
<NCommander> mwhudson: how long does a fullrun take?
<mwhudson> NCommander: about 4 hours, i think
<NCommander> mwhudson: (I'm seeing some codehosting failures which I'm concerend about, since I didn't touch that bit of the code)
<NCommander> mwhudson: this has been going for at least 6-7
<mwhudson> NCommander: what hardware?
<mwhudson> that's pretty slow though
<NCommander> mwhudson: 2.2Ghz dual core intel dual core processor
<NCommander> 8 GiB of RAM running on an SSD
<NCommander> Got a few weird failures
<NCommander> mwhudson: I'm running lucid versus karmic though, which may explain some of the failures
<mwhudson> ah possibly
<NCommander> mwhudson: I know its possible to run the test suit eon EC2; can you shed some light into that?
<mwhudson> NCommander: have you seen https://dev.launchpad.net/EC2Test ?
<NCommander> mwhudson: how much does EC2 cost for something like this?
<mwhudson> a few $ i guess
<mwhudson> it uses a c1.xlarge instance for 4-5 hours
<NCommander> mwhudson: might be worth it although I can't expense it as I'm not on the LP team :-/
<mwhudson> NCommander: i can run it for you if you like
 * mwhudson is currently trying to find stuff to delete so i can upgrade to lucid without scaring myself
<NCommander> mwhudson: that may be worthwhile, although I'm concerned I may have actually broken part of LP (first db changes I ever made)
<mwhudson> NCommander: i don't see why that means you shouldn't run it under ec2 though?
<NCommander> mwhudson: oh, no, that was just a side comment, not a blocker :-)
<mwhudson> :)
<mwhudson> NCommander: where is the branch on lp?
<NCommander> mwhudson: its at ~mcasadevall/launchpad/raw_source_changelog, but I'm thinking I should upload this last round of changes as a new branch
<NCommander> I rewrote the functionality to use librarian rather than storing in teh DB
<mwhudson> NCommander: well, ok, i'm not going anywhere, so if you want me to test a branch, just ping me
 * NCommander also feeds a second code review may be a good idea
<NCommander> mwhudson: give me a sec
<NCommander> mwhudson: lp:~mcasadevall/launchpad/raw_source_changelog_librarian
<mwhudson> NCommander: what email address do you want results sent to?
<mwhudson> (/msg if you like...)
<NCommander> mwhudson: mcasadevall@ubuntu.com
<NCommander> mwhudson: (sorry for the delay in answering)
<mwhudson> NCommander: ok, starting up ec2 test
<NCommander> mwhudson: thanks. I'm thinking I should probably turn my old laptop into running UEC and then run ec2test on it :-)
<mwhudson> heh heh
<mwhudson> not if you want results quickly i suspect
<NCommander> mwhudson: the old laptop is 2.2Ghz 4GiB of RAM
<NCommander> its not that slow
<mwhudson> oh ok
<mwhudson> no :)
<NCommander> mwhudson: Total: 25575 tests, 16 failures, 19 errors in 510 minutes 32.100 seconds.
<NCommander> irony, it just finished
<NCommander> I tee'ed the output, but I don't even know if the failures are due to lucid, or due to something in my environment :-/
<mwhudson> NCommander: well the results from ec2 will tell you some of that
<NCommander> mwhudson: thanks :-/
<wgrant> NCommander: The only known Lucid failure is in apt-ftparchive.
<wgrant> NCommander: Lucid's adds SHA1 and SHA256 hashes in addition to pre-Lucid's MD5s.
<wgrant> Any other failures are probably real problems.
<NCommander> wgrant: I think my postgresql server dumped out towards the end, I got a bunch of *strange* errors with write errors :-/
<wgrant> Heh.
<wgrant> Does anybody happen to have an idea of where the time is taken in the primary archive publisher?
<NCommander> wgrant: well, at least then I know which tests to check
<wgrant> NCommander: Right.
<lifeless> wgrant: not an accurate one; its a problem I'd like to look at. If you choose to look at it I'd be very happy to be a sounding board.
<wgrant> lifeless: From what I can tell, publish-distro.py takes about 20-25 minutes currently.
<wgrant> I've cut two or so minutes out of that locally by optimising domination.
<wgrant> And I think most of the rest mus tbe in the apt-ftparchive integration. But I'd like to know for sure.
<wgrant> (apt-ftparchive integration is the one bit I can't sanely simulate locally, since it requires lots and lots of real packages, not just fake contentless ones)
<persia> wgrant: You could simulate by mirroring karmic and "publishing" bits of current lucid.
<wgrant> persia: But that requires a local copy of the archive, which has a sane Internet connection as a prereq.
<persia> The pacific ocean clearly needs a denser net of high-capacity optical fibre
<wgrant> But anyway, I have just reduced native index generation time for largely-unchanged Ubuntu-sized binary pockets to 2-3 seconds.
<spm> or electrical flows that exceed the speed of light
<wgrant> It would be very interesting to know how much longer than 20 seconds the apt-ftparchive integration code takes to run.
<mwhudson> spm: yeee-haw!
<wgrant> Can anybody see cron.publish-ftpmaster logs?
<persia> Is there some test I can run locally that doesn't require me to setup/install launchpad first?
<spm> mwhudson: indeed
<wgrant> persia: No.
<wgrant> But the times are all in the logs that presumably somebody can see.
<mwhudson> dammit, now i want to watch that movie again, and i only watched it last week or so
<wgrant> Which movie?
<mwhudson> dr strangelove
<wgrant> Ah.
<mwhudson> spm posted http://davideubank.files.wordpress.com/2008/09/slim-pickens_riding-the-bomb_enh-lores-720p.jpg with the comment "the life of a losa" or something like that
<wgrant> Ahhh.
<spm> it' an accurate description.
<wgrant> spm: Does cron.publish-ftpmaster log anywhere that you can check quickly? I am interested in the time taken by index generation (starting with '* Step C', ending with '* Step D')
<spm> looks...
<spm> urg. we don't rotate the poppy log. /me hads a MUST FIX.
<spm> wgrant: http://paste.ubuntu.com/399107/
<wgrant> Aha.
<wgrant> So we can cut 17 minutes of the publisher run-time to probably well under one.
<wgrant> Thanks spm.
<spm> heh; if you can cut the time down? we'll thank you! :-)
<persia> wgrant: This isn't all part of some master plan to release 48 times a day, is it?
 * NCommander waves to persia 
<spm> persia: ??? there's more minutes than 48 in a single day! :-P
<persia> spm: Sure, but one has to run the entire publisher for each release, so it would need lots more optimisation to release every minute.
<wgrant> persia: To at least make it possible to drive it that fast if necessary, yes
<persia> Actually, with the "if necessary" bit, that's a good thing.  I'm just not sure I'd like it that fast by default: sometimes folk catch stuff within a run, which gets hard if it gets smaller.
<wgrant> (this maaaay not be unrelated to Saturday's events)
<persia> heh
<NCommander> persia: well, given a fast enough machine, it should be possible to publish once a minute
<wgrant> NCommander: s/fast enough machine/less pathetic publishing code/
<persia> NCommander: You'd also need a faster network, faster mirrors, etc.
<NCommander> persia: or a longer minute
<persia> feh.
<NCommander> I mean, if you redefine a minute ...
<NCommander> Microsoft redefined stable after all, it could be done :-)
<wgrant> My timeline has a 19 minute gap :(
<persia> My minute is 551,557,906,200 vibrations of cesium-133 at STP.
<NCommander> My minute is 0.06 kiloseconds
<NCommander> My second however is define as 0.0864 standard seconds. What sorta second am I?
<persia> wgrant: Which bit is missing?
<spm> a 2nd rater ?
<spm> NCommander: you really left yourself open to that one :-)
<NCommander> er
<NCommander> *0.864 seconds
<NCommander> spm: its a decimal second :-)
<wgrant> persia: There is a 19 minute hole containing dsync's link-dupes run, cron.germinate and the cocoplum->syncproxy sync. The last probably takes three minutes.
<wgrant> So one of the first two is taking an awful long time.
<persia> link-dupes
<wgrant> I'd suspect so, yes.
<wgrant> I'm not quite sure why it's run.
<persia> It takes > 2 hours to run link-dupes against my mirror set.
<wgrant> Do you know?
<persia> Massively saves space.
<wgrant> Even though we symlink matching files anyway?
<persia> Erm.
<persia> Then no, doesn't do anything at all.
<persia> I'd rather see matching files hardlinked, because it mirrors better, but that's just preference.
<persia> So even though it has no effect, it still takes forever to run fdupes.
<wgrant> I mean, when we need a file published in multiple locations, we symlink the non-master copy as any good archive software should.
<wgrant> This is well before the archive is split into archive/ports/security, so I don't see why there would be any dupes.
<persia> If it's pre-mirror-split, there should be no dupes.
<wgrant> Right.
<persia> Or rather, only known multi-publication.
<wgrant> I'm guessing you use link-dupes because you maintain mirrors of both ports and non-ports (or portions thereof)?
<persia> Actually, I use my own hacked fdupes-based script, and also mirror debian, but yes.
<wgrant> So if there's little point running it, and we fix NMAF to support extra-overrides and migrate the primary archive to use it, *and* my two optimisation branches actually work properly, we can cut more than half an hour off the 40-minute-ish publisher.
<persia> Which could shorten the time-to-fix for something like Saturday's incident down to ~90 minutes with manual publisher runs.
<wgrant> s/90/30
<mwhudson> what happened on saturday?
<persia> wgrant: consider build times too.
<wgrant> Upload CDBS, wait for it to build in a few minutes, publish. Upload two broken packages, built in 10 minutes, publish.
<wgrant> mwhudson: Lucid broke right beta 1.
<persia> mwhudson: some updates were published to lucid that broke gnome-panel autostart on login.
<wgrant> gnome-panel and nautilus broke so they wouldn't autostart.
<wgrant> Leaving all the willing testers with no desktop at all.
<spm> wooo
<persia> (or at least everyone who logged out and logged in after the updates)
<mwhudson> i see....
<wgrant> persia: Which lots of people did, after they couldn't open folders any more :/
<spm> speaking as someone who's 7 mins from downloading for the u/g. is it all fixed on latest updates?
<persia> Oh, yeah.  non-logout/non-reboot is the significant minority.
<mwhudson> just out of totally idle curiosity, what versions of things contain the fixes?
<mwhudson> spm: zing!
<persia> It's fixed.
<wgrant> mwhudson: -0ubuntu2 of both of those is broken.
<wgrant> -0ubuntu3 is safe.
<wgrant> It's been fixed since Saturday night.
<wgrant> It just took a lot longer than it could have if the publisher didn't take an hour.
<mwhudson> mmm
<mwhudson> i have -0ubuntu1 of those
<mwhudson> i guess my mirror aint so fresh
<spm> having my main workstation cactus is a pain; but not unsurvivable. karmic forced me onto my laptop for a week or 2; totally broke my workstation's startup.
<persia> Getting the publisher down to ~10 minutes makes it something people will be more willing to babysit too.
<wgrant> Right.
<wgrant> Plus it'd be less of a pain around releases, when you see release managers tossing up whether to run it now or not.
<persia> Absolutely.
<mwhudson> ha, ha
<mwhudson> 255 updates
<NCommander> mwhudson: thanks for ec2ing my branch. TIme to fix some failures
<NCommander> mwhudson: you got less failrues and errors than I do it
<NCommander> what can cause this: ProgrammingError: permission denied for relation sourcepackagerelease
<mwhudson> NCommander: have you encountered database/schema/security.cfg yet?
<NCommander> mwhudson: nope
<mwhudson> NCommander: in that case "ProgrammingError: permission denied for relation sourcepackagerelease" means you need to make friends with this particular file
<mwhudson> NCommander: different bits of launchpad connect to the database as different database users
<mwhudson> and the different users have permissions to access the various tables in different ways
<NCommander> mwhudson: ugh. I have no idea how to fix this, since there is a SELECT permisison already in place
<mwhudson> NCommander: for which user though?
<NCommander> mwhudson: librariangc. Looking at the code, its trying to DELETE, but I'm kinda confused on how objects leave librarian
<NCommander> mwhudson: it seems to me its trying to delete rows, but I'm not understanding why librariangc isn't happy
<NCommander> mwhudson: oh wait ...
<NCommander> mwhudson: alright, I think I figured out why the librariangc tests are failing :-)
<mwhudson> NCommander: grats :)
<NCommander> mwhudson: I assume I need to re-make schema after editing security.cf
<NCommander> *cfg
<mwhudson> NCommander: i think you can just run python database/schema/security.py launchpad_ftest_template
<mwhudson> or something
<mwhudson> but i usually just run make schema, yeah
<NCommander> mwhudson: thanks, I'm still somewhat new at all this LP hackery. I'll probably ask you to re-run ec2test for me since I got a lot mor efailures on 127.0.0.1 than ec2 did :-/
 * NCommander will probably setup EC2 sometime in the future but can't ATM :-/
<StevenK> NCommander: It's pretty easy, it took me about 10 minutes
<NCommander> StevenK: how much did it cost per run?
<StevenK> Roughly $3.50
<StevenK> But they don't charge until the end of the month
<NCommander> StevenK: that can add up fast -/
<mwhudson> i spent a lot on ec2 in february
<mwhudson> luckily i can expense it...
<NCommander> mwhudson: woo, librarian fixed
 * NCommander now works on registery failures
<mwhudson> NCommander: i'm going to be going afk soon, but europe should be getting up soon ish :)
<NCommander> mwhudson: hrm, is it possible for a test to be failing in db-devel?
 * NCommander can't see how this failure is possible
<NCommander> er
<NCommander> or how is impossible
<StevenK> NCommander: Stuff doesn't land if the tests fail
<mwhudson> StevenK: not true any more, sadly
<NCommander> mwhudson: that was what I was afraid of
<StevenK> Awww
<mwhudson> NCommander: which tests fail for you?
<NCommander> mwhudson: lp.registry.tests.test_distroseries.TestDistroSeriesPackaging
<mwhudson> oh
<mwhudson> that doesn't seem to be failing in buildbot
<NCommander> mwhudson: NameError: global name 'gpg_key' is not defined
<NCommander> Looking at the file in question though, there is only one refernece to gpg_key in the whole thing
<mwhudson> !
<mwhudson> is that in devel or db-devel?
<NCommander> mwhudson: db-devel, but hold on
 * NCommander might have done something stupid
<wgrant> Damn. Turns out my 20 second index generation run had broken bz2 compression. Fixing that blows it out to more than a minute :(
<StevenK> wgrant: A minute versus 24, though?
<mwhudson> NCommander: i don't see gpg_key in that file
<mwhudson> (on either branch)
<wgrant> 17-18, but yes.
<wgrant> The rest of the 24 is being cut away in other places.
<NCommander> mwhudson: I dunno, I think I went a little merge happy at some point, and lack of sleep is not helping
<mwhudson> lack of sleep often doesn't help with hacking
<NCommander> mwhudson: no, but my girlfriend went ona  cleaning spree
<NCommander> and my bed currently isn't where I left it
<mwhudson> NCommander: haha
<wgrant> Are we week 2 or 3 at the moment?
<mwhudson> wgrant: just starting week 3
<wgrant> Aha.
 * wgrant had better get his last PPA download stats branch in order, then.
 * NCommander broke something bad
<NCommander> very bad
<NCommander> anyone around who can run the ec2test script OR run the test suite locally without getting unrelated failures
 * NCommander has it fixed
<wgrant> bigjools: Morning.
<bigjools> g'day
<wgrant> What are the blockers for NMAFing primary? Just extra overrides?
<bigjools> yep, barring any hidden bugs
<bigjools> and possibly performance
<wgrant> Performance is definitely an issue.
<wgrant> But with a bit of caching I can publish all indices for a larger-than-Ubuntu archive in 20 seconds.
<wgrant> (one series thereof, that is, like most normal runs)
<bigjools> wgrant: using NMAF?
<wgrant> bigjools: Yeah.
<bigjools> it would be interesting to do like for like comparisons
<mrevell> Hello
<bigjools> hey mrevell
<bigjools> wgrant: did you see Bug 543162 BTW?
<mup> Bug #543162: launchpad-buildd exception: Try to register an already registered process. <Launchpad Auto Build System:New> <https://launchpad.net/bugs/543162>
<wgrant> bigjools: I was going to, but a-f requires real sources and binaries, whereas NMAF doesn't care if they're garbage.
<bigjools> wgrant: dogfood FTW :)
<wgrant> bigjools: I have, yeah. lamont poked me about it at the time to check if there was anything obvious.
<bigjools> it's an odd one - I can't work out if twisted blew up or the slave code
<wgrant> bigjools: Is dogfood a good performance indicator?
<bigjools> wgrant: not against anything else, but relative to itself of course it is
<noodles775> bigjools: if it's related to tearing down twisted code, that behaviour changed recently.
<wgrant> I mean, a-f is going to hit the DB in different ways to NMAF.
<bigjools> wgrant: indeed
<bigjools> noodles775: was that in lucid though?
<wgrant> I tried to use memcache, but it was far too slow :(
<noodles775> bigjools: nm, not on cesium.
<bigjools> ironic :)
<wgrant> noodles775: It's the buildds themselves, which are in fact pre-Wellington.
<noodles775> wgrant: Right (just reading the bug).
<wgrant> It's only been seen on armel, but it happened on two of them (at a reasonably inconvenient, almost simultaneous time)
<wgrant> bigjools: Also, do you know the purpose of the dsync link-dupes call in cron.publish-ftpmaster? There shouldn't be any duplicates...
<bigjools> no idea, it's always been there
<wgrant> (and it probably takes forever and then a little while longer (or just 15 minutes)
<wgrant> bigjools: Who's likely to know?
<wgrant> 'cause slow publishing makes me sad and breaks lots of users' systems.
<bigjools> wgrant: probably people who have left :/
<bigjools> I can check with IS
<wgrant> It's from dak, so elmo probably does.
<deryck> Morning, all.
<jml> good morning Launchpadders
<wgrant> What do I need to do to get the PPA download stats script running on production?
<bigjools> wgrant: is it a cron job?
<bigjools> where is it?
<wgrant> bigjools: It's a cron job. cronscripts/parse-ppa-apache-access-logs.py
<bigjools> ok
<bigjools> wgrant: we need to run it on dogfood first
<wgrant> Right.
<wgrant> Or staging.
<wgrant> And it needs production configs.
<bigjools> not staging
<wgrant> It doesn't need a real archive. Just logs to be put somewhere.
<bigjools> I'll work on the configs and test it on DF
<wgrant> Thanks.
<bigjools> hmm busy day
<wgrant> bigjools: Ew, history deletion.
<bigjools> wgrant: ?
<wgrant> bigjools: Deleting archives and EVERYTHING with them.
<bigjools> wgrant: yep
<bigjools> like deleting a branch
<wgrant> Yes, both are repulsive.
<bigjools> deletion is deletion, not some half-assed attempt
<bigjools> if you want history, keep the f***ing thing
<bigjools> or it can be disabled
<wgrant> If you want malicious or misled users to be able to inappropriately destroy things irrevocably, let them actually delete thousands of records in one click.
<wgrant> If you want users to be able to remove the only records of their malicious activities on the buildds...
<wgrant> Also, how do binary copies work?
<wgrant> If I delete an archive that has had binaries copied from it, the builds are more than a little awkward.
<bigjools> the binaries won't be deleted
<wgrant> What happens to the builds?
<wgrant> We cannot delete them.
<bigjools> neither those
<wgrant> So we lie about their archive?
<bigjools> no, we change the code to say their original archive was deleted
<bigjools> did you read the bug?
<wgrant> It didn't cover builds in particular. I guess stuff might not break if the archive disappears.
<bigjools> that's what the test suite is for
<elmo> ok, seriously, this sucks
<elmo> I've turned referer back on, and I still can't add attachments to bugs
<wgrant> What's the error? Still 'Unexpected Form Data' with a 500?
<elmo> oh, lala
<elmo> sorry, never mind
<elmo> it's my fascist firefox apparmor profile
<elmo> \o/
<wgrant> Heh.
<kfogel> adeuring: PQM is behaving like it has not seen my submissions (http://paste.ubuntu.com/399248/).  Am I missing something?  Can it have built so fast that I missed it? :-)
<kfogel> adeuring: (oh, I made a cut-and-paste error in that paste -- the submissions actually do match their directories, don't worry)
<adeuring> kfogel: ;)
<kfogel> adeuring: https://pqm.launchpad.net/ shows nothing though
<kfogel> adeuring: bbiab
<bigjools> wgrant: ok I have dogfood updated, what configs does that cronscript need? it looks generic
<wgrant> bigjools:
<wgrant> [ppa_apache_log_parser]
<wgrant> logs_root: /srv/ppa.launchpad.net-logs
<wgrant> Paths will vary.
<bigjools> wgrant: what about where it puts results?
<wgrant> bigjools: The DB?
<bigjools> ah nm the config is ppa specific
<bigjools> how does it know to read that config?
<wgrant> bigjools: It's used in cronscripts/parse-ppa-apache-access-logs.py
 * bigjools slaps face
<bigjools> I was looking at the wrong script, sorry
<wgrant> Heh.
<wgrant> I thought something might have been a bit odd.
<bigjools> yeah :)
<bigjools> wgrant: so does this script depend on only ppa logs being in the logs_root ?
<wgrant> bigjools: It'll ignore them if they don't match a PPA binary for any reason.
<wgrant> So it shouldn't matter.
<bigjools> mmkay
<bigjools> wgrant: well it does if one of the files is not readable :/
<wgrant> Yeah.
<wgrant> Not my code :P
 * bigjools watches the buck flying overhead
<wgrant> It will also do strange things if you run both the librarian and PPA log parsers over the same directory, but that's unlikely to happen.
<adeuring> kfogel: did you perhaps receive a mail from pqm with an error message?
<bigjools> wgrant: it doesn't log much does it :)
<bigjools> and 2.1g resident :/
 * bigjools gets coffee while waiting for it
<maxb> Can someone fix the grammar fail in the PPA description?  https://edge.launchpad.net/~launchpad/+archive/ppa
<wgrant> bigjools: Did you see the warning in the docstring (lovingly copied from the librarian one)
<wgrant> (I didn't think there'd be many logs on dogfood -- sorry)
<bigjools> maxb: done
<bigjools> thanks
<bigjools> wgrant: I fiddled the log so that it only appears as a symlink in another dir
<wgrant> m_o_d: Did you run rocketfuel-setup?
<m_o_d> wgrant: yes
<wgrant> m_o_d: Did you see it installing lots of packages?
<m_o_d> wgrant: yes, my error is when i run make schema
<wgrant> m_o_d: I know. But it's complaining about something that rocketfuel-setup would have instaled.
<wgrant> Do you have python-tickcount installed?
<jml> bigjools, any recommendations for coffee from hasbean?
<m_o_d> yes, 0.1-0ubuntu10launchpad1
<wgrant> m_o_d: Which version of Ubuntu are you using?
<bigjools> jml: oh yes :)
<m_o_d> wgrant: lucid
<bigjools> jml: http://www.hasbean.co.uk/products/Guatemala-Finca-Puerta-Verde-Antigua-2009%252d2010.html
<bigjools> you had that last week
<bigjools> well, on Friday when the other stuff ran out
<wgrant> m_o_d: Try to apt-get upgrade. Does that propose anything python-related?
<jml> bigjools, thanks.
<bigjools> jml: also check out the Bolivia Machacamarca
<bigjools> jml: that one is simply awesome
<wgrant> bigjools: Is the script still doing stuff?
<bigjools> wgrant: no it finished some time ago, I got sidetracked
<bigjools> binarypackagereleasedownloadcount has got 4736 rows now
<wgrant> Excellent.
<jml> bigjools, ta. I'll order the Guatemala one and see how I go.
<wgrant> bigjools: Do you have my latest branch?
<bigjools> wgrant: I'm using db-devel
<wgrant> Ah, the webapp is not up.
<jml> bigjools, I'll have to do a taste test.
<bigjools> jml: most stuff from central America is great
<bigjools> wgrant: noodles775 is playing with iut
<kfogel> adeuring: yeah -- testfix/rc
<kfogel> adeuring: :-(
<barry> bigjools, or maybe anybody else running on lucid+py2.6.  have you seen this error?
<barry> http://paste.ubuntu.com/399288/
<bigjools> barry: no not seen that
<wgrant> maxb: Arrrrgh it's back. ^^
<bigjools> barry: but then I'm not really running 2.6 :)
<wgrant> barry: That one has been plaguing me on and off since I started using Lucid a month in.
<wgrant> maxb has seen it too.
<barry> wgrant, bigjools, maxb arg! :)
<maxb> *blink*
<maxb> UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)   <--- that's a new one
<barry> wgrant, maxb, bigjools i really don't want to install python2.5 if i can help it.
<bigjools> barry: it's part of lp-dev-deps isn't it?
<wgrant> barry: You have to. Launchpad doesn't run on 2.6.
<barry> maxb: i do have python-setuptools installed, which should provide distribute
<bigjools> wot wgrant said
<maxb> barry: You want to install python2.5. Unless you want to dedicate your entire time to massive porting project for the forseeable future, you want to install python2.5
<wgrant> (this is a bug that really needs to be fixed soon)
<bigjools> and I thought barry was fixing it actually :)
<barry> dang.
<barry> it's gotta run on 2.6 pretty soon, though, right?
<kfogel> adeuring, wgrant: have a sec to chat about https://bugs.edge.launchpad.net/malone/+bug/78565/comments/16  (ironically, you'll have to navigate from there to the bug for context, becuase the bug isn't fixed yet).
<mup> Bug #78565: no direct link from bug comment page to corresponding bug <breadcrumbs> <confusing-ui> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/78565>
<wgrant> kfogel: The breadcrumb infrastructure is buggy, and the bug breadcrumbs are buggy.
<kfogel> wgrant: because when the final breadcrumb does not match the page being actually displayed, the breadcrumb is still unlinked?
<kfogel> wgrant: or some other bugginess?
<wgrant> That is, the breadcrumb infrastructure should linkify (and probably whinge) if the target of the final crumb is not the current context.
<wgrant> And bug comments should have breadcrumbs of their own.
<wgrant> Ah, that comment.
<kfogel> wgrant: mmmm.  The latter alone would fix this bug, of course.
<wgrant> Right.
<wgrant> I fixed a few Soyuz cases of this a couple of weeks ago.
<kfogel> wgrant: oh hey, let me find those branches, since I can probably save some time by looking at that diff.
<kfogel> wgrant: or, do you want to just do this one too?  Either way is fine with me.
<wgrant> kfogel: devel r10480
<kfogel> thx
<kfogel> wgrant: ^^
<wgrant> I should sleep and do assignments, not add breadcrumbs.
<kfogel> wgrant: heh, np.
 * maxb lunches
 * wgrant really should get around to starting the page titles, breadcrumbs and root contexts discussion on the ML at some point.
<leonardr> i had to manually install 4 packages (3 dev packages + python-geoip) to get make to work in lucid. is this a problem? would it help other people if i mentioned these packages on the wiki?
<kfogel> leonardr: 'make' in general, or just for lp?
<leonardr> kfogel: just for lp, sorry
<leonardr> i had to install those packages to start doing my work again
<maxb> leonardr: Well, let's figure out whether they ought to be in lp-(dev-)deps, and add them if so
<maxb> What were they all?
<leonardr> maxb: recreating the list
<leonardr> python-geoip
<leonardr> libapr1-dev
<leonardr> python-subversion
<leonardr> python-libsvn
<leonardr> libgpgme11-dev
<leonardr> python-all-dev
<leonardr> i think that's it--i still can't start launchpad but it seems to be a postgres problem
<leonardr> i can build everything
<maxb> python-libsvn ?!
<maxb> leonardr: I conclude that launchpad-dependencies was removed by the upgrade to lucid
<leonardr> maxb: ok, i see what happened
<jml> a thing that stumped me for a while yesterday was forgetting to re-enable the launchpad sources.list lines
<leonardr> i re-enabled the sources.list lines
<leonardr> and i ran update/upgrade
<leonardr> but nothing says you have to have launchpad-dependencies just because it's available
<leonardr> so it never got installed
<maxb> So it wouldn't be a bad idea to remind people on the wiki that the Ubuntu upgrader will disable 3rd-party sources.list entries and potentially remove launchpad-developer-dependencies
<bigjools> the upgrade has always remove lp-deps, it's a royal PITA
<maxb> I like to do something slightly cheeky: When the upgrader pops up the box saying "I've disabled 3rd-party sources.list lines", I re-enable them before I click OK :-)
<maxb> (only for PPAs I *know* to be correctly populated for lucid, obviously)
<leonardr> i knew from last time the repo would be disabled but i didn't make the connection that i would have to reinstall the package
<kfogel> noodles775: you're looking into the pqm stoppage, right?
<bigjools> kfogel: he's at lunch, but yes he was
<kfogel> bigjools: thx
<noodles775> kfogel: Yes, I've got a branch on ec2 test to land for the failure earlier this morning, but buildbot has completed successfully since then.
<kfogel> noodles775: argh
<kfogel> noodles775: but the fix in your branch is still needed?
<noodles775> kfogel: it seems to be timing dependent, so yes.
<kfogel> noodles775: well, I guess this just means you can land your branch without 'testfix' :-).
<jml> wgrant, still around?
<jml> abentley, any movement on https://code.edge.launchpad.net/~abentley/launchpad/multiple-series-recipe/+merge/21257 ?
<intellectronica> jml: is there no way we could figure out which packages to chart programatically?
<jml> intellectronica, I couldn't think of anything, but I wasn't thinking very hard.
<intellectronica> jml: my worry is that this will rot, and when we add new packages we'll forget to add them to the list
<jml> intellectronica, we could, for example, specially mark the __init__ of packages that we care about
<jml> intellectronica, yeah, it's a good worry.
<jml> intellectronica, luckily, it's really obvious when you've missed a package that's been added or removed
<jml> intellectronica, removed packages show up as disconnected islands
<intellectronica> jml: r=me on the code. it would be nice to have a solution for this, so it's worth thinking of, and maybe filing a bug, but i can't think of a solution right now either
<jml> intellectronica, every single file of an added package appears on the graph unless it's included in the cluster list
<intellectronica> yeah, i understand
<jml> intellectronica, thanks.
<jml> intellectronica, do you reckon it's ok to land the code before I update launchpad-developer-dependencies?
<intellectronica> jml: i think it is. it won't block anyone - the worst that can happen is that people won't be able to generate the graph
<jml> intellectronica, cool. I'll land it now then.
<abentley> jml, no.  mars says bjorn is still away.
<jml> abentley, oh, if it's just waiting on BjornT then it's ok to land and then get a post-merge review, iirc.
<jml> abentley, in your last comment you said there was stuff to discuss with the code team.
<abentley> jml, I discussed it, we decided to leave it the way it was.  Stub did not require changing it.
<jml> abentley, cool.
<jml> noodles775, https://code.edge.launchpad.net/~michael.nelson/launchpad/496862-ppa-installable-binaries/+merge/21623 looks good to land. hint hint.
<gmb> jml: Quick question: How do you feel about us adding a next_check column to BugWatch? This would allow us to have a garbo-hourly process that decides when a watch should next be checked based on how much it's error in the past; that saves us complex queries in BugTracker.getBugWatchesNeedingUpdate() and means that we keep that logic out of checkwatches.
<gmb> jml: https://bugs.edge.launchpad.net/malone/+bug/544238 for reference
<mup> Bug #544238: Add a next_check column to BugWatch <story-bug-watch-error-tracking> <story-reliable-bug-syncing> <Launchpad Bugs:In Progress by gmb> <https://launchpad.net/bugs/544238>
<jml> gmb, the quick answer is that it sounds similar to the way the branch puller works and how job tables frequently work
<gmb> Right
<jml> gmb, so, I guess, let's do more of that and avoid gratuitous inconsistencies
<gmb> jml: Cool. We've kinda figured out that the best way to make checkwatches play nice is to take some of its logical toys away, so I think this is a good first step. I'll knock a patch together.
<noodles775> jml: I'd love to, but I don't know that it's ready. Did you seen the screenshot I put on the bug (linked from the MP)?
<jml> noodles775, no I didn't.
<gmb> jml: I've requested a review from you and Stuart for said DB patch. https://code.edge.launchpad.net/~gmb/launchpad/bugwatch-next_check-bug-544238/+merge/21869
<persia`> How painful would it be to convince Soyuz to allow some architecture: all packages to build on non-i386?
<bigjools> very
<bigjools> I was looking at a bug about that yesterday
<bigjools> an old bug :)
<persia`> The one that openbios-sparc and openhackware have FTBFS since warty?
<persia`> So, I know I can construct a package that can trick itself into generating an arch:all binary when told to only build arch:any packages.
<persia`> Would it be easier just to allow Soyuz to accept the uploads, whilst still keeping the standard sbuild arguments?
<bigjools> persia`: I'm not sure I understand
<persia`> So, there's two issues at hand.
<persia`> 1) when Soyuz dispatches to sbuild, it only passes -A for i386
<persia`> 2) when Soyuz accepts an upload, it doesn't like extra packages appearing in the .changes file.
<bigjools> why are they issues?
<persia`> So, if we manage to force the package to build the arch:all package even without sbuild's hint, it fails to upload.
<persia`> Oh.  Some stuff that's useful on any architecture is impossible to build on i386.
<bigjools> that's rather special
<bigjools> and hurts my brain to think about how that can happen
<persia`> openbios-sparc is the classic example.
<persia`> It can only be built on sparc, but is required to run sparc emulation for all architectures.
<bigjools> on inspection of that name, it doesn't seem arch independent :)
<persia`> Why not?
<persia`> It's just a bios blob.
<persia`> So, as data, it's useful for any data consumer, on any architecture.
<persia`> And without it, qemu is broken.
<bigjools> so basically it seems that soyuz is simply not designed to deal with this case
<persia`> Yes.
<bigjools> which makes me sad
<persia`> But since there's only a few packages in the archive (3 or 4) that are affected, it's a low priority bug.
<persia`> If it was easy, and just not done, I'd go find someone to do it.
<persia`> If it's really hard, then I'm not going to be able to get some random person intersted in fixing it.
<bigjools> I guess it would be nice to have a way of overriding the nominated arch-indep builder in the package
<persia`> That would be an ideal way to fix it, yes.
<bigjools> which is a lot easier than making soyuz deal with arch:all on any arch
<persia`> We could insert X-Build-Affinity: (or similar) in the .dsc
<bigjools> yep
<bigjools> would be pretty easy to do I think, barring any dodgy bits of code I've forgotten about
<maxb> Part of the problem is that there's no standard way in the deb formats for this sort of thing (leading to minging hacks like Packages-arch-specific)
<persia`> maxb: hence the X, but that's easy to insert.
<bigjools> p-a-s makes me want to stick needles in my eyes
<persia`> Why do we have P-a-s again?
<persia`> Isn't that why we have Architecture: in debian/control ?
<maxb> Because Packages / Sources files don't contain enough info to compute whether a given source will build a given binary on all architectures or not
<persia`> P-a-s just makes pain for porters, in my experience.
<persia`> maxb: So?  Some stuff FTBFS.
<maxb> huh?
<persia`> And we go fiddle with http://qa.ubuntuwire.com/ftbfs until it builds.
<maxb> Perhaps I should say that they don't contain enough info to compute whether a given source *ought* to build .....#
<persia`> maxb: How doesn't the Architecture: field in the .dsc do that?
 * maxb gets example
<persia`> The vast majority of my experience with P-a-s has been complaining that it was too restrictive, so I may be biased on this point :)
<maxb> So, take gdb, whose .dsc says:
<maxb> Binary: gdb, gdb64, gdbserver, libgdb-dev, gdb-source
<maxb> Architecture: any
<persia`> OK.
<maxb> But, then you look into debian/control inside the unpacked source, and you find:
<maxb> Package: gdb64
<maxb> Architecture: i386 powerpc s390 sparc
<maxb> That's the information which is lost, and then kludged back in via P-a-s so that quinn-diff can figure out whether binaries are missing because they're supposed to be, or because they haven't built yet
<persia`> So you don't actually trust the .changes files for this.
<persia`> Which explains why it wasn't possible to trick openbios-sparc to generate arch:all in the arch:any run.
<maxb> What's the .changes got to do with this?
<persia`> The .changes generated from a build is *supposed* to list all the packages that the build generates.
<persia`> So, if you trusted the changes, you wouldn't need to run quinn-diff to find out if stuff was missing.
<maxb> I'm not necessarily sure LP uses quinn-diff at any point (in fact I doubt it), I'm more describing the original purpose of P-a-s in Debian
<persia`> Which actually makes a lot more sense, since the archive system doesn't have trusted slave buildds.
<maxb> P-a-s is really all about figuring out whether you need to build something, rather than processing the results of a build
<persia`> So, in Soyuz, were we only have source-only uploads, I think it's safe to dispatch the source to the set of architectures defined in the .dsc, and accept the results it generates without P-a-s.
<persia`> Because we know we have to build on at least that set of architectures.
<persia`> The gdb example wouldn't generate gdb64 on amd64, armel, or ia64, but that ought be fine.
<maxb> Another aspect of P-a-s is figuring out whether a source needs a build on a particular architecture at all - i.e. perhaps it will generate zero binary packages on some archs
<persia`> That should never be the case if the .dsc has "any".
<persia`> (barring build failures)
<maxb> I know I've seen real cases of this
<maxb> I agree P-a-s is a total hack, fwiw, I just don't think we're in a position to get rid of it without a fair bit more work
<persia`> If you find a case where some architecture listed in the .dsc really produces 0 binaries intentionally, I'd love to know the package.
<persia`> But for now, I'll leave that alone, and just try to find someone to do X-Build-Affinity: (unless someone gives me a better name)
<bigjools> persia`: X-Arch-Indep-Build-Arch would be a lot more obvious
<persia`> bigjools: Changed in my notes.  Thanks.
<bigjools> thanks
<jml> g'night all
<mrevell> Goodly evelode everyone
<mrevell> Damn, where's the close button in Lucid? :)
<kfogel> intellectronica: ec2 builds known to be failing right now?  https://pastebin.canonical.com/29489/
<thumper> morning
<wgrant> jml: Hi.
<wgrant> jml: Ah, I see the email. That branch got stuck waiting for Soyuz to answer, I think.
<wgrant> Can somebody please land https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats/+merge/21828? The ec2test failed with the two buildfarm spurious failures that have been showing up recently.
<jml> wgrant: sure, I'll do it.
<wgrant> jml: Thanks.
<bdmurray> Is there anybody would could run a simple production db query for me?
<jml> wgrant: can you do me a favour and repeat the URL? I need to use my other computer to land it.
<wgrant> jml: https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats/+merge/21828?
<wgrant> This is why we don't use OS X.
<jml> wgrant, :)
<jml> wgrant, even if I bought a computer with Ubuntu pre-installed, I wouldn't have got Launchpad set up on it today
<wgrant> kfogel: Which were those failures? TestMinTimeToNextBuilder and buildd-slave.txt?
<wgrant> elmo: Do you know about the dsync link-dupes call in cron.publish-ftpmaster? Why would there be dupes? The archive is not yet split at that point.
<jml> wgrant: it should be running through ec2 now.
<wgrant> jml: It'll probably just fail spuriously again...
<jml> wgrant: perhaps.
<kfogel> wgrant: https://pastebin.canonical.com/29489/, but now I have a run that's been going okay for a couple of hours at least
<wgrant> kfogel: ENOTCANONICAL
<kfogel> wgrant: oops, sorry
<kfogel> wgrant: one sec
<kfogel> wgrant: http://paste.ubuntu.com/399555/, and it looked like a file existence error
<kfogel> wgrant: I'm treating it as spurious
<wgrant> kfogel: That's not one I've seen before.
<wgrant> But yes, I find it unlikely that you broke that.
<kfogel> wgrant: by the way, https://code.edge.launchpad.net/~kfogel/launchpad/78565-bug-comment-link-to-bug
<wgrant> kfogel: Ah, good.
<wgrant> I wish we had consistent rules for page titles, labels, breadcrumbs and root contexts.
<kfogel> wgrant: the "comment" vs "message" thing really bugs me :-).  And that's not even in the UI.
<kfogel> wgrant: I think everyone would agree that breadcrumbs should behave this way -- it's a bug in every case where they don't.
<wgrant> kfogel: Well, Messages are used internally for others things as well.
<cody-somerville> kfogel, Thats not spurious. That means your sourcecode tree is out of date.
<cody-somerville> err... maybe spoke too soon.
<kfogel> cody-somerville: oh, really?  I though I did an update this morning...
<wgrant> That's the opposite error to what one would normally get.
<wgrant> Plus ec2test updates download-cache and sourcecode before each build.
<kfogel> cody-somerville: but in any case, it's definitely up-to-date now, and the ec2 run is working.  So even if you're not right, you're not wrong :-).
<cody-somerville> Should file a bug on it. Ya know, the whole stop the line stuff.
<cody-somerville> This is the system speaking to us and we're just not understanding it.
<cody-somerville> <insert our LEAN type comments here>
<kfogel> cody-somerville: I think it may be what noodles775 was already working on though
<lifeless> wgrant: thats a while :P
<wgrant> lifeless: Hm?
<lifeless> 15:21 < wgrant> lifeless: From what I can tell, publish-distro.py takes about 20-25 minutes currently.
<wgrant> lifeless: Yeah, and apt-ftparchive is more than 17 minutes of that.
<wgrant> Fortunately 16 minutes of that can go away.
<wgrant> publish-distro.py itself runs for 20 minutes, normally. 2 minutes of that is inefficient domination (which I have down to <2s), 17 minutes of that is inefficient index generation (which I have down to <60s, or <20s if we ignore bzip2 indices, which we sadly can't), and the remaining <60s is other stuff.
<maxb> domination?
<lifeless> wgrant: cool
<wgrant> maxb: Domination has two related meanings in Soyuz: 1) Marking packages Superseded and 2) Marking Superseded, Deleted or Obsolete packages for removal when they have no references.
#launchpad-dev 2010-03-23
 * thumper EODs (for now), back tonight
<NCommander> Is there a good way to iterate thorugh SourceReleasePackage records? I need to iterate through ALL of them, download the source package, pop out the changelog, upload changelog to librarian, etc.
<NCommander> (or at least, all published ones)
<wgrant> You'll probably have to use query using Storm directly.
<wgrant> There's no way to iterate over them all, AFAIK.
<NCommander> wgrant: ugh .... *groan*
<NCommander> wgrant: do I have to use Storm in writing migrationtools?
 * NCommander is not even sure how to attack this problem in a sane manner
<StevenK> NCommander: Given the number of packages, is there a sane manner?
<NCommander> StevenK: I'm honestly not sure, since we're also including PPA packages in this
<StevenK> NCommander: Do the math -- 9,000 * Dapper, Hardy, Jaunty, Karmic and Lucid times about 40% is still 18,000
<NCommander> StevenK: ugh ....
<NCommander> migration == PAIN
<NCommander> StevenK: (necessary pain, but still pain)
<persia> why 9000?
<NCommander> StevenK: actually, we may want to include every source package so the changelog can be individually queried on a given SRP
<NCommander> That requires discussion with BjornT, and bigjools
<wgrant> We don't have the files for lots of SPRs.
<NCommander> wgrant: we don't?
<NCommander> I thought ever SPR had at least a few files
<NCommander> for the source packages anyway
<wgrant> All PPA sources that have been unpublished for more than a week, and all non-final primary archive versions from obsolete series.
<NCommander> wgrant: oh, that greatly reduces the numbers I was planning
<NCommander> wgrant: is there a special flag on SPR that states if there are files?
<wgrant> NCommander: No.
<NCommander> i.e., published, superseeded, obsolete, etc.
<NCommander> ugh
 * NCommander thought we had a removed from librarian status
<wgrant> You need to scan the LFCs of the LFAs of the SPRFs.
<wgrant> Sorry, the LFCs of the LFAs of the SPRFs of the SPRs.
<NCommander> wgrant: *blink*
<NCommander> ow
<ajmitch> needs more acronyms
 * wgrant gives ajmitch a DASBPR
<mwhudson> i know let's switch launchpad to a nosql database that doesn't support joins!
<ajmitch> wgrant: it's worrying that I know what that is
 * NCommander whacks mwhudson :-P
<NCommander> wgrant: https://edge.launchpad.net/ubuntu/jaunty/+source/hello/2.2-2 - they seem ot be there
<wgrant> mwhudson: My NMAF optimisation does a 9 table join.
<wgrant> NCommander: Jaunty isn't obsolete.
<NCommander> oh, d'oh
<NCommander> fail
<wgrant> It's possible that we only removed obsolete series' binaries. I reviewed the query, but really do not remember.
 * wgrant checks.
<NCommander> wgrant: https://edge.launchpad.net/ubuntu/feisty/+source/hello/2.2-1build1 - seems thats the case
<NCommander> yeah, the binaries were deleted
<wgrant> Oh, right, we kept the old primary sources just in case.
<NCommander> wgrant: that's a LOT of sources :-/
<wgrant> Although they were very very close to being accidentally nuked.
<ajmitch> perhaps to satisfy some legalities?
<wgrant> ajmitch: And to satisfy the bzr importer.
<NCommander> ajmitch: we have to provide sources as long as we provide debs, so as long as old-releases exist ...
<wgrant> NCommander: But old-releases still holds sources, doesn't it?
<NCommander> wgrant: oooh, bzr importer does pretty close to what I'm going to need to do, where's the source
<NCommander> wgrant: eh, I'm notsure
<ajmitch> NCommander: I never knew if you were also required to provide sources for every package version published as well :)
<wgrant> NCommander: It uses the API and stuff, though. You have DB access.
<NCommander> ajmitch: legally speaking, we probably do as long as we distributed it on a release CD
<NCommander> there's only a time limit to how long we provide sources if we use a written offer to get the source
<NCommander> (at least on GPL'ed sources)
<NCommander> Other licenses, YMMV
<NCommander> wgrant: re: bzr importer: but it had to iterate on all SPRs, and if it uses the API, won't be cleaner?
<wgrant> NCommander: But you need to get the SPR from the DB to set the changelog.
<NCommander> wgrant: wait, what?
<wgrant> NCommander: The bzr importer doesn't have DB access.
<wgrant> It uses launchpadlib.
<NCommander> wgrant: ugh
<NCommander> crap
<wgrant> You can't set the changelog with launchpadlib, so you need DB access.
<NCommander> *sigh*
<mrevell> Morning all
<mwhudson> gmb, allenap: have you seen ParallelLimitedTaskConsumer (in the context of bugwatch updating)?
<allenap> mwhudson: Yes, I looked at it briefly yesterday after I saw your message on IRC. That works with the job runner doesn't it?
<mwhudson> allenap: no, it's not at that level
<allenap> mwhudson: Okay, I'll look again.
<mwhudson> (the branch puller uses it, and that doesn't even connect to the database)
<NCommander> mwhudson: your up late :-/
<mwhudson> NCommander: a bit yeah
<mwhudson> NCommander: why don't i look at that branch of yours
<mwhudson> NCommander: it must be a considerably sillier time of night for you
<NCommander> mwhudson: can't sleep unfortunately
<mwhudson> NCommander: :(
<NCommander> mwhudson: insommia == suck
<allenap> mwhudson: Would this be a replacement for the ThreadPool stuff I've done in TwistedThreadScheduler?
<mwhudson> allenap: maybe
 * mwhudson has to go away sorry
<allenap> mwhudson: Thanks for the pointer, bye :)
<mwhudson> allenap: talk to jml maybe
<allenap> Okay.
<deryck> Morning, all.
<wgrant> bigjools: Did you inspect any of the data collected by the script yesterday?
<NCommander> mwhudson: thanks for taking a look at the branch again
<bigjools> wgrant: no
<bigjools> well, I looked at a few rows in the table but didn't closely check
<NCommander> When are data migrations done usually? (i.e., importing changelogs into librarian and updating the SPR record)
<wgrant> NCommander: That's a special one -- it can be done at any time after the release, and will take a long while.
<NCommander> wgrant: I wonder if we're looking at hours, days, or weeks :-/
<wgrant> NCommander: We're probably looking at around 200000 sources.
<NCommander> bigjools, wgrant: so it sounds like to me, to migrate the changelog data, I'm going to need a script that uses Storm to access the DB, access each SRP where changelog = NULL, download the source package, pop out the debian/changelog, upload to librarian, update the SPR and then go on to the next record
<NCommander> wgrant: that seems low ...
<jml> allenap, what?
<bigjools> NCommander: sounds about right to me
<NCommander> bigjools: I assume at some point we may want to do this for copyright data as well?
<wgrant> NCommander: There are only ~18000 primary sources per series, multiplied by a couple for the non-final versions, multiplied by say 10 for the number of releases, excluding most of those before Dapper (they weren't in LP).
<bigjools> NCommander: you can make it faster by using store.execute()
<wgrant> Then 30000 published PPA sources.
<NCommander> (I think stub would be very happy to make the database loose several GiB of data)
<bigjools> I don't care about PPA sources
<NCommander> bigjools: hrm ...
<NCommander> bigjools: we don't? I thought part of native sync sources was to let us go PPA -> main archive with a button click
<allenap> jml: mwhudson mentioned ParallelLimitedTaskConsumer as (maybe) another/better way to solve the problem in checkwatches, of half-way Twistifying it.
<bigjools> we also need to be very, very wary of the increase in librarian disk space required
<wgrant> bigjools: The construction time for the few million Storm objects is going to be negligible compared to extraction time.
<bigjools> I think the losas would quite like an estimate
<jml> allenap, oh yes
<stub> bigjools: Its easier to scale the Librarian than scale PG.
<bigjools> NCommander: well I meant historic PPA sources
<jml> allenap, do you want to run a bunch of tasks in parallel but with limitations until there's nothing left to do?
<allenap> jml: We don't have time for it now, but yes, it does look interesting. It's working for now :)
<NCommander> bigjools: oh, of course.
<NCommander> bigjools: if we did the entire PPA history, I think the datacentre would probably implode
<bigjools> wgrant: I am more concerned about exhausting memory
<wgrant> bigjools: Historic PPA sources don't exist any more, so there's no question of that.
<allenap> jml: Yes, basically. At the moment, because each task must always run in a thread, I'm just using a ThreadPool.
<jml> allenap, :(
<NCommander> stub: oh, so to answer your quesiton, yess, we will have NULLs for changelogs
<bigjools> wgrant: quite true
<allenap> jml: But when we get time to switch to non-blocking IO then something PLTC looks more interesting.
<NCommander> bigjools: it sounds like where I'm sitting we should just attempt a changelog import wherever we have sources available, and make sure the code can handle changelog=NULL
<allenap> jml: I've not looked at how it's used in LP, only at its implementation, so I'm not 100% sure how it's meant to be used.
<wgrant> NCommander: I think that's all it's possible to do.
<bigjools> NCommander: +1
<jml> allenap, the implementation is surprisingly hairy for a simple concept.
<NCommander> bigjools: great, now I need to figure out how to write it :-/
<bigjools> NCommander: good luck :)
<allenap> jml: PLTC or the one in checkwatches? If PLTC, I agree! ;)
<jml> allenap, concurrency is hard. luckily you're using threads so you can pretend it isn't.
<NCommander> bigjools: (I think raw_source_changelog will probably land this week in db-devel, I got a stub +1, just waiting on BjornT and the second code review)
<wgrant> NCommander: Stealing code from archiveuploader will make your job very easy.
<NCommander> wgrant: which code?
<allenap> jml: Yeah, that's the approach we're taking for now. Ear defenders and blinkers at the ready.
 * NCommander notes ripping packages apart isn't that hard
<jml> allenap, actually, PLTC is perhaps an ideal candidate for an informative doctest
<wgrant> NCommander: The code that takes a source package, grabs files into a temp dir (some from the DB), extracts them, and inserts the changelog into the DB.
<NCommander> wgrant: oh, the code we ripped apart last week :-)
<wgrant> Most of it, yes.
<NCommander> wgrant: that wasn't the part I was worried about, I was more worried about iterating through the databsae, but I think thats just due to not knowing much of storm.
<allenap> jml: You're right. I'm so used to doctests being muddled that I didn't even think to look for one.
<jml> allenap, it doesn't have one
 * NCommander isn't even sure how we can test the importer script sanely
<bigjools> NCommander: juse use store.execute() and you can write raw SQL.  This is better in this case otherwise it will exhaust memory when caching objects.
<bigjools> this is a real problem in some scripts that iterate a lot of objects
<bigjools> NCommander: we can test this initally locally on your laptop, then on dogfood
<NCommander> bigjools: fair enough, but I'm going to need quite a few joins to get all the files.
<jml> allenap, http://paste.ubuntu.com/399839/ is probably the best example
<NCommander> bigjools: my laptop has a small SSD. Your going to blow it up
<bigjools> NCommander: welcome to Soyuz SQL
<bigjools> NCommander: I have an SSD too.  Aren't they great? :)
<NCommander> bigjools: is that SQL or SOL :-P
<jml> allenap, the consumer just has an upper limit of parallel tasks and a logger. You tell it to consume from a "source" and it just runs whatever the source gives it until the source is exhausted
<NCommander> bigjools: I dunno, I didn't want to burn my out within a week
<NCommander> bigjools: this laptop takes 510 minutes to run the entire LP test suite (although it didn't swap doing it, which was nice)
<allenap> jml: That's nice.
<bigjools> that's what guarantees are for :)
<bigjools> NCommander: 510?!  jeez, mine takes about 180
<NCommander> bigjools: I dunno why it took so long.
<bigjools> how much free memory and what cpu?
<NCommander> bigjools: 8 GiB of RAM + 2.2Ghz dual core processor
<NCommander> I'm thinking something must have hung for awhile
<bigjools> yes
<NCommander> I got more failures than ec2test
<jml> allenap, the source just has to implement start(task_consumer) and stop(), and to call appropriate methods on the consumer when it has tasks. we've already implemented a polling source.
 * NCommander will probably setup a karmic instance or UEC on my old laptop
 * bigjools needs to go back to working now
<NCommander> bigjools: sorry to distract you :-)
<bigjools> that's my life
<jml> allenap, the nice thing for us is that other than the source, the rest of the code would be identical even if we had some kind of event-driven infrastructure (e.g. message queues)
<bigjools> I have a dead desktop today, I'm upset
<jml> allenap, anyway, I'll let you get to it.
<NCommander> bigjools: ugh, what happened to it?
 * NCommander notes he's run LP onhis ia64 desktop before :-)
<bigjools> NCommander: it powers up but the monitor doesn't wake up, no bios messages etc.
<allenap> jml: I'm can't immediately imagine how that'll fit into checkwatches, but it probably will. Thanks for explaining it.
<jml> allenap, np.
<NCommander> bigjools: sounds like your mainboard died
<bigjools> NCommander: I am thinking gfx card actually
<NCommander> bigjools: try the intergrated graphics?
<bigjools> it was working last night until I tried to suspend it
<bigjools> and hung on the way down
<wgrant> bigjools: lp.services.buildfarm?
<bigjools> wgrant: possibly
<bigjools> increases finger strain though :)
<bigjools> naming is the hardest part of software engineering
<wgrant> Yes :(
<jml> funding is the hardest part of software engineering
<bigjools> fun is the hardest part of software engineering
<NCommander> bigjools: not with python; just import fun
<jml> (or maybe I meant concurrency, or dealing with massive applications)
<bigjools> no, naming stuff is still the hardest :)
<jml> bigjools, I'll respectfully disagree.
<bigjools> jml: it's tongue-in-cheek
<jml> bigjools, :)
<wgrant> What became of the generalised builder history discussion?
<bigjools> ongoing AFAIK
<bigjools> it got a bit stuck on the fact that the translations jobs don't have a build record
<wgrant> Right.
<wgrant> And adding one is going to make everything even more horribly duplicated.
 * wgrant likes http://people.ubuntu.com/~wgrant/launchpad/buildfarm/new-build-model.png as a solution to that.
<bigjools> wgrant: yes, we had something similar in mind
<bigjools> noodles775 and I discussed this
<wgrant> Ah, good.
<bigjools> I want to ditch the use of Job
<bigjools> it's been a royal PITA
<wgrant> Yep.
<bigjools> running intel gfx at 1920x1200 really shows up how awful the performance is :/
<wgrant> Lucid or otherwise?
<bigjools> anything
<bigjools> lucid right now
<wgrant> Hmm.
<NCommander> bigjools: I've nevcer had an issue with intel gfx running two screens at about that resolution
<bigjools> it's fine at laptop resolutions, but on external monitors it's quite sluggish
<wgrant> My old i915 drives a 1920x1080 TV fine.
<bigjools> it could of course be kwin...
 * wgrant is reminded of the oddities in Wellington.
<bigjools> but what about the laptop?
<bigjools> <rimshot>
<gmb> Does anyone have any idea why I'd be getting psycopg2 import errors on db-devel branches? AFAICT it's installed correctly but LP seems not to see it. Could it be connected with the buildbot failures we saw yesterday?
<wgrant> jml: Hm, lp/__init__.py isn't exactly an obvious place for that documentation.
<jml> wgrant, it is if you are browsing API docs, I think.
<jml> wgrant, such as at http://people.canonical.com/~mwh/canonicalapi/index.html
<wgrant> Hm, those are nice.
<jml> wgrant, I reckon we could do a lot worse than emulate Bazaar, and have a doc/hacking/ directory
<jml> or however they spell it -- it's easy to find when you need it.
<jml> I really really need to end this zope testing branch and figure out what the hell to do with these images
<bigjools> gnargh, testfix mode
<wgrant> jml: Any idea why http://people.canonical.com/~mwh/canonicalapi/lp.soyuz.interfaces.build.IBuild.html has only IBuildFarmJob linked, and none of the others?
<wgrant> bigjools: That db_lp failure from four hours ago with me in the blamelist?
<jml> wgrant, backticks, I think.
 * bigjools is checking
<jml> wgrant, alternatively, it's because the others are backticked but lack sufficient context for pydoctor to find them
<wgrant> jml: getFileByName' ILibraryFileAlias is backticked.
<bigjools> wgrant: test_sigusr2 failure?
<wgrant> And IBuildFarmJob isn't imported in the other file.
<wgrant> bigjools: No idea. I can't see buildbot failures.
<bigjools> oh
<bigjools> looks spurious
<jml> wgrant, in that case, I don't know. I *do* know that it's a bit of a black art, and that mwhudson is the person to ask.
<jml> heck, I don't even know if we have a convenience thing in tree to build those docs
 * wgrant didn't find one.
<bigjools> jml: +1 for separate branch but it will be hard until we can split up LP better than it is
<jml> bigjools, it will be hard.
<wgrant> That sounds really really hard.
<bigjools> but totally necessary for a sane development experience
<jml> bigjools, it will always be hard. so will collaborating on a 350k line tightly-coupled Python app.
<bigjools> there are some strong arguments to make it more loose
<bigjools> it also begs the question of why we made lib/lp/xxx in the first place
<jml> bigjools, to make the structure clearer.
<noodles775> step-by-step
<bigjools> what structure?
<jml> bigjools, it then turned out that we had less structure than we'd hoped
<bigjools> exactly :)
<jml> bigjools, but we had even less of an idea of that or the scope of that when everything was in one big directory
<jml> bigjools, also, I think that it's almost always a good idea to change your code to match the way you think about the problem space
<bigjools> no arguments from me there
<jml> bigjools, in our case, we were talking always about "code" this and "bugs" that and "registry" whatever. the lp apocalypse has actually made our code clearer imo
<wgrant> But the API constraints make a complete split difficult.
<wgrant> And the Soyuz<->Registry split is insane.
<bigjools> I agree, but it's also pointed out the nightmare of tight coupling
<wgrant> (although I guess most of the Soyuz-related methods in Registry have belonged on IArchive since archive-rework three years ago)
<bigjools> wgrant: it's a good idea to split it, the problems come because of other reasons
<bigjools> yes
<jml> wgrant, API constraints are the biggest technical blocker I can see.
<jml> wgrant, bigjools: does the buildfarm code actually depend on stuff outside of the buildfarm?
<wgrant> jml: Only a little.
<wgrant> I've removed most of them.
<jml> so it wouldn't be _that_ hard to move to a separate branch
<wgrant> eg. BuildBase stuff depends on Archive.
<wgrant> But BuildBase could reasonably move to Soyuz.
<wgrant> And the builder UI code is still in Soyuz.
<bigjools> well
<jml> I mean, apart from the base level of hard involved in doing anything that changes our build & deployment processes.
<bigjools> there's probably room for another module between the buildfarm and soyuz
<wgrant> jml: But splitting something out when it depends on the DB?
<bigjools> whatever "soyuz" is nowadays anyway
<jml> wgrant, I don't see why that's an issue
<wgrant> jml: Oh, also, buildmaster is currently tested with Soyuz objects, since it has no concrete job implementations of its own.
<wgrant> What is the purpose of splitting it into a separate branch? It can't be run independently.
<bigjools> ideally it should be
<jml> wgrant, can't it? perhaps I misunderstand what it is.
<jml> I'd put the bloody buildd slave code in a separate branch first, tbh.
<wgrant> Ideally it should be.
<wgrant> But it probably depends on DB details and blah blah blah.
<wgrant> Although I guess it really needn't.
<bigjools> the only thing I know about is tachandler.py
<jml> wgrant, ideally it wouldn't, but that's not _actually_ a blocker
<bigjools> otherwise it really should be ripped out
<wgrant> Why is tachandler not somewhere like Twisted?
<bigjools> nfi
<jml> wgrant, a few reasons
<jml> wgrant, in fact, Twisted might have grown better APIs since we wrote tachandler
<jml> wgrant, it's not code that a lot of Twisted apps feel the need for.
<wgrant> jml: Well, a few unrelated bits of LP do. What's so special about them?
<jml> wgrant, tbh, I don't know why we use it.
<jml> wgrant, other than for integration tests.
 * jml greps
<jml> ReadyService, afaict could be in Twisted and is only really there to make testing easier (icbw of course)
<jml> and TacTestSetup is essentially an API for manipulating out-of-process Twisted applications
<jml> wgrant, I guess someone could spend a few hours and end up by getting rid of tachandler
<jml> maybe I'll take on splitting out the buildd code as my hacking task after I land the zope.testing upgrade and my poppy work.
<jml> wgrant, there you go, I asked on #twisted.
<jml> that is progress.
<wgrant> I should probably get my buildmaster cleanup branches into potentially mergable condition.
<jml> I should probably do some strategy today :)
<NCommander> wgrant: jml: any chance you can make the ABORT button work?:-)
<bigjools> so, removing my gfx card and booting works fine - I just don't get a desktop :)
<wgrant> bigjools: Ah, easy fix, good.
<NCommander> bigjools: hope that card is under warrenty?
<bigjools> well easy, but expensive :()
<bigjools> no, it's about 18 months old
<wgrant> NCommander: The slave changes aren't hard. The main difficulty is working out how to transfer the request through the DB.
<wgrant> Do we have new Aborting and Aborted DB statuses?
<bigjools> we also need a CANCELLED
<bigjools> we're abusing SUPERSEDED right now
<wgrant> Right.
<bigjools> lamont will love whoever fixes that
<wgrant> This should probably be thought about in the schema redesign, if that ends up happening.
<wgrant> Since it applies to all job types.
<bigjools> yes
<bigjools> but as much as I hate it, the job status is a separate status to the build status
<wgrant> +ALTER TABLE ONLY build ADD CONSTRAINT build__archive__fk FOREIGN KEY (archive) REFERENCES archive(id) on delete cascade;
 * wgrant cries.
<bigjools> ha, if I had a hdmi cable my desktop would be working, it has an onboard nforce 750a
<wgrant> bigjools: It has onboard HDMI but not VGA?
<bigjools> yep :/
<wgrant> WTF
<bigjools> yep :/
<jml> NCommander, abort button? what do I look like? a programmer?!
<bigjools> wgrant: gar, that one is wrong isn't it. Crapola.
<wgrant> That is a seriously fucking dangerous DB patch.
<bigjools> oO
<wgrant> Oh, no, it does leave a constraint or two that will stop it cascading too far into other archives. Good.
<wgrant> (If BPPH.binarypackagerelease had been altered to cascade... oh dear)
<bigjools> wgrant: so, I could make the Build.archive on delete NULL, but that's gonna break lots of shit
 * bigjools thinks
<wgrant> That's what I said last night :)
<bigjools> so there's a few options
<wgrant> There is no precedent for doing this sort of thing.
<bigjools> 1. don't delete the archive
<bigjools> 2. delete the binaries
<bigjools> 3, ??
<bigjools> 4. profit
 * bigjools is fed up with people insisting that ppa deletion is easy
<wgrant> Deleting the archive means deleting all sorts of stuff. Future copy logging will be difficult, PackageUploads (and thus all upload accountability information and changes files) will have to be removed...
<wgrant> It's not like a branch, which is one object, pontentially with a non-critical tendril attached to another branch.
<bigjools> yes
<bigjools> however I feel very strongly that we should delete them in their entirety if at all possible
<jml> wgrant, stacking is a critical tendril
<wgrant> I am in two minds about it. It's difficult and wrong, but also much cleaner.
<wgrant> jml: But you deny in that case, don't you?
<jml> wgrant, maybe.
<bigjools> we could delay the full deletion until the last PPA that uses another's packages disappears
 * jml really likes the infrastructure abentley set up for deleting branches
<bigjools> I'm starving and I really like the fact that my fridge is full of food, bbiab
<jml> heh heh
<stub> You can delete the things its safe to manually, and leave the constraints there to raise exceptions when necessary.
<kfogel> intellectronica: you saw https://code.edge.launchpad.net/~kfogel/launchpad/78565-bug-comment-link-to-bug/+merge/21896 ?
<dobey> hi all. i'm having some trouble filtering bugs for rhythmbox-ubuntuone-music-store (Ubuntu). they seem to be getting past my procmailrc filter (though all other Ubuntu bugs are being filtered just fine)
<maxb> Hi dobey. Perhaps you mean to be in #launchpad ?
<dobey> but for rhythmbox-ubuntuone-music-store bugs, there seems to be a newline inside the header... perhaps causing procmail to not interpret the header properly?
<dobey> maxb: no, i don't think so. this seems to be a bug in the launchpad bug mail creator
<maxb> Please paste the mail headers to a pastebin.
<bigjools> wgrant: still around
<bigjools> ?
<dobey> maxb: http://pastebin.ubuntu.com/399993/ <- this is from one mail that's failing
<dobey> maxb: and http://pastebin.ubuntu.com/399994/ is from a mail that is successfully filtered
<maxb> dobey: I believe that's header folding, as specified by rfc2822.
<maxb> I.e. it's your filter's fault
<maxb> sorry
<dobey> * ^X-Launchpad-Bug: distribution=ubuntu; sourcepackage=\/[^;]+
<dobey> that's the filter... and it works for the 59 other Ubuntu packages that i've gotten bug mails for
<dobey> i'm just trying to understand what the difference is
<intellectronica> kfogel: i did, and for some reason i thought i approved it. it was early in the morning so i was probably confused. r=me and will tick the mp now
<dobey> and others seem to have 'folded headers' too
<thekorn> dobey: the "\n " between distribution=... and sourcepackage=
<dobey> oh right
<dobey> blah
<maxb> dobey: A fully compliant filtering solution would apply rfc822 unfolding before applying regexps to header values.
<maxb> OOI, do db patches ever get renumbered, or is the idea that the DBA only assigns them a real version number when they will be nearly guaranteed to land in the upcoming dev cycle?
<BjornT> maxb: the db patch numbers aren't version numbers. they are more like identifiers. it doesn't matter of a db patch gets a number and never lands.
<maxb> But, they do control the relative ordering of patch application
<maxb> (right?)
<BjornT> maxb: yes. so in theory it could cause problems not landing them in order, but not really in practice. i can't ever recall that causing a problem.
<bac> hi leonardr
<leonardr> hi bac
<leonardr> i'm on the phone but type away
<bac> leonardr: when using utilities/ec2 i'm getting the "File name too long" error from httplib2.  you did a fix for that in lazr.restfulclient.  should it not be in 0.9.12?
<leonardr> bac, i'd think so
<bac> leonardr: your fix was on 2010-02-09 so it should've been in 0.9.11 and definitely 0.9.12
<leonardr> bac, give me any information about the failure you can
<bac> leonardr: http://paste.ubuntu.com/400029/
<james_w> leonardr: is there a way to test the wadl that is generated for a particular entry from the LP test suite?
<james_w> leonardr: I have to make a change, but the current tests pass and they use the webservice calls in the story tests, and show that the representation is correct. The bug is that the wadl is incorrect.
<leonardr> james_w: unfortunately you are #4 in line
<mars> sinzui, quick question: if I have a bug about a page's HTTP cache headers, should that bug go against launchpad-web or launchpad-foundations?
<sinzui> foudations
<mars> sinzui, so launchpad-web is for UI only?  Not browser mechanics?
<sinzui> mars: web is for css/markup issues that are common to all applications. web is thus about the site design
<mars> right
<sinzui> mars: no mechanics. these bugs should be fixable by designers
<mars> hmm, good category
<mars> I was going through the list of bugs tagged CSS, and a number fell into that category.
<mars> There were three categories IIRC: inconsistent application of the LP style, stuff where the current UI design can't cope with the data, and browser-specific display issues
<sinzui> yes, I came to that understanding when I triaged the bugs gary_poster was uncertain of.
<sinzui> I note that users started putting application specific bugs in launchpad-web. I moved them to the specific app
<mars> hmm. orthogonal concerns: responsibility, and category
<bac> leonardr: i see in your safename replacement you limit to 117 (152-32-1) so it is less than 150.  but that is just limiting the key portion.  httplib2 then adds the cachedir name to it, which in my case is 52 characters
<bac> leonardr:         cacheFullPath = os.path.join(self.cache, self.safe(key))
<leonardr> bac: i see. you have a long cachedir name
<bac> leonardr: not really.  it is the default.  yours would be longer.  (leonardr > bac)
<bac> leonardr: i'll open a bug
<leonardr> is it the path name that is limited or the filename?
<bac> currently on the key is limited
<leonardr> i meant in the underlying ecryptfs filesystem
<leonardr> james_w: i'm not sure what you mean by the wadl generated for a particular entry. do you mean the wadl representation of /~leonardr? i suspect you mean something else but i'm not sure what, nor which wadl is incorrect
<james_w> leonardr: the wadl declaration for ICodeImport is incorrect
<james_w> it asserts there will be a "branch" parameter, when there is in fact a "branch_link" parameter
<james_w> I know the code fix, but I don't know how to test it from that angle
<james_w> I could do a test that ICodeImport.branch is a ReferenceChoice and not a Choice, but that doesn't seem ideal
<james_w> the issue is that lazr.restful changes its behaviour between the declarations and the marshalling when the attribute is an entry, but not declared with the extra information of a ReferenceChoice
 * james_w heads for some lunch, any suggestions welcome
<leonardr> james_w: ok, i need to think about this
<bdmurray> How can I know when my branch is rolled out on edge?
<bigjools> bdmurray: if you know the revision number that it landed with, you can compare against the rNNNNN at the bottom of each edge page
<bac> leonardr: https://bugs.edge.launchpad.net/lazr.restfulclient/+bug/545197
<mup> Bug #545197: Cache file paths are too long for encryptfs <lazr.restfulclient:New> <https://launchpad.net/bugs/545197>
<bdmurray> bigjools: I was hoping for a notification not something I'd have to check ;-)
<bigjools> bdmurray: ah :)  edge is normally gets rolled out each day
<bigjools> and please correct my English
<bigjools> bdmurray: if there was a bug linked to the branch, you'll see its status change when it sees the branch on edge or staging
<bdmurray> bigjools: okay, awesome that'll work
<leonardr> bigjools: the problem is a misconfiguration on dogfood
<leonardr> the service root for the 'api' virtual host needs to have its /beta/ removed
<bigjools> leonardr: ah ok, where do I edit that?
<leonardr> bigjools: launchpad-lazr.conf
<leonardr> [vhost.api].rooturl
<leonardr> you _also_ need to remove /beta/ from your dogfood root url, or launchpadlib's first request will fail
<leonardr> if you remove /beta/ from lputils.py but don't change dogfoot, the first request succeeds and the second one fails
<leonardr> if you fix all this and dogfood credentials still aren't written to disk, let me know
 * bigjools is trying now
<leonardr> james_w: ok, i understand what you're asking for
<leonardr> if you look at lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt you'll see a test that retrieves the wadl description of launchpad and does minimal tests to it
<leonardr> if you look at lazr.restful/src/lazr/restful/example/base/tests/wadl.txt you'll see a more detailed test that dissects a wadl file
<leonardr> you could use an xpath selector to grab the part of the wadl file you want to examine
<leonardr> but you might like this idea better
<leonardr> the launchpad wadl description is transformed into an html document
<leonardr> lib/canonical/launchpad/pagetests/webservice/apidoc.txt grabs that html document and prints it out
<leonardr> you could do something similar to verify that branch_link shows up under code_import
<leonardr> so you have several options
<bigjools> leonardr: ok, success on dogfood!  Thanks very much.
<leonardr> yay
<james_w> leonardr: what about ObjectLookupFieldMarshaller  checking for field.schema? Would that be too intrusive?
<james_w> leonardr: the idea being that it would refuse to unmarshall something that wasn't declared as a ReferenceChoice, and so not cause the skew between the wadl and the representation?
<mrevell> night
<leonardr> james_w: that might be a good way to stop errors from begin committed, since the normal wadl test would fail
<leonardr> try it and see if it interferes with existing tests
<james_w> ok, but I'm not keen on having to deal with eggs and buildout :-)
<leonardr> james_w: if you just want to try it, change the code in your existing lazr.restful egg
<leonardr> ~/.buildout/eggs/lazr.restful.*/...
<leonardr> it'lll take effect immediately
<james_w> ah, cool, thanks
<bac> Ursinha: i approved your MP
<bac> thanks
<Ursinha> bac: oh, thanks :)
<cody-somerville> wgrant, ping
<mwhudson> good morning
<jml> mwhudson, hi
<jml> mwhudson, I need an answer for you re https://code.edge.launchpad.net/~jml/launchpad/update-ec2-image/+merge/21804
 * mwhudson hadn't noticed it was back in question land
<mwhudson> oh, "do we need to support karmic"
<mwhudson> i guess not
<mwhudson> jml: replied
<jml> mwhudson, thanks. :)
<bac> leonardr: so the 144 limit you discussed with dustin is just for the name portion, not including the path?
<mwhudson> jml: "someone" should write a twisted success story for launchpad
<leonardr> bac: right, the path limit is more like 4096 characters
<bac> leonardr: ok.  sorry then for the red herring in the bug report
<leonardr> np, it worked out
<didrocks> gmb: flacoste: hey o/ will it be possible to finish the second merge review on exposing ssh key this week (https://code.edge.launchpad.net/~didrocks/launchpad/expose-sshkeys-bug-357235/+merge/20995)? it prevents me from uploading last Quickly into ubuntu
<james_w> is anyone else seeing a failure in lib/canonical/launchpad/ftests/../doc/webservice-marshallers.txt ?
<james_w> I'm getting a TypeError on devel
<james_w> make clean && make hasn't fixed it, and neither has anything else I have tried
<james_w> would someone try running "./bin/test -cvvt lib/canonical/launchpad/ftests/../doc/webservice-marshallers.txt" in a (fairly) up to date copy of devel?
<gary_poster> sinzui: would you mind helping me categorize the following as registry,  foundations,  or something else? https://bugs.edge.launchpad.net/launchpad-foundations/+bug/231797  It doesn't feel terribly foundations-y to me, but I'll take it if you think it's appropriate.
<mup> Bug #231797: no sensible way to use debian/watch files with launchpad hosted tarballs <Launchpad Foundations:New> <devscripts (Ubuntu):Invalid> <https://launchpad.net/bugs/231797>
<wgrant> cody-somerville: Hi.
<wgrant> james_w: What's the failure?
<james_w> TypeError: a float is required
<wgrant> More traceback?
<james_w> canonical.launchpad.webapp.adapter.set_request_started() isn't being called, or some variation on that
<wgrant> Hm. You've updated sourcecode and download-cache and everything lately?
<james_w> and I can't see what is supposed to call LaunchpadBrowserPublication.beforeTraversal
<james_w> hmm, I haven't done sourcecode
<wgrant> It'll be somewhere in Zope, probably.
<james_w> yeah, that's my problem :-)
<mwhudson> james_w: i get the same
<james_w> hooray!
<james_w> well, not really, but I'm glad I'm not alone in this ;-)
<mwhudson> http://pastebin.ubuntu.com/400246/
<james_w> yep
 * mwhudson tries db-devel too
<james_w> updated sourcecode and download-cache, did make clean && make and still see it
<james_w> also, why isn't buildbot complaining?
<mwhudson> lucid vs hardy maybe?
<james_w> that's possible
<james_w> I haven't upgraded any packages since this test was working, so I think there is a code change associated too
<mwhudson> fails in db-devel too
<mwhudson> james_w: i guess email the list and see what is said there
<mwhudson> james_w: it fails for me too in production-devel which suggests some environmental change to me
<mwhudson> and test failures from ec2
<james_w> right
 * mwhudson grumps off to lunch
 * maxb runs 'lpnochange python-apt', and is very happy to have scripted these rebuilds
<wgrant> Wasn't staging meant to be moved to Lucid reasonably soon?
<mwhudson> "reasonably soon after release" i think?
<wgrant> I thought it was meant to be before release.
<wgrant> But I heard that a few months ago, so things might have changed.
<maxb> It would seem a bit odd to run a beta distro in the datacentre?
<lifeless> maxb: why ?
<maxb> What's the rush?
<wgrant> Test.
<wgrant> Make it less disastrous than Hardy.
<wgrant> That sort of thing.
<lifeless> can't make changes to lucid after it releases
<lifeless> can't tell if its suitable for the dc if we don't deploy it
<wgrant> s/can't/shouldn't/
<wgrant> Much better to get the applications running on it semi-production now.
<lifeless> s/cant'/won't/ then
<wgrant> lifeless: Hardy proved otherwise.
<maxb> ?!
<wgrant> Hardy had far too many SRUs right after release. That showed that there wasn't enough testing, as usual.
<wgrant> The more testing Lucid gets in real environments before release, the better.
#launchpad-dev 2010-03-24
<wgrant> jtv: Are you going to adjust Translations jobs to return a BuildStatus, or the master to not require it?
<jtv> wgrant: my feeling is there'll be all sorts of consequences if I don't provide one...
<wgrant> Heh, yes.
<wgrant> It makes sense and is easy to, and it should unbreak lots of stuff, so...
<jtv> In particular, it'd be nice to know whether it was OK.  :-)
<jtv> btw bigjools ok'ed the idea of verifying slave build ids by re-generating & comparing.
<jtv> So that pares the whole structure down to 1 method.  :)
<jtv> He didn't like the word "bake" for the cookie though.  :(
<wgrant> What about IS?
<wgrant> And is there any reason for the cookie?
<wgrant> BQ + datestarted should be enough.
<jtv> Well it's a major trust boundary... why _should_ we give it all that info?
<jtv> So BQ + datestarted would be enough, but I'd like to hash them.
<wgrant> The build ID is public in logs from the start.
<jtv> Build is, yes, and afaict that's used as the key.
<wgrant> If I'm running a build on a builder and break out of the chroot, I can easily grab the ID from the existing real slave before I take over.
<jtv> But no need to provide starting points for a compromised slave to make well-educated guesses as to the BQ id
<jtv> aiui the slave knows the build id, but atm only knows the bq id because it happens to be encapsulated in the slave build id, right?
<wgrant> Right.
<wgrant> But what's the problem with possessing the BQ ID?
<jtv> Well I'm assuming it's bad for a slave to impersonate the slave build id of another slave.
<wgrant> It's not, really. The BQ is associated with the builder in the DB.
<jtv> wgrant: but is association always checked?
<wgrant> jtv: The slave ID is going to be checked by comparing the slave's ID with the ID generated from the BuildQueue associated with the slave in the DB, isn't it?
<jtv> Yes
<wgrant> So association is always checked.
<wgrant> There are one or two places on the master that use the slave build ID in logs, but they are non-critical bugs.
<wgrant> As long as it's not used for anything other than that check, it doesn't matter if somebody can impersonate another slave's build ID.
<jtv> I meant, is the association between the slave build and the builder always checked?
<jtv> And could a slave ever impersonate a pending or past job on the same builder?
<wgrant> jtv: Past jobs don't have BQs. Pending jobs don't have builders assigned.
<wgrant> there is a unique constraint on BuildQueue.builder.
<jtv> okay, but keep some elbow room for things like changes in replication (as an example).
<wgrant> The slave's build ID doesn't decide any behaviour on the master, except that it will compare the string against the one the builder is meant to have.
<jtv> by the way, scary thought: that all evolves around the build id...  what happens when different types of job have the same build ids (but for different BuildBase classes)?
<wgrant> It doesn't matter. We cannot guard against malice.
<jtv> This would be accident.
<wgrant> Er.
<wgrant> We're not talking about build IDs, are we?
<wgrant> We're talking about BuildQueue IDs.
<jtv> Well looking at the code, my impression is that the real identification happens by build id, and buildqueue id is merely a recurring pattern in the slave build ids.
<wgrant> The current code grabs the BuildQueue ID, and checks that the BQ's build ID matches the one from the slave.
<wgrant> BQ IDs are not reused.
<jtv> That's inside verifySlaveBuildID.  I'm talking about outside it.  But hopefully it's just a wrong impression.
<wgrant> Where outside it?
<jtv> Ah, if only I remembered that.  :)
<wgrant> Nothing outside of that should use the slave build ID.
<jtv> maybe I was just confused by the slave build id being passed as build_id.
<jtv> It's enough to drive one to glue.
<wgrant> Yeah, we should rename that.
<wgrant> jtv: I was confused by this same thing on the day after the open sourcing. It looked like a big hole.
<wgrant> But a little poking around revealed that the slave build ID wasn't actually used for anything important, despite how it appears.
<jtv> I'm still hoping to pay a small price in code, in return for having other people worry about all that after I've gutted the code and fled the scene.
<jtv> In fact I'm sure it's going to be easier to decide when the complexity around it is gone.
<wgrant> You win a thousand ponies if you can remove all references to the slave build ID except in dispatchBuildToSlave, the thing that will compare it, and the slave itself.
<wgrant> (not hard. probably mostly grepping and changing log strings.)
<jtv> Okay, that does sound like a good idea.
<mwhudson> wow
<mwhudson> my crazy no-hosted-area hacking seems to work
<thumper> cool
<thumper> my 'completely change the way merge proposal email' hacking seems to work too :)
<thumper> s/email/email is sent/
<mwhudson> thumper: hooray for redoing the underpinnings of launchpad
<thumper> \o/
<thumper> we rock!
<thumper> mwhudson: http://pastebin.ubuntu.com/400341/
<thumper> mwhudson: that is the new cron script :)
<mwhudson> thumper: cool
<thumper> hmm :(
<thumper> 887 lines for this pipe without new tests
 * thumper considers breaking it up more
<thumper> mwhudson: if you are after a break, we could have a quick call
<thumper> mwhudson: I have a few pieces of news
<mwhudson> thumper: ok
<mwhudson> grr branch-rewrite.py is broken on lucid
<wgrant> mwhudson: the PATH issue?
<mwhudson> for really stupid reasons
<mwhudson> wgrant: yeah
<wgrant> I thought that had been around for longer than that.
<wgrant> I've seen it for more than 6 months, I'm sure.
<wgrant> A bug was filed a couple of weeks ago against -foundations somewhere.
<mwhudson> i think it's time to stop for today
<magcius> Is there a difference in types of code that goes into the canonical.launchpad and lp packages?
<spm> magcius: no. just the config (obviously, server specific). it all comes from the same place.
<magcius> spm: is stuff moving from one package gradually moving to the other
<spm> hrm. I think I may have misunderstood your question and fired from the hip. are you asking if the code that runs launchpad.net is the same as is downloaded from https://code.edge.launchpad.net/~launchpad-pqm eg lp:launchpad/devel ??
<thumper> magcius: stuff is moving from canonical.launchpad into lp packages
<thumper> magcius: we had really big moves early on for the major app bits
<thumper> magcius: but it has been moving slowly since then
<magcius> thumper: what's stopping you from just doing a rope rename?
<thumper> magcius: I don't understand what you mean
<magcius> thumper: http://rope.sourceforge.net/
<thumper> magcius: mostly because we are not sure where we want to move stuff
<noodles775> Hey thumper, did you see the conversation re. the url traversal (and the resulting breadcrumbs) on Aaron's MP?
<thumper> noodles775: no
<thumper> anything I need to know about right now?
<noodles775> thumper: no, it's not urgent, but if you get a chance tomorrow (I'm just wondering if there's a reason we don't put recipes under the (base) branch).
<thumper> there was a reason
<thumper> I'm trying to remember it
<thumper> noodles775: the argument went round about...
<noodles775> The only reason I could think of (which Aaron also highlighted) is that it would be fragile if the base branch changes etc., but the original mockups didn't allow you to change the base branch of a recipe.
<thumper> noodles775: and I picked something
<noodles775> thumper: no problem, as long as you guys have thought about it.
<thumper> noodles775: but what about changing the name of a branch?
<thumper> noodles775: or changing the owner
<thumper> noodles775: both of those change the unique name of the branch
<thumper> noodles775: and since the branch is accessed through the id, the recipe url would chnage
<thumper> noodles775: also ~user/+recipe/name is short
<noodles775> thumper: I see, whereas both of those things wouldn't change the recipe url where it is currently? Great.
 * thumper nods
<noodles775> thumper: yeah, it just came up because I noticed the breadcrumbs are Person >> Branches >> Recipes
<thumper> well...
<thumper> should be: Person >> Branches >> Recipes >> build-my-cool-stuff
<thumper> :)
<noodles775> Yep.
<thumper> noodles775: I'd love for you to finish the review, it is blocking several branches:)
<noodles775> thumper: I'm doing it right now.
<thumper> noodles775: remember that we can iterate the ui
<thumper> noodles775: right now it isn't even enabled on edge
<thumper> or won't be
<noodles775> thumper: sure. It's not that I'm seeing things that are blocked, but more that I can't see all the questions you guys have already asked etc., so need to catch up by asking them :/
 * thumper nods
<thumper> fair enough
<wgrant> bigjools: Morning.
<bigjools> morning!
 * bigjools will disappear in a bit, my power is going out all day
<wgrant> Eww.
<bigjools> I can relocate but my email is on a server at home, so ...
<wgrant> Well, I've got two slight buildd-manager behaviour changes that are blocking buildmaster->soyuz shuffling that I was hoping you could OK quickly for me.
<bigjools> if small
<wgrant> http://bazaar.launchpad.net/~wgrant/launchpad/refactor-slave-architecture-check/revision/10567 <- buildd-manager is about to forget about DASes, so this crack-filled check needs refactoring. I don't think this diff is too bad, considering how wrong the check is.
<thumper> bigjools: why is your power going out?
<bigjools> thumper: power co is cutting trees around lines
<thumper> ah
<wgrant> bigjools: ... that requires turning the power off?
<bigjools> apparently
 * bigjools shrugs
<wgrant> How odd.
<bigjools> not really, if the trees are all over live wires
<wgrant> Heh, doesn't stop them here.
<StevenK> Oh yeah, they'll cut back trees if the wires are live or not.
<persia> Doesn't that have a tendency to cause fires?
<bigjools> wgrant: that looks hackish and I have to run now, I'll look more later
<bigjools> maybe noodles775 can look as well
<noodles775> Sure
 * bigjools -> outta here, back in ~30mins
<noodles775> wgrant: might even be worth creating a 'Work in progress' MP?
<noodles775> ah nice, getting rid of builder dependency on DAS :)
<wgrant> It is hackish, yeah, but the current one is probably just as bad.
<wgrant> (buildd-manager finds all the DASes. buildd-manager collects all buildds with a processor matching the DAS's family. buildd-manager iterates through the builders attached to each DAS, checking their arch tag against the DAS arch tag.)
<wgrant> The problem is that builders have processors in the DB, but the actual slave has an arch tag.
<wgrant> Although that check will go away entirely once we have multi-arch builders.
<wgrant> A branch further along in the series changes the above buildd-manager logic to 'iterate through all of the builders'
<wgrant> Which seems to make an awful lot more sense.
<wgrant> And lets me delete 400 lines of code.
<noodles775> Nice :)
<wgrant> So, basically, we can hugely simplify buildd-manager and remove its direct Soyuz dependency, at the cost of a temporary hack in a single Builder method, which will be fixed once we rethink builder arch affinity soon.
<noodles775> wgrant: I don't see what's worse about your check, than the one you removed? Both check whether the slave's builder_arch matches a das.architecturetag (only difference being that the das used to be passed in)?
<noodles775> So it seems like a good refactoring to me.
<wgrant> Right.
<wgrant> Both are making the best of a terrible situation.
 * wgrant will argue with bigjools when he returns.
<noodles775> wgrant: You're probably planning to anyway, but pls add a bug to that XXX reference at some point :)
<noodles775> wgrant: Why not include the processorfamily in the original query rather than as a separate check?
<wgrant> noodles775: This lets us give a slightly better error.
<noodles775> That would make your new code almost identical in functionality?
<noodles775> Ah, ok.
<wgrant> I suppose I could just say 'Mismatched slave architecture tag: foo'
<wgrant> But the old one gave the correct value too..
<wgrant> noodles775: Can you do a quick search for that bug? Lots of that sort of thing are still private.
<wgrant> There surely is one.
<noodles775> OK, I would have thought it would have been included in the XXX if there was noe. Checking now.
<noodles775> So just back to your raised exception, you could raise "Mismatched slave architecture, Architecture tag: %s, Processor family: %s"?
<wgrant> noodles775: I could, true.
<mrevell> Morgen
<noodles775> Hi mrevell
<noodles775> wgrant: nope, I can't see anything. Closest was bug 2072.
<mup> Bug #2072: Processor/ProcessorFamily inconsistency in current DB model <Soyuz:Triaged> <https://launchpad.net/bugs/2072>
<wgrant> noodles775: OK, thanks.
<wgrant> We need launchpad-builfarm :(
<wgrant> The fact that I have not broken any tests by changing the error message concerns me greatly.
<noodles775> wgrant: yep, grepping for "Architecture tag mismatch" or "BuildDaemonError" doesn't show any results in tests :/, but it's (yet another) chance to improved the codebase!
<wgrant> Yep.
<wgrant> Just working out how to.
<wgrant> But first, dinner.
<noodles775> Enjoy!
<wgrant> bigjools: https://code.edge.launchpad.net/~wgrant/launchpad/refactor-slave-architecture-check/+merge/22010, now with added tests.
<wgrant> Also, you disabled soyuz-upload.txt in the apocalypse because it was failing. It now fails for me only because of post-apocalyptic bitrot.
<wgrant> Should I fix and reenable it?
<noodles775> That would be great :)
<bigjools> wgrant: yes please!
<bigjools> wgrant: IIRC it failed because either poppy or something else was failing to start
<bigjools> and it was unobvious as to why
<wgrant> Seems to work OK now.
 * wgrant proposed a merge.
<bigjools> there's a comment in there about random failures IIRC?
<wgrant> Doesn't look like it.
<bigjools> hmmm
<bigjools> wgrant: there's an XXX from cprov about poppy
<bigjools> the reason I disabled it was that poppy simply would not start in that test and we had no idea why, since it started ok outside the test.
<wgrant> That file has enough XXXs...
<wgrant> Hmm.
<wgrant> It works fine inside now, though :/
<bigjools> the whole test is an XXX
<bigjools> we had a load of lucid test failures last time I ran the whole suite
<wgrant> There is only one persistent one.
<wgrant> The a-f hash changes.
<bigjools> yes
<wgrant> Although a strange webservice-marshaller.txt one seems to have shown up recently.
<wgrant> bigjools: Can you please have a look at that hack at some point so I can get it reviewed?
<wgrant> noodles775: Thanks.
<noodles775> np
<bigjools> wgrant: yep
<bigjools> wgrant: can you mark it "needs review"
<wgrant> bigjools: Oops.
<bigjools> I want to see if I can test a theory about a bug
<bigjools> if I click claim review right now, I get a "not allowed here"
<wgrant> bigjools: 'tis now Needs Review.
<bigjools> haha - noodles775 already reviewed it
<bigjools> so that's why claim review didn't work
<wgrant> Ahh.
<wgrant> that's an odd response to that situation.
<bigjools> it's a buggy response :)
<wgrant> But I guess I can't think of anything better.
<bigjools> it shoudl redirect back to the page with a notification
<wgrant> Probably.
<bigjools> btw we're not doing full ppa deletion in the first cut, to make life easier
<bigjools> just the repo
<wgrant> Ah, good.
<wgrant> Much easier.
<bigjools> very
<wgrant> bigjools: Is the review link happier now?
<bigjools> wgrant: well it was the "claim review" button that I was hitting but it's gone now
<bigjools> I'm just re-reading it to see what was discussed
<wgrant> Ah.
<bigjools> wgrant: so it's kinda yucky I guess but as long as nothing is breaking I am ok with it.  As you point out, the multi-arch builders will shake things up somewhat.
<wgrant> bigjools: There were no tests before, there are now, and it works in practice.
<wgrant> And it isolates the hack to a single method.
<bigjools> yep
<wgrant> Whereas it's currently deeply ingrained embedded in buildd-manager.
 * bigjools grumbles about "make run" bailing out when there's no process running as per a PID file
<bigjools> I also discovered this morning that the local launchpad.dev doesn't work without an internet connection :/
<wgrant> Urgh. What goes wrong?
<bigjools> didn't get far enough to work out why
<wgrant> I've done it lots of times on the train.
<bigjools> apache bails out saying it can't resolve
<bigjools> I've done it before too, but this laptop is a new lucid installation
<bigjools> so I suspect something has changed that's not part of the upgrade manager
<wgrant> Hmm.
<wgrant> bigjools: Thanks for the approval. I suppose you can't test today, so somebody else will have to land it?
<bigjools> yeah sorry
 * noodles775 sends it off.
<wgrant> noodles775: Thanks.
<wgrant> bigjools: np
<noodles775> wgrant: np, wanna add the commit msg?
<wgrant> noodles775: Done.
<deryck> allenap, hi.  Did that CP for checkwatches land?
<pabelanger> Does launchpad's ppa support building powerpc?
<noodles775> Hi pabelanger, the PPA overview has the list of currently supported architecturse: https://help.launchpad.net/Packaging/PPA
<gary_poster> bac, Google thinks we have reviewer mtg in 9.  Is it on Google-Calendar-doesn't-have-a-UTC-timezone crack?
<bac> gary_poster: the meeting has been moved to 1400UTC, which is at the top of the hour
<gary_poster> bac, ah, cool thanks
<bac> gary_poster: is that what you see?
<gary_poster> bac, yes.  btw, if there is not an action item to ask if leonardr has a demo on how to use launchpadlib to test in launchpad, you should add one, and ask him. :-)
<bac> gary_poster: oh, it is ready?  cool!
<gary_poster> yeah :-)
<bac> cool
<leonardr> bac: yes, look at lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt
<bac> reminder, reviewer's meeting in 2 minutes in #launchpad-meeting
<allenap> deryck: Yes, and it's in LPS, ready for CP when possible.
<deryck> allenap, excellent, thanks.
<deryck> I thought the reviewer meeting was in an hour according to UTC, bac ?  if not, I'm late. :-)
<bac> deryck: recall we had the discussion last week to move to 1400UTC.
<deryck> ah, ok
<kfogel> What's our preferred term for The Entity Known Either As A "Person" Or A "Team"?  Do we just say IPerson colloquially?
<kfogel> deryck: ^^
<deryck> kfogel, yeah, I think IPerson, Person, or person is the common usage.
<kfogel> deryck: ok
<barry> gary_poster, flacoste howdy!  what are your thoughts about upgrading lp:lazr.enum to 2a format?
<flacoste> barry: JFDI
<barry> flacoste: i like your style!
<flacoste> barry: you should be able to do that by clicking a button on LP
<barry> flacoste: /should/ being the operative word.  it's failed for me in the past :(
 * barry has his jfdi hat on today... literally
<jml> barry, if it doesn't work, let me know.
<barry> jml: cool
<barry> jml: button pushed
<gary_poster> barry: fwiw, go, go jfdi ;-)
<barry> gary_poster: rock!  thumper asked me to package it for lucid, which i've done and now i want to get it into universe
<gary_poster> great
<barry> jml: can you check to see what's going on.  the button provides no feedback but the page still shows it in 0.92 format
<jml> barry, no, I can't.
<barry> jml: :(
<jml> barry, it shows 2a for me though
<barry> jml: it must have *just* finished! yay.  thanks
<jml> barry, np. I'm glad it actually worked.
<jml> although a little disappointed we still haven't figured out ajax for this sort of thing.
<sinzui> bac: call? coffee and tea first?
<bac> sinzui: i require no refreshments.  give a ring when you're ready
 * jml afk for lunch
<barry> jml: yeah
<leonardr> flacoste, i'm going to write a launchpadlib branch that uses 1.0 instead of beta for the web service. i remember agreeing that changing it across the board was a better solution than having edge use devel and everything else use 1.0. (either because it's more consistent or because it's too big a change to launchpadlib to get into post-freeze lucid)
<leonardr> let me know if you disagree or remember differently
<jml> so
<jml> the weird-ass error I was getting trying to update the ec2 image was caused by a bug in openjdk
<jml> fixed by the wonderful folks at Ubuntu Server
<bdmurray> what's the process for chaning a bug from qa-needstesting to qa-ok?
<bigjools> just edit the tag
<bdmurray> bigjools: after testing it though right?
<bigjools> naw just change it :)
<flacoste> leonardr-lunch: i don't recall that, i don't think using a frozen version on edge makes sense
<cody-somerville> Does anybody run into this issue when running make? http://pastebin.ubuntu.com/400683/
<cody-somerville> bigjools, ^^
<bigjools> cody-somerville: first time you've run make?
<cody-somerville> on this branch? possibly, yes.
<bigjools> i see it sometimes
<bigjools> goes away if you make again
 * bigjools has to run
<bigjools> g'night all
<cody-somerville> bye bye
<mrevell> G'night all
<salgado> does anybody know why/how lib/lp/archivepublisher/tests/archive-signing.txt uses lucille as its db user?
<salgado> as usual, once you ask the question you find the answer:
<salgado> def archivepublisherSetUp(test):
<salgado>     """Setup the archive publisher script tests."""
<salgado>     setUp(test)
<salgado>     LaunchpadZopelessLayer.switchDbUser(config.archivepublisher.dbuser)
<salgado> except that this function is not used anywhere. crap
<salgado> lp/archivepublisher/tests/test_publisher_documentation.py has a copy of that, and this one's used.  nice
<gary_poster> heh
<deryck> sinzui, ping
<sinzui> hi deryck
<deryck> sinzui, hi.  there are 4 unassigned bugs on the 10.03 milestone for malone, which you added.  Are these things you and registry are planning to tackle?  Or was my team supposed to be looking into them?
<sinzui> ah, I was looking for those 30 minutes ago
<sinzui> deryck: There are 5 bugs I think
<deryck> sinzui, yes.  apparently I cannot count.
<sinzui> deryck: Those look like the one I want to fix by creating a single page to configure bug tracking
<deryck> oh, I would love that!
<deryck> We have another bug that could be fixed trivial with that, too.
<sinzui> deryck: I am moving them to 10.04. My teams needs to complete code configuration before we can start that.
<deryck> sinzui, ok, cool.  thanks for taking on this work!
<deryck> bug 538053 can be fixed when that work is done, too.
<mup> Bug #538053: "Enable bug tracking" link takes me to project edit form <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/538053>
<sinzui> yep, that will be fixed
 * sinzui has a single form and new url for that, but need a rocking bug tracker widget to make sense of expiration and remote project fields
<sinzui> deryck: I assigned all the bug tracker configuration bugs to the registry team. I think that will make it clear what we are doing
<deryck> sinzui, thanks.
<jml> sinzui, I'll look at it tomorrow.
<mars> sinzui, ping: question for you about the <style> definition in base-layout-macros.pt
<sinzui> hi mars
<mars> hi sinzui
<mars> sinzui, was looking at the output of bzr annotate lib/lp/app/templates/base-layout-macros.pt -r 8752.2.1
<mars> sinzui, and we pull in the 3.0 stylesheet using an @import directive.  Why was that done?
<sinzui> mars: that is following the beta through 2.0 rules. inheritance and overrides handled better when using that syntax
<mars> sinzui, "handled better"?
<sinzui> some rules are ignored using the common loading technique. That instruction was copied from the old template
<mars> sinzui, what is the comma loading technique?
<sinzui> sorry common way of using a href
<mars> sinzui, so you are saying that browsers ignore some rules if one uses a <link href=""> to load the stylesheet, and that can be avoided by using an @import ?
<cody-somerville> How do I run a specific unit test?
<sinzui> mars: There is some historic problems using simple href when rules need to override or set context like media. The import directive also hides the style rules from older or insane browsers
<mars> cody-somerville, bin/test -t the_specific_unit_test_name
<sinzui> mars: The use of @import may not be as needed in 2010 as it was in 2005
<mars> cody-somerville, be prepared for it to take a bit to load up all the layers.
<cody-somerville> ah
<cody-somerville> holy crap
<cody-somerville> my test passes
<leonardr> flacoste: ok, i'll make it per-server. any other defaults? how about staging?
<mars> sinzui, ok.  I'm investigating IE caching, and one hypothesis for IE not caching the stylesheet is SSL+@import.  So I will look into that.
<mars> sinzui, the other hypothesis is that IE sucks, and because of SSL, will load the stylesheet regardless of what I tell it.
<sinzui> mars: Yes, I think you are on the right trail. I do not think we will be hurt if we change our syntax now. I think we only need to verify the !important instruction
<mars> sinzui, have a page that uses it?
<mars> sinzui, a test case I can follow?
<leonardr> flacoste, here's the rationale for making 1.0 the default on staging
<leonardr> for most people, staging isn't for trying out the new web service
<leonardr> it's for trying out your new application
<sinzui> mars I do not have a test case, search the css for the few rules we have. They are probably color rules
<leonardr> most people will want to just change the service root to switch over to production
<mars> sinzui, ok, thanks for the guidance.
<leonardr> if the default for edge is devel and the default for production is 1.0, everyone will have to know that when you develop against staging you need to also specify version 1.0
<flacoste> leonardr: are you sure? i would think that the common use case for using staging is to try out the app you are developping, not trying out a new application
<flacoste> leonardr: which would hint at using the same default than edge
<mars> sinzui, I would never have guessed as to the reason for the @import technique.  browser code pain.
<leonardr> when i said "trying out a new application" i meant the same thing as you meant when you said "Try out the app you are developing"
<sinzui> mars: it is easier to hide css rules using @import, and IE is still broken :)
<leonardr> flacoste: my basic argument is that an app should not break just because you changed root="edge" to root="lpnet"
<flacoste> leonardr: it won't break
<flacoste> leonardr: hmm, well it may
<flacoste> if you are using features that aren't deployed
<leonardr> yeah
<flacoste> but that's a problem for the app developer
<leonardr> sure, but why cause problems for the app developer?
<flacoste> he's deploying an app for which the server service isn't production ready yet
<leonardr> he wrote an app thinking that edge and production were the same
<flacoste> well
<flacoste> either way the app developer has trouble
<flacoste> either he needs to switch to using an unfrozen API
<flacoste> because he wants to use latest features
<flacoste> hmm
<leonardr> in that case he at least knows what's going on
<flacoste> since 1.0 gets all new features eventually anyway
<flacoste> unless they are backward incompatible
<flacoste> it's probably safe to use 1.0 across the board
<flacoste> and opt-in to using devel
<leonardr> all right
<deryck> kfogel, can any of your branches move to landed now?  I need some room in the bugs lane. ;)
<kfogel> deryck: I believe they are landed, let me just check
<mars> sinzui, does this patch look sane? (4 lines changed) list-style-type:none;
<mars> margin:0 !important;
<mars> padding-left:0 !important;
<mars> }
<mars> argh
<mars> sinzui, http://pastebin.ubuntu.com/400729/
<kfogel> deryck: all landed: bugs <suppressing mup> 78565, 538226, and 481324
<deryck> kfogel, thanks.
<james_w> leonardr: does lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt lie about using canonical/launchpad/apidoc/wadl-testrunner-devel.xml? If not, when is that file generated?
<leonardr> i thought i got rid of any such claims...
<james_w> oh, silly me, it's shipped in the source
<james_w> I just deleted it as there doesn't seem to be an easy way of regenerating just the wadl without a full make clean
<james_w> and the failure mode of that test when you remove that file is rather unhelpful
<leonardr> james_w: as of this morning there should no longer be any wadl-testrunner-devel.xml
<james_w> it just serves you the full development wadl, leading the doctest to print out several hundred lines of CML
<james_w> XML
<james_w> ah, I haven't pulled yet today
<didrocks> kfogel: flacoste: I think I pinged you yesterday about merging my ssh branch, did you have some time to have a look at it so that I can release Quickly 0.4?
<leonardr> james_w: you'll be happier with that test if you pull
<james_w> thanks
<kfogel> didrocks: I don't think I saw that, but let me look now
<leonardr> james_w: if you just want to regenerate the wadl you can delete apidoc/* and make apidoc
<james_w> yeah, I can now
<didrocks> kfogel: https://code.edge.launchpad.net/~didrocks/launchpad/expose-sshkeys-bug-357235/+merge/20995 should do it :)
<kfogel> didrocks: note I'm not a reviewer, but I can comment
<didrocks> kfogel: sure, I guess the branch is quite in a good state (and if it can help to get an hack by flacoste or gmb, it will be good :))
<kfogel> didrocks:  what is the change at @@ -519,8 +520,8 @@ for?  It seems to be unrelated to this change.
<didrocks> kfogel: right, it was when I worked with jml as he liked to have PEP compliant file and it was more than 80 character IIRC :)
<kfogel> didrocks: well, phooey on jml for introducing unrelated changes, but I suppose he's far away over the water and I can't reach him :-).
<didrocks> it should be in a separate commit, not sure as it's quite old now :)
<flacoste> didrocks: we usually refrain from exposing DB id publically (in URL especially) could we use the keyid there instead?
<kfogel> didrocks: I see you have flacoste now, I cede the floor to him as he's actually competent to review this (witness the above comment).
<didrocks> flacoste: I didn't find any keyid key in the file, hence exposing the id, what should be exposed?
<didrocks> kfogel: thanks in any case ;)
<flacoste> didrocks: there is keyid
<flacoste> didrocks: and you have IGPGKeySet.get(key_id)
<flacoste> it will also make the URLs printed in the test stable
<didrocks> flacoste: I'm starting my VM right now and check again on the file, one sec
<james_w> didrocks: your tests don't seem to be doing as they state
<flacoste> didrocks: err
<james_w> didrocks: it says that the user should have no ssh keys to start with, but they have 1, then it says it is adding an ssh key and adds a GPG key
<flacoste> didrocks: i'm looking at GPGKey, you are doing SSH! doh!
<didrocks> flacoste: yes, it's SSH :)
<james_w> plus, should the collection link be <person>/sshkeys or <person>/+ssh-keys?
<flacoste> didrocks: yeah, that's fine then
<flacoste> james_w: it should be sshkeys, collection field have the field name
<flacoste> use the field names
<flacoste> it could be renamed though
<flacoste> the field could be calles ssh_keys
<james_w> ok, I need to fix code-import then I think
<flacoste> james_w: does it work?
<james_w> yes
<flacoste> hmm
<flacoste> then it might not be a problem
<flacoste> i recall that we have special traversal logic to look for a collectionfield
<didrocks> james_w: right, I have to fix the testsuite so
<james_w> oh, sorry, code_imports isn't a collection, it's a single object
<james_w> <branch>/+code-import
<flacoste> james_w: right
<flacoste> that's different
<flacoste> collection field must use the same name
<flacoste> not even sure exported_as would work here
<sinzui> mars: That looks good to me
<flacoste> yeah, it doesn't
<mars> sinzui, ok, I tried the patch, wasn't the issue.  Another hypothesis: the CSS does not come from the same domain as the HTML.
 * flacoste files a bug
<mars> sinzui, and the homepage is caching properly!  (where everything does come from the same URL)
<mars> s/URL/domain/
<mars> sinzui, in a template, how do I pull the current vhost?  base-layout-macros is pulling the mainsite root domain URL right now.
<mars> sinzui, looking at line 281, base-layout-macros.pt
<mars> maybe I could just make the URL relative
<mars> so use /+icing instead of foo.bar.com/+icing
<sinzui> mars: I am confused here. when the url is relative the browser always makes the request to the same domain.
<sinzui> if I load https://launchpad.net/ and am loading https://launchpad.net/+icing/style.css
<mars> sinzui, looks like base-layout-macros.pt is setting up an absolute URL, not a relative URL.  I might be able to change that.
<sinzui> mars: I understand you caching issue when I am on the bugs vhost, but when I am on the mainsite, there the URLs are the same.
<mars> sinzui, yep.  Experimenting now.  I'll let you know how it goes.
<didrocks> james_w: so, name12 has an ssk key, do you know other users that I can put in the testsuite without one?
<james_w> didrocks: you could use the factory
<james_w> person = factory.makePerson(name="ssh-user")
<didrocks> james_w: thanks, sounds easy to create a user :)
<flacoste> didrocks: review reply sent
<flacoste> didrocks: basically I suggest you use another user (no-priv) and use makeSSHKey instead of makeGPGKey in the test (and a couple of related suggestions)
<didrocks> flacoste: ok, thanks, I'm fixing that
<didrocks> flacoste: I'm just wondering how we can find the ssh public key id so. is it the keytext?
<flacoste> didrocks: yes
<didrocks> flacoste: ok, make sense :)
<mars> sinzui, gary, it worked!  http://www.webpagetest.org/result/100324_68A1/1/details/cached/
<gary_poster> mars, heh, awesome :-)
<mars> still fishy though.  Why does Mochikit refuse to die?
<sinzui> because no one has time to convert the old hide/show js rules to yui...
<mars> gary_poster, here is what we had before: http://www.webpagetest.org/result/100324_689B/1/details/cached/
<mars> strange strange.  launchpad.js disappeared all on its own?
<sinzui> mars: since no one is adding a feature to replace the rules, there is little incentive to update the callsites
<gary_poster> fewer bytes, but same net time :-/
<mars> wow
<mars> 40%? fewer bytes
<mars> and two fewer requests.  Something really fishy here.
<mars> see?  The request for the lp-gem 64x64 logo disappeared as well
<mars> and here is what is used to look like.  10 requests, 183KB weight: http://www.webpagetest.org/result/100308_5SEE/1/details/cached/
<mars> It shaved 1.2 seconds off the load time at least.
<mars> Internet Explorer is insane.  It is the only rational explanation.
<gary_poster> mars, you said "here is what we had before: http://www.webpagetest.org/result/100324_689B/1/details/cached/ " and then you said "here is what is used to look like.  10 requests, 183KB weight: http://www.webpagetest.org/result/100308_5SEE/1/details/cached/ "  Quick clarification of what is what?
<sinzui> gary_poster: ping
<gary_poster> sinzui: pong, but high latency
<sinzui> gary_poster: I was going to add a simple link to https://login.launchpad.net/ for users to change their password from the person+edit page. But I see the new SSO page, and I wonder if it would be better to add a "Manage your account" to the person profile page (where the Change password link was)
<gary_poster> sinzui: and where would this link?  We are planning to be a true openid consumer sometime this year, so if the link would be to login.launchpad.net, then that would be a problem.
<sinzui> okay, we have a bigger problem then...
<sinzui> gary: users do not know how to change their password. We took the link away.
<mars> gary_poster, the second link, with 10 requests, was recorded at the beginning of the month.  The first link you replied with has changed the <style>@import()</style> to a <link>.
<gary_poster> mars: ah, cool
<mars> gary_poster, so changing @import to <link> added one cache hit, seemingly without reason.  Changing the <link> domain added yet another cache hit.
<mars> oh, two, actually.
<gary_poster> sinzui: ok how is this for a plan
<mars> once again, without reason.
<gary_poster> sinzui: add a simple link now.
<sinzui> gary_poster: I can mark the bug as wontfix, and let chr deal the frustrated users, or we provide a link for the user to do this. maybe the answer is that we include a link from the profile page to an faq explaining that the user needs to find the login site and we do not know what that is
<gary_poster> sinzui: when we switch to random openid, we sniff.  if canonical, then we show the link
<gary_poster> if otherwise, we link to wiki page as you describe
<sinzui> gary_poster: then maybe we should always link the the faq (we already have one) and the let the user use the link from the faq
<gary_poster> sinzui: ok, +1, easier
<didrocks> flacoste: ok, I guess I fixed what needed to be fixed. When you have some time, if you can give it a look, it would be awesome: https://code.edge.launchpad.net/~didrocks/launchpad/expose-sshkeys-bug-357235/+merge/22067 Thanks! (just saw there is a conflict in configure.zcml now, but it's just removing the conflict tag and add the ssh exposal, if you want me to do this, feel free to ping me :))
<flacoste> didrocks: just approved it
<didrocks> flacoste: sweet, thanks!
<flacoste> didrocks: i'm in the middle of an update and haven't use ec2 land since last month, so i'm not reliable to land this
<flacoste> didrocks: you can bribe anyone on the team to land it for you though
<didrocks> flacoste: ok, I'll ping people tomorrow :) I can ping everyone on the LP team or should they be in a particular area of LP?
<flacoste> didrocks: anyone
<didrocks> flacoste: ok, thanks a lot :) good luck with your update!
<flacoste> thanks, it's finishing downloading, i'll know how lucky i am in a few minutes
<didrocks> heh
<mwhudson> didrocks: i can land your branch
<didrocks> mwhudson: great, do you need anything on my side? (the related bug is in the branch name)
<mwhudson> didrocks: i guess i need you to push the revision that fixes the conflict
<didrocks> mwhudson: sure, so I merge with ~launchpad-pqm/launchpad/devel/ right?
<mwhudson> yes
<mars> sinzui, fixed the JS domains: a 58% speedup overall!  \o/
<mars> sinzui, http://www.webpagetest.org/result/100324_68B9/1/details/cached/
<sinzui> wow
<mars> from 5.28s to 2.24s
<mars> now, tea!
<didrocks> mwhudson: should be ok now
<mwhudson> didrocks: i don't see  a new revision in the branch
<didrocks> mwhudson: sorry, the push wasn't finished, my VM is slow :/ it's ok now
<lifeless> is there a VM suitable for lp development around ?
<rockstar> lifeless, I have one that I stealed from abentley earlier this year.
<rockstar> Of course, it's 10GB, and I think that may cost you a lot if you're in a country that charges by data transfer.
<rockstar> lifeless, I've been working on a script that will shoehorn rocketfuel-setup into a chroot.  I've found the chroot is more helpful for what I want.
<lifeless> cool
<lifeless> Well, I'll roll my own VM for now, then.
 * rockstar nods
<lifeless> I just got a new laptop
<lifeless> which should be able to let me do the odd lp thing without excruciating pain
<rockstar> lifeless, congrats! Boy or girl? Weight? Size?  :)
<lifeless> lenovo x201s, 8G and 128G SSD
 * rockstar wonders if he can convince his wife he needs a new laptop.
<mwhudson> didrocks: argh, it seems you should have merged  ~launchpad-pqm/launchpad/db-devel/ not devel :/
<didrocks> mwhudson: oh isn't what I wrote above? no pb, can uncommit, revert, remerge and push --overwrite
<mwhudson> didrocks: that would be better it think, thanks
 * mwhudson apologizes for launchpad's slightly bonkers development process
<didrocks> mwhudson: I'm used to strike with Launchpad when I thought about contributing. That's ok, still happy that my changes can land :)
<didrocks> mwhudson: seems to be pushed :)
<mars> lifeless, gary_poster and bigjools both run them.  bigjools has instructions that I've been meaning to get from him.
<lifeless> it would be nice if it was as simple as bzr; I don't think thats on the cards anytime soon :)
<mars> lifeless, how about "lp-vmbuilder --from-branch=lp:foo" ?
<mwhudson> didrocks: one last think, can you set a commit message in the merge proposal?
<mwhudson> thing not think
<mwhudson> (the diff looks much saner now)
<mars> lifeless, that is about all you need really, isn't it?  vmbuilder to do the bundling, and a script in a bazaar branch to bootstrap the process.
<didrocks> mwhudson: I don't get the "set a commit message in the merge proposal". On the Launchpad merge proposal page? (don't know where to add that)
<mwhudson> didrocks: just under "Tis proposal supersedes a proposal from 2010-03-10." on the merge proposal page
<lifeless> mars: thats today, or as a vision of the future ?
<mars> lifeless, preferably near future.  bigjools and gary know how to do it, we just have to script it and share it.
<didrocks> mwhudson: oh, never noticed that. sweet :) and done (short but descriptive, I guess)
<mars> vm-based development was one thing the Lean training came up with as a solution to a number of developer issues.
<mwhudson> didrocks: thanks!
<didrocks> mwhudson: thanks to you!
<lifeless> mars: so, that would be interesting, but its still a much higher barrier than bzr has, even bzr-loggerhead
<mars> lifeless, yeah, true.  It could be improved towards a bzr-like setup, but I think dev's need to fiddle with systems-level things like apache get in the way, too.
<mars> lifeless, and adding another level of abstraction is so much easier
<lifeless> mars: all problems in CS can be solved by adding a layer of abstraction, except ...
<lifeless> having too many layers of abstraction
<mars> :D
<jelmer> lifeless: except performance problems :-P
<lifeless> jelmer: sometimes they can be solved that way, othertimes they are instances of 'too many layers' ;)
<jelmer> :-)
<mars> sinzui, oops, my calculations were wrong.  Turns out the IE improvements result in a 66% speedup.
<thumper> someone want to tackle the conflict in stable -> db_devel?
<mwhudson> some kind of user-local /etc/hosts file would sure help with launchpad
<mars> lifeless, off the top of my head, some problems with a bzr-like setup for LP dev: It is difficult to see a path to it; Some devs need Apache, Postgres, etc.;  It takes longer than just wrapping everything in a VM
<mars> two problems of perception, one of technique
<lifeless> mars: so, bzr devs need gcc, pyrex etc too - deps are deps.
<lifeless> apache, pgsql can be privately configured as regular users - there are even [shudder] buildout recipes I think
<mars> lifeless, yeah, but isolating Apache for local use by code?  I think you would have to ask mwhudson how easy that would be :)
<lifeless> I think the key thing I'm agitating for is not needing to alter system parameters to do tests - essentially the same isolation we'll want to do concurrent test runs anyway
<lifeless> mars: its easy
<mars> build recipes?  wow.
<lifeless> mars: isolating apache is pretty straight forward (if a little verbose). Definitely tractable, and once the code is written, its written.
<mars> lifeless, I guess the question to ask then is: If the work has always been straight-forward, why haven't we done it?
<lifeless> mars: people may not have considered it straight forward
<lifeless> fear-of-the-unknown perhaps
<mars> Got repeat-view pages times down to 3.6 seconds in China.  Much better.
<mars> And a bit more work can drop that to 2.6 seconds.
<mars> And if we drop SSL: 1.6 seconds.
<thumper> mars: can you make it faster in nz too please :)
<mars> thumper, dropping SSL is your only hope!
<thumper> :(
<mwhudson> didrocks: your branch is away in ec2 fwiw
<didrocks> mwhudson: sweet! Thanks ;-)
<mars> lifeless, you wouldn't happen to know how Zope serves resources like /@@/product-logo, would you?
<mars> lifeless, I'm looking to add Cache-control headers to the response
<mars> lifeless, actually, nevermind.  It is probably better to turn them into real image resources coming from +icing, same as every other static resource.
<thumper> :(
<mwhudson> oops, i need to merge _stable_ into db-devel
<mwhudson> not devel
<mwhudson> lunch time
 * thumper is installing lucid for the second time today
<thumper> holy crap
<thumper> 1818M will be downloaded
<thumper> onto my poor little laptop
<thumper> that's what you get for having both kubuntu-desktop and ubuntu-desktop installed
 * thumper goes to get some lunch while it downloads
<wgrant> So, I'm guessing that the OCR topic in #launchpad-meeting last night referred mostly to me.
#launchpad-dev 2010-03-25
<lifeless> thumper: :)
<thumper> can anyone remember how to get local mail delivery working?
<thumper> for the dev environment
<mwhudson> apt-get install postfix isn't it?
<mwhudson> i don't think there's anything special to do
<wgrant> I just use postfix with its stock local config.
<wgrant> Then redirect root's mail to my user.
<wgrant> And point Evo at my mbox.
<thumper> wgrant: do you use a .forward file?
 * thumper can't recall
<thumper> best not to install new software while a distro upgrade is in progress
<lifeless> you know
<lifeless> parallel dpkg would be fun
<thumper> hmm postfix is installed
<lifeless> as would fine grained locks
<wgrant> thumper: I just use /etc/aliases
<thumper> hmm, my local postfix install didn't generate a main.cf
<wgrant> dpkg-reconfigure postfix
<spm> postconf -n >> main.cf ;-)
 * thumper thinks of something that will generate an email to test
<lifeless> mail -s foo
<lifeless> also
<lifeless> exim!
<spm> telnet localhost 25
<thumper> spm: ooh hardcore
<spm> heh, more like being on hardened firewalls... :-)
<thumper> spm: do you know about postfix setup?
<spm> thumper: that'd be a yes. getting rusty; but yes.
<spm> in 2001, spm updated ~ 25+ firewalls and rather horrible mail funnel from sendmail to postfix. only 1 (minor) mess in the entire process. whee!
<thumper> spm: I'm trying to get my local postfix to route email through my servers config using the submission port
<thumper> my email client sends email through penhey.net:587
<thumper> hmm...
<thumper> perhaps I'll try something else
<wgrant> thumper: Why do you want it sent externally, to pollute your real mailbox?
<thumper> I was just thinking about that
<thumper> perhaps just relay into local user mail
<wgrant> It's what I do.
<lifeless> thumper: exim can do it trivially
<lifeless> :P
 * lifeless flogs the horse some more
<thumper> wgrant: what do you read your local email with?
<wgrant> thumper: Evolution.
<wgrant> I use it for my normal mail too, though.
 * mwhudson reads launchpad mail with mail(1)
<thumper> spm: how to I remove a deferred email from the postfix queue?
<spm> thumper: sorry - mid debugging a U1 problem. one sec.
<spm> thumper: postsuper -d <MSGID>
<thumper> where the hell is the default mailbox?
<thumper> /var/mail ?
 * thumper looks
<wgrant> thumper: /var/spool/mail/root
<wgrant> Also /var/mail.
<thumper> gah, kontact still won't see it
<thumper> I can see the mail in the file
 * thumper shakes his head
<thumper> haha
<thumper> spam caught it
<spm> rofl
<spm> PEBKA.... spam filter
<mwhudson> grar, another stable->db-devel conflict already
<thumper> hmm... I wonder if I should boot into gnome rather than kde when lucid starts here...
<thumper> hmm
<thumper> lucid decided to uninstall postgre 8.3 and python 2.5
<wgrant> Yeah, they're gone.
<mwhudson> reenable the launchpad ppa and install launchpad-developer-dependencies
<wgrant> Reactivate the ~launchpad PPA, and install lp-dev-deps again.
<wgrant> Damn.
<mwhudson> :)
<wgrant> Is ReviewerSchedule out of date?
<wgrant> I don't think jml is in APAC any more.
<thumper> well...
<thumper> the default font for my irc client is much smaller :)
<wgrant> And al-maisan is no longer around.
<wgrant> And there have been no European reviewers since Monday night, although that might be special for this week.
<mwhudson> thumper: interesting discovery
<mwhudson> thumper: the kernel import has imported about 140 megs of packs so far, which sounds about right
<mwhudson> thumper: the git.db file though is a hair under 6 GIGABYTES
<mwhudson> which suggests something is going a bit wrong
<wgrant> Is that why it is so damn slow?
<mwhudson> who knows
<mwhudson> it doesn't sound very helpful though
<wgrant> 'taking 26 hours' <- niiiiice
<StevenK> Hm. Isn't the topic out of date, and it's in fact week 3 of 10.03?
<wgrant> Even pear looks like it's going to take about 14 hours.
<wgrant> StevenK: I think so.
* mwhudson changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.03 | PQM is in open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes |
<StevenK> mwhudson: :-)
<mwhudson> StevenK: don't know what you're talking about!!
<thumper> mwhudson: OMG
<mwhudson> thumper: indeed
<thumper> mwhudson: where is jelmer_ when you need to smack him around a bit
<thumper> well gnome wasn't pleasent
<mwhudson> thumper: he knows, at least
<thumper> it killed x
<thumper> gwibber wouldn't start
<thumper> and made gdm take over instead of kdm
 * thumper has punted the little blighter back into his corner
<thumper> back into kde now
<StevenK> Hm. What do I do if I see a typo in a branch which I didn't notice until it landed? :-(
<StevenK> The typo is in a debug print, but I suspect it may throw an exception when it's run.
<mwhudson> StevenK: submit a branch to fix it
 * thumper is reinstalling launchpad-developer-dependancies
<thumper> somehow it got lost
<StevenK> thumper: Perhaps it got lost when the upgrade was removing obselete packages?
<thumper> StevenK: probably
 * thumper shurgs
<StevenK> Hm. How does one shurg?
<thumper> it is like shrugging, but different :)
<StevenK> :-)
 * thumper afk
<StevenK> mwhudson: Would you mind reviewing the change, it's a one-line patch
<mwhudson> StevenK: sure
<StevenK> mwhudson: https://code.edge.launchpad.net/~stevenk/launchpad/fix-typo-for-bug-503258/+merge/22098 ; but you'll need to wait for the diff
<StevenK> Which has turned up
<mwhudson> StevenK: is this covered by tests?
<StevenK> mwhudson: I could have sworn it was -- which is why I'm a little concerned how it got landed
<mwhudson> hmm
<StevenK> mwhudson: The original branch (fixes-bug-503258) changes the test to make sure that custom files are not private
<mwhudson> StevenK: ah
<mwhudson> StevenK: want to add a test then?
<StevenK> Is debug printing on when running under a test?
<mwhudson> StevenK: it doesn't matter does it?
<mwhudson> debug(1, foo.bar) will blow up if foo doesn't have a bar attribute
<mwhudson> no matter what debug does
<StevenK> mwhudson: It's a string subst, though
<mwhudson> StevenK: even more so then
<StevenK> Dear bin/test, plz output debug output
<StevenK> mwhudson: Ah ha! It just prints the wrong filename, but doesn't error
<wgrant> (Un)lucky.
<mwhudson> StevenK: heh heh
<StevenK> AttributeError: 'PackageUploadCustom' object has no attribute 'filename'
 * StevenK grumbles
<StevenK> So even my fixed branch is broken
<StevenK> Which does fail the tests, excellent
 * wgrant applies an axe to buildd-slavescanner.txt.
<StevenK> mwhudson: Do you need to re-rubber-stamp the branch now that I've fixed it so that tests actually pass, or shall I land it?
<mwhudson> StevenK: i'll have another look quickly
<StevenK> mwhudson: It's pushed, so you should be able to see it
<mwhudson> StevenK: +1
<mwhudson> land it
<StevenK> Aye, doing so
<lifeless> wheeee vm installs fly
<thumper> stub: ping
<stub> thumper: pong
<thumper> stub: my lucid upgrade installed postgres8.4 and removed 8.3
<thumper> stub: I have 8.3 back from lp dev deps
<thumper> stub: but I've lost all my setup
<thumper> stub: should I remove 8.4?
<stub> It did? Mine didn't, but I had both 8.3 and 8.4 installed.
<thumper> stub: also, I can't find the database setup code
<stub> You can leave it. You can even use it - the tree passes and I would have loved to switch this cycle but not enough time. I just haven't announced it.
<thumper> ah, ok
<thumper> stub: where are the setup instructions now?
<thumper> I looked in rocketfuel-setup but didn't see them
<stub> Dunno :-) Looking in the wiki now
<stub> https://dev.launchpad.net/Running says there is a script in utilities, although I've always done things manually (per the linked DatabaseSetup page)
<stub> ./utilities/launchpad-database-setup
<thumper> stub: will that work with 8.3 and 8.4 installed?
<wgrant> IIRC it will prefer 8.4 if it finds it.
<stub> I have no idea. I've never run it myself. Should be fine though - unless someone tried hard to break things it will use whichever PG is running on port 5432.
<wgrant> I believe it finds the latest version and hopes that it's running on 5432.
<thumper> hmm, I have 8.3 on 5432 and 8.4 on 5433
<wgrant> The script looks like it should die in that case.
<thumper> well that was a big fail
<wgrant> What died?
<thumper> the setup script
<thumper> failed in applying hunks
<thumper> failed to create the user
<thumper> failed on other bits
<thumper> then said...
<thumper> Looks like everything went ok.
<thumper> bullshit
<wgrant> Hahhaa.
<thumper> stub: it is all fucked!
<stub> thumper: https://dev.launchpad.net/DatabaseSetup still seems relevant and should work for 8.3. It should also work for 8.4 if you don't add 'max_fsm_relations = 2000' to postgresql.conf
<thumper> stub: well, 8.4 won't start at all says it has invalid directories et al
<thumper> stub: so I decided to go with 8.3 for now
<thumper> however
<thumper> http://pastebin.ubuntu.com/401000/
<thumper> why can't it get the working directory?
<wgrant> Because you're in the dropped cluster.
<stub> Because that script is destroying the directory you are running it from?
<thumper> oh arse
<stub> I'll update that script for 8.4 soon anyway. I think it is how the ec2 instances get setup.
<thumper> http://pastebin.ubuntu.com/401003/
<thumper> stub: ^^
<thumper> postgres8.3 says it is running
<stub> How does it say that?
<thumper> :(
<stub> It might be running, but not on port 5432
<thumper>  * Restarting PostgreSQL 8.3 database server                                                                                                                                 [ OK ]
<thumper> however
<thumper> $ /etc/init.d/postgresql-8.3 status
<thumper> Error: Invalid data directory
<thumper> WTF?
<wgrant> Why not purge both, install 8.4, and run the script?
<thumper> will that work?
<wgrant> It has for me.
<stub> Did you run step 4?
<wgrant> Except I installed 8.3, because LP didn't support 8.4 yet.
<stub> dropcluster,createcluster?
<stub> jtv updated the script to work with 8.4 IIRC
<thumper> stub: yeah
<stub> I'd go with wgrant's suggestion as it will probably work.
<thumper> ok
<wgrant> And even if it doesn't work, it won't get you into any worse a situation than you're in now.
<thumper> ahh
<thumper> 8.4 can't be removed because it fails to stop
<thumper> because it has an invalid data directory
<thumper> how can I force it harder?
<wgrant> kill it and see if it still tries to stop.
<thumper> it isn't running
 * thumper wonders
<mwhudson> stub: ec2 test uses the script yeah
<wgrant> Make the data directory uninvalid, or hack the script to not whinge.
<thumper> how do I make the data directory uninvalid
<thumper> I don't know where it is looking
<lifeless> 'valid'
<lifeless> the word is 'valid'
<wgrant> lifeless: Well, I didn't want to take it that far.
<wgrant> It probably doesn't need to be a valid postgres data dir. It just needs to not look completely invalid.
<lifeless> wgrant: so, appear valid :P
<wgrant> Pfft.
<thumper> stub: any idea how I can get 8.4 unfucked?
<thumper> I'm losing patience rapidly
<stub> killall postgres ?
<thumper> done
<thumper> but the --remove triggers a stop, which fails
<thumper> which stops the --remove
<stub> --remove?
<thumper> sorry
<thumper> sudo aptitude remove --purge postgresql-8.4
<thumper> that is what fails
<thumper> Removing postgresql-8.4 ...
<thumper>  * Stopping PostgreSQL 8.4 database server                                                                                                                                           * Error: Invalid data directory
<stub> Kill the processes, nuke /etc/postgresql, nuke /var/lib/postgresql
<stub> And file a bug if you can be arsed
<thumper> stub: do you think the uninstall will work then?
<stub> Should. There are no config files to know what to shutdown.
 * thumper thrashes the HDD
<thumper> stub: aptitude tells me that postgresql-8.3 is installed, but there is no /etc/postgresql
<thumper> I nuked it before
<stub> Sounds like aptitude didn't do some setup step. Or maybe its the 8.3 packages on lucid - I thought you where going to use 8.4?
<wgrant> Hmm. If you purged (not just removed) postgresql-8.3, postgresql-8.4 (and maybe postgresql-common), that directory should have been recreated on installation.
<thumper> if I try to remove postgresql-8.3 it removes all the launchpad developer deps
<stub> With synaptic, it would remove the launchpad-developer-deps packages sure. But not the rest of the deps.
<thumper> aptitude did, apt-get won't
<wgrant> stub: Regarding that EmailAddress issue, USSO was creating them until recently. Bug #523346. But I thought they fixed the data too.
<thumper> stub: so if I install the 8.4 packages for slony, contrib, and plpython I should be good?
<thumper> stub: 8.4 didn't add a /etc/postgresql directory either :(
<thumper> stub: before when you said nuke, I just moved them to /junk
<thumper> stub: could that matter?
<stub> You can run pg_createcluster yourself per the wiki docs, but something is screwed
<stub> I don't see how that could matter
<thumper> it is creating the 8.4 cluster now
<thumper> the launchpad database setup script needs to be run as root?
<stub> I expect so, unless it uses sudo internally
<thumper> it uses sudo internally
<thumper> just checked
<thumper> I think I'm getting somewhere
<thumper> making schema
<thumper> on 8.4
<thumper> ok
<thumper> work done for me
<thumper> I think I'm up to a working launchpad environment on lucid
<thumper> stub: thanks for your help
<thumper> and wgrant
<wgrant> Hopefully there's still only that one test failure. I might run the whole suite again tonight.
<StevenK> wgrant: Which test failure?
<wgrant> StevenK: test_generateConfig in Soyuz.
<wgrant> apt-ftparchive outputs extra hashes.
<wgrant> s/Soyuz/archivepublisher/
<StevenK> Ah. I just had test_mark_duplicate_form_overlay fail for me, thought it was related
<wgrant> There was a commit related to that this morning.
 * wgrant WTFs at builder.txt.
<wgrant> Ah.
<wgrant> I was wondering why the start of builder.txt was so shocking.
<wgrant> annotate shows that it hasn't been changed significantly since around r6000.
<wgrant> Er.
<wgrant> r600.
<bigjools> :D
 * wgrant throws a quick rewrite into another branch that touches that file.
<wgrant> bigjools: buxy himself visited earlier and asked that we enable 3.0 (quilt) in Karmic. I discussed this with some distro people a few weeks ago and they tended to agree.
<wgrant> The worst that will happen is that builds will fail if they use one of the couple of things that Karmic's dpkg doesn't support.
<bigjools> wgrant: sounds fine then
 * bigjools considers firewalling all IPs from China, Russia and South America
<wgrant> bigjools: Should I poke a LOSA with instructions, or ...?
<wgrant> bigjools: Good idea.
<bigjools> bastards DoSing me trying to crack my ssh
<wgrant> denyhosts FTW.
<bigjools> yarp
<wgrant> 235 jobs (30 hours)
<wgrant> WOW.
<wgrant> I am glad I haven't tried to use a PPA lately.
<bigjools> I auto-drop after 3 failed attempts but it still screws my connection
<bigjools> builders removed, it'll come back
<wgrant> Um, 30 hours is really quite ridiculous and calls for drastic action.
<bigjools> it's not ideal
<wgrant> Waiting more than a day for a build is well beyond not ideal.
<bigjools> wgrant: umm anyway yes if you prepare some SQL I'm sure we can get a losa to poke it in
<wgrant> bigjools: OK. Thanks.
 * wgrant works it out.
<bigjools> did anyone see https://bugs.launchpad.net/bugs/546017 ?
<mup> Bug #546017: can't add linux-meta-ti-omap to kernel/lucid package set <Soyuz:New> <https://launchpad.net/bugs/546017>
<wgrant> LOSAs: Can you please run this query on production to enable 3.0 (quilt) and 3.0 (native) in Karmic?
<wgrant> INSERT INTO sourcepackageformatselection (distroseries, format) SELECT id, 1 FROM distroseries WHERE name='karmic' UNION SELECT id, 2 FROM distroseries WHERE name='karmic';
<wgrant> bigjools: Yeah, that's an odd one.
<wgrant> Although I don't see that source.
<bigjools> wgrant: can you pastebin that query, I need to get permssion from da management
<mthaddon> wgrant: I'd need approval from someone (BjornT?) to run that on production
<wgrant> Ah, of course.
 * wgrant pastes.
<wgrant> http://pastebin.ubuntu.com/401051/
<bigjools> ta
<bigjools> I ran it on dogfood anyway
<BjornT> bigjools, wgrant: why is that query needed?
<bigjools> BjornT: I just updated LPS
<wgrant> BjornT: See scrollback starting from 25 minutes ago
<BjornT> wgrant: i prefer short summaries :)
<wgrant> Well, the relevant scrollback is two lines.
<wgrant> 20:09:24 < wgrant> bigjools: buxy himself visited earlier and asked that we enable 3.0 (quilt) in Karmic. I discussed this with some distro people a few weeks ago and they tended to agree.
<wgrant> 20:09:40 < wgrant> The worst that will happen is that builds will fail if they use one of the couple of things that Karmic's dpkg doesn't support.
<bigjools> BjornT: there's a short summary in LPS ...
<BjornT> wgrant: well, finding two lines in 25 minutes of scrollback isn't that easy, if you don't know which lines to look for
<BjornT> thanks bigjools
<wgrant> True.
<BjornT> approved
<wgrant> Thanks.
<bigjools> thanks BjornT
<mrevell> owdy
<deryck> Morning, all.
<wgrant> buildd-slavescanner.txt creates builder1-11, sticks them in buildergroup.builders, and then does nothing further with buildergroup.
<wgrant> Before my branch it does touch buildergroup, but only to call rescueBuilderIfLost, which doesn't touch that .builders at all.
<wgrant> So there's a whole lot of dead tests code there that looks potentially important.
<noodles775> wgrant: is it possible that the actual tests have been gradually moved out over time, leaving just the test setup there?
<wgrant> noodles775: I hope so.
<wgrant> It looks like it might now create other equivalent builders as it goes.
<wgrant> The functionality does seem to be tested elsewhere, so I guess the creations and assignments can go.
<noodles775> Great.
<wgrant> (I'm removing buildergroup, so needed to do something with them)
<gmb> stub: Still around?
<stub> yo
<gmb> stub: Ohai. Can you cast your weather eye over this query and tell us if it's completely mad?: http://pastebin.ubuntu.com/401091/
<gmb> stub: We want to use it to schedule the next_check of a bug watch based on the number of errors it has had over the past 5 attempts.
<stub> The coalesce(array( stuff seems needlessly confusing over 'count(*)'
<allenap> stub: I tried with a count but it didn't work after a couple of minutes and the array stuff did, so I didn't spend any more time on it.
<stub> or sum
<stub> But fine. It actually works?
<gmb> stub: Yes. In my head, I imbued that question with a tone of surprise.
<stub> Sticking a subquery in the FROM clause of an UPDATE that references the table being updated - can't recall having tried that
<stub> nm,.. not doing that
<stub> Your missing 'BugWatch.id = bug_watch.id' in the WHERE clause
<stub> sorry... BugWatch.id = counts.id
<gmb> Ah yes, thanks.
<stub> You don't *have* to do it in a single query though ;)
<gmb> stub: It made allenap happy so I was loathe to stop him.
<allenap> stub: Another version of the array stuff with count(*) instead: http://pastebin.ubuntu.com/401099/
 * stub wonders why next_check is ever null
<stub> Wouldn't it just default to NOW and be updated after every check, successful or not?
<allenap> stub: I don't completely understand. Do you mean that we should calculated at the same time the watch is checked?
<gmb> stub: next_check is set to NULL by checkwatches and then updated by a garbo process. We wanted to keep the scheduling logic out of checkwatches since it's complex enough already.
<stub> ok
<stub> Not sure if that is a win, but I'll believe you :)
<gmb> From our point of view, anything that makes checkwatches do less is a win :)
<stub> So how does that query work in the context of a DBLoopTuner loop? It is updating all of them at once.
<stub> That could be tens of thousands if things don't run for a few days.
<gmb> stub: We can run that query against a batch of BugWatch IDs.
<gmb> That would fit nicely with the LoopTuner way of doing things.
<stub> ok
<stub> If you do that though, you are still materializing the recent activity list for all bugwatches each time though, even though you are only updating a batch.
<stub> Maybe PG can optimize that, but I doubt it.
<stub> Oh... you put the list of ids in the subquery
<stub> that would work
<allenap> stub: I was just about to say that ;)
<bigjools> wgrant: karmic now accepts quilt
<wgrant> bigjools: Thanks.
<wgrant> $LOSA: Also thanks.
<bigjools> wgrant: I just thought, we've never pushed a source package branch through dogfood
<wgrant> bigjools: True.
<bigjools> can you think of anything that would prevent that now (apart from lack of a UI)
<wgrant> It should all work now.
<wgrant> As long as the buildd is OK.
<wgrant> All relevant work is long-merged.
<bigjools> cool - have you got a script I can steal to poke something in?
<wgrant> I have somewhere..
 * wgrant hunts.
<wgrant> bigjools: I used http://pastebin.ubuntu.com/401111/
<wgrant> It probably still works.
 * wgrant checks the DF configs for the right branch prefix.
<bigjools> ok
<wgrant> Hm, so plain lp:whatever should work fine.
<bigjools> ok I'll poke it through now
<wgrant> I suppose you can just use any old production package branch.
<wgrant> And hope that dogfood doesn't explode...
 * bigjools starts ripping chunks of hair out
<bigjools> AttributeError: 'NoneType' object has no attribute 'utf_8_decode'
<bigjools> ARGH
<wgrant> That reappears every couple of months.
<wgrant> Yay.
<wgrant> I always forget how to fix it by the next occurrence.
<bigjools> and every f***ing day on dogfood
<bigjools> make clean
<wgrant> Oh.
<bigjools> well it's only doing it on iharness  atm
<bigjools> I guess I'll have to put up with plain harness
<wgrant> Is it reasonably up to date, so the UI won't burn up once the build gets dispatched?
<bigjools> wgrant: http://pastebin.ubuntu.com/401120/
<bigjools> seen that?
<wgrant> bigjools: The branch you referred to probably doesn't exist on DF.
<wgrant> What was your recipe?
<bigjools> ok that makes sense, I blindly copied yours :)
<wgrant> lp:ubuntu/hello should work
<bigjools> yeah
<wgrant> bigjools: How old is the DF archive?
<bigjools> old
<wgrant> If it lacks a reasonably recent bzr-builder, I guess the target archive will need external dependency on production primary.
<bigjools> wgrant: ran the script, b-m not dispatching :/
<wgrant> bigjools: The builders are all disabled...
<bigjools> cobblers, why didn't I check that
<wgrant> You probably also need to twiddle scores, but we'll see.
<bigjools> which I really should have done before watching gcc get dispatched.
<bigjools> cock
<wgrant> Oops.
<wgrant> You can't abort it?
<bigjools> not easily
<bigjools> anyway, there's no sourcepackagerecipebuild
<bigjools> so something is borked
<wgrant> transaction.commit()?
 * bigjools weeps
<bigjools> that really shouldn't be required unless I started a txn >:(
<ricotz> bigjools, hello, are there plans to add more builders cause there is a huge package queue waiting to be built?
<bigjools> ricotz: the missing ones will return at some point, I don't know when
<wgrant> 33 hours and counting...
<ricotz> bigjools, ok, so this is out your hands then
<wgrant> bigjools: Is concordia the one that took 2 hours to do a 2 minute build?
<bigjools> yes, the machines that are missing are not exclusively PPA builders, they get loaned when not in use for other stuff
<bigjools> wgrant: dunno - but I started rubidium up and got a traceback when b-m tried to dispatch the sprb to it
<ricotz> ok, thanks
<bigjools> ricotz: we just have to be patient I'm afraid
<wgrant> bigjools: What was the TB?
<bigjools> DB permissions
<bigjools> 2010-03-25 12:16:42+0000 [-] psycopg2.ProgrammingError: permission denied for re
<bigjools> lation seriessourcepackagebranch
<wgrant> Ah. Easy fix.
<bigjools> yep
<wgrant> I thought I fixed that.
<wgrant> unless you don't regularly run security.py on upgrades.
<wgrant> Hm, maybe not.
<bigjools> I do run it every time
<wgrant> Yeah, i forgot about official source package branches when adding perms.
<wgrant> Oops.
<bigjools> so it's building now - I guess we need to fix: https://dogfood.launchpad.net/builders/rubidium
<bigjools> :)
<wgrant> I thought that was meant to work now.
<wgrant> What's the exception?
<wgrant> No canonical URL for the build?
<bigjools> ForbiddenAttribute: ('title', <lp.code.model.sourcepackagerecipebuild.SourcePackageRecipeBuild object at 0xec71dac>)<br />
<wgrant> Argh.
<wgrant> I wish we had these exposed on the API.
<wgrant> still building... slow...
<bigjools> how long should that take?
<wgrant> Heh. It depends, mainly on disk cache.
<wgrant> On my laptop the time-consuming part is extracting and upgrading the chroot, and installing the build-deps.
<wgrant> But that's because the disk is slow and RAM is eaten by the rest of LP.
<wgrant> It normally takes a couple of minutes here.
<bigjools> k
<wgrant> What's it say it's doing now?
<bigjools> Error accessing Librarian: HTTP Error 404: Not Found
<bigjools> wak wak oops
<wgrant> Fail.
<wgrant> No chroot?
<bigjools> prob the source expiry script
<wgrant> Why would it be getting sources?
<bigjools> nfi
<wgrant> the input to a recipe build is a chroot and recipe text.
<bigjools> you'd think
<wgrant> Hm, that LFA is still OK.
<wgrant> No idea which file it is?
<bigjools> nup
<bigjools> trying to get the log out
<wgrant> The only thing the behavior tries to cache is the chroot.
<wgrant> And the URL I get through the API still works.
<bigjools> it got as far as "Processing triggers for python-support"
<bigjools> no more log available
<wgrant> wut.
<wgrant> Is it still going?
<wgrant> It shouldn't be failing with a librarian error at that point...
<bigjools> seems so
<bigjools> the log must be OOD
<wgrant> OOD?
<bigjools> out of date
<wgrant> Yeah.
<bigjools> oh wait
<bigjools> my bad - I didn't clear the failnotes from last time
<wgrant> I was about to ask that.
<bigjools> builderok is True
<wgrant> So it's still going?
<wgrant> Great
<bigjools> but not doing anything
<wgrant> Can you SSH into the builder?
<bigjools> I think I'll poke another one in shortly on a non-virtual ppa so I can ssh into it
<bigjools> not into virtual ones. no
<wgrant> Or just flip the virtualized flag on the BQ.
<bigjools> yar
<bigjools> need to un-dispatch it though
<wgrant> Mark the builder not-OK.
<bigjools> grrrr
<bigjools> I really really hate the million damn predicates for dispatching
<wgrant> It's somehow excluded?
<bigjools> I'm making a new build, much much easier
<wgrant> I don't think we build SPRBs non-virt even for non-virt PPAs.
<bigjools> it'll get queued and then I can flip the bit before it's dispatched
<wgrant> True.
<bigjools> ffs
<bigjools> IntegrityError: duplicate key value violates unique constraint "sourcepackagerecipe__owner__distroseries__sourcepackagename__na"
<wgrant> Just change any character in the SPR name.
 * wgrant loves constraint name length limits.
<bigjools> 2010-03-25 12:55:36+0000 [-] exceptions.AttributeError: 'SourcePackageRecipeBuild' object has no attribute 'is_virtualized'
<bigjools> awesome
<wgrant> Wha? How did it work last time?
<bigjools> I'm dispatching to a non-virt builder now
<bigjools> that's the only difference
<wgrant> Heh.
<wgrant> My cursor was already over the right line. How odd.
<wgrant>         assert not (not self._builder.virtualized and build.is_virtualized), (
<bigjools> that's the badger
<bigjools> grrr
<wgrant> Yes.
<wgrant> Couldn't you have just turned one of the non-virt builders into a virtual one?
<bigjools> ferraz is building it now
<wgrant> Excellent.
<bigjools> buildrecipe is running
<wgrant> Ah, much better.
<wgrant> Targetting my PPA?
<bigjools> yes
<bigjools> it's still running and not doing anything :/
<bigjools> idle box
<wgrant> Ew.
<wgrant> What's the latest in the log?
<bigjools> and I can't get to the log >:(
<bigjools> rw only for the buildd user
<wgrant> Maybe it's checking out the branch?
<bigjools> 0.0%wa
<bigjools> logtail shows bzr running with the usual locale warning
<bigjools> and that's it
<wgrant> OK, so it's probably checking out.
<wgrant> Or being blocked by a firewall.
<bigjools> it's running bzr dailydeb
<wgrant> Can you telnet to bazaar.launchpad.net:80?
<wgrant> Aha.
<wgrant> Hmm.
<wgrant> it's not that big a package. But let's give it a while.
<bigjools> yes I can telnet there
<wgrant> :(
<bigjools> ummm the command line has:
<bigjools> /usr/bin/python /usr/bin/bzr dailydeb --no-build /home/buildd/work/recipe /home/buildd/work/tree --manifest /home/buildd/work/tree/manifest
<bigjools> but /home/buildd/work doesn't exist
<wgrant> I think that's fine.
<wgrant> buildrecipe constructs the path but never mkdirs it, and it has worked locally.
<bigjools> k
<bigjools> I need to wait for lamont then I can see what's going on
<wgrant> Can you check the buildd-manager log and grab the recipe text?
 * bigjools needs food in the meantime
<bigjools> <bigjools> and I can't get to the log >:(
<bigjools> <bigjools> rw only for the buildd user
<wgrant> on the master.
<bigjools> oh that
<bigjools> sorry
<bigjools> # bzr-builder format 0.2 deb-version 2.4-3+${rev
<bigjools> no}\nlp:ubuntu/hello\n
<wgrant> That's the text that was sent to the slave?
<wgrant> If so, try connecting to xmlrpc.launchpad.net on 443.
<bigjools> ha
<bigjools> ha ha ha
<wgrant> I was assuming it would change it to an http URL.
<wgrant> But I guess not.
<bigjools> :/
<wgrant> The connection is hanging?
<bigjools> yip
<wgrant> Excellent.
<wgrant> So it's not actually broken.
<wgrant> I'm glad we weren't trying to do this on Friday afternoon.
<bigjools> 2010-03-25 13:16:21+0000 [-] Scanning failed with: 'NoneType' object has no attribute 'utf_8_decode'
<bigjools> *weep*
<wgrant> Hahah.
<mars> wgrant, "I'm glad we weren't trying to do this on Friday afternoon" - you are a wise man :)
<bigjools> wgrant: are you doing a security.cfg fix?
<wgrant> bigjools: Since we've identified no other issues there, I might.
<wgrant> And I think that's the only other relevant table.
<bigjools> and the other UI issues need sorting
<wgrant> I forget -- can that go to devel?
<bigjools> no, db-devel
<bigjools> strictly it can, but the regex will kill the landing IIRC
<wgrant> Grmph.
<wgrant> OK.
 * wgrant looks at the UI issue.
<bigjools> I thought jtv had landed something to fix that actually
<wgrant> He fixed it for translations builds.
<bigjools> ah right
<wgrant> SPRBs are broken for other reasons.
<wgrant> SELECT on seriessourcepackagebranch for fiera is all that you added?
<bigjools> we need an interface
<bigjools> yep
<jtv> did I miss something
<jtv> ?
<wgrant> We have one. The view code just doesn't respect it.
<bigjools> jtv: nope - we were just testing recipe builds on dogfood and the ui blew up
 * jtv looks for quick exit
<jtv> something I did?
<bigjools> jtv: nope :)
<jtv> phew
<wgrant> bigjools: How quickly can you get a sysadmin to apply a crowbar to the firewall?
 * bigjools looks which direction the wind is blowing in
<wgrant> buildmailman.py REALLY doesn't like being run concurrently.
<wgrant> noodles775: Do we ave a title scheme for source package recipe builds yet?
<noodles775> wgrant: I'll send another email summarising, but the conversation the other day lead to (1) the build url will sit under the recipe's url, and (2) we need to list the builds in multiple places (ie. obviously under the recipe, but also for the PPA or Builder contexts).
<noodles775> Is (1) what you were after?
<wgrant> Not quite, but I might need that in a minute.
<wgrant> I'm wanting a string to use in the builder UI.
<wgrant> To identify the recipe build.
<wgrant> But I guess the recipe schema in the designs is different, so I'll just make something up for now.
<sinzui> jml: Can you give me your thoughts on my suggestion for ttps://bugs.edge.launchpad.net/bugs/538024
<mup> Bug #538024: No way of saying "This project is not packaged" <Launchpad Registry:Triaged> <https://launchpad.net/bugs/538024>
<mpt> "You are here because the Launchpad Edge environment uses the Launchpad Login Service service."
<noodles775> wgrant: ok... so it looks like there is no title yet on the SPRBuild model. Just add one... something that mentions the recipe name and the target archive and the.... hmm, base-branch short name? Not sure.
<wgrant> noodles775: I won't mention the target archive at this point, since that's handled by other parts of the UI in somewhat revolting ways.
<noodles775> wgrant: huh? isn't it a relevant part of what the SPRBuild represents?
<wgrant> noodles775: Yes, but Builds don't do it, and the builder-related pages include the archive regardless.
<noodles775> wgrant: I think the only reason builds don't do it is because they are always in their context (PPA or a distroseries), it is on the build detail page of course (https://edge.launchpad.net/ubuntu/+source/bash/4.1-2ubuntu1/+build/1580941)
<noodles775> s/don't do it/don't include it in the title/
<wgrant> noodles775: Those pages don't exist for SPRBs yet. If you look at https://launchpad.net/builders, you'll see that the archive name is included outside the link text (which is the title).
<wgrant> The same thing will occur for any BuildBase.
<wgrant> This is all going to have to be revised once the schema is finalised; this is just to make it stop OOPSing.
<noodles775> True.
<Ursinha> bigjools: hello
<Ursinha> bigjools: why did you remove the qa-needstesting tag from bug 541807?
<mup> Bug #541807: Archive should have a status column  <Soyuz:Fix Committed by julian-edwards> <https://launchpad.net/bugs/541807>
<Ursinha> I wonder if I have a weird internet problem that makes me visible only to the rosetta people
<Ursinha> danilos: hi
<danilos> Ursinha, I don't see you, sorry
 * wgrant neither.
<Ursinha> danilos: so you don't have internet problems, you're blind
<wgrant> bigjools: Merging lp:~wgrant/launchpad/fix-sprb-ui should unbreak the UI, making logtails far less invisible.
<bigjools> wgrant: https://dogfood.launchpad.net/builders/ferraz
<wgrant> bigjools: *that's* a bit more encouraging.
<bigjools> we'll see!
<wgrant> bzr is taking its time...
<wgrant> I wonder what its problem is now.
<wgrant> ARGH
<wgrant>     DEFAULT_INSTANCE = 'edge'
<wgrant> lp: uses xmlrpc.edge.launchpad.net by default.
<wgrant> Not xmlrpc.launchpad.net.
<wgrant> bigjools: ^^ :(
<bigjools> wgrant: fuck sake
<bigjools> whyyyyyyy
<wgrant>     # We use edge as the default because:
<wgrant>     # Beta users get redirected to it
<wgrant>     # All users can use it
<wgrant>     # There is a bug in the launchpad side where redirection causes an OOPS.
<wgrant> "Beta users get redirected to it" makes no sense, though, since bzr won't possess the cookie.
<bigjools> indeed
<bigjools> check this out: https://edge.launchpad.net/~ghostonline/+archive/personal/+packages
<bigjools> packages stuck at pending :/
<wgrant> Yay. What's the error?
<bigjools> can't find a mention of that ppa in the logs :/
<wgrant> ...
<bigjools> oooo
<bigjools> its publish flag is "f"
<bigjools> wtf
<wgrant> Haha.
<bigjools> how!
<bigjools> noodles775: do you know if anyone was disabling publishing on any archvies and might have got the wrong one?
<wgrant> I didn't think there was any code that actually set that.
<noodles775> bigjools: in production? Nope.
<wgrant> It was a couple of weeks ago.
<bigjools> we've got 115 archives where publish=false
<bigjools> 110 are ppas
<wgrant> What are the rest? Debian primary and a copy archive or four?
<bigjools> copy
<sinzui> barry: ping
<Ursinha> bigjools: hey
<bigjools> Ursinha: oi
<bigjools> tudo bem?
<Ursinha> bigjools: tudo bem, e vocÃª?
<bigjools> Ursinha: wet
<Ursinha> bigjools: danilos could join this conversation, he's even able to read brazilian newspapers
<bigjools> otherwise great :)
<Ursinha> :P
<Ursinha> bigjools: it's raining here as well
<danilos> Ursinha, bigjools: cuzao
<bigjools> Ursinha: let me guess, they're all complaining about your president?
<bigjools> danilos: rofl
<Ursinha> lol
<Ursinha> man
<Ursinha> hauhau
<Ursinha> so, bigjools, why are you removing qa-needstesting tags from your bugs?
<bigjools> wgrant: *cough* publish is set in the +edit page.... *cough*
<Ursinha> don't you like them
<bigjools> Ursinha: are you stalking me?
<bigjools> they're untestable bugs
<Ursinha> bigjools: you can mark them as qa-untestable then :)
<bigjools> refactoring and suchlike
<bigjools> DETAILS!
<bigjools> :)
<Ursinha> lol
<wgrant> bigjools: W...why?
<Ursinha> bigjools: I can do that for you
<bigjools> wgrant: the question is, why not
 * bigjools hugs Ursinha
<Ursinha> bigjools: done
<Ursinha> :)
<bigjools> Ursinha: how's your sprint?
<Ursinha> bigjools: sorry about the stalking, I just needed to know if you're removing because of a weird use case I haven't thought about
<bigjools> you guys need a dogfood lesson
<Ursinha> bigjools: it's great
<bigjools> ah I was only joking about the stalking :)
<Ursinha> bigjools: :)
<bigjools> is it hot there?
<bac> hi abentley
<abentley> bac, hi.
<abentley> bac, sure.
<bac> ok
<Ursinha> bigjools: it's always hot in here :)
 * wgrant sleeps.
<bigjools> night wgrant
<bac> abentley: i've got a branch A based on devel.  i've got another branch B that depends on A.
<bac> abentley: both A and B have merged from devel and have all recent changes
<abentley> You should not merge B from devel directly.  You should merge devel into A, and A into B.
<abentley> This is one of the things bzr-pipeline gives you.
<bac> abentley: if i run 'bzr diff -r ancestor:../A' while in B i see diffs that don't exist.  it shows a diff in the Makefile even though the Makefile is the same in A, B, and devel
<abentley> bac, you have a criss-cross merge, which means that the revision selected by -r ancestor: will be relatively old.
<bac> abentley: ok
<abentley> bac, it should go away if you merge A into B.  This will warn you that it is a criss-cross merge.  I suggest using merge --lca, because it is a criss-cross merge.
<bac> abentley: ok, i just merged again from A and now the phantom Makefile diff is gone
<bac> abentley: thanks
 * bac vows to start using pipelines
<abentley> bac, no problem.
<mars> matsubara, ping, do you have IE installed for some simple QA work?  I would like to do QA on staging *before* landing my changes.
<abentley> bac, of course, you can just manually make sure not to merge the same branch into A and a branch derived from A.
<bac> abentley: right.  i just had a temporary lapse.
<james_w> If class A has implements(ISomeInterface) then should ISomeInterface.providedBy(A) be True?
<matsubara> mars, nope. there's the windows EC2 machine though. I can use that. I can help you after the meeting.
<abentley> bac, ah, okay.
<james_w> .providedBy(instance of A) I mean
<sinzui> james_w: yes
<leonardr> anyone have any experience with the python memcached client?
<jml> sinzui, sure.
<jml> sinzui, thanks for poking me
<mars> matsubara, instructions are on the wiki for testing with the EC2 instance?
<matsubara> mars, I think so. let me find them
<matsubara> mars, https://wiki.canonical.com/Launchpad/LPAWSAccounts for the win AMI and password. you basically start it up using elasticfox or amazon console
<mars> matsubara, great, thanks
<matsubara> magcius, then $ ec2-authorize default -p 3389 -s your_ip_address/32
<matsubara> mars, and then $ rdesktop -u administrator <ec2-instance-public-dns>:3398
<matsubara> magcius, sorry, above message was for mars, not you
<mars> I wonder if the ec2 toolkit survived the Lucid upgrade?
<matsubara> now is a great time to find out :-)
<mars> matsubara, should the rdesktop port be 3389 or 3398?
<matsubara> mars, 3389
<mars> matsubara, thanks.  3389 works, guess the second command had the typo
<matsubara> mars, yep, sorry about that
<jml> *described*
<thumper> morning
<bigjools> hey thumper how's the ui work going for recipe builds?
<thumper> bigjools: you should ask rockstar and abentley, but they are pretty happy
<rockstar> bigjools, I've spent today wrangling branches to make sure we have all our db patches in this cycle.
<bigjools> I tried putting a recipe build through dogfood today and found quite a few bugs
<rockstar> bigjools, eep.  Well, we should surface all of those in the next few weeks.
<bigjools> yeah Michael is going to work more on the soyuz ui side
<bigjools> it seems as though the builders will need access to xmlrpc.edge.l.n as well
<james_w> sinzui: hi, thanks for the offer, my branch is at https://code.edge.launchpad.net/~james-w/launchpad/export-code-import/+merge/22175
 * sinzui looks
<sinzui> james_w: what kind of test can we write to verify this kind of mistake does not happen again. I agree you fixed these issues, but none of the other webservice tests changed. I think the problem is that the webservice/xx-distroseries.txt test reports that driver_link is correct, but we know that is not true
<james_w> it is true
<james_w> I am working on a fix for lazr.restful that will stop this from ever happening again
<james_w> but that's not going to be ready in the next 18 hours
<sinzui> understood.
<james_w> so it depends if you want to land the fixes with the tests that will show the issue
<sinzui> james_w: I understand  your point. and agree it is a separate issue. Lets land this now. I glad to know there is a comprehensive fix for this issue.
<james_w> thanks
<james_w> it's not pretty, but it throws a big error in your face if you make this mistake, meaning the current tests we do for the webservice will be sufficient and we won't start having to write tests that assert that the wadl contains the right thing for everything we export as well
<sinzui> fab
<james_w> maybe someone that knows zope better can come up with a more elegant solution
<sinzui> that would be gary_poster, flacoste, and leonardr.
 * gary_poster waves
<sinzui> james_w: do you need me to land this for you?
<james_w> and I can't work out how to write the test in lazr.restful as things don't seem to behave the same way there
<james_w> sinzui: yes please
<james_w> gary_poster: hi, I'm confused by all the adapters and indirection that is going in the two codebases.
<sinzui> tests in lazr are different since we do not assume launchpad exists. We build a lot of hypothetical objects and explicitly wire them. launchpad hides that a lot
<james_w> if you look at the merge proposal above you can see the problem is triggered by using a zope.schema.Choice where a lazr.restful.fields.ReferenceChoice is intended
<james_w> so I'm trying to do the same thing in lazr.restful
<james_w> to use a Choice I need to provide a vocabulary, the only one currently defined in lazr.restful is an EnumeratedType, but that seems to give different behaviour, as the test uses getMultiAdapter, and picks a different Marshaller to what we get when the bug occurs in LP
<wgrant> bigjools: Did the test build end up working?
<james_w> so I'm trying to understand how the getMultiAdapter call decides to pick an ObjectLookupFieldMarshaller in LP.
<bigjools> wgrant: nope, I gave up waiting for someone to fix the firewall
<wgrant> Yay.
<wgrant> But it seems to mostly work.
<bigjools> yeah I think it will be fine
<bigjools> I could change the recipe I suppose so it uses http
<wgrant> If you just hack getRecipeData to use getPublicURL instead, yeah.
<wgrant> You can't change the recipe data itself, since the branches are stored as references and URLs are regenerated.
<bigjools> oh :/
<wgrant> bigjools: In lib/lp/code/models/sourcepackagerecipedata.py, s/bzr_identity/composePublicURL()/
<james_w> aha! lib/canonical/launchpad/zcml/webservice.zcml defines an adapter for SQLObjectVocabularyBase
<wgrant> Hmmm. Are we have silent test failures again?
<wgrant> I have three successful ec2test emails, with around 3 failures each.
<wgrant> But they get ignored.
<wgrant> Ah, no, just: lib/canonical/launchpad/pagetests/standalone/xx-form-layout.txt, lib/canonical/launchpad/pagetests/standalone/xx-opstats.txt
<wgrant> Can somebody please re-land https://code.edge.launchpad.net/~wgrant/launchpad/rescueiflost-and-updatestatus-to-model/+merge/22016? It got hit by a spurious failure a few hours ago.
<bigjools> did pqm reject it?
<wgrant> No, it died from the buildd-slave.txt ProtocolError that sometimes appears.
<bigjools> oh in ec2
<wgrant> Yeah.
<wgrant> I've just confirmed that it still passes locally.
<wgrant> bigjools: Also, those two DF fixes from last night landed.
<bigjools> wgrant: coolio
<wgrant> bigjools: Since the only unique failure appears to be spurious, can you please land rescueiflost-and-updatestatus-to-model?
<wgrant> The non-unique failures are more concerning :/
<bigjools> wgrant: it's best if you grab someone else, I am about to go to bed and if it fails again my name will be mud :)
<bigjools> maybe a friendly antipodean can help
<wgrant> bigjools: OK, thanks.
<bigjools> mwhudson, while not an antipodean, is very helpful :)
<mwhudson> mm what's this?  shoving things into ec2 land?
<bigjools> yarp
<wgrant> mwhudson: Well, I have a branch that had two real failures and one spurious failure.
<bigjools> can you land something for wgrant please
<wgrant> The two real failures are in trunk, so they don't matter.
<wgrant> (for that branch, at least -- in wider matters they are rather concerning indeed)
<mwhudson> https://login.ubuntu.com/+openid -> http 500
<mwhudson> yay!!
<mwhudson> and now it works
<mwhudson> wgrant: are you trying to convince me to just pqm-submit it?
<wgrant> mwhudson: I might as well try.
<mwhudson> wgrant: i would rather not, tbh
<wgrant> OK.
<wgrant> Can you EC2 it, then?
<mwhudson> sure
<wgrant> Thanks.
<bigjools> right, glass of single malt and then bedtime, g'night folks
<mwhudson> wgrant: https://code.edge.launchpad.net/~wgrant/launchpad/rescueiflost-and-updatestatus-to-model
<wgrant> Night bigjools.
<mwhudson> bigjools: g'night
<wgrant> mwhudson: That's the one.
<wgrant> mwhudson: Have a look at http://paste.ubuntu.com/401411/. It's a successful test from last night, but with failures.
<wgrant> All the other runs had the same issues, but also ignored them.
<mwhudson> wgrant: that's rather odd
<wgrant> Yes. It's not a feature I really look for in my test infrastructure, personally.
<mwhudson> does it happen locally?
<wgrant> Trying.
<mwhudson> i wonder if there's a way to stop thunderbird 3 tying up a cpu indexing all my mail
<mwhudson> ah, yes
<mwhudson> wgrant: the test fails locally, but also fails the test run
<mwhudson> which is nice
<wgrant> I recall a similar issue a few months ago.
<wgrant> But with a specific type of test case.
<mwhudson> there was a problem with twisted tests
<wgrant> That was it, yeah.
<mwhudson> i think that was somewhat different
<wgrant> :(
<mwhudson> yes
<mwhudson> oh
<mwhudson> hm
<mwhudson> errr
 * mwhudson swivels to point at zope.testing
<mwhudson> wgrant: i am extremely suspicious about the Exception in thread Thread-9:/NameError: global name 'suberr' is not defined traceback
<wgrant> mwhudson: Ah, yes, true.
<mwhudson> the rage is building
<mwhudson> we last changed our zope.testing version in 2009-08-12 though
<mwhudson> oh right, i think it's because the subprocess is dying in a particular way
<mwhudson> zope.testing-3.8.1-py2.5.egg/zope/testing/testrunner/runner.py contains a hilarious number of potential name errors :(
<wgrant> Heh.
<wgrant> So jml's new zope.testing hasn't landed yet?
<mwhudson> no
<mwhudson> though i'm not sure what the blocker is now
<mwhudson> hardy packages of subunit maybe?
<mwhudson>                 if e.errno == errno.EINTR:
<mwhudson>                     # If the subprocess dies before we finish reading its
<mwhudson>                     # output, a SIGCHLD signal can interrupt the reading.
<mwhudson>                     # The correct thing to to in that case is to retry.
<mwhudson>                     continue
<wgrant> Probably buildbot.
<mwhudson> YES BUT ONLY IF YOU'VE IMPORTED THE errno MODULE
<wgrant> Hahah.
<mwhudson> otherwise this won't work so well
<mwhudson> wgrant: the good news is that the newer zope.testing has these bugs fixed
<wgrant> Aha.
<wgrant> Interesting timing.
<mwhudson> it's another thing that will make it harder to land jml's branch
<wgrant> Indeed.
<mwhudson> rage rage rage
<mwhudson> wgrant: ./bin/test -vvct test_dir_construction_and_trivial_running goes bang rather stunningly
<mwhudson> and rather slowly too: it takes about a minute
 * wgrant is waiting...
<wgrant> The missing DB issue?
<mwhudson> yes
<mwhudson> of course the last change to the file containing that test was the addition of the AGPL copyright line
<wgrant> YAY
<mwhudson> trying to run it in production-devel gets the
<mwhudson> exceptions.ImportError: No module named psycopg2
<mwhudson> thing that plagued buildbot for a while
<wgrant> Hmm, I regret using allcaps before, since I can't make it an any bigger 'yay' now.
<jpds> wgrant: Use the figlet.
 * mwhudson does binary chop on buildbot logs
<mwhudson> actually it's rather easy to find
<wgrant> Indeed.
<mwhudson> because it starts where the number of tests reported as being run drops by 2000
<mwhudson> er
<mwhudson> 7000
<mwhudson> i wonder how many of these 7000 tests we're no longer running fail now!
<wgrant> Oh, it's that many!?
<wgrant> Ouch.
<mwhudson> build 716 -> test 26193 tests 26193 passed
<mwhudson> build 718 -> test 19516 tests 19516 passed
<mwhudson> (buildbot vomited on 717)
<wgrant> .... ouch.
<wgrant> That wasn't long ago, was it?
<mwhudson> so between 10574 and 10580
<mwhudson> no, not very long ago
<wgrant> Phew.
<mwhudson> yeah
<mwhudson> i'm suprised noone's noticed the test suite going faster
<mwhudson> http://pastebin.ubuntu.com/401425/
<wgrant> Yeah, I just looked. Nothing there looks suspicious, except for maybe the launchpadlib thing.
<mwhudson> it's hard to see which of these commits could have done it
<mwhudson> so next thought: check the ami versions
<wgrant> Indeed. Didn't jml push his new image a day or two ago?
<mwhudson> 716: ami-296f8340
<mwhudson> 718: ami-296f8340
<mwhudson> not that then
 * wgrant reverts 10575.
 * mwhudson reverts back to 10574
<lifeless> 7K missing tests?
<wgrant> "optimisation"
<wgrant> Reverting 10575 fixes the missing DB error.
<wgrant> WTF
<mwhudson> there's some quote in the company quotes page from jamesh about making the test suite faster by deleting all the tests
<mwhudson> this is just the sneakier version
<mwhudson> wgrant: !!
 * wgrant goes hunk by hunk.
<wgrant> Yeah, it's the least unobvious change.
<wgrant> The Launchpad.login call.
<mwhudson> wgrant: the test is faster when it passes than when it fails
<wgrant> Much, yes.
<mwhudson> wgrant: uhhhhhhhhh
<wgrant> (and no, lifeless, the word is not 'obvious' :P)
<wgrant> mwhudson: Uh?
<mwhudson> wgrant: just thinking about what might be going on here
<wgrant> LEONARD!!!!
<wgrant> It is an odd one.
<wgrant> Although I guess that probably makes requests for WADL and stuff.
<mwhudson> yeah i guess so
<wgrant> And is going to be horrifyingly slow.
<mwhudson> if that's being done for each page test...
<wgrant> I really hope not.
<wgrant> It takes a while to even import.
<wgrant> Ah, not too bad with a hot cache.
<mwhudson> i guess it's the recursive call to suite.run() in test_dir_construction_and_trivial_running that's special about that test
<wgrant> So basically we now have to look at all of launchpadlib to work out what the problem is.
<mwhudson> i dunno
<wgrant> And lazr.restfulclient and wadllib.
<mwhudson> the page tests that test_dir_construction_and_trivial_running runs are exceedingly trivial: empty files
<mwhudson> it's possible that if they made any requests at all, we'd have this effect already
<wgrant> Does that not set up a DB, maybe?
<wgrant> The opstats failures make perfect sense, but this one does not.
<mwhudson> i don't really know
<wgrant> Oh.
<wgrant> PageTestLayer doesn't inherit DatabaseLayer at all.
<mwhudson> hm?
<mwhudson> running PageTestLayer tests certainly sets up the database
<wgrant> Oh, yes it does.
<wgrant> Confusing inheritance hierarchy is too complex.
<mwhudson> wgrant: hah yes
<mwhudson> wgrant: if i change the page tests that test_dir_construction_and_trivial_running runs to actually make a request
<mwhudson> everything goes bang
<wgrant> Given how close we are to freeze, do we want to just revert the two problematic hunks (the Launchpad.login, and the single test that uses launchpadlib)?
<wgrant> Aha.
<wgrant> So somehow the the DB is getting deconstructed. Yay.
<mwhudson> um
<mwhudson> i'm not sure what the best course is
<mwhudson> having a launchpad object in the test globs that's authenticated as a sample data admin is bonkers anyway
<mwhudson> for so many reasons
<mwhudson> wgrant: http://pastebin.ubuntu.com/401435/ ?
<wgrant> mwhudson: Yeah, that.
<mwhudson> wgrant: do you know what the failures we've seen in the log actually are?
<mwhudson> the regular looking page test failures
<wgrant> mwhudson: The opstats one is because of the queries that Launchpad.login makes. I haven't looked at the other one.
 * wgrant runs with Launchpad.login removed.
<wgrant> Still fails. Probably legit.
<wgrant> Ah, field description change.
<mwhudson> well that was nice
<mwhudson> my keyboard stopped working!
<mwhudson> oh
<mwhudson> it was probably seahorse asking for my passphrase :(
<wgrant> mwhudson: The pagetest failure is a team description change yesterday.
 * wgrant fixes.
<mwhudson> wgrant: ah, cool
<mwhudson> i wonder how many other test failures this fix will shake out of the woodwork
<wgrant> It's only been a couple of days.
 * mwhudson is tempted to pqm-submit it so everyone gets to suffer
<mwhudson> yeah true
<mwhudson> wgrant: want to push/pastebin your fix and i'll merge it?
<mwhudson> thumper: want to review a branch?
<wgrant> mwhudson: Just rerunning the test now.
<mwhudson> cool
<wgrant> http://pastebin.ubuntu.com/401444/
<wgrant> So that's all known failures fixed.
<mwhudson> yeah
 * mwhudson shoves it into ec2, starts writing email
<wgrant> Great.
<mwhudson> ah balls, forgot --headless on the ec2 test
<wgrant> Oops.
<mwhudson> well that wasn't what i said i would be doing in the standup
<wgrant> Heh.
<wgrant> Thank goodness for spurious failure, or that would have gone unnoticed for a while longer.
<mwhudson> yeah
<thumper> mwhudson: sure
<wgrant> mwhudson: Regarding point 3), it basically calls DatabaseLayer.testTearDown twice in succession.
<wgrant> No idea why.
<wgrant> Hmm.
<wgrant> I guess it tears down in the sub-suite run, then at the end of the test itself.
<mwhudson> wgrant: but why doesn't that always fall over in a heap?
<wgrant> Doesn't it do some optimisation about not tearing down the DB if it's not dirty?
<mwhudson> maaaybe
<mwhudson> https://code.edge.launchpad.net/~mwhudson/launchpad/aiee-everything-is-broken/+merge/22185
<mwhudson> thumper: ^
<thumper> mwhudson: is your branch one that reverts the launchpad change?
<mwhudson> thumper: it's a little more delicate than that
<thumper> mwhudson: I just read your email thread
<wgrant>         if (ConnectionWrapper.committed and ConnectionWrapper.dirty):
<wgrant>             PgTestSetup._reset_db = True
<wgrant>         ConnectionWrapper.committed = False
<wgrant>         ConnectionWrapper.dirty = False
<wgrant>         if PgTestSetup._reset_db:
<wgrant>             self.dropDb()
<wgrant>             PgTestSetup._reset_db = True
<wgrant> Looks like if you never commit, it probably won't drop the DB.
<mwhudson> though i think my diff still says "XXX outraged XXX goes here"
<thumper> # XXX Outraged XXX goes here.
<thumper> yeah
#launchpad-dev 2010-03-26
<mwhudson> i guess i should file a bug
<thumper> please
<wgrant> That test probably also deserves a bug.
<wgrant> Although it will be less important with the new zope.testing, I guess.
<mwhudson> wgrant: yeah, i guess so
<mwhudson> i'm surprised and a bit scared if "browser.open('http://launchpad/dev')" commits to the db though
<thumper> mwhudson: bonus points for making a glob function that provides a launchpadlib object
<thumper> mwhudson: and if there is no user passed in, make one
<mwhudson> thumper: that sounds a little bit difficult because you have to create an oauth token
<thumper> mwhudson: there is a function for that
 * thumper looks for it
<wgrant> mwhudson: It will at least commit to update the session time.
<wgrant> Although I guess that's on a different DB.
<wgrant> And maybe it doesn't go through the real session machinery anyway.
<mwhudson> wgrant: if it's that sort of thing then i'm going to be even more angry i guess
<thumper> mwhudson:  webservice_for_person has most of what you need
 * thumper remembers writing that
<mwhudson> ah
<mwhudson> a not-surprising coincidence
<wgrant> mwhudson: What was the diff you used to get the pagetests to fail?
<mwhudson> wgrant: i think i pasted the diff didn't it?
<mwhudson> i'd have to read my logs to find it now...
<wgrant> Oh, just a browser.open?
<mwhudson> yep
<thumper> oh FFS
<thumper> make fails
<thumper> ImportError: No module named collection
<thumper> I have updated everything and did a make clean
<thumper> any ideas?
<wgrant> collection isn't in 2.5!
<mwhudson> wgrant: yes it is?
<wgrant> Who's trying to import it?
<thumper> /lib/lp/bugs/configure.zcml", line 935.4
<wgrant> Oh, right, collections.
<mwhudson> oh
<mwhudson> "collection" sounds wrong
<wgrant> mwhudson: So, yeah, that browser.open() commits a couple of times across each database.
<mwhudson> wgrant: AWESOME
<wgrant> So that is indeed the problem.
<wgrant> If that test commits at all, the second testTearDown attempts to destroy the non-existent DB.
<mwhudson> thumper:    <class class="lp.bugs.model.bugheat.CalculateBugHeatJob"> ?
<thumper> i'm looking
<mwhudson> wgrant: do you want to follow up to my mail saying as much
<thumper> can't find the import yet
<wgrant> mwhudson: Will do.
<thumper> hahah
<thumper> pebkac
<thumper> 'twas me
<thumper> from collection import defaultdict
<mwhudson> ah good
<wgrant> (I think I was probably thinking of collections.defaultdict, which wasn't in *2.4*.)
<thumper> wgrant: correct
<thumper> mwhudson: want to review a branch that renames thumper -> dumper?
<thumper> mwhudson: I'm sick of not having make-lp-user work for me
<mwhudson> thumper: :-)
<mwhudson> thumper: ok
 * thumper waits for the scanner
 * mwhudson lunches
<james_w> ok, there is now https://code.edge.launchpad.net/~james-w/launchpad/new-code-import-method-on-branchtarget/+merge/22190
<james_w> and https://code.edge.launchpad.net/~james-w/launchpad/code-imports-for-source-packages/+merge/22191
<thumper> mwhudson: can you go 'make run_codehosting' and push a branch to lp://dev/ using devel?
<thumper> mwhudson: I can't
<thumper> bzr: ERROR: Connection error: while sending POST /bazaar/: [Errno 111] Connection refused
<mwhudson> thumper: err, usually yes
<mwhudson> i usually use the key in the sample data, so i may not have done it in a while
<mwhudson> thumper: is the app server actually running?
<thumper> mwhudson: as far as I can tell
<thumper> ah, I bet I need to make install and apache isn't working
 * thumper looks a bit
<thumper> I've not tried it in lucid yet
<mwhudson> that might do it, i guess
<wgrant> How's the EC2 run looking?
<thumper> yep, sudo make install fixed it
<mwhudson> wgrant: lib/lp/registry/tests/../stories/product/xx-product-add.txt failed
<mwhudson> that's it so far though
<wgrant> OK.
<thumper> ah fuck
<thumper> I thought I had broken something
<thumper> but it is just the make-dummy-branches
<thumper> if you have pushed code, and restart codehosting, it blows away your pushed branches
<thumper> but of a fubar there
<wgrant> thumper: Oh, I thought that was a feature.
<thumper> wgrant: nope
<wgrant> It pisses me off immensely when testing recipe builds.
<spm> argh. ok. who broke pygpgme. <== grumpy losa venting and hence ignorable
<wgrant> spm: Lucid?
<spm> looks like a branch upgrade actually
<mwhudson> spm: or jelmer
 * spm hunts for the bug we raised on jelmer hisself ;-)
<thumper> mwhudson: do you remember what to import in scripts to get around the circurlar import madness?
<mwhudson> thumper: uh, i think there's more than once source of the madness
<thumper> it is mildly insane, but import canonical.launchpad.interfaces before anything else makes it run :(
<thumper> is there a way I can just reapply database/schema/security.cfg without make schema?
<wgrant> database/schema/security.py
<wgrant> Will apply to launchpad_dev by default.
<thumper> ta
 * mwhudson gives some consideration to warsaw's law
<spm> :-)
<wgrant> The second law?
<mwhudson> the one about fridays
<wgrant> Yeah, the second.
<mwhudson> yes, then
 * thumper is having a WTF moment
 * thumper is confused
 * thumper uses pdb
<thumper> huh?!?!?
 * thumper can only guess at stale pyc files
<thumper> doesn't make clean clear pyc files?
<thumper> ARGH!!!!
<thumper> PEBKAC again!
<thumper> it must be friday
 * sidnei hands some band-aid over for thumper's forehead
<thumper> must be time for a cocktail
<thumper> why isn't the email being delivered I wonder...
<thumper> ah crap
<spm> thumper: time for a cocktail? it's after 9am over there eh?
<thumper> somewhere there is a setting saying that scripts don't send email
<thumper> spm: it has been one of those days (or weeks) where things take longer than expected, and you don't get as much done as you expect
<spm> thumper: ahhh! so you've become a losa eh? sweet!
<wgrant> thumper: Zopeless emails do not go anywhere in devmode. config.zopeless.send_email
<thumper> wgrant: so how am I supposed to test this damn script eh?
<thumper> arse
<thumper> also, why don't they go anywhere?
<wgrant> thumper: Because there's no mail utility, so it uses raw SMTP.
<thumper> they should go the same place that the app server emails go
<wgrant> Which is very dangerous, since it can send email to ANYWHERE IN THE WORLD.
<thumper> wgrant: there is a mail utility in the scripts
<thumper> has been for ages
<wgrant> Although it could just override the destination address to root@localhost anyway, i guess.
<wgrant> Not the full Zope queued delivery mail utility.
<thumper> wgrant: yes, the same as the app server
<thumper> well, that is a problem for another day
 * thumper tries something
<thumper> wgrant: http://pastebin.ubuntu.com/401551/ works
<thumper> wgrant: sends email to root@localhost
<thumper> wgrant: please excuse the commenting out :)
<thumper> wgrant: actually this is easier to read http://pastebin.ubuntu.com/401552/
<wgrant> thumper: So raw_sendmail obeys the destination override?
<thumper> yes
 * thumper commits that into his branch
<wgrant> Convenient.
<thumper> bzr commit -m "Zopeless is no longer really zopeless, and to test the local email sending, we want this.  Scripts now send email in the dev environment to root@localhost just like the app server" lib/lp/services/mail/sendmail.py
<thumper> that's what it says
<mwhudson> wgrant: yay that was the only failure
<wgrant> mwhudson: Excellent!
<thumper> mwhudson, wgrant: how long does canonical.testing.layers.ZopelessAppServerLayer take to start up for you?
<thumper> it seems to be taking ages for me
<thumper> perhaps blocking completely
<thumper> bin/test.py -vvt TestCodeHandler
<wgrant> Took 12 seconds here.
<thumper> hmm. not starting here
<mwhudson> ~same here
<thumper> I noticed that my make run didn't kie
<thumper> die
<thumper> killed it
<thumper> trying again
<thumper> ha ha 11 seconds
<thumper> going now
<thumper> thanks anyway
<mwhudson> thumper: still here?
<thumper> mwhudson: just
<thumper> whazzup?
<mwhudson> thumper: https://code.edge.launchpad.net/~mwhudson/launchpad/aiee-everything-is-broken/+merge/22185 will update in a couple of ticks, can you take a look?
<thumper> yep
 * thumper waits for the diff
<mwhudson> grr still not updated
<lifeless> perhaps its broken
<lifeless> :P
<mwhudson> thumper: http://pastebin.ubuntu.com/401568/
<mwhudson> or someone's prosed merge mysql 5 into mysql 6 or something
<thumper> mwhudson: done
<thumper> mwhudson: I'm heading out now
 * thumper EODs
<mwhudson> thumper: thanks
 * mwhudson shoves the branch in the direction of ec2
<NCommander> WHat's the correct way to get the db username and password when working with Storm to access LP's database?
 * NCommander wants to start work on the converter
<wgrant> NCommander: You don't do that manually. You should probably use the normal script infrastructure which sets Zope and Storm up for you.
<NCommander> wgrant: oh, which script is good for an example?
<wgrant> NCommander: utilities/soyuz-sampledata-setup.py is one I know well.
<wgrant>     execute_zcml_for_scripts()
<wgrant>     txn = initZopeless(dbuser='launchpad')
<wgrant> Those are the critical lines.
<wgrant> You can then use the IStoreSelector utility to get your store.
<NCommander> wgrant: cool, thanks. I'm probably going to have a ton more questions in the future
<wgrant> s/questions/sleep/
<maxb> eww. A javascript alert box on ajax-changing ubuntu bug status?!
<wgrant> Yes :(
<wgrant> For anybody who is not a bug supervisor or owner.
<wgrant> It is crack.
<wgrant> Both because of the rules and because it's AN ALERT()!
<maxb> eww
<maxb> That needs an opt-out
<maxb> Or I shall have to learn greasemonkey
<wgrant> It does.
 * maxb wonders who signed off a UI review on an alert() :-)
<wgrant> The excuse was that there wasn't time.
<wgrant> The excuse for that was also that there wasn't enough time.
<wgrant> noodles775: Is there any timeline for the work to extract application information from PPA builds?
<noodles775> wgrant: I'm not sure what you mean by "extract application information"?
<wgrant> noodles775: soyuz is meant to eventually extract .desktop files and icons, and publish them for Software Centre to use.
<wgrant> It seems like it would make the PPA binary listing a lot less problematic.
<noodles775> wgrant: yeah, totally (long-term goal). No, I'm not certain what the timeline is for doing that, but I expect we'll find out at UDS.
<BjornT> gmb, allenap: if you have some time, i think it would be a good exercise rewriting TestBugWatchScheduler not do do any commits. also, why is LaunchpadFunctionalLayer used?
<stub> wgrant, NCommander: IStore(Account), IMasterStore(Account) or ISlaveStore(Account) is the preferred spelling now over using IStoreSelector directly. This will give you the store where the Account objects are stored, and will keep working if things get moved around (probably YAGNI, but the spelling is nicer anyway).
<wgrant> stub: I keep forgetting about that, since it's used in so few places.
<wgrant> But it is much nicer.
<gmb> BjornT: Sure, we'll do that if we get time; LPFL is a copy-paste error, so we'll fix that too.
<stub> So launchpad-dependencies no longer installs for me. It seems to want python2.5-apt and I can't find that. Any clues?
<BjornT> gmb: cool. another question. in what way does this fit into the "Database garbage collector"? it doesn't seem to collect any garbage?
<wgrant> stub: the PPA needs updating.
<wgrant> stub: Force the installation of python-apt IFORGETTHISBITubuntu2launchpad1
<gmb> BjornT: It doesn't. However, basic advice in the past has been to include things like this (and BugHeatUpdater, which doesn't garbage collect either) as part of garbo because we get the benefits of some built in testing (with test_garbo.py). Also, managing DB permissions is slightly less troublesome.
<BjornT> gmb: it sounds a bit dangerous, unless we redefine what garbo is supposed to do. in the past garbo has been broken, and not been running for a while. this is acceptable for garbage collection, but is it acceptable for bug watch updating?
<gmb> BjornT: Hmm. Good point. No, it isn't; if it breaks then bug watches stop getting updated.
<gmb> BjornT: I'll prepare a branch to put the scheduler in a separate cronscript.
<stub> I thought I fixed garbo so if one task fails it keeps going?
<gmb> Oh.
<stub> Even has timeouts so if one task takes too long it is aborted and the others allowed to continue.
<mrevell> Morning
<gmb> stub, BjornT: Ah. So in that case I'd say it *is* an acceptible place to put it, though it doesn't fit under the garbage-collecting umbrella.
<BjornT> stub: could be, i don't know. it still seems a bit dangerous to do a bunch of important things in one script. especially from a scalability point of view.
<stub> Its more than garbage.
<stub> The goal with garbo was to stop the proliferation of all those little cronscripts.
<stub> I personally would rather see scalability type fixes made if/when they become a problem than worry about the deployment issues
<BjornT> stub: i wonder, could this somehow be combined with the job system? they seem quite similar.
<stub> (more cron emails to monitor, more crontab entries, more script monitoring, more database users + permissions)
<stub> BjornT: You could replace garbo-daily.py and garbo-hourly.py with garbo.py which runs regularly, pulling scheduled jobs.
<allenap> stub, BjornT: But then we'd need a garbo job to put jobs into the job system to be run.
<stub> At the moment, that is probably overkill. We also don't have to deal with long running nightly jobs blocking hourly jobs.
<stub> allenap: And some turtles for it to stand on
<allenap> :)
<BjornT> stub: ok. we'll see what we'll do. you've just volunteered to document what garbo does and is supposed to be used for, since you might be the only one that knows enough about it :) (more on that later)
<stub> The other gain we get with garbo is jobs are serialized. We end up with unnecessary load spikes whenever hourly or nightly jobs fire off at the same time - with -hourly and -daily we get a very primative scheduler, which so far has been 'good enough'
<stub> BjornT: Stick it on my kanban board if you want :)
<wgrant> bigjools: Did that firewall fix end up happening?
<bigjools> wgrant: yes, gonna test it RSN
<wgrant> bigjools: Excellent.
<wgrant> We might finally get it done... 2 months after the plan.
<bigjools> well the UI work is still in progress
<bigjools> that's the last piece of the puzzle
<gmb> Does anyone know where mwh got to with his branch to combat the test-suite horrorshow?
<wgrant> gmb: The one where 7000 tests dropped off the face of the earth?
<wgrant> It was shoved into EC2 around 5 hours ago.
<wgrant> With the known failures fixed.
<wgrant> Er, nearly 6 hours ago, now.
<gmb> wgrant: Ah. Just long enough for me to start to worry that it's failed...
<wgrant> Yes...
<wgrant> Although it had just one failure when it was run a couple of hours earlier, and that failure is fixed.
<gmb> Hrm.
<maxb> <stub> So launchpad-dependencies no longer installs for me. It seems to want python2.5-apt and I can't find that. Any clues?
<maxb> I uploaded the new source 10 hours ago, the amd64 build is still pending, "Start in 8 hours"
<maxb> A buildd admin could make it jump the queue O:-)
<wgrant> maxb: Better than the 30 hour queue yesterday..
<maxb> I wonder if it's worth someone liasing with the owners of the hugest daily build PPAs to get them to skip a few days when the buildfarm is borrowed
<wgrant> The problem should go away once builder pools appear.
<wgrant> Which is probably RSN.
<maxb> how is that going to help the current problem, which is just a shortage of builders across all architectures?
<wgrant> maxb: Not architecture pooling Daily builds will be restricted to a set of dedicated builders.
<maxb> ah
<bigjools> I'm working on designing builder pools right now :)
<bigjools> will put it out for comment in due course
<bigjools> wgrant: 2010-03-26 10:34:38 DEBUG   hello: (source) NEW
<bigjools> \o/
<wgrant> bigjools: Nice!
<bigjools> wgrant: one bug is that the builds were created with a score of 0
<wgrant> bigjools: 'Builds' being the SPRBs?
<bigjools> buildqueue to be more specific
<bigjools> but yes
<wgrant> Right, scoring is not yet implemented for SPRBs
<bigjools> https://dogfood.launchpad.net/builders/ferraz
<wgrant> Also, the version on that recipe is screwed. I should really fix that.
<bigjools> it's even building the source now
<wgrant> Excellent.
<wgrant> So it does actually work.
 * bigjools does a little dance
<wgrant> If you fix the recipe, the binary builds should work, too.
<wgrant> (I threw a $ in front of the variable in the version, where it didn't want one)
<wgrant> bigjools: Wouldn't it be nice if we had complete integration tests for all build types?
<bigjools> wgrant: there's a shedload of nice things that could be done :/
<wgrant> Indeed, indeed.
<gmb> How would I go about switching DB user when I'm in the DatabaseFunctionalLayer? switchDbUser is on LaunchpadZopelessLayer and refers speficially to Zopeless stuff.
<gmb> Ignore previous question. I have since hit myself with the clue-bat.
<Ursinha> bigjools: hey, are you still getting emails?
<bigjools> Ursinha: no
<bigjools> does that make it your fault? :)
<Ursinha> oh ffs
<Ursinha> bigjools: yeah, it seems :/ weird thing is that the script seems to be doing what it should, I'll have to dig deeper
<bigjools> Ursinha: also, how can we make that script not do anything to the bug if I so desire?
<bigjools> (as per my email to you earlier)
<Ursinha> bigjools: I haven't seen it yet
<Ursinha> a moment
<Ursinha> bigjools: did you send it directly to me or through some list?
<bigjools> to you
<bigjools> Ursinha: subject "Fwd: [Bug 392887] Re: Cannot delete a PPA"
<mup> Bug #392887: Cannot delete a PPA <feature> <oem-services> <ppa> <Soyuz:In Progress by cody-somerville> <https://launchpad.net/bugs/392887>
<bigjools> sod off mup
<Ursinha> hm
<Ursinha> lol
<Ursinha> bigjools: checking
<Ursinha> slow... internet......zzzzzzzz......
<Ursinha> bigjools: what do you mean? what exactly you don't want to happen with the bug? to be targetted? assigned? marked fix commit?
<bigjools> Ursinha: no action at all
<bigjools> in this case it marked it fixed, but there's still loads of outstanding branches
<Ursinha> bigjools: so the problem would be the mark as fix committed/tagging
<Ursinha> ?
<bigjools> Ursinha: yes, but preferably I want it to do nothing at all
<Ursinha> bigjools: we need to keep track of branches being landed to fix that
<bigjools> targeting and assigning are potentially wrong as well
<Ursinha> so at least a comment pointing that would be nice to have
<bigjools> yeah you could do that
<Ursinha> bigjools: problem is that fits most use cases
<bigjools> but the branch linked on the page shows that too, LP could do that itself
<Ursinha> people forget do assign/target their bugs
<Ursinha> and that ruins how we use the tagging schema
<Ursinha> *forget to
<bigjools> those people should be slapped :)
<Ursinha> bigjools: how hard is to you to unmark it as fix committed? :)
<bigjools> Ursinha: that's not the point - on a popular bug it generates a lot of email spam
<Ursinha> bigjools: what do you suggest?
<bigjools> Ursinha: first thing I guess is to not do anything to the bug if it has multiple branches listed
<Ursinha> anything I don't agree :)
<Ursinha> at least a bug comment
<bigjools> Ursinha: that should not be done by your bot
<bigjools> it should be done by LP
<Ursinha> bigjools: is there a bug for that?
<bigjools> Ursinha: since we're only just talking about it, I guess not :)
<Ursinha> bigjools: I don't know if all projects would like to have comments added to their bugs, but that's not up to me to decide :)
<Ursinha> bigjools: could you open a bug for that, please?
<Ursinha> bigjools: script works now basically with how lp works today, if lp adds new features maybe we could add changes to the current qa tagging process
<bigjools> Ursinha: what triggers the script?
<Ursinha> bigjools: cron
<Ursinha> :P
<Ursinha> bigjools: it checks all the new commits since last run, and if they're on edge or staging
<bigjools> Ursinha: what triggers the changes to bugs I mean
<jpds> http://pastebin.ubuntu.com/401769/ - anyone know why that's happening?
<bigjools> Ursinha: brb
<Ursinha> bigjools: if the commit is already on edge or staging, then the script does his dance with the bug
<Ursinha> *its
<bigjools> Ursinha: how does it know which bug?  does it use the bug= in the commit msg?
<bigjools> or does it inspect the --fixes bzr stuff?
<Ursinha> bigjools: not just that, if the bug# is in the commit message
<Ursinha> bigjools: again, not all people use to remember adding the bug= tag :)
<bigjools> Ursinha: ok
<Ursinha> with ec2 that changed because ec2test/12
<Ursinha> argh
<bigjools> Ursinha: so I can avoid it by not mentioning the bug number
<Ursinha> wait
<bigjools> Ursinha: however, the codehosting code can do something like I suggested earlier and add a comment to the bug if it sees a linked branch merged
<Ursinha> argh
<sinzui> EdwinGrubbs: bac: stand up in two minutes
<deryck> adeuring, if you have time, could you look at the OOPS for Bug #548575?  more ShortListTooBigErrors.
<adeuring> deryck: sure
<deryck> adeuring, thanks!
<mup> Bug #548575: Not able to unsubscribe from a bug <Launchpad Bugs:New> <https://launchpad.net/bugs/548575>
<adeuring> deryck: did we really CP the fix?
<EdwinGrubbs> sinzui: standup?
<deryck> adeuring, I thought we did.  Maybe we didn't.  Did you submit it to production-devel?
<adeuring> deryck: can't remember...
<adeuring> let me check...
<sinzui> EdwinGrubbs: https://code.launchpad.net/~bac/launchpad/bug-524302/+merge/22180
<deryck> adeuring, I assume we didn't CP that fix then?
<adeuring> deryck: no, we didn't cherrypick that branch. Sorry, needed some time to figure out which branch fixed the bug...
<deryck> adeuring, no worries.  Can you mark the bug a dupe of the bug that does fix this, please?
<adeuring> sure
<james_w> gary_poster: here's what I was talking about yesterday: https://code.edge.launchpad.net/~james-w/lazr.restful/object-unmarshal/+merge/22182
 * gary_poster looks
<gary_poster> james_w: do you want a review?
<gary_poster> rephrase:
<gary_poster> do you want me to review
<james_w> yes please
<gary_poster> ok
<james_w> I'd like leonard to look as well
<james_w> I'm not entirely convinced by the approach as it isn't a great example of elegance, and prevents you doing some odd but legal things.
<james_w> but I can't find a better way
<gary_poster> james_w: leonard will not be able to before this release (not around today).  I agree that he should review.  Because he is not around today, this won't make it into the cycle unless we push it as release-critical.  My gut feeling so far is that this is more important for developing (that is, future work) that for production (that is, this release).  Do you agree?
<gary_poster> (IOW, the fact that this will not make the release does not seem like a release-critical problem; but we should get it into the devel branch asap)
<james_w> gary_poster: yes, sinzui landed the branch that fixes the problems yesterday, this is to stop any more from creeping in.
<gary_poster> james_w: cool.  I'll look now to see if I have anything to add, and make sure that reviewing this branch is in leonardr's to-do list.
<james_w> thanks
<james_w> there's a long line in the doctest that I don't know how to wrap
<gary_poster> james_w: you can use #doctest: +ELLIPSIS and elide the text with an ellipsis. #doctest: +NORMALIZE_WHITESPACE might let you wrap the line, but IME sometimes tracebacks are "exceptions" to the rule.  A-ha-ha.  But you could try that first.
<gary_poster> james_w: >>> reference_marshaller.unmarshall(None, cookbook) # doctest: +NORMALIZE_WHITESPACE
<james_w> that goes on the test line, or on the response line?
<james_w> ah, thanks
<gary_poster> or
<gary_poster> >>> reference_marshaller.unmarshall(None, cookbook)
<gary_poster> ... # doctest: +NORMALIZE_WHITESPACE
<james_w> fixed, thanks
<gary_poster> cool, np
<gary_poster> james_w: yeah, that
<gary_poster> 's not fun.  Can you tell me where the adapter is that you mention in the cover notes?
<gary_poster> That uses an Object marshaller for a Choice?
<gary_poster> (with the certain kind of vocab)
<james_w> some webservice zcml
<james_w> lib/canonical/launchpad/zcml/webservice.zcml
<gary_poster> ack thanks
<james_w> I thought the most elegant way to solve it would be for tales.py to just add _link if it would just an ObjectMarshaller for that field, but there's already a comment there saying it can't do that
<gary_poster> james_w: My thought would have been that using an ObjectMarshaller is an abuse.  We should write our own marshaller for this case that does the right thing.
<gary_poster> Does the tales comment (which I have not yet looked up) contradict that?  Going to hunt down tales comment...
<james_w> search for Marshaller in src/lazr/restful/tales.py
<james_w> I'm not sure what "the right thing" for the new marshaller to do would be
<james_w> so, the root cause is that there are two different things, using two different criteria, deciding what the field name should be in the API. When they disagree you get problems.
<gary_poster> james_w: a minimal "right thing" would be to puke, as you are doing here.  Puking where you have it now (in lazr.restful) doesn't make sense because, looking at the code, why would an ObjectMarshaller ever be called with anything other than an IObject?  It shouldn't.
<james_w> A secondary concern is that we want exported objects to be serialised to links, and when that is done for the field name to have _link appended.
<gary_poster> root cause: that sounds valid, but it's been too long since I looked at the code for me to have an informed opinion without staring for awhile more.
<james_w> gary_poster: right, it never is. The issue is when that Marshaller isn't bound to an "object" field type, the name is already wrong in the wadl.
<james_w> so I'm fixing the wrong end in effect, as the problem is already past, but this barfs at you and tells you something is wrong at least, rather than silently doing the wrong thing.
<james_w> I couldn't see how to fix it at the "other end", which would be the field declaration and tales.py, as we need the developer to indicate whether it is a field that references another exported object or not.
<james_w> which they do by using ReferenceChoice instead of Choice
<james_w> perhaps the right thing to do is change the adapter declaration so that it only affects IReferenceChoices?
<gary_poster> james_w: right, which is why I called it an abuse.  We can't protect ourselves from all abuses in lazr.restful.  Here's my proposal for your current motivations.  Since I have this alternative, I would not approve your approach.  Leonard may have an even better answer.
<gary_poster> - we make our own adapter for
<gary_poster>        for="zope.schema.interfaces.IChoice
<gary_poster>             zope.publisher.interfaces.http.IHTTPRequest
<gary_poster>             canonical.launchpad.webapp.vocabulary.SQLObjectVocabularyBase"
<gary_poster> that says "this is broken!" or some other pukage
<gary_poster> - we change the current registration to
<gary_poster>        for="lazr.restful.interfaces.IReferenceChoice
<gary_poster>             zope.publisher.interfaces.http.IHTTPRequest
<gary_poster>             canonical.launchpad.webapp.vocabulary.SQLObjectVocabularyBase"
<gary_poster> Done.
<gary_poster> Actually, the first step could be to return None
<james_w> and zope knows to pick the more specific adapter?
<gary_poster> but then we could provide a helpful error ("Use IReferenceChoice, silly person!")
<gary_poster> I mean, couldn't
<gary_poster> yes
<james_w> right, that makes sense
<gary_poster> james_w: I'll copy some edited version of this over to the MP for leonardr, and move on.  s'ok?
<james_w> yep
<gary_poster> cool.  thank you for doing this.
<james_w> I'll work on that after lunch
<gary_poster> great
<james_w> I think that Leonard may still want to consider a change to lazr.restful, as the discrepancy could still bite people using it.
<james_w> but this is the right fix for LP as the bad registration is what is causing the problem.
<gary_poster> lazr.restful change: OK.  I argue against it because of the "can't stop every abuse" argument, but if you convince me I'll go placidly along. :-)
<gary_poster> I mean, convince him
<james_w> sure, it's not so much about abuse. I wasn't the one that put the comment identifying the issue in tales.py after all :-)
<gary_poster> :-)
<mrevell> Night all
<elmo> wgrant: around?
<gary_poster> james_w: fwiw, I echo Leonard's question to you in https://bugs.edge.launchpad.net/launchpad-foundations/+bug/547216
<mup> Bug #547216: WADL file doesn't work in NetBeans <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/547216>
<james_w> gary_poster: replied
<gary_poster> thanks james_w
<james_w> gary_poster: and https://code.edge.launchpad.net/~james-w/launchpad/export-code-import/+merge/22251 is done
<james_w> just generating the diff
<gary_poster> james_w: ack, claimed the review.
<james_w> thanks
<james_w> it's there now
<gary_poster> yup, got it
<james_w> the main concern was what to call the class and what file to put it in
<gary_poster> lol, ack
<wgrant> elmo: Hi.
<elmo> 18:39 < elmo> so https://bugs.launchpad.net/soyuz/+bug/549041 appears to be either massive lulzwtfomghfs or hilarious PEBKAC
<mup> Bug #549041: source <-> binary reference counting appears to be broken <Soyuz:New> <https://launchpad.net/bugs/549041>
<elmo> wgrant: ^-- was looking for a sanity check on that bug
<wgrant> Let's see.
<wgrant> Eeeeep.
<elmo> quite
<wgrant> Grngh.
<wgrant> Stupid timeouts.
<wgrant> elmo: So, LP says it is still there.
<wgrant> (no "Removal requested" or "Removed from disk" on https://edge.launchpad.net/ubuntu/hardy/+source/linux/2.6.24-26.64)
<wgrant> Are they not in the pre-split cocoplum archive?
<elmo> aha
<elmo> god damn it
<elmo> this is going to be my fault
<wgrant> Your script is dodgy?
<wgrant> Yeah.
<elmo> gack
<elmo> they are indeed on cocoplum
<elmo> damn it all
<elmo> well, yay, I guess that makes it hilarious PEBKAC
<wgrant> I've often wondered how you make the split cope with this.
<wgrant> I guess you don't?
<elmo> apparently not
<elmo> we'll have to change the source syncing to just be unconditional and not based on indices
<wgrant> Yes...
<wgrant> Although is there a good reason that we can't keep the old sources in the indices? dak does that a bit now, and most tools have been taught to deal with it.
<wgrant> But yes, syncing everything is a good quick (but large) fix.
<elmo> I'm fine with keeping sources in indices, but the more I can lobotomize the magic mirror script the better
<wgrant> I guess ports just does dumb matching of arch tags, so it isn't affected?
<elmo> the magic mirror script runs on all 3 (archive, ports and old-releases), it can't do simple arch tag matching because we had different arches supported in different releases
<elmo> so it has to do arch taggging based on Packages files
<wgrant> Oh, crap, yes.
<elmo> but that's fine, I don't mind losing unreferenced binary files so much and we implement a local S-O-E
<elmo> to minimize the pain
<elmo> I just can't believe it took us (or anyone else) this long to notice
<wgrant> It's presumably been around... forever?
<elmo> (it was, unsurprisingly, the gnewsense guys)
<wgrant> Hah.
<elmo> wgrant: not forever, basically since we first had an arch migrate off archive, I think
<elmo> until then, it was just dumb rsync rules
<wgrant> Ah.
<elmo> hmm
<elmo> this does make old-releases near impossible to get right in a non-GPL violating way
<elmo> maybe I should re-open that bug as "please don't de-indices Source till it's got no binaries left"
<wgrant> That's easy to do.
<wgrant> It might just break older tools.
<elmo> ah, that's a bit sucky
<wgrant> Like, say, gina.
<elmo> haha
<elmo> feature
<elmo> which would affect, e.g. dapper-updates
<elmo> ho hum
<wgrant> Yes.
<wgrant> That is my concern.
<elmo> do you know how long Debian's been doing this?
<wgrant> Only a few months.
<elmo> ah, bother
<wgrant> Yes.
<elmo> so this becomes less easy since I really want it to be "please don't de-indices source packages - in DS >> lucid - till it's got no published binaries left"
<elmo> oh well; GPL compliance, much like the tank in halo, beats everything.  I'll reopen the bug and leave it for someone else to figure out how/when/if to do it
<wgrant> Heh.
<wgrant> If only we could sanely do the split in Soyuz.
<wgrant> It has all the knowledge.
<wgrant> It's just... hard.
<elmo> why?
<wgrant> Well, we'd either have to maintain four separate archives and move publications between them, or give the publisher enough intelligence to work over multiple on-disk archives and filter its activities.
<elmo> yeah, more the latter
<elmo> I guess that is hard.  I was going more for messy and involved
<elmo> but I'm probably a little too removed to accurately estimate
<wgrant> The latter is probably hard mostly because of the "when is this source unused in this filter set?" policy.
<elmo> I wouldn't even bother trying
<elmo> for source specifically
<wgrant> Hm.
<elmo> just source reference count as before and de-index + de-publish when your last child binary is removed; across any filter set
<elmo> hmm
<wgrant> elmo: That fails utterly for old-releases.
<elmo> yes, I just realised
<elmo> and so does what I said in the bug
<wgrant> It's actually not that hard, though.
<elmo> mock conkeys
<wgrant> We just need to filter the publications using the same queries we started with.
<wgrant> So it's doable.
<elmo> hmm?
<elmo> how do you mean same queries
<wgrant> Well, the unused source check performs a query to find any binaries for that source in the same archive.
<wgrant> It would just have to also have the same restriction applied that we used to find the source in the first place. So it's not as hard as I first thought.
<elmo> oh, right, binaries aren't floating
<elmo> wow
<wgrant> Hm?
<elmo> I am *so* rusty at this stuff
<wgrant> "floating"?
<elmo> I had it in my head that we might have binaries not attached to DSes
<wgrant> Ah.
<wgrant> No, we're not dak.
<elmo> haha
<elmo> :-P
#launchpad-dev 2010-03-27
<wgrant> YesIamsureIwanttochangethebugstatus.
<wgrant> Grrr.
<wgrant> jtv: Morning. Are you going to have time to fix the /builders icon positioning in the next couple of days, or should I?
<jtv> wgrant: wrapping up a sprint here...  I think I missed the boat on that one, sorry.
<jtv> wgrant: did get an answer though on slave build cookies vs buildqueue ids.
<wgrant> jtv: OK, I'll fix that up today.
<wgrant> What was the answer?
<jtv> Apparently there's talk of having slaves in the feature that may not even be in the buildfarm, in which case a cookie has real value (and what we have now has... less value I guess :)
<wgrant> jtv: "not even be in the buildfarm"?
<wgrant> You mean completely untrusted?
<jtv> wgrant: hey, just telling you what I heard.  :)  Maybe it means upstream packages build their own packages or something.
<wgrant> GAH
<wgrant> YES I DO WANT TO SET THIS TO 'IN PROGRESS'
<wgrant> Does Launchpad not want me to fix its bugs?
<wgrant> (Bug #549323 )
<mup> Bug #549323: Discourages bug fixes <Launchpad Bugs:New> <https://launchpad.net/bugs/549323>
 * jtv swallows flippant speculation about how software feels about its bugs
 * wgrant plays the "poke scripts semi-randomly and see if they make Rosetta do the right stuff" game.
 * wgrant switches to the "oh wow, translation template builds and imports actually completely work now, with a few tweaks on top of trunk" dance.
<wgrant> jtv: Are you actually present?
#launchpad-dev 2010-03-28
<mwhudson> good morning
<thumper> morning
<mwhudson> thumper: morning
<thumper> mwhudson: shall we have our morning call?
<mwhudson> thumper: sure, couple minutes?
<thumper> ok
<mwhudson> thumper: ready now
<mwhudson> thumper: http://xkcd.com/171/
<thumper> http://sourceforge.net/tracker/?group_id=38414&atid=422032
* mwhudson changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.03 | PQM is in open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes |
#launchpad-dev 2011-03-21
<wgrant> lifeless: I believe so.
<lifeless> [there's a few untriaged bugs in launchpad-project]
<wgrant> Ah.
<wgrant> lifeless: We need to delete gandwana and potassium.
<wgrant> So many OOPSes with impossibly high non-SQL time on them.
<huwshimi> wgrant: oh, thanks for triaging bug #737327... I forgot to change the status.
<_mup_> Bug #737327: django should not be running in debug mode on production, it will eat all the RAM <OOPS Tools:Triaged> < https://launchpad.net/bugs/737327 >
<wgrant> Also palladium
<lifeless> wgrant: they are on the list
<michaelh1> Hi there.  If I click on any merge request on staging I get either a timeout or an oops.  Have you seen this before?
<lifeless> the testing librarian does not forward appserver requests to the prod one
<lifeless> so you need to make a new merge proposal to test merge proposal stuff on staging
<wgrant> Well, it's more that the staging librarian can't see restricted files from prod.
<wgrant> And all MP diffs are restricted.
<lifeless> wgrant: its not a token issue, it could forward
<lifeless> its just a bug
<wgrant> Right, but it's specific to restricted files.
<wgrant> Not proxying.
<lifeless> same thing
<michaelh1> I'd like to play with merge requests from launchpadlib.  What would you suggest?
<lifeless> make new ones on staging :)
<lifeless> (actually, qastaging, its a better testbed)
<michaelh1> That should work, ta.
<lifeless> what does branchrevision.sequence is NULL imply ?
<mwhudson> lifeless: off mainline
<wgrant> Whoever named compiz should have thought about how much it would crash.
<wgrant> It's very hard to start again when it dies and there is no z in your terminal buffer.
<StevenK> wgrant: Alt-F1 ; DISPLAY=:0.0 compiz & ?
<wgrant> StevenK: That breaks envvars when you're using unity.
<wgrant> Since it appears to be a Compiz plugin now.
<StevenK> There is no pleasing you.
<lifeless> mwhudson: thanks
<elmo> wgrant: kill unity, and start it again
<elmo> wgrant: it'll bring up a new compiz instance
<elmo> (from the terminal, I mean, i.e. the Alt-F1 scenario)
<wgrant> elmo: With a real environment?
<wgrant> So dbus apps won't cry?
<elmo> wgrant: *shrug*, it WFM on natty without a real environemnt (just DISPLAY) - I just had to do it just now when unity suicided on me
<wgrant> elmo: hm, OK, will try that in future. Thanks.
<wgrant> Mine dies a few times a day at the moment... but I'm using r600g, so that's not too surprising.
<elmo> I'm using Intel :(
<wgrant> My Intel laptop is still safely on maverick.
<poolie> hi all
<wgrant> Afternoon poolie.
<wgrant> ...
<wgrant> Archive.sources_size and binaries_size.
<wgrant> ...
<StevenK> wgrant: What about them?
<wgrant> Look at them.
<StevenK> I don't want my eyes to bleed?
<wgrant> It grabs every LFA and LFC in the archive.
<wgrant> Ah, actually, just the filesizes. But all of them. Into python.
<wgrant> Then sums them.
<wgrant> In Python.
<wgrant> It's probably not *that* slow. But it's stupid.
<StevenK> Er, yeah. Now you that mention it, SQL can do the entire heavy lifting
<StevenK> Hm. I think I can see the bug
<StevenK> (Pdb) print spr.changelog.read()
<StevenK> <security proxied lp.registry.model.sourcepackagename.SourcePackageName instance at 0x106ce390> (1.0-1derived1) unstable; urgency=low
<wgrant> Er, what?
<StevenK> I'm missing a .name
<wgrant> Is this for tests?
<StevenK> Yes
<wgrant> Ah, good.
<StevenK> Haha, I'm not that evil.
<lifeless> man, so many soyuz bugs in 'incomplete wasused to clarify something and never changed' state
<lifeless> lets see
<lifeless> bugtask, dsitribution, cve all as much under control as they will be
<lifeless> i guess its productset time
<spm> bugs don't auto switchout of incomplete status after any additional info added do they? perhaps they could?
<lifeless> pehraps
<lifeless> expiry has had a difficult birth
<lifeless> its not really battle tested
<lifeless> I don't mean to indicate doubt about the idea; just that there are several ways to tackle it
<lifeless> revamping status is bigger but perhaps worth it
<lifeless> solve a bunch of headaches at once
<spm> sure. I guess I've always viewed (anywhere) that bug status fields are project management fields - in my case, I'll update a bug, but typically won't change the status.
<lifeless> indeed
<lifeless> it can be frustrating for folk that identify with a project, when 'users' or bug filers change metadata
<lifeless> wgrant: qa time I think
<lifeless> wallyworld_: you too :)
 * wallyworld_ want to finish some debugging first
<wgrant> lifeless: A good point.
<lifeless> wallyworld_: thats a priority inversion: unqaed stuff on trunk is a -blocker- and comes before all other work except production-is-down situations
<lifeless> wgrant: I'm looking at sinzuis thing
<wallyworld_> lifeless: ok. but it means a context switch. i'll get onto it though....
<lifeless> wallyworld_: it does, and thats why I want us to drive the latency way down so a context switch is less often needed.
<lifeless> wallyworld_: nevertheless, trunk is a critical section :(
<wallyworld_> np.
<wgrant> Baah
<wallyworld_> lifeless: how do we test something that needs to access a diff when the data is not available via qastaging?
<lifeless> mute is still behind a ff right ?
<wgrant> Yes.
<lifeless> wallyworld_: you need to make a new merge proposal or whatever
<wallyworld_> ok
<lifeless> wallyworld_: if you're very confident things will be better not worse, you can punt and go 'qa-untestable'
<lifeless> the goal is to make a good tradeoff between time invested and risk of breaking stuff
<wgrant> I don't think it's worth QAing that one.
<wgrant> It's going to take a while.
<wgrant> Apart from that we're OK to deploy.
<wallyworld_> lifeless: np
<lifeless> wgrant: do you need me to fire up evo and look at the mailbox ?
<wgrant> lifeless: No, I got what I needed.
<wgrant> Thunderbird just took a long time to download all few hundred thousand headers.
<lifeless> wgrant: clear it out
<wgrant> Just delete them?
<lifeless> yeah
<lifeless> its a test environment, not gmail :)
<wgrant> Heh.
<lifeless> hmm
<lifeless> the tagger seems to be missing bugs :(
<lifeless> see rev 12629
<wgrant> The syntax in that commit message is invalid.
<lifeless> wgrant: they are also linked to the MP
<wgrant> Hmm.
<lifeless> 411 exceptions
<lifeless> nice
<wgrant> Not really.
<wgrant> It was less yesterday.
<lifeless> down from 20K :P
<wgrant> Shh.
<lifeless> POFile:+translate is still in trouble
<StevenK> wgrant: I think I may have thought of the only case we want to delete DSDs.
<StevenK> wgrant: If the source is removed from both the child and parent distroseries.
<wgrant> StevenK: Even then it's questionable. But maybe.
<StevenK> wgrant: Questionable why?
<wgrant> StevenK: There are situations in which the package could be deleted from both for a while and then restored. But the diff is unlikely to be important in that case, I guess.
<wallyworld_> huwshimi: ping
<huwshimi> wallyworld_: Hello
<StevenK> wgrant: It's just a thought. I'm going to add two test cases to my end to end testing to see if the DSD gets updated on package deletion too.
<wallyworld_> huwshimi: bug 580404
<_mup_> Bug #580404: pressing enter in tags field doesn't save them <ajax> <javascript> <lp-bugs> <qa-ok> <Launchpad itself:In Progress by huwshimi> <LAZR Javascript Library:Fix Released by huwshimi> < https://launchpad.net/bugs/580404 >
<wgrant> lifeless: You're requesting a deploy?
<lifeless> wgrant: in about 75 minutes
<wallyworld_> i qa-oked it but it doesn't seem to properly fix the issue
<lifeless> wallyworld_: do we need to roll it back ?
<wgrant> lifeless: The portlet fix?
<lifeless> wallyworld_: or is it ok to deploy, just incomplete?
<lifeless> wgrant: yes
<wallyworld_> huwshimi: no, since it doesn't break anything, but it doesn't seem to properly fix it
<lifeless> wallyworld_: then its qa-ok
<wallyworld_> lifeless: i know that
<StevenK> Here Comes The Rain (to the tune of Slayer's Here Comes The Pain)
<lifeless> wallyworld_: ok :) - was unsure what you're saying then
<wallyworld_> i just wanted to see if i was testing proerly
<wallyworld_> in case we needed to raise a new bug
<lifeless> wallyworld_: In this sort of case I add a comment to the bug saying 'does not fix' and whoever does the deploy will put it back to in-progress
<huwshimi> wallyworld_: It isn't testable until we release a new version of lazr-js with Launchpad. Which I've been distracted from doing
<wallyworld_> lifeless: thanks for the tip
<lifeless> thumper: bug 615647 - I've played around for a more efficient query, and it seems to work ok to me; could you eyeball it and check I haven't missed anything?
<_mup_> Bug #615647: BranchMergeProposal:+index timeout <lp-code> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/615647 >
<wallyworld_> huwshimi: i have uploaded rev 206 of lazrjs to the download cache. still getting the branch that needs it fully debugged
<wallyworld_> huwshimi: are your changes done prior to rev 206? if so, they
<wallyworld_> will get rolled out when 206 goes in
<huwshimi> wallyworld_: yeah they're in 204
<huwshimi> wallyworld_: Thanks
<wallyworld_> huwshimi: np. i'll try and get my branch through today. gotta fix some failing tests
<huwshimi> Is there any way to add a class with a condition using TAL. More specifically I want to add the class 'hidden' to a div under a certain condition. The div already has other classes, just to complicate things.
<lifeless> huwshimi: probably best to do that in the view
<huwshimi> lifeless: Really? It seems like the sort of thing that should be done in a template?
<wgrant> It belongs in the template. But it's difficult and ugly to do it there.
<lifeless> huwshimi: if you want to do conditional logic you want a real programming language
<lifeless> huwshimi: either js, or python, not -ever- tal
<huwshimi> lifeless: Oh, but we have heaps of conditional stuff in our templates.
<lifeless> huwshimi: which has a bunch of poor sideeffects
<lifeless> such as:
<lifeless>  - timeouts (tal:when used on a collection that triggers an additional count(*) of an expensive query)
<lifeless>  - complex templates that can't be used in multiple contexts
<lifeless>  - expensive tests (need to test complete rendering - and rendering uses the whole stack- for more code paths)
<lifeless> huwshimi: I'm exaggerating a /little/ for effect, but templates are really not suitable for anything but the simplest of conditionals, and tal inparticular, with its obsession on document structure as a pun for program logic makes this substantially worse when you're not dealing with a dom subtree
<lifeless> wgrant: btw - high nonsql time is (IME) correlated with repeated lookups; I suspect storm GIL handling exacerbates high load situations (e.g. too many handoffs)
<wgrant> lifeless: Quite possibly, yes.
<lifeless> (so fixing such timeouts is easily possible via code  :P)
<lifeless> hmm
<lifeless> makeCodeImport() doesn't create the branch ?
<lifeless> or... not scanned I guess
<huwshimi> lifeless: What I'm doing is purely to do with presentation (and handling broken situations with no javascript). If from what I understand you mean we should be returning css classes from the python view (which I've seen elsewhere in Launchpad) which I think is a bad idea (it tightly couples CSS to views, which makes things less re-usable and a lot harder to change), if anything we should be returning boolean flags wh
<huwshimi> ich we can check against to modify the template accordingly  (which is what we have already in this case).
<huwshimi> lifeless: At this point it looks like my best bet is to duplicated a chunk of HTML under 'condition' and 'condition not:'
<lifeless> huwshimi: I suspect thats more of a maintenance burden than I was suggesting.
<lifeless> huwshimi: in terms of reuse and change, I haven't seen anything in tal that makes it easier to change than a view, and we have nearly universally a 1:1 view->template mapping anyway.
<lifeless> huwshimi: which, I admit, may be because we have things all funky at the moment.
<huwshimi> lifeless: So really we have this limitation of having views return classes due to limitations with our templating language?
<lifeless> huwshimi: I think we have different perspectives.
<lifeless> huwshimi: you say you are working on presentation, right ?
<huwshimi> lifeless: Correct
<lifeless> huwshimi: view objects *are* the presentation layer
<lifeless> huwshimi: templates are bound to a view for convenient editing of the html
<lifeless> huwshimi: but its not a limitation, bug or defect to do html in the view.
<lifeless> including complex logic for deciding on things - like initial visibility
<lifeless> huwshimi: I'll note that duplicating a section of html in the TAL will have a negative impact on performance
<huwshimi> lifeless: OK I see where our opinion differs.
<huwshimi> lifeless: Although I would say that it is a limitation of our templating language that we can't properly manipulate the HTML it ouputs.
<lifeless> huwshimi: TAL is in some ways /extremely/ primitive; the implementation we use is also very slow - the work gary and others have done to make chameleon available will, when complete, help with performance a lot
<lifeless> in other ways TAL is pretty nice
<huwshimi> lifeless: So for this I should move all my classes on this div into the view and return them depending on conditions there?
<lifeless> huwshimi: both wgrant and I reacted that way, without having seen the exact situation
<wgrant> Except I say it's a necessary evil, and lifeless says it's good.
<StevenK> Hm, I wonder how far chameleon is off
<huwshimi> wgrant: I think it's particularly evil cause you can't separate out the classes that should be returned on the condition to the ones that should always be applied :(
<wgrant> huwshimi: Yup.
<mwhudson> you could do tal:attributes="class string: always ${view/optional}" or something?
<wgrant> True.
<huwshimi> mwhudson: Can you explain that to me?
<mwhudson> huwshimi: which bit?
<thumper> lifeless: public holiday here today :)
 * thumper is off all week
<wgrant> Thunderbird has managed to use 6.5GiB of RAM.
<wgrant> I am impressed.
<mwhudson> huwshimi: maybe this bit? http://docs.zope.org/zope2/zope2book/AppendixC.html#tales-string-expressions
<lifeless> thumper: orly ?
<lifeless> thumper: I wants :P
<huwshimi> mwhudson: Thanks I'll take a look
<lifeless> huwshimi: the word always would be a class you want present no matter what
<wgrant> lifeless: Do you have any thoughts on extending BrowsesWithQueryLimit to help with scaling tests?
<wgrant> ie. I want to specify a maximum baseline, test the baseline, then test that it doesn't scale above that.
<lifeless> wgrant: do you mean 'allow it to do LessThan' ?
<StevenK> typos in URLs make me sad.
<wgrant> lifeless: I guess I want the matcher to remember the last value it checked, and use that as the max for future calls.
<lifeless> wgrant: so, this would permit accidental increases in baseline
<lifeless> wgrant: but be more tolerant of deliberate increases in core infrastructure
<wgrant> lifeless: It would still have an initial value.
<wgrant> But would use the last checked value in preference.
<lifeless> wgrant: so it would ratchet down
<wgrant> Right.
<lifeless> so the failure you are preventing is 'someone set the initial value way to high'
<wgrant> More "someone improved other stuff, so the initial value is now way too high and nobody noticed because we use LessThan instead of Equals"
<lifeless> wgrant: you can do this if you want, should be a small matter of code; I don't have a strong opinion any which way.
<lifeless> wgrant: as an alternative, perhaps error if the query count is *too low*
<wgrant> lifeless: That's what I suggested last week, but you seemed neutral/negative.
<lifeless> wgrant: did I? sorry.
<wgrant> Quite reasonably, because it's more sensitive to widespread query reductions.
<wgrant> I'm not sure which concern should win.
<lifeless> wgrant: did you suggest users supplying a minimum, switching to Equals, or having a heuristic ?
<wgrant> lifeless: Switching to Equals.
<wgrant> Users supplying a minimum is silly.
<lifeless> wgrant: ah
<wgrant> And a heuristic is a heuristic.
<lifeless> wgrant: so I'm suggesting using a heuristic, not switching to equals
<lifeless> wgrant: one such heuristic would be 'allow off by 1'
<lifeless> another would be
<lifeless> 'alllow a 10% margin below the count given'
<lifeless> wgrant: I'm still neutral/negative on using equals. I think we need some slack.
<lifeless> thumper: whats the holiday?
<lifeless> I can has review? https://code.launchpad.net/~lifeless/launchpad/bug-734751/+merge/54146
<wgrant> lifeless: "allow off by 1" rots, "allow a 10% margin below the count given" is going to miss scaling issues.
<wgrant> Unless the 10% margin only applies to the first time.
<lifeless> wgrant: i like the idea of combing a heuristic and an observed cap
<lifeless> combining*
<lifeless> 5 second portlets still suck, but at least its less suck
<wallyworld_> anyone feel like reviewing https://code.edge.launchpad.net/~wallyworld/launchpad/unassign-private-bug/+merge/53945
 * StevenK slaps wallyworld_ for pasting edge URLs
<wallyworld_> ouch
<wallyworld_> stupid browser cache
<StevenK> wallyworld_: Why do you have a return at the end of _cacheViewPermission()?
<wallyworld_> StevenK: cause i'm stupid. it's a typo
<wallyworld_> StevenK: actually no
<wallyworld_> it's to exit the loop as soon as a match is found
<StevenK> Ah, right
<wallyworld_> StevenK: notice how my first reaction was to think how dumb i am :-)
<lifeless> wgrant: wallyworld_: StevenK: I'm tired of my merge proposals timing out. So have fixed. Could one of you eyeball this please: https://code.launchpad.net/~lifeless/launchpad/bug-615647/+merge/54150
 * wallyworld_ looks
<lifeless> if you're not a full reviewer thats ok - I'm going to self review, this thing is going down ;)
<StevenK> No diff yet
<StevenK> wallyworld_: If you can confirm self.bug in the context of those two tests is private, r=me
 * wallyworld_ looks
<lifeless> wallyworld_: reviewed.
<wallyworld_> thanks.
<wallyworld_> StevenK: self.bug is created in the test setup
<StevenK> wallyworld_: I'm abstaining, due to lifeless' review of your MP
<StevenK> WTF, how hard can it be to switch DB users
<wallyworld_> lifeless: do we really know that the current user has permission? what if they were removed from a team after they had already opened the page?
<StevenK> self.layer.switchDbUser() insists that LaunchpadFunctionalLayer doesn't have that method, and LaunchpadZopelessLayer.switchDbUser() gives ZopelessTransactionManager not installed
<lifeless> wallyworld_: then how do they reach the code to call unassign?
<wallyworld_> lifeless: might they have the page open and before they go to unassign, they are removed from the team by someone else. on their browser, they are still assigned
<wallyworld_> so they could concievably stil lmake the api request to the back end
<lifeless> wallyworld_: then the publishing code will return a 404 to the POST
<wallyworld_> ok
<wallyworld_> so i'll take away the checks and just add the user to the cache
<lifeless> wallyworld_: and move the code
<wallyworld_> to?
<lifeless> wallyworld_: in fact, this is a self inflicted bug :)
<lifeless> wallyworld_: have you got transitionToAssignee open ?
<wallyworld_> lifeless: no
<wallyworld_> but i can
<lifeless> grab it now
<wallyworld_> open
<lifeless> look at the last line of the third if block
<wallyworld_> you mean clearing the cahce?
<lifeless> it explicitly enforces cache coherency on the viewers, *discarding* the fact that the active user can see the bugtask.
<lifeless> this is in conflict with the intent of your change which is to 'permit the user to view stuff that they *could* even if they *can't now*
<wallyworld_> yeah, i didn't really know this bit of the code base that well. just found what looked like would work
 * StevenK glares at the apartment above him
<lifeless> wallyworld_: http://pastebin.com/iwLt6zhw
<StevenK> Some ingrateful muppet is using a hammer drill
<lifeless> wallyworld_: seems to be all that is needed to me - + a test that tests the launchpadlib call works
<wallyworld_> lifeless: i think it's more than just changing the assignee though. from memory, the same issue cam up when unsubscribing oneself
<wallyworld_> and the change i did sorted out that issue too
<wallyworld_> but that was last week. i may have misremembered
<wallyworld_> i'll look into it
<lifeless> wallyworld_: there is a separate method for unsubscribe
<lifeless> looks like an identical bug
<wallyworld_> yeah, probs
<lifeless> the reason I think the lifecycle change, while elegant, is a problem, is because we have direct cache management - and the two will stomp on each other.
<lifeless> the first like of unsubscribe needs to either be more selective about what it clears, or to keep the known viewers list.
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-615647/+merge/54150 has a diff up
<wallyworld_> lifeless: your unmerged revisisions change - is the order by BranchRevision.branch desc necessary?
<wallyworld_> lifeless: i'm late to pick up the kid from school. back soon
<lifeless> it selects the right index
<lifeless> wallyworld_: the index is on (branch, sequence)
<lifeless> wallyworld_: it /may/ work without it. It /does/ work with it - see the various test plans in the bug
<wgrant> Anyone want to review https://code.launchpad.net/~wgrant/launchpad/trim-asp/+merge/54152?
<StevenK> wgrant: r=me
<wgrant> StevenK: Thanks.
<StevenK> wgrant: Can haz help switching db user?
<wgrant> StevenK: Use LaunchpadZopelessLayer.
<wgrant> Call LaunchpadZopelessLayer.switchDbUser
<StevenK> wgrant: Does that allow me to upload to the librarian?
<wgrant> StevenK: Yes.
<wgrant> "Launchpad" generally means that the librarian is up.
<lifeless>  bah, no stubs
<StevenK> He was here
<lifeless> yes
<lifeless> was
<lifeless> timed out 2 minutes back
<lifeless> StevenK: can you do a review for me ?https://code.launchpad.net/~lifeless/launchpad/bug-734751/+merge/54146
<StevenK> lifeless: rme
<lifeless> StevenK: thanks
<StevenK> Sprinkle = to taste
<poolie> wgrant, re the mail mangling
<poolie> i have to say tim's thing a while ago that spam causes the headers to be inserted into the body gives me a queasy feeling about lp mail handling
<wgrant> poolie: Yes.
<wgrant> poolie: This is a good excuse to examine those configs and unbreak it.
<wgrant> I don't entirely know what the incoming mail path is :/
<StevenK> wgrant: Ask a friendly* sysadmin?
<poolie> is that a Kleen star? :-)
<poolie> lifeless, it's not top of my list (nor i dare say yours) but i was wondering after friday's mail what your actual thoughts on triaged vs confirmed were
<poolie> are you also hoping to merge them?
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge | https://code.launchpad.net/launchpad-project/+activereviews/
<StevenK> henninge: O hai! I have one for you :-)
<henninge> Hi StevenK! Let me see it ;-)
<StevenK> henninge: https://code.launchpad.net/~stevenk/launchpad/dsdj-runner/+merge/54159
<lifeless> poolie: that 3 page bug update is about as far as my thinking has advanced
<lifeless> poolie: I think that 'triaged' as it stands is entirely redundant, but that piecemeal solutions probably aren't [solutions]
<adeuring> good morning
<StevenK> henninge: No news means "It's terrible, redo from start" ? :-)
<henninge> StevenK: no, it looks good
<henninge> StevenK: Still a lot to read ...
<rvba> henninge: Hi, once you're finished with StevenK's branch, I'd appreciate if you could take a look at https://code.launchpad.net/~rvb/launchpad/dds-sampledata/+merge/53988
<StevenK> rvba: Julian approved that MP?
<rvba> StevenK: yes
<henninge> rvba: why would you want another review if it has already been approved?
<rvba> henninge: well ... I thought the procedure was to have it reviewed by someone outside the circle of people who discussed the branch in question
<henninge> rvba: good point
<rvba> henninge: but in this case, it's really tiny and does not involve code change (only sql in current-dev.sql) so it might not be worth you time
<rvba> s/you/your/
<henninge> rvba: yeah, there is really not much I can review here.
<rvba> henninge: all right then
<henninge> rvba: how did you create the sample data?
<rvba> henninge: I confess I did that manually (fiddling with the database), then I ran make newsampledata
<rvba> probably not the most clever way to do this I know
<henninge> rvba: well yeah, usually it's better to do that through the normal procedures (i.e. UI and scripts)
<henninge> rvba: actualy it's *best* to avoid creating new sample data at all.
<henninge> well, actually it's more about using it in tests, I guess.
<rvba> henninge: yes but the scripts to create those objects are in the works now
<henninge> which is not the case here, of course.
<henninge> I figured
<rvba> yes, I only modified the sampledata for local instances
<henninge> oh, right
<henninge> rvba: r=me in that case ;-)
<rvba> the goal is to populate a form on which I'll have some js work to do pretty soon, before the scripts to populate this with real data exist
<rvba> henninge: thx
<henninge> StevenK: Isn't there some existing infrastructure to test scripts so you don't have to do the subprocess dance?
<henninge> StevenK: what about "from canonical.launchpad.scripts.tests import run_script"
<danilos> henninge, good morning, one very complex branch for you when you can find 10 minutes: https://code.launchpad.net/~danilo/launchpad/mail-header-discrepancy/+merge/54172 :)
<henninge> Hi danilos!
<henninge> and "No" :-P
<henninge> (which means "Yes", of course)
<StevenK> henninge: Hm, wasn't sure about that. I can certainly look at changing it.
<henninge> StevenK: I'll give you an example in the review.
<henninge> see if that works for you
<StevenK> henninge: Awesome.
<danilos> henninge, excellent, I prefer "no" meaning "yes" than the other way around :)
<henninge> ;)
<danilos> henninge, and I love you :)
 * henninge blushes
<henninge> StevenK: review sent
<henninge> danilos: sorry, too complex to do in one OCR shift, you will hear back from me tomorrow night.
<danilos> henninge, I understand, thank you for trying anyway!
<wgrant> Erm.
<wgrant> Why is ~registry not a member of ~vcs-imports any more?
<lifeless> it was via launchpad-chr, which matsubara deleted for some reason a couple of weeks back
<wgrant> Ah. Can you readd?
<wgrant> ALthough I guess we then need a ~launchpad admin :(
<jelmer> doesn't ~canonical-bazaar own it now for some reason?
<wgrant> lifeless is an admin.
<wgrant> But adding a team to another requires confirmation from both ends.
<lifeless> sure, one sec
<lifeless> hangon, ~registry or ~launchpad into vcs-imports ?
<lifeless> wgrant: ^
<wgrant> ~registry is basically the semi-admin team now... it's basically ~launchpad + ~canonical-bazaar plus a few others. I would go with ~registry.
<wgrant> But it doesn't really matter.
<lifeless> huh, I own but am not in registry. shrug
<lifeless> anyhow, added.
<wgrant> You are in registry.
<wgrant> It's a display bug.
<wgrant> Thanks.
<wgrant> lifeless: https://lists.launchpad.net/registry/msg32632.html
<wgrant> Odd.
<lifeless> yes, that was sent out when he deleted launchpad-chr
<lifeless> the message is confusing because it doesn't describe the action taken
<wgrant> That makes very little sense.
<lifeless> that he deleted -chr, or that it took it out of the team ?
<wgrant> That is sent when a teammembership is deactivated. launchpad-chr deletion would not have done that.
<lifeless> deleting the team merges it to registry right? and deactivates all previous team memberships ...
<wgrant> Right. But it deactivates before the merge.
<lifeless> right
<wgrant> Hmm.
<wgrant> Ahh.
<henninge_> jtv: Hi!
<deryck> Morning, folks.
<henninge_> Hi deryck!
<jtv> Hi henninge!
<jtv> Just looking at the import queue, as it happens.
<henninge_> uh-oh
<henninge_> jtv: what's up with it?
<jtv> Nothing, that's the great thing.
<henninge_> cool
<jtv> I'm just emailing the uploader of an RT template about branch synchronization.
<henninge_> jtv: I was going to ask you for a review but now that deryck is here I can ask him , too. ;-)
<jtv> oh sweet :)
<jtv> Just out of interest, what are you working on?
<deryck> henninge_: sure, I can do a review.
<henninge_> jtv: still finishing of upstream sharing UI
<jtv> Ah
<jtv> Saw that you'd been working on thatâvery nice to know that we're getting some visibility into sharing.
<henninge> jtv, deryck: Here is the mp. I have to go out now for an hour so it might not be optimal for you, jtv, since you are towards EOD.
<henninge> jtv: but it's using POTemplateCollections! ;-)
<henninge> https://code.launchpad.net/~henninge/launchpad/devel-737422-remove-translationsharinginfo/+merge/54184
<jtv> henninge: yes, I'm EOD now and have to rush to a dinner appointmentâ¦  but cool stuff!
<henninge> jtv: enjoy it!
<henninge> deryck: thanks, that OCR is ignoring me ...
<deryck> henninge: heh.  np.
<jtv> henninge: dankeschÃ¶n, und bis morgen :)
<jtv> hi salgado!
<salgado> hi jtv!
<jtv> everything good?
<salgado> yep! and you?
<jtv> Great, thanks!  Narrowly avoided arrest over the two empty cartridges I brought home from Dallas.  :)  Gotta run now!
<salgado> that sounds like lots of fun. ;)
<cjwatson> I'd like to have a pre-implementation chat with somebody about adding support for .orig.tar.xz files in package uploads
<cjwatson> henninge-out: you're listed as OCR, but I guess not because you're out? :)
<wgrant> cjwatson: I did the last round of changes there, so I might be a reasonable person to talk to.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge, benji | https://code.launchpad.net/launchpad-project/+activereviews/
<gmb> benji: Morning. Can I get a review for https://code.launchpad.net/~gmb/launchpad/finish-it-bug-737648/+merge/54189 please?
<benji> gmb: sure
<gmb> Ta
<benji> gmb: done: https://code.launchpad.net/~gmb/launchpad/finish-it-bug-737648/+merge/54189
<gmb> benji: Thanks. WRT your point about tests, I can only say: Oh boy, you don't want to know.
<cjwatson> wgrant: OK; well, I just went through and grepped for bzip2, bz2, and orig.tar, and added xz to anything that currently handles bz2
<benji> heh
<cjwatson> wgrant: the changes were confined to archiveuploader, except for a comment change in soyuz/enums.py; does that sound reasonable?
<cjwatson> wgrant:
<cjwatson> http://paste.ubuntu.com/583299/
<wgrant> cjwatson: Do we need additional series restrictions on that? 3.0 is permitted in >= Karmic.
<wgrant> +        # each component, and can use gzip, bzip2, or compression.
<wgrant> missing 'xz'
<wgrant> It almost seems worth generalising the compression type check, but not quite, so that looks reasonable.
<cjwatson> missing> whoops
<cjwatson> series> hmm.  orig.tar.xz requires dpkg >= 1.15.5, so we probably want >= lucid here.
<wgrant> :(
<wgrant> Meh.
<wgrant> karmic can burn.
<wgrant> Worst case the buildds will explode and fail the build in bad ways.
<wgrant> Plus Karmic dies in a month, right?
<sidnei> lifeless or gary_poster, around by any chance?
<gary_poster> sidnei, hi
<sidnei> hey gary_poster
<sidnei> gary_poster, would you know from memory why the page performance reports are generated from tracelog instead of using the oops system?
<gary_poster> sidnei, yes.
<gary_poster> 1) oops system only has exceptional circumstances
<gary_poster> (we want all)
<gary_poster> 2) tracelog has inter-request numbers that no other output has (how long request has been waiting)
<gary_poster> mm, "no other output" == "no other output I know of" :-P
<gary_poster> 3) we were starting from existing similar work with the tracelog
<gary_poster> that's all I can think of sidnei
<sidnei> gary_poster, ok. we're considering running ppr for landscape, but therve suggests that we enable oops for all requests, which would address (1), and afaict, we're limiting the number of concurrent requests to the number of threads in haproxy, which would address (2)
<sidnei> i think we might still got with tracelog just because it requires less changes though
<gary_poster> sidnei, ack with #2, same for us; tracelog gives you visibility into Zope bits, which as you said, should be controlled by haproxy, but might not.  I think we are only looking at haproxy results too, tbh and afaik.
<gary_poster> (for #2 values)
<gary_poster> sidnei, I vaguely would like to share code with you all, but don't strongly care otherwise.  Another thing to consider maybe is cost of generating oopses.  tracelog is designed to be very fast; oops reports do a lot more, so I would expect them to be a lot more expensive.  Just guessing though, dunno.
<henninge> cjwatson: I am back
<sidnei> gary_poster, thanks!
<gary_poster> np
<cjwatson> wgrant: where is the 3.0 >= karmic check implemented?
<wgrant> cjwatson: A table, SourcePackageFormatSelection.
<cjwatson> henninge: thanks - I think I can continue with wgrant
<cjwatson> wgrant: if you don't feel bad about spuriously allowing xz in karmic, I don't either
<wgrant> cjwatson: So adding another is a fairly heavyweight thing.
<cjwatson> as you say, it will almost certainly simply fail to build
<wgrant> cjwatson: Sounds good.
<henninge> cjwatson: oh, that's cool. I did not read the backscroll ;-)
 * deryck reboots, back shortly.....
 * maxb idly wonders if there is a reason why anyone can give away projects to arbitrary launchpad users, but can't give away branches similarly
<jelmer> maxb: I don't think there is a reason..
<wgrant> It's possibly an impersonation risk.
<wgrant> (so is project ownership transferral, but that's a bug, like team memberships)
<danilos> mrevell, hi, can you perhaps come up with something that looks more like English for what I am trying to achieve here: https://pastebin.canonical.com/44947/ :)
<danilos> mrevell, what's going on is that we've got a X-Launchpad-Subscription header, *and* "Matching subscriptions: subscription1, subscription2" text in the body of the email for those poor gmail users
<dobey> "an X-Launchpad-Subscription header"
<wgrant> Can we please have a "My email provider is stupid" checkbox to declutter text for those of us with sensible filtering mechanisms?
<wgrant> It could even default to true for gmail.com accounts.
 * henninge relocates
<allenap> benji: Do you have time to review a mostly-javascript branch? https://code.launchpad.net/~allenap/launchpad/dd-initseries-bug-727105-architecture-picker/+merge/54173
<benji> allenap: sure
<allenap> benji: Thanks :)
<deryck> henninge: this is a nice refactor. Don't really have any questions about it. r=me
<henninge> deryck: thanks
<henninge> deryck: I have some failing tests, though ... :(
<deryck> henninge: oh, bummer
<mrevell> danilos, Will take a look now.
<danilos> mrevell, thanks
<deryck> henninge, abentley -- I'm having serious connection issues this morning and working with cable company to debug.... may not make standup time.
<abentley> deryck: I've got SkypeOut, so we can do it on POTS if that helps.
<deryck> abentley: I'm on the phone with cable company now, sorry
<abentley> deryck: cool.
<deryck> abentley, henninge -- sorry.  if you guys will just carry on without me.
<deryck> hi adeuring.  see ^^
<adeuring> deryck: ok, noted
<deryck> I may loose connection here some as I debug this, too.
<abentley> deryck: Sure, if that's what you want.
<henninge> deryck: ok, good luck with the cable company. ;)
<deryck> henninge: thanks
<abentley> leonardr: is there a version of testing._webservice.api_url which is suitable for production use?
<leonardr> abentley: i don't know. what exactly do you want to do?
<leonardr> i think in production you can convert the current browser request to a web service request and then use canonical_url
<abentley> I want to create a view which returns a json-serialized dict that I will use to initialize the javascript functionality of a page.
<abentley> deryck: This serialized dict would need to include the API urls of several objects.
<leonardr> abentley: if i understand you correctly, we have a mechanism for that already. let me look it up
<abentley> leonardr: cool.
<deryck> abentley: was that info for me, or just chatting with leonardr?
<abentley> deryck: for leonardr.  Sorry.
<deryck> np
<leonardr> abentley: the thing we have already is: you can put an object in the IJSONRequestCache and when the launchpad page is rendered, the json representation of that object will be available through lp.cache.objects (or something like that)
<leonardr> lib/lp/bugs/browser/bugtask.py:        IJSONRequestCache(self.request).objects['bug'] = bug
<leonardr> oh, even better
<leonardr> you can put the object in IJSONRequestCache(self.request).links['key']
<leonardr> and only the link will show up
<leonardr> see lib/canonical/launchpad/rest/me.py
<abentley> leonardr: you are a beautiful man.
<leonardr> abentley: i've always thought so
<benji> allenap: done (https://code.launchpad.net/~allenap/launchpad/dd-initseries-bug-727105-architecture-picker/+merge/54173)
<allenap> benji: Awesome, thank you :)
<jelmer> jcsackett: it looks like the instructions for vcs imports are outdated - sourceforge doesn't work with non-https at the moment
<jcsackett> jelmer: oh fantastic. :-P
<jcsackett> thanks for letting me know. :-)
<jcsackett> i'll update.
<jelmer> jcsackett: I guess technically that still means plain http is faster than https ;-)
 * jcsackett laughs.
<jcsackett> and the offending line is removed.
<leonardr> benji, i have a zope question
<benji> leonardr: hit me
<leonardr> take a look at lazr.restful metazcml.py
<leonardr> there's a 'register_webservice' option
<leonardr> er, action
<leonardr> the kind of thing you put in a zcml file
<leonardr> i added a 'webservice sanity checks' action to the end of it
<leonardr> which is supposed to run after the web service is fully registered
<leonardr> the problem is that in launchpad, register_webservice is called no less than 12 times
<leonardr> the sanity checks will only succeed if they are called _last_
<leonardr> i could add a separate sanity_checks option to go into the zcml file, but how can i guarantee that it's run after everything else?
<benji> hmm, that's a good question; let me look at it for a second
<leonardr> benji: the other problem is that during register_webservice i build up lists of the things that need to be sanity-checked. is there any safe place to store that information for another zcml action to pick it up?
<benji> leonardr: on that front a module global would appear to be your best bet; that approach would cause problems if ZCML processing was ever parallelized, but I bet lots of other implementation details would fall over too so doubt that will ever happen
<allenap> jml: Do you have time to review a very small testing-related branch?
<jml> allenap: how could I say no?
<allenap> jml: I hoped you'd say that :) https://code.launchpad.net/~allenap/launchpad/id-for-yui-test-cases/+merge/54229
<allenap> jml: I'm asking you because you'll be able to tell me if it's sane or not.
<jml> allenap: Looks basically sane to me. Two things come to mind.
<jml> allenap: first is that you should add a test for this.
<jml> allenap: second is that the ideal id() allows you to load the test based on the id. I don't know enough about how JS tests are run to know if that's the case here. It seems likely though.
<allenap> jml: Fair. This whole class is untested so I was hoping to sneak this one through too.
<jml> allenap: self.assertEqual(test.test_path, test.id())
<jml> allenap: there you go, I've written the test for you :P
<allenap> jml: Cheers :)
<jml> also, in random spam news, I may be entitled to receive Â£3750 for the accident I had. (I didn't have an accident)
<jml> allenap: looks like a very useful change to make
<allenap> jml: Cool. Thanks for looking at it.
<benji> leonardr: is it ok to require that all the register_webservice elements be in the same file?
<leonardr> benji: no, they've been split out so that every component (bugs, etc.) registers its own stuff
<benji> leonardr: the only other thing I can think of would be to respond to the process start event (i.e., do it outside of ZCML handling)
<leonardr> benji: can you point to some code i could copy? is the process start event a zope thing?
<benji> leonardr: yeah, the standard Zope thing to do is to fire some events on process start so you can do startup tasks; let me find an example
<abentley> leonardr: When I add something to IJSONRequestCache(self.request).objects, is that meant to appear in the HTML?  I don't see it, only LP.cache['context'].
<deryck> abentley: hey.  Working on your review.  Is lib/lp/translations/templates/translation_sharing_config.pt the template you used until the sharing info page landed?  Is it used in this branch now?
<benji> leonardr: lib/canonical/launchpad/configure.zcml line 92 has <subscriber handler=".webapp.initialization.handle_process_start" />
<benji> ...which is implemented in lib/canonical/launchpad/webapp/initialization.py
<abentley> deryck: yes.  Did I forget to remove it?
<deryck> abentley: yeah, it's still showing up in the diff on the MP at least.
<abentley> deryck: okay, removed now.
<deryck> abentley: cool
<leonardr> benji: ok, i'll mess around with it
<deryck> abentley: looking at the windmill test -- can self.client.click(xpath='//*[@id="branch-incomplete-picker"]/a') be written as self.client.click(id=u"branch-incomplete-picker") and work?
<abentley> deryck: I don't know, but don't you want to click a hyperlink rather than its surrounding div?
<abentley> Or span, I guess.
<deryck> abentley: I think Windmill is actually smart about this.  *I think* :-)  in other words, if it works, it's Windmill helping you and keeps you from relying on xpath.
<deryck> or maybe it depends on the css, i.e. if the link is block and fills the div.
<deryck> abentley: at any rate, it's a suggestion, not a requirement.  if it works, I'd use it and save the xpath.  but then you know my xpath is evil feelings. ;) :)
<deryck> abentley: and finally for this test, the waits.forElementProperty needs to supply a timeout argument.
<abentley> deryck: I prefer to be more precise.  If I could assign an id to the link, I'd do that.
<deryck> abentley: ok
<abentley> deryck: Sure.  But can we please default to suitable timeouts in the future?
<deryck> abentley: what do you mean?  The constants aren't suitable for you?
<abentley> deryck: No, then constants aren't the defaults.
<deryck> abentley: I'm not following.  There are no defaults.  Windmill requires you specify this with each waits statement, no?
<henninge> deryck: About the branch you reviewed earlier today.
<deryck> henninge: hi. yes?
<henninge> deryck: the tests failures I now see have their root in the fact that TranslationTemplatesCollection does not have an interface and therefore no security configuration.
<henninge> I get lots of "Forbidden Attribute"
<abentley> deryck: It appears that we never want the default behaviour and we're always specifying timeouts.  If so, we should fix the default behaviour to be the behaviour we usually want.
<deryck> henninge: ah, oops
<henninge> deryck: I could add the interface and such but the next branch will remove the need for that anyway.
<henninge> deryck: so I'd like to leave this branch as it is now (it's gotten quite big anyway) and simply not land it.
<henninge> The next branch will build on it and fix the issues.
<deryck> henninge: that's sounds fine
<henninge> cool, thanks
<deryck> abentley: what is the "default behavior" that we don't want?  I'm sorry to be so dense here.
<abentley> deryck: the default behaviour, according to you, is that there is no timeout.
<deryck> abentley: right.  and that's the way windmill is designed.  There is no default value for the various wait methods.  Are you suggesting we change the windmill waits methods to include some default value?
<abentley> deryck: yes.
<abentley> deryck: or provide wrappers that are more suitable for us.
<deryck> abentley: I could see providing wrappers, if that's helpful
<abentley> I think it would be.  The best way to get people to do stuff the right way is to make the right way the easy way.
<dpm> hey launchpadders, I'm trying to wrap up the timetable for https://wiki.ubuntu.com/UbuntuAppDeveloperWeek, and I've noticed that there are no Launchpad sessions yet. I think it would be great to have one or two to give LP more community exposure, and perhaps it would be interesting to talk about things like the integration features LP provides when developing an app, and he LP api. Has anyone got any other suggestions? Who'd be up for running a session?
<deryck> abentley: I need to eat lunch, but will return to this after that.
<abentley> deryck: cool.
<leonardr> benji: which package defines the 'subscriber' directive?
<leonardr> it's hard for me to find anything in the maze of includes
<benji> leonardr: zope.component, src/zope/component/zcml.py has ISubscriberDirective
<benji> and zope.component, src/zope/component/meta.zcml registers the directive
<leonardr> thanks
<leonardr> so i should do <include package="zope.component.meta" /> ?
<leonardr> i think <include package="zope.component" file="meta.zcml" /> is right, but i still get unknown directive...
<leonardr> i should have lunch and come back to this
<benji> leonardr: yeah the second one looks right to me; it in itself may need some other directives though (probably also from zope.component)
<dpm> jml, wold you be up for a talk during Ubuntu AppDeveloperWeek on using the Launchpad integration features when developing an application? ^^
<jml> dpm: maybe. I'm going to try to find someone else, but will do it if I can't.
<leonardr> jml: maybe benji could do it
<dpm> jml, awesome, thanks. You can just add yourself or the person doing it on https://wiki.ubuntu.com/UbuntuAppDeveloperWeek/Timetable or simply ping me and I can do it for you
<jml> dpm: yes.
<jml> leonardr: not a bad idea.
<jml> gotta run. will have questions later.
<jml> bye
<c10ud\> hello, any chance to get this fixed somehow? i'm trying to get launchpad clone my git repo, but it fails because of a submodule (which i dont need in lp, btw). here's the error: http://launchpadlibrarian.net/66911794/emesene-team-emesene-master.log
<deryck> abentley: review is done. r=me, with minor note about variables in the js test.
<abentley> deryck: cool.
<c10ud\> see #400805 and #402814 they have the same issue as me.
<_mup_> Bug #400805: Unhelpful error when importing a project with submodules <code-import> <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/400805 >
<_mup_> Bug #402814: Importing revisions with submodules is not supported <lp-code> <udd> <Bazaar:Confirmed> <Bazaar Git Plugin:Fix Released> <Launchpad itself:Triaged> < https://launchpad.net/bugs/402814 >
<abentley> deryck: Yeah, I did not deliberately create any globals, so I'll take another pass and try to catch them all.
<deryck> abentley: awesome, thanks.  I spotted lci and ci, I think.
<c10ud\> err.. *2009*, can you just print a warning or whatsoever and make the import succeed? i need it for launchpad recipes...
<abentley> deryck: yeah, and I also found "complete", "incomplete", and "link".
<deryck> well spotted
<abentley> deryck: any other thoughts on the code so far?
<deryck> abentley: looks good to me.  I think most of it we went over already.  It's shaping up nicely..
<deryck> abentley: I think it's hard to know perfectly how the rest will fit in with the other links/settings until you do it, but it looks like it should be an easy follow on for the test.
<deryck> s/test/rest/
 * deryck has testing stuck in the brain
<abentley> deryck: yes, I think so.
<abentley> deryck: I feel like at this point, I know everything I need to get stuff working, but I don't know what the usual idioms are.
<deryck> abentley: I think you have everything you need now, too. This reads like it was written by someone who writes js regularly.
<abentley> deryck: thanks.
<deryck> np!
<abentley> deryck: Do you know about the IJSONRequestCache stuff?  There's an example in bugtask.py
<deryck> abentley: no, I don't know anything about that.
<deryck> abentley: maybe new work from what thumper and leonardr did recently.
<abentley> deryck: okay.  I haven't got it working yet, but it sounds promising.
<leonardr> deryck: it's actually very old
<deryck> ah, ok
<abentley> deryck: it's a python-side way of getting values into the page cache, AIUI.
<deryck> right
<abentley> deryck: and presumably it would also be able to get that as a pure json response, though that may not yet be implemented.
<lifeless> sidnei: yeah, all that gary said
<lifeless> sidnei: I agree on oops cost: the more diagnostics, the more overheads
<sidnei> lifeless, thanks for confirming
<abentley> leonardr: is anything I stick in IJSONRequestCache.object supposed to appear automatically, or is there something I have to do to make it show?
<lifeless> morning all
<leonardr> abentley: putting it in there should be enough
<leonardr> benji: ok, i think there's just one bit of this i don't understand
<leonardr> here's launchpad:
<leonardr> @adapter(IDatabaseOpened)
<leonardr> def handle_process_start(ev):
<leonardr> where does that IDatabaseOpened come from? what should i have as the equivalent?
<benji> leonardr: that's the interface you're subscribing to; that one may work for you or there may be one that feels more like "the process is starting" than IDatabaseOpened, let me look
<benji> leonardr: IProcessStarting looks to be the one you want
<leonardr> benji: hm, looks like i need to add zope.processlifetime as a dependency
<benji> yep, that's where those guys live; it's essentially a one file package
<abentley> leonardr: it's not working.  failing test: test_cache_foo http://pastebin.ubuntu.com/583453/
<abentley> leonardr: hmm, perhaps it only works with logged-in users?
<leonardr> abentley: it should work if you have permission to see the object
<abentley> leonardr: the code is wrapped with condition="view/user|nothing"
<leonardr> abentley: possibly because the only link up to this point has been the 'me' link
<leonardr> abentley: i think we should un-guard that zcml and make it not print a link if the object is none
<leonardr> which file is that condition= in?
<abentley> leonardr: Only for the links array?  Okay, but why special-case None?
<abentley> leonardr: lib/lp/app/templates/base-layout-macros.pt
<lifeless> leonardr: hi, when you're finished with abentley, I'd like to learn a bit about the representation cache (where does it live, is it shared between appservers, that sort of thing)
<leonardr> lifeless: the representation cache is disabled because it turned out not to save us much time
<lifeless> leonardr: ah, ok
<leonardr> when it's turned on, i believe it lives in memcached
<lifeless> leonardr: great, thanks! thats all I needed :)
<leonardr> abentley: ok, i think the condition there is not to stop a crash when there is no me link, but to save time rendering the page for anonymous users
<leonardr> let's suppose we want to change that, and always display the links
<leonardr> in that case we would take out the conditional from the tal:cache tag, and  add one to the <script> tag
<leonardr> the new conditional would omit the <script> tag if $key was None
<leonardr> does that make sense?
<leonardr> or, if $key is None, we can print 'null' instead of fmt:api_url
<abentley> leonardr: Yes, that makes sense.
<leonardr> benji: the IProcessStarting event never seems to happen, either in my tests or when i start up launchpad for real
<benji> leonardr: hmm, I wonder if that's why LP was using IDatabaseOpened (or whatever it was)
<leonardr> i'll try the other event
<leonardr> benji: the other event isn't firing either! here's my zcml
<benji> leonardr: I grepped for IProcessStarting and found it in several places in LP
<leonardr> oh, wait
<leonardr> my zcml is obviously dumb
<benji> :)
<leonardr> benji: same result with non-dumb zcml:
<leonardr>     <subscriber
<leonardr>     handler="lazr.restful.metazcml.webservice_sanity_checks"
<leonardr>     for="zope.processlifetime.IProcessStarting"/>
<benji> leonardr: try using the event interface: IProcessStartingEvent
<leonardr> ah
 * benji tries to remember why both exist.
<bac> hi sinzui, i've got a question about DistributionSourcePackage
<bac> sinzui: it is marked as being an IStructuralSubscriptionTarget ... but i cannot find where that is supported in the UI.  you know anything about it?
<leonardr> benji: no difference
<benji> leonardr: I don't think that'll make a difference, IProcessStartingEvent is just a backward-compatible name for...
<benji> right
<benji> leonardr: how are you starting the process? the events sure do seem to work, that or we spend a lot of time wiring up subscribers for them that don't do anything: lib/canonical/launchpad/webapp/configure.zcml
<leonardr> benji: i am running 'make run'
<benji> that's simple enough
<benji> leonardr: I just put a pdb in one of LP's IProcessStarting handlers and did make run and got a pdb prompt
<leonardr> argh
<leonardr> i can't think about this anymore, i need to come back to it later
<leonardr> benji: is this something i could punt over to you? do you have time?
<benji> leonardr: sure, I have time to take a couple of swings at it
<leonardr> benji: ok...
<leonardr> benji: lp:~leonardr/lazr.restful/integrate-strict-versioning
<benji> looking
<leonardr> you're trying to make metazcml.py#webservice_sanity_checks() run
<lifeless> flacoste: what time is our call?
<flacoste> lifeless: now according to my calendar, but according to URC it's in 1h, i can do either
<lifeless> now is fine :)
<flacoste> s/URC/UTC/
<flacoste> let's talk now then
<leonardr> wallyworld_, are you around for a quick standup? i have to leave pretty soon for an appointment
<lifeless> https://code.launchpad.net/~lifeless/storm/with/+merge/52630
<benji> leonardr: is there a test that I can run or do I need to get an LP instance to use this version of lazr.restful?
<leonardr> benji: i don't know whether it should be running during any of the existing tests, but it's not
<leonardr> i'm sure it should be running during lp startup, and it's not
<leonardr> so i recommend working in an lp context
<benji> ok
<leonardr> i have to go to an appointment, but i'll stay online in case you find anything
<benji> leonardr-afk: the problem was that LP doesn't use lazr/restful/basic-site.zcml; I put the handler registration in lazr/restful/configure.zcml (which LP does use) and bin/run drops me into pdb
<wallyworld_> gary_poster: are you still online? I have a zope question
<gary_poster> wallyworld, fire away.  benji is great for this stuff too, fwiw.
<wallyworld_> gary_poster: thanks. i am writing a test and want to hook up an object modified event listener for object modification. afaict, these are not invoked when tests are run. i have some code, but it doesn't work. i'll pastebin the code
<gary_poster> wallyworld_, it depends on what is set up.  zope.event.notify is always called
<gary_poster> that is a wildly simple function
<wallyworld_> https://pastebin.canonical.com/44984/
<gary_poster> usually we have zope.component register a handler in zope.event
<gary_poster> which does the whole dispatch-by-classes-or-interfaces thing
<gary_poster> looking...
<wallyworld_> gary_poster: from what i can see, the this listener is not invoked in tests:
<wallyworld_>     <subscriber
<wallyworld_>         for="lp.bugs.interfaces.bugtask.IBugTask                 lazr.lifecycle.interfaces.IObjectModifiedEvent"
<wallyworld_>         handler="lp.bugs.subscribers.bugtask.notify_bugtask_edited"/>
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge, benji | https://code.launchpad.net/launchpad-project/+activereviews/
<gary_poster> wallyworld_, is this being run in a layer, or with some other kind of setup that sets zope.component up?  I suspect the first part (layer) is the most likely approach.  If you are not running in any layer and you don't have any special setup, that's probably the story
<wallyworld_> gary_poster:  i'm using layer=DatabaseFunctionalLayer
<wallyworld_> gary_poster: from what i can see from other usages of TestEventListener, this is sufficient
<gary_poster> I don't remember what layer does what, but I'd expect that to work.
 * gary_poster looking at code again
<gary_poster> yours, that is
<wallyworld_> TestEventListener comes from canonical.lazr.testing
<wallyworld_> gary_poster: i'd also be happy if i could make the actual event listener fire during the test ie notify_bugtask_edited
<gary_poster> I figured :-)
<jml> wallyworld_: are you importing zope.component.event?
 * wallyworld_ looks
<gary_poster> sadly, my brain juice has dropped to precipitously low levels...
<jml> wallyworld_: you have to import it to make zope event->adapter stuff work
<jml> it's bit me and others before.
<wallyworld_> jml: not importing that.
<gary_poster> that's the hook I was talking about, yeah.
<gary_poster> I'd expect DatabaseFunctionalLayer to import that
<wallyworld_> jml: will try that. if that's the solution, thanks * 1000
<gary_poster> if it does not, you should perhaps choose another layer
<jml> gary_poster: yeah. or FunctionalLayer.
<jml> gary_poster: the problem is that it can also bite production scripts
<wallyworld_> gary_poster: i've tried a few layers
<jml> (as it did memorably for the branch scanner at one point)
<gary_poster> because by doing that you are changing the environment for all subsequent tests
<wallyworld_> jml: DatabaseFunctionalLayer extends Functional layer so it should make no difference?
<gary_poster> and thus hiding problems for other people, possibly
<gary_poster> I'd try AppLayer as a smoke test
<jml> wallyworld_: not practically, it's just more appropriate. FunctionalLayer sets up all the Zope stuff.
<gary_poster> Well
<jml> wallyworld_: but at this point, the importing doesn't take place in any layer
<gary_poster> I guess I'd try what jml suggested as a smoke test
<jml> # This non-standard import is necessary to hook up the event system.
<jml> import zope.component.event
<gary_poster> no layer: yuck :-/
<wallyworld_> gary_poster: i'm doing this in a separate, new test class
<jml> wallyworld_: do the import. see if it works.
<wallyworld_> i've tried AppServerLayer
<jml> wallyworld_: layers are a red herring for now.
<wallyworld_> will do. trying now
<gary_poster> wallyworld_, the change is in global state
<gary_poster> jml, well, not a red herring for the proper fix IMO
<jml> (incidentally, I always hear that as "hewwing" when typed)
<gary_poster> heh
<jml> gary_poster: yeah. I guess ideally FLayer would set up the hook manually and then remove the hook when done.
<gary_poster> either that or have our code always set up the hook ourselves
<jml> hmm.
<jml> that's not a bad idea either.
<gary_poster> but putting it in a test setup is a bad idea
<gary_poster> well
<jml> although, it's got to be undoable
<gary_poster> not the best idea :-)
<jml> (as in, reversible)
<gary_poster> why do we need to reverse it?  no challenge, just no idea
 * wallyworld_ waits for disk to stop thrashing so test can run
 * gary_poster is also feeling very sleepy :-/
<jml> e.g. our logging code does some useful set up, but then it makes subsequent tests print stuff to stderr uncontrollably
<jml> because it mutates global state
<gary_poster> ah-ish
<gary_poster> well, if the global state is our expected underlying state then that's OK it seems to me
<gary_poster> mutating the global state after it has been set up is the bug bear
<jml> we have a non-trivial amount of code that doesn't use the site.zcml or exec_zcml_for_scripts
<jml> obviously it's the minority, but still
<gary_poster> if importing the module is idempotent it still seems not so bad
<gary_poster> but eh
<jml> indeed :)
<jml> I just got from training. time for a shower before my call w/ huwshimi
<gary_poster> I'm not usually one to run away at the sound of the factory bell, but I'm feeling pretty zoned.
<gary_poster> bye
<gary_poster> wallyworld_ are you set for now?
<wallyworld_> gary_poster: just finished rerunning test. no good but i'll keep looking. thanks for helping
<lifeless> flacoste: https://devpad.canonical.com/~lpqateam/ppr/lpnet/2011-03/daily_2011-03-14_2011-03-15/
<gary_poster> benji, you happen to have brain juice left to help wallyworld_?
<lifeless> flacoste: https://devpad.canonical.com/~lpqateam/ppr/lpnet/2011-03/daily_2011-03-16_2011-03-17/
<flacoste> lifeless: https://wiki.canonical.com/InformationInfrastructure/Review-LL-20110317
<benji> wallyworld_: what's up? (not much brain juice left here either, got a bit of a headache)
<wallyworld_> benji: i am trying to hook up an event listener in a test to be invoked when an object is modified, but i can't get it to be called
<wallyworld_> benji:
<wallyworld_>   https://pastebin.canonical.com/44984/
<benji> have you double-checked that the ObjectModified event is being fired?  is there another handler you can put a pdb in to check?
<wallyworld_> i an using TestEventListener. jml suggested that i needed to import zope.component.event but that didn't help in my case
<wallyworld_> benji: i'm not sure how to check that it is being fired. if i could get the actual notify_bugtask_edited listener to fire during the test, i would need to hook up a fake one
<wallyworld_> wouldn't
<wallyworld_> benji: it's eod for you so if your batteries are dead, that's ok, i'll try a few other things
<benji> wallyworld_: ok, a few parting shots: I don't know why the event wouldn't be fired in a test; you can put a PDB in zope.event.notify to see when events get fired (and you can enjoy the beauty that is the implementation of zope.event)
<wallyworld_> benji: will do. thanks.
<jml> huwshimi: hello
<huwshimi> jml: Morning
<jml> huwshimi: skype?
<huwshimi> jml: Indeed
<lifeless> jml: Want to catch up sometime this week?
<jml> lifeless: yes, but it might be tricky to find the time. Will get in touch when I'm off the phone.
<lifeless> jml: kk
<lifeless> wgrant: oem no longer use buildd-secret
<wgrant> lifeless: That was less slow than I expected.
<wgrant> Excellent.
<lifeless> flacoste: you has mail
<wgrant> lifeless: Are you going to QA your branches thing?
<lifeless> wgrant: I did
<lifeless> http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/files/head:/lib/lp/blueprints/model/specification.py is breaking for me
<lifeless> oh, the next one
<lifeless> sure, doing now - was on phone etc earlier didn't see the notification
<lifeless> frell, that was snappy
<jml> lifeless: my morning would be easier for me. evenings are kind of packed atm.
<lifeless> jml: I am generally up late tuesday
<lifeless> jml: would you like to talk tomorrow morning?
<jml> lifeless: sure.
<lifeless> lets do that
<jml> lifeless: 0900UTC ok?
<jml> lifeless: alternatively, I could do after the TL call, but I'll be on a bus.
<lifeless> its currently UTC 2230 ?
<lifeless> so in 10h30m ?
<jml> lifeless: yes.
<lifeless> that makes it 10pm here - sure, can do.
<jml> lifeless: ok.
<jml> lifeless: also, in case it helps, the UK moves to summer time this Sunday.
<lifeless> \o/ confusion
<jml> yeah.
<lifeless> I blame NZ for this whole disaster
<lifeless> ok, time for another timeout fix
<wgrant> lifeless: We have to look for "ever published" :(
<wgrant> lifeless: Because it's reasonable to file bugs on removed pacakgse.
<lifeless> is it?
<wgrant> For SRUs.
<lifeless> series specific
<wgrant> Until we can file directly on series, we can't forbid filing on the distribution.
<lifeless> so there is a conflict in requirements
<wgrant> A conflict in present requirements, or a conflict in requirements 5 years ago and now?
<lifeless> shrug, will think about it more when we finish with our critical bugs
<lifeless> wgrant: probably resolvable, definitely has a temporal component
<lifeless> 5 seconds - https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-bugfilters-stats
<lifeless> \o/
<wgrant> Excellent.
<lifeless> 5 seconds to render https://code.launchpad.net/~bryce/launchpad/lp-617698-forwarding/+merge/41003/+index
<lifeless> sinzui: is the person merge job configure to run ?
<lifeless> is anyone on the db-devel conflict?
<wgrant> lifeless: It's not already fixed?
<sinzui> I am
<lifeless> sinzui: cool, thanks
<sinzui> It was a little awkward but make schema says I am done
<lifeless> jcsackett: btw - bug 727023 needs some more examination
<_mup_> Bug #727023: Cve:+index timeouts <qa-ok> <timeout> <Launchpad itself:Triaged by jcsackett> < https://launchpad.net/bugs/727023 >
<wgrant> Huh, why is it not still spamming us, then?
<sinzui> lifeless: the merge-person-job is not configured to run. Nothing uses it yet
<lifeless> https://code.launchpad.net/~openerp-commiter/openobject-addons/trunk-extra-addons/+merge/6722/+index makes me sad - 312 queries
<lifeless> sinzui: we just deployed your commit to make things use it
<lifeless> sinzui: or is it still guarded in some fashion?
<sinzui> The ui does not use it. I just started that branch. My lat branch lets us query the state and fixed the db permission exposed when I constructed a sane test
<lifeless> sinzui: I'm talking about bug 728471
<_mup_> Bug #728471: Person:+delete timeouts : Add sanity checks to mergeAsync <merge-deactivate> <qa-ok> <timeout> <Launchpad itself:Fix Released by sinzui> < https://launchpad.net/bugs/728471 >
<lifeless> sinzui: if that helps explain my confusion
<lifeless> its so nice seeing bug pages render in < 2 seconds.
<lifeless> need to get that to < 1 second though
<sinzui> lifeless: sorry. look like I reversed the description of the bug I split
<lifeless> sinzui: heh, that would explain it
<sinzui> lifeless: I fixed the description to https://bugs.launchpad.net/launchpad/+bug/728471 That is what I tested for release
<_mup_> Bug #728471: Person:+delete timeouts : Add sanity checks to mergeAsync <merge-deactivate> <qa-ok> <timeout> <Launchpad itself:In Progress by sinzui> < https://launchpad.net/bugs/728471 >
<lifeless> sinzui: thanks
#launchpad-dev 2011-03-22
<lifeless> desperately seeking reviewers
<wgrant> lifeless: Where?
<wgrant> I don't see a lifeless review pending.
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-739799/+merge/54287
<wgrant> Oh, it didn't actually exist yet.
<wgrant> I see.
<wgrant> lifeless: Still 100ms? :/
<lifeless> wgrant: see the oops: 9000ms cold, new query is 900ms cold, 100ms hot
<wgrant> No thumper, and no StevenK yet I guess :(
<StevenK> Hmm?
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: wgrant | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> StevenK: https://code.launchpad.net/~lifeless/launchpad/bug-739799/+merge/54287
<lifeless> next thing is to permit bulk_load chainging
<lifeless> and we'll have a poor man persistence layer
<wgrant> Indeed.
<wgrant> I also need to look at a helper for collections.
<StevenK> wgrant: Done.
<wgrant> StevenK: Thanks.
<wgrant> StevenK, lifeless: https://code.launchpad.net/~cjwatson/launchpad/tar-xz/+merge/54215
<lifeless> wgrant / sinzui can you confirm jc is still on Cve:+index ?
<wgrant> lifeless: He mentioned it on the call this morning.
<lifeless> kk
<wgrant> So I think so.
<lifeless> sob
<lifeless> I don't want to touch 27 /   26  ProductSet:CollectionResource:#projects
<wgrant> It's not that bad.
<wgrant> Just needs a few things preloaded.
<wgrant> A couple of them are collections, though.
<lifeless> its a nuisance determining all the traversals being hopped to
<wgrant> Indeed.
<wgrant> We may want to fix lazr.restful.
<wgrant> To provide a hook.
<lifeless> productseries should be sufficient to make the timeout go away
<wgrant> That was my impression, but officialbugtag might be bad too.
<wgrant> StevenK, lifeless: Thanks.
<lifeless> sinzui:
<lifeless> Traceback (most recent call last):
<lifeless>   File "bin/sprite-util", line 54, in <module>
<lifeless>     url_prefix_substitutions={'/@@/': '../images/'})
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/services/spriteutils.py", line 94, in __init__
<lifeless>     css_template_file, group_name, url_prefix_substitutions)
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/services/spriteutils.py", line 101, in _loadCSSTemplate
<lifeless>     self.css_object = cssutils.parseFile(css_template_file)
<lifeless>   File "/home/robertc/launchpad/lp-sourcedeps/eggs/cssutils-0.9.6-py2.6.egg/cssutils/__init__.py", line 156, in parseFile
<lifeless>     return CSSParser().parseFile(*a, **k)
<lifeless>   File "/home/robertc/launchpad/lp-sourcedeps/eggs/cssutils-0.9.6-py2.6.egg/cssutils/parse.py", line 130, in parseFile
<lifeless>     return self.parseString(open(filename, 'rb').read(),
<lifeless> IOError: [Errno 2] No such file or directory: '/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/icing/style-3-0.css.in'
 * StevenK glares at run_script. Something looks to be running DELETE FROM FeatureFlag first
<lifeless> see jtv's recent landing?
<lifeless> StevenK: wgrant: any idea about this issue I have ?
<lifeless> or should I back 12638 out?
<wgrant> lifeless: Reproduced.
<lifeless> bah, not that rev. other rev
<wgrant> lifeless: We have another 3 hours left on buildbot.
<lifeless> no, actually, that rev
<wgrant> Let's debug slightly first.
<lifeless> its trying to build the style-3.0 regardless
<wgrant> Ah.
<wgrant> I wonder if bin/sprite-util didn't get rebuilt...
<lifeless> so the easiest fix is to rename it
<lifeless> indeed
<wgrant> Or touch something that buildout depends on.
<lifeless> removing it
<lifeless> and it fails to run
<lifeless> 'make' doesn't know that bin/sprite-util should be rebuilt
<lifeless> bin/buildout fixed
<sinzui> style-3-0.css.in should not exit /bin/buildout will fix the utils
<wgrant> lifeless: Fixed?
<lifeless> wgrant: made 'make' work
<sinzui> Looks like the buildout templates were not regenerated during the deploy
<lifeless> we need to land a dependency rule to rerun buildout
<lifeless> it may|maynot fail on deploy
<wgrant> lifeless: It shouldn't fail on deploy.
<lifeless> wgrant: we don't start with a clean tree
<lifeless> wgrant: do we explicitly run buildout?
<wgrant> Then why does it take 30 minutse to run buildout?
<sinzui> what
<sinzui> ?
<lifeless> bbr
<lifeless> brb
<wgrant> sinzui: Your change to sprite-util.in wasn't enough to cause buildout to be rerun.
<wgrant> sinzui: So bin/sprite-util is still old.
<sinzui> ./bin/buildout will remake the bin dir from the new bin templates
<wgrant> Right, but until then everyone's tree is broken.
<wgrant> Maybe I should just upgrade loggerhead.
<wgrant> Kill two birds with one stone.
<wgrant> Oh, no.
<wgrant> versions.cfg, not sourcedeps.conf :(
<sinzui> Mine wasn't. I think this is case of different setups. I branch and make new trees every tine
<wgrant> sinzui: I normally do, but sometimes run stuff from devel.
<wgrant> And as of yesterday I'm trying bzr-colo, so all my one tree is broken.
<poolie> thanks for trying it
<poolie> i do use it for lp development fwiw, but not as often as you will be
<sinzui> I stopped running from devel when we switched to eggs. It became unstrust worthy
<poolie> what happened?
<sinzui> I miss simple python and debs
<wgrant> sinzui: Indeed.
<wgrant> poolie: Makefile + buildout == confusion
<wgrant> Compounded by running with a single tree.
<poolie> sinzui, what's the alternative?
<poolie> to running from devel, i mean
<poolie> my frustration here is that i only touch lp stuff every 1-3 weeks and so every time i do, i need to fight with dependencies
<lifeless> so minimally
<lifeless> someone needs to mail the dev list
<lifeless> secondly, we should just fix Makefile to invoke buildout here
<sinzui> poolie: yes :( I run rocketfuel-setup the moment I start my day, merge and rebuild all my trees. I can review projects, check spam and answer questions, before all that work is complete some days
<poolie> wgrant, so the breakage is not specifically from bzr-colo?
<wgrant> poolie: No.
<wgrant> poolie: colo is working OK, but the fact that it shares trees means that our screwed up build system is more exposed.
<lifeless> jcsackett: I have a change to lib/lp/bugs/browser/cve.py that may trivially fix the CVE:+index bug, though I'm not sure if it is/isn't related
<lifeless> jcsackett: actually nvm, I was on (hopeful) crack
<lifeless> huh
<lifeless> sinzui: looks to me like there is only one user of ProductSet.search : the API.
<lifeless> sinzui: if I'm right, could make it siiiiimpler
<jtv> morning folks
<lifeless> wgrant: fyi, offiical bug tags may be an issue, but they are also locked behind a mixing behind a property hidden in a dilemma.
<lifeless> wgrant: e.g. TooHardBin
<wgrant> lifeless: Perhaps.
<lifeless> wgrant: certainly not low hanging fruit
<jtv> hi StevenK, are you here?
<sinzui> lifeless: I would expect ProductSetView to use IProductSet.search(). I see it uses IPillarNameSet. I think this is because users do not care care that we divide things into distros, projects, and project groups
<lifeless> sinzui: I had such expectations too
<lifeless> sinzui: I found a total of 1 use: bugalsoaffects
<lifeless> can has review? https://code.launchpad.net/~lifeless/launchpad/bug-727020/+merge/54290
<StevenK> jtv: I am now. What are you after?
<jtv> StevenK: I figured it out for myself now.
<StevenK> jtv: Nice work. :-)
<jtv> Well I still haven't solved my problem.  :)  The difficulty was finding out where Sources gets generated.  Turns out the answer depends on the archive purpose.  In my case it should be done by a-f.
<lifeless> StevenK: https://code.launchpad.net/~lifeless/launchpad/bug-727020/+merge/54290, if you would be so kind
<lifeless> wgrant: how is Archive:+index looking perf wise now, do you think ?
<StevenK> lifeless: Done.
<lifeless> thanks
<wgrant> lifeless: getBuild(Status)?Summar(y|ies)(ForSource(Id)?s(AndArchive)?)? is still being problematic.
<wgrant> Across +index, +packages, +copy-packages and +delete-packages
<lifeless> wgrant: want me to look ?
<lifeless> or I can fix ScopedCollection:CollectionResource:#person-page-resource
<lifeless> hmm, might do that one
<wgrant> lifeless: I'm looking.
<wgrant> Well, was looking before, and will be looking again in a while, I imagine...
<wgrant> lifeless: It needs to be rewritten to not execute four 300ms queries, basically.
<lifeless> wgrant: thats only 1.2 seconds
<wgrant> lifeless: The queries are at best 300ms. They're often much more.
<StevenK> jtv: Hi! :-)
<jtv> uh-oh
<jtv> yes?
<StevenK> jtv: Is there anything special I need to do to see feature flags in a script?
<jtv> StevenK: if it's a LaunchpadScript, no.
<jtv> If it's not a LaunchpadScript, you're still on your own.
<StevenK> I don't think it is. Can you point me at the scaffolding I need to set up?
<jtv> Surprisingly, you can find the answer in LaunchpadScript.  :)
<jtv> Have a look at lp.services.scripts.base: LaunchpadScript.run
<jtv> In a nutshell:
<jtv> from lp.services.features import install_feature_controller, make_script_feature_controller
<StevenK> Ah ha, it's a JobCronScript, which bases from LaunchpadCronScript, which is based off LaunchpadScript.
<jtv> install_script_feature_controller(make_script_feature_controller("identifying script name"))
<jtv> Well then there you go.
<jtv> The feature controller is installed only during "run."
<jtv> After that, it restores whatever controller was previously active (which probably means "none").
<jtv> If you call main directly in a test, it uses whatever controller the test had set up.
<StevenK> I call the cronscript directly using run_script
<jtv> Then it should just work (apart from ZCML taking #$%@ ages to parse during script startup)
<StevenK> It insists the feature flag isn't set
<StevenK> I shall worry about it after lunch
<jtv> StevenK: my branch landed only recently, so make sure it's included.
<StevenK> jtv: This is with latest devel merged in
<StevenK> jtv: Is it because the test set up uses a fixture?
<jtv> Uses a fixture for what?
<StevenK> self.useFixture(FeatureFixture({FEATURE_FLAG_ENABLE_MODULE: u'on'}))
<jtv> Does FeatureFixture actually change your feature-flag settings?  Because it'd have to do that to have any influence on your script run.
<StevenK> jtv: Running the cronscript test by itself with LP_DEBUG_SQL=1, it runs "DELETE FROM FeatureFlag"
<StevenK> The job gets added, then there is a pause, which is ZCML and such like, and then two statements get executed, the first is the delete, and the second is checking the feature flag which is now unset
<jtv> No idea where that comes from; does the FeatureFixture also INSERT your test flag (and do you commit before the script runs)?
<jtv> My impression was that the FeatureFixture is an in-memory trick only, in which case it's not going to do anything for scripts.
<StevenK> jtv: So this is using the testcase you added :-)
<jtv> How does that follow?
<StevenK> If FeatureFixture is in memory only, how do I go about enabling the flag?
<lifeless> its not in memory only AFAIK
<lifeless> it was discussed, not done.
<lifeless> IMBW
<lifeless> check the code
<StevenK> lifeless: So how can I tell what is running the DELETE?
<lifeless> LP_DEBUG_SQL_EXTRA=1
<jtv> I'm already running that
<StevenK> I was right, it is the fixture.
<lifeless> it should show a backtrace then
<StevenK> It's cleaning itself up before the script runs
<lifeless> StevenK: pastebin the code?
<StevenK> lifeless: It's in an MP: https://code.launchpad.net/~stevenk/launchpad/dsdj-runner/+merge/54159 ; I'm just pushing latest changes
<StevenK> Unless I cause the fixture to cleanup by running transaction.commit()
<StevenK> Anyway, lunch
<StevenK> (Finally)
<lifeless> grrrrr
<lifeless>     self._fd.close()
<lifeless> IOError: [Errno 28] No space left on device
<lifeless> trying to setup a test env
<lifeless> because sill factory triggers
<lifeless>   File "/home/robertc/launchpad/lp-sourcedeps/eggs/zope.sendmail-3.7.1-py2.6.egg/zope/sendmail/delivery.py", line 136, in createDataManager
<lifeless>     msg.close()
<lifeless>   File "/home/robertc/launchpad/lp-sourcedeps/eggs/zope.sendmail-3.7.1-py2.6.egg/zope/sendmail/maildir.py", line 136, in close
<lifeless> StevenK: looks like the fixture DELETE should happen at the end of the test, not before
<jtv> wgrant: I'm driving myself up the walls trying to find out why a-f isn't generating any Sources files for me.  One thing I notice is that there's no trace of binary packages in the setup that the test publisher has produced for meâ¦ is that normal?
<jtv> I do get a <distroseries>_main_source referring to an existing dsc fileâ¦ what else is needed?
<wgrant> jtv: What does the a-f conf look like?
 * jtv finds pastebin
<jtv> wgrant: I hope apt.conf is the one you needâ¦ http://paste.ubuntu.com/583629/
<wgrant> jtv: That's the one.
<wgrant> So, that should work. But I wonder if it's not working because you have no DASes.
<wgrant> I'm not sure anyone has tested that.
<jtv> So the test publisher doesn't set any up?
<wgrant> Not if you just makeDistroSeries, no.
<jtv> Also, the contents.header that that config refers to doesn't exist prior to the a-f run.
<wgrant> It's not designed to placate buggy a-f stuff.
<wgrant> Right, that's fine.
<jtv> Trying it with a DAS in place, just for the hell of itâ¦
<wgrant> It shouldn't make a difference, but you never know.
<jtv> I'm just getting more error messages out of a-f.  :)
<jtv> It now also says "could not open file" about the Packages files.
<wgrant> Could you pastebin the full output?
<wgrant> Or throw the branch my way?
<jtv> wgrant: the branch is lp:~jtv/launchpad/bug-55798 ; run ./bin/test lp.soyuz.scripts.tests.test_publish_ftpmaster -t publishes
<jtv> I've set it up to tar up directory contents in /tmp, though not all of that is committed (because I don't want it in my final branch)
<StevenK> lifeless: It looks to me that the commit is setting it off
<jtv> wgrant: another fairly wild guess: if none of this has been tested with distros other than Ubuntu, could something be reacting badly to the test distro not having (or populating) all the pockets that ubuntu has?
<lifeless> StevenK: the commit will commit all pending stuff
<lifeless> StevenK: which will include a cleanout of *prior* featureflags
<lifeless> StevenK: and should include an add of yours
<lifeless> StevenK: check the FeatureFixture code
<wgrant> jtv: Sorry, got distracted by uncontrollable anti-Microsoft rage.
<wgrant> Let's see.
<jtv> wgrant: never apologize for that.  It's Microsoft's fault and don't stop until they're gone.
<lifeless> argh
<lifeless> activemembers unions with adminmembers. why
<StevenK> lifeless: That doesn't explain what I see with LP_DEBUG_SQL_EXTRA. http://pastebin.ubuntu.com/583636/
<lifeless> StevenK: thats after the test ran
<lifeless> File "/home/steven/launchpad/lp-sourcedeps/eggs/testtools-0.9.8-py2.6.egg/testtools/runtest.py", line 154, in _run_cleanups
<lifeless> StevenK: so its irrelevant to you
<StevenK> Drat, pasted the wrong bit
<StevenK> lifeless: http://pastebin.ubuntu.com/583637/
<StevenK> lifeless: The traceback from the commit is half-way through the test
<wgrant> jtv: Ah
<lifeless> StevenK: still in *test* cleanups
<lifeless>   File "/home/steven/launchpad/lp-sourcedeps/eggs/testtools-0.9.8-py2.6.egg/testtools/runtest.py", line 154, in _run_cleanups
<StevenK> lifeless: So the code for the statement is before the statement, http://pastebin.ubuntu.com/583640/ shows the entire test, and you can plainly see it runs the clean ups after line 6 of the test
<lifeless> StevenK: this suggests that that commit is failing
<lifeless> StevenK: raising an exception, which triggers cleanups
<lifeless> StevenK: i promise you, you've left your entire test according the backtraces you are showing me
<lifeless> StevenK: if I was debugging this, I would break in with pdb about now
<LPCIBot> Project db-devel build #477: FAILURE in 5 hr 1 min: https://lpci.wedontsleep.org/job/db-devel/477/
<lifeless> argh
<lifeless> ----------------------------------------------------------------------
<lifeless> SELECT 1 FROM (SELECT PersonTransferJob.json_data, PersonTransferJob.id, PersonTransferJob.job, PersonTransferJob.job_type, PersonTransferJob.major_person, PersonTransferJob.minor_person FROM Job, PersonTransferJob WHERE PersonTransferJob.job_type = 1 AND PersonTransferJob.job = Job.id AND Job.status IN (0, 1, 4) AND (PersonTransferJob.minor_person = 251673 OR PersonTransferJob.major_person = 251673) LIMIT 1) AS "_tmp" LIMIT 1
<lifeless> ----------------------------------------------------------------------
<lifeless> (repeat for the batch size)
<lifeless> SELECT 1 FROM (SELECT PersonTransferJob.json_data, PersonTransferJob.id, PersonTransferJob.job, PersonTransferJob.job_type, PersonTransferJob.major_person, PersonTransferJob.minor_person FROM Job, PersonTransferJob WHERE PersonTransferJob.job_type = 1 AND PersonTransferJob.job = Job.id AND Job.status IN (0, 1, 4) AND (PersonTransferJob.minor_person = 251672 OR PersonTransferJob.major_person = 251672) LIMIT 1) AS "_tmp" LIMIT 1
<wgrant> jtv:   Ran 1 tests with 0 failures and 0 errors in 32.236 seconds.
<jtv> wgrant: yes, but no Sources files.  Check /tmp/var_archive.tar.gz.
<wgrant> jtv: It's empty, but it's there.
<wgrant> (because I fixed it)
<jtv> wgrant: you do know how to keep the tension up, don't you?
<wgrant> Indeed.
<jtv> How did you fix it, and why is it empty?
<wgrant> You had your paths wrong
<wgrant> config.archivepublisher.root on prod is /srv/launchpad.net/ubuntu-archive, not /srv/launchpad.net.
<wgrant> And what's all this PPA business?
<wgrant> lp:~wgrant/launchpad/jtv-bug-55798
<jtv> What is what PPA business?
<wgrant> Your test and cron.publish-ftpmaster talk about PPAs.
<wgrant> cron.publish-ftpmaster does not go near PPAs.
<wgrant> Oh, right, there was that alternate codepath that could never have worked.
<jtv> You speak in riddles.
<wgrant> -    ARCHIVEROOT_PARTNER=$PUBLISHER_ROOT/ppa/$DISTRONAME-partner
<jtv> All my test did was create some ppa directories that appeared to be needed for anything to run.
<wgrant> You added that.
<wgrant> Possibly because the original did this:
<wgrant> -    ARCHIVEROOT_PARTNER=/srv/launchpad.net/ppa/ubuntu-partner
<jtv> How is that different?
<wgrant> That branch has the minor issue of not making any sense at all.
<wgrant> So I removed it.
<jtv> What branch?
<wgrant> -if [ "$LPCONFIG" = "$PRODUCTION_CONFIG" ]; then
<wgrant> -    ARCHIVEROOT_PARTNER=/srv/launchpad.net/ubuntu-archive/ubuntu-partner
<wgrant> -    GNUPGHOME=/srv/launchpad.net/ubuntu-archive/gnupg-home
<wgrant> -else
<wgrant> -    # GNUPGHOME does not need to be set, keys can come from ~/.gnupg.
<wgrant> -    ARCHIVEROOT_PARTNER=/srv/launchpad.net/ppa/ubuntu-partner
<wgrant> -fi
<wgrant> That's how it is in devel now.
<wgrant> The second part of that is broken.
<jtv> You mean "branch" as in "if"?
<wgrant> Yes.
<wgrant> Sorry.
<lifeless> bah
<lifeless> wgrant: sinzui is gone, can you show https://bugs.launchpad.net/launchpad/+bug/739961 to him tomorrow ?
<_mup_> Bug #739961: person collections in API late evaluate PersonTransferJob <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/739961 >
<jtv> wgrant: Okay, but does this change really affect the Sources issue?  I can see how the _other_ change affects it.
<wgrant> lifeless: Sure.
<lifeless> review plox https://code.launchpad.net/~lifeless/launchpad/bug-727540/+merge/54293
<wgrant> jtv: I don't recall exactly. But the partner dists move was crashing until I fixed that.
<jtv> wgrant: your change does mean that ARCHIVEROOT becomes /srv/launchpad.net/ubuntu instead of /srv/launchpad.net/ubuntu-archive/ubuntu as it currently is, doesn't it?
<wgrant> jtv: Yes.
<wgrant> Er.
<jtv> Er.
<wgrant> Where does cron.publish-ftpmaster get this info?
<jtv> It duplicates what it finds in the lazr config.
<wgrant> So it should be fine, then.
<jtv> Which is one of the things that made this untestable.
<wgrant> Because it will see config.archivepublisher.root == /srv/launchpad.net/ubuntu-archive
<jtv> But that's different yet again.
<jtv> On development systems, by the way, it's all in /var/tmp/archive
<wgrant> Right.
<wgrant> But how is it different yet again?
<wgrant> Assuming that you give cron.publish-ftpmaster config.archivepublisher.root, it should behave the same on prod as the one in devel does.
<lifeless> StevenK: perhaps you could review https://code.launchpad.net/~lifeless/launchpad/bug-727540/+merge/54293 ? its very simple and has its diff now
<lifeless> StevenK: I'm going to grab dinner, after that I can perhaps pull your branch down and have a poke for you
<jtv> wgrant: except that the script duplicates it all in its variables.  You just said the production config said /srv/launchpad.net/ubuntu-archive, but you changed the script to use /srv/launchpad.net/ubuntu instead.
<jtv> This is all so confusing.
<StevenK> lifeless: So where is the eager loading?
<wgrant> jtv: The production dists tree lives in /srv/launchpad.net/ubuntu-archive/ubuntu/dists
<wgrant> jtv: The dev one traditionally lives in /var/tmp/archive/ubuntu/dists
<jtv> oic!
<jtv> So then you missed a change:
<jtv> PUBLISHER_ROOT="${PUBLISHER_ROOT:-/srv/launchpad.net/ubuntu-archive}"
<jtv> Right?
<lifeless> StevenK: _members(need_validity..) etc
<jtv> wgrant: in other words, the "ubuntu-archive" goes into the $PUBLISHER_ROOT instead of being added in ARCHIVEPARENT.
<jtv> And in fact PUBLISHER_ROOT and ARCHIVEPARENT are the same thing
<wgrant> jtv: I did enough to make the tests pass.
<jtv> (which the script previously hid by using $ARCHIVEROOT/..)
<wgrant> I didn't actually look at what changes you really made.
<wgrant> Enough to get the tests to pass with a sane path structure, that is.
<jtv> It's pernicious because there's no way of testing it.  Eventually it'll all go; right now I'm concerned with the "chalk outline" that the new version must fill.
<wgrant> Righ.
<jtv> Rhymes with "Sigh."  :)
 * jtv looks up production config
<wgrant> Also note that Julian's config work will change how this works, but the values should stay the same.
<wgrant> Just your tests will need to change, if they decide to override configs.
<jtv> They don'tâwouldn't work for external script runs.
<wgrant> Well, it can.
<wgrant> We have facilities for that now.
<wgrant> Indeed you probably should.
<wgrant> (eg. the librarian and DB fixtures push config overlays then write them to disk)
<jtv> That may become necessary later as we move this script to python and create more appropriate tests.  Then again, at that point it won't involve script runs any more so we should be able to do more in-memory.
<jtv> wgrant: you were right by the way: the archivepublisher's "root" setting on production is /srv/launchpad.net/ubuntu-archive.
<wgrant> jtv: Yup.
<jtv> Thanks for helping me with this.  I'm reluctant to drop the ppa stuff until I've got something of a proper test hammered out, but it'd be good to streamline things then.
<wgrant> Right.
<jtv> Food.
<StevenK> Right, something odd is going on, since I've added the flag manually, and the cron script still can't see it.
 * StevenK doesn't understand feature flags
<lifeless> StevenK: does your script install a flag handler? [jtv's work was in this area]
<StevenK> lifeless: jtv added it to LaunchpadScript's run() method from what I can see, and my script (through 2 layers of indirection) bases off it
<StevenK> lifeless: Unless I'm wrong. What's a flag handler?
<lifeless> sadness
<lifeless> 1112 timeouts
<lifeless> StevenK: featurecontroller or whatever
<StevenK> install_feature_controller(make_script_feature_controller(self.name))
<StevenK> Is that enough?
<StevenK> lifeless: ^
<lifeless> yeah, should be
<lifeless> StevenK: whats the feature controller present when your code does a flag lookup ?
<StevenK> How do I check that?
<lifeless> print ?
<StevenK> Sure, but print *what* ? :-)
<lifeless> lp.services.features.mumble
<StevenK> Guessing get_relevant_feature_controller(), which gives None
<StevenK> lifeless: ^ Which is likely sub-optimal
<lifeless> StevenK: that is indeed likely to be a problem
<lifeless> StevenK: is your script threaded?
<StevenK> lifeless: I have no idea, but I doubt it.
<lifeless> timeouts top to bottom - ugh, ugh, ugh, fixed fixed fixed ugh ugh in-progress ugh ugh ugh ugh
<lifeless> StevenK: so, keep spattering prints around - like after install_feature_controller - until you can see what goes wrong
<huwshimi> ok seriously did someone rename style-3-0.css.in?
<lifeless> huwshimi: run bin/buildout
<StevenK> lifeless: So directly after install_feature_controller, it's still None
<lifeless> so, either you're looking in the wrong place, or that method is failing ;)
<StevenK> I don't get what install_feature_controller is supposed to do
<jtv-eat> StevenK: it's a simple setter for the feature controller, though it only applies to the running thread.
<StevenK> I'm close to ripping this out for now
<huwshimi> lifeless: That didn't help. I think it has been renamed.
<StevenK> I don't understand it, and it seems quite opaque
<lifeless> huwshimi: delete bin/sprite-util
<lifeless> huwshimi: then run it
<StevenK> Or make clean; make
<jtv> StevenK: let's start by establishing that you're talking to the right feature controller.
<StevenK> But that's a bigger hammer
<jtv> StevenK: if get_relevant_feature_controller returns None right after install_feature_controller, then are you sure you didn't pass it a None?
<StevenK> 'install_feature_controller(make_script_feature_controller(self.name))' is the call
<StevenK> Which is the line you added to LaunchpadScript
<jtv> Yes
<jtv> Oh actually you don't need to install it; make_script_feature_controller calls install_feature_controller.
<jtv> After which in fact it returns None.
<StevenK> Oh, so it's broken
<StevenK> :-P
<jtv> Yup.
<jtv> Fixing.
<StevenK> It's fine, I can do it
<jtv> You can replace install_feature_controller(make_script_feature_controller(self.name)) with just make_script_feature_controller; or better yet,
<jtv> make make_script_feature_controller return the controller instead of installing it.
<StevenK> We don't need the controller, so if it installs it, that's enough
<StevenK> jtv: This was the failing test for me, so I can just fix the bug in my branch if you wish
<jtv> True, it's just nice to export just a single function to set the thing.
<jtv> Needs a test also.
<StevenK> Which you're working on?
<jtv> Yes
<jtv> StevenK: pushing fix
<jtv> bug 739986
<_mup_> Bug #739986: LaunchpadScript fails to install feature controller <derivation> <feature-flags> <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/739986 >
<huwshimi> lifeless: just fyi that file was renamed
<jtv> StevenK: care to review?  https://code.launchpad.net/~jtv/launchpad/bug-739986/+merge/54299
<StevenK> jtv: r=me
<jtv> thanks
<jtv> and sorry for theâ¦
<jtv> oh wait I already said that didn't I
<StevenK> A few times :-P
<jtv> possibly not enough :)
<jtv> This is something that still bothers me about our testing.  Quis custodiet ipsos custodies?
<StevenK> jtv: You'll toss that through ec2?
<jtv> I was considering it, yesâmeanwhile you could just merge it in
<StevenK> Yup
<lifeless> huwshimi: it was, but the reason you had a problem was due to a stale built file
<lifeless> huwshimi: if you fixed it locally by renaming again, you've done it wrong
<huwshimi> lifeless: I haven't renamed it, I was just getting conflicts within the file and it was renamed and now I'm in a world of hurt with strange diffs, but I think it's completely unrelated to the buildout stuff. I just happened to run into it at the same time.
<huwshimi> lifeless: I wasn't really expecting to come across the changes and renaming of the files without some kind of warning (like an email)
<jtv> heh.  wgrant, was that you who wrote "I do not care about sources"?  Because that seems to be what's bothering a-f now.  :)  Not sure that we need to fix that though, as long as a Sources file gets generated.
<wgrant> jtv: I don't recall saying that, but I may have.
<wgrant> Anyone want to review https://code.launchpad.net/~wgrant/launchpad/restore-archivepublisher-config/+merge/54301?
<jtv> It just sounded like you somehow.  :)
<wgrant> Probably.
<jtv> I suppose I owe you one, so sure.
<wgrant> It's not very difficult :)
<spm> wgrant: hit jtv for Cake. that way you can start paying us back. /unsubtle-hint
<wgrant> jtv: (that was cprov)
<jtv> Cake?
<wgrant> But yes, it does sound like something I'd say.
<jtv> wgrant: I just approved without comment.  May be a first.
<jtv> (I did check the bug reference)
<StevenK> jtv: I'm tossing my branch at ec2. Which means your bug fix might inadvertly land
<jtv> StevenK: oh woe is us
<wgrant> jtv: Thanks.
<jtv> And in case the sarcasm didn't drip quite where everyone can see it, I'll add what BBC teletext page 888 does: "(!)"
<jtv> (I hope I'm not so much older that I now have to explain what teletext is)
<StevenK> I'm so tempted to ask, now
<wgrant> Even I know what Teletext is.
<wgrant> although only because I didn't know what a button on the old TV did.
<jtv> StevenK: there you go, ask william just like you would with any soyuz question.
<jtv> The BBC has excellent subtitling on page 888.
<jtv> Someone hums half a bar of an ill-remembered melody, off tune, in an anvil factory and the subtitles will tell you it's the 3rd movement from Vivaldi's opus #128 etc.
<wgrant> jtv: That is surely crucial to the plot.
<jtv> Surely.
<jtv> They also use different colours for different characters in films, as I recall.
<jtv> It was a great help in learning English.
<lifeless> huwshimi: renames are normally pretty painless; also having short lived branches avoids a lot of pain
<jtv> StevenK: in replacing cron.publish-ftpmaster, do I need to call getPubConfig on every archive that the script is to work on?
<wgrant> publish-distro will do that for you.
<wgrant> But the dists musical chairs probably needs it run before publish-distro.
<wgrant> But youi'll be moving that into publish-distro soon.
<jtv> I expect I'll probably need to do some setup before then, yes
<jtv> Will I?
<wgrant> I hope so.
<wgrant> It belongs there.
<jtv> For now I've been far too busy getting it to run at all.  Probably wouldn't have been wise to start hunting for cleanup opportunities too soon.  :)
<jtv> My next step is to pythonify the shell script as-is.
<wgrant> I'm not sure that Pythonifying it directly is particularly valuable.
<jtv> It's just one of the steps.
<jtv> It means that I can use config instead of shell variables that must match the config, for one.
<jtv> It also means that I can use dedicated temporary directories again, which helps avoid pollution by previous test runs.
<jtv> And of course it means I can run unit tests and get the results the same day.  :)
<LPCIBot> Project windmill build #85: FAILURE in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill/85/
<lifeless> have I mentioned how much I love that Storm and sqlobject result sets are incompatible with each other ?
 * jkakar hugs lifeless
<jkakar> lifeless: Isn't the problem just that the SQLObject one is incompatible with Storm? ;b
<lifeless> jkakar: they both ship in the storm tarball ><
<jkakar> lifeless: I know, I was just being facetious... sorry, coffee hasn't set in yet.
<lifeless> jkakar: it makes migration tricky
<lifeless> jkakar: :)
<lifeless> jkakar: end of long day for me, so the reverse...
<jkakar> lifeless: Heh
<spm> heya jkakar!
<jkakar> spm: Hi! :)
<jtv> jkakar: so you miss us after all, do you?  :)
<jkakar> jtv: I'm having fun at the new gig, but yes, I do miss you. :)
<jtv> :)
<rvba> wgrant: Hi, I'd appreciate a review of https://code.launchpad.net/~rvb/launchpad/db-dds-fix-filter-form-add-parent-name/+merge/54239
 * lifeless wonders if http://dev.launchpad.net/ReviewingCodeImports?action=diff&rev1=9&rev2=10 was intended
<lifeless> wgrant: I need some blurb aboutthe change for oem
<wgrant> rvba: Done.
<wgrant> lifeless: :(
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project windmill build #86: FIXED in 43 min: https://lpci.wedontsleep.org/job/windmill/86/
<lifeless> wgrant: I can just say 'xss hole and escaping blah', but perhaps a little more accuracy would help?
<rvba> wgrant: thx
<lifeless> wgrant: if you're doing up an incident report, just make a clear summary, and I'll copy that across - avoids you duplicating effort
<lifeless> lib/lp/bugs/tests/../doc/bugnotification-sending.txt", line 1260, in bugnotification-sending.txt
<lifeless> ^ db-devel failure.
<lifeless> ring any bells?
<lifeless> ah
<lifeless>    - Matching filters: Allow-comments filter
<lifeless>     ?          ^ ^ ^^
<lifeless>     + Matching subscriptions: Allow-comments filter
<lifeless>     ?          ^^^^^^ ^ ^^^
<lifeless> someone didn't use ec2 :)
<lifeless> (or ran into a landing race I guess)
<wgrant> lifeless: Merge issue. Fix already landed.
<wgrant> lifeless: https://wiki.canonical.com/IncidentReports/2011-03-22-LP-unescaped-json, but the summary is not very good.
<henninge> jtv: Hi!
<jtv> hi henninge
<henninge> jtv: I picked up bug 605924 as you may have noticed. I just commented on what I plan to do.
<_mup_> Bug #605924: Clean up IHasTranslationTemplates <lp-translations> <tech-debt> <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/605924 >
<henninge> if you are interested.
 * jtv has a look
<jtv> henninge: looks goodâI agree that the hard part is to find all uses and make sensible decisions for them.  The tests probably aren't going to find all of those, but that's no excuse for ignoring the issue forever.
<henninge> jtv: well, I am picking it up because it is hindering me in using this properly.
<jtv> That's as good a reason as any other, isn't it?  :)
<henninge> well, it certainly raises its priority ... or urgency? whatever
<lifeless> wgrant: cool; are you forcing a build?
<wgrant> lifeless: It's already building again.
<lifeless> cool
<wgrant> Because there was the testfix rev in the queue.
<adeuring> good morning
<jtv> morning adeuring
<jml> lifeless: hello
<adeuring> hi jtv
<lifeless> jml: hi
<jml> lifeless: skype?
<lifeless> jml: aye
<jml> lifeless: hello, I can now hear you
<jml> lifeless: Total: 231 (+42, -48 since 2011-03-15)
<wgrant> lifeless: Thanks.
<lifeless> wgrant: de nada, thank you!
<poolie> is staging down?
<poolie> i'm getting a 504 gateway timeout trying to make api calls against it
<poolie> it might be just me
<wgrant> staging is down, yes.
<wgrant> Fix landed.
<poolie> thanks
<wgrant> We might be able to get it going again mid-morning tomorrow.
<wgrant> Maybe.
<poolie> is there any point filing a bug saying the api error is less clear than the web ui one?
<lifeless> nope
<poolie> this seems like another reason to make lplib default away from staging
<bigjools> morgen
<wgrant> Morning bigjools.
<wgrant> bigjools: I've restored those archivepublisher config schema entries in db-devel, so the configs work again.
<bigjools> wgrant: why?
<wgrant> bigjools: staging broke.
<bigjools> yes, it was broke before and Chex cowyboyed it
<wgrant> Also because having to merge configs in lockstep with deployments sucks.
<wgrant> We're going to have to cowboy it every time it updates...
<bigjools> grar
<wgrant> Anyway, fixed now.
<bigjools> I detest this config system
<wgrant> I just have to remove the entries right after the next deployment.
<bigjools> yes, but it's hassle and pain and annoyance
<wgrant> It is.
<StevenK> Bloody hell. If it isn't the backup killing garbo-hourly, it's a vacuum
<bigjools> wgrant: nice security fix earlier :)
<wgrant> bigjools: Yeah :/
<bigjools> you'd think it having been broken once before there'd be a test case
<wgrant> I plan to add one tomorrow.
<wgrant> Also I might upgrade our simplejson and fix it properly.
<wgrant> Since IE prevents any sensible fix.
<bigjools> heh, that explains your FB update earlier
<wgrant> I mean, XHTML was only standardised 11 years ago.
<poolie> wow, hooray for bug expiry
<LPCIBot> Project db-devel build #478: STILL FAILING in 5 hr 4 min: https://lpci.wedontsleep.org/job/db-devel/478/
<lifeless> jml: 11 /   18  Product:+code-index
<lifeless> jml: https://code.qastaging.launchpad.net/++profile++show/launchpad
<lifeless> jml: 06699-12526@SQL-launchpad-main-slave SELECT DISTINCT BugBranch.branch FROM Bug, BugBranch WHERE BugBranch.branch IN (24637, 34755, 34756, 352362, 423464, 423252, 438596, 423710, 438533, 28862, 421710, 425429, 425609, 424278, 422318, 423317, 423707, 423704, 422533, 421039, 423618, 422191, 423555, 423341, 423342, 423273, 423359, 423465, 419565, 423403, 423337, 30466, 421067, 422299, 421762, 422907, 422741, 422712, 319807, 422
<lifeless> is 6000ms
<lifeless> jml: https://lp-oops.canonical.com/oops.py/?oopsid=1906G1635#repeatedstatements
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: gmb | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> jml: https://lp-oops.canonical.com/oops.py/?oopsid=1906G1635#statementlog - query 39 and 40 appear identical to me
<lifeless> jml: both are 'bmps where the source is for this product and both branches are visible
<lifeless> jml: I think we could get a little fuzzy and use 'the source is for this product and the source is visible' - because you cannot /create/ a bmp targeting a branch you cannot see:
<lifeless> some person subscribed to the source might see the *count* of a bmp they cannot otherwise see
<lifeless> but would that matter? anonymous users wouldn't get wrong data, nor would authenticated users with no access to the source branches
<lifeless> doing that makes the query run in 70ms
<sladen> sorry to bother people---I've been searching for about 15 minutes and haven't turned it up yet
<sladen> I've had an email passed to me about visual/side-by-side diffing in code-hosting of non-code
<sladen> along similiar lines to  https://github.com/cameronmcefee/Image-Diff-View-Modes/commit/8e95f70c9c47168305970e91021072673d7cdad8
<sladen> I remember this being discussed at length ~3 years ago, but I haven't turned up any blueprints, mailing list posts or even bugs against launchpad itself
<jml> lifeless: looking...
<sladen> ah  https://answers.launchpad.net/bzr/+question/121049
<sladen> "Can Bazaar meet requirements for game-development workflows?"
<lifeless> sladen: yeah, discussed on canonical-launchpad and canonical-bazaar too:)
<sladen> lifeless: my apologies.  Guess I need to join those to read the archives
<lifeless> jml: actually, I've got a 130ms plan
<lifeless> jml: with half the cost estimate
<jml> lifeless: nice
<lifeless> https://bugs.launchpad.net/launchpad/+bug/736008
<_mup_> Bug #736008: Product:+code-index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/736008 >
<deryck> Morning, all.
<henninge> danilos, jtv: Any idea why we have this in our code (for productseries and distroseries) ? http://paste.ubuntu.com/583755/
<henninge> danilos, jtv: I mean, why override the usage of the parent  object?
<jtv> ah
<henninge> Asside from the fact that we get a nice loop here
<henninge> because the template cound is dependend on the usage ...
<jtv> Ah, that's funâ¦
<jtv> In any case it seems like a strange fallback.
<jtv> It was decided at the time that there is basically no usage as long as there are no templates, and now we do allow that again.
<henninge> jtv: I understand the "usage" more like an expression of the maintainers intend.
<jtv> That was historically the idea behind official_rosetta, yes.
<henninge> well, the question is what are the consequences of the usage value?
<jtv> But in UI terms, I thought the idea had shifted (as part of the cross-app "usage" renovation) to "there is a stated intent and there is a usable setup."
<henninge> ok, but this is doing "a stated intend OR a usable setup"
<henninge> which is strange.
<henninge> I'd rather have LP say "The user wants to use launchpad for translations but has not yet configured it fully."
<henninge> Also, the presence of templates does not imply "LAUNCHPAD" usage, it might also be external templates being mirrored in.
<henninge> through a branch
<henninge> which is actually a common setup we are trying to achieve for upstream message sharing.
<jtv> I agree with your desire, but I'd talk to sinzui andâ¦ was it jcsackett who re-did the usage flags?
<henninge> ok
<henninge> thanks, jtv
<henninge> jtv: oh, I just saw that there is wrong line in my paste
<jtv> ?
<henninge> line 5
<henninge> I hope you were not discussing that? ;-)
<jtv> Now I have to look againâ¦
<henninge> Here is the correct version that reflects the current code: http://paste.ubuntu.com/583761/
<jtv> henninge: oh, that was so weird my eyes just skipped over it.
<henninge> phew
<henninge> danilos: ^ in case you read the backscroll
<jtv> henninge: and "usage" is "self.usage"?
<henninge> argh
<henninge> jtv: this is the correct paste. really now. http://paste.ubuntu.com/583762/
<henninge> usage = self.product.translations_usage
<henninge> sorry for the confusion
<jtv> Ah, that was the sort of line my eyes assumed must be there in previous versions.  :)
<benji> leonardr: did my info help with your start up event problem?
<leonardr> benji: yes, thanks!
<benji> good
<jcsackett> henninge: i see that jtv recommends talking with me and/or sinzui? something about translations_usage?
<bigjools> TypeError: getPublishedSources() got an unexpected keyword argument 'status'
 * bigjools blinks
<abentley> leonardr: enabling the IJSONRequestCache.objects functionality for anonymous users causes xx-bug-obfuscation.txt to fail.  I think this mechanism may be too revealing.
<leonardr> abentley: what's the failure?
<abentley> leonardr: http://pastebin.ubuntu.com/583778/
<leonardr> abentley: so the problem is not that None gets into the cache, it's that something gets into the cache that shouldn't
<abentley> leonardr: yes.
<leonardr> i don't think "don't show the cache at all" is a winning long-term strategy. that cache is becoming more and more important
<leonardr> the code that populates the cache needs to learn some discretion
<leonardr> do you agree?
<abentley> leonardr: no, I don't.
<abentley> leonardr: I think it suggests that we should be specific about the data we pass on.
<leonardr> you may not agree with me, but i agree with you
<abentley> leonardr: rather than passing whole objects.
<abentley> leonardr: We should just pass the specific attributes we need.
<bigjools> :q
<leonardr> if we had the expand/filter system in place we could exploit that
<leonardr> what attributes do you want to pass?
<abentley> leonardr: For ProductSeries, title, web link, autoimport status, project link.  For Branch, unique_name, web link.  For Project, translations_usage, title, web link.
<leonardr> abentley: i think you may have uncovered a larger bug. presumably the anonymous user is seeing the uncensored email address in the representation of the bug
<leonardr> what happens if the anonymous user just requests that bugtask through the web service?
<abentley> leonardr: no idea.
<leonardr> abentley: i'm going to speak with the authority of someone who won't be working here next week :)
<leonardr> putting entire objects into the cache when you don't need the whole thing is inefficient, but it should not cause data breaches
<leonardr> there's something wrong with the way the cache is populated that's causing xx-bug-obfuscation to fail
<abentley> leonardr: the email address is appearing in the bug description.
<leonardr> right. but why is launchpad giving an uncensored bug description to an anonymous user? either the censoring isn't happening when a json representation is requested, in which case the web service as a whole is leaking data
<leonardr> or when we make an internal request for the json representation of the bug, we are for whatever reason escalating the user's privileges
<leonardr> so they see an uncensored version
<leonardr> either way, refusing to show the cache at all is just papering over the problem
<abentley> leonardr: if the privileges are being escalated, they're being escalated after the tal condition used to be evaluated, which is very late.
<abentley> leonardr: Well, I suppose they could be escalated and then dropped by the time the tal is evaluated.  But the template knows we've got an unauthenticated user.
<henninge> jcsackett: Hi, yes about its intended meaning.
<henninge> jcsackett: I understand the usage flags to reflect the intention of the maintainer of how to use that particular part of LP for their project.
<jcsackett> henninge: right. as i recall, we had some confusion of what best reflected that for translations, as translations has quite a few more "nobs" than other parts of launchpad.
<jcsackett> at least when we initially put the enums into use, translations used the enums plus some other information to determine what would be showed.
 * jcsackett doesn't know what's become of that since.
<henninge> jcsackett: I was particularly surprised to find this in the code http://paste.ubuntu.com/583762/
<henninge> jcsackett: ok, I guess I can feel free to redefine that, then ... ;-)
<dpm> hi jcsackett, bug 740153 looks more like a LP Translations bug to me (possibly related to message sharing?) than a synaptic bug. Shouldn't it be reassigned to LP?
<_mup_> Bug #740153: Translations skewed for synaptic <synaptic> <ubuntu> <upgrade> <NULL Project:Invalid> <synaptic (Ubuntu):New> < https://launchpad.net/bugs/740153 >
<jcsackett> dpm: that could be. i may have misunderstood the complaint.
<dpm> jcsackett, the reported was saying that the translations in LP were overwritten by others done a long time ago, which I'd guess couldn't have happened in Synaptic itself, but in LP
<jcsackett> dpm: ah, yes. okay, that should be targeted back to LP.
<jcsackett> henninge: correct. there was some ongoing work on translations when we were phasing that in (perhaps recife?) that was going to require changes to the translations_usage property.
<jcsackett> does that jive with what concerns you?
<henninge> jcsackett: possibly.
<henninge> jcsackett: I am trying this out here to see what happens.
<dpm> jcsackett, would you mind reassigning it to LP when you've got a minute? I don't seem to be able to make it, I get a "Too many matches. Please try to narrow your search." when trying to change the product
<jcsackett> dpm: i'm getting the same issue. :-P i'll reassign it soon.
<dpm> jcsackett, ah, and I thought you launchpadders had superpowers to overcome the issue :P
<jcsackett> dpm: we have many superpowers. in this case, the super power of being very patient with launchpad may be most applicable. :-)
<dpm> :-)
<henninge> abentley, adeuring: I will relocate quickly now, might be a bit late for the stand-up.
<jcsackett> dpm: and we're good.
<jcsackett> thanks for catching my mistake. :-)
<dpm> jcsackett, awesome, thanks!
<bac> hi deryck, can i quiz you about some structural subscription minutiae?
* leonardr changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: gmb, leonardr | https://code.launchpad.net/launchpad-project/+activereviews
<deryck> bac: you can, but I'm about to do a call with abentley.  Can we chat after that?
<bac> deryck: sure
<deryck> ok, cool.  I'll ping when I'm free.
<sinzui> henninge: danilos: can either of you assist me in answering https://answers.launchpad.net/launchpad/+question/148141
<henninge> sinzui: looking
<henninge> sinzui: unfortunately he is right. That feature has been removed.
<henninge> unfortunate for his project, that is.
<sinzui> henninge: Why did we remove this for projects. I think I understand the issue for Ubuntu
<henninge> sinzui: I am not sure, actually. It may be pure oversight.
<sinzui> henninge: Should I report a bug asking that the new/translated status for upstreams be restored for projects?
<henninge> sinzui: Yes, at least to have a point of discussion why this was done.
<sinzui> okay. Thank you
<LPCIBot> Project db-devel build #479: STILL FAILING in 5 hr 1 min: https://lpci.wedontsleep.org/job/db-devel/479/
<dpm> sinzui, I've just caught the conversation re: new/translated status in projects. When you file the bug, would you mind subscribing me or pointing me to it? I'd be interested in following the discussion. Thanks!
<adeuring> gmb: leonardr: could one of you please review this MP: https://code.launchpad.net/~adeuring/launchpad/no-package-translation-uploads-when-sharing/+merge/54356 ?
<leonardr> i got it
<sinzui> dpm bug 740225
<_mup_> Bug #740225: The differences between New and Translated for upstreams was removed <rosetta-imports> <upstream-translations-sharing> <Launchpad itself:New> < https://launchpad.net/bugs/740225 >
<jam> I just confirmed another XSS vulnerability in loggerhead for bug #740142
<jam> how do we roll out fixes for that sort of thing?
<dpm> thanks sinzui
<adeuring> leonardr: thanks!
<jam> I'd also like to avoid pushing the fix making it public
<leonardr> adeuring: r=me
<adeuring> leonardr: thanks!
<deryck> bac: about ready now.  voice or chat?
<bac> deryck: mumble might be quickest
<deryck> ok.  firing it up....
<deryck> bac: what room?
<deryck> bac: just drag me around ;)
<jcsackett> sinzui: do you know much about how security adapters work?
<jcsackett> or, anyone, same question? i'm having trouble getting an adapter (or really a new security.py file) working.
<bigjools> we really need to fix the ajax-forms-lose-data problem
<bigjools> just got burned big time
<sinzui> jcsackett: I do now quite a bit
<bigjools> by clicking a non ajax find-person link on the bug filing page :/
<sinzui> jcsackett: is this about adaption to IFAQTarget or IQuestionTarget
<jcsackett> it's a security adapter for IQuestion.
<jcsackett> sinzui ^
 * sinzui starts mumble
 * jcsackett starts it too.
<jcsackett> sinzui: i suspect you cannot hear me. i cannot hear you either. :-P
 * sinzui stabs mumble
<jcsackett> i heard you.
<jcsackett> i will restart mumble as well.
<jcsackett> sinzui: unrelated, are there still long term plans to deprecate polls (there's a question in #launchpad)
<sinzui> jcsackett: I see the mic setting went missing. I was there 18 hours ago
<sinzui> They are deprecated. We do not support them. There exist only for the Ubuntu community
<jcsackett> sinzui: i can hear you.
<jcsackett> but clearly you cannot hear me. :-P
<sinzui> no I cannot
<jcsackett> hm.
<jml> abentley: I have some questions about the TwistedJobRunner
<abentley> jml: shoot
<jml> abentley: in getTaskSource, why does the producer exhaust the iterReady() iterator before the loop begins?
<jml> abentley: hmm. actually another way of putting this is, what are you trying to get out of using PollingTaskSource?
<abentley> jml: I believe that's so that it runs for a finite time, instead of pulling jobs infinitely.
<abentley> jml: PollingTaskSource was existing Twisted code for doing this kind of thing.
<jml> abentley: I think we can achieve the same effect more simply by using DeferredSemaphore instead of PollingTaskSource and ParallelLimitedTaskConsumer, but I'm a little bit hesitant because I don't see many tests.
<LPCIBot> Project windmill build #89: FAILURE in 1 hr 17 min: https://lpci.wedontsleep.org/job/windmill/89/
<jml> abentley: also, that reminds me, TwistedJobRunner uses ParallelLimitedTaskConsumer with a hard-coded maximum number of 1 task at a time. What's the reason behind that?
<bigjools> lifeless: and as if by magic, bug 740250 appears
<abentley> jml: The idea was that we could relax the maximum when we felt that would improve performance.
<_mup_> Bug #740250: Timeout accepting packages  <Launchpad itself:New> < https://launchpad.net/bugs/740250 >
<jml> abentley: fair enough.
<jml> abentley: last question, I think...
<jml> abentley: I think I've isolated the intermittent failure in TestTwistedJobRunner.test_timeout. It comes from ampoule killing the job before the custom signal handler is installed
<abentley> I don't think that's possible.
<jml> abentley: I can prove it with science.
<abentley> the signal handler is installed when the first job runs, right?
<jml> hmm. interesting point.
<jml> OK. I can say with much greater confidence that if I reduce the lease_length in the test from 1 to a very small number, like 0 or 0.01, then the "intermittent" failure becomes deterministic.
<jml> abentley: is it guaranteed to re-use the same process?
<abentley> jml, that's what it's supposed to do, and I didn't see anything to make me believe otherwise.
<jml> abentley: good answer :)
<abentley> jml: I haven't dug through ampoule's internals thoroughly, and it's been a while since I looked at them at all.
<jml> abentley: I had a bit of a poke. There's definitely something racy going on with the lease_length (ampoule's "deadline").
<abentley> jml: Okay, IIRC 0 is treated completely stupidly.
<abentley> jml: I don't remember the details.
<jml> abentley: it's a bit obscured by the abstraction in job.
<jml> abentley: ok. I'll keep poking.
<jml> abentley: are there tests for the TwistedJobRunner other than TestTwistedJobRunner?
<abentley> jml: There are tests for the branch_merge_proposal_jobs script, which uses it.
<jml> abentley: thanks. I'll use those as backup.
<bigjools> g'night everyone
<jml> bigjools: g'night.
<abentley> jml: I wonder if setting the value to a very small number would cause the deadlineCall to fire before the callRemote?
<jml> abentley: that does happen. Although interestingly, the failure in the test is that the job *does* complete successfully (you can tell because the wait goes up to 30s+)
<abentley> jml: That's the failure case I hadn't been able to pin down-- the unexpected success.
<jml> abentley: I guess I'll instrument ampoule & see what happens.
<abentley> jml: I think I was wrong about the 0 handling, or else it was improved.  In any case, that's the timeout handling, and we're using deadlines, IIRC.
<jml> oh.
<jml> hm.
<jml> abentley: it uses the same mechanism, afaict.
<jml> http://paste.ubuntu.com/583909/
<abentley> jml: well, similar, but without special-casing 0.
<jml> right.
<abentley> leonardr: I'm writing some code that wants to treat an object the same whether it's from the LP.cache or retrieved from the web service.
<leonardr> abentley: that's a good place to be going. what problem are you having?
<abentley> leonardr: But WS objects use .get('foo') and LP.cache objects use ['foo'].
<leonardr> probably because ws objects are wrapped in a LP.Entry class (or whatever it's called), and LP.cache objects are just data structures
<abentley> leonardr: right.
<leonardr> you should be able to wrap the data structure yourself, and i'd even say that should be done automatically
<abentley> leonardr: sounds reasonable.
<leonardr> benji, i've once again reached the point where i can't stand to work on this multiversion stuff anymore. do you want to talk about filter/expand?
<benji> leonardr: sure, say 10 minutes from now?
<leonardr> ok
<jml> abentley: OK. I've got it figured out. http://paste.ubuntu.com/583931/
<jml> abentley: the signal is being sent before the callRemote call...
<jml> (as you said)
<jml> abentley: what this means is that the signal handler is triggered, raises an error, and then ... well, who cares? it's not going to terminate the process.
<abentley> jml: Cool.  That's great you were able to track it down.
<abentley> jml: so we need to tweak ampoule so it doesn't try to kill processes until they have launched?
<jml> abentley: not sure. in this case the process has launched, it's just that the command isn't running. tbh I'm not sure by what mechanism an exception that is raised in a signal handler is propagated out.
<jml> abentley: experimenting now...
<jml> *maybe* something to do with hooking up the OOPS system to the Twisted log?
<jml> errors raised in sighup handlers while reactor is running get treated as Unhandled errors, I think.
<leonardr> benji: i'd like to start the chat in irc if that's ok
<benji> leonardr: sounds good
<leonardr> i'm vacillating on having it here vs. in another room where we won't flood everyone else's conversations
<abentley> jml: I see what you mean.
<leonardr> like the defunct #launchpad-meeting
<leonardr> anyway, let me try to find the stuff i wrote on this before
<leonardr> here we go: https://dev.launchpad.net/Foundations/Webservice/DraftProposal
<leonardr> i think the only thing not mentioned in that document
<benji> leonardr: we can do it in PM, or we can just fabricate a chan., like #expand-and-filter
<leonardr> let's do it in pm, i don't think there's all that much to discuss now that i've found that document
<leonardr> since it never got much further than that document
<benji> k
<leonardr> basically i just think this is really important to the future of the web service and i don't want it to fall through the cracks when i leave
<benji> yep, agreed
<jml> abentley: so actually, it's just that the exception is raised in whatever code happens to be running at the time. maybe we can store some state ("hey, we got HUPped") and then raise during runJob if we see that state?
<abentley> jml: The exception being TimeoutError?
<jml> abentley: yes, or indeed anything that 'handler' might raise.
<abentley> jml: And if the job hasn't started, that would be raised in ampoule code, I guess.
<abentley> jml: And then, for some reason, it's not causing the process to terminate.
<jml> abentley: sort of. It will be raised as an unhandled error, because it's during the reactor.run() in BOOTSTRAP
<jml> abentley: e.g. http://paste.ubuntu.com/583943/ (simplified version)
<jml> the reactor keeps running
<jml> and whatever things were set up to listen to connections are still set up
<jml> in this case, the AMP protocol handlers
<abentley> jml: The reactor in the child process keeps running?
<jml> abentley: yes.
<jml> abentley: if the SIGHUP happens between jobs, it's only the child process that ever learns about the raised error.
<abentley> jml: I guess your solution makes some sense, but it doesn't feel satisfying.  What if no job ever gets run in the child?
<jml> abentley: I agree that it doesn't feel satisfying. I don't know what would happen in that case, or what we desire to happen.
<jml> abentley: I guess the timeout/deadline/lease is tied to the job rather than the child process.
<jml> abentley: so maybe it wouldn't matter.
<abentley> jml: true.
<abentley> jml: What about the case where the timeout fires after the job completes, but before the next is dispatched?
<jml> abentley: then in the simplest implementation, the next job would "time out" instantly
<jml> or at least, that's what it would look like.
<jml> I'm also worried about making timeout code too complex. It's supposed to be something you can rely on when other things fail.
<abentley> jml: Fully agreed.
<abentley> jml: Can we tweak this so that the child always dies when we raise TimeoutError?
<jml> abentley: probably. it's easy enough to sys.exit or reactor.stop or something. how does ampoule handle dead children?
<jcsackett> sinzui: can i set you as help contact in #launchpad?
<jml> even if we stopped the process, we'd have to figure out how to tell the parent that we did it because of a timeout. I guess we have OOPS logging available to us from BaseJobRunner.
<abentley> jml: I don't know how it handles dead children, but it expects timeouts to kill children, so it should expect this.
<jml> abentley: OK. I'll give it a try, see how it goes.
<jml> relatedly, I'd like to have a few more direct unit tests for this code.
<jml> not tonight though.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #480: FIXED in 5 hr 5 min: https://lpci.wedontsleep.org/job/db-devel/480/
<abentley> jml: I guess at the time it seemed like there weren't a lot of units to test.
<jml> abentley: I know the feeling. maybe there aren't, but after debugging this for a while I'd like to give it a try.
<jml> abentley: we'll have to do some work to translate the death into a TimeoutError, but I think it can be done
<jml> but that's enough from me for this evening
<jml> abentley: thanks for the help.
<abentley> jml: no problem.
<abentley> jml: good to get some fresh eyes on this.
<jml> g'night all.
<sinzui> jcsackett: I am so sorry. I forgot :(
<jcsackett> sinzui: no worries, it's been a quiet day.
<jcsackett> i just didn't want to switch you over if you were busy.
<bac> hi sinzui
<sinzui> bac
<wallyworld_> benji: hi. did you get a chance to look at my lazr restful enhancement? i was hoping to check with leonard but he's not around atm it seems
<benji> wallyworld_: yep I looked at it and didn't see anything bad jump out at me, I'd like to give it some more thought and get back to you tomorrow, how's that?
<bac> hi sinzui
<sinzui> hello
<bac> i wanted to ask you about the new hidden menu items and the use of the invisible-link class
<bac> it doesn't seem to do what i thought it did, i.e. make the link invisible
<wallyworld_> benji: that's fine thanks. i'll continue on with a branch to use the new stuff to fix the actual bug and we can revisit when you are ready
<bac> sinzui: was i correct in understanding the link would be present but invisible until i removed the class in JS?
<sinzui> bac: yes
<bac> hmm, ok
<sinzui> bac: let me look for a page with such as link
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project windmill build #90: FIXED in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/90/
<lifeless> wgrant: ping me when you're back
<lifeless> wgrant: back yet?
<wgrant> lifeless: Hi.
<lifeless> wgrant: so, why is the id desc sort important?
<wgrant> lifeless: For the POST? It's probably not; that query shouldn't be happening at all.
<wgrant> For the GET... what other order is there?
<wgrant> Apart from date_created, which is the same issue.
<lifeless> wgrant: so I proposed distroseries desc, status desc, archive desc, PackageUpload.id DESC
<lifeless> wgrant: distroseries and status are constants
<lifeless> so this is known as 'noddy fool the planner stuff'
<lifeless> archive has two values - 1, 534
<wgrant> It's fooling the planner into making a bad decision.
<lifeless> wgrant: good decision
<wgrant> Bad, because the ID sort is unindexed in that case.
<wgrant> This will probably exacerbate timeouts in the DONE case.
<lifeless> wgrant: huh, no, id is in the index
<wgrant> Oh, so it doesn't use it even with the full index?
<wgrant> I suspect your index is buggy.
<wgrant> packageupload(archive, distroseries, status, id) is what I think is ideal.
<lifeless> create index packageupload__distroseries__status__archive__idx on packageupload(distroseries, status, archive, id);
<wgrant> Ahh.
<lifeless> is what we tried
<wgrant> Interesting.
<wgrant> That's better for this particular query, but worse for others, but mine worked fine on mawson.
<wgrant> Can you tell why the planner is avoiding the index?
<lifeless> Limit (cost=0.00..105.89 rows=31 width=36) (actual time=0.106..0.108 rows=1 loops=1)
<lifeless>    -> Index Scan Backward using packageupload__distroseries__status__archive__idx on packageupload (cost=0.00..5365.99 rows=1571 width=36) (actual time=0.104..0.106 rows=1 loops=1)
<lifeless>          Index Cond: ((distroseries = 104) AND (status = 0))
<lifeless>          Filter: (archive = ANY ('{1,534}'::integer[]))
<lifeless>  Total runtime: 0.159 ms
<lifeless> wgrant: what are the other queries? theres no reason for use to ues the same query for different cases
<lifeless> bah, you know what I mean
<wgrant> lifeless: Everything except that page is driven by archive, not distroseries. So having archive first is going to be more generally useful.
<lifeless> wgrant: anyhow, *why* do we need id to sort before archive *on this page*
<wgrant> lifeless: Because it's meant to be a chronological ordering, not have every partner upload ever first.
<wgrant> Perhaps if primary was first.
<wgrant> But definitely not partner.
<lifeless> wgrant: this is for processing NEW right?
<wgrant> lifeless: Not just NEW, but yes.
<wgrant> lifeless: The other statuses get looked at to see what has happened to various packages.
<wgrant> And having to scroll through an indeterminate number of pages of partner to get to the latest uploads is not going to make people happy.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/276950/comments/23
<_mup_> Bug #276950: DistroSeries:+queue Timeout accepting many packages queue page <lp-soyuz> <queue-page> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/276950 >
<wgrant> Both because it's stupid, and because it's partner.
<wgrant> lifeless: What if you try another series?
<wgrant> The planner is being really stupid here.
<lifeless> wgrant: this is ubuntu
<lifeless> no ?
<wgrant> lifeless: Ubuntu and Canonical Partner.
<lifeless> yes
<lifeless> wgrant: so, I propose we take a different approach to fixing this
<lifeless> wgrant: a) define the scope
<lifeless> b) solve that scaope
<lifeless> c) wait for others to have issues
<lifeless> theres no a-priori requirement to have the same query generated by python for different use cases.
<wgrant> The same query is fine.
<wgrant> PostgreSQL's planner is not.
<lifeless> folk shouldn't be scrolling through pages and pages to get to things
<lifeless> thats nuts anyway
<lifeless> so I really don't think we should care about status not in (0,1)
<lifeless> and for status in (0,1) its a work queue thats meant to be driven to zero; it shouldn't normally matter if we group on archive or not
<wgrant> These postgres workarounds are piling up.
<wgrant> But OK.
<lifeless> we can do a partial index - (date_created desc, archive, distroseries) where status in (0,1)
<lifeless> that models the work queue case accurately.
<lifeless> which is *always* going to be lopsided
<lifeless> the non-queue case is 2.7MILLION rows
<wgrant> lifeless: And the indexes handle that trivially, when they are properly used.
<wgrant> But we cannot make them be properly used.
<lifeless> wgrant: storing a work queue in a historical table is improper use
<wgrant> Because postgres doesn't let us override its bad judgement.
<lifeless> wgrant: thats why postgresql is having trouble
<wgrant> lifeless: It depends on your definition of "proper", but probably.
<lifeless> proper = will work well for the DB stack we're using
<wgrant> I would s/proper/hackish/
<lifeless> wgrant: working against the stack will be hard
<lifeless> wgrant: its not adroit :P
<wgrant> Sure.
<wgrant> But it's still a really bad hack.
<wgrant> We probably have no choice.
<lifeless> wgrant: what makes it that ?
<wgrant> But that doesn't make it good.
<lifeless> wgrant: seriously, we're not using bdb
<wgrant> lifeless: We are changing data models and rewording queries to convince the postgres planner to do exactly what we want it to do.
<wgrant> When we know exactly what we want it to do, but we can't tell it that.
<lifeless> wgrant: you're talking about query hinting ?
<lifeless> wgrant: how would you describe what we want the planner to do?
<lifeless> wgrant: (in english) - because looking at the indices *and the query constraints* I think its being pretty sane
<lifeless> specifically, its looking for a small number of rows near the end of the table by walking backwards.
<lifeless> its been told:
<lifeless>  - we don't want every row (limit 31) and we want the beginning of the sequence (offset 0)
<lifeless> so its a very smart choice.
<wgrant> lifeless: It's avoiding using an index which is perfect for that query.
<wgrant> We know that.
<wgrant> But the skewed data prevents it from knowing that.
<lifeless> packageupload__distroseries__status__idx isn't perfect for the query
<wgrant> With archive and id it is.
<lifeless> wgrant: results of hte partial on the bug
<wgrant> Thanks.
<lifeless> wgrant: the index isn't perfect - archive 1 + 534 - 1M rows
<wgrant> Is comment #24 corect?
<wgrant> How is the index condition distroseries, when date_created is first in the index?
<lifeless> its scanning the index
<lifeless> it can't select at the root
<lifeless> but because its only looking at status 0,1 its very cheap
<wgrant> Ah, true.
<lifeless> it can satisfy the distroseries rule from the index
<lifeless> before looking at table row
<wgrant> We will probably need to revisit this in a year or so.
<wgrant> Once we have a derivative distro or two.
<lifeless> Index cond -- index tells us, filter -- table row is used.
<lifeless> and - win
<lifeless> we can make the current query use it
<wgrant> How fast?
<lifeless> 7ms
<wgrant> Not bad.
<lifeless> I can live with it
<wgrant> It still leaves accepted/rejected/done timing out often, but I guess that is less of a user-affecting issue.
<lifeless> wgrant: that use case needs to be revisted anyway
<lifeless> wgrant: it should be a search etc, not a freaking pagination
<lifeless> wgrant: showing 1M rows of done in a pagination view is very hard to do at all, let alone well
<wgrant> Indeed.
<lifeless> wgrant: I'm going to prep a db patch to add the index, and get it applied live
<wgrant> lifeless: Thanks.
<lifeless> I think jam has halted(), we should do loggerhead ourselves
#launchpad-dev 2011-03-23
* StevenK changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> sinzui: ping
<LPCIBot> Project windmill build #91: FAILURE in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/91/
<lifeless> thats better,
<lifeless> more closed bugs
<lifeless> and one more hidden critical unearthed
 * lifeless resists the urge to start at bug oldest and walk forward
<lifeless> wgrant: ping
<lifeless> I'm pinging you cause curtis had better be asleep by now :) -  where is Cve:+index - is jc still on it, or would it be ok if I pick it up? its our #2 frequency OOPS
<wgrant> lifeless: He's still on it.
<lifeless> kk
<lifeless> wgrant: have you poked at +needs-packaging at all ?
<lifeless> wgrant: I'm struggling to find top-10 timeouts that are not fixed-undeployed, bugtask:+index, or in-progress by someone
<wgrant> lifeless: Not at all.
<wgrant> You seem to love BugTask:+index
<lifeless> wgrant: Not at all.
<lifeless> wgrant: the next obvious thing is a paginated message display
<lifeless> wgrant: which is to big for me: I get called away all over the place.
<lifeless> wgrant: I mean, making a vertical page-reloading batchnav is easy.
<lifeless> wgrant: making one that uses js to incrementally pull in new messages and actions and shows the comment form once you read to the bottom - a bit harder.
<wgrant> Yeah.
<lifeless> wgrant: so other than ratcheting scaling down a bit more, I'm not likely to really touch it for a bit
<lifeless> dammit, I need to finish my analysis of deletable bugtasks etc
 * lifeless adds a note to talk to jml about said
<lifeless> wgrant: (oh, and I'd be delighted for someone else to do the next iteration on bugtask:+index)
<lifeless> wow
<lifeless> 400ms overhead per page atm
<lifeless> just doing traversal
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1907C1055
<lifeless> probably a busy appserver contributing
<mwhudson> i worry a bit sometimes about the minimum time launchpad can produce a page in
<mwhudson> (i think it's quite a while, like 100ms)
<mwhudson> maybe not that bad
<lifeless> mwhudson: well, the page I'm looking at - 400ms minimum.
<lifeless> mwhudson: trace the queries down to the session update after traversal
<mwhudson> lifeless: erk
<lifeless> we're still in phase one though - drive the 99th% percentile down by fixing poor scaling in pages
<mwhudson> oh that's the server machine wgrant is always complaining about isn't it?
<lifeless> mwhudson: well, its not the hardware.
<wgrant> It's partly the hardware.
<lifeless> mwhudson: its a 4-thread appserver with maybe 2-4 additional requests parsed and awaiting handoff
<lifeless> wgrant: it really isn't :)
<wgrant> lifeless: soybean has lots of them as well.
<wgrant> But we see most of our timeouts from the old appservers.
<mwhudson> lifeless: yeah, something smells off about that that's not entirely related to the code in lp:launchpad
<lifeless> anyhow
<lifeless> tonight is pounce on the sysadmins night
<StevenK> Poor sysadmins
<lifeless> (not nastily)
<lifeless> just that tom is the guy working this progression
<wallyworld_> lifeless: if there's a bug with total crap in it (random text), do we just mark it as incomplete or do we need to get it deleted?
<lifeless> invalid probably.
<lifeless> whats the bug #?
<lifeless> wallyworld_: ^
<wgrant> wallyworld_: Sorry, I meant to look at that earlier, but got sorta distracted :/
<wallyworld_> 738023
<lifeless> lol - classic - http://programming-motherfucker.com/
<lifeless> wallyworld_: that seems to be outside the launchpad project?
<wallyworld_> yes
<wallyworld_> but it's an open lp question
<lifeless> ah
<wallyworld_> so i thought i'd clean it up since it's clearly a crap bug
<lifeless> so there is no spam in it
<lifeless> incomplete leaves it open
<lifeless> I would make it invalid
<wallyworld_> no. just crap
<wallyworld_> ok
<wallyworld_> thanks
<cody-somerville> Why does the series filter on the main pages of PPAs have such huge text? lol
<wgrant> Someone embedded a form inside a heading, and this was exposed in the recent style unifications.
<lifeless> H1 is big
<lifeless> I can has review https://code.launchpad.net/~lifeless/launchpad/bug-722794/+merge/54466 ?
<StevenK> lifeless: Done, r=me
<lifeless> thanks
<lifeless> wgrant: wtf: https://lp-oops.canonical.com/oops.py/?oopsid=1907C2337
<lifeless> wgrant: I'm pinging you cause you added the transaction status to the timeline :)
<wgrant> lifeless: Huh.
<wgrant> But API timeouts normally work fine :/
<wgrant> Although leonardr did change them recently.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/730396
<_mup_> Bug #730396: DistroSeries:EntryResource:getBuildRecords timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/730396 >
<lifeless> what does BuildFarmJob.status <> 1 OR BuildFarmJob.date_finished IS NOT NULL imply
<lifeless> not fullybuilt and finished ?
<lifeless> the query I'm looking at also checks for FAILEDTOBUILD
<lifeless> can I drop the BuildFarmJob.date_finished IS NOT NULL condition without breaking semantics ?
<lifeless> thumper: https://bugs.launchpad.net/launchpad/+bug/730396/comments/9
<_mup_> Bug #730396: DistroSeries:EntryResource:getBuildRecords timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/730396 >
<wgrant> lifeless: That query is stupid, yes. For some of the cases you can drop the first BFJ <> AND date_finished IS NOT NULL check, yes.
<wgrant> lifeless: But we need to keep date_finished IS NOT NULL if status == 1.
<lifeless> sure
<lifeless> so over here
<lifeless> uhm
<lifeless> if we don't gc builds
<lifeless> there is no point reading all *FTBFS* rows every time
<lifeless> full stop
<StevenK> Hmmm, looks like I didn't kill dogfood
<wgrant> lifeless: If we move to an incremental model, sure.
<wgrant> lifeless: But then we'd need to read all recent successes too.
<lifeless> wgrant: why?
<wgrant> lifeless: Because we need to be able to remove failures that are no longer failures.
<lifeless> wgrant: do they get removed from this collection?
<wgrant> If we're not reading the whole failure collection, we need to read every other collection.
<lifeless> wgrant: do they get removed from this collection?
<wgrant> lifeless: Which collection?
<lifeless> fbtfs
<wgrant> lifeless: A retried build is not a new build.
<wgrant> A retried build will disappear from this page.
<wgrant> (unless it fails again)
<lifeless> wgrant: so, we do gc
<wgrant> Pardon?
<lifeless> or are you asying we reuse the BFJ ?
<wgrant> We reuse the BFJ.
 * lifeless headdesks
<wgrant> Yes.
<wgrant> Someone made this decision 6 years ago, and uh, yeah.
<lifeless> anyhow
<lifeless> I'd read all collections incrementally
<lifeless> that seems cheap and effective as long as you don't shutdown for a month
<lifeless> wgrant: what rev has leonards change for timeouts ?
<wgrant> lifeless: It was a lazr.restful upgrade.
<lifeless> meh, I'll raise a critical
<wgrant> I'm pretty sure this worked after my changes, although it's clearly somewhat related.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/740750
<_mup_> Bug #740750: API timeouts broken and returning no useful data... <Launchpad itself:Triaged> <lazr.restful:Triaged> < https://launchpad.net/bugs/740750 >
<poolie> big f* jools?
<lifeless> poolie: ?
<poolie> <wgrant> We reuse the BFJ.
<wgrant> Heh'
<lifeless> poolie: before his time
<StevenK> poolie: Build Farm Job
<poolie> https://bugs.launchpad.net/launchpad/+bug/740759 might amuse you
<_mup_> Bug #740759: creating bzrdir on launchpad is slow (BzrDirFormat.initialize_ex_1.16) <codehosting-ssh> <hpss> <lp-code> <performance> <Launchpad itself:Triaged> < https://launchpad.net/bugs/740759 >
<adeuring> good morning
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> stub: hi
<lifeless> stub: how are you feeling?
<stub> yo
<stub> Better today... slept 13 hours straight last night.
<lifeless> stub: good timing - we tried your live index docs but got the patch number wrong: https://code.launchpad.net/~wgrant/launchpad/fix-patch-57/+merge/54481 is meant to fix it, but wanted to confirm: is the db patch *in the tree* meant to be a .1 patch, or a .0 with the same major as the one done live ?
<wgrant> I think the numbers have to match./
<wgrant> Or it will be reapplied on the next db-deploy and then blow up.
<wgrant> I think.
<stub> db patch in the tree is supposed to be a 2208-xx-01 (or -02 etc.) patch. The number should match what you inserted into LaunchpadDatabaseRevision on production
<poolie> hi
<poolie> would i be right in expecting that the qastaging database will be generally faster than lpnet?
<poolie> because of less load
<wgrant> It's a much smaller DB server.
<wgrant> And the cache will be colder.
<stub> Why this works - Launchpad appservers check LaunchpadDatabaseRevision for patches that have been applied that are not in the local source tree, and for patches in the local source tree that have not been applied. It ignores all patches that are non-zero in the 'patch' field.
<StevenK> So it's generally slower
<poolie> hm
<stub> lifeless, wgrant: So that patch seems correct, and matches what I see on the production db. Did the patch also get applied to launchpad_prod_2 and launchpad_prod_1 ?
<lifeless> stub: yup
<stub> So it worked out best with id in the first field of the index?
<wgrant> stub: It was applied as -0 to all three.
<lifeless> stub: it was a lopsided problem
<stub> I don't see -0 in LaunchpadDatabaseRevision - I only see 57-1
<wgrant> stub: Right, we had to fix it when an appserver failed to start.
<lifeless> stub: we tried a number of permutations on qastaging, and found that a partial index on the queues that are driven to 0 (the NEW and INCOMPLETE) performed very well
<wgrant> lifeless: NEW and UNAPPROVED
<stub> I thought the index would have worked better with id in the last field, as the ordering would have been done last, but I hadn't checked and wasn't sure.
<wgrant> stub: As would I. I didn't follow lifeless' trials very closely, though.
<lifeless> can't do id last
<lifeless> not without a bunch of headaches
<stub> interesting
<lifeless> right now we use one query for both:
<lifeless>  - queue processing
<lifeless>  - historical data analsis
<lifeless> pg won't use the indices we tried with id last, unless the full index sort was given
<lifeless> archive desc distroseries desc setc
<lifeless> it *will*, but not for status=0 [or 1 we are pretty sure]
<lifeless> because its a lopsided data problem
<lifeless>  status |  count
<lifeless> --------+---------
<lifeless>       4 |    5777
<lifeless>       1 |     182
<lifeless>       2 |   47494
<lifeless>       3 | 2289831
<lifeless>       0 |   40961
<wgrant> Um what?
<wgrant> That's about 41000 more with status 0 than I would expect.
<lifeless> wgrant: indeed, but not enough to worry the index
<wgrant> Enough to worry me, though...
<lifeless> wgrant: sure, separate problem
<bigjools> *blink*
<wgrant> Similar on mawson.
<wgrant> Hmm.
<wgrant>       0 |           97 | 40877
<wgrant> Distroseries 97
<lifeless> stub: so the headaches are: we would have to either: split out a separate query for the historical data, if we changed the status 0 case to use archive desc distroseries desc status 0 desc id desc
<wgrant> karmic
<lifeless>      (because the historical data /wants/ id as the first sort key - and pg does a decent job there today)
<wgrant> Oh.
<lifeless>  or
<wgrant> bigjools: Old archive rebuilds :(
<lifeless> well, no or
<stub> lifeless: Sure. I recall the query. I'm interested in you found it works best with the id in the sort key. I think we have similar cases elsewhere - bugs or bugtask I think.
<lifeless> basically, there are two totally different use cases in the one table
<lifeless> stub: only because its essentially a micro-table embedded in the bigger one
<lifeless> stub: thats why adding a partial index gave good results - -very- fast results once applied [to that particular query]
<bigjools> wgrant: grar
<wgrant> bigjools: 2009ish
<bigjools> let's clear those out
<wgrant> So long-fixed.
<wgrant> But yes.
<lifeless> stub: its possible you can come up with something better
<wgrant> Also, accepted is pretty big... probably newer copy archives.
<wgrant> Indeed, the newer copy archives.
<stub> lifeless: If you tried the partial index on (distroseries, archive, id) instead of (id, distroseries, archive), that would be all I'm interested in at the moment.
<lifeless> stub: but the nice thing here was that the partial index was small - 1280 kB
<stub> lifeless: I think in earlier versions of PG, the index would not have been used at all because the first field in the index was not used until the very end.
<lifeless> stub: we didn't try that variation, no
<bigjools> wgrant: we need a way of cleaning out copy archives entirely
<wgrant> bigjools: Yes.
<bigjools> wgrant: I want to make derived distros supersede them
<lifeless> stub:  Limit  (cost=0.00..104.34 rows=31 width=36) (actual time=0.132..83.965 rows=1 loops=1)
<lifeless>    ->  Index Scan Backward using packageupload__distroseries__status__archive__idx on packageupload  (cost=0.00..5287.65 rows=1571 width=36) (actual time=0.131..83.961 rows=1 loops=1)
<wgrant> bigjools: An interesting plan.
<lifeless>          Index Cond: (distroseries = 104)
<lifeless>          Filter: ((archive = ANY ('{1,534}'::integer[])) AND (status = 0))
<lifeless>  Total runtime: 84.023 ms
<lifeless> stub: ignore the index name, thats on qastaging where we were experimenting
<bigjools> wgrant: they're essentially the same thing, but copy archives have a spethial UI
<lifeless> stub: also - thats *cold* runtime
<wgrant> bigjools: Indeed.
<stub> lifeless: Right. So in PG 8.4, it might well be more efficient to put the field we are ordering on first, and the ordering and filtering done in one index scan.
<lifeless> stub: at least sometimes, yeah
<stub> Which is different to how we optimized stuff before elsewhere in the schema. Might be some room for improvement there.
<bigjools> wgrant: OOPS-1907BUILDMASTER3 ....
<stub> lifeless: Where the live rollout docs good enough for the losas to apply the patch without hiccups?
<wgrant> bigjools: What about it?
<lifeless> stub: yup, except we got the patch # wrong
<wgrant> bigjools: It's annoying, but ordinary and harmless.
<bigjools> wgrant: it should not be an oops
<wgrant> bigjools: Sure.
<bigjools> well, we could see if someone deleted the source
<wgrant> Someone uploaded one to supersede it.
<bigjools> or that
<stub> lifeless: Right. Applying a -0 patch live would mean the appservers fail to restart, so wouldn't even be immediately noticeable.
<bigjools> that oops happened 3 times
<wgrant> bigjools: Different archs?
<bigjools> ah might be
<wgrant> Hm, no.
<wgrant> Different series.
<wgrant> https://launchpad.net/~kxstudio-team/+archive/ppa/+packages?field.name_filter=lv2core&field.status_filter=&field.series_filter=
<lifeless> stub: we only caught it when we tested dropping the appserver name from the lpnet configs and found it broke
<bigjools> ok
<lifeless> stub: so the name in db-devel is meant to be -1 ?
<stub> lifeless: Yes.
<lifeless> cool
<stub> lifeless: Anything non-zero in that field. A non-zero in the 'patch' field means ignore this on runtime checks - it is a minor patch and you shouldn't care if it has been applied or not. Matches our life updates usecase exactly, so yay for prescience.
<lifeless> stub: kk
<henninge> jtv: Hi!
<jtv> hi henninge
<henninge> jtv: Are your reviewing today?
<jtv> henninge: oh, completely forgot about it.  :(  OTP now.
<henninge> np
<stub> lifeless: Hmm.... in fact, I can't see why -1+ patches can't land on launchpad/devel ....
<henninge> danilos should be reviewing today, too. ;-)
<lifeless> stub: that would be great
<stub> Where does that rule get changed now?
<lifeless> the pqm config
<lifeless> its out of the tree
<wgrant> Heh, the first time it's needed a change in almost three years, a month after I argued that it never needed changing...
<stub> :-)
<stub> I think we just need to block *-0.sql from landing on launchpad/devel now. No problems landing updates to the scripts, security.cfg or minor db patches.
<wgrant> Right.
<stub> Anyone see a problem with that rule?
<wgrant> Nope. We should also drop the conflict check while we're there, since it doesn't work.
<wgrant> And is pointless.
<lifeless> fine with me
<stub> wgrant: Do you want to file an RT explaining that conflict check thing, or do you want to explain the conflict check thing to me so I can file the RT ?
<wgrant> stub: https://pastebin.canonical.com/45073/ is the current hook.
<stub> MY EYES!
<wgrant> stub: The second bit (bzr ls -0 [...]) is meant to check for conflicts, but doesn't actually work. I considered removing it when we migrated the check from the tree to PQM, since it seems pretty pointless, but decided to make as few changes as possible.
<stub> So precommit_hook=[ -z "$(bzr status -S database/schema/ | grep '-0.sql$')" ]     ??
<wgrant> stub: Don't need to protect fti.py or anything?
<stub> wgrant: fti.py is 50/50 - adding a new index could futz things, changing an existing one will not (the column will exist, it just won't have the expected data in it)
<wgrant> Sounds good, then.
<wgrant> RT time.
<stub> I guess we should block it - we likely won't change anything there anyway until we know where our search is heading.
<wgrant> stub: Thanks for the review.
<stub> precommit_hook=[ -z "$(bzr status -S database/schema/ | grep -e '\(-0\.sql\|/fti.py\)$')" ]
<stub> is what I end up with
<wgrant> Looks sensible, as long as it works.
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: danilos | https://code.launchpad.net/launchpad-project/+activereviews
<henninge> danilos! ;)
<danilos> henninge, :)
<danilos> henninge, I don't know what you've been talking about ;)
<henninge> danilos: I do, though! Can I please have a review?
<danilos> henninge, sure thing
<henninge> danilos: diff is only 1075 lines
<henninge> https://code.launchpad.net/~henninge/launchpad/devel-605924-hastranslationtemplates/+merge/54470
<danilos> henninge, ha-ha-ha
<danilos> henninge, ok, I'll take it
<henninge> danilos: thanks
<danilos> henninge, btw, just looking through proposal text, "shortlist" is especially recommended when you expect something to return a short list
<danilos> henninge, because then it catches bugs early
<henninge> danilos: oh, i thought it was about cutting a list off.
<danilos> henninge, no, it throws an OOPS/exception when hard limit is broken
<henninge> ah
<henninge> so using shortlist is saying "tell me if this ever gets longer than X"
<henninge> danilos: thanks, I did not know that. I am happy to put it back in, then.
<danilos> henninge, yeah, it's a developer tool :)
<danilos> henninge, I don't know in what context did you drop it, so it all depends on that, but just clarifying what shortlist is about
<LPCIBot> Project windmill build #92: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/92/
<deryck> Morning, folks.
<danilos> henninge, good job clearing the inter-dependency between template count and uses-translations
<henninge> yeah, that was strange to see
<danilos> henninge, not yours, but line 226 of the diff: "hidding"
<danilos> henninge, in TranslationTemplatesCollection, I am surprised target_pillar is not used anywhere, but since it's apparently not, good cleanup there as well :)
<henninge> danilos: yes, that was only used to access translation_usage in getCurrentTranslationTemplates
<danilos> henninge, also, instead of listifying stuff in tests, I suggest you use assertContentEqual instead
<danilos> henninge, unless where you want to compare the order as well
<danilos> henninge, fwiw, this is probably irrelevant comment when you get shortlist back :)
<danilos> henninge, also, I'd rather see test_has_obsolete_translation_templates as 3 separate tests
<danilos> henninge, or 4
<danilos> henninge, and also, you probably inserted list() on diff line 116 because you removed the shortlist as well, so that can probably go (and if you decided not to re-insert shortlist, this is where you should use it)
<danilos> henninge, r=me
<henninge> danilos: yup, that's just because of shortlist.
<henninge> danilos: thanks
<danilos> henninge, up to you to decide if you want shortlist back or not, not a big deal either way
<henninge> danilos: I felt the same about test_translation_templates but I played along with what the rest of the file was doing.
<henninge> I may just split up the other tests, too.
<danilos> henninge, right, we did write tests that way in the past
<henninge> danilos: And I was always the one saying "one assert per test"!
<danilos> henninge, that'd be great, but if it gets complex, it might mean a new branch
 * bigjools waves at danilos
<henninge> I'll have a look
<danilos> henninge, heh, well, that is tricky because we sometimes use asserts to make sure input data is right
<danilos> bigjools, hi there
<henninge> danilos: ok, that is only a guideline.
<henninge> ;-)
<danilos> bigjools, I hope you just look forward to hearing my lovely voice :)
<bigjools> danilos: are you reviewing?  I have a short branch
<danilos> bigjools, yeah, I do (but I am disappointed ;)
<danilos> bigjools, pass it on, just done with henninge's
<bigjools> danilos: I miss your booming presence on the TL calls :)
<danilos> henninge, yep, cheers
<danilos> bigjools, ok, now I feel better, thanks :)
<bigjools> danilos: https://code.launchpad.net/~julian-edwards/launchpad/localdiff-sync-bug-739525/+merge/54496
<danilos> bigjools, branch sounds like a nice feature, but you've got a single lint issue so please fix it (I understand it may not be from your changes :)
<danilos> looking at the code now
<bigjools> :/
<LPCIBot> Project windmill build #93: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill/93/
<bigjools> it was my lint
<danilos> bigjools, I am sad to see a message like "The following sources *would have been* synced if..." go from the real UI
<bigjools> :)
<wgrant> bigjools: Have you worked out how permissions are going to work?
<bigjools> no
<wgrant> Heh
<danilos> bigjools, r=me, but I'd also remove the "XXX" from a comment about a thing you are not really going to fix unless it proves to be broken
<bigjools> ok
<bigjools> thanks danilos
<bigjools> wgrant: right now it's launchpad.edit on the series
<wgrant> bigjools: launchpad.Edit and launchpad.Append (since you need Append to run syncSources)
<bigjools> right
<bigjools> this will all fall out in the wash when we start doing intensive testing
<wgrant> Yeah.
<wgrant> And then we realise we need queues.
<bigjools> which Steve is doing tomorrow :)
<wgrant> And then you have an excuse to fix copying.
<bigjools> queues would be nice
<bigjools> I might have to badger flacoste and jml about that
<jml> mushroom! mushroom!
<bigjools> snaaaaaaaake
<sinzui> jcsackett: I Untriaged bug 740225. The former rosetta team need to explain if this was inteded and why or if it is a regression. The behaviour is correct for Ubuntu so it appears to be by design
<_mup_> Bug #740225: The differences between New and Translated for upstreams was removed <rosetta-imports> <upstream-translations-sharing> <Launchpad itself:New> < https://launchpad.net/bugs/740225 >
<stub> Are lp-propose-merge bugs bzr bugs or something else?
<jcsackett> sinzui: understood.
<jcsackett> sinzui: do you remember the issue you helped oem with regarding filing bugs in IE?
<jcsackett> i believe it was an issue with an error popping up on the summary field? i've got an open question about it, and was wondering if there was a bug filed. i can't find one.
<sinzui> jcsackett: The JS to populate the matches and enable the continue actions does not work in IE. It is impossible to report a bug since the user is forced to use the guided process
<sinzui> jcsackett: The bug is filed and has many dupes. tag: javascript, text ie or ie8 or ie7
<jcsackett> ah right, tags!
 * jcsackett never remembers tags when bug searching.
<sinzui> first match
<sinzui> I always use tags but I hate the default ANY, I am not sure I have ever used ANY in a search
<abentley> adeuring1: bzr+ssh://bazaar.launchpad.net/~abentley/launchpad/js-translation
<abentley> adeuring1: https://code.launchpad.net/~abentley/launchpad/js-translation/+merge/54067
<henninge> deryck: I'll restart
<abentley> adeuring1: do you want to chat in mumble about the template changes?
<adeuring1> abentley: no yet, let me first read a bit more
<abentley> adeuring1: okay, ping me whenever.
<leonardr> danilos, would you take a look at https://code.launchpad.net/~leonardr/lazr.restful/integrate-strict-versioning/+merge/54530 ?
<adeuring1> abentley: so your idea is to always render both variants of "X is configured/not configured" and to make the "wrong" variant invisible via CSS, right?
<abentley> adeuring1: right.
<adeuring1> ok, so no need to mess with the menu:overview stuff, i think
<adeuring1> abentley: ^^^
<abentley> adeuring1: Actually, we probably do.
<adeuring1> abentley: why?
<danilos> leonardr, after lunch, sorry
<abentley> adeuring1: The icon is supposed to vary between "plus" and "edit".
<adeuring1> abentley: ah, right
<abentley> adeuring1: Depending on whether there's already a value.
<abentley> adeuring1: But that state is frozen when the template is rendered.
<adeuring1> right
<abentley> adeuring1: So either we get two plusses or we get two edits.
<adeuring1> yeah
<jam> dear codehosting devs, why is my +junk branch trying to stack on /~sathishmanohar/junk/main
<jam> lp:~jameinel/+junk/one-rev-1
<jam> ah, stupid me, doing '/junk/" without the +
<abentley> jam, right.
<jam> I only noticed because it fails on qastaging because 'junk' doesn't exist
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: danilos, abentley | https://code.launchpad.net/launchpad-project/+activereviews
<jcsackett> danilos or abentley: could i request a review of https://code.launchpad.net/~jcsackett/launchpad/set-question-message-visibility/+merge/54408?
<jam> any way to bulk upload attachments? I have 6 attachments I want to add to a bug, and it is a pain to do them one-by-one
<benji> jam: not that I know of; I wonder if a little launchpadlib script would be helpful, but it will take longer to write than just doing the uploads
<abentley> jcsackett: sure.
<abentley> jcsackett: You say "the branch partially relies on a branch that exposes Question on the webservice".  This sounds like a prerequisite branch to me.
<abentley> jcsackett: Do you know about prerequisite branches?
<LPCIBot> Project windmill build #94: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/94/
<jml> abentley: I learnt a thing about the job system today. It uses Python imports as a registry.
<abentley> jml: that doesn't parse.
<jml> abentley: as in, it very strongly encourages job sources to be defined as classes that have all of their state on the class so that it can then load these classes using import/module-lookup/getattr-fu in other processes.
<abentley> jml: this is true.
<jml> abentley: so in this regard, it's using Python's namespace/import system as a registry of job sources. sort of.
<jml> abentley: I bumped into this because I thought having an instance job source would make testing easier, so I gave it a try.
<abentley> jml: Seems like a stretch to me, unless you think that most python functionality uses imports (by which you mean packages/modules?) as a registry.
<jml> abentley: well, maybe it is a stretch. it's just that I can imagine alternative implementations that make as much sense as importing that are more obviously registry-like
<abentley> jml: In which case, it doesn't seem worth mentioning.
<jml> e.g. pickling the job source to a known directory and passing the filename through to the child process
<jml> or having a config file that maps names to ways of acquiring job source objects.
<abentley> jml: we actually do have a config that maps names to job source classes.
<jml> abentley: class names in Python modules?
<abentley> jml: No, when we use cronscripts/run_jobs.py, we pass in a config name, which includes the job source as one of its variables.
<abentley> jml: e.g.  packaging_translations
<jml> abentley: hmm. that's a more flexible than the TwistedJobRunner. Still assumes a particular mechanism for acquiring the IJobSource object though (it has to be a member of a module, available on import).
<abentley> jml: I don't think it's more flexible.  I can easily imagine using it with the TwistedJobRunner.
<jml> abentley: you'd have to change TwistedJobRunner first, surely
<abentley> jml: what would I need to change?
<jml> abentley: the bit where it figures out the import name based on job_source.__name__
<abentley> jml: why?
<jml> abentley: to use a non-class object as a job source
<abentley> jml: I expect an interface to correspond to a class.  Don't you?
<jml> abentley: Yes, but that's not what I mean.
<jcsackett> abentley: sorry i missed your comment earlier. i know about prereq branches, but the the prereq branch had other stuff that messed up the diff, as i only brought over the webservice bits.
<jml> abentley: the TwistedJobRunner theoretically expects a job source object that provides ITwistedJobSource. The implementation demands that it is a class that classProvides ITwistedJobSource.
<abentley> jml: From an interface, we get a class, from the class, we get job_source.__name__, what's the issue?
<abentley> jml: still not following why I'd need to change TwistedJobRunner to make it work with run_jobs.
<jml> oh right.
<jml> abentley: run_jobs takes an arbitrary name to import, TwistedJobRunner infers the name from the passed in job source object (which has to be a class)
<LPCIBot> Project devel build #569: FAILURE in 4 hr 52 min: https://lpci.wedontsleep.org/job/devel/569/
<abentley> jml: right, but these conditions are met for all of our job sources, so I don't need to address that in order to use TwistedJobRunner.
<jml> job_source = somethingThatMakesAJobSource(); runner = TwistedJobRunner(job_source, ...); ... will explode unless job_source has a __module__ and __name__ and can be reconstructed by importing "%s.%s" % (job_source.__module__, job_source.__name__)
<sinzui> jcsackett: set  ~jcsackett/launchpad/questions-on-webservice  as a dependency of https://code.launchpad.net/~jcsackett/launchpad/set-question-message-visibility/+merge/54408
<jcsackett> sinzui: ok.
<jml> abentley: of course. I was trying something different and discovered these conditions.
<adeuring1> abentley: I think there is a more fundamental problem with the approach to hide  the "wrong" <li> entries via CSS: If no packaging link is set, we can't generate the links "set source branch", "enable translations", "enable auto sync" at all: We don't know which target to use...
<jcsackett> sinzui: is there a way to do that without redoing the proposal?
<abentley> adeuring1: I've already addressed that for the branch link.
<abentley> jcsackett: You resubmit the proposal.
<adeuring1> let me look...
<abentley> adeuring1: It creates an empty link.
<adeuring1> abentley: ah, right!
<sinzui> jcsackett: you may hate me. We should not export Question.priority. It is unused
<adeuring1> abentley:  regarding the icons: I think we always have the pencil icon. So I think we don't have to tinker with the menu: stuff
<jcsackett> sinzui: that's an easy fix. no need for hate. :-)
<abentley> adeuring1: that's nice, but it's inconsistent with how we're handling branches.
<abentley> deryck: adeuring1 reports that productseries always has an edit icon, never an add icon, unlike branches.  Should we make them consistent?
<deryck> abentley: I would think consistency would be better, yes.
<leonardr> benji, i'd like to go over lp:~leonardr/launchpad/explicit-versions with you, both to do review and to bring you up to speed on my changes
<abentley> jcsackett:   If you base your branch on the other branch, it's much clearer what came from what.  Why did you only bring over the webservice bits from the other branch?
<jcsackett> abentley: prereq was based on devel, this is db-devel. didn't want all the changes coming across before pqm brought them over automatically.
<LPCIBot> Project windmill build #95: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill/95/
<benji> leonardr: sure; I'm deep in something at the moment; how about after lunch (1 ish)
<abentley> deryck: Do you want to change branch so it only uses edit, or productseries so it uses add and edit?
<jcsackett> abentley: in retrospect, questions-on-webservice probably should have just been based on db-devel as well.
<leonardr> o
<leonardr> k
<deryck> abentley: can productseries be none intially?  (I assume so)
<deryck> abentley: if so, that should behave like branch then.
<abentley> deryck: Yes, and this is the most important case.
<deryck> abentley: yes, so it should get an add icon to start with, I think.
<abentley> adeuring1: ^^ could you do this as part of your work, please?
<adeuring1> deryck. abentley: yes.
<abentley> adeuring1: thanks.
<danilos> leonardr, looks good, r=me
<leonardr> great
<abentley> jcsackett: I wouldn't worry overmuch about applying the changes before PQM brings them across automatically.  You might get a criss-cross merge, but --lca or --weave merge should handle it fine.
<jcsackett> abentley: dig.
<jcsackett> if you like, i can merge the prereq completely in before you review.
<jcsackett> abentley ^
<abentley> jcsackett: I don't think I need that.  It just stuck out as weird, and I didn't want to start reviewing until I understood what was going on.
<jcsackett> abentley: got it. sorry for the oddities.:-)
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<abentley> jcsackett: For your tests, did you consider using WebServiceTestCase?
<jcsackett> abentley: i did not. i am not aware of WebServiceTestCase
<abentley> jcsackett: It allows you to test your code using launchpadlib.
<abentley> jcsackett: So you're testing something closer to what users will actually use, and it's closer to native code, too.
<sinzui> jcsackett: I had a few notes about the branch too. I added them to the MP
<jcsackett> abentley: seems worth looking at then. :-)
<jcsackett> there are a lot of tests i've seen that could benefit from that.
<abentley> jcsackett: it's similar to what you're doing in TestSetCommentVisibility, only generic.
<sinzui> jcsackett: wgrant. allhands should know about your participation in the 2011 campaign now.
<abentley> jcsackett: Why are you testing the XHTML?  This seems like your are testing the webservice implementation, not your exports.
<jcsackett> abentley: it looks like even the "clean" diff i pasted in has some noise--if you're looking at TestQuestionRepresentation, that's from the other branch. sorry.
<abentley> jcsackett: yes, I was.
<jcsackett> i'm not sure why that made it into the diff i generated.
<sinzui> abentley: jcsackett: this testing point relates to my experience with anon users and representations. The simple doctests for the representation easily documents what anon users can see
<sinzui> jcsackett: and about my anon + IQuestionMessage remark on the Mp, may be we do not want them public, see bug 740208
<abentley> jcsackett: Was there a preimplementation discussion?
<jcsackett> yes, sinzui and i have talked a lot about how to go about this.
 * jcsackett sees he skipped that bit in the proposal.
<abentley> jcsackett: 1 out of 1 abentleys recommend lp-propose for all your merge-proposing needs.
<stub> abentley: Are lp-propose bugs bzr bugs or some other package?
<abentley> stub: bzr, I would say.
<jcsackett> abentley: i have actually used lp-propose before, and while it opens a local editor and then pushes the whole thing to lp, it doesn't do so from a template. is there a config step i missed for that?
<abentley> jcsackett: You may be missing the lpreview-body plugin.
<jcsackett> i am indeed.
<abentley> jcsackett: try apt-get install bzr-lpreview-body
<jcsackett> i have it now. i'll see if alls well on the next branch.
<abentley> jcsackett: cool.
<benji> leonardr: ready when you are
<leonardr> benji, let's go
<leonardr> have you read the mp?
<benji> looking at it now
<benji> leonardr: ok, the MP makes sense
<leonardr> benji: and you understand the basic require_explicit_versions rule, right?
<benji> leonardr: yep, I believe so (its intent if not yet its implementation)
<leonardr> ok, take a look at apihelpers.py to see how i hack into the implementation
<benji> why did you have to add all the patch_entry_explicit_version calls in _schema_circular_imports?  Is it really circular imports or something else?
<leonardr> benji: it's something else, but it's the same kind of 'patch everything' logic
<leonardr> i'm ok with moving it though the imports would be a pain
<benji> just I wonder if making a new module and copying over all the imports and then running that through the linter to show unused imports would work well enough to make it worth it
<benji> would those ideally be in the same file where the interface is defined?
<leonardr> benji: yeah, but if you're going to go through that much trouble it approaches the amount of work necessary to just stop using it
<benji> that makes sense; I'm cool on leaving them where they are.  I might try the move-and-lint trick if I get 15 minutes.
<bigjools> night folks
<abentley> deryck: chat?
<deryck> abentley: I can't sorry.  Sorry.  Having to pull double duty now, since wife just came home sick.  Kids need moderate supervision.
<abentley> deryck: okay.  Will want to chat tormorrow about branch picker and vocabulary context.
<deryck> abentley: sure.  we can do it first thing tomorrow.
<sinzui> jcsackett: mumble?
<jcsackett> sinzui: oh, it is 3. one sec.
<jcsackett> abentley: my earlier mp is updated, both with new diff and comments.
<abentley> jcsackett: cool.  I'll have a look.
<sinzui> jcsackett: http://pastebin.ubuntu.com/584448/
<jcsackett> abentley: ws_object is now used in the test, and your question about endInteraction was answered.
<LPCIBot> Project devel build #570: STILL FAILING in 4 hr 53 min: https://lpci.wedontsleep.org/job/devel/570/
<lifeless> gary_poster: hi
<lifeless> gary_poster: I noticed this - https://bugs.launchpad.net/launchpad/+bug/741234 - by chance when looking at a timeout
<gary_poster> lifeless hi
<lifeless> gary_poster: I don't know if you want to pick it up or not, I may well be misremembering which squad changed the visibility rules
<_mup_> Bug #741234: product:+code-index merge proposal queries do not show private bugs visible by assignment <Launchpad itself:Triaged> < https://launchpad.net/bugs/741234 >
<lifeless> [I probably am misremembering]
<gary_poster> lifeless, I don't *think* we touched that...it certainly doesn't ring any bells
<lifeless> gary_poster: de nada :)
<gary_poster> :-)
<jml> lifeless: Red squad will be doing disclosure/visibility once one of the current squads finishes up
<lifeless> jml: cool
<lifeless> jml: this was missed during a change in the last few months which is why I mentioned it
<lifeless> anyhow
<lifeless> -> dentist
<jml> \o/
<sinzui> jcsackett: how goes CVEs?
<jml> https://code.launchpad.net/~jml/launchpad/test-timeout-505913/+merge/54598 â fix for an intermittent failure (I think)
<jml> wgrant: ^^
<jcsackett> sinzui: i got derailed by a hardware issue, but i think what we chatted about is showing promise.
<jcsackett> i'm having to fight my desire to do something about the nested chain that is the various nomination-rows &c.
<sinzui> jcsackett: are you working with BugLinksListingView ?
<sinzui> I see the pairing of the view to the template is broken. That view ensures can_view_bug is available to avoid security checks, but the template ignores it in two cases. I think that view should also use the trick I showed you to change the security checker on the object so subsequent checks are fast
<jcsackett> sinzui: how can you tell it is broken? i recall odd behavior when i was looking at this last which that might explain.
<jcsackett> though i just have the recollection of said behavior, and can't recall exactly what it was.
 * jcsackett needs to take better notes.
<sinzui> jcsackett: view build a dict of bug info to show. It recovered from an Unauthorized error and adds 'can_view_bug': False. The template does not need to call required:launchpad.View on the bug!
<sinzui> jcsackett: I am sure the adaption to +bugtasks-and-nominations-table also calls permission checks. Since the view has done the work, we want a fast permission checker on the good ones. We can update the cve-index.pt template to always use can_view_bug
<jcsackett> sinzui: dig.
<sinzui> jcsackett: I can send you a diff of the changes actually, so you can have an adventure with milestones
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<jcsackett> sinzui: i'm not sure that's what i would call "adventuring" :-P
<wallyworld_> leonardr: mumble?
<jml> g'night all.
<leonardr> wallyworld, 1 minute, sorry
<wallyworld_> np
<leonardr> wallyworld, i can hear you
<leonardr> wallyworld, go ahead and give me your status, i'll try to figure it out
<lifeless> sinzui: jcsackett: are you talking bug security checks?
<lifeless> sinzui: jcsackett: if so there is no need for that now, our searches inject the rules.
<leonardr> wallyworld: sorry, still not working
<lifeless> jml: I wrote the fixture you asked for
<sinzui> lifeless: okay good to know. We could reimplement the oddness in the view
<lifeless> sinzui: what symptoms are you seeing ?
<sinzui> No symptoms. The does very odd things to build a dict of bugs, and the template ignores the work the view did hald the time. We have bool in the dict, which must be faster than adaption
<lifeless> sinzui: any bug task gotten via IBugTaskSet.searchTasks will not trigger security lookups; bugs gotten via other methods that do not indirect to searchTasks should be updated to do so
<lifeless> sinzui: ok
<jml> lifeless: yeah, I saw, thanks.
<leonardr> wallyworld_: take a look at lazr.restful.interfaces.IWebBrowserOriginatingRequest
<leonardr> launchpad defines an adapter from WebServiceLayer (ie. a web service request) to IWebBrowserOriginatingRequest, in webapp/configure.zcml
<leonardr> in _resource.py, applyChanges, lazr.restful casts the current request to IWebBrowserOriginatingRequest
<leonardr> if the result is None (because the application never defined an adapter), one thing happens
<leonardr> if it's not none (because this is launchpad), another thing happens
<wgrant> jml: !
<wallyworld_> leonardr: thanks!
<leonardr> np
<rockstar> huwshimi, hi
<huwshimi> rockstar: Hey
<rockstar> huwshimi, what particularly about YUI are you struggling with?
<huwshimi> rockstar: Oh, nothing in particular (I've just replied to your email).
<rockstar> huwshimi, yeah, I saw it.
<huwshimi> rockstar: I just don't like the way it does certain things, but I can get over that and use it anyway
<rockstar> huwshimi, most of the concerns I've seen from people about YUI (and javascript in general) are because there's something idiomatic that they don't understand.
<rockstar> huwshimi, what particularly do you not like though?
<rockstar> (It's quite possible that we are doing it wrong it YUI, and I'd like to help fix it if need be)
<huwshimi> rockstar: I'm not sure exactly. I need to spend some more time with YUI3 to really be able to help you out with that
<sinzui> wgrant: I think you should block about the private bug as 404 change. Many users are confused by this. They think Lp is broken (making bad links)
<sinzui> wgrant: s/block/blog/
<cody-somerville> In that case why is it 404 instead of 403?
<wgrant> cody-somerville: We are moving to deny the existence of private objects, because it is much simpler. :/
<wgrant> Private bugs are just about impossible to do sensibly.
<wgrant> Without 404ing.
<wgrant> (because of the redirects)
<sinzui> wgrant: actually, I think switching to NotFound is a bug. We cannot make links that will 404. We need to ensure that the user has permission to see the link we are making
<sinzui> wgrant: and yes, because of redirects this can be near impossible
<wgrant> sinzui: We don't make links, AFAIK.
<wgrant> Hmm.
<cody-somerville> what redirects?
<wgrant> Except perhaps if the bug is referenced in text.
<wgrant> cody-somerville: /bugs/1 -> /ubuntu/+bug/1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<wgrant> cody-somerville: /otherproject/+bug/1 -> /ubuntu/+bug/1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<poolie> good thing that line is truncated :)
<sinzui> wgrant: bug https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/739380 certainly does link to a private bug. I am sure it shows up in the 404 reports
<_mup_> Bug #739380: update-manager crashed with AuthorizationFailed in _run(): org.freedesktop.PolicyKit.Error.Failed: ('system-bus-name', {'name':  ':1.129'}): org.debian.apt.update-cache <apport-crash> <i386> <natty> <running-unity> <update-manager (Ubuntu):New> < https://launchpad.net/bugs/739380 >
<sinzui> wgrant: I think this is another case where the is a right way to do something, but it is more fun to creaft your own links and play with interpolation
<sinzui> sorry about the spelling. I am a wreck. I should end my day
<lifeless> sinzui: we don't create links to hidden bugs though
<cody-somerville> why not just handle this the same way as oh, I dunno... most other websites? :P lol. Sorry to be facetious but LP can't be the first website trying to solve these kinds of problems, can it? :P
<lifeless> sinzui: (because we filter all hidden bug objects so they don't show up in miletones, search results etc)
<sinzui> lifeless: But we do https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/739380
<_mup_> Bug #739380: update-manager crashed with AuthorizationFailed in _run(): org.freedesktop.PolicyKit.Error.Failed: ('system-bus-name', {'name':  ':1.129'}): org.debian.apt.update-cache <apport-crash> <i386> <natty> <running-unity> <update-manager (Ubuntu):New> < https://launchpad.net/bugs/739380 >
<lifeless> cody-somerville: this is how other websites handle it
<sinzui> lifeless: look above the comment box
<lifeless> the 'Remember, this bug report is a duplicate of bug #710026. ' ?
<lifeless> sinzui: so, this is related to the disclosure work - remember how you were saying that we shouldn't permit adding tasks outside of visibility zones
<lifeless> sinzui: same thing - when something is duped on a private bug we need to take disclosure into account
<wgrant> lifeless: I thought the duplicate warning avoided linking. Maybe only some of them do.
<sinzui> lifeless: I think this relates to an even older rule I have when reviewing code. Do not craft a link that you are not will to write a test for to verify it behaves like the canonical link code.
<lifeless> sinzui: sounds plausible to me
<lifeless> interestingly
<lifeless> sinzui: I can see that private bug
<sinzui> your special
<lifeless> I'm not an admin
<wgrant> Any more, at least.
<lifeless> sigh, rub it in :)
<sinzui> I am just one of the proles.
<lifeless> I'm in ubuntu-bugcontrol
<wgrant> If it's apport, your MOTU membership will let you see it.
<lifeless> wgrant: ah
<lifeless> k
<wgrant> Indeed, it is apport.
<lifeless> sinzui: so you see a link there
<lifeless> sinzui: and you can't see the master bug
<lifeless> wgrant: looks to me like apport should have unprivated this
<sinzui> I see "Remember, this bug report is a duplicate of bug #710026. Comment here only if you think the duplicate status is wrong."
<lifeless> sinzui: and that is linkified ?
<sinzui> yes
<wgrant> lifeless: Indeed, nothing private there.
<lifeless> wgrant: worth filing a bug about this?
<wgrant> lifeless: Not sure.
<sinzui> I think this instance I see is bug. Isn't the reason the oops reports have a NotFound section is to encourage us to not do stupid thinks to frustrate users.
<lifeless> sinzui: indeed, we shouldn't generate links that break
<lifeless> we need to teach the linkifier code to rewrite bug references too
<lifeless> e,g. https://bugs.l.n./foo/+bugs/NNN -> check visibility and don't link otherwise.
<sinzui> The message I see is wrong. Since the bug is private and I cannot see it, I cannot say I think the duplication was a mistake. I have to know to locate a bug contact for the zero-vector bug I cannot see and ask for it to be make pulic of be subscribed
<lifeless> which also implies a batch scan-and-check for efficiency
<sinzui> That is an daunting task for most users
<lifeless> sinzui: I totally agree. I filed a couple of bugs about usability here earlier this week
<lifeless> sinzui: let me dig them up
<sinzui> oh, yes I do realised that my grievance is goes back to when we issued a 403. Users are just more confused since bugs privacy is volatile, unlike team privacy.
<lifeless> thumper: bug 740750 - I think I mailed you.
<_mup_> Bug #740750: API timeouts broken and returning no useful data... <regression> <Launchpad itself:Triaged> <lazr.restful:Triaged> < https://launchpad.net/bugs/740750 >
<lifeless> bug 334130
<_mup_> Bug #334130: misbehaviour in marking duplicates of a private bug <lp-bugs> <privacy> <Launchpad itself:Triaged> < https://launchpad.net/bugs/334130 >
<lifeless> bug 434733
<_mup_> Bug #434733: marking public bug as duplicate of private bug leads to confusing UI <confusing-ui> <disclosure> <lp-bugs> <privacy> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/434733 >
<lifeless> I can't find the one I thought I filed
<lifeless> bug  157899
<poolie> lifeless, oh just one more thing
<_mup_> Bug #157899: Shouldn't need access to private bug report to reverse a public duplicate marking <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/157899 >
<lifeless> bug 136570
<_mup_> Bug #136570: Can't unsubscribe from duplicates if dupe is private <duplicate-subscribing> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/136570 >
<poolie> i see people adding new apis for admin use etc
<lifeless> poolie: yes?
<poolie> it seems like it's kind of half-baked without a cli
<poolie> i don't want to hold things hostage but i was wondering if people could add things to lptools for admin use
<poolie> it shouldn't be much more incremental work
<lifeless> poolie: I don't think they should
<poolie> or, at least, to file a bugtask against lptools
<lifeless> poolie: following reasons:
<lifeless>  - these apis are not productised; we don't want to have to care about them in the stable api, and lptools is a productised project
<lifeless>  - the api client for them should be the web ui, not a cli
<poolie> oh
<lifeless> I think its fine if a dev things lptools is the best place to do it, but its not where I would encourage them to go with these apis
<poolie> are these meant to be internal-use things for ajax only?
<poolie> i misunderstood
<lifeless> poolie: we have /one/ api, it serves both launchpadlib (python) and a yui launchpad object that all the web ui ajax uses
<lifeless> so theres no clear separation, for right or wrong
<poolie> what does "not productised" mean in this context?
<lifeless> we don't care about API churn
<lifeless> we'll change them willynilly as needed
<poolie> what is the intended client for them? ajax?
<lifeless> I don't know
<lifeless> the syadmin request is a button in the ui
<lifeless> but the work is being done in spare timeslots
<poolie> ok
<poolie> i was confused because the bug seemed to say "please add an api"
<poolie> i'll add lptools bugtasks
<lifeless> the bug was added to reflect the unit of work, for tracking qa
<lifeless> poolie: if the api is in devel, which I htink it is, you may not want lptools using it
<lifeless> poolie: because lptools is a packaged product
<lifeless> I wouldn't want bugs filed on lp if we change this api
<poolie> ok
<poolie> better to have the function than not at all though
<lifeless> poolie: this doesn't seem like a thing that will broadly interest lptools users; I'm confused by your interest in it
<poolie> anyhow
<poolie> i had the misapprehension that people were adding an api so that other people could write ad hoc scripts to call it
<poolie> and in that case i would rather those scripts went into lptools
<sinzui> lifeless: one of those bugs may be a duplicate of a bug I reported 5 months ago. I would need to see a TB to be certain. Maybe Gary's team will fix the defect before they declare bug subscriptions ready.
<lifeless> they may be; even if that is the case, why is it appropriate for lptools ?
<poolie> it's more the general pattern than the specific case
<poolie> sorry i've got to go get a haircut, biab
<lifeless> poolie: I'm worried about the problem we have with edge and bzr happening here. Talk to you later about it.
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/434733/comments/4
<_mup_> Bug #434733: marking public bug as duplicate of private bug leads to confusing UI <confusing-ui> <disclosure> <lp-bugs> <privacy> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/434733 >
<lifeless> wgrant: it happens a lot, I think we need to talk to pitti
<lifeless> do we still send mail to all duplicates when a master bug is altered, even if the master is privte?
<wgrant> lifeless: I'm pretty sure not.
<wgrant> Since only direct subscriptions function for private bugs.
<lifeless> hmm, then I think I can close one of those bugs off completely
<wgrant> Oh?
<wgrant> Which?
<lifeless> bug 136570
<_mup_> Bug #136570: bug mail explaining cause of message has an inaccessible link when the the master is private and the subscription is is via duplicates <duplicate-subscribing> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/136570 >
<lifeless> don't need to unsubscribe if you don't get mail
<lifeless> bug 589878 looks nice, I was thinking of a branch of yours recently I'd love the bug subscribers to all be able to see
<_mup_> Bug #589878: bzr branches should be private by default if linked to a private bug <lp-code> <privacy> <Launchpad itself:Triaged> < https://launchpad.net/bugs/589878 >
#launchpad-dev 2011-03-24
<lifeless> wgrant: what do you think? closable?
<wgrant> lifeless: Needs testing. If it's not closable, then it's a security bug.
 * spm needs to fix his losa highlight. "closable" shoudn't be a highlighter....
<lifeless> how about closeable?
<wgrant> lifeless: It's possibly from back in the days when one got notified of duplicates of the master bug if one was subscribed to a duplicate.
<lifeless> :P
<spm> ha
<lifeless> wgrant: yes, its old; thus my question ;)
<lifeless> wgrant: are you on launchpad-security now?
<lifeless> wgrant: if not that would explain your confusion when I mentioned the bug before
<wgrant> lifeless: I have been for a while.
<wgrant> I get launchpad security mail.
<lifeless> sinzui: I guess you're off to family time, just saw your reply to my review; if you do want to discuss it now, I'd be delighted
<lifeless> otherwise, if I don't hear from you I'll reply as much as I can think of via mail and look for you my morning tomorrow
<lifeless> hmm
<lifeless> where are my daily oops reports for prod :(
<lifeless> no landings to devel since tuesday? wtf
<poolie> hi lifeless
<lifeless> hiya
<poolie> shall we talk now? pots?
<poolie> schooners?
<lifeless> sure
<lifeless> skype is easiest for me
<poolie> lifeless, https://bugs.launchpad.net/bugs/739772
<_mup_> Bug #739772: Question should be available via the API <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/739772 >
<lifeless> jcsackett: you're going to hate me :)
<lifeless> jcsackett: https://bugs.launchpad.net/launchpad/+bug/741440
<_mup_> Bug #741440: merge proposal diff only shows once per page load <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/741440 >
<LPCIBot> Project windmill build #96: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill/96/
<StevenK> Windmill, you make me sad.
<StevenK> Actually, gina's doctest makes me positively suicidal.
<StevenK> I think I broke it, gina has been running for 11 minutes.
<StevenK> No, it's just OOPSing constantly
<wgrant> :( fake forms
 * StevenK burns gina.txt, and then douses it with kerosene.
<wgrant> Watch out for gina-multiple-arch.txt
<wgrant> It may avenge its brother.
<StevenK> Haha
<StevenK>   Ran 1 tests with 1 failures and 0 errors in 18 minutes 25.669 seconds.
<lifeless> ok, time to write more deploy scripts
<wgrant> Hmm. Lots of theme updates.
<wgrant> New Ubuntu panel logo, new speaker icons, messaging indicate is now blue...
<wgrant> Wallpaper is... very slightly different.
<wgrant> What.
<wgrant> It is almost unnoticably different :/
<StevenK> Then how did you notice?
<wgrant> "almost"
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #571: FIXED in 5 hr 22 min: https://lpci.wedontsleep.org/job/devel/571/
<StevenK> Uhhh, why is gina's doctest testing Python math?
<lifeless> because twisted numpy accelerator for ssh changes global fnuctions
<StevenK> That's so not related.
<StevenK> lifeless: Can haz review?
<StevenK> Methinks that means 'No'
<lifeless> StevenK: check your mail
<StevenK> lifeless: Thanks
<poolie> do mps no longer show the diff inline in the page and only in the popup?
<poolie> or is this just the problem that the diff is sometimes missing?
<wgrant> The diff should always be there.
<lifeless> poolie: mps don't have a popup; bug pages referencing mps have a popup
<poolie> bug search timed out again
<poolie> ok, i think it was just https://bugs.launchpad.net/launchpad/+bug/734629 then
<_mup_> Bug #734629: empty/useless diff shown for merged proposals <code-review> <confusing-ui> <mail> <Launchpad itself:Triaged> < https://launchpad.net/bugs/734629 >
<huwshimi> lifeless: I have a branch that adds an ajax time log. Here's a screen cap: http://people.canonical.com/~huwshimi/ajax_time_log.ogv
<huwshimi> lifeless: I haven't figured out a way to add a useful identifier to that log yet though
<huwshimi> lifeless: I'd like some suggestions about what other useful info you'd like to see there
<lifeless> huwshimi: ----awesome----
<lifeless> huwshimi: url
<lifeless> huwshimi: and if it oopses the oops id
<poolie> huwshimi, wow
<huwshimi> lifeless: Yeah the url is the one I was trying to get before and couldn't. I can get back headers and data though so I should be able to oops info etc.
<huwshimi> lifeless: I'll have a look again at the url
<lifeless> perhaps we could stash the url in a response header
<huwshimi> lifeless: well I'll try without that first, but it's good to know we can do that
<stub> BinaryPackageReleaseDownloadCount is updated just over 8 times a second.
<lifeless> 24M rows on qastaging.
<lifeless> wow
<stub> That is a batch job or live?
<lifeless> batch I believe
<stub> yer... I was wondering if you can translate that directly to downloads, but I don't think it does
<StevenK> That's the job that parses librarian logs, I thought
<stub> About a million downloads counted per day
<lifeless> stub: hows the dbr looking - have we pushed the numbers down at all recently ?
<stub> Which I think means every line counted results in an update. That could be much more efficient if we sacrificed a little RAM.
<wgrant> stub: We batch the updates.
<stub> lifeless: Roughly the same. Some wins - bug notification has dropped off from chewing ~17% of a core to <1%
<wgrant> stub: At least it's meant to.
<wgrant> We build up a massive dict.
<stub> wgrant: But the same row might be updated multiple times in the run?
<stub> wgrant: Or does the dict count everything, then the updates are made at the end?
<wgrant> stub: It's possible, if the package has been copied.
<stub> archivepublisher has dropped off the cpu chewing list too. Was it split into components, or just made awesome?
<wgrant> stub: Just made a bit less awful.
<lifeless> stub: when was it doing 17%?
<LPCIBot> Project windmill build #97: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/97/
<wgrant> stub: Each BPRDC should be updated at most once in each batch.
<stub> https://devpad.canonical.com/~lpqateam/dbr/weekly/db-report-2011-03-11-2011-03-18.txt from 2 months ago is what I'm using as a baseline
<lifeless> stub: thats probably when we had the bad filter code
<jtv> stub: what say you I lower statistics-logging in LoopTuner to DEBUG2?
<stub> wgrant: Ok. That makes sense. If it becomes a problem, we could count everything first then do the updates in batches after the count is in.
<lifeless> we're going to put the filters back in on monday
<stub> jtv: I'm not fussed. If it makes things easier to work with, go for it.
<wgrant> stub: We count everything first, but I guess Storm updates each row separately.
<jtv> thanks stub
<stub> wgrant: oic. It is just there are 1000000 rows to update :)
<wgrant> stub: Heh, not quite that many, I hope.
<lifeless> wgrant: batch update: insert into temp table (VALUES); select into ...
<stub> Hmm... no... only 250k. So each tuple gets updated 4 times per day. Not sure 100% if that translates to 1 insert and 3 updates.
<stub> Anyway... not anything to worry about atm....
<stub> So 'less aweful' translates to a drop from 70% of a CPU core 2 months ago to no longer being on the list at all.
<wgrant> stub: Huh, nice.
<stub> appservers have dropped by 10% too, but that might not be statistically significant.
<wgrant> stub: ew, distributionmirror shouldn't be that high.
<stub> wgrant: no, and I think I have open bugs from me bitching about it in the past ;)
<wgrant> stub: Where are you seeing the archivepublisher drop?
<wgrant> It's still on yesterday's, at 70%...
<bigjools> morning people
<wgrant> Morning bigjools.
<stub> Ahh crap.... please reverse everything I just said. The tabs where reversed :)
<stub> So I guess the old lucille == archivepublisher and no change there
<wgrant> stub: lucille turned into archivepublisher
<wgrant> With no significant change.
<wgrant> Right.
<stub> And the appservers are chewing 10% more
<stub> And bug notifications sucks dogs balls.
<wgrant> Indeed, bugnotification is 40x what it was.
<wgrant> Yellow!
<lifeless> stub: this is worth a mail to the list +gary -> they want to add the mass filters to prod again; this high utilisation is a red flag for that
<stub> k
<jtv> wgrant: do we still need commercial-compat.sh?
<wgrant> jtv: Yes, until Dapper dies in three months.
<jtv> Three months, egad
<bigjools> run-parts ...
<jtv> hi bigjools.  wgrant: it's run at a somewhat awkward point in cron.publish-ftpmaster, and things would become a lot simpler if I could move it up or down just a bit.
<jtv> So I was wondering: does it have to be run after the new dist directories are moved into place?
<jtv> Or alternatively, does it have to be run before the ls-lR.gz files are generated?
<wgrant> It should be able to run after ls-lR, I think.
<jtv> bigjools: seeing 4 streaks of runparts invocations at the moment, 3 of which run twice.  Looking at how that number might be reduced.
<wgrant> Let me check.
<wgrant> runparts?
<bigjools> jtv: hmmm we can surely consolidate some
<jtv> I should hope so
<bigjools> I thought there should only be 2
<bigjools> remember that run-parts orders stuff if you want
<jtv> wgrant: I'm rewriting cron.publish-ftpmaster in python, and the plan is to "outsource" a lot of the work to run-parts directories.
<wgrant> jtv: Ah.
<jtv> bigjools: yes, but there's stuff inbetween that doesn't look like it belongs in run-parts.
<bigjools> it's more like adding hooks for non-generic stuff to live in
<jtv> Such as moving the dists directory into place: can we move that into run-parts?
<bigjools> no, that's generic
<bigjools> but we might be able to move things around
<wgrant> jtv: So, commercial-compat has to run before ls-lR, and after dists
<wgrant> jtv: Unless you make it work on dists.new instead.
<jtv> :/
<jtv> oh
<wgrant> Or delete partner
<bigjools> it's odd that nobody has had errors about incomplete PPA dists
<bigjools> since we don't do them atomically
<wgrant> bigjools: buildds do occasionally.
<wgrant> But they update infrequently and quickly.
<bigjools> ah interesting
<bigjools> on another note, isn't it a royal pita to see your ec2 land succeed and then pqm is in testfix!
<wgrant> Yes, someone left it in testfix overnight :(
<bigjools> :/
<wgrant> It failed like half an hour after I last checked it, and was failed when I started this morning...
<wgrant> Which makes me rather sad.
<bigjools> StevenK: hello
<jtv> bigjools, wgrant: right now the basic structure looks like http://paste.ubuntu.com/584690/
<wgrant> jtv: You will ideally be able to move signReleases into publishDistro (for PPAs it's already there)
<jtv> 4 stretches of "runparts" code, which is a bit much.  The smallest change I see that would help with that is to move installDists() down one step.
<wgrant> We can also hopefully get rid of copyIndices, because it's stupid.
<wgrant> What's installDists?
<jtv> Moving signReleases into publishDistro doesn't help.
<bigjools> why is 4 too much?
<jtv> wgrant: installDists moves the dists directories into place.
<wgrant> jtv: So, installDists and removeUncompressedListings and signReleases all go into publishDistro.
<jtv> bigjools: there's no such thing as "too" much, but maintenance is going to be easier when it's fewer.
<bigjools> why?
<jtv> Because it's hard to explain exactly what each of those 4 run-parts directories is going to be fore.
<bigjools> so naming stuff is hard? :)
<wgrant> jtv: I think there's only one run-parts.
<jtv> wgrant: what about copyIndices?
<wgrant> Post-publication stuff.
<wgrant> jtv: Does that put stuff in dists?
<jtv> Err
<jtv> â¦
<wgrant> That would be a no.
<jtv> if [ "$LPCONFIG" = "$PRODUCTION_CONFIG" ]; then
<jtv>     echo "$(date -R): Copying the indices into place."
<jtv>     rm -f $INDICES/override.*
<jtv>     cp $OVERRIDEROOT/override.* $INDICES
<jtv> fi
<wgrant> So it doesn't have to run before installdists.
<jtv> is what it does.
<wgrant> So we have this:
<wgrant> publishDistro
<wgrant> run-parts
<wgrant> The end.
<wgrant> "if not self.options.security" is implemented by the Python script calling run-parts with an env-var which tells them whether it's a security run or not.
<jtv> Well, it's not quite that simple of course because we need to do most of this twice per script run.
<wgrant> Long-running scripts can avoid running if the security flag is set.
<wgrant> We could also pretend that the concept of a security run doesn't exist.
<wgrant> jtv: Why must we do it twice?
<jtv> To expedite security uploads.
<wgrant> Is that actually used?
<jtv> You're in a better position to answer that than I am.  But it's in the original script and it's in the design for its replacement.
<jtv> So you could say it's already automatically used, but we don't know whether we rely on it in any way.
<wgrant> Keep in mind that this script was written in 2005. Most of the assumptions it makes can probably be brought into question.
<wgrant> eg. dsync kill it with fire.
<jtv> That's exactly what I'm doing here: questioning the assumptions.
<jtv> dsync goes into run-parts where it's not my problem.
<wgrant> True.
<jtv> Signing of releases is something I'd like to divest myself off as well though.
<wgrant> Why?
<wgrant> We already have the code to do that.
<wgrant> But the primary archive doesn't use it.
<bigjools> I don't want jtv's job complicated with that right now
<jtv> We already have the code to divest ourselves of it?  Or we already ahve the code to sign releases?
<wgrant> jtv: The signing code.
<wgrant> PPAs.
<bigjools> by all means file a bug and do it, but not in this change
<jtv> I also imagine there's the variable of what GPG key is to be used.
<wgrant> I think we should be fixing publish-distro before we try to make its wrapper sensible...
<bigjools> I disagree
<wgrant> The wrapper can be a lot more sensible if it doesn't duplicate functionality that's already in publish-distro.
<jtv> What I'm doing is at least bringing out the structure in what the old script does, and allows us to re-think these assumptions.
<lifeless> bigjools: btw - we got the go ahead to combine the soyuz (s)ftp services and uploader into one
<jtv> Also, at least now we can run tests.  In temporary directories.  Without global setup and writing to /srv/launchpad.net.
<bigjools> lifeless: I am fearful but ok
<lifeless> bigjools: we can always split it into two again - or optimise
<bigjools> lifeless: this will break the "you get an email in a few minutes" thing more often
<lifeless> but it will be a simpler initial config, which is important at the moment
<bigjools> lifeless: did you ask anyone why they were split like that in the first place?
<lifeless> bigjools: you asked elmo, I didn't see a reply from him
<bigjools> that's a no then? :)
<lifeless> bigjools: if he doesn't know, I suspect noone is left that knows ;)
<lifeless> bigjools: we can dig further - I've no objection to more info
<bigjools> lifeless: I think it would be wise
<lifeless> bigjools: the main point for me is that we have the stakeholder buy in to do whatever tradeoff make the most sense for now
<bigjools> just from being burned in the past by seemingly innocuous changes like this
<lifeless> bigjools: this one seems pretty big to me :) - if we do it it would be staged, easily reversible etc
<bigjools> lifeless: did we get a reply from platform?
<lifeless> bigjools: silence (neither a 'whaaa' nor anything specific)
<bigjools> I think a direct question would be good then
<lifeless> bigjools: marjo I think, said to me that the mail was forwarded to platform directly
<bigjools> they may not have seen it
<lifeless> bigjools: he cc'd me; I think they have seen it - he pulled out the TL;DR summary for us
<bigjools> what is this tl;dr fad
<lifeless> bigjools: I'll check with elmo and flacoste and so on when we've finished the single-threaded stack
<lifeless> bigjools: too long didn't read?
<bigjools> yes I know what it means :)
<lifeless> :P
<bigjools> lifeless: anyway, go ahead with the change, I've raised any concerns I may have
<adeuring> good morning
<lifeless> bigjools: I'll drag you into it when we get up to making the uploaders HA during deploys - i don't think any decision needs to be made yet.
<lifeless> bigjools: I'm just happy we have a broad set of options
<bigjools> lifeless: indeed!
<jtv> wgrant: what does dsync-flist do exactly?
<bigjools> options are good
<bigjools> that's why I run KDE
<wgrant> jtv: hardlinks common files.
<wgrant> jtv: It saves roughly a megabyte, I believe.
<bigjools> ha
<jtv> wgrant: sorry just to be clear, that's a megabyte of L2 CPU cache right?  ;-)
<wgrant> Heh
<jtv> Soâ¦ could I just simply kill it, drop it from the script, do away with it, no questions asked?
<bigjools> I doubt anyone would notice it missing
<jtv> Gone.
<jtv> Don't security uploads need to germinate?
<bigjools> no
<jtv> (And by the way, triggering mirrors looks like a prime candidate for "&" in the run-parts script)
<jtv> Germination is not generic to all distros, right?
<wgrant> Most will probably want it. But they should run the script themselves, since it's fairly specific, and I wish it would get out of our tree.
<bigjools> jtv: mirror trigger is very quick
<bigjools> germination is distro-specific
<bigjools> that script is not really owned by LP even though it's in our tree
<jtv> Oh, ISTR back in 2006 we tried to run some of this stuff on a non-Canonical machine and it just froze up trying to connect to each mirror.
<jtv> So I sort of remembered it as being slow.
<jtv> bigjools: what was your verdict on signing?  run-parts or generic?
<bigjools> jtv: run-parts for now since it's less work and won't mess up this change.
<jtv> Well I'm not 100% sure it's less work.  I see 2 options:
<jtv> 1. Do this in run-parts, and have another script in the same dir that finds and removes the uncompressed Sources/Packages files.
<jtv> 2. Implement it in the script itself, and have one run-parts directory less.  (The removing of uncompressed files could also go into the script, or into the next run-parts stretch)
<wgrant> bigjools: Oh yeah, it turns out that delayed copies are now much slower than direct copies.
<bigjools> wgrant: !
<bigjools> "delayed" would imply that :)
<wgrant> Since delayed copies still scale by number of BPRs.
<wgrant> Heh
<bigjools> so if you can guarantee that the security team can unembargo a kernel using a direct copy, please remove delayed copies
<bigjools> we can get rid of that re-upload crap too
<wgrant> Can't yet. Need to move notifications and overrides into direct copies :(
<wgrant> And that, yes.
<bigjools> we can get rid of that anyway
<wgrant> I'm going to see if doing constant-query overrides is easy.
<wgrant> If so, I'll JFDI.
<bigjools> hooah
<mrevell> Hey
<jtv> hey!
<wgrant> bigjools: I guess this will also have the convenient effect of stopping copy-package from breaking the archive.
<poolie> hi all
<henninge> Hi adeuring!
<henninge> adeuring: your soyuz translations upload branch landed! Good job on finding the right spot. ;-)
<adeuring> henninge: thanks :)
<henninge> adeuring: Unfortunately it seems to be doing a little too much ... :(
<adeuring> henninge: really?
<henninge> adeuring: we still need the *templates* to be uploaded
<adeuring> henninge: ah, right
<henninge> adeuring: so addOrUpdateEntriesFromTarball needs to filter only for pot files.
<adeuring> henninge: right
<henninge> actually, there is a function that checks that
<henninge> I was just looking for that
<henninge> adeuring: TranslationImporter.isTemplateName
<henninge> adeuring: you should be able to add a flag 'only_templates' to addOrUpdateEntriesFromTarball and use that in _isTranslationFile.
<adeuring> henninge: right
<henninge> adeuring: I am sorry but this really needs to be done before this gets rolled out. Let me file a bug for it.
<poolie> jml, hi?
<henninge> adeuring: bug 741571
<_mup_> Bug #741571: Translation templates should always be uploaded from soyuz builds. <regression> <upstream-translations-sharing> <Launchpad itself:Triaged> < https://launchpad.net/bugs/741571 >
<bigjools> wgrant: sorry was otp, to which breakage are you referring, for there are many :)
<wgrant> bigjools: The "oh, you wanted to copy a universe package from a PPA? let me just silently promote that for you" one, mainly.
<bigjools> ah
<henninge> adeuring: I touched that code, too, because I have removed the translationsharinginfo module. But that branch is just landing.
<adeuring> ok
<henninge> adeuring: are you busy now or should I fix that?
<adeuring> henninge: let's dice ;)
<henninge> adeuring: I roll a W20 ...
<henninge> ...
<henninge> 20!
<henninge> adeuring: I win ;)
<adeuring> henninge: 20*19*18... is a huge number
<henninge> :(
<henninge> oh
<adeuring> henninge: ok, so you'll fix the bug?
<henninge> adeuring: yeah, I just thought of some other optimization for my last branch that I could include.
<adeuring> henninge: cool, thanks
<dpm> hi jml, did you have the chance to have a look at signing up for a session on Launchpad for Ubuntu AppDeveloperWeek or finding someone else to run it?
<wgrant> bigjools: Your sync button is buggy.
<bigjools> ffs
<wgrant> bigjools: syncSources copies the latest published source in the archive, which could be, say, a security upload.
<wgrant> It does not discriminate by series or pocket.
<wgrant> :D
<bigjools> I expected I'd need to fix it later to sync the versions specified
<jtv> Any reviewers available?  https://code.launchpad.net/~jtv/launchpad/bug-741585/+merge/54674
<bigjools> wgrant is like the eye of Sauron
 * jtv tries to see a way that comparison works
<jtv> âbut fails
<wgrant> bigjools: Sadly not.
<bigjools> I have a vision of an eye sweeping the Launchpad landscape looking for bugs
<jtv> bigjools, you've been reading too much of that "LotR from Sauron's perspective" stuff
<bigjools> or maybe I feel like Frodo trying to sneak bugs past the eye :)
<wgrant> One does not simply land hacks into Mordor.
<wgrant> StevenK: Do you have a solution to your version issue? I see two possibilities.
<jtv> bigjools, another Q about publish-ftpmaster: we only need to call process-accepted once, despite the "run once for security, run once for everything" structure, right?
<jtv> stub, care to review my little logging cleanup?  https://bugs.launchpad.net/launchpad/+bug/741585
<_mup_> Bug #741585: FakeLogger doesn't quite conform to logging API <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/741585 >
<wgrant> :(
<jtv1> bigjools: if we won't be needing commercial-compat.sh for much longer, maybe I should just hard-code it in and we can drop it from the script later.  It actually makes things simpler.
<jtv1> (In the long run, that is)
<jtv> I can just add an "if we're doing ubuntu" guard around it, and add an XXX.
<deryck> Morning, all.
<stub> jtv: k
<jtv> thanks
<jtv> hi deryck
<stub> jtv: I thought I fixed it so our custom logger class is registered so we shouldn't see any 'standard' loggers?
<jtv> stub: but by "our custom logger class" you don't mean the FakeLogger, do you?
<jtv> Oh, I see what you mean
<jtv> you mean I could just call debug2.
<stub> That should work, yes.
<jtv> If that's safe, then I'll do it that way.
<wgrant> eeeeeep
<stub> If it doesn't work, there is a bug somewhere (retrieving a logger through some weird mechanism?), but that could wait for another branch
<jtv> stub: I take it that debug2 will call self.log(DEBUG2, *x, **y)
<stub> Yes
<jtv> I didn't run into a problem with that; I was just being conservative.
<jtv> Running tests.
<stub> Actually, according to lp/services/log/loglevels.py it is _log for reasons I don't recall but might be documented in the python reference
<stub> Or maybe it was the sourcecode...
<stub> Its registered with the logging module in lp_sitecustomize.py anyway
<jtv> stub: AttributeError: RootLogger instance has no attribute 'debug2'
<stub> With the normal logger or your FakeLogger with debug2() added?
<jtv> Probably the normal logger.
<jtv> I don't think my FakeLogger has debug2 though.
<jtv> There's nothing overriding the logger in that test.
<stub> You might need to add it, or make it inherit from LaunchpadLogger instead of the logging module class.
<jtv> Or call log() and have it just work.  :)
<stub> I fixed this so I didn't have to keep seeing log(DEBUGX, 'foo') everywhere ;)
<jtv> Also, I don't _think_ the FakeLogger has a debug2 method but I'm not sure
<stub> Yes, you will need to add debug2 to FakeLogger if it isn't there. If it isn't there, it is broken as it isn't a complete mock of the real thing.
<jtv> I could fix this up, but that would start to look like a rabbit hole and I'd prefer to be unblocked on my main job (which is a scary soyuz script overhaul)
<stub> You have a branch that mainly consists of making things less readable, when adding a three line method on FakeLogger would keep the readability and fix a bug.
<stub> Well.. not mainly. You have only changed two .debug2 callsites to use the old syntax.
<jtv> But the problem I just ran into is not the FakeLogger.  So fixing up the FakeLogger wouldn't fix that one; there is yet another bug to fix.
<stub> ok
<jtv> The logger I seem to be getting (and which doesn't seem to have debug2) is canonical.launchpad.scripts.log.
<stub> jtv: In what cases do you find you have None as a level?
<jtv> I'm not aware of any.  Not sure that deserves to be in there.
<stub> ok. Add a comment there mentioning that, or remove it.
<jtv> I'll remove it.  It's a bit of legacy crud from jml's merging of fake loggers.
<jtv> At least one of the old fake loggers printed "log>" instead of a log level.
<stub> jtv: So it looks like lp_sitecustomize.py installs the custom class, and instances are correctly returned for new loggers but the root logger isn't updated.
<jtv> I have no idea what you just said.
<stub> (testing with 'make harness')
<stub> The registration of our custom logger class works for everything except the root logger, which is bogus.
<jtv> Sounds like itâ¦ what can we do about it?
<stub> Surprised this hasn't fallen over before.
<stub> I'll have a quick poke to see if it is an easy fix, or open a bug which you can cite in your branch.
<jtv> Thanks.
<jtv> I've added the new log levels.
<bigjools> jtv: sorry just saw your questions
<jtv> tsk
<bigjools> jtv: p-a only needs to run once
<jtv> Running it from the script does introduce the PATH question again, of coursee.
<jtv> Oh, different question :)
<jtv> Thanks.
<bigjools> jtv: embedding commercial compat is ok
<bigjools> but file a bug of course :)
<jtv> Naturally.
<jtv> That brings us down to 2 run-parts dirs: after publish-distro and at the very end.
<bigjools> rock on
<jtv> For the latter, the scripts will be told in some way whether this is a security-only expedited publishing or a proper full one.
<bigjools> so post-publication.d and pre-exit.d ?
<jtv> I still need to check whether all of the parts break down neatly into archives.
<jtv> Something like that, I guess.
<jtv> bigjools: what about the renaming of dists to dists.new?  That looks like it needn't be repeated either.
<jtv> I should say: renaming of dists to dists.new plus subsequent rsync from distsroot.
<bigjools> mmmm
<bigjools> not sure
<stub> jtv: So https://pastebin.canonical.com/45170/ looks like the fix, and works with bin/harness. Want to dig the rabbit hole a little deeper and save me making a separate branch?
 * jtv waits for that page to loadâ¦
 * jtv waits for openidâ¦
<jtv> ah, there's my page!
<jtv> stub: I realize this is piling work onto you, which pains me, but I'd really rather not risk the potential Q/A fallout while I'm doing soyuz feature work.
 * stub opens a bug
 * jtv gets ready to xxx the bug number
<stub> Bug #741650
<_mup_> Bug #741650: root logger is not a LaunchpadLogger <Launchpad itself:Triaged> < https://launchpad.net/bugs/741650 >
<jml> jtv: thumper's merging
<stub> jtv: Copywrite in logger.py should be 2010-2011, not 2011
<jtv> my apologies
<jtv> stub: fixing
<henninge> adeuring: I am done. Want to review?
<henninge> adeuring: https://code.launchpad.net/~henninge/launchpad/devel-741571-filter-templates-in-upload/+merge/54688
<adeuring> henninge: sure
<henninge> adeuring: I'll be out to lunch now, you can ask questions later.
<adeuring> ok
<henninge> adeuring: but I think this is a pretty clear branch, I really enjoyed coding this.
<stub> jtv: Thats it. r=stub.
<jtv> stub: thanksâchanges pushed, on the way to EC2.
<stub> jtv: For lint issues like the one you mentioned, I usually go with whatever makes lint shutup.
<jtv> stub: me too, usually.  This is the one exception.  I won't add warnings like that, but I'm hesitant to clean them up.
<stub> jtv: What test are you using for this? I want to test my branch
<gary_poster> stub, thanks for heads up.  Remind me where the db utilization report is, please?  I'd probably like to talk to you about your concerns too, once I've gotten more familiar with the data.
<jtv> stub: I simply replaced the log calls in looptuner.py with debug2 calls and then ran looptuner.txt.
<stub> gary_poster: https://devpad.canonical.com/~lpqateam/dbr/
<gary_poster> thanks
<stub> jtv: ta
<stub> gary_poster: Its not causing problems yet, but it certainly should be looked into since it is a big jump and affects our scalability. Robert asked me to raise it because of the work I believe you are doing.
<gary_poster> stub, right.  We are doing more work now on the asynchronous side.  It is doing features that have been requested, of course.  Do you have any concrete suggestions on next steps?  The only thing I know of is to try to take a glance at the pertinent code and see if it can be simplified, but that seems fairly vague.
<stub> gary_poster: If we know what extra work it has started doing in the last two months, then we can tell if the extra load is justified or not. I don't know what it is doing - I just see the percentage.
<stub> gary_poster: I think Robert's main concern is that we are currently in a performance drive, and shouldn't be landing code that makes performance worse (which it seems we have)
<jtv> bigjools: do we copy overrides only for the main archive?
<gary_poster> stub, even asynchronously, for requested features?  How do we know that this is a problem?
<bigjools> jtv: copy?
<gary_poster> Amusingly to me, two months ago would be about the week when we started working on this, of course. :-)
<jtv> bigjools:     cp $OVERRIDEROOT/override.* $INDICES
<jtv> $INDICES is in the main archiveroot
<bigjools> jtv: overrides only apply to main archives
<jtv> Perfect.  Thanks.
<bigjools> so far anyway :)
<gary_poster> ah it's less than 2 mo though...
<jtv> Well, not perfect, but then I know what to implement.  :)
<stub> gary_poster: You would have to discuss features vs. performance drive with lifeless :-) I don't think it is affecting production performance yet, but it has lessened our scalability somewhat.
<gary_poster> ack stub.  ok thanks.
<stub> gary_poster: It might just be we are missing an index or similar if we can work out what has changed and what we are doing differently now. Or maybe we just have to put up with it, because the extra work is required for some reason.
<stub> gary_poster: The two months figure I got because today I compared this weeks report with a report from two months ago. It could have spiked any time between then and now.
<gary_poster> understood stub.  I have a suspicion I know the cause, and will investigate and report back on the list.  Thanks again for the heads up.  It started rising in early March.  https://devpad.canonical.com/~lpqateam/dbr/weekly/db-report-2011-03-04-2011-03-11.txt
<stub> (In fact, first of all I erroneously compared them in reverse and was singing the praises of whoever stopped bugnotifications from chewing on the cpu :) )
<gary_poster> lol
<stub> jtv: https://code.launchpad.net/~stub/launchpad/trivial/+merge/54692 (to try landing tomorrow after your branch)
<jtv> stub: I know, I'm already looking
<jtv> stub: does lp_sitecustomize get run by scripts as well?
<jtv> Not sure it matters,
<jtv> since we seem to have some logging magic for scripts anyway that probably already sets things right there
<stub> jtv: Its buildout magic. Everything in our tree gets it
<stub> (unless you forget python -S or _pythonpath)
<jtv> stub: approved
<stub> ta. I guess I can punt it to ec2 land to see what explodes now - if your branch fails, this one will fail too since I merged yours in.
<jtv> stub: looks like my instance never fully set up, but since my review is also approved, "ec2 land" should just land them both.
<jtv> (Unless it doesn't want to do that across users)
<stub> jtv: Actually, I'll wait. Better two separate landings in case mine needs to be backed out.
<jtv> OK
<jtv> I should get a chance to send it in tonight.
<jml> Can I get a review of this please: https://code.launchpad.net/~jml/launchpad/test-timeout-505913/+merge/54598
<abentley> deryck: chat?
<deryck> abentley: sure.
<deryck> abentley: let me head to the mumbler
<jml> 'ec2 test' is complaining about me not having an SSH agent running. But I do.
<jtv> bigjools: I'm about to head off.  If you look at lp:~jtv/launchpad/bug-55798 at the last dozen lines of lib/lp/soyuz/scripts/publish_ftpmaster.py you'll see the structure I have now, which is thankfully shaping up to be pretty simple.
<jml> maybe I need to set an env var or something
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
<jml> huh. my keys weren't in it. odd.
<wgrant> jml: Natty?
<jml> wgrant: indeedly
<wgrant> jml: Unity been crashing?
<jml> wgrant: Does a bear... well, you know the rest.
<wgrant> It sometimes doesn't restore the environment properly.
<jml> wgrant: ok.
<jml> anyway, running ssh-add makes everything better. just going to test my branch that fixes the intermittent test_timeout failure
<jml> maybe it'll be done by the time it gets reviewed.
<leonardr> abentley: if you can take another look at https://code.launchpad.net/~leonardr/launchpad/explicit-versions/+merge/54558, i can get it into ec2 land
<leonardr> wallyworld, you run tests with bin/test
<abentley> leonardr: looking
<abentley> leonardr: r=me
<gmb> jcsackett: Can I get a review of https://code.launchpad.net/~gmb/launchpad/fix-fragile-link-bug-741645/+merge/54698 please? Only 20 lines.
<jcsackett> gmb: sure thing.
<gmb> Ta
<jml> abentley: could you please review https://code.launchpad.net/~jml/launchpad/test-timeout-505913/+merge/54598
<abentley> jml: sure, but I've got a standup first.
<jcsackett> gmb: r=me.
<gmb> Thanks jcsackett
<jml> abentley: thanks.
<jml> why am I allowed to change translations settings for empathy (upstream) on lpnet but not qastaging?
<jml> (https://translations.launchpad.net/empathy vs https://translations.qastaging.launchpad.net/empathy)
<bigjools> jml, wgrant: I have to ssh-add every session because the stupid script assumes I'm running Gnome desktop
<jml> bigjools: "stupid script" == utilities/ec2?
<bigjools> jml: I'm not sure which part exactly, something to do with one of the included modules I expect
<bigjools> keyring wants to use the gnome agent as well, instead of the kwallet
<jml> I'm guessing this isn't https://bugs.launchpad.net/launchpad/+bug/577118
<_mup_> Bug #577118: ec2test barfs if the ssh-agent keys are stored in the wrong order <build-infrastructure> <ec2test> <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/577118 >
<bigjools> don't think so
<benji> bigjools: I've added a section to https://dev.launchpad.net/LandingChanges that has instructions for telling keyring to use unencrypted flat files for credential storage so you don't have to type in passwords or fiddle with keyrings
<bigjools> benji: yep, I saw that thanks.  I don't mind that so much as the problem with the ssh keys
<sinzui> jcsackett: ping
<jcsackett> sinzui: good morning.
<sinzui> jcsackett: do you have time to talk about a branch I am starting: http://pastebin.ubuntu.com/584441/
<jcsackett> sinzui: sure.
 * jcsackett starts mumble
<bigjools> deryck: so, looking at Selenium yet? :)
<deryck> bigjools: I would be interested in Selenium, definetly. :-)
<bigjools> :)
<bigjools> it was in reference to your last tweet/fb
<deryck> bigjools: other parts of Canonical are using it now.  Windmill is all but dead, except for us
<deryck> bigjools: right :-)
<bigjools> yeah, it's not developed any more even, I heard?
<deryck> right, it's pretty dead, I think
<deryck> I just don't have an easy path off of Windmill
<bigjools> except we have a twitching limb
<deryck> heh
<bigjools> I'll see if my team can help with this when we get to the feature review stage
<deryck> bigjools: oh, dude!  That would be wonderful.
<bigjools> don't get too excited, it's a month away or so :)
<deryck> heh, fair enough :-)
<bigjools> when we discussed it we were all of the opinion that we should not flog the dead
<deryck> I think the important thing is to get it added as an option for these sort of tests and then get an example test going.
<deryck> and not make the same mistakes again :-)
<deryck> e.g. maybe we should consider running them separtely from our normal tests, decoupled from the Zope test runner
<deryck> they have to block in some way to be useful, but exactly what that blocking mechanism is good be looked at, too.
<bigjools> I'd be willing to do that
<deryck> s/good/could/ (for my last one)
<bigjools> we need to look at how many of our landed branches would need that sort of test
<deryck> yes
<bigjools> worth an experiment!
<deryck> very much so!
<deryck> selenium would, I think, allow us to have real web page integration tests that were more stable, rather than the minimal XHR integration test I'm preaching for now
<bigjools> unfortunately I know next to bugger all about the whole JS environment so I can't be very helpful right now.  I aim to delve into this in more detail soon though.
<deryck> I'm just thrilled to have someone express an interest :-)
<bigjools> ha :)
<abentley> jml: what's _logTimeout for?
<bigjools> james_w: it struck me that there's nothing in the derived distros LEP about considering pockets when working out package differences.  Right now it'll just grab whatever was most recently published but I'm not sure if that will always be what's required.  Any thoughts?
<james_w> bigjools, could you expand please?
<bigjools> james_w: ok for example say the most recent upload is to -backports.  It'll show up as a diff but won't say it's from -backports.
<bigjools> you may not want to sync from backports :)
<james_w> hmm, indeed
<bigjools> so I'm trying to flesh out the use cases here so we can work out what to change to make this work better
<james_w> my initial guess is that you would want an item for each pocket that had a new version published
<bigjools> that's what I thought - you'd end up with multiple rows for the same package showing in the table of diffs
<james_w> "-security has <this> and -updates has <that>" and you can choose which to sync
<bigjools> yup
<james_w> if you chose the later one then the other would go away
<bigjools> interesting
<abentley> jml: it seems test_timeout_short has a lot in common with test_timeout.  Could we extract the common code to a helper?
<james_w> it's not quite right, but I don't see a better way
<abentley> jml: IIRC, raising an exception meant that the oops would contain a traceback showing what code had timed out.  It seems a shame to lose that.
<bigjools> james_w: I'll ask a few more people and get some consensus
<james_w> ok
<bigjools> cheers
<bac> gary_poster, jcsackett: either of you going to trizpug tonight?
<jcsackett> bac: i'm not sure. there's a decent chance of 'no'.
<gary_poster> no. previous commitment, bac (to painting a mural in the baby's room)
<bac> jcsackett: that's my inclination too
<jcsackett> bac: in my case, i need to contact one of the other splatspace members. he's technically the point man for the meeting, but is not always on time, and someone has to be there to open doors.
<bac> jcsackett, gary_poster: on an unrelated note, andrea from lanter is a james beard finalist.
<bac> lantern
<jcsackett> cool.
<gary_poster> cool :-)
<benji> Does allhands eat newlines in objective "Progress Notes" for anyone else.  It really hurts the readability of my text.
<jcsackett> bac: confirmed, i won't be there tonight.
 * jcsackett is having a braindead day.
<adeuring1> abentley: my branch lp:~adeuring/launchpad/js-translation now also shows the corect "add packaging" icon
<bigjools> you can't beat a good refactoring for satisfaction
<jml> abentley: have you had a chance to look at that branch?
<LPCIBot> Project windmill build #98: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/98/
<bigjools> jml: too late
<bigjools> I already submitted a fix
<abentley> jml: I did, and I asked you some questions about it.
<jml> bigjools: oops. missed your mail (Gmail put it into a different thread :()
<bigjools> jml: awesome Gmail :)
<bigjools> jml: except PQM just blew me out, grar
<jml> abentley: on the channel?
<jml> ahh yes.
<jml> abentley: logTimeout is for making sure the logged oops is a TimeoutError, not ProcessDone(42)
<jml> abentley: the tests are fairly short. I guess I could put the logger stuff in setUp and make a custom assertion. Not sure whether that would be better.
<jml> abentley: re <abentley> jml: IIRC, raising an exception meant that the oops would contain a traceback showing what code had timed out.  It seems a shame to lose that.
<jml> abentley: I agree, it is a shame. I'm not sure we can do that and avoid the race.
 * jml tries something
<jml> (although I don't really know how to verify the traceback
<jml> )
<abentley> jml: would it be crazy to log the oops in a signal handler?
<jml> abentley: I don't *think* so. The caveats about complexity that we discussed the other day would apply.
<jml> abentley: I was thinking of try: raise TimeoutError; finally: os._exit(TIMEOUT_CODE), or something similar.
<abentley> jml: I don't understand how that would result in a traceback in the oops.
<bigjools> right, time to make like a shepherd.  Good night all.
<jml> abentley: well, probably something more like finally: reactor.callLater(0, os._exit(TIMEOUT_CODE))
<abentley> jml: Ah.
<jml> err... reactor.callLater(0, os._exit, TIMEOUT_CODE)
<jml> abentley: so the error handling would get a chance to run, and if it doesn't then the next thing that happens is a hard stop.
<abentley> jml: works for me.  Works for you?
<jml> abentley: if it works at all :)
<jml> hmm. the test passes, but I suspect it's logging two TimeoutError oopses.
<jml> (really hard to actually check for that)
<jml> abentley: yeah. it logs two oopses in that case.
<abentley> jml: I guess we want that, since the first oops isn't guaranteed to work?
<jml> abentley: maybe. I honestly don't know. Two oopses seems less good than one. But maybe the extra information is worth it.
<abentley> jml: You could use a different exit status if the exception handler succeeds.
<jml> I don't know how to log oopses safely from the signal handler (both because of signal handling but also because logging an oops requires a substantial amount of infrastructure that might not be present, such as parsed config)
<jml> abentley: hmm.
<jml> abentley: I'll give that a try.
<jml> abentley: this is approaching the complexity of the "set some state" solution I discussed the other day, though
<jml> abentley: I can't make this work without making a mess.
<abentley> jml: I think getting the traceback is pretty important, so I can accept the two-oops solution.
<jml> abentley: OK. I'll add some comments.
<jml> abentley: http://pastebin.ubuntu.com/584950/
<abentley> jml: r=me.
<jml> abentley: thanks.
<LPCIBot> Project windmill build #99: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill/99/
<LPCIBot> Project windmill build #100: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill/100/
<leonardr> benji, i'm having a problem receiving zope events. can you help?
<leonardr> lp:~leonardr/lazr.restful/449561
<benji> leonardr: sure, looking now
<leonardr> look at EventTestCase in test_webservice.py
<benji> leonardr: ahh, the zope.component subscriber (the one that takes the super simple events from zope.event and dispatches the interface based events) isn't wired up in the test
<benji> let me see how you do that
<benji> well, that was at least part of the problem, but it's still not working
<leonardr> benji: i know it's working in webservice.txt, you might look at that
<benji> leonardr: here you go http://pastebin.ubuntu.com/585015/
<leonardr> benji, thanks, will look in just a minute
<benji> also, the zope.component.eventtesting.getEvents function would probably simplify your tests a bit (remove the need to register your own handler)
<lifeless> 'sender-time Sent at 4:56 AM (UTC). Current time there: 8:10 PM'
<deryck> later on, everyone.
<leonardr> jcsackett: very quick: https://code.launchpad.net/~leonardr/lazr.restful/fix-test-failures/+merge/54773
<jcsackett> leonardr: looking now.
<jcsackett> leonardr: r=me
<leonardr> jcsackett: i'm still getting a failure. it's that damn list of things that need to be sanity-checked
<leonardr> it needs to be cleared out all the time, not just for those tests
<jcsackett> leonardr: ok. feel free to ping me after uploading those changes.
<leonardr> jcsackett: ok, i think i know how to do it
<jcsackett> leonardr: how does one go about exporting something only into devel?
<leonardr> jcsackett: for an entry or field, you use as_of="devel"
<leonardr> for a named operation, you use @operation_for_version("devel")
<leonardr> this is once my branch lands
<leonardr> ah, it just landed, so my advice is accurate
<jcsackett> but until your branch lands it's not doable?
<jcsackett> ah, excellent.
<leonardr> before my branch landed it was doable but ugly
<lifeless> ok, logs have updated
<lifeless> I see 2011-03-24 20:07:25 INFO    Notifying robertc@robertcollins.net about bug 741821.
<_mup_> Bug #741821: "Mute bug mail" in new bug reports is odd <confusing-ui> <story-better-bug-notification> <Launchpad itself:Triaged> < https://launchpad.net/bugs/741821 >
<lifeless> and the initial one was
<lifeless> 2011-03-24 15:45:43 INFO    Notifying robertc@robertcollins.net about bug 741821.
<_mup_> Bug #741821: "Mute bug mail" in new bug reports is odd <confusing-ui> <story-better-bug-notification> <Launchpad itself:Triaged> < https://launchpad.net/bugs/741821 >
<lifeless> thats 4h20 minutes later
<lifeless> the bug activity is
<lifeless> 2011-03-24 15:32:53	Andrea Corbellini	bug			added bug
<lifeless> 2011-03-24 15:56:44	Graham Binns	tags		confusing-ui story-better-bug-notification	
 * jcsackett nods. that works with what we've seen.
<lifeless> 2011-03-24 15:56:48	Graham Binns	launchpad: status	New	Triaged	
<lifeless> 2011-03-24 15:56:52	Graham Binns	launchpad: importance	Undecided	High	
 * gmb needs to stop "Graham" and "Binns" from highlighting
<lifeless> so 20 minutes would have been reasonable
<lifeless> the fact that the interval is 4h20 minutes makes me wonder
<lifeless> is there a 4 hour grace period on actions being reverted or something ?
<lifeless> gary_poster: ^
<gary_poster> I was just talking in ops about all this...
<gary_poster> lifeless, no
<lifeless> oh
<lifeless> CHANNEL FAIL
<gary_poster> :-)
<bac> hi sinzui, you have a minute to chat about menus
<lifeless> sorry
<gary_poster> np
<jkakar> This is a curious position to be in: You are the owner of this team. You are not currently an active member.
<jkakar> I guess one never gives up ownership of a team...?
<jkakar> https://launchpad.net/~clouddeck
<lifeless> jkakar: ownership can be handed off
<jkakar> lifeless: Ah, okay.  That should probably happen in this case.
<jkakar> lifeless: Aha, found it... thanks for the hint. :)
<sinzui> bac: I will in 2 minutes
<sinzui> bac: mumble?
<bac> sinzui: yes please
* flacoste changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: https://wiki.canonical.com/IncidentReports/2011-03-24-LP-bugmail-extremely-slow-or-offline | On call reviewer: jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> is anyone fixing the db-devel conflict ?
<jcsackett> lifeless: is there a new one? bigjools or jml killed one awhile ago.
<leonardr> jcsackett: ok, take another look. i made the sanity check responsible for cleaning up after itself, and i added code to the test framework to wipe any test that imported a module without running the sanity check
<jcsackett> leonardr: while the while len > 0: pop? wouldn't just reassigning to [] be faster? the contained objects lose their reference all the same so should still be cleaned up normally...
<leonardr> jcsackett: i would love to reassign to [], but that doesn't work
<lifeless> jcsackett: leonardr: cleaning a list ?
<leonardr> yeah
<lifeless> del list[:]
<jcsackett> leonardr: ooh, that works nicely.
<jcsackett> lifeless, rather.
<lifeless> also spellable as list[:] = []
<lifeless> but I'd use del
<leonardr> change made
<jcsackett> leonardr: does that work in your instance?
<leonardr> yeah
<jcsackett> awesome.
<jcsackett> r=me,
<leonardr> i just can't change the reference
<jcsackett> leonardr: ah, right.
<jcsackett> well, with the del, all seems well by me. land it. :-)
<leonardr> ok
<sinzui> wgrant:  mumble?
<wallyworld> leonardr: sorry i missed you this morning - got sucked straight into a prod incident. saw your mp comment. thanks. can you tell me how to run the tests?
<leonardr> wallyworld: bin/test should work
<leonardr> if you don't have bin/test you need to run 'python bootstrap.py; bin/buildout'
<wallyworld> leonardr: i there was no bin/test. i started running buildout etc and it went to download all this zope stuff so i killed it cause i was using my 3g modem at the time
<wallyworld> i will let it finish running the next time
<lifeless> sinzui: you wanted to talk about the person merge branch I reviewed
<leonardr> wallyworld: ok, do this too:
<leonardr> create a ~/.buildout/default.cfg
<leonardr> mine looks like this
<leonardr> [buildout]
<leonardr> eggs-directory=/home/leonardr/.buildout/eggs
<leonardr> download-cache=/home/leonardr/.buildout/download-cache
<leonardr> everything it downloads will be stored so it will only download once
<sinzui> lifeless: Yes please
<leonardr> not for every branch
<lifeless> sinzui: did you want voice or irc or ... ?
<wallyworld> leonardr: ah, that's what i need. thanks
<sinzui> lifeless: I am staring skype
<lifeless> kk
<wallyworld> leonardr: i'm working on the prod incident atm but will make the changes you suggested and let you know. i should be able to make myself available for a standup tomorrow morning to hopefully finalise stuff before you go
<leonardr> ok, great
<leonardr> talk to you then
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: https://wiki.canonical.com/IncidentReports/2011-03-24-LP-bugmail-extremely-slow-or-offline | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> Ooh, explicit API versioning.
<wgrant> Excellent!
<lifeless> sinzui: mailed.
#launchpad-dev 2011-03-25
<poolie> jelmer, hi
<jelmer> hello again :)
<poolie> jelmer, so i think the lep is as far forward as makes sense for something that's just queued up
<poolie> if we're going to actually start it, it should be made a bit tighter about requirements
<poolie> probably make sure all the issues people raised are addressed
<poolie> and then break out some particular bugs
<poolie> all with a tag
<jelmer> poolie: ah, ok
<jelmer> poolie: is it still waiting for anything, or should one or more of us just go over it to make sure all requirements are addressed?
<poolie> nothing else
<poolie> that would be great
<poolie> if you'd like to start on it
<jelmer> might as well
<LPCIBot> Project windmill build #101: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/101/
<jelmer> sleep first though, g'night!
<lifeless> wgrant: how much work is sitting unreleased on lh trunk ?
<wgrant> lifeless: 11 revs... start/stop-loggerhead are gone (deprecated long ago), a couple of logging changes that seem harmless.
<wgrant> Otherwise just optimisations and UI tweaks.
<lifeless> wgrant: feel like doing a 1.19 ?
<lifeless> wgrant: my thinking is: we have radical stuff in the pipeline
<lifeless> be nice to get the incremental stuff 'out there'
<wgrant> lifeless: i might talk to jam about that tonight.
<lifeless> and you're already fiddling with all the policy etc etc
<wgrant> lifeless: Oh, you mean "unreleased" as in everything since 1.18, not since our last rev?
<lifeless> wgrant: yes
<lifeless> wgrant: the stuff on trunk we should deploy, thats not done because of unfamiliarity with the mechanism for jam
<wgrant> There's only a few revs more. 1.18 was 421. There's the new theme, what look like bzr compat fixes, and the HEAD changes.
<wgrant> And a favicon change.
<wgrant> So all releasable.
<wgrant> If we're going to start doing more radical stuff, we should indeed release.
<lifeless> historydb is more radical
<wgrant> Right, but I'm not sure how close that is.
<lifeless> I think it would be nice to have a modern loggerhead equivalentish to our deploy before that tracks forward
<lifeless> wgrant: not all that far off
<james_w> who did the new theme?
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/new-loggerhead/+merge/54803
<wgrant> james_w: huwshimi, I believe.
<james_w> thanks huwshimi
<huwshimi> james_w: Yeah you can blame me :)
<james_w> huwshimi, far from it :-) it's a great and sorely needed improvement
<huwshimi> james_w: Hopefully it'll be integrated with Launchpad further at some point, but it's a start
<james_w> yeah
<james_w> huwshimi, two questions, do we need all the duplication of the (i) icons? and can we have the commit message wrap if it's too long?
<james_w> e.g. http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/files
<james_w> (for me at least with 1280px width)
<huwshimi> james_w: Do you mean the icons for "view revision" and "view branch changes"?
<james_w> yeah
<james_w> there's more on the file pages
<james_w> http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/view/head:/Makefile
<james_w> that's a lot of informative links
<huwshimi> james_w: It's not great, but that's the way we do sub navigation elsewhere in Launchpad
<huwshimi> james_w: Can you please file a bug about the commit message length?
<james_w> huwshimi, fair enough
<james_w> sure
<huwshimi> james_w: I will be address a lot of things like that soon
<huwshimi> *addressing
<james_w> huwshimi, do you want a separate one for http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/changes
<james_w> where if you expand the top revision it gets loooooong
<james_w> and http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/revision/60 I guess, which is the worst yet :-)
<james_w> also, bugs against loggerhead or launchpad? tags? subscribers?
<huwshimi> they should be bugs against loggerhead
<huwshimi> just tag them as ui and you can add me as a subscriber
<james_w> ok, thanks
<huwshimi> james_w: Thank you for noticing (and caring)
<james_w> it's great to have someone to raise these with again :-)
<huwshimi> james_w: oh and with the lengths you can probably put it all in one bug but link to all those pages
<james_w> bug 742178
<_mup_> Bug #742178: doesn't handle long commit messages well <ui> <loggerhead:New> < https://launchpad.net/bugs/742178 >
<huwshimi> james_w: Thanks heaps
<mwhudson> ah it would be good to have someone who knows some css look at those
<lifeless> I want a teddy bear
<lifeless> to talk webservice collection offset tuning with
<lifeless> to the sound of o/~ I need a hero
<spm> nothing here but us crickets
<lifeless> crickets are good
<lifeless> spm: hey, can you explain analyze a query on prod for me - looking for cold cache again
<lifeless> https://bugs.launchpad.net/launchpad/+bug/736008/comments/1
<_mup_> Bug #736008: Product:+code-index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/736008 >
<spm> lifeless: https://pastebin.canonical.com/45213/
<lifeless> booyah
<lifeless> spm: thanks
<spm> np
<lifeless> spm: could you run the same query, not explained - the actual query - on another server?
<spm> sure, one sec.
<spm> lifeless: count: 168, Time: 125.128 ms lp_1
<lifeless> spm: woo,t hanks.
<lifeless> hot shit query
<spm> as opposed to warm?
<lifeless> run it again :)
<spm> Time: 62.076 ms
<lifeless> yup
<lifeless> I have oopses with it at 700ms
<spm> 110; then 3 x 171's.
<lifeless> the older version
<lifeless> let me grab that
<spm> is stabilising at 171ish.
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1906G1635#longstatements
<lifeless> 1521ms + 589ms
<lifeless> http://pastebin.com/8H8tha0g
<lifeless> is the original
<lifeless> which is 4.6 seconds coldish on qas
<spm> ouchy
<lifeless> and 650 hot
<lifeless> so yeah, I'm pretty happy with the improvment
<lifeless> the new plan is missing the sequential scan on branch :)
<mwhudson> !
<mwhudson> the tuple read rates that stub produced seem to indicate that something was sequential scanning branch about once a second
<mwhudson> lifeless: could you have found that?
<lifeless> yes
<lifeless> code:+index is the page
<lifeless> ppr will tell us hits
<lifeless> I would expect multiple causes
<mwhudson> i guess any branch filtering is similar queries
<lifeless> yeah
<lifeless> the double subqueries were either too complex or constrained for some corner case - I think the thing was that they were different
 * mwhudson wonders what fraction of branches are private... 
<lifeless> so pgsql wasn't coalescing them
<lifeless> mwhudson: 3 5ths of bugger all
<mwhudson> yeah
<mwhudson> if they hit 10% of the table or whatever the threshold is i wildly speculate that we'd have the sequential scans back
<mwhudson> but that's unlikely
<lifeless> problem for another day
<mwhudson> ye
<lifeless> you'd actually need 10% of the table for *one context*
<mwhudson> oh righ t:)
<lifeless> mwhudson: I'm so glad you are here though, you can answer something flummoxing me
<lifeless> mwhudson: why does the original (and the new one because i kept it) only constraint the source branch to the product
<mwhudson> lifeless: don't know
<mwhudson> i guess you can end up with the products being different if you move one of the branches after the mp is created
<lifeless> right, but why does moving the target not hide the count; but moving the source does ?
<mwhudson> constraining the target branch would seem a bit less wrong though ...
<lifeless> *hige it from the count*
<mwhudson> lifeless: i bet that whoever wrote this (possibly named after a disney rabbit) assumed that source target and target target are always the same and 'optimized' it
<mwhudson> lifeless: just a guess though
<lifeless> hahaha
<lifeless> tactfully put
<lifeless> see the thing is
<lifeless> if the target was constrained the original query is fine
<lifeless> uhm, if 'source and target are constrained'
<mwhudson> heh heh heh
<lifeless> DistributionSourcePackage:+code-index 540
<mwhudson> yes, my guess doesn't make much sense on that level does it?
<lifeless> Product:+code-index 4965
<mwhudson> more constraints -> faster query (generally)
<lifeless> mwhudson: yes
<lifeless> mwhudson: so, 5500 hits a day
<lifeless> mwhudson: 2 runs of the query each hit
<lifeless> 11K query runs a day
<lifeless> 710983.85 is pretty high
<mwhudson> lifeless: it's one of these slightly horrible things, we /try/ fairly hard to enforce target target == source target, but i don't think we can completely
<lifeless> mwhudson: we can choose to just not report the exceptions :>
<mwhudson> yeah, i think that would be fine
<mwhudson> lifeless: what is 710983.85 ?
<lifeless> branch tuples read per second
<mwhudson> ah right
<lifeless> https://devpad.canonical.com/~lpqateam/dbr/last-15-mins.txt
<mwhudson> yes it is
<lifeless> > 3 * bugtask reads
<lifeless> and bugs are much higher user activity than branches
<lifeless> oh
<lifeless> I remember one of the branch thoughts
<lifeless> 'url -> disk mapping'
<lifeless> still several OOM too high
<StevenK> You should not import patch_entry_explicit_version from canonical.launchpad.components.apihelpers:
<mwhudson> i'm pretty sure that doesn't seqscan...
<StevenK> Did that change recently?
<lifeless> StevenK: add it to __all__
<lifeless> mwhudson: it doesn't
<mwhudson> or even read more than a handful of tuples, so yeah, like you said, an OOM out
<mwhudson> at least
<mwhudson> i wonder how much cpu 396125.47 tuples/sec uses, it must be more than 0.01% though, and 'rewrite' isn't a user on https://devpad.canonical.com/~lpqateam/dbr/last-15-mins.txt
<mwhudson> lifeless: the pillarname read count is demented too, but at least it's obviously a table that gets accessed an awful lot
<lifeless> remember that foreign keys on updates take locks
<lifeless> possibly on reads too,but I'd need to double check that
<lifeless> sinzui: ping
<lifeless> sinzui: if you're around; how would you and jc feel about my stabbing at CVE:+index; its consitently 2nd or third on the oops report at the moment
<wgrant> What is the scanner doing these last few days?
<wgrant> 1600 OOPSes...
<LPCIBot> Project db-devel build #488: FAILURE in 4 hr 58 min: https://lpci.wedontsleep.org/job/db-devel/488/
<lifeless> wgrant: bug is filed ;)
<sinzui> lifeless: go ahead on the cve:+index jcsacket and I were to busy to work on it today
<huwshimi> lifeless: Just taking a look at the AJAX logging stuff. What's involved in getting the URL into the header?
<lifeless> huwshimi: one cheap hack would be to change the publisher to add an X-header unconditionally on every reponse with the url it got
<lifeless> huwshimi: e.g. probably not much
<huwshimi> lifeless: Is this going to be bad for us in any way?
<lifeless> huwshimi: slightly more response overhead
<lifeless> huwshimi: if someone sends a 4K request, they'll get 4K more back
<lifeless> huwshimi: (4K URL I mean)
<huwshimi> lifeless: Will it be enough of an impact for us not to do it?
<wgrant> You can't get at the request from the response?
<huwshimi> wgrant: Well this is not really the response. It's an event that fires every time an AJAX even completes, with no other knowledge about the request except what is returned
<lifeless> huwshimi: I think we can do it, will just want a brief walk-by the usual suspects to look for gotchas
<wgrant> huwshimi: :(
<huwshimi> lifeless: I may have just thought of a way to do it without the header, thanks to wgrant's question. I'll have a quick try
<lifeless> huwshimi: care to file a bug asking for an X-LP-URL header, and mail the -dev list for feedback.
<lifeless> huwshimi: if you can't get it, of course
<huwshimi> lifeless: Sure
<lifeless> grrr
<lifeless> Collections may be great for adhoc assembly
<lifeless> but they are terrible for optimisation.
 * lifeless puts on the wet blanket hat
 * StevenK grumbles.
<StevenK> I think python-debian has a bug in it's regex to parse valid versions
<mwhudson> oof
<wgrant> StevenK: Is it meant to only parse valid versions?
<wgrant> Also, why is the solution to AJAX being too slow to limit it to only a few bugtasks? :(
<StevenK> wgrant: It accepts : in the upstream version all the time, but that's only valid if there is a epoch.
<huwshimi> lifeless: OK I can't reliably get it this way. Is the code for managing the headers outside of Launchpad?
<StevenK> (Pulling from my memory, so I could be wrong.)
<StevenK> wgrant: It has a re_valid_version regex defined, so I doubt it. I think the regex is naive.
<wgrant> StevenK: I don't think it's valuable for it to be terribly pedantic, really.
<lifeless> huwshimi: we'd put it in lib/canonical/launchpad/(publisher|publication).py I think.
<StevenK> wgrant: In this case, the version that sorts the highest is invalid, and so we can't set it in the DB, due to integry rules.
<lifeless> huwshimi: there are some interactions with the api, but I'm 99% sure that starting there would get what we need easily enough
<StevenK> wgrant: So I'd like to filter invalid versions from the ancestry sets.
<wgrant> StevenK: Right.
<StevenK> wgrant: And I'm having trouble convincing debian_support.BaseVersion that "a1:1.8.8-070403-1~priv1" really is invalid.
<StevenK> Due to it accepting : in upstream version, it thinks epoch is None, and the upstream version is "a1:1.8.8-070403", which is just wrong.
<wgrant> Yeah.
<wgrant> Er, thinks the epoch is None?
<wgrant> That's a bug.
<wgrant> Missing epoch is 0.
<StevenK> But it isn't missing, it's invalid :-)
<StevenK> % bin/py foo
<StevenK> Version: a1:1.8.8-070403-1~priv1
<StevenK> Epoch: None
<wgrant> Ah, true.
<StevenK> IIRC, you can only have :'s in the upstream version iff there is an epoch.
<wgrant> That's right.
<wgrant> Otherwise it's ambiguous.
<wgrant> "This is a single (generally small) unsigned integer. It may be omitted, in which case zero is assumed. If it is omitted then the upstream_version may not contain any colons."
<StevenK> wgrant: Have a look at python-debian in sourcecode, the re_valid_version is wrong, then.
<wgrant> StevenK: I can't really fault it on that. Be conservative in what you do, be liberal in what you accept from others, and all that.
<StevenK> wgrant: Suggestions on how to filter invalid versions out of ancestry, then?
<lifeless> ohdogohdogohdog
<wgrant> StevenK: I'm not sure what's best.
<wgrant> lifeless: Oh?
<wgrant> lifeless: What completely naÃ¯ve and non-performant code have you found now?
<lifeless> wgrant: GenericBranchCollection uses storm expresses to express constraints
<lifeless> wgrant: so I have to unwind them to get the clause to regenerate it with class aliases.
<lifeless> because it discards the original context
<lifeless> in the adapter
<wgrant> :(
<StevenK> wgrant: I'm struggling to come up with *any* solution that doesn't involve hacking debian_support.BaseVersion
<lifeless> wgrant: its Yet Another Compiler.
<lifeless> I should suggest a code review rule : no more compilers.
<StevenK> gcc 4 lyfe?
<lifeless> StevenK: collections are compilers; wadl creation is a compiler, storm is a compiler.
<lifeless> StevenK: BugTaskSet.searchTasks is a compiler
<lifeless> StevenK: nearly everything that is easy to use and hard to make fast, is a compiler.
<StevenK> I didn't need the explanation, I was trolling.
<poolie> lifeless, i'm glad you asked for a simple reproduction because i no longer seem to be able to hit it :)
<lifeless> poolie: heh
<poolie> this is actually astonishingly fast
<poolie> i think something must be wrong
<StevenK> Haha
<poolie> mm i wasn't measuring the hard part
<poolie> damn all software that tries to pretend the network is transparent
<poolie> i wonder if i was just measuring some transient misbehaviour before
<poolie> anyhow, the good thing is that if this works, it may make kanbans much faster
<poolie> lifeless, the numbers from that script are just astonishingly good
<poolie> 0.96s for some queries
<poolie> i never saw sub-second calls from australia before
<poolie> (i guess that's on a warm socket)
<spiv> Happiness is a warm socket.
<poolie> yeah apparently so
<wgrant> Maybe gandwana and potassium finally decided to die.
<poolie> ?
<poolie> what would that mean, wgrant?
<wgrant> poolie: Well, it would speed up requests.
<wgrant> Since those two are multithreaded, slow, and the source of most of our timeouts.
<poolie> because they're slow appservers?
<wgrant> Slow and misconfigured.
<StevenK> wgrant: Didn't we port some version validator from somewhere in Soyuz?
<wgrant> StevenK: You mean the one that's in the database?
<StevenK> wgrant: That one feels a little heavyweight for validating versions
<StevenK> wgrant: Dpkg::Version::version_check would be nice, but I can't use that
<LPCIBot> Project windmill build #102: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/102/
<wgrant> wallyworld: Those two Windmill failures look like yours, I think.
<wallyworld> wgrant: oh :-(
<wallyworld> i'msure they passed locally :-(
<wallyworld> wgrant: https://code.launchpad.net/~wallyworld/launchpad/recipe-text-xss/+merge/54638
<lifeless> ahhh
<lifeless>  1 file changed, 31 insertions(+), 35 deletions(-)
<lifeless> \o/
 * wallyworld has to go get kid from school
<lifeless> deleting code *and* making things faster
<huwshimi> lifeless: Any suggestions on how I can break an AJAX request so it returns an exception?
<huwshimi> lifeless: I mean returns an oops id
<lifeless> break the server side code
<lifeless> add
<lifeless> raise Exception('zomg')
<lifeless> to the code path
<huwshimi> lifeless: ok
<poolie> lifeless, i think some other change must have fixed it
<lifeless> huwshimi: your demo was -really really shiny-
<poolie> i'm running the same code i was then and i no longer see the effect
<lifeless> poolie: sure
<lifeless> poolie: I believe that you saw it
<lifeless> poolie: I *suspect* it was doing a full text search for '' or something equally daft
<lifeless> (we repacked the fti index for bug recently)
<lifeless> [amongst other fixes such as improving component fixes, related bugs yada yada yada]
<poolie> i certainly wasn't asking for that
<poolie> it's possible that lplib or the server did end up doing it
<lifeless> poolie: thats what I'm sketching
<poolie> yeah
<lifeless> its the only thing I can think of that might vary
<poolie> anyhow, the bottom line on performance is actually quite interesting:
<lifeless> because the collections are really just function calls to searchTasks
<poolie> single calls are quite impressively fast
<poolie> however, if you have to do 600 of them, not so much
<poolie> thus my braindump
<lifeless> poolie: sure; that aspect has been heavily sketched out in the filter+expand API LEP
<poolie> yep
<huwshimi> lifeless: OK, I don't know enough about Launchpad's internals to break this request. Any clues about where to look?
<huwshimi> lifeless: Also are there any time recommendations that we have for AJAX requests? E.g. over 5 seconds is very bad under 2 seconds is good
<lifeless> huwshimi: ajax requests are no different than others
<lifeless> huwshimi: we're (long term) aiming at subsecond for 99% of requests and under 5 seconds for all
<lifeless> huwshimi: current timeout limit is 11 seconds
<lifeless> huwshimi: I think colouring anything > 1 second a vivid shade would be sensible ;)
<huwshimi> lifeless: OK, I'll do >1 second for now
<lifeless> most ajax things should be very targeted operations
<lifeless> unlike e.g. a bug listing page
<lifeless> at least for now
<micahg> hi lifeless
<lifeless> hiya
<micahg> lifeless: so, several years ago, I wrote some javascript code to output an HTML table that needed to be displayed multiple times on a page (i.e. output with document.write) and that sped up load time tremendously, could something like that work as a stop gap for the repetitive JS code until it can be refactored properly?
<lifeless> micahg: that sounds like it would be refactoring it properly
<huwshimi> lifeless: so, any clues on how API internals work. Like, where does the API code for bugs live?
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> huwshimi: lib/lp/bugs/model
<lifeless> huwshimi: whats the url you are calling ?
<micahg> lifeless: well, no this would just output the javascript inline as opposed to using a function w/params and have one copy of the JS code in the rendered page
<huwshimi> lifeless: But if I raise the exception there won't it be broken for the normal request as well?
<lifeless> huwshimi: yes
<lifeless> huwshimi: are you trying to check it works, or write an automated test ?
<huwshimi> lifeless: I'm trying to check it works
<lifeless> huwshimi: because, for the latter, its a bit complex: we don't want /any/ broken apis in the system, so you have to register a broken one temporarily, use it, then unregister in your test.
<wgrant> lifeless: Are you sure your +needs-packaging change is OK?
<lifeless> huwshimi: right, so pick something - like mark as duplicate
<lifeless> wgrant: if I marked it as such, I tested it on qastaging
<lifeless> wgrant: why do you ask ?
<wgrant> lifeless: It's not QA'd yet, but it's gone from timing out on lpnet to 0.2s on qas.
<wgrant> So I'm a bit suspicious.
<lifeless> wgrant: sounds right
<wgrant> Seriously?
<wgrant> The results look fine.
<wgrant> But the time is not plausible.
<lifeless> oh ye of little faith
<lifeless> https://answers.qastaging.launchpad.net/ubuntu/karmic/+needs-packaging was 2.43 on qastaging
 * lifeless looks at a profile
<wgrant> natty's first batch is 0.20 to 0.25s.
<wgrant> 0.17 just now.
<lifeless> 00294-01256 SELECT COUNT(*) FROM SourcePackageName, (SELECT ...
<lifeless> 01259-02197@SQL-launchpad-main-slave SELECT DISTINCT ON (score ...
<lifeless> it looks plausible to me
<lifeless> wgrant: you do realise it has memcache in there
<lifeless> wgrant: which I didn't touch
<wgrant> Oh.
<wgrant> Indeed. a more plausible 1.7s on the next batch..
<lifeless> wgrant: 2.5 seconds cold, 0.2 in memcache
<lifeless> I think its qa ok indeed
<lifeless> marked as such
<lifeless> wgrant: I'll whip through and do all mine
<wgrant> lifeless: Great. I'm doing the others'.
<lifeless> done
<micahg> lifeless: so, I commented on bug 735966, but almost instantly regretted it, is that the same bug?
<_mup_> Bug #735966: Product:+bugs timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/735966 >
<micahg> I hit the crash again, so I figured I'd confirm
 * StevenK nails wgrant to the channel
<wgrant> Huh.
<wgrant> Thanks Optus.
<wgrant> We were throttled for almost 10 minutes.
<wgrant> To 28kbps.
<wgrant> Despite being at 70% of our quota...
<StevenK> Haha
<huwshimi> lifeless: Gotta run in a minute, but this is where I'm leaving it for today: http://people.canonical.com/~huwshimi/ajax_time_log.ogv
 * huwshimi is gone
<lifeless> micahg: may be, may not be :P
<micahg> lifeless: should I file a new bug with my new oops code and you can dupe if appropriate?
<lifeless> no need
<micahg> ok, do you want the oops?
<lifeless> you put it in the comment already
<micahg> lifeless: no, I got a new one today
<StevenK> wgrant: http://pastebin.ubuntu.com/585234/ makes me sad.
<wgrant> StevenK: Doesn't that forbid upstream versions with colons?
<StevenK> wgrant: Probably.
<StevenK> wgrant: I looked at the check_version sub in Dpkg::Version, and I didn't feel like porting it to Python so I can use it. But I suspect it's the better answer.
<wgrant> StevenK: So python-debian doesn't have a useful checker? What about python-apt?
<StevenK> wgrant: I couldn't see one in apt_pkg
<StevenK> wgrant: python-debian's checker is too lax for our needs, since the DB's own checker is strict.
<StevenK> Doing it in one regex is not the way
<wgrant> :( qa-bad
<StevenK> wgrant: Which rev? :-(
<wgrant> StevenK: The Soyuz translations change.
<wgrant> Missing DB perms.
<StevenK> Which points to missing tests, too.
<StevenK> I think the qa-tagger needs to run more often
<lifeless> micahg: probably hasn't changed
<micahg> lifeless: ok
<wgrant> Huh, translations sharing works.
<henninge> wgrant: Hi!
<wgrant> henninge: Hi.
<henninge> wgrant: why did you mark bug 741571 bad?
<_mup_> Bug #741571: Translation templates should always be uploaded from soyuz builds. <qa-bad> <regression> <upstream-translations-sharing> <Launchpad itself:Fix Committed by henninge> < https://launchpad.net/bugs/741571 >
<wgrant> Just testing out the Soyuz translations changes on DF.
<wgrant> henninge: I'm going to comment in a sec.
<henninge> ok
<wgrant> But there's a missing DB perm.
<wgrant> queued needs SELECT on packaging.
<wgrant> Apart from that it seems to be OK.
<wgrant> I think I've just got an upstream template in place, so we'll see now if the POs are skipped...
<lifeless> wgrant: is it for the jon thats not running et ?
<henninge> yeah, it's actually a pretty simple change, so I'd have been surprised if it went wrong.
<wgrant> lifeless: It's running.
<lifeless> ah
<wgrant> lifeless: But on germanium/cocoplum.
<wgrant> So we can nodowntime with a little process skipping.
<henninge> wgrant: thanks, I was about to try to find out how I could QA this best.
<wgrant> henninge: Could you QA your other change? The big cleanup thing?
<henninge> wgrant: yes, I am doing that atm.
<wgrant> Great.
<wgrant> buildbot should finish shortly.
<henninge> although it's a lot of refactoring, so this is mostly "see that nothing broke".
<wgrant> Yup.
<StevenK> How many revs will the buildbot-poller pull in?
<wgrant> Success.
<wgrant> StevenK: 1
<wgrant> henninge: https://translations.dogfood.launchpad.net/ubuntu/maverick/+source/hello/+imports?field.filter_status=NEEDS_REVIEW&field.filter_extension=all
<wgrant> No POs.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #489: FIXED in 5 hr 0 min: https://lpci.wedontsleep.org/job/db-devel/489/
<lifeless> \o/ private branch query on lp with me is still only 500ms
<lifeless> we need to redo the privacy queries comprehensively, but this will do for now
<henninge> wgrant: cool, thanks!
<henninge> wgrant: so, we need a branch for the db permission?
<wgrant> henninge: Yeah, ideally with tests.
<wgrant> henninge: Can your squad handle that soonish?
<henninge> wgrant: yeah, sure
<wgrant> henninge: Thanks.
<wgrant> lifeless: Should we deploy even with the missing DB perm?
<wgrant> win 3
<lifeless> wgrant: will it break ?
<lifeless> wgrant: can we avoid it breaking ?
<wgrant> lifeless: It only affects cocoplum.
<wgrant> Which is way outside nodowntime.
<lifeless> wgrant: +1
<wgrant> lifeless: Thanks.
<wgrant> henninge: Should we just qa-untestable that refactor?
<henninge> wgrant: yeah, I was thinking about that too. I am busy now with the missing db permissions.
<wgrant> henninge: Have you filed a bug?
<henninge> wgrant: do I need to? I was going to do it on the existing bug.
<henninge> I can, np.
<wgrant> henninge: We are going to deploy this broken code, so I'd like a Critical one targetted to 11.04 to make sure we fix it before it's deployed to the relevant machines.
<henninge> wgrant: makes sense. I'll file it.
<wgrant> henninge: Thanks.
<henninge> wgrant: are both cocoplum and germanium production machines?
<wgrant> henninge: Yes.
<wgrant> henninge: cocoplum = ftpmaster, germanium = ppa
<wgrant> Neither is nodowntime.
<henninge> ok
<henninge> wgrant: bug 742309
<_mup_> Bug #742309: Missing db permissions for soyuz translation uploads. <upstream-translations-sharing> <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/742309 >
<wgrant> henninge: Thanks.
<lifeless> right, works
<lifeless>  branchcollection.py |   85 +++++++++++++++++++++++-----------------------------
<lifeless>  1 file changed, 39 insertions(+), 46 deletions(-)
<adeuring> good morning
<henninge> Hi adeuring!
<adeuring> hi henninge
<henninge> adeuring: bug 742309
<_mup_> Bug #742309: Missing db permissions for soyuz translation uploads. <upstream-translations-sharing> <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/742309 >
<adeuring> yes?
<lifeless> can has review? https://code.launchpad.net/~lifeless/launchpad/bug-736008/+merge/54820
<adeuring> lifeless: sure
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring | https://code.launchpad.net/launchpad-project/+activereviews
<henninge> adeuring: just so you know. ;)
<adeuring> henninge: ah, thanks, now i understand
<lifeless> adeuring: https://bugs.launchpad.net/launchpad/+bug/736008 has all the experimentation for that branch, if you want the backstory
<_mup_> Bug #736008: Product:+code-index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/736008 >
<LPCIBot> Project windmill build #103: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/103/
<mrevell> Morning
<lifeless> oh god
<lifeless> no wonder jc had trouble on cve:+index. its bugtask:+index all over again
<henninge> wgrant, adeuring: https://code.launchpad.net/~henninge/launchpad/devel-741571-db-permissions/+merge/54823
<henninge> If either of you could be so kind.
<henninge> I have to run an errand now but will be back later to land it.
<adeuring> henninge: i'll once I've finished robert's mp
<henninge> I don't see how this branch would profit from a full test suite run.
<henninge> adeuring: thanks
<wgrant> henninge: As long as you've run the new test then lp-land it.
<henninge> of course ;-)
<lifeless> wow
<lifeless> https://bugs.launchpad.dev/bugs/cve/1999-8979
<lifeless> 262 queries/external actions issued in 6.15 seconds
<lifeless> locally
<lifeless> oh, rotfl
<wgrant> ?
<lifeless> I crewated 200 or so tasks for it
<StevenK> "crewated"
<StevenK> :-P
<lifeless> Product-name903534, Product-name903544 etc
<wgrant> I'm surprised shortlist didn't slap you.
<adeuring> lifeless: r=me
<lifeless> adeuring: thanks
<lifeless> wow
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1904QS73 query 304
<spiv> lifeless: at least it's quick!
<lifeless> spiv: indeed
<adeuring> henninge-afk: r=me, 1 typo spotted
<lifeless> stub: ping
<stub> lifeless: pong
<lifeless> bug 618406
<_mup_> Bug #618406: Distribution:+bugs-index time outs <lp-bugs> <pg83> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618406 >
<lifeless> stub: I'm wondering if a matching index is better than modifying the query
<lifeless> bugheat__id__idx on (Bug.heat DESC, BugTask.id)
<stub> looking
<stub> lifeless: You would have to denormalize the heat onto bugtask for that index.
<lifeless> stub: bwah, of course
<lifeless> I'm just tired and forgetting
<stub> A (Bug.heat, Bug.id) might help though - join and order in one step
<stub> Although we would likely need to change the ordering to (Bug.heat DESC, Bug.id DESC), and get rows ordered inconsistently if there happen to be more than one bug task being matched.
<lifeless> stub: (bug.heat desc, bug.id) would work fine
<lifeless> stub: the index can sort different columns differently afte rall
<stub> Thats new too
<lifeless> nah
<lifeless> not sure when it was added, but definitely existed in 8.3
<stub> You used to have to make the index match. 8.3 is new - This all used to run under 7.4 remember, and I started with PG 7.0 :)
<lifeless> you do need it to match
<lifeless> you have to tell the index that one column is up and one down
<stub> ok. we are on the same page :)
<stub> I could rationalise that newer bugs are hotter, all else being equal
<lifeless> or older bugs need fixing more ;)
<lifeless> http://www.postgresql.org/docs/8.4/static/indexes-ordering.html is the docs
<lifeless> why oh why do cve pages show bug editing controls.
<henninge> adeuring: thanks
<adeuring> welcome
<henninge> wgrant: fix has been merged
 * henninge relocates.
<wgrant> herb: Fantastic. Thanks.
<wgrant> Er.
<wgrant> henninge.
<leonardr> wallyworld_, i'm up if you want to talk (on irc, my wife is asleep)
<deryck> Morning, all.
<mrevell> Should debhelper be a dependency of bzr-builder?
<lifeless> stub: will you add such an index?
<stub> lifeless: I will test later - doing my paperwork atm. I think it unlikely to help, but worth giving it a shot anyway.
<stub> (This is the bug.head, bug.id index?)
<lifeless> bug.heat, bug.id desc
<lifeless> yeah
<lifeless> sorted by heat alone its a 3ms query
<lifeless> or something rad like that
<lifeless> stub: thanks
<stub> lifeless: I doubt we can improve on that. I guess the results might be more stable if we add more fields, but I think faster queries will be the preference.
<lifeless> stub: right, the question is, is heat desc + id fast enough - if it is, its a trivial change, add the index, done.
<stub> Agreed.
<lifeless> if its not, we have to see if we break the test suite etc etc
<lifeless> mmm, I guess we may anyway with bug.id being needed
<stub> Heat + Bug.id still can break tests - multiple bug tasks will be returned in arbitrary order
<lifeless> yeah
<lifeless> on distro I guess
<lifeless> should probably only return one task for this search
<stub> I think the only thing that really cares is the test suite, so it will be sucky to have worse performance just to keep it happy
<lifeless> blah, we have to do a code change regardless
<lifeless> agreed
<lifeless> I keep forgetting the cross table aspect
<leonardr> adeuring: a small branch, https://code.launchpad.net/~leonardr/lazr.restful/449561/+merge/54849
<adeuring> leonardr: I'll look
<cjwatson> I have two approved branches to land, but I don't have direct landing access (AFAIK).  Can somebody help me with this?
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/dpkg-xz-support-619152
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/tar-xz
<wgrant> cjwatson: Sorry, forgot about those.
<wgrant> I'll send them to ec2.
<cjwatson> no problem, the first one only just got the IS side done
<cjwatson> thanks
<wgrant> cjwatson: Which IS side? Getting a new dpkg out?
<wgrant> We still probably can't land the first one, unless the new dpkg is also in our ec2 AMIs.
<wgrant> And buildbot machines.
<wgrant> It's also needed on germanium and cesium.
<cjwatson> New dpkg, yes.  Don't the buildbot machines use dpkg from chroots?
<wgrant> buildbot != buildd
<cjwatson> We don't actually need the new dpkg to pass LP tests, do we?
<cjwatson> oh, maybe
<wgrant> testXZDebUpload would test that, I hope.
<cjwatson> I guess we do, since bar_1.0-1_xz has 'dpkg-deb -Zxz -b debian/tmp ..'.
<wgrant> Yeah.
<cjwatson> so what's involved in getting a new dpkg into AMIs and buildbots?
<wgrant> buildbot needs the new dpkg in CAT, and a LOSA poke.
<wgrant> ec2test AMIs need the new dpkg in ppa:launchpad, and a Launchpad engineer poke.
<cjwatson> I've reopened the RT ticket asking for germanium and cesium to be upgraded.
<cjwatson> new dpkg is already in CAT.
<wgrant> Great.
<cjwatson> (lucid-cat)
<cjwatson> what's the relevant distroseries for ppa:launchpad?
<cjwatson> and how should I ask for buildbot to be upgraded?  RT or IRC ping?
<wgrant> I've already poked.
<wgrant> Do you have a copy of the package handy? I'll upload it to the PPA.
<wgrant> I guess I could grab it from cat.
<cjwatson> it's in my home directory on chinstrap too
<cjwatson> dpkg_1.15.5.6ubuntu4.5.IS.PATCHED.10.04_source.changes (you may want to reversion, of course)
<wgrant> Yup.
<wgrant> thanks.
<wgrant> Could you set a commit message for both?
<cjwatson> do I need any r= type runes in it?
<wgrant> No, ec2 land will add those.
<cjwatson> set for both, then
<wgrant> Thanks.
<adeuring> leonardr: r=me
<wgrant> cjwatson: dpkg_1.15.5.6ubuntu4.5.IS.PATCHED.10.04.tar.bz2 is 600
<cjwatson> hah, lamont clearly cheated
<cjwatson> fixed
<wgrant> Heh.
<dpm> hi jml, did you have the chance to look at the AppDeveloperWeek LP session?
<jml> dpm: busy right now, sorry.
<dpm> jml, no worries. When you've got some time, if you could tell me if you or someone else would be up for it, I'd add it to the timetable. It would be great to have at least a LP session
<dpm> thanks
<jml> dpm: will do. trust me, it's not forgotten.
<dpm> thanks jml :)
<wgrant> jam: Hi.
<jam> hey wgrant
<jam> thanks for your work on loggerhead
<wgrant> jam: Bug #742390
<_mup_> Bug #742390: page generation problem <codebrowse> <oops> <regression> <Launchpad itself:Triaged> <loggerhead:New> < https://launchpad.net/bugs/742390 >
<wgrant> Wondered if you had any ideas on that.
<jam> wgrant: mismatch between version of bzrlib and loggerhead
<jam> wgrant: that page must have a "tag" in it, that the other pages don't have
<jam> wgrant: I'm checking when we introduced it in bzr now
<jam> wgrant: looks like it was introduced in bzr 2.3
<jam> and I'm guessing LP is still using 2.2?
<jam> wgrant: note that "bzr-2.2 selftest -s bp.loggerhead" would have caught this. Is there a reasonable way to integrate that into the LP test suite?
<jam> (run the loggerhead test suite from the bzr that is going to be deployed?)
<wgrant> jam: We can (and should) probably do that somehow.
<jam> wgrant: I think part of the confusion is that loggerhead used to be standalone, and then became a bzr plugin
<wgrant> Yeah.
<wgrant> (we use 2.2.3dev, apparently. not been updated lately)
<jam> wgrant: the really easy fix is to just comment out that line
<jam> we don't *have* to sort the tags
<jam> I did it to get stable results in the test suite
<jam> since otherwise it is 'random'
<jam> I thought it would be nice to have a sort order for them anyway
<jam> wgrant: https://bugs.launchpad.net/launchpad/+bug/742390/comments/5
<_mup_> Bug #742390: page generation problem <codebrowse> <oops> <regression> <Launchpad itself:Triaged> <loggerhead:New> < https://launchpad.net/bugs/742390 >
<jam> Possible fix
<jam> it will make codebrowse stop OOPSing
<jam> though it also fails the test suite
<jam> because, you know, it isn't natural sorting :)
<wgrant> jam: It's just about midnight here. Can you fix trunk?
<jam> wgrant: Pushing a change for review now
<jam> what do we need to do to get it rolled out?
<wgrant> A buildbot run started 2 minutes ago, so without a cowboy it will be 10 hours, at which point we have no LOSA.
<jam> wgrant: buildbot takes 10 hours to run? yeouch
<wgrant> jam: Only 4.5 or so.
<wgrant> But if we land it now then it will be waiting for the next one.
<jam> wgrant: https://code.launchpad.net/~jameinel/loggerhead/sort-742390/+merge/54855
<jam> I can land and propose an update to LP, or do we want to 'lp-land' the sourcecode update?
<jam> wgrant: I confirmed that the test suite passes on bzr 2.2, 2.3, and dev with that patch
<jam> (not that the loggerhead test suite is all that complete at this point, but it is getting better)
<wgrant> I think we get this into LH trunk, land the sourcedeps.conf change to LP, pull the new LH rev onto guava, restart codebrowse.
<wgrant> Yeah, I noticed it's much better now.
<wgrant> Is the LH review policy the same as the LP one?
<wgrant> ie. can I not approve it on my own?
<jam> wgrant: it is probably closer to the bazaar one, but you certainly can approve it yourself
<wgrant> (since I'm not a full reviewer yet)
<jam> wgrant: you've got more experience in loggerhead than most of the rest of them :)
<wgrant> True.
<wgrant> jam: So, looks fine to lend as long as the tests pass with all three bzrs.
<jam> wgrant: k, how do we get it landed in LP ,then?
<wgrant> jam: Set the revno in utilities/sourcedeps.conf.
<jam> sure, I know that
<jam> the *landing* is the part for me
<wgrant> You mean getting it onto prod right now?
<wgrant> Since it's landed no differently than normal...
<jam> wgrant: /me => not a lp dev, don't have the tools or authority for stuff like lp-land or ec2land
<jam> I happen to play one on tv
<jam> I *only* happen to play one on tv
<wgrant> Oh, I thought you would by now.
<wgrant> OK.
<wgrant> So, propose, get it reviewed by a full reviewer, poke them to land it if I'm already asleep.
<wgrant> Bah.
<wgrant> The last tag on lp:loggerhead was on r421. My quick QA involved only the first page of results for that branch, which only went back to r423 :/
<wgrant> So close.
<jam> wgrant: so you would have noticed in QA if it had gone back another couple revs. That's life, I think, and why you want a nice stable automated test suite, rather than expecting human QA to find stuff
<jam> (they are complimentary, as Human QA detects things that are hard to automate, but is really bad at repeatability and checking all edge cases)
<wgrant> jam: Yeah, and the other branches I tested were lp:launchpad and one with crafted filenames, neither of which have any tags.
<wgrant> Fail.
<wgrant> Anyway, confirmed that fix works on a local instance.
<wgrant> (of LP codebrowse)
<leonardr> lifeless: quick question on bug 61531 if you're still around. does the behavior i describe fall within the "404-from-lp-action logic" you mention?
<_mup_> Bug #61531: Forbidden error when trying to mark a bug as private <lp-bugs> <oops> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/61531 >
<jam> wgrant: how *do* you qa loggerhead? Given that you can't actually connect to qastaging?
<jam> I think you said something about firewall holes?
<wgrant> jam: There are convenient firewall holes.
<jam> adeuring: can you have a look at: https://code.launchpad.net/~jameinel/launchpad/loggerhead-oops-742390/+merge/54858
<jam> wgrant: ^^
<jam> anyone?
<jam> wgrant: can you lp-land the fix then? Or you have to wait for a full reviewer?
<wgrant> jam: My code* is worth nothing, sadly.
<wgrant> Needs a full reviewer.
<jam> know anyone who would be up at this time?
<jam> jml: ^^
<wgrant> Half the team is up at this time :)
<jam> wgrant: though apparently not reading IRC :)
<jcsackett> jam: looking now.
<jam> thanks jcsackett
<wallyworld_> leonardr: the latest larz restful mp is showing conflicts but i'm not sure why. none on my system and if i run a merge --preview it's clean
<leonardr> wallyworld, give me a url
<jcsackett> jam: r=me. not sure about lp-landing instead of ec2ing it. i understand your rationale, but i am a timid man.
<wgrant> lp-land is safe because of what allowed this to break (the test suite isn't run). But there's no time benefit at this time of week, so might as well ec2 it.
<wallyworld_> leonardr: https://code.launchpad.net/~wallyworld/lazr.restful/propogate-notifications/+merge/54690
<jam> jcsackett: up to you, but there are no tests being run for loggerhead at the moment
<jam> hopefully will be fixed soon (by me)
<jam> bug #742446
<_mup_> Bug #742446: Launchpad should include loggerhead's test suite <Launchpad itself:Triaged by jameinel> < https://launchpad.net/bugs/742446 >
<jam> jcsackett: if it helps, it was just rolled out in production
<deryck> wallyworld_: hi
<leonardr> wallyworld: you're conflicting with a change i just landed. have you updated your copy of lazr.restful trunk?
<wallyworld_> deryck: hi there
<jcsackett> jam: huh. given that and what wgrant just said, i guess there's not much concern.
<wallyworld_> yes
<wallyworld_> leonardr: and merge into my branch and resolved conflicts and committed and pushed
<deryck> wallyworld_: test_multicheckbox is failing in windmill test run....
<leonardr> that is weird
<deryck> wallyworld_: I think I was horribly wrong about the disappearing console not mattering.
<jam> jcsackett: anyway, if you can run ec2land or lpland for me, I would appreciate it.
<jam> since the fix is in production, you have all the time you want to do it either way
<jcsackett> jam: ah, right, you need someone to land it for you. forgot that bit. i'll get it sent out.
<wallyworld_> deryck: yeah :-( i haven't had time to look yet since my plate today has overflowed due to the bug notification incident etc
<wgrant> We won't have time this week to deploy, so it's not time-critical.
<deryck> wallyworld_: no worries.  I'm debugging it a bit for you.  Looks like it's the activator's fault.  Probably that flash a custom node stuff.
<jam> wgrant: I can confirm that http://bazaar.launchpad.net/~loggerhead-team/loggerhead/trunk-rich/revision/421 works
<wallyworld_> deryck: i did have a quick look and couldn't figure out how to stop the disappearing stuff
<wgrant> jam: I checked the URLs in the bug too.
<wgrant> jam: All looks good.
<wgrant> jam: Thanks for the very quick fix :)
<adeuring1> jam: I'll look
<wallyworld_> deryck: you're awesome for doing that. thanks
<jam> adeuring1: we got it done, jcsackett handled it, thanks, though
<adeuring1> ak, ok. sorry was out for lunch
<wallyworld_> deryck: if it is the activator, i guess it's a bug or at least a test environment incompatibility that needs fixing
<deryck> wallyworld_: yeah, I think it's a bug in handling custom supplied nodes.  This will fix the test:  http://pastebin.ubuntu.com/585380/
 * wallyworld_ looks
<deryck> wallyworld_: but I'm not sure how to fix the real  issue yet.
<deryck> ah, interesting!  It's not the missing console causing the failure.
<deryck> I get real output now though....
<wallyworld_> deryck: ah ok. remove those lines and i think it won't flash anymore :-)
<deryck> right
<wallyworld_> deryck: wonder why we haven't seen this before. maybe not enough js tests
<deryck> wallyworld_: I think it's a misdirection.... getting a new paste now that I have a real error
<deryck> wallyworld_: here's the real failure now:  http://pastebin.ubuntu.com/585384/.  I guess the console stuff was masking it.
<wallyworld_> leonardr: i may just pull another branch and try again. but in the meantime you could look at my latest changes. it's late here now and we can talk in a few hours for our last standup
 * wallyworld_ looks at pastebin
<leonardr> will do
<wallyworld_> deryck: thanks. i'll get it sorted and put up a mp. but not tonight. i'll try and get it done over the w/e
<leonardr> wallyworld: i get the conflicts when i check out your branch and merge trunk. i'll resolve them and push for you
<jml> qastaging down?
<wgrant> jml: It's updating.
<jml> wgrant: ok.
<wgrant> Hmmmm.
<wallyworld_> leonardr: thanks, that's great. not sure why my local copy is clean.
<jml> deryck: I'm going to need to have a chat with you today to make further progress on this testing
<deryck> wallyworld_: no worries, man.  I'll fix it for you.  It looks easy.
<deryck> jml: ok, cool.  progress FTW!  Any particular time?
<jml> deryck: but not right now. should clear up a few things locally first.
<wallyworld_> deryck: cool. i'll owe you a beer at uds or the next epic
<deryck> jml: ok.  just ping when you're ready.
<jml> deryck: will do.
<deryck> wallyworld_: don't worry about it.  but, of course, I'll not turn down free drink. ;)
<wgrant> jml: It's back.
<leonardr> wallyworld: lp:~leonardr/lazr.restful/propogate-notifications
<wallyworld_> deryck: i am enjoying my forays into yui land etc. just wish there were better debugging tools. firebug is ok but...
 * wallyworld_ looks at leonards mp
<wallyworld_> i mean branch
<deryck> wallyworld_: yeah, debugging is hard.  rockstar was supposed to ask about pro debugging tips at the YUI conf he attended, but I never checked back about that.
<deryck> rockstar: anything more than "use firebug" for that ^^ ?
<wgrant> jcsackett, jam: I'll land the branch, since sleep is blocked on that.
<rockstar> deryck, wallyworld_, generally I've just got used to all the common screw ups, and know what they mean.
<jam> wgrant: thanks
<jcsackett> wgrant: that would be good, as bzr is not working for me at the moment.
 * jcsackett prepares to stab his computer
<wgrant> jcsackett: Fun.
<deryck> rockstar: yeah, me too.  I suspect what wallyworld_ means is better facility for working out what is wrong when you don't know.
<jcsackett> wgrant: always. :-P
<wallyworld_> rockstar: i rely  alot on Y.log(). takes be back 15 years ago before there were decent java ide's :-)
<wallyworld_> deryck: rockstar: yeah, the ability to step through code and look at variables and evaluate expressions etc
<deryck> rockstar, wallyworld_ -- I think filter debug is nice for log noise and working it out.  Liberal Y.log is your friend, etc.  I'm not sure what else to suggest.
<deryck> wallyworld_: you can't step through with firebug?
<rockstar> deryck, wallyworld_, that sounds unhelpful, I know.  However, I've had some, er, "fun" forays into the bowels of YUI trying to find out what's wrong, and that usually ends up showing me the problem is very simple.
<deryck> yeah
<rockstar> wallyworld_, do you know how to use the debugger?
<deryck> yeah, I do step-wise debugging in firebug often.
<wallyworld_> deryck: rockstar: clearly not :-)
<rockstar> wallyworld_, through 'debugger;' it at the line where you want the breakpoint, and run the code.
<deryck> wallyworld_: add a "debugger;" statement in any line and have fun :-)
 * wallyworld_ will have to RTFM
<rockstar> Firebug will take care of the rest.
<rockstar> wallyworld_, I also prefer chrome's tools to Firebug now, as Firebug's XUL-land code often confuses Firefox greatly.
<wallyworld_> if only i had figured that out 1 month ago :-)
<rockstar> Like, refreshing while still in the breakpoint causes madness.
<wallyworld_> rockstar: cool. i try it. actually i have found firefox 4 to be a huuuuge memory hog :-(
<rockstar> wallyworld_, with or without firebug?
<deryck> I'm slowly coming over the Chromium, too.  Now that I'm learning the UI differences better, and changing muscle memory.
<wallyworld_> rockstar: without. but of course its always just an F12 away
<wallyworld_> so it must have stuff loaded all the time etc
<wgrant> jam: Landed, thanks.
<wallyworld_> deryck: yeah, muscle memory. that's why i can't use another ide besides PyCharm. too much time with Intellij for Java
<deryck> heh, understood
<wgrant> jam: I'm off to bed now.
<wgrant> jam: Hopefully nothing will break!
<wallyworld_> leonardr: thanks for fixing branch. talk again in 7 hours :-)
<leonardr> k
<jam> wgrant: sleep well
 * wallyworld_ off to bed too but not same bed as wgrant
<rockstar> wallyworld_, indeed it does.  That's a big reason I switched to Chrome for development.
<wallyworld_> rockstar: i like chrome but last time i looked their adblocking sucked due to lack of content filtering api
<wgrant> wallyworld_: Ad blocking is much better now
<wgrant> It used to really suck, though, yes.
<rockstar> I just wish they'd block popunders.
 * wallyworld_ really going away now :-)
 * jcsackett holds breath to see if irc client has stopped misbehaving
 * jcsackett exhales, slowly
<leonardr> adeuring: https://code.launchpad.net/~leonardr/launchpad/520412-from-thekorn/+merge/54868
<adeuring> leonardr: I'll look
<deryck> adeuring, abentley, and henninge -- I forgot... we're all on DST next week, right?  Which should put standup back at 1300 UTC?
<adeuring> deryck: yes
<abentley> deryck: I believe so, and yes.
<jam> adeuring: any chance you could look at the follow up patch: https://code.launchpad.net/~jameinel/launchpad/loggerhead-test-suite-742446/+merge/54870
<adeuring> jam; sure
<adeuring> leonardr: r=me
<henninge> deryck: since it won't change the time for us here, I am all for it.
<deryck> ok, cool.
<rvba> sinzui: Hi ... if you have a few minutes, I'd be happy to have your opinion on the UI side of a branch I'm working on right now.
<sinzui> rvba: I can talk in a few minutes
<rvba> sinzui: https://pastebin.canonical.com/45240/
<jml> deryck: how's now?
<rvba> sinzui: I'd be happy to talk about it yes ... just ping me when you're ready
<deryck> jml: now's good.  mumble it?
<jml> deryck: yep
<danilos> deryck, s/chromium inspector/webkit inspector/ (it works in epiphany as well)
<deryck> jml: drag me where you like
<danilos> deryck, btw, cool tip, I was setting a breakpoint from the inspector directly :)
<deryck> danilos: heh, ok :-)
<jml> deryck: http://paste.ubuntu.com/585427/
<deryck> jml: https://translations.qastaging.launchpad.net/ubuntu/natty/+source/gnuhello
<deryck> jml: https://translations.qastaging.launchpad.net/ubuntu/natty/+source/gnuhello/+sharing-details
<LPCIBot> Project windmill build #104: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/104/
<adeuring> jam: r=me
<jam> adeuring: can you 'ec2land' it for me?
<adeuring> jam: sure
<jam> thanks
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | https://code.launchpad.net/launchpad-project/+activereviews
<sinzui> rvba: why does 2_details.png show the expander arrow, but the computes replaces it with a cog? Cannot I not expand the entry anymore?
<rvba> sinzui: the greyed arrow cannot actually be expanded ...
<sinzui> Why are we telling users It can be expanded?
<sinzui> Is that arrow being used as a bullet?
<rvba> yes
<sinzui> easy fix
<rvba> indeed
<sinzui> Should we have a grey cog instead?
<rvba> it acts as a placeholder so that when it's replaced by real status icon (processing/failed) the text does not jump
<sinzui> good
<rvba> if there is a grey cog, it would be very nice ... I've reused existing icons
<sinzui> I think a cog with a questionmark emblem might convey we do not know the status yet
<rvba> yes
<rvba> what's the process for creating new icons?
<jml> hah
<jml> rvba: ask huwshimi
<rvba> jml: ok
<sinzui> I have not noticed the cog before. The page title says source package, I think the data is actually a source package release. I do not want to confuse the users about out precise use of terms, BUT I think expect an icon that implies source package. cogs are used for binaries.
<rvba> yes, the data is source packages
<rvba> (even if there is a remaining 'binary' somewhere)
<rvba> the icons would be for packages diff more than source packages
<sinzui> rvba: I think the interaction is very good. I think we need to note we want the icon developed for the interaction before the final release
<rvba> it's the diff that can be a) not there / b) pending / c) failed
<cjwatson> wgrant: apparently the buildbots run hardy-cat, so I'll need to backport this all the way back to hardy
<sinzui> rvba: understood. You might want to report a bug to record the requirements for the icons. You can then focus on functional side of your branch
<rvba> sinzui: ok
<rvba> jml: should I report a bug about the missing icons and assign that to huwshimi?
<cjwatson> wgrant: sorry, just the staging buildbot
<jml> rvba: sounds good.
<cjwatson> wgrant: I'm having a hard time seeing how 3.0 format support works in that case, though ...
<rvba> great
<cjwatson> wgrant: wouldn't current LP tests fail without 3.0 support in dpkg?
<rvba> sinzui: ok, thanks for your time.
<sinzui> bac: do you have time to review https://code.launchpad.net/~sinzui/launchpad/person-merge-job-2/+merge/54873 today?
<bac> sinzui: yes
<cjwatson> wgrant: never mind; as far as I can tell, if the staging buildbot is running hardy (and doesn't have a lucid chroot or whatever), then AFAICT it can't be testing LP
<sinzui> jcsackett: do you have a few minutes to mumble? We can pretend it is our daily talk
<jcsackett> sinzui: sure, one second.
<jcsackett> sinzui: i'm on mumble.
<bac> sinzui: i made a super-trimmed down test to demonstrate the problem i'm having. it can be run, and fails, in trunk.  would you have a look and see if you spot anything obviously wrong when you have time?  http://pastebin.ubuntu.com/585455/
<sinzui> bac: OTP, will look in a few minutes
<cjwatson> leonardr: hmm, can I suggest that maybe https://help.launchpad.net/API/Examples shouldn't recommend connecting to edge? :-)
<leonardr> cjwatson: yeah, i'll change it
<cjwatson> thanks
<bac> hi sinzui, your code says membership in a merged team is not transferable.  really?
<danilos> jml, hi, did you by any chance change the summary of https://bugs.launchpad.net/bugs/742490 and then change it back? (because if you did, we've got a bug; if you didn't, I am just having trouble reading the differences in the email :)
<_mup_> Bug #742490: HTML is constructed using string concatenation in the structural subscription JS. <story-better-bug-notification> <Launchpad itself:New for yellow> < https://launchpad.net/bugs/742490 >
<jml> danilos: no, just a spelling correction
<jml> concatination => concatenation
<danilos> jml, ah, right, concatInation
<danilos> jml, it was hard to spot, and I was worried our "do not email about reverted changes" didn't catch it
<jml> understandable.
<danilos> jml, sorry for interruption, cheers :)
<allenap> adeuring, bac: Do you guys fancy any branches to review? :) I have a small one, a medium one, and a large one to choose from.
<bac> allenap: i'll be happy to look at them after my lunch
<adeuring> allenap: I'll start with the small one :)
<allenap> bac: Thank you. I hope you have a delicious lunch that leaves you in an pleasantly accommodating mood :)
<allenap> adeuring: Cheers! https://code.launchpad.net/~allenap/launchpad/dd-initseries-bug-727105-feature-flag/+merge/54884
<bac> me too
<sinzui> hi bac
<bac> hi
<sinzui> do you want to mumble?
<bac> sinzui: ok
<sinzui> bac: http://pastebin.ubuntu.com/585467/
<adeuring> allenap: there is a merge conflict
<cjwatson> wgrant: germanium, cesium, and the buildbots other than staging should all have the new dpkg now, according to lamont
<allenap> adeuring: Gah! I'll fix it now.
<adeuring> allenap: thanks.
<allenap> adeuring: I think it's confused. The correct diff - http://paste.ubuntu.com/585475/ - is actually shorter, and has no conflicts. Mmm :-/
<adeuring> allenap: ok, thanks. allenap: r=me.  just one suggestion: I don't like generic names like is_featrue_enabled very much, What about is_derived_series_feature_enabled?
<allenap> adeuring: Okay, makes sense. Thank you :)
<adeuring> cheers
<adeuring> allenap: next mp? (the mid-size one, please. Since my EOD is relatively close, I'd like to pass the big  MP to bac)
<allenap> adeuring: https://code.launchpad.net/~allenap/launchpad/dd-initseries-bug-727105-copy-options/+merge/54667 though I'll check the diff on that one now.
<bac> bac: +1 adeuring
 * bac -> lunch
<allenap> adeuring: It seems to be sane.
<adeuring> allenap: right
<jcsackett> leonardr: sorry to bug you about this on your last day, but is there a way to mark a field readonly on the api that's not read only otherwise?
<leonardr> jcsackett: sure, pass readonly=True into export()
<jcsackett> leonardr: someday i'm going to learn this all easier than i expect. :-)
<leonardr> it's not _that_ tough
<jcsackett> leonardr: indeed. i conflate webservice stuff with dealing with the wadl, i think. which as i recall was a major pain.
<jcsackett> obviously such conflation is silly.
<leonardr> jcsackett, can you elaborate on the pain?
<jcsackett> leonardr: remember the issues you, sinzui, and to some extent i had when you were modifying it?
<jcsackett> really, i mean that was xsl issues. not wadl per se.
<jcsackett> but it set the tone in my head. :-P
<leonardr> oh, xsl is the problem there :)
 * jcsackett nods.
<jcsackett> xsl is always a problem. :-)
<jkakar> leonardr: Just noticed "last day" above.  I hope you have fun wherever you end up. :)
<leonardr> jkakar: thank you!
<adeuring> allenap: r=me
<allenap> adeuring: Thanks!
<bac> sinzui: i've tested and create_view is setting the BugsLayer properly based on rootsite
* adeuring changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer:  bac | https://code.launchpad.net/launchpad-project/+activereviews
<sinzui> bac: okay. I am starting my adventures with distros, adaption, and menus
<bac> cool, sinzui
<sinzui> bac: I see that the menu cannot get the facet, so cannot actually select a menu. We expect  bugs as was set by the rootsite.. facet should be layer, but Lp does not do this right. I think the setup is wrong. I see that path is missing, so I suspect we got another request object.
<bac> sinzui: so i need to manually set facet?
<bac> and path?
<sinzui> That is not really possible, Menu gets it via adaption
<sinzui> facet is moneypatched into the view by our ZCML hacks.
<bac> sinzui: my battery is dying so i need to relocate.  i'll ping you in 15 minutes.
<sinzui> okay
<jml> g'night all. have a great weekend.
<bac> sinzui: i am resettled and recharging
<LPCIBot> Project db-devel build #491: FAILURE in 5 hr 20 min: https://lpci.wedontsleep.org/job/db-devel/491/
<sinzui> c
<lifeless> moin
<lifeless> leonardr: hi
<sinzui> bac: create_initialized_view uses getMultiAdapter directly. It bypasses the publisher. The publisher is what monkeypatches the facet into the view. So we will not have a facet...
<bac> sinzui: ah.  boo.
<sinzui> bac: we are working with an model object of course, but the failover method cannot get a facet...
<lifeless> leonardr: I'm not sure what you're asking
<lifeless> oh, I think I've figured it out
<sinzui> bac: for expediency you might want to use test browser to for the test or...
<leonardr> lifeless: you triaged the bug as critical because of a situation in which a 404 caused an oops
<leonardr> i'm wondering if the situation i described falls into that category
<lifeless> I think so, I just commented on the bug
<sinzui> bac: we could finally fix the bug facet/layer bug. eg, Use the layer when we do not get a facet
<lifeless> leonardr: the rules for 404 oopsing are:
<lifeless>  - its a 404 (duh)
<lifeless>  - and the referrer is a launchpad vhost or one of a select number of canonical/ubuntu sites
<sinzui> bac: this is the hack I did to fix the test: http://pastebin.ubuntu.com/585562/
<cody-somerville> hey. have you guys ever had any problems with chromium thinking lp is TSLv1 intolerant and falling back to SSLv3 or erroring out with non-descriptive SSL error?
<sinzui> ^ the comment exposed in the diff mean we really wanted to use layer
<lifeless> leonardr: I suspect you have oopses in /var/tmp/lperr/$day
<bac> sinzui: but your hack won't work on other layers, right?
<lifeless> cody-somerville: other than tls being a little bit broken ;)
<lifeless> cody-somerville: anyhow, I saw someone mentioning a tls error with something yesterday in #is
<sinzui> No, but I think in your case, the menu is not associated with a layer. I think the menu you are testing is on the default layer overview
<sinzui> oh hell, I need to get children
<lifeless> cody-somerville: see -ops right now in fact
<sinzui> bac: I was going to walk through pdb next to see how to get the correct layer and map it to the selectedfacetname so that we have a final fix
<bac> sinzui: the portlet i'm looking for only appears on the bugs page
<bac> sinzui: ok, go get your kids
<sinzui> bac: I do not think get_current_browser_request does the right thing in this kind of test. I am certain layer is set by create_initialized_view, but it do not set when we work with the menu. I think the menus are getting another request object
<bac> ok
<bac> sinzui: for now i think i'm going to take your advice and rewrite the test to use a browser
<sinzui> I think will continue my distraction to learn what is wrong with some small hope that we will get one step closer to killing facet.
<bac> great, sinzui
<benji> see ya, leonardr
<leonardr> bye, benji
<leonardr> wallyworld_, i'm here sporadically, let me know when you're ready for standup
<wallyworld_> leonardr: hi. i'm here now
<leonardr> wallyworld_: ok, getting on mumble
<leonardr> lp:~leonardr/launchpad/520412-from-thekorn
<leonardr> wallyworld_: http://pastebin.ubuntu.com/585604/
<leonardr> ok, i'm out of here. thanks, everybody. it's been great working with you
<bac> sinzui: i have discovered a problem with my test construction
<sinzui> current_request=True ?
<bac> sinzui: simpler.  for rootsite='bugs' i was using '+index'.  that page renders as the main project page
<bac> i should use '+bugs-index'
<sinzui> bac: I discovered that the request we pass/make in create_view is not setup for the interaction be default. We need to set current_request=True to ensure the menu gets what is setup
<sinzui> I am making a helper function that gets the facet name form the layer
<bac> sinzui: i had variously tried setting current_request=True
<bac> sinzui: i have to run but feel confident correcting my simple mistake will help greatly
<sinzui> fab. I hope to have a spike to remove facets in a hour
<wgrant> cjwatson: I'm confused. What staging buildbot?
<wgrant> cjwatson: There are staging buildds, but no staging buildbot that i know of.
<cjwatson> wgrant: lamont said the machine name was marambio
<cjwatson> 16:00 <lamont> drwxr-xr-x 3 importd importd 20480 2011-03-25 00:01 staging-logs
<cjwatson> 16:00 <lamont> OTOH, /srv/importd.staging.launchpad.net/staging/launchpad has nothing newer than 60 days ago
<cjwatson> 16:00 <lamont> so the cronjobs are running, but no one is home
<cjwatson> so I think it can reasonably safely be declared irrelevant for the moment anyway
<cjwatson> whatever it is
<wgrant> ... oh. That's a code importd. I guess they did run buildbot in like 2006, but I meant the lpbuildbot.canonical.com buildbots.
<wgrant> cjwatson: I've sent your branch back to ec2.
<cjwatson> ah, right.  thanks
#launchpad-dev 2011-03-26
<wgrant> Ew.
<wgrant> Lots of odd MP OOPSes...
<wgrant> Can't see how it could be a regression, though.
<lifeless> mp creation was timing out for the package importer yesterday
<wgrant> It was OOPSing, not timing out.
<lifeless> true
<lifeless> they were getting 500's and not seeing the oops id
<lifeless> so I was guessing
<wgrant> Ah.
<lifeless> bah
<lifeless> BMP index still timing out
<wgrant> :(
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1910B1069#longstatements
<lifeless> oh right, that wasn't deployed
<wgrant> You mean the one that landed 64 minutes ago?
<lifeless> yeah
<wgrant> HYeh
<lifeless> 1	2079.0	1	SQL-launchpad-main-slave	
<lifeless> SELECT BranchMergeProposal.commit_
<lifeless> bah, terrible plan
<lifeless> bug 742916 if you're interested
<_mup_> Bug #742916: BranchMergeProposal:+index timeouts - slow query plan <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/742916 >
<lifeless> BranchRevision needs revision_date denormalised into it
<wgrant> lifeless: No.
<wgrant> That will double the size of our biggest table, so I think not :)
<lifeless> its only 34GB
<wgrant> True. It's the indices that are really big :/
<lifeless> wgrant: we're currently bringing 10K rows in at a time
<lifeless> wgrant: a bigger table (and another index - shrug) - which stays mostly on disk, would be a net win.
<lifeless> wgrant: select count(*) from revision where Revision.revision_date >= '2009-11-05 00:21:45.372993+00:00' and Revision.revision_date <= '2009-11-06 15:52:25.310401';
<lifeless>  count
<lifeless> -------
<lifeless>   9722
<lifeless> I suspect a recursive graph traversal query will be faster
<lifeless> and we should investigate that before denormalising
<wgrant> lifeless: I personally prefer the "don't store 50000 times more data than necessary" approach.
<wgrant> lifeless: What do you know about bugnotification?
<wgrant> https://lpstats.canonical.com/graphs/TableRowCountbugnotification/20100327/20110327/ is somewhat interesting.
<wgrant> The trend from the start of Feb hasn't been seen before.
<lifeless> https://lpstats.canonical.com/graphs/TableRowCountbugnotification/20090327/20100327/
<wgrant> I guess the pre-release rush could just have been brought all the way back to the start of Feb rather than March.
<wgrant> But mmm.
<lifeless> so its a short lived audit trail + work queue
<lifeless> on qastaging
<lifeless> select min(date_emailed) from bugnotification;
<lifeless>             min
<lifeless> ---------------------------
<lifeless>  2011-01-14 23:05:53.30206
<LPCIBot> Project windmill build #105: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/105/
<wallyworld> thumper: ping
<LPCIBot> Project windmill build #106: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/windmill/106/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #492: FIXED in 5 hr 19 min: https://lpci.wedontsleep.org/job/db-devel/492/
<cjwatson> 7990 N T Mar 26 William Grant          Test results: tar-xz => devel: FAILURE
<cjwatson> 7991 N   Mar 26 noreply@launchp        [Merge] lp:~cjwatson/launchpad/tar-xz into lp:launchpad
<cjwatson> wgrant: ^- do I need to do something about that failure, or did it get resolved before merge?
<StevenK> cjwatson: It has hit devel as of r12667, and the LP buildbot is currently building that rev -- so we'll find out in about 37 minutes.
<StevenK> There are currently no test failures, though.
<cjwatson> the failures look legit, investigating
<cjwatson> unfortunately I don't appear to be able to run the uploadprocessor tests locally
<cjwatson> I've pushed what I think is a fix to lp:~cjwatson/launchpad/tar-xz
<wgrant> cjwatson: Ah, sorry, forgot to reply. I fixed that myself and landed it.
<wgrant> cjwatson: Why can't you run it locally?
<cjwatson> setting up all the necessary postgres stuff for full LP test runs impedes my ability to do Ubuntu boot speed work
<cjwatson> ah, if you fixed it yourself, great
<cjwatson> wgrant: you might want to check my tar-xz branch all the same - I had apparently forgotten to add the .tar.xz file to a second set of test data that hangs off the first
<cjwatson> it seemed to pass anyway but it's more correct this way, I think
<wgrant> cjwatson: Lots of us run LP in a VM.
<wgrant> Because it is slightly intrusive.
<wgrant> (Unity and r600g are really not good friends at the moment)
<LPCIBot> Project windmill build #107: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/107/
<james_w> is there a bug for whatever is causing the package importer to fail with a 500 error? I was told it was a timeout error that a fix was available for
<wgrant> james_w: It's not a timeout. I suspect that it's a bug in the importer... it's OOPSing with a mismatch between the number of reviewers and review types in the request.
<wgrant> james_w: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1910A1001
<wgrant> james_w: It looks like change landed in lp:udd three weeks ago, but perhaps it was only deployed a couple of days back?
<cjwatson> wgrant: I don't work on LP often enough to have spent time setting that kind of thing up, TBH
<james_w> wgrant, that's quite possible
<lifeless> moin
<lifeless> we should upgrade to bzr 2.3
#launchpad-dev 2011-03-27
<LPCIBot> Project windmill build #108: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/108/
<LPCIBot> Project windmill build #109: STILL FAILING in 53 min: https://lpci.wedontsleep.org/job/windmill/109/
<lifeless> wgrant: if you're around, I'd like a second eyeballs over https://bugs.launchpad.net/launchpad/+bug/736008/comments/12
<_mup_> Bug #736008: Product:+code-index timeouts <qa-ok> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/736008 >
<LPCIBot> Project db-devel build #496: FAILURE in 4 hr 55 min: https://lpci.wedontsleep.org/job/db-devel/496/
<lifeless> moin
<jkakar> lifeless: Moin. :)
<lifeless> jkakar: how goes the new job?
<mwhudson> morning
<lifeless> mwhudson: hi
<lifeless> mwhudson: got a sec to talk (irc) about Product:+code-index
<mwhudson> lifeless: sure
<lifeless> so https://bugs.launchpad.net/launchpad/+bug/736008/comments/12 is a new query I've put together that really does look better for determining outstanding merge proposals
<_mup_> Bug #736008: Product:+code-index timeouts <qa-ok> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/736008 >
<mwhudson> (if you'll excuse small gaps in which i make toast)
 * mwhudson looks
<lifeless> there is one *definitional* difference between that and what runs today: both the source and target branch contexts are constrained
<mwhudson> well sure, as we discussed before that sounds find
<mwhudson> *fine
<mwhudson> lifeless: it looks pretty sane to me, the with stuff makes it nice and readable
<mwhudson> lifeless: i guess like you said, we should actually dump that query into a temp table, then count it and select .. limit $N from it
<lifeless> lp is the worst project in lp for this query AFAICT
<lifeless> its 5.5 seconds cold on qastaging
<lifeless> the bit I don't like is we're defining unbounded-growth here
<mwhudson> lifeless: does lp have the most private branches?
<lifeless>    CTE scope_branches
<lifeless>      ->  Bitmap Heap Scan on branch  (cost=143.45..11180.08 rows=7634 width=9) (actual time=4.790..29.200 rows=7634 loops=1)
<lifeless>            Recheck Cond: (product = 10294)
<lifeless>            ->  Bitmap Index Scan on branch__product__id__idx  (cost=0.00..141.55 rows=7634 width=0) (actual time=2.862..2.862 rows=7634 loops=1)
<lifeless>                  Index Cond: (product = 10294)
<lifeless> we're defining this on all branches for lp ever
<lifeless> rather than 'all unmerged' or something similar
 * thumper is off to the physio :-|
<lifeless> thumper: be well
<thumper> sprained my ankle Thursday last week
<thumper> playing at camp
<mwhudson> lifeless: well sure, but i'm (probably naively) assuming that most of the cost comes from checking the subsciptions of private branches; maybe not
<mwhudson> thumper: at least bigjools wasn't there to land on :)
<lifeless> mwhudson: yeah, AFAICT it does, yes.
<mwhudson> lifeless: i guess you could filter scope_branches with the status filter -- i guess this is what you meant in comment #8
<lifeless> mwhudson: thats the sort of thing, yes
<mwhudson> which is still theoretically unbounded but much less likely to be a problem in practice
<lifeless> mwhudson: a naive query can run this in a few ms; the with clause approaxh is always about 100ms [on lp branches] - but handles the privacy permission check better
<lifeless> mm, not as fast as I thought it an
<lifeless> *can*
<lifeless> anyhow, I think this is an incremental win
<mwhudson> yeah definitely
 * mwhudson speculates
<mwhudson> what would a completely blue sky fast-from-scratch design of this look like... maintaining the set of visible branches in cassandra or something?
<mwhudson> i guess the space requirements would be a bit daft for that
<lifeless> dropping the owner check halves the query time
<lifeless> so the basic costs are:
<lifeless>  - merge proposal visibility is independent of the merge proposal
<lifeless>  - merge proposal scope is independent of the merge proposal
<mwhudson> would not have guessed the owner check as expensive
<lifeless> oh, I made a booboo on that test
<lifeless> owner isn't incrementally expensive
<lifeless> dropping the subscriber check is 40ms off the hot time
<lifeless> and 45% off the cost estimate in the plan
<jkakar> lifeless: The new gig is turning out to be really fun.  The team is great and we've begun a process to tackle a bunch of tech debt, which is moving quickly and will improve Fluidinfo in many ways.
<lifeless> jkakar: cool
<lifeless> mwhudson: so I think in cassandra you'd have a merge proposal column family
<lifeless> mwhudson: which stores the visibility inline
<lifeless> mwhudson: probably tie visibility to the user
<mwhudson> and then fun games to update that when the user joins/leaves a team?
<lifeless> mwhudson: e.g. users know what they can see [except admins]
<lifeless> mwhudson: we already do updates of denormalised stuff
<lifeless> mwhudson: e.g. teamparticipation as the primary exemplar
<mwhudson> yes indeed
<lifeless> which in fact is the only thing making the performance here tolerable
<mwhudson> so you'd have to map something like "place that accesses TP today" -> "something that has to be updated when team membership changes"
<lifeless> yes
<lifeless> a halfway house might be useful - for particularly big teams, dereference to the team
<lifeless> mwhudson: or alternatively, 'what is the disk space cost of storing a primary key for every private branchmergeproposal that a user is given access to'
<lifeless> for large teams - hundreds or thousands of members, this could be quite large
<mwhudson> yeah, indeed
<lifeless> e.g. a MB per merge proposal
<mwhudson> and similar for bugs, etc
<lifeless> an interesting little thing we could do now though
<lifeless> is to add a denormalised 'private' field to the bmp
<mwhudson> (big table) x (big table) numbers are pretty scary :)
<lifeless> that would let us do a union query, and only pay the privacy lookup for merge proposals of private branches
<lifeless> anyhow, I'm going to code up this with clause now
<lifeless> I might tap you for a review in a few
<mwhudson> sure, sorry for the distraction :)
<mwhudson> lifeless: sure, note that i'm only working a half day today
<lifeless> no problem, totally on topic
<mwhudson> cool
<lifeless> mwhudson: kk
<lifeless> hmm, that reminds me, I still haven't mailed flacoste about memorial day
<wallyworld> thumper: morning. did you want a standup?
<lifeless> mwhudson: also - https://bugs.launchpad.net/launchpad/+bug/742916
<_mup_> Bug #742916: BranchMergeProposal:+index timeouts - slow query plan <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/742916 >
<lifeless> wallyworld: he's at the physio
<wallyworld> lifeless: ah ok. thanks
 * wallyworld goes to have breakfast then
<mwhudson> lifeless: well, branchrevision is its own well known kettle of fail
<mwhudson> one might be able to do something clever with a recursive with query on revision there
<mwhudson> hang on, why are we filtering by date?
<mwhudson> oh right, the comment makes sense now
<mwhudson> won't all the revs have already been accessed for the 'unmerged revisions' section?
<mwhudson> i guess that query is limited to 10
<lifeless> mwhudson: right, so if we can rephrase it to be a traversal on branch
<lifeless> mwhudson: or sequence limited or some such, we can probably make it massively more efficient
<lifeless> mwhudson: some branches in lp have /huge/ revision counts
<mwhudson> lifeless: it would be interesting to see how a recursive with that walked revision parents performed
<lifeless> mwhudson: we seem to bring in maybe 10K fresh revision s aday
<lifeless> mwhudson: during some days
<mwhudson> ah, you mean lp overall here, not lp-the-project?
<lifeless> yeah
<lifeless> so if you look at the plan
<lifeless> the 9772 or whatever it is, is new-revisions-in-that-period
<lifeless> which then get looked up in branchrevision
<mwhudson> oh
<mwhudson> yeah, that doesn't seem very good
 * thumper is trawling through email
<lifeless> the obvious other vector - loading all revs in br first then filtering, is also not very good
<mwhudson> right, that would work better for a smaller project than lp i guess
<mwhudson> s/would/might/
<lifeless> and cripple mr onion on bigger ones
<lifeless> select branch, count(*) from branchrevision group by branch order by count(*) desc limit 10; - still running
<mwhudson> i'm not surprised
<mwhudson> launchpad is over half of that table iiuc
<mwhudson> err
<mwhudson> oh i expect the top few will be kernel imports though
<wgrant> Morning.
<lifeless> mwhudson: the table is 34GB
<lifeless> mwhudson: 10% of LP
<lifeless> mwhudson: the *indices* are bigger
<lifeless> mwhudson: still -  branch | count
<lifeless> --------+--------
<lifeless>  275684 | 233206
<lifeless>  401221 | 222008
<lifeless>  409550 | 212471
<lifeless> mwhudson: is where we need to scale up to
<lifeless> jkakar: still up ?
<thumper> lifeless: abentley and jam worked together for a plan to replace branchrevision
<thumper> lifeless: we should really bump up the priority on getting that actually in and working
<thumper> lifeless: using bzr-historydb
<thumper> lifeless: backed by postgresql
<thumper> lifeless: all the existing uses for branch revision would be catered for
<thumper> lifeless: and much less disk space
<mwhudson> then it would be awesome to hook loggerhead into the same data
<lifeless> that would be cool, yes.
<lifeless> uhm
<wgrant> thumper: Is it safe to use on such an untrusted scale?
<lifeless> I haven't looked at the detail of that plan
<thumper> mwhudson: yes, the plan was to have loggerhead use the same data
<mwhudson> wgrant: it's not like the approach we're using is safe :)
<thumper> wgrant: don't know
<lifeless> I'd like to do so before talking prioritisation of it - we're not using the existing data structure anywhere near as well as we can
<wgrant> eg. what happens if I deliberately introduce revid collisons?
<wgrant> mwhudson: True.
<thumper> lifeless: go and look at it and chat with abentley and jam, they know everything :)
<lifeless> mwhudson: you finish in 45 right?
<mwhudson> lifeless: probably more like 75
<lifeless> mwhudson: so, I've just figured out the voodoo to reuse select expressions in With clauses
<thumper> phew
<thumper> unread emails all looked at
<lifeless> but I'd like your review on the patch proper; how about I push up [when I've done it] the lp changes without the new storm, you review that expecting me to bump the storm egg trivially, and on we go ?
<lifeless> this is the bit for storm:
<lifeless> https://pastebin.canonical.com/45270/
<mwhudson> lifeless: that sounds sane enough
<wgrant> What :/
<wgrant> I'm running process-upload.py on DF with LP_DEBUG_SQL=1
<wgrant> And processing this source upload is going through hundreds of unrelated binarypackagenames.
<lifeless> wgrant: \o/
<wgrant> Oh.
<wgrant> P-a-s, you need to die.
<wgrant> So *that's* why it takes several minutes to run.
<lifeless> oh damn
<lifeless> some constraints should apply to just the source branch
<lifeless> e.g. owned by
<lifeless> and branchcollection doesn't distinguish
 * lifeless swears about BranchCollection some more
<thumper> don't dis the collection
<lifeless> wgrant: you're qaing?
<wgrant> lifeless: We're up to date.
<wgrant> There was only one item that wasn't covered over the weekend.
<thumper> lifeless: I have a question for you about a review you did before I was off last week
#launchpad-dev 2012-03-19
<lifeless> StevenK: hey
<lifeless> StevenK: how is the audit service coming along; do you need any input on it ?
<james_w> any zope experts around?
<lifeless> !ask
<james_w> I've lost my question now
<james_w> I think grokcore.component.global_adapter(DjangoLocation, IDjangoLocation, ILocation) isn't doing anything
<james_w> but I don't know how debug that
<StevenK> lifeless: Input would be awesome
<james_w> hmm, it's not that it seems
<james_w> it's "directives.location_interface(IDjangoLocation)" whatever that means
<james_w> ah, it seems my requests don't implement IWebServiceLayer
<james_w> no, that's not either
<james_w> marvellous
<james_w> it was a zope.traversing version incompatibility
<james_w> http://pypi.python.org/pypi/zope.traversing/3.13 breaks lazr.restful it seems
<nigelb> wgrant: Nice!
<huwshimi> wgrant: Hi, the change you made to the line height on Loggerhead seems to have created a mismatch between the code and the line numbers.
<wgrant> huwshimi: Which browser? The only one I've heard bad things about is whatever iOS has.
<huwshimi> wgrant: Oh right. This is with Chromium 17.0.963.79 (Developer Build 125985 Linux) Ubuntu 12.04
<wgrant> Hmm, indeed.
<huwshimi> wgrant: It looks fine on Firefox
<huwshimi> (whatever version I have)
<wgrant> Yeah, Opera and IE too.
<wgrant> Just WebKit is broken.
<huwshimi> :(
<wgrant> I'm sure I tested it in Chromium, though. Maybe it was an old version.
<nigelb> maybe the version number updated right after you checked... :P
<huwshimi> wgrant: My guess is that it has something to do with the <pre>
<huwshimi> wgrant: You might be able to do it without the <pre> and use .viewLine a { display: block;}
<wgrant> It's the line-height on the pre
<wgrant> Dropping that makes it work everywhere.
<wgrant> Just makes it slightly more obese in Firefox
<huwshimi> ah right
<lifeless> StevenK: if you want a chat or whatever, I'm free
<StevenK> lifeless: Chat works
<lifeless> skype then?
<StevenK> I knew you were going to say the evil s word
<nigelb> hahah
<lifeless> â¥ skynet
<nigelb> I thought Mumble was more evil anyway?
<lifeless> its technically poorer thats for sure
<wgrant> You know, it would be really awesome if testing in OS X and iOS browsers didn't require purchasing Apple hardware.
<StevenK> Hahaha
<nigelb> wgrant: need help with testing?
<StevenK> lifeless: Let me get a cup of tea and setting up skype
 * nigelb is on the mac.
<wgrant> Nah, I'll just hope that Win32 Safari is vaguely similar.
<wgrant> Thanks for the offer, though :)
<nigelb> :)
<bigjools> not sure if hangouts are more evil either
<wgrant> hangouts are a binary plugin that I have yet to convince to not hang Firefox or Chromium.
<wgrant> They are more evil than Skype.
<wgrant> And that's saying something.
<bigjools> I never had any probs like that with it
<nigelb> heh
<nigelb> skype on linux is good. BUt on mac it's severly pita.
<bigjools> well both are closed source... which do you trust more? :)
<bigjools> sound quality on skype is bloody excellent though
<wgrant> TBH I trust Microsoft more than Google right now...
<bigjools> their echo cancellation is something to behold
<nigelb> lol, me too.
<nigelb> and skype scales better than google hangout.
<StevenK> wgrant: BugsInformationTypeMigrator is getting blocked a lot
<bigjools> my data usage under hangouts has been insane
<wgrant> StevenK: Sure, but surely not all weekend.
<StevenK> wgrant: Pretty much
<StevenK> VACUUMs, backups, PRF and retry_depwait
<wgrant> Hmm
<wgrant> Sigh
<wgrant> Loggerhead's YUI is an embedded copy of 3.0.0pr2
<wgrant> Truly the pinnacle of good practice and modern technology.
<wgrant> bigjools: How is MaaS handling its YUI3 dep?
<wgrant> I don't see it packaged in Ubuntu.
<bigjools> convoy
<wgrant> How does it get into convoy?
<rick_h> it's in precise now and I uploaded it to pypi on friday finally
<bigjools> files are in maas
<wgrant> O_O
<rick_h> rvba said it's going into precise
<bigjools> python-django-maas
<wgrant> Embedding JS libraries... aaaaaaaaaaaaaa
<bigjools> until they're packaged.... any better idea?
<wgrant> Package it so you don't have to embed multiple copies of 400KLOC JS libraries? :)
<wgrant> I'm surprised the archive admins let that through.
<bigjools> meanwhile, in the real world
<wgrant> StevenK: I am sad.
<StevenK> wgrant: Why this time?
<wgrant> StevenK: Your kin let MaaS through NEW with an embedded third-party 400KLOC JS library.
<StevenK> lol
<StevenK> python-django-maas is not a source package ?
<wgrant> Source is just maas
<huwshimi> wgrant: We wanted to have our js libraries as dependencies, but we need security audits etc. and there's just not the time\
<wgrant> $ du -sh
<wgrant> 34M	.
<wgrant> $ rm -r src/maasserver/static/jslibs/yui
<wgrant> $ du -sh
<wgrant> 1.9M	.
<bigjools> meanwhile, in the real world.... people need to get work done
<wgrant> huwshimi: I'm not sure that gluing multiple packages together is an accepted way to bypass security audits :)
<StevenK> bigjools: And it's that attitude in 2007 that got LP into the state it's in. So think very carefully.
<wgrant> It's similar to the Go static linking crap a couple of weeks back.
 * bigjools sighs heavily
<bigjools> you think we don't know this?
<bigjools> you think we're stupid?
<StevenK> Oh clearly, that's *exactly* what I'm saying.
<bigjools> please, feel free to package it for us then, while meeting all the deadlines we have
<StevenK> Last time I offered suggestions on your packaging, you told me to shut up, so I don't think so.
<bigjools> thought not
<huwshimi> wgrant: Actually, it seems like exactly the way to bypass all that :)
<wgrant> huwshimi: It certainly works, but it's not accepted :)
<huwshimi> wgrant: Sure, we all wanted to do this properly (in fact I spent this morning ripping out a js dependency and putting it in our tree). But with 4 days to go, it's become unavoidable.
<huwshimi> wgrant: So I've done EXACTLY what you're talking about, but I can't be in control of the packages in main.
<wgrant> Things that security corners shouldn't be cut on, in descending order of importance:
<wgrant>  1) Platforms for managing your corporation's entire computing infrastructure.
<wgrant>  2) Everything else :)
<StevenK> huwshimi: Sure you can. If it's not in main, file a MIR
<lifeless> what this really shows is that maas is the first YUI using thing we've got for main inclusion
<StevenK> There is a process for how source and binary packages move between components
<huwshimi> StevenK: I know there's a process for getting it into main, but that doesn't mean we have the time to get it done
<StevenK> huwshimi: It takes about an hour to write the MIR, and you can talk to someone in ubuntu-mir about an urgent request
<huwshimi> lifeless: It's not just YUI
<huwshimi> StevenK: The people responsible for packaging MAAS have already told us we don't have enough time (on top of all the packages we already have to get included).
<StevenK> lifeless: objects.find() you say?
<lifeless> StevenK: erm, I think Imeant filter.
<lifeless> StevenK: https://docs.djangoproject.com/en/dev/topics/db/queries/
<lifeless> https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.get vs https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.filter
<StevenK> lifeless: Yeah, I've figured that out
<StevenK> Now I'm trying to work out why I can't pass lists or tuples as the value of a thing to search by
<StevenK> It seems to get forced to unicode
<mwhudson> StevenK: what are you trying to do?
<mwhudson> ah, bad time to offer help, i'm about to run away
<StevenK> Haha
<lifeless> StevenK: Entry.objects.filter(id__in=[1, 3, 4])
<StevenK> lifeless: Yes, I got that bit.
<lifeless> StevenK: so whats the code that is going wrong look like ?
<StevenK> lifeless: My issue is c.get('/fetch/', {'operation': ['foo', 'bar']}) ends up with request.GET['operation'] == u'bar'
<wgrant> Yes.
<lifeless> StevenK: hangon, is this webservice marshalling, or ORM stuff ?
<wgrant> Lists of HTTP GET arguments are serialised as operation=foo&operation=bar
<StevenK> lifeless: The former
<lifeless> StevenK: as wgrant says, HTTP params should be listified by the django framework; what are you using to submit the request ?
<StevenK> Django's Client class
<wgrant> You may have to use a special syntax instead of request.GET['operation']
<wgrant> Or you may have to call with [('operation', 'foo'), ('operation', 'bar')]
<StevenK> It wants a dict, though
<wgrant> What is the generated URL?
<StevenK> How do I pull that out of Client?
<wgrant> request.GET.getlist('operation') is what you want.
<StevenK> Indeed. Thanks.
<StevenK> It even works, neat.
<huwshimi> I love that you can decide that the code you're working on should be in a new branch and you can just add a new pipe and all your uncommitted changes get pushed into a new branch
<bigjools> yeah I get the same thing with co-located checkouts
<wgrant> Slow TALES is slow.
<stub> wgrant: I have built a Slony package under Lucid for PG9.1. I neglected to rename the source package. Do things explode if I copy the built packages across to the Launchpad PPA?
<wgrant> stub: Let me see.
<wgrant> stub: It appears to be built for both 8.4 and 9.1, which is probably want we want
<wgrant> (looking at 2.0.7-3ppa1 in ppa:stub/launchpad)
<wgrant> s/want we want/what we want/
<stub> wgrant: Gah - debversion
<wgrant> stub: It won't blow up, but it will be difficult to update either of them and be very confusing and leave orphaned binaries around.
<wgrant> So I would rename.
<wgrant> As, IIRC, I did with 8.4 for oneiric.
<stub> Ok. Ta.
<stub> Yer, just realized on the weekend. I did it last time (when I built under Oneric and copied back to Lucid, which was a #fail)
<lifeless> wgrant: you could try pybars
<lifeless> StevenK: unless the internet, or #isd, have answers, you'll need to step through with pdb
<wgrant> lifeless: It's actually the TALES in browser:url that's upsetting me right now.
<wgrant> (canonical_url on a person can take more than 200Âµs in a LaunchpadSecurityPolicy environment, 80Âµs in PermissiveSecurityPolicy, and about 25Âµs of that goes away when I replace the TALES-based IPerson browser:url with a Python adapter.
<wgrant> And it takes 500Âµs to load a single Person from a DB result :/
<wgrant> So to generate two links to a person you've got 1ms :)
<wgrant> Speedy Zope is speedy.
<lifeless> canonical_url is known for its lightning fast speed and efficient operation
<adeuring> good morning
<stub> Getting a bzr error with a recipe build: http://paste.ubuntu.com/890439/
<jelmer> stub: you have  tag that's pointing at a revision that doesn't exist
<jelmer> stub: it's trying to export that tag to create the upstream tarball
<stub> htf did I manage that I wonder?
<stub> Oh - push --override might do that?
<jelmer> yeah, it won't push history that's not related to the branch tip
<stub> Should there be a bug on this?
<stub> (push --override leaving old tags pointing to revisions that no longer exist?)
<jelmer> it's not necessarily wrong to leave that tag around - the revision is still accessible in your local branch
<jelmer> it just won't be pushed along
<jelmer> there's a few bugs about tag handling in this regard already
<jelmer> it would be nice to file a bug about the bzr-builder error though, it should at least tell you what tag it tried to resolve
<stub> huh - cosmic ray. "bzr: ERROR: Corruption while decompressing repository file, zlib: Error -3 while decompressing data: incorrect data check" but repeating the command, works fine
<czajkowski> jtv: morning
<jtv> Good morning czajkowski
<czajkowski> jtv: would you have some time free to help me with 2 lanauge questions either now or later on
<czajkowski> please
<jtv> Let's get it done quickly, since night just fell here.
<jtv> Where are they?
<czajkowski> jtv: https://answers.launchpad.net/launchpad/+question/191090
<czajkowski> https://answers.launchpad.net/launchpad/+question/173909
<czajkowski> the latter had a bug linked to it and has been released but they are still having issues.
<jtv> czajkowski: the first one is for Ubuntu, not for us.
<czajkowski> grand
<czajkowski> jtv: thanks for the help. Also does this make sense to you?  https://bugs.launchpad.net/launchpad/+bug/957884
<jtv> czajkowski: the second one, I think, may need manual intervention from someone more knowledgeable such as sinzui.  I did propose something that would resolve a lot of these problems in practice, but it was considered too radical.
<czajkowski> nods
<czajkowski> I'll bring it up later as the fix that is currently in place isn't fixing things
<jtv> Yes, IIRC the bug is about not producing dangling membership requests any more.  Which is not the same as dealing with existing ones.
<czajkowski> jtv: thanks for the help.
<jtv> Good luck :)
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugtasks: 4*10
<deryck> Morning, all.
<czajkowski> deryck: howdy doody
<wgrant> czajkowski: A timeout on production is always a bug.
<wgrant> Triaged/Critical/timeout
<wgrant> qastaging/staging are slow and non-production, so it doesn't apply there.
<deryck> adeuring, abentley, rick_h -- coming for standup, just taking a second to fire up hangout
<abentley> adeuring: Can we chat?
<adeuring> abentley: sure. mumble?
<abentley> adeuring: sure.
<mrevell> Hey deryck, czajkowski kindly pointed me to bug 138775. Some bug mail is going out with truncated subjects, so no bug summary shows. czajkowski says that complaints about this issue have increased since Wednesday, suggesting something may be making it worse. Any thoughts?
<deryck> mrevell, hey, let me look at the bug.
<deryck> mrevell, hmmm, I have no idea what could have changed to make it worse.  Let me look around a minute or two.
<deryck> mrevell, do you know if it's just watched bugs, or happening on any kind of bug?
<mrevell> czajkowski has just gone on lunch. I'm guessing she'll see her name highlighted when she gets back. She can probably fill you in better than I can.
<czajkowski> deryck: happens a lot on https://bugs.launchpad.net/ubuntu/+source/gnome-terminal/+bug/190848
<czajkowski> deryck: and https://bugs.launchpad.net/xserver-xorg-video-intel/+bug/926379
<czajkowski> <-- running out door
<deryck> ok, thanks czajkowski
<tumbleweed> poke: https://code.launchpad.net/~stefanor/wadllib/datetime-924240/+merge/97099
<tumbleweed> I'm about to apply that patch in Debian (it works fine for me) but would obviousuly prefer it if it was already merged into trunk
<deryck> tumbleweed, benji is the on call reviewer.  perhaps he could look at that branch for you?
<deryck> tumbleweed, rick_h would have an interest in this, too, since he did some of the original work but he isn't around right now.
<tumbleweed> deryck: yeah, I see LP made him the reviewer
<deryck> czajkowski, so I don't see anything recent that could be to blame for the increased instances of this....
<deryck> czajkowski, and this seems related to freedesktop.org watched bugs, based on your two examples....
<deryck> czajkowski, so either something changed on their end to make this worse....
<deryck> czajkowski, or else it's always been happening and the importance or attention given to these bugs is making it seem more common.
<deryck> czajkowski, so I'd suggest getting some of the mails showing this attached to the bug report.  I've subscribed to stay informed.
<deryck> czajkowski, but if I don't reply timely enough, another ping to me if fair and appreciated :)
<deryck> s/if/is
<czajkowski> deryck: thanks for the help
<czajkowski> I've taken a screen capture of my bug mail
<czajkowski> and added it to the bug
<deryck> czajkowski, can you add full emails with headers and all?
<czajkowski> deryck: done I think
<deryck> czajkowski, thanks.  I'll try to dig in more later.
<czajkowski> cheers
<czajkowski> deryck: you about?
<czajkowski> I'm pretty sure I just made a large boo boo error :/
<deryck> czajkowski, hey, yes, I'm around.  what's up?
<czajkowski> and heart failure over
<rick_h> deryck: tumbleweed sorry for the delay, it's on my todo for today. I need to get py3 setup to retest it out.
<rick_h> tumbleweed: did you run the tests on both py2 and 3?
<rick_h> tumbleweed: we had to hack that in to get things happy on both sides and if that fixes it, we should probably add a test to verify it works with the right date strings on both versions of python
<tumbleweed> rick_h: np
<tumbleweed> yes, ran it on py2 and 3
<tumbleweed> yeah, it could definitly use a test case, but I don't know the code well enough for that
<abentley> adeuring: I am working on adding Celery & dependencies to the download cache.  I don't think you need to do it in your branch.
<adeuring> abentley: cool
<jcsackett> benji: can i throw https://code.launchpad.net/~jcsackett/launchpad/sharing-details/+merge/98223 on to your queue?
 * jelmer wished testr gave some progress output..
<benji> jcsackett: I'm back from lunch and will look at your MP shortly.
<rick_h> abentley: ping, I'm trying to see how I update/get lp-dev-utils correctly?
<abentley> rick_h: Okay, how can I help?
<rick_h> abentley: so this doesn't appear to be a python pacakge, or a .deb package?
<rick_h> abentley: so do I just download it somewhere and add it to path? I'm a bit confused on how it's meant to be used/run
<abentley> rick_h: Right, it's a bzr branch.  lp:lp-dev-utils.
<abentley> rick_h: Yes, you download it somewhere, and add it to a path.
<rick_h> abentley: ok, so it's not meant to set itself up then, just up to me to add it somewhere I can assess it
<rick_h> abentley: ok, gotcha, thanks
<abentley> rick_h: I have a symlink to ~/launchpad/lp-dev-utils/ec2 in ~/bin
<rick_h> abentley: thanks, just wanted to make sure I wasn't missing some package or something
<deryck> abentley, hey, man. Since you started to look at it the first time, would you be willing to review my preload people branch now it's ready again?
<abentley> deryck: sure.
<deryck> abentley, thanks!  https://code.launchpad.net/~deryck/launchpad/buglistings-preload-people-901122/+merge/97041
<abentley> deryck: r=me
<deryck> abentley, awesome, thanks!
<abentley> deryck: I've written dict(foo.id, foo for foo in bar)  a lot, too.
<abentley> I'm getting a repeated assertion failure when I do "make run_all".  Anyone else seeing this?  http://pastebin.ubuntu.com/891134/
 * deryck heads out for the day with family
<sinzui> benji, do you have time to review https://code.launchpad.net/~sinzui/launchpad/project-notify-3/+merge/98282
<benji> sinzui: sure
<benji> sinzui: done
<sinzui> wow. Thank's benji
<benji> sinzui: anything to keep me from thinking about performance reviews
<sinzui> benji, Indeed. I was depressed writing mine last week. benji Keep in mind that the most important thing you did was help your squad and the Lp team. All personal goals are not more important than the group goals. One year my team missed our personal goals, but it was okay because we responded to the group goals that emerged during the year
<benji> sinzui: thanks for the encouragement
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<lifeless> flacoste: did you want a call ?
 * jelmer is reminded why he contributes to launchpad in bursts
<salgado> lifeless, we seem to have encountered a weird issue with db updates not being flushed by storm before it executes a query... if you have a couple minutes to help us, it's at https://code.launchpad.net/~linaro-infrastructure/launchpad/notify-workitems-changes/+merge/98231
<lifeless> jelmer: because of the excruiating overheads?
<lifeless> salgado: looking
<jelmer> lifeless: yeahp
<lifeless> salgado: the transaction.commit() in prod code is a problem
<lifeless> salgado: as in, its a blocking problem; it will cause incorrect behaviour on retries
<salgado> lifeless, right, we're not keeping that
<lifeless> ok
<lifeless> so I have a few thoughts so far
<lifeless> firstly, event based email sending is terrible.
<lifeless> I do mean that literally, it isn't hyperbole.
<lifeless> its a wonderful concept that performs poorly and is hard to customise, extend and reason about
<lifeless> salgado: that said, I can see you're building on an existing hook, which means you'd have work to do to change things
<salgado> yeah, and probably a lot of work given that this is blueprints
<lifeless> (also, the object delta stuff in lazr.restful is a direct cause for dozens of different timeouts)
<salgado> heh
<lifeless> because doing an in python diff to determine that one message was added to a bug loads multiple MB's of data in an inefficient manner, to reconstruct the knowledge that the API layer had discarded about what it did.
<lifeless> I have sketched a new subscriptions API
<lifeless> on the list
<lifeless> noone has implemented it, and its a bit big to slide into what you're doing, your awesome notwithstanding
<lifeless> salgado: anyhow, the diff on that page has a commit() and no flush calls
<lifeless> salgado: what happens if the commit is removed?
<salgado> lifeless, https://code.launchpad.net/~linaro-infrastructure/launchpad/notify-workitems-changes/+merge/98231/comments/212211
<salgado> "Now, it looks like the work item changes are not always flushed to the DB when notify_specification_modified is fired (as the pdb session below shows), so that would explain why the diff looks wrong in some cases."
<lifeless> salgado: IIRC the only reason a select wouldn't see new objects is if both the select and the store.add for the objects *both* happen within a block_implicit_flushes() decorator
<salgado> it does!
<salgado> the subscriber has that
<lifeless> well
<lifeless> that will cause the behaviour you see :)
<lifeless> you might want to figure out why it has that decorator
<lifeless> :cn
<lifeless> a quick grep shows a lot of event subscribers having it
<lifeless> this bothers me
<salgado> yeah, all the blueprints-related ones have it
<lifeless> and karma
<lifeless> this smells to me of some bug patched over
<lifeless> if it was e.g. a truism that all event handlers must not permit implicit flushes, we'd have glued it into the event subscription code to avoid the repetition
<lifeless> salgado: I would look in annotate for the commit adding the decorator and see what it was part of; look at the MP too probably; and definitely try removing the decorator and seeing what happens
<salgado> there seems to be plenty of subscribers not using that
<salgado> lifeless, it was added when we migrated to storm, I'm pretty sure
<lifeless> sure, but why :)
<lifeless> salgado: ok, I'm going to context switch, if you want more chat, ping me again :)
<salgado> lifeless, with the reorganization of the code, how I'd go about finding the revision where that was introduced?  the latest commits that touched that code were actually just moving files around... is there a way I can find the first commit which contains a block_implicit_flushes?
<lifeless> well, file renames won't affect annotate
<lifeless> but code copies will
<lifeless> so you can jump back - gannotate will let you do this easil, or you can annotate -r before:lastcommityouknowabout filename
<salgado> lifeless, ok, will try that.  thanks a bunch
<lifeless> no worries
<lifeless> you can also pass a regex to bzr log (and a filename too)
<lifeless> or use bzr-search if you have a search index built
<salgado> lifeless, I've found, but it doesn't say anything about why it was needed so I emailed the list
<wallyworld_> sinzui: bug 959784
<_mup_> Bug #959784: Cannot add access to new information type without destoying existing 'Some' access <disclosure> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/959784 >
<wgrant> sinzui: https://code.launchpad.net/~wgrant/launchpad/sharing-help-fixes/+merge/98300
<sinzui> r=me
<wgrant> Thanks.
#launchpad-dev 2012-03-20
<StevenK> $ grep BugsInformationTypeMigrator garbo-hourly.log-20120320 | grep -c Blocked
<StevenK> 117
<StevenK> wgrant: ^
<wgrant> StevenK: Well, the problem here is clear.
<wgrant> The PRF has gone insane. Get it killed.
<wgrant> wallyworld: So, using StormRangeFactory will generate a query something like https://pastebin.canonical.com/62588/
<wgrant> wallyworld: Note the slight oddness in that you have to include the sort key in the SELECT list, due to the DISTINCT.
<wallyworld> wgrant: ok, seems simple enough. how do i tell the batching infrastructure to use it?
<wgrant> wallyworld: There's only a couple of places that use it so far, but see eg. ArchiveSubscribersView
<wgrant>         self.batchnav = BatchNavigator(
<wgrant>             self.subscriptions, self.request,
<wgrant>             range_factory=StormRangeFactory(self.subscriptions))
<wallyworld> ok, thanks
<wgrant> That is up to 30% faster than the original query.
<wgrant> Then you just need https://pastebin.canonical.com/62590/, which takes <1ms for 100 people.
<wgrant> Intriguing.
<wgrant> Chromium is of the opinion that my launchpad.dev wildcard cert has been revoked.
<wgrant> Despite it not having a CRL or OCSP endpoint specified.
<wgrant> Anyone feel like reviewing https://code.launchpad.net/~wgrant/launchpad/involvement-config-validity/+merge/98328?
<StevenK> wgrant: r=me
<StevenK> wgrant: In my branch to change IBug.{private,security_related} to make use of information_type most of the TestBugMirrorAccessTriggers fail with AttributeError: 'NoneType' object has no attribute 'concrete_artifact'
<wgrant> StevenK: You'll need to change the triggers to use information_type instead of private and security_related
<wgrant> StevenK: There's a card in Next for that.
<wgrant> I was going to do it once production's column was populated.
<StevenK> Right.
<StevenK> But is sort of seperate to this work to move IBug definition
<wgrant> It means that setting .private and .security_related still has to set the DB flags.
<StevenK> wgrant: I'm tempted to re-write the garbo jobs __call__ to be faster and not call bug._setInformationType() in a loop
<wgrant> See how it goes overnight.
<wgrant> Killing PRF before you sleep if necessary.
<StevenK> wgrant: What about the VACUUMs that keep turning up?
<wgrant> They shouldn't be too terrible most of the time.
<lifeless> StevenK: what about it ?
<StevenK> lifeless: They seem to keep blocking garbo jobs
<StevenK> Just look at the trail of fail in garbo-hourly.log
<lifeless> autovacuum is more a symptom not a cause
<lifeless> it canm be blocked by other writes
<lifeless> even if it shows up, its almost certainly not the reason for lag
<wgrant> Lag is not relevant here.
<wgrant> Long transactions.
<lifeless> samesame
<lifeless> vacuum is only long when its blocked on another xaction
<wgrant> StevenK: /win 6
<wgrant> Bah
<StevenK> Haha
<wgrant> StevenK: How are you structuring your branches?
<StevenK> wgrant: I'm only working on one at the moment, and it's only 200 lines.
<wgrant> StevenK: Hm
<StevenK> wgrant: What are you thinking?
<wgrant> StevenK: But you appear to have a branch which stops writing to Bug.private and Bug.security_related
<wgrant> Which can't be in the same branch.
<StevenK> Not any more.
<wgrant> Ah
<wgrant> Good good.
<StevenK> IBug.private is now IBug._private and stuff is looking good.
<StevenK> Just running test -vvm bugs to see if there is anything heinous before I put up an MP.
<wgrant> This branch still keeps the flags and information_type in sync, just from the opposite direction?
<StevenK> Roughly
<StevenK> setPrivacyAndSecurityRelated is utterly horrid
<StevenK> So the less I touch it, the better.
<StevenK> I'm thinking we want to replace it with IBug.transitionInformationType() like the methods on IBugTask.
<wgrant> Yes.
<wgrant> 1) Get things keeping information_type in sync with private and security_related (done)
<wgrant> 2) Populate information_type (nearly done)
<wgrant> 3) Treat information_type as canonical, mirroring to private and security_related
<wgrant> 4) Get that deployed everywhere (including ftpmaster)
<wgrant> 5) Replace the triggers (I'm just testing that now)
<wgrant> 6) Stop touching private and security_related in any way
<wgrant> 7) Drop private and security_related
<StevenK> 8) Kill IBug.set{Private,SecurityRelated,PrivacyAndSecurityRelated}, replacing it with IBug.transistionToInformationType()
<wgrant> That's part of 6
<StevenK> Ah
<StevenK> I did wonder
<StevenK> Right, I do like that plan.
<StevenK> We have cards for part 6 and 7?
<wgrant> Not yet.
<StevenK> Shall I create two in Backlog, sans bugs for now?
<wgrant> Sure.
<wgrant> Also, I missed a step
<wgrant> 2.5) ALTER TABLE Bug ALTER COLUMN information_type SET NOT NULL;
 * StevenK stabs Kanban
<StevenK> Ah yes
<StevenK> Let me create a card for that and stuff into next
<StevenK> We'll need to fix the model/interface code seperately, but that's a trivial patch.
<wgrant> Are you fixing test_bug_mirror_access_triggers to not set private/security_related directly?
<StevenK> Already done
<wgrant> s/need/potentially and non-critically want/
<wgrant> StevenK: Can you push? My trigger branch needs that.
<StevenK> wgrant: What I mean is sprinkle things like notNull=True to information_type in the model
<wgrant> StevenK: I'm aware. We probably want to do that at some point, but it's not in the critical path.
<wgrant> Nor important at all.
<StevenK> I'd prefer to do it just after the DB patch is deployed, but meh
<StevenK> The following 1 account(s) were found. Select the account you wish to access.
<StevenK> <list of two things>
 * StevenK stabs Kanban harder
<wgrant> I only had one.
<StevenK> wgrant: Sorry, got distracted by Kanban. lp:~stevenk/launchpad/bugs-use-information_type
<wgrant> Thankyou sir.
<wgrant> StevenK: We should complete (and test as a whole) parts 3-7 before we land any of them.
<wgrant> In case we've missed something
<wgrant> 5 is done, asounds like you're close to 3 and 6, and 7 is pretty trivial :)
<StevenK> 3 yes, 6 no.
<wgrant> StevenK: Isn't 6 mostly a matter of deleting code?
<wgrant> Or does 3 not change the queries that use it?
<StevenK> 3 does not. I'm trying to keep it simple.
<wgrant> Ah, sure.
<StevenK> 3 falls back to _private and _security_related if information_type is not set, so it's not dependant on the garbo job being complete.
<StevenK> Hmmmm. That could not be helping.
<StevenK> wgrant: What should I set information_type to if IProduct.private_bugs -- I want to say PROPRIETARY.
<wgrant> StevenK: USERDATA
<wgrant> PROPRIETARY doesn't exist yet.
<wgrant> It's defined, but it doesn't exist.
<StevenK> wgrant: Are you still reviewing since you're OCR and everything.
<stub>  LINE 3: ... Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  WHERE bt.bug=bug.id ORDER BY bt.name LI...
<stub> Could have sworn we fixed that one already (BugTask.name does not exist...)
<stub> Scoping rules seem subtly different in 9.1 if you use the same tag to alias more than one table in a nested query
<adeuring> good morning
<czajkowski> aloha
<jelmer_> moin adeuring, czajkowski
<dpm> hi jtv, are you around?
<jtv> Hi dpm!  Trying to have a conference call atm, but can get back to you in half an hour or so if that's okay.
<dpm> jtv, sure! if you've got 5 minutes for me later on, that'd be perfect
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Firefighting: - | Critical bugtasks: 4*10
<rick_h> morning
<deryck> Morning, everyone.
<rick_h> morning
<deryck> abentley, adeuring, rick_h -- I'm coming.  joining hangouts is slooooowww here for some reason.
<rick_h> deryck: all the beach air interfering with your bandwidth :P
<adeuring> abentley: can't hear you
<abentley> adeuring: My mic shows levels.  Let's both rejoin.
<adeuring> ok
<adeuring> abentley: let's use muble. Google claims that have to install an new plugin...
<abentley> adeuring: I think job.fail needs transaction.abort before it.  Here's how it fails without it: http://pastebin.ubuntu.com/892304/
<adeuring> abentley: ouc, of course, yes
<abentley> adeuring: Maybe we should go back to having commit/abort methods on the runner?  I know it's not pretty.
<adeuring> abentley: I think we should first try to fix the bugs we have currnently. I really don't like to have DB commit into lazr.jobrunner...
<abentley> adeuring: Or maybe the runner could accept a transaction manager.
<abentley> as a parameter.
<adeuring> abentley: well, that's bascially teh same ;)
<abentley> adeuring: This is a bug that we have currently.
<abentley> adeuring: I was getting this, so I had to disable oops handling to get the real error.
<adeuring> abentley: does this mean that the OOPS handling is also broken?
<abentley> adeuring: The oops handling was broken because job.fail was not aborting the transaction, so when it tried to invoke methods on job, it got InternalError: current transaction is aborted, commands ignored until end of transaction block
<adeuring> abentley: ah, ok.
<abentley> adeuring: I've worked around all of this by doing "import transaction; transaction.abort()" immediately before job.fail, but this is not a long-term solution.
<adeuring> abentley: the branch I'm working on should fix the problem: it changes the methods fail(), queue(), suspend() etc
<abentley> adeuring: Great.
<deryck> abentley, rick_h -- Just FYI for the YUI history curious, our phantom history events is a bug fixed in YUI 3.4.1.
<rick_h> deryck: awesome
<abentley> deryck: Ah, great.
<deryck> abentley, rick_h -- http://pastebin.ubuntu.com/892344/ shows the issue.  simple initialState bug.
<deryck> abentley, rick_h -- I vote for waiting on upgrade to fix it. :)
<rick_h> deryck: yea, guess so
<deryck> only other option is to fork our 3.3.0 and patch the history module.
<rick_h> right, messy
<deryck> then we have a diff against 3.3.0, have to maintain our own build, etc. when the issue is annoying but relatively minor.
<sinzui> deryck, rick_h what is stopping us from using 3.4.1?
<deryck> sinzui, upgrade time && too many js irons in the fire.
<rick_h> sinzui: hopefully the combo loader will make it much easier to swap out versions, run all our tests, and find out what issues pop up
<sinzui> :( this is why we stopped updating zope for 3 years
<sinzui> rick_h, well if the calendar dies, we have incentive to use 3.4.1 with a hack to make the popup work in firefox
<rick_h> sinzui: that and 3.5 is due next month so the hope is more to jump to 3.5
<sinzui> That would make me happy
<deryck> I don't think it's far off actually.  Just not NOW()! :)
<rick_h> sinzui: yea, so the plan isn't to not upgrade, but try tomake one big jump with combo loader + 3.5 in sucession
<deryck> that's the chief driver of the combo loader, to make upgrades much easier.
<rick_h> yea, it's all kind of tied together with combo loader, 3.5, handlbars/pybars. Lots of parts moving that hopefully we can snap together
<deryck> exactly.
<deryck> and that's also what gives me hope that an upgrade is not just vapor, but something that will happen.
<rick_h> tumbleweed: ping
<sinzui> jcsackett, do you have a few minutes to talk about the auditor editing case in the the mockups
<jcsackett> sinzui: sure.
<jcsackett> sinzui: can you start the hangout? i do not have a functioning webcam, so will have to use my phone.
<sinzui> jcsackett, starting now
<sinzui> jcsackett, does you phone see a hangout with me in the messenger?
<tumbleweed> rick_h: pong
<rick_h> tumbleweed: wanted to see if you could help me out with the test data so I can get the test working for that MP
<rick_h> I posted back on the MP what I got going, and need to see if the test data in wadllib is out of date
<tumbleweed> rick_h: I can't see pastebin.canonical.com
<rick_h> tumbleweed: k, sec
<rick_h> http://paste.mitechie.com/show/569/
<rick_h> basically checking the date formats given the test json data in wadllib works out, so curious if the json out of that bug is different and needs updating
<tumbleweed> thanks for actually doing a test. I was just testing it from launchpadlib, against lp
<lifeless> flacoste: hey
<rick_h> tumbleweed: ok, I wasn't sure what the test is hitting and wondered if you could shoot me a json file to check against
<flacoste> hi lifeless!
<flacoste> yes, it would be a good time to talk!
<rick_h> tumbleweed: end of the day, the change seems good, but I'd like to get a good test to make sure we don't break things again
<tumbleweed> rick_h: IIRC the problem was that namespace_url (line 483 of application.py) was None
<lifeless> cool, @ 30 past? Would like to nab a snack first
 * lifeless is astonished that pg 8.4 handles bugtag.name without blowing up
<rick_h> tumbleweed: ugh, ok, so deeper issue than it seemed
<rick_h> tumbleweed: ok, the source for that test stuff is open right? if so I guess I'll just pull that down and trace through to be able to duplicate something
<tumbleweed> rick_h: you should be able to test it by checking isinstance(lp.distributions['ubuntu'].date_created, datetime.datetime)
<rick_h> tumbleweed: right, but I need to get that into a unit tests with a json api response and an existing known wadl file. but yea, I'll pull that down and try to get it going
<rick_h> tumbleweed: thank
<tumbleweed> of course :)
<rick_h> it helps to know that's what the bug was hiting, couldn't tell that input the test file in the pastebin from the bug was using as input
<lifeless> flacoste: okies
 * flacoste starts hang out
<lifeless> gary_poster: do you have a little time today, once I'm finished with flacoste, to talk paralleltest ?
<gary_poster> lifeless, I have my call with Francis in 24 min, and am trying to get a machine ready for analysis right now.  I know it would be good to talk.  We also have our scheduled call tomorrow though
<gary_poster> IOW, not easily, now
<gary_poster> no
<lifeless> gary_poster: no worries; perhaps we can talk a little tomorrow 1:1
<gary_poster> ok cool
<gary_poster> lifeless I'll ping if I have time before EoD today in case you are around/available
<lifeless> cool, thanks
<gary_poster> lifeless, if we can make it fast we could talk now
<gary_poster> lifeless, https://talkgadget.google.com/hangouts/extras/canonical.com/goldenhordeoneonone ?
<kirkland> getting some weird LP oopses ... anyone around to take a look?
<lifeless> kirkland: gotta link the OOPS man :P
<lifeless> gary_poster: sure, trying to get in
<lifeless> this multi account stuff really is a nuisannce
<kirkland> lifeless: copying over from #launchpad (where I did) ...
<kirkland> Sorry, something went wrong with your search. (Error ID: OOPS-3611440967e645c2a3d8c68345be9b24)
<gary_poster> lifeless, I tried inviting your @gmail.com
<lifeless> yeah
<kirkland> I'm also getting a "not found" on a bug that I've been looking at/working on all day
<lifeless> that used to work
<lifeless> seems to not now
<lifeless> gary_poster: sending you an invite instead
<gary_poster> :-) cool
<lifeless> kirkland: thats a 404; it is oopsing because you have a local referrer
<kirkland> lifeless: what does that mean, sorry for being dense?
<kirkland> lifeless: I've been working on this private bug all day long https://bugs.launchpad.net/ztrustee/+bug/959650
<kirkland> lifeless: and it's now disappeared for all I can tell
<mwhudson> i guess you no longer have permission to see it
<mwhudson> for whatever reason
<lifeless> kirkland: you may have been unsubscribed from it
<lifeless> kirkland: hmm
<lifeless> ztrustee may have been renamed :)
<lifeless> https://launchpad.net/ztrustree
<kirkland> lifeless: interesting;  I'm on the team that owns the project
<lifeless> kirkland: can you see the project?
<kirkland> lifeless: I can see ztrustee (which is right);  ztrustree is 404
<lifeless> hah, yes ok
<lifeless> so there are two possibilities
<kirkland> Include the error ID OOPS-e72a16ca0af50bf4338dddc14d2ba243 in your message.
<lifeless> we've nerfed something in private projects
<kirkland> lifeless: any info there?
<lifeless> or you / your team have been unsubscribed from that bug, leading to no access
<kirkland> lifeless: how odd, okay, thanks
<kirkland> lifeless: we can recreate the bug, that's no problem
<lifeless> its failing in   File "/srv/launchpad.net/production/launchpad-rev-14969/lib/lp/bugs/security.py", line 127, in checkAuthenticated
<lifeless>     return self.obj.userCanView(user.person)
<lifeless> which is a cannot-access situation
<kirkland> lifeless: this is the first time something like this has happened, though, so I thought I'd mention it in case it's systemic
<lifeless> kirkland: I'm 99% certain someone in your team unsubscribed the team from the bug
<kirkland> lifeless: okay, we'll go with that
<kirkland> lifeless: I'll re-raise if it happens unexpectedly again
<lifeless> what they need to do if they don't want notifications is to use the *mute* feature
<lifeless> until the disclosure project is rolled out
<lifeless> the disclosure project is decoupling access controll and notification
<kirkland> lifeless: k
<lifeless> which will avoid this rather spectacular foot gun
<kirkland> lifeless: btw, can you tell me who to thank for the recent "Affiliations" feature, when searching for a person or team?
<kirkland> lifeless: cool
<lifeless> purple panthers
<lifeless> the same folk working on disclosure
<lifeless> e.g. sinzui & co
<kirkland> lifeless: lovely
<kirkland> sinzui: thanks!!!
<sinzui> kirkland, I will pass that on to the squad. they did the testing and work to make it appear in real time
<kirkland> sinzui: it's *awesome*
<kirkland> sinzui: and exactly what I was looking for in Bug #930364, which was marked wont-fix
<_mup_> Bug #930364: show badges when searching/selecting a launchpad user <disclosure> <person-picker> <Launchpad itself:Won't Fix> < https://launchpad.net/bugs/930364 >
<kirkland> sinzui: I think you noted that in Comment 7, which is good enough for me!
<sinzui> fab
<sinzui> kirkland, we have started the UI for showing who has access to private data: https://qastaging.launchpad.net/ecryptfs/+sharing is a draft of a what we will announce in a few weeks
 * sinzui cannot see that page so cannot answer question about who has access and how, but the views to do that will be in place next week
<sinzui> StevenK, http://blog.launchpad.net/general/reimagining-the-nature-of-privacy-in-launchpad-part-1
#launchpad-dev 2012-03-21
<wgrant> StevenK: Hm, you didn't move the two constants?
<wgrant> AFAICS you just renamed them.
<wgrant> PRIVATE_BUG_TYPES etc.
<StevenK> I was supposed to move them?
<wgrant> I said they should be alongside the enum.
<wgrant> Perhaps I was unclear.
<wgrant> But there's nothing bug-specific about them.
<StevenK> wgrant: Sorry, I've moved them to registry.enums
<wgrant> Thanks.
<cody-somerville> sinzui, That page seems to scroll really slowly in my Firefox.
<wgrant> cody-somerville: For which project?
<cody-somerville> bugsy
<wgrant> Approximately how many rows does it have?
<cody-somerville> a lot.
<wgrant> Ubuntu has nearly 20000, which is a lot.
<wgrant> Launchpad has a few hundred, which is a lot.
<wgrant> What is a lot?
<cody-somerville> probably less than 100
<wgrant> Ah
<wgrant> Not a lot :)
<StevenK> wgrant: I guess I should kill IBug.set{Private,SecurityRelated} and related gubbins?
<wgrant> StevenK: Something like them is still necessary for the API.
<wgrant> (and +secrecy)
<StevenK> Rargh, setPrivacyAndSecurityRelated is still exported.
<wallyworld_> StevenK: you ocr?
<wgrant> StevenK: Here, I made three code reviews, just for you :)
<wallyworld_> and one from me
<wallyworld_> https://code.launchpad.net/~wallyworld/launchpad/tri-state-sharee-picker-959784/+merge/98556
<wallyworld_> it's mainly javascript too, and i know StevenK loves javascript :-)
<wallyworld_> and if he does them soon, we can make the next buildbot run :-)
<StevenK> I was reviewing my lunch.
<lifeless> nom
<StevenK> wgrant: Three? I can only see two.
<wgrant> It would help if I submitted the MP for the last one.
<StevenK> Haha
<StevenK> wgrant: All three done.
<StevenK> wallyworld_: You're up now.
<wgrant> StevenK: Thanks.
<StevenK> wallyworld_: Three comments, the first two are not optional, the last one is.
<wallyworld_> StevenK: thanks
<StevenK> wallyworld: If you hold down the "Propose merge" button it brings up 3 or 4 spinners and then errors horribly
<wallyworld> StevenK: so don't hold the button down :-P
<nigelb> lol
<huwshimi> StevenK, wallyworld: Clearly we need to have a notice that says "do not press more than once". I think that's how the banks handle it.
<wallyworld> huwshimi: yes, or something like that. disable on click even
<StevenK> No, I think we need do it on mouseUp
<wallyworld> yes, you are right
 * huwshimi thinks wallyworld missed the joke
<StevenK> I didn't think you got mulitple mouseDown events if you held the button down
<wallyworld> me wither
<wallyworld> either
<lifeless> stub: wgrant: have the two of you spoken about not using FK's in the FACT tables ?
<wgrant> No, but indeed we should.
<wgrant> stub: So I heard you like foreign keys.
<nigelb> ("Yo dawg")
<nigelb> I just imagined wgrant say it with that added.
<stub> wgrant: I shirley dooly
<wgrant> stub: I'm not sure they're valuable on fact tables.
<wgrant> And in some cases they're not possible.
<wgrant> eg. denormed arrays
<stub> wgrant: Looks like they will be in 9.2 or 9.3 :)
<stub> But sure, if PG doesn't support it that is a good reason to not use that case.
<wgrant> Given that they're maintained by tested triggers from FKed columns, the only purpose they serve seems to be to cause slow writes :)
<wgrant> Oh really?
<wgrant> Hmm
<wgrant> Oh, I see a patch. Nice.
<stub> I'm fine with not having them if the table is driven by triggers.
<stub> It is a harder decision to make if writes are distributed over more code, as we are more likely to screw up.
<wgrant> Right. For tables that are fairly simple, written only by triggers, and easy to regenerate quickly if something does break horribly, I think they're not worth it.
<wgrant> If app code is touching them, agreed.
<stub> So are the fact tables going to reference libraryfilealias (for, say, patches) or person?
<wgrant> They reference Person, but not LFA.
<wgrant> Which is slightly interesting.
<stub> Will need some special well commented code in person merge then
<wgrant> But the triggers handle the person merge case fine.
<stub> Ok.
<stub> oh, of course.
<stub> is all fine
<wgrant> That array FKs thing is really nice, though.
<wgrant> Might actually make them practical for normal use.
<lifeless> poolie: hi
<lifeless> wgrant: small arrays only probably, don't want them TOASTed (to be better than a separate table.. perhaps)
<wgrant> lifeless: TOAST isn't too bad in some situations.
<lifeless> its fine as long as you're not using an ORM that requests every field always
<wgrant> Ahem
<lifeless> ok
<lifeless> so thats hyperbole
<lifeless> but not all that far off
<wgrant> (fortunately the ORM isn't going near my array table :))
<wgrant> Also, isn't the TOAST threshold like 8KiB?
<wgrant> That's one large array.
<lifeless> nah
<lifeless> that would be 2 pages
<poolie> lifeless, hi
<lifeless> 2K I think
<wgrant> Pages are 8KiB.
<poolie> does lp actually dislike docstrings on tests, or does it just tend to not have them?
<lifeless> bit of both
<lifeless> they aren't very useful, and the rules for docstrings are rather awkward for comments on many tests
<lifeless> also they tend to be redundant with well chosen test names
<StevenK> I prefer a comment at the top of a test, but I may forget to add one.
<lifeless> wgrant: I have no idea why I thought 4KiB
<wgrant> lifeless: Because every other computing definition of "page" is 4KiB.
<lifeless> yes
<lifeless> The TOAST code is triggered only when a row value to be stored in a table is wider than TOAST_TUPLE_THRESHOLD bytes (normally 2 kB). The TOAST code will compress and/or move
<lifeless> http://www.postgresql.org/docs/8.4/static/storage-toast.html
<wgrant> Ah
<wgrant> I concede.
<poolie> i know unittest used to have stupid behaviour where it would not print the test name if there was a docstring
<poolie> but i'm pretty sure that's fixed in lp's test runner
<wgrant> poolie: I think our testrunner still does.
<wgrant> Hmm
<wgrant> Maybe that was fixed.
<wgrant> Or I'm thinking of another project.
<wgrant> Oh goody.
<wgrant> code.launchpad.net is magically faster now.
<wgrant> And Bugs is slowly speeding up.
<wgrant> wtf launchpad why are you doing this
<wgrant> don't get better without us doing anything :/
 * wgrant breaks the world with HTML5.
<lifeless> stub: so wtf with bugtag.name
<lifeless> stub: how was 8.4 working at all ?
<stub> lifeless: I have no idea
<lifeless> stub: tests failed for you ?
<stub> lifeless: Either somehow that code path was never reached, or there is some odd scoping going on with 8.4 that allowed it to work (maybe another table with the same alias in the outer scope?)
<stub> Yes, picked up with a failed test run. Testing my 9.1 ec2 image.
<wgrant> That code path should have been reached, though :/
<stub> I'd seen it before, but thought I'd fixed it already.
<wgrant> stub: That code is fairly new
<stub> There are still a few other tests failing I want to fix later today.
<wgrant> And should really just be deleted.
<stub> salgado's email re: conjoined bugs confuses me, and leads me to fail to discover any actual definition of a conjoined bug.
<stub>     _NON_CONJOINED_STATUSES = (BugTaskStatus.WONTFIX, ) is probably what confuses me the most
<wgrant> stub: That's the bit of the definition that nobody remembers.
<wgrant> A bugtask conjoinment relationship has two halves: the master and the slave.
<wgrant> The master task is the development series task: a productseries, distroseries, or sourcepackage task.
<wgrant> The slave task is the non-series task: a product, distribution or distributionsourcepackage task.
<wgrant> Conjoinment implicitly occurs when there is a task for the development series.
<wgrant> The slave is no longer directly editable, instead being mirrored from the master.
<wgrant> *Except* when the master is marked Won't Fix, in which case the conjoinment relationship is broken, and the slave is separate again.
<wgrant> It's like somebody tried to devise the most obscure set of rules they could think of.
<stub> Yer. Your definition makes sense. I suspect it would have benefited from rethinking the data model around those rules at the time.
<wgrant> You'd think.
<lifeless> well
<lifeless> the whole problem was one of model leaking
<lifeless> nominations contribute to the pain too
<lifeless> skaet would love only-series tasks to be implemented :)
<wgrant> And most of the rest of the world would not :)
<lifeless> well
<lifeless> I don't think thats true
<lifeless> most of the rest of the world wouldn't ever notice </pedantic>
<adeuring> good morning
<czajkowski> aloha
<mabac> gmb, hi there! are you up for reviewing a fix for me? :) https://code.launchpad.net/~linaro-infrastructure/launchpad/notify-workitems-changes/+merge/98231
<mabac> I probably didn't mean gmb since it looks like he's not online. :)
<gmb> mabac: I'm mobile ATM. Also, it's
<gmb> Not
<gmb> My review day.
<mabac> gmb, sorry, I just thought I'd bug you since the topic listed you as OCR. :)
<gmb> mabac: Check http://dev.launchpad.net/ReviewerSchedule for details of who's on today.
<gmb> It's out of date. Apologies.
<mabac> gmb, thanks! I didn't know about that page
* gmb changed the topic of #launchpad-dev to: https:/â/âdev.launchpad.net/â | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<gmb> Doing that on my phone was hard :/
<czajkowski> :/
<czajkowski> gmb: ello ello
<mabac> gmb, :) thanks
<gmb> Hi czajkowski.
<czajkowski> gmb: fancy a UDS photo walk one of the evenings?
<gmb> czajkowski: Definitely!
<czajkowski> hop on the bart and walk around san fran take pics and grab foods?
<czajkowski> plan it in advance and get some numbers?
<czajkowski> and you cna plan a route :)
<czajkowski> know there are a lotta photo folks in ubuntu could be fun
<czajkowski> something different to do
<gmb> Haha. My plans often fail, hard. But it seems like a good principle. I'll put something together and we can refine it together; sound cool?
<czajkowski> am gonna update the uds wiki page
<czajkowski> will mail you in a bit
<gmb> Cool.
<czajkowski> and then get it posted to uds organisers :)
 * gmb afk to give someone a large amount of money
<czajkowski> if people know in advance then it wont clash with team dinners
<gmb> czajkowski: Right; makes perfect sense.
<salgado> stub, who's working on the private bugs overhaul you mentioned?
<stub> Curtis' team
<stub> sinzui: ^
<czajkowski> purple squad working their magic
<salgado> cool, thanks stub
<wgrant> salgado: It's not so much as an overhaul as a complete redesign and rewrite from scratch.
<wgrant> get_bug_privacy_filter is what you should be using, indeed.
<wgrant> We'll have to rewrite your query soon, though.
<salgado> wgrant, ok, cool.  but why rewriting my query is going to be needed?
<wgrant> salgado: Because there's a new, faster bug search schema coming soon, and privacy is completely different.
<salgado> oh, ok
<mabac> I'm waiting for abentley who seems to be ocr in the next time-slot. Anyone know when his day starts? Or even anyone who would like to review this for me? :) https://code.launchpad.net/~linaro-infrastructure/launchpad/notify-workitems-changes/+merge/98231
<rick_h> mabac: he should be in soon.
<rick_h> our stand up is in 30
<rick_h> so will have to catch him after that probably
<mabac> thanks rick_h
* abentley changed the topic of #launchpad-dev to: https:/â/âdev.launchpad.net/â | On call reviewer: abentley | Firefighting: - | Critical bugtasks: 4*10
<mabac> abentley, good to see you. I hope you're happy to learn that I'm hoping to add to your work load this early ;)
<mabac> abentley, https://code.launchpad.net/~linaro-infrastructure/launchpad/notify-workitems-changes/+merge/98231 is in need of a review
<abentley> mabac: On The Phone
<czajkowski> mabac: wow you really want this code reviewed!
<mabac> czajkowski, sorry if I'm being a pain. just trying to get this off my plate since I'll be offline the rest of the week. I'll shut up now ;)
<mabac> abentley, sure no rush. really. I'm sorry for jumping on you like that
<sinzui> jcsackett, I am indeed trying to get you branch merged.
<sinzui> jcsackett, I decided to PQM landed it after getting another false error in ec2 test
<jcsackett> sinzui: got it, so i should not attempt to reland on my end. :-P
<sinzui> There is now a text conflict though
<jcsackett> sinzui: if you have not pqm-landed it, i should be able to land on my end now.
<jcsackett> or if you're already dealing with it, i'm happy to leave it to you, if you would prefer.
<sinzui> jcsackett, Was fixing the conflict. let me try for 2 more minutes
<jcsackett> sinzui: sure.
<sinzui> jcsackett, I am going to run the pillar tests locally. You and Ian collided
<jcsackett> sinzui: ok.
<abentley> adeuring: http://pastebin.ubuntu.com/892304/
<adeuring> abentley: can't header you any more
<adeuring> abentley: mumble instead?
<abentley> adeuring: Sure.
<sinzui> jcsackett, your branch is merged.
<jcsackett> sinzui: awesome, thanks. :-)
<jcsackett> glad i sorted out what you were doing before lp-landing on my own. :p
<sinzui> jcsackett, the conflicts were nasty. merge is an idiot. I reviewed both your branch and Ian's so I just pasted your classes into the conflicting module instead of editing the spaghetti.
<jcsackett> sinzui: that seems very reasonable, and having just updated from devel it all looks right to me.
<abentley> mabac: Your functionality looks okay.
<abentley> mabac: You are adding LOC.  Do you have permission to do that?
<mabac> abentley, I hope this is considered offset by the tech debt payed off by salgado for the work items feature
<salgado> abentley, this is a bug fix, not a new feature, btw
<mabac> abentley, so my answer is, yes I believe so
<abentley> salgado: Is there a blanket exemption for bug fixes?  I didn't think there was.
<mabac> abentley, still this should have been part of the initial feature, just neglect on my part that it wasn't
<abentley> mabac: Okay.
<abentley> mabac: Some of the indentation doesn't match our style.  40-41 should be indented four spaces.  42-43 should be indented another four spaces.
<czajkowski> bac: if you get a chance would you mind looking over https://answers.launchpad.net/launchpad/+question/169663  please.
<salgado> abentley, maybe there isn't, but I remember someone saying that in the end it's at the reviewer's discretion to decide and requiring contributors to remove unused code before they can fix bugs sounds like a very good way to chase them off
<abentley> mabac: at 56, since the arguments don't all fit on one line, they should all be placed on the next line, indented four spaces.
<mabac> abentley, ok cool.
<abentley> salgado: No, there's no room for reviewers discretion.  waivers are granted by flacoste and lifeless.
<flacoste> abentley: yes, salgado's credit is still good on that one
<salgado> flacoste, that's cool, but do you think it make sense to require people to remove unused code in order for their bug fix to be accepted?
<abentley> salgado: No, there's no room for reviewers discretion.  waivers are granted by flacoste and lifeless.
<abentley> mabac: Also 138-142, 148-149, 170-171, 185-190, 191-196, 205-206, 210-213
<abentley> mabac: Is the lint clean?
<salgado> <flacoste> abentley: yes, salgado's credit is still good on that one
<salgado> <-- abentley has quit (Ping timeout: 264 seconds)
<mabac> abentley, apparently not. looks me relying on my emacs mode indentation and pep8 wasn't a good idea. can you tell me how I run lint to check the lp style?
<abentley> mabac: I don't know of a linter that check's Launchpad's style, but you should run "make lint" to ensure your code is lint-free.
<mabac> abentley, ok sorry I hadn't done that. I get a couple of errors but they are not caused by this particular change. they're still my doing so if I can sneak in fixes for those here that'd be great.
<abentley> mabac: Please do.
<abentley> adeuring: The problem still exists, but storing job.fail as a local variable avoids it.
<adeuring> abentley: ah, ok. So you were right that the problem is caused by late loading job.
<mabac> abentley, I have pushed changes, I hope it looks better now
<abentley> mabac: Thanks.  That's better.  At 57, 150, 172, 209, the line should be indented four spaces.
<bac> czajkowski: that's interesting.  it's also interesting that we've not seen a general problem with key import so i'm not sure what is happening.
<abentley> mabac: At 143, 192, 199, I believe the } is indented 4 spaces too far.  I.e. it should have the same indentation as the line containing the matching {.
<czajkowski> bac: aye I wonder is it something specifically wrong with that key and why they don't generate a new one rather than wait 6 months. but how and ever
<bac> czajkowski: yes, it is odd
<abentley> mabac: Did you investigate salgado's comment "However, if someone edits the workitems using the +workitems page, no notification will be sent because that code doesn't call updateStatus() via lazr.restful and so no event is fired. We might need to have that page itself fire an ObjectModifiedEvent..."
<abentley> ?
<bac> czajkowski: i do see his key is crazy old...1996
<mabac> abentley, yes. that's handled by change line 56
<abentley> mabac: great.
<czajkowski> bac: makes no sense we'd surely have others with the same issue
<mabac> abentley, I've pushed another revision with the indentation fixes
<abentley> mabac: I think your comment in tests, "In production this notification is fired by lazr.restful..." is now stale.
<abentley> mabac: Also, you should add a description of each test-- either a docstring or a comment.
<abentley> mabac: Do the tests still need transaction.commit?
<mabac> abentley, the comment was intended to explain why I fire a notification in the tests but not in the production code. if that's not useful I'll remove it
<mabac> abentley, they do. and I've followed an existing example where it's done exactly like that
<mabac> abentley, ack on the docstrings
<abentley> mabac: But "this notification is fired by lazr.restful" is true, but not complete, now that it's also fired in another location.
<mabac> abentley, true. I'll fix it
<abentley> mabac: r=me with those changes.
<mabac> abentley, thank you very much!
<abentley> mabac: np.  Sorry for the delay.
<mabac> abentley, that was very quick so I'm very happy :)
<abentley> mabac: line 57 still looks wrong.
<mabac> abentley, sorry about that, I had forgotten to save that file
<mabac> abentley, fixed and docstrings added
<abentley> mabac: great.  Do you need me to land this for you?
<mabac> abentley, if you would. thanks!
<mabac> abentley, that is, yes please :)
<abentley> mabac: okay, will do.
<mabac> abentley, thank you
<sinzui> jcsackett, I sent you an image to add to your mockup directoy
<sinzui> jcsackett, no change from your work. We are adding a picture based on wallyworld's discovery that the suggested design overwrites permissions or fails to inform about existing permissions
<jcsackett> sinzui: rereading your email now, sorry. :-P
<salgado> danilos, mabac, the call is in 20 minutes.  are you guys joining?
<jcsackett> sinzui: pushing it up to the directory now.
<sinzui> thanks.
<danilos> salgado, I am, mabac won't be able to
<danilos> mrevell, salgado: hi, I suppose mumble or hangout will do?
<mrevell> salgado, flacoste, danilos: Mumble is fine.
<mrevell> salgado, flacoste: We're in the Product channel on Mumble
<danilos> flacoste, joining us?
<mrevell> danilos, salgado: https://dev.launchpad.net/Projects/WorkItems/
<flacoste> mrevell: sorry, i had a dr appointment, i forgot to tell you
<mrevell> flacoste, Ah no problem. I'll send notes for the checkpoint later. The actions are for Linaro to produce the milestone team view and test internally, with danhg doing some testing with a broader group of users. danilos also had a question about how we handle groups of work items within an indiviual blueprint, which he posted to the -dev list.
<flacoste> ok
<czajkowski> gmb: https://wiki.ubuntu.com/UDS-Q/OtherEvents
<gmb> czajkowski: Ah, thankee. I've already turned up one or two potential routes. Really depends what we're after (landmarks vs interestingness).
<czajkowski> gmb: factor in a food stop
<gmb> czajkowski: Right. And also travel time from t'hotel, which, this being the BART, could be substantial.
<gmb> Anyroadup, I'm sure we'll come up with something.
<czajkowski> gmb: 30 mins I think
<gmb> czajkowski: Yeah, it's not so much the travel time as the waiting-for-a-train time.
<czajkowski> nods
<lifeless> flacoste: hi
<flacoste> hi lifeless
<lifeless> flacoste: did you see skaets in-line request for escalation on the audit trail bug ?
<flacoste> yes
<lifeless> salgado: hi, when is your EOD ?
<lifeless> flacoste: cool
<flacoste> lifeless: i replied to her via irc, but i think i'll need to follow-up more
<flacoste> lifeless: basically, i don't think we have time to do it before the release
<flacoste> so not sure it's worth escalating at this point
<lifeless> yeah, I had a private chat too
<lifeless> the bug wasn't on our management radar
<salgado> lifeless, in about 20 minutes, why?
<lifeless> salgado: I thought I might try and help you with this bug search thing
<lifeless> salgado: if you would like to talk about it, irc or voice or whatever
<salgado> lifeless, sure, mumble, hangout or something else?
<lifeless> skype or hangout work best for me
<lifeless> mumbles lack of echo cancellation -> pain
<lifeless> flacoste: if you need anything from me (that I'm not already doing :P) on that bug, let me know.
<flacoste> i will
<salgado> lifeless, agreed. starting a hangout now.
<lifeless> wow, unity's alt-tab is reallly hard to use
<lifeless> just breaking me all the time atm
<salgado> lifeless, http://paste.ubuntu.com/894246/
<salgado> lifeless, https://dev.launchpad.net/Projects/WorkItems
<poolie> has anyone else seen an error about
<poolie> 2012-03-21 06:14:39 UTC FATAL:  could not access private key file "server.key": Permission denied
<poolie> running current lp?
<wgrant> poolie: No. Possibly an SSH server key?
<poolie> google suggests something about an ssl key
<poolie> which would make more sense for postgres
<poolie> (i should have mentioned that's where it is)
<poolie> probably something local
<wgrant> Oh, if that's postgres, then yes, it's probably postgres :)
<sinzui> wallyworld, Surely there is something in this list that irks you: https://bugs.launchpad.net/launchpad/+bugs?field.tag=trivial
<nigelb> Morning.
<poolie> if i grant a team access to a pppa, will they all get spam^Wmailed?
<poolie> o/ sinzui
<poolie> o/ nigelb
<lifeless> yes
<nigelb> Hey poolie! Got a new motorbike yet? :)
<poolie> not yet, but i did get the payout for the old one
<poolie> i'm not going to get the chance to ride much for the next while so i might wait
<sinzui> hi poolie
<nigelb> Oh, nice.
 * huwshimi goes digging to find out who broke the homepage
<poolie> lifeless, that 'yes' was to me?
<poolie> :/
<sinzui> poolie, do you think the emails should go to the team admins? or none at all?
<poolie> not sure
<poolie> maybe it would be nice to have a tick box for "mail people to tell them they have access", like google docs does for instance
<poolie> or, just silently give them access
<poolie> presuming they'll find out some other way
<wgrant> huwshimi: You mean the header being wider than the rest of the page?
<poolie> it doesn't seem like a thing where there's a strong reason they _must_ be told
<huwshimi> wgrant: Yeah...
<wgrant> huwshimi: sinzui removed locationless a while ago.
<wgrant> That's what changed it.
<wgrant> revno: 14867 [merge]
<wgrant> timestamp: Sat 2012-02-25 07:26:36 +0000
<huwshimi> wgrant: Oh, so this has been live for a while!?
<wgrant> message:
<wgrant>   [r=bac][bug=432889,435832,580240,938889] Update error pages to look
<wgrant>         and read like other Lp pages; locationless is gone.
<sinzui> poolie, there is a similar bug request for mailing lists. It would be nice if Lp could show you new things on your pages that you got access to when you arrive
<poolie> indeed, i guess that connects to a more generally nuanced view of notification
<poolie> so "tell them now" would perhaps
<poolie> mean
<poolie> "it's important they know now", which would push it up to being sent by mail
<poolie> or, maybe there should just be a per-user "tell me when i get a new ppa" flag
<poolie> s/tell/mail
<poolie> anyhow, generic notifications ++
<sinzui> I pioneered a mechanism to show notifications from non-browser code last week. It is possible to store them then unqueue them when the user arrives
<wgrant> StevenK: Should 933759 be in QA-Landing?
<lifeless> poolie: yes
<StevenK> wgrant: So I'd like to add information_type to branch's model code and then use that branch to close the bug.
<wgrant> StevenK: Mmm.
<wgrant> StevenK: It's not useful yet, and won't be for a while.
<StevenK> wgrant: Or we can just close it saying the column exists and other bugs deal with switching to it.
<wgrant> Yes.
<wgrant> That was my intent.
<StevenK> Doit
<wallyworld> wgrant: with the missing sorting in getPillarShareeData, why wouldn't I just add .order_by("person_sort_key(Person.displayname, Person.name)") to the findGranteePermissionsByPolicy() query?
<wallyworld> you mentioned not to do that but i think it's the way to go
<wallyworld> that's effectively how it was before i split the query
<wgrant> wallyworld: You shouldn't be loading people in that query.
<wgrant> Given how slow loading people is.
<wgrant> I guess you could do it, given that it's batched now.
<wallyworld> yes i think so
<wallyworld> it's only a few records eg 75
<wallyworld> we can optimise later if really needed
<wgrant> That's still around 40ms :/
<wallyworld> that's negligable
<lifeless> mm
<lifeless> I think we need to recalibrate our speed expectations :)
<lifeless> 40ms is actually quite slow
<wgrant> Yes.
<wgrant> That's nearly 10% of the average render time of Product:+index
<wgrant> Which is still slow.
<wallyworld> in the context of a round trip it's ok surely
<wallyworld> and the rest of lp loads people in queries as needed
<wgrant> Sure, but the rest of LP is fucking terrible.
<wgrant> This stuff doesn't have to be :)
<wallyworld> and we're talking max batch_size records
<wallyworld> let's just get sorting fixed first and then optimise if needed
<wallyworld> i sure wouldn't notice if a page rendered 40ms slower
<wgrant> It's a 40ms delay for a non-significant query.
<wgrant> On its own it's not terrible.
<wallyworld> so? in the content of the overall page render time, it doesn't matter
<wgrant> But if there are 3 others like it, that's more than 100ms.
<wallyworld> and without the missing order_by, we are still loading people
<wallyworld> so adding the order_by now is the right thing to do
<wallyworld> to fix the core issue
<wgrant> OK, didn't notice that.
<wgrant> But still, 40ms is not negligible.
<wgrant> We should be getting the page's core query down that low if at all possible.
<wallyworld> if a page takes say 1s round trip time to render it is
<wgrant> And no other query on the page has any business being more than 10ms.
<wgrant> wallyworld: In Australia things will be slow, sure.
<wgrant> But Australia is negligible.
<wgrant> Western US -> UK latency is ~130mx
<wgrant> ms
<wgrant> 40ms is a third of that.
<wallyworld> i'm not arguing against optimisation. just the right time to do it
<wgrant> That's what LP said for the first 6 years.
<wgrant> Look where we are now :)
<wallyworld> sigh. the page is already faster than most others. i'm more concerned with getting the right data displayed initislly
<wallyworld> then we can optimise
<wallyworld> i'd also rather initially follow established lp patterns
<wallyworld> and see if they are good enough
<wallyworld> before diverging
<wgrant> It depends on your definition of "good enough".
<wgrant> I argue that our traditional definition of good enough is not good enough.
<wallyworld> if i can open my browser and load a page in < 1s that's good ebough for me
<wallyworld> and worrying about 40ms at the expense of code divergence etc is not worth it
<wallyworld> not saying we shouldn't fix that 40ms
<wallyworld> but priorities...
<wgrant> We can very easily avoid that 40ms now with about two lines of code.
<wallyworld> it's more than that since we would not be loading whole Persons anymore
<wallyworld> so the model query needs to change as does the code to assemble the data for the view
<wgrant> Oh, I wasn't saying to avoid loading the persons at all.
<wgrant> That would indeed be a bit radical for such a small improvement right now.
<wgrant> But you're currently loading them twice.
<wgrant> getPillarShareeData takes a list of person, then queries them back out of the DB.
<wgrant> When it really just needs the IDs to match against the provided list.
<wallyworld> that method can be called standalone
<wallyworld> so it needs to be defined consistently with the getPillarSharees method
<wgrant> Ah, true, in this design.
<wgrant> I had envisaged that these would be a single method, using something like DecoratedResultSet to add the permissions into the sliced ResultSet.
<wgrant> Which would be roughly the same interface as the original, just more efficient and sliceable.
<bigjools> morning people
<wallyworld> but we would lose the ability for users to have available that 2nd method if they wanted to use it
<nigelb> ohai bigjools
<wallyworld> so we have done a lot of arguing over a drop in the ocean in terms of total time a user sees a page to render
<wgrant> wallyworld: But four drops in the ocean add more than 150ms to the page.
<wgrant> wallyworld: Which is more than the RTT for the vast majority of our users.
<wallyworld> why 4?
<wgrant> It's just the number of 40ms-overhead operations that add up to >RTT.
<wallyworld> but it's only one. and there'a lot more to rendering a page than RTT
<wgrant> Not this page.
<wallyworld> mustache on the client takes a bit of time
<wgrant> That's a bug that can't be trivially fixed right now.
<wallyworld> sure, so 40ms just doesn't matter right now
<wallyworld> since other things swamp it
<wgrant> But then we replace mustache with handlebars globally.
<wgrant> And this page is still slow.
<wgrant> Because the server-side is crap.
<wallyworld> not slow
<wgrant> And now we have to go and fix *every page in Launchpad* separately.
<wgrant> Because we knowingly wrote slow code.
<wgrant> Because it was dominated by slow client-side rendering.
<wallyworld> we have different definitions of slow
#launchpad-dev 2012-03-22
<StevenK> 13 cards in QA-Landing, because buildbot sucks and needs to die.
* StevenK changed the topic of #launchpad-dev to: https:/â/âdev.launchpad.net/â | On call reviewer: -ey | Firefighting: - | Critical bugtasks: 4*10
* StevenK changed the topic of #launchpad-dev to: https:/â/âdev.launchpad.net/â | On call reviewer-ey | Firefighting: - | Critical bugtasks: 4*10
 * StevenK stabs irssi
<rick_h> StevenK: ping, it came up in a call that you and Steven hit a bug where codehosting was giving bad responses during log rotation?
<rick_h> StevenK: did that end up in a bug somewhere that I can reference?
<StevenK> That I and me?
<StevenK> :-)
<rick_h> :)
<rick_h> just going off of what I'm told from the higher power, Steven and Steve K were involved somehow
<StevenK> It was investigated by poolie, wgrant and I.
<StevenK> Only one of us is called 'Steve'
<rick_h> ok, well then I was lied to, my apologies
<wgrant> https://bugs.launchpad.net/bugs/724750
<_mup_> Bug #724750: launchpad redirects pack files to https://launchpad.net (while repacking?) <Launchpad itself:Triaged> < https://launchpad.net/bugs/724750 >
<StevenK> That's the one.
<rick_h> ty sir
<StevenK> It doesn't impact buildbot since that uses bzr+ssh.
<wgrant> rick_h: BTW, thanks for fixing the YUI2 thing.
<rick_h> wgrant: sorry it got hung up with my travels and such
<rick_h> had hoped to get it landed before I left, but various tests hated me
<wgrant> Yeah, that often happens :(
<wgrant> Now maybe I can implement Content-Security-Policy this weekend.
<StevenK> rick_h: You've recovered from PyCon?
<wgrant> Still, 30 minute test suite soon :)
<rick_h> StevenK: working on it, have the post pycon flu so on meds and need a nap something fierce
<rick_h> wgrant: yea, heard there was some serious progresson that, exciting!
<StevenK> wgrant: So should I replace bug.private with bug.information_type in (2, 4, 5) in the triggers?
<wgrant> StevenK: (3, 4, 5), but yes.
<wgrant> StevenK: That's what I've done, although it is a bit awful.
<StevenK> Yup
<StevenK> wgrant: The DB branch that makes information_type not null failed due to the garbo job. :-(
<wgrant> Heh.
<StevenK> The garbo job has just been killed in devel
<wgrant> So I aw.
<wgrant> s
<StevenK> Now to wait for buildbot so I can re-land the db patch
<wgrant> That's about 11 hours away.
<StevenK> I'm aware. :-(
<StevenK> at now + 11 hours bzr lp-land
<StevenK> That has no hope of working, but it would be nice.
 * StevenK claims another -12 patch
<StevenK> wgrant, wallyworld: I think we should suggest to sinzui that we need a new lane: 'QA-Fucked over by buildbot'
<wallyworld> +1 :-)
<StevenK> Haha
<wgrant> Yeah
<wgrant> I'd really like there to be a separate column.
<wgrant> For when it is actually landed.
<wgrant> Since once it's landed it can usually be dealt with by anyone.
<StevenK> To be fair, QA-Landing is conflating ec2 with buildbot.
<StevenK> Perhaps we need two: "This is working through ec2, and isn't landed yet." and "This has hit devel/db-devel and can be dealt with by anyone when it hits qas."
<wgrant> That's what I meant, yes.
<StevenK> Right.
<StevenK> wgrant: CREATE FUNCTION changes to DROP/CREATE or CREATE OR REPLACE ?
<wgrant> StevenK: CREATE OR REPLACE
<wgrant> DROP will probably not work because there are references.
<wgrant> In general there's very little reason not to use CREATE OR REPLACE every time.
<wgrant> wallyworld: Hi, did you see my comment on your MP about half an hour ago?
<wallyworld> wgrant: no. but i already fixed the typo
<wallyworld> is person_sort_key case insensitive?
<wgrant> It is.
<wallyworld> ffs
<wallyworld> why would it do that?
<wgrant> Ah, hm, I see you did indeed fix the typo a few minutes ago, but I don't have an email about it :/
<wgrant> Because case insensitive sorting is what most of the world expects.
<wallyworld> luckily for the next landing it's not really an issue
<wallyworld> regardless of what the world expects, it should match the case sensitivity of normal order bys
<wgrant> It's also rather trivial to fix without rerunning tests.
<wallyworld> yes
<wgrant> Hm?
<wgrant> That *is* the normal order by.
<wgrant> Ordering by Person.displayname is not supported.
<wallyworld> but Person.displayname, Person.name is
<wgrant> Its sort order is not a recognised one.
<wgrant> No
<wgrant> It's just a rule we have to break here until StormRangeFactory is fixed.
<wgrant> person_sort_key is, as the name suggests, the person sort key.
<wgrant> (Person.displayname, Person.name) is not, but it's close enough for the initial branch.
<wallyworld> there should be no rule against sorting on multiple columns or am i missing something?
<wgrant> It's not a rule against it.
<wgrant> The rule is that the natural ordering for persons is person_sort_key.
<wgrant> Which is roughly lower(displayname) + ', ' + lower(name)
<wallyworld> sure, ok
<wgrant> Having slightly different orderings around the place is rather confusing.
<wgrant> (Person.displayname, Person.name) will do for now, but the ordering for the batch slicing and intra-batch sorting must be identical, or it will be very confusing.
<StevenK> 18degC? What the heck. This isn't winter.
<wgrant> It got down to 11 here yesterday.
<StevenK> Bloody heck
<wgrant> http://www.baywx.com/melbtempyestII.html
<wgrant> Veeeery odd day.
<mwhudson> isn't that graph basically the wrong way up?
<wgrant> Yes.
<bigjools> bloody freezing here too, it's 22C
<wallyworld> wgrant: agreed about sorting needing to be consistent - i thought it was, didn't realise person_sort_key was case insensitive
<wgrant> wallyworld: Yep
<wgrant> bigjools: Yes, but you're in the past.
<wallyworld> wgrant: StormRangeFactory doesn't accept lower either
<wallyworld> so we have to live with it till StormRangeFactory is enhanced
<wgrant> wallyworld: Sure, the only way to get it consistent now is to use plain (displayname, name)
<wallyworld> which will mess up if there's names with non ascii chars
<wgrant> wallyworld: We don't really handle that case very well normally.
<wallyworld> i'll tweak the sort order, raise a bug for StormRangeFactory, and put in an xxx
<wgrant> Thanks.
<wgrant> I'll probably end up fixing SRF soonish.
<wgrant> Since we likely want it for bugs.
<wallyworld> wgrant:  bug 961836
<_mup_> Bug #961836: StormRangeFactory should support functional sort keys <batching> <Launchpad itself:New> < https://launchpad.net/bugs/961836 >
<wgrant> wallyworld: Bug #961840
<_mup_> Bug #961840: "Share with someone" makes it easy to accidentally revoke access <disclosure> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/961840 >
<wallyworld> wgrant: currently "by design" since there's no vocab to support that sort of thing yet
<wgrant> wallyworld: k
<wgrant> wallyworld: Is it easy enough to use the normal batch spinner on initial page load?
<wallyworld> but good that's it's filed so we can track it
<wgrant> Currently it uses a different one for initial vs batch load.
<wallyworld> no since the batcher expects trhe initial batch to be rendered
<wallyworld> server side
<wgrant> Yeah, it would obviously need some changes, but might be nice if it's simpleish.
<wallyworld> i guess the client side code could be changed
<wgrant> Apart from that, this UI is pretty damn nice.
<wallyworld> and faster than other things
<wgrant> lifeless: I'm seeing overhead of about 10% on +sharing from traceback collection from just 27 queries.
<lifeless> wgrant: what are the times
<wgrant> Sigh, libvte upgrade has broken termcaps
<wgrant> lifeless: Rendering without profiling is about 0.20s vs 0.22s. With profiling is 0.42s vs 0.48s, with about 0.05s shown in format_stack
<wgrant> But that's 2ms per query, which is sort of insane unless it's doing IO.
<wgrant> Ah, it is.
<wgrant> linecache.checkcache
<wgrant> I wonder how much of it is that.
<StevenK> wgrant: Can you link me to your branch that changes the triggers?
<lifeless> wgrant: so, I'm willing to pay 2ms on a 20ms call to have good data on the bad calls
<lifeless> wgrant: if you can come up with a sensible way to achieve the good data on bad calls without the 2ms on the 20ms call, I'd be delighted
<wgrant> lifeless: That adds 15ms to every authentication attempt :)
<lifeless> one thing we could do is get the line nos but not the line content
<wgrant> Yeah
<lifeless> and if we oops only then convert to lines
<wgrant> Exactly.
<wgrant> Since hopefully the file won't change much...
<lifeless> if it does we have a different problem
<wgrant> But I don't know how much is actually the formatting.
<lifeless> so our auth calls are 150ms?
<wgrant> We do have some pretty horrific traces due to TAL.
<lifeless> with 15ms in data gathering overhead? Still doesn't bother me :)
<wgrant> Request startup and auth is 50-60ms on production.
<lifeless> sorry, 165ms w/15ms in data gathering
<lifeless> wgrant: are you talking about the validation of credentials per request
<wgrant> Yes.
<lifeless> ok
<lifeless> so look, I'm totally happy with tweaks to make it better
<lifeless> I'm glad you're working on sufficiently lean stuff that this is showing up
<wgrant> It's not big, but it's not exactly insignificant as you initially stated :)
<StevenK> I think we're the only team that is following the directive that new pages need to be <1s
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/legacy-access-type/+merge/98342
<wgrant> StevenK: I think it's <5s
<wgrant> <1s is lifeless' eventually 99% goal, IIRC
<lifeless> the directive is for new pages to be <1 99% of the time
<wgrant> Ah
<lifeless> new stuff should be where we want to be
<wgrant> Oh right
<wgrant> Timeout of 5s.
<lifeless> right
<lifeless> could you please add feature rules to ensure that as you add page ids? that would be awesome.
<wgrant> The batching changes should bring this comfortably to 99% <0.5s, except for Ubuntu where who knows.
<lifeless> it will be brilliant to have front end problems
<wgrant> Indeed.
<wgrant> Would love a flaggable soft timeout...
<lifeless> doit
<wgrant> I tried.
<wgrant> Requires reworking lots of things.
<lifeless> oh yeah end of the world sort of thing
<lifeless> you could cheat
<wgrant> And it's probably worth taking it that little bit further to make UserRequestOops work in the same way.
<wgrant> At present ++oops++ doesn't work for requests that don't render the base template.
<lifeless> oh my, checkcache isn't written for speed is it.
<wgrant> Nooooo it is not.
<lifeless> so
<lifeless> you have my total blessing to monkey patch that to return.
<lifeless> we /don't/ support reload() in lp.
<lifeless> we could, in principle, use inotify but I don't see the practical value
<spm> maybe if you changed it to be iNotify, then the value becomes obvious. ???
<wgrant> Heh
<lifeless> spm: keep your day job :)
<wgrant> 828 calls to checkcache from 27 queries
<spm> :'-(
<wgrant> And I guess each of those causes at least 1 stat
<wgrant> And those are all really shallow queries.
<lifeless> whats calling checkcache ?
<lifeless> getlines?
<wgrant> _get_frame_data
<wgrant>     # End part adapted from zope.exceptions.
<wgrant>     co = f.f_code
<wgrant>     filename = co.co_filename
<wgrant>     name = co.co_name
<wgrant>     linecache.checkcache(filename)
<wgrant>     line = linecache.getline(filename, lineno, f.f_globals)
<wgrant> Seems like at worst we'd want a clearcache at the start of the request.
<lifeless> its not threadsafe
<lifeless> or rather
<lifeless> I'll take money that there are check-and-then-reference patterns in the code
<wgrant> True.
<lifeless> which will blow up if the cache clears between two bytecodes in the model
<lifeless> module
<lifeless> like
<lifeless> ok
<lifeless>     if filename in cache:
<lifeless>         return cache[filename][2]
<lifeless>     else:
<lifeless>         return updatecache(filename, module_globals)
<lifeless> (in getlines, called from getline)
<wgrant> :D
<lifeless> now, if you want to rewrite the whole module more-or-less, sure.
<lifeless> the right place to check the cache is import
<lifeless> 'if you replace this module please be so kind as to discard its cached entries'
<lifeless> ditto tal, if tal uses the linecache to cache lines for tracebacks
<lifeless> and really, module -> lines should be a module held reference so that reload(),w hich leavces the old module to be gc'd, doesn't cause bad stack traces on old call stacks.
 * lifeless stops before he gets critical 
<StevenK> Before?
<wgrant> Mysterious.
 * StevenK stabs MPs
<wgrant> Nothing is calling checkcache
<wgrant> But it's still being called.
<mwhudson> zope is good at hiding behaviour
<wgrant> Oh
<wgrant> extract_tb does it
<wgrant> In stdlib's traceback module
<wgrant> Why were we calling it directly too :/
<StevenK> To be certain.
<StevenK> Or something.
<lifeless> thats probably a bug
<lifeless> the intent was to have the oops core get tb's
<lifeless> but the lp consumer opt in for the zope pretties
* StevenK changed the topic of #launchpad-dev to: https:/â/âdev.launchpad.net/â | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<wgrant> lifeless: A custom traceback collector without touching the files at all is 10x faster.
<wgrant> So <1% overhead.
<wgrant> Means we need to have a somewhat more hacked up traceback system, but it seems like it's probably worth it to do the formatting only on demand.
<StevenK> wgrant: I still get failures in TestBugMirrorAccessTriggers with your DB patch applied. :-(
<lifeless> wgrant: cool, if you're going to execute on this wicked, if not can you file bugs to capture the issue and your digging ?
<wgrant> StevenK: Do you also have my test fixes merged?
<StevenK> Oh, there's test fixes for TestBugMirrorAccessTriggers too?
<wgrant> yes
<wgrant> Best to merge the whole branch, probably.
<StevenK> I only grabbed the DB patch
<StevenK> And indeed, that was the only change in the branch you linked.
<wgrant> Oh, yes, different branch.
<wgrant> I was thinking of bugtaskflat.
<wgrant> Which had test changes.
<wgrant> What's the erorr?
<StevenK> wgrant: http://pastebin.ubuntu.com/894657/
<wgrant> StevenK: Somehow the bug is acquiring an extra subscription.
<wgrant> In every case.
<StevenK> I wonder if that points to a bug in transistionToInformationType()
<wgrant> It may be implicitly subscribing the actor.
<wgrant> So if the actor's not the usual person, bad things happen.
<wgrant> Where "bad things happen" == "making this pedantic test fail"
<wgrant> But it's possibly good that it picked it up.
<StevenK> wgrant: Figured it out.
<StevenK> wgrant: I had changed createBug() to unilaterly subscribe the bug supervisor if the information_type was USERDATA instead of 'params.product and params.product.private_bugs' before I changed it.
<wgrant> StevenK: Ah
<wgrant> That would break apport.
<wgrant> No other tests failed?
<wgrant> s/break/leak/
<StevenK> I was only running TestBug as a smoke test
<StevenK> Next up is -vvm bugs
<wgrant> lifeless: Bug #961875, and qastaging agrees with my numbers. 10% for the average well-optimised sub-second page.
<_mup_> Bug #961875: Timeline traceback collection has significant overhead <performance> <Launchpad itself:Triaged> < https://launchpad.net/bugs/961875 >
<wgrant> More like 5% for eg. +bugs
<wgrant> StevenK: So, my tests are happy now?
<StevenK> wgrant: Yeah, that fixed them. If I was doing -vvm bugs as my smoketest, I suspect I would have seen other fallout.
<StevenK> But -vvm bugs takes an hour
<wgrant> Yeah.
<StevenK> wgrant: TestBug == Total: 874 tests, 0 failures, 0 errors in 12 minutes 6.871 seconds.
<wgrant> Excellent.
<StevenK> wgrant: Running through -vvm bugs now, already at 27 failures
<wgrant> Heh
<lifeless> wgrant: which is ok :)
<wgrant> Hmm
<wgrant> We'll probably hit r15000 tomorrow.
<wgrant> wallyworld: Hum
<wgrant> wallyworld: The batching doesn't actually batch.
<wgrant> Ah, nevermind, the real batching rev isn't on qastaging yet.
<wgrant> Just the UI half.
<wgrant> 600ms in _load_object :/
<wallyworld> only a few more hours to go till it gets to qas
<wgrant> wallyworld: Still, most of the QA can be done now.
 * wgrant QAs his stuff.
<wallyworld> wgrant: which pillar were you looking at for the 600ms
<wgrant> wallyworld: launchpad
<wgrant> ++profile++show is handy
<wgrant> https://qastaging.launchpad.net/launchpad/+sharing/++profile++show
<wallyworld> sure, i've been using it a lot this week
<wgrant> Actually, looks like 600ms was exception
<wgrant> 450ms is more common.
<wgrant> But asuka seems to be rather unstable :(
<adeuring> good morning
<jml> :(
<jml> hello launchpad do you know what is a good thing libraries that do not assume that a user is only running one program that uses that library at a time that is a good thing.
<jml> hello launchpad do you know what is a good thing libraries that do not assume that a user is only running one program that uses that library at a time that is a good thing.
<jml> sorry for the repeat
<wgrant> jml: Just disable the cache, I guess :/
<wgrant> You can also pass a custom cache dir to the Launchpad constructor
<czajkowski> jtv: good morning/evening
<jtv> hi czajkowski
<jtv> It's not the one I already commented on, is it?
<czajkowski> nope
<czajkowski> just reading that
<czajkowski> thanks for looking at it
<czajkowski> jtv: https://answers.launchpad.net/launchpad/+question/191340
<czajkowski> jtv: I am getting better at figuring the translations out, just I do seem to find unusal ones I need your help still
<jtv> I'm a bit busy with a deadline, a review for rvba, and some SQL magic for dpm, and having network trouble so expect me to be a bit sporadic today.
<jtv> The question page, for instance, isn't loading for me.  This may take some time.
<czajkowski> jtv: not urgent sorry wil try and find out
<jtv> czajkowski: I just answered.
<stub> lol. "When complete", "A moment ago (estimated)"
<czajkowski> jtv: thanks
<jtv> czajkowski: if you do find time to look into this further, it's scripts/rosetta/remove-translations-by.py
<jtv> It _may_ be possible to select something like âonly by the given user.â
<czajkowski> nods
<jtv> czajkowski: (Yes, I know, I wrote that script.  And come to think of it yes, you should be able to select the submitter.  But the script also has some safety catches against mass deletions that you wouldn't want to do accidentally.)
<jml> wgrant: yeah, I passed a custom cache dir, but then I accidentally filled my (EC2 instance's) disk.
<jml> wgrant: I'm tempted to just fix lazr.restfulclient.
<wgrant> jml: Brave man.
<jml> well, let's see if I can actually run the tests.
<wgrant> I tried to do that once.
<jml> :(
<jml> If core developers can't patch it, then it's literally unmaintainable.
<wgrant> This was years ago. Not sure how it is now.
<jml> Then that just makes it literally unmaintained. :P
<wgrant> That is entirely uncontested.
<czajkowski> danilos: ping
<jml> heh heh
<jml> so, a couple of weeks ago maybe, httplib2 decided that no one should be allowed to use 0.6.0 any more
<jml> I'm not sure why buildout is looking on the internet for this rather than in ./download-cache/dist
<wgrant> Oh
<wgrant> Does that mean that they admitted the certificate thing was actually a problem?
<jml> I honestly don't know.
<jml> Why isn't buildout fetching files from download-cache...
<jml> ah
<jml> it needs a setting in buildout.cfg that lazr.restfulclient doesn't have set.
<jml> zope.testing. there's no school like the old school.
* rick_h changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h* | Firefighting: - | Critical bugtasks: 4*10
<mhall119> did something change with launchpadlib recenty?  I'm getting unicode strings for datetimes now
<StevenK> rick_h: O hai. Can you please prod deryck to do his QA when you do that whole stand-up thing with him?
<rick_h> StevenK: email pls :)
<rick_h> mhall119: yes, there was an update to wadllib a bit back, I just ok'd a fix this morning
<StevenK> rick_h: Why?
<rick_h> StevenK: because then it's on his todo list and not something I can forget to bug him about during standup
<mhall119> rick_h: thanks, so it should be fixed in Precise soon?
<rick_h> mhall119: yes
<StevenK> I'd bug him directly, but he isn't on IRC.
<mhall119> rick_h: is there a bug # I can subscribe to about this?
<deryck> Morning, all.
<StevenK> deryck: O hai. Would you mind doing the QA for r14983? :-)
<deryck> StevenK, sure, first thing on my list today.
<sinzui> czajkowski, Can you change the definition and implementation status https://blueprints.qastaging.launchpad.net/gdp/+spec/gdplaunchpad
<deryck> woo-hoo, we now have the same number of queries issued in both old bug listings and new.
<czajkowski> sinzui: nope
<abentley> deryck: Woo!
<czajkowski> sinzui: I checked all my old blueprints before I created the bug
<czajkowski> as that's kinda annoying not to be able to change the status give this time of cycle
<sinzui> czajkowski, no yellow edit icons next to the two fields>
<czajkowski> sinzui: nope
<czajkowski> there used to be :)
<sinzui> czajkowski, the model says you can
 * sinzui looks for old form
<czajkowski> I can edit subscription
<czajkowski> thats it
<sinzui> czajkowski, can you access and use https://blueprints.qastaging.launchpad.net/gdp/+spec/gdplaunchpad/+status
<czajkowski> sinzui: yes
<czajkowski> sinzui: how did you get there?
<sinzui> anyone can change subscription. I think the permissions on the links are different from the real fields
<sinzui> czajkowski, I help my pointer over the edit Icon I could see to learn the non-javascript location
<czajkowski> sinzui: only edit filed I cna get to is https://blueprints.qastaging.launchpad.net/gdp/+spec/gdplaunchpad/+subscribe
<sinzui> czajkowski, this is a regression. It may be related to the javascript choice-picker, but I think the real issue is that the code that decides to render them thinks you have to be a driver
<sinzui> czajkowski, The code really did you to be a driver two years ago, that was fixed, but I think the links were never updated
<czajkowski> hmm
<sinzui> czajkowski, this broke 12 months ago when the blueprints switched to the choice picker for everything
<czajkowski> sinzui: nods well at this time of cycle it's probably going to be used a lot
<sinzui> czajkowski, do you see the assignee edit icon next to your name?
<czajkowski> sinzui: I do now on https://blueprints.qastaging.launchpad.net/gdp/+spec/gdplaunchpad
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett, rick_h* | Firefighting: - | Critical bugtasks: 4*10
 * jcsackett almost forgot it was thursday.
<sinzui> czajkowski, do you manage a project on qastaging that would could create a blueprint for and make me the assignee?
 * sinzui wrote a test that shows the assignee gets the edit link
<czajkowski> I dont manage any project
<sinzui> czajkowski, https://qastaging.launchpad.net/projects/+new register a project, configure blueprints, create one, make me the assignee
<sinzui> czajkowski, This widget we are looking at happen to be the only widget that does not work for IE...maybe the problem is bigger than we thought
<czajkowski> ok
<sinzui> PS I care because we are using this widget to replace the privacy/security checkboxes
<czajkowski> ok created now to configure blueprint
<czajkowski> sinzui: dont see anything to configure bp
<sinzui> czajkowski, its on the right side of the project page, and the link is repeated on the blueprint page
<czajkowski> sinzui: otp
<sinzui> ack
<abentley> adeuring: You wanted me to review https://code.launchpad.net/~adeuring/lazr.jobrunner/use_job_repr_in_logging/+merge/98821 ?
<adeuring> abentley: right
<adeuring> abentley: and https://code.launchpad.net/~adeuring/launchpad/lp-lazr.jobrunner/+merge/97458
<abentley> adeuring: It looks a bit strange.  Most reprs either resemble instance construction, like "BranchScanJob(id=5)" or have angle brackets like "<BranchScanJob>".
<czajkowski> sinzui: https://blueprints.qastaging.launchpad.net/conference-planning/+spec/test
<adeuring> abentley: well, we can use that too; I just wanted minimal changes that don't require related changes in the tests ;)
<abentley> adeuring: Just "BranchScanJob" doesn't look like a repr to me.
<sinzui> czajkowski, :( it looks right. I see the definition and implementation edit icons
<sinzui> and I just changed one
 * sinzui switches to firefox
<adeuring> abentley: I agree. But that's how it looked so far ;) (unless the derived class had its own __repr__ method)
<sinzui> bugger. Why does this work for me
<abentley> adeuring: Is the job_str method really related to this?
<adeuring> abentley: yes, is it used in the logger calls
<adeuring> ...in runJob() and runJobHandleError()
<abentley> adeuring: It doesn't seem to be used in the messages about MembershipNotificationJob that you changed, for example.
<adeuring> abentley: job_str() is used in runJob() and runJobHandlerError(), in self.logger.info().
<abentley> adeuring: You've changed it to use job_str in the "Running " message.  It used to use %r.  And that's why you're now changing job_str to use %r.
<sinzui> czajkowski, I think I sussed it. the bug is about the "Status:" field only. No one can edit "status" because it is a summary of "Direction", "Definition", and "Implementation"
<sinzui> czajkowski, so back to https://blueprints.qastaging.launchpad.net/gdp/+spec/gdplaunchpad . Do you see a yellow edit icon next to Definition Approved and Implementation started?
<czajkowski> sinzui: yes
<sinzui> thanks. I will update this bug, then stop panicing
<czajkowski> heh
<czajkowski> cheers the help
<czajkowski> why are there so many erp projects trying to triage one outta our queue and into theirs
<sinzui> yuck. That is messy
<sinzui> czajkowski, choose openerp
<czajkowski> sinzui: cheers
<jml> https://code.launchpad.net/~jml/lazr.restfulclient/multiple-instance-safe/+merge/98873
<jml> rick_h: thanks for the review. I'm guessing that '*' is still Launchpad secret squirrel code for the fact that it needs a review by another full fledged reviewer?
<rick_h> jml: yes, I'm in training. I've asked my mentor to peek as well
<jml> rick_h: thanks :)
<lifeless> gary_poster: I wonder if regular ec2 land is suffering entropy issues
<lifeless>  / ec2 test
<gary_poster> lifeless, yeah, me too.  I brought that up with Francis.  There are reasons why parallel testing woud have it worse, but yeah, I thought it might be an interesting experiment
<gary_poster> on call right now though :-)
<lifeless> gary_poster: cool cool, did you get anywhere on the hanging-at-the-end thing? or did that turn out to be entropy too? (reply when you can, no rush)
<gary_poster> lifeless, was entropy
<lifeless> fascinating
<lifeless> thanks
<lifeless> salgado: hi, how did you go ?
<lifeless> (I'm about to afk for 10 to give lynne a hand, so just braindump :P)
<salgado> lifeless, haven't had much time for it, but I was able to extend search_bugs() without breaking any tests.  now I'll try and use it in my method
 * deryck changes work locations
<czajkowski> sinzui: so folks are going to get confused if they are used to being able to change their status of a blueprint and not can't
<sinzui> czajkowski, lp never let someone change specification.status. it is a summary calculates by Lp with the definition and implementation status change
<sinzui> The rules have not change for 6 years
<sinzui> s/calculates/calculated/
<lifeless> salgado: great
<sinzui> czajkowski, that url I showed you (https://blueprints.qastaging.launchpad.net/gdp/+spec/gdplaunchpad/+status) is called +status because the assignee/driver needs to change those two fields to set the overall status
<czajkowski> hm ok
<czajkowski> thanks
<czajkowski> sinzui: thanks
<salgado> lifeless, http://paste.ubuntu.com/895534/
<lifeless> salgado: cool
<salgado> lifeless, it doesn't seem to do eager loading, though.  maybe I'm missing something
<mhall119> is there a way to reference a project's development focus bzr branch using the HTTP urls?
<salgado> lifeless, hmm, looks like it's doing it but with one query for every related table.  is that how search_bugs() work?
<lifeless> salgado: one per related table is fine ;)
<salgado> lifeless, right, but BugTaskSet.search() is hard-coded to do it only for Product/SourcePackageName, so I need to prejoin the others.  either that or use search_bugs() directly
<lifeless> salgado: you can extend bugtaskset.search to do more; you can eager load after you get the batch back
<lifeless> prejoining is bad
<lifeless> it makes postgresql slow
<amorphous1> jcsackett, hi there. Is there going on with launchpad? I'm truing to 'Propose Merge' but it's not going through
<amorphous1> rick_h, ^
<rick_h> amorphous1: what mp is this? do you get an oops number back?
* rick_h changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugtasks: 4*10
<amorphous1> rick_h, hm...it seems that's ok now...but it stayed like that for over 90 minutes
<rick_h> amorphous1: ok, if you happened to have an oops number or something I can go off of error wise I can look into it, but not sure without
<amorphous1> thanks rick_h , I'll look into it next time if it happens
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<wgrant> rick_h: Probably bug #961126
<_mup_> Bug #961126: +register-merge JS doesn't handle form validation errors <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/961126 >
<wgrant> wallyworld: ^^
 * wallyworld looks
<wgrant> wallyworld: http://www.youtube.com/watch?v=qKc7mhy6t2s
<rick_h> wgrant: when the tests run, is apache setup on the test box?
<wgrant> rick_h: No.
<rick_h> wgrant: ok, didn't think so, but wanted to dbl ckeck
<wgrant> wallyworld: I don't understand the "'NOTHING' as permission
<wgrant> " bit in findGranteePermissionsByPolicy
<wallyworld> wgrant: it's a placeholder
<wgrant> Is that just so Storm knows about the column?
<wallyworld> yes
<wallyworld> and a failsafe
<wgrant> Right.
<wgrant> I still don't see a DISTINCT there.
<StevenK> wgrant: .set() might not work -- https://bugs.launchpad.net/storm/+bug/625071 ?
<_mup_> Bug #625071: permit updates to fields on the far side of joins <Storm:New> < https://launchpad.net/bugs/625071 >
<wallyworld> wgrant: look harder
<wallyworld> wgrant: stormrangefactory and batching and decorated result sets suck - the main query is executed twice
<wgrant> StevenK: You have to construct the query manually.
<wgrant> StevenK: You can use Storm, but you can't use the existing attachments collection.
<StevenK> wgrant: Why not, it's a DRS. :-(
<wgrant> StevenK: wut?
<wgrant> Oh, attachments is?
<wgrant> Then you may be able to.
<wgrant> You can if you can.
<wgrant> You can't if you can't.
<wgrant> wallyworld: Ah, right, I was misled by it still taking a grantees arg.
<StevenK> wgrant: Yes, attachments_unpopulated returns a DRS directly.
<wgrant> wallyworld: Forgot that was the full query now.
<wgrant> Looks reasonable.
<wgrant> Will check qastaging in 10 hours :/
<wallyworld> wgrant: the grantees arg is used by a callsite in the service
<wgrant> Yep.
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Firefighting: - | Critical bugtasks: 4*10
#launchpad-dev 2012-03-23
<StevenK> wgrant: And you end up getting r15000
<wgrant> StevenK: Indeed.
<StevenK> wallyworld_: https://code.launchpad.net/~stevenk/launchpad/imports-once-more-with-feeling/+merge/98944
<wallyworld_> on it
<wallyworld_> StevenK: r=me. i'm sure you'll find some more next week
<StevenK> Haha, probably.
<wgrant> wallyworld_: How do you feel about 650ms for a batch of Ubuntu grantees on dogfood?
<wallyworld_> wgrant: i think that's ok, no? i have nothing to compare it to though
<wallyworld_> what does the profiling say take the time?
<wgrant> I mean in SQL.
<wgrant> Isn't it taking about 2s on prod atm?
<wgrant> s/prod/qas/
<wallyworld_> total, not sure about the breakdown
<wgrant> Can you check?
<wgrant> Expand the in-page query listing.
<wgrant> Since oops-tools chokes on these oopses.
<wallyworld_> i/m sort of in the middle of some coding right atm
<wgrant> k
<lifeless> wgrant: wait, what ?
<wallyworld_> messing with lp form infrastructure
<wgrant> lifeless: sqlparse crashes when asked to reindent some of wallyworld's SQL
<lifeless> aieee win
<wgrant> lifeless: It doesn't like MIN(CASE [...])
<lifeless> wgrant: you've opened a bug ?
<wgrant> No, was too busy trying not to throw buildbot and rosetta into a fire pit.
<wallyworld_> wgrant: i wasn't going to look too closely at the nimbers till the next branch lands
<wgrant> wallyworld_: I've tried the new primary DISTINCT query -- it takes 8s on DF.
<wgrant> Rewording it gets it down to 650ms.
<wgrant> Will see how it is on qas.
<wallyworld_> 8s? what does explain say?
<lifeless> ask again later
<wgrant> wallyworld_: It does the sort before the unique
<wgrant> So it's sorting like 140000 people
<wallyworld_> well that sucks
<wgrant> Forcing it to do the distinct first (by moving it into a subquery) works nicely, but it'd be nice if it was smarter.
<wallyworld_> you should land that
<lifeless> distinct will always sort before unique unless the data is being generated in-order for selective skipping
<lifeless> or the output sort is wholly different to the distinct columns, at which point brains assplode
<wgrant> Yeah.
<wgrant> Oh
<wgrant> It's the extra parens
<wgrant> >>> sqlparse.format('SELECT CASE (foo) WHEN 0 THEN 1 END', reindent=True)
<wgrant> u'SELECT CASE (foo)\n           WHEN 0 THEN 1\n       END'
<wgrant> >>> sqlparse.format('SELECT CASE(foo) WHEN 0 THEN 1 END', reindent=True)
<wgrant> Traceback (most recent call last):
<wgrant> When there's (unncessary) parens after CASE without a space, boom.
 * wgrant lands a workaround to LP
<lifeless> wgrant: you will file a bug upstream though?
<wgrant> lifeless: Yeah
<wgrant> Just confirming that it still exists.
<lifeless> thanks
<lifeless> I hsould have had fairh
<wgrant> orly?
<wgrant> Oh
<wgrant> faith
<wgrant> lifeless: Fixed upstream, it seems.
<wgrant> >>> sqlparse.format('SELECT CASE(foo) WHEN 0 THEN 1 END', reindent=True)
<wgrant> u'SELECT CASE(foo)\n           WHEN 0 THEN 1\n       END'
<wgrant> Our version's more than a year old.
<wgrant> But there's only been one release since.
 * wgrant upgrades us to 0.1.3.
<wgrant> lifeless: Does python-oops-tools do reviews?
<lifeless> humans do
<wgrant> Ah, I see there are MPs.
<lifeless> lp reviewers review them
<lifeless> or should be
<lifeless> its in the same group etc etc - /launchpad-project
<lifeless> e.g. wallyworld as OCR is interruptible
<lifeless> wgrant: if its a dep bump, self review though
<wgrant> Sure
<wgrant> Still wants an MP by policy, though.
<wallyworld_> am i needed?
<wgrant> Nope.
<wallyworld_> good :-)
<wgrant> I can self-review an s/2/3/ :)
<wgrant> Oh
<wgrant> It's tarmac
<wgrant> So I couldn't have committed directly anyway.
<StevenK> wgrant: Has your -11-4 triggers branch landed?
<StevenK> Grumble. 917 line branch. Perhaps I will have to split this thing.
<wgrant> StevenK: No, it was waiting for your information_type NOT NULL branch.
<wgrant> Which I believe landed this morning.
<StevenK> It did, yes.
<wgrant> But we already have two patches in the queue, and it won't clear until Tuesday, so I won't land it yet.
<StevenK> This code branch depends on it and -12-3
<wgrant> So Monday will be FKs, Tuesday will be NOT NULL, Wednesday will probably be my legacy access triggers
<wgrant> Then what do you have/
<StevenK> BugSummary triggers
<wgrant> Ah
<wgrant> wallyworld_: Ah, findGranteePermissionsByPolicy still returns oddly sized batches.
<wgrant> Because it's distinct on (person, policy)
 * wgrant fixes in the performance branch.
<wallyworld_> bollocks, thanks
<wallyworld_> wgrant: if you fix, can you add a test too?
<wgrant> wallyworld_: It necessitates a reworking of the return type, so I have to rewrite the tests anyway.
<wallyworld_> ok
<wgrant> (currently it returns 75 (person, policy, permission) rows, whereas it probably wants to return 75 (person, {policy: permission})
<wallyworld_> i like the 3-tuple form better
<wgrant> Indeed, but it doesn't work for slicing by person.
<StevenK> wgrant: So why does the langpack export take 22 hours? And no fair if you say because translations is horrible.
<wgrant> 'cause you'd slice by :75 and get 93 rows.
<wgrant> StevenK: Because translations is horrible.
<wgrant> I filed a bug on it a couple of weeks ago.
<StevenK> wgrant: No fair.
<wgrant> The bug mentions a specific thing that might be a performance problem.
<StevenK> 971 line diff :-(
<StevenK> wallyworld_: Are you going to hate me if my branch that drops private and security_related from CreateBugParams, IBug._private and IBug._security_related is roughly 1,000 lines?
<wallyworld_> no
<wallyworld_> i hte you anyway
<StevenK> :-(
<StevenK> wgrant: Turns out bugsummary *and* calculate_bug_heat has to change.
<wgrant> :D
<wallyworld_> StevenK: i was joking :-P
<StevenK> 220 database/schema/patch-2209-12-3.sql
<StevenK> stub is going to hate me too.
<wgrant> 523 ../bugtaskflat-db/database/schema/patch-2209-16-0.sql
<StevenK> Haha
<StevenK> One bug to fix, then I get to re-run -vvm bugs
<wgrant> Has diff generation been oddly fast or anyone else lately?
<wgrant> I'm regularly getting diffs within 20 seconds now.
<wgrant> wallyworld: https://code.launchpad.net/~wgrant/launchpad/sharing-prettier-sql/+merge/98967
<wgrant> wallyworld_: ^^
<wgrant> underscorefail
<StevenK> Haha
<wallyworld_> wgrant: r=me. will be good to see how it goes on qas
<wgrant> Marvellously I hope.
<wgrant> Then we can set the timeout to 1s and declare victory over the universe.
<nigelb> I want to setup a qdb just for all the fun quotes in #lp-dev
<StevenK> You'll keep adding branch names.
<nigelb> I'll just have everything StevenK and wgrant says.
<nigelb> maybe I should just make it a tumblr. shitlpdevssay.tumblr.com :P
<wgrant> !!
<wgrant> http://validator.w3.org/check?uri=https%3A%2F%2Flaunchpad.net&charset=%28detect+automatically%29&doctype=Inline&group=0
<wgrant> We are valid!
<wgrant> Finally.
 * wallyworld_ goes away for a bit - bookclub with kid after school
<StevenK> With 2 warnings?
<huwshimi> wgrant: So we've gone LIVE with changing over to html5?!
<wgrant> huwshimi: Yeah, this morning. I tested the site fairly thoroughly in Firefox/Chromium/Opera/Safari/IE8/IE9/IE10, and it looks good.
<huwshimi> wgrant: I've clearly missed the discussion about doing that...
<wgrant> It was just a doctype change :)
<wgrant> So the only rendering change is moving from Almost Standards to Standards.
<huwshimi> wgrant: Have you tested in ie6?
<huwshimi> (it's not JUST a doctype change)
<wgrant> huwshimi: The watermark is slightly broken, but we no longer support IE6.
<huwshimi> wgrant: When was that decided?
<wgrant> (only difference is that the logo is on a separate line)
 * wgrant finds a reference.
<wgrant> 06:56 < lifeless> rockstar: we just blew IE6 away
<wgrant> 06:56 < lifeless> rockstar: so gnar gnar gnar :P
<wgrant> And various other discussions.
<wgrant> But the doctype change makes no visible difference in IE6.
<huwshimi> wgrant: Have you check that?
<huwshimi> wgrant: You're going to have to cite something better than that. Last I heard we have significant stakeholders who require IE6. If that's changed it's not been announced publicly.
<wgrant> huwshimi: That restriction came from OEM a little over two years ago, and is apparently no longer true. https://dev.launchpad.net/GradedBrowserSupport was recently updated to remove any reference to IE.
<wgrant> And it has been stated on IRC that IE6 is no longer required.
<wgrant> lifeless: ^^ right?
<wgrant> Anyway, still works fine on IE6, except for the logo and title being on different lines.
<wgrant> (and the fact that we use PNG transparency for sprites, which makes everything look pretty hideous, but that's been the case for years)
<huwshimi> wgrant: There needs to at least have been a discussion about migration, ramifications, future implications etc.
<wgrant> huwshimi: For the doctype change that doesn't change the markup that engineers must write at all, brings the existing HTML5 elements and attributes that we use into validity, and in browsers causes a single rendering change (the removal of the shrinkwrap effect from inline images)?
<wgrant> It's not as if we're moving from XHTML to HTML rendering.
<wgrant> There was some discussion on our call, however.
<huwshimi> Right, those are SOME of the implications of changing to HTML5.
<wgrant> The only change is the doctype, and the only user-visible difference is the inline img shrinkwrap removal, which has already been fixed everywhere that it was used.
<wgrant> There may be implications of using more HTML5-only features.
<wgrant> But that wasn't the change here.
<wgrant> We've always been polyglot HTML/XHTML out of necessity, so there is no markup difference.
<wgrant> So the implications are remarkably close to 0.
<huwshimi> wgrant: I think you're missing the point.
<wgrant> huwshimi: Oh?
<huwshimi> wgrant: "16:01:32 huwshimi: wgrant: There needs to at least have been a discussion about migration, ramifications, future implications etc."
<wgrant> Migration: the shrinkwrapping change. Ramifications: the shrinkwrapping change, and our existing usage of HTML5 elements is no longer invalid. Future implications: nothing new.
<huwshimi> (and by discussion I mean on the mailing list)
<wgrant> Particularly boring.
<lifeless> wgrant: huwshimi: we are waiting on a final ack from OEM *but*
<lifeless> statik has said that us supporting IE6 isn't a good use of time
<lifeless> and mrevell has said we won't do it - on the stakeholders list
<lifeless> the final ack we're waiting on is in case someone comes back and says its part of getting X revenue to make it worth while
<wgrant> Ahh, that's why I couldn't find the discussion.
<wgrant> There it is, indeed.
<lifeless> IE6 is now at most best effort
<wgrant> "Thanks. Let me explain a little more clearly where we are. We no
<wgrant> longer support IE6 and we won't make any effort to ensure a good
<wgrant> experience in IE6."
<wgrant> (from mrevell)
<huwshimi> lifeless: Thanks (clearly I'm not on that list).
<huwshimi> we need to at least communicate that to our users if we do go ahead
<wgrant> huwshimi: We knowingly released dynamic bug listings when they're completely broken in all versions of IE.
<huwshimi> wgrant: We did!?
<huwshimi> Who knew?
<wgrant> I'm pretty sure it was knowingly, but regardless it's clear that nobody tried them even though they were clearly using new JS features unlikely to be supported in IE.
<lifeless> the team explicitly put in a check for IE short circuiting the code, because they hadn't tested it
<wgrant> Hm?
<wgrant> It's not short-circuited.
<wgrant> It runs, and is broken.
<lifeless> wgrant: there was a new if (...IE) thing you ripped out just this week
<lifeless> anyhow, I have to run
<wgrant> Nope.
<wgrant> Those were all old.
<lifeless> I have no objection to us announcing that we're not caring for IE6 anymore
<wgrant> I only ripped out things that work in IE>=8.
<lifeless> wgrant: ah, my misunderstanding
<lifeless> OTOH I don't think we need to shout it from the rooftop that we don't support a given browser; most sites just track current releases and make no fanfare about changes
<wgrant> Exactly.
<wgrant> And our IE6 and IE7 support has been in a pretty severe state of disrepair since 3.0.
<wgrant> Anybody using them significantly since then has probably torn their eyes out by now.
<wgrant> So they wouldn't be able to read the notification anyway :)
<wgrant> (functionality critical for OEM *worked*, and still does, but it's always been fugly)
<huwshimi> Well the fact that we had (or have) stakeholders that we were maintaining support for means we need to be a little polite
<wgrant> huwshimi: mrevell asked someone in PES if a polite notification was desired, and received no reply AFAICS.
<wgrant> This was 6 weeks ago.
<wgrant> Ah, you should be able to see the ML archive.
<wgrant> https://lists.launchpad.net/private-canonical-launchpad-stakeholders/msg00658.html
<StevenK> wgrant: I think I've broken calculate_bug_heat(), can I use 'in' in a plpy block?
<wgrant> StevenK: In what context?
<StevenK>     if bug['information_type'] in (3, 4):
<StevenK>         heat['privacy'] = BugHeatConstants.PRIVACY
<wgrant> That should work, but you actually want (3, 4, 5)
<wgrant> And it's possible it's some magical type that doesn't work like that.
<StevenK> It's getting pulled out by SELECT
<StevenK> But I'm lost how to debug it
<wgrant> Pastebin the whole function
<StevenK> wgrant: http://pastebin.ubuntu.com/896042/
<wgrant> StevenK: wut
<wgrant> That's prehistoric
<wgrant> You need to check patches too :)
<wgrant> database/schema/patch-2209-07-1.sql
<wgrant> It's a little bit simpler.
<StevenK> A little?
<StevenK> wgrant: How do I replace CASE private WHEN true -- CASE information_type IN (3, 4, 5) ?
<wgrant> StevenK: For now I guess CASE information_type WHEN 1 THEN 0 WHEN 2 THEN 250 WHEN 3 THEN 400 ELSE 150 END
<wgrant> That will match the current behaviour.
<StevenK> WHEN 4 THEN 150
<StevenK> Oh, ELSE
<StevenK> Nevermind me
<wgrant> To catch USERDATA and PROPRIETARY, yes.
<wgrant> I think it makes sense.
<StevenK> wgrant: Indeed. It even works.
<StevenK> wallyworld: Are you still reviewing?
<wallyworld> StevenK: when are you expecting your db patches to land?
<wallyworld> StevenK: yes to your question, i didn;t see it
<StevenK> wallyworld: https://code.launchpad.net/~stevenk/launchpad/bugs-remove-old-privacy/+merge/98974
<wallyworld> StevenK: already looking
<wallyworld> StevenK: 69	# Security bugs are always private when filed, but can be disclosed
<wallyworld> i think that comment needs updating
<StevenK> It
<StevenK> Sigh
<StevenK> It's current for the moment, it will change when the UI does.
<wallyworld> hmm. i'd just s/private/embargoed
<wallyworld> i think that's more accurate given the code change just below it
<StevenK> wallyworld: -12-3 is in review, -11-4 is reviewed but not landed, but we have Monday and Tuesday for FDTs already so if -11-4 and -12-3 land on Tuesday, it gives us time.
<StevenK> wallyworld: So I'd expect this branch in ec2 next Friday morning
<wallyworld> too bad we can't do more than one patch per FDT
<wallyworld> we have to wait a week to get your branch in ec2
<StevenK> wallyworld: I need the patches in devel (otherwise a large amount of tests go bang), and that doesn't happen after it has been applied to prod.
<StevenK> wallyworld: I'm tempted to just slam the DB patches I need into this branch and toss it at ec2 test on Tuesday just so I can fix all the other failures.
<wgrant> The DB patch branches should be merged into that one; it depends on them.
<wallyworld> StevenK: you could do that if you run ec2 test
<wallyworld> StevenK: i don't understand the need for the conditional on line 95
<wallyworld> 95	+ if params.information_type is InformationType.PUBLIC:
<wallyworld> 96	+ params.information_type = InformationType.USERDATA
<StevenK> wallyworld: We don't want to clobber EMBARGOEDSECURITY
<StevenK> Which may be set higher up
<wallyworld> ok, makes sense
<wallyworld> StevenK: return "Bug.information_type IN (1, 2)", _nocache_bug_decorator
<wallyworld> instead of 1 and 2, should we use InformationType.PUBLIC.value etc
<StevenK> wallyworld: So, I think that method should be entirely rewritten because it's horrid
<StevenK> wallyworld: So I did the smallest amount of work to get the method working
<wallyworld> StevenK: ok. i think you could probably also change transitionToInformationType to not return a tuple
<wallyworld> it would only be a few extra loc
<StevenK> transitionToInformationType doesn't return a tuple :-)
<wallyworld> sorry, setPrivacyAndSecurityRelated
<wallyworld> since the tuple values would always be equal
<StevenK> Right
<wallyworld> StevenK: other than that, i think it looks ok. but as with most things lp, i'm sure there will be a few dark corners we have forgotten to look in
<StevenK> Haha
<StevenK> Yeah
<wallyworld> let's hope the test coverage is adequate
<StevenK> Yeah. :-(
<wallyworld> StevenK: r=me but i had a thought. why don't you just delete setPrivacyAndSecurityRelated entirely?
<StevenK> Because it's bloody exported!
<wallyworld> but there's only a few places left in the code base so i would remove those at least
<wallyworld> eg bugimport
<wgrant> Isn't it new?
<wgrant> Didn't we decide it could go away because it's not part of 1.0?
<wallyworld> and yes, i think we did
<StevenK> Hmmm.
<StevenK> I could kill it ...
<wallyworld> what's a few extra changes? it's already quite large :-)
<wallyworld> go on, you know you want to
<stub> StevenK: Your confident your code was based off the latest versions of those stored procedures? We unfortunately have versions scattered throughout the DB patches and we broke things once before by accidently using old code as a basis for a change.
<StevenK> stub: I'm confident about calculate_bug_heat(). Would you mind checking for the other three?
<stub> k. I'll have a trawl too.
<stub> But that might end up after lunch :)
<wgrant> I generally check \df+ just to be sure.
<stub> yer
<wgrant> Particularly since this touches bugsummary...
<stub> Hmm... I suspect the best way of doing these changes is to use '\df+' to generate current functions into your db patch. commit. then edit, and we can generate a nice diff.
<wgrant> BugSummary probably hasn't been touched since the last schema snapshot, so you can likely just pull them straight from launchpad-2209-00-0.sql
<wgrant> But yeah.
<wgrant> That might be a good way.
<wgrant> Currently it's pretty horrible.
<stub> Oh, if it hasn't been touched we don't have to worry about accidentally cargo culting old versions any more.
<wgrant> Yeah
<wgrant> I think before the last snapshot production had a different version than dev.
<wgrant> Due to massive confusion around those emergency fixes.
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<StevenK> wallyworld: Dropping setPrivacyAndSecurityRelated means that branch deletes more code than it adds.
<stub> StevenK: What are you using the bugsummary_viewers() function for? You might find a VIEW more suitable.
<wallyworld> StevenK: that's good that we end up with a net code deletion :-)
<adeuring> good morning
<StevenK> stub: I have no idea at all -- it was using private, so I changed it.
<stub> oh, thought that was something new you were adding :)
<StevenK> stub: Nope, those are 4 triggers that should already being used and called on prod today -- I'm converting their use of bug.private and bug.security_related to bug.information_type
<StevenK> stub: Thank you for the approval, I'll toss it at ec2 on Monday.
<wgrant> wallyworld: Bug #962912 :(
<_mup_> Bug #962912: StormRangeFactory evaluates DecoratedResultSet slices twice <disclosure> <performance> <Launchpad itself:Triaged> < https://launchpad.net/bugs/962912 >
<wallyworld> wgrant: yes, i noted that in my mp
<wallyworld> but didn;t raise a bug
<wallyworld> because there's a reason for it
<wallyworld> so we would need to find a work around
<wgrant> wallyworld: I think it should be pretty easy. Will look at it next week
<wgrant> wallyworld: Just needs to use a custom version of __iter__ on DRS that returns (base, decorator(base)) rather than just decorator(base)
<wallyworld> yeah probs. my brain was too full to think about it this week
<wallyworld> and we can fix the other issue too
<wgrant> WHich other issue?
<wgrant> I didn't actually read the text of your MP, just the diff.
 * wgrant finds.
<wgrant> Hm, don't see another issue.
<wallyworld> wgrant: SQL tracing shows that the batching infrastructure executes the core batching query twice due to a decorated result set being used. This is unfortunate.
<wallyworld> https://code.launchpad.net/~wallyworld/launchpad/sharing-view-batching4-957663/+merge/98820
<wgrant> That was the first issue, I was looking for the second one that you mentioned.
<wallyworld> wgrant: ah, that the range factory can't handle person_sort_key
<wallyworld> bug 961836
<_mup_> Bug #961836: StormRangeFactory should support functional sort keys <batching> <infrastructure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/961836 >
<wgrant> wallyworld: ah, right.
<wgrant> wallyworld: That's a bit harder.
<wallyworld> yeah
<wgrant> But similar indeed.
<wallyworld> maybe
<rick_h> morning
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: bac | Firefighting: - | Critical bugtasks: 4*10
<deryck> Morning, all.
<rick_h> party on
<rick_h> deryck: https://code.launchpad.net/~rharding/launchpad/yuixhr_combo/+merge/99023
 * deryck switches work locations
<jcsackett> are we applying the new LoC maintenance costs experiment to all code, or just the LP tree? my understanding was the latter, but i'm curious.
<macagua> hi guys
<rick_h> jcsackett: hmm, that conversation came up last night. /me checks how that went
<adeuring> abentley: r=me for the latr.jobrunner mp
<abentley> adeuring: Cool.  Thanks.
<abentley> adeuring: Merged and pushed.
<adeuring> abentley: cool. but I can't yet see your other MP -- LP is terribly slow for me at the moment...
<abentley> adeuring: I haven't made a proposal because I'm still trying to sort out this configuration issue.
<adeuring> abentley: ah, ok
<abentley> adeuring: So with my branch merged, we have my job.fail fix landed without a test.
<adeuring> abentley: I know but that does not mean that can't test it ;)
<jcsackett> rick_h: ctrlp is pretty hot. :-P
<rick_h> jcsackett: yea, I'm liking it. Not used to starting in the lib directory, but with multiple tmux sessions going I'm getting over it slowly
<jcsackett> rick_h: i didn't need to start in lib; just added the non-code dirs to the custom filter list.
<rick_h> bac: can I get this on your list please? https://code.launchpad.net/~rharding/launchpad/yuixhr_combo/+merge/99023
<jcsackett> were you having problems with that working, or did you not want that globally set?
<rick_h> jcsackett: yea, but since I do other projects I didn't want to drop some other dirs
<jcsackett> rick_h: dig.
<bac> rick_h: sure
<rick_h> lib, build, I use on other code projects
<rick_h> jcsackett: but yea, starting in lib seems like a winner, just have to switch to another tmux window to fire off make/test commands and such
<jcsackett> yeah; it can run a custom filter command..you could probably rig up something nice with sourcing a command from a .ctrlpignore file.
 * jcsackett makes note to do that at some point.
<rick_h> ah, true. or even just a shortcut to set lpmode or something
<rick_h> I do get when it keeps reloading the cache more often than I'd expect, but not nailed down why yet
<rick_h> ssd ftw on that I guess, pretty fast
 * jcsackett nods
<rick_h> next up I need to get syntastic running vs my pyflakes plugin, maybe this weekend
<jcsackett> "vim: the neverending plugin and configuration rabbit hole" :-P
 * deryck is afk at accountant
<bac> rick_h: done.  sorry for the sashimi-induced delay.
<rick_h> bac: no problems, thanks for the peek
<rick_h> bac: thanks for the comment catch
<czajkowski> <---------- EOD nn
<rick_h> have fun czajkowski
<mhall119> does anybody know if there's a Django app that mirrors Launchpad objects?
<mhall119> Project/Team/Branch, etc
<sinzui> bac: do you have time to review https://code.launchpad.net/~sinzui/launchpad/project-notify-4/+merge/99094
<bac> sinzui: yes
<bac> sinzui: that branch has conflicts, though.  would you resolve them first?
<sinzui> okay
<sinzui> bac: I pushed the resolution of the import conflicts
<abentley> deryck: I am especially free to chat now, as I've just put my branch up for review.
<deryck> abentley, I'm not further along enough to articulate ideas yet. unfortunately, distracted by other work.
<abentley> deryck: cool.
<deryck> abentley, I'll try to knuckle down the rest of the day on it, and then let's chat first thing Monday.
<bac> sinzui: done.  neat branch and idea.  it's going to be awesome.
<deryck> abentley, if that's cool with you.
<abentley> deryck: sure thing.
<deryck> abentley, thanks, man!
<abentley> deryck: np.
<sinzui> bac, thanks for the encouragement. That branch was very hard. Reading all the other ways lp.registry sends emails really depressed me
<bac> no doubt
<sinzui> bac assertTrue(1) passes, but it is not a boolean. I used assertIs to avoid coercion.
<bac> sinzui: oh, gotcha.
<sinzui> bac do you have a few minutes to look at this small branch: https://code.launchpad.net/~sinzui/launchpad/front-page-layout/+merge/99121
<sinzui> it has a picture too
<bac> sinzui: i don't.
<sinzui> okay. thanks bac
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<wgrant> Yay
<wgrant> /ubuntu/+sharing down to 1.2s
<wgrant> And /launchpad/+sharing is faster enough that the AJAX batchnav feels very nice.
<lifeless> nice
<jelmer> 'evening launchpadders
<wgrant> Morning jelmer.
 * jelmer is liking the new +sharing pages
<jelmer> I guess it's intentional that they're read-only at the moment?
<wallyworld_> wgrant: ubuntu/+sharing seems to work well
<wgrant> wallyworld_: Indeed, 1.2s even with the dupe query
<wgrant> And launchpad/+sharing is about 600ms, 113ms SQL
<wgrant> But the AJAX batchnav feels pretty nice, even with 350ms of latency.
<wallyworld_> wgrant: i think the supe query does not load person objects, so it may not be too bad
<wgrant> wallyworld_: 25% of the page time is in load_objects
<wallyworld_> yes, i think it's quite usable
<wallyworld_> 25% but i don't think that's due to the 2nd query being done
<wgrant> Ubuntu should feel nice too once we remove the dupe query.
<wgrant> Indeed.
<wgrant> There's only about 110 load_object calls
<wgrant> Hm
<wgrant> Actually, that could be, then.
<wgrant> But we'll see.
<wallyworld_> so i haven't looked so i'm not sure how much time the dupe query takes
<wgrant> /launchpad/+sharing's queries are like 15ms
<wgrant> It's awesome.
<wgrant> In other news, lifeless makes EnumCols slow :(
<wallyworld_> how?
<wallyworld_> i like the service stuff because it's easy to tweak things
<wgrant> Indeed.
<wgrant> Well, about 5% of the total page time is creating and loading EnumCols.
<wgrant> I get a 50% speedup on that by reverting EnumCols to take a single DBEnum rather than multiple.
<wallyworld_> we could change that - just fetch the ids
<wgrant> Support for multiple enums in a single col was added for the bug search incomplete thing.
<wgrant> Nah, we should fix this properly.
<wgrant> Will be a universal speedup.
<wallyworld_> ok, i'm not across the root cause really. need to grok what's happening
<wallyworld_> wgrant: i also fixed the register merge thing - we had no support at all for XHR form error handling
<wgrant> Heh
<wgrant> Yeah, glanced at that vaguely.
<wgrant> Do you really need to use setContent?
<wallyworld_> it looks and works nice now
<wgrant> That's basically innerHTML
<wallyworld_> i could use set('text', xxx)
<wallyworld_> i'll change that
<wgrant> I know in some circumstances we assume that we can spit HTML out.
<wgrant> eg. if you try to register a spec with a dupe URL
<wgrant> But I'm not sure at what level the escaping is normally done.
<wgrant> Hmm
<wgrant> So, profiling shows that the total SQL execution time for /launchpad/+sharing is 55ms.
<wgrant> So 90% of the time is Python :)
<wallyworld_> cool, good that the db is not the bottleneck
<wgrant> For Ubuntu it is, but everything else should be Python.
#launchpad-dev 2012-03-24
<wgrant> Mmmmmm sub-second batching
<lifeless> wgrant: including rtt?
<wgrant> lifeless: Yes
<wgrant> lifeless: 0.85 browser time for an ubuntuone-servers batch on qas
<wgrant> 0.85
<wgrant> s
<lifeless> nice
<lifeless> hmm, I suspect we need to move to 2pc
<lifeless> will solve a raft of failure modes around SOA
<nigelb> bigjools: You need to taste good turkish delights :D
<czajkowski> morning
<wgrant> Morning czajkowski.
<czajkowski> wgrant: howdy
<lifeless> http://www.webperformancetoday.com/2012/03/21/neuroscience-page-speed-web-performance/
#launchpad-dev 2012-03-25
<cody-somerville> python-launchpadlib is not currently installable from launchpad ppa on natty:
<cody-somerville>  python-launchpadlib : Depends: python-lazr.restfulclient (>= 0.11.2-1) but 0.11.1-1build1 is to be installed
<cody-somerville>                        Depends: python-lazr.uri (>= 1.0.2-4) but 1.0.2-3build1 is to be installed
<wgrant> cody-somerville: Natty's no longer supported for Launchpad development.
<cody-somerville> It hasn't even been a year since Natty was released.
<lifeless> cody-somerville: its best effort only
<lifeless> cody-somerville: not deliberately broken, but no effort made to spport it
<wgrant> Everyone else who has wanted to run Launchpad has been running either Lucid or the latest stable.
<wgrant> Nobody complained when we proposed not actively maintaining Natty in ppa:launchpad, so we stopped.
<cody-somerville> is python-launchpadlib getting backported in another PPA?
<lifeless> you can use the one in natty itself
 * cody-somerville manually patches his launchpadlib with https://code.launchpad.net/~leonardr/launchpadlib/handle-unknown-token/+merge/51563
<StevenK> cody-somerville: Are you subscribed to launchpad-dev? That's where I announced that Maverick was removed from the LP PPA.
<cody-somerville> StevenK, I am. I use natty not maverick though.
<StevenK> I announced that Natty might get shot in the face, though.
<StevenK> wgrant: I figured out why my rev broke. IBugSet.createBug() calls notify() *then* _setInformationType()
<StevenK> wgrant: But I'm not sure how to test that. The fix, of course, is to call _setInformationType() right after the bug is created.
<wgrant> StevenK: Right, that's what I suspected immediately :)
<wgrant> StevenK: Should be very easy to test.
<wgrant> StevenK: You just have to create a private bug on a product with a structsub, and check the notification recipients.
<wgrant> StevenK: And no, the fix is to merge _setInformationType into whatever creates the bug.
<wgrant> StevenK: private/security_related exist only for compatibility; they should not permeate that far into the model.
<StevenK> I'm still questioning if they will die completly.
<StevenK> But yes -- I can bring forward convert_to_information_type() in adapters.bug from the next branch and pass in private, security_related and information_type to the Bug() call.
<wgrant> Differences (ndiff with -expected +actual): - Time zone: Europe/London (UTC+0000) + Time zone: + Europe/London + (UTC+0100)
<wgrant> Hah
<wgrant> I guess that'll affect all builds for the next view months...
<wgrant> few
<StevenK> wgrant: Oh sigh
<lifeless> StevenK: yo can subscribe, create a bug, in your handler check the information type
<wgrant> But you also need the integration test that I outlined above.
<StevenK> I tried to write it, it fails either way.
<czajkowski> aloha
<rick_h> 080254
<lifeless> http://www.soapatterns.org/masterlist_c.php
<bigjools> lifeless: people still run php!
<lifeless> very many
<lifeless> probably generates most pages on the internets
 * bigjools was being tongue-in-cheek
<bigjools> glad you are thinking about the patterns anyway
<bigjools> it's nice to see some abstractions written down
<wallyworld_> bigjools: boo!
<bigjools> wallyworld_: enjoy your drag club?
<wallyworld_> how's your ticker this morning?
<wallyworld_> yes, *very* informative
<StevenK> wgrant: http://pastebin.ubuntu.com/899767/  is my test complete bong?
<wgrant> StevenK: Does it work?
<wgrant> (no, it doesn't -- so yes, complete bong :))
<wgrant> StevenK: You need to check the recipients *at creation time*.
<wgrant> This is well after.
<wgrant> You could do what lifeless said, or you could create a bug and inspect the BugNotificationRecipient table afterwards, or preferably both.
<StevenK> I'd rather just one nice test.
<StevenK> But okay, I need to reach into BugNotificationRecipient with Storm.
<wgrant> StevenK: You need an integration test.
<wgrant> But both would be preferable.
<StevenK> wgrant: I thought the one I pasted was?
<lifeless> wgrant: StevenK: checking the BNR table afterwards is a test I would delete
<wgrant> lifeless: Why?
<lifeless> notificationservice won't have that table
<wgrant> .............
<wgrant> notsureifserious
<lifeless> totally
<wgrant> Then you'd change the test to check the notification service?
<lifeless> use abstractions, don't test across them
<bigjools> completely
<wgrant> lifeless: THen mock out the notification service.
<wgrant> But don't just mock out one of the Zope subscribers and check that single callsite.
<wgrant> Not for something like this.
<StevenK> wgrant: Right, and there should be *one* notification to the reporter?
<StevenK> (Currently, I have two)
<wgrant> StevenK: And potentially one to the security contact or bug supervisor.
<lifeless> StevenK: you probably have two due to bug
<StevenK> lifeless: Yes, I know, thank you.
<lifeless> bug 132300
<_mup_> Bug #132300: mail for new bugs inconsistent with other bug mail (ignores mail-on-my-actions disabled, missing footers, duplicate mail vs structural subscriptions) <email> <lp-bugs> <notifications> <Launchpad itself:Triaged> < https://launchpad.net/bugs/132300 >
<lifeless> StevenK: ^ was digging the number up
<StevenK> lifeless: I have two due to the critical filed on the weekend.
<StevenK> The one I've been thinking about, and kicking myself for since I found about it.
<lifeless> StevenK: bug 963539 ?
<StevenK> lifeless: Yah.
<StevenK> wgrant: http://pastebin.ubuntu.com/899792/  Given that test (which passes), what other test do I need?
<wgrant> StevenK: There's no existing API for that?
<StevenK> Not that I could see.
<wgrant> And that test fails before the reversion?
<StevenK> wgrant: Yeah, I resurrected the changes, and the test failed. Applying the diff I wrote in anger on Sunday fixes it.
<lifeless> wgrant: equally, or better, mock out the existing notification api to check its private. Oh wait, thats what I suggested.
<wgrant> lifeless: Mocking out the API called by the subscriber, maybe.
<wgrant> But the subscribers are too muddy now to safely mock out directly.
<wgrant> Erm
<wgrant> Who is this Eike character, and why did they readd the crap that you removed from that bug description? :/
<StevenK> wgrant: I can reach into BugNotification and pull out the record and then call .recipients() on it, or I can just Join() through to BNR.
<StevenK> I picked the latter.
<lifeless> wgrant: i suspect a bug in 'bug-repo-syncer'
<wgrant> Indeed it is.
<wgrant> ./bug_repo_syncer/common.py:        #Match discovery bug link: "[@[@BUGLINK "bug #123"@trac @]@]"
<_mup_> Bug #123: There's no direct way to see the project info when translating it <lp-translations> <Launchpad itself:Fix Released> < https://launchpad.net/bugs/123 >
<lifeless> bug 964862
<_mup_> Bug #964862: something is damaging the bug description in bug 936502 <bug-repo-syncer:New> < https://launchpad.net/bugs/964862 >
<wgrant> lifeless: So, yes, ideally we'd mock out the notification API, except that LP's "lol what's an abstraction" policy makes that difficult to do reliably without checking the DB tables.
<StevenK> wgrant: So I've fixed and can put this up for review again, or I need another test?
<wgrant> StevenK: Is information_type now handled natively?
<StevenK> wgrant: information_type is now set correctly when the bug is created.
<lifeless> thats a different question :P
<wgrant> All three attributes need to be set correctly in the initial INSERT
<wgrant> And preferably the method only takes information_type
<wgrant> Calculating private and security_related from that.
<StevenK> wgrant: All three attributes are set correctly when the bug is created.
<wgrant> They were last time too.
<wgrant> What we care about is *in the initial INSERT statement*.
<StevenK> Sorry, yes, set correctly during the initial INSERT.
<wgrant> Sounds reviewable :)
<StevenK> wgrant: http://pastebin.ubuntu.com/899809/
<wgrant> Indeed.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/bugs-use-information_type-redux/+merge/99236 if you're offering
<wgrant> StevenK: Will look in a sec.
<wgrant> Trying to convince the staging mailbox to function...
<wgrant> "Downloading message header 1 of 345783"
<StevenK> Haha
<StevenK> Hmmmm. Mail. I should read that.
<wgrant> StevenK: btw, you may have seen that cocoplum scripts are nodowntime now.
<StevenK> I have, yes.
<wgrant> StevenK: This means that everywhere that touches the DB is nodowntime.
<wgrant> So we no longer have to inject cocoplum deployment sequence points.
<wgrant> For this sort of thing.
<lifeless> poppy still has a DB connection
<wgrant> lifeless: Yes, but it doesn't use it.
<wgrant> So it doesn't matter.
<lifeless> wouldn't it be a lot safer to remove it
<lifeless> so that there is no timebom
<wgrant> Oh yes.
<lifeless> b
<wgrant> I actually looked last week at deleting poppy from the tree.
<lifeless> o\o/
<wgrant> Sadly it would require the extraction of the SSH server stuff.
<lifeless> so ?
<wgrant> Which is slightly less than trivial, so I didn't JFDI.
<lifeless> that needs extraction anyhow
<wgrant> Sure.
<lifeless> fair enough
<wgrant> Had it not required that, it would have been an obvious JFDI task.
<wgrant> StevenK: Can I expect a followup in the near future that makes information_type the native form?
<StevenK> wgrant: That's in the branch that was reviewed on Friday
<wgrant> Ah
<wgrant> Indeed.
<StevenK> Which depends on two DB patches
<wgrant> Does not.
<wgrant> The change I want is purely internal API changes.
<wgrant> Currently things still pass private/security_related flags around.
<wgrant> When they should only deal with information_type.
<StevenK> Right. That's the next branch.
<wgrant> The Bug model can deal with mapping that to private/security_related.
<wgrant> StevenK: I think I mentioned this in the last review, but from the diff it looks like bug.private writes were previously permitted by the security policy, but I don't see any ZCML changes in this brnach to prevent that.
<StevenK> wgrant: wgrant: private/security_related aren't in the ZCML, so I'm not sure how to tell it "forbid anyone setting these"
#launchpad-dev 2013-03-18
<StevenK> Somebody should blow sampledata up
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/performant-getSourcesForDeletion/+merge/153704
<wgrant> StevenK: Have you checked that my theory is correct?
<StevenK> It sounded plausible to me
<wgrant> A query to check that scheduleddeletiondate is set iff (published or has published (or unremoved? can't remember) binaries) would give me more confidence
<wgrant> I guess if you check scheduleddeletiondate on the binaries it might work
<wgrant> Though it's not going to corrupt data if my assumption is incorrect, we should still try to verify it against existing data.
<StevenK> wgrant: Query is http://pastebin.ubuntu.com/5624173/, results http://pastebin.ubuntu.com/5624172/
<wgrant> StevenK: https://pastebin.canonical.com/86983/
<StevenK> wgrant: http://pastebin.ubuntu.com/5624196/
<wgrant> I know
<StevenK> Right
<wgrant> So it looks like it might be right, but maybe not
<wgrant> We should probably investigate those cases to see what might be up
<wgrant> They may indicate publisher bugs
<StevenK> Which cases?
<wgrant> Those 20
<wgrant> It's also possible that a publisher run will fix those
<wgrant> Because this invariant is only eventually consistent
<StevenK> Well, let's see on prod
<StevenK> Since qas is a bit odd
<wgrant> Oh
<wgrant> You also check that the binary is actually published
<wgrant> Rather than condemned
<StevenK> That was what the old method did
<wgrant> And I only changed the archive in one part of the query
<StevenK> Haha
<StevenK> Oh
<StevenK> Right, so the results are fairly useless
<wgrant> That also explains why it ran so quickly
<StevenK> If you're running it on sour, I'll kill mine
<wgrant> I am
<StevenK> Mine has been killed
<StevenK> wgrant: That query with 32 for both archives is 0 rows
<StevenK> wgrant: What did that query return for PRIMARY?
<wgrant> StevenK: 20000 rows
<StevenK> Bloody hell
<StevenK> I don't think Archive:+delete-packages works for PRIMARY anyway?
<wgrant> Sure
<wgrant> But the primary archive and PPAs use the same domination code.
<wgrant> Any problem in the primary archive can apply to another archive.
<wgrant> So it is a useful pathological test case.
<StevenK> Right, so do you want to split up the work, or what is your plan?
<wgrant> Hmm, overrides, maybe
 * wgrant gives up
<StevenK> Overrides will indeed fuck that up
<StevenK> Thinking about it
<StevenK> No overrides for PPAs
<StevenK> wgrant: So I think it's plausible that overrides won't be doing that query any favours.
<wgrant> Running it over the first 10k PPAs gives about 500 results
<wgrant> Possibly explainable by copies
<wgrant> I guess it'll do
<StevenK> wgrant: Wait, how do copies explain it?
<wgrant> StevenK: Undeletions
<wgrant> If I delete something, let it be removed, then revive it
<StevenK> Oh, the necromancy style of copies
<wgrant> The old pubs may have been removed from disk, despite there now being live binaries
<StevenK> wgrant: If it will do, can I get a review, then?
<wgrant> StevenK: r=me with a comment
<StevenK> wgrant: Your comment assumes I understand the reasons :-P
<wgrant> StevenK: We only expose deletion of sources in the UI
<wgrant> But you may need to delete a superseded source's binaries
<wgrant> So we want to show a source if can itself be deleted, or if it has binaries that can be.
<wgrant> Due to the GPL and sanity, we don't condemn sources until their binaries are gone.
<StevenK> Right
<StevenK> Yeah, I just recalled GPL compliance
<StevenK> +        # We will return sources that can be deleted, or deleted sources that
<StevenK> +        # still have published binaries. We can use scheduleddeletiondate
<StevenK> +        # rather than linking through BPB, BPR and BPPH since we don't condemn
<StevenK> +        # sources until their binaries are all gone due to GPL compliance.
<StevenK> wgrant: ^
<wgrant> Sounds reasonable.
<StevenK> wgrant: How goes your TranslationMessage denorm?
<wgrant> StevenK: Given that it's a fairly large table, I'm looking for other things I can do at the same time to avoid rewriting it more than once
<StevenK> wgrant: It requires a full table rewrite?
<wgrant> StevenK: Denorming a column does, yes.
<wgrant> We need to populate it.
<wgrant> Resolving the message sharing performance debacle is probably a much more difficult matter, but we'll see
<wgrant> I'm reasonably convinced that the feature was primarily designed to make me cry years later.
<StevenK> Haha
<StevenK> Given TM is a 6GiB table, we should only rewrite it once, yes.
<wgrant> A statement count of 328 doesn't seem too bad until you realise that the page only shows 10 items and times out before it's done half the queries.
<StevenK> Hah
<StevenK> wgrant: I wonder if we want tmpreaper for ackee and loganberry to combat bug 881255 ?
<_mup_> Bug #881255: bzr-limbo-?????? and gpg-?????? in /tmp is not being cleaned up and taking up a lot of space <canonical-losa-lp> <Launchpad itself:Triaged> < https://launchpad.net/bugs/881255 >
<wgrant> StevenK: Combat? Conceal.
<StevenK> Conceal works too
<StevenK> wgrant: Hmmmm
<wgrant> The failure?
<StevenK> wgrant: Deleting publications does not set scheduleddeletiondate, so the sources turn up in the view again
<wgrant> StevenK: Bah, yes.
<wgrant> Damn
<wgrant> Although that's probably not necessarily a huge problem
<wgrant> Except for tests.
<StevenK> It will get set by the publisher?
<wgrant> It'll show the status correctly.
<wgrant> Yes
<wgrant> It's done during domination.
<StevenK> It will show the status correctly, but there is probably nothing stopping users from selecting a DELETED publication and trying to do so again and then getting a OOPS
<wgrant> It shouldn't oops
<StevenK> Hm, indeed
<wgrant> Recall that we show them when they have published binaries exactly so that an already deleted source can be deleted again
<StevenK> I thought I saw an assert for !DELETED
<StevenK> But I can't see one
<StevenK> wgrant: I think the easiest thing for the tests is to set scheduleddeletiondate
<wgrant> Depends on what the test wants to achieve
<wgrant> But probably, yes.
<StevenK> archive-views.txt is checking a double POST
<StevenK> We can probably just destroy the view.has_sources_for_display check
#launchpad-dev 2013-03-19
<StevenK> wgrant: So, WRT to the Archive:+repository-size critical, what if we denorm LFC + some squinting onto {B,S}PPH. A new size column directly on the publication, which is the sum of the 3 LFC sizes for a source, and a copy of the LFC size for a binary? Then we just need the publications for the archive, and can SUM() the sizes easily
<wgrant> StevenK: That's still a good number of rows. It would improve things, but only by a smallish constant factor.
<wgrant> Remember that those joins aren't really the big problem here; we're only filtering on one end.
<StevenK> The number of LFC rows is a problem
<StevenK> Which that denorming will solve
<StevenK> And if we add columns onto Archive, we need to deal with population issues, locking and updating issues
<wgrant> You still have to look up all the publications, which is less bad, but still not good.
<wgrant> Sure
<StevenK> Surely this problem isn't intractable :-P
<wgrant> It's not intractable, but there's no easy solution
<StevenK> At the moment, there is no solution I can think of that doesn't have you or I poking holes in from the very start. :-)
<wgrant> Your suggestion would improve things substantially, but it's a fair bit of work for something that isn't a proper fix.
<wgrant> And it's not clear that even that will scale.
<wgrant> To what we have now
<StevenK> wgrant: ArchiveSize table? That removes the locking problems, is probably scaleable with indexing, but leaves the population and updating issues.
<wgrant> StevenK: The problem is not storing the int somewhere -- the entire problem is calculating it
<wgrant> StevenK: What does the query look like today?
<wgrant> Whoa
<wgrant> StevenK: Have you... have you read the code?
<wgrant>         size = sum([lfc.filesize for lfc in result])
<wgrant> No wonder it's dreadful for even moderately sized archives
<StevenK> Well, yeah
<wgrant> Fixing that should be a substantial and just about trivial improvement
<wgrant> It certainly won't fix everything, but it's much less work than the other solution which also won't fix everything.
<wgrant> So it may be tolerable for now.
<StevenK> wgrant: Which is {S,B}PPH.size ?
<wgrant> Yeah
<wgrant> StevenK: Also, I just remembered why I completely discounted that last time
<wgrant> orig.tar.gz sharing
<wgrant> And arch-indep
<StevenK> arch-indep is not a issue, since they have BPPHs
<wgrant> Exactly
<wgrant> So each arch-indep binary will be counted for each arch
<wgrant> Rather than once
<StevenK> We don't count per arch
<StevenK> Just all binary + all source
<wgrant> You do if you use BPPH.size
<StevenK> Why? Fetch all BPPH's and SUM() their size
<StevenK> We don't care about their archtag, we only care about the archive
<wgrant> Publications regularly share files, so the sum of the publication sizes is not equivalent to the size of the archives.
<StevenK> We could use a FK straight to LFC, and distinct on that, but I like that even less
<wgrant> Have you checked how terrible the current queries are?
<wgrant> On a large PPA
<wgrant> You can't get that info from the OOPS reliably, due to the overhead of returning all the objects.
<StevenK> They are pretty terrible, yes
<wgrant> They're not terrible in the usual sense
<wgrant> They just deal with a lot of data
<StevenK> Yeah
<StevenK> Dear qas, load Archive:+delete-packages :-(
<StevenK> wgrant: So, rewriting a query
<wgrant> StevenK: Which query? +delete-packages?
<StevenK> wgrant: No, that query is actually performant on qas, but the page doesn't load. Not sure if it's due to the current issues.
<wgrant> StevenK: what needs rewriting?
<StevenK> wgrant: Archive:+repository-size
<wgrant> Oh
<wgrant> Work the sum into the query
<StevenK> That was your suggestion, no?
<wgrant> Rather than pulling back a bazillion Storm objects and summing an attribute of them
<wgrant> in Python
<StevenK> Yeah
<wgrant> There's a comment about that not working
<wgrant> But we can make it work
<StevenK> How do we do the distinct for SHA1/filename, though?
<StevenK> That bit I can't work out
<wgrant> StevenK: See the comment in sources_size
<wgrant> It already does the distinct
<StevenK> Yeah, I saw that
<StevenK> Sorry, let me be more clear.
<wgrant> Hopefully storm's sum() knows to do it after a subselect
<wgrant> I haven't used it like that before
<StevenK> SELECT SUM(lfc.filesize) FROM ....
<wgrant> You will most likely need a subselect
<wgrant> Storm should automatically construct it
<StevenK> wgrant: I thought there was a 'make sure that this is distinct' thing
<wgrant> StevenK: What do you mean?
<StevenK> Ah, I can't use DISTINCT ON
<wgrant> No, because that will happen after the SUM
<wgrant> Unless you use a subselect
<StevenK> SELECT SUM(filesize) FROM libraryfilecontent WHERE id IN (SELECT DISTINCT(id, lfa.filename, lfc.sha1) ....) ?
<wgrant> Nonono
<wgrant> Slow
<wgrant> SELECT SUM(filesize) FROM (SELECT DISTINCT ON (id, filename, sha1) filesize FROM ...) AS donotcare;
<StevenK> Do we have a similar callsite? So I can try and work out how to fit the Storm we have into that pattern?
<wgrant> -        return sum(result.values(LibraryFileContent.filesize))
<wgrant> +        return result.sum(LibraryFileContent.filesize)
<wgrant> +        return result.sum(LibraryFileContent.filesize) or 0
<wgrant> perhaps
<wgrant> As it'll otherwise be None when there are no rows.
<wgrant> SELECT SUM("_expr") FROM (SELECT DISTINCT LibraryFileAlias.filename, [...]
<wgrant> As expected
<StevenK> 46 queries/external actions issued in 11.74 seconds
<StevenK> Success
<wgrant> Why so slow?
<StevenK> The queries look good, at least
<StevenK> wgrant: From the stack of OOPSes, it's all in non-SQL time
<StevenK> Maybe we should move to SRF
<wgrant> SRF doesn't affect non-SQL time
<wgrant> And it's not a good fit for this problem
<wgrant> How long is the primary query?
<StevenK> 400ms
<wgrant> On ppa:ubuntu-langpack/ppa?
<StevenK> Yeah
<wgrant> Not dreadful
<StevenK> It's what we expected when we tested on sourcherry
<StevenK> wgrant: http://pastebin.ubuntu.com/5627366/
<StevenK> wgrant: Would you prefer DISTINCT ON filename, SHA1 or it doesn't hurt?
<wgrant> StevenK: It should behave identically, so no need to change
<StevenK> wgrant: Do we need to go through the EXPLAIN ANALYZE dance, or shall I just push this branch up?
<wgrant> StevenK: The plan is pretty trivial, but still something of potential interest
<wgrant> So I'd get one just in case
<StevenK> wgrant: http://pastebin.ubuntu.com/5627380/
<StevenK> Using my favourite archive ever
<StevenK> wgrant: It's 20ms for archive 71, which is one of yours
<wgrant> StevenK: How is a subsequent run?
<wgrant> Some of those indices will be pretty cold
<StevenK> wgrant: ~ 380ms
<StevenK> I ran it a few times afterwards without the EXPLAIN ANALYZE, and that was as fast as it got
<wgrant> Right
<wgrant> Right
<wgrant> Yeah
<wgrant> So
<wgrant> Since it's not pulling back $lots of storm objects it should be much faster
<StevenK> Yeah
<wgrant> The underlying query still sucks
<wgrant> But it can't be fixed without a lot of design work
<wgrant> And it should be fast after a couple of refreshes
<wgrant> Unlike the Storm bits that are problematic before your branch
<StevenK> It's loaded by AJAX already
<wgrant> Yes
<StevenK> It won't end up realizing any from those methods, right?
<StevenK> Since it will return a number, which might be 0
<wgrant> Right
<wgrant> .sum(...) will return an int or None
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/archive-repo-size/+merge/153997
<wgrant> StevenK: r=me
<StevenK> wgrant: I think the test failure in buildbot is actually a bong test.
#launchpad-dev 2013-03-20
<StevenK> wgrant: You have two LFAs with identical SHA1s and filesizes and the old binaries_size was counting them twice (so 18 + 18 = 36); whereas we now weed out duplicates so 18
<wgrant> StevenK: LFAs or LFCs?
<StevenK> wgrant: The LFA ids were different, I'm not sure if they were FK'd to the same LFC.
<wgrant> StevenK: We were just distincting on LFC before, weren't we?
<wgrant> I don't quite remember.
<wgrant> But anyway, yes, it seems likely the test is buggy
<StevenK> wgrant: Both LFA and LFC are distinct.
<wgrant> StevenK: Now or before?
<wgrant> Now they are
<StevenK> wgrant: Sorry, in the test, the two publications have distinct LFAs and LFCs.
<wgrant> But the LFCs are identical?
<StevenK> wgrant: http://pastebin.ubuntu.com/5629884/
<wgrant> Right, so normally librarian-gc would have coalesced them
<wgrant> The test is wrong
<StevenK> So I have no idea how it was coming up with 54.
<StevenK> Since that's mentioned in a comment
<StevenK> 36 I get, but it's still wrong
<StevenK> wgrant: http://pastebin.ubuntu.com/5629892/
<wgrant> StevenK: Sounds good
<StevenK> wgrant: That file was a bit crazy
<StevenK> Less so now
<StevenK> wgrant: The source query takes ~ 400 ms, the binary query is more like 1 second
<StevenK> (For ubuntu-langpack)
<wgrant> StevenK: Sounds OK
<wgrant> StevenK: Tried on some Kubuntu PPAs?
<StevenK> wgrant: You assume I know where some of them live.
<wgrant> StevenK: ~kubuntu-ppa
<wgrant> ppa:kubuntu-ppa/backports is quick once the cache is hot
<wgrant> When cold, the binary query took 11s. 6s of that was just finding the BPPHs, so inlining the size would have only saved 50%, even though it would also have been wrong.
<StevenK> I thought about it a bit more
<StevenK> We could solve the dupe issue by also denorming a checksum
<wgrant>    ->  HashAggregate  (cost=189318.78..189411.75 rows=9297 width=93) (actual time=11545.856..11547.182 rows=1719 loops=1)
<wgrant>          ->  Nested Loop  (cost=737.92..189225.81 rows=9297 width=93) (actual time=3700.384..11534.203 rows=2587 loops=1)
<wgrant> Not very effective
<wgrant> ~30% decrease in work
<StevenK> So the queries still need a little tuning, or there isn't much we can do?
<StevenK> Without a massive amount of design, that is.
<wgrant> There's not much we can do.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/back-to-agpl/+merge/154251
<wgrant> StevenK: most of that is the former lazr-js?
<wgrant> Yes
<wgrant> r=me
<StevenK> Bleh, I do not understand these JS failures
<wgrant> StevenK: Have you tried it locally?
<wgrant> It's possible that it makes the files too big or something
<wgrant> We've had that problem before, though it's meant to be fixed
<wgrant> Ah
<wgrant> StevenK: You have some syntax issues
<wgrant> === modified file 'lib/lp/code/javascript/branchmergeproposal.nominate.js'
<wgrant> === modified file 'lib/lp/code/javascript/branchmergeproposal.status.js'
<wgrant> === modified file 'lib/lp/registry/javascript/team.js'
<wgrant> === modified file 'lib/lp/registry/javascript/team_mailinglists.js'
<wgrant> === modified file 'lib/lp/translations/javascript/pofile.js'
<StevenK> wgrant: I've fixed all of those
<StevenK> http://pastebin.ubuntu.com/5630415/
<wgrant> StevenK: Does it work?
<StevenK> wgrant: It only fixes 3 of the 7 failures
<wgrant> StevenK: Have you examined the common files included by those broken tests?
<StevenK> Yes, they look fine to me
<adeuring> good morning
<stub_> http://pgfoundry.org/projects/emaj/
<stub_> If that is what I think it is, it might give us an easier way to reset db state after running a test.
<stub_> Not sure if it would be a win though from test speed perspective (faster doing the db reset, but inserts, updates and deletes will be slower)
<czajkowski> aloha
<czajkowski> adeuring: where is the best place to point a canonical person at to fix something in LP
<adeuring> czajkowski: the most up to date knowledge have wgrant and StevenK. But i can help perhaps too
<adeuring> or other people who retired fromLP development.
<czajkowski> mfisch: what area do you want to try and fix I guess would be the best start
<czajkowski> adeuring: aye just timezones makes it kinda hard
<adeuring> czajkowski: yeah, that's why i sugegsted "retired people" too ;)
<mfisch> hey czajkowski, I ran into a problem yesterday where I changed permissions on a script by accident, it was lost in the review because it's hidden in one line in all black text and looks like a "file header", all the reviewers missed it too
<mfisch> I was thinking this morning that permissions changes could be called out with colorization as well
<mfisch> I don't know anything about the LP code base, but I thought something like: foo.sh <red>-x</red> <green>+x</green>
<mfisch> that seemed to be simple enough to call attention
<adeuring> mfisch: so, that's about reviews? abentley might be able to help
<mfisch> yeah, about reviews
<mfisch> I'd be happy to try and fix it myself if I can get a kick-start in the right direction
<adeuring> mfisch: a very vague pointer: the directory lib/lp/code/browser
<adeuring> in the LP source code.
<adeuring> but you'd need to read how to build LP locally on dev.launchpad.net first
<mfisch> ok
<mfisch> I'll grab the code and poke around and save this for 10% time
<adeuring> mfisch: there are also some procedures, like having a pre-imp conversation (what we are beginning now, but I am not sure how far we can get, since i am not very familiar with this part of LP)
<mfisch> adeuring: ok, let me look into whether this is feasible too and then I'll talk to abentley too
<cjwatson> wouldn't that be in loggerhead?
<mfisch> still pulling the code...
<cjwatson> oh, ok, not for the review interface I guess
<abentley> mfisch: I believe you want to look at FormattersAPI.format_diff in lib/lp/app/browser/stringformatter.py
<cjwatson> general advice: reproducing it in the test suite early is generally much more fun than trying to run a local LP instance.
<cjwatson> IME.
<cjwatson> and you'll need a test anyway
<mfisch> okay good advice
<mfisch> at this rate I'll have the code by tomorrow
 * abentley is somewhat irked that the code parses the diff manually instead of using bzrlib.patches.parse_patches.
#launchpad-dev 2013-03-21
<StevenK> wgrant: Thoughts about http://pastebin.ubuntu.com/5632908/ for bug 888049 ?
<_mup_> Bug #888049: garbo-frequently logged an oops <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/888049 >
<StevenK> If you say it's good, I'm going to self-review the branch and land, but want your thoughts first.
<wgrant> StevenK: It's not ideal, but it's probably OK
<wgrant> We might want some sort of threshold
<wgrant> But this is probably OK for now
<StevenK> wgrant: Threshold? For what?
<StevenK> That warning only fires if a tuner is getting killed because the script is.
<wgrant> StevenK: Sure
<wgrant> StevenK: But we probably sort of want to warn if it legitimately doesn't finish
<wgrant> But not if it was blocked all the time
<wgrant> Anyway, this seems OK
<wgrant> for now
<StevenK> Oh, wait a moment.
<StevenK> The bug is about the looptuner warning, not the script warning.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/garbo-abort-info/+merge/154592
<wgrant> StevenK: Can you adjust the logger's level so the tests can test what they're testing?
<StevenK> wgrant: It's a nice plan, but doesn't love me much.
<StevenK> wgrant: http://pastebin.ubuntu.com/5633274/ fails
<StevenK> wgrant: The INFO lines aren't printed, but everything else for those blocks is
<wgrant> StevenK: What if you pass a custom logger into the tuner?
<StevenK> If I create a BufferLogger, set its level to INFO and then make use of loop.log.use(), it also doesn't print INFO line
<wgrant> StevenK: Even if you print the content of the buffer?
<StevenK> wgrant: http://pastebin.ubuntu.com/5633308/
<wgrant> StevenK: I would suggest overriding the tuner's log argument
<wgrant> """        Why a custom BatchNavigator is required is a great mystery and
<wgrant>         should be documented here.
<wgrant>         """
<StevenK> wgrant: http://pastebin.ubuntu.com/5633323/
<StevenK> And oh dear
<wgrant> StevenK: Have you looked at the script logging infrastructure to see how it does the level override?
<wgrant> It probably has some custom filter preventing setLevel from working
<StevenK> wgrant: I had a little bit of a dig, but not much.
<StevenK> The logger stuff is ... odd
<czajkowski> aloha
<jam> frankban: looking here: https://launchpad.net/~gophers/+related-projects there is "golxc" is that something that should stay in ~gophers or move to ~juju?
<frankban> jam: I don't know, I suggest to ask on #juju-dev
<jam> sorry frankban, I was in the wrong window. Thanks for understanding
<jam> (I was looking for Frank in #juju-dev, but didn't notice the irc window)
<frankban> :-)
<marcoceppi> Hi all, I'm having a weird issue using launchpadlib on Windows
<marcoceppi> Whenever I run the script using launchpadlib I get this SSL error http://paste.ubuntu.com/5635134/
<marcoceppi> Happens on Windows Server 2012 on EC2 and a local Windows 7 machine
<wgrant> marcoceppi: You probably need to override the CA certificate list path
<marcoceppi> wgrant sounds straight forward, would this need to happen in the code or on the machine?
<wgrant> marcoceppi: SYSTEM_CA_CERTS in lazr.restfulclient._browser
<marcoceppi> wgrant I replaced C:\Python27\Lib\site-packages\httplib2\cacerts.txt with latest from upstream without success.
<wgrant> marcoceppi: You'll need to change the path in the module that I referenced.
<marcoceppi> wgrant: thanks
<marcoceppi> wgrant: so, bascially because SYSTEM_CA_CERTS is hardcoded n lazr.resfulclient._browser it'll just never be portable beyond Ubuntu?
<wgrant> marcoceppi: There's no standard way to find a list of CA certifactes, unfortunately. A list of fallbacks could be added if someone interested did it, but that's all we have today.
<marcoceppi> wgrant I've got a small patch that'll at least make lazr.resfulclient work on non-debian/ubuntu platforms while maintaining proper ubuntu support https://code.launchpad.net/~marcoceppi/lazr.restfulclient/system-ssl-certs/+merge/154821
<wgrant> erm
<wgrant> When did httplib2 become not totally insane...
<wgrant> Huh, that actually works
<wgrant> I guess someone must have finally convinced Python people that making it extremely awkward to use HTTPS was actually a stupid idea :)
<wgrant> httplib2.CA_CERTS has only existed for about a year, it seems
<wgrant> marcoceppi: It might be worth having it the other way around
<marcoceppi> wgrant FWIW, CA_CERTS works on Windows, though I haven't tested that path directly on Ubuntu
<wgrant> See if httplib2.CA_CERTS is importable, and if it is use that. Otherwise catch the ImportError and use the hardcoded path
<marcoceppi> wgrant, ah that makes more sense. I'll update my merge request
<wgrant> marcoceppi: The path is correct on Ubuntu too; the reason we didn't use it is that it didn't exist
<wgrant> Everyone hardcoded their own paths because httplib2 was stupid :/
<wgrant> But they finally saw sense
<marcoceppi> glad they wised upppppp
<marcoceppi> wow, keyboard lag
<wgrant> Heh
<marcoceppi> wgrant merge updated
<wgrant> marcoceppi: Hm, actually, look at the NEWS entry in http://bazaar.launchpad.net/+branch/lazr.restfulclient/revision/134
<wgrant> marcoceppi: Maybe httplib2's been fixed, if it works on Windows for you?
<marcoceppi> It does work on Windows for me now, however I realize I don't have a standard cacerts.txt file. I downloaded a new one from upstream. Let me see what was bundled
<marcoceppi> *let me try what came bundled
<marcoceppi> Nope, the one that came with httplib2 that was downloaded earlier today works fine
<marcoceppi> I'm able to get responses from lp, etc. So rev 134 just needs to be reversed?
<wgrant> Which version of httplib2 are you using?
<wgrant> For safety we might want to go with what you originally had, since old Ubuntu releases might have a bad httplib2.
<marcoceppi> 0.8-py2.7
<wgrant> Ah
<wgrant> We have 0.7.2ish
<marcoceppi> Ah, so the original commit would have been better
<marcoceppi> Actually, shouldn't the import of CA_CERTS still throw an ImportError if it's not in 7.2ish
<wgrant> Er yeah
<wgrant> With the ImportError catching
<wgrant> Indeed
 * marcoceppi double checks
<wgrant> So if the existing hardcoded path exists, use that. Otherwise try to import httplib2.CA_CERTS and use that.
<wgrant> So basically what you had before, except with the CA_CERTS import guarded
<marcoceppi> wgrant cool, I'll update the merge again
<wgrant> Thanks.
<marcoceppi> So, for the catch, shouldn't it actually raise an error. Because there's no CA_CERTS, and no hardcoded path available
<marcoceppi> I feel like it'd be better to throw a more sane error there than just having the current error which is really misleading
<wgrant> Possibly. Even the ImportError would probably be more sensible than what's there now
<wgrant> So maybe don't catch the exception, but only try the import if the hardcoded path doesn't exist?
<marcoceppi> wgrant: ack
<StevenK> Which then results in people having to know why _browser is throwing an ImportError
<wgrant> StevenK: It's going to be fairly clear
<wgrant> Because the line will be 'from httplib2 import CA_CERTS' :)
<czajkowski> StevenK: morning
<StevenK> czajkowski: HAI
<marcoceppi> wgrant: rebased and resubmitted the merge request https://code.launchpad.net/~marcoceppi/lazr.restfulclient/system-ssl-certs/+merge/154826
<marcoceppi> Thanks for the help!
<wgrant> marcoceppi: looks good, but I have suggested an optional change in the MP
<wgrant> I see you've already signed the contributor agreement, so I can land this right away if you want
<marcoceppi> wgrant: I'll just implement your change, and you can feel free to land
<wgrant> Thanks, will do
<marcoceppi> wgrant: awesome, this will help move things along
<wgrant> marcoceppi: Actually, I've just had yet another thought
<wgrant> It presumably defaults to CA_CERTS internally...
<marcoceppi> presumably
<wgrant> But it wouldn't really be any cleaner to avoid overriding the default, so I guess this is fine
<wgrant> Nevermind :)
<marcoceppi> Hopefully when all ubuntu distros are at >httplib2-0.8 this whole workaround can just be dropped. At least until httplib2 does something silly again
<wgrant> All supported releases should have httplib2.CA_CERTS already, I think, but I don't want to rely on it just yet.
<wgrant> (0.7.2 was backported to -security on the basis that the old one was basically impossible to secure)
 * marcoceppi nods
<StevenK> wgrant: What was your objection about http://pastebin.ubuntu.com/5635613/ ?
<wgrant> StevenK: I don't think I saw that.
<wgrant> 'cause that's basically ideal :)
#launchpad-dev 2013-03-22
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/garbo-abort-info/+merge/154592 is updated
<wgrant> StevenK: Much better, thanks
<wgrant> r=me
<xnox> mgz: i'd like to quickly chat to you about jubany when you have a moment or two =)
<mgz> xnox: lets do it
<xnox> mgz: i've read all the README and README_DEPLOYMENTS info. Roughly, my understanding is sudo into pkg_import user, schroot into quantal, source / update path to include the magic scripts directory & run those.
<xnox> mgz: but for example, i'm not quite sure what requeue_package --full will do and whether it is "safe" to run on out of date packages.
<xnox> mgz: also I'm curious about what will need to be done at s-series opening.
<mgz> --full is not safe as such
<xnox> that was my thinking.
<mgz> as it discards all existing history, and attempts to recreate it from what's published
<xnox> that's not what I want =)
<mgz> generally packages are out of date for one of two reasons:
<mgz> * a bug or limitation of some kind in constructing the history from a certain point, often packaging error, has also been stuff like lack of support of latest .xz format changes
<mgz> * some history got rewritten after an import ran, so the importer can't reconcile what it has with what now exists
<mgz> really #2 is just a subcase of #1. but #2 is the only kind where --full might actually fix things
<xnox> I think I have a latter case, where --full might actually help. All revisions were committed by pkg-importer only, ye
<xnox> yet the latest upload by me, is not being imported.
<xnox> mgz: has anything been done after raring switched to uploads to -proposed with a copy to -release? as I bet the importer is going crazy with packages appearing & disappearing in -proposed.
<xnox> and later reappearing in -releases (as this is the course of the events usually)
<mgz> nothing has been done, and that does sound like it might break things
<cjwatson> well, *usually* they disappear in -proposed and appear in release in the same publisher cycle
<xnox> cjwatson: sure, unless is haskell and it's in -proposed for a while before it migrates, or if the package is huge and takes time to build on armhf, by which time the source package in -proposed is published already.
<cjwatson> if it's in -proposed for a while before it migrates then it hasn't disappeared from -proposed
<cjwatson> and how long it takes to build has no bearing on the relative timing of disappearance from -proposed vs. appearance in release
<xnox> ok.
<cjwatson> if that *is* the problem then it's plain bad luck on when proposed-migration finishes
<cjwatson> if it happens to finish very close to the start of the publisher then it might race in that particular way.  but it's not clear whether that is in fact the problem anyway :)
<xnox> right. I just wonder how come we have 417 packages failing with bug 714622, with some of them fairly straight forward (e.g. lp:ubuntu/libsemanage where nobody but package-importer has touched the branches)
<_mup_> Bug #714622: import fails when lp branch has been push --overwrite'n <Ubuntu Distributed Development:Confirmed for canonical-bazaar> < https://launchpad.net/bugs/714622 >
<xnox> and there was no --overwrite.
<mgz> that does sound like something else is tickling that symptom
<xnox> mgz: let me run pkg-import here locally for libsemanage to see if there are any other bugs with it, and if there aren't I'll just requeue full it.
<mgz> xnox: on the updates needed for S front, I got myself in a muddle last time, there's a thread on the udd list from October you can refer to
<mgz> in short, there are just some hardcoded values in lp:udd to change
<mgz> part of the release process should be stopping the importer before release, then enabling again after the copy-over script has finished
<xnox> mgz: https://wiki.ubuntu.com/NewReleaseCycleProcess only has "notify to restart package-import" Can you expand that check-list with other steps that are needed to be done?
<mgz> wiki/sso is being slow over letting me log in...
<mgz> xnox: done. also updated ReleaseProcess page
<xnox> mgz: cjwatson: let's see if I can do those bzr importer tasks this release.
 * xnox has access to jubany ;-)
<mgz> sounds good to me
#launchpad-dev 2014-03-17
<psivaa_> hello, We sometimes see that installing pkges from ppa's take about 20+ mins,
<psivaa_> just curious if this happens normally at peak times or is there anything special going on
#launchpad-dev 2014-03-18
<bigjools> wgrant: any known problems with mailing lists?  I've had two messages bounce recently with "retry timeout" and "retry time not reached for any host after a long failure period"
<wgrant> bigjools: Can you forward the failures?
<bigjools> wgrant: yes one moment
<bigjools> wgrant: oh fuck me, ignore them I see why
<bigjools> how embarrassing
<wgrant> bigjools: lanchpad, yeah
<bigjools> and ~@Â£$% MRU lists in email clients
#launchpad-dev 2015-03-16
<cjwatson> wgrant: git-ref-scanner bulk updateRef - do you just mean having it be a single method call for many refs?  There's no actual concise way to do a bulk update in SQL, AFAICS, unless I want to bother creating a temporary table which I think is probably overkill here
<wgrant> cjwatson: VALUES
<wgrant> cjwatson: We should have a few bulk updates around, using Update and Values storm expressions. Let me find one.
<wgrant> Oh, BulkUpdate.
<wgrant> PopulateLatestPersonSourcePackageReleaseCache
<wgrant> cjwatson: But I'd consider just having an upsert, like Python's dict.update.
<cjwatson> Oh huh, OK
<cjwatson> Thanks
<wgrant> I'm afraid you're going to have to learn a bit of non-trivial SQL.
<wgrant> Though it can probably be cargo-culted.
<cjwatson> Yeah, fine with learning it, just needed the starting point
<cjwatson> BulkUpdate seems like a reasonable interface, though, and it's not going to be a problem for the upsert to be sometimes two queries rather than one.
<wgrant> Two is still O(1) :)
<wgrant> IIRC Ian wrote that with a tonne of sqlvalues originally and I rewrote it to use the custom Storm expression. But it's only been used in that one place, so it may need a bit of adjustment to fit well for you.
<cjwatson> I suppose I could do the http://www.postgresql.org/docs/current/static/plpgsql-control-structures.html#PLPGSQL-UPSERT-EXAMPLE thing, but it seems like overkill.
<wgrant> We should have retry logic in the job infrastructure anyway, though I don't think we do now.
<wgrant> (the webapp does, though)
<cjwatson> I guess concurrent notifies could be a thing, even without any other way to create refs.
<stub> We can do upsert in python now I think, using savepoints.
<cjwatson> https://wiki.postgresql.org/wiki/UPSERT  oww my head
<stub> cjwatson: Will there potentially be concurrent connections trying to insert the same row? If not, the writable-CTE variant is easiest.
<cjwatson> It's possible, yes: two notifies might be scheduled in rapid succession due to two git pushes, and those might be run concurrently by different celery workers.
<cjwatson> (Although we could work around this by having notify check whether there's already a job and not schedule another one, which might be worthwhile if it makes this significantly easier)
<stub> mmm, so the stored procedure or savepoints...
<cjwatson> Savepoints seem cumbersome for a bulk upsert
<stub> Indeedy
<cjwatson> I think the stored procedure needs a temporary table, though, if I'm reading this right
<wgrant> cjwatson: Branch scans use postgres advisory locks to avoid two running on the same branch simultaneously.
<stub> Or do the CTE variant if you don't mind two jobs potentially conflicting and one aborting with a violation
<wgrant> It may be useful to do the same here. It's not going to totally exclude all possible conflicts, but it should fix most of them.
<stub> or yeah, advisory lock.
<cjwatson> what's the CTE variant?
<stub> The stored procedure version doesn't need a temporary table. https://dba.stackexchange.com/questions/13468/most-idiomatic-way-to-implement-upsert-in-postgresql-nowadays has a simple example of that
<stub> http://www.xzilla.net/blog/2011/Mar/Upserting-via-Writeable-CTE.html has an example of the CTE variant
<stub> CTE is the fastest approach
<wgrant> I don't see a huge problem with the easy client-side select+insert+update approach.
<stub> speed, no of queries. Depends on how bulk is bulk.
<stub> if you have a lock, it is faster doing the update first, then if now rows modified, do the insert
<cjwatson> Advisory lock plus something like https://stackoverflow.com/questions/1109061/insert-on-duplicate-update-in-postgresql/8702291#8702291 looks tolerable
<cjwatson> And maybe a retry loop around the whole thing
<cjwatson> The worst case without a retry loop AFAICS is that you might have to send another notify to get it to catch up properly.
<cjwatson> The "if no rows modified" strategy surely only works if I'm trying to upsert one row at a time
<cjwatson> (?)
<wgrant> Right, all of this is just a cache, so the worst case is we have to request the cache be refreshed again.
<wgrant> UPDATE ... RETURNING
<wgrant> You can't just use rowcount, but you can use RETURNING to work out which rows were changed.
<cjwatson> "Bulk" is unlikely to be more than 100 rows or so, usually much less.
<wgrant> However, SELECT+INSERT+UPDATE is just as correct as the CTE and probably quicker to implement now.
<stub> cjwatson: yeah, "if no rows modified" is one row at a time.
<cjwatson> Well, maybe more for lots of tags in a newly-pushed repo.
<wgrant> Mm, lots of repos have lots of tags
<cjwatson> Yeah, my linux tree here has 2648 tags
<cjwatson> Rare for most of them not to be either "insert and never change again", though.
<cjwatson> s/either //
<cjwatson> wgrant: I'd appreciate a re-review of https://code.launchpad.net/~cjwatson/launchpad/git-ref-scanner/+merge/252763, since the changes were non-trivial.
<blr> cjwatson: we have the same string -> GITOBJ.type mapping in turnip - would it be useful to have that in pygit2?
<blr> bizarre that the library doesn't provide that... might send them a PR this week, time allowing.
<blr> cjwatson: context - in your branch scanning branch.
<cjwatson> blr: Hm, possibly, but I think LP wants to have its own enumeration for anything that's in the database, really.
<cjwatson> blr: Although doing the string parsing bit wouldn't hurt, I suppose.
#launchpad-dev 2015-03-17
<wgrant> cjwatson: 2015-03-17 05:33:24,847 INFO    Outage complete. 0:00:01.254161
<wgrant> cjwatson: So you should be apply to apply your patch today.
<wgrant> (yes, I ignored the defined outage windows, because who cares, really.)
<StevenK> Wow, FDT is down to 1.2 seconds
<StevenK> Nice
<wgrant> StevenK: Turns out that DB servers that aren't powered by potatoes are handy.
<StevenK> wgrant: That sniggering is GLaDOS.
<wgrant> (though the fdt is still driven from wildcherry, as we haven't moved the code etc. yet)
<StevenK> wgrant: Ah, so wildcherry is only used for FDT and isn't actually answering queries any more?
<wgrant> StevenK: Right
<wgrant> (it would have ENOSPCed if it were still replicating, even)
<StevenK> Haha
<cjwatson> wgrant: Thanks, scheduling.
<cjwatson> wgrant: https://github.com/kevinw/pyflakes-vim/issues/66 is the main reason I avoid dict comprehensions, even though they're possible now
<cjwatson> wgrant: Do you use something other than pyflakes-vim?
 * Spads idly wonders how well it handles set comprehensions
<Spads> ah yes, I moved from pyflakes-vim to syntastic a while back, but it doesn't seem to do some of the same warnings, possibly because my environment never got straightened out fully for the new way
<cjwatson> Spads: same issue, roughly
<Spads> disappointing
<Spads> it seems the pyflakes-vim folks are arguing everyone should move to syntastic
<cjwatson> possibly I could try a homebrew pyflakes upgrade within pyflakes-vim
<cjwatson> oh, hang on, http://www.vim.org/scripts/script.php?script_id=2441
<cjwatson> claims pyflakes-vim >= 3.0 does set/dict comprehensions
<Spads> also says that was released in 2010
<cjwatson> yeah, well, pretty sure I unpacked it once and never upgraded
<cjwatson> because non-packaged software FTL
<cjwatson> \o/
<cjwatson> there we go, pyflakes-vim 3.01 fixes dict comprehensions, so I'm just four years behind
<cjwatson> (and set)
<Spads> cool!
<Spads> so the syntastic system uses some kind of vim scratch buffer or status buffer to list troubles, and I never did get the hang of navigating it
<wgrant> cjwatson: I use an ancient thing from IIRC Edwin.
<wgrant> Been meaning to upgrade, but what I have works...
<wgrant> Hm, it's been updated!
<wgrant> https://code.launchpad.net/~edwin-grubbs/+junk/canonical-vim-new
<wgrant> Fixing my main issue with it, too.
<wgrant> In fact my two top issues.
<wgrant> I have angered lazr.restful.
<wgrant> I know not how.
<wgrant> Perhaps I need to change everything to implements() rather than inherit IFAQTarget.
<wgrant> cjwatson: There is a gotcha here, but I don't quite remember the details. Do you happen to?
<wgrant> The problem is promoting IFAQTarget to a webservice entry in its own right. It kills at least some of the exposed methods from eg. IProduct.
<wgrant> I haven't run into this in years.
<wgrant> And I did not note the solution.
<cjwatson> Might it be the other way round?  Product currently does implements(IFAQTarget), but perhaps IProductSomething could inherit from it.
<cjwatson> Things like IHasBranches and IHasGitRepositories work the latter way.
<cjwatson> IHasBranches isn't relevant, I guess, but IHasGitRepositories is a webservice entry in its own right.
<blr> cjwatson: thanks for adding basenode support to the charms, that was next on the list after stackstack deployment.
#launchpad-dev 2015-03-18
<blr> cjwatson: thread was reasonably constructive actually
<blr> I am surprised.
<cjwatson> oh, the reddit one?  yeah
 * blr heads off for lunch
<blr> wgrant: so I destroyed the juju environment and re-ran the spec and turnip is successfully deployed because fuck computers.
<wgrant> blr: Heh, helpful.
<blr> no floating ip yet however
<blr> do we want to attend to that later?
<wgrant> I think it's important to get it sensibly deployable first, before investigating floating IPs.
<wgrant> So ensure it's working reliably, and not just accidentally working once.
<wgrant> Get basenode, LS, ksplice, etc. known to be working.
<blr> yep
<blr> wgrant: sudo uptrack-uname -r reporting, assuming ksplice is working.
<blr> wgrant: a few nrpe failures I need to fix
<blr> but everything appears to be deploying nicely.
<blr> only issue remaining is the ubuntu bootstrap node - suspect it will work on wendigo, but I would feel more comfortable if fetching that charm didn't fail against the local provider.
<wgrant> blr: What's the local error?
<blr> wgrant: branch permissions/NotABranch
<blr> which seems very weird
<blr> if you have a moment to test it, add this following to collect and run the spec against the local provider
<blr> wgrant: see ops
<wgrant> blr: I'm trying to speed up juju deployment, and turnip seems to run apt-get update three times for a deploy without any relations.
<wgrant> Do you have any ideas on cutting that down a bit?
<blr> wgrant: set the following in ~/.juju/environments:
<blr> enable-os-upgrade: false
<wgrant> Oh no, this is run by the charm's hooks, AFAICT.
<wgrant> Like, each time the hook fires it tries to reinstall python-virtualenv etc.
<blr> hmm
<blr> I'm not certain, perhaps I can have a look at that tomorrow.
<blr> a 30s wait between adding services and relations unfortunately didn't work.
<wgrant> Hmm.
<wgrant> I need to look at how block-storage-broker and storage work.
<wgrant> blr: Can I switch juju into some debug mode so it will tell me when it fires hooks, when hooks complete, etc.?
<wgrant> Like, I can see hook output, so I can try to guess.
<wgrant> But I can't see what it thinks it's trying to do.
<blr> I might revert that last commit - should probably be making these changed directly on wendigo and testing, just ackward working there
<blr> yep there's hook debugging via tmux
<blr> https://jujucharms.com/docs/authors-hook-debug
<blr> hmm at this point I may be doing more damage than good... will have a fresh look in the morning
<wgrant> Yeah, I've used that, but I just really want it to log when it invokes a hook, which arguments it used, and when the hook finished.
<wgrant> Not trap them, exactly.
<wgrant> Heh, it is late.
<wgrant> Night!
<wgrant> We are close.
<blr> yes! shame about the false positive with the inverted relationships hah
<blr> will get b-s-b sorted tomorrow
<wgrant> Well, that's one of the problems, at least!
<blr> night
#launchpad-dev 2015-03-19
<mpt> Well, thatâs amusing
<mpt> Launchpad e-mailed me last night to invite me to moderate a mailing list message that was sent 3 years and 3 days ago
<mpt> (reported bug 1433959)
<mup> Bug #1433959: "New mailing list message requiring approval" is years old <Launchpad itself:New> <https://launchpad.net/bugs/1433959>
<wgrant> mpt: Is there any suggestion that it wasn't actually a new message?
#launchpad-dev 2015-03-20
<sil2100> Hello!
<sil2100> Does anyone know if I could get a person's upload permissions through launchpad API somehow?
<sil2100> Since I'm afraid it's not exported to LP in any way
<wgrant> sil2100: Sure, though you can only get them on a per-archive basis. See for example the edit-acl tool in lp:ubuntu-archive-tools.
<sil2100> wgrant: excellent news then, that's exactly what I would need
<sil2100> Let me look it up
<wgrant> But if you just want to check whether someone can upload a package, use archive.checkUpload instead.
<sil2100> Thanks!
 * sil2100 likes when things are so easy
<sil2100> Yeah, checkUpload seems like the perfect thing, I can just use it when someone will perform an upload for all packages in a silo
<wgrant> Please don't try to reimplement the upload permission logic yourself. If checkUpload doesn't do exactly what you need, we should adapt it to avoid security issues.
<cjwatson> Indeed, archive.checkUpload is IIRC what I suggested in Malta when somebody asked in a CI Engine session about checking this externally.
<cjwatson> wgrant: What do you think of this layout?  http://people.canonical.com/~cjwatson/tmp/new-builders.png
<cjwatson> wgrant: My thinking is that, once scalingstack builders can dispatch both i386/amd64/armel/armhf/arm64 virt and i386/amd64 non-virt, the heading row above them can be two lines and read "Non-virtual: i386 amd64\nVirtual: i386 amd64 armel armhf arm64"
<cjwatson> wgrant: This sacrifices table sorting, but honestly that was mostly a footgun on that page anyway; "oh, I clicked somewhere random, oh god where did all the builders go, reload page"
<cjwatson> Hm, might shuffle the names right a little bit and give the icons their own column back.  They don't have much mnemonic value, but I do find they help break things up visually a bit.
<cjwatson> And I hate that sampledata has "386".  But you get the general idea.
<cjwatson> It's a bit non-semantic-web given that it's blatantly tables-for-layout, with the artificial rows.  But it's that or multiple tables, I think, and then we have to deal with keeping column widths the same.
<cjwatson> <tr class="category"> seems to style a bit nicer for this.  Updated that image
<cjwatson> Also have started on the Processor etc. changes needed
<cjwatson> But EOW now, have a good weekend all
<wgrant> cjwatson: I don't think they should be labeled as non-virtual at all.
<wgrant> Rather the x86 Processors have a flag that says they are never virtualised.
<wgrant> Er, always.
<wgrant> (but yes, that's roughly how I think the page should look. It may make sense to have filtering, but it's not critical and still much easier to scan)
<cjwatson> wgrant: to be clear, that screenshot's with the current model; I'm thinking of that as something we could do straight away, that's easier to adapt to the new model
<cjwatson> I was indeed planning to add Processor.always_build_virtualized - that seems the natural approach
<wgrant> cjwatson: Right. I'm wondering how that interacts with DAS.supports_virtualized.
<wgrant> It probably doesn't change the semantics of Processor.restricted.
<cjwatson> Why was that on DAS to begin with?
<wgrant> cjwatson: Only some distributions support PPAs, and we used to not enable PPAs for new distroseries until they were usable.
<wgrant> Distribution now has a flag that could be used to forbid PPA uploads, and new distroseries are less of an issue nowadays.
<cjwatson> Mm.  This is about builder dispatch rather than whether it's a PPA, though.
<cjwatson> (But yeah, those two used to be conflated.)
<cjwatson> BinaryPackageBuildSet._getAllowedArchitectures filters out cases where DAS.supports_virtualized and Archive.require_virtualized contradict, so perhaps it would be logically consistent to behave similarly here.
<cjwatson> That is, if the build farm is set up so that we only do virtual builder dispatch for a processor, and DAS.supports_virtualized is false, you just don't get builds.
<cjwatson> Or they never get dispatched.
<cjwatson> The _getAllowedArchitectures semantics would be that you don't get builds at all, I guess.
<wgrant> They still are conflated to some extent, so the flags are relevant for discussing how this changes things.
<wgrant> It may make sense to abolish supports_virtualized.
<wgrant> The unsupported distro filter could then use distribution.official_packages or distribution.supports_ppas.
<wgrant> The unsupported distro filter could then use distribution.official_packages or distribution.supports_ppas.
<wgrant> Er, Processor needs supports_virtualized and only_virtualized, or supports_virtualized and supports_nonvirtualized.
<wgrant> I think it makes sense to have a consistent virtualisation setting on the BuildQueues for a single Archive.
<wgrant> That is, the flag indicates whether virtualisation is *required*.
<wgrant> But for Processors with only_virtualized, virtual builders search for non-virtualised builds too.
#launchpad-dev 2017-03-20
<NikoKrause> Hello?
<NikoKrause> Is somebody active on this channel?
<cjwatson> Yes, but I'm afraid I don't know Translations well enough to be able to answer your .po review question (https://answers.launchpad.net/launchpad/+question/575853).  Perhaps wgrant knows.
#launchpad-dev 2017-03-22
<somedev> attempting to run a local instance of launchpad based on these instructions : https://dev.launchpad.net/Running
<somedev> the software starts and is available, but clicking "login/register" link produces an error : DiscoveryFailure: Error fetching XRDS document: urlopen() got an unexpected keyword argument 'cafile'<br />
<somedev> "__traceback_info__: <bound method OpenIDLogin.__call__ of <zope.browserpage.metaconfigure.OpenIDLogin object at 0x2b26efec3810>>"
<somedev> thanks in advance for any assistance, there does not seem to be any results in the bug database for this easily reproducible error condition
<cjwatson> (In case 'somedev' reads channel logs) I don't see how that could happen unless you were running a very specific revision of Launchpad, namely r18316; I fixed that bug in r18317.  If it's still occurring in some edge case, we'd need to know what revision of the Launchpad code base you're running and exactly what version of Python you're running it with.
#launchpad-dev 2018-03-22
<ral> I've either done something stupid, or have found a problem in launchpad. I reckon it's the former which is why I'm here not on bugs.l.n.
<ral> I've got a snap that is automatically built from a lp branch imported from github. I wanted to migrate that snap to a new name (not title), so I've registered the new snap and done a manual upload and have been configuring the automatic build.
<ral> I created a new snap package on lp using the existing lp git branch and used the new name. I deleted the lp snap package with the old name.
<ral> The builds for the new lp snap package are ok, except that in the build log they use the old name for some reason.
<cjwatson> Well, you could have just renamed the old one, but OK.  Can you point to the new snap?
<ral> When I click on the "Successfully built" link, I get an Oops! Error ID: OOPS-c22975b8e333068c64a6cd0030f20ba9
<ral> https://code.launchpad.net/~mosquitto-dev/+snap/mosquitto/
<cjwatson> An OOPS is definitely a bug; please file that.
<ral> Maybe so, but the snap dashboard hints strongly that renaming the snap isn't possible.
<cjwatson> In quite recently-written code, in this case.
<ral> ok, will do.
<cjwatson> You can't rename the snap in the *dashboard*, no, but it's possible to rename an LP snap object and point it at a new name.
<cjwatson> But maybe you didn't want that.
<cjwatson> So are you referring to things like: Snapping 'mosquitto-simple'
<ral> It's the snap name on snapcraft.io that I wanted to rename, not the lp one really.
<ral> Yes, mosquitto-simple is the old, just mosquitto is the new.
<cjwatson> That's entirely controlled by the tree you're building from.  You need to change that in snap/snapcraft.yaml.
<cjwatson> That should match the "Registered store package name" in LP, otherwise things will get confusing.
<ral> Oh, duh, I've spotted the stupid thing then.
<ral> I pushed the change of name in snapcraft.yml but don't think I triggered a reimport on lp.
<ral> Yep, that looks like the problem.
<cjwatson> Oh, right, I hadn't noticed that was an import.
<ral> I've submitted a bug and set my import to run. Hopefully I'll not have to report back.
<cjwatson> Thanks.  It's an easy fix, already underway.
<ral> The best kind of bug :)
<ral> The upload worked after the import and rebuild, thanks for your help.
<cjwatson> Great.  I just proposed the fix for that OOPS too.
 * cjwatson manages to reproduce the annoying occasional lp.services.daemons.tests.test_tachandler.TacTestSetupTestCase.test_couldNotListenTac failure in buildbot by strategic insertion of time.sleep
#launchpad-dev 2018-03-23
<juliank> cjwatson: Do you think it's possible to have launchpad directly provide us binary GPG public keys for PPAs to store in trusted.gpg.d? Avoids GPG on the client
<juliank> xnox: ^ here we go
<juliank> It should end in .gpg I think. I have some weird things I'm thinking about regarding TOFU repository adding. :D (list Key URI in Release file, and generalize add-apt-repository)
<xnox> the "best" thing I could come of with, to approximate that is:
<xnox> $ curl 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=<FINGERPRINT>' | gpg --no-default-keyring --import --import-options import-minimal,import-export > /etc/apt/trusted.gpg.d/<FINGERPRINT>.gpg
<xnox> but ideally we would just do a REST call to launchpad, to get a minimal, clean public key.
<cjwatson> juliank: That would be https://bugs.launchpad.net/launchpad/+bug/1667725 .  I don't think it's particularly terrible, just needs code
<mup> Bug #1667725: [feature request] make full ppa signing public key available over https <cpe-onsite> <Launchpad itself:New> <https://launchpad.net/bugs/1667725>
<cjwatson> juliank: Why the constraint on the URL?
<juliank> cjwatson: APT supports both .asc and .gpg files (armored and binary); and I'd like to know what I'm fetching. I'm thinking about extending Release files with a "link to key" field.
<cjwatson> Right, but is there any particular reason to have that as an extension-style suffix in the URL?
<cjwatson> We should clearly define which is which.  Is there a positive reason to have the binary variant?
<juliank> It does not need to be dearmored on the client side
<juliank> older apts support it
<cjwatson> OK, so you'd actually need to process that?  Fair enough
<cjwatson> In which case, is there a positive reason for LP to expose the armoured variant? :)
<juliank> I don't think so. Unless you want to provide human-consumable variants
<cjwatson> I think I care more about having less code
<juliank> :D
<juliank> cjwatson: I don't think we need to worry about the extension, I can encode that differently if I come up with some format
<juliank> (for Release file metadata)
<xnox> the argument that armored is somehow is more human readable, than binary, is dubious -> both look like jibberish; one is simply formatted to be justified.
<xnox> it would be more useful to have the .asc format; if the comments had URLs where one can get the updated new key; or which repository it belongs to.
<cjwatson> This sounds like feature creep.
<juliank> xnox: not readable, but consumable - e.g. can be copy pasted into something
#launchpad-dev 2018-03-25
<lifeless_> cjwatson: where the differences inconsequential when you did that?
<cjwatson> lifeless: Ah, sorry, I forgot about it.  Done now, and indeed they were.
