#launchpad-dev 2009-02-18
<jml> good grief
<jml> a cabal!
<joey> yes
<joey> and you are invited jml
<jml> oooh goody. I like cabals.
<barry> can i has open sauce lunchpad?
<joey> yes barry
<joey> you can!
<joey> I hear those LP devs are working on it and we could expect something by July
<barry> joey: great!  because i just got lunchbag.tv and would like to create a fork based on SCCS branches
<joey> I think I just found the first use case for tags on irc.
<joey> barry, linkshare from devpad.info? :-D
<joey> I want tag the user jml with "cabal groupie"
<joey> :-D
<barry> phew!
 * barry though you were going to tag barry as nuisance pita
<joey> I like pitas
<joey> especially with humus
<Ursinha> lol
#launchpad-dev 2010-02-22
<thumper> wgrant: I have a branch in progress that sends the email on needs review
<thumper> wgrant: but it is blocked on two others
<thumper> wgrant: to avoid the horrible edge cases
<thumper> wgrant: one of them is done
<thumper> wgrant: the one that adds a description to the MP
<thumper> wgrant: the other isn't too hard
 * thumper waves at beuno
 * beuno waves back at thumper 
<wgrant> thumper: Great.
<wgrant> thumper: Do you know about the Zopeless email thing I asked earlier?
<thumper> wgrant: didn't see your question
<wgrant> thumper: Where do Zopeless emails go in the dev config? I cannot find them anywhere.
<thumper> wgrant: haha, I remember looking for it too
<thumper> wgrant: root@localhost I think
<wgrant> thumper: Normal emails go there.
<wgrant> But not Zopeless ones.
<thumper> really?
<wgrant> AFAICT.
<wgrant> Which is perhaps not surprising, since it looks like Zopeless uses SMTP directly, and there is a zopeless/send_email config key which is set to false.
<wgrant> Hmmm.
<wgrant> How annoying.
<wgrant> AAAAAAA
 * wgrant stabs ISD in the face repeatedly.
<wgrant> Gnarnrewwfwfew
<jml> please don't.
<jml> they are very nice people.
 * ajmitch detects some rage there
<thumper> hi jml
<thumper> jml: how's pycon?
<jml> thumper, great!
<jml> thumper, I'm at the Twisted sprint now
<jml> thumper, it's exciting to be able to personally fix multiple bugs in a single day :)
<wgrant> (bug #525556, for anybody ISDish)
<thumper> jml: :)
<lifeless> jml: \o/
 * thumper wonders how often the staging scanner is running
<mwhudson> thumper: */15 i think?
<wgrant> The production scanner seems to be pretty slow at the moment.
<wgrant> (or maybe the puller)
<wgrant> beuno: Since you seem to have access, can you have a look at bug #525556?
<beuno> sure
<wgrant> It seems like it should be fairly critical.
 * beuno facepalms
<wgrant> That's what I said.
<beuno> spm, if you or any other losa is around ^
<beuno> just to be aware, I know you can't do anything about it
<wgrant> beuno: Thanks.
<beuno> wgrant, thanks for raising it, I'll make sure the appropriate people know ASAP
<wgrant> Great.
<beuno> elmo, ^
<wgrant> I'm trying to write a Storm query to return a resultset of objects ordered by a column on an optional reference.
<wgrant> ie. I have a BinaryPackageReleaseDownloadCount, which has an optional Country associated.
<wgrant> I want to order them by Country.name.
<wgrant> any hints?
<mwhudson> do a leftouterjoin on country
<mwhudson> then sort on country.name
<wgrant> yes, but how do I do that in Storm?
<mwhudson> maybe coalesced with something depending on which end of the results you want things without country to end up at
<wgrant> Hm, I guess a more manual select with a DecoratedResultSet might work.
<wgrant> http://paste.ubuntu.com/381372/ is the code at the moment, but it of course doesn't return objects with a NULL Country.
<mwhudson> wgrant: http://paste.ubuntu.com/381376/ or similar i think
<wgrant> mwhudson: Oh, yes, of course.
<wgrant> Thanks.
 * thumper settles down again
<mwhudson> thumper: hi, i have some reviews for you :-)
<thumper> mwhudson: sure, paste the links
<mwhudson> https://code.edge.launchpad.net/~mwhudson/launchpad/smarter-code-import-scheduling-bug-236973/+merge/19841
<mwhudson> thumper: https://code.edge.launchpad.net/~mwhudson/launchpad/incremental-code-imports-bug-512683/+merge/19674
 * thumper looks now
<thumper> mwhudson: are you around for a question?
<mwhudson> thumper: just about, i should stop pretty soon
<thumper> mwhudson: why does the interdiff for the incremental imports actually change anything?
<mwhudson> thumper: because the tearDown raises NotImplementerError
<thumper> mwhudson: and that does what?
<mwhudson> thumper: and this is the signal to the zope test runner to not run any more tests in that process
<lifeless> so bad
<lifeless> so very very bad
<lifeless> have I mentioned bad?
<mwhudson> thumper: it's what the functionallayer (i.e. the one that sets up the component architecture) does
<mwhudson> lifeless: hey, if bzr supported unloading plugins...
<thumper> mwhudson: so because the teardown isn't implemented, the test runner starts another process?
<mwhudson> thumper: it seems so yes
<mwhudson> thumper: i just cargo culted from layers.py
<lifeless> mwhudson: why do you need to unload plugins? [and it does to an extent]
<thumper> mwhudson: can you add a comment saying as much?
<lifeless> mwhudson: I'd really like to avoid you needing this feature. Can we discuss?
<mwhudson> lifeless: right now, no we can't
<thumper> lifeless: some of our other tests fail if the plugins are loaded
<mwhudson> thumper: ok
<thumper> lifeless: some tests need it
<thumper> lifeless: we'd like to be able to have plugins loaded for only some tests
<lifeless> is there a doc somewhere explaining this, so I don't en up playing twenty questions?
<thumper> lifeless: probably not
<lifeless> so, lets play twenty questions.
<lifeless> why do some tests fail
 * thumper shrugs
<thumper> lifeless: probably worth playing 20 questions when mwhudson is working tomorrow
<thumper> lifeless: as mwhudson is trying to finish up
<lifeless> mwhudson: can you ping me at a good time tomorrow?
<thumper> lifeless: and I won't know answers to your questions
<mwhudson> lifeless: ok
<lifeless> mwhudson: thanks
 * mwhudson EODs
<lifeless> good idea
<wgrant> thumper: Whatever happened to my branch?
<thumper> wgrant: not heard anything
 * thumper checks ec2 thingy in firefox
<thumper> wgrant: I have one instance that is terminated
<thumper> wgrant: I submitted yours and one of mine
<wgrant> So it hates my branch again :(
<thumper> wgrant: heard from one...
<wgrant> I ec2tested a branch last night and it emailed me OK.
<thumper> wgrant: check your email
<thumper> wgrant: id emails you not me
<thumper> wgrant: I double checked, and it at least started off right
<thumper> wgrant: there was a test fix issue
<thumper> wgrant: so I'm resubmitting mine
<thumper> wgrant: interesting
<thumper> wgrant: I didn't get an email from ec2 either
<thumper> wgrant: even though it submitted mine to pqm
<thumper> wgrant: I should have got one but didn't
<wgrant> I haven't had an ec2 success notification since mid-December, although I have had successful submissions since then.
<thumper> wgrant: your branch hasn't landed if that helps...
 * wgrant looks through mail server logs.
<wgrant> thumper: It doesn't look like it tried to email me.
<thumper> wgrant: sorry dude, no idea at all
<wgrant> thumper: Thanks for trying.
<wgrant> Lots of branches have been going missing lately.
<wgrant> noodles775: Morning. Do you have access to buildd-manager logs? It seems to have stopped dispatching builds about an hour ago.
<noodles775> wgrant: Yeah, but a losa will need to restart it I'd say.
 * noodles775 checks logs
<noodles775> wgrant: ouch, another job without a builder: http://pastebin.ubuntu.com/381433/
<wgrant> Gnarrrgh.
<wgrant> There are also two builders stuck (the usual two: samarium and bohrium), but that's probably unrelated.
<wgrant> Do we have no LOSAs today? I thought there was a Nagios check for that.
<noodles775> yeah, there should be... I'm just checking the bug and then I'll try to find someone.
<thumper> wgrant: they are sprinting this week in europe
<thumper> noodles775: morning
<noodles775> Hi thumper
<wgrant> thumper: Ahh.
<thumper> noodles775: I'm working on a wiki page BuildBranchToArchiveUI/InitialCut
<thumper> noodles775: nothing there yet, but I'm working on it
<noodles775> Great.
<thumper> noodles775: actually I just hit save, but I'm still editing :)
<noodles775> thumper: sure. Do you think it wasn't clear enough on the other pages? (ie. what wouldn't be included in the initial cut?)
<noodles775> wgrant: the buildd-manager should be dispatching again now (as of a min. ago).
<noodles775> wgrant: thanks for the heads-up.
<thumper> noodles775: I'm going to have a little more detail
<thumper> noodles775: ordering of work
<noodles775> thumper: ah, great.
<wgrant> noodles775: Thanks.
<thumper> noodles775: stuff broken up into ready to code chunks
<thumper> noodles775: something where I can say "hey <dev>, do this chunk" and it has what they need to know
<thumper> noodles775: at least, that's the plan :)
<noodles775> thumper: yeah, I was hoping the list of urls required could be turned into bugs for that reason, but you're right, they're a bit too coarse grained (and have interdependencies).
 * thumper nods
<adeuring> good morning
<mrevell> Morning
<danilos> bigjools, hey, good morning :)
<bigjools> hey danilos - what do you want? :)
<danilos> bigjools, ha!
<danilos> bigjools, do you know if there have been any firewall rules set up to allow access to our bzr branches from the build farm?
<danilos> bigjools, is that something build-from-recipe requires as well, or are translations special in that regard :)
<bigjools> danilos: no, BFB needs it too.  So far I have it set up on dogfood so we can test stuff, but it's not done in production.
<danilos> bigjools, ok, I am considering writing up an RT, but I am sure you can give me a few pointers on how to best word it so IS actually understands it :)
<danilos> bigjools, i.e. what machines need what privileges :)
<bigjools> danilos: you don't need it yet
<wgrant> Do we actually know that it works on dogfood?
<danilos> bigjools, heh, ok, if you say so :)
<bigjools> and you won't get it until you've testing on DF :)
<wgrant> It didn't after the first reuqest.
<bigjools> s/testing/tested/
<bigjools> wgrant: huh?
<danilos> bigjools, ok, makes a lot of sense, then I'll get on with the next phase to actually do it, and then we can see what happens
<bigjools> danilos: in all seriousness, dogfood is the way to go here so that we can establish that stuff works before altering production
<wgrant> bigjools: I remember that codehosting access from DF was supposedly set up during the sprint, but it didn't work.
<bigjools> wgrant: I don't remember us even trying at the sprint!
<danilos> bigjools, of course, I am not disagreeing
<bigjools> it was all on your laptop
<bigjools> danilos: I'm always right.... 'cept when I'm wrong.
<wgrant> bigjools: We didn't try the code, but we tried branch access in anticipation.
<bigjools> wgrant: we only opened up access for the builders IIRC
<wgrant> bigjools: Isn't that what we're discussing?
<bigjools> wgrant: I just tried and it works fine
<wgrant> OK, great.
<bigjools> I even remember testing it at the time it was done
<wgrant> Hm. I remember you had to make a second request, but didn't remember if that was resolved.
<bigjools> I don't remember that but then I don't remember at lot of things at my age :)
<wgrant> Haha.
<wgrant> BjornT: Can you please re-review that branch know that we've clarified that ILFA isn't exportable?
<wgrant> s/know/now/
<stub> Anyone used the with statement in a doctest yet?
<stub> I think I need to inject some flags somewhere
<wgrant> stub: Why not just use the import?
<stub> Because it doesn't work
<wgrant> Ah. Good rationale.
<stub> I think it is http://www.python.org/dev/peps/pep-0264/, but no english translation in the __future__ or doctest bits of the reference manual.
<BjornT> wgrant: sure, done.
<wgrant> bigjools: No violent objections to https://code.edge.launchpad.net/~wgrant/launchpad/export-das-chroot/+merge/19759?
<BjornT> stub: have you tried putting the __future__ import in the test harness (i.e., the file that sets up the doctest)?
<BjornT> stub: or maybe in the test runner, although that's a very big hammer...
<danilos> bigjools, btw, does BFB already have the code which checks the branch out?
<stub> BjornT: Looks like more than that. I need to do the import at the top of the harness, and then pass a globs={'with_statement': with_statement} when constructing the suite.
<stub> icky
<wgrant> danilos: It's done by bzr-builder, so it's probably not useful for your purposes.
<danilos> wgrant, right, so it's outside the LP tree?
<wgrant> danilos: Yes, and inside a script that takes a recipe.
<bigjools> wgrant: one sec
<bigjools> danilos: it's done in the builder code
<danilos> bigjools, wgrant: ok, excuse me for not being certain where to look now, but... is there some code in LP I could generalize and reuse for translations?
<wgrant> danilos: If you really want to look at it, try sourcecode/bzr-builder. But it's probably just as easy for you to do it yourself.
<bigjools> danilos: nothing in LP - as wgrant says it's done in bzr-builder
<danilos> bigjools, wgrant: ok, thanks, that clarifies it
<bigjools> wgrant: MP is ok, I commented on it
<stub> bzr pish
<wgrant> Is the branch scanner broken?
<wgrant> I pushed a new rev 7 minutes ago.
<wgrant> It has still not scanned.
<wgrant> BjornT/bigjools: Branch finally scanned -- can one of you please check the new revs and land it?
<bigjools> sure
<wgrant> bigjools: Thanks. Do you have a moment to talk about those test changes you requested last week?
<bigjools> wgrant: sure
<wgrant> bigjools: So, regarding your last comment on https://code.edge.launchpad.net/~wgrant/launchpad/p-u-sprb-handling/+merge/19380...
<wgrant> The security is there so the storeUploadLog test works.
<wgrant> It just needs some declaration of writeability. It does not care what.
<wgrant> The tests will fail if the security is removed, and I'm not sure it can be usefully tested otherwise.
<bigjools> wgrant: ah ok, so there is a test :)  In that case, please document that fact on the test.
 * bigjools is about to order a new Thinkpad
<bigjools> would love to get one w/Ubuntu pre-installed but the stuff at LinuxEmporium is all old models :(
<wgrant> :(
<bigjools> windows tax here I come
 * wgrant removes the security to work out which test it was.
<deryck> Morning, all.
<adiroiban> danilos: can you give me a hint for how can I make a TranslationMessage to be the current translation? Basically I have this test http://paste.ubuntu.com/381520/, and the contributors for makeTranslationMessages are listed in the POFile details page, but not on the language global statistics
<bigjools> hi deryck
<adiroiban> jtv: hi. when you have time, can you please test the fix for bug #127171 ?
<mup> Bug #127171: Rosetta experts not allowed to "Change translators" <qa-needstesting> <Launchpad Translations:Fix Committed by adiroiban> <https://launchpad.net/bugs/127171>
<danilos> adiroiban, looking at your test
<danilos> adiroiban, fwiw, a lot of that data is maintained through triggers or nightly scripts, so that might be part of the problem
<danilos> adiroiban, so, list of top contributors is based on karma, and karma is updated only with a nightly script
<adiroiban> hm... how do you suggest I should write that test?
<stub> You need to populate the relevant Karma and KarmaCache tables. I don't think there are factory methods for that though :-/
<jtv> adiroiban: hang on, some high-priority stuff going on :)
<jtv> adiroiban: ah, I guess danilos took care of it while I was otp  :)
<danilos> adiroiban, I am not exactly sure what would be the best way to do it
<danilos> adiroiban, you can just add a method which updates karma for one particular user, or you can fake it by adding it directly
<bigjools> wgrant: hmmm, I am thinking about your security zcml
<bigjools> The upload processor is zopeless and doesn't need security, so what is falling over exactly?
<wgrant> bigjools: The test that I added the comment to. It appears to run in that semi-secure mode which doesn't care about users.
<bigjools> wgrant: your diff has got merge conflict markers in it
<bigjools> in the spr model
<wgrant> bigjools: Argh.
 * wgrant fixes.
<bigjools> wgrant: which test?  the comment was on the zcml itself
<bigjools> testSetsBuildAndState?
<wgrant> bigjools: I added a comment at both ends.
<wgrant> Yes, IIRC.
<bigjools> ah I see it
<wgrant> The one that tests UploadProcessor
<bigjools> buried in other comments :)
<wgrant> Indeed.
<bigjools> hmmm
<wgrant> Remember that Soyuz's Build has the same sort of thing, with little explanation. It would be nice to work out both.
<bigjools> hmm yes
<bigjools> it's very odd
<bigjools> ah I know I think
<wgrant> Hm?
<bigjools> those Build attributes get reset in the webapp when you retry the build
<bigjools> which explains why you need lp.edit
<bigjools> so I am still confused as to wehy SPRB needs it :/
<wgrant> But shouldn't they be set from within retry()?
<wgrant> Once you're inside the object, security is irrelevant.
<bigjools> I didn't think it was but now you have me thinking
<bigjools> since IBuild is exported, lp.edit still makes sense anyway
<wgrant> For a dangerous value of sense that has caused security issues in the past!
<bigjools> yeah, we have a bad record there :(
<wgrant> I mean that particular line of ZCML has been problematic.
<wgrant> bigjools: That conflict's resolved.
<wgrant> Argh, wrong push location though.
<bigjools> heh
<wgrant> bigjools: That looks a bit healthier.
<bigjools> wgrant: approved
<bigjools> I can land it, it's db-devel right?
<wgrant> bigjools: It's db-devel.
<wgrant> bigjools: But watch out: thumper tried to land the prereq earlier, and the instance disappeared silently.
<bigjools> :/
<bigjools> what happened to the pre-req?
<wgrant> Hm?
<bigjools> it landed, or is it still in limbo?
<wgrant> bigjools: It's not landed. The instance died, and we're not sure why.
<bigjools> ok I'll try merging it and having another go
<wgrant> So you might not want to land this headlessly.
<bigjools> I'll run tests on my box
<wgrant> Ah, good point.
<bigjools> I don't need no steenkin ec2
<bigjools> I have a fork whore
<wgrant> bigjools: It hasn't exploded yet?
<bigjools> wgrant: there's a test failure
<wgrant> Hrmph.
<bigjools> ProgrammingError: relation "sourcepackagerecipebuildupload" does not exist
<bigjools> for a few tests
<bigjools> test_DeleteExpiredBlobs test_cronscript test_PersonPruner and others - non soyuz, which is odd
<wgrant> Oh, damn.
<wgrant> I know about and have fixed that.
<bigjools> lib/canonical/librarian/tests/librarian-report.txt now
<bigjools> if you have a fix I'll quit this test run
<wgrant> I forgot to pump one of the DB patch changes through. Oops.
<wgrant> Sorry.
<bigjools> ok
<wgrant> (we can't drop SPRBU entirely due to replication limitations, so it has been moved to another schema, which confuses things that look for foreign keys referencing Person)
<wgrant> bigjools: r8934 has that DB change pumped through (it's already reviewed) and pushed.
<wgrant> But I guess it'll take about 10 minutes to appear on the public branch.
<bigjools> okay
<bigjools> wgrant: on your lp:~wgrant/launchpad/sprb-new-model-columns branch?
<wgrant> Stuart Metcalfe doesn't exist on Freenode, does he?
<wgrant> bigjools: The missing piece started on https://code.edge.launchpad.net/~wgrant/launchpad/sprbu-columns-to-sprb, but is now in p-u-sprb-handling
<bigjools> ok got it
<bigjools> ok off with another test run
<wgrant> Thanks.
<bigjools> doubt you'll be up in 3h huh? :)
<wgrant> I really hope not.
<wgrant> 3am while sick is not advisable.
<bigjools> hope you're feeling better tomorrow
<bigjools> wgrant: still failing
<wgrant> bigjools: Is this line in database/schema/patch-2207-32-0.sql?
<wgrant> ALTER TABLE SourcePackageRecipeBuildUpload DROP CONSTRAINT sourcepackagerecipebuildupload_registrant_fkey;
<bigjools> wgrant: yep
<bigjools> I suspect that when I run "make check" it deletes the table when doing make schema
<bigjools> anyway, I need food, back in a bit
<stub> If dropping the table is too painful, comment out that line and open a bug.
<wgrant> See, the tests pass here.
<wgrant> Ah, no, there's one failure.
 * wgrant pokes around.
 * wgrant drops the rest of the foreign keys.
<wgrant> stub: No objections to dropping the other three foreign keys?
<stub> No - all the foreign key constraints on that table should die.
<wgrant> bigjools: OK, r8935 pushed, with all the foreign keys dropped. All the failures you've mentioned are fixed by that.
<bigjools> wgrant: ok trying again
<deryck> adeuring, gmb, intellectronica, kfogel -- just a reminder, update the board before the standup.
<kfogel> deryck: oh, thanks!
<deryck> I did some QA and moved things there.
<kfogel> deryck: I saw.  We don't have to hit "save" or anything, right?  Just move the post-it and it's done?
<deryck> kfogel, right.  Just move the card.
<kfogel> deryck: *nod*
<maxb> ooi, why does the LaunchpadDatabaseRevision table contains multiple rows, instead of just the current revision?
<kfogel> intellectronica: know a good example of a usage of a standard Zope form somewhere?  (I'm in bug #515584)
<mup> Bug #515584: BugsPatchesView:batchedPatchTasks() should use a Zope form, instead of validating form values manually <story-patch-report> <Launchpad Bugs:Confirmed for kfogel> <https://launchpad.net/bugs/515584>
<wgrant> maxb: I don't know for sure, but it's probably something to do with the fact that a DB patch can live in a branch for ages.
<intellectronica> kfogel: it's probably worth fixing that together with fixing it for the search form
<maxb> ah - so in theory they might not actually be sequential?
<wgrant> maxb: Exactly.
<intellectronica> kfogel: there's a bug for that. let me try and find that
<wgrant> maxb: In practice, too.
<intellectronica> kfogel: https://bugs.edge.launchpad.net/malone/+bug/322130
<mup> Bug #322130: Convert IHasBugs.searchTask(order_by) to use a real enumeration <api> <tech-debt> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/322130>
<kfogel> intellectronica: thank!
 * kfogel opens it up
<intellectronica> kfogel: it's quite a lot of work, i think
<intellectronica> but it would be even more work if we end up doing it twice
<kfogel> intellectronica: it's a lot of work?  Just because of all the callers?
<wgrant> bigjools: If it's still looking good, I will head to bed.
<kfogel> intellectronica: if it's just a bunch of caller updates, well, Emacs and I can do that together.  If you think it's a lot of work in terms of inherent complexity, then I'm not sure lumping the two bugs together is a good idea.
<bigjools> wgrant: yes, no issues so far
<bigjools> sleep well
<intellectronica> kfogel: yes. also you need to arrange for backwards compatibility
<wgrant> bigjools: Excellent. Thanks.
<kfogel> intellectronica: backwards compatibility?  This is an API issue?
<kfogel> mmmmm
<intellectronica> kfogel: maybe the API, also the search form, since many users save URLs for later
<kfogel> intellectronica: okay, then I think we should keep these two bugs separate.  I'm all for fixing 322130, but 515584 is its own separate piece of tech debt, and doesn't have any compat issues.
<kfogel> intellectronica: *nod* I see.
<intellectronica> kfogel: ok, maybe you're right and it's worth fixing this first, don't know
<kfogel> intellectronica: This *is* a fair amount of work, then.  Not against it, but I don't think we should just randomly start on it vs some other thing -- it needs to be triaged, and we probably want team input, or at least deryck's.  There might be more presing stuff to do.
<kfogel> intellectronica: which we can discuss on the call now :-)
<intellectronica> kfogel: indeed :)
<kfogel> intellectronica: in the meantime... know a good example of a zope form? :-)
<al-maisan> are thunderbird 3 and enigmail conflicting packages these days
<al-maisan> enigmail de-installs thunderbird 3 and vice versa
<al-maisan> ECHAN
<al-maisan> sorry
<deryck> kfogel, looking at QA page, can you update bug 513608, to link branch, status, milestone, etc.  I think this landed?
<mup> Bug #513608: community-contributions.py script should use Launchpad to determine who is not a Canonical employee <Launchpad Foundations:Confirmed for kfogel> <https://launchpad.net/bugs/513608>
<kfogel> deryck: yup
<kfogel> deryck: oh, no -- wrong bug number there.  513608 is still open, it's just mentioned in an XXX comment now.  the fix that landed is for another bug.  I'll fix it all up.
<kfogel> (the qa page I mean)
<deryck> kfogel, excellent.  Thanks, sir.  I think I marked something as OK, so sorry about that.
<kfogel> deryck: well, the change is indeed OK, it just needs to be associated with the right bug number.  no worries.
<deryck> ok, cool
<kfogel> deryck: so, it kind of disturbs me that if I do a bugs search across all projects, and put "community-contributions.py" in the search box, it shows no matches -- even though I know of bugs where that exact string is in the *summary* of the bug.
<kfogel> deryck: is this expected?
<deryck> by "search across all projects" what exactly do you mean?
<kfogel> deryck: https://bugs.launchpad.net/, put that string in the search box, make sure "All projects" is checked, hit the button.
<kfogel> deryck: it also turns up nothing if I do "One project:" and put launchpad or launchpad-projects as the project name.
<deryck> kfogel, seems like a bug.  This should work.  Maybe something with the way full text queries handle the ".py"
<kfogel> deryck: urgk, sigh.  I'll push stack once more :-) and ensure there's a bug report for this.
<kfogel> may already be one, we'll see
<deryck> kfogel, note that search on "launchpad" shouldn't work but search on "launchpad-project" should, since that is the project group.  yes, this is confusing, I know.
<kfogel> deryck: I knew the latter should work.  I can never remember what the former is supposed to represent.
<deryck> kfogel, it's a thing to catch bugs that no one knows where to file, *I think*. :-)
<kfogel> deryck: oic :-)
<kfogel> deryck: so, it turns out that in addition to that change (the one that is not for 513608), I also need to land https://code.edge.launchpad.net/~kfogel/launchpad/cc-script-new-world, which is approved.  To land that, I need to find the bug number(s) relevant to it.  On the way to finding those, I may be filing a bug about how I can't find them.  There.  I think that's everything at the current stack level.
<kfogel> :-)
<deryck> heh
<deryck> Not too bad for a monday, kfogel ;)
<kfogel> deryck: oy
<kfogel> deryck: okay!  that's in the queue (https://pqm.launchpad.net/).  No need to respond.  I'm just feeling the work-at-home-cabin-fever this morning and so I'm yammering what I do at you.
<kfogel> :-)
<kfogel> deryck: are bug searches supposed to be case-insensitive?
<deryck> kfogel, I'm sure they are supposed to be, but I can't confirm for certain yet.  Never checked on that yet.
<kfogel> deryck: I'm assuming they should be too.  Doesn't matter for the purposes of the bug consolidation I'm doing right now of some search failure bugs, but I was just curious.
<adiroiban> gmb: hi. any news about the ever disappering branch ?
<gmb> adiroiban: No, it's running on ec2 at the moment and hasn't died yet.
<gmb> adiroiban: I'll let you know how it goes.
<adiroiban> thanks
<kfogel> deryck: okay, it's all in bug #29713 now, which I think maybe should be high priority (search is supposed to work reliably!)
<mup> Bug #29713: bug search fails to find results despite exact search string being in bug titles <Launchpad Bugs:Triaged by stub> <https://launchpad.net/bugs/29713>
<deryck> kfogel, yeah, jml is beating the search drum too. :-)
<kfogel> deryck: oh, I wonder if he knows of any dup bugs, then.  jml?  ^^
<jml> kfogel, I don't.
<kfogel> jml: ok, thx
<noodles775> I've got an ec2 instance that's hung during windmill tests... is this a known issue? http://pastebin.ubuntu.com/381681/
<noodles775> BjornT: ^^ ?
<sinzui> jtv: thanks for doing the triage
<jtv> sinzui: chr
<sinzui> jtv: I am thankful none-the-less
<jtv> very gracious
<jtv> sinzui: here, have a free desktop background made of 9th-century ceiling design: http://people.canonical.com/~jtv/DSCF0570.JPG
<jtv> sinzui: took that in the Dome of Charles the Great at Aachen yesterday
<noodles775> Wow...
<jtv> it was pretty stunning, yes
<jtv> also the bust of the big man himself
<sinzui> jtv: thanks.
<kfogel> adeuring: looking for an example of standard, best-practices Zope form usage (for bug #515584) in our bugs code.  Do you know a good place?
<mup> Bug #515584: BugsPatchesView:batchedPatchTasks() should use a Zope form, instead of validating form values manually <story-patch-report> <Launchpad Bugs:Confirmed for kfogel> <https://launchpad.net/bugs/515584>
<adeuring> kfogel: not out of the box...
<kfogel> adeuring: np.  If you spot anything, or you know of some good docs on this, let me know.  I'll figure it out eventually.
<kfogel> .oO (he said, optimistically)
<mramirez> can help me configure my mail server launchpad
<mramirez> help mail
 * adeuring is somewhat confused today. I contenated mentally at first "standard, best"  to "bastard"...
<adeuring> kfogel: the best approach is to search for a simple from, but, in our case, one that does not use POST but GET
<kfogel> adeuring: I guess what I'm really asking is, how will I know a good example when I see one?
<kfogel> I'm not sure how to recognize the standard/best practice.
<adeuring> kfogel: when it is simple ;)
<kfogel> adeuring: It's a zen afternoon for you, I see.
<adeuring> kfogel: ;) let me try to find something
<kfogel> adeuring: btw, how long are you on today?  (reason: bad cabin fever, really need to just go outside and walk to the store and get a muffin, get some sunshine)
<kfogel> if you'll be around for a bit, I'll do that and come back
<adeuring> kfogel: an hour or so, i guess, the I'll leave for one or two hour; after that (around 9pm my time) I'll be back
<kfogel> adeuring: you're coming back at 9pm??
<adeuring> kfogel: yes
<kfogel> adeuring: ok; I'll be out for a few, then back here.  If you happen to dig up a great example using an enum and standard zope form stuff, paste it to me.  If not, no worries; I'm grepping around for method="get" etc, and I'm sure I'll find the right thing eventually.
<maxb> mbarnett: If you happen to be around, what are your thoughts on the launchpad-dependencies Slony-I thread?
<sinzui> bac: I updated bug 523891 with my findings. Do you think my proposal is sane?
<mup> Bug #523891: +needs-packaging includes packages that has upstream links <Launchpad Registry:Triaged by sinzui> <https://launchpad.net/bugs/523891>
<bac> sinzui: yes, what you propose does make sense...i think
<sinzui> bac: My fix may also fix bug 512831. I have tried to fix the gtk+2.0 packaging link, but I cannot. Every time I try to edit Lucid's link, I get Karmic's, so I set the same object to the same state. It is not possible to correct the the data from the dsp/sp pages.
<mup> Bug #512831: Packaging records not editable <package-link> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/512831>
<sinzui> I just confirmed I can fix the lucid/+source/gtk+2.0 by creating the link from the upstream project. I think I will make my goal of week 3 to close the last of the packaging-link bugs
<mwhudson> good morning
<kfogel> mwhudson: good morning
<thumper> morning
<jpds> thumper: Morning.
<kfogel> adeuring: I'm noticing a problem with +patches view, but it might just be an artifact of my local dev instance.  After patching database/sampledata/current-dev.sql with the db diff I've been using for weeks to test +patches view, and then running 'make schema', I visited https://bugs.launchpad.dev/patches-view-test/+patches.  There were no patch tasks listed, though!  There should be six or seven of them -- the ones in my custom samp
<kfogel> ledata.  Puzzled, I created a new bug on the 'patches-view-test' product, attached a patch to it, and refreshed the view.  Now that bug alone shows up, but the old bugs in my sampledata still don't.  I tried adding a new comment on one of those old bugs, to see if maybe that would cause a refresh in the 'latest_patch_uploaded' column, but it didn't cause the bug to appear in the view.
<adeuring> kfogel: that sounds really odd. Do you see the bugs missing on the +patches view on other bug pages for the test project?
<kfogel> adeuring: yes, they are missing on other entities
<kfogel> adeuring: assuming by "test project" you meant my test instance
<adeuring> kfogel: I mean that thingy (IProduct, IIRC) that you created first, and where you then added the bugs for
<kfogel> adeuring: btw, my db sampledata patch is here: http://paste.ubuntu.com/381809/
<kfogel> adeuring: so when you say "+patches view on other bug pages for the test project", what does that mean?  The product ('patches-view-test') has only one bugs page....
<adeuring> kfogel: OK, let me look. As a guess without lookng: if the test bugs are missing on the other bug related pages for the test project as well, I'd guess that the ID of the test project changed
<adeuring> soryy...
<adeuring> I mean: Do you see the bugs on the http://bugs.launchpad.dev/testproject , for example?
<kfogel> adeuring: this isn't actually holding me up on bug #515584 work, since I can just create a bit of new data easily, but ... it's worrying, obviously :-).
<mup> Bug #515584: BugsPatchesView:batchedPatchTasks() should use a Zope form, instead of validating form values manually <story-patch-report> <Launchpad Bugs:Confirmed for kfogel> <https://launchpad.net/bugs/515584>
<kfogel> adeuring: oh, yes, the bugs are there on bugs.launchpad.dev/patches-view-test, sorry, I should have said that.
<adeuring> odd... let me look at the sampledata
<kfogel> But the bugfilters stats portlet on that page says "0 bugs with patches", and going to ".../+patches" shows no patches.
<adeuring> kfogel: Trying to apply you patch to trunk r10326, I get the message "Hunk #1 FAILED at 1377", Did you get perhaps something similar?
<kfogel> adeuring: no, I did not.  You're using db-devel, right?
<kfogel> adeuring: I branched my branch from an up-to-date db-devel this morning, AFAIK
<adeuring> kfogel: argh, no... let me try again
<kfogel> adeuring: :-)
<kfogel> adeuring: hey, hmm, how would I do a python "if ... else ..." in a tal:content tag using the "python" magic word?  I can't use any indentation, obviously...
<adeuring> kfogel: use a browser class mehod?
<adeuring> ...method...
<kfogel> adeuring: oh, I see :-)
<kfogel> adeuring: sigh.  ok.
<adeuring> ;)
<kfogel> adeuring: .oO (But why can't I just implement all of Launchpad in this .pt file?)
<adeuring> kfogel: but... what is your use case? perhaops there are other options
<kfogel> adeuring: in the menu that offers sort orders, the actual orders we offer sometimes begin with "-" and sometimes don't.  But what we show do the user should never show the "-".  E.g., although it might be "-importance" behind the scenes, it should display to the user as "importance" in the dropdown.  So I was going to test the first char and do "value[1:]" if it was "-" ...
<kfogel> you see why it's tempting to  just take care of that in the .pt file
<adeuring> kfogel: yes. But I think the "technical" values (with or without a leading '-') differ anyway from what we display to the user. the "technical" bvalues are all lower-case, while the displayed text might be capitalized
<kfogel> adeuring: hmmmm, ok.  I'll run with that.  I think the patch I first show you (soon) will be very much in need of review.  but that's okay :-).
<adeuring> kfogel: I think I know why your old attachments/patches don't show up: We have meanwhile a cached column "latest patch uploaded" or somesuch in the tabkle bugs. And this column is normally updated by DB triggers. But your test data is probably "too old" to be "aware" of the triggers...
<adeuring> kfogel: but there is a simple way to fix this, let me looks
<kfogel> adeuring: thanks
<adeuring> kfogel: in the file database/schema/patch-2207-29-0.sql, there is an UPDATE command. Just run this command via psql.
<kfogel> adeuring: wow!  thank you.
<adeuring> kfogel: welcome :) But I am the one to blame to for all the mess with this column ;)
<kfogel> adeuring: I think what I'll do after I run that is regenerate my db sampledata diff
<kfogel> that old one's getting stale
<adeuring> kfogel: rigth, makes sense. a classical case of bit rot?
<thumper> jml: ping
<kfogel> adeuring: ezzatly
<jml> thumper, hi
<thumper> jml: what was the state of your factory no-commit branch?
<thumper> jml: landed, or work needed?
<jml> thumper, huge chunks of work needed.
<jml> thumper, it didn't even make it all the way through ec2 test -- it crashed it or hanged it or something
<jml> thumper, all I did was go through factory and delete all of the calls to commit
<kfogel> adeuring: ok, rtfm'ing but I think you know the answer quickly: what user and db do I invoke psql as?
<kfogel> adeuring: (I've never had to do it locally before for Launchpad)
<adeuring> kfogel: "psql -d launchpad_dev" .
<adeuring> you don't need to specify a user
<kfogel> adeuring: thanks
<kfogel> adeuring: that update command worked, thank you
<adeuring> kfogel: welcome. As I said, I am trhe one to blame for all this mess. background: the +patches view timed out for ubuntu, so I added this cached column bug.lastest_patch_uploaded .
<kfogel> adeuring: well, you're not to blame -- ubuntu is!
<poolie> hi there kfogel
<adeuring> yeah, too many bugs ;)
<poolie> how are the bugs?
<kfogel> poolie: hey, there.
<kfogel> poolie: we're down to the last couple of bugs in launchpad.  Pretty cool.  I guess we might all be able to go home after today.
<poolie> good to hear :)
<kfogel> poolie: Oh, wait -- we already are home.  And there are 2000 bugs, not 2.  My bad.
<kfogel> adeuring: hunh, I just notice (now that I have all my old data back!) that the batch size for patch tasks is, like, 5, instead of 50.  While that's useful for development, I'm worried that we built it into the code somewhere (temporarily for testing) and forgot to take it out.  What controls the batch size again?
<kfogel> oh
<kfogel> adeuring: it's a personal pref, isn't it?
 * kfogel sees
<adeuring> kfogel: do you perhaps use a bookmarked URL?
<kfogel> adeuring: nope
<adeuring> kfogel: but the page https://staging.launchpad.net/ubuntu/+patches shows more than 5 bugs
<kfogel> adeuring: not for me :-)
<kfogel> adeuring: sorry, i mean not in my launchpad.dev instance
<kfogel> adeuring: I wonder if we have a lower batch size in general for .dev instances, just for easier testing?
<adeuring> kfogel: odd. are you sure that you don't use a parameter like batchsize=5 or so?
<adeuring> kfogel: could be, I am not sure...
<kfogel> adeuring: if I do, it's not in the URL
<kfogel> adeuring: https://code.edge.launchpad.net/~kfogel/launchpad/515584-use-zope-form
<kfogel> adeuring: would love early review from you on the route I'm taking
<kfogel> adeuring: AFAICT, what I'm doing isn't more "Zope-y" than any other way, it just reduces some code duplication.
<adeuring> kfogel: I'll look
<kfogel> adeuring: thank you
<adeuring> kfogel: you can create a vocabulary with the right options and display that as a drop-down list. Tha's done more or less automatically by LPFormView
<adeuring> IOW, you don't need patchTaskOrderings()
<adeuring> kfogel: and for the form view, you need and interface class which defines the values you want to display onn the form
<kfogel> adeuring: aaaaah
<kfogel> adeuring: much clearer now, thank you
<kfogel> adeuring: (I don't think this work was wasted, though, as it still consolidated stuff outside the .pt file.  Now it's just  matter of moving code around.)
 * maxb is unconvinced by mwhudson's docstring nargery
<maxb> Does "Calculate the appropriate value for the BZR_PLUGIN_PATH environment." feel right to anyone?
<mwhudson> oh oops
<mwhudson> at least it follows the coding standard now :-)
<kfogel> jml: hey, something's come up, can I move our tomorrow call to Wednesday?  (Or, hmm, it's not on cal for tomorrow anyway, so maybe it's cancelled this week anyway?)
<jml> kfogel, it's cancelled this week, I'm away.
<kfogel> jml: *nod*
<thumper> jml: did you end up talking to james_w about multi-distroseries recipes?
<kfogel> adeuring: when does trunk freeze?  Wednesday?
<thumper> kfogel: Friday normally
<kfogel> thumper: oh, that's not so bad, okay
<james_w> thumper: not yet
<jml> thumper, no.
<thumper> james_w: so... what's the story with that then?
<james_w> it's not clear yet
<james_w> there will be some recipe format work needed to support it if that is the way we want to fix it
<james_w> otherwise we could do it soyuz side
<thumper> james_w: in which case, I'm going to say it is all go with what we have
<thumper> james_w: so a recipe is going to have a base branch and an associated distroseries
<james_w> thumper: I think you may be making a decision that makes it harder to provide the user experience that you want, but I can understand the desire to move forwards
<thumper> james_w: well...
<thumper> james_w: lets design for what we want then
<thumper> james_w: if we really want multi distroseries recipes
<thumper> james_w: then lets work out what we need to make that happen, and JFDI
<james_w> yes
<thumper> james_w: do you know what needs to happen?  I don't
<james_w> it's not me that is driving that requirement
<thumper> who is?
<james_w> someone on the LP side
<james_w> thumper: no, I haven't thought it through yet.
<thumper> ok
<james_w> I would like to do so, but the water is a little over my head right now
<james_w> it would only be a few hours work to design something and implement it though
<thumper> james_w: ok... can I do anything to help?
<james_w> either schedule something to force the issue, or lighten the load elsewhere to give me the time to do it
<thumper> james_w: who do I need to talk to about your schedule and load?
<james_w> thumper: that's not particularly clear right now
<poolie> hi james_w
<james_w> hi poolie
<poolie> i'd like to catch up sometime
<poolie> realize it's late right now
<james_w> well, I am here
<thumper> james_w: :-) however that doesn't help me help you
<james_w> thumper: I realise that :-)
 * thumper reboots server
<james_w> thumper: we don't really have a manager right now, in the conventional sense at least. I guess it is split between Hugh and Robbie.
 * rockstar reboots
 * mwhudson heads out for an early lunch -- unproductive morning
<lifeless> mwhudson: when you get back
<lifeless> mwhudson: lets talk
<jelmer> wgrant, hi
<wgrant> jelmer: Hi.
<jelmer> wgrant: you have launchpad working on lucid right?
<wgrant> jelmer: Yes.
<jelmer> wgrant: what did you need to do to get it working?
<wgrant> And I tried to run the test suite overnight, but it died early on for un-Lucid reasons.
<wgrant> jelmer: Install an old launchpad-dependencies (since the new one needs slony for postgres 8.3 -- that's about to be reverted)
<wgrant> You will also need to install python-egenix-mx{datetime,tools} from ppa:launchpad, or better still merge the versions in there so they're up to date.
<wgrant> That's about it.
<wgrant> Also, Canonical's non-Launchpad webapp teams are utter failures at security. Grrr.
<lifeless> wgrant: heh
<jelmer> wgrant, thanks
<lifeless> I'm not entirely sure why the login.ubuntu.com became a different team
<wgrant> lifeless: Not just that, but yes.
<jelmer> wgrant, is that documented somewhere yet?
<wgrant> jelmer: Which?
<jelmer> how to use lp on lucid?
<wgrant> No, since the two blockers are bugs that are easily fixed by anybody with upload privileges to ppa:launchpad.
<wgrant> ie. anybody in ~launchpad or maxb.
<jelmer> hmm
<jelmer> wgrant: so I guess I should fix it rather than document it on the wiki?
<wgrant> jelmer: Definitely.
<wgrant> Don't fix lp-dependencies yourself for now, since that's a bit harder, but the two outdated packages are just trivial merges.
<mwhudson> lifeless: hi
<lifeless> mwhudson: hai
<lifeless> mwhudson: IRC or voice. either is fine for me - but this did sound a little thorny.
<mwhudson> lifeless: so plugin unloading/layers are bad etc
<mwhudson> lifeless: i think irc will do, to start at least
<lifeless> ok
<lifeless> so plugin unloading is 'unregister + delete from sys.modules'
<lifeless> what interests me is why you need it
<lifeless> layers are just a poor implementation of testresources; we don't need to talk about that immediately :)
<mwhudson> so there's a specific problem -- probably really a launchpad bug -- that some tests fail when bzr-git is loaded
<lifeless> is there a bug showing the failure?
<lifeless> or a pastebin or something I can see?
<mwhudson> (it's because LaunchpadTransport rejects filenames at one level apart from '.bzr' and 'backup.bzr' and so on, and bzr-git probes for some files)
<lifeless> mwhudson: this suggests to me that bzr-git on the smart server may cause explosions?
<mwhudson> however, there is a more general issue that we don't want bzr-git to be loaded when we don't expect it to be, i guess mostly because bad things could happen in the branch puller
<mwhudson> (mirrored branches and imported branches should be made more similar, yes, but not by accident)
<mwhudson> lifeless: it's possible, yes
<lifeless> mwhudson: so only loading a plugin when you want it is fairly easy: don't import it :>
<mwhudson> lifeless: sure, that part works fine
<lifeless> mwhudson: but that doesn't help with the test failures
 * mwhudson rummages for the failure
<mwhudson> lifeless: http://pastebin.ubuntu.com/381912/
<mwhudson> there are many things one could do to make this failure go away
<mwhudson> you could fix the codehosting vfs to not explode on attempts to _read_ disallowed filenames
<mwhudson> (which we should probably do anyway)
<mwhudson> or you could unregister the git formats
<lifeless> oh looks; test tools :)
<lifeless> so
<mwhudson> or we could run all tests that use bzr-git in subprocesses i guess
<lifeless> I think you shouldn't blow up internally on read attempts to unexpected files: NoSuchFile is more appropriate
<lifeless> 'these are not the files you are looking for', I mean. That, or bzr-git should be catching the permission denied error (by making it subclass something it catches)
<lifeless> mmm
<lifeless> * change the vfs
<lifeless> * catch the error in bzr-git
<lifeless> * use subunit to run the git using tests
<lifeless> * use a resource to load git and unload it for tests that need it
<lifeless> the last one is pretty easy
<mwhudson> if it is, that must be the better option
<mwhudson> the first two are probably good ideas but only solve the specific problem
<lifeless> It seems like a workaround to me - is there evidence for or against having othe specific problems?
<mwhudson> we probably wouldn't need the resource, it's only one testcase class (i think)
<lifeless> you'll probably need to reach around inside bzrlib guts in the first place, but its not hard.
<lifeless> and we can dress it up later.
<lifeless> I suggest encapsulating it as follows:
<maxb> wgrant, jelmer: It's not even a merge, just a no-change rebuild. Also, it's not required
<lifeless> change bzr-git's __init__.py to not register stuff directly, rather to have a 'register()' function, which is called at the end of the module.
<mwhudson> lifeless: i guess the only other specific problem i can think of is the branch puller
<wgrant> maxb: Not required?
<lifeless> and a deregister() function, which undoes - however ugly it is - what register did.
<mwhudson> lifeless: but we're into 'unknown unknowns' on the cheney scale here
<lifeless> make them both idempotent
<mwhudson> (or was it rumsfeld?)
<lifeless> mwhudson: I can write this for you in an hour or so, ifyou need
<maxb> wgrant: the egenix thing is only a build-dep, not a dep
<wgrant> maxb: It causes me problems here if I don't have a 2.5-supporting one.
<lifeless> mwhudson: your test case would then do 'import bzrlib.plugins.git; bzrlib.plugins.git.register()' in setUp, and deregister in tearDown
 * wgrant finds the import.
<mwhudson> lifeless: that would work
<maxb> wgrant: oh! Well we should add it to lp-deps then
<lifeless> put this conversation in a bug on bzr-git; assign to me
<wgrant> maxb: psycopg2 needs it.
<maxb> the only reason I've been lazy about updating it is that aptitude wasn't whining at me
<wgrant> maxb: It depends on it, but not a specific version.
<mwhudson> lifeless: to generalise slightly, you could say that the next level down of problem is that it's hard to be confident of what's going to happen when you call "Branch.open" on a URL you don't entirely trust
<lifeless> I don't think this is the best option, but it has clear limits which making things work with bzr-git being loaded doesn't.
<lifeless> mwhudson: uhm, I don't get what that means
 * maxb runs 'lpnochange egenix-mx-base' ..... I should put that script somewhere :-)
<mwhudson> lifeless: if you set up a branch reference at some url that points to http://bazaar.lp.internal/$id_of_private_branch and asked for a mirror of the branch
<lifeless> mwhudson: thats what the open hooks are for, no ?
<mwhudson> if the puller wasn't careful, it would suck the private branch into public area
<mwhudson> in the bzr-git case it would just be a performance screw up
<lifeless> mwhudson: and similarly for git, I guess.
<mwhudson> mmm, maybe the open hooks are enough these days
<lifeless> mwhudson: so, I think that 'check policy' and 'plugin installed' are orthogonal.
<lifeless> mwhudson: we should make sure you can enforce whatever policy is needed, but that should be entirely different from 'is bzr-svn installed and loaded'.
<maxb> wgrant: I can't find the import, where is it?
<mwhudson> lifeless: that certainly sounds like a worthy goal
<wgrant> maxb: Probably _psycopg.so
<lifeless> mwhudson: so, with respect to bzr-git, opening a random url will 'try all the things that can open branches'
<lifeless> mwhudson: if you want a policy of 'we only support X types when we open a url' - file a bug on bzr, explain why you want it, and we can figure out how to do it.
<wgrant> maxb: That does have an mx.DateTime reference.
<wgrant> maxb: I would upgrade and see what breaks, but I have a test run going at the moment.
<mwhudson> lifeless: if you know what the formats are precisely, that's easy enough already
<mwhudson> i guess having an open hook that rejected non BzrBranch subclasses would do well enough for us
#launchpad-dev 2010-02-23
<lifeless> well you might want to be sure that this happens before the probe
<lifeless> for instance
<lifeless> but I cam think of things that will make you cry
<mwhudson> hm yes, that would be a bit backwards
<mwhudson> lifeless: i think i've run out of tears in this area, but try me
<lifeless> for instance, I can setup a bzr:// server serving a git branch over VFS only
<lifeless> which will cause your local obejct to be a RemoteBzrBranch
<mwhudson> ah true
<lifeless> but the conversion to happen on your machine, not mine
<lifeless> ditto svn
<lifeless> so this is a little complex and worth thinking about some more.
<lifeless> I think we should get rid of your new-process need in tests in the first instance.
<lifeless> and fix the vfs/git after that, to get rid of the short term hack.
<lifeless> for the 'deployed services need to be robust in xyz case' stuff we've been discussing, thats a longer term topic, right?
<maxb> wgrant: No, I believe you. I'm adjusting lp-deps now.
<wgrant> maxb: Thanks. Going to drop the slony deps while you're there?
<mwhudson> lifeless: yes, sure
<lifeless> ok
<lifeless> well I'll volunteer to do the bzr-git change
<lifeless> ifyou make a bug for it
<wgrant> maxb: Please also depend on python2.5-egenix-mxtools.
<wgrant> maxb: It contains the mx package.
<mwhudson> lifeless: of course, using an open hook to limit the urls we access leads to a vaguely similar concern
<mwhudson> as uninstalling hooks isn't support either, aiui
<lifeless> mwhudson: hooks are reset by the test framework
<mwhudson> ah ok
<lifeless> you start in a new test with no hooks installed at all
<lifeless> calling self.run_bzr installed the 'find command hooks'
<lifeless> thats about it
<wgrant> 2
<wgrant> Gah.
<mwhudson> lifeless: ok, i filed a few bugs, https://bugs.edge.launchpad.net/bzr-git/+bug/526133 is the bzr-git one
<mup> Bug #526133: need to be able to unload bzr-git <Bazaar Git Plugin:New> <https://launchpad.net/bugs/526133>
<poolie> kfogel: this is also true for the help homepage
<maxb> wgrant: gah, oops. failed to see that and dputted already
<kfogel> poolie: hmrm.  I wonder if there's a policy here, and if so, where it's documented.
<wgrant> maxb: Nyeh. It's not really important.
<wgrant> It'll just break in reasonably obscure ways if the PPA gets outdated again.
<maxb> heh
<maxb> it's annoying that 'import psycopg2' doesn't break
<poolie> and you can't change that either, karl?
<maxb> oh, yes it does, if you use the right version of python
<wgrant> maxb: Yeah.
<wgrant> maxb: Have you tried a full test run on Lucid?
<wgrant> I've one running now, and there are a couple of the usual Soyuz failures.
<wgrant> But otherwise it's looking good.
<mwhudson> staging is down :(
<mwhudson> and sysadmins are in europe
<wgrant> mwhudson: It should be back in 15 minutes, no>?
<wgrant> Maybe 10.
<mwhudson> i've not caught it up yet today
<mwhudson> admittedly i haven't tried all that hard
<wgrant> I suspect it went for an update at 23:44Z
<mwhudson> oh perhaps, i guess db-stable got updated around then
<maxb> wgrant: I tried, and it was all hideous. I gave up and decided to lower my sights to python2.6 on karmic to start with
<wgrant> I sort of wish it didn't pull.
<maxb> Except then things broke more, and I spent some time fixing python2.5 on karmic. Got that branch landed today
<wgrant> maxb: That bzr plugins thing?
<maxb> ye
<maxb> s
<wgrant> Only one failure so far, and that's just extra hash type on apt-ftparchive output.
<wgrant> So this is looking pretty good.
<wgrant> Much better than Karmic was, at least.
<wgrant> mwhudson: There aren't going to be any more recipe model changes, are there?
<mwhudson> wgrant: not completely sure
<mwhudson> wgrant: nothing fundamental aiui, apart from adding derived recipes
<mwhudson> wgrant: there might be changes to do with multi-distroseries
<mwhudson> yay staging is back
<wgrant> Since the backend in trunk now works, it seems almost feasible to add API exports.
<wgrant> mwhudson: Early!
<maxb> wgrant: "so far" :-)
 * mwhudson finds an old no-codeimportworker-db-access branch, merges 2000 revs of trunk into it
<maxb> impressive
<poolie> thumper: hi, quick call?
<thumper> poolie: yeah, sure
 * maxb waits for the publisher :-/
<maxb> jelmer: Once the publisher gets around to it, the launchpad PPA should be fully installable on Lucid again
<wgrant> maxb: I suppose you had to wait a cycle for the Lucid one to publish, then copy to other series, then wait for it to publish again?
<maxb> no, I'm waiting for my second upload of lp-deps :-/
<wgrant> Ah.
<wgrant> I really should look at killing process-accepted.py for the normal case.
<wgrant> Since it's only needed for binaries with custom uploads.
<wgrant> Anything without custom uploads can be realised immediately on upload, allowing a copy as soon as it's built.
<wgrant> We already skip it for sources.
<maxb> btw, I have a fix locally for importfascist on py2.6, your testrun will probably run into it at some point
<wgrant> maxb: I'm running it on 2.5, not 2.6 yet.
<maxb> oh I see, the inverse to me, I tried 2.6/karmic
<wgrant> I figure that we will need it running on Lucid in a month or so, so it probably deserves a full test run.
<maxb> hrm, installing lp-deps has now started me a slony daemon :-/
<wgrant> maxb: You saw the talk about that on the list?
<maxb> yes - and: ah, no, it claimed it did, but it was lying to me
<wgrant> Heh.
<wgrant> mwhudson: There is a way to tell ec2test to take a standard Ubuntu AMI, Launchpadify it and run the test suite, isn't there?
<mwhudson> wgrant: yess
<mwhudson> wgrant: however the code in trunk assumes you can log in as root, not ubuntu
<wgrant> I wondered if that was the case.
<wgrant> But I'm sure I used it on a Karmic daily. Maybe I hacked it.
<mwhudson> wgrant: lp:~mwhudson/launchpad/ec-log-in-as-ubuntu should work with a newer image
<mwhudson> i should look at getting that merged i guess
<wgrant> Probably.
<wgrant> Official images are good.
 * mwhudson writes it on a list
 * thumper afk for a few minutes collecting the girls
<thumper> anyone else not getting emails from EC2?
<thumper> it seems that I'm not getting any
<wgrant> I received a failure of my own a couple of days ago.
<wgrant> I tried to ec2test something this morning, but the instance hung before it had finished booting.
<thumper> mwhudson: launchpad-code has no New bugs :)
<thumper> wgrant: :(
<mwhudson> thumper: congrats
<wgrant> (btw, the successor to the branch you tried to land yesterday landed over night)
<mwhudson> thumper: i'm getting them
<thumper> hmm
 * thumper checks spam box
<mwhudson> bugger
<thumper> nope, not there
<wgrant> thumper: So you're reproducibly not getting emails from it?
<thumper> wgrant: haven't had an email from ec2 in several days
<thumper> wgrant: can you send an email to me at tim.penhey@canonical.com?
<thumper> wgrant: it could be being caught there
 * thumper taps fingers
<thumper> parent teacher interviews in 12 minutes
<mwhudson> thumper: the incremental kernel import failed :( https://code.staging.launchpad.net/~kiko/linux/2.6.31
<thumper> mwhudson: just saw the email
<thumper> mwhudson: perhaps ordering on the revisions it is choosing to import?
<wgrant> thumper: Sent.
<thumper> mwhudson: just a wild arse guess
<thumper> wgrant: ta
<thumper> wgrant: the email from mwhudson was addressed to that too and arrived fine
<mwhudson> thumper: well, the import list we're slicing into is topo sorted, so that shouldn't be a problem
<thumper> wgrant: got your email :(
<thumper> wgrant: so it isn't that
<thumper> it seems ec2 hates me
 * thumper stopping for a few hours,
<thumper> back later
<mwhudson> thumper: i guess run ec2 non-headless with --postmortem and log in and poke around after the tests are done
<mwhudson> gararar i'd forgotten how annoying working with the internal xmlrpc server was
 * mwhudson EODs
 * wgrant has never had the pleasure.
<mwhudson> the main reason it sucks is that requests are not authenticated
<mwhudson> so either you have to weaken permissions or use removeSecurityProxy all over
<mwhudson> oh and faults are a crummy way to report errors
<wgrant> Oh, nice.
<wgrant> Yeah, I know that much from buildd work.
<mwhudson> i should look (again) at using the PermissiveSecurityPolicy for xmlrpc server requests i guess...
<wgrant> That seems like a really good idea.
<mwhudson> oh guess what
<mwhudson> the security policy is global, not per-request
<mwhudson> i think
<wgrant> Ah ha ha.
<mwhudson> mmmmmmmmm
<mwhudson> maybe not actually
<mwhudson> aaaa
 * mwhudson stares into the zope vortex
<wgrant> Nooooooooo.
 * wgrant throws xx-resetpassword-of-sso-account.txt out the window.
<mwhudson> well, that seems to not be really true
<mwhudson> but still overriding the policy per request does seem quite hard
<wgrant> Is update-sourcecode broken for anyone else? It can't find bzrlib.branch.
<wgrant> Ah, probably because it's running with 2.5, and it's meant to use the system's bzr.
<jelmer> mwhudson: ping
<lifeless> wgrant: system bzr should be installed for all python versions
<wgrant> lifeless: Yes, and Python 2.5 doesn't exist.
<wgrant> Lucid hates it and wants it to die.
<wgrant> Odd that it's only broken recently, though.
<adeuring> good morning
<wgrant> bigjools: Thanks for getting that landed.
<bigjools> wgrant: np
<wgrant> bigjools: Has somebody poked you about the eaten binary?
<bigjools> umm no
<wgrant> bigjools: create-resources in lucid was eaten today by the arch-indep reverted override bug.
<wgrant> It wants to be revived.
<bigjools> which bug is that?
<wgrant> If you take an arch-indep binary, override it, then override it back within one publishing cycle, it gets eaten.
<bigjools> oh that
<wgrant> It's very difficult to fix, so it hasn't been yet.
<wgrant> (I think somebody should just fix change-override.py to die if it's going to get into that situation, really)
<bigjools> we could record overrides as an intermediate step and refuse further overrides
<wgrant> We could either reject further conflicting overrides, or just mutate the existing Pending publishing record.
<wgrant> Although I don't trust the publisher to be sufficiently respectful that that would be safe.
<bigjools> it needs time for analysis and given the frequency of occurrence it's not exactly a priority for us, unfortunately
<wgrant> Indeed.
<bigjools> I am deep in a refactor for process-accepted
<bigjools> converting it to a LaunchpadScript sounded a good idea at the time
<wgrant> Ah.
<wgrant> I didn't think it was otherwise much work.
<wgrant> Are you going with per-copy-archive selection, or just all copy archives?
<bigjools> it should not be but I seem to be suffering from pebcak
<bigjools> all
<wgrant> OK. So it was about a four line change :P
<bigjools> yes, it should have been ....
<bigjools> but see if you can find any existing tests that I can add a 4 line change to
<bigjools> the testing is shambolic
<bigjools> which is why I am refactoring
<wgrant> Ha ha ha.
<bigjools> it'll make testing easier
<bigjools> Popen on a script is not a test, it's a sledgehammer
<wgrant> I'd like to discuss PPA download stats at some point when you're free. I got something reasonably working and tested over the weekend.
<bigjools> oh awesome
<wgrant> Only for binaries, since it is completely unclear how index counts will work.
<bigjools> ok let me finish this code first
<wgrant> Certainly.
<deryck> Morning, everyone.
<bigjools> howdy deryck
<adeuring> what happened to the bug heat icons? I don't see anything about bug heat in trunk or in db-devel...
<wgrant> adeuring: It looks quite present in both to me.
<deryck> adeuring, what do you mean?  You can't find how the icons are used in code?  or on LP itself?
<adeuring> deryck: no, I don't see anything in the browser....
<adeuring> Or am I looking at the wrong pages?
<intellectronica> adeuring: maybe that's the same problem joey had the other day, with icons not displaying?
<adeuring> like bugs.launchpad.dev/ubuntu
<adeuring> intellectronica: may be, I don't know about joey's problem...
<wgrant> The hot bugs list doesn't show them.
<wgrant> Normal bug listings and bug pages do, though.
 * adeuring is completely confused...
<intellectronica> adeuring: try any normal bug listing, they should be there
<deryck> adeuring, yeah, the actual hot bugs list doesn't have them.  I think it should, though we've had some disagreement about this.
<deryck> intellectronica, what do you think about this ^^?  i.e. adding them to the hot bugs list?
<intellectronica> deryck: the problem is that for any project with a reasonable amount of bugs, all the bugs on the hot bugs list will show four flames
<intellectronica> and with no new information, shame about the space
<intellectronica> then again, there's an argument from consistency
<wgrant> Does the hot bugs lists have a purpose?
<adeuring> consufison reolved...
<deryck> intellectronica, yeah, I agree with the problem statement.  But it is a confusing UI, since they appear everywhere else.
<deryck> wgrant, show the 10 hottest bugs.
<wgrant> deryck: Why?
<wgrant> deryck: What does that achieve that showing +bugs ordered by hotness doesn't?
<wgrant> Apart from requiring an extra page load to get more than 10 bugs.
<deryck> wgrant, as an "overview" bugs page it shows you an overview of the hottest bugs.
<intellectronica> wgrant: i think it's only really useful for users who aren't heavily involved in the project they're viewing. i almost never look at this list.
<deryck> wgrant, everything on that page is a link to something else, no?
<wgrant> deryck: How is https://bugs.edge.launchpad.net/ubuntu more useful than https://bugs.edge.launchpad.net/ubuntu/+bugs?orderby=-heat?
<deryck> wgrant, I'm not arguing it is. :-)  But in "theory" there is room for more info on the top-level page, should we want to add it.  i.e. a "Recent books" top 10, etc.
<wgrant> deryck: Possibly.
<wgrant> But this suffers from the same problem as lots of other features.
<wgrant> it gets added in a completely useless fashion.
<wgrant> Angers users.
<wgrant> And then maybe eventually gets fixed a year later.
<wgrant> (see also bug heat)
<deryck> wgrant, first, I don't get the sense anyone is angry about this or that it's useless.  And second, we're not done, and I assure you, we won't leave this work until people are happy.  Can we make *every* Ubuntu dev or LP user happy?  No.
<intellectronica> wgrant: one way in which it is more useful is that it renders faster
<wgrant> intellectronica: I considered that.
<wgrant> deryck: It irritates everybody on my team.
<wgrant> because it means an extra click to get where they want to go.
<intellectronica> wgrant: i find it surprising that people on your team even go to that page. i never do, and i was under the impression that most serious users don't.
<wgrant> A feature looks a lot less bad if it's deployed fantastic rather than useless and fixed gradually later.
<wgrant> intellectronica: The Bugs tab and breadcrumb go there.
<deryck> wgrant, and we're trying to fix this.  BjornT is working on a solution to make this possible.  Right now, with monthly releases and incremental work in small branches, it's actually hard to do this.
<intellectronica> wgrant: we're actually moving towards doing more work separated from edge before landing a feature
<wgrant> That would be very good.
<wgrant> Bug heat had the potential to be a pretty exciting feature. But it was rolled out too early, and is now just a set of four gray unexplained flames wasting space. Even when it becomes awesome later, it will not have the same effect.
<deryck> wgrant, will have the effect of "wow, bug heat!"  No.  Will it be a nice, useful feature?  yes.  So what's the harm?
<deryck> wgrant, (I'm being a bit sarcastic to make my point.  I agree we could do better at this.  But I think you're being a bit hyperbolic and unfair about he harm here.)
<intellectronica> wgrant: to some extent there's agreement about that, which is why a lot of effort is being put now into providing the infrastructure for working like that, but i also have a lot to say in favour of working in the open, and letting features evolve with input from users
<deryck> s/he harm/the harm/
<wgrant> intellectronica: Deploying it to edge is fine and great for that.
<wgrant> deryck: Quite a few people really don't like Launchpad very much.
<wgrant> Impressions like that do matter, sadly.
<intellectronica> quite a few people don't like coffee too
<deryck> wgrant, I agree we could make a better impression.  We're trying to fix this.
<wgrant> So, I've just had a look at MergeWorkflowDraft. And it indeed looks like a much more flexible setup.
* salgado changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.02 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | Channel logs: http://irclogs.ubuntu
<james_w> is it intentional for a single file in lazr.enum to be LGPL with (or later), when the rest of lazr seems to just be LGPL?
<kfogel> james_w: seems unlikely to be intentional, unless that file has special provenance...
<james_w> http://bazaar.launchpad.net/~lazr-developers/lazr.enum/trunk/annotate/head:/src/lazr/enum/_enum.py
<james_w> no other declared copyright holders and no statement of code taken from elsewhere
<kfogel> james_w: looks completely like an accident to me.  Check with barry maybe?
<james_w> it looks like leonardr started lazr.enum
<kfogel> james_w: oh, maybe leonardr would be better
<leonardr> james_w: i'm looking
<leonardr> james_w: i believe that code was refactored out of launchpad
<wgrant> jtv: Shouldn't start-soyuz.sh be Makefile targets, and the result of make-ubuntu-sane.py be added to the dev sampledata?
<jtv> hi wgrant!  The latter is exactly what I'm looking at now.
<jtv> The big question is how many tests it'll break.
<adiroiban> sinzui: hi. do you remember the form next_url bug when you suggest using HTTP_REFERER ? The is a problem when the form is changing the object name and so the URL is changed. I have this code but I am not happy with it http://paste.ubuntu.com/382280/ ?
<leonardr> kfogel: flacoste is the final authority but i think that's just an oversight and it can be changed
<wgrant> None, because they don't use the dev sampledata.
<sinzui> hmm
<wgrant> They use the other sampledata.
<adiroiban> do you know a better solution?
<jtv> wgrant: gah!  Of course.
<leonardr> kfogel: see the README. it still says "Enumerated types are used primarily in two distinct places in the Launchpad code"
 * sinzui looks
<jtv> wgrant: my brain doesn't seem to be working this week
<flacoste> kfogel, leonardr, james_w: it's an error, everything should be LGPLv3 only (no or later)
<kfogel> flacoste: thx
<wgrant> jtv: I've not been maintaining the script since I don't use the sampledata any more.
<jtv> wgrant: why is that?
<kfogel> james_w: feel free to fix up the README as leonardr points out too :-).
<sinzui> adiroiban: take a look at .lp.registry.browser.product.ProductReviewLicenseView.next_url
<wgrant> jtv: Too much bad idea in Ubuntu.
<jtv> wgrant: EPARSE
<james_w> kfogel: I wasn't planning to fix the license statement :-)
<wgrant> jtv: bad data
<jtv> wgrant: so what do you use?  Just dogfood?
<adiroiban> sinzui: but you can not change the product name from ProductReviewLicenseView
<adiroiban> so the form action is not causing any URL change
<sinzui> adiroiban: sorry, I misunderstood your issue
<wgrant> jtv: I have a similar script which uses the factory to populate an empty DB with celebrities and Ubuntu stuff.
<sinzui> adiroiban: yes, that is not pretty code, but at a glance, it looks correct.
<adiroiban> sinzui: yes. is is ugly. the correct one is here http://paste.ubuntu.com/382283/
<adiroiban> sinzui: I can not think at a prettier solution
<adiroiban> :(
<jtv> wgrant: that sounds interesting... though I don't want to get into a full-scale sample data rewrite, of course.  :)
<wgrant> jtv: No. That sounds messy.
<wgrant> I think you should be able to get away with just running that script and making newsampledata.
<sinzui> adiroiban: there is an old bug in launchpad-foundations about broken next_urls  caused by a rename. Launchpad does not handle the situation well in some circumstances. Most engineers never experience the problem because most objects cannot be renamed.
<jtv> wgrant: yes, that's what I was thinking too.
<wgrant> Although deleting all the primary publications probably isn't advisable, since it will break other parts of LP.
<wgrant> (eg. bug filing on packages)
<adiroiban> sinzui: hm... is this bug 34228 ?
<mup> Bug #34228: Implement a "teleportation" or "jump-to" feature <feature> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/34228>
<jtv> wgrant: it's really only just a status change, isn't it?
<wgrant> jtv: Yes, but that might be sufficient to break things.
<jtv> wgrant: I'd prefer to delete the old ones and set up some new, entirely up-to-date ones.
<wgrant> And it will certainly be sufficient to render parts of the UI not easily testable.
<jtv> The catch is, I have no idea what's required.
<wgrant> So, the reason I delete them is that they reference librarian files without data.
<jtv> wgrant: can we provide those?
<wgrant> jtv: I suppose so.
<wgrant> That would let us have a reasonable set of default publications for the UI, and also be able to actually publish the archive if required.
<bigjools> one day, death to sampledata apart from major celebs
<sinzui> adiroiban: no, the one am thinking of contains a step by step scenario of changing a person or product name and the user gets a 404.
<wgrant> bigjools: I've already declared death to sampledata except for the bits of my script that I elect to run... it works pretty well.
<adiroiban> sinzui: i tried searching launchpad-foundations after ârenameâ or ânext_urlâ ... but no luch
<adiroiban> luck
<wgrant> No 5-year-old impossibly broken objects!
<bigjools> heh
<bigjools> some of the original sample data is.... special
<jtv> wgrant: so... what would we need to provide?
<wgrant> jtv: A tarball of the library files for the published sources and binaries.
<deryck> adeuring, there are some issues with your icon branch, I believe.  hold on submitting to ec2 til I check something.
<wgrant> But it may just be easier to tell people to delete everything if they want to do anything Soyuzy.
<adeuring> deryck: ok. What are the problems?
<jtv> wgrant: I'm such a non-Soyuz person that what you just said doesn't mean much to me.
<deryck> adeuring, there were new color icons as well and the icons were prepared to use a single image for each degree of heat.
<deryck> adeuring, and these should be done via sprites, too.
<adeuring> deryck: Ah, i see.
<wgrant> jtv: So, publish-distro.py will whine and crash if there are published packages for which the corresponding librarian files are missing.
<deryck> adeuring, I'll update the MP with comments and info.  Sorry to not have caught you earlier about these.
<adeuring> I thought that the five images were just another way to show how things should look
<jtv> wgrant: what needs to be in those files?
<deryck> adeuring, no, those are actually the images to use.
<adeuring> ok
<wgrant> jtv: Source and binary packages.
<jtv> wgrant: how painful would it be to get those into the Librarian out of the box?
<wgrant> jtv: Out of the box sounds branch-bloating. But providing a tarball that can be extracted into /var/tmp/fatsam for interested would be easy enough.
<jtv> wgrant: true
<jtv> wgrant: but is that better than having make-ubuntu-sane.py in utilities?
<wgrant> jtv: I do not know.
<wgrant> One-size-fits-all sample data cannot really work.
<deryck> adeuring, MP is updated now.  Sorry to send you back to work. :-)
<adeuring> np
<jtv> wgrant: I was hoping it might, but I don't want to make it the goal of the month either.  :)
 * wgrant mumbles something about 2am, and sleeps.
<jtv> wgrant: I was wondering about that...  in fact that's why I didn't ask you about it today  :)
<wgrant> Heh.
<jtv> good night, and thanks.  :)
<wgrant> np
<deryck> adeuring, so the alt and title attributes are required for heat icons and you note.  Sorry for my mistake...
<deryck> adeuring, alt can be something like "3 out of 4 heat flames" I think.  I'm open to better suggestions there.
<deryck> adeuring, and the title attribute is bug 513219
<mup> Bug #513219: flames have no tooltip <story-bug-heat> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/513219>
<adeuring> deryck: OK, that works. But I though it would make sense to use something "super hot bug", "very hot bug" etc
<deryck> adeuring, yeah, that works well too.
<deryck> adeuring, and you could fix bug 513219 the same time as this one. :-)
<mup> Bug #513219: flames have no tooltip <story-bug-heat> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/513219>
<adeuring> deryck: yure. But again, any suggestion what to display tehre ;)?
<deryck> adeuring, I think this could be the heat number itself for now.  Something like: "Heat: 420"
<adeuring> deryck: thnkas, that's a nice idea!
<deryck> adeuring, when I finish my branch, we can include the context max_heat, so "Heat: 420 out of 10,000"
<adeuring> ok
<deryck> adeuring, I think the tooltip could include "base on X subscribers, X dupes, X affected users" (or something similar)  But I worry that gets noisy.
<deryck> s/base/based
<adeuring> deryck: yeah, let's keep it simple ;)
<deryck> adeuring, agreed :-)
<deryck> some have asked for this, though.
<kfogel> adeuring: ready.  You might want to have my branch at hand (lp:~kfogel/launchpad/515584-use-zope-form) while we're talking.
 * adeuring is pulling the branch-...
<adeuring> kfogel: shall i call you?
<kfogel> adeuring: sure.  skype?
<adeuring> kfogel: yes
<kfogel> adeuring: lib/lp/bugs/browser/bugtarget.py
<kfogel> adeuring: class BugsPatchesSortFormView(LaunchpadFormView):
<kfogel> """Browser view class for sorting bugtasks with patch attachments."""
<kfogel> lib/canonical/launchpad/webapp/launchpadform.py
<kfogel> intellectronica: ayt?
<intellectronica> kfogel: what does ayt mean?
<intellectronica> and in what language?
<intellectronica> are you there?
<kfogel> intellectronica: "are you there", yes.  And the real question is: do you have time for a quick call about lp:~kfogel/launchpad/515584-use-zope-form? :-)
<intellectronica> kfogel: sure
<intellectronica> kfogel: give me 5 minutes, though, to get skype going
<kfogel> intellectronica: sure, ping me when ready
<intellectronica> kfogel: ready when you are
<kfogel> intellectronica: ok, one sec to gt headphones on
<kfogel> intellectronica: dailing you
<kfogel> lp:~kfogel/launchpad/515584-use-zope-form
<kfogel> http://paste.ubuntu.com/382345/
<kfogel> intellectronica: search for this in the file: def patchTaskOrderings(self)
<kfogel> lib/lp/bugs/templates/bugtarget-patches.pt
<kfogel> lib/lp/answers/interfaces/faqcollection.py:class FAQSort(EnumeratedType):
<kfogel> intellectronica: I'm looking in the code for an example of a form that works the way you described.  If I grep for "(EnumeratedType" there are about 20 hits throughout the code (http://paste.ubuntu.com/382372/).  But I don't see new variables being created with those enumerated types.  Instead, the enum classes (or values in them) are used directly.  For example, in class IBranchListingFilter, see this line:
<kfogel>     sort_by = Choice(
<kfogel>         title=_('ordered by'), vocabulary=BranchListingSort,
<kfogel>         default=BranchListingSort.LIFECYCLE)
<kfogel> intellectronica: "BranchListingSort" is a class that inherits from EnumeratedType.
<intellectronica> kfogel: yes, that's the way it's used
<kfogel> intellectronica: ok
<intellectronica> kfogel: EnumeratedType isa zope vocabulary
<kfogel> intellectronica: *nod*
<kfogel> okay, intellectronica -- please sanity check my writeup here (I mainly want to make sure that the example I chose is a textbook example of what we were talking about): http://paste.ubuntu.com/382378/
<intellectronica> kfogel: yes, that's perfect
<kfogel> intellectronica: thanks!
<kfogel> deryck: ping for 5-10 min phone chat when you have a chance
<kfogel> re 515584 and some general prioritization questions
<deryck> kfogel, ok.  Give me 5 minutes to get to stopping point, if that's cool.
<kfogel> deryck: yup
<abentley> wgrant, around?
<deryck> kfogel, I can chat now if you like.  Skype?  Phone?
<kfogel> deryck: +1 (347) 591-7738
<deryck> kfogel, ok, calling now.
<kfogel> deryck: no you're not.  you're typing :-)
<deryck> one handed
<kfogel> wow
<kfogel> https://bugs.edge.launchpad.net/malone/+bug/515584
<mup> Bug #515584: BugsPatchesView:batchedPatchTasks() should use a Zope form, instead of validating form values manually <story-patch-report> <Launchpad Bugs:In Progress by kfogel> <https://launchpad.net/bugs/515584>
<kfogel> https://bugs.edge.launchpad.net/malone/+bug/322130
<mup> Bug #322130: Convert IHasBugs.searchTask(order_by) to use a real enumeration <api> <tech-debt> <Launchpad Bugs:Confirmed> <https://launchpad.net/bugs/322130>
<thekorn> ^ ha! - it would be great if the last one could be fixed ;)
<kfogel> intellectronica: can I submit lp:~kfogel/launchpad/515584-fix-DRY-violation to you for review?  Note it has a very trivial UI effect -- the sort orderings have better names now -- so I'd list you as the UI reviewer too, though I'm not even sure it needs UI reviw.
<intellectronica> kfogel: i'm just on my way out, so it's probably better if you try and find someone else. if you don't, submit it to me anyway and i'll review it when i'm back later in the evening
<kfogel> intellectronica: np, thanxs
<kfogel> gmb: mind if I submit lp:~kfogel/launchpad/515584-fix-DRY-violation to you for review?  Small change.
<bac> hi abentley.  i want to lp-land a branch for someone else.  if i use 'bzr lp-land <url-of-MP>' will it work, even if i don't have a local copy of the branch or a copy on LP?  should it just get all of that info from the MP?
<abentley> bac, I haven't tried it yet, so I think it would fail.
<bac> abentley: indeed it does, but with a bzr error
<bac> it would be handy if it worked
<abentley> bac, I agree.
<bac> abentley: i'm getting: bzr: ERROR: exceptions.AttributeError: 'Entry' object has no attribute 'source_branch'
<abentley> bac, that sounds pretty desperate.  source_branch would be on the merge proposal.  What's the traceback?
<bac> abentley: i got no traceback but have a crash report
<bac> which i haven't looked at yet
<abentley> bac, rerun with -Derror.
<bac> ok
<bac> abentley: still no traceback
<bac> ah, it's in the crash report
<bac> abentley: http://paste.ubuntu.com/382450/
<abentley> bac, that's very strange-- the object that's supposed to be a merge proposal has no source branch.
<bac> abentley: OOPS -- i pasted in the URL to the branch not the MP!
<bac> abentley: by that i mean, in the lp-land command i'd inadvertantly used the URL of the branch not the URL of the MP
<abentley> bac, right, I'd forgotten about that option.
<abentley> bac, we should probably add a -d option to specify the branch.
<mwhudson> jelmer: hi?
<mwhudson> argh argh argh git isn't installed on any data centre machiens
<jelmer> mwhudson, hello
<mwhudson> jelmer: you can probably guess what i want to talk about
 * mwhudson fires up an ec2 instance to test stuff on
<mwhudson> jelmer: :(
<jelmer> mwhudson: Cricket? ;-)
<mwhudson> heh
<mwhudson> no
<jelmer> mwhudson: I saw the URL you pasted. My guess is that we are somehow updating the git map before we've added the revisions to bzr
<mwhudson> jelmer: oh, so the git map contains all the objects in the git repo, but we only imported some of them
<mwhudson> jelmer: oh, something else that might have happened
<mwhudson> we imported 1000 revisions
<jelmer> mwhudson, I'm just speculating
<mwhudson> but the branch only contains 930
<jelmer> but that's what I would guess based on the error
<jelmer> mwhudson, oh
<jelmer> mwhudson, that would explain it as well
<mwhudson> presumably because not all of the imported revisions are in the ancestry of the last one we imported
<jelmer> perhaps we imported those revisions but they didn't up in the repo for some reason?
<mwhudson> well, sure
<mwhudson> we only move the branch around between import attemps with 'push'
<jelmer> mwhudson, does the repo contain 1k revisions even if the branch doesn't?
<mwhudson> i doubt it
<mwhudson> i can't access the repo directly in any case
<mwhudson> ENOLOSA
<mwhudson> jelmer: A way to fix this, though perhaps not a totally sensible one
<mwhudson> is to, in import_git_objects, compute revision_ids as now, then take revision_ids[limit] as the new desired head and recompute revision_ids
<mwhudson> alternatively, we can try harder to preserve all revisions in the repo between import attempts
<lifeless> rockstar: what are you looking at ?
<mwhudson> jelmer: thanks for the hint at least!
<mwhudson> bah
<poolie> hello mwhudson, lifeless, jelmer
<lifeless> mwhudson: this is incremental fetching?
<mwhudson> lifeless: yar
<lifeless> mwhudson: you can use targetb.repository.fetch(sourceb.repository)
<lifeless> mwhudson: that will copy all heads
<mwhudson> lifeless: oh that transfer all revs?
<lifeless> not just referenced ones
<mwhudson> lifeless: iinteresting
<lifeless> just call that after branch.push
<mwhudson> sweet
<lifeless> hi poolie
<mwhudson> and pull i guess?
<mwhudson> or sprout, in this case, i think
<mwhudson> lifeless: what does BzrDir.sprout return? is it the branch?
 * mwhudson must try not to fix every oddness in one go
<lifeless> mwhudson: yes and pull
<lifeless> sprout returns the new bzr as per its docstring
<lifeless> I'm pretty sure [without checking code, that is]
<mwhudson> lifeless: the docstring is not super clear about this
<lifeless> mwhudson: if this works around it for you, file a bug on bzr-git, you shouldn't need to do this, and other folk doing imports may run into it
<mwhudson> (i did read it first)
<lifeless> mwhudson: ok; let me check then
<mwhudson> lifeless: um, noone else will be using the bzr-git feature for this
<mwhudson> (as i wrote it last week)
<mwhudson> and there's no command line interface
<lifeless> it returns the new bzrdir
<lifeless> bzrdir.sprout, that is
<mwhudson> cool, that makes sense
<mwhudson> yeah
<thumper> Ursinha-food: ping
<Ursinha-food> thumper: pong
<thumper> Ursinha-food: http://dev.launchpad.net/CodeTeamTestPlans/10.02?action=diff&amp;rev1=39&amp;rev2=40
<mwhudson> thumper: should be able to fix incremental imports easily enough \o/
<thumper> Ursinha: just got a whole heap of old commits added to the test plan
<thumper> mwhudson: awesome
<Ursinha> argh
<Ursinha> *sigh*
<Ursinha> thanks thumper
<thumper> Ursinha: np
<Ursinha> thumper: this is because I've implemented the changes to add only edge/staging items to the testplans but the files that contain the last revision numbers were overwritten
<thumper> oops
<rockstar> lifeless, I'm not sure what you mean.
<thumper> rockstar: are you back?
<rockstar> thumper, I am.
<thumper> rockstar: talk now?
<rockstar> thumper, 5 minutes?  My car is running in the driveway right now, and I need it to run just a little longer before I shut it off.
<thumper> ok
<mwhudson> gosh launchpad takes a long time to import
<thumper> mwhudson: from what?
<thumper> rockstar: I have a call with flacoste in 5 minutes
<mwhudson> thumper: just the time between starting ./bin/test and tests actually start running
<mwhudson> it'
<mwhudson> it
<thumper> ah
<mwhudson> it's like 4 seconds
<mwhudson> and the test i'm running takes 0.1...
<rockstar> thumper, okay, I'll grab you afterwards.
<thumper> rockstar: take a look at the code first then :)
<rockstar> thumper, doing that now.
<thumper> flacoste: ready?
<flacoste> thumper: yep
<thumper> abentley: is there an RT for the staging email issue?
<abentley> thumper, yes, but I don't know the number.  Chex said he had an open ticket.
<flacoste> abentley, thumper: RT #37601
<mup> Bug #37601: unable to burn dvd .iso <gnomebaker (Ubuntu):Fix Released by motu> <https://launchpad.net/bugs/37601>
<kfogel> intellectronica: thanks.  incorporating your suggestions and resubmitting for review in a bit
<intellectronica> cool
<maxb> I wish the testsuite logged somewhere useful from its background components :-(
<maxb> wgrant: Reckon we should start a lucid status wiki page?
<wgrant> maxb: I was thinking that last night, but there are only three tests plus one or two utilities.
<wgrant> (plus that bootstrapping setuptools/distribute issue which only sometimes shows up)
<wgrant> (question-target.txt, test_ftparchive, and update-sourcecode are the problems I know of)
<wgrant> maxb: Did you dsee that xx-resetpassword-of-sso-account.txt is gone?
 * mwhudson lunches
<wgrant> I can't create a user any more, can I?
<lifeless> wgrant: why not?
 * wgrant sighs at offloading of functionality to external proprietary applications.
<wgrant> lifeless: LP uses OpenID now, so I can't create new users locally for testing.
 * wgrant gets out the factory.
<lifeless> wgrant: so, you should setup an authorising party, and point your test lp at it for auth :>
<wgrant> lifeless: It's producer-locked, sadly.
<wgrant> And the test producer doesn't have functionality to create a user.
<lifeless> yes, but you can change that :)
<wgrant> That functionality is reserved for that quaity piece of software that is c-i-p.
<abentley> wgrant, launchpad does not yet use OpenID AFAIK.
<lifeless> even in your local, test lp ?
<wgrant> abentley: As of 7 hours ago it does. Which is good.
<abentley> wgrant, in stable or db-stable?
<wgrant> abentley: devel. Might not be in stable yet.
#launchpad-dev 2010-02-24
<thumper> wgrant: eh?
<thumper> wgrant: launchpad.dev uses openid?
<wgrant> thumper: Yeah.
<wgrant> And edge will tonight.
<thumper> why?
<wgrant> Because c-i-p is the new hotness.
<thumper> cip?
<wgrant> canonical-insecurity-provider
<thumper> and wtf is that?
<wgrant> login.ubuntu.com
<lifeless> thumper: the i is identity
<wgrant> Not after 48 hours.
<lifeless> wgrant: do we need to stop the rollout ?
<thumper> wgrant: does this mean you can't create a user?
<thumper> wgrant: locally?
<wgrant> thumper: The OpenID thing does, but that's easily resolved with the factory.
<wgrant> Just a little annoying that all this functionality is being pushed into an external proprietary application the development team for which isn't doing a very good job.
<thumper> my current branch is 1600 revisions behind devel... :(
<lifeless> thumper: wheee
<thumper> lifeless: and I merged trunk with no conflicts \o/
<lifeless> nice
<lifeless> what vcs are you using ? :)
<thumper> lifeless: perforce
<thumper> heh
<lifeless> haha
<thumper> NOT!
 * mwhudson writes a LEP: using clearcase for launchpad development
<lifeless> mwhudson: E stands for enhancement
<mwhudson> lifeless: spoilsport!
<lifeless> mwhudson: :>
<wgrant> Are Clearcase and Perforce really that bad?
<wgrant> I know VSS is, but I haven't had the pleasure of using the other two.
<lifeless> differently bad
<lifeless> not dataloss bad like vss
<wgrant> Pfft. Where's the fun in that?
<thumper> wgrant: perforce is quite good
<thumper> wgrant: clearcase is fcuking aweful
 * thumper checks his spelling
<thumper> no e
<thumper> awful
<thumper> horrible
<thumper> makes you want to slit your wrists
<lifeless> so, a lphant then ?
<thumper> perforce at least does what you expect
<thumper> clearcase takes forever, slow, complex and doesn't always do what you want
<thumper> doesn't even have atomic commits
<wgrant> Ew.
<mwhudson> apparently it gained atomic commits in december 2009!!
<thumper> mwhudson: really?
<mwhudson> thumper: it's what wikipedia says
<thumper> ok, didn't have atomic commits when I used it
<thumper> which was 2006
<maxb> hmm. I get a different set of test failures on lucid to wgrant
<wgrant> maxb: What failed?
<wgrant> (this brings back bad, bad memories of mid-Karmic)
<maxb> AppServerLayer oopsed all over the place, and I have one bugs windmill failure
<wgrant> Which Windmill test failed?
<wgrant> And what were the AppServerLayer oopses?
<maxb> ohh.... wait a moment
<maxb> it might work better if I had testopenid.dev in my /etc/hosts
<wgrant> Yeah.
<maxb> the oopsing just gave me a load of 500 internal server errors, and nothing more helpful in a visible log
<wgrant> It'll be the OpenID XRDS hostname lookup exception.
<maxb> hmm, so now I just have test_bug_tags_entry (lp.bugs.windmill.tests.test_bug_tags_entry.TestBugTagsEntry)
<wgrant> maxb: that passed here. What's the error?
 * wgrant tries on recent devel.
<maxb> rerunning, I suppose it could have been openid issues too
<maxb> yeah, it's happy now
<maxb> well, that's nice. Lucid here we come... next up python2.6
<wgrant> maxb: Can you reproduce the two identical question-target.txt failures, and the test_ftparchive one?
<maxb> question-target.txt did not fail for me
<wgrant> The question-target.txt one looks really odd -- the query should ensure order.
 * wgrant reruns.
<maxb> I see the ftparchive one
<wgrant> I haven't looked at that one at all.
<wgrant> I also need to test utilities/*, since some of it is broken.
<maxb> oh, you mean the "You need to invoke update-sourcecode using the system python, not what its shebang says" problem?
<wgrant> yes.
<wgrant> And maybe others too
<wgrant> Have you had the bootstrapping issue lately?
<wgrant> I ran into it for the first time in weeks last night when trying the python2.6 branch.
<maxb> I've been working in karmic still quite a lot
<maxb> so no, I've not seen it
<wgrant> OK.
<wgrant> And what do you know, questiontarget.txt passes for me now.
<wgrant> Actually, the failure was on db-devel.
 * wgrant tries there.
<wgrant> I would really love it if the DB and librarian data directories were per-branch.
<lifeless> doable
<wgrant> OK, questiontarget.txt works on db-devel too.
<wgrant> It must have just hated me yesterday.
 * wgrant looks at test_ftparchive.
 * wgrant vomits in the direction of lp.archiveuploader.ftparchive.
<wgrant> s/uploader/publisher/
<maxb> uhoh, postgres 8.3 just got kicked out of lucid - so much for launchpad-dependencies being installable
 * maxb copies it back into the ppa
<wgrant> Urgh.
<wgrant> still, it seems that stub wants to move to 8.4 really soon.
<maxb> I love how easy it is to use PPAs to rescue removed packages
<wgrant> Yeah, it's pretty handy.
<maxb> ... pending publication on amd64 i386 powerpc sparc and ia64 :-)
<maxb> not often you see the ppa build status column being stretched that far
<wgrant> No armel?
<maxb> it ftbfs, apparently
<wgrant> Ah.
<wgrant> maxb: Oh. I just realised that questiontarget.txt does still fail on db-devel. I just had my test fix in place.
<wgrant> maxb: can you try it on db-devel?
<thumper> AARRGGGHHHH!!!!!
<thumper> YUI madness
 * thumper heads to #yui
<wgrant> Anybone else running LP on Lucid?
<poolie> wgrant: only under a chroot
<wgrant> poolie: That's fine. Could you please 'bin/test -vvt questiontarget.txt' in db-devel?
<poolie> it will take a few minutes, but yes
<poolie> wgrant: just to be clear, i mean in a karmic chroot on a lucid host
<poolie> wgrant: do you still want it?
<wgrant> poolie: Oh, right. That's probably not what I want, then.
<mwhudson> gaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaar
<lifeless> ?
<wgrant> Normally I would assume that was a JavaScript scream.
<mwhudson> it's a zope scream
<mwhudson> xml-rpc methods can't take *args
<lifeless> haha
<lifeless> sorry.
<wgrant> Better than lazr.restful methods, which can't take positional args at all.
<lifeless> 'omg I'm so surprised'
<mwhudson> and it'
<mwhudson> s basically because of stupid LBYL-ism
<mwhudson> wgrant: i _could_ be writing js!
<mwhudson> theoretically
<wgrant> Gngrg stupid doctests.
<wgrant> -D is almost completely useless with them.
<wgrant> Also, WTF. I have a test failure in db-devel on Lucid that does not appear in devel. It's an ordering difference, but executing the query manually in the transaction gets the right order.
<poolie> lifeless: so it's bug 59846
<poolie> hullo bot?
 * mwhudson EODs
<swarna> http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/utilities is not opening
<wgrant> swarna: Go to http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel and browse to utilities/rocketfuel-setup from there.
<lifeless> poolie: well, that was about an earlier behaviour
<lifeless> poolie: I think perhaps the last subscriber can't unsubscribe.
<poolie> is it? that looks like what we're hitting here
<lifeless> no, the bug in question had plenty of subscribers
<wgrant> Ah, it's private. That's why there's no bot.
<lifeless> and I've unprivated it.
<poolie> how is having lots of subscribers relevant?
<lifeless> in reference to bug 59846, where we dealt with bugs having only one subscriber (the filer) due to a bug in the bug filing logic
<poolie> i think kiko felt that bug was closed by dealing by cleaning up some particular cases
<poolie> but the description says
<poolie> >  it seems wrong that (a member of) the *security contact* for a product can be prohibited from viewing a bug report about that product, whether they are subscribed or not.
<poolie> and i agree with that
<poolie> the underlying bug still exists
<lifeless> so, this isn't a quick topic, and I started 9 1/2 hours ago.
<poolie> np
<poolie> it's not urgent
<poolie> i was just surprised
<poolie> i understand what's causing this now, so thanks
<poolie> you can followup on 59846 if you want
<lifeless> I appreciate you feel that there is a bug here; I don't see how to reconcile what you are asking for with CVE support and multiple  organisations both contributing to one project.
<lifeless> the bug you found wasn't filed on bzr; it was half-connected to bzr, and should have been made unprivate at the same time
<poolie> are you saying that's a case where there are bugs that affect a project, that the project's security team is not allowed to know about?
<lifeless> poolie: yup
<lifeless> as I understand it anyhow.
<poolie> hm
<lifeless> two separate cases; also note that private and 'security' are nowadays separate flags.
<poolie> well, nobody has yet made a case for that on that bug
<lifeless> so I wouldn't comment on bug 59846 regardless, I'd file a new one.
<poolie> so it looks like just a plain bug
<lifeless> poolie: [meta] this looks like a case where a new bug would really have been better
<poolie> yes, you said that
<lifeless> poolie: I've commented there anyhow
<lifeless> gnight
<poolie> to me it seems like the underlying bug has not actually been fixed
<lifeless> poolie: it is deliberate, by design. Rejected would be a better term, for that aspect of it.
<poolie> that may be true but nobody has actually said so
<poolie> except you :)
<poolie> and it sounds more like defense of an accidental behaviour
<lifeless> poolie: comment #2 describes it
<lifeless> I agree that a clear unambiguous statement wasn't made.
<poolie> it's not actually the same thing
<lifeless> And I'm not in a position to do that either.
<lifeless> I'm just sharing the context and history I know of with you. Privacy was /agonised/ over.
<stub> wgrant: If the ordering is depended on but not explicit, its broken. implicit order is effectively random (you can't even trust it to come out in disk order, as it might get pulled from a cache).
<wgrant> stub: The order is in the query, sorry.
<wgrant> stub: What is the timeframe for moving to PostgreSQL 8.4?
<wgrant> stub: 8.3 is gone from Lucid now.
<poolie> lifeless: (when you're back)
<poolie> i don't think this is a critical bug
<poolie> but it is pretty surprising
<poolie> and i think there is some kind of bugginess here
<poolie> it's internally inconsistent
<wgrant> What is the bug?
<poolie> the security team can't see all bugs on the product
<poolie> and, depending on the history of the bug, they may not get subscribed by default
<poolie> for instance adding a new bugtask does subscribe them
<stub> As soon as possible. Last I looked, we had maybe three or four tests failing that looked trivial. I need to get slony 1.2.20 packages backported from lucid to hardy, at which point we can test staging and migration procedures.
<poolie> retargeting an existing bugtask does not
<wgrant> Ah. Hm.
<wgrant> stub: Oh, excellent.
<wgrant> poolie: Really? I've found that moving a task explicitly subscribes the security contact even when the bug is public.
<stub> wgrant: Bug #426823 is tracking this
<mup> Bug #426823: Update to PostgreSQL 8.4 <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/426823>
<poolie> wgrant: nup, definitely not
<poolie> look at https://bugs.staging.launchpad.net/launchpad/+bug/453104
<mup> Bug #453104: sftp user/password don't work <Bazaar:Incomplete> <https://launchpad.net/bugs/453104>
<poolie> oh heh, you can't :)
<poolie> but try now
<wgrant> If mup can see it, it's probably public...
<poolie> it's public in reality, i changed it on staging
<wgrant> Ah.
 * wgrant curses 64kbps throttling.
<lifeless> poolie: so I think this hinge on what 'security team' means
<lifeless> poolie: I think it means 'people to be told about new security bugs, by default'
<wgrant> I think that is the assumption.
<poolie> k
<lifeless> poolie: I'm open to 'adding a product task to a secured disto bug should subscribe the product security team'
<poolie> right
<poolie> or retargeting a task
<lifeless> poolie: but that is very different to 'the security team can always see bugs'
<lifeless> right
<poolie> and i dont' think it's specific to whether they're distro or product or series tasks etc
<lifeless> conceptually perhaps not
<poolie> lifeless: so 'the security team should always see bugs' is a different bug
<poolie> perhaps a closer match for this existing one
<poolie> i'm kind of ok if that's marked wontfix
<poolie> i would like there to be a public bug for it, as documentation
<poolie> and explanation
<poolie> i do think it's a bit weird that potentially absolutely no one from the product can see the bug
<lifeless> I think thats fine, but something we want to have happen explicitly not accidentally
<wgrant> It's handy if you want to hide a bug, like I did last week.
<poolie> i mean some bugs seem to at least accidentally have got into this state
<wgrant> When does edge update these days?
<noodles775> wgrant: I didn't realise it had changed (but haven't been watching), so I'd assume between 8:30-9:00UTC?
<wgrant> noodles775: buildd-manager has again been broken for three or four hours.
<wgrant> noodles775: It probably hasn't changed. I just didn't remember.
<noodles775> wgrant: no losa? OK, they're still sprinting and probably having brekkie.
 * noodles775 tries to find out.
 * wgrant wonders if this actually happens quite often, but there's normally a LOSA around to respond to the Nagios scream.
<noodles775> wgrant: I don't think so, there's a note on their page to always let us know (which is why I keep the bug up to date).
<wgrant> noodles775: Ah, good.
<noodles775> If it is happening regularly now, it might be worth CP'ing the change so that it doesn't block the b-m.
<wgrant> It seems like it might be worth it to log all SQL queries executed by b-m to see if we can work out WTF is happening.
<noodles775> yeah, let's chat about that with bigjools. Great, the losa's are on it :)
<wgrant> Great.
<noodles775> wgrant: back to normal now, thanks to the losa's, and we'll check the sql log to see if it can point us back to a problem in the code.
<wgrant> noodles775, $LOSA: Thanks.
 * wgrant has a look through the new code for anything suspicious.
<wgrant> It was the building build without a builder again, right?
<adeuring> good morning
<noodles775> wgrant: Thanks, and yes, a build_queue item that is dispatched (ie. has a related job with non-null date_started), but build_queue.builder is None.
<wgrant> noodles775: Do we have the state of build/buildqueue/job before it was repaired?
<noodles775> wgrant: other than the above, no, but I'll update the wiki page so that we have it next time.
<wgrant> noodles775: Great.
<wgrant> It might tell us where in the build it actually was.
<bigjools> g'morning
<wgrant> Morning bigjools.
<mwhudson> hello europe
<bigjools> morning wellington
<wgrant> You know you're in trouble when the Americans (apart from deryck) show up.
<mwhudson> wgrant: even more so for me i guess
<mwhudson> haven't stayed up that late in a while
<wgrant> bigjools: Speaking of download stats...
<bigjools> heh
 * mwhudson stares at buildbot, trying to figure out if his change will be in db-stable in time for the code import update in more than twelve hours time
<wgrant> mwhudson: In time to get the incremental import repo pull fix onto staging?
<mwhudson> wgrant: exactly
<jelmer> moin mwhudson
<jelmer> hi wgrant
<wgrant> Morning jelmer.
<wgrant> jelmer: Are you running LP on Lucid?
<jelmer> wgrant, yep
<wgrant> jelmer: Can you 'bin/test -t questiontarget.txt' on db-devel, please?
<wgrant> jelmer: I get a strange ordering failure here, but not on devel, and I wonder if anybody else can reproduce it.
<jelmer> wgrant: ok
<wgrant> It's one of only two lucid test failures.
 * jelmer hates how slow 'make schema' is on this machine
<wgrant> It's slow everywhere.
<jelmer> wgrant, running now
<wgrant> jelmer: Thanks.
<jelmer> wgrant, failing here too because of the ordering
<wgrant> jelmer: excellent except not.
<jelmer> "daf" is now the first entry rather than the second
<wgrant> jelmer: Yep, same failure is seen here.
<wgrant> But the query is explicitly ordered, and running it in that transaction gets the right result.
<jtv> henninge: feel free to replace the intltool-debian line with plain intltool, or use them both.
<henninge> jtv: I am still not convinced they are the same.
<henninge> jtv: ah, now I see
<jtv> henninge: when I wrote that line, I had no reason to care.  I might as well have installed Quake and added a comment for you to replace it with the right package.  :)
<henninge> jtv: so why is it that you install it manually from the script and not just add a dependency to the package?
<jtv> henninge: to what package?  I didn't want to pull in another package for all slaves when we only need it for this one job.
<jtv> Also, IIRC this gets installed into the chroot whereas the buildd package lives outside it.
<henninge> jtv: oh, we have a chroot inside the buildd?
<henninge> to build
<jtv> henninge: yes, the slave sets up a chroot, and then sets up its build environment in there.  That's what I did in translationtemplates.py
<henninge> jtv: frankly, translationtemplates.py looks a bit over-engineered ... ;) Mind if I change it or is this some common pattern?
<henninge> jtv: using a state machine this way, I mean.
<jtv> henninge: that's how the base class wants it.  The job runs asynchronously in twisted, so it doesn't hold everything up.
<jtv> It is _possible_ to do the whole thing in one go, but not much easier.
<jtv> And this way helps provide progress information, I believe.
<wgrant> You need to either use the state machine or have a single subprocess that does all of the work.
<wgrant> bigjools: Given the earlier discussion, I guess you don't have time to discuss PPA download stats today?
<bigjools> wgrant: next week would be better - I really need to get the rebuild work done
<bigjools> or maybe on Friday if I am done by then
<wgrant> Yep, OK.
<wgrant> I found some odd behaviour when testing them locally -- the binaries ended up in NEW. But it looks like it's OK on production.
<wgrant> Hm, rebuild archives have the same permission rules as PPAs when they're disabled. Damn.
<wgrant> bigjools: Is there a good reason for that, or can I submit a branch to fix it?
<bigjools> wgrant: which rule is a problem?
<wgrant> bigjools: Disabled copy archives are not accessible except by their owners and admins.
<wgrant> This is awkward, since they normally get disabled when they're done.
<bigjools> there's a bug about not being able to see them when they get disabled
<wgrant> Bug #488468
<mup> Bug #488468: Disabled rebuild archives are not shown anywhere <Soyuz:Triaged by jelmer> <https://launchpad.net/bugs/488468>
<bigjools> there ya go
<deryck> Morning, all.
<deryck> BjornT, ping.
<deryck> adeuring, hi.  I think noodles775 suggestion about using real text and hiding it off screen is nice, assuming this does indeed play well with screen readers.
<deryck> I say "assuming" not to question noodles775, just that I don't know much about screen readers.
<adeuring> deryck: yes and no. I think it is a bit like expoiting a bug in screen readers ;)
<adeuring> they should not present non-visible text, I'd argue
<deryck> adeuring, hmmm, that would make me concerned then.  img alt tags are the standard way to handle this.
<adeuring> deryck: that's my main point indeed.
<deryck> adeuring, let me look closer at the comments in the MP again.
<deryck> adeuring, did you see the link noodles775 provided?  The webaim site notes pushing content off screen as an acceptable way to hide content.  I would assume they know screen readers well.
<adeuring> deryck: no, I had not looked yet.
<deryck> adeuring, it makes me confident in this approach, and using a sprite and a fmt'er would make this nicer to use in tal and for front-end performance.
<adeuring> deryck: well, ok. I am not 100% convinced, but it does not matter that much.
<deryck> adeuring, ok, thanks.  sorry to send you back to work again.
<adiroiban> henninge: hi. when you have time, can you please take a look at the last comment from https://code.edge.launchpad.net/~adiroiban/launchpad/bug-509252-take-2/+merge/19484
<BjornT> deryck: pong
<jtv> Oh wow, a "here's your compensation from the Nigerian government for being scammed" Nigerian scam.
<gmb> intellectronica: If you want to chat about a logarithmic scale for bug heat, I'll be free in 5.
<intellectronica> gmb: excellent. i'm just firing an MP for my first branch and will ping you in ~5m
<gmb> Cool.
<beaver> www.search2.net
<intellectronica> now that must be the mossad ^^^^^
<intellectronica> gmb: ready when you are
<gmb> intellectronica: Cool. Skype?
<intellectronica> gmb: yup
<deryck> intellectronica, just FYI, I'm changing my max_heat branch to have max_bug_heat where the default is NULL not 0.
<deryck> intellectronica, but you could review as is, and I can fix it up in my branch when merging in.
<kfogel> intellectronica: hey, you're totally right about that targetName thing -- I feel dumb.  Will fix, thanks.
<intellectronica> cool
<intellectronica> and it was easy to confuse that one
<intellectronica> maybe it's even worth a comment, so that the next reader won't get confused. i nearly missed it myself
<intellectronica> kfogel: ^^^
<kfogel> intellectronica: well, I think the code will make it clear, since the used-visible display name is going to be generated instead of hardcoded.
<kfogel> intellectronica: maybe an explanation of why, though, yeah.
<intellectronica> deryck: i didn't really update on my bug heat work. the branch for using max_heat is in review. i am now looking at scaling the results. went over it with gmb and we've reached the conclusion that just using a logarithmic scale is not obviously a good solution, so now i'm trying out a few formulas to see what makes sense. i'll try not to get stuck on that for too long and come up with something by the end of the day. it will be much easier 
<deryck> intellectronica, sounds good.  thanks for the info.
<deryck> intellectronica, sorry to leave you out, too.  Still trying to find a good rhythm with the board.
<intellectronica> that's oright. it's also not like the call is our one and only opportunity to update
<deryck> right
<intellectronica> i wonder if we can get a feed or something of the updates on the kanban and spew it here
<deryck> intellectronica, there is RSS for the board.
<intellectronica> it's nice how the kanban shows a notification when someone changes something, but you can't really follow it
<deryck> yeah
<deryck> we need some kind of commit email or something.
<deryck> this would also make sharing in the open better.
<deryck> flacoste, is there something like the above for the board? ^^  Or some way to come up with that.
<flacoste> deryck: there is
<flacoste> each line has an email icon
<flacoste> to subscribe to events there
<flacoste> there is also a RSS feed for the whole board
<flacoste> and there is an email notification option for the whole board also
<flacoste> look under the Options tab
<deryck> ah, ok.  thanks flacoste
<deryck> intellectronica, see that ^^.  You can subscribe your email to board events.
<intellectronica> deryck: thanks. i'll give that a try.
<deryck> I find the per lane rss or email useless, really.  But the global option is nice.
<intellectronica> ah, and i found out why i didn't see avatars. there's a global option to show or hide them
<flacoste> deryck, intellectronica: the option to turn off the white board markers is also nice to get more screen real-estate
<deryck> ah, right.  sorry.  I didn't realize that was per user.
<intellectronica> flacoste: indeed
<deryck> yes, nice.
<deryck> flacoste, I wonder if we couldn't have a generic user account and subscribe some email to all boards and have this go to a mailing list.  Or something like this.
<deryck> flacoste, would make it less private.
<flacoste> deryck: they should implement a "public board" option soon
<deryck> flacoste, ah, ok.  cool.
<kfogel> intellectronica: new version of the branch pushed up.  It was actually not as trivial to fix as I thought -- the uppercase/lowercase needs are different between the sort order display and the column name on the right, which wasn't an issue before.
<kfogel> intellectronica: anyway, review when ready
<intellectronica> kfogel: looks good. r=me
<kfogel> intellectronica: thx
<kfogel> intellectronica: you think any need for UI review on this one?
 * kfogel answers his own question
<kfogel> no.  the ui change is trivial and unambiguously an improvement :-)
<intellectronica> kfogel: never hurts, but since this was such a small change i don't think it's strictly necessary
<intellectronica> kfogel: normally i wouldn't even ask for a first ui review for something like this, tbh
<kfogel> intellectronica: what, you mean for the whole feature?
<intellectronica> i only added the keyword because we did discuss it a bit in the review
<kfogel> intellectronica: keyword?
 * kfogel confused
<kfogel> intellectronica: where?
<intellectronica> kfogel: for the whole feature definitely, but for something like the capitalization of the option in a combo ... depending on how confident i'd feel about the change
<intellectronica> kfogel: keyword in the review signature. i used 'code ui'.
<intellectronica> maybe it's called review type
<kfogel> intellectronica: oh, didn't refresh, didn't see that yet
<kfogel> intellectronica: so if I use 'ec2 land', it'll automatically use the commit message from the merge proposal, right?  That is, I can just run 'utilities/ec2 land -v --headless -b launchpad=db-devel' ?
<intellectronica> kfogel: if so then that's a complete surprise to me. i never used it like that.
<kfogel> hmmm.
<intellectronica> let me know if it works for you :)
<kfogel> intellectronica: well, I don't want to risk a change landing with a bogus message.  I'll ask around.
<kfogel> deryck: do you regularly use 'ec2 land'?
<deryck> kfogel, I do
<kfogel> deryck: so the 'ec2 help land' text is a bit thin.  I'm trying to figure out if, when a merge proposal has a commit text set (as https://code.edge.launchpad.net/~kfogel/launchpad/515584-fix-DRY-violation/+merge/19990 does) whether 'ec2 land' will automatically use that as the PQM submit message.
<deryck> kfogel, it will.  do `ec2 land --dry-run` and you can see what it will do.
<kfogel> deryck: and if so, whether it auto-generates the reviewer metadata "[r=foo][etc]" or whether it figures all that out from the merge proposal.
<deryck> kfogel, it does
<kfogel> deryck: thx
<deryck> kfogel, np
<cbx33> hey peeps
<bdmurray> could somebody review my branch for bug 527174?
<mup> Bug #527174: logo_link missing from "team" object in API <Launchpad Registry:In Progress by brian-murray> <https://launchpad.net/bugs/527174>
<thumper> rockstar: I have a branch now that refreshes the status table on the mp page when the status is updated
<thumper> rockstar: it is awesome
<rockstar> thumper, we're refreshing the whole table?
<thumper> rockstar: I'll make a movie :)
<rockstar> bdmurray, #launchpad-reviews is what you want.
<kfogel> intellectronica: in my launchpad.dev instance, default batch size for +patches view seems to be 5 bugs per page instead of (say) 50.  Is this a dev thing we do?  I don't remember setting that param anywhere, and it's not in my URL.
<intellectronica> kfogel: yes, that's a dev env setting
<kfogel> intellectronica: thanks
<jtv> Wasn't there an easy way to add a person to a team?
<salgado> jtv, the team's home page should allow you to do that.  there's an ajaxy link there
<jtv> salgado: I should've said: in code.
<salgado> oh, code
<salgado> jtv, team.addMember(person), then?
<jtv> salgado: ah, thanks!  Yes, that's what I was looking for.  I missed it somehow (did grep for addPerson of course)
<maxb> OOI, are there any estimations on when the datacentre Lucid upgrades are planned?
<sinzui> jtv: you are still awake?
<thumper> maxb: IIRC some get upgraded at beta time
<thumper> maxb: others shortly after release i think
<maxb> To put it another way: When is edge or staging likely to running on lucid? :-)
<maxb> * to be
<mwhudson> i would guess staging would be pretty soon after release
<mwhudson> totally a guess though
<sinzui> bac: EdwinGrubbs: I just sent a request for sql/ddl help to launchpad-dev. Can both of you read it and provide your thoughts.
<bac> ok
 * sinzui is now thinking about how to move the packaging portlet to staging-only and update the bug heat script to add info to sourcepackagename
<lifeless> mwhudson: was the bzr-git patch sufficient?
<mwhudson> lifeless: i haven'
<mwhudson> t tested it actually
<mwhudson> i guess it would be easy enough
<thumper> mwhudson: you have much QA to mark off :)
<mwhudson> thumper: see earlier griping about staging
<thumper> mwhudson: yeah, but there are some we can check off right?
<mwhudson> thumper: no?
<mwhudson> thumper: all the 'landed' things are code import related
<lifeless> mwhudson: it would be good to get it tested and done
<lifeless> mwhudson: one less thing in progress ;)
<EdwinGrubbs> sinzui: can you do an EXPLAIN ANALYZE on that query? EXPLAIN can often be misleading with its cost numbers.
<sinzui> EdwinGrubbs: just explain as I showed
<EdwinGrubbs> sinzui: I'm asking whether you can rerun it with EXPLAIN ANALYZE?
<sinzui> sent
<sinzui> EdwinGrubbs: The numbers look roughly the same.
<EdwinGrubbs> sinzui: the costs will look the same, but I would like to see the times, which are easier to understand. How many times is this query run on the page? just once?
<sinzui> EdwinGrubbs: We pay a terrible cost scanning all bugs and the join to bugtask is also expensive
<sinzui> check your email
 * sinzui is trying to understand how bugs are queued in the new bugjob system.
<sinzui> EdwinGrubbs: the query is run once for every page that uses it.
<EdwinGrubbs> sinzui: I don't think it will be possible to get much better results if 3/5 of the bugs have a sourcepackage. Caching the heat in sourcepackage is definitely the best bet if it is ok for the data to be a little stale. If it needs to be absolutely up to date, caching it on the BugTask table would be safer to implement via a trigger, and eliminating the join should speed it up greatly.
<EdwinGrubbs> sinzui: as a shot in the dark, you could try "set enable_seqscan = false;" before you run explain analyze to see if it cuts down the times. It probably won't help since so many of the bugs in the table are being read.
<sinzui> EdwinGrubbs: agreed. bug heat is updated using the bugjob system. The mechanism for jobs is quite elaborate. I was hoping to append a rule for sourcepackagename to something.
<EdwinGrubbs> makes sense
<sinzui> I bug heat and po messages scoring is fuzzy, I think it is fine to cache it. I think I need to consider disabling the problem calls in the views; the work to build a cronscript correctly may be mroe than 8 hours. I need a schema change approval too
<thumper> rockstar: ping
<rockstar> thumper, pong
<thumper> rockstar: skype?
<rockstar> thumper, sure.
<thumper> rockstar: https://code.edge.launchpad.net/~thumper/launchpad/use-last-rev-id/+merge/20023
<sinzui> is salgado an admin in sample data?
<mwhudson> sinzui: i think so, not sure though
<sinzui> We cannot expire our oauth tokens. I see a typo in the security checker, yet the test passes, implying it works because salgado has god-like qualities.
<mwhudson> yep, he's an admin
<mwhudson> i remember griping at this because the webservice tests are/were done with salgado by default
<mwhudson> or something like that
<sinzui> yes, the same is true for oauth
<thumper> rockstar: http://people.canonical.com/~tim/magic.ogv
<jtv> sinzui: I'm here
<sinzui> jtv: I was looking for help improving a query. I sent a request to launchpad-dev for help. EdwinGrubbs concluded that changing the schema was the only viable option to improve the query performance.
<jtv> sinzui: the fix for the translations was simple; cp'ed yesterday
<jtv> sinzui: where is this query you posted used?
<sinzui> jtv: [Launchpad-dev] SQL/DDL help wanted to fix a distrosseries timeouts
<jtv> sinzui: yes, I just read it.  But where is the query that you posted used in the code?
<sinzui> registry.model.distroseries._current_sourcepackage_joins_and_conditions() it is used in the packaging portlet on the distroseries, and in the +needs-packaging report
<sinzui> ^ jtv https://edge.launchpad.net/ubuntu/lucid/+needs-packaging
<thumper> Quick straw poll: Rename +junk to +personal?
 * thumper leaves to collect Maia
<lifeless> 'meh'
<lifeless> more clear
<lifeless> less sexy
<wgrant> thumper: Hasn't this been discussed to death lots of times?
<sinzui> thumper: +1
<wgrant> Also, +personal doesn't really work for teams.
<lifeless> +misc
<lifeless> actually, delete +junk
<jtv> sinzui: but that view is batched.
<lifeless> create a junkcode project; move everything to it.
<sinzui> yes, but the query behind it is a monster
<thumper> wgrant: yes, but I'm willing to move now
<jtv> sinzui: then leave the counting to the browser code.
<thumper> +personal doesn't work so well for team branches does it?
<jtv> sinzui: You're probably doing the counting for _all_ sourcepackages even though you're only showing 20.
<sinzui> jtv: This is an actual query used by the portlet...it only wants the top 10, but learning the top ten by bug heat and po messages is terrible: http://pastebin.ubuntu.com/383302/
 * jtv whistles
<sinzui> jtv: that came from https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1516S98
<sinzui> thumper, I close 3 projects a week because the user does not know +junk. The word they use to describe it is "personal" and "sandbox"
<jtv> sinzui: that is a monster.
<sinzui> jtv: yes, it took me four days to create a query that had viable results. It was a sad week for me
<lifeless> sinzui: so we should create a sandbox project perhaps?
<sinzui> jtv: I think I want to change sourcpackagename to store reporing data that Is updated daily.
<jtv> sinzui: I see that total_bugs and total_messages can be left out... does that help any?
<jtv> We have a cache like that for POFiles.  It's necessary, but keeping it up to date is sheer hell.
<sinzui> lifeless: I have approved a few when the user decided to focus the code on a single problem. I think project registration to explain to the user what a project is in launchpad and point him to +junk when he wants a sandbix
<mwhudson> sinzui: sourcepackagename doesn't have any connection to a distro
<sinzui> jtv, no, but yes they can be left out now
<mwhudson> sinzui: are you proposing just caching the values for ubuntu on there?
<lifeless> sinzui: I'm saying lets delete +junk /from launchpad/ and replace it with an actual project, called - 'sandbox' or 'personal' (or even a few such things)
<lifeless> sinzui: owned by registry; bugs turned off; no trunk series, or an empty one.
<lifeless> sinzui: we can simplify our code and make it more discoverable at the same time.
<sinzui> mwhudson: sourcepackagename is nothing but a key in our schema, but we do not think about it that way. We think about source packages having bugs and being in a distro, but launchpad does not think that way
<wgrant> Excellent.
<wgrant> Er.
<wgrant> Gah, thought the other Lucid failure had fixed itself :(
<wgrant> But no.
<mwhudson> sinzui: what you say is undoubtedly true, but i don't see the connection with what i said
<sinzui> mwhudson: I am proposing that sourcepackagename have new columns that can be used to store the total bug heat or total messages so that these do not need to be recalculated.
<mwhudson> sinzui: are you proposing actually having a "source package" concept in the databgase?
<jtv> sinzui: I take it the part you posted took the bulk of the time though?
<lifeless> jtv: look at the oops itself. (and yes)
<mwhudson> sinzui: but nowhere do we show the bugs targeted to a sourcepackagename
<lifeless> next highest question was 576ms
<lifeless> sorry, 592
<mwhudson> sinzui: we show those targeted to a sourcepackage
<sinzui> lifeless: I think your general proposal is right, but creating exceptions is costly. I think we want this repo space more visible, and have a mechanism to promote a branch to be the basis of a new project
<mwhudson> sinzui: so i guess i'm saying i don't really understand your proposal
<sinzui> mwhudson: no, just extending the sourcepackagename.
<sinzui> jtv: yes, the bugheat for a sourcepackagename is the main problem by a vast magnitude
<sinzui> mwhudson: I am using sourcepackagename as a key to join many objects (because there is not really a source package object). I think it would be easier for many queries if the sourcepackagename had additional data with it that would allow me to make simpler queries
<jtv> sinzui: I don't suppose you could ignore (or treat separately) packages without any bugtasks?
<jtv> At least you'd have an inner join instead of an outer one
<sinzui> jtv: bug heat is the best indicator that a package needs linking to an upstream project, but we do not have that data in a convenient place.
<jtv> sinzui: why is Bug in that join?
<jtv> Is there no way to get it out?
<jtv> It's used for counting heat and ids.  For ids you don't actually need it.
<jtv> (Because you can count the bugs' primary keys as the bugtasks' foreign keys)
<sinzui> jtv: bugs have heat, not tasks, which are linked to a sourcepackagename, that is in a publishing table that says the package is in a series
<jtv> sinzui: ok, so can you limit the Bug join to ones that have heat?
<sinzui> jtv: removing count is inconsequential: http://pastebin.ubuntu.com/383310/
<mwhudson> sinzui: so what data would you attach?
<sinzui> all bugs have heat, Would limiting it to heat > 0 woek
<sinzui> mwhudson: sum of bugheat and po messages
<jtv> Looks like most bugs have heat.  :(
<mwhudson> sinzui: across all distros?
<sinzui> all? yes, because only Ubuntu is really present
<mwhudson> ok
<mwhudson> that's all i wanted to check
<jtv> sinzui: if you dropped bugs with heat <= 10 from the heat count, you would make a serious dent in the number of Bugs the query has to churn through.
<sinzui> mwhudson: we claim to support all distro, but we only support Ubuntu. If we change distro(series).getSourcePackage() to only return what we know exists, several hundred bugtasks disapear, and 60 packaging links tpp
<wgrant> abentley: You pinged me yesterday?
 * sinzui tries
 * sinzui hugs jtv
<jtv> sinzui: or maybe it's not me; in your post with the subquery, "explain analyze" only reported a cost of about 2 seconds.  The larger query as shown in the pastebin and the timeout takes 12.5 seconds.
<jtv> So are we barking up the wrong tree because there are too many digits in the millisecond counts?
<sinzui> jtv: This is the largest junk from the whole query: http://pastebin.ubuntu.com/383314/
<jtv> whoa, thanks for splashing _that_ across my eyes in the wee hours of the night!
<sinzui> jtv: I did try to show only what was needed
<sinzui> jtv heat > 200 is great. I can lower it a bit
<wgrant> jtv: Hm, you probably don't want to use makeGPGKey. The user needs to hold the matching private to make an upload...
<jtv> sinzui: I'm just messing with you.  But AFAICS the part you posted accounted for 2 seconds out of 13.
<jtv> wgrant: drat.  Any alternatives you know of?
<sinzui> jtv: This listing is a heuristic. We just want the most problematic packages being seen by users
<jtv> sinzui: don't forget to count(BugTask.bug) instead of count(Bug.id) though
<wgrant> jtv: Not really.
<jtv> wgrant: so makeGPGKey doesn't even create a proper key pair and hide the private side somewhere?
<sinzui> jtv: did you ping-bomb me? I saw your name then pidgen went belly-up
<jtv> sinzui: I didn't ping you at all
<sinzui> jtv: yes, makeGPGKey() is bogus.
<EdwinGrubbs> bac: I have a question regarding the formatting of the upstream assoc portlet.
<wgrant> jtv: No.
<sinzui> jtv: I had to hack it last release to get a test to work
<wgrant> Maybe sinzui knows what to do.
<jtv> wgrant: would it be possible to let the user provide an email address that they have a proper key for?
<jtv> Just fork gpg to do the interesting work?
<sinzui> wgrant: repeat the question, I lost my scrollback when irc restarted
<jtv> (Or invoke some gpg library, but to the same effect)
<jtv> sinzui: a script needs to create a user with a proper GPG key
<wgrant> jtv: There is code already in LP to take a fingerprint and look it up. That's how we add gpg keys.
<wgrant> jtv: So you can find that code, and take a fingerprint on the command line.
<wgrant> The only caveat is that zeca needs to be running before that will work.
<jtv> drat
<jtv> what _is_ zeca, anyway?
<jtv> the keyserver?
<wgrant> See PersonGPGView.claim_gpg for the code you need to replicate.
<sinzui> pr_BR for test?
<jtv> wgrant: thanks
<wgrant> It's a mostly braindead test keyserve.r
<jtv> sinzui: meanwhile, did you say there was progress?
<wgrant> I would have liked to have had all this functionality in the script at the start, but I'd had the LP code for less than 24 hours and didn't really know my way around yet.
<wgrant> Thanks for adding it.
<sinzui> jtv: yes, filtering the lower bug.heat  will make the query fast enough for now.
<jtv> wgrant: pretty impressive.
<jtv> One day.
<jtv> sinzui: \o/
<sinzui> wgrant: jtv: I do not understand the gpg question, I only know that the kerys we make in testing are not unique. I had a test failure and I make a quick hack to the factory to make them unique enough for me to test
<wgrant> sinzui: jtv is extending one of my scripts to add a user's real GPG key.
<sinzui> ah
<jtv> or at least, _a_ real GPG key that the LP user can use.
<wgrant> True.
<jtv> sinzui: oh, I said count(BugTask.bug) but I guess that should be count(DISTINCT BugTask.bug)
 * sinzui tries that too
#launchpad-dev 2010-02-25
<jtv> Otherwise you're counting bugtasks instead of bugs.
<jtv> sinzui: still ok?
<sinzui> jtv: I can tune this. I think 150 should be the minimum heat threshold because that is the minimum number for a private bug
<sinzui> well, 250 might be best because private bugs are not community driven, but security are, which is 250
<jtv> sinzui: how does bug privacy factor into it all?
<sinzui> jtv: I am reading the heat calculator. It add 150 points
<jtv> oh ah
<sinzui> I am sure the numbers are arbitrary. so are mine. I just want to get the packages that need the most love to be listed first
<jtv> sinzui: do you see a big performance difference between those threshold?
<jtv> s
<sinzui> jtv: 67.82..27941.89 is better than 85994.40..85994.40
<jtv> sinzui: uh yes, looks like
<jtv> sinzui: you could multiply the bug count by some factor to compensate.  The ideal factor might be something like the average bug heat for bugs that stay below your chosen threshold.
<sinzui> jtv: maybe: The heat filter keeps the top packages 100 very distinct
<jtv> sinzui: you mean it sets them very far apart from the packages further down the list?
<sinzui> yes, there is less variation in the mid range of packages because their bug heat scores are 0
<jtv> In the context of this list, do you care a lot about those packages?
<jtv> Because maybe the list should just stop at some pseudo-arbitrary point.
<thumper> Is anyone tackling the merge conflict from stable -> db_devel?
<mwhudson> thumper: i'd got as far as thinking about it
<mwhudson> thumper: seems trivial to fix, i'll do it
<thumper> mwhudson: ta
<thumper> mwhudson: rs=me if you don't want to rs=you :)
<mwhudson> heh thanks :)
<jtv> sinzui: I'm off... good night & good luck!
<sinzui> me too
 * mwhudson lunches
<lifeless> https://wiki.ubuntu.com/PackagingGuide/PatchSystems
<lifeless> bah sorry
<thumper> mwhudson: Add a new code import result for success of partial import
<thumper> mwhudson: was that the commit that you fixed after
<thumper> ?
<thumper> mwhudson: just wondering about the qa status
<mwhudson> thumper: yeah, that's the commit that was fixed in two followup branches
<thumper> mwhudson: we should change it's status to FIXED
<thumper> mwhudson: I'm not sure I like the RCFIXED, as it wasn't an RC
<thumper> mwhudson: perhaps just move it to OK now
<mwhudson> thumper: ok
<mwhudson> thumper: well, the fix hasn't been QAed
<thumper> ah
<thumper> poo
<mwhudson> yep
<kfogel> jml: you on?
 * kfogel guessing not...
<kfogel> thumper: ah, but you might know: do we have any bugs filed for work on patch/branch equivalence?
<kfogel> or story pages, etc?
<kfogel> mwhudson: ^^   (if you know)
<mwhudson> kfogel: i don't know of anything formal
<kfogel> mwhudson: thanks
<kfogel> mwhudson: I'm looking at bugs and branches under ~jelmer, just to see
<thumper> kfogel: what exactly are you interested in?
<thumper> kfogel: worth skyping about?
<kfogel> thumper: hmrmrm.  It is worth skyping about, but I can't do a skype right now (need to wrap up this draft blog post about the new patch report feature).  I'm just listing, at the end of the blog post, some of the stuff we haven't gotten to yet, and that's on the list.  If there were a bug or a spec to link to, I'd link to it.
<thumper> I don't think there is a bug about it
<thumper> kfogel: one thing we haven't gotten around to
<thumper> kfogel: is automatically turning a patch into a branch if it applies to trunk cleanly
<kfogel> thumper: :-)  yes, that's one of the things I'm mentioning in the post
<kfogel> thumper: you can't see this, right?  http://blog.launchpad.net/?p=1356
 * thumper looks
<thumper> nope
<thumper> mwhudson: ping
<mwhudson> thumper: pong
<thumper> mwhudson: what version of pygments are we using ATM?
<mwhudson> thumper: good question
<thumper> mwhudson: easy to find out?
<thumper> mwhudson: is it an egg?
<mwhudson> no, i think it's from the package
<mwhudson> could/should switch to an egg though
<mwhudson> thumper: seems like 0.9-2
<thumper> hmm...
<thumper> bug 392406
<lifeless> mwhudson: why should?
<mup> Bug #392406: Update Pygments library to version 1.1 or later <codebrowse> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/392406>
<mwhudson> lifeless: easier for us to upgrade, more in line with what we do for everything else
<lifeless> mwhudson: I don't really like what we do for everything else ;)
<lifeless> its odd for us, a large chunk of an OS vendor, to ignore the OS for our products.
<mwhudson> lifeless: i am having a little difficultly composing a civil reply here
<lifeless> :>
<lifeless> troll:1 burdened-programmer:0 ?
<mwhudson> i guess
<lifeless> I wasn't intending to troll, but I realise picking one package isn't a good way to change the systemic approach
<mwhudson> right
<lifeless> so - sorry; and please get back to what you're doing
<mwhudson> thanks!
<thumper> mwhudson: bug 153779 still an issue?
 * thumper wanders off, back UK morning time
<mwhudson> thumper: not really i guess
 * mwhudson closes it
 * mwhudson EODS
<mwhudson> not the most productive day, oh well
<lifeless> heh
<lifeless> I have had similar
<lifeless> slow progress
<mwhudson> maybe staging will be magically up to date when i start tomorrow
<wgrant> It does seem particularly strange to me that a product from an OS vendor does not use a custom archive of dependencies, when a large part of the product is designed to produce custom archives.
<adeuring> good morning
<wgrant> When are the buildds expected back?
<wgrant> This is the longest queue I have ever seen.
<noodles775> I haven't heard. bigjools ^^
<bigjools> I don't know
 * noodles775 tries to find out.
<wgrant> 12 hours is way beyond "absolutely ridiculous" territory.
<bigjools> FWIW the PPA service only has 4 builders, the others are a bonus
<bigjools> (per arch)
<wgrant> I thought it had only three.
<bigjools> you could be right
<bigjools> we're getting some more but they will be reserved for daily builds
<wgrant> Hmmm.
<adiroiban> henninge: Hi. do you have time to land my last branch review or should I ask the OCR ? https://code.edge.launchpad.net/~adiroiban/launchpad/bug-509252-take-2/+merge/19484
<henninge> adiroiban: oh sorry, forgot about that. Sure. Does it have a commit message?
<henninge> adiroiban: ah, yes it does ;)
<adiroiban> henninge: yes. and I have updated it
<henninge> adiroiban: ok, landing it now.
<adiroiban> henninge: if you have time, can you also QA the fix for this bug https://bugs.edge.launchpad.net/rosetta/+bug/127171 ?
<mup> Bug #127171: Rosetta experts not allowed to "Change translators" <qa-needstesting> <Launchpad Translations:Fix Committed by adiroiban> <https://launchpad.net/bugs/127171>
<adiroiban> I don't have rosetta admin rights
<adiroiban> so I don't know how to test it on edge
<henninge> adiroiban: we can probably make you an admin on staging
<adiroiban> henninge: that is also OK
<henninge> danilo__, danilos: ^^?
<henninge> adiroiban: as a reminder: Please don't push to the branch anymore now that it is landing.
<adiroiban> henninge: ok
<danilos> henninge, /me is thinking... are there any privacy concerns? is rosetta-admins subteam of any generic team?
<adiroiban> danilos: I just need that to test that fix
<adiroiban> if some of you can qa it
<adiroiban> it's ok for me
<danilos> henninge, adiroiban: I've looked and it seems it's ok, I'll just do it and let Adi QA this :)
<danilos> adiroiban, haha, there's also "Poor Adi" account, nice name for what I assume is a testing privilege-less account :)
<danilos> adiroiban, anyway, you should be in rosetta-admins on staging now
<adiroiban> danilos: yep :)
<danilos> adiroiban, fwiw, staging will be refreshed sometime on weekend, so that's until when you've got the chance :)
<adiroiban> danilos: ok. well I will try to qa it now as I will not have time on friday weekend
<danilos> adiroiban, cool, thanks
<adiroiban> and I don't want to fix it when pqm is closed
<deryck> Morning, all.
<adiroiban> danilos: can you please target this bug for the next release https://bugs.edge.launchpad.net/rosetta/+bug/462013
<mup> Bug #462013: Don't ask me to import obsolete templates <qa-ok> <trivial> <ui> <Launchpad Translations:Fix Committed by adiroiban> <https://launchpad.net/bugs/462013>
<adiroiban> I think this is requred to me marked as âfix-releasedâ
<gmb> deryck: Do you have access to run a SELECT on staging?
<deryck> gmb, I do, yeah.
<gmb> deryck: Could you run http://pastebin.ubuntu.com/383617/ and paste me the output?
 * gmb neatly sidesteps a) the LOSAs being sprinty and b) the fact that staging is 10 revs behind where he needs it to be.
<deryck> gmb, sure.  doing now....
<gmb> Thanks.
<adiroiban> gmb: when you have time, can you please try a new landing for https://code.edge.launchpad.net/~adiroiban/launchpad/bug-522188/+merge/19395 ? Thanks!
<gmb> adiroiban: Sure, it's already on my list for today :).
<adiroiban> :)
<deryck> intellectronica, https://code.edge.launchpad.net/~deryck/launchpad/max-heat-by-target-511382/+merge/19998
<intellectronica> deryck: as it happens i already read through it and ran it working on the patch for updating the column, so r=me :)
<deryck> intellectronica, excellent!  easy-peasy. Thanks! :-)
<intellectronica> deryck: actually, scratch that. did you want me to review just the db patch, or the rest of the code?
<deryck> intellectronica, the code
<intellectronica> if the latter then i need to go through it more thoroughly
<intellectronica> ok
<deryck> intellectronica, yeah, the code.  stub and BjornT are on the patch.
<intellectronica> deryck: i'm wondering, isn't there a more general getOrCreate sort of thing for DSPs? this can't be the first time we do this in the code
<intellectronica> and even in your test you repeat the code twice
<intellectronica> ehrm, not in the test, in the distro model object
<intellectronica> maybe bigjools will know if there's something like that
<deryck> intellectronica, hmmmm, let me look at the diff....  and I greped for something and didn't find anything, and I was also following what is already there for another bit of code.
<deryck> intellectronica, ah, I see what you mean, to get the DB version of the class.
<bigjools> whu what?
<intellectronica> deryck: yes
<intellectronica> bigjools: is there a standard getOrCreate thingster for DistributionSourcePackageInDatabase ?
<bigjools> I have no idea, Gavin wrote it!
<intellectronica> a getOrCreateDistributionSourcePackageInDatabase :)
<bigjools> I'd be looking for it
<deryck> intellectronica, I copied the bits from what I assume Gavin had before, but I can refactor into a nicer getInDB method.
<deryck> I did think about that and never went back.
<intellectronica> deryck: if there isn't anything already in existence i'm fine with just filing a bug for doing it later (if you don't want to spend time on it now)
<intellectronica> deryck: why do we need a set on distroseries if all it does it delegate it to the distro?
<intellectronica> isn't it better to just not implement it?
<deryck> intellectronica, but in the distro series bug view you'll have to get at the max_bug_heat for that, right?  I thought it made using max_bug_heat less tricky if everything implemented it.
<intellectronica> deryck: yes, it makes sense to have a getter, but does it make sense to have a setter
<intellectronica> since we don't currently have any code that relies on it, i think it might be better to throw a NotImplementedError or something
<deryck> intellectronica, ah, right.  Yeah, I went back and forth on that.  I don't know that it makes sense.  You would never set on the series, but there's no harm in doing so.
<deryck> intellectronica, and given past bugs I fixed I have a nervousness about NotImplementedError :-)
<intellectronica> deryck: in fact, i'm not even sure we need a setter anywhere. as i just discussed with stub, it looks like we'll update this column from a trigger.
<intellectronica> though i guess at the very least it's useful for testing
<deryck> intellectronica, yeah, that was my reaction.  without a setter, from the test's pov it's just a useless attribute.
<intellectronica> deryck: so, i rather not set if it's done through delegation, because it can be confusing and lead to subtle errors. if you throw an error then whoever is trying to write to the attribute will become aware of the way it's implemented and do the right thing
<deryck> intellectronica, good point.  I'll do that then.
<intellectronica> cool
<intellectronica> deryck: r=me with that change
<deryck> intellectronica, excellent.  thanks!
<wgrant> adiroiban: What is the 'archive name' of a source package?
<adiroiban> wgrant: repository
<adiroiban> maybe the info is already available in API... but it is not obvious how to retreive it
<wgrant> A SourcePackage doesn't really have an archive.
<wgrant> Except potentially primary vs. partner.
<adiroiban> ah...
<adiroiban> looks like in the UI is called component
<adiroiban> https://edge.launchpad.net/ubuntu/karmic/+source/kubuntu-docs
<wgrant> Oh. that's something completely different to an archive or a repository.
<adiroiban> it may be...
<adiroiban> but on the Internet
<adiroiban> it is said that an Ubuntu package is part of main or universe repository
<wgrant> They are wrong :)
<wgrant> The term has always been component, both inside and outside Launchpad.
<adiroiban> yep. someone is always wrong on the Internet :)
<adiroiban> I will update the bug report
 * wgrant looks around for somebody who can be bribed into bumping a build score.
<deryck> three cheers for adeuring killing 4 bugs with one branch.  errr, 4 cheers, then.
<adeuring> deryck: thanks but the cheers should go to sinzui
<deryck> he does know our bugs well now.
<sinzui> adeuring: It would have been cruel for me to ask you to fix it without first finding out why the page has been broken for 18 months
<leonardr> wgrant, i'm looking at https://code.edge.launchpad.net/~wgrant/launchpad/export-das-chroot/+merge/19759
<leonardr> can you explain to me what exactly a chroot is? some sort of massive file stored in the librarian?
<james_w> leonardr: yep, it's a tarball of a filesystem stored on the librarian and used by the buildds when building packages
<wgrant> leonardr: "massive" being about 60MiB a the moment, but potentially some hundreds of megabytes.
<leonardr> wgrant: i haven't swapped in all the details yet, but to publish a librarian file on the web service you should export a Bytes field
<leonardr> the example i'm working from is IBugAttachment.data
<wgrant> leonardr: But that's not going to do anything useful like giving me hashes, is it?
<leonardr> wgrant: it will not give you hashes, but it will let users read the librarian file from within launchpadlib rather than having to fire up httplib2 to grab the librarian url
<BjornT> leonardr: what's stopping us from exporting ILibraryFileAlias?
<leonardr> BjornT: well, there's no code in lazr.restful for that because the library file alias is launchpad-specific
<leonardr> but there is code in lazr.restful for publishing 'bytes' fields as generic hosted files, and code in launchpad for hooking up LibraryFileAlias to that implementation
<wgrant> Could one not easily export ILFA with a Bytes field inside it?
<leonardr> wgrant, what information does your end-user want?
<wgrant> leonardr: I would like to be able to verify the content of the files, given that I downloaded them over HTTP.
<leonardr> so like the md5 sum?
<wgrant> Yes.
<leonardr> ok, i filed a bug at one point to publish a json representation of a hosted file resource which would contain that information
<wgrant> Thanks.
<leonardr> this would be a change to lazr.restful (and to the launchpad implementation of the hosted file), and publishing ILibraryFileAlias vs publishing IBytes would not change anything
<leonardr> i'm trying to find that bug
<leonardr> i really hate how that bug could have been filed in one of four projects
<wgrant> Hmm.
<leonardr> wgrant: see if this is what you want
<leonardr> https://bugs.edge.launchpad.net/lazr.restful/+bug/406912
<mup> Bug #406912: Publish a JSON representation of a binary file containing metadata <lazr.restful:New> <Ubuntu:Invalid> <https://launchpad.net/bugs/406912>
<wgrant> leonardr: Looks good to me.
<leonardr> ok, i'll respond to the review
<wgrant> Thanks.
<leonardr> wgrant: shall i add on to this bug that you requested the md5 sum of the file?
<BjornT> leonardr: i just find it odd that in python code we say that an attribute is an ILFA, while we have to export it as Bytes. seems easier and more intuitive if we could export ILFA themselves, with a Bytes field inside them.
<wgrant> leonardr: And the other hashes, too.
<leonardr> wgrant: i don't know what other hashes you're refering to
<leonardr> oh, sorry
<leonardr> i thoguht you meant hash as in the data structure
<wgrant> Ah, no, the cryptographic variety, sorry.
<leonardr> bjornt: i'm doing research to understand what you mean, but lazr.restful wouldn't care what is inside an exported field. if we got some way to export ILFA, then some representation of the ILFA would go into the entry's representation, and that's is
<leonardr> s/is/it/
<jml> good morning Launchpadders
<jml> wgrant, are you visiting New Zealand or perhaps some Pacific island?
<jml> wait.
<jml> wgrant, I mean, are you visiting Perth or perhaps some island in the Indian Ocean?
<wgrant> jml: No..
<wgrant> It's not yet 1:30.
<jml> wgrant, that's still kind of late :)
<wgrant> jml: I blame Enablement for making PPA builds take too long.
<wgrant> But it looks like my build still isn't going to happen for at least another hour, so i might as well sleep.
<leonardr> BjornT: the basic reason is that lazr.restful is not part of launchpad, so it can't know anything about ILibraryFileAlias
<leonardr> if this was a huge problem i could put some indirection in place such that ILibraryFileAlias could substitute for Bytes, but i just don't consider it that big a problem
<BjornT> leonardr: right. i was talking about how we export things in LP.
<leonardr> me too
<leonardr> there's a slight possibility you could get it to work by making ILibraryFileAlias subclass Bytes, but if that doesn't work i don't think it's worth putting in additional work
<leonardr> s/Bytes/IBytes/
<BjornT> leonardr: i don't get why we need to hide ILFA. Bytes and ILFA are different things. why does the exported API need to be different from our internal API?
<leonardr> BjornT: when the web service starts up, the launchpad classes are run through lazr.restful
<leonardr> when lazr.restful encounters a published Bytes field, it thinks "this is a hosted binary file"
<leonardr> and when the time comes to create a representation of it, lazr.restful performs a lookup, adapting the field to some interface (provided by launchpad)
<leonardr> when lazr.restful encounters a published ILibraryFileAlias field, it thinks "i've never heard of this", and crashes
<adiroiban> hi. do you know where I cound find some info about how to write API tests? I was looking at this API https://api.launchpad.net/+apidoc/#translation_import_queue_entries , and grepping the whole source tree after âtranslation_import_queue_entriesâ did not reveal any test
<BjornT> leonardr: but ILibraryFileAlias have (or should have) a Bytes field for its content, and ILibraryFileAlias could be exported just like any other interface, like IBug, IPerson, etc. no?
<leonardr> bjornt: ok, you want to publish ILibraryFieldAlias as an entry, not a field
<leonardr> i didn't understand that
<leonardr> bjornt: what you describe is technically possible, and it's the launchpad team's decision how to do that, but i think publishing the hosted file as a Bytes field would give a better end-user experience
<leonardr> bjornt: because right now you have, eg. a bugattachment object that has a 'data' field
<leonardr> right now you can do something analogous to open(bugtask.attachment.data).read()
<leonardr> if data was an ILibraryFileAlias then that would be open(bugtask.attachment.data.file).read()
<leonardr> but it should still be possible
<BjornT> leonardr: well, i see now that IBugAttachment is lying. it says that IBugAttachment.data is a Bytes field, yet BugAttachment.data is an ILibraryFileAlias
<BjornT> we should make sure that our content objects provide what the claim to provide
<leonardr> bjornt: unless ILibraryFileAlias can subclass IBytes i don't see any good way to resolve that problem
<leonardr> flacoste might be able to help
<BjornT> leonardr: i think there are a few options, although i'm not sure how feasible they are. one is of course to have our exported API to behave the same way as our internal API
<leonardr> actually, we could define a new interface in lazr.restful and have ILibraryFileAlias subclass that
<BjornT> leonardr: if we want to have it work like it works today, we could mark ILibraryFileAlias, and say that this is our hosted file, and provide an adapter, or something like that
<leonardr> bjornt: how about this. define lazr.restful.interfaces.IHostedFile
<leonardr> have ILibraryFileAlias subclass IHostedFile
<leonardr> change lazr.restful to treat IHostedFile the same way it treats Bytes
<leonardr> use IHostedFile instead of Bytes in launchpad
<BjornT> leonardr: what would IHostedFile look like?
<leonardr> BjornT: i don't think it has to have any fields
<leonardr> this is work i don't want to do, but it should work
<flacoste> leonardr: i think your idea sounds good, the end result is that we want to be able to export a Bytes or LibraryFileAlias field as a hosted file
<BjornT> leonardr: why do we call export_as_webservice_entry() when exporting interfaces, rather then subclassing marker intefaces?
<BjornT> leonardr: or a better question: does lazr.restful deal with ILibraryFileAlias, or LibraryFileAlias to make the decision that it's a hosted file?
<leonardr> BjornT: lazr.restful deals with Bytes to make that decision
<leonardr> the only Launchpad-specific code it uses in this process is lib.canonical.launchpad.rest.bytestorage.LibraryBackedByteStorage
<leonardr> basically it adapts the Bytes object to IByteStorage
<leonardr> and launchpad says the IByteStorage adapter to use is LibraryBackedByteStorage
<leonardr> LBBS hides all the details of LibraryFileAliases, how to create and delete them, etc
<leonardr> so the change i'm proposing is to replace Bytes with IHostedFile, and adapt IHostedFile to IByteStorage
<BjornT> leonardr: so it doesn't make sense to say that LibraryFileAlias is an IHostedFile then? if so, it seems like you should mark ILibraryFileAlias, rather then subclassing. but i'm not sure i follow completely
<leonardr> bjornt: it makes sense to say that LibraryFileAlias is an IHostedFile, but it doesn't make sense to tell lazr.restful anything about LibraryFileAlias
<adeuring> intellectronica: http://paste.ubuntu.com/383753/ For larger maxet values, you'll get a sort of "gap" after the relative heat 2/3
<adeuring> intellectronica: but i think ii have a proposal for a better function
<adeuring> just a second
<adeuring> intellectronica: http://paste.ubuntu.com/383761/ is at least "smoother"
<adeuring> intellectronica: I'm getting quite bad error in lib/lp/bugs/tests/../stories/bugtask-searches/xx-listing-basics.txt with your branch
<intellectronica> adeuring: damn
<intellectronica> let me check
<james_w> noodles775: still around?
<intellectronica> adeuring: yeah, i'm getting the errors too. looking at them now
<rockstar> abentley, your superseded comments stuff is made of awesome.
<abentley> Thanks, but you designed about half of it.
<abentley> rockstar, chat?
<rockstar> abentley, gimme a bit to clear the backlog of reviews here.
<abentley> rockstar, sure.
<deryck> intellectronica, I just merged you branch and will push up now.  If you're fixing errors could you merge our combined branch, or else branch new from me?
<deryck> intellectronica, since I fixed some of the changed elements.
<intellectronica> deryck: sure, i'm remerging your branch
<intellectronica> will ping you when i've got a fix for you guys to pull
<deryck> intellectronica, still running a test.... a minute more and will push up the version from me.  sorry.
<deryck> I started the test before I saw the scrollback here. ;)
<deryck> intellectronica, it's up now.  thanks!
<intellectronica> cheers
<jtv> eod! eod! eod!
<intellectronica> deryck: strange, i get ForbiddenAttribute: ('max_bug_heat', <lp.registry.model.distributionsourcepackage.DistributionSourcePackage object at 0xd224aec>)
<intellectronica> could it be that wer're missing the zcml to allow the new attribute or something?
<deryck> intellectronica, hmmm, maybe. and I just realized I haven't done the NotImplementedError you suggested yet.  Sorry.
<deryck> intellectronica, yeah, I bet DSP needs some zcml.  Sorry.  because I used zopeless layer in test it by passes that check, right?
<intellectronica> deryck: yeah, i now see that dsp uses the verbose attribute-by-attribute format for permissions
<intellectronica> i'll fix it in my branch, since you have to re-merge it anyways
<deryck> excellent.  thanks!
<intellectronica> whoa, it gets weirder. after fixing the permission problem i get 'InternalError: transaction is read-only'
<intellectronica> deryck: too early to tell, but my first guess is that something is not ok with your getOrCreate trick
<intellectronica> deryck: i pushed the fixes in my branch, if you want to look
<deryck> intellectronica, ok, thanks.
<intellectronica> deryck: my guess is we're using the read-only store, and then you try to create the dsp but you can't, so somehow we need to force it to use the read-write store
<deryck> ah, right
<deryck> intellectronica, ok, I'll look closely here in just a sec.  Trying to go over the latest bug about log() with adeuring.
<deryck> intellectronica, I think we may have to adjust your branch a bit, too.  Sorry to send more work you way again.
<adeuring> deryck: intellectronicagive me a second for a proposal of a better function
<intellectronica> deryck: in what way?
<deryck> intellectronica, in that the current branch will make anything 2/3 and up 4 flames.  We want something more like:  if max_heat is 100, use intervals 0..40, 41..70, 71..80, 91..100
<intellectronica> deryck: i don't think it will. see the documentation
 * deryck looks again
<intellectronica> but as i said this is hardly my expertise. if you have a better transformation go for it
 * deryck is playing with numbers in the doc test
<deryck> intellectronica, I'm actually getting the numbers of how many bugs will get what flames right now.  That might help us decide.
<intellectronica> cool. nothing like real data :)
<adeuring> deryck, intellectronica: http://paste.ubuntu.com/ (no lint check ;)
<intellectronica> adeuring: are you asking for input? ;)
<adeuring> intellectronica: well, that's a proposaal for other thresholds in calculate_heat_display()
<intellectronica> adeuring: right, but you only gave the url to the pastebin, not to the pasted item
<adeuring> intellectronica: argh, I am idiot... http://paste.ubuntu.com/383823/
<adeuring> another option are median statstics, BTW
<intellectronica> adeuring: yes, that looks like a good way to do it
<adeuring> intellectronica: how/when do we calculate heat values (or max_heat for targets)?
<adeuring> if this is done by a script, we could set the threshold based on actual data to build thresholds that match the data.
<intellectronica> adeuring: i'm creating a trigger that calculates it whenever a heat value is entered for a bug
<adeuring> so that we have the same number of bugs with 0, 1, 2,... flames
<intellectronica> yes, that was the original idea, but i think for now it's too complicated
<adeuring> intellectronica: ah, no, running median statistics tehre is not a good idea, I think.
<adeuring> intellectronica: I suspect that the function can be simplified....
<adeuring> let me see
<deryck> adeuring, intellectronica -- I'm preparing real numbers with both approaches for what ubuntu and malone (extreme and average) would do in terms of bug distribution in these scales.
<adeuring> deryck: cool, I'm curious
<intellectronica> deryck, adeuring: so i fixed the problem with getting max_heat from a dsp. you can now re-merge my branch
<adeuring> intellectronica: too late: EC2test is already running...
<adeuring> ...for my branch...
<intellectronica> adeuring: well, it will fail
<intellectronica> unless you fixed it independently
<adeuring> yours is already merged?
<intellectronica> deryck: ftr, i did that by removing the bit that creates the record in the getter. thinking about it it really shouldn't have been there, since it potentially allows anonymous users to create new records
<intellectronica> adeuring: no, that's the branch you presumably branched from. you notified me of errors and i fixed them.
<deryck> intellectronica, ok.  I'll look closer here in a minute.
<adeuring> intellectronica: yeah, but that's not what I got reviewed, and so deryck suggested I just go ahead with my branch.
<intellectronica> deryck, adeuring: i have to go out for a few hours soon. if you need me to update the scaling, or whatever, drop me a line and i'll do that later in the evening when i'm back. sorry to have to leave you like that.
<deryck> intellectronica, no worries.  So far, your scaling is looking better.  Though both have issues.  Will email.
<adeuring> intellectronica: seems that I'm also getting tired. I think I can proposed a more efficient varaint than what is in the pastebin, but i need a bit more time ;)
<intellectronica> adeuring: i'm not sure i understand. in my scale-heat branch there was one error that originated in my branch, and another error the originated in deryck's branch. i fixed both and pushed, and thought that if you branched from it you might want to re-merge.
<adeuring> intellectronica: no, I simply stared from a devel branch...
<deryck> intellectronica, abel's branch is based on devel.  He's just changing icons, no?
<adeuring> ...started...
<adeuring> deryck: well, no...
<adeuring> i changed also the HTML rendering
<adeuring> and the place where that happens changed
<adeuring> in intellectronica branch
<deryck> ah, right.
<intellectronica> adeuring: right, and i told you that you will have to look at my branch since the place where the rendering is done has changed in my branch
<intellectronica> heh
<deryck> heh.
<adeuring> argh... I missed that....
<deryck> adeuring, can you just kill your run. and we can merge your branch into intellectronica's.  Or vice versa.  And one of you can fix conflicts.  yes/no?  Likey? :)
<intellectronica> adeuring: well, if your branch lands in devel then i guess i'll just have to merge and resolve the conflict. no big deal
<adeuring> intellectronica: ok, thanks
<deryck> yeah, that works.
<deryck> intellectronica is waiting on me anyway :-)
<intellectronica> i'm not. i'm going to a concert :)
<deryck> heh
<deryck> intellectronica, adeuring -- so enjoy the night.  I'll send my numbers in email.  We'll stick with Tom's scale now and re-evaluate later if need be.
<deryck> we still end up with large numbers in 0 flames with both.
<intellectronica> deryck: but that's desirable, no?
<intellectronica> most bugs are not hot
<deryck> intellectronica, maybe so.
<deryck> intellectronica, I worry in Ubuntu that the drop off is too quickly.  For average projects it looks nice.  But we can revisit if we need.
<deryck> but the package view should be nice, I imagine.
 * intellectronica --> pere ubu
<EdwinGrubbs> sinzui, bac: Can you give me your feedback on these two mockups on the Upstream connections portlet? It looks like Brad recently moved name/value pairs onto a single line, which has some advantages, although it is inconsistent with other portlets. https://chinstrap.canonical.com/~egrubbs/upstream/
<sinzui> EdwinGrubbs: yes, he did, it is inconsistent, but it solved a space and scanning problem. We can do the same if we have the same problem again
<sinzui> EdwinGrubbs: I started to read the blueprint a few minutes ago
<bac> EdwinGrubbs: i didn't know i was introducing an inconsistency but i think it looks better
<sinzui> bac: I think I mentioned it was odd in my comments in the blueprint
<bac> sinzui: perhaps.
<EdwinGrubbs> sinzui, bac: I'm not against it, but it is a little confusing on how to move forward with it. It also might be nice to structure projectgroup/project/series so that it is left to right and possibly a little terser similar to the breadcrumbs.
<paki> hi all
<paki> i have a problem
<paki> how to show changelog in update manager?
<paki> anyone can help me?
<sinzui> paki: update-manager? the desktop application?
<paki> yes
<sinzui> paki: this channel is for building the launchpad.net website
 * sinzui checks his update-manager
<paki> uff
<paki> where could i ask?
<paki> is a very stupid thing
<sinzui> paki: You are asking why some packages do not include a meaning changelog? The answe most of the time is that packager did not prepare one
 * sinzui has looked at the package in launchpad often and not found a log
<paki> no, i'am a mantainer..can i include my changelog for update-manager?
<paki> how to do?
<sinzui> paki: okay, now I understand. This is a motu question. let me look for a channel
<paki> excuse my crappy english :S
<paki> ok, where is this channel?
<paki> ok, thanks
<sinzui> paki: try #ubuntu-motu
<paki> thanks sinzui :)
<EdwinGrubbs> sinzui, bac: I added two more screenshots. https://chinstrap.canonical.com/~egrubbs/upstream/
<sinzui> EdwinGrubbs: I think we need to be clear what is set by the series and what is set by the project. The reason fixing this is so effing hard is that the locations are not obvious, and changing a project changes the information for all series, and all packages that come from that series
<sinzui> EdwinGrubbs: have you noticed the new behaviour of the bug app? https://bugs.staging.launchpad.net/gedit
<mwhudson> good morning
<sinzui> EdwinGrubbs: bac: your blueprint extends scope to allow jumping to alternate services. I am not sure we want to do this, but it is worthy of discussion. Conversely, we (the registry team) may want to put a small barrier like the bugs app on Answers and Blueprints to make communities explicitly set the app locations
<EdwinGrubbs> sinzui: I had not seen the new behavior of the bug app before. That seems relevant to the involvement portlet but unrelated to the upstream connections portlet that I'm working on now.
<sinzui> Well, no, it is not
<paki> one question
<paki> how can update my pot file to launchpad?
<paki> excuse me..
<paki> upload my pot file..
<sinzui> EdwinGrubbs: Our goals are to capture this information for the Ubuntu community, and they need to see it. I agree that Answers and blueprints are separate issues. But by locking the first page (/gedit does have bugs, but they cannot be viewed until someone clearly states how bugs should be used)
<sinzui> EdwinGrubbs: Translations is already locked, but code is not
<paki> in documentation i don't find a button to upload pot file
<paki> explained in documentation but i don't find a button to upload pot file
<sinzui> EdwinGrubbs: and per your portlet, we do not have to allow links to alternate services for users, that does not help ubuntu. What we need to do is callout information that Ubuntu needs to know
<sinzui> paki: I think you want to use the #launchpad channel, which has many translations users tending it
<paki> ok thanks
<paki> bye
<sinzui> EdwinGrubbs: I think the project homepage and freshmeat url can stay where they are. Your design hints at a problem in the UI right now. I think we should move the external downloads link to the downlaods portlet.
<EdwinGrubbs> sinzui: as far as the information that Ubuntu needs to know, are you saying that just the Bugs and Source Code links in the involvement portlet should complain when they are not set?
<sinzui> EdwinGrubbs: And translations
<EdwinGrubbs> sinzui: I can updated the wiki and mockups and then send it to the mailing list.
<sinzui> okay
<sinzui> EdwinGrubbs: I really want to move the wiki link and change blueprints to use it when prompting for the blueprint URL. That will have to be on my own time though
<EdwinGrubbs> sinzui: do you have any opinions on these screenshots: https://chinstrap.canonical.com/~egrubbs/upstream/vertical_hierarchy.png    https://chinstrap.canonical.com/~egrubbs/upstream/horizontal_hierarchy.png
<sinzui> EdwinGrubbs: I saw those. As I said they do not make it clear what information is on the series, and what is on the project. That information will require different permissions to change...
<leonardr> gary, flacoste: there's a multiversion test case it's proving difficult to handle. i don't think it's worth it to make it work properly right now (though i have a test case that i can use to let people know that the behavior is the way it is)
<sinzui> EdwinGrubbs: ...Setting the project affects all series, and thus affects all packages from that series. Some series produce 10 different packages
<leonardr> let me know if you agree
<gary_poster> leonardr: back soon
<leonardr> the test case is this:
<leonardr> i'll use launchpad versions to make it concrete
<leonardr> we stop publishing mutator methods as named operations in version 1.0
<sinzui> EdwinGrubbs: Maybe this will not be an issue if project-reconfiguration via AJAX rocks.
<leonardr> named operations like 'transitionToStatus'
<leonardr> right now, it is impossible to publish *any* named operation called 'transitionToStatus' in version 1.0
<sinzui> EdwinGrubbs: I need to get another child from school. I'll be back in a bit
<leonardr> we can publish a transitionToStatus in version 2.0, but not in the version with the mutator named operation cutoff
<leonardr> again, i don't think it's worth it, and if you agree, i can get this branch into review today
<gary_poster> leonardr, back and reading
<gary_poster> leonardr: by "2.0" in your description you meant "any version that is not 1.0" right?
<bac> sinzui: have time for a quick call?  i'm a bit wedged on a testing issue
<sinzui> bac: sure, I'll get something to drink first
<bac> ok, call when ready
<leonardr> gary: yes, exactly
<gary_poster> leonardr: sounds fine
<leonardr> ok, cool
<leonardr> also ready for call when you are
<gary_poster> cool, will ping (on another call)
<sinzui> bac: I am ready
<bac> sinzui: ok, can you dial?
<sinzui> calling, skype say you do no exist
<mwhudson> james_w: around?
<mwhudson> wgrant: or you
 * mwhudson just wants to ask some questions about building packages
<bac> sinzui: can you hear me?
<sinzui> no
<bac> i could
<bac> stupid skype on phone
<james_w> hi mwhudson
<mwhudson> james_w: hi
<james_w> how do?
<mwhudson> james_w: i think i just wanted to ask about the mangling that needs to be done to a source package to build a different version
<mwhudson> james_w: pretty good!
<mwhudson> almost entirely unpacked in the new house...
<james_w> excellent
<james_w> the easiest way to do it is just to add a new changelog entry
<james_w> let me see how they do it in Debian
<mwhudson> this is where i get a teeny bit confused by questions like "what is a source package, again"
<wgrant> mwhudson: Hi.
<mwhudson> isn't the version specified in the .changes file?
<wgrant> The binary build process cannot see the changes file.
<mwhudson> ah ok
<mwhudson> and 'make' seems to be broken in devel
<wgrant> The changes file isn't really part of the package.
<mwhudson> Error: There is a version conflict.
<mwhudson> We already have: zope.interface 3.4.0
<mwhudson> anyone seeing this?
<james_w> mwhudson: the .dsc has the version, taken from the debian/changelog inside the unpacked source package (unless you are nasty)
<james_w> the .changes is a description of the upload, so is more about the delta than the state at the end
<wgrant> mwhudson: I saw that around a month ago. A clean fixed it.
<mwhudson> james_w: ah ok
<mwhudson> so mangling the dsc is *an* option, although possibly not a very good one
<wgrant> It won't work, will it?
<mwhudson> for https://dev.launchpad.net/BuildBranchToArchive/MultipleSeriesBuilds
<wgrant> It won't affect the unpackaged source, so binary builds will ignore it.
<wgrant> s/unpackaged/unpacked
<mwhudson> wgrant: make clean not helping :(
<wgrant> james_w: Soyuz has no work towards binary rebuilds yet, and supporting them would require quite a few changes on both slave and master.
<james_w> hmm
<james_w> Celso told me there was some work towards it
<wgrant> This is the same infrastructure that is needed for native backports.
<james_w> it might have been very early steps though
<james_w> wgrant: happen to know of any package in Debian with +b1 currently? ;-)
<wgrant> james_w: No. Why?
<james_w> just want to examine one
<wgrant> I'm pretty sure they just magically somehow gain the extra changelog entry at the start of the build.
<wgrant> It's added on the buildd side.
<james_w> yeah, that's what I think too
<james_w> just want to confirm before asserting it to be true
<wgrant> Examining sbuild 0.60.0 might work.
<wgrant> I would, but my connection really sucks at the moment.
<james_w> http://packages.debian.org/sid/i386/libfacile-ocaml-dev/download
<james_w> mwhudson: so yeah, we just tell the buildd to write a new changelog stanza as it starts the build
<james_w> as I said, not exactly elegant, but does buy us several other useful features
<wgrant> Presenting this in the UI is going to be interesting.
<mwhudson> james_w: oh ok
<wgrant> But it is a very useful feature.
<wgrant> And probably a bit less evil than mutating the source.
<james_w> yes
<james_w> easier to implement
<wgrant> james_w: Do you know how distro people feel about binNMUs?
<james_w> but it still leaves us without no-change rebuilds and native backports
<wgrant> I've heard some negativity around that area before.
<james_w> it's mixed I think
<wgrant> Does it?
<james_w> done right I think it would be accepted
<james_w> wgrant: you are saying we could use the same infrastructure to do them?
<wgrant> james_w: Native backports and multi-series recipe builds without source changes seem to be the same concept.
<mwhudson> james_w: for the "What happens when we open a new development release, there will be zero recipes associated with that series for people to choose from." question
<mwhudson> i think that "build for all supported series" should be a distinct option from (currently) build for "lucid, karmic, jaunty and hardy"
<mwhudson> and uh intreprid i guess
<wgrant> I'm not sure we do.
<wgrant> There's probably no point doing daily builds for the first little while in each series.
<james_w> mwhudson: yeah, the question may be moot, but we need to consider what happens when we open a new seris
<james_w> wgrant: quite possibly
<mwhudson> wgrant: well, maybe, but that's not actually in too much opposition to what i said
<james_w> wgrant: I was going to say that it doesn't fit no-change rebuilds as it rebuilds the source package, so it may not be no-change, but that's the status quo anyway
<mwhudson> maybe not "all supported series" but "some default set of series"
<wgrant> james_w: It doesn't really rebuild the source package; it unpacks it, adds a changelog entry, and builds a binary from it.
<james_w> wgrant: sorry, status quo in Ubuntu
<james_w> apt-get source; dch -i; debuild -s; dput
<wgrant> james_w: Oh, you meant the status quote may not be no-change.
<wgrant> True.
<james_w> yes
<james_w> we hope it is no substantive change
<wgrant> But autofoo or blah blah.
<james_w> wgrant: but regardless of the implementation we will be able to do no-change rebuilds server side with build from branch :-)
<wgrant> True.
<mwhudson> oops, i probably didn't want to delete download-cache in my trunk directory...
<EdwinGrubbs> maxb: ping
<mwhudson> hm
<mwhudson> thumper: does 'make' work in devel for you?
#launchpad-dev 2010-02-26
 * thumper looks
<thumper> mwhudson: first try no, updating source code and download cache now
<thumper> mwhudson: can you look at   https://dev.launchpad.net/BuildBranchToArchive/MultipleSeriesBuilds and determine if we know enough to change the model?
<mwhudson> thumper: i think we know basically enough
<thumper> ok
<mwhudson> thumper: i guess it reopens the question of "what namespace do recipes live in"
<thumper> mwhudson: can you plan a branch count (1 or more as necessary) to change the model?
<thumper> mwhudson: I think we are going to have recipes at ~user/distro/+source/packagename/+recipe/name
<mwhudson> thumper: i'd have thought one would be enough
<thumper> ok
<mwhudson> thumper: ok that makes sense
<thumper> mwhudson: the interesting bit is how to make ~user/distro and ~user/distro/+source/packagename (and possibly ~user/distro/distroseries and ~user/distro/distroseries/packagename) useful objects
<thumper> mwhudson: consider ~user/project
<mwhudson> thumper: 'useful objects' ?
<mwhudson> i can see useful pages
<thumper> mwhudson: at code.lp.net/~thumper/distribution ?
<thumper> mwhudson: PersonDistro c.f. PersonProject
<thumper> mwhudson: PersonDistroSeries
<thumper> et al
<mwhudson> right
<mwhudson> maybe i'm unimaginative, but that doesn't seem very interesting
<thumper> what useful pages can you see?
<mwhudson> it's just work isn't it?
<thumper> yes
<mwhudson> thumper: i think we're saying the same thing in different ways, ignore my objection
<thumper> ok
<thumper> jml suggested there may be some multi-object adapter thing that we could use instead of having explicit objects
<thumper> so (Person, Product) instead of PersonProduct
<mwhudson> oh right
<thumper> I'm not so sure there is
<thumper> or even if there is that it is a better solution
<thumper> simple is better than complex
<mwhudson> well
<mwhudson> um
<thumper> although which is simple is a matter of opinion
<mwhudson> i think you have to have something like PersonProduct
<mwhudson> that you can hang a view off
<thumper> mwhudson: btw, make worked with updated download-cache
<mwhudson> it doesn't have to be anything more than just a container object though
<mwhudson> thumper: :(
<mwhudson> well i guess yay that launchpad isn't broken for _everyone_
<thumper> what are your symptoms?
<mwhudson> mwh@grond:trunk$ make
<mwhudson> utilities/shhh.py PYTHONPATH= ./bin/buildout \
<mwhudson>                 configuration:instance_name=development -c buildout.cfg
<mwhudson> While:
<mwhudson>   Installing.
<mwhudson>   Getting section i18n.
<mwhudson>   Initializing section i18n.
<mwhudson>   Installing recipe z3c.recipe.i18n.
<mwhudson> Error: There is a version conflict.
<mwhudson> We already have: zope.interface 3.4.0
<thumper> hmm..
<thumper> my buildout bit isn't running
<mwhudson> it seems to be finding the system zope.interface
<thumper> and now I'm wondering what'll happen if I remove my eggs
<mwhudson> thumper: dpkg -l python-zopeinterface
<mwhudson> ?
<thumper> un
<mwhudson> !
<thumper> however in a python shell I can import zope.interface
<mwhudson> !!
<mwhudson> lots of things seem to depend on it here
<thumper> ii  python-zope.interface                   3.5.2-1
<mwhudson> hmmm
<mwhudson> HMMM
<mwhudson> well
<mwhudson>  apt-get install python-zope.interface
<mwhudson> which replaces the package
<mwhudson> and now it works
<mwhudson> W
<mwhudson> T
<mwhudson> F
<mwhudson> anyway
 * thumper shrugs
<thumper> weird
<mwhudson> well about four hours later than it should have been, let's test that git import out...
<thumper> mwhudson: :)
<thumper> mwhudson: I've set my preferred email back to the @c.c one
<thumper> mwhudson: we'll see if I get an email this time
<thumper> mwhudson: perhaps we should add an --email-from tag too
<thumper> mwhudson: that way it might work for the community people too
<wgrant> thumper: It works fine if I run it myself, though.
<wgrant> Probably because I have no SMTP server configured.
<thumper> wgrant: could be
<wgrant> And my mail server is not sufficiently SPF-policing.
<thumper> wgrant: if the canonical one is set to use the canonical smtp server
<thumper> wgrant: it only allows emails from a canonical.com address
<thumper> wgrant: and drops others
<wgrant> thumper: It doesn't even allow emails to a canonical.com address from a non-Canonical one?
<thumper> wgrant: it may not be the general one... not sure
<thumper> wgrant: maybe?
<wgrant> Yay email.
<mwhudson> ec2 test should just drop a letter in the mail
<mwhudson> i'm sure there's a printer somewhere in amazon's data centre?
<thumper> :)
<thumper> mwhudson: an unproductive afternoon just got worse with visitors
<thumper> mwhudson: I'm going to get a real coffee
<mwhudson> thumper: ok
<mwhudson> thumper: i'm going to be finishing a little early today, btw
<thumper> mwhudson: ok
<thumper> like 4?
<mwhudson> thumper: about 4:30
 * thumper nods
<thumper> ok, sounds fine
<mwhudson> cool
<mwhudson> ec2 demo seems pretty broken by the openid changes
<wgrant> mwhudson: What goes wrong?
<wgrant> mwhudson: Does it hardcode hostnames like rocketfuel-setup?
<mwhudson> wgrant: i got fed up and didn't really dig
<mwhudson> but pages that require you to log in oops
<wgrant> Yeah, it's not adding testopenid.dev to /etc/hosts.
<mwhudson> oh right
<mwhudson> hm it is when creating a new image
<mwhudson> maybe a new one is needed
<wgrant> Does the image creation use rf-setup?
<mwhudson> wgrant: no
<mwhudson> rf-setup does rather more than is needed i think
<wgrant> Ah, I didn't see the devscripts change in the diff.
<wgrant> Ah, because the diff is 5000 lines.
<jml> thumper, wrt "simple is better than complex", "simple" is often defined as "fewer interacting entities", in which case a multi-adapter would be simpler
 * thumper pokes jml in the eye
 * thumper looks around to see who saw that
<jml> thumper, entia non sunt multiplicanda praeter necessitatem
<thumper> jml: when are you back at work?
<jml> thumper, I was back all today.
<thumper> ah, home again?
<jml> thumper, no, in Massachusetts
<lifeless> jml: em or um?
<jml> thumper, working w/ cody-somerville & kiko, talking about OEM requirements and things
<jml> lifeless, em.
<thumper> jml: ok
<cody-somerville> s/OEM/NewCore (the new shiny) ;)
<cody-somerville> err
<lifeless> 'thingy'
<thumper> hi cody-somerville
<thumper> cody-somerville: still working?
<thumper> getting kinda late isn't it?
<cody-somerville> It is
<cody-somerville> 21:19
<cody-somerville> Just got back from dinner.
<thumper> and you couldn't wait to see what happened while you were out?
<jml> thumper, well, I couldn't
<jml> thumper, and also there's nothing like taking a break from my job and doing a spot of document writing and process improvement for an open source project :P
 * cody-somerville is an addict to work. :(
<thumper> jml: I've got my own pet project now
<jml> thumper, does it involve web and dvcs?
<thumper> jml: yeah
<thumper> jml: looking at mako for templating
<thumper> jml: looks more intuitive to me than genshi
<thumper> jml: still using twisted.web
<jml> thumper, what does django use normally?
<thumper> kid I thought
<jml> thumper, have you looked at nevow?
<thumper> or maybe that was something else
<thumper> jml: nope
<thumper> mako is super fast
<jml> thumper, I'm not suggesting that you _use_ it, mind, just that it's worth a look
<thumper> and what the python website uses
<thumper> jml: understood
<thumper> jml: there are a gazillion different templating systems out there
<thumper> I've been playing around with python 3.1 a bit too
<jml> thumper, nevow is one that was built to work with twisted.web
<wgrant> Django uses its own thing.
<wgrant> I like genshi more than mako, I think.
<wgrant> but there's not much in it.
<thumper> wgrant: why?
<thumper> I'm not strongly stuck on any one
<wgrant> I'm not sure.
<thumper> I just want something that can easily apply a common skin to pages
<thumper> I'm not wanting to invest a lot of time into it
<thumper> (just yet)
<thumper> jml: do you know of a python library that renders moin syntax?
<lifeless> thumper: like, oh, moinmoin ?
<thumper> lifeless: I heard that it wasn't simple to extract the rendering code from the entire wiki project
<jml> thumper, there's a name on the tip of my tongue (other than moinmoin)
<thumper> lifeless: but I haven't looked
<lifeless> thumper: I'd look; theres been a lot of work done on moin
<jml> thumper, also, I humbly suggest that you should use mediawiki formatting
<thumper> jml: I was going to have a choice
<jml> thumper, good good.
<thumper> jml: what is the name of the mediawiki format?
<thumper> is there another name?
<jml> thumper, no, I don't think so
<jml> thumper, there's a wiki interchange library thing that I'm struggling to remember the name of
<thumper> jml: I think I did a code import for that ...
<thumper> jml: creole?
<jml> thumper, yeah, that's it.
<thumper> lp:creoleparser
<jml> thumper, phew, I'll be able to sleep now.
<mwhudson> thumper: is https://dev.launchpad.net/Code/PostThreeDotO much relevant these days?
<thumper> mwhudson: I don't think so
<thumper> mwhudson: we know about it all
<thumper> mwhudson: and they are tracked elsewhere
<mwhudson> thumper: right
<thumper> mwhudson: so kill it
<mwhudson> thumper: delete it? replace with a link to somewhere else?
<thumper> umm...
<thumper> not sure where to link it to
<thumper> is there a general four dot oh page somewhere
<thumper> or perhaps link to the dev wiki strategy pages?
<mwhudson> https://dev.launchpad.net/VersionFourDotO/Themes ?
<thumper> yeah, seems reasonable
<thumper> mwhudson: I'm off to go shop for BBQ
<thumper> mwhudson: see you monday
<mwhudson> thumper: see you monday, have a good weekend
<adeuring> good morning
<jelmer> mwhudson, hi
<jelmer> stub: ping
<adeuring> when i run "make build", i get "Error: There is a version conflict.
<adeuring> We already have: zope.interface 3.4.0". Any suggestions how to fix this?
<wgrant> mwhudson had that this morning.
<adeuring> wgrant: tahnks for the hint!
<wgrant> adeuring: It looks like he might have fixed it by install the python-zopeinterface package.
<wgrant> Er, python-zope.interface
<adeuring> wgrant: thanks! that helped indeed
<wgrant> Urgh. It shouldn't.
<jtv> Did the +login page just stop working in devel?
<wgrant> jtv: It's OOPSing?
<jtv> wgrant: yup, for me it is
<wgrant> jtv: Add testopenid.dev to /etc/hosts.
<jtv> wgrant: thanks
<wgrant> An email should really have been sent about that.
<jtv> wgrant: thanks, that did the trick
<wgrant> Great.
<stub> jelmer: pong
<jelmer> stub: Any chance you can review this patch that changes the db: https://code.edge.launchpad.net/~jelmer/launchpad/bug471148/+merge/20142 ?
<allenap> Is it really week 3? I've been out of touch for a while :)
<allenap> Yes, yes it is. (Looked at the milestone.)
<stub> jelmer: What architectures does an archive get built for if there are no entries in ArchiveArch ?
<jelmer> stub: Just the unrestricted architectures
<stub> So we add this flag to avoid needing 8 entries in ArchiveArch for every archive. I see.
<jelmer> stub: yep
<stub> We are assuming we won't need further granularity like 'build hppa in lucid only'
<stub> ?
<bigjools> stub: that's already handled through distroarchseries
<al-maisan> stub: this is not a requirement at present; if needed we could add a DistroSeries FK to ArchiveArch
<al-maisan> to put the restricted processor family override into a distro series context
<wgrant> How does this work with require_virtualized?
<al-maisan> wgrant: the latter is orthogonal to this
<bigjools> al-maisan: no FK needed, distroarchseries man :)
<al-maisan> je
<al-maisan> sorry
<al-maisan> bigjools: I guess we're fine for the moment
<bigjools> al-maisan: yep
<wgrant> Ah, so this is for restricting the architectures of non-virtual PPAs?
<jelmer> stub: thanks for the review
<stub> np
<al-maisan> I don't quite see a use case where we'd want to have a PPA with builds for armel in say karmic but not in lucid
<bigjools> wgrant: it's for restricting for any archive
<bigjools> virtuality is orthogonal
<intellectronica> i can't build after branching from db-devel because of "Couldn't find a distribution for 'lazr.restful==0.9.21'". anyone knows what's going on?
<intellectronica> ah, nm, rocketfuel-got it
<bigjools> al-maisan: that's a very valid use case actually
<bigjools> for example, lpia is not in lucid
<al-maisan> the lpia case is not really so pertinent since lucid has no lpia DistroArchSeries anyway
<jtv> bdmurray: your branch has landed.  Now you need to Q/A it.  I'm not sure what the process for keeping track of that is in this case; in Translations we've been using an experimental process.
<jtv> bdmurray: it needs to be marked as Q/Aed in order to be included in the release.
<wgrant> Argh, all the buildds are gone again.
<noodles775> gar, the cherry-pick is still waiting to land :/
<noodles775> wgrant: the i386 ppa builders are all full?
<wgrant> And amd64 :(
<noodles775> wgrant: ah, I thought you meant that the buildd manager had died again... *phew* :)
<mwhudson> jelmer: incremental imports seem to fail with
<mwhudson> AssertionError: Invalid sha for <Commit 8ab9c1e964cd0c36bb79f95fc229e49b7e296684>: 4638aef40ba9ebb9734caeed1f373c24015259fd
<mwhudson> type errors sometimes
<jelmer> mwhudson: is this just of the kernel or of other branches as well?
<mwhudson> (only running locally, staging is still out of date
<mwhudson> )
<mwhudson> jelmer: kernel and gedit
<jelmer> mwhudson: if it is just in the kernel, it might be an unrelated bug
<mwhudson> only on the import of revs 3000-4000 too, not very far in
<mwhudson> hm, it's not actually during revision fetch
<jelmer> mwhudson: The way I usually debug this is to compare the contents of the objects
<jelmer> mwhudson: git cat-file <id> and bzr git-object <id>
<mwhudson> jelmer: well at 5 to midnight on a friday i'm mostly thinking about bed...
<jelmer> mwhudson: ahh, right :-) sorry, I hadn't considered the timezone
<jelmer> mwhudson: is there an easy way to reproduce this outside of launchpad, using command-line bzr-git?
<mwhudson> jelmer: not sure tbh
<jelmer> guess we should get that limit parameter upstream to make testing easier ;-)
<mwhudson> jelmer: maybe you can try gedit locally?
<jelmer> yeah, sure
<mwhudson> https://code.launchpad.dev/+code-imports/+new
<mwhudson> then run ./cronscripts/code-import-dispatcher.py as required
<mwhudson> with make run_codehosting in another window
<jelmer> ah, thanks
<mwhudson> if it doesn't work let's not worry too much, i can try to debug it on monday
<mwhudson> and if we can't fix it we can always neuter the feature by setting the revision limit really really high
 * mwhudson zzzs
<deryck> Morning, all.
<gmb> Is pushing branches to launchpad being slow for anyone else? I'm getting ~1KB/s, apparently, but my connection seems otherwise quite nippy.
<bigjools> morning deryck
<bigjools> gmb: bit slower than normal, yeah
<gmb> Hrm.
<gmb> (This is where my knowledge about how codehosting works ends)
<intellectronica> stub: around? got a mo to review my db patch adding a trigger to update max_bug_heat?
<stub> k
<intellectronica> excellent, just creating an MP
<intellectronica> stub: https://code.edge.launchpad.net/~intellectronica/launchpad/update-max-heat/+merge/20204
<intellectronica> gmb: did you have any thoughts on how to implement bug #505849 ?
<mup> Bug #505849: Bug heat should decay over time <story-bug-heat> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/505849>
<gmb> intellectronica: Some. deryck and I chatted about this and our general idea was that heat should decay by 10 points for each week since a bug's last been touched.
<stub> Hmm... we don't have max heat for a distroseries?
<intellectronica> adeuring: looks like the new flame icons implementation is a bit broken. on a 1024 wide screen they new break and appear on a new line
<intellectronica> stub: no, we only use the distro's max_heat
<adeuring> intellectronica: sigh...
<adeuring> intellectronica: do you mean listings or the icons on a bug index page?
<intellectronica> gmb: gotcha. sounds like an easy solution.
<gmb> adeuring: See bug 528374
<intellectronica> adeuring: let me take a screenshot
<mup> Bug #528374: Flames misplaced on bug report page <Launchpad Bugs:New> <https://launchpad.net/bugs/528374>
<gmb> intellectronica: mpt already did that for you :)
<intellectronica> thanks mpt
<stub> intellectronica: I think the distribution max_heat is incorrect, as BugTask's linked to a distroseries are not directly linked to a distribution (Only one of BugTask.distroseries and BugTask.distribution can be set)
<intellectronica> stub: oh, good point. didn't realise that.
<stub> (I'm sure there was a reason for that, but I doubt it was a good one and hopefully not mine ;) )
<stub> intellectronica: Do you want to fix or should I?
<intellectronica> stub: go ahead and fix it if you can.
<stub> DistributionSourcePackage.max_bug_heat calculation doesn't support multiple distributions yet either, but I guess we don't have to worry about that now :)
<intellectronica> :)
<adeuring> intellectronica: I can't reproduce this... which browser are you using?
<intellectronica> adeuring: chrome
 * adeuring nedds to install chrome...
<intellectronica> adeuring: oh, and in FF it's even worse. the flames appear to the left of the bug info line instead of to the right
<adeuring> intellectronica: I thought that was intended ...
<intellectronica> adeuring: oh, i didn't realise that. why did we change the location?
<adeuring> intellectronica: i have no idea what changed or when it changed.
<intellectronica> adeuring: i thought it was your branch, since it was ok until yesterday, but maybe something else
<intellectronica> the light icons do look much better, though
<adeuring> intellectronica: well, the icons were in firefox left of "bug #123 reported" even before my change
<mup> Bug #123: There's no direct way to see the project info when translating it <Launchpad Translations:Fix Released> <https://launchpad.net/bugs/123>
<intellectronica> adeuring: huh. i can't believe we missed that for so long
<deryck> adeuring, intellectronica -- the flames aren't orginally left of the reporter info.  See:  https://bugs.staging.launchpad.net/ubuntu/+source/nspluginwrapper/+bug/141613
<mup> Bug #141613: npviewer.bin crashed with SIGSEGV <apport-crash> <apport-failed-retrace> <iso-testing> <metabug> <npviewer> <npviewer.bin> <nspluginwrapper> <nspluginwrapper (Ubuntu):Confirmed> <nspluginwrapper (Mandriva):Fix Released> <https://launchpad.net/bugs/141613>
 * adeuring is now completely confused...
<stub> intellectronica: looks like it all needs refactoring. If a BugTask.product changes for instance, both the old product and the new product need their maximums updated.
<stub> oh.. hang on... i'm confusing myself
<deryck> heh, "7 out of 4 heat flames"  oops. :-)
<stub> Yer - if we do this with triggers, we need triggers on both bug and bugtask, because changes to bugtask link and unlink bugs from stuff and that stuff needs their maximums updated.
<stub> And technically on distroseries too incase a distroseries.distribution gets changed, but I think we can ignore that one.
<intellectronica> stub: oright. i'll add triggers for the change of target. i guess in that case it would make sense to move most of the code out to a procedure and use the triggers to call that?
<stub> If there is enough that can be factored out, sure.
<stub> I'd go for the cron job personally :)
<wgrant> bigjools: Isn't that branch going to cause all the copy archive binaries to land in universe, so the second run won't actually use them?
<henninge> bigjools: Hi ;)
<henninge> bigjools: can you help me with build farm jobs or is that somebody else's domain?
<adeuring> intellectronica: can you take care of the flame icons buG? I OCR today and have some reviews in the quere...
<intellectronica> adeuring: i think i can have a look at it after i finish refactoring my max_heat db patch
<henninge> wgrant: Hi :)
<adeuring> intellectronica: thanks!
<intellectronica> deryck: does that sound like a reasonable plan to you? ^^^^^
<wgrant> henninge: Hi.
<henninge> wgrant: are you knowlegable on build farm jobs?
<wgrant> henninge: Probably.
<wgrant> I know most of how it works.
<intellectronica> it might mean not getting any other heat bugs done, though :/
<henninge> wgrant: I'd like to try out our template generate job in as much the same way as it will be run in real life.
<bigjools> wgrant: it's not applying any overrides
<stub> Error: There is a version conflict.
<stub> We already have: zope.interface 3.4.0
<stub> ?
<bigjools> maybe it should look in the primary archive
<stub> Buildout error
<henninge> stub: apt-get install python-zope.interface
<wgrant> bigjools: It should.
<bigjools> meh
<wgrant> I think.
<bigjools> I did have a nagging thought about it
<wgrant> henninge: Do you have a local buildd set up?
<henninge> wgrant: I don't think so. What would that entail?
<henninge> I have build the launchpad-buildd package, or I can at least.
<wgrant> henninge: So, you'll need to install that. What sort of arguments does your generation job take?
<henninge> wgrant: a branch url to fetch the package branch from. That's all.
<stub> henninge: Seems to be working. ta.
<deryck> intellectronica, looking back
<wgrant> henninge: OK, so, you can probably use Python's xmlrpclib to start the job without involving the horrible buildd-manager and Launchpad at all.
<henninge> stub: oghers have gone before me ... ;-)
<deryck> intellectronica, so the plan being you'll refactor the trigger branch and then fix the icons?
<intellectronica> deryck: yes
<stub> buildout isn't reporting an error code on failure, which screws things up in the makefile...
<deryck> intellectronica, good plan.
<henninge> wgrant: in real life the build slaves are ec2instances, right?
<bigjools> no
<henninge> oh
<wgrant> henninge: No. They're mostly Xen machines in the DC.
<wgrant> The rest are real machines in the DC.
<henninge> ok, but they are like their own machines.
<wgrant> Yes.
<henninge> but you are saying I should not try and simulate that part, just run it on my machine.
<wgrant> Right.
<henninge> I mean, "setting up a buildd" is not setting up a xen instance or the like.
<henninge> ok
<wgrant> No, no. You just need to install the package.
<wgrant> It'll run a daemon which makes the Soyuz tests fail, but that's about it.
<henninge> good. Which is the top-level script I need to start?
<wgrant> Have you installed launchpad-buildd?
<wgrant> That should have started it already.
<henninge> oh, I see. NOt atm, I had it installed in a pbuilder chroot. I'll try that again and look at it.
<wgrant> That's fine too.
<wgrant> The daemon just needs to be running somewhere.
<henninge> so when it runs I can use xmlrpc to have it start my job.
<henninge> ?
<wgrant> Right.
<henninge> wgrant: ok, thanks. Let me see how far I get with that.
<wgrant> import xmlrpclib; xmlrpclib.ServerProxy('http://localhost:8221/rpc').build('1-1', 'translation-templates', 'sha1ofthechrootinyourlibrarian', {}, {'branch_url': 'http://url.to/branch'})
<henninge> wgrant: thanks, that'll help a lot ;-)
<henninge> wgrant: where do I find 'sha1ofthechrootinyourlibrarian' ?
<wgrant> henninge: Grab http://launchpadlibrarian.net/39617064/chroot-ubuntu-lucid-i386.tar.bz2, get it into your librarian, and use f1f10b8402ed686aaf0307676c76f07b45af2a09 as the SHA1.
<ricotz> salgado, hello
<salgado> hi ricotz
<ricotz> salgado, is it possible for you to restart this build https://edge.launchpad.net/~ricotz/+archive/testing/+build/1532157
<ricotz> it seems to be jammed
<salgado> ricotz, I don't seem to have permission to do that.  should I have?
<ricotz> salgado, sorry, thought you have
<ricotz> can you delegate this request to someone who can
<ricotz> bigjools, hello ^
<bigjools> ricotz: let me lookl
<ricotz> thank you
<bigjools> ricotz: why do you think it's jammed?
<ricotz> cause its running for an hour and doesnt show any progress
<bigjools> ok I'll investigate
<ricotz> this build should take less than 10min like https://edge.launchpad.net/~ricotz/+archive/testing/+build/1532162
<bigjools> ricotz: it's reset and on a different builder now
<bigjools> see how it goes
<ricotz> bigjools, thank you!
<wgrant> henninge: Any luck?
<techrascal> we are a bunch of undergrads and want set up a launchpad clone for projects we have in our college
<techrascal> we have a server access with jailed ssh
<techrascal> which doesnt give us sudo access and probably not even yum install kind of operations
<techrascal> can anyone help me identify the bare minimum access we need for installing launchpad source code on a server
<jml> techrascal, you'll need root, I'm afraid.
<jml> techrascal, can you not just use launchpad.net?
<deryck> BjornT, ping
<jtv-afk> Why doesn't gpgme find my keys when I "make harness"?  (It does when I ./bin/py)
<jtv> deryck: on an entirely arbitrary whim, I decide that you sound like the sort of person who would know this.
<BjornT> deryck: pong
<deryck> Hi BjornT.  So I need to rollout a new lazr-js to update with a change Tom had.  Want to make sure I remember right...
<deryck> BjornT, it's just roll an sdist, ci it to download-cache, and update the versions.cfg in an lp branch.  is that right?
<gmb> Anyone know how to fix this http://pastebin.ubuntu.com/384423/ when I'm trying to run `make`?
<gmb> rocketfuel-get and make clean haven't fixed it.
<deryck> gmb, do you need to clean up download-cache by hand, i.e. remove something from it that bzr up is trying to add?
<deryck> pure guess that is.
<gmb> Hmm.
<stub> (19:09:04) henninge: stub: apt-get install python-zope.interface
<stub> gmb: ^^ I got a similar error, but zope.interface. That was the solution.
<gmb> stub: I'll give that a shot, thanks. (Except with the zope package, obviously)
<BjornT> deryck: yes, that's right
<deryck> BjornT, thanks
<mars> ok, there is something fishy with the Repeat view load times of BugTasks on edge: http://www.webpagetest.org/result/100226_5F6Q/#run3
<mars> They have increased a lot since December: http://www.webpagetest.org/result/091221_3RNM/#run2
<mars> is the server under load or something?
<mars> Dec time to first byte: 0.955s
<mars> Feb time to first byte: 2.961s
<gmb> Grrrrrrr.
<mars> Must be something messed up with the cached view code on webpagetest's end.  The first-view times are still sane across the board.
<gmb> gary_poster: I'm running into this whilst running make: http://pastebin.ubuntu.com/384423/. stub suggested apt-get installing zope.testing, but that didn't help. My download-cache is up to date; any suggestions about how I can get past this?
<gary_poster> gmb: yes.  As a work-around, manually edit your bin/buildout file to have a " -S" at the end of the shebang line.  I have a (rather invasive, but fully reviewed) branch that I'll land at the beginning of next cycle to address.  If you don't plan to land this until next cycle then you could merge that.  Getting branch...
<gmb> gary_poster: This needs to land today, so I'll use the workaround for now.
<gary_poster> https://code.edge.launchpad.net/~gary/launchpad/bug491705
<gary_poster> gmb: ack
<gmb> gary_poster: Thanks, though :)
<gary_poster> np :-)
<gmb> gary_poster: Er, that -S didn't seem to help any. I'll try merging your branch and see what happens.
<gary_poster> gmb: mm, did you run make clean after applying the -S?  If so, it would have been wiped away
<gary_poster> gmb: steps would be as follows:
<gary_poster> 1) run make
<gary_poster> 2) curse at error
<gary_poster> 3) edit bin/buildout
<gary_poster> 4) run make again
<gmb> gary_poster: I don't think I did; let me just try again.
<gary_poster> if that doesn't help I'll be surprised and a bit concerned TBH
<gmb> gary_poster: Hmm. Same problem. Let me try  a clean branch of db-devel (this branch was fine before I merged db-devel to resolve some conflicts, btw)
<gary_poster> gmb: argh.  pastebin your edited bin/buildout for me to make sure we are talking about the same thing?
<gmb> gary_poster: http://pastebin.ubuntu.com/384448/
<gary_poster> gmb: ok alternate approach, have you infact tried make clean && make?
<gmb> gary_poster: Yes. No dice.
<gmb> gary_poster: In fact it complains about bin/py not existing when I do that
<gmb> Running link-external-sourcecode fixes that though.
<gary_poster> gmb: that doesn't make a lot of sense in my world view, so I don't understand something that you are doing. :-/
<gary_poster> um
<gmb> gary_poster: So, everything was fine until I merged db-devel. Then it broke. I updated my download-cache, ran make clean && make.
<gmb> And everything else has revolved around trying to get the latter to work.
<gary_poster> gmb, ok, I'll try to dupe.
<gmb> gary_poster: The branch is at lp:~gmb/launchpad/processapportblobjob-api-bug-513191, ftr.
<gary_poster> ack, on ity
<gary_poster> or it
<stub> gary_poster: I did notice that buildout must not be reporting an error code on this failure, as the shhh.py ate the output and the Makefile continued (hence the 'no bin/py' error)
<gary_poster> stub, huh
<gmb> gary_poster: This is also happening in a fresh branch of devel.
<gmb> (the no bin/py thing, that is)
<gmb> And also the zope.testing thing, too.
<gary_poster> gmb, a full make WFM without having to edit bin/buildout (history, for reference: http://pastebin.ubuntu.com/384454/ ).  I did not have zope.testing installed from apt, which is what your showed, so I just added it and will try
<gary_poster> retry
<gary_poster> gmb, sigh, still, worked for me.  Can you think of anything else I should maybe do to try to supe our environments?  Similarly, can you try merging in the branch I mentioned and see if it makes the problem go away for you?
<gary_poster> s/supe/dupe/
<gmb> gary_poster: I'll try merging your branch; I can't think of anything else you need to do. I might end up just switching machines for the sake of getting this landed...
<gary_poster> :-(
<gary_poster> (If my branch doesn't help, then I'll have to try to dig into this later once the release pressure has passed)
<gary_poster> (because it is supposed to make this kind of thing go away)
<gary_poster> note that your download-cache will need to be up-to-date
<deryck> abentley, qa-ready is very nice.  many thanks!
<abentley> deryck, you're welcome.
<gmb> gary_poster: Same error but with extra messages after your branch is merged: http://pastebin.ubuntu.com/384458/
<gmb> gary_poster: I'll switch machines; I don't have time to debug this properly.
<gmb> Thanks anyway.
<gary_poster> gmb: ack, and sorry.  I'll need your time next week.
<gmb> gary_poster: Sure.
 * gmb -> switching machines; brb
<gmb> gary_poster: Looks like it's all working fine on my laptop; no idea what's up with my desktop machine's environment. Ping me next week when you want to try and debug this.
<jtv> bigjools: did you dazzle the crowds?
<gary_poster> gmb: glad you are unblocked at least.  Another guess is http://pastebin.ubuntu.com/384463/ combined with my branch, FWIW.  I don't have enough information so far to do much else than guess.  I'll definitely ping you next week, thanks.
<bigjools> jtv: I did - the head wants me to install Edubuntu there now
<jtv> sinzui, maybe you can help me with this?  I'm trying to get a GPG key out of the current system user's real GPG, and it works if I run it in a plain python, but I see no keys at all in "make harness"
<jtv> bigjools: we'd hate to lose you to tech sales
<bigjools> jtv: I feel I have missed my calling
<bigjools> my true calling
<jtv> bigjools: I get missed calls all the time... I'm also using True prepay.
<sinzui> jtv: I have no experience with this...
<jtv> drat
<jtv> darn
<jtv> smeg
<bigjools> jtv: I watch True Blood
<jtv> bigjools: speaking of bleeding hearts, you wouldn't know about this either would you?
<henninge> bigjools: I followed wgrant's helpful hints but when I try to initiate the buld I get an error back (on xmlrpc).
<jtv> the gpgme problem
<sinzui> jtv: I think that tests that work with package uploading may provide a clue how to do this
<jtv> sinzui: I'd prefer not to rely on zeca though
<henninge> bigjools: what could be wrong here? http://paste.ubuntu.com/384472/
 * bigjools looks
<bigjools> henninge: there's a chroot checksum mismatch somehow
<bigjools> so what the buildmaster told it it was didn't match what it calculated when it unpacked it
<henninge> bigjools: when I Iook at the code, though, it does not seem to even try to get the file from the librarian.
<bigjools> henninge: it could be cached
<bigjools> and the cache doesn't match
<bigjools> it's supposed to re-fetch in that case though :/
<henninge> bigjools: the cache is empty
<bigjools> then I am stumped :(
<henninge> bigjools: also, I am a bit confused by this: http://paste.ubuntu.com/384483/
<henninge> bigjools: trying to get the file by its sha1 value gives me an empty result.
<bigjools> hmm
<henninge> but accessing it by its id is possible - the sha1 is correct.
<bigjools> henninge: findBySHA1("blah")[0] ... ?
<henninge> oh, .... right
<bigjools> it's a proxied result set
<henninge> yup, that was too obvious for me to see ...
<henninge> ok, so it's not an error in the librarian.
<henninge> bigjools: does the librarian need to be running in any way for the slave to be able to access it?
<bigjools> henninge: yeesssss :)
<bigjools> unless it's some magic librarian
<henninge> bigjools: I have tried with LP running in another terminal, though.
<jtv> henninge: but what you need from the Librarian AFAIK is the chroot, and didn't you get stuck at that point?
<henninge> bigjools: what else could keep the slave from being able to access the librarian? I am running it from a chroot.
<bigjools> henninge: firewall?
<henninge> ;)
<bigjools> can you telnet the address/port it's trying to access?
<bigjools> as your slave user
<henninge> how do I know what it's trying to access
<henninge> ?
<bigjools> the URL is passed to it from the buildmaster
<henninge> lemme try
<bigjools> so it might not be passing the right URL
<henninge> bigjools: do you mean the librarian url?
<bigjools> henninge: the path should be ok, I suspect the domain part isn't
<henninge> bigjools: ok, but I am simulating the master by an rpc call that wgrant gave me. So maybe the call is missing something.
<bigjools> ah
<bigjools> tell me what it was?
<henninge> http://paste.ubuntu.com/384472/
<henninge> bigjools: ^
<bigjools> oh that
<bigjools> hmmm
<bigjools> I can't remember and honestly I think I'd rather run the b-m locally
<henninge> bigjools: np, how do I do that? ;-)
<bigjools> henninge: /bin/twistd --python daemons/buildd-manager.tac
<bigjools> henninge: assuming you have the builder defined in LP.dev
<bigjools> and that your job is ready to dispatch, etc
<henninge> not sure if that is assuming too much ....
<henninge> bigjools: I'll try something else first.
<bigjools> henninge: poking xmlrpc is fine but given that only wgrant has done that then I would be scratching around the code a bit myself to remember what's going on
<henninge> that's what I am doing ... ;)
<henninge> bigjools: http://paste.ubuntu.com/384494/ ;-)
<bigjools> henninge: \o/
<bigjools> henninge: awesome, let me know how it goes.  When you get it all working locally we can try it out on dogfood.
<henninge> bigjools: "can't find ntp.buildd" What should that be?
<bigjools> I need more context
<henninge> bigjools: sorry
<henninge> bigjools: build log on the slave: http://paste.ubuntu.com/384500/
<bigjools> henninge: heh
<bigjools> ok
<bigjools> easy fix is to define that in /etc/hosts
<bigjools> along with the other million .dev ones
<bigjools> and point it at ntp.ubuntu.com
<henninge> bigjools: oh, that is how the buildd's are synced ...
<bigjools> yeah, the DC has that domain defined
<sinzui> I think the QA wiki script is behind. I just QAed my changes on staging, but the wiki does not know that they have landed and been deployed to staging
<henninge> jtv: are you sure the "doUpdate" step is required?
<jtv> henninge: no.  Only if something dramatically breaks intltool I expect.
<henninge> jtv: it calls "update-debian-chroot" but that is not included in the chroot.
<henninge> so it breaks.
<henninge> jtv: also any cleanup fails because it tries to remove proc instead of unmounting it (and dev/pts and sys).
<jtv> henninge: I don't think it's supposed to call that from within the chroot
<jtv> but it's something I could easily have gotten wrong.
<magcius> eeeek!
<magcius> Why can't I visit edge.launchpad.net without this new Launchpad Login Service?
<sinzui> magcius: because launchpad is going to use that for login from now on.
<magcius> sinzui, but why do I need an account to view Edge?
<magcius> (Is it just testing Edge?)
<magcius> also, I see an +openid in the URL, but there's no place I can put in my existing OpenID
<sinzui> magcius: except that the login service will really be Ubuntu.
<magcius> sinzui, ? what does that mean?
<magcius> (I don't use Ubuntu at all)
<sinzui> magcius: I believe next week all launchpad will be using openid for login
<magcius> sinzui, um, I still don't follow what you're saying
<magcius> Next week when 10.2 releases I'll need to login to do anything?
<magcius> also, I don't know who to talk to about this, but your sliding door needs more length
<magcius> i.e. right here: http://imgur.com/67fmj.png
<sinzui> magcius: Yes. but to be clear, launchpad-login code became the launchpad SSO, but users did not to use it launchpad authentication, so we removed the feature last year. Ubuntu is the real SSO server, we have been running a copy and allowing the old login to continue for about year
<deryck> intellectronica, I have some nervous energy about this branch.  This updates in real time with changes, right?
<intellectronica> deryck: yes
<magcius> sinzui, Launchpad authentication? What was that?
<intellectronica> not much different from a trigger, really
<sinzui> magcius: login
<magcius> sinzui, so there were two ways to log in?
<magcius> sinzui, what were they?
<intellectronica> deryck: are you worried about timing?
<deryck> intellectronica, except this happens in a view, right?  So if it takes a while to complete it timesout a view.  I thought triggers were independent of the view.
<deryck> yeah, timing is always my fear :-)
<intellectronica> deryck: why in a view?
<intellectronica> oh right, when you create a task or change the target it's in a view, you're right
<sinzui> Launchpad has not manages user accounts for a year. We allowed users to continue to use the launchpad copy of SSO, but the Ubuntu version is the master. when you login to a page in launchpad, you are really logging into ubuntu's sso
<deryck> intellectronica, that's what I meant.  I realize it's model code, but called from a view.
<deryck> intellectronica, you could create a job then, but you still need something to process it offline.
<deryck> intellectronica, do you feel really confident this won't timeout?
<magcius> sinzui, what is the Ubuntu SSO?
<intellectronica> deryck: we can remove the code that updates when you touch a bugtask. that will have pretty much the same effect, and wait for the max_heat to get updated when the bug's heat is next updated
<sinzui> magcius: https://login.ubuntu.com/
<intellectronica> deryck: it's hard for me to guesstimate whether it will timeout. i can't really know without trying this on staging
<magcius> sinzui, so if I have a Launchpad account, I can log in there?
<deryck> intellectronica, can we just move all of it to update when heat is recalculated?
<deryck> or is that what you mean?
<magcius> sinzui, what exactly does the Ubuntu SSO allow me to do?
<intellectronica> deryck: it will update when heat is calculated anyway
<sinzui> magcius: yes, because you really do have an ubuntu account, launchpad only keeps a profile of your hacking activity. For example, Launchpad does not really know your email address.
<intellectronica> deryck: it updates all targets whenever setHeat is called on a bug
<deryck> intellectronica, which happens offline, right?
<intellectronica> yes
<magcius> sinzui, okay, what can I use the Ubuntu SSO for?
<deryck> intellectronica, so the only live update is when a task is re-targeted?
<magcius> sinzui, like, I see my OpenID site in there for "Sites you last authenticated to"
<intellectronica> deryck: yes, or when a new task is created
<sinzui> magcius: You can use the openid url on your launchpad profile page to login into other openid enabled sites without registration
<magcius> sinzui, I already have an OpenID account, I don't need another one
<sinzui> magcius: you have had one from launchpad for about 2 years, it is shown on you profile page
<deryck> intellectronica, yeah, so I think either remove those bits.  Because those will trigger setHeat anyway, right?
<magcius> sinzui, I know, I have had another one from ClaimID.com for about 3 or 4
<deryck> s/either//
<intellectronica> deryck: i don't see how they will trigger setHeat
<intellectronica> deryck: how about, we remove triggering this when touching bugtasks and open a bug for looking at it later?
<intellectronica> deryck: my worry with all this stuff that happens offline is that we'll be accumulating inaccurate data
<intellectronica> it's very hard to follow what happens when
<deryck> intellectronica, yeah, that sounds good.  Also, if we're not adjusting heat with bugtask changes, then maybe we shouldn't do max_bug_heat either then.
<intellectronica> so without triggering the calculation directly, we'll probably have reasonable results, but i just don't want to leave it at that
<intellectronica> deryck: i'm sorry, i don't really understand your last sentence
<deryck> intellectronica, I'm asking why do we even need to recalculate max_bug_heat on bugtask changes?  If we don't change heat on the bug on bugtask changes, then why change max_bug_heat?
<intellectronica> deryck: because the bug might already have heat
<deryck> intellectronica, ah, but the project of the task might not yet have a max_bug_heat?
<intellectronica> deryck: exactly
<deryck> hmmmmm
<intellectronica> deryck: how screwed are we if this gets so staging and we find it times out? it will be really nice to know whether this is actually a problem
<deryck> intellectronica, thinking about that.  is the only time we would call this "live" would be on changing a bugtask?  which is via AJAX?
<intellectronica> deryck: also when adding a new task. both are not necessarily ajaxed (changing a target can be ajaxed but there's also a non-ajax interface)
<deryck> intellectronica, right, but it's through a new page, which is not as a bad as a bug list or page.  I think this probably isn't a problem in practice.
<intellectronica> deryck: right, i wouldn't bet on a timeout. but as always, without testing this on ubuntu on staging it's very hard to tell
<deryck> intellectronica, I'm willing to risk it.  allenap has to cowboy a patch to test.  perhaps we could create a combined diff and test both.
<intellectronica> great!
<intellectronica> worst case, we know exactly what are the bits we need to back-out, so it shouldn't be too much work to recover from it
<deryck> right.  but I do want to avoid having to do that.
<intellectronica> but if we get to cowboy a patch then even that won't be necessary
<deryck> exactly.
<deryck> intellectronica, so can you and allenap coordinate about that combined diff when he's available.
<intellectronica> sure thing
<intellectronica> deryck: apart from that, any comments on the code?
<deryck> intellectronica, looks good.  Just doing a careful scan of the sql, and then I'll be ready.
<intellectronica> cool
<deryck> intellectronica, only comment really is that line 145 has a commented out bit of code, which I assume you don't need anymore.
<intellectronica> deryck: whoops, no, of course i don't need that bit.
<intellectronica> removed
<deryck> intellectronica, cool.  running a couple queries on staging just for paranoia.
<deryck> hmmm, it's slow.
<deryck> intellectronica, http://pastebin.ubuntu.com/384562/
<intellectronica> oh yes, that's too slow
<intellectronica> deryck: so, remove the call when touching a bugtask for now? i do think we'll have to revisit this if we want reliable results, but it probably won't be too terrible for now
<intellectronica> b.t.w i think it will also be very slow when calculating heat from a script
<deryck> intellectronica, just a second more and let me play and then we can decide.  not meaning to delay, I know it's late for you.
<intellectronica> deryck: that's oright, i'm working on other heat bugs in the meantime. no rush
<intellectronica> and thanks for running these queries. nothing beats real data
<deryck> yeah, I'm glad I can do them on staging.
<deryck> intellectronica, I may have something.  http://pastebin.ubuntu.com/384585/
<deryck> removing the union and doing a limit.  and I get the right result.  it makes sense to me.
<intellectronica> deryck: wow. i'm impressed.
<intellectronica> not by postgres :)
<deryck> my head hurts now.
<intellectronica> i'm amazed the difference is so bug
<intellectronica> big
<deryck> yeah, it's huge.
<deryck> because it doesn't have to scan with the limit and order by.
<intellectronica> so this is just one of the queries in the union, but even running both and maxing the result in python would be faster than the 4s we got with the union
<deryck> intellectronica, no, this can replace the union.
<deryck> intellectronica, you just want the highest bug heat for any bug that has a bugtask.distribution or bugtask.distroseries, right?
<intellectronica> deryck: right, i missed the OR
<intellectronica> it doesn't limit by distribution. don't know if that will make much of a difference
<deryck> intellectronica, right.  so we can add a distribution = %s.  Let me check times on that.
<intellectronica> deryck: can you run the same with AND  Bugtask.distribution = 1 ?@
<deryck> intellectronica, quicker even!
<intellectronica> awesome
<deryck> intellectronica, http://pastebin.ubuntu.com/384589/
<intellectronica> wonderful
<intellectronica> and the sql is much more readable too
<deryck> intellectronica, cool.  So r=me with something like this to replace each UNION sql statement.  If that seems acceptable to you, too.
<intellectronica> i'm still surprised. my guess is that it's not the max() that was making it small, but the nested queries
<intellectronica> deryck: for sure. thanks a bunch!
<deryck> intellectronica, cool. np!  Very sick of timeouts.  One day we'll do this everywhere and bring our counts do on the OOPS report.
<sinzui> beuno: adeuring: landed his fix for bug advanced search: https://bugs.edge.launchpad.net/launchpad-registry/+bugs?advanced=1
<sinzui> ^ You can now see what the default criteria is for searching, and the legends a have enough space so that you can see the parts of the form.
 * sinzui thinks the checkboxes at the end are silly though
<beuno> sinzui, sure
<sinzui> beuno: I wanted them to be radio-button
<beuno> so
<beuno> almost everything on that page needs vertical spacing
<beuno> the sorting drop down is confusing when layed out that way
<beuno> the input box as well
<beuno> this is advanced search, so you probably want to explain that the input box is optional, and to search for text in X or Y parts of the bug
<beuno> the "Doesnât matter" option for asignee could be named better
<sinzui> beuno: I agree. I think Abel has done a good job closing a number of UI issues in one branch. He may hate me for it, but I can finally read that form
<beuno> sinzui, YES. It's a million times better now
<beuno> the any/all radio buttons on tags should be right next to the tag field
<beuno> just "Help" may also suffice there
<beuno> and having:
<sinzui> In their previous position, they looked like they were subordinate to the item above. The form layour you introduced 9 months places subordinates below, not to the right
<beuno>   Show only bugs with linked branches   	
<beuno> Show only bugs without linked branches 	
<beuno> both ticked by default is a bit confusing
<beuno> ah
<beuno> well, if it can't be easily tweaked to be to the right of the input, at least change the labels to "All tags" or "Any tags"
<beuno> all labels on that page could do with .5-1em of padding-top
<sinzui> beuno: yes, I was too, as was abel, We and I wanted a radio button show exclusivity or state your do not care. I had to run a search to prove I would get results with that insane condition
<sinzui> beuno: yes, padding can fix the radiobutton on right issue.
<jml> kfogel, ol buddy ol pal
<jml> kfogel, I want an elisp function that behaves like upcase-word but switches from under_score to camelCase, and another one that goes in the other direction.
<beuno> sinzui, did all that help?  or where you asking something else?
<sinzui> beuno: yes it helped. I wanted your opinion on the improvement.
<beuno> sinzui, long overdue, and very nice!
<jml> kfogel, It would be great if you could write this, but I say this in a way that has absolutely nothing to do with Canonical, just that it would help my occasional free software LP development
<EdwinGrubbs> bac, sinzui: what do you think about this simplified upstream connections portlet? https://chinstrap.canonical.com/~egrubbs/upstream/upstream_simplified.png
<sinzui> EdwinGrubbs: I think that can get long
<sinzui> EdwinGrubbs: This may be easier if you keep in mind your proposing a change to two pages, not one...
<bac> EdwinGrubbs: 'mozilla', 'firefox', and 'blog series' are three different links?
<sinzui> EdwinGrubbs: The difference between the packaging listing and the source package page is that the listing does not get the change operations
<sinzui> https://edge.launchpad.net/ubuntu/lucid/+packaging
<sinzui> https://edge.launchpad.net/ubuntu/lucid/+source/openoffice.org
<sinzui> EdwinGrubbs: if we can keep the branch and translations in a separate column, and keep the 3 objects linked, the layout may work for both cases. Your https://chinstrap.canonical.com/~egrubbs/upstream/horizontal_hierarchy.png is more practical for users who want to visit the project or project group
<intellectronica> what's a good way to test something time-related?
<intellectronica> is there a way to temper with the current time for a test?
<EdwinGrubbs> sinzui, bac: I had actually removed the project group entirely after reading Curtis' email. Then I removed the project link, since the user can always make to clicks, and it makes the display cleaner and clearer.
<EdwinGrubbs> s/to clicks/two clicks/
<sinzui> EdwinGrubbs: I cannot fix the project bug tracker unless I have a link to the project. We need links to projects an series. The project group link is dubious, but needed until the bugs team sorts out their upstream contact contradiction
<sinzui> bac: EdwinGrubbs: look at this: https://bugs.edge.launchpad.net/openoffice
<sinzui> ^ I need to enable bug tracking to set the value, then disable it again?
<EdwinGrubbs> bac, sinzui: I'll use the horizontal layout but with the Branch in a separate column then.
<sinzui> EdwinGrubbs: I think that is what is there now (and translations is there too)
<EdwinGrubbs> yes
<wgrant> Ooh. Somebody fixed the advanced search form.
<beuno> wgrant, hug sinzui and adeuring for that
<wgrant> Hey, ISD finally got their act together.
<lifeless> ?
<wgrant> The security fix has finally been rolled out.
<bdmurray> make build is failing for me in devel at finding a distribution for lazr-js==0.9.2DEV-r169
<wgrant> bdmurray: Run rocketfuel-get lately?
<wgrant> How is login.ubuntu.com going to provide the OpenID team data that Launchpad provides?
<lifeless> bdmurray: update-sourcecode probably
<wgrant> This one needs an updated download-cache.
<wgrant> rocketfuel-get updates both.
<lifeless> ah
<lifeless> wgrant: is there a UEC AMI for doing LP dev?
<wgrant> lifeless: No.
<bdmurray> wgrant: okay I ran rocketfuel-get and it got the new version of lazr-js but its still failing
<wgrant> Hmmm.
<bdmurray> oh, that's not quite right
<bdmurray> I have 0.9.2DEV not -r169
 * bdmurray takes it back
<wgrant> I can see it was added to the branch 7 hours ago.
<bdmurray> okay I'm all squared away now thanks
<lifeless> wgrant: do you know, is the EC2 test AMI suitable for adapting to be a development AMI?
<wgrant> lifeless: I considered that. Quite possibly -- it can already do 'ec2 demo'.
<wgrant> (although it needs an image update for the OpenID stuff)
<lifeless> do you know the size of the AMI?
<wgrant> No.
<mwhudson> jelmer: any luck with incremental imports?
<jelmer> mwhudson: I wrote a small plugin to add --limit to bzr pull - lp:~jelmer/+junk/bzr-pull-limit
<jelmer> mwhudson, but I haven't really been able to reproduce the problem yet
<wgrant> sinzui: Around?
<sinzui> I am
<wgrant> sinzui: I'm interested in making the mirror prober less inflexible by converting it to a job-based system.
<sinzui> That will make several people happy
<sinzui> wgrant: jpds may also be interested in the job system for mirrors. There is a small chance the two of you could step on each other.
<jpds> sinzui: We do that all the time. ;)
<wgrant> sinzui: Oh, don't worry, I'm talking to him.
<mwhudson> jelmer: :/
<sinzui> fab.
<mwhudson> jelmer: oh well, something for monday then
<jelmer> mwhudson: If I have a spare moment on the train I might have a look during the weekend
<mwhudson> ooh, staging is updated
<mwhudson> let's try it there
<mwhudson> jelmer: ok, don't sweat it
<lifeless> mwhudson: so a full repo fetch wasn't it ?
<wgrant> sinzui: It seems to me that there are two options for doing this: adding a next_probe_time to DistirbutionMirror, or adding a full-blown DistributionMirrorProbeJob.
<sinzui> wgrant: In the former to do image a default time and allowing users to choose probe now so that the existing proc will pull all those that are due for a probe?
<wgrant> sinzui: Something like that, yes.
<wgrant> The former probably makes more sense.
<sinzui> I think so to, but I am thinking of reason that might happen in the next year that make us want a job instead
<wgrant> Oh?
<wgrant> I couldn't think of anything compelling.
<sinzui> wgrant: nor me
<lifeless> ugh
<lifeless> http://lanchpad.net/testresources
#launchpad-dev 2010-02-27
<mwhudson> lifeless: different problem now
<lifeless> mwhudson: tell me if you want to talk about it
<mwhudson> lifeless: keep an eye on https://code.staging.launchpad.net/~vcs-imports/linux/trunk if you're curious how it goes
<lifeless> mondayish
<mwhudson> lifeless: sure
<mwhudson> right
<mwhudson> certainly not now :-)
<lifeless> its mah birthday :P
<mwhudson> oh, happy bday!
<lifeless> ciao
 * wgrant is slightly scared by Launchpad classes that inherit private Twisted classes.
<wgrant> sinzui: Hmm. The default advanced bug search page has both "Show only bugs with linked branches" and "Show only bugs without linked branches", yet it still returns all of the bugs.
<sinzui> Yep
<wgrant> Has both of those selected, that is.
<sinzui> It is insane
<wgrant> Less insane than before, though.
<sinzui> abel and I wanted radio buttons to who mutually exclusive states
<wgrant> Ah.
<sinzui> I had to do a search to prove I would get results
<sinzui> I intend to report a bug about it
<Pilky> sinzui: you around?
<sinzui> I am
<Pilky> just going to do some mockups for the team page
<Pilky> based on the current design
<Pilky> got two questions before I start though
<Pilky> 1. you said on the list that the emphasis on certain information is wrong, what bits would you consider the most important to emphasise?
<Pilky> 2. Does the list of pending members need to be shown to people who can't accept a new member?
<sinzui> 1. The projects and packages that a team owns or creates.
<sinzui>    What are the latest bugs, blueprints, questions, and branches that the team is assigned...We should find a away to indicate if a team is involved in translations.
<sinzui> 2. All team members should be able to see who is pending, or if the user is himself in the pending list. As for everyone else, we show it to make the possible change visible, but I do not think it is necessary to be visible on the index page
<Pilky> ok
<Pilky> well, I'll have a go at mocking something up and then post it to the list. I'll try a few variations.
<sinzui> Pilky: I really do not like the approach we have taken to show the latest <bug|blueprint|...> on pages. Some items might be months old. I have suggested a changelog or facebook wall approach to show recent activity. This may work for projet, but not necessarilly teams
<Pilky> yeah it does cause quite a bit of clutter, I'll try looking at ways to simplify it
<sinzui> What many users want to know (about themselves or others) is what has happened recently, what do I need to do next.
<Pilky> the wall approach is slightly iffy legally now, seeing as facebook just got a patent on combined news feeds for activity on a site :/
<sinzui> changelog is an metaphor that cannot be challenged.
<Pilky> yeah
<Pilky> well I'm pretty sure there's some prior art as with half of software patents, but I'll have a look at some sort of change log for the team
<Pilky> well I'll hopefully have a few mockups in a few hours
<sinzui> The design has been limited by implementation details, such each box has only one type of object. I want the code to 1. allow me to have a mixed list of objects to show. 2. allow me to set filters on the page so that I see only the kinds of objects|events that interest me
<sinzui> My hope is that if we can have mixed objects and filters, we can solve a lot of wasted space on the page--either empty spaces, or boxes of information that the user does not care about.
<Pilky> yeah
<Pilky> well I'll try some ideas, if we can get the team page cleaned up a bit then hopefully a lot of that can be transferred to other pages.
<Pilky> sinzui: you around? I've got some mockups
<lifeless> moin
<jelmer> mwhudson, hi
<mwhudson> jelmer: hello
<jelmer> mwhudson, I've tried to reproduce the bug you were seeing with gedit
<jelmer> but I can't reproduce it by pulling 1k revisions at a time from a local git repo into a local bzr repo
<mwhudson> jelmer: and it just worked?
<mwhudson> jelmer: hmm
<mwhudson> jelmer: linux failed on staging in the same way as locally: https://code.staging.launchpad.net/~vcs-imports/linux/trunk
<jelmer> mwhudson, yes
<jelmer> mwhudson, this was from not inside of launchpad though but from the commandline
<mwhudson> well
<mwhudson> that shouldn't make a difference
<jelmer> mwhudson, So you could reproduce the issue with both gedit and linux?
<mwhudson> jelmer: i didn't try gedit on staging actually
<jelmer> mwhudson: ah, ok
<jelmer> mwhudson: staging is running a recent bzr-git, isn't it?
<jelmer> so maybe that happened to have the bug that gedit is breaking on fixed
<jelmer> mwhudson: I'm running bzr pull --limit=1000 for the kernel now
<jelmer> do you know after how many revisions it broke?
<wgrant> IIRC it was on the third run.
<jelmer> wgrant, thanks
#launchpad-dev 2010-02-28
<jelmer> mwhudson, wgrant: unfortunately I'm already past 5000 revisions :-/
<wgrant> jelmer: I just checked IRC logs, and it was between r3000 and r4000.
<jelmer> wgrant: Thanks
<jelmer> So I wonder what's different between Michaels' setup and mine
<wgrant> jelmer: Did you import 5000 revs, or up to r5000?
<jelmer> ah, good point
<jelmer> I imported 5000 revisions
<jelmer> r1008
<jelmer> so there's still "hope"
<wgrant> I'm not sure which Michael meant. I guess there's one easy way that will probably find out.
<jelmer> mwhudson: Hmm, it definitely seems to slow down over time
<jelmer> mwhudson: but no failures yet.
<jelmer> mwhudson, reproduced
<wgrant> Yay.
<jelmer> strangely enough I only seem to be able to reproduce it on fetches from remote hosts
<jelmer> and it's related to encodings in commit messages/authors
<wgrant> Aha.
<jelmer> The issue only crops up when we try to recreate the original git object from the data we have in Bazaar
<jelmer> We need to create the original git object again to use it as the delta base, but of course that's only necessary during a pull (during a clone or 'full' pull we already have all base texts available in git packs)
<wgrant> Ah, yes. So is the bzr revision bad, or just not being recreated properly?
<jelmer> it's just not being created properly - it strips off characters that are invalid for the expected encoding at the moment
<jelmer> so we lose a few characters
<jelmer> and then of course the sha1 of the newly created commit is different
<wgrant> Ah.
<jelmer> anybody around to review the patch to unbreak the merge from stable into db-devel?
<jelmer> mwhudson: I got to 18k revisions without problems
<jelmer> mwhudson, hi
<lifeless> jelmer: another hour perhaps
<jelmer> lifeless, Isn't he in NZ while you're in AU ?
<lifeless> yes
<jelmer> lifeless, or are you just up at a crazy hour?
<lifeless> I woke a little...early today
<jelmer> ah :-)
<mwhudson> jelmer: hello
<jelmer> mwhudson: I just sent you an email. We should make sure to get a new version of bzr-git onto lp.
<mwhudson> jelmer: subject?
<mwhudson> jelmer: i guess it's r727 we want, right?
<jelmer> mwhudson: there's bzr-git in the subject, don't recall what it is exactly
<jelmer> mwhudson, yeah
<mwhudson> jelmer: it doesn't look like i received it :/
 * jelmer flushes his mail queue
<jelmer> mwhudson, should be sent now
<mwhudson> jelmer: it's interesting that bzr-git spends nearly 30% of it's time shoving things into sqlite
<mwhudson> jelmer: i got your mail now :-)
<mwhudson> jelmer: if a branch is suffering from this problem the branch is broken and needs to be re-imported from scratch, right?
<jelmer> mwhudson: yes
<lifeless> jelmer: btw, new testrepository is sexy :) and packaged in debian [in NEW]
<jelmer> lifeless, ah, awesome. I need to give it a try
<jelmer> lifeless, I saw your addition of .testr.conf to bzr
<lifeless> jelmer: you  could write a .testr.conf for lp :)
<jelmer> I might
<lifeless> \o/
<mwhudson> jelmer: out of curiosity, what do you think a sensible revision pull limit is for lp?
<mwhudson> it's at 1000 now, which i think might be a bit low
<mwhudson> i actually wonder if a time-based limit might almost make more sense
<mwhudson> import revisions for an hour, then stop
<mwhudson> or something
<jelmer> mwhudson: as long as you reschedule imports I don't think the limit is really important
<mwhudson> jelmer: ok
<jelmer> anywhere between 1k and 10k seems reasonable
<mwhudson> testing the kernel locally sure used up quite a lot of my bandwidth :-)
<mwhudson> we should be getting some new code import slaves soon-ish btw
<mwhudson> they will be much much better speced that the machines we currently have
<mwhudson> *than
<jelmer> ouch, you have a limit on your bandwidth?
<mwhudson> only a practical one
<mwhudson> but it takes a lot longer here to get the kernel.org repo than it does the data centre machines, that's all i meant
<jelmer> ah
<jelmer> mwhudson: oh, we'll need a newer dulwich too
<mwhudson> jelmer: oh
<mwhudson> jelmer: you haven't rebased dulwich again have you?
<jelmer> mwhudson, no
<mwhudson> hm
<mwhudson> must have had old local branches then
<mwhudson> oh great, someone broke buildbot?
<lifeless> say its not so
<mwhudson> looks like r9055
<mwhudson> or maybe it's a merge integration thing
<mwhudson> oh i see there's already list discussion about it
<mwhudson> (it is an integration thing)
<mwhudson> jelmer: fwiw the git import succeeded locally without the new dulwich
<mwhudson> s/git/gedit/
<thumper> morning
<thumper> mwhudson: when do you want to chat?
<mwhudson> thumper: in a few minutes?
<thumper> sure, just call
<mwhudson> thumper: https://staging.launchpad.net/successful-updates.txt
<mwhudson> jelmer: wow, that bzr-svn symlink thing is breaking so many imports
<mwhudson> jelmer: is the fix landed already?
<wgrant> Argh.
<lifeless> ?
 * wgrant rechristens Soyuz "who could possibly think that was a good idea?"
<lifeless> jelmer: as incentive for doing a .testr.conf
<lifeless> https://code.edge.launchpad.net/~lifeless/tribunal/testrepository/+merge/20321
<thumper> mwhudson: https://bugs.edge.launchpad.net/launchpad-code/+bug/297398
<mup> Bug #297398: support password/passphrase authentication for bazaar <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/297398>
<wgrant> (bug #529710 was the cause of my horror)
 * thumper taps his fingers waiting for the staging email to arrive in the imap box
<thumper> mwhudson: how's the qa on the import changes going?
<mwhudson> thumper: well
<mwhudson> as in, it all seems to work
<mwhudson> i'm not sure there's much more i can do without losa help
 * thumper now waits for staging incoming email processor to kick in at noon
<thumper> damn it
 * thumper needs a losa too
<thumper> mwhudson: I'm walking up to collect Maia, afk for a bit
<mwhudson> thumper: ok
 * mwhudson lunches
#launchpad-dev 2011-02-21
<lifeless> changes to shipit itself? via PQM I believe
<wgrant> I mean reviewer-wise.
<wgrant> Since it's ISD.
<lifeless> dunno
<lifeless> is there even a project/
<wgrant> There is.
<wgrant> That's where the branch lives.
<jelmer_> wgrant: last time I changed it I just proposed a merge in Launchpad and then pinged somebody in ISD (Anthony Lenton I believe) about it.
<jelmer_> wgrant: He reviewed and took care of landing my changes.
<wgrant> :(
<wgrant> Thanks.
<jml> lifeless: I'd be interested in disagreeing with you about that some time.
<jml> lifeless: but not now.
<lifeless> jml: about what? [ambiguous that]
<jml> lifeless: moving devscripts out of tree
<jml> good night!
<wgrant> lifeless: Also, you could export id in the interface rather than using rSP.
<huwshimi> jml: Night
<lifeless> jml: gnight
<wgrant> Night jml.
<lifeless> wgrant: err, the ids are exported
<lifeless> wgrant: they get wrapped, because thats what security proxies do
<lifeless> then sqlvalues goes boom
<wgrant> lifeless: Huh? ints are not proxied.
<wgrant> If it's on the interface it should be fine :/
<lifeless> lists of ints are
<wgrant> Oh, right, all_distro_archive_ids.
<lifeless> oh man
<lifeless> launchpad_dev=# SELECT COUNT(*) FROM BugTask JOIN Bug ON BugTask.bug = Bug.id WHERE Bug.id = BugTask.bug AND ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25)) AND BugTask.milestone = 14 AND Bug.duplicateof is NULL;
<lifeless> 2
<lifeless> launchpad_dev=# SELECT COUNT(distinct bugtask.bug) FROM BugTask JOIN Bug ON BugTask.bug = Bug.id WHERE Bug.id = BugTask.bug AND ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25)) AND BugTask.milestone = 14 AND Bug.duplicateof is NULL;
<lifeless> 1
<lifeless> our production milestone bug counts are a little inflated
<lifeless> give you one guess why
<lifeless> I'm going to go beat my head against a wall for a few minutes
<wgrant> Hmm.
<wgrant> librariangc creates dozens of cursors.
<wgrant> Can anyone see a good reason to keep that?
<lifeless> I'd check with stub
<wgrant> I can see no benefit over passing a single one around, instead of the connection that it currently passes.
<wgrant> k
<lifeless> anyone remember where the storm expr for Distinct is
<wgrant> .config(distinct=True) isn't good enough?
<wgrant> Ah, you want that second query?
<lifeless> well
<lifeless> I'm not going to fix the current for loop
<lifeless> but countBugs also has the same defect
<lifeless> it just isn't showing it in the current code path because its constrained to only consider series tasks
<lifeless> the reason conjoined masters mess things up is because they have /two/ tasks
<lifeless> rather than rendering as two tasks but actually having one.
<wgrant> Right.
<lifeless> sigh, we definitely need two queries.
<lifeless> mmm
<lifeless> unless, I group by - bingo
<lifeless> mmm
<lifeless> no
<lifeless> our compiler just isn't good enough, and I think we'll get enough benefit for now with two queries
<lifeless> right, thats another second or so off of bug searching in ubuntu
<lifeless> + more accurate counts
<wgrant> Excellent.
<lifeless> [should be marginally cheaper too]
<lifeless> we have laze evaluation of milestones too
<lifeless> going to look at that in a minute
<lifeless> I can has review number two? https://code.launchpad.net/~lifeless/launchpad/bug-717394/+merge/50541
<lifeless> poolie: hi, I will be in syd evening of the 4th,5th and through early avo 6th.
<StevenK> lifeless: \o/
<poolie> hi lifeless
<poolie> great! i'll be here too
<wgrant> lifeless: milestones are targets now?
<lifeless> wgrant: for bug search, yes
<wgrant> :(
<poolie> 98	-            group_on + (Count(BugTask.bugID),), [], None, params).result_set
<poolie> 99	+            group_on + (SQL("COUNT(Distinct BugTask.bug)"),), [], None, params).result_set
<poolie> it seems like a storm bug if that's any faster
<poolie> but it is?
<lifeless> poolie: how  would it be a storm bug?
<lifeless> wgrant: I can take that out if it doesn't make sense
<poolie> does it generate different sql? only becaues of the 'distinct' constraint?
<wgrant> lifeless: It doesn't seem to be used and doesn't comply with the definition of 'bug target' everywhere else.
<lifeless> wgrant: it just made sense at the time; I guess rather than setTarget a better name would be 'constrainBy'
<wgrant> Right.
<lifeless> wgrant: its used in browser/bugtarget.py
<lifeless> poolie: yes, different sql
<poolie> lifeless, i'm just curious what it does differently
<lifeless> SELECT COUNT(*) FROM BugTask -> SELECT COUNT(distinct bugtask.bug)
<lifeless> erm
<lifeless> SELECT COUNT(bugtask.bug) FROM BugTask -> SELECT COUNT(distinct bugtask.bug)
<lifeless> ^ thats the change
<poolie> and that's faster in psql?
<lifeless> slightly smaller intermediary table
<lifeless> the big win is using the countBugs function rather than doing a loop in python
<poolie> sure
<poolie> there's no way to say 'distinct' in storm?
<lifeless> doing a distinct forces a sort
<lifeless> but so does grouping
<lifeless> so in this case it just lets it return less data internally before the aggregation
<lifeless> poolie: you can define an Expr for it
<lifeless> but it likes to put () around things
<lifeless> and COUNT((Distinct BugTask.bug)) isn't valid SQL
<lifeless> s/it/storm/
<wgrant> Storm will do distinct, but not in a group by.
<wgrant> Oh.
<wgrant> I misread.
<wgrant> But still, no way to do that in Storm right now.
<poolie> i wonder if lp should ban branch renames until more of the bugs related to it are fixed
<poolie> eg https://answers.launchpad.net/launchpad/+question/146165
<wgrant> Is there more than one bug?
<poolie> renaming branches that are stacked upon causes troubel
<wgrant> That is the only real bug that I know around renaming.
<poolie> renaming branches causes confusing errors in a bunch of cases
<lifeless> wgrant: IIRC recipes decided to not link to branches
<lifeless> wgrant: because modelling recipes was too hard
<lifeless> or something
<wgrant> lifeless: False.
<wgrant> See SourcePackageRecipeDataInstruction.branch
<wgrant> I think we need a SourcePackageRecipeDataInstructionPartToken, or something like that.
<lifeless> wgrant: interesting
<lifeless> wgrant: so why did that guys branch start barfing? or is it a stacking issue
<wgrant> lifeless: Stacking.
<wgrant> So, I am all for banning renames of stacked-on branches.
<wgrant> Because it causes a lot of trouble.
<lifeless> I think thats appropriate
<lifeless> but not a desirable constraint per se
<wgrant> Sure.
<wgrant> But it *will* break things.
<wgrant> So we might as well ban it until it's fixed.
<spiv> +1
 * spiv blinks.  Isn't "product" a deprecated term?
<wgrant> Yes.
<wgrant> Is this on +newrecipe?
<spiv> wgrant: no, on clicking the ajaxy bug task affects to reassign something from bzr to bzr-loom
<wgrant> Ah.
<spiv> Thanks; I'll file/find a bug.
<wgrant> A new one was introduced recently (bug #714536)
<_mup_> Bug #714536: Branch:+new-recipe related branches listing has formatting issues <recipe> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/714536 >
<wgrant> I don't know a bug for yours.
<thumper> spiv: yes
<thumper> spiv: there is a bug already
<thumper> spiv: bug 714534
<_mup_> Bug #714534: Branch:+new-recipe refers to "product series" <recipe> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/714534 >
<spiv> Just filed https://bugs.launchpad.net/launchpad/+bug/722412
<_mup_> Bug #722412: Changing project of bug task prompts to change/search "product" <Launchpad itself:New> < https://launchpad.net/bugs/722412 >
<spiv> thumper: hmm, that bug to my eyes doesn't seem obviously the same
<thumper> spiv: ah... differrent one
<spiv> thumper: unless it is meant as a global s/product/project/ bug?
<thumper> spiv, no it wasn't
<thumper> spiv: just my misunderstanding of your bug
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/use-script-isolation-arg/+merge/50543
<spiv> thumper: Heh, I'm guessing you've been working on recipes a lot lately, because I didn't mention them at all :)
<thumper> spiv: perhaps it was wgrant mentioning it in passing interleaved with your messages :)
<spiv> thumper: he also guessed recipes were involved... I'm detecting a pattern!
<thumper> I fixed the bug page as part of some recipe work
<thumper> but I didn't notice the product term used
 * thumper is a little confused
<thumper> lifeless: are you familiar with the DocTestMatches matcher?
<thumper> lifeless: nm
<lifeless> yes
<lifeless> I wrotes it
<thumper> the whole point of using the DocTestMatches was to have it normalize the whitespace.  A little frustrating that I have to pass in the flag
<lifeless> when I wrote it I wasn't sure what sensible defaults would be
<lifeless> if we were to put the flag on by default
<lifeless> I think we'd need a way to turn it off too
<lifeless> but we could change the default, or you could use functools.partial to make a version that has the flag supplied for all your tests
<thumper> hmm...
<wgrant> lifeless: Thanks.
<lifeless> thumper: idioms that work in functional environments will work well with matches
 * thumper off for school run
<lifeless> thumper: its a very similar environment
<lifeless> ok, that was too easy.
<lifeless> if I put a method 'getPillar' on I*Series and I*[things that also have series variants]
<lifeless> what would you expect it to do?
<wgrant> Isn't there a .parent or similar to do that?
<lifeless> ah, but IProduct.getPillar() -> self
<lifeless> possibly I want IPillar(thing)
<lifeless> do we have that already
<lifeless> ?
<wgrant> We have IRootContext or similar, which does that.
<wgrant> But it's probably not a good idea to reuse it.
<lifeless> another review plox https://code.launchpad.net/~lifeless/launchpad/bug-717394-2/+merge/50545
<poolie> o/ spiv
 * spiv waves
<lifeless> wgrant: if you look at my new branch you can see it would be rather more pithy and clear
<lifeless> wgrant: I don't know how often we do this
 * lifeless headdesks
<lifeless> lib/lp/registry/model/distribution.py
<wgrant> lifeless: Sure, I would support an IPillar. Take stuff out IRootContext.
<lifeless> line 724 - __getitem__
<wgrant> Wow.
<wgrant> OTOH it does save queries.
<wgrant> Hm, except not.
<wgrant> But Distribution.series is cached, so yes, it does save queries.
<wgrant> Slightly.
<poolie> huwshimi, does https://bugs.launchpad.net/launchpad/+bug/722419 ring any bells?
<_mup_> Bug #722419: bug title edit buttons go under portlets, can't be clicked <bugs> <css> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722419 >
<lifeless> it triggers the query I'm seeing
<huwshimi> poolie: I can reproduce it if that's what you mean?
<lifeless> whether it saves queries or not is a little dicy at this point
<lifeless> I'm not sure thats a regression, its been like that since ajaxification
<huwshimi> poolie: Oh actually that's a dupe
<poolie> huwshimi, i guess i was politely asking "i wonder if you broke it" :-)
<wgrant> lifeless: I like Distribution._sort_key
<poolie> not really a big deal
<huwshimi> poolie: haha, ok. No I didn't. It's actually been like that for ages. It has to do with how big your browser window is
<poolie> yes
<lifeless> wgrant: facedesk
<huwshimi> poolie: Have you changed your res or browser size recently?
<huwshimi> poolie: Like in the last week :P
<lifeless> wgrant: you'll love 722421
<lifeless> bug 722421 that is
<_mup_> Bug #722421: lib/lp/bugs/model/bug.py _known_viewers causes per-row lookups on public bugs in bug searches. <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722421 >
<wgrant> It's also wrong.
<lifeless> argh
<lifeless> wgrant: no, its not
<lifeless> bac broke the optimisaiton
<wgrant> lifeless: Pillar owners cannot always see bugs...
<lifeless> either way
<lifeless> whether its correct to do so or not
<lifeless> the defect is real
<wgrant> Also some of the manipulations of that cached property seem to be a little screwed.
<wgrant>             # The bugtask is unassigned, so clear the _known_viewer cached
<wgrant>             # property for the bug.
<wgrant>             get_property_cache(self.bug)._known_viewers = set()
<lifeless> rev 12156.8.20
<lifeless> the clearing is conservative
<wgrant> That's not clearing it.
<wgrant> That's *emptying* it.
<lifeless> indeed
<lifeless> so
<lifeless> the branch to set assignee as a viewer as broken
<lifeless> bug 702429
<_mup_> Bug #702429: Pillar owners and private bug visibility <Launchpad itself:Triaged> < https://launchpad.net/bugs/702429 >
<lifeless> wgrant: however the contract for it is that it can be an empty set and is added to
<lifeless> wgrant: so del() would be equally glitchy
<wgrant> Yes, but it would at least crash instead of being wrong.
<lifeless> wgrant: its not wrong
<lifeless> wgrant: if a user is not in the set the rest of the 'can see' code triggers and it will repopulate
<wgrant> Ah, I see.
<lifeless> this is fallout from folk still learning about how eager loading and late evaluation interact
<lifeless> another 75 queries a page removed
<wgrant> Assuming each pillar is distinct.
<wgrant> Which will happen approximately never.
<lifeless> what was also buggy
<lifeless> was the missing TP lookup
<lifeless> wgrant: uhm, no, it requeries the *bugtasks* per bug
<lifeless> wgrant: so you don't need distint pillars, you just need multiple bugs
<wgrant> Oh.
<wgrant> Nice.
<wgrant> I didn't actually read the bug past the summary, FWIW.
<wgrant> I am looking to see if there is a sensible way to put tracebacks in the OOPS format.
<wgrant> I suspect not.
<lifeless> there isn't
<lifeless> I'm kicking off a multi-team requirements list for oops tools
<lifeless> give it a couple of days and I should have their needs and be able to write it up for public consumption
<wgrant> Great.
<wgrant> For now I guess I'll just write invalid OOPSes locally...
<huwshimi> poolie: that bug is a dupe of #692291 which was reported in December. I'
<_mup_> Bug #692291: Impossible to modify the title of the report <css> <javascript> <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/692291 >
<huwshimi> poolie: I'll mark yours as a dupe. Feel free to elevate the existing bug if you feel the need
<poolie> huwshimi, i don't think i changed my screen resolution recently
<poolie> hm, i did tweak fontconfig to use the ubuntu mono beta font
<poolie> it's possible that changed the layout
<poolie> my bug title was better :)
<poolie> where do people get these things
<huwshimi> poolie: Yes, your description was probably better too
<huwshimi> poolie: welcome to the problem of using a combination of relative and fixed sizing :)
<wgrant> lifeless: It looks like the problematic change was actually a little earlier... 12126.4.1
<wgrant> I don't know why 12156.8.20 seems to have that change too...
<huwshimi> poolie: Although the mono font doesn't look like it should have affected that
<lifeless> it was meant to be reverted according to the bug bac filed
<lifeless> I wonder if someone remerged it in accidentally
<wgrant> THe diff in the MP shows it being changed, not restored or removed.
<wgrant> Odd.
<poolie> maybe i just noticed it today
<wgrant> Oh yeah, github's branch browser is nice. We should totally have one that doesn't suck.
<wgrant> lifeless: Hmm. The SQL statement value substitution has broken duplicate statement detection completely.
<wgrant> That's a bit sad.
<wgrant> I think.
<wgrant> Or maybe this view just does lots of raw SQL.
<lifeless> wgrant: python strings send to python-pgsql are u"foo" - this threw lpoops off for a few days
<lifeless> but it should be fixed now
<lifeless> whats ths oops
<lifeless> ah
<lifeless> milestones also trigger lazy evaluation on bug searches
<wgrant> Huh.
<wgrant> In OOPS-1878QS4 I see lots of queries for people who are message owners... but those comments came out of memcache.
<lifeless> the bind variables stuff captures at our layer, not what the serialised form of real sql would be
<lifeless> if yo usee what I mean
<lifeless> wgrant: makes sense
<wgrant> Why?
<lifeless> wgrant: memcache is incompatible with eager loading
<wgrant> Sure.
<wgrant> But this is a separate query for each message.
<lifeless> hmm
<lifeless> yes
<wgrant> Which means something is accessing message.owner.
<lifeless> right
<wgrant> Something other than the message view.
<lifeless> permission checks
<wgrant> On a message?
<wgrant> Messages have no security.
<lifeless> read get_visible_messages
<wgrant> wut
 * wgrant needs a sharper desk.
<lifeless> now you see why I was like \o/ when I got shit working
<wgrant> Are you burning that, or should I?
<lifeless> wgrant: I took it out of the key path for performance
<lifeless> wgrant: by ignoring it for range query purposes
<lifeless> wgrant: we need to push it down to the db, delete things that we don't want, etc.
<lifeless> wgrant: are you seeing VPC calls ?
<lifeless> wgrant: if so, the getMessagesForView eager loading may be incorrect
<wgrant> lifeless: There are also VPC calls, which I was about to look into.
<wgrant> There are also another 60 person queries unaccounted for, even for anonymous requests.
<wgrant> I'm trying to track those down.
<lifeless> cool
<lifeless> another review plox https://code.launchpad.net/~lifeless/launchpad/bug-717394-4/+merge/50548
 * StevenK grumbles at tal
<StevenK> It works for one case, but not the other? :-(
<StevenK> thumper: Still around?
<lifeless> hmm
<lifeless> 5
<lifeless> branches today
<wgrant> Not bad.
<lifeless> Should I do more ?
<wgrant> Yes.
<wgrant> Always.
<lifeless> it is nice to be doing stuff
<StevenK> lifeless: Can haz a little TAL help?
<lifeless> sure
<lifeless> how do you like it ?
<StevenK> lifeless: I can pastebin a diff and a traceback?
<lifeless> sure
<lifeless> wgrant: ahha, I think I know why the subscriptions are needed to do bug pages
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/webapp/menu.py", line 262, in _buildLink
<lifeless>     linkdata = method()
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/structuralsubscription.py", line 338, in subscribe
<lifeless>     if sst.userHasBugSubscriptions(self.user):
<StevenK> lifeless: Just had a thought, let me try something
<wgrant> Hah.
<StevenK> lifeless: Right, I have it working, but not how I want
<lifeless> garh fail
<lifeless> the vocabulary to change bug targets
<lifeless> type 'launchpad' in there
<StevenK> lifeless: http://pastebin.ubuntu.com/569920/
<StevenK> lifeless: Looking at the page in my browser, I get: "Built by recipe generic-string548131 for Person-name548129" which is brilliant, but I'd like the recipe to be a link, and it isn't.
<lifeless> StevenK: and what do you want it to do?
<lifeless> you don't want tal:replace
<StevenK> lifeless: And given this is first time ever touching tal, I suspect I'm missing something fundamental
<lifeless> replace replaces a tag by something
<lifeless> e.g. structure, a string or a context
<lifeless> you want to set the href
<lifeless> e.g.
<lifeless> tal:attributes="href context/bug/duplicateof/fmt:url">bug
<lifeless>              #<span tal:replace="context/bug/duplicateof/id">42</span></a>
<lifeless> bah
<lifeless> <a
<lifeless> before that
<StevenK> Right
<StevenK> lifeless: Excellent, thanks
<lifeless> cool
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/722455 may entertain you, amidst more headbanging
<_mup_> Bug #722455: bug task target selector widget cannot be used to select 'launchpad' <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722455 >
<wgrant> lifeless: Oh, that's even more awesome than last time I tried.
<StevenK> lifeless: It does the same thing for 'launchpad-project' too
<lifeless> win
<lifeless> though launchpad-project can't have bugs, it would be nice to list all the elements of it there
<StevenK> wgrant: Did you see https://bugs.launchpad.net/launchpad/+bug/722344 ?
<_mup_> Bug #722344: PPA traversal assumes distribution is ubuntu <Launchpad itself:New> < https://launchpad.net/bugs/722344 >
<wgrant> StevenK: Yes.
<wgrant> Let's design URLs without the distribution in them! It's going to be awesome!
<StevenK> And no wails of dispair. Awesome.
<lifeless> ok
<lifeless> why does bugtask need the main archive and archive admin person as json
<wgrant> Hah.
<lifeless> however
<lifeless> I've now audited distro bug search
<wgrant> I didn't realise we pracached the context as JSON.
<lifeless> that might explain the what of it. Tho not the why
<StevenK> lifeless: http://people.canonical.com/~stevenk/built-by-recipe.png
<lifeless> StevenK: that looks pretty nice, but why is the word 'for' in ?blue?
<StevenK> lifeless: The whole thing is one link to the recipe
<lifeless> I wouldn't link the 'for' or person name then
<StevenK> Right
<lifeless> I think folk would expect the person to be a link to the person, if anything
<StevenK> Okay, fixed
<StevenK> lifeless: Just wanted your thoughts before I put together an MP and beg for a UI review
<lifeless> StevenK: de nada
<StevenK> (My first, surprisingly)
<wgrant> UI review? Surely you jest.
<StevenK> wgrant: I have your manager on the phone, he wants a word.
<StevenK> :-P
<wgrant> TALES, please stop swallowing every exception under the sun with LocationError...
<StevenK> wgrant: Ah yes, I've been hitting that today too
<wgrant> It even swallows NameErrors.
<wgrant> That I did not expect.
<StevenK> "Feature"
<wgrant> I bet it would swallow SyntaxErrors if they weren't fatal to compilation...
<StevenK> wgrant: Try it and raise one
<lifeless> StevenK: whats the milestone for ?
<StevenK> lifeless: Which milestone?
<lifeless> the one you set on the launchpad selection bug
<StevenK> If it was 722455 that was by accident
<lifeless> indeed, it is
<StevenK> wgrant already fixed
<lifeless> well
<lifeless> wgrant: why the milestone?
<StevenK> Haha
<wgrant> Assuming StevenK's assignment to the milestone indicated urgency, 11.04 was clearly a mistake.
<wgrant> For that is some time away.
<lifeless> ah
<lifeless> so milestones aren't part of our what to work on workflow
<StevenK> False assumption, sadly
<lifeless> I really need something that will flash my browser window when an SSL handshake occurs
<wgrant> The added latency isn't enough of a stab in the face?
<lifeless> I'm in NZ
<lifeless> and we have other queue points
<lifeless> so it can be hard to be /sure/
<lifeless> stub: good morning!
<lifeless> stub: you're down as asiapac OCR for mondays.
<stub> yo
<stub> I am?
<lifeless> stub: so I'm going to pounce on you now:
<StevenK> Read as: "I have latency to everything, except if it's hosted in my house."
<lifeless> apparently
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-636158/+merge/50534
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-717394/+merge/50541
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-717394-2/+merge/50545
<stub> I never entered the program, and in general spoke out against it. I've never been OCR :)
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-717394-4/+merge/50548
<stub> Oh... OCR... different acronym. Yes
<lifeless> stub: https://dev.launchpad.net/ReviewerSchedule is where I found this, if you need to change it
<lifeless> stub: anyhow, I can has some reviews?
<stub> Nah... just too early to deal with TLAs
<stub> Yes, I am OCR :)
<lifeless> cool
<lifeless> I think I have ubuntu context bug searches down to a constant number - 40 queries (contrast with 118 on lpnet)
<lifeless> hmm
<lifeless> we need more qa
<wgrant> Oh?
<wgrant> We do, but do you have a specific example?
<lifeless> we're only half qaed
<lifeless> also
<lifeless> can you check my deployment request for other gotchas in your feature flag changing patch
<wgrant> Ah, I'm blocking on a LOSA for mine :(
<wgrant> lifeless: Doesn't the value need to be ''?
<lifeless> wgrant: it gets stripped
<wgrant> Oh, handy.
<lifeless> long as there is a trailing space it will be good
<lifeless> I'm /pretty/ sure
<wgrant> Let's see.
<lifeless> flag, scope, priority_str, value = re.split('[ \t]+', line, 3)
<wgrant> Indeed.
<wgrant> So it all looks good.
<lifeless> stub: btw, mthaddon says that apache can be told to use one haproxy as failover
<lifeless> so we don't need to chain the haproxies at al
<stub> yay.
<lifeless> wgrant: ping
<wgrant> lifeless: Hi.
<lifeless> -> other
<lifeless> oh I love getting ISE's from wiki.u.c
<thumper> https://code.launchpad.net/~thumper/launchpad/choosing-recipe-name/+merge/50530 anyone... anyone...
<thumper> not that I'll be around to respond...
<wgrant> stub: Hi.
<stub> yo
<wgrant> stub: I started destroying Zopeless yesterday. I got rid of it from most scripts and got all passing their tests except librariangc.
<wgrant> It passes around a connection internally and creates dozens of cursors from it.
<wgrant> And its tests use this fact to maintain parallel transactions.
<wgrant> This doesn't work so well when everything is using the global transaction manager.
<stub> I don't recall doing that but I guess I must have :)
<wgrant> I think I can sensibly collapse it into one cursor and transaction.
<wgrant> But I was wondering if there was a reason I should not.
<stub> Can we just steal the connection from Storm internals instead of Zopeless internals?
<stub> If the tests pass, it is fine.
<stub> Oh...
<stub> wgrant: Each step looks like it commits at the end already. But changing this to use transaction.commit() and Storm's internal connection fails?
<wgrant> The tests seem very crufty and are possibly already not testing what they should, since the transaction lifetimes are identical and implicit begin is on globally.
<wgrant> stub: It's not that it fails. It's that I'm scared I'm missing a reason for this strangeness.
<StevenK> wgrant: How big is that diff, out of interest?
<wgrant> StevenK: The first step (which I'm about to re-EC2) removes use of specialised Zopeless features from scripts. It's 400 lines.
<wgrant> The big diff is going to be in making the tests work on the non-ZTM.
<wgrant> The second branch is renaming Zopeless mail and detaching it from ZTM. Then another branch introduces a new database configuration thing and shifts scripts onto that.
<wgrant> Then I just need test helpers to replace switchDbUser.
<wgrant> And that will be done.
<wgrant> Because production code doesn't use it any more.
<wgrant> *cough* buildd-manager
<wgrant> I was surprised at how few test failures there were after replacing initZopeless and ZTM with 10 lines...
<stub> wgrant: The whole process runs serially, so there won't be odd fallout. If it is wierd, it is because it is some of the oldest code in our codebase.
<lifeless> stub: btw, for your interest - https://code.launchpad.net/~lifeless/launchpad/rabbit/+merge/50488 - failed on ec2, but the diagnostics were insufficient
<stub> The test might be crufty, but they have good coverage.
<wgrant> stub: Right, but some comments in the tests suggest that it is trying to check its transaction usage.
<wgrant> But that is already false.
<wgrant> So I'm not loosening them any further.
<stub> wgrant: Which tests are these? I don't see it.
<wgrant> stub: ftests/test_gc.py
<wgrant> It does lots of explicit begins and aborts while saying "merge_duplicates should have committed"
<wgrant> It may just be explaining that it is expected, or it may be explaining that the following lines verify that. I can't really see how, but I am going to be paranoid about something like this :)
<stub> That is a comment why the following ztm dance is needed I think (I don't think that was me - I always use implicit begins)
<wgrant> Right.
<wgrant> Thanks.
<stub> Other transaction stuff seems to be to deliberately break things, leaving disk and db inconsistent. Most of these have already been switched to using transaction.abort()
<adeuring> good morning
<lifeless> stub: thanks for the reviews
<bigjools> morning
<henninge> Hi wallyworld!
<wallyworld> hi there
<henninge> wallyworld: someting changed in your branch, the link to create a recipe is gone.
<henninge> wallyworld: do you know where it went? I guess that is from something you merged?
<wallyworld> henninge: the build now branch?
<henninge> yes
<henninge> I go to https://code.launchpad.dev/~vcs-imports/evolution/main
<wallyworld> i'll check the code - it's possible. tim also checked in stuff which touched the same pt file
<wallyworld> and there were conflicts
<henninge> and it used to have a "Related source package recipes" section in r12389 but that is fone now.
<wallyworld> so perhaps i made a mistake dealing with those
<henninge> s/fone/gone/
<wallyworld> give me 1/2 an hour or so - i've got to get the kids to bed :-)
<mrevell> Hello
<henninge> wallyworld: sure ;-)
<henninge> Hi mrevell ;)
<jml> good morning
<henninge> Hi jtv!
<henninge> jtv: how many more mornings at this time UTC?
<jtv> hi henningeâ¦ just this week
<wallyworld> henninge: i think i remember what happened - tim landed a branch which put all recipe stuff behind a feature flag
<wallyworld> you need to do this: insert into FeatureFlag (flag, priority, scope, value) values ('code.recipes_enabled', 1, 'default', 'o
<wallyworld> n')
<henninge> wallyworld: I figured something like that ;-)
<henninge> let me try
<wallyworld> ok
<henninge> wallyworld: looks perfect! r=me (ui) ;-)
<wallyworld> henninge: \o/ thanks!
<henninge> wallyworld: the MP is timing out for me atm.
<henninge> I will add the approval later.
<wallyworld> henninge: np. launchpad has been slooooow for me today too
<bigjools> something is horribly borked in the DB
<wallyworld> henninge: thanks for all the effort you put into reviewing it
<wgrant> henninge: Which URL timed out?
<wallyworld> bigjools: is borked the technical term for fucked?
<bigjools> wallyworld: fucked *is* the technical term
<wallyworld> :-)
<jtv> it's spelled b0rked
<jtv> henninge, danilos: did you see the problem with the wiserearth zh_cn stats?
<henninge> nope
<danilos> jtv, I didn't
<jtv> Reporting 95.x% translated, whereas in reality it's just under half.
<jtv> I wonder if maybe we're just adding up rosettacount and currentcount in the display.
<jtv> Yes, we are.
<henninge> wgrant: looks like any mp
<jtv> danilos, henninge: rosettastats is still computing translatedCount as currentCount + rosettaCount.  Do you concur that that is wrong in the recife world?
<jtv> Or is it the interpretation of currentCount that's wrong maybe?
<danilos> jtv, currentCount should be translations that are active in both ubuntu and upstream
<danilos> jtv, with that, it should still be correct
<jtv> danilos: then it looks as if we're probably computing the wrong thing
<danilos> jtv, calculation of currentCount might be wrong
<jtv> we never updated the interface docstring for currentCount.  It basically says "current translations."  :(
<jtv> Well, "current upstream."
<danilos> jtv, oh, it was incorrect for ages then :)
<jtv> No surprise with RS!
<jtv> I mean RosettaStats, not Serbia.
<danilos> jtv, yeah, basically, we need to do away with RosettaStats, as we all know :)
<danilos> jtv, the mapping on ITranslatedLanguage.translation_statistics (or something) should clarify it all
<danilos> jtv, TranslatedCount.recalculateCounts() transforms RosettaStats counts to useful counts, fwiw
<danilos> jtv, RosettaStats meaning of "current" was always "synced with upstream"
<jtv> Yes
<jtv> Unfortunately the documentation remained confused all over the place
<danilos> jtv, right; so, it's likely that TranslatedLanguage (uhm, not TranslatedCount :) needs to be updated to be side-aware
<jtv> TranslatedLanguageMixin still takes about "imported" messages
<danilos> jtv, and others
<danilos> jtv, yeah, that seems to be the problem
<wgrant> henninge: Thanks, that helped to track the issue down.
<lifeless> stub: update to the query: http://bazaar.launchpad.net/~lifeless/launchpad/bug-636158/revision/12415
<jtv> danilos: it's the actual pofile.currentcount that seems to be wrong.
<jtv> so not a TL problem
<danilos> jtv, :(
<danilos> jtv, that means that updateStatistics is wrong
<jtv> yes
<danilos> jtv, current count is the only one not based on an actual POFile:+translate filter, iirc
<jtv> danilos: that's a long time agoâwe (I, I'm afraid) rewrote to compute basically everything in one query.
<danilos> jtv, oh
<danilos> jtv, well, then you probably know much more about it than I do :)
<jtv> yar
<jtv> danilos, one thing you do know better probably: do we still run the full stats update runs?
<jtv> ISTR you stopped them at some point
<jtv> And this is data that hasn't changed in a whileâ¦
<danilos> jtv, yeah, the daily ones are not working because of DBLoopTuner craziness
<danilos> jtv, but weekly ones run on Thu evenings or something
<jtv> so what about the weekly ones?
<jtv> :/
<lifeless> stub: can you update your vote please (assuming the new query makes you happy ;))
<jtv> danilos: the stats query selectsâ¦
<jtv>                     (Other.id = Current.id) AS same_on_both_sides
<jtv> so the intention is certainly right
<danilos> jtv, you can always confirm that in
<danilos> jtv, ...crontabs
<jtv> danilos: yes, but having some painful wrist trouble, trying to minimize movement
<danilos> jtv, you probably had hard time mapping to rosetta stats from sane values: I am pretty sure it's not even a bijection
<jtv> danilos: why would it have to be?
<danilos> jtv, because when it isn't, it means that in some cases you can't even get a correct value back
<jtv> danilos: think I have it
<jtv> rosettacount includes both "translated differently" and "translated this side only"
<jtv> but no
<jtv> that wouldn't affect currentcount
<jtv> Damn, it's empty TMs.  :(
<jtv> They're shared upstream/ubuntu, but "not translated on the other side"
<wgrant> jml: Did you get a response to your PQM submission?
<jml> wgrant: which one?
<jml> wgrant: more-canonical-cleanups got rejected because of testfix
<wgrant> Ahh, so we are testfix.
<jml> with that new-fangled less-useful email that PQM sends these days.
<wgrant> I keep getting no response to my shipit branches.
<wgrant> Perhaps that is relevant.
<wgrant> Forced.
<bigjools> do we have a "right" way of removing the hundreds of old eggs/dependencies?
<jml> bigjools: perhaps doc/buildout.txt has the answer?
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge | https://code.launchpad.net/launchpad-project/+activereviews
<bigjools> real documentation!
<bigjools> unfortunately it doesn't say
<bigjools> I suspect this is like the linux kernel package problem
<bigjools> I have 21 of them installed here :/
<jml> bigjools: yeah, doc/buildout.txt is one of the more useful technical docs in the tree
<stub> lifeless: Not seeing the new diff in the mp :-(
<wgrant> stub: mizuho was broked... it's possible the diff update job failed.
<bigjools> jml: it is
<wgrant> It turns out that PQM's response to an invisible branch is nothing.
<wgrant> Excellent.
<jml> it's not the most polished of tools.
<danilo> allenap, hi, got a minute perhaps?
<jtv> Reviewer wanted!  https://code.launchpad.net/~jtv/launchpad/bug-722568/+merge/50593
<jtv> Hmm strange error from bzr lp-propose:
<jtv> ERROR: Too many attempts to read from the player!
<henninge> adeuring: Hi
<adeuring> moin henninge!
<henninge> adeuring: wanna do a stand-up call?
<adeuring> henninge: , sure let's do it. I have a small question about a factory method
<jml> gary is not here :\
<jml> hm.
<jml> President's Day
<beuno> gary's the president now?
<danilos> :)
<danilos> henninge, hey, I see you are OCR, and I am fighting a zope permissions problem
<danilos> henninge, and I remember you knowing a lot about those :)
<jtv> Still need a reviewer!  https://code.launchpad.net/~jtv/launchpad/bug-722568/+merge/50593
<jtv> henninge:            ^^^^^^^^^^   from when you were at lunch probably
<henninge> jtv: oh, missed that.
<jtv> henninge: you weren't connected at the time
<henninge> jtv: I am working on another review but I can take yours next.
<jtv> great, thanks
<henninge> danilos: what is the question?
<danilos> henninge, I figured it out, thanks :)
<henninge> adeuring: Please have a look if this affects your current work in any way: https://code.launchpad.net/~jtv/launchpad/bug-722568/+merge/50593
<adeuring> henninge: yeah,  Ithanks, I've seen it. jtv spotted an issue I hadn't yet
<jtv> adeuring: you're also working on stats?
<adeuring> jtv: bug 688130
<_mup_> Bug #688130: Statistics clean-ups and extra tests <lp-translations> <upstream-translations-sharing> <Launchpad itself:In Progress by adeuring> < https://launchpad.net/bugs/688130 >
<jtv> "Test for empty messages"â¦ that's something my branch does
<jtv> "Treat incomplete messages as untranslated" will be affected.
<adeuring> right
<danilos> henninge, will you be able to fit in https://code.launchpad.net/~danilo/launchpad/bug-722626/+merge/50618 as well soon? (it is very short branch)
<henninge> danilos: No.
<henninge> :-P
<danilos> :(
<henninge> danilos: let me look first ... ;-)
<danilos> henninge, 98 lines of diff according to "wc -l", so, pleaaaseee
<henninge> danilos: why is it depending on a branch that has no mp?
<danilos> henninge, my overeager pipelining
<danilos> henninge, nothing in that branch
<henninge> ah!
<henninge> danilos: I will look at it as soon as I am done with jtv's branch.
<danilos> henninge, this bug was uncovered while trying to land a different branch
<danilos> henninge, thanks
<henninge> jtv: abentley recently added a "side" parameter to makePOFile, did you know that?
<henninge> This way it is easier to create messages on the other side.
<jtv> henninge: ah yes, I saw thatâ¦ did I neglect to use it?
<jtv> You mean, create an extra POFile and then create a current message in that?
<henninge> jtv: exactly but I am not sure it is worth it
<jtv> It does seem a bit roundabout.
<henninge> jtv: remind me how COALESCE works, please.
 * henninge goes rtfm
<henninge> jtv: r=me
<jtv> thanks!
<jtv> hehâ¦ no proof-of-read?  :)
<jtv> tsk
<henninge> danilos: I am pretty sure that using check_permision in security.py is broken
<henninge> because check_permission assuems a user and that might (theoretically I admit) not be the same as is requested by checkAuthenticated
<henninge> so you might end up checking permissions for the wrong user ... theoretically.
 * henninge wants to fix LP security so badly!
<danilos> henninge, well, what would you propose to be used instead?
<danilos> henninge, (not to mention that it's already used elsewhere in security.py)
<henninge> yes, I feared you where going to ask that ...
<henninge> That is no excuse! :-P
<henninge> danilos: Maybe you cannot do anything different (easily) but you have to admit that it is broken.
<henninge> checkAuthenticated asks for a specific user which is ignored.
<danilos> henninge, heh, well, it's broken but it's convenient
 * henninge looks closer
<danilos> henninge, the right approach for my particular problem would be to check all target types (IProduct, IProductSeries, IDistribution, ...) and use their own checkers instead
<danilos> henninge, and doing it that way is even uglier and even more broken imo
<henninge> yes, I can see that
<henninge> danilos: r=me
<danilos> henninge, thanks
<danilos> henninge, I did learn about IPersonRoles playing with security.py right now (by having to use user.person to pass in :)
<danilos> henninge, also, only now I realize your point about check_permission not having a user parameter
<danilos> henninge, I am a bit more worried right now
<henninge> danilos: check_permission goes through to zope.security.checkPermission which in term uses the "current interaction"
<henninge> if no other is passed in.
<danilos> henninge, yeah, I realize that
<henninge> I am not sure how likely it might be that "user" is differnt from the "principal" in the current interaction.
<henninge> s/in term/in turn/ ... JFTR
<bigjools> jml: if I wanted to force the ftp scenario in the test_poppy tests, how would I do that?
<jml> bigjools: looking.
<bigjools> jml: thanks.  the best I could think of was to include "ftp" as the test match but that doesn't exclude sftp... I was hoping there would be a magic matcher for scenarios
<jml> my machine is slow today.
<jml> 23s to start the database
<jml> bigjools: you need to match the parenthesis
<bigjools> ah ok so it is part of the test name
<bigjools> thanks
<jml> yeah.
<bigjools> jml: you and StevenK did a great job of writing new tests for the older FTP stuff - I can just replace the setup phase - trivial - and run them!
<jml> There are some subtleties there, tests have at least three things that could be called "name",
<jml> bigjools: glad to hear it :)
<bigjools> I just changed 2 lines to use a different fixture!
<bigjools> jml: unfortunately I think the test runner hates parentheses :/
<bigjools> no tests matched
<jml> bigjools: try this: ./bin/test -cvv test_poppy '\(ftp\)'
<bigjools> ah escape
<jml> it's a regex.
<bigjools> now I really do have problems
<jml> bigjools: :)
<bigjools> jml: one more thing - were you involved with the actual tests in that file?
<bigjools> I'm trying to work out the rather opaque "self.assertEqual(os.stat(wanted_path).st_mode, 0102674)"
<jml> bigjools: I *think* I was, but I don't remember.
<bigjools> since it fails now :)
<bigjools> but that is such a magic number I've no idea what it's trying to test :/
<jml> bigjools: Don't know off the top of my head. I'd look up os.stat documentation for the constants and start making some educated guesses.
<bigjools> yeah I was going to head there - just wondered if you could save time :)
<jml> https://code.launchpad.net/~jml/launchpad/list-ec2-test-runs-721784/+merge/50537 might interest some folk
 * jml has to dash off
<jml> g'night all.
<bigjools> nn jml
<jml> be back later to talk to Aussies
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> hi
<jtv> hi lifeless!
<jtv> say, would it be okay with you to set a much longer timeout for the approval form for translations import queue entries?  It's blocking queue review.
<lifeless> whats the page id
<jtv> TranslationImportQueueEntry:+index
<lifeless> hmm, thats interesting
<lifeless> according to the histogram on  https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-pageids.html
<lifeless> its not had any requests near the timeout
<lifeless> though of course, thats probably showing the weekend
<lifeless> how long do you think it needs?
<jtv> This is a page only a handful of people can access, so you won't be seeing many timeoutsâespecially during the weekends.
<jtv> I'm taking a wild stab at 30 seconds for now.  Terrible, I know.
<lifeless> we can't set it to that
<lifeless> (note, not 'I don't want to' : cannot)
<jtv> Right, I noticed thatâwhy not?
<lifeless> haproxy will shutdown a request with a total in-dc queue time of 30 seconds
<lifeless> so will apache
<lifeless> ha proxy currently has 1second +- 0.5 seconds queuing on most requests afaict through adhoc measurement
<lifeless> setting anything to > 28 seconds has a high probability of running into the 30 second time on either of apache or haproxy
<jtv> Okayâ¦ so how about 25 seconds?
<lifeless> that said, this page worked when we had a 15 second timeout, no ?
<jtv> Not to sure of that.
<jtv> You see, the work that this form does hasn't been getting done for a while.
<jtv> We're in the process of fixing that.
<lifeless> ah
<lifeless> ok - add a request to LPS to set it to 20 seconds
<lifeless> lets see if thats sufficient
<lifeless> I'm +1 on that
<jtv> We can also fix the code to be faster, but a bit of goodness will have to be sacrificed until we can work something out with rabbit or somesuch.
<jtv> Thanks.
<lifeless> looking at the oops
<jtv> In any case it'll allow me to get _more_ done, and that's something.
<lifeless> it has 600ms of query repeated 19 times
<jtv> You'll see a lot of repetition in the oops; not something that can be batched very easily.
<lifeless> the plan will be the key
<bigjools> night all
<jtv> night bigjools!
<lifeless> if its getting into scans, doing a larger query may be no slower
<jtv> It's not a matter of slower so much as of complexity.
<jtv> This query, believe it or not, is already pretty well-optimized compared to what we used to have.
<lifeless> you say in the bug that its statistics
<jtv> It is, yes.
<lifeless> I thought a background job did that already?
<jtv> Yes, but it's slow.
<jtv> But I am indeed thinking of just not calculating them in this scenario.
<lifeless> does that matter for approval ?
<jtv> After all, since the end user isn't hitting this form, there's no expectation of an interactive update.
<jtv> There is also already an accepted situation where the template is empty until the import happens.
<jtv> And *blink* when that happens, the statistics need to be recomputed anyway.
<jtv> In fact they are, IIRC.
<jtv> I don't see any point to calculating them at this particular point!
<jtv> Why don't I just go fix thatâ¦
<lifeless> cool
<lifeless> btw
<lifeless> when I add an additional language to the query
<lifeless> it stays at 300ms
<lifeless> with the same plan
<lifeless> of course this is on qastaging, so may be useless for production prediction
<lifeless> ...and I didn't finish the conversion. bah.
<jtv> May well be different there, yes, though you're getting a similar time and that suggests it may be a similar plan.
<jtv> Conversion?
<lifeless> adding group by, and = -> IN ()
<lifeless> jtv: I've added it to the bug, in the offchance that its helpful
<jtv> thanks lifelessâthough AFAICS we can just eliminate this use-case and then only template imports will need to do this calculation in bulk.
<lifeless> I think thats better still :)
<jtv> There may still be a few interactive cases though that may run several of these, so who knows.
<lifeless> we can look at their performance when we get timeout protection on backend services
 * jtv looks forward to that day in trepidation
<lifeless> heh
<lifeless> we need it - rogue backends regularly cause frontend timeouts at the moment
<lifeless> even with tunable loop
<jtv> And I beheld the sixth terrible horseman, and his shirt said "Internal Server Error"
<lifeless> :(
<lifeless> what site ?
<jtv> Just a general observation.
<jtv> lifeless: btw the postgres-R presentation at FOSDEM was impressiveâbut alas not production-ready yet.
<lifeless> http://www.fosdem.org/2011/schedule/event/clustered_databases ?
<jtv> Optimistic multi-masterâ¦ it felt a lot like distributed 2PC
<lifeless> was it taped?
<jtv> Yes, and I think it was taped.  If you find a recording, you may recognize the voice of the heckler in the front.
<lifeless> :)
<lifeless> uhm
<lifeless> does it partition as well?
<lifeless> what we need for scaling is either much less bulk/historical data (gives us a lower coefficient per user-year)
<jtv> I don't think it does that, no.
<lifeless> or the ability to scale everything horizontal - lets us choose an good price-point per machine and add machines
<jtv> Although if disk space isn't the issue, I suppose one could get similar effects by implementing some kind of affinity layer on top.
<lifeless> scaling horizontally requires capping the local data per node in some fashion
<jtv> Try to get locality for caches through clever load distribution.
<lifeless> (e.g. a nfs backend that can be raided across many servers - meep!)
<jtv> aaiggh!
<elmo> ..
<lifeless> now that I've made your eyes bleed
<elmo> lifeless: did you just suggest nfs in the context of DB servers?
<jtv> it was him!  it was him!
<jtv> I didn't do anything!
<lifeless> elmo: Not in any serious sense
<jtv> Maybe with sqliteâI hear nfs is really good with locks
<lifeless> that would be a win
<jtv> Anyway, as long as we're blue-skying, we already have slaves, so I don't see a huge difference with multi-master when it comes to resource allocation.
<lifeless> so -xc is referenced in the slides as doing partitioning
<jtv> I was just saying it mightn't even matter so much how much is on each disk, as long as you could limit how much needs to be in memory on a given node.
<jtv> Not limit actuallyâmore like guide.
<lifeless> jtv: our cluster isn't symmetric: the load isn't evenly balanced atm, though multimaster might help with that substantially, the master and slave machines are not equivalently sized
<lifeless> elmo: so what are you guys sprinting on - puppet ?
<jtv> lifeless: postgres-R has no problem with that, with the obvious exception (which someone insisted on lament^H^H^H^Hdiscussing at length) of increased lock contention.
<elmo> lifeless: i'm not there, so I'm at best an idle spectator, but it's just our bi-annual sprint - puppet is one of the focuses this time though, yeah
<lifeless> elmo: cool
<lifeless> jtv: speaking of rabbit, I tried to land an incremental step towards it in the weekend but it failed on ec2; I haven't dug into why, yet.
<jtv> lifeless: I wish I'd gotten around to looking into itâ¦  but this is something that we've long, long wanted message queues for.
<jtv> It'll let us lift these queries out of interactive transactions and be more aggressive about recalculating.
<lifeless> you can use the regular task system today
<lifeless> or put in some energy towards rabbit
<lifeless> :)
<jtv> I wonder if the existing task system would get us to the desired response time of "a few seconds" though.
<lifeless> if you can't do it in a 13 second web request
<lifeless> then a task / queue system won't do it either ;)
<lifeless> james_w: what project are your soup matchers in ?
<lifeless> ah found it
<Ursinha> bug 711064
<_mup_> Bug #711064: POFile:+translate timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/711064 >
<lifeless> Ursinha: >
<lifeless> Ursinha: ?
<Ursinha> lifeless, seeking help of the bot to get the bug link
<lifeless> ah :)
 * thumper relocates to somewhere with coffee to hand
<jtv> No OCR?
<jtv> Reviewer wanted!  https://code.launchpad.net/~jtv/launchpad/bug-719267/+merge/50651
<jtv> It's small.
<jtv> Very small.
<jtv> thanks once more lifeless!
<lifeless> jtv: no worries
<lifeless> jtv: hey
<lifeless> do you know why +needs-packaging does pofile lookups ?
<lifeless> bug 722794
<_mup_> Bug #722794: DistroSeries:+needs-packaging timeouts <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722794 >
<jtv> lifeless: priority computation IIRC
<jtv> There's some kind of rating system that includes a wide variety of factorsâ¦ sinzui will know the details.
<jtv> As so often in line.
<jtv> life.
<lifeless> thanks
<flacoste> lifeless: call?
<flacoste> lifeless: https://lpstats.canonical.com/graphs/LPProjectCriticalBurndown/
<flacoste> https://lpstats.canonical.com/graphs/LPProjectBugsClosed/
<flacoste> https://lpstats.canonical.com/graphs/LPProjectCriticalFixed/
<wallyworld> thumper: we having standup?
<thumper> aye
<StevenK> http://people.canonical.com/~stevenk/built-by-recipe.png
<huwshimi> Morning
<jml> huwshimi: hi
<jml> huwshimi: will brb.
<huwshimi> jml: Sure
 * StevenK grumbles at the lack of UI reviewers in this timezone
<StevenK> And no, I'm not volunteering. :-P
 * jml in back
<flacoste> lifeless: https://lpstats.canonical.com/graphs/PPR/20100222/20110222/
<lifeless> 6312026
<lifeless> 2.78
<lifeless> flacoste: https://lpstats.canonical.com/graphs/PPR/20110208/20110215/
<poolie> hi flacoste
<flacoste> hi poolie
<flacoste> poolie want to chat earlier?
<poolie> lifeless, teddy, just thinking about flags for controlling dkim
<poolie> i think a per-email-domain scope may be yagni for now
<poolie> we can add one when there's a second thing controlled by email
<poolie> does thta make sense to you?
<poolie> flacoste, sure, give me a few minutes to get set up
<lifeless> poolie: how would you do it today then?
<poolie> just a flag that gives a space-separated list of trusted domains
<poolie> just wondering if is this enlightened laziness or the regular kind
<lifeless> it might be a bit awkward for dealing with concurrent updates
<lifeless> a row-per-thing would be easier (once we stop regenerating all the rules)
<lifeless> poolie: so I think a per email scope would be quite beneficial
<poolie> ok
<lifeless> jml: welcome back
<lifeless> jml: looked at the testtools thing yet ?
<lifeless> thumper: hi; did you know about soupmatchers?
<thumper> lifeless: yes
<lifeless> thumper: needed something different again ?
<thumper> somewhat
 * thumper has head down right now
<jml> lifeless: no, going to after this call.
<lifeless> thumper: I'd like us to not have generic matchers in the lp tree - they should be in testtools, or in soupmatchers etc - just to keep our code base focused
<lifeless> thumper: gl with your head downing
<thumper> I agree in principle
<lifeless> (there is a but there, isn't there?)
<jml> satisfice!
<jml> lifeless: No, I haven't sadly. I don't really get time for much discretionary hacking on Mondays.
<jml> lifeless: I'll probably be able to take a look tomorrow. Right now am too knackered to be of any use.
<lifeless> jml: sleep well
<wgrant> Night jml.
<jml> g'night
<lifeless> wgrant: hey
<lifeless> wgrant: what timeout are you beating up on today?
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: Finishing tracking down all the Person queries.
<wgrant> I've fixed the comment ones. But there are still lots from permission checks.
<lifeless> bugtask:+index?
<wgrant> Yes.
<lifeless> cool
<wgrant> Bug #1 has ~140 Person queries, which means 100 aren't from comments :(
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> wgrant: activities perhaps
<wgrant> Possibly.
<wgrant> We'll find out soon.
<wgrant> You?
<lifeless> working on the test fixes ec2 found for me last night
<lifeless> waiting for stub to discuss the bug search query change
<wgrant> Aha
<lifeless> oh
<lifeless> and writing leps
<wgrant> For?
<lifeless> oops
<lifeless> parallel testing
<poolie> lifeless, so istm the incoming mail processor needs to be able to establish a flag scope from something like a context manager
<poolie> which is eminently possible, but there is no infrastructure at the moment
<lifeless> bug 717394 should get through ec2 this time
<_mup_> Bug #717394: Distribution:+bugs timeouts <qa-ok> <timeout> <Launchpad itself:In Progress by lifeless> < https://launchpad.net/bugs/717394 >
<poolie> all of them work from global state
<poolie> i'm thinking something like
<poolie> with feature_scopes(MailFromDomain(domain)): ....
<lifeless> poolie: as I understand the caching, you'll want to push the existing controller completely to the side
<lifeless> poolie: anyhow, it sounds fie
<lifeless> *fine*
<poolie> it does have to deal with caching
<poolie> having seen some of the bugs people hit, i think having multiple controllers around may not be such a good idea
<lifeless> wow
<lifeless> massive quake
<lifeless> internet is up
<lifeless> phones & cell towers down
<spiv> lifeless: hooray for internet
#launchpad-dev 2011-02-22
 * maxb hopes the cell towers are only metaphorically down :-)
<lifeless> maxb: at least logically down
<huwshimi> Anyone have any ideas why my tests failed? http://paste.ubuntu.com/570317/ (URLError: <urlopen error [Errno 111] Connection refused>)
<wgrant> huwshimi: Both known occasional spurious failures.
<wgrant> huwshimi: Land the branch manually.
<lifeless> http://maps.google.co.nz/maps?f=q&source=s_q&hl=en&geocode=&q=http://magma.geonet.org.nz/services/quake/kml/2.2/search?externalRef=3468575
<huwshimi> wgrant: You mean just 'lp-land'? Is there anything else I need to do?'
<wgrant> huwshimi: lp-land, yeah.
<wgrant> lifeless: 6.3?
<lifeless> yeah
<lifeless> but shallow
<lifeless> so felt it more
<lifeless> mum, down in dunedin, felt it
<lifeless> thumper: did you ?
<wgrant> Ah, very shallow, yes.
<wgrant> Any major damage known yet?
<lifeless> http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10707996
<lifeless> water mains burst, some buildings down, more road cracks
<wgrant> .au news sources haven't mentioned it yet...
<lifeless> phone lines are down
<wallyworld> lifeless: surely you're not asking thumper if the earth moved for him?
<wgrant> 5km deep!?
<wgrant> That's awfully shallow.
<lifeless> wgrant: yup
<lifeless> its being called an aftershock for now
<thumper> lifeless: yes I felt it
<spiv> Ah, I see geonet.org.nz has it as 6.3 rather than 3.1 now.
<thumper> lifeless: quite a shaker here
<spiv> That's quite a difference!
<lifeless> it was felt in wellington too
<spiv> USGS says 4km deep +/- 0.5km.
<wgrant> lifeless: Why do you invoke PersonSet directly instead of using the utility?
 * lifeless shrugs
<lifeless> I think we need an object for 'group of <type>'
<lifeless> the Sets look ideal for that, if parameterised
<lifeless> I was heading slightly in that direction at the time
<wgrant> Sure.
<lifeless> feel free to do it any which way
<lifeless> doing DI on these singletons seems fairly uninteresting
<thumper> rumours abound in the office here that the christchurch cathedral is down
<lifeless> yup
<lifeless> police are saying that
<lifeless> though the degree of down isn't clear
<wgrant> Ah, now it's hitting the news here.
<wgrant> 'Witnesses said there were buildings down all around Cathedral Square in the city, with the church destroyed.'
<wgrant> Hmmm.
<lifeless> another shock now
<lifeless> relatively gentle
<wgrant> Ah, the cathedral has just lost its steeple. Not tooo destroyed.
<lifeless> the bugger of it is that we're still waiting on a plumber having time to fix some shit for us - but other folk are more badly in need
<wallyworld> anyone know why adding @operation_returns_collection_of(Interface) to a method with an existing @export_read_operation() decorator results in:
<wallyworld> UnknownEntryAdapter: No IEntry adapter found for Interface (web service version: 1.0).  Encountered as a result of the entry interface None, field ''.
<wallyworld> there seems to be several other places in the code where the same pattern is used
<wallyworld> and i can't find anywhere where adaptors are defined for those places
<wgrant> wallyworld: You need to patch them in lib/canonical/launchpad/interfaces/_schema_circular_imports.py
 * wallyworld looks
<wallyworld> wgrant: ah that looks like it. thanks.
<wallyworld> i was getting a circular import when I tried using ISourcePackageRecipe instead of Interface
<wallyworld> wasn't sure how to fix it but now i know :-)
<wgrant> http://www.youtube.com/watch?v=qt0iIHXFnR0
<wgrant> That's quite a rock.
 * thumper stabs the compressed javascript
<wgrant> wallyworld: Yeah, it's a bit awful :(
<wallyworld> awful is one way to describe it :-)
<lifeless> wallyworld: thats sumner
<lifeless> bah
<lifeless> wgrant: thats sumner; just around the hills from where our rental property was
 * wallyworld was wondering what sumner was :-)
<lifeless> wallyworld: its a suburb
<wallyworld> i know that now :-)
<wallyworld> but had to wait for you to post a subsequent message to figure it out :-)
<wgrant> lifeless: Congrats!
<lifeless> wgrant: thanks
<huwshimi> lifeless: Just saw your post: congrats!
<lifeless> thanks
<huwshimi> lifeless: My wife is three weeks ahead of yours.
<lifeless> congrats
<huwshimi> lifeless: Thanks. My first. You have others right?
<lifeless> nope
<lifeless> we're late to the sprogging game
<huwshimi> lifeless: Oh, I'm thinking of someone else. Exciting times then
<spiv> lifeless: congrats!
<lifeless> spiv: thanks
<wallyworld> lifeless: you are always on irc etc, don't know where you found the time for species propagation :-P
<spiv> Multitasking, perhaps.
<wallyworld> spiv: but python is single threaded :-)
<lifeless> wallyworld: doesn't mean *I* am :>
<wallyworld> lifeless: so you can type and do "other" things at the same time. must have been hard to set up your desk that way :-)
<lifeless> no comment, keeping the workplace clean since 1436
<lifeless> bbiab, going to try and top up our supplies before supermarket shelves run dry - we have enough but its bland fare, better to have the remainder of fresh meats etc
<thumper> what does this do?  --> ${objects/?key/webservice:json}
<thumper> the ?key has me wondering
<wgrant> Literally '?key'?
<wgrant> Ah, yes.
<wgrant> thumper: It uses the value of the 'key' variable to continue traversal.
<thumper> ah... ok
<thumper> thanks
<thumper> that makes sense
<wallyworld> i can has review? https://code.launchpad.net/~wallyworld/launchpad/recipe-webservice-api/+merge/50684
<wgrant> 90	def getBuilds():
<wgrant> 91	"""Return a ResultSet of all the non-pending builds."""
<wgrant> That's an odd method name, if the docstring is correct.
<wallyworld> wgrant: i didn't write it, but it does make sense if you realise there's also a getPendingBuilds() method
<wallyworld> which returns the pending builds :-)
<wgrant> wallyworld: We should export a sane API if at all possible, and that method name is insane.
<wgrant> Does the method need to not return pending builds?
<wallyworld> wgrant: yes, it needs to not return the pending builds
<wgrant> Hmm.
<wgrant> Is that useful for API users, though?
<wallyworld> wgrant: there was a refactoring previously which changed those method names. i can't recall the exact circumstances now since i didn't do it
<wallyworld> i guess the method names could be changed
<wallyworld> again
<wgrant> We shouldn't be exposing suboptimal interfaces to people who are going to use them forever so we can never change them.
<wgrant> We can rename just the exports if required.
<wallyworld> wgrant: so given there's 2 methods - getPendingBuilds() and getBuilds() - is it just the getBuilds() name you have a problem with?
<wgrant> wallyworld: I think a single method with arguments may be better. But getPendingBuilds on its own is possibly OK.
<wallyworld> wgrant: there used to be a single method with args and the refactoring converted it to 2 methods :-)
<wgrant> Why? TAL?
<wallyworld> wgrant: not sure exactly. i was never involved in the decision, just noticed that it had happened
<wallyworld> the getBuilds() method returns all builds where status != NEEDSBUILD
<wallyworld> so that could be successful, failed, whatever
<wgrant> Hmm.
<wallyworld> getNonPendingBuilds() sounds a bit funny, or?
<lifeless> getFinalisedBuilds?
<lifeless> Terminal
<lifeless> Complete
<wgrant> Ahh, it's used by builds_for_recipe.
<wallyworld> complete implies success
<wgrant> I don't think exporting them as separate methods is useful.
<wallyworld> i like finalised perhaps
<lifeless> wallyworld: complete does not imply success
<wallyworld> wgrant: so we could have an api just just gets all builds
<wallyworld> i'll do this: getBuilds() returns everything and is exported. i'll add a getCompletedBuilds() which does what getBuilds() does now
<wallyworld> sound ok?
<wgrant> Maybe have getBuilds(include_pending)?
<wallyworld> wgrant: that's what we had before the refactoring :-)
<wgrant> lifeless: Tracebacks in OOPSes are very enlightening.
<lifeless> yes
<wgrant> lifeless: Could not work out where some queries were coming from... turns out they were in __storm_loaded__ in various places.
<wgrant> Grr.
<lifeless> wgrant: the 'is it a team' check ?
<wgrant> lifeless: And is the distribution a derivative or not.
<wgrant> Queries for Ubuntu on every distro load.
<lifeless> let me just say arrrrrrrrrrrrrgh
<wgrant> Oh?
<wgrant> ahh, inTeam often issues two queries. That sort of sucks.
<wgrant> It's very tempting to preload all participation IDs at the start of the request...
<wgrant> We can get rid of a lot of queries if we just prefetch the user's participations and a few celebrities.
<wgrant> lifeless: Have you thought about that at all?
<lifeless> yes
<lifeless> net loss I think
<lifeless> getting one TP entry is very fast
<lifeless> some users have hundreds of TP entries and we'd only be looking at 2 or 3
<lifeless> wgrant: consider lying to objects instead.
<lifeless> wgrant: that is, if we load an object from a context where view/edit is known a priori, grant it during the load, the way bugs _known_viewers gets data injected into it
<lifeless> also
<lifeless> bug 722447
<_mup_> Bug #722447: checking 'well known' teams requires two queries at startup and in tests <Launchpad itself:Triaged> < https://launchpad.net/bugs/722447 >
<wgrant> lifeless: Getting 200 TPs takes 200 microseconds in the DB.
<wgrant> I tried with person=1
<lifeless> wgrant: hot or cold
<wgrant> That was cold.
<lifeless> consider storm deserialisation - its not brillianrtly fast
<wgrant> This is probably the most trivial table in the DB.
<lifeless> wgrant: perhaps you mean milliseconds ?
<wgrant> It's deserialising ints... it goes nowhere near Storm.
<wgrant> No, 0.2ms.
<lifeless> nice
<lifeless> well, lets say I'm open to data on this
<wgrant> I guess the table is so small that it may have been hot enough even on mawson.
<wgrant> It is just two ints...
<lifeless> launchpad_qastaging=> select count(*) from teamparticipation where person=1;
<lifeless> Time: 3.323 ms
<lifeless> wgrant: its got 1.4m rows, so fairly small
<lifeless> and yes, a narrow table - like bugmessage
<lifeless> wgrant: I'm not saying no, I'm saying its not a no-brainer
<lifeless> wgrant: if you bring back team names:
<lifeless> Time: 1694.335 ms
<lifeless> wgrant: it becomes a lot slower
<lifeless> that was 'select team.name from person as team, teamparticipation where teamparticipation.person=317562 and teamparticipation.team=team.id;'
<wgrant> lifeless: Sure. But why would we want team names?
<wgrant> Of course a join across a wideish table is going to be slow.
<lifeless> wgrant: just bringing back ids is pretty useful.
<lifeless> s/useful/useless/
 * wallyworld wants to stab windmill through the heart
<wallyworld> AttributeError: 'AppYUIUnitTestCase' object has no attribute 'client'
<wallyworld> WTF.
<wgrant> lifeless: It's completely sufficient for inTeam, though.
<wgrant> Just not FF.
<lifeless> its only sufficient for inTeam if the team id is known a priori
<lifeless> wgrant: we can instead do an inTeam on PersonSet which would issue one fast query.
<lifeless> interestingly, teamparticipation seems to have redundant entries for code in offspring hackers - three rows
<lifeless> doh, bad query
<wgrant> lifeless: But that would require restructuring huge amounts of code.
<wgrant> If we prefetch participations then later prefetch relevant persons, we can make all of the permission checks free with very little code.
<lifeless> wgrant: my concern is three fold:
<lifeless>  - This sort of thing eventually becomes an albatross : its hard to undo an optimisation like you're proposal
<lifeless>  - its not inherently scalable
<lifeless>  - its very data set specific
<lifeless> we should be able to get down to 4 or 5 teamparticipation checks per page without any special case
<lifeless> I think we should do that first - thats a whole 30ms : we're dealing with 10000ms pages at the moment, so lets reserve the special cases until they are a big win, not a little win
<wgrant> Very data set specific?
<wgrant> The inTeam thing needs to be done in a single place, and it will optimise every permission check in the project.
<lifeless> at the cost of loading a potentially unbounded dataset on every request
<lifeless> thats a risky design
<lifeless> now, if you were to load just the tps relevant to the teams you'd also preload, that addresses that scaling factor
<lifeless> but it still doesn't address the data set specific nature
<lifeless> by which I mean that the teams relevant to a given page depend on the pillar, the privacy settings, the bug settings etc
<lifeless> and we should be doing *at most* 2 queries for the same stuff today
<lifeless> so the maximum possible win is 30-40ms on a page.
<lifeless> i don't understand - given the existing caching - how you can save more than that
<lifeless> \o/Test results: bug-717394 => devel: SUCCESS
<wgrant> lifeless: BugTask:+index does several for each task and nomination.
<lifeless> so, you'll need to eager load all the /teams/ for that , right ?
<lifeless> even if we had the persons ids, we'd still trigger - in some cases - 100 queries
<wgrant> Right, so I could precache participations at the same time.
<lifeless> exactly
<wgrant> But getting all of them is *so* cheap for even our worst case right now.
<lifeless> but its not incrementally useful given that we have to do the team eager loading anyway
<wgrant> For this particular case, no.
<wgrant> But it's harder to make it specific to this case, and it is less globally useful.
<lifeless> mmm
<lifeless> I really think its better to sneak up on this
<wgrant> We can solve all of them ever with less work than solving this specific case, with the only risk being that someone joins a million teams, in which case we have bigger problems.
<lifeless> perhaps I'm misunderstanding bu that sounds completely false
<wgrant> Oh?
<lifeless> see above under having to special case this situation anyway
<lifeless> we've no particular reason to think that any of the late evaluation triggers would not need special casing to load their teams.
<lifeless> And lots of reason to think that they *will*, even with a seed-at-start-tp-list-for-the-current-user
<lifeless> its less work to write one eager-load-for-doing-a-inTeam-check helper and use it where we need it than to write a tp loader, change inTeam to be handle that extra cache, write a eager loader for groups of teams, and apply that in the same places
<wgrant> What part of my statement above was completely false?
<lifeless> that it was less work
<wgrant> If we have a global one, all we need to do is preload persons every time.
<wgrant> If we don't, we have to preload persons every time and then possibly preload participations if we think they will be necessary.
<wgrant> We *will* need at least one participation check on edge authenticated page.
<wgrant> I think that one check might as well do the trivial extra work that lets it fulfil *every* subsequent check.
<lifeless> s/edge/each/ ?
<wgrant> Uh, yes.
<wgrant> WTF
<lifeless> so
<lifeless> this isn't the way I'd do it
<lifeless> I think its a poor design
<lifeless> if you're doing the work, you get to choose
<lifeless> I've laid out my concerns
<lifeless> at this point we're clearly missing each others point, or something.
<wgrant> I think so.
<lifeless> I *will* insist that if you do it, you instrument it
 * StevenK suspects dogfood is now missing recipes
<lifeless> so that we can see the overhead clearly in the PPR or a similar system
<StevenK> Where can I see feature flags?
<wgrant> StevenK: Watch out, DF will be a bit broken.
<lifeless> the one guaranteed thing is that users will do crazy things with teams and memberships and so on
<wgrant> It needs a new DB.
<lifeless> StevenK: /+feature-rules
<StevenK> wgrant: Rargh, I need it :-(
<wgrant> StevenK: Add 'code.recipes_enabled default 0 true'
<wgrant> StevenK: It should mostly work.
<wgrant> Just bugs will do very strange things.
<wgrant> And it can't run latest db-devel.
<wgrant> I merged devel into it this morning, though.
<StevenK> Heh
<StevenK> By accident? :-)
<wgrant> No.
<StevenK> wgrant: Recipes enabled, let's see
<StevenK> wgrant: We can probably sort out a new DB for it soonish.
<lifeless> if you've still got disk space
<wgrant> lifeless: So, I agree that unconditionally loading an unbounded and excessive dataset is a bad thing.
<wgrant> lifeless: We still have 110GB free, with a DB dump and one full DB.
<lifeless> wgrant: whats \l+ show
<wgrant>  launchpad_dogfood | postgres | UTF8     | C         | C     |                       | 222 GB  | pg_default |
<lifeless> so, you're going to lose 80GB of that 110
<lifeless> the db is ~300GB now
<wgrant> I was thinking it was ~260.
<wgrant> But maybe that's old.
<lifeless> it was
<lifeless> it was 290GB during the last db deploy IIRC
<StevenK> TBH, we tend to update dogfood by scorched earthing the DB and re-importing
<StevenK> Which is a little ... brutal
<lifeless> sure
<lifeless> point is, you can't run a db and a dump now
<wgrant> Hm, I wonder if I can order \d+ by size, or if I have to go querying manually...
<StevenK> lifeless: If I'm reading what wgrant said correctly, we already have a full DB and a full DB dump on dogfood. So if we delete both, we can get another dump and import it
<wgrant> That's how it's normally done.
<wgrant> Since the dump compresses well.
<StevenK> Pity it takes mawson 40 hours to import
<lifeless> StevenK: you can get another dump, but the dump will be bigger too; if its 30GB bigger you'll run out of disk space mid import
<lifeless> well, 90% or so
<StevenK> lifeless: Er, we have 110GiB free?
<wgrant> It's not 30GB bigger, fortunately.
<StevenK> How does a 30GiB increase to both eat 110?
<wgrant> StevenK: The dump is much smaller than the full DB.
<wgrant> The DB itself has grown by 80GB.
<wgrant> (how!?)
<wgrant> The dump will also have grown by a few.
<StevenK> Either I'm missing something or lifeless is. I'm not sure who yet.
<wgrant> 110 - 80 - a bit is almost 0
<lifeless> StevenK: you have 110GB free
<lifeless> StevenK: the db is 80GB bigger, that means you'll have 30GB free for the dumps increase in size.
<wgrant>  public | branchrevision                            | table    | postgres | 26 GB      |
<StevenK> I think there's a good 12 or so GiB I could free in a hurry if i need to.
<wgrant> And then there are four revision-related indices that are about 15GB each :/
<wgrant> That's like a third of the DB.
<StevenK> launchpad@mawson:/srv/launchpad.net$ ls -lh | head -n 2 | tail -n 1
<StevenK> -rw-r--r--   1 launchpad launchpad  8.0K Aug 27 16:50 ,x
 * StevenK chuckles
<wgrant> Yeah, 100GB goes to branchrevision.
<wgrant> Awesome.
<StevenK> Can't we delete that monstrosity yet?
<wgrant> Wait.
<wgrant> That's 100GB of 222GB.
<wgrant> Not 300GB.
<wgrant> That is huge.
<StevenK> lifeless: I suspect you didn't close bugs after the deploy you requested.
<lifeless> StevenK: it was midnight. I suspect that too
<lifeless> sigh buildbot
<lifeless> StevenK: did you get any traction on why jenkins appeared more fragile than buildbot?
<wgrant> Because windmill.
<wgrant> It was fine until windmill :(
<StevenK> It's been behaving fine this week and last, mind you.
<StevenK> But Windmill makes me very sad.
<lifeless> windmill makes its mother sad
<wgrant>     "revisionnumber_branch_id_unique" UNIQUE, btree (branch, id)
<wgrant>     "revisionnumber_branch_sequence_unique" UNIQUE, btree (branch, sequence)
<lifeless> grrr
<lifeless> bug 1 timeout ><
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<wgrant> That first index is a bit off, since id is the pkey...
<lifeless> it may be used
<wgrant> lifeless: Which query?
<lifeless> alternatively we could rely on heap joins and just do single column indices
<lifeless> dunno yet
<lifeless> OOPS-1879G332
<wgrant> Here too. That's a bit upsetting.
<wgrant> Death by a couple of hundred Person queries, I guess.
<StevenK> lifeless: Thanks for the bug closures.
<lifeless> StevenK: thanks for the reminder; the list was at the top of LPS so easy to do (you could have done it :P)
<StevenK> "Meh" :-)
<wgrant> lifeless: I realise I'm blocking the QA pipeline, but I need to check with sinzui on the best way to QA the DMP.
<wgrant> And he wasn't around yesterday.
<lifeless> wgrant: we can't run it in staging
<wgrant> Exactly.
<lifeless> so, the answer is 'hope'
<StevenK> Haha
<wgrant> Mm, I guess.
 * wgrant untestables.
<lifeless> same with branch mirroring
<wgrant> We can QA that.
<wgrant> Just need to find a consenting victim and disable the rest.
<lifeless> looks new: 23 /    1  BinaryPackageBuild:+retry
<StevenK> On prod?
<wgrant> Argh, not again.
<wgrant> That could well have been during the fuckage last night.
<StevenK> Could it possibly be fallout?
<wgrant> With buildd-manager and p-u holding locks open for an hour.
 * StevenK high fives wgrant for having the same thought.
<lifeless> wheeee
<StevenK> Hmmm
<wgrant> I am really disturbed that nagios didn't notify us within a minute of the incident beginning...
<lifeless> it was probably swapping
<wgrant> And? A missing probe is a failed probe...
<lifeless> actually
<lifeless> its more nuance than that ;)
<StevenK> wgrant: As a thought: a way to handle cancelling distro builds: add a new status: BuildStatus.REFUSED or so which process-upload will silently reject binaries. And then instruct buildd-manager to shoot the buildd in the head. Thoughts?
<wgrant> StevenK: We can't cancel builds on the non-virt builders.
<wgrant> lifeless: It *is*, but it shouldn't be.
<lifeless> wgrant: talk to the netsaint designer
<wgrant> My Nagios days are hopefully over.
<StevenK> Nagios makes me sad, too.
<StevenK> I used that enough at my last job.
<wgrant> StevenK: Anyway, what were you doing to mawson?
<StevenK> wgrant: Testing https://code.launchpad.net/~stevenk/launchpad/link-recipe-on-ppa-page/+merge/50694
<StevenK> wgrant: Can has review? :-)
<wgrant> StevenK: Could you QA thumper's thing while you're there?
<StevenK> Which 'thing'?
<wgrant> r12410
<wgrant> Recipe FF changes.
<wgrant> Main thing that needs testing is r-d-b
<wgrant> OOPS syncing seems to be really slow.
<wgrant> G332 still isn't there.
<StevenK> wgrant: With the flag set, it runs, but there are no daily builds to build. I can reach into the db and set one if you wish
<wgrant> StevenK: Could remove the flag and try again?
<wgrant> It needs to run fine without it.
<wgrant> And be able to request builds.
<StevenK> wgrant: Same output as with the flag
<wgrant> StevenK: Could you staleify a recipe and try without the flag?
<lifeless> wgrant: you were interested in OOPS: https://dev.launchpad.net/LEP/OopsDisplay
<wgrant> lifeless: Excellent, looking.
<StevenK> Oh, bah, the recipe I forced to be stale has built recently
 * StevenK reaches back into the db
<wgrant> Die forster die.
<thumper> wgrant: we need an admin for that
<wgrant> thumper: Or a mawson.
<wgrant> And given the lack of admins this week, it is tempting to use mawson.
<wgrant> It's where we normally test daily builds, anyway.
<wgrant> thumper: Also, do you plan to remove the unused key from the production configs?
<wgrant> If you don't, it will make me sad.
<thumper> wgrant: yes I do...
<thumper> wgrant: it will be part of the whole release process
<wgrant> Great. Just checking. There are lots of deprecated code keys left around from years ago.
<StevenK> wgrant: Right, with flag disabled, and a recipe forced to be stale:
<StevenK> 2011-02-22 05:22:32 DEBUG   Recipe stevenk/terminator-test is stale
<StevenK> 2011-02-22 05:22:32 DEBUG    - build requested for Lucid (10.04)
<StevenK> 2011-02-22 05:22:32 INFO    Requested 1 daily builds.
<wgrant> StevenK: Great, both the UI and script work. qa-ok it is.
<wgrant> Thanks.
<StevenK> wgrant: Review? :-)
<wgrant> StevenK: Oh, right, forgot about that.
 * wgrant does it.
<wgrant> StevenK: That nasty lifeless wrote a distracting LEP.
<StevenK> Haha
<StevenK> thumper: For the description branch, I just need the database patch?
<StevenK> Oh, no, I don't
 * StevenK fixes the code too
<thumper> StevenK: primarily the flag on the storm column, and on the interface
<thumper> that should be it
<StevenK> thumper: Both done
<StevenK> thumper: In the model: description = Unicode(allow_none=False)
<StevenK> thumper: So I guess that has to change too :-)
 * thumper nods
<thumper> BTW, I'm done now
<StevenK> thumper: Awww
<StevenK> thumper: I was about to point you at the MP
<StevenK> Hm. ENOSTUB
<wallyworld> latest build failure - windmill timeout. can't see any obvious errors and the last reported test which timed out runs fine locally. i guess a force build is in order?
<lifeless> StevenK: whats up ?
<StevenK> lifeless: I was after a db number
<StevenK> But it can wait until stub does turn up
<lifeless> probably 49
<lifeless> but yeah, he should be aorund soonish
<StevenK> wallyworld: Force away
 * wallyworld shakes fist at ec2 or windmill or someone, anyone :-(
<lifeless> hmm
<lifeless> branchrevision ||     136.01 tuples/sec
<lifeless> whee, that deployment glitch really messed the stats - 4.58 99th percentile
<lifeless> flacoste: oh, the PPR needs to exclude the haproxy status page too
<lifeless> flacoste: as well as opstats :)
<wgrant> StevenK: Reviewed.
 * StevenK reads, hoping wgrant was gentle
<wgrant> Heh. Not a chance.
<wgrant> Sorry.
<wgrant> lifeless: Where's the dbr?
<lifeless> https://devpad.canonical.com/~lpqateam/dbr/last-15-mins.txt
<wgrant> Thanks.
<StevenK> wgrant: Hmmm. I think I will test how it behaves with deleted recipes
<StevenK> wgrant: Second point. I wasn't sure how to get at that view directly, so I used the route I knew worked
<wgrant> I think it may work as of this morning.
<wgrant> But it would have last week.
<wgrant> Er.
<wgrant> *wouldn't* have last week.
<StevenK> wgrant: First point: The method was a good place to start with, and did start out longer, I just didn't bump it from the model and interface into the TAL directly
<StevenK> wgrant: Do you think it's worth the bother to move it?
<wgrant> StevenK: I like to avoid bloating already bloated interfaces.
<StevenK> tal:condition="context/sourcepackagerelease/source_package_recipe_build"> makes me sad.
<wgrant> StevenK: As for getting the right view directly, canonical_url(spph, view_name='+listing-archive-extra') should do it.
<StevenK> wgrant: Thinking it about, would it address your concerns if it moved from @property to @cachedproperty ? Since I doubt it's going to actually change
<wgrant> StevenK: The speed is of no concern whatsoever.
<StevenK> According to Julian, it is
<wgrant> It is no extra queries.
<wgrant> So it's no issue.
<wgrant> You can have the method on the view if you really want it.
<wgrant> But I would just have a long tal:define and base everything on that.
<StevenK> wgrant: If you don't think it's ugly, like I wrote before, I can do that and remove it from the model and interface.
<wgrant> StevenK: SPPH.sourcepackagerelease, SPRelease.source_package_recipe_build, and SPRB.recipe are all FKs, so subsequent invocations are not expensive at all.
<wgrant> StevenK: It is ugly.
<wgrant> But it's ugly hidden away in a template.
<StevenK> And it doesn't add code to SPPH and so therefore automatically good? :-)
<wgrant> Yes.
<wgrant> It's also not *that* ugly.
<StevenK> steven@liquified:~/launchpad/lp-branches/link-recipe-on-ppa-page% grep -rc '+listing-archive-extra' lib/lp/soyuz/browser/tests/* | grep -vc ':0$'
<StevenK> 0
<StevenK> That concerns me.
<wgrant> StevenK: Probably all from Archive:+packages doctests.
<StevenK> wgrant: No matches in lib/lp/soyuz/doc or lib/lp/soyuz/tests
<wgrant> StevenK: Link-clicking, probably.
<StevenK> wgrant: You mean, like I'm doing?
<wgrant> StevenK: Yes.
<wgrant> But you are not a doctest.
<lifeless> so tal defines can (in some situations) evaluate multiple times
<lifeless> *if* the thing is expensive, thats a concern
<StevenK> Don't make me turn it into one.
<StevenK> :-P
<lifeless> julian and I didn't figure out what causes that
<wgrant> It is three FK lookups.
<wgrant> So it's trivial.
<StevenK> wgrant: Besides, I couldn't come up with a fool-proof way to get at the SPPH given a BPPH I just created.
<wgrant> StevenK: BPB.current_source_publication doesn't work?
<wgrant> I don't remember if I fixed the factory to create it in the right context.
<StevenK> Meh, I think I'll ignore BPPH completly and just create an SPPH
<wgrant> Also, you don't technically need a BPPH.
<wgrant> Right.
<StevenK> wgrant: Anyway, I shall fix it up tomorrow morning, and ping you
<wgrant> Thanks.
<wgrant> Do you have any comments on the UI stuff?
<StevenK> You want a full stop at the end of the line I ended, that's it?
<wgrant> I'd also like you to consider linking to the build instead of or as well as the recipe.
<wgrant> And maybe turn the person into a link.
<StevenK> That view is already crawling with links
<wgrant> It is.
<wgrant> But at least the build is probably useful.
<StevenK> I fail to see how, exactly
<wgrant> So I can check the build logs, mostly.
<wgrant> To see why it built strangely.
<wgrant> And to see which revs it used.
<StevenK> I already have the build, so I'm going to push back on turning the person into a link, but with suggested wording will look at linking the build in
<wgrant> You could possibly link "Built" to the recipe.
<wgrant> Er
<wgrant> To the build, not the recipe.
<wgrant> So we have "_Built_ from recipe _Launchpad trunk_ for William Grant"
<StevenK> You forgot the full stop
 * StevenK ducks
<wgrant> SILENCE
<lifeless> StevenK: why shouldn't the person be a link?
<wgrant> The icon can look slightly cluttersome.
<wgrant> But I would prefer it to be a link.
<wgrant> Even with the icon.
<StevenK> I wouldn't, and thumper agrees with me
<lifeless> -why-
<lifeless> also
<lifeless> if its not a link
<lifeless> you need to show the unique username, not the display name
<lifeless> to prevent spoofing
<StevenK> What spoofing? We're showing information from something that's already been requested?
<lifeless> so if the user doesn't matter, why show it at all ?
<StevenK> Because it may have requested by someone to build into their own PPA
<lifeless> then you should show who they are, not who they claim to be, no ?
<lifeless> StevenK: whats the /objection/ to linking
<StevenK> lifeless: As wgrant says, the icon can make the line look cluttered
<wgrant> StevenK: But it shouldn't be too bad here, since there are no other icons.
<wgrant> Unless you add the recipe icon to the start.
<wgrant> Which I might recommend.
<lifeless> so, thats a reason to discuss iconless formatters, its not a reason to not link the person - we link person everywhere else.
<wgrant> Exactly.
<lifeless> I guarantee if you don't link it, day one, someone will file a bug saying 'this is not linked and I cannot figure out who the >>> it actually is'
<wallyworld> wgrant: i've done the changes to the method names: getBuilds, getCompletedBuilds, getPendingBuilds
<wgrant> lifeless: I suppose I should ask if you have any further comments on that MP.
<wgrant> wallyworld: Thanks.
<lifeless> link ?
<wgrant> wallyworld: I'm not sure if that's better than getBuilds(completed_only, pending_only), but it's certainly better than it was before.
<wgrant> lifeless: https://code.launchpad.net/~stevenk/launchpad/link-recipe-on-ppa-page/+merge/50694
<wallyworld> lifeless: https://code.edge.launchpad.net/~wallyworld/launchpad/recipe-webservice-api/+merge/50684
<wallyworld> lifeless: sorry, wrong context
<wgrant> Sorry, I was ambiguous.
<wallyworld> wgrant: the reason the methods were changed from getBuilds(pending_only) etc is that having code like builds = getBuilds(True) blows chunks
<wallyworld> much better and clearer to explicitly name the methods in this case
<wgrant> wallyworld: But we have kwargs for that purpose.
<wgrant> lifeless: That librarian meliae analysis is intriguing... could that be ZCA internals that we are seeing there?
<lifeless> I htink os
<wallyworld> wgrant: yes, i see your point but it's 2 against one :-)
<lifeless> wallyworld: is this a democracy ?:P
<lifeless> hey
<lifeless> whats the name of the law ab..
<lifeless> ah, demeter I think
<wallyworld> lifeless: well, even dictatorships can fall :-) egypt, etc
<wallyworld> lifeless: you manage to buy some food? i read that all the Countdowns are closed
<stub> Loving my indicator icons. It is 31â with a chance of bacon.
<lifeless> wallyworld: yeah, 60 minutes in a queue though
<lifeless> wallyworld: at least we have some shinier stuff than baked beans * forever
<wallyworld> lifeless: when we had the floods here a month back, people paniced and bought out the whole supermarket - no bread, veges or anything like that left. even canned foos all gone
<lifeless> yeah
<lifeless> panic buying *after* the fact
<lifeless> lynne is getting pissed at the 4.0 after shocks
<lifeless> there are lots
<lifeless> there was a block long queue at the service station for petrol
<wgrant> The petrol station wasn't closed?
<wgrant> I heard many were.
<lifeless> we're in rangiora
<lifeless> 30K NW of the cbd
<lifeless> 40K NW of the epicentre
<wgrant> lifeless: Oh, I thought it was more like 15km.
<wgrant> Oops.
<wgrant> lifeless: Scaling-wise we are already doing all the queries.
<wgrant> To determine the uploader.
<wallyworld> i heard that petrol was being made available to police and military only
<wgrant> I'm not sure if that's prejoined or not.
<wgrant> But it's already there.
<wgrant> Oh, except those are subviews.
<wgrant> They're separate requests.
<wgrant> So this will cause three extra queries, but on-request and in a separate request.
<lifeless> wgrant: it wasn't discussed, so I didn't know
<lifeless> wgrant: its not clear to me that the source package recipe build link is dereferenced in determining the uploader
<lifeless> stub: hey
<lifeless> stub: so this branch of mine; I'd like to use the query I have, because its uglier to do what you suggested and use a big literal as a subquery with the reset in storm
<lifeless> we need to bring out two separate columns
<lifeless> so we'd need to join to it as a subquery table
<lifeless> wallyworld: I've commented anyhow
<wgrant> :(
<wgrant> TeamParticipation lies.
<lifeless> wgrant: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1879G332
<lifeless> wgrant: how so?
<wgrant> lifeless: You are in a team if you are in the owning team.
<lifeless> wgrant: the owner distinction?
<lifeless> yeah
<wgrant> Yes.
<lifeless> sinzui and I agree that that is a bug
<wgrant> But there is no participation in that case.
<wgrant> This makes it impossible to do in one query.
<wgrant> Because you may need to recurse near-infinitely.
<lifeless> you are /not/ meant to be in a team by virtue of owning it
<wgrant> Right.
<wgrant> Just have its privs.
<lifeless> no
<wgrant> Well.
<wgrant> Privs over *it*
<lifeless> right
<wgrant> Not its privs.
<lifeless> you should be able to add yourself to the team
<lifeless> or kick someone off it
<stub> lifeless: Yup. I think my query just exposed the distro id.
<wgrant> In fact.
<wgrant> It's already only partial.
<lifeless> but you don't get to do what the team permits people to do.
<lifeless> wgrant: there are some places that are confused about this
<wgrant> Since some things look up permissions from your teams, not vice-versa.
<lifeless> wgrant: the /intent/ is 'must be in, not own, to be granted stuff *by* a team'
<wgrant> I think we should look towards fixing this in the immediate future.
<lifeless> wgrant: feel free to fix in that direction
<lifeless> stub: it just exposed the spr
<StevenK> stub: Hi! When you have a tick, could you review https://code.launchpad.net/~stevenk/launchpad/db-recipe-description/+merge/50697
<lifeless> stub: we'd have to rejoin lots
<wgrant> lifeless: Can I give you a query to find affected teams?
<lifeless> stub: which is why I want to land what I have  - it works, it passes all tests
<lifeless> wgrant: sure, but I can't check on prod; ask stub to do that if you need comprehensive data.
<wgrant> lifeless: Staging's newish.
<lifeless> sure
<wgrant> I just want a rough idea.
<lifeless> stub: what do you think
<wgrant> For now.
<stub> Gah... diff still not updated on the mp. I guess we don't retry on failures :-(
<wgrant> stub: We retry a few times, I think. At least we do with normal BranchJobs.
<wgrant> 5 times for scans.
<stub> lifeless: I don't really follow. You have a query that is returning SourcePacakgeReleases and Distributions. Why does more need to be returned from the Store.find()  than what you already have?
<stub> The hack seems particularly bogus to me, and difficult to decifer with half the main query being injected in the origin(), and then relying on some magic for the DISTINCT ON to end up in the right place by inserting it in the find, and a magic column being ignored...
<stub> (And the indentation of the SQL blocks went left, but that might be a matter of taste)
<lifeless> stub: the variant you put on the MP doesn't work
<lifeless> stub: to make it work we need to rejoin all the tables again
<lifeless> stub: spr -> spph -> distroseries
<lifeless> stub: its nearly twice as long once you fix your variant to do that
<stub> So spr.distroseries != spr -> spph -> distroseries ?
<lifeless> the indenting was taste, I can fix that
<stub> Sorry - spr -> distro -> distroseries != spr -> spph -> distroseries ?
<lifeless> spr->upload_distroseries != spr->spph->distroseries
<lifeless> there is no spr->distroseries
<lifeless> package copies keep the spr and make a new spph
<lifeless> making a new series of ubuntu does the same thing
<stub> I see
<lifeless> so to get distroseriesID (and we need to use the ID, Reference columns have nasty compile errors - I
<lifeless> 've filed bugs on them in storm)
<wgrant> mawson has 244 teams with members but the owner not in the team, and 207 teams with no members at all. Odd.
<lifeless> we need to redo all the joins in the origin query
<lifeless> stub: the reason the diff didn't update was the librarian outage
<lifeless> it retried but all within the downtime
<stub> I think similar to my approach can still work - just need to reconstruct the correct query from your branch (told you it was indecipherable :)
<lifeless> stub: I did that, but its much longer
<lifeless> stub: the SQL("DISTINCT ON () 0 as ignored") is used elsehwere
<stub> Nah.... sublty different construct. Hang on...
<lifeless> wgrant: 6 seconds of python time on bug 1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> wgrant: I'll be in 2 weeks when we have 1-threaded appservers that it gets a lot better
<wgrant> :(
<lifeless> 6.7 seconds getting the first message out
<lifeless> thats either a /really/ busy slave or bogus
<lifeless> FROM BugMessage,
<lifeless>      Message
<lifeless> WHERE BugMessage.bug = 1
<lifeless>   AND BugMessage.message = Message.id
<lifeless> ORDER BY BugMessage.INDEX, Message.datecreated,
<lifeless>                            Message.id LIMIT 1
<wgrant> lifeless: https://pastebin.canonical.com/43694/
<wgrant> Could you run that on staging please?
<wgrant> Seems to be lots of OEM and DMB and loco.
<wgrant> And not too much else.
<lifeless> oh grrr
<lifeless> that query (the bugmessage one) just took 9 seconds cold on staging
<wgrant> How bad hot?
<lifeless> 23ms
<wgrant> :(
<lifeless> yeah
<wgrant> Although it is restoring.
<wgrant> And it is asuka.
<stub> So the branchscanner is too fast, because it does all the retries before we even notice critical systems are down :-) Think we need a better retry mechanism...
<lifeless> :>
<lifeless> wgrant: whats this query reporting exactly - too late ot be arsed figuring it out
<wgrant> lifeless: Unmerged teams that have members but the owner is not a member.
<lifeless> wgrant: why is this interesting?
<wgrant> lifeless: Because it's teams that will be affected when inTeam no longer considers owners.
<lifeless> 269 rows
<lifeless> oh rihgt
<lifeless> *folk to notify*
<wgrant> Right, probably.
<wgrant> We will see.
<lifeless> it will make some delegations tricky
<wgrant> Oh?
<lifeless> 'I and my delegate can do this'
<lifeless> could be done as 'I am in my delegated team'
<wgrant> You delegate it to yourself and someone else.
<lifeless> or with a sideways team that has the delegated team and the owning team as both members
<wgrant> By putting you and that someone else in a team.
<wgrant> lifeless: Can I have those rows? :)
<lifeless> meh?
<lifeless> nonowners.txt in devpad:~robertc
<wgrant> Thanks.
<lifeless> wgrant: the sort order is the problem in the query
<lifeless> just sorting on index makes the plan simple
<lifeless> and drops it to 2ms
<wgrant> lifeless: Ah. That's an easy fix.
<lifeless> I'll fix indexed_messages
<wgrant> 29 teams are among the problematic owners. Need to work out how to contact them.
<wgrant> Lots of them are Canonical/Ubuntu, though, which is nice and easy.
<thumper> StevenK: I did say the person should be a link, just a link to the person who requested the build, not necessarily the recipe owner
<lifeless> stub: so, hows it going?
<stub> lifeless: What test should I run?
<lifeless> stub: I've been assessing this by loading a bug from lp.dev
<lifeless> https://bugs.launchpad.dev/ubuntu/+source/mozilla-firefox/+bug/1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> that has (for me) a spr on 'ubuntu' and on 'debian'
<stub> So we don't have a suitable test, or we don't know which test is suitable?
<lifeless> the latter
<lifeless> actually
<lifeless> which thing is this
 * lifeless scratches head
<lifeless> series lookups
<lifeless> yes, that bug
<lifeless> stub: the mozilla-firefox(ubuntu) task should highlight with a build by mark
<lifeless> and the mozilla-firefox(debian) task should mouse over with 'no current source release ...'
<lifeless> stub: a trivial thing that may save you some time - its DistroSeries SourcePackag... - every word capitalised
<stub> We never could seem to decide if they were one word or two - English language vs. domain language.
<stub> What would save me time is faster 'make build' :-)
<stub> mailman build, css assembling....
<stub> rebooting hamster...
<StevenK> mailman out of the tree would make me so happy
<wgrant> stub: I use 'make -j3 compile' now. Not enough for the appserver, but enough for tests, and it's fairly fast.
<wgrant> More -j3 kills things :(
<stub> Ohh.. cool. forgot about -j
<StevenK> Barry thinks it's very doable with Mailman 3.
<wgrant> stub: compile skips mailman and WADL and other stuff.
<StevenK> I think he's waiting for one or two things to hit core.
<StevenK> stub: Can haz review and db patch number? :-)
<lifeless> wgrant: yeah, like byte support in python3 :)
<StevenK> The thought of moving LP to Python 3 makes me sad.
<stub> lifeless: win. pushing.
<lifeless> stub: thanks!
<lifeless> stub: did you change both ?
<stub> Ah sod... forgot to commit your work before modifying... mushed together in the diff
<stub> Is there magic to repair that 'commit changes I merged in' or 'commit changes that come from this branch over there'?
<stub> http://paste.ubuntu.com/570442/
<stub> I've just done one so far.
<stub> StevenK: Are you dropping the NOT NULL to make the UI less sucky and not forcing users to enter rubbish, or is there something more subtle here?
<StevenK> stub: I suspect the reason is most users aren't setting a description, or don't really know what to put in, so we're dropping the constraint.
<lifeless> stub: when I merge from you it should be a clear diff
<stub> Thought so. Approved.
<StevenK> stub: Can has number?
<StevenK> I daresay -99 is bad
<stub> StevenK: 2208-49-0, which will appear on the review when this ajax request submits... yay Thai ISPs.
<lifeless> stub: how would you feel about people grabbing their own numbers from the wiki page as a minor optimisation to their workflow
<stub> lifeless: Sure. As long as approved status is in the same table so we can garbage collect if we want
<StevenK> stub: Hehe, thanks
<lifeless> stub: I was thinking just let us have holes in the sequence
<stub> lifeless: patchnum || who || description || approved
<lifeless> stub: is that what you track at the moment ?
<stub> lifeless: So we either need a new baseline generated before we hit 99, or we need to hack upgrade.py and the Makefile and switch to 3 digits in the middle field.
<lifeless> stub: re http://paste.ubuntu.com/570442/
<stub> I don't track who at the moment, but we should if this is a shared pool.
<lifeless> stub: you're grabbing DistroSeries.id - we need DistroSeries.distributionID
<lifeless> oh bah
<lifeless> nvm I had the wrong one of the two queries
<stub> lifeless: ok. Want me to fix that and the other callsite on my branch to see if it is working?
<stub> The technique seems to work though and I think it is clearer.
<lifeless> I think its quite nice
<lifeless> yes please
<lifeless> stub: it will be less nice on the other one though
<lifeless> stub: that bug doesn't exercise the one you've edited
<lifeless> oh, hmm, yes it should work, little opaque at first reading
<StevenK> wgrant: Still here?
<wgrant> StevenK: Slightly.
<StevenK> wgrant: I think I have it sorted out -- trying to fit tal:define in
<lifeless> jml: WARNING: intersphinx inventory 'http://docs.python.org/objects.inv' not fetchable due to <class 'urllib2.URLError'>: <urlopen error [Errno 101] Network is unreachable>
<lifeless> jml: from carob
<lifeless> jml: the update-launchpad-built-code.sh stable  script - given you added the dep, and are in the right tz to liase with losas to fix, herewith I hand it to you ;)
<StevenK> Er, why does it need a losa? The script needs to use http_proxy
<StevenK> wgrant: <li tal:define="sprb context/sourcepackagerelease/source_package_recipe_build" tal:condition="sprb">
<lifeless> StevenK: it runs out of cron in a losa maintained account?
<lifeless> StevenK: that said, I'm not at all sure 'make' should -ever- need internet access.
<lifeless> particularly not in the deployment pipeline
<lifeless> 'fixing' it to do internet access would be bad
<StevenK> lifeless: Ah ha. Fair enough
<lifeless> hmmm
<StevenK> lifeless: Is there a special testtools matcher that ignores whitespace changes?
<lifeless> I might file a critical bug instead
<StevenK> Since I'm trying to match HTML, and TALES is helpfully inserting linebreaks
<lifeless> for string matching? DocTestMatches(,,,. doctest.IGNORE_WHITESPACE) or something like that.
<lifeless> however
<lifeless> to do html
<StevenK> Ew
<lifeless> import soupmaches
<lifeless> bah
<lifeless> jamesw's soupmatchers
<StevenK> lifeless: Thanks, I've made a note to investigate.
<lifeless> the doctest docs document the flags to doctest to control its behaviour
<lifeless> ah
<lifeless> ELLIPSIS I think is th eone
<lifeless> then you can write ... like you do in 'page tests'
<lifeless> jml - fyi - https://bugs.launchpad.net/launchpad/+bug/723003
<_mup_> Bug #723003: make is trying to connect to the internet <Launchpad itself:Triaged> < https://launchpad.net/bugs/723003 >
<wgrant> StevenK: Back.
<lifeless> stub: I think you need to commit and push again ?
<lifeless> :!bzr merge lp:~stub/launchpad/trivial
<lifeless> Nothing to do.
<stub> $ bzr push
<stub> Using saved push location: lp:~stub/launchpad/trivial
<stub> No new revisions to push.
<lifeless> ah, there it goes
<lifeless> urgh
<lifeless> you rebased onto trunk :<
<lifeless> stub: future ref for doing something like this: start your branch; do bzr pull --overwrite <my brancH>
<stub> k
<lifeless> then you're hacking directly on what I'd done, rather than merging it into trunk etc
<lifeless> avoids that issue you had with uncommitted changes from me as well
 * stub doesn't recall...
<lifeless> 20:44 < stub> Ah sod... forgot to commit your work before modifying... mushed together in the diff
<lifeless> 20:44 < stub> Is there magic to repair that 'commit changes I merged in' or 'commit changes that come from this branch over there'?
<StevenK> wgrant: Do you think my <li tal:define... is suitable enough?
<stub> Right
<wgrant> StevenK: LGTM
<StevenK> wgrant: Okay, excellent. I think I've addressed most of your concerns, except for one -- do you think I should rename the test?
<wgrant> StevenK: Yes, the test needs to be reworked, renamed and moved.
<stub> I've been avoiding pull --overwrite as I'm usually reusing branches and --overwrite nukes that old history
<StevenK> wgrant: I've done the re-work, using self.getViewBrowser(spph, ... that should be the reworking done, surely?
<wgrant> StevenK: That's great.
<wgrant> Now rename and move, and I may be happy :)
<StevenK> wgrant: Yes, I was going to ask for a suggestion.
<wgrant> StevenK: test_publishing.py
<lifeless> stub: once the branch is merged, that history becomes uninteresting
<StevenK> wgrant: Just move the entire class into there?
<wgrant> StevenK: Yes, and maybe rename it to TestSourcePublicationListingExtra or something like that.
<wgrant> But I'm not so fussed about the class name, as long as it doesn't mention ArchivePackages.
<StevenK> Yes, the test was written when I believed what needed to be changed was in ArchivePackagesView.
<wgrant> I know.
<StevenK> wgrant: At risk of being wrong: TestSourcePublicationListingRecipes
<wgrant> StevenK: It depends how specific you want to make the class.
<wgrant> Do those tests deserve their own?
<wgrant> It's possible they do.
<wgrant> But there are only two.
<StevenK> There is no other BrowserTestCase in test_publishing
<wgrant> Is there a test_publishing?
<wgrant> I don't see one.
<StevenK> Oh, I thought you meant tests, not browser/tests
<wgrant> If anything in lib/lp/*/tests/ is importing from lib/lp/*/browser, we have a bug.
<StevenK> wgrant: Fixed, it's now lib/lp/soyuz/browser/tests/test_publishing.py
<lifeless> stub: sql quick check
<lifeless> when you select from two tables and they aren't joined
<StevenK> And I understand what you were talking about, so it's also TestSourcePublicationListingExtra
<lifeless> is it an implicit cross join ?
<wgrant> StevenK: Thanks.
<lifeless> stub: aren't joined, and there are no constraints joining them
<lifeless>  https://pastebin.canonical.com/43697/
<lifeless> is what we're actually executing now
<stub> lifeless: Its an odd sort of join. the subquery only returns the (spr.id, distroseries) we are interested in.
<lifeless> I wnat to make sure a) it will do the right thing, b) I understand why it will do the right thing
<adeuring> good morning
<stub> lifeless: So without an explicit join condition between the two tables, it is a cross join. But all tuples apart from the ones with the given (spr.id, distroseries) are discarded.
<stub> lifeless: Consider SELECT spr.*, distroseries.* FROM SourcePackageRelease AS spr, DistroSeries WHERE (spr.id, DistroSeries.id) IN (1,2)
<stub> lifeless: Or the eqivalent SELECT spr.*, distroseries.* FROM SourcePackageRelease AS spr, DistroSeries WHERE spr.id = 1 and DistroSeries.id = 2;
<stub> Bah... although that first one has to be written SELECT spr.*, distroseries.* FROM SourcePackageRelease AS spr, DistroSeries WHERE (spr.id, DistroSeries.id) IN (SELECT 1,2)
<stub> https://pastebin.canonical.com/43698/
<mrevell> Hi
<lifeless> jml: ping
<lifeless> stub: thankds
<lifeless> stub: I think we can make this a little more efficient
<lifeless> stub: by using distroseries not distribution, so distribution isn't used at all
<stub> expanding the list of all-distroseries-for-this-distribution because we know that already?
<lifeless> no
<lifeless> you selected Distribution.id
<lifeless> but we don't join to distribution in the inner query
<lifeless> we use distroseries.distribution
<lifeless> which is DistroSeries.distributionID outside it
<lifeless> s/outside/ to storm/
<stub> I'm not sure how that would speed anything up - Distribution.id and DistroSeries.distribution are just integers. DistroSeries might be slightly hotter in the cache, but Distribution is smaller. Not that it matters for those tiny tables.
<lifeless> explain says 105 vs 108
<lifeless> so not worth worrying about
<lifeless> the 105 is using distroseries.distribution and a distinct on the outer (because the cross join nature permits multiple identical matching rows when we don't use something with a unique constraint)
<lifeless> stub: sending what you had to ec2
<lifeless> bigjools: what do you think of bug https://bugs.launchpad.net/launchpad/+bug/279513
<_mup_> Bug #279513: Distribution source package tooltip in bugtask table shows most recent SPPH in any series <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/279513 >
<bigjools> lifeless: reading
<bigjools> gimme 5
<lifeless> wgrant: once hot, bug one is now 8.3 seconds to render on prod
<lifeless> wgrant: which isn't too bad at all
<lifeless> wgrant: given all the other improvements we have pipelined
<bigjools> l
<bigjools> lifeless: do we even need tooltips?
<lifeless> bigjools: thats an interesting question. I'll note that we can generate them in about 30ms now
<bigjools> lifeless: I have never used that tooltip :)
<bigjools> didn't even know about it
<wgrant> bigjools: I used to use it very often.
<lifeless> bigjools: less if we move the 'distro.getCurrentSourceReleases' stuff to be done via distro.development_focus.getCurrentSourceReleases
<wgrant> lifeless: That's without the description check fix?
<lifeless> bigjools: which is how I would propose to fix that bug
<bigjools> lifeless: +1
<lifeless> wgrant: the bug one times - yes
<lifeless> bigjools: ok, cool, I'm in the area, I may polish it a little
<bigjools> turds can only be polished so much
<wgrant> Heh.
<wgrant> lifeless: So, you said you'd discussed the owner participation thing with sinzui?
<lifeless> yes
<lifeless> IIRC the idea was this:
<lifeless>  - owner would be made visually distinct from member, and we'd squash any bugs around this - e.g. inTeam
<lifeless>  - we'd make it really easy for an owner to become a member, and to leave again
<wgrant> Yep.
<wgrant> Bug #227494
<_mup_> Bug #227494: Do not special case the owner in IPerson.inTeam() <disclosure> <lp-registry> <Launchpad itself:Triaged> < https://launchpad.net/bugs/227494 >
<stub> jtv: You might be interested in reviewing https://code.launchpad.net/~stub/launchpad/garbo/+merge/50595
<jtv> looking
<jtv> stub: I'll happily review it, but mind if I do after lunch?  Visa.
<stub> jtv: Sure.
<jtv> cool
<stub> Its not an urgent branch
<wgrant> lifeless: The test fallout from this could be amusing.
<wgrant> I guess I will talk to sinzui about it tonight.
<bigjools> jml: can I trouble you to cast an eye over my twisted ftp stuff please?
<cody-somerville> Is there something going on with staging.l.n?
<lifeless> by something, you mean its down?
<cody-somerville> something like that, yea :)
<lifeless> [its being restored at the moment, use qastaging if you need to play with something]
<lifeless> its generally a better playground anyhow
<cody-somerville> hrm... whats the process if it appears a legitimate person's account has been compromised by a spammer?
<lifeless> file a question on answers.launchpad.net/launchpad
<lifeless> CHR will talk to them, get the spam removed etc
<cody-somerville> I assume I should escalate if for example said compromised account has upload permissions to the Ubuntu archive?
<lifeless> most spam is javascript automation driving their browser
<bigjools> cody-somerville: it's usually someone impersonating an email address
<bigjools> or what rob said
<lifeless> that won't get transparent access to ssh or gpg keys
<lifeless> so, I don't think you necessarily need to hit the panic button
<lifeless> if its someone that can upload, talk to them directly too - at least they will be clueful enough not to need educating
<cody-somerville> Looking at the recent actions for the user, whatever is doing it is doing it quite quickly. ;) so most likely as you suggest lifeless
<lifeless> whats the user
<bigjools> lifeless: launchpad needs a sudo feature
<lifeless> bigjools: yes
<wgrant> I filed a few bugs about that a long time ago.
<cody-somerville> https://launchpad.net/~nickj-fox
<lifeless> I've suspended it
<cody-somerville> https://answers.launchpad.net/launchpad/+question/146393
<wgrant> Huh, ~registry can't see suspended users?
<wgrant> That seems dangerous.
<cody-somerville> https://answers.launchpad.net/launchpad/+question/146394 is a duplicate
<lifeless> night
<jml> bigjools: definitely, already flagged.
<bigjools> jml: excellent, thanks.  https://code.launchpad.net/~julian-edwards/launchpad/twisted-ftp-poppy/+merge/50721
<jml> lifeless: I saw the bug. dealing with it as soon as I may.
<lifeless> jml: oh hai
<lifeless> jml: I had an action item to talk to you
<lifeless> about uhm
<lifeless> bugs I think
<jml> lifeless: I think I know what that's about.
<jml> lifeless: I have to go very soon (I can't tell you how much I want this driving stuff to be over with)
<lifeless> jml: its midnight
<lifeless> perhaps tomorrow
<jml> lifeless: but I'll be around when you are up next.
<lifeless> that would be good.
<lifeless> Hopefully I can remember stuff after sleeping :) - wouldn't remember much right now at all
<lifeless> jml: oh, also - parallel testing an doops leps
<jml> lifeless: saw them, flagged them :)
<lifeless> one is drafted [pending other team inputs, but they don't affect our direct plans]
<lifeless> the other is open in my browser with stuff
<lifeless> anyhow
<lifeless> tomorrow. Later today. Whatver ;)
<jml> :)
 * jml is off.
<soren> lifeless: You know what's funny? I woke up this morning, found a notice on my phone about the quake in Christchurch and 65 people dead. I wonder if you're ok, wander over to my laptop that was left in this channel from yesterday. I search for "quake", scroll up to find the context, and this is all I find:
<soren> 23:56 < lifeless> wow
<soren> 23:56 < lifeless> massive quake
<soren> That's just classic.
<soren> 23:56 < lifeless> internet is up
<bigjools> soren: lol
<wgrant> Come on, that was like 5 minutes after the quake :P
 * bigjools quotes :)
<soren> Someone ought to put that on a certain Quotes page if it's not already there. :)
<soren> bigjools: snap :)
<bigjools> heh
<jtv> danilos, henninge: yay!  Fixed the new-template approval timeouts in just about the simplest way imaginable.
<henninge> jtv: cool!
<danilos> jtv, marked them all as 'deleted'? :)
<jtv> Damn.
<jtv> *Second-simplest* way imaginable.
<jtv> No actually, this is simpler.
<danilos> hehe
<jtv> It was timing out because it creates all those POFiles.
<jtv> And specifically, computing stats for them.
<danilos> jtv, so you just removed updateStatistics call?
<jtv> Which doesn't make all that much sense for a freshly-created template becauseâ¦
<jtv> there's nothing in them.
<jtv> Simpler!
<danilos> jtv, cool stuff
<jtv> I just made it go "if template.messageCount() == 0: return 0"
<jtv> heh heh heh
<danilos> jtv, ah, you optimized updateStatistics :)
<jtv> Yup
<danilos> and heavily at that (at least for this special case :)
<jtv> Yup
<henninge> sounds great ;)
<jtv> saves something like half a second per POFile
<danilos> jtv, now you only need to do that for other cases updateStatistics handles, and you are done :P
<danilos> hum, "operation timed out"? :)
<StevenK> Would someone mind reviewing https://code.launchpad.net/~stevenk/launchpad/db-recipe-description/+merge/50697 ; it's tiny, and I can't see gmb.
<deryck> Morning, all.
<stub> StevenK: Looks like a candidate for a self review (I would approve, but lp doesn't want a code review in addition to db review :) )
<wgrant> stub: Mentats sadly cannot self-review.
<benji> jml: hi, I'm the RM for this round and the wiki says that I should ask you to be added to the stakeholder's list before sending emails
<jml> benji: done.
<benji> jml: thanks!
<bigjools> jml: I am off to lunch now so can't talk much but I made another branch on top of that ftp one that will veryify gpg signatures on uploaded files and send an error back to the ftp client if it's invalid.  However, I need to patch twisted :)
<bigjools> so I'll bug you about that later
<jml> bigjools: ok.
<jml> I'm still swamped beneath email.
 * bigjools hears you
<henninge2> deryck: I cant make the stand-up right now. Can we do it in 30 min?
<deryck> henninge2, that's fine
<henninge2> cool
<benji> jml: I have another RM-related query for you, the procedure says that msm (Michelle Surtees-Myers) can approve my subscription to the ubuntu-platform list, but she's out of office until the 28th; do you know of anyone else that can approve it?
<jml> benji: my guess would be any of the Ubuntu Platform managers.
<jml> benji: mdz, rickspencer3, robbiew etc.
<benji> jml: thanks, that'll help
* leonardr changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: leonardr | https://code.launchpad.net/launchpad-project/+activereviews
<jcsackett> benji: you doing RM?
<benji> jcsackett: I am.
<jcsackett> benji: cool! let me know if you have any other issues; i see elmo already told you to just email the list with reply-to rather than await moderation. :-)
<benji> jcsackett: thanks
<deryck> henninge, abentley, adeuring -- shall we try the standup now?
<adeuring> sure
<abentley> deryck, sure.
<EzraR> on the launchpad wiki it says "Running a stable production instance would be much harder than running a single-developer test instance, and we don't recommend it."
<EzraR> how much harder would it be? or would it just be the usual problems with any site?
<EzraR> my school is looking to set something up for CS students to use and wanted to use open source, i suggested LP
<jtv> danilos, henninge: good & bad news.  The pofile stats script's been failing in production because it accesses productseries.  Turns out it doesn't need toâanother easy optimization.  :)
<henninge> deryck: when shall we have our chat? voice or type?
<deryck> henninge, ah, crap.  should have stayed on, sorry.
<deryck> henninge, I'll jump back in mumble.
<henninge> deryck: https://translations.launchpad.net/ubuntu/natty/+source/gedit/+pots/gedit
<gary_poster> EzraR, harder.  LP is not designed for it, and (the vast majority of) devs are not interested in helping because we want people using the central LP site, and satellite versions actually fight our goals.
<maxb> EzraR: The main problems (AFAIK) are that parts of the code are specifically customized on the assumption that they are running on launchpad.net (like, linking off to the Ubuntu Shipit CD distribution site, for example), that the full documentation of all the cronjobs and various different component deployments are not public, and that there isn't really any such thing as a Launchpad release.
<maxb> So, as much as I love Launchpad, it's not really viable to run separate instances unless you first found a project to derive a packaged version of the software
<jml> some of these problems we are interested in solving. others less so.
<maxb> An idea lurks in my mind that it might be a viable project to fork out the registry + core code.lp.net bits as a way to give Bazaar a realistic "Here's how you can host your company's branches internally" solution
<deryck> henninge, do you know if bug 720673 has been deployed to lpnet yet?
<_mup_> Bug #720673: TranslationMessage.origin field set incorrectly during import. <qa-ok> <upstream-translations-sharing> <Launchpad itself:Fix Committed by henninge> < https://launchpad.net/bugs/720673 >
<jml> maxb: I think thumper has been working on something similarish, although not by forking registry code.
<maxb> You'd need at least the concept of projects, teams and people, wouldn't you?
<jml> teams are optional, really.
<bigjools> jml: regarding that sleep - I could not figure out how to avoid it other than to try and hack some crap in like the original poppy
<henninge> deryck: it's not. It's the next revision ready to be deployed.
<deryck> henninge, ok, great thanks!
<deryck> abentley, and your stuff that's in the deploy ready lane is all db-devel stuff, right?
<jml> bigjools: yeah. I've said how I'd do it, and I think that's pretty similar to how original poppy worked.
<henninge> deryck: I can request a deployment of revision 12410.
<jml> we already have something hacked in to make tachandler work anyway.
<abentley> deryck: no, some of it is devel stuff.
<bigjools> jml: regarding the avatar, I can't return checkers.ANONYMOUS as it makes the FTP stuff go read-only
<abentley> deryck: actually, none of it is db-devel stuff.
<jml> bigjools: oh, I see. why do you return a different string there to the long string in the tac file?
<deryck> abentley, and all qa-untestable, or there are bugs for any of them?
<abentley> deryck: qa-untestable.
<bigjools> jml: which long string?
<deryck> abentley, ok thanks
<deryck> I'll move the cards across then
<jml> @$!#!@
<jml> excuse me
<jml> it's very hard to concentrate with this Ubuntu One bug.
<jml> ok. my computer is usable again.
<EzraR> thanks for the responses maxb and gary_poster
<jml> bigjools: re "which long string", "userthatwillneverhappen"
<bigjools> jml: IIRC, it makes the ftp server go anonymous since it matches the string
<bigjools> or does it ...
<bigjools> anyway, it seemed the wrong thing to do because conceptually they are not the same user
<jml> are they not?
<jml> oh, I see.
<jml> bigjools: have you filed a bug about the anonymous => read-only thing?
<bigjools> jml: no, I don't see that it's particularly wrong
<bigjools> we just have a weird ass thing we're doing in poppy
<jml> Twisted should probably still support that.
 * jml is looking deeper at the code
<bigjools> maybe - or at least let me say whether it's a r/o session without it matching the user name
<bigjools> maybe it should be a property of the avatar
<jml> oh, also, you can pass portal and userAnonymous to the FTPFactory constructor.
<jml> bigjools: I can't find the code in Twisted that does the read-only bit
<bigjools> jml: search for userAnonymous
<bigjools> in protocols/ftp.py
<jml> I see that.
<jml> but not where it's used to prevent writes.
<bigjools> ok, one sec
<bigjools> so, 1st, notice it sets the creds to credentials.Anonymous()
<jml> but that's ok, because you get to decide in the Realm how we handle anonymous credentials
<bigjools> then there's a FTPAnonymousShell that FTPShell inherits from
<bigjools> most of the former throws AnonUserDeniedError
<jml> but you aren't using either shell, you're returning your own.
<bigjools> I know
<bigjools> I can't remember wtf it goes wrong now!
<bigjools> but it does
<bigjools> ah
<bigjools> it was to do with the Realm
<bigjools> I'm going to have to comment that line to see where it breaks
<jml> yeah, I'm running the tests now against an unmodified version of your branch, in preparation for some experimentation.
<bigjools> ok, perhaps you can do it, I'm in the middle of making your suggested changes
<jml> thanks.
 * bigjools works out why we have both lib/lp/poppy/server.py and lib/lp/poppy/daemon.py
<jml> they are both cruft now, right?
<bigjools> yeha
<bigjools> I'll be wielding a large axe at some point
<bigjools> but I need somewhere to put a function common to both ftp and sftp
<bigjools> __init__ it is :)
<jml> this ZopelessAppServerLayer had better be worth the wait.
<jml> actually, better to reboot first get these updates out of the way.
<bigjools> reading every single zcml file to just get at one is annoying
<bigjools> jml: so removing my override of the anonymous user prevents anonymous from actually logging in
<bigjools> I decided it was easier to move it out the way that bugger about with another checker
<bigjools> than*
<thumper> maxb: check out lp:sloecode
<thumper> maxb: we have it installed now, and I'll be writing something up soon
<jml> turned out ZAPL was deadlocked somehow.
<jml> in natty, I've lost my cpu usage indicator.
<jml> bigjools: test_full_source_upload fails for me in an unmodified version of that branch.
<bigjools> jml: it works here
<bigjools> maybe the 5 seconds is not enough
<jml> as does test_single_upload, both because of mode checking failures.
<bigjools> hmm
<jml> Difference: 34236 != 34228
<bigjools> yeah I fixed that
<jml>     self.assertEqual(os.stat(wanted_path).st_mode, 0102664)
<jml> oh, maybe I need to pull again.
<bigjools> hmm
<bigjools> well it was fixed before the MP went up
<jml> that's 0102674 != 0102664
 * bigjools scratches head
<jml> bigjools: what's your umask?
<bigjools> ugh
<bigjools> 0022
<jml> same as mine
<bigjools> just re-running the test
<jml> in any case, the 7 means g+s, iirc.
<bigjools> it's failing here now....
 * bigjools boggles
<jml> perhaps you were only running the ftp tests?
<jml> it fails only for sftp for me.
<bigjools> AH
<bigjools> yeah that's it
<bigjools> arse
<bigjools> it should not be setting g+x, the test is right
<jml> +x, of course. I'm a nong.
 * bigjools googles nong
<bigjools> ah, Australian slang.
<bigjools> SFTPFile.writeChunk() was wrong
<jcsackett> any chance someone could help me figure out what's going on with a traceback/test error? http://pastebin.ubuntu.com/570668/
<jcsackett> something to do with me adding a where clause and join to a bug search, but i can't figure out how it's related.
<bigjools> jcsackett: diff?
<jcsackett> bigjools: http://pastebin.ubuntu.com/570669/
<jcsackett> the test in that diff is fine; the one failing (amongst others) is the xx-front-page-search
<jcsackett> thought i had gotten everything sorted, and ec2 kindly let me know i was quite wrong.
<jml> bigjools: http://pastebin.ubuntu.com/570671/ seems to work. removes the need for the literal strings.
<bigjools> jcsackett: it's hard to tell from your diff, but I suspect you can figure it out by selectively removing diff chunks until it works
<jml> and/or writing a test that does a more focused version of whatever xx-front-page-search does.
<jcsackett> jml, bigjools: yeah, i've tried removing things. if i get rid of my product clause adding segment it works. which confuses me, as i don't see what the connection is.
<jcsackett> i will continue chipping away at it. thanks.
<jml> jcsackett: oh, you mean the commented out section?
<bigjools> jcsackett: which clause?
<bigjools> storm errors suck :/
<jcsackett> jml: that's not a commented out section, that's a very large comment block before the clause i'm talking about.
<jml> oh
<jml> my bad. green is how I colour comments in my editor :)
<bigjools> heh
 * jcsackett laughs.
<jcsackett> i could see where that would be confusing.
<bigjools> is the extra_clauses right?
<jcsackett> bigjools: i believe so. it seems to follow how some others do it.
<bigjools> you quoted it
<bigjools> ah it's literal sql rather than storm
<jcsackett> bigjools: yes; as you roll through this section an enormous query is built here and there.
<bigjools> I wonder if that's buggering up the bit (that's not in the diff) where it's adding extra clauses?
<bigjools> jcsackett: in the past I've debugged these by using pdb to step into the storm compilation
<bigjools> it would be nice if it threw decent errors
<jcsackett> bigjools: yes.
<jml> or you could see the failing sql
<bigjools> it's not getting that far jml
<jml> bigjools: yeah it is. "statement" is defined in the traceback.
<jml> bigjools: it's just not sending it to the db.
<jcsackett> yeah, i tried pdb diving, but didn't find anything terribly useful. couldn't even easily get through to the part where this exception bubbled up from. many executions between a sensible point and the failure.
<bigjools> jml: ah so it is
<bigjools> my bad, it's generating duff sql somehow
<bigjools> jcsackett: which means you can turn on LP_DEBUG_SQL_EXTRA
<bigjools> and see what it's trying to do
<jcsackett> is that an environment var to set, or what?
<gary_poster> leonardr: https://code.launchpad.net/~gary/launchpad/subscriptions_for_bug-1/+merge/50788 when you get a chance, though I plan to get a bit of lunch soon
<leonardr> gary, sure
<bigjools> jcsackett: yes
<gary_poster> thank you
<jcsackett> bigjools: thanks.
<jcsackett> i'll give that a try.
<bigjools> jcsackett: you need LP_DEBUG_SQL, the latter gives a traceback too
<jcsackett> bigjools: dig. just set to true, i assume?
<bigjools> set anything, yeah
<jtv> grrr why can't I push my branch to staging?
<jtv> exec request failed on channel 0
<jtv> bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
<jml> jtv: what does 'ssh bazaar.staging.launchpad.net' tell you?
<jtv> jml: err that it tries to ssh to bazaar.staging.launchpad.net?
<jtv> jml: No shells on this server.
<jml> jtv: hmm. so basic connectivity & authentication is fine.
<jml> jtv: anything interesting in your ~/.bzr.log?
<bigjools> jml: your patch does not work
<jml> bigjools: in what fashion?
<bigjools> you can't log in as "anonymous"
<jml> bigjools: no tests failed.
<bigjools> the tests suck then!
<jml> bigjools: seriously, are you manually testing this stuff?
<bigjools> jml: in some cases yes, because it's loads quicker than waiting for our shitty test harness to start up
<bigjools> I thought this had a test case
<bigjools> obviously not :/
<jml> maybe it's not in test_poppy
 * jml runs all the poppy tests.
<bigjools> jml: that will just use the *old* server
<jtv> jml: there's the traceback, and then the next attempt shows nothing that strikes me as suspicious.
<jml> jtv: ok. you're getting the error consistently?
<jtv> yesâand there it is again.
<jtv> I think the non-suspicious one was a different request.
<jml> bigjools: I'm confused. I think I have misunderstood the point of your branch.
<jtv> It logs the traceback.
<jml> bigjools: you aren't changing poppy to use Twisted for ftp in this branch, are you?
<bigjools> jml: the point is to add a new ftp service
<jml> but not stop the old service
<bigjools> the old one is not removed yet so we can switch over in parallel
<jml> ok.
<jml> that makes sense.
<bigjools> other benefits include using your recently-landed haproxy stuff
<bigjools> ok, what are these tests logging in with?!
<jml> not mine
<bigjools> oh flacoste said it was yours
<bigjools> some tac stuff for load balancing?
<jml> definitely not mine.
<bigjools> oooo kay :)
<jml> only coding I've done recently has been deleting old crap and adding sphinx doc gen stuff.
<jml> anyway, what are these tests logging in with, I wonder.
<leonardr> gary, review is in. i think you have some stuff you forgot to remove, but it's good otherwise
<jml> ftp://ubuntu:@localhost:%s/
<bigjools> jml: so the tests are using the bzrlib ftp transport
<bigjools> ah you beat me to it
<bigjools> sigh
<jml> I've got to drop offline to test something. brb.
<jml> back.
<bigjools> jml: so making the user anonymous in the URL has the result of the test failing
<jml> bigjools: yay
<jml> bigjools: I guess we should have a couple of tests that try user:pass, anonymous and ubuntu w/out pass.
<bigjools> yes
<bigjools> poking my override back makes it pass
<jml> that the override is needed to make them pass means there's a bug somewhere.
 * jml keeps digging
<bigjools> no doubt
<bigjools> sorry I can't remember where was causing it :/
<bigjools> I did dig deep to find it
<bigjools> but I think it was because there's a separate checker for anonymous access
<jml> there aren't any checkers in t/protocols/ftp.py
 * jml really does prefer old standard_test_template
<jml> bigjools: got it.
<bigjools> jml: \o/
<jml> bigjools: it was my bad. credentials.username doesn't exist for Anonymous().
<bigjools> ok
<jml> bigjools: and ftp.py reacts to *any* failure in the realm as an auth failure
<bigjools> ha
<jml> wait, no I don't.
<jml> but that is definitely *a* bug.
<jml> adding direct tests for some of this stuff as I go.
<bigjools> jml: I am using credentials.username and it's fine
<jml> bigjools: that's because you aren't logging in with the username that ftp has assigned to the anonymous user.
<bigjools> ok
<bigjools> yes
<bigjools> meh
<jml> got it for real.
<bigjools> just pushed up fixed tests
<jml> http://pastebin.ubuntu.com/570671/plain/
<jml> bigjools: errr, wrong url.
<jml> bigjools: http://pastebin.ubuntu.com/570708/
<jml> bigjools: the actual, actual, actual issue was that the checker we'd defined was not being used for anonymous logins, because it hadn't declared that it could handle such things.
<bigjools> \o/
<bigjools> jml: this stuff is too complicated
<jml> bigjools: it's certainly not easy to understand.
<bigjools> so I see from your comments in #twisted :)
<jml> bigjools: but I don't think that's quite the same as "too complicated". I'm not sure what I'd remove.
<bigjools> if it's not easy to understand it must be too complicated
<bigjools> I suspect a lot of this is down to snow blindness
<jml> cred is actually an elegant design, and surprisingly simple. it's just hard to get one's head around at first.
<jml> we really would have been helped here by unit tests instead of integration tests.
<jml> (won't say it again, promise)
<bigjools> jml: ok so I've fixed everything in the review apart from the strports stuff.  I really, really hate seeing "tcp:NNNN" in the config and I want to change that to just NNN but it means fixing production configs in parallel
<flacoste> bigjools: i meant spiv
<bigjools> so I will do it all as a simpler change later
<bigjools> flacoste: heh, ok
<jml> bigjools: ok. sounds good.
<jml> bigjools: you've also applied the patch in the pastebin above?
<bigjools> jml: the extra credentials?
<bigjools> yes, if so
<jml> bigjools: yeah, extra creds, return credentials.ANONYMOUS instead of "poppy", don't bother setting factory.userAnonymous
<bigjools> all done
<bigjools> now I need to so the sleep removal thing
<bigjools> but, after sleeping myself
<bigjools> night all
<bigjools> thanks for the help jml
<jml> bigjools: my pleasure.
<bigjools> twisted is starting to make me think of regexes now :)
<jml> the ftp server isn't its strongest point.
<jml> spiv has spent the last ten years apologizing for it.
<lifeless> moin
<lifeless> jml: hi
<jml> lifeless: hi
<jml> lifeless: I haven't looked at your branch yet. it's third item in my queue, counting the thing I'm doing now.
<lifeless> thats fine; its for your patch after all :)
<lifeless> jml: how much longer are you around for this evening ?
<jml> lifeless: probably at least another half hour to an hour
<lifeless> jml: so, I should be ready for that call in about 10/15 if you can still spare the time
<jml> lifeless: that would be ideal
<jml> lifeless: hi
<lifeless> jml: ready
<jml> lifeless: ok
<leonardr> i have a yui question, on the off chance anyone here knows anything about yui
<leonardr> i'm trying to figure out why an event fired from one source file is not received by the handler in another file. is it possible they are running in different sandboxes?
<thumper> leonardr: hi
<leonardr> thumper:hey
<thumper> leonardr: I've just asked that on #yui
<leonardr> i've been poking at your branch for a while
<leonardr> oh, great
<leonardr> are you ok w/r/t the earthquake?
<thumper> got my answer
<thumper> yes, all fine here
<thumper> no damage in Dunedin
<leonardr> thumper: i may be doing something wrong, but changing Y.on to Y.Global.on doesn't change anything for me
<thumper> leonardr: we need to use Y.publish(...)
<thumper> leonardr: I'm just wondering how to pass the args through
<leonardr> oh, we need to do the whole publisher thing
<leonardr> thumper: maybe explicitly create the event object and assign the arguments to its attributes?
<thumper> I think we need a fancy context
<thumper> leonardr: aye
<thumper> I've got a call with flacoste now, I'll continue looking after that
<leonardr> thumper: ok, i'll hack at it a bit
<thumper> I think I may have it...
 * thumper makes run
<jtv> mbarnett: no luck at all reproducing the permissions problem with that stats scriptâ¦ did it finish yet by the way?
<thumper> hmm...
<thumper> :(
<thumper> not working yet
<jtv> Anyone know why I can't push my branch to staging?  "exec request failed on channel 0"
<leonardr> thumper: push what you have?
<thumper> leonardr: done
<leonardr> k
<lifeless> gary_poster: hi
<gary_poster> lifeless, hey
<lifeless> gary_poster: I'd like to catch up a little, if you have time
<gary_poster> lifeless, I have 20 minutes today, right now, and then I'm booked till EoD.  Tomorrow is good after team lead call (and a bit of lunch).  Which would you prefer?
<lifeless> gary_poster: right now is best for me
<gary_poster> ok
<gary_poster> lifeless, I'm on mumble and draggable to where you'd like to talk (or tell me and I'll try to get there)
<lifeless> I'll go grab a headset then
<gary_poster> cool
<gary_poster> I can
<lifeless> may we switch to pots?
<lifeless> internet voice fail - another quake going through at the moment
<lifeless> gary_poster:
<lifeless> gary_poster: ^
<gary_poster> lifeless, skype?
<gary_poster> lifeless, I can heat you fine, fwiw
<gary_poster> hear
<lifeless> gary_poster: I could hear you -much- better with skype, FWIW
<jtv> leonardr: got another review for you if you're available!  https://code.launchpad.net/~jtv/launchpad/bug-723168/+merge/50808
<leonardr> jtv, ok, it'll be a minute
<jtv> thanks
<deryck> Later on, everyone.
<jtv> lifeless: any idea why I might not be able to push my branch to staging?  "exec request failed on channel 0"
<mwhudson> jtv: sounds broken server side
<jtv> no kidding :)
<lifeless> jtv: I'd guess that the codehosting server is configured for the fork server but the fork server isn't
<lifeless> (serving that is)
<jtv> I'll pretend I understood that
<mwhudson> without a losa around i guess that's not going to be possible to debug :/
<jtv> :(
<jtv> We had mbarnett earlierâ¦
<wgrant> sinzui: Hm, when did you ask me to do that?
<sinzui> wgrant: ask what?
<leonardr> jtv, r=me
<jtv> thanks!
<StevenK> leonardr: Hi! What's the next step with my lazr.restful branch?
<leonardr> StevenK: it's approved, so you can merge it or i can do it for you
<wgrant> sinzui: Me to look at what needs fixing in the person picker.
<StevenK> leonardr: I've not done that before, but happy to merge it.
<leonardr> StevenK: it's easy. just do a bzr co of trunk, then bzr commit with [r=leonardr]
<leonardr> trunk being lp:lazr.restful
<sinzui> wgrant:  oh. 1.5 weeks ago when we discussed goal. I think I cited the person-picker as something you should propose because of your experience with the issues
<StevenK> leonardr: Ah. What about a release so I can drop about 90 lines from the LP tree? :-)
<leonardr> StevenK: when you commit, bump the version number in src/lazr/restful/version.txt to 0.17.1, and add a summary in the NEWS.txt. then ping me and i'll run my release script
<StevenK> leonardr: So merge my branch, adding [r=...] at the front, and then commit the version and summary information? Sounds fine
<leonardr> y
<StevenK> leonardr: Or I can change my move-test branch to also change the version, which would you prefer?
<sinzui> hand up who has a IE that they can point at a dev instance
 * sinzui expects no hands
<leonardr> StevenK: change the move-test branch and merge the whole thing at once
<StevenK> sinzui: I think you also need to ask "and will admit to it"
<sinzui> StevenK: I really do not expect anyone to have windows. I don't
<StevenK> leonardr: 0.16.1 -> 0.17.1, right?
<leonardr> StevenK: we are now at 0.17.0, but launchpad is probably on 0.16.1
<wgrant> sinzui: There is a machine here that I can boot into Windows if required.
<wgrant> sinzui: I believe it has IE8.
<sinzui> wgrant: the exact version I need.
<wgrant> sinzui: What needs testing?
<StevenK> leonardr: Hm, should I be concerned trunk's NEWS file has no mention of 0.17.0? :-)
<leonardr> StevenK: what does bzr info say about your copy of trunk?
<leonardr> you should be at revision 174
<StevenK> leonardr: Yes, I was looking at my branch, not trunk :-(
<sinzui> wgrant: I think I will attempt to create a simple page and script to exercise bug 500015. I hope it is a one line fix
<_mup_> Bug #500015: IE js error in filebug <dhrb> <javascript> <lp-bugs> <oem-services> <story-ajaxify-dupe-finder> <Launchpad itself:Triaged> < https://launchpad.net/bugs/500015 >
* leonardr changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
* StevenK changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<StevenK> leonardr: Done. Do you need me to tag, or does your release script do that?
<leonardr> StevenK: my script does that
<wgrant> sinzui: Hm? Is it not easy enough to patch the page directly?
<sinzui> wgrant: I thought so. I saved the page, but I need to do some hacking because the function is not wired to the button
<leonardr> StevenK: all right, http://launchpad.net/lazr.restful/trunk/0.17.1/+download/lazr.restful-0.17.1.tar.gz is ready to go into launchpad dependencies
<wgrant> sinzui: OK, I have the machine booted into Windows and explorer stopped hanging after I killed it 5 times.
<wgrant> sinzui: It seems to connect to my LP instance OK.
<StevenK> Haha
<StevenK> leonardr: Thanks! If you need to go, I can work with someone else to update the download-cache and so on. Or I can keep asking you. :-)
<wgrant> Hmm, that's not good.
<leonardr> StevenK: copy it into lp-sourcedeps/download-cache/dist and commit it before your land your branch
<leonardr> that's about it
<leonardr> you'll need to update versions.cfg in launchpad
<lifeless> flacoste: hi
<flacoste> hi lifeless
<lifeless> flacoste: whats the url for the haproxy status page?
<lifeless> flacoste: the ppr needs to exclude that as well as +opstats
<flacoste> lifeless: it does
<flacoste> already
<flacoste> :-)
<flacoste> part of my initial patch
<lifeless> hmm
<flacoste> but for the record, the URL is /+haproxy
<lifeless> launchpad except opstats
<lifeless> (?<!\+opstats)$	
<lifeless> doesn't look like it :)
<flacoste> ???
<wgrant> sinzui: Clicking Next on +filebug seems to just do refresh the page.
<lifeless> https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-categories.html
<flacoste> lifeless: All Launchpad except operational pages=(?<!\+opstats|\+haproxy)$
<StevenK> leonardr: Yeah, that part I worked out, thanks.
<lifeless> flacoste: ?!
<sinzui> wgrant: excellent. The bug lives
<flacoste> that's from from utilities/page-performance-report.ini
<flacoste> hmm
<lifeless> flacoste: maybe it needs to be deployed ?
<flacoste> are we using a custom configuration on devpad...
 * flacoste looks
<lifeless> flacoste: should bug 451390 be 'escalated' ?
<_mup_> Bug #451390: limited upload rights no longer give series nomination accept/decline rights <dhrb> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/451390 >
<wgrant> Ah, there's the error.
<flacoste> lifeless: nobody asked me for it<
<sinzui> wgrant: Y.one('#bug_reporting_form') ?
<flacoste> lifeless, the lpqateam launchpad tree is outdated...
 * flacoste updates
<lifeless> flacoste: we should get that done by nodowntime deploys
<wgrant> sinzui: No, it's calling event.stopPropagation()
<wgrant> sinzui: Which isn't supported by IE, AIUI.
<sinzui> \o/
<wgrant> Let's try cancelBubble instead.
<wgrant> sinzui: Oh, no, this._event is undefined.
<wgrant> How do I unminify this JS...
<sinzui> I do not know.
<spiv> jml: actually, the ftp server in Twisted isn't my fault, I wrote the ftp client
<wgrant> Grar.
<wgrant> The call stack is a dozen "JScript anonymous function"s.
<wgrant> Not helpful.
<flacoste> wgrant: see Deryck's email to the list
<flacoste> wgrant: there is an env var you can set
<flacoste> to have it build unminified
<wgrant> flacoste: Ah, missed that one. Perfect.
<wgrant> Thanks.
<sinzui> wgrant: e.cancelBubble = true; is what I would have tried
<flacoste> $ make clean_js
<flacoste> $ make jsbuild JSFLAGS='--filetype=raw'
<flacoste> wgrant: ^^^
<flacoste> from 'State of lazr-js/YUI in Launchpad' email
<flacoste> iirc, that only works for our JS
<flacoste> not YUI itself
<StevenK> wgrant: I've updated the screenshot for link-recipe-on-ppa-page
<flacoste> bug 720097 tracks making it work for YUI
<_mup_> Bug #720097: Support jsbuild's JSFLAGS filetype raw for YUI deps <javascript> <Launchpad itself:Triaged> < https://launchpad.net/bugs/720097 >
<wgrant> StevenK: Link?
<StevenK> wgrant: http://people.canonical.com/~stevenk/built-by-recipe.png
<wgrant> StevenK: Could you publish that?
<wgrant> So we can see how it looks alongside the timestamps.
<sinzui> wgrant: mumble?
 * StevenK tries to remember how to run the PPA publisher on mawson
<wgrant> StevenK: ./scripts/publish-distro --ppa -vvomgbbqvvvvwefwepfiojwegfwrigwrgerg
<wgrant> +.py
<flacoste> for real???!???
<StevenK> flacoste: No, wgrant is kidding :-)
<sinzui> what that is too close to Ekke Ekke Ekke Ekke Ptang Zoo Boing Zow Zing!
<flacoste> lol
<flacoste> it still goes on the quotes page...
<StevenK> sinzui: When you're undistracted, would you be able to UI review a branch of mine. wgrant has context, too.
<sinzui> StevenK: thanks I will do it shortly
<sinzui> wgrant: still no sound
<wgrant> Hm.
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> Did you hear me say hi earlier?
<sinzui> I wish mumble let me read lips
<sinzui> oh
<huwshimi> sinzui: Lost internet for a second.
<flacoste> lifeless: fixed
<StevenK> wgrant: Running the publisher had no effect. :-(
<lifeless> flacoste: \o/
<wgrant> StevenK: It's not a P3A?
<lifeless> flacoste: will it be updated by deploys?
<flacoste> i also found out that i never commited the change i made to page-performance-daily-report.sh
<flacoste> that's not corrected
<StevenK> wgrant: No, it's public
<flacoste> lifeless: nope
<lifeless> flacoste: RT that ?
<flacoste> lifeless: yes
<flacoste> please
<wgrant> StevenK: The publisher didn't crash? The publish flag isn't off?
<StevenK> wgrant: It published the source, but not i386
<wgrant> StevenK: I don't care about i386
<wgrant> StevenK: But you need to run p-a to get binaries published.
<StevenK> wgrant: Ah, if all you care about is source, then: https://code.dogfood.launchpad.net/~stevenk/+archive/daily/+packages
<wgrant> StevenK: :(
<StevenK> wgrant: Hm?
<wgrant> StevenK: It looks bad next to the timestamps.
<wgrant> It needs to be moved or rewritten.
<wgrant> Possibly under Builds.
<wgrant> sinzui: What do you think?
<lifeless> timestamps ?
<lifeless> looks like it should replace the 'Publishing details' text
<wgrant> lifeless: The rest of that section is timestamps.
<wgrant> So it looks out of place.
<sinzui> sorry, I am still catching up
<lifeless> hmm
<lifeless> Bug 719249 needs qa
<_mup_> Bug #719249: The new direct subscription overlay takes too long to load <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gmb> < https://launchpad.net/bugs/719249 >
<wgrant> It does.
<lifeless> I don't think anyone in yellow is still here, and gmb is out for the week
<wgrant> I'm tempted to not-ok it for being insane.
<wgrant> But I think it's actually ok.
<lifeless> wgrant: its ok to go out for a few days
<lifeless> it will get rolled back soon
<lifeless> Bug 722568 also needs love
<_mup_> Bug #722568: Empty TranslationMessages confuse stats <qa-needstesting> <Launchpad itself:Fix Committed by jtv> < https://launchpad.net/bugs/722568 >
<lifeless> and Bug 720826
<_mup_> Bug #720826: Add subscription description header for bug notifications <qa-needstesting> <story-better-bug-notification> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/720826 >
<lifeless> and we can deploy everything
<wgrant> Oh good.
<wgrant> Even loading BugTask:+index in IE8 causes JS errors.
<wallyworld_> wgrant: you admit to using IE? :-P
<wgrant> wallyworld_: I'm attempting to debug +filebug in IE.
<sinzui> wgrant: I see that pofile.js is using e.halt()
<wgrant> I think our YUI setup is entirely broken in IE :/
 * wallyworld_ hooks a big fisk
<wgrant> The event facade is not being loaded.
<wallyworld_> fish
<wallyworld_> wgrant: i was just messing with you :-P
<wgrant> Ah
<lifeless> wgrant: are you in the malone beta team?
<wgrant> lifeless: The alpha team? No.
<lifeless> I'll join on qas and qa then
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/less-lazr-security/+merge/50824 should make you happy
<lifeless> oh
<lifeless> its moderated
<lifeless> flacoste: hey
<wgrant> lifeless: Yes :(
<lifeless> flacoste: can you check that Bug 719249 hasn't broken the subscription form on qastaging ?
<_mup_> Bug #719249: The new direct subscription overlay takes too long to load <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gmb> < https://launchpad.net/bugs/719249 >
<wgrant> StevenK: Yay. But what's in lazr.restful 0.17?
<StevenK> wgrant: Something that thumper was after. I was going to wait until that lands, which should be soon.
<flacoste> lifeless: sorry, i need to run
<lifeless> flacoste: no worries
<lifeless> mmm
<lifeless> cody-somerville: ^
<lifeless> these teams should be owned by ~launchpad
<cody-somerville> hmm?
<wgrant> lifeless: He's not an admin.
<lifeless> doesn't need to be
<wgrant> Oh.
<wgrant> True.
<lifeless> cody-somerville: would love it if you could help us with qa
<cody-somerville> Sure. What do you need me to do?
<sinzui> wgrant: StevenK: I am looking at https://code.dogfood.launchpad.net/~stevenk/+archive/daily/+packages . I got confused about what I should be looking at. Instead I just looked. found the "built by" line, then realised I do not know why there are two black bold chunks of text.
<lifeless> cody-somerville: on qastaging
<lifeless> visit a bug
<lifeless> and click on the advanced subscroption thingy that you should see as a member of malone-aplha
<lifeless> tell me that it works
<lifeless> that is that it loads ok
<lifeless> you don't need to actually change your subscription
<wgrant> sinzui: The first one's boldness is not useful. The second one may be.
<sinzui> StevenK: I suspect the chunks of text are reused, and the bold "Published" and "Pending publication" mix badly on this page
<StevenK> I can stop it being pending :-)
<cody-somerville> Does qastaging.launchpad.net/bugs/##### not work on qastaging?
<sinzui> StevenK: I think the "Built by" is not important next to the "Published". I think both are is equal value
<wgrant> cody-somerville: It works for me.
<lifeless> cody-somerville: it should
<lifeless> http://qastaging.launchpad.net/bugs/23 just worked for me
<_mup_> Bug #23: baz redo should use merge3 for conflicts like most other commands do. <Baz (deprecated):Won't Fix> < https://launchpad.net/bugs/23 >
<cody-somerville> hmm... maybe the bug I'm trying to access is too new
<lifeless> cody-somerville: doesn't need to be an interesting bug - use 23 ;)
<sinzui> StevenK: the <h3>s are victims of the stylesheet. I think I can fix that this week
<wgrant> sinzui: I sort of want to remove all bold text from Launchpad, except maybe the occasional "this will kill something" warning.
<wallyworld_> thumper: i like the stuff you did to the spr browser tests. removing the (fragile) huge text pattern matching strings
<wgrant> It looks terrible.
<cody-somerville> lifeless, If I click 'Subscribe' I get a modal dialogue with three options.
<lifeless> great, thanks
<cody-somerville> Its too bad that flexibility isn't available when subscribing someone else.
<wgrant> cody-somerville: Yet.
<cody-somerville> :]
<wgrant> lp.registry.browser.person is too big.
<wgrant> lifeless: I guess jtv was trying to QA that empty messages thing last night when he ran into the permissions issue.
<lifeless> indeed
<StevenK> lifeless: Are you aware if I can mix soupmatcher tags with text?
<lifeless> I don't know
<wgrant> sinzui: I am thinking about UI for the ownership change.
 * sinzui look
<wgrant> https://launchpad.net/~wgrant/+participation is going to be particularly problematic.
<wgrant> We may need to split ownership into a separate table.
<sinzui> Yes. That may be. I considered that too recently.
<wgrant> It already lies.
<sinzui> wgrant: also. I want to know what teams I own. Lp does not tell me
<sinzui> I own more than I remember.
<wgrant> Right.
<thumper> wallyworld_: :-)
<sinzui> wgrant: I was thinking I could avoid the owner column by showing "owner/admin" in membership. I want to list owned teams separately on a related page. I do not think about the teams I only own as something I participate in.
<wallyworld_> thumper: that big blob of text was the real reason my branch failed, not the db crap i talked about this morning. i got mixed up with 2 separate issues
<thumper> wallyworld_: ah...
<thumper> wallyworld_: should be easier now
<wallyworld_> but your changes mean that it wouldn't have failed if they had been done first
<wallyworld_> so more robust moving forward :-)
<wgrant> sinzui: Hmm. I don't like the idea of having them on separate pages, really.
<cody-somerville> the less pages the better IMHO :-)
<sinzui> wgrant: Adding it to a second table might work. Since non-member-owner cannot will not be able to participate (change the project the team owns for example), I think mixing these teams in a listing will lead to confusion.
<wgrant> sinzui: That was my intention.
<wgrant> Placing them in the same table would be very awkward.
<cody-somerville> wait a second
<sinzui> wgrant: consider the team that losas own. No one should think they participate in rosetta-admins or ubuntu-mirror-admins
<cody-somerville> Are you saying that you're changing it so that being the owner of a team will no longer give you the permissions associated with said team?
<sinzui> yes
<sinzui> we are
<wgrant> cody-somerville: It already gives you only a partial set.
<cody-somerville> OEM Services *specifically* relies on this.
<wgrant> You can add the owning team to your teams.
<cody-somerville> Its intentional that they are not a member
<sinzui> you can be a member, or you can just own, delegating the responsibilities to the members
<wgrant> cody-somerville: Why?
<wgrant> If they want the privileges of the team, they can be a member of the team...
<cody-somerville> Membership causes bug mail
<wgrant> Sigh,.
<lifeless> cody-somerville: what benefit do you get by having the owner not listed as a member but getting the benefits of being a member
<sinzui> cody-somerville: then we want to fix the bug mail case
<wgrant> The bug mail case is actually just about fixed.
<cody-somerville> lifeless, We have a pmteam that is able to edit and maintain all of our projects without actually being members of the individual teams
<wgrant> Mute subscriptions are nearly here.
<lifeless> cody-somerville: this is a pretty big issue
<lifeless> cody-somerville: it causes lots of confusion at the moment, and reports on 'who can do X' are not possible until the ambiguity is resolved.
<cody-somerville> I don't disagree that there is confusion. Its a huge issue for us. However, we intentionally exploit the current behavior for our benevolent needs so changing it without coordination would cause major disruption.
<wgrant> We are not changing it wihtout coordination.
<wgrant> We will communicate with all affected owners.
<sinzui> cody-somerville: the pm case is a part of the project group non-sense that need fixing, probably with an axe
<wgrant> sinzui: A Green axe, I suppose.
<cody-somerville> wgrant, I had no doubt. I'm just doing a bit of that communicating right now to highlight that this in particular would be problematic for us ;)
#launchpad-dev 2011-02-23
<wgrant> So, I think that non-ML team subscriptions are a bug.
<wgrant_> Hah.
<sinzui> wgrant: what to you mean by "non-ML team subscriptions"? mailing list subscriptions for teams that one had a mailing list?
<thumper> w00t
 * thumper does a little dance
<thumper> now to just edit the other 200 odd references to LP.client methods in the LP tree
<StevenK> wgrant: I saw you discussing my traversal change with Julian. Revert it or keep it?
 * StevenK nails wgrant to the channel
<StevenK> sinzui: If you look at lib/lp/soyuz/templates/packagepublishing-details.pt, a lot of the status information is in <strong> tags
 * thumper does another little dance
<sinzui> StevenK: yes. I think that is defeating the purpose of the list.
<StevenK> I know that the statuses have been strong for as long as I remember
<StevenK> So I'm little nervous changing them
<wgrant> They have been strong forever.
<wgrant> So there is no reason to keep them.
<wgrant> cody-somerville: Is there a reason you can't set a /dev/null email address on ~pmteam?
<wgrant> Yellow will be sorting out subscription muting soon, I believe.
<StevenK> wgrant: So I should remove all of the strong?
<wgrant> And we will be sorting out project permissions a little less soon.
<wgrant> StevenK: Except maybe the "pending publication" bit next to builds, yes.
<wgrant> StevenK: Someone decided in 2005 that it should be strong, and nobody ever looked at it again.
<cody-somerville> wgrant, Is there a bug or blueprint for this change regarding the owner no longer being endowed permissions associated with teams owned?
<StevenK> wgrant: I just have this feeling that bigjools will like that change even less
<wgrant> cody-somerville: Bug #227494
<_mup_> Bug #227494: Do not special case the owner in IPerson.inTeam() <disclosure> <lp-registry> <Launchpad itself:Triaged> < https://launchpad.net/bugs/227494 >
<lifeless> is bug 181401 fixed?
<_mup_> Bug #181401: push to lp:project or lp:project/series should to set the development focus and/or create the series <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/181401 >
<wgrant> lifeless: I've seen breakage which suggests that it is.
<lifeless> wgrant: sinzui: also bug 191870
<_mup_> Bug #191870: Participation in a team as the owner is non-obvious <lp-registry> <registry> <Launchpad itself:Triaged> < https://launchpad.net/bugs/191870 >
<lifeless> bug 206058
<_mup_> Bug #206058: Team Administrators cannot change their membership expiration date <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/206058 >
<lifeless> bug 409241
<_mup_> Bug #409241: Team member listings show "Owner" as "Administrator" <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/409241 >
<lifeless> bug 539903
<_mup_> Bug #539903: Team Owner seems to show ownership, should show leadership - or options <lp-registry> <needs-decision> <team-roles> <teams> <Launchpad itself:Triaged> <loco-directory:Fix Released by chrisjohnston> < https://launchpad.net/bugs/539903 >
<wgrant> I had all of those open somewhere, but tooooo many tabs :/
<wgrant> Bug #409241 should be fixed by allowing administrators to bless people as administrators, I think.
<_mup_> Bug #409241: Team member listings show "Owner" as "Administrator" <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/409241 >
<lifeless> also https://bugs.launchpad.net/launchpad/+bug/669643/comments/2 was relevant to this discussion
<_mup_> Bug #669643: Avoid extra queries by making inTeam() accept an ID <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/669643 >
<wgrant> Admins separate is a good idea, but requires reworking TeamMembership (in a good way... but it's a bit of work)
<StevenK> wgrant: I'd appreciate a second pair of eyes: http://pastebin.ubuntu.com/570855/
<StevenK> wallyworld_: BAH! Why do you have four things in QA-Landing and QA-In process?
<wallyworld_> StevenK: some of the qa in progress things can be moved across. the qa was done this morning and/or last night. i haven't got it moving the cards yet
<wgrant> StevenK: Well, that appears to be a lie.
<StevenK> wgrant: Exactly. :-(
<wallyworld_> StevenK: the landing stuff is waiting on ec2 and/or buildbot
<StevenK> wallyworld_: There's nine cards there, and I have a card in Coding-Review that I'd to move.
 * wallyworld_ looks
<StevenK> Sorry, eight
<StevenK> wgrant: Even 'print expected_contents in contents
<StevenK> ' gives False :-(
<wallyworld_> StevenK: i've moved 2 cards from QA to deployment ready. the board was only a few hours out of date.
<StevenK> wallyworld_: Thanks
<wallyworld_> StevenK: np. sorry for the delay. the 2 cards in progress are being done in the one branch. if only ec2/bb didn;t take 8 hours end-end stuff wouldn't have to langusih in the Landing section for so long
<StevenK> wallyworld_: It would be okay if you didn't have four things going at once. :-)
<wallyworld_> StevenK: what i do is, i push stuff to ec2, start a new branch, get that done, ask for a review, and while waiting for that, start a new branch etc. so it's all pipelined...
<wallyworld_> StevenK: and if ec2 takes a while and/or you're waiting for a particular person to review of whatever, you may as well start a new branch
<StevenK> It's been ending up that I've been tossing branches at ec2 and then EODing
<lifeless> StevenK: your mp commit message is naffed
<StevenK> lifeless: Which one?
<lifeless> just got the mail on the chnage
<lifeless> is now just [r=][bug=]
<lifeless> deleted the message already
<StevenK> lp-land, you fail
<StevenK> Commit message [[r=stub][bug=713518] SourcePackageRecipe.description is now optional.]
<StevenK> However, we're in testfix anyway
<StevenK> test_404 and test_oldurl failed again
<wgrant> I still have not managed to reproduce those locally.
<StevenK> wgrant: Can you have another look at https://code.launchpad.net/~stevenk/launchpad/link-recipe-on-ppa-page/+merge/50694 ?
<StevenK> wgrant: I've addressed all of your concerns, removed the strong like sinzui suggested
<wgrant> StevenK: Looking.
<StevenK> PQM, you fail.
<wgrant> What has it done now?
<StevenK> wgrant: If we're in testfix the body of the mail says:
<StevenK> Command failed!
<StevenK> running 0 tests...
<wgrant> Yes,
<wgrant> but the attachments say more.
<StevenK> And then you need to read the two gzip'd attachments
<StevenK> And Thunderbird hates gzip
<StevenK> :-(
<wgrant> StevenK: Could you put some linebreaks in that TAL?
<wgrant> Also that span is redundant.
<wgrant> Put tal:content="sprb/recipe/name" on the a.
<wgrant> Same with the other one.
<StevenK> wgrant: <a tal:attributes="href sprb/recipe/fmt:url" tal:content="sprb/recipe/name" /></a> ?
<wgrant> StevenK: That. You may also be able to self-close the a.
<StevenK> I already have />, so maybe that is sufficent.
<StevenK> Vim disagrees.
<wgrant> Ah, so you do.
<wgrant> You can't have both that and the </a>, obviously.
<StevenK> wgrant: Linebreaking the tal makes me sad, since it makes my naive string matching break.
<wgrant> StevenK: Whitespace normalisation is cool.
<StevenK> Which I can't figure out how to do with a string
<wgrant> Matchers don't have such a facility?
<StevenK> I have no idea
<lifeless> StevenK: DocTestMatcher(...doctest.ELLIPSIS)
<lifeless> and in your reference, use '...'
<lifeless> -or-
<lifeless> use soupmatchers
<wgrant> Doctests have whitespace normalisation, not just ellipsis... does DocTestMatcher not do that?
<lifeless> it doesn't set any flags by defailt
<lifeless> just set the flag you want
<wgrant> Ah, handy.
<wgrant> cody-somerville: Still around?
<lifeless> so matcher=DocTestMatches('foo bar', NORMALIZE_WHITESPACE)
<lifeless> StevenK: ^
<StevenK> lifeless: Which gives AttributeError: 'str' object has no attribute 'match'
<StevenK> lifeless: Since what I'm comparing to is just a string
<lifeless> StevenK: self.assertThat(yourstring, DocTestMatches('foo bar', doctest.NORMALIZE_WHITESPACE)
<lifeless> )
<lifeless> StevenK: or paste your code so that I'm not making random guesses ;)
<StevenK> lifeless: http://pastebin.ubuntu.com/570869/
<lifeless> StevenK: so a small style thing that made me wtf when I read this
<lifeless> StevenK: a matcher isn't static, its an active object
<lifeless> so calling one expected_contents is mind wrenching
<StevenK> lifeless: A small note: I've never used matchers before
<lifeless> I would probably create the string separately
<StevenK> lifeless: Renamed to recipe_matcher
<lifeless> expected_contents = '...'
<lifeless> browser =
<lifeless> self.assertThat(browser.contents, DocTestMatches(expected_contents, NORMALIZE_WHITESPACE))
<lifeless> or rename it as you have, that fine too
<lifeless> ok, and is it still giving you this error ?
<StevenK> Yes, because the newlines aren't ignored
<lifeless> it might be clearer to read if you put the doctest.NORMALIZE_WHITESPACE left alighted with the '<a href'
<lifeless> http://docs.python.org/library/doctest.html#doctest.NORMALIZE_WHITESPACE
<lifeless> StevenK: you said it was giving you a str has not attribute error
<lifeless> is that resolved?
<wgrant> jam: Around?
<StevenK> Yes, if the string is first in assertThat, it's fine
<lifeless> >>> print DocTestMatches('foo\nbar').match('foo bar')
<lifeless> <testtools.matchers.Mismatch object at 169b950 attributes={'matcher': <testtools.matchers.DocTestMatches object at 0x7f3b8471fd50>, 'with_nl': 'foo bar\n'}>
<lifeless> >>> print DocTestMatches('foo\nbar', NORMALIZE_WHITESPACE).match('foo bar')
<lifeless> None
<lifeless> StevenK: ^ works for me
<lifeless> StevenK: paste the error you get
<StevenK> Oh, sure, let me just edit the tal to inclue 'foo bar' and all my problems will be solved. :-P
<lifeless> StevenK: oh, and a tiny tweak - s/recipe_matcher/recipe_matches/
<lifeless> the intent is to be able to read the assertion as a sentence
<StevenK> lifeless: http://pastebin.ubuntu.com/570870/
<lifeless> StevenK: you're comparing totally different things
<lifeless> the matchee is the entire portlet
<lifeless> the thing you're matching against is a tiny fragment of that
<StevenK> Now I guess, that the matcher is expecting the content to be exactly what I want, but I only care that the string is in browser.contents
<lifeless> right
<lifeless> so for that, you need ELLIPSIS
<lifeless> -or- to use soupmatches, which is structured and much better for this sort of thing
<lifeless> anyhow
<lifeless> add
<lifeless> ... at the start and end of your expected text
<lifeless> and add | ELLIPSIS
<lifeless> to the flags
<StevenK> lifeless: My issue with soupmatchers was I wasn't sure how to add the free-form text into the mix, such as 'by recipe'
<lifeless> james_w: ^
<wgrant> lifeless: Could you please mentor https://code.launchpad.net/~stevenk/launchpad/less-lazr-security/+merge/50824?
<lifeless> looking at it already
<lifeless> if I get a test timeout in ec2
<lifeless> but no other errors
<lifeless> does that suggest my changes are ok ?
<wgrant> lifeless: Windmill layers tend to run late in the suite.
<wgrant> But not necessarily right at the end.
<wgrant> It suggests that are OK, but does not prove anything.
<wgrant> I would retest.
<wgrant> Unless the changes are really simple.
<lifeless> ttps://code.launchpad.net/~lifeless/launchpad/bug-717394-3/+merge/50546
<wgrant> Eeeh. I'd retest.
 * StevenK switches back to soupmatchers
<wgrant> You never know what bad things people are doing.
<lifeless> StevenK: so the ... and ELLIPISIS at the start and end will work
<lifeless> StevenK: if soupmatchers isn't easy to approach, just do the ... for now
<StevenK> But they didn't ...
<lifeless> pastebin
<lifeless> both your new code
<lifeless> and the new error
<StevenK> Too late was the cry as the ship went down?
<StevenK> lifeless: I had the soupmatchers code in my undo history, so I've switched back to it
<lifeless> StevenK: the doctest matcher is in your pastebin
<lifeless> ;)
<StevenK> Yes, I'm fixing it without your help :-P
<StevenK> And fixed!
<lifeless> cool
<lifeless> what does it look like with soupmatchers?
<StevenK> I'd like to also check that the strings 'by recipe' and 'for' appear in the right place, but I don't know how to put them in
<StevenK> lifeless: http://pastebin.ubuntu.com/570872/
<StevenK> lifeless: Sadly, I agree with your less-lazr-security comment.
<StevenK> thumper: Ping
<lifeless> HTMLContains
<lifeless> StevenK: ^ looks like what you need to get the by recipe and for bits evaluated
<lifeless> HTMLContains(Equals('by recipe'))
<StevenK> Equals is from testtools.matchers?
<lifeless> yes
<StevenK> And where did you see that pattern?
<lifeless> I read the pydoc for soupmatches
<lifeless> and am guessing
 * StevenK tries it
<lifeless> http://bazaar.launchpad.net/~soupmatchers-dev/soupmatchers/trunk/annotate/head:/README
<StevenK> Actually, Equals() might just do it by itself
<StevenK> Since it has a .match() attribute
<StevenK> Er, method
<lifeless> yes, Equals is a matcher
<lifeless> as is Not
<lifeless> RaisesException
<lifeless> and so on
<lifeless> StevenK: actuall
<lifeless> the readme suggests
<lifeless> Tag("...", text="built for") or some such
<lifeless> might want to wrap your entire bit in a span or something. I suggest reading the readme
<StevenK> I have, twice
<lifeless> does text= not meet your needs?
<StevenK> I haven't tried it
<lifeless> ok
<StevenK> Oh, no tagname or attrs, just text=''
<StevenK> soupmatchers.Tag('first text', text='by recipe'),
<StevenK> TypeError: __init__() takes at least 3 non-keyword arguments (2 given)
<StevenK> lifeless: I've just read the README again and couldn't distill any clues about matching plain text
<lifeless> StevenK: well its not entirely plain i sit? its embedded in an html structure
<lifeless> so you need to match the containing span
<lifeless> or try equals
<lifeless> or do the doctestmatches thing - personally I'd do that simply to get unstuck and moving
<StevenK> Equals() didn't work and neither did HTMLContains(Equals())
<StevenK> lifeless: I can just not check for the text and the test passes. I'd just like to actually check it
<huwshimi> lifeless: When you've got a chance I'd like to figure out how to land my Loggerhead changes.
<StevenK> lifeless: Oh, further to your shipit comment: That will turn up during the test suite?
<wgrant> StevenK: It will, and it won't break.
<lifeless> ec2 land will run the shipit tests
<wgrant> ShipIt doesn't use those bits.
<lifeless> wgrant: any idea how we can qa Bug 720826 ?
<_mup_> Bug #720826: Add subscription description header for bug notifications <qa-needstesting> <story-better-bug-notification> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/720826 >
<lifeless> \o/ bug down down by 70 queries on prod
<lifeless> vs prod I mean
<lifeless> 4.42 seconds hot on staging!
<lifeless> booyha
<thumper> StevenK: pong
<wgrant> Why does strucsub live in Bugs now? :/
<lifeless> because it was started as a bugs project
<lifeless> before the desiloisation of teams
<wgrant> Yes.
<wgrant> But it lived in Registry.
<wgrant> Then was moved to Bugs post-desiloisation.
<lifeless> by folk already fixed in the bugs silo - its only future work I really expect this to change
<lifeless> anyhow, registry is equally wrong ;) <domain> is wrong :)
<StevenK> thumper: Are you fine with me landing a branch that pulls in lazr.restful 0.17.1?
<thumper> there is a 0.17.1?
<StevenK> thumper: IE: It won't break anything, or I should wait until your work lands?
<thumper> StevenK: it makes no difference AFAIK if you land it now
<thumper> StevenK: mine just builds on new features
<thumper> StevenK: what is the update in 0.17.1?
<StevenK> thumper: It adds a test, that's it.
<StevenK> So I can remove it from the Launchpad tree.
<thumper> ah...
<thumper> that's fine with me
<thumper> land it
<StevenK> Doing so
<thumper> I'll deal with the merge conflicts
<StevenK> thumper: My recipe description DROP NOT NULL fix should have hit db-devel, too
<wgrant> lifeless: That's qa-ok. The UI really sucks, but it works.
<wgrant> And it's not linked.
<wgrant> Which means we only need that pofilestats thing.
<wgrant> And I think we can probably OK that, since apparently it's broken on production anyway and no more broken now.
<thumper> how can I quickly record a screen cast?
<thumper> StevenK: did you test the UI?
<thumper> StevenK: do you know if required defaults to True, or False ?
<thumper> StevenK: I have a feeling it defaults to True
<huwshimi> thumper: I've used 'gtk-recordmydesktop' in the past with success
<thumper> huwshimi: ta
<huwshimi> thumper: It's worth noting that it will record sound out of your mic unless you uncheck the sound option (great if you want to narrate it, not so great if you are talking to yourself).
<thumper> huwshimi: hmm... where did it save it?
<huwshimi> thumper: Home folder?
<lifeless> huwshimi: hi
<thumper> yep
<huwshimi> lifeless: Hello.
<lifeless> huwshimi: uhm, propose them, get them reviewed, and land manually if its on trunk
<lifeless> running the test suite, of course
<thumper> Behold the magic! http://people.canonical.com/~tim/js-update.ogv
<huwshimi> lifeless: OK, so should I just try to get my changes landed directly to trunk?
<thumper> And the code: http://pastebin.ubuntu.com/570890/
<thumper> \o/
<wgrant> thumper: Yay, that fixes two bugs I filed.
 * thumper takes a bow
<wgrant> That is awesome.
<wgrant> And needs to be applied everywhere.
<thumper> wgrant: yes. yes it does
<thumper> I'll be emailing the list
<thumper> wgrant: which two bugs?
<lifeless> huwshimi: that was the conclusion from my discussions with poolie and jml: that we'll go with one theme for loggerhead upstream & lp, make it nice for standalone or lp users
<thumper> wgrant: check out the underlying code :)
<lifeless> huwshimi: for things that we need to be lp specific we'll need to add callbacks inside loggerhead that can call into the launchpad_loggerhead stuff
<lifeless> wgrant: Bug 722568 needs a losa doesn't it ?
<_mup_> Bug #722568: Empty TranslationMessages confuse stats <qa-needstesting> <Launchpad itself:Fix Committed by jtv> < https://launchpad.net/bugs/722568 >
<wgrant> thumper: Ah, I put both in one. Bug #721064.
<_mup_> Bug #721064: Inline recipe text editor doesn't update the "Debian version" field on the page <recipe> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/721064 >
<StevenK> thumper: That's awesome
<huwshimi> lifeless: OK. I have one change that is LP specific. What should I do with it? Remove it for now?
<lifeless> huwshimi: what is it
<StevenK> wgrant: Can you look at link-recipe-on-ppa-page again?
<wgrant> lifeless: Yes, but it's already broken on production.
<thumper> StevenK: which one?
<thumper> StevenK: link
<thumper> ?
<lifeless> wgrant: well it runs with incorrect results sometimes
<lifeless> wgrant: thats not totally broken, and the change might totally break it in theory
<lifeless> wgrant: so I think wait for losa
<huwshimi> lifeless: it's a backlink to the branch summary page in Launchpad, and labelled specifically so.
<wgrant> lifeless: No, it crashes.
<StevenK> thumper: The screencast looks awesome
<lifeless> wgrant: orly? I thought that was staging only that crashes
<wgrant> lifeless: That's what we thought too.
<wgrant> lifeless: But check carob:/srv/launchpad.net-logs/scripts/loganberry/rosetta/pofile-stats.log-20110217
<wgrant> People became aware of this after we were both asleep, I believe.
<wgrant> thumper: How many requests does it need?
<thumper> wgrant: there is only one patch request
<wgrant> thumper: Or does it request all the relevant representations?
<wgrant> Nice.
<thumper> wgrant: lazr.restful was updated to return the full entry info + html
<wgrant> thumper: The duplication sort of sucks, though.
<thumper> wgrant: I'm fixing the duplication
<lifeless> huwshimi: you should make that configurable in some way (we can talk about how to do that separately); land it with it configured to not show, and change launchpad_loggerhead in the launchpad tree to make it show when we deploy
<wgrant> thumper: Great.
<wgrant> thumper: So, one line per field for dynamic updates? That is fantastic.
<lifeless> huwshimi: we want to end up running a stock loggerhead with (conceptual) plugins to it, rather than a patched loggerhead, to reduce engineering overheads
<wgrant> I like AJAX that is not fucking painful to code.
<lifeless> thumper: this lazr.restful change
<lifeless> thumper: its optional, right ?
<thumper> lifeless: optional in which way?
<lifeless> server side generation of the html
<lifeless> thumper: I mean, launchpadlib api requests shouldn't trigger it
<thumper> not unless they ask for it
<wgrant> I presume the client can pass multiple Accepts.
<thumper> actually...
<lifeless> because its fantastically slow by comparison to just a json dict
<lifeless> *fantastically slow*
<thumper> no I don't think launchpadlib gets it
<lifeless> cool
<lifeless> nice work
<thumper> you have to specify Accept: application/json;include=lp_html
<huwshimi> lifeless: OK great. So, how to make it configurable?
<wgrant> Excellent.
<lifeless> huwshimi: How does it work?
<wgrant> thumper: It might be nice if you could specify particular fields.
<wgrant> But this shouldn't be too bad.
<thumper> wgrant: we specifically only do one event for the change to avoid race conditions
<lifeless> wheeee
<wgrant> thumper: btw, do you know what's up with the empty grid cell on recipe pages?
<lifeless> 11 seconds python time - wtf
<thumper> wgrant: each page could break it apart more easily
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1880QS43
<wgrant> Above "Debian version"
<thumper> wgrant: yes
<thumper> wgrant: it has to do with the yui grid flow
<wgrant> thumper: Easily fixable?
<thumper> wgrant: if the height of the top left is greater than the height of the top right, then the next one is below right
<wgrant> Ah :/
<thumper> yui grid layout is deprecated
<thumper> huwshimi will fix :)
<wgrant> Hopefully before they are out of beta.
<wgrant> They want to look awesome before they are pushed out to everyone.
<wgrant> lifeless: Wow. I hope that's just asuka being crap.
<huwshimi> thumper: I'll do what now? If it's to do with positioning I'm not help you :)
<lifeless> wgrant: look at query 20
<wgrant> lifeless: You are blocking QA now :P
<wgrant> Awesome.
<wgrant> That is really awesome.
<lifeless> wgrant: not at all, I've been qaing it
<wgrant> Also, WTF.
<wgrant> thumper: No inline name editing :(
<thumper> wgrant: it is coming
<wgrant> !
<wgrant> No inline series editing :(
<thumper> wgrant: I wanted my fix for the auto-url change
<thumper> wgrant: series editing needs more work
<wgrant> :(
<huwshimi> lifeless: In the templates there are links (in a few templates). That's pretty much it. E.g. lines 19-25 of http://bazaar.launchpad.net/~huwshimi/loggerhead/launchpad-theme/view/head:/loggerhead/templates/changelog.pt
<thumper> as we don't have any reasonable selectors
<wgrant> thumper: Oh?
<wgrant> thumper: Can't it just be a form like the build request one?
<lifeless> huwshimi: how are you calculating the link
<huwshimi> lifeless: The link already existed.
<lifeless> huwshimi: you have a condition there already
<huwshimi> lifeless: We already had a backlink, but now it's more obvious
<lifeless> huwshimi: I suggest running it up standalone and seeing if the links already hide themselves automatically
<huwshimi> lifeless: They do
<lifeless> huwshimi: so that seems fine for trunk
<lifeless> its data driven
<lifeless> and $otherdeployment of loggerhead can inject their own branch_link attribute
<huwshimi> lifeless: Ok, well I'll put together an MP then. Thanks
<thumper> wgrant: probably, but that makes it not a simple patch selector...
<thumper> wgrant: I'll look at it
<thumper> wgrant: it'd be good to be able to change everything
<lifeless> ok, 37 /  407  POFile:+translate wasn't librarian, its up again
<lifeless> 16 /   11  Archive:+copy-packages
<lifeless> etc
<lifeless> I'll go bug filing spree
<lifeless> I think a lot are GIL issues, we'll see when the ppr refreshes in a few hours
<wgrant> thumper: That page is just so close to being a perfect model for everything else.
<wgrant> thumper: It seems sad to leave one thing un-AJAXed.
<wgrant> +copy-packages already has a bug.
<lifeless> wgrant: I know
<lifeless> but first, stuff
<thumper> rockstar: check this out: http://people.canonical.com/~tim/js-update.ogv
 * thumper sighs
<thumper> rockstar: should have been in the other channel, but my client can be dumb
<StevenK> wgrant: O hai? You missed my prod?
<wgrant> StevenK: Got distracted, looked a couple of minutes ago. Just one more formatting issue that I am looking for pretty solutions to.
<StevenK> wgrant: And there's the issue of the traversal. Leave it, Julian be damned, or kill it?
<wgrant> StevenK: I argued with him last night and he facepalmed. Leave it.
<StevenK> I don't think I can land it with his "Needs Fixing"
<lifeless> sure you can
<StevenK> Fair enough.
<wgrant> Most side reviewers do not rescind their Needs Fixing.
<wgrant> StevenK: Reviewed.
<StevenK> wgrant: Your diff is wrong, it's sprb.requester, and that pushes it over the length limit
<wgrant> StevenK: Look a few lines earlier.
<StevenK> Bleh
<StevenK> wgrant: Then why is text still sprb.requester.displayname? :-)
<wgrant> StevenK: Because I didn't notice that one because its formatting didn't suck!
<wgrant> pls fix
<StevenK> wgrant: As you keep saying to me: SILENCE!
<wgrant> That's the one.
<StevenK> wgrant: http://pastebin.ubuntu.com/570905/
<StevenK> Er, ignore the lack of context at the end of the diff, my mouse ate it
<lifeless> StevenK: how many instances can you reliabilty bring up in jenkins on our test cloud
<wgrant> Looks good.
<wgrant> Thanks.
<StevenK> lifeless: I have no idea. Why?
<lifeless> thinking about parallel testing
<StevenK> Er?
<lifeless> we can trivially multi machine parallel test on jenkins
<wgrant> So we disable Windmill, move to Jenkins, and have things in stable in 30 minutes? :)
<StevenK> lifeless: Parallel test what, exactly?
<StevenK> I'm still confused.
<lifeless> lp
<lifeless> if we provide a fractional filter to run 1/M tests
<lifeless> and start up M machines
<StevenK> lifeless: 1: We can't use the UEC if we move to Jenkins to prod usage. 2: I'm not certain how many instances I can run in parallel
<lifeless> we can't ?
<StevenK> No
<StevenK> elmo forbids it.
<lifeless> oh
<lifeless> did he say why?
<StevenK> Yes. It's used by everyone who has an account, and there are no resource limits available on our UEC currently, which means we can get 'locked out'
<StevenK> lifeless: When you get undistracted, can has another look at https://code.launchpad.net/~stevenk/launchpad/link-recipe-on-ppa-page/+merge/50694 ?
<lifeless> isn't it requestor ?
<wgrant> It varies.
<wgrant> I've seen both, and I don't think one is particularly American :(
<lifeless> small style thing
<lifeless> optionak
<lifeless> bt I'd indent the <a ... lines in tal
<lifeless> if you look at the other <li> regions that is done there
<huwshimi> lifeless: For the loggerhead review it will be a Launchpad dev who does it, right? So should I just ask here?
<lifeless> huwshimi: yes, normal process - request on the project, ask the OCR here
<lifeless> huwshimi: that applies for *everything* listed on launchpad.net/launchpad-project
<huwshimi> lifeless: Right, got it. Thanks
<StevenK> lifeless: Done locally, anything else?
<wgrant> I suppose I should take an OCR slot.
 * wgrant takes an OCR slot.
<StevenK> \o/
<lifeless> StevenK: i commect on the mp
<huwshimi> StevenK: A review if you can: https://code.launchpad.net/~huwshimi/loggerhead/launchpad-theme/+merge/50863
<StevenK> lifeless: So, file a bug saying IPerson.getPPAByName() has to die or be fixed?
<StevenK> huwshimi: You know we have a branch limit of 800 lines, right? :-)
<StevenK> But, uh, lots of CSS
<StevenK> huwshimi: Can has screenshot?
<huwshimi> StevenK: Oh I did not. I'm not even sure I know what you mean by that.
<StevenK> Diff against target: 1415 lines (+606/-636) 10 files modified
<StevenK> We prefer that 1415 to be under 800
<StevenK> But it is a preference
<huwshimi> StevenK: Right
<huwshimi> StevenK: Would you like me to do something about that?
<StevenK> huwshimi: No, it's fine. As you say, it's a massive CSS cleanup.
<huwshimi> StevenK: Screenshot coming up. Actually how do I (can I?) push to people/~huwshimi
<StevenK> huwshimi: scp to people.canonical.com:public_html
<huwshimi> StevenK: Great thanks
<lifeless> StevenK: it will depend on what the fix for 'ppas are only for ubuntu' is
<StevenK> huwshimi: As an employee, you ought to have access.
<StevenK> lifeless: I suspect derived distributions will also grow PPAs
<lifeless> StevenK: for instance, we could say that a ppa called 'testtools' can *either* be for Ubuntu
<lifeless> *or* for Linaro
<lifeless> which would leave that API completely safe to use.
<lifeless> or we could say that the various suites in a PPA can extend across any support distro in LP
<lifeless> or we could support multiple ppas with the same name on a person, and different distros.
<StevenK> My thought is that if the distribution is not in the URL, it's Ubuntu, and if it is, it's that.
<lifeless> I think its prejudging it to decide that that API is a problem.
<wgrant> One thing we need to say is that this is going to be the last solution.
<wgrant> Because every previous one has been designed ignoring everything except the immediate issue.
<StevenK> huwshimi: You *might* need to ssh into people.canonical.com first and 'mkdir public_html'
<StevenK> wgrant: Then you file a bug? :-)
<StevenK> wgrant: Did you file a bug, or shall I?
<wgrant> StevenK: I didn't. I'm not sure it needs one.
<StevenK> lifeless seems to think so.
<lifeless> erm
<lifeless> 16:59 < lifeless> I think its prejudging it to decide that that API is a problem.
<lifeless> did that get through ?
<StevenK> Yes
<wgrant> I think it should be left to the design work next week.
<lifeless> You *can* file a bug if you agree with julian that that api is a problem.
<wgrant> The issue is going to be clear as soon as someone goes near multi-distro PPAs.
<wgrant> So a bug does not add very much at all.
<wgrant> Particularly as it cannot recommend a solution, nor indeed clearly identify the problem.
<lifeless> there's at least 4 different answers to the question 'what should ppas interaction with linaro be'
<lifeless> the api isn't a problem today
<lifeless> it might be a problem in the future
<StevenK> It, and others *will* be when we want to add PPAs for distributions that aren't Ubuntu.
<StevenK> Significant parts of Soyuz are coded in such way to blithely believe that the only thing we ever build for and publish for is Ubuntu.
<lifeless> hang on
<lifeless> those are unrelated assertions
<wgrant> StevenK: No, they actually aren't, but anyway.
<lifeless> StevenK: nothing says one PPA can't be for both linaro and ubuntu
<lifeless> StevenK: similarly, nothing says we have to permit a PPA name to be shared across distros
<lifeless> which would equally leave that API safe
<StevenK> Right, I'll land it, and won't file a bug. wgrant, Julian and I are aware of the issues surrounding multi-distro PPAs
<wgrant> And anybody who tries to do multi-distro PPAs will become aware very rapidly.
<wgrant> :( KDE's bugzilla lies.
<wgrant> It says it uses P1 through P5 as priorities.
<wgrant> But some of the bugs have priority 'NOR'...
<lifeless> jml: https://dev.launchpad.net/LEP/ParallelTesting
<lifeless> wgrant: fun
<StevenK> thumper: Can has review of my review: https://code.launchpad.net/~huwshimi/loggerhead/launchpad-theme/+merge/50863
<lifeless> huwshimi: nice
<StevenK> I think it looks brilliant
<StevenK> lifeless: Did Launchpad lose it's picture?
<StevenK> In the top left corner of https://launchpad.net/launchpad , that is
<huwshimi> lifeless: It's a start. Just need someone to make it part of Launchpad proper now :)
<lifeless> huwshimi: it has an ajax api
<wgrant> That's a nice start.
<lifeless> huwshimi: I don't think we want to fold the code into LP; we do want it to be better integrated
<lifeless> less of a box-of-bits feeling to ut.
<lifeless> *it*
<lifeless> StevenK: no, its there for me.
<huwshimi> lifeless: Yeah, agreed
<lifeless> one think I'd like to do is folk the loggerhead api into the main website urlspace
<lifeless> so that we can use it without cross domain stuff
<lifeless> s/folk/fold
<StevenK> lifeless: You're using Chrome, and I'm using Firefox. This may be related.
<wgrant> StevenK: Works OK for me in both.
<lifeless> StevenK: there for me in ff too
<wgrant> Firefox 4b11, though.
<lifeless> 3.6.13 for me
<wgrant> StevenK: What was it showing?
<wgrant> mizuho may be being crap again.
<wgrant> It is very good at that.
<StevenK> wgrant: Nothing
<wgrant> StevenK: What did Firebug say about the image load?
<lifeless> wgrant: thanks for prepping the deploy
<wgrant> lifeless: I just couldn't let you do two in a row.
<wgrant> Or my inbox was crying.
<StevenK> Haha
<wgrant> One of those.
<lifeless> 5.4 million pages yesterday (+ opstats/haproxy)
<StevenK> wgrant: How can I tell?
<StevenK> I've found the logo <img> in firebug
<wgrant> StevenK: There should be a Network tab or something.
<wgrant> You don't want the DOM browser.
<lifeless> 3M api.*
<lifeless> so thats what 60% just on launchpadlib :(
<wgrant> lifeless: Do we log users?
<lifeless> its determinable somehow
<wgrant> I wonder if anyone will notice/complain if I move the security deploy stuff to the bottom of the page in a trivial edit...
<lifeless> wgrant: I think that would be fine
<StevenK> wgrant: Content-Typetext/html
<wgrant> I might also precede it with a \0 to indicate how much I love it.
<wgrant> StevenK: Oh dear.
<wgrant> StevenK: Which frontend?
<wgrant> And which request URL?
<wgrant> Length: 2050 (2.0K) [text/html]
<wgrant> :(
<StevenK> GET /50084288/launchpad-logo HTTP/1.1
<StevenK> Host: launchpadlibrarian.net
<wgrant> Bypassing the cache gets an octet-stream as expected.
<StevenK> wgrant: Looks like bananananana
<StevenK> launchpad_dogfood=> SELECT mimetype from libraryfilealias where id = 50084288;
<StevenK> application/octet-stream
<wgrant> It is both :(
 * StevenK facepalms
<wgrant> Both somehow got text/html.
<wgrant> Squid-log-diving time, I guess...
<StevenK> Orsum
<lifeless> relatedly
<lifeless> there is a bug open
<lifeless> about the librarian and mimetypes - and comments in the code too
<lifeless> the idea is that we honour the mimetype and stop guessing
<lifeless> and then the librarian server *only ever* hands out what the mimetype says it should be
<StevenK> Can we force it be image/png, then?
<lifeless>  if you update the LFA yes
<lifeless> thats the point, the LFA has crap in it
<lifeless> a) garbo hourly task to fix those
<lifeless> b) migrate librarian to use that data
<lifeless> c) win
<wgrant> application/octet-stream is not optimal, but also OK.
<LPCIBot> Project devel build (465): FAILURE in 5 hr 24 min: https://hudson.wedontsleep.org/job/devel/465/
<LPCIBot> * Launchpad Patch Queue Manager: [r=benji][bug=134063] Updates DateTime widget to kick back formats it
<LPCIBot> can't possibly cope with.
<LPCIBot> * Launchpad Patch Queue Manager: [r=sinzui][ui=sinzui][bug=687379] Changes alignment of diff overlay
<LPCIBot> popup to align on the diff popup link, rather than on the viewport,
<StevenK> Right, but image/png let's the browser plan a little better
<LPCIBot> so the diff won't expand above the top of the page.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][ui=none][bug=723003] Make the Sphinx doc build process not
<LPCIBot> try to connect to the Internet.
<StevenK> test_404 and test_oldurl again
<StevenK> ANNOYANCE
<wgrant> Yes.
<wgrant> Hmm.
<wgrant> I wonder if we should catch that exception, cause the test suite to hang in an unkillable way, and spam people to look at the instance.
<StevenK> In an Hudson run? There's one person that has keys to the builders
<wgrant> Or buildbot.
<wgrant> Or ec2, although that is far less frequent.
<StevenK> No signs of OOM killage
<StevenK> I wish I knew what was going on
<wallyworld_> bollocks. if i've already created a mp, there's no way to go back and set a dependent branch?
<lifeless> resubmit
<lifeless> and then set the dependent branch during that step
 * wallyworld_ shakes fist at stupid mp page :-(
<StevenK> resubmitting doesn't allow you to do that?
<lifeless> this is a bit suboptimal
<wgrant> StevenK: I think that changed recently.
<StevenK> Bout time
 * wallyworld_ takes it all back. it's actually quite easy
<wallyworld_> i had thought i would have to start again, but i forgot all this stuff was enhanced at some point
<wgrant> Yay, I crashed KDE bugzilla.
<StevenK> No big loss
<mwhudson> wgrant: that makes feature equivalence between KDE and gnome now?
<wgrant> Heh.
<wgrant> Ahaaaa
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> :(
<wgrant> We have two separate Bugzilla status maps.
<lifeless> \o/
<wgrant> Here I was wondering why this test wasn't failing when someone broke status mapping a couple of months ago...
<lifeless> hmm
<lifeless> 7:30 at night
<lifeless> I might finally get to write some code
<wgrant> Heh.
<lifeless> ta is a balancing act ;)
<wallyworld_> anyone seen this windmill error on ec2? 'BugsYUIUnitTestCase' object has no attribute 'client'
<wallyworld_> that's not even related to my changes
<wgrant> That's a new one to me.
<wgrant> But probably still spurious.
<wallyworld_> 2nd time it's happened :-(
<wallyworld_> got a bunch of them
<lifeless> race condition?
<wallyworld_>   lp.app.windmill.tests.test_yuitests.AppYUIUnitTestCase.runTest
<wallyworld_>   lp.bugs.windmill.tests.test_yuitests.BugsYUIUnitTestCase.runTest
<wallyworld_>   lp.bugs.windmill.tests.test_yuitests.BugsYUIUnitTestCase.runTest
<wallyworld_>   lp.bugs.windmill.tests.test_yuitests.BugsYUIUnitTestCase.runTest
<wallyworld_>   lp.code.windmill.tests.test_yuitests.CodeYUIUnitTestCase.runTest
<wallyworld_>   lp.registry.windmill.tests.test_yuitests.RegistryYUIUnitTestCase.runTest
<wallyworld_>   lp.registry.windmill.tests.test_yuitests.RegistryYUIUnitTestCase.runTest
<wallyworld_>   lp.soyuz.windmill.tests.test_yuitests.SoyuzYUIUnitTestCase.runTest
<wallyworld_>   lp.soyuz.windmill.tests.test_yuitests.SoyuzYUIUnitTestCase.runTest
 * wallyworld_ is disliking windmill a lot today
<wallyworld_> lifeless:  could be a race condition. but you'd think an error like "has no attribute" would be quite deterministic
 * wallyworld_ will have to look a little deeper.....
<wallyworld_> just had to get it all off my chest. nothing like venting a little to relieve the stress
<lifeless> still, another branch landed. \o/
<lifeless> wgrant: why closed duplicate -> invalid ?
<lifeless> wgrant: shouldn't we dereference to the target ?
<wgrant> lifeless: Yes, we should. Bug #56644
<_mup_> Bug #56644: Remote bugs that are duplicates are shown as "Invalid" <bugwatch> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/56644 >
<wgrant> But this is reverting to the old behaviour that was lost in r11434
<lifeless> by accident?
<wgrant> Yes.
<wgrant> All unknown CLOSED/RESOLVED/VERIFIED statuses used to map to Invalid.
<wgrant> But that rev listed a few known ones, mapped them to Invalid, and mapped the rest to Unknown.
<wgrant> s/mapped them to Invalid/mapped them to appropriate statuses/
<lifeless> ah
<lifeless> and the map to unknown was bong
<wgrant> It was explicit, so nothing got logged.
<wgrant> Invalid isn't a great mapping, but it's better than Unknown.
<wgrant> Thanks.
<wgrant> jtv: Hi.
<lifeless> ok, I can make bug search 40 times faster
<lifeless> for the ubuntu homepage query
<wgrant> :(
<lifeless> see my comments on https://bugs.launchpad.net/launchpad/+bug/618406
<_mup_> Bug #618406: Distribution:+bugs-index timing out in ~2% of requests <lp-bugs> <pg83> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618406 >
<lifeless> wgrant: trivial fix
<wgrant> lifeless: Well, it makes it non-deterministic.
<lifeless> in a fashion noone should care an iota about
<stub> very non-deterministic, since IIRC we only have 4 levels of heat
<lifeless> 5
<wgrant> 5 flames.
<wgrant> But they are buckets.
<lifeless> 4 flames
<lifeless> 0,1,2,3,4
<wgrant> 5 levels of flame.
<wgrant> Right.
<lifeless> anyhow
<lifeless> we can change the index
<lifeless> (heat desc, id)
<lifeless> or (heat desc, importance desc)
<lifeless> or (heat desc, importance desc, id) - all on bug
<lifeless> however
<stub> ordering by heat, bug.id, bugtask.id might be better
<lifeless> stub: nope
<lifeless> stub: forces materialisation
<lifeless> stub: see the plan - its doing a sort() - i tried with heat desc, bug.id - still did the sort
<lifeless> however, and this is important
<lifeless> this is for this hot bugs list
<lifeless> *that* list has solid heat variation
<stub> It won't - you are retrieving 40 items and there are only 5 buckets
<lifeless> the buckets are not evenly populated
<lifeless> its a decay function
<lifeless> (the bucket allocator is a decay function)
<lifeless> the actual heat value is unconstrainted
<lifeless> and does not have buckets
<stub> The index we seem to want to use here is (heat, last_updated), so ORDER BY Bug.heat DESC, Bug.last_updated DESC LIMIT 40 should hit the index and be deterministic enough
<lifeless> (check the data)
<stub> ok
<lifeless>  Limit  (cost=6957590.92..6957591.02 rows=40 width=1017) (actual time=2761.605..2761.621 rows=40 loops=1)
<lifeless>    ->  Sort  (cost=6957590.92..6958004.71 rows=165515 width=1017) (actual time=2761.603..2761.613 rows=40 loops=1)
<lifeless>          Sort Key: bug.heat, bug.date_last_updated
<lifeless> causes materialisation
<lifeless>          Sort Method:  top-N heapsort  Memory: 92kB
<lifeless> I'm totally fine with us changing the index.
<lifeless> but I think we'll have very deterministic results for ubuntu as a whole with just heat.
<stub> oh... that is the heat_last_updated index not the (heat, last_updated) :-(
<lifeless>  Limit  (cost=6957590.92..6957591.02 rows=40 width=1017) (actual time=2649.104..2649.119 rows=40 loops=1)
<lifeless>    ->  Sort  (cost=6957590.92..6958004.71 rows=165515 width=1017) (actual time=2649.101..2649.106 rows=40 loops=1)
<lifeless>          Sort Key: bug.heat, bug.heat_last_updated
<lifeless> not to mention, but the hot bugs list is broken anyway
<lifeless>          Sort Method:  top-N heapsort  Memory: 92kB
<lifeless> its not including series bugs
<stub> So yeah, it seems deterministic enough for production. Tests might have difficulties though as you mentioned.
<stub> Yeah - I skimmed the indexes too quickly. We don't have a suitable multicolumn index at the moment for guaranteeing determinism and remain fast. We could add one if we want but that is always annoying to add indexes only to keep the test suite happy.
<lifeless> right
<lifeless> the top 40 bugs by heat in ubuntu are deterministic
<lifeless> based on staging data
<lifeless> possibly the last 2 will jump around a little, because there is a quantisation effect due to the algorithm.
<lifeless> stub: https://bugs.launchpad.net/launchpad/+bug/618406/comments/4
<_mup_> Bug #618406: Distribution:+bugs-index timing out in ~2% of requests <lp-bugs> <pg83> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618406 >
<lifeless> stub: make sense?
<stub> Apart from importance, which we don't have on Bug.
<lifeless> stub: grah, yes.
<lifeless> we could push heat down to the tasks - denorm is
<lifeless> or rather, make it context specific
<lifeless> which might give better results
<stub> We just create an index on (heat, id) and ORDER BY heat DESC, id DESC
<stub> That will give the same sort of determinism we have at the moment, and be nearly as fast as the heat only lookup
<lifeless> sure
<lifeless> wasn't sure if it was worth the index
<lifeless> I wonder if the heat index is being used at all at the moment
<stub> I suspect better an index than denormalizing :)
<stub> I think it is - it wasn't in the unused indexes lists I was looking at the other day. Probably being combined with other indexes.
<lifeless> stub: denormalisation isn't that bad. How big would this new index be?
<stub> double the size of the existing heat index...
 * stub forgets how wide timestamps are
<stub> oh.. its another int. double.
<lifeless> 60MB
<lifeless> lets do it
<lifeless> stub: I can't find any uses of bugtask.heat_rank
<lifeless> stub: am I missing something
<stub> lifeless: That column isn't used. I don't know if this was a column for a feature never implemented, or a column that should have been dropped.
<stub> We need to ask a bugger.
<wgrant> Doctest you suck and I hate you.
<wgrant> Yes, let's define the logging level in a file in another directory... what could possibly go wrong.
<lifeless> stub: I think just drop, it won't have valid data if its not being update
<lifeless> stub: can always add it if/when someone actually works up a feature for it
<stub> The only value in the production db is 0
<lifeless> exactly
<lifeless> grrr
<lifeless> public branches linked to private bugs -> boom
<wgrant> Hm? Works fine, modulo a leak or two.
<lifeless> https://code.launchpad.net/~oops-tools-dev/wsgi-oops/trunk
<wgrant> Oh, that one.
<wgrant> New bug.
<lifeless> Module lp.app.browser.tales, line 1529, in _link_summary_values
<lifeless> return {'id': str(self._context.id), 'title': self._context.title}
<lifeless> Unauthorized: (<Bug at 0x17001050>, 'title', 'launchpad.View')<br />
<wgrant> Right, it's trying to show the bug linked to the MP.
<wgrant> A bug was filed a week or so ago... trying to find it.
<wgrant> lifeless: Bug 719901
<_mup_> Bug #719901: Cannot view branch with revisions fixing private bugs <bug-branch-links> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/719901 >
<adeuring> good morning
<mrevell> Hello
<stub> Any opinions on where an ICleanup interface should live (for declaring adapters for arbitrary objects to a cleanup method)
<lifeless> stub: outside the lp tree? :P
<lifeless> stub: whats this for - do fixtures or context managers meet your needs already
<stub> I want to make the loop tuner call a cleanup method on the loop instance, if it is defined.
<stub> Nicer to use adaption rather than sniff for a well known name. And __del__ is fragile and IMO should never be used.
<lifeless> I'd make the loops a Fixture
<lifeless> add that into the base class
<lifeless> call setUp before using the loop
<lifeless> cleanUp at the end of the loop
<stub> The loops implement an interface - they don't subclass from any particular root.
<stub> there is no base class.
<lifeless> mmm
<lifeless> well you could interface this up
<lifeless> personally I'd just make them all subclass from fixture
<lifeless> the cleanup facilities in it are mature and useful
<stub> There doesn't seem to be any sane location in lp.* for such an interface :-P (nor a load of the other stuff still in canonical.*, as I'm sure jml is finding)
<lifeless> I'd be quite happy to have a zope interface for fixture in the fixtures tree
<lifeless> if that woul dhelp
<stub> Where is the fixtures tree?
<lifeless> https://launchpad.net/python-fixtures
<lifeless> is the upstream
<lifeless> its widely useful beyond unit testing, so ignore that bit of the blurb
<danilos> lifeless, fwiw, bug 720826 was already qad yesterday (before I added special rollout requirements for that), but since it was only [incr] change, I didn't consider it a big deal to mark the bug as such (i.e. it wouldn't block the rollout, wouldn't it?)
<_mup_> Bug #720826: Add subscription description header for bug notifications <qa-ok> <story-better-bug-notification> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/720826 >
<lifeless> danilos: --incr seems to be broken in some corner cases
<lifeless> danilos: I touched it because it showed up in the deployment-stable rpeort
<danilos> lifeless, ah, ok then, sorry for causing more work for others then (I did put it properly through QA on the kanban board, but unfortunately, that's still not linked to LP)
<wgrant> Is that the one where the [incr] was at the start, separated from the rest by a space?
<lifeless> danilos: no worries
<stub> lifeless: Not sure I want the 'exceptions are swallowed' bit.
<lifeless> wgrant: --incr combine with something or other results and doesn't send the right thing to pqm IIRC
<lifeless> stub: they aren't anymore
* jtv changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: jtv | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> stub: thats stale - where did you see that ?
<stub> lifeless: ic. The docstring for addCleanup says they are swallowed.
<lifeless> stub: I shall fix that
<lifeless> stub: if multiple excpetions are raised (one per cleanup, say), then one exception is raised, with its args being the individual exceptions that got raised
<lifeless> stub: if only one is raised, it is reraised asis
<lifeless> right, those docstrings are really stale. naughty lifeless
<stub> So declaring an IFixture interface seems overkill, as the sanest way to use this would be to just subclass Fixture and then isinstance is fine (which isn't the component architecture way, but that is often a good thing ;) )
<stub> jtv: Any objection to the loop tuner runner sniffing for isinstance(Fixture), and invoking the setUp and cleanUp methods?
<stub> Or perhaps that is better left for the garbo
<jtv> stub: otp for a minute moreâ¦ what is it you actually want to do?
<stub> Tell a loop tuner instance 'you were aborted so clean up now'
<lifeless> stub: personally, I'd just subclass Fixture on all our loops
<lifeless> stub: and then not even worry about the isinstance sniff. Seems simpler.
<stub> Without adding a new method to ILoopTuner, and without making all the existing loops inherit from a subclass
 * jtv reads backscroll
<lifeless> stub: I've pushed rev 22 explaining things better
<jtv> Are we supposed to use fixtures in production code?  I thought they were just for tests.
<stub> We have dozens of implementations and this is the first time we have wanted cleanup. Adding setup, cleanup, reset (addsetup etc.) to everything seems overkill.
<stub> jtv: Its generic code, not test specific, but alot of it makes no sense for this use case.
<jtv> Yesâ¦ maybe we should just have an extra method in the ITunableLoop, with an empty default implementation.
<lifeless> jtv: the inspiration was a specific set of testing problems; they are much more general than that.
<jtv> By the way stub, are you using the DBLoopTuner or the plain LoopTuner?
<stub> jtv: dblooptuner
<jtv> Cool.
<jtv> Anyway, in the spirit of duck typing, why not def cleanUp(self) to the tunable-loop API, guarantee that it gets called at least once by the loop tuner, and provide a blank default?
<jtv> I agree that a full fixture interface seems a bit much.
<jtv> After all, all we really want here is a *cough* *cough*destructor*cough* *cough*
<lifeless> jtv: the 'full fixture interface' is just: call setUp before doing stuff with it, and cleanUp afterwards.
<lifeless> jtv: so if we do the duck typing thing, its nearly no extra overhead to use the full fixture interface
<lifeless> I don't really mind whats done here, just throwing my 20c in the ring
<jtv> But then what does setUp _mean_?  It's still a method too many.
<lifeless> jtv: from that argument, cleanUp is unneeded to - the inner loops just need a finally
<jtv> I don't see how that worksâthe whole point is that this spans the interface between looptuner and tunableloop.
<jtv> So we're talking about how the tunable loop tells the looptuner, "I need you to promise that you'll call this cleanup method at the end, no matter what happens."
<jtv> One way we could do it is to make the caller use either "with" or a fixture (from my point of view they do the same thing, just with static/dynamic lifetime) but that burdens the call site with the needs of the type.
<stub> jtv: I started with ICleanup(loop) to do that. More complex than duck typing, but catches typos when you declare a 'cleanup' method instead of a 'cleanUp' method.
<jtv> Clever.
<stub> But I guess this is Python
<jtv> I wouldn't worry about it _too_ much, no.
<jtv> I guess it's easier to get the adapter wrong than to get the method name wrong.
<stub> I'll add cleanUp() to ITunableLoop as an optional method, which will be compatible with loops inheriting from Fixture.
<jtv> Yes, that's nice.  Do we have any of those?
<stub> No, but we might if this becomes a more common problem.
 * stub wonders if  we have any loops that don't inherit from TunableLoop, and we could just use inheritance
<lifeless> stub: I didn't see any when I was last looking at the code - thats why I suggested adding Fixture to the TuneableLoop base class
<wgrant> jtv: Could I throw a review your way?
<wgrant> Oh.
<wgrant> lifeless already did.
<wgrant> Sneaky lifeless
<jtv> wgrant: yes, but otp now
<stub> there are a few in translations
<StevenK> OMGWTFBBQ
<wgrant> ?
<StevenK> We need to QA and deploy r12433 *right* *now*
<jml> StevenK: patience.
<StevenK> That bug has been annoying me for *weeks8.
<StevenK> s/8/*/
<wgrant> Is that the MP diff thing?
<StevenK> wgrant: Yes
<wgrant> Heh.
<wgrant> We could QA and deploy that in 5 minutes if you really want to.
<StevenK> We could, yes ...
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (466): FIXED in 5 hr 27 min: https://hudson.wedontsleep.org/job/devel/466/
<LPCIBot> Launchpad Patch Queue Manager: [r=leonardr][no-qa] add an API to get all of the subscriptions for a
<LPCIBot> set of bugtasks.
<jml> that reminds me
<jml> sometime this week, I'd like Ursinha & matsubara to get a chance to orchestrate a nodowntime rollout
<wgrant> There's not that much orchestration to do, which is nice.
<jml> yeah. I get that impression.
<wgrant> Although if they were doing them I think they would make it even more trivial.
<jml> yeah, that's also my thinking :)
<wgrant> The largest task (apart from sorting out QA stragglers) is turning the deployment report into a list of bugs to put on the wiki.
<jml> plus, we really need to figure out this QA vs not-abysmal-to-deploy thing
 * jml goes to get food.
<wgrant> Out of order bug notifications make me sad.
<bigjools> jml: when using the logger in the poppy daemon, everything seems to get swallowed. I vaguely recall some weird trick you need to do to get it working?
<jml> bigjools: which logger? poppy uses two.
<bigjools> jml: gah
<jml> bigjools: or rather, there's Python logging and Twisted logging.
<bigjools> jml: logging.getLogger("poppy-sftp")
<bigjools> so the former
<jml> right.
<bigjools> presumably they need to be connected?
<jml> bigjools: you need to add a handler to it, I think.
<bigjools> twisted.... always needs something different :)
<jml> bigjools: umm... I guess they should be.
<jml> bigjools: historical reasons, when Twisted was first written, Python didn't have logging.
<bigjools> jml: ah I can copy what the buildd-manager does
<jml> bigjools: that sounds like a great plan.
<bigjools> sorted
<wgrant> jml: Hah, I just got hit by that LockWarner thing for the first time.
<jml> wgrant: yay, now we can be partners in hatred
<jml> wgrant: https://bugs.launchpad.net/launchpad/+bug/721166
<_mup_> Bug #721166: Tests sometimes fail on EC2 due to _LockWarner garbage <build-infrastructure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/721166 >
<wgrant> jml: The prime target of my hatred at the moment is Zopeless.
<wgrant> It has almost perished.
<jml> wgrant: I'm glad to hear it.
<wgrant> Afterwards it may be redirected to this garbage.
<jml> heh.
<jml> wgrant: there are three separate spurious test failures that bit me last week. I think I'll try to tackle one of them on my next hacking day.
<bigjools> jml: so, the idea for waiting on the log file output works a treat.  Except for the isolation tests :)
<jml> bigjools: you mean, ones that call code in-process?
<bigjools> jml: no, the simultaneous upload
<bigjools> I need to wait for multiple occurrences of the magic string
<jml> right.
<jml> a parameter to waitForClose() perhaps?
<bigjools> something like that yeah
<bigjools> I freaking love Python today
<deryck> Morning, all.
<bigjools> hi deryck
<jml> bigjools: yay
<bigjools> jml: string.count() FTW
<jml> bigjools: ahh yeah.
<jml> bigjools: you know you can do that directly off a string, no need to import string
<bigjools> I didn't
<jml> 'abba'.count('b') => 2
<jml> I want to spend all of today programming.
<jml> But I am going to have meetings and read specs instead.
<bigjools> jml: heh, doing this fix has highlighted a bug in the hooks.  If the uploads happen *really* close to each other then the path containing the timestamp overlaps
<jml> bigjools: yay finding bugs. that sounds like a fairly shallow one, which is even better :)
<bigjools> jml: not entirely sure how to fix it minds
<jml> bigjools: won't the targetcount uniquify them already?
<bigjools> yeah I just saw that
<bigjools> wtf
<jml> or are there separate "Hooks" instances?
<bigjools> urgh, that'll be it
<bigjools> nicely spotted
<bigjools> the cheap fix is to make targetcount static :)
<jml> yeah. the slightly less cheap but oh-such-good-value-for-money thing is to put the count on an object with a longer lifetime, such as an object that represents the server
<wgrant> For everything else, there's mktmp
 * bigjools spots tumbleweed rolling past
<bigjools> unfortunately the shell can't see the server it seems
<jtv> bigjools: since you're just tumbleweed-spotting, maybe you could review https://code.launchpad.net/~jtv/launchpad/bug-181368-ami/+merge/50910
<bigjools> jtv: self-review such a small change
<jtv> It's not the size, it's that I don't know if I did it right.  :)
<jtv> Also, there's a useful bash tip in the MP.
<bigjools> you'll find out when you try and use it :)
<jtv> Yeah I guess.
<bigjools> that is a useful tip - I've seen it before, never got around to setting it up
<jtv> For all those oh-sh*eep moments when you wish you'd remembered to echo $*
<jtv> Sorry, $?
<jtv> Why do we hard-code a geographical location into our choice of AMI?  I'm tempted to self-review a change to the Singapore one.  :)
<jtv> Or is there something in our script that picks the nearest?
<jml> jtv: approved.
<jtv> thanks jml, a sanity check is always welcome
<jtv> I find them more reliable than sheep entrails lately
<jml> jtv: find a better augur.
<jtv> Well I thought I did, after Arthur Andersen went down.
<jtv> Oh, au_gur_â¦
<jml> jtv: re geographical location, don't know. Maybe the ec2 api docs have something.
<jtv> It could probably help a lot to pick the right locationâ¦ to illustrate, apparently we'd get more bandwidth to the US through a proxy running in EC2 Singapore than we do directly.
<bigjools> the bandwidth that counts is between the ec2 instance and the DC
<jtv> bigjools: so we should all be using the one near the DC?
<bigjools> jtv: well I would have thought so, no idea where that is though
<bigjools> but the b/w between the US and London is immense anyway
<jtv> lucky bastards
<jtv> bigjools, was that you who had trouble with the LC_TIME setting going into the image that the image can't deal with?
<bigjools> jtv: yes
<bigjools> I mentioned it in the wiki
<bigjools> it expects en_US
<jtv> that's why I'm askingâ¦ turns out the script code removes LC_ALL from the env.  It could just do the same for LC_TIME I suppose.
<jtv> Oh, food gong
<bigjools> jtv-eat: ah yes that'd be better
 * bigjools away lunch
* danilos changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: jtv, danilos | https://code.launchpad.net/launchpad-project/+activereviews
<danilos> forgot about that bit :)
<gary_poster> ugh, staging down for code update again. :-(  Hopefully this one does not become another marathon outage.
<gary_poster> anyone happen to know if this one has lasted awhile?  (this is staging, not qastaging)
<jml> hmm.
<jml> ec2 is sometimes sending emails that are 33,557,808 bytes long.
<gary_poster> that seems excessive
<jml> yes.
 * jml finds out whether a partial base64 string can be decoded
<jml> not by standard python tools, is the answer.
<benji> jml: if this is a one-off, this thing appears to handle truncated base64 strings: http://www.motobit.com/util/base64-decoder-encoder.asp
<jml> benji: thanks. it doesn't seem to work for me. I'll try to bump up the limits on my mail server and hope for the best.
<danilos> jml, well, you can always make it in 4 tries at most
<jml> danilos: not sure what you mean.
<danilos> jml, you just try with round_to_4(string), round_to_4(string[1:]), round_to_4(string[2:]) and round_to_4(string[3:]) where round_to_4 rounds the length to zero modulus 4
<jml> danilos: oh, you're using an actual knowledge of what's going on. that's cheating.
<danilos> jml, base64 decodes each 4 characters into 3 bytes
<danilos> jml, heh, ok then :)
<jml> hmm. just looks like a data file. can't recognize the format.
<jml> deryck: we're really using YUI 3.3 on prod now, right?
<deryck> jml, indeed we are
<jml> deryck: sweet. thanks.
<deryck> np
 * jml adds a hacking task.
<allenap> jtv, danilos: Do either of you have time to review https://code.launchpad.net/~allenap/launchpad/async-person-merge-162510/+merge/50919?
<danilos> allenap, I'll take it :)
<allenap> danilos: Cheers :)
<danilos> allenap, I am very happy to see this happening, btw! :)
<allenap> danilos: Awesome!
<danilos> (talking about bug 162510, fwiw)
<_mup_> Bug #162510: Person:+delete timeouts : Person merging needs to be done asynchronously <canonical-losa-lp> <chr> <feature> <lp-registry> <merge-deactivate> <tech-debt> <timeout> <Launchpad itself:Triaged by allenap> < https://launchpad.net/bugs/162510 >
<danilos> allenap, line 195 of the diff: "nor is it not guaranteed" sounds kinda weird
<danilos> allenap, but, first things first: this is going to be backed up by a DB patch for is_pending_merge as well?
<danilos> allenap, s/is_pending_merge/is_merge_pending/
<allenap> danilos: No, it doesn't need a patch.
<danilos> allenap, how's it going to protect against race conditions?
<allenap> danilos: Do you mean against a merge being requested twice?
<danilos> allenap, yeah, possibly even to a different person
<abentley> henninge: would you be free to mumble soon?
<allenap> danilos: I haven't actually checked if it'll break in that case. I assume PersonSet.merge() will break, but perhaps I ought to add a specific check.
<allenap> danilos: My intention with Person.is_merge_pending is to use it to either display a message on Person:+index, or to disable all mutations from that page.
<danilos> allenap, right, but you can do it only inside the session if it's not persistent
<allenap> danilos: It's derived from the database.
<danilos> allenap, i.e. if somebody reloads the page, they won't see a warning and they'll be able to change stuff
<danilos> allenap, it is? oh
<danilos> allenap, right, I only got to that point now
<allenap> danilos: Adding notifications via the request object is session-bound, or something like that, but I wasn't planning on using that.
<danilos> allenap, yep, I see how you get it, the presence of a merge job indicates that the merge is pending, all good :)
<allenap> danilos: I've fixed the docstring, thanks for spotting that.
<danilos> allenap, I suppose framework to run PersonTransferJobs is already there? (or membership notifications wouldn't even go off :)
<allenap> danilos: Yeah, it'll get run with: cronscripts/process-job-source-groups.py MAIN
<danilos> allenap, cool, so the only other thing that I can ask you to do is to make sure DB privileges for that script are suitable for whatever the merging needs (it probably needs a lot more than what it used to); if you want to add a minimal test that runs the script in appropriate environment, I wouldn't mind
<allenap> danilos: Good idea :)
<danilos> allenap, it'd also be nice to get rid of the web server DB user privileges that are not going to be used, but I am sure that's very hard to figure out and very easy to get wrong, so if you feel like it's worth it, just file a bug
<allenap> danilos: I think that'll be a case of trial and error. Lots of error. I also suspect that I may end removing nothing.
<danilos> allenap, and it's a very nice branch, btw, I like it :) r=me
<allenap> danilos: Cheers, thank you!
<henninge> abentley: let's do it in 30 min if that's ok.
<abentley> henninge: sure.
<jcsackett> abentley: do you know how one gets full mp diffs available on qastaging? i need to qa a bug about diff-overlays.
<abentley> jcsackett: You need to run the merge-proposal-jobs script.
<abentley> jcsackett: though if your code has landed on staging, it's easier to qa there because the scripts already run.
<jcsackett> abentley: i'll see if it's there; thanks. if it's not, i assume i have to bug losas for the script?
<abentley> jcsackett: yes, you'd have to bug the losas.
<jcsackett> abentley: thanks.
<abentley> jcsackett: also, I think you'll need to run scan-branches before running the merge-proposal-jobs script.
<jcsackett> abentley: dig.
<bigjools> jml: your suggestion does not work
<abentley> jkakar: That issue I was having turns out to be a known bug that References don't know they're associated with ClassAliases.  Bug #682989
<jtv> bigjools, could you do something for me?  â
<jtv> echo $LC_TIME
<jml> bigjools: it does.
<bigjools> jtv: en_GB.utf8
<jtv> thanks
<jkakar> abentley: Ah, interesting, thanks for letting me know.
<bigjools> jml: pebkac :(
<abentley> jkakar: no worries.
<leonardr> lifeless: apropos our conversation of last week, https://bugs.edge.launchpad.net/lazr.restful/+bug/723753
<_mup_> Bug #723753: Continuous deployment makes web service mistakes much more dangerous <lazr.restful:New> < https://launchpad.net/bugs/723753 >
<henninge> abentley: let's chat! ;-)
<danilos> allenap, hi, do you know if BugNotificationRecipientReason gets used anyhow? should BugNotificationRecipientSet instead use it?
<allenap> danilos: Not off the top of my head. I'll have a look.
<danilos> allenap, thanks
<allenap> danilos: What's BugNotificationRecipientSet?
<danilos> allenap, s/Set/s/ :)
<allenap> Ah ha :)
* abentley changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: jtv, danilos, abentley | https://code.launchpad.net/launchpad-project/+activereviews
<allenap> danilos: Yes, I guess it was meant to be used by BugNotificationRecipients, but that looks like it would just complicate things now.
<allenap> danilos: It's dead code, may as well remove it. I wonder if the reason+header generation in BugNotificationRecipients is tested?
<allenap> I'm almost certain it is, but I can't remember where.
<danilos> allenap, what's interesting is that BugNotificationRecipientReason is well tested in lib/lp/bugs/mail/tests/test_bugnotificationrecipients.py
<abentley> danilos or jtv: could you please review https://code.launchpad.net/~abentley/launchpad/class-alias-mergable/+merge/50940 ?
<allenap> danilos: Yeah, odd isn't it. I guess it was never plumbed in.
<jtv> abentley: coming
<danilos> abentley, ok, I am letting jtv have it
<jtv> he's so generous
<abentley> jtv, danilos: thanks.
<abentley> jtv: still in the Netherlands?
<jtv> Yes
<danilos> abentley, fwiw, I remember writing a complex query with ClassAlias, but perhaps you need to explicitely add it to the store.using() list (at least I think it used to work)
<abentley> danilos: The issue was that I was using a Reference, and the Reference thought it was part of the class, not the ClassAlias.
<abentley> danilos: So I switched to the column property instead, which worked properly.
<danilos> abentley, ah, ok
<jtv> abentley: done
<jtv> ClassAliases are pretty shallow AIUI
<abentley> jtv, thanks.
<jtv> abentley: by the way, care to review a similarly small branch for me as well?  https://code.launchpad.net/~jtv/launchpad/bug-723733/+merge/50935
<abentley> jtv: the set_trace_if function seems a bit weird to me.  Is its main purpose is to reduce the number of lint warnings?
<jtv> abentley: yes, though I do also see those as a hint that this needs encapsulating.  If we're doing something weird, repeatedly, then it might as well have a name.
<abentley> jtv: I see what you're saying.  I think I might have written it into EC2Command.run instead, but this is fine.
<jtv> thanksâ¦ TBH I was a bit lazy because I didn't want it to overshadow the main change
<jtv> (And didn't want to spend much time on it)
<abentley> jtv: r=me
<jtv> thanks
<jtv> danilos, henninge: the approval page is no longer timing outâ¦ I just dealt with a few of the remaining oldest entries.
<LPCIBot> Project db-devel build (390): FAILURE in 5 hr 50 min: https://hudson.wedontsleep.org/job/db-devel/390/
<jcsackett> jtv: any chance you might be able to help me figure out why a user can't save any tranlations for the loco-teams directory?
<jtv> jcsackett: "can't save" any translations?
<jcsackett> she puts in some strings, tries
<danilos> jtv, cool
<jcsackett> tries to save them, and gets the "try again in a few minutes" message.
<jtv> jcsackett: sounds like a timeoutâ¦ got an oops number?
<jcsackett> jtv: no, just the "Try again in a few minutes, if it persists ask someone in #launchpad" message.
<jcsackett> jtv: full details are in #launchpad now, if you're interested in hopping in--i'm well into my zone of ignorance on just about anything translation related.
<jtv> jcsackett: I'll have a look
<abentley> henninge: Does this look reasonable to you? http://pastebin.ubuntu.com/571203/
<abentley> henninge: I was getting ForbiddenAttribute, and switching security policies doesn't fix that.
<jcsackett> jtv: ok, thanks.
<abentley> henninge: But once an attribute is writable by *someone*, then scripts can write to it.
<henninge> abentley: yes, looks reasonable. ;)
<henninge> abentley: The reason may be that these attributes are marked as "readonly" in the interfaces.
<henninge> I was wondering why the "set_schema" config  does not suffice.
<henninge> so "set_attribute" seems to override the "readonly" flag on the attribute.
<abentley> henninge: Yes, for TranslationMessage, setting it readonly=False makes set_schema provide it.
<abentley> TranslationTemplateItem doesn't use set_schema.
<henninge> interesting
<henninge> abentley: I was wondering how setSequence updates the sequence without a set_schema.
<henninge> abentley: Looking at it, it gets its TransalationTemplateItem instance from a database call, so I guess that returns naked objects.
<abentley> henninge: looks correct to me.
<bigjools> I find it really satisfying that when I want to force a package upload with a bad signature, the dput options are -fu
<jml> :)
<abentley> danilos or jtv: Could you please review https://code.launchpad.net/~abentley/launchpad/translation-splitting/+merge/50949 ?
<jtv> abentley: bit busy atm, and may get called away.  I'm afraid I'll have to bow out soon.
<abentley> jtv: understood.
<danilos> otp
<lifeless> jml: for Ursinha / matsubara to do a deploy - I'd be delighted to walk them through it whenever
<jml> lifeless: thanks.
<jml> lifeless: I think the key thing is getting wgrant to *not* do one
* jtv changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: danilos, abentley | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> jml: why?
<jml> lifeless: because there are rarely revisions to deploy when Brazil wakes up
<lifeless> there are 5 revs queued up already since last nights deploy
<Ursinha> lifeless, thanks, I'd love to do that
<jml> then I am misinformed :)
<Ursinha> (before wgrant :P)
<lifeless> jml: there are rarely revisions ready to go when wgrant and I get up as well - its proactive qa of revs in the queue that gets it ready for a deploy
<lifeless> right now for instance, a patch from jc is blocking the qa pieline, there are 2 more ready after that, and then the 4th blocked on a patch from me.
<lifeless> Ursinha: well, lets do one after the team lead call ?
<Ursinha> lifeless, yes!
<lifeless> \o/ 74 queries/external actions issued in 4.15 seconds for a bug search on qastaging / ubuntu
<jml> lifeless: down from...?
<lifeless> jml: last week it was timeouts :)
* danilos changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<danilos> abentley, sorry, I am wrapping up for the day, and your branch is not on the small scale (though, I am intrigued by it, since this is something we've been missing for general message-sharing as well :)
<thumper> deryck: I want to chat with you some time later about some yui / windmill tests, but when I'm more awake and caffeinated
<deryck> thumper, sure.
<lifeless> Ursinha: would you like to schedule a deploy ?
<Ursinha> lifeless, yes.
<lifeless> Ursinha: how would you like to do this - irc, voice, ?
<Ursinha> lifeless, I'll have a call with jml and then I'll be ready
<lifeless> kk
<Ursinha> works for you?
<lifeless> I'll go grab breakfast
<Ursinha> argh
<Ursinha> lifeless, I'm back
<lifeless> cool
<lifeless> so
<lifeless> Ursinha: how would you like to do this - irc, voice, ?
<Ursinha> lifeless, I'm fine with anything, do you think we need voice?
<Ursinha> as in too much information to type
<lifeless> dunno :)
<lifeless> ok
<lifeless> so - lets start
<lifeless> first thing is to assess whether there is enough to deploy - we don't want to overload the losas.
<lifeless> deploys take about an hour of wallclock time
<Ursinha> right
<lifeless> and a losa needs to spend ~ 5-10 minutes during that period kicking parts of it off
<lifeless> we do 10 commits a day, so 5 commits is a reasonable number to deploy (assuming nothing really urgent - if there was urgent stuff we wouldn't wait for 5:)
<lifeless> https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<lifeless> shows that we have nothing ready to deploy
<Ursinha> I see
<Ursinha> I'm pretty familiar with that report at least :P
<lifeless> but, if we check to ensure the first commit is safe to deploy
<lifeless> then we will have 5 revisions ready to go
<lifeless> so lets do that
<Ursinha> yep
<Ursinha> so do you ask the landing owner to qa that or you do that yourself?
<lifeless> if they are around, you might ask.
<lifeless> as it happens jcsackett is trying to get it qa'd now, but the losas will be gone.
<jcsackett> lifeless: :-P
<lifeless> Ursinha: one of the htings I look for is whether it can possibly be worse
<lifeless> Ursinha: if it can't be, then deploying it is ok :)
<lifeless> Ursinha: so, I'd visit https://bugs.launchpad.net/launchpad/+bug/687379 - the linked bug
<Ursinha> hehe
<_mup_> Bug #687379: Merge proposal diff overlay is bigger than the screen and can't be closed <confusing-ui> <javascript> <lp-bugs> <qa-needstesting> <regression> <Launchpad itself:Fix Committed by jcsackett> < https://launchpad.net/bugs/687379 >
<lifeless> Ursinha: click through to the merge proposal
<lifeless> and read whats changed, what the reviewer etc said, and consider the existing behaviour
 * jml is off
<Ursinha> bye jml
<Ursinha> lifeless, right
<lifeless> and in this case, it looks like:
<lifeless>  - the old behaviour is unusable
<lifeless>  - the change is tiny and can't break other things - its isolated the branchmergeproposal javascript
<lifeless> Ursinha: so I would mark this qa-ok immediately
<Ursinha>   hm, right
<Ursinha> we should consider the qa-deployable or something that just says the branch is deployable
<lifeless> (if losas were around doing a more thorough test like jcsackett tried to, would be awesome. This is a risk balance thing)
<Ursinha> right
<lifeless> Ursinha: yes, exactly
<lifeless> Ursinha: so, go ahead and qa-untestable or qa-ok the bug
 * Ursinha marks the bug qa-untestable
<lifeless> now we switch back to the -stable report
<Ursinha> lifeless, I'll poke the script to generate that now
<lifeless> I typically repeat this for all the bugs - qaing any that can be, assessing others for deployability
<lifeless> I talk to the engineers / team leads where it seems doubtful
<Ursinha> ok
<lifeless> e.g. distro mirror probing script changes are very hard to qa at all
<lifeless> then, yes - the script taking a long time to update is a real concern :)
<lifeless> you're privileged
<lifeless> Ursinha: then, once it updates, rev 12437 will be deployable
<lifeless> Ursinha: so stage two - prepare a deployment request on LaunchpadProductionStatus
<lifeless> Ursinha: what I do is:
<lifeless>  - open the page, click edit, use ctrl-f and search for 'requested stable depl' - that gets me to the right section
<lifeless>  - and we can see that last nights deploy didn't get moved correctly when it was done, the losas generally do that.
<lifeless> there is a big table higher up the page that contains the recently complete deployments
<lifeless> so we should move last nights deploy up to that table
<Ursinha> ok, looking
<Ursinha> lifeless, deployment is already in the big table
<Ursinha> requested stable deployments is empty
<Ursinha> ok, now
<lifeless> ah cool
<lifeless> my copy of the page was stale I guess
<lifeless> so, now, we edit the requested stable deployments section
<lifeless> this is very mechanical
<lifeless> we take the revno from the stable report
<lifeless> the 'can be deployed to production' one - second row at the top
<lifeless> for the bug #'s column
<lifeless> we go through each row in the deployment report and copy the Bug 12345 text
<_mup_> Bug #12345: isdn does not work, fritz avm (pnp?) <isdnutils (Ubuntu):Fix Released by doko> < https://launchpad.net/bugs/12345 >
<lifeless> change it to Bug:1234
<lifeless> which makes it a wiki ling
<lifeless> *link*
<lifeless> for requested by put your own name
<lifeless> for approved by 'qatagger'
<Ursinha> right, I took last one as example
<lifeless> for roll out to, put 'nodowntime'
<lifeless> and for comments put any special things that the revisions being deployed need
<Ursinha> right
<lifeless> I figure that out by reading all the revisions one at a time in the stable list, clicking through to the bug etc depending on how familiar I am with it
<lifeless> for instance, some things need a new feature flag added *before* the deploy starts
<lifeless> engineers will often arrange for that
<lifeless> or code so that it doesn't need to happen at the same time
<lifeless> but its not always like that, so a small check is good
<lifeless> save the page
<lifeless> ping a losa
<Ursinha> right
<lifeless> <wait an hour>
<lifeless> then click on all the bug links you put in the wiki page to open the bugs
<lifeless> for each one, if its fixed by a nodowntime deploy, toggle it to fix released
<lifeless> if it needs other actions
<lifeless> or if its a incremental branch
<lifeless> toggle it back to in progress
<Ursinha> ok. right
<lifeless> (or leave at fix committed if the remaining work is e.g. a deploy to germanium)
<lifeless> -done-
<lifeless> there are bugs open on qatagger for all the pain points :)
<Ursinha> ah, I know!
<Ursinha> about not being able to run it on demand, don't you have access to lpqateam user on devpad?
<lifeless> I tried to sudo to it the other day
<lifeless> didn't work
<lifeless> will try and sort that out when the losas aren't sprinting
<Ursinha> hm
<Ursinha> lifeless, file an RT asking for your user to be added to lpqateam on devpad
<lifeless> yeah, had done
<Ursinha> that should be enough
<Ursinha> ah, cool
<lifeless> late last year
<lifeless> need to find it, talk to is etc
<lifeless> next week :)
<lifeless> Ursinha: I'd love it if this were documented well enough that losas can operate it easil
<lifeless> y
<Ursinha> access to lpqateam?
<Ursinha> I can arrange that
<lifeless> access + instructions
<lifeless> how to deploy
<lifeless> how to do an on-demand run
<lifeless> where it logs
<Ursinha> I'll do that
<lifeless> \o/ thanks
<Ursinha> as you gave me such detailed information, it's half written already :)
<Ursinha> thanks for your time lifeless
<lifeless> Ursinha: oh, perhaps we've confused each other - I was talking about qatagger operation - I expect deployment requests to come from devs always.
<Ursinha> ah
<Ursinha> sure :)
<lifeless> Ursinha: although I'm happy if you're writing up the deployment request stuff too.
<lifeless> I thought it was already on the wiki, let me have a look
<lifeless> huh, no its not
<lifeless> downtime deploys are, but a bit sketchily
<Ursinha> lifeless, where?
<lifeless> mergeworkflow page
<jcsackett> is there a compelling reason to not move canonical.launchpad.validators to lp.services.validators or similar?
<thumper> jcsackett: no
<thumper> no reason at all
<thumper> jcsackett: they could though go to lp.app.validators
<thumper> which I think would be more fitting
<thumper> they are more a core part of the app than a service
<jcsackett> thumper: agreed, that's a much better place.
<thumper> leonardr: want to chat on mumble about the series work?
<leonardr> thumper, sure
<leonardr> let me push what i have
<leonardr> lp:~leonardr/launchpad/publish-distro-series-for-recipe
<lifeless> sinzui: around ?
<lifeless> sinzui: I've been thinking about your javascriptified batch navigator
<leonardr> https://bugs.launchpad.net/lazr.restful/+bug/723753
<_mup_> Bug #723753: defaulting to exporting new web service methods on all web service versions makes web service mistakes dangerous <lazr.restful:Triaged> < https://launchpad.net/bugs/723753 >
<leonardr> thumper:
<leonardr> you can say whether or not a field is published in the very first version
<leonardr> with exported(field, exported=False)
<leonardr> or exported(field, exported=True)
<leonardr> we could change it so that the value was the first version in which it's exported
<leonardr> or exported(field, exported="devel")
<leonardr> or add a new argument exported_as_of
<sinzui> lifeless: I just got back from picking up my children
<sinzui> I can talk
<lifeless> sinzui: cool
<lifeless> sinzui: I'm wondering about two things
<lifeless> admist eathquakes
<lifeless> sinzui: should we teach batchnavigator to handle sparse batches
<lifeless> (the first, last 40 thing that bugs do)
<lifeless> sinzui: and how much work would javascriptifying the batch navigator be
<sinzui> lifeless: I do not know if there is a lot of gain for that. We talked a little about that with google page results, but did not pursue it since google does not promise there really will be another batch
<sinzui> I am strongly in favor of ajaxing the batch navigator.
<lifeless> thirdly (monty python count to 2 :P)
<lifeless> there is this thing about 'only show the comment form if the user has read relevant messages'
<lifeless> where a proxy metric for relevant messages is pretty poor at the moment
<sinzui> That seems like a difficult assertion to make on a user
<lifeless> indeed
<lifeless> but thats why it doesn't show always
<sinzui> oh?
<lifeless> yeah
<lifeless> its so nice that bug one renders now
<lifeless> https://bugs.launchpad.net/ubuntu/+bug/1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> note that there is no comment form at the bottom of the page
<sinzui> I surmised.
<lifeless> if you click add a comment, it tries to load every comment
<sinzui> I worked on a site once that put a timer on the submit button to encourage users to read what was above it
<lifeless> but its nonsense to argue that this stops poor comments, because if there are 200 comments, what inexperienced user will read them all
<lifeless> let alone 400, or 1000
<lifeless> so I think I'm thinking alone these lines:
<lifeless>  - short term and long term
<lifeless> short term:
<lifeless>  - stop showing first 40, last 40, instead show first 100 in a standard batch navigator
<lifeless>  - if we're on the last batch, show the comment widget
<lifeless> medium term:
<lifeless>  - ajaxify the navigator, and unhide the comment widget at the end of the batch
<lifeless> long term:
<lifeless>  - some mechanism for really relevant / important comments (e.g. 'do not comment on this bug. File a new one if you have battery problems, no matter how similar you think it is') so that they are more visible/likely to be read.
<lifeless> [this may imply nested conversations, a voting system, or some other stuff]
<sinzui> Does someone really need to see the beginning  or middle of a conversation started 6 years ago. epic bugs needs a summary of the story so far, the current batch of comments (the last page I suppose) and provide a link to see the hirstorical record
<lifeless> sinzui: I don't know. Certainly we could start on the last page not the first and let people walk back
<lifeless> sinzui: for summary of story so far, folk should edit the description
<lifeless> sinzui: the historical record should be done via batching/incremental display - its just to big to do quickly in one request.
<sinzui> lifeless: yes that is why we can edit the description
<lifeless> sinzui: sounds like we're in agreement
<sinzui> lifeless: I think so. would like short term should start on the last page, but I think that could be medium term if it is costly to know we have more than a reasonable number of messages
<sinzui> That is a bad edit
<lifeless> we can do last page equally easily
<lifeless> I was showing a preconception by suggesting first 100 rather than last 100
<StevenK> lifeless: So staging is still broken?
<thumper> StevenK: mumble?
<StevenK> th	Aye, sec
<LPCIBot> Project db-devel build (391): STILL FAILING in 5 hr 24 min: https://hudson.wedontsleep.org/job/db-devel/391/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12436,
<LPCIBot> 12437 included.
<thumper> https://bugs.launchpad.net/launchpad/+bug/682516
<_mup_> Bug #682516: No link to recipes for packages <lp-code> <recipe> <Launchpad itself:Triaged> < https://launchpad.net/bugs/682516 >
<huwshimi> Morning
<leonardr> thumper: https://bugs.edge.launchpad.net/lazr.restful/+bug/723959
<_mup_> Bug #723959: Separate the floating "latest" version from the latest version-in-progress <lazr.restful:New> < https://launchpad.net/bugs/723959 >
<StevenK> Moaning
* abentley changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: yes, but heat updating is insane, but it's not *that* insane. Copies only update the head of the bugs that they close, then calculate the max heat of the source package.
<lifeless> wgrant: well
<lifeless> you might think so
<wgrant> It's still thoroughly ridiculous.
<lifeless> but it would be a lie
<lifeless> this is what they calculate:
<wgrant> But not utterly ridiculous.
<lifeless> Max(Bug.heat), Sum(Bug.heat), Count(Bug.id))
<wgrant> Right, but it's not recalculating each bug's heat.
<lifeless> no, but its querying it
<lifeless> this pages every distro series source package bug in the context in
<wgrant> Right.
<lifeless> this is that insane.
<lifeless> reading in 400000 bugs is not a good idea.
<lifeless> [yes its filtered to open status, but it may still be doing a seq scan. I'll need to check]
<wgrant> I'm of the opinion that bug heat in general is insane, and this is not significantly less sane than the rest of it.
<lifeless> bug heat has lots of room for improvement, thats true.
* barjavel.freenode.net changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<jml> build is broken.
<lifeless> jml: windmill timeout
<lifeless> build forced
<jml> lifeless: thanks.
<lifeless> meepe
<lifeless> bugs with 300 attachments
<sinzui> wgrant: mumble?
<lifeless> reference = 'Making output directory...\n'
<lifeless> actual = "Making output directory...\nException IndexError: IndexError('list index out of range',) in <bound method _LockWarner.__del__ of <bzrlib.lockable_files._LockWarner object at 0x1378e290>> ignored\n"
<lifeless> forced
<StevenK> lifeless: Latest build failure looked like Sphinx to me?
<StevenK> Ah
<jcsackett> wgrant: bug 723274; set it fix committed/released whatevs as appropriate. it's been assigned to you.
<_mup_> Bug #723274: Bug status changes to unknown when upstream status changes to dupe <bugwatch> <trivial> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/723274 >
<wgrant> jcsackett: Thanks.
<StevenK> wgrant: I can't see a recent landing from you that enabled logging commits?
<lifeless> 4.8 seocnds in structural subscriptions
<wgrant> StevenK: It was a side-effect of the ZTM refactor that I thought might affect a lot. But the test suite suggested that only a couple of tests didn't already pass a transaction in. Which might mean that we have an isolation issue.
<wgrant> StevenK: See the lib/canonical/launchpad/webapp/adapter.py diff in r12418
<StevenK> wgrant: I'm concerned that I tossed 2 branches at ec2 yesterday, with roughly 2 hours between them -- 1 passed completly and 1 failed
<wgrant> StevenK: You've seen this error on ec2?
<lifeless> gary_poster: hi
<StevenK> wgrant: Yes, it failed for one of the branches, but also fails locally
<lifeless> gary_poster: heads up on a bug that may be a side effect of, or related to your current project
<lifeless> https://bugs.launchpad.net/launchpad/+bug/723999
<_mup_> Bug #723999: structural subscriptions taking 4.8 seconds during nomination editing POST <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/723999 >
<wgrant> StevenK: Was the test order different between the success and the failure?
<lifeless> wgrant: do you have a branch working on bugtask:+index person late evaluation ?
<StevenK> wgrant: Yes
<wgrant> lifeless: I have a fix for the one-person-query-per-visible-comment one.
<wgrant> I was going to put more in there. But maybe I should just land it.
<wgrant> StevenK: What changed? Everything?
<lifeless> wgrant: just land it, or I'll be colliding with you
<StevenK> wgrant: "Some". I'm checking with a naive diff, and that has 400 lines different
<wgrant> lifeless: If you have a related branch, you might as well take my 6 character fix.
<lifeless> wgrant: paste it somewhere then
<lifeless> or push o rwhatever
<lifeless> bbiab, lunch breaking
<wgrant> lifeless: http://paste.ubuntu.com/571411/
<gary_poster> lifeless, ack, thanks for heads up.  It is certainly related one way or another, and I'll add the bug to the story and to the kanban board.
<lifeless> gary_poster: thanks
<thumper> ec2 error ?!?! ImportError: No module named pqm.pqm_submit
<thumper> anyone know what is going on?
<jcsackett> thumper: oh thank goodness it's not just me.
<jcsackett> no idea what's going on, but i finally just threw a branch to ec2 test b/c ec2 land was failing with that msg.
<thumper> hmm...
 * thumper does ec2 test
<wgrant> Do you have bzr-pqm installed?
<wgrant> There was a launchpad-dependencies update last night which may have gone fairly wrong.
<thumper> wgrant: I do
<jcsackett> me too.
<thumper> wgrant: it is the ec2 image
<StevenK> thumper: Which number did it use?
 * thumper looks
<thumper> Using machine image version 508
<wgrant> Oh, in the image?
<wgrant> There was a new one of those last night too.
<StevenK> My two landings yesterday used 507. Hmmm, perhaps 508 is busted
<wgrant> Remove jtv's ID from the list, I guess.
<jml> ok. now I'm really gone for the day
<lifeless> ciao
<jml> have a super wednesday
<wgrant> Night jml.
<jml> please fix all of the critical bugs.
<lifeless> I Can Has Fix ?
<jml> Thursday, I mean.
<lifeless> wgrant: -lol-
<wgrant> lifeless: Yes.
<wgrant> lifeless: Terribly difficult.
<lifeless> wgrant: did you spot load_people
* barjavel.freenode.net changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<StevenK> Bad server!
<wgrant> lifeless: I didn't.
<lifeless> wgrant: I suspect we're going to be coming back to it
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> wgrant: so this 6 character fix is easy
<lifeless> I wonder how many others are like this
<lifeless> this hsould drop bug 1 with all comments down to 200 queries.
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> theres some poor python scaling in there
<wgrant> lifeless: Huh? It's at like 490 for only 40 comments.
<wgrant> This will take it down to 450 for 40.
<wgrant> lifeless: Could you mentor https://code.launchpad.net/~jameinel/launchpad/loggerhead-disconnect-701329/+merge/48665?
<StevenK> wgrant: So, in regards to test_source_query_counts, I should re-ec2?
<lifeless> wgrant: there https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1880N2148
<lifeless> wgrant: 309 reps of j-i-t person lookup on comments=all
<lifeless> wgrant: thats 80 comments in your metric, not 40.
<wgrant> StevenK: Could you send me both subunit logs?
<wgrant> lifeless: Fair point.
<wgrant> 40 on each end.
<lifeless> if the commenters are distinct, 80 less
<lifeless> but they aren't
<wgrant> lifeless: Note that lots of those people are not comment lookups, I suspect.
<StevenK> wgrant: wget http://people.canonical.com/~stevenk/{less-lazr-security-r12400.subunit.gz,link-recipe-on-ppa-page-r12407.subunit.gz}
<lifeless> wgrant: select count(distinct message.owner) from bugmessage,message where bug=1 and bugmessage.message=message.id;
<lifeless>  count
<lifeless> -------
<lifeless>    535
<wgrant> Hmm. That's odd.
<wgrant> No manual entry for subunit-diff
<wgrant> Gr.
<StevenK> steven@liquified:~% subunit-diff --help
<StevenK> steven@liquified:~%
<wgrant> Yes.
<lifeless> thats jelmers perl special
<StevenK> And lifeless doesn't believe me when I tell him the subunit tools are as user-friendly as git
<lifeless> I would like it to be done in python
<lifeless> jelmer: ^ confusion ftw
<lifeless> however its better to have it than not
<wgrant> True.
<wgrant> Didn't do exactly what I wanted, but was still handy.
<jelmer> lifeless: I might redo it in python at some point
<StevenK> jelmer: Or write some help!
<StevenK> Or some *roff!
<jelmer> I am looking forward to the day I can write more *roff
#launchpad-dev 2011-02-24
<wgrant> StevenK: The only ordering differences there are doctests :(
<wgrant> StevenK: I wonder if the value already fluctuated, and the test had a buffer built in that my extra line exceeds.
<wgrant> StevenK: One way to test that is to decrease the limit by a few and throw it at ec2 two or three times, and see what values come back and which queries differ.
<lifeless> interesting
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1880F2434
<lifeless> 2.5 seconds sql
<lifeless> 12 python
<lifeless> ><
<wgrant> lifeless: You really need to acquire some LOSAs next week.
<lifeless> it will help
<lifeless> hmm, thats what I was going to do.
<wgrant> lifeless: Did you see my review poke?
<lifeless> no
<wgrant> https://code.launchpad.net/~jameinel/launchpad/loggerhead-disconnect-701329/+merge/48665
<wgrant> It is our friendly 16k OOPS bug.
<lifeless> done
<lifeless> we need a bug, if there isn't one, about the lack of timeout checking on that exception check.
* barjavel.freenode.net changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: Because haproxy will kill the connection after a while?
<lifeless> wgrant: that or users yes
<lifeless> basically we don't have the timeline trigger we do in the zope stack
<wgrant> Right.
<lifeless> so if the client goes away and we render at 8 seconds in
<lifeless> we'll hit a socket error rather than generating a timeout
<lifeless> we should record them as oopses, as timeouts
<StevenK> Er, wasn't the entire point of the branch to *stop* oops on socket errors?
<lifeless> yes, but we shouldn't stop them all
<lifeless> its better to land this branch now
<lifeless> but we *know* we're discarding valid, important oopses as a result. Its overcorrecting.
<StevenK> So, the 16k oops aren't all the haproxy ping?
<lifeless> theres two different exception times
<lifeless> one is haproxy, a few hundred a day are not haproxy
<lifeless> they are all going to go when this lands
<StevenK> wgrant: So subunit-diff helped, or you got distracted?
<wgrant> StevenK: I used subunit-ls then normal diff.
<wgrant> StevenK: There are no significant non-doctest order changes.
<StevenK> wgrant: Which means toss it through ec2 again?
<wgrant> StevenK: Maybe not.
<wgrant> Checking something locally.
<wgrant> StevenK: Try running the whole test_archive_packages locally.
<wgrant> It seems to succeed.
<wgrant> Can you confirm?
<StevenK> wgrant: YEs
<wgrant> StevenK: Throw it at ec2 again, and lp-land if it fails again.
<StevenK> So .... yay
<wgrant> I can debug it locally from here.
<lifeless> wgrant: had you done a test run with your eager loading person patch
<timrc> StevenK: Hey... I followed your example in https://bugs.launchpad.net/launchpad/+bug/440652/comments/4 using changing out values 'ppa-foo' and 'edge in login_with() and my e-mail for getByEmail().  When I do a dir(me), I do not see createPPA... was wondering if you may have an idea why
<_mup_> Bug #440652: Allow creation of PPAs via API <api> <bad-commit-11820> <lp-soyuz> <oem-services> <ppa> <qa-ok> <Launchpad itself:Fix Released by stevenk> < https://launchpad.net/bugs/440652 >
<wgrant> lifeless: No.
<wgrant> StevenK: Actually, if it fails ec2 again and I haven't landed a fix yet, increase the limit in the test by 1 and lp-land.
<wgrant> Just to be safe.
<lifeless> wgrant: ah. Well, I may break ec2 then
<wgrant> lifeless: Oh?
<wgrant> It's very unlikely to break anything...
<lifeless> adds another query to bug page loading
<lifeless> some of the scaling tests are exactly on the minimum query count
<StevenK> timrc: Having a look
<wgrant> lifeless: Only if there are no comments.
<wgrant> No comments with people other than the reporter, I guess.
<wgrant> StevenK: Ah, I see the issue.
<lifeless> it doesn't discard things in the storm cache from the queyr
<lifeless> so it will query them all
<StevenK> wgrant: With test_source_query_counts?
<wgrant> StevenK: Yes.
<wgrant> StevenK: It was already fluctuating, and my change made it exceed the buffer.
<wgrant> StevenK: The first webservice test does a SELECT secret FROM secret
<wgrant> Subsequent ones do not.
<StevenK> I did see that in the traceback.
<wgrant> Will fix properly after lunch -- increment the buffer and lp-land for now.
<StevenK> wgrant: The branch has been in ec2 for 10 minutes :-(
<wgrant> Ah.
<wgrant> In happier news, the mirror prober has now shut up.
<wgrant> In less happy news, lucid_db_lp has librarian-broken again
<StevenK> wgrant: I can kill the instance with fire, bump the 42 to 44(?) and lp-land
<wgrant> StevenK: 43
<StevenK> Or I can just wait
<wgrant> But yes, that's a good plan.
<wgrant> My change added (626, 626, 'SQL-nostore', 'Transaction completed, status: Active') to the end.
<wgrant> That's all.
<StevenK> That's a COMMIT or actually something else?
 * thumper takes a break
<wgrant> I am not entirely sure.
<wgrant> Normally it says something like "Committed" or "Doomed"
<StevenK> wgrant: Mind you, the branch had 3 other failures, which I've fixed, so not sure that impacts your "use lp-land" comment
<wgrant> Ah.
<StevenK> timrc: This is strange. The production API doesn't mention createPPA at all.
<StevenK> But qastaging does
<wgrant> StevenK: I see createPPA in lpnet's apidoc...
<StevenK> wgrant: Looking at my api.launchpad.net cached WADL, I can't see it
<wgrant> StevenK: What if you move the cached WADL away?
<wgrant> Or use devel instead?
<StevenK> timrc: Are you using edge or production?
<StevenK> wgrant: Still happy for me to kill the instance and lp-land directly?
<wgrant> StevenK: If you've fixed the other failures and incremented the value, sure.
<StevenK> wgrant: http://pastebin.ubuntu.com/571446/
<wgrant> StevenK: If it passes on its own locally, go for it.
<StevenK> It does, pushing and landing
<wgrant> Thanks.
<wgrant> I will fix the test infrastructure to clear the session machinery's secret cache, I think.
<StevenK> Rargh, lp-land uses edge
<StevenK> EDGE!
 * wgrant â lunch
<timrc> wgrant, StevenK: my apologies, I had to dip out
<timrc> StevenK: I was testing with edge
<StevenK> timrc: I had to delete my launchpadlib cache, but using production I could see createPPA()
<timrc> StevenK: okay, let me give it a try
 * cody-somerville wondered if it was a cache issue.
<StevenK> cody-somerville: I was hoping it wasn't, and my cache of the production WADL was only 24 hours old
<cody-somerville> interesting
<StevenK> ITYM "disturbing"
<timrc> StevenK: okay, switching to production, and it worked as expected
<timrc> StevenK: and clearing my cache for edge also worked
<timrc> StevenK, wgrant: thanks!
<StevenK> timrc: You're welcome
<lifeless> hmm
<lifeless> is there an ISourcePackage adapter for IDistributionSourcePackage
<lifeless> wgrant: btw
<lifeless> test_source_query_counts - the expected_count += 1 due to getCurrentSourceReleases can be trivially fixed now.
<StevenK> It can?
<lifeless> yes
<lifeless> StevenK: look at the implementation of DistroSeries.getCurrentSourceReleases
<wgrant> lifeless: How does it make sense to adapt a DistributionSourcePackage to an ISourcePackage?
<lifeless> get the current series sourcepackage
<lifeless> https://bugs.launchpad.net/launchpad/+bug/279513
<_mup_> Bug #279513: Distribution source package tooltip in bugtask table shows most recent SPPH in any series <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/279513 >
<wgrant> Ah.
<lifeless> bunch of code in dsp would be simpler if it indirected to ISourcePackage(self).<same>
<lifeless> for instance
<wgrant> If you add that adapter then you have to rename it to DistroSeriesSourcePackage.
<lifeless> bah
<lifeless> whats in a name
<lifeless> also the sample data is bong
<lifeless> hoary has no publications
<wgrant> Yes, the sample data is really bad.
<wgrant>   44 IntegrityError: duplicate key value violates unique constraint "tm__potmsgset__language__shared__ubuntu__key"
<wgrant> Not ideal.
<lifeless> oops?
<wgrant> OOPS-1880A1578
<wgrant> session_dev=# SELECT secret FROM secret;
<wgrant>           secret
<wgrant> --------------------------
<wgrant>  thooper thpetial theqwet
<wgrant> Hah
<lifeless> oh noes
<lifeless> lp has been hacked
<wgrant> I know!
<wgrant> lifeless: lib/canonical/launchpad/webapp/session.py, _get_secret is a test isolation issue.
<wgrant> I am wondering about a good place to clear that.
<lifeless> sec
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-279513/+merge/51063
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: Looking.
<lifeless> so re the secret
<lifeless> I've worked around it by doing one request before actual test requiests
<lifeless> thats a bit fugly
<wgrant> I am tempted to clear it in getUserBrowser.
<wgrant> Well, setupBrowser.
<lifeless> forcing it on every time?
<wgrant> Empty the cache so it is queried for every test.
<lifeless> won't help folk doing 2 queries per test
<lifeless> what about populating it when the layer is brought up
<wgrant> But it will isolate the tests.
<wgrant> I guess.
<wgrant> Yeah, that might work.
<wgrant> Just need to be sure the DB is in place at that point.
<LPCIBot> Project db-devel build (392): STILL FAILING in 5 hr 23 min: https://hudson.wedontsleep.org/job/db-devel/392/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12438,
<LPCIBot> 12439 included.
<thumper> wgrant, wallyworld: update script evolved to http://pastebin.ubuntu.com/571485/
<wallyworld> thumper: cool. otp. will look soon
<wgrant> thumper: e.name vs e.new_value?
<thumper> e.name is the name of the field updated
<thumper> e.new_value is the value of the field
<wgrant> Oh, right, separate events.
<thumper> wgrant: yup, very simple update methods
<wgrant> Looks really great.
<wgrant> Please deploy it EVERYWHERE>
<wgrant> Now.
<thumper> wgrant: would you like to review the interesting change?
<thumper> wgrant: like Javascript?
<wgrant> thumper: I would.
<wgrant> I don't know YUI well at all, but I need to learn it.
<thumper> wgrant: well... there are three branches here
<thumper> wgrant: I've self approved the first one
<wgrant> I saw that.
<thumper> it changed LP.client.cache -> LP.cache
<thumper> and LP.client.links -> LP.links
<thumper> and that was it
<thumper> the second makes LP.client methods a YUI module
<thumper> I was about to self approve, but if you want to take a look, it is mind-numbingly boring
<thumper> https://code.launchpad.net/~thumper/launchpad/lp-client-yui-module/+merge/51038
<thumper> and 2.2k lines
<wgrant> I already had that open, and it is 1.5k lines... has it grown?
<wgrant> Hah, it has.
<thumper> https://code.launchpad.net/~thumper/launchpad/client-cache-sync/+merge/51049
<thumper> that is the intersting one
<LPCIBot> Project devel build (469): FAILURE in 5 hr 21 min: https://hudson.wedontsleep.org/job/devel/469/
<LPCIBot> Launchpad Patch Queue Manager: [r=jcsackett,sinzui][no-qa] Add a test to prevent new Sphinx errors
<thumper> wgrant: where is that bug of yours that says the base branch and deb version get out of date?
<wgrant> thumper: Bug #721064
<_mup_> Bug #721064: Inline recipe text editor doesn't update the "Debian version" field on the page <recipe> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/721064 >
<wgrant> I actually thought you'd just laugh and ignore it.
<wgrant> I didn't imagine a fix was in progress.
<thumper> :)
<wallyworld> thumper: looks ok on first read. i think i need to actually wire something up though and experiment to get a full feel for it
<wgrant> lifeless: These structural subscription queries upset me.
<wgrant> Just looking at some checkwatches OOPSes.
<lifeless> there is a bug now
<wgrant> SQL queries that are 323 lines long: Australia says no.
<lifeless> oh good
<lifeless> question:+index is back
<thumper> wgrant: I've added a description to the MP
<wgrant> Hmm.
<wgrant> That LockWarner thing just killed devel for a second time.
<wgrant> Maybe not spurious.
<wgrant> Well, it seems to at least occur there with far higher frequency than the other places it shows up.
<wgrant> I think we should disable that test.
<wgrant> lifeless: Do you know why NullBugTask still exists?
<wgrant> thumper: hm, why do you redirect on location change?
<lifeless> its high frequency
<lifeless> we should fix it
<thumper> wgrant: ahh... whut?
<lifeless> its the same as the oopses we're seeing I think
<wgrant> thumper: erm, I mean, why do you reload the page when the web_link changes?
<lifeless> wgrant: nullbugtask exists for when someone puts in an invalid context for a bug
<wgrant> lifeless: I know that's why it existed.
<wgrant> But it doesn't explain why it still exists.
<thumper> wgrant: because the current page is now invalid
<thumper> wgrant: it doesn't exist any more
<thumper> wgrant: it would be a 404
<lifeless> wgrant: whats more interesting is 'why isn't the NBT redirect working'
<thumper> wgrant: for example, if I change the owner of a recipe
<thumper> wgrant: the recipe now has a new url
<lifeless> I think this may be related to it being a source package, or something
<wgrant> lifeless: There is one case left.
<wgrant> lifeless: It is.
<wgrant> lifeless: It checks if it's reported in the distribution at all.
<wgrant> If it is, it's OK.
<wgrant> I am wondering why that is OK.
<lifeless> I think this is oversight
<wgrant> I hope so.
<wgrant> I meant to check with Bugs people about it.
<lifeless> gmb muttered words to the effect of 'wtf' when I opened the NBT timeout bug
<wgrant> thumper: Can't we update things that reference the base URL and push a new history entry?
<wgrant> The AJAX product switcher refreshing the page is really disconcerting.
<wgrant> lifeless: Ah, good.;
<wgrant> lifeless: 'tis easily destroyed, then.
<thumper> wgrant: I'm not sure... can we?
<wgrant> Will you murder it?
<lifeless> I don't believe you'd meet any objections to nuking it
<thumper> wgrant: in a reasonable way?
<lifeless> wgrant: no, I'm staying on the top two timeouts
<wgrant> thumper: github does it reasonably nicely.
<wgrant> Twitter does it awfully badly.
<lifeless> I'm going to poke at Branch:+index for a bit, it seems to be running into grief
<thumper> wgrant: I'll take a look
<wgrant> thumper: See https://github.com/jquery/jquery
<wgrant> Click on a directory there.
<wgrant> It changes the URL without refreshing the page.
<thumper> wgrant: it is refreshing the page, or at least firebug thinks so
<wgrant> thumper: Firefox 3?
<thumper> yep
<wgrant> I guess that doesn't support some new HTML5 API that it needs :(
<wgrant> Sad.
<StevenK> lifeless: Your forced build failed in exactly the same way ...
<lifeless> StevenK: no
<lifeless> StevenK: db-devel failed
<lifeless> then devel failed
<lifeless> devels previous failure was a different error
 * lifeless is pretty but not 100% sure that this is what happened
 * StevenK checks again
<thumper> I am so finishing a little early today
<thumper> 6am call, and working through lunch
<lifeless> yeah
<lifeless> now if only that call was at 7am
<StevenK> Yes, I'm right. Both 653 and 654 both failed with the _LockWarner garbage in the middle of the Sphnix test.
<wgrant> That's what I thought.
<lifeless> ah well
<lifeless> 3rd time lucky
<StevenK> And if it fails then you might admit there is an issue?
<lifeless> if you want to disable it I think thats a reasonable precaution
<lifeless> I've already agreed to that, no need to get snarky
<lifeless> all intermittent failures are issues
<lifeless> there is a bug open on this one
<lifeless> marked critical
<StevenK> lifeless: You already said I was wrong once ... :-)
<lifeless> StevenK: I thought the identical failure was on db-devel
<lifeless> hmm
<lifeless> wallyworld: do you show bugs in the linked mps you did on branch indexes ?
<StevenK> No, db-devel was test_404 and test_old librarian breakage
<lifeless> StevenK: thanks for digging
<StevenK> In fact, Hudson is failing builds in exactly the same way.
<StevenK> actual = "Making output directory...\nException IndexError: IndexError('list index out of range',) in <bound method _LockWarner.__del__ of <bzrlib.lockable_files._LockWarner object at 0x10ea8790>> ignored\n"
<wgrant> Huh.
 * thumper walks away for today
<thumper> branches are up for review
<wallyworld> lifeless: yes.
<StevenK> lifeless: I thought we had upgraded all of our sourcecode branches? http://pastebin.ubuntu.com/571505/
<wallyworld> lifeless: there are a couple of minor enhancements/issue that martin raised. can't recall what they are off hand
<wgrant> StevenK: Your local branch is an old format.
<lifeless> wallyworld: https://bugs.launchpad.net/launchpad/+bug/710685
<StevenK> steven@liquified:.../sourcecode/loggerhead% bzr info
<StevenK> Standalone tree (format: unnamed)
<_mup_> Bug #710685: Branch:+index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/710685 >
<StevenK> Hm. Orsum
<lifeless> wallyworld: *300* bug lookups on Branch:+index pages.
 * wallyworld looks
<lifeless> wallyworld: I suspect, totally without evidence, your patch.
<wallyworld> lifeless: could be me for sure. it was a while ago, but there already was one round of optimisation to avoid the 1+N query problem. could be something else as well. i'll look at it once i finish my current branch
<lifeless> wallyworld: you're on maintenance next week
<lifeless> wallyworld: if I were you, I'd stay focused on recipes this week to leave them as awesome as possible
<StevenK> Hm, what does format: unnamed in bzr info mean?
<wallyworld> lifeless: yep. wasn't sure if you wanted it to wait til next week but that suits me fine
<lifeless> wallyworld: I was running it up the flagpole to see if it triggered an 'oh yes' or a 'hmm didn't think of that' or a 'its been changed since me cause I checked for that'
<wallyworld> lifeless: nothing pops to mind, since work was done to reduce the query count at the time of implementation and i *thought* it was left in a sensible state. something might jump out when i have another look though
<lifeless> I've just generated OOPS-1881C305 on https://code.launchpad.net/++oops++/~bzr-pqm/bzr/bzr.dev to get an overview
<lifeless> that branch has no linked bugs
<wgrant> Oh.
<wgrant> OH
<wgrant> FFS
 * wgrant headdesks repeatedly.
<wgrant> REPEATEDLY
<wgrant> So. Fucking. Obvious.
<wgrant> Garrr
<wgrant> The librarian mystery is not so mysterious.
<lifeless> wgrant: 'sup
<StevenK> wgrant: Is this test_404 and such?
<wgrant> StevenK: yes.
<StevenK> SHARE!
<wgrant> I will have a branch in a sec.
<wgrant> You don't want to see it.
<wgrant> You will never forgive yourself for not seeing it earlier.
<StevenK> Hahaha
<wgrant> Now I just have to kick Windmill in the face a few times.
<StevenK> Only if you have knives on the bottom of your shoes
<StevenK> lifeless: I can't see this bug about Sphinx
<wgrant> There's a bug about the _LockWarner issue in general.
<wgrant> Sphinx just exposes it better.
<lifeless> bug 721166
<_mup_> Bug #721166: Tests sometimes fail on EC2 due to _LockWarner garbage <build-infrastructure> <spurious-test-failure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/721166 >
<StevenK> I'm strongly leaning towards disabling test_docs_build_without_error
<wgrant> If it fails again, do it,.
<wgrant> It is not Sphinx's issue, but Sphinx seems to expose it reliably... except locally.
<wgrant> ... and except on ec2.
<StevenK> It's failed twice on BuildBot and once on Hudson
<wgrant> Kill it.
<StevenK> -    def test_docs_build_without_error(self):
<StevenK> +    def DISABLED_test_docs_build_without_error(self):
<StevenK> wgrant: Tossing to PQM with "[testfix][rs=stevenk,wgrant] Disable building the Sphinx docs due to spurious failures."
<wgrant> StevenK: You're disabling the *test*.
<wgrant> Not building the docs.
<wgrant> Also, we're not actually in testfix atm, are we?
<wgrant> So you'll need QA tags.
<StevenK> [rs=stevenk,wgrant][no-qa] Disable the Sphinx docs test due to spurious failures.
<wgrant> StevenK: Looks good.
<StevenK> Tossing
<StevenK> wgrant: Where's this librarian branch?
<wgrant> StevenK: Generating a diff.;
<wgrant> Description of the Change
<wgrant> So it turns out that str.replace()ing integers in a URL with a random port is not such a good idea.
<StevenK> *facepalm*
<wgrant> Codehosting, you suck.
<wgrant> I discounted months ago the idea that there was an obvious issue in the tests themselves, because a subclass worked fine...
 * StevenK discovers the branch
<wgrant> Diff is mostly there now.
<StevenK> I note the branch scanner hasn't run over it yet
 * wgrant goes logdiving.
<StevenK> It has now
<wgrant> The diff took 6 minutes :/
<wgrant> lifeless: Still around?
<StevenK> wgrant: Why a top level function?
<wgrant> StevenK: Because there is no benefit in making it a method.
<wgrant> We are not Java.
<StevenK> Next question, why weren't they failing locally? :-)
<lifeless> wgrant: sup
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/fix-librarian-spuriousity/+merge/51068
<StevenK> lifeless: I've already reviewed, so you need to review my review
<lifeless> so, when the port == the alias, boom
<lifeless> yes?
<wgrant> lifeless: When the alias is any substring of the port.
<lifeless> right
<wgrant> I am guessing the ec2 and hudson pids rarely get high enough.
<wgrant> And locally the tests are rarely run.
<StevenK> OH, it's from the pid?
<wgrant> Hm, except it's ports, not pids.
<wgrant> Fail.
<wgrant> I wonder what the algorithm is.
<StevenK> It does seem to hit buildbot more
<lifeless> you need a low ephemeral port #
<lifeless> and a high lfa id
<StevenK> Right
<StevenK> Laaaaaaaaaaand it
<wgrant> Thanks.
<wgrant> If only we could restart test runs.
<lifeless> import smalltalk; smalltalk.start()
<lifeless> think of the good news
<lifeless> if I broke the test suite, we'll see those failures as well
<wgrant> What good news? There is still Windmill and _LockWarner.
<wgrant> Hah
<lifeless> now to figure out why @adapter(IBeforeTraversedEvent) isn't triggering
<wgrant> What are you doing and why?
<lifeless> ++profile++ on staging
<lifeless> we'll miss very early setup
<lifeless> oh wow
<lifeless> we call set_developer_in_launchbag_before_traversal a few times too many
<lifeless> (becaues the event doesn't do what folk think it does)
<wgrant> Is it before every publishTraverse?
<lifeless> its before each step of traversal, yes
<lifeless> I'm going to be using the same event, but explictly handle that case
<lifeless> its easier than patching zope
<wgrant> I suppose I should exterminate the remaining third of the checkwatches OOPSes.
<StevenK> How many OOPSes is that?
<lifeless> 3 brazillion
<wgrant> It varies.
<wgrant> 1000-7000 a day.
<wgrant> 15000 in the last three days.
<wgrant> 9000 of which are fixed.
<StevenK> staging doesn't love me :-(
<wgrant> The update failed due to some slony stuff.
<wgrant> Speaking of slony.
<wgrant>         if not is_ca_available():
<wgrant>             raise LayerInvariantError(
<wgrant>                 "Component architecture not loaded or totally screwed"
<wgrant>                 )
<lifeless> wgrant: speaking of fish, monitors ?
<wgrant> lifeless: The slony comment was related to the arrival of stub, not the CA screwedness.
<wgrant> No fish, sadly.
<stub> slony is a computer program, not me typing furiously
<lifeless> a small patch is a good patch
<wgrant> stub: Anyway, do you know why the staging update exploded yesterday?
<stub> (Replication lag is spiking. Make stub another coffee!)
<wgrant> Some slony error.
<wgrant> That is not entirely obvious.
<wgrant> stub: Ahh, I noticed there was multi-minute lag on some stuff an hour ago.
<stub> the slony error will be that it obfuscates the real error most likely
<wgrant> The forwarded email didn't look like the usual patching failure.
<wgrant> 2011-02-23 07:35:19 INFO    Waiting for cluster to sync.
<wgrant> /tmp/slonik5hZrbc.sk:27: timeout exceeded while waiting for event confirmation
<wgrant> 2011-02-23 07:45:20 ERROR   slonik script failed
<wgrant> Not a patching issue :/
<stub> Yup
<wgrant> Hmm, one long transaction.
<wgrant> Except it wasn't long.
<stub> Its sad when I consider an automated system that works most of the time a successful improvement
<wgrant> Ahh, pofilestats was running.
<wgrant> That could do it.
<wgrant> It probably still is.
<wgrant> That's quite a serious transaction it has there.
<wgrant> Are qastaging and staging in the same replication thingy?
<StevenK> soren: Do you know what plugins the OpenStack Hudson is using? I'm interested in the merging one that posted to the MP
<lifeless> mtaylor: ^
<lifeless> wgrant: no
<lifeless> wgrant: qas has no replica
<lifeless> wgrant: staging has one replica
<StevenK> Then we can replace both PQM and BuildBot with Jenkins!
<wgrant> StevenK: Once we unbreak windmill.
<wgrant> Which could be approximately infinitely far in the future.
<StevenK> Windmill makes me sad.
<wgrant> It also doesn't work at all with Firefox 4.
<wgrant> StevenK:
<wgrant> Test-module import failures:
<wgrant> Module: lp.scripts.tests.test_sphinxdocs
<wgrant> TypeError: Module lp.scripts.tests.test_sphinxdocs does not define any tests
<wgrant> Is that going to break anything?
<StevenK> Oh, DAMN IT
<StevenK> TestCase, I hate you
<StevenK> Yes, it is
<wgrant> That's a little stupid.
<lifeless> \o/
<lifeless> stub: just organising dinner; call after?
<stub> Sure.
<wgrant> Hmmm.
<wgrant> I want to increment this rollout request.
<stub> Caffeinated. Still stuck with silly sleep cycle though.
<lifeless> wgrant: do so
<wgrant> lifeless: But that will make jml and Ursinha sad :P
<lifeless> wgrant: it was a learning exercise for Ursinha - and she has learnt
<lifeless> wgrant: I would increment yours, if you were asleep and more was ready
<lifeless> just put Ursula, william as the requestors
<wgrant> Hopefully we can do two tonight.
<wgrant> Since there are 9 revs stuck in buildbot.
 * StevenK waits for bzr
 * wgrant cuts off shipit's fingers as they go near anywhere except lp.shipit
<StevenK> Oh?
<wgrant> Removing the canonical.widgets imports, mainly.
 * StevenK cheats
<wgrant> That's cheating.
<StevenK> Uh, wgrant, I think the import facist hates you
<StevenK> ** 1 import policy violations **
<StevenK> There were 1 imports of names not appearing in the __all__.
<StevenK> You should not import UnknownRemoteValueError from lp.bugs.externalbugtracker:
<wgrant> It does, yes.
<StevenK>     lp.bugs.scripts.checkwatches.remotebugupdater
<wgrant> I noticed that earlier and was going to slip a fix into my query counts branch.
<mtaylor> StevenK: aroo?
<StevenK> mtaylor: O hai
<mtaylor> hey - so, we're just having hudson trigger tarmac
<mtaylor> I _REALLY_ want to finish some work on plugins for jenkins to allow it to not need tarmac but instead directly interface with launchpad
<mtaylor> but that stalls every time I try to make progress onit
<StevenK> mtaylor: How? I was strugging to figure out how to get Jenkins to do anything after a build finished
<StevenK> wgrant: I can't test this cheat with that import error :-(
<mtaylor> StevenK: there are post-build triggers
<wgrant> StevenK: Hm? That's not an error...
<lifeless> \o/ working
<lifeless> stub: skype?
<StevenK> mtaylor: I see them, but there doesn't seem to be a "Go and run this script for me" one
<mtaylor> StevenK: I _think_ you can do that, but I don't do that so I don't know
<StevenK> mtaylor: Also, does that mean you have Jenkins testing random branches?
<lifeless> StevenK: thats easy; parameterised :)
<StevenK> wgrant: http://pastebin.ubuntu.com/571547/
<wgrant> StevenK: Success.
<mtaylor> StevenK: I do have jenkins set up with parameterized builders both for drizzle and openstack
<mtaylor> StevenK: although the openstack one is less useful since we're just using jenkins to trigger tarmac for openstack. for drizzle, the param-build jobs are essential and excellent
<mtaylor> lifeless: have I mentioned how sad I am that I havne't gotten those jenkins plugins done yet?
<wgrant> production deployments don't respect the revisions in sourcedeps.conf, do they?
<lifeless> mtaylor: you have
<StevenK> wgrant: My cheating makes me sad: http://pastebin.ubuntu.com/571557/
<wgrant> StevenK: Can't you just rename the file?
<StevenK> oh, disabled_test_... ?
<wgrant> .py.goaway
<wgrant> Or maybe disabled_test...
<wgrant> Not sure if that works.
<wgrant> Preferably the prefix.
<lifeless> uhm
<lifeless> prefix preferred for sure
<wgrant> For doctests we suffix.
<StevenK> Fix pushed into PQM
<StevenK> It would be nice if the deployment report also said how far behind qas is, which might hint at a buildbot stall or os
<StevenK> s/os$/so/
<StevenK> Er, how far behind stable tip qas is
<wgrant> I'd really like the report to list all revisions in devel, split into what is on qas, what is waiting for deploy to qas, what is in buildbot, what is waiting for buildbot.
<wgrant> And somewhere listing in-progress ec2 runs would be handy.
<lifeless> lp:~lifeless/launchpad/profile
<lifeless> pushing now
<StevenK> wgrant: In-progress ec2 runs is *hard*
<lifeless> stub: https://code.launchpad.net/~lifeless/lp-production-configs/single-threaded/+merge/41554 has a script to add configs
<lifeless> stub: https://bugs.launchpad.net/launchpad-project/+bugs?field.tag=timeout
<lifeless> stub: if you wanted to review https://code.launchpad.net/~lifeless/launchpad/profile/+merge/51078 that might be nice
 * lifeless goes eats
<wgrant> Interesting.
<wgrant> I can set the importance of ShipIt bugs, but can't mark them Triaged.
<lifeless> backs
<lifeless> nomification vacuumed
<wgrant> Excellent.
<lifeless> stub: hey; the staging and qastaging restores
<lifeless> stub: can they inject a feature flag ?
<wgrant> Can't we add a server:qastaging?
<lifeless> wgrant: still wouldn't want that on production
<lifeless> in event of a mishap it would be bad
<stub> Sure. Its just a shell script. It could also be injected in database/replication/Makefile but for a hack just for staging the shell script is a better fit.
<lifeless> wgrant: oh, and scopes have no boolean operators
<lifeless> wgrant: so it wouldn't help.
<lifeless> stub: I'd like to get 'profiling.enabled team:launchpad 0 on' added when the staging and qastaging systems come online
<stub> lp:~canonical-losas/lp-staging-scripts/trunk is where the staging restore script lives
<lifeless> stub: we also should add those to the current databases
<stub> Do you want these flags added via config file items, or just hard coded in the rebuild scripts?
<stub> scrap that... config file items would need to handle removal too and becomes a much bigger problem...
<lifeless> stub: exactly, just added in the rebuilt scripts
<lifeless> and manually nowish to the existing [qa]staging servers
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/everything-into-lp.shipit/+merge/51080
<wgrant> Thanks.
<stub> We have a nice web ui for adding feature flags to live systems somewhere...
<wgrant> stub: /+feature-rules?
<stub> This script is a piece of poo
<wgrant> Wiiiiindmiiiiiiiiiiiiiiil
<stub> lifeless: https://code.launchpad.net/~stub/lp-staging-scripts/devel/+merge/51082
<lifeless> stub: will that affect qastaging as well?
<LPCIBot> Project devel build (470): STILL FAILING in 4 hr 49 min: https://hudson.wedontsleep.org/job/devel/470/
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless, wgrant][ui=sinzui][bug=712773, 722344] Show which recipe,
<LPCIBot> its build and the requester are responsible for the creation of the
<LPCIBot> publishing record in the detail of a PPA's +packages page.
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][bug=724013] Actually eager load message owners in
<LPCIBot> Bug._indexed_messages.
<LPCIBot> * Launchpad Patch Queue Manager: [r=stub][bug=722429] Eager load milestones used in bug search result
<LPCIBot> tables.
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][bug=592345,
<LPCIBot> 719288][incr] Several checkwatches OOPSes have been demoted to INFO
<LPCIBot> log entries.
<lifeless> stub: I've lowered the oauthnonce thing to low given its not making trouble at the moment; feel free to make it high or $whatever if I've assessed it wrongly
<soren> StevenK: That would be Tarmac.
<stub> lifeless: thats fine
<stub> lifeless: No. I'm not sure where the qastaging rebuild scripts live.
<wgrant> Is there a qastaging rebuild script?
<lifeless> there is a means
<lifeless> I don't know the rest
<lifeless> stub: could I beg a review from you - https://code.launchpad.net/~lifeless/launchpad/profile/+merge/51078
<lifeless> stub: I forgot to highlight https://dev.launchpad.net/LEP/OopsDisplay for a once over
<stub> k
<stub> lifeless: reviewed
<lifeless> thanks
<stub> lifeless: nice to have: short OOPS codes. If you are going for md5 for oops id, might as well just use uuid
 * stub tries to recall how well an md5 can be compressed
<lifeless> stub: uuids are ~ the same length
<lifeless> stub: we could just use the first N bits of (any) hash and be (probably) unique
<lifeless> e.g. if we're doing 6M in a month
<lifeless> and gcing at 30 days
<lifeless> then taking, oh, 40 bits, would be pretty unlikely to collide within the lifespan of the oops
 * stub wonders where his base.py disappeared too
<lifeless> it belongs to us
<lifeless> you are right though, uuids would also do the job, using libuuid or whathave you
<lifeless> (40 bits is 10 characters in hex - 0123456789
<lifeless> which is pretty short
<stub> lp.services.utils.compress_hash shrinks an md5 to 22 ascii characters (converts the number to base62)
<lifeless> cool
<stub> I guess that is fine given our oops prefixes are getting looong already
<lifeless> is that available outside the lp tree?
<stub> not yet, but it probably should since the base() function it uses is PSF licence and pulled from the Python Cookbook.
<lifeless> I want to write oops-repository as a lightweight project not needing the lp tree, so that other teams can deploy it sanely
<stub> yup
<mrevell> Hello
<stub> I suspect local rabbits on each server is overkill. I think we don't care about occasional lost OOPSes enough to warrent the extra complexity.
<lifeless> stub: I'm torn
<lifeless> on one hand, it is slightly more setup (only slightly - rabbit, durable queue, shovel - done.)
<stub> But then we have three moving parts to fail rather than one.
<lifeless> but less dependencies
<lifeless> if we do just one rabbit, we have more complex code
<lifeless> the appserver code would have more reason to be tolerant of that rabbit being down for maintenance (whereas with a federation we can just bring the local rabbit up before the appserver)
<lifeless> anyway, it won't change the code we write
<lifeless> the contract is 'speak to a rabbit' on both ends
<lifeless> as long as we can configure the rabbbit node to use, its flexible enough to run in either config without change
<stub> I'm used to more moving parts increasing downtime
<stub> Too many systems attempt to reduce downtime by creating redundancy and end up increasing it because there are more things that can screw up.
<lifeless> thats certainly possible!
<lifeless> we can afford to lose some oopses
<lifeless> I'd hesitate to lose hours though, in the event of (say) devpad failing
<stub> Of course, if casandra is distributed and never goes down we could have the appservers stuff OOPSes directly in there.
<lifeless> we could
<lifeless> I'm totally open to that as well
<lifeless> one advantage for u1 of rabbit is that they have a lot of network glitches, which cassandra wouldn't help with
<wgrant> Is that because they are partly on EC2?
<lifeless> entirely because of that
<stub> if we want short OOPS ids, systems could pull unique integers from rabbit (fed from some source maintaining the central counter)
<lifeless> stub: I don't particularly care about short OOPS ids - and having a central allocator would force an api-call rather than a fire and forget model
<lifeless> what would short oops ids help us with ?
<stub> lifeless: Making them citable. bug reports, irc conversations, even phone calls.
<lifeless> all but phone calls are copy and pastable
<stub> a script or appserver could pull a unique prefix and then use it to generate unique ids
<lifeless> for phone calls, once they are in the database, we can use the prefix logic git and hg do to let you cite just a unique prefix
<stub> They are copy and pastable, but not readable
<stub> Bug reports with 'See OOPS-X542327' vs. 'See OOPS-ACBD18DB4CC2F85CEDEF654FCCC4A4D8'
<stub> Much more space chewed up, much less readable == sucky ui
<lifeless> mmm
<lifeless> I see the point, but it doesn't feel very compelling
<lifeless> I am possibly overreacting to the current headaches
<stub> manually allocated prefixes suck cause people mess up (even if they are slightly nicer in that you can use them to identify the system the OOPS came from without looking at the content). But long ids suck too because of the sheer volume we are constantly wading through.
<lifeless> OOPS-68b329da9893e34099c7d8ad5cb9c940 - thats an actual md5sum
<lifeless> OOPS-1202CBB1234
<lifeless> thats our current oopses
<lifeless> so its about twice as long
<bigjools> nicely rounded down :)
<lifeless> OOPS-1202CBB1234OOPS-1202CBB1234
<lifeless> OOPS-68b329da9893e34099c7d8ad5cb9c940
<stub> 6 digit bug numbers suck too. Should have made the ids alphanumeric :)
<jtv> mwhudson: you still here by any chance?  Got a problem with the ec2 image, and heard you might know.
<jtv> WTF?  WTF?  WTFF?
<jtv> bigjools: /etc/apt/sources.liste
<jtv> sic
<bigjools> jtv: that pulsing  vein is getting bigger
<jtv> They are identical though
<bigjools> that sounds suboptimal
<jtv> But shouldn't there be a sources.list[.d] entry for the Launchpad PPA?
<bigjools> yes
<bigjools> I think
<bigjools> stub: if we add a ulimit on memory for the librarian, how big should it be?
<stub> Dunno. Guestimate by losa?
<bigjools> ok
<stub> If we spool uploads and/or downloads into RAM, fucking huge (and a bug report opened)
<mthaddon> stub: enough so that it's not using swap I guess
<bigjools> well it was at 15GB when it went awol
<stub> mthaddon: Sounds sane
<mthaddon> bigjools: we have mizuho configured with a ton of swap as when we didn't the process was dying before we could debug it, but that doesn't mean we *want* it to be using that much
<bigjools> indeed
<stub> It *should* be small, even with a dozen threads and handling large files.
<bigjools> I'm filing an RT to get a ulimit
<mthaddon> bigjools: you realise this is just going to mean the librarian process crashes when it hits the ulimit rather than crashing later?
<bigjools> mthaddon: yes - I am just doing what I've been asked to do :)
 * mthaddon nods
<bigjools> mthaddon: I don't think it crashed before BTW, it just went very slow
<bigjools> it's preferable to have it crash so you get an alert
<mthaddon> the problem we had before was it crashed before we could get debug info, which is why we changed it - it sounds like setting ulimit is going to mean it goes back to crashing before we can do any debugging
<bigjools> ok, you might want to put that on the RT
<mthaddon> k
<bigjools> jml: man I am so jealous of the twisted test suite:  real    1m44.882s
<jml> bigjools: yeah. I was going to reply to your comment about the Twisted landing process.
<bigjools> it was somewhat tongue in cheek :)
<jml> bigjools: even though Twisted has stricter standards than Launchpad in many ways, it's still way easier to land stuff.
<bigjools> jml: they need to start using bzr/lp for hosting the code - using svn again was kinda painful!
<jml> bigjools: tell me about it.
<bigjools> I didn't realise how much I relied on local committing
<jml> or shelve, or revert working, etc.
<adeuring> allenap: do you have time for a review? https://code.launchpad.net/~adeuring/launchpad/bug-688130/+merge/51107
<allenap> adeuring: Sure.
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
<adeuring> allenap: thanks!
<bigjools> it pains me that poppy, which was one of the last truly free scripts, now has to load the zcml
<wgrant> gina :D
<wgrant> Or does that load ZCML, but just completely ignore its existence?
<bigjools> it uses getUtility, so no
<bigjools> poppy-sftp will be doing that also, so I had to call execute_zcml_for_scripts
<jml> bigjools: codehosting held out for a while.
<deryck> Morning, all.
<bigjools> jml: resistance is futile ...
<wgrant> deryck: Morning.
<jtv-afk> Could someone try an "ec2 land" to see if it works now?
<wgrant> jtv-afk: What have you changed?
<jtv-afk> Generated new image, from newer lucid
<wgrant> It will still have the new bzr.
<wgrant> 'import bzrlib.plugins.pqm' needs to work in a fresh python
<wgrant> Or we need to patch ec2test-remote.py
<jml> Has this error been fixed:
<jml> You should not import UnknownRemoteValueError from lp.bugs.externalbugtracker:
<jml>     lp.bugs.scripts.checkwatches.remotebugupdater
<wgrant> jml: It's in an MP of mine.
<wgrant> https://code.launchpad.net/~wgrant/launchpad/secret-query-count-determinism/+merge/51086
<StevenK> wgrant: Can you also revert that 42 to 43 business in that branch?
<jml> wgrant: approve w/ tweak.
<wgrant> StevenK: I can't, sadly.
<wgrant> StevenK: That bump was legit.
<wgrant> StevenK: There is actually one more query now.
<wgrant> You added one when you changed PPA traversal.
<wgrant> So the failure in ec2 was not spurious, but the local failure was.
<StevenK> Ah
<wgrant> Same test, same failure, different cause.
<StevenK> Heh
<wgrant> (yes, that took a while to work out)
<elmo> so - I'm confused, am I blind or is +copy-packages really a hidden URL?
<wgrant> elmo: For the primary archive it is deliberately unlinked.
<wgrant> For PPAs you need to "View package details" first.
<elmo> also - how does the 'rebuild the copied sources' option work when you're copying within the same PPA?  is it a binonlyNMU style thing?
<wgrant> elmo: It doesn't.
<wgrant> It will refuse to rebuild into the same archive.
<elmo> ok
<elmo> bother
<wgrant> Roughly, yes.
<elmo> also, is there anyway to transition a PPA from one person to a group?  or should I just copy all the packages (including binaries) from one ppa to the other and then delete the original?
<wgrant> You cannot reassign a PPA. Your suggested method is the best workaround.
<bigjools> the latter
<elmo> cool - thanks guys
<LPCIBot> Project devel build (471): STILL FAILING in 5 hr 25 min: https://hudson.wedontsleep.org/job/devel/471/
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][bug=724138][incr] Add remaining shipit canonical.*
<LPCIBot> imports to lp.shipit.
<LPCIBot> * Launchpad Patch Queue Manager: [rs=stevenk][no-qa] Unbreak the testsuite,
<LPCIBot> disable the entire test_sphinxdocs file.
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless,
<LPCIBot> stevenk][bug=706992][no-qa] Fix the spurious LibrarianWebTestCase
<LPCIBot> test failures by restricting LFA ID replacement to the path.
<LPCIBot> * Launchpad Patch Queue Manager: [rs=stevenk,
<LPCIBot> wgrant][no-qa] Disable the Sphinx docs test due to spurious failures.
<leonardr> geser: while we wait for benjamin (stephano is away), i'll summarize
<jml> I didn't see a bug filed about that last one.
<leonardr> in previous versions of launchpadlib, you had to get a separate credential for every application, and the credentials were stored unencrypted on disk
<jml> oh well. I guess I know now.
<wgrant> jml: It was the _LockWarner one.
<wgrant> jml: The Sphinx test just breaks on it more often than the rest.
<leonardr> geser: now the user gets a single credential that authorizes every application they run on their computer, and the single credential is stored in the gnome keyring or kde waller (with a fallback to an encrypted file on disk if neither is running)
<wgrant> jml: Three builds in a row got it, plus one on Hudson, so we decided to kill it.
<bdrung> leonardr: hi
<jml> wgrant: fair enough. thanks.
<leonardr> bdrung: hi, let me re-paste what i just told geser
<leonardr> in previous versions of launchpadlib, you had to get a separate credential for every application, and the credentials were stored unencrypted on disk
<geser> leonardr: ah that change, I've read about it
<leonardr> now the user gets a single credential that authorizes every application they run on their computer, and the single credential is stored in the gnome keyring or kde waller (with a fallback to an encrypted file on disk if neither is running)
<leonardr> in ubuntu-dev-tools there's some code (find_credentials) to look for a credential file on disk
<leonardr> there's also some code to automatically approve an oauth request token, which 1) is something we're trying to get rid of, and 2) will only work for users who got their launchpad accounts before we started using the openid login service
<wgrant> jml: Can I get a rereview of that branch in a sec? I was going to fix ec2 land in another branch, but since it's trivial and I can't land this one without it...
<jml> wgrant: sure, np.
<leonardr> there's also some minor cleanup: you use a method get_token_and_login which has been deprecated
<geser> leonardr: does the new launchpadlib auth method also work in Debian without problems? (server installs or people not using KDE or gnome)
<leonardr> and in the documentation, and in at least one place in the code, you assume that the server is either 'staging' or 'edge'. there are many more choices now, and 'edge' no longer exists
<leonardr> geser: if you're not using kde or gnome, the keyring library stores the keys in an encrypted file on disk. it's not ideal, but it seems to work
<leonardr> the keyring system does not work for scripts that must run unattended (such as cron scripts)
<wgrant> jml: Diff updated.
<leonardr> for those you still must get a credential, write it unencrypted to a file, and pass the filename into Launchpad.login_with()
<leonardr> hi, tumbleweed
<tumbleweed> leonardr: hi, yes this has been on my todolist for a while
<leonardr> tumbleweed: i just summarized the changes i think need to be made. let me pastebin the conversation to avoid repeating it in this channel
<jml> wgrant: just thinking if there's a sensible way to test the plugin change.
<leonardr> http://pastebin.ubuntu.com/571711/
<wgrant> jml: I don't think that can really be tested. Apart from by landing it.
 * leonardr refreshing his memory of how you're supposed to get a credential for a cron script
<leonardr> good. you just run your code, and if the credential file doesn't exist, you get a browser open, and the credentials are written to that file.
<jml> wgrant: ok. approved w/ comments.
<wgrant> jml: It needs to be run in a subprocess in the right Python environment :/
<wgrant> jml: On vanilla Lucid it works fine without it.
<jml> wgrant: oh yuck.
<wgrant> It's the new bzr PPA packaging that killed things.
<leonardr> tumbleweed, geser, bdrung: i've blocked out my whole day for talking to developers. once i locate everybody i need to locate, i can sketch out some code changes for u-d-t if you like
<leonardr> do you have any questions about these changes?
<tumbleweed> leonardr: sorry I've just sat down at my desk, and people are coming at me left, right and centre
<tumbleweed> and now the power has failed
<leonardr> tumbleweed: np, like i said, i'm here all day
<jml> wgrant: today, it's slightly harder to believe that software can make the world a better place.
<tumbleweed> but I followed the LP bug about the previous upgrade, so I know vaguely what's happeneing
<tumbleweed> for u-d-t it should be quite simple, because the login code is centralised in acouple of places
<wgrant> jml: Yes.
<wgrant> But on the other hand I tracked down the librarian spurious failures.
<wgrant> And that makes me very happy.
<jml> wgrant: yeah, I saw that last night and was very happy. Well done.
<leonardr> tumbleweed: one more thing i noticed (i may just say these random things)
<leonardr> translate_api_web is no longer necessary, since all web service objects that are also published on the website have a web_link. you could add a deprecation warning
<geser> leonardr: does u-d-t just simply use Launchpad.login_with(...) and launchpadlib will take care of everything or do we need some additional handling code?
<jml> I'm going to go outside for lunch & errands and to enjoy the blue sky.
<jml> back soon.
<leonardr> geser: the goal is that you use login_with and launchpad will take care of everything
<leonardr> but i don't know what the people who use your library are expecting
<geser> leonardr: we use already web_link where possible, only one place left (manage-credentials) but it looks like we can dump that script completely soon
<leonardr> wow, that caught on quickly
<bdrung> leonardr: translate_api_web is only used in manage-credentials
<leonardr> oh, it's not even translating a self link. i see
<leonardr> i thought it was a service you provided your users to work around the lack of web_link
<jml> wgrant: I also saw your shipit change. Thanks for that.
<jml> wgrant: are you planning on patching shipit to only import from lp.shipit?
<wgrant> jml: It's part 1 of 3.
<bdrung> leonardr: to summarise: the script just have to use login_with and don't care about the credentials?
<wgrant> jml: Yes, the branch is up.
<wgrant> But I dare not land it until this is deployed.
<jml> wgrant: cool. makes sense. I think I can guess what part 3 is.
<jml> really off now.
<bdrung> leonardr: with version of launchpadlib is required for playing with the new code?
<tumbleweed> leonardr: ok, things have calmed down a bit. I was hoping we'd be able to get rid of manage-credentials (which is why it still has edge somewhere), but I guess we'll need it for non-interactive u-d-t use
<tumbleweed> bdrung, leonardr: and is this available packaged somewhere?
<leonardr> bdrung: i recommend 1.9.3, the latest version. it makes get_token_and_login work with a deprecation warning (in previous versions it was just broken)
<leonardr> tumbleweed: it's in natty
<geser> leonardr: translate_api_web was used that way before web_link appeared, but we switched to web_link soon after an entry about it appeared on planet
<leonardr> barry also has it in a ppa
<leonardr> but i was never able to get his ppa version to work because of a problem with his package of python-keyring
<tumbleweed> yeah I think translate_api_web is only used in manage-credentials now
<geser> tumbleweed: which non-interactive scripts are in u-d-t?
<tumbleweed> we still have translate_web_api, though
<leonardr> tumbleweed: i think you can get rid of manage-credentials altogether. i don't see any difference between calling manage-credentials and calling login_with() and passing in a credential file
<tumbleweed> geser: nothing I can think of
<wgrant> jtv: Hi.
<jtv> hi
<jtv> just looking into what you said
<wgrant> jtv: Fix already in ec2.
<tumbleweed> geser: although, what about using these scripts on a remote box over ssh, I do that quite a lot...
<jtv> !
<wgrant> Since I needed to land a branch :P
<jtv> wgrant: that's fantastic, thanks!
<wgrant> jtv: Is your rev likely to be QA'd today?
<jtv> wgrant: which rev?
<wgrant> jtv: pofilestats perms.
<wgrant> It is the sole thing between us and complete deployment.
<geser> tumbleweed: better ask leonardr, I don't know who that works in this case
<jtv> Then I'll do that.  So many urgent things going onâ¦
<leonardr> tumbleweed: you'll need to set something up to get gnome-keyring running over the x session. it's pretty easy, let me find it
<wgrant> jtv: That would be great. If I'm still around I'll request the deploy, but if not then it'd be great if someone else could.
<wgrant> Thanks.
<jtv> I'll expedite it.
<Ursinha> I can
<jtv> Ursinha: it's blocked on my Q/A
<jtv> which I'm doing right now
<Ursinha> right :)
<tumbleweed> leonardr: that doesn't sound like it'd play well with screen
<geser> leonardr: I'm using a natty chroot (with bind-mounted /home and /tmp) for development (to not pollute my main system with -dev packages and to be able to upgrade it earlier on the next development release). Will it continue to work in such an environment?
<tumbleweed> leonardr: will it fall back to an on-disk credential? And would there be an easy way to create such a credential, included in launchpadlib / lptools?
<leonardr> geser: tumbleweed: i don't know. try it. if there's going to be a problem, better to find out now
<leonardr> if it can't access a keyring it should use an encrypted file on disk
<leonardr> but there might be a problem if the keyring is there but communication with it is impossible
<tumbleweed> yeah I remember issues like that with bzr gtk
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap, jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
<leonardr> tumbleweed: fwiw, the workaround is to put "export `gnome-keyring-daemon`" in your .bashrc
<leonardr> and that's useful in general
<bdrung> leonardr: can sketch out some code changes in a separate branch?
<leonardr> bdrung: i can, but it'll need to wait until i contact the rest of the develoeprs
<bdrung> leonardr: why do you have to wait?
<leonardr> bdrung: sorry, are you asking me to sketch out the code changes, or are you saying you'll do it?
<bdrung> leonardr: bug #724327
<bdrung> leonardr: i was asking you
<leonardr> bdrung: that bug is private
<leonardr> my priority is making sure everyone knows about this change
<bdrung> leonardr: public now
<leonardr> bdrung: did you install from the ppa? python-keyring should now be a dependency of python-launchpadlib
<bdrung> leonardr: the official package in natty
<leonardr> bdrung: ok, sounds like a dependency problem. try installing python-keyring and see if that helps
<bdrung> leonardr: it works then
<leonardr> bdrung: ok, i'll turn the bug into a launchpadlib-in-ubuntu bug
<tumbleweed> leonardr: OOPS-1881A1326
<tumbleweed> seems to have authed, though
<leonardr> i can't see the oops yet, but i bet i know what it is
<bigjools> jml: do our test fixtures give any way of having a fixture that hang around longer than one test?  I've got a slow setup phase running up a  KeyServerTac  process that doesn't really need to go up and down
<leonardr> tumbleweed: can you paste me the exception so i can find the bug?
<tumbleweed> leonardr: there was no exception, I got that after confirming the oauth in my browser
<tumbleweed> on a +token-authorized page
<leonardr> tumbleweed: it's bug 271010
<_mup_> Bug #271010: OOPS accessing a non-existent oauth token page. <api> <lp-foundations> <oauth> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/271010 >
<leonardr> we can fix it on the server side
<gary_poster> allenap hi.  I just saw your reply to bug 723999.  Thank you!  Do I correctly understand your reply to say that, to the best of your knowledge, the query is doing the right/expected thing right now?  Moreover, do I also understand correctly that you do not see a way to improve it (at this time) without offlining the recipient calculation, as discussed at the end?
<_mup_> Bug #723999: structural subscriptions taking 4.8 seconds during nomination editing POST <story-better-bug-notification> <timeout> <Launchpad itself:Triaged by yellow> < https://launchpad.net/bugs/723999 >
<jml> bigjools: not yet.
<bigjools> jml: ok I'll file a bug, which would be the right project, testtools?
<jml> bigjools: it's a good enough place to start.
<bigjools> ok cheers
<allenap> gary_poster: Yes, on both counts.
<gary_poster> allenap, ok, thank you again.
<jml> bigjools: a cheap fix might be to write a fixture -> layer adapter.
<tumbleweed> bdrung, geser: pushing my quick porting to lp:~stefanor/ubuntu-dev-tools/launchpadlib-1.9 (but I don't have too much free time this week)
<jml> bigjools: I would like to do it properly though (i.e. have something that could replace layers).
<bigjools> hellyes
<jcsackett> henninge: any chance i can get your input to answer a question on answers.lp? (per danilos suggestion of trying to get some knowledge transfer going. :-))
<henninge> jcsackett: what's the question?
<jcsackett> question is https://answers.launchpad.net/launchpad/+question/146556
<jcsackett> my belief is the answer is "no, we can't support that very specific language code"
<jcsackett> i saw the faq you wrote about supported language codes, but it mentions exceptions can be made.
<jcsackett> my belief, henninge, is that this is not one of the exceptions lp can support. :-)
<henninge> jcsackett: well, for one be-1959acad is probably not an ISO lang code.
<jcsackett> yeah, it's not in any list, which is why i figured "nope" was the answer.
<henninge> so you would have to enter it as a variant: be@1959acad
<henninge> He is suggesting, I think, to replace "be" with the two variants but we won't do that.
<henninge> The translator community has to decide which variant of the language to use for "be" translations.
<henninge> that is not our call.
<henninge> But from what he is writing it seems that they settled on be-tarask, so the other would be the exception.
<jcsackett> henninge: ah, okay. and the use of a variant is just a community decision, right? we don't have infrastructure to enter it as a variant or anything?
<henninge> jcsackett: you can offer him to add "be@1959acad"
<henninge> jcsackett: we used to have a special database field for the variant but nowadays it's just part of the language code.
<gary_poster> mrevell, thank you for the user testing report.  I'm 2/5 through reading. :-) You said you could help with the text.  I'd be happy to take advantage of that help as much as you'd like.  For instance, if you have concrete suggestions for one or all of the pages, I'd be thrilled to get them and I expect Graham would as well.  Is that what you had in mind, or some other mechanism?
<henninge> jcsackett: The real problem is: will many people do translations in that language? Is it worth it? Maybe he should be warned about that.
<jcsackett> henninge: cool. okay, i'll give him the info and ask if adding be@1959acad is necessary.
<jcsackett> thanks!
<henninge> jcsackett: welcome.
<jtv> wgrant, Ursinha: my branch just went qa-ok
<Ursinha> yay
<wgrant> jtv: Thanks.
<wgrant> Just requested the deploy.
<jtv> wgrant: trying to find your ec2test fixâ¦ did you have a bug for it?
<wgrant> jtv: I stole your bug for it.
<jtv> That's fine, thanks
<jtv> But please attach the branch there!
<wgrant> I thought it was...
<wgrant> ** Branch linked: lp:~wgrant/launchpad/secret-query-count-determinism
<jtv> wgrant: the branch name doesn't sound related
<wgrant> jtv: It isn't, but I needed to land that through ec2, so I put it in the same branch.
<jtv> I see
<mrevell> jml, Yeah, that's what I had in mind. I'll look at that this afternoon.
<mrevell> sorry jml
<mrevell> gary_poster, that was meant for you ^^^
<jml> heh.
<jml> np at all.
<gary_poster> mrevell :-) awesome thank you
<abentley> allenap or jcsackett: could you please review https://code.launchpad.net/~abentley/launchpad/translation-splitting/+merge/50949 ?
<allenap> abentley: It's a Translations branch so I'm not going to be much more than a glorified linter. If that's what you want, I'll take it :)
<abentley> allenap: Is it something about Translations in particular?  Because I'm always reviewing stuff outside Code.
<allenap> abentley: I'm happy reviewing outside of Bugs, but Translations has always eluded me for some reason.
<allenap> I can do it, but I am always left with the feeling that I didn't really get it.
<abentley> allenap: you could try doing a kiko-style "ask a bunch of questions" review.  I've had to learn all this recently, so I might be better at explaining.
<allenap> abentley: Hehe, yeah. I'll do the review, though I doubt I'll fire too many questions at you.
<abentley> allenap: cool.
<flacoste> jcsackett: congrulations on achieving reviewrhood!
<jcsackett> flacoste: thanks! :-)
<danilos> allenap, jcsackett: hey-hey, I've got a branch up for review, and I'd be happy to let you guys fight over it: https://code.launchpad.net/~danilo/launchpad/bug-720826-db/+merge/51146 :)
<danilos> (am I optimistic or what?)
<jcsackett> danilos: i'm looking at a huge javascripty branch; i'll take a look at yours after that. :-)
<danilos> jcsackett, cheers
<allenap> danilos: I'm doing one for Aaron right now, but I'll take a look after that unless jcsackett gets there first.
<danilos> excellent, that's what I like to see :) thanks guys
<jcsackett> allenap: thanks for the review on my validator migration, btw.
<allenap> jcsackett: You're welcome.
<jml> build is broken
<abentley> sinzui: Did I mention that I created a new Job type for a Packaging?  Now that I'm generalizing the code for more than just merging translations, I'm thinking of moving it to Registry.
<sinzui> abentley: that sounds good
<abentley> sinzui: Cool.
<jtv> henninge: I think you've fought this as well in the pastâ¦ unique violations on TM.
<henninge> jtv: Hm ... not sure I was victorious, though.
<jtv> henninge: wanted to bounce this off you, in case I'm missing something big: _setTranslation clears the "current" flag(s) on the incumbent(s), then sets it/them on the new message.  But we're not setting an explicit flush order.
<henninge> jtv: I don't remember that we do, no.
<jtv> So I added some.
<henninge> and do they help?
<jtv> Don't know yet!  Can't think of a meaningful way to test this.
<jtv> henninge: also came across a bit of weirdness in the "if/ifelse" that implements the "first upstream translation overrides ubuntu translation" exceptionâ¦ would appreciate a second opinion to confirm that as it stands, the "if" _inside_ the ifelse clause has a condition that's always true.
<henninge> jtv: can you point me to the code, please?
<jtv> henninge: potmsgset, about line 1040
<henninge> ah, yes
<henninge> jtv: yes, it looks like that condition can go away.
<jtv> cool, thanks
<bigjools> why why why is bzrlib's ftp transport uploading files with a different name to the local one and then renaming it
<allenap> bigjools: So it can detect half-uploaded files in case of a network failure, and so it can atomically replace existing files (again, protecting against network failure)?
<bigjools> allenap: all very nice but it totally fucks up my tests
<bigjools> not to mention the server-side upload checks
<allenap> bigjools: I was about to ask if that was the bother.
<bigjools> oh well I will use ftplib instead of bzrlib
<jtv> jcsackett: congrats on becoming a reviewer!  Want to exercise your new power?
<jcsackett> jtv: sure, especially if it's the branch to deal with that weird translation issue. :-)
<jtv> jcsackett: ah! :)  This one?  https://code.launchpad.net/~jtv/launchpad/bug-708385/+merge/51169
<jcsackett> jtv: indeed.
<allenap> sinzui: I have a branch to allow async merging of people and teams. It's not wired into any of the existing call-sites yet. I wonder if you have a few minutes to discuss the best way to do that?
<allenap> sinzui: https://code.launchpad.net/~allenap/launchpad/async-person-merge-162510/+merge/50919
<jtv> jcsackett: I got the branch, you got the skillz, let's do this thing
<jcsackett> jtv: i'm looking now. :-)
<jcsackett> danilos: r=me, but there's a not about a test doc string you should change. :-)
<jcsackett> jtv: that was the smallest most straightforward branch i've seen today.
<jcsackett> r=me, no comments. thanks for dealing with that.
<jtv> jcsackett: thanks!
<jml> sinzui: I actually quite like the way the Light font looks on Launchpad.
<sinzui> allenap: this is great. I think you mean you want delete team, merge, team merge admin merge to show a message that this job is in progress. delete team and merge are most crucial because they are used by users
<danilos> jcsackett, great, thanks
<sinzui> jml: I agree. official font-size appear a little smaller in my browser and screen size. Light also has narrower headings.
<bigjools> I borked buildbot
<bigjools> no idea how
<allenap> sinzui: Yes, something like that :) I have three concerns: how and where to show that a job is in progress, how and where to report on failure, and how to get the complete set of db permissions required for the person-merge-job db user.
<sinzui> yuck
<sinzui> well it would be nice to show /!\ on the two persons to explain what is happening. Failures is a challenge because the job can be setup by ~registry or ~admins. allenap, we send emails to the user to confirm the merge should happen. Maybe can email the user to let him know something is wrong
<allenap> sinzui: Yeah, that concurs with what I was thinking, except I was only going to show a /!\ on the person being merged (the "from" person).
<sinzui> allenap: the compete set of db permissions is implied in _merge that looks for all columns that reference person. It is an awesome list
<sinzui> allenap: while a job is in progress, the new porfile is gain parts of another. Users see some information transferred and some they now is missing. The user reports a bug or asks a question that I then explain
<allenap> sinzui: Yes, and I assume there will be others, like tables that are touched as a side-effect of other things. I have a first run of the necessary permissions, so I thought I could make async merging available via a feature flag so that we can converge on the correct permissions. Or we could shove it in staging and keep hacking at it until we seem to have everything.
<allenap> sinzui: It shouldn't see anything until the job is done though? It only commits at the end?
<sinzui> allenap: and we will need to update these permission each time a person reference is added to the schema
<allenap> sinzui: Yes. There is a test that exercises the new user, so, apart from side-effects, we should catch that before landing.
<sinzui> allenap: I think there are parts that do not. Consider that the emaila addresses are moved before the job will start
<allenap> sinzui: Ah, true. Eventually those actions could/should be moved into the job too.
<sinzui> I am not sure about the emails. Merge does not actually start until the user has confirmed each email address. that can take days from the initial send
<manish> how can I download the latest Launchpad WADL file?
<manish> is it hosted somewhere?
<manish> Ursinha, lifeless jml gary_poster is launchpad's WADL file hosted somewhere from where I can download it?
<gary_poster> manish hi.  Try leonardr?
<manish> gary_poster, I was not sure if he is there
<leonardr> manish: the wadl file is the wadl representation of the service root
<manish> so this means that I need to authorize the application, and then download it?
<leonardr> so GET /[version] and ask for application/vnd.sun.wadl+xml
<leonardr> you should not need to authorize
<manish> leonardr, possible to download it via curl?
<sinzui> allenap: may it would be possible to generate the list of tables in the proc that reads security.cfg. It could call listReferences(cur, 'person', 'id') to build the list. We know that the code has to declare what it will handle itself. I think stub will have to advise though
<manish> I am working on the API after months
<leonardr> manish: yes, let me figure out the syntax
<sinzui> allenap: oh. there is a skip list that we know can be invalid after a merge (team poll for example).
<leonardr> manish: curl https://api.launchpad.net/1.0/ -H "Accept: application/vnd.sun.wadl+xml"
<manish> thanks
<manish> trying
<allenap> sinzui: Interesting idea :) See my last comment in https://code.launchpad.net/~allenap/launchpad/async-person-merge-162510/+merge/50919 for how I've calculated the list so far.
<allenap> sinzui: s/list/permissions/
<allenap> sinzui: Is your team on maintenance next week? The red squad is moving over to feature work. If you are, then would one of your team be able to take over this work? I don't think I'll get it all done tomorrow!
<sinzui> allenap: green will be on maintenance for 4+ weeks I believe
<sinzui> allenap: I think you can land the core work, then my squad can take up the integration issue
<jtv> bigjools: I see you're having one of those good days with buildbot.
<bigjools> jtv: FSVO
<allenap> sinzui: That's great news. Okay, I'll do that, and email you to hand over.
<jtv> bigjools: I do TLAs, not ETLAs
<sinzui> thanks! really. This was the last timeout left in the old registry teams queue
<bigjools> jtv: it's a shame :)
<jtv> bigjools: I just noticed it's evening hereâI ought to go.
<bigjools> that happens to all of us
<jcsackett> sinzui: i notice you have nested <ul> elements in your diff rather than <ul><li>; was that intentional? https://pastebin.canonical.com/43911/
<sinzui> I suck
<lifeless> morning
<jcsackett> also, i just realized, you have this set to code reviewers, did you want UI, since it's mostly ui stuff?
<sinzui> If I did it once, then I think I may have done it 3 times. I am very consistent :)
<sinzui> oh, good
<jml> I've got these branches here that someone might want to take over: https://code.launchpad.net/~jml/launchpad/reported-by-me-121646 , https://code.launchpad.net/~jml/launchpad/advanced-search
<sinzui> jcsackett: The replace ensure fixes my stupidity. The template should use li as you noted
<jml> if not, maybe I'll magically finish them.
<deryck> Morning, lifeless
<jml> g'night all
<jcsackett> sinzui: ok. i'll continue reviewing with that in mind. you want me to request a ui review for this, as long as i have it open?
<sinzui> yes, I will prepare screen caps of the root pages for henninge or huw
<jcsackett> ok.
<lifeless> hi deryck
<lifeless> night jml
<jcsackett> sinzui: you didn't actually have any other typos like that. i've highlighted the diff in a comment, but aside from the typo, r=me.
<sinzui> jcsackett: thanks I will look the diff over again. since I am prone to repeating my mistakes exactly
<lifeless> flacoste: ping
<flacoste> hi lifeless
<lifeless> quick call ?
<flacoste> lifeless: sure
<LPCIBot> Project devel build (472): STILL FAILING in 5 hr 27 min: https://hudson.wedontsleep.org/job/devel/472/
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml, leonardr][bug=251685,
<LPCIBot> 586695] An implementation of the Poppy FTP server using Twisted.
<LPCIBot> * Launchpad Patch Queue Manager: [r=abentley][no-qa][bug=723733] Clear LC_TIME for update-image.
<abentley> LPCIBot: You mention me, but not even the original committer?
<lifeless> it hasn't been taught to do that. should be reasonably straight forward to do so
<LPCIBot> Project db-devel build (393): STILL FAILING in 5 hr 54 min: https://hudson.wedontsleep.org/job/db-devel/393/
<flacoste> leonardr: can you take a look at https://code.launchpad.net/~thumper/launchpad/lp-client-yui-module/+merge/51038
<flacoste> and see if you agree with my assessment
<leonardr> sure
<lifeless> flacoste: +oops and +haproxy not being allowed db access is a little annoying ;(
<flacoste> lifeless: why? we could revert that policy
<lifeless> lp:~lifeless/launchpad/profile
<flacoste> it wasn't like that in the beginning
<flacoste> what's the issue with that?
<lifeless> it does a feature rule lookup to enable profiling
<lifeless> it happens in beforetraversal, but we inject the null feature controller in callObject
<flacoste> any reason it isn't injected in beforeTraversal?
<flacoste> at the same time than the DBPolicy is set
<lifeless> yes
<lifeless> we need the team selector to use this on staging.
<flacoste> instead of looking at the pageid
<flacoste> in callObject
<lifeless> oh, the db policy
<flacoste> let's simply look at the DBPoilicy
<lifeless> sorry
<lifeless> feature controller
<flacoste> we could even set the FeatureController using adaptation
<lifeless> that might work
<lifeless> I've got a very small change to my patch to get by for now
<bac> hi deryck
<deryck> Hi bac
<thumper> flacoste: the problem with the current launchpad_ajax.js test is I have no idea how to wrap it
<thumper> flacoste: also... we don't want to use sample data
<flacoste> thumper: you mean automate it's running?
<flacoste> or run it
<thumper> flacoste: I'd happily fix it if we can work out how
<flacoste> hang on
<flacoste> so the bug has the command line to run the tests
<thumper> flacoste: also, why do you say that the yui tests can't test the api?
<flacoste> because they don't have the LP app server running
<thumper> hmm...
<flacoste> we might fix that
<flacoste> but I'm not too sure how the io part would interact with YUI tests
<thumper> and to answer your other question, yes I was wanting to add some tests for the cache updating :)
<flacoste> i know yuitest supports some form of asynchronous testing
<flacoste> but not sure it would work in practice
<thumper> flacoste: my problem was a complete lack of understanding of the windmill tests and how to get them to integrate with the new YUI module
<flacoste> it's a different test infrastructure
<flacoste> the jsttests
<flacoste> it's different from both our regular windmill tests (which uses python)
<flacoste> and from yuitest
<thumper> sure, but since the launchpadlib client code is now in a YUI module, is it still possible to test?
<flacoste> it will yes
<thumper> I'm referring to the launchpad_ajax.js test
<thumper> flacoste: would mumble be better?
<flacoste> thumper: we can yes
<flacoste> thumper: i'm in blue one-on-one
<mwhudson> jtv: no
<flacoste> thumper: ./bin/tags -e
<thumper> $ ./bin/tags -e
<thumper> /usr/bin/ctags: invalid option -- 'e'
<thumper> 	Try `/usr/bin/ctags --help' for a complete list of options.
<flacoste> thumper: update-alternatives --list ctags
<flacoste>  /usr/bin/ctags-exuberant
<leonardr> thumper, flacoste, did you come to an agreement on the javascript tests? i'm late to this party
<flacoste> leonardr: i think so, on the phone with thumper right now
<lifeless> any reviewers around ?
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-279513/+merge/51063
<lifeless> I can't mentor wgrants review of mine own branch
<lifeless> jcsackett: ^
<jcsackett> lifeless: looking now.
<lifeless> awesome, thanks
<StevenK> thumper: Still on the phone with flacoste? Trying to decide to make breakfast or wait
<thumper> StevenK: lets do now
<thumper> leonardr: mumble?
<leonardr> thumper, sure
<leonardr> hm, the server's timing out
<jcsackett> lifeless: in case you missed my wrong channel message, r=me.
<thumper> wallyworld: bug 680497
<_mup_> Bug #680497: jstests for LP JavaScript client are not running automatically <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/680497 >
<lifeless> jcsackett: cool, thanks!
<leonardr> thumper, when we move to maintenance, i'd really like to address this bug: https://bugs.launchpad.net/launchpad/+bug/271010
<_mup_> Bug #271010: OOPS accessing a non-existent oauth token page. <api> <lp-foundations> <oauth> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/271010 >
<leonardr> now that launchpadlib has been upgraded in natty, people will start getting that bug all the time
<thumper> leonardr: sounds great, it is critical even :)
<thumper> leonardr: I'd also like to see the lazr.restful change we talked about yesterday
<leonardr> thumper: let me find the bug, i  think lifeless pushed back on it a little
<lifeless> leonardr: I'd be happy to talk about it
<leonardr> thumper: https://bugs.launchpad.net/lazr.restful/+bug/723959
<_mup_> Bug #723959: Separate the floating "latest" version from the latest version-in-progress <lazr.restful:Incomplete> < https://launchpad.net/bugs/723959 >
<thumper> lifeless: how about we all talk about it next week?
<leonardr> sure
<lifeless> thumper: sure; you know where to find me :)
<lifeless> thumper: if you want to make a specific time, I can do that too. monday/tuesday are best; i have a hosital visit on wed and a short friday
<thumper> lifeless: we should try early tuesday so we can get leonardr on his monday afternoon
 * leonardr approves
<lifeless> sure
<lifeless> uhm, my 8am with elliot is cancelled as he's in sa
<StevenK> South Australia?
<lifeless> africa
<lifeless> ensemble sprint
<flacoste> thumper: i'm back
<lifeless> I wonder
<lifeless> perhaps we should have a batchnavigator for linked bugs on branches
<wgrant> lifeless: Why?
<wgrant> Any branch that has lots of bug links probably shouldn't have any shown at all.
<lifeless> https://code.launchpad.net/%7Esoftware-store-developers/software-center/trunk/+index has 375 linked bugs
<wgrant> lifeless: Hmm.
<wgrant> Bug 721591 is Natty-critical, and we have only one or two cocoplum deployments left before release.
<_mup_> Bug #721591: backports release files in natty and above don't have new apt metadata <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/721591 >
<wgrant> lifeless: I thought we didn't show linked bugs on dev focuses...
<lifeless> wgrant: we might not show it. We're doing the work.
<lifeless> anyhow
<lifeless> for now, I think batching will handle it fine
<lifeless> the actual bugbranch relation should be snappy and grabbing a bug description is shallow
<lifeless> anyone feeling LP is slower than usual *right now* ?
<wgrant> I think I have some idea why.
<wgrant> Testing a theory as we speak.
<wgrant> I suspect we have extreme request backlogs.
<lifeless> if we need to escalate, we should do so asap so as not to be waking elmo up
<wgrant> lifeless: Requests that end in an IntegrityError hang.
<wgrant> on staging and lpnet.
<wgrant> I wonder if our retry limit is broken.
<wgrant> It first appeared within the last two days.
<wgrant> Possibly the last day.
<lifeless> that sounds a little worrying
<lifeless> how long do they hang for ?
<wgrant> Ever.
<wgrant> AFAICT
<wgrant> We get a 502 back.
<wgrant> After a long time.
<wgrant> Let's see if it happens locally.
<lifeless> 502 will be apache?
<wgrant> server: Apache/2.2.14 (Ubuntu)
<wgrant> Hangs locally too.
<wgrant> Bisection time!
<wgrant> Ah, found it.
<lifeless> don't we have a test for this ?
<lifeless> you're going to blame me, aren't you :)
<wgrant> No, it was me!
<lifeless> what is it
<lifeless> rev #?
<wgrant> The ZTM thing.
<wgrant>   File "/home/wgrant/launchpad/lp-branches/secret-query-count-determinism/lib/canonical/launchpad/webapp/adapter.py", line 201, in clear_request_started
<wgrant>     transaction.manager.unregisterSynch(_local.commit_logger)
<wgrant> AttributeError: 'thread._local' object has no attribute 'commit_logger'
<wgrant> Does that, looping over and over, forever.
<wgrant> With pauses, though.
<wgrant> The tests must replace some of the error handling machinery...
<lifeless> \o/
<StevenK> Hm. Four test failures on buildbot, all in devscripts.ec2test
<jml> Sorry!
<jml> I'll fix those now.
<lifeless> wgrant: so, we'll have hung threads right ?
 * jml dons the pointy hat of shame
<wgrant> lifeless: It's not entirely clear.
<wgrant> Diff is easy, why it exists and how to test it is not.
<jml> StevenK: btw, did you see my suggestion to make your bot report the failing tests rather than the failing commits?
<wgrant> Also, why it keeps retrying infinitely is also unclear.
<lifeless> wgrant: I'm thinking about the operational impact:
<wgrant> Because it does.
<lifeless>  - are we on a death spiral?
<StevenK> jml: Yes, it's a good idea, but I couldn't see a knob to tweak in the Jenkins config.
<lifeless>  - will a service reboot *now* tide us over until the losas are up in 12 hours
<lifeless>  - if (True, True) lets ring the escalation number and get that done
<jml> StevenK: can't you get the junitxml & parse it in your bot?
<lifeless> jml: it has an object model
<StevenK> jml: It isn't my bot, it's a Jenkins plugin -- I didn't write it, I just run it :-)
<lifeless> jml: no need to get that close to the plumbing; the bot is a plugin to hudson.
<wgrant> lifeless: Do we have data to confirm that it is actually slow?
<jml> lifeless: don't tell me, I'm not going to do it
<lifeless> wgrant: not without escalating to IS
<jml> I see.
<lifeless> jml: the risk in speculating about implementation is that others think you're interested in implementation :)
<StevenK> jml: I think an upstream bug report against the IRC plugin would be awesome, though
<lifeless> wgrant: I'm going to escalate now before it gets really late
<wgrant> lifeless: OK. We've been running the problematic code for two days now, so a restart should be fine.
<lifeless> wgrant: can you also prepare a branch against the current deployed version, so that we can do a custom deploy if it hasn't gotten through buildbot + qa in the next 12 hours.
<wgrant> lifeless: Sure.
<thumper> is ec2 land fixed now?
<wgrant> thumper: No, Windmill killed my branch.
<wgrant> thumper: Easiest workaround is to delete jtv from your list of allowed image owners in lib/devscripts/ec2/account.py
<sinzui> wgrant: mumble?
<wgrant> But my thing should land in four hoursish.
<wgrant> sinzui: Mumble production is broken mumble, but OK.
 * thumper reverts to ec2 test and pqm-submit
<sinzui> wgrant: That reminds. me. I think it messed with my mic again
 * jml plays the regex slot machine game
<jml> sinzui: I've been having troubles w/ mumble and natty since upgrading a couple of days ago.
<StevenK> jml: Your bug about printing failing tests has been fixed upstream. I just need to actually get some time and upgrade our install
<wgrant> Ahh, I guess tests might reuse the same thread.
<wgrant> So the thread-local exists already.
<sinzui> jml: yes that is when I started for me too
<jml> sinzui: I honestly have "fix this or file a bug" on my todo list.
<jml> OK. I think that one has taken.
<wgrant> jml: What has it done? Muting the mic every time you reboot?
<jml> wgrant: yeah, but also other things
<wgrant> That's probably the alsa volume changes.
<jml> wgrant: I can't actually get mumble to pick up sound any more
<wgrant> Hah.
<wgrant> Nice.
<wgrant> My push-to-talk key broke a couple of months ago.
<wgrant> But it works OK otherwise.
<jml> wgrant: when I opened it, it prompted me with an audio wizard
<wgrant> Me too.
<wgrant> Yesterday.
<jml> yeah
<lifeless> wgrant: that theory is plausible.
<jml> I didn't go through it though
<jml> and maybe that's the problem
<jml> I don't know.
<jml> I just need to have half-an-hour free where that's the problem I want to solve
<jml> I was intending to hack on Launchpad some more this evening, but I think I'm not going to.
<jcsackett> sinzui: is there a bug # for the bugmessages => imessage thing we talked about yesterday?
<sinzui> Oh many, let me look for one we could steal
<jml> g'night
<lifeless> jml: gnight
<lifeless> wgrant: series branches show open bugs.
<wgrant> lifeless: :(
<sinzui> jcsackett: bug 201121 is the goal
<_mup_> Bug #201121: Option to delete comments <chr> <feature> <lp-answers> <lp-bugs> <ubuntu-platform> <Launchpad itself:Triaged> < https://launchpad.net/bugs/201121 >
<lifeless> \o/
<jcsackett> sinzui: cool. it's too late to do so now, can i ping you tomorrow morning to have a bit more of a chat about it?
<lifeless> sinzui: hide or actually delete?
<sinzui> jcsackett: Since this is about providing ~registry and ~admins a universal script to hide offensive comments, lets set the first goal to be bug 115322
 * lifeless is easy but reckons hide is probably sufficient for now
<_mup_> Bug #115322: Need better protection against offensive comments and attachments from a user <canonical-losa-lp> <infrastructure> <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/115322 >
<sinzui> hide, not delete
<jcsackett> lifeless: hide.
<lifeless> sinzui: do we need a script? What about just a button on the comment ?
<sinzui> jcsackett: lifeless, phase 1 move the flag on IBugMessage to IMessage and allow use to change the flag. Update three comment systems to honour the flag
<lifeless> sinzui: I'm confused
<sinzui> jcsackett: lifeless phase 2, allow users to mark a comment as inappropriate. Let use review the queue to hide or unmark
<lifeless> sinzui: -very- few messages are shared between bugs. Why do we need to move the flag ?
<sinzui> Because I will not implement something 3 times
<lifeless> sinzui: this will have performance implications
<sinzui> jcsackett: is welcome to implement it 3 or 5 times if he agrees to implement
<lifeless> sinzui: the query plan filters on bugmessage at the moment, which is very very efficient
<jcsackett> i would rather not implement 3 times, but if we need to hack something for performance, i can look at it.
<sinzui> then I think you want to talk to jcsackett
<lifeless> sinzui: message is a massive table by comparison; its likely to change the performance of retrieving messages in all the systems if message gets wider
<jcsackett> lifeless: i'll have to have a longer talk with you tomorrow, i'm 5 min from EOD and need to meet people across town for an evening meeting.
<jcsackett> but i'm definitely happy to talk.
<lifeless> sinzui: I can certainly agree that message is the logical place
<sinzui> lifeless:  so if we had one comment system, would you be crying?
<lifeless> jcsackett: Its saturday tomorrow, so no promises. But ping me; if I'm around I'll be happy to talk.
<lifeless> sinzui: I would be ecstatic if we had that *and* cruft like 'BugComment' - wtf - were gone. *and* we sort the performance stuff out.
<jcsackett> oh right. i always remember different hours of the day. forget diff days of the week. :-p
<lifeless> jcsackett: I probably will be around; but it will depend on lynne etc etc
<jcsackett> right.
<sinzui> jcsackett: keep in mind that to two bugs I showed you talk about people spamming via the UI. 90% or more are via the email gateway this year
<sinzui> My goal is to not assign a question to a losa to get rid of the message
<lifeless> sinzui: whats the thing one calls on a view to make it render the template (and subtemplates/portlets etc) in tests
<lifeless> sinzui: its not just 'create_initialized_template' is it
<sinzui> view = create_initialized_view(obj, '+name')
<sinzui> view()
<sinzui> well view.render() to be explicit
 * wallyworld hates doctests
<sinzui> lifeless: many templates want you to pass the principal kwarg to create_initialized_view()
<lifeless> principal=a_person ?
<wallyworld> when a test fails, it prints as "errors" all the places where there's "..." and it's hard to see the real error :-(
<sinzui> lifeless: sorry, I got distracted. yes principal=a_person is used to setup the request
<lifeless> sinzui: thanks!
<lifeless> sinzui: I get this - ComponentLookupError: ((<lp.code.browser.decorations.DecoratedBranch object at 0x100bd5d0>, None), <InterfaceClass zope.interface.Interface>, '+hierarchy')
<lifeless> sinzui: I haven't [yet] added the principal argument. Doing so now
<sinzui> lifeless: you probably don't need the principal yet, some tales code (menus?) need it
<lifeless> sinzui: any idea what causes that component lookup error then ?
<sinzui> yes, I have seen that with +hierarchy. I am going to look at the lp tree. I think we need to pass a PATH arg
<lifeless> maybe layer, or root_site?
<sinzui> lifeless: here is an example from an old test, and an alternate how I would write it http://pastebin.ubuntu.com/571959/
<lifeless> is path_info the canonical_url ?
<sinzui> yes it is
<lifeless> sinzui: http://pastebin.com/ZqKwvs2M
<lifeless> sinzui: https://bugs.launchpad.net/launchpad/+bug/710685 is the bug I'm working on
<_mup_> Bug #710685: Branch:+index timeouts - 3 queries triggered per linked bug <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/710685 >
<sinzui> oh effing pagetitle
<lifeless> create_initialized_view has constant query counts, but the page doesn't per the oops, so I want to render
<lifeless> I could use getUserBrowser
<lifeless> but I'm trying to find the thin line below that
<lifeless> sinzui: where should it get the current request from ?
<lifeless> sinzui: I see its using get_current_browser_request
<wgrant> lifeless: In the interests of getting this out ASAP, should I land without tests? They are going to require a bit of thought and work.
<wgrant> There's only one un-QAd rev in devel at the moment.
<lifeless> wgrant: soit
<sinzui> lifeless: I do not think DecoratedBranch has a registered hierarchy adapter and there is nothing in the default setup that is being used
<sinzui> and this is death is render()
<lifeless> yes
 * sinzui looks at DecoratedBranch
<lifeless> are there reasonable alternatives to a user browser within reach of a morning fiddle ?
<sinzui> lifeless: I know what to do. I faced this when working with menus
<sinzui> We need a helper.
<lifeless> o/~ I need a hero o/~
<wgrant> Total: 7 tests, 0 failures, 0 errors in 27.508 seconds.
<wgrant> General warnings.
<wgrant> The doctest <doctest test_adapter.txt[98]>, at the line: >>> clear_request_started()
<sinzui> oh bugger, I may be in a Klein bottle
<wgrant> That warning is so general that it has no content at all!
<sinzui> lifeless: http://pastebin.com/3Bkr9ntz
<sinzui> We need the traversed objects, so we build a request with them first
<wgrant> lifeless: Ah, so it turns out the retry machinery is buggy.
<wgrant> lifeless: But we have just been swallowing the warnings about the illegal clear_request_started call for years.
<wgrant> Does anybody know how to check warnings in a doctest?
<lifeless> they show as 'output'
<wgrant> No :(
<wgrant> They show as "General warnings." at the end of the test run, apparently.
<wgrant> I guess I will try the catch_warnings decorator.
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/unbreak-request-retry/+merge/51234
#launchpad-dev 2011-02-25
<LPCIBot> Project devel build (473): STILL FAILING in 5 hr 24 min: https://hudson.wedontsleep.org/job/devel/473/
<LPCIBot> * Launchpad Patch Queue Manager: [testfix][r=jml, leonardr][bug=251685,
<LPCIBot> 586695] An implementation of the Poppy FTP server using Twisted.
<LPCIBot> * Launchpad Patch Queue Manager: [r=henninge][no-qa] Replace DistroSeries/SourcePackageName with
<LPCIBot> SourcePackage in translationsharinginfo.
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=721784][no-qa] [no-qa]
<LPCIBot> [r=leonardr][bug=721784] Add ec2 list command to list running ec2 instances
<LPCIBot> and test status
<wgrant> Any other reviewers around?
<wgrant> sinzui: ^^?
<lifeless> sinzui: thanks, was otp
<lifeless> wgrant: is that the right way around?
<lifeless> shouldn't it be 'is not None' ?
<lifeless> sinzui: trying it now
<wgrant> lifeless: Yes.
<wgrant> So, then, why does it work.
<lifeless> wgrant: because its not none so it leaves it installed
<lifeless> wgrant: you're testing the external contract 'do not warn on this case' - but thats not comprehensive ;)
<lifeless> sinzui: ><
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/browser/launchpad.py", line 283, in makeBreadcrumbForRequestedPage
<lifeless>     if view.__name__ != default_view_name:
<lifeless> AttributeError: 'Branch' object has no attribute '__name__'
<lifeless>             request = make_fake_request(
<lifeless>                 'http://code.launchpad.dev/~%s/%s/%s/+index' % (branch.owner.name, product.name, branch.name),
<lifeless>                     [branch.owner, product, branch])
<lifeless>             view = create_initialized_view(branch, '+index', request=request, principal=branch.owner)
<lifeless>             view.render()
<wgrant> lifeless: Heh, what this shows is that the synchronizer can just be registered at startup.
<wgrant> It doesn't matter if it's removed any more, because with Storm there is only ever one.
<wgrant> Will investigate later.
<lifeless> you're inverting the if for now ?
<wgrant> Yes.
<wgrant> Just testing that it behaves properly in the app.
<wgrant> It does.
<wgrant> lifeless: DOne.
<lifeless> approved
<wgrant> Thanks.
<lifeless> sinzui: something really odd is going on here
<lifeless> in pdb I get this:
<lifeless> (Pdb) self
<lifeless> <zope.browserpage.metaconfigure.SimpleViewClass from /home/robertc/launchpad/lp-branches/working/lib/lp/code/browser/../../app/templates/launchpad-hierarchy.pt object at 0x7879450>
<lifeless> (Pdb) view
<lifeless> <zope.browserpage.metaconfigure.SimpleViewClass from /home/robertc/launchpad/lp-branches/working/lib/lp/code/browser/../templates/branch-index.pt object at 0x2f409d0>
<lifeless> but in the test, the second is the branch /object/
<lifeless> not the view
<lifeless> ah
<lifeless> sinzui:
<lifeless> (Pdb) self.request.traversed_objects[4]
<lifeless> <zope.browserpage.metaconfigure.SimpleViewClass from /home/robertc/launchpad/lp-branches/working/lib/lp/code/browser/../templates/branch-index.pt object at 0x402a9d0>
<lifeless> sinzui: I think create_initialized_view should poke that in
<lifeless> fingers crossed!
<lifeless> sinzui: so I made it work, but its showing a query count of 19, which is implausibly small
<lifeless> sinzui: http://pastebin.com/nB3ZSn2t for your reference, I'm going to use getUserBrowser now
<lifeless> sinzui: getUserBrowser finds - Difference: queries do not match: 50 is >= 53
<lifeless> wgrant: have you landed that webapp session cache preloading yet ?
<wgrant> lifeless: It's in ec2.
<wgrant> Windmill killed it last night.
<lifeless> its new matcher time, baby
<wgrant> Oh?
<lifeless> rewriting assert_milestone_page_query_count as a matcher for reuse
<wgrant> Excellent.
<wgrant> 0 tests run in 3:51:18.052785, 0 failures, 0 errors
<wgrant> Ah, windmill.
<lifeless> .,
<wgrant> Die Windmill die.
<wgrant> OK, I am going to lunch and then beat Windmill to death with a few ec2 instances.
<lifeless> wgrant: are you cleaning up test_milestone in that change?
<lifeless> lp:~lifeless/launchpad/bug-710685 if anyone is interested
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: What needs cleaning up?
<lifeless> wgrant: accomdations for the session issue
<lifeless> I have a test in there that does a request to seed it
<wgrant> Ah.
<lifeless> may be others scattered around the code base
<wgrant> I have two unexplained failures that *appear* to be fallout, but look fairly irrelevant, and only show up on ec2. I am sorting them out first.
<wgrant> +opstats OOPSes unexpectedly :/
<wgrant> But this is in PageTestLayer...
<lifeless> http://pastebin.com/f6xQ1J8L
<lifeless> ^- thats what writing scaling tests is down to now ;)
<lifeless> +opstats - whats the oops ?
<wgrant> Not sure yet.
<wgrant> Cannot reproduce locally.
<lifeless> ><
<lifeless> it runs w/o a db
<wgrant> But it's slightly crazy, since PageTestLayer runs an appserver in a subprocess...
<wgrant> Ah, hm.
<james_w`> do you not get the oops in the subunit stream now?
<wgrant> True.
<wgrant> james_w`: Not this one :/
<james_w`> damn
<wgrant> It's a doctest, which may be relevant.
<lifeless> james_w`: only if the generator is in the same process
<lifeless> wgrant: this is xx-oops.txt ?
<wgrant> lifeless: xx-opstats.txt
<lifeless> wgrant: cause I saw an ec2 failure with it, but I made sure it passed locally before submitting
<lifeless> bah, thats the one I meant
<wgrant> Also librarian.txt failed.
<lifeless> wgrant: thats in-process FWIW, but its a doctest
<wgrant> But not on a second run.
<wgrant> I guess I will run just those tests in devel on ec2.
<wgrant> Interestingly they passed yesterday.
<wgrant> All I've changed since it loading bzr plugins.
<wgrant> So perhaps they are yet more wonderful spurious failures for me to crush this afternoon.
 * lifeless facepalms
<lifeless>     @cachedproperty
<lifeless>     def linked_bugs(self):
<lifeless>         """Return a list of DecoratedBugs linked to the branch."""
<lifeless>         bugs = self.context.linked_bugs
<lifeless>         if self.context.is_series_branch:
<lifeless>             bugs = [
<lifeless>                 bug for bug in bugs
<lifeless>                 if bug.bugtask.status in UNRESOLVED_BUGTASK_STATUSES]
<lifeless>         return bugs
<wgrant> Hahahah
<wgrant> That's been done before.
<lifeless> there are two bugs here.
<wgrant> Not cached if it's not a series branch, and lots of queries if it is.
<lifeless> yes!
<lifeless> well, not -quite-
<lifeless> because its a DecoratedBranch
<lifeless> which needs to be unwound and pushed down
<wgrant> utilities/ec2 list is great.
<lifeless> yep, both cases scale poorly
<wgrant> Heh
<wgrant> lifeless: Nice Matcher.
<lifeless> wgrant: thanks
<lifeless> UI check
<lifeless> currently the menu shows
<lifeless> 'Link to another bug report'
<lifeless> or 'Link to a bug report'
<lifeless> depending on - you guessed it - a query
<wgrant> Nice.
<wgrant> Drop it.
<lifeless> which doesn't matches the is_series_branch change
<lifeless> *check*
<lifeless> so, its a) broken.
<wgrant> b) pointless
<lifeless> yeah.
<lifeless> 'Link a bug report' is how I'd like to phrase it
<wgrant> Yeah.
<james_w`> The tags applied to this bug. Web service: The list of tags is whitespace delimited. Launchpadlib: The list of tags is represented as a sequence of strings.
<wgrant> james_w`: huwshimi has a branch for that.
<james_w`> great
<wgrant> Also, you seem to have a parasite.
 * lifeless sobs
<lifeless> isBugBadgeVisible
<wgrant> Hah.
<wgrant> Nice.
<lifeless> not when you conside rthat context is a decorated branch
<lifeless> which means that this requires /all/ the bugs to evaluate
<lifeless> regardless of what it claims
<lifeless> hmm
<lifeless> the linked_bugs api call probably takes forever on these branches too
<lifeless> because its not eager loading anything, it will do just-in-time permission checks for every bug
<lifeless> *meep*
<lifeless> related_bugs
<lifeless> repeatedly queries the target branch bug list for every bug in the source branch
<thumper> that kinda sucks
<thumper> I'm sure that wasn't the intention :)
<lifeless> I'm sure it wasn't either
<huwshimi> wgrant: Just sending that off to land now :)
<thumper> given that I wrote it :)
<wgrant> huwshimi: Good luck with that...
<huwshimi> wgrant: Is stuff still not working?
<lifeless> its a symptom of us not having a persistence layer and data access being mixed with business logic as python code
<wgrant> lifeless: Confirmed xx-opstats.txt occasionally shows up on devel... but only on ec2.
<wgrant> huwshimi: ec2 land is still a bit broken.
<wgrant> pqm-submitting the branch now.
<lifeless> wgrant: I'd lke to see the error
<huwshimi> wgrant: Should I just wait for Monday
<huwshimi> ?
<lifeless> thumper: I know you spent a lot of time on this, its really tricky with the tools we /had/ and still pretty darn tricky
<lifeless> thumper: https://bugs.launchpad.net/launchpad/+bug/710685 is the bug I'm chasing
<_mup_> Bug #710685: Branch:+index timeouts - 3 queries triggered per linked bug <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/710685 >
<wgrant> huwshimi: 40 minutes should do it.
<huwshimi> wgrant: Oh awesome
 * thumper sighs
<wgrant> lifeless: Forwarded.
<lifeless> thumper: http://pastebin.com/uf0FCV1G is my work in progress - currently that test shows 58 new queries in the second assert - for both the series and nonseries case.
<wgrant> lifeless: That's plain devel except for my fixed ec2test-remote.py
<wgrant> Can't reproduce the librarian failure anyway, but it only showed up in one of the three runs.
<huwshimi> wgrant: I assume these errors are related to those issues? http://paste.ubuntu.com/572017/
<wgrant> So it's probably old and spurious too.
<lifeless> the good news is we squashed nearly 200 timeouts since last week
<huwshimi> wgrant: From another branch
<lifeless> 600 overnight vs 800
<wgrant> huwshimi: That's fixed now.
<lifeless> wgrant: your mua sucks
<wgrant> lifeless: Oh?
<lifeless> wgrant: I got a .eml attachment with base64 encoded internal structure + headers
<wgrant> lifeless: I use Thunderbird on my desktop and Evolution on my laptop, and they both suck.
<huwshimi> wgrant: Is that just related to landing the branch? As in, can I just manually land that branch or do I need to resubmit for testing?
<wgrant> huwshimi: I'd lp-land that.
<wgrant> No point retesting.
<huwshimi> wgrant: Great, thanks.
<wgrant> The fact that the original change got through in the first place shows that there are no tests.
<wgrant> buildbot approaches the moment of truth...
<wgrant> It is past it!
<wgrant> Windmill did not blow up!
<wgrant> lifeless: Were you able to read the attachment?
<wgrant> Or shall I forward with Evo instead?
<wgrant> And add Thunderbird to my list of things to eliminate after Windmill and xx-opstats.txt
<StevenK> Hmmm. Can someone try to click the diff link on https://code.launchpad.net/~wgrant/launchpad/secret-query-count-determinism and then try and close the diff popup?
<thumper> anyone seeing ec2 errors on xx-opstats.txt ?
<StevenK> I thought wgrant just mentioned that
<lifeless> wgrant: not without more futzing than I have mental space for right now
<StevenK> [13:52] < wgrant> lifeless: Confirmed xx-opstats.txt occasionally shows up on devel... but only on ec2.
<wgrant> StevenK: Yes, broken here too.
<wgrant> thumper: When did you first see that.
<wgrant> StevenK: (the diff popup, that is)
<StevenK> wgrant: Excellent. Let me file a bug
<thumper> now
<thumper> well, on the ec2 run that just finished
<wgrant> thumper: Which rev of devel?
<thumper> works locally now though
 * thumper checks the email
<wgrant> thumper: It always works locally.
<thumper> r12457
<wgrant> But fails around half the time on ec2.
<wgrant> Thanks.
<thumper> half?
<wallyworld> anyone want to claim a review? https://code.launchpad.net/~wallyworld/launchpad/recipe-request-build-partial-success/+merge/50873
<lifeless> thumper: hi
<lifeless> thumper: question for you about linked bugs
<lifeless> thumper: which bugtask do we /intend/ to show ?
<thumper> lifeless: shoot
<thumper> lifeless: it should be the bugtask relating to the target of the branch
<lifeless> tal:define="bugtask bug/bugtask
<thumper> sometimes it is linked to a bug that doesn't have a task for that
<lifeless> is the lookup
<thumper> in which case, we get the first
<lifeless> I can't find a bugtask attribute on bug
<thumper> is it on the annotated bug?
<lifeless> ah
<lifeless> DecoratedBug
<lifeless> right
<lifeless> is branch.target the context of the branch
<lifeless> e.g. the product/sourcepackage ?
<thumper> it is a BranchTarget object
<wgrant> lifeless: Can I blame you?
<wgrant> I think I might.
<lifeless> thumper: I'm sure you considered this, but what about linking to bugtask ? It would push this complexity to where we add links, but we wouldn't pay it after
<lifeless> wgrant: it might be me
<lifeless> wgrant: pastebin it
<wgrant> lifeless: I'm thinking that your profiling stuff is suspicious.
<wgrant> But I still can't repro locally.
<thumper> lifeless: it was considered but just not done, we could also link revisions as fixes
<wgrant> lifeless: http://paste.ubuntu.com/572027/
<StevenK> Ew, doctest diffs
<wgrant> I guess I will add some getLastOops stuff to it and rerun.
<wgrant> Because it does not want to happen locally.
<wgrant> Has anyone seen it before r12457?
<wgrant> Confirmed.
<wgrant> +opstats OOPSes on launchpad.dev
<lifeless> its throwing a 503
<wgrant> Hm, not 500?
<lifeless> wgrant: thanks for finding and digging
<lifeless> + 503s:1@1298603161
<lifeless> + 5XXs:1@1298603161
<wgrant> Oh.
<lifeless> and there is one error higher up
<wgrant> +opstats explodes when you are authenticated regardless.
<lifeless> yes
<wgrant> 503... it times out?
<wgrant> lifeless: Can you check how many BugSubscriptionFilters staging and qastaging have?
<wgrant> We added them manually on production, so I think they may be missing on the stagings.
<wgrant> Which would explain the quick query.
<wgrant> The plan suggests that there are very fwe.
<lifeless> 4
<wgrant> Hah.
<wgrant> So the plan is completely invalid.
<wgrant> That would do it.
<lifeless> how many filters does prod have?
<lifeless> wgrant: ^ ballpark
<wgrant> lifeless: SELECT COUNT(*) StructuralSubscription
<wgrant> It has one for every strucsub.
<lifeless> wgrant: that was the conversion thing ?
<lifeless> so, unknown number. did you happen to see the actual row count inserted ?
<wgrant> lifeless: Yes, we added a no-op filter for every subscription.
<wgrant> Yes.
<lifeless> [yes unknown?]
<wgrant> INSERT 0 24056
<wgrant> lifeless: I have a traceback for you.
<wgrant> http://paste.ubuntu.com/572033/
<wgrant> It is *timing out* in the flag check.
<wgrant> Ah.
<wgrant> I see.
<wgrant> The tests override the timeouts.
<wgrant> This wasn't a problem before because there was no DB access.
<wgrant> But now it checks the timeout before it can raise a DisallowedStore.
<wgrant> Impressive.
<lifeless> the flag layer checks the timeout
<lifeless> because if you just query and get timed out by the db - in other tests - you end up with a doomed transaction
<lifeless> which is a world of pain when that happens *in the error handler*
<wgrant> Ah, right.
<lifeless> however
<wgrant> Have you filed a bug, or shall I?
<lifeless> its passing that (not timed out), then doing the query, and is slow enough it has then timed out
<lifeless> wgrant: if you would
<lifeless> that would be great
<wgrant> lifeless: But there should be a DisallowedStore in there somewhere, right?
<wgrant> Before it gets to the query.
<lifeless> I would have expected that, yes.
<lifeless> except
<lifeless> uhm
<lifeless> it probably hasn't traversed to +opstts yet
<lifeless> *+opstats* yet.
<wgrant> Ha ha ha.
<wgrant> Nice.
<wgrant> Bug #724718
<_mup_> Bug #724718: xx-opstats.txt failing occasionally <spurious-test-failure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/724718 >
<lifeless> those tests are basically bong
<wgrant> Almost all of our tests are basically bong.
<lifeless> every time I change instrumentation gathering, or things like features, I meet them again.
<lifeless> wgrant: uhm, it may be we need to patch into zope publication more and switch to a 'straight after traversal before callObject' point
<lifeless> wgrant: and/or call out from right after the dbpolicy is installed
<lifeless> which would be a shame, because traversal is nonfree
<wgrant> Yes :(
<wgrant> Anyway, sounds like we may want to roll it back, or possibly just mangle the test a bit.
<wgrant> lifeless: +1 on your structuralsubscription suggestion.
<lifeless> wgrant: do what you think best; I think pushing the timeout up -just a little- will probably make it reliable
<lifeless> thumper: so whats the branch target for a junk branch? I'm very confused about the difference between a branch target and a bugtask target
<thumper> lifeless: it is a PersonTarget
<wgrant> lifeless: It looks like the timeout-causing views will adapt themselves to the configured timeout values.
<thumper> lifeless: BranchTaget objects are found in lp.code.model.branchtarget
<wgrant> Which is handy, albeit in a "I think I'm going to be sick" way.
<lifeless> thumper: interesting thing
<lifeless> thumper: linked bugs show *no* bugtask if the task is on a different context
<lifeless> thumper: so I can just push that down to the bug search
<thumper> lifeless: that seems wrong...
<lifeless> lib/lp/code/browser/decorations.py
<lifeless> the bugtask property returns
<lifeless> the result of getBugTask
<lifeless> which looks for an exact match and on failure returns None
<lifeless> which is excellent, I can push this all down to the db
<lifeless> refactor to return tasks not bugs
<lifeless> \o/
<wgrant> lifeless: OTOH I can fix this permanently by popping the config before requesting +opstats
<wgrant> I might do that.
<lifeless> wgrant: \o/
<lifeless> lets see how badly I've broken this
<lifeless> thumper: ah, I see what its doing; I will preserve it. Some epic confusion is possible here ;)
 * wgrant narrows eyes at Windmill.
<wgrant> --layer=WindmillLayer has passed 5 times on ec2 so far.
<wgrant> Bad Windmill.
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/bump-opstats-test-timeouts/+merge/51249
<thumper> wgrant: passed or failed?
<wgrant> thumper: passed.
<wgrant> thumper: I want it to fail so I can minimise the problem from "the whole test suite"
<wgrant> Because it's much quicker to run WindmillLayer than everything.
<wgrant> In fact it takes only 46 minutes.
<wgrant> lifeless: Thanks.
<thumper> wgrant: comments welcome https://code.launchpad.net/~thumper/launchpad/inline-recipe-name-edit/+merge/51248
 * thumper lands self reviewed branch
<thumper> https://code.launchpad.net/~thumper/launchpad/recipe-heading-overlap/+merge/51250 in case anyone cares...
<thumper> fixes bug 721134
<_mup_> Bug #721134: Description and Recipe Text heading overlap when logged out <recipe> <regression> <ui> <Launchpad itself:In Progress by thumper> < https://launchpad.net/bugs/721134 >
 * thumper EODs
<thumper> I'll check in later on ec2 test runs
<wgrant> lifeless: It has the usual line editor glitches, but apart from that looks great.
<wgrant> Er.
<wgrant> thumper: ^^
<lifeless> wgrant: what does ?
<thumper> wgrant: what line editor glitches?
<wgrant> thumper: Clicking the edit link next to the name shifts the whole page down slightly.
<thumper> wgrant: file a bug and I'll fix it
<thumper> wgrant: I did fix the multi-line editor 1px down-shift
<thumper> wgrant: that it got on hover
<thumper> wgrant: those are the sort of bugs that most people never notice, but once you have, it annoys the shit out of you
<wgrant> thumper: The single-line editor one is really obvious.
<wgrant> Have you seen it?
<thumper> wgrant: not focused on it
<wgrant> thumper: lazr-js or launchpad?
<thumper> wgrant: both?
<thumper> I fixed the multiline in lazr-js
<wgrant> thumper: Bug #724727
<_mup_> Bug #724727: Single-line inline editor causes page to shift when editing <javascript> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/724727 >
<thumper> wgrant: ta
 * StevenK peers at dogfood
<wgrant> StevenK: Oh, are you doing stuff to it?
<wgrant> I am testing the trio of poppys.
<StevenK> KeyError: 'request_daily_build'<br />
<wgrant> StevenK: New templates, old code?
<wgrant> Well, what is the full traceback?
<StevenK> Could be. Unsure
<StevenK> wgrant: Let me dig up the OOPS
<wgrant> StevenK: Shall I restart the appserver?
<StevenK> wgrant: 17471.DF5 in the usual place
<wgrant> Ja, needs appserver restart.
 * wgrant restarts the appserver.
<StevenK> wgrant: I'm worried that Julian is no longer testing Zope's poppy
<wgrant> StevenK: I just did.
<wgrant> Still works.
<StevenK> wgrant: Er, sorry. That due to his changes the testsuite no longer is.
<wgrant> Ah/
<wgrant> Maybe.
<StevenK> Has the appserver restarted?
<wgrant> No.
<wgrant> Still building the WADL so it can stop.
<StevenK> Fail
<wgrant> Is down.
<wgrant> Coming back up.
<StevenK> Yes, he no longer is.
<wgrant> :(
<lifeless> \o/ down to one bugtask query
<lifeless> 2 related lookups to squash
<StevenK> wgrant: I'm not sure when Julian plans to kill the Zope FTP server, though, so it could be a non-issue
<StevenK> wgrant: I take it by the lack of make processes, the appserver is back?
<wgrant> StevenK: It should be.
<wgrant> I cannot auth to poppy-sftp, though :(
<wgrant> Hm.
<wgrant> Appserver did not come back up.
<wgrant> Did Julian pull db-devel?
<wgrant> Probably.
<wgrant> wut
<StevenK> wgrant: That reminds me. When I was in Adelaide there was one wireless network in range of where I was staying. The ESSID was 'lolwut'
<StevenK> wgrant: Usually when the appserver fails like this, dogfood-nohup.out fills with tracebacks
<wgrant> StevenK: It did.
<wgrant> The 'wut' was at a very strange error.
<wgrant> But I was looking in development-nohup.out instead.
<wgrant> dogfood-nohup.out has the usual patch failure.
<wgrant> Faking it now...
<wgrant> StevenK: We're back.
<lifeless> lp:~lifeless/launchpad/bug-710685 handles products properly now, still have productseries and distroseries to capture
<StevenK> wgrant: Confirmed, thanks.
<huwshimi> wgrant: Any luck with ec2 land or still working on it?
<StevenK> huwshimi: The fix landed, merge devel
<huwshimi> StevenK: Awesome thanks.
 * StevenK grumbles at the lack of logtailing on dogfood buildds
<wgrant> Ah, poppy-sftp works better when it's not using the former production xmlrpc server.
<wgrant> StevenK: Do it manually.
<StevenK> Heh, former production?
<StevenK> wgrant: I don't know how :-(
<wgrant> StevenK: It was still using xmlrpc.lp.internal:8097, which died a couple of months ago.
<wgrant> StevenK: import xmlrpclib; xmlrpclib.ServerProxy('http://something.buildd:8221/rpc').status()
<StevenK> Which says IDLE
<wgrant> That would suggest idleness.
<StevenK> Clearly, but the build page itself and /builders disagrees.
<StevenK> Now it says building. WTF
<wgrant> Could have been resuming.
<wgrant> Or just being shit.
<StevenK> I did say karmic, it could have been digging out a chroot tarball
<wgrant> Indeed.
<wgrant> StevenK: I am done with mawson.
<wgrant> all three poppys work!
<LPCIBot> Project devel build (474): STILL FAILING in 5 hr 20 min: https://hudson.wedontsleep.org/job/devel/474/
<LPCIBot> * Launchpad Patch Queue Manager: [testfix][r=jml][no-qa] Fix test failures in
<LPCIBot> devscripts.ec2test.tests.test_remote
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][no-qa] Allow overriding the profiling_allowed config
<LPCIBot> entry via a feature flag.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jtv][no-qa] Work around Storm ClassAlias bug more cleanly.
<StevenK> wgrant: As am I. Incidently, you'll get some mail.
<wgrant> Which mail?
<wgrant> I got two more disappointing ec2 successes.
<wgrant> But is there anything else?
<StevenK> I changed some bug statues
<StevenK> *statuses
<wgrant> :( short months make me sad.
<StevenK> wgrant: Oh?
<StevenK> wgrant: Based on your experience do you think bug 579870 can be nailed shut?
<_mup_> Bug #579870: Packages built from daily builds say 'No signer' <lp-soyuz> <recipe> <Launchpad itself:Triaged> < https://launchpad.net/bugs/579870 >
<wgrant> StevenK: Yes, it's fixed.
<wgrant> StevenK: Less time to fix bugs.
<wgrant> The milestone is looking rather sad, yet we are a week from release.
<StevenK> wallyworld: O hai?
<wallyworld> StevenK: de ho
<StevenK> wallyworld: We were going to add a button "Build right now" to daily built recipes, were you working on that?
<wgrant> It's done.
<wgrant> Will be deployed in 4 hours.
<StevenK> Ah, what's the bug number, I haz dupe
 * wallyworld looks
<wgrant> bug 687623
<_mup_> Bug #687623: No ability to manually trigger a build 'like the daily build system' <lp-code> <qa-ok> <recipe> <Launchpad itself:Fix Committed by wallyworld> < https://launchpad.net/bugs/687623 >
 * wallyworld unlooks
<StevenK> So marked, thanks
<StevenK> 3 bugs knocked off
<wallyworld> there's also some enhancements to that stuff waiting for review https://code.launchpad.net/~wallyworld/launchpad/recipe-request-build-partial-success/+merge/50873
<wgrant> wallyworld: I also filed bug 724679 earlier.
<_mup_> Bug #724679: Not obvious why "Build now" button is missing <recipe> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/724679 >
<wallyworld> wgrant: you reall think that's a bug?
<wallyworld> pbkbac
<wallyworld> :-P
<wgrant> wallyworld: It had me confused for a while.
<wallyworld> wgrant: i'm not sure what we can do about it though
<wgrant> wallyworld: Problem Between Keyboard And Blood Alcohol Content?
<wallyworld> wgrant: you should know by now i can't type :-)
<StevenK> lifeless: What you want to do with bug 721460?
<_mup_> Bug #721460: Cannot use 'lp:bzr' as a branch in a recipe <recipe> <Launchpad itself:Incomplete> < https://launchpad.net/bugs/721460 >
<StevenK> wgrant: Thoughts on bug 515918?
<_mup_> Bug #515918: package_name and suite not passed to dailydeb <lp-soyuz> <recipe> <Launchpad itself:Triaged> < https://launchpad.net/bugs/515918 >
<wgrant> wallyworld: I put some concrete suggestions in the bug... do you have any thoughts?
 * wallyworld looks at the bug
<wgrant> StevenK: I'm not sure.
<wgrant> StevenK: That needs testing.
<wallyworld> wgrant: it's not quite that simple. there are other reasons why the build now link won't show, besides the recipe being stale
<wallyworld> wgrant: eg if the user doesn't have upload permissions to the archive
<wgrant> wallyworld: Right, but one of those already has a warning at the top of the page.
<wgrant> And the other is a bug.
<wallyworld> hmmmm. if you can convince thumper to assign it to me, i'll do it :-) he had some definite ideas about how he wanted that to work
<wgrant> I only know of two other cases it won't show:
<wgrant>  1) No permissions, for which there is already a big warning at the top of the page.
<wgrant>  2) Not a daily build, for which we should show the link.
<wgrant> Because that's still useful even for non-daily builds.
<wgrant> eg. when updating stuff in the ~launchpad PPA... we don't want daily builds, but a single-click build request would be wonderful.
<wgrant> In fact, you are more likely to want the "Build now" button if you are not a daily build.
<wallyworld> there's already a "request builds" link for adhoc builds
<wgrant> But that is a more complex issue than merely presenting it better for daily builds.
<wgrant> wallyworld: Right, but these aren't ad-hoc builds.
<wallyworld> the daily "Build now" link has a specific purpose
<StevenK> wgrant: http://pastebin.ubuntu.com/572056/ from Hudson
<wgrant> wallyworld: Right, and I want that purpose.
<wallyworld> and you get it, no? but not if the recipe isn't stale
<wallyworld> which is what the requirement stated
<wgrant> StevenK: Which revs is that?
<wallyworld> so for the case "when updating stuff in the ~launchpad PPA,....", you wouls just use the "Request Builds" link?
<StevenK> wgrant: r12458
<wgrant> wallyworld: But it's already preconfigured.
<wgrant> StevenK: Are you sure?
<wgrant> That could have plausibly been introduced by r12460
<StevenK> wgrant: Er, a little
<wgrant> But it passes here.
<wallyworld> wgrant: by "it's", you mean the "Build now" link is already configured to do the same thing as a daily build would if it were to run?
<wgrant> wallyworld: The ~launchpad recipes want the same functionality as a daily build, except on request.
<LPCIBot> Project devel build (475): ABORTED in 25 min: https://hudson.wedontsleep.org/job/devel/475/
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][no-qa] Implement splitting Package/Product translations
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][bug=724004][no-qa][incr] Rename LP.client.cache and
<LPCIBot> LP.client.links to LP.cache and LP.links in preparation of
<LPCIBot> LP.client becoming a YUI module.
<LPCIBot> * Launchpad Patch Queue Manager: [r=mbp,
<LPCIBot> wgrant][ui=sinzui][bug=591274] Modified the bug tag field width so
<LPCIBot> that it is a more appropriate length for the content.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][bug=724039, 724232][no-qa] Prepopulate the cookie secret cache,
<LPCIBot> preventing tests from seeing an extra query when they are the first
<LPCIBot> in a layer. Also unbreak ec2 land and silence the import fascist.
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][bug=724522] Make clear_request_started() work when
<LPCIBot> called twice in a row, unbreaking request retrying.
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][bug=712329,
<LPCIBot> 712331] Add extra recipe methods to the web service.
<wgrant> wallyworld: Daily builds get to build daily, but they also now get this nice single-click button to do everything.
<StevenK> Sorry for the spam, I'm upgrading to Jenkins
<wallyworld> wgrant: and you get that but not if the recipe isn't stale :-)
<wallyworld> wgrant: so your issue is you want to request a daily build even if the recipe isn;t stale?
<wgrant> wallyworld: Even if it's not configured to build daily?
<wallyworld> ah, so when it's configured as " build on request", you want to trigger the build_daily() method
<wallyworld> ?
<wgrant> wallyworld: Yes.
<wallyworld> that's confusing perhaps?
<wgrant> The name is confusing, but the action is not.
<wgrant> I guess I don't understand why only daily builds get the one-click functionality.
<wallyworld> so the problem scope is larger then since the naming of the boolean choices for "build daily/build on request" would need rework etc
<wgrant> Right.
<wgrant> That's why I separated it from the daily build status issue.
<wgrant> Mostly.
<wallyworld> wgrant: i only came to recipes late so haven't got all the history of the requirements discussions etc
<wallyworld> so i can't really tell you for sure why certain decisions were made
<wallyworld> i can follow up with thumper
<wgrant> I was only really involved in the backend stuff, not the later frontend development :(
<wallyworld> sure, but here we're talking about the semantics of a recipe attribute
<thumper> wgrant: it is my suggestion, but I'm about to take the girls out to watch the rugby
<wgrant> thumper: What is your suggestion?
<lifeless> StevenK: I think its invalid
<wgrant> Won't Fix, if anything.
<wgrant> It is very valid.
<lifeless> StevenK: its definitely confusing on qastaging
 * StevenK waits for aptitude
<StevenK> Sigh, the jenkins package conflicts with hudson
<StevenK> This is not convincing me of a pain-free upgrade process
<StevenK> wgrant: You know, I think that rabbitmq install error is due to DEBCONF_FRONTEND=noninteractive
<wgrant> StevenK: That rabbitmq install error?
<StevenK> wgrant: When Jenkins installs rabbitmq on a slave it errors
<lifeless> :(
<wgrant> Ah.
<lifeless> StevenK: I have a branch using rabbit
<lifeless> so thats probably important to fix ;)
<wgrant> lifeless: It's not quite that easy to get rid of the buildd secret, sadly.
<wgrant> lifeless: apt still needs it.
<wgrant> Even if lp-buildd does not.
<lifeless> wgrant: theres an alternative anyhow.
<huwshimi> Have a nice friday everyone
<lifeless> wgrant: add and subscriber 'buildd' to all private ppas
<wgrant> Night huwshimi.
<wgrant> lifeless: Yeah.
<wgrant> lifeless: Do we have to use config-manager in PQM?
<wgrant> Can't we just use the precommit hook to pull download-cache?"
<wgrant> Without c-m it doesn't have to redownload the whole tree.
<StevenK> lifeless: It actually drops files on disk, the postinst fails
<wgrant> And it takes about 10 seconds instead of 20 minutes.
<lifeless> wgrant: yes
<StevenK> And I think it's because of noninteractive
<lifeless> wgrant: why wouldn't it need the whole tree?
<wgrant> lifeless: It already has the whole tree.
<wgrant> lifeless: eg. look at shipit merges.
<lifeless> wgrant: after the rm -rf  ?
<wgrant> lifeless: Why would it rm -rf?
<lifeless> because it doesn't trust the build before
<lifeless> it might have been hostile
<wgrant> shipit merges have most of the history, and take like 2 minutes.
<wgrant> Ah, not 2 minutes, 25 seconds!
<wgrant> 25 second PQM!
<lifeless> config-manager can do a lightweight checkout
<lifeless> the thing thats slow is grabbing the whole history because *make check* uses bzr operations
<lifeless> fix *that* and we can switch to lightweight checkouts
<wgrant>         [ `PYTHONPATH= bzr status -S database/schema/ | \
<wgrant>                 grep -v "\(^P\|pending\|security.cfg\|Makefile\|unautovacuumable\|_pythonpath.py\)" | wc -l` -eq 0 ]
<wgrant> That's the only bzr operation I can see.
<lifeless> right
<lifeless> it opens the tree
<lifeless> the tree opens the branch
<lifeless> the branch opens the repo
<wgrant> And the lightweight checkout works.
<lifeless> it wants the pending head revision to show the last commit
<lifeless> wgrant: not once you chroot
<lifeless> wgrant: the ssh key for lp being outside the chroot
<lifeless> (and likewise the local disk copy in ~/archives/rocketfuel being outside the chroot)
<wgrant> lifeless: :(
<spiv> lifeless: I'm late to this discussion, and have to hit the road in a sec, but stacked commit would help there.
<wgrant> lifeless: We could still make it approximately lots of times faster by just cping the tree across...
<lifeless> wgrant: except we don't start from a known good treee and we don't have any operation to give us one reliably.
<StevenK> But the local disk might be hostile!
<wgrant> lifeless: Hm? The on-disk copy is good.
<lifeless> spiv: stacked branches still open the parent
<lifeless> wgrant: I thought you meant the build tree
<wgrant> lifeless: If PQM's checkout outside the chroot is fucked, we are all fucked.
<spiv> Hmm, even for 'bzr status'?  That might be fixable.
<lifeless> wgrant: because we're not lightweighting it, *all* of sourcecode is copied.
<wgrant> lifeless: Right. And if we can't lightweight it, then we should just cp -a the tree into the chroot.
<lifeless> spiv: mixed blessing to change that
<wgrant> Then we can stop running buildout.
<wgrant> And have 30s PQM.
<lifeless> wgrant: its more complex than that; its in a shared repo
<wgrant> lifeless: That can be fixed.
<wgrant> Or the shared repo can be copied.
<lifeless> wgrant: and we don't want information leakage of embargoed revisions if someone gets smart
<lifeless> wgrant: its really a lot simpler to stop calling into bzr
<wgrant> lifeless: You know...
<wgrant> We don't need to chroot, really.
<wgrant> We don't need buildout.
<wgrant> We don't need to run any code from the tree.
<wgrant> We just need to run bzr st, check that no DB stuff is modified, and grep for conflict markers.
<wgrant> That's all check_merge does now.
<lifeless> so
<lifeless> change pqm
<lifeless> it does all of that except the db check
<wgrant> Just need to change the config, even.
<lifeless> add a feature to exclude changes to a subtree, add that to the config for devel, done.
<wgrant> We could just add the grep to the precommit hook.
<wgrant> It's sitting in Makefile at the moment.
<lifeless> wgrant: thats possible too
<lifeless> wgrant: and the pqm precommit hook runs outside the chroot
<wgrant> lifeless: Exactly.
<wgrant> No PQM code change required.
<wgrant> But no chroot required.
<LPCIBot> Project db-devel build (394): STILL FAILING in 2 min 9 sec: https://hudson.wedontsleep.org/job/db-devel/394/
<wgrant> And no config-manager required.
<wgrant> So no half-hour required!
<wgrant> I'm not sure if we want the conflict marker test still.
 * StevenK peers at Jenkins
<wgrant> We would lose the guarantee that buildout works.
<wgrant> But I think that is a reasonable sacrifice for 60x faster landing.
<lifeless> buildbot will tell us that
<StevenK> Uhhh
<StevenK> $ bzr branch http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel /mnt/test/launchpad/workspace/db-devel
<StevenK> bzr: ERROR: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/.bzr/repository/packs/09fb66d3dfb9df78e2bb1ca7b6c7aac9.pack is redirected to https://launchpad.net
<StevenK> ERROR: Failed to clone http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel
<lifeless> \o/
<lifeless> also wtf
<lifeless> i guess its a 404
<StevenK> It works now ...
<lifeless> so you hit a repack
<lifeless> (during a)
<lifeless> file a bug
<spiv> StevenK: what lifeless said
<spiv> StevenK: especially \o/
<StevenK> It was repacking while downloading?
<lifeless> pqm did it, yes
<spiv> StevenK: concurrent push while you pulled
<StevenK> It's a bzr bug or codehosting?
<spiv> I'd say both :)
<lifeless> codehosting
<spiv> Mainly codehostin
<lifeless> bzr can't really sensible handle a redirect there
<lifeless> because it points at not-a-pack
<spiv> But I think it would be ok for bzr to understand that a redirect while reading a pack indicates "probably not there"; if bzr has gotten as far as trying to read packs it's already successfully fetched a bunch of other stuff under .bzr/
<lifeless> spiv: 'ewww'
<spiv> It might be too nasty to code though.
 * lifeless puts on the http nazi hat
<StevenK> lifeless: https://bugs.launchpad.net/launchpad/+bug/724750
<_mup_> Bug #724750: repacking during checkout results in wierd error <Launchpad itself:New> < https://launchpad.net/bugs/724750 >
<lifeless> StevenK: go on, triage it.
<StevenK> Bah, sorry, forget to set status and importance
<lifeless> :)
<lifeless> StevenK: jinx
<StevenK> Triaged/Low. Not sure if you agree
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-710685/+merge/51258
<lifeless> StevenK: I agree
<lifeless> ^ thats I can has review, plox
<StevenK> Waiting for diff
<wgrant> What.
<StevenK> wgrant: Hm?
<wgrant> lib/lp/tests/test_no_conflict_marker.py
<wgrant> 49 lines of Python wrapped in two of make wrapped in one of shell.
<wgrant> To run one line of shell.
<StevenK> EEEEEK
<StevenK> lifeless: You stop importing TestCase in lib/lp/testing/tests/test_matchers.py and one class still bases off it?
<StevenK> lifeless: Your multi imports require ( and )
<StevenK> lifeless: You didn't drop the unittest import from lib/lp/code/browser/tests/test_branch.py
<StevenK> As well as lib/lp/code/browser/decorations.py
<StevenK> I'm not sure about the move into setupBrowserForUser(), but I'm willing to humour you
<wgrant> https://pastebin.canonical.com/43935/
<wgrant> I think that does the same as we do now, except without the test that buildout works, because I hate buildout.
<lifeless> StevenK: I don't stop importing TestCase
<StevenK> Ah. Fix your imports anyway
<StevenK> :-)
<lifeless> where are you saying () is needed ?
<lifeless> I've removed the two useless imports of unittest
<lifeless> StevenK: ^
<StevenK> lifeless: from lp.testing import TestCase, TestCaseWithFactory , for instance.
<lifeless> blah
<lifeless> I *actually* intended that to be ok when I argued about this
<lifeless> but the nuance got lost somewhere
<lifeless> changed
<StevenK> Just run the import formatter and stress less
<lifeless> StevenK: anything else ?
<StevenK> lifeless: Not that I can think of right now
<lifeless> man, oracle are really not dealing with hudson stuff well
<wgrant> Oh?
<StevenK> Oh?
<wgrant> What have they broken now?
<lifeless> the long thread with charles rhys
<wgrant> lifeless: So, any flaws in my PQM plan?
<wgrant> I admit to not knowing PQM very well at all.
<lifeless> wgrant: pqm already rejects conflicts
<wgrant> lifeless: Sure, but people sometimes commit them.
<wgrant> I've seen them in MPs :/
<wgrant> But we could probably drop that check, true.
<lifeless> other than that, it seems fine
<lifeless> while you're at it
<lifeless> care to look at why prod config landings take an hour
<wgrant> Same reason.
<wgrant> They grab and build LP, then verify all of the configs against the schema.
<wgrant> We could probably install lazr.config on prasÃ© and do it without using the tree, though.
<wgrant> Let's see how easy that would be.
<StevenK> lifeless: You missed the imports in lib/lp/registry/browser/tests/test_milestone.py
<wgrant> Mmm, not really feasible.
<wgrant> We may have to live with long prod configs for a while :(
<lifeless> StevenK: which imports
<StevenK> from testtools.matchers import LessThan, Matcher ; from lp.testing.matchers import HasQueryCount, BrowsesWithQueryLimit
 * wgrant foods.
<wgrant> I will return and request a deploy to finish off that incident.
<wgrant> Unless buildbot chokes on Windmill again, in which case I will be tempted to rs=me disablement of WindmillLayer.
<StevenK> Haha
<lifeless> StevenK: thanks
<StevenK> lifeless: If you've made that change, r=me
<lifeless> I have
<lifeless> needs a non-* review
<StevenK> Indeed
<StevenK> Can't help you, sadly
<lifeless> oh, there is a stub
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-710685/+merge/51258
<wgrant> Yay.
<lifeless> stub: could I get a review of https://code.launchpad.net/~lifeless/launchpad/bug-710685/+merge/51258
<stub> lifeless: yer - looking already.
<lifeless> stub: cool, thanks
<stub> lifeless: I don't particularly like how you are having to sniff params.linked_branches for type in model code (browser code I could forgive when things are coming from HTML forms).
<lifeless> stub: it doesn't make sense to have a different parameter
<lifeless> stub: I don't like the sniff either, but the original linked_branches code was bong
<lifeless> it should have used not(None) and None with a normal searchbuilder right from the get go.
<lifeless> stub: we can fix that up separately
<stub> lifeless: So what is zope_isinstance(params.linked_branches, (any, all, int)) doing? any and all are builtin functions
<lifeless> stub: not here they aren't
<lifeless> canonical.launchpad.searchbuilder
<lifeless> this is the standard way for passing in constraints to bug searches
<stub> I see now.
<stub> Might be better to use a namespace since we are overriding builtins (searchbuilder.any)
<lifeless> stub: perhaps; its used bare /throughout/ bugtask.py though
<lifeless> stub: what I mean is, it wouldn't reduce the burden of that override at all to do different here
<stub> yup. Just thinking about 'best practice', and if it really matters (they may be builtins, but does anyone use 'em?)
<lifeless> and it would be inconsistent
<lifeless> I think we should improve it - perhaps move to Any rather than any, as a separate, large mechanical change.
<stub> lifeless: Ok. Better sooner than later though, as people will start cargo culting this code.
<lifeless> ah, the use of it in browser code I can change easily.
<lifeless> and will do so now
<lifeless> done
<stub> lifeless: line 139 of the diff, you should use shortlist rather than list
<lifeless> stub: doesn't that blow up at 1000
<stub> (Not that I'm sure why you need to materialize the result set apart from doing what the docstring says)
<stub> lifeless: It should emit warnings for smaller lists than that.
<lifeless> stub: we're doing this because we have bugs with 400+ bugs
<stub> lifeless: But if we are materializing large lists, we have a different problem.
<lifeless> branches with 400_ bugs
<lifeless> stub: yah, I'm wondering about paginating these
<lifeless> I can put shortlist in there
<stub> lifeless: You can specify the expected sane limit
<lifeless> where does it live?
<stub> canonical.launchpad.helpers (for now...)
<lifeless> I'll run with the default
<lifeless> an oops will be no worse than the current timeouts
<stub> lifeless: So 'linked_bugtasks' has a docstring stating it returns DecoratedBugs
<lifeless> oh, shall fix.
<lifeless> I got kindof mired in the restructuring mess here - was tired at the end
<stub> lifeless: And further down, you do 'for bugtask in self.branch.linked_bugs'
<lifeless> hah - UserWarning: shortlist() should not be used here. It's meant to listify sequences with no more than 15 items.  There were 36 items.
<stub> shortlist should be emitting soft oopses now soft oopses exist.
<stub> But the goal is good :)
<lifeless> i've given it a 1000 expected length.
<lifeless> thats a great point.
<lifeless> fixing that bugrask in linked_bugs code -   http://pastebin.com/bw7Cv4UM
<lifeless> it is late evaluation
<lifeless> but AFAICT that code path isn't actually used anymore
<lifeless> I just haven't tracked down every case to be sure of it.
<stub> that looks fine
<stub> What does self.assertThat(branch, browses_under_limit) actually check? I'm not sure how a branch can be used to check a query limit.
<lifeless> it renders the default view for the branch
<lifeless> and checks the queries made doing that
<lifeless> it uses userBrowser
<lifeless> so we get the full stack, all the portlets etc
<lifeless> see the matcher, its new in the branch - it wraps a QueryCollector + HasQueryCounts
<jtv> hi folks
<stub> jtv: Yo. I need you to better describe how you want those tests to look so I can land that branch.
<jtv> stub: you could have a pruner that takes a list of ids for its constructorâ¦ makes it easy to create pruners that will or won't clean up specific objects.  Or you could do it on some other specific attribute value.
<jtv> Then you can do things like "create 3 objects that will get cleaned up; run one pruner iteration with a batch size of 2; now 2 are gone but one's still there."
<jtv> Presto: no reliance on sample data.
<stub> I'm not relying on sample data
<jtv> I thought you were?
<jtv> The IrcID thingies?
<stub> Last iteration changed. Instead of using a random table with some arbitrary data (IrcId), I create a new table with some arbitrary data (FooSomething)
<jtv> Oh cool
<jtv> (Meanwhile, I'm browsing to the MP)
<stub> The data never mattered - I just needed a table with some data in it (> chunksize items actually)
<stub> lifeless: we probably can do the 'filter unnecessary bugtasks' in SQL, but I doubt it will be much of a win over the current approach. Not dealing with large numbers of items to prune here.
<jtv> Wow that's a large indentation.  Wasn't it easier to put the method body in a separate method, and wrap the call to that method in a try/finally?
<lifeless> stub: i think we should leave that for a later iteration
<lifeless> stub: *but* - some bugs have 100's of tasks
<lifeless> pathological ones to be sure
<stub> lifeless: Yes. I'm suggesting that later iteration might never happen.
<stub> Outliers, that are probably indicative of another bug (something is wrong if a bug has 100's of tasks)
<lifeless> stub: indeed. Lets be guided by data :)
<lifeless> stub: yes, something is wrong... we have users
<stub> Status: 666 (Electrocute Luser)
<lifeless> in fact looking at this I see another possible bug, Should we really show 'wontfix', 'invalid' and 'opinion' bugs
<lifeless> (we do currently, but it seems like an easy way to deal with these pathological bugs, just don't bring back those tasks.
<stub> lifeless: This is model code, so you really shouldn't be making decisions for the browser code here.
<lifeless> stub: the browser code passes the filter in
<lifeless> stub: I'm saying the filter may be bong
<lifeless> I've preserved the behaviour, but its not necessarily defined correctly to start with
<stub> Ok. So different, existing bug :)
<jtv> stub: big improvement in that you no longer rely on pre-existing itemsâ¦ is it safe to make BulkFoo a non-temp table?
<stub> jtv: Yes, as the test suite will clean everything up. But I can't see why I can't use a temporary table in any case.
<jtv> stub: that's the thingâyou're not using a temp table
<lifeless> temp table would be faster to test with
<jtv> Typo?
<stub> I know. By double negative was attempting to say I can change that without ill effects.
<jtv> Ah sorry
<jtv> Seems cleaner to make it a temp table and have cleanUp drop it.
<jtv> If the base class didn't do the query, this would be easy to unit-test without a table; you could just pass a list.
<stub> jtv: I need a table so I can define a Storm class so I can add it as an attribute to the BulkPruner
<jtv> Ah, the bulkpruner also does the delete in SQL of course.  Stupid of me.
<jtv> I'll have a coffee after this.
<stub> Get me one too please. My eyes blur over when I look at lifeless's filter code.
<jtv> For that, I recommend beer.
<jtv> (Any other religions we can offend with simple beverages?)
<stub> ok. I see why it is done that way and how it works now.
<stub> I don't need props to offend religions
 * stub is naturally offensive
<jtv> Plus, you have a T-shirt!
<adeuring> good morning
<jtv> stub: I still think it'd help to break that test down into chunks that prove separate things.  "isDone initially returns zero."  "A run with a batch size of 2 removes exactly 2 objects."  "Iterating until isDone() clears up everything."  (I don't see why there are two of those bits in the test.)
<stub> jtv: I don't think the test is particularly large or unwieldy, and I know every test is going to take about 1.5 seconds as the database will need to be reset. But I can change that if you insist.
<stub> lifeless: approved
<lifeless> stub: thanks
<jtv> stub: that long?
<jtv> Where does the time go?
<stub> Last I checked. dropdb, createdb --template (in a loop, waiting for dropdb to complete and pg to complete)
<stub> Although if I use a temporary table, I can tell the test suite the db isn't dirty and avoid that
<lifeless> stub: you can use a temp table with storm
<stub> So yes, we can split the tests without time increases with a little magic.
<jtv> Maybe try the timing with the temp table first.  :)
<stub> lifeless: Yes, but I'm defining a Storm subclass to access it.
<lifeless> stub: sure, I don't understand why that implies any trauma
<stub> lifeless: Ideally I could create a Storm class on the fly but I don't know if I can.
<lifeless> stub: you can - curry it in your test
<jtv> AFAIR you can
<lifeless> [not that you need to, but you definitely can]
<stub> I mean easily. temporary_table = StormTempTable("SELECT * FROM FooBar WHERE wibble IS FALSE")
<lifeless> I was thinking - definte the temp class once
<lifeless> do store.execute() once in the test to create the temp table
<stub> lifeless: That is what I'm doing. I just need to add 'TEMPORARY' to my 'CREATE TABLE' statement in the test.
<lifeless> cool
<lifeless> doesn't sound magic then :)
<mrevell> hi
<jtv> hi mrevell!
<lifeless> \o/ profiling on qastaging. -win-
<wgrant> lifeless: Do they go somewhere useful now?
<jtv> lifeless: don't play with our feelingsâ¦ seriously?
<lifeless> wgrant: I suspect the rsync isn't in cron or whatever yet
<lifeless> wgrant: will worry about that when losas are not sprinting
<jtv> Do we still need to get the config swizzled in order to use them?
<lifeless> jtv: no
<lifeless> jtv: just need to be in team launchpad
<jtv> \o/
<lifeless> jtv: it will still lock every other concurrent request out
<jtv> so not something we're going to do in production then
<lifeless> (or it should anyhow...) - but thats a profiling limitation
<lifeless> we could actually, once we're single threaded in production
<jtv> eh :)
<lifeless> thats coming with the new servers
<lifeless> and haproxy reconfiguration
<lifeless> jtv: anyhow - https://bugs.qastaging.launchpad.net/++profile++show/ubuntu/+bug/1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<stub> jtv: This really isn't better. I've replaced a simple, linear test with several less simple tests for a net increase in complexity and decrease in readability
<jtv> stub: oh well, at least you gave it a try.  Thanks for thatâI won't hold you up any longer.
<lifeless> wow, 1/3 of a second in 0.3811      0.0110   +storm.references:133(__get__)
<jtv> lifeless: yes, they're expensive even when they come out of the cache.
<lifeless> jtv: I knew that in principle, but I didn't have data to support it
<lifeless> this isn't conclusive yet either because it may still be lazy loading on that page
<jtv> You could have asked.  :)  I fixed a timeout recently that was largely based on that.
<lifeless> jtv: I remember the review, I forget the page id
<jtv> So do I, unfortunatelyâbut IIRC it was more or less two consecutive lines in the code, with ridiculous time spent python-side without queries.
<jtv> Sometimes I wonder if it'd be worthwhile to have per-request caches in front of the object-info dict in storm.
<wgrant> lifeless: OK, ++show is extremely awesome.
<jtv> What's that do?
<lifeless> jtv: click on the url I linked before
<lifeless> wgrant: that was gary's magic
<lifeless> I'll let everyone know about this and the improving scaling count helper on monday
<jtv> Oh wait I've seen this before
<jtv> Didn't realize you got the page underneath as well though.  Awesome!
<jtv> I wonder if I could hack up my browser to add a "profile this request" link to all my LP pagesâ¦
<wgrant> jtv: You were getting 502s from that error, right?
<stub> jtv: Can you push the relevant buttons on https://code.launchpad.net/~stub/launchpad/garbo/+merge/50595 to keep ec2 land happy?
<jtv> wgrant: from what error, sorry?
<wgrant> jtv: The one you're discussing in #launchpad
<jtv> stub:  I was in the process of doing that, actually.
<jtv> wgrant: ah, they were getting those non-oops "can't connect to Launchpad server" errors.
<jtv> stub: see?  It's people like wgrant keeping me from approving your work.
<wgrant> jtv: Yeah, the 502s. They're fixed now.
<jtv> Great!
<wgrant> But they do indicate an IntegrityError.
<jtv> Which is exactly what I currently have a fix in EC2 for.  And thanks for fixing "ec2 land" btw.  :)
<wgrant> It finally landed.
<wgrant> After Windmill kicked it out twice.
<jtv> Grr
<wgrant> and a new spurious failure (now fixed) kicked it out a third time.
<wgrant> r12426 started creating the no-op filters.
<wgrant> r12435 is the only rev since that affects bug notifications.
<lifeless> night all
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: adeuring | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> adeuring: Hi.
<adeuring> hi wgrant
<wgrant> adeuring: https://code.launchpad.net/~wgrant/launchpad/remove-canonical.widgets/+merge/51277 should be a nice quick one.
<wgrant> If you could.
<adeuring> wgrant: I'll look
<adeuring> wgrant: r=me.
<wgrant> adeuring, jml: Thanks.
<jml> jelmer: thanks for the email.
<jelmer> jml: np. Let's get this show on the road :)
<LPCIBot> Project devel build (476): STILL FAILING in 5 hr 44 min: https://hudson.wedontsleep.org/job/devel/476/
<deryck> Morning, all.
 * jml is off for a bit, last driving lesson in a while, I hope.
<jtv> good luck jml, good morning deryck
<jtv> Ahhhh an empty translations import queue for once.  Lovely.
<danilo> stub, hi, just one q re your review of https://code.launchpad.net/~danilo/launchpad/bug-720826-db/+merge/51146
<danilo> stub, we are mostly going to do joins through bugnotification, is _pkey as you suggest going to be enough for that?
<stub> danilo: yes
<danilo> stub, (fwiw, we might not even need the index on bug_subscription_filter, so how bad for the DB is it to add it "just in case"?)
<danilo> stub, ah, interesting, so the BTREE index on a tuple will be a BTREE on the first element of the tuple "first"?
<stub> there is a good chance we won't need it, as the primary key index should be usable going in that direction too.
<stub> but it will be faster, and just in case.
<stub> danilo: I don't know the details of the structure.
<danilo> stub, well, what I am saying that we are not going to be joining through this table from the direction of a bugsubscriptionfilter (iow, we won't be taking a set of bugsubscriptionfilters and look for all bugnotifications that it caused; at least not for now)
<stub> So that is two reasons we might not need it :) I just put it there because your patch had it too.
<danilo> stub, right, I just put it in for "just in case" (i.e. we might want to do that when we start trying to estimate how much mails is each filter generating)
<danilo> stub, but I wonder if it is a drag on the DB and we shouldn't create such indexes "just in case" (iow, I am asking how bad is it to have an index that's not used?)
<stub> It slows down inserts and chews some disk space, apart from that not much.
<danilo> stub, right, thanks
<danilo> stub, I think I'll leave this one in
<stub> If the filters can be removed, we want the index.
<LPCIBot> Project db-devel build (395): STILL FAILING in 5 hr 50 min: https://hudson.wedontsleep.org/job/db-devel/395/
<danilo> stub, right, good point
 * jtv wants hypothetical indexes
<danilo> jtv, I want a hypothetical database ("if you need it, it'll be there" :)
<jtv> Go buy hardware
<danilo> jtv, hypothetical hardware?
<jtv> hypotheticall, yes
<danilo> jtv, there's not enough "all" for what we might need :)
<LPCIBot> Project devel build (477): ABORTED in 1 hr 12 min: https://hudson.wedontsleep.org/job/devel/477/
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][bug=721134] Fix the heading alignment for the multi-line
<LPCIBot> editors on the recipe index page.
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: adeuring, bac | https://code.launchpad.net/launchpad-project/+activereviews
<bac> hi adeuring
<adeuring> morning bac
<bigjools> adeuring or bac, no rush, but I have a branch up for review if one of you can take it please
<adeuring> bigjools: I'll look
<bigjools> thank you
<sinzui> So begins another day where the first 1.5 hours is spent working natty bugs
<bigjools> natty is not yet released, right? :)
<sinzui> I do not think people will be happy if it were released today
<jcsackett> sinzui: you free to mumble about the messages stuff in about half an hour?
<sinzui> sure
<sinzui> if mumble/pulse works
<sinzui> I had to fix the mic again this morning
<adeuring> bigjools: PoppyAnonymousShell.openForWriting() has a "catch-all" except. I think we do not want to catch KeyboardInterrupt and SystemExit. And I think we should log an error for other exceptions.
<adeuring> otherwise, the branch looks good, I think
<bigjools> adeuring: that was cargo-culted from the parent class... I wasn't sure what to do about it TBH
<jcsackett> sinzui: dig. i think all meeting plans these days involve a "natty-errors" fudge factor. :-P
<adeuring> bigjools: admittedly, I am not familiar with twisted, so I am not sure about KBInterrupt and SystemExit. But nevertheless I think we should log other exceptions.
<adeuring> "eating" exceptions simply scares me ;)
<bigjools> adeuring: yeah
<bigjools> adeuring: so I'll re-raise KeyboardInterrupt and SystemExit
<bigjools> adeuring: oh wait
<bigjools> adeuring: yes twisted has confused us both momentarily - the defer.fail() is effectively the same as re-raising the exception, it gets dealt with later
<adeuring> bigjools: intersting. So,  IOError and OSError are re-raised too?
<adeuring> ah, no, there no fail() call
<bigjools> adeuring: no, they are converted to ftp errors
<bigjools> ftp.errnoToFailure will return a defer.fail
<adeuring> bigjools: right, thanks for the explanation, so r=me. But perhaps a small comment that twisted properly deals with the exceptions? (just to avoid the same question I asked again)?
<bigjools> adeuring: yup, will do, thanks for the review
<bigjools> I am blocked on landing this right now anyway
<bigjools> talking of which, does it matter if we lay our own eggs in lieu of a proper Twisted release?
<jml> gary_poster: hi
<gary_poster> jml, hi on call but maybe will postpone call if your conversation will change the tenor of the call :-)  what is your topic?
<jml> gary_poster: general mgmt stuff. It can wait.
<gary_poster> ok cool will ping jml thanks
<jml> gary_poster: np
 * jml fetches tea & chocolate.
<gary_poster> jml, ping?
<jml> gary_poster: hi.
<jml> gary_poster: skype? (mumble's broken for me atm)
<gary_poster> hey.  Skype or mumble?
<gary_poster> ok
<abentley> jkakar: I'd like to be notified via a Zope event whenever objects of a particular type are removed from the store.  It looks like Storm does emit an event then, but I can't find any documentation of how that works.
<abentley> jkakar: Do you have any pointers?
<jkakar> Hmm, let me look at some code, one sec.
<jkakar> abentley: Okay, so the only thing I can recommend is not really part of the public Storm library.
<jkakar> abentley: I wouldn't rely on it not changing under you.
<jkakar> abentley: Anyway, Storm has an internal event system which it uses to keep track of things.  You can hook into it, but as I said, it's not recommended.
<abentley> jkakar: understood.  I already have a 95% solution, by overriding Foo.destroySelf.  Maybe that's enough.
<jkakar> abentley: That's what I was going to recommend... providing a wrapper/hook on the public API.
<abentley> jkakar: Yeah, it just wouldn't surprise me if someone did Store.remove instead of destroySelf at some point.
<abentley> jkakar: It's more Storm-y.
<jcsackett> sinzui: mumble?
<jkakar> abentley: Right.
<abentley> jkakar: Perhaps you could put a docstring in storm.event that says it's for internal use only?
<jkakar> abentley: That's a good idea.  We desperately need to improve our documentation system... no one has taken up the charge to do it, though... I've made some small starts, but haven't managed to push through.
<benji> anyone done a rockefuel-get lately and then not been able to make?  SyntaxError: invalid syntax (shipit.py, line 51)
<benji> hmm, conflict markers in the file, trying to see if it's just a local problem or not
<henninge> adeuring, bac: It would be great if either of you could look at my branch! It's UI work.
<henninge> https://code.launchpad.net/~henninge/launchpad/devel-705176-sharing-info-on-page/+merge/51283
<henninge> sinzui: Can you do a UI review on that, please? ^
<sinzui> okay
<adeuring> henninge: I'll look
<henninge> adeuring, sinzui: thanks. I gotta run out now for 30 min but will be back to answer any questions ... ;-)
 * henninge is back
<henninge> adeuring: are you still reviewing?
<adeuring> henninge: nearly done, give me five more minutes
 * henninge sets count down on watch ;)
<henninge> adeuring: cool, thanks ;)
<adeuring> henninge: r=me with a few nitpicks
<danilo> allenap, hey, sorry for missing out on that bit of extra review you asked for, had a nice, wild day â +1 for ignoring my lazy ass :)
<allenap> danilo: :)
<henninge> adeuring: Danke! SchÃ¶nes Wochenende!
<danilo> allenap, fwiw, if you run into problems, you can always change the user to be defined as group=launchpad when it inherits all the permissions of the "launchpad" user (though, you might know that already, but I am just saying :)
<adeuring> henninge: Dir auch auch ein schÃ¶nes Wochenede!
<lifeless> moin
<sinzui> looks like I win. I get to try a bug import
<LPCIBot> Project db-devel build #396: STILL FAILING in 5 hr 7 min: https://hudson.wedontsleep.org/job/db-devel/396/
 * jml has to go
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: bac | https://code.launchpad.net/launchpad-project/+activereviews
<jcsackett> are we still having ec2 problems? i just got back three ec2 runs which seem to have taken the full time, but show 0 tests run, 0 errors, 0 failures. log looks like everything actually ran.
 * jcsackett is confused.
<lifeless> jcsackett: hi
<lifeless> jcsackett: windmill
<jcsackett> lifeless: oh goodie.
<jcsackett> (ah, the problems with splitpanes)
<lifeless> I prefer F16's myself
 * jcsackett laughs.
<jcsackett> so, we're having windmill issues in the tree, still, or i have somehow blown up windmill in my branch?
<lifeless> bit from column A
<lifeless> bit from column B
<jcsackett> righto.
<lifeless> shiny: http://caniuse.com/
<lifeless> sinzui: ping
<sinzui> hi lifeless
<lifeless> hi
<lifeless> I'd like to mark bug 719182 wontfix
<_mup_> Bug #719182: boolean interpretation for feature flags <feature-flags> <Launchpad itself:In Progress by sinzui> < https://launchpad.net/bugs/719182 >
<lifeless> but I just noticed you're working on it
<lifeless> and I don't want to be a nuisance
<lifeless> the problem statement at the top is false: we don't have varying interpretations for booleans; that all got cleaned up by wgrants work
<LPCIBot> Project devel build #478: STILL FAILING in 5 hr 52 min: https://hudson.wedontsleep.org/job/devel/478/
<sinzui> fab
<lifeless> thats ok with you ?
<sinzui> Indeed it is.
<lifeless> cool. I'll twiddle the knobs.
<gary_poster> lifeless do you happen to know if there is a way to use the WITH statement in Storm?  Mucking about with SQL, I can get the 1200-1500ms query from bug 723999 down to 180ms simply, and if I can use the WITH statement somehow I can get it down to 50ms.
<_mup_> Bug #723999: BugNomination:+edit-form structural subscriptions taking 4.8 seconds during nomination editing POST <story-better-bug-notification> <timeout> <Launchpad itself:Triaged by yellow> < https://launchpad.net/bugs/723999 >
<lifeless> gary_poster: thats a decent improvement :)
<gary_poster> :-)
<lifeless> gary_poster: The short answer is 'probably not'
<gary_poster> :-(
<gary_poster> oh well
<gary_poster> 180 is the big improvement, so I'll run with that
<gary_poster> thanks
<lifeless> storms compiler will put the SELECT at the start
<gary_poster> :-/
<lifeless> so we'd need to hack into that to actually support with (or similar) directly.
<lifeless> you could use a temp table explicitly
<gary_poster> how would I do that?
<gary_poster> (pointing to an example or telling me what to grep for would be great)
<lifeless> have you seen gavins hack to do distinct on ?
<gary_poster> I don't think so
<lifeless> SQL("distinct on foo 0 as ignore")
<gary_poster> it didn't stivk with me if so
<lifeless> and use that as the first thing in the select clause
<lifeless> store.find((SQL("distinct on foo 0 as ignore"), Foo, Bar), ...)
<lifeless> for _, foo, bar in result:
<gary_poster> not entirely sure I follow but will grep to see if that helps
<lifeless> you can probably inject a INTO TEMPORARY TABLE (I forget the syntax offhand ...) in the same way
<lifeless> however
<gary_poster> ah ok
 * gary_poster waits for however ;-)
<lifeless> then you've make the temp table, and use it in followups
<lifeless> oh the however: 180*4=720ms, which is tolerable for a month or two
<gary_poster> sure
<gary_poster> since I'm focusing on it I want to see if I can get it down
<gary_poster> 'cause 200ms is even better ;-)
<lifeless> I am a little scared by the size of query, and if you have any concerns about the modelling that you inherited with this project, I stand ready to consult
<lifeless> yeah, 200ms is even better
<lifeless> if the same basic temp table is used by all 4 queries
<gary_poster> the query is down to, say, 25 lines lifeless.  no worries there
<lifeless> then a temp table can be an even bigger win than with - make it once, use it 4 times
<gary_poster> (25 nicely formatted lines)
<lifeless> gary_poster: \o/ that will be a significant benefit all of its own.
<gary_poster> yeah
<gary_poster> ok, I'll read up on temporary tables; I didn't realize you could use them within a transaction
<lifeless> oh yeah, they are awesome
<lifeless> variables FTW
<gary_poster> :-)
<abentley> deryck: could you have a look at https://code.launchpad.net/~abentley/launchpad/refactor-merge-job/+merge/51359 ?  I may have used too much magic.
<gary_poster> ok thank you very much lifeless.  I'll dig, and come back up for air with you if I encounter confusion.
<lifeless> cool
<deryck> abentley, sure.  I'll take a look.
<abentley> deryck: thanks.
<abentley> bac: Could you please review https://code.launchpad.net/~abentley/launchpad/refactor-merge-job/+merge/51359 ?
<lifeless> jcsackett: you wanted to talk about spam
<jcsackett> yes, but i think it's no longer necessary.
<lifeless> ?
<jcsackett> you were concerned about performance implications for pushing the visible attr down into imessage and having multiple message systems use it, yeah?
<lifeless> yes
<lifeless> message is fat, link tables are pretty darn narrow
<lifeless> we probably need an overhaul on message
<lifeless> e.g. drop the fti index
<jcsackett> right. i'm thinking as far as not having to rewrite code too much, going for an IHideable or similar, and having shared rules there. it means some reimplemenation, but at least reliably linked.
<bac> abentley: yes
<jcsackett> rather than questions being entirely different from bugs, and so forth.
<lifeless> jcsackett: bugmessage is 188MB, message is 2.4GB
<abentley> bac: thanks!
<lifeless> jcsackett: postgresql has to read every row that indexes say *might* be a match, because the liveness of the row is only in the row
<jcsackett> lifeless: right.
<lifeless> jcsackett: this doesn't mean we /can't/ put it on message, but it does mean we need to try the migration, measure the impact
<lifeless> jcsackett: it -might- be faster than it is now. It might be slower.
<jcsackett> lifeless: yeah, i'm going to give it a try.
<lifeless> jcsackett: what I know *for sure* is that we have issues in this area, and we doing things that are hard to undo (e.g. schema schanges) should be done with care.
<lifeless> jcsackett: as a test case, i give you http://bugs.launchpad.net/bugs/1
<lifeless> oops
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> jcsackett: https://bugs.launchpad.net/ubuntu/+bug/1?comments=all
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<lifeless> jcsackett: that url is (now) (largely) driven by messaging system performance
<jcsackett> i'm still impressed the non-all comments version renders, btw. :-P
<lifeless> jcsackett: actually, thats a little unfair of me : we're looking at pagination - but still - the count(*) in a batchnavigator isn't free
<jcsackett> no indeed.
<jcsackett> i'll probably getting into that stuff more on monday. today has been hunting down permissions stuff, since the visible attribute being used for culling spam on bug comments is admin only, and we would like registry to be able to deal with it.
<lifeless> jcsackett: its not reliable yet, I need to keep pushing it
<bac> abentley: why do you use a staticmethod for _register_subclass instead of a classmethod?
<abentley> bac: it had to be invoked in TranslationPackagingJob, and that didn't work with a classmethod.
<abentley> bac: There's no syntax for "invoke this class's classmethod, but with this other class as the first parameter."
 * bac looks
<lifeless> that would be a definitional problem - unless the otherclass is a subclass
<abentley> lifeless: otherclass is indeed a subclass.
<lifeless> abentley: does it define its own version of that method ?
<abentley> lifeless: yes.
<lifeless> ah
<lifeless> that will indeed be tricky
<abentley> bac: The follow-on is here: https://code.launchpad.net/~abentley/launchpad/translation-splitting-job/+merge/51371
<abentley> bac: You can see that these refactorings make the actual definition of TranslationSplitJob very small.
<abentley> bac: just noticed schedule_merge needed to be deleted.  Pushed now.
<abentley> bac: thanks for your review.  Updates pushed.
<lifeless> also, I hate sample data
<lifeless> lib/lp/soyuz/doc/closing-bugs-from-changelogs.txt is going boom because of the dsp.currentrelease change to not pick a crazy random package.
<jcsackett> lifeless: sample data is the devil. kill the doctest--you have solid unittests.
<lifeless> jcsackett: we do? kill the whole thing?
 * jcsackett considers.
<jcsackett> i *thought* we did.
<jcsackett> perhaps do a check, and then kill it.
<lifeless> where would I find the unit tests?
<jcsackett> crap, i don't actually see a test that exactly handles that.
<jcsackett> sorry for the false hope.
<jcsackett> fair bit of that does look to me like it would be better covered by unit tests, but i imagine you don't really want to write a ton more tests if you can just update the doc.
<jcsackett> sinzui: are we doing standup this evening?
<sinzui> jcsackett: yes.
<lifeless> wgrant: while you're around, whats the easiest way in a doctest of doing an upload to the currentseries of ubuntu - see the error i've just forwarded you
<wgrant> lifeless: Use the factory.
<wgrant> Or maybe SoyuzTestPublisher.
<wgrant> Depending on what exactly you want.
<lifeless> wgrant: well, thats why I'm asking :)
<lifeless> I had tw oissues in my branch to nuke Distro.getCurrentSourceReleases()
<wgrant> Aha.
<wgrant> I see now.
<lifeless> one was a type items() rather than values()
<lifeless> thats fixed
<lifeless> the closing bugs upload one however is epic fail sample data
<lifeless> this isn't a critical bug, I want to sweep it under the carpet for later polish
<lifeless> so I was figuring that if I do a new upload of cdrkit to hoary
<lifeless> it will work
<wgrant> lifeless: Try just using factory.makeSourcePackageRelease
<wgrant> It may need more than that.
<wgrant> But we'll see.
<lifeless> is there a global factory in doctests?
 * lifeless doesn't do doc tests :)
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: I think 'factory' may be a factory.
<wgrant> But not entirely sure.
<wgrant> I don't write new ones.
#launchpad-dev 2011-02-26
<lifeless> wgrant: whats a 'delayed' copy
<wgrant> lifeless: Hee hee.
<lifeless> closing-bugs does a copy into *hoary* (the dev series) of 1.0
<lifeless> which of course I've now created as part of fixing the start of the test.
<wgrant> lifeless: Delayed copies are used to copy from private to public archives. They create a fresh PackageUpload referencing the existing SPR.
<lifeless> Its triggered multiple results in .package_upload
<lifeless> lib/lp/soyuz/model/sourcepackagerelease.py
<wgrant> lifeless: What's the actual problem?
<wgrant> Oh.
<wgrant> I see.
<lifeless>  http://pastebin.com/c4QZn2RE
<lifeless> + my branch
<wgrant> I am guessing that the test creates a PackageUpload for an existing SPR with a changesfile?
<lifeless> it uploads into hoary, which only worked previously because hoary is badly damaged in the sample data
<wgrant> Heh.
<lifeless> on the up side
<lifeless> I have a librarian process leaking if I run just this one test
<wgrant> Ah, excellent!
<LPCIBot> Project devel build #479: STILL FAILING in 5 hr 23 min: https://hudson.wedontsleep.org/job/devel/479/
<lifeless> wgrant: bug 725373
<_mup_> Bug #725373: reproducable test librarian (and sometimes memcached) leak <Launchpad itself:Triaged> < https://launchpad.net/bugs/725373 >
<wgrant> lifeless: Thanks, I'll look at that on Monday if you don't.
<lifeless> that would be cool
<lifeless> I'm actually going to leave this branch for a bit
<lifeless> I have other yaks to shave; if you wanted to have a brief poke monday, and let me know whats probably easiest - its soyuz stuff i haven't done a lot with - in terms of moving forward
<lifeless> that would be /awesome/ [more awesome still would be a patch to resolve the test fail]
<LPCIBot> Project db-devel build #397: STILL FAILING in 5 hr 50 min: https://hudson.wedontsleep.org/job/db-devel/397/
<lifeless> Hard / Soft  Page ID
<lifeless>      130 /  214  BugTask:+index
<lifeless>       62 /    2  Cve:+index
<lifeless> ok, thats odd
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1882E511
<lifeless> 1.8 seconds getting *one* LFA
<lifeless> otoh we're down by another 100 oopses \o/
<Ursinha> \o/
<jelmer> lifeless: wow.
<lifeless> jelmer: ?
<jelmer> lifeless: euhm, I meant to say
<jelmer> lifeless: wow!
<lifeless> jelmer: about ?
<jelmer> lifeless: I wouldn't have expected that to happen this quickly.
<jelmer> lifeless: the OOPSes
<lifeless> jelmer: 100 frequency, not distinct pageids
<jelmer> oh
<jelmer> sorry, never mind me
<lifeless> jelmer: we are making progress :)
<lifeless> the frequency drop is the removal of the bugnotification filters I think overnight
<lifeless> was only half the day
<lifeless> so expect to see more tomorrow
<LPCIBot> Project devel build #480: STILL FAILING in 5 hr 19 min: https://hudson.wedontsleep.org/job/devel/480/
<LPCIBot> Project db-devel build #398: STILL FAILING in 5 hr 22 min: https://hudson.wedontsleep.org/job/db-devel/398/
<wgrant> lifeless: Also the checkwatches stuff.
<wgrant> About 2/3 of them should be gone.
<lifeless> \o/
<wgrant> About 1/2 of the remaining will be fixed on Monday.
<wgrant> Just need to attack doctests violently.
<wgrant> buildbot is being oddly reliable today.
<wgrant> No full-scale Windmill failures.
<wgrant> Just that one buggy test.
<StevenK> Jenkins keeps failing on checkwatches.txt
<wgrant> It does.
<wgrant> No idea why.
<wgrant> ec2 and buildbot are fine.
<LPCIBot> Project db-devel build #399: STILL FAILING in 5 hr 30 min: https://hudson.wedontsleep.org/job/db-devel/399/
<LPCIBot> Project db-devel build #400: STILL FAILING in 5 hr 30 min: https://hudson.wedontsleep.org/job/db-devel/400/
<jelmer> the recent daily build enhancements are awesome, thanks :)
<lifeless> its looking really great isn't it
<lifeless> the project is finished now
<jelmer> yeah, the recent polishing of it was really good
#launchpad-dev 2011-02-27
<jelmer> hmm, I'm seeing weird behaviour talking to the qastaging server
<jelmer> retrieving a wadl resource I sometimes (I don't see a pattern yet) only get 3486 bytes back
<wgrant> How big are the headers?
<wgrant> And what is the content?
<jelmer> the content is JSON (but cut off halfway through, so the JSON parser barfs)
<wgrant> We occasionally see that when production is overloaded.
<wgrant> The response, including headers, gets truncated to a power of two.
<wgrant> I do not remember which power of two.
<jelmer> it seems to consistently be 3486 here, and it appears to happen randomly
<jelmer> I wouldn't be surprised if the size of the headers was (4096-3486)
<wgrant> That sounds about right.
<wgrant> Which resource?
<wgrant> Authenticated or not?
<jelmer> yes, should be authenticated
<jelmer> would anonymous access make a difference in any way?
<wgrant> Probably not on qastaging.
<jelmer> do you know if there is a bug of some sort?
<jelmer> ehm, bug /report/
<wgrant> jelmer: Bug #636713
<_mup_> Bug #636713: during sustained overload situation replies are truncated in production <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/636713 >
<jelmer> wgrant: thanks!
<jelmer> unfortunately my script can only run against qastaging at the moment, so I'll try again later
<wgrant> Why can it only run against qas?
<jelmer> it uses some new stuff that Ian introduced recently
<jelmer> http://samba.org/~jelmer/recipe-status/
<wgrant> It's not even on staging?
<jelmer> has some of the output
<wgrant> ah, the new recipe build collections?
<jelmer> yep
<jelmer> it wasn't on staging a couple of days ago, let me check again..
<wgrant> Ah, staging's update failed overnight.
<wgrant> Will look tomorrow.
<wgrant> Too many incidents already this weekend.
<jelmer> thanks, it does actually appear to be on staging now
<wgrant> Odd.
<wgrant> I was expecting staging to be down.
<wgrant> Hmm.
<wgrant> jelmer: You may be interested to know that they are on prod now.
<wgrant> But they only show up in 1.0, not devel...
<wgrant> Ah, no, they're in devel too, but you have to kill your cache.
<jelmer> oh, cool
<jelmer> they don't show up in launchpad.net/+apidoc/devel
<wgrant> They do if you avoid your cache.
<wgrant> eg. Ctrl+Shift+R.
<jelmer> oh, wow. I didn't realize Launchpad did heavy caching on that too. I thought it was just the wadl.
<wgrant> Neither did I.
<wgrant> But I thought it was worth a try.
<lifeless> jelmer: isn't https://bugs.launchpad.net/bugs/725811 a dupe?
<_mup_> Bug #725811: expose ISourcePackageRecipeBuild.binary_builds on the web UI <api> <recipes> <Launchpad itself:Triaged> < https://launchpad.net/bugs/725811 >
<jelmer> lifeless: a dupe of which other bug?
<lifeless> ians work
<wgrant> No.
<jelmer> lifeless: No, Ian exposed the source package builds
<lifeless> can you not traverse to the binaries ?
<jelmer> this is about the binary builds created from those source builds
<lifeless> yes, I get that
<jelmer> lifeless: I can't find any way to traverse to the binaries, the source_package_recipe_build doesn't have anything that links it to the source package it created
<wgrant> We should probably guess at that.
<jelmer> wgrant: the source_package_recipe_build has a binary_builds attribute that's not exposed
<jelmer> I currently parse the build log for the source package name and the version, and then call archive.getPublishedSources(source_name=source_name,version=version)[0].getBuilds()
<wgrant> jelmer: Ah, so it already does guess.
<wgrant> We should expose that and the guessed SPPH.
<jelmer> wgrant: yep - the web page already shows that information
<wgrant> Yay, fpc built correctly on i386 and amd64 this time.
<jelmer> at least it works against production now :) e.g. http://samba.org/~jelmer/recipe-status/testing-cabal.html
<jelmer> wgrant: fpc?
<wgrant> jelmer: The last bit of fallout from https://wiki.ubuntu.com/IncidentReports/2011-02-25-Permissions-build-failures
<lifeless> woo - statement count gropuing useful again
<wgrant> Oh?
<lifeless> you've killed enough of the checkwatches stuff
<lifeless> 1992  OOPS-1883CCW1185  https://bugzilla.novell.com/show_bug.cgi?id=604251
<wgrant> Almost.
<lifeless> is still in the list
<wgrant> Still got lots of ProtocolErrors.
<wgrant> Hmm.
<wgrant> That's odd.
<wgrant> Looking.
<wgrant> It should be mostly bugtracker-wide errors now.
<wgrant> Woah.
<wgrant> Looks like LP now uses the Light variant of the font...
<lifeless> hmm
<lifeless> I -really- have to fix this design issue preventing test optimisation with baseline fixtures.
<lifeless> making a new keyspace and schema with cassandra is a little slow to do per-test.
<LPCIBot> Project db-devel build #401: STILL FAILING in 5 hr 48 min: https://hudson.wedontsleep.org/job/db-devel/401/
<maxb> The front-page of staging OOPSes - yay :-(
<lifeless> maxb: what is hte oops ids
<maxb> OOPS-1884S112844 for example, but it's entirely repeatable
<maxb> geser mentioned an "S" OOPS id in #launchpad recently too
<lifeless> maxb: the reason I asked for anexisting one is that the oops only sync every $hours on staging
<lifeless> looking at gesers
<lifeless> https://api.staging.launchpad.net/1.0
<SpamapS> lifeless: o/
<lifeless> oh hai
<SpamapS> lifeless: so I wanted to setup a bug tracker for the packages...
<lifeless> SpamapS: upload them once
<SpamapS> and at one time I did have the project cassandra-packages registered
<SpamapS> but now its like that project disappeared
<SpamapS> I can't create a new cassandra-packages ... but I can add branches of it.
<lifeless> 2010-06-16 Curtis Hovey: Project disabled and Clint Byrum (Clint Byrum) notified
<lifeless> I can reenable it
<SpamapS> Doh maybe I missed that notification
<lifeless> but why not just upload to natty
<SpamapS> its not suitable
<lifeless> that will activate the package bug tracker
<SpamapS> thrift is embedded..
<SpamapS> avro..
<lifeless> ok
<SpamapS> :(
<SpamapS> we're a lot closer to distro packages... but there are still things embedded. :-P
 * SpamapS notes that the .debian.tar.gz is bigger than the .orig.tar.gz
<lifeless> reenabled
<lifeless> I suggest a more clear description and title
<lifeless> 'e.g. 'staging for cassandra packages' and 'cassandra embeds too much stuff to be blah blah blah
<lifeless> you'll want to configure the bug tracker
<lifeless> SpamapS: ^
<lifeless> SpamapS: it was a bit disapointing that cassandra can't do schema changes in *separate keyspaces* concurrently :>
<lifeless> SpamapS: I have some /awfully/ crufty code you'll want to see :>
<SpamapS> lifeless: everybody using cassandra has some awfully crufty code to deal with their still-evoling admin capabilities
<lifeless> OTOH I have most of a cassandra based oops system
<lifeless> but their inability to do range requests is plain annoying
<lifeless> I've mailed our contact at riptano again :)
<SpamapS> wait ranges for reads are supported
<SpamapS> as long as you use OPP
<SpamapS> which, given the OOPS focus, would make sense, and be only slightly confusing.
<lifeless> not on secondary indices because they are bound to the node of the primary row
<SpamapS> right. not much you can do there.
<lifeless> sure there is; they can fix the broken assumption :>
<lifeless> for now I've use a trick
<lifeless> you have a column with value ''
<lifeless> and you index it
<lifeless> then you also index the thing you want ranged
<lifeless> and you do a search for '' and $range
<lifeless> it reads every row and does the range stuff as a loop
<lifeless> this will suffice for a while
<SpamapS> Right.. is that worth it?
<lifeless> given I only need that for gc
<lifeless> for aggregates and reports I'm keeping running totales
<SpamapS> there's some rudimentary count suppot in 0.7 .. and 0.8 is supposed to have powerful counting
<lifeless> one of the weirdnesses
<lifeless> in twissandra they use a timestamp to build the timeline
<lifeless> but timestamps - even @ microseconds - are not unique
<lifeless> it might be /hard/ to collide, but get enough activity and frontends and it will happen
<SpamapS> yeah there's some work on making it a vector clock instead of a straight microsecond timestamp
<lifeless> see, I could ues that
<lifeless> the other thing I could use is a ttl
<lifeless> but open bug references need to impede gc
<lifeless> I'm thinking of having two cfs
<lifeless> identical
<lifeless> but one referenced by bugs and no ttl
<lifeless> that would leave aggregated cruft behind though, because AIUI I wouldn't get a callback on ttl expiry
<SpamapS> Couldn't you just remove the ttl when a reference is reated?
<lifeless> wasn't 100% sure that doing a new write with no ttl would do that
<lifeless> and was coding at us unfriendly times
<lifeless> SpamapS: so I reactivated https://launchpad.net/cassandra-packages
<lifeless> SpamapS: but you need to click on the bug tracker and configure it
<lifeless> it thinks you don't use LP at the moment
<SpamapS> lifeless: thanks. :) I looked in my nbox and I must have inadvertently deleted the notification
<lifeless> are you familiar with the bug settings bit I'm refering to ?
<SpamapS> very
<lifeless> kk
<SpamapS> just having dinner now.. somebody reminded me that there is no food to be had in capetown this late on sunday ;)
<lifeless> I'm just polling till I can file these bugs for you :)
<SpamapS> EAGAIN
<SpamapS> lifeless: ok, bug reporter configured
<lifeless> cool
<lifeless> want all the things I mailed you as bugs?
<lifeless> SpamapS: ^
<SpamapS> yes please!
<SpamapS> I think some of them will be solved by doing actual lucid/maverick backports rather than copying the bin packages
<lifeless> done
<lifeless> SpamapS: ^
<SpamapS> awesome... I sent my reply as well
<SpamapS> to your email I mean
<lifeless> cool
<SpamapS> These Afrikaaners have filled me up with delicious food.. hopefully I'll pass out soon. :)
<lifeless> nice
<SpamapS> its good to see that you're making some headway w/ the cassandra packages.
<SpamapS> I was beginning to wonder if anybody would ever actually use them. :)
<lifeless> heh
<lifeless> so the only critical one is the classdefnotfound
<lifeless> not being able to compact is, uhm, operationally significant
<SpamapS> indeed
<SpamapS> $ ./get-ppa-stats cassandra-ubuntu stable
<SpamapS> cassandra	0.7.0-0ubuntu1	42
<SpamapS> 42 downloads.. not too shabby ;)
<maxb> Where does get-ppa-stats live?
<SpamapS> in my home dir ;)
<SpamapS> actually
<SpamapS> lp:~clint-fewbar/+junk/lptools
<SpamapS> maxb: ^^
<SpamapS> lifeless: one point of pain right now is that they up and changed the build system entirely from 0.7.0 -> 0.7.1
<maxb> thanks
<lifeless> SpamapS: of course they did
<lifeless> SpamapS: get-ppa-stats should be in lptools ;)
<SpamapS> ok.. passing out eminent
<lifeless> night
<lifeless> hi flacoste
<flacoste> hi lifeless
<lifeless> flacoste: I spiked a oops-in-cassandra in the weekend; its on my local disk only at the moment
<lifeless> flacoste: ok with you if I make a project in lp, agpl3 the code and push publicly?
<lifeless> flacoste: also, how was your flight?
<flacoste> lifeless: fine by me
<flacoste> lifeless: my flight was awesome, even though it started by a cancelling flight
<lifeless> ><
<flacoste> cancelled
<lifeless> so the replacement they upgraded you or something ?
<flacoste> lifeless: kind of, instead of flying continental to London, i got an air canada flight
<lifeless> oh nice
<flacoste> and i managed to use one of my upgrade ticket
<lifeless> \o/
<flacoste> so i first business for the first time
<flacoste> slept better than at home, 5 hours straight without interruptions
<lifeless> no kids :)
<flacoste> exactly
<flacoste> :-)
<flacoste> and on the london cape town flight, it wasn't fully booked, so i got an empty seat besides me
<flacoste> which made sleeping also easier
<flacoste> so one of the best flight experience overall!
<flacoste> feel fresh and ready for this week sprint
<lifeless> excellent
<lifeless> we have fresh water again, came on on friday
<lifeless> :)
<flacoste> that's always useful
<lifeless> very
<lifeless> all software sucks ;)
<wallyworld> thumper: mumble?
<lifeless> wallyworld: two bugs you might like to know about - model defects exposed by your nice patch to show merge proposals in the revision lists for branches
<lifeless> https://bugs.launchpad.net/launchpad/+bug/726190
<_mup_> Bug #726190: bugs shown against merge proposals in revision list lists all bugs ever linked to branch and differ to those the MP itself show <code-review> <Launchpad itself:Triaged> < https://launchpad.net/bugs/726190 >
<lifeless> https://bugs.launchpad.net/launchpad/+bug/726195
<_mup_> Bug #726195: merge proposals show closed bugs <code-review> <Launchpad itself:Triaged> < https://launchpad.net/bugs/726195 >
<lifeless> rhythmbox just rickrolled me
<lifeless> mornign wgrant
<lifeless> hmm, no sinzui
<lifeless> anyone know if there is a bug about things like ubiquity showing up in +needs-packaging at all ?
<wgrant> lifeless: What's special about ubiquity?
<wgrant> It still has a project, doesn't it?
<lifeless> no
<lifeless> you may be htinking unity
<lifeless> dpkg, ubiquity, apport - those things the distro folk often explicitly do not want 'upstream' projects
<wgrant> Erm.
<wgrant> Have you seen devel lately?
<wgrant> It needs a new apt-ftparchive that is not everywhere yet.
<wgrant> I suspect we want to roll that back.
<lifeless> you just changed topic, right ?
<wgrant> I did.
<wgrant> So, there is a bug about how we don't really support Ubuntu-native packages well at all.
<lifeless> for context
<lifeless> I'm looking at https://bugs.launchpad.net/launchpad/+bug/722794
<_mup_> Bug #722794: DistroSeries:+needs-packaging timeouts <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722794 >
<lifeless> which does crazy stuff to get bugs and strings
<lifeless> and bug heat?!
<wallyworld> lifeless: thanks for the heads up on those bugs - that post implementation clean up stuff is on my radar for this maintenance phase
<lifeless> wallyworld: the timeout I mentioned last week preexisted
<lifeless> wallyworld: I have a branch playing to amliorate it
<wallyworld> lifeless: yeah, i had a quick look and had come to the same conclusion
<lifeless> wallyworld: https://code.launchpad.net/~lifeless/launchpad/bug-710685/+merge/51258
<wallyworld> was going to get in today and try and fix it but you already are on it
<huwshimi> Uh what? 0 tests run in 4:12:24.883708, 0 failures, 0 errors. It took four hours to do exactly what then?
<StevenK> Haha. That's the question.
<wgrant> huwshimi: Bug #726243
<_mup_> Bug #726243: ec2 no longer keeps track of failed tests <Launchpad itself:Triaged by wgrant> < https://launchpad.net/bugs/726243 >
<huwshimi> wgrant: Oh, that's not very helpful.
<wgrant> It does at least detect whether it has failed or not, though.
<huwshimi> wgrant: How do I know what failed then? The log is not obviously showing anything that went wrong, but there's a lot of the log that I haven't looked at :)
<wgrant> huwshimi: You'll need to grep it.
<wgrant> Or run subunit/testr over it
<huwshimi> wgrant: For what?
<wgrant> I presume there's some subunit tool, but I tend to search for "error:" and "failure:", or just pipe it into 'testr load'.
<StevenK> Grab the log from the e-mail and: zcat <file> | subunit-filter --no-passthrough | subunit-ls
<huwshimi> I'm guess it is this: http://paste.ubuntu.com/573231/
<StevenK> wgrant: subunit *tool*? Why have one confusing UI when you can have ten!
<wgrant> huwshimi: I fixed that on Friday.
<wgrant> huwshimi: If it's the only failure, lp-land it.
<wgrant> Except that we're in testfix.
<wgrant> Because buildbot doesn't have the new apt.
<wgrant> And I guess we are ISless today.
<huwshimi> wgrant: Ah right, it appears to be. Do I need to merge devel first?
<wgrant> No.
<lifeless> wgrant: the new apt is in our ppa
<lifeless> wgrant: IS will be here when spm shows up
<StevenK> Given he flew back from London on Saturday? You tell funny jokes.
<wgrant> lifeless: Right, but given that he may well still be in the air...
<lifeless> he should get in at 9am canberra time or thereabouts
<lifeless> and he will want to stay up to beat jetlag
<lifeless> [whether he will succeed... different question]
<wgrant> Heh
<wgrant> lifeless: What does subunit.TestProtocolServer do? You give it a result object and a stream, and it plays the stream's events through the result?
<wgrant> Ah, no, that stream is something else...
<lifeless> wgrant: pydoc subunit
<lifeless> '    Subunit has support for non-blocking usage too, for use with asyncore or
<lifeless>     Twisted. See the ``TestProtocolServer`` parser class for more details.
<lifeless> '
<lifeless> wgrant: what do you want to achieve?
<wgrant> lifeless: Just working out how ec2test.remote does its stuff.
<wgrant> So I can unbreak it.
<lifeless> ok
<lifeless> well  - class TestProtocolServer( - find that section in pydoc
<lifeless> I just reread it and it seems useful to me :)
<lifeless> BranchSet:CollectionResource:#branches never completes in < 4 seconds :(
<LPCIBot> Project devel build #481: STILL FAILING in 5 hr 56 min: https://hudson.wedontsleep.org/job/devel/481/
<wgrant> StevenK: Jenkins needs the new apt.
<wgrant> StevenK: And you need to tell me what the checkwatches.txt failure is, since it 404s in the web UI...
<StevenK> Yes, it doesn't like doctests.
<wgrant> :(
<StevenK> http://pastebin.ubuntu.com/573238/
<wgrant> Thanks.
<StevenK> Just checking the current build slave out
<StevenK> ubuntu@ip-172-56-124-250:~$ COLUMNS=150 dpkg -l apt | tail -n 1
<StevenK> ii  apt                             0.7.25.3ubuntu9.3+ppa1          Advanced front-end for dpkg
<StevenK> wgrant: Is that the new apt? ^
<wgrant> I presume so.
<wgrant> But looking.
<wgrant> Yes
<wgrant> :(
<StevenK> Hm?
<wgrant> Well, if it has the new apt and is still failing...
<StevenK> Oh, sorry.
<StevenK> That output is from the current build slave
<StevenK> As in, the test run that started ~ 30 minutes ago
<wgrant> Ah. I guess we'll see.
<wgrant> Thanks.
<lifeless> whats the ? to set webservice batch size ?
<wgrant> ?ws.size
<lifeless> thanks
<lifeless> hmm, I think I need more sample branches to fix properly.
<lifeless> shrug
<michaelh1> Morning.  Is there a URL to get the latest download release from a series?  I'm writing a Makefile and would like to point wget at one URL that always resolves to the latest tarball.
<michaelh1> (such as the latest from the 4.5 series at https://launchpad.net/gcc-linaro/+download)
<lifeless> I don't know
<lifeless> you could use the uwatch stuff that packages (like bzr) use to track upstream
<lifeless> wgrant: do you happen to know where the top level collections are glued into the web service?
<wgrant> lifeless: LaunchpadRootNavigation?
<michaelh1> lifeless: OK, ta.  I'll file a wishlist feature request
<StevenK> michaelh1: I'd suggest uwatch as the easiest way currently
<lifeless> indefinitely
<wgrant> lifeless: export_as_webservice_collection() makes it appear in launchpadlib, though.
<lifeless> realistically, such a link is going to be a long way away
<lifeless> we're sorting out many rather heavy lifting issues atm
<lifeless> wgrant: yes, I see.
<lifeless> wgrant: and then
<lifeless> *another* decorat
<lifeless>  @collection_default_content
<lifeless> to select what is shown.
<lifeless> This is just so byzantine.
<wgrant> Right.
<StevenK> Ran 56 tests with 0 failures and 48 errors in 17.653 seconds.
<StevenK> "Bugger"
<wgrant> At least it was quick.
<StevenK> Total: 77 tests, 2 failures, 3 errors in 2 minutes 13.821 seconds.
<StevenK> Progress!
<huwshimi> Is anyone else getting make errors (ImportError: No module named widgets) when they try and make on a new branch?
<lifeless> run utilities/update-sourcecode
<lifeless> wgrant: it would be nice for folk if when changing shipit you make a small song and dance on the dev list
<huwshimi> lifeless: Thanks all working now.
#launchpad-dev 2012-02-20
 * StevenK takes ILaunchBag out the back, shoots it in the head, and then stabs it for good measure.
 * mwhudson cheers StevenK on
<mwhudson> StevenK: for real?  or is that just wishful thinking?
<StevenK> mwhudson: Wishful thinking
<mwhudson> boo
<wgrant> Damn
<wgrant> branch is down more than 90%, but it's still there :/
<lifeless> http://dilbert.com/strips/comic/2007-11-26/
<lifeless> wgrant: 100000 loops, best of 3: 7.32 usec per loop
<lifeless> bah, typo
<lifeless> ah, thats better
<lifeless> :!bash ./profile
<lifeless> 10 loops, best of 3: 21.7 msec per loop
<wgrant> lifeless: Heh, that's slightly less implausible.
<wgrant> Is that compile?
<lifeless> pybars
<lifeless> 10 loops, best of 3: 21.9 msec per loop
<lifeless> pystache
<lifeless> 10 loops, best of 3: 30.8 msec per loop
<wgrant> lols
<lifeless> comfortably faster b/4 optimisation
<lifeless> could do 1500 items of that scale without worry within a second, I suspect
<lifeless> will want some detailed tuning
<wgrant> The previous timings were compile, I assume?
<lifeless> no
<lifeless> running the same template
<lifeless> but with no data
<lifeless> because lp.cache != what we feed the template
<lifeless> so all the top level lookups found nothing
<wgrant> 1500 * 30.8 > 1000
<lifeless> thats pystache
<wgrant> 1500 * 21.9 > 1000
<lifeless> you are forgetting the /75
<wgrant> Oh, so that's the template for the whole list?
<lifeless> yes
<wgrant> (also, lpnet average down to 395% from the 560% it was 2 weeks ago)
<StevenK> wgrant: Do you want to review https://code.launchpad.net/~stevenk/launchpad/no-active-for-dsd-spr/+merge/93762 ?
<wgrant> StevenK: r=me
<wgrant> So, that's OOPS #3 (not #2 as I thought earlier) fixed.
<wgrant> If you fix #1 (easy) and #2 (probably trivial) as well, we'll be at around 100 exceptions per day.
<wgrant> And #4 isn't real
<wgrant> StevenK: Actually
<StevenK> wgrant: Hmm?
<wgrant> Mm, I guess it's fine.
<wgrant> Since it picks the most recent.
<wgrant> And filters by version
<wgrant> So there shouldn't be an issue with too many or non-latest resolds.
<wgrant> results
<StevenK> Right
 * StevenK looks for our #1 OOPS
<wgrant> It's None: None
<wgrant> Which is usually log.warning or log.error
<wgrant> In most cases it's garbo whinging that a task timed out.
<wgrant> Can probably be demoted.
<StevenK> Ah, bug 888049
<_mup_> Bug #888049: garbo-frequently logged an oops <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/888049 >
<wgrant> Yeah
<wgrant> gnargh I hate security proxies breaking object identity.
<stub> Garbo tasks timing out are a concern - we need to ensure it keeps up, even if that means increasing time slices or moving hourly jobs to daily or on both lists
<wgrant> stub: Mm
<wgrant> We care that it keeps up in the end.
<wgrant> At present to care that it keeps up daily we have to put it in garbo-daily.
<wgrant> When in fact we may want to do it -hourly, but don't care if it lags for two hours.
<StevenK> Or more, thanks to the backups
<wgrant> Right
<stub> Yes, and the only way we have of doing that without adding features is by ensuring it always finishes in its allotted time slice (which might be changed)
<stub> Backups were an issue, and we fixed that.
<stub> They where a problem
<wgrant> They still are AFAIK
<stub> Backups are happening on a slave and garbo isn't looking at that slave afaik. If it is, that is a problem we need to fix.
<wgrant> I thought that didn't matter.
<StevenK> The OOPSes don't look like a problem of not running
<StevenK> They are hitting the slice limit and being killed
<StevenK> <oops-message-0>: [BugHeatUpdater] Task aborted after 3299 seconds.
<wgrant> Ah
<wgrant> keeps getting blocked by librariangc and autovacuum now :/
<wgrant> Instead of backups.
<wgrant> But yes, bugheatupdater is terrible.
<wgrant> I'm not sure why.
<stub> librariangc? That is a new one. How is that blocking? Long running transactions?
<wgrant> But the whole lot often get blocked by autovacuum for an hour
<wgrant> Long running transactions on both counts, yeah
<wgrant> 2012-02-20 02:53:23 INFO    [DuplicateSessionPruner] Blocked on 1:42:21.965885 old xact postgres@launchpad_prod_3/25895 - <HIDDEN>.
<wgrant> 2012-02-20 02:53:23 INFO    [DuplicateSessionPruner] Blocked on 0:44:56.982871 old xact postgres@launchpad_prod_3/11353 - autovacuum: VACUUM pg_catalog.pg_index.
<stub> oh... librariangc is a victim, not a cause
<wgrant> 2012-02-20 00:02:06 INFO    [BugHeatUpdater] Blocked on 1:07:30.239287 old xact librariangc@launchpad_prod_3/32758 - <IDLE>.
<wgrant> Possibly true, indeed.
<StevenK> Right, so it's using self.log.warn()
<stub> So we can silence these (don't report a task timeout if we got blocked for reasons beyond our control)
<stub> Or we can make dblooptuner ignore autovacuum, which scares me a little.
<wgrant> Perhaps we record the total time each task has been blocked.
<StevenK> stub: My plan was to fix looptuner to not warn, so it stops causing >100 OOPSes a day
<wgrant> And if it's more than $threshold, don't warn.
<stub> That is easy enough to do IIRC
<stub> We can go with that.
<stub> Monitoring long running transactions can be done better than watching batch jobs produce errors
<StevenK> Haha
<nigelb> haha, I just read flacoste's email babout serial tab openers team. lol.
<StevenK> nigelb: Did you see the cricket? :-P
<wgrant> stub: That BranchLookup.getByUniqueName fix freed 1.5 cores on wildcherry.
<wgrant> Which might almost mean that we're not screwed if we have to failover.
<nigelb> StevenK: Nope, been offline helping run an  conference.
<StevenK> nigelb: Aus 5/288, India 178
<nigelb> fuuuuu.
<nigelb> now?
<wgrant> Heh
<StevenK> That was last night
<nigelb> Ah. I was a zombie last night after being on the road all day.
<nigelb> Just woke up after a 12-hour glorious sleep.
<StevenK> Hmm, switching to self.log.info() means it doesn't get printed by the doctest now. :-(
 * StevenK can't even work out how the warnings are injected into the stream
<stub> wgrant: \o/
<nigelb> StevenK: http://www.devsigh.com/sigh/67
<StevenK> Hah
 * StevenK sighs at looptuner.txt
<wgrant> grar
<wgrant> how does AMD manage to make fglrx into such a piece of shit
<StevenK> Bwaha
<lifeless> one bug at a time
<wgrant> Perhaps they're actually benevolent
<wgrant> They just make it unstable in order to improve FS crash handling cases.
<StevenK> Or they just can't write drivers for their own hardware
<wgrant> Never attribute to stupidity that which can be adequately explained by them being proprietary arses.
<StevenK> And yet you say AMD are *better*? :-P
<wgrant> In some ways :)
<poolie> what's fashionable these days for running lp dev?
<poolie> lxc, bare (precise) metal?
<wgrant> LXC is in fashion, and there's even a script to set it up, but I'm not sure if it works.
<wgrant> Yellow is making it really easy to set up at present.
<poolie> hm
<poolie> does it run ok on precise?
<wgrant> That's the target.
<poolie> perhaps i'll just go with bare metal for now
<wgrant> Oh
<wgrant> LP on precise?
<wgrant> Mostly, yes.
<wgrant> A few tests fail, but it otherwise works.
<lifeless> poolie: the target for the paralleltests project is lxc (lucid) hosted on precise
<lifeless> poolie: the setuplxc script in tree should do a decent job of putting that together, starting from a precise 'bare metal'
<poolie> lifeless, i reinstalled my precise laptop a while ago so i have to choose between putting lp deps on the base os, or sometihng else
<poolie> perhasp i'll try that
<poolie> *setuplxc
<poolie> would i be right in guessing less-oops.sh is obsolete now?
<poolie> setuplxc is stuck at
<poolie> mbp@lptests's password:
<wgrant> Hm. It should automatically let your key in, AFAIK.
<poolie> or rather, i'm not sure what password it's expecting
<wgrant> But it's the same as your normal system password.
<poolie> if it is, ssh is rejecting it
<wgrant> :/
<poolie> ok
<poolie> Feb 20 07:05:19 localhost sshd[457]: User mbp not allowed because shell /usr/bin/zsh does not exist
<wgrant> Ah, heh.
<poolie> i can fix that, but later
<wgrant> Oh no
<wgrant> Surely not
<lifeless> ?
<wgrant> lifeless: Found out why the heat updater is slow
<wgrant> lifeless: Goal time is 2s, as normal.
<wgrant> The initial query is 2.8s.
<wgrant> So it degrades to roughly 1 update per 3 seconds.
<wgrant> And reads 154000 bug rows every 3 seconds.
<wgrant> I bet if I fix that it updates the whole lot in 5 minutes.
<lifeless> why is it reading 154K rows?
<wgrant> Probably because it doesn't update duplicates, but the index doesn't exclude duplicates.
<lifeless> wgrant: as in, why doesn't it have a LIMIT batch_size there ?
<poolie> lifeless, well, i pruned one small bit of cruft for you
<poolie> not yet the timeline stuff though, due to EYAK
<lifeless> thank you
<lifeless> wgrant: uhm, you know we're running a branch of storm already....
<lifeless> wgrant: so why you copy code?
<wgrant> Mmm, I guess.
<wgrant> I prefer to be non-invasive where possible.
<lifeless> dropping the non-null requirement on duplicates, and updating them appropriate, makes it fast
<lifeless> launchpad_qastaging=> EXPLAIN ANALYZE SELECT id FROM bug WHERE (heat_last_updated IS NULL OR heat_last_updated < '2012-02-13')  LIMIT 1;
<lifeless> Time: 1.906 ms
<wgrant> lifeless: What if you add back the duplicateof and leave the LIMIT 1?
<lifeless> 7.7 seconds, table scan
<wgrant> Yep
<lifeless> with dup is null and order by heat_last_updated nulls last -> 2.2 sec
<wgrant> I think we probably just fix the index to be duplicateof IS NULL.
 * wgrant tries.
<lifeless> erm
<lifeless> actually
<lifeless> did you get the actual results as well ?
<lifeless> there is a pattern, when you see a big slow scan even with a limit
<wgrant> Oh, there's meant to be an ORDER BY id there too
<wgrant> Missed that
<wgrant> Just to probably make it even slower.
<lifeless> nah, not that
<wgrant> I didn't get the results.
<wgrant> I know
<wgrant> But thought I should note it.
<lifeless> so, ordering by the index will force it to use it
<lifeless> (often)
<lifeless> the reason its slow is that it is finished
<lifeless> tada.
<lifeless> SELECT id FROM bug WHERE duplicateof IS NULL AND (heat_last_updated IS NULL OR heat_last_updated < '2012-02-13')  order by heat_last_updated desc LIMIT 1;
<lifeless>  id
<lifeless> ----
<lifeless> (0 rows)
<lifeless> wgrant: it is slow, and consulting all rows, because it can't fulfill the request; even when limited
<wgrant> Hah, true.
<lifeless> wgrant: it has an estimate of 155K rows which is bong; it *is* examining all million rows
<lifeless> so, I think there are two bugs: if it is not limiting its results to the current batch size, that is bong.
<lifeless> And if it continues to try to do shit when nothing is returned, that is bong.
<lifeless> wgrant: this is the same slowness that batches that run past the end of big data sets like BPR exhibit
<lifeless> wgrant: where it suddenly goes off into lala land and reads through to the end of the table
<wgrant> I think it is limiting, actually, I misread.
<lifeless> ok cool
<wgrant> Yeah
<lifeless> in which case, this is a nuisance but also a symptom - it should just stfu and stop :)
<wgrant> But all this would not be an issue if the index was right.
<lifeless> the index is fine
<lifeless> change the query to have order by heat_last_updated
<stub> 3 bongs, a shit and lala land.
<lifeless> that will update the oldest rows first (which is strictly correct anyhow)
<wgrant> lifeless: The index is less than ideal.
<wgrant> Because it has to do a bitmap heap scan to determine duplicateness.
<stub> Avoiding order by heat_last_updated as that isn't stable ordering
<wgrant> And it will have to read all those rows first.
<lifeless> wgrant: I don't see a bitmap heat scan in the explain
<wgrant> Because they'll be older.
<wgrant>  Bitmap Heap Scan on bug  (cost=12576.08..141492.57 rows=154605 width=4) (actual time=2861.017..2861.017 rows=0 loops=1)
<wgrant>    Recheck Cond: ((heat_last_updated IS NULL) OR (heat_last_updated < '2012-02-13 00:00:00'::timestamp without time zone))
<wgrant>    Filter: (duplicateof IS NULL)
<wgrant> That looks like a bitmap heap scan filtering on duplicateof to me :)
<lifeless> that was without the limit right ?
<lifeless> explain analyze SELECT id FROM bug WHERE duplicateof IS NULL AND (heat_last_updated IS NULL OR heat_last_updated < '2012-02-13') LIMIT 100;
<lifeless>                                                                       QUERY PLAN
<lifeless> -------------------------------------------------------------------------------------------------------------------------------------------------------
<lifeless>  Limit  (cost=0.00..100.82 rows=100 width=4) (actual time=1261.132..1261.132 rows=0 loops=1)
<lifeless>    ->  Seq Scan on bug  (cost=0.00..155602.61 rows=154338 width=4) (actual time=1261.131..1261.131 rows=0 loops=1)
<lifeless>          Filter: ((duplicateof IS NULL) AND ((heat_last_updated IS NULL) OR (heat_last_updated < '2012-02-13 00:00:00'::timestamp without time zone)))
<wgrant> With the limit it seq scans here
<lifeless>  Total runtime: 1261.219 ms
<wgrant> Which is hardly better.
<lifeless> shrug, I'm fine with you improving it.
<lifeless> Just saying, doesn't seem terrible given there are no rows
<wgrant> So, it either has to do a seq scan or it has to do an index scan.
<wgrant> If it does a seq scan it's probably going to take forever.
<stub> If we make the index partial, we have to ensure that all of the queries also have the duplicateof IS NULL clause
<wgrant> If it does an index scan, it has every dupe bug ever earlier on in the index.
<wgrant> stub: All one query? :)
<lifeless> stub: tis already partial
<wgrant> It's not partial.
<lifeless> oh, not its not
<wgrant> But the one query that uses it will work if it's partial.
<lifeless> Right, I should sleep.
<wgrant> That's why I want it partial :)
<stub>     "bug__heat_last_updated__idx" btree (heat_last_updated)
<wgrant> Anyway, I'm right, will submit DB patch with a few more indices later.
 * wgrant dines.
<lifeless> its very odd that its not using the index today
<lifeless> as the index is a superset of the data needed; it must think there are an overwhelming number of stale-hat duplicates
<lifeless> wgrant: I suggest changing the code to update the heat on duplicates.
<lifeless> wgrant: make it simpler, leave the index as-is
<lifeless> wgrant: (yes, the update will be trivial..)
<stub> As there are 0 rows matching the clause, there should be 0 rows in the stats matching that clause. So the planner should think that, if they exist, they are unlikely and hit the indexes.
<lifeless> wgrant: so yea,
<lifeless> select id, duplicateof, heat, heat_last_updated from bug where heat_last_updated in (select min(heat_last_updated) from bug);
<lifeless>    id   | duplicateof | heat |     heat_last_updated
<lifeless> --------+-------------+------+----------------------------
<lifeless>  199626 |      199600 |    0 | 2010-05-28 17:50:04.055114
<lifeless> (1 row)
<lifeless> we've been carefully shoving all the duplicate rows to the front of the desired index
<lifeless> stub : but the planner doesn't know this
<lifeless> stub: what it knows is that the index *starts* with a lot of data - 196K rows -
<stub> lifeless: I'm talking from the pov of the planner. It has to assume that the rows might be there, and that they are uncommon because they are not in the stats, so index lookups would be a win unless it needs to scan for some other reason.
<lifeless> stub: me too
<lifeless> stub: the thing is, they are common :)
<lifeless> stub: the first 196K rows in the index are for duplicateof is null bugs
<poolie> setuplxc has new doctests?
<lifeless> wgrant: updating duplicates would be more in line with using this as an occasional re-fresh-the-data script too - we might not always have duplicate imply 0, so this would remove a booby trap.
<lifeless> stub: select count(*) from bug where heat_last_updated < '2012-02-13';
<lifeless>  count
<lifeless> --------
<lifeless>  196094
<czajkowski> Aloha
<stub> Oh right.
<lifeless> so I think the planner is more or less saying '1/5 of the bugs in the table are relevant, which means I probably will find what they want on the first page -> scan'
<lifeless> of course, then it gets horribly disappointed :)
<wgrant> lifeless: That's another option.
<lifeless> wgrant: I favour it, for the reasons I mention above
<wgrant> Yep
<wgrant> Glad you saw the light about the index :)
<stub> Its saying there are 196094 bugs where heat_last_updated < '2012-02-13', and zero bugs where heat_last_updated is null. But it is too stupid to know that there are 0 bugs where head_last_updated is null or < '2012-02-13'
<lifeless> stub: s/heat_last_updated/duplicateof/ :P and yes.
<wgrant> stub: The assumption in that second sentence is false.
<wgrant> Right
<wgrant> What lifeless said.
<lifeless> wgrant: well, I see that the index isn't performing, I don't think we should change it, all things considered.
<lifeless> anyhoo; i leave you and stub to debate it :> I'm offtosleep
<wgrant> Night.
<wgrant> Thanks!
<wgrant> Ah, crap.
<wgrant> That will probably need a DB patch.
<lifeless> wgrant: what will?
<wgrant> Or I workaround it by wrapping the calculate_bug_heat calls with a guard for dupeness.
<wgrant> lifeless: calculate_bug_heat doesn't return 0 for dupes.
<stub> https://pastebin.canonical.com/60572/
<wgrant> That's implemented in updateHeat
<lifeless> wgrant: you wouldn't just call updateHeat(bug) ?
<wgrant> lifeless: It currently uses RS.set()
<wgrant> I guess this will still be a thousand times faster.
<wgrant> So could just use updateHeat
<lifeless> yeah
<stub> think I'm on the wrong query
<lifeless> I mean, we're doing two things
<lifeless> a) moving to update-once, ever, so not caring about 'older than one week' rather 'older than constant value'
<lifeless> and b) making the operation decently efficient
<lifeless> wgrant: or we could store non-zero heat for duplicates
<lifeless> wgrant: I don't think it would break anything
<mrevell> Hello
<wgrant> lifeless: That's true.
<lifeless> It has always seemed odd to me that a bug suddenly gets 0 heat when it becomes a dupe; dupes are filtered from results, so the heat should be orthogonal; and if someone searches for bugs including dupes, sorted by heat...
<wgrant> Would remove a fair chunk of the remaining heat code
<wgrant> (ie. 5 lines)
<lifeless> why would we shove all the dupes to the bottom of the list
<lifeless> hi mrevell
<wgrant> So, I'll remove the dupe => 0, and remove the non-dupe filter?
<lifeless> mrevell: I've been meaning to catchup with you, but ETERRIBLETIMING
<lifeless> wgrant: that works for me; see if you can convince yourrandomreviewer of the sanity of it ;)
<lifeless> wgrant: you should also land either a knob, or a hard coded value, to replace the '1 week old' thing, unless you've done that already.
<adeuring> good morning
<wgrant> lifeless: I haven't. Might do it in the same branch.
<wgrant> A flag to set the earliest valid timestamp, if null then nothing happens.
<lifeless> yah
<lifeless> -> be
<lifeless> d
<wgrant> noight
<thumper> night wgrant
<wgrant> Not me, lifeless :)
<nigelb> meh, wgrant as no days and nights. he's awake throughout.
<wgrant> Heh
<mrevell> lifeless, I'm available 22.30 UTC my Monday (11.30 your Tuesday, I believe). Does that work for you?
<StevenK> bigjools: But only because Queenslanders wouldn't know what daylight savings time was if it bit them on the bum. :-)
<bigjools> StevenK: quite!
<bigjools> StevenK: I am quite happy with that though, means I stay closer to UK time
<czajkowski> daft question time, I do aplogise in advance
<czajkowski> if a  person requests ownership of a project and the registery are already the owners and seems to have some clue as to what they are talking about and an interest in helping the project
<czajkowski> is it ok to hand it over once fully checking they will maintain it?
<flacoste> czajkowski: yes
<czajkowski> flacoste: aloha there :)
<czajkowski> thanks
<flacoste> hi!
<czajkowski> someone wants to take over ownership of Mandriva
<flacoste> interesting
<flacoste> czajkowski: note that Mandriva is a distribution, and what you can do with this in LP is kind of limited
<flacoste> distribution not using Soyuz
<flacoste> mainly you can only track bugs
<czajkowski> nods
<flacoste> which doesn't make sense here since Mandriva doesn't use LP for bug tracking :-)
<czajkowski> flacoste: https://answers.launchpad.net/launchpad/+question/188064
<czajkowski> flacoste: which is why I'm kinda confused and trying to work  it out, will get there eventually :)
<flacoste> czajkowski: seems like a reasonable request, what is interesting is that he thinks that the Mandriva community would use LP for project planning, like the derivative he's involve dwith
<czajkowski> nods
<czajkowski> I did meet one of the folks leading the derivative at FOSDEM and he was asking how useful LP was. So maybe there is a plan B
<czajkowski> I'm really not sure how to asnwer this question or if even is a lp Question : https://answers.launchpad.net/launchpad/+question/188094
<rick_h> czajkowski: yea, that's a major 'not applicable' to LP. Might suggest they try ask ubuntu if they've having issues with getting some PHP magic to work with ubuntu versios of php/apache.
<czajkowski> rick_h: thanks
<czajkowski> want to start on a clena page tomorrow so trying to clear up the last few
<czajkowski> bene looking at that trying to work it out
<rick_h> yea, basically if register globals is on, then things sent in a POST request to the server should be auto variables in his script and it's not working. but that stuff has been so bad practice for so long, it might just not even work in 5.3 I don't know
 * rick_h has php dev flashbacks ...shudder 
<czajkowski> I was wondering if people knew the reasons behind the lack of opera support on LP - https://answers.launchpad.net/launchpad/+question/188200
<lifeless> manpower
<lifeless>  / womenpower
<czajkowski> lifeless: while I'd agreew ith you, writing that on a questions page isn't really going to be helpful now is it :)
<czajkowski> and the question shall remain open and I dont like open things in my queue :)
 * czajkowski has beaten RT into submission today 
<rick_h> czajkowski: we've got a bug for the opera issue and I hope to take a peek at it, as we'll need it to fix IE8 support as well, so it's not that we're not going to fix it. Just not yet
<czajkowski> ok that seems more of a reasonable answer to give someone
<rick_h> czajkowski: yea, looking for the bug now, thought I was watching it
<lifeless> czajkowski: so, opera isn't in our list of target browsers.
<lifeless> czajkowski: its not in the list because (usually) what works for chromium (which is in the list) works for it, and we don't have the resources to test all browsers; opera isn't in the top-3 by volume browsers out there.
<lifeless> czajkowski: I think its a fine answer to give, its true :). Also, please don't commit to us fixing opera, because chances are we won't, unless or until someone steps up and maintains opera support as a volunteer.
<czajkowski> nods
<lifeless> let me forward you the stakeholders thread on this
<czajkowski> lifeless: thanks
<lifeless> czajkowski: there was also a related discussion, on -dev I think, about how YUI no longer categorise browers the same way
<czajkowski> nods
<czajkowski> thanks for the mail
<czajkowski> right time for some foods and relaxing befor eI switch brain over to locoteams work
<czajkowski> nn
<lifeless> nn
<lifeless> flacoste: can you hear me ?
<lifeless> flacoste: http://bazaar.launchpad.net/~canonical-launchpad-branches/pybars/trunk/view/head:/pybars/_compiler.py
<flacoste> lifeless: ISP woes again?
<lifeless> ADSL apparently stands for approximately down standard link
<lifeless> flacoste: http://www.yuiblog.com/blog/2011/07/12/gbs-update/#grades-deprecated is what I was babbling about
<flacoste> lifeless: yes, i saw, but their main page still explains the three tier grade approach, they just don't specify which browser gets which experience
<flacoste> because that should be left to each team to dictate
<flacoste> they just specify which browsers they test
<flacoste> i think for us, it still makes sense
<flacoste> since we are one such team that has to decide of the experience users get based on their browser and our resources to support it
<lifeless> kk
<mrevell> Hey lifeless. Did you want to talk today?
<lifeless> mrevell: sometime, but not today ;)
<lifeless> mrevell: i have to pop out for a bit, and it's already way late for you
<mrevell> lifeless, Okay, great :) I shall go to bed.
<mrevell> Night all
 * wgrant needs opinions.
<wgrant> Bug #933911 is somewhat qa-bad
<_mup_> Bug #933911: Remove PersonLocation-related code <qa-needstesting> <tech-debt> <users> <Launchpad itself:Fix Committed by salgado> < https://launchpad.net/bugs/933911 >
<wgrant> Because https://qastaging.launchpad.net/~/+editlocation doesn't preset its sole field to the existing value.
<wgrant> I suspect we don't care enough to block things.
<thumper> hey
<thumper> are we ever going to get ARM recipe builds?
<wgrant> They're no different from normal ARM PPA builds.
<StevenK> thumper: If the PPA the recipe is built into supports ARM, you'll get it.
<wgrant> Ah, crap.
 * wgrant curses cocoplum to hell.
<lifeless> wgrant: 'meh' (+editlocation)
<wgrant> lifeless: Thanks,.
<thumper> StevenK: how do you get a ppa to support arm?
<bigjools> admin option
<wgrant> Ah, *ning bigjools.
<StevenK> bigjools: Packing furiously?
<bigjools> yes. I now have my laptop in bed to catch up on email before I sleep
<bigjools> getting some snorts from the missus
<bigjools> wgrant: nice glob
<StevenK> The only thing that glob doesn't support is afternoon
<wgrant> It's not afternoon anywhere important :)
#launchpad-dev 2012-02-21
<wgrant> lifeless: https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883 <- I think that could/should be done as an API script, what's your opinion?
<wgrant> poolie: Do you want to redo https://code.launchpad.net/~mbp/launchpad/remove-bsondump2/+merge/93778 with a proper whoami?
<wgrant> You seem to have been setuplxc'd.
 * StevenK prods wgrant to fix the topic
<wgrant> I'm bac, what are you talking about.
<StevenK> Haha
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wgrant | Firefighting: - | Critical bugtasks: 4*10^2
<poolie> wgrant, ok
<poolie> that was icky
<poolie> i'm glad i caught it
<poolie> these scripts that do lots of random modifications to your host machine :(
<StevenK> You mean a lot like rf-setup?
<poolie> yeah, but lxc ought to be a chance to get away from that
<poolie> perhaps i should run setuplxc inside a vm :/
<thumper> wallyworld_: oi, can I please have a link on the team code home page for the +recipes please
<thumper> wallyworld_: I'll buy you a drink
<thumper> wallyworld_: maybe a backrub?
<wallyworld_> thumper: oooh, now you're talking
 * StevenK reviews his breakfast
<StevenK> TypeError: ('Incompatible metatypes', (<class 'lp.bugs.interfaces.bug.IBugLimitedView'>, <InterfaceClass lp.bugs.interfaces.bug.IBugView>))
 * StevenK stabs Zope
<thumper> StevenK or wgrant: do either of you know what is causing W: GPG error: http://archive.canonical.com precise Release: The following signatures were invalid: BADSIG 40976EAF437D05B5 Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>
<lifeless> thumper: intercepting proxy cache
<wgrant> Or scandium being terrible again.
<wgrant> But I think that's mostly fixed, so probably your ISP is shite.
<lifeless> in NZ proxy caches you.
<mwhudson> thumper: it can also be the file in /var/lib/apt/lists being corrupt
<wgrant> lifeless: I'd like to remove feature rules for these removed flags: https://pastebin.canonical.com/60613/
<wgrant> lifeless: And add this rule for my heat changes (not yet deployed): https://pastebin.canonical.com/60614/
<lifeless> if the code is gone, thats a no-brainer
<wgrant> The code is indeed gone.
<wgrant> And it is a no-brainer, but I still need a +1 :)
<lifeless> +1 on them both
<wgrant> Thanks.
<wgrant> Totally not cool, fglrx.
<wgrant> Ooh
<wgrant> New apport in precise, though.
<wgrant> does this mean LP won't have 9 bazillion bugs filed daily...
<wgrant> Ah, no, still uploads to LP
<wgrant> lifeless: Bug comment searching has never been enabled on prod AFAICT. Can I remove the code, since the current implementation is never going to perform even vaguely acceptably?
<lifeless> yes
<wgrant> Thanks.
<lifeless> or should I say 'no, I want the quota' :P
<wgrant> I'm removing targetnamesearch too
<wgrant> Heh
<lifeless> flacoste: I'm not sure we define 'core functionality' anywhere :)
 * wgrant massacres the branchscammer
<StevenK> Sounds lke an apt name.
<StevenK> Put up a branch renaming it and it'll mark it r=me
<wgrant> Maybe it's choking on MySQL again.
<StevenK> Plausible.
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/obliterate-targetnamesearch/+merge/93914
 * wgrant is preparing for the LOCTS
<StevenK> wgrant: Sorry, I was lunching. Looking now.
<wgrant> k
<StevenK> wgrant: r=me
 * StevenK approves of all this code death
<wgrant> Thanks.
<lifeless> see #launchpad for some related ;P
<wgrant> Makes my disclosure work easier, and puts me in a good position for the LOCTS that will surely eventuate.
<StevenK> Haha
<lifeless> TS?
<StevenK> I've not wanted to find out how much code I've ripped out versus added.
<StevenK> Specifying multiple revspecs would help
<wgrant> lifeless: Lines of Code Trading Scheme
<wgrant> lifeless: Like an ETS, but for maintenance burden.
<StevenK> Sigh, defeated by PQM
<StevenK> Since it's the committer for each rev.
<StevenK> Because auditing is for chumps
<lifeless> wgrant: so, you're committed to futures
<lifeless> StevenK: pqm isn't the author though, which is what you care about; just uses the merged in revs. or annotate.
<lifeless> wgrant: you've checked prodconf for the heat config key ?
<wgrant> lifeless: Yup, nothing there
<StevenK> lifeless: How does annotate help? What I'd like is a large diff of any rev I have or someone else has landed to devel
<lifeless> StevenK: so, perhaps it doesn't :) but it can tell you loc last touched by you, for the current tree, which is a related q
<wgrant> Ohloh gives lines changed, but not +-
<lifeless> anyhow, use the merged in revs to determine author, and mainline diffs for the content
<StevenK> Perhaps I should learn bzrlib
<StevenK> I was hoping I could loop over bzr log
<StevenK> wgrant: So, given IBug's definition in configure.zcml, I don't need to split it into IBugView and IBugLimitedView, I can just move id to LimitedView ?
<wgrant> StevenK: Unfortunately, yes.
 * StevenK reverts the split he did
<wgrant> If you've got a split that works, by all means do it.
<StevenK> I do -- but only in terms of Python. I wasn't sure how to shoehorn it into the ZCML
<StevenK> wgrant: Given the branch I just reviewed, can MessageChunk.fti die a horrible death?
<wgrant> I'd prefer not.
<wgrant> It's write-once so pretty cheap.
<wgrant> And I hope we'll do comment searching in future, even if we don't move to Solr soonish.
<wgrant> tsvector concat should make it pretty easy to do comment searches.
<lifeless> didn't you just remove comment searching?
<wgrant> Yes
<wgrant> But if we want to do a non-stupid implementation in future, it might well make use of MessageChunk.fti
<wgrant> Since that table is never updated it's cheap to keep.
<poolie> http://37signals.com/svn/posts/3112-how-basecamp-next-got-to-be-so-damn-fast-without-using-much-client-side-ui -- interesting
<wgrant> Yeah, saw that a couple of days ago.
<wgrant> Pretty interesting indeed.
<wgrant> But it relies on having a cascading last_updated column on pretty much everything.
<wgrant> Which has to increase write contention massively.
<poolie> i wonder if it's in plain old sql
<lifeless> they are caching into memcache I believe
<wgrant> They cache into memcached, yes.
<wgrant> But IIRC it's all ActiveRecord otherwise.
<lifeless> so contention == redundant writes more than lock-slows
<wgrant> lifeless: no, no
<poolie> also i guess unlike lp they have relatively few objects read by millions of people
<wgrant> lifeless: The whole caching scheme relies on having a last_updated field to generate the cache key from
<poolie> though, would that really be more mtime fields than lp?
<poolie> perhaps
<wgrant> There's no project mtime field.
<wgrant> max_heat is the closest, and I deleted that two weeks ago :)
<wgrant> Partly because of contention, partly because of terrible implementation.
<poolie> hm
<poolie> an mtime on /ubuntu would probably suck
<wgrant> Exactly.
<poolie> but i wonder if you would really need it
<lifeless> wgrant: I missed that; it looked like they just update the cache on any write, recursively, on all places affected.
<wgrant> lifeless: They update the mtime all the way up the tree.
<wgrant> lifeless: Cache contents never change for a given key.
<lifeless> wgrant: sure, but that's just the mtime of the write
<wgrant> The key includes in the update time.
<poolie> anyhow, so it's interesting that they have headed down the course lp was going to take ~2y ago, and that lp abandoned
<lifeless> wgrant: doesn't imply table storage of said value
<poolie> and, the opposite of lp's handlebars approach
<wgrant> poolie: Did you see the HN discussion?
<lifeless> wgrant: or are you saying 'to find the cached entry they need a consistent mtime on all things' ?
<poolie> no
<wgrant> poolie: It goes into stuff like this. Let me find it.
<wgrant> lifeless: Right.
<wgrant> lifeless: The invalidation mechanism is that the object's mtime changes.
<wgrant> So to change something 5 levels down, they need to write a new mtime on all 5 levels.
<wgrant> http://news.ycombinator.com/item?id=3603367
<wgrant> poolie: ^^
<poolie> wgrant, so (perhaps hn answers this), i don't think you would necessarily have to have an mtime all the way up
<wgrant> There is a bit of an argument over whether forcing everything to be server-side like this is smart.
<poolie> in the db
<wgrant> So, you could probably store the mtime in something more sensible.
<poolie> for instance if you change a bugtask in /ubuntu, you would need to invalidate any caches of bugs/ubuntu
<wgrant> Right.
<poolie> it seems like the important thing is to have a pubsub-ish tree of "I changed!"
<wgrant> And you only care about last-update-wins, so ACID is not useful.
<poolie> <wgrant> There is a bit of an argument over whether forcing everything to be server-side like this is smart.
<poolie> it's useful data that it can be made to work well
<wgrant> Indeed.
<poolie> i guess they need the FE to be close to the users
<wgrant> Yep
<poolie> but perhaps no more so than for a more ajaxy approach, if they are doing few requests
<wgrant> DHH goes into the latency aspects.
<lifeless> they are using ajax
<lifeless> according to the HN thing, 50% of their code is coffeescript, or thereabouts
<wgrant> Just over half-way down the HN comments.
<poolie> lifeless, well i said 'more ajaxy', don't you think this is a less ajaxy approach?
<poolie> if you accept a definition of ajax that the x means data comes down as xml or json
<lifeless> I think this is equivalently ajaxy; it does what lp does in large part (which is one of the things that is terrible about LP client side support)
<lifeless> we render on the server, ship html to the browser, the browser shoves it into the dom
<lifeless> their big claim is less client-side /stuff/ - no MVC on the client, no template rendering on the client, so less surface area of e.g. browser variation to deal with
<poolie> sure
<poolie> what i meant was http://news.ycombinator.com/item?id=3604732
<wgrant> I don't think LP's template caching thing should be abandoned.
<wgrant> It was just applied in completely the wrong way.
<wgrant> It was used to make slow things fast.
<wgrant> Rather than fast things very fast.
<wgrant> It was used as a means to stop timeouts.
<wgrant> I think it still has a place in LP.
<wgrant> But not for any of its current uses, except the blog caching thing.
<lifeless> its causing plenty of complaints today :) - e.g. jono's why-is-the-list-of-projects-stale qeru from a few days back
<wgrant> Oh yes, as I said, the current implementation is fucking terrible.
<wgrant> rargh
<wgrant> Must smash multitask bugs
<wgrant> Slow branch vocab is slow
<poolie> wgrant, i'm still shaving bzr yaks towards rebasing the bsondmp patch
<poolie> and ubuntu bugs ftm
<wgrant> poolie: There's a mostly working rewrite branch that I used for that purpose last week.
<wgrant> translationstobranch and the gardener need a few good amputations :(
<poolie> wgrant, https://code.launchpad.net/~mbp/launchpad/remove-bsondump2/+merge/93778 has the right userid now
<poolie> any other comments?
<wgrant> "good riddance"
<wgrant> poolie: Can you land it yourself?
<poolie> yes
<wgrant> Great.
 * wgrant touches Storm in inappropriate places.
 * StevenK attempts to resist the urge to /nick Storm
<nigelb> http://en.wikipedia.org/wiki/Storm_%28Marvel_Comics%29
<nigelb> 33
<nigelb> bah
<wgrant> Ah, heh
<wgrant> garbo runs tasks in threads, so of course the default script feature controller isn't always visible.
<stub> wgrant: I think garbo's architecture would suite switching to multiprocess now that the new stdlib library is available.
<wgrant> stub: Yeah, multiprocessing is pretty nice
<wgrant> Might look at that once we're on 2.7
<wgrant> Oh, it's in 2.6 too
<stub> Oh... was thinking it was 2.6
<stub> yer
<wgrant> stub: https://code.launchpad.net/~wgrant/launchpad/garbo-feature-flags/+merge/93928 is a quick fix for now
<wgrant> It's one line of significant change, if you want to review...
<stub> sure. not sure what it is fixing yet...
<stub> oic
<wgrant> Feature controllers are thread-local.
<wgrant> garbo calls login to set up an interaction in the fresh thread, but it also needs a feature controller now
<stub> So what is that constant for anyway? I'm not sure how we calculate bug heat any more.
<stub> (r=stub btw)
<wgrant> Thanks.
<wgrant> Bug heat no longer degrades over time, so it doesn't need to be updated weekly, just on changes to the bug.
<wgrant> But we change the algorithm occasionally.
<wgrant> So I adjusted the garbo job to read a timestamp from a feature flag -- anything older than that flag is considered invalid and gets recalculated.
<stub> So the timestamp is for the last time we changed the algorithm. I see.
<stub> I could argue that it gets hardcoded in the code somewhere rather than abusing feature flags, as it will change when the code changes.
<wgrant> We don't know when the code is deployed, though.
<stub> You don't need to know - pick a date a few days in the future and she'll be right.
<wgrant> Heh
<wgrant> Not quite
<wgrant> It will then recalculate every bug every hour until then.
<wgrant> Which sounds less than ideal for bloat.
<stub> But it won't, as it will take a while to recalculate. Anyway... this has already been discussed I'm sure.
<wgrant> An alternative is a version column, but meeeh
<wgrant> It won't take long.
<wgrant> Yeah, lifeless suggested this approach.
<stub> Looks like rather than feature flags, we need some generic shared runtime configuration. Feature flags could be built on that. Zookeeper almost meets the needs, but I'd rather not call Zookeeper from inside stored procedures.
<lifeless> aren't feature flags generic, shared and at runtime ?
<stub> Yes, and if you renamed them you would have an implementation :)
<stub> Like washing my dishes in a toilet. Technically, it is doable and done right no problems. But it is still called a toilet, and people still go euwww.
<stub> What are we going to do about ProductSeries.name ? It is actually a version number, and should be using the debversion type.
<stub> Quick fix and change the type with a dbpatch, or a correct fix and rename the column and change its type?
<stub> Bug #718660
<_mup_> Bug #718660: Drop version_sort_key, changing columns to debversion type <Launchpad itself:Triaged> < https://launchpad.net/bugs/718660 >
<stub> I guess we can rename the column and keep 'name' around for compatibility, API clients etc.
<wgrant> I'm not sure the rename is valuable.
<wgrant> Or even correct.
<stub> Mainly thinking about maintaining API compatibility. I haven't played with it so not sure what the policies and such are.
 * wgrant feels dirty, but hopes for a review of https://code.launchpad.net/~wgrant/launchpad/bulk-insert/+merge/93930
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10^2
<lifeless> stub: api compat - its fairly loosely typed
<lifeless> stub: that would cast into a string for instance (or at least I'd hope it does ;P)
<stub> It will be a string from Python - I don't think we have a debversion type?
<StevenK> We do
<StevenK> We use it extensively in Soyuz
<wgrant> But versions are normally handled as strings.
<stub> wgrant: Your bulk insert - shouldn't Storm do that optimization, using multi row insert if it has several objects of the same type to create in sequence?
<wgrant> stub: Yeah, but there are questions around how that works with ordering guarantees.
<wgrant> And I need this now, not in 6 months when the Storm devs have worked out what the guarantees are :)
<wgrant> Bug #411556
<_mup_> Bug #411556: Storm should support multi-row inserts <Storm:New> < https://launchpad.net/bugs/411556 >
<wgrant> Syntax also differs between DBMSes.
<stub> wgrant: I don't see what the ordering problem is.
<stub> wgrant: You have a list of things to flush. If you are about to do an INSERT, and the next item is also an INSERT of the same type, you do them together as one statement.
<wgrant> There are also issues around populating the primary keys.
<wgrant> MySQL provides no way to do it AFAICT
<wgrant> And I'm not sure if postgres guarantees order.
<stub> Not sure who on the team can review branches poking Storm privates anymore - allanap or bac?
<wgrant> (Storm relies on knowing the PK for each inserted object)
<wgrant> They're read-only storm privates, so they're mostly harmless :)
<stub> You would invoke the database multiinsert you are landing, or aarons executemany
<wgrant> stub: But the RETURNING order is undefined.
<wgrant> AFAICT
<wgrant> So we can't map them back to the objects.
<stub> Really? That seems somewhat silly :-(
<stub> Its a PostgreSQL extension, so I assume it is defined as the same order of the rows you are inserting. It isn't explicitly documented, but the extension would have been added for exactly this use case.
<stub> (Inserting a set of records, and you want to know what the DB set your NULLs to and how the triggers munged your data)
<wgrant> I guess.
<mrevell> Hello
 * jelmer is still strugging with his scanner branch
<wgrant> jelmer: What's up?
<adeuring> good morning
<jelmer> wgrant: I'm trying to reproduce the timeout issue we've been seeing on qastaging but have failed so far
<wgrant> jelmer: It's not just qastaging being terrible/
<wgrant> jelmer: abentley is working on a scanner DB performance branch at present.
<jelmer> I doubt it, we saw similar issues on production earlier (the incident)
<wgrant> True.
<jelmer> wgrant: oh, I didn't realize abentley was working on that bug now
<jelmer> it was assigned to Graham for a while and then unassigned I think
<wgrant> jelmer: He's batching the BranchRevision inserts AIUI
<jelmer> I should talk to Aaron and make sure our changes don't conflict too badly
<wgrant> stub: Also, about executemany, it seems it's implemented in psycopg2 as just a sequence of normal executes.
<wgrant> stub: Which seems like it would have negligible benefit.
<stub> If it isn't doing anything with prepared statements, yer
<stub> It might be using prepared statements under the hood btw.
<wgrant> It didn't seem to be.
 * wgrant rechecks.
<stub> If it is interpolating its own variables into the query string, it isn't.
<stub> If the query string and parameters are being sent to libpq separately, it might :)
<wgrant> https://bazaar.launchpad.net/+branch/ubuntu/psycopg2/view/head:/psycopg/cursor_type.c#L491
<wgrant> 524 looks pretty suspicious.
<wgrant> It parses the args into 'operation', then passes that straight into execute.
<stub> That is really ugly Python
<wgrant> Heh
<stub> It isn't doing interpolation there... if it is happening, it is down in _psyco_curs_execute
<stub> Might be write on multi value insert returning rows in an arbitrary order :-(
<stub> c/write/correct
<wgrant> stub: I didn't have any evidence either way, apart from the docs not specifying it and it seeming too good to be true.
<wgrant> stub: Have you already created those indices?
<stub> wgrant: No. Was waiting for the backups to complete before suggesting
<wgrant> Yeah
<wgrant> Finished early tonight.
<wgrant> Actually, it's been quick for the last week
<wgrant> Hmm
<stub> wgrant: So https://code.launchpad.net/~wgrant/launchpad/more-indices/+merge/93916 ?
<wgrant> stub: Yup
<stub> k. I'll apply that now.
<wgrant> Thanks.
<lcanas> hey guys, good morning from Madrid
<lcanas> one question about launchpadlib, how can I get the complete list of bugs for a given project? As far as I saw I have to get all the bugs and examine them one by one .. I'm pretty sure I'm missing something
<wgrant> lcanas: lp.projects['fooproject'].searchTasks()
<lcanas> wgrant, oh! great! thank you :D
<rick_h> morning
<ESphynx> hey guys, why do my daily builds show up as successful, yet I get these also what is up with 'ERROR: ld.so: object 'libfakeroot-sysv.so' from LD_PRELOAD cannot be preloaded: ignored.' errors?
<deryck> Morning, everyone.
<jelmer> hey Deryck
<ESphynx> 'morning deryck :)
<czajkowski> aloha
<ESphynx> hey guys, what's with the 'You have not informed bzr of your Launchpad ID, and you must do this to...' ?
<deryck> abentley, adeuring -- https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<ESphynx>    Remove the following packages:
<ESphynx> 1)     pbuilder-satisfydepends-dummy ?
<czajkowski> ESphynx: copying and pasting the same question into multiple channels is not necesary, when someone can answer it they will.
<ESphynx> czajkowski sorry I thought you were telling me I should ask here as well :P
<czajkowski> ESphynx: not at the same time, pick one place perhaps :)
<ESphynx> well it all depends whether the same people are in both places or not ;P
<ESphynx> i'm trying to move ia32-libs ahead :|
<rick_h>  https://launchpad.net/lpjsmin
<nigelb> man, someone should tel stub his part message probably needs some cleaning/fixing.
<rick_h> adeuring: http://pypi.python.org/pypi/modern-package-template as well, <3
<adeuring> rick_h: thanks!
<rick_h> deryck: https://dev.launchpad.net/JavaScriptIntegrationTesting?highlight=%28yuixhr%29 looks like what you were talking about?
<deryck> rick_h, indeed.  Was having a hard time finding it myself.
<deryck> rick_h, didn't think to try the wiki. :)
<rick_h> deryck: ok cool, will go from there. Thanks for the heads up.
<deryck> rick_h, np.  Thanks for handling it.
<salgado> sinzui, can you land https://code.launchpad.net/~salgado/launchpad/bug-937310/+merge/93987 for me?
<sinzui> Yes I can
<salgado> thanks!
<ESphynx>  <tumbleweed> launchpad doesn't use pbuilder -- meh ?
<czajkowski> sinzui: aloha there! thanks for the mail over the weekend
<sinzui> youy welcome
<czajkowski> sinzui: worked my way through the review projects and manged ok
<czajkowski> there are 11 there am which I am sunsure of
<czajkowski> and didnt want to just ack them
<sinzui> czajkowski, I ran this : ./disable_projects.py ielix campus vinfolio ptchiang+test
<czajkowski> ahh
<sinzui> czajkowski, I am deactivating the three ~registry projects
<czajkowski> nods those were the ones I wasn't sure about
<sinzui> This one needs a fix first https://launchpad.net/madrasah
<sinzui> madrasah is not the upstream for edubuntu-live. We need to remove the package link to deactivate it.
<czajkowski> yes they had picked too many liciences
<sinzui> The licenses are fine
<sinzui> Anything like a distro will have them all
<sinzui> czajkowski,  I followed the All packages link from the project page to find https://launchpad.net/madrasah/+packages
<sinzui> We want to remove the package, then this will be an empty, meaningless project that we will deactivate
<czajkowski> ok
<sinzui> and I have done that
<sinzui> The four remaining projects can be approved
<czajkowski> sinzui: thanks
<czajkowski> glad I checked with you
<rick_h> deryck: so just a heads up, added a card to see about updating the yui xhr tests to be combo loader friendly. Not sure the best way to do that off the top of my head yet.
<salgado> flacoste, around? when you moved me to ~launchpad-emeritus I ceased being a member of https://launchpad.net/~launchpad-dev and because of that lost my subscription to launchpad-dev@
<salgado> what's the correct way to get me subscribed again? do I need to join https://launchpad.net/~launchpad-dev?
<lifeless> moin
<lifeless> salgado: just join it, its an open list
<lifeless> -> no privileges are granted by membership in it :>
<salgado> oh, right, missed that
<salgado> lifeless, btw, I've sent a mail a few hours ago so it's probably stuck waiting for moderation. care to let it through?
<salgado> lifeless, you don't want to let my message through? ;)
<lifeless> salgado: haven't seen it - I'm doing my morning mail scan
<lifeless> salgado: plus giving czajkowski a hand
<czajkowski> salgado: I dont see you mail in the queue either
<salgado> lifeless, oh, ok, thought you didn't see my previous msg
<salgado> hm, maybe it's dropped if I'm not subscribed to the list
<salgado> but I should have good "standing", no?  do we still have the concept of good standing in LP?
<jcsackett> sinzui: got a second to mumble?
<lifeless> salgado: I think so :)
<sinzui> jcsackett, I can mumble. X just gave me some problems, but I think I am stable again
<jcsackett> sinzui: cool.
<lifeless> salgado: I can't see the moderate mail for you
<lifeless> salgado: I have no idea why not. I suspect a bug due to your odd state.
<sinzui> jcsackett, http://people.canonical.com/~ianb/picker-demo.ogv
<lifeless> sinzui may know.
<salgado> lifeless, my odd state?
<lifeless> salgado: you didn't leave the team directly
<sinzui> salgado, yes we have standing
<sinzui> We can make you excellent so that you can post to any list
<sinzui> salgado, you have excellent standing now
<salgado> sinzui, heh, no need to worry; was just wondering because a message sent to launchpad-dev@ while I was not subscribed seems to have been discarded
<salgado> oh, well, thanks
<sinzui> salgado, I see messages I moderate arrive. Maybe out list is ours/days behind again
<salgado> sinzui, I resent it and it's arrived already, so I think that's unlikely?
<sinzui> I agree
<lifeless> right, the lack of moderation notice is what was odd
<lifeless> I'm thinking its because of a glitch in that you didn't 'leave' the team - you had indirect membership. I am probably wrong.
<lifeless> sinzui: speaking of IE9 937722
<lifeless> bah
<lifeless> sinzui: speaking of IE9 - bug 937722
<_mup_> Bug #937722: error loading Inkscape page on IE9 <bugs> <ie> <javascript> <Launchpad itself:Triaged> < https://launchpad.net/bugs/937722 >
<lifeless> sinzui: I'd like to get your thoughts on IE* + js compat.
<lifeless> sinzui: and chrome frame. Do you have time to talk (voice) today ?
<sinzui> in a few minutes
<lifeless> cool
<rick_h> lifeless: that looks like a dupe of bug 894797, should be pretty simple fix for that one at least
<_mup_> Bug #894797: bug portlet ajax calls break in IE and break js for other features <bugs> <ie> <javascript> <markup> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894797 >
<rick_h> though not sure since that ie9 and my test/bug was ie8 I believe
<lifeless> rick_h: sure, its more the overall challenge we're setting ourselves I'm interested in
<rick_h> lifeless: gotcha, ok.
<lifeless> rick_h: (I mean, please do dupe it :P)
<rick_h> well, I'm not 100% sure, it's a different error string it looks like
<lifeless> rick_h: do you have any thoughts; is supporting IE9 harder than running parallel form implementations for all of LP ?
 * rick_h is trying to follow the screenshot vs error message
<rick_h> no, supporting IE9 isn't a huge issue. There's really just a couple of things we hit like this portlet stuff
<rick_h> which should never have happened anyway
<rick_h> the big thing is that as we get more client side and worry about the html5 history things that's currently causing issues for IE and opera
<rick_h> that's a larger chunk of work, but still think that should be done
<rick_h> imo
<lifeless> what do you think of chrome frame ?
<rick_h> I've never used it, and not sure it's always available as an option.
<rick_h> should be good/nice, but then again it's kind of like tossing a towel over some issues and allowing things to be a bit lazier in a way.
<lifeless> you're thinking locked down corporate w/activeX white-listing ?
<rick_h> lifeless: yea, I know you can install it sans-root now, but still not tested to see if it's availin all situtations
<rick_h> plus just the whole default non-technical user experience of first time ubuntu users/etc
<rick_h> that might not have chrome frame in their heads anyway
<lifeless> I don't think users really see it do they ?
<rick_h> there's a certain amount of just 'good practice' that I feel falls away
<lifeless> its just a signed control, then the whole browser engine is handed over
<rick_h> lifeless: no, I just mean that they don't know to go find frame and install it
<lifeless> rick_h: they don't have to
<lifeless> oh hmm
<lifeless> I *thought* you didn't have to but perhaps recent IE makes it a separate step
<rick_h> my understanding is that they did?
<rick_h> right, you can hint to them to go install it
<lifeless> http://www.chromium.org/developers/how-tos/chrome-frame-getting-started#TOC-Detecting-Google-Chrome-Frame-and-P
<rick_h> but it's a local change to the client's browser
<rick_h> If Google Chrome Frame is not installed, you can direct your users to an installation page.
<rick_h> If Google Chrome Frame is installed, it detects the tag you added and works automatically.
<sinzui> lifeless, sorry. X/Xinit is dying and not restoring.
<lifeless> sinzui: \o/
<lifeless> sinzui: I swear, you have a magic magnet
<sinzui> I am going to restart to hope I get 6 hours of stability.
<rick_h> lifeless: yea, so this link you sent loads up the "install chrome frame" bit in a modal popup served via them (iframe?) and then once the install is done it can try to reload your page or another page
<lifeless> I see that now :)
<lifeless> I was sadly hopeful for magic
<rick_h> yea, it's magic once you get past the install parts, but that's a big step
<rick_h> comparing UA in GA it looks like nearly 10% of users don't get the buglisting history stuff
<rick_h> 9.86ish
<rick_h> heh, and 3% of FF users are still 3.6 which doesn't either ugh. Not bad though I guess, that 96% of FF users are 7.0 or greater
<sinzui> lifeless, I can talk now. Which technology...preferably something I can use my phone with. skype or hangout
<lifeless> skype is easy
<sinzui> I am on skype
<wallyworld> thumper: your +recipes change is merged and in buildbot now. will be deployed tomorrow at the latest
<rick_h> lifeless: if you get a sec can you ok this: https://code.launchpad.net/~rharding/lp-source-dependencies/add_lpjsmin/+merge/94051 ? As part of the moving the js min into a package in pypi as we discussed
<lifeless> rick_h: we don't review changes to lp-source-deps, just commit :)
<rick_h> lifeless: ok, wasn't sure if I needed sign off based on the comments in buildout.cfg there
<lifeless> rick_h: you may need a review of the change in LP to use it
<lifeless> rick_h: or it may be up to you, based on the OptionalReviews policy
<rick_h> lifeless: yea, I'll definitely have that
<rick_h> I'll probably even ask StevenK to do it since it's around stuff he's done recently and he'll kick me into place best :)
<rick_h> StevenK: so yea, when you're around can you peek at this and make sure it's ok? I've updated the lp-source-deps repo with the two new packages. https://code.launchpad.net/~rharding/launchpad/use_lpjsmin/+merge/94054
<thumper> wallyworld: so, backrub?
<sinzui> backrub? I didn't know this was the channel to get some man-on-man action
<james_w> can anyone see the link to download the file on http://pypi.python.org/pypi/oops_amqp ?
<lifeless> sinzui: you call them men ? :P
<lifeless> james_w: its missing, possibly wallyworld did something unusual
<lifeless> wallyworld: when yo udid the release what exact command did you run ?
<wallyworld> thumper: ok. you bring the oil
<wallyworld> lifeless: i can't recall exactly. something like "setup.py sdist upload --sign"
<wallyworld> sinzui: thumper had to bribe me to get that +recipes fix done :-)
<sinzui> wallyworld, You mustn't manage many. I would have done that for free so that I could stop looking through my browser history
<wallyworld> sinzui: i would also have done it anyway "for free". i was joking about the bribe
<sinzui> wallyworld, I know you well enough to see the joke
<lifeless> wallyworld: that should have done it, but th efile is missing
<lifeless> wallyworld: if you login and go to the files view for it, you can upload it now ;)
<wallyworld> lifeless: will do. not sure what happened
<lifeless> me neither
 * sinzui takes a break to listen to Sonic Youth on rte 2fm
<lifeless> wallyworld: doing 'setup register' would have updated the metadata but not uploaded the file
<wallyworld> lifeless: i did that also
<lifeless> wallyworld: ah; and let me guess, you ran that second ?
<wallyworld> lifeless: perhaps, not sure
<lifeless> don't
<lifeless> only register once
<lifeless> when claiming the name, after that sdist upload -s will update the metadata *and* attach the file
<wallyworld> that makes sense. i thought i ran the upload last, but not sure now
<james_w> https://dev.launchpad.net/HackingLazrLibraries
<james_w> https://dev.launchpad.net/HackingLazrLibraries#releases is wrong then
<james_w> or is right but expects the tarball to be uploaded to LP and to be picked up from there
<wallyworld> james_w: file uploaded. can you see if you can access it?
<james_w> thanks wallyworld
<wallyworld> np
<wallyworld> sorry about the stuff up
<james_w> np
<james_w> and it's in the ppa
 * james_w moves on to patch datedir-repo
<wgrant> lifeless: Chrome frame is pretty pointless now
<wgrant> lifeless: IE8 and 9 are sufficiently unterrible now that it is simply embarrassing to not support them reasonably.
<lifeless> wgrant: indeed; see my reply to curtis
<wgrant> Didn't actually read the conversation, just saw a mention of /chrome frame/i and decided you were insane.
<lifeless> I'm so glad you're making informed judgements now
<wgrant> :)
<wgrant> But our browser support policy seems stuck in 2008
<sinzui> wgrant, stuck in 2008? I was surprise Lp supported IE 6 when I joined in 2007
<wgrant> sinzui: Sorry, stuck in the common world of 2008.
<wgrant> Where people had given up on IE6
<wgrant> But now treated IE8 with less disdain.
<StevenK> rick_h: Beh?
<sinzui> jcsackett, mumble?
<rick_h> StevenK: I can't translate beh
<StevenK> rick_h: What do you need me to do?
<rick_h> StevenK: just wondered if you can review that branch for me. You've done so much of the combo dir and such that you're best to sanity check I've caught all the places and the buidlout/etc should go together right?
<StevenK> The source-deps change?
<rick_h> StevenK: the lp branch for using lpjsmin (that depends on the source deps change)
<lifeless> StevenK: whats the thing to setup combo ?
<lifeless> (in an existing environment)
<StevenK> lifeless: You should read the mail I sent about it
<lifeless> StevenK: I should, but you're closer.
<StevenK> lifeless: I am on the stand-up, so the mail will be quicker
<rick_h> lifeless: make copy-apache-config install libapache2-mod-wsgi, and make jsbuild
<rick_h> and set the FF
<StevenK> No
<rick_h> no?
<StevenK> No.
<rick_h> ok, nvm then. I'm recalling incorrectly
<rick_h> ok, so make and not make jsbuild: https://pastebin.canonical.com/60685/
<StevenK> And libapache2-mod-wsgi is not a make target
<StevenK> And you need root for the first make
<rick_h> sorry, that was a missing comma
<rick_h> my bad, long day typo, please forgive :)
<rick_h> ok, well lifeless there's the email in the pastebin. Sorry for leading astray.
<lifeless> bah
<lifeless> touch /var/tmp/bazaar.launchpad.dev/rewrite.log
<lifeless> chmod 777 /var/tmp/bazaar.launchpad.dev/rewrite.log
<lifeless> chmod: changing permissions of `/var/tmp/bazaar.launchpad.dev/rewrite.log': Operation not permitted
<StevenK> Remove it
<lifeless> StevenK: oh, I figured that was a bug so guarded it in the makefile
<lifeless> StevenK: if [ -d /srv/launchpad.dev ]; then \
<lifeless>                 ln -sfn /home/robertc/source/launchpad/lp-branches/working/build/js /srv/launchpad.dev/convoy; \
<lifeless>         fi
<lifeless> ln: creating symbolic link `/srv/launchpad.dev/convoy': Permission denied
<StevenK> for j in be the for we will of ; do for i in $(bzr grep -l " $j $j ") ; do sed -e "s/\( $j \)$j /\1/" < $i | sponge $i ; done ; done
<lifeless> WTF
<StevenK> lifeless: /srv/launchpad.dev would have been created as root and then chowned to $SUDO_USER
<StevenK> That was for sinzui's benefit
<lifeless> ls -l /srv/launchpad.dev/ -d
<lifeless> drwxr-xr-x 2 root root 4096 2012-02-21 22:31 /srv/launchpad.dev/
<lifeless> echo $SUDO_USER
<lifeless> (nothing)
<StevenK> Oh, sorry, $SUDO_UID
<lifeless> echo $SUDO_UID
<lifeless> (nothing)
<StevenK> It's an existing pattern in the Makefile
<lifeless> that is, I'm ssh'd into my lxc container, I'm not root
<lifeless> sudo should have no reason to be confused
<lifeless> StevenK: so what should exist and what should it look like
<StevenK> lifeless: It should be owned by your user
<lifeless> ok, no errors now
<StevenK> sinzui: https://code.launchpad.net/~stevenk/launchpad/double-word-death/+merge/94065
<lifeless> ahahahahahahahaahahahah http://www.rabbitmq.com/memory.html
<lifeless> and this is why appservers hung after rabbit saturated
<sinzui> StevenK, while yuou are fixing spelling...https://bugs.launchpad.net/launchpad/+bug/931575
<_mup_> Bug #931575: typo "There is no package name blah". <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/931575 >
<wgrant> lifeless: You didn't actually approve.
<wgrant> Blargh
<wgrant> lifeless: It seems that mortals can't see the Ubuntu subscribe link still.
 * wgrant links to the page directly.
<wgrant> baaaaaah
<wgrant> The form on +subscriptions doesn't have a team selector.
 * wgrant gives up.
#launchpad-dev 2012-02-22
<StevenK> wallyworld_: I've had breakfast, but I think I need at least one cup of tea before our chat.
<wallyworld_> StevenK: np
<StevenK> Also got distracted by finishing my double-word-death branch
 * StevenK tosses it at ec2
<lifeless> wgrant: ?
<wgrant> lifeless: You said "heh, doit." but didn't actually vote.
<lifeless> twill be clear to onlookers :>
<wgrant> And the subsequent 6 lines were about that tag subscription bug that you commented on.
<lifeless> ah yes
<lifeless> which is beautiful win
<StevenK> wallyworld: Sorry, now I'm waiting for the jerk with the leaf blower to move
<wallyworld> StevenK: what? can't hear you over the noise
<StevenK> Or I get annoyed enough to go downstairs and ... re-position the nozzle
<wgrant> lifeless: Can we migrate to SQLAlchemy now please?
<StevenK> Add it sourcedeps. I dare you.
<lifeless> wgrant: raise it on the list
<wgrant> Hah
<StevenK> lifeless: You have no objections?
 * StevenK deals lifeless a penalty card for 'Lying'
<lifeless> StevenK: oh?
<lifeless> StevenK: deal yourself one
<lifeless> StevenK: then ask Francis what I've said in the past.
<StevenK> lifeless: You have strong opinions on everything we use, so I find it hard to believe you don't have one about this.
<lifeless> I never claimed no opinion
<rick_h> wgrant: is my new hero for that comment
<StevenK> if( vs if (
<rick_h> pep8 says if (
<StevenK> wallyworld: http://pastebin.ubuntu.com/852074/
 * StevenK stabs Firefox
<abentley> wgrant: it looks like bulk.create with a 0-length values will perform an execute and get a ProgrammingError.  What do you think about changing it to handle 0-length values by returning?
<abentley> wgrant: Also, is it efficient to load when the values aren't needed?
<wgrant> abentley: Much like the rest of Storm's SQL compilation stuff it is a bit fragile, yeah.
<wgrant> And loading should indeed probably be made optional.
<wgrant> So, yeah, 0-length values should probably just be a no-op, and loading should be disablable.
<abentley> wgrant: Cool.
<abentley> wgrant: The 0-length values is the only reason I kept insert_many in my latest branch.
<wgrant> abentley: I'm intrigued as to how your experiment was so slow.
<wgrant> Perhaps sqlvalues is awful.
<wgrant> abentley: How fast is the original code, btw?
<wgrant> Without any bulk optimisations.
<abentley> wgrant: I'm curious too, but I've got lots of things piling up at the moment.
<wgrant> s/fast/slow./
<wgrant> Yeah
<abentley> wgrant: I took over from gmb, so I didn't try it.  I know it exceeded the branch scanner job timeout.
<wgrant> Heh
<mwhudson> which bit of the scanner is timing out?
<lifeless> the wrapper
<wgrant> Both BranchRevision and Revision insertion have been problematic.
<mwhudson> ok
<wgrant> And there was a libreoffice non-DB timeout yesterday.
<wgrant> I didn't investigate.
<abentley> mwhudson: The db insertions of Revisions and BranchRevisions.
<mwhudson> i worked on BranchRevision pretty hard at one point
<mwhudson> as you have probably noticed
<wgrant> Yeah
<wgrant> IIRC it uses manual string SQL now?
<wgrant> Like the rest of LP's existing bulk inserts.
<wgrant> Which I'm going to port to the new shiny tomorrow, probably.
<mwhudson> yeah
<mwhudson> that'll be good
<abentley> wgrant: Are you fixing the same bug as me?
<mwhudson> (even if it doesn't end up being faster)
<wgrant> abentley: No.
<wgrant> abentley: I'm working on disclosure stuff.
<wgrant> abentley: jelmer is working on the bzr side of scanner timeouts, though.
<abentley> wgrant: I've ported it over to insert_many already.
<wgrant> abentley: I informed him of your potential conflicts.
<wgrant> abentley: Yeah, but there are other places that do manual bulk inserts.
<wgrant> BPPH creation and a few others.
<abentley> wgrant: Oh, I misread you.
<wgrant> I'm not going to touch your stuff :)
<abentley> wgrant: Well, if you insist on doing it better than  me, I'll grumble, but not too much :-)
<wgrant> Heh
<wgrant> abentley: You've wrapped create() rather than fixed it directly?
<wgrant> The initial version will land in about 90 minutes, so I'll land a quick followup with the changes you suggested unless you already have one.
<abentley> wgrant: No, I hadn't seen create.  I was using "Store.execute(Insert( ..."
<wgrant> Ah
<wgrant> I'll make those two changes now, then.
<wgrant> Thanks.
<wgrant> Hopefully this will make people avoid bulk creation a bit less.
<wgrant> 'cause there are a lot of places that could use it.
<wgrant> abentley: I apologise for colliding with you. I didn't realise you were working on this until I was proposing my Storm branch and saw yours.
<abentley> wgrant: No worries.  Strange coincidence that we both tried to fix this longstanding gap in the same month.
<wgrant> Indeed.
 * wgrant benchmarks the old one too, just for curiosity's sake.
<abentley> wgrant, jelmer: In last week's profiles of lp:s~irar/gcc-linaro/slp-for-any-nops-4.6/, only 13% of branch scanner time was spent in getBazaarRevisions, so I wasn't too concerned about it.
<abentley> wgrant, jelmer: Though wgrant's bulk-insert will make the db side faster.
<wgrant> wgrant@lucid-test-lp:~/launchpad/lp-branches/devel$ ls -lh /var/tmp/lperr
<wgrant> total 191M
<wgrant> drwxr-xr-x 2 wgrant wgrant 189M 2012-02-22 13:24 2011-12-05
<wgrant> That can't be good...
<StevenK> A 189M dentry? Niiiiiiiiice
<wgrant> Yeah
<wgrant> I think that's the day I reproduced the poppy rabbit explosion.
<wgrant> Which could mean there are millions of files in there.
<StevenK> Likely
<StevenK> ls -1 /var/tmp/lperr/2011-12-05 | wc -l ? :-)
<wgrant> I unfortunately already deleted part of it, but I'm already doing that.
<wgrant> Been going for 3 minutes...
<StevenK> Haha
<wgrant> It's probably statting all the children :*(
<StevenK> It will, yes
<wgrant> Might write something to just readdir instead.
<wgrant> Oh no
<wgrant> It's still calling getdents
<wgrant> 512 at a time
<StevenK> Bwahaha
<wgrant> ls is only using 3% of my RAM so far, though.
<wgrant> Ew
<wgrant> At least 4 statements per Revision insert in devel.
<wgrant> And often 7
<wgrant> And oh look it timed out.
<wgrant> Less than half way through.
 * wgrant starts it without a timeout and goes away for 15 minutes.
<wgrant> Huh
<wgrant> It doesn't really seem to be inserting them in any kind of order.
<wgrant> abentley: Just tested on lp:launchpad. 9:00 without bulk inserts, 1:45 with mine.
<wgrant> Probably doesn't improve the normal case much, but the initial scan of a large new history is 5x faster even with no DB latency.
<wgrant> Yeah, doesn't help the 0:47 new branch with existing Revisions case at all.
<wgrant> mwhudson: Did you ever look at sharing history?
<wgrant> eg. bisect to find the latest shared history and then delegate to that.
<lifeless> wgrant: it was examined, based on the work jam had done in history-db
<lifeless> lol, pybars has had 65 downloads
<lifeless> interesting
<wgrant> s/interesting/google/
<lifeless> you think?
 * wgrant finds out.
<mwhudson> wgrant: for branch revision?
<wgrant> mwhudson: Yeah
<wgrant> It would presumably reduce the table size by something like 99%.
<wgrant> lifeless: Oh, bah, hosted on PyPI rather than LP?
<wgrant> No referers for me :(
<lifeless> pypi -> centre of the python universe
<wgrant> Yeah, but often our downloads are hosted on LP.
<StevenK> wgrant, lifeless: Can I Disapprove https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883 ?
<StevenK> garbo job or API script, damn it
<wgrant> As I said a couple of days agoo, I think it should be an API script. But it needs discussion.
<lifeless> I recommended garbo to salgado a week or two back
<StevenK> All of three of us are in agreement about "Oh god, not that>"
<StevenK> s/>/./
<lifeless> I think its reasonable to say that you want to discuss it
<lifeless> and yes, I agree, API or garbo makes much more sense
<StevenK> lifeless: "I'm going to disapprove this. This requires discussion, and certainly doesn't require a one-shot script -- the usual modus operandi for migrations like this is either a garbo job or an API script."
<StevenK> lifeless: Sound fine?
<mwhudson> wgrant: i'm not sure how that would help things like mapping a revision to a branch
<lifeless> StevenK: personally, I would needs-info rather than disapprove, but if you're reviewing, do what you prefer
<mwhudson> you can do cleverer things that BR of course, but you end up doing something like historydb i think
<lifeless> StevenK: I'll note that its a tunableloop already
<mwhudson> there is a wiki page somewhere about being cleverer than BR, come to think of it
<lifeless> StevenK: which means it can trivially sit in the garbo
<wgrant> wallyworld__: Om nom nom XSS
<wallyworld__> hmm?
<wgrant> Your new sharing picker thing is nice and XSSable.
<StevenK> lifeless: Meh, needs fixing
<wgrant> And not feature-flagged :(
<lifeless> StevenK: sure, or that
<wallyworld__> wgrant: it's on a view that is not used
<mwhudson> wgrant: https://dev.launchpad.net/Code/BranchRevisions
<wgrant> wallyworld__: That's only barely mitigation.
<lifeless> wallyworld__: if someone can craft a url to it, they can exploit it
<lifeless> (if there is an xss hole...)
<wallyworld__> i'm not sure if the view is featured flags or not
<wgrant> It's not.
<wallyworld__> so we need to ff the view
<StevenK> lifeless, wgrant: You both chimed in on https://code.launchpad.net/~sinzui/launchpad/team-titles/+merge/93675 ? Do you have any remaining objections or were you just commenting?
<lifeless> StevenK: sinzui took my suggestion into account
<wgrant> I'm OK as long as the bug title changes are reverted.
<lifeless> StevenK: the title is now preserved - see his QA instructions
<lifeless> wgrant: they aren't reverted, but titles are shown in the page title
<wallyworld__> lifeless: i'm in iharness and using Launchpad.login_with(). but there's no get() or named_get() method on the lp instance that is returned. what's the best way to get an instance of WebServiceCaller with those methods?
<wgrant> Launchpad.login_with() is a full launchpadlib
<wgrant> Heh
<wgrant> This is actually a two-layer XSS
<wgrant> Cool
<StevenK> wgrant: Are you okay with the MP given what lifeless said?
<wallyworld__> wgrant: there's no get or named_get methods though it seems
<wgrant> wallyworld__: No, launchpadlib uses attribute access instead.
<wallyworld__> so how to i get something i can invoke named_get or get on?
<wgrant> Do you want to do that, or do you just want to experiment with the API?
<wallyworld__> i have a service i think  have exposed and  i want to try and call it
<StevenK> Write a webservice test, then?
<wgrant> Might as well just use launchpadlib. It's a nicer interface than WebServiceCaller
<wgrant> And I don't know if one can use WebServiceCaller from outside AppServerLayer
<wgrant> lp = Launchpad.login_with(fewfwefwehfuwef)
<wgrant> obj = lp.load('/path/to/your/resource')
<wgrant> obj.someattribute
<wgrant> obj.someMethod(some=arg)
<wallyworld__> .load is what i was missing
<wallyworld__> it's not a domain object though
<wgrant> Have you regenerated your WADL?
<wallyworld__> yes
<wallyworld__> i'm treating it as a service
<wgrant> You should be able to access it by lp.<whatever name you registered it as>
<wgrant> But note it's the top-level collection name, not its URL.
<wallyworld__> so i assume export_as_webservice_entry() is the correct way to register it
<wgrant> wallyworld__: Anyway, these XSSes are now blocking deployment.
<wgrant> No.
<wgrant> Collections aren't entries.
<wallyworld__> it's not a collection
<wallyworld__> it's a service
<wgrant> Hm, so it's a singleton entry?
<wgrant> I guess that's OK, then.
<wgrant> These things generally get exported as collections, but I guess your way makes more sense.
<wallyworld__> the ws api is geared to export entries and collections sadly
<wgrant> Just means there's no way to access it without lp.load
<wallyworld__> ideally i'd like to say something like lp.get('/myservicename?ws.op=something')
<wallyworld__> or lp.post(/myservicename?ws.op=something&data=foo)
<wallyworld__> it that possible?
<wallyworld__> wgrant: so where's the xss hole? in the picker config passed to the client from the view?
<wgrant> wallyworld__: LaunchpadWebServiceCaller is the interface you get in tests.
<wgrant> It looks like it speaks TCP
<wgrant> So you can probably use it.
<wgrant> LaunchpadWebServiceCaller(protocol='https') should work
<wgrant> I hope
<lifeless> wallyworld__: so why do you want that rather than lp.thing.something() and lp.thing.something(foo) ?
<wallyworld__> lifeless: whatever works. i tried that and couldn;t get it to work
<wallyworld__> that would be my preferred way
<wgrant> Oh
<wgrant> Then make that work :)
<wgrant> What's the error?
<lifeless> wallyworld__: so, suggestion, and I'm not being snarky - don't talk about what you don't want. Talk about what you do want!
<wallyworld__> sure.
<wallyworld__> the error is that lp has no attribute thing
<wgrant> What's the expression?
 * StevenK distracts wgrant with something shiny
<wgrant> StevenK: Oh?
<StevenK> wgrant: Are you okay with the MP given what lifeless said?
<wallyworld__> so i'm not registering something properly. actually it's using the system launchpadlib
 * wgrant reads the diff
<wgrant> wallyworld__: You sure you're connected to dev and not prod?
<wallyworld__> yep lp = Launchpad.login_with('testing', service_root='https://api.launchpad.dev', version='devel')
<wgrant> And what is failing, and what is the exact text of the error?
<wallyworld__> <launchpadlib.launchpad.Launchpad object at 0x85c4e90> object has no attribute 'accesspolicies'
<wgrant> Right, you didn't register it as a top-level collection, so it won't appear there.
<wallyworld__> clearly i haven't set something up correctly
<wgrant> You have to use lp.load to get at it.
<wallyworld__> so how do i avoid the lp.load() step?
<wgrant> You can't.
<wgrant> Unless you say define it as an empty collection.
<lifeless> well
<wallyworld__> ideally lp.accesspolicies would map to my AccessPolicyService
<lifeless> what are the methods that you will have on this thing
<wallyworld__> at the moment, i have foo()
<wallyworld__> just to test it
<lifeless> ok, but really
<lifeless> help me understand
<wgrant> StevenK: Bug titles are missing a colon
<wgrant> It's now "Bug #1 blah blah blah"
<wallyworld__> ok, getAccessPoliciesForProducts(product_collection, user)
<_mup_> Bug #1: Microsoft has a majority market share <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <LibreOffice Productivity Suite:New> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:In Progress> <The Linux OS Project:In Progress> <metacity:In Prog
<wgrant> Which doesn't make sense.
<wallyworld__> lifeless:  so i want to go lp.accesspolicies.getAccessPoliciesForProducts(product_collection, user)
<wallyworld__> or something like that
<StevenK> wgrant: Right, noted in my +1, thanks
<wallyworld__> lifeless: and in the browser, there'd be an XHR call to the same api to get the data for the view
<lifeless> wallyworld__: you want to pass *in* a vector of products?
<wallyworld__> a set based interface
<wallyworld__> but there'd be just the one normally
<lifeless> so, I'm hugely in favour of set based calls, as you know
<lifeless> but all our existing api stuff would just make that a method on IProduct and IProject and IDistribution
<wallyworld__> noooooooo
<lifeless> (or IPillar probably)
<wallyworld__> putting service methods in domain objects is just so wrong
<wallyworld__> and i'd rather avoid that in this new tranch of work if possible
<wallyworld__> so i want to try a service based approach
<lifeless> That sounds good, but I htink then you need to commit to fixing the environment so that your new work is no worse than the current (which swimming upstream would be)
<lifeless> specifically, you'll need to teach launchpadlib about named objects or some such
<lifeless> one hack you could use is to create a collection of services - IService or something
<wallyworld__> right, that's what i didn;t know - i didn't realise it couldn't do that
<wgrant> Or just define this as a collection with [] as the default content.
<wgrant> collections have methods too
<lifeless> you'd need to fix the launchpadlib bug where the interface the wadl specifies the type of collection members even when they return a more specific type (and self-annotate as such)
<wgrant> I'm not actually sure if /branches is usefully iterable, for example.
<lifeless> or as wgrant says, an empty collection will get you a named service fairly cheaply
<wallyworld__> and i'd still need .load() right?
<wgrant> No
<wgrant> Top-level collections show up as lp.foo
<lifeless> wallyworld__: note though that the web API /only/ exports domain objects as far as its concerned, there isn't a separation there - and in most restful things I've seen there isn't a service layer as such
<wgrant> See eg. IBranchSet.
<lifeless> wallyworld__: so I'm not-at-all-sure your approach makes sense for the *external* API.
<lifeless> wallyworld__: I'd kind of like to talk it through with you and curtis
<lifeless> wallyworld__: I'd like to understand the vision
<lifeless> wallyworld__: where you want to take it
<wallyworld__> lifeless: i haven't talked to much to curtis about it
<wallyworld__> lifeless: want to jhoin our standup tomorrow?
<wgrant> This is very un-lazr.restful-ish.
<wgrant> But I think that's probably a good thing.
<wallyworld__> i'm experimenting a bit first to see what can be done
<lifeless> wallyworld__: I probably have conflicting calls; I have 3 in a row; what time is it ?
<wallyworld__> 8:00am AEST
<wgrant> So 11:00am
<wgrant> Damn Queenslanders.
<wgrant> Oohoohoo
<lifeless> 11am? surely you jest, he's not 3 hours out from you is he ?
<wgrant> Just realised what time the TL call is going to be for bigjools.
<wgrant> lifeless: He's 3 hours from you.
<lifeless> so, 11am I can do
<wgrant> One hour from us.
<StevenK> wgrant: Do share?
<lifeless> 6am
<lifeless> at the moment
<wgrant> lifeless: 6am for you, right?
<lifeless> no
<wgrant> Which makes it 3am for bigjools :)
<wgrant> Oh
<lifeless> 9am at the moment
<lifeless> hugely civilised
<wgrant> When did that change?
<wgrant> Disappointing.
<lifeless> when mrevell said roughly 'uhm, I need to put my kids to bed at the current time, can we change it'
<wallyworld__> lifeless: so i want to try and adopt a service oriented approach, using popo for the business model objects, separate from the storm db layer objects, and views getting flattened data from api calls etc
<wallyworld__> and still support "sensible" api access from launchpadlib
<lifeless> so this in some ways fits with where I want to take LP itself: remember I want to gut the server render layer to have no DB access at all
<wallyworld__> it's how i've always built these types of systems previously
<lifeless> however I think it would be good to separate the discussion into what you want the API to do and look and feel, and what you want to do in the appserver; because all the appserver work will be irrelevant as we move to more SOA backends
<lifeless> thats not a reason not to do appserver work - because some things pay for themselves very quickly, and improvements now are improvements now
<lifeless> but I still want to get a handle on the specifics you're intending
<wallyworld__> sure, np. it's still handwavy
<lifeless> and how we'll measure the success of the experiment; and how we'll roll all the way forward, or roll back (depending on the assessed success)
<wgrant> Anyway, we need to revert this picker thing.
<wallyworld__> wgrant: what's the xss issue?
<wgrant> wallyworld__: Put some stuff in a product title
<wgrant> s/title/displayname/
<wgrant> And visit +sharing
<wallyworld__> so the json from the view doesn't get escaped properly?
<wgrant> Correct.
<wallyworld__> i thought it did?
<wgrant> You may recall that I strongly discourage ever encoding JSON manually.
<wgrant> And always using JSONRequestCache wherever possible.
<wallyworld__> wgrant: rather than revert, it's a simple one line fix to get the product title out of the picker header
<wgrant> That's not a fix.
<wallyworld__> why?
<wgrant> It's getting the one bit of bad data that I've found so far out of the unescaped section of the page.
<wgrant> Rather than removing the injection of user-provided data into unescaped section of the page.
<wallyworld__> we do that same pattern in a few other places in lp too
<wgrant> Yes, and they're all bugs.
<wallyworld__> eg the inline picker widget
<wgrant> Just because something is embarrasingly terrible already doesn't mean we should perpetuate it.
<wallyworld__> sure. that's the trouble when it's in the code - the pattern can be reused
<wgrant> Hmm, I'm sure I've posed that point before and you've debated it :)
<wallyworld__> so given the product title is the only user editable exploit, we could simply remove that for now
<lifeless> wallyworld__: it is the problem, and thats why we need to finish our transitions and migrations more
<wallyworld__> lifeless: well, i'm sure people thought the lazr picker work was finished when they did it
<lifeless> wallyworld__: software is never finished ;)
<wallyworld__> that's an answer to a slightly different question
<lifeless> wallyworld__: we can only hope to put some of it in the ground soon :)
<lifeless> wallyworld__: ok, so what was the question ?
<wallyworld__> i was merely saying that the work in question was probably considered to be "finished", without the expectation there was still stuff to do
<wgrant> It was finished.
<wgrant> It was just wrong.
<wgrant> fg
<StevenK> No jobs running
<wallyworld__> wgrant: or the jsonrequestcache encoder could be used in this case
<wallyworld__> rather than stuffing new stuff in the cache
<wallyworld__> so the json would be properly escaped
<wgrant> That's what we've done in the past.
<wgrant> But nobody remembers.
<wgrant> This is why manual encoding is no longer permitted.
<wallyworld__> i'll do that then for now
<wgrant> (this is all because Microsoft are fucking morons, btw)
<wallyworld__> so manual encoding works if done properly using the right encoder
<wgrant> Yes.
<wgrant> But I wasn't going to say that, because it's always a bad idea.
<wallyworld__> and in this case, i forgot to use the right encoder
<wgrant> That's why it's always a bad idea.
<wallyworld__> wonder if we patch something  to force the correct encoder to be used by default
<wgrant> There's also a second layer of XSS after this.
<wgrant> So please just revert the whole thing :)
<wallyworld__> wgat's the other xss?
<wgrant> (an XSS I pointed out a few months ago during code review, but I was assured it would never be a problem)
<wallyworld__> which is?
<wgrant> The picker title is treated as HTML.
<wgrant> So even once you encode the JSON properly, the product title is then injected unescaped into the picker.
<wgrant> => revert it all until someone looks through it thoroughly.
<wallyworld__> i'd have to look to check, but the picker title is in the json data so should be escaped
<wgrant> Bonus points for deleting the rest of the pickers, but that seems unlikely.
<wgrant> That's not how escaping works :)
<wallyworld__> hmm? we use the correct encoder and the json data is all properly escaped
<wgrant> Right, when using the correct encoder the JSON is escaped fine.
<wgrant> But the JSON isn't really relevant for the second one.
<wallyworld__> so that's what i was saying - i'd have to check but i thought the picker title was in the json data
<wallyworld__> so would be escaped
<wgrant> It is.
<wgrant> That doesn't mean it's escaped properly in the picker.
<wallyworld__> but if it comes from json data that is escaped....
 * wgrant reverts it.
<wallyworld__> hang on a minute
<wallyworld__> the issue can be easily corrected using the proper encoder
<wgrant> The first issue.
<wallyworld__> and thye second issue is moot
<wgrant> Oh?
<wgrant> It happens, so it's not moot.
<wallyworld__> it is since the title is escaped in thje json
<wgrant> The JSON is escaped.
<wgrant> The title is not escaped in the JSON.
<wgrant> The point of escaping the JSON is to get the JSON through unharmed.
<wallyworld__> if that's the case, then ALL of our pickers have this issue
<wgrant> That doesn't meant he contents of the JSON are escaped.
<wgrant> Yes.
<wgrant> But few/none have variable data in the title.
<wgrant> Most have no variable data in the config at all.
<wgrant> Otherwise I would have started reverting every picker branch a year ago until it was fixed.
<wallyworld__> ok, i'll revert this one.
<wgrant> Thanks.
<wgrant> This is similar to the reason I want to patch our mustache implementations to replace {{{ with {IAcknowledgeThatWgrantWillProbablyMaulMe{
<StevenK> s/Probably//
<lifeless> wgrant: s/mustache/handlebars/
<lifeless> wgrant: and I know where the plumbing is to do that
<wgrant> Heh
<wgrant> Tempting to try making the factory more declarative at some point.
<wgrant> Probably cut 40% off the test suite time.
<StevenK> 2.4 hours is better than 4 :-)
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bulk-insert-2/+merge/94084
<StevenK> wgrant: +1
 * StevenK wonders if he can force a bug to be a dupe
<StevenK> Since dupe.markAsDuplicate(bug) doesn't work since dupe.owner can't see bug
<wgrant> (one X crash later): StevenK: Thanks
<StevenK> Haha
<StevenK> Perhaps we need to bribe huwshimi to 'pay a visit' to RAOF.
<huwshimi> StevenK: I can be there in 7 minutes
<StevenK> Haha
<wgrant> huwshimi: You worked on the current loggerhead theme, right?
<huwshimi> wgrant: I'm not sure how to appropriately answer that question...
<huwshimi> wgrant: Hypothetically, yes. If-stabbing-is-a-possibility, no.
<wgrant> Heh
<wgrant> Just wondering if you'd complain if I removed the 4px of padding around each line.
<wgrant> I only stab for security holes :)
<huwshimi> wgrant: Let me take a look
<StevenK> (Pdb) p user
<StevenK> <lp.registry.model.personroles.PersonRoles object at 0xfd81990>
<StevenK> BAH
<wgrant> wallyworld__: Any luck with bending launchpadlib to your will?
<wallyworld__> wgrant: not yet, had to revert that branch and redo the mp
<wgrant> Ah
<huwshimi> wgrant: So you want to remove the 4px from the tds?
<wgrant> huwshimi: Yeah
<huwshimi> wgrant: Why would you want to do that?
<wgrant> The code is pretty unreadable at present.
<wgrant> It is currently at a density of 0.5
<huwshimi> wgrant: Oh, you're not talking about the file listing then...
<wgrant> Oh, no, http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/bootstrap.py
<wgrant> Sorry
<StevenK> wgrant, wallyworld__: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/bug-limitedview/+merge/94088 ?
<wallyworld__> i can do it
<wgrant> The file listing and changelog are pretty dense.
<StevenK> wallyworld__: Thanks. How long before you need to disappear?
<wallyworld__> StevenK: gotta go to soccer in about 90 mins
<huwshimi> wgrant: I have no problem with that at all :)
<wallyworld__> got a reprieve from picking up the kid today
<wgrant> huwshimi: Thanks.
<huwshimi> wgrant: It may require a little padding, but I'll leave that up to you
<huwshimi> wgrant: Just check you don't break the other listings at the same time
<StevenK> wallyworld__: If it takes you 90 minutes to review, I'd be concerned. :-P
<wallyworld__> yep
<wgrant> huwshimi: Yeah, I think 1px either side is possibly OK
 * wgrant checks what other sites do.
<huwshimi> wgrant: (I might have inadvertently added that spacing when I was trying to fix the other padding)
<wgrant> Ah, indeed.
<huwshimi> wgrant: I'll leave it to your judgement :)
<wgrant> Thanks.
<huwshimi> wgrant: Thanks that had been bothering me too
<wallyworld__> StevenK: typo line 88
<wallyworld__> StevenK: are there existing tests for the bugs security adaptor?
<wgrant> wallyworld__: Oh, that reminds me, you misspelt duration in that commit too :P
<wgrant> huwshimi: I might also try to make that view more friendly for copying.
<wgrant> I wonder if side-by-side <pre>s works.
<StevenK> wallyworld__: I'm not certain
<wgrant> Rather than a terrifying table.
<huwshimi> wgrant: yeah, you could easily float divs next to each other with numbers in one and code in the other
<StevenK> wallyworld__: I've added another test: http://pastebin.ubuntu.com/852269/
<wallyworld__> StevenK: the new unit tests that are there are good but are incomplete. perhaps a comment on the test case would be good explaining that mainly limitedView is being tested
<StevenK> wallyworld__: Incomplete? You think I'm missing a scenario?
<wallyworld__> StevenK: get_bug_privacy_filter() is the method that is used
<wallyworld__> to see if someone can view the bug
<wallyworld__> it covers subscribers also i think
<wallyworld__> IBug.userCanView() calls get_bug_privacy_filter()
<StevenK> wallyworld__: You've lost me
<wallyworld__> StevenK: so the security adaptor calls IBug.userCanView() to check for view permission
<wallyworld__> and IBug.userCanView() calls get_bug_privacy_filter()
<wallyworld__> which checks for subscribers
<wallyworld__> which the tests don't currently do
<StevenK> wallyworld__: Okay, I'm fairly sure that should be well-tested, so I can drop my unittests, except for the last one?
<wallyworld__> StevenK: i think so. just add a comment on the test case that just limitedview access is being tested
<wallyworld__> and leave it as an exercise for the reader to figure out where the other stuff is tested :-)
<StevenK> wallyworld__: http://pastebin.ubuntu.com/852278/
<wallyworld__> StevenK: looks good to me
<StevenK> wallyworld__: Pushing
 * wallyworld__ still waiting for diff
<StevenK> wallyworld__: Its updated for me
<wallyworld__> StevenK: just updated for me too. took a while. r=me
 * StevenK fills up QA-Landing
<wgrant> huwshimi: http://people.canonical.com/~wgrant/launchpad/lh-view/old-view.png -> http://people.canonical.com/~wgrant/launchpad/lh-view/new-view.png
<wgrant> (the new one is copyable, since it's a table of two <pre>s)
<huwshimi> wgrant: Awesome
<wgrant> Although there's still too much space above the top line
<wgrant> StevenK, wallyworld__: Any comments?
<wgrant> Trying to make loggerhead a bit less painful for reading code
<wallyworld__> wgrant: +1 from me, looks nice
<wgrant> fail
<wgrant> The great LP bug migration corrupted things.
<mrevell> Hi!
<poolie> mrevell, hi there
<adeuring> good morning
<al-maisan> Good morning adeuring, how are things?
<adeuring> hi al-maisan! things are fine here. how about you?
<al-maisan> doing well, thanks :)
<gmb> lifeless, Around?
 * jml keeps hitting refresh
<jelmer> 'morning abentley
<abentley> jelmer: morning.
<jelmer> abentley: I heard you're working on fixing the scaling issue in the branch scanner?
<abentley> jelmer: Yes, mainly by fixing the DB access.
<jelmer> abentley: cool
<jelmer> abentley: I'm working on removing the use of Branch.revision_history and Repository.get_ancestry from the lp codebase, and as such was also touching that bit of the code
<jelmer> it sounds like our changes shouldn't overlap though
<abentley> jelmer: No, I don't think I touched the bzr side.
<abentley> jelmer: It was only 13% of runtime when I profiled it.
<abentley> jelmer: lp:~abentley/launchpad/bulk-insert
<abentley> jelmer: You're just wanting to avoid deprecated interfaces?
<czajkowski> are there any plans in the works to support proxy on launchpad? have two bugs today with issues about using launchpad in one country and having to use a proxy
<jelmer> abentley: mostly, yeah. In the process I'm also changing things to not access all of the branch history/ancestry if it doesn't have to
<abentley> czajkowski: I haven't heard of any plans, but I hadn't heard of any issues, either.
<czajkowski> abentley: https://bugs.launchpad.net/launchpadlib/+bug/938542  and https://bugs.launchpad.net/launchpad/+bug/938580
<_mup_> Bug #938542: launchpadlib doesn't support system proxy <launchpadlib :Triaged> < https://launchpad.net/bugs/938542 >
<_mup_> Bug #938580: launchpad not opening in Syria.. <Launchpad itself:Triaged> < https://launchpad.net/bugs/938580 >
<abentley> czajkowski: Oh, launchpadlib.
<czajkowski> well one is launchpadlib and the other cant use LP without going through a proxy
<abentley> czajkowski: I would be surprised if they were related.
<abentley> czajkowski: So it sounds like one issue is that lazr.restfulclient (which launchpadlib uses) may not respect the system proxy settings.
<abentley> czajkowski: And the other issue is that access to Launchpad in Syria doesn't work, *except* when a proxy is used.  Which may be due to government interference with the Internet.
<czajkowski> nods
<abentley> czajkowski: I would ask webops about the second one.
<czajkowski> ok cheers
<abentley> czajkowski: The first one needs investigation, but can certainly be fixed if true.
<czajkowski> abentley: thanks
<salgado> mrevell, hey there. did you have a chance to talk to huw/dan about that new page we'd like to implement in LP?
<deryck> adeuring, abentley -- https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<deryck> rick_h, I'm recalling you being away today, if not, see ^^
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugtasks: 4*10^2
<danilos> mrevell, heya, we have a chat scheduled in ~11 mins
<mrevell> danilos, We do
<danilos> mrevell, ok, just confirming, let's do mumble
<mrevell> danilos, Dude, it's not 2010 any more.
<mrevell> :)
<danilos> mrevell, ah, right, sorry, proprietary software is "in" again ;)
<mrevell> heh, Mumble it is, then.
<salgado> mrevell, oh, is that the recurring one for which I've never gotten the invite?
<danilos> salgado, you don't deserve one
<mrevell> salgado, It's the recurring one, yeah. Let me see if I can fix the invite problem.
<salgado> danilos, I know I don't but with mrevell being English and all I thought I'd get one if only for his politeness ;)
<danilos> salgado, it's supposed to be in your canonical.com calendar
<mrevell> haha
<danilos> salgado, fair point
<mrevell> I've sent an invite to your Linaro.com address, salgado
<salgado> aha, now I got it
<mrevell> great
<lifeless> gmb: hi
<gmb> lifeless, Hi. W.r.t my earlier ping, I was checking whether the fix for bug 518016 was in the subunit 0.0.8beta that's in the LP sourcedeps. From what I can tell from commit timestamps it is (you added the snapshot 9 minutes after committing the bugfix) but I just wanted to be sure.
<_mup_> Bug #518016: No public API for tagging on TestProtocolClient <paralleltest> <Launchpad itself:Fix Committed> <subunit:Fix Committed> < https://launchpad.net/bugs/518016 >
<gmb> (I assumed "yes" for the purposes of marking the bug fixed)
<lifeless> gmb: yes makes sense to me :)
<lifeless> gmb: or even fix released, as its not something that needs deploying to be fixed
<gmb> lifeless, Righto, works for me. Thanks.
<lifeless> abentley: AWS SWF looks interesting
<abentley> lifeless: yes, indeed.
<danhg> When is Pay Day? Does anyone know?
<lifeless> http://blog.fastmail.fm/2012/02/20/building-the-new-ajax-mail-ui-part-2-better-than-templates-building-highly-dynamic-web-pages/
<lifeless> gary_poster: 24061
<lifeless> bah
<lifeless> bug 24061
<_mup_> Bug #24061: GPG error with apt-get/aptitude/update-manager behind proxy (BADSIG 40976EAF437D05B5) <apt (Ubuntu):In Progress> <update-manager (Ubuntu):Won't Fix> <apt (Ubuntu Precise):In Progress> <update-manager (Ubuntu Precise):Won't Fix> < https://launchpad.net/bugs/24061 >
<gary_poster> lifeless, ah-ha, with workaround included! thanks
<lifeless> gary_poster: basic troubleshooting is - remove the files from /var/cache/apt/ etc
<lifeless> gary_poster: if that doesn't work, you're into figuring out what host is serving inconsistent data
<lifeless> HTH
<gary_poster> lifeless, do you agree with workaround listed in bug?
<lifeless> the Acquire::BrokenProxy thing? if it works yes ;)
<lifeless> gary_poster: ^
<gary_poster> lifeless, heh
<lifeless> I don't recall the code changes that triggers, but working >> most anything else
<lifeless> you may, when running precise, naturaly encounter skew mid-archive-push, but the archive update scripts work hard to make the skew interval sub-second
<abentley> lifeless_: Is https://code.launchpad.net/~abentley/launchpad/callgrind/+merge/94283 also worthy of a LOC waiver?
<czajkowski> flacoste: should all questions end up being 'solved' rather than marked as 'answered' after a length of time
<lifeless_> abentley: It seems to me that if the bzrlib lsprof module supported just a little more glue, you could delete the entire profiling support for scripts from LP and not miss it
<wgrant> sinzui, jcsackett: https://wiki.ubuntu.com/Bugs/Status
<sinzui> StevenK, wallyworld, jcsackett, wgrant: http://pastebin.ubuntu.com/853310/
<lifeless> james_w: hey, so, django
<james_w> hey
<lifeless> james_w: what does your group do when getting a new django API/site up and running; do you use packages of django (are they in lucid-backports or a PPA) , ....
<james_w> yes, we use packages
<james_w> from lucid-cat currently
<james_w> (to get django 1.3)
<lifeless> do you use buildout or anything, or do sysadmins manually manage settings.py?
<lifeless> (+ mod-wsgi glue?)
<james_w> django.wsgi is delivered by puppet
<james_w> we *may* use buildout, but we don't currently
<james_w> settings are stored in a hierarchical way
<james_w> main.cfg for settings used everywhere
<james_w> dev.cfg and production.cfg for settings that are only appropriate in those environments, with a switch between them
<hingo> Hello #launchpad-dev. Is it possible my launchpad ppa ignores the epoch number given in .changes file?
<james_w> production_credentials.cfg delivered by puppet for secrets
<james_w> hingo, I doubt it
<james_w> all settings via django-configglue
<hingo> I'm trying to: dput ppa:drizzle-developers/ppa drizzle_7.1.31~rc-1~oneiric1_source.changes
<james_w> dependencies via a mix of packages (most things) and branches (things we develop that are tightly bound to the service)
<hingo> ...but nothing (not even a failure) appears at https://launchpad.net/~drizzle-developers/+archive/ppa/+builds
<james_w> devstaging deployments in ec2/canonistack using fab and some custom puppet (not related to memento unfortunately)
<james_w> hingo, do you get an email about the upload?
<hingo> The thing is, oneiric ships with drizzle_2011.03.13-something. So I have increased epoch to 1.
<hingo> james_w: No.
<flacoste> czajkowski: nah, answered is fine in the end, since it's up to the reported to mark them solved
<wgrant> czajkowski: Users mark questions as solved once they confirm the answer.
<wgrant> czajkowski: We should generally not set them to Solved ourselves.
<james_w> hingo, then it is 99% likely that the .changes file is incorrectly signed
<james_w> hingo, what is your LP username?
<hingo> james_w: Ok. I created a generig GPG key. Do I need to use my own / matching email address?
<hingo> james_w: hingo
<czajkowski> flacoste: wgrant ok, just wondered as there are many not set as solved there. just curious
<wgrant> czajkowski: A lot of users don't do it.
<wgrant> But it doesn't matter much.
<james_w> hingo, you need to register the GPG key that you are signing the upload with: https://launchpad.net/~hingo
<wgrant> GPG verification of /srv/launchpad.net/ppa-queue/incoming/upload-ftp-20120222-215212-001713/~drizzle-devel
<wgrant> opers/ppa/ubuntu/drizzle_7.1.31-rc-1_source.changes failed: Verification failed 3 times: ["(7, 9, u'No public key')", "(7, 9, u'No public key')", "(7, 9
<wgrant> , u'No public key')"]
<hingo> james_w: Ok, this makes sense. I'll upload the public key.
<hingo> james_w: That explains it. Thanks a lot!
<james_w> np
<wgrant> StevenK: You also need to QA your double word thing.
<wallyworld_> what happened to buildbot?
<wgrant> webops: did someone kill it?
<thedac> wgrant: that was me. We had defunct processes that needed cleaning up
 * wgrant forces.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10^2
<abentley> lifeless: I can see how you could reduce the number of lines of profiling support for scripts, but how would you remove it entirely?
<hingo> There she goes: https://launchpad.net/~drizzle-developers/+archive/ppa/+builds?build_text=&build_state=pending
<hingo> james_w: Thanks.
<lifeless> abentley: If bin/py -m bzrlib.lsprof cronscripts/foo .... profiled cronscripts/foo in the same way bin/py -m pdb cronscripts/foo ... profiles it, then you don't need the script to know that it is being profiled
<lifeless> .
<james_w> lifeless, does that cover your questions?
<wgrant> lifeless, poolie: I want to make the soft timeout flaggable like the hard timeout is. But unfortunately the feature controller is destroyed before the soft timeout is checked.
<wgrant> I'm considering simplifying things by just storing the hard and soft timeouts in thread-locals at the start of the request.
<wgrant> Rather than relying on the feature controller for every timeout check.
<abentley> lifeless: Ah, I hadn't seen that before.  Yes, that might be worthwhile.  (Though profililing via bzrlib is a somewhat strange idea anyhow.)
<wgrant> A lot of bzrlib seems like it should be called "this should really be in the standard library"
<poolie> yeah
<poolie> especially osutils
<poolie> arguably also transport
<abentley> wgrant: IIRC, the lsprof module was rejected from the standard library.
<wgrant> Hah
<poolie> wgrant, so you're saying you'll basically get it from the feature controller and remember it
<poolie> oh also selftest
<wgrant> poolie: Yeah
<wgrant> poolie: We already preread it at the start of the request
<james_w> if a language had excellent distribution support then I think it would take over the world, but that may be naive
<wgrant> To get it cached so it doesn't require DB access later.
<wgrant> But it relies on the feature controller surviving.
<lifeless> james_w: I think so, basically if you look at oops-tools is uses buildout to drop a local copy of django
<lifeless> so its not getting bugfixes
<lifeless> james_w: is there a ppa with django 1.3 in it ?
<james_w> lifeless, yes
<wgrant> I could make the controller destruct later, but the way it's done now is reasonably nice.
<james_w> https://launchpad.net/~canonical-ca-hackers/+archive/production/+packages contains the stuff we use from CAT
<abentley> wgrant: The problem with the "batteries included" approach is when the batteries are all the same size :-)
<wgrant> Heh
<lifeless> james_w: any objection to e.g. LP using that, or nabbing packages from it ?
<james_w> lifeless, none, we stole them from CAT after all :-)
<lifeless> :P
<james_w> obviously co-ordination is encouraged, as we all end up having to share again when things are pushed back to CAT
<lifeless> yah
<lifeless> I was more checking you're not tossing known-broken stuff in there
<james_w> lifeless, I have all the dependencies of oops-tools packaged fwiw
<lifeless> cool
<james_w> lifeless, that PPA contains only what is deployed in production
<james_w> lifeless, we have a staging PPA for the packages we are waiting for copying to CAT
<james_w> so it's all very much things we expect to work
<poolie> wgrant, thanks for the loggerhead improvements
<wgrant> :)
#launchpad-dev 2012-02-23
<wgrant> lifeless: Thoughts on the timeout thing?
<lifeless> which thing
<wgrant> See here 50 minutes ago
<lifeless> uhm, why is the controller destroyed?
<lifeless> oh, paging it
<lifeless> in
<lifeless> ..
<lifeless> ..
<lifeless> ..
<wgrant> Controller destruction and soft request timeouts are both end-request events.
<lifeless> soft time out is checked at end of request
<mwhudson> it's removed in the endRequest event handler right?
<wgrant> Yep
<lifeless> you want the controller info in the oops
<mwhudson> woo components
<lifeless> so no, thread local values won't help
<lifeless> you either want the events ordered, or one event
<StevenK> Ah, so I can't delete a team I own? WTF?
<wgrant> Well, we don't have the controller info now, and I wanted a quick way out :)
<wgrant> But OK
<lifeless> wgrant: poolie put the flag stuff into the oopses for hard oopses
<poolie> lifeless, i haven't forgotten the timeline stuff :)
<poolie> just so you know
<poolie> it's still within shouting distance of the top of stack
<lifeless> poolie: thanks, I know; it was only last week we chatted
<wgrant> Maybe I can do the soft timeout in afterCall or so instead.
<wgrant> Unfortunately I've mostly forgotten how publication works.
<wgrant> Bah
<wgrant> GitHub has hung Firefox
<lifeless> o/
<mwhudson> clearly you should be using safari
<wgrant> wallyworld_: Ready to talk?
<wallyworld_> wgrant: sure, i'll start mumble
<wgrant> In [4]: lp = Launchpad.login_anonymously('fewfw', 'dev', 'devel')
<wgrant> In [5]: lp.access_policy_services
<wgrant> Out[5]: <lazr.restfulclient.resource.Collection object at 0x106abcec>
<wgrant> wallyworld_: ^^
 * StevenK wonders why deleting a team he owns brings up "There is 1 error"
<wgrant> wallyworld_: http://pastebin.ubuntu.com/853451/
<wallyworld_> wgrant: i can hear you
<wallyworld_> wgrant: i'll restart mumble
<StevenK> Now, why can't I delete a team I just created?
<lifeless> delete? Delete? DELETE?
<wgrant> LPCONFIG=development utilities/create-lp-wadl-and-apidoc.py --force lib/canonical/launchpad/apidoc/
<StevenK> lifeless: It says "There is one error" but gives no indication what that is
<lifeless> StevenK: prod? qastating? dev?
<StevenK> lifeless: qas
<StevenK> https://qastaging.launchpad.net/~foobarbazquux
<wgrant> Ah
<wgrant> Team deletion won't work
<wgrant> At all
<wgrant> Because qastaging's DB is beyond all reasonably definitions of fuckedness.
<StevenK> Right
<StevenK> Which means person merge is utterly fucked too
<lifeless> its being rebuilt as we speak
<wgrant> StevenK: Deletion is merge.
<StevenK> wgrant: I'll mark bug 934778 as untestable
<_mup_> Bug #934778: Typo in informational message after deleting a team <docs> <merge-deactivate> <qa-needstesting> <trivial> <Launchpad itself:Fix Committed by stevenk> < https://launchpad.net/bugs/934778 >
<StevenK> At the end of the day, it was a mechnical change anyway
<lifeless> the robots are revolting ?
<StevenK> sourcherry is, anyway
<StevenK> wgrant: Are you done with wallyworld_?
<wgrant> Not even close :(
<wgrant> StevenK: We're done now.
<wgrant> But I'm about to have lunch unless you need something.
<StevenK> wgrant: Go and have lunch, we can chat about LimitedView after lunch.
<wgrant> StevenK: Does it provide any immediate benefits?
<wgrant> Or should we deploy now without it?
<StevenK> It's not flagged, so if it works as intended it should remove the confusing UI
<wgrant> Oh, has the template been modified?
<StevenK> Not as yet
<wgrant> Then how can it remove the confusing UI?
<wgrant> Nothing uses LimitedView on Bug, does it?
<StevenK> I thought the template had been fixed by sinzui
<wgrant> Didn't you just say the template hadn't been modified?
<wgrant> How was it tested without an adapter?
 * StevenK was going by what wallyworld_ told him
<wallyworld_> hmm?
<wallyworld_> yes, the base page template supports rendering entites with limited view
<wallyworld_> it won't render the main or side slots for example
<wgrant> Ah
<wallyworld_> just put in a message saying "you cannot see this"
<wgrant> Not sure it will actually work, though.
<wgrant> The view init will probably trip over not having View
<wgrant> Needs testing
<wallyworld_> works for teams, and yes, if init touches anything meeding view, you're stuffed
<wallyworld_> need to add permission checks if that's the case
<wallyworld_> in the init
<StevenK> wgrant, wallyworld_: http://nowhiring.com.au/424936+job+Prime+Minister+of+Australia+ACT.aspx
<wallyworld_> urgh, asp
<wallyworld_> i might forward the link to krudd
<StevenK> Hah
<StevenK> I hear he has a bunch of free time now.
<wallyworld_> wgrant: the top level stuff and navigation rules all work nicely
<wgrant> wallyworld_: You have a trivial service?
<wgrant> StevenK: Did you change the traversal check?
<wgrant> StevenK: I can't traverse to bugs on which I should have LimitedView.
<StevenK> wgrant: There's a traversal check? :-(
<wallyworld_> wgrant:  yes, there's a trivial service
<wgrant> StevenK: That's how it 404s.
<wgrant> StevenK: You know that this is testable locally, and the view will probably need changes, right?
<wallyworld_> wgrant: StevenK: the traversal check has already been changed to only require limitedView IIRC
<wgrant> The launchpadlib collection check was. I don't know about the bug traversal check.
<wallyworld_> i thought it had but could be wrong
<StevenK> wgrant: So, I think I was led down the garden path a bit by wallyworld_
<wallyworld_> StevenK: it all works fine for teams
<wallyworld_> and bugs should be similar
<StevenK> wgrant: If you point me at the traversal check, I'll look into fixing it
<wallyworld_> if it needs fixing
<StevenK> wallyworld_: wgrant has already stated it does
<wallyworld_> do teams use different traversal?
<StevenK> wgrant: As it is, r14855 is safe to deploy, but it doesn't work.
<wallyworld_> the traversal stuff that was changed was in the base publisher
<wallyworld_> so it should work for most things, no?
<StevenK> [13:37] < wgrant> StevenK: I can't traverse to bugs on which I should have LimitedView.
<wgrant> wallyworld_: Yes, it's separate.
<StevenK> wallyworld_: ^
<wgrant> Only two types of objects have the forbidden-equals-404 check AFAICT
<wallyworld_> hopefully only bugs is different then
<wgrant>         # Get out now if the user cannot view the bug. Continuing may
<wgrant>         # reveal information about its context
<wgrant>         if not check_permission('launchpad.View', bug):
<wgrant>             return None
<wgrant> Ah crap.
<wgrant> StevenK: You are in for a world of pain.
<wgrant> StevenK: You need to introduce a new view, basically.
<wgrant> StevenK: On IBug
<wgrant> Since IBug doesn't have a +index of its own -- it redirects to the default task.
<wgrant> Which is private information.
<StevenK> Perhaps we should introduce a NotSharedWithUserView or something?
<wgrant> In related news, multitask bugs are hideous.
<wgrant> Anyway, let's deploy.
 * wgrant deploys.
<StevenK> wgrant: Including or excluding r14855?
<wgrant> Including.
<wgrant> No reason to exclude, really.
<StevenK> Or a reason to rollback
<wgrant> Well
<wgrant> Bug's security declarations are terrible.
<wgrant> But I don't think it's too holy.
 * StevenK sighs
<StevenK> Where did my route to the DC go?
<wgrant> lolxetel
<wgrant> StevenK: you need to fix Bug's security declarations before you continue.
<wgrant> We can deploy now, but you must do nothing further on this until you fix them.
<StevenK> Which you just said were terrible.
<wgrant> Yes
<wgrant> They are more terrible than I expected.
<wgrant> Not fatal, but not good.
<StevenK> So, my current plan was to look at Curtis' bug and branch so I can unblock him
<StevenK> But I can't reach the DC
<StevenK> This is reminding me of when wgrant couldn't.
<wgrant> qa-meh-itll-do
<StevenK> qa-wcpgw
<StevenK> Ah. I think I'm getting TTL expired
<StevenK> Because L3 are a bunch of idiots
<StevenK> I think v6 traffic is immune, which is nice
<wgrant> The DC is also immune to v6, though :(
<StevenK> Hmmm, the route has changed. I think Exetel may have noticed, since their core is now dropping everything
<StevenK>  19.|-- 4.69.134.77                                 88.9%     9  244.9 244.9 244.9 244.9   0.0
<StevenK> Muppets.
<StevenK>  20.|-- 4.69.137.77                                 88.9%     9  312.1 312.1 312.1 312.1   0.0
<StevenK>  21.|-- ???                                         100.0     8    0.0   0.0   0.0   0.0   0.0
<StevenK> Yay, I can hit the DC again
<StevenK> Nice how it took Exetel 30 minutes to drop their route to Telstra, since they are the cause.
<StevenK> What the?
<StevenK> Look at any bug reported against LP and hover over "Launchpad itself"
<wgrant> StevenK: Mmm, delicious 2007.
<wgrant> (2007 + bugs, but still 2007)
<StevenK> wgrant: So, what needs to happen with the bug security adapters?
<wgrant> StevenK: They leak tonnes of stuff that needn't be leaked.
<wgrant> Grab https://bugs.qastaging.launchpad.net/api/devel/bugs/718213/duplicate_of?ws.accept=application/json as anonymous
<_mup_> Bug #718213: Can't access due to content. <Internet Archive - Tech Support:New> < https://launchpad.net/bugs/718213 >
<wgrant> It's a dupe of bug #1234, which I've made private.
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<StevenK> Lots of that JSON is redacted
<wgrant> And lots is not.
<StevenK> wgrant: What do you want to see disappear?
<wgrant> Pretty much all of it.
<wgrant> I don't have access to the bug.
<wgrant> I don't really need to see much.
<StevenK> I think the issue is the <allow ... in the ZCML
<wgrant> Indeed.
<StevenK> Most of them should drop to launchpad.View ?
<StevenK> Which means anonymous users can't fetch them for public bugs eithwer
<wgrant> I would suspect so.
<wgrant> Oh?
<StevenK> Oh, I'm wrong?
<wgrant> launchpad.View is held by anonymous users for public bugs.
<wgrant> I would hope.
<wgrant> If it's not it's a bug.
<StevenK> So, id needs to stay, as does private.
<wgrant> Regrettably.
<StevenK> [16:06] < Solver> Dodo announced THE INTERNET to Telstra and they accepted it
<wgrant> Heh
<StevenK> wgrant: ^ Cause of my DC unreachability
 * StevenK kicks wallyworld_
<wallyworld_> ouch
<StevenK> That's what you get for importing ILaunchpadCelebrities into bugs.security
<wallyworld_> did i do that?
<StevenK> 14186.3.3   ian.boo | from zope.component import getUtility
<StevenK>                     |
<StevenK>                     | from lp.app.interfaces.launchpad import ILaunchpadCelebrities
<wallyworld_> i should have used personroles
<wallyworld_> not sure why i didn't
<wallyworld_> moment of madness
<StevenK> user is already an role
<StevenK> wallyworld_: I'm putting up a small branch to fix
<wallyworld_> ok. i thought by what you were saying you'd do it as a drive by
<StevenK> No, I don't want to cause other issues in a branch where I'm moving 30 attributes
<wgrant> StevenK: Be warned, there may be 10 or more tonnes of test fallout.
<StevenK> wgrant: Yes, which is why I don't want to do any drive-bys
 * wallyworld_ stabs unity/compiz/whatever... restart time :-(
<StevenK> wallyworld_: https://code.launchpad.net/~stevenk/launchpad/less-celebrites-bug-security/+merge/94313
<wallyworld_> looking
<wallyworld_> StevenK: r=me
<wallyworld_> thanks for fixing
<wgrant> Bah
<wgrant> I wish we were on postgres 9.1
<StevenK> Tossing through ec2 just to see if there are any failures
<wgrant> Bah
<wgrant> BugHeatUpdater is only 300 times faster.
<StevenK> Only
<wgrant> Yes, only.
<wgrant> Was hoping for more than 1000x.
<wgrant> Must still be something wrong.
<stub> A penchant for wishful thinking? :-)
<wgrant> Oh, blah, the bug->task heat mirror trigger.
<wgrant> Forgot about that.
<nigelb> wgrant: https://twitter.com/#!/nigelbabu/status/172561507148763136 :D
<nigelb> I hope I didn't convey that lp is only 300 times faster...
<wgrant> Heh
<wgrant> Oh oops.
<nigelb> Man, "What Could Possibly Go Wrong" should be like the key phrase for LP.
<StevenK> wgrant: Only 41 failing tests.
<StevenK> wgrant: Right, http://pastebin.ubuntu.com/853644/ looks pretty good, and has no failing tests.
<StevenK> Pity default_bugtask, getBugTask and userCanView had to be made public too
<wgrant> StevenK: default_bugtask does not need to be public.
<StevenK> The factory goes boom if it is not
<wgrant> That's not an excuse.
<wgrant> The factory can log in as an admin or remove the security proxy :)
<StevenK> userCanView is the excusable.
<StevenK> s/the //
<wgrant> Yes.
<StevenK> getBugTask is not, right?
<wgrant> Unless you are extremely convincing, it is indeed not excusable.
<wgrant> I would expect the only public attributes to be id, private, and userCanView
<wgrant> Possibly not even private :)
<StevenK> I'm leaving private for the time being
<wgrant> Yeah
<wgrant> private isn't leaking
<wgrant> It's just dirty
<StevenK>   File "/home/steven/launchpad/lp-branches/less-bug-leakage/lib/lp/testing/factory.py", line 1682, in makeBug
<StevenK>     bugtask = bug.default_bugtask
<StevenK> ForbiddenAttribute: ('default_bugtask', <Bug at 0xd21c150>)
<wgrant> with admin_logged_in()
<wgrant> Or rSP
<wgrant> Probably rSP
<wgrant> Depending on whether you're returning the task or not.
<StevenK> No, we're not returning it
<StevenK> Using it for date_closed and milestone only
<wgrant> Then you can just unproxy it and nobody will care :)
<StevenK> This branch is *tiny*
<StevenK> +7/-9
<G> StevenK: and you've got 2 lines for the future :P
<StevenK> I suspect I have a lot more than that.
<StevenK> Both wgrant and I tend to remove a lot of code.
<wgrant> zomg
<wgrant> BugHeatUpdater caught up
<wgrant> For the first time in more than a year
<wgrant> Maybe we will stop having so many oopses now.
<StevenK> steven@liquified:~/launchpad/lp-branches/less-bug-leakage% subunit-stats < lp.log | grep Failed
<StevenK> Failed tests:    307
<wgrant> Uhoh
<wgrant> Ah
<wgrant> Not so bad
<StevenK> Almost all of them are default_bugtask
<wgrant> That's why I haven't preemptively euthanised you yet.
<StevenK> Oh?
<StevenK> Why would you want to do that?
<wgrant> Because if they were 307 different failures, you'd want it.
<StevenK> Haha
<wgrant> stub: http://people.canonical.com/~wgrant/launchpad/bug-legacy-mirror.sql is my current plan for migration of bugs to the new privacy model. Lets us safely run both schemas in parallel during the migration without worrying about syncing in the app. Rather abhorrent, I know, but it's fast and temporary.
<stub> We have too much data to migrate in a 5 minute window?
<adeuring> good morning
<wgrant> stub: Given the interactions with already unperformant bug search, we'd like to be able to revert to the old ways quickly if things don't perform optimally.
<stub> If it is good enough to get past staging, can we tell if any performance issues on production are because of the new schema or because of existing issues?
<wgrant> Heh, have you tried a bug search on staging? :)
<stub> So we have a new feature we have no way of qaing ?
<wgrant> Most recent risky performance tweaks have been done by deploying them with feature flags, releasing to launchpad, then beta-testers, then everyone.
<wgrant> Because staging just isn't good enough.
<stub> I see a fairly complex trigger, so I'm wondering if we can avoid it and the potential for bugs it brings.
<wgrant> And for stuff like bug search, where there are thousands of different queries...
<wgrant> Heh, yeah. There is that issue.
<stub> Is the compatibility code more likely to cause troubles than the new work :)
<stub> Will we be hiding use of the new schema behind a feature flag, so most users continue to use the old schema?
 * stub can't even load bugs.staging.launchpad.net...
<wgrant> stub: Yes, it will all be behind feature flags.
<stub> So we must have the two models running in parallel then.
<wgrant> Well, we basically have to. Even if we want to just cut over, we'd have to have horrifying views or hybrid appserver code to cope after the change.
<wgrant> Because we don't have code changes at the same time.
<stub> Populating one schema asynchonously is sucky, so I think we are stuck with a trigger like you have
<wgrant> I figure doing it this way makes that much easier and safer.
<wgrant> And lets us back out.
<stub> yup
<wgrant> The trigger is sucky, yes.
* czajkowski changed the topic of #launchpad-dev to: https://launchpad.net/ | Help contact: czajkowski | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide: https://help.launchpad.net/ | Support: https://answers.launchpad.net/launchpad | For packaging help: join #ubuntu-packaging
<czajkowski> gah
* czajkowski changed the topic of #launchpad-dev to:  
<stub> Much better :)
<wgrant> stub: The idea is that we get the new schema running in parallel, adjust the bug search code to optionally use it, do lots of performance testing. Based on the results of that, probably implement a flattened bug+bugtask table (which I've already done some testing on, with very promising results). Once we're comfortably on the new schema, dump the old one and make the new one writable and we're in a happy place.
<czajkowski> stub: great start to my day eh
* stub changed the topic of #launchpad-dev to: <Insert Coffee to continue>
<stub> wgrant: Yes, this seems the sanest approach.
<wgrant> stub: Thanks, just wanting to get your veto before I do much more :)
<stub> czajkowski: I have your attempts in scrollback if you don't have the history :)
* elmo changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10^2
<czajkowski> stub: elmo thanks.
<wgrant> Aw
 * czajkowski goes and makes another cuppa tea 
<stub> wgrant: urgh... using EXECUTE to ensure the query gets planned, rather than using the static plan generated when the procedure was compiled :-(
<wgrant> stub: I know :(
<wgrant> I'm not sure what the actual prepared plan is.
<wgrant> But it takes 20ms rather than 150us
<stub> I just make elmo buy me more hardware.
<wgrant> :)
<wgrant> stub: The indices are partial on product IS NOT NULL and distribution IS NOT NULL, so I guess it can't use them unless it knows there aren't any nulls.
<stub> wgrant: Adding redundant 'product is not null' clauses might help there.
<wgrant> Indeed.
 * wgrant tests.
<wgrant> Indeed, that works.
<wgrant> Even a few hundred microseconds faster.
<wgrant> I guess it will avoid the present prod planning problems, too.
<wgrant> Ideally I'd use conditional triggers rather than filtering in plpgsql, but that's not available until 9.0 :(
<StevenK> Heh, wallyworld_ has landed his two step picker again
<wgrant> Yep
<StevenK> Shall we go for two rollbacks? :-)
<wallyworld_> i fixed it
<wgrant> Looked over it this morning, and there're no glaring issues this time.
<StevenK> s/this time/yet/ ? :-P
<wgrant> And a global potential XSS fix.
<wgrant> Can't argue with that :)
<StevenK> We might even deploy it tomorrow
<wallyworld_> wgrant: btw, the launchpadlib api doesn't work with the new services stuff :-(
<wgrant> wallyworld_: Does so. What doesn't?
<wallyworld_> wgrant: lp.load('/services') fails
<wgrant> There's no reason this shouldn't work, but it might take a bit of poking.
<wgrant> wallyworld_: What error?
<wallyworld_> invalid xml id #service_factory
<wallyworld_> because there's no collection exported
<wgrant> Sinister.
<wgrant> Do you have a branch?
<mrevell> Hello
<wallyworld_> wgrant: just writing a test or two, give me a few minutes
<wgrant> wallyworld_: Sure
<wallyworld_> mrevell: g'day
<wgrant> wallyworld_: We should be able to make lp.services.access_policies work as well.
<wallyworld_> wgrant: hope so
<wallyworld_> mrevell: only a few more days for you to put up with bigjools \o/
<mrevell> wallyworld_, We have a band waiting at the air port to see him off. That and big banners with a "Never darken these shores again".
<wallyworld_> hah
<wallyworld_> mrevell: but now *I* need to deal with him
<wallyworld_> no fair
<wgrant> stub: Want to rebuild bug_fti for hopefully the last time for a while? :)
<wgrant> stub: The new BugHeatUpdater took 1.5 iterations to do the entire backlog, so that should be it.
<wgrant> (currently at 67% bloat)
<wgrant> Hm
<wgrant> I wonder if I can make update-pkgcache massively faster with SPPH.spn
<wgrant> That would be nice.
<wgrant> StevenK: Those 307 errors look pretty easily fixable?
<wgrant> blink
<wgrant> There's no SPPH index on (archive, distroseries) :(
<StevenK> wgrant: I'm not sure, I haven't investigated. I can rSP a lot, but that doesn't fill me with happiness.
<wgrant> StevenK: In the factory it's fine.
<wgrant> The factory can commit any evil it wants, as long as the objects it returns are proxied.
<StevenK> Those 307 are with the factory fixed.
<wgrant> Ah
<wgrant> I'll be making the factory commit a loooot of evil in coming months.
<wgrant> But it should be very fast evil :)
<StevenK> wgrant: Oh, and that 307 is just -m bugs
<StevenK> I'm suspecting more fallout
<wgrant> Should be relatively minor elsewhere.
<wgrant> I actually expected more.
<wgrant> Given that you're fixing buggy security declarations from 2005.
<wallyworld_> wgrant: wip, https://code.launchpad.net/~wallyworld/launchpad/pillar-access-service-infrastructure/+merge/94338
<wallyworld_> wgrant: i used named utilities to get the services
<wallyworld_> so the servicefactory is very simple
<wallyworld_> to add a new service, just define it in the zcml
<wallyworld_> and that's it
<wgrant> wallyworld_: Ah
<wgrant> That's why it won't work in launchpadlib.
<wgrant> It won't be in the WADL
<wgrant> You should still be able to say lp.load('/services/accesspolicy'), but not lp.services.accesspolicy
<wallyworld_> the servicefactory  attribute
<wallyworld_> load doesn't work
<wallyworld_> lp.load('/services') neither
<wgrant> Hm
<wallyworld_> undefined xml id
 * wgrant tries.
<wgrant> Did you actually rebuild your WADL?
<wallyworld_> yes, several times
<wgrant> :(
<wallyworld_> i may have missed something
<wallyworld_> i'm happy how the webservice side of things is working
<wgrant> It's also possible that launchpadlib is being hilarious with WADL caching.
<wallyworld_> maybe
<wallyworld_> bbiab, gotta feed the dog
<wgrant> wallyworld_: In [5]: lp.load('/services/accesspolicy')
<wgrant> Out[5]: <access_policy_service at https://api.launchpad.dev/devel/services/accesspolicy>
<wgrant> wfm
<wallyworld_> wgrant: what, it worked for you?
<wallyworld_> one thing i had to do was define a url for the service
<wallyworld_> otherwise it would complain
<wgrant> Indeed, it will do that.
<wallyworld_> i guess i'll try make clean and do it again
<wallyworld_> but i've done that already
<wgrant> Just blow away lib/canonical/launchpad/apidoc and make build again
<wallyworld_> wgrant: can you invoke a method?
<wgrant> wallyworld_: No. There's a launchpadlib bug about that, IIRC>
<wgrant> Bug #681767
<_mup_> Bug #681767: Can't use relative URLs with launchpadlib.Launchpad - generates URI object that breaks wadllib <lp-foundations> <launchpadlib :Triaged> < https://launchpad.net/bugs/681767 >
<wgrant> Reasonable workaround.
<wgrant> In [14]: aps = lp.load(str(lp._root_uri) + '/services/accesspolicy')
<wgrant> In [15]: aps.getAccessPolicies()
<wgrant> Out[15]: u'[{"index": 0, "description": "Everyone can see this information.\\n",
<wallyworld_> wgrant: so, you can't go lp.load('/services/accesspolicy').get('xxxx') ?
<wgrant> etc
<wallyworld_> ah ok
<wallyworld_> well, i'm pleased it all works for you
<wallyworld_> now to try for me
<wgrant> wallyworld_: Heh
<wallyworld_> this is good
<wgrant> wallyworld_: So, you can probably define a generic browser:url for services.
<wallyworld_> i'm liking what we can do with this
<wgrant> Let me try.
<wallyworld_> i have
<wallyworld_> already
<wgrant> Ah
<wgrant> I thought you meant a specific one.
 * wgrant actually reads the code.
<wgrant> Ah, cool.
<wallyworld_> wgrant: all you need to do to add a service is add a bit of zcml for it
<wallyworld_> done
<wallyworld_> really easy :-)
<wgrant> Would be nice if we could get it into the WADL, but that would require being able to define custom adapters, or some runtime interface patching.
<wallyworld_> yeah, a future branch
<wgrant> Still, the current launchpadlib way is not too bad :)
<wallyworld_> yes, it works
<wallyworld_> not too fugly
<wallyworld_> wgrant: it fails for version='devel'
<wallyworld_> when i do version='beta' it works
<wgrant> wallyworld_: That's what I'm using.
<wgrant> lp = Launchpad.login_anonymously('fewfw', 'dev', version='devel')
<wgrant> Blow away your launchpadlib cache
<wallyworld_> In [1]: from launchpadlib.launchpad import Launchpad
<wallyworld_> In [2]: lp = Launchpad.login_with('testing', service_root='https://api.launchpad.dev', version='devel')
<wallyworld_> In [3]: ap = lp.load('/services/accesspolicy')
<wallyworld_> where's the cache?
<wgrant> ~/.launchpadlib/api.launchpad.dev/cache
<wallyworld_> ok, didn't even know that was there
<wgrant> Might as well dispose of ~/.launchpadlib/cache as well, if it's there
<wgrant> I think it's deprecated.
<wallyworld_> wgrant: yay, it was the supid cache al lalong :-(
 * wallyworld_ add a launchpadlib test to the branch
<wallyworld_> s/add/adds
<wgrant> Heh
<wallyworld_> i wasted sooo much time due to that :-(
<wgrant> That's what caches are for :)
<wallyworld_> but cahces need to be properly maintained :-/
<wgrant> Pssht, details.
<wallyworld_> sigh
 * wallyworld_ has more unity / compiz grief
<wgrant> Oh
<wgrant> This just gets better and better
<wallyworld_> wgrant:  i was offline briefly. what's "this" ?
<wgrant> wallyworld_: update-pkgcache
<wgrant> Unrelated :)
<wallyworld_> ok :-)
<wgrant> Just a 20h daily script that I want to fix.
<wallyworld_> 20h. there's still 4 to go :-)
<wgrant> It used to be 25h
<wgrant> Until I fixed it a bit.
<wgrant> But it still does about 1000x too many queries.
<wallyworld_> 25h is ok if we slow down the rotation of the earth
<wgrant> That's a good idea.
<wgrant> Oh look that query is now 400 times faster
<wgrant> sigh
 * wgrant aims to get it down to 5 minutes or less.
<wgrant> Ah, it's back up to 23 hours :(
* rick_h_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h* | Firefighting: - | Critical bugtasks: 4*10^2
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett, rick_h* | Firefighting: - | Critical bugtasks: 4*10^2
<abentley> rick_h_: Could you please review https://code.launchpad.net/~abentley/launchpad/bulk-insert/+merge/94271 ?
<rick_h_> abentley: sure thing
<abentley> adeuring: I'm ready to join in on the Job system revamp.  Can you bring me up to speed?
<adeuring> abentley: i only created a basic branch:
 * adeuring os tryingto fins it again...
<adeuring> abentley: lp:lazr.jobrunner
<adeuring> abentley: mumble
<adeuring> abentley: sorry,mylaptop crashed
<abentley> adeuring: no worries.
<czajkowski> https://bugs.launchpad.net/launchpad/+bug/936989  could someone point me in the right direction of where I'd find the answer to this ?
<_mup_> Bug #936989: Launchpad doesn't properly redirect if ubuntu-bug user is allready logged <apport-bug> <i386> <oneiric> <Launchpad itself:New> < https://launchpad.net/bugs/936989 >
<czajkowski> please
<sinzui> czajkowski, I do not know what to say. I can confirm that Lp did redirect me 15 minutes ago
<czajkowski> sinzui: likewise, but i do know from time to time i've had glitches also
<czajkowski> :/
<czajkowski> not today when I try and replicate it though
<czajkowski> sinzui: good morning btw :)
<sinzui> is it? I do not think I have ever had a good morning. I am happy to have survived the night though
<sinzui> czajkowski, I wonder if the user has multiple profiles. The console and launcher are selecting different profiles
<flacoste> adeuring, deryck: are we done with bug #
<flacoste> exec'ing bin/maas in the run script would be enough to satisfy
<flacoste> supervise.
<flacoste> raarh
<flacoste> paste, i hate you
<deryck> heh
<czajkowski> sinzui: I'm not a great sleeper and can function on about 4hrs sleep pretty well.
<deryck> adeuring, did you chat with bryce yesterday about it?
<flacoste> adeuring, deryck: i meant to say: are we done with bug #829074? any news from the stakeholders?
<_mup_> Bug #829074: Show bugs that are not known to affect "official" upstream <bugs> <escalated> <qa-ok> <Launchpad itself:Fix Released by adeuring> < https://launchpad.net/bugs/829074 >
<adeuring> deryck,flno, i did not yet talk with bryce
<deryck> adeuring, please try to get that done today so flacoste can close out the request completely.
<adeuring> ok
<sinzui> czajkowski, are you gloating or challenging me. On my 25th birthday my metabolisms slipped from 5th gear to 3rd. I need 6 hours of sleep or more.
<flacoste> thanks adeuring
<czajkowski> sinzui: hey I'd like more don't get me wrong. I just seem to have missed the lesson on how to sleep through a night!
<deryck> video games and melatonin
<deryck> czajkowski, lesson learned ^^ :)
<czajkowski> deryck: I do my loco work at night that helps, next week I get the go ahead to go back to the gym, which should kill me and help me sleep! :)
<sinzui> deryck, Does that combat children hoping into bed? What about fighting cats? I wake up with scars, maybe I referred a cat fight in my sleep
<deryck> see cats, that's your problem right there.
<deryck> and I've grown immune to kids toes in my ribs somehow. :)
<czajkowski>  lol
<lifeless> flacoste: did you have an opinion on the oops-on-local-404 thing ?
<adeuring> abentley: new attempt to mumble?
<abentley> adeuring: sure.
<abentley> adeuring: http://pastebin.ubuntu.com/854075/
<flacoste> lifeless: whether to show OOPS id on those?
<lifeless> flacoste: and whether to even make an OOPS
<lifeless> flacoste: because of linkification, its not uncommon to have bad links in-site that we don't control
<flacoste> lifeless: right
<flacoste> we are not doing much with those OOPS right now
<lifeless> I've been torn for a while, I think its not really a win ATM
<flacoste> so dropping them wouldn't have any impact
<lifeless> can always turn it on again later
<flacoste> exactly
<lifeless> ok, cool
<rick_h_> abentley: did jelmer review the other one for you or was that just a drive by?
<abentley> rick_h_: OTP
<czajkowski> sinzui: in the LP licience reviews, I've 2 maintained by us, both look a bit dubious , later on when you can would
<czajkowski> you please
<sinzui> tuxubuntu can be deactivated
 * sinzui hopes weiqi is real
<czajkowski> nods
<sinzui> czajkowski, I think there is a problem with gnome-go. It is real, we should own it.
<sinzui> czajkowski, The code import is odd, not wrong, just odd
<czajkowski> sinzui: I dont feel so bad now for asking, I managed to do the others this morning/so far but those two just looked odd to me
<sinzui> I happen to know the registrant is the baltix maintainer, and his branch in owned by baltix. I assume that branch is his distro version. The branch is also the upstream import.
<sinzui> czajkowski, I see the branch has a recipe and looking at the source, it has a debian dir: http://bazaar.launchpad.net/~baltix/gnomego/trunk/files/head:/src/
<sinzui> czajkowski, We want to set this branch as the focus of development for gnomego. I think baltic will want to create a separate branch in time and maybe an extra recipe to make upstream integrate into baltix
<sinzui> czajkowski, visit https://code.launchpad.net/gnomego
<czajkowski> ok
<sinzui> czajkowski,  copy the branch URL (lp:~baltix/gnomego/trunk) then follow the link to configure code hosting
<sinzui> Paste the link to the branch field and Update to be done
<czajkowski> lets hope I dont break things!
<sinzui> czajkowski, I think the second option in that form is fundementally broken (Create a new, empty branch in Launchpad and link to this series). Creating an empty branch does not really help anyone. We should remove the feature to register an empty branch
<czajkowski> sinzui: I do the change code hosting, but owner, It picks the teams I'm on, so I select the registery ?
<sinzui> It want's to change the owner? That is odd
<sinzui> I would pick ~registry, but I do not think we should be changing the owner of the branch unless we are asked to
<sinzui> That would break recipes
<sinzui> czajkowski, maybe I am doing this the hard wary. See the " set it now" link on https://code.launchpad.net/gnomego
<sinzui> czajkowski, This simpler for want us to paste ~baltix/gnomego/trunk into the field an Update
<czajkowski> sinzui: ahh thats a lot easier
<czajkowski> thanks sinzui
<czajkowski> can I have a brain dump of sinzui and lifeless please on launchpad topics!
<abentley> rick_h_: I thought it was a review, but it looks like technically, he didn't vote.  Anyhow, it didn't get a LOC waiver from lifeless, so it won't be going in.
<sinzui> I think my information is ordered by iChing, broken into topics about food. I do not think many people can make sense of it.
<czajkowski> sinzui: oh food, food I cna understand!
<rick_h_> jcsackett: if you get a sec can you peek at https://code.launchpad.net/~rharding/launchpad/watch_jsbuild/+merge/94379 ?
<rick_h_> jcsackett: note that it's dep on a branch that's in ec2 land atm
<jcsackett> rick_h_: sure. talking with danhg right now about some text changes. after that i can take a look at yours.
<rick_h_> np, ty much
<abentley> adeuring: As of r7, you don't need virtualenv, "make develop" will set you up and "make check" or bin/python setup test will run the test suite.
<adeuring> abentley: cool
<jcsackett> rick_h_: i see pyinotify added to the versions; you're going to update sourcedeps with that before landing, i assume?
<rick_h_> jcsackett: yes, should be there alreday
<rick_h_> already*
<jcsackett> hrm.
<jcsackett> perhaps i didn't update today.
<rick_h_> just did it before the MP
<rick_h_> but I did update and make sure it came down
<jcsackett> rick_h_: dig.
<jcsackett> rick_h_: I'm unfamiliar with pyinotify; i see you have a method for IN_CREATE and IN_MODIFY but they both use event.mask == pyinotify.IN_MODIFY; is that correct?
<rick_h_> jcsackett: ah, let me update that sorry.
<jcsackett> rick_h_: no worries. lemme know when the fix goes up.
<rick_h_> jcsackett: yea, will be a few. I actually had bigger plans there I got sidetracked and forgot about oops
<czajkowski> nn folks
<lifeless> morning for reals
<rick_h_> howdy lifeless
<lifeless> hey rick_h_
<james_w> does anyone know where the session db sql sanitising code that LP uses lives?
<lifeless> yes
<james_w> ah, got it
<lifeless> and now you do too :)
<james_w> unfortunately it's not all that helpful
<lifeless> its very simple
<james_w> because I can't assume a separate db
<lifeless> I'm sure django will have different needs
<lifeless> can you assume a particular substring match
<james_w> I'm not sure what a good substring would be
<lifeless> well, whats your session table ?
<james_w> we can assume the session table name
<james_w> probably
<james_w> 'django_session'
<lifeless> seems unlikely to mask legitimate traffic, and if it does unlikely to be harmful
<james_w> with 'session key' and 'session data'
<james_w> just if "django_session" in query: query = "<session query>" ?
<lifeless> thats what I'd do
<lifeless> I mean, you can be more fancy, but will it make a real difference to you?
<james_w> probably not
<lifeless> if someone wants to attack the system, *and* cause oopses, *and* hide a query, they can put django_session in as user data
<lifeless> but that wouldn't mask every query I expect
<lifeless> you could look at what django compiles for the query and do a tighter match - e.g. 'FROM django_session'
<james_w> the other one would be user passwords
<james_w> which could be more annoying to have redacted
<lifeless> you're not using openid ?
<james_w> yes
<james_w> but it doesn't mean that it should be ignored in a librar
<lifeless> so, same principle will apply
<lifeless> possibly admins will have local accounts for you
<rick_h_> jcsackett: ok, updated https://code.launchpad.net/~rharding/launchpad/watch_jsbuild/+merge/94379
<lifeless> rick_h_: did you consider putting your watch thing as a separate tool? It seems totally unrelated to LP itself
<rick_h_> jcsackett: I meant to go back and implement the meta.js generation when you add a new file
<rick_h_> lifeless: yes, except that the "knowledge" of how things need to get built isn't easy to pull out some how that I can think of
<rick_h_> since it needs to know to regex the filename, copy over the bug/code/etc app part, and map it over to the build/js/lp/$app/xxx
<rick_h_> lifeless: but yea, I'm hoping this and hte lpjsmin can enable the other projects to keep common good practice JS/combo load build patterns
<lifeless> that logic seems needed for the main build part too; so presumably there is a mapper function we call, which it could use.
<rick_h_> lifeless: yea, except currently that 'logic' is in a combo-rootdir shell script
<lifeless> so, this duplicates it into python ?
<rick_h_> a little bit, this requires convoy package to be avail to run and such
<rick_h_> this ONLY takes care of the combo loader, it doesn't touch launchpad.js
 * lifeless thinks you're building up a LOC debt rapidly :)
<rick_h_> that's too slow since it has to be completely rebuilt
<rick_h_> well, but the idea was that I ripped out the jsmin code, decrease maint by making JS dev faster
<lifeless> I like the idea
<rick_h_> so yes, this adds a new script file that's + LOC
<rick_h_> but with the jsmin branch I've - LOC as well
<lifeless> I suggest having a mapper function, putting the watch tool somewhere separate (e.g. convoy perhaps? they might benefit from it) and having it be trivially callable with our custom mapper
<lifeless> I worry - a lot - when we add dev tools to the LP tree
<lifeless> they are typically entrenched and very hard to reuse when we do that.
<rick_h_> lifeless: understand, and thought of the callable. Do you have a locationsuggestion for that? I didn't see one in my initial looking
<lifeless> (which existing entrenchment you're running into)
<lifeless> rick_h_: lp.services.js.something ?
<rick_h_> lifeless: yea, I think we're in agreement in principle for sure, just hard time implementing
<rick_h_> would a new service dir with just an __init__.py exposing the callable be ok then?
<lifeless> fine by me
<rick_h_> and about the variable defs? the paths? just move those into that callable then? They're a bit buried
<lifeless> where are they defined at the moment, and are the alterable ?
<rick_h_> I guess I can start there, no worse off.
<lifeless> I nearly said variable but then the world exploded
<rick_h_> right now they're in the top of the script
<rick_h_> so we'd still need something Make can call, but I guess that can be python -m lib.lp.services.js.run_jswatch or something
<rick_h_> right now those build paths and such are part of all the build/makefile/buidlout process
<rick_h_> so it just feels strange to move them down into tree
<lifeless> rick_h_: a buildout template can be just: #!\nfrom lib.lp.services.js import path_map\nfrom convoy.buildwatch import watcher\ndef __main__():watcher(path_map)\n
<rick_h_> lifeless: right, yea. Let me know it out and I'll see if things fall into place.
<lifeless> make can then just call bin/jsbuildwatch or whatever the name is again
<rick_h_> so I'm looking to add a jsbuild_watch python module that's external/reusable requiring a callback to be sent to it right?
<rick_h_> lifeless: right, cool
<lifeless> yeah
<lifeless> something which you can write a tiny bit of code to use in e.g. maas, or u1, or ...
<rick_h_> lifeless: right, that might make sense to take back to convoy. I'll check it out.
<rick_h_> jcsackett: ok, so nvm on the review :) thx
<jcsackett> rick_h_: fair. :-p
<james_w> https://code.launchpad.net/~james-w/python-timeline-django/obscure-session-info/+merge/94439
<rick_h_> james_w: I don't think we can review that one for you
<james_w> rick_h_, that's ok thanks
<james_w> rick_h_, I was just noticing it here because I was discussing it a little earlier in here
<james_w> someone else on my team will do the actual review
<james_w> sorry for not being clear
<rick_h_> james_w: ah ok cool. Wanted to try to get my OCR hours in but no buttons there
<james_w> :-)
<rick_h_> lifeless: re last-mod header, what header would you send the content hash back with?
<rick_h_> lifeless: I was noticing that I was geting 304 not-modified responses from some static files, but nothing out of the combo loader so got down this path of why/how to get the combo loader files treated the same as static files
<rick_h_> lifeless: nvm, so you mean set the hash as the etag vs last-mod header
<lifeless> ayup
<lifeless> of course this means you have to honour INM as well - or just set good cache headers and put a squid in front
<lifeless> rick_h_: actually, thats better - for *us* we never edit a file, so we want to just set cache-control: 100000 or something
<lifeless> and expires: now+1 year
<rick_h_> lifeless: yea, so I ran into this on an outside app I'm using convoy for
<lifeless> (more or less, I can dig up the canonical make-it-never-expire reference fo you layer)
<rick_h_> the trouble I was getting is that while I set expires headers way out, it would still request because it didn't have a last-mod date in the resp
<lifeless> rick_h_: ah cool. So, LMT can be done, but IMO etag is safer
<lifeless> up to you
<rick_h_> I didn't try an etag header though. Trouble with that is that convoy streams the response, so getting the etag header is going to be a pita
<lifeless> technically you can set it at the end, but I know of -no- web stacks that handle trailers correctly
<rick_h_> lifeless: yea, thanks for the heads up. I'll have to look into it some more.
<lifeless> no worries
<rick_h_> lifeless: do you have 5 to do Q&A on that though? I'm wrapping my head around part of it and think it might be faster to voice vs type?
<lifeless> sure, gimme a few and I'll ping you
<rick_h_> ty
<lifeless> rick_h_: ok
<rick_h_> lifeless: tech of choice?
<lifeless> one that works? skype? hangouts? you know, anything proprietary
<rick_h_> hah, gotcha
<abentley> gary_poster: adeuring and I are working on a new lazr package: lp:lazr.jobrunner.  I am having trouble getting it working in developer mode: http://pastebin.ubuntu.com/854512/  Do you have any advice?
<gary_poster> abentley, on call, will look soon
<abentley> gary_poster: thanks.
<lifeless> abentley: is it lazr.runjobs or lazr.jobrunner ?
<abentley> lifeless: Doh!
<lifeless> abentley: I don't know if you and adeuring have seen https://dev.launchpad.net/CreatingNewProjects, but https://launchpad.net/lazr.jobrunner suggests you haven't
<lifeless> abentley: could you run through that checklist and update the project and metadata and so forth? thanks!
<abentley> lifeless: sure thing.
<lifeless> abentley: (I assume buildout worked for you now ?)
<abentley> lifeless: Yes, thanks.
<lifeless> cool
<lifeless> rick_h_: btw have you had a chance to play with pybars?
<gary_poster> abentley, heh, I had just reviewed and was going to ask the same name question.  Glad it's working.
<abentley> lifeless: btw, going with handlebars/pybars makes a lot of sense to me.
<abentley> lifeless: esp. given that handlebars is an expected YUI 3 technology.
<lifeless> abentley: thanks
<lifeless> abentley: http://blog.fastmail.fm/2012/02/20/building-the-new-ajax-mail-ui-part-2-better-than-templates-building-highly-dynamic-web-pages/ - seems to me that handlebars could compile down to DOM operations just as well
<lifeless> which would be nice
<sinzui> wallyworld_, We should update this page with your recent experience https://dev.launchpad.net/API/ImplementingAPIs
<lifeless> sinzui: wallyworld_: when is your standup?
<StevenK> Now
<sinzui> right now
<lifeless> I would have joined yesterday but it was a trauma
<StevenK> You're welcome to join
<sinzui> we are in mumble in purple
<abentley> lifeless: It could, and that could be an advantage for big loops.  Handling unescaped might be a bit of a challenge, though.
<sinzui> wallyworld_,  here are the cluster of bugs. I think they overlap and maybe some are duplicates, Bug #590705, Bug #615244, Bug #723753, Bug #723959
<_mup_> Bug #590705: New Build api exposed via 'beta' rather than 'devel' <api> <lp-soyuz> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/590705 >
<_mup_> Bug #615244: syntax for declaring *when* an API is exported <lp-foundations> <Launchpad itself:Triaged> <lazr.restful:Triaged> < https://launchpad.net/bugs/615244 >
<_mup_> Bug #723753: defaulting to exporting new web service methods on all web service versions makes web service mistakes dangerous <lazr.restful:Triaged> < https://launchpad.net/bugs/723753 >
<_mup_> Bug #723959: Separate the floating "latest" version from the latest version-in-progress <lazr.restful:Incomplete> < https://launchpad.net/bugs/723959 >
<wallyworld_> sinzui: thanks
<wallyworld_> hmmm. none of those appear to exactly describe the issue.
 * wallyworld_ considers filing a new bug
<wallyworld_> bug 939910
<_mup_> Bug #939910: Need to export entry in version "beta" but it's only needed for "devel"  <lazr.restful:Triaged> < https://launchpad.net/bugs/939910 >
#launchpad-dev 2012-02-24
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer:wallyworld | Firefighting: - | Critical bugtasks: 4*10^2
<wgrant> Mmmm, readable loggerhead
<wgrant> A nice change.
<StevenK> I did wonder if a NDT would update loggerhead
<wgrant> Still needs to be degnomed a bit more
<wgrant> But much better
<poolie> is it just me or does the 'add an attachment' link on bugs now lose the text you already entered?
<lifeless> wallyworld_: yes
<wallyworld_> lifeless: cool. cause i was thinking we would have a standard set of helpers to do the equivalent of out standard tal formatters for person etc
<lifeless> you'd probably use partials for that
 * wallyworld_ hasn't read the doco yet
<lifeless> helpers should be used when you need host language logic
<lifeless> partials should be used when you want to reuse a template elsewhere
<wallyworld_> ok
<lifeless> {{> person assignee}
<lifeless> for instance
<wallyworld_> i am thinking about the +sharing view
<lifeless> bah, }} at the end, but you get the idea
<wallyworld_> for now, i might just hard code the formatting
<lifeless> {{> <partialname> <contextpath>}}
<wallyworld_> lifeless: is it on anyone's todo list to do the handlebars bundling?
<lifeless> so, handlebars is in yui3.5
<lifeless> I imagine rick or deryck are likely to step up and do that
<StevenK> Blocked on combo loader
<wallyworld_> ah, cool. but i guess we need something to use till then, or we can wait
<lifeless> indeed
<lifeless> you can use mustache now (and pystache || pybars on the server)
<wallyworld_> it's just prototyping atm so will do
<lifeless> pystache has some wtf's apparently :) - pybars is in the download cache, if you wanted to play with it and tell me what you think
<wallyworld_> lifeless: i was looking to do the rendering on the client
<lifeless> wallyworld_: oh sure
<wallyworld_> lifeless: still, server side vs client side rendering is something that needs some thought etc
<wallyworld_> one argument is that all rendering should be done client side
<wallyworld_> after getting json data from the server
<StevenK> wgrant: Is qas's database back to FK goodness?
<wallyworld_> s/from the server/using the api
<lifeless> wallyworld_: you have been following the thread on that right ?
<wgrant> StevenK: No
<lifeless> wallyworld_: there is a proposed standard for us
<wgrant> StevenK: Hopefully tonight.
<wallyworld_> lifeless: i was about to re-read it, i think it was advocating getting as much stuff from the api as possible
<lifeless> if you use innerHTML with something containing a script section, does that script section run ?
<wallyworld_> lifeless: don't think so, not in a consistent and portable way
<wallyworld_> lifeless: if you need it i would claim the design is broken
<lifeless> wallyworld_: its the reverse, I'm assessing some security risks
<wallyworld_> in our stuff?
<lifeless> it would terrify me if it worked ever
<wallyworld_> it can be made to work
<wallyworld_> for not in a portable way afaik
<lifeless> if it ever works, thats sufficiently terrifying
<wallyworld_> lifeless: one example, possibly old, for ie, you can mark a script tag with DEFER and it will execute when set vua innerhtml
<wallyworld_> not sure if that's still the case, ymmv etc
<wallyworld_> what i'm saying is people have found ways to do it i think
<lifeless> sure
<lifeless> -> it means folk /wanting/ it to work can probably do so, but folk can't (and shouldn't rely on it just working
<wallyworld_> yes
<wallyworld_> and we should actively discourage/forbid it :-)
<wgrant> lifeless: How is that terrifying?
<lifeless> it also means that folk which want to us js to customise delivered html to another site, can't reliably do so (and those sites would be crazy to accept it)
<lifeless> wgrant: its off-site js all over again
<wgrant> ...., yes that's why you don't use innerHTML
<wgrant> <script> blocks not running is not a security feature.
<wgrant> It's just because of how things work
<wgrant> You can still use onclick/onhover etc to execute code
<wgrant> innerHTML is in no way safe to use with untrusted data.
<lifeless> I know
<lifeless> I wasn't thinking it was, I was seeking confirmation that it wasn't
 * wallyworld_ wishes we didn't use innerHTML so often in our code
<lifeless> so onlick/onhover etc events *will* run ? With frame and frameset being gone in html5...
<wgrant> If you're using innerHTML in your code, you are doing it wrong.
<wgrant> lifeless: Yes
<wallyworld_> wgrant:  s/are/were
<wgrant> wallyworld_: Hm?
<lifeless> wallyworld_: tense mismatch there
<wallyworld_> we don't do it anymore i don't think, what's there is mostly lazrjs stuff
<wgrant> wallyworld_: That code is still doing it wrong.
<wallyworld_> yes, not disagreeing
<wgrant> lifeless: HTML5 still has iframe, doesn't it?
<wallyworld_> just saying it's legacy and we don't do it for new code
<lifeless> ah indeed, missed that
<wgrant> And nobody real has used a frameset for a long time.
 * wgrant has never understood why HTML5 doesn't make XHTML mandatory :(
<lifeless> because then they would have N+1 problems :P
<lifeless> anyhoo
<lifeless> iframe execution context is that of the originating server IIRC, so thats somewhat sane
<lifeless> bbiab
<wgrant> lifeless: Yes
<wgrant> No non-infrastructure code is permitted to use innerHTML.
<wgrant> And infrastructure code that uses it will be looked at with great scrutiny.
<wgrant> And anybody who uses innerHTML will be slapped, hard :)
<wgrant> lifeless: HTML5 still permits crap like unquoted attributes, implicit empty elements, etc.
<wgrant> That helps *noone*.
<StevenK> Bah. +36/-36
<StevenK> I think I've fixed all the failures in bugs at least
<lifeless> wgrant: so, what does mustache use ;)
<lifeless>  https://github.com/amoffat/pbs#readme
<lifeless> anyone here played more than trivially with juju ?
<StevenK> wgrant: Is there a bug for this IBug leaking?
<StevenK> wgrant: And can I have that API URL you used yesterday so I can see what .dev against this branch gives.
<wgrant> StevenK: Just look at the duplicate_of of a dupe of a private bug.
<wgrant> lifeless: Does mustache use anything?
<wgrant> lifeless: It generates HTML
<wgrant> It doesn't inject it AFAIK
<StevenK> wgrant: I was talking about the API URL you gave me to show me just how much IBug leaks.
<wgrant> 15:53:30 < wgrant> Grab https://bugs.qastaging.launchpad.net/api/devel/bugs/718213/duplicate_of?ws.accept=application/json as anonymous
<_mup_> Bug #718213: Can't access due to content. <Internet Archive - Tech Support:New> < https://launchpad.net/bugs/718213 >
<StevenK> Right, so to test on .dev I need to make a private bug, and have a public one as a duplicate of it, and then massage that URL against the public bug?
<wgrant> Yep
<wgrant> Ensuring that you can't see the private bug.
<StevenK> Right
<StevenK> Waiting to see if I've fixed all the -m bugs failures first
<lifeless> wgrant: so, if someone *uses* mustache, how do they expose their html?
<wgrant> lifeless: Probably using their framework's methods that are similar to innerHTML.
<wgrant> Which will internally use innerHTML.
<wgrant> And are pretty much as unsafe.
<lifeless> tada
<wgrant> But this would normally be encapsulated in your infrastructure.
<wgrant> You're unlikely to be rendering templates directly in your day-to-day JS.
<wgrant> https://code.launchpad.net/~wgrant/launchpad/bulk-insert-errywhere/+merge/94490
<lifeless> gary_poster: https://github.com/amoffat/pbs#readme might be interesting for setuplxc
<wgrant> lifeless: That looks like a really good way to be vulnerable.
<gary_poster> lifeless, looks fun.  More magical than I'm interested in atm, but maybe my tastes will change. :-)
<lifeless> :)
<lifeless> elmo pointed me at it when looking at the setuplxc ticket
<gary_poster> heh
<huwshimi> wallyworld_: Hi, on lp.net the "affects me" selection dialogue looks like it might have been affected by your changes to add the descriptions to bug statuses...
<huwshimi> (I'm not sure if this is some kind of feature flag fallout)
<wallyworld_> huwshimi: the empty span?
<huwshimi> wallyworld_: Yea, there's an extra empty row after the title
<wallyworld_> could be a side effect of choicesource widget changes
<wallyworld_> i'll file a bug etc
<huwshimi> wallyworld_: Thanks :)
<wallyworld_> np, thanks for letting me know
<huwshimi> wallyworld_: I'm a little preoccupied otherwise I would have filed a bug myself :)
<wallyworld_> np
<StevenK> Hmmm, it still returns 37 things
<StevenK> I was expecting less
<wgrant> StevenK: The API? It replaces forbidden things with the redacted tag
<wgrant> Rather than omitting them.
<StevenK> wgrant: Ah, so it's going to continue to return 37 things, but a lot more will be redacted?
<wgrant> Yes
<StevenK> Bleh, okay.
 * StevenK pushes up this branch
<wallyworld> wgrant: what were your thoughts on bulk insert returning entire instantiated objects just to get the ids?
<wgrant> wallyworld: Fixed.
<wallyworld> ok, cool. thanks
<wgrant> wallyworld: I considered doing that initially, but decided it wasn't worth it. But you convinced me.
<wgrant> There's now get_objects and get_primary_keys args
<wgrant> mutually exclusive.
<wallyworld> np. i just saw the mp->approved and didn't realise you had changed to code
<wgrant> Heh
<StevenK> Is there a bug for IBug being leaky?
<wgrant> Don't think so.
<StevenK> wgrant: prod-revnos is AssertionErroring
<wgrant> I blame acamar
<StevenK> Haha
<StevenK> I thought you might
 * wgrant waits for openid...
<wgrant> 'tis an awful, awful script.
<wgrant> ssh: connect to host acamar port 22: Connection refused
<wgrant> But it's meant to handle that.
<StevenK> Refused is a bit harsh
<wgrant> Ah
<wgrant> It's because it was crashing the script.
<wgrant> The deploymgr revno check thing doesn't log at the start of its run, only at the end.
<wgrant> So I split the log based on the final line, which isn't there
<wgrant> Should work next hour.
<StevenK> Bug 940044
<_mup_> Bug #940044: IBug is leaky as a rusty sieve <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/940044 >
<StevenK> wallyworld: O hai, Mr OCR.
<wallyworld> yeeees?
<wallyworld> has review for me?
<StevenK> Lurch: https://code.launchpad.net/~stevenk/launchpad/less-bug-leakage/+merge/94495
<StevenK> Pity wgrant won't get that joke
<wallyworld> you rang?
<StevenK> ;-)
<wgrant> Ha ha
<wallyworld> StevenK: lines 22,23 - why not use rSP there also?
<wallyworld> also 84,85
<wgrant> rSP 4 eva
<StevenK> Because I didn't want to overuse it
<wallyworld> in tests, to get data needed to run the test, i think it's ok
<wgrant> Agreed, in those contexts it makes sense.
<wgrant> And it's much faster.
<wallyworld> and more consistent with 1. the changes in this mp and 2. other usages in factory
<wgrant> I'm pleasantly surprised there's so little fallout.
<StevenK> wallyworld: Okay, I've made that change locally
<wgrant> It won't be quite so easy when it comes to private projects in a couple of months :(
<StevenK> wgrant: I'm pleasantly surprised you're happy with my changes. :-P
<wgrant> I am usually happy with good changes :P
<wallyworld> StevenK: r=meeeeeeee
<StevenK> wallyworld: Pushing up the rSP change
<StevenK> I'm happy the branch ends up as +37/-42
<lifeless> \o/
<StevenK> loltpg
<lifeless> oh?
<wgrant> More likely to be Unity.
<StevenK> True, but I can hope.
<StevenK> I think I may have to switch to making fun of dodo users, rather than TPG users.
<wgrant> webservice tests are slow :(
<StevenK> wgrant: Can haz pointer to IBug traversal?
<wgrant> BugTargetTraversalMixin for one.
<StevenK> There's more than one? I am disappoint.
<wgrant> Yeah, that's +bug
<wgrant>  /bugs is somewhere else, probably MaloneApplicationNavigation
<wgrant> Now, the question we must all ask ourselves eventually.
<wgrant> Will Unity crash before I open my third terminal this time...
<wgrant> Apparently not.
<StevenK> Haha
 * StevenK tries to figure out where BugTargetTraversalMixin is tested
<StevenK> lib/lp/bugs/browser/tests/test_bugtask_navigation.py lies, that is testing MaloneApplicationNavigation
<wallyworld> StevenK: it was unity / compiz. been quite bad last couple of days :-(
 * wallyworld does school run
 * StevenK blinks
<StevenK> AssertionError: Name "+bug/16" is not registered as a view or navigation step for "Product" on "bugs".
<wallyworld> wgrant: my new services branch which merged earlier today will conflict with your mp
<wallyworld> you will want to merge trunk if you haven't already done so
<wallyworld> and rename the InformationVisibility enum in the services test
<wgrant> wallyworld: Ah, I think I was one rev behind that.
 * wgrant fixes.
<wallyworld> wgrant: why are there bulk insert changes for eg BinaryPackagePublishingHistory in the mp?
<wallyworld> not related to the core work in the mp
<wgrant> wallyworld: Ah, I guess I forgot to set the prereq.
 * wgrant fixes.
<wallyworld> thanks :-)
<wgrant> https://code.launchpad.net/~wgrant/launchpad/multipolicy-3/+merge/94501 will hopefully be better
 * wallyworld looks
<wallyworld> ah 600 lines smaller :-)
<wgrant> InformationVisibilityPolicy replaced.
<wallyworld> wgrant: the IAccessXXX interfaces and attributes are very light on doc strings
<wgrant> Indeed. You probably want some.
<wallyworld> yes please, not just for me necessarily
<wallyworld> since this is new for everyone outside purple, feel free to be verbose :-)
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10^2
<adeuring> good morning
<mrevell> Hi
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring | Firefighting: - | Critical bugtasks: 4*10^2
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring,bac | Firefighting: - | Critical bugtasks: 4*10^2
<bac> good morning abel.  much going on today?
<czajkowski> for launchpad mailing lists, is there a way to see who is a moderator on a ml or is it just the team owner who can see it ?
<wgrant> czajkowski: The team admins are the moderators.
<czajkowski> wgrant: thats what I thought
<czajkowski> thanks
<adeuring> morning bac, quiet day so far.
<bac> adeuring: cool.  i think i'll tackle william's MP shortly
<bac> adeuring: heads up -- i'll be out for the next three fridays.
<adeuring> bac: ok, thanks for the warning ;)
<deryck> Morning, all.
<czajkowski> deryck: hello how is the wife is she feeling better?
<deryck> czajkowski, yes, much better
<abentley> adeuring: Good morning.
<adeuring> morning abentley
<abentley> adeuring: How's it going?
<adeuring> abentley: fine, though I haven't yet done much on the card you created yesterday
<abentley> adeuring: Cool.
<deryck> adeuring, /extras/talk.google.com/orange-standup
<deryck> adeuring, sorry, https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<jtv> flacoste: allenap, rvba & I were discussing the naming scheme for Maas API versions.  What would you prefer â "v1", or "1.0", or â¦?
<flacoste> jtv: blue
<flacoste> :-)
<jtv> frankban: Come on, help us a bit here, this is our copout plan.  :-)
<jtv> Ahem.  I meant flacoste.
<flacoste> you are not the first to make the association :-)
<flacoste> then use 23
<flacoste> and power of 23 from then on
<jtv> flacoste: just thinking because of the long-term implications, you might have some grand standard scheme.
<jtv> Uniformity and all that.
<flacoste> 1.0 is fine
<jtv> Since we're a fun company of individuals.
<jtv> Thanks!
<flacoste> similar to what we do in lp
<rvba> /api/v1.0/ then?
<jtv> See, that wasn't so hard.  Ask the Boss Engineering works.  :)
<rvba> or /v1.0/api ?
<jtv> rvba: definitely /api/ first.
<jtv> Or we'll be inviting an unholy mess of paths.
<rvba> Not what twitter does fwiw. Nor http://musicsearch.ubuntu.com/.
<rvba> Ok, twitter uses api.twitter.com ;)
<jtv> That makes all the difference.  We on the other hand have a bunch of path trees on one hostname.
<rvba> True,  /api/1.0/ it is then.
<jtv> \o/
<salgado> bac, adeuring, I've just added a small one for review :)
<adeuring> salgado: I'll look
<salgado> thanks adeuring!
<abentley> adeuring: can we chat about the jobs stuff?
<adeuring> abentley: give me 10 minutes or so, I'm just finishing a review
<abentley> adeuring: cool.
<adeuring> salgado-lunch: r=me, some minor nitpicks
<adeuring> abentley: mumble?
<abentley> adeuring: sure.
<sinzui> bac: adeuring: Do either of you have time to review https://code.launchpad.net/~sinzui/launchpad/error-pages/+merge/94574 <-there are a lot of find and replace changes in it
<bac> sinzui: i can
<salgado> adeuring, thanks for the review; I've done the changes you suggested and it'd be great if you could land it for me
<adeuring> salgado: sure, I'll land it
<salgado> adeuring, oh, should I create a bug and link to that so that we can track its qa-untestability or can we just tag the commit as qa-untestable?
<adeuring> salgado: right, good idea
<salgado> oh, hmm. I can no longer assign bugs to arbitrary people?
<abentley> deryck: I'm trying to follow https://dev.launchpad.net/EC2Test but the instructions for getting the access credentials don't look right.  Should I be doing something with "Key Pairs"?
<deryck> let me look....
<abentley> deryck: I think I found it.  Got confused because there was no "Account" link.
<deryck> abentley, yeah, so I do have my credentials in ~/.ec2/aws_id
<deryck> abentley, but not sure if the how to about getting those is still right. it's been awhile since I did it.
<flacoste> bac: didn't you report a bug similar to bug 939910 in the past?
<_mup_> Bug #939910: Need to export entry in version "beta" but it's only needed for "devel"  <lazr.restful:Triaged> < https://launchpad.net/bugs/939910 >
<bac> flacoste: yes, i believe i did
<bac> flacoste: but i don't see it
<flacoste> ah i know!
<flacoste> bug=760849
<flacoste> from IProcessorFamily!
<flacoste> deryck: any news on bug 829074?
<_mup_> Bug #829074: Show bugs that are not known to affect "official" upstream <bugs> <escalated> <qa-ok> <Launchpad itself:Fix Released by adeuring> < https://launchpad.net/bugs/829074 >
<deryck> gah.  Forgot to ask again this morning.
<deryck> flacoste, I'll check with bryce myself now.
<deryck> flacoste, issue is fixed, stakeholders happy. :)
<flacoste> awesome!
<flacoste> deryck: you can drop the follow-up from your board :-)
<deryck> ha
<deryck> yeah
<deryck> lot good that did me. :)
<deryck> flacoste, what follow up, I don't see it.  or you mean the bug abel was working on?
<flacoste> deryck: yes, the one where you had concerns with the additoinal maintenance costs
<deryck> flacoste, gotchas.  got it.
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10^2
#launchpad-dev 2013-02-18
<StevenK> wgrant: Unauthorized: (<zope.browserpage.metaconfigure.SimpleViewClass from /home/steven/launchpad/lp-branches/archive-limitedview/lib/lp/soyuz/browser/../templates/archive-packages.pt object at 0x2b1e9815e290>, 'browserDefault', 'launchpad.View')<br /> I guess from traversal, which create_initialized_view doesn't call into?
<RoelV> Hi all
<RoelV> bzr: ERROR: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
<RoelV> how do I fix this?
<wgrant> StevenK: That usually just means you don't hold launchpad.View on the object, and the view requires launchpad.View
<StevenK> wgrant: Right. Which is expected. But the test uses create_initialized_view, which shows me the page content
<wgrant> Right, that might skip it. Normally the page would 403 anyway because it accesses launchpad.View attributes
<StevenK> wgrant: So this change looks okay to me, except for the +packages test for subscribers failing
<wgrant> StevenK: What if you call the view?
<StevenK> view.render() returns the page content
<wgrant> Are you sure you set up the interaction correctly?
<wgrant> Surely it needs to access some attributes that need launchpad.View
<StevenK> The test calls login_person(self.subscriber) and c_i_v is called with self.subscriber as the principal
<StevenK> wgrant: The P3A is empty, so maybe I need to publish a source into it first
<wgrant> StevenK: Doesn't getPublishedSources need launchpad.View?
<wgrant> Oh, I guess not
<wgrant> Since it's necessary for +index
<wgrant> So you may be right
<StevenK> Right
<wgrant> But stuff like the build counters might be protected
<StevenK> I've moved description, is_active, is_copy, num_pkgs_building, publish, series_with_sources, signing_key and getPublishedSources()
<StevenK> wgrant: So I can destroy the test, but I'd like to keep it
<StevenK> Maybe I should switch it to the browser test
<wgrant> StevenK: You probably want to adjust it so it triggers the view permission failure
<StevenK> wgrant: Right, and it seems neither c_i_v() or c_v() will do so for me
<StevenK> Whereas +packages requires View in the ZCML
<StevenK> +        browser = setupBrowserForUser(self.subscriber)
<StevenK> +        self.assertRaises(Unauthorized, browser.open, url)
<StevenK> That works fine, since it uses the traversal rules
<wgrant> Right
<StevenK> So maybe I'll push this crap up and you can sob
<StevenK> lib/lp/security.py:120: 'IAbstractMilestone' imported but unused
<wgrant> Oops
<wgrant> My fault
<wgrant> A few weeks ago
<StevenK> But I got burned last time I removed useless imports :-)
<StevenK> I shall pull it out
<wgrant> lp.security is not webservice definitions :)
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/new-perm-for-archive-subscribers/+merge/148974
<wgrant> StevenK: Doesn't SubscriberView want to short-circuit if View is held?
<StevenK> Probably
<wgrant> Also, SubscriberView might want to imply LimitedView
<wgrant> s/might/does/
<StevenK> wgrant: How do I imply LimitedView?
<wgrant> StevenK: The current default LimitedView adapter delegates to View, I believe
<wgrant> It probably wants to fall back to SubscriberView
<wgrant> StevenK: You also seem to be seriously allergic to VWS
<wgrant> There's no gap before getPulbishedSources in the interface
<wgrant> And you'll want tests for the permissions
<StevenK> There is one before the API params
<wgrant> eg. check directly that a subscriber has SubscriberView but not View
<wgrant> 242	+ )
<wgrant> 243	+ # Really returns ISourcePackagePublishingHistory, see below for
<wgrant> 244	+ # patch to avoid circular import.
<wgrant> 245	+ @call_with(eager_load=True)
<wgrant> No space
<wgrant> Oh
<wgrant> So there is
<wgrant> That's just a really huge operation_parameters
<StevenK> Yes
<StevenK> wgrant: Why do I want to imply LimitedView ?
<wgrant> StevenK: Some things (eg. lazr.restful) won't acknowledge an object's existence without LimitedView
<wgrant> eg. objects will not appear in collections unless LimitedView is held
<StevenK> wgrant: http://pastebin.ubuntu.com/1675575/ is the permission test
<StevenK> wgrant: Oh, ViewArchive backs onto get_enabled_archive_filter anyway, so it should be fine, no?
<wgrant> StevenK: What do you mean?
<StevenK> get_enabled_archive_filter is used for checking for launchpad.View, and we use the same method for SubscriberView, so we don't need a short-cut?
<wgrant> StevenK: ViewArchive has several shortcuts
<wgrant> I'm not sure if get_enabled_archive_filter also has all of them, but it's almost certainly slower too
<StevenK> Right, I want to return True if ViewArchive does?
<wgrant> Right
<StevenK> wgrant: http://pastebin.ubuntu.com/1675592/
<wgrant> if check_permission(...):
<wgrant>    return True
<wgrant> if check_permission(...):
<wgrant>    return True
<wgrant> return False
<wgrant> Or or, I guess
<StevenK> wgrant: In which bit of that diff?
<StevenK> Oh, right
<StevenK> Fixing
<wgrant> All of it
<StevenK> wgrant: http://pastebin.ubuntu.com/1675600/
<wgrant> StevenK: Also, what happens when there's no SubscriberView adapter?
<wgrant> I assume it just evaluates to False, but you should confirm
<wgrant>      def checkAuthenticated(self, user):
<wgrant> +        view_permission = ViewArchive.checkAuthenticated(self, user)
<wgrant> +        if view_permission:
<wgrant> +            return view_permission
<wgrant> That's a bit silly
<StevenK> So I considered calling if check_permission or self.store.find, but that means we need to do the work to calculate the filter if the user has View anyway
<wgrant> foo = bar
<wgrant> if foo:
<wgrant>    return foo
<wgrant> Where foo is boolean
<wgrant> Is equivalent to:
<wgrant> if bar:
<wgrant>     return true
<wgrant> s/t/T/
<StevenK> wgrant: No SubscriberView adapter for IArchive, or something like IProduct?
<wgrant> StevenK: Anything
<wgrant> Because if something asks for LimitedView and View fails, it'll defer to SubscriberView
<StevenK> wgrant: Hmmm, still trying to toss up how to test that
<StevenK> And grumbling at the construction crew outside, who sound like they're grinding up cars
<wgrant> StevenK: check_permission('launchpad.SubscriberView', someproduct)
<wgrant> done
<wgrant> No need for an actual test, just do it in a harness
<StevenK> Yeah
<StevenK> ------> print(check_permission('launchpad.SubscriberView', factory.makeProduct()))
<StevenK> True
<wgrant> Erm
<wgrant> So, work out why :)
<StevenK> Sorry, dealing with the fun of car ownership
<StevenK> wgrant: Due to PermissiveSecurityPolicy
<StevenK> Which returns True for anything
<StevenK> And interaction being none results in <lp.services.webapp.authorization.LaunchpadPermissiveSecurityPolicy object at 0x8072d90>
<wgrant> StevenK: Ah, right.
<wgrant> Might have to hack the harness to use_web_security=True
<StevenK> In [2]: print(check_permission('launchpad.SubscriberView', factory.makeProduct()))
<StevenK> False
<wgrant> Right
<StevenK> With the following diff:
<wgrant> That is less terrifyingly incorrect
<StevenK> -    execute_zcml_for_scripts()
<StevenK> +    execute_zcml_for_scripts(use_web_security=True)
<StevenK> Tempted to hack iharness to respect LP_HARNESS_WEB_SECURITY=1
<StevenK> (Or so)
<StevenK> wgrant: Diff updated
<wgrant> StevenK: Done
<StevenK> wgrant: Does http://pastebin.ubuntu.com/1675739/ address your concerns?
<wgrant> StevenK: Indeed.
<StevenK> wgrant: Where's your GC branch + Archive:+delete timeout fix up to?
<wgrant> StevenK: Doing more archive deletion rework
<StevenK> Oh, so it won't suck. That would be nice.
<wgrant> Actually, might propose the first stage now
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/archive-defer-deletion/+merge/148988
<wgrant> I may not be around to answer grillings
 * StevenK gets the chair, desk lamp and dripping tap ready
<StevenK> wgrant: Did you consider only calling getUtility(IPublishingSet).requestDeletion() once in the publisher?
<wgrant> StevenK: That'd delete most binaries twice
 * wgrant goes
<StevenK> wgrant: So, you did consider it. r=me
<adeuring> good morning
<StevenK> Unauthorized: (<Archive at 0xef0f490>, 'dependencies', 'launchpad.View')
<StevenK> Bah! This was the *whole* point of the exercise
#launchpad-dev 2013-02-19
<adeuring> good morning
#launchpad-dev 2013-02-20
<StevenK> wgrant: Can you see https://qastaging.launchpad.net/~stevenk/+archive/private ?
<wgrant> StevenK: I'm god
<wgrant> StevenK: You probably want to try with an unprivileged account
<StevenK> Yeah
<StevenK> I'm doing so
<wgrant> eg. me+testing
<StevenK> I'm in the middle of using my other account
<wgrant> Ah
<StevenK> Response body:
<StevenK> ---
<StevenK> (<Archive at 0x116cf510>, 'getBuildRecords', 'launchpad.View')
<StevenK> I think I can cope with that
<StevenK> And external_dependencies is returned as redacted
<StevenK> wgrant: I can add me+testing if you want to poke around
<wgrant> StevenK: As long as +index renders with some packages it's probably OK
<StevenK> Hmm
<StevenK> How to inject some packages into it
<wgrant> You can use syncSource to copy stuff on qas
<wgrant> Although PCJs might work there too; I don't quite recall.
<StevenK> I don't know if the PCJ runner is working
<StevenK> s/working/set up to run/
<wgrant> So use syncSource :
<wgrant> )
<StevenK> wgrant: My qas auth token is my test account which has no privs
<cjwatson> I think PCJs are set up there
<cjwatson> Vaguely recall from the last time I cared
<cjwatson> You would find out soon enough
<StevenK> Oh, that reminds
<StevenK> *me
 * StevenK gets a bunch of flags removed from qas
<cjwatson> wgrant: Did you see my updates to https://code.launchpad.net/~cjwatson/launchpad/bpph-phase/+merge/144154 ?
<StevenK> Hmph, now Archive:+index gives Not allowed here
<wgrant> cjwatson: Oh, sorry, completely missed that email
<wgrant> StevenK: Exactly what I was hoping for
<StevenK> :-(
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/archive-defer-deletion/+merge/148988
<wgrant> Change is that I now set dateremoved on anything that doesn't already have it set, rather than just setting it on formerly active pubs
<wgrant> Otherwise stuff that was superseded/deleted but hadn't been p-d-r'd yet would stick around forever
<StevenK> wgrant: _getBinaryPublishingBaseClauses has include_removed=True, and getPublishedSources has it as False, which is it?
<wgrant> StevenK: Oops
<wgrant> StevenK: I initially had it as exclude_removed=False, but inverted it but missed one default
<wgrant> That would have failed a lot of tests
<wgrant> Fortunately
<wgrant> Fixed
<wgrant> StevenK: Any other objections?
<StevenK> wgrant: No, no other objections.
<wgrant> Thanks
<StevenK> Unauthorized: (<SourcePackagePublishingHistory at 0x111008d0>, 'id', 'launchpad.View')
<StevenK> This displeases me
<StevenK> That I had to move dependencies to SubscriberView also displeases me
<StevenK> wgrant: So I need to change the SPPH adapter to say you have View if you hold SubscriberView on the archive?
<wgrant> StevenK: I suppose so
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/ppa-permadeath/+merge/149462
<StevenK> No fair stealing r16500
<StevenK> renamable looks wrong
<wgrant> Oh?
<wgrant> The is_copy bit is slightly insane now that they can be published
<wgrant> But I didn't really want to go near that, and there's no way to actually trigger the problematic rename today
<StevenK> Sorry, the actual variable name, not the calculation for it
<wgrant> Maybe it deserves an extra e
<wgrant> http://en.wiktionary.org/wiki/renamable
<wgrant> renameable is listed as an alternative form, but doesn't have a page
<wgrant> I win
<StevenK> I didn't say it was wrong, I said it *looks* wrong.
<StevenK> wgrant: Bleh, IArchive.dependencies is on +indexx
<StevenK> *+index
<wgrant> StevenK: Sure, and it's not hugely sensitive.
<StevenK> wgrant: It isn't?
<wgrant> StevenK: I don't think so
<wgrant> It's a list of PPAs
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/new-perm-for-archive-subscribers-fixed/+merge/149464
<wgrant> StevenK: Have you tested this more thoroughly locally?
<StevenK> Spoil all my fun
<wgrant> StevenK: Why does checkUnauthenticated forward using check_permission (which uses global state)?
<StevenK> wgrant: Since I don't really understand security adapters and copied from the LimitedView one
<wgrant> I'd check to see if there's a reason it doesn't just use forwardCheckUnauthenticated or similar
<StevenK> wgrant: I have approved your MP
<wgrant> StevenK: Thanks
<wgrant> StevenK: For cleanup, I think we should just reset the status to active if it's deleted but enabled, and set the rest to deleting
<wgrant> And reenable the publish flag, I guess. So they get redeleted by the publisher, setting dateremoved and renaming them
<StevenK> Right
<StevenK> How many are we talking about?
<wgrant>  status | enabled | publish | count
<wgrant> --------+---------+---------+-------
<wgrant>       2 | f       | f       |  7803
<wgrant>       2 | f       | t       |     3
<wgrant>       2 | t       | f       |   241
<wgrant>       2 | t       | t       |   132
<wgrant> So ~400 get actived, and 8000 back to deleting
<StevenK> Right
<StevenK> It might be okay to do that via SQL
<StevenK> The numbers are a bit high for my liking
<wgrant> Not really. There's likely to be crap on disk
<StevenK> Right, so we need a garbo to clean this crap up?
<wgrant> And the number of pubs that would need updating is large
<wgrant> No
<wgrant> Oh
<wgrant> You mean set the archive status with SQL?
<wgrant> Yes, that was the plan
<StevenK> Right
<wgrant> I thought you meant avoiding the publisher
<StevenK> No no
<StevenK> I'm not sure about updating 8000 Archive rows in 2.5 seconds
<StevenK> With a temp table it's probably doable
<wgrant> No need for a temp table
<wgrant> Calculating the set takes about 15ms
<StevenK> No triggers on archive?
<wgrant> Apart from the fti update
<wgrant> So nothing of consequence
<StevenK> So, once we deploy we fix stuff up?
<wgrant> Right
<StevenK> Sounds like a plan
<StevenK> I'm testing this stuff
<wgrant> Great
<wgrant> Hopefully it won't need a 5th emergency landing
<StevenK> My stuff or yours?
<StevenK> Bah, forgot to subscribe myself
<wgrant> Yours
<StevenK> Don't remind me
<StevenK> Already feel like a moron
<wgrant> The UPDATE takes ~3s on DF, so should be fine on prod
<wgrant> And given the rows it's touching I don't really care if it holds a lock for slightly too long
<wgrant> The update that touches active rows is around 300ms
<wgrant> So all fine
<StevenK> wgrant: There is no expander for sources on Archive:+index, right?
<StevenK> There are not
<StevenK> So it looks good
<wgrant> Yeah
<wgrant> That's on +packages
<StevenK> Which is Forbidden
<StevenK> http://pastebin.ubuntu.com/1687066/
<wgrant> I'd hope so, given it needs View
<wgrant> StevenK: Any other calls to check_permission in lp.security that want fixing
<wgrant> ?
<StevenK> wgrant: Only LimitedViewDeferredToView
<StevenK> And it has a comment
<StevenK>         # The forward adapter approach is not reliable because the object
<StevenK>         # might not define a permission checker for launchpad.View.
<StevenK>         # eg. IHasMilestones is implicitly public to anonymous users,
<StevenK>         #     there is no nearest adapter to call checkUnauthenticated.
<wgrant> If there's no adapter then how is it implicitly public?
<StevenK> zope.Public I guess
<wgrant> Sure, but then the adapter won't be checked
<wgrant> So it doesn't matter if it delegates to nothing
<wgrant> StevenK: When was that comment added?
<StevenK> 14646.4.1 14646.4.6 and 14646.4.2
<StevenK> r14669 on devel
<wgrant> StevenK: I'd try replacing that and seeing what ec2 thigns
<wgrant> thinks
<wgrant> you can certainly use forwardCheckUnauthenticated in yours
<StevenK> forwardCheckUnauthenticated is not a method on AuthorizationBase
<StevenK> wgrant: So I guess I can't, if the method doesn't exist? :-)
<wgrant> StevenK: It's probably mostly used for edit permissions
<wgrant> Where checkUnauthenticated is always false
<wgrant> I sense an obvious solution
<StevenK> wgrant: Grep over the tree for forwardCheckUnauthenticated
<wgrant> I know it doesn't exist today
<wgrant> But there was also no way to rename a PPA three hours ago
<wgrant> :)
<StevenK> Ah, Edit don't use checkUnauthenticated
<wgrant> Right
<wgrant> Since anonymous users don't have edit/admin/etc. perms
<wgrant> Which is the primary use of delegation
<StevenK> wgrant: http://pastebin.ubuntu.com/1687202/
<wgrant> StevenK: Right
<StevenK> Tempted to refactor slightly
<wgrant> Probably a good idea
<StevenK> Since both forward methods are roughly identical
<StevenK> wgrant: http://pastebin.ubuntu.com/1687235/
<wgrant> StevenK: I'd say _getNext, but yeah
<StevenK> Sure, but it does a bunch of asserts and checks
<wgrant> But it doesn't actually do the forwarding
<wgrant> The current method name says it does
<StevenK> So the docstring lies too?
<wgrant> The forwarding bit is the call to checkAuthenticated
<StevenK> wgrant: Tossing it at ec2
<StevenK> I may forgive you at some point for stealing r16500 :-P
<wgrant> Unlikely
<lifeless> surely 16384 was more important
<wgrant> That's why I always get both 1000 and 1024 on new projects
<StevenK> bzr log doesn't tell me the commit id
<StevenK> Just that PQM commited it, which is unhelpful
<StevenK> But I think it's Curtis
<wgrant> Why do you want the ID?
<lifeless> StevenK: -n0 ? --show-revids ?
<StevenK> wgrant: Because I don't keep the commit history in my head, like you do :-P
<wgrant> If you want to know who it was, then -n0 is what you want, as lifeless says
<wgrant> Though --show-ids would probably also work, as you could see the RHS parent
<StevenK> parent: curtis.hovey@canonical.com-20121220230354-8t4pd8cx4jkztgga
<StevenK> Right
<StevenK> wgrant: I thought we destroyed the blueprint dropdown for being horrible?
<wgrant> StevenK: I thought it was removed from +supersede, but it may just have had the timeout fixed
<StevenK> I can recall a critical about ubuntu blueprints and a dropdown
<stub> Can we use a cookie as a quick hack to fix the longpoll blocking issue?
<stub> Just a counter - if it is higher than a threshold, do no long poll. If it is lower, increment it, do longpoll, decrement it when result in
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/new-perm-for-archive-subscribers-fixed/+merge/149464 is ready for a review
#launchpad-dev 2013-02-21
<wgrant> StevenK: r=me, thanks
<StevenK> wgrant: QA for r16500 ?
<StevenK> Not that it matters for approximately 70 minutes
<wgrant> StevenK: It's going to take quite a while
<StevenK> Mmmm
<StevenK> And requires DF
<StevenK> Since you have to run the PPA publisher
<wgrant> Yes
<wgrant> It still works. I checked yesterday.
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/hide-deleted-ppas/+merge/149737
<StevenK> Hm, don't we have a array for ACTIVE, DELETING already?
<wgrant> Don't think so
<StevenK> wgrant: There's a check in model/person that wants to check the same thing, it can likely be simplified
<wgrant> StevenK: Not really sure it's worth it
<StevenK> wgrant: Eh. r=me
<wgrant> StevenK: Thanks
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/filter-spec-lists/+merge/149748
<wgrant> StevenK: "specifications" isn't really a valid method name
<StevenK> getSpecifications ?
<StevenK> Hm, and I forgot to tell the interface that it's a method and not a property
<wgrant> StevenK: Does Branch.getSpecificationLinks provide value over Branch.getLinkedSpecifications?
<StevenK> wgrant: All the branch stuff wants the actual SpecificationBranch
<StevenK> Which took far too long to work out, since I originally had it returning the specs itselfs
<wgrant> OK
<StevenK> wgrant: So IBug.specifications -> IBug.getSpecifications() ?
<wgrant> StevenK: Probably nicer than getLinkedSpecifications, yeah
<StevenK> wgrant: http://pastebin.ubuntu.com/1697353/
<wgrant> StevenK: Right. I'd also like the tests to check that an unauthorised but non-anonymous user can't see private things that they can't see
<wgrant> And also, what's an accepted user?
<StevenK> I had a brainfade
<wgrant> And why has p-d-r slowed down so much...
<StevenK> wgrant: visible_user doesn't work either
<wgrant> authorised
<wgrant> shared
<StevenK> wgrant: http://pastebin.ubuntu.com/1697385/
<wgrant> StevenK: Sounds more reasonable
<wgrant> I think that's it
<StevenK> wgrant: Oh, that 'OK' meant your objection of getSpecificationLinks was withdrawn?
<wgrant> StevenK: Yes
<wgrant> StevenK: Given you can't easily avoid it
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/filter-spec-lists/+merge/149748 is updated
<StevenK> wgrant: So .using(Specification, SpecificationBug).find(Specification, SpecificationBug.bugID == self.id, Specification.id == SpecificationBug.specificationID, *get... ?
<wgrant> StevenK: No need for .using
<wgrant> It'll use autotables if there's no .using
<StevenK> wgrant: http://pastebin.ubuntu.com/1697683/
<wgrant> StevenK: Right
<StevenK> Let me land it then
<wgrant> StevenK: Bug #1131407 may be of interest
<_mup_> Bug #1131407: BranchView.landing_candidates needs preloading <easy> <lp-code> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1131407 >
<StevenK> Ah, indeed
#launchpad-dev 2013-02-22
 * StevenK grumbles at this test
<StevenK> ARGH
<StevenK> propertycache I will destroy you
<adeuring> good morning
#launchpad-dev 2014-02-17
<kindjal> Are there docs for setting up one's own Soyuz based build farm?
<kindjal> Anyone here have any experience with automated build farms, say with buildd, wanna-build, etc?
#launchpad-dev 2014-02-18
<xnox> cjwatson: wgrant: I believe i've address colin's review comments about https://code.launchpad.net/~xnox/launchpad/update-maintenance-check-3/+merge/194778 is there anything else needed to land it?
#launchpad-dev 2014-02-21
 * mpt watches rocketfuel-setup install a bunch of updates to Lucid
<mpt> âNeed to get 3,155kB of archives.â That sounds like a lotâ¦
<cjwatson> Didn't you switch your container to precise when we had an issue a few days back?
<mpt> I thought I had
<cjwatson> If not, you probably ought to now, as LP is on precise
<cjwatson> (finally)
<mpt> ok
<cjwatson> I've changed Running/LXC
<mpt> A minute before I could :)
<cjwatson> (and a few other pages)
<mpt> Hm, âlpdev login: <4>init: setvtrgb main process (270) terminated with status 1â doesnât look like a login prompt, but it is
<mpt> There are fewer of those errors in the precise version than the lucid version, at least
#launchpad-dev 2015-02-16
<wgrant> cjwatson: Thanks for the review. Didn't get to yours today, I'm afraid. Too much buildd and related amusement.
<cjwatson> NP, I have plenty to do with splitting out the next in the series and working out why it makes one of the sharingservice tests fail.
<wgrant> Heh
<cjwatson> Which would be fine if we didn't also have simultaneous drain blockage and tumble-dryer failure because why does everything hate me.
<cjwatson> So I think I will be working from the launderette this afternoon.
<wgrant> You should move to Australia, where we have air conditioner failures but it's warm enough to avoid tumble dryers.
<cjwatson> I think Kirsten would expire on the spot.
<wgrant> Likely.
<cjwatson> She's one of the few people I know who can be too hot in the middle of an English winter.
<wgrant> ...
<cjwatson> wgrant: Just noticed a problem with the default-repository methods in git-basic-model.  My code requires that a target default must also be the default for the combination of the repository owner and its target, to avoid what I think would be the very confusing situation where lp:launchpad is owned by ~launchpad-pqm but lp:launchpad != lp:~launchpad-pqm/launchpad.  But the project owner is allowed to set the default repository for ...
<cjwatson> ... their project even if they don't own that repository, to handle the bot-owned repository case; and in general setting an owner-target default requires launchpad.Edit on the owner, which I think is sensible in isolation.  So do I (a) give special dispensation to setTargetDefault to set the owner-target default (still provided that there isn't one already set) even if you aren't the owner/admin, (b) require you to set the ...
<cjwatson> ... owner-target default first in this case (but that would be awkward in the bot case which is the whole point here) (c) something else?
<cjwatson> Oh or (d) compute owner-target defaults dynamically, by looking first for an explicit owner-target default but falling back to a target default with the same owner
<wgrant> cjwatson: A good question. I'll have to think about that one...
#launchpad-dev 2015-02-18
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/db-git-no-target-implies-owner/+merge/250114 per discussion last night
<wgrant> We should be able to do that hot (and I don't think we currently have the ability to do cold ones).
<cjwatson> Because wildcherry/aurora?
<wgrant> Right.
<cjwatson> Shall I just ask a webop to sort that out after landing?
<wgrant> That or stub.
<cjwatson> Hm, is step 11 of the procedure in https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess still valid?
<wgrant> cjwatson: I'm not too fussed. This won't break any tests.
<stub> fdt should still be working, although I should probably be there for the next one.
<stub> dropping a contraint on an unused table is no problem live.
<stub> wgrant, cjwatson : Run that now?
<stub> I'm being called away, let me know.
<cjwatson> stub: Please do.
<stub> cjwatson: done
<cjwatson> stub: Thanks.  Any reason I shouldn't merge it into devel once db-stable updates?
<cjwatson> Well, staging
<cjwatson> stub: Did you update LaunchpadDatabaseRevision?
<stub> cjwatson: It should be merged everywhere it isn't sooner rather than later. Yes, LaunchpadDatabaseRevision was updated.
<cjwatson> Right, I'll get it into devel
<cjwatson> stub: It's on devel now.
<thomi> wgrant: what's this mail header about, do you know? "X-Katie: Launchpad actually"
<cjwatson> thomi: It's for compatibility with mail filtering designed for Debian's dak archive management software, formerly called "katie".
<thomi> huh - what is 'dak'?
<cjwatson> Debian's archive management software
<thomi> ahh., ok
<cjwatson> Sort of a distant spiritual ancestor of Soyuz
<thomi> thanks :D
<blr> cjwatson: there are config knobs for virtinfo_endpoint and authentication_endpoint in the turnip charm now.
<cjwatson> Ah cool, will give that a try tomorrow thanks!
<cjwatson> Also want to spend some of tomorrow on the non-LP work of finishing getting proposed-migration set up for SRUs
<blr> cjwatson: won't work until the turnip config branch has been merged however.
<cjwatson> That shouldn't take too long to fix up, I guess
<blr> yep, just let me know if there's anything particularly egregious :)
<cjwatson> Just the things in the diff comments on my review.  It looked pretty good for the most part.
<blr> cjwatson: oh I missed that, apologies.
<cjwatson> Ah, I thought your reply was brief :)
<cjwatson> Yeah, can be a little easy to miss in the mail if you're not looking out for it
#launchpad-dev 2015-02-19
<cjwatson> wgrant: Is there a reason to kill the UPLOADING status from the TTBB case in https://code.launchpad.net/~wgrant/launchpad/buildd-manager-handoff/+merge/250246 ?  The tarball processing is potentially slow, so it's worth having an indication of what's going on, and I can't see how archiveuploader could end up processing something from a TTBJ.
<wgrant> cjwatson: I guess I could restore that. But it'd better not often be too slow, since it blocks all of buildd-manager...
<cjwatson> Indeed.
#launchpad-dev 2015-02-22
<thomi> anyone feel like reviewing a simple branch for me? https://code.launchpad.net/~thomir/launchpad/devel-fix-package-rebuild-language/+merge/250568
#launchpad-dev 2018-02-20
<juliank> I'm using launchpad lib.
<juliank> I'm trying to associate teams subscribed to packages.
<juliank> So, for example, fb=launchpad.people["foundations-bugs"]
<juliank> I try ps=fb.getBugSubscriberPackages()
<juliank> len(ps) is 1441, but list(ps) is empty and ps[0] fails with IndexError
<juliank> Or given, a source package, like apt, I do apt.getSubscriptions()
<juliank> it's len is 11, but indexing at 0 produces an IndexError as well
<wgrant> juliank: There is an odd bug where for some collections you need to be authenticated to view their contents
<wgrant> Authenticated as anyone, but authenticated.
<wgrant> So login_anonymously won't let you see them.
<juliank> wgrant: Hmm, that's bad.
<juliank> wgrant: But yes, that's it
<wgrant> juliank: It is a trivial fix, if it is actually inconvenient for you.
<wgrant> grep for AnonymousAuthorization
<juliank> wgrant: I'm trying to make that ftbfs page I asked for in #ubuntuwire report by team.
<juliank> Oh, I'll just use http://people.canonical.com/~ubuntu-archive/package-team-mapping.json like MoM does, that's much faster.
<wgrant> https://code.launchpad.net/~wgrant/launchpad/anon-dsp will fix the API
<juliank> wgrant: Is that ubuntuwire thing still running on precise?
<juliank> because, um, it uses the old python-apt API that's not in trusty anymore
<wgrant> juliank: No, the local diff is purely that port.
<wgrant> I hoped nobody would notice, but I didn't expect we'd hire you :)
<wgrant> juliank: Pushed.
<juliank> wgrant: Sent you a merge request to add per-team lists.
<juliank> * proposal
<wgrant> juliank: Thanks, will look tomorrow!
#launchpad-dev 2020-02-17
<tomwardill> cjwatson, wgrant: I'm running into a problem trying to get an OCI build (in ocirecipebuildbehaviour.extraBuildArgs) to have the Universe component enabled. The OCIRecipeBuild doesn't have distro_arch_series, so none of the tooling in archivedependencies appears to fit. Any suggestions for workarounds?
<tomwardill> I need Universe as that's where the Docker package lives
<tomwardill> if I'm understanding this right, we'd need that infrastructure even if it was in a PPA though
<wgrant> tomwardill: I haven't thought it through in full detail, but this sounds somewhat analogous to TTBs, where the series is implicit (or configured at the system level). Except in this case you select the DAS based on OCIRB.processor rather than nominatedarchindep
<lifeless> yay bug 7
<mup> Bug #7: Need help for novice translators <lp-translations> <verification-needed-eoan> <Launchpad itself:Fix Released by danilo> <Launchpad Documentation:Fix Released by matthew.revell> <https://launchpad.net/bugs/7>
#launchpad-dev 2020-02-18
<tomwardill> wgrant: sorry, what's a TTB?
<tomwardill> I can follow the rest of that
<wgrant> tomwardill: TranslationTemplatesBuild
<tomwardill> aha!
<tomwardill> yes, that feels roughly analogous
<tomwardill> thanks
<wgrant> Takes a branch, runs some predefined stuff on it, uploads translations to LP
<wgrant> So has no related Ubuntu series, even.
<tomwardill> yeah, that makes sense
<cjwatson> Agreed, I think that's the best approach
<cjwatson> Ideally make distro_arch_series / distro_series / pocket properties of the build work based on that
<tomwardill> yeah, that's what I think I've got
<tomwardill> (ish, it's a bit of a mess atm, currently tidying up)
<tomwardill> No url for <lp.oci.model.ocirecipebuild.OCIRecipeBuild object at 0x7f17e1d7d750> because <lp.oci.model.ocirecipebuild.OCIRecipeBuild object at 0x7f17e1d7d750> broke the chain.
<tomwardill> well.
<tomwardill> cjwatson: is that part of the work you did with the views?
<cjwatson> tomwardill: one sec
 * cjwatson peers at +activereviews
<tomwardill> yeah, I just had a look through that
<cjwatson> I could have sworn I did something here, but where
<tomwardill> I definitely remember seeing a branch for it
<tomwardill> because we had the discussion about the rebase chain becoming a tree
<cjwatson> Maybe that was something else
<cjwatson> Oh, I did OCI *project* views (4a1852da9d872d815daa0bed1dd36ef9bcc2f699)
<cjwatson> I haven't done any recipe views, and they have no URLs defined at the moment
<cjwatson> Do you need me to work something out there?
<tomwardill> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/376106
<tomwardill> ah, yes
<tomwardill> cjwatson: ideally if you have time, as I suspect it'd be quicker for you to do it agian than me
<cjwatson> OK
<cjwatson> Let me fix buildbot first
<tomwardill> cool, I'll work around this for now and continue implementing extraBuildArgs
<cjwatson> buildbot issues were exposed by converting away from urllib.urlopen, but I think they're fundamentally a txfixtures bug.  I've pushed https://github.com/testing-cabal/txfixtures/pull/13 which should fix that (test failures there are fixed by https://github.com/testing-cabal/txfixtures/pull/12), and I've poked Free about reviewing them
<tomwardill> Ran 1 tests with 0 failures, 0 errors, 0 skipped in 1.038 seconds.
<tomwardill> weeeeeee
<tomwardill> extraBuildArgs for OCIRecipeBuildBehaviour.
<tomwardill> Now to unpick the mess I've made, and ensure that it's actually the values that I wanted
<tomwardill> cjwatson: that doesn't look like it was a whole lot of fun to find
<cjwatson> tomwardill: Not exactly
<cjwatson> lifeless: Thanks for the testing-cabal invite.  Do you think I could have PyPI access to txfixtures?
<lifeless> cjwatson: of course
<lifeless> I wish pypi had teams
<cjwatson> Yeah, totally
<lifeless> added you to a few random projects
<cjwatson> Thanks, I like randomness
<lifeless> speaking of random, you should consider porting launchpad to rust
<cjwatson> A nice small project
<lifeless> side project for when you're not busy
<cjwatson> :)
<lifeless> but this is actually a serious suggestion
<lifeless> I've been watching the way you've been adding things in - more and more microservice style additions
<lifeless> which is great
<lifeless> I presume you're ported or nearly so to run live on py3 at this point
<lifeless> which means you'll have mypy gradual typing
<lifeless> anyhow, point is that rust productivity has been steadily climbing (as has golangs for fairness), but in the last 6-12 months I think things have gotten to the point where for production services, developer productivity now exceeds that of python
<cjwatson> Released txfixtures 0.4.3
<lifeless> so I think if Launchpad wants to be around for another 5-10 years, a sensible question to ask is what is the most developer efficient platform for it to be in, and a dead-end fork of an experimental branch of Zope probably isn't it :)
<cjwatson> It's conceivable that we'd look at Rust for an experimental service at some point, though there's some politics too of course
<lifeless> the rest of the conclusions pretty much follow :P
<cjwatson> I don't think it's really true that we're on a dead-end fork of Zope any more - I upgraded to newest versions of everything late last year
<cjwatson> Certainly Zope has less mindshare and I want to get off e.g. zope.server
<lifeless> I retract my statements then
<lifeless> I had thought that a lot of the stuff we had was experiments that never gained traction
<cjwatson> I don't disagree with everything you say, although there is the minor practical note that I haven't learned Rust yet :)
<cjwatson> I think we are probably in a better situation upstream-wise than you remember
<lifeless> well done, thats fantastic news
<cjwatson> At the moment I think we have two small zope.* forks
<lifeless> I think you will very much like rust
<lifeless> and hey - its Gnome aligned :P
<cjwatson> zope.session to stick with an old session cookie algorithm, and zope.testrunner for a couple of tweaks to test ID handling and some changes to thread leak handling (I got all the subunit stuff back upstream again, which was the big chunk)
<cjwatson> I think I take a more incrementalist attitude (it was ever thus); there's still a lot we can do to move LP onto things developers find familiar, and at the moment (modulo finishing the py3 port) I think we're in a good position to migrate to them gradually
<cjwatson> while possibly looking to entirely different platforms for microservices
<lifeless> certainly any migration would have to be incremental :P
 * cjwatson sleeps
<lifeless> night! thanks for the chat
#launchpad-dev 2020-02-19
<wgrant> lifeless: As Colin says, ZTK has become a lot less dead than it was when you left.
<wgrant> Still not massively thriving, but by no means dead
<wgrant> Zope 3, though, heh.
<wgrant> I haven't tried Rust for a network service in a couple of years.
<wgrant> It's all been more low level stuff.
<lifeless> wgrant: yeah, I'd been doing similar things. I wrote an API server in it last week though, and was very productive.
<lifeless> wgrant: rocket.rs + diesel
<lifeless> wgrant: and some of the smarts like the new adaptive scheduler in std-async.rs are just mwah.
<cjwatson> I think https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379456 should fix buildbot; could anyone review?
<cjwatson> And if anyone could look over https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/378828, that's pure UI
<tomwardill> cjwatson: +1 to the buildbot bits
<tomwardill> and a comment on the UI ones
 * tomwardill -> lunch
<cjwatson> thanks
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379464   Mailman integration removal \o/
<tomwardill> weeeee
<wgrant> Burn it all.
<tomwardill> wgrant: I'll leave that review to you to +1 ;)
<wgrant> Too late
<tomwardill> i feel like you might appreciate it
<tomwardill> hah
<tomwardill> good work
<cjwatson> Just waiting for IS to land the corresponding lp-production-configs change before I land that
<cjwatson> tomwardill: I've been making a decent amount of progress on OCI recipe views; next is to write tests
<tomwardill> cool :)
<cjwatson> I say "write", I mean "shamelessly copy"
<tomwardill> I'm just working out how to get universe enabled, then need to move some copy and paste stuff into mixins (snap + oci test code), and then "write" more tests similarly :)
<tomwardill> got the extraBuildArgs doing stuff and creating things that looks right though
<cjwatson> wgrant: BTW I don't suppose you've had a chance to look at lazr.restful py3-declarations?
<cjwatson> I was otherwise thinking of a lazr.restful release soon even though the py3 port isn't complete, mainly because it makes it easier to build an LP py3 virtualenv
<cjwatson> Quick review of https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379473 anyone?
<cjwatson> (Send script-monitor emails to launchpad-error-reports@)
<cjwatson> Also some reasonably straightforward py3 bits: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379469 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379470 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379471
<tomwardill> +1 to mail list change
<pappacena> I'll take a look
<cjwatson> lazr.restful upgrade: https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+merge/379486 / https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379487
<tomwardill> cjwatson: I have a test that is trying to ensure that the Universe sources get added to the buildd args (via extraBuildArgs). My current_component in the build is 'multiverse' (currently, will change that), but I'm still only getting the Pocket sources (Updates) added.
<cjwatson> updates and universe are different categories, so I'm confused about what you mean
<tomwardill> ah, sorry, badly worded maybe (or misunderstood)
<tomwardill> essentially: I get no universe deb lines added
<cjwatson> OK, can I see your code?
<cjwatson> And the test failure
<tomwardill> I think because archive.getComponentsForSeries is returning None in _get_sources_list_for_dependencies (soyuz.adaptors.archivedependencies)
<tomwardill> one sec, I'll check this in and push. The test isn't testing for the presence of the line yet, I've established it via pdb on the relevant line
<tomwardill> cjwatson:  https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379201 (test_extraBuildArgs_git)
<tomwardill> the expected_archives parameter there doesn't include universe
<tomwardill> I _think_ because the distribution that is being created for the test isn't configured correctly?
<cjwatson> OK, I'll have a look
<tomwardill> thanks :)
<cjwatson> tomwardill: So this is broken in TestAsyncSnapBuildBehaviour as well.  I think the problem is that the distroseries in use for the test has no ComponentSelection rows telling it what components exist.  You could set that up in the test (see near the top of lib/lp/soyuz/adapters/tests/test_archivedependencies.py) or use sampledata; probably better the former
<tomwardill> aha, I think that test_archivedependencies was what I was looking for
<tomwardill> I figured there must be some way to set this up
 * tomwardill tries
<tomwardill> thanks!
<cjwatson> np.  Feel free to fix the snap case too :)
<cjwatson> Might be easier to start there since you know the rest of that test already works
<cjwatson> Total: 24 tests, 3 failures, 20 errors, 0 skipped in 32.581 seconds.
<cjwatson> On the plus side, that means one test worked
<tomwardill> ... probably a start?
<cjwatson> I'm leaving out the +request-builds view for now
<tomwardill> "deb unique-from-factory-py-line489-100020/distro unstable main universe"
<tomwardill> weee!
<tomwardill> cjwatson: add check for none-default components to test_snapbuildbehaviour: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379495
<cjwatson> tomwardill: +1 with minor comment
<tomwardill> cjwatson: ah, lovely
 * tomwardill changes
<tomwardill> pushed fix, landing
#launchpad-dev 2020-02-20
<cjwatson> tomwardill: I think I have basic views for OCI recipes that aren't entirely broken.  I needed to fix up a few things at the interface/model level too; should have some MPs for you later today.
<tomwardill> cjwatson: excellent, thanks!
<cjwatson> Several bits left out for now due to not enough model support yet
<cjwatson> But it's a skeleton at least
<tomwardill> I'm just working out why my interface has stopped being an interface and then the Build Behaviour stuff should be ready again
<cjwatson> Properties that raise exceptions are a great way for interface verification to stop working.  Not that I ever do that.
<tomwardill> yeah, I think that is literally the cause
 * tomwardill breaks out pdb
<cjwatson> tomwardill: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379539 (smallish) https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379540 (quite large)
 * tomwardill will look in a mo
<tomwardill> Total: 87 tests, 0 failures, 0 errors, 0 skipped in 49.559 seconds
<tomwardill> finally!
<tomwardill> now to add more tests
<cjwatson> wgrant: I've been looking at the python-openid fork situation (https://bugs.launchpad.net/launchpad/+bug/676372, https://bugs.launchpad.net/launchpad/+bug/1034376).  I believe that the bug that led to this can no longer be relevant, because as of https://code.launchpad.net/~cjwatson/launchpad/testopenid-certificate/+merge/314768 we force the OpenID fetcher to Urllib2Fetcher in loggerhead as ...
<cjwatson> ... well as in lp.services.webapp.login, and the fix was only ever needed in CurlFetcher
<mup> Bug #676372: "Server denied check_authentication" from bazaar.launchpad.net private branch since 11926 deployed <bad-commit-12290> <lp-foundations> <qa-ok> <regression> <Launchpad itself:Fix Released by wgrant> <https://launchpad.net/bugs/676372>
<mup> Bug #1034376: Unable to see private branch contents on https://bazaar.launchpad.net <codebrowse> <openid> <qa-ok> <regression> <Launchpad itself:Fix Released by deryck> <https://launchpad.net/bugs/1034376>
<cjwatson> wgrant: Do you agree with this analysis?  If so I'd like to try dropping the patch (probably in the process of moving to python-openid2, which is maintained and supports Python 3)
<tomwardill> Total: 123 tests, 0 failures, 0 errors, 0 skipped in 1 minutes 23.144 seconds.
<tomwardill> woooo
<pappacena> ð
<SpecialK|Canon> Nice!
<tomwardill> cjwatson: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379201 is ready for another round of review. Areas of concern are 'have i implemented enough of buildbehaviour', and the methods for distro_arch_series and where I'm pulling the values for that from.
<cjwatson> Ack, queueing up
<tomwardill> ta :)
<cjwatson> tomwardill: getting closer :)  reviewed
<cjwatson> Thiago's HTTPS mirrors stuff is next in my queue but I'm not going to get to it today, so tomorrow morning instead
<pappacena> Thanks, cjwatson! No rush. :-)
<wgrant> cjwatson: That sounds like a reasonable analysis and plan.
#launchpad-dev 2020-02-21
<cjwatson> tomwardill: Have you had any further thoughts on https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/378828 ?
<cjwatson> (tour-git)
<tomwardill> looking
<tomwardill> cjwatson: also can't think of an alternaitve, so have a +1 :)
<cjwatson> heh, thanks
<pappacena> tomwardill, is the python3 port for turnip done? I saw we have a card for that on Trello, and you have a branch for it with some commits. Do you think I can continue from your branch?
<tomwardill> pappacena: it's not done, feel free to pick it up from my branch!
<tomwardill> I was hoping to get to it after OCI, but then that took longer than expected, so grab it :)
<pappacena> :-) I'll try to work a bit on that. Thanks!
<tomwardill> one day, I will work on something twisted based without just randomly sticking defer.inlineCallbacks on everything
<tomwardill> today is not that day
 * SpecialK|Canon defers tomwardill 
 * tomwardill defernestrates
<tomwardill> cjwatson: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379201 ready for round... I've lost count... of reviews :)
<tomwardill> "last edit was 14 years ago"
<tomwardill> well.
<SpecialK|Canon> heh what?
<pappacena> But it seems like yesterday...
<tomwardill> login.py is old, apparently
<tomwardill> also, thanks python-openid for burying/wrapping a certificate error 4 layers deep, that was a fun one to find
<tomwardill> (why I can't login in the buildtheworld launchpad)
<tomwardill> hopefully a relatively straightforward review to allow apache configs to be installed from not the development directory: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379644
<cjwatson> tomwardill: How about doing copy-certificates too?
<cjwatson> Hm, well, I guess there isn't necessarily a launchpad.{crt,key} anywhere else
<cjwatson> Ignore me
<tomwardill> well, there will be
<tomwardill> so that's not a bad idea
<cjwatson> tomwardill: r=me
<cjwatson> (either way)
<pappacena> cjwatson, do you have time for a quick 4 lines review? It might fix a bug on turnip: https://code.launchpad.net/~pappacena/turnip/+git/turnip/+merge/379645
<pappacena> (btw, thanks for the review on the HTTPS thing. I totally forgot about interface tests for Brody's changes...)
<cjwatson> pappacena: It's very unlikely to make a difference to too-many-open-files bugs; I'm fairly sure TurnipConfig is only instantiated a few times at startup at most, except in the test suite.  But that's fine anyway, r=me
<pappacena> It's actually instantiated at BaseApi's __init__...
<cjwatson> pappacena: Oh, I guess that is instantiated per-request, fair enough
<cjwatson> Should probably make that a singleton at some point too
<cjwatson> Could just be "from turnip.config import config" or similar
<pappacena> We do not need "on the fly" configuration changes in any way, right? I'll add the singleton to this MP, then...
<cjwatson> We don't
<SpecialK|Canon> nice
<tomwardill> ooh, I can login!
<cjwatson> The charm restarts the service when it applies config changes
<pappacena> Great! I'll push a change in few minutes
<cjwatson> Could anyone look at https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379618 for that GDPR handling thing I mentioned in the standup?
<pappacena> I can take a look. Give me 10 minutes
<SpecialK|Canon> r=me
<SpecialK|Canon> cjwatson: ^ (sorry pappacena didn't see your message!)
<pappacena> Pushed the TurnipConfig singleton implementation, cjwatson. Another round of review, pls? :)
<cjwatson> pappacena: There are a bunch of other instantiations found by git grep
<cjwatson> *.tac
<cjwatson> doita
<cjwatson> err
<cjwatson> SpecialK|Canon: ta
<pappacena> Ha. Maybe my IDE is not searching the whole project. Let me double check.
<SpecialK|Canon> cjwatson: heh sorry
<tomwardill> vscode doesn't see .tac as a code file by default
<pappacena> yep... neither pycharm :-(
<cjwatson> You lot and your IDEs ;-)
<tomwardill> heh
<pappacena> hahahah
<cjwatson> "git config --global alias.cp cherry-pick" is one of the better things I ever did
 * doismellburning lurks for tab-completion porpoises
<cjwatson> heh
<SpecialK|Canon> cjwatson: I don't know how I feel about that alias...do you `git mv`?? ;P
<SpecialK|Canon> I guess it is quite like a cp
<cjwatson> I do.  OTOH ... yes, that
<cjwatson> And I'm doing it a lot at the moment
<cjwatson> But that's the great thing about local aliases, nobody else has to like them
<SpecialK|Canon> Hah absolutely!
<pappacena> Now the singleton thing should be fine...
<cjwatson> LGTM, thanks
<pappacena> Thank you!
<pappacena> Ah, I don't have permission to top-approve on turnip. If you can do it, or give me permission...
<cjwatson> SpecialK|Canon: ^- how come pappacena isn't in ~ols ?
<pappacena> Â¯\_(ã)_/Â¯
<cjwatson> I've also changed that reviewer team to ~launchpad-reviewers (which won't help with this MP, but will with the next one)
<cjwatson> pappacena: I've top-approved that now
<pappacena> Thanks!
<SpecialK|Canon> cjwatson: pappacena: process bug, sorry, fixing
<SpecialK|Canon> actually hm where _is_ that process
<cjwatson> Yeah I figured it was a missing checklist item somewhere
<SpecialK|Canon> I think we don't actually have an explicit checklist for this because rate of change is probably a bit higher than rate of hiring
<cjwatson> Point
<pappacena> It makes sense...
<tomwardill> tehre was some element of it in the snapstore onboarding trello
<tomwardill> I think there was a card along the lines of 'check you have the right permissions'
<SpecialK|Canon> tomwardill: that rings a bell, yes!
<tomwardill> I remember checking my list against glower's list
<SpecialK|Canon> tomwardill: I didn't give that to pappacena because it's 99% Store/Snap
<tomwardill> yeah
<cjwatson> Python 3 reviews for today: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379509 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379648 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379650 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379654 ...
<cjwatson> ... https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379657
<cjwatson> If anyone's interested, my strategy here at the moment is that I have a very very rebasey branch with a bunch of smallish hacks to make it possible to build a py3 virtualenv at all, and then I'm repeatedly running "make", seeing the next error, identifying a common theme to fix across the codebase in a single logical commit (where feasible), committing that cold (since I can't run more than ...
<cjwatson> ... trivial tests yet), and doing that several times over; then, separately, cherry-pick individual commits from that onto branches off master, test them, and file MPs
<cjwatson> This makes it sort of vaguely practical to make progress
<SpecialK|Canon> Nice
<cjwatson> Several of the hacks are fundamentally unlandable at the moment because they're of the form of "rip out $dependency in the full knowledge that this breaks some subsystem"
<SpecialK|Canon> Can you...right that
<cjwatson> But doing it this way means that those hacks don't completely block doing other stuff
<SpecialK|Canon> Yup yup makes sense
<cjwatson> When it gets a bit more reasonable I'll probably push it as a py3-wip branch or something
<SpecialK|Canon> I'm going to stop even half-typing questions for a bit here as I keep getting pre-empted ;P
<SpecialK|Canon> Sounds great :)
<SpecialK|Canon> ooi what are the main bits of hackery?
<cjwatson> https://people.canonical.com/~cjwatson/tmp/py3-tig.png
<cjwatson> Gives you an idea
<SpecialK|Canon> Can we drop CVS codeimport functionality?
<SpecialK|Canon> Better question, sorry
<SpecialK|Canon> Do we have any feels/data as to the value of CVS codeimport functionality?
<cjwatson> https://code.launchpad.net/+code-imports/?field.review_status=&field.review_status-empty-marker=1&field.rcs_type=CVS&field.rcs_type-empty-marker=1&field.target_rcs_type=BZR&field.target_rcs_type-empty-marker=1&submit=Submit+Query
<cjwatson> Quite a lot of failed but there are some that are still used
<cjwatson> It's an option
<cjwatson> Main constituency that still inexplicably hasn't moved is some of the BSDs
<cjwatson> So there's things like https://code.launchpad.net/~vcs-imports/pmake/main
<cjwatson> https://code.launchpad.net/+code-imports/+index?field.review_status=REVIEWED&field.review_status-empty-marker=1&field.rcs_type=CVS&field.rcs_type-empty-marker=1&field.target_rcs_type=BZR&field.target_rcs_type-empty-marker=1&submit=Submit+Query is a more reasonable query, so not that many still working
<cjwatson> https://code.launchpad.net/~mirabilos/mksh/MAIN is another one that's active
<cjwatson> The other option there is that we probably want to separate codeimport anyway
<cjwatson> So it doesn't have to block the rest of LP
<SpecialK|Canon> Right, yup
<cjwatson> Anyway, I've been making like 500% faster progress since adopting this sort of approach.  I totally recommend it for anything where starting the project is kinda blocked on some dependencies but in fact if you temporarily rip stuff out you can find a lot of things behind it
<cjwatson> Right.  Coffee, finish reviewing Tom's stuff, and then probably weekend
 * tomwardill is arguing with apache and ssl certificates, then weekend :)
<tomwardill> turns out shouting 'but it's RIGHT THERE' at apache isn't all that effective
<SpecialK|Canon> effective for getting results? sure
<SpecialK|Canon> effective as an outlet for frustration? I bet it is...
<SpecialK|Canon> er, make that first one a "not effective" sigh
<cjwatson> When Kirsten was a TA she used to respond to the school computers producing "cannot access printer" errors by shouting "*I* CAN ACCESS PRINTER" at them
<tomwardill> that seems entirely reasonable tbh
<tomwardill> especially when printers are involved
<tomwardill> aah, libxml2, we meet again
<tomwardill> that's why my initial make takes forever
<cjwatson> All right, EOW I think.  Have a good weekend
<pappacena> Have a good weekend!
