#launchpad-dev 2010-06-07
<spm> wgrant: awesome. will do what I can.
<james_w> lifeless: plus, hiccup with test discovery, discover on 'foo' that finds 'foo.test_foo' will load it as 'test_foo', so that becomes the root of the test id, and then that can't be run by subunit.run.
<james_w> lifeless: I think that's a bug in discover, but I'd like your opinion.
<spm> "<Fault 8002: 'error'>" <== this is not the error message you're looking for....
<spm> wgrant: just set 'actinium' back to ok, see if that sorts itself; if it does, I'll do the rest
<wgrant> spm: Great, thanks.
<wgrant> Some of the others were out with 'No route to host', which suggests a network glitch of some kind, which could also have caused the 'error's, I guess.
<wgrant> Although the fact that it was all i386 hints that it might have been a rogue recipe build -- I guess we'll see soon.
<spm> heh
<james_w> lifeless: ah, it works if you use "." rather than "foo" as the discovery root. I don't think that's ideal, but I can see that discover may want to act the way it does so that you can discover on a root of ../../../somewhere if you want. I'd be happy to chat about this tomorrow if you like.
<spm> wgrant: nope. looks more serious. it's dropped straight back into "I'm broken" state.
<wgrant> spm: Argh, anything more useful in the b-m log than good ol' fault 8002?
<spm> wgrant: you want me to look in logs for meaning vs just making something up!!?!?! good lord man, it's too early on a monday morning for that level of competence!
<spm> bleh. that seems... unusual.
<lifeless> james_w: its a python limitation I suspect
<lifeless> james_w: it can't tell, due to the flexability of importing, where in the python namespace '.' is; it may need a second parameter to say 'here is the root', which may not be built in.
<lifeless> james_w: so I think discover can be improved
<adeuring> good morning
<wgrant> noodles775: Morning.
<noodles775> Hi wgrant and adeuring
<adeuring> hi noodles775!
<wgrant> noodles775: Are you going to have time to arrange a CP of the log parser branch?
<noodles775> wgrant: We need to QA it first, but yes, it should happen soon. I'm currently organising the CP for the builders timeout (edge is now working with the fix).
<wgrant> noodles775: Yep, I noticed edge was looking good.
<wgrant> (well, apart from the i386 builders, but that's hopefully unrelated)
<noodles775> wgrant: yeah, has there been any discussion there? I just tried enabling actinium, but it went straight back to Fault 8002. I'll check the logs.
<spm> noodles775: https://pastebin.canonical.com/33033/
<spm> I tried exactly the same earlier today
<wgrant> Yeah, I was wondering what was in the logs.
<wgrant> Given that it's only i386, I'm suspicious that a recipe build may have snuck through.
<noodles775> Last mention of actinium seems fine: 2010-06-07 07:44:58+0100 [-] Processing successful build amd64 build of kwin-effect-bereflected 0.1~lucid~ppa1 in ubuntu lucid RELEASE from builder protactin
<noodles775> quite recent.
<wgrant> protactin != actinium?
<wgrant> Hm.
<spm> heh, that caught me out as well :-)
<noodles775> right, I just search backwards...
<wgrant> protactin doesn't exist.
<noodles775> and cut the line while copying...
<wgrant> Ah.
<wgrant> So what about actinium?
<wgrant> And who could possibly have thought that having those two names was a good idea? :P
<noodles775> heh
<noodles775> wgrant: just the traceback that spm posted... ah, I'll post it on the public one.
<noodles775> http://pastebin.ubuntu.com/445954/
<wgrant> Oh, useful...
<wgrant> I guess that firing the rest trigger will fix them, but it might be nice to grab logs first if possible...
<wgrant> s/rest/reset/
<kb9vqf> I'm looking at setting up a local copy of Launchpad for a project I'm working on, but am having trouble getting Launchpad to accept a different OpenID URL than the default
<wgrant> Ohhhhhhh, rescueIfLost.
<wgrant> So it was a network glitch.
<kb9vqf> How can I set a custom OpenID URL?  Something in launchpad-lazr.conf?  Is there any documentation for that file?
<wgrant> spm, noodles775: So, we have a whole lot of builders that need resetting. Is that easy enough?
<noodles775> Is that something you can do spm? lamont normally does it afaik.
<spm> noodles775: no unf, only lamont or one of the gsa's. and it's a NZ holiday, so paul hasn't been around.
<wgrant> (Basically, the builders dropped of the network for a while. The master noticed, and unassigned their builds. They came back, and the master noticed they were doing something they shouldn't be doing. The master tried to abort the builds. Aborting them is known to be... um.... buggy).
<spm> ha
<wgrant> Just the slaves need to be reset -- the thing that's done before every build.
<bigjools> morning all
<wgrant> Evening bigjools.
<maxb> jelmer: Hi, about that fretsonfire vcs-import. Should we be complaining that the import of the upstream fretsonfire project was requested into the LP fretsonfire-severedfifth namespace instead of the LP fretsonfire namespace?
<jelmer> maxb: yeah, we probably should
<jelmer> maxb: I don't think we're able to change the project of a branch though
<jelmer> maxb: also, this wouldn't be a problem if URLs didn't have to be unique..
<maxb> Well, even if not that, it's a minor waste of resources to be importing the same branch twice
<maxb> I guess we have the option of deleting the branch and requesting a replacement import of our own
<wgrant> jelmer, maxb: You can change the product through the API.
<jelmer> wgrant: not when you don't have permission to do so I hope?
<wgrant> jelmer: Heh, no.
<mwhudson> ~vcs-imports has edit on all import branches though
<wgrant> bigjools: Ugh, that log parser issue is in the apachelog contrib module.
<bigjools> yeah noodles775 just noticed that too
<wgrant> It looks likt it might not be a real issue.
<wgrant> Since it's trying to parse from somewhere other than the start of the line.
<wgrant> So... either the exisitng offsets in the DB confused it, or something put bad offsets there.
<wgrant> You've not changed the log file content at all?>
<noodles775> But those lines are passed in by our code (specifically: next_line = fd.next())... so it could be a bad offset... that's a good point.
<wgrant> If two have the same first line, the world will end like this.
<bigjools> it should be smarter about offsets and seek to the nearest \n
<wgrant> Well, it should probably actually complain if there isn't a \n right there.
<wgrant> Since we want to know if something has gone wrong.
<mars> gary_poster, here is an in-depth summary of the issue and outstanding questions for bug 587886
<mup> Bug #587886: ec2 test mail reports SUCCESS when the suite fails <build-infrastructure> <Launchpad Foundations:In Progress by mars> <https://launchpad.net/bugs/587886>
<mars> gary_poster, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/587886/comments/3
<mup> Bug #587886: ec2 test mail reports SUCCESS when the suite fails <build-infrastructure> <Launchpad Foundations:In Progress by mars> <https://launchpad.net/bugs/587886>
<gary_poster> reading, thanks
<gary_poster> mars, looks like shakeout from the subunit integration
<mars> yep
<mars> like I said, we need to address the root cause here after the fix goes through.  Otherwise changing the testrunner is always going to be risky and painful.
<gary_poster> mars..isn't the root cause the fact that we have inter-process communication, which is by its nature more fragile for this sort of thing?
<gary_poster> (and I don't see fixing that particular root cause)
<mars> gary_poster, yes, but the testrunner claims to have error handling mechanisms in place.  Also, it has no concept of a child layer timeout.
<mars> so its current mechanism is broken, and it is missing at least one possible safeguard.
<adeuring> can somebody give me a clue about this test failure: http://paste.ubuntu.com/446120/ ? it occurs for this branch: lp:~adeuring/launchpad/expectations-bug-556499-model
<mars> leonardr or gary_poster, ^ haven't we seen adeuring's error before?
<leonardr> checking
<leonardr> it looks like an error occured when generating the json representation of an object (which is stuffed into the HTML for use in ajax)
<mars> adeuring, yes, check this: https://dev.launchpad.net/Debugging#Getting past "LocationError: 'json'" in TAL template tracebacks
<adeuring> mars: thanks, will lok
<leonardr> yeah, you need to find the underlying error
<adeuring> leonardr: the main change in this branch is the adition of a new property that is exposed to the webservice
<adeuring> though it is defined via get/set methods
<leonardr> i'll take a look at the branch
<adeuring> leonardr: but... the error occurs also for r9391 of the branch
<gary_poster> mars, I don't see how a timeout could be coded reasonably for running arbitrary code that has no guarantee of generating anything on stdout or stderr in any particular amount of time.  If you do, cool, show me when you write it, cause I'll want to see what you have in mind (no need to explain now).
<gary_poster> On the other topic, an you point me to the "error handling mechanisms" claims?  If the subprocess is running arbitrary code (which it does), and it does not return an error code it seems to me like you're going to have trouble.  The subprocess wrapper code (the code running the arbitrary code) is the place where you have to code very defensively to make sure it catches just about everything.
<gary_poster> Is that what you mean you need to improve?
<gary_poster> mars: or do you just mean your last point?"Why is the parent testrunner process not failing the suite when it knows that the child process died due to a testrunner error? (I.e. the parent it got a 'Could not communicate with subprocess' flag)"
<mars> gary_poster, I think the code on either side of the subprocess gap needs to be examined to see why this exception did not cross the IPC channel
<mars> gary_poster, the latter, what you just quoted
<gary_poster> gotcha mars, thank you.
<mars> gary_poster, if it knows "my subprocess just died, uh oh", then why the heck didn't the suite error out?
<gary_poster> mars: ack, agree
<leonardr> adeuring: i don't know. can you find the underlying exception? i hope it will make everything clear
<adeuring> leonardr: yeah, I think mars gave me the right clue
<gary_poster> noodles775, bigjools: bug 590766 sounds like a high priority.  is it?  noodles775, are you going to take it since it appears to be a continuation of your work for bug 588288, or are you tossing this over?
<mup> Bug #590766: Apache log parser crashes out on the PPA logs <Launchpad Foundations:New> <https://launchpad.net/bugs/590766>
<mup> Bug #588288: log parser should not read entire file into memory <qa-needstesting> <Launchpad Foundations:Fix Committed by michael.nelson> <https://launchpad.net/bugs/588288>
<bigjools> gary_poster: I think it's happened because the offset it's using is out of sync with the file
<gary_poster> ah
<bigjools> gary_poster: and I'd like to toss it over if that's ok :)
<bigjools> the code should probably look for the nearest \n to the offset
<gary_poster> bigjools: and it's high priority?
<bigjools> gary_poster: fairly yes
<gary_poster> ok
<bigjools> thanks
<gary_poster> bigjools: sure.  do you have an easy way to dupe, perchance?  Otherwise I'll try to figure out a way to dupe with the clues you just gave.
<bigjools> gary_poster: it's reproducible on dogfood
<bigjools> but I suspect that's because the offset that's stored is out of whack for some reason
<bigjools> I'm happy to help test on DF for you
<bigjools> or probe for any more info you need
<gary_poster> bigjools: ok, thanks, yeah.  I'll probably write up a hack patch soon for you to try to see if we are on the right track before I pursue a full proper branch.
<bigjools> cool
<noodles775> Thanks gary_poster !
<gary_poster> :-) np
<wgrant> bigjools: I don't think being more robust and looking for a \n nearby is at all the right solution.
<wgrant> It should never happen.
<bigjools> do expand ...
<wgrant> Unless we have a bug, or somebody mangles the log files.
<bigjools> never != will never
<bigjools> and you hit on the right thing with your latter point
<wgrant> Yes, and when it does happen we have a problem, so we probably want to know that it exists.
<bigjools> it should not make the parser bail out permanently
<wgrant> Why would somebody be mangling log files?
<bigjools> PEBKAC
<gary_poster> I agree I'd like to know what the root cause is
<bigjools> never underestimate human fuckups
<wgrant> It only kills the rest of that file, right?
<bigjools> in my case I suspect it's because the dogfood database was restored from production
<gary_poster> Vaguely, I'd prefer to reread the file if we get out of sync, but I don't understand what is going on in this code well enough yet--the diff doesn't give me enough context.
<wgrant> We can't safely reread it.
<wgrant> We may have already put stuff into the DB from previous runs.
<gary_poster> I figured we would, but I had hoped we would have enough information around to know what we had already read.  That's obviously uninformed hope; perhaps it can't work.
<wgrant> We have the information: the offset.
<gary_poster> Which is broken.
<wgrant> But the problem here is the case where that information is wrong.
<wgrant> Right.
<gary_poster> So the offset is too fragile.  I thought we might have a datetime of the last read line somewhere.  If not, we could store it.
<gary_poster> bigjools: could you email me the specific file that is broken?  I'd like to experiment with it.
<bigjools> I can
<gary_poster> thank you
<bigjools> gary_poster: ummm it's 1.7G
<gary_poster> bigjools: ... :-/ devpad, and I
<gary_poster> '
<bigjools> I can copy it to.... devpad!
<gary_poster> I will tell you when I have it?
 * bigjools needs 2 more arms
<gary_poster> He would look funny then.
<nigelb> Is there plan for LP to have a "press foo to forward to debian" thingy?
<james_w> leonardr: you might be interested in https://edge.launchpad.net/soupmatchers see http://bazaar.launchpad.net/~soupmatchers-dev/soupmatchers/trunk/annotate/head:/README
<jml> nigelb, it's definitely within scope, but there aren't concrete plans to implement it right now.
<henninge> Can somebody tell me where we import the sortable tables from that we use in Launchpad?
<henninge> Are they LAZR-JS?
<mars> gary_poster or sidnei, Zope question: if I want to add a test to zope.testing for a specific corner case, do I have to use a doctest to do it?
<mars> gary_poster, sidnei, I would normally use a unit test for corner cases and test to increase code coverage.  The test in question is to ensure a bug is not present.
<sidnei> mars, no, you don't need to use a doctest. in fact, if you're willing to convert all doctests to unittests you would be very welcome :)
<mars> sidnei, alright, where in zope.testing would this unit test go?  "testrunner-ex" doesn't sound right
<mars> sidnei, oops, have to go, lunch is served, back in 1hr
<nigelb> jml: I'm trying to write something that links with reportbug
<nigelb> bug anyway, hooking with debian bts should be easier than most, since its easy to opena  new bug.
<nigelb> but its the searching that might be a bit challenging
<ScottK> reportbug has searching code and it's in Python ....
<nigelb> ScottK: oh, reportbug is python? if only I knew, I wouldn't be pulling my hair out last weekend
<ScottK> ;-)
<nigelb> ScottK: in that case its only trivial for me to add the querying LP part to reportbug as such as a new feature!
<nigelb> I shuold try doing that this weekend
<mars> sidnei, back
<sidnei> mars, so tests are registered in tests.py. i see they are pretty much all doctests. but that shouldn't prevent you from writing a unittest
<sidnei> mars, testrunner-ex contains sample tests used by the doctests to trigger failure conditions and edge cases apparenlty
<mars> sidnei, yes, it looks that way
<mars> sidnei, ok, I see the subunit tests block near the end of tests.py.  Looks like that is the place to put unit tests.
<sidnei> yeah
<mars> sidnei, should I create a new subdirectory just for unit tests? 'unit', or 'tests' or something?
<mars> sidnei, I am a bit hobbled here by my ignorance of zope project conventions
<sidnei> mars, uhm. instead of tests.py it should really be a tests directory there, and maybe a test_doctests.py inside. i think no-one bothered to fix that.
<sidnei> mars, maybe just add the test to tests.py
<mars> I'm not sure that is a good idea.  That file is large enough already.
<mars> Adding an unknown number of xUnit tests to it would just pollute the module.
<mars> sidnei, I could create a tests/ directory and put test_subunit.py inside it
<mars> it would be a partial refactoring
<sidnei> that sounds good too
<mars> ok
<mars> sidnei, ah, won't work: a 'tests' package directory would raise a name conflict with the existing 'tests' module
<sidnei> yeah, so i thought.
<mars> testtestrunner :P
<sidnei> mars, best way would be to move tests.py to tests/test_doctest.py as i suggested.
<mars> sidnei, and what about testrunner-ex and friends?
<sidnei> mars, as i said, that's sample code used by the doctests
<mars> leonardr, ping, just did a rf-get and build, am getting an AssertionError from the apidoc/index.html build step.  Known issue?
<leonardr> mars, i don't know about it. pastebin?
<jml> mars, btw, can I look at your patch for zope.testing when it's done?
<mars> jml, sure.  may be a bit, I have to craft a reproduction of the error
<jml> mars, that's ok. I'm on vacation so not in any kind of rush :)
<jml> (yes I suck at vacation)
<mars> lol
<mars> leonardr, here is the output of 'make clean && make' on a fresh branch of devel: http://pastebin.ubuntu.com/446264/
<leonardr> mars: hypothesis: someone landed a web service branch recently that broke this
<leonardr> i don't know why they were able to land it, since apidoc.txt should have failed
<leonardr> but that's the most likely explanation
<mars> leonardr, fake success mail from ec2?
<leonardr> "web service branch" = "branch that messes with or publishes something new on the web service"
<leonardr> yeah, i guess
<mars> leonardr, do you know how to fix it?
<mars> I don't :)
<leonardr> i know how to fix it on a high level, ie. 1) find the branch that caused the problem, 2) run the test, 3) fix the failure
<leonardr> but i have no idea what the actual problem is--it could be anything anywhere in the web service
<leonardr> mars: i'd start looking at rev 10953, adiroiban's POTemplate change
<mars> leonardr, we'll find out in 27 minutes :)
<mars> if buildbot is accurate
<leonardr> mars, i was able to make revision 10952
<mars> I don't trust the buildbot estimates any more.  Refreshing the page removed 2 minutes from the estimate since I last spoke of it, 20 minutes ago.
<bryceh> https://bugs.edge.launchpad.net/ubuntu is oopsing for bdmurray and I, is this a known issue?
<mars> bryceh, I was having similar problems with my branch listings page OOPSing earlier today.  There was mention of increased database loads due to some OAuth code.  gary_poster, could they be related?
<gary_poster> mars, no what stub was talking about was systemic, not specific to today or this week or this month.
<mars> gary_poster, ok
<bryceh> the error I see is:  Module lp.bugs.model.bugtarget, line 211, in getBugCounts
<bryceh> ', '.join(select_columns), ' AND '.join(conditions)))
<bryceh> QueryCanceledError('canceling statement due to statement timeout\n',)
<bryceh> (https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1619ED1527)
<leonardr> mars, binary search has revealed it's got to be either 10958 or 10959
<mars> leonardr, ah, have to love the binary bug search :)
<bdmurray> well it couldn't be 10959 ;-)
<mars> hehe
<maxb> jelmer: Did you follow up about that fretsonfire import or did jonobacon apparently delete it himself for another reason?
<leonardr> mars, bdmurray, it's 10959
<jelmer> maxb: I haven't followed up
<bdmurray> leonardr: can you provide some guidance for fixing it?
<maxb> ok, I'll write to him since he's deleted the old import and requested a new one
<jelmer> maxb: you can also msg him on irc, he's 'jono' on this network
<leonardr> bdmurray, first step is to find the underlying problem
<mars> leonardr, gary_poster, r10958 built fine for me.
<lifeless> my gyess
<lifeless> he couldn't find the edit-details button to fix it
<leonardr> bdmurray
<leonardr> this should be easy to fix
<leonardr> you define a new argument to searchTasks, which is a reference(schema=Interface)
<leonardr> you did what other, similar arguments did
<leonardr> but what you didn't see is that that value doesn't stay Interface forever
<leonardr> it only starts out as Interface to avoid a circular import
<leonardr> take a look at lib/canonical/launchpad/interfaces/_schema_circular_imports.py
<leonardr> you'll see # IHasBugs
<leonardr> and then a number of calls to patch_plain_parameter_type()
<kb9vqf> How can I set a custom OpenID URL?  Something in launchpad-lazr.conf?  Is there any documentation for that file?
<leonardr> each of those replaces the 'Interface' in one Reference(schema=Interface)
<leonardr> you need to add a call to patch_plain_parameter_type(IHasBugs, 'structural_subscriber', ???)
<leonardr> the question is, what goes in that ???? What type of object is structural_subscriber, really?
<leonardr> it looks like it's an IPerson
<leonardr> so you would add patch_plain_parameter_type(IHasBugs, 'structural_subscriber', IPerson)
<bdmurray> leonardr: right, it is.  and then would I just resubmit it to pqm?
<leonardr> yes
<leonardr> make sure that 'make' works after you make your change
<gary_poster> leonardr, bdmurray: awesome, thank you
<bdmurray> leonardr: yes, of course.
<leonardr> gary, np
<bdmurray> leonardr: thanks for the help
<leonardr> sure
<gary_poster> bryceh: did you make a bug for that?  If not, I will.
<gary_poster> bryceh: (the timeout on https://bugs.launchpad.net/ubuntu and https://bugs.edge.launchpad.net/ubuntu)
<bryceh> gary_poster, nope I didn't
<gary_poster> bryceh: on it
<maxb> Does anyone know why https://code.edge.launchpad.net/~chanwit/groovy/ck1 would not be showing a 'Branch metadata' section?
<mars> maxb, thumper might
<thumper> maxb: what do you mean?
<Lonniebiz> Some questions:  Does launch pad solve the same problems that bugzilla does?
<Lonniebiz> Can you install it on you own Ubuntu server, and make it work the same way launchpad.net works?
<Lonniebiz> Can a software development company use Launch Pad for project management?
<lifeless> yes
<lifeless> sorry, in order
<lifeless> no (solves more than)
<lifeless> yes you can install it yourself, if you have the time and willpower to admin it (its not .... lightweight); you *must* rebrand it if you do that.
<lifeless> Yes, you can use launchpad for some project management tasks, either hosted (launchpad.net, speak to bac for commercial subscription info), or on your own one (see my immediately previous answer)
<Lonniebiz> My company is looking for a Software Project Manangement solution.
<Lonniebiz> Been looking at Jira, but thought about how much I liked how Ubuntu allows you to submit bugs and stuff.
<Lonniebiz> We develop software that is commercial, but need something to track us through the Software Development Life Cycle.
<lifeless> sure
<lifeless> launchpad.net can host such things
<Lonniebiz> Think installing LaunchPad on our Ubuntu server would be too involved a solution. I'm a big Ubuntu fan myself....
<maxb> thumper: Normally a branch page shows a 'Branch metadata' section showing its format, and if its stacked
<maxb> But it's missing on that one branch
<Lonniebiz> I've read this: https://dev.launchpad.net/Getting
<Lonniebiz> After I complete those steps, will it be as simply as navigating to an internal ip such as http://localhost/LaunchPad   ?
<thumper> maxb: the branch hasn't been scanned since we added the metadata, so nothing to show
<maxb> oh, I see
<Lonniebiz> then just add a project through the web interface?  Is it that simple or much much more involved.
<maxb> It is MUCH MUCH more involved
<maxb> For a start, you *must* rebrand it
<maxb> And also learn lots about how the many components fit together, and how to run it in production, which isn't documented anywhere public
<Lonniebiz> Yeah, but at first, we'd be just using it internally; If we were to let customers see it, we'd surely take the step of rebranding it.
<maxb> To use it internally _legally_ you'd have to rebrand it first
<Lonniebiz> For now, we're just looking at it more as a development tool than a customer feedback tool.
<maxb> so says the licence
<Lonniebiz> oh, I see.
<maxb> (This annoys me too)
<gary_poster> Lonniebiz: FWIW, https://answers.edge.launchpad.net/launchpad/+faq/208
<Lonniebiz> Well, that doesn't seem hard, just change the logo right, and replace any word "launchpad" with "CompanyA", right?
<maxb> and design replacements for all of the many icons....
<gary_poster> (Again FWIW, I also think you are significantly underestimating the setup involved, as maxb suggested.)
<Lonniebiz> that could be laborious...
<rockstar> wgrant, ping
#launchpad-dev 2010-06-08
<krkhan> in testing layer, after creating a bugattachment with factory.makeBugAttachment, i get 404 whenever i try to access the attachment's url
<krkhan> that is, doing just a = self.factory.makeBugAttachment and then accessing a.data.read() results in a lookuperror
<kb9vqf> Anyone have recommendations for an OpenID server that will work with a local installation of Launchpad?
<thumper> krkhan: that is because the attachment is stored in the librarian
<thumper> krkhan: which is a different process
<thumper> krkhan: so the local transaction needs to be committed for the librarian to see it
<thumper> kb9vqf: I just use the local test openid server
 * thumper afk for lunch
<spm> configs question: we have a script: scripts/process-accepted.py which is oops'ing. and then failing to write said oops as it's going to the wrong place; which is then aborting the shell script it's running from. Q, how do I find which config stanza I need to fix for that specific script *only*? I've found the master it's inheriting from ftpmaster/launchpad-lazr.conf but that controls a bunch of semi-related things I don't want to break. ???
<wgrant> rockstar: Hi.
<krkhan> thumper: how do i commit the local transaction?
<krkhan> thumper: nvm. transaction.commit. thanks a bunch for the quick help :)
<rockstar> wgrant, can you take a look at this: https://code.edge.launchpad.net/~rockstar/launchpad/fix-buildds/+merge/26990
<wgrant> rockstar: Good description, good patch.
<rockstar> wgrant, yeah, I awoke to find shit all over me this morning.
<rockstar> :)
<wgrant> Sorry -- I would have submitted a branch myself, but exams are eating lots of time at the moment.
<wgrant> Ah, so it was recipe builds that killed everything over the weekend?
<rockstar> wgrant, yea, but it's not your job..  :)
<rockstar> wgrant, yup, we're assuming this is why.
<poolie> hi all
<rockstar> wgrant, so do you feel like this fixes the bug you reported (I'll test as well, but you know this better than I do)
<wgrant> rockstar: I believe it does. I'll fire buildd-manager up and get a build through.
<wgrant> Although fixing this right now may not be a good idea.
<wgrant> Since it means the master will start crashing instead of the slave.
<rockstar> wgrant, yeah, I tried to do it locally, and my launchpad chroot shit the bed twice, so I tried to test on dogfood, but I don't really have access to it.
<wgrant> Bug 587113
<mup> Bug #587113: BuildBase result handling broken <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/587113>
<wgrant> OK, I'll test locally.
<poolie> that's vivid :)
<rockstar> wgrant, why will the master start crashing?
<wgrant> rockstar: The big model refactor last master broke the master's handling of completed recipe builds.
<wgrant> (the bug I referenced above)
<wgrant> But we can at least test that the slave works.
<rockstar> wgrant, wait, so this isn't going to be able to be cherry picked to production?
<wgrant> rockstar: Unless we get noodles' master fix in quickly too, no.
<wgrant> But that fix should't be too big.
<rockstar> wgrant, hm, I'll chat with noodles about that tomorrow then.
<wgrant> For now, can't you just disable the recipe UI?
<rockstar> wgrant, that's what we did.
<wgrant> Ah, good.
<rockstar> wgrant, but we would like to be testing recipes through edge this whole cycle.
<rockstar> We've found that dogfood is a terrible place to be testing recipes
<wgrant> Hmmm, production appservers need a CP before that can happen.
<wgrant> Bug #587110
<mup> Bug #587110: Need fmt:link for SourcePackageRecipeBuild <recipe> <ui> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/587110>
<wgrant> What's wrong with dogfood?
<wgrant> Besides being dogfood?
<rockstar> wgrant, can you compile this into a mail to the list.
<rockstar> wgrant, we ran recipes through dogfood and it went well.  As soon as they got into edge, they just became a pain.
<wgrant> rockstar: Well, all of this broke during week 3.
<wgrant> Every one of these bugs.
<wgrant> rockstar: As expected, the recipe build works fine with your branch merged, and then the master has a fit.
<wgrant> So your branch is good.
<rockstar> wgrant, okay, so there are three needed cherry picks before we can re-enable recipes, correct?
<wgrant> 1) Your buildd branch to all of the buildds -> lamont
<wgrant> 2) noodles master fix, which is yet to be written -> cesium
<wgrant> 3) The SPRB fmt:link -> production appservers
<rockstar> wgrant, okay, I'm compiling a this in an email to the list then.
<wgrant> Thanks.
<rockstar> wgrant, no, thank you for the help.
<wgrant> I was able to get them going 1.5 weeks ago with those three changes.
<wgrant> And it probably hasn't broken much since then.
<wgrant> (although my master fix was a hack)
<wgrant> rockstar: Is it known that the distroseries are in an arbitrary order on +request-builds?
<rockstar> wgrant, for recipes?
<wgrant> rockstar: Yes.
<rockstar> wgrant, I think abentley had some issue with ordering by distroseries, but I don't remember what it was.
<rockstar> But there was method to the madness.
<wgrant> I really wish dogfood was less omgslow.
<rockstar> wgrant, and that the buildmanager was actually running?
<wgrant> rockstar: Well, I don't care much about that.
<wgrant> I was hoping to log in and check if the distroseries were in any obvious order.
<wgrant> But it turns out you can't log in either.
<wgrant> And editing a recipe would probably time out, too...
<rockstar> wgrant, yeah, dogfood is Soyuz's baby...
<jml> james_w, you are maybe asleep now, but http://mumak.net:8080/job/txrestfulclient/
<spm> hey jml, how's the big M doing?
<thumper> spm: what? McDonalds?
<rockstar> wgrant, technically bug# 587110 doesn't need to be cherry picked, because the UI is not enabled on production.
<mup> Bug #587110: Need fmt:link for SourcePackageRecipeBuild <recipe> <ui> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/587110>
<spm> thumper: heh, close. Tho I believe MontrÃ©al is more accurate. ??
 * rockstar shudders at McDonald's...
<wgrant> rockstar: Erm, well.
<wgrant>  /builders will start crashing if an SPRB gets dispatched.
<rockstar> wgrant, ?
<wgrant> So the index views themselves are disabled too?
<rockstar> wgrant, well, the views are available, but in the words of abentley, "if you url hack, you get to keep both pieces"
<rockstar> There are no links to those views though.
<wgrant> OK. Maybe we should cherrypick a special fmt:link that doesn't actually generate a link.
<rockstar> wgrant, sigh, I hadn't thought about that.
<rockstar> wgrant, please reply to the my email about that.
<wgrant> rockstar: Your third point has a hanging "This does"
<rockstar> wgrant, gimme a break, it's 2000 here.  :)
<wgrant> Heh.
<wgrant> Was there anything important omitted?
<wgrant> Replied, anyway.
<jml> spm, Montreal is great
<lifeless> jml: are you on leave now ?
<noodles775> Hi ppl
<adeuring> good morning
<bigjools> hello world
<wgrant> Morning.
<mwhudson_> morning
<bigjools> hi guys
<bigjools> how's the service station mwhudson? :)
<mwhudson> bigjools: terrible
<mwhudson> the fire alarm went off at about 01:30 last night
<bigjools> I looked on google maps and it's in. the. middle. of. nowhere!
<mwhudson> yeah
<mwhudson> you thought dolce la hulpe was isolated?
<wgrant> It was in a forest!
<wgrant> Is this even worse?
<mwhudson> yes
<wgrant> HOW!?
<mwhudson> wgrant: it's a travel hotel, with a mcdonalds next door and about 5km from anywhere
<wgrant> Oh.
<wgrant> Awesome.
<cody-somerville> There is a bloody handprint above my bed.
<wgrant> ... that is concerning.
<bigjools> cody-somerville: lol
<deryck> Morning, folks.
<noodles775> When using codehosting locally, I'm trying to push a branch for a user I've created, but it's trying to authenticate as sabdfl... can someone point me in the right direction? http://pastebin.ubuntu.com/446558/
<wgrant> noodles775: You have an old ~/.ssh/config.
<wgrant> rf-setup used to put an extra stanza in there.
<noodles775> Thanks wgrant.
<bilalakhtar> Hi there, people. Can anyone tell me what does this line mean in https://dev.launchpad.net/Running/Schroot "Don't rm -rf /chroot/karmic-lp, because this will descend into /chroot/karmic-lp/home and delete your actual home directory!"?
<bilalakhtar> Sorry I am a newbie when i comes to chroots
<bilalakhtar> sorry have to go
<noodles775> wgrant: still around? I've got failed SPRecipeBuilds going through properly (ie. log gets appended, etc.). The log shows they're failing with "bzr: ERROR: no such option: --append-version". I assume I need to make sure the buildd has a newer version of bzr-builder?
<wgrant> noodles775: Right, you need to steal a new bzr-builder from production.
<wgrant> Or add the right sources.list entries to the config option.
<wgrant> Either steal from the prod configs, or download and upload from https://edge.launchpad.net/~launchpad/+archive/bzr-builder-dev
<noodles775> Yep, doing so now. Thanks.
<bilalakhtar> Hi there, noodles775 is it possible to run lp on a computer already running LAMP? (apache2-mpm-prefork)
<bilalakhtar> Rocketfuel-setup runs apt-get install apache2-mpm-worker. This removes php
<noodles775> bilalakhtar: I've not tried.
<bilalakhtar> noodles775: What apache module does launchpad use? mod-python or cgi?
<wgrant> Neither.
<wgrant> It's mostly proxied.
<bilalakhtar> wgrant: proxied?
<wgrant> It mostly uses mod_proxy.
<wgrant> Apache does not run the actual application.
<wgrant> prefork should be fine, if you are PHP-inclined.
<bilalakhtar> wgrant: oh. Thanks. I will modify the line in rocketfuel-setup so that it tries to install prefork, which is actually installled. thanks
 * bigjools finds the problem why PPAs are not getting deleted and groans
<bilalakhtar> Hi people, I am confused aboout which branch should I download to begin hacking. db-devel or devel? I just want to make a minor change (no db change). Acoording to the wiki, I should work on devel, but it is said "db-devel is default stacking branch". What should I do?
<beuno> bilalakhtar, go for devel
<bilalakhtar> beuno: And stack on db-devel?
<beuno> bilalakhtar, yes
<bilalakhtar> beuno: then merge with devel?
<beuno> don't worry about the stacking
<beuno> it's not important
<beuno> use devel, propose merging into devel
<beuno> ignore db-devel
<bilalakhtar> beuno: Last question: I am not using rocketfuel. I am manually downloading db-devel right now. I installed apache and lp-developer-deps. Now, what should I do about locations.conf?
<beuno> I think locations.conf is a convenience to push branches
<bilalakhtar> beuno: What should I set it to, if I want to take advantage of the "convenience"?
<lifeless> nothing
<lifeless> you're worrying about a bunch of automatic details
<lifeless> don't
<lifeless> just follow the wiki :)
<bilalakhtar> thanks, beuno and lifeless
<bilalakhtar> beuno and lifeless: What is bzr pqm-submit? Is it necessary to run that as well when proposing merge? Or proposing via lp web UI is enough?
<lifeless> bilalakhtar: just run bzr lp-propose
<lifeless> pqm-submit is used by folk with commit-to-devel privileges
<bilalakhtar> lifeless: ahha, a new plugin. Is it the same as proposing via web?
<lifeless> yes
<lifeless> not a new plugin, its built in
<bilalakhtar> lifeless: thanks
 * bilalakhtar is bumming around with the code :)
<bilalakhtar> beuno: running bzr branch lp:launchpad/devel is stacking on the devel branch. is it fine?
<bilalakhtar> hehe, most of the people here are lp-ers
<bilalakhtar> I mean members of ~launchpad team
<maxb> <bilalakhtar> running bzr branch lp:launchpad/devel is stacking on the devel branch
<maxb> That is meaningless/impossible. The act of branching locally can't stack on a remote launchpad branch
<bilalakhtar> maxb: I cannot run link-external-sourcecode. The problem is that the sourcecode folder is empty. What to do?
<maxb> The sourcecode folder should not be empty if you've successfully completed rocketfuel-setup
<bilalakhtar> maxb: I didn't use rocketfuel only.
<maxb> If you do not, then you are responsible for reading it through carefully and doing what it would have done
<bilalakhtar> maxb: ok, now I will use it. Many probs are coming thanks.
<maxb> I do not run rocketfuel either
<maxb> But I do treat it as a reference document
<deryck> So what's the rule for using memcached in tal and testing?
<deryck> Cache and move on, trusting the memached testing itself?
<deryck> gary_poster, have any thoughts there ^^ ?
<gary_poster> deryck: yes, a few thoughts.  So, first of all, there's a bug on making testing easier.  I could find it; the gist is that we should be able to verify that memcache is working, and there's an idea on what that spelling might be.
<gary_poster> That probably should be pretty easy to do but we haven't gotten around to it yet
<deryck> ok, np.  I do understand.
<gary_poster> So, looking into that is an option.  However, meanwhile, depending on what you are doing, there may be some other approaches for testing this second.
<gary_poster> In particular, curtis wanted to show that one snippet (with memcache sharing characteristic X) was used for one situation, and another was used in another (with characteristic Y).
<gary_poster> He did that with simple comments in the code that actually didn't have too much to do with memcache except by convention.
<deryck> gary_poster, yeah, I saw Curtis' test but it seemed to be just testing html comment.  but I guess that does ensure that the right caching is in effect.
<deryck> right
<deryck> I guess that would work for me.
<deryck> gary_poster, and normal page tests are run outside cache, right?  So adding caching has no effect on those tests?
<gary_poster> OK.  Let me find the bug for improving memcache testing, and then maybe you can add your example there so we can change it when we add the better testing story.  Normal page tests are run outside cache: actually, I think memcached is available and running
<adiroiban> noodles775: do you have time to discuss an Firefox/HTML-accesskey issue affecting bug 359180?
<gary_poster> deryck: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/586466 FWIW
<mup> Bug #586466: Testing tal cache states could be easier <Launchpad Foundations:Triaged by stub> <https://launchpad.net/bugs/586466>
<noodles775> adiroiban: in 0.5hr or so?
<noodles775> Will you still be around?
<adiroiban> noodles775: yep. we can also do it tomorrow ...no hurry
<adiroiban> I just wanted to schedule it
<stub> pagetests use memcache
<deryck> gary_poster, you say memcached is available and running.  Does it affect tests then?  i.e. if I fetch a page that is cached, do some work, and then want to verify a cached section, should the test fail?
<deryck> stub, ^^
<noodles775> adiroiban: today is good, I'm just finishing something off.
<gary_poster> deryck: AIUI, yes, it affects tests.  That is certainly the intent.
<deryck> ah, ok.
<stub> memcached affects tests. You might need to clear the cache if you need changes to be visible - there is a helper on MemcacheLayer I think to do that easily
<deryck> that works perfectly for me and my concerns then.  Thanks, gary_poster and stub.  And many thanks for this work from you guys, too! :-)
<gary_poster> cool :-)
 * deryck plans to cache liberally over the next couple months
<stub> deryck: And use the slave db liberally too.
<stub> deryck: I'd like to see all bug searches hitting the slave instead of the master - they are pretty heavy weight.
<deryck> stub, right.  That's a good point.
 * deryck makes reminder note to discuss with team
<stub> lib/canonical/launchpad/doc/db-policy.txt goes over how to install a db policy to make the slave a default for a section of code, or just ISlaveStore(Bug).find(...) to be explicit.
<bigjools> db-policy.txt ROCKS
<deryck> note made
<bigjools> lifeless: is lp-propose a new bzr command?  I don't have it.
<jml> lifeless, I am.
<noodles775> adiroiban: ok, ready when you are.
<adiroiban> noodles775: ok. I start working on keyboard shortcuts for Launchpad translations page
<adiroiban> and I went for Shift+Alt+KEY
<adiroiban> as they are also used for HTML Access key
<adiroiban> and thinking that Web browser will not overlap with them
<adiroiban> I have implemented them using YUI3 key events and Chromium / Webkit family is fine
<adiroiban> but in Firefox, if there is no HTML Access key on the page, Shift+Alt+Key will behave as Alt+Key
<noodles775> Great!
<noodles775> (except for the last bit :) ).
<adiroiban> and instead of performing the YUI3 event
<adiroiban> it will pop up a menu
<adiroiban> as I workaround, I found out that if I add an empty <a accesskey=F> in the page
<adiroiban> Firefox will not popup the menu
<adiroiban> and will trigger the YUI3 event
<adiroiban> and I would like to ask you if this is an acceptable workaround
<adiroiban> or there are other side effect for this solution
<adiroiban> or do you have other ideas for implementing it
<noodles775> adiroiban: could it not be a feature if it was for a skip to content-type link?
<adiroiban> noodles775: I lost the conversation... what is a feature?
<noodles775> Have I understood correctly: you're asking if it would be acceptable to add a default access key to all pages (ie. the base template) so that FF won't pop up the window?
<adiroiban> yes. that what I was asking.
<adiroiban> but it will requre an âblindâ access key for each keyboard shortcut
<noodles775> So, couldn't that be combined with providing a "skip-to-content" link with an access key (eg. http://diveintoaccessibility.org/day_11_skipping_over_navigation_links.html )
<noodles775> OK, I'm not sure what you mean by a blind access key.
<adiroiban> a blind accesskey is just a tag like <a accesskey=KEY/>
<adiroiban> it will not be a proper anchor
<adiroiban> as it does not have a name
<noodles775> But why couldn't it be a proper anchor if it was providing useful functionality (but not displayed) as in the above link?
<adiroiban> because the YUI3 event is doing more than just focusing a link
<adiroiban> adding an Skip to content accesskey will not solve this problem
<adiroiban> as we need an accesskey for each YUI3 key event
<adiroiban> and the accesskey should not have an attached ID or name
<adiroiban> noodles775: to demonstrate the behaviour, go to https://translations.edge.launchpad.net/ubuntu/lucid/+source/ubiquity/+pots/ubiquity-debconf/de/+translate using Firefox
 * noodles775 opens FF
<adiroiban> and while the focus is on the first translation field (it should be auto focused)
<adiroiban> press Shift+Alt+F (if you are using FF in English)
<adiroiban> if you are repeating these steps in Chromium, the Search field will be focused
<adiroiban> but in Firefox it opens the File menu
<noodles775> (as does simply ALT-F)
<adiroiban> yes
<noodles775> Which is the first translation field? (It didn't seem to auto focus...)
<noodles775> nm... I wasn't logged in with FF, so no fields :)
<adiroiban> ah
<adiroiban> yes
<adiroiban> you need to log in
<noodles775> OK, and you say that can be avoided specifically by adding *one* dummy <a accesskey=KEY/> to the template?
<adiroiban> one dummy for *each* YUI3 event
<noodles775> ah... OK, now it's clearer.
<noodles775> hrm.
<adiroiban> do you know other ways for solving this problem
<noodles775> Nope, but I'm just reading atm.
<adiroiban> Ctrl+Key, Alt+Key and Shift+Ctrl events are already well established in browsers menus
<adiroiban> and I think that Ctrl+Alt is used by the window managers
<noodles775> adiroiban: ok, another point where I'm confused: why not add a valid anchor with a valid access key for each?
<adiroiban> HTML accees key actions can only focus or activate (depending on the browser) a link
<adiroiban> in the translations page we have Shift+Alt+C
<bilalakhtar> People, How bug is the launchpad branch? I have been downloading it from an hour. Its more than 100MB by now
<adiroiban> which triggers a javascript function
<noodles775> adiroiban: yes, but I was hoping that the YUI code would override that if present?
<bilalakhtar> *big
<adiroiban> also, for some access key there is no associated anchor
<noodles775> But ok, it makes more sense now, and no I don't know of a better option (nor can I find one). If you add the empty ones to your template only I wouldn't see it as an issue. When you've got the MP, please ping me with it, as I'd be keen to play with it a bit.
<adiroiban> noodles775: another option is overwriting Ctrl+KEY
<adiroiban> I have tested with Chromium and Firefox
<adiroiban> and it looks like Ctrl+Key events are first passed to javascript and only if not handled there
<adiroiban> they are triggering menu actions
<noodles775> aha, seems quite standard (at least, gmail keyboard shortcuts use it)
<adiroiban> yes... but user will no longer have access to their shortcuts
<noodles775> Is there any drawback to Ctrl+Key? Why weren't they the original option?
<adiroiban> like Ctrl+C
 * noodles775 reads bug again.
<adiroiban> noodles775: list of keyboard shortcuts in other web apps http://mashable.com/2007/06/29/keyboard-shortcuts/
<adiroiban> noodles775: the main reason for using Shift+Alt was to avoid overlapping with other key shortcuts
<adiroiban> and Ctrl+Key are widely usind in browsers
<adiroiban> also afaik IE 7.0 will not allow the overiding of Ctrl+P, Ctrl+O and Ctrl+F .... maybe there are more restrictions
<noodles775> adiroiban: OK, I'm tending towards keeping the keys you've got and updating just your template so it also works in FF (adding real accesskeys where possible, but blank otherwise). As this is the first time LP has had accesskeys (I can't see any other egs?), it's probably worth a discussion on the dev list... there may be people on there in a better position to make a more permanent recommendation.
<gary_poster> deryck, sinzui: in a fit of pique, I tried for a fix to bug 586466, which gives a test convenience for memcached bits.  I'm running it through ec2 test now to see if my change has any nasty side effects, but otherwise it seemed to work.
<gary_poster> Once I get that to pass, I'm going to ask Curtis to review, for several reasons (making sure I'm doing something reasonable with my lpconfig check, for instance, or if he thinks I should be more defensive).
<gary_poster> However, as a sort of pre-impl or mid-impl or something, would be happy to get any feedback from you two on the change.  Essentially, if you go to http://bazaar.launchpad.net/~gary/launchpad/bug586466/revision/10964 and look at milestone-views.txt, you will see that, if you are in test or dev modes, the memcached integration will now add comments describing the start (and end) of the cache hit.
<gary_poster> Does that look good to you two? 
<mup> Bug #586466: Testing tal cache states could be easier <Launchpad Foundations:Triaged by stub> <https://launchpad.net/bugs/586466>
<adiroiban> noodles775: I think this is the first place to use accesskeys. I will also send an email to dev list. Thanks! I will let you know when the MP is ready
<deryck> gary_poster, oh, wow.  JFDI FTW! :-)  looking at your rev from above now...
<gary_poster> :-)
<deryck> gary_poster, I like that a lot.  That works for my concerns, too.  It's more explict about whether you've hit cache or not, which is what I wanted.
<gary_poster> Great, deryck.  Thank you for looking at it.
<deryck> np
<sinzui> gary_poster, deryck: remove the end comment or put them before the call. I personally do not think they are needed...it know what purge does. r=me, as deryck said jfdi
<gary_poster> sinzui: ack, cool, thank you
<sinzui> gary_poster, deryck: My initial landing of cached content was disappointing. I thought my second would also be disappointing, but most of the problems are now gone from the oopses. I will add this tactic to person and teams next
<gary_poster> great
<deryck> yes, I appreciate your initial work, sinzui.  Makes my use easier.
<deryck> and I'm on a memcached onslaught now
<sinzui> take a look at the milestone bug task listings. I use two different cache times (which required a macro for the content) to handle stable verses unstable information. eg, cache old comments 6 hours, but cache new ones 10 minutes (so losas can delete them)
<noodles775> abentley: I'll include that change in the branch that I'm landing.
<abentley> noodles775, tx
<noodles775> Night all.
<adiroiban> hi. it looks like LP API (https://launchpad.net/+apidoc/devel.html) is not updated together with edge. Is there a way where I can find the revision of the deployed LP API code
<jml> adiroiban, https://edge.launchpad.net/+apidoc/devel.html ?
<adiroiban> jml: true. stupid me :)
<jml> is there a stable ppa for launchpadlib?
<dobey> hey guys
<dobey> what are the metrics for determining what reviews show up on my launchpad.net/~me/+activereviews page?
<beuno> dobey, I think it's "reviews that you can do, or reviews that you have been asked to do"
<beuno> the "you can do" part inherits from your teams
<jelmer> unfortunately it doesn't include reviews for teams that you are in
<beuno> but this is totally out of memory
<beuno> ah
<beuno> ok
<beuno> so that was the plan
<dobey> jelmer: didn't it used to?
<jelmer> dobey: not that I remember, but I have only been using the +activereviews page since bzr switched to using launchpad for reviews.
<dobey> ooh. no more bundle buggy?
<jelmer> no, we moved away to launchpad quite a while ago (two years?)
<dobey> hmm
<beuno> jelmer, a bit under 2 years, when we actually made code reviews usable  :)
 * beuno gets nostalgic
<dobey> it's definitely less than 2 years, becuase i haven't been at canonical that long yet, and the branches i submitted to bzr had to go to bundle buggy :)
<jelmer> time flies, just realized I've been hacking on bzr for almost 5 years now
<dobey> heh
<mars> gary_poster, want to switch the Windmill test suite back on?  https://code.edge.launchpad.net/~mars/launchpad/re-enable-windmill-ec2-suite/+merge/27077
<gary_poster> mars, sure, once LP loads :-P
<jml> leonardr, is it deliberate that NamedOperation.__call__ has a variable named "request" that is probably supposed to be a ResponseDefinition instance? (I'm saying that because get_representation_definition is called on it)
<leonardr> jml: i think it's probably a RequestDefinition that _contains_ a ResponseDefinition
<jml> leonardr, RequestDefinition doesn't have a get_representation_definition method though.
<jml> leonardr, it has a representation_definition method :)
<kb9vqf> Does Launchpad have something like an admin console where projects and users can be deleted by an administrator?
<kb9vqf> This is for a local installation of course ;-)
<jml> leonardr, oh, never mind. I'm pulling from an old trunk.
<jml> leonardr, sorry for the noise.
<leonardr> jml, np
<krkhan> for local launchpad, what service_root should i give to the login_with method of launchpadlib?
<jml> leonardr, ok, now I have another question :) how come https://bugs.edge.launchpad.net/wadllib/+bug/274074 is marked as fix released, but https://code.edge.launchpad.net/~leonardr/wadllib/fix-bind-to-anon-collection/+merge/24613 isn't merged into wadllib trunk?
<mup> Bug #274074: Missing total_size on collections returned by named operations <api> <wadllib:Fix Released by leonardr> <https://launchpad.net/bugs/274074>
<leonardr> jml, most likely the code is there but the branch wasn't marked merged for unknown reasons
<leonardr> i'm checking
<jml> leonardr, thank you
<jml> leonardr, fwiw, when I merge the branch into trunk, it changes stuff and makes the failing test that triggered james_w's initial report pass
<mars> jml, btw, I have that patch now for zope.testing.  I can push the branch, see if you think it's sane.
<jml> mars, thanks.
<mars> jml, lp:~mars/zope.testing/fix-subunit-utf8-traceback-reporting
<jml> mars, ta. is that related to ec2 land being broken?
<mars> jml, that is the cause.  It isn't just ec2 land, it causes the test suite to outright forget previous test failures, so it ends up lying.
 * jml nods
<Ursinha> sinzui, hi
<sinzui> Hi Ursinha
<Ursinha> sinzui, I see you landed something that was a testfix and a fix for three bugs at the same time, right?
<Ursinha> r10951 on devel
<Ursinha> sinzui, problem is tagging script ignores testfixes
<sinzui> Ursinha, understood
<sinzui> I also had no idea that my testfix would fix a bug I did not know was reported
<mars> jml, I don't think it is a stellar fix, but it works, and should be forward-compatible.
<jml> mars, that branch wasn't made from lp:zope.testing :)
<Ursinha> sinzui, that was unexpected
<mars> jml, ?  Says "Created new stacked branch referring to /~ztk-steering-group/zope.testing/trunk"
<jml> mars, it'll say that regardless of what you push up.
<Ursinha> sinzui, I wonder if that could happen too often that would be worth changing the script's behaviour
<mars> jml, looking at the recent revisions on https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting, it still looks right?  Unless lp:zope.testing uses something different?
<jml> mars, the fix itself looks good to me (and damn, I wish I felt confident enough to use unit tests when originally contributing the zope support)
<leonardr> jml: looks like the branch i thought was lp:wadllib was just a random branch
<jml> leonardr, heh
<mars> jml, sidnei said it was OK ;)
<jml> leonardr, the ~launchpad-pqm owned one?
<sinzui> Ursinha, My actions were extraordinary. I do not think it will happen often
<Ursinha> sinzui, right
<Ursinha> thanks
<sinzui> Just knowing is enough for me to ensure they get updated myhand
<jml> mars, yeah, there shouldn't be a problem.
<leonardr> jml: no, it was something in ~leonardr/wadllib
<sinzui> by hand
<mars> jml, and lp:zope.testing points to https://code.edge.launchpad.net/~ztk-steering-group/zope.testing/trunk - still OK I would think
<jml> mars, yeah, it's ok, it's just that you can't merge one branch into the other with bzr
<jml> mars, which isn't a big deal, because it'll be applied as a text patch in svn anyway :)
<mars> jml, oh, you mean I didn't use svn!  Right.  Nope, I figured lifeless did it, so it must get through the contributor process somehow
<jml> mars, no, I just mean that when I try to merge your branch into my local copy of lp:zope.testing, bzr tells me there's no common ancestors.
<leonardr> jml, should be fixed now
<jml> ahh
<mars> jml, ok, that's weird
<jml> mars, lp:zope.testing has changed since I last pulled.
<jml> mars, my bad
<mars> no problem
<jml> leonardr, thanks! was the latest wadllib release based on the actual trunk?
<leonardr> the latest wadllib release was based on my oddball branch. but since i think i'm the only one who's ever modified wadllib, it should contain all the latest code
<jml> leonardr, good to know. Also, do you know if there's a stable PPA?
<leonardr> jml: not to my knowledge
<jml> leonardr, thanks.
<kb9vqf> Does Launchpad have something like an admin console where projects and users can be deleted by an administrator?
<kb9vqf> This is for a local installation of course ;-)
<mars> kb9vqf, log in as an administrator, visit the project or user page and hit the "Administer Foo" link in the upper-right.  There is no central page for doing so.
<kb9vqf> OK, thanks
 * kb9vqf is still learning...
<kb9vqf> second question: How do I promote a standard user to administrator status?
<krkhan> i have exported a method for Bug in lp.bugs.model.bug, it is accessible from lp.bugs.model.bug.Bug but it doesn't show up in lp_operations of the lazr Entry. any ideas what i'm doing wrong?
<krkhan> leonardr: ping
<leonardr> krkhan, does it show up in /+apidoc/devel.html in your local launchpad?
<leonardr> how did you export it? typically exporting happens on the interface level
<leonardr> if you give me a link to your branch i can take al ook
<krkhan> leonardr: i exported it this way: http://bazaar.launchpad.net/~inspirated/launchpad/bug-findAttachment/revision/10937
<krkhan> let me check apidoc
<krkhan> leonardr: no. it doesn't show up in apidoc
<leonardr> ok, so it's not published in the wadl, so lazr.restful doesn't really think it's published
<krkhan> how do i publish it in wadl?
<leonardr> once lazr.restful recognizes what you're trying to do, it will show up in the wadl, and launchpadlib and apidoc will pick it up from there
<krkhan> that's what i'm stuck at :-( why isn't lazr.restful picking up what i'm trying to do. i've tried reading implementation of other read operations which do show up and i don't see anything different
<leonardr> it also looks ok to me
<mars> sidnei, ping
<sidnei> mars, aye
<mars> Hi sidnei, just wondering if there is anything I need to do in the way of copyright assignment before or after submitting code to zope.testing?
<sidnei> mars, oh, good one. there's a contributor agreement that you have to sign to get commit access. i could checkin the patch for you though, since i signed the contributor agreement.
<mars> sidnei, well, the patch is in bzr, so I guess you would have to land it for me anyway :)
<sidnei> true
<mars> sidnei, ok, I'll prep a merge proposal then
<leonardr> krkhan, afaict an adapter is being generated for your new method
<leonardr> i put a breakpoint in the lazr.restful egg, in declarations.py#generate_operation_adapter
<leonardr>    if "Attachments" in method.__name__:
<leonardr>     import pdb; pdb.set_trace()
<leonardr> you can try it yourself
<leonardr> if you don't make any progress i'll come back to it tomorrow
<jkakar> leonardr: Thanks for looking at my fake-launchpad branch!
<jkakar> leonardr: Your comments look very sensible.  I need to digest them a bit, but will respond tomorrow.
<mars> sidnei, https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting/+merge/27086
<jkakar> leonardr: Also, totally agree that this should ideally live in restfulclient and also that FakeResource is doing too much...
<jkakar> leonardr: I was basically discovering the way the WADL and launchpadlib itself works as I went, so I'm not surprised that someone with clue about them see that the design I've come up with has problems.  Thanks for explaining them, I'm happy to work on fixes based on your suggestions.
<mars> jml, you may be interested in this: https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting/+merge/27086
<krkhan> leonardr: indeed, both canBeNominatedFor (example read operation that works) and findAttachments pass through generate_operation_adapter. so lazr.restful is seeing the method too but it still isn't getting exported. anyways, will ask you tomorrow then
<thumper> morning
<mars> morning thumper
<sinzui> bac, are you really online?
<bac> sinzui: i am now
<sinzui> bac: I think there are 4 or more bugs about bug field descriptions and documentation. I think we should fix these this week in a single branch
<bac> sinzui: sounds good
<sinzui> I can find the bugs, but I think it is best we commit to reviewing all the fields so that they are consistent. I think doing this will help us define what we intend to do with the new forms
<krkhan> leonardr: fwiw, i'm also noticing that changing the method names in IBug interface doesn't affect anything in the apidoc
<jml> what's bugtask.date_left_new mean?
#launchpad-dev 2010-06-09
<james_w> the date the bug first/last (?) left the New state I believe
<james_w> I think it was something that bryceh requested?
<bryceh> did I?
<james_w> no then :-)
<bryceh> well maybe, I've got an absolutely horrible memory
<james_w> I thought it was something you wanted to use in your scripts
<james_w> maybe Brian for bug gravity?
<bryceh> yeah could have been bdmurray
<james_w> hi bryceh, btw. Will we have the pleasure of your company at the platform sprint this time around?
<bryceh> hi james_w, actually no I'll be going to the launchpad sprint instead this go-around
<bryceh> but raof is well up to speed on anything X related you need help with
<james_w> bryceh: of course, but we don't just like you for your X knowledge :-)
<bryceh> how uncommon!
<james_w> there's your inkscape knowledge too... ;-)
<bdmurray> jml: I'm pretty sure its the first time it left the new state
<bryceh> sadly atrophied
<bryceh> james_w, but lately I have been doing some low level cairo experimentation for visualizing bug tasks that I'd like to show you some day
<james_w> thanks for the tip about searching for 'widgets' on openclipart.org, I'll try and make use of that sometime
<james_w> bryceh: that sounds sweet
<bryceh> yeah there's not too much there yet, but I would bet that one or two people posting some additional widgets there could enable inkscape for mockups pretty quickly
<bryceh> (further enable)
<leonardr> krkhan: the apidoc is generated once, you should see changes if you 'make clean'
<bdmurray> what creates _pythonpath.py?
<thumper> make build?
<thumper> gary_poster: still around?/
 * thumper is running make schema and cpu is spinning, but no apparent progress
<gary_poster> thumper, bdmurrary, bin/buildout does.  make compile should generate that along with a bunch of other stuff
<thumper> running buildout
<gary_poster> will run away in 60 seconds :-)
<bdmurray> when running make I'm getting the following error
<bdmurray> src/zope/security/_proxy.c:19:20: error: Python.h: No such file or directory
<bdmurray> In file included from src/zope/security/_proxy.c:20:
<thumper> gary_poster: still running...
<thumper> 3 minutes plus now
<thumper> oh
<thumper> and now its off
<thumper> why so slow today?
<thumper> I don't remember it being so bad
 * thumper knows
<thumper> 2.6 right?
<gary_poster> bdmurray: almost certainly python 2.5 -> python 2.6 change
<thumper> new buildout
<gary_poster> I need to run
<gary_poster> I'm sorry
 * thumper nods
<thumper> ok
<thumper> ttfn
<gary_poster> bdmurray: make clean then make build
<wgrant> Why do we only run Windmill tests on a single browser?
<thumper> wgrant: as opposed to?
<wgrant> thumper: More browsers. So we don't have half the world broken on IE and Chromium.
<thumper> wgrant: I'm under the impression that windmill only works with firefox
<wgrant> thumper: That wouldn't be very useful.
<wgrant> It supports at least IE/Firefox/Chrome.
<wgrant> Probably Safari as well.
<wgrant> What's the point of it if you can only run your tests in a single browser?
<thumper> ok, well we should be able to at least support chrome easily
<thumper> IE is a bit of a problem for our test environment :)
<lifeless> http://www.getwindmill.com/features
<lifeless> that ism it supports IE/Safari/Firefox/Chrome
<lifeless> thumper: ie can run under wine
<lifeless> I think
<lifeless> or we can run it on ec2
<wgrant> IE6/7 can mostly run OK in Wine.
<spm> can we run IE legally; vs technically?
<lifeless> spm: on ec2, yes
<spm> really? we don't have windows licenses aiui. ??
<lifeless> really
<lifeless> not uec
<lifeless> ec2
<thumper> wgrant: take it to the dev mailing list, and it should become a foundations issue
<mars> QA has licensed EC2 images for running IE tests.  Ask QA if you need to use them.
<thumper> wgrant: I'd suggest a buildbot builder per different browser
<lifeless> spm: on ec2 windows images are windows-per-hour charged
<lifeless> spm: its all bundleablemagic
<thumper> wgrant: we could run them in parallel initially
<spm> lifeless: granted; but we're not using windows instances on our test images; expect per mars' noting above. aiui, wgrant is referring to testing within the one image/instance. ??
<lifeless> distributed is the now
 * spm refrains from making rude comment about (oh to pick a name at random) oracle who haven't yet got that memo
<lifeless> spm: interesting you should say that
<lifeless> spm: given their clustering backend $magic they've been doing for a decade+ :)
 * spm was working with systems in the early 90's that could be physically moved across a city without any downtime :-)
<lifeless> nice
<lifeless> vendor ?
<spm> Digital Equipment Corporation
<lifeless> ah
<spm> VMS clusters ftw
<thumper> spm: plz fix launchpad, kthxbye
<lifeless> I went to uni at otago... largest dec networking install in the southern hemisphere
<spm> thumper: when you've ported it to vms? sure!
<thumper> spm: surely a small issue like that won't stop you
<spm> lifeless: was always amusing at decus conferences; "Oh? you work in Canberra too? Which Department?" Resp:Attorney Generals "Hrm. But AG's is all Mac's. Oh wait. You're ASIO. NM, you can't confirm/deny." :-D
<mars> thumper, you can start using HDFS for branch storage while you're at it!
<mars> shouldn't be hard, right?
<thumper> mars: we use Hard Disk for File Storage now :-) WFM
<mars> hehe
<spm> haha
<lifeless> stable to db-devel is failing at the moment
<lifeless> calculate-bug-heat
<thumper> lifeless: yeah
<thumper> I was going to get to it after my DB analysis
<lifeless> ok cool
 * thumper is starting to file bugs and tweak the view
<thumper> to make it suck less
<kb9vqf> Just to verify, all the icons in launchpad/icing are under Canonical copyright, correct?
<wgrant> It appears that way.
<kb9vqf> Can I still use the Ubuntu/Debian/Redhat icons in lauchpad/images though?
<wgrant> Well, just about everything is under Canonical copyright. But the icons and other images are under a much more restrictive license.
<kb9vqf> OK, I mean non-free licensing then
<wgrant> I have no idea why the Debian/Red Hat ones are there.
<kb9vqf> Me either ;-)
<kb9vqf> Just wanted to know if they were free or if I should just delete them
<lifeless> we should probably split that into a theme pack
<kb9vqf> How about the ones under icing/contrib?
<kb9vqf> Free or non-free?
<kb9vqf> I mean icing-contrib, sorry
<kb9vqf> Never mind, there are no images there!
 * kb9vqf needs some sleep
<wgrant> sinzui: That new +participation of yours is awesome.
<sinzui> wgrant, almost
<sinzui> wgrant, awesome would let me change my mailing list subscriptions and leave a team
<lifeless> +1
<sinzui> I was driven to make the changes because I could not be sure I dealt with ~drsganesh carnival of teams
<sinzui> :)
<wgrant> Haha.
<wgrant> There is a bit of a bug due to the Role column's conflation of ownership and membership, but it's still a great step.
<sinzui> I hope to see if I am done with him when I wake up in 8 hours
<kb9vqf> Is the file icon-sprites automatically or manually generated?
<thumper> kb9vqf: no idea
<wgrant> kb9vqf: There's a script to generate it.
<wgrant> Let me find it.
 * kb9vqf breathes a sigh of relief
<wgrant> make sprite_image
<wgrant> I think.
<kb9vqf> Let me try it real quick
<wgrant> Although you may actually need to make sprite_css.
<lifeless> sinzui: so did someone speak to drs ?
<kb9vqf> wgrant: make sprite_image worked, thanks!
<sinzui> He did not directly reply to my email. I had remarked that he appeared to be registering team that were already projects.
<sinzui> the next day he started creating projects that were already registered
<wgrant> sinzui: Oh, nice.
<lifeless> wow
<lifeless> thats really non cooperative
<lifeless> I think I'm going to have to take iwlagn out back and shoot it
<sinzui> maybe. I think he is clueless and thought he was helping. Since he didn't remove the teams I decided to do it for him
<lifeless> spm: code import dispatcher just blew up
<lifeless> spm: on pear
<kb9vqf> Are the images sprinkled in lp-sourcedeps/eggs/lazr_js-0.9.2DEVr170-py2.5.egg/lazrjs/inlineedit/assets/skins/sam free or non-free?
<kb9vqf> I am unsure because they are in the dependencies branch, not the actual Launchpad branch
<wgrant> LAZR/LP licensing is all a bit broken.
<spm> lifeless: bleh, ta.
<wgrant> eg. the LP branch is not redistributable. Possibly not even legally possessible. And the image license restriction is too vague, and the LAZR stuff is even less clear.
<lifeless> wgrant: please file bugs; thats not a situation we want.
<kb9vqf> Great, so I could be on legally shaky territory if I allow public access to a custom installation even if I replaced all the image files?
<spm> lifeless: what's the error message? or where?
<lifeless> spm: in cron, xmlrpc fault
<wgrant> I don't think access to it is much of a concern. But there's code in the history of the branch that is proprietary, files are missing license headers, and nobody is sure which images count and which do not.
<kb9vqf> Well I guess I'll have to replace all of them then...I'm most of the way there anyway :-P
<spm> lifeless: gar. I had that folder on a 'loganberry' search. no wonder I couldn't see it.
<kb9vqf> I did notice the missing headers, all over the LAZR stuff
<kb9vqf> Not good
<lifeless> wgrant: the history doesn't bother me, for all that it is clearly unclear
<wgrant> lifeless: But there is code in there that Canonical doesn't even own.
<lifeless> wgrant: if a tarball of tip isn't clearly redistributable, thats more of an issue
<lifeless> wgrant: a situation many open source projects are in - configure scripts being the most common :)
<wgrant> A tarball of tip is pretty much OK, although some of the files have partially incorrect licensing headers.
<spm> lifeless: heh. https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1621XMLP2 \o/
<lifeless> wgrant: so, my refrain - please file bugs, or patches :)
<wgrant> lifeless: OK, proprietary code that Canonical doesn't own, which I believe required a paid license.
<lifeless> wgrant: bzr has a tool to check headers automatically; might be useful for lp
<lifeless> wgrant: really? that surprises me
<lifeless> spm: ah, load load load
<wgrant> The old calendar widget, IIRC.
<wgrant> kb9vqf: I suspect that the lazr-js images are OK, but there aren't many so you might as well replace them anyway...
<wgrant> And, well, I might file bugs after exams.
<spm> lifeless: huh. interesting, given we were stumbling on that last week(??): https://answers.edge.launchpad.net/launchpad-registry/+question/113354
<lifeless> spm: the bug I filed got converted to a question until i convinced sinzui that it was really a bug
<lifeless> spm: it didn't need renaming :)
<lifeless> spm: gary_poster found the bug, in zope guts
<spm> oh, it's a real bug eh?
<lifeless> yes, fix already merged
<lifeless> the ++oops++ handler
<spm> typical, well, the project is renamed anyhoo.
<lifeless> gets treated as a traversal adapter
<lifeless> which it isn't
<spm> heh
<lifeless> so any namespace we add, like ++oops++, breaks all traversals that match its name.
<lifeless> as they say
<lifeless> 'oops'
<lifeless> spm: how did you go about renaming it? just via db ?
<spm> yup. like all blacklisted names
<lifeless> kk
<lifeless> (its not blacklisted :P)
<spm> :-) details. don't bore me wit hthe details. ;-)
<lifeless> spm: ok, take #34 - please pull.
<spm> :-)
<lifeless> whose on after you - tom ?
<spm> lifeless: hrm. we're running a U1 whatsit at the moment. problematic?
<spm> yup
<lifeless> no problem
<spm> kk
<lifeless> unless they are really really unlucky. Which they won't be.
<spm> rev 239
<lifeless> thanks
 * spm is tempted to quotes page that line. something suitably context misleading....
<lifeless> I think I need a shower now
<spm> hahaha
<wgrant> gmb: Thanks for fixing bug #373683. Looking at the diff, though, it appears that in the case where the old bug was public you've forgotten to actually % in the number and summary (bugchange.py:374).
<adeuring> good morning
<wgrant> Hmm.
<wgrant> "No recipes based off of this branch."
<wgrant> Is 'based off of' valid non-colloquial English?
<wgrant> I would have thought it was 'based on'.
<noodles775> Me too... wgrant, can you comment on the bug... we could change it at the same time (assuming you're looking at bug 591613)
<mup> Bug #591613: Recipes view oopses with +junk branches <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/591613>
 * noodles775 updates the description with a note.
<wgrant> noodles775: Heh, yes, I did find it while looking at that bug.
<wgrant> That bug looks pretty critical, given that they are *source package* recipes.
<noodles775> Yep... Just found both while trying to record a quick screencast.
<bigjools> g'day
<wgrant> Morning bigjools.
<jkakar> Maybe this is a known issue, but I'm commenting (via the web UI) on my merge proposal for a launchpadlib branch and getting "Your message was rejected" messages from launchpad-bugs-owner@l.c.c.  Seems odd. :)
<mwhudson> there was a thread about that
<mwhudson> it's because ~launchpad has that as a contact address
<lifeless> yes
<lifeless> mwhudson: oh, I wanted to talk to you.
<lifeless> mwhudson: ... and I've no idea why.
<mwhudson> lifeless: awesome
<lifeless> I think so.
<jkakar> mwhudson: Ah, okay, thanks.  I just wanted to make sure it was a known issue and not something weird happening just to me.
<wgrant> noodles775: Looks like we should really have some check constraints on the build tables...
<noodles775> wgrant: yes... did you look at the code to see where date_started is set? It's only in the one place, when the status is also set :/
<wgrant> noodles775: Hm, well, there are a couple of places where it's set to None.
<wgrant> One using setDateStarted, so it wasn't trivially greppable.
<wgrant> But it's in the unknown status handler, so should never be invoked.
<wgrant> (really we should assert that, I think)
<wgrant> But apart from that it's only set to None in retry, reset and jobReset. There's a bit of duplication there, which suggests that the code hasn't been looked at all together closely...
<wgrant> Sometimes I'd really like buildd-manage to keep a full statement log.
<wgrant> Oh. BuildFarmJob doesn't delegate all the date_* to Job yet?
<noodles775> wgrant: no, I wasn't prepared to bloat that branch further... as you pointed out, it can be done just as easily afterwards.
<noodles775> (it would have required more schema changes).
<wgrant> noodles775: Um, it looks like date_started wasn't unset. jobStarted was never called in the first place.
<wgrant> (date_first_dispatched is None, and it's much easier to audit all callsites of that)
<noodles775> wgrant: that's another bug...
 * noodles775 finds it.
<noodles775> bug 590699
<mup> Bug #590699: IBuildFarmJob.date_first_dispatched is None <Soyuz:Triaged> <https://launchpad.net/bugs/590699>
<wgrant> Are we sure it's not the same bug?
<noodles775> I can't see how it would be possible for jobStarted to never be called and yet result in a FULLYBUILT build.
<noodles775> And no, not sure, but I'm also not sure that we can assume jobStarted was never called because the api (?) has date_first_dispatched as None.
<wgrant> Ah.
<wgrant> https://edge.launchpad.net/~chromium-daily/+archive/beta/+build/1750601 has no date_first_dispatched, but it does have a date_started.
<wgrant> So it's a separate issue.
<wgrant> But that was built before 10.05...
<wgrant> And it still strongly suggests that jobStarted was never called, since it's pretty clear on what it does.
<wgrant> Oh. I wonder...
<wgrant> Hm, no.
<wgrant> noodles775: Ah...
<noodles775> ?
<wgrant> The specific_job is BuildPackageJob, which inherits BuildFarmJobOldDerived, which delegates to BuildFarmJobOld, and BuildFarmJobOld.jobStarted is a no-op.
<wgrant> So date_first_dispatched is not getting set anywhere.
 * noodles775 looks
<wgrant> And date_started isn't being set there, but is being set elsewhere sometimes, it appears.
<noodles775> wgrant: BuildPackageJob *should* be delegating to self.build_farm_job, which is  BuildFarmBuildJob, which implements jobStarted()?
<wgrant> But this network of classes and interfaces is seriously confusing at the moment.
<wgrant> Hm, let's see.
<noodles775> It is... it will be great to get rid of all the intermediate classes (ie. once the queue is refactored).
<wgrant> Yeah, you're right, it looks like it is being called.
<wgrant> Hm.
<wgrant> And indeed, it is set sometimes.
<wgrant> Gnargh.
<wgrant> Just the few I looked at had it None.
<wgrant> Also, i386 builders have had a fit again.
<wgrant> Yay recipes?
<noodles775> yep. bigjools is aware of it. And yes, the logs show recipes being dispatched.
<wgrant> The transaction is clearly being committed, or the builder and job state wouldn't be set, and the slave would end up rescued next scan.
<wgrant> So jobStarted must somehow not be called in some circumstances. But this seems impossible.
<wgrant> noodles775: What's the purpose of all these Old classes?
<noodles775> wgrant: BuildFarmJob already existed as an in-memory object used by the queue infrustructure, and yet provided a lot of the interface we needed.
<wgrant> Oh, right.
<wgrant> Like DistributionSourcePackage and DistributionSourcePackageInDatabase.
<noodles775> Yes, that would have been a better naming scheme.
<wgrant> Well, I think an uglier and more obscure one is probably better -- gives everyone more incentive to destroy it at the first possible opportunity.
<noodles775> But in retrospect, we should have named it something completely different, IMO.
<noodles775> Yep :)
<noodles775> wgrant: how many builds are you aware of that have a null date_started? I can only see 4, and there's something in common about them all that might help explain:
<noodles775> Finished at 20100602-1154
<noodles775> :)
<wgrant> I found four or five, I think. A query to find them (excluding gina-generated ones, obviously) shouldn't be difficult, though...
<wgrant> Are there any that have no date_started but do have a date_first_dispatched?
<noodles775> yeah, I'm doing that now (I expect more, but it certainly looks like it was rollout-related).
<wgrant> There are some that have both, some that have a date_first_dispatched and no date_started, and some that have neither.
<wgrant> Er.
<wgrant> Some do have a date_started but no date_first_dispatched, not the other way around.
<noodles775> Yep, I'll find out and updated the bug.
<wgrant> Oh, so they were running during the rollout?
<noodles775> See the timestamps at the end of the logs for the builds without date_started.
<wgrant> Heh, yes, the migration script bug is obvious.
<wgrant> So we only have one real bug -- the date_first_dispatched one. Phew.
<noodles775> Yep... :D, one less fire to put out.
<wgrant> And it's even easy enough to manually populate the field.
<wgrant> And, um, I don't see anything in the DB patch that takes date_first_dispatched from BuildInfo and puts it into a permanent table.
<wgrant> So we may in fact have no real bugs at all.
<wgrant> noodles775: ^^ have you found any missing date_first_dispatched values from after the rollout?
<noodles775> hangon.
<noodles775> wgrant: so, as expected, 9 completed builds since rollout with date_started not set.
<noodles775> and 2415 completed builds with date_first_dispatched not set.
<wgrant> 2415 since the rollout, or ever?
<noodles775> since the rollout (so it's not being set).
<wgrant> Curses.
<wgrant> But are there any set from before the rollout?
<wgrant> Probably not, but best to be sure...
<noodles775> Nope.
<wgrant> OK.
<wgrant> So we accidentally obliterated a whole lot of useless data, but we're still losing it somehow, so the massive confusion from earlier continues.
<wgrant> Yay.
<noodles775> Let's see (what has it been used for historically? I can't see it displayed in the ui)
<wgrant> date_first_dispatched? Nothing whatsoever.
<wgrant> Its value may be useful in the case of retried builds.
<wgrant> But it's only ever been exposed over the API, and the way Soyuz handles retries is completely braindead anyway.
<bilalakhtar> Hi there, people, how big is the launchpad branch?
<maxb> several hundred megabytes
<bilalakhtar> maxb: Over 200MB ?
<maxb> $ du -hs launchpad/lp-branches/.bzr/
<maxb> 180M	launchpad/lp-branches/.bzr/
<maxb> huh, not as big as I thought
<wgrant> The LP branch itself is less than 200MB.
<bilalakhtar> maxb: Wierd... Yesterday when I branched devel it took only an hour and downloaded only 95MB. Now, its getting more than 200MB and is still in "Fetching Revisions: Inserting stream"
<wgrant> But there are many hundreds of megabytes of dependencies.
<wgrant> Right, it will probably have to download 250-300.
<bilalakhtar> wgrant: deps? I have already installed lp-devel-deps
<wgrant> bilalakhtar: There are several other dependency branches which rocketfuel-setup will download.
<bilalakhtar> wgrant: you mean lp:lp-source-dependencies or there are even other ones?
<wgrant> bilalakhtar: That, plus perhaps a hundred megabytes of other smaller branches which will be downloaded into ~/launchpad/lp-sourcedeps/sourcecode
<bilalakhtar> wgrant: rocketfuel-setup downloads lp:launchpad/devel first. Am I right?
<maxb> yes. And so any extra bandwidth must be bzr being ineffiicent
<wgrant> Right.
<wgrant> I believe it then calls rocketfuel-get, which downloads the rest.
<bilalakhtar> wgrant: I downloaded lp:launchpad/devel yesterday, and it did not take as much time as it is taking now.
<bilalakhtar> i downloaded manually yesterday
<bilalakhtar> and later deleted and used rocketfuel
<maxb> that was unnecessary
 * maxb is disturbed to find that urls like https://edge.launchpad.net/~launchpad/+archive/ppa/+files/slony1_1.2.20.orig.tar.bz2 404
<bigjools> there's a bug for that
<bigjools> never got around to looking at it
<bigjools> I suspect it's where a package was copied and the original file expired
<Ursinha> hi sinzui, after the reviewers meeting: what QA problem exactly are you talking about there?
<sinzui> bug-tag qa report verses kanban qa coiumns
<Ursinha> sinzui, ah, I see, that
<Ursinha> sinzui, the QA report is updated every 15 minutes, I guess
 * Ursinha checks
<Ursinha> that's correct, 15 minutes
<sinzui> The cards on Kanban disagreed with the qa tags. some bugs were reported qa-needstestings, but they were fore the previous release
<Ursinha> sinzui, how is the kanban board updated, manually?
<sinzui> Manually
<sinzui> and the card may represent many bugs, or no bugs
<Ursinha> sinzui, but if the bugs were from past releases, they don't show in current cycle QA reports
<Ursinha> sinzui, I see
<sinzui> Ursinha, malone had a lot of 10.04 bugs and they did show up in launchpad-project/?tag=qa-needstesting
<Ursinha> sinzui, ah, that's what you're calling QA report?
<sinzui> No, that is pointing out we are not using Lp as we claim. That list should be near zero when we are closing PQM
<micahg> deryck: ping
<Ursinha> sinzui, agreed
<deryck> micahg, hi
<micahg> deryck: hi, I wanted to chat about the multi dupe move in LP
<deryck> sinzui, that was largely just oversite on my part.  I'm trying to watch my team and milestones closer.
<deryck> micahg, ok, please do.  bend my ear. :-)
<micahg> deryck: so, I was thinking perhaps if dupes > 3, only bug control should be able to in case there are issues
<micahg> deryck: i.e. either malicious or mistakes
<deryck> micahg, ah, right.  Someone did mention limiting to bug supervisor since there is potential for huge mistakes now.
<deryck> micahg, I'm happy with this change.  in the case of a bug with other dupes only, though.
<micahg> deryck: k, is there a bug I should comment on/mark as affecting me?
 * deryck looks for number....
<deryck> micahg, please add a comment to bug 591705 saying we discussed and we need some limit for this, etc.
<mup> Bug #591705: Add confirmation dialog when marking duplicate a bug that itself has duplicates <dupe-handling> <javascript> <Launchpad Bugs:Triaged by deryck> <https://launchpad.net/bugs/591705>
<deryck> and, of course, subscribe if you like. :-)
<micahg> deryck: I'm actually subscribed to Malone :)  I figure since I'm a heavy user of LP, it's good to know what's going on
<deryck> oh, cool.
<micahg> deryck: what about a "reverse dupe merge" option?
<deryck> micahg, elaborate, please.  What do you mean?  Undo the mess automatically?
<micahg> deryck: well, a button for bug supervisor to reverse the duplication as a group
<deryck> micahg, interesting idea.  It's a non-small amount of work.
<micahg> deryck: k, is it worth filing a place holder bug and adding it to my comment on the dupe bug?
<deryck> micahg, yes, I think so.
<deryck> micahg, do you see a downside to limiting moving a bug with dupes to just bug supervisors?
<micahg> deryck: only that regular users can't help, but given the possible damage, I think it's a good tradeoff
<deryck> micahg, ok, I agree.  I wonder if we limit to bug supervisor, if the conf dialog is even required.
<micahg> deryck: yes :) bug supervisors are human
<micahg> deryck: bug 591749
<mup> Bug #591749: Allow mass deduplication option for bug supervisors <Launchpad Bugs:New> <https://launchpad.net/bugs/591749>
<deryck> micahg, fair enough :-)
<micahg> deryck: so if the bug I just filed gets fixed, maybe we can open the ACL
<micahg> thanks deryck
<deryck> micahg, np.  Thanks for the feedback and input.
<mpt> Right at this moment, there are exactly 6000 open bug reports about Launchpad.
<mars> flacoste, ^ :)
<flacoste> we should just close randomly 5500 of those and see how many gets re-opened :-)
<mpt> Probably 100 or 200 of them have been fixed or become invalid without anyone realizing
<mwhudson> i bet it's more than that
<mwhudson> but i'm not about to start trying to prove it :)
 * jml has plans
<mwhudson> jml: go away, supposedly on leave person
<mpt> oh, jml, I should send you some sketches I've been doing this week
<mpt> of "How do we get people excited about the possibility of contributing to Ubuntu", in the medium of Launchpad's distribution series overview page
<deryck> adeuring, the OPINION status is completely un-acl'ed, right?  Like Invalid, right?
<adeuring> deryck: yes
<deryck> adeuring, excellent.  thanks.
<mpt> Oh, lordy, that OPINION thing went ahead?
<deryck> mpt, yes, it did.
<deryck> mpt, were trying to have careful tracking of its usefulness, so that if it doesn't prove good, we can verify that and remove the status.
<mpt> ok, good
<mars> If there was some way to mark comments as either "evidence" or "discussion", then you can change the visual design based on which type the user wants to see.
<mpt> see also bug 1734
<mup> Bug #1734: Need ability to mark bug comments as obsolete <feature> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/1734>
<mars> a four-digit issue, nice
<abentley> noodles775, any tips on how to avoid 'read committed' assertion when testing handl_status_OK?
 * noodles775 looks
<mars> mpt, reading that thread reiterated in my mind how a simple 'fault' tag may be the best filter for engineers like myself.  I think on some days we would be happiest to ignore everything else.
<noodles775> abentley: I'm not sure... i've never had it triggered.
<abentley> noodles775, it looks like changing the layer from LaunchpadFunctionalLayer to LaunchpadZopelessLayer fixes it.
<noodles775> ok.
<mpt> mars, if I was designing it we'd just have states {Unconfirmed, Confirmed, Ready, In Progress, Done, Declined}. On your hack days you'd concentrate just on {Ready, In Progress}. Everything else would be the job of either design sprints or QA.
<bigjools> abentley: buildmaster runs as zopeless
<mars> gary_poster, ping, any idea what we should do about the DeprecationWarnings being raised by ec2 test now that we are on Python2.6?  Silence them or upgrade the offending library?
<gary_poster> mars, I suggest that we silence them in lp_sitecustomize.py and add a bug.  However, if you are up for it and there's a new version of the library that claims to address the issue, you can give it a whirl in ec2 test pretty easily.
<mars> gary_poster, just checked, pycrypto is the culprit, still on 2.0.1
<gary_poster> silence is the order of the day then :-)
<mars> lp_sitecustomize it is then
<sinzui> gary_poster, ping
<gary_poster> sinzui: pong
<sinzui> gary_poster, do you have time to talk about canonical.launchpad and shipit?
<gary_poster> sinzui: ...in about 30, at 5PM?
<sinzui> thanks
<gary_poster> cool
<maxb> allenap: I like your simile
<maxb> :-)
<jelmer> aas/win 23
<gary_poster> back to ya ;-)
<gary_poster> sinzui: hey.  now?
<sinzui> hi gary, mumble or skype
<gary_poster> sinzui, either fine, but mumble is my default
<thumper> sinzui: can I have a call with you after gary_poster?
<gary_poster> we're done :-)
 * sinzui starts skype
<gary_poster> I will call it, "a day".  Look, a day!
<gary_poster> bye
<sinzui> thumper, https://bugs.edge.launchpad.net/launchpad-registry/+bug/58297
<mup> Bug #58297: Making a project part of a project group should require project group owner's approval <feature> <oem-services> <registry> <Launchpad Registry:Triaged by sinzui> <https://launchpad.net/bugs/58297>
<jml> http://people.canonical.com/~jml/convergence/
#launchpad-dev 2010-06-10
<wgrant> Hm, I didn't know enough data was exposed to create such a thing.
<kb9vqf> Is the PPA build system open sourced as well?
<wgrant> Yes.
<wgrant> https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally
<kb9vqf> Thanks!
<jml> wgrant, that's 100% through the API
<jml> wgrant, using anonymous access only.
<jml> wgrant, although the numbers are wrong, for some reason that yet eludes me
<wgrant> jml: Oh, using date_confirmed and its kin?
<jml> yep
<wgrant> kb9vqf: I've just tweaked that page a bit, trimming obsolete sections and that sort of thing.
<kb9vqf> OK, thanks--I was still in the configuration section anyway ;-)
<kb9vqf> wgrant: Any idea what might have caused this: http://pastebin.ubuntu.com/447456/
<kb9vqf> All other pages seem to work fine, but the home page doesn't
<kb9vqf> (https://launchpad.dev/)
<wgrant> kb9vqf: You're using an empty DB without sample data?
<kb9vqf> I trimmed out a lot of the sample data
<kb9vqf> Guess I trimmed something I shouldn't have ;-)
<wgrant> Try cronscripts/update-stats.py
<wgrant> That may fix it, but it may not work unless you have at least one project.
<kb9vqf> I was wondering if that would be the case
<kb9vqf> Thanks for the cron hint as well
<kb9vqf> One more thing...regarding changing all the instances of the word "Launchpad" to something else, would I be lucky enough for that to be controlled by a single file somewhere?
<kb9vqf> Or will I have to resort to sed
<wgrant> sed, probably.
<wgrant> I use http://pastebin.ubuntu.com/447458/ and http://pastebin.ubuntu.com/447459/ on an empty DB (without any sample data at all).
<wgrant> Creates most of the critical celebrities.
<wgrant> Updates the statistics, that sort of thing.
<wgrant> And the second populates Ubuntu.
 * kb9vqf spent an entire day trying to get the celebrities in place
<kb9vqf> I'll try the scripts as soon as Launchpad finishes rebuilding
<ajmitch> wgrant: you have those scripts somewhere other than pastebin?
<ajmitch> they look useful
<wgrant> ajmitch: Not really.
<wgrant> They just sit around in ~/launchpad/utils and occasionally get unbroken when I get fed up with how terrible the sample data is.
<lifeless> jml: wow
<lifeless> jml: lovely visualisation; can you do the bazaar project group too ?
<jml> lifeless, I want to. You might be able to figure out how to do it yourself if you branch lp:~jml/+junk/convergence
<lifeless> jml: I'm sure I can :) . Uhm this seems like a lptools thing, if you want to make a home for it
<jml> lifeless, I'm deferring the problem of figuring out how to have couch views that I can give a parameter to.
<jml> (probably the answer is something horrible)
<lifeless> jml: this is using offlinelaunchpadstuff ?
<jml> lifeless, no.
<jml> I wanted to, and then I spent a bunch of time making the offlinelaunchpad stuff better in subtle, peripheral, important ways and then I got on with this.
<wgrant> jml: Launchpad didn't collapse under the load of generating that graph?
<lifeless> heh
<lifeless> wgrant: no
<jml> wgrant, not as far as I can tell.
<lifeless> wgrant: we generate more load internally than the API servers can cause ;P
<wgrant> What was it that was causing the DB issues?
<lifeless> more load on master than desired, not enough on the slaves.
<lifeless> which was due to replication lag being stale
<wgrant> Ah.
<lifeless> which was due to the replication lag monitor not running.
<wgrant> Oops.
<lifeless> which wasn't fixed immediately because its not, itself, monitored.
<lifeless> Which is because we don't have a 'must have monitoring on day 1' rule.
<lifeless> or something like htat.
<lifeless> (such a rule might be more trouble than its worth)
<jml> meh
<lifeless> jml: ? I'm saying not everything is important enough to monitor, in my suggestion that a hard rule might be trouble.
<jml> apart from the 162 bug tasks that are counted as "New" on that graph due to API exposure, I don't know why there are 500 bug tasks that have no "date_foo" field but are not actually "New"
<jml> lifeless, the "meh" was in response to my internal angst :)
<lifeless> ah :)
<lifeless> by all means, mehangst away
<wgrant> jml: Incomplete?
<jml> wgrant, that's the 162
<wgrant> Ah.
<thumper> jml: hi
<thumper> jml: are you skypeable?
<jml> thumper, only for fun and social reasons :)
<thumper> jml: I have something of strategic importance
<thumper> heh
<thumper> probably neither fun nor social
<thumper> jml: are you on leave?
<jml> thumper, yeah.
<thumper> when are you back?
<jml> thumper, thus dicking about with desktopcouch and graphs
<jml> thumper, back Monday
<thumper> ok
<thumper> remind me Monday
<jml> thumper, will do. look at the shiny. http://people.canonical.com/~jml/convergence/
<thumper> I saw
<thumper> lots of bugs
<thumper> you didn't graph won't fix :)
<jml> thumper, yeah I did!
<thumper> really?
<jml> thumper, it's the invisible bit at the bottom :)
<wgrant> What about Opinion?
<jml> thumper, or rather, wontfix are considered done for the purpose of this graph
<wgrant> Or is that not actually in production yet.
<thumper> jml: your key only shows five statuses
<jml> thumper, it deliberately doesn't show non-open bugs.
<thumper> It'd be interesting to see our rate of fixing ...
<thumper> has it slowed down / sped up ...
<thumper> jml: a first derivative graph of the bugs graph would also be interesting
<jml> thumper, well, you can kind of see that by the "Fix committed" line at the bottom.
<thumper> jml: at what rate are bugs being filed
<jml> thumper, some of that stuff is on this (old) set of graphs: http://people.canonical.com/~jml/lp-bugs/
<jml> thumper, I guess now that I've figured out the relevant technologies and don't have to do staging database queries to get this information, I can add those.
 * thumper nods
 * thumper goes back to wrapping bugs
<jml> thumper, first derivative isn't useful with so many lines -- I think you only want "open" and "closed" then
 * thumper stabs ORMs in the face
<wgrant> Ridiculously inefficient SQL usage?
<thumper> yes
<thumper> and the hoop jumping to make it more efficient
<thumper> part of what I need to talk to jml about
<wgrant> I wish I could tell it that I wanted it to precalculate various attributes.
<wgrant> Rather than having to do that manually, which makes everything about 10000000000x times uglier.
<thumper> like what?
<wgrant> For example, the publisher needs to retrieve lots of SourcePackagePublishingHistorys, their SourcePackageReleases, their SourcePackageReleaseFiles, and various other collections.
<wgrant> If I want to do that in less than hundreds of thousands queries (quite literally), I have to formulate a query manually and can't use the objects' normal attributes.
<lifeless> yes
<lifeless> orms are evil
<lifeless> db query interfaces - like storm /has/ - are very nice
<wgrant> I should be able to tell Storm to precalculate SPPH.spr, SPR.files, and that sort of thing.
<lifeless> but magic attributes - very hard to get the right balance
<lifeless> wgrant: you can
<wgrant> There is the prejoin implementation in the SQLObject wrapper.
<lifeless> wgrant: there's a couple of hooks; not sure if the precise one you need is present
<wgrant> But it doesn't do collections.
<lifeless> wgrant: theres a 'when getting this from the DB do xyz' hook
<lifeless> wgrant: I'm pretty sure
<maxb> heh, did someone give launchpad/ppa a biased buildscore? :-)
<lifeless> don't think so
<maxb> I just uploaded stuff and it seems to have bypassed the entire queue :-)
<lifeless> rotfl
<wgrant> That is a little suspicious.
<wgrant> I suppose we can delete most of the PPA now.
<maxb> s/suspicious/convenient/ :-)
<wgrant> Mm, although I guess we need to wait until 10.06 is out the door.
<lifeless> is there a karma<->priority matchup ?
<wgrant> lifeless: Only inasmuch as people with more karma are more likely to be able to hit up buildd admins :P
<maxb> "Automatic retry of builds waiting on dependencies is disabled pending resolution of a performance problem" -- o rly?
<wgrant> It's back now.
<wgrant> Was announced on launchpadstatus, I thought.
<maxb> Also, https://edge.launchpad.net/~lamont/+archive/psql8.3/+packages seems to have gone into an endless loop of depwait -> retry -> depwait -> retry ...
<maxb> i.e. they're being erroneously retried
<wgrant> If you ever catch a build log or dependencies value from any of them, please keep a note of it.
<wgrant> There are some known bugs, but not any that should affect PPA builds.
<maxb> the lpia build there is building again now
<wgrant> Yeah.
<wgrant> Oh, if it depwaits I guess it will finish soon.
<wgrant> Forgot that detail.
<wgrant> If buildd-manager ever wakes up, that is.
<wgrant> I must fix it one day.
<maxb> If there's a buildd-admin willing, can I get a rescore on at least one build in https://edge.launchpad.net/~maxb/+archive/launchpad/+packages ? I'd like to get a confirmed success before I sleep, so I can give them to stub tomorrow
<thumper> ââ¹
<thumper> ââ¹
<thumper> ââ¹
<thumper> ââ¹
<thumper> ââ¹
<wgrant> Ah, debhelper >=7.
<thumper> grrrrraaaaahhhhhhhhhhhh!!!!!!!!!!!!!
<wgrant> So it's probably the pocket-naÃ¯vetÃ©.
<wgrant> thumper: What has Storm done now?
 * jelmer waves to thumper and wgrant 
<wgrant> Or Soyuz?
<wgrant> Morning jelmer.
 * jelmer ducks
<thumper> I'm using lazr.delegate to wrap a bug so it doesn't hit the db for its tasks
<thumper> but getBugTask(self, target) uses self.bug_tasks
<thumper> so in order to get that method to the cached tasks,
<thumper> I either have to copy the implementation
<thumper> or violate one of our rules for imports
<thumper> and import Bug into code.browser
<thumper> to go Bug.getBugTask(my_wrapped_bug, target)
<wgrant> So you're precalculating the collection?
<thumper> yes
<wgrant> Well.
<thumper> in a single query to get the linked bugs and all their tasks
<wgrant> One easy way out is to abuse the fact that Storm doesn't notice when you assign a list.
<wgrant> So you clobber bug.bug_tasks with your precalculated list.
<wgrant> Evil, yes.
<wgrant> But so are the other methods.
<thumper> eh?
<thumper> that is evil
<spm> maxb: did you get that rescored? I'll do it now?
<maxb> spm: yes please
<spm> ahh. slony. hell yes I'l rescore those.
 * thumper goes for the lesser of three evils
<maxb> :-)
<spm> maxb: powa has been abused and used: https://edge.launchpad.net/~maxb/+archive/launchpad/+builds
<maxb> neat, and they've been dispatched already :-)
<spm> see. I am useful for *some* things. ;-)
<maxb> Now I just need to talk lamont into using my builds instead of redoing it
<wgrant> Doesn't Hardy have 8.3 already?
<spm> the backport for hady-cat that we do? generally that's little more than trivial textual changes and getting a localised rebuild of same.
<maxb> An 8.3 what?
<maxb> These builds were about getting the most recent upstream of slony built for (hardy,lucid)*(8.3,8.4)
<wgrant> Ah, right, Slony.
<maxb> which it now has :-)
<thumper> queries down from 294 to 93 using cached bug tasks
<thumper> (for my test branch)
<ajmitch> thumper: you managed to mangle it to use the right property?
<thumper> no
<thumper> well kinda
<thumper> too much special code IMO
<thumper> now I see most of the queries being in the menu link formatters :(
<thumper> it should not be doing that :(
<thumper> I thought that some old fix of salgado fixed that
 * thumper wishes sinzui was around to explain
<sinzui> Yes it should not be doing that
<sinzui> I crafted a menu two weeks ago to avoid subscription checks. each menu call in the template was  a call to the db
<sinzui> thumper, I think menus should be singletons or we use a utility that manages to menus for each object to ensure there is only for instance for each request
<thumper> sinzui: ah, so it is known that it is somewhat fucked then?
<sinzui> You and I know this
<thumper> heh
<thumper> ok, how to fix this in my branch?
<thumper> ah
<thumper> I know why it is screwed in this moment
<sinzui> I was very tempted to risk toppling all menus to fix the performance issues working with milestones on pages
<thumper> the context object is the unwrapped object
<thumper> the view.context is the decorated object
<thumper> I bet the pages are using context/menu rather than view/context/menu...
<thumper> I'm caching the queries needed by the menu
<thumper> in my wrapped object
 * thumper checks the templates
<sinzui> I think each template fragment is creating the menu on demand or for it's chunk, but that same menu for the context may be instantiated 30 times
<thumper> right
<sinzui> I think salgado had a hack to put the data in the request so that templates shared the menu, but I do not see evidence of that
<thumper> sinzui: can I override the globs for the template?
<thumper> sinzui: I want to set the context to be my wrapped context
<thumper> rather than the unwrapped one
<thumper> sinzui: do you know where the template render code is actually called?
<sinzui> render() for the fmt:link?
<thumper> I mean the code that sets the 'context' for the page template render
<sinzui> ah
<sinzui> That would be sidnei's chameleon engine
<thumper> are we using chameleon?
<sinzui> yes
 * sidnei raises an eyebrow
<thumper> where is the code?
<sinzui> in an egg
<thumper> :(
 * sinzui saw it while purging py2.5 eggs today
<thumper> surely we call it somewhere with a dictionary of locals right?
<thumper> that is what I want to override
<sidnei> sinzui, chameleon is not enabled, though the code is there
<sinzui> :(
 * thumper hunts in webapp for calling pagetemplates
<thumper> sidnei: unless you can tell me where to look
<sinzui> thumper, are these LaunchpadView views?
<thumper> yep
<sinzui> I believe we hacked __call__
<sidnei> thumper, i might if sinzui doesn't beat me to it
<thumper> :(
<thumper> there seems to be much magic there
<sidnei> there's a ton of magic involved yes. it left me scratching my had last time i looked at it.
<sidnei> s/had/head
<thumper> __call__ calls render
<thumper> render calls self.template()
<thumper> template is a property that returns self.index
<thumper> self.index seems opaque
<sinzui> thumper, during traversal, the view is construct. adapters from zcml marry the named view to a template and create the final view call, and it monekypatches __facet__ because using ILayer would have been too easy
<sidnei> thumper, self.index is the template that gets registered by the zcml directive iirc
<thumper> sinzui: do I have to read the entire publisher file?
<sinzui> No, look for __facet__ I scream every time I see that monster
<thumper> ENOTFOUND
 * sinzui ponders
<thumper> __launchpad_facet__
<sidnei> sorry for not being more helpful, it's almost midnight here and im trying to wrap up a large refactoring
<sinzui> thumper, yes, in metazcml
<thumper> ââ¹
<ajmitch> are there any tests around for scripts/ftpmaster-tools ?
<lifeless> hah
<ajmitch> I was afraid of that
<thumper>  return mapply(ob, request.getPositionalArguments(), request)
<thumper> publication.py line 436
<ajmitch> on a related note, is launchpad.net running on lucid
<sinzui> yes
<sinzui> as of today, dev is on 2.6
<sinzui> python 2.6
<ajmitch> ok, I just saw some duplicated code in sync-source.py to remove then, I'm just changing some of the version matching in there :)
<spm> sinzui: it is? oh *dev* is. right. prod is still running on hardy.
<sinzui> all lpnet is hardy?
<lifeless> ajmitch: different questions
<lifeless> ajmitch: 'works on 2.6'
<lifeless> ajmitch: != 'deployed on 2.6'
<lifeless> ajmitch: we can't remove compat stuff until its deployed on 2.6
<sinzui> We this will be a very exciting rollout. I am glad I am not the release manager for 10.06
<spm> sinzui: I hope so, or else it's been upgraded without me knowing about it; which'd be fun :-)
<ajmitch> lifeless: in this case, it's code duplicated from python-debian that is in lucid
<ajmitch> rather than anything python 2.6-specific
<lifeless> ajmitch: we're runniong on hardy
<ajmitch> which is mostly what I was asking
<lifeless> yeah, you asked the right question; but a little more clarity that it wasn't a proxy for python version would have helped ;)
<ajmitch> I did clarify, just a little too late :)
<spm> :-)
<ajmitch> it was bug 520508, fwiw
<mup> Bug #520508: Duplicate copy of split_gpg_and_payload() <soyuz-upload> <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/520508>
 * thumper frowns
<thumper> I can't find it!
<thumper> damn it
<thumper> the publication code calls request.getPositionalArgs
<thumper> which defers to self._args of the request
<thumper> which gets set with positional= kwarg
 * thumper is frustrated
<thumper> going back to changing context/menu to view/context/menu
<thumper> for some reason my context is getting wrapped twice
<thumper> \o/
<thumper> down to 63 queries
 * thumper looks for dupes
<lifeless> nice
<thumper> down from almost 300
<thumper> still dupes though
<thumper> and we have 20 queries before we even hit object traversal
<thumper> I think with some effort we could drop that to under 1-
<thumper> 10
<wgrant> thumper: Ew, what are the pre-traversal queries?
<wgrant> That's mildly insane.
<thumper> :)
<thumper> want me to pastebin it?
<wgrant> Well, I have an exam in 20 minutes, so it's probably best that I don't look at it :P
<thumper> http://pastebin.ubuntu.com/447557/
<thumper> sorry
<thumper> there goes the study
<wgrant> Heh.
<thumper> which exam?
<wgrant> Logic Completeness and Incompleteness.
<ajmitch> oh that sounds fun
<wgrant> SELECT AccountPassword.account, AccountPassword.id, AccountPassword.password FROM AccountPassword WHERE AccountPassword.account = %s
<wgrant> Why would it being doing that? It's not meant to know that passwords exist.
<thumper> wgrant: go to your exam
<thumper> ew
<thumper> removeSecurityProxy(db.branch).__class__.displayname.__get__(db)
<thumper> I think that passes the wrapped object (db) into the property getter as self
<thumper> it is pretty revolting
<stub> Hmm.... """sourcepackagerelease components "not null" site:bugs.launchpad.net """ in google finds me the bug I want on index pages, but not the bug itself.
<madscientist159> wgrant: You still around?
<madscientist159> wgrant: I tried to run your script to initialize the empty database, but all I got was this:
<madscientist159> http://paste.ubuntu.com/447607/
<kb9vqf> Anyone have an idea why I am getting that permission error?  I've tried as root and as the postgresql user...
<kb9vqf> wgrant: I managed to get around the problem by temporarily granting all access to the karmacategory table and sequence, so my above comments are just an FYI
<kb9vqf> Also FYI, the second script will notrun, complaining about "lp.registry.interfaces.distroseries" not existing
<kb9vqf> Ideas for debugging this oops?
<kb9vqf> v
<kb9vqf> http://pastebin.ubuntu.com/447619/
<kb9vqf> It happens only when trying to go to a project page such as https://launchpad.dev/launchpad
<kb9vqf> Everything else seems to work great
<adeuring> good morning
<kb9vqf> wgrant: Here's a fixed version of the second script: http://pastebin.ubuntu.com/447622/
* thumper changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.06 | PQM is open but ec2 land/-s is broken | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<wgrant> kb9vqf: Ah, yeah, sorry, forgot about the karmacategory perm fix.
<wgrant> And DistroSeriesStatus was recently moved and renamed to SeriesStatus, and apparently that version of the script hasn't been fixed yet.
<wgrant> And it looks like you need to run rmy second script now, or all projects explode.
<wgrant> That's pretty inconvenient.
<kb9vqf> wgrant: Did you see my fixed version of the second script ;-)
<wgrant> Aha.
<kb9vqf> I figured out the projects exploding...my local installation is actually working quite well ATM
<kb9vqf> Thanks to your scripts ;-)
<kb9vqf> Still working on the PPA builders though
 * kb9vqf crosses fingers
<wgrant> What's wrong with them?
<kb9vqf> Nothing yet
<kb9vqf> Just don't have the software installed yet
<wgrant> Ah.
<kb9vqf> I fumble-fingered a move command earlier and deleted the past 12 hours of work
<wgrant> Well, beware, I wrote those instructions too, so...
<kb9vqf> :-)
<wgrant> I might throw the scripts into a branch and push it up.
<kb9vqf> Be aware that there is an issue (syntax error) in one of the Launchpad python files the first script calls...something to do with the distribution IIRC
<kb9vqf> Sorry I don't have the filename handy, that was several hours ago
<kb9vqf> There was a colon in the function declaration when it was really supposed to be a comma
<wgrant> Hmm, odd. I guess I'll find out in a few minutes when I test the whole lot.
<noodles775> kb9vqf: yes, a testfix branch was just landed. (The issue was the result of a bad-but-non-conflicting merge)
<wgrant> However, I think the ubuntu.currentseries.displayname crash is a legitimate bug.
<wgrant> Let's see...
<kb9vqf> Well, it went away after the second script was modified and run
<wgrant> Yeah.
<wgrant> But I meant the second one to be optional; that's why it's separate.
<kb9vqf> Oh, OK
<wgrant> Ah, it's not a real bug that will show up in production :(
<kb9vqf> How does the blog widget on the Launchpad home page operate?  Can I redirect it to my blog?
<wgrant> It's... hardcoded.
<kb9vqf> Oh
<kb9vqf> How can I remove the entries that are there right now?
<wgrant> grep around and find the template that has them.
<kb9vqf> In the DB or in the files?
<wgrant> The files.
<wgrant> The entries themselves are hardcoded.
<kb9vqf> Ohhhh
<wgrant> Not the blog feed that it pulls from.
<kb9vqf> Shall I say that seems rather...odd?
<kb9vqf> ;-)
<wgrant> Yes.
<kb9vqf> Is there a list anywhere of how often each cron script should be run?
 * kb9vqf doesn't want to waste system resources by running every minute if its not needed
<wgrant> That's something they've not revealed.
<kb9vqf> OK, trial and error it is then
<kb9vqf> wgrant: Are there any instructions for setting up a build daemon on a remote machine?
<kb9vqf> I have a rather large cluster here of identical machines I'd like to use
<wgrant> kb9vqf: Just install launchpad-buildd on them and point LP at them.
<wgrant> But watch out. lp-buildd is unauthenticated.
<kb9vqf> I have a dedicated private network, so that's not a problem ;-)
<kb9vqf> (Physically secured too)
<kb9vqf> So I have to install the lp-branches/devel stuff on each build machine then?
<wgrant> No. Just grab the launchpad-buildd .deb that is created.
<kb9vqf> What about the chroot image?
<kb9vqf> Oh..is that image system wide and uploaded to each builder on-demand?
<wgrant> Right.
<wgrant> The chroot is sent out for each build.
<wgrant> (although it's cached on the slave)
<kb9vqf> That makes a ton more sense
<kb9vqf> Thanks!
<wgrant> https://code.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch contains all the changes and fixed scripts for bootstrapping.
<kb9vqf> Subscribed myself so I won't lose it :-)
<kb9vqf> Has anyone else actuallydeployed Launchpad other than Canonical?
<kb9vqf> Just out of curiosity
<wgrant> I don't believe so.
<kb9vqf> So I might be the first
<kb9vqf> :-)
<wgrant> Anyone who tried would have probably run into huge problems and asked here.
<wgrant> Everyone who has asked here has given up.
<kb9vqf> Hmmm....Librarian
<kb9vqf> It doesn't seem to be started from "make run"
<wgrant> noodles775: Have the getBuildRecords timeouts been fixed? I've only had a few emails from cron today, rather than the usual several dozen.
<wgrant> It still times out sometimes, but not as much.
<wgrant> kb9vqf: It probably only listens on localhost.
<kb9vqf> Yes; I'm getting this though
<kb9vqf> Librarian upload failed: [localhost:58090]: [Errno 111] Connection refused
<wgrant> Hm.
<wgrant> Is that from manage-chroot?
<kb9vqf> Yup
<wgrant> You've not touched the config?
<kb9vqf> Nope
<wgrant> make run definitely normally starts it.
<kb9vqf> That's what I thought too; here's what is in the config file right now:
<kb9vqf> [librarian]
<kb9vqf> download_url: http://launchpad.dev:58080/
<wgrant> Hm, that's not the default, is it?
<kb9vqf> Yes it is
<wgrant> Mm, so it is.
<kb9vqf> At least that's what came from Bazaar
<kb9vqf> Should it be localhost?
<wgrant> Well, if you want to make really sure that it doesn't work from any of your other buildds :P
<kb9vqf> No thanks, I'll pass :-P
<kb9vqf> It doesn't seem to be running at all; there is nothing with "lib" in ps aux
<wgrant> It's a twistd with librarian.tac in argv.
<wgrant> So try killing and restarting make run.
<kb9vqf> That's what I'm doing now
<wgrant> So.
 * kb9vqf is reminded of Windows..."When acting strangely, simply reboot!"
<wgrant> I guess you need to override upload_host, download_host, restricted_upload_host and restricted_download_host in the librarian section in your config file.
<wgrant> As well as making it somehow not listen on localhost only.
<wgrant> Or stick a proxy in front of it.
<kb9vqf> The reboot fixed it
<kb9vqf> And I was thinking of doing a proxy, it's three lines in Apache ;-)
<noodles775> wgrant: no, well not specifically. There has been some changes since yesterday that reduce the db load, so that might be the reason you're seeing less timeouts.
<kb9vqf> Can I run  i386 buildd on a 64-bit installation of Ubuntu?
<wgrant> kb9vqf: Yes.
<kb9vqf> Good
<wgrant> You'll need to tweak the arch in /etc/launchpad-buildd/default, though.
<wgrant> It defaults to the system arch.
<kb9vqf> Yeah, I noticed that
<kb9vqf> Wasn't sure if it was a read-only attribute so to speak
<wgrant> You can also drop additional config files alongside 'default' to run multiple lp-buildds on one host.
<wgrant> Handy for handling multiple archs.
<wgrant> They cannot, however, share a filecache.
<kb9vqf> Cool
<kb9vqf> Now I can put those 8 cores to good use
<kb9vqf> Things not to do...try to run all of the cron scripts in parallel at once
<kb9vqf> :-{
<wgrant> Um, yeah, bad idea.
<kb9vqf> wgrant: FYI, I got a "canonical.launchpad.utilities.celebrities.MissingCelebrityError: ubuntu-bugzilla" from one of those scripts
<kb9vqf> You might want to add it to the DB building script
<wgrant> kb9vqf: Note that the DB bootstrap script has lots of comments in the celebrity list for stuff that isn't created yet.
<wgrant> You really don't want to be running scripts unless you know what they do.
<kb9vqf> I glanced through it and verified the database; just didn't look real hard at which celebrities were being created
<kb9vqf> Since that is a bit of a black box to me right now
<wgrant> Right. But you should be working out what the cronscripts do before you run them.
<kb9vqf> That is true
<wgrant> Ewwwwwwwwwwwwwwwww.
<kb9vqf> The main problem was launching them all at once; I will probably have to create groups of them (e.g. the PPA updating scripts could run every minute, etc)
<wgrant> I'd never dared to look at why ubuntu-bugzilla was a celebrity.
<wgrant> I... regret discovering why.
<kb9vqf> Do I want to know?
<wgrant> Well, once upon a time, Ubuntu used Bugzilla. Then all the bugs were imported into LP, with bugwatches pointing at the old Bugzilla bugs.
<wgrant> Now, normally bugwatches are updated every day or so.
<wgrant> But this doesn't make sense for Ubuntu Bugzilla, since it doesn't exist any more.
<wgrant> So.... there's a hack to skip updating bugwatches if their tracker's name is the same as the ubuntu-bugzilla celebrity's.
<kb9vqf> Ewwww
<ajmitch> almost as ewww as the old ubuntu bugzilla was
<wgrant> It wasn't *that* bad.
<kb9vqf> If /var/tmp/poppy does not exist, should I just create it?
<wgrant> Yes.
<wgrant> I thought that was documented. Hmm.
<wgrant> Yeah, the first process-upload.py call in the "Upload a source to the PPA" section will create /var/tmp/poppy.
<wgrant> Or should, at least.
<kb9vqf> It gave me an error until I created the directory manually
<wgrant> Hrmph.
<kb9vqf> One other question...how does Launchpad handle Email sending?  Can I just point it to my SMTP server?
<wgrant> Pretty much. But you'll need to make sure any email addresses in the code or DB are really what you want, and aren't going to spam people.
<kb9vqf> OK.  And the configuration file for that is schema-lazr.conf or something else?
<wgrant> You shouldn't alter schema-lazr.conf itself.
<wgrant> You should probably create a new config, perhaps based on the development one.
<wgrant> Overriding more things.
<kb9vqf> Ah, OK
<wgrant> If you look really hard you might be able to find the old production configs and use bits of them as examples ;)
<kb9vqf> I need to catch some sleep...the build farm looks like it is active and cron jobs are partially set up...I'll do the actual build test tomorrow
<kb9vqf> Thanks for all your help so far!
<deryck> wgrant, ping.  still around?
<wgrant> deryck: Sure.
<wgrant> Hm, translation template jobs are on now?
<henninge> wgrant: you mean if they are working?
<wgrant> I haven't seen dozens on the farm before.
<wgrant> But most of the i386 builders are doing them at the moment.
<henninge> wgrant: well, they are restricted to run only on i386
<wgrant> Well, yes, but I hadn't seen any on production for weeks.
<wgrant> Also, are they restricted to only happen every $INTERVAL?
<wgrant> Or will they run for every rev?
<henninge> AFAIK they run on every rev.
<henninge> jtv might be surer about the details.
<wgrant> erm.
<jtv> hi wgrant
<wgrant> Evening jtv.
 * henninge goes to lunch
<jtv> wgrant: we still need a lot more logging and tracking for these jobs.  :/  One thing we found some time ago was that there's been a change in gettext that makes our front-end vetting think a lot of pure-gettext branches are intltool-based.
<wgrant> jtv: Yeah, well, we have the infrastructure to keep track of them after they're done now.
<wgrant> Once that's done, it probably becomes easier to impose quotas, too.
<wgrant> Since you can't actually tell at the moment if an attempt has been made lately...
<jtv> It'd be nice to have some form of bundling.
<wgrant> Hm?
<jtv> Where we don't record separate jobs for every revision, but the need to re-process a given branch.
<wgrant> Ah.
<jtv> If you get a dozen revisions in quick succession, there shouldn't be a need to run a dozen jobs.
<wgrant> Right.
<wgrant> That seems easy enough to do, even now.
<jtv> Perhaps, yes.
<wgrant> Is the porting of SPRBs and TTBJs to the new infrastructure going to happen in the foreseeable future?
<jtv> Frankly, I don't know.  We're focused on something else at the moment.
<bigjools> wgrant, jtv: I expect buildd admins will insist so that those jobs show up in the history
<jtv> bigjools: it'd definitely make sense to do, yes.  And we want to do it... question is when.
<wgrant> Plus it means we can delete a whole lot of code.
<wgrant> And simplify a whole lot of other code.
<wgrant> And make logging suck less.
<bigjools> win win win
<bigjools> henninge has a decision to make ;)
<henninge> bigjools: I would love to if I knew about what ...
<bigjools> henninge: "porting of SPRBs and TTBJs to the new infrastructure"
<bigjools> TTBJs in your case
<henninge> also, I am currently listening to a Microsoft general manager talking about "Microsoft and Opennes" ...
<bigjools> Microsoft has always been open
<bigjools> ...to making more money
<Ursinha> Chex, gary_poster, rockstar, bigjools, henning, sinzui, gmb: production meeting on #launchpad-meeting @ Freenode in 30 minutes
<Chex> Ursinha: thanks for reminder
<rockstar> Ursinha, correction: 28 minutes.
<gary_poster> thanks
<rockstar> :)
<Ursinha> rockstar, thanks for that :)
<gmb> Ursinha, Thanks.
<Ursinha> Chex, gary_poster, rockstar, bigjools, henning, sinzui, gmb: production meeting on #launchpad-meeting @ Freenode now
<kb9vqf> Any idea why Launchpad won't accept my key import?
<kb9vqf> "Launchpad could not import your OpenPGP key"
<kb9vqf> Any idea why Launchpad won't accept my key import?
<kb9vqf> "Launchpad could not import your OpenPGP key"
<kb9vqf> This is weird... "format '1.0' is not permitted in lucid."
<kb9vqf> What does that mean?
<kb9vqf> It came from a PPA upload rejection notice
<kb9vqf> I have sucessfully uploaded a package to my PPA on my local Launchpad installation and it is marked as published, but for some reason it won't start building
<kb9vqf> It's stuck on "Needs building"
<kb9vqf> Any thoughts?
<maxb> Have you read wgrant's documentation on making a local soyuz work?
<kb9vqf> Yes, I followed it
<kb9vqf> It seems like is almost working, the build machines show up in /+builds as idle, I added both the i386 and amd64 chroot images
<kb9vqf> But the build itself does not fire off
<kb9vqf> maxb: is there anywhere I should look for logging info?
<maxb> Not sure, sorry.
<kb9vqf> maxb: If I haven't signed the code of conduct could it do thise?
<kb9vqf> this?
<maxb> huh. I'd expect it to refuse the upload
<kb9vqf> maxb: I'd like to try signing it and see what happens, but I get "Launchpad could not import your OpenPGP key" when I try to add my key
<kb9vqf> Any thoughts on that?
<maxb> I'm sure wgrant documented how to sign the CoC
<kb9vqf> maxb: Well I think I know why it accepted my upload; I ran the upload acceptor with absolutely-anything
<kb9vqf> But I don't know if the CoC is checked again afterwards before the package is built
<kb9vqf> So what piece of Launchpad actually dispatches the builds?
<mars> lifeless, ping, is there any chance you could approve https://code.edge.launchpad.net/~tseaver/subunit/add_distutils_support/+merge/26653 ?
<mars> lifeless, s/approve/review/
<lifeless> yup its on my todo
<mars> lifeless, cool, thank you.
<lifeless> is this work relasted for you? its currently in my personal pile
<lifeless> mars: ^
<mars> lifeless, relasted?
<mars> ah, related
<lifeless> related
<mars> lifeless, yes, it is blocking my landing of https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting/+merge/27086
<lifeless> how so ?
<mars> if you read Tres' comment, you'll see that he is waiting for PyPI subunit.
<mars> When that is available, then zope.testing's setup.py and buildout.cfg can be updated with conditional subunit[test] dependencies
<mars> And then my patch will be sane: you will need subunit to run the zope.testing test suite
<lifeless> ah
<lifeless> there is a test tools bug thats related
<mars> as Tres put it, then we can "ensure that the [subunit] features don't bitrot"
<mars> gary_poster, ec2 land did not eat my branch.  It just took an hour to deliver my change from ec2 to PQM :/
<mars> It fell through a timerift on the way over or something
<gary_poster> mars, well, time rifts are interesting, and a lock of eating the branch is a good thing :-)
<gary_poster> lack
<kb9vqf> So what piece of Launchpad actually dispatches the builds?
<lifeless> sidnei: around ?
<lifeless> tres has said he is ok with https://code.edge.launchpad.net/~mars/zope.testing/fix-subunit-utf8-traceback-reporting/+merge/27086 landing as long as you or someone else with subunit tests it; mars has said it may break other builds right now as the additional tests he adds aren't conditional.
<lifeless> mars: you're going to fix that right ?
<mars> lifeless, yes, I certainly will now
<sidnei> lifeless, im about to leave. i can land it in the morning.
<lifeless> sidnei: cool
<lifeless> thanks
<thumper> rockstar: call now?
<rockstar> thumper, sure.
<thumper> rockstar: and I suppose we have to mumble?
<rockstar> thumper, yes.... :(
<jml> lifeless: fwiw, that thing I did to make a pretty graph has grown a script that makes similarly pretty graphs given a csv file like YYYY-MM-DD,number,number,number,...
<lifeless> nice
<jml> I guess I should productize it, or something.
<lifeless> meh
<lifeless> tweet it
<lifeless> your job is done
<jml> hah
<ajmitch> yay for documentation about testing soyuz
<jml> at least now it has a README
<jml> good bye
<lifeless> fly well
<wgrant> ajmitch: What about it?
<ajmitch> wgrant: that I'm glad it's there, it was helpful
<wgrant> Ah, you've actually tried it?
<ajmitch> yes
<ajmitch> at least the accepting source packages & publishing them, not the building
<ajmitch> so that I could test out this change to sync-source.py:
<ajmitch> http://bazaar.launchpad.net/~ajmitch/launchpad/fakesyncs/revision/10992
<wgrant> Ah, right.
#launchpad-dev 2010-06-11
<maxb> createlang: language installation failed: ERROR:  could not access file "$libdir/plpython": No such file or directory
<maxb> hrm
<maxb> got past that, only to run into
<maxb> AssertionError: /usr/share/postgresql/8.4/contrib/tsearch2.sql does not exist
<maxb> ah
<maxb> with two versions of pgsql in play, lp-dev-deps does not necessarily ensure you have all the addons for each version installed
<thumper> do I still need to use from __future__ import with_statement?
<thumper> prolly do right?
<lifeless> http://www.python.org/dev/peps/pep-0343/
<lifeless> while we deploy with 2.5, you do
<lifeless> when we deploy with 2.6, its always present
<lifeless> (see under transition plan)
<thumper> lifeless: yeah I knew that
<thumper> I was kinda thinking out loud
<thumper> since we are still deploying on 2.5 for now
 * thumper EODs for now, back for the interview in 3 hours
<lifeless> thumper: interview ?
<thumper> lifeless: yeah, talking to an ozzie in 2.5 hours
<lifeless> the code team position ? good luck
<kb9vqf> Why is my buildd remote builder complaining about "/var/debbuild/srcdep-lock is not a directory"?
<kb9vqf> I verified that it is a directory in the chroot environment
 * kb9vqf realized he was yakking in the wrong channel for the past several hours :-P
<kb9vqf> More worrying is this:
<kb9vqf> 2010-06-11 01:41:35-0500 [-] Build log: cat:
<kb9vqf> 2010-06-11 01:41:35-0500 [-] Build log: build-progress
<kb9vqf> 2010-06-11 01:41:35-0500 [-] Build log: : No such file or directory
<lifeless> kb9vqf: I'd assume that you don't have that directory etc
<kb9vqf> My troubleshooting is indicating that sbuild is exiting immediately upon invocation
<kb9vqf> Which would explain the missing status files
<kb9vqf> Manually running it also causes an immediate exit with no error messages
<kb9vqf> Here we go...die "conf::mailto not set\n" if !$conf::mailto;
<kb9vqf> ^^^ That was in /usr/bin/sbuild
<kb9vqf> Commenting that line out fixed the builder
<kb9vqf> So, what should I have done instead of commenting it out? ;-)
<lifeless> file a bug
<lifeless> saying the following:
<lifeless>  - the situation should have been more obviously debuggable
<lifeless>  - and include enough pointers for you to have found docs
<lifeless> what you should do to fix it, is to set the admin address to send mail to
<kb9vqf> Yeah, silent exiting is not fun :-P
<kb9vqf> How would I set that address
<lifeless> Dunno
<kb9vqf> Well, the builder is hanging now at this
<kb9vqf> Finished at 20100611-0158
<kb9vqf> Build needed 00:00:17, 133368k disk space
<kb9vqf> Can't open average time db /var/debbuild/avg-build-times
<kb9vqf> Can't open average space db /var/debbuild/avg-build-space
<kb9vqf> So I guess I'll need to find some way to set it
<kb9vqf> Hmmm....actually I'm not sure that has anything to do with the mail address--I don't see sbuild in ps aux any more
<kb9vqf> After the package is built it just uploads to Librarian, correct?
<kb9vqf> Something isn't generating the changes file
<kb9vqf> Causing this:
<kb9vqf> 2010-06-11 01:58:51-0500 [-] Iterating with success flag 0 against stage SBUILD
<kb9vqf> 2010-06-11 01:58:51-0500 [-] Returning build status: OK
<kb9vqf> 2010-06-11 01:58:51-0500 [-] unexpected error in processEnded
<kb9vqf> Followed by this:
<kb9vqf> exceptions.IOError: [Errno 2] No such file or directory: '/home/buildd/build-bb6a61d51a36b9892717c7a7f03321b7024338e9/<packagename>.changes
<kb9vqf> Looks like gatherResults is failing as a result, and the builder gets stuck
<kb9vqf> On further inspection this shows up in the builder log
<kb9vqf> Can't find redmond-default-settings-kde3_10.04-7_amd64.changes -- can't dump infos
<adeuring> good morning
<kb9vqf> More fun...the changes file is present in /home/buildd/build-bb6a61d51a36b9892717c7a7f03321b7024338e9/chroot-autobuild/build/buildd/<packagename>.changes
<kb9vqf> So either a copy or symlink operation is not being run
<kb9vqf> My hunch is a symlink is missing
<kb9vqf> Maybe someone who knows the buildd system could be of help here ^^^ ;-)
 * wgrant returns.
<kb9vqf> wgrant: Any idea why the buildd would not be symlinking or copying the .changes file as described above?
 * kb9vqf is so close to having this working...
<wgrant> kb9vqf: Which version of Ubuntu is this?
<kb9vqf> Lucid
<wgrant> Your earlier mailto error suggests that you don't have a /home/buildd/.sbuildrc, which the package does create, so you've probably done something very strange.
<kb9vqf> I just ran the commands in your howto
<kb9vqf> Dunno what would be strange there
<wgrant> Do you have a /home/buildd/.sbuildrc?
<kb9vqf> No!
<kb9vqf> Odd
<kb9vqf> How is it supposed to be created?
<kb9vqf> Oh wait a minute...I know how this broke
<kb9vqf> I tried moving the home directory around a few times
<wgrant> launchpad-buildd's postinst creates it.
<wgrant> noodles775: Morning.
<kb9vqf> I did a dpkg-reconfigure and the file was recreated successfully
<kb9vqf> Now I get exceptions.AttributeError: 'BinaryPackageBuildManager' object has no attribute '_subprocess'
<wgrant> Restart the buildd.
<wgrant> You may have to also unmount and remove /home/buildd/build-SOMESHA1.
<wgrant> And then maybe restart it again.
<wgrant> Depending on how bad a start it's wedged in.
<wgrant> s/start/state/
<kb9vqf> OK
<kb9vqf> Now that I know about the dot files I'll be more careful with the home directory
<kb9vqf> Build is in progress...
<noodles775> Hi wgrant
<kb9vqf> wgrant: Same problem: http://pastebin.ubuntu.com/448134/
<kb9vqf> At least the Email is resolved though
<kb9vqf>  /Email/Email problem/
<wgrant> noodles775: So, I had a look at that new massive getBuildQueueSizes query that landed a few days ago... and it seems to me that it can be replaced with a join over just BuildQueue, Job and Processor, rather than BuildPackageJob, BinaryPackageBuild, PackageBuild, DistroArchSeries, BuildFarmJob, Archive, Processor and BuildQueue.
<wgrant> Since BuildQueue has virtualized and processor fields, and Job has the estimated duration, we don't need to join anything else.
<wgrant> (plus the smaller solution works for generalised jobs)
<mwhudson> morning
<wgrant> So I'm wondering if that was just missed, or if there is some reason that I've missed.
<wgrant> kb9vqf: Your chroot is the architecture it says it is?
<kb9vqf> wgrant: Pretty sure
<wgrant> Can you pastebin the whole build log?
<kb9vqf> The problem seems to be that /home/buildd/build-bb6a61d51a36b9892717c7a7f03321b7024338e9/chroot-autobuild/build/buildd/redmond-default-settings-kde3_10.04-7_i386.changes is not linked to /home/buildd/build-bb6a61d51a36b9892717c7a7f03321b7024338e9/redmond-default-settings-kde3_10.04-7_i386.changes
<kb9vqf> Yeah, hang on
<noodles775> wgrant: sounds great. It may have just been missed - I asked stub for help and he just optimised what the query was currently doing. Do you want to update the bug with that info and we can try it out when there's time. (or if you've already got a branch that passes those tests, just link it).
<noodles775> wgrant: and thanks for looking deeper rather than just accepting a solution that works!
<kb9vqf> wgrant: The log files are huge; pastebin won't accept them
<wgrant> Uh, well, the tests are pretty bad, but it passes them.
<wgrant> kb9vqf: OK. So /home/buildd/build-bb6a61d51a36b9892717c7a7f03321b7024338e9/chroot-autobuild/build/buildd/redmond-default-settings-kde3_10.04-7_i386.changes exists, with the right arch and everything?
<kb9vqf> Yes
<kb9vqf> There's even a correct .deb file sitting in /home/buildd/build-bb6a61d51a36b9892717c7a7f03321b7024338e9
<wgrant> Um.
<wgrant> So the .deb got copied/linked across, but not the .changes?
<kb9vqf> Yes
<kb9vqf> Weird, huh?
<kb9vqf> And this with no error messages in the logs
<wgrant> sbuild says nothing odd about it?
<kb9vqf> Nope
<kb9vqf> The first sign of trouble is when the .changes file can't be found
<kb9vqf> wgrant: That might not be entirely true:
<kb9vqf> 2010-06-11 01:58:50-0500 [-] Build log: Can't find redmond-default-settings-kde3_10.04-7_amd64.changes -- can't dump infos
<kb9vqf> 2010-06-11 01:58:50-0500 [-] ******************************************************************************
<kb9vqf> 2010-06-11 01:58:50-0500 [-] Build log: Built successfully
<kb9vqf> 2010-06-11 01:58:51-0500 [-] Build log: Purging chroot-autobuild/build/buildd/redmond-default-settings-kde3-10.04
<kb9vqf> 2010-06-11 01:58:51-0500 [-] Build log: ------------------------------------------------------------------------------
<kb9vqf> I missed it earlier
<wgrant> Aha.
<wgrant> So it is an arch conflict.
<kb9vqf> So I have the wrong chroot image?
<wgrant> Hm. Do you have i386 and amd64 lp-buildds on the same system?
<kb9vqf> Yes
<wgrant> Ah.
<kb9vqf> Well, I configured them that way at any rate, all based off of the single amd64 launchpad-buildd installation
<kb9vqf> Separate systems in the future?
<wgrant> I think that when I told you earlier that that was OK, I was remembering slightly incorrectly: /home/buildd/.sbuildrc contains an arch definition. I must have hacked that out to use 'dpkg --print-architecture' or similar.
<kb9vqf> Yeah, I noticed that when you originally mentioned the .sbuildrc file
<kb9vqf> I was wondering if it might cause problems
<kb9vqf> I have another system I can use for i386
<kb9vqf> I'll just set that up tomorrow
<kb9vqf> wgrant: I temporarily changed the arch type in .sbuildrc and it actually works!
<kb9vqf> Thanks a bunch!
 * kb9vqf has learned a lot about Launchpad in the process
<wgrant> Great.
<wgrant> noodles775: Given the recent history, it might be nice to see that this query doesn't perform ridiculously badly on production like the old one. http://paste.ubuntu.com/448145/ has the new and old queries -- how can I go about verifying there's no regression?
<noodles775> wgrant: Given that the current query will need to be updated to support general builds, I'd personally create a new bug, attach your branch and as we go through the normal process, part of it would be to involve stub to test the query on production.
<noodles775> If you don't have time, let me know and I'll do it.
<wgrant> I've just finished my last exam for a week, so I've time.
<wgrant> Preparing a branch now.
<noodles775> Great. Thanks!
<bigjools> congrats wgrant
<bigjools> you get a week off :)
<wgrant> Heh.
<bigjools> wgrant: Any jobs with no processor are currently always dispatched to i386  builders aren't they?
<bigjools> oh actually maybe we don't create those any more thinking about it
<wgrant> bigjools: Right, they are dispatched to any builder.
<wgrant> But, um, we can't do that any more.
<bigjools> we fixed something to send all TTBJ and SPRBJ to i386
<bigjools> I can;'t remember if we forced i386 or changed the dispatcher code
<wgrant> Right, they set their BQ.processor to i386.
<bigjools> to assume it
<bigjools> ok
<wgrant> We might be better off just implementing multi-arch builders, I think.
<maxb> wgrant: I seem to remember you wrote a more up to date version of http://williamgrant.id.au/f/1/2009/soyuzness.html, but I can't seem to find it. Can you direct me?
<wgrant> maxb: https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally may interest you.
<wgrant> It has evolved into that.
<maxb> ah. See, I just remembered I needed to google for "Soyuzness" :-)
<mwhudson> wgrant: do you know where the code in lp is to make copy archives is?
<wgrant> mwhudson: scripts/populate-archive.py
<mwhudson> wgrant: ta
<wgrant> And lp.soyuz.scripts.populate_archive
<mwhudson> right, given a start pulling the thread is easier :)
<wgrant> Right.
<wgrant> maxb: What's causing you to stray into dangerous Soyuz territory?
<maxb> curiosity :-)
<maxb> potentially of the "killed the cat" variety :-)
<wgrant> Heh.
<deryck> Morning, all.
<wgrant> gmb: Hm, so this proposed solution will see the From column in my mail client rendered useless?
<gmb> wgrant, bear with me a sec, OTP
<wgrant> I can perhaps see some benefit in avoiding use of the author's email address.
<wgrant> But filling a field with completely redundant and useless information probably isn't the right solution.
<gmb> wgrant, Hmm. I can see your point, though I confess that if the body of the email told me who had made a change or comment I wouldn't care too much about the From field. As it is, I only care about it when I get a message from the bug email address that doesn't tell me who actually made a change.
<gmb> wgrant, It would be worth you adding your voice to the lp-dev discussion so that other people with more brains than I could discuss your points.
<wgrant> gmb: When I'm looking through my bug mail, I am going to be more interested in responding quickly to a bug reported by a user than a developer.
<wgrant> If I want to look back through a thread for an email, I can now do it easily simply by remembering who wrote it.
<gmb> wgrant, Yes, I see your point.
<wgrant> If you obliterate the sane From value, every email relating to a bug looks identical until you open it.
<gmb> wgrant, True. And yet, the person whose name the email is sent in didn't actually send an email; it's an auto-generated notification sent by the application. Most webapps don't purport to send notifications from a user directly, AFAIK (I'm trying to think of one that isn't Launchpad that does).
<wgrant> gmb: Not many do, no. And it's something that I love about LP.
<wgrant> Because it doesn't gratuitously stuff its name in prominent places on all of my mail.
<wgrant> It's in my Launchpad Bugs folder -- of course it's from Launchpad.
<wgrant> I don't need another column to tell me that.
<wgrant> The user performed the actions. They wrote the text.
<wgrant> They are surely the author.
<wgrant> And the author should be in From.
<deryck> wgrant, I really think this is the correct way to send email from Launchpad bugs.  barry agrees.  There are technical benefits to doing so (e.g. batching).  The question really now is:  does the correctness/benefits outweight the loss of current functionality if we go this route?  Or does current functionality trump correctness/benefits?
<deryck> wgrant, I know what you would say on this. :-)  We need others input, too.
<wgrant> Erm, you're going to batch multiple comments together?
<deryck> wgrant, potentially, yes.  But I suspect if so, it would be a configurable option.
<wgrant> Can we please have a "please don't screw with my mail, I like it sane" flag?
<bigjools> one man's sane is another man's madness
<deryck> wgrant, shall I call it the "do you prefer your Launchpad like wgrant" option?
<wgrant> bigjools: But this madness has been in place for five years and caused few complaints...
<wgrant> And it makes LP a lot more transparent.
<wgrant> Which is probably a good thing.
<deryck> forging an email address is transparent? ;-)
<bigjools> it's equivalent to people who like ML batch
<wgrant> It means I can treat a bug as an email conversation. The bug tracker has a really nice email interface.
<bigjools> I personally prefer individual emails too, but many people don't
<deryck> bigjools, individual emails for comments, to be precise, no?
<bigjools> yep
<bigjools> you already coalesce status changes
<didrocks> hey
<deryck> bigjools, right.  but just pointing out it's a preference for a certain kind of individual email.
<bigjools> absolutely
<bigjools> I'm not disagreeing :)
<deryck> :-)
<wgrant> What are the benefits in batching more?
<deryck> wgrant, less mail sent
<wgrant> Er.
<wgrant> So I don't get notified about bug activity for hours?
<wgrant> That seems like a bug.
<deryck> wgrant, I take your point, I really do.  And I agree if we change this, it is a serious change, and we should not do that lightly and get lots of input on it.
<wgrant> Perhaps Launchpad needs to abandon its one-size-fits-all policy.
<deryck> wgrant, no minutes still.  Just the comment is no longer the boundary.  And as I said, if we go this route, I seriously doubt we make it the default.  It's an option.
<wgrant> I am trying to think of parts of the application that are configurable.
<wgrant> All I can think of is the thing to show descriptions at the bottom of bugmail.
<deryck> wgrant, branch subscriptions.
<wgrant> Ah, true.
<wgrant> I am glad that you are going to get lots of input.
<wgrant> Because it didn't look like it a few days ago when a branch was proposed for merging without any discussion whatsoever.
<deryck> wgrant, yes, we jumped the gun there, but didn't think this would be controversial.  Now we know and have changed our plans.
<wgrant> Great.
<deryck> wgrant, though to be fair, you are the only one passionate about the current form.  Most say "I like the current form" but don't argue why it is the one true way.
<deryck> most who take this opinion, rather
<wgrant> noodles775: Is the date_started on those 9 ill-fated builds going to be populated, in addition to the code fix?
<ScottK> wgrant: Another point I would make is that the conversational nature of bugmail that you like may also encourage people to think of LP bugs like a forum.  That can certainly be problematic.
<noodles775> wgrant: I wasn't planning to - unless there's a need. If the situation can happen again during the next release, my main concern is that it doesn't block peoples development efforts. Do you see a need?
<noodles775> I'd be keen to spend any more time on that bug figuring out how we can ensure the data inconsistency doesn't happen at the db level.
<wgrant> noodles775: It's not going to happen again, is it? It was a one-off issue in the migration.
<wgrant> We can make the data less inconsistent easily with one-of setting of 9 values now. There is probably no other data like this. Eliminating it has to be a good thing.
<noodles775> wgrant: I wasn't certain. As the data should have been consistent when the migration started. But haven't looked further.
<noodles775> True.
<noodles775> (Although, there is a lot of old data in a similar state... no date_started but successfully built).
<wgrant> No date_started but successfully built is fine.
<wgrant> No date_started, but successfully built *and with date_finished or builder* is not.
<wgrant> gina creates successful builds with no dates or builders.
<wgrant> My vision is that we will eventually be able to instead make it create BinaryPackageBuilds without a corresponding BuildFarmJob, but we'll see...
<deryck> hmmm, we don't seem to be minifying js anymore.  at least in the lp.APP.javascript version.
<maxb> deryck: In a clean branch or not? I've noticed some silly errors coming from that part of the build step when it's run in a branch where it has already run once.
<maxb> Of course stupid stupid shhh.py will probably obscure this unless you turn it off :-(
<deryck> maxb, yeah, I wonder if I have something hanging around.  Ran make clean_js though
<deryck> good call about turning off SHHH though.  The noise may help me understand here.
<gmb> mars, Is ec2 land safe to use now? I don't remember.
<mars> gmb, it should be
<gmb> mars, Cool, thanks.
<mars> gmb, just haven't written the list yet.  Was waiting for buildbot to pass it
<gmb> mars, Okay. In that case I'll do it the old fashioned way until you've confirmed it's fixed.
<mars> sinzui or gary_poster, ping, do you know if any work was done on the +openid URL path recently?
<gary_poster> mars, on call, but not to my knowledge
<gary_poster> ask salgado?
<sinzui> mars, no work on path, but I changed merge rules to avoid attempts to login to old Lp accounts
<mars> sinzui, ok.  Not sure if that would do it.  buildbot and local are failing because the +openid URL has started returning a 404.  Strange because none of the recent landings appear related.
<sinzui> I saw that before
<sinzui> mars, did someone remove the old openid domain from the testrunner?
<mars> The domain itself?  I don't know.  This tries to stay on the same domain by the looks of things
<sinzui> mars, When we switched to Ubuntu SSO, we added the new domain for dev, but did not remove the old domain. Losts of tests use it. To remove the old domain, the tests need to switch to the new domain.
<mars> salgado, ping, would you happen to know if any work was done on the launchpad +openid URL path recently?
<mars> sinzui, do you know where that would be configured?
<salgado> mars, don't know
<sinzui> mars, I think +openid is the defunct url. it will not work in dev, but can work if the old vhost is started
<sinzui> mars, testopenid.dev is new and in dev
<sinzui> mars, openid.launchpad.dev and ubuntu-openid.launchpad.dev are old
<mars> sinzui, odd.  IIRC the windmill tests don't cross domains when negotiating OpenID.
<sinzui> That may be why we kept the old urls
<sinzui> but someone could have disabled the servers in the testrunner
<mars> now it looks like the logout button my be broken on dev as well
<sinzui> mars, it has been broken for 4 weeks,, maybe more
<mars> sinzui, I wonder why it is failing now then...
<sinzui> Was there a config change made recently?
<mars> going back to 10988 - no, nothing
<kb9vqf> Is there a web interface where I can manage distroseries?
<kb9vqf> In the database where are the pockets defined?
<kb9vqf> Creating a new distribution...I figured out the distroseries, distroarch, and distribution tables, but poppy still won't accept uploads (says it can't find the distroseries)
<kb9vqf> Did I miss a table?
<kb9vqf> Creating a new distribution...I figured out the distroseries, distroarch, and distribution tables, but poppy still won't accept uploads (says it can't find the distroseries)
<kb9vqf> Did I miss a table?
<kb9vqf> Or, how would I create a new distribution and make it accept uploads?
<kb9vqf> Any ideas on why this might happen when trying to upload to a manually created distroseries?
<kb9vqf> 2010-06-11 17:30:11 INFO    Creating lockfile: /var/lock/process-upload-insecure.lock
<kb9vqf> 2010-06-11 17:30:21 INFO    Processing upload redmond-default-settings-kde3_10.04-75656t_source.changes
<kb9vqf> 2010-06-11 17:30:31 ERROR   Exception while accepting:
<kb9vqf>  'NoneType' object has no attribute 'id'
<kb9vqf>  -> http://quickbuild.pearsoncomputing.net:58080/155/cKHh1gV2I4VAtvWCgZvVlrkgYAI.txt ('NoneType' object has no attribute 'id')
 * kb9vqf cant seem to find what is missing in the DB
<deryck> rockstar, ping
<rockstar> deryck, pong
<deryck> rockstar, hey, dude.  You did the work to get js building from lp.$APP locations, right?  the lp-dep.py script, I mean.
<rockstar> kb9vqf, is there a distro connected to the distroseries?
<rockstar> deryck, yup, that's me.
<deryck> rockstar, it doesn't minify the files.  I happen to be the lucky team to hit the 500k limit today because of it. :-)  So I'm playing with:  http://pastebin.ubuntu.com/448352/
<deryck> rockstar, what do you think of that?
<rockstar> deryck, wait, you want to minify the files that are only used in dev?
<deryck> rockstar, no, I want launchpad.js to combine minified files, not the main files.
<rockstar> deryck, ah, crap, I didn't realize it was doing that...  :/
<rockstar> deryck, so I'm happy with launchpad.js combining minified files, but I was using a symlink because I don't want minified files used during development.
<deryck> rockstar, yeah, ./bin/jsbuild normally takes care of it by passing in lib/canonical/launchpad/javascript as the BUILD_DIR
<deryck> rockstar, right.  that makes sense.  This still links the non-min files in a real "bugs" or "code" directory alongside the debug and min files.
<rockstar> deryck, okay, so long as we still have access to those unminified files during dev (otherwise, debugging would be a bitch)
<deryck> rockstar, this is what link_and_minify does:  http://pastebin.ubuntu.com/448354/  (for the bugs directory as an example)
<rockstar> deryck, ah, okay.  I'm good with that change then.
<deryck> rockstar, ok, cool.  Thanks!
<deryck> mars, can you look at the scrollback about this build change, since it relates to js.  You feel okay about what I'm proposing?
<mars> deryck, those are ugly symlink paths, but if that gets things minified again, then +1 from me.
<deryck> mars, ok, cool.  thanks.  The paths are just generated from lazr js builder.
<mtaylor> hey all - I'm trying to run make schema from a recent pull and I'm getting Error: Couldn't find a distribution for 'zope.testing==3.9.4-p1'.
<mars> mtaylor, checking
<maxb> mtaylor: cd ~/launchpad/lp-sourcedeps/download-cache; bzr update
<mars> mtaylor, a pull?  You didn't update the source dependencies
<mars> (what maxb said :)
<mtaylor> gotcha. thanks
<deryck> mars, care to review my build changes for js?
<kb9vqf> rockstar: Yes
<rockstar> kb9vqf, what's the distribution?
<kb9vqf> rockstar: Ubuntu
<mars> deryck, sure
<kb9vqf> I wanted it to be Debian, but that just pain wasn't workink
<rockstar> kb9vqf, so I guess that eliminates the easy stuff.
<deryck> mars, thanks!  https://code.edge.launchpad.net/~deryck/launchpad/minify-app-specific-js-592795/+merge/27395
 * kb9vqf can't type
<rockstar> kb9vqf, yeah, Launchpad doesn't support any distribution outside Ubuntu.
<kb9vqf> Yeah I know
 * kb9vqf has been hacking some along those lines
 * maxb stares at the bzr history of launchpad-buildd and says "What?!"
<maxb> Does lamont keep some other branch somewhere which is what really gets deployed?
<mars> rockstar, your 500Kb JS warning just saved us a fair bit of trouble :)
<mars> Kudos for that
<rockstar> mars, That makes me quite happy then.  :)
<rockstar> maxb, what's the issue?
<maxb> only that it's really confusing to find unrelated production changes to launchpad-buildd landing back into devel via arbitrary feature branches
<rockstar> maxb, only the packaging information.
<rockstar> maxb, and that change keeps the buildds from killing itself.
<maxb> "Do not check implicit pointers on 32-bit architectures" landed back onto devel as part of a merge "Enable building from recipe in the builder."
<rockstar> maxb, oh, no idea there...
<elmo> maxb: launchpad-buildd development is divorced from launchpad development; it sucks, and I'd love for it to stop, and I believe that is the medium term plan
<rockstar> elmo, it's more "estranged" and we need to actually get the divorce papers signed.
<mars> gary_poster, buildbot is green, guess the restart worked!
<gary_poster> mars, cool.  Yeah, ec2 gets like that sometime. :-/
<jelmer> win 23
<wgrant> maxb: It's mostly developed in tree now.
<wgrant> maxb: Since LP has mostly taken ownership.
<wgrant> This has happened mostly within the last 6 months.
<wgrant> But yes, historically the LP tree would occasionally get an update to the production version.
#launchpad-dev 2010-06-12
<kb9vqf> wgrant: You around?
<kb9vqf> Everything is working very well, except that package uploads with arch=all only build under i386
<kb9vqf> I can't seem to get the system to accept an amd64 build
<kb9vqf> distroarchseries looks OK
<kb9vqf> There are 8 active amd64 builders
<kb9vqf> s/active/idle/g
<kb9vqf> It would have helped to look in the architecture admin page before asking stupid questions ;-)
<kb9vqf> amd64 PPA builds were disabled
<wgrant> kb9vqf: Also, 'Architecture: all' packages are only built on i386.
<wgrant> Perhaps you were looking for 'Architecture: any'
<kb9vqf> Yeah, that was it
<kb9vqf> I wrote the wrong thing here by accident ;-)
<wgrant> Otherwise it's all running OK?
<kb9vqf> Working great!
<kb9vqf> I'd send you a link but I don't have external access up yet
<wgrant> Excellent.
<kb9vqf> Its primary purpose is to build Debian packages for Trinity
<kb9vqf> lenny/squeeze/etc
<wgrant> It takes a bit of effort to get Debian builds working adequately.
<kb9vqf> I've almost got it...386 works well, amd64 chroot still has a couple bugs
<wgrant> There are a couple of things hardcoded to Ubuntu, and the archive model is slightly different, but apart from that it works.
<kb9vqf> sed is my friend
<kb9vqf> ;-)
<wgrant> (once you force the correct external dependencies)
<wgrant> Heh.
<kb9vqf> wgrant: You can try accessing it at http://quickbuild.pearsoncomputing.net
<kb9vqf> Hang on...somehting got fouled up
<wgrant> Heh, it wants a MythTV password...
<kb9vqf> Yeah, the proxy isn't up
<kb9vqf> Hang on while I recreate the missing config file
<kb9vqf> wgrant: OK, try it now: http://quickbuild.pearsoncomputing.net/
<kb9vqf> bugs./code. etc. won't work yet because of the proxy, but you get the idea :-)
<kb9vqf> Looks like a bunch of stuff isn't making it past the proxy actually
<kb9vqf> Hmmm
<wgrant> Not bad.
<kb9vqf> Oh well, a project for tomorrow
<kb9vqf> See any copyright problems"?
<wgrant> You're not using canonical-identity-provider for OpenID, and it still works!?
<wgrant> I am surprised.
<kb9vqf> :)
<kb9vqf> Took me a day to figure that whole system out
<wgrant> I guess lots of stuff is trying to use HTTPS.
<kb9vqf> Yeah, I really have to give this thing my second IP addres
<kb9vqf> Which I can't use yet because the gateway machine isn't online ATM
<kb9vqf> Hence tomorrow's project ;-)
<wgrant> Ah, the OpenID provider is on 443 already?
<kb9vqf> Yeah
<kb9vqf> http://openid.pearsoncomputing.net will give you the openid provider in non-secure mode though
<wgrant> Hmm. It's a wildcard cert, so you can still use vhosts on the same IP.
<kb9vqf> Apache is the problem really
<kb9vqf> I have other SSL (non-proxy) sites on this public IP
<kb9vqf> so it barfs when I try to do a secure proxy
<wgrant> Ah.
<wgrant> Hm, so you've created a lenny series in Ubuntu?
<wgrant> It's easy enough to construct a proper Debian PPA.
<wgrant> Only a couple of lines of code need changing.
<kb9vqf> I couldn't find them...the PPA kept rejecting my uploads (distroseries not found)
<kb9vqf> Hints are appreciated ;-)
<wgrant> You need to create the Debian distroserieses, create a PPA, tweak the PPA in the DB to point to Debian instead... and then upload to ~somebody/someppa/debian instead.
<wgrant> I think that should just about do it.
<wgrant> It's possible that you won't be able to access that through the web UI, though. The PPA traversal might be restricted to Ubuntu archives.
<kb9vqf> Well I can track that down easily enough
<kb9vqf> I'm going to take the public site down for now as it's completely broken ATM
<wgrant> Ah, yes, lp.soyuz.browser.archive.traverse_named_ppa.
<wgrant> That's where the URL traversal is hardcoded to Ubuntu.
<kb9vqf> OK...for now I might just leave it alone, as it works pretty well as-is (minus the claims that Lenny is in Ubuntu)
<kb9vqf> The web interface is critical
<kb9vqf> Makes my life a whole lot easier ;-)
<kb9vqf> And I'd like Debian and Ubuntu to operate side by side on the same machine for now (which they do ATM)
<wgrant> In that case, you could just remove the distribution constraint.
<kb9vqf> Not sure what you mean
<wgrant> traverse_named_ppa finds a PPA by (owner, distro, name).
<wgrant> If you remove the distro from the equation, you can traverse to both as long as they have different names.
<wgrant> So you can properly have PPAs for both distros.
<kb9vqf> Oh, OK.  That sounds good
<kb9vqf> Out of curiosity, did you see any potential copyright problems in your brief glance?
<kb9vqf> I don't want to step on Canonical's toes accidentally here
<sinzui> wgrant, does this look like an acceptable fix for the team participation page: http://pastebin.ubuntu.com/448553/
<wgrant> kb9vqf: Well, it's AGPLv3 and you didn't have the code up AFAICT, but apart from that it looks pretty well rebranded (but I am no Canonical guy).
<kb9vqf> A quick tarball (minus config directory) should fix that though, right?
<wgrant> sinzui: Apart from the fact that I loathe using teamowner rather than something like is_team, looks great.
<kb9vqf> :-)
<wgrant> kb9vqf: Ideally push the branch up somewhere, but I guess a tarball works too.
<sinzui> We should have is_team
<kb9vqf> Once I get the public interface up maybe I'll throw it in a bazaar branch
<kb9vqf> I'm still learning about all of that
<kb9vqf> It's off the public Internet for now, I'll put something together before I reconnect it
<wgrant> Yep.
<kb9vqf> Thanks for the help once again!
<wgrant> sinzui: Hm, +participation OOPSes on some of my teams.
<wgrant> (I was going to check if the 'Manage mailing list subscriptions' link was shown)
<sinzui> can you show me a url?
<wgrant> In fact, any team I'm an admin of.
<wgrant> KeyError: 'editemailaddresses'<br />
<wgrant> So, yes, we need to hide that link as well.
 * sinzui looks
<sinzui> wgrant, I am an idiot
<sinzui> I can fix that now.
<sinzui> wgrant, That is really the same as my oversight about teams
<wgrant> sinzui: Yeah, I just thought to check if we needed to hide that.
<wgrant> I didn't think it would actually OOPS -- just be broken.
<sinzui> wgrant:I intended those links to be on your page, but my permission check also applied to your teams
<wgrant> Yep.
<wgrant> But 'Register team' should probably be present anyway.
<sinzui> switching to admin_browser gives my the expected error. Adding a team check to the action links fixes the issue
<wgrant> Perfect. Thanks.
<wgrant> sinzui: So, I'm looking at the fmt:link for IPPA... it returns the empty string if it's private and you can't see it. Is it just me, or is that a completely insane way of doing things?
<sinzui> That is a safety measure to ensure it does not link
<wgrant> I would have expected it to end in an Unauthorized.
<sinzui> You should look before you leap with private ppas and teams
<wgrant> Probably, but if it's private then a proper implementation will end in a noticable Unauthorized, rather than a silent missing link which probably indicates an underlying lack of security filtering.
<magcius> does loggerhead have a raw mode?
#launchpad-dev 2010-06-13
<thumper> morning
#launchpad-dev 2011-06-06
<wgrant> StevenK: rvb has two pending items on https://devpad.canonical.com/~lpqateam/qa_reports/deployment-db-stable.html -- could you please try to QA those?
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:208 - 0:[######=_]:256
<wallyworld> wgrant: the work you are doing on product picker - have you made any changes to ProductVocabulary in vocabularies.py?
<LPCIBot> Project windmill-db-devel build #364: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/364/
<wgrant> wallyworld: Just http://paste.ubuntu.com/619573/
<wallyworld> wgrant: thanks. i'm looking at altering the json so that exact matches (or matches above a certain ranking) are returned in a separate attribute so that if there are "too many" we can stil display the exact ones
<wallyworld> bug 255282
<_mup_> Bug #255282: project name picker search / vocabulary is hard/impossible to use (too many results on exact searches) <disclosure> <lp-registry> <project-picker> <projects> <Launchpad itself:Triaged by wallyworld> < https://launchpad.net/bugs/255282 >
<wgrant> wallyworld: Probably better to limit?
<wgrant> wallyworld: That's what that diff does.
<wgrant> wallyworld: It ranks exact matches first, then by FTI, and limits as well.
<wallyworld> wgrant: ah yes, i didn't see the limit
<wgrant> Limiting in the current implementation is bad, because it ranks by displayname.
<wallyworld> in that case, that bug is moot
<wgrant> But ordering by rank makes limiting sensible.
<wallyworld> yes agreed
<wallyworld> i spent some time cleaning up a fix previously landed which didn't work properly :-(
<wgrant> Oh?
<wallyworld> there was one recently to deal with the picker handling too many matches but it was all wrong (i didn't do the fix, just cleaned it up :-)
<wgrant> Ah, right, I remember that one.
<wallyworld> wgrant: so i should assign that bug to you since your change deals with the problem
<wgrant> wallyworld: Probably, yeah. I think this two line change solves most of the product picker problems.
<wallyworld> ok. you can link a few bugs to your branch :-)
<wgrant> Yup.
<wgrant> The prettification is a separate, more complicated beast.
<wgrant> wallyworld: The link doesn't seem to do anything.
<wgrant> The one you just landed.
<wgrant> And the spacing is off.
<wallyworld> works for me?
<wallyworld> your sure your browser is not blocking popups?
<wgrant> Which browser?
<wallyworld> i use ff4
<wgrant> Ew, popups?
<wallyworld> well the browser says its a popup but it's a window.open(...) call
<wallyworld> and i have mine set up to open in a new tab
<wgrant> Hm, then it shouldn't be green.
<wgrant> Green means an inline thing.
<wallyworld> well, i was told to make it green
<wgrant> href is not defined
<wgrant> When I try it in Firefox 4.
<wallyworld> green meaning "this link doesn't kill the current page"
<wallyworld> ?
<wallyworld> i'll have another look, but it worked when I tested before ec2'ing
<wgrant> Hmm. Where is href defined?
<wgrant> I think you might have meant data.alt_title_link
<wallyworld> i need a bit more context. let me open the branch
<wgrant> window.open(href) is failing, since href is not defined. In the 'if (data.alt_title)' branch.
<wgrant> Anyway, it's all behind flags, so it doesn't really matter much right now.
<wallyworld> that sucks. yes, it is broken. not sure how i let that happen.
<wallyworld> i'll fix
<wgrant> Thanks.
<wgrant> We should also huwshimi about the link style, I think.
<wgrant> It looks a bit odd being in the parenthesised section.
<wallyworld> sure. i'll do what i am told :-) i wanted to make it something different but was told the styles are fixed and not easily changed
<wgrant> Well, it's easier to change now it's there.
<wgrant> The hard bit is doen :)
<wallyworld> yep
<wgrant> wallyworld: Did you intend to reuse the commit message?
<wgrant> I see you did that on lazr-js a few times too.
<wgrant> It's rather confusing.
<wallyworld> probably should have changed it
<wallyworld> didn't think you would be looking so closely at it
<wgrant> I read most diffs, so I can keep track of what's going on and hit people when they introduce security holes :)
<wallyworld> fair enough :-)
<LPCIBot> Project windmill-devel build #171: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/windmill-devel/171/
<LPCIBot> Project windmill-devel build #172: STILL FAILING in 45 min: https://lpci.wedontsleep.org/job/windmill-devel/172/
<jtv> Who's up for a pre-imp chat?
<LPCIBot> Project windmill-db-devel build #365: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-db-devel/365/
<jtv> lifeless: got another fun problem that I'd love to discuss with you â creating multiple Jobs in one query.
<wgrant> jtv: lifeless isn't here today.
<jtv> Ah blast
<wgrant> jtv: But that's always fun.
<jtv> Yes
<wgrant> I think you have to use strings.
<wgrant> That's what I do, at least.
<wgrant> eg. in PublishingSet.publishBinaries
<jtv> You mean, SQL strings?
<wgrant> Ja.
<jtv> That's not really the problem I'm pondering though.
<jtv> I never really assumed that Storm would support what I wanted directly.  The real questions is how to make this work well API-wise.
<jtv> It seems easy: extend IJobSource with a createMultiple() that returns an iterable of job ids.
<jtv> Then feed those into job-type-specific equivalents of createMultiple().
<jtv> The part I dislike is how some IJobSource-derivatives operate on Job, and classes implemented purely in that table; and others like to refer to other tables that in turn refer to Job.
<jtv> So IJobSource.createMultiple() would end up in job utilities that are expected to return something else than Job.
<lifeless> I would like to delete 'Job'
<jtv> You're not here.
<jtv> Shoo.
<lifeless> its a generally poor pattern for sql to do inheritance like that
<lifeless> I pass through
<jtv> Not using direct inheritance support in the DB does make things feel very awkward and old-fashioned,
<jtv> almost like inheritance in JavaScript.
<jtv> Maybe we should've just used db-native table inheritance.  :)
<lifeless> well
<lifeless>  I would have done code inheritance but not db inheritance
<lifeless> for most jobs there isn't anything in common, but we've created a bottleneck as it stands
<wgrant> :(
<lifeless> by analogy, why don't we have a base *table* with the id column :>
<wgrant> Indeed.
<wgrant> Also, why does the test suite and staging update take so long :(
<wgrant> And why does something conflicting with db-devel have to have just landed :(
<jtv> The answer, my friend, is blowing in the wind.
<jtv> lifeless: you're right though â design of our jobs is driven by commonality of contents, not commonality of purpose.
<jtv> It'd be nice if we could at least have a pattern along the lines of "put everything in the json metadata, but we can also duplicate data that needs to be searchable in proper columns."
<jtv> I think I need to keep multi-job creation out of IJobSource because it isn't even a utility interface; it's an interface definition pur sang.
<adeuring> good morning
<jtv> hi adeuring
<adeuring> hi jtv!
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs:208 - 0:[######=_]:256
<rvba> wgrant: Hi, I'm in the progress of QAing r10636.
<wgrant> rvba: And r10641?
<rvba> wgrant: yes, that too
<wgrant> Thanks.
<wgrant> rvba: Have you talked to Stevenk?
<StevenK> At length.
<rvba> wgrant: Yes I have ...
<mrevell> Happy Monday
<wgrant> Heh
<rvba> Hello mrevell!
<LPCIBot> Project windmill-devel build #173: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/windmill-devel/173/
<LPCIBot> Project windmill-db-devel build #366: STILL FAILING in 1 hr 14 min: https://lpci.wedontsleep.org/job/windmill-db-devel/366/
<jelmer> wgrant: hi
<wgrant> jelmer: Hello.
<jelmer> wgrant: you mentioned in #bzr you were QA'ing some of the bugs in my newer-bzr branch; are you still working on that and/or is there something I can help with?
<wgrant> jelmer: I tried a few things on qastaging, but I'll probably need to do importds on staging tomorrow, since qastaging doesn't have any.
<wgrant> (and it's not on staging yet)
 * wgrant -> food
<jelmer> ah, ok
<jelmer> wgrant: if you need any help, you know where to find me :)
<wgrant> jelmer: Yup, thanks.
<wgrant> rvba: How's it going?
<rvba> wgrant: good so far, I've found a bug.
<rvba> wgrant: is this blocking you in any way?
<wgrant> rvba: This is potentially disastrous.
<wgrant> rvba: We need to merge db-stable into devel today.
<wgrant> What's the bug?
<rvba> wgrant: a simple UI bug.
<wgrant> In +initseries?
<rvba> yes
<wgrant> Ah, OK, fine then.
<rvba> Nothing disastrous :)
<wgrant> I'd like to do the merge in the next 30 minutes.
<wgrant> To catch the next devel run.
<wgrant> So it'd be great if you could have the QA done by then.
<rvba> Since the changes in question are (like always) derivation related ... I suppose we could mark these bug QAok.
<rvba> s/bug/bugs/
<wgrant> Well, they also have the potential to break Ubuntu initialisation, but yeah, that's a while off.
<wgrant> So if you can't get them done soon, qa-ok it is.
<rvba> wgrant: that's right, they have potential to break ubuntu init ... I think StevenK is checking these changes very seriously.
<rvba> StevenK: I suppose +initseries won't be used "for real" right now, the only dangerous thing is the initialisation of ubuntu. what is the conclusion of your conversation with Julian this morning about this? Will you check the ubuntu initialisation?
<rvba> wgrant: We found another bug in initialisedistroseriesjob ... if you pass packagesets=None the job will fail because of the way parameters are passed inside a metadata attribute.
<rvba> It will be very easy to fix but it's a stupid bug.
<rvba> https://bugs.launchpad.net/launchpad/+bug/793434
<_mup_> Bug #793434: Passing packagesets=None triggers an error when initializing a series. <derivation> <Launchpad itself:In Progress by rvb> < https://launchpad.net/bugs/793434 >
<rvba> wgrant: I suppose I should wait for you to merge db-stable into devel and then fix this in devel ...?
<wgrant> rvba: Probably, yes. We'll merge shortly, then be frozen until QA'd past the merge.
<wgrant> Which will be tomorrow some time.
<rvba> Ok, I'll get the fix ready then.
<wgrant> Thanks.
<wgrant> rvba: So, can we qa-ok them?
<rvba> wgrant: I *just* did.
<rvba> Right in time I suppose ;)
<wgrant> Heh.
<wgrant> Thanks.
<bigjools> I wish make schema would do just exactly that, not a million other things
<stub> bigjools: I use 'make -C database/schema'
<bigjools> stub: do you know why the top-level Makefile's schema target depends on build then?
 * henninge lunches
<stub> bigjools: Don't know. Technically that is correct I guess, since we need /bin/py built and stuff, so the problem is that 'make build' is too busy
<maxb> Also that 'make build && make schema' re-does bits of the build
<bigjools> ah, that Makefile uses bin/py ... :/
<stub> probably something to do with wadl depending on *.py... perhaps that should only rebuild if explicitly asked or if it is non-existent
<bigjools> jeebus, the test failure attachments are huge now
<stub> Yer... all those librarian logs...
<bigjools> and SQL now it seems
<stub> that I suspect are not reset between tests...
<wgrant> Hmm.
<wgrant> schema doesn't need build.
<wgrant> Just compile.
<wgrant> Which is much smaller, and runs OK with -j3.
* wgrant changed the topic of #launchpad-dev to: devel is release-critical until r13163 deployable | https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs:208 - 0:[######=_]:256
<bigjools> that was slower than -j2 last time I checked
<wgrant> ... bah.
<stub> What does 'orphaned' mean on the qa'd patch deployment report
<stub> ?
<wgrant> stub: You used [testfix] when you shouldn't have.
<wgrant> I decided it was safe to deploy, since it passed buildbot and seems harmless.
<wgrant> So it is in devel now.
<wgrant> staging seems happy enough, so if it's not OK please yell really really soon.
<stub> That branch lifelesslanded would be qa-untestable anyway. Nothing is using the new table yet, so the fallout would be in related areas that don't cope with the triggers.
<wgrant> stub: Yep.
<bigjools> have we got a run_jobs.py cronned in on staging?
<jtv> hi henninge!  Can I trouble you for a review?  https://code.launchpad.net/~jtv/launchpad/db-bug-793382/+merge/63545
 * henninge tries not to let anything trouble him ...
<jtv> Is that good or bad?  :)
<henninge> jtv: sure, I am happy to look at your review. ;-)
<jtv> \o/
<henninge> good for me, saves me the troubling.
<henninge> s/at your review/at your proposal/
<gmb> flacoste: Can I get an RC for https://code.launchpad.net/~gmb/launchpad/bug-772609-hopefully-without-breaking-anything-this-time/+merge/63432? It should have landed Friday night but bounced due to a [testfix].
<wgrant> gary_poster: Morning.
<gary_poster> morning wgrant
<wgrant> gary_poster: You're still planning to release the new subscriptions stuff during the downtime deploy this week?
<gary_poster> or evening, sorry
<wgrant> (the bugmail footer change is in devel now, so will be deployed unless it's reverted)
<wgrant> (but we still have the strucsub listing regression, so I'm not sure)
<gary_poster> wgrant, yes
<wgrant> Great.
<gary_poster> (the structsub listing thing has branches to be addressed--that also rip out the feature flags--after at least a day or two of making sure that the subscriptions stuff is ok when exposed to everyone)
<wgrant> OK. We've only had a few complaints about its absence, so I guess it's not too bad.
<gary_poster> cool
<wgrant> Despite the mild security implications.
<gary_poster> wgrant, security?  Do "other subscribers" affect visibility of private bugs?
<wgrant> gary_poster: No. But people inevitably mistakenly put private info in public bugs, and want to know who it was emailed out to.
<wgrant> Hopefully that's less common now.
<gary_poster> ah, k.
<wgrant> gmb: Hmm, I'm not sure who can grant r-c these days. Since there's no RM...
<gmb> Hmm.
<gmb> Traditionally it was flacoste, lifeless or <RM>. So I guess we have to wait on one of the other two. Or maybe gary_poster can sign off on it.
<wgrant> This seems pretty clearly necessary and really wants to land in the next 3 hours, or I will be sad tomorrow.
<gmb> Yes.
 * gary_poster has vague memories of team leads having privs like this?
<wgrant> It hasn't been used since the RM position was abolished,
<gmb> gary_poster: Let's say that your memories are right :)
<wgrant> Heh.
<gary_poster> :-)
<stub> Because we are minimizing the stuff that *has* to rollout during downtime deployment. Does this come with a db patch?
<gary_poster> gmb, ok, I'll go fiddle with the MP then
<gmb> stub: No.
 * gary_poster waits...
<wgrant> stub: Yeah, this is the issue with deploying features at the same time.
<wgrant> Which lifeless is vehemently against.
<gmb> stub: But if stuff is rolled out without this, we will have breakages.
<gmb> And that will make both wgrant and I go <sadface>
<wgrant> This release prep is chaotic enough already, let's not make it worse by delaying this.
<gary_poster> stub, barring your quick strong objection, I'll approve
<stub> Sure. Might want to make that clear to Francis and/or robert then
<stub> I don't object, not that it would matter if I did :)
<gary_poster> heh, I dunno
<stub> I would wonder if not landing it could be worked around by just disabling the feature using a flag, which also isn't clear from the MP
<gary_poster> gmb, rc approved with all the authority of my title behind it ;-)
<gmb> Cool, thanks.
<stub> Its actually pretty awesome the last few releases haven't needed RC's...
<stub> Used to be a truck load of 'OMG its broken' fixes picked up by last minute QA
<wgrant> Well, given that we deploy stuff every couple of days, there's less space for stuff to explode. It's been working pretty well!
<lifeless> stub: wgrant: gary_poster: FWIW I think 'rc-approval' is fine from squad leads
<lifeless> I'm not sure we -need- an rc-approval bit other than 'this has to be landed to make the downtime safe'
<gary_poster> cool, thanks lifeless
<gary_poster> and agreed
<gary_poster> a team review approval seems pretty good to me
<lifeless> this is fixing the anonymous viewing of bug pages right ?
<lifeless> gary_poster: re: releasing the new subscription stuff - thats slated for Monday right ?
<wgrant> It has to be Thursday, I think :/
<wgrant> The bugmail footer change is not flagged.
<lifeless> why ?
<wgrant> So it will be deployed on Thursday.
<wgrant> Unless that page works for everyone?
<gary_poster> lifeless, correct.  we had the flag discussion on the bug tracker, I think
<lifeless> I think we should wait for an hour or two after the deploy before setting the flag
<lifeless> or monday per gary_poster's comment just now ;) \o/
<gary_poster> no no
<gary_poster> I was agreeing with wgrant
<lifeless> ok
<gary_poster> we can wait, but then there will be a couple of hours in which the links will point to a page without valuable info
<gary_poster> page will appear
<lifeless> sure
<gary_poster> but no info
<lifeless> lets just wait ~1 hour - enough time to see that nothing else is truely messed up
<gary_poster> I'm fine with that
<lifeless> if something is, we can open the flag wide and know its not the flags fault
<lifeless> vice versa, if it looks good and we open the flag wide and it falls over, we know it probably is the flag.
<lifeless> anyhoo... its now late, so gnight all :)
<gary_poster> sounds good lifeless.  'night
* wgrant changed the topic of #launchpad-dev to: devel is release-critical until r13164 deployable | https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs:208 - 0:[######=_]:256
<wgrant> Night lifeless.
<LPCIBot> Project windmill-db-devel build #367: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/367/
<deryck> adeuring, henninge -- ping for standup.
<LPCIBot> Project windmill-devel build #174: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/174/
<henninge> deryck: yup
<henninge> X crash. Unity?
<deryck> reboot, brb
<gary_poster> mrevell, hi.  are you going to be around to give some UI opinions in the next half hour or so if I send you some screen shots?
<mrevell> gary_poster, Yes, certainly.
<gary_poster> mrevell, thank you
<mrevell> np :)
<henninge> jtv: reviewed
<jtv> Thanks!
<jtv> henninge: apologizing for not finding anything wrong may be going a bit far :)
<henninge> ....
<henninge> yeah, let me see if I can find something wrong, then
<jtv> uh-oh
<henninge> jtv: the indentation of the parenthesis on line 319 and 332 looks strange ... but I may be wrong.
<henninge> same on line 350
<henninge> jtv:  there you go! ;)
<jtv> Thank you, that's better.  :)
<jtv> The indentation looks a bit weird there because we rarely open a parenthesis on a line of its own.
<jtv> In this case though I do think it makes sense.
<henninge> yeah, I guess I was wrong, then ...
<henninge> well, actually I wasn't as we agree about the strangeness.
<jtv> I'm not sure there's any perfect answer.
<jtv> I generally like what I think is GNU style, with opening and closing braces identically indented.
* flacoste changed the topic of #launchpad-dev to: devel is release-critical until r13163 deployable | https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs:208 - 0:[######=_]:256
<flacoste> gary_poster: you probably want to land your bug 792493 branch on devel once PQM gets out of RC-mode
<_mup_> Bug #792493: Creating a structural subscription for a team with a preferredemail makes a mute link that oopses <oops> <Launchpad itself:Triaged by gary> < https://launchpad.net/bugs/792493 >
<flacoste> gary_poster: also, i downgraded the revision to deploy to 13163 since that's the one that include the merge, your other RC can be deploy nodowntime afterward
<flacoste> so it's not a blocker per se i think
<wgrant> It is unless we want to delay the unflagging for hours and have bugmail links broken.
<wgrant> We cannot turn the flag on by default without 13164.
<gary_poster> flacoste, what wgrant said.  My branch is not that important, but his is
* flacoste changed the topic of #launchpad-dev to: devel is release-critical until r13164 deployable | https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs:208 - 0:[######=_]:256
<flacoste> ok, makes sense
<flacoste> shouldn't be a big problem
<gary_poster> cool
<flacoste> anybody took care of the conflicts yet?
<wgrant> I can prepare a branch if you want, but it's pretty trivial.
<wgrant> Since you can just take the files from devel.
<flacoste> yeah, no problem
<wgrant> There's only three.
<wgrant> flacoste: We'll be remerging db-stable tomorrow once devel reopens anyway.
<wgrant> So no need to reland it on devel.
<flacoste> yep
<gary_poster> mrevell, mail sent (with two screenshots inline; hope that's not too annoying)
<mrevell> Thanks gary_poster, looking now. Not at all annoying :)
<gary_poster> cool :-)
<nigelb> Does running launchpad impair the ability for my apache to server php?
<wgrant> nigelb: rocketfuel-setup installs apache2-mpm-worker, and PHP isn't threadsafe so it must be removed. But LP works fine with apache2-mpm-prefork instead of -worker, so you should be able to install libapache2-mod-php5 again.
<wgrant> (it will remove -worker and install -prefork)
<nigelb> wgrant: I tried that and it ended with an unsucessfull install, but its fine.  I'll just use nginx :)
<wgrant> nigelb: What went wrong?
<wgrant> I've done it in the past.
<LPCIBot> Project windmill-devel build #175: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-devel/175/
<nigelb> wgrant: dpkg failures
<wgrant> nigelb: Huh.
<flacoste> jelmer: were you able to QA the bzr upgrade?
<deryck> henninge, I see you saw I marked bug 410864 as related.  I agree it looked like a dupe too...
<_mup_> Bug #410864: 'choose a source package' suggests undefined when search has too many results <qa-ok> <Launchpad itself:Fix Committed by henninge> < https://launchpad.net/bugs/410864 >
<deryck> henninge, but I thought I'd be cautious and leave it separate since the widget was used differently.
<henninge> deryck: argh!
<henninge> deryck: I just read the bug properly. It is not the same bug and it is not fixed.
<henninge> Actually, I had wondered about what is described there while fixing the bug.
<deryck> henninge, I walked through the steps in the bug on qastaging and had a nice list of packages to choose from.  over 35 steps, IIRC.
<deryck> 35 steps in the results picker, I mean
 * henninge has to try this now
<henninge> deryck: :(
<deryck> Why the frown?
<henninge> deryck: qa-bad
<deryck> ah sadness indeed.  How is it bad?
<henninge> deryck: It broke the check for picker with search boxes.
<deryck> I don't get meaning of "the check for picker with search boxes"
<henninge> hm
<deryck> henninge, how is it broken?  Sorry to be dumb.
<henninge> I just got a result box with 220 batches.
<henninge> deryck: https://bugs.qastaging.launchpad.net/ubuntu/+source/linux/+bug/687392
<_mup_> Bug #687392: Kernel panic with 2.6.32-26 security upgrade <Linux:Confirmed> <linux (Ubuntu):Incomplete> < https://launchpad.net/bugs/687392 >
<deryck> heh a ha!
<henninge> deryck: edit "Linux" and enter "linux" into the search box.
<deryck> I see it now.
<deryck> only 220 pages! :)
<henninge> deryck: Unfortunately I have to leave really soon and have no time to fix or rollback.
<deryck> henninge, so depending on where we are in the qa process, i.e. if qa-ok revisions ahead of you, which is likely.  I'd still leave it qa ok and do a follow on fix....
<deryck> henninge, it's a corner case that this many results will turn up normally.  and ugly, big corner case.  but at least the inital bug is fixed and the picker is usable...
<deryck> if ugly.
<henninge> oookaaay
<deryck> henninge, so either constrain the widget width, or limit the number of pages we show.  is anything beyond 40-50 pages useful?
<henninge> deryck: the limit is 20
<henninge> deryck: I removed it for Robert's special case but unfortunately it leaked into other cases it seems.
<deryck> henninge, right, but you could only display 20 and not limit on 20.
<henninge> deryck: I filed two other bugs for a proper fix.
<henninge> I think they are linked from my mp?
<henninge> deryck: I will fix this tomorrow and maybe include a fix for bug 410864.
<_mup_> Bug #410864: 'choose a source package' suggests undefined when search has too many results <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/410864 >
<deryck> henninge, great, thanks.
<deryck> though I'm still not clear why 410864 isn't fix since we just looked at 220 results on that page. ;)
<deryck> but we can talk about it tomorrow.
<sinzui> I am 2 hours into my day. I am already burned. No doubt because I worked 3 days to see progress.
<sinzui> 1. We do not need a special JS for linting/evaluating. I can make seed do it
<sinzui> 2. I can run YUI tests without firefox, or any browser, I do not require a database or the web server. The test runner still needs  xvfb,
<sinzui> jcsackett: do you have time to talk?
<jcsackett> sinzui: sure, but one moment. i need to reboot.
<jelmer> flacoste: wgrant said we'd have to wait until staging gets updated as it has a importd we can use for testing
<jcsackett> sinzui: back, calling you now.
<flacoste> jelmer: don't we have a bazaar.qastaging ? i'm sure we can test at least basic bzr functionality there
<jelmer> flacoste: the basic bzr functionality we can test there - and that works, but all the bugs associated with my branch require running of a importd with the new code
<flacoste> jelmer: staging is running 10647 and you require 10649 right?
<jelmer> flacoste: what branch are those revno's for?
<flacoste> jelmer: db-stable, that's what staging runs, i see that your stable landing was merged on db-stable at 10649
<jelmer> (on another note, why are there 5 people with recipes that build lp:~launchpad-pqm/launchpad/stable)
<jelmer> flacoste: ah, yep
<flacoste> jelmer: recipe: i hav eno idea
<flacoste> people trying to autobuild launchpad .deb?
<maxb> Oh dear. That's an impressive lack of clue
<bigjools> yet hilarious at the same time
<jelmer> I guess.. none of them merge in any packaging branches though, so that's doomed to fail
<maxb> It's about as ridiculous as requesting subversion code imports from bazaar.launchpad.net, which people also seem to do from time to time
<jelmer> maxb: we should fix that by just letting them enter a URL and Doing The Right Thing
<maxb> But, what is The Right Thing?
<maxb> It's not even clear what they are hoping to achieve
<jelmer> maxb: In some cases I guess a mirror (if it's a bzr branch elsewhere), if they're trying to import from bazaar.launchpad.ent we should tell them to "bzr branch && hack && bzr push"
<jelmer> maxb: I think they're looking for the fork button
<maxb> Could be
<maxb> On a related note, we could use a "Do you really want to be doing this?" banner for people entering questions on Launchpad itself
<maxb> Back on the code import issue, I think https://code.launchpad.net/+code-imports/+new is wrong to default to Subversion
<maxb> It should default to nothing and force you to pick one
<jelmer> maxb: we should not require you to specify the VCS at all, just let you enter a URL
<maxb> Hm. I am of the opinion that there is no point writing the code to do that, because if a user can't even figure out what VCS they are importing from, they don't deserve an import :-)
<flacoste> maxb: that's a little harsh :-)
<flacoste> maybe they onbly care about bzr
<flacoste> whatever, just give me a bzr repo
<flacoste> !
<jelmer> maxb: *I* would appreciate having to click one radio button less for each import
<jelmer> if we have the data we shouldn't bother to ask our users for it
<flacoste> jelmer: staging is updating now, should be available in ~90 minutes
<jelmer> flacoste: great - thanks :)
<rvba> bac: Hi ... I'm not sure I should have done it this way, but I created another branch ("a rebased" version of the branch you've been reviewing) so that it targets devel.
<rvba> bac: sorry to bother you with that, but perhaps you could review/approve it ...?
<rvba> https://code.launchpad.net/~rvb/launchpad/init-multiple-parents-bug-777120-devel/+merge/63572
<LPCIBot> Project windmill-devel build #176: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/windmill-devel/176/
* henninge changed the topic of #launchpad-dev to: devel is release-critical until r13164 deployable | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:208 - 0:[######=_]:256
<timrc> wgrant, I might  be seeing a regression of https://bugs.launchpad.net/launchpad/+bug/750640
<_mup_> Bug #750640: If the source version differs from the binary version, it is not specified in the Package index for that binary <oem-services> <qa-ok> <soyuz-publish> <Launchpad itself:Fix Released by wgrant> < https://launchpad.net/bugs/750640 >
<maxb> timrc: It's 4:30am in Melbourne...
<timrc> maxb, $1 USD says he has logging enabled and will see it when he wakes up
<timrc> wgrant, nm, I think I misread comment #2
<lifeless> morning y'all
<lifeless> flacoste: yo
<flacoste> hi lifeless
<mwhudson> jelmer: does launchpad no longer depend on pysvn?
<mwhudson> (i guess cscvs still does)
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | devel is release-critical until r13164 deployable | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:208 - 0:[######=_]:256
<mwhudson> hm, it seems the update of sourcedeps.cache depends on dictionary order
<abentley> mwhudson: yes, that's likely.
<mwhudson> shall i file a bug?
<abentley> mwhudson: But I thought it would only perform the update if the data had changed.
<mwhudson> ah
<abentley> mwhudson: I experienced a bug this morning where it said the cache had been updated, so I'm now no longer sure that's right.
<mwhudson> i think that's what it just did to me
<abentley> mwhudson: Okay, what did it do?  Are the cache files the same data in different orders?
<mwhudson> abentley: it looks at a quick glance as if it will update the cache if any branches were updated, not if any branch ended up with a different tip than that in the cache
<mwhudson> so i think if i revert and run update-sourcecode again, it will not be changed
<mwhudson> abentley: yeah
<abentley> mwhudson: okay, that's useful.
<jelmer> mwhudson: hi
<jelmer> mwhudson: cscvs indeed still does, and there codehosting code also still uses oo_svn
<mwhudson> jelmer: ah well
<mwhudson> i guess hypothetically we could kill off the remaining cscvs-svn imports and then delete all the svn code from cscvs
<mwhudson> but well, fails to beat the effort/reward test i guess
<jelmer> mwhudson: yeah, probably
<jelmer> mwhudson: I was going to ask the other day - the codeimport workers don't have direct/API access to the database, right?
<mwhudson> jelmer: no
<mwhudson> the machines still do, for sodding oops-prune
<jelmer> mwhudson: I was looking at bug 432217, I guess that's a fair bit harder in that case
<_mup_> Bug #432217: Import branches have auto-format upgrade disabled <code-import> <lp-code> <tech-debt> <Launchpad itself:In Progress by jelmer> < https://launchpad.net/bugs/432217 >
<jelmer> mwhudson: it seems though, that at least for bzr-{git,svn,hg} branches we could just nuke the branch if it's in an old format rather than bothering with upgrading?
<mwhudson> jelmer: yes, for deterministic imports that's totally the way to go
<mwhudson> for the others, ugh, i guess there's no good answer really
<mwhudson> jelmer: we can give the machines access to the db again if we want, but i've always regarded them as a bit more likely to get compromised than the average machine
<jelmer> the code has a FIXME saying a upgrade job should be created, though I wonder how that would have to happen (XML-RPC request??)
<mwhudson> (i seem to be more paranoid than most people about this)
<mwhudson> uhh
<mwhudson> a branchupgradejob wouldn't help
<mwhudson> because of the storage of the import branches on escudero
<mwhudson> so if you upgraded the branch on crowberry, it would just get downgraded again the next time the import puller ran
<jelmer> ah, urgh
<mwhudson> (the real answer to this side of things is to have the code import worker have some kind of token-mediated access to crowberry that allows write access to exactly one branch)
<mwhudson> i think on staging the code import worker can write to any branch on the codehost in effect
<mwhudson> this doesn't seem like a good idea for prod :)
<jelmer> :)
<jelmer> I'll have a look at nuking the bzr-{hg,svn,git} branches; we can nuke and recreate cscvs-based svn import branches
<jelmer> that just leaves CVS imports
<jcsackett> sinzui: did you ping me earlier?
<sinzui> jcsackett: no, but a kitten may have. She slept on my keyboard during lunch
<jcsackett> sinzui: dig. the cats are after your computer again.
<sinzui> jcsackett: I just approved your yui test branch
<mwhudson> jelmer: wow the ratio of failed to running says something here: https://lpstats.canonical.com/graphs/CodeImportsCVSBreakdown/
<jcsackett> sinzui: cool, thanks.
<jelmer> garr, I thought my internet connection was just bad today
<jelmer> mwhudson: oh, wow (those colors are.. unusual)
<mwhudson> jelmer: sysadmins are not visual designers, usually :p
<jelmer> mwhudson: while you're here.. am I correct in thinking that the code import workers don't have any access to e.g. information on what the development focus for a project is?
<mwhudson> jelmer: yes, that's true too
<mwhudson> it's not hard to arrange to pass more information along though
<mwhudson> (are you thinking about stacking?)
<jelmer> mwhudson: yes, exactly
<cr3> if I add a new method to the restful api, do I need to add @operation_for_version("beta")?
 * timrc would think "devel" would be a more appropriate version *duck*
<lifeless> devel
<cr3> timrc: I'm thinking "not backward compatible == beta", "backward compatible == devel"
<lifeless> beta *was* a beta
<lifeless> it is not now
<lifeless> ignore it
<lifeless> you want for devel
<lifeless> mwhudson: jelmer: those machines are not getting db access.
<timrc> can the suggestion that gets spit at you if you do not include @operation_for_version be changed to suggest 'devel' rather than 'beta'? I stumbled upon this recently as well and some head scratching ensued
<mwhudson> ah, now we have someone who is as paranoid as me apparently
<lifeless> timrc: patches gratefully...
<timrc> lifeless, =)
<jelmer> lifeless: fair enough
<mwhudson> lifeless: has the access they have for oops-prune been removed then?
<timrc> lifeless, hey while you're here... I did not really have a reviewer for https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-2 -- I discussed some w/ bigjools, though... should I just add the next on call reviewer to the merge proposal ?
<jelmer> can it be that launchpad is firewalled by sourceforge?
<mwhudson> jelmer: in the past they've been happy with us doing what we do, but it's possible for sure
<lifeless> mwhudson: I thought that went *out* to them / was a different DB
<lifeless> timrc: ping them here if you can
<lifeless> jelmer: mwhudson: so its not for security I don't want the importds talking to the DB
<lifeless> rather its that long running processes with DB access have a history of terrible transaction handling.
<lifeless> these things have no call to know about our DB schema.
<lifeless> and if they did, downtime deploys would get that much harder.
<flacoste> after merging devel i have an error with sprite generation:
<flacoste> KeyError: u'../images/mute.png'
<flacoste> any ideas?
<flacoste> ok, make clean solved the issue
<LPCIBot> Project windmill-db-devel build #368: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/368/
<lifeless> back
<thumper> what's with all the failing import emails?
<thumper> there seems to be an uncharacteristically large number of them
<mwhudson> thumper: maybe this is related? <jelmer> can it be that launchpad is firewalled by sourceforge?
<thumper> haha
<thumper> maybe
#launchpad-dev 2011-06-07
<wallyworld> sinzui: stupid mic
<sinzui> wallyworld: I had to toggle mute on Sound Preferences > Input. Even though it said it was on, the live check clear said it was not
 * wallyworld tries that
<wallyworld> sinzui: here but mic still broken
 * sinzui is now deafened
<lifeless> we should do a ndt today
<wgrant> lifeless: I plan to, once I attack staging.
<wgrant> importds need QA.
<lifeless>  oh cool
<wgrant> Then I need to remerge db-stable.
<StevenK> WebCat sounds like a fun project
<StevenK> utilities/shhh.py bin/sprite-util create-css
<StevenK> Traceback (most recent call last):
<StevenK> ...
<StevenK> KeyError: u'../images/mute.png'
 * StevenK pouts.
<jcsackett> wallyworld: so, what's the bit that needs dealing with picker wise?
<wallyworld> jcsackett: sent you an email :-)
<jcsackett> ah, fantastic.
<wallyworld> just move a coupl eof line sof js
 * jcsackett wades through codeimport emails to find the relevant one.
<wallyworld> jcsackett: i have code impor temails go to /dev/nul :-)
<wgrant> StevenK: rm ./lib/canonical/launchpad/icing/icon-sprites.positioning
<jcsackett> wallyworld: that definitely fits within the scope of what i'm working on now. i'll incorporate it into my personpicker-fixes branch.
<wallyworld> jcsackett: thanks :-)
<jcsackett> wallyworld: you're welcome.
<StevenK> wgrant: Thanks
<StevenK> And another 113 code import mails :-(
<huwshimi> StevenK: That's what mail filters are for
<StevenK> Yes, I just fixed sieve
<StevenK> I think it's working, too.
<wgrant> jelmer: Still around?
<jelmer> wgrant: barely
<wgrant> jelmer: http://staging.launchpadlibrarian.net/72500040/vcs-imports-virt-manager-trunk.log
<wgrant> Something to worry about?
<jelmer> wgrant: somewhat - I tagged one of the bugs associated with the newer-bzr branch qa-bad for this reason
<lifeless> jelmer: qa-bad meaning 'must rollback'
<wgrant> At this stage it is more "must panic"
<lifeless> yup
<lifeless> so, rollback that rev.
<wgrant> jelmer: How critical is this?
<wgrant> Can we just roll back bzr-hg, or do we have to do the whole thing?
<wgrant> I guess the whole thing.
<lifeless> rollback the lot, its simplest.
<lifeless> known good.
<jelmer> wgrant: the whole thing, the older bzr-hg won't work (at least isn't tested) with the older bzr
<wgrant> OK, rolling back.
<wgrant> Thanks.
<wgrant> buildbot is even waiting for us.
<lifeless> \o/
<StevenK> Hah
<wgrant> Although we'll just miss the following db-devel run.
<wgrant> So QA will be this evening.
<wgrant> What else can go wrong...
<wgrant> Build breakage, qastaging breakage, staging breakage, three RC revs, two bad revs, major QA backlog
<lifeless> 8000000 estimated plans make me cry
<wgrant> + deploying a big new feature at the same time
<lifeless> well
<lifeless> shortly after
<lifeless> I agree, it would be nice not to have it bundled in
<lifeless> wgrant: you redid the tag portlet queries recently right ?
<wgrant> lifeless: Yes.
<wgrant> They still suck.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/793809
<_mup_> Bug #793809: Distribution:+bugtarget-portlet-tags-content timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/793809 >
<wgrant> Will need bugsummary.
<wgrant> (which is in!)
<lifeless> yes
<lifeless> I'm hacking on that now
<wgrant> Fantastic.
<lifeless> just gthering data on th ecurrent behaviour
<wgrant> jelmer: Do you have a bzr-hg bug for that?
<lifeless> (actual time=0.164..0.164 rows=1 loops=323890)
<lifeless> what is 0.16ms * 300K :P
<jelmer> wgrant: bug 793812
<_mup_> Bug #793812: doesn't lock target repository when reading tags during fetch <affects-launchpad> <Bazaar Hg Plugin:In Progress by jelmer> < https://launchpad.net/bugs/793812 >
<wgrant> jelmer: Hm, weren't tags not supported before?
<wgrant> So this will only affect branches that were already broken?
<jelmer> wgrant: tags were ignored before, and *most* of these branches were not working earlier
<StevenK> lifeless: 48 seconds?
<wgrant> Hmmmm.
<wgrant> lifeless: Perhaps we should not roll back.
<wgrant> Given that we can roll out to the importds later fairly easily.
<jelmer> wgrant: (hence my answer "somewhat" to your question earlier; the risk is that there's a couple of branches out there that were previously working that have tags)
<lifeless> StevenK: yes, and thats what bug privacy costs us
<lifeless> jelmer: wgrant: I'm fine with saying that this only affects previously broken branches
<lifeless> but in that case its not qa-bad
<lifeless> its qa-ok
<wgrant> lifeless: It looks like it will break some existing branches.
<lifeless> fsvo some
<wgrant> But bzr-hg is still beta.
<wgrant> Well.
<wgrant> It looks like there are 15 succeeding bzr-hg branches.
<wgrant> In all of LP.
<wgrant> So it can't break more than 15 branches.
<lifeless> some of them may have tags
<StevenK> lifeless: "Orsum"
<lifeless> jelmer: when do you think this will be fixed ?
<jelmer> lifeless: tomorrow morning
<lifeless> StevenK: see the plan in the bug I linked if you are interested
<wgrant> I say we deploy with this, and rerelease to the importds on Friday or Monday.
<lifeless> jelmer: wgrant: FWIW I wouldn't rollback given this analysis
<wgrant> Right.
<jelmer> Yeah, I agree.
<wgrant> I was trying to convince you of that, but it seems it wasn't necessary.
<wgrant> Great.
 * mwhudson too, if anyone cares
<wgrant> jelmer: You've tested the others?
<jelmer> Sorry, I probably should've communicated that a bit better than just saying qa-bad and "somewhat"...
<wgrant> bzr-git seems happy, but I've not tried bzr-svn.
<wgrant> Heh.
<wgrant> It's late :)
<jelmer> I've tried bzr-svn and bzr-git
<wgrant> jelmer: qa-bad today means zomgpanic, since we have downtime in <48 hours.
<jelmer> wgrant: I realize that, but I also didn't want to say qa-ok while it could break existing branches (I hadn't bothered looking at the graph yet)
<wgrant> Yep.
<wgrant> Heh.
<wgrant> jelmer: Should we qa-ok all the bugs?
<jelmer> wgrant: yep
<wgrant> Hopefuly 13162 is good.
<lifeless> wgrant: btw, a window function eliminates the union
<wgrant> lifeless: Oh?
<wgrant> How?
<lifeless> tag in (official) or rank() < 10, or something like that
<lifeless> I won't need to do it
<wgrant> Hmm, true.
<lifeless> just noting another technique we could use to tune further
<lifeless> (49 rows)
<lifeless> Time: 595.781 ms
<wgrant> Nice.
<lifeless> thats for distro=1
<lifeless> https://bugs.launchpad.net/launchpad/+bug/793809/comments/3
<_mup_> Bug #793809: Distribution:+bugtarget-portlet-tags-content timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/793809 >
<cjohnston> wgrant: thanks for looking at my qa-needstesting's
<StevenK> cjohnston: Hah. wgrant looks at *everyone's* qa-needstesting's revisions :-)
<cjohnston> lol
<cjohnston> in that case, there's still one pending.. lol
<StevenK> cjohnston: He couldn't possibly steal all of your fun.
<wgrant> cjohnston: I'm waiting on a script to run on qastaging.
<cjohnston> :-)
<wgrant> hloeung: How's send-bug-notifications.py going?
<hloeung> wgrant: Still going... and going... and going...
<wgrant> hloeung: Is branchmail eating the CPU or something?
<StevenK> I had no idea send-bug-notifications.py was the energizer bunny
<wgrant> I see nothing new in the inbox.
<hloeung> wgrant: err, I'm seeing it print out lots of "Notifying <email@address> about bug XXXXXX" being displayed on my screen
<wgrant> Ah, yes, Thunderbird was just being slow.
<hloeung> It's not just the branchmail, there's quite a few jobs running right now
<hloeung> so yes, the load is pretty high on that server
<wgrant> :(
<StevenK> hloeung: Define pretty high? 10? 50?
<hloeung> StevenK: anything over 10 is high for me, and it's at 12
<LPCIBot> Project windmill-db-devel build #369: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-db-devel/369/
<wgrant> StevenK: Can I have mawson?
<StevenK> wgrant: Take it, I'm done with it until after lunch.
<wgrant> Thanks.
<wgrant> lifeless: We are green.
<wgrant> Shall we unfreeze?
<wgrant> wallyworld: The pickers are healthy when the flags are all disabled?
<wallyworld> wgrant: afaik
<wgrant> wallyworld: Great.
<wallyworld> i last tested that a day or two ago
<wallyworld> but there have been code changes since
<wgrant> Yeah,.
<wgrant> I'll check once hloeung's back.
<wgrant> By disabling them on qas.
<wallyworld> i think prod has an earlier version deployed that is fine
<wgrant> Yeah.
<wgrant> But that's about to change.
<wgrant> Well, the earlier version thing is about to change. The fineness hopefully won't.
<wallyworld> i tested when i put in support for the result ordering to be behind a flag
<wgrant> But we've had lazr-js changes and such since.
<wallyworld> yes, but the lazr-js changes don't use a ff
<wgrant> So they are more likely to break even when the FF is off.
<wallyworld> lazr-js will use json attributes if they are there, which they won't be if the ff is off
<wgrant> Sure. But this is Launchpad.
<wgrant> I'm still going to test it :)
<wallyworld> ok
<lifeless> wgrant: fully, really ?
<wgrant> lifeless: Hm?
<wgrant> Oh, QA? Yes.
<wgrant> Both are fully green.
<lifeless> \o/
<lifeless> daaaaploy
<wgrant> Already requested.
<lifeless> love your work
<wgrant> Will do it and unfreeze and remerge once hloeung is delunched.
<wgrant> Heh.
<lifeless> garrrr
<lifeless> storm reference columns paaaain
<wgrant> Yes.
<wgrant> ~/launchpad/lp-branches/devel$ bzr diff -c13163 | wc -l
<wgrant> 20211
<wgrant> D:
<lifeless>  lib/lp/bugs/doc/bug-tags.txt
<lifeless>   Ran 1 tests with 0 failures and 0 errors in 0.679 seconds.
<lifeless> \o/
<lifeless> wgrant: (48 rows)
<lifeless> Time: 165.065 ms
<wgrant> lifeless: Where'd the 49th row go?
<wgrant> And the other 340ms.
<lifeless> I had forgotten we aggregate up by sourcepackagename
<lifeless> so it was counting a few hundred K rows too many
<wgrant> Ahh
<lifeless> the extra row was apport-collected
<lifeless> not sure why its gone, looking
<lifeless> is there with spn is null
<lifeless> my theory - it was 11th with the duplicates fixed.
<lifeless> because its /all/ source package targeted bugs it had 100% inflation
<wgrant> Ahh
<wgrant> That could do it, I guess.
<lifeless>  3626
<lifeless> (null)
<lifeless> and
<lifeless>  3869
<lifeless> (not null)
<lifeless> discrepancy handwaved away by multi-task bugs
<lifeless> that takes its count to 7K
<lifeless> and that was 10th place
<lifeless> hmm,t hats ever
 * lifeless checks with open
<lifeless> 2100
<lifeless>  rank |          tag           |  sum
<lifeless> ------+------------------------+-------
<lifeless>     1 | i386                   | 34609
<lifeless>     2 | apport-crash           | 29323
<lifeless> ...
<lifeless>    14 | apport-collected       |  2003
<michaelh1> Hey, the 'Packaging/SourceBuildsdaily builds knowledge base' link at the bottom of 'https://help.launchpad.net/Packaging/SourceBuilds/BzrBuilder' is broken.  I can't seem to log in to fix it...
<lifeless> can has review ? https://code.launchpad.net/~lifeless/launchpad/bug-793809/+merge/63635
<wgrant> michaelh1: login attempts seem to hang for me :/
<lifeless> sso is having trouble probably; we have a bug open on that
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | devel is release-critical until r13164 deployable | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:212 - 0:[######=_]:256
<wgrant> Ah, finally worked.
<wgrant> michaelh1: Fixed, thanks.
<michaelh1> Ta
<hloeung> wgrant: You called?
<wgrant> hloeung: Hi! Three things:
<hloeung> okay, hit me
<wgrant> 1) Could you please bring PQM out of RC mode (I think both devel and db-devel are RC at the moment)?
<wgrant> 2) Could you remove the disclosure.picker_enhancements.enabled feature rule from qastaging, so we can test how it will behave on production without it tomorrow?
<wgrant> 3) There's a nodowntime deployment request on LPS, to deploy all the stuff before the db-stable merge.
<wgrant> lifeless: Reviewed.
<lifeless> wgrant: dunno whats wrong with the indentation
<jtv> wgrant: if I have a PackageUpload pu, and I do pu.addSource(â¦), should that be enough to make contains_source become True?
<wgrant> jtv: Yes.
<wgrant> jtv: Minus caches, but there might not be any.
<wgrant> lifeless: Multi-line function calls are meant to be indented like this:
<wgrant> something(
<wgrant>     foo, bar)
<wgrant> Not
<wgrant> something(foo,
<wgrant>     bar)
<wgrant> And multi-line function definitions are to be indented like:
<wgrant> def something(foo,
<wgrant>              bar)
<wgrant> I forgot a space.
<wgrant> But you get the idea.
<lifeless> problem is both of those are fugly.
<lifeless> you'll find indenting like I did already present in the code base
<lifeless> -> vet, back soon
<wgrant> lifeless: Hm, I think the postgres planner needs a good killing.
<jtv> Nice argument there: the same person who won't listen to his reviewer and created optional reviews says "I can do it the way I like because we already have places where it's done that way."  :-)
<wgrant> Heh.
<jtv> What's the problem with the postgres planner?
<wgrant> jtv: See the last comment in the MP.
<wgrant> https://code.launchpad.net/~lifeless/launchpad/bug-793809/+merge/63635
<jtv> The answer to that is not killing the planner.
<wgrant> It would help.
<jtv> No it wouldn't â then we'd have to hand-tune _every_ query at the cost of legibility.
<jtv> What Robert mostly uses WITH for AFAICS is as a new optimization barrier.
<wgrant> Yeah.
<StevenK> I was tempted to read the slides about the query planner from the last Postgres meetup, but I chickened out
<jtv> It's just an ever-shifting battleground: we find we can get a query faster (at least for the cases we're looking at) by making it too hard for the optimizer to restructure.
<jtv> Then the optimizer gets smarter, and most queries get faster â but it breaks that trick.
<jtv> The fights over planner hints are endless.
<jtv> Even though the decision is already made: we're not going to have them.
<wgrant> Heh.
<wgrant> Yay, I can self-review now.
<StevenK> But you have been for months with rs=wgrant
<StevenK> Or was I not supposed to point that out?
<wgrant> That was always for testfixes/merges.
<wgrant> No new code :)
<wgrant> But yes, shh.
 * jtv worriedly pretends he had not heard that
<wgrant> Hmm.
<wgrant> Pickers with dozens of pages look a bit bad.
<wgrant> But it'll do.
<wgrant> eg. search for 'launchpad' in the product picker on https://bugs.qastaging.launchpad.net/launchpad/+bug/1234
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<jtv> This is a bit of python syntax I'm still missing: the "if" clause from list comprehensions, but without the "for"
<jtv> Names = [self.name if self.name is not None]
<wgrant> Yes :(
<wgrant> filter()'s deprecation upset me.
<wgrant> Same with map().
<jtv> Those have been deprecated!?
<StevenK> What jtv said?!
<jtv> Surely that's only in MS Python.NET enterprise?
<jtv> Or some other thing that we don't care about?
<wgrant> Hm, looks like they weren't actually killed from py3k. There were going to be at one point.
<jtv> Phew.  So their obsolescence was deprecated?
<wgrant> map/filter appear to be iterators in py3k, though.
<jtv> That makes more sense.
 * jtv dreams for a moment about the cool vectorizations we could have if this wonderful language were a compiled one
<jtv> Then again, modern compilers can do that without it being explicit in the code can't they
<wgrant> Yes :(
<lifeless> -> vet, back soon]]]]]({{{{{kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkK0;2~")Y9;2~<GF[3;2~String formatting in SQL: Australia says no.
<lifeless> Also, calling that "store" is a bit too much of a lie for my tastes.
<wgrant> Did you forget to take the cat?
<StevenK> Hahaha
<StevenK> wgrant: Perhaps that was cat clawing despertly in a vain attempt to not be scooped up
<wgrant> We can only hope!
<lifeless> jtv: uhm, I do listen to reviewers - a lot.
<jtv> Shhh you're spoiling it
<StevenK> Never let the facts get in the way of a good punch line
<wgrant> hloeung: PQM is indeed un-RC'd. Thanks.
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | On call reviewer: wgrant | Critical bugs:212 - 0:[######=_]:256
<hloeung> wgrant: How did you check? Just so I know in the future how to check
<wgrant> hloeung: I landed something :/
<hloeung> Oh hahah
<lifeless> so, if you look at the query plans *we get*, the planner often treats exists / subqueries as things to run multiple times
<wgrant> lifeless: Sure. But it should be smart enough to not do that.
<lifeless> jtv: if you can find a way to get it to not do that, without a with, that would be cool.
<wgrant> At least sometimes.
<lifeless> AFAICT using WITH as a way to declare what is essentially a variable is good style SQL wise.
<jtv> lifeless: I don't know of a way â I think it's one of those cases where it just wasn't the extra optimizer time.
<jtv> Yes, it's fine.
<wgrant> lifeless: Perhaps good style, but unconventional.
<jtv> Well no, that's exactly what it's for.
<wgrant> Right. But it's not very much.
<wgrant> I've rarely seen it used.
<wgrant> And it's ugly in Storm.
<jtv> It's relatively new.
<wgrant> Right. New => unconventional.
<lifeless> wgrant: so, help make it better in storm?
<lifeless> wgrant: unconventional in our code base perhaps.
<wgrant> I've read enough horror stories in that MP.
<jtv> The main "good" reason to use it is to remove such duplication.  The main practical reason is as an optimization barrier.
<lifeless> anyhow, when storm is three times the size of direct queries, I really start to wonder why we use it.
<jtv> How does it work in storm?
<lifeless> jtv: see the use of with_ in https://code.launchpad.net/~lifeless/launchpad/bug-793809/+merge/63635
<jtv> Ah, great to have that as a keyword isn't it?  :)
<lifeless> yes (no)
<lifeless> wgrant: so, I
<lifeless> 've changed the browser/ indenting as its much of a muchness
<lifeless> (though its still fugly)
<wgrant> Not a muchness!
<wgrant> Heh.
<lifeless> the bug.py indent is really harder to read
<wgrant> Oh?
<lifeless> def foo............
<lifeless>                            fwejijw
<lifeless>     """.....
<lifeless> dunno about you but that plays havoc with my eyes
<wgrant> It's not ideal.
<wgrant> But not much is.
<lifeless> I can raise a discussion on list about this if you like, but this is the sort of thing I really hate about coding standards.
<jtv> We've had that discussion.
<StevenK> We have
<lifeless> jtv: I don't think the discussion is over
<jtv> I hate it too, but having the consistency and stability is probably more valuable than making it prettier in that case.
<StevenK> Can the bikeshed not change colour again?
<jtv> Exactly.
<lifeless> StevenK: I've been consistently arguing one thing.
<StevenK> lifeless: And you keep bringing it up because the bikeshed is a different colour, it seems.
<lifeless> StevenK: no, I keep bringing it up because I have neither been convinced that the status quo is a good place to be, nor that what I am arguing for is a bad place to be.
<StevenK> IOW, the bikeshed needs a new coat of paint
<lifeless> or to not be a bikeshed
<jtv> The great thing is, if we change it now, the people who argued the other way get their say again.
<lifeless> hah
<jtv> That can't be bad, right?
<lifeless> interestingly PEP8 says we are wrong.
<lifeless> not for function definitions
<lifeless> but for function calls
<lifeless> jtv: healthy discussion is fine.
<StevenK> lifeless: In the words of the Debian Policy maintainers, why must we make hundreds (if not thousands) packages (read as: lines of code) insta-buggy?
<lifeless> StevenK: you seem you have your troll hat on. Can you take it off please?
<jtv> If healthy discussion is fine, by all means let's have more of it.  Now excuse me while I get some work done.
<lifeless> so I was discussing this with my reviewer; the rest of you just piled in :)
<StevenK> lifeless: Do I have to? :-(
<jtv> lifeless: Because you didn't want to listen to your reviewer, which you deny.  Seriously, it's pretty clear to me that you're the one with the troll hat on today.
<lifeless> right, breath taken.
<lifeless> jtv: there is a huge difference between dialogue and instruction.
<lifeless> jtv: if our reviewers are having a dialogue, then its up to the coder to listen and choose; making it an absolute must-be-X stops it being a dialogue and makes it instructions.
<StevenK> lifeless: In my experience, reviewers can do either.
<StevenK> lifeless: Must-be-X is usually of the form "That is horrid! *needs fixing*"
<lifeless> I have been wondering whether removing that from reviewers entirely would help folk understand the cultural distinction
<lifeless> I'm going to go off to the side and finish this specific discussion with wgrant, we can chew the fat about big picture stuff another time
<wgrant> lifeless: Hm, so anybody can r-c bless truly r-c stuff now?
<lifeless> yes
<wgrant> Great.
<wgrant> Makes sense.
<lifeless> more centralised authority undone
<wgrant> Yup.
<lifeless> let folk -shock, horror- make decisions.
<wgrant> Except for formatting :P
<lifeless> I have a list.
<wgrant> Eep.
<lifeless> wgrant: ?
<wgrant> An OOPS editing a bug.
<wgrant> Trying to find it.
<wgrant> I can't find CE anywhere...
<wgrant> Ah, wampee's rsync config has changed.
<wgrant> AssertionError: Attribute bug_subscribers not in object <BugTask for bug 97266 on <Product at 0xd8588d0>>
<_mup_> Bug #97266: Suggestions menu next to "Choose..." doesn't do anything <javascrcipt> <lp-web> <project-picker> <qa-ok> <webkit> <Launchpad itself:Fix Committed by sinzui> < https://launchpad.net/bugs/97266 >
<wgrant> Panic.
<lifeless> ?!?!
<lifeless> that seems undesirable
<wgrant> OOPS-1984CE15
<wgrant> Yes.
<lifeless> HTH did it get through tests
<wgrant> It doesn't affect all bugs.
<wgrant> I bet it's the thing I've rolled back three times already.
<wgrant> The dupe subscriptions muting thing.
<wgrant> I guess one of getDirectSubscribers and getIndirectSubscribers is raising an AttributeError...
<wgrant> Reproducible on qastaging, I guess we should roll back.
<wgrant> And then panic.
<lifeless> yup
<wgrant> hloeung: Hi :(
<hloeung> wgrant: Hi
<lifeless> will this prevent unhiding the subscriptions feature?
<wgrant> lifeless: It prevents deployment.
<wgrant> Of anything.
<wgrant> Hence panic.
<lifeless> even after reverting the rev ?
<wgrant> Ah. No, probably not. It's ugly, but not entirely critical.
<wgrant> The issue is that there won't be a mute link if you only have a dupe subscription.
<lifeless> we can deal for a few days lke that I think
<wgrant> Possibly it must even be a team subscription to a dupe.
<wgrant> Yes.
<wgrant> hloeung: We need to roll back that deployment, I'm afraid.
<hloeung> wgrant: okay sure, to what REVNO?
<wgrant> hloeung: 13144, the previous nodowntime.
<wgrant> lifeless: Fourth time lucky :/
<hloeung> that last rollout actually had issues, not sure if it's related - https://pastebin.canonical.com/48230/
<hloeung> neumayer rebooted itself
<wgrant> hloeung: Not related, but still concerning.
<wgrant> Ah...
<wgrant> That's even more concerning.
<lifeless> wgrant: I need to prep dinner
<lifeless> I will drop by in a bit to touch base
<wgrant> lifeless: Thanks, but it should be OK.
<wgrant> I do wish we had a gary now.
<hloeung> wgrant: https://pastebin.canonical.com/48232/
<wgrant> hloeung: Hm, you're doing a full redeploy?
<wgrant> hloeung: The deployment tool has an option to rollback to a previous rev.
<wgrant> That's already built.
<hloeung> oh?
<wgrant> --rollback-to or somesuch... let me find it.
<wgrant> 00:48 < mthaddon> Chex: should just be ./deploymgr.py --config=lp --revert-to-revno=XXXX lpnet edge
<wgrant> But we want all of nodowntime this time.
<wgrant> Note that the tree has to already be built.
<wgrant> So it may be too late if you've already killed it.
<wgrant> In which case we will have to recover from that conflict, which is easy enough.
<wgrant> Grarrrrrr
<wgrant> Conflicts reverting this.
<hloeung> what should we do?
<StevenK> Hm, perhaps the deployment manager should remove the sourcedeps.cache
<wgrant> StevenK: More like we should fix the bug which makes it depend on dict ordering.
<wgrant> But we have bigger crises than that right now.
<wgrant> hloeung: Oh, sorry, missed that message.
<wgrant> hloeung: is the old tree still present, so --revert-to-revno will work?
<hloeung> where do I check?
<wgrant> I'm... not sure. See if you can find the directories with revnos in them, and see if there's an existing one for r13144. Not sure whether it needs to be on prasÃ© or everywhere else.
<wgrant> deploymgr is still black magic to me.
<hloeung> checked out a couple of servers and it looks to be there
<wgrant> I guess try a --revert-to-revno and see if it works.
<wgrant> I think it should.
<wgrant> The tree should even still be on prasÃ©, even though that shouldn't matter.
<LPCIBot> Project windmill-devel build #177: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/windmill-devel/177/
<wgrant> I guess we need to re-RC PQM :/
<wgrant> Or we could just not bother, since we're sorta screwed either way.
<hloeung> so cold today
<wgrant> Yes :(
<hloeung> Seems to be working
<hloeung> soybean is back at 13144
<hloeung> it's currently doin wampee
<hloeung> *doing
<wgrant> hloeung: Fantastic, thanks.
<wgrant> Much quicker than a full rebuild, see :)
<hloeung> definitely
<wgrant> hloeung: I guess we should fix that conflict.
<wgrant> Might as well just revert the whole tree, I guess.
<wgrant> But I don't really know how that works...
<hloeung> just leave it with me, I'll ask Tom when he gets online
<hloeung> now what about pqm in RC mode? leave that as is?
<wgrant> I guess we should put it back in RC, just in case. It's just about trivial.
<lifeless> wgrant: hloeung: re-rc please
<lifeless> may as well mitigate the damage
<wgrant> Yeah.
<hloeung> lifeless: okay!
<LPCIBot> Project windmill-devel build #178: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill-devel/178/
<adeuring> good morning
<jtv> hi adeuring
<adeuring> hi jtv!
<lifeless> jml`: guten morgen
<jml`> lifeless: good morning
<jml`> and good morning to all
<jml`> huwshimi: good morning
<huwshimi> jml`: Hello! Welcome back :)
<jml`> thanks.
<jml`> hmm
<jml`> who's the other jml...
<jml> there we go.
<huwshimi> jml: Someone had to be you while you were gone
<StevenK> Haha
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday! | devel release-critical again until r13168 deployable | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:212 - 0:[######=_]:256
<jml> huwshimi: firing up skype now.
<jml> huwshimi: I can hear you. gimme a sec.
<mrevell> Good morning everyone.
<wgrant> gmb: Dupe muting seems to be somewhat ill-fated :/
<gmb> Yeah.
<gmb> I feel like I'm stuck in a Python version of Groundhog Day.
<jml> o/` Just put your little hand in mine... o/`
<bigjools> jml is having a relapse
<jml> ... back to Groundhog Day
* gmb changed the topic of #launchpad-dev to: Performance Tuesday! | devel release-critical again until r13168 deployable | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs:212 - 0:[######=_]:256
<jml> why might I get a bug task that has a closed status but a null date_closed?
<lifeless> very old data?
<jml> lifeless: Perhaps. the query is modified_since=2011-05-11, status=CLOSED_STATUSES
<jml> lifeless: it's conceivable that an old bug was modified but not closed in that time
<lifeless> bug # ?
<jml> lifeless: I don't know.
<lifeless> k
<jml> afk for a bit.
<lifeless> meep
<lifeless> Bugs fixed elsewhere 125
<lifeless> takes ~ 1 second to generate on its own
<jml> lifeless: that's pretty bad.
<danilos> though, if it took 1 second to fix 125 bugs elsewhere, that'd be pretty good
<danilos> (not saying this is useful or relevant observation :)
<lifeless> danilos: :P
<lifeless> jml: it is
<lifeless> jml: I'm wondering if its sufficiently useful to show a number at all
<lifeless> jml: should we talk about the wiki briefly ?
<jml> lifeless: some other day, if that's ok. I'd rather catch up on it via email along with all of the other stuff.
<lifeless> sure
<lifeless> I have done nothing
<lifeless> on the basis that all my concerns are technical
<lifeless> and we don't seem even close to that sort of discussion yet
<lifeless> s/all my concerns/all the things I'm concerned about *and should be concerned aout*
<jml> lifeless: yeah.
<lifeless> jml: but perhaps us talking about it may help with scope or clarity. I dunno.
<jml> lifeless: my main concern is giving these guys a fighting chance of actually helping with the wiki without smothering them in blockers. honestly, it's pretty low on my priority queue atm.
<lifeless> jml: yes, there are some competing concerns here
<LPCIBot> Project windmill-db-devel build #370: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-db-devel/370/
<LPCIBot> Project devel build #783: FAILURE in 5 hr 47 min: https://lpci.wedontsleep.org/job/devel/783/
<gmb> Anyone got a spare minute to give me a review on https://code.launchpad.net/~gmb/launchpad/bug-61531/+merge/63672?
<bigjools> wgrant: are you managing this rollout?
<wgrant> bigjools: In no official sense, but I have done much of the preparation.
<bigjools> wgrant: ok, do you know if db-devel 10648 should have been merged to devel?  It didn't make it since 10647 was merged
<bigjools> not even sure why it landed there, they didn't change anything under database/
<wgrant> bigjools: I remerged db-stable earlier.
<wgrant> Up to 10651, IIRC.
<bigjools> \o/
<bigjools> which saves me a shitload of work!
<wgrant> 10648 depended on stuff in db-devel.
<wgrant> But only 10647 was on staging when I needed to do the first merge.
<wgrant> Why?
<bigjools> because I merged one too many revisions from db-devel yesterday and then did a load of work on top of that
<wgrant> Ahh.
<bigjools> see #bzr :)
<danilos> gmb, fwiw, the Bug:+subscribe page doesn't offer mute functionality, you have to go to Bug:+mute page for that (regarding your MP, I don't have the time to review it right now, but just noticed this)
<gmb> danilos: Ah, bugger. I forgot that had changed.
<gmb> Thanks. I'll update my branch.
<danilos> gmb, yw
<gary_poster> danilos, thanks for mail about bug 788874.  Do you (or wgrant) know if the task is to simply review and shepherd the branch poolie has?  Or is there something more that needs to be done?
 * gary_poster looks at branch
<gary_poster> ah
<gary_poster> nm
<bigjools> please make the code import emails stop :(
<jelmer> bigjools: sorry about that :-/
<jelmer> I wonder if there's an easy way we can clean them up without generating so much noise
<bigjools> put the contact address back until the bug is fixed? :)
 * jelmer wasn't aware anything had changed, been getting these for years
<bigjools> oh
<bigjools> it's vcs-imports I think
<maxb> I don't mind getting ~vcs-imports mails. They are a bit spammy, but it can be useful to review my saved folder of them as a log at times
<jelmer> canonical-launchpad is a member of vcs-imports, and vcs-imports is set to mail each member individually
<jelmer> I can certainly restrict it while I do the cleanup
<bigjools> I honestly don't know what the best thing to do is
<bigjools> I just know that some of us are getting more email than we want :)
<benji> gary_poster: I am indeed.
<benji> pfft
<jml> benji: good morning.
<benji> morning, jml; I trust all is well with you.
<jml> benji: almost everything is very well, thank you.
<jml> benji: however there's a current production issue (see recent discussion on #launchpad-ops), and I was wondering if you'd be able to take care of it.
<jml> benji: wgrant says that it's https://wiki.canonical.com/IncidentReports/2011-05-27-LP-2011-05-27-LP-incoming-mail-wedged come again.
<benji> jml: sure, I'll be glad to take a look
<jml> benji: thank you.
<StevenK> Is the topic to date in terms of revision number?
<wgrant> StevenK: Unless something else has broken since I EODed.
<wgrant> Which is not unlikely.
<wgrant> But let's see.
<wgrant> StevenK: It is up to date.
<jelmer> wgrant: could it be that nothing was rolled out to the importds yet?
<bac> gmb: after our call could you look at https://code.launchpad.net/~bac/launchpad/bug-789383/+merge/63695
<wgrant> jelmer: Ah, yeah, about that...
<wgrant> jelmer: We had deployed it.
<wgrant> jelmer: Then I got half-way through closing the bugs.
<wgrant> jelmer: When one of them OOPSed.
<wgrant> So we had to roll back.
<wgrant> Sorry, entirely forgot I'd closed a few of them by that point.
<jelmer> wgrant: ah, ok - what was the OOPS?
<wgrant> jelmer: See the end of bug #772609
<jelmer> wgrant: thanks
<jelmer> wgrant: sounds like you had a tough day... have a good evening :)
<Ursinha> hello
<jml> hello
 * jml out for lunch & errands.
<flacoste> gmb: so what's the story? is 13154 a blocker or not? from deployment-stable.html and the bug report it looks like it, but i'm not sure from your email exchange with wgrant on the list
<wgrant> flacoste: It's a blocker.
<wgrant> flacoste: We had to revert the nodowntime deployment. I guess I forgot to put that bit in the email.
<wgrant> But simply reverting it is OK; it need not be fixed and relanded.
<allenap> Eyup gmb, got time for a shortish straightforwardish review?
<gmb> allenap: Sure
<allenap> gmb: Thanks :) https://code.launchpad.net/~allenap/launchpad/derives-from-portlet-bug-793547/+merge/63703
 * gmb looks
<allenap> gmb: Oops, I forgot to retarget to db-devel... one moment.
<gmb> Woah, conflcty.
<wgrant> allenap: Why's that db-devel?
<flacoste> wgrant: ok thanks, and i guess gmb is handling the reversal?
<wgrant> allenap: Unless that has a DB patch, it can land on devel.
<wgrant> flacoste: I reverted it hours ago, and it just passed buildbot.
<gmb> flacoste: I thought it had been reverted
<wgrant> flacoste: Should be on qastaging in half an hour or so.
<gmb> AH, which it has.
<gmb> Molto bene.
<allenap> wgrant: Has db-devel been merged into devel recently? If not, then I strongly suspect this depends on stuff that's only in db-devel.
<wgrant> allenap: We release in <36 hours. db-devel has nothing that devel doesn't have.
<wgrant> allenap: It was merged into devel twice yesterday.
<allenap> wgrant: Okay. I wonder why so many conflicts then :-/
<wgrant> Possibly a criss-cross... but I deliberately tried to avoid that.
<wgrant> Maybe you merged devel recently?
<flacoste> wgrant. gmb: thanks a lot!
<wgrant> flacoste: So we should be deployable again in an hour.
<flacoste> awesome!
<wgrant> And hopefully nothing more will crop up tomorrow, because we've really had enough of that sort of stuff already.
<allenap> gmb: https://code.launchpad.net/~allenap/launchpad/derives-from-portlet-bug-793547/+merge/63704 is targeted to db-devel, which gives a sensible diff and can be reviewed. I'll try and land it to devel though, if I can figure out wtf is going on.
<gmb> RIGHT, TA.
<allenap> Thanks :)
<gmb> Woah.
<gmb> That was a bit old-man-of-Lancashire.
<gmb> RIGHTO LAD, TA.
<allenap> gmb: That's exactly as I interpreted it :)
<gmb> I shall go and get me cap and pipe.
<wgrant> allenap: I suspect things should go away if you merge db-devel in a few minutes.
<gmb> And then tek a sken at thy branch.
<wgrant> Or devel.
<wgrant> Hm.
<wgrant> PQM sees a conflict merging stable into db-devel. I do not.
<wgrant> Ahh, there.
<allenap> wgrant: Cool, cheers.
<gmb> allenap: Approved
<allenap> gmb: Thanks :)
<StevenK> jml: O hai, can you recall off the top of your head if there is a bug about ec2 land giving an ugly traceback if the URL provided is not an MP?
<jml> StevenK: I don't *think* so. I've tried to make sure bugs like that are tagged ec2land, and I can't see any.
<StevenK> jml: I'm happy to file one -- I just wanted to avoid the "Oh, damn, someone has had that idea already."
<jml> StevenK: that'd be great. thanks.
<StevenK> jml: Either that, or maybe I fix it.
<jml> StevenK: that'd work too :)
<StevenK> jml: Could not think of an elegant way, sadly. :-(
<StevenK> jml: Bug 794067
<_mup_> Bug #794067: ec2 land gives traceback when URL isn't an MP <ec2land> <Launchpad itself:Confirmed> < https://launchpad.net/bugs/794067 >
<jml> StevenK: funny. I could have sworn that bug had been filed.
<RawChid> Hey, from Python I try to load a pot from LP (staging). The code should work and I should have the right permission, nevertheless I get HTTP Error 401: Unauthorized  (Unknown access token). Any ideas?
<StevenK> jml: Bah
<RawChid> By "the code should work" I mean I just checked it out and i works for other people
<jml> StevenK: but I can't find it.
<StevenK> jml: I doubt we can use isinstance() in this case, though
<jml> StevenK: to handle this case, you could do this:
<jml> StevenK: assume the URL is for a branch; if that branch has exactly one merge proposal that is Approved (or maybe Needs review), then use that merge proposal
<jml> StevenK: otherwise assume the URL is for a merge proposal
<jml> still doesn't handle cases where the user specifies http://example.com/whatever or LP object URLs that are neither branches nor MPs.
<jml> how come I still don't have food yet?
<jml> gnuh.
<jml> bbiab.
<StevenK> jml: Yes, I was hoping to figure out an elegant solution that handled both cases.
<henninge> gmb: Hi! Can you please review this branch of mine?
<henninge> gmb: https://code.launchpad.net/~henninge/launchpad/bug-347117-duplicate-template-name/+merge/63710
<henninge> gmb: it claims to be a 629 line diff but it's only 238. See the comment. ;-)
<cjohnston> henninge: they are done! yay
<henninge> cjohnston: yes. Thanks for your contribution and sorry it took so long.
<wgrant> cjohnston: Uh, well, sort of.
<cjohnston> Not a problem.. Part of it taking so long was me and my not knowing, then me getting too busy.
<wgrant> cjohnston: They were deployed for about 20 minutes.
<cjohnston> uh
<wgrant> cjohnston: When we had to revert for unrelated reasons.
<wgrant> They will be redeployed on Thursday.
<cjohnston> ok.. well close enough to being done
<cjohnston> lol
<RawChid> Hm, when running I get "The kwalletd service has been disabled". Maybe this has something to do with the authentication error?
<wgrant> I forgot to reopen the bugs, sorry.
<cjohnston> oh well
<cjohnston> They are ready and able to go, and that's as much as I can hope for
<cjohnston> Although it will be cool when my text is spamming a gazillion people on each bug email.. lol
<bac> gmb:  can you take a look at https://code.launchpad.net/~bac/launchpad/bug-789383/+merge/63695 ?
<gmb> bac: Syre
<gmb> *Sure
<gmb> bac: Approved.
<henninge> deryck: Hey!
<henninge> gmb: If you could also have a look at my branch, please, I'd be very happy. ;-)
<gmb> henninge: Ah, sorry. I missed your ping. Looking now.
<henninge> gmb: np
<gmb> henninge: I like this approach of saying "Everything after line XXX isn't interesting." I'll try and land more changes that way...
<jml> sinzui: hello
<jml> sinzui: I'm moving the disclosure roadmap doc that we worked out to the wiki. Any place that would make sense for you?
<sinzui> jml, do we have a page for current features? I think it would be helpful to have pages showing the derived distros and disclosure roadmaps
<jml> sinzui: no we don't. we are exploring uncharted realms of documentation.
<jml> sinzui: and yes I fully agree.
<jml> sinzui: but I'm afraid the disclosure feature is the guinea pig for whatever doc / proj. planning experiments we're doing.
<jml> sinzui: if you don't have a preference for a location on the wiki, I'll just add it somewhere that makes sense to me.
<sinzui> jml: forward me the URL after your decision
<jml> sinzui: will do.
<gmb> henninge: Approved.
<henninge> gmb: thank you very much ;-)
<gmb> np
* gmb changed the topic of #launchpad-dev to: Performance Tuesday! | devel release-critical again until r13168 deployable | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:212 - 0:[######=_]:256
<jml> hey, what's that word for "post-mortem" that people use to avoid saying "mortem" when they want to talk about a project that's just finished?
<timrc> on does the on call reviewer process work? I need to assign a reviewer to a merge proposal... original pre-implementation discussion happened mostly w/ bigjools
<timrc> on does? how does..
<bigjools> timrc: looks like there's no OCR at the moment
<bigjools> I would review your work but I am really swamped, sorry
<timrc> bigjools, I've seen one for awhile... is there a certain time of day or night I should be here?
<timrc> gah... I haven't seen one
<bigjools> it depends on the timezone they are in
<timrc> bigjools, is there a schedule on the wiki, perhaps?
<bigjools> timrc: https://dev.launchpad.net/ReviewerSchedule
<bigjools> :)
<timrc> nice
<bigjools> I think you missed both of today's
<timrc> bigjools, naturally :)
<timrc> I'll wait 'til Wednesday, that's fine
<bigjools> you'll have to wait, or sweet talk someone
 * timrc accidentally drops a hundred dollar bill near abentley's foot... oops... can you get that for me?
<bigjools> bad luck actually, today is the only day where there's no reviewer in north america
<bigjools> timrc: that could be interpreted more than one way ...
<timrc> bigjools, doh
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #784: FIXED in 5 hr 23 min: https://lpci.wedontsleep.org/job/devel/784/
<jcsackett> timrc: there's a launchpad reviewers team that you should assign your MP too; many OCR people just work off a queue of projects in +activereviews on their day, so it increases the liklihood that someone will look at it without you needing to nag anyone.
<jcsackett> it's not a guarantee that someone will look at it without them pinging you, but it's been known to happen.
<jml> jcsackett: there's a team? OCRs don't just use https://code.launchpad.net/launchpad-project/+activereviews?
<jcsackett> jml: that's the queue. assigning to the reviewer team means anyone who might be OCR can look at your MP.
<jcsackett> i conflated two parts of how things work. sorry. :-P
<jml> jcsackett: isn't that the default anyway?
<jcsackett> jml: it may well be, but timrc was asking about assigning it to someone.
<jml> jcsackett: oh right.
<jml> sorry. carry on.
<timrc> jcsackett, I don't this the review team in the directory..
<timrc> oh man typing skills seriously lacking today
<jcsackett> timrc: happens to all of us. i'm assuming you meant "i don't think the review team is in the directory" ?
<timrc> jcsackett, yep :)
<jcsackett> timrc: don't worry about it. as jml pointed out, if you leave that blank launchpad-reviewers is assigned automatically.
<jcsackett> if i were better at providing help on irc, i would have told you that first before babbling about +activereviews. :-P
<timrc> ok, let me remove cody-somerville from the review... I'm sad because I put this merge proposal in last week which means 100,000+ lines have probably changed :)
<cody-somerville> timrc, merge from trunk and push then
<timrc> cody-somerville, yeah yeah, I'm just trying to guilt-trip you :)
<cr3> when making database changes, as per /PolicyAndProcess/DatabaseSchemaChangesProcess, where do I make the database changes considering a patch number is only assigned after review?
<cr3> I'm thinking database/schema/launchpad-2208-00-0.sql but that feels weird, so might as well ask
<deryck> cr3, make a launchpad-XX.sql and when you have a number rename the file.
<deryck> cr3, see #10 under "making a database patch" on that page
<cr3> deryck: excellent, thanks
<deryck> np
<nigelb> StevenK: Happy Birthday!
<lifeless> morning
<lifeless> gary_poster: hi
<lifeless> gary_poster: can we talk about bug 772609 and the impact on the downtime deploy ?
<gary_poster> hey on call
<_mup_> Bug #772609: bug subscription mute link is not shown for membership in a team with a direct or structural subscription <bad-commit-13003> <bad-commit-13154> <qa-bad> <story-better-bug-notification> <Launchpad itself:Triaged by gmb> < https://launchpad.net/bugs/772609 >
<lifeless> gary_poster: after your call is fine, if you have time.
<deryck> sinzui, if you're curious, here's the IE filebug issue fix:  https://code.launchpad.net/~deryck/launchpad/ie-filebug-error-500015/+merge/63763
<sinzui> wow
<sinzui> deryck: I have a working yui test suite on the functional layer. I cannot land it without fixing 5 broken tests that we could not see
<deryck> oh, cool.  running yui tests via Windmill?
<deryck> I had forgot you were trying to fix that.
<deryck> I don't think this will affect that, though.  It was never a problem in yui_tests, only with the big Windmill tests.
<sinzui> No windmill
<sinzui> deryck: I wrote a simple browser using pywebkit. It loads a page from the filesystem and returns when the command completed
<sinzui> no polling, no waits
<deryck> very cool!
<sinzui> We do have js tests that run in 20 seconds. They violate our fast test policy though
<deryck> ouch
<sinzui> All but 7 tests pass in 45 seconds on my computer. the remaining take the suite into 3 minutes
<deryck> I can't wait for this to land!  sinzui, I can help you fix the remaining yui tests if you like.
<sinzui> I might be able to fix them. two were bad setup/teardowns on our part
<deryck> cool.  well don't hesitate to ask if you need help.
<sinzui> thanks
<deryck> alright, and with that, I'm out.  until tomorrow everyone....!
<gary_poster> lifeless: hi.  I was nervous about that bug, but gmb seemed to think it was under control.  I didn't dig into it enough with him to give you any more information than you have already, and in fact I suspect you know more about it than I do.  I am vaguely nervous about it.  f you think a call would help with something, I'm happy to have one
<lifeless> gary_poster: I don't think we need a call
<gary_poster> I thought that he needed to land that branch in order to fix something else critical
<gary_poster> and I thought that this meant that our current state was Really Bad
<lifeless> AIUI if we roll the patch back again, then we will just be missing that mute klink
<gary_poster> But he didn't think so
<gary_poster> lifeless, I think you are right.  I think I am getting confused about a follow-on bug that a previous branch caused
<gary_poster> (a previous branch for that bug)
<lifeless> ok, so wgranthas been doing the legwork. I think either he fluffed a little bit of metadata
<lifeless> -or- qatagger is confused because of too many joined bugs
<lifeless> either way, I will get the patch backed out if it isn't
<lifeless> and that mute button can be fixed post-release-of-the-feature
<gary_poster> ok thanks lifeless.  Completely agreed that this bug is not critical to its release.
<lifeless> no worries
 * gary_poster runs away now.  Have a good night all
<lifeless> its a shame we're running into such friction. night night
<wgrant> lifeless: Huh?
<wgrant> Oh, bah, that bug was linked to 13164 too.
<wgrant> Fixed.
<wgrant> Should be green in 30s.
<lifeless> wgrant: actually, I added the bad-commit-13164 after looking at the deploy report
<lifeless> wgrant: did 13164 (the follow-on-oops) also need backing out ?
<wgrant> lifeless: No, it's unrelated.
<wgrant> Well, slightly related.
<wgrant> But not really.
<wgrant> You should be able to see from the diff that's it's safe :)
<lifeless> I'd only just started looking at stuff
<wgrant> .... conflict for 7 hours, awesome.
<wgrant> Has someone QA'd the rollback?
<lifeless> no
<wgrant> I pre-QA'd on DF, but let's see anyway.
<lifeless> when I started looking deploy-manager thought things were bad
<lifeless> and the rollback has no linked bug to qa against from the deploy-report perspective
<wgrant> It always considers a rollback to be deployable, anyway.
<lifeless> wgrant: rev 13164 is linked to bug 772609
<_mup_> Bug #772609: bug subscription mute link is not shown for membership in a team with a direct or structural subscription <bad-commit-13003> <bad-commit-13154> <qa-ok> <story-better-bug-notification> <Launchpad itself:Triaged by gmb> < https://launchpad.net/bugs/772609 >
<wgrant> lifeless: Yes. I've marked that qa-ok and removed bad-commit-13164
<lifeless> ok
<wgrant> Hmm. Is the Subscribe/Unsubscribe link meant to do nothing when not logged in?
<lifeless> I don't think we have JIT login yet
<wgrant> Yeah, but normally it's a non-AJAX link which prompts for login.
<wgrant> Bug #792502
<_mup_> Bug #792502: Bug page subscribe/unsubscribe link doesn't work for anonymous users <exploratory-testing> <qa-ok> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/792502 >
<wgrant> lifeless: It is green now.
<wgrant> So, I think the confusion has stemmed from three things: 13154 vs. 13164 is not very obvious; 13164 shouldn't have had 772609 linked to it; I didn't notice that it did.
<wgrant> Fail.
<wgrant> Anyway, sorted now.
<lifeless> yup, agreed on that
<lifeless> I thought you had missed 13164
<lifeless> (because they were similar)
<wgrant> It's still good on qastaging.
<wgrant> So I think we're fine.
<lifeless> so we need rc= and testfix to land atm right ?
<wgrant> Are we in testfix!?
 * lifeless was wagging
<lifeless> ok, so rc it is
<wgrant> Just need RC.
<wgrant> I have a conflict resolution branch.
<wgrant> For stable->db-devel.
<wgrant> Do I have your blessing?
<lifeless> aaron had one
<wgrant> Ah.
<wgrant> k
<lifeless> I've just shoved it at pqm
<wgrant> Thanks.
<lifeless> with the right headers
<wgrant> I actually prepared one last night.
<wgrant> But decided it would be entertaining to see what happened if I left it overnight.
<wgrant> Plus it was 1am so meh.
<lifeless> I wonder if telling postgresql to block %foo% searches would break too much
<wgrant> Or we could upgrade to 9.1.
<lifeless> it has indexable substring searches ?
<wgrant> IIRC.
 * wgrant googles.
<wgrant> http://www.postgresonline.com/journal/archives/212-PostgreSQL-9.1-Trigrams-teaching-LIKE-and-ILIKE-new-tricks.html
<lifeless> right
<lifeless> thats available in 8.4
<lifeless> its just an addon
<lifeless> (and needs the query to be mangled a little)
<wgrant> lifeless: DId you forget your QA tags again?
<wgrant> It doesn't seem to have landed.
<lifeless> [rc=lifeless][no-qa][r=lifeless] ...
<lifeless> is what i did
<wgrant> release-critical=
<wgrant> Not rc=
<lifeless> bwah
<wgrant> [no-qa] is also traditionally at the end, but unlike [testfix] I don't think the order matters too much.
<wgrant> Something to consider if it fails again.
<lifeless> I wonder if julian will facepalm when he gets my reply.
<poolie> hi all
<wgrant> lifeless: How is the conflict?
<lifeless> checking for cron errors
<lifeless> ah
<lifeless> abentley: your branch was private and not accessible to canonical-launchpad-branches
<lifeless> flacoste: ping; would you consider bug 794008 critical ? It's not js, rather css, but still...
<_mup_> Bug #794008: Opera displays Launchpad _without text_ <opera> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794008 >
<sinzui> wallyworld: wgrant: I just got called to pickup my daughter. I will be 20 minutes late
<wallyworld> sinzui: ack
<wgrant> sinzui: Sure.
#launchpad-dev 2011-06-08
<sinzui> wallyworld: wgrant, I am back
 * wallyworld opens mumble
<lifeless> win - 1567  OOPS-1983AS138  Bug:EntryResource
<lifeless> wgrant: guess what
<wgrant> lifeless: What?
<lifeless>  http://pastebin.com/ZpWvaAgr
<wgrant> lifeless: Heh. Yay
<lifeless> thats why list() was around official bug tags before
<lifeless> not because it wasn't a list
<lifeless> can you suggeset any improvements ?
<wgrant> lifeless: Not really :/
<lifeless> are we coming out of rc now ?
<wgrant> We can't QA on prod any more. But I guess so.
<lifeless> prod ?
<wgrant> Well, prod worked well as a QA environment yesterday... ahem.
<lifeless> *blink*
<wgrant> lifeless: It found a bug that would have really screwed us over on Thursday.
<wgrant> Because we wouldn't have been able to roll back.
<lifeless> right
<lifeless> but we don't have a branch with that rolled back and without db changes
<wgrant> No.
<lifeless> and we aren't aware of any damage in rev 13168
<lifeless> that we haven't already agreed to
<wgrant> Correct.
<lifeless> so, unless we have some more testing we want to do
<lifeless> we should un-rc
<wgrant> Indeed.
<wgrant> But we said that yesterday too.
<wgrant> And then the world melted.
<lifeless> yes
<lifeless> unknown unknowns, they suck.
<lifeless> still
<lifeless> I've very glad I have been conservative about moving the freeze time closer to the downtime
<wgrant> Yes...
<wgrant> Once we have a faster test suite and no stupid feature deadlines, we can improve.
<LPCIBot> Project windmill-db-devel build #371: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-db-devel/371/
<lifeless> allenap: are you still trying to get rabbit stable ?
<wgrant> lifeless: I don't think so.
<wgrant> StevenK might know.
<StevenK> He is not, no.
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:212 - 0:[######=_]:256
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugs:212 - 0:[######=_]:256
<StevenK> sinzui: I've picked bug 307620 off that list
<_mup_> Bug #307620: Confusing message "There is a no package named 'xserver-xorg' in Debian" when filing a bug <confusing-ui> <filebug-package> <lp-bugs> <lp-soyuz> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/307620 >
<abentley> lifeless: ah.  Hysterical raisins.
<StevenK> It's so lightly tested it's scary
<wgrant> StevenK: How do you propose to fix that?
<wgrant> lifeless, abentley: Do we know why PQM is silent about that?
<StevenK> wgrant: By doing what sinzui suggested in comment 18
<abentley> wgrant: I do not.
<wgrant> StevenK: Ah, heh.
<StevenK> wgrant: You think that's a good idea or a bad one?
<wgrant> StevenK: Mmm. We will have a proper solution to this soon.
<wgrant> StevenK: Ensemble needs it.
<wgrant> (hi lifeless)
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/fix-sprite-rebuild-rules/+merge/63791
<StevenK> But Ensemble is only concerned about SPNs
<wgrant> StevenK: It's a similar issue.
<wgrant> But true.
<StevenK> wgrant: r=me, awesome work
<wgrant> Thankyou sir.
<LPCIBot> Project windmill-devel build #179: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-devel/179/
<wgrant> Hmm. The OOPSes are creeping back up.
<lifeless> wgrant: it blows an exception through to cron
<lifeless> comment 18 ?
<wgrant> The Internet hasn't ended yet?
<wgrant> What with World IPv6 Day and all that.
<StevenK> Doesn't seem to have
<wgrant> Tomboy needs bzr integration.
<LPCIBot> Project windmill-devel build #180: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/180/
<StevenK> Can I set a distribution that I've just created to use LP as a bug tracker?
<wgrant> StevenK: As a mortal? Probably not.
<wgrant> Although +edit may work...
<wgrant> Some distro stuff is only accessible to admins.
<wgrant> Yeah, it's on +edit still.
<StevenK> wgrant: In a test case
<wgrant> StevenK: Set official_malone = True
<wgrant> Grar sampledata.
<StevenK> wgrant: Bah, setting offical_malone to True gives me the same message
<wgrant> StevenK: Which message?
<StevenK> wgrant: u"Distribution-774925 doesn't use Launchpad as its bug tracker."
<wgrant> grep may help.
<StevenK> wgrant: Spelling correctly would also help. :-(
<wgrant> StevenK: Heh.
<LPCIBot> Project windmill-devel build #181: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/181/
<jtv> Good day gentlebeings.
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, jtv | Critical bugs:212 - 0:[######=_]:256
<wgrant> Greeting jtv.
<jtv> greeting back
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, jtv | Critical bugs:214 - 0:[######=_]:256
<lifeless> (yes, its going the wrong way)
<wgrant> The graphs are quite depressing.
<lifeless> jml: I would like to catch up with you this week; my tonight is doable for me if it works for you.
 * wgrant stabs SQLObject
<lifeless> oh
<wgrant> Oh?
<lifeless> what provoked the stabbing ?
<wgrant> The old cls.select thingy doesn't seem to behave just like Storm, particularly around using SQL(..., params=...) in orderby.
<lifeless> thats a SQLObject result set, it is a little different
<wgrant> Yeah.
<wgrant> I'm notting stabbing SQLObject for being buggy and inconsistent -- merely for existing at all.
<wgrant> Er.
<wgrant> s/notting/not/
<lifeless> poolie: in canonical wikis 'Bug:1234' will link to lp automatically
<poolie> that's good to know
<poolie> i guess you realized my edit was fixing a typo, not just inserting the pad bit
<wgrant> Changing person vocab sorting is fuuuuun.
<StevenK> wgrant: The type of fun that is spelt with no n, and an extra c, k, e and d?
<wgrant> Something like that.
<StevenK> Like Help in CS is spelt with no p and an extra l
<StevenK> jtv: Are you free to review one of my branches?
<jtv> StevenK: yes, just finished the last one.
<jtv> StevenK:  the also-affects one?
<StevenK> jtv: Yes.
<StevenK> jtv: I'm in the middle of allenap's MP.
<StevenK> rvb's MP looks okay to me, but I'm comfortable about reviewing a JS-only branch
<StevenK> Er. Not comfortable
 * StevenK goes to inject tea
<lifeless> poolie: I didn't realise that,no.
<LPCIBot> Project windmill-devel build #182: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/182/
<wgrant> Tests normally run as the 'launchpad' DB user, right?
<StevenK> Yes, I think so.
<StevenK> jtv: No news is good news?
<jtv> StevenK: sorry for the delays â lots of distractions so threw in a spot of lunch as well
<lifeless> jtv: do you know anything about the storage-and-performance of arrays vs related tables in pg ?
<lifeless> jtv: I'm wondering if we should make subscriptions into an array (or duplicate them into one) on (perhaps just) private bugs
<lifeless> jtv: specifically, every time we access a private bug, we want to check the condition (private is false or has-subscription)
<jtv> lifeless: I know very little except what's obvious.  This sounds like a good idea; I've been quietly wishing for arrays in several places but not daring to speak out.
<jtv> However front-end library support is not a given.
<jtv> Generalized array parsing is an annoying little job that I decided not to tackle in libpqxx because it was coming in libpq, and AFAIK it never came.
<lifeless> ah
<lifeless> so I believe we do array processing in some (limited) places already
<jtv> Nice
<lifeless> storm etc support is a good question though
<lifeless> this may be done via 'raw' sql today.
<jtv> I did crib a bit of Storm support from somewhere and put it in the codebase IIRCâ¦ let me dig it up
<lifeless> no need, I'm not hacking on this just now
<jtv> ok
<lifeless> was curious about the performance implications more than higher level stuff
<jtv> I think you can index and search arrays for specific elements.
<jtv> But duplicating into arrays as you say would give us the best of both, at the cost of writing a bit more data.
<lifeless> we're massively read heavy
<jtv> There you go.
<lifeless> other than crazy things like bug heat, I've no issue with us writing more to read more efficiently
<lifeless> well
<lifeless> once we get the DB server disk upgrades done.
<jtv> Although (advocate o/t devil) we've also got scaling headroom for reads that we don't have for writes
<lifeless> until then we need to be juidicious about it ;)
<LPCIBot> Project windmill-db-devel build #372: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/windmill-db-devel/372/
 * lifeless makes a 800K row temp table
<lifeless> oh -wow-
<lifeless> we have a slightly brain dead privacy check for assignee
<wgrant> The new one?
<lifeless> yes
<poolie> well, https://code.launchpad.net/~mbp/launchpad/790902-mail-librarian/+merge/63816 makes it a uuid
<lifeless> goes from bugtask -> bug -> bugtask.assignee -> teamparticipation
<poolie> the lp typo test to make the initial patch seems fairly consistent at about 45m for me
<lifeless> this is accurate but fiee.
<lifeless> so, denorming 3 fields onto bugtask and drop the assignee visibility -> 2 second table scan query.
<lifeless> still an OOM slower than bugsummary
<lifeless> poolie: is that friction, familiarity, not knowing where to find things?
<poolie> StevenK, jtv, could you have a look at https://code.launchpad.net/~mbp/launchpad/790902-mail-librarian/+merge/63816 for me please?
<Peng> /1/1
<jtv> poolie: I'm just reviewing StevenK but shouldn't take long, so I can take it after that
<Peng> erk, damn. I hadn't done that all day!
<poolie> about 5m getting the environment up to date, eg updating lp-sourcedeps, make schema etc
<poolie> ah, recently i've had a run of things where there is redundant code that makes the fix more complicated
<poolie> like the "decoy" mail sending code in the ppa notification stuff
<poolie> perhaps none of them are truly typo-level fixes
<jtv> StevenK: review done â I'm recommending some changes though
<jtv> poolie: looking at yours nowâ¦ why the trans.begin() in incoming.py?
<StevenK> jtv: structured() is smart enough to not escape things that are themselves an instance of structured() -- since I wrote that bit
<poolie> jtv: rather than the other
<jtv> StevenK: ahhh!  But then where does that displayname get escaped?
<poolie> it seemed to me only incoming.py really has an opinion about when things hould be committed
<StevenK> jtv: So, there are two displayname, both are inside a structured() of some form
<wgrant> StevenK: You fail.
<wgrant> 20	+                binary_tracking = structured(
<wgrant> 21	+                    ' Launchpad does not track binary package names '
<wgrant> 22	+                    'in %s.' % distribution.displayname)
<wgrant> s/ %/,/
<jtv> poolie: I've bumped my head into this myselfâ¦ turns out a transaction is started automatically whenever we access the database while not in a transaction and not in autocommit mode.  So for the general case (and I don't know whether that includes this of course) all that begin() really does for us is increase the length of the running transaction.
<jtv> wgrant: see my review and our ongoing discussion.
<StevenK> wgrant: Sigh, so I do.
<jtv> poolie: I may be misunderstanding you though because I'm trying to carry two conversations at once.  :)
<poolie> ok
<poolie> i didn't add that line, i just moved it
<poolie> your logic makes sense
<jtv> That's a fine answer AFAIC.
<poolie> does "start if not already started" actually take much time?
<jtv> If you don't feel comfortable changing it, leaving it as-is is valid.
<jtv> No, that's really really quick.
<poolie> that's good to know for the future
<poolie> later i may try to clean up incoming.py more
<jtv> Good call keeping that separate.
<StevenK> jtv: I have another one, if you're free.
<jtv> StevenK: that wasn't enough to put you off?  :)  Sure, I'm almost done with Martin's.
<StevenK> jtv: This is one is 1: Much smaller, and 2: Needs more thought, but this is quick solution to the problem.
<jtv> StevenK: being talked at from 5 directions; will be with you shortly.
<jtv> poolie: you're done
<jtv> StevenK: waitâ¦ the vocab currently only returns descriptions, and you're changing it not to return descriptions?
<StevenK> jtv: How useful do you think it is for people to pick out a source or binary package purely on their description only?
<StevenK> It's utterly utterly pointless
<jtv> I don't think I've ever knowingly read a package description.  What's in them, in practice?
<jtv> (Just out of interest)
<StevenK> libc6's description is: Embedded GNU C Library: Shared libraries
<lifeless> hmm, 150ms stats portlets appear possible
<wgrant> StevenK: That's the summar.
<wgrant> StevenK: Not the description.
<StevenK> Oh, in which case the picker will show "Contains the standard libraries that are used by nearly all programs on"
<wgrant> Right.
<StevenK> jtv: ^
<jtv> Wow, that _does_ sound very helpful.
<wgrant> And that's one of the most useful descriptions.
<jtv> (irony there)
<wgrant>  This package contains debugging files used to investigate problems with
<wgrant> That's another example.
<jtv> StevenK: I can see how you'd want to think about the proper solution some moreâ¦ is it still worth setting a title though, or testing for it?
<jtv> You can just leave out the third arg to SimpleTerm()
<jtv> â¦and cut it out of the doctest.
<jtv> If you're going to replace it very soon with something more sensible, it makes sense to keep it.  Or maybe something in the picker requires it?  Otherwise, I'd just drop it.
<StevenK> jtv: The picker does not seem to use the term, only the title, so we need to return it twice
<lifeless> hmm, where is the stubster
<jtv> I see
<jtv> StevenK: approved
<StevenK> jtv: Thanks!
<jtv> rvba: I see you have a review waiting.  I may get to that soon.
<jtv> (There's one more in the queue before you)
<rvba> jtv: great, thanks (it's a js fix).
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv | Critical bugs:214 - 0:[######=_]:256
<jtv> rvba: that's a nasty little bug you're tackling there.  As they say, only two things are hard in software engineering: naming things and cache invalidation.
<jtv> And off-by-one errors.
<adeuring> good morning
<jtv> hi adeuring!
<jtv> need refreshment, brb
<poolie> thanks jtv
<poolie> it would be cool if lp did ipv6 day next year
<wgrant> Hahahah
<wgrant> indeed, but...
<poolie> in all your spare time :)
<wgrant> It's got nothing to do with LP :)
<jtv-afk> And then someone in China pops up and says "here's a version of IP that just uses 64-bit addresses, and we NAT it into a hole in the IPv4 space, and we already talked to the equipment manufacturers" and then all that effort is going to be wasted.  :)
<wgrant> I think there's like two significant Australian ISPs that provide native v6 :/
<poolie> i'm all right jack :)
<poolie> wgrant: meaning it's mostly a canonical-network thing?
<wgrant> poolie: Entirely.
<wgrant> Well. Unless you call the frontends LP.
<wgrant> Which I guess you almost could.
<lifeless> poolie: I thought we chatted about this on the phone
<lifeless> poolie: when I closed that bug
<poolie> oh about ipv6?
<poolie> yes, we did
<lifeless> stub: hola!
<stub> Yo
<lifeless> stub: I want to add 3 columns to bugsummary :>
<stub> Oh joy :)
<lifeless> https://bugs.launchpad.net/launchpad/+bug/793848/comments/4
<_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/793848 >
<stub> lifeless: You are quadrupling the number of rows. Not sure if the distro queries are going to survive with their tag counts.
<lifeless> stub: https://bugs.launchpad.net/launchpad/+bug/793848/comments/5
<lifeless> :P
<_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/793848 >
<lifeless> stub: its about 50ms slower on the tag counting case
<stub> lifeless: Will it be exploded by tags and milestones, or can we add CHECK (has_patch IS NULL != tag IS NULL) etc.
<mrevell> Hello devers
<lifeless> stub: its more dimensions
<lifeless> stub: so more cross products
<stub> Right. So we have to be careful, as every addition could put us over the top with this model
<lifeless> we do:
<lifeless>  select count(*) from bugsummary2;
<lifeless>  count
<lifeless> --------
<lifeless>  758533
<lifeless> \dt+ bugsummary2;
<lifeless>                        List of relations
<lifeless>   Schema   |    Name     | Type  | Owner | Size  | Description
<lifeless> -----------+-------------+-------+-------+-------+-------------
<lifeless>  pg_temp_1 | bugsummary2 | table | ro    | 54 MB |
<stub> Did you check the existing queries for distribution and distroseries use cases?
<lifeless> distribution yes, haven't checked distroseries
<stub> I think we will be ok too, this time....
<lifeless> I'd expect distribution to be worst, and its 200ms
<lifeless> stub: we could make the booleans and importance nullable and have two separate summaries - one for use in this query, one for the tags one
<lifeless> but I don't think the complexity is worth it - this seems pretty straight forward and the performance (hot) is still extremely shiny
<stub> lifeless: If we took that approach, I'd just stick them in a separate table.
<lifeless> yeah
<lifeless> so, whats the right way forward to apply this change
<stub> lifeless: The usual. db patch to add the new columns and extend all the triggers, test on staging...
<stub> lifeless: Probably one for me.
<lifeless> stub: would you?
<stub> although it will likely be cut & paste work with the existing code as a template.
<stub> sure
<lifeless> yah
<lifeless> I have a test script in that bug, which is an adapted version of the original patch (but no triggers)
<lifeless> oh btw
<lifeless> stub: were you aware that copying a template db locks the source ?
<lifeless> exclusive-locks
<lifeless> anyhow, I put a spinlock-with-random-backoff into the db layer when I found this some months ago
<stub> Your template db can't be in use, so not surprising.
<lifeless> I'm thinking of making a test-run-specific template
<stub> You get an error if there are any open connections to the template, or at least you used to.
<lifeless> so launchpad_ftest_template -> launchpad_ftest_template_$PID -> launchpad_ftest_$PID
<lifeless> do you see any problem with such a double-indirection ?
<stub> This is to avoid contention I guess? No problem except twice as many databases to clean up on a cancelled run. We should probably automatically do that somehow... might need to maintain shared state in some well known location, such as a dedicated PG database for lp test run metadata.
<stub> lifeless: Is db setup time worth optimizing yet though? Last I checked, teardown and setup was only 10-15 minutes or so of our total test run time.
<lifeless> stub: parallel testing contention...
<lifeless> stub: I have no solid figures, but I'd like to be at least theoretically interaction-free.
<stub> Yer, it will be better. Just wondering if there is other low hanging fruit that should be tackled first.
<lifeless> stub: possibly. Feel free to try parallel testing anytime you like :)
<lifeless> StevenK: which reminds me, does jenkins parallel test devel nowadays ?
<stub> There used to be some backoff time required *anyway*, because it took a short while after you closed your connection before PG had cleaned up the backend and the connection was really dead.
<stub> I think that is better now under 8.4, where it just blocks until the db is available. Might be possible to pull out the retry code (?)
<lifeless> definitely isn't possible :)
<lifeless> running 8 test threads at once, they all blew up on template copying until I added the random backoff
<stub> I mean after a separate template per thread.
<lifeless> oh right
<lifeless> up
<lifeless> uhm
<stub> I guess if it isn't needed, the retry will never happen so no need to pull it out yet.
<lifeless> if we're using the same helper to copy from l-f-t to l-f-t-$pid and l-f-t-$pid to l-f-$pid then it needs to be in for the first copy
<stub> right
<stub> mark had the idea of a background task that created fresh databases for tests to consume, but I never chased it once we saw db teardown/setup was not as significant as we thought.
<lifeless> *blink*
<lifeless> that would create IO and CPU contention
<lifeless> unless you had less test runners than the machines concurrency could support
<lifeless> grr 14492 tests
<lifeless> 14491 passed
<stub> this was singlethreaded tests, so building the db would chew io while the test was chewing cpu
<lifeless> whats the bet I failed to push
<stub> and most tests don't need new dbs, so it would be idle a lot of the time anyway.
<lifeless> yeah
<lifeless> I wouldn't pursue the idea :)
<StevenK> lifeless: What do you mean?
<lifeless> StevenK: there was a jenkins job doing parallel test runs
<StevenK> There still is
<lifeless> what branch does it run
<StevenK> http://bazaar.launchpad.net/~lifeless/launchpad/librarian
<lifeless> StevenK: can we switch that to running stable ?
<lifeless> and sorry for breaking trunk, I had not pushed :(
<StevenK> Done
<lifeless> StevenK: thanks!
<stub> lifeless: Although extending BugSummary will fix counts, I think there will still be problems elsewhere discovering bugs that have been fixed upstream. If people are interested in how many, they are also interested in a set of links. Is this search also problematic? Perhaps being able to flag a bugtask as 'fixed upstream' will make both the counts and search acceptable. And if the counts still are not acceptable, make the triggers much easier.
<lifeless> stub: I'd be fine with denormalising 'fixed_upstream' onto bugtask
<stub> garbo task or checkwatches could do that
<lifeless> stub: or a trigger could
<stub> yer.
<lifeless> stub: I did an experiment with a wider bugtask
<poolie> thanks for the kind words jtv
<lifeless> stub: it can get down to 2 seconds if we can drop the join onto bug
<stub> but checkwatches is the only thing that updates the relevant rows I think (?), so there might be simplest.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/793848/comments/1
<_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts <dba> <timeout> <Launchpad itself:Triaged by stub> < https://launchpad.net/bugs/793848 >
<jtv> poolie: you know me â you earned them.  :)
<lifeless> stub: there are 2 cases to fixed upstream, a) bugwatch, b) a product task
<lifeless> stub: b) is changed by users.
<stub> k
<poolie> lifeless: so just briefly i'm merging those two mail->librarian things into just one, that uses a uuid
<lifeless> poolie: cool, I saw. Looks good.
<lifeless> stub: comment 3 gets a 2 second query with reflexive-join to find fixed_upstream
<lifeless> stub: by denormalising duplicateof, latest_patch_uploaded and private onto bugtask
<poolie> thanks
<lifeless> stub: but we have to join to bug to get assignee
<lifeless> stub: so we're still looking at 4.5 seconds minimum
<lifeless> stub: -> 3.5 seconds too slow
<stub> Unless we use triggers to mush together bug and bugtask into one flat BugSearch table...
<stub> maybe too fat for sanity... but I guess fat for a reason.
<lifeless> stub: I doubt it would get down to the 100ms that bugsummary does
<stub> right, but useful for stuff besides counts
<lifeless> stub: the thing about the portlets is that they want multiple criteria shown at once
<stub> or is this only a problem with counts and I'm fixing a problem we don't have?
<lifeless> stub: indeed, and I'm totally happy to have a bugsearch table too, but its a different scenario
<lifeless> stub: so these counts link to searches
<lifeless> and bug search is also slow
<lifeless> but the one portlet shows 9 different searches summarised
<lifeless> so an individual search has rather more headroom than the portlet
<stub> Right, so we will need bugsummary for this sort of things in every case, unless we can make bug search super fast (10x more than we are currently aiming at)
<stub> Or something similar... we already know BugSummary isn't going to scale to too many more dimensions
<stub> But there are a finite number of possibilities that I can't recall of the top of my head.
<RawChid> Hey, from Python I try to load a pot from LP (staging). The code should work and I should have the right permission, nevertheless I get HTTP Error 401: Unauthorized  (Unknown access token). If I paste the URL in my browser I DO get the correct response. Any ideas?
<lifeless> your python code hasn't logged in via either oauth or openid
<wgrant> Or anonymously.
<RawChid> It does launchpadlib.login_with()
<RawChid> How can I check if it's done right
<RawChid> lifeless, this is the output: http://pastebin.com/kZ9wr6BL
<adeuring> jtv: could you please have a look at this MP: https://code.launchpad.net/~adeuring/launchpad/bug-735979/+merge/63832 ?
<poolie> RawChid: can you paste the code too?
<jtv> adeuring: ok, but that's the last one for today.  :)
* jtv changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:214 - 0:[######=_]:256
<adeuring> jtv: thanks!
<poolie> RawChid: one issue is that the url you're printing is on lpnet, but you say you're trying to load from staging
<poolie> i wonder if you are only authenticated on one of them?
<RawChid> poolie, I just doscovered I can load the pot from production without a problem.
<RawChid> You're right! I authenticate to staging, but the URL is production. Thanks a lot
<poolie> you're very welcome
<lifeless> adeuring: hi
<lifeless> adeuring: you know about ++profile++ right ?
<adeuring> hi lifeless
<adeuring> lifeless: argh, yeah, right!
<adeuring> thanks for the reminder
<lifeless> no probs!
<lifeless> I was debugging a CPU case the other week
<lifeless> with code like:
<lifeless> foo = list(resultset)
<lifeless> for quux in bar:
<lifeless>     if quux in foo:
<lifeless>         ....
<lifeless> that triggered O(N^2) storm object comparisons
<lifeless> and __eq__ on storm objects is -slow-
<lifeless> showed up very clearly on qastaging with ++profile++show
<jtv> adeuring: done
<adeuring> jtv: thanks!
<adeuring> jtv: thanks for the reminders of the "usual suspects" for "mysterious delays" :) Though GIL contention for more than a second sounds unlikely... or, well, it could be a kind of accumulated shorter delays
<jtv> adeuring: you'd be amazed.  I was.
<wgrant> The GIL isn't so much of a problem these days.
<wgrant> Thanks to lifeless' ruthless campaign.
<jtv> He optimized it in the interpreter?
<wgrant> He minimised multithreading.
 * adeuring wonders if we could record the time a thread waits for the GIL
<jtv> wgrant: if you're talking about *in Launchpad*, then see the context: the starting point is "I asked for non-threaded app servers because of the GIL."
<jtv> adeuring: someone-or-other (I should have bookmarked!) did really extensive research.
<jtv> The results were really stunning.
<wgrant> jtv: Sure, but that means that the GIL is not a major concern for our code any more.
<jtv> wgrant: see the context.  :)  We were talking about how _now that we have nonthreaded appservers_, we have a better base for evaluating python-side time sinks.
<wgrant> jtv: Ahhh, I see.
<wgrant> I'm not actually sure where the context is.
<wgrant> I guess a review somewhere in backscroll.
<wgrant> Anyway, sorry.
<jtv> Yes, it's a review.  Hard to follow without that, I know.  :-)
<jtv> But since you are aware of the issue, you may want to help me convince abel that there were (are, I guess) serious problems with the GIL in general.  :-)
<lifeless> jtv: wgrant: note that we're still 2-threaded
<lifeless> xmlrpc-internal is the second thread
<adeuring> jtv: well, I know about these problems in general...
<wgrant> lifeless: Yeah, but I pretend XML-RPC doesn't exit.
<wgrant> +s
<jtv> Annnnd there goes my good mood, thanks.  :-)
<lifeless> this is a tolerable compromise
<wgrant> adeuring: We had 8-second gaps sometimes, and they magically vanished once we went single-threaded.
<wgrant> :/
<jtv> adeuring: sorry, just meant to point out the part where it's really ridiculously bad sometimes.
<jtv> Wow, 8 seconds?
<wgrant> There were lots of 5-6. But I saw at least a few 8s.
<wgrant> Possibly it was bad code on both sides, I guess.
<poolie> is there anyone who specially cares for lazr.restful these days?
<wgrant> No.
<poolie> and could answer questions
<poolie> hm
<lifeless> s/specially//
<lifeless> jtv: the internal xmlrpc calls are (today) relatively low frequency; if we see more than 1-timeout-a-day its not going to be GIL (and thus we get the better analysis environment as you say)
<jtv> lifeless: yeah I figured you wouldn't have done it if it were a likely source of problems.  :)
<LPCIBot> Project windmill-devel build #183: STILL FAILING in 1 hr 19 min: https://lpci.wedontsleep.org/job/windmill-devel/183/
<poolie> lifeless: i wonder if it would be worth trying 0mq just within a machine or something
<poolie> hm
<LPCIBot> Project parallel-test build #19: STILL FAILING in 1 hr 20 min: https://lpci.wedontsleep.org/job/parallel-test/19/
<jtv> suite == series-pocket?
<jtv> (I always forget this after being out of soyuz for a while)
<wgrant> jtv: Yes.
<bigjools> jtv: yes
<jtv> Thanks
<wgrant> stub: Can we turn off notifications for parallel-test?
<wgrant> Er./
<wgrant> StevenK: ^^
<bigjools> anyone else who's built an image for ec2 get an email from Amazon saying it's been made private?
<bigjools> they don't like having our SSH keys in it
<wgrant> There are SSH keys in it? :/
<bigjools> yeah, your public key if you built it
<bigjools> which means you can log into anyone's instance
<wgrant> Oh, right, in authorized_keys?
<bigjools> they don't like that :)
<wgrant> Yeah, that is less than ideal.
<bigjools> it means if they do this with all our ec2 images then we can't use ec2  any more
<bigjools> so we'd better prod a maintenance dude to fix this asap
<wgrant> Does it affect current images too?
<wgrant> some bug about SSH keys was fixed a year or so ago.
<bigjools> they only emailed about one that I built recently-ish but I expect so
<wgrant> :/
<jtv> bigjools: what architecture do you want to show for PCJ uploads?
<bigjools> jtv: Source
<jtv> thx
<bigjools> we're not syncing binaries
<LPCIBot> Project devel build #786: FAILURE in 5 hr 39 min: https://lpci.wedontsleep.org/job/devel/786/
<jtv> allenap: this baffles.  I create a PlainPackageCopyJob, and its id is None.
<jtv> Maybe I still need to add it to the store.
<jtv> Yup, that does it
<jtv> How unspeakably frustrating.
<stub> So we could do that automatically if we created a subclass of Storm that added the object to the correct store automatically in its __init__, and used that instead.
<jtv> Sounds good.
<jtv> oh what is it _now_?  I'm still getting None ids in one place.
<stub> Its not done automatically so projects like Landscape can do sharding. Or Launchpad either, since we have sharding support even if we are no longer using it.
<wgrant> jtv: Have you flushed?
<jtv> No.  I figured that was implicit when I read the attribute!
<wgrant> No.
<StevenK> Er, so who broke devel? I had two branches fail ec2 with errors in xx-official-bug-tags.txt
<wgrant> StevenK: lifeless
<stub> Adding it to the store should flag the id column to be refreshed from the db
<wgrant> StevenK: Fix is landed already.
<stub> I thought
<jtv> That's what I expected, yes.
<jtv> Or that it would happen somewhere.
<wgrant> jtv, stub: I don't think so...
<wgrant> That could cause a very early flush.
<stub> one of these objects is not like the other. Perhaps not everything got added to the store?
<jtv> wgrant: it could, but if the alternative is merrily reading a meaningless value from the attribute..?
<stub> wgrant: It delays as late as possible - the id column is set to a magic value, and when you attempt to access a magic value the flush happens.
<wgrant> https://storm.canonical.com/Tutorial#References%20and%20subclassing
<wgrant>    1 >>> ben = store.add(Employee(u"Ben Bill"))
<wgrant>    2
<wgrant>    3 >>> print "%r, %r, %r" % (ben.id, ben.name, ben.company_id)
<wgrant>    4 None, u'Ben Bill', None
<jtv> Grrr
<wgrant> If whatever is pending is flushed to the database (implicitly or explicitly), objects will get their ids, and any references are updated as well (before being flushed!).
 * stub wonders what he is thinking of
<jtv> bigjools: will a PackageUpload with a PCJ have a Component?
<bigjools> yes
<bigjools> jtv: it'll be the same as any other source
<bigjools> jtv: with the exception that there's no changes file
<jtv> I guess we'll have to look up the SPR then.
<bigjools> jtv: so component, section will come from the job metadata
<bigjools> don't use the SPR
<jtv> When I looked (yesterday I think) the metadata did not contain component and section yet; I think there's something like an addOverride() that you'd need to call first.
<bigjools> jtv: there's an addSourceOverride and getSourceOverride on the PCJ
<bigjools> jtv: when the job creates the PU it will guarantee that an override is present
<jtv> That's it.  When I looked, we weren't calling addSourceOverride yet.
<bigjools> yeah it's not landed yet it's in my branch :)
<jtv> So I'm testing for the presence of something that's absent.
<bigjools> jtv: if you want you can merge my branch and use it as a pre-req
<bigjools> we can land both at the same time
<jtv> I suppose I'll want that then.
<jtv> BTW I'm off db-devel.
<bigjools> jtv: use devel
<jtv> Bit late!
<wgrant> not late.
<bigjools> everything you need is in devel
<wgrant> devel has all of db-devel.
<wgrant> Except for a couple of automerges.
<bigjools> jtv: lp:~julian-edwards/launchpad/copies-must-use-queue-bug-789502
<bigjools> that branch has one TODO left, which I am landing some pre-req stuff for separately
<jtv> OK, will have to look further tomorrow.
<bigjools> namely, letting IDistroSeries.createQueueItem() work without a changesfile
 * jtv is away, away!
<jtv> good night people
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilos | Critical bugs:214 - 0:[######=_]:256
<bigjools> I wish diff was clever enough to show stuff that was indented or moved
<danilos> bigjools, how about "diff -b" (though it's the opposite of what you are asking about)
<bigjools> yeah
<bigjools> kdiff3 does it nicely
<bigjools> not sure that's in the distro any more though :/
<flacoste> bigjools: bzr qdiff is your friend
<flacoste> bigjools: install qbzr
<deryck> Morning, everyone.
<bigjools> flacoste: I already had that installed, had no idea it had improved this much!  Now, can we do the same colouring on the MP diff page? :)
<bigjools> morning deryck
<flacoste> bigjools: qt plugin for mozilla ;-)
<cr3> is there a way to use launchpadlib in a cron and handle openid/oauth automatically somehow?
<bigjools> flacoste: I can't tell if you're joking!
<flacoste> lol
<flacoste> i was
<flacoste> cr3: yes, you need to run the script one interactively and save the credentials to a file
<flacoste> cr3: you can ask pitti how he does it for apport-retracer
<cr3> flacoste: cheers!
 * bigjools wonders if the derived distros work on package copy jobs has fixed all the PPA copying timeouts
<henninge> Hey deryck! I am back.
<deryck> henninge, ok, coming.
<henninge> deryck: otp
<deryck> ok
<bigjools> do we have a revno for the release yet?
<rvba> bigjools: https://dev.launchpad.net/DowntimeDeploymentSchedule
<flacoste> bigjools: we should have, PQM Is open
<bigjools> \o/
<bigjools> thought it was closed still
<flacoste> bigjools: 13168 is the one
* flacoste changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | PQM is open for business | 13168 is rolling out at 22:00UTC | On call reviewer: danilos | Critical bugs:214 - 0:[######=_]:256
<LPCIBot> Project windmill-db-devel build #373: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/373/
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | PQM is open for business | 13168 is rolling out at 22:00UTC | On call reviewer: - | Critical bugs:214 - 0:[######=_]:256
<LPCIBot> Project parallel-test build #20: STILL FAILING in 1 hr 28 min: https://lpci.wedontsleep.org/job/parallel-test/20/
<lifeless> man, this is waaay to early to be awake
<bigjools> lifeless: !
<bigjools> nobody, ever, has had a better nick than you
<sinzui> deryck_: Can you review https://code.launchpad.net/~sinzui/launchpad/webkit-yuitest-love-1/+merge/63884 so that we can discuss the future of JS testing
<deryck_> indeed
<deryck_> sinzui, looking now....
<lifeless> bigjools: IRC nicks, you gotta own them :)
<abentley> lifeless: or else pwn them using sniffed credentials :-)
<lifeless> bac: if you can qa rev 13170 we can get it deployed today
<deryck_> sinzui, I like this very much.  No concerns or comments really.  r=me.
<bac> lifeless: looking
<sinzui> deryck_: thanks. I will make my first attempt to land. I may need to tune html5browser for lucid because the dependencies are different.
<gary_poster> abentley, could you kick https://code.launchpad.net/~chromium-team/chromium-browser/channels , per https://answers.launchpad.net/launchpad/+question/160629 ?  I'm assuming this is related to bug 648075, per your comment in https://answers.launchpad.net/launchpad/+question/159857
<_mup_> Bug #648075: Automatic translations export fails intermittently <branch-scanner> <lp-translations> <oops> <qa-untestable> <Launchpad itself:Fix Committed by abentley> < https://launchpad.net/bugs/648075 >
<deryck> sinzui, Once it lands, it would be nice to move the test_yuitests files out of the windmill dir, just to make it clear this is separate...
<deryck> sinzui, but that's easy and not a huge issue either.
<LPCIBot> Project windmill-devel build #184: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/184/
<sinzui> deryck: agreed.
<abentley> gary_poster: kicked.
<gary_poster> thanks abentley!
<jcsackett> sinzui, could you look at https://code.launchpad.net/~jcsackett/launchpad/oh-oh-pick-me-pick-me-3/+merge/63885 for me?
<sinzui> okay
<LPCIBot> Project windmill-devel build #185: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/185/
<LPCIBot> Project devel build #787: STILL FAILING in 5 hr 40 min: https://lpci.wedontsleep.org/job/devel/787/
<henninge> sinzui: Hi! Do you have a minute?
<lifeless> I recall there being an existing bug asking for a strike-out style on closed bugs
<lifeless> (not bug 109113)
<_mup_> Bug #109113: closed bugs are not clearly marked in the recently filed/touched lists <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/109113 >
<lifeless> I cannot find it
<sinzui> jcsackett: I replied with a question. I hope you can reassure me that the comment is wrong
<lifeless> found it, it had become a dupe
<LPCIBot> Project windmill-devel build #186: STILL FAILING in 45 min: https://lpci.wedontsleep.org/job/windmill-devel/186/
<lifeless> sinzui: hi
<sinzui> hi lifeless
<lifeless> lib/lp/bugs/templates/bugtarget-portlet-tags-content.pt to remove memcache I just need to delete the <tal:tags content="cache:public noparam, 60 minutes"> line (and its closing line) ?
<sinzui> yes
<lifeless> like so - http://pastebin.com/UseELrCv ?
<sinzui> lifeless: exactly so
<lifeless> great
 * lifeless does it
<lifeless> (we don't need memcache for a 150ms page :)
<lifeless> not a 60 minute one anyhow
<jcsackett> sinzui: i see your comment on my MP. do you want confirmation that it's not possible period, or not possible for someone who doesn't own/moderate said private team?
<sinzui> jcsackett: register a private team, then try to set the team as the driver of /firefox
<jcsackett> sinzui: so, if i do that, as say admin, i can assign the private team. if i log in as name12 afterwards i cannot.
<jcsackett> this is, i take it, not ideal.
<sinzui> jcsackett: No admin can do this. the picker should not suggest you can
<jcsackett> ok.
<sinzui> jcsackett:  I believe is mispoke. Please read Brad's private-team-roles.txt doctest that clearly states our expectation.
<jcsackett> sinzui: ok.
<jcsackett> sinzui: so it looks like provided you can see the private team, you should be able to pick it as maintainer?
<sinzui> yes
<sinzui> This is actually good news because lifeless and I agree that you should be able to place a private team in a pubic role (and we will tell users who the team is) This check the bug assignee example to ensure nothing odd happens
<jcsackett> okay, i'm looking at that now.
<jcsackett> sinzui: it's a little weird when a product has a private team assigned. the display if you can't see the team is "Maintainer: <hidden>"
<lifeless> bigjools: package copy
<bigjools> lifeless: you're like a rabid dog :)
<lifeless> bigjools: does it count the user-ticked-boxes (e.g. sources) or expand to source + binary count and use that ?
<lifeless> bigjools: woof!
<bigjools> lifeless: I asked for source+bin but let me check.  Although ideally we need *file* count as that's the badly scaling bit
<lifeless> bigjools: so if its source + bin can I suggest a threshold of 40 ?
<bigjools> lifeless: it's set in a FF so you can suggest wth you like :)
<lifeless> :P
<lifeless> I mean, do you think that that would be too low?
<bigjools> gah, it's only # of sources
<bigjools> easy enough to fix
<lifeless> I'll leave it with you :)
<sinzui> jcsackett: That is correct for the code in the current state
<bigjools> lifeless: it was done with syncing in mind for DDs but DDs will always use PCJs now so I am going to leave this for a maintenance team
<bigjools> maybe that will still be me :)
<jcsackett> sinzui: right, didn't think it was wrong. just peculiar. :-)
<lifeless> bigjools: ok, so we can set the threshold to 0 for now to fix unembargo of linux
<bigjools> lifeless: yes but I'd like to see that tested on staging first
<lifeless> indeed!
<bigjools> :)
<bigjools> lifeless: the other thing is, most people expect the copies to complete immediately, this will be a culture shock and possibly even break api scripts
<lifeless> we can socialise it a little first
<bigjools> give it a few beers?
<lifeless> take it out to dinner!
<bigjools> fnar
<jcsackett> sinzui: bug assignee stuff is working on my branch as well.
<sinzui> okay. r=me
<sinzui> with the fix of the comment
<jcsackett> sinzui: dig.
<jcsackett> making that fix now.
<LPCIBot> Project windmill-db-devel build #374: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-db-devel/374/
<henninge> sinzui: Hi!
<henninge> sinzui: can you please review this branch?
<henninge> https://code.launchpad.net/~henninge/launchpad/bug-410864-undefined-in-picker/+merge/63911
<henninge> sinzui: It is fixing a bug in the picker widget which I heard your squad is working on, too.
<henninge> sinzui: So I hope you can check if this is colliding with any work that is going on
<sinzui> henninge: thank you very much
<lifeless> gary_poster: while you're here, is there anything you need from me / that I can do for you ?
<henninge> sinzui: but it is only a small fix.
<lifeless> bigjools: ditto^
<gary_poster> lifeless, hi, thanks for offer.  thinking...
<gary_poster> lifeless, nothing offhand, but I will keep in mind that you are here unusually early :-)
<lifeless> :)
<LPCIBot> Project parallel-test build #21: STILL FAILING in 53 min: https://lpci.wedontsleep.org/job/parallel-test/21/
<sinzui> henninge: r=me , but your branch conflicts with jcsackett's current branch that is landing
<henninge> sinzui: thanks, I'll have a look
<sinzui> henninge: jcsacket moved modules and code so it may not be obvious. You may need to paste line 8 from the diff into the new code
<LPCIBot> Project windmill-devel build #187: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/187/
<henninge> sinzui: I'll EOD now but I will look into it in the morning. Thanks for your review and comments.
<LPCIBot> Project db-devel build #621: FAILURE in 6 hr 29 min: https://lpci.wedontsleep.org/job/db-devel/621/
<bac> hi lifeless, you around and have a minute?
<lifeless> sure
<lifeless> how can I help ?
<bac> lifeless: great.  i'm trying to QA the branch for bug 788874 -- the too big email oops
<_mup_> Bug #788874: mail handling stalls when a large message is received and not deliverable <canonical-losa-lp> <qa-needstesting> <Launchpad itself:Fix Committed by bac> < https://launchpad.net/bugs/788874 >
<bac> on qastaging i've seen the log message written, and an OOPS report generated but i cannot find the outbound OOPS email to verify it got truncated as intended
<lifeless> ok
<bac> i triggered it by sending a bug message to qastaging with an 11MB attachment
<bac> and had cronscripts/process-mail and send-bug-nofications run on demand
<bac> lifeless: so, can you think of something i'm missing?
<lifeless> 2011-02-26 06:20:56 ERROR   No mail box is configured. Please see mailbox.txt for info on how to configure one.
<lifeless> the logs for qastaging are in carob:/srv/launchpad.net-logs/qastaging/asuka
<lifeless> process-mail is logging that output
<bac> right, and if you look at 16:10 from today it has the log message i referred to
<lifeless> I think this means there is a config issue of some sort in the way its being run
<lifeless> oh, thats feb
<lifeless> nvm
<lifeless> so, I thought that when process-mail generates an oops
<lifeless> it reported it to the log
<lifeless> so, if thats not happening
<bac> lifeless: this is the generated OOPS: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1985QASTAGING40
<lifeless> ah, I wasn't looking at the right point in th elog
<lifeless> *now* I'm more together
<lifeless> and you've looked in the staging mailbox?
<bac> yes
<LPCIBot> Project windmill-devel build #188: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill-devel/188/
<bac> lifeless: but the odd thing abt the staging mbox is it has the same message repeated hundreds of times, unrelated to what i'm doing
<lifeless> I think paolo is testing a script
<lifeless> there are several different bugs with duplicate messages
<lifeless> 718985, 718986, ..
<lifeless> I've cleaned those out
<lifeless> so, possibilities are:
<lifeless>  - the NDR isn't getting captured by qastaging mail rules
<lifeless>    (check the account you sent the big message from)
<lifeless>  - the patch is not sending the mail (and not causing a log message to be sent)
<lifeless>  - something else?
<lifeless> the change to delete the email before sending the NDR makes this a bit harder to check
<bac> lifeless: i was wondering if there was some other process that needed to be kicked.  but it doesn't sound like it
<lifeless> because we can't check that its been deleted as a way to see where the code got to
<lifeless> at least on prod this is definitely in-process
<gary_poster> lifeless, I was thinking about your perception that all or most of yellow was on the feature well after we switched to maintenance.  FWIW, it is true that danilo and I were the people I expected to be consumed, and we were; but it is also true that gmb and benji each spent a bit of time on the feature.
<gary_poster> benji was only about a week in my estimation; gmb is fixing generic critical bugs but one he's tried to fix repeatedly is tied into the whole feature thing.  So I think there's a good explanation for your perception--there's something behind it.
<gary_poster> If this clarification means you think it is important to reraise your concern later, I'll be happy to reiterate this (and perhaps get concrete data if you desire).
<gary_poster> (I personally think we should simply remember this for the future, and if it happens again, raise the concern again.)
<lifeless> gary_poster: thanks!
<gary_poster> np
<bac> lifeless: i'm going to try another and ask that process-mail be run with INFO level logging...it won't tell us much more but is worth trying
<lifeless> gary_poster: I will talk to flacoste about it; I'm not sure I can suggest anything /different/
<lifeless> gary_poster: estimation is hard
<gary_poster> ack
<lifeless> gary_poster: did my reply about the account-merge-feedback thing help you?
<gary_poster> lifeless, ah, sorry!  that was yesterday, and so is no longer relevant. ;-)  I forgot.  How about I bring up the question on -dev and include your reply, to see if we have any comments?  The issue is a small thing, but I expect you can see that it is still relevant in CHR-land (and a bit annoying that we ask for feedback without any particular plan on what to do about it).
<gary_poster> I generally agree that we should not follow up on each case.  I'd go further and say that we could simply say, in the email, "if you receive many of these, please contact feedback"
<gary_poster> That way, if we get a ping on feedback, it's more likely to be actionable
<gary_poster> (and if it is not actionable, let's not ask for feedback)
<lifeless> gary_poster: +1
<LPCIBot> Project windmill-devel build #189: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/189/
<bac> lifeless: i have this now:  https://pastebin.canonical.com/48325/ with INFO level debugging
<bac> lifeless: followed by the results of send-bug-notifications.py -vv at https://pastebin.canonical.com/48327/
<bac> lifeless: aha!  the mail did show up in the staging mailbox
<lifeless> \o/
<bac> and the 11MB attachment is 60KB long
<bac> whee
<gary_poster> yay!!!!
<lifeless> sweet
<lifeless> so thats qa-ok
<bac> i was afraid i was going to have to roll-back a branch i really thought was good...
<lifeless> bac: I'm glad you persevered ! I dunno what went wrong on the first test
<bac> dunno.  thanks for your help, though.
<lifeless> no probs, anytime
<persia> Good day.  I just noticed that I had a workitem from UDS to "rocketfuel setup - dev recommendation in a chroot - http://dev.launchpad.net/Running/Schroot".  Does anyone happen to know what this means?
<persia> Am I to clean up the page, with recommendations for automated means to create schroots?  Just test the procedure and file bugs?
<lifeless> both of those things sound good
<lifeless> you know what would be super awesome
<persia> What?
<lifeless> instructions to get lp dev environment going with lxc
<lifeless> with the container natty/oni and the contained thing lucid
<persia> I agree that would be super-awesome.  If I work on that, I'll spend the next three months yak shaving, take care of some of the precedents, and get distracted.
<persia> My recommendation there is to wait for LXC support to be added to libvirt and the ubuntu-security-tools stuff (both planned), and then ask someone to document how to do rocketfuel-setup therein at that point.
<lifeless> oohhh lixc libvirt sounds shiny
<persia> It's getting really close.  Some architectures don't have the necessary HW support for KVM, and qemu is *slow*.
<persia> Unless I misread my tea leaves, I suspect we should have nice automation for LXC in oneiric, making the super awesome procedural documentation something that should be able to happen sometime next Ubuntu cycle (which ought make it easier to cross-test as LP migrates to new LTS)
<lifeless> we're unlikely (due to sheer manpower constraints) to migrate to the next LTS for 6-12 months after its release
<lifeless> we'll have the dev environment move
<lifeless> but moving to the next LTS means:
<lifeless>  - moving to python2.7 on lucid
<lifeless>  - then moving LTS
<lifeless> neither are trivial
<persia> Indeed.
<lifeless> and we have no manpower to do them for their own sake; so stakeholder requests || dealing with criticals have to be sacrificed to do them ahead of 'must do them now'
<LPCIBot> Project windmill-devel build #190: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/190/
<persia> Lucid will have another ~36 months of support when oneiric+1 is released, so it's not precisely a critical transition.
#launchpad-dev 2011-06-09
<wallyworld> sinzui: @##@@@# mic acting up again :-(
<poolie> hi all
<poolie> lifeless, anyone, can you suggest how (if at all) i should qa the librarian-mail-naming change?
<LPCIBot> Project devel build #788: STILL FAILING in 6 hr 13 min: https://lpci.wedontsleep.org/job/devel/788/
<LPCIBot> Project devel build #789: STILL FAILING in 5.6 sec: https://lpci.wedontsleep.org/job/devel/789/
<poolie> i guess by sending it mail, syncing the logs, then peeking in there?
<lifeless> send mail to qas, sget someone with qas db access to check the .raw attribute and follow that back to make sure its accessible in the librarian
<poolie> by handcrafting a url from the database row?
<lifeless> more-or-less :)
<lifeless> rather more than less
<sinzui> wgrant: wallyworld, StevenK bug 1334, bug 80902
<lifeless> not 1337 ?
<gary_poster> :-P
<gary_poster> night
<sinzui> wallyworld: are you saying you fixed bug 86861
<_mup_> Bug #86861: SinglePopupWidget only works with vocabulary registered by name <infrastructure> <lp-foundations> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/86861 >
<wallyworld> sinzui: it appears so - i would have to read the bug details to be sure
<poolie> o/ wallyworld
<wallyworld> ?
<StevenK> Hm. buildbot has failed to check out devel -- I suspect since it tried during the rollout
<StevenK> Do I need to force a build to get it to start again?
<wgrant> StevenK: I already did.
<wgrant> 17 seconds before you asked, it would seem.
<StevenK> Haha
<wgrant> jelmer: Are bzr-svn imports meant to be taking ages now?
<wgrant> jelmer: Perhaps it is only the first run with the new version, but it is still slightly worrying.
<wgrant> eg. https://code.launchpad.net/~vcs-imports/ljcode/trunk and https://code.launchpad.net/~vcs-imports/wxwidgets2.6/trunk
<StevenK> 30 seconds to an hour? Orsum.
<wgrant> They do seem to eventually finish.
<wgrant> Let's see if the second run is faster.
<LPCIBot> Project windmill-devel build #191: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/191/
<wgrant> lifeless: Erk.
<wgrant> https://lp-oops.canonical.com/oops.py/?oopsid=1985CE168
<wgrant> Trigger contention on bugsummary?
<lifeless> I hope not
<wgrant> As do I, but what else?
<lifeless> so, a bug on lp is fine
<lifeless> gcc-linaro is a similar size
<lifeless> wgrant: its possible; the ubuntu row specifically will be fairly high volume, but 8 seconds -- argh
<lifeless> so, we need to make write transactions faster
<lifeless> for bugs
<lifeless> ><
<wgrant> s/argh/oh shit shit shit/'
<lifeless> I will talk with stub this evening
<wgrant> We'll have to see how bad this gets.
<lifeless> I think it will survive until then
<wgrant> But it may be pretty dire.
<lifeless> we may be have to disable the triggers which can be done pretty quickly
<lifeless> and then rollback my use of bugsummary.
<wgrant>        20 /    1  Bug:EntryResource:subscribe
<lifeless> but I hope not
<wgrant> That was in half an hour?
<lifeless> subscribing to public bugs won't lock any rows
<lifeless> (in theory)
<lifeless> but we're going on theory to blame bugsummary
<wgrant> It's a pretty reasonable theory.
<wgrant> Hm, OK, only 5 subscribe timeouts today so far.
<lifeless> yesterday - 3 /    0  Bug:EntryResource:newMessage
<wgrant> May be tolerable for now.
<lifeless> that new task
<lifeless> was on a product
<lifeless> 258
<wgrant> Yes.
<lifeless> which is gcc
<lifeless> choose-affected-product is 2.16 99th percentile historically
<wgrant> ... are there enough triggers involved here?
<lifeless> to get 8 seconds due to contention you'd need simultaneous edits of 4 gcc bugs serialised
<lifeless> the new task won't affect the ubuntu rows in bugsummary
<lifeless> now, we *may* have a problem where we don't shortcircuit no-op changes and we need to
<wgrant>     -- Grab a suitable lock before we start calculating bug summary data
<wgrant>     -- to avoid race conditions. This lock allows SELECT but blocks writes.
<wgrant>     LOCK TABLE BugSummary IN ROW EXCLUSIVE MODE;
<wgrant> bugsummary_tasks deals with a bug, not a bugtask.
<wgrant> It will be locking all tasks.
<lifeless> yes
<lifeless> I was just looking at that
<wgrant> == we are fucked
<wgrant> Pretty much
<lifeless> we're going to have higher than desirable contention on the primary row in Ubuntu
<wgrant> == 9s insert queries == aaaaaaaaa
<lifeless> so what we need to do is capture the bug rows (rather than summarise), capture after, take a diff, and then apply the diff
<wgrant> No, what we need to do is drop the triggers.
<wgrant> And fix them later.
<lifeless> this is non trivial to implement
<lifeless> we need to wait for stub, to determine the right way to disable them with slony
<lifeless> First thing, ident.ica and irc notices
<wgrant> 16:55 < stub> wgrant: That one is a 'possibly, but very problematic'. We need to drop and recreate a trigger, which is fast enough if we manage to grab a lock but grabbing the lock is problematic. And then we need to apply a db patch *to the slaves only* during the next rollout that updates the trigger.
<wgrant> So it looks like it needs a lock and drop on the master.
<lifeless> well there are two ways
<lifeless> we can redefine the trigger function
<lifeless> or we can drop the use of the trigger
<wgrant> lifeless: Do you know why it uses all the tasks?
<wgrant> I'm not really keen on reading 500 lines of PL/pgSQL...
<lifeless> wgrant:
<lifeless> its pretty simple code
<lifeless> see bugtask_maintain_bug_summary
<wgrant> Yes, but it's PL/pgSQL.
<lifeless>     IF TG_OP = 'INSERT' THEN
<lifeless>         IF TG_WHEN = 'BEFORE' THEN
<lifeless>             PERFORM unsummarise_bug(bug_row(NEW.bug));
<lifeless>         ELSE
<lifeless>             PERFORM summarise_bug(bug_row(NEW.bug));
<lifeless>         END IF;
<lifeless>         RETURN NEW;
<lifeless> wgrant: thus its clearer than python :P
<wgrant> lifeless: Yes, that says how it calls it, but not why.
<wgrant> Is it because it's lazy and just updates everything?
<wgrant> By removing the bug from everywhere and then readding it?
<lifeless> yes, remove the bug from the table, let the task be added, summarise the bug into the table
<lifeless> 13:56 < lifeless> so what we need to do is capture the bug rows (rather than summarise), capture after, take a diff, and then apply the diff
<wgrant> Looks somewhat buggy to me.
<lifeless> if you consider undue contention buggy
<wgrant> When unsummarising it uses the new rows.
<lifeless> no
<wgrant> So I don't think adding a task will actually increment...
<lifeless> this is an insert
<lifeless> it only has the NEW bug id
<wgrant> Oh, this is a before. Right.
<lifeless> http://www.postgresql.org/docs/8.4/static/trigger-datachanges.html
<wgrant> So, are we going to wake up stub or hope he arrives in a timely manner or wing it?
<lifeless> we're not winging it
<lifeless> its bad but its not disastrous
<lifeless> (its not too far off disastrous)
<wgrant> It needs exclusive locks on either side :/
<wgrant> It's only not disastrous because most people have EODed already.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #622: FIXED in 6 hr 24 min: https://lpci.wedontsleep.org/job/db-devel/622/
<lifeless> no, its not disastrous because it hasn't gone -completely- dead
<lifeless> what do you mean by exclusive locks either side ?
<wgrant> Both sides of each bugtask INSERT require an exclusive lock on BugSummary.
<wgrant> How ugly.
<lifeless> are you worried about deadlock ?
<wgrant> Possibly.
<lifeless> the lock once taken applies through to commit
<lifeless> its per-row
<wgrant>     LOCK TABLE BugSummary IN ROW EXCLUSIVE MODE;
<wgrant> That's not per-row.
<wgrant> Oh.
<lifeless> ... iz not?
<wgrant> I missed the "ROW" because the comment is slightly misleading.
<wgrant> Ahem.
<wgrant> LOCK TABLE only deals with table-level locks, and so the mode names involving ROW are all misnomers. These mode names should generally be read as indicating the intention of the user to acquire row-level locks within the locked table. Also, ROW EXCLUSIVE mode is a sharable table lock.
<wgrant> How odd.
<lifeless> yeah
<lifeless> I am refreshing this too
<wgrant> I'm still not sure how this isn't disastrous.
<lifeless> An exclusive row-level lock on a specific row is automatically acquired when the row is updated or deleted
<lifeless> its 8am for stub
<wgrant> Oh, true, it is later than I had thought.
<lifeless> I'm considering raising the default timeout to 20 seconds
<lifeless> the contention is self limiting bounded on permitted transaction time
<lifeless> if the change rate is below some N (unknown) then we won't backoff indefinitely, it will just spike up to some number of seconds at high concurrency changes
<lifeless> if the change rate is above N, it will cascade and push back past whatever timeout we set
<wgrant> Yes, that's my worry.
<lifeless> ok, the lock mode is wrong I think
<wgrant> 18 subscribe timeouts so far.
<wgrant> On single-task Ubuntu bugs, too :/
<lifeless> oh hell
<lifeless> subscribe changes the bug last-updated field doesn't it.
<lifeless> lalalalaalalaala
<wgrant> Hee hee so it does.
<wgrant> 00186-09119@SQL-launchpad-main-master INSERT INTO BugSubscription (bug, bug_notification_level, date_created, subscribed_by, person) VALUES (317370, 40, CURRENT_TIMESTAMP AT TIME ZONE 'UTC', 690731, 690731) RETURNING BugSubscription.id
<wgrant> Hmm.
<wgrant> But it's a single-task bug.
<wgrant> That shouldn't be that bad.
<lifeless> shouldn't matter:
<lifeless>     IF TG_OP = 'UPDATE' THEN
<lifeless>         IF OLD.duplicateof IS DISTINCT FROM NEW.duplicateof
<lifeless>             OR OLD.private IS DISTINCT FROM NEW.private THEN
<lifeless> and hah - where is status?
<lifeless> oh, bugsubsctiption might not be fast-pathing
<lifeless> yes, thats it
<lifeless> whats the bug #f or this ?
<wgrant> Bug #794802?
<_mup_> Bug #794802: OOPS-1986EA9 trying to add 'linux' task to a bug <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794802 >
<lifeless> no answer on either phone
<lifeless> trying phone again
<lifeless> I have a fixed bugsubscription trigger
<lifeless> which is likely to be a rather huge component.
<wgrant> It was updating even for public bugs?
<lifeless> yes
<lifeless> [un]summarise doesn't know that it could skip for subscription triggered summarisation
<wgrant> stub!
<hloeung> he's on
<StevenK> I think that is what wgrant was commenting on
<hloeung> yeah, I'm just saying....
<wgrant> lifeless: Around?
<lifeless> stub: hi
<stub> yo
<lifeless> stub: can I nab you for a moderately urgent voice call about fallot from bugsummary?
<stub> k
<lifeless> stub: (and sorry for repeatedly trying your mobile, all will become clear)
<lifeless> https://bugs.launchpad.net/launchpad/+bug/794802 and the topic in -ops
<_mup_> Bug #794802: many bug activities timing out due to contention on bugsummary <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794802 >
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-794802
<lifeless> oh, and http://www.postgresql.org/docs/8.4/static/explicit-locking.html#LOCKING-ROWS is going to be referenced
<lifeless> stub: skype or your mobile?
<lifeless> ah, skype. :)
<StevenK> lifeless: Is there going to be a test?
<lifeless> stub: http://bazaar.launchpad.net/~lifeless/launchpad/bug-794802/revision/13176
<lifeless> stuart and I are talking through it
<lifeless> the table level lock is deliberate due to edge cases
<lifeless>  we'll need to consider that in future
<lifeless> we are doing 0.19 busummary updates / second
 * StevenK suspects the topic here is out of date
<lifeless> this may be contention (5 second transactions) or it may be that we're mostly filtered through
<lifeless> we're updating the bugsubscription trigger
<lifeless> stub: http://pastebin.com/RZi7gfkq
<lifeless> ok, stuart has applied the new subscription filter
<lifeless> wgrant: whats the oops frequency looking like ?
<wgrant> lifeless: Only 5 subscription timeouts in the last hour.
<poolie> lifeless: one idea for lp people in Dublin is to to a bit of work towards turning loggerhead into a service, and then making it be hooked into the main lp web app
<lifeless> poolie: its already a service
<lifeless> poolie: I think hooking it in better and expanding its web service to be more useful to LP would be great
<poolie> well, "used as a service"
<poolie> is it already?
<poolie> do you think this would be realistic to stab at in a one week sprint?
<lifeless> its json api is used to show diffs / revs when yo uclick on expanders
<lifeless> poolie: I wouldn't want the lp app servers doing callouts to loggerhead today:
<lifeless>  - we are not in position to parallelise
<lifeless>  - without parallelism it will make the appservers tiiiime out
<lifeless> wgrant: what was the last one ?
<lifeless> poolie:  and loggerhead isn't consistently fast yet, and we don't have reliable timeout reports for it yet
<poolie> ok, so the only path would be to have the browser pull data from it directly?
<lifeless> right, which it does now but from a different domain
<lifeless> so calculating the right url to pass and putting that in the lp pages, and making those urls available under the main hostnames to avoid SSL - thats tractable
<wgrant> lifeless: 2011-06-09T03:01:31.659696+00:00
<lifeless> wgrant: thanks
<poolie> "under the main hostname" meaning having apach proxy them or something?
<cody-somerville> Why is the Ubuntu branches celebrity being removed?
<wgrant> cody-somerville: Ubuntu's owners and uploaders are fulfilling the role instead.
<cody-somerville> LP #524173 - There is a need for a 'bot' to have write access to the branches but not upload permissions. Would it make sense to create a 'bot account' and make that a celebrity?
<wgrant> There will be no more celebrities.
<poolie> so iow if we did this, we would model it by having an acl-type slot on the ubuntu distribution for "people who can write to branches but not upload"?
<cody-somerville> So whats the recommended way for a process like package-import to get elevated privileges?
<wgrant> cody-somerville: Give it upload privs, I suppose.
<wgrant> cody-somerville: It will effectively have them anyway.
<poolie> that was francis's approach
<cody-somerville> wgrant, How?
<wgrant> cody-somerville: How what?
<cody-somerville> wgrant, That is, how will it effectively have upload permissions if it has write permissions to the branch?
<wgrant> cody-somerville: BFBIP
<wgrant> If you can alter the branch, you can compromise it.
<wgrant> The next person to use it will get your exploit into the primary archive.
<lifeless> stub: https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content
<lifeless> stub: https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content
<cody-somerville> wgrant, lots of teams in Debian maintain packages in branches. Folks can have write privs to the branch but not upload. Its a legitimate use case by its self. The fact that the upload could potentially not review changes others have made before uploading does not mean having write privs is the same as having upload privs.
<wgrant> cody-somerville: Launchpad official branches operate under the rule that upload permissions == edit permissions
<wgrant> To change that would be a redesign.
<poolie> i'll propose this on the tb
<lifeless> ok, we're rolling forward
<StevenK> poolie: To the TB, during their meeting, or on the TB's mailing list?
<poolie> i think moving from james to a role account would be a step forward
<poolie> on the list
<lifeless> poolie: yes, apache rewrite rules ftw
<cody-somerville> I imagine The Developer Membership Board is the appropriate body to approach as the TB has delegated controlling upload permissions to that body.
<StevenK> However, the TB has not delegated the branch permission, so I think it should go before the TB, not the DMB.
<lifeless> poolie: what are you proposing to the tb ?
<persia> Well, LP is kinda special: if LP decides to implement an internal role uploader, it doesn't really match the Ubuntu processes.
<wgrant> persia: package-import isn't part of LP.
<poolie> to change the package importer from impersonating james to having its own account, per bug 524173
<_mup_> Bug #524173: package-import uses james_w credentials <Ubuntu Distributed Development:Triaged by mbp> < https://launchpad.net/bugs/524173 >
<persia> LP has more opportunity to be malicious with Ubuntu contents than any uploader.
<persia> wgrant, Should it be?
<cody-somerville> persia, +1
<wgrant> persia: Depends on your definition of "part of LP".
<lifeless> so, I would love to participate in this
<lifeless> but I have a critical regression to fix.
<lifeless> This has been discussed with at least various TB members already
<poolie> you can reply to my mail or on the bug
<poolie> it will not be settled today
<persia> I'd consider a role account running services in a LOSA-controlled environment to be "part of LP" for the purposes of this discussion.
<poolie> cjw at least has commented there
<poolie> i just want to get it unblocked
<lifeless> at the moment you have access to operate as james_w
<lifeless> so you can be as malicious as you like
<poolie> so, generally, lp will perhaps move into less-trusted interacting services
<lifeless> a dedicated account will mitigate that
<lifeless> by making it clear who uploaded etc
<lifeless> TB can easily have a bot that watches this account and makes sure it has no gpg key :)
<cody-somerville> What about the principle of least privilege?
<persia> cody-somerville, See comment #8 on the bug: there isn't really any semantic difference between commit-to-branch and upload.
<poolie> it's a good principle
<lifeless> within the year push to the branch will do the build
<poolie> this moves us closer towards it
<lifeless> so the principle will say 'this is the least privilege'
<lifeless> (modulo various details about how BFBIP all works)
<cody-somerville> if the role account is a member of Ubuntu Core Developers team then it still has tons of permissions it does not legitimately require
<StevenK> Agreed.
<poolie> so,
<lifeless> we can always add a dedicated role in future if desired... but note that *right now* its running as jamesw
<persia> cody-somerville, Easier is to have the role account just have upload to everything, rather than making it a core-dev.
<lifeless> so it has those permissions,.
<lifeless> Moving to a dedicated losa administered account is an improvement.
<lifeless> you're welcome to argue that more improvement is needed. But that is a separate discussion.
<lifeless> nothing is *regressing*
<lifeless> (and I am sure that me, poolie, wgrant etc are all open to such an argument)
<poolie> i completely agree
<poolie> this is a step forward and not a step back
<persia> And there are more steps, but they need to be thought about more, and not having thought about them yet shouldn't block this.
<cody-somerville> Who would have access to the account?
<poolie> canonical staff who maintain the importer
<lifeless> (that is, the people that currently have access to james_w's account)
<cody-somerville> If you say only LOSAs then I'd be alot happier about this
<cody-somerville> (as they can do anything they want already)
<poolie> right
<poolie> that is not true at the moment but it will become so
<lifeless> cody-somerville: there is an RT to move it to losa-only.
<poolie> there is an RT asking for it in the queeu
<wgrant> Eventually it should only be the LOSAs, yes.
<lifeless> cody-somerville: its also in-progress.
<wgrant> And the role account should only have ArchivePermissions for the primary archive, not any others.
<poolie> http://pastebin.ubuntu.com/622270/
<poolie> ^ draft email; comments welcome
<persia> poolie, Enough of TB has read access to RT that it may be worth mentioning the ticket number.
<hloeung> ^ great, more mail to ignore ;-)
<poolie> :)
<poolie> ok
<poolie> good idea persia
<persia> poolie, Also, it's probably worth phrasing the request differently.  The TB acts as a deliberative body, but is intentionally not expected to be a bottleneck.  The expectation is that people do stuff, and the TB provides observance and guidance.
<poolie> so "i plan to do this next week"?
<persia> I'd suggest either announcing to the TB that LP plans to do this, and asking for feedback, *OR* structuring it as a request for the TB to grant the appropriate permissions to the robot account.
<persia> In either case, I recommend cc: bryceh as the representative stakeholder
<poolie> ok, ka-thunk
<LPCIBot> Project devel build #790: STILL FAILING in 5 hr 56 min: https://lpci.wedontsleep.org/job/devel/790/
<StevenK> Hmmmm
<lifeless> and it passes tests. --woot--
<lifeless> maybe not all :P
<lifeless> wgrant: lp:~lifeless/launchpad/bug-794802
<lifeless> if you're interested
<LPCIBot> Project windmill-devel build #192: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/192/
<lifeless> wgrant: AWOL for ~ 40; if stub turns up point him at his email.
<wgrant> lifeless: Sure.
<adeuring> good morning
<wgrant> lifeless: FWIW we're still seeing some +choose-affected-product timeouts, but no subscribe ones.
<wgrant> And lots of BugTask:EntryResource timeouts now :/
<wgrant> Mostly status changes on Ubuntu bugs.
<wgrant> Particularly linux.
<wgrant> Almost entirely on a single bug. Perhaps lots of retries.
<wgrant> Tag changes taking 5s...
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:214 - 0:[######=_]:256
<LPCIBot> Project windmill-devel build #193: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/193/
<lifeless> wgrant: yeah
<lifeless> (back)
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:209 - 0:[######=_]:256
<lifeless> \o/
<spiv> Yay!
<wgrant> Still well above where we were three weeks ago :(
<lifeless> need to stop adding bugs
<wgrant> We've only had like one new critical today.
<lifeless> 3 ?
<lifeless> bugsummary
<wgrant> Did I miss some?
<lifeless> francis escalated the accessibility-in-bugtask bug
<lifeless> and I think there was another
<wgrant> :(
<lifeless> certainly 2
<lifeless> another day w/no microservice :(
<lifeless> wgrant: thanks
<wgrant> lifeless: Huh?
<lifeless> closing off the bugs
<wgrant> Oh, right.
<wgrant> I left the subs ones open for Yellow to close or not.
<lifeless> https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content is still fast.
<lifeless> I'm having to pinch myself :)
<wgrant> Yes, but it made everything else slow :(
<lifeless> gotta pick your battles
<lifeless> we should chang that to json
<wgrant> We should change a lot of things to JSON.
<lifeless> yes
<lifeless> this is one of them
<mrevell> Morning
 * jelmer waves
<bigjools> morning mrevell, jelmer
<jelmer> hi bigjools
<jelmer> wgrant: it's odd, some bzr-svn imports seem a lot slower; locally I don't see that effect though
<wgrant> jelmer: Some got fast again.
<wgrant> After a few tries.
<wgrant> I retried one 4 times.
<jelmer> wgrant: which one?
<wgrant> First was 40 minutes, second 25ish, third 10, fourth 1.
<wgrant> I can't remember... chromium crashed.
<wgrant> Let me see if I can find it in history.
<lifeless> ahh alpha software
<jelmer> wgrant: it looks to me like it's all bzr-svn imports that are affected - have you seen any bzr-git or bzr-hg imports becoming slower?
<wgrant> jelmer: https://code.launchpad.net/~vcs-imports/flylinkdc/trunk
<wgrant> jelmer: No, only bzr-svn.
<wgrant> jelmer: ANd only bzr-svn has been eating swap.
<wgrant> AFAIK
<jelmer> ah, so it's a memory thing?
<wgrant> Well, pear was swapping heavily.
<wgrant> One import eating 40% of the RAM.
<lifeless> \o/
<wgrant> But it had vanished before the next ps.
<wgrant> So we don't know which it is.
<lifeless> I thought we had ulimit on it
<jelmer> even with some swapping, it seems like there shouldn't be a 10 second vs 1 hour difference
<jelmer> wgrant: how did you get to the ps output, losa interaction?
<wgrant> jelmer: Yes, LOSA.
<jelmer> wgrant: ahh, I think it's got to do with the fix for bug 309682
<_mup_> Bug #309682: tags are copied but their revisions may not be <gnome> <udd> <Bazaar:Fix Released by spiv> <Bazaar Git Plugin:Fix Released by jelmer> <Bazaar Hg Plugin:Triaged by jelmer> <Bazaar Subversion Plugin:Fix Committed by jelmer> < https://launchpad.net/bugs/309682 >
<wgrant> jelmer: It looks rather like that, yes.
<jelmer> the fact that they become increasingly faster (rather than just being slow once) is really weird though
<wgrant> jelmer: It's not going to try to pull all the revs mentioned by tags in the whole repo, is it? :)
<wgrant> Yes...
<jelmer> wgrant, I'm wondering if we should make that an optional thing, there are situations in which it's not correct and causes a lot of extra data
<wgrant> jelmer: Possibly. Anyway, it should settle down eventually, I guess.
<LPCIBot> Project windmill-devel build #194: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-devel/194/
<bigjools> is there any reason why a "bzr pull" would update loads of files and then finish with "bzr: ERROR: [Errno 13] Permission denied" ?
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #791: FIXED in 5 hr 25 min: https://lpci.wedontsleep.org/job/devel/791/
<LPCIBot> Project parallel-test build #22: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/22/
 * henninge lunches
<LPCIBot> Project windmill-db-devel build #375: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/windmill-db-devel/375/
<StevenK> bigjools: O hai. You haz many lots QA to do.
<bigjools> me personally?
<StevenK> bigjools: Yes, four revisions are marked as assigned to you on the deployment report.
<bigjools> StevenK: and they were all submitted with --no-qa,
<StevenK> I see.
<StevenK> rvba: O hai. You haz many lots QA to do.
<rvba> StevenK: indeed.
<rvba> StevenK: please don't forget to take a look at the multiple parents init stuff when you get a chance :)
<LPCIBot> Project windmill-devel build #195: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/195/
<jml> lifeless: it might have to be next week. sorry.
<deryck> Morning, all.
<jcsackett> morning folks.
<bigjools> morning
<deryck> Morning, jcsackett
<jcsackett> heya henninge: did the branch you merged my stuff into land?
<jml> jelmer: hi
<huwshimi> jml: Hey
<jml> huwshimi: hello
<jml> huwshimi: let's have a call.
<huwshimi> jml: Sure
<jelmer> jml: hi
<jml> jelmer: was going to talk to you about BFBIP, but need to talk to huwshimi first.
<jelmer> jml: ah, ok. you know where to find me :)
<henninge> jcsackett: yes, it did, your branches say so, too. ;-)
<henninge> jcsackett: it's on the next buildbot ride
<jcsackett> henninge: cool. do you know if qa-tagger will be able to keep track of this, or should i keep an eye on your branch to qa my related bugs?
<henninge> jcsackett: I linked my branch to bug 787595, too, so it should update it.
<_mup_> Bug #787595: person picker could have a link to choose myself when I am a valid choice <disclosure> <person-picker> <qa-untestable> <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/787595 >
<jcsackett> ah, cool. thanks, henninge.
<henninge> np
<jcsackett> and sorry for the confusion regarding my branch names.
<LPCIBot> Project windmill-devel build #196: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-devel/196/
<jcsackett> deryck: in YUI widgets the "initializer" method is the right place to parse and deal with passed in config stuff, right?
<deryck> jcsackett, yup
<jcsackett> cool. wanted to make sure i wasn't embarking on something goofy (as has so always been the case). :-P
<deryck> jcsackett, if you need to, of course.  the widget infrastructure does all of the default values and changing values based on passed in config for you.
<deryck> jcsackett, but if you need something based on the value of two config options, for example, then the initializer would be the place to do that.
<jcsackett> deryck: yeah, this is dealing with non-default config options.
<deryck> right
<deryck> yup, you're on the right track
<jcsackett> cool. thanks, deryck.
<deryck> np
<sinzui> henninge: test_picker_displays_empty_list does not pass in devel
<sinzui> I am debugging this now.
<henninge> sinzui: oh
<henninge> sinzui: what do you mean?
<sinzui> The test fails in my browser and in the YUI test layer I just fixed
<henninge> ah
<henninge> strange, it passed for me
<sinzui> We expect an empty string, but get undefined
<sinzui> henninge: does it still pass in devel for you? I think this may be a merge compliation
<henninge> let me try
<sinzui> henninge: I tested with fireforx and chromium
<sinzui> The test runner will use webkit
<henninge> sinzui: all tests pass
<sinzui> how many tests passed?
<sinzui> I have 9 pass and 1 fail
<henninge> 10 pass
<sinzui> have you pulled devel in that last hour?
<henninge> sinzui: what does your line 22 to 25 of lib/lp/app/javascript/widgets.js look like?
<henninge> I just pulled right now
<sinzui> henninge: It looked like the line in the diff
<sinzui>         if (data.title === undefined) {
<sinzui>             // Display an empty element if data is empty.
<sinzui>             return li_title;
<henninge> ok
<sinzui>         }
<henninge> that should pass
<henninge> the failure you described looked like those lines were missing.
<LPCIBot> Project parallel-test build #23: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/23/
<henninge> sinzui: 10 pass, both in firefox and chromium.
<sinzui> I can see the for the innerHTML for yui3-picker-results, yet the innerTEXT is undefined
<sinzui> oh.
<sinzui> maybe *I* need to rebuild
<henninge> yes, this must be something to do with your local setup
<sinzui> deryck: I see that tests spend more than 1/5 of the time setting individual YUI test layers for each app. Do we really want a BugsYUITestLayer if we can run all the tests in 2 minutes
<deryck> sinzui, no, we don't need app specific yui layers.
<deryck> app-specific anything is old school anyway ;)
<sinzui> I will remove them.
<deryck> especially with yui tests it makes less sense anyway.... say I change the picker.... I need to know all uses of the picker are safe, not just one app's version.
<sinzui> henninge: I think the merge is the issue. Your block of code is never entered. I need to review the test setup I think.
<LPCIBot> Project windmill-db-devel build #376: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/376/
<LPCIBot> Project db-devel build #623: FAILURE in 5 hr 57 min: https://lpci.wedontsleep.org/job/db-devel/623/
<jml> sorry it's been such a brief day folks, but I have to go. back tomorrow for Action Friday.
<LPCIBot> Project windmill-devel build #197: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/197/
<sinzui> I need help with an ec2 image. I am following the steps in https://dev.launchpad.net/EC2Test/Image
<sinzui> I have uploaded the image and I see it in S3. I do not know how to make it public
<bigjools> sinzui: is that a new thing?  I don't remember having to do that
<sinzui> I added my name to lib/devscripts/ec2test/account.py, but I do not see my ami listed. I assumed maybe that was because it is private.
<bigjools> maybe
<bigjools> you probably need the AWS console
<bigjools> right click the AMI and select "edit permissions"
<bigjools> I have a public/private radio selection there
<sinzui> bigjools: I do not see that. I see a bucket in s3 with with image name, but it does not say it is an ami
<bigjools> sinzui: click on "AMIs" on the left
<sinzui> My browser says there is no "AMI" on the page :(
<bigjools> https://console.aws.amazon.com/ec2/home?region=us-east-1#s=Images
<sinzui> bigjools: thank you. aws started in s3, not ec2
<bigjools> heh
<bigjools> it's a wall of jargon on that page
<gary_poster> hi abentley.  Without having to do any research, can you confirm that we do not support importing private github branches?  If so, please do.
<abentley> gary_poster: I can't be 100% sure without research, but I think it's very unlikely.
<gary_poster> ok, thanks abentley
<lifeless> morning y'all
<cody-somerville> There seems like there used to always be an on call reviewer but now every time I've checked the topic its just '-'. : (
<lifeless> what do you need reviewed?
<cody-somerville> https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-2/+merge/63950
<cody-somerville> lifeless, I notice that the associated bug (LP #724740) is listing the superseded MP instead of the new one. Is there something that needs to be done to update that or is that a bug?
<_mup_> Bug #724740: setting a ppa private cannot be done over the API <api> <oem-services> <ppa> <Launchpad itself:In Progress by timrchavez> < https://launchpad.net/bugs/724740 >
<lifeless> cody-somerville: file a bug, include a screenshot and the url to the old mp and the new mp
<lifeless> that should get reviewed today, unf mid-crisis still
<cody-somerville> and of course for some reason it now shows the new mp, lol. page must have been cached in my browser.
<jcsackett> cody-somerville: I am OCR, it would seem my morning topic change didn't take.
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs:209 - 0:[######=_]:256
<jcsackett> cody-somerville: i'm looking at it now, and sorry about the topic confusion.
<cody-somerville> jcsackett, No problem! :) Kudos!
<lifeless> k: special?
<nigelb> Argh.
<nigelb> Just found a bug in LP.
<nigelb> Well, possible bug.
<lifeless> no, really?
<nigelb> lifeless: I headdesked for an hour trying to figure this out :-)
<nigelb> https://launchpad.net/~ubuntumembers/+members
<nigelb> There are 521 direct members of the "Ubuntu Members" team, and 695 people
<nigelb> the 695 is not entirely true.
<nigelb> So, if you have 10 teams
<lifeless> teams are members too
<nigelb> if counts the unique members in those 10 teams
<nigelb> lifeless: ah.
<lifeless> this could be presented better
<nigelb> Yes
<lifeless> and a case can be made that a team shouldn't count as a member
<nigelb> exactly
<nigelb> I was trying to figure out why my script wasn't catching all the members
<lifeless> however our db doesn't know the difference today in teamparticipation
<lifeless> so its not cheap to do differently
<nigelb> So, doing this would be expensive.
<lifeless> on your script shouldn't be iterating members
<nigelb> Right.
<lifeless> it should be interating participants
<LPCIBot> Project parallel-test build #24: STILL FAILING in 1 hr 15 min: https://lpci.wedontsleep.org/job/parallel-test/24/
<nigelb> members was my use case. I wanted to get a list of all the members including sub teams.
<nigelb> I can just exclude the teams out of it and I could get a clean list.
<nigelb> or does participants do that for me?
<lifeless> https://launchpad.net/api/1.0/~launchpad/participants
<lifeless> [yes]
<lifeless> hi stub
<SpamapS> still fighting the perf regression?
<stub> yo
<lifeless> SpamapS: yeah
<lifeless> SpamapS: did you see atimeout ?
<SpamapS> It renders the sru-accept script we use in ubuntu-archive-tools unusable
<lifeless> what does the script do
<nigelb> lifeless: "members = team.participants" heh, I was already using participants :)
<lifeless> nigelb: then that is recursing into subteams for you
<nigelb> yup \m/
<lifeless> nigelb: transitive closure, but I think it excludes teams. IMBW, check the Person interface and source ;)
<lifeless> stub: I have some feedback on the -journal case
<SpamapS> lifeless: looks through the tasks of given bug #'s for the given package name/series, and marks it to Fix Committed, then tags the bug verification-needed and comments
<lifeless> stub: and am looking into the test failure
<nigelb> lifeless: Thanks!
<stub> cool. It didn't seem obvious, but looks legit.
<SpamapS> lifeless: I'm not sure which operation is timing out
<stub> Only seems the one failure though, but I couldn't see what is different between the bugtag case and others
<lifeless> stub: is it safe to chang the python stuff around person merge without adding the references metadata to the db ?
<stub> lifeless: yes, that is fine
<SpamapS> lifeless: but I have the HTML of the failure
<lifeless> stub: oh, bugsummary.txt is blowing up for me just now
<lifeless> SpamapS: got the oops ?
<stub> lifeless: I think it is one failure (off by one, as if a bugtag didn't get created), and fallout from that
<SpamapS> OOPS-1986AO121
<lifeless> stub: I'm concerned that we're going to journal a complete-remove and complete-add to the journal even when nothing interesting changed - 50% of the rows or more could be noise
<stub> lifeless: My assumption is that just spitting inserts to the journal is faster than trying to do updates and avoids the locking issues.
<SpamapS> and OOPS-1986DY97
<lifeless> stub: the temp journal won't have locking issues
<stub> lifeless: Using the temp file seems a good idea though
<stub> temp table I mean
<lifeless> stub: ok, I have a patch that does that, let me commit and push for your hilarity
<SpamapS> lifeless: only look into it if it helps with the solution. If not, I can manually accept bugs until the problem is solved. :)
<stub> btw. there are no mixed case tablenames - FooBar is cast to lowercase. "FooBar" is a mixed case table name.
 * lifeless headdesks
<lifeless> stub: lp:~lifeless/launchpad/bug-794802
<lifeless> SpamapS: it hasn't synced yet; did you get a backtrace in the html ?
<lifeless> stub: there are two reasons I can see slow queries - contention, or we're just adding too much work
<lifeless> stub: we're kindof betting its contention
<stub> the queries I've checked are only attempting to update a handful of rows, so I'm betting on contention
<lifeless> stub: argh, *really* not awake
<SpamapS> lifeless: no sorry just the oops number.
<lifeless> stub: yeah, if its slow on the db, its contention
<lifeless> (now that the seq scans are gone)
<lifeless> stub: they are, aren't they?
<lifeless> stub: as in the slow occurences you see when explained show index use ?
<stub> My scenario is some dude triaging a big and clicking three of four things - change status, target, add a comment. Each is a separate ajax request being launched around the same time and wanting to lock the same rows in BugSummary.
<benji> jcsackett: thanks for the review
<lifeless> stub: so status, target would serialise normally
<jcsackett> benji: you're welcome.
<lifeless> stub: (same row in bug)
<stub> Yes. We have fixed index use. Queries are fast, except when they are real slow. Looking at the plans they seem fine.
<lifeless> stub: tag + status would serialise on bugsummary
<lifeless> stub: as would tag + target
<lifeless> commenting won't change the aggregates so won't lock anything
<stub> So if one takes 4 seconds, the other is blocked for 4 seconds before it gets to issue its query.
<lifeless> well, before the flush gets the row its trying to update
<stub> And this blocking is happening inside the trigger, so counts as SQL evaluation time.
<lifeless> yeah
<lifeless> so this failure
<lifeless> we're looking for tag is null
<stub> Best theory I've come up with anyway :)
<lifeless> (line 136 of the doctest)
<lifeless> 3 new bugs should result in an increase of 3 in the tag=null rows
<lifeless> but we're seeing a total of three
<lifeless> if I comment out line 128 I see the same thing
<stub> So given how similar the current branch is to the previous, wtf is is failing now and not before?
<lifeless> I'm trying with the rollup() call commented out
<stub> I guess it is in the combinedbug view... maybe rollup
<lifeless> which results in the right raw data
<lifeless> so the journal is capturing everything
<lifeless> the rollup appears to be discarding a row
<lifeless> I'm going to toss a few sample bugs into launchpad_dev and look at the rollup by hand
<lifeless> yeah its messed up I think
<lifeless>  id | sum | product | productseries | distribution | distroseries | sourcepackagename | viewed_by | tag | status | milestone
<lifeless> ----+-----+---------+---------------+--------------+--------------+-------------------+-----------+-----+--------+-----------
<lifeless>     |   2 |      21 |               |              |              |                   |           |     |     10 |
<lifeless>     |   1 |      21 |               |              |              |                   |           | moo |     10 |
<lifeless> (2 rows)
<lifeless> launchpad_dev=# select * from bugsummaryjournal;
<lifeless>  id | count | product | productseries | distribution | distroseries | sourcepackagename | viewed_by | tag | status | milestone
<lifeless> ----+-------+---------+---------------+--------------+--------------+-------------------+-----------+-----+--------+-----------
<lifeless>   1 |     1 |      21 |               |              |              |                   |           |     |     10 |
<lifeless>   2 |     1 |      21 |               |              |              |                   |           |     |     10 |
<lifeless>   3 |    -1 |      21 |               |              |              |                   |           |     |     10 |
<lifeless>   4 |     1 |      21 |               |              |              |                   |           | moo |     10 |
<lifeless>   5 |     1 |      21 |               |              |              |                   |           |     |     10 |
<lifeless> (5 rows)
<lifeless> no, actually, that sums to 3
<lifeless> and we get three
<lifeless> though adding the tag should really have only journalld one row
<lifeless> but the rollout output was
<lifeless>  65 |     1 |      21 |               |              |              |                   |           |     |     10 |
<lifeless>  66 |     1 |      21 |               |              |              |                   |           | moo |     10 |
<lifeless> which is clearly wrong
<stub> because we expand the tag to the null and the tag - two rows.
<lifeless> stub: no
<lifeless> because _dec and _inc only update by 1
<lifeless> they don't use the vector
<lifeless> stub: UPDATE BugSummary SET count = count - 1
<lifeless> easy fix
<stub> Yer
<stub> I see. count might be 2 or -2, but we dec or inc by one only
<stub> 'Cause I was a smart arse and wanted to minimize updates :)
<stub> You fixing on your branch?
<lifeless> yes
<lifeless> test running now
<lifeless> worked for the doctest
<lifeless> pushing now
<lifeless> we're running on the temp journal atm right ?
<lifeless> stub: :!bzr push
<lifeless> Using saved push location: lp:~lifeless/launchpad/bug-794802
<lifeless> Pushed up to revision 13189.
<stub> Nothing in test_bugsummary is invoking rollup, so all those tests passed.
<lifeless> thank mercy for doctests :P
<stub> lifeless: Yes. The first MP is what is running right now, with patches -1 and -2
<lifeless> so I think we're busy corrupting by off-by-ones just now
<lifeless> anyhow, my branch is pushed
<stub> So in patch-2208-75-0.sql we should delete * from bugsummary and reinitialize using the SQL from 63-0
<lifeless> http://bazaar.launchpad.net/~lifeless/launchpad/bug-794802/revision/13188 and http://bazaar.launchpad.net/~lifeless/launchpad/bug-794802/revision/13189 are my incremental changes over your code
<lifeless> stub: nah - we've got those new dimensions to add
<lifeless> stub: do the reinit in that patch
<stub> lifeless: The bug is in rollup, and that isn't what is running atm. So data should be fine.
<lifeless> stub: no, the bug was in _inc and _dec, which the temp journal code uses
<stub> oh... your journal has the same problem
<stub> yer.
<lifeless> so how should we proceed; its late for you I know.
<stub> lifeless: ahh... but is there anyway to get anything but -1 or +1 in the count atm?
<lifeless> stub: there is
<lifeless> distro bug tasks and bugs with tags
<lifeless> erm, scratch tags.
<lifeless> distro source package bug tasks
<stub> invoking multiple triggers in one transaction
<lifeless> actually no I think its fine
<lifeless> because each _row_ changed flushes separately to the table
<lifeless> that that had damn well better be unary or we are so screwed.
<stub> yes, I think that is right.
<stub> which is why the doctest used to pass with the same bug
<lifeless> yer
<stub> lifeless: So count is positive when we call _dec? Yay.
<lifeless> *blink*
<stub> +    UPDATE BugSummary SET count = count - $1.count
<lifeless> can't be
<lifeless> thats a thinko - and uncovered by tests
<lifeless> we only call _dec from rollup now
<stub> lifeless: Might want to invoke rollout in the test_bugsummary helper just after the flush and invalidate, like the doctest.
<stub> it has much more coverage
<lifeless> ok
<lifeless> perhaps we should run it twice
<lifeless> once without and once with
<lifeless> that can wait
<stub> I'm more interested in seeing if there are other lurking edge cases
<lifeless> running all the tests with the rollup call added and the buggy _dec
<lifeless> I want to see if we have coverage
<stub> So in theory, something should fail now or the laws of mathematics are being warped.
<lifeless> non euclidean geometry FTW
<lifeless> SpamapS: your oops is
<lifeless> SELECT  bug_summary_dec( $1 )"\\nPL/pgSQL function "bug_summary_flush_temp_journal" line 9 at PERFORM\\nSQL statement "SELECT  bug_summary_flush_temp_journal
<lifeless> SpamapS: e.g. the contention we're working on
<lifeless> yeah, we get fail
<lifeless> stub: all tests pass with that patch
<stub> woot
<stub> Push again?
<lifeless> 13191 available at your local bzr server.
<lifeless> stub: I'm going to fry up some brekky
<stub> k
<lifeless> stub: while you review
<stub> Yup
<stub> Don't worry about me, I'm already stuff full of dead pig.
<lifeless> yum
<lifeless> stub: so, this safe to progress?
<lifeless> the tag portlets will freeze until we get the new view code deployed
<lifeless> but thats fine IMO
<stub> You cook too fast
<lifeless> they are for most projects pretty static
<lifeless> <- optimiser
<stub> I thought views on this were still behind a feature flag?
<lifeless> no
<flacoste> bac: now that we have IServiceUsage, shouldn't we kill ILaunchpadUsage?
<lifeless> stub: but there is only one using it
<lifeless> stub: https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content
<bac> flacoste: looking
<lifeless> stub: I'm going to delete patch 75 from the branch, so that we can land on devel
<stub> lifeless: DELETE FROM bugsummary_temp_journal; is better spelt TRUNCATE bugsummary_temp_journal (but only with temporary tables until we modernize our slony)
<stub> lifeless: Sure.
<jcsackett> bac: flacoste: i seem to recall there being a uses_malone use case that buggered our desire to completely kill ILaunchpadUsage
<jcsackett> i may be confusing things though.
<flacoste> jcsackett: we could have trim the interface though to only keep the uses_malone attribute in that case?
<lifeless> 13192 pushed (just deletes that patch)
<jcsackett> flacoste: if i'm remembering correctly, yes.
<bac> flacoste, jcsackett: i have the same recollection.  they certainly look like they could be combined
<stub> lifeless: Seems fine. Extra points if you switch to the truncate for probably unnoticable performance boost, and subclass the tests in test_bugsummary so they get run twice, once with rollup, once without.
<flacoste> bac, jcsackett: anyway, just wondered when I saw both, thanks for the clarifications
<lifeless> stub: s/subclass/parameterise :P
<flacoste> might yak shave it at some point
<jcsackett> flacoste: probably worth filing a bug against it.
 * flacoste only files yak shaving bug when actually doing the shave...
<lifeless> stub: I would like to do that separately as a tech debt issue, because we've run them both with and without
<jcsackett> flacoste: seems like a reasonable policy. :-)
<lifeless> stub: I will change to the truncate right now
<stub> I'd just put the rollup into a method, subclass, and override it with a noop. But whatever.
<stub> cool.
<lifeless> stub: well theres a bunch of small classes
<stub> I thought all the tests were on one class? never mind then.
<lifeless> stub: so all of them need to subclass too - testscenarios is designed to multiply it out
<lifeless> ahm you are right
<lifeless> they are
<lifeless> so its easy and I'll do that
<stub> So... lets see if I can get these patches applied live to staging.
<stub> I guess qastaging first since it isn't replicated, then confirm I can add the table to the existing replication set on staging, then push the big red button
<lifeless> those changes are done, testing
<lifeless> \o/ big red buttons
<stub> Yell when you have pushed so I don't get confused versions of stuff on things
 * stub loves technical jargon like 'stuff' and 'things'
<lifeless> sure thing
<stub> lifeless: production is blocked until the new replica is built, unless I kill it (which is a pita to clean up)
<stub> but we can get staging etc. done
<lifeless> so I believe there is a dsd script to run thats also blocked
<lifeless> perhaps we should kill it, run that scrpit, do this on prod, and then start the replica over
<stub> yes, but it was declared non urgent
<lifeless> right, it isn't urgent, but this is
<lifeless> stub: and see -ops, the script may be urgent
<lifeless> ok tests passed
<lifeless> 13193 pushing
<lifeless> statik: I think your cal update to weekly went awol :)
<lifeless> statik: pushed
<lifeless> bah stub: pushed
<sinzui> jcsackett: do you have time to mumble about yui tests?
<jcsackett> sinzui: sure.
<jcsackett> i am in mumble now.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #624: FIXED in 5 hr 42 min: https://lpci.wedontsleep.org/job/db-devel/624/
<jcsackett> sinzui: you broke up on me.
<LPCIBot> Project devel build #793: FAILURE in 5 hr 32 min: https://lpci.wedontsleep.org/job/devel/793/
<poolie> hi all
<lifeless> SpamapS: please try your script now
<SpamapS> lifeless: good timing I delayed an accept just in case you fixed things. :)
<lifeless> SpamapS: hows it looking ?
<SpamapS> lifeless: unfortunately, the queue is full of stuff to reject today.. I haven't found something to accept just yet.. moment.
<SpamapS> Ok running now
<SpamapS> \o/
<SpamapS> lifeless: fixed. :)
<lifeless> woot
#launchpad-dev 2011-06-10
<sinzui> StevenK: mumble?
<jcsackett> sinzui: mp is finally up. https://code.launchpad.net/~jcsackett/launchpad/picker-patcher-picked-a-patch-of-personpickers/+merge/64094
<lifeless> bac: if you're still around, bug 794802 is addressed on qastaging
<_mup_> Bug #794802: many bug activities timing out due to contention on bugsummary <canonical-losa-lp> <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794802 >
<StevenK> https://bugs.launchpad.net/launchpad/+bug/408698
<_mup_> Bug #408698: Search when reporting a bug fails when spaces are used <lp-bugs> <package> <package-picker> <vocabulary> <Launchpad itself:Triaged> < https://launchpad.net/bugs/408698 >
<StevenK> https://bugs.launchpad.net/launchpad/+bug/376671
<sinzui> StevenK: I agree that th use case for 408698  is fixed. mark it as release
<sinzui> s
<sinzui> d
<bac> hi lifeless
<bac> lifeless: do you cowboy the fix for bug 794802 to qastaging b/c the bug branch does not show that it has been deployed
<_mup_> Bug #794802: many bug activities timing out due to contention on bugsummary <canonical-losa-lp> <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794802 >
<bac> s/do you/did you
<lifeless> bac: yeah, its live
<bac> lifeless: darn
<lifeless> cowboy of the year
<lifeless> it wasn't when you were qaing
<bac> lifeless: my QA is still failing
<bac> i just tried
<lifeless> ah
<lifeless> on qastaging ?
<bac> yes
<lifeless> timeout? what statement is timing out ?
<bac> OOPS-1987QASTAGING7
<bac> just happened so it
<bac> will be a while before it shows up
<bac> lifeless:  my OOPS from earlier today is https://lp-oops.canonical.com/oops.py/?oopsid=1986QASTAGING120
<lifeless> bac: yeah, that not biug 794802
<bac> lifeless: i'll prepare a rollback
<lifeless> what was the change you made ?
<lifeless> found it
<wgrant> findSimilarBugs timeouts aren't exactly uncommon...
<wgrant> I'm not sure bac's change is relevant here.
<lifeless> yeah
<lifeless> thats what I want to assess
<lifeless> gaaarh lint fixes
 * lifeless still looking for the actual change
<lifeless> bac: don't rollback
<StevenK> sinzui: Ready when you're back.
<lifeless> bac: the actual issue - a security error - is fixed
<wgrant> Yeah, this is unrelated, assuming that it's the hunk starting at line 82 of the MP diff.
<lifeless> bac: this remaining issue wasn't introduced by your change.
<bac> ok
<lifeless> pre existing condition; like the old guy that goes into the doctor complaining that other people smell bad
<bac> lifeless: how do you want me to mark that bug?
<lifeless> doctor removes wax from his ear, and can hear is own flatulence
<lifeless> boom-tish
<bac> qa-smell-old-guy
<lifeless> bac: I think its qa-ok
<bac> lifeless: done
<poolie> spiv, did your failing-faster twisted get released, or can it get into lp?
<poolie> re https://bugs.launchpad.net/launchpad/+bug/740759
<_mup_> Bug #740759: creating bzrdir on launchpad is slow (BzrDirFormat.initialize_ex_1.16) <codehosting-ssh> <hpss> <lp-code> <performance> <Launchpad itself:In Progress by jameinel> < https://launchpad.net/bugs/740759 >
<spiv> poolie: I think it's been on LP for a while
<spiv> Hmm, maybe not, just the workaround branch.
<sinzui> StevenK: I am looking at createBug in model.bug
<lifeless> ok, it could have been worse - * 1548 Time Outs
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld (*jtv) | Critical bugs:209 - 0:[######=_]:256
<jtv> hi wallyworld_ â I'm afraid I'm not well today
<poolie> hi wallyworld_
<poolie> get well soon jtv
<jtv> thanks
<wallyworld_> jtv: hi, sorry to hear that
<wallyworld_> hi poolie
<wallyworld_> jtv: there's only been one or two reviews posted and they were taken so nothing for you to check so far anyway
<adeuring> good morning
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld (*jtv), adeuring | Critical bugs:209 - 0:[######=_]:256
<cody-somerville> Hey. I was going to land https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-2/+merge/63950 but I notice some benign changes to utilities/sourcedeps.cache (looks like maybe a script regenerated it?). I assume I should ask timrc to get rid of that cruft before I land?
<lifeless> its fine
<lifeless> the cache is well, a cache.
<cody-somerville> alrighty, I'll go ahead and land it then.
<StevenK> cody-somerville: If you could suggest to Tim in future that he uses self.assertIs(<something>, None) or self.assertIsNot(<something>, None) when comparing to None?
<cody-somerville> StevenK, Sure thing. I'll pass that on in our next standup.
<jtv> cody-somerville, StevenK: in launchpad we pass the expected value first, actual value second.
<lifeless> or assertThat(thing, Is(None))
<cody-somerville> Has anyone else noticed how signing into the AWS Management Console defaults to the S3 tab, incurring you a list operation (which of course gets rounded up and you get charged atleast a cent that month even if all you did was login to the management console).
<cody-somerville> How long does the test suite take to run these days on ec2?
<jtv> I would say about 4 hours.  Wild guess.
 * StevenK kicks buildbot until pieces fall off.
<LPCIBot> Project windmill-devel build #198: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/198/
<lifeless> night everyone
<bigjools> hello, is anyone feeling brave enough to review an 800 line soyuz branch?
<bigjools> adeuring? :)
<adeuring> bigjools: sure, but let me first have a lunch break :)
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugs:209 - 0:[######=_]:256
<Ursinha> conferencing system is nice, but only for the first five minutes
<Ursinha> matsubara, jml, want to join me there?
<jml> oh hi
<Ursinha> *conferencing system music
<jml> matsubara: is that you?
<benji> Ursinha: we should write a script to pull new on-hold music from http://freemusicarchive.org every day
<jml> or some kind of U1 streaming thing.
<adeuring> bac: could you have a look at this MP: https://code.launchpad.net/~adeuring/launchpad/bug-735991/+merge/64171 ?
<bac> adeuring: sure
<adeuring> thanks!
<flacoste> james_w: hi, can i request your help for qa (should take 5 mins at most)
<james_w> flacoste, yep
<james_w> might have to delay by 30 minutes, but I'm happy to help
<flacoste> james_w: can you just try setting a Ubuntu official branch on qastaging
<flacoste> james_w: just to make sure that the package-importer will continue working after my permission change is rolled up
<flacoste> james_w: there is no hurry, so 30 mins delay is fine
<flacoste> thanks
<flacoste> it works for me on the ensemble distribution, but i'd like a confirmation for the Ubuntu case
<bac> hi adeuring
<bigjools> adeuring: did my branch scare you?
<bac> looking at getBugSubscriberPackages i don't see how it filters based on the user
<adeuring> bigjools: well, a bit, but I'm still reading..
<adeuring> bac: let me check...
<jml> gror. still haven't eaten :(
<adeuring> bac: self.structural_subscriptions_clause . That's StructuralSubscription.subscriberID == self.id
<bac> adeuring: it might be enlightening to change test_getBugSubscriberPackages to create other subscribed DSPs for another user
<adeuring> bac: sure, good idea
<bac> adeuring: thanks, i didn't see that ref to self.structural_subscriptions_clause
<bac> adeuring: and i don't really see the benefit of it either...  saves some typing but adds confusion.  but it is an existing pattern.
<adeuring> bac: the idea is that we want all DSP for which a structural subscription exists, i.e., things that are related to the result of structrural_subscriptions, but without calling this property
<bac> adeuring: i understand
<bac> my point is using self.structural_subscriptions_clause replaces a one line constraint that is obvious with a one-line property use that is not clear
<bac> or at least easily overlooked
<bac> adeuring: i suggest you undo that change
<danilos> adeuring, bac: hi guys, anyone fancy reviewing some JS code? a few branches available, so feel free to pick one and make sure to claim it so somebody else doesn't do it before you guys :)
<adeuring> bac: yeah... ok
<bac> danilos: ok, i'll grab one when free
<danilos> bac, cool, thanks
<danilos> some are oversized but mostly due to tedious JS test code :/
<bac> adeuring: approved.  a *really* nice piece of work!
<adeuring> bac: thanks!
<james_w> flacoste, still works
<flacoste> james_w: awesome, thanks!
<deryck> Hi, everyone.
<flacoste> hey deryck
<adeuring> bigjools: r=me
<bigjools> adeuring: cool, thanks.  Any questions on what's going on?
<adeuring> bigjools: I tried to figure it out by looking around in the sourceode, outside the plain diff
<bigjools> adeuring: it's not the easiest of branches to grok
<danilos> bac, adeuring: I am about to leave now, I'd still appreciate a review or two, but if you don't feel like doing it with me not available for questions, I won't mind; thanks again :)
<bac> danilos, adeuring: i am about to start on https://code.edge.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-activity/+merge/64180
<danilos> bac, cool, thanks
<bac> danilos: is that the one i should start with?
<danilos> bac, the best one to start with is probably -sections, then -subscribers and then -activity, but I did split them up so they could be looked at separately, so any is good
<danilos> bac, the order of dependencies is listed in bug 772754 (i.e. earliest ones are near the top of linked branches), fwiw
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-ok> <story-better-bug-notification> <Launchpad itself:In Progress by gary> < https://launchpad.net/bugs/772754 >
<bac> danilos, adeuring: ok, i've claimed https://code.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-sections/+merge/64176
<bac> danilos: have a good weekend
<LPCIBot> Project devel build #794: STILL FAILING in 5 hr 43 min: https://lpci.wedontsleep.org/job/devel/794/
<adeuring> danilos: sorry, I'd like to cop out from a review. It's quite late for me, and I did not sleep that well last night.
<danilos> adeuring, that's ok, thanks
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer:  bac | Critical bugs:209 - 0:[######=_]:256
<LPCIBot> Project windmill-db-devel build #377: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/377/
<LPCIBot> Project parallel-test build #25: STILL FAILING in 1 hr 18 min: https://lpci.wedontsleep.org/job/parallel-test/25/
<LPCIBot> Project windmill-db-devel build #378: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-db-devel/378/
<LPCIBot> Project windmill-devel build #199: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/199/
<LPCIBot> Project windmill-devel build #200: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/200/
<benji> bac: Will do do a pre-review of https://pastebin.canonical.com/48394/ for me?  I'm going to switch from an attribute on the exception to a marker interface and move the monkey patch from lib/lp/app/__init__.py to lib/lp_sitecustomize.py per Gary's suggestions but I wanted your opinions while I'm moving things around.
<bac> benji: ok, just a sec?  i'm trying to finish up danilo's JS review
<benji> absolutely, thanks
<bac> hi benji i've read your paste
<bac> benji: i'm confused by some terminology.  you said above you were going from an attribute on the exception to a marker interface...but that isn't what i see in the paste
<benji> bac: sorry, I meant that the paste uses an attribute on the exception and I'm changing that to a marker interface on the exception instead
<bac> benji: ok, so the paste is what, a first attempt that you're abandoning in favor of a marker interface?
<benji> bac: right, but not quite abandoning, refactoring
<bac> ok
<bac> benji: well, what you have in the paste looks reaonsable to me
<benji> (well, "refactoring" isn't technically the right word either, make that "reworking")
<benji> bac: cool, thanks for looking at it for me
<bac> benji: will it be done today?  i look forward to seeing how it morphs
<benji> I hope it will be.
<LPCIBot> Project windmill-devel build #201: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/201/
<gary_poster> bac, you'll see in https://code.launchpad.net/~gary/launchpad/bug553368/+merge/64108 that lifeless and I had a conversation that significantly changed the goal of the branch, but I think he'd be comfortable now with someone else--say, you--giving a code review, if you are willing and able.
<bac> gary_poster: i'll do so now
<gary_poster> thank you
<bac> gary_poster: your replacement fixture has a comment that it won't work with the appserverlayer.  can you enforce that?
<gary_poster> bac, uh, I hadn't thought of that. um....I don't remember if the layer is a global accessible sanely.  I don't think it is.  The test suite knows, I think: I can check if fixtures have access to the test suite.
<bac> might be nice
<bac> unclear what the failure mode currently would look like
<bac> if it is informative then i'd say not to bother
<bac> neat branch, though.  i'll approve it momentarily.  thanks.
<gary_poster> bac, failure mode is that your replacement doesn't seem to work--you've replaced the view locally, but not in the other process or whatever that the appserverlayer uses, so nothing happens.  The fixture does not get access to the test class, so I don't know of a way to do it right now.
<bac> ok
<bac> the devs have been warned!
<gary_poster> :-)
<lifeless> morning
<lifeless> gary_poster: you can check whether the layer is setup I guess, but FWIW not-working is pretty clear :)
<gary_poster> lifeless, yeah I guess the layer itself is a global :-)  but yeah, I'm inclined to leave it
<lifeless> and yeah, *totally* happy for another person to do the review; FWIW I agree with bac's comments
<gary_poster> cool, changed them, pushing
<bac> hi sinzui, can you briefly remind me of the process for unsuspending a user?
<lifeless> statik: hi
<lifeless> statik: how did your microservice go?
<bac> sound like you had a tiny funeral
<gary_poster> heh
<gary_poster> or very small communion, wedding, etc.
<lifeless> he was hacking on a node.js microservice based on the gpg signature checking requirements
<lifeless> bac: http://twitter.com/#!/sstatik/status/76831129654657024
<statik> lifeless: i have not had time to get back to it, i have vagrant building the whole machine and node.js/coffee script spitting out some boilerplate JSON, next I need to wire in some tests, a gpg layer, and connect it all together.
<statik> lp:~statik/+junk/gpg-val
<statik> it's probably horrible in 5 or more ways
<lifeless> statik: http://vagrantup.com/ ?
<lifeless> statik: I think its a great experiment
<statik> vagrant is a gem that autobuilds a VM using virtualbox (atm, libvirt support planned) and then provisions it using chef, puppet, or bash
<lifeless> the vagrant aspect surprises me a little but its neither here nor there
<lifeless> statik: vagrantup.com looks like it
<statik> yep, thats the one.
<statik> subst vagrant for ensemble, openstack, whatever the appropriate tech is when the time comes. for now it's the only one that actually works ;)
<statik> it automated the entire process from a raw CD install
<lifeless> yeah
<statik> it's only for devs though, doesn't work for managing production. it does give a nice way to try out puppet or chef recipes though, and provides a nice isolated environment to work in.
<lifeless> I guess for me, for dev environment, we want 15-20 services on one dev laptop
<lifeless> VM's just seem too heavyweight
<lifeless> one VM - sure, thats how I do my dev for LP, but one-per-service, mmmm, perhaps I'm too cautious
<lifeless> that said, the idea I have of having a network fake for each microservice means doing one VM per service would work
<statik> well, i'm only prototyping one service :) it could easily be one VM with LXC containers inside it to wall of each individual service
<lifeless> just need to bring the fake into different VM's
<lifeless> and one can shutoff a VM when finished hacking on that service
<statik> oh thats a nice idea too
<lifeless> statik: anyhow, its a good thing to play with; we don't do nearly enough playing
<statik> yep, my goal was to make it dead simple for someone else to bring up the experiment and poke at it without polluting their machine
<lifeless> benji: hi
<lifeless> benji: I'm confused by the duplicating of bug 795180 (about the subscription popup UI) onto bug 772754 (about us not showing the list of bug subscribers)
<benji> lifeless: hmm, let me look... oh, it looks like more work is being done under the auspices of that bug than the description would suggest
<benji> I'll now wonder out loud if my above interpretation is correct so gary_poster can verify for me.
<gary_poster> la la la
<gary_poster> me wonders where mup is
<lifeless> sulking?
<gary_poster> :-)
 * benji secretly hates mup.
<gary_poster> hhe
<gary_poster> heh
<lifeless> benji: also, separately, what makes the mute text bug critical ?
<gary_poster> lifeless, benji, yes, in order to address bug 772754 we had to address what mpt describes.  More properly, perhaps, you would say that the branches we have addressing that bug also address bug 795180.
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-ok> <story-better-bug-notification> <Launchpad itself:In Progress by gary> < https://launchpad.net/bugs/772754 >
<gary_poster> so there you are
<benji> lifeless: it seemed like an actual bug that we (yellow) would likely do soon (being on maintenance rotation)
<gary_poster> (you == mup)
<lifeless> benji: ah
<gary_poster> benji rules are if oops, regression or security, critical
<gary_poster> otherwise high or low
<gary_poster> this could maybe be high
<lifeless> or operational FUBAR
<gary_poster> yeah
<benji> k
<lifeless> benji: we're trying to make queue jumping work evidence based
<lifeless> benji: high/low are whether we're planning on working on it in < 6 months or not
<lifeless> and then folk pulling work from a bucket (critical/high/low) can choose whatever bit of work they want subject to however the squad is internally doing stuff
<lifeless> benji: https://dev.launchpad.net/BugTriage and https://dev.launchpad.net/BugTriage/Background
<benji> ah, I confused High and Critical
<lifeless> benji: gary_poster: thanks; -> afk
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #795: FIXED in 5 hr 20 min: https://lpci.wedontsleep.org/job/devel/795/
<lifeless> sinzui: hi; bug 794008 - I don't know if you noticed but one of the screenshots is from LP itself, so its both c-i-p and lp
<_mup_> Bug #794008: Opera displays Launchpad _without text_ <opera> <Canonical SSO provider:Triaged> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794008 >
<soren> A bit earlier today I attempted to create a new package set. The tool I used didn't make it obvious that it wasn't archive, but rather distribution specific. Wondering why it failed I went and looked at the code implementing the packagesets.new method in the API, and I'm really having trouble spotting the bit of code the rejects my request. Can anyone provide some hints?
<LPCIBot> Project windmill-devel build #202: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/windmill-devel/202/
<sinzui> lifeless: that page, with css and markup was not served my lp
<sinzui> lifeless: so if there is a browser issue the fix will be in c-i-p
<lifeless> sinzui: there is a broken page served by lp
<lifeless> https://bugs.launchpad.net/launchpad/+bug/794008/+attachment/2159063/+files/loool.png
<_mup_> Bug #794008: Opera displays Launchpad _without text_ <opera> <Canonical SSO provider:Triaged> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794008 >
<lifeless> sinzui: link 2
<lifeless> bah
<lifeless> sinzui: comment two
<sinzui> sorry, I did not see that one
<sinzui> sorry
<lifeless> sinzui: no worries; thats why I'm bringing it to your attention
<lifeless> clearly something is wrong, and I suspect their browser
<lifeless> but it might be something we're doing
<sinzui> What does this mean? It does not work with Lp's or c-i-p's markup and css. They are not related anymore
<sinzui> ha. I see the sprites render. opera had issues with that in the past
<sinzui> ph
<sinzui> lifeless: both lp and c-i-p must have the same font rules per UX. I wonder if the browser dies on "UbuntuBeta Regular",Ubuntu,"Bitstream Vera Sans","DejaVu Sans",Tahoma,sans-serif
<lifeless> sinzui: yeah, I thought that was common
<sinzui> lifeless: look at the image again. only the monospace font renders!
<lifeless> yeah
<sinzui> I suspect the user has messed with his fonts because sans-serif is the ultimate failover
<sinzui> Now might be the time to switch to oneiric. I am at EOD and I have confirmed html5browser runs on both lucid and natty.
<lifeless> \o/
<lifeless> EOW even
<LPCIBot> Project windmill-db-devel build #379: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/379/
<LPCIBot> Project parallel-test build #26: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/parallel-test/26/
<LPCIBot> Project windmill-devel build #203: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/203/
#launchpad-dev 2011-06-11
<LPCIBot> Project windmill-devel build #204: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/204/
<LPCIBot> Project windmill-devel build #205: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/205/
<LPCIBot> Project db-devel build #626: FAILURE in 5 hr 51 min: https://lpci.wedontsleep.org/job/db-devel/626/
<LPCIBot> Project parallel-test build #27: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/27/
<LPCIBot> Project windmill-db-devel build #380: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/380/
<LPCIBot> Project devel build #796: FAILURE in 6 hr 54 min: https://lpci.wedontsleep.org/job/devel/796/
<LPCIBot> Project windmill-devel build #206: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/206/
<LPCIBot> Project windmill-db-devel build #381: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/381/
<LPCIBot> Project parallel-test build #28: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/parallel-test/28/
<LPCIBot> Project db-devel build #627: STILL FAILING in 6 hr 32 min: https://lpci.wedontsleep.org/job/db-devel/627/
<LPCIBot> Project devel build #797: STILL FAILING in 6 hr 3 min: https://lpci.wedontsleep.org/job/devel/797/
<LPCIBot> Project parallel-test build #29: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/29/
<LPCIBot> Project windmill-db-devel build #382: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/windmill-db-devel/382/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #628: FIXED in 5 hr 51 min: https://lpci.wedontsleep.org/job/db-devel/628/
#launchpad-dev 2011-06-12
<lifeless> jml: I have done a release of testtools 0.9.11 on Launchpad; I can't do one on pypi,
<lifeless> s/,/./
<lifeless> jml: if you can push the tarball + sig from LP to pypi that would be great.
<lifeless> booyah
<lifeless> 91 Time Outs
<wgrant> lifeless: Yeah, forgot to drop it again :/
<wgrant> Oops.
<lifeless> hah :)
<lifeless> our bad :>
<lifeless> 20 /  687  HWDevice:EntryResource:getSubmissions ><
<lifeless> OTOH, tags portlet is -fixed- :)
<jml> lifeless: thanks.
<jml> lifeless: I've fixed pypi to allow you to release to it
<jml> lifeless: fwiw, I had to untar the tarball, run 'python setup.py register' and then manually upload the tarball & sig.
<lifeless> jml: thanks!
<lifeless> jml: I've also done a subunit release - 0.0.7; that should let your jenkins do python3 for the testing toolchain
<lifeless> typical
<lifeless> release some software
<lifeless> and instantly get 2 bug reports and a patch
<nigelb> Isn't the patch bit rare?
<lifeless> nigelb: depends on the project
<lifeless> nigelb: the annoying bit is that I did the release, *then* got the patch.
<lifeless> nigelb: so the patch isn't in the release
<nigelb> lifeless: haha :-)
<nigelb> On that note, good night.
<lifeless> night :)
#launchpad-dev 2012-06-04
<wallyworld_> wgrant: the recent apa model code - i haven't looked in detail, but it doesn't seem to update the access_policies or access_grants columns of bug task flat?
<wgrant> wallyworld_: BugTaskFlat will continue to be trigger-maintained in perpetuity, as it's still not feasible to hook or even identify all the relevant app code.
<wallyworld_> wgrant: so we still need to separate out the btf triggers then
<wgrant> wallyworld_: They're deliberately already separate
<wgrant>     bug_maintain_bug_summary_trigger AFTER DELETE OR UPDATE ON bug FOR EACH ROW EXECUTE PROCEDURE bug_maintain_bug_summary()
<wgrant>     bug_mirror_legacy_access_t AFTER INSERT OR UPDATE ON bug FOR EACH ROW EXECUTE PROCEDURE bug_mirror_legacy_access_trig()
<wallyworld_> hmmm. i thought the btf code and apa stuff just added to the model was in the same trigger
<wgrant> Nope.
<wgrant> BTF is entirely separate.
<wgrant>     accessartifactgrant_maintain_bugtaskflat_trigger AFTER INSERT OR DELETE OR UPDATE ON accessartifactgrant FOR EACH ROW EXECUTE PROCEDURE accessartifact_maintain_bugtaskflat_trig()
<wgrant>     accesspolicyartifact_maintain_bugtaskflat_trigger AFTER INSERT OR DELETE OR UPDATE ON accesspolicyartifact FOR EACH ROW EXECUTE PROCEDURE accessartifact_maintain_bugtaskflat_trig()
<wgrant> etc.
<wallyworld_> ok, i'll look elsewhere to see why btf is not being maintained with legacy triggers disabled
<wgrant> What sort of operation are you performing?
<wallyworld_> a search
<wgrant> And what does the failure look like?
<wgrant> And which triggers are you disabling?
<wallyworld_> not a failure - a former grantee is still being returned
<wallyworld_> after calling transition to target
<wgrant> transitionToTarget won't change grantees.
<wgrant> It might change policies.
<wallyworld_> am disabling bugtask_mirror_legacy_access_t etc
<wallyworld_> yes, an a person with a olicy grant should then no longer see the bug
<wgrant> Do you have a diff?
<wgrant> Right.
<wallyworld_> i'll look into it a bit to see what's happening
<wgrant> transitionToTarget will call updateAccessPolicyArtifacts, which will remove the row from AccessPolicyArtifact, which will cause the trigger to update BugTaskFlat.
<wallyworld_> that's what i was expecting
<StevenK> wgrant, wallyworld_: https://code.launchpad.net/~stevenk/launchpad/set-spph-packageupload/+merge/108513
<wallyworld_> ugh, soyuz, pass
<StevenK> Hahaha
<StevenK> wallyworld_: You said you want to learn it :-P
<wallyworld_> not today though
<StevenK> wallyworld_: It's only a 17 line query, that's *tiny* :-P
<wgrant> StevenK: I'd check that PU.changesfile IS NOT NULL to make sure it's a real upload.
<wgrant> Also perhaps check that it's the earliest SPPH for that SPR
<StevenK> wgrant: Do we care? We'd get a hold of some delayed copies PUs too, but I don't think it matters
<wgrant> StevenK: We want to be conservative here. We can easily detect data that we haven't filled in, but it's harder to detect and correct data that exists but is wrong.
<StevenK> Hmm, I'm not sure how to check it's the earliest SPPH
<wgrant> So I'd add both the checks I mentioned. changesfile IS NOT NULL ensures it's a real upload, and restricting to the first SPPH prevents it from incorrectly matching post-upload overrides.
<wgrant> spph.id = (SELECT MIN(spph2.id) FROM sourcepackagepublishinghistory spph2 WHERE spph2.archive = spph.archive AND spph2.sourcepackagerelease = spph.sourcepackagerelease) or so
<StevenK> wgrant: Hmm, make sense. Now there's this Storm thing. Let me see.
<wgrant> Easy in Storm.
<StevenK> wgrant: Hm, really? My 5 minutes of playing around has yielded nothing.
<wgrant> spph2 = ClassAlias(SourcePackagePublishingHistory)
<wgrant> SourcePackagePublishingHistory.id == Select(Min(spph2.id), tables=spph2, where=And(spph2.sourcepackagereleaseID == SourcePackagePublishingHistory.sourcepackagereleaseID, spph2.archiveID == SourcePackagePublishingHistory.archiveID))
<wgrant> or so
<StevenK> Oh Select()
<wgrant> That is how one does a subselect.
<StevenK> I was trying to do it directly :-(
<wgrant> (a quick query here shows that there are about 294 publications in oneiric that that will exclude, which seems a little high, but perhaps it's correct)
<wgrant> Ah, forgot to match section
<wgrant> 31, that's more plausible
<wgrant> StevenK: dogfood confirms that I am correct.
<wgrant> There are 30 cases in oneiric where the MIN(spph2.id) bit is required.
<wgrant> eg https://launchpad.net/ubuntu/+source/im-config/+publishinghistory
<wgrant> Note how it started in universe, was promoted, and then demoted again
<wgrant> Your query would set both universe publications to the PU
<wgrant> Which is a lie.
<StevenK> wgrant: Yeah. Writing a test for it
<wgrant> https://pastebin.canonical.com/67361/
<wgrant> Note that they all have 2 or 3 publications with the same overrides, except for ppl which was a delayed copy (because I omitted the pu.changesfile IS NOT NULL clause)
<wgrant> launchpad_dogfood=# SELECT COUNT(*) FROM packageupload pu JOIN packageuploadsource pus ON pus.packageupload = pu.id JOIN sourcepackagerelease spr ON spr.id = pus.sourcepackagerelease WHERE pu.changesfile IS NOT NULL AND pu.archive <> spr.upload_archive;
<wgrant>  count
<wgrant> -------
<wgrant> So my other assumption is correct too
<wgrant>      0
<wgrant> (1 row)
<wgrant> (PU.changesfile == original upload)
<wgrant> (PU.changesfile IS NOT NULL => original upload)
<lifeless> oh right, you guys are working today. Doh! ;)
<StevenK> wgrant: pu.changesfiles IS NOT NULL will miss PCJs, won't it?
<wgrant> StevenK: Yes. We can't match PCJs this way.
<wgrant> We'd have to match the PCJ to match the PCJ.
<StevenK> wgrant: http://pastebin.ubuntu.com/1022402/
<StevenK> Not sure why bzr di things I replaced one of the methods wholesale. :-(
<StevenK> s/g/k/
<wgrant> StevenK: Let's see
<wgrant> wallyworld_: Any luck?
<StevenK> wgrant: The tests all pass, too, so I don't feel like a complete idiot after my subselect fail.
<wgrant> StevenK: Possibly better to call spph2 matching_spph or something like that.
<wgrant> +    def test_PopulateSPPHPU_no_changesfile(self):
<wgrant> +        # PopulateSourcePackagePublishingHistoryPackageUpload will not
<wgrant> +        # set a SPPH's packageupload to a non-original upload.
<wgrant> that comment could be a bit clearer
<wgrant> And I suspect that common bits of the tests can be factored out
<wgrant> But otherwise looks good.
<StevenK> Hmmm, you're right
<StevenK> Let me do that
<wgrant> I'd like comments in the non-test code too, though
<wgrant> It's currently not exactly obvious to anyone but me why those two bits are there :)
<wallyworld_> wgrant: no. logging the sql shows an insert and then a delete from AccessPolicyArtifact, but a subsequent search selecting from btf testing (BugTaskFlat.access_policies)&&(SELECT ARRAY_AGG(AccessPolicyGrant.policy) returns true
<cody-somerville> Just Curious. Is there a particular reason that is a comment instead of a docstring?
<wgrant> Not sure. It's Launchpad convention to use a comment.
<wgrant> wallyworld_: odd. What if you break in the test or commit and inspect the tables?
<cody-somerville> Do you guys use nose?
<wallyworld_> wgrant: yeah, looking into that now
<StevenK> cody-somerville: We do not.
<wgrant> wallyworld_: Otherwise throw me a branch/diff and I'll have a poke.
<wallyworld_> ok, will do after i look into things a bit more
<StevenK> wgrant: http://pastebin.ubuntu.com/1022422/
<wgrant> StevenK: "(ie. not a delayed copy or copy job)"
<StevenK>             # If the changesfile is not None, it is an original upload
<StevenK>             # (IE. not a delayed copy, or package copy job.)
<wgrant> Sounds reasonable.
<StevenK> wgrant: Diff updated.
<wgrant> Looking
<wgrant> StevenK: You'll need indices
<StevenK> :-(
<wgrant> Otherwise it seems to just go for a seq scan every time.
<wgrant> "CREATE INDEX temp_spph ON sourcepackagepublishinghistory USING btree (id) WHERE packageupload IS NULL;"
<wgrant> is probably all that's going to give us a wonderful result.
<wallyworld_> wgrant: i found the problem - there was some previous code which may have been cut and pasted and which did [self] instead of [self.bug] and so the wrong id was being used in the searches
<wgrant> wallyworld_: Ouch, where?
<wgrant> Using a task instead of a bug?
<wallyworld_> it was my code - i just can't quite recall which of the pipes in went into first - it is not landed yet, and the test it was in was failing due to waiting for the apa stuff to go in
<wgrant> Ah, right.
<wgrant> I didn't think I did anything like that anywhere.
<wallyworld_> so oen failure was masking another problem
<wallyworld_> although i have found an issue - the createAccessGrants method in the sharing service will fail if the grants already exist
<wallyworld_> so i need to fix that or else it will blow up when f flag is turned off but the triggers are still enabled
<wgrant> wallyworld_: btw, did you see lp.testing.dbtriggers? I guess you probably won't want to merge them now, so I'll sort it out once your stuff is landed.
<wallyworld_> wgrant: no, didn't see that yet
<wallyworld_> wgrant: you obviously didn't see my DisableTriggerFixture
<wgrant> wallyworld_: Not until afterwards -- it's not in trunk yet.
<wallyworld_> oh really!? :-(
<wallyworld_> didn't realise that
<wallyworld_> but makes sense
<wgrant> wallyworld_: Bug #1007984 may be of interest
<_mup_> Bug #1007984: BugTask:+index information type widget doesn't handle failure <disclosure> <javascript> <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1007984 >
<wallyworld_> me or StevenK?
<wgrant> Both
<wallyworld_> but it's more fun to make him to javascript stuff :-)
<wallyworld_> s/to/do
<wgrant> True.
<wallyworld_> i know how much he loves it
<bigjools> JS was invented by Lucifer himself
<wallyworld_> i don't mind it now
<wgrant> You've clearly been infected.
<bigjools> wallyworld_:  I figured out that it's a really good idea to re-flush the Breville after the clean cycle.  Bleargh....
<wallyworld_> bigjools: that's what she said in -ops this morning
<wallyworld_> wgrant: when do you plan on doing updateAccessPolicyArtifacts for branches?
<StevenK> wgrant: Did you see https://code.launchpad.net/~stevenk/launchpad/spph-packageupload-index/+merge/108515 ?
<StevenK> wgrant: I'm going to hold off on landing set-spph-packageupload until the index is on prod, thanks for the +1
<wgrant> wallyworld_: Refactoring mostly done.
<wgrant> Shuffling around tests to make them less heinous now
<StevenK> wallyworld_: I don't understand enough about ChoiceSource, I guess, which is why that bug exists.
<wallyworld_> wgrant: ok, afaict as soon as that gets in all the tests in my ueber branch should pass hipefully
<wgrant> wallyworld_: Marvellous.
<wallyworld_> StevenK: i'll do it, i just wanted to get a reaction from you
<StevenK> wallyworld_: I'd be interested to see the diff
<wallyworld_> np, will do
<StevenK> wallyworld_: Besides, it's Ã¼ber ...
<wallyworld_> StevenK: ue or Ã¼ are aceptable
<wgrant> Only if you're on Windows.
<wgrant> Everywhere else has easy ways to enter Ã¼, so there's no excuse.
<StevenK> I had to set my compose key :-(
<wallyworld_> sigh, i can type ue fatser than Ã¼
<wallyworld_> same with ae, oe
<wallyworld_> you would have had a point if ue was somehow wrong
<wallyworld_> wgrant: when you have a minute, a small one, hopefully last branch in the ueber branch pipeline https://code.launchpad.net/~wallyworld/launchpad/createAccessGrants-robust/+merge/108518
<wgrant> RAlt+" u isn't that much slower :)
<wallyworld_> extra key presses
<wgrant> wallyworld_: approved
<StevenK> wallyworld_: What about Ã§?
<wallyworld_> smartarse :-P
<wallyworld_> wgrant: thanks
<StevenK> wallyworld_: Or Ã¶?
<wgrant> wallyworld_: What's the diffstat for the end of the pipe up to?
<wallyworld_> wgrant: i forget the command to get a merge preview diff
<wallyworld_> StevenK: oe ===  Ã¶
<wgrant> Now now
<wgrant> == maybe
<wgrant> Not ===
<wallyworld_> sticky fingers
<StevenK> JS disease
<StevenK> wallyworld_: Ã¤? :-)
<wgrant> ae
<wallyworld_> what he said
<wgrant> Not to be confused with Ã¦
<StevenK> Haha
<StevenK> I've forgotten how to type that one
<StevenK> Ã¦
<wgrant> Compose+a e
<wgrant> Not too hard :)
<StevenK> Hm, so that works. :-)
<StevenK> wgrant: I wonder if spph.source_package_name can be murdered yet.
<wgrant> StevenK: API
<StevenK> What? It's exported?
<wgrant> It's the only way to get the name of the source package through the API.
<wgrant> That's the reason it was created.
<StevenK> Can't we just point it at self.sourcepackagename?
<wgrant> That's no string
<wgrant> That's an object
<wgrant> An unexported object!
<lifeless> you want to export SPN? die fiend
<StevenK> self.sourcepackagename.name then
<wgrant> You'll find that looks remarkably like source_package_name
<wgrant> ie. that should be identical to what it is today.
<StevenK> wgrant: It's still using self.sourcepackagerelease
<wgrant> Ah, right.
<wgrant> You can fix that if you really want.
<StevenK> It's that, or beating my head against Django
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/denorm-spph-source_package_name/+merge/108520
<wgrant> StevenK: Did you identify and fix the preloading callsites?
<StevenK> There are preloading callsites? :-(
<wgrant> There should be.
<wgrant> Anything that exposes that over the API is going to want to preload the SPN
<wgrant> It's possible that it doesn't; if you can't find any, file a bug.
<StevenK> wgrant: greping for \.source_package_name is mostly turning up DSDs, which are a whole other barrel of laughs, and tests.
<wgrant> StevenK: I'd hope so
<wgrant> Since they won't be touching that directly.
<StevenK> The current code wouldn't have pre-loaded SPR?
<wgrant> wut?
<StevenK> wgrant: SPPH.source_package_name is exported and grabs it via SPR.
<wgrant> StevenK: Yes.
<wgrant> So code will be preloading SPR and SPR.spn
<StevenK> wgrant: Right, it's SPR.sourcepackagename, so maybe a grep for \.sourcepackagename will be better
<StevenK> wgrant: I can not see anything related besides the dominator that uses load_related() over SPR
<wgrant> :(
<wgrant> Ignore, I guess
<StevenK> BMP and bugtask
<StevenK> But I'm not sure how that fits in
<wgrant> It doesn't.
<StevenK> wgrant: Can haz +1, or you no longer care and I should self-review?
<wgrant> StevenK: I'm not sure that it's an immensely valuable change.
<wgrant> But if you want to do it then it's probably a self-review.
<StevenK> wgrant: You don't think dropping SPR out of the chain is valuable?
<wgrant> We don't have a really good set of policies around the use of denormalised data.
<StevenK> I thought the policy was effectively "denormalised data is good, we should use it"
<wgrant> win 3
 * StevenK stabs Django
<StevenK> Error: Can't find the file 'settings.py' in the directory containing '/home/steven/auditorfixture/eggs/auditor-0.0.1-py2.7.egg/auditor/manage.pyc'. It appears you've customized things.
<StevenK> -rw-rw-r-- 1 steven steven 4.0K Jun  4 16:39 /home/steven/auditorfixture/eggs/auditor-0.0.1-py2.7.egg/auditor/settings.py
<adeuring> good morning
<frankban> lifeless: ping
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 3.47*10^2
<deryck> Morning, all.
<cjwatson> Could I have "Status: Approved" on https://code.launchpad.net/~cjwatson/launchpad/db-packageset-score-rename/+merge/108336 so that I can land it?
<james_w> lifeless, should cjwatson be added to https://launchpad.net/~canonical-launchpad-committers ?
<james_w> cjwatson, toggled, assuming it was just a formality
<cjwatson> So I understand - thanks
<sinzui> jcsackett, StevenK, mumble
<wgrant> wallyworld_: http://pastebin.ubuntu.com/882822/
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<lifeless> james_w: I'm not here today; sick, but he probably meets the criteria I set out in my mail to canonical-launchpad
#launchpad-dev 2012-06-05
<wgrant> wallyworld_: My branch APA branch has landed.
<james_w> lifeless, get well soon
<wallyworld_> wgrant:  thanks, will take a look
<wallyworld_> wgrant: i just pointed chromium at https://launchpad.net/launchpad/+filebug and it works fine
<wallyworld_> 18.0.1025.168
<wgrant> wallyworld_: Ahh, indeed.
<wgrant> the problem in fact affects both.
<wgrant> wallyworld_: You have to get to the second step using non-AJAX
<wallyworld_> wgrant: what project were you filing a bug against?
 * wallyworld_ tries that
<wgrant> That is, enter a summary and click Continue on the first step before the JS changes it to an AJAX Next.
<wgrant> You have to be very quick
<wallyworld_> hmmmm
<wgrant> That then POSTs to +filebug to return the second step without AJAX
<wgrant> Which presumably lacks some of the JS or something.
<wgrant> "choices is undefined"
<wallyworld_> corner case :-)
<wgrant> Yeah.
<wallyworld_> so i guess i should try disabling the button untill all ajax is loaded or something
<wgrant> No.
<wgrant> The JS on the second stage should just not crash :)
<wgrant> Although it would be nice to solve the slowness on the first stage too.
<wallyworld_> i'll see what can be done
<wallyworld_> the trouble is doing the post before the ajax is loaded sort of violates the expected behaviour
<wallyworld_> ie it was never designed to handle that use case
<StevenK> Bleh, why does branch deletion always toss me to the "Please try again" page
<wallyworld_> timeout?
<wgrant> StevenK: The timeout is too low to delete all 80k branchrevisions
<StevenK> DROP TABLE branchrevision;
<wgrant> wallyworld_: The rest of the JS on the second step works fine...
<wallyworld_> i need to look into it - the whole bug form setup is "complicated"
<rick_h_> wallyworld_: wgrant which bug is this? (curious)
<wallyworld_> bug 1008543
<_mup_> Bug #1008543: ChoiceSources on +filebug are empty in Chromium <disclosure> <javascript> <regression> <ui> <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/1008543 >
<rick_h_> oh sucky, hadn't seen that one
<wgrant> It's actually nothing to do with Chromium. It just happened that I tested it slightly differently in Chromium and Firefox.
<wgrant> Was apparently slightly slower in Firefox.
<rick_h_> gotcha, well that's a good thing at least, consistant
<wallyworld_> yep
<rick_h_> is this combo loader related? did it work without it?
<wgrant> Unrelated
<wgrant> This is code that was new last week
<rick_h_> k, cool
<wallyworld_> rick_h_: we need to support devs running combo loader with un-minified js ie add filter: 'raw', to the config just for devs
<rick_h_> wallyworld_: yea, agree. I've been doing it manually, but would be nice to have it in config
<rick_h_> no real reason not to run ith with always raw for devs
<wallyworld_> rick_h_: we do need devs to run minified if required
<wallyworld_> for testing etc
<rick_h_> yea, but really QA would blow up if minifying messed anything up
<wallyworld_> but we can add a property to the view which used a feature flag
<wallyworld_> liek we do for yui version
<StevenK> I think we should just set raw for devmode
<wgrant> devmode is evil
<wgrant> use feature flag
<lifeless> james_w: thanks
<StevenK> Hmmm. Can't reproduce bug 919252 on dev :-(
<_mup_> Bug #919252: blueprints: can't reset Assignee etc to 'None' via popup <disclosure> <person-picker> <regression> <specifications> <Launchpad itself:Triaged> < https://launchpad.net/bugs/919252 >
<wgrant> StevenK: Privs?
<StevenK> wgrant: I'm doing as owner, but I have no idea of the Linaro set up.
<StevenK> wgrant: And he says he can do it via +people
<wgrant> StevenK: It's possible that it's been accidentally fixed, I supposed. Have you tried both Firefox and Chromium?
<StevenK> I don't have Chromium to try
<StevenK> It works in Firefox at least
<wgrant> sudo apt-get install chromium-browser
<StevenK> wgrant: *sadface*
<wgrant> You can't expect to work on a webapp without it.
<StevenK> wgrant: What about IE, Opera and Konquerer? :-)
<wgrant> Covering both WebKit and Gecko successfully makes it a lot more likely that Trident, Presto and KHTML will work.
<wgrant> It's still nice to test in the other three
<wgrant> But testing in two is far far better than testing in one.
<wgrant> Particularly since they're the dominant browsers
<StevenK> wgrant: I thought Trident was only 9? Wasn't the engine in 8 something different?
<wgrant> StevenK: Nope, since IE4.
<wgrant> Except for Mac OS versions
<StevenK> Chromium has no Help->About
<StevenK> I guess Google considers indentification too hard
<StevenK> wgrant: Works for me on dev with Chromium too
<wgrant> StevenK: Huh, it doesn't have a classical menubar at all.
<wgrant> They must have added it for globalmenus.
<wgrant> Click the in-window menu icon, then About Chromium.
<StevenK> wgrant: I'm suspecting it's already fixed, but I don't have acess as a driver to stuff on prod
<StevenK> wallyworld_: How is information_type ChoiceSource going?
<wallyworld_> StevenK: it's done, i have to finished the tests. but the tests are breaking due to animation issues so i need to sort that out
<wallyworld_> StevenK: the code (minus tests) if you are interested, https://code.launchpad.net/~wallyworld/launchpad/infotype-widget-1007984/+merge/108665
<adeuring> good morning
<wallyworld_> StevenK: i've finished updating the tests
<wallyworld_> https://code.launchpad.net/~wallyworld/launchpad/infotype-widget-1007984/+merge/108665
<StevenK> wallyworld_: Looks good.
<wallyworld_> coolio. seems to work nicely
<StevenK> wallyworld_: Approved.
<StevenK> stub: https://code.launchpad.net/~stevenk/launchpad/spph-packageupload-index/+merge/108515 would love a review.
<stub> StevenK: Your partial index WHERE clause is backwards
<StevenK> Oh, IS NOT NULL
<StevenK> Bleh
* gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: gmb | Firefighting: - | Critical bugs: 3.47*10^2
<lifeless> stub: I'm not here today, sick - I may be a few days getting over this
<lifeless> stub: the storm and sockets thing; I think my main concern is group abnormal stuff into magical storm handling codepaths
<stub> lifeless: I think I have food poisoning, or some sort of stomach bug.
<lifeless> stub: like IntegrityError we know the db is live; SocketError - thats kindof a WTF moment.
<lifeless> stub: ugh.
<lifeless> stub: my sympathies; I have horrid stomach cramps at the moment - side effect of the virus
<lifeless> stub: anyhoo, just wanted to put that out there - I'm not totally hostile to storm capturing the error, if those risks are assessed.
<lifeless> stub: probably we need two distinct bugs though:
<lifeless>  - one for handling the error
<lifeless>  - one for -why- is it occuring
<stub> lifeless: The goal of the Storm code is to reconnect when the db dies or is uncontactable or whatever. By design, this means it will hide failures if we neglect to log them.
<lifeless> [and the folk that are stressing, (OEM), will only care about the second one]
<stub> I think we log them though.
<stub> lifeless: yup
<lifeless> stub: so, if you need sick leave, JFDI, drop uhm elliot francis and I a mail (elliot cause I'm sick ATM, francis cause of this librarian issue so he knows its on hold while you recover)
<lifeless> I'm going to crawl back off into my little hole and think healing thoughts.
<StevenK> lifeless: Or stomach amuptation?
<stub> lifeless: IIRC what happens is that Storm catches all the possible crazy i-can't-talk-to-the-database exceptions and reraises them as DatabaseDisconectionErrors, flagging the connection for reconnection. We log the DatabaseDisconnectionErrors. Trick is to make sure we are logging what we should be logging. I think this is handled by storm manipulating the psycopg2 base classes so the disconnection error we catch is actually a psycopg2 exception.
<lifeless> StevenK: did I mention the razor blade throat with laryngitis ?
<StevenK> lifeless: :-(
<stub> lifeless: you sound worse than me. I'll probably take sick leave once I've investigated this landscape replication issue.
<lifeless> stub: I leave it in your capable hands; my main concern is about the 2-bug separation, not one bug w/2 tasks
<stub> yup
<lifeless> stub: thanks!
<lifeless> Lynne and Cynthia had this, seems to be a 5-day thing
<stub> erm... u1 replication issue
<lifeless> day 2 of it for me (yes, sick on a pub holiday... win)
<StevenK> stub: That index has landed as r15360 on devel.
<wgrant> StevenK: That index looks backwards
<wgrant> It's packageupload IS NOT NULL, when we want to search for ones where it IS NULL.
<stub> did he go an land the backwards version?
<StevenK> wgrant: stub said it was backwards :-(
<wgrant> StevenK: stub probably mistook it for the PackageUpload.changesfile != None
<StevenK> And ENOSTUB
<StevenK> Let me revert
<wgrant> StevenK: Can't revert
<wgrant> StevenK: Wasn't it applied live?
<wgrant> (I don't know if it was, but it probably was)
<StevenK> And we have no UK webops to check
<wgrant> jjo is around now
<deryck> Morning, everyone.
<wgrant> rick_h_: Thanks.
<wgrant> rick_h_: A scan of 22000ish rows is much quicker than a scan of 22000ish rows then a join through 5 multi-million row tables :)
<rick_h_> wgrant: yea, just got beat into my head that LIKE was evil too much over the years
<sinzui> Sweet. I just fixed a post-3-ui-cleanup bug. It only took 2.5 years
<rick_h_> sad that regex was slow, but oh well
<rick_h_> sinzui: ummm woooo!
<wgrant> rick_h_: The regex is still pretty fast, but 3x slower so meh.
<wgrant> rick_h_: (do note though that postgres 9.1 can use trigram indices to satisfy LIKE '%foo%' efficiently)
<wgrant> It's just not worth it here.
<rick_h_> yea, something tells me dropping an 8 table join is probably a good plan regardless of the new plan lol
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: gmb, rick_h | Firefighting: - | Critical bugs: 3.47*10^2
<jcsackett> sinzui: i'm looking at bug 1007208. do we want to change the character limits for all pickers, or just the person picker?
<_mup_> Bug #1007208: Searching for 'ev' in the 'add a member' picker fails <disclosure> <person-picker> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1007208 >
<sinzui> jcsackett, excellent question
<sinzui> jcsackett, The crux of the issue is that Lp Ids can be 3 characters. Lp has a nasty habit of creating Lp-ids from user email addresses, which are often 2-characters.
<sinzui> jcsackett, Thus person pickers/searches must support 2-characters.
<jcsackett> sinzui: right, i totally agree with the ">1" constraint for person pickers. i assume the bug picker doesn't do any checking regarding length, b/c that would be silly. that leaves...branch pickers?
 * jcsackett slaps head.
<sinzui> jcsackett, I think we can say projects and distros are less of a problem because users name them,
<jcsackett> sinzui: yeah. and as i just realized, branch pickers aren't an issue b/c of branch naming patterns.
<jcsackett> so this probably is just a person picker issue.
<sinzui> jcsackett, branch pickers are buggered by design. We might make them workable in a month when we optimise the slow queries to not blindly search all Lp
 * sinzui is looking at projects
<sinzui> jcsackett, we 204 2-character project names, and no complaints. Lets no touch projects
<jcsackett> sinzui: sounds good to me. i'll leave them alone.
<cjwatson> sinzui: of course the case at hand was one where the user deliberately changed his username to a two-character one, rather than it being automatically created from an e-mail address :-)
<sinzui> cjwatson, I think that is fine. Email addresses and IRC nicks can be two characters, and Lp allows them. It was ridiculous to require users to enter 3 characters to search.
* gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: -, rick_h | Firefighting: - | Critical bugs: 3.47*10^2
<bdmurray> I've noticed that the list of subscribers is in reverse alphabetical order in the subscribers portlet and I've a fix for this
<bdmurray> however it seems like this might have been intentional
<bdmurray>              // With multiple subscribers being added to the same section,
<bdmurray>             // the last one is listed first.
<bdmurray> so before carrying on I wanted to see if it is intentional and if so find out why
<bdmurray> oh well there is a bug about it - 912137 so it must be wrong
<timrc> When I review commits in Launchpad I sometimes see 'Timothy R. Chavez <timothy.chavez@canonical.com?' and so no hyperlink is produced for my name... is there something I'm doing wrong?
<beuno> timrc,
<beuno> <timothy.chavez@canonical.com?
<beuno> wrong closing
<beuno> >
<beuno> need to fix bzr whoami
<timrc> right, but where do I specify that... I'm using bzr launchpad-login timrchavez
<timrc> hm ok
<beuno> timrc, can't change it retroactively, btw
<timrc> yeah, bummer
<maxb> timrc: 'bzr whoami'
<timrc> you sort of can with the last commit by uncommitting though
<beuno> yeah
<timrc> thanks for helping me with a bzr prob in the #launchpad-dev channel
<beuno> it's 30 cents extra in here!
<timrc> ;)
<sinzui> wallyworld_, looks like mumble cannot connect properly
<sinzui> wallyworld_, wgrant. I am having networking issues with mumble
<wallyworld_> sinzui: should we try skype?
<sinzui> I am killing UbuntuOne
<sinzui> wallyworld_, your voice is 3 octaves lower. Did you balls drop?
<wgrant> sinzui: Sounds like you just need a little bit more upstream bandwidth.
<wallyworld_> lol
<sinzui> My wife is having trouble too. I think our provider switched us to 1st gear
<sinzui> StevenK, I think as a driver you can mess with this https://blueprints.launchpad.net/gdp/+spec/gdplaunchpad
<wgrant> Hm
<wgrant> Why is icon-sprites.png in the tree now...
<StevenK> I just saw that too
<wgrant> sinzui changed the filename and ignore rule, and I guess someone blindly added the unknown file.
 * wgrant fixes.
<wgrant> Except that buildbot-poller apparently hates me.
<StevenK> wgrant: Has your fixed landed now?
<wgrant> StevenK: No. i suspect that buildbot-poller will remain confused until devel finishes.
<StevenK> Bleh
<StevenK> wgrant: devel just finished.
<StevenK> wgrant: Can haz fix landed?
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
#launchpad-dev 2012-06-06
<wallyworld_> StevenK: why did you stab me?
<wgrant> wallyworld_: :(
<StevenK> wallyworld_: Run "head lib/lp/code/model/branch.py" in an up-to-date devel and spot the problem.
<wgrant> Why is the new 'user' argument on the methods at random positions?
<wgrant> -    def deletePillarSharee(pillar, sharee, information_types):
<wgrant> +    def deletePillarSharee(pillar, sharee, user, information_types):
<wgrant> -    def revokeAccessGrants(pillar, sharee, branches=None, bugs=None):
<wgrant> +    def revokeAccessGrants(pillar, user, sharee, branches=None, bugs=None):
<wgrant> +    def _populatePermissionsCache(cls, permissions_cache,
<wgrant> +                                  shared_artifact_info_types, grantee_ids,
<wgrant> +                                  policies_by_id, persons_by_id): all_permission_term = SQL("bool_or(artifact IS NULL) as all")
<wgrant> Bah, stupid irssi
<wgrant> But that function is doubly-indented.
<wallyworld_> one is an internal method, so meh
<wallyworld_> and was likely written at a different time
<wgrant> Two Person arguments with different meanings swapped in two very similar methods is rather perilous and PHP
<wallyworld_> StevenK: ah, sorry, will fix as a driveby
<wgrant> The chance of someone not getting the order wrong is roughly 0
<StevenK> wallyworld_: I have it fixed locally
<StevenK> wallyworld_: And I need to touch IBranch anyway, so I'll sort it out
<wallyworld_> thanks
<StevenK> wallyworld_: Strangely, all 4 imports are not used.
<wallyworld_> StevenK: i may have happened after devel was merged in and i was resolving merge conflicts
<StevenK> wallyworld_, wgrant: Do I want to do the same thing as IBug.subscribe in IBranch.subscribe?
<wallyworld_> i think so
<wgrant> StevenK: For what? Creating grants?
<wallyworld_> if user can't see branch they are being subscribed to, add an AAG
<StevenK> wgrant: Yes.
<wgrant> Yeah, sounds good.
<StevenK> I wonder if we want the same check in IBranch.subscribe() too
<wgrant> StevenK: Even better if you fix Branch.unsubscribe()'s new **kwargs while you're there.
<wgrant> Which check?
<StevenK> If it's a private bug, you can not subscribe open and delegated teams.
<wgrant> Probably.
<StevenK>         ignore_permissions = kwargs.get('ignore_permissions', False)
<StevenK> *staba*
<StevenK> s/a.$/
<wgrant> wallyworld_: How do I set up celery to run the jobs locally?
<lifeless> cool - http://googleonlinesecurity.blogspot.co.nz/2012/06/security-warnings-for-suspected-state.html
<lifeless> we need a warning bar like that ;>
<StevenK> "We suspect you are trying to do something silly" ?
<wallyworld_> wgrant: in tests, you use CeleryLayer, for dev, start the deleryd from bin
<wallyworld_> celeryd
<wallyworld_> and there's a fflag you add your job class name to
<wallyworld_> the flag takes a space separated list
<wallyworld_> of jobs which celery should run
 * StevenK peers at create_initialized_view()
<StevenK> Oh, team.name, sigh
<StevenK> Hmmm, where are the tests that make sure that a AAG is created when someone is subscribed to a bug?
<wallyworld_> StevenK: test_bugvisibility
<mwhudson> random buildout question
<mwhudson> where does bin/twisted come from?
<StevenK> mwhudson: It's created a buildout rule
<mwhudson> StevenK: right, i got that far
<mwhudson> which one, though?
<mwhudson> oh
<mwhudson> the scripts one i guess
<mwhudson> zc.recipe.egg:scripts != z3c.recipe.scripts i guess?
<bigjools> I really wish Python were more informative when there's an import error
<mwhudson> no, that's not it
<bigjools> mwhudson: setup.py?
<StevenK> Yeah, it's setup.py
 * StevenK rips his eyes out after reading it
<mwhudson> bigjools: aaah
<mwhudson> thanks
<bigjools> it's always fun when we randomly put the same stuff in different files
<StevenK> wallyworld_: Can you glance at https://code.launchpad.net/~stevenk/launchpad/branch-subscribe-aag/+merge/108881 ?
<wallyworld_> yes
<wallyworld_> StevenK: TestBranchSubscriptionAddOtherView could iterate over all types of open team and test with each one
<wallyworld_> StevenK: also getVisibleArtifacts - you have the return tuple mixed up
<wallyworld_> StevenK: also, there are no legacy branch triggers so that check isn't needed
<wallyworld_> StevenK: and so the reason the test_subscribe_gives_access test passes is that the branch collection assumes subscribed = visible
<wallyworld_> so that bit of code is sort of pointless atm
<StevenK> wallyworld_: Do you want to put all of that into the MP?
<wallyworld_> ok
<StevenK> (And anything else you find)
<adeuring> good morning
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban* (rvba) | Firefighting: - | Critical bugs: 3.47*10^2
<StevenK> stub: The index is for speeding up the findSPPHs() method of the garbo job, not the evil monster query in __call__.
<stub> StevenK: Still not sure if that index would work :) Has it been tested, or shall we try on staging?
<wgrant> stub: Why wouldn't it work?
<stub> wgrant: Because the planner can be stubborn :)
<wgrant> We're finding SPPHs without a PU, ordered by ID
<wgrant> I'm pretty sure I tested it on DF
<StevenK> stub: The query doesn't involve joins, it's just "SELECT id FROM spph WHERE packageupload IS NULL ORDER BY id"
<wgrant> The index is completely perfect for it, and the query is simple.
<stub> Oh, ordered by id should force it. If you have tested it, fine.
<stub> So this should go live on production?
<StevenK> Let me land it first :-)
<wgrant> Do show me the CREATE INDEX invocation first :)
<stub> https://code.launchpad.net/~stevenk/launchpad/spph-packageupload-index-redux/+merge/108852
<stub> Blocked on backups atm anyway.
<wgrant>    ->  Index Scan using sourcepackagepublishinghistory__packageupload__idx on sourcepackagepublishinghistory  (cost=0.00..52499.86 rows=11359 width=4) (actual time=0.138..0.228 rows=60 loops=1)
<wgrant>  Total runtime: 0.369 ms
<wgrant> :)
<StevenK> The commit message should please both wgrant and bigjools.
<StevenK> stub: r15367 on devel
<StevenK> I guess the backups should be done soonish?
<wgrant> Could be another hour
<StevenK> wgrant: I'm going to toss the SPPH+PU garbo job at ec2.
<wgrant> StevenK: Sounds good.
<stub> I think the control we have for not accidently deplying in fdt a db patch that should have been deployed live is the qa-ok tag on the corresponding bug? I guess we don't always have a bug.
<wgrant> stub: The normal control is that I don't make mistakes like that.
<wgrant> stub: Although live patches tend to be landed on devel.
<wgrant> So it's not an issue.
<stub> wgrant: Which makes you a single point of failure :)
<stub> Seems to be working well enough though, no point complicating things. Just musing.
<wgrant> stub: Oh, it's certainly suboptimal.
<wgrant> But quite effective for now.
<frankban> hi, I have to land an approved branch of launchpad-buildd, how to do that? I guess I could merge it locally, commit with "[r=reviewer] ..." message and then push to trunk. Am I right?
<wgrant> frankban: That's right.
<frankban> cool wgrant, thank you
 * jml resubmits branches for landing
<jml> third time. both previous times no actual feedback.
<lifeless> jml: that may mean they failed so spectacularly that failure mail got nommed
<jml> lifeless: yes. or that my email config was broken. or that something else failed.
<jml> oh actually
<jml> my bad. hadn't quite caught up with long weekend email.
<jml> I have a successful ec2 test run for lp:~jml/launchpad/archive-commercial-rename-support. Would someone please land it for me/
<jml> frankban? rvba?
<frankban> jml: ok
<jml> frankban: thank you.
<jml> frankban: thanks :)
<frankban> jml: welcome
<cjwatson> Is it just me or is WebServiceCaller.named_get busted?
<cjwatson>         return self.get("%s?%s" % (path_or_url, data), data, headers,
<cjwatson>                         api_version=api_version)
<cjwatson> but:
<cjwatson>     def get(self, path, media_type='application/json', headers=None,
<cjwatson>             api_version=None):
<cjwatson> I don't see how data can possibly be a sensible thing to pass as media_type
<mgz> does look dodgy.
<mgz> write a failing test?
<wgrant> That indeed looks a little broken.
<cjwatson> How do I run its test suite?  I seem to need to link in LP's download-cache somehow, but can't quite figure out how ...
<rick_h> cjwatson:  ./utilities/link-external-sourcecode ../devel
<rick_h> cjwatson: so I run that from a branch to link over the download cache/etc bits into my current branch
<rick_h> cjwatson: or maybe I'm not reading enough scrollback and you're talking something different...
<wgrant> cjwatson: Symlink ~/launchpad/lp-sourcedeps/download-cache into the tree as download-cache
<wgrant> And, for speed, probably ~/launchpad/lp-sourcedeps/eggs as well
<wgrant> Then python ./bootstrap.py
<wgrant> I use './bootstrap.py --version 1.5.2 --setup-source=download-cache/ez_setup.py --download-base=download-cache/dist' to avoid slow PyPI access, but you might get away without it.
<cjwatson> rick_h: no utilities/ in lazr.restful
<rick_h> cjwatson: yea, gotcha. Missed that part.
<rick_h> cjwatson: so yea, with lazr.restful I think I had to just mkdir the download-cache and then just take the first hit in downloading packages
<cjwatson> wgrant: Yeah, that's not happy.  http://paste.ubuntu.com/1026635/
<wgrant> Oh wow
<wgrant> It builds its own special lxml in buildout
<wgrant> What an odd piece of software
<wgrant> cjwatson: Oh!
<wgrant> You're using setup.py build
<wgrant> You want to run bin/buildout
<wgrant> So you get versions that work.
<wgrant> Rather than just the latest versions of everything.
<wgrant> Oh how I love Python distribution mechanisms.
<cjwatson> Ah, I see
<mgz> buildout is its own special kind of mess
<wgrant> cjwatson: lazr.restful's tests still pass, which is odd.
<wgrant> Surprised it hasn't bitrotten.
<czajkowski> rick_h: hot desk is where people have a spare desk in an office and either rent out or some just leave it free so they open it up to people travelling :)
<rick_h> czajkowski: ah, ok. That makes sense I guess.
<czajkowski> rick_h: usually found in co-working spaces
<cjwatson> wgrant: And the Accept header seems to make little difference (the obvious test that might fail in fact passes), so I suspect actually this media_type brokenness isn't my problem
<cjwatson> If I could get a little more than "HTTP/1.1 400 Bad Request" out of the damn thing it might help
<wgrant> cjwatson: What're you trying?
<cjwatson> Trying to convert test_distroseries_dist_upgrader to use my new PU API rather than CommandRunner.  The request/response pair is http://paste.ubuntu.com/1026681/
<wgrant> cjwatson: Huh, that's pretty useful.
<cjwatson> Current untidy code diff is http://paste.ubuntu.com/1026682/
<cjwatson> (I can probably go back to .named_get there; it fails the same way either way ...)
<wgrant> Yeah, named_get would be a bit more clear.
 * wgrant tries.
<cjwatson> Oh, that won't be desperately useful without http://paste.ubuntu.com/1026688/ too.
<wgrant> Well
<wgrant> Shouldn't be relevant to that error :)
<cjwatson> Yeah
<wgrant> cjwatson: Does that tests actually do anything useful?
<wgrant> It's probably because the test is not running in AppServerLayer
<wgrant> So there's no appserver to talk to.
<cjwatson> Aha, that would make sense
<wgrant> But there's no point having that as a webservice test.
<cjwatson> It's an integration test for whether dist-upgrader uploads can be accepted/rejected.  There was a bug about that once upon a time.
<cjwatson> It's not really testing the webservice - but it's currently using stuff from soyuz/scripts/queue.py, which I want to kill
<cjwatson> And since it's basically doing the equivalent of "queue accept", it seemed to make some sense to transfer that over to the new equivalent of the same tool
<wgrant> cjwatson: See r3691.1.200
<wgrant> cjwatson: The only significant changes in that rev were to the queue tool itself
<wgrant> The formatting and such
<wgrant> I'd delete the test.
<cjwatson> Wait, how come say TestBuildPackageJobScore works then?  It's not running in AppServerLayer either.
<wgrant> Hm
<wgrant> Perhaps WebServiceCaller works in FunctionalLayer
<wgrant> Your test is currently Zopeless
<wgrant> FunctionalLayer calls some ancient Zope stuff which adds fake HTTP hooks and stuff like that.
<wgrant> Anyway, that test isn't useful if the queue tool doesn't exist any more.
<cjwatson> I guess I just feel uncomfortable about having nothing that does a straight-through test of accepting custom uploads.
<wgrant> Ah
<cjwatson> But you think the original bug was purely presentational in the queue tool/
<cjwatson> ?
<wgrant> If there's really nothing else that tests that sort of thing, replace it with a model test.
<cjwatson> Right.  I'll have a look, maybe there is
<wgrant> Webservice tests are slow and awkward, so they should be avoided whenever possible.
<wgrant> And in this case the webserviceness is entirely irrelevant.
<cjwatson> Makes sense.  Thanks.
<bac> hi sinzui
<czajkowski> sinzui: aloha!
<czajkowski> sinzui: I've 2 projects being created by one person for two seperate things, however using a licence I've never heard of, would you mimnd if you get a chance having a look please?  https://launchpad.net/projects/+review-licenses
<wgrant> czajkowski: Non-free
<wgrant> "The user may use the work for any good purpose and he may not use it to
<wgrant> harm others or violate the permissive principles of Islam"
<czajkowski> wgrant: it must be sleep time for you!
<wgrant> It's never sleep time when there are stupid, stupid licenses to tell to GTFO,.
<czajkowski> :/
<bac> wgrant: :)
<wgrant> Hm
<wgrant> Although the English version isn't binding, so that's relying on precision of the translation.
<czajkowski> wgrant: aye it's been the strangest one I've seen so far
<wgrant> Debian says it's non-free.
<wgrant> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=579796
 * benji contemplates the craziest possible license he could come up with.
<czajkowski> well I've seen the ones that are peoples own license, but they just come across as rude
<czajkowski> the above one seemed well thought out and written but made no sense for a lp project
<benji> I'm remninded of the ExtTLD license which includes "You agree you are not involved in or profit from the use of animals for entertainment such as circuses, hunts, rodeos and races etc."
<benji> All the open-source-using clowns add an extra tear to their makeup when they read that.
<nigelb> wtfpl
<nigelb> haha
<nigelb> I like wtfpl.
<nigelb> "Do whatever the f you want with this code"
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban* (rvba), abentley | Firefighting: - | Critical bugs: 3.47*10^2
<cjwatson> These licences are all basically well-intentioned but fall afoul of not everyone agreeing on general principles of "good".
<cjwatson> And in general trying to use software licences to control things that aren't actually much to do with software.
<stub> I recall a beer licence, which granted the user permission to buy the author a beer if they wanted.
<nigelb> stub: I think I like that license, but it should be s/beer/booze/g
<benji> yeah, fortunately the "field of endeavor" restrictions that used to be popular have mostly gone away.
<wgrant> benji: That's not one I've run into before.
<wgrant> Quite impressive.
<wgrant> ly misguided
<benji> heh
<bac> czajkowski: note that the project is dual licensed with GPLv3.  it is not triggering the 'you must buy a subscription' warning.
<bac> the two licenses are incompatible, but that's not our concern
<wgrant> bac: Is it dual-licensed, or did they just say that?
<bac> wgrant: the GPL 3 text file is in their branch
<benji> wgrant: you would probably appreciate _why's "Death and Repudiation" license which states that you only have the right to use the softwre after you are dead.
<wgrant> bac: Surprising. I find people mostly say GPL to get around the Launchpad checks, but it seems that it is indeed the case here.
<bac> i'd be in favor of at least communicating to the owner that waqf is not OSI-approved and does not meet our "field of endeavor" restrictions.
 * bac wonders wth the project actually does
<benji> bac: ironically it's a guidance system for nuclear warheads
<czajkowski> yeah...
<sinzui> czajkowski, That license is not the first thing I want to read in the morning
<czajkowski> sinzui: aplogies
<czajkowski> sinzui: wasnt my cup of tea on a wednesday either :)
<sinzui> The "good" aspect always leads to a discusion of discriminating against the evil doers of this world.
<sinzui> Debian agrees, they accepted a package with the same license as non-free https://launchpad.net/ubuntu/quantal/+source/othman/+copyright
<sinzui> czajkowski, I will send a license-conflict-notice to maintainer
<czajkowski> sinzui: okie dokie thanks
<czajkowski> sinzui: sorry for the bad start but I did want to check with folks first
<wgrant> sinzui: bac points out that it's GPL too, so it's strange but fine.
<sinzui> wgrant, It had to be relicensed
<sinzui> wgrant, there was a very long thread and a bug about it. The agreement was it was non-free
<wgrant> Hah
<sinzui> wgrant, do disagree?
<wgrant> http://www.ojuba.org/wiki/waqf/faq is quite hilarious.
<wgrant> sinzui: Waqf is clearly non-free. No disagreement there.
<sinzui> okay. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=579796 and this thread are in agreement http://www.archivum.info/debian-devel@lists.debian.org/2010-07/00051/-quot-Waqf-quot-General-Public-License-in-Debian.html
<wgrant> sinzui: Right, but it's moot since it's dual-licensed.
<sinzui> yes indeed.
<jml> need another branch landed: https://code.launchpad.net/~jml/launchpad/generate-ppa-logging/+merge/108040
<jml> did a test run. three doctests failed and my last push just fixed those.
<sinzui> czajkowski, you can approve the two projects since they are gpl. They can join the older Microsoft licenses we also have in the system.
<jml> this is necessary to remove a cowboy so would appreciate a swift landing.
<wgrant> jml: That cowboy has been deployed over like 4 times :)
<wgrant> It's long gone.
<jml> wgrant: oh ok.
<jml> wgrant: well, the branch still needs landing :)
<wgrant> jml: You don't have privs?
<jml> wgrant: pqm bounced my last attempt (last week)
 * jml tries again
<wgrant> That's not very nice of it.
<wgrant> Worth another try now we're not in testfix or anything.
<jml> hmm.
<jml> wrong key is being used :\
<czajkowski> sinzui: bac wgrant thanks for the help with that one
<jml> haoesuhseo
<jml> gpg.conf has default key set to the correct key but lp-land uses the old key. wtf.
<sinzui> wgrant, where am I going to put all the bug  you reported?
<wgrant> jml: bazaar.conf has a gpg_signing_key setting
<wgrant> sinzui: I'll hopefully fix most of them tomorrow.
<jml> which I don't have set... doing so now
<wgrant> jml: Hm, odd then.
<jml> \o/
<wgrant> jml: Marvellous.
<czajkowski> sinzui: are you spring cleaning answers?
<sinzui> czajkowski, I fulfilled my promise to let other people do answers for a year
<czajkowski> sinzui: mrevell hangout link ?
<cjwatson> wgrant: I didn't go as far as setting you as the only reviewer, but I think https://code.launchpad.net/~cjwatson/launchpad/queue-api/+merge/108967 could particularly use your review.
<jcsackett> sinzui: i see that you have moved it to the backlog since this morning, but i spent a little time trying some changes to the bugstatus display for bug 997646
<_mup_> Bug #997646: confusing colour in the same picker with different enums? <disclosure> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/997646 >
<jcsackett> sinzui: http://people.canonical.com/~jc/images/bugstatus-color-issue.png
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: 3.47*10^2
<jcsackett> it's just a bit of differentiation, but i think it may be enough. gray *is* a pretty good color choice for those options.
<sinzui> jcsackett, don't be concerned about where I moved it. It stil needs to be worked on
<sinzui> jcsackett, I wanted to add wgrant critical and high bugs since they are the items that block the sharing beta
<sinzui> jcsackett, +1 for the bold
<sinzui> jcsackett, I think some of our shades of grey violate WCAG. We need more contrast
<sinzui> jcsackett, #666 is the lightest text can be. I bet we have shades that are higher.
<sinzui> jcsackett, sweet, PSU has a cheatsheet for us! http://accessibility.psu.edu/contrasthtml
<sinzui> jcsackett, I see in colour.css that we use #999, #bebebe, and gray. We use these for status colours. I see in form.css that .formHelp used for descriptions is #777. I think the last must be changed to #666, and we may want the others to be #555
<jcsackett> sinzui: sorry we keep missing each other. i've read your comments here and on the MP; i'll give some other colors a try.
<sinzui> thank you
<james_w> feature flag convention is None for not set right?
<james_w> what should be done for booleans?
<james_w> just bool() and get on with it?
<lifeless> there is a faq on this
<lifeless> I odn't remember if its in the code or the wiki
<lifeless> but yeah, bool() - so unset, and "" both evaluate to 'off' and anything else to 'on'
<wgrant> Most notably, 'off' evaluates to 'on'
<lifeless> as does False and false
<lifeless> and 0
<james_w> I don't see it on https://dev.launchpad.net/FeatureFlags
<lifeless> check the memcache flag
<maxb> Does anyone remember what the final state of fixing branch stacking URLs to not break if branches were renamed, was?
<maxb> I had somehow got the idea that all existing branches had been fixed up to point at +branch-id URLs, but apparently not
<lifeless> they were
<lifeless> branch-distro regressed it
<lifeless> as its code isn't updated
<maxb> *ah*
<james_w> lifeless, https://dev.launchpad.net/FeatureFlags#Boolean_values if you care to check
<james_w> hmm, this test is running as a different dbuser that doesn't have DELETE on featureflag
<lifeless> you need to switch dbuser back before dropping the flag
<james_w> implicit teardown doesn't make it that easy to switch for that bit of code
<lifeless> so you need two context managers
<lifeless> there is a context manager for dbuser and one for flags, should work fine
<lifeless> if you're using useFixture, you should be able to useFixture the dbuser context manager.
<lifeless> If its not a fixture, a trivial adapter is probably in order
<james_w> useFixture to the non-standard user
<james_w> or to a standard user for this one particular test?
<lifeless> nonstandard user
<lifeless> usefixture(a); usefixture(b); do stuff is equivalent to
<lifeless> with a,b: dostuff
<lifeless> a needs to do the flags
<james_w> yes
<lifeless> b needs to do the dbuser switch
<james_w> but currently this TestCase just uses dbuser = config.generateppahtaccess.dbuser
<james_w> so I'll have to unpack that, or move this test to a different TestCase
<lifeless> what, as a class variable ?
<james_w> ah
<james_w> that's not a standard thing it turns out
<lifeless> if so, yes, unpack it (personally I find that horrible: its not as flexable as resources=... and you can't express dependencies etc)
<wgrant> lifeless, james_w: What about my dbuser context manager?
<wgrant> from lp.testing.dbuser import dbuser
<wgrant> with dbuser('testadmin'):
<wgrant>     # hahah i am god
<james_w> wgrant, I've used that
<james_w> I couldn't just use it directly though
<wgrant> Ohhh
<wgrant> FeatureFixture, right
<wgrant> That'll teach me to read more than 5 lines of scrollback
<lifeless> :)
<lifeless> its possible
<lifeless> perhaps
<lifeless> that featurefixture should auto switch as needed
<lifeless> I'd worry about magic side effect syndrome
<james_w> well, that's sent the diffstat the wrong way
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1027740/
<wgrant> wallyworld_: Oh
<wgrant>         subscribed_invisible_bugs = store.find(
<wgrant>             Bug,
<wgrant>             BugSubscription.bug_id == Bug.id,
<wgrant>             BugSubscription.person == self.grantee,
<wgrant>             bug_filter)
<wgrant> RemoveBugSubscriptionsJob will remove members.
<wgrant> RemoveGranteeSubscriptionsJob does not
<wallyworld_> yes
<wallyworld_> i think so
<wallyworld_> will have to relook at the code
<wallyworld_> to rehresh my memory
<wgrant> In fact I think I said that in the bug.
<wgrant> But that was like a day ago
<wallyworld_> so you saying removing a team member calls the wrong job?
<wallyworld_> wgrant: i just re-rea dthe bug report: "If I subscribe a team and then one of its members to a bug...."
<wallyworld_> if the member is separately subscribed, then they will retain access
<wallyworld_> even if they are removed from the team
<wgrant> wallyworld_: Not quite
#launchpad-dev 2012-06-07
<wgrant> wallyworld_: Because the team has access, a grant is not created for the member.
<wgrant> wallyworld_: So they only have access through the team.
<wgrant> When the team's access is revoked from +sharing, a REmoveGranteeSubscriptionsJob removes the team's subscription.
<wgrant> But doesn't remove the person's subscription, even though they no longer have access.
<wallyworld_> oh, from +sharing. the remove subscription for former members works when using the team page to revoke membership
<wallyworld_> so there could be a case to drop the RemoveGranteeSubscriptionsJob and just rely on the RemoveBugSubscriptionsJob
<wgrant> wallyworld_: I think they should indeed be merged.
<wgrant> but RemoveBugSubscriptionsJob would need to not require bug_ids.
<wallyworld_> probably, it will need to be looked at
<wallyworld_> the RemoveGranteeSubscriptionsJob was written first and then several pipes later the RemoveBugSubscriptionsJob came along
<wgrant> Ah, I see.
<StevenK> Perhaps we should just use RemoveGranteeSubscriptionsJob and have it for both branches and bugs
<wallyworld_> StevenK: that job was written to handle both
<StevenK> Then why does RemoveBugSubscriptionsJob exist? :-)
<wallyworld_> StevenK: the former removes access for an individual subscriber from the specified bug/branches for a specified pillar
<wallyworld_> the later removes subscriptions for any artifacts any person can no longer see
<wallyworld_> for any pillar
<wallyworld_> the former was written early on to work will the +sharing page
<wallyworld_> the latter was written to handle team membership revocation and evolved from there
<wgrant> The rules for both should be the same.
<wgrant> But one is information_type-wide
<wgrant> And the other is restricted to a set of bugs.
<wgrant> Both need to consider teamparticipation.
<wallyworld_> one is potentially more efficient because it is more constrained
<wallyworld_> so perhaps both are needed, but the remove gants one needs to include team membership
<wgrant> Why are both needed?
<wgrant> One has bugtaskflat.information_type = 3
<wgrant> The other has bugtaskflat.bug IN (1, 2, 3, 4)
<wgrant> They should otherwise be identical, AFAICT
<wallyworld_> i did say "potentially" - i need to review the code
<wgrant> Yeah
<wallyworld_> the one from the +sharing page is constrained to that pillar so potentially may have less work to do in the db
<wallyworld_> bugtaskflat.bug IN (1, 2, 3, 4) will return far less rows that infotype = 3
<wallyworld_> hence the downstream processing may be less
<wgrant> Er yeah, I mean (bugtaskflat.information_type = 3 and bugtaskflat.product = 1) or (bugtaskflat.bug IN (1, 2, 3, 4))
<wgrant> The two types of constraints are required for efficiency.
<wallyworld_> still less rows
<wgrant> But the implementation can otherwise be identical.
<wgrant> Just one filters by infotype and pillar, the other by a predefined list of bug IDs.
<wallyworld_> yes, the implementations can likely be merged now that both have been written
<wallyworld_> but the different filtering should remain probably
<wgrant> Yup
<wgrant> wallyworld_: btw, bin/celeryd -l DEBUG -Q job --config lp.services.job.celeryconfig
<wgrant> Handy for testing stuff outside tests.
<wgrant> Without waiting 50s for the cronjob to start
<wallyworld_> cool, i did know that a couple of weeks ago but forgot
<lifeless> ahahaha http://www.leakedin.com/
<lifeless> also leakedin.org :>
<wallyworld_> StevenK: what did we decide about have a public A stacked on public B but then making B userdata - A should also become userdata?
<StevenK> wallyworld_: Hm, I think I've swapped that out. Let me think.
<wallyworld_> ok. i have a test which does that and fails
<wallyworld_> it used to set private = true and the triggers would update everything
<wallyworld_> but setting info type = userdata fails since A remains public
<StevenK> wallyworld_: Right, so the information_type won't change, but we check if the stacked_on branches information_type is private in places where we were depending on transivitely_private.
<wallyworld_> StevenK: sure, we now check for branches "under" the one being changed, but the old transitively private stuff also went "up" the chain
<StevenK> Yes, we don't do that.
<wallyworld_> i think we need to
<StevenK> Why?
<wallyworld_> or else we will have public stacked on user data and will be in the same mess
<wallyworld_> as we had before transitively private
<wgrant> wallyworld_: It's less than practical to do that universally, and the cases where it's useful are less than common.
<wallyworld_> it's not impractical if we use triggers
<wgrant> wallyworld_: If you're changing a stacked-on branch from public to private, you should know you're already in strife.
<StevenK> Neither wgrant or I want to use triggers.
<wgrant> wallyworld_: It's still impractical.
<wallyworld_> why impractical?
<wgrant> wallyworld_: It's just that it's so unused nowadays that the impractical cases haven't shown up yet.
<wallyworld_> and if we don't fix this, users will shoot themselves in the foot
<wgrant> wallyworld_: Writing to a few thousand branches because of a write on one branch is silly.
<wallyworld_> that's what we do now with the triggers and it works fine
<wgrant> wallyworld_: Users have shot themselves in the foot in the past due to Launchpad staff misconfiguring branch visibility policies.
<wgrant> sinzui has since fixed them, and in the new model it's impossible.
<wallyworld_> so are you sating allowing a public branch to stack on a private branch is ok?
<wallyworld_> because the code as written allows that
<wgrant> wallyworld_: The normal footgun situation is handled here, as it was before. If you push up a new public branch stacked on a private branch, it will be overridden to private.
<wgrant> What's not handled is making an existing public stacked-on branch private.
<wallyworld_> wgrant: sure, but now the other way arpund
<wallyworld_> not
<wgrant> Now, we should probably warn in that case.
<wgrant> But it's going to be extremely rare.
<wgrant> Extremely.
<wallyworld_> you can make a stacked on branch private
<wallyworld_> the code allows it
<wgrant> Sure.
<wallyworld_> so people will do it
<wallyworld_> there's a test which does it
<wgrant> It's possible to do.
<wallyworld_> i think we are leaving a huge hole
<wgrant> +sharing will say "you have n private branches"
<wallyworld_> and don't understand why we are regressing
<wgrant> The public branch will probably have a warning say "wtf are you crazy"
<wgrant> But the situation should never occur again.
<wgrant> Unless someone forgets to set up private branches before they push their code.
<wallyworld_> should !+ won't
<wallyworld_> it will and can occur via the api i think (need to check)
<wgrant> I'm not saying that it's impossible to do.
<wgrant> I'm saying that it is very unlikely to happen.
<wallyworld_> the system should not allow people to do this
<wgrant> Unless someone deliberately does it.
<wallyworld_> unlikely != won't
<wallyworld_> well they may not realise
<wallyworld_> what's to stopme from changing my branch to private not realisng it is stacked on
<wgrant> Nothing.
<wgrant> But why would you do that?
<wallyworld_> what, change a public branch to private?
<wgrant> Yes.
<wallyworld_> why not?
<wgrant> Why?
<wallyworld_> you may want to do future work in secret
<wgrant> Anyway. Doing unbounded cascading writes is foolish, and this case is sufficiently harmless and so extraordinarily uncommon that I'm pretty sure it's not worth it.
<wallyworld_> it's not unbounded really
<wgrant> There may be some sense in adding a warning to the stacked branch.
<wgrant> It is unbounded.
<wallyworld_> not in practice
<wallyworld_> and it works just fine now
<wallyworld_> so we are causing a deliberate regresiion
<wgrant> Has it ever been used?
<wallyworld_> in tests
<wallyworld_> i have no way of knowing whether on prod
<wgrant> It wasn't used during the several week information_type transition on production.
<wallyworld_> ingrequent != won't ever
<wallyworld_> we should not allow the system to get into a bad state
<wgrant> launchpad_dogfood=# SELECT COUNT(*) FROM branch WHERE stacked_on = (SELECT id FROM branch WHERE unique_name = '~launchpad-pqm/launchpad/db-devel');
<wallyworld_> and that's what we wil be doing in introducing this regression in functionality
<wgrant>  count
<wgrant> -------
<wgrant>   5018
<wgrant> (1 row)
<wgrant> That's pretty unbounded.
<wgrant> wallyworld_: What is wrong with the state?
<wallyworld_> 5018 < infinity
<wallyworld_> public branch stacked on private branch doesn't make sense
<wgrant> It doesn't make sense, no.
<wallyworld_> and yet we will be allowing it
<wgrant> But why do we need to prevent it?
<wallyworld_> because it doesn't make sense
<wgrant> We should prevent things that are harmful.
<wallyworld_> and the whole reason transitively private was done was to stop this
<wgrant> Not quite
<wallyworld_> that's the only reason transitivelt private was done
<wgrant> transitively_private's main use case is handled by information_type
<wgrant> That case is the one where a branch is pushed as public, but is stacked on a private branch.
<wgrant> In that case the user is probably unknowingly making a mistake.
<wgrant> We need to handle that, because it happens often due to misconfiguration.
<wallyworld_> and the other case will also happen
<wgrant> The case that transitively_private handles now, but we are dropping with information_type, is where a branch and its stacked-on are public, and the stacked-on transitions to private.
<wgrant> In that case the branch owner is already aware of some information disclosure.
<wgrant> They're going to want to audit their project's sharing once they've fixed the immediate hole.
<wallyworld_> yes, and i totally disagree with that regresiion
<wallyworld_> it is insane we are allowing bad data from one direction and stopping  it from the other
<wgrant> Hm?
<wgrant> We're not doing it to stop bad data.
<wgrant> We're doing it to stop people from a footgun that they can't know about.
<wallyworld_> and yet we will allow it still
<wgrant> No.
<wallyworld_> yes
<wallyworld_> change the stacked on to private and boom
<wgrant> Boom? What boom?
<wgrant> Nothing breaks.
<wgrant> Some stuff may be left public.
<wgrant> But it's not unknown.
<wallyworld_> what breaks is that people will want to work with the public branch but cant
<wgrant> Public->private of a stacked-on happens as a result of an incorrect configuration being discovered.
<wgrant> Ah!
<wallyworld_> because they can't see the stacked on private branch
<wgrant> Now that's something we can't solve.
<wgrant> They can't work with it either way.
<wgrant> If it's public, they can no longer work with it since it's stacked on a private branch.
<wgrant> If we make it private, they can longer work with it since it's private.
<StevenK> 2012-06-06 23:26:57 DEBUG2  [PopulateSourcePackagePublishingHistoryPackageUpload] Done. 231995 items in 337 iterations, 807.064977 seconds, average size 688.415397 (287.45639486/s)
<wgrant> So it's no more unusable this way.
<wallyworld_> yes, and so it should be marked private
<StevenK> wgrant: ^
<wgrant> wallyworld_: Why?
<wallyworld_> so that people realise it's private. as i said, that why tp was introduced
<wgrant> wallyworld_: This is not a regression which makes those forgotten branches less usable. And this is not a regression which prevents an audit from working, since the other branches will clearly show up as public too.
<wgrant> wallyworld_: tp's main usecase is, as I said, to prevent the initial footgun case.
<wallyworld_> it is a regression which removes previously agreed useful behaviour
<wgrant> *Not* to handle this change later.
<wgrant> It's very expensive and probably not worthwhile behaviour.
<wallyworld_> and we will now be allowing the initial footgun again
<StevenK> wgrant: And racy.
<wgrant> No
<wgrant> How?
<wallyworld_> by allowing a stacked on branch to be marked private
<wgrant> We explicitly do handle the initial push case.
<wgrant> That's not the initial footgun.
<wallyworld_> it's a way to create the problem without people knowing
<wgrant> Making a branch private later is a very explicit operation.
<wallyworld_> and one which is easy to mess up
<wallyworld_> if you don't realise yor branch is stacked on
<wgrant> Sure.
<StevenK> Then let's warn in the UI and move on
<wgrant> And what is the fallout from it?
<wallyworld_> we just simply cannot alllw this
<wgrant> It is messy, but not a significant security issue.
<wallyworld_> ui != api
<wgrant> So a warning is probably sufficient.
<wallyworld_> it's a significant functional issue
<wgrant> In an extraordinarily rare situation.
<wgrant> Where someone has already broken shit.
<wallyworld_> since it inadvertantly stops peope working wit the public branch
<wgrant> Huh?
<wgrant> That's irrelevant.
<wallyworld_> not rare enough that tp wasn't needed
<wgrant> We'd just be advertantly stopping people from working with it.
<wgrant> wallyworld_: tp solves the footgun case.
<wgrant> This also solves the footgun case.
<wallyworld_> tp does more than that
<wgrant> tp's implementation happens to also solve the later transition case
<wallyworld_> we are deliberately regressing
<wgrant> This does not. It consciously does not.
<wgrant> Correct.
<wgrant> But it's not a substantial regression in any way.
<wallyworld_> not "happens", it was a key part of the implementation
<wallyworld_> it is a substantioal regrsiion
<wallyworld_> we must not allow users to do things whuch break the system
<wgrant> It does not break the system.
<wallyworld_> it breaks the ability for people other than themselves to work wit the stacked branches
<wgrant> I can make a public branch unusable in other ways.
<wgrant> eg. by deleting the repository from it.
<wallyworld_> so?
<wallyworld_> 2 wrongs doesn't make a right
<wgrant> And all it does is avert a tiny bit of user confusion.
<wgrant> Since if we follow tp, we just make the branch even less usable.
<wallyworld_> it does a lot more than that
<wgrant> Not more usable.
<wallyworld_> why is the branch less usable wit tp?
<wgrant> wallyworld_: Your complaint is that with information_type the branch is unusable by people who don't have access to the stacked-on branch.
<wallyworld_> without tp, the ui lies
<wgrant> wallyworld_: With tp the branch is just as unusable.
<wgrant> Sure.
<wgrant> But the UI can say "uh, your branch is stacked on a private one, but it's public. you're probably wrong"
<wallyworld_> but it the private branch could be several layers down
<wgrant> That's true.
<wallyworld_> and model code also needs to treat the branch like it is private
<wallyworld_> to exlcude it from search results etc
<wgrant> Why?
<wgrant> It doesn't break anything.
<wallyworld_> if i change a staced on branch to private, any publiv branche sstaked on it must not show up in search results
<wgrant> It just makes it a little confusing.
<wgrant> Why not?
<wgrant> https://pastebin.canonical.com/67615/
<wallyworld_> because the stacked branch is to be considered private
<wallyworld_> that was the required tp functionality
<wallyworld_> when that work was speced out
<wgrant> canonical-payment-service and ztrustee were misconfigurations that sinzui fixed. information_type handles those identically, because it's the initial footgun case. libunity-webapps was a misconfiguration that required a full project audit afterwards. Everything else has less than 5 branches in this state.
<wgrant> Which proves that no large transition has ever made use of the tp ability that we're regressing on.
<wgrant> Except *possibly* libunity-webapps.
<wallyworld_> sure, but just because it's uncommon, we can't just unilaterally decide to regress
<wgrant> That paste covers every branch that has made use of tp, except for those where explicity_private was set afterwards.
<wgrant> wallyworld_: We're regressing on a feature that has never been used.
<wgrant> Not once.
<wallyworld_> what about libunity-webapps
<wgrant> wallyworld_: What if we don't consider the stacked branch private?
<wgrant> wallyworld_: What if we consider it misconfigured?
<wallyworld_> and it's not so much that it's used, it is designed to prevent inconsistent data
<wallyworld_> and confusion
<wgrant> libunity-webapps I shall not go into here, but it was a complete misconfiguration of a private project.
<wgrant> In the case of a misconfigured project like that one, you should be fully auditing +sharing anyway.
<wallyworld_> even if we consider it misconfisgured, why would we deliverately design out a saftely check to prevent the misvonfiguration in the first place?
<wgrant> wallyworld_: It's not proven useful even once, it requires unbounded write cascades, and it is non-trivial to implement with information_type.
<wallyworld_> it was useful because of the state lp was in befre it was implemented
<wgrant> The main usecase for tp is extremely useful.
<wallyworld_> there were a large number of public branches stacked on private
<wgrant> This one is not.
<wgrant> Sure.
<wallyworld_> and now we can have the all over again
<wgrant> No!
<wgrant> No!
<wallyworld_> yes
<wgrant> Only if a project changes their stacked-on branch without touching the rest.
<wallyworld_> why do you say no when we have already agreed it can happen?
<wgrant> In that case it's not a vulnerability.
<wgrant> It's a project being stupid.
<wallyworld_> but they may no realise their branch is stacked on
<wgrant> wallyworld_: As we've seen, the vast, vast majority of cases where it can happen is the initial push.
<wgrant> So we won't have a large number of public branches stacked on private.
<wallyworld_> it can happen anytime
<wgrant> We might have 1% of what we had before
<wgrant> wallyworld_: Technically it can.
<wgrant> Practically it won't unless the project owner is a moron.
<wallyworld_> and therefore it will
<wgrant> Is it a problem?
<wallyworld_> moron != careless
<wallyworld_> yes
<wallyworld_> otherwise tp would not have been approved to be resourced
<wgrant> Huh?
<wgrant> TP solved a very real problem
<wgrant> It was totally worth it
<wgrant> It also solved another problem that wasn't real.
<wgrant> We have hard data on that now.
<wallyworld_> it was rel
<wallyworld_> i can't recall the exact figures, but a garbo job was needed to clean up the data
<wgrant> That is entirely irrelevant to the nature of that data.
<wgrant> The volume and split are unrelated.
<wgrant> The data I just obtained from dogfood confirms my assumptions: that the case we're dropping is unused.
<wgrant> It's a case I was always suspicious about.
<wallyworld_> used now != used ever
<wallyworld_> and we are taking away a safety net
<wallyworld_> to prevent bad data
<wallyworld_> deliberately
<wgrant> OK
<wgrant> It's not bad data.
<wallyworld_> it is
<wallyworld_> because it causes confusion
<wallyworld_> and stuff to break
<wgrant> Preventing bad data is not itself a useful goal
<wallyworld_> huh?
<wgrant> The "bad data" has to have some negative effect
<wallyworld_> it does
<wgrant> The goal is to eliminate the negative effect.
<wgrant> Right
<wallyworld_> confusion, no longer being able to use the stacked branch
<wgrant> So let's stop talking about bad data, and start talking about the negative effects :)
<wgrant> Confusion: granted, we can fix that with messages on the branch page, as we do already if the stacked-on location is invalid.
<wallyworld_> or we could just prevent the bad data and not introduce a deliberate regresiion
<wallyworld_> we would need to drill down the entire stack
<wgrant> No longer being able to use the stacked branch: that's a false statement; you can't use it if it's private either.
<wallyworld_> but it's not pretending to be usable
<wgrant> So that goes back to confusion.
<wgrant> Which is very similar to the invalid stacked-on issue
<wgrant> For which we currently warn
<wgrant> With a warning on Branch:+index
<wallyworld_> we never used to warn
<wallyworld_> is that new
<wgrant> We've always warned.
<wgrant> Invalid meaning not a valid URL, or the URL points to something that doesn't exist or is outside Launchpad
<wgrant> Not meaning private
<wgrant> Yet.
<wallyworld_> nope, not in the case of public stacked on private afair
<wallyworld_> we used to just show the stacked branch as if it were public
<wgrant> 12:49:24 < wgrant> Which is very similar to the invalid stacked-on issue
<wgrant> 12:49:27 < wgrant> For which we currently warn
<wgrant> "is very similar to"
<wgrant> Not "is"
<wallyworld_> so it's new then
<wgrant> Hm?
<wgrant> That warning is not new.
<wgrant> Extending it to warn when the stacked-on is private would be new.
<wallyworld_> but we never used to warn people their public branch was stacked on a private one
<wgrant> No.
<wgrant> I never said we did.
<wgrant> I'm saying it's what we should do
<wallyworld_> hence my assertion it will be new
<wgrant> Sure, warning *on privacy violation* is new.
<wgrant> Warning on bad stacked-on in general is not.
<wgrant> So it's quite anomalous that we don't warn in the case of a privacy violation.
<wallyworld_> even recusive stacked on
<wallyworld_> ?
<wgrant> I'm not sure about that.
<wallyworld_> we may warn on bad stacked on, but perhaps not recusrively
<wallyworld_> i guess it comes down to in part philosophy - should software prevent a known bad situation
<wallyworld_> or should software regress deliberately
<wgrant> It's a tradeoff.
<wallyworld_> given the current implementation works fine, i'm not sure i see why we are doing this
<wgrant> We can maintain messy code of unbounded computational complexity that will probably never be usefully used.
<StevenK> wallyworld_: Are you done yelling at wgrant? :-)
<wgrant> Or we can not.
<wallyworld_> it's not messy
<wallyworld_> the algorithm is very simple
<wgrant> With transitively_private it's not messy.
<wgrant> With information_type it is slightly moreso.
<wgrant> Because it's not boolean.
<wallyworld_> so what i would like to see is an exception raised
<wgrant> Hahahaha
<wallyworld_> if someone attempts to change a stacked on branch to an invalid state
<wgrant> We can't do that every time.
<wallyworld_> why?
<wgrant> You may recall we discussed this for weeks.
<wgrant> The information_type of a stacked on branch could change directly.
<StevenK> stacking can change via an sftp client
<wgrant> Or it could change by its stacked-on branch changing.
<wgrant> Precisely.
<wallyworld_> sure, but i'm talking about public stacked on not public
<wgrant> So.
<wgrant> Listen to this case:
<wallyworld_> via sftp we can't detect sure
<wgrant> I have a stacking chain. A stacked on B stacked on C
<wallyworld_> but that's the case now with tp also
<wgrant> They're all public.
<wgrant> Now, the owner of B changes it to stack on D, a private branch.
<wgrant> What happens now?
<wgrant> We can't block that.
<wgrant> So A and B have to become private, or we have to let them be inconsistent.
<wgrant> Blocking is the ideal solution.
<wgrant> But it's not feasible.
<wgrant> So we have to either propagate or accept.
<wallyworld_> what happens now is A and B become private
<wgrant> Yes.
<wallyworld_> as they should
<wgrant> That is arguable.
<wallyworld_> the argument was had before tp was approved
<wgrant> And even if we decide that they should, it becomes is it worth it.
<wallyworld_> and it came out in favour of tp
<wgrant> Curtis agreed with my view on several calls, and I'm pretty sure you were there too :)
<wallyworld_> i'm not sure it was agreed to intorduce a deleibate regression
<wgrant> It may have been agreed during TP's design on the absence of useful data.
<wgrant> But we have useful data now.
<wgrant> It was.
<wgrant> It was acknowledged as a regression.
<wgrant> And I now have data which proves beyond reasonable doubt that nobody will miss it, because nobody has used it.
<wallyworld_> it's not that nobody has used it i'm worried about - it's the ability to allow a misconfigured sysem, deliberately
<wgrant> Sure
<wgrant> It's less than ideal.
<wallyworld_> because nobody uses it is even more a case not to allow it
<wgrant> But the misconfiguration is not fatal.
<wallyworld_> fsdo
<wgrant> There are lots of other misconfigurations that we permit and can't reasonably forbid.
<wgrant> And this misconfiguration is expensive to prevent.
<wallyworld_> well we can reasonably prevent this one because we already do
<wgrant> Only in a way that will probably time out if it's ever used.
<wallyworld_> nope, it's used now
<wgrant> On any project of significance.
<wgrant> It's *NOT*
<wgrant> The data I gave proves that.
<wallyworld_> the triggers are run everytime a branch changes
<wgrant> Hmmmmmm?
<wgrant> Yes.
<wgrant> But we're not talking about the triggers' overall behaviour.
<wgrant> We're talking about the particular aspect that we're dropping.
<wgrant> The unbounded cascade from the lower branch.
<wallyworld_> i don't want to redo the cascade bit
<wallyworld_> just the bit to prevent bad data
<wgrant> Hm?
<wgrant> Confused
<wgrant> The bit we're regressing on is the cascade bit.
<wallyworld_> to prevent changing a stacked on branch to userdata
<wgrant> Ah
<wallyworld_> if the stacked branch is public
<wgrant> Now, we could prevent that at the UI and API level.
<wgrant> However, we can't prevent that at the bzr level.
<wallyworld_> yes, that's all i am wanting
<wallyworld_> sure
<wgrant> OK, that is a more reasonable proposal.
<wallyworld_> we don't do anything at the bzr level now for tp
<wgrant> We do
<wallyworld_> wthat's all i've been arguing for here
<wgrant> When the stacked-on changes via bzr, it will change in the DB.
<wgrant> And transitively_private will work
<wgrant> It is no problem there because it's not trying to prevent the transition.
<wgrant> Just react to it.
<wallyworld_> ah right. well we should do the same check in the  new world also
<wgrant> We don't want to react to it.
<wgrant> wallyworld_: Which is exactly the cascading thing.
<wgrant> We have three options:
<wallyworld_> i'm not talking about cascading any changes as we do now, just now allowing the transiitin
<wallyworld_> not
<wgrant>  1) Do nothing. Let people shoot themselves in the foot if they shuffle branches around in strange ways.
<wgrant>  2) Cascade privacy changes through the tree.
<wgrant>  3) Block transitions that violate the rules.
<wgrant> TP does (2)
<wallyworld_> i'm arguing for 3, and have been for 30 minutes
<wgrant> information_type does (1)
<wgrant> (3) is impossible, since we can't block bzr changes.
<wallyworld_> why can't we block the stacked on change?
<wallyworld_> does bzr write straight to the db?
<wgrant> No, which is exactly the problem.
<wgrant> It's FS access
<wallyworld_> when in that case the triggers won't run
<wallyworld_> so how are the stacked on changes propogated?
<wgrant> The branch scanner.
<wgrant> After the push has completed.
<wallyworld_> so the branch scanner could reject the change
<wgrant> So we cannot error at the time the client sets stacked-on
<wgrant> Reject how?
<wgrant> We could fail the scan.
<wgrant> But you'll never guess what that does.
<wallyworld_> raise an oops?
<wgrant> It leaves the branch public, but puts a warning on its page :)
<wgrant> Which is exactly what I proposed.
<lifeless> wgrant: you can block stacked on changes, just reject the IO operation
<wallyworld_> leaves the stacked branch public?
<wgrant> lifeless: lol
<wgrant> lifeless: I don't feel like rewriting bzr lp-serve today
<lifeless> wgrant: why would you need to rewrite it?
<lifeless> wgrant: it has a lp VFS module already
<lifeless> wgrant: I'm not saying you should, I'm disputing your assertion that its impractical.
<wgrant> lifeless: If it's practical to do that (eg. due to restrictions around RPC from the lp-serve process), why wasn't that done from the start instead of doing it at scan time which is clearly inferior?
<lifeless> wgrant: what restrictions, lp-serve makes RPC calls all the time
<wgrant> lifeless: It's possible.
<wgrant> It's probably not practical.
<mwhudson> it's not scan time
<mwhudson> it's when the branch is unlocked
<wgrant> mwhudson: branchChanged time, sorry
<wgrant> After the end of the operation.
<mwhudson> (which is admittedly too late to tell the writer to bugger off)
<wgrant> Precisely.
<lifeless> wgrant: there was no initial requirement to patrol stacking
<lifeless> wgrant: you are adding one, which implies change.
<lifeless> wgrant: so why wasn't it done this way before, because it wasn't done before at all.
<wallyworld_> and a change i think we should do
<wgrant> lifeless: We break branches in LP when the stacked-on is invalid.
<wgrant> That's not new functionality.
<wgrant> wallyworld_: Why?
<wgrant> I'm pretty sure it's completely not worth the extra code.
<lifeless> wgrant: what do you mean break them ?
<wallyworld_> do i really have to explain myself all over again?
<wgrant> And the data confirms it.
<wgrant> It's an extraordinarily rare situation.
<wgrant> And it will become even rarer once people can't break private project setup.
<wallyworld_> rare != we shouldn't prevent it
<wgrant> Nothing goes terribly wrong if we don't prevent it.
<wgrant> And preventing it requires lots of extra code.
<wallyworld_> lots?
<wgrant> difficult to implement + very rare + largely inconsequential == not worth it
<lifeless> wgrant: what do you mean break them ?
<wgrant> lifeless: It stops scanning and throws warnings on the branch page.
<wgrant> lifeless: Which is pretty unpleasant if it could just reject the push instead.
<lifeless> it can
<lifeless> however thats not break-the-branch, its aknowledge-that-its-broken
<lifeless> we're not altering the branch data are we?
<lifeless> wgrant: wallyworld_: Can I ask, whats the context - you have a long and interesting discussion, but whats the core issue ?
<wallyworld_> lifeless: we now handle the case if a stacked on branch is changed from public -> private
<wallyworld_> any stacked branches will become private also
<wgrant> recursively
<wallyworld_> but the new world will see use allow a user to change a stacked on branch to private will no warning and the stacked branches will remain public
<wallyworld_> and the stacked branches will become unusable
<wgrant> The stacked branches will become unusable to unprivileged users, same as they would be if they were made private. The only issue is confusion if the branch says it's public but isn't usable.
<wallyworld_> so we are allowing a footgun with no warning
<lifeless> hmm
<wgrant> I gathered evidence from the database that this feature hasn't been used.
<lifeless> so, I would say, that changing other peoples branches to private is a misfeature
<wallyworld_> i'm not arguing it hasn't been used, i'm worried about the footgun
<lifeless> wallyworld_: its fixable by the same user right ?
<wallyworld_> ie allowing someone to do something dumb with no warning
<wallyworld_> yes
<lifeless> so the footgun specifically, is it:
<wallyworld_> fsvo fixabe
<lifeless>  a) the user changes the privacy setting for the stacked-on branch to private
<lifeless> or
<lifeless>  b) in a public branch they set a stacked-on which is private
<lifeless> ?
<wallyworld_> a)
<wallyworld_> we handle b)
<wallyworld_> and it's a regresion because we handle a) now
<lifeless> with the general case of c) being 'setting a stacked-on which has a different visibility rule' (and applies if per-artifact access is given to a branch but not its stacked-on)
<lifeless> do we handle c)? how do we handle it ?
<wallyworld_> triggers
<wallyworld_> transitively private
<lifeless> I thought that column was deleted ?
<wallyworld_> but it's complicated with info type
<lifeless> (being deleted)
<wgrant> lifeless: It's about to be deleted.
<wallyworld_> yes, and so we need to figure out how to prevent the footgun
<lifeless> so transitively private propogates outwards
<wallyworld_> if you mean up the stack , yes
<lifeless> a) is a propogates-outward case, c) is a propogates inwards case.
<lifeless> outward == up the stack, yes.
<wallyworld_> yes
<wgrant> information_type is currently independent, except that if you push a new public branch and it's stacked on a private one, then you're probably doing something stupid and disclosing previously private code, so we make it private.
<wallyworld_> but we can't propogate info type
<wallyworld_> we should just prevent bad things
<wallyworld_> ie veto the info type change
<wallyworld_> or stacking change
<lifeless> ok, so I'm much more worried about c than a.
<wgrant> With the propagation behaviour we're not disclosing previously private code.
<wgrant> We're just not automatically unleaking it.
<wgrant> Um, with*out* the propagation behaviour
<lifeless> a) very very few users will run into and they can restore functionality by unprivating the branch; or they can file a ticket to get a mass-migration of information type done (assuming they don't own all the branches)
<lifeless> c) users will run into all the time, because feature branches are fairly common, and contractors are also fairly common
<wgrant> This is upsetting
<wgrant> I don't get to argue with lifeless this time :(
<wallyworld_> and c) is handled now but soon won't be
<lifeless> wgrant: see, sometimes you are right ;)
<wgrant> Hm?
<wallyworld_> my view is that we should not allow the users to enter bad data
<wgrant> Isn't (c) the case we *do* handle?
<wallyworld_> if a user changes a branch to stack on a private one, the any public branches stacked on top "break"
<wgrant> That's case (a)
<wallyworld_> with transitively private, they didn't
<wallyworld_> a) is changing the privacy, not the stacked on
<wallyworld_> or am i wrong
<lifeless> ah, I think the phrasing is poor.
<lifeless> let me rephrase.
<lifeless> General case there are three branch sets to consider.
<lifeless> stacked-on branche(s) - its transitive following the pointers.
<lifeless> <the branch>
<lifeless> stacking-on-this branch(es) - anything that stacks on this branch, or transitively outwards
<wgrant> wallyworld_: Ah right, misunderstood you.
<wgrant> wallyworld_: Right, if the change an intermediate branch it still breaks.
<wallyworld_> yes
<lifeless> changing the pointer in <the branch> to point to a branch with different visibility rules (e.g. different info type) is equivalent to to changing the visibility rule on a member of the stacked-on branch set.
<lifeless> in terms of impact on the stacking-in-this branches.
<lifeless> The difference is whether its web ui (change info type) or bzr (change stacking pointer)
<lifeless> from the perspective of any member of stacking-on-this, its identical.
<lifeless> ^ does this make sense ?
<wgrant> Yes
<wallyworld_> yes
<wallyworld_> we can detect that we are going to stack onto an incompatible branch
<wallyworld_> but we are saying that we will not detect changing a branch which will cause outward brancvhes to become invalid
<lifeless> and we can detect when changing info type that there is a branch stacked on this that is incompatible (again transitively)
<wallyworld_> yes
<lifeless> so, what user stories do we have around this
<wallyworld_> but we the proposal is not to do that anymore
<lifeless> one story is 'I have a branch that should be different info-type *and* it has stacked-on branches that I do not control'
<wallyworld_> or 'has stacked on branches i do not know about'
<lifeless> right
<wallyworld_> hence i may break them inadvertantly
<wallyworld_> with no warning
<lifeless> so what can we do here
<lifeless> we could stop the user changing the setting (be that stacked-on *or* information-type = we've established identity)
<wallyworld_> yes
<wallyworld_> so with transitively private, we allowed the change because it was transparently propogated
<wallyworld_> but now it won't be
<lifeless> we could allow the branches that are no longer compatible [for users that don't meet the union of visibility rules] to become inoperable for those users
<lifeless> we could overlay the new information type into all the affected branches [which is what transitively private did]
<wallyworld_> ipractical
<wallyworld_> with allowing the branches to become inoperable, i'm arguing we should warn the user first
<lifeless> here is my suggestion
<wallyworld_> via the api, not sure how to handle
<lifeless> I think you should just do the change requested.
<lifeless> tell the user the *impact* of the change.
<lifeless> They can switch it back in a heartbeat if they made a mistake, API *or* web UI.
<lifeless> or bzr for that matter.
<wallyworld_> that works i think, at least then it's an informed decision, not something done that they dont know about, and that was my main issue
<lifeless> I totally agree that silent breakage at a temporal distance from the act is terrible
<lifeless> this will avoid that; we can, if you desire, send a warning to bzr too.
<wallyworld_> yes, and this is what i was arguing against
<lifeless> I think though, that anyone doing bzr set-stacked manually, is in a class of their own : I wouldn't bother handling it.
<lifeless> wallyworld_: this == 'silent breakage at a temporal distance from the act is terrible' ?
<wallyworld_> yes
<lifeless> righto
<lifeless> wgrant: ^ what do you think of my proposal?
<wallyworld_> i have no problem with the new limitations in behaviour
<wgrant> We can't forbid, but we can warn on the branches at both ends, which is my preference.
<wallyworld_> but the 'silent' breakage
<wallyworld_> we can forbid it, no?
<wallyworld_> sorry, i mean warn
<lifeless> wallyworld_: so, the silent-breakage via bzr case is excruciatingly rare
<wallyworld_> after they have done it
<lifeless> wallyworld_: bzr makes it super hard to change
<lifeless> wallyworld_: and lp sets the default behavior (to the branch of the default series)
<wallyworld_> no problem there. but we can reasonably handle the other cases
<wallyworld_> and i would argue that peope doing it via bzr are more expert anyway
<lifeless> warning in the web ui is easy; the API can return something with a warning too I expect.
<wallyworld_> yes
<wgrant> The API can't return a warning that people will see.
<wgrant> But in the web UI we can.
<wallyworld_> in the ui, warn as it is done as also after wards
<lifeless> wgrant: OTOH folk doing API scripting are probably changing every branch they have.
<lifeless> wgrant: e.g. the warning would be superfluous
<wgrant> wallyworld_: Well
<wgrant> wallyworld_: The branch edit page returns to the branch index page on submission.
<wgrant> wallyworld_: So a bright warning at the top of the branch index page serves both purposes :)
<wallyworld_> we can ask for a confirmation
<lifeless> wallyworld_: nix on that
<wallyworld_> in the web ui
<wgrant> I don't think we should.
<wgrant> It's trivial to revert
<lifeless> wallyworld_: confirmations for things that are easily reversible is an antipattern of usability.
<wgrant> And more expensive to add confirmation
<wallyworld_> ok
<wgrant> Like
<wgrant> It'll be a matter of clicking the edit icon
<wgrant> and clicking Public in the picker
<wallyworld_> with the api, we could introduce a 'i really want to do this parameter'
<wallyworld_> so it raises an exception unless that parameter is set
<wallyworld_> that way, it's an informed choice
<wallyworld_> since that is a case of breakage from a distance
<lifeless> ugh
<StevenK> wallyworld_: Can you peer at http://pastebin.ubuntu.com/1027943/ ?
<lifeless> expose the attribute as readonly, use an API call to set it rather than direct mutation, and have the return value describe the impact.
<lifeless> wallyworld_: ^
<wallyworld_> lifeless: that works for me
<wallyworld_> so long as the impact is communicated
<lifeless> wallyworld_: wgrant: making a branch *public* that wasn't before is something that you might justify a confirmation for.
<wallyworld_> StevenK: looking
<lifeless> because unhiding previously hidden data is infeasible
<wgrant> lifeless: Sure.
<wgrant> lifeless: But that is precisely the opposite of this case.
<wgrant> lifeless: That one is a general issue with the information type picker, unrelated to stacking.
<lifeless> actually, its tightly related :P
<StevenK> Hmm
<wgrant> Oh?
<lifeless> which is that fixing a public->private change involves a private->public change :)
<wgrant> Uh
<wgrant> yes
<wgrant> But that's barely relevant.
<wgrant> It's a special case that will be fixed by the general fix.
<wgrant> There's nothing even really special about it :)
<StevenK> Lots of test fallout of forbidding open/delegated teams to subscribe to private branches. :-(
<wgrant> Yeah
<wgrant> We really need to split the two types of teams up better.
<wgrant> So they're more obviously different types of objects.
<StevenK> I think ec2 has found most of them
<wgrant> makeTeam and makeBadgeSeekersGroup
<StevenK> Hahaha
<StevenK> Oh, of course it has, the instance has been up for 4:04
<StevenK> wgrant: Some of the fixes are trivial
<wgrant> All of them should be.
<StevenK> test_branchnamespace breaks in strange ways
<StevenK> (When I make the 3 teams MODERATED)
<wallyworld_> StevenK: i'm not sure what caused that error ottomh
<wgrant> Hmm
<wallyworld_> i'd need to debug it
<wgrant> StevenK: What's the diff?
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/branch-subscribe-aag/+merge/108881 is the MP
<wgrant> StevenK: So, the branch owner is trying to subscribe a private team that they can't see.
<wgrant> I'm surprised that ever worked.
<wgrant> The test is clearly on crack.
<StevenK> There are a *bunch* of them
<wgrant> Replace person_logged_in(...) with admin_logged_in()
<wgrant> Problem solved
<wgrant> (the write permission is not relevant in that test)
 * wallyworld_ has to do school pickup
<StevenK> It isn't relevant to check that the person subscribing can see the team?
<wgrant> StevenK: Look at the test.
<wgrant> StevenK: It is testing that an anonymous user can't see a private subscriber.
<wgrant> Not that a private subscriber can or can't be created.
<wgrant> So the principal behind the creation is irrelevant.
<StevenK> wgrant: That issue can't affect prod?
<wgrant> StevenK: Only if someone is subscribing someone that they don't have View on, which I don't think is allowed since it discloses the team name.
<wgrant> But it's possible that someone with LimitedView can subscribe, I suppose.
<StevenK> It will oops
<StevenK> So we can find out
<wgrant> Right.
<StevenK> Right, 18 failures, let's see how few I fixed
<StevenK> Hmmmmm. A test creates a private team and then creates a branch with that owner and it blows up too.
<wgrant> StevenK: The branch owner is irrelevant.
<wgrant> StevenK: The current principal is important.
<StevenK> It just calls into the factory twice, I'm not sure which principal is involed.
<StevenK> *involved
<StevenK> wgrant: I've thought a problem -- create a project and have an open team as a reviewer. Create a private branch and propose it. Boom.
<wgrant> StevenK: Reviewers aren't subscribed unless the branch is private.
<wgrant> And that will largely go away now we have APGs.
<StevenK> wgrant: Yes, and you can't subscribe an open team to a private branch ...
<StevenK> And if the reviewer is an open team ...
<wgrant> StevenK: Right, that's stupid and should be forbidden :)
<wgrant> It doesn't make sense.
<wgrant> Forbidding something that shouldn't be permitted is not a bug :)
<StevenK> Yes, so I was going to chase that rabbit hole for a bit.
<StevenK> Trying to figure out where reviewers are in the DB
<StevenK> Hmmm, can't see it in the UI either currently, that's disappointing
<wgrant> What do you mean?
<StevenK> wgrant: So when I propose a branch for LP, a team is set as the reviewer. I'm trying to figure out where that is in the DB.
<wgrant> StevenK: branch.reviewer
<StevenK> I discounted that because I thought it was per-branch
<wgrant> It is per-branch.
<wgrant> The default reviewer for an MP is the target branch's default reviewer.
<StevenK> Which is in .reviewer for that branch, right.
<StevenK> wallyworld: ^ Distracted from my other branch by a different problem.
<StevenK> People that have set an open team as a reviewer need to be shot.
<wallyworld> StevenK: i think so
<StevenK> wallyworld: About which?
<wallyworld> the default mp reviewer is the branche's default reviewer
<lifeless> StevenK: again, its our oversight
<lifeless> s/again, //
<StevenK> lifeless: Which is?
<lifeless> failing to consider the implications of open teams on integrity
<StevenK> Yup.
<StevenK> I'm going to bring it up on the call tomorrow.
<lifeless> StevenK: you're aware of all the other cases we've fixed so far?
<lifeless> StevenK: we should probably just do an audit
<lifeless> ppas, project roles, ....
<StevenK> lifeless: I was involved in both of them closely.
<StevenK> lifeless: However, these situations are not as cut-n-dried as PPAs and project roles.
<StevenK> There are arguments both ways, which there wasn't for the two earlier cases.
<wallyworld> wgrant: https://code.launchpad.net/~wallyworld/launchpad/sharing-job-fflag-protection/+merge/109072
<lifeless> StevenK: well, there weren't for ppas, because sudo on machines, but the rest there are :)
<lifeless> StevenK: if you consider tarmac, this case becomes roughly === ppas
<wgrant> lifeless: branch.reviewer is different, since most projects don't use Tarmac, so there are uses where an open team doesn't mean root for everyone.
<wgrant> lifeless: For PPAs there's no use that doesn't mean root for everyone.
<wgrant> wallyworld: That's the third feature flag for the same thing.
<wallyworld> not really
<wallyworld> i reused one
<wgrant> wallyworld: All the other stuff is guarded by one of the two feature flags that you introduced two days ago.
<wallyworld> for this
<wgrant> And, critically, we don't want to turn writing on before we confirm that these work on production.
<wallyworld> different semantics
<wgrant> So turning this on only when we turn writing on is less than ideal.
<wgrant> I'd have all the job creation under a flag like disclosure.that_job_stuff.enabled
<wgrant> Rather than 3 with misleading names.
<wallyworld> misleading?
<wallyworld> to whom?
<wallyworld> for this branch, i just wanted a fast way to disable the jobs
<wallyworld> so we can deploy
<wallyworld> without having to add another flag
<wallyworld> the closest existing flag is perhaps disclosure.legacy_subscription_visibility.enabled but that one is for when we want unsubscribing to revoke access. the jobs are for the opposite direction so different
<StevenK> lifeless: Tarmac and BMQs are a compelling argument indeed.
<wallyworld> wgrant: so which flag did you want me to use if you disagree with the one i chose. remember i just want something quick to stop the jobs running on prod
<wgrant> wallyworld: I'd probably just use disclosure.legacy_subscription_visibility.enabled. But I guess we use writable temporarily just to disable it, you're right.
<wallyworld> i can clean up next branch
<wallyworld> i just want to unblock deployment
<wallyworld> which has been blocked for over a day :=(
<wgrant> wallyworld: I'm on a call right now, but is that all the callsites for both types of jobs?
<wallyworld> wgrant: unless i missed anything but i double checked
<wallyworld> there are 2 places in the service but the methods they are inside are protected
<wgrant> wallyworld: r=me, thanks.
<wallyworld> thank you
<lifeless> wgrant: can you remind me tomorrow to send a summary to l-dev ?
<wgrant> lifeless: Sure.
<lifeless> also ahahaha https://lp-oops.canonical.com/oops.py/?oopsid=61493187d8286292a72f12956de8b3cb#longstatements
<lifeless> branch fti query strikes again
<lifeless> on +addseries
 * czajkowski peers at wgrant are you doing stuff to lp atm 
<wgrant> lifeless: Ew
<lifeless> https://github.com/cjdelisle/cjdns/blob/master/rfcs/Whitepaper.md
<lifeless> elmo: ^ may make you laugh, may make you cry
<czajkowski> https://lp-oops.canonical.com/oops.py/?oopsid=61493187d8286292a72f12956de8b3cb  am getting these while doing translations imports
<jml> hello
<jml> I'd like to QA https://bugs.launchpad.net/launchpad/+bug/1006295
<_mup_> Bug #1006295: Archive's 'suppress_subscription_notifications' attribute is called 'commercial' in the database <qa-needstesting> <tech-debt> <Launchpad itself:In Progress by jml> < https://launchpad.net/bugs/1006295 >
<jml> I *think* the right thing to do is to set the attribute on an archive on qastaging and then perhaps check that 'commercial' was set in the database
<jml> can someone confirm?
<wgrant> jml: I'd just check that you can set it and read it.
<wgrant> If it reads back correctly, it probably wrote to the database, and probably to the commercial column :)
<jml> ok.
<jml> I guess I'll just use one of my own crappy testing PPAs
<jml> Python sucks.
<jml> Let's rewrite everything in Clojure.
<wgrant> Good plan.
<jml> You know what else sucks?
<jml> transparent network operation syntax
<wgrant> Indeed
<jml> Hmm.
<jml> neither suppress_subscription_notifications nor commercial appear in the PPA dict for me (for a PPA I own)
<wgrant> jml: Are you sure you're using the right API version?
<jml> wgrant: no.
<wgrant> It's probably only exported in devel, not 1.0
<jml> Hmm.
<wgrant> Launchpad.login_with('fooconsumer', 'qastaging', version='devel')
<jml> wgrant: thanks.
<jml> wgrant: hmm. no change.
<wgrant> jml: Perhaps you need to remove your WADL cache
<wgrant> In ~/.cache/launchpadlib or ~/.launchpadlib
<jml> wadl. cache.
 * jml parties like it's 1999
<wgrant> It's on https://api.qastaging.launchpad.net/devel/~wgrant/+archive/ppa, so the server knows about it...
<wgrant> Heh
<jml> HHmm.
<jml> getPPAByName doesn't work the way I think it should.
<jml> Or maybe createPPA doesn't.
<wgrant> jml: Are you calling lp.me.somemethod?
<jml> wgrant: yes.
<wgrant> That doesn't work
<jml> ffs
<wgrant> It'll just return the person instead.
<jml> thanks.
<jml> I'd noticed.
<wgrant> lp.people[lp.me.name].somemethod
<wgrant> will work
<wgrant> If you're not dead yet.
<jml> \o/
<jml> wgrant: sanity check this script for me?
<jml> http://paste.ubuntu.com/1028403/
<jml> output: http://paste.ubuntu.com/1028404/
<jml> is there a way I can get an alert when a revno gets deployed?
<wgrant> jml: I can comment in the bug.
<jml> wgrant: thanks.
<wgrant> But probably in about 13 hours.
<jml> cool.
<stub> wgrant: Did you see anything horrible happen with the GIN index on staging?
<wgrant> stub: It seems to not be broken.
<stub> Might do the rest of them then
* rick_h changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<jml> hello
<jml> can someone in ~launchpad please claim a number for me in lp:~launchpad/+junk/dbpatches
<rick_h> looking jml
<jml> rick_h: thank you.
<rick_h> jml: looks like I need a comment for it?
<jml> rick_h: Rename Archive.commercial to Archive.suppress_subscription_notifications (bug #1006295)
<_mup_> Bug #1006295: Archive's 'suppress_subscription_notifications' attribute is called 'commercial' in the database <qa-ok> <tech-debt> <Launchpad itself:In Progress by jml> < https://launchpad.net/bugs/1006295 >
<rick_h> jml: ok, think I got that
<jml> rick_h: thanks.
<jml> I got a lot of spew when I ran 'make newsampledata': http://paste.ubuntu.com/1028563/
<jml> mostly of the form, "NOTICE: there are circular foreign-key constraints among these table(s)"
<jml> is this my fault?
<jml> Also, there appear to be changes to ArchivePermission and SPPH after running 'make newsampledata'. I wasn't expecting those. Should I not be doing this from db-devel?
<rick_h> sorry jml, no idea there. Have yet to do a db patch myself. stub is the master, you know?
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugs: 3.47*10^2
<rick_h> oh hai jcsackett
<jcsackett> morning, rick_h.
<rick_h> morning, -ops
 * jcsackett idly wonders if the 1400 line diff in the MP queue is a harbinger of the day to come.
 * jml sends an email
<stub> jml: You can ignore that I think. That make sampledata stuff is an old hack and has never been pretty.
<jcsackett> sinzui: i need a refresher from last night. which of the many cards was the one wallyworld was looking at you suggested i try for today? my mind is sort of mush, evidently.
<sinzui> jcsackett, https://bugs.launchpad.net/launchpad/+bug/999596
<jcsackett> sinzui: ah, dig. yes, i should totally be able to fit that in today.
<jcsackett> huh. looks like william is already doing it though.
<sinzui> jcsackett, I think the complication is that the vocabs and personset have competing implementation
<sinzui> jcsackett, that is my mistake. No one is working on it
<jcsackett> sinzui: dig. i'll assign myself then.
<cjwatson> oww, soyuz/doc/archive.txt makes me feel physically ill
<cjwatson> 2762 lines of solid doctest
<sinzui> cjwatson, I bet half of it is tested in a unit test, but removing the duplicate parts topple the doctest
<cjwatson> Bingo
<cjwatson> And so it requires actual thought to dissect.  Grr
 * jml wants to take a swipe at it.
<cjwatson> I'm at -253 on it so far
<cjwatson> Though I strongly suspect there'll still be some left when I'm done
<bdmurray> Could I get my merge proposal review? https://code.launchpad.net/~brian-murray/launchpad/bug-912137/+merge/109045
<jcsackett> bdmurray: i'm in the middle of running to somewhere that has better internet than my currently failing home spot. i'll check it as soon as i'm moved.
<bdmurray> jcsackett: okay, thanks!
<jml> stub: https://code.launchpad.net/~jml/launchpad/db-rename-archive-commercial-to-suppress/+merge/109179
<stub> jml: The horrible hack to allow both the old and new column name has landed?
<jml> stub: indeed it has.
<jml> stub: and appears to work.
<stub> I feel dirty
<jml> stub: it's not deployed yet.
<stub> r=stub, with a comment that the code needs to be deployed before the patch can go live.
<stub> Easiest way to guarantee that is to not land it until after deployment.
<jml> stub: that'll probably happen, since I need lifeless's patch before landing anyway, right?
<stub> jml: No, you only need one of our approvals.
<jml> stub: ah ok. I'll wait until after I hear about the deployment then.
<stub> That would be FDT tomorrow I guess
<cjwatson> NDT, wouldn't it be?
<stub> er, yer
<jml> sorry, I'm not familiar with those acronyms.
<stub> fast downtime, no downtime
<cjwatson> fastdowntime -> db deploy, nodowntime -> code deploy
<jml> oh right. sorry.
<jml> yeah, NDT
<jml> stub: oh, the patch number ends with -0. Is that a problem?
<stub> No problem. Numbers currently have no meaning besides ordering and uniqueness.
<jml>  stub: oh, the patch number ends with -0. Is that a problem?
<jml> stub: ah ok. cool.
<jcsackett> bdmurray: are you still around? took me a good bit longer to get setup in new location than i anticipated, but i've looked at your MP and have some comments/questions.
<bdmurray> jcsackett: yes still here
<jcsackett> bdmurray: awesome. so, your modification does change the sort order, but it has the side effect of putting any user that is subscribed via the "Subscribe someone else" button at the bottom of the list. I *think* this is probably why prepend was being used--so in long subscriber lists a user could still see the person they added was in fact added.
<bdmurray> jcsackett: ah, how did you go about determining the side effect?
<jcsackett> bdmurray: i just used the "subscribe someone else" button above the subscribers list. given the change was commented as "add user to start of list" i wondered if that wasn't intentional.
<bdmurray> jcsackett: oh right, running it makes sense.  Do you any suggestions are where to go from here?
<jcsackett> bdmurray: i do. i think you can probably update addSubscriber to take an argument named, say "add_new" that tells addSubscriber whether it's adding something being loaded from a list, and should be appended, or is adding a brand new person from the UI, and should be prepended.
<jcsackett> you could default that to 'false' and then just update the call that's bound to the "Subscribe someone else" button to set it to true.
<bdmurray> jcsackett: okay, thanks I'll have a go at it in a bit
<jcsackett> bdmurray: awesome. i'll update your MP with these notes. ping me if you run into any problems--i'm happy to help. :-)
<jml> james_w: I've ec2 landed your branch
<james_w> thanks
<james_w> I hope it stays landed
<sinzui> jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/unlock-sprite/+merge/109206
<jcsackett> sinzui: i just opened the MP, actually. :-)
<sinzui> sorry about the svg
<jcsackett> no worries.
<jcsackett> sinzui: r=me. looks good, thanks. :-)
<sinzui> fab
<bdmurray> I forget what is username / password for launchpad.dev?
<bdmurray> jcsackett: ^?
<rick_h> bdmurray: you can use mark@example.com with test
<bdmurray> jcsackett: I've updated my branch
 * jcsackett looks
<jcsackett> bdmurray: r=me. do you have landing rights, or do you need an assist?
<bdmurray> jcsackett: its been a long time, so I could use some handholding landing it.  I mean I'd like to see if I still can.
<jcsackett> bdmurray: dig. well, first off, are you set up for ec2 test/land? (https://dev.launchpad.net/EC2Test)
<bdmurray> jcsackett: yes, I should be set what is the proper comamnd to test and land?
<jcsackett> bdmurray: If lp-dev-utils is on your path you just use "ec2 land".
<sinzui> bdmurray, that would be "ec2 land https://code.launchpad.net/~brian-murray/launchpad/bug-912137/+merge/109045"
<sinzui> jcsackett, are you available for a pre-imp talk. I am trying to teach Lp that projects can go uncommercial
<jcsackett> I am actually in a coffee shop since wifi died at home. I don't think I could hear you over background noise.
<jcsackett> I am told wifi should be restored in about an hour?
<jcsackett> Well, Internet, not wifi.
<sinzui> okay
<bdmurray> tests are faster now right?
<sinzui> bdmurray, yui tests are, the suite itself takes 6 hours and must be run on ec2, the buildbot twice
<sinzui> bdmurray, expect a day for it to arrive on qastaging
 * sinzui looks at diff
<sinzui> bdmurray, I suggest an alternate plan since you fixed a js-only issue
<sinzui> bdmurray,  run ./bin/test -vvc --layer=YUI
<sinzui> if that layer passes, we can submit your branch directly to PQM
<bdmurray> sinzui: I'm not in a hurry and want to make sure my ec2 setup is good so I think I'll let it run
<sinzui> okay
<jcsackett> sinzui: i am home, and i appear to have full connectivity. do you still want to chat, or have you moved on?
<sinzui> oh, fab, yes lets talk
<jcsackett> sinzui: i've started a hangout.
<sinzui> okay
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
#launchpad-dev 2012-06-08
<lifeless> http://arstechnica.com/security/2012/06/flame-crypto-breakthrough/
<lifeless> cjwatson: ^
<mwhudson> i.e. it was NSA wot did it?
<lifeless> something like
<mwhudson> i like the first comment:
<mwhudson> It is going to be both funny and sad if it turns out this thing was written by some 15 year old in his parent's basement.
<cjwatson> I wonder how well the maths behind that will transfer to the SHA family :-/
<wgrant> cjwatson: SHA-1 perhaps, but isn't SHA-2 fairly different?
<wgrant> But I guess if it's an entirely novel form of the attack it may have some relevance.
<cjwatson> They're similar enough for the multicollisions approach to still apply.
<cjwatson> All the current hash families aside maybe from some elliptic curve ones have roughly the same general shape.
<cjwatson> Which isn't to say that it wouldn't be quite a bit more world-class maths to transfer it ...
<wgrant> Indeed.
<cjwatson> My general takeaway from hash developments over the last ~10 years is that every deployed system needs a structure for moving to new hashes, anyway.
<cjwatson> So meh :-)
<cjwatson> Pretty surprised Microsoft is still vulnerable to MD5-based attacks, though.
<cody-somerville> I'm curious. Might be a stupid question but why would the flame malware have this new world class crypto breakthrough in it anyhow? AFAIU, even though it's been discovered how to crack md5 it still takes a little while plus why would you need the crypto breakthrough in the malware yourself? Wouldn't you just use the math to create the fake cert then you use that to sign stuff?
<mwhudson> i was wondering that
<mwhudson> i think that the deduction that a new approach was used by looking at the way the form the colliding data has
<mwhudson> reading between the lines of http://www.cwi.nl/news/2012/cwi-cryptanalist-discovers-new-cryptographic-attack-variant-in-flame-spy-malware
<mwhudson> so there's not a novel md5 collision generating algorithm in flame
<cjwatson> Ah, it's only Terminal Server that's still MD5y
<wgrant> The collision code isn't in flame.
<wgrant> They were just able to determine it was novel from the cert it uses.
<wgrant> The known attacks are fairly distinctive :)
<wgrant> wallyworld: Thanks for cleaning that up.
<wallyworld> had to be done
<cjwatson> wgrant: Have you had a chance to look at my queue-api branch?
<wgrant> cjwatson: A little. Have you considered the performance impact of the additional exported properties on PackageUpload?
<wgrant> Unless they're preloaded they will likely make it impossible to call getPackageUploads
<lifeless> cjwatson: I want to talk to you about deb deltas
<lifeless> cjwatson: are you up for a while, or is your circadian rhythm not ?
<cjwatson> wgrant: I haven't.  Advice welcome
<cjwatson> lifeless: Meh - I'm kind of here but mostly in a "hacking until my brain lets me go back to sleep" kind of way
<cjwatson> So maybe now isn't the ideal time for deep thought
<lifeless> cjwatson: ok, ping me when we cross paths and you're up for deep though
<lifeless> t
<cjwatson> lifeless: OK - can you send me an e-mail reminder?
<lifeless> yes, when I'm not quite so crook
<cjwatson> (Then at least it can get lost in the depths of my inbox as well as the depths of my brain ...)
 * cjwatson nods
<StevenK> lifeless: You have the entire weekend to be sick now :-P
<StevenK> lifeless: Do you know anything about stub's GIN work?
<cjwatson> wgrant: Is DecoratedResultSet(query, pre_iter_hook=...) the mechanism for this?
<cjwatson> I find storm a little difficult to navigate unassisted.
<StevenK> Welcome to the club
<bigjools> it's not the most pleasant code
<StevenK> Everytime I get a traceback in the depths of Storm, I tend to yell for help
<wgrant> cjwatson: Yeah.
<wgrant> Note that DRS is one of our layers on top of Storm.
<wgrant> Storm rejected it.
<wgrant> StevenK: What about GIN?
<StevenK> wgrant: DB r11636 is up next -- [r=stub][bug=306201][incr] Switch BugTaskFlat.fti index from GiST to GIN
<StevenK> But then DB r11640 says --- [r=stub][bug=1007333] pgstattuple doesn't support GIN indexes, so don't do that
<wgrant> StevenK: Bah, I typoed the deployment request
<wgrant> Should be 11639
 * wgrant fixes.
<StevenK> wgrant: Haha, what did you have?
<wgrant> 11640
<StevenK> WCPGW
<wgrant> Nothing at all, since I believe 11640's was probably applied live.
<wgrant> But I'm not sure.
<wgrant> Part of 11636 was as well, but not all.
<StevenK> We can probably check that and make db-stable's deployment report a bit happier?
 * StevenK stabs wallyworld.
<StevenK> Just when I want him, he buggers off.
<StevenK> wallyworld_, wgrant: Test fixes for branch-subscribe-aag and checking that the reviewer isn't an open team for private branches: http://pastebin.ubuntu.com/1029885/
 * wallyworld_ looks
<wallyworld_> StevenK: test_open_reviewer_private_branch, i'd like to see test_closed_team_reviewer_private_branch also even thoug the functionality is implicitly tested elsewhere, it would be good to have the two matching tests for this together
<wallyworld_> StevenK: also, add a comment to _acceptable_to_give_visibility
<lifeless> wgrant: I don't know that DRS was ever offered to storm
<wgrant> http://comments.gmane.org/gmane.comp.python.storm/1319
<lifeless> oh right
<lifeless> so yes, I remember, and disagree with them :)
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1029896/ should address both comments.
<wallyworld_> StevenK: thanks, looks good
<wallyworld_> StevenK: i have to go to soccer - i can +1 your mp when i get back
<cjwatson> wgrant: Well, http://paste.ubuntu.com/1029949/ demonstrates the problem, but adding more load_referencing/load_related doesn't seem to reduce the query count that the test complains about, so I must be doing something wrong ...
<StevenK> wallyworld_: It already was, I've tossed it to ec2
<cjwatson> (The test output is http://paste.ubuntu.com/1029951/, which does look like a moderately plausible set of queries that might be happening here)
<wgrant> cjwatson: Those load_relateds should be sufficient (Reference columns look up by primary key, so if an object is already in Storms cache it won't cause a DB query), but the load_referencing things aren't, since the problematic properties issue queries directly or use a ReferenceSet or SQLMultipleJoin, none of which can be cached in Storm currently.
<wgrant> So you need to use @cachedproperty
<wgrant> And populate the property cache explicitly with the result of load_referencing
<wgrant> And then sob.
<wgrant> cjwatson: eg. search for load_referencing in lib/lp/registry/vocabularies.py
<cjwatson> ... I think I need sleep, then caffeine, then food, then beer, *then* to attack this
<cjwatson> But thanks, that's exactly the kind of reference I was looking for
<wgrant> Heh
<wgrant> cjwatson: PersonSet._getPrecachedPersons may also be of interest.
<wgrant> cjwatson: It's one of the earlier cases of this, so it's a bit terrible but roughly a sort of correct idea.
<jam> Hey all, I'm getting a timeout trying to submit a merge proposal: The following errors were encountered: Timeout error, please try again in a few minutes.
<jam> is there a downtime I'm unaware of?
<wgrant> jam: No, that's a bug.
<wgrant> What's the OOPS ID?
<jam> wgrant: no oops
<wgrant> (perhaps check the AJAX log in the top right corner.
<jam> just failure
<wgrant> There is an OOPS
<jam> and it just succeeded :(
<wgrant> It might just not be shown.
<jam> I reproduced it 3 times, but yeah, no log of the oops for me to go check.
<jam> I'll try back, but I think that will wipe ajax stuff
<jam> yeah
<adeuring> good morning
<jtv> Hi adeuring
<adeuring> hi jtv
<stub> Test fix mode due to  lp.services.job.tests.test_celeryjob.TestRunMissingJobs.test_find_missing_ready on db-devel
<stub> Pretty certain it is spurious - no sampledata landings in the timeframe and test passed on devel
* adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: adeuring | Firefighting: - | Critical bugs: 3.47*10^2
 * jml feels dumb for not running tests on db-rename-archive-commercial-to-suppress last night
<jml> hmm. last build failed. looks like a network error.
<jml> https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/2028/steps/shell_5/logs/stdio
<gmb> adeuring, Do you have time to review https://code.launchpad.net/~gmb/launchpad/bug-1009712/+merge/109315 ?
<adeuring> gmb: sure
<gmb> Thanks
<adeuring> gmb: the feature retlaed changes look good. but what is test_noise supposed to do?
<gmb> adeuring, Wups, that wasn't meant to be there. I'll remove it.
<adeuring> gmb: ok, then r=me
<gmb> Danke.
<jml> grr.
<jml> database patches aren't hard, but iterating on them is a pain.
<stub> Is the estimated time before a build completes based on all builds, or successful builds?
<wgrant> stub: Successful
<rick_h> rvba: oops sorry, thought you guys were using 3.5 for some reason.
<czajkowski> rick_h: you're up early
<rick_h> naw, this is pretty normal for me
<rick_h> just normally quiet reading email and MP stuff for the first bit :)
<rvba> rick_h: no worries.  Thanks for the review.
<jml> may I have another db patch number please?
<jml> https://code.launchpad.net/~jml/launchpad/remove-archive-commercial/+merge/109332 is an easy MP, btw.
<lifeless_> jml: you can self allocate these days
<jml> lifeless_: I'm not in ~launchpad
<jml> lifeless_: so I don't have write access to the branch in which patch numbers are allocated.
<stub> jml: Gimme a short comment for the log
<jml> stub: "Add column to Distribution controlling who can create private PPAs"
<stub> jml: 2209-22-1
<jml> stub: thanks.
<ev> are there any plans to support OAuth 1.0a in launchpadlib? Specifically, oauth_callback.
<jml> ev: not that I know of.
<ev> jml: cheers
<jml> db patch up for review: https://code.launchpad.net/~jml/launchpad/db-distro-level-ppa-privacy/+merge/109336
<jml> lifeless_: if you're still kibbitzing, you might be interested in that.
<StevenK> jml: You want stub for that
<jml> StevenK: sure. but lifeless has been involved in implementation discussions, and might want to be sure that I'm doing what we agreed on, rather than what I think we agreed on
<wgrant> jml: You want to talk to sinzui.
<wgrant> jml: That DB patch is probably incorrect.
<jml> wgrant: I thought I had.
<jml> anyway.
<jml> wgrant: I'll talk to him again.
<wgrant> It's not really anything to do with the distro.
<jml> it is now :)
<wgrant> How?
<wgrant> We generally grant private PPAs to people with commercial subscriptions.
<wgrant> Not because they are vaguely related to some distro.
<wgrant> It's not a security-sensitive option.
<wgrant> So it makes little sense to have it as a distro-delegated operation.
<wgrant> Similar to the way we now let anybody with a commercial subscription create a private team.
<wgrant> Which implicitly has private PPAs.
<jml> Grr.
<jml> a) lifeless said "make a new celebrity". I don't know how it became "add a column" in my notes.
<jml> b) I emailed launchpad-dev about this 5 weeks ago. No one replied.
<wgrant> jml: The thread about the technical bits of protecting it differently in the Zope security model?
<wgrant> I don't recall a thread about how it should be modelled.
<jml> wgrant: right, the one titled "permissions for creating private objects"
<jml> (there was also the email to stakeholders in Jan asking if there'd be "any problems with a PPA having, say, 100,000 subscribers?", but that's a different badger)
<jml> sinzui: are you around?
<wgrant> jml: Well, that thread didn't go near deciding who can do it. It was around the probably impossible problem of doing it cleanly in the Zope security model.
<wgrant> I suspect nobody replied because anybody with an opinion already knows it's hopeless.
<jml> wgrant: it opened describing our planned approach. if it was incorrect, that was the time to tell us so.
<wgrant> Oh
<wgrant> Indeed, the second paragraph of the first email mentions the celebrity approach.
<wgrant> Missed it among the other 20 paragraphs of hopeless Zopeness :)
<wgrant> Sorry.
<wgrant> So, a celebrity is a lightweight and potentially sensible approach for now.
<wgrant> A DB column is not.
<jml> I put it first (just after what we're trying to achieve) because it was most important (after what we're trying to achieve)
<wgrant> But sinzui is Lord of Entitlement.
<jml> and is apparently not around.
<wgrant> It's not quite 9am for him yet.
 * czajkowski peers at wgrant how are you still awake !
<wgrant> I'm waiting to talk to a US user who is making several million API requests a day.
<czajkowski> wgrant: ah lovely that reminds me
<jml> sinzui: I'm getting some food. ping me when you're around and want to talk about creating private PPAs.
<cjwatson> wgrant: queue-api: I've converted some of the expensive properties into methods, and arranged to preload everything else.  I think that should address your performance concerns.
<cjwatson> (Where "everything else" was just the referencing PackageUpload{Source,Build,Custom} objects.)
<jml> sinzui: ping
<sinzui> hi jml
<jml> sinzui: hi
<jml> sinzui: I'd like to change LP so I can create private PPAs without being in commercial_admin
<jml> sinzui: consensus from other LP devs is that I should create a new celebrity team that includes everyone who is allowed to do this.
<sinzui> jml: Yes, I think that meets your needs.
<jml> sinzui: ok. is this in line with other entitlement / privacy work going on?
<sinzui> jml: its orthogonal. for entitlement...I think any user/team that maintains a project  with a  commercial subscription can create a p3a.
<sinzui> I do not think your rules to maintain the system should be connection to user entitlement
<jml> sinzui: well, then why bother making a celebrity team? why don't I just query to see if they maintain a project with a commercial subscription?
<jml> sinzui: in fact, why don't I just assume that if they have a private team they can make private PPAs?
<sinzui> jml: private teams can only have p3as. maybe I misunderstand you
<jml> hmm. thinking...
<sinzui> jml: but your statements are true. both will work
<sinzui> We do not dismantle private teams when subscriptions expire
 * sinzui has no idea to handle that case
<jml> sinzui: so, AIUI, currently if you're creating a PPA under a private team you also need to have launchpad.Commercial to make it private.
<jml> sinzui: which is silly, I think.
<sinzui> oh, jml, there is a method, maybe on personset that will tell you of the user has commercial privs
<jml> sinzui: sorry, I shouldn't have said launchpad.Commercial, I should have said "in commercial admin"
<sinzui> jml. I hope not...that would be a bug. private teams can not have public ppas
<jml> sinzui: well I think it's just that you're not allowed to create any PPA. Let me check
<sinzui> that too would be a bug. Any team can have a ppa, private teams are p3a. public teams may only have p3a when someone intervenes.
<jml> ok.
<jml> I think it is a bug (validatePPA doesn't bother checking the privacy status of the owning team, it just checks admin & commercial_admin)
<jml> sinzui: so now we need some way to say who is allowed to create private PPAs on public teams
<jml> sinzui: and I guess that can be a celebrity
<jml> sinzui: but if there's already some mechanism in Launchpad saying "they've paid for super-powers" then I'd rather use that.
<jml> anyway
<sinzui> ah, I agree
<sinzui> if the user paid for privs, he should see the checkbox or API to make the archive private
 * sinzui is still hunting for the method that checks if the user have paid for privs
<jml> sinzui: of course, this sort of assumes commercial is all-or-nothing.
<jml> sinzui: I guess that's an OK assumption.
<sinzui> jml: please assume that
<sinzui> we want this very simple for users and us to understand
<sinzui> if you find a contradiction, we need to fix it
<jml> sinzui: ok, thanks. let me know if you find that API
<sinzui> jml, person.checkAllowVisibility() is checks if the user has permission to make things private. It is used to know when to show users the the team visibility field. It would also apply to p3as
<jml> sinzui: thanks.
<jml> sinzui: please verify: https://code.launchpad.net/~jml/launchpad/db-distro-level-ppa-privacy/+merge/109336/comments/234987
<sinzui> jml that is correct with one subtle point you might find in the code...I think we mean users = team admin in some cases.
<sinzui> user ~= team admins
<sinzui> I do not think team members can create ppas
<jml> sinzui: I think they can.
<jml> hmm.
<jml> maybe not.
 * jml can't seem to log in to staging.
<jml> sinzui: qastaging uses a different db to prod, right?
<sinzui> jml: yes, and it is months older :(
<jml> sinzui: you can create PPAs over the API without being a team admin.
<jml> sinzui: but not in the UI
<sinzui> I think that contradicts ui
<sinzui> yep
<sinzui> That is a bug, but I do not know how disruptive it is to fix
<jml> lbyl bites again.
<sinzui> I do not think there is a bug about this. I image it would only exist if teams felt that team privs were being abused
<jml> well, it's something we might fix as a follow up, or as a drive-by, but we'll preserve the existing behaviour all things being equal
<sinzui> jml: I agree. I would not fix this without an audit of the db
<jml> sinzui: how come?
<sinzui> Canonical and other commercial users may be taking advantage of this bug.
<sinzui> jml: I don't this I could audit this case. We do not know who created the archive. So I think this issue requires a conversation with PES and openstack
<jml> sinzui: fair enough.
<ev> Before I go too far down the rabbit hole, what's the suggested approach for logging into Launchpad with existing OAuth credentials from a web client?
<ev> Should I use Launchpad.login_with and a CredentialStore subclass that provides the already obtained oauth_token and oauth_token_secret?
<ev> the existing documentation creates a Launchpad() object from the default constructor with now-missing arguments (https://help.launchpad.net/API/launchpadlib)
<ev> err rather: https://help.launchpad.net/API/launchpadlib#Authenticated_access_for_website_integration
<sinzui> ev: I am not very familiar with the OAuth code, but I write a lot of API scripts. I think the subclass approach is right so that scripts can be adapted or be written support multiple stores
<ev> sinzui: multiple stores? This is only ever going to use cookies. Or have I misread?
<ev> back in a bit, it's toasting the new office time
<sinzui> ev, As a tester, I would like to switch which cookies I am using.
<nigelb> rick_h: Hey, you around?
<rick_h> nigelb: sure, what's up?
<nigelb> rick_h: Hey, when you worked on SA, what did you guys do for db migration?
<rick_h> we used sqlalchemy-migate but I
<rick_h> 've got on my todo to port my app over to alembic
<nigelb> ah
<rick_h> and squash down the migations/etc
<nigelb> rick_h: thanks! I look up both of them :)
<rick_h> nigelb: yea, definitely suggest alembic, it's done by the SA author so it'll be kept in sync/etc much better
<rick_h> nigelb: and he looked at django's south so should be a bit 'learned' from the competition
<nigelb> heh
<bdmurray> sinzui: so I tried testing via bin/test -vvc --layer=YUI per your suggestion and I keep getting tests timing out
<sinzui> That is odd. The tests run in a few minutes for me
<sinzui> maybe you are missing dep
<rick_h> bdmurray: what's the exact error? It might be that you didn't run the build steps or failed to setup
<rick_h> bdmurray: try manually running make jsbuild
<bdmurray> AssertionError: JS timed out
<sinzui> ah, right, the build files need to be on the filesystem
<rick_h> right
<ev> sinzui: ah, good point. Thanks for the tips!
<sinzui> bdmurray, you can test any YUI module in your browser: xdg-open lib/lp/app/javascript/tests/test_lp_links.html
<sinzui> if your browser cannot run the test, the tree was not build right
<bdmurray> okay thanks, make jsbuild seems to have helped
<rick_h> awesome
<lifeless> http://www.infoq.com/presentations/Storm
<jelmer> 'morning lifeless
<lifeless> hi jelmer
#launchpad-dev 2012-06-09
<wgrant> ev: What is this launchpadlib-using web service doing?
<ev> wgrant: creating bug reports for error reports on errors.ubuntu.com
<ev> wgrant: there will be a "create" link wherever there isn't a LP bug report already in the "report" column
<ev> clicking on that will take you through the OAuth dance, then create a new launchpad bug for the data in the error tracker (daisy.ubuntu.com)
<ev> we thought about just doing this automatically - and indeed I thought that was the consensus at UDS, but other developers don't want to buried in Launchpad bugs any more than they already are
<ev> so they would prefer it to be a semi-manual operation
<ev> I had it first implemented as a tastypie API, called asynchronously from YUI, but then realised it'd probably be better to avoid having a bot user's credentials do the heavy lifting
<wgrant> ev: Well, anyone granting a non-read-only LP OAuth token to a non-desktop application should probably have their privileges revoked, since they provide insufficient scope restriction facilities.
<ev> wgrant: it only holds the keys in cookies. In practice, I don't see then how it's different than a desktop application
<wgrant> eg. any writable token from an archive admin can root every Ubuntu machine in the world in fairly short order.
<ev> wgrant: would your suggestion then be to go back to creating a bot user for this sort of thing?
<wgrant> ev: I'm not quite sure.
<wgrant> ev: But teaching people that it's OK to give random webapps access to their Launchpad accounts is not a good idea.
<wgrant> (Launchpad could make this a non-issue by actually providing sensible OAuth functionality, but today it does not)
<ev> sure, but alas it does not and I have deadlines to meet :)
<ev> (and other work items dependent on being able to associate an error with a bug number)
<lifeless> wgrant: what do you mean by sensible - tightly controlled actions ?
<lifeless> ev: Why did you stop using a service account? they are the 'stock' answer we have for that sort of thing...
<cjwatson> wgrant: What about ~ubuntu-archive-robot, then (which isn't actually being used yet, but I'd like it to be soon enough to replace stuff currently being run with direct DB access on cocoplum and should-be-automated stuff run by hand)?  I discussed that with sinzui a while back and we agreed that it wasn't really any worse than the present situation ...
<cjwatson> (Restricting its scope to particular API requests, say, might be nice but given the kinds of things we need to do I'm not sure it would actually be a meaningful security improvement ...)
<cjwatson> If it weren't for bug 1006917, that could (I think reasonably) be running fully-automatic auto-syncs from Debian right now.
<_mup_> Bug #1006917: Distribution archive owners cannot necessarily copy packages <feature> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1006917 >
<ev> lifeless: at the time it seemed like having the bug created by the individual clicking the link on errors.ubuntu.com was the sensible thing to do, but equally because the documentation (https://help.launchpad.net/API/launchpadlib#Authenticated_access_for_website_integration) didn't say to not do it and I didn't find anything saying that I should definitely use a bot account for this sort of thing.
<wgrant> lifeless: On any other OAuth service the tokens grant partial access depending on what the application requires.
<wgrant> cjwatson: It's clearly not a problem for a service that is intended perform archive administration tasks to possess permission to perform archive administration tasks.
<wgrant> cjwatson: It is probably a problem for a widely-accessed bug filing service to possess them.
<ev> wgrant, lifeless: so to be clear, I should be using a bot account for this? Any thoughts on whether clicking the "create bug" link should also sit behind open ID, or can we get away with letting anyone click on it? It's javascript, so presumably crawlers wouldn't hit it.
<lifeless> ev: what data will it ask the user for ?
<lifeless> ev: I suspect my answer is going to be 'use a bot and do the thing automatically with no button presses at all'
<ev> lifeless: it doesn't need to ask the user anything. It will take the error metadata and create a bug, then write that into the duplicates db
<lifeless> ev: so why does there need to be a button ?
<lifeless> wgrant: I don't suppose you got the LEP together on friday afternoon after I EOD'd ?
<ev> yeah, I thought doing it automatically was the understanding coming out of UDS, but developers were against flooding Launchpad with more bug reports that they wont be able to process
<ev> and it's really only needed for the things we'll fix
<ev> so if a crash happens once, creating a bug for it is just noise
<lifeless> ev: there is a button for that: 'invalid'
<ev> the specific problem this is trying to solve is having a way of marking a crash as being fixed. We also discussed putting SHA-1 hashes in the changelog, but that was overwhelmingly rejected
<ev> lifeless: sure, but that requires human processing :)
<ev> SHA-1 hashes of the crash signature, that is. So you could avoid creating a bug report entirely.
<lifeless> so the workflow is at-or-after fixing the issue, someone realises its on errors.ubuntu.com but didn't have a bug
<lifeless> they click the button, which creates a bug, they go to the bug, mark it a duplicate of the one they used during their upload, and the system then knows its fixed ?
<lifeless> if someone is working from errors.u.c, they might start with the button clicking
<ev> half of that. Someone sees an issue on errors.ubuntu.com, they fix it. They click the "create bug button" then reference it in the debian/changelog and upload
<lifeless> ev: folk *will* fix things accidentally ;)
<ev> that then goes into our facy bugs -> fixes UI
<lifeless> ev: whats the workflow when someone fixes it without telling the system in advance ?
<ev> lifeless: sure, and indeed that should do the right thing when faced with the scenario you've presented
<ev> it can do exactly that. Follow the bug to its parent and write that into the DB
<lifeless> ok
<ev> or follow the bug to its parent as it's reading the bug -> fixed binary versions data later on, since the duping of the created bug could happen much later in time
<lifeless> cool
<lifeless> ok
<lifeless> uhm
<lifeless> what do you think folk would feel if:
<lifeless>  - the top10 frequency signatures automatically got bugs made
<lifeless>  - other signatures allow you to specify a bug # or request one get made
<lifeless> ev: btw, you can't safely use launchpadlib within a web service.
<lifeless> ev: you need to farm it out to a single threaded own-cache-owning worker.
<ev> eep
<ev> I think creating bugs for the top 10 frequency signatures automatically is reasonable
<lifeless> oh, I forgot, its (privisionally) fixed.
<lifeless> bug 459418
<_mup_> Bug #459418: Cache is not safe for concurrent use (by processes or threads) <lp-foundations> <oem-services> <Launchpad itself:Triaged> <lazr.restfulclient:Fix Committed by jml> < https://launchpad.net/bugs/459418 >
<lifeless> I don't think its been released.
<ev> beauty of web services - I don't have to wait for it to be
<lifeless> thats in lazr.restful
<lifeless> erm restfulclient
<lifeless> so yes, you do ;)
<lifeless> or you need to run your own tarball
<ev> well I mean I can throw it into a PPA and add it to the dependencies package for webops
<ev> of course, it might be released faster than the PPA builds it :-P
<ev> ah, but presumably that's blocked on bug 513116
<_mup_> Bug #513116: launchpadlib is not thread-safe <lp-foundations> <oem-services> <Launchpad itself:Triaged> <launchpadlib :Triaged> < https://launchpad.net/bugs/513116 >
<lifeless> releasing lazr.restfulclient blocked? no. But using launchpadlib in a web server - yes, thats blocked ;)
<lifeless> ev: if I may make a suggestion
<lifeless> ev: for make-a-bug, use a parameterised link directly to launchpad, and the user can go through the filing form themselves.
<lifeless> ev: for automatically-make-it do it in celery/rabbit consumer/backend script/whatever
<lifeless> ev: its not quite as nice, but it will be a lot faster to put together, and you can iterate later
<ev> lifeless: we need to be able to write the bug number into Cassanda. We could put an input box there, but that is quite ugly. It sounds like celery is the best option until launchpadlib becomes thread safe.
<lifeless> ev: 'forever'
<ev> hahaha, oh dear
<ev> well, using celery isn't a bad thing
<lifeless> sure :)
<lifeless> ev: anyhow, does that give you enough to go on?
<ev> lifeless: it surely gives me enough to keep me busy for a while :)
<lifeless> ev: was my reply to the just-in-time thread sane ?
<ev> lifeless: the librarian one?
<lifeless> yes
<ev> yes, and thank you for following up to that
<lifeless> my pleasure
<lifeless> it had slipped past me for a bit
<ev> and a big thanks for pointing out that we want to delivery code to execute from this thing, so it's a security problem we'll have to solve one way or another. That had slipped my mind.
<lifeless> yeah, it makes quie a difference to the incremental overheads :)
#launchpad-dev 2012-06-10
<wgrant> lifeless: Is there a tarball for testresources 0.2.5 somewhere?
<wgrant> It's apparently released, but I see no files.
<lifeless> wgrant: http://pypi.python.org/pypi/testresources
<wgrant> lifeless: Ah, thanks.
<lifeless> wgrant: np
<jml_> anyone around?
<jelmer> hey jml_
<lifeless> jml_: FSVO
<jml_> lifeless: thanks. following up w/ jelmer
<lifeless> anyone around ?
 * lifeless needs to double check a small zcml/zope thing
<lifeless> specifically, how can I be sure a template is not being used ?
 * mwhudson waves
<mwhudson> lifeless: grepping for the template name should work i think?
<lifeless> is just grepping on the configured name from the zcml sufficient ?
<lifeless> e.g. lib/lp/answers/templates/questiontarget-portlet-latestquestions.pt is what I was looking at (which I found a use) - but the template name is not what is referenced :)
<lifeless> yay indirection
<mwhudson> i guess it can be pretty hard to tell
<mwhudson> that template is pretty clearly used though
<lifeless> indeed
<lifeless> it is one of the culprits using memcache.
<lifeless> (not that I object to memcache in principle, but its not being very helpful today
<lifeless> low hit rate, some inappropriate uses, and we're not using it for *scale* we're using it for performance, which entirely misses the point of random access usage patterns.
<mwhudson> yes
<lifeless> yay, huge list of conflict :(
<lifeless> oh, and no invalidation protocol
<lifeless> so, lots of repeated user confusion
#launchpad-dev 2013-06-03
<StevenK> wgrant: /srv/launchpad.net/builddmaster is also used on mawson by buildd-manager as queue for binaries from the buildds?
<wgrant> StevenK: Sure, and it should be on alphecca do
<wgrant> too
<wgrant> It's used on builddmaster hosts
<StevenK> Right
<wgrant> But it only exists on other upload hosts because of P-a-s.
<StevenK> Yeah
<StevenK> wgrant: So my QA plan is to destroy the P-a-s checkout on mawson, and upload acpi to my ppa and see what happens
<wgrant> That would be correct.
<StevenK> 2013-06-03 01:13:46 DEBUG   Creating PENDING publishing record.
<StevenK> 2013-06-03 01:13:47 DEBUG   Created amd64 build of acpi 0.09-3ubuntu2 in ubuntu precise RELEASE [4560808] in PPA for Steve Kowalik (2505)
<StevenK> 2013-06-03 01:13:47 DEBUG   Created i386 build of acpi 0.09-3ubuntu2 in ubuntu precise RELEASE [4560809] in PPA for Steve Kowalik (2505)
<StevenK> 2013-06-03 01:13:47 DEBUG   Not closing bugs for PPA source.
<StevenK> So that works fine
<wgrant> StevenK: Not quite
<wgrant> StevenK: Only i386 and amd64 chroots exist for saucy
<wgrant> So that's not necessarily a good test
<StevenK> wgrant: But I didn't upload to saucy
<wgrant> Ah, true
<wgrant> And I don't think I cleaned out the precise chroots
<StevenK> wgrant: You didn't, there are 5 chroots for precise on DF
<StevenK> wgrant: Right, I uploaded acpi to a devirt PPA in precise and only get amd64 and i386 too
 * StevenK nails wallyworld to IRC.
<wallyworld> i think my wireless router is fighting with my laptop
<StevenK> And which is winning?
<wallyworld> neither :-(
<wallyworld> reboot time, but i'm in the middle of something
<wgrant> Ohhh, clever
<StevenK> wgrant: aoetools to a devirt PPA creates all five build records, too
<wgrant> StevenK: Excellent
<StevenK> wgrant: So, previewdiff work, or shall I create an API to update a chroot like infinity wants?
<wgrant> StevenK: We're not very good about accepting large file uploads atm, but it might work
<wgrant> See if we regularly have apport attachments that are roughly chroot-sized.
<wgrant> Oh Zope
<wgrant> You are so awful sometimes
<wgrant> I just cut 20% off the Launchpad start time by fixing zope.component.registry to not be completely moronic
<wgrant> (registering or unregistering a utility compares each new registration against every existing one)
<StevenK> Haha
<StevenK> wgrant: Still waiting for my terrible query to finish hurting mawson.
<nigelb> < wgrant> You are so awful sometimes <-- sometimes?!
<wgrant> There we go, registerUtility down from 47.73% (7.19%) to 5.29% (0.27%), and I doubt I broke anything.
<StevenK> wgrant: What's the diff, so I can laugh and/or cry?
<wgrant> http://pastebin.ubuntu.com/5728141/
<wgrant> InterfaceClass.__hash__ is also pretty slow now
<wgrant> 5% of no-op script runtime
<wgrant> Actually, that patch is a bit buggy now
<wgrant> But the fix shouldn't change performance
<StevenK> wgrant: Interesting choice of variable name
<wgrant> I was none too pleased to see the original code :)
<wgrant> Ah, the new code is actually marginally faster, even.
<StevenK> I bet
<wgrant> http://pastebin.ubuntu.com/5728149/
<StevenK> wgrant: Could it be due to older Python?
<wgrant> StevenK: No
<wgrant> O(nÂ²) is never a good idea :)
<wgrant> Interface.__eq__ was being called a few million times.
<StevenK> WCPGW
<StevenK> wgrant: So, you plan on upstreaming it?
<wgrant> Seeing if I can get start time below 50% first
<StevenK> wgrant: No fair if your patch contains speech like GLaDOS addressing Wheatley.
<wgrant> Ah
<wgrant> Maybe if I drop lazr.restful registrations from scripts
<wgrant> That could give me another 20%
<wgrant> :(
<wgrant> Barely a second
<wgrant> And zope.sendmail adds 3s to shutdown
<StevenK> wgrant: http://pastebin.ubuntu.com/5728202/
<wgrant> StevenK: That just shows that 16 requests ever exceeded 200MB, which isn't exactly encouraging.
<wgrant> StevenK: Perhaps try uploading some chroots to +storeblob and see if they always succeed
<StevenK> wgrant: Right, hitting up RootObject:+storeblob (I guess) on DF as soon as I can grab a chroot
<StevenK> Oh look, there's on
<StevenK> *one
<wgrant> StevenK: Needs to be prod
<wgrant> DF means nothing
<wgrant> It's likely to succeed on DF/staging but fail on prod
<StevenK> wgrant: I'm a little restiscent to toss a few hundred Mb at prod
<wgrant> StevenK: Why?
<wgrant> +storeblob will get pruned in ~48h
<StevenK> Right
<wgrant> Near-sane test startup time is near my grasp :)
<StevenK> Heh
<wgrant> No-op script time is below 3s
<wgrant> (moving webservice and browser stuff to appserver-only, and with the zope.sendmail and zope.component fixes. zope.interface is also a bit slow, but the slowness was added for a good reason)
<wgrant> It was about 9.8s before
<StevenK> wgrant: 'Sending request to launchpad.net...' -- this could involve a bit of waiting
<wgrant> Quite
<wgrant> It'll probably time out from here
<wgrant> But we'll see
<StevenK> Some way to gauge progress would be nice, but I can't have everything
<wgrant> http://pastebin.ubuntu.com/5728220/ is the zope.sendmail diff
<wgrant> Previously every LP process would have to wait 3s for that thread to shutdown
<wgrant> That's why everything takes so long to finish
<StevenK> Come ON firefox
<wgrant> So yeah, 11.6s to 3.6s for a script run
<wgrant> I mostly want this for less awful test times
<wgrant> librarian start times, subprocess startup/teardown times
<StevenK> Hmmm
<StevenK> 'Waiting for launchpad.net'
<StevenK> wgrant: Is the blob ticket secret?
<wgrant> StevenK: What do you mean?
<wgrant> There's no way to look up the blob from the outside without it
<wgrant> And you can only get it from the response to a POST to +storeblob
<StevenK> wgrant: Your ticket is "2f66004a-cc06-11e2-a37c-002481e7f48a"
<StevenK> From a chroot upload
<wgrant> So it worked?
<StevenK> Yeah, it just took 25 minutes to upload.
<StevenK> wgrant: Shall I repeat the test?
<wgrant> StevenK: Might as well
<wgrant> I wonder if the timeout is 30 minutes
<StevenK> wgrant: Right, that works as well
<wgrant> StevenK: Not sure how well launchpadlib exposes uploads. I'd try to work out that next.
<StevenK> wgrant: In terms of progress or how they're exposed?
<StevenK> IBug.addAttachment() was my plan
<wgrant> There's certainly no progress
<StevenK> wgrant: Current plan is DAS.setChroot(data (exported as Bytes), sha1sum)
<wgrant> StevenK: Why sha1sum?
<StevenK> wgrant: Because infinity would like to be damn sure the client and server agree about a checksum before we set PocketChroot.chroot
<wgrant> OK.
<StevenK> Hmm
<StevenK> wgrant: Going to write a security adapter that limits to the owner of the relevant distribution main_archive, too
#launchpad-dev 2013-06-04
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/celery-tweaks/+merge/167197
<StevenK> wgrant: I was late lunching, looking.
<StevenK> wgrant: Doesn't the import on lines 8-10 fit on one line?
<wgrant> StevenK: Hm, indeed
<wgrant> I thought so, so ran format-new-and-modified-imports, but it didn't do anything
<wgrant> StevenK: The diff is updated
<StevenK> I was just about to look at that again since I've answered that question.
<StevenK> wgrant: r=me
<wgrant> Thanks
#launchpad-dev 2013-06-05
<StevenK> Hmm, when YUI 3.10.2 come out ...
<StevenK> Haha, yesterday.
<StevenK> wgrant: http://pastebin.ubuntu.com/5734688/
<wgrant_> StevenK: Going to upgrade?
<StevenK> wgrant: To YUI 3.10.2? I can quickly put together a branch for it and toss it at ec2 if you wish.
<wgrant> StevenK: I don't think there are any notable compat breaks, so it might be worth a try
 * StevenK tosses 33Mb of a new YUI zip at sourcedeps
<StevenK> wgrant: Yeah, YUI upstream seem to be focusing clean-up and speed since 3.8.1
<StevenK> wgrant: I have a branch ready, tossing it at ec2.
<StevenK> BAH
<StevenK> Damn it, Thunderbird, lose your interest in the staging mailbox
<StevenK> wgrant: Did you peer at the pastebin?
<wgrant> StevenK: I probably didn't see it
<wgrant> My freenode connection died
<StevenK> Then your connection is a miserable failure.
<StevenK> wgrant: http://pastebin.ubuntu.com/5734688/
<wgrant> That is perhaps the least descriptive problem description I've seen
<wgrant> Oh, the setChroot call
<StevenK> I can paste the diff too if you wish
<wgrant> have you debugged it?
<StevenK> Tried, I drowned somewhere in lazr.restful
<wgrant> That's not immensely helpful
<wgrant> Which field was it? What is v?
<wgrant> What does the request look like?
<StevenK> wgrant: It's data, the Bytes field, and v is u' '
<wgrant> StevenK: How is v u' ' when it should be wrapped in []?
<StevenK> It's a Bytes field, why should it be?
<wgrant> StevenK: Look at the code
<StevenK> wgrant: Ah ha
<StevenK> wgrant: lib/lp/soyuz/browser/tests/test_distroarchseries_webservice.py -- shouldn't that be under tests, rather than browser/tests ?
<wgrant> StevenK: webservice tests are usually under browser (or stories, which is the browser equivalent of doc)
<wgrant> StevenK: I'm not a huge fan of the meaning of launchpad.Edit on DAS and DS differing so greatly.
<angelica> hla
#launchpad-dev 2013-06-06
<lifeless> so, whos up for kicking bug mangler Tobias B ?
<wgrant> I've fixed those and emailed him
<wgrant> If he continues, a suspension will swiftly follow.
<StevenK> wgrant: I'm confused. The test that fails is actually firing an event which ends up tossing an exception that is caught and then returns false. That code path for the event framework is not called for 3.10.2, so there is no exception, therefore the function returns true.
<wgrant> StevenK: Have you traced through it in both versions?
<StevenK> There is a maze of event functions, but I set a breakpoint on the line that causes the exception in 3.9.1, and it doesn't get hit in 3.10.2
<StevenK> wgrant: The fire() function in 3.10.2 looks very very different -- in 3.10.2, the call to getSiblings() (which is what dies horribly in 3.9.1) is guarded by this._hasSiblings
<wgrant> StevenK: You'll probably want to step through it and work out what's going wrong.
<StevenK> http://pastebin.ubuntu.com/5737687/ fixes it in 3.10.2
<wgrant> What does it mean?
<wgrant> Shouldn't the response always have an event_key?
<StevenK> Ah, but this test is for malformed data
<StevenK> Which only has { something: 6 } and no event_key
<wgrant> Y.fire behaves differently in the face of undefined?
<StevenK> Yes, it fails to find silblings in 3.9.1, hence tosses an error
<StevenK> It doesn't look for silblings in 3.10.2 and so behaves
<wgrant> Can you see the rationale for the change?
<StevenK> wgrant: Based on a quick Googling, it is make fire() more performant
<wgrant> StevenK: What's the intent of the test?
<StevenK> wgrant: That manager.sucessPoll() with that responseText returns false
<wgrant> StevenK: Oh, so it catches the exception, right. That diff sounds OK, then
<StevenK> Well, in fact it fires a namespace.longpoll_fail_event with the exception, so maybe mine should do the same too
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/add-yui-3.10.2/+merge/167689
<wgrant> StevenK: Can you remove the try/except?
<wgrant> Surely that check will work for 3.9.2 as well
<wgrant> Or are there other potential errors?
<wgrant> 3.9.1
<StevenK> wgrant: Just in case there are other errors, like failure to decode
<wgrant> StevenK: Would it be better to just throw an exception rather than duplicating the failure firing?
<StevenK> Oh
<StevenK> Could do that
<StevenK> wgrant: http://pastebin.ubuntu.com/5737890/
<wgrant> StevenK: Do we prefer to throw strings or objects (eg. Error)?
<StevenK> wgrant: I see evidence of both, even in the same JS files
<wgrant> Yay
<StevenK> lib/lp/registry/javascript/timeline.js:                throw new Error("resize_frame must not be empty.");
<StevenK> lib/lp/registry/javascript/timeline.js:            throw "Invalid y_align argument: " + y_align;
<wgrant> Yayer
<StevenK> 38 for Error and kin, 11 for bare strings
<StevenK> wgrant: Diff updated
<wgrant> StevenK: k
<StevenK> wgrant: So, launchpad.Edit for DAS.
<StevenK> wgrant: Shall I use launchpad.Moderate or so then?
<wgrant> Moderate might work indeed
<wgrant> In other news, person has too many FKs referencing it.
<StevenK> News at 11
<wgrant> Somewhere around 200
<StevenK> wgrant: http://pastebin.ubuntu.com/5737952/
<wgrant> StevenK: Sounds reasonable
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/das-setchroot-api/+merge/167464
<wgrant> StevenK: That docstring is not good. "Set the chroot tarball used for builds in this architecture. The SHA-1 checksum must match the chroot file.", maybe.
<wgrant> That even lets you format the docstring legally.
<wgrant> Otherwise that looks OK, except for the file upload bits, but not much we can do about that because lazr.restful and Zope.
<StevenK> Oh, do you want the XXX there like the other callsites to that horrible method that I will hopefully pay for in blood one day?
<wgrant> That would be nice.
<StevenK> wgrant: ... That I add the XXX, or pay in blood for that horrible method? Both?
<wgrant> Both would be ideal.
<StevenK> http://pastebin.ubuntu.com/5737987/
<wgrant> Docstring should be formatted correctly.
<StevenK> wgrant: http://pastebin.ubuntu.com/5737994/
<wgrant> Right.
<StevenK> wgrant: Diff updated
<wgrant> StevenK: r=me
<StevenK> wgrant: Thanks
#launchpad-dev 2013-06-07
<wgrant> StevenK: Are you still touching mawson?
<StevenK> wgrant: I've been doing bad things to qas, not mawson
<wgrant> StevenK: But your script is qa-mawson!
<StevenK> And it talks to qas, just to be confusing
<StevenK> wgrant: So I can't get progress from a lplib POST?
<wgrant> StevenK: No
<StevenK> This is upsetting
<wgrant> Your kernel can probably tell you :)
<StevenK> Oh, how?
<wgrant> StevenK: I'm not sure how to get it from the local endpoint, but I usually just grab it from ip_conntrack on the router
<StevenK> wgrant: Uploading at 75KiB/s, apparently
<StevenK> wgrant: And there's 3MiB extra uploaded
<StevenK> Oooh, it just hit CLOSE_WAIT
<StevenK> wgrant: This seems to upload it twice? First connection uploaded 77MiB or thereabouts, second connection is 22MiB in and counting
<wgrant> StevenK: That would be hilarious
<wgrant> StevenK: You don't have HTTP debugging enabled?
<StevenK> I do not
 * StevenK waits for his terminal to recover
#launchpad-dev 2013-06-08
<nigelb> StevenK: Happy Birthday! (and I hope you aren't on IRC today)
#launchpad-dev 2014-06-02
<bigjools> wgrant, cprov: *loving* the inline MP stuff, excellent work guys.
<cprov> bigjools: thx, cjohnston has also worked on it. Lots of improvements and suggestions to come
<bigjools> thanks to cjohnston too!
<bigjools> cprov: next job, git hosting? :)
<cprov> bigjools: no kidding, git support is coming.
#launchpad-dev 2014-06-03
<bigjools> cprov: along with its famous usability
<jelmer> bigjools: Stockholm syndrome is here to help
<bigjools> jelmer: :)
<dpm> hi wgrant, have you had the chance to update the translation exports crontab for RT #72011? I'd like to roll out the utopic language packs this week for the MAE image
<_mup_> Bug #72011: adonthell-data: Please review/upload this merged package. <adonthell-data (Ubuntu):Fix Released by crimsun> <https://launchpad.net/bugs/72011>
<wgrant> dpm: It's on my list, I've only been a non-zombie for three hours since the disastrous flight back from Malta.
<wgrant> Let me see if there are people around now to do it.
<dpm> oh, you were caught up in that too, sorry to hear that :(
<wgrant> Suspected bomb (actually diving equipment) shut down the airport for two hours, missed connection in Dubai, arrived in Melbourne at 5am yesterday, got no sleep on flights, yeah.
<wgrant> But home and more alive now.
<dpm> yeah, I read that, glad you finally made it home
<wgrant> dpm: So we only want one devel export per week now?
<wgrant> We used to run Trusty Tuesday and Thursday
<wgrant> Well, currently do
<dpm> wgrant, yeah, pitti suggested that and I decided to go along to make the crontab simpler, especially the changes we do when we open a new release
<wgrant> Replied to the RT.
<dpm> wgrant, awesome. Replied too. Do I need to ping anyone in particular to get your change applied?
<wgrant> I might see if bradm can apply that quickly, given how trivial it is.
<dpm> cool, thanks. I've also added a comment regarding the lp-get-ul10nstats.py exports - there's RT 72012 for that, but in a nutshell, happy with your suggestion of exporting precise, trusty and utopic
<wgrant> I think those got merged.
<wgrant> Yeah
<dpm> ah, I see
<wgrant> Might want to reply to point that out, in case that half gets missed.
<dpm> I pointed that out in my last reply. Let me know if that's enough, and if not, happy to add more info
<wgrant> jtv: Is there a way to delete/hide a POFile? I can't see one.
<wgrant> For https://answers.launchpad.net/launchpad/+question/249542, specifically.
<jtv> wgrant: no, I don't think there is â you'd have to remove the translations.
#launchpad-dev 2014-06-04
<evfool> hey all, I would be interested in implementing a launchpad bugtracker plugin, and need to know whether a bug tracker plugin can modify the bug title on launchpad
<wgrant> evfool: Launchpad bugwatches can only import status and importance.
<wgrant> cjohnston, cprov: Joining?
<cjohnston> I'm in there
<wgrant> Are you sure?
<evfool> wgrant: meaning that if I would like an integration with an external site which does change the bug title, I would have to modify the launchpad core, and that's probably not going to get accepted (just for the sake of providing better integration with an external bugtracker)
<evfool> ?
<wgrant> evfool: Bugwatches aren't really meant to do that. They're usually used by projects or distributions that use Launchpad's bugtracker, and they mostly want Launchpad to tell them when the same bug on a remote bugtracker is fixed.
<wgrant> So it shouldn't automatically change the Launchpad title.
<wgrant> What are you trying to do?
<evfool> wgrant: bountysource integration, but the only pluggable thing I have found in launchpad are bugtrackers, which bountysource is
<evfool> see bug 1325045
<_mup_> Bug #1325045: Bountysource integration [$100] <Launchpad itself:New> <https://launchpad.net/bugs/1325045>
<wgrant> That doesn't seem like a good fit for bugwatch integration.
<wgrant> Why do you particularly want it integrated in Bugs, or indeed Launchpad at all?
<evfool> wgrant: the elementary project uses bountysource to provide bug bounties, and uses launchpad for bug tracking, code reviews, merge proposals and others
<evfool> wgrant: when adding a bounty, that appears only on bountysource, and they would like better integration with launchpad to have some sort of status on bugs with bounties about the bounty
<evfool> currently, each bug gets manually commented with a link to the bountysource issue, the title gets updated to contain the bounty value, and occasionally a bounty tag is added too
<wgrant> I'd suggest a bot using the Launchpad API instead
<wgrant> I think that makes more sense than a bugwatch here.
<evfool> wgrant: thanks, I'll look into integrating from the other side, through a bot making the changes
<wgrant> You can file a bug using the createBug method on https://api.launchpad.net/devel.html, using https://help.launchpad.net/API/launchpadlib
<evfool> wgrant: bounties get added to already existing launchpad bugs, so I'm not gonna create tickets, but I'll look for updateBug then
<evfool> thanks for your help
<wgrant> cprov, cjohnston: http://pastebin.ubuntu.com/7587437/ gets it down to six queries per translated string.
<wgrant> I'm not sure the second change is totally correct in the presence of shared POTMsgSets; we need to entire that the TranslationMessage has the correct POFile set, not just getOnePOFile
<cprov> wgrant: CurrentTranslationMessageZoomedView.zoom_url calls self.sequence too
<cprov> wgrant: self.context.sequence should fix it
<wgrant> cprov: I'm not seeing duplicated queries from that here.
<wgrant> Though we shouldn't be using the ZoomedView multiple times one on page anyway
<wgrant> That's why it's zoomed.
<wgrant> I'm not sure how much further we can reasonably go here without combining the views.
<wgrant> So turn the CurrentTranslationMessageView thing into a view that actually renders somewhere between 0 and infinity of them
<wgrant> Running all the suggestion queries etc. in bulk
<wgrant> We could hack this into the model, but it's going to be ugly and I'm not sure it's worth it.
<wgrant> cprov: What do you think?
<cprov> wgrant: I think it will be a long journey, considering the current code shape
<wgrant> It's going to be either way
<wgrant> The view code does all the suggestion etc. calculations itself
<wgrant> We need to move that somewhere higher up
<wgrant> I'm not sure how much per-message code is left once we do that
<wgrant> And anything that is left is probably crap :)
<cprov> agreed, minor amendings are not getting us anywhere
<cprov> they will probably take as much time and break as many tests as writing a string-batch-loader from scratch.
<wgrant> The code in question is mostly browser/{pofile,translationmessage}.py, right?
<wgrant> It's big and ugly, but I think it's probably worth basically doing a massive refactor which turns it into a megaview.
<wgrant> Hacking the bulk loading in is only going to make it worse, and it's already like 10 years of hacks
<cprov> wgrant: well, the megaview would still be in charge of doing the bulk loading of subviews, no ?
<wgrant> cprov: I'm not sure there'd be many subviews.
<cprov> or do you mean to collapse those too ?
<wgrant> Almost all of the logic would end up in the megaview that renders n messages
<wgrant> I suspect the per-message templates would end up as macros, not entire views.
<cprov> makes sense
<wgrant> cjohnston, cprov: I'm done, be back in a sec if you want to hang out again yo.
<cjohnston> k
<wgrant> So many queries :(
#launchpad-dev 2014-06-05
<bigjools> wgrant: still looking for informal inline comments feedback? Or prefer bugs?
<wgrant> bigjools: Depends on whether it fits well as a bug.
<wgrant> But informal is fine.
<bigjools> wgrant: ok so:
<bigjools> replying to inline comments sends an email out without the original comments as context
<wgrant> Yup. That's a bit difficult to do until we're parsing the diff to show only relevant hunks, which is one of the two top bits left to do.
<wgrant> We're not entirely sure how we'll show the existing comment thread, though
<bigjools> ok
<bigjools> it's great work though, we're loving it
<wgrant> That's why we got it out early.
<wgrant> The amount of nagging in person last week...
<bigjools> heh
<dpm> wgrant, could you give me a hand to close RT 72011? Essentially, I need the lp-get-ul10nstats.py cron job updated to get exports for precise, trusty and utopic, but I need someone with access to loganberry.canonical.com-launchpad to do it
<dpm> I'd be happy to provide the diff myself if I can get access to a copy of that file
<wgrant> dpm: Only IS can change that file.
<wgrant> I reopened the RT, but I guess it's low prio or something.
<wgrant> bradm: ^^ feel like a trivial loganberry crontab change before you disappear?
<dpm> thanks wgrant. I'll see if I can ping someone at #is too
<wgrant> dpm: It'd be #webops
<dpm> ah, ok
<dpm> wgrant, quick LP question: I'm trying to set the maintainer of the ubuntudeveloperportal project to the ~ubuntudeveloperportal team, but LP does not seem to allow me to do that. When I try to change it and pick the team from the search box, the only result for 'https://launchpad.net/ubuntudeveloperportal' is 'ubuntudeveloperportal-editors', which is a completely different team
<dpm> any ideas on how to set the maintainer to the ubuntudeveloperportal team? thanks
<wgrant> dpm: https://launchpad.net/~ubuntudeveloperportal is an open team. LP won't let you use an open team as a project maintainer, as anyone could join that team and commandeer the project.
<dpm> oh, hadn't realised that!
<dpm> I'll change the permissions
<dpm> thanks!
<wgrant> You probably also want to check that it only has trusted members.
<wgrant> There are some badge collectors there.
<dpm> yeah, I see what happened
<dpm> so Jono created the open team to have a mailing list, then he handed it over to me
<wgrant> Ah, right.
<dpm> the ml is not really being used, so I think I'll just change the permissions for the team to be more restrictive, rather than creating a new one
<wgrant> The team description suggests it's mostly to collect bugmail, but that has't been necessary since 2011.
<wgrant> So it's probably reasonable to repurpose it and clean up the members.
<dpm> yeah
#launchpad-dev 2014-06-06
<cjwatson> wgrant: Hmm, if I move generate_extra_overrides.py to ubuntu-archive-publishing, then I obviously can't use the database directly to extract the list of operable series.  Should I arrange to pass that in the environment, or would it be better to use launchpadlib from pepo?
<wgrant> cjwatson: Hrm.
<wgrant> cjwatson: launchpadlib wouldn't be horrible, given it's all RO.
<wgrant> Different scripts have different rules about the series they operate on, so it's possibly best left up to them to calculate the list.
<cjwatson> Yeah, indeed I'll need to move out maintenance-check at the same time and it has different rules
<cjwatson> We'll be able to drop germinateroot from the publisher config, since nothing else cares
#launchpad-dev 2015-06-02
<cjwatson> wgrant: should bug 1244868 be fix-released now?
<mup> Bug #1244868: all architectures should be togglable for PPAs <ppa> <qa-ok> <soyuz-core> <Launchpad itself:Fix Committed by wgrant> <https://launchpad.net/bugs/1244868>
<cjwatson> and similarly the asana stakeholders task
<wgrant> cjwatson: Ah, indeed.
#launchpad-dev 2015-06-03
<cjwatson> blr: Hm, you made h be next comment and l be previous comment?  sort of feels backwards to me, perhaps because of Western left-to-right reading conventions
<cjwatson> (yes, I should have looked at it during review ...)
<blr> cjwatson: if it feel weird, easy enough to change before people become too accustomed to it, although the 'forward' keys tend to be to the left of their backwards counterparts (realise there are vimisms there too)
<blr> cjwatson: wgrant: should `required=False` be sufficient to prevent clientside validation on a TextLine? It seems to be expecting a value, and I'm not certain why... would like the 'Optional' behaviour the copy_fields like branch_location have
<wgrant> blr: Our forms do no client validation by default.
<wgrant> Well, no client validation at all.
<blr> "Required input is missing."?
<wgrant> That's server-side.
<wgrant> required=False would normally fix that.
<blr> that's what I thought... hrm
<wgrant> Which field is it?
<wgrant> There's no vocab involved?
<blr> wgrant: just a barebones TextLine
<blr> with a title, required and description
<blr> might need to look at what the launchpad_form/widget_row is doing more closely
<wgrant> Huh, that's quite surprising.
<wgrant> blr: If you can't work it out this evening, throw a diff/branch at me or Colin and we can have a look.
<blr> wgrant: thanks
<cjwatson> blr: Yeah, it may also be a vimism for me in that 'l' is very definitely a forward action in a file
<blr> cjwatson: hmm yes, perhaps it wouldn't hurt to swap them
<wgrant> h is certainly nicer to type, but there is precedent for the other way.
<blr> well given I like weird emacs chords, I'm probably the worst person to decide.
<StevenK> blr: Ah, so forward will be M-x Super-F12 Ctrl-Alt-f then?
<blr> StevenK: that sounds perfectly reasonable.
<blr> wgrant: cjwatson: did you see lifeless's new constraints feature for pip, pretty cool: https://github.com/pypa/pip/pull/2857/files
<cjwatson> cute
<redixin> hi all. any idea how to set up new git repository on launchpad?
<cjwatson> redixin: https://help.launchpad.net/Code/Git
<redixin> cjwatson: yes, but there is no info about setting up new repo. I can see only "import from git" option on launchpad project's page
<redixin> but it does import from git to bzr
<cjwatson> redixin: You just push to it
<redixin> cjwatson: hm. I'll try. thanks
<cjwatson> redixin: So for example if you want to push a default repository for an existing project named foo, "git remote add origin lp:foo && git push --all origin" or whatever
<cjwatson> otherwise use one of the other URL forms
<redixin> cjwatson: it works. thanks
<cjwatson> Cool
<rpadovani> Hey guys, I love to see how much effort are you pulling in lp development in last months - thank you :-)
<cjwatson> you're welcome :)
<blr> rpadovani: lots more to come!
<rpadovani> \o/
<blr> cjwatson: still about?
<cjwatson> blr: just
<blr> trying to understand the target argument for IGitRepositorySet.setDefaultRepository
<blr> presumably in the git context, not an IBranchTarget
<cjwatson> blr: That's the target (IProduct or IDistributionSourcePackage; IPerson targets don't have defaults) that you're setting the default for
<cjwatson> IBranchTarget is a confusing thing that I would like to refactor away soon
<cjwatson> I didn't replicate it in the Git model
<cjwatson> To the extent that it exists, it's mostly in GitNamespace
<blr> right okay
<cjwatson> But anyway, here target means the actual target of the repository
<cjwatson> For usage you could look at the GitRepositorySet tests or lp.code.xmlrpc.git
<cjwatson> Or the interface docs :)
<blr> right, I think I was slightly thrown by IBranchTarget, I'll ignore it.
<cjwatson> Yeah, I don't blame you
<cjwatson> It took me quite a while to understand what it was doing
<cjwatson> And you get awesome stuff like
<cjwatson> lib/lp/code/stories/webservice/xx-branchmergeproposal.txt:    >>> source = factory.makeBranchTargetBranch(target.target)
<cjwatson> *crosses eyes*
<blr> cjwatson: wow, I'm strangely reminded of old java codebases there heh
<cjwatson> Indeed!  BranchTargetFactoryTargetBranchFactoryFactory
<cjwatson> Bean
<blr> hah right, was going to say, forgot the Bean
<blr> wgrant: cjwatson: what's the best way to determine if product.development_focus is a git repo?
<blr> just check the type, or is there a helper somewhere?
<wgrant> blr: development_focus is a series.
<wgrant> It can't be a bzr branch or a git repo.
<wgrant> (a series can have a bzr branch, though)
<blr> err development_focus.branch
<wgrant> The project's main bzr branch is product.development_focus.branch, and the project's main git repo is getUtility(IGitRepositorySet).getDefaultRepository(product)
<wgrant> branch is always a branch.
<wgrant> (later on we'll probably allow a series to specify a branch name within the project's default git repo, but not at the moment)
<blr> okay, thanks
<wgrant> It's an unfortunate difference in the real-world model which complicates everything :)
<blr> wgrant: is there an equivalent of getUtility(IBranchLookup).getByUrl(branch_url) for a product?
<blr> where branch_url is in the lp:foo form?
<wgrant> blr: For a product?
<wgrant> You're in the context of a product. Why do you need to look one up?
<blr> wgrant: required to find the defaultrepository for 'another' product
<wgrant> blr: Oh, why are you doing that?
<wgrant> (you probably want getUtility(IProductSet).getByName('foo'), but I'm interested to know what you're doing.)
<blr> wgrant: hmm in the context of allowing a user to provide the url of another product to set their product's default git repository
<blr> perhaps my thinking around this is muddled.
<wgrant> blr: That's impossible.
<wgrant> It doesn't make sense, as is impossible in the model.
<wgrant> Do you have a use case for that?
<blr> in the mirror/fork case?
<wgrant> We don't support forking by creating another project. There are some situations where that is valid, but it requires an extensive rework of our project model to properly make sense.
<blr> ok, I suppose that clarifies that :)
<wgrant> We only support mirroring from external sites today, to discourage people from trying a GitHub-style model on the LP model where it doesn't fit.
<wgrant> (also to avoid security issues)
<blr> I wonder if that is a possible source of confusion for new users, given familiarity with github's model.
<wgrant> It is, but there are much worse ones.
<wgrant> We can mitigate it in a couple of obvious ways:
<wgrant>  - Make it very clear how to push up your own branches to an existing project.
<wgrant>  - Make the project creation form less insane, so people actually read the warnings on it.
<blr> right
<wgrant> Rather than "oh god there are 20 fields what am I doing let's just fill them in"
<blr> yep, there's far too much to think about up-front.
<blr> do you think we could safely set a default for licensing, but provide feedback to review it afterwards?
<wgrant> Yes, now that the commercial project model isn't nonsense, it's quite feasible.
<wgrant> We just ask private or public, and then bug them to select a license later.
<wgrant> (in fact, the "bug them to select a license later" workflow already exists, though it sucks, when you select "I don't know yet")
<blr> true
<blr> wgrant: I have a git 'foo', however getUtility(IGitRepositorySet).getByPath(self.user, '~kit/+git/foo') return None
<blr> git ^repo
<wgrant> blr: What's the git repo's target? That's a personal, not project, repo named 'foo'.
<blr> wgrant: a product
<wgrant> blr: getByPath('~kit/foo') is probably what you want.
<wgrant> Or just getByPath('foo') if it's the project's default.
<blr> oddly all None
<wgrant> ~kit/foo/+git/foo?
<wgrant> How did you create it?
<blr> the factory
<wgrant> ah, it'll have a weird name.
<wgrant> What was the exact command you used to create it?
<wgrant> It's probably ~kit/foo/+git/some-reasonably-random-string
<blr> just makeGitRepository() no args
<wgrant> Oh
<wgrant> Then it's not going to be foo, or owned by you, is it?
<blr> wgrant: it is, just using foo for illustrative purpose
<blr> ah ownership ugh
<blr> thanks, no doubt that's it :P
<wgrant> Heh
<wgrant> blr: Remember that repo.git_identity can get you the shortest path that will work for getByPath
<blr> oh that's handy
<wgrant> Not if you don't remember to store the repo in a local variable first, though :P
#launchpad-dev 2015-06-04
<blr> wgrant: do you see anything obviously wrong with the validation or call to getByPath on l.829? https://code.launchpad.net/~blr/launchpad/ui-project-setbranch/+merge/259069
<blr> wgrant: user 'kit' owns 'gitrepository-100046' yet getUtility(IGitRepositorySet).getByPath(kit, 'gitrepository-100046') is None (have tried with git_identity and other forms as well)
<wgrant> blr: getByPath takes a user for privilege checking, and a full repository path.
<wgrant> eg. the user should enter ~kit/project
<wgrant> Or ~kit/project/+git/some-non-default-repo
<blr> wgrant: is the interface comment misleading?
<blr> s/comment/docstring/
<wgrant> blr: Looks fine to me. It specifies the valid forms.
<wgrant> What's confusing about it?
<blr> '~kit/product-name-100049/+git/gitrepository-100046' returns a repo, however '~kit/+git/gitrepository-100046' does not
<wgrant> blr: The latter is a personal repo; one not related to a project.
<blr> aah
<blr> of course, sorry.
<blr> wgrant: pity zope IField doesn't support arbitrary attributes, or more specifically 'placeholder'
<blr> default isn't quite the same
<wgrant> blr: Right, that's more of an attribute of the widget, I think.
<wgrant> eg. a widget could use the title as a placeholder.
<blr> it would be nice to have the expected form for the git repo path as a placeholder, but perhaps I'll just add it the label
<blr> I suppose I could set it in the js, but that might be a little odd.
<blr> will be less of an issue once we have a search modal for git.
<wgrant> Yep, search will make it much better. It's also a relatively uncommon case.
<blr> wgrant: from [code]/project configure hosting is still for the series, do we want that to also point to project setbranch? That would leave project/[series] as the only means of configuring a series branch
<wgrant> blr: I think that's very reasonable.
<blr> yep, possibly even confusing in its current state.
<cjwatson> wgrant: Nice, I didn't know about variable_name_prefix in batchnavs.
<wgrant> cjwatson: I only knew about it from +members
<wgrant> That's quite possibly the only other callsite.
<cjwatson> wgrant: There's one other, BugTrackerSet:+index
<wgrant> Ah, so there is.
<cjwatson> https://bugs.launchpad.net/launchpad/+bug/1455907 suggests that the +members use doesn't work ideally, but that isn't really a batchnav problem as such.
<mup> Bug #1455907: Every "Former members" batch page starts with a list of "Active members" <Launchpad itself:New> <https://launchpad.net/bugs/1455907>
<wgrant> Yeah
<cjwatson> Except in that maybe batchnav Next should do AJAX rather than reloading the whole page
<wgrant> It can be fixed by anchors, only showing one half on subsequent batches, or using the AJAX batchnav stuff.
<cjwatson> I see the FF for the last is disabled on prod
<cjwatson> Indeed, everywhere
<wgrant> I think AJAX batchnav is only enabled for +bugs
<cjwatson> Branches too
<wgrant> I'd forgotten it was even implemented for +branches until I saw the code.
<cjwatson> But it's behind a disabled feature flag in both cases
<wgrant> No, +bugs has been AJAXed since 2013.
<cjwatson> Huh, that's interesting because I see ajax.batch_navigator.enabled mentioned in lib/lp/bugs/templates/bugs-listing-table.pt and that flag is definitely off.
<wgrant> Is that the right template?
<cjwatson> Perhaps not.
<wgrant> Hm, I think it is.
<wgrant> But there's a whole lot of custom JS for that page.
<danilos> launchpad consistently times out when I attempt to comment on my MP (ajax just mentiones a server problem after ~30s, but if I go to +comment and try to submit, I get http://people.canonical.com/~danilo/tmp/lp-problem.png): any known issues?
<danilos> MP in question is https://code.launchpad.net/~danilo/landscape-charm/actions/+merge/260916
<wgrant> danilos: Yep, looks like a weird postgres issue, we're investigating.
<danilos> wgrant, ok, thanks (I've even tried sending an email, still hasn't shown up)
<cjwatson> danilos: The mail processor will be writing to the same table, indeed.
<danilos> cjwatson, yeah, I was hoping the problem was somewhere between haproxy and lp appservers :)
<cjwatson> No such luck.
<danilos> yeah, got it
<wgrant> danilos: Should all be OK again now.
<danilos> wgrant, repeated comments have gotten through, fwiw (not a big deal, just letting you guys know)
<wgrant> cjwatson: Oh, I didn't realise IGitCollection handled PersonProduct etc.
<cjwatson> Yup, I overachieved slightly on the adapters since it was easy.
<cjwatson> Top tip to self: it's more likely that somebody will review my merge proposals if I actually remember to propose them.
<blr> wgrant: cjwatson: any suggestions for a user response for GitTargetError that is likely to make sense to them?
<blr> "repo attached to another target" may not be meaningful
<blr> hmm, well I'll go with that for now given it isn't the typical case.
<cjwatson> blr: I think that your form field should be using a vocabulary that only offers repositories attached to the correct target.
<blr> cjwatson: I think we decided to defer the repository search modal
<blr> cjwatson: so the case here is just providing the user with a git_repository_location TextLine
<blr> I believe the only error paths are repo not found, and target error
<blr> cjwatson: just updating the configure code link on [code]/project, but perhaps you could have a look in review if there's a better way.
<blr> I'll ping you
<wgrant> blr: It could still have a non-searchable vocab.
<blr> wgrant: ok, so presumably we want something like ValidTargetDefault?
<wgrant> blr: Any Git repository for the target is valid.
<wgrant> blr: What does the branch equivalent do? It has pretty much the same rules, but we don't need searching yet.
<blr> hmm branch_location doesn't have a vocabulary afaict
<wgrant> blr: How's it validated?
<wgrant> It's possible that it isn't; there are some cross-project relations that shouldn't exist, but they might be from branches that have been moved later.
<blr> wgrant: yep only validated for presence
<wgrant> blr: OK, then it might not be worth using a vocab just yet. Manually validate the target and use an error message like "Repository is in a different project." or similar?
<blr> okay, that's what I have atm
<wgrant> Really weird that the branch case wouldn't use a vocab already!
<blr> branch_type does
#launchpad-dev 2015-06-05
<blr> wgrant: have you ever looked at having un-proxied objects by default in the harness?
<blr> presumably there's something that makes that impossible or difficult
<blr> hmm weird, have some doctests failing with proxied instance reprs instead of a diff, have you seen that before?
<wgrant> blr: What are the failures?
<blr> wgrant: very uninformative https://pastebin.canonical.com/132631/
<wgrant> blr: Hum, that's certainly odd.
<wgrant> (also, you might want to merge devel -- +all-branches no longer exists)
<blr> probably a result of changing the menu context from a product.development_focus (branch) to a product
<wgrant> Oh
<wgrant> You have a print somewhere
<blr> wgrant: hmm ok, maybe it will go away :P
<wgrant> browser.open isn't meant to print anything.
<wgrant> You have some leftover debugging code.
<blr> ah
<blr> I do to, thank you :)
<blr> s/to/too/
<blr> wgrant: small branch to review here: https://code.launchpad.net/~blr/launchpad/dont-fear-the-repr/+merge/261174
<blr> also if you're able to revisit project setbranch at some point (monday is okay)
<cjwatson> wgrant: https://code.qastaging.launchpad.net/germinate/+git shouldn't show lp:germinate at the bottom, should it?
<cjwatson> It's not an "other repository".
<wgrant> cjwatson: Probably not, but it required a GitCollection enhancement.
<cjwatson> Or perhaps "All repositories" if it's hard to exclude that.
<wgrant> It's hard-ish for bzr.
<wgrant> For git we can just say "NOT target_default"
<wgrant> Trying to avoid the hideousness that plagues +branches.
<cjwatson> Sure
<wgrant> Luckily we don't need a whole lot of stupid joins for git, so it's doable.
<cjwatson> wgrant: There are still several users of DAS.supports_virtualized in the tree
<cjwatson> wgrant: Oh, never mind, that's not the actual DB column
<wgrant> cjwatson: Indeed, all a property now. I might go back and update them all at some point, but I don't see much need.
<wgrant> cjwatson: Thanks.
<cjwatson> wgrant: I wonder if we should have some way to get to the default repository's edit views from Project:+git, since we're generally trying to present that as a superset
<wgrant> cjwatson: Right, I believe we want to include all of the portlets, given the approach we're taking.
<wgrant> cjwatson: But that gets a bit weird with subscribers and two privacy portlets.
<wgrant> So I haven't done it yet.
#launchpad-dev 2015-06-06
<cjwatson> wgrant: So I was looking at https://bugs.launchpad.net/launchpad/+bug/42298 after a friend whinged at me about it, which took me to https://code.launchpad.net/~stevenk/launchpad/destroy-dsp_picker-ff/+merge/128138, and I believe I have a modification to the query which produces reasonable results and executes in more like 160ms than the previous one which took multiple seconds
<mup> Bug #42298: package picker lists unpublished (invalid) packages <lp-bugs> <target-picker> <vocabulary> <Launchpad itself:Triaged> <https://launchpad.net/bugs/42298>
<cjwatson> wgrant: The piece I don't understand is the comment that it "breaks the branch case".  Do you remember background on that?
<wgrant> cjwatson: charms
<wgrant> cjwatson: /charms is a distro which has no actual packages.
<wgrant> It tracks bugs for things that have no publications, just branches.
<cjwatson> aha
<wgrant> Since we don't materialise which package names are actually valid in a given context, this makes things stupid.
<wgrant> There are hacks in place specifically for /charms that allow a bug to be targeted to a non-existent package if there is an official branch, IIRC.
<wgrant> Maybe any branch at all.
<cjwatson> guessPublishedSourcePackageName does a thing with branches, indeed
<wgrant> Yep
<wgrant> Just found that.
<cjwatson> Perhaps I have enough headroom in my modified query to deal with that
<wgrant> Official only.
<wgrant> The query should be very quick now.
<cjwatson> I basically replaced the join constraints with dspc.fti @@ to_tsquery('default', 'blah')
<wgrant> Oh
<wgrant> Heh
<wgrant> That index very nearly disappeared yesterday
<cjwatson> Was it just below the cut?
<wgrant> No, it was entirely unused AFAICS, but small and cold enough that I didn't bother deleting it.
<cjwatson> Well, perhaps you could see your way clear to leaving it there, it appears to be potentially quite handy :)
<cjwatson> (There's also a bug in the rank-25 case in the old query, but easy to fix)
<wgrant> Right, it was also potentially useful. The other victims not so much.
<wgrant> What does the new query look like?
<cjwatson> The official-only thing is fine for guessPublishedSourcePackageName, because that just results in a complaint in the UI if you file against an unofficial thing, but the vocabulary probably has to be looser.
<wgrant> SPN/BPN prefix match plus FTI?
<wgrant> Oh wow
<wgrant> That old query is sort of impressive.
<cjwatson> http://paste.ubuntu.com/11602694/
<wgrant> I forget when it was added, but that's unjustifiably bad if post-2011.
<cjwatson> An earlier version of it was https://code.launchpad.net/~stevenk/launchpad/dsp-vocab/+merge/65762; I didn't trace its full mutation
<cjwatson> I haven't tried my modification with pathological cases like single characters yet.
<wgrant> cjwatson: Does DSPC actually buy us anything there?
<wgrant> Avoiding it would probably actually be faster.
<cjwatson> Really?  I assumed the FTI would massively reduce the search space
<wgrant> xPPH.xpn exist now
<cjwatson> And avoid having to join through *PPH
<cjwatson> Hm, that's true.  I'll give that a try later
<wgrant> So you can do a very cheap query for an active publication with the right name in the right place in a fraction of a millisecond.
<wgrant> The entire search should take less than 50ms.
<wgrant> FTI is silly here when we don't want stemming or anythgin.
<wgrant> GIN is slow and not partitioned.
<StevenK> Ah yeah, xPPH.xpn got added later, and I think we decided to ignore/remove the DSP vocab
<cjwatson> The picker needs to be able to handle the case where the user didn't get the name exactly right thoug.
<cjwatson> *though
<cjwatson> I don't think exact match only is good enough
<wgrant> That's true.
<wgrant> But normal English stemming isn't likely to give a good result.
<cjwatson> I thought DSPC.fti was handled in a fairly custom way already, but haven't deciphered it in full
<cjwatson> Hm, maybe not
<cjwatson> Normal English stemming does give a fair bit of junk in this case, but the rank function sorts most of that to the bottom
<wgrant> Right, I'm not so concerned about the junk, but whether it actually gives any good results.
<cjwatson> It seems to, though I only tried a couple.
<cjwatson> linux-image, nvidia-graphics-drivers
<wgrant> Ah
<wgrant> Because it splits on -
<wgrant> So I guess that's not totally invalid.
<wgrant> http://paste.ubuntu.com/11602762/
<wgrant> Must be matching on linux and imag, I guess.
<wgrant> I didn't think it'd split on -, so it's more useful than I believed.
<wgrant> It's also a lot faster and less dodgy than the substring matching.
<wgrant> So probably worth a branch, since it's quick.
#launchpad-dev 2015-06-07
<blr> orning
<blr> morning even
<wgrant> blr: Morning.
<wgrant> blr: Turns out I'm actually not here today.
<blr> wgrant: even when you're not here, you're still here! :)
<blr> wgrant: talk to you tomorrow... I'm just chasing doctest failures this morning.
<wgrant> blr: In buildbot?
<blr> wgrant: no, just regressions from the UI changes to the code configuration on [code]/project
<wgrant> blr: buildbot is broken too, but they're easy fixes.
<blr> ah the smoke test
<blr> will have a look
<blr> thanks
#launchpad-dev 2016-06-07
<cjwatson> wgrant: you said "the usual 3-character threshold" the other day - do you know where that's implemented?  it is, perhaps obviously, near-impossible to search for
<wgrant> cjwatson: That's a good question. It could be in the JS picker frontend.
<cjwatson> thanks, will hunt
<cjwatson> min_search_chars, indeed.
<cjwatson> Hm.  This is all very reminiscent of the DSP picker work from 2011 that I tried to resurrect once ... maybe it makes more sense to try to finish that than to continue with the BinaryAndSourcePackageName vocabulary.
<cjwatson> SourcePackage.setBranch ensures a DistributionSourcePackage row, so that would be charms-safe.
 * cjwatson digs up year-old branch for that and is pleasantly surprised to find it not entirely exploding
<wgrant> cjwatson: Still using the updated cache tables, though?
<wgrant> DistributionSourcePackageInDatabase doesn't know about binary names AFAICR
<cjwatson> The stuff I dug up uses DistributionSourcePackageCache
<wgrant> (and the table isn't necessary for DistributionSourcePackage objects)
<cjwatson> But it goes through that for binary names rather than using DistroSeriesPackageCache
<cjwatson> Which seems potentially likely to perform better; will have to see
<cjwatson> I didn't at all understand the constraints at the time I was last working on this (I was just resurrecting some old work by Purple in an attempt to deal with https://bugs.launchpad.net/launchpad/+bug/42298), but there's a query here that's ranking exact source name, exact binary name, partial source name, partial binary name
<mup> Bug #42298: package picker lists unpublished (invalid) packages <lp-bugs> <target-picker> <vocabulary> <Launchpad itself:Triaged> <https://launchpad.net/bugs/42298>
<cjwatson> cache-aware and archive-aware
#launchpad-dev 2016-06-08
<timrc> wgrant, cjwatson Did I ever show you this? https://gist.github.com/penk/8415099
<timrc> That was a front-end to a tool they made to cache all their LP bugs locally at the Taipei office because querying Launchpad from Taipei was a painfully slow thing to do for them.
<cjwatson> I guess it was inevitable :)
#launchpad-dev 2017-06-07
<gQuigs> decided to try something smaller (https://bugs.launchpad.net/launchpad/+bug/1696567) before tackling adding HTTPS to mirrors..
<mup> Bug #1696567: Remove slow (outdated) speeds from mirror options <Launchpad itself:New> <https://launchpad.net/bugs/1696567>
<gQuigs> is it still running 12.04 in prod (per https://dev.launchpad.net/Running) ?
<cjwatson> gQuigs: We're part-way upgraded to 16.04.
#launchpad-dev 2017-06-08
<gQuigs> ty
<mz_> which package include "bzr import-package" command?
<cjwatson> mz_: Do you possibly mean "bzr import-dsc", which is in bzr-builddeb?
<mz_> cjwaston: https://wiki.ubuntu.com/DistributedDevelopment/Specification , describe that "The bulk of the import code already exists in the bzr builddeb plugin". But the package "bzr-builddeb" not include the comand of bzr import-package
<cjwatson> mz_: Well, that project is abandoned, so I can't help if it doesn't quite follow the spec.
<cjwatson> Maybe that bit of the design was never implemented, or implemented in a slightly different shape.  I don't remember.
<cjwatson> But that is a software design document, not user documentation for something existing.
<mz_> the Phase1 , packages can't be imported to lauchpad?
<cjwatson> The import was run for a while, but has now been abandoned.  There are too many problems with it and bzr is not sufficiently maintained to fix them.
<cjwatson> The Ubuntu server team is working on a replacement process using git.
<cjwatson> Unless you happen to have a spare multi-person bzr maintenance team lying around, I would not advise trying to revive the bzr import project.
<mz_> But how to importing Ubuntu packages in to Bazaar?  Is there any specification for that?
<cjwatson> I've said everything I can say.
<cjwatson> The abandoned project is in lp:udd (which come to think of it does seem to have something called import-package).
<cjwatson> But if you try to use it you will be entirely on your own.  Expect fatal bugs and no support.
<mz_> Ok, I understand, thanks for your help.
<sil2100> Hey! A quick question: is this still a thing? LP: #254901 - just checking if anyone knows instantly ;)
<mup> Bug #254901: appending tags to bug.tags is not supported properly on lp_save() <oem-services> <launchpadlib :Triaged> <https://launchpad.net/bugs/254901>
<sil2100> Ok, I see it's still the case, nvm ;)
<xnox> wgrant, i want to be able for a private bug in a private project, to have a second status with a remote bug tracker.
<xnox> as far as i can tell, that's not possible.
<xnox> is it at all possible to create private remote bug trackers and add "affects remote bug tracker" on private bugs in private projects?
<wgrant> xnox: No, a private project can't have bugs that affect other projects.
<wgrant> xnox: The owner of the other project would then have a bug in their project that they couldn't see because it would reveal the existence of the private project.
#launchpad-dev 2020-06-01
<tomwardill> Going to start landing the yarn/sass upgrades and testing on dogfood
<tomwardill> object now if this is a bad plan :)
<SpecialK|Canon> SGTM!
<tomwardill> okay, last buildbot run failed with a bunch of JS timeouts, going to retry
<tomwardill> bah, failed again
<tomwardill> cjwatson: is there anything special I need to do to ensure the nodeJS version is upgraded in the buildbot base images?
<cjwatson> tomwardill: https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/Buildbot#Updating_the_LXC_environment
<cjwatson> tomwardill: You'll need to ask IS to do that
<tomwardill> aha, that would probably be the casue
<cjwatson> (On both workers)
<tomwardill> looking more promising now
<tomwardill> \o/ success
 * tomwardill lands more things
<tomwardill> okay, still getting timeouts
<tomwardill> got an actual problem then
<tomwardill> cjwatson: how do I run these tests via firefox?
<cjwatson> find the absolute path to the associated .html, paste that in
<cjwatson> [4/4] Building fresh packages...
<cjwatson> [1/1] â  node-sass: g++ '-DNODE_GYP_MODULE_NAME=libsass' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DLIBSASS_VERSION="3.5.5"' -I/home/cjwatson/.node-g
<cjwatson> tomwardill: ^- is node-sass meant to be doing that?  I thought you'd squashed that
<tomwardill> yes, I had/have
<tomwardill> cjwatson: have you got the latest source-dependencies?
<cjwatson> Yes
<cjwatson> After an initial false start anyway
<cjwatson> And I upgraded my base container
<cjwatson> Maybe a cache that defeats your attempts to tell whether your fix worked?
<tomwardill> cjwatson: can you add --verbose to the end of the yarn line in the css_combine Makefile
<tomwardill> it should output whether it's found the .node binding before it starts compilation
<cjwatson> Oh.  That overflowed my terminal's scrollback
 * cjwatson tries again with |less
<tomwardill> oh, yes
<tomwardill> it is very verbose
<tomwardill> but somehwere in the middle will be the output about the binding
<tomwardill> my local lxd has decided it's going to drop all of it's internet connection
<tomwardill> which is not ideal
<cjwatson> verbose 3.23 Copying "/home/cjwatson/.cache/yarn/v6/npm-node-sass-4.14.1-99c87ec2efb7047ed638fb4c9db7f3a42e2217b5-integrity/node_modules/node-sass/binding.gyp" to "/home/cjwatson/src/canonical/launchpad/git/launchpad/yarn/node_modules/node-sass/binding.gyp".
<cjwatson> looks possibly relevant
<cjwatson> Hm, maybe, not sure
<cjwatson> [4/4] Building fresh packages...
<cjwatson> verbose 6.952 node-sass build Binary found at /home/cjwatson/src/canonical/launchpad/git/launchpad/download-cache/yarn/node-sass-4.14.4-linux-x64-57_binding.node
<cjwatson> verbose 7.674 Binary found at /home/cjwatson/src/canonical/launchpad/git/launchpad/download-cache/yarn/node-sass-4.14.4-linux-x64-57_binding.node
<cjwatson> That file shows as modified in git
<tomwardill> ah, so if it ever compiles it, it will replace your existing one
 * cjwatson copies it aside and restores it
<tomwardill> becasue input and output are synonymous, apparently
<cjwatson> Just going to try nuking ~/.cache/yarn/
<cjwatson> Ah of course
<cjwatson> My container is 32-bit
<tomwardill> ah
<tomwardill> yes, that would do it
<cjwatson> But it helpfully writes it back to the path you gave anyway, so replaces *-x64-* with a 32-bit build
<cjwatson> tomwardill: So um where did the 4.14.4 bit come from?
<cjwatson> https://www.npmjs.com/package/node-sass <- newest version is 4.14.1
<tomwardill> yes, I just looked at that and realised my mistake there
 * tomwardill gives up on today, it's clearly faulty and needs to be returned
<tomwardill> cjwatson: okay, now I've finally managed to test it (via X11 forwarding firefox), the JS tests still pass locally
<cjwatson> tomwardill: I'd suggest building two worktrees, one from before it started failing, one from current master
<cjwatson> tomwardill: And then diff the two after running 'make' in each
<cjwatson> I'll fix up the versions and arches and stuff
<cjwatson> https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+merge/384913 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/384914
<tomwardill> cjwatson: diff doesn't show anything immediately obvious and I can't make the JS tests fail locally
<tomwardill> cjwatson: I'm out of ideas for how to reproduce this
<cjwatson> Trying to see if I can find anything
 * tomwardill fetches more biscuits
<cjwatson> tomwardill: Ah look
<cjwatson> tomwardill: http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1295/steps/shell_8/logs/stdio
<cjwatson> From the build step before the test step
<tomwardill> hah
<cjwatson> tomwardill: So same problem as in my setup before the MPs I posted above
<tomwardill> LOOK A GIANT BLOCK OF RED TEST
<tomwardill> *TEXT
<cjwatson> Except it doesn't have the bits necessary to build it
<tomwardill> serves me right for just ctrl-f for 'Failure'
<cjwatson> Or rather, it tries to connect to the network and network says no
<tomwardill> right, so it's a 32 bit container
<cjwatson> It's not often you need to look at the output of the build step, but when you've just frobbed the build system and things are being weird ...
<tomwardill> indeed
<tomwardill> I've just +1'd your MPs
<tomwardill> are we actually deployed in a 32 bit world then?
<cjwatson> No
<cjwatson> I suspect buildbot does this because it's doing something like 20 tests in parallel and running them in 32-bit mode uses less memory
<tomwardill> ah
<tomwardill> yeah, that would make sense
<cjwatson> I admit I hadn't actually realised buildbot was 32-bit (or maybe I knew once and forgot)
 * cjwatson adds a note about that to https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/Buildbot
<cjwatson> Also I kind of hate that the test proceeds even though the build fails.  If somebody wants to figure out how to fix that, it might be a good intro to lpbuildbot
<tomwardill> cjwatson: I've just mentally added it to my 'work out buildbot and move it to bionic/focal' task
<tomwardill> which I might do next, I think it's sufficiently annoyed me for me to have a look at it now :)
<cjwatson> It is *meant* to halt the build on failure
<cjwatson> Look at bzrbuildbot.shell.ShellCommand (yes, bzrbuildbot is now horribly misnamed but ...)
<tomwardill> can you point me at the code/setup/docs for buildbot?
<cjwatson> Public branch in lp:lpbuildbot, some private customisations in lp:~canonical-launchpad-branches/lpbuildbot/production (make changes in the former if you can - it gets merged into the latter)
<cjwatson> I think all other documentation that exists is probably in https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/Buildbot
<cjwatson> oh.  Actually the bug is probably in lp-setup-lxd-build
<cjwatson> lp-setup-lxc-build, rather
<cjwatson> Which is in the most horribly confusing bit of the whole setup
<tomwardill> righto :)
<cjwatson> When parallel buildbot tests were being developed, there was an effort to get all the code needed for that sort of thing out of puppet and into an "lpsetup" package
<cjwatson> And this was sort of 80% done
<cjwatson> It's in lp:lpsetup and deployed via a recipe build into ppa:launchpad/ubuntu/ppa
<cjwatson> Hmm
<SpecialK|Canon> Can this end up on a wiki page if it isn't already please?
<cjwatson> Or maybe those scripts aren't
<cjwatson> If I understood it well enough to not be misleading I would have documented it a long time ago :-/
<cjwatson> I think possibly lp-setup-lxc-* never got fully extracted from puppet
<cjwatson> Which is bad because we kind of need to fix those, partly for this but also to use lxd
<SpecialK|Canon> "this is a thing probably worth looking at" / "here is possibly what happened" are still valuable documentation
<SpecialK|Canon> Er, sentence fail, but hopefully you get the idea
<SpecialK|Canon> I'd rather have your vague half-understandings on a page (with appropriate notes as to confidence in the quality) than not at all!
<cjwatson> OK, I'll see what I can do
<SpecialK|Canon> Thank you!
<cjwatson> tomwardill: I'd suggest getting an SRE to give you the current contents of /usr/local/bin/lp-setup-* in the appropriate container on e.g. sluagh, and then comparing that with lp:lpsetup and lp:canonical-is-puppet to work out which one of those it comes from
<cjwatson> Should be able to tell from the comments at the top of lp-setup-lxc-build, say
<tomwardill> that makes sense
 * tomwardill does that now
<cjwatson> And if they aren't the lpsetup versions, we should check the effective diffs between them, and probably make them be the lpsetup version
<cjwatson> Because while lpsetup is kinda weird it's better than a defunct-for-us config management system
<tomwardill> poked buildbot
<cjwatson> Thanks
<cjwatson> Build step worked now, so that should hopefully work better
<tomwardill> excellent :)
<tomwardill> wonder why it worked that once
<cjwatson> It's interesting, since it failed before the sass move
<cjwatson> http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1291/steps/shell_8/logs/stdio is a somewhat different failure
<cjwatson> Oh right
<cjwatson> So two different failures
<cjwatson> It failed because you landed the yarn upgrade before buildbot workers had a newer nodejs
<cjwatson> You got that fixed, and then nodejs on the worker was able to parse yarn and the build passed
<cjwatson> Then you landed the node-sass upgrade, which failed due to the 32-bit thing
<tomwardill> aah, right
<tomwardill> of course
<cjwatson> All appropriately deterministic :)
<tomwardill> but because I never looked at the build step, they had the same apparent failure mode to me
<cjwatson> Right
<tomwardill> wonder what new and exciting failure I can come up with the next landing!
<tomwardill> stay tuned to find out!
<cjwatson> I'd have let you at least fix it had I realised that it was the buildbot failure rather than just weird thing on my machine :)
<tomwardill> heh, no worries :)
 * tomwardill has also broken his glasses today, so it's not like it's been the best day all roudn
<cjwatson> Ugh
<tomwardill> buildbot success!
<tomwardill> landing the next in the series
<SpecialK|Canon> \o/
<tomwardill> anyone got time for this horrible diff? https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/384834
<pappacena> I can check that
 * pappacena immediatelly regret this decision
<pappacena> https://usercontent.irccloud-cdn.com/file/S535aaXN/image.png
<pappacena> hahaha
#launchpad-dev 2020-06-02
<tomwardill> landing the last of the sass branches
<tomwardill> wgrant: if you get a mo for a hopefully simple DB review: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/384129
<wgrant> tomwardill: Reviewed.
<tomwardill> ta
<tomwardill> blergh, actual test failures for master
<tomwardill> looking now
<wgrant> Hm, I thought I'd cleaned up the buildbot failures earlier
<wgrant> Oh a landing broke it, nvm
<tomwardill> yeah, at least one of them is reproducible locally
<wgrant> ... slightly odd landing to break it, though.
<tomwardill> lib/lp/app/javascript/picker/tests/test_personpicker.html
<tomwardill> JS failure somehow, so not entirely weird
<tomwardill> although it does mean I have to work out how to debug these tests
<wgrant> You can usually run them in your normal browser just by opening the listed HTML file
<tomwardill> yeah, I've got that
<tomwardill> "pickerPatcherPersonPickerTests"
<tomwardill> ... picked a pickled pepper?
<tomwardill> well, found 2 other bugs, but not fixed that one yet
<tomwardill> found it
<SpecialK|Canon> \o/
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/384954 fix css imports in tests
<cjwatson> Making decent progress with the OCI registry upload status UI, but having to fix a few bits of model along the way first
<tomwardill> slightly surprised that change only broke two tests, but there we go.
<cjwatson> r=me
<tomwardill> cjwatson: just pushed a change to flip the order of 'base' and 'components' (and fix the comment), as in the old method the components css was appended to the list
<tomwardill> so base was never actually the last import, despite what the comment said
<cjwatson> I wondered about that.  Also fine.
<cjwatson> Oh also
<tomwardill> thanks, landing
<cjwatson> Can you fix the node-sass version in that comment?
<tomwardill> .. not landing
<tomwardill> yes!
<tomwardill> done
<cjwatson> Thanks, go ahead
 * tomwardill -> coffee and then DB patch
<tomwardill> \\Ã¸/ build success
<cjwatson> Will probably be worth having a manual look over qastaging once it updates
<cjwatson> Just to make sure there are no obvious oddities from the uplift
<tomwardill> wgrant: the ocifile date_last_used shouldn't contend too much, I can't see us building enough oci images often enough to hit lock contention on that table
<tomwardill> and the garbage job collects every 7 days, so it shouldn't bloat with 'dead' images either
<tomwardill> cjwatson: yeah, I'll be doing that
<wgrant> Oh, that's not what I mean by bloat.
<wgrant> I mean PostgreSQL relation tuple bloat.
 * tomwardill does not know what that means, I'm afraid
<wgrant> Oh right, Mr. MongoDB Background :)
<tomwardill> it's hard to have bloat when you have no data
<wgrant> tomwardill: PostgreSQL's MVCC implementation means that a row's data on disk is actually immutable. The physical manifestations of a row are often referred to as "tuples", and contain the actual row data, plus metadata like the transaction IDs where the tuple was created and deleted. An update to a row actually marks the old tuple as deleted and creates a new one, and updates any indexes to point at
<wgrant> both. VACUUM is the process that runs (usually automatically) to find old tuples that aren't visible to any current transactions, and properly deletes them.
<wgrant> This works nicely, except when you have long-running transactions like, say, a backup. Since the backup runs in a transaction that references an old snapshot of the DB, tuples that were deleted during the backup cannot be properly deleted until the backup completes.
<wgrant> This doesn't just cause the table to grow in size - it can also slow things down significantly, as the index will reference all the tuples and queries will have to fetch them all to work out which one is alive.
<wgrant> This has mostly been a problem on tables like Builder and BuildQueue, where existing rows get updated frequently and automatically by buildd-manager.
<tomwardill> aah, right
<wgrant> It is also seen occasionally when an appserver hangs in the middle of a transaction for a couple of days. The whole app can start to slow down due to all the dead tuples lying around.
<tomwardill> that makes sense
<tomwardill> I don't think we're going to be updating that table with that level of frequency
<tomwardill> (pending an OCI based team decided they need minutely rebuilds or something)
<wgrant> But what if there are a million users building things based on ubuntu:18.04!
<wgrant> (but yes, unlikely to be a problem at the start, and we can always only update it if it's at least an hour old and SKIP LOCKED, or something like that, down the line if we need to)
<wgrant> But something to think about in future.
<tomwardill> yes, and not something I knew to consider, so thanks for the explanations :)
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/384952 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/384957 are a couple of fixes to various bits of the OCI upload arrangements.  They're independent, but I'd like to get at least one of them reviewed please so that I can linearise work based on both of them
<tomwardill> looking now
<tomwardill> qastaging is showing a 500 error again
<cjwatson> Right, but this is a normal one with a proper error page
<cjwatson> qastaging doesn't update itself in a nodowntime kind of way, so it routinely does this in the middle of updates
<cjwatson> However - atemoya.canonical.com::qastaging-logs/qastaging-update.log doesn't show it as being in the middle of an update right now
<cjwatson> Oh, hm, the sudoers fix apparently didn't quite take
<cjwatson> -> #launchpad-ops
#launchpad-dev 2020-06-03
<tomwardill> Fix my silly in compressed CSS output: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/385040
<cjwatson> tomwardill: r=me
<cjwatson> Is "--output-style compressed" what other people call minimised?
<tomwardill> yeah
<cjwatson> Right
 * tomwardill -> lunch and afk for errands
<tomwardill> skins should be consistent and have the files we expect: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/385050
<tomwardill> in a choice between hacking the build system around even more and just making the two components that were odd more consistent, I chose the latter
<cjwatson> r=me
<cjwatson> rubber-stampily
<tomwardill> thanks, landing
 * tomwardill is unsure why our skin is called sam
<tomwardill> but there we go
<cjwatson> There's a skin by that name in the YUI docs, so I assume something to do with that
<tomwardill> ah, that'd probably be it
#launchpad-dev 2020-06-04
<tomwardill> I am debugging bash
<tomwardill> this is not a fun time
 * pappacena echo. Echo everywhere...
<tomwardill> 2020-06-04 15:39:37+0000 [Uninitialized] Connection to buildbot-master:9989 failed: Connection Refused
<tomwardill> that's... progress
<SpecialK|Canon> a _new_ error! :D
<pappacena> haha. One step at a time...
<SpecialK|Canon> then you end up running literally everything with `set -ex` at the top and just grepping over terminal output...
<cjwatson> BTDT
<tomwardill> A THING! https://usercontent.irccloud-cdn.com/file/WUYV7QGs/image.png
<tomwardill> it appears to be checking out git code...
<tomwardill> attempting a build...
<tomwardill> not done anything special with lxc scripts yet, so no idea how badly this will go
<tomwardill> failed at the second step!
<SpecialK|Canon> but passed the first step! nice :)
#launchpad-dev 2020-06-05
<tomwardill> "failed shell_8 "
<tomwardill> progress!
 * tomwardill -> lunch and afk for errands
<tomwardill> Did not anticipate quite such a queue.
<tomwardill> wow, the LXC cli experience is ... something
<tomwardill> perhaps unsuprisingly, there are no available guides for 'lxc in lxd'
