#launchpad-dev 2010-06-14
<lifeless> is it just me, or are pages like https://bugs.edge.launchpad.net/bzr/+bug/4800 rendering really ugly for other folk ?
<_mup_> Bug #4800: Unwanted side effect of umask and 'bzr rspush' <patch-forwarded-upstream> <Bazaar:Invalid> <BzrTools:New> <bzrtools (Ubuntu):Triaged> <https://launchpad.net/bugs/4800>
<lifeless> the right hand side particularly
<wgrant> Already fixed.
<wgrant> Broke yesterday.
<ajmitch> the overlapping areas?
<wgrant> Yep.
<thumper> lifeless: sinzui has submitted a fix already
<lifeless> cool cool
<wgrant> Already in stable, so it'll be on edge tonight.
<stub> Should our test harness set the random number seed, or would that allow bugs to leak through?
<lifeless> Neither
<lifeless> it should be random by default
<lifeless> and you should be able to specify it if you need to reproduce a specific test run.
<lifeless> if you need a particular non-random-from-random-source sequence in a test, make the randomness provider replaceable for that code path.
<lifeless> IMO, YMMV, HTH
<stub> Because bugs might leak through?
<lifeless> largely
<stub> I know it wouldn't be an ideal architecture, but it might be practical. There are a few timebombs (which we triggered recently) we wouldn't have to worry about. I can't recall any bugs where random randomness saved us, but we don't have a large sample size.
<lifeless> more importantly though
<lifeless> if you set the random seed, you'd have to do it for every test separately
<thumper> the randomness was for subsequent make harness runs (I believe)
<thumper> so the factory didn't bitch
<thumper> about a user already existing
<thumper> when we just wanted some user
<lifeless> otherwise every time a test that uses randomness is added before an existing test using randomness, that latter test will blow up, if it depends on the output.
<thumper> I'm with lifeless that the factory should have a seed set when created
<thumper> if it needs to be tweaked
<thumper> it should be alterable
<thumper> but that would be enough IMO
<lifeless> sorry, are we talking test runs, or local-instance stuff ?
<lifeless> local-instance I don't care a hoot about in terms of this discussion - whatever works is fine
<lifeless> but test run changes can have serious maintenance impact
<thumper> lifeless: we are getting spuriour test failures because factory created 'things'
<thumper> lifeless: are now getting 'random' string names
<thumper> which when searching for the non-existance of 'foo' sometimes fails to work
<lifeless> random random, or serial-numbers from an in-python generator ?
<thumper> I've not looked recently
<thumper> but random enough that 'foo' turns up in the random characters
<lifeless> this sounds like an isolation issue between tests ?
<thumper> no
<thumper> it is the factory
<thumper> and a bug in that
<thumper> although a bug in the fact that it isn't doing what we want
<lifeless> ok, so a given test starts with a clean sample-db; makes something using the factory, and then asserts that 'foo' doesn't exist, and sometimes it does.
<thumper> in creating a project that won't match by name
<thumper> lifeless: yes
<lifeless> ok
<thumper> abentley and I talked about this on Friday and I think he had a branch in progress that fixed this
<thumper> back to appending numbers to strings
<lifeless> if you're suggesting that you should be able to say 'do not call it foo' in the calls to the factory, that sounds appropriate to me.
<lifeless> alternatively, you could ask the factory 'what is an unused name'
<thumper> and not random characters
<thumper> I don't think so
<thumper> the factory doesn't know what is an unused name
<thumper> and it isn't matching 'foo' exactly
<thumper> what we might say
<thumper> is 'foo' in some unique branch name
<thumper> one failure was a vocab search test
<lifeless> sure, just putting it out there.
<lifeless> in that 'make a unique thing' is a general factory function
<lifeless> and 'does not match' is still a case of making a unique thing
<lifeless> oh, and I hate my wifi driver
<stub> Or we can seed the random module (one line in one of the layers, or pass the seed as a parameter to Factory.__init__ and have readable tests...
<stub> vs. possibly not picking up genuine failures (although these failures may never show up anyway due to probability)
<stub> Still, first time it happened so maybe not worth changing anything and just tweaking the test
<lifeless> stub: I don't see that seeding the random module will avoid these issues
<lifeless> stub: can you lay out how it would help for me ?
<stub> The tests become deterministic
<lifeless> so if you pick a seed that works for existing tests, and every test has the same seed, what stops new tests from running into bad data at a different point in the sequence ?
<stub> You know what random.random() returns, thus Factory.makeSomethingWIthRandomName() returns an object with a random but known name,
<stub> The test will fail consistently so will never land
<lifeless> ok
<lifeless> I think I'd be more comfortable with the factory holding its own random generator and not futzing with the global generator
<lifeless> which sounds like what thumper was saying abentley was working on
<stub> If calling random.seed(xxx) in BaseLayer.testSetUp is futzing, then sure. BaseLayer would fix other occurrences too but maybe YAGNI
<stub> I have no idea if Aaron was fixing it that way or some other mechanism (generating random names from a smaller set of characters for instance, or prefixes or suffixes (although that doesn't help for substring matches, which I think is what we hit), or just refactoring the tests)
<lifeless> better to do
<lifeless> generator = random.Random(xxx)
<lifeless> and then return values from generator
<lifeless> that way you don't interact with any modules using the global random generator
<lifeless> and conversely, won't be broken if lp code itself were to start using more random values - throwing off the determinism you're going to be relying on.
<thumper> I'm not suggesting that every value be random
<thumper> just the first one
<thumper> we used to add a unique integer to the end of the base string
<thumper> this would create person1, person2 etc
<thumper> what I suggest is that we seed the initial value with a random number
<thumper> (which can be overridden if necessar)
<thumper> and still do the increment and add approach
<lifeless> I like that
<thumper> it is simple
<thumper> and simple is good
<lifeless> ok, nose.against(grindstone)
<adeuring> good morning
<kb9vqf> wgrant: The local Launchpad installation is online and should now be AGPL compliant...  https://quickbuild.pearsoncomputing.net/
<kb9vqf> In case you're interested ;-)
<kb9vqf> There is still the Ubuntu/Debian issue, but I'm working on that as I have time
<kb9vqf> Let me know if you see any potential copyright issues (e.g. I missed an image or something isn't quite right about the GPL stuff) and I'll fix them right away
<mrevell> Goodly morning
<thumper> hi mrevell
<deryck> Morning, all.
<maxb> Hi. I need a second opinion on a couple of vcs-imports reviews
<maxb> I did email the list, but no one responded
<maxb> So I'm prodding :-)
<jelmer> maxb: what import?
<maxb> jelmer: the blender win??libs ones
<jelmer> maxb: ah
<jelmer> maxb: I think it would be ok to have those imports as long as the license meets Launchpad's requirements and as long as the disk space isn't an issue.
<jelmer> maxb: Assuming the branch was requested by somebody who needed to have it imported.
<jml> hello launchpadders
<jelmer> 'morning Jono
<krkhan> in interface definitions, some times methods return collections of Interface. that was patched later on in a separate file to refer to the derived Interface (IBug etc.). i forgot where the patching process took place. anyone recalls?
<krkhan> nvm, found it at lib/canonical/launchpad/interfaces/_schema_circular_imports.py
<mars> jml, I think the topic is stale - thumper set it last Thursday
* ChanServ changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2 of 10.06 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<noodles775> gary_poster: Hi! If you have a spare minute, could you please take a look at the following zope security issue that I'm seeing: http://pastebin.ubuntu.com/449690/
<gary_poster> looking noodles775
<noodles775> Thanks gary_poster . I'm confused, as it seems to recognise the correct proxy, and have the correct permission, but cannot access package_build.
<noodles775> Ah... or does it have *two* security proxies... and it's using the first?
<gary_poster> noodles775: I'm still trying to grok the pastebin yougave me, but a common cause for this sort of behavior is *another* security proxied object
<gary_poster> yeah, I didn't understand that either
<gary_poster> noodles775: to try your hypothesis, see if you can access something else mentioned by the second proxy and not by the first.  Meanwhile, I'll go try to see what debug_proxy does
<noodles775> The first is actually a subset of the second... but I'll dig a bit too.
<sinzui> matsubara, ping
<matsubara> hi sinzui
<sinzui> matsubara, I do not see any staging oops reports for the last 14 days
<sinzui> matsubara, I want to see reports for today mostly, and may some from May 30-31
<matsubara> sinzui, you can see today's here: https://devpad.canonical.com/~lpqateam/staging-oops.html
<matsubara> sinzui, I'll look into why the previous days were empty
<sinzui> thankjs
<sinzui> matsubara, it may relate to the db update
<matsubara> sinzui, https://devpad.canonical.com/~lpqateam/oops-summaries/staging-2010-05-30.html and https://devpad.canonical.com/~lpqateam/oops-summaries/staging-2010-05-31.html
<matsubara> for the 30th and 31st
<sinzui> I see them in the folder thanks
<sinzui> matsubara, This is all I need to see that our MailingListAPIView optimisation does not work
<gary_poster> noodles775: I believe that the nested security proxies would explain this (and I checked the code reporting it, and it looked sane).  I have not encountered security proxies around security proxies before, so I don't know why or when they would occur, but maybe it has happened in LP before.
<sinzui> gary_poster, noodles775, thumper encounter this last week
<noodles775> gary_poster: yep, I can see why it's happening now. I'm manually setting a ProxyFactory() for the result of my adapter, but then it seems that getAdapter() also wraps the result in a proxy for the specified interface (which is different).
<gary_poster> sinzui: interesting.  was there a resolution?  I was about to ask you and Francis if you had encountered
<gary_poster> noodles775: ah-ha.
<gary_poster> noodles775: so the getAdapter proxy is the outer one?
<sinzui> I think he unwrapped it twice. but we was very dissatisfied and may have come up with a solution or at least an explanation
<noodles775> gary_poster: yes. I'd like getAdapter to not add its own :/
<sinzui> gary_poster, thumper was trying to reduce the number of queries we call for every page
<gary_poster> noodles775: when you register your adapter, is it registered as "trusted"?
<gary_poster>   If it is, then that might be a solution
<gary_poster> say it is not trusted
<gary_poster> and then rip off the proxies yourself on entrance to your adapter
<gary_poster> and then I don't think you will have external code that will add a proxy on exit
<gary_poster> If that's not the case, and not a solution, I'll look deeper, but that's my first guess
<noodles775> gary_poster: I tried with trusted="True", but it didn't seem to change (hrm, maybe it's "true").
<gary_poster> noodles775: you want False
 * noodles775 tries
<gary_poster> (don't remember fals vs. False)
<noodles775> How do I rip off the proxies?
<gary_poster> removeSecurityProxy
<noodles775> Ah, I'd not used it just for one proxy before... great...
 * noodles775 tries
<gary_poster> If this is the problem/solution maybe we can convince upstream to make the "trusted" behavior say "if the output is already security proxied then *leave it alone*".  (Right now it says, IIRC, proxy the output, period.)  Or maybe there's some other upstream solution
<gary_poster> (like, why is it making a proxy based on an interface, when we already have a registered proxy?)
<sinzui> matsubara, I just noticed that https://devpad.canonical.com/~lpqateam/staging-oops.html is from May 31. We do not have a report for what is happening today
<matsubara> sinzui, so it's very likely that oopses are not being synced from staging to devpad or there are no oopses on staging at all
<matsubara> sinzui, I'm pinging the losas to find out what's up with the oopses
<sinzui> thanks
<noodles775> gary_poster: Thanks! Confirming that removing the outer proxy ensures that the returned result is just proxied with my specified one. Thanks sinzui too for the hint.
<gary_poster> noodles775: great, so that is also confirmation that the "trusted" thing was the source of the additional proxy?
<noodles775> gary_poster: erm, no... I'll re-check now, but when I tried both trusted with true and false, I was still getting the wrapped proxy.
 * noodles775 checks again.
 * gary_poster would like to know the source of the naughty proxy
<noodles775> gary_poster: confirming that either way (trusted true or false), the calle to getAdapter() is proxied with the passed IFace... and actually, the docs seem to agree with this? http://apidoc.zope.org/++apidoc++/Interface/zope.component.zcml.IAdapterDirective/index.html
<noodles775> That is, trused just gives the adapter itself unfettered access to the object, it doesn't seem to change whether the result is proxied?
<gary_poster> noodles775: my understanding was that, for trusted adapters, the output was supposed to be proxied if the input was.  Checking implementation
<gary_poster> noodles775: my understanding is confirmed that a "trusted" adapter has custom code to handle reproxying the output if the input was proxied.  However, my guess is that your adapter registration also specifies permissions on it?
<gary_poster> noodles775: sorry for dwelling on this, but someone else encountered this, and it pretty clearly is something that is lame somewhere.  We should understand the cause at least, and quite possibly address it.
<matsubara> sinzui, syncing was disabled. it's now re-enabled and staging-oops.html should be updated within the hour
<sinzui> thanks!
<barry> sinzui: could you explain a little bit what was done to fix bug 590840?
<_mup_> Bug #590840: timeout MailingListAPIView <oops> <qa-ok> <timeout> <Launchpad Registry:Fix Committed by edwin-grubbs> <https://launchpad.net/bugs/590840>
<sinzui> barry, EdwinGrubbs optimised the set sender and receiver address in the ML model
<sinzui> barry, last db update we had 248 timeouts on staging. We have about 9  this tine
<sinzui> barry, The loop you added worked and still works, but the queries work more often on the first try now
<EdwinGrubbs> barry: the main improvement was retrieving only the three columns necessary from the db instead of all the columns from four tables.
<barry> sinzui: ah, so it's not trying to collate all the lists together, like the last (unsuccesful - i suck) attempt?
<sinzui> s/set sender and receiver/GET sender and receiver/
<sinzui> barry, right
<sinzui> Lp in py2.6 ready. I do not know when lpnet will be Lucid, but we may be ready for MM3 soon
<barry> sinzui: excellent.
<sinzui> Lp /in/is/
<barry> sinzui: very cool
<barry> sinzui: i'll have to allot some background time to that work now
 * sinzui has got to stop typing with kittens next to me
<barry> me. ow!
<sinzui> I have a deaf kitten who likes to sleep on my keyboard.
<maxb> I'm quite looking forward to the importds getting to be lucid, for Subversion 1.6
<sinzui> She often sets the volume to 11 and scares the other cats away
<mrevell> Night all
<matsubara> sinzui, I recreated staging oops summaries for the 7th to 13th. there are no oopses for 1st to 6th. You can see them at https://devpad.canonical.com/~lpqateam/oops-summaries/staging-YYYY-mm-dd.htmll
<sinzui> matsubara, thanks
<sinzui> I was able to verify that edwin's mailing list changes reduces timeouts by 95%
<matsubara> cool!
<jml> g'night all
<mars> gary_poster or leonardr, do you know if the lp-land command works for non-launchpad PQM projects as well?
<gary_poster> I don't mars
<leonardr> nor do i. i always land manually
<gary_poster> mars, it does not:
<gary_poster> $ bzr lp-land --help
<gary_poster> Purpose: Land the merge proposal for this branch via PQM.
<gary_poster> ...
<gary_poster> From:     plugin "pqm"
<mars> yes, so it should work, since the information comes from the proposal
<gary_poster> :-)
<zyga> is package building service still available?
<gary_poster> oh I misread your question
<gary_poster> so, maybe so
<gary_poster> zyga, you mean PPAs?  If so, definitely
<zyga> gary_poster, no, packages from bzr + recipe
<gary_poster> 'fraid I don't know, sorry
<zyga> https://wiki.ubuntu.com/DailyBuilds/GettingStarted
<mars> zyga, abentley or rockstar may know
<maxb> <abentley> jcastro, you mean with source package recipe builds?  I'm running a stress-test right now.  If that goes well, we could see it deployed on edge soon.
<maxb> (from #launchpad, not long ago)
<zyga> james_w, ^^ do you know something about this, you seem to (https://wiki.ubuntu.com/DailyBuilds/AvailableDailyBuilds) maintain lots of them
<zyga> abentley, rockstar: can I setup daily builds of my branch?
<abentley> zyga, that service is not currently available.
<zyga> ok, thanks
<abentley> zyga, we had hoped to have it deployed by now, but some changes were made that caused it to wreak havoc on the buildfarm, so we had to disable it.
<abentley> zyga, but we have fixes in place now, and we're in the middle of testing them.
 * mars jots down another case for something like http://www.google.com/appsstatus#hl=en
<zyga> abentley, thanks for the explanation
<maxb> Can anyone confirm, launchpad-database-dependencies needs to stay on 8.3 until such time as production migrates to 8.4? Or not?
<kb9vqf> So...any good reason why the keyserver deleted all my keys when the computer was restarted?
<wgrant> kb9vqf: The dev configuration keeps things in /var/tmp, and there's probably a Makefile target somewhere that purges /var/tmp/zeca, which contains the keys.
<kb9vqf> Ah, OK.  Thanks!
<kb9vqf> Nothing else was affected, so the keyserver must be unique that way
<wgrant> make clean erases some stuff.
<wgrant> Like that.
<wgrant> Maybe it shouldn't. But then it probably also shouldn't erase PPAs.
<kb9vqf> And I stupidly ran it to get the AGPL source tarball
<thumper> kb9vqf: can I ask what the purpose is of setting up your own Launchpad instance?
<kb9vqf> :-P
<kb9vqf> thumper: Debian packages
<thumper> kb9vqf: what isn't the public Launchpad offering?
<kb9vqf> thumper: Debian support
<kb9vqf> And possibly other distros
<thumper> how do you envision keeping things in sync?
<kb9vqf> Not sure yet
<kb9vqf> I might create a diff between my instance and the branch it originated from, and then keep applying that to the latest LP trunk
<kb9vqf> Don't know if that will work though
<kb9vqf> wgrant: Anything special about the ppa_key_guard celebrity?
<kb9vqf> e.g. how would I create it?
<wgrant> kb9vqf: It's where the PPA keys are attached.
<wgrant> It's just a normal unactivated user.
<kb9vqf> OK
<wgrant> Similar to the janitor.
<thumper> kb9vqf: and so I'm clear, this is intended to be a public service?
<kb9vqf> thumper: I'm considering it
<kb9vqf> At the very least the built PPA packages will need to be publically accessable for Trinity
<thumper> kb9vqf: what is Trinity?
<kb9vqf> http://trinity.pearsoncomputing.net
<kb9vqf> Continuation of KDE3
<kb9vqf> Lots of packages, would be a nightmare without Launchpad
<thumper> kb9vqf: and your reason for not using the public one is lack of debian build support?
<kb9vqf> thumper: Yes
<thumper> ok
<kb9vqf> thumper: And I was also considering extending the build system to handle RedHat
<kb9vqf> Although that will be far in the future
<kb9vqf> Do you see any legal problems with what I'm trying to do now?
<thumper> I'm not a lawyer
<thumper> the code is AGPL
<thumper> the code you run has to be publicly accessable
<kb9vqf> I know; just don't want to step on any toes
<thumper> perhaps just social fragmentation
<thumper> the intent of Launchpad is to be a central location and project registry
<thumper> having multiple public launchpad instances dilutes this ideal
<kb9vqf> Ah...
<kb9vqf> My purpose isn't really to deal with Ubuntu stuff though
<kb9vqf> Not on this instance
<kb9vqf> This is more for Debian and other distros
<kb9vqf> I'm still using the public Launchpad for Ubuntu packages
 * kb9vqf notes that the code is up as a tarball under the QuickBuild project at https://quickbuild.pearsoncomputing.net/quickbuild for AGPL compiance
<kb9vqf> wgrant: I tried to run this code:
<kb9vqf>     ppa_key_guard = factory.makePerson(
<kb9vqf>         name='ppa_key_guard', displayname='Launchpad PPA Key Guard',
<kb9vqf>         email='ppa_key_guard@example.com')
<kb9vqf> And I got: lp.registry.interfaces.person.InvalidName: ppa_key_guard is not a valid name for a person.
<kb9vqf> Which doesn't make much sense
<kb9vqf> wgrant: Is it supposed to be ppa-key-guard?
<kb9vqf> I guess it was
<wgrant> kb9vqf: Right. underscores in the Python name, hyphens in the Launchpad name.
<wgrant> Underscores are ugly.
<kb9vqf> Makes sense.  And it worked right up until when I ran out of disk space
<kb9vqf> :-P
<wgrant> Heh.
#launchpad-dev 2010-06-15
<krkhan> merges for implementing api calls should be proposed against devel or db-devel?
<krkhan> wgrant: branching lp:launchpad gives me a copy of db-devel. that's confusing me as to whether i should branch should i target for merge proposal
<wgrant> krkhan: If they need DB changes, db-devel.
<wgrant> Otherwise, devel.
<krkhan> wgrant: ok, thanks
<ajmitch> how should tests be written for extending the API?
<wgrant> ajmitch: See lib/lp/APP/stories/webservice
<ajmitch> thanks
<wgrant> You can use launchpadlib in tests now, though, so try to find a new one that does.
<wgrant> It makes everything much prettier.
<ajmitch> & less likely to write a bad test
<krkhan> ajmitch: if using launchpadlib for testing webservice, "make clean" is required for lplib to recognize new methods. wasted a few hours on that thing
<wgrant> 'make build' should do it.
<ajmitch> krkhan: yeah I knew that one, or at least removing & recreating the apidoc target
<bryceh> wgrant, yes, would be nice if it did
<ajmitch> but it still thiks the apidoc target is up-to-date
<wgrant> Hmm.
<ajmitch> krkhan: I was just looking at that getBugSubscriberPackages() method when I stumbled across what you'd done on it :)
<krkhan> ajmitch: i'm just going to propose a branch for exporting that method. it's useful :)
<ajmitch> yes, it came up in #ubuntu-devel this morning
<wgrant> bdmurray: That branch of yours says that it exports suppress_notify... but @call_with declares that it's not exported. I am confused.
<bdmurray> wgrant: well its still usable by launchpad itself but not API clients
<wgrant> Ah, so it was deliberately *un*exported?
<wgrant> That makes more sense.
<bdmurray> yes, sorry for any confusion
<krkhan> what would be the command to push a personal launchpad devel branch. bzr push lp:~user/launchpad/devel/branchxyz gives permission denied error
<wgrant> krkhan: Drop the devel/
<krkhan> wgrant: dropping devel just pushes a branch with the message "Using default stacking branch /~launchpad-pqm/launchpad/db-devel"
<krkhan> i don't want db-devel
<krkhan> that is bzr push lp:~user/launchpad/branchxyz creates a stacking branch referring to /~launchpad-pqm/launchpad/db-devel
<wgrant> That's fine.
<wgrant> Stacking just means it shares revisions, so you have to upload less.
<wgrant> It has no further meaning.
<krkhan> but then the merge proposal is created against db-devel too. i'm confused, i want the proposal to be against devel
<wgrant> You need to select devel when creating the merge proposal.
<krkhan> wgrant: oh okay, i got it. thanks
<lajjr> Hello jml and flacoste
<adeuring> good morning
<kb9vqf> So which cron script purges deleted pacakges from Librarian?
<kb9vqf> Or technically I guess from disk?
<wgrant> expire-archive-files, then librarian-gc
<wgrant> Are you also running process-death-row regularly?
<wgrant> But expire-archive-files should only remove stuff that's been gone a week, IIRC.
<kb9vqf> There it is; I had librarian-gc but not the other two
<wgrant> p-d-r removes the files from the archive.
<wgrant> e-a-f expires the librarian files once they're removed from the archive disk.
<wgrant> And l-gc removes expired librarian files from disk.
<kb9vqf> Actually it looks like I was only missing p-d-r
<kb9vqf> The other two I had found and was running on a cron job
<wgrant> The e-a-f stay of execution is configurable.
<kb9vqf> p-d-r didn't do anything; I still have 1.3gb of deleted packages floating in the ether
<kb9vqf> It said it found 0B to delete
<wgrant> What said that?
<wgrant> p-d-r?
<StevenK> process-death-row
<kb9vqf> Yes
<wgrant> I know what it expands to -- I was just wondering if that was what said that.
<StevenK> wgrant: Oh, never mind, too little context, too much context switching
<kb9vqf> Technically it said this:
<kb9vqf> 2010-06-15 07:26:43 INFO    Creating lockfile: /var/lock/launchpad-process-death-row.lock
<kb9vqf> 2010-06-15 07:26:52 INFO    Processing http://archive.quickbuild.pearsoncomputing.net/ubuntu
<kb9vqf> 2010-06-15 07:26:52 INFO    Removing 0 files marked for reaping
<kb9vqf> 2010-06-15 07:26:52 INFO    Total bytes freed: 0
<wgrant> kb9vqf: The packages are actually Deleted, not Superseded?
<kb9vqf> I deleted them manually
<wgrant> Superseded packages will only be considered for removal from disk after 24 hours.
<wgrant> Hmm.
<wgrant> You've run the publisher since deleting them?
<kb9vqf> Maybe the counter isn't updated?
<kb9vqf> Yeah, I have the publisher on a 1 minute timer right now
<kb9vqf> So I'm sure it ran ;-)
<wgrant> Can you point me at the PPA in question?
<kb9vqf> https://quickbuild.pearsoncomputing.net/~trinity/+archive/trinity/+packages
<kb9vqf> The size should be around a few hundred MB at most
<wgrant> Oh, you are running p-d-r with --ppa, right?
<kb9vqf> No!
<kb9vqf> Let me try that :-)
<wgrant> All the publishing scripts need to be called with that.
<wgrant> mmm, not using proper Debian PPAs yet? :(
<kb9vqf> OK, I wasn't sure if it was just the ones that were close to poppy (i.e. in the upload->build->publishing chain) or not
<kb9vqf> Not yet
<kb9vqf> Got some build issues to work out first
<wgrant> Anything that needs to touch the archive, basically. Since the PPA and primary archives are on separate machines in production.
<kb9vqf> That worked; thanks!
<StevenK> wgrant: I suspect that isn't the only reason.
<wgrant> Well, true. The primary archive publisher takes three eternities.
<wgrant> And there are probably other reasons lost in the mysteries of time.
<kb9vqf> Regarding Debian PPAs, I'm going to get the build system functioning correctly, then I have a couple weeks of other work to do, then I'll come back and try to hack in correct Debian support
<kb9vqf> There are a bunch of problems that are really kinda weird and may require significant rewrites
<kb9vqf> Starting with the fact that the upload processor doesn't find any distroseries not belonging to Ubuntu
<wgrant> Hmmm. I've not had that problem.
<wgrant> As long as the PPA's distribution is set correctly, and the upload path is right, there's no reason it shouldn't work.
<wgrant> And I'm pretty sure I tried that, although it was nearly a year ago.
<kb9vqf> I might have done something wrong but I'm fairly certain I tried debian in the upload path to no effect
<kb9vqf> I spent most of a day trying to get it to work
<wgrant> Maybe I should try that tonight.
<StevenK> kb9vqf: You know https://quickbuild.pearsoncomputing.net/debian doesn't work, right?
<StevenK> Whereas /ubuntu does, for example
<wgrant> I suspect the DB was reset 4 hours ago.
<kb9vqf> Why yes it was
<StevenK> Heh
<kb9vqf> I was hoping no one would notice
<kb9vqf> :-P
<StevenK> Because scorched earth solves everything
<kb9vqf> When I ran out of disk space a bunch of stuff broke
<kb9vqf> I couldn't get all the pieces back together again, so....
 * kb9vqf notes it's a really bad sign when the ubuntu celebrity is suddenly not found
<kb9vqf> StevenK: https://quickbuild.pearsoncomputing.net/debian works now ;-)
 * wgrant resets his DB and tries Debian PPAs again.
<wgrant> kb9vqf: You're manually setting external dependencies on each PPA to the real Debian archive, I guess?
<kb9vqf> wgrant: Yes; I set a flag file in the Debian chroot and when it is detected the builder overrides the provided sources.list with sed s/ubuntu/debian/g or similar
<kb9vqf> There's a bit more to it but it works pretty well
<kb9vqf> as hacks go, anyway... :-P
<wgrant> Ahahah. That is horrible :)
<kb9vqf> But it's allowing me to proceed to fixing the Trinity build system for the moment
<kb9vqf> It will be fixed ASAP :)
<wgrant> Yep.
 * kb9vqf does not miss the PPA build queue on broken package resubmission
<wgrant> Heh.
<kb9vqf> So much nicer for figuring out these weird libtool errors ;-)
<kb9vqf> Almost makes the past week worth it :-P
<wgrant> Oh lucilleconfig, I hate you so so much.
<mrevell> Morning
<wgrant> kb9vqf: So, it's working OK for me. I did get a missing distroseries error, but it was misleading: I'd used a bad upload path, so it went into an error handler which assumed Ubuntu to make error messages nicer.
<wgrant> So make sure you're looking at the first error message.
<kb9vqf> OK, I will try that when I have time.  Thanks for checking it out; my Python skills really stink
<kb9vqf> Come to think of it I might have actually created the distroseries incorrectly in the database the first time
<kb9vqf> We shall see in a couple weeks
<wgrant> Uhoh, the Launchpad Buildbot appears to have become sentient.
<wgrant> It's commenting on bugs.
<ajmitch> that's a bit worrying
<wgrant> Bug #593522
<_mup_> Bug #593522: Don't send out real email from staging codehosting <Launchpad Bazaar Integration:Fix Released by canonical-losas> <https://launchpad.net/bugs/593522>
<ajmitch> Now if only you could get it to fix bugs as well
<kb9vqf> Well it already appears to have an IQ higher than most Internet users...it speaks in complete sentences ;-)
<spm> that's coming version 2. version 3 is when it starts patching itself. live. version 4 is when it invents a new killer language + api's; rewrites itself into that. declares the rest of the world as backwards and hence obsolete; and turns the entire planet into a giant supercomputer. something pity about 42 in there as well I'm sure.
<ajmitch> spm: great, when can it write tests for LP code as well?
<spm> s/pity/pithy/
<spm> ajmitch: that'd be version 8 based on the above timeline. simple extrapolation.
 * ajmitch has been slacking just a little bit & has finally got back to bug 146389, apart from the little issue of writing tests
<_mup_> Bug #146389: api for blueprint tracker <api> <feature> <Launchpad Blueprints:In Progress by ajmitch> <https://launchpad.net/bugs/146389>
<wgrant> It's much less painful with launchpadlib.
<thumper> ajmitch: oh dear
<thumper> ajmitch: I started looking into that
<ajmitch> yeah, given that I've been testing manually with it as well
<thumper> ajmitch: and ran screaming
<ajmitch> thumper: great to know!
<ajmitch> how far did you get?
<thumper> ajmitch: as I felt that it needed a machette
<ajmitch> it does, really
<thumper> ajmitch: too much weird shit logic in the wrong place
<thumper> ajmitch: the model really needs an overhaul
<wgrant> Well.
<wgrant> It's Blueprint.
<thumper> wgrant: agreed
<wgrant> It needs a machette. To the whole app.
<ajmitch> but I've just been exporting properties & methods, going WTF in a few places
<thumper> wgrant: with wikis, I'll be better ;-)
<thumper> maybe
<ajmitch> wondering how I make an exported field editable, all that fun stuff
 * wgrant throws stuff at buildd-manager.
<thumper> ajmitch: there is special logic in most of the other setter methods
<thumper> ajmitch: which is why I didn't go ahead and make the methods writable
 * ajmitch just made a big, big mistake
<ajmitch> running 'bzr viz' on LP
<thumper> ajmitch: recording things like started date, and who did it
<thumper> finished date et al.
<wgrant> ajmitch: qlog works OK.
<ajmitch> thumper: one thing that the distro team appears to use, is splitting a single blueprint into multiple work items using the whiteboard
<ajmitch> ideally that'd be supported better than just a big text mess
<ajmitch> if someone will give some love to blueprints :)
<thumper> ajmitch: I'll mentor you :)
<ajmitch> shared drinking sessions for blueprint code? sounds like a plan
 * ajmitch thinks that it'll need a lot more thought put into the design
<wgrant> Or a good rm -r lib/lp/blueprints
<wgrant> Why are they separate?
<ajmitch> separate from bugs?
<wgrant> Yes.
<ajmitch> I think because they were at least meant to have features that didn't really map well to the bug workflow
<ajmitch> some things like the dependencies between blueprints would be nice to have in bugs
<wgrant> Exactly.
<wgrant> Most features unique to blueprints would be useful in bugs too.
<wgrant> Oh.
<wgrant> I hate you Twisted.
<wgrant> Why must you hang when attempting to serialise an XML-RPC request containing None...
<wgrant> mthaddon: So we needn't worry that the DC is going to come and conquer the world?
<mthaddon> wgrant: the machines are on the march
<wgrant> kb9vqf: Once I set up lucilleconfig on Debian and its series and configured nominatedarchindep on lenny, and hacked traverse_named_ppa to not care about the distribution, it all works fine.
<wgrant> bigjools: You don't object strongly to https://code.edge.launchpad.net/~wgrant/launchpad/bug-592935-hide-disabled-ppas/+merge/27411  or https://code.edge.launchpad.net/~wgrant/launchpad/no-buildd-ogre-model/+merge/27410 , do you?
 * bigjools OTP, gimme some 
<wgrant> k
<bigjools> wgrant: +1 to both
<bigjools> provided that the user can still see his own disabled PPAs
<wgrant> They can.
<wgrant> Thanks.
<bigjools> we need a better solution for that in the long term, but it'll do for now
<bigjools> thanks for the change
<wgrant> Yep.
<wgrant> bigjools: Is the log parser CP blocking on that dogfood issue?
<bigjools> wgrant: it's blocking because it doesn't work
<didrocks> jml: hey, to implement https://blueprints.launchpad.net/ubuntu/+spec/desktop-maverick-quickly as discussed at UDS (for pushing ssh/gpg key, signing CoC and creating a ppa), I will try to connect "the bad way" to launchpad :/ Is there any way to make it less hackish than screenscraping and avoiding breaking the user?
<wgrant> bigjools: It doesn't?
<jml> didrocks, I'm on the phone.
<didrocks> jml: no pb, will idle there :)
<bigjools> wgrant: no, I removed all existing db entries and re-ran it.  Same issue.
<bigjools> so there's a definite code problem
<wgrant> Ah, I didn't know that bit.
<wgrant> I wonder why it doesn't use fd.tell() instead of working it out manually.
<wgrant> bigjools: Any chance you can stick something like a "assert parsed_bytes == fd.tell()" after "parsed_bytes += len(line)", delete the ParsedApacheLog for that file, then rerun?
<wgrant> Would be nice to see where it actually starts going wrong...
<bigjools> wgrant: not in the near future, I'm way too busy
<wgrant> OK.
<bigjools> sorry
<wgrant> No worries.
<krkhan> is there a way for me to debug doc tests quickly? while writing unittests i did pdb.set_trace() inside one of the tests and then played around. for doc tests, pdb.set_trace() loses all context
<bigjools> krkhan: go up one in the stack and the context is there
<bigjools> so hit "u" after it breaks
<krkhan> bigjools: thanks :-D
<bigjools> it used to stop in the right stack frame, I wonder what changed :(
<didrocks> jml: still on the phone? :)
<jml> didrocks, no, doing performance reviews. :)
<jml> didrocks, I have to confess I forgot about your question
<didrocks> jml: just ping me when you will some free time, ok? ;)
<jml> didrocks, sure.
<jml> didrocks, actually, looking at your question, can you please just ask it on the launchpad-dev@lists.lp.net mailing list?
<jml> didrocks, I'm really not the best person to ask.
<didrocks> jml: sure, doing it now. Thanks!
<didrocks> jml: did you book for RMLL, btw?
<jml> maybe
<jml> I'm pretty sure I'm speaking there.
<didrocks> right, on wednesday IIRC :)
<jml> I'm dealing with all of that stuff tomorrow.
<didrocks> good luck for your reviewing time :)
<krkhan> in a doc test, login(..); team = factory.makeTeam(); response = webservice.get("/~"+team.name); returns a http response with 500 internal server error. any ideas on what i'm doing wrong?
<jelmer> krkhan: you might have to commit() the change
<krkhan> jelmer: login(...); team=factory.makeTeam(); transaction.commit(); response = webservice.get(...); still returns 500 internal server rror
<jelmer> krkhan: with what error message in the 500 internal server error?
<krkhan> jelmer: empty body, but with ('x-lazr-oopsid', 'OOPS-1627T162') in header
<allenap> sinzui: Are milestone listing in memcached now? How long for? Is there any way to force a refresh?
<sinzui> allenap, 10 minutes active milestones, 3 hours inactive
<sinzui> the status are 1 hour
<allenap> sinzui: bug status?
<sinzui> allenap, bug listing are 10 minutes in active milestones, and 3 hours inactive milestones
<allenap> sinzui: You said "the status are 1 hour". I didn't understand what that meant.
<sinzui> The status and assignment summary above the listings caches for 1 hour
<sinzui> see blog.launchpad.net
<krkhan> found the fix. had to call logout() before using webservice
<mrevell> Night all
<krkhan> how should i encode get queries which contain newline characters? "var=line1%0Aline2" gives a HTTP 400 Bad Request response
<lifeless> krkhan: what variables are you using that you want newlines in ?
<lifeless> I suspect that we'd be looking for POST form encoding for mutate operations
<krkhan> lifeless: for a text search. i was using restclient before but using urllib from python works fine
<lifeless> krkhan: interesting
<lifeless> I'm fairly sure that \r and \n will be ignored by the search engine. IMBW.
<krkhan> lifeless: i was trying to implement my own search method for bug attachments. anyways, python's urllib works fine as far as request is concerned (it doesn't give http 400) but on the server side i still only see the part before the linebreak
<krkhan> ws.op=abc&text=line1%0Aline2 results in text=line1 at the server side
<lifeless> please file a bug on that
<lifeless> it suggests a possible security hole
<krkhan> lifeless: but i can't be sure this is a launchpad issue. i mean, perhaps get requests aren't supposed to have newlines? (IMBW as well)
<lifeless> krkhan: either way ;)
<lifeless> krkhan: there is a bug, and we should get to the bottom of it
<krkhan> lifeless: okay, doing that now. btw, i'm curious as to how this could be a possible security hole
<lifeless> well
<lifeless> imagine that line2 isn't actually lost
<lifeless> what is it doing? does a subsequent request get it? does it enter via an unknown code path?
<lifeless> you could use it to craft a link to launchpad that wouldn't do what the user things it would, for instance.
<lifeless> I'm probably not devious enough, buts it that sort of inconsistent behaviour between a client and a server that leads to many vulnerabilities
<krkhan> ah. i see. i'm a little unsure as to how to present the issue in the bug report. i'm using my own method here and inserting a pdb.set_trace() at the server side to see that the text isn't received properly
<lifeless> document what you know
<krkhan> okay
<lifeless> thats all that one can ask
<krkhan> lifeless: i think it's an issue in httplib2 :) i just tried using launchpadlib to send newline characters and they reached fine
<krkhan> so the problem was only with httplib2 and restclient
<lifeless> ok
<lifeless> thanks for digging
<krkhan> np
<sinzui> abentley, I think https://lpbuildbot.canonical.com/builders/lp/builds/982/steps/shell_7/logs/summary shows that Warty\nHoary\nSecret Squirrel does not match, but I cannot say the test is broken
<abentley> sinzui, I think the problem is there's no defined order for the distroseries.
<sinzui> ah
<wgrant> sinzui, abentley: I have an approved branch which fixes that.
<wgrant> (sorting them by version)
<wgrant> Also, neither of the two ec2tests that were running last night bothered to email me.
<wgrant> Although one submitted to PQM successfully.
#launchpad-dev 2010-06-16
<wgrant> stub: Yes yes yes kill BST.
<stub> Annoying DBA's since 2006
<stub> spurious-apostrophes-is-us
<lifeless> BST ?
<stub> Bull Shit Time
<stub> Better to run in Unambiguous Computer Time
<lifeless> heh
<lifeless> I know what it meant, I was wondering what the impact was
<lifeless> didn't you set the test tz to be argentina or something ages back ?
<stub> Launchpad copes, but it confuses users. wgrant was referring to the recent bug that revision timestamps are in BST (and we don't display the timezone either).
<stub> https://bugs.edge.launchpad.net/launchpad-code/+bug/594591
<_mup_> Bug #594591: What timezone does codebrowse display? <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/594591>
<lifeless> oh
<lifeless> in loggerhead or lp core ?
<lifeless> sounds like loggerhead
<stub> Yer. Elsewhere we use 'x hours ago' or just the date, which is human friendly I guess but not cache friendly.
<lifeless> its friendly enough
<lifeless> pull the cached object, format, ship
<stub> Not when you are trying to cache the formatting
<lifeless> thats true, but formatting single revs isn't where the bottlenecks for loggerhead are
<lifeless> its also cachable for the granularity of your format
<lifeless> e.g. 60 seconds for 'last X minutes
<lifeless> '60 minutes for 'last Y hours
<lifeless> and days for 'Z days ago'
<lifeless> if you're getting multiple requests a second, a 60 second cache is plenty
<lifeless> particularly as its self bounding - after an hour it will scale back, and so on
<stub> I'm thinking more about Launchpad. Marking up out content is slow.
<wgrant> Do we have a good analysis of where time is spent?
<stub> Less so now that thumper switched off bug-link tooltips so rendering text containing a bug tag no longer issues database queries
<thumper> stub: it does for faq references still
<stub> There is a dropin TAL replacement called Chameleon, but unfortunately isn't dropin-enough so we are still using the default TAL rendering engine.
<wgrant> stub: That sounds like it could easily be done with memcached...
<stub> thumper: it won't when I get done
<wgrant> Although python-memcached seems to be very slow.
<thumper> lifeless: I'm in general agreement with you about privacy
<thumper> lifeless: I think we need careful analysis of it as it will touch everything
<lifeless> stub: understood; I don't want to make things slow :). We could still get 60 second caches out using the above strategy for LP.
<stub> wgrant: Still sucks the first time you render it, unless you prepopulate memcache and have some way of ensuring objects don't get evicted.
 * thumper is still trying to do personal 360 reviews even though they are late
<wgrant> stub: Right.
<lifeless> thumper: year, I had two 360's turn up late, forgot about em, and remembdered today
<lifeless> better todo than not todo
<thumper> lifeless: do you have a few minutes for a skype call?
<lifeless> sure
<lifeless> let me grab my mic from lynne so you don't get feedback
<thumper> ok
<lifeless> thumper: you have mail
<mtaylor> don't use python-memcached
<mtaylor> you want to use the one based on libmemcached
<lifeless> what is it called?
<mtaylor> python-libmemcached I think
<mtaylor> also, I started a new one at one point - can't remember if I ever released it
<lifeless> have you played with gizzard /
<mtaylor> no...
<mtaylor> ew. they're using thrift
<stub> The other memcached library seemed flakier than python-memcached. Not sure what the performance issue is - it seems very thin.
<stub> It would be trivial to switch though - we are only using the minimal API at the moment if they are different (get/set single keys at a time)
<mtaylor> the main reason to switch would be getting distributed support that's in libmemcached
<stub> Possibly because of versions of libmemcached in hardy too - not sure what is there.
<mtaylor> that's where all the partitioning/load-balancing is handled  - oh you need hardy
<mtaylor> yeah, that's gonna be old
<mtaylor> I've got libmemcached backports in my ppa I believe
<mtaylor> I may have deleted hardy though...
<mtaylor> yeah, sorry. no hardy
<stub> If it is worth upgrading we can worry about it after the lucid upgrade.
<stub> What is the distributed support?
<mtaylor> the key hashing algorithms to distribute across multiple memcached's is implemented in libmemcached rather than server side
<mtaylor> it's conceivable that the pure-python one could have re-implemented all of the hashing stuff, but rather unlikely
<stub> The pure python library calculates the hashes client side, and there is a hook to enable server side if you want.
<stub> no... all client side. that hook is to be compatible with an older server I think.
<gmb> Morning kids.
<spm> yo dad
<wgrant> gmb: Morning.
<gmb> spm, This morning, I feel old enough to be your dad.
<gmb> Morning wgrant.
<spm> gmb: that's a scary scary place to be, given how old my dad actually is
<wgrant> Did you get an email from that ec2 test of mine you ran?
<wgrant> Because it landed.
<wgrant> But I didn't get an email from it.
<wgrant> Nor the other one, which probably failed.
<gmb> wgrant, Yes, I did; I misspelt your email on the command line, that's why you didn't get one :).
<gmb> (I wrote wrant, it turned out)
<wgrant> gmb: Ah. So maybe mars just didn't fire my other one off, and EC2 isn't being crap again.
<spm> wrant seems more accurate somehow...
<wgrant> Heh.
<gmb> wgrant, Possibly. You have to deliberately put emails in as CLI arguments, so maybe he ran the tests but forgot to add you as a recipient.
<wgrant> gmb: Doesn't ec2 land do that automagically?
 * spm is depressed he hasn't noticed that potential childish name misspelling tease. right under his nose all this time. shame.
<wgrant> spm: Maybe I need to get wgrant.com, and start a blog named WG-rant.
<spm> BWHAHAhahahahahaha
<gmb> wgrant, Yes, but I ran (and I suspect mars ran) ec2 test, just out of habit when running others' code through ec2.
<gmb> wgrant, That would be full of win.
<gmb> You could have a deathmatch with Matt Garrett.
<wgrant> He'd surely win.
<gmb> Starbucks appear to have picked "bizaare Ska tracks" as their CD this morning. Time for Rhythmbox.
<adeuring> good morning
<stub> I need some crappy javascript to sort tables by clicking on the column headers
<stub> Or some good javascript - I'm not fussy.
<lifeless> we had that
<lifeless> 5 years back, just uncommit
<lifeless> in fact, I think some tables *still* have it
<stub> But I'll worry about that after a swim
<wgrant> Can someone please ec2 land https://code.edge.launchpad.net/~wgrant/launchpad/bug-592935-hide-disabled-ppas/+merge/27411 and https://code.edge.launchpad.net/~wgrant/launchpad/bug-592914-recipe-distroseries-order/+merge/27415?
<jelmer> wgrant: sure
<wgrant> jelmer: Thanks.
<mrevell> Good morning
<noodles775> Hi stub (or anyone with sql-foo), if you have a chance, can you take a look at the following options and let me know which of the two (left join or except) would be better? http://pastebin.ubuntu.com/450490/
<noodles775> erm, sql-fu ;)
<noodles775> (with both queries the result will be limited to a batch)
<gmb> Right. Modules without an __all__, which are used in indirect imports, first against the wall when I become President of the World, okay?
<wgrant> You have my vote, as long as removing all glob imports is also one of your core promises.
<lifeless> where is the ec2 land code these days ?
<james_w> ./utilities/ec2
<james_w> ./lib/devscripts/ec2test for the meat.
<lifeless> I meant as a branch url )
<lifeless> what branch is that in
<james_w> it's in the launchpad codebase
<lifeless> thanks
<james_w> http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/files/head:/lib/devscripts/ec2test/
<lifeless> hah
<lifeless> bad mime type, naughty bobo
<lifeless> or,.
<lifeless> email module being confusion, no worries
<jtv> hnyarg testfix mode?
<wgrant> bigjools: Is the package metadata thing being designed with the PPA index page as a possible future use case?
<bigjools> no
<wgrant> :(
<bigjools> but it would be easy to do
<bigjools> just not a priority
<wgrant> Well, as long as the format isn't unowkrable.
<noodles775> It'd be great to be able to say, if the meta data is there describing the apps, use it, otherwise use the installable binaries...
<wgrant> Exactly.
<jtv> Anyone got time to testfix lp.code.browser.tests.test_sourcepackagerecipe.TestSourcePackageRecipeView.test_request_builds_page?
<wgrant> The order thing?
<jtv> wgrant: looks like
<jtv> Warty & Secret changed places in some test output.
<wgrant> I have a branch in EC2 which orders by distroseries.version. I wasn't aware the order wasn't stable... I was just irritated over the weekend that it was arbitrary.
<wgrant> So it will probably be rejected for the thing that it fixes :(
<jtv> wgrant: the irony.
<jtv> wgrant: I guess this was Aaron's branch, and he's not here yet...  any other interesting background you can give me?
<wgrant> I don't know why it's just started breaking now.
<jtv> wgrant: hmmyeah... I don't see anything in the past day of changse that would particularly have effected this.
<jtv> It may be an indeterminate ordering.
<jtv> What was that sound?  Ah, darkness fell.  Goes so fast here.
<wgrant> Oh.
<wgrant> Oh. I hate you.
<wgrant> Gaaaah.
<wgrant> So, I was told to use attrgetter('name.version') rather than lambda ds: ds.name.version
<wgrant> That worked locally, although I was surprised.
<wgrant> And it just failed ec2.
<jelmer> wgrant: did you see those emails?
<wgrant> Turns out that python2.5 doesn't like that.
<wgrant> python2.6 is fine with it.
<wgrant> Gahl
<wgrant> So, https://code.edge.launchpad.net/~wgrant/launchpad/bug-592914-recipe-distroseries-order/+merge/27415
<jtv> wgrant: errr when you say "I hate you," who exactly are you talking to?
<wgrant> That should work on 2.5.
<wgrant> jtv: Python.
<jtv> ah :)
<wgrant> Does anybody have a 2.5-using system that they can test that on?
<wgrant> 'bin/test -t sourcepackagerecipe' is sufficient.
<jtv> (silence)
 * mwhudson thinks attrgetter is an abomination
<jtv> wgrant: I can run it through ec2, and then just land it
 * jtv wonders: if it works for "foo.bar" will it also work for "foo.invokeSomeMethod(param, arg).bar"?
<wgrant> mwhudson: This new variant seems to be.
<wgrant> The old one wasn't too bad.
<mwhudson> maybe i exaggerated
<mwhudson> s/abomination/solution to a non-problem/
<wgrant> What, you don't hate lambdas like Launchpad policy does?
<mwhudson> correct
<wgrant> Good.
<jtv> noodles775, I'm sure you know this: can I give utilities/ec2 options like bin/test's -t to control which tests will run?
<noodles775> I don't actually... I'd use the source to find out...
<jtv> I did that, but it's not incredibly transparent.
<noodles775> jtv: utilities/ec2 --help test
<noodles775> Shows how to pass test options.
<maxb> <wgrant> So, I was told to use attrgetter('name.version') rather than lambda ds: ds.name.version
<maxb> What is the rationale for that? The lambda looks so much nicer to my mind
<jtv> noodles775: damn, I should've thought of that.  :)  Thanks.
<wgrant> maxb: The style guide just about forbids lambdas.
<jtv> And the exception to that is recent.
 * noodles775 thinks it's another case of "use it sensibly"
<noodles775> s/it's/it should be/
<wgrant> It explicitly says that one should use attrgetter instead if possible.
<maxb> And fails to provide plausible rationale for this :-(
<noodles775> which broke your test on 2.5 :/
<jtv> ISTR the review docs on the old wiki saying something about attrgetters as well...  something about them being unsafe?  Never really got what it was about.
<maxb> "Use of the built-in hasattr function should be avoided since it swallows exceptions." ?
<wgrant> http://bugs.python.org/issue504714
<mars> wgrant, I did not get a mail for that branch.  Not sure why, maybe the run itself did not start, and I no longer have the console I started it in.
<mars> wgrant, do you need me to do another run?  Or did gmb pick it up?
<gmb> mars, I didn't pick it up since I didn't know what happened to it.
<gmb> Probably best to run it again.
<wgrant> mars, gmb: jelmer reran it.
<wgrant> But it failed.
<wgrant> Since attrgetter('foo.bar') doesn't work on Python 2.5.
<wgrant> So I've reverted to the lambda.
<mars> wgrant, yeah, I thought of that when I saw your reply later that night
<wgrant> But now devel is in testfix.
<mars> I have a hard-earned "check in Python2.5" mental switch after trying to work in 2.6 for a while
<mars> 2.6 has a surprisingly large number of obviously useful features that are not in 2.5.  You get addicted to them.
<wgrant> Heh.
<wgrant> Yes...
<wgrant> I think 2.4->2.5 might have been worse, though.
<wgrant> But we can do away with 2.5 in a month or two...
<mars> yes, looking forward to that
<wgrant> Hm.
<wgrant> Someone has landed that branch now, anyway.
<wgrant> jtv: Was that you?
<jtv> wgrant: yes
<jtv> like I said
<wgrant> Thanks.
<jtv> Thank _you_.  Easy testfix.  :)
<maxb> ooi, do we know when we will know when LP datacentre machines start going lucid?
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/bug-592935-hide-disabled-ppas/+merge/27411 failed EC2 because of the devel breakage.
<wgrant> Can somebody please send it back?
<mars> wgrant, I'll try and lp-land it
<wgrant> I wonder if it will bother to email anybody this time.
<mars> wgrant, it is on its way
<mars> but the buildbot pass is a while away - 7 hours
<wgrant> But it should be untestfixed already, right?
<mars> yes
<mars> jtv's branch did that
<jtv> mars: it was wgrant's branch... I just landed it as a testfix
<sinzui> gary_poster, flacoste: I need help with a CheckBoxWidget: http://pastebin.ubuntu.com/450614/
<gary_poster> sinzui: not sure.  I'll look at setUpWidget.
<sinzui> gary_poster, flacoste: I neglected to mention that I am using GhostWidget to suppress rendering. I now think (after making my failure public) that the problem is that GhostWidget is really a text widget, it does not covert values of 'on' to True
<sinzui> That must be the case. I should subclass the GhostWidget or implement a BooleanGhostWidet
<gary_poster> sinzui: that jibes with the suggestion that I was trying to validate before making: make the widget handle the value.
<gary_poster> so, agree
<sinzui> thanks gary
<flacoste> sinzui: why do we need a ghostwidget in the first place?
<gary_poster> welcome
<sinzui> flacoste, it is a mechanism we use to allow a form to manage manage a field in a normal fashion, but permit another widget to do the rendering. We do this when we want to compose a compound widget that interleaves subordinate fields into the widget's primary field
<sinzui> flacoste, eg, we show the licence_info field adjacent to the license widget because it is only relevant when the user choose two of the 25 items
<flacoste> ok
<sinzui> sweat a 6 line fix by converting the Ghost into a mixin
<sinzui> ow sweet. I hate English
<jml> how do I disable edge redirection
<jpds> jml: Footer of any page on the right: "Disable edge redirect".
<jml> jpds, thanks.
<mrevell> night
<henninge> matsubara: you were saying you something like bug 595163 fixed elsewhere? Do remember where?
<_mup_> Bug #595163: TypeError raised when query string contains a list <oops> <Launchpad Translations:New> <https://launchpad.net/bugs/595163>
<matsubara> henninge, just a sec. let me find it
<matsubara> henninge, https://bugs.edge.launchpad.net/bugs/213366
<_mup_> Bug #213366: OOPS in +builds page passing a list as the query string <oops> <Soyuz:Fix Released by cprov> <https://launchpad.net/bugs/213366>
<henninge> matsubara: so, that fix turns the TypeError into an UnexpectedFormData. Is that an acceptable fix?
<matsubara> henninge, yes, I think so. that way the user will be informed that what he's trying to do is wrong and we won't record an oops for it
<henninge> ok, cool ;-)
<henninge> matsubara: thanks, that's really easy now.
<henninge> I hope
<henninge> ;)
<matsubara> :-)
<maxb> thumper: Hi. Given the status of bug 327126 is it OK if I go ahead and create a small bzr-svn import from apache.org ?
<thumper> maxb: yep
<maxb> lets give it a go... (I have doubts)
<maxb> bzrlib.errors.NotBranchError: Not a branch: "/subversion/trunk/subversion/tests/cmdline/svntest".
<maxb> boo
<maxb> (yes I did want just the svntest library in its own branch)
<thumper> maxb: we do have some apache imports working
<maxb> yeah, I think I've been caught out by bzr-svn being too clever for its own good
<lifeless> skynet is on the way
#launchpad-dev 2010-06-17
<wgrant> spm: Users can deactivate their accounts, right..?
<spm> wg-rant: deactivate yes; purge no
<spm> if you appreciate the difference. we get very few requests for the latter, but enough.
<wgrant> How do you purge? Merge them away?
<spm> sql nullify the relevant bits in the appropriate tables
<wgrant> Ah.
<spm> not ... pretty, but.
<spm> actually.. I think that's correct. verifying...
<spm> bleh. it's not docco'd. but that'd be the likely way.
<spm> wgrant: heh, ref the 'ssh codebounce' I'd typed that much and was starting to get frustrated at why ssh tab completion wasn't. then the light came on. whee. :-D
<wgrant> Heh.
<wgrant> It's not a DNS alias yet? :(
<spm> that would be .. unwise :-)
<spm> i know we have cocobanana as an alias for a rollout target. but generally we do limit our moments of ... public silliness :-)
<wgrant> Pfft.
<lifeless> thumper: hi
<lifeless> thumper: Can we have a 5 minute pre-impl call ?
<thumper> lifeless: sure
<lifeless> when suits? now would be great for me
<thumper> lifeless: now is fine
<thumper> lifeless: you dropped
<lifeless> can you hear me ?
<lifeless> [120531.779865] iwlagn 0000:02:00.0: Microcode SW error detected.  Restarting 0x2000000.
<ajmitch> iwlagn isn't the most stable of wireless drivers, I've had problems with it at times that required rmmod & modprobe
<adeuring> good morning
<mrevell_> Morning
<kb9vqf> wgrant: You around?
<kb9vqf> Any idea why this might be happening? https://quickbuild.pearsoncomputing.net/builders/vidar
 * kb9vqf wonders if it's unique to Debian
<wgrant> kb9vqf: Not sure. Maybe check Ubuntu's console-data diff?
<kb9vqf> OK, just wondering if you guys had seen it before ;-)
<kb9vqf> Those almost look like drawing commands for the text-based interactive mode
<kb9vqf> Hmmm...why does the build farm ignore the remaining idle builders insead of assinging them chunks of the build queue?
<kb9vqf> I have two active and two idle with a long build queue...
<kb9vqf> https://quickbuild.pearsoncomputing.net/builders/
<kb9vqf> I don't see any errors anywhere
<wgrant> Check the buildd-manager log for references to those builders.
<kb9vqf> Nothing
<kb9vqf> wgrant: The only thing in the log are references to the broken builders
<wgrant> What if you edit lib/lp/buildmaster/manager.py, replace logging.INFO with logging.DEBUG, then restart buildd-manager?
<kb9vqf> wgrant: Not much, just a bunch of
<kb9vqf> 2010-06-17 04:11:32-0500 [-] Considering odin
<kb9vqf> 2010-06-17 04:11:32-0500 [-] No build candidates available for builder.
<kb9vqf> Strangely both builders work, just not simultaneously
<wgrant> Ahh.
<wgrant> I bet it's the 80% restriction.
<kb9vqf> ?
<wgrant> That is, no PPA is allowed to use more than 80% of the builders for any architecture.
<kb9vqf> Makes sense
<kb9vqf> Can I override it?
<wgrant> No, but you can delete it.
<wgrant> See the end of addCandidateSelectionCriteria in lp.soyuz.model.buildpackagejob
<kb9vqf> OK, thanks (Each of my builders is a separate physical machine, so adding more builders isn't a very palatable solution)
<wgrant> remove the num_arch_builders stuff and everything from there until the return.
<kb9vqf> It looks like I could override it (sort of) by just changing the 80 to something else
<kb9vqf> That way it's a small change from production Launchpad, not a big one ;-)
<wgrant> Or just delete the bit that adds the clause.
<kb9vqf> I assume a simple restart of the buildd manager will apply the change?
<wgrant> Correct.
 * kb9vqf needs to get to bed...it's way to early here
<kb9vqf> Thanks for your help!
<wgrant> Night.
<deryck> Morning, all.
<leonardr> Chex, i filed an rt earlier this morning to ask for help running a performance test. it should only take a few minutes, so let me know when you can spare the time
<leonardr> argh, where's my receipt?
<sinzui> mars: I did not get an email from ec2 this morning. The instance finished, but not success or fail email
<mars> sinzui, did it finish in a reasonable amount of time?  Also, was it an 'ec2 land' mail?  Those can take forever sometimes.
<sinzui> How could I know it finished in a reasonable amount of time?
<mars> (just noticed how ridiculous that sounds - if it took forever, how would I know when it finished?)
<sinzui> I used ec2 test
<mars> ok
<mars> sinzui, ok, so there may still be problems, barring a few-hours delivery delay.  I have a follow-up branch that should address that.
<sinzui> I resubmitted the test. I hope it to be complete in 4 hours
<mars> ok, please let me know what the result is.  If it is a success, then we have an intermittent failure.  If it eats it again, then I might have something I can reproduce.
<Ursinha> sinzui, hi
<sinzui> Hi Ursinha
 * sinzui was lost and could not find the window that wanted his attention
<Ursinha> sinzui, :)
<Ursinha> sinzui, about your email: the bugs marked as fix committed without qa tags weren't marked fix committed by my script, but manually
<Ursinha> that's why they don't have the tags
<Ursinha> when the script runs on them, it will mark them properly
 * sinzui nods
<Ursinha> I've stopped the script for now, I'm putting newer code there, but it should happen rsn
<sinzui> Ursinha, is it a concern that they are not being tracked?
 * sinzui QAs every fix-committed in answers, blueprint and registry regardless of tags
<Ursinha> sinzui, I'll run the script and check if they will be properly tagged
<Ursinha> sinzui, yes, I often monitor those
<Ursinha> the fixed but untagged
<sinzui> fab
<sinzui> I will not worry then
<Ursinha> same way I monitor the orphaned ones
<Ursinha> :)
<sinzui> mars, ec2 sent me emails this time. Both instances ran about 4 hours
<mrevell> Night!
<cody-somerville> If anyone asks, it appears PPA publishing is broken at the moment. Working on fixing it.
<jml> g'night all. see you next week.
<jml> cody-somerville, btw, if it's down for more than 15 minutes, it's an incident. Please find someone to manage it.
 * jml flees the scene
<cody-somerville> jml, trying to. It seems all the Soyuz people have EODed
<mars> flacoste, ^ ?
<cody-somerville> It appears to have been broken now for four hours and 5 minutes now.
<cody-somerville> * 10 minutes
<sinzui> It looks like the devel build will fail. I have a fix for the two failing tests that I know about
 * sinzui waits for the official failure message
<wgrant> U1 is very good at 'fixing' major security flaws but somehow managing to leave them wide open.
<beuno> wgrant, that's not exactly useful
<thumper> beuno: hi
<beuno> wgrant, there is a roll-out to production that has been delayed 3 times already
<beuno> wgrant, that contain the actual fixes
<beuno> heya thumper!
<wgrant> beuno: Ah. That makes more sense.
<beuno> wgrant, hence, the fix-committed and not fix-released
<wgrant> Just jdo said it was fixed... and it's not. Which reminds of last time someone said it was fixed, where the fourth fix finally worked.
<wgrant> Mmm, no, the bug in question here is Fix Released.
<beuno> wgrant, I would of expected you to know that by now  ;)
<beuno> oh?
<beuno> it's wrong then
<beuno> he probably thinks the roll-out happened
<wgrant> Bug 590540
<beuno> wgrant, right, no deployed
<beuno> tomorrow, hopefully
<wgrant> Ahhh, good. Sorry about that.
<beuno> I see where the confusion came up
<beuno> if PQM behaves the roll out tomorow will include all fixes
<beuno> the branch for #535651 is bouncing with an unrelated error
<beuno> the rest are all landed
<lifeless> beuno: 'if' :)
<beuno> heh, yes
<beuno> may be all of them minus one
<wgrant> OK, great. Thanks.
<beuno> wgrant, thanks for the reports
<beuno> it was a bit of a shock to see those open
<beuno> we did some db mangling to make sure they hadn't been exploited
<beuno> they haven't
 * thumper afk
<wgrant> beuno: It returned no results at all?
<wgrant> Because I did try it.
<beuno> wgrant, I didn't see the query, but it may of been "nobody except wgrant"  ;)
<lifeless> wgrant: now try % encoded path seperators
<wgrant> sinzui: +participation 403s sometimes.
<sinzui> ouch
<wgrant> You, for example.
<sinzui> me
<sinzui> I will investigaste
<wgrant> Thanks.
<sinzui> ha ha, there are 17 deprecated private membership teams, and I appear to be a members of a few. and since the new page shows the path to membership, it breaks
<sinzui> I will land a fix for this tomorrow. I wish I could remove the teams too
<wgrant> Why can't you?
#launchpad-dev 2010-06-18
<wgrant> Sounds like a fair bit of code doing nothing useful...
<adeuring> good morning
<mrevell> Morning!
<deryck> Morning, all.
<mrevell> Night
#launchpad-dev 2010-06-19
<wgrant> sinzui: ISD doesn't feel that email deletion is important, so SSO doesn't do it.
#launchpad-dev 2010-06-20
<lifeless> hmm
<lifeless> we need a crash gantry or something - fail-whale equivalent ;)
<lifeless> thumper: whats the plan for this use case
<lifeless> mbp puts up a MP X
<lifeless> I say 'mbp is on leave, I will fix and land'
<lifeless> do I 'supersede mbp's proposal'
<lifeless> or do we decouple proposal from branch, so I can push a branch based on it and just change the branch in the MP web ui
<lifeless> or something else ?
<lifeless> oh thats right, thumper-awol
#launchpad-dev 2011-06-13
<lifeless> huwshimi: is this you - http://huw.ugbox.net/ ?
<huwshimi> lifeless: It was, a loooong time ago, yes
<huwshimi> lifeless: How come?
<lifeless> huwshimi: its linked from the bottom of http://twistedmatrix.com/trac/wiki/IPv6
<huwshimi> lifeless: Oh yeah :)
<LPCIBot> Project windmill-devel build #207: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/207/
<lifeless> stub: \o/ thanks for fix-releasing that bug
<lifeless> stub: the followup branch with the journal etc is all landed and just passed bb
<stub> One down...
<stub> I'll be starting on the extension. Hopefully importance and the two booleans won't bloat things to impractical levels.
<lifeless> stub: I think it ended up being a 54MB table
<lifeless> stub: e.g. still quite compact
<stub> How many discreet importances are there?
<lifeless> unknown wishlist low medium high critical (+ perhaps 2 backwards compat ?)
<stub> huh... we have some bugtasks with priority set
<stub> Only 30k or so... must have been a bug in bugs
<stub> Or is that settable via the API?
<LPCIBot> Project windmill-db-devel build #383: STILL FAILING in 2 min 43 sec: https://lpci.wedontsleep.org/job/windmill-db-devel/383/
<lifeless> stub: unmigrated early-days stuff?
<lifeless> I'm not sure python even knows about the column now
<lifeless> stub: ah, your branches were targeted at db-devel
<nigelb> Someday I need to get around to fixing that sorting. I've been holding it off.
<jtv> Any reviewers in the house?  I'm pretty sure the topic is out of date!  https://code.launchpad.net/~jtv/launchpad/pre-bug-791204-getPackageUploads-name-filter/+merge/64345
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer:  - | Critical bugs:209 - 0:[######=_]:256
<jtv> stub: are you ocr'ing today?
<nigelb> Aww, the branding wiki linked in the email isn't accessible outside of Canonical :(
<huwshimi> nigelb: Oh, apologies about that. I'll see if there's another link I can use
<nigelb> huwshimi: Ah, thanks!
<jtv> Anybody want to review an extensive but relatively straightforward lint branch?  https://code.launchpad.net/~jtv/launchpad/more-pre-791204-lint/+merge/64365
<stub> jtv: yo
<jtv> stub: got time to review this one?  Not hard: https://code.launchpad.net/~jtv/launchpad/more-pre-791204-lint/+merge/64365
<jtv> and hi, btw :)
<stub> If it is just lint, r=jtv would be fine. Or do you want another pair of eyeballs on it?
<jtv> stub: there's a few places where I went beyond the entirely trivial.  Not far, but still.
<stub> One place I see, and that is pretty obviously an improvement. r=stub
<stub> Off for a quick bite to eat
<jtv> thanks stub!
<stub> np :)
<LPCIBot> Project devel build #798: STILL FAILING in 5 hr 50 min: https://lpci.wedontsleep.org/job/devel/798/
<LPCIBot> Project parallel-test build #30: STILL FAILING in 1 hr 21 min: https://lpci.wedontsleep.org/job/parallel-test/30/
<stub> Yay. http://pgreplay.projects.postgresql.org/ might let us actually create a DB benchmark.
<LPCIBot> Project windmill-db-devel build #384: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/windmill-db-devel/384/
<jml> huwshimi: hello
<huwshimi> jml: Hey
<jml> huwshimi: skype?
<huwshimi> jml: Just logging in
<huwshimi> jml: Just one sec
<mrevell> Hello!
<huwshimi> jml: back
<jml> mrevell: hi
<huwshimi> nigelb: Here is that link: https://dev.launchpad.net/Projects/Rebranding
<nigelb> huwshimi: thanks :-)
<nigelb> Can we introduce a bug into launchpad that will report the number of critical bugs against lauchpad as zero? ;-)
<jml> nigelb: no.
<nigelb> jml: Drat. I tried :-)
<lifeless> nigelb: we could, but it would be a regression and so critical :P
<lifeless> jml: btw, hi
<jml> lifeless: hello
<lifeless> stub: that looks interesting
<lifeless> jml: hows today looking ?
<nigelb> lifeless: Hehe
<jml> huwshimi: http://lpqateam.canonical.com/doc/scope.html
<jml> lifeless: otp
<jml> lifeless: should have time for a call if you'd like one
<lifeless> that would be nice
<jml> lifeless: now? skype?
<jml> lifeless: Since you're not answering. I'm going to go wash the dishes. Will be back soon.
 * bigjools adds a bug tag subscription and rejoices
<danilos> bigjools, nooooo, we didn't make it to be used!
<bigjools> (ab)used
<jml> back
<lifeless> jml: yes, and yes
<jtv> wgrant, StevenK: has either of you touched the Upgrade button on df's oneiric/+localpackagediffs lately?  Mind if I hit it in a few minutes?
<StevenK> jtv: I seriously doubt it, since .au has a public holiday today.
<jtv> that's never stopped you before
<jtv> you geocontrarian bunch of workaholics
<LPCIBot> Project windmill-devel build #208: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-devel/208/
<jtv> Review needed!  https://code.launchpad.net/~jtv/launchpad/pre-bug-791204-getPackageUploads-name-filter/+merge/64345
<bigjools> jtv: looking at it now
<jtv> thanks!
<bigjools> jtv: are you doing full substring on the name_filter?
<jtv> No, prefix.
<bigjools> jtv: the old code did full substring
<jtv> Didn't look like it.
<bigjools>                 "sourcepackagename.name LIKE '%%' || %s || '%%'"
<jtv> argh
<bigjools> there's also an exact_match param in the old code
<jtv> oh well, good thing I isolated that in a function.
<jtv> Yes
<jtv> I'll have to mimick that as well, for the script.
<bigjools> not sure where that was used though
<jtv> In the script.  :(
<jtv> I just found it.
<jtv> But it won't be hard, again because I isolated the match in a separate function.
<bigjools> ok
<jtv> Maybe I should tackle both of these issues together in a separate branch: instead of prefix match, give a choice of exact match or full substring match.
<jtv> (There were some intricacies w.r.t. pattern escaping)
<bigjools> well you'd be landing a regression...
<jtv> No, because we're not actually calling that method yet.
<jtv> Or at least not with that argument!
<bigjools> oh you've not changed code to call it yet
<jtv> No, because that requires both this branch and the other one that I'm currently writing an MP for.
<bigjools> ok
<bigjools> the substring query is going to cack up the performance something rotten
<bigjools> jtv: also the old getQueueItems had some messed up ordering that was discarded when it did the union at the bottom
<bigjools> I wonder if we can fix that
<bigjools> but the problem is that the old query ordered on different things depending on the upload type
<jtv> What kind of ordering do we want, from the user's perspective?
<bigjools> not entirely sure tbh, but anything consistent will be better than the current code
<bigjools> it's quite hard to order because of the different types of upload
<jtv> Yeahâ¦ if it weren't for custom uploads, I'd say we clearly need to sort by package.
<bigjools> sorting by packageupload.id is a good start, it just won't order the items that share the same PU
<jtv> And then maybe have a separate more queue-oriented page for seeing how the work goes.
<jtv> Isn't it just the builds that share a PU?
<jtv> Anyway, it's probably not going to be _that_ hard to treat that as a separate issue?
<bigjools> jtv: a single upload can have builds, sources AND custom files
<bigjools> there's not many of those in practice any more, but the model allows it
<jtv> Oh ewll
<jtv> well
<jtv> That's for a separate query though.
<jtv> This query only returns the PUs.
<jtv> When we display those, I have a feeling the sorting of items within a PU will work out without too much difficulty.
<jtv> Oh this just had to happen to me, didn't it?
<jtv> Diff against target:666 lines (+391/-50) 8 files modified (has conflicts)
<jtv> Woe to you, oh earth and seaâ¦
<bigjools> jtv: reviewed, needs-fixing
<jtv> thanks
<bigjools> jtv: so FYI there's one PCJ with a PU in the dogfood DB now, since I synced an unknown package.  It'll be good QA fodder :)
 * jtv cackles nastily
<benji> /topichttps://dev.launchpad.net/ | On call reviewer:  benji | Critical bugs:209 - 0:[######=_]:256
<benji> /topichttps://dev.launchpad.net/ | On call reviewer:  benji | Critical bugs:209 - 0:[######=_]:256
<benji> pfft
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer:  benji | Critical bugs:209 - 0:[######=_]:256
<danilos> benji, hey, hey, I've got a few branches awaiting review, the first one at https://code.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-lp-names/+merge/64178
<benji> danilos: I'll take a look
<danilos> benji, cool, thanks
<lifeless> jml: sorry, cut the last bit off
<lifeless> jml: was good talking; have a great day
<LPCIBot> Project db-devel build #630: FAILURE in 5 hr 51 min: https://lpci.wedontsleep.org/job/db-devel/630/
<rvba> benji: Hi! Could you take a look at https://code.launchpad.net/~rvb/launchpad/initseries-bug-795537/+merge/64231 ?
<benji> rvba: sure
<rvba> benji: Thanks!
<LPCIBot> Project windmill-devel build #209: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/209/
<benji> danilos: I'm done with https://code.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-lp-names/+merge/64178
<danilos> benji, cool, thanks, care to take on the next one: https://code.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-subscribers/+merge/64179? :)
<benji> danilos: sure, you're second in line, but I don't think it'll be a very long wait
<danilos> benji, great, thanks
<benji> rvba: I'm done with https://code.launchpad.net/~rvb/launchpad/initseries-bug-795537/+merge/64231
<rvba> benji: Thanks!
<jelmer> benji: Hi! Can I add another branch to your queue?
<benji> jelmer: absolutely
<jelmer> benji: Thanks :) MP is at  https://code.launchpad.net/~jelmer/launchpad/auto-upgrade/+merge/64398
<allenap> benji: Can I push another merge proposal (shortish, some javascript) on your stack? https://code.launchpad.net/~allenap/launchpad/localpackagediffs-latest-comment-bug-746379/+merge/64334
<benji> allenap: sure
<allenap> Thanks :)
<cr3> can someone explain the purpose of lazr.canonical.timeout.urlfetch when urllib2.urlopen already supports a timeout parameter?
<benji> danilos and jelmer,done with your reviews
<LPCIBot> Project parallel-test build #31: STILL FAILING in 1 hr 0 min: https://lpci.wedontsleep.org/job/parallel-test/31/
<danilos> benji, cool, thanks again
<benji> np
<jelmer> benji: thanks :)
<benji> np
<jml> sinzui: hi
<sinzui> hi
<jml> sinzui: would you like to have a call about disclosure?
<sinzui> okay
<LPCIBot> Project windmill-db-devel build #385: STILL FAILING in 1 hr 14 min: https://lpci.wedontsleep.org/job/windmill-db-devel/385/
<sinzui> jml: I am on mumble
<jml> sinzui: mumble hates my guts. I have dared to change my hardware. Give me a sec.
<sinzui> jml: I will also learn is mumble works for me on oneiric
<benji> allenap: I'm done with https://code.launchpad.net/~allenap/launchpad/localpackagediffs-latest-comment-bug-746379/+merge/64334
<allenap> benji: Thank you :)
<benji> my pleasure
<jcsackett_> sinzui: just saw bug 707234 bumped up. think i might tackle that today.
<_mup_> Bug #707234: Ajax controls disabled on bugs with many tasks due to multiple redundant copies of person picker make_picker function in view-source:https://bugs.launchpad.net/launchpad-project/+bugs?advanced=1 <disclosure> <javascript> <person-picker> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/707234 >
<jcsackett_> up for chatting about it when you're done talking with jml?
<LPCIBot> Project devel build #799: STILL FAILING in 6 hr 2 min: https://lpci.wedontsleep.org/job/devel/799/
<sinzui> jcsackett: I am available to chat now
<cr3> nevermind my question about urlfetch, I suspect the reason is that the timeout parameter was only added to urllib2.urlopen in 2.6
<bigjools> mrevell: you might find this interesting reading (then again it may put you to sleep) https://dev.launchpad.net/Soyuz/DerivativeDistributions
<mrevell> bigjools, Thanks :)
<LPCIBot> Project windmill-devel build #210: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/210/
<timrc> bigjools, Cody ran the complete test suite before landing my fix for #724740 and I had one failure... does that have to mean I introduced a regression?  If so it's causing some serious head scratching, because the failure seems completely unrelated to my change (and if there is a relationship, I'm scared :))
<_mup_> Bug #724740: setting a ppa private cannot be done over the API <api> <oem-services> <ppa> <Launchpad itself:In Progress by timrchavez> < https://launchpad.net/bugs/724740 >
<bigjools> timrc: can you paste the failure?  It's not unheard of to have random failures :/
<timrc> bigjools, http://pastebin.ubuntu.com/625962/
<timrc> bigjools, I'll dig deeper into this, myself... it just struck me as odd as this being the one place of failure
<bigjools> timrc: ah that does look related
<bigjools> timrc: where's your merge proposal?
<timrc> bigjools, https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-2/+merge/63950
<bigjools> timrc: if you run that test locally does it fail again?
<timrc> bigjools, yes, when I run the test locally it fails
<bigjools> timrc: if you revert your changes does it still fail?
<allenap> benji: Got time for another shortish javascript branch? https://code.launchpad.net/~allenap/launchpad/localpackagediffs-commenting-bug-795573/+merge/64418
<benji> allenap: sure
<allenap> Thanks again :)
 * jml has to run.
<jml> see y'all later.
<bigjools> timrc: I'm testing locally too
<timrc> bigjools, testing now
<timrc> bigjools, yeah, must be a regression caused by me :( test ran fine with my changes removed
<bigjools> ok let's work out why
<bigjools> and indeed it fails for me too
<benji> allenap: I'm done with https://code.launchpad.net/~allenap/launchpad/localpackagediffs-commenting-bug-795573/+merge/64418
<allenap> Thanks benji.
<cr3> is there a naming convention for urls in the web and webservice interfaces for referring to sets and individual elements within sets, eg plural for the former (/foos) and singular for the latter (/foo/1)
<bigjools> good question
<cr3> what I'm observing from the apidoc is plural all the way, eg /foos and /foos/1 taking my above example. I can understand that motivation for people playing with urls manually, so removing /1 from the latter would immediately resolve into a working url
<cr3> I would appreciate some confirmation whether I'm observing a legacy decision or one that still holds today
<bigjools> timrc: I can see where that test is failing but I can't work out why
<bigjools> timrc: IArchiveSet.getArchivesForDistribution() is not returning the copy_archive created at the top of the test
<bigjools> so the script that needs it doesn't do anything
<bigjools> timrc: I have to go now, maybe someone else can help you or grab wgrant when he's around later in your day
<bigjools> cr3: I think /foos is a good pattern
<sinzui> launchpad tells me that it cannot import subunit using python 2.6 when running on oneiric :(
<bigjools> the singular is up for debate
<bigjools> it depends on the domain
<timrc> bigjools, alrighty... thanks for the help
<bigjools> timrc: you're welcome.  Good luck.
<cr3> bigjools: implying /foos/1 for an individual element under /foos, right?
<bigjools> cr3: well one example is /builders and /builders/<builder>
<cr3> bigjools: I just tried accessing /people/cr3 on launchpad, which redirects me to /~cr3 whereas /person/cr3 asks me if I lost somthing :)
 * bigjools has to run for the day
<bigjools> there you go thenb
<bigjools> good night all
<mrevell> Good night my comrades.
<Ursinha> matsubara: hi :) did you file bugs for any of your dd exploratory testing notes?
<matsubara> Ursinha, not yet
<LPCIBot> Project windmill-devel build #211: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/211/
<abentley> benji: could you please review https://code.launchpad.net/~abentley/launchpad/ppa-api-errors/+merge/64425 ?
<benji> abentley: sure
<bdmurray> I'm having an issue running 'make schema' on my launchpad dev system
<bdmurray> http://pastebin.ubuntu.com/626039/
<benji> abentley: I'm done with https://code.launchpad.net/~abentley/launchpad/ppa-api-errors/+merge/64425 I had one thought that you might want to consider.
<abentley> benji: ?
<benji> abentley: I don't understand what you don't understand. :)
<abentley> benji: You said there was something I might want to consider.  I was saying "Hmmm?  Go on..."
<benji> ah, it's in a comment on the MP
<abentley> benji: BadRequest is only emitted for status 400 AFAIK.
<benji> abentley: since the test docstring explicityly says 400, it would be good to actually test for that, and I'm paranoid enough to expect that same message to one day be given a different status code
<abentley> benji: Very well, I'll change the test to say "Bad Request".
<benji> k
<abentley> benji: I submit that if you're afraid standards bodies are out to get you, you truly are paranoid.
<benji> heh; it's more that the people writing the software are fallible
<abentley> benji: I think that if I tested the status code of BadRequest, I'd be violating the principle of "Test only one thing", because I'd be testing the behaviour of lazr.restfulclient.
<benji> abentley: I'd buy that.
<bac> hi james_w, i need to create a new test data suite for archiveuploader.  do you know how to do that, specifically working out getting them properly signed?
<bac> abentley, flacoste: ^^
<flacoste> bac: what do you want to do exactly?
<bac> flacoste: create a test for a maintainer with a bad email address (bug 519857)
<_mup_> Bug #519857: Upload processor OOPSes with bad email addresses <lp-soyuz> <oops> <soyuz-upload> <Launchpad itself:In Progress by bac> < https://launchpad.net/bugs/519857 >
<flacoste> benji: if you have time, could you review https://code.launchpad.net/~flacoste/launchpad/bug-781993-1/+merge/64258 pretty please?
<bac> flacoste: so i figure i need to create a new suite, with a bad maintainer or Changed-by email address...but it has to be properly signed
<benji> flacoste: sure
<flacoste> thanks benji
<flacoste> bac: why don't you write this as a unit test to safe_fix_maintainer
<flacoste> bac: which operates on a hash
<flacoste> err, dictionary
 * flacoste again showed he's a former perl hacker
<bac> flacoste: i'll look into that.  thanks.
<flacoste> bac: if you want to go end-to-end, i think there are probably some helper method in soyuz for that, but would be best to ask bigjools, StevenK or wgrant about it
<bac> flacoste: actually, no.  i was intending for safe_fix_maintainer to still raise the same error, but i'd just handle it in process
<bac> flacoste: which is why i wanted an end-to-end test
<bac> and it looked like the way things are done in test_uploadprocessor.py is to create novel suites...
<james_w> bac, from what I remember the checking of real packages tends to operate on packages contained in the LP source tree
<flacoste> which is more a case of missing test architecture i think
<flacoste> but i may be mistaken
<flacoste> bac: you want to make the fix in processChanges?
<bac> james_w: what i think i need to do is in lib/lp/archiveuploader/tests/data/suite create a new suite, say, bar_1.0-1_bad_maintainer_email and in it change bar_1.0-1_source.changes to futz with the maintainer or changed-by email address to invoke the failure.  but when i do that i need to sign that file
<LPCIBot> Project db-devel build #631: STILL FAILING in 5 hr 39 min: https://lpci.wedontsleep.org/job/db-devel/631/
<bac> flacoste: i was intending something like http://pastebin.ubuntu.com/626072/
<LPCIBot> Project parallel-test build #32: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/parallel-test/32/
<james_w> flacoste, there's no support in the test suite for signing packages that I know of, which is why all of those are already signed
<bac> james_w, flacoste: right, so that's what i'm trying to figure out.  i see a secring.gpg but it doesn't seem to have in it what i need.  so i'm trying to figure out where the keys are and perhaps some instructions...
<james_w> bac, ah, got it
<bac> james_w: sorry it was so circuitous
<james_w> I've never had to do it myself
<james_w> so I can't help you I'm afraid
<bac> james_w: ah, right then.  of the people awake you seemed like the best candidate
<bac> james_w: thanks anway!
<benji> flacoste: I'm done with https://code.launchpad.net/~allenap/launchpad/initialise-to-initialize/+merge/64121
<bac> flacoste: perhaps i should put this aside until i can talk to bigjools
<allenap> benji: EWRONGURL I suspect.
<benji> pfft!
<flacoste> bac: yeah, probably a good idea
<benji> allenap: the curse of having multiple tabs open at once
<allenap> benji: But thanks for reviewing that one :)
<benji> flacoste: I actually meant that I'm done with this https://code.launchpad.net/~flacoste/launchpad/bug-781993-1/+merge/64258 ;)
<flacoste> benji: ah, cool, thanks!
<benji> allenap: I just had a thought about your branch that I want to check out, so if you're panning on jumping on the merge, hold off for one sec
<allenap> benji: Sure.
<LPCIBot> Project windmill-db-devel build #386: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/windmill-db-devel/386/
<benji> allenap: ok, we're good, my negative thought didn't pan out (I was concerned that one or more symbols exposed by the web service might have been changed)
<allenap> benji: Thanks for thinking of it and checking :)
<lifeless> flacoste: this explains a lot :P
<flacoste> lifeless: ???
<lifeless> 07:02  * flacoste again showed he's a former perl hacker
<lifeless> :P
<flacoste> lol
<LPCIBot> Project windmill-devel build #212: STILL FAILING in 47 min: https://lpci.wedontsleep.org/job/windmill-devel/212/
<flacoste> lifeless: call time?
<lifeless> yup
<poolie> hi flacoste, lifeless
<bdmurray> I sorted out my question
<cr3> might there be an interface for accessing oops reports? I'm wondering what might be wrong with this one: OOPS-1990AP106
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #800: FIXED in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/800/
<LPCIBot> Project windmill-devel build #213: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/213/
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer:  - | Critical bugs:209 - 0:[######=_]:256
<LPCIBot> Project windmill-devel build #214: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/214/
<lifeless> o/ poolie
<lifeless> sinzui: Was your team touching on bug 66552 ? I think its a dupe or something
<_mup_> Bug #66552: Unhelpful error when reporting bug with non-existent package entered and "I don't know" chosen <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/66552 >
<bdmurray> I'm now getting an error trying to run most tests
<bdmurray> http://pastebin.ubuntu.com/626204/
<lifeless> fun
#launchpad-dev 2011-06-14
<sinzui> wgrant: mumble?
<jcsackett> wallyworld: bug 707234
<_mup_> Bug #707234: Ajax controls disabled on bugs with many tasks due to multiple redundant copies of person picker make_picker function in view-source:https://bugs.launchpad.net/launchpad-project/+bugs?advanced=1 <disclosure> <javascript> <person-picker> <regression> <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/707234 >
<sinzui> wallyworld: jcsackett: StevenK: http://www.markshuttleworth.com/wp-content/uploads/2008/12/link-blueprint_web-version2.swf https://devpad.canonical.com/~beuno/ui_movies/Search%20Packages.1.swf    https://devpad.canonical.com/~beuno/ui_movies/Subscribe_create_team_3.swf
<sinzui> wgrant: ^
 * wallyworld off to bank to try and get working credit card
<StevenK> Hah
<wgrant> poolie: Hi.
 * StevenK attempts to work out when his SSL certificates expired.
<wgrant> StevenK: Going back to Friday, was DSP the only extra perm required?
<LPCIBot> Project windmill-devel build #215: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/windmill-devel/215/
<StevenK> wgrant: It seemed to be, yes.
 * wgrant qa-oks
<wgrant> abentley: Hi.
<wgrant> abentley: It the 500 on some error cases the only reason bug #776449 is qa-bad?
<_mup_> Bug #776449: Set Ubuntu dependencies for PPA via API <api> <bad-commit-13157> <escalated> <not-pie-critical> <oem-services> <ppa> <qa-bad> <Launchpad itself:Fix Committed by abentley> < https://launchpad.net/bugs/776449 >
<LPCIBot> Project parallel-test build #33: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/33/
<jtv> Morning paduans
<wgrant> Morning jtv.
<jtv> hi1
<wgrant> Aherm.
<wgrant> The new picker has a bug.
<wgrant> Everyone seems to have all IRC nicks.
<wgrant> I guess the preloading is buggy.
<wgrant>             nicks_by_person = dict((nick.personID, nicks)
<wgrant>                 for nick in nicks)
<wgrant> Oops.
<mwhudson> haha
<LPCIBot> Project windmill-db-devel build #387: STILL FAILING in 1 hr 18 min: https://lpci.wedontsleep.org/job/windmill-db-devel/387/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #632: FIXED in 5 hr 53 min: https://lpci.wedontsleep.org/job/db-devel/632/
<wallyworld> wgrant: what's the problem with the nicks?
<wgrant> wallyworld: The preloading preloads all of the found nicks into everyone that is found to have a nick.
<wgrant> So if the batch has two people with nicks, both of them will have all of the nicks.
<LPCIBot> Project windmill-devel build #216: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/216/
<wgrant> Bug #796889
<_mup_> Bug #796889: ValidPersonOrTeamVocabulary preloads all IRC nicks into everybody <disclosure> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/796889 >
<wallyworld> i can't see how? the lines you refer to above set up a dict used by the lines that follow to preload the cache
<wgrant> wallyworld: (nick.personID, nicks)
<wgrant> wallyworld: It creates a dict mapping each nick's person to the whole set of nicks.
<wallyworld> ah. shit.
<wgrant> Heh
<wallyworld> that should be *nuck*
<wallyworld> nick
<wallyworld> without the s
<wgrant> That won't quite work either.
<wgrant> Since that will only allow one for each.
<wallyworld> yep. can't argue there.
<wallyworld> something was in the coolaid that day :-(
<LPCIBot> Project devel build #801: FAILURE in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/801/
<abentley> wgrant: yes.
<wgrant> abentley: Great, thanks.
<poolie> hi wgrant, all
<wgrant> poolie: Hi. Was going to check with you on QA for your librarian email fix, but I think I did it.
<poolie> you think you did the qa?
<wgrant> Yes.
<poolie> thanks!
<poolie> +filebug seems pretty screwed
<poolie> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1991BA20
<poolie> cannot get throuh at all
<wgrant> Erk.
<wgrant> BugSummaryJournal contention.
<wgrant> lifeless: Have you seen that before?
<wgrant> A timeout on a BSJ INSERT during a BugTask INSERT.
<wgrant> (but +filebug worked fine for me earlier)
<wgrant> But there's no UNIQUE :/
<wgrant> Must be a table lock somewhere...
<poolie> that's like ~20 failures over 20 minutes
<poolie> unless this is just me, i think it's whatevers-above-critical
<wgrant> It's not just you, and it's not just Ubuntu.
<wgrant> lifeless: Do you know what was cowboyed?
<wgrant> lifeless: BugSummaryJournal is somewhat stuck.
<lifeless> wgrant: how do you mean?
<wgrant> lifeless: We are seeing dozens of timeouts inserting into BugSummaryJournal.
<lifeless> wgrant: all of bsj was cowbowed
<wgrant> lifeless: All *what*?
<wgrant> Is it exactly what was landed?
<lifeless> wgrant: except the appserver code to query (read only, always) when doing tag portlets
<lifeless> the schema, triggers
<wgrant> 202 timeouts of this kind in the last two hours.
<wgrant> Three hours.
<lifeless> thats a problem
<lifeless> we've done the ndt now right ?
<wgrant> It doesn't seem to be done yet.
<lifeless> did this start before the ndt ?
<wgrant> Ah, the appservers have updated in the last 5 minutes.
<wgrant> They were not updated when the problem started.
<wgrant> Let me find the first occurrence.
<wgrant> There were none yesterday, it seems.
<lifeless> ok
<wgrant> chaenomeles/2011-06-14/05755.AR6:Date: 2011-06-14T01:35:55.028700+00:00
<wgrant> First occurrence.
<lifeless> oops id ?
<wgrant> It's in the path
<wgrant> AR6
<wgrant> Today.
<wgrant> To OOPS-1991AR6
<wgrant> s/To/So
<LPCIBot> Project parallel-test build #34: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/parallel-test/34/
<wgrant> Does someone feel like reviewing https://code.launchpad.net/~wgrant/launchpad/bug-796889/+merge/64490?
<StevenK> wgrant: Nice work, r=me
<wgrant> Thanks.
<LPCIBot> Project windmill-db-devel build #388: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/388/
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer:  - | Critical bugs: 204 - 0:[######=_]:256
<LPCIBot> Project windmill-devel build #217: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-devel/217/
<wgrant> Looks like codeimports have almost caught up.
<wgrant> And the failure counts are still small.
<wgrant> This is nice.
<LPCIBot> Project windmill-devel build #218: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/218/
<wgrant> We need a faster test suite.
<lifeless> wgrant: I'm amazed that that picker passed qa with those seqscans
<nigelb> can I run my own launchpad in ec2 and talk to it? Purely for testing applications that use launchpad data where I don't want to mess up real launchpad.
<StevenK> nigelb: ec2 demo
<nigelb> StevenK: there's a script?
<lifeless> that may be broken, but yeah thats the idea
<wgrant> lifeless: I knew it was slow, but decided it might just be qas. And it was behind flags and didn't time out too much.
<lifeless> or you can mess up qastaging as much as you like
<wgrant> nigelb: You could also use a local instance, or staging or qastaging.
<StevenK> nigelb: Sure, bin/ec2 demo from a local branch that is pushed
<lifeless> wgrant: rule 34: its never just [qa]staging
<wgrant> lifeless: Sure, but given it was behind flags I decided not to investigate.
<nigelb> wgrant: I want to set it up so that developers working on the community projects can mess up with the same install.
<lifeless> oh, I missed that. cool.
<lifeless> wgrant: nice work teal
<StevenK> nigelb: *Expensive*
<StevenK> nigelb: Like, hellishy
<nigelb> StevenK: Oh. :(
<nigelb> StevenK: Lp takes a ton of ram?
<nigelb> ram/cpu cycles
<wgrant> lifeless: ValidPersonOrTeamVocabulary has three variants now.
<StevenK> nigelb: We run the LP test suite on c1.xlarge instances. $300+ a month
<nigelb> Oh :|
<StevenK> nigelb: I'd point developers at qastaging, personally
<nigelb> StevenK: I asked jml about that at UDS.
<nigelb> StevenK: basically, we want to mess up with sprints, which we don't have permissions for.
<nigelb> And since staging and qastaging data is cleared every few weeks, we'll need to keep asking for permissions in either of them
<StevenK> staging is every week
<StevenK> qastaging is ... an interesting question
<nigelb> I use my own launchpad. I was just wondering if I can run a common install. Apparently not :)
<nigelb> *launchpad instance
<StevenK> nigelb: You can, for a few hours
<StevenK> After that, it gets expensive
<lifeless> wgrant: 3?
<nigelb> I guess its just easier to run your own launchpad instance.
<StevenK> nigelb: That $300+ figure is running an instance in ec2 for the entire month
<lifeless> nigelb: huh, sprints - create a custom project of your own
<lifeless> nigelb: on qastaging
<lifeless> nigelb: then you can fiddle with sprints for it to your hearts content
<wgrant> lifeless: The affiliation rank stuff is under a second flag, in case we wanted to deploy the other improvements while tweaking this.
<nigelb> \o/
<wgrant> lifeless: Of course the affiliation stuff is fine, it's the rest that's buggy :P
<nigelb> lifeless: thanks. doh. why didn't I think of this.
<lifeless> nigelb: so you can run one on ec2 if you really want, though if its public acccess you'd need to rebrand it :( - but yeah, qastaging should meet your needs Just Fine
<nigelb> lifeless: Oh rebranding. I don't event want to go there, too painful :-)
<StevenK> wgrant: Up for reviewing a short branch of mine?
<wgrant> StevenK: Sure.
<wgrant> Ah, diff there already.
<wgrant> Handy.
<StevenK> Sorry, was just linking a bug.
<wgrant> StevenK: I'm not sure we really want to link to that page.
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer:  - | Critical bugs: 204 - 0:[######=_]:256
<wgrant> It's pretty crap.
<wgrant> Particularly for private PPAs, where the technical details section is a bunch of lies.
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer:  - | Critical bugs: 202 - 0:[######=_]:256
<wgrant> StevenK: We could possibly link to it from +archivesubcriptions.
<wgrant> lifeless: Huh, where'd those two go?
<lifeless> wgrant: http://webnumbr.com/launchpad-critical-bugs is what I'm tracking
<wgrant> Oh.
<wgrant> Excludes private ones.
<wgrant> Right.
<lifeless> yes
<lifeless> gone
<StevenK> wgrant: It was the only good solution that occured, and it sucks.
<nigelb> Bah, no creampie I guess
<nigelb> erm
<nigelb> cream pie on the face.
<wgrant> StevenK: I don't think it's beneficial.
<wgrant> Do you?
<StevenK> I don't think the e-mail is beneficial at all, but sladen obviously thinks it adds value.
<wgrant> A lot of people think that a lot of things add value.
<LPCIBot> Project windmill-devel build #219: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/219/
<rvba> wgrant: Hi William, I've got an MP that really could use your Soyuz expertise ... could you review it? (If you cannot do it I'll grab someone else ;))
<rvba> https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676
<wgrant> Wow that is large.
<wgrant> Hmmm, so we now have 'sequence', 'ordering' and 'priority' as different names for the same concept :/
<StevenK> rvba: But this MP still doesn't address the second use-case for IDS?
<rvba> StevenK: wgrant that's right, the second use case is in a second branch (almost ready)
<rvba> StevenK: https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-previous-series/+merge/64414
<StevenK> "will be *derived* from the previous_series' parents."
<rvba> StevenK: and the UI fix for the second use case is in a third branch.
<StevenK> That makes me very nervous.
<StevenK> It *should* be derived from previous_series. Only.
<rvba> StevenK: you're right, it is only the doc that is wrong.
<rvba> wgrant: I confess this branch has been a real work for me, so don't hesitate to recommend any fixing you'd like to see. It's a big branch so if you agree to review it, I'd be happy to follow your advices to fix it, even if they are plenty ;)
<wgrant> rvba: I am reading it.
<rvba> wgrant: thanks a lot.
<wgrant> rvb: "In result, in case the same package exists in multiple parents, the package from the first parent with this package is the one that will end up in the derived series."
<wgrant> rvba: That's different from the rest of the behaviour.
<wgrant> Where the highest version tends to win.
<wgrant> At least for overlays.
<wgrant> I don't know how it behaves for non-overlays.
<rvba> wgrant: right, Julian says it was good enough for now and we could fix this later if we need to .... but that's a good point ...
<rvba> I'm not sure we can come up with a solution to take the highest version with simple sql queries though ...
<wgrant> They'd get fairly nasty, yeah.
<rvba> And this initialisation thing already takes a rather long time to run sometimes.
<rvba> The advantage of this is that it's pretty simple.
<lifeless> back
<rvba> StevenK: doc fixed. MP needs review ;).
<adeuring> good morning
<LPCIBot> Project db-devel build #633: FAILURE in 6 hr 36 min: https://lpci.wedontsleep.org/job/db-devel/633/
<wgrant> rvba: I've made a first attempt at it.
<rvba> wgrant: thanks!
<allenap> wgrant: Thanks for QAing my stuff over(my)night. And for all the other times you've done it :)
<LPCIBot> Project parallel-test build #35: STILL FAILING in 39 min: https://lpci.wedontsleep.org/job/parallel-test/35/
<LPCIBot> Launchpad Patch Queue Manager: [r=benji][bug=746379] Update the "Latest message" column in
<LPCIBot> DistroSeries:+localpackagediffs when a comment is added in the
<LPCIBot> expander.
<rvba> wgrant: Yeah, thanks for that too ;)
<lifeless> stub: hey, I saw some nice progress towards bugsummary-2
<stub> Yer. Triggers and functions written. If they worked, that would be great :-)
<lifeless> stub: I've proposed nuking readonly mode on the list
<stub> upstream bug fixes are not sticking atm. need to look at it today.
<stub> lifeless: I'm just responding.
<lifeless> cool
<lifeless> stub: How would we fix having to restart appservers, and the time to rebuild the replica after schema change ?
<stub> Appservers could automatically failover when the master db connection is unavailable, and automatically switch back when it is available.
<lifeless> I think thats a different thing isn't it?
<stub> At the moment it is a manual switch.
<stub> At the moment we break out the replica from the cluster so we can upgrade the whole cluster. We could leave it in the cluster, but shut down its replication daemon.
<lifeless> stub: I realised something the other day - the slave rebuild is contributing to our bloat
<stub> If we then had smarter scripts, instead of detecting 'the cluster is in sync' we would detect 'the nodes we care about for now are in sync', we could do the upgrade live. When done, we bring the dead daemon back and the readonly node should then get the updates applied.
<stub> lifeless: Yes, it does.
<lifeless> stub: so, the core of the proposal I am making is 'lets focus on minimal actual downtime'; if we get buyin on doing that (and so far responses have been positive); and if we decide the best way to do that is to leave the readonly codepaths in the appservers - thats fine with me.
<stub> Right. I'd rather not invest time in making RO mode better if we only care about it as a failover mode, because making the extra complexity reliable will be a lot of work and probably cause downtime as we learn.
<lifeless> exactly
<lifeless> stub: I'd like to aim for 4-5 second downtime windows
<lifeless> stub: if we reconfigure-db-connections; kill all current backends (except slony); apply patches; reconfigure-db-connections again
<mrevell> Hello!
<lifeless> stub: based on our previous discussions it seems to me we can get pretty darn close
<stub> We are not a credit card processing system losing thousands of dollars a minute of downtime, and doing real HA is expensive.
<stub> Right. So better to just pause things in HAProxy.
<stub> And deal with an exploded master with real downtime if the master explodes.
<stub> Yer. Appservers need to get more intelligent though and learn to switch themselves over live.
<lifeless> stub: I wouldn't even pause in haproxy
<lifeless> stub: its another moving part to change and deal with
<stub> We could possibly pause everything in pgbouncer
<lifeless> certainly
<lifeless> thats something we can iterate towards or even do upfront
<lifeless> I'm kindof of hopeful to be able to start doing this next week
<stub> Sounds excellent since I'm on holiday next week ;)
<lifeless> bleep
<lifeless> ok, week after :>
<lifeless> I'd like you around for the first few times
<stub> I hear there is a sprint on or something
<lifeless> till we get the quirks out
<nigelb> My inbox seems to thinki any mail from lifeless is Priority Inbox mail. Interesting.
<lifeless> good inbox
<stub> After a while of trying to train it, I realized that there is no such thing as priority email ;)
<nigelb> It works okay for me so far. Mostly.
 * stub runs off to the ATM and to grab blinner
<lifeless> blinner?!
<bigjools> I'm guessing he's *really* hungry
<LPCIBot> Project windmill-db-devel build #389: STILL FAILING in 49 min: https://lpci.wedontsleep.org/job/windmill-db-devel/389/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #802: FIXED in 6 hr 40 min: https://lpci.wedontsleep.org/job/devel/802/
 * bigjools reads lifeless's wall of email
<lifeless> one more coming up
<bigjools> lifeless: it might be helpful if you could format the emails into distinct sections, I usually end up re-reading your emails as it's hard to take 'em all in in one go
<lifeless> bigjools: thats good to know - I do sometimes, but I kindof run into 'is this an email or a wiki page' feeling sometimes too
<bigjools> lifeless: they do read like a steam of consciousness :)
<bigjools> stream*
<lifeless> well, I do edit and shuffle and so on; trying to paint a narrative
<lifeless> my head is nowhere near as organised
<bigjools> lifeless: in any case, I've been wanting to sort out the schema change downtime story for a loooong time
<bigjools> most schema changes don't involve interdependent code
<jtv> gmb: got time for a review?  I have two MPs ready including https://code.launchpad.net/~jtv/launchpad/bug-791204/+merge/64390
<gmb> jtv: I'm just working on one for danilos, but I'll take a look when I'm done with that.
* gmb changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer:  gmb | Critical bugs: 202 - 0:[######=_]:256
<stub> A steaming pile of consciousness
<jtv> great, thanks
<jtv> A stream of <garble>ss
<jtv> (Just for the sake of the play on words of course, not my actual opinion!)
<gmb> danilos: Accepting your "it is thoroughly tested" assertion, r=me.
<gmb> (Although the evidence for that is pretty strong, I'll grant you)
<danilos> gmb, heh, thanks :)
<danilos> gmb, after you get relaxed a bit with the one from jtv (ha), you can pick the following one from me as well: https://code.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-loading/+merge/64186 (that one is oversized a bit though, hope you don't mind)
<jtv> danilos: hey hey hey, I had _two_!
<gmb> danilos: jtv speaks the truth. But I'll try to come to yours early this afternoon.
<jtv> This could be a fun exercise in scheduling algorithms
<danilos> jtv, hey, I had three but didn't want to take all the available "room" for others to get their code reviewed :)
<danilos> jtv, unlike, ahem, some other people
<lifeless> jtv: the docstrings for match_exact_string etc are misleadning
<lifeless> jtv: the :return: is incorrect - it returns a storm expression whose result will be True if ...
<jtv> danilos: hey, if you leave room for other people, don't get all disappointed when other people use it!
<jtv> thanks lifeless â I'll have a look right away.
<lifeless> jtv: (AIU the code)
<gmb> danilos, jtv : http://oo00.eu/
<gmb> Anyway.
<jtv> lifeless: ISWMâyou're right
<jtv> danilos: we've just heard the URL of the banshee.
<jtv> Go on, ask for whom the bell tolls: it tolled for chromium!
<jtv> It died.
<danilos> jtv, gmb: :)
<lifeless> allenap: hi
<lifeless> allenap: rabbit: do we reset SIGPIPE when spawning it ?
<bigjools> lifeless: he's out for a bit, will be back soon
<lifeless> thanks
<danilos> jtv, I've been hitting a lot of webkit crashes with epiphany lately, but just on this computer (even though it's identical arch to my laptop and all the same versions are used)
<danilos> jtv, so I just figured out it's probably your fault and learned to live with it (or maybe hardware, but it's so much nicer to blame you again)
<jtv> danilos: agreed, it makes perfect sense given that I see the same problem.
<jtv> See that everyone?  This is why I liked the transition to squads!  :-)
<danilos> basically the only reason, I am sure :)
<jtv> oh yes
<allenap> lifeless: I haven't done anything special with signals.
<lifeless> http://bugs.python.org/issue1652
<bigjools> danilos: whatever the problem is, it's probably jtv's fault
<jtv> All of a sudden I'm not so sure about this squads thing any more.
<danilos> bigjools, glad to hear that, that's how it used to work in the previous team structure as well!
<allenap> lifeless: That's a great find. I'll try that simple change. (I'll do it in a separate branch from the subprocess changes I made.)
<lifeless> cool
<gmb> jtv: r=me on your first branch; your tests need some comments though.
<jtv> gmb: thanks, will do.  I always try writing 'em clear enough that they don't need comments, but after that I don't mind adding them if they're still needed.  :)
<gmb> jtv: Well, I'm lazy, so I tend to say they all need comments otherwise I need to think.
<jtv> I try to encode the whole story in the function name.  :)
<gmb> jtv: Oh, I know. And that often works. Sometimes though it requires context that I don't have to be able to understand the encoding.
<jtv> Right â problem is it's very hard for the person who wrote a text to judge that, so that's where I rely heavily upon the reviewer.
<StevenK> gmb: I have a short branch for you if you're free.
<gmb> StevenK: Sadly, there's a queue at the moment. I may get to it later but I can't promise anything
<StevenK> That's okay. I will beg bigjools, since he whinged at me about the bug.
<StevenK> bigjools: https://code.launchpad.net/~stevenk/launchpad/reject-changes-file-no-email/+merge/64523 please
 * gmb -> changing locations. BBIW.
<jml> lifeless: ping
<lifeless> hi
<jml> lifeless: oh hi, was going to ask for a quick chat, but maybe we could do it tomorrow instead
<lifeless> jml: tonight would be better; tomorrow night leads into thursday morning which is at an uncivilised time
<lifeless> jml: I need to take recycling out and stuff like that, but in 15-20 I can chat with you
<jml> lifeless: perfect. skype me when you're ready.
<bac> hi bigjools
<bigjools> bac: hey
<bac> bigjools: could i have a mid-imp call with you re: bug 519857
<_mup_> Bug #519857: Upload processor OOPSes with bad email addresses <lp-soyuz> <oops> <soyuz-upload> <Launchpad itself:In Progress by bac> < https://launchpad.net/bugs/519857 >
<bigjools> bac: stevenk already fixed it.
<bigjools> ah sorry
<bigjools> no he didn't
<bigjools> different bug!
<bigjools> bac: so
<bigjools> what needs to happen is that no oops gets thrown, and we send an email to whoever signed the changes file
<bigjools> I just noticed you said mid-imp...
<lifeless> bigjools: what did you mean by '
<lifeless> 21:02 < bigjools> lifeless: in any case, I've been wanting to sort out the schema change downtime story for a loooong time
<lifeless> 21:02 < bigjools> most schema changes don't involve interdependent code
<lifeless> '
<bac> bigjools: yeah, i was planning to catch the ParseMaintError in process()
<bigjools> lifeless: we should be able to make schema changes on a live DB
<bigjools> bac: in this case it's already sending a rejection for another reason
<bigjools> bac: so we don't really need to do anything other than catch and ignore the error
<bac> bigjools: what do you mean by "in this case"?
<bigjools> bac: the traceback on the bug
<bigjools> it's in the middle of processing a rejection
<bigjools> which I also suspect is because a previous bit of code already noticed the bogus email address
<bac> bigjools: i intended the code change to look something like: http://pastebin.ubuntu.com/626527/
<lifeless> bigjools: do you mean that most schema changes don't alter the meaning of existing fields, nor require new fields to be populated by appservers (at least initially) ?
<lifeless> bigjools: if thats what you mean, I agree :)
<bigjools> lifeless: yes!
<bigjools> lifeless: and we should structure all our patches to be like that as much as possible
<lifeless> bigjools: so there are some slony limits that make applying while truely live impossible today, but we can come close.
<bigjools> bac: I would not do that
<bac> bigjools: i think the code parts are easy.  where i need some guidance is creating new package suite in lib/lp/archiveuploader/tests/data/suite
<lifeless> bigjools: yes, if this proposal gets off the ground stub and I will start rejecting patches that are /not/ like that
<bac> bigjools: ok
<bigjools> bac: the code here has changed *significantly* since the bug was filed, because steve re-wrote the notification stuff
<bac> bigjools: yes, that was obvious as nothing matched up
<bac> bigjools: but the problem still exists
<bigjools> bac: yes - what I would do is catch the error in lib/lp/soyuz/adapters/notification.py - get_recipients  ()
<bigjools> and just not add any invalid emails to the recipient list
<lifeless> benji___: lp:~benji/launchpad/bug-697735 seems to be deleting some db patches. ..
<bigjools> the changesfile signer will always be valid if we get this far so that's the last-ditch option
<bigjools> lifeless: excellent
<bigjools> bac: "blamer" in that code is the signer
<benji> lifeless: ooh, that doesn't sound right; thanks, I'll take a look
<lifeless> benji: === removed file 'database/schema/patch-2208-63-0.sql'
<lifeless> ...
<bigjools> bac: so we need a test.  For that we make a dodgy package and try to upload it.
<bac> bigjools: yes, *that* is the part that i got hung up on yesterday
<bigjools> bac: however thinking about it that's overkill for this case, so we could just do a unit test case for get_recipients
<bac> bigjools: when i make a bad email address in the changes file i need to sign it
<bac> bigjools: yes, that may be true now that i'll not be changing process()
<bigjools> bac: right.  fix get_recipients + unit test = simples.
<bac> bigjools: if i did need to create a new test/data/suite are there any instructions on how to create and sign one?
<bigjools> then we can QA it with an upload that has a dodgy maintainer
<bigjools> bac: no :(  but it's not too hard.
<bac> bigjools: i couldn't find any keys, etc
<bigjools> there's a private key in the tree
<bigjools> you need to add it to your local keyring
<bac> i found secring.gpg but it didn't have the right keys
<bac> bigjools: ok, i'll go down that route and it should be much simpler
<bigjools> bac: huh so it doesn't, that's odd.  I have the key in my keyring, I should add it to the tree then.
<bac> bigjools: win
<bigjools> can't remember wtf I got it from!
<bigjools> probably celso
<bigjools> bac: ok cool, is that all you need?
<bac> bigjools: part of soyuz's oral history
<bac> bigjools: a-ok.  thanks for your time.
<bigjools> nae prob
<lifeless> night all
<bigjools> nn
 * bigjools -> lunch
<jtv> gmb: got time for a really small one?  It might be fun.  https://code.launchpad.net/~jtv/launchpad/bug-797151/+merge/64537
<jtv> (It's lower priority than my other one though)
<gmb> jtv: I'm still working through your other one - got distracted by vittles. I'll take a look after I've looked at Danilo's 900-liner, assuming I have time.
<jtv> gmb: don't force yourself â the small one is really low-priority, but you might find it a nice "lite bite" after danilo's huge one
<gmb> jtv: That's what I'm thinking :)
<jtv> :)
<LPCIBot> Project parallel-test build #36: STILL FAILING in 1 hr 12 min: https://lpci.wedontsleep.org/job/parallel-test/36/
<StevenK> bac, bigjools: I've already fixed the bug?
<bigjools> StevenK: different bug, ignore
<jml> brb
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #634: FIXED in 5 hr 52 min: https://lpci.wedontsleep.org/job/db-devel/634/
<jml> ok. that was a confusing update.
<bigjools> gmb: may I avail of your reviewership please: https://code.launchpad.net/~julian-edwards/launchpad/link-LCV-bug-783442/+merge/64550
<bigjools> ah cock seen some lint already
<gmb> bigjools: I might be a while; working on a 900-liner from Danilo and then a smaller one from Jeroen, but I'll try to get to it before EoD.
<bigjools> gmb: ok np
 * danilos is happy gmb is ignoring the fact that it's nine hundred sixty something, which is closer to 1k ,)
<gmb> Sssssh.
<bigjools> mine's a small branch :)
<jml> mrevell: do you have any notes from any rounds of disclosure user testing?
<mrevell> jml, Let me check what I have.
<gmb> danilos: Your not-quite-1ker is approved :)
<gmb> Now then, something smaller...
<danilos> gmb, cool (I read that first as "you are not quite approved" :))
<gmb> Heh.
<danilos> gmb, thanks again, I ain't bothering you with any more reviews then :)
<gmb> danilos: Heh. I doubt I'll have the time or brain power for them anyway :).
<james_w> flacoste, seen bug 797088?
<_mup_> Bug #797088: Launchpad has removed privileges that UDD importer requires to function <Launchpad itself:New> <Ubuntu Distributed Development:New> < https://launchpad.net/bugs/797088 >
<flacoste> james_w: now, i did
<flacoste> james_w: we just need to give core-dev permission to upload to Debian
<flacoste> or make them a maintainer
<flacoste> james_w: it's a configuration problem iow
<james_w> flacoste, he reports a problem with lucid too?
<flacoste> james_w: that would be surprising
<james_w> flacoste, http://package-import.ubuntu.com/status/language-pack-nds.html#2011-06-14 14:45:28.882406
<james_w> maverick
<flacoste> james_w: that script is still run as you, right?
<flacoste> hmm
<flacoste> it may be checkUpload permission that isn't getting this right
<flacoste> actually, it might just be that
<james_w> yep
<flacoste> the easiest thing would be to make you a maintainer of both Ubuntu and Debian temporarily
<gmb> bigjools: r=me
<flacoste> that would solve the permission problem immediately
<bigjools> gmb: ta
<flacoste> bigjools: call time
<flacoste> bigjools: https://launchpad.net/baltix/+series
<flacoste> bigjools: you cut off
<flacoste> bigjools: call me back
<LPCIBot> Project windmill-devel build #220: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/220/
<benji> gmb: boo!  I have a review for you: https://code.launchpad.net/~benji/launchpad/bug-697735-2/+merge/64563
<gmb> benji: Looking; waiting for the diff.
<benji> gmb: cool
<rvba> gmb: Thanks for the review!
<gmb> rvba: np
<gmb> benji: I got fed up of waiting for the scanner and looked at the revisions in loggerhead. r=me. Nice work.
<benji> gmb: cool, thanks for the review
<gmb> np
* gmb changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer:  - | Critical bugs: 202 - 0:[######=_]:256
 * gmb -> afk for a bit
<LPCIBot> Project windmill-devel build #221: STILL FAILING in 45 min: https://lpci.wedontsleep.org/job/windmill-devel/221/
<jml> I'm doing some LEP restructuring. Let me know if I'm getting in your way.
<james_w> flacoste, maxb has a workaround that lessens the impact, but the importer will still be unable to deal with new packages, SRUs etc.
<flacoste> james_w: i have a code solution
<james_w> flacoste, cool
<flacoste> for frozen series, we need to check the UPDATE pocket
<james_w> flacoste, that will prevent setting official branches for obsolete series?
<flacoste> probably
<flacoste> i don't know
<flacoste> actually
<flacoste> i don't know the intimates of checkUpload
<flacoste> but would that be a problem?
<james_w> it sounds better than what we have, but I'm wondering if we actually want a different set of rules for this
<flacoste> why should we care about obsolete series
<flacoste> ?
<james_w> c.f. changing the packaging links
<james_w> well, the importer currently sets them up when it does the initial import of an old package
<flacoste> it could simply not set them up
<james_w> obviously most will be done already
<james_w> it could
<flacoste> obsolete is obsolete
<james_w> do you still propose setting me/pkg_import to be maintainer of Debian?
<flacoste> in a way
<flacoste> either that
<james_w> yeah, but it's still useful to easily refer to it sometimes
<flacoste> but i think we should simply mirror the upload privileges from Ubuntu
<james_w> obviously less useful over time
<james_w> that sounds sensible
<flacoste> jml: wow, LEP cleanup spree! nice!
<flacoste> sinzui: call?
<sinzui> starting mumble
 * flacoste bets he's already in mumble
<flacoste> and lost
<flacoste> sinzui: we are in mumble-not-working-first-time-land i think
<flacoste> i'm hearing you
<flacoste> but you are obviously
<flacoste> i just did
<jml> flacoste: yeah. only a little bit more to go.
<jml> flacoste: you might want to look over https://dev.launchpad.net/Projects/DerivativeDistributions & add info
<jml> flacoste: please feel free to do things like add columns or change structure
<maxb> flacoste: Even if a series is obsolete, there is still some value in constructing a bazaar branch for its history and registering it with launchpad. We also still have a large number of "broken" imports that are being worked through, so retroactively producing history isn't an especially abnormal case.
<jml> g'night all.
<timrc> I'm having a head-scratching moment... I've modified lp/lib/soyuz/model/archive.py to turn "private" into a getter/setter and by doing this, I fail a test in lib/lp/soyuz/tests/test_processaccepted.py (test_accept_copy_archives:133, error: http://pastebin.ubuntu.com/626756/).  This particular test runs lib/lp/soyuz/scripts/processaccepted.py which (when I look at output) returns a failure (https://code.launchpad.net/~timrchavez/l
<timrc> aunchpad/set_ppa_private_from_api_724740-2/+merge/63950) -- I'm not sure if that's actually intended or not (looks like: http://pastebin.ubuntu.com/626766/), which begs the question: how can I retrieve the oops report from the id ?
<timrc> trying this again..
<timrc> I'm having a head-scratching moment... I've modified lp/lib/soyuz/model/archive.py to turn "private" into a getter/setter (https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-2/+merge/63950) and by doing this, I fail a te
<timrc> st in lib/lp/soyuz/tests/test_processaccepted.py (test_accept_copy_archives:133, error: http://pastebin.ubuntu.com/626756/)
<timrc> This particular test runs lib/lp/soyuz/scripts/processaccepted.py which (when I look at output) returns a failure / generates an oops.  I tried to extract details from the 'request' object which gives me an oops report (http://pastebin.ubuntu.com/626766/) -- is there a better way of getting the oops report from the id?
<Ursinha> +15
<Ursinha> argh
<LPCIBot> Project windmill-db-devel build #390: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/390/
<lifeless> poolie: what TZ are you in now :)
<matsubara> timrc, oops reports are kept in /var/tmp/lperr/ for your local LP instance
<timrc> matsubara, ah thank you!
<matsubara> np
<poolie> hi lifeless, i'm in us/pacific
<flacoste> james_w, maxb: how about if we always do the check for the development series? would that make sense?
<lifeless> flacoste: allo allo
<flacoste> hi lifeless
<maxb> flacoste: That feels like it could do the right thing
<james_w> flacoste, are upload permissions series specific?
<lifeless> IArchive.canUpload is
<flacoste> james_w: they are archive specific
<flacoste> plus the series status check
<james_w> then it sounds ok
<james_w> if you can e.g. upload to lucid without being able to upload to devel then we probably want the union?
<lifeless> hang on a second though
<sinzui> jcsackett: do you have time to mumble
 * flacoste hangs
<lifeless> these branches are meant to end up being what is built
<flacoste> lifeless: we are talking about the source package metadata really
<flacoste> so it's not the permission to upload to the branch
<lifeless> so push to them (and by extension setting the official pointer to an existing branch) is meant to be equivalent to upload
<lifeless> flacoste: this is the 'setting official branch 401s' right ?
<flacoste> yes
<jcsackett> sinzui: sure. i'm on now.
<flacoste> but they want to set official branch for obsolete series also
<sinzui> jcsackett: sorry. I fell off
<maxb> Do I misunderstand? I thought package sets were series-specific? So that would imply package set upload permissions are implicitly series-specific?
<flacoste> for frozen, one, we should check against the UPDATE pcoket
<lifeless> so, I'm saying that perhaps setting the official branch should be treated as an upload, even for frozen. Who can upload to obsolete series ?
<flacoste> i don't think anybody can upload to obsolete series
<maxb> No one should be able to upload to obsolete series, but the UDD importer should be able to register that it has created an official Bazaar representation of what was uploaded when the series was active.
<lifeless> righg
<lifeless> right
<flacoste> do we really care?
<flacoste> i mean obsolete is obsolete
<lifeless> so what we're doing at the moment is getting rid of the special case 'folk in ~ubuntu-branches can do anything' rule; perhaps this suggests that the importer really should be a role.
<flacoste> we sometime delete the published files there
<flacoste> not automatically
<maxb> At the moment we are still at the maturity level in the UDD importer where wanting to erase all the current branches for a package and redo from saved published files is not entirely out of the question.
<flacoste> sure
<flacoste> but obsolete series?
<maxb> Yes
<flacoste> frozen and current i get
<flacoste> but obsolete
<flacoste> i don't
<flacoste> obsolete is obsolete
<flacoste> if it's not obsolete it's not obsolete
<flacoste> if you see what i mean
<flacoste> we shouldn't care about obsolete series at all in a way
<flacoste> that's before our time
<flacoste> i'd want strong justification for introducing special cases just for supporting "uploads" to obsolete series
<maxb> uploads would be silly, yes. I think this is a demonstration that official branch linkage is _not_ wholly equivalent to uploads
<flacoste> still
<flacoste> why do we care
<flacoste> ?
<flacoste> the whole point of obsolete, is that we don't have to care anymore
<maxb> We care because we consider history of packages to be important.
<maxb> The importer tries to import Debian woody!
<flacoste> that's misguided to me
<lifeless> well
<lifeless> we can import something without linking it to the official series.
<lifeless> so I think its a separate question
<flacoste> right
<flacoste> in a way, it's no business to me if UDD should try to import woody or not
<flacoste> i do have an opinion, but it's irrelevant unless you ask for it :-)
<lifeless> Everyone has one :)
<maxb> A secondary aspect of this is: Launchpad has broken the importer through insufficiently well understood changes - please provide sufficient time to recode the importer if you're going to forcibly withdraw rights to do something it could do before.
<flacoste> maxb: well, it's more a case of missing qa than anything else
<flacoste> we didn't intend to remove rights
<flacoste> but the check we did (could james_w still set official package branch on staging) wasn't the whole story it seems
<lifeless> maxb: thats a bit harsh
<lifeless> maxb: we suggested ages ago it should be an LP managed service
<lifeless> maxb: and this change was discussed with the folk running the importer
<lifeless> lol
<lifeless> flacoste: check out https://bugs.launchpad.net/null/+bug/420937
<flacoste> in a away
<lifeless> 'Directly subscribed
<lifeless> No subscribers'
<flacoste> in a way, i could special case OBSOLETE
<flacoste> and if the distroseries is OBSOLETE, use the current devel one
<flacoste> with a XXX to a bug in the importer
<flacoste> or no bug if we don't want to support changing sourcepackage metadata for OBSOLETE
<flacoste> i'd like that the same permission check would be used for the other regression we had (can't change packaging link when there is an upstream with shared translations)
<flacoste> lifeless: what do you think of the above^^^
<maxb> I don't mean to be harsh, I'm just a bit confused why, having discovered this issue, continuing to deny the importer these permissions seems to be an option under consideration
<lifeless> maxb: we're basically doing development that was skipped in the initial implementation
<flacoste> yeah
<flacoste> it's not clear if we want to support those permissions
<lifeless> these are questions that were invisible with the celebrity and identified as work that should be done
<flacoste> but i agree that removing functionality should allow for a smooth transition on your side
<flacoste> that's why i'm suggesting the OBSOLETE special cases (instead of the ubuntu-branhces special case, which I really don't want to reintroduce)
<lifeless> flacoste: uhm, so I think the translations link is less risky than setting these branches
<lifeless> translations link could well use the rule the branch setting uses, just not the other way around
<flacoste> agreed, but i'd rather we still one logic
<flacoste> the more strict one
<lifeless> yeah
<flacoste> which should work fine for the other case also
<flacoste> agreed
<lifeless> I think that we should ask the TB
<lifeless> erm
<lifeless> actually
<flacoste> about setting OBSOLETE series
<flacoste> ?
<lifeless> the importer isn't a debian uploader
<lifeless> lets talk debian for just a minute
<lifeless> ubuntu upload rights won't let the importer upload to debian
<flacoste> no
<lifeless> so we're going - if someone queries LP - to have an 'uploader' to Debian-in-LP
<flacoste> but we can give it upload righ on launchpad
<lifeless> this is at best weird.
<flacoste> i agree
<lifeless> so I have an alternative proposal
<flacoste> but not weirded than having Registry Administrator be the maintainer
<flacoste> for Debian in Launchpad
<lifeless> add a role to distribution called 'package importer'
<lifeless> flacoste: I think its weirder, or at least its additional explanation.
<flacoste> for Debian, i was thinking of creating a team called 'LAunchpad Debian Maintainers'
<flacoste> owned by Registry Admin
<flacoste> and with the importer in it
<flacoste> that would actually clarify things a bit
<lifeless> So the question is - is that simpler than directly representing 'there is this service that can write to any official branch and set any official branch for any series in $distro'
<lifeless> I'm not a big fan of directly modelling every little thing
 * flacoste neither
<lifeless> this seems to be a biggish thing though
<lifeless> consider that:
<lifeless>  - long term we don't want the package importer triggering builds [this came up in tb discussion I think]
<lifeless>  - the package importer would like to edit obsolete packages which normal uploaders cannot
<flacoste> (I'm -1 on that second item FTR)
<flacoste> gardening obsolete data isn't a use-case i'd like to support
<lifeless>  - and it would like to edit debian and linaro and perhaps more in future but normal uploaders have no use case for doing this
<lifeless> flacoste: I'd like to put that gardening question aside; it is something that was previously deliberately supported; we should ask the TB whether they really care IF we decide its too hard to support
<flacoste> that sounds like YAGNI
<lifeless> flacoste: editing debian is a current job for it; when linaro goes all derived distro that will also be a job for it
<bac> flacoste: :(
<flacoste> bac: i pushed your guilt-trip button? ;-)
<bac> yeah...
<bac> where is that reset?
<LPCIBot> Project parallel-test build #37: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/parallel-test/37/
<flacoste> bac: it's called daily meditation
<flacoste> lifeless: it don't think 'too hard' is the criteria here
<flacoste> lifeless: it's more of a "limt the set of things we support" criteria
<maxb> Regards "gardening" - I think gardening deserves to be special - for example, the importer MUST be able to push to branches for the release pocket of a released distroseries, but normal uploaders probably shouldn't
<lifeless> when we started this
<lifeless> we said 'lets use upload rights, they exist and should include the right people and the importer is essentially getting upload rights'
<lifeless> my points above were saying that we're running into feedback of various sorts from the folk who will be using what we create, that says upload isn't all that good a fit
<lifeless> In terms of what we support, I think the distro really drive that
<flacoste> i'm going to fix to special case OBSOLETE series
<flacoste> and punt on whether to model a package-importer or not
<flacoste> that way the importer can work tomorrow
<flacoste> and eventually we'll iterate again to make this 'correct' at some point
<flacoste> what we'll have short-term is better than what we had before
<flacoste> it might need further tweaks eventually
<lifeless> sure; I think in parallel we should ask the TB their thoughts - they may love having the ability to fiddle with this data for obsolete [by any uploader]
<lifeless> or they may hate it
<lifeless> either way we'l get input into how urgently we should come back to it
<flacoste> sure
<lifeless> sinzui: any idea about the cause of https://launchpadlibrarian.net/73532151/screenshot4.png ?
<lifeless> (bug 797413) - I'm trying to assess for triage
<sinzui> wow that is new
<_mup_> Bug #797413: Launchpad 'h' font broken <Launchpad itself:New> < https://launchpad.net/bugs/797413 >
<sinzui> lifeless: I cannot say what is wrong.
<sinzui> lifeless: 'h', 'L' and 'i' are the tallest letters in Ubuntu font. since the i is not broken, I do not know what is wrong with the 'h'
<lifeless> speaking of fonts
<lifeless> any chance we can make our text darker ?
<lifeless> (asking as a user, not TA)
<flacoste> lifeless: the contrast we have is within the W3C accessibility chart
<flacoste> interesting that my tests to check permission on frozen and obsolete series is passing :-/ something fishy with the tests i guess
<flacoste> but that's for tomorrow
<lifeless> flacoste: ><
<flacoste> time to go
<lifeless> flacoste: well, chart or not, we do get a stream of bugs about it, and at least for me, I have to run on max brightness to read the text easily.
<sinzui> lifeless:  The font colour is #333 which is defined in both the web standards and Ubuntu light themes. They should be the same. We know something is amiss when they are not
<sinzui> lifeless: I believe the goal is to make the Ubuntu web experience look like a desktop experience
<lifeless> sinzui: ah, so I don't run the normal ubuntu theme... because its too hard on my eyes
<sinzui> lifeless: I hope the new drive for accessibility will be addressed in the light themes then. I personally miss them. Oneiric alpha 1 does not ship with a working default theme.
<poolie> abentley, hi
<poolie> bryce was asking why you retargeted https://bugs.launchpad.net/python-keyring/+bug/796873
<_mup_> Bug #796873: ec2 land generates gnomekeyring.IOError if run over an ssh session <Python Keyring:New> < https://launchpad.net/bugs/796873 >
<abentley> poolie: it looks like the appropriate project to me.
<lifeless> wow
<lifeless> > 50% of our active reviews are approved-but-not-landed
#launchpad-dev 2011-06-15
<sinzui> StevenK: mumble?
<sinzui> sorry.
<StevenK> sinzui: http://pastebin.ubuntu.com/626955/
<sinzui> I think I have isolated the cause of several X session crashes. Typing certain number sequences in email and my editor causes me to drop into the console
<sinzui> I am going to test this now in IRC by typing my phone number, which I cannot do in email.
<sinzui> 301.346.9229
<sinzui> okay. I can type it here.
<LPCIBot> Project parallel-test build #38: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/38/
<wgrant> So can't test/release the pickers further :(
<wgrant> Bad buildbot.
<wgrant> Blocking codeimport QA :(
<lifeless> *blink* 1209 Time Outs
<wgrant> I would tend to doubt that.
<wgrant> But I guess.
<lifeless> yesterday was ndt-fail was't it
<StevenK> It was
<wgrant> It was. I'd expected more.
<wgrant> I guess it was a quiet time.
 * wgrant cleans up dogfood's shelf.
<LPCIBot> Project windmill-devel build #222: STILL FAILING in 14 min: https://lpci.wedontsleep.org/job/windmill-devel/222/
<StevenK> Uh oh
<StevenK> bzrlib.errors.ConnectionError: Connection error: while sending GET /~launchpad-pqm/bzr-git/devel/.bzr/repository/packs/8dff80eb7a417959127398ef4268cdd3.pack: [Errno 110] Connection timed out
<StevenK> Whee!
<LPCIBot> Project windmill-devel build #223: STILL FAILING in 7.1 sec: https://lpci.wedontsleep.org/job/windmill-devel/223/
<StevenK> Sigh
 * StevenK shoots the builder in the face
<wgrant> sinzui: Have you updated buildbot and convinced StevenK to fix jenkins?
<wgrant> (although jenkins might fix itself if you've fixed launchpad-dependencies)
<StevenK> It ought to
<spiv> StevenK: incidentally the package importer got some timeouts from Launchpad a few minutes before you said that.
<wgrant> spiv: Is it OK now?
<spiv> wgrant: yes, it just had a few at 01:41 (jubany time)
<wgrant> Erk.
<spiv> wgrant: see the most recent failures on http://package-import.ubuntu.com/status/
<wgrant> http://package-import.ubuntu.com/status/libgit-ruby.html
<wgrant>         <h1>Sorry</h1>
<wgrant>         <p>
<wgrant>           Launchpad is offline for scheduled maintenance.
<wgrant>           We should be back soon.
<wgrant> What.
<wgrant> LOSA ping.
<hloeung> wgrant: hi
<wgrant> hloeung: Did something happen to the network/haproxy/something at 11:41?
<hloeung> I don't think so
<hloeung> what's the problem?
<wgrant> hloeung: jubany got a scheduled maintenance 503 from haproxy.
<wgrant> And jenkins (on canonicloud) got a connection timeout to crowberry.
* jtv changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer:  jtv | Critical bugs: 202 - 0:[######=_]:256
<wgrant> huwshimi: Hi.
<huwshimi> wgrant: Hey there
<wgrant> huwshimi: Have ytou looked at the new person picker?
<wgrant> huwshimi: Try searching in the assignee picker on https://bugs.qastaging.launchpad.net/launchpad/+bug/1234
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<wgrant> huwshimi: I'm wondering about the "Details..." link... it seems less than ideal.
<huwshimi> wgrant: No I haven't. I'll take a look
<huwshimi> wgrant: Is it supposed to open in a new window?
<wgrant> huwshimi: I believe that is the intention. I'm not convinced it's the ideal solution, but it was the easiest at the time.
<huwshimi> wgrant: It shouldn't be a green link then
<wgrant> That's what I said.
<wgrant> huwshimi: But it also doesn't navigate away from the current page.
<wgrant> So it might not be blue either.
<huwshimi> wgrant: I believe green links are for doing an action that will affect the current page.
<huwshimi> wgrant: Also the ellipsis would imply that it's going to an action on the page too
<huwshimi> wgrant: So this is to help people get some further information about who the person and to know they are selecting the correct person?
<wgrant> huwshimi: I believe so.
<wgrant> huwshimi: I think it should really expand the picker entry inline.
<wgrant> But that's a bit difficult, AIUI.
<huwshimi> wgrant: What's the difficulty?
<wallyworld___> wgrant: huwshimi: sinzui wants to navigate to the person page since we can't pre-guess the info they will need to make their decision
<wgrant> Jumping out of an inline picker into a new page seems... questionable.
<wallyworld___> green was chosen as the colour because the action of clicking does not navigate away from the current page - that's what was communicated to me anyway
<wallyworld___> questionable maybe, but how else do we do it?
<wallyworld___> it's not that uncommon a paradigm
<wallyworld___> a lot of the web pages i visit do that
<wgrant> Popups that spawn new pages? Odd.
<wgrant> I mean, these popups exist just to return data to the original page.
<wallyworld___> yes, but you need to be able to tell you are picking the correct thing - that's one of the tenants of the disclosure work
<jtv> Not quite the same, but doesn't the help system already face this problem?  Any ideas that could be cribbed from there?
<jtv> wallyworld___: tenets, I hope.  :)
 * wallyworld___ can't spell
 * jtv was worried about LCP for a moment
<LPCIBot> Project windmill-devel build #224: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/224/
<wallyworld> jtv: i haven't used the help system. that would be RTFM and men don't read instructions :-)
<jtv> wallyworld: eerie how you identify the reason for my uncertainty there
<jtv> But still
<jtv> I think it also pops up a window with text that may have links, which may open new browser windows.
<jtv> In which case, crib copy & steal!
<wallyworld> jtv: well the picker does that now so there's no need to steal anything
<jtv> What I mean is, doing whatever it does would technically make it a UI pattern.  :)
<wallyworld> ah right
<huwshimi> Opening a new window is certainly suboptimal. Can someone can give me some more information on why we can't guess what info might be useful for making this descision?
<wallyworld> huwshimi: what info is needed depends on the context of what is being picked and what it is being picked for, and also the user's prior knowledge
<wallyworld> there's no easy 2 or 3 lines of info what can be displayed in every case that is guaranteed to be what the current user wants to see
<wallyworld> so we have to take them to the indx page of the thing they think they want to pick so they can poke around and see if it's what they want
<huwshimi> wallyworld: Can you give me an example?
<wallyworld> picking an assignee for a bug task
<wallyworld> or a source package for ....
<wallyworld> i haven't got a good example of the latter one
<huwshimi> wallyworld: And what is the difference in information that might be required between those two?
<wallyworld> so picking an assignee, you may want to check what other bugs they've worked on before giveng them the new one
<wallyworld> huwshimi: i know little about source packages in lp so can't really given a good example
<wallyworld> huwshimi: i'm more or less paraphrasing what was told to me when i asked these questions
<huwshimi> wallyworld: No problems, I'm just trying to understand the problem.
<wallyworld> sure, that's offering additional info :-)
<wallyworld> /that's/just
<wallyworld> huwshimi: you maybe want to join our standup tomorrow and we can discuss :-)
<huwshimi> So really this is more about a convenient link to the person's profile rather than just providing additional info for the picker
<huwshimi> Which is outside the scope of what you'd want to accomplish with the picker
<wallyworld> yes
<huwshimi> As in, it's not the job of the picker to provide that level of information about the user. That's additional work you might want to do to figure out who this person is.
<wallyworld> because there's no one size fits all solution for what extra info could or should be displayed in the picker for each context
<wallyworld> yes
<wallyworld> i think you have it :-)
<wallyworld> the picker should try and provide as much as it can so the user doesn't have to click the link :-)
<wallyworld> and we do that with the newly added work to show the nicks and correct ordering and badges etc
<wallyworld> but we also need to offer a way to get extra info if what's there doesn't suffice
<huwshimi> OK, I think the "Details..." link implies that you're going to get a little bit more information and what might need to happen is just to rename the link and drop the ellipses and change the colour. I'm not sure what our pattern should be for external links.
<wallyworld> huwshimi: we can implement whatever is deemed necessary, just let us know what is needed in terms of styling :-)
<huwshimi> So that link could be something like "View profile"
<wallyworld> and get agreement from the stakeholders
<wallyworld> sounds reasonable for a person picker
<huwshimi> And make the link blue
<wallyworld> perhaps different wording is needed for other vocabularies
<huwshimi> wallyworld: Sure.
<StevenK> huwshimi: But not pink with purple polka dots? :-)
<wallyworld> huwshimi: i was told not to make the link blue, so that would need to be a discussion point :-)
<huwshimi> StevenK: Only under a feature flag just for you.
<StevenK> I don't know about that :-)
<StevenK> But nice comeback :-P
<wallyworld> what's wrong with pink and purple?
<huwshimi> wallyworld: I think it does need to be blue because it _does_ take you to a new page
<huwshimi> wallyworld: It just doesn't change the url of the current page
<wallyworld> huwshimi: but it *doesn't* trash the current page
<huwshimi> wallyworld: Unless you have your browser set up to not open new window links in new windows
<wallyworld> huwshimi: yes, but even if you do (why anyone would nowadays with tabbed browsing is beyond me), your current page is still there
<huwshimi> wallyworld: And I think that breaks the "do something to the current page" which is what the green links are supposed to represent
<wallyworld> huwshimi: that was my argument initially too.
<StevenK> wallyworld: Er. So, we still support IE6. And IE6 has no tabs.
<wallyworld> StevenK: and why anyone still uses IE6...... (i know, i know brain dead corporates)
<wallyworld> huwshimi: anyways, it's just a css attribute. i'm sure we will do whatever the stakeholders say we need to :-)
<huwshimi> wallyworld: I think it needs a wording change too
<wallyworld> ok
<wallyworld> huwshimi: so you watching SOO tonight? I know StevenK and wgrant aren't man enough to enjoy it :-P
<StevenK> Meh, SoO
<StevenK> wallyworld: http://www.theie6countdown.com/default.aspx is telling, but I suspect you've seen it.
<huwshimi> wallyworld: I'm catching up with some friends and then probably catch the end of it... if seeing friends doesn't mean watching SOO with them
 * wallyworld looks
<StevenK> Putting the banner on would make me happy, but only if it links to getfirefox or so
<wallyworld> StevenK: wow, china and india surprise me
<StevenK> wallyworld: And South Korea
<wallyworld> StevenK: that's because south koreas has a propriatary encryption solution for online banking that *requires* IE6 and is incompatible with every standard
<wallyworld> AFAIUI
<wgrant> Most South Korean bank sites use ActiveX, yeah :/
<StevenK> Then I'd expect a higher percentage, TBH
<wallyworld> or maybe it's just *requires* IEx
<StevenK> wallyworld: If you want to know the funniest thing -- that website was written by the IE team themselves
<wallyworld> well, i think even they realise now what they did :-)
<LPCIBot> Project windmill-devel build #225: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/225/
<StevenK> IE6 was during Microsoft 3-E approach to the Internet, wasn't it?
<StevenK> (Embrace. Extend. Extingush.)
<wallyworld> yep
<wallyworld> i think of it as s/Extinguish/Exterminate (said with a Darlec accent)
<lifeless> SoO ?
<wallyworld> State Of Origin
<wallyworld> QLD vs NSW
<wallyworld> the ultimate battle :-)
<StevenK> lifeless lived in NSW for a number of years, he probably got quite good at filtering it out.
<huwshimi> wallyworld: Well not really. It would have to be a little more evenly matched to be the ultimate battle.
<wallyworld> :-D
<StevenK> And not be based on thugby
<wallyworld> i can watch NSW get beaten anytime. I don't care if it's even or not :-)
<huwshimi> wallyworld: haha
<wallyworld> what's wrong with Rugby League|Union
<wallyworld> does StevenK prefer watching soccer? or netball? perhaps synchronised swimming?
<lifeless> wallyworld: oh, league. Not union :>
<wallyworld> lifeless: yes, SOO is League
<StevenK> wallyworld: AFL
<mwhudson> it's even more slabs of beef running into each other than most league afaict
<wallyworld> ah, the game of tight shorts
<wallyworld> mwhudson: you mean AFL?
<mwhudson> wallyworld: state of origin
<wallyworld> mwhudson: ah yes, the intensity is pretty high. and they hit *hard*
<wallyworld> lifeless: i may have found a bug with the bug subscription portlet - specifically the team membership check. well it's an issue in dev, but not production perhaps due to the really small batch size in dev
<wallyworld> the client makes an ajx call to a team's member_details property via the web service
<wallyworld> but this only returns the first batch of members
<wallyworld> and so the membership check may erroneously fail
<wallyworld> if the person being checked is not in the first batch of results
<wallyworld> that's what's happening on my dev setup anyway
<LPCIBot> Project windmill-devel build #226: STILL FAILING in 39 min: https://lpci.wedontsleep.org/job/windmill-devel/226/
<wallyworld> the membership check is done in javascript by loading the team members. not sure why it was implemented that was instead of asking the server "is this person a member". it's still an RPC either way
<wallyworld> has something changed recently to limit gets to web service CollectionField attributes to return only one batch at a time? or is this a long standing issue that we have avoided hitting in prod?
<wgrant> wallyworld: Don't go near that code.
<wgrant> wallyworld: Yellow is still rewriting it.
<wgrant> I fixed it myself a couple of weeks ago, but set the branch aside once Danilo informed me that it was being completely reworked.
<wallyworld> shit. i've based the blueprint subscription stuff on it. that's where i've found the bug
<wgrant> wallyworld: CollectionFields have always been batched.
<wallyworld> wgrant: so the code was broekn to begin with :-(
<wgrant> Of course.
<wgrant> That code is nauseating and buggy.
<wallyworld> bollocks
<wgrant> eg. you can't AJAX-subscribe someone with the same displayname as an existing subscriber.
<wallyworld> my version is very much trimmed down to the bare essentials since there's no direct vs indirect subscriptions etc
<wgrant> And it spends longer working out where to place the spinner (three HTTP requests, IIRC) than it does between showing the spinner and finishing the operation (one HTTP request).
<wallyworld> yes, all true :-(
<lifeless> Gary and I both had doubts about that spinner stuff
<lifeless> if its the one I think it is ;)
<wallyworld> i didn't realise the code i was reusing was so new :-(
<wgrant> wallyworld: Hm, it's not new.
<wgrant> wallyworld: It's just yet to be reworked as part of the... completed and released feature.
<wallyworld> well, i may just clean up this issue in my new blueprints stuff then and revisit it later when bugs is done
<wallyworld> it's also impossible to fully test without resorting to windmill
<LPCIBot> Project windmill-db-devel build #391: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/391/
<StevenK> Windmill, you make me sad.
<lifeless> gmb: yo
<lifeless> gmb: in https://code.launchpad.net/~gmb/launchpad/bug-61531/+merge/63672 I had a question for you; can has answer?
<lifeless> wallyworld: is https://code.launchpad.net/~wallyworld/launchpad/blueprint-subscriptions-tales-refactor/+merge/63949 or needs-another-review ?
<poolie> lifeless, hi
<poolie> thanks for resolving those mps, that's so great
<poolie> \heart
<lifeless> np
<poolie> i don't know if you saw the conversation on bzr recently
<poolie> but i think wip mps are a graveyard
<poolie> you might as well just reject community contributions than put them there
<lifeless> I saw it
<lifeless> I don't agree
<lifeless> :)
<poolie> therefore, specifically, i would flip nigelb's mp back to needsreview
<poolie> ok :)
<lifeless> that said, even if I ungreed unreservedly, we are not yet in a patch-pilotable position
<lifeless> so for now, I would still move stuff to WIP, to get a clean slate so we can get a drive-to-zero thats approachable, and bring up a patch pilot model.
<wgrant> Hmm, I've always seen WIP as coming before Needs Review. And it's something that I think should be toggled by the owner.
<lifeless> wgrant: did you see the discussion poolie is referencing ?
<wgrant> No.
<poolie> https://lists.ubuntu.com/archives/bazaar/2011q2/073028.html
<lifeless> so for bzr, with a patch pilot, WIP conflates 'can be piloted' and 'someone is hacking on this'
<poolie> yes
<poolie> ideally lp would separate these
<lifeless> with apilot working through proposals that were nearly-there this conflation is arguably a problem
<poolie> by, hypothetically, having assignment on mps
<poolie> yes
<wgrant> lifeless: Regarding launchpad-reviewers' contact address, isn't that necessary to stop all review mail from going to everyone (since it's the default reviewer, and reviewers always get mailed)?
<lifeless> I say that I don't agree; its more nuanced than disagreement, but not something I want to drill into today.
<poolie> it's ok
<lifeless> wgrant: I don't believe so
<wgrant> lifeless: What stops the mail?
<poolie> i'm confident enough that it's worth trying in bzr, but not so much i'd advocate it to every one else yet
<lifeless> wgrant: not being subscribed. IMBW
<poolie> we can see if it changes things
<wgrant> --
<wgrant> https://code.launchpad.net/~cr3/launchpad/hwsubmissionset_search/+merge/63768
<lifeless> wgrant: we can see how bzr's experiment turns out before changing anything in LP; I believe it is done.
<wgrant> Your team Canonical Launchpad Engineering is requested to review the proposed merge of lp:~cr3/launchpad/hwsubmissionset_search into lp:launchpad/db-devel.
<lifeless> wgrant: anyhow, we need to figure something out. the backscatter is pissing folk off ;)
<wgrant> lifeless: Yes. We should stop working around our bugs :/
<poolie> which one is done?
<poolie> fwiw i think removing that list along the lines you describe makes sense
<lifeless> poolie: the bazaar codereviewers thing
<lifeless> poolie: wgrant is saying that you may still find all new MP's trigger mail to ~bzr
<lifeless> darn, where is that bug about pushing-to-make-releases gone
<poolie> it would be hard for me to tell
<poolie> that change has been made, anyhow
<lifeless> wgrant: I had the wrong team
<lifeless> wgrant: its launchpad-canonical-reviewers I want to contact-ectomy
<wgrant> Oh, those are separate lists.
<wgrant> "WTF kill it kill it" is the answer that comes to mind.
<wgrant> lifeless: I no longer think you are crazy :)
<lifeless> wgrant: I'm so glad ;)
<poolie> just curious why you say "not yet in a patch-pilotable position"
<lifeless> https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> just the size of the queue vs the time allotted to reviews
<LPCIBot> Project windmill-db-devel build #392: STILL FAILING in 14 min: https://lpci.wedontsleep.org/job/windmill-db-devel/392/
<StevenK> And another redirect.
<StevenK> bzrlib.errors.RedirectRequested: http://bazaar.launchpad.net/~launchpad-pqm/bzr-builder/trunk/.bzr/repository/indices/aa01e422f8699f10641354d6fa588e29.cix is redirected to https://launchpad.net
<lifeless> I think there is a bug on that
<lifeless> it *should* 404
<lifeless> someone needs to tweak the apache rules
<lifeless> stub: hiya
<stub> lifeless: Hi
<stub> Just looking at 0mq
<lifeless> stub: I had a question for you in https://code.launchpad.net/~cr3/launchpad/hwsubmissionset_search/+merge/63768
<lifeless> stub: nice
<lifeless> stub: http://zguide.zeromq.org/page:all is a pretty good read about it
<stub> lifeless: I mentioned in my comment that we could apply the patch live. I didn't want to complicate Marc's life though by splitting patches out of his branch and stuff unless necessary.
<stub> lifeless: Although now I've decided to start giving out -1 style patch numbers even if we aren't planning on applying them live, just to make it easier if we change our minds.
<lifeless> stub: Right, I'm not suggesting splitting
<lifeless> stub: merely to use a -1 :)
<stub> Right. I guess it doesn't matter which branch they land on, just as long as they land and don't get lost.
<lifeless> stub: I heartily endorse your decision
<stub> I'll add a comment and request a renumbering
<stub> lifeless: So on initial skim, 0mq might be suitable for an RPC mechanism, but not suitable without effort for our async jobs as we would need to build a queue with it.
<stub> lifeless: Kestrel sounded nice. You looked at that yet?
<lifeless> stub: huh, It queues
<lifeless> stub: it does spill to disk
<lifeless> stub: high water thresholds, pub-sub etc
<lifeless> stub: I'm not saying we should use it, but a quick skim won't touch its facilities enough to assess
<lifeless> I need to reply to gavin
<lifeless> yes, I've looked at kestrel
<lifeless> and its also very interesting
<lifeless> simple api
<stub> lifeless: I thought it queued at the receiver or client, which assumes the receiver is up, and the client doesn't go down.
<stub> lifeless: So for the 'fire and forget reliable messages' story, we would need a persistent queue in the middle.
<stub> (But maybe that is trivial with the spill-to-disk behaviour)
<lifeless> stub: right, I haven't finished getting to grips with it
<lifeless> elliot has got me in contact with some rabbit staff
<lifeless> to pull on rabbits ha story
<lifeless> and gavin has landed a new stab at the test fixture
<stub> I'll still vote for an API for implementation agnostic messaging so we can run with whatever-works-with-the-testsuite first for potentially unreliable delivery and move up from there.
<lifeless> I'm not objection to that, but there are risks
<lifeless> they centre around the definition of agnostic messaging :)
<stub> getUtility(IMessageQueue, 'logging').send(python_object)
<lifeless> if its that minimal its low risk; getting into priorities, persistence, etc - thats where I was worrying :)
<stub> Right. Stuff priorities for now, and persistence is hidden by 'implementation agnostic' and irrelevant anyway because of 'potentially unreliable'
<rvba> lifeless: Hi ... Thanks for the review!
<lifeless> and, we're done: approximately all reviews done
<lifeless> https://code.launchpad.net/launchpad-project/+activereviews
<rvba> wgrant: Hi William, what did you think about my replies/code changes for https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676 ?
<LPCIBot> Project parallel-test build #39: STILL FAILING in 1 hr 18 min: https://lpci.wedontsleep.org/job/parallel-test/39/
<wgrant> rvba: Looking now.
<wgrant> rvba: Re-reviewed. Much better, but there are still a few issues.
<rvba> wgrant: Thanks a lot, I'll address your issues today.
<wgrant> rvba: Thanks!
<wgrant> Is blog.launchpad.net meant to feature stuff that is only relevant to Launchpad developers?
<lifeless> sure
<wgrant> Hmm.
<lifeless> everyone can be an lp dev :)
<wgrant> Why?
<lifeless> more seriously
<lifeless> stuff for the front page should be more carefully selected
<lifeless> most stuff for devs is at least interesting for highly technical folk
<wgrant> We could perhaps put them in another category.
<lifeless> and most LP-as-host-decision-makers are highly technical
<wgrant> "Initializing page JavaScript from the JSONCache" would be nice on a LP planet or something.
<wgrant> But it's not useful for users.
<wgrant> So it shouldn't drown out other stuff on the blog, IMO.
<lifeless> mrevell would be a good person to talk to about this
<lifeless> my understanding is that its meant to just get communication out there
<lifeless> breaking the code of silence we got into for a few years ;)
<wgrant> This sounds like blogging every day for the sake of blogging.
<lifeless> I don't think thats very useful
<lifeless> OTOH we have 20ish folk
<wgrant> It was a serious proposal two months ago.
<lifeless> I think we have interesting things to say; one per person per month at least
<LPCIBot> Project devel build #806: FAILURE in 6 hr 3 min: https://lpci.wedontsleep.org/job/devel/806/
<adeuring> good morning
<bigjools> morning all
<lifeless> hi bigjools
<bigjools> wotcher
<lifeless> bigjools: is there anything you'd like from me at the moment ?
<bigjools> all your base
<lifeless> belong to me!
<bigjools> you can fix all my bugs so I can just go to the beach?
<lifeless> do I look like miracle max ?
<lifeless>  :)
<bigjools> also, it's very disconcerting to find a load of earwigs in my keyboard today
<lifeless> ugh
<lifeless> seen khan recently ?
<bigjools> heh
<lifeless> 8 days to my new desktop. shiiiiny
<bigjools> I shall feed them to the 5cm-across spider who's made a nest just above me
<wgrant> bigjools: Only 5cm?
<nigelb> lifeless: Its on my list, just that life took a bigger priority :)
<nigelb> I'll make the change today and request merge again
<bigjools> wgrant: yes, it's a small one
<lifeless> stub: my ec2 land failed to start up for https://code.launchpad.net/~stub/launchpad/bugsummary/+merge/64583
<lifeless> stub: so you probably want to do that yourself
<LPCIBot> Project windmill-devel build #227: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/227/
<cjwatson> Could somebody be reviewer for https://code.launchpad.net/~cjwatson/launchpad/germinate-lubuntu?  Fairly trivial change ...
<bigjools> cjwatson: you need a merge proposal
<cjwatson> yup, creating now
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/germinate-lubuntu/+merge/64650
<wgrant> cjwatson: Approved. Want it landed now?
<cjwatson> that would be good, thanks
<cjwatson> I can't start building CDs right away because I'm blocked on disk space, but that's supposed to get sorted out this week
<wgrant> cjwatson: cocoplum updates still require downtime.
<cjwatson> when's the next one due?
<wgrant> Probably 3 weeks.
<wgrant> The last was a week ago today.
<wgrant> But we can do cocoplum separately in a few minutes of poppy downtime.
<cjwatson> Hmm.  Lubuntu wants to be building CDs in time for Alpha 2; three weeks would blow that, I think
<cjwatson> (my fault for not getting this trivial tweak in earlier)
<wgrant> July 13 is the next scheduled downtime. I'll get a downtime window for cocoplum for probably Fri/Mon and deploy then.
<stub> lifeless: k
<wgrant> cjwatson: It has landed (r13238)
<lifeless> stub: are my estimates for getting the (disable access, do a single schema patch, restore db access) realistic (I'm thinking if we go down -fast- [e.g. pg_cancel_backend all backends in parallel and don't restart pg itself], we should be down to sub-minute of overhead
<stub> lifeless: So one thing I realized - it takes about 7 seconds to startup one of our scripts. So overhead due to zcml cruft is 21 seconds to start with.
<lifeless> stub: one of the admin scripts?
<stub> lifeless: update.py, fti.py, security.py
<lifeless> stub: lets kick zcml out of them
<stub> Yes, so we need to restructure that stuff.
<stub> Otherwise I think we are fine for subminute, assuming negligable time to run the actual DDL.
<rvba> wgrant: Just so you know, I've replied to your comments on https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676.
<rvba> (I'm a little bit pushy with this because I have two branches that depends on this one ... but let me say that I really appreciate your detailed reviews on this rather complicated (at least to me) matter).
<cjwatson> wgrant: great, thanks
<wgrant> rvba: Thanks, I've replied.
<rvba> wgrant: Thanks.
<lifeless> still, 21 seconds in the first round is tolerable, it still leaves 4 minutes for the schema + disable + enable + kill backends
<henninge> when did we remove Blog information from a user's profile page?
<lifeless> stub: perhaps we should make up a tag for things to make this better, and/or things that we need to get started
<wgrant> henninge: It was never there, AFAICR...
<wgrant> henninge: Wiki information was removed a week or two ago.
<henninge> wgrant: maybe that is what I remember.
<henninge> I am aware that it can easily be included in the description.
<stub> lifeless: Sounds sane.
<rvba> wgrant: About the 1-based counter: the only problem I see is that the db column defaults to 1.
<lifeless> gmb: oh hai
<henninge> wgrant: was spam the reason to remove it?
<gmb> lifeless: Hi.
<wgrant> henninge: No. Bug #186660
<_mup_> Bug #186660: Launchpad shouldn't store wiki names <lp-registry> <qa-ok> <users> <Launchpad itself:Fix Committed by bac> < https://launchpad.net/bugs/186660 >
<rvba> wgrant: But of course, if I explicitly set it, I can make it start with 0 and everything is good (apart for the sql comments that says the counter starts at 1 :( )
<henninge> wgrant: thanks, I wonder why it is not "Fixed Released", though.
<wgrant> henninge: I have avoided closing it because they are not entirely gone.
<rvba> wgrant: My only argument is that to fix that properly, a db-patch will be needed.
<wgrant> henninge: They are merely hidden from the UI.
<wgrant> I hope bac will garden it soon :)
<henninge> ok
<bigjools> lifeless: something I'd like to discuss is script startup time
<henninge> Could somebody with bugs foo please have a look at this question? I don't think I understand what the problem is.
<lifeless> gmb: yeah, that branch - do you agree with my point about unintention explicit subscriptions
<henninge> https://answers.launchpad.net/launchpad/+question/161503
<lifeless> bigjools: I'd love to discuss that; not right now (TL meeting tomorrow)
<bigjools> lifeless: ok.  AIUI it's to do with zcml parsing but I think we can fix that
<gmb> lifeless: You mean "is it a no-op if you're subscribed via a team?" Yes - good call. I'm updating it now to make sure that being in a subscribed team means you won't be subscribed directly.
<lifeless> gmb: ah, just saw your mail
<lifeless> gmb: cool
<lifeless> bigjools: feel free to fix it ;)
<bigjools> lifeless: well it's not trivial :)
<bigjools> but the basic problem is that why should a script need to worry about browser zcml for example
* danilos changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer:  jtv, danilos | Critical bugs: 202 - 0:[######=_]:256
<danilos> jtv, hi, do you think you'd have time to review https://code.launchpad.net/~danilo/launchpad/bug-772754-other-subscribers-actions/+merge/64187 (it's slightly over-sized, but mostly due to JS tests)
<bigjools> lifeless: still around?
<lifeless> bigjools: barely; whats up?
<bigjools> I'm looking at this rabbit test failure
<bigjools>         self.useFixture(EnvironmentVariableFixture('HOME', '/nonsense/value'))
<bigjools> wtf?
<LPCIBot> Project windmill-devel build #228: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/228/
<bigjools> given that the test is failing trying to write cookies, that looks very suspect :)
<lifeless> bigjools: see the 3 line comment before that
<lifeless> bigjools: the fixture is meant to self isolate and export HOME itself whenever it runs erlang stuff
<lifeless> bigjools: this makes it fail-fast if the fixture doesn't do that.
<lifeless> bigjools: the test fails intermittently, so that isn't the issue.
<bigjools> ah ok
<bigjools> the comment didn't say that
<lifeless> bigjools: I added that in after I found that the soyuz test fixtures chdired
<bigjools> yay for soyuz
<lifeless> nn
<bigjools> thanks lifeless, nn
<lifeless> uhm
<lifeless> I suggest getting it to fail locally :)
<lifeless> or getting more data on what goes on when it fails.
<lifeless> the current code *can* pass, in a full test run. It can also fail in a full test run : so its not trivially broken or trivially correct.
<lifeless> elliot has put me in touch with vmware
<lifeless> I will be dropping one of their guys a brain dump tomorrow
<lifeless> bigjools: I'm happy that you're looking at it, just noting we have some backup coming in.
<lifeless> anyhow, *nn* for reals, I'm hosting tomorrow, so can't sleep through it :>
<bigjools> lifeless: yeah well I'm not looking too hard, just curious
<bigjools> lifeless: that's ok we'll sleep through it too
<bigjools> does our test runner always run tests in the same order?
<danilos> bigjools, my experience is that it does
<bigjools> ok ta
<bigjools> allenap: so in the "daemon" function there's a few print statements that are not in the test output
<bigjools> some of them trying to print $HOME
<allenap> bigjools: I'm looking but I have to go now.
<LPCIBot> Project windmill-devel build #229: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/229/
<adeuring> bigjools: do you have time to talk about bug 793630?
<_mup_> Bug #793630: better cronscript setup: remove hard-coded paths from cronscripts/publishing/cron.base-ppa <Launchpad itself:Triaged> < https://launchpad.net/bugs/793630 >
<bigjools> adeuring: not right now I am heading to lunch, later is ok
<adeuring> bigjools: ok; ping me when you have time
<bigjools> ok
<danilos> jtv, I am going to suppose you are off already since you haven't responded to my review request ;) cheers
* danilos changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: danilos | Critical bugs: 202 - 0:[######=_]:256
<Ursinha> good morning launchpadders
<danilos> Ursinha, good morning :)
<cr3> interesting, some site proxying launchpad but adding its own javascript is being indexed by google: http://www.anticensure.com/?__new_url=aHR0cHM6Ly9idWdzLmxhdW5jaHBhZC5uZXQvbHAtdXBzdHJlYW0tdG9vbHMvK2J1Zy8zMzQ0NTY=
<benji> they could use a robots.txt file
<dobey> nice; a big red box covering everything up
* abentley changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: danilos, abentley | Critical bugs: 202 - 0:[######=_]:256
<jcsackett> sinzui: up for mumbling?
 * sinzui tries
<LPCIBot> Project windmill-devel build #230: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/230/
<jcsackett> sinzui: is sip working better for you?
<danilos> mrevell, hi
<danilos> mrevell, how's it going?
<mrevell> Hello danilos
<mrevell> I'm okay thanks, how are you danilos?
<danilos> mrevell, pretty good as well, thanks :)
<mrevell> That is excellent news.
<danilos> mrevell, that's all I needed from you, thank you :P
<danilos> mrevell, but then again, do you have a minute or two to discuss one nice little branch I've got going
<mrevell> Yes, certainly :) Some kind of phone call, or is IRC okay ?
<danilos> mrevell, phone call is probably easier for start
<danilos> mrevell, you can pick mumble or skype :)
<danilos> mrevell, this is about bug 772754, fwiw
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-ok> <story-better-bug-notification> <Launchpad itself:In Progress by gary> < https://launchpad.net/bugs/772754 >
<mrevell> I'll strike a blow for freedom and go with Mumble. Just a sec.
<mrevell> danilos, What channel are you in?
<danilos> mrevell, yellow, but it seems mumble doesn't work for me
<mrevell> Skype it is.
<danilos> mrevell, yeah :/
<LPCIBot> Project devel build #807: STILL FAILING in 6 hr 0 min: https://lpci.wedontsleep.org/job/devel/807/
<jml> sinzui: ping
<sinzui> hi jml
<jml> sinzui: I'm otp atm, but would you be able to have a call w/ me sometime in the next hour and a half?
<sinzui> yes
<jml> sinzui: cool. ok if I ping you when I'm ready?
<sinzui> yep
<jml> Ursinha: https://dev.launchpad.net/Projects/Disclosure
<jml> bah
<cr3> hi folks, I would appreciate if someone could land the branch that was just approved in this merge request: https://code.launchpad.net/~cr3/launchpad/hwsubmissionset_search/+merge/63768
<jml> sinzui: https://dev.launchpad.net/PolicyAndProcess/FeatureDevelopmentCheckpoint
<bac> hi bigjools -- care to chat/pre-imp about bug 776437
<_mup_> Bug #776437: Enable ARM builders for PPA via API <api> <escalated> <not-pie-critical> <oem-services> <ppa> <Launchpad itself:Triaged> < https://launchpad.net/bugs/776437 >
<bigjools> ok
<bac> bigjools: so are you ok with lifeless' suggestion, which is basically to do the thing your first comment said not to do ?  :)
<bigjools> heh
 * bac needs to review the lp.commercial explosion
<bigjools> given the fact they're desperate for this, JFDI
<bigjools> lp.commercial is a pox
 * bac hopes no one 'bzr blames' that one
<bigjools> ;)
<bac> well, i created it...i didn't over use it.
<bigjools> well I think it's fine for commercial stuff
<bigjools> but to use it for anything else is simply crazy
<bac> so the basics for this bug should be straightforward, i assume
<bigjools> let me see
<bigjools> bac: see enabled_restricted_families in the model
<bigjools> expose it on the API.  It already has a setter/getter
<bigjools> and ensure the usual privileges
<bac> bigjools: rt
<timrc> I keep running into problems... when I type 'make run' now I get the following error: http://paste.ubuntu.com/627411/
<timrc> has anyone seen this?  I think it started when I started running launchpad out of multiple branches on the same system
<jml> hmm.
<jml> timrc: is there another Launchpad instance running?
<timrc> jml, no, and I've rebooted the VM
<jml> timrc: I haven't encountered the problem before, so I'm winging it... does /var/tmp/mailman/data/master-qrunner.pid exist? what about /var/tmp/mailman/data/?
<timrc> jml, the directory does indeed exist, but not the pid file
<jml> ok, so first up that's a lousy error from get_pid, probably bug report worthy
<jml> timrc: I guess I don't know why 'make run' is trying to stop mailman.
<flacoste> bigjools: i'm trying to fix bug 797088
<_mup_> Bug #797088: Launchpad has removed privileges that UDD importer requires to function <regression> <udd> <Launchpad itself:In Progress> <Ubuntu Distributed Development:New> < https://launchpad.net/bugs/797088 >
<flacoste> bigjools: and my simple tests isn't reproducing the problem
<flacoste> the security uses checkUpload
<flacoste> security checker
<flacoste> you can see it in security.py:2600
<bigjools> flacoste: is this the updates pocket that you need to pass?
<flacoste> bigjools: couldn't I just use verifyUpload() and not bother with the pockets check in that case?
<flacoste> since i think that's the difference between verifyUpload and checkUpload, no?
<flacoste> which is very confusing to be honest
<bigjools> upload permissions are confusing generally
<bigjools> so yes use verifyUpload
<jml> flacoste: sorry about the names. I  was running out of ideas.
<jml> timrc: oh sorry, I got distracted. I'm not sure how to help further though :(
<timrc> jml, I think mailman is a red herring... the exception is actually occurring w/ /var/tmp/development-librarian.pid
<bigjools> flacoste: checkUpload is the same as verifyUpload but with the pocket check
<flacoste> i know
<jml> timrc: ahh
<timrc> the file exists, but it's empty, which is why we get the ValueError and not the IOError
<flacoste> bigjools: after reading the implementation code :-)
<bigjools> flacoste: shit, we can READ!
<jml> timrc: man that's a lousy error
<jml> flacoste: fix the docs! change the names! storm the castle!
<bigjools> man, the damn air force is dogfighting over my house again
<flacoste> bigjools: time to move ;-)
<timrc> jml, I'll file the bug on improving the exception to at least include the file name... is there a make command that removes all the pid files?
<flacoste> jml: but why two different checks in the first place? why not one with pocket=None for when you don't care about the pocket? or does the client should always care about the pocket
<bigjools> flacoste: good idea
<timrc> (not that removing the file solved my problem,  argh)
<jml> timrc: not that I know of.
<bigjools> flacoste: it's a conflict of requirements
<jml> flacoste: can't recall, sorry. maybe I was trying to minimize the size of the change
<bigjools> some stuff cares deeply about the pocket, other stuff doesn't give a rats
<bigjools> and I suspect it's a half-done refactoring
<flacoste> 76 references to checkUpload
<flacoste> 8 to verifyUpload
<flacoste> so I think yeah, it's a minimize changes kind of thing
<flacoste> would be better to only use checkUpload with pocket=None for when you want to skip the pocket check
<bigjools> checkUpload is more fopr the use of the upload processor
<bigjools> and package copier
<bigjools> nothing else is going to have a pocket
<flacoste> the only client of verifyUpload are bugnomination! and test_branch?
<flacoste> !!!
<LPCIBot> Project windmill-devel build #231: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/231/
<flacoste> and archive.checkUpload
<jml> flacoste: ahh, bugnomination
<jml> flacoste: yeah, I remember this.
<jml> flacoste: it was too hairy to change at the time, and I was already deep in yak shaving by giving soyuz an actual API for checking this stuff.
<bigjools> something in the bugs code is using SPPH directly too.  it needs smacking
<jml> bigjools: package branches have pockets.
<bigjools> oh boy
<jml> well, some of them do.
<bigjools> I can see the generalised suite work getting bigger and bigger :(
<jml> which sort of makes sense because writing to certain package branches is (will be?) equivalent to an upload
<bigjools> right
<jml> bigjools: I like to think of it as becoming more and more valuable
<bigjools> as in costing more? :)
<jml> no, that other thnig
<jml> benefit.
<bigjools> more work is benefit?
<flacoste> bigjools: translations is also referencing SPPH direclty
<bigjools> flacoste: RARGH
<flacoste> jml: how package branches are tied to pockets?
<bigjools> we need better APIs between components
<jml> flacoste: SeriesSourcePackageBranch
<timrc> so odd, make start_librarian and make stop_librarian seem to work just fine :/
<jml> flacoste: official branches are tied to a pocket
<jml> timrc: check to see if a librarian is running, kill them all, delete the pid files, try again.
<jml> bigjools: *sigh*
<bigjools> jml: tired? :)
<jml> bigjools: having a Suite object would be a good start toward having better APIs between components
<bigjools> totally
<bigjools> but I was referring to stuff in bugs and translations poking around with publishing records, that's just wrong
<jml> yeah. that's often because it's the only way provided by existing APIs to do a thing, and because sometimes it feels easier to use existing APIs than add new ones.
<bigjools> we need to better differentiate between internal and external APIs then
<matsubara> jml, lunch just arrived. could you give me a few to eat and then we can have our 1-on-1?
<jml> matsubara: sure.
<matsubara> thanks, brb
<jml> bigjools: that'd be nice
<bigjools> jml: services-orientated architecture! :)
<jml> bigjools: Ooh. I read about those in _Wired_
<jml> bigjools: I hear they're really cloudy.
<bigjools> lol
<flacoste> bigjools: are you -1 on getting rid of verifyUpload and allowing pocket=None to checkUpload for case where you only care about SourcePackage (and not the SuiteSourcePackage)?
<flacoste> if you are not, i can shave that yak on my branch
<bigjools> flacoste: I'd favour renaming the methods so that the confusion is removed.  I'm a bit wary of someone making a change to the upload processor and forgetting to pass pocket.
<bigjools> because it *is* required in that case or trouble will ensue
<bigjools> something like verifyIncomingUpload() and the other one could be personCanUpload()
<flacoste> bigjools: actually, currently if you pass pocket=None, it works :-)
<flacoste> iow, there is no check that pocket is actually a valid one :-)
<flacoste> so if the distro isn't in SUPPORTED or CURRENT, it's basically fine :-)
<flacoste> (return True)
<bigjools>  /o\
<bigjools> what a mess
<flacoste> there are 800 checkUpload reference
<flacoste> not all of them are to IArchive.checkUpload but still
<flacoste> i'm not shaving that yak today :-)
<bigjools> it's a hairy yak
<flacoste> i could be convinced of renaming verifyUpload
<flacoste> only 8 references
<bigjools> right
<flacoste> checkUploadWeDontCareAboutPocket?
<flacoste> would make it obviously silly
<bigjools> looking at it, that one is called from the upload processor as well.  Hooray.
<bigjools> so it needs to be doesPersonHaveUploadRights or similar I think
<flacoste> bigjools: indirectly though
<bigjools> naming is hard :)
<flacoste> bigjools: i don't see references to verifyUpload in lib/lp/archiveuploader
<bigjools> but you get my drift I hope
<bigjools> yeah it's indirect
<flacoste> through checkUpload
<flacoste> because checkUpload is verifyUpload + pocket check
<bigjools> verifyPersonCanUpload() maybe?
<flacoste> i really don't think two APIs here pull their weight to be honest
<bigjools> I like that one
<flacoste> checkUpload and verifyPersonCanUpload
<flacoste> not sure it makes a lot of sense
<bigjools> it would if you knew soyuz code more :)
<flacoste> especially when you grep
<flacoste> and see that everything uses checkUpload apart 3 call sites
<flacoste> actually 2
<flacoste> bugnomination
<bigjools> yeah those are the ones that just need to know that person is an uploader
<flacoste> and that new launchpad.Edit permission checker on sourcepackage
<timrc> jml, looks like I had to use a big stick approach and just whack everything in /var/tmp
<bigjools> which is why I suggested verifyPersonCanUpload
<flacoste> but that's also what checkUpload does
<bigjools> not exactly
<flacoste> verify that the person has upload permission
<flacoste> plus a pocket check
<flacoste> which is person independant
<bigjools> it checks that the person can upload that package at that time
<flacoste> the series status representing time here?
<bigjools> yes
<matsubara> jml, I'm back. voip?
<jml> matsubara: sure.
<jml> matsubara: 7608
<bigjools> flacoste: so for me these are 2 conceptually different things.  One is "is this person an uploader?", the other is "should we let this upload in?"
<bigjools> it's just that the parameter list is quite similar
<flacoste> bigjools: there is another API for should we let this upload in, it's the checkUpload(upload) method in the archiveuploader
<flacoste> which makes more sense as a signature
<flacoste> and why the heck are we always passing sourcepackage and suitesourcepackage as tuples?
<bigjools> flacoste: no, those are policies
<flacoste> (distroseries, sourcepackagename) or (distroseries, sourcepackagename, pocket)
<bigjools> policy checks are completely different beasts
<bigjools> they are one aspect of the overall check
<flacoste> wouldn't the policy check delegate to the archive.checkUpload at some point?
<bigjools> the other way around
<flacoste> i don't see any call to policy checks in archive.checkUpload
<bigjools> you were talking about the one in archiveuploader I thought?
<bigjools> the code is in a bit of flux at the moment since we're ripping out a lot of the policy stuff in archiveuploader and replacing it with a more general framework that works with derived distros syncing
<LPCIBot> Project parallel-test build #40: STILL FAILING in 1 hr 2 min: https://lpci.wedontsleep.org/job/parallel-test/40/
<bigjools> .checkUploadToPocket() is a policy check, it's just hard-coded instead of using the policy framework
<flacoste> ah
<flacoste> then that's the real fix
<flacoste> move checkUploadToPocket to a policy
<jml> '<bigjools> flacoste: so for me these are 2 conceptually different things.  One is "is this person an uploader?", the other is "should we let this upload in?"' â that's a really good way of putting it.
<flacoste> and at that point IArchive.checkUpload and IArchive.verifyUpload are identical
<flacoste> and we can merge them (and possibly rename it)
<flacoste> bigjools: does that sound sensible?
<flacoste> bigjools, jml: if any of you want to review https://code.launchpad.net/~flacoste/launchpad/bug-797088/+merge/64711
<bigjools> flacoste: yes but I think I'd prefer you left it TBH since I want to do more than that
<flacoste> bigjools: don't worry, i wasn't goind to tackle it
<bigjools> the work to make the uploader use the new policy framework is not trivial
<bigjools> flacoste: ok :)
<flacoste> just wanted to have a clear conscience by leaving this mess behind :-)
<flacoste> i saw the mess, didn't do anything about it
<flacoste> but it's because the professionals are coming in to clean it up ;-)
<bigjools> ha
<bigjools> it's one of the smallest messes in soyuz :)
<bigjools> flacoste: failUnless -> assertTrue
<flacoste> bigjools: ok, think positive ;-)
<bigjools> right :)
<flacoste> i can change that
<bigjools> I think we deprecated the failXXX methods
 * flacoste wasn't aware
<flacoste> and fixed
<bigjools> flacoste: r=me!
<flacoste> bigjools: thanks!
<jml> matsubara: https://dev.launchpad.net/Projects/DerivativeDistributions
<bigjools> jml: did you see this: https://dev.launchpad.net/Soyuz/DerivativeDistributions
<jml> bigjools: no,  I hadn't.
<bigjools> mostly written for devs but useful for techy users
<jml> cool. I'll look at that soon.
<bigjools> ah balls, call in an hour
<LPCIBot> Project db-devel build #637: FAILURE in 6 hr 1 min: https://lpci.wedontsleep.org/job/db-devel/637/
<LPCIBot> Project windmill-db-devel build #393: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-db-devel/393/
<timrc> lifeless, ping
<jml> timrc: he's out.
 * jml is off for the day.
<Ursinha> bye jml
<timrc> jml, thanks...
<jml> g'bye
<timrc> abentley, you there?
<abentley> timrc: hi.
<timrc> abentley, I'm trying to adapt the archive api to enable setting of the private attribute, this method seems to break a lot of unrelated tests... can you easily identify anything I may be doing wrong?  It's not a lot of code, http://paste.ubuntu.com/627484/
<timrc> the tests I'm failing are: http://paste.ubuntu.com/627486/ -- it's odd, I'm able to create ppas from launchpad.dev and I'm able to set them private via the api
<abentley> timrc: I'd expect Archive.private is used in composing a lot of Storm expressions, and a python property that's not a Storm column wouldn't work for that.
<timrc> abentley, how would it know any differently if I expose private as a getter / setter ?
<timrc> would think it would be completely transparent
<timrc> but I don't know much of anything about Storm, so... :/
<abentley> timrc: It's not at all transparent.  You're dealing with the class of the attribute, not the attribute itself in a Storm expression.
<timrc> abentley, let me ask you this, do you know of any other places in launchpad where we try to keep the internal and external interface to an attribute consistent with Python properties?  I originally implemented this with a mutator but lifeless suggested this other approach with the Python property
<abentley> timrc: e.g. IStore(Archive).find(Archive, Archive.private == True) is used to find all private archives.
<abentley> timrc: generally a property is better than a mutator, and a storm column is better than a property.
<abentley> timrc: I've used properties myself, but that was before I found out about storm_validator, which allows columns to be used in many places where properties would otherwise be required.
<abentley> timrc: e.g. Job.status used to be a property, but was recently changed to use a storm_validator instead.
<cody-somerville> Hey. I'm getting "ProgrammingError: text search configuration "default" does not exist" when my local dev instance attempts to execute sql. Anyone know what I need to do to fix this?
<abentley> cody-somerville: perhaps you need to run utilities/launchpad-database-setup
<abentley> cody-somerville: beware that this will trash any existing Postgres databases on your system.
<cody-somerville> abentley, Yea. It doens't look like there is anything new in there for me to run
<cody-somerville> it has to do with the text search stuff I think that lifeless did
<abentley> cody-somerville: And have you done "make schema"?
<cody-somerville> Yes.
<abentley> cody-somerville: Okay, I've never seen the error you describe, so I don't know what the problem is.
<abentley> generally, those two are enough to hard-reset your database.
<timrc> abentley, so to make this work for Storm expressions I'll need to use storm_validator?  Did you encounter this issue I'm having when you used properties?  Is there a work around?
<abentley> timrc: I didn't experience the issue you're having, because I originally implemented Job.status as a property, so there were no existing uses to break.
<timrc> abentley, ah hah
<abentley> I don't believe there's a work around.
<timrc> abentley, I think I'd need to basically rename private to _private
<abentley> I don't know whether it's appropriate for a validator to have side-effects.
<timrc> abentley, but that seems like a big change
<abentley> If it is, then a validator would be the right solution.  If not, then I'd go back to the mutator.
<timrc> There are 15 places that need to be updated, I'll try that, re-test and submit a merge proposal and see where that goes :)
<abentley> timrc: The motivation for changing Job.status from a property to a column was so that Job.status could be used in storm expressions, instead of Job._status.  It's certainly possible that a branch which introduced Archive._private into Storm expressions would be rejected.
<timrc> abentley, ok
<timrc> abentley, I will look at Job.status for how to use storm_validator
<jcsackett> sinzui: up to chat a bit?
<jcsackett> sinzui: time to chat about some package-picker stuff?
<sinzui>  I do.
<sinzui> is mumble working?
<jcsackett> i heard you.
<jcsackett> sinzui: i can hear you. it would appear you can't hear me.
<sinzui> sip?
<sinzui> I have empathy back up
<jcsackett> calling now.
<jcsackett> sinzui: bug 698020 bug 698022 and bug 698024
<sinzui> jcsackett, http://pastebin.ubuntu.com/627558/
<jcsackett> sinzui: disconnected. calling you again now.
<sinzui> okay
<matsubara> danilos, hi
<matsubara> danilos, could you review two small oops-tools branch for me?
<LPCIBot> Project devel build #808: STILL FAILING in 5 hr 57 min: https://lpci.wedontsleep.org/job/devel/808/
<cody-somerville> Does that mean trunk is broken ^^?
<lifeless> maybe
<matsubara> abentley, hi there, could you review two small oops-tools branch for me?
<abentley> matsubara: sure.
<matsubara> abentley, first one: https://code.launchpad.net/~matsubara/oops-tools/html-report-urls-broken/+merge/64731
<matsubara> abentley, second one: https://code.launchpad.net/~matsubara/oops-tools/update-deprecated-setttings/+merge/64739
<abentley> matsubara: Any reason not to use the URL generation stuff in Python?
<matsubara> abentley, how do you mean?
<abentley> matsubara: get_absolute_url uses string concatenation rather than actual url handling to generate its urls, which makes it more brittle.  Is there a reason why that's a good tradeoff?
<matsubara> abentley, just seemed like the easiest thing to do and it seems to be the recommended way in django docs: https://docs.djangoproject.com/en/dev/ref/models/instances/?from=olddocs#get-absolute-url
<abentley> matsubara: in that example, they're dealing with an int, so it's not string concatenation.
<matsubara> abentley, I didn't add the get_absolute_url method to the model as suggested in the docs because the dbsummary.py code doesn't use the Oops object itself. It has some optimization to fetch all necessary oopses using a single SQL query and the data structure returned doesn't have the oops object, just the oops id
<matsubara> abentley, or you mean I should be using urlparse.urljoin()?
<abentley> matsubara: Yes, but ideally you'd be escaping the oops-id.
<abentley> matsubara: e.g. with urllib.urlencode
<flacoste> what does it mean when i do ec2 land and after
<flacoste> Running tests... (output is available on http://ec2-50-19-153-15.compute-1.amazonaws.com/)
<flacoste> i get
<flacoste>      8kB     1kB/s /
<flacoste> like a stalled bzr progress bar
<flacoste> that's two run that are stalled like that
<bac> ec2 instance is having difficulty talking to LP?
<lifeless> yup
<flacoste> so i should just kill the run and try again later?
<lifeless> or let it continue
<flacoste> sinzui: do you recall why you mapped NoSuchSeries exception to 400 on the webservice (i would have expected 404)?
<sinzui> flacoste, typo?
<flacoste> https://code.launchpad.net/~sinzui/launchpad/api-additions/+merge/6329
<flacoste> well, it's spelled BAD_REQUEST
<flacoste> so that's a pretty big typo :-)
<mwhudson> jelmer: what's the story with lots of large bzr-svn imports failing after the last rollout?
<mwhudson> https://code.launchpad.net/~vcs-imports/gcc/trunk failing is bad news for linaro
<flacoste> "
<flacoste> When calling distribution.getSeries() with an invalid name or version I get
<flacoste> an "HTTP Error 500: Internal Server Error". The content of the exception
<flacoste> ('NoSuchDistroSeries') is correct about the fact that there is no such
<flacoste> distro series but I would have preferred if the method returns [400]
<flacoste> "
<jelmer> mwhudson: it's the new fetching of tags :-/
<mwhudson> jelmer: ah
<mwhudson> jelmer: is there a bug for it?
<flacoste> sinzui: if you can't recall the justification for this, i'll just file a new bug and change it to 404
<flacoste> https://bugs.launchpad.net/launchpad/+bug/358332
<_mup_> Bug #358332: [API] OOPS when distribution.getSeries() is called with an invalid name or version <api> <lp-registry> <oops> <story-series-milestones-releases> <Launchpad itself:Fix Released by sinzui> < https://launchpad.net/bugs/358332 >
<flacoste> seems that the user suggested 400
<flacoste> matsubara suggested NotFound, but you went ahead and took the user suggestion
<jelmer> mwhudson, not that I'm aware of, please file one
<jelmer> mwhudson, when does a user timeout happen on the import slaves exactly/
<jelmer> ?
<mwhudson> jelmer: when there is no progress reported for ... the value of some config variable ... seconds
<mwhudson> jelmer: i think it's an hour
<sinzui> flacoste, I cannot think of reason, so file a new bug
<jelmer> oh, wow
<mwhudson> jelmer: well, no output on stdout, and we have a special progress bar thing
<lifeless> flacoste: +1 on making it 404
 * flacoste gets the razor out
<mwhudson> jelmer: although, um: http://launchpadlibrarian.net/73372571/vcs-imports-gcc-trunk.log
<mwhudson> jelmer: yeah, it's an hour
<mwhudson> jelmer: https://bugs.launchpad.net/launchpad/+bug/797915
<_mup_> Bug #797915: large bzr-svn imports failing <code-import> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/797915 >
<LPCIBot> Project windmill-db-devel build #394: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-db-devel/394/
<LPCIBot> Project parallel-test build #41: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/parallel-test/41/
<jelmer> mwhudson: thanks
<mwhudson> jelmer: i've also seen imports crash with MemoryError -- is that possibly related?
* abentley changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 202 - 0:[######=_]:256
<jelmer> mwhudson: possibly - finding the tags requires more history analysis so possibly more memory usage
<lifeless> flacoste: I'm planning on splitting my day into two today; there is a sale on in the city for various things we need
<flacoste> lifeless: sure, no problem
<flacoste> lifeless: changing from 401 to 404 will break backward incompatible change
<flacoste> unless we versioned the exception error code, which i suspect we don't
<flacoste> do we care?
 * flacoste does not
<sinzui> wgrant, wallyworld_, StevenK. I am told I am picking up a child. I believe I will more than an hour late. I will make a showing when I return to see if I am needed.
<wallyworld_> sinzui: we can delay the call till you return?
<thumper> hi hackers
<wallyworld_> thumper: how's they hanging?
<mwhudson> jelmer: https://code.launchpad.net/~vcs-imports/gcc/trunk seems pretty stuck
<jelmer> mwhudson, I'm trying it locally, but it seems to be working here
<mwhudson> jelmer: oh good!
<jelmer> mwhudson, no :(
<mwhudson> yeah
<mwhudson> jelmer: tdb vs sqlite or something?
<mwhudson> jelmer: it doesn't seem to be using a lot of cpu https://pastebin.canonical.com/48542/
<jelmer> mwhudson: hmm
<jelmer> mwhudson, swapping a lot perhaps?
<mwhudson> yeah
<jelmer> mwhudson, the home directories are just on local disk right?
<mwhudson> jelmer: yeah
<mwhudson> maybe we should be scaling the concurrency down a bit
<mwhudson> jelmer: see #launchpad-ops for, well, ops stuff
<LPCIBot> Project db-devel build #638: STILL FAILING in 5 hr 55 min: https://lpci.wedontsleep.org/job/db-devel/638/
<thumper> wallyworld_: hanging mostly normally, you?
<wallyworld_> thumper: mired in javascript. plus kindle screen just gone bad :-(
<thumper> wallyworld_: what's the problem with your kindle?
<thumper> wallyworld_: Rachel suggests charging it
<wallyworld_> screen is all messed up. about 1/4 of the screen is all corrupt
<wallyworld_> displays garbage, random pixels
<thumper> wallyworld_: have you tried rebooting?
<wallyworld_> holding power button across for 15 seconds before releasing? yes. aything else?
 * thumper shrugs
<thumper> send it back?
<wallyworld_> bit hard since it was bought over the counter in dallas
<thumper> wallyworld_: but amazon still handle the warrenty
<wallyworld_> i don't think i can find the receipt :-(
<thumper> you shouldn't have to find it
<thumper> they have serial numbers
<wallyworld_> ah right. i'll email them and see what they say
 * wallyworld_ wonders if there's a place to stick a paperclip
#launchpad-dev 2011-06-16
<wallyworld_> thumper: just talked to amazon. they are sending out a replacement. they are charging me but will refund the money when i send back the broken one
<thumper> wallyworld_: awesome news
<wallyworld_> thumper:  yeah. they asked several times if it had been dropped or pressure put on the screen. but it lives in the amazon hard covered case....
<wgrant> So much QA :/
<wgrant> lifeless: Can we restore qastaging?
<lifeless> +1
<wgrant> jelmer: Hi.
<wgrant> jelmer: http://staging.launchpadlibrarian.net/73288606/vcs-imports-pydoctor-trunk.log -- I don't think cscvs likes the autoupgrade stuff.
<lifeless> I will be back later; today is in 2 parts :)
<jelmer> wgrant: darn
<jelmer> wgrant: should be a trivial change to clean it up after upgrade
<wgrant> jelmer: Rather.
<wgrant> jelmer: Trivial enough that it's worth fixing rather than rolling it back?
<jelmer> wgrant, https://code.launchpad.net/~jelmer/launchpad/auto-upgrade-qafix/+merge/64771
<wgrant> jelmer: Perfect, thanks.
<wgrant> I think we shall skip ec2 this time.
<StevenK> Hm. No wallyworld
<StevenK> I guess he did see the State of Origin, then ...
<jelmer> wgrant: are you landing it, or should I?
<wgrant> jelmer: I've approved it, and can land it if you want.
<wgrant> I realise it's late.
<jelmer> wgrant: if you can, that would be great
 * jelmer gets some sleep
<wgrant> jelmer: Thanks for the fix.
<wgrant> Night!
<poolie> flacoste: ping?
<92AADA0EO> StevenK: hi. did you ping me?
<92AADA0EO> stupid nick :-(
<StevenK> 92AADA0EO: Yes. To gloat.
<92AADA0EO> ns
<wallyworld_> StevenK: well, i bet you didn't even watch the game?
<StevenK> Right.
<wallyworld_> so nothing to gloat about :-P
<StevenK> But QLD lost, so that makes me happy.
<wallyworld_> but you don't care you said
<LPCIBot> Project devel build #809: STILL FAILING in 5 hr 51 min: https://lpci.wedontsleep.org/job/devel/809/
<LPCIBot> Project windmill-devel build #232: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/232/
<wgrant> wallyworld_: Hi.
<wallyworld_> hello
<wgrant> Have you seen bug #797820?
<_mup_> Bug #797820: searching for the last name makes still hard to find the person <disclosure> <exploratory-testing> <person-picker> <Launchpad itself:Triaged> < https://launchpad.net/bugs/797820 >
<wgrant> The name-prefix matches drown everything else out :/
<wallyworld_> no, not yet
<wgrant> Even after I reduced the numbers.
<wallyworld_> let me have a quick read
<wgrant> I think we may need to take them right down :(
<wallyworld_> wgrant: so that bug report implies there are lots of names starting with "lange"?
<wgrant> wallyworld_: It doesn't say that, but when you try it it's clear.
 * wallyworld_ tries it
<wgrant> https://pastebin.canonical.com/48548/ are the results of that search.
<wgrant> The second column is the rank.
<wgrant> jml's FTI match is 0.6, but the karma scales it up.
<wgrant> But not high enough.
<wgrant> (0.6 is about right for a single name match on a two-part name)
<wallyworld_> wgrant: i tried it on lp.net and yes there are lots of usernames starting with "lange". so i think we just de-prioritise name prefix matches
<wallyworld_> adjust the scaling factor
<wgrant> Adjust the scaling factor, or reduce the hardcoded ranks?
<wallyworld_> reduce the hard coded ranks
<wallyworld_> for name prefix
<wgrant> name prefix = 5, displayname prefix = 4, IRC nick match = 3, email prefix = 2
<wgrant> That's where we are now.
<wallyworld_> where is exact name match in that list?
<wgrant> Exact name match is always 100
<wgrant> There is nobody with username 'lange', so it doesn't show up here.
<wgrant> (yeah, having it hardcoded to 100 is bad, but it will work unless someone has 10^20 karma, in which case we are probably screwed anyway)
<wallyworld_> i don't like 5,4,3,2. i think ranks needs to be between 0,100 and sacling factor applied to fti to put it in the 0,100 range
<wgrant> Why 0-100 rather than 0-1?
<wallyworld_> so that we can make minor adjustments using whole number arithmatic
<wgrant> I wonder what happens if we just reduce them to a perfect FTI match... 1.
<wgrant> That would probably be good for full email address local part matches.
<wgrant> But not general prefix matches.
<wallyworld_> i think we need to discriminate between them though, not just make all = 1
<wallyworld_> hence the 0-100 range
<wgrant> Probably, but how?
<wallyworld_> conceptually it's easier to think about adjustments in that range
<wgrant> Not for me, but OK :)
<wallyworld_> i mean whole number arithmetic is easier than decimals between 0..1
<wallyworld_> so we scale the fti result by 100
<wgrant> Hardly :)
<wgrant> But this is not really crucial to the problem.
<wgrant> We don't know where to put these non-FTI matches.
<wgrant> That is the important bit.
<wallyworld_> sure, butno we have a 0-100 range to play in, we can experiment
<wallyworld_> but now
<wallyworld_> so start by reducing the weight of name prefix matches
<wallyworld_> but then again, if we scale fti by 100 maybe no adjustemtn needed
<wallyworld_> to the name prefix weighting if it's say 50
<wallyworld_> since 0.6 * 100 = 60 > 50
<wgrant> But is that what we want?
<wallyworld_> well, i think an exact match on lastname should be > name prefix match
<wallyworld_> too bad we can't weight based on how closely the name prefix matches
<wgrant> Indeed.
<wallyworld_> that would help a lot
<wallyworld_> but in any case, if someone types the full lastname and it matches, then surely that should place those results near the top?
<wallyworld_> above any name prefix matches
<wgrant> What if I'm not typing a full lastname, but a name prefix?
<wallyworld_> then you see that result below the display name matches :-)
<wallyworld_> i think it will be more common for people to type last name
<wallyworld_> if they search on lp id (name), then surely they will typr the whole id?
<wallyworld_> i would :-)
<wgrant> Possibly.
<wgrant> Perhaps we should remove the name prefix matching and see how it goes.
<wgrant> Not sure about displayname.
<wgrant> There are still 9 displayname match results above jml.
<wallyworld_> i just think we need to reduce the name prefix weight
<wgrant> But I wonder if FTI deals with them OK.
<wgrant> Yes, but reduce it to where?
<wallyworld_> wgrant: so imho, i think we should: scale fti by 100 and adjust the 5,4,3,2 to 50, 40, 35 or whatever
<wgrant> There are tonnes of FTI matches.
<wallyworld_> tonnes yes, but all with different rankings
<wgrant> Lots at the same rankings.
<wallyworld_> so if we scale correctly, they will fall somewhere in the range also occupied by those other matches
<wgrant> The visible non-jml FTI matches are 0.75990885..., and there are dozens of them.
<StevenK> What FTI match does jml get for 'lange'?
<wgrant> 0.6
<wgrant> 0.607927
<wallyworld_> winder why a full lastname match is lower than those other results
<wgrant> wallyworld_: lange stems to lang.
<wgrant> alexander-lang's displayname is Alexander Lang
<wgrant> So he gets a double lang match.
<LPCIBot> Project db-devel build #639: STILL FAILING in 5 hr 27 min: https://lpci.wedontsleep.org/job/db-devel/639/
<wgrant> launchpad_dogfood=> SELECT name, fti, rank(fti, ftq('lange')) FROM person WHERE name IN ('jml', 'alexander-lang');
<wgrant>       name      |                       fti                        |   rank
<wgrant> ----------------+--------------------------------------------------+----------
<wgrant>  jml            | 'jml':1A 'jonathan':2A 'lang':3A                 | 0.607927
<wgrant>  alexander-lang | 'alexand':2A,4A 'alexander-lang':1A 'lang':3A,5A | 0.759909
<wallyworld_> why does it chop off the last e?
<StevenK> So, in this case, I don't think jml has to be *first*, I think he has to be in the top 10
<wgrant> Stemming does not work well on names :)
<wgrant> StevenK: He should be first.
<wgrant> StevenK: He's the only one related to the project.
<StevenK> That's a point
<wallyworld_> wgrant: so perhaps we can do in memory affiliation sorting once the db query has been done
<wallyworld_> use the affiliation to fine tune the result
<wgrant> We can do that in the DB just as easily.
<wgrant> We have no choice.
<wgrant> We have to do it there.
<wgrant> Or we have to pull back EVERYTHING from the DB.
<wallyworld_> i didn't think the affiliation was easily queryable
<wallyworld_> and i mean we use affiliation to fine tune the results
<wgrant> "fine tune" is a rather broad term.
<wallyworld_> so we still limit the db query to 100
<wallyworld_> and then "fine tune" the final order of those 100 records, for whatever mean of "fine tune"
<wallyworld_> meaning
<wgrant> What if there are 500 0.7 FTI matches?
<wgrant> We pull back the top 100 sorted by displayname.
<wgrant> And bump the affiliated people.
<wallyworld_> well we limit the results to 100 now don't we?
<wgrant> But the person I wanted was named Zack.
<wgrant> So he's number 499.
<wgrant> We do. But we do affiliation in the query.
<wgrant> All ordering is done in the query.
<wgrant> And we take the top 100.
<wallyworld_> you mean karma = affiliation?
<wallyworld_> i haven't seen the latest query
<wgrant> For the purposes of search, yes.
<wgrant> ORDER BY rank * COALESCE( (SELECT LOG(karmavalue) FROM KarmaCache WHERE person = Person.id AND product = 10294 AND category IS NULL AND karmavalue > 10), 1) DESC, Person.displayname, Person.name LIMIT 100
<wallyworld_> i think that's perhaps a false assumption that karma == affiliation
<wgrant> Yes, but it was quick and it's pretty effective.
<wgrant> And it handles unofficial relationships well.
<wgrant> And it's logarithmic, so infrequent contributors aren't penalised too much.
<wgrant> And if you're searching for someone to assign them to a bugtask, that's probably been done before, so they have karma.
<wallyworld_> ok. so we have affiliation in the query. we "just" need to tweak the ranking factors :-)
<wallyworld_> but it will be hard
<wallyworld_> for every search we "fix", we will "break" another
<wgrant> Yes.
<wgrant> I question the utility of the prefix matches.
<wallyworld_> so i think so long as the intended person appears on the first page every time, that wil suffice
<wgrant> But perhaps there is a reason.
<wgrant> Yes.
<wgrant> The correct person will rarely be first.
<wgrant> Because exact name matches always win.
<wgrant> and I don't think that's negotiable.
<wallyworld_> yes
<wallyworld_> if we are unsure about prefix matches we just need to devalue them
<wallyworld_> but not exclude them
<wallyworld_> totally
<wgrant> Devaluation is pretty much exclusion, unless there are no FTI matches at all.
<wgrant> Mm.
<wgrant> Although I guess relevant prefix matches will be bumped up a bit.
<wallyworld_> surely bad fti matches will be < 0.3 or whatever
<wallyworld_> so prefix matches can be made to be higher than 0.5 or whatever
<wallyworld_> so that only close fti matches 'win"
<wgrant> "close"
<wallyworld_> :-)
<wgrant> We have to choose arbitrary constants :/
<wallyworld_> yep
<wallyworld_> unless we can run the data through some statistical analysis
<wgrant> s/statistical/magical/
<wallyworld_> well, if we had expected search terms and results, and a lot of those, then we could do something. but we don't
<wgrant> Even if we did, there will always be other cases.
<wallyworld_> outliers. sure. but that's unavoidable
<wgrant> What if the cases we test are the outliers?
<wgrant> :)
<wallyworld_> up to us to make sure they're not
<wallyworld_> i think there's wiggle room. desired person on first page seems reasonably doable hopefully. or very least, on page 2.
<wallyworld_> i'd like to be on page 3 personally :-)
<lifeless> wallyworld_: *blind*
<wallyworld_> lifeless: maybe we should do a naked geeks calendar for charity :-) i'll be Mr December.
<StevenK> And none of us will enjoy Christmas again
<lifeless> -ever-
 * StevenK high fives lifeless 
 * wallyworld_ lols
<wgrant> jtv: Around yet?
<jtv> far too round, thank you
<jtv> Had to go make some travel preps etc.
<wgrant> jtv: Ah, hi! Bug #797151 needs QA, when you have time.
<_mup_> Bug #797151: Display DSD copy errors differently from comments <derivation> <qa-needstesting> <Launchpad itself:Fix Committed by jtv> < https://launchpad.net/bugs/797151 >
<jtv> thanks for bringing it to my attention.  Won't take long.
<wgrant> mwhudson: How much effort is setting up importds for qastaging?
<wgrant> mwhudson: Is it just a matter of configuring the importd with a qastaging tree too?
<mwhudson> wgrant: um, probably
<mwhudson> the process is documented, would you believe
<wgrant> I don't appreciate your lies.
<mwhudson> wgrant: https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/SetUpCodeImportSlave
<wgrant> I don't appreciate lies with evidence, either.
<wgrant> But OK.
<mwhudson> most of those steps will have been done already for the staging importds, i don't know what modifications will be needed for qastaging
<wgrant> mwhudson: There's no master setup needed? It just does xmlrpc?
<mwhudson> wgrant: yeah
<wgrant> Great, thanks.
<mwhudson> the production importds connect to the database for oops-prune
<wgrant> Well, at the moment some of them don't.
<wgrant> And they cronspam about it :)
<spm> pear? it should now.
<wgrant> Indeed! It stopped a week ago.
<wgrant> This is good.
<spm> yah. just needed same rules to allow those connections thru and we were gold.
<spm> some*
<jtv> Anyone mind if I sync some packages on staging's oneiric?  wgrant, StevenK?
<lifeless> wgrant: do me a favour?
<jtv> Oh darn, can't do it on staging â not privileged
<lifeless> wgrant: file bugs for any db access you run across that isn't from an appserver, db-stats or security/py/upgrade.py/fti.py.
<StevenK> lifeless: The publisher? :-)
<wgrant> lifeless: Um, you know we have dozens of scripts, right? :)
<wgrant> Like StevenK said.
<StevenK> loganberry
<StevenK> process-accepted, queue, change-overrides, process-death-row ...
<wgrant> jtv: StevenK or I can do it there, but I don't think the flag is enabled.
<jtv> ahhh
<jtv> nm I'll stick with df
<jtv> wgrant, StevenK: mind if I restart the app server on df?
<StevenK> Go ahead
<wgrant> Doit.
<jtv> thx
<jtv> the +localpackagediffs is currently oopsing on df because it picks up my tal change on the fly, but needs a restart for the code change
<jtv> wgrant: it's out of your way.
<huwshimi> wallyworld_: Can't believe I missed the game last night
<jtv> wgrant: did you Q/A my update to the queue.py script?  Julian said it was broken.
<wgrant> jtv: It seemed to work for me.
<wgrant> jtv: I initially thought the code was wrong, but it seems OK.
<jtv> Any idea whether you made it print out any package uploads with PCJs attached?
<jtv> That's the part that was in question.
<wgrant> No, but they don't exist yet, so I don't care :)
<jtv> They don't exist?
<lifeless> wgrant: yes, I just mean whenever you think of one.
<lifeless> wgrant: particularly ones where we're close to nuking all db access
<wgrant> jtv: PCJ PUs don't exist in any real environment yet, and they are not close to done.
<jtv> wgrant: in the current code we create PUs with PCJs by creating a fresh PPCJ, and .run()ing it.
<jtv> wgrant: that's no reason to deploy a branch that may or may not have spectacularly failed Q/A!
<wgrant> jtv: It's not going to affect production.
<wgrant> So it has not failed QA.
<wgrant> Production does not yet use PCJs.
<jtv> Until these jobs exist, and suddenly it may break.
<jtv> The whole point is to notice that _before_ it hits production.
<wgrant> There is a fuckload more QA to do of the whole thing before PCJs are going anywhere near production.
<wgrant> Once it's done.
<wgrant> And working.
<jtv> Yes, but this could be an easy thing to miss at that point.
<wgrant> It's one of the more obvious things to test.
<jtv> And it can be tested now.
<wgrant> It can be.
<jtv> Anyway, I think I'm conflating qa-tagging with q/a
<wgrant> Probably.
<jtv> It'd be really nice to have that process cleaned up.  You're right that it can be safely deployed; I've still got the ingrained habit of qa-needstesting meaning what it did.
<wgrant> It needs fixing, yes.
<jtv> deploy-needstesting / deploy-ok / deploy-bad
<LPCIBot> Project parallel-test build #42: STILL FAILING in 1 hr 3 min: https://lpci.wedontsleep.org/job/parallel-test/42/
<StevenK> stub: O hai!
<stub> yo
<StevenK> stub: Mind looking at https://code.launchpad.net/~stevenk/launchpad/db-add-bprc/+merge/64783 for a db review when you have a sec?
<stub> k
<stub> StevenK: So we don't stuff the files in the Librarian because they already exist somewhere on disk, and we can find that from the filename and the bpr?
<stub> oic. The filename is the data, and we generate a file containing all the filenames
<StevenK> Right
<stub> The tablename makes me think 'This row represents the contents of a BinaryPackageRelease', which would be a binary blob or maybe the TOC of a tarball.
<StevenK> It will be many rows
<StevenK> Which is why the primary key is both columns
<wgrant> This is the TOC of a tarball.
<stub> Which answers my next question on why this is a separate table too (not that it would be a bad idea even if it was 1:1, just interested in reasoning)
<lifeless> StevenK: stub: denorm the filename
<StevenK> Hm?
<lifeless> theres only ~2M unique filenames in Ubuntu
<StevenK> Oh
<lifeless> sorry
<lifeless> I mean 'normalise the filename'
<StevenK> I like that idea even less
<stub> normalize
<wgrant> It will save lots of disk.
<stub> Hmm... its always a tough call that. It might save a little space, but will be slower and harder to query.
<wgrant> Probably a good idea.
<wgrant> But...
<stub> How much we looking at?
<wgrant> Makes it hard to search.
<wgrant> This table will be huge.
<StevenK> stub: MANY MANY LOTS
<lifeless> well
<wgrant> Probably >100m rows.
<lifeless> it changes the search dynamic
<lifeless> sometimes its better
<lifeless> sometimes its worse.
<wgrant> No way we can do it before we have more disk.
<StevenK> At this point, I only want the table there. I will look at populating it later
<lifeless> conflictchecker has this table
<lifeless> done normalised style
<stub> StevenK: Will these files already be in the librarian?
<lifeless> it can find conflicting filenames -extremely- fast
<lifeless> stub: no, they are the unpacked contents
<StevenK> stub: No, they won't.
<lifeless> StevenK: so, rather than liking or not liking the table
<StevenK> stub: It will be a list of the contents of the files in the librarian
<lifeless> StevenK: lets analyze. What are the queries you want to do on it?
<StevenK> lifeless: In pseudo-SQL: SELECT filename FROM bprc WHERE binarypackagerelease IN (lots);
<lifeless> StevenK: so for that, we'll be more efficient with two tables.
<StevenK> With some possible bulk loading
<lifeless> StevenK: because 8 bytes are << 50 bytes.
<lifeless> StevenK: the whole 2-4M unique filenames can page into memory, this can't happen with the table you have today because of the 100's of M of rows will be huge.
<stub> Although we don't care if 100's of millions of filenames fit - we only care if the index fits.
<StevenK> A id, filename table, and BPRC links between them?
 * stub tries to recall if each node in the btree contains the text
<lifeless> stub: agreed, but the PK index will be large (because it has to have every filename hundreds or thousands of times)
<lifeless> stub: yes
<lifeless> bah
<lifeless> StevenK: yes
<StevenK> I doubt it is thousands
<lifeless> debian + ubuntu + ppa builds
<lifeless> dailies of some packages, with a couple of series
<StevenK> That's still only 500 to 600
<stub> Just stuff it in Cassandra and stop bothering me before my coffee has kicked in.
<lifeless> StevenK: I'm sure we will have hundreds; we may have thousands for some.
<lifeless> StevenK: 600 * 2M = 1.2B :) I don't think 600 is the common case though.
<lifeless> stub: :)
<lifeless> StevenK: the point is, you aren't building a search schema, you are building an export schema.
<StevenK> lifeless: If it's two tables, it's much harder to populate, too.
<lifeless> not at all
<stub> For some values of 'much'...
<StevenK> I can no longer be done in one query
<StevenK> Er, It
<stub> StevenK: So technically, it could, but we don't want to go there
<stub> I suspect lifeless is right in that we want a separate table. Its 100,000,000 integers + 2,000,000 strings vs 100,000,000 strings. I could do real calculations if we knew the average filename length.
<wgrant> I also concur with lifeless.
<wgrant> FWIW
<StevenK> 38 characters, picking on yelp
<stub> average or max?
<lifeless> StevenK: average length?
<wgrant> I suspect they're going to be well over than 4 characters :)
<StevenK> That is the average
<stub> wow.
<lifeless> if you want, I can dig up my conflict checker credentials and tell you the histogram of sizes for all ubuntu uploads
<stub> So yeah, we would break even around an average length of 3 I think...
<wgrant> stub: These are absolute FS paths.
<StevenK> So it's 3, we have a problem
<stub> Does that mess up the 2,000,000 unique guestimate?
<wgrant> No.
<wgrant> But it's at least 3.2 million.
<StevenK> Suggestions for a filename table? If I use BinaryPackageReleaseFilename, wgrant will hurt me.
<wgrant> But it's roughly around that order of magnitude.
<wgrant> I like Contents.
<wgrant> But I won't murder over Filename :)
<lifeless> StevenK: so the filename table is just filenames
<wgrant> Oh.
<wgrant> Right, that table.
 * stub checks up on toast tables
<wgrant> Well, paths, not filenames.
<lifeless> sure
<lifeless> so anyhow, it's an optimisation table not a semantic table really today; BinaryPaths ? PackagePaths? whatever
<LPCIBot> Project windmill-db-devel build #395: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-db-devel/395/
<lifeless> stub: funky test failure
<StevenK> lifeless, stub: http://pastebin.ubuntu.com/627812/
<wgrant> InternedString? :)
<StevenK> Hm?
<lifeless> wgrant: diaf ? :>
<wgrant> lifeless: :(
<wgrant> I was only partially joking.
<lifeless> I know. Uhm.... needs thought.
<stub> BinaryPackageReleaseContentsFilename.... bleh
<lifeless> and I have a virus or someshite
<StevenK> stub: BPRCF makes me sad
<wgrant> Should I be concerned that I immediately wondered if you were running Windows, and didn't consider it might be a biological entity?
<StevenK> I've gone with BinaryPackagePaths
<stub> It makes me sad two. In hindsight, there would be common acronyms we use like SPR, BPR, SPN
<wgrant> But Code destroyed SPR.
<StevenK> This gives us BPP and BPRC
<StevenK> They overloaded it
<lifeless> StevenK: how about PRC ?
 * lifeless just wants to get country codes into the codebase
<StevenK> Package is ambiguous.
<lifeless> yes, but it would be such a cool acronym
<StevenK> "Meh" :-P
<StevenK> lifeless, stub: Changes pushed, diff updated
<lifeless> stub: I don't see those tests
<stub> lifeless: eh?
<lifeless> you had 3 failures
<lifeless> only one of the tests matches a -t parameter to bin/test
<lifeless> stub: anyhow, lib/canonical/launchpad/doc/unicode_csv.txt passes for me
<stub> Hmmm... and I thought it was weird just because the tests were totally unrelated...
<lifeless> stub: does it pass locally for you ?
<lifeless> ah, I've got the test_encryptor one to run
<lifeless> ok, all three pass
<LPCIBot> Project devel build #810: STILL FAILING in 5 hr 35 min: https://lpci.wedontsleep.org/job/devel/810/
<lifeless> it could be a dependency skew, or a locale thing
<stub> StevenK: bprc.path.path :-(
<stub> lifeless: yes, unicode_csv.txt passes locally
<StevenK> unicode_csv.txt also fails in Jenkins
<stub> Ahh...
<lifeless> did the test environment change recently or something?
<lifeless> like, did someone make the VMs run with a UTF8 locale?
<wgrant> Hmm, it only just failed now.
<wgrant> "Failing for the past 1 build"
<StevenK> stub: Yes, bprc.path.path makes me sad
<lifeless> stub got a failure 9 hours back from ec2
<stub> and 4 hours before that too!
<wgrant> Hm. Jenkins lies.
<wgrant> It failed in 809 too.
<stub> Anyone else seeing issues with ec2 test/land though?
<stub> Maybe I'm the only one trying to use it for db-devel?
<lifeless> stub: so, this is clearly unreleated.
<lifeless> stub: lp-land time.
<wgrant> I guess it's probably related to sinzui's YUITestLayer thing.
<wgrant> But I'm not sure when it first appeared...
<wgrant> A-ha.
<wgrant> Yes.
<wgrant> Those three tests first failed in the build with the new YUITestLayer.
<jtv> lifeless, can I trouble you for a review?  It's nice & short: https://code.launchpad.net/~jtv/launchpad/bug-394645/+merge/64787
<lifeless> wgrant: do you have a feeling for Time to fix?
<wgrant> lifeless: NFI
<wgrant> I can't even start the test suite now.
<lifeless> wgrant: roll it back then, if you would
<wgrant> gi.RepositoryError: Requiring namespace 'Gtk' version '2.0', but '3.0' is already loaded
<lifeless> win?
<lifeless> wgrant: are you on oneiric or natty ?
<wgrant> Natty.
<lifeless> or lucid?
 * lifeless hugs his dev vm
<adeuring> good morning
 * stub wonders if we should use a view to hide bprc.path.path as an implementation detail
<jtv> hi adeuring!
<adeuring> hi jtv!
<jtv> and hi dpm as well :)
<dpm> hey jtv :)
<jml> good morning
<jtv> good morning jml
<jml> I thought huwshimi would be around.
<jtv> jml: your wish, mylord..?
<jml> :P
<jml> huwshimi: http://www.timeanddate.com/worldclock/meetingtime.html?day=16&month=6&year=2011&p1=165&p2=240&p3=-1&p4=-1
<jml> huwshimi: https://dev.launchpad.net/Projects/Disclosure
<mrevell> Hello!
<jml> mrevell: hi
<mrevell> Hello jml
<lifeless> jml: bug 797697 is an example of bugs not quite fitting our critical rules, which I think should jump queue ... what do you think ?
<_mup_> Bug #797697: Creating a bug in a private project via email does not make the bug private <Launchpad itself:Triaged> < https://launchpad.net/bugs/797697 >
<jml> lifeless: looking.
<jml> lifeless: I think it's a bug that doesn't quite fit our critical rules that should jump queue
<jml> lifeless: as a practical matter, it might be addressed by the Teal squad sooner than a maintenance squad.
<rvba> wgrant: bigjools Do you have a moment to talk about the duplication of binary packages in my multi parent initialization branch?
<wgrant> Hi.
<rvba> wgrant: Hi William!
<bigjools> wgrant: we'd like you to expand on the comment you made in the MP that is still unaddressed
<bigjools> if only I could find the MP
<rvba> https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676
<rvba> bigjools: ^
<bigjools> thanks
<bigjools> "It seems to me that there's no binary de-duplication at all"
<jml> bigjools: btw, did you see bryce's email to launchpad-dev@ about Derived distros worksheet?
<jml> jelmer: hello
<bigjools> jml: I havenow
<jelmer> jml: 'morning
<jml> jelmer: how do you wish to communicate, mortal?
<lifeless> wgrant: how do I put something in teals queue?
<wgrant> bigjools, rvba: no_duplicates prevents source conflicts. It doesn't prevent binary conflicts, AFAICT.
<wgrant> lifeless: Poke sinzui, I guess.
<wgrant> lifeless: You could probably throw something on the board.
<rvba> wgrant: correct.
<wgrant> But that's not really happened before.
<jelmer> jml: by some magic coincidence, mumble or skype both appear to work for me at the moment. I'm already on mumble
<wgrant> rvba: That's Really Badâ¢
<bigjools> wgrant: oh where difference sources generate the same binary
<wgrant> bigjools: Or even the same source, right?
<jml> jelmer: mumble hangs for me. skype?
<lifeless> wgrant: bug 798081
<_mup_> Bug #798081: unicode tests failing since yuitest landing <build-infrastructure> <Launchpad itself:Triaged by sinzui> < https://launchpad.net/bugs/798081 >
<bigjools> wgrant: same source?
<wgrant> bigjools: The dedup code is in _clone_source_packages.
<wgrant> That does not provide input to _clone_binary_packages, AFAICT.
<jelmer> jml: Sure - I'm jvernooij
<bigjools> ah ISWYM
<wgrant> What, "that there's no binary de-duplication at all"? :)
<bigjools> no, I was wondering about the "from the same source" comment
<bigjools> having not looked at that code for a long time
<rvba> I suppose I could add the same kind of trick that I used in _clone_source_packages to  _clone_binary_packages to avoid duplicates ...
<rvba> Since the copies are done in sequence AFAIUI
<rvba> Are the db constraints enough to ensure all is well?
<wgrant> rvba: No :(
<wgrant> rvba: They would be huge slow multi-table triggers.
<lifeless> rvba: db constraints never are - because you'd trigger an exception in the db
<lifeless> rvba: (at best)
<jml> jelmer: http://launchpad.leankitkanban.com/Boards/Show/12720553
<lifeless> sinzui: I've assigned a bug to you - its a yuitestlayer regression :(
<lifeless> sinzui: making ec2 & jenkons fail
<jml> jelmer: https://dev.launchpad.net/PolicyAndProcess/FeatureDevelopment
<rvba> lifeless: wgrant: I think you misunderstood me: what I meant was: can I just take a look at the constraints of the binary package table and enforce those in the code?
<stub> lifeless: Any objections to a bzr branch to maintain database patch numbers, or would a wiki page be better?
<jml> jelmer: https://dev.launchpad.net/Projects
<lifeless> stub: We could just use trunk :)
<stub> lp:~launchpad/+junk/dbpatches ?
<rvba> I was not even thinking about letting the db do the job :)
<lifeless> stub: but I have no objections to $whatever-you-want-to-do
<lifeless> stub: there is a number on the policy wiki page
<stub> lifeless: Which trunk? devel or db-devel?
<lifeless> stub: its a bit fiddly to edit
<lifeless> stub: {yes} (db- for -0, devel for -!0)
<stub> lifeless: Yes. I want something better that is shared, and can't be arsed retrieving my google docs password.
<wgrant> rvba: No :(
<stub> lifeless: no, we are not splitting our documentation on what we plan to land into two.
<rvba> wgrant: hum ... after taking a look at the table there is no constraint ...
<wgrant> rvba: There are far more complex constraints that cannot be feasibly enforced in the DB.
<wgrant> rvba: Right, because it's cross-table.
<rvba> wgrant: Right ... so, what do you reckon should be enforced on the data during the copy? What kind of "uniqueness"?
<wgrant> rvba: A very good question, and the reason I didn't provide a more concrete answer in the review.
<wgrant> rvba: We should ensure that there are no duplicates within the copy. If it's not a new archive, we also need to ensure there are no file conflicts with anything in the archive. And we should also ensure that there are no orphaned binaries.
<rvba> wgrant: ok, a distinct should take care of "no duplicates within the copy"
<wgrant> rvba: Probably, yes.
<rvba> wgrant: what do you mean exactly by "no file conflicts"?
<wgrant> rvba: I can't have two different foo_1.2-3_i386.deb files in the same archive.
<rvba> wgrant: no orphaned binaries: I'll see what can be done ... but unless I'm mistaken bigjools sayed this is Bad But Not Too Badâ¢ ... at least for now.
<rvba> arf s/sayed/said/
<wgrant> rvba: We really want to avoid initialising an archive into a bad state, I think.
<wgrant> It's not fatal, unlike the other two.
<wgrant> But it's not good.
<rvba> wgrant: Understood ... but I'm not sure it's easy to do with a simple sql query
<rvba> wgrant: and the initialisation already takes ages ...
<bigjools> distinct on the (name, sha1) I think
<LPCIBot> Project windmill-db-devel build #396: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/windmill-db-devel/396/
<bigjools> ah yes StevenK, around?
<lifeless> stub: anyhow, a +junk branch sounds fine, but I don't know if teams are permitted +junk
<stub> lifeless: It worked...
<lifeless> stub: cool
<lifeless> stub: can you subscribe me to the branch ?
<jelmer> jml: One more quick question - do I keep the current LEP as is (since it's already approved), or can I remove the bits from there that I've moved to the new LEP?
<lifeless> stub: also I had an idea
<jml> jelmer: remove the bits. leave it as RTC
<lifeless> stub: can we remove the INSERT INTO LaunchpadRevision ... line from patches ?
<lifeless> stub: and have the applying wrapper supply it ?
<jelmer> jml: thanks
<jml> lifeless: teams are permitted junk.
<adeuring> bigjools: ping
<rvba> wgrant: bigjools I think I've enough to get going with this duplication thing ... thanks.
<bigjools> adeuring: hello
<bigjools> rvba: great
<adeuring> bigjools: do you have to talk about bug 793630?
<stub> lifeless: Yes, but there were reasons I had it in there explicitly.
<_mup_> Bug #793630: better cronscript setup: remove hard-coded paths from cronscripts/publishing/cron.base-ppa <Launchpad itself:Triaged> < https://launchpad.net/bugs/793630 >
<lifeless> stub: do they still apply? Is the trade off still worth it?
<stub> lifeless: I like having it encoded in the content for when things get passed around or applied through means other than update.py
<stub> lifeless: Also, it is a bit of a checksum against typos and the version number is important.
<stub> c/version/patch
<lifeless> stub: it being important is why I'm wondering about removing the duplication
<StevenK> bigjools?
<stub> Removing duplication means scope for error when it needs to be manually entered, such as when I'm applying a db patch live at 4am while drunk.
<stub> c/when/if :-)
<lifeless> stub: there seem to be a -raft- of issues in that proposition :0
<stub> So I don't think the duplication solves anything, but I think it helps a little.
<stub> And I don't think the overhead is bad and ensures any patch renumbering is done carefully.
<lifeless> stub: does upgrade.py cross-check the patch number?
<stub> yes
<lifeless> kk
<stub> It was deliberately done this way, but I have no idea if my original reasoning matches my current reasoning.
<adeuring> bigjools: ?
<bigjools> sorry had a call
<rvba> bigjools: I see a sha1 field only in libraryfilecontent ... and I'm not really sure how to find those from binarypackagepublishinghistory ...  I suppose making sure (binarypackagename, version) of the related binarypackagerelease is another way to avoid this kind of duplicates ...?
<bigjools> StevenK: I wanted to chat briefly about where the packagecloner could be going slow, I think you identified it as the slow bit in IDS right?
<bigjools> adeuring: let me take a quick look
<StevenK> bigjools: Yes, it's the package cloner. But I have no idea about its internals
<StevenK> bigjools: Perhaps it's the .createMissingBuilds() calls
<bigjools> StevenK: ok.  it's a job for a PG expert then :)
<bigjools> StevenK: no, on DF I observed a VERY long INSERT
<wgrant> StevenK: But IDS shouldn't be using createMissingBuilds
<wgrant> DF SPPH (I think) inserts often take a long time.
<wgrant> You can see it in p-u too.
<wgrant> Creating PENDING publication record.
<wgrant> [... wait for a couple of minutes]
<StevenK> The PackageCloner calls .createMissingBuilds()
<wgrant> But it's possible cMB.
<bigjools> it was BPPH
<bigjools> rvba: one sec, the join is easy
<bigjools> what['s the normal way of scripts finding the root of the code tree?
<StevenK> canonical.config.root ? Or something?
<bigjools> in a shell script
<bigjools> hmmm it comes from the crontab I think
<StevenK> pwd?
<bigjools> adeuring: so, in this case I think we need to shift the definition of LPCURRENT to something that comes from the crontab
<rvba> bigjools: yeah, but if we can avoid joining too many tables it's all for the best so my question is: is the uniqueness of binarypackagerelease(binarypackagename, version) sufficient?
<bigjools> rvba: no it's not
<adeuring> bigjools: yeah, that's what wanted to ask: the regular config stuff would result in a hen/egg problem, wouldn't it?
<bigjools> rvba: BPPH -> BPR -> BPF -> LFA
<rvba> bigjools: ok, thanks!
<bigjools> adeuring: LPCONFIG is already set in the crontab
<bigjools> adeuring: unless you can get config.root out in bash?
<bigjools> rvba: I hope hose abbreviations make sense :)
<rvba> bigjools: I think it's going to be ok ;)
<adeuring> bigjools: we could switch to a python script.
<StevenK> stub: You said both columns in BPRC should be NOT NULL -- they're both in the primary key, so it shouldn't be needed?
<bigjools> adeuring: yes, we already did that for the ubuntu wrapper
<bigjools> adeuring: the base-ppa is sourced from less cronscripts/publishing/cron.publish-
<stub> StevenK: Yes, but I like being explicit. We don't want to change the primary key constraint in the future and lose the implicit NOT NULL
<bigjools> ppa
<StevenK> stub: Fair enough, I shall add it.
<bigjools> adeuring: let me try that again!  it's sourced from cronscripts/publishing/cron.publish-ppa
<adeuring> right
<adeuring> and that's done in two scriptd.
<bigjools> adeuring: which is a trivial bash script that calls three python scripts in turn
<stub> (Not sure if it matters here - just consider it lint)
<adeuring> bigjools: right, but there is no scriptactivity logging, at least for one of the scripts
<bigjools> adeuring: the cron.daily-ppa does some cleanup
<adeuring> so another reason to change to apython script
<bigjools> adeuring: there is scriptactivity
<bigjools> from the 3 python scripts
<bigjools> that's what matters
<adeuring> in cron.daily-ppa?
<bigjools> not that one
<stub> lifeless: So 0mq looks really cool and reading the docs well worth while, even if it becomes clear it doesn't meet some of our existing use cases without development work.
<bigjools> that one's not critical
<adeuring> ok
<stub> lifeless: The authors also share some of my opinions on HA setups, and argue some points better than I do.
<adeuring> but still: If the wrong value of LPCURRENT was a problem, why not define it in the regular configuration?
<stub> lifeless: But if we want a queue with persistence that survives restarts, we would need to implement it in 0mq (they provide designs and most of an implementation in the docs), stick with rabbit, or go with kestrel or activemq or something.
<adeuring> bigjools: can we talk on mumble about it?
<lifeless> stub: yes, thats true
<lifeless> stub: gavin seems to have some traction on rabbit
<stub> lifeless: Cool. If he has no luck next week, I'll implement a trivial messaging system with PG that we can swap out for something better later.
<lifeless> stub: hmm, given our initial use cases don't need persistence, I'd suggest holding off
<lifeless> stub: Its up to you, but I think there are other things more pressing for us
<stub> lifeless: I mean if we can't get rabbit stable in the test environment, I can throw together a simple high level API backed by PG that will work and allow us to do basic messaging while we work out the low level details.
<stub> It won't advertise itself as being reliable or persistent, because we may well swap the implementation for something unreliable and ephemeral.
<lifeless> stub: sure, thats what I thought you meant
<stub> 0mq does look ideal for centralised logging though
<stub> And centralised OOPS collecting
<stub> Lets us kill OOPS prefix too
<lifeless> we can kill the prefix by changing the naming to a hash of the oops file
<stub> Or we can keep them short and have the id allocated by the OOPS collector
<lifeless> we need the name allocated on the server if we want to show the id to the user even if the collector is down
<stub> We can have multiple collectors
<lifeless> stub: yes, but that won't help when the collector is 'down' due to network issues
<allenap> Or use base64.urlsafe_b64encode(uuid.uuid1().bytes).rstrip("=") for the OOPS number.
<stub> turtles all the way down
<lifeless> stub: hah yes.
<stub> If the network is having issues bad enough to not submit an OOPS, I'd argue the OOPS is just noise.
<lifeless> stub: I guess my basic thinking here is that its better to have fire-and-forget during error handling
<lifeless> stub: actually I care a great deal about oopses at such times; short-transient-network glitches are hell to diagnose
<lifeless> stub: lots of data is a big win
<allenap> stub: I'm 90% confident that I've found the source of the Rabbit fixture problems we were having.
<stub> Ok. So we could fallback to using a UUID if the OOPS server can't reply immediately.
<lifeless> stub: wouldn't it be simpler to just have one way of doing it that is reliable-by-approach  ?
<stub> allenap: Yay :)
<stub> lifeless: I like short OOPS numbers. The ones we have are already too long. They are human readable codes.
<lifeless> stub: Why does human readable matter?
<stub> Because we see them all the time in bug reports, we paste them into IRC sessions, we email them, we might even end up memorizing some of them.
<lifeless> all of those are robustly copy-pastable
<bigjools> allenap: yay!
<lifeless> except memorising & speaking
<bigjools> lifeless: I really want us to sort out a MQ strategy because I intend to do the long poll stuff in Dublin
<stub> And having to cut and paste is suckier than having the option of cutting and pasting, and a 40 character id is going to be less readable than a 6 character id when embedded into a bug report.
* allenap changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugs: 202 - 0:[######=_]:256
<lifeless> stub: when was the last time you spoke an OOPS ID ?
<lifeless> bigjools: yes, I know :(.
<bigjools> lifeless: I am assuming it's going to be Rabbit
<stub> lifeless: Can't recall, but I'm sure I've had them read out in skype conversations.
<stub> Maybe as I used to do standups with the QA team
<lifeless> stub: a prefix match will support reading-out easily - just use the first 6 digits
<lifeless> bigjools: well, as I said in prague, lets unblock and use it.
<lifeless> bigjools: just not [for now] for persistent data; your use case doesn't need that.
<stub> Which is still suckier than just being able to read the whole code.
<lifeless> stub: all things are tradeoffs
<stub> We can do client generated oops codes, but they are less usable than server allocated codes.
<bigjools> lifeless: exactly
<stub> And usability matters as they are the interface between humans and our qa databases.
<stub> I don't think we want to throw away that usability for pathological situations, given there will always be a more pathological situation we will never be able to cope with
<lifeless> stub: I agree that usability matters; I am not at all convinced that modest length is an issue
<stub> (oops... disk full... )
<lifeless> wgrant: can you put a link to the damaged feed in https://bugs.launchpad.net/launchpad/+bug/78942 ?
<bigjools> allenap: so what is the rabbit issue you found?
<bigjools> lifeless: why does the rabbit fixture double-fork?
<allenap> bigjools: See my reply to "rabbit, where art thou?" in launchpad-dev. I think it's a race between rabbitmqctl and the rabbit server to create the cookie file.
<lifeless> bigjools: thats how U1 do it
<bigjools> lifeless: why does U1 make the rabbit fixture double-fork? :)
<lifeless> bigjools: I made the minimal changes on the assumption their code had reasons to be the way it was.
<bigjools> it seems - odd
<wgrant> lifeless: Their discontinuation was announced three weeks ago.
<bigjools> and there's no comments
<lifeless> bigjools: no tests and no docs make answering that hard.
<bigjools> quite
<bigjools> having seen some other Python projects recently I really appreciate our coding standards
<lifeless> stub: I've a few bugs that contributed to downtime post bugsummary; would you look at them for me?
<wgrant> I questioned the double-fork when I semi-reviewed it. But decided it was better to get the code in and iterate.
<lifeless> stub: and how would you like me to put them in your queue.
<bigjools> allenap: ah ok, I am currently ploughing through 25 unread messages in that thread
<allenap> wgrant: I have a branch that removes the double-fork and uses subprocess.
<wgrant> allenap: Excellent.
<bigjools> yay!
<wgrant> allenap: Does it remove any of the other crapness?
<wgrant> This crapness may include bad style, and not working at all.
<bigjools> you'd think that U1 would get this race too
<allenap> wgrant: I hope so... :)
<allenap> wgrant: I'll get it landed, today hopefully, and I promise not to mind if you improve on that :)
<wgrant> allenap: I fear the improvement will be to disable the test again, but I can hope I'm wrong :)
<allenap> wgrant: Ah, I think I've figured out the cause of the problems.
<wgrant> I saw that.
<lifeless> bigjools: they probably do occasionally
<lifeless> wgrant: which were discontinued?
<wgrant> lifeless: http://feeds.ubuntu-nl.org/MaverickChanges
<lifeless> ok, so wontfix ?
<wgrant> No.
<wgrant> Not necessarily.
<wgrant> Well.
<wgrant> The redirects are still broken.
<wgrant> I don't know of any remaining non-antique sources of those URLs.
<lifeless> but nothing outputs that url
<lifeless> right
<lifeless> I'm going to close it; if someone finds an active source we can reevaluate
<wgrant> Sounds good.
<lifeless> stub: please ping when you are bakc
<stub> lifeless: back
<lifeless> catchup call ?
<stub> lifeless: Give 'em a tag (dba is fine), or assign them to me.
<lifeless> stub: i've tagged em ops
<stub> lifeless: Sure, but needs to be quick. I need to go buy some UK power adapters.
<lifeless> stub: there are 3; the fourth ops a maintenance squad will get
<lifeless> stub: kk, skype?
 * stub boots his laptop
<lifeless> clearly not 5 second boot :)
<bigjools> wgrant: can I borrow your brain and eyeballs for a bit please
<wgrant> bigjools: Probably.
<bigjools> http://pastebin.ubuntu.com/627895/
<wgrant> aaaaa
<bigjools> I wrote that test for the SPR.getBuildByArch problem and it passes unexpectedly
<bigjools> I've been staring at it too long to see why
<wgrant> bigjools: You're sure that derived_series is in a new distribution?
<wgrant> I mean, it looks like it should be, but still.
<wgrant> Oh.
<wgrant> The binary is published.
<wgrant> That would do it.
<bigjools> sorry, why?  I am feeling particularly slow today
<wgrant> bigjools: IIRC getBuildByArch looks for anything published first.
<wgrant> Otherwise copies would duplicate builds.
<wgrant> Let me check.
<wgrant> Yes.
<wgrant> It queries for any binary publications, and returns the single build that produced them.
<wgrant> Only afterwards does it do the whole-archive match on (spr, archtag)
<bigjools> meh
<bigjools> thanks.  I am blind
<bigjools> wgrant: hmmm wait though
<bigjools> it's looking for publications in the passed DAS
<wgrant> :(
<wgrant>  do_copy(
<wgrant> +            [parent_source_pub], derived_archive, dsp.derived_series,
<wgrant> +            PackagePublishingPocket.RELEASE, include_binaries=True,
<wgrant> +            check_permissions=False)
<wgrant> You copied binaries.
<wgrant> There are pubs :)
<bigjools> yes the intention was to copy binaries
<bigjools> to get the build from the parent
<wgrant> Yes.
<wgrant> 21:02:01 < bigjools> it's looking for publications in the passed DAS
<wgrant> Is that a surprise?
<bigjools> gah
<bigjools> so none of this may be a problem at all
<wgrant> Well.
<wgrant> Maybe.
<bigjools> wtf would someone query for a build, passing in a context where it's not published
<wgrant> Well.
<bigjools> well :)
<wgrant> This works to get a failed build from the source of a copy.
<wgrant> I'm not sure it's ever used for that.
<bigjools> I think I am going to invalidate this bug because the scenario I was concerned about is not a problem
<bigjools> I might keep my test though :)
<wgrant> A good plan.
 * bigjools sighs at the headset volume control STILL controlling the speak volume
<bigjools> speaker*
<LPCIBot> Project windmill-devel build #233: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/233/
 * bigjools grumbles at patch 75
<wgrant> bigjools: Is that the bugsummary expansion?
<bigjools> yeah
<bigjools> df is going to be busy for a while
<LPCIBot> Project db-devel build #640: STILL FAILING in 6 hr 20 min: https://lpci.wedontsleep.org/job/db-devel/640/
<wgrant> jelmer: Your fix QAed fine. Thanks for the quick work.
<LPCIBot> Project windmill-devel build #234: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/234/
<jelmer> wgrant: awesome - thanks for pinging me about it
 * bigjools holds on to desk while overflying Chinook shakes the office to bits
<jelmer> wgrant: is there any way to easily notice that a new revision has been deployed on staging?
<wgrant> jelmer: If the rev needs QA its bugs will be tagged qa-needstesting once it's on (qa)staging.
<wgrant> jelmer: But rollback= revs don't need QA, so that didn't happen here.
<jelmer> wgrant: I knew about qastaging, but do deployments to staging happen at the same time?
<wgrant> And this can only be QAed on staging, because qastaging has no importds yet :(
<wgrant> No.
<wgrant> It's separate.
<wgrant> But it normally happens within a couple of hours of db-stable tip changing.
<jelmer> wgrant: That's useful to know - thanks
<jtv> Has anybody else been seeing failures on the unicode tests in PQM that don't happen locally?
<wgrant> jtv: Yes.
<wgrant> There are three. Ignore them for now. sinzui's YUITestLayer changes somehow break them.
<jtv> Thanks!
<wgrant> (but not on buildbot, and not locally)
<lifeless> hmm, bugsummary 2 isn't on staging yet ;(
<wgrant> renamed: lib/lp/bugs/stories/bugs/xx-bug-personal-subscriptions.txt => lib/lp/bugs/stories/bugs/xx-bug-personal-subscriptions.txt.THIS
<wgrant> danilos: ^^
<wgrant> r13243
<danilos> wgrant, :/
<danilos> wgrant, I'll land a fix for that (it's supposed to be removed)
<wgrant> Also, 12k-line unflagged changes on fragile code upset me.
<wgrant> But OK.
<danilos> wgrant, what do you mean with "unflagged"?
<StevenK> Non-feature flagged
<danilos> ah
<wgrant> So there is no escape.
<danilos> well, I am not disagreeing with that, but 4k of those lines are removals, and another 3k are the actual feature flag removal
<wgrant> Branches are cool :)
<danilos> wgrant, yeah, it was all spread out in like 8-9 branches, but for easier "escape", we did it as one branch
<danilos> we landed it as one branch
<danilos> wgrant, btw, are you reverting that revision or should I?
<wgrant> danilos: I'd prefer not to fix that move.
<wgrant> It is late here.
<danilos> wgrant, it seems there's a bunch of PackageCopyJob failures, not sure what they are about
<danilos> wgrant, right, just checking, thanks
<wgrant> Oh.
<wgrant> What?
<wgrant> PackageCopyJob failures? Where?
<wgrant> Huh.
<danilos> wgrant, buildbot run that's just ending for my rev: https://lpbuildbot.canonical.com/builders/lucid_lp/builds/1048/steps/shell_6/logs/summary
<danilos> tests, not production :)
<danilos> wtf, "NameError: global name 'ISourcePackageNameSet' is not defined"
<wgrant> This is why we don't land 12000 line revisions.
<wgrant> They are unreviewable.
<wgrant> And un-QAable.
<wgrant> And... they do that.
<wgrant> :)
<danilos> wgrant, well, these same tests fail for me in devel
<danilos> not sure how it got through ec2 :/
<spiv> Land 13000 line revisions instead.  13k is a lucky number!
<danilos> spiv, good point, thanks
<wgrant> danilos: Yes, there was a merge issue somewhere. :(
<wgrant> They are less bad if you review every diff before you land them. Or if people are watching diffs as they land. Which is a bit impossible when they are so huge :/
<danilos> wgrant, yeah, I'll have to revert and look into it
<wgrant> Thanks.
<wgrant> Also, one of the MPs that was merged was rejected, which is somewhat concerning.
<wgrant> I'm not sure you merged what you thought you did.
<danilos> wgrant, was it? which one was that?
<wgrant> danilos: You can't land the FF removal first?
<wgrant> https://code.launchpad.net/~benji/launchpad/bug-784575-message/+merge/62157
<wgrant> Something like that landed on db-devel last month.
<danilos> wgrant, well, that seems to have been included in a different branch from Benji then
<danilos> I am guessing the problem with packagecopyjob merge is that someone reverted something using patch instead of bzr merge
<wgrant> danilos: Reversions are just patches.
<wgrant> danilos: Since they are cherrypicks, which aren't tracked in the DAG.
<danilos> wgrant, doesn't bzr track them (I've had bzr nicely recognize a reverted *revision* before and not worry about it, so I thought it was smarter than that)
<wgrant> danilos: No.
<danilos> wgrant, that's surprising
<wgrant> A little.
<wgrant> But it normally works fine.
<spiv> Well, a commit of a revert is a "rich patch" if you like, in that it copes with renames and deletes a little better than a raw diff/patch.
<wgrant> True.
<spiv> But there's no record kept in the revision graph to indicate the fact that a particular revision has been undone.  But as you say mostly things work fine without explicitly tracking that.
<LPCIBot> Project windmill-devel build #235: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/235/
<danilos> spiv, oh, and I thought "bzr merge" (and especially "-r" option) did more than that
<spiv> danilos: usually it does
<danilos> spiv, but specifically not for "-rREVNO..REVNO-1"?
<spiv> danilos: but giving it a reversed revision range is a cherrypick
<danilos> spiv, right, understood
<spiv> danilos: at a low-level each revision has a field to record a sequence of one or more parent revisions
<spiv> danilos: and that's all the data kept for the revision graph
<jtv> allenap: got time for a quick review?
<spiv> So there's no way to represent "oh actually I took revisions 3 and 4 back out" in that structure.
<jtv> It's something I'm hoping to polish off very soon.
<spiv> (and similarly no way to represent a cherrypick in general.  Just a statement that revision X includes Y and Z, and so transitively all the ancestry of Y and Z)
<danilos> spiv, right, that's a shame, but if it mostly works fine, I guess it's no big deal
<spiv> Or if you like, if 'bzr qlog' can't plot it that relationship, bzr can't record it ;)
<danilos> spiv, I generally attributed "smart" behaviour to it recording more data, but I guess it's intrinsic to the merge algorithm (i.e. if I land a branch which includes a "reverted" change, that change generally is not re-applied on the landing; so, now I know to be a bit more wary of these things :)
<spiv> Right, because after all if someone commits a revert you probably want that change preserved :)
<danilos> hum, I keep getting the test failures in my devel copy for PackageCopyJob tests, anyone else can confirm/dispute that?
<wgrant> danilos: Even after you've reverted it?
<danilos> wgrant, well, I definitely had it earlier, but not with the revert now (I had it before I refreshed my devel to include r13243)
<jtv> allenap: I've got a review I'm hoping to polish off very quickly before I leaveâ¦ got time for it?  It's not big: https://code.launchpad.net/~jtv/launchpad/bug-798179/+merge/64813
<allenap> jtv: Yes indeed.
<jtv> Great!
<wgrant> danilos: 13243 was clearly the cause. It removed the import.
<danilos> wgrant, right, but I had it even without it, so it seems some of my devel merges have gotten that in (I, naturally, haven't touched the file)
<danilos> though, I was exclusively merging stable
<jtv> so... you ran into that one as well?
<danilos> jtv, so I am not the only one?
<jtv> Nope.  Missing ISourcePackageNameSet?
<jtv> In packagecopyjob.py?
<danilos> jtv, or are you perhaps hitting it because of my recent landing earlier today?
<jtv> No ideaâ¦ I thought I'd caused it but couldn't find where I'd done it.
<allenap> jtv: Why do you need to call view.setupQueueList() explicitly in the test?
<danilos> jtv, hum, my branch landed on r13243, can you please check if you have that in your revision history?
<jtv> allenap: that initializes the batch navigatorâ¦ is the call redundant?
<jtv> danilos: I do have the revision
<jtv> otherwise I wouldn't have known we were hitting the same thing.  :)
<danilos> jtv, heh, well, what I am asking is if you had that problem prior to that revision as well :)
<jtv> Oh.  I don't _think_ I did; I've been branching a lot these past few days and only hit it today.
<allenap> jtv: Maybe not. I would assume that happens when you call the view. I mean, it *has* to happen IRL, outside of a test case, but perhaps there's something missing that means it's needed in the test.
<danilos> jtv, that revision has obviously caused it, but I got it into my branches somehow without ever touching the file (so due to some weird merging, I gues)
<jtv> allenap: why don't I just try it?  :)
<allenap> Okay :)
<jtv> danilos: as if the revision that added it maybe got lost?
<danilos> yeah
<jtv> Frightening.
<jtv> allenap: you're right â renders fine without.
<jtv> Pushing change.
<allenap> jtv: Cool. r=me.
<jtv> thanks!
* jcsackett changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugs: 202 - 0:[######=_]:256
<jtv> good morning jcsackett
<jcsackett> good morning jtv.
<danilos> it gets even better, I checked the branches I landed and *none* of them have changes in packagecopyjob.py file that my branch supposedly modifies
<wgrant> danilos: No warnings of criss-cross merges as you merge them?
<danilos> wgrant, no, but some of them have been live for a few weeks already and they've been merged by other people
<danilos> well, had devel merged in by other people
<LPCIBot> Project windmill-devel build #236: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/236/
<danilos> spiv, so, now that I reverted one change, if I want to land it and keep the revision graph, is there a way to do that? (considering it's a 12k line change, I'd like to keep actual revision log in the history)
<jtv> allenap, jcsackett: I have one more candidate for review, but I'm afraid I can't stay around to answer questions.  It's a large-scale removal of an obsolete method: https://code.launchpad.net/~jtv/launchpad/post-394645/+merge/64829
<allenap> jtv: I'll take it.
<jtv> wonderful, thanks
<jtv> allenap: I'm afraid I have no idea what to do about the pre-existing lint that I didn't remove.
<jtv> I did land 7â8K of lint cleanup for those tests earlier though.
<jtv> 7â8K *lines of diff*, that is.
<allenap> jtv: No worries, I'll ignore lint.
<jtv> Hmmm.  :)
<allenap> Blimey.
<jtv> Yes, I've really been pushing it.
<jtv> Lazily continuing to let the tech debt pile up on soyuz didn't seem like the right thing to do somehow.  :-)
<spiv> danilos: I'm not sure what you mean exactly by keep the revision graph
<allenap> jtv: Agreed :)
<danilos> spiv, well, "bzr log -n" should show the actual revisions
<jtv> allenap: Anyway, I'll be off then.  Thanks again.
<spiv> Revisions are only added, not taken away
<allenap> Cheerio jtv.
<spiv> (Unless you forcibly uncommit or push/pull --overwrite with an older revision)
<spiv> So if you add a commit, you are strictly adding to history, not removing any of it.
<danilos> spiv, well, if I try to "stupidly" remerge my branch into trunk (which now has a reverted change), my changes are ignored; basically, these revisions are already in the graph for the previous commit, so I guess bzr can't internally link to it either?
<spiv> Right, those revisions are already present.
<danilos> spiv, and there is no way to indicate that revision I am committing includes changes from that revision? so basically, it'll be a simple "patch" again?
<spiv> If/when you want to revert the revert, you can do just that: bzr merge -r REVERT_REV..REVERT_REV-1
<danilos> spiv, right, that's what I am doing, and losing all the subrevision history that the original commit had
<spiv> What does "it" in "it'll be a simple..." refer to?  The revision you are about to commit, or some other revision?
<danilos> spiv, revision I am about to commit
<danilos> spiv, ideally, it'd be a revision listing previously reverted revision as the parent, at least imho :)
<spiv> In general: if "bzr st" shows "pending merges:", then the tips of those merges will be included as parents in the new commit.
<danilos> spiv, (just wondering if that'd be at all possible and whether it makes sense, not a big deal)
<spiv> It would be technically possible to have revision where one of the extra parents is in the ancestry of the left-hand (a.k.a. mainline) parent.
<spiv> I'm not sure if you can easily convince the CLI to create that situation, though.
<spiv> Basically, the short answer is: just say what you're reverting or unreverting in the commit message :)
<spiv> That's probably going to be good enough :)
<danilos> spiv, heh, right enough, so in theory it should be possible to create a cyclic revision graph in bzr? :))
<danilos> spiv, anyway, sure thing, I'll be detailed enough in the commit message
<spiv> No, I don't see how that allows cycles?
<spiv> A cycle would require a situation where you somehow make an older commit point to the new commit
<spiv> Which I'm pretty sure is impossible with the CLI :)
<spiv> You could of course construct all sorts of invalid data by hand.
<spiv> (Or by abusing sufficiently low-level parts of bzrlib)
<LPCIBot> Project windmill-devel build #237: STILL FAILING in 39 min: https://lpci.wedontsleep.org/job/windmill-devel/237/
<timrc> jcsackett, hey I re-submitted my merge proposal[1] -- the original change I made broke Storm expressions using Archive.private, so I had to address that
<timrc> [1] https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-4/+merge/64840
<jcsackett> timrc: cool. i will look at it in just a moment.
<timrc> jcsackett, cool, I spot checked some of the tests that were failing yesterday afternoon and am now running the full test suite right now
<rvba> allenap: jcsackett(hi!) Could you have a look at https://code.launchpad.net/~rvb/launchpad/initseries-no-arch-bug-795396/+merge/64843 ?
<allenap> I'm doing a big review for jtv, going to be a while longer.
<allenap> But I can do it after that unless jcsackett has got to it :)
<rvba> Great! (it's a tiny MP)
<bigjools> timrc: ah you found your bug then?
<timrc> bigjools, yah, abentley helped me out there... turns out using Python getter/setters breaks Storm expressions that use the attribute (e.g. Archive.private) -- I'm not entirely sure why, to be honest
<bigjools> ha!
<bigjools> yeah it makes sense
<bigjools> I didn't spot that.... doh
<bigjools> good job that one unrelated test caught that
<abentley> timrc: It's because Archive.private is a reference to the BoolCol class, but if you change it into a property, it's not.
<timrc> well there were a few failures and one error and that was the common thread
<timrc> abentley, ah okay, yeah that makes sense
<timrc> abentley, which is why using the Archive._private satisfies Storm
<abentley> timrc: right.
<jcsackett> timrc: only changes between this and the last version is the updates from using .private to ._private, yeah?
<timrc> jcsackett, yeah
<adeuring> allenap: fancy a review with a tiny diff? https://code.launchpad.net/~adeuring/launchpad/bug-793630/+merge/64847
<timrc> jcsackett, I believe I caught all the instances where the change was needed, testing should confirm that (should)
<jcsackett> timrc: dig.
<jcsackett> timrc: this looks good; i've approved the mp.
<jcsackett> you said the full suite is running now?
<timrc> jcsackett, great, thanks... yes full suite running
<jcsackett> timrc: sounds good.
<timrc> jcsackett, will we have to re-test to land?
<jcsackett> timrc: generally i'm loathe to land without ec2 landing. how long have you had the suite running?
<timrc> jcsackett, not very long, 20-30 minutes
<jcsackett> timrc: if you want to kill the current test run, i can send your branch out via ec2 to land now.
<timrc> jcsackett, okay, sounds good
<jcsackett> timrc: cool. i'll send it out momentarily then. :-)
<allenap> adeuring: Looks good.
<adeuring> allenap: thanks!
<cody-somerville> timrc, terminating ec2 test
<timrc> jcsackett, doh, you'll be mad at me... the merge proposal did not include the "note" explaining the lint's failure to recognize the correctness of the way I implemented the getter/setter... what shall we do? http://paste.ubuntu.com/628006/
<jcsackett> timrc: if you want to push that change up, it's fine. i've had to do a quick update and link your branch to the bug it's fixing, so i'm just now at issuing the land command.
<jcsackett> push up the change with the comment, and then i'll send it out.
<timrc> ok, doing so now
<LPCIBot> Project windmill-devel build #238: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/238/
<timrc> jcsackett, okay, think everything is there now -- thanks for your patience!
<jcsackett> timrc: looks good. sending it off now.
<jcsackett> should it fail, you'll get the traditional email telling you things are broken. :-)
<timrc> jcsackett, sounds good, thanks
<jcsackett> rvba: i'm looking at your branch now.
<rvba> jcsackett: thanks!
<flacoste> allenap, jcsackett: https://code.launchpad.net/~flacoste/launchpad/bug-797917/+merge/64852
<flacoste> should be a quick review
<flacoste> pretty please :-)
<allenap> flacoste: Okay, you ask so nicely :)
<flacoste> thanks allenap
<allenap> flacoste: r=me
<flacoste> allenap: thanks!
<flacoste> any vim users who integrated make lint output? (so that I can use cn to go to all of them?
* allenap changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 202 - 0:[######=_]:256
<jml> I have to head off.
<jml> see you all tomorrow
<maxb> flacoste: Hello. What is the landing status of ~flacoste/launchpad/bug-797088 ? Thanks
<flacoste> maxb: running through EC2
<flacoste> maxb: EC2 didn't love me yesterday, all my attemps failed in stalled bzr
<LPCIBot> Project devel build #811: STILL FAILING in 5 hr 46 min: https://lpci.wedontsleep.org/job/devel/811/
<maxb> Ah, I did notice some conversation about that
<jcsackett> rvba: comments on your diff; i think you can cut down the query count you boosted.
<rvba> jcsackett: great, thanks. I'll have a look.
<flacoste> maxb: with luck it will be deployed before tomorrow, otherwise, it will probably go to Sunday (antepodeans Monday)
<flacoste> maxb: i could fix the Debian issue immediately though
<flacoste> as this only requires chaning the maintainer team
<maxb> I think most of the problems are SRUs wanting to create their natty-updates branches
<flacoste> that requires my branch
<maxb> Right - so no need to hurry on fixing just debian
<timrc> I forgot to pay Amazon $10 for 2 or so years and now they are mad at me
<timrc> need to take it out to dinner and buy it flowers, I think
<timrc> s/forgot/neglected/
<sinzui> flacoste, ping
<rvba> jcsackett: thanks for your suggestion, I've fixed the branch.
<jcsackett> rvba: r=me.
<jcsackett> and i concur that ordinarily is_empty is better. :-)
<rvba> thanks!
<sinzui> hands up who knows how to set/force the default encoding in python. My use of sys.setdefaultencoding('ascii') does not seem to work.
<LPCIBot> Project db-devel build #641: STILL FAILING in 5 hr 27 min: https://lpci.wedontsleep.org/job/db-devel/641/
<flacoste> hi sinzui
<sinzui> flacoste, I was struggling with an import issue. importing gtk sets python's default encoding to utf-8. barry has been helping me.
<sinzui> flacoste, I am going to trying reload(sys), the only known way to correct the corruption
<sinzui> my next choice is to make lucid run modern gtk
<flacoste> is this related to the the UnicodeEncodeError test failures?
<sinzui> flacoste, yes it it. I updated https://bugs.launchpad.net/launchpad/+bug/798081 to report my findings
<_mup_> Bug #798081: unicode tests failing since yuitest landing <build-infrastructure> <Launchpad itself:Triaged by sinzui> < https://launchpad.net/bugs/798081 >
<flacoste> sinzui: that would explains the error i got back from ec2test
<flacoste> sinzui: but it doesn't seem to tbe a problem from buildbot though?
<sinzui> flacoste, It wont be cause my rt is not closed
<flacoste> did somebody took care of the buildbot failure that happened earlier?
<flacoste> i see a testfix and a remerge from danilo
<flacoste> is that it?
<flacoste> sinzui: ah, ok
<flacoste> sinzui: so it's only a problem for ec2test
<flacoste> which means i can safely submit my branch
<sinzui> I can remove my ami so that the import is not used
<sinzui> flacoste, you certainly can submit
<deryck> sinzui, hi.  I get seg fault running YUI layer tests.  About 6-7 tests into the run.
<deryck> sinzui, anyone else reported this?
<bac> hi flacoste, i've got some questions about the fix for bug 776437
<_mup_> Bug #776437: Enable ARM builders for PPA via API <api> <escalated> <not-pie-critical> <oem-services> <ppa> <Launchpad itself:In Progress by bac> < https://launchpad.net/bugs/776437 >
<sinzui> deryck, I have never seen this
<deryck> sinzui, hmm, I wonder what's happening for me then.  I updated developer deps.  Anything else required?
<sinzui> no
<sinzui> but maybe I borked the deps
<sinzui> deryck, are you on maverick?
<sinzui> sorry natty?
<deryck> sinzui, natty, yes
<flacoste> bac: shoot, irc or voice?
<bac> skype?
<sinzui> deryck, does dpkg-query -s python-html5-browser libwebkitgtk-1.0-0
<sinzui> version: 0.0.3-0~15-6 for html5browser and
<sinzui> version: 1.4.1-0ubuntu1 for webkitgtk
<deryck> sinzui, hmmm, no, I get 1.3.13-0ubuntu2 for webkitgtk.
<cr3> is rabbitmq used for anything other than testing in launchpad?
<deryck> sinzui, but the html5browser is right.
<sinzui> deryck, that looks fine. I updated to oneiric so I am on a higher rev od webkit
<deryck> ah, ok
<sinzui> deryck, my tree is a day old
<sinzui> deryck, which is the last test you see run?
<sinzui> deryck, test_subscription.html is test 7 in my list
<deryck> sinzui, it's always the 7th test, regardless of which test is run 7th.  test_filebug_dupfinder.html or test_subscription.html usually.
<deryck> sinzui, they don't seem to be run in the same order to me.
<deryck> hmmm, yeah, a few runs here now confirm the order isn't the same, and the 7th test always dies.
 * sinzui runs tests with current tree
<sinzui> deryck, can you pastebin the fault?
<deryck> I didn't update today. this is update from yesterday.  wifi is very spotty here at velocity.
<deryck> sinzui, I'll try.  if pastebin will load.
<sinzui> I see test_lp_names timesout
<sinzui> so does test_bug_subscription_portlet
<sinzui> I expected the latter since it is a massive mosule
<sinzui> module
<deryck> sinzui, here's a couple runs:  http://pastebin.ubuntu.com/628105/
<sinzui> deryck, test_lp_names.js is in the old format. It never signals that the test is compelte
<sinzui> and  test_bug_subscription_portlet was reverted to the old format I think
<sinzui> So the two failures I see are bad tests
<sinzui> deryck, your failure is quite cryptic
<deryck> ah, ok.
<deryck> they run fine by themselves.
<deryck> but that should be easy enough to fix up, if they're old format.
<sinzui> deryck, your tests always die in /bugs I see
<sinzui> deryck, actually the test dies in the first bugs test each time
<sinzui> so maybe there is a teardown issue in the previous test suite (/app) or in bugs itself
<deryck> ah, interesting.  that makes sense.
<sinzui> deryck, can you confirm this package is installed dpkg-query -s gir1.2-webkit-1.0
<sinzui> I am also working on a fix for lucid's import for gtk from pygtk. I want to be certain you are getting the gir version of Gtk
<deryck> sinzui, yup, I have that installed.
<sinzui> deryck, I am going to fix the lucid ec2 image first, then look at your segfault.
<abentley> sinzui: Also, when html5browser is imported, it tries to inisialize gtk, and if that fails, it raises a RunTimeError, which breaks some of the TwistedJobRunner tests.  I'm moving the import to YUIUnitTestCase so we don't start gtk when we don't need it.
<deryck> sinzui, ok, thanks.  not meaning to create work for you.  just trying to work out what's up.
<sinzui> deryck, this isn't a distraction. I really want this test framework to rock
<sinzui> abentley, thanks
<sinzui> I was pondering the same to solve the encoding breakage in gtk. This is not an issue with modern code that  imports from git.
<sinzui> maybe I should send another day trying to make lucid use gir properly
<abentley> sinzui: we solved it the old-fashioned way in bzr-gtk.
<abentley> sinzui: reload(sys); sys.setdefaultencoding()
<sinzui> abentley, that is what I added to lucid's import of gtk
<sinzui> abentley, gtk is only used in two Browser() methods. would moving the import into those mthods make eveyone happy?
<abentley> sinzui: That would make me happy, but I suspect it would not fix the unicode issues.
<sinzui> abentley, the unicode fix is separate
<sinzui> the unicode issue is only lucid
<sinzui> I think the issue you bring up affects natty/oneiric
<abentley> sinzui: It only affects systems where html5browser is installed, at least.
<sinzui> abentley, bzr-gtk does not work with gtk3.
<sinzui> I will learn more about how bzr-gtk solved the issue soon since I am creating a gtk3 branch
<sinzui> abentley, I do not see anything special about importing gtk in bzr-gtk
<sinzui> abentley, what twisted thing can I run in Lp to see the issue?
<abentley> sinzui: bin/test -t test_timeout_short
<sinzui> thanks
<abentley> sinzui: the bzr-gtk code is wrapped around the test suite in /__init__.py
<sinzui> hmm, natty does not use pygtk
<sinzui> oh, I love the segfault though
<sinzui> abentley, moving the import of html5browser into the one testcase that uses it is much easier than changing html5browser. importing anything form gir seems  to cause the problem
<LPCIBot> Project windmill-devel build #239: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/239/
<lifeless> morning
<abentley> sinzui: I agree.  Hate modules with side effects on import.
<abentley> lifeless: morning.
<lifeless> sinzui: perhaps we should rollback the yui layer until these issues are fixed on a branch ?
<lifeless> sinzui: ah, catching up with mail; I see its not run but is an import side effect
<lifeless> flacoste: hi
<flacoste> hi lifeless
<lifeless> flacoste: reviews: reducing inventory, OCR drive-to-zero thread: I'd like your opinion on us adding 'drive nearly-there non-paid-launchpad-hacker patches to zero by helping' to the CHR collection-of-duties; and 'please put things which are "tweaks" into state approved, and things which need rework into "WIP"
<lifeless> (the last two items being for reviewers)
<flacoste> lifeless: -1 to adding to CHR
<lifeless> flacoste: oh, and OCR to officially drive the 'needs-review' collection to zero
<flacoste> OCR focus makes sense though
<flacoste> i can reply to the thread
<lifeless> flacoste: so the reason I suggest the rotating squads get the helping aspect is because OCR is perhaps too short a turnaround
<flacoste> i see
<lifeless> flacoste: 5 hour test run + followup : there will be lots of person-helping-churn if we ask OCR to do it.
<flacoste> indeed
<lifeless> flacoste: I would *like* to ask OCR to do it, but not -yet-
<flacoste> i'm still -0 on adding maintenance rotation responsibility
<flacoste> we should ditch some if we want to add some
<lifeless> flacoste: so instead asking a squad thats on for a week to do it would at least allow for continuity.
<lifeless> flacoste: this is something we kindof have as a 'we should be doing' but its not structured well [because of the churn issue]
<lifeless> flacoste: so I'd argue we are reducing churn on an existing todo item, not adding a whole new item.
<flacoste> maybe, but i'm not sure that how it would be perceived
<flacoste> we are adding a line item to their responsibility list
<flacoste> now, the todo item is diffuse across the team
<lifeless> (and doesn't happen)
<lifeless> flacoste: ok, so concretely - we have 4 social things I think need to happen if we want to reach a 'merge proposals are reviewed quickly' state of nirvana; there are technical things we can do to change those social things but thats going to take longer to bring about.
<lifeless> flacoste:  we need to make both approved and needs review lists in the lp-project/+activereviews page get driven to zero without it being frustrating for the folk doing the driving (2 items)
<lifeless> flacoste: we need to move things that don't need another review when the next OCR starts to either WIP or approved or rejected (1 item)
<lifeless> flacoste: ok so 3 things :).
<lifeless> flacoste: how do we move forward with this ?
<lifeless> (by which I mean resolve the bugs in the current process one way or another)
<flacoste> the weekly reviewer meeting was the forum where these were resolved previously
<flacoste> we could call one to discuss this
<flacoste> bac: ^^^
 * bac reads
<sinzui> lifeless, I have a fix the the YUITestlLayer
<lifeless> sinzui: great
<sinzui> lifeless, I was hoping to make everything perfect, but that is not one branch :(
<sinzui> lifeless,  the issue is only in the latest image. I will delete my lucid image in a few minutes, and make another one.
<lifeless> great
<bac> lifeless, flacoste: we can have a reviewers meeting next week to discuss this.  i'm -1 on it becoming another CHR task.  point #3 is trivially done, just ensuring we remind people to do it.
<bac> i like the patch pilot idea but would like to see OCRs try to tackle it at the beginning of their day.  set a goal of trying to get one of the branches landed and see if we can drive it to zero
<flacoste> bac: we could also do this live in Dublin
<lifeless> bac: That seems like it ignores the friction around landing things
<bac> lifeless:  perhaps we ask the OCRs on maintenance to claim at least one during their shift and see it through over the next couple of days.
<lifeless> bac: we could; that is in tension with 'do project work' though
<lifeless> bac: and we have a lot of experience that says that project work really crowds out, well, just about everything else.
<lifeless> bac: I'd like us to learn from that experience
<bac> flacoste: if we can keep the discussion focused doing it in dublin would make sense.  i don't want another ramble-ass discussion on reviewing.
<flacoste> bac: agreed, we can have a 1h reviewers meeting with an agenda
<lifeless> flacoste: bac: I won't be there in Dublin; I will just raise these angles on the list thread and suggest we keep discussing it there.
<flacoste> ok
<bac> lifeless: ok.  thanks for bringing it up.
<lifeless> bac: its my job :)
<LPCIBot> Project parallel-test build #43: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/parallel-test/43/
<LPCIBot> Project windmill-db-devel build #397: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/397/
<allenap> sinzui: I've recently started getting http://pastebin.ubuntu.com/628166/ when I try to use bin/ec2 land. It references your ec2 image (516). Any ideas?
<sinzui> allenap, I deleted that image 30 minutes ago
<allenap> sinzui: Interesting! bin/ec2 images still reports that it's there.
<sinzui> allenap, That image has a version of html5browser that was changing the default encoding
<sinzui> allenap, okay, I deregistered the name
<sinzui> I think it is fixed now
<allenap> sinzui: Yes, bin/ec2 images is happy. Thanks :)
 * flacoste concurs
<sinzui> jcsackett, mumble?
<jelmer> mwhudson, hi
<mwhudson> jelmer: hey
<mwhudson> did i just leave the channel?  my machine was in a strange state when i got in to the office this morning...
<jelmer> you quit and rejoined twice, once because of "Remote host closed the connection", second time "Changing host"
<LPCIBot> Project devel build #812: STILL FAILING in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/812/
<jelmer> mwhudson: I made some progress on the svn import thing today, hopefully will have a fix up tomorrow
<jelmer> mwhudson, I was also looking at some of the workermonitor code
<mwhudson> ok so i guess x was running but just not displaying anything then :)
<mwhudson> jelmer: woo
<mwhudson> jelmer: does the problem have a simple description?
<jelmer> mwhudson, for the tags code?
<mwhudson> yeah
<jelmer> mwhudson: bzr-svn has a simple way to discover tags - it just looks at a portion of the log and keeps track of when tags were created and when they were added
<jelmer> mwhudson: usually it only does this for the fraction of the log related to the piece of history you're importing. So, about 5000 revisions
<mwhudson> ah, but if you haven't imported tags before, it chews over everything?
<jelmer> It's now look at the entire history to discover tags, probably because of the way we look at the tags before actually fetching anything now
<jelmer> the tag discovery also uses way more memory than it needs to, which never was much of an issue but we can very well notice it now.
<jelmer> mwhudson, some of the values in CodeImportResultStatus don't appear to be used. Do you know if they ever were?
<mwhudson> jelmer: no, i think that was a big dose of big design up front
<mwhudson> in some ways it would still be nice to distinguish if e.g. it's the pull from escudero that's failing
<mwhudson> but it's probably not worth the effort
<jelmer> mwhudson: I'll leave them in, I was just curious
<jelmer> mwhudson, I'm adding FAILURE_UNSUPPORTED_FEATURE and FAILURE_INVALID
<mwhudson> if you're looking for things to rip out, please remove CodeImport.updateFromData
<mwhudson> that was definitely a mistake :)
<mwhudson> jelmer: ah right, signalled via different exit codes?
<jelmer> mwhudson, yep, exactly
<mwhudson> cool
<jelmer> and determined by the exception classes
<mwhudson> what will failure_invalid mean?  connection refused, authentication required, that sort of thing?
<jelmer> mwhudson, yeah - ConnectionError, NotBranchError, PermissionDenied
<mwhudson> jelmer: something i've talked about before is that the review_status kinda represents two things: sort of an intention and a result
<mwhudson> i.e. reviewed == we think this should be imported
<jelmer> mwhudson: yeah
<mwhudson> failed == we can't
<mwhudson> so really, FAILURE_UNSUPPORTED_FEATURE in particular should result in some kind of reviewed/failed combo
<jelmer> yep
<jelmer> Hopefully that's the next step
<LPCIBot> Project windmill-devel build #240: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-devel/240/
<sinzui> deryck, I would like to know if you still get segfaults when the changes I landed in devel.
<deryck> sinzui, sure.  are they in devel now?
<sinzui> yes
<jelmer> mwhudson: I would also really like to get rid of notion of what VCS you're importing from (and whether it's foreign or a mirror), that seems harder to fix though.
<deryck> sinzui, ok, let me see if I can get an update here.
<mwhudson> jelmer: yeah
<lifeless> jelmer: welcome back to lp :)
<mwhudson> jelmer: really, the fact that Branch has columns to do with mirroring is wrong
<mwhudson> (but very understandable, given history)
<lifeless> IIRC it didn't use to; it got refactored to have them. IMBW
<jelmer> lifeless: :)
<mwhudson> before my time then
<deryck> sinzui, so I get further now.  12 tests in.  re-running to see if it's consistent.
<maxb> If someone has a moment, could you check whether devel r13249 is likely to become deployable before the weekend?
<sinzui> deryck, Since you get a segfault, is is still at the start of an new testcasde, such as bugs
<deryck> sinzui, no, it's a few test into bugs now.  and 12 does seems the consistent number, if 4 runs can be trusted.
<sinzui> deryck, what of layer=BugsYUI
 * deryck tries....
<deryck> sinzui, that doesn't run any tests.  I thought you went with a single YUI layer anyway?
<lifeless> sinzui: its probably fork() related
<sinzui> oh yes, that was the testcase name :)
<lifeless> sinzui: gui libraries and fork tend to have a hate-hate relationship
<sinzui> what for we fork?
<sinzui> Why do not not see it?
<lifeless> zope.testrunner forks whenever a layer that cannot be torn down finishes
<lifeless> the CA layer cannot be torn down
<lifeless> maxb: probability 0
<lifeless> maxb: its not through BB yet
<sinzui> maxb, I think It could happen, I so not think we will have more than 5 things that need qa
<sinzui> lifeless, is there no hope that matsubara-afk or Ursinha could do a release?
<lifeless> Revision 13240 can not be deployed: needstesting, Revision 13243 can not be deployed: needstesting, Revision 13244 can not be deployed: needstesting, Revision 13248 can not be deployed: needstesting
<lifeless> sinzui: doing a deploy on friday afternoon means we have no safety net
<sinzui> understood
<lifeless> sinzui: we'd be deploying 30 revisions - a lot of work.
<maxb> Right, thanks, that was what I was wondering - there's enough QA already pending before it that I shouldn't expect anything
<lifeless> sinzui: which increases the likelyhood of needing the safety net
<lifeless> sinzui: if we can get it done in the next couple of hours it should be fine - that would leave nearly 24 hours of losa coverage before EOW
<deryck> sinzui, so I got the bugs tests with -- ./bin/test -cvv --layer=YUI -m bugs -- and still get the seg fault.
<sinzui> I think qaing bug 772754 is a lot of work
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
<lifeless> sinzui: yes
<sinzui> deryck, thanks. I think I need to add something brutal to the layer. deryck I suspect the issue may related to timeouts. Those tests are really slow
<deryck> sinzui, yeah, I was wondering that, too.  My new branch speeds them up, but maybe not enough.
<sinzui> deryck, The test case does handle the 30 second timeout, but something else might be deciding hat the test is taking too long to rrin
<deryck> all that dom redrawing is super slow.
<sinzui> deryck, all tests passed for my in 43 seconds 30 minutes ago
<sinzui> But I can see that the subscription tests are monsterd
<sinzui> monsters
<deryck> yeah, they're pretty fast for me, too.  I was just trying to make them even faster.  cleaning up some of that waits stuff.
<sinzui> rock
<deryck> the subscriptions tests are slow because of all the dom manipulation.
<deryck> that is slow and ugly for users too.
<sinzui> There are a lot of testcase in those modules too.
<deryck> yeah, agreed.  they are long tests.
<deryck> ok, back soon.  break time and changing session rooms.
<jelmer> Hmm, are those OOPSes for code imports still actually generated (and thus Critical bug worthy) ?
<mwhudson> bah, my login to devpad has been disabled again
#launchpad-dev 2011-06-17
<wgrant> lifeless, sinzui: I was planning to do a deployment this morning.
<wgrant> But then a certain monstrous completely un-QAable undisable hopeless merge failure buggy criss-cross branch landed.
<wgrant> We won't be deploying for a few days :/
* wallyworld_ changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld (*jtv) | Critical bugs: 202 - 0:[######=_]:256
* wallyworld_ changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 202 - 0:[######=_]:256
<wallyworld_> wgrant: ^^^^^^ which branch out of interest?
<wgrant> It's tempting to roll it back on principle.
<wgrant> There are bugs.
<wgrant> Not fatal if it was flagged.
<wgrant> But...
<wgrant> It's not.
<StevenK> Wasn't it rolled back once already?
<wgrant> Yes.
<wgrant> Yay
<wgrant> It crashes in Firefox 3.5, at least sometimes.
<wgrant> :/ this is a big UI change.
<wgrant> No longer obvious how to subscribe, and you can't tell who is directly subscribed to the bug...
<LPCIBot> Project parallel-test build #44: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/44/
<LPCIBot> Project windmill-db-devel build #398: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/398/
<lifeless> wallyworld_: the bugsubscription one
<wallyworld_> the 12000 line one i assume
<StevenK> wgrant: Can haz link to notify() bug?
<wgrant> StevenK: There doesn't seem to be one. Check #launchpad from 21:48 yesterday.
<LPCIBot> Project db-devel build #642: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/642/
<lifeless> wgrant: if its buggy, its a rollback
<wgrant> lifeless: The ordering is apparently arbitrary and the JS crashes when not logged in.
<wgrant> And it completely changes the UI of a feature we announced a week ago :/
<lifeless> crashing when not logged in is a pretty big blocker
<wgrant> It's not a regression from what we have now.
<wgrant> Ohhh, yes it is, actually.
<wgrant> Since at least now the direct subscribers are shown.
<lifeless> wgrant: crashing is a regression
<lifeless> wgrant: the UI aspect is deliberate AIUI. IMBW
<wgrant> lifeless: Deliberate, but untested and a break from what we announced :/
<huwshimi> lifeless: ITMFTFOWTFYW
<lifeless> huwshimi: ?
<huwshimi> lifeless: It takes me forever to figure what you write
<huwshimi> lifeless: Sentences like this: "lifeless: wgrant: the UI aspect is deliberate AIUI. IMBW"
<lifeless> huwshimi: sorry !
<huwshimi> lifeless: haha, it's ok
<huwshimi> lifeless: Most people on Launchpad do it, I just spend half my time opening a browser and googling what everyone is saying
<wgrant> s/Launchpad/IRC/
<huwshimi> wgrant: True
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 205 - 0:[######=_]:256
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 205 - 0:[######=_]:256
<LPCIBot> Project windmill-devel build #241: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/241/
<wgrant> StevenK: Any luck with the notifications?
<StevenK> wgrant: I've been a little distracted, sorry. Still digging.
<StevenK> wgrant: Unsigned changes file, since it comes from a buildd, so it adds Changed-By.
<wgrant> StevenK: Ah.
<wgrant> StevenK: How did this not happen before?
<StevenK> I suspect the old code only did that if it wasn't a PPA.
<wgrant> Probably.
<wgrant> I also suspect we need more unit tests.
<StevenK> wgrant: I think the bug is firmly in get_recipients(), I was just going to read the old code
<StevenK> wgrant: Hm, the bug seems present in the old code too
<wgrant> StevenK: But I know it didn't happen :/
<wgrant> StevenK: Possibly archiveuploader just didn't notify?
<wgrant> I'll look.
<StevenK> But I didn't change that bit!
<StevenK> wgrant: What I mean is _getRecipients() always added Changed-By if packageupload.signing_key is None
<wgrant> Yeah.
<wgrant>         # If this is not a PPA, we also consider maintainer and changed-by.
<wgrant>         if self.signing_key and not self.isPPA():
<wgrant> StevenK: From the original code.
<StevenK> wgrant: But look above that
<StevenK> wgrant:         debug(packageupload.logger,
<StevenK>             "Changes file is unsigned, adding changer as recipient")
<StevenK>         candidate_recipients.append(changer)
<wgrant> Fair point.
<wgrant> Oh.
<wgrant> sec
<wgrant> Hm.
<wgrant> The buildds override Maintainer, but not Changed-By.
<wgrant> So Changed-By should be the source Changed-By... hm.
<wgrant> This will all have to be rewritten soon anyway :/
<wgrant> 2011-06-17 00:49:35 DEBUG     Subject: [PPA stackapplet-dev-stackapplet] stackapplet_1.5+147~maverick1_source.changes rejected
<wgrant> 2011-06-17 00:49:35 DEBUG     Sender: Ubuntu Installer <archive@ubuntu.com>
<wgrant> 2011-06-17 00:49:35 DEBUG     Recipients: Launchpad Beta Testers <noreply@launchpad.net>
<wgrant> what
<StevenK> Another rewrite? :-/
<wgrant> StevenK: For DDs, yes.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #813: FIXED in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/813/
<wgrant> StevenK: Once we're copying between primary archives.
<wgrant> StevenK: Otherwise Debian people will probably be notified, which would be Bad.
<wgrant> And Ubuntu people would definitely be notified about Linaro builds.
<StevenK> Indeed
<StevenK> wgrant: So, shall we just remove the else bit that adds Changed-By for unsigned uploads?
<wgrant> StevenK: No.
<wgrant> StevenK: We need to work out what has changed.
<StevenK> Between a refactor a month ago, a rewrite 2 weeks ago, and 2 tweaks. :-/
<wgrant> Interesting. Logs start on the 19th, and the first occurence was the 30th.
<wgrant> It was broken then.
<StevenK> The 30th of May? That's the rewrite
<wgrant> So it was.
<wgrant> So.
<wgrant> I can't see why it wasn't notifying before :/
<StevenK> What do the DEBUG logs show for occurances of this problem prior to the rewrite?
<wgrant> There are none.
<wgrant> But they only go back 11 days earlier.
<StevenK> Bugger.
<wgrant> That exception is somehow not seen at all before then.
<wgrant> (which is not unthinkable; it also wasn't seen between the 3rd and the 10th of this month)
<StevenK> Yes, they don't happen very often.
<wgrant> https://lists.ubuntu.com/archives/ubuntu-reviews/2010-April/000281.html
<wgrant> That's a log from a year ago.
<wgrant> There are more recent ones, but that came up quickly in Google.
<StevenK> That doesn't trip the Changes file is unsigned, adding changer as recipien
<StevenK> Sigh. But you get the idea.
<StevenK> In fact, that doesn't send a notification at all.
<wgrant> I wonder...
<wgrant> It's possible that this is unrelated.
<wgrant> process-upload was originally run by b-m, with -M
<wgrant> So no mail was sent.
<wgrant> But then late last year that changed...
<wgrant> Could this have been happening for 6 months? Odd that we only heard about it now.
<StevenK> Indeed.
<wgrant> If only we kept logs for a while :/
<wgrant> What we keep is only 13MB :(
<wgrant> I don't understand why we don't keep pretty much all our logs for just about ever :(
<jtv> hi wallyworld_, I'm back.  Any reviews to mentor?
<wallyworld_> jtv: hi. none yet
<jtv> ok
<jtv> wgrant: is there something similar to the qa-tagging going on with bug statuses?  I notice you've marked a few bugs Fix Released before all their branches had landed.
<StevenK> wgrant: Since we don't have logs, we have no way to be certain -- but the work that divorced process-upload and b-m could have caused this? So I'm off the hook?
<wgrant> StevenK: I think so :/
<wgrant> jtv: Those didn't look like multi-branch bugs, and they weren't landed with [incr], so I didn't notice, sorry.
<wallyworld_> jtv: i'm off to pick up the kid from school. still no reviews. will check again later
<jtv> ok wallyworld_
<jtv> wgrant: I just knew there was something I wasn't thinking ofâ¦ incr.  It's getting altogether a bit finicky for me, this tagged-double-bookkeeping.
<wgrant> Indeed.
<jtv> OMG my branch made it in.  Starting at 07:30 pays off.
<jtv> I think that's about 8K of today's 9K lines landed.  :)
<nigelb> so, I have the sort bug still pending. where is a good place to add a util function?
<LPCIBot> Project parallel-test build #45: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/parallel-test/45/
<LPCIBot> Project windmill-db-devel build #399: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/399/
<StevenK> Does anyone know lib/lp/{bugs,code,soyuz}/notifications/ is used for? All three of them contain a zero-length __init__.py
<lifeless> stub: hi
<stub> Yo
<lifeless> stub: I did that lock experiment; the blocked alter table blocked out new readers.
<lifeless> session one: begin; set transaction isolation level serializable; select id from bug;
<lifeless> session two: being; alter table bug add column foo int;
<lifeless> session three: begin; set transaction isolation level serializable; select id from bug;
<lifeless> two and three both block
<lifeless> doing a rollbackl in session one - andtwo and three unblock
<lifeless> well, once two is committed.
<stub> ok. There might be something more complex going on (transactions working with related tables causing the ALTER TABLE to block).
<stub> Or not
<lifeless> stub: staging has displayed one of these typical 'fail to apply' all day
<lifeless> stub: hloeung and I have been gathering data and poking at it
<stub> But so far, I've never had success doing anything that needed to get a lock on Person on production. 'ALTER TABLE BugSummary ADD CONSTRAINT bugsummary__viewed_by__fk FOREIGN KEY (viewed_by) REFERENCES Person' for instance.
<lifeless> stub: the appservers and scripts were running
<stub> Has the lag check rung any alarms?
<lifeless> yeah
<stub> So the script is either blocked for some reason (eg. a DB patch trying to grab a lock on the Person table ;) ), or the daemons have fallen over, or replication is broken
<stub> Good the lag check is alerting us
<lifeless> https://pastebin.canonical.com/48634/
<lifeless> pending locks
<lifeless> https://pastebin.canonical.com/48634/
<lifeless> bah
<lifeless> https://pastebin.canonical.com/48632/
<stub> It seems to be currently running my bugsummary update. This is the first run on a production like system... the query might just be way slow.
<lifeless> stub: 5 hours ?
<StevenK> stub: 2208-00-0.sql is current, or is that the amalgamation of 2207? Happy to wait until you're done digging.
<lifeless> stub: if its the one I prepared, it runs into a temp table in a few minutes
<stub> There is no 2207 in our tree. launchpad-2208-00-0.sql is the current baseline.
<StevenK> stub: Right, but 2207 used to exist
<stub> lifeless: I didn't notice or decode your one (need to feed that stuff through a formatter).
<stub> yes
<LPCIBot> Project windmill-devel build #242: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/242/
<StevenK> stub: IE, where did 2207-* move to? 2208-00-0, or somewhere else?
<stub> It was either moved to 2208 or deleted
<stub> annotate on 2208 should give you the revno
<StevenK> I'd like to drop a view from 2208-00-0
<StevenK> (Not yet, mind)
<stub> That needs a DB patch. You don't modify the baseline - it is a snapshot of production at a point in time.
<StevenK> stub: Okay. So for an ec2 run (for instance), the view will get created and then removed?
<stub> If you hack the baseline, it will never get applied to production
<stub> Yes
<StevenK> stub: Okay, thanks.
<stub> Until we rollup the next baseline.
<stub> (probably next month - we are in the 70's already)
<StevenK> Bring on 2209
<stub> Any particular reason? Applying those patches takes about a second.
<StevenK> No, just forward progress :-)
<stub> Going back to 1 is backwards progress ;)
<StevenK> Haha
<stub> lifeless: So it is my bugsummary population query running. Possibly an earlier one was slow - I won't get timings until (if) it completes.
<lifeless> stub: no, it was that one
<lifeless> just digging up the query I had
<stub> lifeless: We can either wait or kill it - things should rollback fine at the current point in time.
<lifeless> stub: I'd like to debug whats up with it
<stub> Its chewing 100% of a core.
<stub> the EXISTS always is dangerous to performance.
<stub> Your version had that?
<stub> Anyway - we can fix it directly in the existing patch since it hasn't landed yet...
<lifeless> yeah
<lifeless> https://bugs.launchpad.net/launchpad/+bug/793848/comments/4
<_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts <dba> <timeout> <Launchpad itself:Triaged by stub> < https://launchpad.net/bugs/793848 >
<lifeless> had my query
<stub> But the cpu utilization means it isn't blocked.
<lifeless> yah
<stub> Maybe something stoopid like a missing join condition
<lifeless> so we need to diff the two queries
<lifeless> stub: it was blocked for a while
<lifeless> stub: was waiting for a lock on the sl_event
<stub> Some other process inserting data that needed to be replicated.
<lifeless> stub: it looked like a slon had started up badly and was deadlocking itself
<stub> lifeless: So how long has this patch been running do you think?
<lifeless> stub: https://pastebin.canonical.com/48632/
<lifeless> stub: submitted 11 hours or so ago
<lifeless> postgres 32012 99.9  2.2 2280364 726944 ?      Rs   Jun16 567:55  \_ postgres: slony lpmain_staging [local] INSERT
<stub> lifeless: looking at the slony stuff being executed, it was blocked then on the event lock
<lifeless> stub: so, tha db patch has used 567 cpu minutes
<stub> One slony process had been waiting 4 minutes
<stub> ok, and nothing else would have had that process before hand.
<lifeless> right
<stub> (won't be the case after we install pgbouncer)
<lifeless> stub: slonik will be direct won't it ?
<stub> True
<stub> lifeless: So yeah, the queries are different where it matters. I created a new CTE with clause for bugs_fixed_upstream, and that obviously sucks.
<lifeless> bbs, grabbing dinner
<stub> Yer, so the CTE will be executed once for the query (which is why we normally use them), then every insert is doing a full scan on that set to see if the EXISTS clause works. Your version with it inline will be doing an index lookup.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #643: FIXED in 5 hr 47 min: https://lpci.wedontsleep.org/job/db-devel/643/
<lifeless> stub: yah
<lifeless> stub: you probably need to merge db-devel and push
<lifeless> stub: the diff looks horrible
<stub> Whoops... lp-land already away
<lifeless> ah well
<lifeless> we'll see what happens ;)
<stub> Pumped db-devel... must have had an older version?
<lifeless> something :)
<stub> It landed (!)
<stub> I think it is wierd because I ran bzr pump --from db-devel in my second pipe, so it had more db-devel than my first pipe
<stub> Landed diff is sane anyway
<stub> So how to we get this on staging?
<lifeless> 4 hours for it to get through BB
<stub> I can kill the existing job which should fail safely
<lifeless> its a bit hacky
<lifeless> but I'd kill the existing slonik run
<stub> Given it has taken so long so far to apply the patch on the master, I'll assume we can't be arsed waiting for it to complete and run a second time on the slave.
<stub> yup
<lifeless> drio your new file in
<lifeless> and do the deploy thing yourself
<stub> select pg_cancel_backend(32012);
<stub> That should have killed the existing run
<stub> losaping: Can we kick of a staging code update without the code update?
<lifeless> stub: what do you mean ?
<stub> After modifying the live tree, we don't want to have the scripts blat it
<lifeless> stub: so, disable the cron job for a few minutes?
<lifeless> stub: oh, I see where you are going; ignore me
<stub> I can try running upgrade.py manually live. Not sure exactly what happens :)
<lifeless> stub: as long as we avoid staging-restore auto running the old patch I'm happy ;)
<adeuring> good morning
<stub> I can't see how to override the code update.
<stub> apart from directly editing the script
<lifeless> +1?
<lifeless> stub: think you've got the patch right now ;>
<stub> I'm sure there are other exciting ways I can screw up
<lifeless> Was Not A Challenge! :)
<LPCIBot> Project windmill-db-devel build #400: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/400/
<bigjools> stub: did you benchmark 2208.75.0 on staging yet?
<stub> not the new version, no
<bigjools> running for 24h on dogfood so far :/
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv), adeuring | Critical bugs: 205 - 0:[######=_]:256
<rvba> wgrant: Hi, I know you're in fire-fighting mode right now ... but if you have time after that (I know it's going to be late for you), I'll be happy if you could take a look at my (never-ending) multi parent thing.
<rvba> wgrant: I you don't have time, that's really fine and we could talk about it on monday.
<lifeless> bigjools: yes, kill it
<bigjools> lifeless: ?
<bigjools> you are reverting it?
<lifeless> no, stub landed a fix
<lifeless> its in db-devel
<bigjools> grar
<wgrant> rvba: I'll try to get to it tonight, but no promises :/
<rvba> wgrant: that's fine, thanks a lot.
<lifeless> stub: whats the state of staging, given you're going ?
<stub> Needs a losa
<lifeless> ping for one in -ops ? ;)
<stub> I've patched the tree but not sure how to continue. I could improvise... but I'd rather someone who knows the correct processes does it
<stub> losaping here from ages ago...
<lifeless> they don't watch -dev
<lifeless> tom requested that we only do such pings in -ops
<LPCIBot> Project windmill-db-devel build #401: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-db-devel/401/
<jml> good morning
<bigjools> stub: how do I tell the difference between the old and new patch?  I just diffed DF's version against the one in db-devel and it's the same still.
<bigjools> jml: morning
<lifeless> bigjools: the fixedtasks CTE should be gone
<lifeless> line 78 of the patch
<bigjools> lifeless: 78 for me says "OR ("
<lifeless> bigjools: did you pull db-stable or db-devel
<bigjools> devel
<lifeless> *db-devel*
<bigjools> yes db-devel
<lifeless> what revno do you have?
<bigjools> it's hardly likely to be in devel is it :)
<bigjools> 10679
<bigjools> is the one I am using now, but it says no diff to 10685
<wgrant> danilos: I played around with the new subscription listing thing this morning, and noticed a couple of things. The JS crashes for anonymous users, the subscriptions don't seem to be sorted, and I can't work out who is directly subscribed.
<lifeless> bigjools: I see a huge diff
<danilos> wgrant, well, everyone except the "Maybe notified" section are direct subscribers
<wgrant> danilos: Oh.
<wgrant> That's less than obvious.
<danilos> wgrant, some of them might choose not to receive some emails
<wgrant> I assumed they were just people who had selective structural subscriptions :(
<danilos> wgrant, you can have selective direct subscriptions as well
<wgrant> True.
<wgrant> So "Always notified" is sort of a misnomer?
<danilos> wgrant, no, always notified is those who are always notified (i.e. "receive all"
<danilos> wgrant, see eg. https://bugs.qastaging.launchpad.net/ubuntu/+source/network-manager/+bug/152754 for example of other sections
<_mup_> Bug #152754: NetworkManager fails and crashes after resume from suspend <network-manager (Ubuntu):Incomplete> < https://launchpad.net/bugs/152754 >
<danilos> wgrant, (you can see my different subscriptions on different levels)
<wgrant> danilos: aaaaaa
<wgrant> So "Maybe notified" is all the structsubs? I seeeee.
<wgrant> Hmm.
<danilos> wgrant, yeah, "maybe" is all struct subs, all dupe subs
<danilos> wgrant, for your own subscriptions, you are supposed to use "edit bug mail" link instead
<bigjools> lifeless: weird, a bzr ancestor diff shows nothing, but bzr missing shows 6 revisions
<wgrant> danilos: I've never liked that link... shouldn't it be "Edit subscriptions" or something that doesn't sound like it's editing emails? :/
<danilos> wgrant, basically, this is something that's hard (and potentially expensive) to display in succint form with enough details (while staying correct)
<wgrant> But yeah.
<lifeless> bigjools: uhm an ancestor diff should show nothing
<lifeless> bigjools: because your current rev is the common ancestor
<wgrant> I understand it's hard to calculate, yes. But it's not obvious what it's actually showing.
<bigjools> gah, roight
<bigjools> lifeless: I am stupid
<lifeless> :P
<danilos> wgrant, yeah, I think the attempt was to clarify that this is about controlling the email you get from that particular bug
<lifeless> bigjools: didn't you just implement a vcs layer for distros?
<danilos> wgrant, and I think we just copied whatever was there already on the pillar pages which was "Edit bug mail"
<bigjools> I'd like to think not :)
<danilos> wgrant, I do see your point, but maybe you are thinking too much in terms of implementation, rather than trusting what the text says?
<LPCIBot> Project parallel-test build #46: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/parallel-test/46/
<danilos> wgrant, "Notified of all changes" are, indeed, people/teams who get all the bug mail for this bug
<wgrant> danilos: Links like that have always talked about subscriptions before, or "Subscribe to bug mail", not "Edit bug mail"
<wgrant> Since editing bug mail doesn't really make sense.
<lifeless> danilos: wgrant: was saying earlier that the js crashes for anonymous users
<danilos> wgrant, hey, I am not going to argue over that, I don't like "Edit bug mail" anymore than you do
<wgrant> Heh
<wgrant> k
<danilos> lifeless, yeah, I'm going to look into it
<wgrant> But yes, what lifeless said.
<danilos> wgrant, ack
<lifeless> danilos: if it needs fixing we should roll the branch back
<wgrant> ... and land the unrelated pieces separately? :)
<lifeless> danilos: we have 30! revisions to deploy already, and more will queue up over the weekend
<lifeless> wgrant: orthogonal discussion
<wgrant> lifeless: Not quite... I rolled out up to that rev this morning.
<lifeless> wgrant: danilos: grah sorry; stale info
<wgrant> There's only 8 in the queue ATM
<danilos> lifeless, if it needs fixing, we'll fix it in the next few hours
<danilos> lifeless, if subscribers list doesn't render for anonymous users, it's not a big deal since none of the JS works for them anyway
<danilos> lifeless, iow, that should not block the rollout, but it should be fixed asap
<wgrant> Why shouldn't it block the rollout?
<danilos> lifeless, I was also hoping to get more QA on this and try to deploy it on Monday instead (and any such problems should be fixed by then)
<danilos> wgrant, why should it? I don't consider it a critical feature, and I value the time it takes to get something landed much more than this particular problem (if it's not as severe, like I said)
<wgrant> danilos: Some browser react to JS errors with violent popups.
<wgrant> And we treat JS errors as OOPSes now.
<wgrant> I think we'd consider it a blocker if the most commonly accessed page on LP OOPSed for anonymous users.
<danilos> wgrant, right, and fixing the problem is probably going to take 30 mins as soon as I get to it, whereas just doing a revert is going to take me 10-20 mins
<wgrant> I don't place much value on rushing a 12000 line branch through, TBH.
<wgrant> It deserves a lot of caution.
<danilos> wgrant, I agree, and I want to exercise some
<danilos> wgrant, but I want to be realistic as well
<allenap> stub: Do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5?
<stub> allenap: Nope. Last minute scramble for a plane.
<allenap> stub: Okay, have a good trip :)
<allenap> lifeless: I know it's silly o'clock for you, but do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5?
<lifeless> allenap: lets get an analyse for the 3 second one off of prod, after gnuoy finishes fixing staging
<allenap> Okay, cool. I have it ready on devpad.
<bigjools> lifeless: do you know how much quicker that new db patch is?  30 minutes and counting on DF so far.
<lifeless> 157 seconds when it ran on a temp table
<lifeless> are you sure you got the new one ?
<bigjools> yes
<bigjools> :)
<lifeless> allenap: ok so
<lifeless> 6.6 seconds
<allenap> Ouch.
<lifeless> see the actual time
<lifeless> hash join on spr to dsd
<lifeless> 351K rows
<danilos> adeuring, wallyworld_: hi, I need a review of a short branch (151 line), any takers? :)
<lifeless> the stats suggested to postgresql it would see 2.2K rows
<adeuring> danilos: I'll look
<lifeless> could be a stats issue
<danilos> adeuring, thanks
<allenap> lifeless: Can we get stats recalculated online?
<lifeless> yeah
<lifeless> analyze tablename;
<lifeless> allenap: how many rows are you expecting to return ?
<allenap> lifeless: Around 1:1 with DistroSeriesDifference rows.
<allenap> So around the same as the big IN (5, 7, ...) clause.
<lifeless> right, so the plan is problematic, or the model is problematic or the query is
<allenap> Hehe.
<LPCIBot> Project windmill-devel build #243: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/243/
<allenap> lifeless: Is an analyze cheap enough that we could try that first? DistroSeriesDifference until recently was empty for example, that might be a candidate for refreshing stats?
<allenap> Although that doesn't appear in the bad part of the plan :-/
<lifeless> yes
<lifeless> bigjools: in other news, DF sucks.
<bigjools> lifeless: tell me something I don't know
<lifeless> staging appears to have run the update
<bigjools> how slow?
<lifeless> times will be in the logs
<bigjools> what, you're making me look myself?  tsk :)
<lifeless> 11pm and I'm sick.... so yes :)
<lifeless> also woo
<lifeless> 27ms stats portlets here we come
<bigjools> which file has the timings?
<lifeless> the staging-restore-$date IIRC
<bigjools> 2011-06-17 07:24:24 ERROR   slonik script failed
<bigjools> Fri Jun 17 07:24:24 UTC 2011 ERROR: Failed to run upgrade.py
<lifeless> it won't have rsynced yet so you'll need to pull it over by hand
<bigjools> k
<lifeless> thats the run we canned
<adeuring> danilos: nice small branch, r=me
<danilos> adeuring, thanks
<bigjools>  lifeless: so it's far easy to look in LaunchpadDatabaseRevision.... ;)
<bigjools> 7 mins on staging it seems
<lifeless> allenap: so not stats
<allenap> No.
<lifeless> 34*10547
<lifeless> 358598
<lifeless> that explains the 351K rows I think
<lifeless> 10K spph * 34 archive
<allenap> I wonder what we can do to narrow the spph set down.
<lifeless> well
<lifeless> its a cross join atm
<lifeless> so one thing to try is an inner join
<lifeless> rather than a constrained cross join
<allenap> lifeless: Okay, I don't know enough to know how to do that, but I can go and find out. In fact, I'd be happy to do that. But if you want to see some swifter progress you'll have to help me out on that one :)
<allenap> lifeless: Okay, I think I know how.
<lifeless> query in ~robertc on carob
<allenap> lifeless: Thanks :)
<lifeless> ah, I misread
<lifeless> problem doing this tired
<lifeless>                            ->  Index Scan using sourcepackagerelease_pkey on sourcepackagerelease  (cost=0.00..0.87 rows=1 width=8) (actual time=0.012..0.013 rows=1 loops=351423)
<lifeless>                                  Index Cond: (sourcepackagerelease.id = sourcepackagepublishinghistory.sourcepackagerelease)
<lifeless> thats the 350K
<lifeless> spr
<lifeless> I have to crash
<lifeless> however
<lifeless> some thoughts:
<lifeless>  - we know the distroseries, so can constrain the query further
<lifeless>  - there are only 34 primary archives, so we can eliminate the archive join if we want (may not matter)
<lifeless> and that would let us drop the distroseries join too
<wgrant> jelmer: Thanks!
<allenap> lifeless: Thanks for your help :) Have a good weekend.
<jelmer> wgrant: For what, wasting your morning ? :)
<lifeless> erm, no first analysis was right I think
<lifeless> we're expanding out through archive
<lifeless> because you have this
<lifeless> constraints - join - join - join - join - constraints
<jelmer> wgrant: Sorry about the importd upgrade trouble, I wonder how neither of us had caught this on staging though.
<lifeless> each join, even inner ones, will multiply until it can constraint
<wgrant> jelmer: For fixing that :) (and it was my afternoon/evening, not my morning!)
<wgrant> jelmer: Not sure... I wonder if it lied about the format.
<wgrant> Possibly the import was new, but still had the old metadata, or something.
<wgrant> Anyway, was easily fixed.
<LPCIBot> Project windmill-devel build #244: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/244/
<danilos> wgrant, fwiw, the fix for the anonymous users JS is playing in ec2
<wgrant> Although it was a major panic moment before I realised there would probably still be a copy on escudero...
<wgrant> danilos: Ah, great! That was quick.
<danilos> wgrant, if I did more integration testing for JS stuff (I did loads of JS unit tests, with the ratio of tests to code being roughly 2:1 in number of lines), I would not have let it happen; unfortunately, I don't and I did
<danilos> wgrant, so, I hope to start on more windmill tests if they are the way to go for this
<lifeless> nn, must go
<jelmer> wgrant: Yeah, I can imagine..
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugs: 205 - 0:[######=_]:256
<matsubara> morning
<allenap> Sorry lifeless, had to go. My missus had been badgering me for half an hour already.
<LPCIBot> Project windmill-devel build #245: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/245/
<jml> matsubara: good morning
<matsubara> hi jml
<danilos> matsubara, good morning as well :)
<jml> bigjools: can  we have a call at 4pm to touch base about derivative distros?
<matsubara> hey danilos
<matsubara> danilos, btw, did that branch land?
<danilos> matsubara, fwiw, new subscription stuff is up on qastaging, there is a bug with anon users, but the fix is playing in ec2 already
<danilos> matsubara, yeah, it did, it should be good to test
<matsubara> danilos, cool. I'll give it a try today.
<danilos> matsubara, excellent, you can also report any problems you find on bug 772754, I'll be reading it
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
<matsubara> will do
<danilos> matsubara, thanks!
<matsubara> danilos, np. Yesterday you mentioned r13248 as the one you were waiting. Was that a typo or that's the one with the subsequent fix playing on ec2?
<danilos> matsubara, no, that's the one which includes everything but the fix for anonymous users (basically, there is a JS error for anonymous users), which is in bug 798622
<_mup_> Bug #798622: JS failure for anonymous users on bug page <story-better-bug-notification> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/798622 >
<danilos> matsubara, so, qastaging should be good for QAing for logged in users (I already did some, as did wgrant: he found the anon user problem)
<matsubara> danilos, there's no r13248 on qastaging. the one available is r13243 linked to bug #772754 so I assume that's the right number not 13248 :-)
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
<danilos> matsubara, qastaging for me has "r13251" in the footer
<matsubara> danilos, I'm looking at the dashboard. that might be a bit outdated
 * danilos looks at his dashboard, and thinks redline at 7000 rpm is too low
<matsubara> actually, I'm confusing the qastaging with what's in the dashboard
<danilos> matsubara, I am thoroughly confused since I don't know what the dashboard is :)
<matsubara> 13243 is block the rollout and qastaging has 13251
<matsubara> danilos, http://lpqateam.canonical.com/devops/
<danilos> matsubara, ah, nice
<danilos> matsubara, right, so we still haven't marked it as qa-ok because I'd prefer to get a little bit more QA done first
<matsubara> I see now, should have looked at the development-stable report. I can see that 13243 was reverted by 13247 and then 13248 landed afterwards
<matsubara> no worries. I unconfused myself and things should be ok to start now. thanks!
<danilos> matsubara, cool, thanks
<LPCIBot> Project windmill-devel build #246: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/246/
<bigjools> jml: not feeling too well but if I am here, yes
<jml> bigjools: cool. thanks.
<LPCIBot> Project windmill-devel build #247: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/247/
<bac> hi adeuring -- i've just grabbed wallyworld_'s branch.  you're not already looking at it are you?
<adeuring> bac: no
<LPCIBot> Project parallel-test build #47: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/47/
 * jml ducking out for a bit, back in ~20m
<nigelb> bigjools: heh, lol That as a fun commit :-)
<nigelb> *was
<bigjools> jml: you no wanna call then?
<bigjools> nigelb: the comments are hilarious
<flacoste> james_w: i again need your help to qa a change, could you confirm that you are able to call setBranch with the UPDATES or RELEASE pocket on lucid/maverick on qastaging? pretty please
<jml> bigjools: I do, just need to get some other stuff out of the way first.
<LPCIBot> Project windmill-db-devel build #402: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/402/
<nigelb> bigjools: Yup, with new memes!
<sinzui> jcsackett, mumble?
<jcsackett> sinzui: sure.
<sinzui> jcsackett, If you cannot hear me I think I need to restart mumble
<jml> bigjools: ok. pie safely consumed, call can proceed
<jcsackett> sinzui: i heard you. seems you could not hear me. i'm restarting.
<bigjools> the pie should be on ya face
<jml> bigjools: 200 bugs stand between my face and pie.
<bigjools> jml: gimme a couple of mins
<jml> also, a hot steak & stilton pie would be inappropriate for face-planting
<sinzui> maybe I can get empiphay to work
<jcsackett> sinzui: i'm having a connection hiccup.
<sinzui> empathy say I have sip, but I cannot use it, maybe you can call me
<jcsackett> sinzui: i hear you, it seems you cannot hear me. i will try sip.
<flacoste> maxb: the fix for the importer is deployed to qastaging, i asked james_w to confirm that it works, maybe you can do it also?
<maxb> flacoste: No, I'm not a core-dev (or any sort of uploader actually)
<jcsackett> sinzui: lost call. recalling in a moment.
<cr3> what part of launchpad adds the utc timezone to a datetime between getting it from the database (which always has "without time zone") to the restful interface (which has the timezone as "+00:00", therefore explicitly defined somewhere)?
<bigjools> jml: sorry.  what is your call platform of choice?
<jml> bigjools: voip
<jml> bigjools: skype for second preference
<bigjools> gah
<jml> (am I allowed to have a second preference?)
<bigjools> pulseaudio keeps usurping Twinkle's alsa connection
<bigjools> and twinkle doesn't know about PA
<jml> ahh, I see.
<bigjools> I hate PA
<jml> pulseaudio is great in natty for me
<jml> I guess that's because everything is using it.
<bigjools> I hate it because it limits my options
<bigjools> I am on skype
<jml> ok
<jml> bigjools: have you seen https://dev.launchpad.net/Projects/DerivativeDistributions ?
<cr3> nevermind folks, I found that it's UtcDateTimeCol that sets the tzinfo. oddly though, that doesn't seem to behave the same as non-sqlobject derived storm variables which only set the tzinfo when writing to the db
<LPCIBot> Project windmill-devel build #248: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/248/
<LPCIBot> Project windmill-devel build #249: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill-devel/249/
<bigjools> bye, have a good weekend everyone
<LPCIBot> Project db-devel build #645: FAILURE in 5 hr 35 min: https://lpci.wedontsleep.org/job/db-devel/645/
<LPCIBot> Project windmill-db-devel build #403: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-db-devel/403/
<fjlacoste> maxb: james_w confirmed the fix is working, I've changed the Debian distribution maintainers, so everything should be back to normal at the next nodowntime deploy
<fjlacoste> maxb: probably Sunday (antepodean Monday) as we don't roll-out just before the week-end
<maxb> Awesome, thanks, so on Monday we should be back in business
<fjlacoste> maxb: again, i'm very sorry for the breakage
<fjlacoste> and apologize for it
<maxb> Sometimes regressions happen - thanks for pushing through a fix promptly
<fjlacoste> thanks for your understanding
<LPCIBot> Project parallel-test build #48: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/48/
<fjlacoste> anyone knows what LostObjectError: means?
<fjlacoste> i'm writing a view test and getting this
<fjlacoste> wow, that's intersting
<fjlacoste> it happens because there is actually a validation error in the form
<fjlacoste> which resets the transaction
<fjlacoste> and then when then a storm attribute is accessed after the end of transaction
<fjlacoste> and boom
<bac> i'm trying to export a new class to the api and i'm getting:
<bac> UnknownEntryAdapter: No IEntry adapter found for IProcessorFamily (web service version: devel).  Encountered as a result of the entry interface <InterfaceClass lp.soyuz.interfaces.archive.IArchiveEntry_devel>, field 'enabled_restricted_families'.
<bac> normally that would be due to a circular dependency, but i've handled that (i think)
<bac> fjlacoste: you now what ^^ could be?
<fjlacoste> bac: that's because the IProcessFamily isn't exported
<fjlacoste> you need to import it into lp.soyuz.interfaces.webservice probably
<bac> fjlacoste: ah, but it is:  http://pastebin.ubuntu.com/628552/
<bac> er
<fjlacoste> it's not :-)
<fjlacoste> see lp.soyuz.interfaces.webservice
<fjlacoste> you need to import it in there
<fjlacoste> because only definitions in that file are processed for export
<bac> oh!
<bac> fjlacoste: you are a super-genius!  thanks!
<flacoste> abentley: is it possible to set the pre-requisite branch from lp-propose?
<abentley> flacoste: not directly, but it's done automatically if you use pipelines.
<flacoste> i didn't
<flacoste> is it possible to set the pre-requisite branch on the created merge proposal?
<flacoste> or i need to resubmit
<abentley> flacoste: You can set the pre-requisite branch on the created merge proposal by re-submitting.
<flacoste> thanks
<flacoste> bac: a review for you pretty please: https://code.launchpad.net/~flacoste/launchpad/bug-781993-2/+merge/65058
<flacoste> should be simple
<bac> flacoste: sure
<bac> flacoste: can you get rid of the conflicts?
<bac> flacoste: actually the conflicts don't show up in the MP.  looks good.
<flacoste> thanks for the review bac
<soren> Hi. The person listed as help contact in #launchpad isn't online and noone seems to respond there, so I'm escalating here. Hope you can forgive me. Uploads to PPA's don't work. I get "Changes file must be signed with a valid GPG signature: Verification failed 3 times: ['General error', 'General error', 'General error'] : Permission denied." Ampelbein in #launchpad pointed me to bug 757248.
<_mup_> Bug #757248: poppy-sftp's signature checking relies on long-term survival of a directory in /tmp <poppy> <qa-ok> <regression> <Launchpad itself:Fix Released by wallyworld> < https://launchpad.net/bugs/757248 >
<soren> ...which, interestingly, was marked "fixed in stable" exactly one month ago today.
 * soren <- saaaad panda
<flacoste> soren: i asked IS to check what's up
<soren> flacoste: ta very much.
<flacoste> soren: someone internal says that the upload should still be accepted
<flacoste> it is an avatar of the old bug
<lifeless> well
<flacoste> but it shouldn't reject your upload
<lifeless> it might be
<soren> flacoste: Ah, yes, you seem to be right. I didn't even bother checking. It looked pretty darn fatal to me :)
<lifeless> it does; we're going to have lots of folk double-upload and panic if this is happening again
<flacoste> indeed, soren can you file a new bug (one might already exists though)
<soren> flacoste: Sure
<flacoste> thanks
<soren> https://bugs.launchpad.net/launchpad/+bug/798957
<_mup_> Bug #798957: Uploads are seemingly (but not actually) rejected <Launchpad itself:New> < https://launchpad.net/bugs/798957 >
<lifeless> soren: perhaps your gpg key expired recently or something?
<LPCIBot> Project windmill-devel build #250: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/250/
<soren> lifeless: It started happening a couple of hours ago for both the automated openstack package uploader thingamajig and myself.
<soren> lifeless: ...and they both never expire. I checked :)
<lifeless> k
#launchpad-dev 2011-06-18
<LPCIBot> Project parallel-test build #49: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/parallel-test/49/
<LPCIBot> Project windmill-db-devel build #404: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/404/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #646: FIXED in 5 hr 42 min: https://lpci.wedontsleep.org/job/db-devel/646/
<LPCIBot> Project windmill-db-devel build #405: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-db-devel/405/
<LPCIBot> Project parallel-test build #50: STILL FAILING in 1 hr 14 min: https://lpci.wedontsleep.org/job/parallel-test/50/
<lifeless> allenap: oh hai
<LPCIBot> Project db-devel build #648: FAILURE in 5 hr 25 min: https://lpci.wedontsleep.org/job/db-devel/648/
<allenap> lifeless: Something I discovered with that query is that it's looking at a batch of 300 DistroSeriesDifferences. If the batch is reduced to the default (for +localpackagediffs) of 75, the query appears to run 10x faster. So, one solution is to split the query into smaller batches - say 50 - and union them back together.
<LPCIBot> Project windmill-devel build #251: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/251/
<LPCIBot> Project devel build #819: FAILURE in 5 hr 30 min: https://lpci.wedontsleep.org/job/devel/819/
<LPCIBot> Project parallel-test build #51: STILL FAILING in 1 hr 20 min: https://lpci.wedontsleep.org/job/parallel-test/51/
<lifeless> allenap: thats interesting; looks like the plan goes screwy ? - one cause for that can be large IN clauses
<lifeless> allenap: another approach there is to make a temp table, put the ids you want into that table, and then join against it.
<LPCIBot> Project windmill-db-devel build #406: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/406/
#launchpad-dev 2011-06-19
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #649: FIXED in 5 hr 45 min: https://lpci.wedontsleep.org/job/db-devel/649/
<lifeless> moin
<nigelb> moinmoin
#launchpad-dev 2012-06-11
<lifeless> lib/lp/app/templates/root-index.pt is hilarious
<lifeless>             <div id="homepage-featured" class="homepage-portlet"
<lifeless>                 tal:content="cache:anonymous, 1 hour">
<lifeless>               <tal:cache
<lifeless>                   tal:content="cache:public, 5 minutes" tal:omit-tag="">
<lifeless> yes, thats two conflicting cache rules in the same file, adjacent.
<mwhudson> is the latter nested in the former?
<lifeless> why yes, funny you should ask that.
<mwhudson> awesome
<lifeless> line 215,216,217,218 there
 * james_w waves to NZ
<lifeless> I thought we had something that injected values into memcache for the home page
<lifeless> but I cannae find it
<lifeless> hi james_w
<mwhudson> hi james_w
<mwhudson> lifeless: that sounds like something that would be suspiciously well designed for lp memcache
<lifeless> bitter bitter
 * james_w now has tarmac updating sourcedeps.conf for config-manager, so that staging can autorollout changes across any branch involved in a deployment
<james_w> the next target should be the latency before deployment-manager sees that
<lifeless> nice
<lifeless> I would like to drop update-sourcedeps and put LP back to config-manager
<james_w> yeah
<lifeless> since the functionality is basically identical, except that the sysadmins use config-manager :)
<james_w> I was pondering about replacing config-manager with bzr-builder the other day
<james_w> but I'm not sure there's any benefit
<lifeless> 6/1 1/2 of the other
<james_w> except for having one fewer codebase
<lifeless> well, neither is a superset of the other
<lifeless> so I don't think you'd achieve that in the first instance.
<james_w> the reason I was thinking of that is that I thought it would be useful to have the "only do it if something is changed" feature of bzr-builder
<lifeless> ah, so there is something like that in update-sourcedeps
<james_w> but talking with web-ops led me to believe that having tarmac update things rather than relying on that is a better way to go
<lifeless> should be straight forward enough to give config-manager a similar hinting facility
<wgrant> lifeless: Those memcache rules you quoted aren't necessarily wrong. They could be quite reasonable.
<lifeless> wgrant: orly
<james_w> the only other small benefit I could see is the "manifest" that bzr-builder writes
<wgrant> lifeless: And no, nothing injects. That was a proposal to remove HTTP access from the appservers.
<lifeless> wgrant: you mean if the inner one ends short
<james_w> that is valid input, so you could deploy exactly to production what was in staging by linking up the output of one to the input of the other
<james_w> but we are close enough to that already I think
<lifeless> wgrant: and even if it ends short, its wrong
<lifeless> wgrant: you have to have your *longest* cache times at the innermost context.
<wgrant> lifeless: It may be *misguided*.
<wgrant> lifeless: But it's perfectly valid.
<wgrant> lifeless: Why?
<wgrant> The top-level one caches for an hour for anonymous users.
<wgrant> Non-anonymous users get lower expiration times.
<lifeless> anyhow, gones now.
<lifeless> there are precisely two uses of memcache I'm a little wary of removing, but I'm willing to take the risk.
<lifeless> I may need to go back and feature flag this change, somehow.
<lifeless> will see
<wgrant> The frontpage blogpost one can't really be removed.
<wgrant> The others can.
<lifeless> that is one of them
<lifeless> and it can be
<wgrant> lol
<wgrant> Let's not call into a frequently down Wordpress on every RootObject:+index load,l please.
<lifeless> the other one is the global counts on RootObject:index.html
<lifeless> aieee pageidblindness RootObject:+help-blueprints for /srv/launchpad.net/production/launchpad-rev-15190/lib/lp/blueprints/help
<wgrant> lifeless: SELECT MAX(id) :)
<lifeless> wgrant: they are currently cached, I'm assuming there is some insanity lurking.
<wgrant> There is.
<lifeless> wgrant: so, we're meant to be using squid in the DC everywhere
<wgrant> Which is why you should SELECT MAX(id)
<wgrant> lifeless: Ha ha ha
<wgrant> lifeless: We do.
<lifeless> wgrant: we get sensible headers from this wordpress:
<lifeless>   Last-Modified: Wed, 06 Jun 2012 15:15:54 GMT
<lifeless>   ETag: "3fafd55d87f99a8d01f330c4402a3b75"
<wgrant> lifeless: Whether it's not broken or actually useful is another quesiton.
<wgrant> Oh
<wgrant> That's surprising.
<lifeless> wgrant: so other than a check with webops that appservers are actually using it
<lifeless> I'm not concerned.
<lifeless> it is 37K of html to parse.
<wgrant> lifeless: We also need to fix it to not crash if it can't retrieve it.
<lifeless> but we can move the caching into the model rather than template if we want to
<wgrant> lifeless: We've used this as fault-tolerance in the past, and everything goes to shit if the cache happens to expire while it's down.
<lifeless> wgrant: since you are here.
<lifeless> wgrant: I'd like your opinion on the tal:cache in lib/lp/app/templates/base-layout-macros.pt
<lifeless> AFAICT the entire block is removable as something insane
<lifeless> but, I'm likely misunderstanding it
<wgrant> You are completely misunderstanding it
<wgrant> It's unrelated to memcache; run away
<lifeless>   <tal:cache condition="view/user|nothing"
<lifeless> ^ that memcaches its output
<wgrant> Nope
<wgrant> tal:FOO
<wgrant> The FOO means nothing.
<wgrant> Only the NS is important.
<lifeless> Oh, freaking lies.
<lifeless> lies lies lies.
<lifeless> Have I mentioned I loath some parts of tal ?
<wgrant> It just implies a no-op element and removes the need for the NS on attribute names.
<lifeless> grep caught it
<lifeless> cause tal:cache was used by folk to setup memcache rules.
<lifeless> thanks for 'splaining.
<lifeless> be nice to give it a clearer name.
<wgrant> np
<lifeless> my lp:memcache branch will be massive LoC win soon :)
<lifeless> well, s/massive/enoughtodosomethinginterestingwith/
<lifeless> and pushing \o/
<lifeless> mwhudson: bug 623199 lies
<_mup_> Bug #623199: scripts do not establish valid zope participations <lp-foundations> <qa-untestable> <Launchpad itself:In Progress by mwhudson> < https://launchpad.net/bugs/623199 >
<mwhudson> the 'in progress' part?
<lifeless> and the assignee I suspect, part and parcel
<lifeless> unless you are ...
<mwhudson> yes certainly that
<mwhudson> i guess it's a classic of not really specifying a condition of when it can be called done
<mwhudson> as i pointed out early on the title is also a lie, after all
<lifeless> wgrant: bug 939055 - its alredy mmmmmmmmmm horrible
<_mup_> Bug #939055: front page blog feed updates are done in-line with requests <canonical-losa-lp> <Launchpad itself:Triaged> < https://launchpad.net/bugs/939055 >
<lifeless> sorry, cat got in there
<wgrant> lifeless: Horrible but working.
<lifeless> wgrant: yes, and when the blog is down, and memcache times out, the behaviour we see will be the same
<wgrant> No.
<mwhudson> lifeless: if you want to retitle it as "timelines are not automatically cleared when a new interaction is set up" then it's back to triaged
<lifeless> wgrant: you'll need to unpack that.
<wgrant> lifeless: We'll see it when the blog is down.
<wgrant> No memcache timed out required.
<wgrant> Currently a memcache timeout is required.
<lifeless> wgrant: which is a mild increase in frequency increase
<wgrant> lifeless: Before it can be removed we need to prevent it from crashing.
<lifeless> wgrant: a 10 minute outage today has a 1 in 6 chance of showing the same crash.
<mwhudson> lifeless: if you want to retitle it "scripts use interactions which are not suitable for storing interesting things on" then its fix released
<lifeless> wgrant: so no, we don't 'need' to.
<mwhudson> (and you should file/find a new bug for that first thing)
<wgrant> lifeless: This means it's impossible to perform any maintenance on the blog without bringing parts of Launchpad down.
<lifeless> mwhudson: FR it please.
<lifeless> wgrant: no, it doesn't, because it will still be cached, and we can trivially make that page a 60 minute cache as well.
<lifeless> wgrant: via a one-line config change to squid.
<wgrant> Ah good, SPOFing ourselves even more on druzhnaya :)
<mwhudson> lifeless: https://bugs.launchpad.net/launchpad/+bug/623199
<_mup_> Bug #623199: there is no way to store interesting data on the participation scripts set up <lp-foundations> <qa-untestable> <Launchpad itself:Fix Released by mwhudson> < https://launchpad.net/bugs/623199 >
<lifeless> wgrant: vs memcaches non-resilient cluster, no significant difference.
<lifeless> mwhudson: thanks
<wgrant> lifeless: Well, the memcached cluster is not an Antarctic base and we haven't had major problems with it in the past.
<lifeless> wgrant: so put numbers on this; whats the expect change, in % of our OOPS count per month.
<lifeless> wgrant: did you remove the webservice entity cache stuf?
<wgrant> lifeless: It's gone from LP, but not from lazr.restful.
<lifeless> kk
<lifeless> darrens has confirmed, appservers are using druz for it.
<lifeless> and we're getting -lots- of hits on it
<lifeless> wgrant: another thing you've missed; you're assuming the front page is staying in memcache
<lifeless> last I checked our stats, we have a ridiculously low hit rate, partly due to high evictions.
<lifeless> allenap: are you really hacking on bug 704585 ?
<_mup_> Bug #704585: canonical_url performs poorly <timeout> <Launchpad itself:In Progress by allenap> < https://launchpad.net/bugs/704585 >
<lifeless> wgrant: bug 618019
<_mup_> Bug #618019: storm processing of result sets can be very very slow <lp-foundations> <Launchpad itself:Triaged> <Storm:Confirmed> < https://launchpad.net/bugs/618019 >
<lifeless> wgrant: just retitled. Thats the one about cache handling and columns etc.
<lifeless> wgrant: and/or bug 632145
<_mup_> Bug #632145: handling of Specification result sets with 3000 rows is slow <Storm:New> < https://launchpad.net/bugs/632145 >
<lifeless> hmm, can't find the bug about series/milestones having caching
<james_w> anyone know if there's a standard for publishing structured content in an atom feed?
<lifeless> hmm, we may have fun :) - 24 /    0  RootObject:index.html
<james_w> i.e. for PuSH
<lifeless> james_w: 'atom'
<james_w> for something that isn't author/description/etc.
<james_w> description = json.dumps(whatever) perhaps
<lifeless> james_w: its xml
<lifeless> james_w: putting json in xml would be fffffugly at best :)
<lifeless> james_w: however, atom has no DTD or validator
<lifeless> james_w: so, IIRC, you can just add any keys you want
<lifeless> http://tools.ietf.org/html/rfc4287#section-6
<lifeless> james_w: ^
<lifeless> james_w: are you looking at PSHB ?
<lifeless> mwhudson: https://code.launchpad.net/~lifeless/launchpad/memcache/+merge/109551 if yo ufeel like a treat.
<james_w> lifeless, yes
<lifeless> james_w: I'm very keen on PSHB, FWIW.
<james_w> lifeless, the next question is how to convince a library to add arbitrary content :-)
<james_w> lifeless, I know, the two are related :-)
<lifeless> james_w: :)
* lifeless changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<lifeless> james_w: did your oops-tools patch fail ?
<lifeless> james_w: or we just haven't deployed.
<james_w> lifeless, the branding one?
<lifeless> yes
<james_w> lifeless, it's waiting on tarmac or something
<lifeless> james_w: webops run that, and don't want the MP's, so a ping on -ops is the thing (which I've just done)
<james_w> ok
<lifeless> can has revu https://code.launchpad.net/~lifeless/launchpad/bug-1011390/+merge/109559
<stub> diff updating seems stuck there
<lifeless> I see a diff
<stub> r=stub
<stub> I saw a diff. I also saw the yellow diff-is-updating blurb. Odd since there is only one unmerged revision?
<lifeless> huh, yes
<lifeless> that is indeed odd
<lifeless> I only pushed to it the once
<lifeless> stub: this may be more interesting https://code.launchpad.net/~lifeless/launchpad/memcache/+merge/109551
<lifeless> stub: could you ec2-land the former branch? Still on my TODO is fixing my landing environment.
<stub> haha... when I move upstairs, yes :)
<lifeless> stub: no rush
<lifeless> stub: have a good weekend ?
<stub> good enough ;)
<lifeless> brb
<stub> Did sod all besides a movie.
<lifeless> nice
<lifeless> I'm finally, as of late sunday, feeling human again
<lifeless> small bit of sniffles is all that remains
<lifeless> of course, cynthia is now teething again and didn't sleep at all last night ><
 * lifeless is spaced out
<stub> huh... I also see just 'updating diff' on that memcache mp
<adeuring> good morning
<allenap> lifeless: No, I'm not. I've set it back to triaged.
<jml> Review of https://code.launchpad.net/~jml/launchpad/validate-ppa-owner/+merge/109529 sought.
<jml> also, how can I tell when my db patch has been deployed?
<cjwatson> Watch http://lpqateam.canonical.com/qa-reports/deployment-db-stable.html; also IME one of the .au cabal will often tell you
<cjwatson> You can also watch https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadProductionStatus
<jml> cjwatson: thanks.
<jml> cjwatson: how did you qa your db patch?
<Gwaihir> hello all, I have a small problema with a django app that is using launchpad lib to get user info, it gives an HTTPError code 401, saying the the token has expired, does anyone have an idea on how to this could be fixed? bug is #1010455 (repeating question from #launchpad channel)
<_mup_> Bug #1010455: Patchmetric Login Token Expired <Linaro patch metrics:New> < https://launchpad.net/bugs/1010455 >
<cjwatson> jml: ran test suite on corresponding code branch with db patch both unapplied and applied; asked on -ops for somebody to tell me how long the db patch took to apply on (qa)staging so that I could ensure it was within the time limit
<jml> cjwatson: ok. thanks.
<cjwatson> (which looks like /srv/launchpad.net-logs/staging/sourcherry/*-staging_restore.log on carob)
<jml> LP's permission system :(
<jml> I can figure out how to allow a new group of people the ability to *create* private PPAs, but I can't figure out how to let them *set* private.
<jml> Hmm. Hmm.
<czajkowski> jml: do you need any branches now made private if you do just shout and I can do it for you.
<jml> czajkowski: I think I'm OK thanks.
<rick_h> morning
<rick_h> ivorykr8: howdy, welcome to the squad
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 3.47*10^2
<gary_poster> ivorykr8, welcome.  don't believe allenap, we're bots.
<rick_h> gary_poster: that's just what a non-bot would say!
<gary_poster> rick_h, my brother always lies.  I always tell the truth.
<jml> oh hey
<jml> on call reviewers!
<rick_h> anyone recall if there's a way to python -m module:function ?
<rick_h> I can get the module part, but not the function in the module
<jml> benji: https://code.launchpad.net/~jml/launchpad/validate-ppa-owner/+merge/109529 is the first of three patches that needs review.
<jml> rick_h: python -c 'import module; module.function()'
<benji> jml: I'll take a look.
<jml> benji: thanks.
<rick_h> jml: thanks
<jml> benji: sinzui has just approved that branch. https://code.launchpad.net/~jml/launchpad/p3a-private-team/+merge/109600 is the next one in the series.
<benji> jml: ok
* fjlacoste changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivorykr8 | On call reviewer: benji | Firefighting: - | Critical bugs: 3.47*10^2
<fjlacoste> ivorykr8: welcome!
<jelmer> 'morning fjlacoste
<jelmer> hi ivorykr8
<flacoste> hi jelmer
<rvba> Hi ivorykr8 and welcome to the team!
<deryck> ivorykr8, welcome!  Here's the link for the stand-up: https://plus.google.com/hangouts/_/4b599a50a7ebb329230ced7d9caef1ba95835bfc?authuser=0&hl=en
<deryck> adeuring, likewise ^^
<mterry> Hello!  If I can see a private bug, I had thought that meant I can edit its privacy.  But I've encountered a crasher where that's not the case.  Is there a way to find out why I can't edit the privacy?
<mterry> (this is bug 1011293)
<mterry> No bug bot?  :(  https://bugs.launchpad.net/ubuntu/+source/gnome-settings-daemon/+bug/1011293
<bac> welcome ivorykr8
<czajkowski> ivorykr8: how are you doing on day 1 anything we can help you with ?
<ivorykr8> well actually no i just try to get myself into python :)
<jml> https://code.launchpad.net/~jml/launchpad/remove-top-tests/+merge/109664 up for review
<jml> (Heals 477 lines of damage)
<cjwatson> benji: Thanks for your review on https://code.launchpad.net/~cjwatson/launchpad/pocket-permissions/+merge/109192; can you "Status: Approved" and I'll land it?
<benji> cjwatson: sure
<james_w> jml, delete it in the other branch too and score double!
<jml> james_w: well, ISTR objecting when it was first added.
<james_w> I assume that was before testr slowest was added?
<jml> james_w: yeah, but it was after subunit reports were available from bin/test, I think.
<lifeless> morning everyone
<jml> is there a LoC score board?
<lifeless> other than ohlo, no.
<lifeless> and cjwatsons thing
<jml> ta
 * jml has been mucking around with dmg
<lifeless> benji: hi
<benji> hi lifeless
<lifeless> benji: if you're still on the review hook, perhaps you can do me a related favour
<lifeless> 05:59 < lifeless> anyone around to run an ec2 test command for me? Will get my environment fixed soon, I swear.
<lifeless> 05:59 < lifeless> https://code.launchpad.net/~lifeless/launchpad/memcache/+merge/109551
<lifeless> (I asked in the wrong channel initially :P)
<benji> lifeless: Instance starting.  I'll give you the URL when it presents itself.
<lifeless> benji: oh doh
<lifeless> benji: I mean 'ec2 land'
<lifeless> not ec2 test.
<benji> heh
<benji> ok, restarting
 * lifeless stabs his lying fingers.
<benji> heh
<lifeless> benji: thanks
<jml> g'night
<lifeless> night jml
<lifeless> benji: could you try again, if you're still around ?
<lifeless> benji: I've pushed a fix
<jcsackett> wallyworld_, StevenK, wgrant: i don't think i'm making standup--i've been feeling progressively iller as the afternoon has worn on and i'm at the point where i just need to crash.
<jcsackett> s/iller/more ill/
<wallyworld_> jcsackett: sorry to hear that, get well soon
<benji> lifeless: I was AFK.  Running ec2 land for you now.
<lifeless> benji: thanks!
<StevenK> wgrant: Brian's branch also needs QA.
<StevenK> And lifeless' r15388
<wgrant> StevenK: Yes, but they're rather simple.
 * StevenK looks for an open team on qas to abuse
<lifeless> StevenK: mine doesn't, just qa-ok it; no code changes
<lifeless> config default value change + increased test coverage.
<StevenK> wgrant: So my change sort of works. If you select/type an open team and hit Subscribe, the form submits and dumps you back to Branch:+index with no feedback, but they are not subscribed.
<wgrant> StevenK: So the form has no error handling?
<StevenK> wgrant: My code calls setFieldError(), so I'm not sure why that doesn't fire.
<wgrant> StevenK: Simply setting a fielderror won't do much if the form isn't rendered.
<wgrant> StevenK: You probably want to not set next_url in that case
<wgrant> So it doesn't redirect away
<wgrant> Normally the validation is done in a validate method, not the action itself.
<StevenK> wgrant: So I think the code may need a little more polish -- use a validate method on Branch:+subscribe like you say, and probably change the JS popup if the branch is private and the reviewer is an open team when proposing an MP.
<wgrant> validate may be less appropriate here.
<wgrant> LBYL is perhaps not a good idea.
<StevenK> wgrant: Oh? What do you suggest then? A notification that "so-and-so not subscribed, this branch is private and they are an open/delegated team." ?
<wgrant> Personally I would alter the JS popup to not change anything, and just tell you that you're doing something stupid. Since it should become rare that people don't have access.
<wgrant> StevenK: I'd probably have the action catch the exception and display an error.
<wgrant> Like you do now, except in a way that actually works.
<StevenK> Haha
<StevenK> I thought setFieldError() would work :-(
<wgrant> It sets a field error.
<StevenK> Creating an MP with an open team as a reviwer has them pop up as the reviewer, but they don't get subscribed.
<StevenK> wallyworld_, wgrant: Parramatta got 90mm of rain yesterday.
<wallyworld_> is that all?
<StevenK> Hah
<StevenK> wgrant: So I think this is qa-ok for now, but I will come back to it after the DB patch o' doom.
<wgrant> StevenK: Sounds good.
<wgrant> Seems like you have a tonne of missing test coverage, thoguh.
<StevenK> A test for the field error is there, and it passed. :-(
#launchpad-dev 2012-06-12
<StevenK> OMG, actual sun.
<nigelb> heh
<nigelb> I thought you guys had plenty of sun. oh wait, because winter?
<wgrant> Because winter and because Sydney.
<nigelb> Ah!
<StevenK> nigelb: The suburb I live in got 90mm of rain yesterday.
<nigelb> Woah.
<StevenK> wgrant: We need at least two DB patches. We need to drop NOT NULL from private and transitively_private
<wgrant> StevenK: False. They both have defaults.
<StevenK> Ah
<StevenK> Right, so they get set to false irrespectively to information_type and they can just get dumped.
<wgrant> StevenK: transitively_private default to true, but yes.
<wgrant> StevenK: We can just stop setting them and they will default to relatively sensible things that nothing will see anyway.
<wgrant> Then we can drop them.
<StevenK> wgrant: You were talking about bug_build_access_cache in 16-0 that I should look at?
<wgrant> StevenK: That sort of thing, yes.
<lifeless> wgrant: ping (LEP, impl notes)
<lifeless> ugh
<lifeless> this post-merge-dif-nuking really is disruptive.
<lifeless> https://code.launchpad.net/~lifeless/launchpad/memcache/+merge/109551
<StevenK> lifeless: It doesn't even look like that branch has been scanned.
<lifeless> indeed
<lifeless> wedged-is-U
<lifeless> S
<StevenK> lifeless: bzr push -r -2 ...  ?
<StevenK> lifeless: I approve of +49/-1182
<StevenK> wgrant: Am I on the right track? http://pastebin.ubuntu.com/1036633/
<lifeless> StevenK: that thing is landed :>
<lifeless> garyposter: I have checkmarkified https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess#preview
<lifeless> jml: I'd like to know if https://checkmarkable.com/team/sthederj/checklist/1263/view would have helped your confusion around db schema qa etc the other day.
<StevenK> lifeless: Hmmm, memcache for garbo is useful. If we had some other way to record stuff between runs it could die a horrible death
 * StevenK stabs the branch scanner
<stub> StevenK: Do you mean, like, some sort of persistent database?
<StevenK> stub: Haha
<stub> Its a pretty bad fit for garbo. If that is the last use of it, we should replace it with something suitable.
<stub> Keep the IMemcache interface for lazyness
<StevenK> stub: I think it's the last use.
<StevenK> And only a few jobs use it
<StevenK> One job, in fact.
<stub> And it could be replaced with sqlite easily enough or even something less like a hack
<StevenK> wallyworld_: Do you have a sec to look over https://code.launchpad.net/~stevenk/launchpad/drop-set-spph-packageupload/+merge/109776 ?
<nigelb> woah, LP has a new intern?
<nigelb> You guys are going to break him, aren't you? ;-)
<wallyworld_> StevenK: a sec yes, i'm about to go see aus vs japan :-)
<StevenK> wallyworld_: It's +1/-133, so it should be very quick
<nigelb> wallyworld_: rugby?
<StevenK> Soccer
<StevenK> World Cup
<nigelb> wat.
<nigelb> Soccer World Cup?
<wallyworld_> nigelb: football
<nigelb> o_O
<wallyworld_> world cup qualifier
<nigelb> aha
<nigelb> that makes better sense
<StevenK> We are going to lose horribly
<wallyworld_> yep :-(
<nigelb> I was following Eurocup and couldn't believe there was world cup going on.
<wallyworld_> StevenK: r=me
<StevenK> wallyworld_: Thanks. Go and enjoy the soccer.
<wallyworld_> will do. hope we at least make a decent game of it
<lifeless> StevenK: two jobs when I looked in trunk
<StevenK> lifeless: Yup, I just removed one in the MP wallyworld_ reviewed.
<lifeless> StevenK: ah, kk.
<lifeless> we could do a persistent transactional k:v store in the DB
<lifeless> or we could grab redis or something with persistent backing and use that.
<lifeless> memcache is fine I think today.
<lifeless> stub: sqlite in production would be more, not less of a hack :)
<stub> lifeless: yer, I'm not suggesting it is a briliant idea. just that there are alternatives.
<stub> There is a new hstore datatype in PG 9.1 for k,v stuff.
<stub> And json to match the existing xml datatype
<wgrant> lifeless: Hi, just added them. https://dev.launchpad.net/DisklessArchives and https://dev.launchpad.net/LEP/DisklessArchives
<lifeless> I'll do an edit pass and then let elmo etc know
<lifeless> wgrant: we may want two leps; a horizontal scaling one covering the uploader bug etc, and this one. *may*
<wgrant> lifeless: Possibly, yes.
<wgrant> But we also need *something* for the whole picture, I think.
<lifeless> Can't fault your logic
<lifeless> uhm, we'll see. I'm applying humanizing to this doc just now :)
<lifeless> thank you for putting them together.
<lifeless> wgrant: you've let a lot of implementation detail leak into the LEP (I mention this so you don't think I'm crazy for deleting it). I'll ensure it survives somewhere.
<wgrant> lifeless: Probably.
<adeuring> good mornin
<lifeless> wgrant: could you check (e.g. on dogfood, or give me a query) the number of unique people uploading to ppa.l.n ?
<jelmer> g'morning adeuring
<adeuring> hi jelmer
<wgrant> lifeless: SELECTing.
<lifeless> wgrant: https://dev.launchpad.net/LEP/DisklessArchives edited, please check I didn't trash your intent :)
<wgrant> lifeless: Looks reasonable.
<wgrant> Thanks.
<lifeless> wgrant: still around ?
<wgrant> lifeless: A little.
<wgrant> lifeless: 'sup?
<lifeless> so you claim external acls are cached on the full request path.
<lifeless> But
<lifeless> makeExternalAclKey
<lifeless> claims otherwise
<lifeless> I want to check why you say that before I delete it as wrong :)
<lifeless> wgrant: ^
<wgrant> lifeless: makeExternalAclKey? Is that some Squid thing?
<lifeless> yes
<wgrant> lifeless: Squid can't know that we only care about the first two segments of the path.
<lifeless> its the thing that makes the helper cache key from the helper parameter format string
<lifeless> wgrant: thats what the third helper is for.
<lifeless> wgrant: it does that, trivially, and tags the request.
<wgrant> Well, Squid itself doesn't cache it. One of the helpers does, if there are two.
<wgrant> Does the acl helper get the tags?
<wgrant> The rewrite helper doesn't.
<lifeless> so, there are three helpers
<lifeless> mapper
<lifeless> it takes url, returns ppa string, does no DB lookup at all. Purely local.
<lifeless> it is cached by squid on full url.
<lifeless> authenticator
<wgrant> That's the ideal situation.
<lifeless> it takes the auth header + the ppa tag, does DB lookup (via webservice).
<lifeless> its cache by squid on the auth header value + the ppa tag value.
<wgrant> But I didn't see a way for the mapper to execute before the others and tag the request in a way they could see it.
<lifeless> you use it in an acl earlier.
<lifeless> we may need four helpers in fact, we need to detect ppa.l.n requests to private ppas.
<wgrant> Ahh. There's an %EXT_TAG new in 3.2
<lifeless> the mapper always succeeds, and you say http_access deny !mapper
<wgrant> (which isn't in Ubuntu)
<lifeless> or some such.
<lifeless> just %TAG should be fine, no ?
<wgrant> Is that a thing?
<lifeless> ah no
<lifeless> misread the code.
<lifeless> So we need that ported, or to run 3.2.
<lifeless> noting.
<wgrant> "%EXT_TAG tag= value returned by previous external ACL calls. Tag may not be altered once set."
<wgrant> lifeless: So we can't stack tags.
* gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivorykr8 | On call reviewer: gmb | Firefighting: - | Critical bugs: 3.47*10^2
<wgrant> lifeless: I was handling public-on-private by an arg on the the librarian mapper func, but something nicer might be better.
<lifeless> wgrant: oh, two rewriters?
<wgrant> lifeless: Or just pass the hostname in.
<wgrant> But two rewriters might be cleaner.
<lifeless> well, a public and private ppa may map to the same file
<lifeless> I propose just having one rewriter that doesn't care about public/private, and do the public/private check separately.
<lifeless> for public requests we check the ppa is public, for private ones we check its authed and their creds match
<wgrant> I guess the extra public roundtrip doesn't suck too much if we're using a separate API service that isn't LP.
<jml> lifeless:
<jml> lifeless: that's the checklist at the top of the process page, but with some detailed notes
<jml> lifeless: I made edits to the wiki page as I went, correcting sources of my confusion or uncertainty
<wgrant> lifeless: So we just have a separate verify_public external_acl_type?
<wgrant> Which uses a separate helper
<lifeless> jml: it is, but its also a stateful checklist so you can work through it as yor patch progresses.
<lifeless> jml: bayme thats not helpful.
<lifeless> wgrant: yes, and use squid acl rules to route to one or tother.
<wgrant> lifeless: Right, if you look at my sample config it's trivial to do.
<jml> lifeless: that might help. I couldn't say. I don't feel I lost track of where I was at any point, despite it being a multi-day process
<lifeless> wgrant: we're going to want a small patch to squid
<lifeless> wgrant: to let the helper turn off caching for failures.
<jml> incidentally, I made this: https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AsE2-aW-RE8adG5vbW1nN3dJV0o0WUVKSElFRE9UUFE#gid=0
<wgrant> lifeless: negative_ttl=0?
<lifeless> jml: we're running an experiment with checkmarkable, if you want to help that you could use it next time.
<jml> lifeless: ok.
<lifeless> wgrant: oh, that will do fine :)
<lifeless> wgrant: I forgots.
<wgrant> lifeless: external_acl_type has it, not sure if url_rewrite_helper does
<wgrant> But I assume so
<jml> lifeless: I can't say it's grabbed me.
 * jml *sighs*
<lifeless> jml: its trying hard :)
<jml> I'm just a little sick of rattling my tin cup every time I want a review.
 * jml drags it out again
<jml> Alms for the poor? https://code.launchpad.net/~jml/launchpad/remove-top-tests/+merge/109664
<wgrant> url_rewrite_program doesn't have any such config. Probably somewhere else determines it.
<lifeless> gmb: ^
<jml> Spare a penny for poor ex-leper? https://code.launchpad.net/~jml/lp-dev-utils/apply-top-tests-fixes/+merge/109665
<StevenK> jml: remove-top-tests r=me
<lifeless> wgrant: rewriter gets user= passed to it, if we want it
<gmb> jml, Looking now.
<lifeless> wgrant: src/redirect.cc redirectStart(ClientHttpRequest * http, RH * handler, void *data)
<StevenK> gmb: I grabbed the LP one already
<gmb> StevenK, Ok.
<lifeless> wgrant: I'm not sure the rewriter has caching; we may need a aggressive cache layer just for it.
<gmb> ... the worrying implication, carrying on from jml's script, is that StevenK is somehow the messiah. This has far too many unpleasant implications for me, so I'm just going to look at the lp-dev-utils branch and STFU.
<wgrant> lifeless: I don't think user= is useful.extuser= might be.
<wgrant> lifeless: Right, we'd basically need completely custom caching for the rewriter anyway.
<lifeless> wgrant: rephrase, I'm reasonably sure it doesn't have caching.
<lifeless> wgrant: custom, how so ?
<wgrant> lifeless: As I detail in my musings on the different cache coherency constraints.
<wgrant> lifeless: pool/ should cache forever and then some.
<wgrant> dists/ should not
<lifeless> wgrant: you forgot that the cache keys are based on the rewrite value
<StevenK> gmb: Unpleasant for you
<lifeless> wgrant: unless you disproved that.
<StevenK> ? What about ME!?
<wgrant> lifeless: I mean the lookups.
<gmb> StevenK, You're the one that's going to end up nailed to a tree.
<jml> gmb: he's not the messiah. he's a very naughty boy.
<wgrant> lifeless: Pool file rewrites never have to hit the DB ever again.
<StevenK> jml: I was waiting for that.
<wgrant> lifeless: Because they never change./
<lifeless> wgrant: ah! That was entirely unclear.
<jml> StevenK: thanks for the review.
 * lifeless reedits.
<gmb> jml, I'm so, so sad to say that I didn't actually see that coming :)
<StevenK> Hahaha
<czajkowski> gmb: you need to get out more
<gmb> jml, Anyroad up, your lp-dev-utils branch is approved. Want me to merge it for you?
<wgrant> lifeless: "Caching of archive path to librarian URL rewrites is a bit of an unresolved issue. While pool/ lookups should be cachable forever without requiring revalidation"
<wgrant> lifeless: Pretty explicit :)
<jml> gmb: yes please. I'll land the removal branch.
<gmb> czajkowski, This is not, in fact, something with which I would disagree.
<lifeless> wgrant: and yes, entirely unclear :)
<lifeless> s/yes/yet/
<wgrant> lifeless: I should have mentioned, though, that we'd confirmed Squid cached the content based on the post-rewrite URL
<wgrant> lifeless: Howso? :)
<lifeless> wgrant: I had no sleep sunday night. *anything* may fail to be unclear.
<wgrant> Heh
<wgrant> May fail to be unclear?
<wgrant> Heh
<wgrant> Anyway, I await your edits.
<gmb> czajkowski, Still, only 1 week and I get to catch the bollocks o'clock train to our shiny new HQ.
<czajkowski> gmb: whoo shall see you there with bells on
<gmb> \0/
<lifeless> wgrant: soon, I think.
<gmb> jml, Merged and pushed.
<lifeless> wgrant: I think we want memcache in there.
<lifeless> wgrant: if you disagree, I'll step you through it.
<lifeless> wgrant: either that, or we make the mapping step so lightweight we don't need a cache at all.
<wgrant> lifeless: memcache is probably a good place to cache this, yes.
<lifeless> wgrant: I haven't gotten into the schema implications of that as yet.
<wgrant> Hm?
<wgrant> I deliberately left the caching mechanism unspecified.
<wgrant> memcached may well be a good option.
<wgrant> Not sure what you mean by schema implementations.
<wgrant> implications
<lifeless> well, what does it take to make url->librarian url mapping take 3ms
<lifeless> or, lets be generous, 5.
<wgrant> With a bit of work the DB can answer it is about 2-3ms.
<wgrant> s/is/in/
<wgrant> But we should cache it to avoid hitting it too often.
<jml> gmb: thanks.
<wgrant> If we use something shared like memcache we can even have the publisher shootdown the index caches.
<lifeless> wgrant: well, at 3K rps, we're looking at 9 seconds worth of load/second.
<wgrant> Rather than revalidating every time.
<wgrant> Or having a TTL of like 5s.
<lifeless> so we'd need 9 cores at peak.
<wgrant> We can easily take that, but let's not.
<lifeless> and 3 at normal, which we can trivially do.
<lifeless> wgrant: lets also avoid writing something we don't need :). I am going to call it out, but also let it be a thing to test and validate, not blindly do.
<wgrant> That's true.
<jml> it looks like the database rollouts have been blocked for a few days on a non-QAd bug: http://launchpad.net/bugs/1007333
<_mup_> Bug #1007333: update_database_disk_utilization fails with GIN indexes <qa-needstesting> <Launchpad itself:Fix Committed by stub> < https://launchpad.net/bugs/1007333 >
<jml> anything I can do to push that along?
 * jml confesses to knowing nothing as yet about GIN
<wgrant> jml: Not blocked for a few days.
<wgrant> Just that we have other things to do tonight, and .au wasn't here yesterday.
<jml> wgrant: perhaps I'm misreading lpqateam and the bug report, but it says it was marked as needing QA on Jun 1
<wgrant> THat's possible -- we've had a lot of backlog to get through.
<jml> ok.
<lifeless> wgrant: what do you mean by 'migration path for PPAs' ?
<lifeless> wgrant: do you mean 'we need to be able to bring up and validate the new environment before breaking what works on germanium?
<stub1> It was also a two part landing which would have confused things.
<stub1> Second landing didn't happen until Friday?
 * lifeless assumes he does
<wgrant> lifeless: We probably want the ability to run both concurrently for a time, but also we need to be able to migrate to the new one.
<wgrant> Sorry, distracted by pgbouncer :)
<lifeless> wgrant: what does that mean though ?
<lifeless> wgrant: we have two sets of boxes
<wgrant> lifeless: eg. we need to load all the Packages and Sources and everything into the DB.
<wgrant> lifeless: And keep them up to date.
<wgrant> Without taking down PPA publishing for a day.
<lifeless> wgrant: I don't think we do :)
<stub1> qa-ok since staging still works
<wgrant> Oh?
<wgrant> lifeless: What is your cunning plan?
<lifeless> wgrant: If we bring up the new system, and run a 'garbo' job across it to regenerate everything, filling in the missing data, what is the impact on germanium
<wgrant> lifeless: The garbo job will take more than 5 minutes.
<lifeless> wgrant: so...
<wgrant> lifeless: So there won't be holes any more, but stuff will be out of date.
<lifeless> wgrant: we make the existing publisher make holes.
<wgrant> lifeless: So the publisher knows about the new DB structures, but only to shoot them in the head?
<lifeless> its a thought
<wgrant> Perhaps.
<lifeless> I don't want to make publishing on germanium *slower*
<lifeless> that scares me, for hopefully obvious reasons.
<wgrant> That's a reasonable point.
<lifeless> so my proposal is to bring up a parallel structure
<wgrant> But the extra load should be minimal.
<lifeless> and teach it to cope with germanium doing its thing
<lifeless> but ignore that germanium did its thing, or detect stale, or something.
<wgrant> I think that's probably more work.
<wgrant> But it is indeed a reasonable thought.
<lifeless> anyhow, now I know what you mean, I can clarify.
<wgrant> Great.
<jml> any tips on how to use the LP web ui when it's running within an LXC container?
<jml> I'm guessing I should just copy /etc/hosts but set it to the lxc instance's ip
<wgrant> jml: https://dev.launchpad.net/Running/LXC
<wgrant> Step 10 is https://dev.launchpad.net/Running/RemoteAccess
<jml> wgrant: thanks.
<jml> buh wuh
<jml> apache2 isn't installed?
<wgrant> That sounds quite extraordinary.
<wgrant> Sure you ran rocketfuel-setup?
<wgrant> Is launchpad-developer-dependencies installed?
<jml> In order: Yes it does. I think I did but I am not certain. Yes.
<jml> observe: http://paste.ubuntu.com/1036945/
<wgrant> Oh right, rocketfuel-setup installs apache separately later on.
<wgrant> Perhaps it died early and you failed to notice?
<wgrant> REQUIRED_PACKAGES="launchpad-developer-dependencies apache2 apache2-mpm-worker libapache2-mod-wsgi"
<wgrant> for pkg in $REQUIRED_PACKAGES; do do_install;
<wgrant> done
<jml> heh.
<jml> wow.
<wgrant> jml: Re. bug #1011611 I wasn't saying it was reasonable, just that it might be helpful for diagnosis.
<_mup_> Bug #1011611: Not possible to file embargoed security bug <bugs> <disclosure> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1011611 >
<wgrant> ?
<wgrant> wow?
<jml> it's like dependencies, but in shell.
<wgrant> Yeah
<wgrant> Pretty awesome.
<wgrant> We don't need no packaging system.
<nigelb> ...
<jml> wgrant: (have added this to the bug report) To clarify: clicking on "Public" brings up the chooser in my browser (Chrome) and from there I can change it and submit. I would never have thought to click on it, since there's absolutely no indication in Chrome that it's anything other than a declaration of fact (This bug will be Public).
<wgrant> jml: Yeah.
<wgrant> If you mouseover it it gets an underline, but that's it.
<wgrant> in Firefox it has a drunken exclamation mark.
<jml> :\
<jml> I wonder if there's a more declarative way to say "LP needs these apache modules enabled in order to run"
<wgrant> I don't believe so.
<jml> yay running LP
<wgrant> But can you browse to it?
<jml> (maybe I'll need to do this again next time I lxc-start?)
<wgrant> Need to do what?
<wgrant> The Apache setup is one-off
<wgrant> Running make run or make start needs to be done every time.
<jml> wgrant: install apache, enable 5 modules etc.
<jml> these virtual machines confuse me a little. one of the ones I use resets to a template whenever I use it.
<wgrant> lxc-start shouldn't.
<wgrant> lxc-start-ephemeral is the ephemeral variant.
<lifeless> wgrant: so
 * jml decides that the only sane thing to do is to eat the last of the Wensleydale.
<lifeless> wgrant: primary archive + diskless
<wgrant> lifeless: lalalalala
<lifeless> wgrant: I just wrote this:
<lifeless> split the work in these scripts between on-disk logic and non-disk logic, and migrate the non-disk logic component off of germanium early in the development process, reducing its current load, and giving a single source of logic for the new schema as it comes of age.
<lifeless> wgrant: how would that interact with primary archive - if we did this, would it *interfere* with it.
<wgrant> lifeless: It'd mean redesigning the primary archive publisher.
<lifeless> wgrant: e.g. I'm thinking the first stage would generate jobs to do the spooling to disk etc
<wgrant> Well, that could work.
<lifeless> wgrant: because its still apt-ftparchive ?
<wgrant> lifeless: Right, we can't generate primary archive indices outside the actual on-disk archive.
<wgrant> lifeless: And because we need to maintain the on-disk archive we need a queue of disk files to publish.
<lifeless> wgrant: so, we'd need a kludge for the primary archive, to not generate indices upfront, but the rest of the work could be partitioned sanely ?
<wgrant> lifeless: That's traditionally SPPH WHERE datepublished IS NULL
<wgrant> lifeless: But in your proposal that would be maintained separately.
<lifeless> wgrant: that queue could be explicit as job entries though, yes?
<wgrant> So we'd a set of events to publish the files.
<wgrant> Yes.
<lifeless> I think this is the best approach so far.
<wgrant> Yeah
<wgrant> It means rewriting a bit more.
<wgrant> It's by far the cleanest.
<wgrant> It's a significant amount of extra work.
<wgrant> I discounted it as being too much work, so I don't really have a good estimate of how much work it would be.
<wgrant> But it basically means rewriting the entire backend.
<wgrant> Rather than just extending it in a couple of places.
<lifeless> we can let james_w and jml decide.
<lifeless> I suspect that 'more work' here giving a more debuggable and understandable system is well worth it.
<lifeless> probably a smaller system too when I think about it.
<lifeless> bigger change from what we have today - sure.
<wgrant> It's also throwing away 7-year-old tried and battle-tested code, and forcing people who don't know Soyuz to rewrite Soyuz :)
<lifeless> wgrant: can you expand a little on the concurrency issue with parse-ppa-apache-access-logs ?
<wgrant> But they do have packaging experience, so I guess all is not lost :)
<lifeless> wgrant: the latter is a good thing
<lifeless> wgrant: fresh eyes, no stockholm syndrome.
<wgrant> lifeless: Redesigning Soyuz perhaps.
<lifeless> wgrant: precisely.
<wgrant> Rewriting Soyuz with the same data model and all the boobytraps it has? No.
<lifeless> now, -logs thingy.
<wgrant> lifeless: Well, we currently parse n lines at a time.
<wgrant> lifeless: And then update the counts all at once.
<wgrant> lifeless: If n is too large, we'll get lots of rollbacks.
<wgrant> As there'll be conflicting count updates.
<lifeless> so by problems you mean 'lots of waste' vs 'bad data'
<wgrant> There's also the ParsedApacheLog problem, but I may be too paranoid there.
<wgrant> We currently identify a log file by its first line.
<lifeless> antything else ?
<wgrant> If two logs happen to have the same first line, they'll be confused.
<wgrant> THat's all I know about.
<lifeless> wgrant: whats the cycle for log parsing
<lifeless> cron? /15 ?
<wgrant> lifeless: cron, disabled
<wgrant> But something around there.
 * wgrant checks.
<lifeless> ok
<lifeless> wgrant: and how long does it take when it runs? 2-3m ?
<wgrant> It's */30
<wgrant> I don't know how long it takes nowadays.
<wgrant> It was set up before I worked here, and I haven't had cause to look at it since.
<wgrant> So I have no clue how long it ever took on prod.
<wgrant> Ah, we still have logs.
 * wgrant looks
<wgrant> lifeless: Seems to take 5-10 minutes at load of about 6.
<jml> man, it would be great if pagetests handled unexpected form submission errors by showing the unexpected form submission errors.
<wgrant> jml: The appserver just says "Unexpected form data". It doesn't give any extra data.
<jml> wgrant: I mean in cases where validate() shows errors but the test wasn't expected it to
<wgrant> Oh
<wgrant> Handy.
<lifeless> jml: ...pagetests...
<lifeless> jml: is all I saw.
<jml> well, it'd also be great if Launchpad had an app server that wasn't also a web server.
<jml> i.e. if the web UI was an api client
<lifeless> jml: funny you should mention that.
<nigelb> wallyworld_: \o/ suprise surprise!
<jml> some names appear twice (need to do proper fuzzing), but here's a lines-of-code high score table: http://paste.ubuntu.com/1037100/
<wgrant> jml: Curses.
<jml> wgrant: that you aren't on top, or that the numbers add up to 10k+ LoC added since the policy?
<jml> (incidentally lp:bzr-damage; 'bzr high-scores -r 14780..')
<wgrant> That I'm not on top, indeed.
<wgrant> I wonder what Aaron deleted.
<wgrant> Maybe that was ec2.
<jml> I reckon so.
<jml> wgrant: 'bzr loc' (takes log args) will get you that info
<wgrant>  30 files changed, 679 insertions(+), 6078 deletions(-)
<wgrant> Yup
<wgrant> jml: Nice, thanks.
 * jml is just adding a bunch of todos to the code base before taking a break.
<jml> wgrant: my pleasure.
<jml> wgrant: although it's not nice, it's hideous.
<cjwatson> jml: Oh, fantastic, I'd been meaning to get round to doing that.
<jml> cjwatson: my pleasure. I got stuck in a mental groove.
<jml> cjwatson: I might merge loc-contributors into bzr-damage at some point, if you don't mind
 * jml might also merge bzr-damage into bzr-stats
<jml> the really annoying thing is the way pqm sets itself as author & committer.
<wgrant> Yeah, it would be pretty handy if it didn't do that.
<jml> I know lifeless says that it's the right thing for it to do, but accumulating authors from the merged revisions is expensive and hard to slot into bzr's other mechanisms
<cjwatson> I found that too.
<cjwatson> And it makes logs tedious to read.
<jml> although to be fair, the bzr developers would probably admit that bzrlib.log is in need of love
<StevenK> jml: I personally think that PQM needs to die a horrid, painful and *SLOW* death, but lifeless disagrees. :-(
<wgrant> jml: Does lifeless say that?
<wgrant> Surely I'm the author; I told it what to do.
<jml> I'm thinking of spinning out the pqm hackery I've done here into a separate plugin that tweaks 'bzr log' behaviour by default.
<jml> wgrant: IIRC. It was a conversation long, long ago. Perhaps on IRC, perhaps not.
<jml> or I guess I could patch bzr proper
<jml> what I want to know is what wallyworld_ has been up to
<wgrant> jml: Oh?
<wgrant> What has wallyworld_ been up to?
<StevenK> jml: Why are you stalking wallyworld_?
<jml> StevenK: because my script says he has added 12k+ lines of code, which makes him an outlier
<wgrant> Ah.
<wgrant> He's been writing disclosure stuff.
<wgrant> Lots of frontend and backend stuff.
<jml> wgrant: and that's under a LoC exemption?
 * jml finds that surprising.
<wgrant> I believe so.
<StevenK> disclosure is, yes
<wgrant> And there's a lot of code that will be deleted once this is done.
<StevenK> jml: A lot of wallyworld_'s LoC was from writing a whole new services - ISharing
<wgrant> StevenK: That's hardly relevant.
<wgrant> It's still LoC.
<wgrant> All LoC is equal.
<StevenK> He has also been pilling on bunch of JS
<wgrant> That's the idea of the policy.
<jml> Hmm. I'm also not sure how to count the commits from PQM
<jml> I'm fairly sure they are merges from db-stable
<wgrant> That was a problem with community-contributions.py.
<wgrant> I'm pretty sure i fixed it years ago.
<wgrant> Might be able to steal stuff from that.
<wgrant> Oh right, no, it's hideous.
<wgrant> Don't go near that.
<wgrant> It does work, though.
<jml> yeah.
<jml> I'm thinking of merging some of that into bzr-(damage|stats) too.
<jml> I wish bzr were more popular
<jml> then I'd feel happier about doing that sort of thing
<jml> rather than slightly constrained by a weird ocd variant.
<wgrant> Heh
<wallyworld_> jml: what's wrong with adding code? it's sort of impossible to write a new feature without adding code to implement it don't you think?
<jml> wallyworld_: Certainly it requires new lines. However, those new lines can also be accompanied by removed lines.
<jml> wallyworld_: have you followed Drizzle at all?
<wallyworld_> jml: why does it follow that one must remove lines just because a new feature has been added?
<wallyworld_> i haven't followed drizzle
<jml> wallyworld_: it doesn't necessarily follow, however it's possible and perhaps desirable to insist that you do so.
<wallyworld_> hmmm.
<wallyworld_> i don't see a correlation, but that's just me
<wgrant> wallyworld_: The idea of the policy is that new code can't be added.
<jml> wallyworld_: Launchpad has a lines-of-code freeze policy. It's possible to write a new feature but also "pay" for it by removing other pointless code as you're adding the new feature.
<wgrant> Therefore from adding code it follows that you must remove a corresponding amount.
<jml> wallyworld_: However, the team doing disclosure has decided not to do it.
<wallyworld_> yes, for good reason :-)
<jml> wallyworld_: what's that reason?
<wallyworld_> it's a bit silly to artifically find stuff to delete just because a new feature is being done'
<wgrant> Hm?
<wgrant> That's not silly.
<wallyworld_> sort of adds unnecessary scope to the feature writing
<wgrant> That's the whole point of the policy.
<jml> oh right, you disagree with the lines of code policy
<wallyworld_> in some circumstances yes
<wallyworld_> like bespoke feature development
<jml> hah!
<wallyworld_> which we have an exemtiion for
<wallyworld_> for good reason imho :-)
<wgrant> IMO the only excuse we have is that we started 6 months before the policy.
<wgrant> New features will be started with the policy in mind.
<wallyworld_> i've never ever seen such a loc policy applied to bespoke feature devellopment
<wgrant> Ours was not.
<wgrant> wallyworld_: Really?
<jml> wallyworld_: your good reason being that it adds unnecessary scope?
<wallyworld_> yes, really
<wgrant> wallyworld_: workitems was done under LoC freeze.
<wgrant> wallyworld_: cjwatson's work is under LoC freeze.
<wgrant> celery was done under LoC freeze.
<wallyworld_> jml: yes, and completity and muddles the cleat thought on how best to architect the feature
<jml> and I believe all the work on drizzle (or one of the mysql forks) over the last few years was done under a LoC freeze.
<wgrant> Launchpad is far too big considering what it does.
<wallyworld_> it may be possible, doesn't make it right thing to do
<wgrant> Without the LoC freeze there's nothing convincing people to fix that.
<wgrant> The last 7 years have shown that very well.
<wallyworld_> that's what maintenance should be for
<jml> and it is, but  it's hard with a leaky bucket.
<wallyworld_> imagine starting a new project from scratch. bit hard for a loc freeze there. a new bespoke feature sort of falls into that category. it shouldn;t be tied to sins of the past
<cjwatson> It clearly wouldn't work for a newer project, but for a mature project that's well-known to contain lots of cruft I think it's an interesting enough policy that I'm happy to support it by finding stuff to delete.
<cjwatson> I do waver between thinking it's insanity and thinking it's genius
<wallyworld_> i agree somewhat if the new work builds on or it related to previous cruft
<czajkowski> cjwatson: fine line is it :)
<cjwatson> Yeah
<cjwatson> But there is something to be said for requiring refactoring along the way; rather like how review often doesn't happen unless it's a requirement, because it's not the obvious shortest path to churning out lots of code
<wallyworld_> i'm all for refactoring, but not as a required, mandatory cost of adding a new unrelated feature
<jml> wallyworld_: you're saying disclosure *isn't* related to previous cruft?
<cjwatson> Sometimes if it isn't mandatory it won't happen.
<wallyworld_> jml: there's arguments both ways
<cjwatson> I mean, maybe maintenance should be doing refactoring, but there's those 300-odd critical bugs whose line has been trending the wrong way ...
<wallyworld_> i do think it is a primarily a maintenance thing
<jml> wallyworld_: if it's not related, you should be implementing it as a separate service from Launchpad
<wallyworld_> and a best effort thing whre it makes sense for feature work
<cjwatson> Admittedly I'm doing much smaller feature work than you are, but I also found that the policy forced me to think about economy of implementation, generally for the better.
<wallyworld_> jml: the new sharing service stuff is essentially a separate bit of infrastructure, a new paradigm
<cjwatson> People doing feature work can very likely churn out lines of code much faster than maintenance programmers can refactor them, so saying it's primarily maintenance is (I think) dooming the project to unbounded growth
<jml> cjwatson: +1
<wallyworld_> cjwatson: i would argue that one should be thinking about that sort of stuff regardless :-)
<cjwatson> And since one of the initial axioms here appears to be that Launchpad is too large ...
<wallyworld_> i wish we were having this discussion over a few beers down at the pub :-)
<cjwatson> Yeah :)
 * wallyworld_ would then challenge jml to another arm wrestle
<cjwatson> Oh, speaking of branches that add stuff to be traded off against later deletions, I could use a review of https://code.launchpad.net/~cjwatson/launchpad/export-change-override/+merge/109549
<czajkowski> gmb: ^^
<jml> wallyworld_: you're more than welcome to next time we meet.
 * wallyworld_ is scared now
<jml> http://wiki.drizzle.org/FAQ#Can_I_get_involved.3F
 * jml â lunch
<gmb> cjwatson, czajkowski: Noted, will take a look, ta.
<sinzui> gmb, do you have time to review https://code.launchpad.net/~sinzui/launchpad/uncommercial-projects/+merge/109404
<gmb> sinzui, I will do shortly.
<sinzui> thank you
<deryck> adeuring, abentley, rick_h -- coming for standup sorry
<deryck> adeuring, https://plus.google.com/hangouts/_/4f2b2de401df7d5f9167b0f44c74671388ee17b4?hl=en
<gmb> sinzui, So, benji looked at the code yesterday and has given it r=, but he's asked me to double-check some of the logic... I'll take a look presently and get back to you.
<benji> gmb: here's an odd MP for you: https://code.launchpad.net/~benji/launchpad/bug-1011793/+merge/109871
<gmb> Ooh, lovely. I loves me some oddness.
<gmb> Hah.
<gmb> Nice.
<gmb> benji, r=me
<benji> thanks
<jml> benji: it might be a good idea to add a README to that directory so someone doesn't remove it thinking "what the heck is an empty directory in VCS for?"
<benji> jml: good call, I'll do so
<gmb> sinzui, r=me
<sinzui> thank you gmb
 * cjwatson upgrades mawson
<lifeless> gary_poster: hey, it might be nice if your buildbot log wiki page included the duration.
<lifeless> gary_poster: just for my prurient interest :)
<gary_poster> lifeless, heh, ok will do
<cjwatson> Have to go now, but I've upgraded dogfood and will QA my patch on it later.
<lifeless> wgrant: can you update the diagram? take apache out.
<wgrant> lifeless: Not without justification :)
<lifeless> wgrant: we don't need it; elmo surprised me and said 'lets use squid directly for http', then we spoke about SSL and he decided that squid would be fine for that too, if I was telling the truth about how squid does SSL.
<wgrant> lifeless: Hopefully we can also do the header mangling with need in Squid, then :)
<wgrant> But elmo was the main reason I thought we'd need Apache, so sounds good.
<lifeless> wgrant: we can at a pinch ignore the outbound side cache headers, I suspect ppa.l.n doesn't do stuff there anyhow.
<lifeless> wgrant: there is a header replacement system though
<lifeless> http://www.squid-cache.org/Doc/config/reply_header_replace/
<lifeless> may need to tweak that to permit acl based rules I guess
<wgrant> lifeless: ppa.launchpad.net doesn't need to do it at present.
<wgrant> lifeless: Since the static files don't have Expires: $FOREVER
<wgrant> Librarian responses wil
<lifeless> point is, we can set a global value trivially
<lifeless> we can't do sophisticated setting today.
<wgrant> Right.
<lifeless> what ppa.l.n does is roughly equiv to a global value
<wgrant> lifeless: That's done, btw.
<cjwatson> Bug 914779 - broken, but is that qa-ok based on my comments?  I've checked that things like Archive.newComponentUploader and Archive.newPackageUploader still works, so it's a matter of attempted new functionality being broken rather than a regression.
<_mup_> Bug #914779: Pocket maintainers cannot always upload to their pocket <qa-needstesting> <Launchpad itself:Fix Committed by cjwatson> < https://launchpad.net/bugs/914779 >
<cjwatson> *still work
<StevenK> cjwatson: If it does not impact existing functionality and is safe to deploy, then you can mark as qa-ok.
<cjwatson> Right, thanks.  I'll fix it properly tomorrow.
<cjwatson> Modulo being in a sprint for something else.
<lifeless> wgrant: elmo raises a concern about FDT impact
<lifeless> wgrant: I'll talk with him tomorrow; I think its doable offhand, because a) SC can't complete during FDT *anyway*, and most users have the GUI update-manager which is doing background updates - they won't notice
<wgrant> lifeless: Yeah.
<wgrant> I think it's manageable for now.
<wgrant> And we can eliminate it entirely later.
#launchpad-dev 2012-06-13
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/fix-branch-subscribe-open-team/+merge/109960
<bigjools> yay, spammers are harvesting email addresses from LP
<bigjools> how?
<bigjools> ah wait - pgp keys.  bastards.
<StevenK> wgrant: So, DB patch, since I can't have breakfast. :-/
<wallyworld_> StevenK: you still want the cancel_url set all the time
<StevenK> wallyworld_: Right.
<wgrant> StevenK: The bugtaskflat cache is ([policies], [grantees])
<wgrant> StevenK: The branch cache can be (policy, [grantees])
<StevenK> wgrant: In which case the subselect from APA just dies?
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1038217/
<wgrant> StevenK: It still exists, but it only returns a single value, not an array.
<wallyworld_> StevenK: i'd consider self.next_url = canonical_url(...) in the init() and in the one place where needed, set it to None
<StevenK> wgrant: Oh, SELECT INTO _access_policy FROM (SELECT ...
<wgrant> +        SELECT COALESCE(array_agg(policy), ARRAY[]::integer[])
<wgrant> +            INTO _access_policies
<wgrant> ->
<wgrant> "SELECT policy INTO _access_policy
<StevenK> wgrant: Could we fix the comment too? Since now we're on 9.1
<StevenK> wallyworld_: Ah, I can't set it directly in the function, it breaks.
<wallyworld_> which bit?
<StevenK> wallyworld_: My test blows up with something about can't set next_url directly
<wallyworld_> self.next_url is set in many other places
<wgrant> That view probably has it as a property.
<StevenK> It does
<wgrant> Then that's obviously not going to work.
<wallyworld_> the diff showed the property being removed
<wgrant> StevenK: This should work, rather than the subquery madness: SELECT COALESCE(array_agg(policy ORDER BY policy)) FROM accesspolicyartifact WHERE artifact = 5;
<wgrant> Good way to make sure everyone's upgraded to 9.1, too.
<StevenK> wgrant: http://pastebin.ubuntu.com/1038229/
<wgrant> StevenK: 227 is useless
<wgrant> It doesn't do anything,
<StevenK> wgrant: But that's what you said? :-)
<wgrant> That's the SQL query.
<wgrant> It's clearly not a useful PL/pgSQL statement.
<wgrant> Since it doesn't put the value anywhere.
<wgrant> StevenK: An alternative may be to have a general function which returns the arrays for a given artifact.
<wgrant> BugTaskFlat can use them directly, but Branch can just extract the first policy
<wgrant> Also, brb.
<lifeless> wgrant: lp:~lifeless/squid/3.1-ext-tag
<lifeless> bigjools: what about it ?
<bigjools> lifeless: expiring librarian files - doesn't seem to be mentioned apart from an oblique reference in process-death-row changes
<bigjools> make it explicit
<lifeless> bigjools: do we gc the files two different ways?
<bigjools> lifeless: what files? :)
<bigjools> there's two lots of files at present
<bigjools> and reaped separately
<bigjools> death-row deletes the archive's files and the expiration script deletes the librarian files
<bigjools> both to a different schedule
<lifeless> where deleting a librarian file == unreferencing it ?
<bigjools> right
<bigjools> actually
<bigjools> we set the date for deletion
<lifeless> I believe pdr does that
<bigjools> they both do
<lifeless> or at least, what wgrant wrote claims it is what does it
<lifeless> bigjools: when would one set it vs the other ?
<bigjools> the expiration script deletes the librarian files something like 7 days later
<bigjools> it gives people a chance to recover old files after they get auto-superseded from disk
<lifeless> why don't we just set a 7 day later deletion date?
<bigjools> because it affects quotas
<bigjools> whereas the librarian doesn't
<wgrant> bigjools, lifeless: Three phases: publisher sets SPPH.scheduleddeletiondate; p-d-r deletes from on-disk archive and sets SPPH.dateremoved; e-a-f sets LFA expiry to now
<bigjools> wgrant: WFM
<wgrant> But e-a-f excludes things with dateremoved within the last week
<bigjools> but make it explicit in the wiki page
<wgrant> It'd better work for you, since I was describing the current process :)
<bigjools> given that I know the current process rather well that seems redundant
<wgrant> lifeless doesn't.
<wgrant> Well, didn't.
<bigjools> which is why I was just explaining it ;)
<wgrant> So, DisklessArchives doesn't affect expiry, since p-d-r will continue to set dateremoved.
<wgrant> Or a p-d-r equivalent.
<wgrant> dateremoved still exists -- it's probably used to determine whether a file is still visible or not.
<wgrant> Since we can't just rely on publication state.
<bigjools> presumably the proxy will serve 404s for dateremoved files?
<wgrant> Precisely.
<wgrant> The rewriter will fail the request.
<bigjools> I think that's wrong
<wgrant> Why?
<bigjools> actually, scratch that
<bigjools> it's fine
<wgrant> It's correct, so it had better not be wrong :)
<bigjools> the coffee kicked in and I remembered that the UI has librarian links :)
<wgrant> So it doesn't affect expiry at all. I should probably have mentioned that.
<bigjools> yeah, make it explicit
<wgrant> (although if jml and james_w want to implement my e-a-f rewrite proposal they are welcome to :P)
<bigjools> how do we currently work out quotas, is it adding up PUBLISHED files?
<wgrant> No.
<wgrant> datepublished IS NOT NULL AND dateremoved IS NOT NULL, or something like that.
<lifeless> nasty horrible timingout query
 * wgrant checks.
<lifeless> hits the LFC IIRC.
<wgrant> It's possible that datepublished isn't checked.
<wgrant> But dateremoved is.
<bigjools> ok
<wgrant> lifeless: Of course; only LFC has the size.
<bigjools> yes it checks LFC of course
<bigjools> :)
<lifeless> this might be something to dernorm as part of the project; solve quota performance
<wgrant> It just checks dateremoved IS NOT NULL
<lifeless> just a random though
<wgrant> lifeless: On the contrary, if you're going to suggest that I'll add it to the out of scope section :)
<wgrant> It's completely irrelevant.
<bigjools> focus!
<wallyworld> StevenK: just for you https://code.launchpad.net/~wallyworld/launchpad/bug-javascript-1011611/+merge/109966
<wgrant> lifeless: Your lock has expired -- are you still writing?
<lifeless> yes, sec
<StevenK> wallyworld: r=me with one comment
<wallyworld> StevenK: thanks
<lifeless> wgrant: all done btw
<wgrant> lifeless: Thanks.
<wallyworld> StevenK: i've responded
<lifeless> wgrant: so, 3.2 is in an interesting state just now; trunk has some acl breakage,w hich alex thinks he has fixed, but...
<lifeless> there are folk running 3.2 in prod
<lifeless> basically we'd need to validate it does what we want fairly carefully, so I think 3.1+patches is probably the way to go. I've backported the ext-tag support already for this.
<wgrant> Hmmm
<wgrant> Agreed, the ext_tag stuff looked fairly uninvasive.
<wgrant> But I haven't seen your patch
<wgrant> So I may not have seen everything.
<lifeless> see the wiki page, tis all linked up
<wgrant> StevenK: How goes the DB patch?
<StevenK> wgrant: http://pastebin.ubuntu.com/1038289/
<StevenK> I've been distracted trying to figure out how to disable next_url when it's a property
<wgrant> StevenK: Make it not a property.
<StevenK> wgrant: Zope will deal either way?
<wgrant> Confused.
<wgrant> Zope doesn't know it's a property.
<wgrant> It's a property.
<StevenK> wgrant: Zope will call it properly if it isn't a property?
<wgrant> Make it an attribute.
<wgrant> It doesn't need to be a method or property if you're going to set it explicitly.
<wgrant> (it doesn't need to be, and it can't bd)
<StevenK> wgrant: next_url = canonical_url(context) ?
<wgrant> Right:
<wgrant> if all_is_good:
<wgrant>    self.next_url = canonical_url(context)
<StevenK> wgrant: Not that easy.
<StevenK> next_url is set in _BranchSubscriptionView, which the three subscription views subclass off
<StevenK> Right, Zope does not love next_url if it's not a property
<wgrant> What is "not a property"?
<wgrant> It doesn't require it to be a property.
<StevenK> wgrant: The function is not decorated with @property
<wgrant> Well yes.
<wgrant> Making it a method isn't going to change anything, apart from breaking the interface.
<StevenK> Oh, it fixes the case where you're subscribing an open team nicely. It just breaks everything else. :-)
<wgrant> Erm.
<wgrant> What?
<wgrant> How does it fix that?
<StevenK> Because I can then just set self.next_url = None
<wgrant> Ah.
<wgrant> No.
<wgrant> Mmmm, postgres 9.1
<wgrant> So nice.
<wgrant> GIN, column triggers, ordered aggregates...
<wgrant> StevenK: http://paste.ubuntu.com/1038383/ or so
<lifeless> -sonly
<StevenK> wgrant: Hahah, 12-6, not 16-6 :-P
<wgrant> StevenK: 12-6 is before BugTaskFlat was created :)
<wgrant> So it doesn't work back then.
<wgrant> Has to be after 16-0.
<wgrant> lifeless: sonly?
<StevenK> wgrant: Really?
<wgrant> StevenK: It'll work on prod and staging.
<wgrant> But not in make schema.
<StevenK> Oh, of course, the patches will be applied in order
<wgrant> Because make schema applies in numerical order.
<StevenK> Yeah, just realised
<lifeless> wgrant: -slony
<wgrant> Ah
<wgrant> Right
<StevenK> I guess lifeless means no slony for 9.1
<wgrant> Eventually.
<StevenK> 10 second FDT sounds good
<StevenK> wgrant: So I get the trigger on information_type changes on branch, is there another entry point I'm missing?
<wgrant> StevenK: accessartifact_flatten_bug possibly wants to be renamed to reflect the fact that it now does branches too -- it's an INSERT OR UPDATE OR DELETE trigger on AAG and APA, so it handles the other two entrypoints.
<StevenK> Right.
<StevenK> accessartifact_update_artifacts ?
<wgrant> Or accessartifact_denorm_to_artifacts, or something like that.
<StevenK> I like that, let's do that.
<StevenK> I have to DROP the trigger, I can't just replace which function it calls?
<wgrant> Right, you'll need to drop the trig and func and recreate them with the new names.
<StevenK> Yup
<StevenK> And the comment
<StevenK> There are four PERFORM accessartifact_flatten_bug in 16-0, do they need to change too?
<wgrant> Oh right, there's an extra layer of indirection.
<wgrant> Don't need to recreate the trigger.
<wgrant> Just rename accessartifact_flatten_bug and replace its name in the trigger func
<wgrant> Bah, but the trigger func is called accessartifact_maintain_bugtaskflat_trig
<wgrant> So
<wgrant> Drop trigger
<wgrant> Drop accessartifact_maintain_bugtaskflat_trig
<wgrant> So needs renaming.
<wgrant> Drop accessartifact_flatten_bug
<wgrant> Recreate the three with new names.
<StevenK> wgrant: http://pastebin.ubuntu.com/1038425/
<wgrant> StevenK: Pretty good. We can probably dump some of the SECURITY DEFINERs now, though.
<wgrant> The internal functions shouldn't have them.
<StevenK> So accessartifact_maintain_denorm_to_artifacts_trig can lose it, at least.
<wgrant> branch_denorm_access, bug_flatten_access and bug_build_access_cache are probably the only ones that need it.
<wgrant> All the other functions either do nothing interesting or are internal.
<wgrant> Can possibly even do without bug_build_access_cache.
<wgrant> Since the only other place that is called is the main bugtaskflat function.
<wgrant> Which is itself a definer.
<wgrant> StevenK: Do you know what SECURITY DEFINER does? :)
<StevenK> wgrant: Nope, I was trusting your judgement.
<wgrant> StevenK: It's setuid.
<StevenK> Oh, argh.
<wgrant> It causes the function to execute as the role that created it.
<wgrant> Rather than the role that is executing it.
<StevenK> Right, so the less functions that have it, the better.
<wgrant> It's handy to put it everywhere for testing. And it's not a huge issue in our DB, since all the users are pretty much trustworthy.
<wgrant> But yes, should be minimised.
<wgrant> StevenK: Functions without SECURITY DEFINER also don't need the custom search_path.
<wgrant> (same as you don't let a setuid binary execute things from PATH)
<StevenK> wgrant: Is the converse also true?
<StevenK> Do functions with SECURITY DEFINER need SET search_path = ... ?
<wgrant> StevenK: Yes.
<wgrant> Since names without an explicit schema (ie. everything in all of our DB functions) use the search_path.
<StevenK> wgrant: http://pastebin.ubuntu.com/1038442/
<wgrant> StevenK: CREATE TRIGGER accesspolicyartifact_maintain_denorm_to_artifacts_trigger
<wgrant> That "maintain_" probably shouldn't be there
<wgrant> Ditto in the second trigger.
<wgrant> StevenK: I'd drop the SECURITY DEFINER from bug_build_access_cache
<StevenK> wgrant: bug_build_access_cache was in your list of three that need it :-)
<wgrant>     LANGUAGE plpgsql SET search_path = public AS $$
<wgrant> That search_path can die.
<wgrant> 14:52:40 < wgrant> Can possibly even do without bug_build_access_cache.
<wgrant> 14:52:50 < wgrant> Since the only other place that is called is the main bugtaskflat function.
<wgrant> 14:52:59 < wgrant> Which is itself a definer.
<StevenK> wgrant: http://pastebin.ubuntu.com/1038447/
<wgrant> StevenK: Plausible.
<StevenK> wallyworld: Where's your branch that drops use of branch.explicitly_private and friend up to?
<lifeless> wgrant: basic_fake_auth - 3.2 helper, don't need your python bit; another backport from 3.2 / build as separate package.
<StevenK> wgrant: Shall I push up the DB patch, or do you want to ponder a bit more?
<wgrant> StevenK: Push it up.
<wgrant> StevenK: Well
<wgrant> StevenK: First check that it works.
<StevenK> wgrant: make schema works
<wgrant> That doesn't mean much :)
<wgrant> lifeless: Huh
<wgrant> lifeless: Indeed, rather handy.
<wgrant> http://wiki.squid-cache.org/ConfigExamples/Authenticate/MultipleSources describes a fair bit of what I discovered on Friday.
<wgrant> It seems.
<StevenK> wgrant: test_bug_mirror_access_triggers passes, still doesn't mean much? :-)
<lifeless> wgrant: yeah, shocking, we have docs
<wgrant> lifeless: I wouldn't really call that docs.
<wgrant> StevenK: I'd check it works for branches too.
<wgrant> StevenK: in psql
<StevenK> wgrant: http://pastebin.ubuntu.com/1038476/
<wgrant> StevenK: Check the toggling information_type toggles access_grants between {7} and NULL
<StevenK> wgrant: http://pastebin.ubuntu.com/1038480/
<wgrant> Marvellous.
<StevenK> wgrant: Happy to check APA too, just not sure what policy should be.
<wgrant> Anything.
<wgrant> Anything at all.
<wallyworld> StevenK: it's merged
<wgrant> Deployed too, isn't it?
<wallyworld> probably
<wgrant> If it was deployed yesterday, rerun the usual information_type vs transitively_private consistency checks on prod, and drop the columns from code entirely.
<StevenK> wallyworld: Do you want to handle that?
<StevenK> Or shall I?
<wallyworld> StevenK: i've got a few branches on the go, so feel free to take it. i thought you wanted to be the one to drop the columns etc?
<StevenK> wgrant: http://pastebin.ubuntu.com/1038483/ -- not sure if that's what usually happens.
<StevenK> wallyworld: Yes, where drop the columns is ALTER TABLE branch DROP COLUMN :-)
<wgrant> StevenK: bug
<wgrant> StevenK: Or you're mis-testing.
<wgrant> Or my code is crap.
<StevenK> wgrant: I'm happy to accept that I'm mis-testing.
<StevenK> wgrant: How do I check?
<wgrant> It's unfortunately not your fault, I don't think.
<StevenK> Anything I can fix in 16-6?
<wgrant> It's a bug in 16-6, so I hope it's fixable there, indeed.
<StevenK> Ah. Can I have a clue? :-)
<wgrant> Not until I do.
<StevenK> I'll quickly put together a branch that drops *_private from the branch model.
<wgrant> Great.
<wgrant> lalala i am stupid
<wgrant> Well
<wgrant> lalalal postgres is visual basic
<wgrant> (its array indices are based at 1, not 0)
<wgrant> StevenK: Does that tell you how to fix it? :)
<StevenK> Bwahaha
<StevenK> One character bug
 * wgrant fixes two timeouts.
<StevenK> Hm, this could be difficult to rip out.
<wgrant> Howso?
<wgrant> Should be delet
<wgrant> delete
<wgrant> delete
<wgrant> delete
<wgrant> commit
<wgrant> push
<wgrant> bzr pqm-submit
<StevenK> Given the branch browser code uses copy_field(IBranch['explicitly_private'])
<StevenK> And one test still uses transitively_private, so wallyworld fails.
<wgrant> Oh.
<wallyworld> StevenK: i told you about that one on the call the other day
<wgrant> wallyworld's branch didn't turn them into properties?
<wgrant> SO the columns are still being read :(
<wgrant> Fixing that is top priority and pretty trivial.
<StevenK> Right.
<wgrant> Only after that's deployed to appservers can we stop writing to the columns.
<StevenK> Yeah
<wallyworld> StevenK: i thought your next branch would be the one to drop them from the model code
<StevenK> wgrant: Give me a few minutes and I'll throw up a branch.
 * StevenK blinks, just having re-read that sentence.
<wallyworld> hence i left that one behind since it can be done at the same time
<StevenK> wallyworld: As wgrant says, we can't. They need to depend fully on information_type under the covers.
<wallyworld> but the one remaining case is just some bug text view
<wallyworld> which is not really used
<wgrant> Plus the API :)
<wallyworld> which i thought would be done when the class properties were dropped
<wallyworld> i basically changed all the sql queries to use unfo type
<wgrant> wallyworld: There's a race there.
<wgrant> We need to stop reading before or at the same time as we stop writing.
<wgrant> But deploys aren't atomic and isolated.
<wgrant> So the read removal needs to be done in an earlier deploy.
<wallyworld> sure
<wallyworld> that's what i though StevenK was doing next
<wallyworld> since he expressed an interest in doing the remainign bits
<StevenK> I think lib/lp/code/model/tests/test_branch_privacy_triggers.py will fail with this change.
<wgrant> It will.
<StevenK> Nuke from orbit?
<wgrant> Ideally replace with something that tests the new ones :)
<StevenK> wgrant: Oh, sorry, I don't mean the branch that adds 16-6, I mean the branch that replaces explicitly_private and transitively_private with properties.
<wgrant> StevenK: Hm?
<StevenK> wgrant: http://pastebin.ubuntu.com/1038507/
<wgrant> StevenK: You need to preseve the cols, just under different names.
<StevenK> But nothing reads them anymore
<wgrant> Not after that diff is deployed, no.
<StevenK> Bug._private was for Storm's benefit
<wgrant> But as I said above, we need to keep setting the DB columns until everything has stopped reading them.
<wgrant> Now, if you try to stop both reading and writing in the one deploy, you're breaking things.
<wgrant> Since one appserver will have stopped writing them while others are still reading them.
<wgrant> == leak
<StevenK> Ah.
<StevenK> Curses.
<wgrant> I have that effect on people.
<StevenK> wgrant: http://pastebin.ubuntu.com/1038511/
<StevenK> wgrant: And 16-6 psql test: http://pastebin.ubuntu.com/1038516/
<wgrant> +    # These can die after the UI is dropped.
<wgrant> I forget -- do we care about API?
<wgrant> StevenK: That looks unbroken indeed.
<StevenK> transitively_private is not exported, so can die.
<StevenK> explicitly_private is exported
<StevenK> As is private
<StevenK> Because a non-duplicated interface is a bad one.
<wgrant> explicitly_private is private
<StevenK> wgrant: They're both on the interface. Seperately.
<wgrant> Ah
<wgrant> Erm
<wgrant> Storm, what are you doing.
<wgrant> Bad Storm.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/branch-privacy-properties/+merge/109987
<StevenK> wgrant: Can haz review so I can do stuff while ec2 munches on it?
<wgrant> StevenK: Done, with comment.
<adeuring> good morning
<rvba> Hi frankban, are we not on review duty today?
<frankban> rvba: yes we are.
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivorykr8 | On call reviewer: frankban* (rvba) | Firefighting: - | Critical bugs: 3.47*10^2
<rvba> frankban: ;)
<jml> benji: thanks for adding that readme :)
<cjwatson> benji: Isn't it a bit odd that logs is still in .bzrignore?  (And also logs/README.txt via the logs/* entry.)
<benji> cjwatson: yea it is a bit odd; I'll remove the directory itself.  I wonder how to ignore the non-README contents of the directory though
<jml> benji: one (crappy) way, rename README to _README and ignore [A-Za-z]*
<adeuring> stub: do you know the reason why ftq('aaa.bbb') returns the tsquery 'aaa' & 'bbb' | 'aaabbb'? Can't we simply drop this logic?
<stub> adeuring: We can certainly drop and/or change the logic. Nobody has been able to really articulate what a search is actually supposed to do beyond 'be like google, but with substring matching' in the past.
<adeuring> stub: ok, I'll see what we can remove or if we can make the function at lkeast simpler
<stub> adeuring: I don't think any tests beyond the text search ones test the weird cases, and feel free to rewrite that horrible ftq() function entirely. Nobody should ever let me write parsers because I don't know what the hell I'm doing.
<adeuring> stub: ;)
<stub> adeuring: I do recommend reading the text search docs in the PG manual if you haven't already - if nothing else, it will get you familiar with the language and what problems you should be considering
<adeuring> stub: yeah, I'm already looking into the docs
<ivory> frankban: could you please look at this mp: https://code.launchpad.net/~ivo-kracht/launchpad/bug-353097/+merge/110018
<frankban> sure ivory
<jml> so a private team can't be a product owner?
<jml> so, I was thinking about Launchpad's microservice idea last night
<jml> and thinking about splitting out codehosting
<jml> and then I thought that you'd probably want to split out the xml-rpc service that codehosting talks to
<jml> but then I thought, LP used to have a separate xml-rpc service that codehosting talked to, and we rolled it into the appserver
<jml> What has changed between then and now such that splitting it out again is a good idea? (Is it a good idea?)
<jml> If a private team can't be a product owner, then I guess that means that private teams cannot have commercial subscriptions
<czajkowski> jml: you did a lot of thinking last night :)
<wgrant> jml: Private teams can be project owners.
<wgrant> jml: I'm not sure if it's meant to be allowed yet, but we intend to allow it in future, and there is at least one case on production already that I know about.
<jml> wgrant: http://paste.ubuntu.com/1038782/ says otherwise.
<wgrant> jml: Yeah, as I thought, it's not currently allowed. But https://launchpad.net/convoy is the production example I was thinking of.
<jml> wgrant: ok, thanks
<jml> we're in testfix mode now. I'm guessing it's due to the failure on the db_lp slave.
<wgrant> It was more than likely spurious, can you retry?
<wgrant> If you're there.
<jml> I am. Just did.
<wgrant> Thanks.
 * jml wonders how long it takes for pqm to switch out of testfix
<lifeless> jml: splitting out the bzr codehosting stuff doesn't imply splitting out the xmlrpc service
<jml> lifeless: I know that.
<lifeless> jml: the xmlrpc service talks to the LP db, it needs to stay unsplit, most vigorously.
<lifeless> 22:18 < jml> and then I thought that you'd probably want to split out the xml-rpc service that codehosting talks to
<lifeless> jml: ^ what did you mean by that then ?
<jml> lifeless: I meant "want" rather than "need"
<lifeless> jml: so I'm asserting that want is wrong.
<jml> lifeless: so what's the principle, that only the appserver should talk directly to the db?
<lifeless> yes.
<lifeless> thats declared in the SOA docs FWIW.
<lifeless> most of our scripts should be moved out of the core tree into API clients, for instance.
<lifeless> the buildmaster likewise.
<jml> lifeless: where abouts?
<lifeless> one or more anciliary projects
<jml> lifeless: yeah, the buildmaster is the other one I think about from time to time
<lifeless> validated test doubles to preserve the contracts they need and let them be tested robustly.
<jml> lifeless: it's fruit is higher up the tree
<jml> lifeless: I mean, whereabouts is the "only appserver should talk to the DB" written? I don't recall it and a quick grep doesn't find it.
<jml> s/it's/its/
<lifeless> its not a rule-rule yet, because it would create massive instantaneous debt, but its a meme I keep harping on. The closest actual rule-thing for it is...
<lifeless> https://dev.launchpad.net/ArchitectureGuide/ServicesAnalysis#Contracts_and_protocols
<jml> lifeless: I guess I'm mostly interested in why it's a desirable thing.
<lifeless> which has 'Access to a particular form of persistence (e.g. the librarians files-on-disk, or a particular postgresql schema) requires being in the same source tree as the definition of that persistence mechanism.'
<lifeless> jml: managing complexity
<jml> "Access to a particular form of persistence (e.g. the librarians files-on-disk, or a particular postgresql schema) requires being in the same source tree as the definition of that persistence mechanism."
<lifeless> jml: two different servers means either db model code running in two different (perhaps wildly so) contexts, or just redundancy in layout
<jml> lifeless: that makes sense, thanks.
<lifeless> jml: anytime, I'm sure I put that in there somewhere
<wgrant> However, most of codehosting's model probably doesn't belong in the main LP DB.
<wgrant> :)
<lifeless> wgrant: lp:~wgrant/launchpad/dspc-trgm-indices into lp:launchpad
<lifeless> sholdn't thast be db-devel ?
<wgrant> lifeless: Should be doable hot.
<lifeless> absolutely, but,..
<wgrant> Hot patches go to devel.
<lifeless> unless jml butchered it, no they don't.
<wgrant> We've done it this way since last year...
<wgrant> Why wouldn't they?
<wgrant> Hot index patches have always been applied live and landed on devel.
<lifeless> nearly 11pm, not the time for me to be thinking about this.
<lifeless> worth checking the history on https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess and seeing if a 'cleanup' actually made it incorrect.
<wgrant> I haven't read that page this year :)
<wgrant> So a recent cleanup certainly hasn't changed my view of it.
<wgrant> Anyway, landing live-applied DB patches on db-devel provides no benefit and lots of confusion, so we shouldn't do it.
<wgrant> But tomorrow :)
<lifeless> wgrant: qa might be part of it, I'll have to page it in. Tomorrow
<lifeless> also, you edited it in may, fibber ;)
 * lifeless goes to sleep
<wgrant> I didn't read it.
<wgrant> I knew exactly the part that needed fixing :)
<jml> lifeless: I'm pretty sure I didn't touch hot vs cold patches.
<jml> g'night.
<wgrant> cjwatson: Your column rename is done.
<wgrant> cjwatson: You can remove the compat code.
<wgrant> jml: Yours should happen on Friday.
<jml> wgrant: hurrah.
<jml> I'm keen to mark the LP side of our LP integration work as done.
<stub> wgrant: We already have a GIN index on distributionsourcepackagecache(fti)
 * stub checks if his branch landed
<wgrant> stub: Oh right, just not in devel yet.
<wgrant> It's in db-devel
<wgrant> Next up
 * wgrant removes that one.
<ivory> frankban, rvba: thanks for the reviews
<rvba> You're welcome ivory.
<frankban> ivory: welcome
<cjwatson> wgrant: Great, thanks.
<cjwatson> I could use reviews of https://code.launchpad.net/~cjwatson/launchpad/pocket-permissions/+merge/110037 and https://code.launchpad.net/~cjwatson/launchpad/packageset-score-rename-finish/+merge/110046
<cjwatson> gmb: I tried your suggestion in https://code.launchpad.net/~cjwatson/launchpad/export-change-override/+merge/109549 (http://paste.ubuntu.com/1038961/), but two problems: firstly, the title= override doesn't seem to take so the title is wrong in apidoc (it says "The priority being published into", which is less helpful); secondly, tests fail with http://paste.ubuntu.com/1038963/ suggesting that the magic translation ...
<cjwatson> ... doesn't actually happen.
<cjwatson> gmb: Do you know what I might be doing wrong?
<deryck> adeuring, https://plus.google.com/hangouts/_/5aac2d43bc403742ae3a25334fb32476561cab82?authuser=0&hl=en
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivorykr8 | On call reviewer: frankban* (rvba), abentley | Firefighting: - | Critical bugs: 3.47*10^2
<jamestunnicliffe> Hi, anyone got any idea why the expander icons have vanished from https://launchpad.net/~linaro-infrastructure/+upcomingwork ? The work with a recently merged (an hour ago) local instance.
<jml> jamestunnicliffe: I think there might be a  known issue with sprites & chrome
<jml> jamestunnicliffe: but I'm not 100% sure, and I don't have a bug number.
<jamestunnicliffe> jml: Thanks. Firefox does show the sprites.
<rick_h> looking, I wasn't aware we had any sprite issues.
<sinzui> jamestunnicliffe, I think wallyworld has a fix for that. This affects webkit browsers.
<rick_h> ah, it's the empty content
<rick_h> if I put any content in there I get the sprite
<sinzui> rick_h, it is more subtle than that. There is a leading space *before* the anchor. New webkit can show empty content, but space normalisation and stripping interfere
<sinzui> rick_h, I still believe empty anchors are invalid though
<rick_h> sinzui: yea, I'm pretty sure they are as well.
<rick_h> wheee, the browser dance here's your chance...browser dance...do the browser dance... :)
<sinzui> webkit wins one technical merit. gecko wins because it does what you expects
<sinzui> jamestunnicliffe, report the missing expanders as a new bug and add the regression tag. wallyworld's fix does not fix the missing expanders
<jamestunnicliffe> sinzui: Will do. Thanks for looking into it.
<sinzui> deryck, abentley, mrevell, flacoste: I have written all that I recall proprietary projects need to do with bug linking. I think we can find the essential use cases from this http://blog.launchpad.net/general/bug-linking-part-2
<mrevell> Thanks sinzui
<abentley> sinzui: thanks.
<czajkowski> sinzui: just posting it t places now
<czajkowski> thanks
<flacoste> adeuring: congratulate ivorykr8 for his first landing!
<sinzui> oh, did he use my suggested text?
<adeuring> flacoste: thanks, but congrats shoukd go to ivory
<flacoste> ah, i wasn't sure if he renamed himself
<flacoste> ivory: congrats on your first landing! should be out to qastaging pretty soon
<czajkowski> ivory: well done
* flacoste changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: frankban* (rvba), abentley | Firefighting: - | Critical bugs: 3.47*10^2
<sinzui> it only took me 18 months to write that sentence
<deryck> sinzui, thanks for writing that up.
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: abentley | Firefighting: - | Critical bugs: 3.47*10^2
<czajkowski> jamestunnicliffe: have you seen  https://bugs.launchpad.net/launchpad/+bug/1004416   wondering how similar it is to the one you just reported.
<_mup_> Bug #1004416: Work Items not allowing users to edit them properly <work-item-tracker> <Launchpad itself:Triaged by linaro-infrastructure> < https://launchpad.net/bugs/1004416 >
<sinzui> czajkowski, I think they are the same issue
<czajkowski> sinzui: I thought so too but wanted to check as I've tried to recreate the blueprint one but not seeing the issue, but I know others are having issues with them
<timrc> Hi, my understanding is that subscribing an individual to a Private PPA via the Manage Access view gives them "read" access to the PPA.  Someone just reported that they attempted to do this for an individual, but they were unable to access +packages? How would I go about giving an individual read-only access to the PPA in terms of installing packages and accessing the UI?
<lifeless> sinzui: bug 1012787 - what mails are going to the team membership ? {I'm wondering if its unneeded complexity and that the actual problem is unnecessary notification emails/implicit subscriptions}
<_mup_> Bug #1012787: cannot team restrict emails to go to the team admin <email> <linaro> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1012787 >
<sinzui> lifeless, joey can provide example of emails that he does not want members to get
<joey> aloha
<lifeless> aloha
<lifeless> so, what are the symptoms you're seeing ?
<joey> I'll email you, lifeless, and sinzui,  a typical example
<sinzui> thank you
<joey> ok it's on it's way to your @c.c ids
<joey> sinzui: lifeless - btw there's one more thing
<joey> sinzui: lifeless - when IS makes a change to the teams for example, I get a raft of "Security exposure! This guy isn't part of Linaro. He's touching our stuff!" emails
<joey> sinzui: lifeless - It's a headache since it happens a lot
<joey> sinzui: lifeless - that's part of the reason I went to ~linaro-sysadmins so they see my name on it and mostly they ignore it
<sinzui> joey. That last part is very interesting
<joey> sinzui: yeah I have to imagine other projects may get that from time to time
<joey> sinzui: especially when they touch private teams
<joey> czajkowski: ^^
<lifeless> joey: that refers to mail from linaro folk that don't recognise e.g. stephanie miller ?
<joey> lifeless: correct
<joey> lifeless: they rightfully query me which is good but you see the overall issue
<joey> I guess to some extend Linaro might be a special case but if was with, say, Mozilla, and someone did that  they didn't know they would probably have a similar reaction
<czajkowski> sinzui: in all the disclosure work you're doing, is there anything being done to tell people the bug they are trying to view is a private bug so they understand when they cant find it ?
<sinzui> czajkowski, We are still debating this. private things are officially 404 and we do not talk about privacy
<czajkowski> well it would cut down on questions we get a lot of https://answers.launchpad.net/launchpad/+question/200348
<sinzui> czajkowski, the underlying problem is the people (and bots) are inconsiderate of others when marking bugs duplicates
<czajkowski> it's not clear to users why they cant see it but it's being referenced so it's confusing
<sinzui> I hope that Lp will force users to be considerate when the Orange Mafia work on bug linkings
<czajkowski> if there is no bug open on this will create one as I think it'd be good to address
<sinzui> czajkowski, private bugs in non-commercial projects should never be the master of a duplicate. People and bots are being complete and utter bastards to other users
<czajkowski> the current error or lack of information is not ideal.
<czajkowski> sinzui: no arguements from me there :)
<sinzui> We have several open bugs
<czajkowski> sinzui: I only understand this now from working on LP, before I used to find it rude and annoying, now I just find it annoying
<sinzui> bug 434733 I cited in my blog bost today is the one users most care about
<_mup_> Bug #434733: marking public bug as duplicate of private bug leads to confusing UI <chr> <confusing-ui> <lp-bugs> <privacy> <qa-untestable> <ui> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/434733 >
<joey> sinzui: the way I've begun to look at things now is with a UX hat on.  It's not sufficient to say "my service isn't responsible".  If the service is able to help mitigate a social problem, oftentimes it's worth the time and effort (assuming it's done correctly)
<joey> sinzui: and it helps to clear up ill perceptions
<lifeless> czajkowski: https://bugs.launchpad.net/ubuntu/+source/apport/+bug/764414
<_mup_> Bug #764414: private master bugs are confusing and lead to more duplicate filings <pet-bug> <apport (Ubuntu):Triaged by pitti> < https://launchpad.net/bugs/764414 >
<lifeless> joey: ^
<lifeless> thats the root cause; its what causes the bad experience.
<lifeless> We could fix it in a heartbeat, by stopping apport doing it.
<lifeless> But that would be a litle unreasonable to do unilaterally.
<czajkowski> lifeless: thanks.
<czajkowski> never seen a tag - pet-bug before
<joey> yeah I remember that bug :-)
<joey> caused me grief in OEM
<czajkowski> joey: why ?
<joey> in the OEM days public was public and private was NDA'd so you couldn't even acknowledge the two
<joey> so you might have a public Ubuntu bug that was created and a dup of a private OEM bug, but you could do touch it
<czajkowski> ah gotcha
<joey> but anyway, there is usually a solution if you can think out of the box and, crucially, have some resource to work on it
<joey> in the bug lifeless mentioned, you can't futz with apport because it'll break things
<joey> but you could have a background service process that interrogates bugs and does bug matching...but that is very expensive
<joey> and I fear the algorithm would be more prone to failure than success
<lifeless> joey: we can change apport, pitti in principle agreed to what I suggested.
<lifeless> joey: the problem is resources.
<joey> lifeless: Good to know. I guess you solved it somehow other than making apport aware of private bugs?
<joey> apport on the client machine that is
<lifeless> joey: huh? this isn't client side.
<lifeless> joey: its all server side + how devs do their bug management.
<joey> lifeless: ok you just confirmed my suspicion
<lifeless> joey: the apport retracer is the thing that makes bug A a dupe of bug Master.
<lifeless> And bug Master is allowed to be private.
<joey> ah right, that's where I was headed with the service process comment above...was just being a bit general
<joey> still, you need dev time to make it happen
<joey> anyway, dead horse :-)
<lifeless> joey: *we* don't, Ubuntu do :)
<lifeless> joey: this isn't something we can directly fix :(
<joey> lol
<lifeless> Even supplying a patch isn't enough, the stuff that needs finesse and time is social.
<lifeless> sinzui: so joeys mail; the point solution there is perhaps 'only tell admins about team join events', that is there is no action for members to take.
<lifeless> or maybe even only owners.
<sinzui> lifeless, admins I think. the owner can be an admin if he chooses
<lifeless> members are not expected to do anything as a result of a team join (ignoring mailing lists for a second)
<sinzui> lifeless, and if this is the case, I suspect we can duplicate this bug
<lifeless> sinzui: for mailing lists, they still don't need to do anything, unless their list preference is 'ask me about subscribing when I join a list team'
<lifeless> joey: are there other sorts of mails than that one that you sent us ?
<sinzui> lifeless, agreed
<joey> lifeless: that's the standard mail I receive along with the "who's Stephanie Miller and why is she touching our Linaro stuff?" emails.
<joey> sinzui: lifeless - I'm ok with admins vs owners.
<joey> sinzui: lifeless - just anything other than rank and file members
<joey> If I think about it, admin is the probably the correct decision
<sinzui> joey owners can leave the team, they have delegate the running of it to the admins.
<sinzui> owner can take back control if they want
<joey> sinzui: well in my case, ~linaro-sysadmins are not members, only owners
<joey> sinzui: so I guess I'd need both owner and admin
<sinzui> ah, so they should not ever get notifications.
<joey> I'm sure if you do it one way someone will complain that they want it the other
<joey> sinzui: in my case, having ~linaro-sysadmins as part of the team screws up the milestone view in LP so we have to remove them for metrics and such
<sinzui> joey, can I see en example of a team that ~linaro-sysadmin owns
<joey> sinzui: think  status.linaro.org
<joey> sinzui: ~linaro
<joey> sinzui: actually, anything ~linaro and ~linaro-*
<joey> sinzui: also we have some private or access restricted teams and we don't want members/partners/etc looking at the team list and wondering who the sysadmins are
<joey> just prevents questions
<joey> is that a firm requirement? no
<sinzui> joey, https://launchpad.net/~linaro/+members says ~linaro-sysadmin are the only admins
<joey> sinzui: yeah that's the one case :-)
<joey> sinzui: look at any of the sub-teams
<joey> sinzui: I suppose the safer approach is admins & owners but the more expensive approach is a tick box for each.  Just me thinking out loud. You're the right brain to noodle on it.
<sinzui> joey, we really make a point of not contact owners that are not admins
<joey> sinzui: I think I can live that
<joey> sinzui: I miss the change notifications but frankly, if something is not working, someone is going to me
<joey> sinzui: and eventually ping the linaro sysadmins
<joey> sinzui: and I do want the team admins to be fully engaged in their teams
<joey> sinzui: so I can deal with admin only
<joey> sinzui: I've often wanted a "reason box" when adding people just like deleting them but that's another story for another day :-)
<sinzui> joey. I too see a cluster of issues here that would be nice to fix in a single branch. Limit who gets the email, fix the headers and rationale, state in the UI who gets the emails
<sinzui> jcsackett, mumble?
<wgrant> benji: Is your removal of logs/ just a day after its creation deliberate?
<cjwatson> Hm, OCR not on channel
<cjwatson> I'd like to get https://code.launchpad.net/~cjwatson/launchpad/pocket-permissions/+merge/110037 landed in the not too distant future, since the code on prod is broken (not a regression, but still).  Anyone up for a review of some URL traversal code?
<wgrant> cjwatson: Looking.
* wgrant changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<wgrant> 37	item = ("type=packageset&item=%s&series=%s" %
<wgrant> 38	(self.context.package_set_name,
<wgrant> 39	self.context.distro_series_name))
<wgrant> Ew
<StevenK> wgrant: If you make some broad statement about that line wrapping being as bad as mine, I might have to get offended. :-P
<wgrant> Not line-wrapping.
<wgrant> That code is just wrong :)
<cjwatson> wgrant: That's the pre-existing bit isn't it?
<wgrant> It is, yes.
<wgrant> It's also horrible.
<wgrant> But not your fault.
<cjwatson> (For my edification, what's horrible?)
<wgrant> You can't just stuff arbitrary strings into application/x-www-form-encoded and hope it works.
<wgrant> eg. if a name contains + that will break.
<cjwatson> Entertaining
<wgrant> application/x-www-form-urlencoded, that is
<cjwatson> Should it be escaped or totally rearranged? :-)
<cjwatson> (Academic since I hope not to have to touch that again.)
<wgrant> "escaped" is the wrong way to think about it. The query string should be formed using urllib.urlencode({'type': 'packageset', 'item': self.context.package_set_name, 'series': self.context.distro_series_name}) or so.
<wgrant> cjwatson: r=me
<wgrant> Feel free to rewrite that doctest to be unit tests that are about 90% smaller, but I doubt I can convince you to do that...
<cjwatson> Well, I've done it for a bunch of my other branches recently
<cjwatson> In this case I sort of want to get the fix in quickly rather than spending another half a day rewriting doctests, though
<wgrant> Exactly :)
<cjwatson> How about I queue it up as first against the wall for the next time I want LoC?
<wgrant> It's fine as it is.
<wgrant> But yes, that would indeed be a good idea.
<cjwatson> "fine" in the limited sense that any doctest can be fine.
<wgrant> doctest-slaying is a pretty good way to get LoC
<cjwatson> Most of the time, yeah.
<cjwatson> I'm not convinced my xx-*-package-publishing.txt rewrite is going to be much of a wine
<cjwatson> *win
<cjwatson> wgrant: Thanks, sending to EC2.  I hope it sticks this time.
<wgrant> cjwatson: Seems like it should. Thanks for fixing that.
<lifeless> hmm, js.js for foreign sourced scripts
<cjwatson> wgrant: I'd still like to get the queue-api review finished, but that might be a bit much for first thing in the morning. :-)
<wgrant> lifeless: Along with its nodejs integration, js.js.js?
<wgrant> cjwatson: Looking at your others new.
<wgrant> queue-api will be done gradually over the day, I suspect :)
<wgrant> packageset-score-rename-finish approved
<cjwatson> Heh, yeah
<wgrant> What's export-change-override up to?
<cjwatson> I queried gmb about his comment earlier today, but haven't heard back.
<cjwatson> 13:58 <cjwatson> gmb: I tried your suggestion in https://code.launchpad.net/~cjwatson/launchpad/export-change-override/+merge/109549 (http://paste.ubuntu.com/1038961/), but two problems: firstly, the title=
<cjwatson>                  override doesn't seem to take so the title is wrong in apidoc (it says "The priority being published into", which is less helpful); secondly, tests fail with http://paste.ubuntu.com/1038963/
<cjwatson>                  suggesting that the magic translation ...
<wgrant> Ah
<cjwatson> 13:58 <cjwatson> ... doesn't actually happen.
<cjwatson> 13:59 <cjwatson> gmb: Do you know what I might be doing wrong?
<wgrant> Saw that but forgot about it, indeed.
<wgrant> cjwatson: If you're going to make it more widely available, you might consider fixing bug #530020 while you're there
<_mup_> Bug #530020: change-override.py should forbid modifications of the RELEASE pocket of non-development series <lp-soyuz> <soyuz-ftpmaster-tools> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/530020 >
<cjwatson> p-s-r-f: thanks, can I have Status: Approved and I'll land it?
<cjwatson> Oh, interesting
<wgrant> Hm, I did.
<wgrant> Maybe I have too many MPs open so longpoll is killing the world.
<wgrant> Indeed.
 * StevenK stabs longpoll
<wgrant> Fixed.
<lifeless> wgrant: js.js.js would be most fun
<cjwatson> Happy to stall export-change-override on 530020, though probably not tonight.
<wgrant> cjwatson: Should be a ~2-line fix + tests, and it seems like a good idea to do it before exposing it, so that'd be nice, thanks.
<cjwatson> Well, ~4 'cos there are two changeOverride methods to consider, but yes :-)
<wgrant> True.
<StevenK> wgrant: Oh, HAH.
<StevenK> wgrant: IBranchNamespace.createBranch() still had explicitly_private in it.
<wgrant> That would do it.
<StevenK> I'm using test_branchmergeproposal as my canary, and I've dropped from 305 failures to 24. Something else is touching transitively_private inappropiately.
<StevenK> Right, 1 failure, and that's 2.7
<StevenK> wgrant: So I want to stop setting them too?
<wgrant> The DB columns still need to be set in the first branch.
<wgrant> But they must not be read.
<wgrant> wallyworld: Bug #1012912 might be of interest. Found it during QA of your changes from yesterday, but it's not a regression in them.
<_mup_> Bug #1012912: +filebug choice popups break when the user is unprivileged <disclosure> <javascript> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1012912 >
<wallyworld> ok, will fix thanks
<StevenK> wgrant: http://pastebin.ubuntu.com/1039933/
<wgrant> StevenK: If that fixes it.
<StevenK> wgrant: Drops the failures from 305 to 1, and that one is a 2.7 thing.
<wgrant> Sounds good.
 * StevenK re-tosses at ec2
#launchpad-dev 2012-06-14
 * StevenK stabs the lack of completion for bzr rm
<wgrant> StevenK: How's the ec2 run looking?
<StevenK> branch-privacy-properties => devel        [OK]       (up for 2:33:55) i-2ca43755
<wgrant> Good good
 * StevenK smacks wgrant.
<StevenK> steven@undermined:~/launchpad/lp-branches/devel% grep -c '^# ' lib/lp/bugs/model/bugtaskflat.py
<StevenK> 0
<wgrant> Oops, indeed.
<StevenK> wgrant: Do I need to add access_policy and access_grants to {I,}Branch, or are they internal only?
<StevenK> I note BTF is missing access_policies and access_grants on its model.
<wgrant> StevenK: We'll likely add them later, but you can't add them in this branch.
<StevenK> wgrant: Sure, they have to wait until the DB patch hits prod and devel
<wgrant> And then you'll add them if you need them.
<wgrant> And not add them if you don't need them.
<wgrant> Same as any other piece of code :)
<StevenK> I was plotting writing tests for the DB patch, which made me ponder model code.
<wgrant> lib/lp/bugs/tests/test_bugtaskflat_triggers.py may be of some interest.
<wgrant> wallyworld: Why'd you delete that branch? It attracted the Scanner's Curse?
<wallyworld> yes
<wgrant> :(
<wgrant> I still haven't been able to reproduce that locally.
<wallyworld> it won't delete properly either it seems
<wgrant> Yeah, you can't delete lp:launchpad branches any more.
<wgrant> Not once they've started to scan.
<wallyworld> well it lets me try
<wgrant> Sure.
<wallyworld> and then errors in a funny way
<wgrant> Then it 502s.
<wallyworld> why?
<wallyworld> it sholdn't let me delete
<wallyworld> if i'm not supposed to
<wgrant> It should let you delete.
<wgrant> You are supposed to.
<wgrant> It just doesn't work.
<StevenK> Because BranchRevision exists
<wallyworld> hooray!
<wgrant> I'm not sure exactly what's going on, but I *believe* that the BranchRevision delete is unflushed at the end of the request.
<wgrant> So the commit() in the response post-processing flushes the 90krow delete outside the timeout.
<wgrant> And crashes.
<wallyworld> not good
<wgrant> s/outside the timeout/outside the timeout exception handler/
<wgrant> It probably turns into a normal timeout if we flush explicitly after the delete.
<wgrant> wallyworld: I'll review that branch after lunch, btw.
<wallyworld> wgrant: no problem. whenever :-)
<StevenK> Wow, a smile from wallyworld.
<wallyworld> >:-(
<mwhudson> isn't branchrevision deleted by an ON DELETE CASCADE?
<mwhudson> which is another way of saying the same thing i guess
<mwhudson> btw buildout is very slow if you strace it :)
<StevenK> s/ if you strace it//
<wgrant> mwhudson: Ah, indeed.
<wgrant> That explains why it's unflushed.
<mwhudson> could this be a pg9.1 regression?
<wgrant> No
<mwhudson> or has it been a problem for a while?
<wgrant> It's been a problem forever, it's just more prevalent now with the lower timeout.
<wgrant> You used to have to retry a couple of times to delete big branches.
<StevenK> Can we bump the timeout for Branch:+delete?
<StevenK> The 'proper' fix is the death of destruction of most of BranchRevision
<mwhudson> yes
<wgrant> wallyworld: 70+ if safe_hasattr(user_or_reference, 'id'):
<wgrant> wallyworld: Can that be replaced by IPerson.providedBy(user_or_reference)?
<wallyworld> no, cause it's not always a person, could be PersonRoles
<wgrant> wallyworld: Perhaps try adapting it to IPersonRoles?
<wgrant> Then you can just say IPersonRoles(user_or_reference).in_admin
<wgrant> If the adaption fails, it's not a person or an IPersonRoles.
<wallyworld> i guess it will fail by returning None?
<wgrant> wallyworld: By default it'll raise a TypeError
<wgrant> But the second arg specifies a default
<wgrant> So IPersonRoles(1, None) will return None
<wallyworld> that's the one i'll use, i don't like flow control by exception
<wgrant> It is good to avoid where possible, indeed.
<wgrant> wallyworld: Looks like the branch fixes bug #1009360, bug #1009358 (maybe?), bug #1009281?
<_mup_> Bug #1009360: RemoveBugSubscriptionsJob duplicates get_bug_privacy_filter <disclosure> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1009360 >
<_mup_> Bug #1009358: Unsharing information from a team doesn't remove the members' bug subscriptions <disclosure> <Launchpad itself:Triaged by wallyworld> < https://launchpad.net/bugs/1009358 >
<_mup_> Bug #1009281: SharingService.deletePillarSharee revokes all access to any involved artifacts <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1009281 >
<wgrant> 1009358 is linked already, nevermind
<wgrant> The other two aren't, though.
<wallyworld> ah, didn't realise thise were filed
<wallyworld> but yes
<wallyworld> i'll link those to the branch
<wgrant> Great, thanks
<wallyworld> wgrant: that should be fixed now
<wgrant> wallyworld: Great. I'm experimenting with refactoring the admin specialcase.
<wgrant> Since everywhere that needs to do it ends up mildly hideous.
<wallyworld> wgrant: should that be done in a separate branch? i'd like to get this one (and the next) landed
<wgrant> 318	+ getUtility(IRemoveBugSubscriptionsJobSource).create(
<wgrant> 319	+ user, bugs=None, information_types=information_types)
<wgrant> wallyworld: Doesn't that need a pillar restriction?
<wgrant> Otherwise it's going to do a full consistency check over every bug of that information type.
<wallyworld> yes, that's the intent at the moment
<wallyworld> but i could add a pillar check in
<wgrant> We can't run it on production like that, but I guess it's a reasonable intermediate.
<wallyworld> that bit was ported across from the deleted remove grantee job
<wgrant> In the long-term we probably want to restrict to bugs with the relevant AP and a subscriber in the team.
<wgrant> The join through BugSubscription + TeamParticipation is not completely cheap, but it's far cheaper than a full consistency check.
<wallyworld> wgrant: i have to go pick up the kid, be back later. i'll add the pillar check before landing
<wgrant> wallyworld: Thanks, I'm almost done.
<wallyworld> ok, no rush
<wgrant> Would be done quicker if the celery job tests weren't hanging...
<wallyworld> yeah, that happens sometimes to me too
<StevenK> wgrant: branch-privacy-properties => devel        [OK]       (up for 4:05:35) i-2ca43755
<wgrant> StevenK: Excellent. It's landing rather than testing?
<StevenK> It will land
<StevenK> wgrant: r15414
<wgrant> Marvellous.
<StevenK> It's about seven hours away from stable.
<wgrant> We're fixing that.
<StevenK> We are?
<wgrant> I also filed RTs this morning to make qastaging updates faster.
<wgrant> Well, the test suite will take half an hour soon :)
<StevenK> wgrant: More than one RT?
<wgrant> Well, one thing was only tangentially related.
<wgrant> Oh
<wgrant> Bah
<wgrant> Test was failing because the job was crashing because of a typo, and the test didn't notice
<micahg> Bug #590523 doesn't seem to have been triaged to a specific type of failure on that page, if I have a new OOPS, should I just add it to the page or file another bug?
<_mup_> Bug #590523: Archive:+delete-packages times out <boobytrap> <lp-soyuz> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/590523 >
<micahg> s/page/bug/
<wgrant> micahg: Add it to that page.
<micahg> thanks
<StevenK> wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/fix-copyrights/+merge/110229
<wgrant> StevenK: lib/lp/app/__init__.py wants reverting
<wgrant> It's a file of monkeypatches without an import section, really.
<StevenK> wgrant: Done.
<wgrant> StevenK: lib/lp/services/database/locking.py's __all__ is in the wrong place.
<StevenK> wgrant: Also fixed
<StevenK> The branch is now at +1
<wgrant> Otherwise it's good
<wgrant> Thanks
<wgrant> micahg: Hah
<wgrant> micahg: Copying 4 sources with a total of more than 250 binaries in one operation.
<wgrant> micahg: I am not surprised it timed out.
<micahg> I'm special :)
<wgrant> So, the solution to the bug is to not do that.
<wgrant> Because it's not sensible :)
<micahg> wgrant: umm, to copy 4 source packages to 4 different series in the same PPA shouldn't require 4 times through the page
<wgrant> For a sensible source package, sure.
<micahg> maybe it should just be done asynchronously
<wgrant> You can't make any assumptions about sources averaging 62 binaries each
<wgrant> Probably, yes.
<StevenK> PCJ?
<wgrant> die
<bigjools> I have a plan for async ppa copies
 * micahg loves copyPackage
<bigjools> just never got around to implementing it
<wgrant> micahg: So, how can there possibly be more than 250 binaries?
<wgrant> l10n?
<micahg> wgrant: yes, the l10n packages were made arch:any because of skew issues
<wgrant> blink
<wgrant> When was this?
<StevenK> micahg: !!!
<micahg> or was there another reason for that...
<wgrant> Recall that we now keep arch-all published until all the arch-any are gone.
<micahg> StevenK: I didn't do it :)
<wgrant> But making them all arch-any is just completely ridiculous and not justifiable :)
<micahg> wgrant: http://bazaar.launchpad.net/~mozillateam/thunderbird/thunderbird.lucid/view/head:/debian/changelog#L123
<micahg> it's a waste of mirror space and bandwidth as well
<wgrant> Ew
<lifeless> wgrant: ok, so, hot patches
<wgrant> Oh right.
<wgrant> lifeless: Indeed
<lifeless> they have the following properties:
<lifeless>  - they may break running code
<wgrant> micahg: Amusingly, the copy actually succeeded a few milliseconds before the timeout, but generating the notification timed out.
<lifeless>  - they will apply as regular dbpatches
<wgrant> I think it might be loading too many objects for the cache to survive.
<lifeless>  - but we can apply them by-hand superfast.
<wgrant> lifeless: Right.
<lifeless> so, I'm not sure why we would land them on devel.
<wgrant> lifeless: Though generally they're just index additions, so cannot break running code (unless postgres misplans and performance suffers)
<lifeless> I can't find any reference to landing them on devel in the wiki page history
<lifeless> though I'm not quite at rev 0
<micahg> wgrant: interesting
<wgrant> I quote:
<wgrant> 10:51 < wgrant> lifeless: What's the process for live index creation these days?
<wgrant> 10:51 < wgrant> Apply before or after landing?
<wgrant> 10:51 < wgrant> It should take about 60ms to create :)
<wgrant> 10:52 < lifeless> wgrant: land on qastaging, add and QA there, then prod.
<wgrant> 10:53 < wgrant> land on devel, add and QA on qastaging, then prod?
<wgrant> 10:53 < lifeless> yes
<lifeless> wgrant: so clearly I'm inconsistent
<lifeless> hmm
<lifeless> stub went back to sleep
<wgrant> Landing on db-devel provides no benefit.
<wgrant> And it's extremely awkward to deal with, because db-stable can be undeployed for weeks.
<wgrant> Because of the one-patch-per-downtime.
<wgrant> So we'd end up with db-stable being deployed out of sequence.
<wgrant> For no benefit whatsoever.
<lifeless> well, thats a very absolute statement.
<wgrant> It's also absolutely correct :)
 * lifeless wonders if disproving it is worthwhile or a distraction
<lifeless> ELOCAL, bbiaw
<wgrant> k
<lifeless> wgrant: so what we originally wrote was to have the hot patch qaed by application to qastaging
<wgrant> lifeless: Right, that makes sense.
<wgrant> lifeless: In practice we often go live directly to production with indices, because there's no meaningful QA until the code is there.
<wgrant> (though they usually have been tested with real queries on one of the three stagings first)
<StevenK> qastaging, staging, and the staging that is spelt with no s, t, a, i, n, one less g, and with two d's, three o's, and an f?
<wgrant> Those, indeed.
<wgrant> lifeless: So, I don't see that there's anything achieved by landing on db-devel.
<wgrant> We shouldn't do something just because the policy that's never been executed in full says we should do it.
<wgrant> There has to actually be a reason :)
<wgrant> wallyworld: 204	+ return [
<wgrant> 205	+ enumerated_type_registry[InformationType.name].items[value]
<wgrant> 206	+ for value in self.metadata['information_types']]
<wgrant> wallyworld: Isn't that just this:
<wgrant> return [InformationType.items[value] for value in self.metadata['information_types']]?
<wallyworld> not sure, i can try it
<wallyworld> i wrote that code ages ago
<wgrant> IIRC the registry was created for de-JSONing things.
<wgrant> But you have the actual enum here
<wgrant> So you don't need the registry.
<wallyworld> no i don't :-)
<wallyworld> the job state is saved
<wallyworld> and then read later from json
<wgrant> Hmmm?
<wgrant> You do!
<wgrant> You get the name of the enum from the enum
<wgrant> And then use it to look up the enum
<wgrant> in the enum registry
<wallyworld> i'll try it
<wallyworld> i may have copied the code from when i didn't have the enum name
<wgrant> That sounds most plausible.
<wgrant> # The enumerated_type_registry is a mapping of all enumerated types to the
<wgrant> # actual class.
<wgrant> So it indeed just returns the class
<wgrant> So enumerated_type_registry[InformationType.name] is just InformationType
<wallyworld> sounds about right, i have to go and re-read the source
<wgrant> It's also tempting to ignore Guido and replace the overlong list comprehension with "map(attrgetter('value'), information_types or [])"
<wallyworld> does he not like map?
<wgrant> He tried to remove it from Python 3
<wallyworld> why?
<wgrant> http://www.artima.com/weblogs/viewpost.jsp?thread=98196
<wgrant> Basically, Guido doesn't like functional-style code :)
<wallyworld> wgrant: seems like an opinion on style rather than anything of substance at first glance
<wgrant> Sure
<wgrant> But he actually tried to remove them from the language :(
<wallyworld> that's one of the good things about pythin, that it's multiparidigm
<wgrant> Precisely.
<wgrant> (I think listcomps might not be so bad if we didn't have a policy against using single-letter variable names)
<lifeless> wgrant: I agree, I'm trying to reconstruct our reasoning
<lifeless> wgrant: we failed to write some of it down
<wgrant> lifeless: Perhaps stub remembers?
<wgrant> I don't remember much of it from IRC, so it was probably mostly between you two
<lifeless> ondeed
<lifeless> he is asleep just now
<wgrant> :(
<lifeless> I think there are at least minor downsides to going straight to devel, but also less duplicate work merging stuff up etc
<lifeless> one of the undone longer term things is making hot patch application be -done- (in make schema etc) using the same sql we use to apply it live.
<lifeless> to avoid manual handling
<wgrant> wallyworld, StevenK: Looks like the in-progress branch denorm will make lp:launchpad branch search about 12x faster.
<wgrant> Yay
<wallyworld> great
<wgrant> lifeless: Right, it'd be nice to automate that.
<wgrant> lifeless: It is infeasible at present due to slony.
<lifeless> its totally doable
<lifeless> just not done
<wgrant> Well.
<wgrant> Awkward.
<lifeless> and not worth doing now.
<wgrant> And probably not worth it given slony's demise
<wgrant> Right
<wgrant> The remaining big issue is CONCURRENTLY. Not sure how best to work around that.
<adeuring> good morning
<wallyworld> wgrant: fancy a quick +1 on a +16/-471 mp (deletes RemoveGranteeJob). then i can chuck at ec2 https://code.launchpad.net/~wallyworld/launchpad/delete-RemoveGranteeSubscriptionsJob/+merge/110256
<wgrant> wallyworld: Looking
<wallyworld> thanks
<wallyworld> the additonal lines are just in a generic base class test since the test job has been replaced
<wgrant> wallyworld: Well that was challenging.
<wgrant> r=me
<wgrant> Also just looked over your amendments to the one I reviewed earlier. That looks good too.
<wgrant> Thanks for applying my suggestions.
<wallyworld> np, thanks for the reviews
<wallyworld> wgrant: so, i think if i add the filter for grantee when leaving a team the job should be performant
<wgrant> The pillar filtering isn't perfect, but in another iteration or two it should be good.
<wallyworld> wgrant: you think it should be on AP?
<wgrant> wallyworld: That's probably best. eg. now it will miss series bugs.
<wgrant> AP is probably more direct.
<wgrant> And accurate.
<wgrant> We'll need grantee filtering for normal removals too
<wgrant> eg. if I revoke someone's access to Ubuntu userdata.
<wgrant> That's a couple of hundred thousand bugs.
<wallyworld> yes, makes sense
<wgrant> We care about the bugs that have the relevant AP and have a subscriber that is a participant in the grantee.
<wallyworld> the grantee will be removed first though
<wallyworld> before running the job
<wallyworld> but that shouldn't matter i don't think
<wgrant> Right.
<wgrant> That certainly makes things look a bit nicer.
<wallyworld> for team membership removal, the filtering will be a bit different
<wgrant> Yeah
<wgrant> That has no AP filter.
<wallyworld> since the team itself still has access
<wgrant> It's just any private bug with a participating subscriber.
<wgrant> Right, the team isn't relevant to the job.
<wgrant> The removed member is.
<wgrant> Any subscriber participating in the former member is affected.
<wallyworld> the team could be used to narrow the search
<wgrant> Hmm.
<wgrant> Indeed.
<wgrant> I'm not sure if that's worth it, but it is indeed a good thought.
<wgrant> In general the set of subscriptions should be very small.
<wallyworld> so we need to record in the metadata (former) grantee and optionally team
<wgrant> Well
<wgrant> The job only really cares about a set of criteria to identify the bugs to reconcile.
<wgrant> I don't think it needs to know about grantees directly.
<wgrant> We just need to say "please reconcile subscriptions for participants of ~launchpad in Ubuntu's user data bugs"
<wgrant> That could mean that ~launchpad had a grant revoked.
<wgrant> Or it could mean that ~launchpad was removed from a team.
<wallyworld> sure, but i'm more specifically talking about removing a team member
<wgrant> Er, in the team removal case there wouldn't be the AP restriction, but yes.
<wgrant> wallyworld: That's the same.
<wallyworld> so we only care about bugs the former member has access to via the team
<wgrant> Right.
<wallyworld> had
<wallyworld> anyways, i'll code it up and see what falls out
<wgrant> In most cases it won't be worth filtering by "had access to via the team"
<wgrant> It will be more efficient just to consider all subscriptions.
<wgrant> But it is something to consider for bad cases, and it may not be too bad to do it all the time.
<wallyworld> ok, i wasn't going to assume that, but i'll defer to you for that
<wallyworld> i think if the cost of doing it is small, we may as well
<wgrant> wallyworld: Subscriptions are going to become pretty rare, I think.
<wgrant> Structural subscriptions handle widespread notification.
<wgrant> APGs handle widespread access.
<wgrant> There's no more use case for widespread subscriptions :)
<wallyworld> but the subquery to get the subscriptions
<StevenK> I like the idea that subscriptions are going to die.
<wallyworld> would benfit from the extra filtering
<wgrant> wallyworld: So
<wgrant> Maybe.
<wgrant> But, for example, I'm removing a user with 5 subscriptions from a team with access to Ubuntu's user data bugs.
<wgrant> It's clearly more efficient there to just look at the subscribed bugs.
<wgrant> The converse case can also be argued, of course.
<wgrant> But huge numbers of subscriptions shouldn't be common at all.
<wgrant> Huge access grants will be.
<wallyworld> the structure of the job though is to select bugsubscription where bug in (...)
<wallyworld> and it's the subsquery that needs to be fast
<StevenK> We still need to transition to APGs/APAs.
<StevenK> That is going to be ... fun.
<wallyworld> yep :-)
<lifeless> also subscriptions will be a separate service in future too.
<StevenK> So you keep saying
<lifeless> I do
<lifeless> how is auditor ?
<StevenK> I've not touched it this week, due to Monday and I'm still stuck having the fixture being able to run manage.py or some other form of it.
<wgrant> StevenK: Huh? What transition?
<wgrant> StevenK: We're already using them internally for all bug access checks.
<wgrant> It's just the UI that's not rolled out yet.
<wgrant> And it mostly works.
<wgrant> These are the final pieces coming together.
<StevenK> Perhaps I am misrecalling the Marvellous, Foolproof and Excellent Disclosure Plan.
<StevenK> wgrant: However, branch access is another matter. But, that is progressing, slowly.
<wgrant> Not particularly slowly.
<wgrant> We just have to wait for the bug stuff to mature and then refactor it to do branches too.
<wgrant> Branches are a faaaar simpler problem.
<wgrant> Once you discard stacking.
<StevenK> I was about to say "*cough* stacking" :-)
<lifeless> 'discard'
<jml> heh heh
<StevenK> wgrant: I say slowly, because I have a DB patch that drags it forward and I can't land it for a week.
<StevenK> And when it deploys is another matter entirely.
<wgrant> StevenK: You can probably land the DB patch on Monday night.
<wgrant> It can be deployed on Tuesday.
<StevenK> wgrant: That plan requires that the model code is dropped in a deployment on Monday.
<StevenK> But okay.
<wgrant> StevenK: That's the plan.
<wgrant> Although if we push it we can drop the model code tomorrow.
<wgrant> Let's push it :)
<StevenK> I haven't written the branch yet.
<wgrant> It's about -10
<wgrant> We can deploy today's branch in about 3 hours
<wgrant> Land the model removal
<wgrant> Deploy model removal tomorrow
<wgrant> Land DB patch
<wgrant> Deploy DB patch Monday night :)
<StevenK> There are at least seven branches that need QA before we can deploy in three hours.
<wgrant> The ones with non-trivial QA are â
<wgrant> Well, r15407 and r15410 require a small amount of thought.
<StevenK> wgrant: So, model droppage now, or tomorrow morning?
<wgrant> StevenK: I'd prepare the branch now, since it's trivial.
<wgrant> It shouldn't be landed until today's is deployed.
<wgrant> I think you can just delete any statement mentioning _transitively_private or _explicitly_private.
<StevenK> Of which there are two ...
<wgrant> 3 lines mention explicitly, 8 transitively
<wgrant> But close enough
<wgrant> So it's going to be like -13
<wgrant> Not -10 as I said :(
<StevenK> wgrant: I'm at -18/+0 already
<jml> https://bugs.launchpad.net/launchpad/+bug/1013056
<_mup_> Bug #1013056: Creating a PPA requires one to be a team admin <ppa> <Launchpad itself:New> < https://launchpad.net/bugs/1013056 >
<wgrant> jml: Argh
<wgrant> Make it go away
<jml> I'm trying.
<wgrant> That one comes up occasionally
<wgrant> I don't have a good answer.
<czajkowski> I was thinking something similar
<jml> I filed it.
<wgrant> I know you filed it.
<jml> Here are some answers I can think of. I won't claim they are good.
<wgrant> The issue comes up in discussion occasionally.
<jml> make it launchpad.Append
<jml> make a new object, say, ArchiveCollection, and hook permissions on to that.
<bigjools> append on what?
<wgrant> So, allowing it technically is easy.
<wgrant> Socially it's less obvious whether it's sensible.
<jml> bigjools: IPerson
<bigjools> ah ok
<wgrant> jml: I'm a member of ~ubuntu-bugcontrol
<wgrant> Should I be able to create a PPA?
<wgrant> No.
<wgrant> ubuntu-bugcontrol isn't a PPA team.
<jml> wgrant: right. similar for mailing list teams.
<StevenK> wgrant: http://pastebin.ubuntu.com/1040482/
<jml> and the last thing Launchpad needs is another knob.
<StevenK> jml: No, we have enough of those working on it already. :-P
<bigjools> badumtish
<wgrant> jml: Precisely.
<wgrant> jml: So I would just make your robot a team admin.
<wgrant> And ignore the question.
<jml> wgrant: we have a contractor too.
<bigjools> can we apply a Knob-count approach like LOC? :)
<wgrant> jml: Can't SCA have an API to do it or something?
<wgrant> There's no sane way to do it in LP without adding a knob.
<jml> wgrant: we can probably get by, but I'd rather leave the bug open for the moment. I don't like bots with team admin privs either.
<wgrant> Sure.
<wgrant> We need something more flexible.
<wgrant> But we have no way to do that without making everything suck right now :/
<jml> wgrant: by 'have an API', you mean write a front-end and secure that?
<wgrant> jml: Maybe.
<wgrant> jml: I mean, SCA already accepts API requests from webapps to manage subscriptions.
<wgrant> And those webapps have admin UIs
<wgrant> That could be used to ask SCA to perform admin operations like setting up a PPA for a new app.
<jml> wgrant: yeah, we could possibly do that. We weren't planning on having SCA create PPAs, but that might work.
<wgrant> StevenK: Ah, comments, I see. That's cheating.
<StevenK> Hahaha
<StevenK> wgrant: Do you think it's reasonable?
<wgrant> StevenK: It does what we want, it works, it doesn't fail tests, it doesn't add ugly code.
<wgrant> I don't see how it could not be reasonable.
<wgrant> StevenK: Deployment report is green and the two remaining revs before yours are 30s QA.
<StevenK> wgrant: I think this branch model may also have to kill the transitively_private triggers tests. I'm not certain.
<wgrant> Oh, indeed.
<wgrant> bzr rm lib/lp/code/model/tests/test_branch_privacy_triggers.py
<cjwatson> wgrant: map/listcomp> the tradeoff has changed a bit in Python 3 though - map returns an iterator, so if you need a concrete list then the map approach is an extra six characters versus Python 2
<wgrant> cjwatson: True.
<wgrant> StevenK: Actually, it shouldn't matter. The trigger tests don't use the model.
<wgrant> StevenK: It's all string-based SQL
<wgrant> StevenK: But might as well delete them anyway.
<cjwatson> 15412 is a bit more than 30s QA since it's gone wrong before, but hopefully not overly much.
<wgrant> Meh, it's only worth two API calls.
<cjwatson> Actually, I could do that on dogfood now couldn't I
<wgrant> You could indeed.
<wgrant> It won't be on qastaging for a couple more hours.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-branch-privacy/+merge/110274
<wgrant> StevenK: r=me, ec2test time.
<StevenK> wgrant: Tossed at ec2 test.
<wgrant> StevenK: Great.
<StevenK> It will be great if it doesn't fail any tests. Not before.
<wgrant> Meh
<bigjools> jml: I saw a car today with a sticker in the back window that said "Beached as!"
<bigjools> and I thought of you
<jml> bigjools: :)
<al-maisan> hmm .. can someone please take a look at this build : https://launchpad.net/~openquake/+archive/testing/+build/3575134
<al-maisan> the upload of the resulting binaries seems stuck
<wgrant> al-maisan: Cronjobs are disabled for a DB upgrade in a few seconds.
<wgrant> Should be processed in a few minutes.
<al-maisan> ah, I see, thanks!
<wgrant> al-maisan: The webapp downtime is down to 70s, but we're still disabling cronjobs about 15min before for safety.
<al-maisan> no problem at all -- at least I know what's going on now :)
<wgrant> al-maisan: That uploaded a few minutes ago.
<al-maisan> thank you William!
<jml> ooh, a db upgrade
<jml> will this include my patch?
<wgrant> jml: This was stub's index work. Yours is nearly 24 hours away.
<jml> wgrant: ta
<wgrant> cjwatson: Did you get around to doing that QA on DF?
<rick_h> StevenK: you get +10 internet points for your email signature I noticed today
<cjwatson> wgrant: Yes; I've been waiting for qastaging to update.
<wgrant> It'll be there in about 30s.
<cjwatson> (So I can mark it qa-ok.)
<cjwatson> Cool.
<wgrant> Ah right
<wgrant> That'll be another 5 minutes.
<wgrant> Thanks.
<wgrant> There we go, right on time.
<cjwatson> wgrant: Bug 648611: any particular reason you feel it necessary to include upload status granularity in that?
<wgrant> cjwatson: I don't remember who that was discussed with. ScottK and others. But if it's no longer an issue for you, then there's no need for it.
<jml> wgrant: do you know if the codehosting puller is still in use?
<cjwatson> Maybe I should talk with ScottK then.
<wgrant> jml: Mirrors and imports still use it. Mirrors can't be created any more, but there are still some old ones in use.
<cjwatson> It'd be easier to just make it per-pocket, since that's almost exactly analogous to per-pocket upload.
<jml> wgrant: ah ok. I'm surprised imports use it.
<jml> oh wait
<wgrant> jml: Same as ever.
<jml> to pull it from internal location where branch is
<wgrant> importds push their branches up to escudero
<wgrant> LP pulls from escudero
<wgrant> Because who cares about sanity :)
<jml> wgrant: I thought I recalled some discussion about using the codeimport system to do pulling
<jml> ddaa had a reason for that, I'm sure.
<jml> huh
<jml> I wonder. Could / should the ScriptActivity table have a place in the web ui, exposed to LP developers & admins.
<StevenK> rick_h: Haha
<wgrant> jml: s/table/service/
<jml> wgrant: even better.
<StevenK> wgrant: Are you going to be awesome and do my QA, since dinner for Sarah and I is about 3 minutes away from serving?
<wgrant> StevenK: Mostly done already
<jml> hmm. hmm. hmm.
<StevenK> wgrant: ec2 has another ~ 110 minutes, but it should be good to lp-land when it finishes.
<wgrant> jml: Hm hm?
<jml> wgrant: I was thinking about starting work to pull codehosting out of LP
<jml> wgrant: and then about how I wonder how much work it would be to make & deploy a scriptactivity service as we just described
<cjwatson> Oh, hey, maybe I lied when I said to technical-board@ that my permissions branch would probably land tomorrow or failing that Monday.
<jml> wgrant: and then about how I don't know whether anything *has* been pulled out of LP into a separate service successfully
<wgrant> cjwatson: Pocket permissions? With a bit of luck it'll be deployed 40 minutes after gnuoy returns from lunch.
<wgrant> jml: It depends on how much of codehosting you want to pull out of LP.
<cjwatson> Yeah, I missed your chatter about pushing it.
<wgrant> How much of codehosting do you want to pull out of LP?
<jml> wgrant: and how if I started I'd be afraid that the work would spiral out of control with lifeless, yourself and others telling me that I'm doing it wrong or adding new requirements or elucidating requirements that I'm not fully conscious of yet.
<jml> wgrant: I think the ssh server / vfs bit makes sense.
<jml> wgrant: maybe the apache rewrite support too. I'm not sure.
<wgrant> Nothing like this has been extracted before.
<jml> wgrant: an obvious pre-req would be extracting lp.services.ssh into a library.
<wgrant> I was going to do poppy to start. But I ran into trouble with naming the libraries.
<wgrant> Right.
<wgrant> That's pretty easy to do.
<jml> and I don't know how I'd answer questions like what to use for config.
<wgrant> And it's probably most of the integration points.
<jml> wgrant: poppy would also be a good place to start.
<wgrant> I'd probably just use a separate minimal lazr.config-based thing. But configglue or something might be easier and better.
<wgrant> jml: I'd suggest starting with poppy, as it uses the sshserver and is much simpler in every other respect.
<jml> wgrant: it seems to me that doing one small thing, and (crucially) writing down the steps taken would be a giant leap for Launchpad.
<wgrant> Most definitely.
<wgrant> poppy is probably the smallest, easiest piece to rip out like that.
<wgrant> It's extremely isolated.
<jml> Yeah. The list for me goes: poppy, codehosting ssh, maybe codeimport, builddmaster.
<jml> Although you mentioned scriptactivity, which probably goes somewhere toward the right.
<wgrant> scriptactivity is sort of the opposite.
<wgrant> It's an internal service used by everything, but it's very isolated and an excellent example of something that should be a microservice.
<wgrant> scriptactivity is probably a prereq for getting rid of other things.
<wgrant> Since they'll want a way to record activity.
<jam> anyone have much experience with lxc? I'm running low on space on '/' so I'm trying to put the rootfs on an alternate drive. But once I do 'lxc-start' just returns without starting anything, or giving me anything informative.
<jam> I've tried playing with the config file, to point the root to another location
<jam> maybe if you know of an error log or something like that?
<wgrant> jam: Tried 'lxc-start -n foo -o /tmp/lxclog -l DEBUG'?
<jml> wgrant: yeah. it's realizing that it's a pre-req that got me 'hmming' in the first place
<wgrant> Yeah
<jml> Specs fall apart, the scope it cannot hold; more work items are loosed upon the world
<jml> wgrant: anyway, I'll keep it in mind as a possible skunkworks project
<wgrant> Sounds good.
<wgrant> So, poppy and scriptactivity, in that order, would be my recommendation for anyone wanting to do this sort of thing. poppy because it's simple, and scriptactivity because it's simple and a core pre-req from the other end.
<jml> wgrant: doesn't poppy need to use scriptactivity
 * jml just remembered something.
<wgrant> jml: It's a daemon.
<wgrant> Not a cronjob.
<jml> ah, ok.
<jam> wgrant: it appears /sbin/init was just dying, but that appears to be because I had that device mounted with "nosuid,nodev"
<jam> it seems to work now
<wgrant> jam: That would be unhelpful, indeed.
<wgrant> It was GNOME-mounted removable media?
<jam> wgrant: I added it to fstab, using the params done by GNOME disk-utility
<cjwatson> udisks mounts everything with nosuid,nodev.
<jam> I couldn't find a Gnome utility that would let me say "always mount this disk"
<wgrant> Ah yes, udisks, I had forgotten what was underneath today.
<cjwatson> udisks2 in quantal, because it would be a shame to leave anything the same.
<jml> wgrant: you don't appear to be subscribed to DisklessArchives. I made an edit, and james_w added a comment the other day for which we'd welcome your feedback. (Can wait until tomorrow though)
<wgrant> jml: I couldn't find a subscribe button, and /UserPreferences is like all the way over there.
<jml> wgrant: URL hack! ?action=subscribe
<wgrant> Ah, didn't think that would still work.
<wgrant> Surely new moin has fixed mutating GETd
<wgrant> s
<wgrant> Evidently not.
<wgrant> Thanks.
<jml> wgrant: np.
<wgrant> jml: The API service will live in the LP tree.
<wgrant> The squid bits will not.
<wgrant> Ideally the API service would be part of the main Zope stack, but the main Zope stack is so terrible that it's not feasible at present, so we have to hack.
<jml> wgrant: that makes sense. it's a shame from a test cycle pov.
<wgrant> Giving the testing hostname to real clients is an interesting idea. I'm not sure if it's a good one.
<wgrant> jml: Certainly.
<lifeless> wgrant: we an use the testing hostname as the new, browser-safe permanent hostname
<lifeless> wgrant: and initiate a very slow migration.
<wgrant> True.
<jml> wgrant: however, once we have one non-zope api server in LP to facilitate fast requests, why not move the existing api services to that same technology
<lifeless> see #ca-internal for a chat between me and james_w about this
<wgrant> jml: That's the plan.
<lifeless> wgrant: btw, LP's stack *can* answer in 3ms.
<wgrant> jml: Whether the plan will happen this decade is unclear :)
<wgrant> lifeless: hahahah
<jml> I hope not using zope stack means that at least the unit tests will be fast
<wgrant> +opstats maybe
<lifeless> wgrant: definitely, see the haproxy stats page.
<lifeless> wgrant: I noting that the publication stack through to render and return is likely fast enough for our needs; there is definitely potential issues below that.
<lifeless> wgrant: I'm inclined to suggest trying in the main stack in the first instance; it means we know fairly likely places for the rot to be located.
<wgrant> Right, for a no-op publication to a no-op view that has hardcoded hacks to avoid most of the stack, it can be fast.
<jml> alas for my TDD cycle time!
<lifeless> jml: the TDD cycle time for the horizontally scaling nodes should be fine.
<lifeless> jml: it will need a test harness, but that by definition has to be fast to start and use.
<wgrant> That's like 3% of the effort.
<jml> btw, at some point before we actually start work it'd be good if you two, james_w and I could have a call to walk through the implementation plan. I realize that TZ wise it's a huge pita.
<wgrant> lifeless: Are you sure you didn't hallucinate the conversation with james_w?
<jml> also, it looks more like 6-8 weeks than 4-6.
<wgrant> I don't see it.
<wgrant> jml: It mostly depends on how far you want to rewrite the backend publication stack.
<wgrant> lifeless wants you to rewrite them a lot.
<wgrant> I don't.
<wgrant> lifeless: Which day was it?
<lifeless> wgrant: today
<lifeless> may have been here. Lets see
<lifeless> wgrant: u1-internal
<wgrant> Oh, sneaky.
<wgrant> I don't think I'm in there.
<lifeless> very very so
 * wgrant grabs logs.
<stub> jml: Yes, I think scriptactivity should definitely be exposed. I always imagined a page of green/red/yellow blobs for each component we can track.
<stub> jml: I think the trick is to start simple, as bikeshedding and feature creep turn it into a big problem very quickly
<jml> stub: agreed
<jml> stub: the main thing I'm interested in is knowing which cronjobs are still being run and with what options so I can delete more code.
<jml> although I don't think SA tracks options
<wgrant> jml: bzr branch lp:lp-production-crontabs
<jml> mirable dictu!
<lifeless> deleting scriptactivity itself would be a start
<wgrant> cjwatson: Perhaps we should revert your revision quickly, just to prevent the rest of UE from getting the idea that LP deadlines don't always slip.
<cjwatson> wgrant: haha
<wgrant> Also, we're not really in a hurry for something else, I just wanted to stop StevenK complaining that things are too slow.
<StevenK> wgrant: You know, scriptactivity could probably be replaced by auditor ...
<wgrant> Potentially.
<stub> and the bikeshedding and feature creep starts...
<lifeless> jml: pretotyping
<StevenK> stub: For auditor? I don't think the feature could creep much :-)
<wgrant> StevenK: That just means it's already crept.
<StevenK> wgrant: But it hasn't?
<StevenK> Log events via POST, grab them via GET. Done.
<StevenK> It might need some squinting in terms of actor/object acted on, but shrug.
<StevenK> wgrant: It was good natured complaining about things going slowly. :-)
<wgrant> Sure.
<wgrant> But we might as well make things faster where we can.
<stub> StevenK: It started as exposing some information in the DB, now you are posting information into it :)
<StevenK> stub: No, it was always the plan that we had to post information into it, so nyah.
<wgrant> jml: How soon are you likely to want that call?
<jml> wgrant: before we start implementing, after you guys think it's ready for us to start implementing.
<jml> wgrant: modulo some scheduling stuff that I need to talk to james_w and mvo about.
<wgrant> Right.
<ivory> jelmer: could you review this please? https://code.launchpad.net/~ivo-kracht/launchpad/bug-999662/+merge/110322
<jelmer> ivory: sure, I'll add it to my queue
<ivory> jelmer: thanks
<flacoste> gary_poster: i really liked the format of your latest parallel tests update
<flacoste> gary_poster: it read very well!
<gary_poster> flacoste, thanks!  I'll go review it to see what I did differently. ;-)
<mgz> ivory: skim-by, is that change wrap the <a> in xx-distirbution-changes.txt valid? I'd think id'd break a doctest
<wgrant> ivory: Did you consider using fmt:approximatedate? It handles timestamps within the last 24 hours a little more useful, by saying eg. "2 hours ago"
<flacoste> gary_poster: i think the summary rocked, or maybe is that the news are particularly good this week ;-)
<wgrant> For more than a day it just uses the date.
<StevenK> "a moment ago" \o/
<gary_poster> flacoste, lol, cool, I'll take either one :-)
<wgrant> mgz: Doctests default to normalising whitespace.
<wgrant> mgz: So you can normally wrap however you want.
<ivory> wgrant: i did but somehow it didn't work, maybe i did something wrong but then abel proposed to do it like that ...
<cjwatson> Not that you'd know it by looking at lots of our existing doctests.
<wgrant> cjwatson: Shhh
<mgz> wgrant: they don't generally, but does launchpad set that flag?
<wgrant> mgz: I meant our doctests, yeah.
<mgz> thanks, good to know.
<wgrant> default_optionflags = (doctest.REPORT_NDIFF | doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS)
<StevenK> Some of our unit tests also make use of doctest.NORMALIZE_WHITESPACE, which I find hideous.
<cjwatson> doctest.ELLIPSIS is handy when converting doc to unit though.
<ivory> mgz: i don't really understand what you mean.i didn't change xx-distribution-changes.txt
<mgz> ivory: *-packages.txt, but wgrant settled that question
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: jcsackett | Firefighting: - | Critical bugs: 3.47*10^2
<wgrant> cjwatson: It is done.
<cjwatson> Neat.
<StevenK> wgrant: ec2 is done.
<wgrant> StevenK: Might as well land it. The deploy isn't completely done yet, but the appservers are and nothing's going to cause a revert.
<StevenK> wgrant: r15417
<wgrant> StevenK: Excellent, thanks.
<wgrant> Hopefully by the end of the year we'll be doing this in two hours, not two days.
<ivory> jelmer: thank you :)
<danilos> flacoste, mrevell, jamestunnicliffe, matsubara: hi, shall we try hangout again: https://plus.google.com/hangouts/_/40c23fb864bd02615f8a2bc4d8919084f9053f90?authuser=0&hl=sr
<danilos> mrevell: hi, shall we try hangout again: https://plus.google.com/hangouts/_/40c23fb864bd02615f8a2bc4d8919084f9053f90?authuser=0&hl=sr
<mrevell> Th
<mrevell> danilos, Cheers!
<mrevell> flacoste, Hangout: ^^^
<mrevell> danilos, Google Plus is now talking to me in Serbian.
<danilos> mrevell, don't you feel happy about it? just get hl=sr off the URL, sorry :)
<matsubara> https://dev.launchpad.net/Projects/WorkItems
<mrevell> czajkowski, In case you're interested, hangout ^^^
<jamestunnicliffe> https://launchpad.net/~linaro-infrastructure/+upcomingwork
<danilos> matsubara, and https://launchpad.net/people/+me/+upcomingwork
<danilos> mrevell, https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/engineering-view-fixes
<danilos> mrevell, https://bugs.launchpad.net/launchpad/+bug/1004416
<_mup_> Bug #1004416: Work Items not allowing users to edit them properly <work-item-tracker> <Launchpad itself:Triaged by linaro-infrastructure> < https://launchpad.net/bugs/1004416 >
<sinzui> jcsackett, can you review https://code.launchpad.net/~sinzui/launchpad/commercial-jobsource-zcml/+merge/110365 today?
<jcsackett> sinzui: sure.
<jcsackett> sinzui: r=me, but i'm confused by moving the lp.services imports? is that a merge artifact, or was there a problem with standard sort order?
<sinzui> jcsackett, someone sorted imports and I merged
<sinzui> I can resort
<jcsackett> sinzui: cool, thanks. :-)
<jml> dist/zope.testing-3.9.4-p13.tar.gz. p13? my sympathies!
<jml> httplib2.SSLHandshakeError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
<jml> ^^ got that error while trying to do API call against a dev instance
<jml> how can I work around?
<jml> (alternatively, how can I test creating a PPA on Launchpad as a user who does *not* have a commercial subscription?)
<cjwatson> Can anyone advise on the problem I'm having in https://code.launchpad.net/~cjwatson/launchpad/export-change-override/+merge/109549 ?
<jml> cjwatson: looking.
<jml> cjwatson: perplexing. copy_field doesn't seem to have any positional args beyond the first, so I'm in favour of the "Not being called" idea. Maybe it's something to do with the circular import hackery?
<cjwatson> I thought I used a keyword arg.
<jml> cjwatson: you did. yes. I checked for an error that in retrospect couldn't be possible.
<cjwatson> I'm not really convinced this proposed change actually makes the code any easier to maintain, but I'm not the reviewer ...
<jml> why do lazr.restful, lazr.restfulclient and launchpadlib only include the first 6 points of the GPLv3 license?
<cjwatson> I suppose I could start print-debugging through copy_field.
<jml> http://www.gnu.org/licenses/gpl.txt vs http://bazaar.launchpad.net/~lazr-developers/launchpadlib/trunk/view/head:/COPYING.txt
<jml> same date & version, I think.
<jml> oh. lgpl.
<jml> never mind.
<jml> sinzui: fwiw, my build was not from a public source.
<jml> mischan
<sinzui> jml: the same issue is happening. builds are not private, so they cannot issue a 403 or show a privacy banner. the privacy is very smart as it looks at the object, then the view to ensure the banner is shown
<jml> sinzui: then the bug has the wrong title.
<sinzui> fix it
<jml> sinzui: ok.
<ev> lifeless: if I may briefly go back to the conversation of not being able to use launchpadlib directly from a wsgi app because it's not thread safe.
<ev> What are your thoughts on talking straight HTTP to the webservice, rather than going through lplib?
<ev> this being an alternative to adding a rabbit/celery layer
<lifeless> ev: love it.
<ev> so that's a yes, avoid lplib for this then? :)
<ev> lifeless: and good morning
<lifeless> ev: I have no soft spot for it :P
<ev> heh
<ev> awesome
<ev> that makes things much simpler
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
#launchpad-dev 2012-06-15
<wgrant> StevenK: You have QA :)
<wgrant> sinzui, benji: logs/ isn't having much luck -- it's been added and then accidentally removed twice this week. How'd each of your removals happen, so we can make it stick this time?
<StevenK> % grep logs .bzrignore
<StevenK> logs/*
<StevenK> wgrant: ^
<wgrant> logs/*
<wgrant> Not logs
<StevenK> Bleh
<wgrant> logs isn't ignored
<benji> wgrant: heh
<StevenK> wgrant: Not sure how to QA that branch
<benji> wgrant: I don't remember the circumstances of my removal (nor did I understand it at the time).
<benji> and I found the "!" pattern for .bzrignore that lets us ignore the contents of logs other than README.txt
<StevenK> benji: But logs/* is already ignored
<benji> StevenK: my understanding is that either order is important or negative patterns are higher priority
<benji> I may be mistaken, but if so, I don't see how the negative patterns accomplish much
<wgrant> wallyworld: When landing the end of a pipe, it might be a good idea to adjust the commit message to cover the whole thing, not just the last branch.
<wgrant> Your two megalandings have understated their actions by about a factor of 10 :)
<wallyworld> it would be a very long message then
<wallyworld> shouldn't all the commit messages along the way be aggregated? that would make sense to me
<wgrant> lp-land is not designed for megalandings like that.
<wgrant> So you have to do it manually.
<wallyworld> wgrant: could you possibly take a look at https://code.launchpad.net/~wallyworld/launchpad/improved-bugremovesubscriptions-job/+merge/110447
<wgrant> wallyworld: of course.
<wallyworld> thanks
<wallyworld> it would be good to see if the queries run ok on dogfood so that we can see if it can be turned on for prod
<wgrant> They won't run OK, but I can make them run OK.
<wgrant> Is this branch relatively final?
<wallyworld> well, unless the queries need fixing
<wgrant> Yeah, I mean you don't like plan additional constraints.
<wallyworld> i think everything is covered funcionality  wise
<wallyworld> i can add constraints if it will help
<wallyworld> performance
<wallyworld> but it's funcionally correct as is
<wgrant> Right.
<wgrant> Agreed.
<wallyworld> i have no idea whether the current filtering on product/distro and info types as separate terms is better than an ap filter for example
<wgrant> The existing one is probably better atm.
<wgrant> But we can possibly gain a win from a GIN index on BTF.access_policies and filtering on that.
<wgrant> I'll find out after lunch.;
<wallyworld> ok, hopefully we can get this turned on early next week i guess
<wgrant> wallyworld: Have you played around with this in full locally?
<wgrant> Last time I tried I filed 11 bugs.
<wgrant> Looks like about half, and all the major ones, have been fixed.
<wallyworld> i 've done some testing, mainly tried to get unit test coverage
<wgrant> The major omissions are triggering the jobs on information_type and target changes.
<wgrant> Oh, also creating AAGs when making a bug private. We haven't decided on rules for that.
<wallyworld> ? i thought that was covered
<wallyworld> the jobs are triggered on info type and target changes and there are tests for that
<wgrant> Oh, are they. That's handy.
<wgrant> Forgot that detail, apparently.
<wallyworld> that bit was never missing, unless i'm totally wrong
<wgrant> Yeah, you're right, I think.
<wallyworld> so, creating AAG for bug -> private should be the last bit
<wgrant> Yeah
<wgrant> Bug #1009364
<_mup_> Bug #1009364: AccessArtifactGrants not created on bug privacy transitions without triggers <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1009364 >
<wgrant> Is what I was thinking of.
<wallyworld> yep
<wgrant> wallyworld: Bug #1009363 also looks fixed now, since that job is gone.
<_mup_> Bug #1009363: RemoveGranteeSubscriptionsJob._unsubscribe_pillar_artifacts has a slow reimplementation of bugtasksearch <disclosure> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1009363 >
<wallyworld> yes
<wallyworld> i'll link that bug to this branch
<wgrant> Although it also probably applies to the RemoveBugSubscriptions, it's not quite as bad or gratuitous there.
<wallyworld> not any more
<wallyworld> that was fixed a couple of branches ago
<wgrant> Do you have thoughts on bug #1009356? Pretty minor, not sure if it's easy.
<_mup_> Bug #1009356: Sharing a subset of information types generates a subscription removal job for the others <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1009356 >
<wallyworld> oh, it's harmless but unnecessary
<wgrant> Sure.
<wgrant> Might not be worth avoiding.
<wgrant> But if it's a one-liner, might as well.
<wallyworld> i can look at it,
<wgrant> To avoid all the extra jobs.
<wgrant> r=me, anyway.
<wgrant> Thanks for fixing that.
<wallyworld> so the AAG thing,  why do we want to create AAG for subscribers to the formerly public bug?
<wallyworld> thans for the review
<wallyworld> if the bug goes from public to private, then people who are not allowed to see it should have their subscriptions removed, those who can see it will have APG or AAG already, no?
<wgrant> wallyworld: When it becomes private nobody will have an AAG.
<wgrant> wallyworld: It's not clear who should have one.
<wgrant> But in the current implementation nobody will.l
<wgrant> Including the person who is making it private.
<wallyworld> those who have an APG will be ok though
<wgrant> So eg. a reporter can't flip their bug to private if they misfile it as public.
<wallyworld> so for now can we just create one for the key roles - reporter and person doing the flipping perhaps
<wgrant> Reporter isn't a key role.
<wgrant> But probably, yeah.
<wgrant> We almost need to make it explicit.
<wgrant> Which is why I was thinking we should just create one for all the subscribers.
<wallyworld> as a reporter i think i should be able to see the bug
<wallyworld> i think we should start narrow
<wgrant> I forget what we do now.
<wgrant> You changed the behaviour last year, but I don't remember if it stuck.
<wallyworld> i think it was bug supervisor and all direct subscribers, but am unsure
<wallyworld> what do the triggers do?
<wgrant> wallyworld: The triggers do subscription == visibility
<wgrant> So there's a 1-1 mapping.
<wgrant> So there's a grant for anyone that the app leaves a subscription for.
<wallyworld> do we know what our stakeholders expect for the new world order?
<wallyworld> it seems dangerous to just convert all existing subscriptions
<wgrant> The stakeholders are likely to care much; they all want private projects, so they don't care about public->private
<wgrant> wallyworld: It can't leak the bug any more than it has already leaked.
<wgrant> Because it was previously public.
<wallyworld> but new conversations will be invisible
<wgrant> Sure.
<wgrant> And before you have the new conversation you should ensure that only people who should see the bug can see the bug.
<wallyworld> so in that case we should just convert all existing direct subscriptions
<wgrant> I think that's correct for now.
<wgrant> It's certainly correct for the underlying layer.
<wgrant> I'm not sure what the application does currently.
<wgrant> It may already remove some of the subscriptions.
<wallyworld> it may do, can't recall, i'll have a look
<wgrant> wallyworld: :(
<wgrant> Looks like I won't be filing 10 bugs about the jobs today.
<wgrant> It seems correct, apart from needing some query optimisation.
<wallyworld> some of those 10 were questionable
<wallyworld> what query optimisation is needed?
<wallyworld> oh shit, gotta duck out to drop off something before the office closes, biab
<wgrant> wallyworld: A quick rewording from an IN to a JOIN gets it down to 10s for identifying subscriptions affected by removing a hypothetical ~ubuntu-bugcontrol grant to Ubuntu.
<wgrant> (that's some 399500 subscriptions)
<wgrant> So that's quite acceptable.
<wgrant> Given that it's about the worst conceivable case.
<StevenK> wgrant: http://pastebin.ubuntu.com/1041847/
<lifeless> wgrant: thats not-inline with the webapp request though ?
<wgrant> lifeless: No, this is a job.
<wgrant> StevenK: It'd be nice to test the triggers, rather than just that the underlying function works.
<wgrant> StevenK: This just tests the information_type UPDATE trigger.
<wgrant> StevenK: You can probably test the AAG/APA triggers in about 3 lines.
<wgrant> No need for INSERT/DELETE really. Just a quick test to check that INSERTing each updates the columns.
<wgrant> Although APA is a bit complicated, since we only expect one. So you'll need to UPDATE for the APA test.
 * StevenK stabs rhythmbox, and then twists the knife.
<StevenK> 2854 steven    20   0 9895m 5.3g  12m S    1 68.6  14:20.41 rhythmbox
<wgrant> 5.3G RSS?
<wgrant> And 10G virt?
<wgrant> So it's swapping?
<StevenK> 8G of RAM, so it was swapping heavily
<wgrant> Yeah
<wgrant> Thank god for amd64.
<wgrant> Or it wouldn't be able to get all the RAM it needs :P
<StevenK> So Quod Libet will randomly stop playing after a song finishes, and it looks like rhythmbox leaks like a sieve.
<StevenK> wgrant: Three lines? Really?
<wgrant> makeBranch
<wgrant> assertAccess
<wgrant> makeAccessArtifactGrant
<wgrant> assertAccess
<wgrant> 4 lines
<StevenK> So a USERDATA branch, or PUBLIC?
<wgrant> StevenK: It has to be private.
<StevenK> wgrant: http://pastebin.ubuntu.com/1041889/
<wgrant> StevenK: makeAccessArtifact should fail, since the branch should already have one.
<wgrant> Does it not?
<StevenK> Hm, not sure if I ran the tests before pasting.
<StevenK> wgrant: It does not fail.
<wgrant> That is a matter of some concern.
<wgrant> StevenK: See Branch.transitionToInformationType's _reconcileAccess call
<wgrant> That should be creating the AA
<StevenK> wgrant: Hm, I see ....
<StevenK> wgrant: How can I check if it has an AA?
<wgrant> StevenK: I'd check postgres logs to see if you can see the INSERT
<wgrant> But if you want to programatically check, getUtility(IAccessArtifactSource).find([branch])
<StevenK> INSERT INTO AccessArtifact (bug, branch) VALUES (NULL, 77) RETURNING AccessArtifact.id
<StevenK> Not sure if that's my makeAA call
<StevenK> artifacts = getUtility(IAccessArtifactSource).ensure([concrete])
<StevenK> That sounds like it will just return one if it exists ...
<wgrant> StevenK: ensure creates any that are missing.
<StevenK> wgrant: Hmmm, which does return one. That is most perplexing
<wgrant>     "accessartifact__branch__key" UNIQUE, btree (branch) WHERE branch IS NOT NULL
<wgrant> So that's not the problem.
<StevenK> wgrant: I have a 261K psql log you can grep through if you wish.
<StevenK> Changing it to artifact = getUtility(IAccessArtifactSource).find([branch])[0]
<StevenK> Also has the test pass
<wgrant> Oh heh
<wgrant> makeAccessArtifact calls ensure
<wgrant> That explains it.
<StevenK> That was what I tried to say earlier, and failed.
<wgrant> Oh
<wgrant> I misunderstood.
<wgrant> I thought you meant that was part of _reconcileAccess
<StevenK> wgrant: With that cleared up, are you happy with the tests?
<wgrant> If they pass, indeed.
<StevenK> All two of them
<wgrant> Then it sounds reasonable.
<StevenK> Ran 2 tests with 0 failures and 0 errors in 0.539 seconds.
<StevenK> wgrant: Thanks for all your help with this massive thing.
<StevenK> stub: https://code.launchpad.net/~stevenk/launchpad/db-branch-new-access-policy/+merge/110464 would love a DB review.
<wgrant> Hah, not massive.
<StevenK> wgrant: It's felt massive at times.
<StevenK> It's one of the largest DB patches I've done.
<stub> I still keep getting 'Updating diff' on mps that already seem to have perfectly good diffs
<wgrant> stub: I think it can take a while to go away.
<stub> I thought I nuked comments.sql?
<wgrant> We don't apply it any more.
<wgrant> But it continues to exist.
<StevenK> wallyworld: You have eight cards in QA-Landing, can you have a look?
<StevenK> stub: comments.sql is still applied by make schema
<stub> It isn't on production anymore.
<wgrant> Right, that's what I meant.
<stub> Right. Thought I tossed it from dev too along with trusted.sql
<wgrant> I think trusted.sql might still be applied, it's just empty.
<wgrant> Even on prod
<wgrant> Yeah
<stub> I'll leave nuking comments.sql for someone who needs the LOC :)
<StevenK> % loc-contributions 'Ian Booth'
<StevenK> 12535
<wgrant> That's what you get for writing code.
<StevenK> I've been writing code, and I'm at 473
<StevenK> This DB branch will probably push me to 700
<StevenK> wgrant: You're only at -28
<wgrant> Huh
<wgrant> Oh
<wgrant> I'll get the blame for all the DB patches.
<wgrant> Since I do the merge.
<StevenK> That's a bit mean.
<wallyworld> StevenK: they are all in progress
<wallyworld> but will be moved after the next bb run hopefully
<StevenK> wallyworld: We've had three deployments in the past two days, I'd hope some of them are done. :-(
<wgrant> Most of them were in the pipe that I reviewed yesterday
<wallyworld> StevenK: sure, but my pipe only just got through ec2
<wallyworld> that one is landing now
<wallyworld> i had to fix a test failure this morning
<wgrant> Need covering indices :(
<StevenK> stub: How is the review?
<stub> going
<stub> StevenK: It looks fine. I haven't had time to fully grok it for bugs, but there are tests for that.
<stub> StevenK: You want to land this today?
<StevenK> stub: I think the FDT queue is empty, so this could hopefully deploy Monday night
<stub> Ok. wgrant has already gone over the logic?
<wgrant> I wrote it.
<stub> ok. r=stub. I'll go over it again later but it seems fine.
<wgrant> Well, I gave a draft to StevenK and he polished it up. It seems good to me.
<stub> yer, I just need to grok the logic more to see if I can come up with any missed edge cases. What is there seems fine.
<wgrant> Most of the non-trivial logic is just generalised from the existing bug functions.
<wgrant> wallyworld: If you're still around, you're probably well-placed to consider https://code.launchpad.net/~wgrant/launchpad/improved-bugremovesubscriptions-job/+merge/110480
<wallyworld> wgrant: looking
<wgrant> Feel free to object to the repr changes.
<wgrant> But I think we need something more detailed than we have now.
<wallyworld> wgrant: i did the join in my current branch, i'll back it out
<wallyworld> the repr changes are fine, i initially just wanted to get the jobs working. there were issues with repr not being correct for derived classes so i left them generic initially
<wgrant> Yep
<lifeless> speaking of security
<lifeless> http://blog.xen.org/index.php/2012/06/13/the-intel-sysret-privilege-escalation/
<lifeless> should give us chills
<wgrant> Yeah, that was a pretty good one.
<wgrant> AMD invented that instruction
<wgrant> So I'm pretty sure Intel's implementation is just buggy
<lifeless> differently valid ?
<wallyworld> wgrant: do we need the distinct=true
<wgrant> wallyworld: Yes, since BTF can have multiple rows per bug.
<wgrant> lifeless: Well
<wgrant> lifeless: Intel's definition doesn't make much sense.
<wgrant> lifeless: And EM64T intended to be compatible with AMD64.
<wgrant> Oh
<wgrant> That article actually has a section on that sort of thing
<wallyworld> wgrant: r=me, thanks. i'll have the aag one finished after soccer later tonight, i'll get yoo to review
<wgrant> wallyworld: Thanks.
<wgrant> And will do.
<lifeless> did you see http://io9.com/5918453/cooked-squid-inseminates-womans-tongue-cheek-and-gums ?
<wgrant> Heh
<lifeless> weird shit
<adeuring> good morning
* adeuring changed the topic of #launchpad-dev to:  http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: adeuring | Firefighting: - | Critical bugs: 3.47*10^2
<cjwatson> lifeless: the-intel-sysret-privilege-escalation> I suspect you will enjoy the discussion about the disclosure process, when that comes out
<StevenK> How do you file a bug about a instruction set.
<cjwatson> Why does contents-generation generate its own dists/ rather than copying the most recent published version, anyway?  I get why it has a separate dists/, but I don't really see why it has to go to the effort of generating its own.
<cjwatson> Oh, maybe it's just hard to get apt-ftparchive not to do so.
<wgrant> Because I hate apt-ftparchive.
<wgrant> But yeah, it's probably difficult to tell it otherwise.
<wgrant> Or was in 2006.
<cjwatson> It takes it 100+ minutes to generate all the Packages and Sources again, so avoiding that would be a nice improvement.
<cjwatson> Then Contents takes 90 minutes.
 * cjwatson files a bug.
<wgrant> Oh
<wgrant> It doesn't preserve?
<cjwatson> Not so you'd notice.
<wgrant> Heh
<czajkowski> I love cjwatson bugs, as I know they are never going to be invalid or dupes (in most cases)
<czajkowski> so I can traige them easily
<jml> heh
<czajkowski> jml: less so yours :s
<jml> that's cos I live on the edge, man
<jml> my stylez are too intense for the mainstream
<czajkowski> yes thats the reason :)
<jml> see what I did there? I put a 'z' in stylez. Backatcha!
<czajkowski> lol
<czajkowski> ivory: you're really making progress
<ivory> czajkowski: i feel like i learned more useful things in this week than in 12 years of school :)
<czajkowski> ivory: excellent!
<jml> ivory: :D
<jml> open source hacking ftw
<jml> I uploaded a package to a PPA that I accidentally made private. I then create a public PPA and uploaded that same package to it.
<jml> However, https://launchpad.net/~jml/+archive/consumer-apps-tools has no record of any build or upload, and when I try to dput again I get told that Package has already been uploaded to ppa on ppa.launchpad.net
<jml> Nothing more to do for create-usc-ppa_0.1_source.changes
<jml> what's going on?
<wgrant> jml: dput -f
<jml> wgrant: why does LP think it's already uploaded?
<wgrant> jml: dput (for not very good reasons) creates a .upload file locally when it completes an upload.
<wgrant> LP doesn't.
<wgrant> That's dput
<jml> ahh.
<wgrant> I have a theory that it was a feature added just to confuse people.
<jml> my brain, she grows!
<czajkowski> wgrant: you don't say!
<jelmer> wgrant: is there another explanation for those files?
<wgrant> Hah
<jam> I'm trying to add a simple 'does the script load' sort of smoketest for the fix-translations script (I'm also adding it to the launchpad scripts/ folder). Can anyone point me to where that test should live?
<jam> I guess I see some doc tests for 'copy-translations-from-parent' script under: lib/lp/translations/doc/...
<jam> or maybe that isn't a doctest
<jam> no, they are
<wgrant> jam: lib/lp/services/scripts/tests/test_all_scripts.py may be of interest
<jam> wgrant: thanks, that looks like the smoke test I wante
<jam> wanted
<StevenK> wgrant: db-devel r11683
<jml> nyah ah ah ah ah! my patch got into this run of buildbot
<jml> now I'll get results before the end of my working day
<StevenK> jml: Lies.
<wgrant> StevenK: Excellent.
<wgrant> Monday it is.
<StevenK> It has missed buildbot by about 15 minutes, butmeh.
<wgrant> It has nearly three days.
<StevenK> Yes, hence 'meh' :-)
<jml> oh right.
<jml> I can't seem to use the API against launchpad.dev
<jml> I get an error about SSL.
<wgrant> Yeah, cert verification will break that.
<jml> httplib2.SSLHandshakeError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
<jml> wgrant: is there a work around?
<wgrant> I either hack launchpadlib to not verify the cert, or generate a new cert for launchpad.dev and tell it to trust that.
<cjwatson> I often end up hacking httplib2.
<cjwatson> (search for disable_ssl and change False defaults to True)
<cjwatson> I wish there were an environment variable override.
<cjwatson> dogfood has the same problem.
<jml> wgrant, cjwatson: thanks.
<wgrant> jml: In lazr.restfulclient._browser.RestfulHttp.__init__, add disable_ssl_certificate_validation=True to the super call
<jml> wgrant: even better, thanks.
<cjwatson> There are three sites you need to change, IME.
<cjwatson> I always forget the full list.
<cjwatson> Which is why I've started editing httplib2 instead.
<wgrant> Yeah
<wgrant> Trial and error works for me...
<cjwatson> Because not all the call sites are in the same file.
<wgrant> Yep
<wgrant> jml: That's cheating.
<jml> wgrant: it's not cheating if you win.
<jml> wgrant: anyway, I haven't tested at all, or written automated tests, and it needs a comment and I can't find those other call sites you mentioned.
<wgrant> I don't remember what they look like.
<wgrant> It's just a matter of killing things until you run out of tracebacks.
<jml> yeah. I'll poke at that.
<jml> although I really need to move on from LP hacking to my next project.
 * jml has lunch instead
<czajkowski> wgrant: any idea what back end project this belong so
<czajkowski> 12:30 < popey> if you search in the video lens, it goes off to that website to get results
<czajkowski> 12:30 < popey> http://videosearch.ubuntu.com/v0/search/
<czajkowski> 12:30 < popey> see^^
<czajkowski> 12:30 < popey> and AIUI that backend is in lp somewhere, but I dunno what it's called
<popey> it's okay I'll ask u1 people
<wgrant> popey: I think that's candiru
<popey> nice timing!
<wgrant> But don't quote me on that.
<popey> 12:36:22 < popey> found it!
<popey> 12:36:23 < popey> https://launchpad.net/candiru
<popey> :D
<wgrant> Heh
<popey> thank you.
<czajkowski> popey: I di say the only person that might know would be wgrant
<czajkowski> *did
<popey> hehe
<benji> bac: I just noticed your message.  So the tests try to capture sys.stdout but now they should capture sys.__stdout__, right?
<nigelb> How the hell did wgrant reach -18281 :)
<wgrant> Hm?
<wgrant> That sounds off by a factor of 10
<wgrant> or more
<wgrant> Maybe in my all-time commit history.
<wgrant> But not since the policy was introduced...
<nigelb> I mean.... you're higher than StevenK by an order of magnitude
<wgrant> Oh
<wgrant> That was indeed from before the policy.
<wgrant> That explains it.
<jml> wgrant: oh yeah, two things
<jml> removing the debbugs db from the tree
<wgrant> Yay
<wgrant> Kill
<wgrant> But how?
<jml> also, rolling back other people's commits
<wgrant> Oh, did I delete it already?
<wgrant> That's right, I deleted a duplicate copy.
<jml> as in, those are two things you did that inflated your pre-policy score
<jml> the debbugs db most significantly
<wgrant> There's still one copy lurking there.
<jml> right.
<deryck> abentley, adeuring, ivory -- let's try this:  https://plus.google.com/hangouts/_/62374aa6d5f5280d63ca32c5288e194fd3ed8ba5?authuser=0&hl=en
* bac changed the topic of #launchpad-dev to:  http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: adeuring, bac | Firefighting: - | Critical bugs: 3.47*10^2
<timrc> engineering trying to look at a failed  build log for a package with tildes in the version must manually encode the tilde and this method apparently only works in Firefox as Chrome does not allow you to do this? For packages with tildes is there a way we can provide an alternative link with the filename hashed?
<sinzui> I think I have finally found a way to make sprite links that work properly in all browsers
<sinzui> I see my copy of " YUI 3 Cookbook" has finally shipped.
<rick_h> sinzui: just downloaded that to the kindle last night myself
<sinzui> rick_h, It was not available two weeks ago when I checked. I don't want to dead tree
<rick_h> sinzui: ah, yea just saw it when davglass posted it to twitter that the book arrived
<rick_h> bad thing about the kindle setup is you can't 'wishlist' it like a paper version so have to wait
<rick_h> pre-order that is
<sinzui> :/
<sinzui> I now hope the book is not that good so I am not tempted to buy it again for my nook
<rick_h> heh, we'll see. Hopefully start digging into it tonight.
<rick_h> it is oreilly, you can upgrade on their end for $5
<rick_h> and get epub/pdf/mobi
<rick_h> http://oreilly.com/register/?CMP=PAC-8JY296190230
 * sinzui looks
<lifeless> cjwatson: oh? I shall look out for it
<timrc-exercise> sedentary lifestyle is starting to show
* adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: bac | Firefighting: - | Critical bugs: 3.47*10^2
<wgrant> timrc: You don't have to manually encode the tilde. You just have to use Firefox to click the link.
<timrc> wgrant, that didn't work for me
<timrc> wgrant, not in FF 13.0 at least
<wgrant> timrc: The launchpad.net link, not the launchpadlibrarian.net link
<wgrant> Chromium mangles during the redirect.
<timrc> wgrant, let me try again, I believe that didn't work
<timrc> wgrant, ah, it does work from the launchpad.net link (I guess I hadn't tried since I wasn't logged in via FF)
<wgrant> Yeah
<wgrant> We send the link with ~ encoded.
<wgrant> Chromium decodes it during the redirect.
<wgrant> So copying the URL out of the address bar gets the broken one.
#launchpad-dev 2013-06-12
<StevenK> wgrant: httplib2.debuglevel = 1 produces 200MiB of output
<StevenK> steven@undermined:~% tail -n 1 qa-mawson-output | wc -c
<StevenK> 206338762
<wgrant> As you'd expect
<StevenK> wgrant: Blah. http://pastebin.ubuntu.com/5756839/
<StevenK> wgrant: That's 3 POSTs in a row, so I don't know what the hell lplib is doing
<wgrant> StevenK: What are the responses?
<StevenK> reply: ''
<StevenK> For the first one
<StevenK> 502 for the second
<StevenK> reply: '' for the third
<StevenK> wgrant: reply: '' and 502s make me sad
<wgrant> StevenK: Indeed
<wgrant> StevenK: The second call is probably a retry of the 502
<StevenK> reply: '' was first
<StevenK> So I suspect all but the first are retries
<wgrant> StevenK: Wasn't that the response to the initial GET, not the POST?
<wgrant> Oh, or you mean that there was a third POST after your paste?
<StevenK> No
<StevenK> send: 'POST /devel/ubuntu/raring/i386 HTTP/1.1\r\nHost: api.qastaging.launchpad.net\r\nContent-Length: 71817072\r\nAuthorization: OAuth realm="OAuth", oauth_nonce="73131629", oauth_timestamp="13710006
<StevenK> reply: ''
<StevenK> connect: (api.qastaging.launchpad.net, 443)
<StevenK> send: 'POST ...
<StevenK> wgrant: So I'm not sure what would cause an empty reply to a POST
<wgrant> StevenK: There was no separate status?
<StevenK> wgrant: No status, no headers, nothing
<StevenK> wgrant: I'm a bit stuck where I should fix this invalid transition critical that you filed a dupe of
<wgrant> StevenK: What are the options?
<StevenK> wgrant: We can hack lazr.jobrunner to log "Skipping Completed job" in runJob, but I don't think we can impose a cronscript to rip out the entry from the rabbit queue when it completes a job
<StevenK> We could also do that in lp.services.job.runner, rather than lazr.jobrunner
<wgrant> StevenK: No, we can't dequeue the job
<wgrant> We have to handle it once celery receives it
<StevenK> Right
<StevenK> So runJob can just log it and move on
<wgrant> StevenK: Probably. The normal jobrunners only fetch jobs that are WAITING, so it would make sense for celery to skip them similarly
<wgrant> Before it even gets into the job ode
<wgrant> code
<StevenK> Well, we don't have that luxury, since we're pulling jobs off the queue
<wgrant> I know
<wgrant> But you can possibly check as soon as you pull a job off the queue
<wgrant> The job should ideally be thrown away before we reach the common celery+cronjob code
<StevenK> wgrant: I can do it in acquireLease() which should be early enough
<StevenK> wgrant: But is probably the wrong place. lp.services.job.runner.BaseJobRunner.runJob() should be okay to log and return
<wgrant> StevenK: Sounds reasonable
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/no-completed-jobs-for-celery/+merge/168859
<fginther> what causes a new password to be generated for access to a private ppa?
<fginther> I'm working with some ppa automation which accesses private ppas. I'm seeing an issue where sometimes it works and sometimes it gets a 401. While debugging I've noticed that the password is different each time. Each time the script runs, it uses the same credentials, but a different cache dir
<fginther> please disregard my earlier question. It doesn't look like the ppa access was setup correctly.
#launchpad-dev 2013-06-13
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/no-completed-jobs-for-celery/+merge/168859 with http://pastebin.ubuntu.com/5760296/ to claw it back to net-negative if you approve
<wgrant> StevenK: I'm still not entirely sure that that's the right place to put it
<wgrant> And we also need to handle FAILED
<wgrant> And RUNNING
<wgrant> We handle that in the normal runner by only retrieving jobs that are WAITING
<wgrant> Ideally the list of jobs from celery should be filtered at a similar level -- above runJob
<StevenK> RUNNING should be handled by acquireLease
<StevenK> wgrant: I wasn't able to figure out which code path pulls job off the queue, so we could make it skip and drop jobs that aren't WAITING
<wgrant> StevenK: lazr.jobrunner.celerytask.RunJob.run is the relevant method.
<wgrant> Difficult to override without just copying it
<StevenK> It just calls job_source.get, we could make it return None if the job is not WAITING and then have run pike if the job is None
<wgrant> That sounds less than ideal
<StevenK> Right
<wgrant> StevenK: I'd put a waitingness check between the get and acquireLease
<wgrant> Probably just in a local override for now
<StevenK> wgrant: But isn't that too high again? That isn't pulling stuff off the queue, it's running the job
<wgrant> StevenK: That's the celery task
<wgrant> It's a RunJob task that is given to celery
<wgrant> It examines and runs the job
<wgrant> It's the appropriate level
<wgrant> And indeed the highest level that isn't inside celery
 * StevenK stabs LP
<nigelb> LP will stab you back... :p
<StevenK> Now this script is causing an OOPS because it REALLY REALLY REALLY wants to send a mail
<StevenK> nigelb: Duh. I have scars.
<nigelb> Heh
<StevenK> wgrant: lp.services.job.celeryjob.CeleryRunJob is a subclass of RunJob, we could check the status is WAITING pretty easily and then not call super() ?
<wgrant> StevenK: Except that the superclass method is the bit that calls job.get
<wgrant> I'd just override the method and we can push it upstream into that project that nobody else uses eventually
<StevenK> wgrant: Or I could just fix it in lazr.jobrunner?
<wgrant> StevenK: You could. I guess adding the extra bit to the interface isn't a problem, as there's only one implementation.
<StevenK> wgrant: http://pastebin.ubuntu.com/5760341/
<StevenK> wgrant: I couldn't work out how to get the enum into jobrunner
<StevenK> wgrant: And I can't work out how to test it.
<wgrant> StevenK: I'd consider 'if not job.canRun()', perhaps
<StevenK> wgrant: Then it can go live in jobrunner if it's that.
<wgrant> StevenK: That's the point :)
<StevenK> wgrant: Then it turns into a two liner, because I can't work out how to get at the logger
<wgrant> StevenK: :(
<StevenK> wgrant: https://code.launchpad.net/~stevenk/lazr.jobrunner/only-run-if-canrun/+merge/169105
<wgrant> StevenK: That doesn't break its tests?
 * StevenK finds out
<StevenK> FileJob\\\' object has no attribute \\\'canRun\
<StevenK> Pity
<StevenK> wgrant: This would be easier if r49 didn't fail tests locally.
<StevenK> wgrant: From what I can see, both r49 and r51 with my changes fail locally in the same way.
<wgrant> StevenK: Can you fix the test failure?
<StevenK> wgrant: I don't get why r49 fails
<StevenK> r49 being 'trunk'
<StevenK> wgrant: So you're not happy wih https://code.launchpad.net/~stevenk/lazr.jobrunner/only-run-if-canrun/+merge/169105 ?
<wgrant> StevenK: Ah, that's right, I got distracted by the test suite hanging and then other things
<StevenK> wgrant: The test suite breaks for me locally, but the failures from trunk are identical at least
<wgrant> StevenK: Right, now that you've fixed the test job implementation
<StevenK> wgrant: Happy to build 0.12, stuff it into sourcedeps and then toss a branch that jumps to 0.12 at ec2
<wgrant> StevenK: As long as you also add a canRun method to the LP job implementations...
<StevenK> Oh, I thought that exists
<StevenK> *existed
<StevenK> Let me find the base class
<wgrant> No
<wgrant> There's actually already an is_pending method, which is WAITING/RUNNING/SUSPENDED
<wgrant> Might be better to make canRun() is_runnable
#launchpad-dev 2013-06-14
<StevenK> Right
<StevenK> wgrant: But no matter the method name it's still status == WAITING ?
<wgrant> StevenK: Yes
<wgrant> It doesn't really make sense to run anything else :)
<StevenK> wgrant: I've shifted canRun() to is_runnable
<wgrant> StevenK: Most useless attribute title ever.
<wgrant> "Whether or not this job is ready to be run immediately", possibly.
<wgrant> :)
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/sha256-model/+merge/169345 and http://pastebin.ubuntu.com/5763741/
<StevenK> sha256_digester = hashlib.sha256() and friends makes me sad, but you're contining the bad
<wgrant> StevenK: What is sad about them?
<wgrant> Besides the slight repetition for md5/sha1/sha256
<StevenK> It's repeated in a few places, makes me think it should perhaps be refactored into a helper method
<StevenK> wgrant: But, r=me, and that script looks excellent.
<StevenK> wgrant: jobrunner-012's diff has updated
#launchpad-dev 2014-06-09
<wgrant> cjwatson: <3
<wgrant> cjwatson: You've tested that the split-out extra-overrides scripts actually work, I assume?
<wgrant> cjwatson: You say that it needs ~ubuntu-archive read access, but why can't it use login_anonymously()?
<cjwatson> wgrant: Oh yeah that would work too
<cjwatson> willfix
<cjwatson> wgrant: I'm going to test end-to-end now that python-launchpadlib is installed on pepo
<wgrant> Great
<wgrant> Thanks
<wgrant> Some of us sadly do not have that capability.
<cjwatson> Oh, hey, I already use login_anonymously
<wgrant> Heh
<cjwatson> Well, I could test it somewhere else, but it's handy to be able to diff the result and check that I'm not missing some other random dependency
<cjwatson> The ported tests should be pretty close to good coverage, but it's hard to do an out-of-process run with sufficient mock objects
<cjwatson> wgrant: Glad you asked - I found and fixed one bug in the translation to webservice-speak.  It works now after a bit of bodging of the shell script to write to somewhere different.
<cjwatson> (And a change to the lockfilepath to avoid arguing with production ...)
<wgrant> cjwatson: Excellent
<wgrant> cjwatson: Do you intend to kill off lp-query-distro.py soonish? It seems like it should be just about trivial.
<cjwatson> wgrant: Yes, just wanted to do it in a separate step
<wgrant> cjwatson: Sure, just checking.
<cjwatson> Ideally along with moving that shell loop into maintenance-check.py, but if I don't have time for that then writing a launchpadlib version of lp-query-distro is trivial.
<cjwatson> And it can hardly take longer to execute than lp-query-distro does :P
<cjwatson> lp_publish@pepo:~$ time /srv/launchpad.net/production/launchpad/scripts/ftpmaster-tools/lp-query-distro.py supported
<cjwatson> utopic trusty saucy precise lucid
<cjwatson> real    0m8.588s
<cjwatson> user    0m7.968s
<cjwatson> sys     0m0.400s
<cjwatson> I mean FFS
<wgrant> Yeah, Zope startup time is nice.
<wgrant> Well, and importing a million lines.
<wgrant> Can probably be done in one launchpadlib request.
<cjwatson> [series.name for series in ubuntu.series if series.status not in ("Experimental", "Obsolete")]  or some such
<wgrant> Yep
<wgrant> Also Future
<wgrant> I think
<cjwatson> So two requests because you need the distribution first, but whatever
<wgrant> If that still exists.
<wgrant> True.
<cjwatson> querydistro only excludes those two statuses
<wgrant> Heh
<cjwatson> Not to say that it's correct
<wgrant> In fact, it should really be Future/Obsolete
<wgrant> Experimental is a live status...
<wgrant> Not that Ubuntu uses it.
<wgrant> In fact, how does that not generate things for v-series etc?
<cjwatson> Pretty sure it does
<wgrant> Unless it's just because there are no packages...
<cjwatson> It's awesome
<cjwatson> Oh, maybe it just used to
<cr3> hi folks, might there be a coding guideline for launchpad that describes when to use the various logging levels? (I asked in #launchpad but here is more appropriate)
<cjwatson> wgrant: Oh, because we haven't created series beyond utopic
<wgrant> cjwatson: Right, but we had before.
<wgrant> So u-series files should still exist.
<cjwatson> Yep, and I think it used to generate for u-series.
<cjwatson> lp_publish@pepo:~$ ls -l /srv/launchpad.net/ubuntu-archive/ubuntu-misc/ | grep series
<cjwatson> -rw-r--r-- 1 lp_publish lp_publish        0 Apr 23  2012 more-extra.override.q-series.main.supported
<wgrant> cr3: There's no formal distinction between DEBUG/INFO and WARNING/ERROR, but the latter two generate OOPSes and generate cronspam.
<cjwatson> -rw-r--r-- 1 lp_publish lp_publish        0 Oct 18  2012 more-extra.override.r-series.main.supported
<cjwatson> -rw-r--r-- 1 lp_publish lp_publish        0 Apr 25  2013 more-extra.override.s-series.main.supported
<cjwatson> -rw-r--r-- 1 lp_publish lp_publish        0 Oct 17  2013 more-extra.override.t-series.main.supported
<cjwatson> -rw-r--r-- 1 lp_publish lp_publish        0 Apr 17 17:19 more-extra.override.u-series.main.supported
<wgrant> Within the two categories is basically up to the engineer; we normally have scripts logging at DEBUG anyway.
<wgrant> cjwatson: Hah
<cjwatson> wgrant: Do you have an opinion on the matter at the tail end of https://rt.admin.canonical.com/Ticket/Display.html?id=72323 ?
<wgrant> cjwatson: Is that the puppet vs juju thing?
<cjwatson> Yeah
<wgrant> Meh; we'll know pretty quickly if it breaks. As long as it doesn't get autoremoved.
<cjwatson> I'm assuming that (a) the publisher isn't moving to juju in a long time and (b) maybe it's not worth arguing about
<cjwatson> We can just keep it in -soyuz-deps until then I suppose.
<wgrant> Indeed.
<cjwatson> cr3: What are you working on, out of interest?
<cjwatson> Oh, useful thing I noticed in my RSS feed: Linux 3.15 adds an atomic exchange-two-files (and I think also directories) operation
<wgrant> The renameat extension actually went through?
<wgrant> I remember discussion about that like a year ago.
<wgrant> But it stalled
<wgrant> Forever
<wgrant> Ah, renameat2, nice.
<cjwatson> Yeah.  It's still the annoying renameat2(..., RENAME_EXCHANGE) API rather than exchangeat(), but whatever
<cjwatson> So I guess we just need to make sure support for that is in Python and then we can use it in 2020 or something
<cjwatson> Will deploy that ubuntu-archive-publishing change after this publisher run finishes, and monitor the next run.
<wgrant> I'm glad we avoided having sensitive data like an ~ubuntu-archive OAuth token on pepo...
<cjwatson> Eh, this option is better, but it's not like pepo isn't already a much scarier machine than snakefruit (which has such a token).
<wgrant> Indeed, I intend to convince people to gradually fix snakefruit :)
<cjwatson> What would you suggest there?  It's kind of integral to how most of it works
<cjwatson> Absent things like operation-limited oauth tokens
<wgrant> Random people not having logins to it :)
<cjwatson> Oh well yeah
<cjwatson> Though I think it's just us and the graphite server
<wgrant> It is always going to be scary, but it needn't also be the errors log machine or whatever it is.
<cjwatson> It's a lot less scary than lillypilly was :)
<wgrant> lillypilly was retarded, not scary.
<wgrant> But yes
<wgrant> Now we just need to get the errors stuff onto some new UE equivalent of carob, and all will be marginally less dreadfully.
<wgrant> -ly
<cjwatson> Oh, right, I think app-logs is new since I last checked /srv
<cr3> cjwatson: sorry for the lag, I'm working on another project using Zope3 and I wanted to adopt similar coding as Launchpad like those described here: https://dev.launchpad.net/Hacking
<cjwatson> Ah, I thought it might be in Launchpad itself, OK
<stub> cr3: ERROR for broken things that we can continue from. FATAL for things that mean termination. WARNING for things someone should look at. INFO so you can follow normal operation. DEBUG when you are debugging a problem, DEBUG2->DEBUG9 when you need more DEBUG
<stub> cr3: Hows life?
<cr3> stub: hey dude, life is still awesome in montreal. how about you, still loving thailand?
<stub> cr3: Just another usual coup, same as always really.
<cr3> stub: you probably have as many coups as we have demonstrations in the streets :)
<stub> Somewhat itchy feet... might move next year. Still, every time we talk about it we end up deciding Thailand suits us best for now.
<stub> cr3: Oh, and as wgrant mentioned WARNING and ERROR additionally get an OOPS report generated
 * cjwatson deploys ubuntu-archive-publishing
<cr3> stub: I'd wait until the end of Game of Thrones before moving back to Australia :)
 * cjwatson has a brief moment of "but Game of Thrones is being filmed in Northern Ireland", but I see what you mean :)
<cjwatson> OK, new ubuntu-archive-publishing looks good
#launchpad-dev 2014-06-10
<wgrant> cjohnston: How'd you go today?
<cjohnston> wgrant: so I'm a little confused about how we want to do this...
<cjohnston> From talking to cprov, my understanding is that LP just gets diff's as plain text from librarian...
<wgrant> Correct.
<cjohnston> so LP doesn't have the notion of hunks...
<wgrant> Correct.
<cjohnston> He mentioned there may be code in bzrlib that we could reuse..
<cjohnston> But other than that, I'm not even really sure how we want to go about attempting to do this?
<wgrant> What do you mean by that last bit?
<wgrant> bzrlib has some useful patch parsing utilities, eg in bzrlib.patches.
<wgrant> It may be worth using them, but parsing manually isn't terribly difficult if not.
<cjohnston> We had initially discussed emailing out relevant hunks for IC's
<wgrant> Yes.
<cjohnston> so if only one hunk has an IC, only email the one hunk
<cjohnston> If it's plain text that LP has, how do we determine what one hunk is?
<wgrant> The same way the 'patch' tool does.
<wgrant> Diffs have a machine-readable format; that's what they're for.
<wgrant> So we can work out what a hunk is.
<wgrant> bzrlib.patches has utilities to do that, but the format is obviously simple enough that it's not a huge challenge to do it ourselves.
<cjohnston> ok
<wgrant> So we probably need to work out where the hunk dividers are, then map the comments we have onto hunks, so we can work out which hunks we care about.
<cjohnston> ok..
 * cjohnston will bbiab, guest just showed up
<StevenK> I *think* codehosting has some code using bzrlib.patches for detecting hunks.
<wgrant> cjwatson: Why didn't you use multiprocessing again?
<wgrant> Also, os._exit? :)
<cjwatson> wgrant: Because hell on earth getting the file descriptors set up properly
<cjwatson> wgrant: os._exit is the correct thing to use in forked-but-not-execed child processes
<cjwatson> I think the particular problem was that multiprocessing kind of wants you to use a Queue to pass things around between processes, and I couldn't make that work - I was getting can't-write-to-closed-fileobj IOErrors all over the place
<cjwatson> and it was taking an unnecessarily long time to debug
<wgrant> Hrm, OK.
<cjwatson> I use os._exit in this kind of situation because I don't want double calls to atexit-registered functions happening under my feet
<cjwatson> It just requires care to make sure that any fds that have been written to are flushed or closed first
<wgrant> Yeah.
<cjwatson> Wow, these constant hwdb test failures aren't incredibly annoying at all.
<wgrant> Not at all!
<wgrant> I deferred investigating them because we had approval to remove hwdb, but then that was sadly rescinded.
<cjwatson> Sigh
<wgrant> So maybe it's worth actually working out what's going on there at some point.
<cjwatson> Didn't remove it fast enough :)
<wgrant> At least it's not like polls, where we removed it and then had to put it back.
<cjwatson> I'll spend a bit of time testing the apt source-caching backport today, I think.
<cjwatson> 16:26 <cjwatson> yeah, I was going to just feed it a different db path
<cjwatson> 16:27 <cjwatson> copy in the binary dbs and time source generation, then compare Packages/Sources
<cjwatson> 16:27 <cjwatson> then time cached source generation
<wgrant> :)
<cjwatson> Built the tip of http://anonscm.debian.org/gitweb/?p=apt/apt.git;a=summary on precise, on advice from Michael; if that works then he'll do a backport to trusty
<cjwatson> And then I think we'll SRU that into trusty and build the same source for precise
<cjwatson> Until such time as pepo is on trusty
<cjwatson> Michael wants to get most of this code into precise for other reasons, but since this is the riskiest bit of the trusty upgrade we might as well QA it in one step
<wgrant> Agreed, a-f is the most problematic and least testable bit of the upgrade historically.
<jtv> Does anyone know why "bzr branch lp:ubuntu/trusty/apache2" breaks?
<jtv> This post says that lowering bzr's output verbosity works around the problem!  http://bridge.grumpy-troll.org/2014/06/four-miscellaneous-things/
<jtv> I hit the same problem with lp:ubuntu/trusty/firefox but not with some other branches.
<wgrant> jtv: Breaks with an error about LazyParentGraphCacheSomething having no such revision?
<wgrant> Graph(StackedParentsProvider(bzrlib.repository._LazyListJoin(([CachingParentsProvider(None)], []))))
<jtv> Yup.
<wgrant> https://bugs.launchpad.net/bzr/+bug/888615
<_mup_> Bug #888615: UDD branch freshness checker breaks on incomplete history <Bazaar:Confirmed> <https://launchpad.net/bugs/888615>
<jtv> Nasty.  It's clearly not just the freshness checker...
<wgrant> It is.
<wgrant> If you disable it, it'll work fine.
<jtv> Oh, and the verbosity setting controls that?
<wgrant> I assume it makes it sufficiently quiet that the freshness checker doesn't bother running, as it won't display its message.
 * cjwatson deploys germinate parallelisation
#launchpad-dev 2014-06-11
<wgrant> cjwatson: How's the performance looking?
<cjwatson> wgrant: About a minute for each run
<wgrant> cjwatson: Nice.
<cjwatson> Down from three and a bit
<cjwatson> Looks like reasonably consistently 53s or nearby
<cjwatson> wgrant: Is there any way to sensibly preload things that are modelled using SQLMultipleJoin(..., orderBy=)?  Storm doesn't seem to cache that in a way that can be satisfied by a previous load_referencing, not even if I expand out load_referencing's query to include a matching ORDER BY.
<wgrant> cjwatson: No, Storm only caches by PK
<wgrant> You'll need to turn the SQLMultipleJoin into a cachedproperty
<cjwatson> Aha
<wgrant> Depending on the callsites, that can be very easy or very hard.
<wgrant> Which one is this?
<cjwatson> SPR.files
<wgrant> Ah, most things should be happy with that being a list.
<wgrant> What are you trying to accelerate?
<cjwatson> I got pissed off with PPA publisher performance and thought I'd try the obvious preloads
<wgrant> Ah, yeah
<wgrant> OK, this is really confusing. I'm watching a UOS session where you're talking, while talking to you about something totally different on IRC.
<cjwatson> Everything else from source publication is easy to make constant query count by number of source packages.  Haven't tried binaries yet
<cjwatson> Heh
<wgrant> It's not a huge amount of work now that model cachedproperties aren't verboten.
<wgrant> eg. SPR.files probably only needs invalidation in SPR.addFile
<wgrant> But gina needs checking.
<cjwatson> Right, I assume I just grep for SourcePackageReleaseFile as the starting point
<wgrant> Yep.
<cjwatson> Yay, that makes my record_two_runs test for source publishing pass.  Thanks.
<cjwatson> I'll try the same for binaries in the morning.
<cjwatson> Or maybe I'll have insomnia and JFDI.  Should actually try sleeping while more of the tests run though ...
<wgrant> cjwatson: Did you go to bed? :)
<cjwatson> wgrant: For a bit, yeah
<wgrant> Disappointing.
<cjwatson> Well, I'd written the binaries code before saying that above
<cjwatson> Should have that up for you to look at a bit later today
<cjwatson> Any luck with livefs*?
<wgrant> Spent most of the day on log analysis and scalingstack stuff, but I'll hopefully have that sorted out tomorrow in the unlikely event that nothing else comes up :/
<wgrant> cjwatson: How many queries were actually executed per pub?
<wgrant> I guess one for the xPR, one for the xPR?Fs, then one for each LFA and one for each LFC.
<wgrant> Plus the xPN
<wgrant> And sometimes the component and section.
<wgrant> And maybe one for the BPB, its SPPH, its SPR, its SPN and its component in the case of binaries?
<wgrant> Though that might be done by postgres the view, I can't quite recall.
<cjwatson> I think it was about five or six per pub
<wgrant> in the view
<wgrant> Ugh, some of that lint is mine.
<cjwatson> Seven per source and six per binary if I'm reading my terminal scrollback correctly
<cjwatson> SPR, SPN, SPRF, number of files * (LFA, LFC)
<cjwatson> BPR, BPB, BPF, LFA, LFC, (maybe SPN), BPN
<cjwatson> Also sometimes Section.  Component is effectively constant
<wgrant> I was wondering how SPRs had more. Of course, multiple files.
<cjwatson> Two in my test but of course often more.
<wgrant> Hm
<wgrant> We need to check the Storm cache config.
<wgrant> I think it's capped at 10000 objects
<wgrant> It is for the
<wgrant> webapp, at least, I think
<wgrant> I might have uncapped it for scripts a few years ago/
<cjwatson> How many packages are there in the largest PPA?
<wgrant> A few thousand
<wgrant> 3000ish IIRC
<wgrant> We probably want to do this in reasonably sized chunks, but your diff is better than what we have now, so meh.
<cjwatson> I guess if we were to chunk it the preloading would have to move into archivepublisher
<wgrant> As it bloody well should.
<wgrant> None of that ridiculous publishing code belongs in registry :)
<wgrant> Or even soyuz.
<wgrant> Oh yes let's have the webapp publication objects know how to serialise themselves to disk, that's a good idea.
<cjwatson> Well, I mean into the caller rather than the callee
<cjwatson> Those get*PackagePublishing methods only have one call site each
<wgrant> Yeah
<cjwatson> (going out for a bit before UOS)
<wgrant> I've been meaning to investigate chunking DecoratedResultSet.__iter__ for a while.
<wgrant> It already does a bit of evil, so it wouldn't be too bad.
<wgrant> Have it iterate through chunks of a caller-specified size, so it does 10 bulk queries each over 1000 objects rather than one over 100000.
<wgrant> Ignore the missing order of magnitude there.
<wgrant> Inline comments are nice.
<stub> wgrant: it is capped, but you can override it in the launchpad.conf per script
<wgrant> cjwatson: How'd the source caching test go, modulo ENOSPC?
<cjwatson> wgrant: Looked fine, was about four minutes quicker
<cjwatson> Results looked correct
<cjwatson> I've asked mvo to do a sourceful trusty backport so that we can then build that lot on precise
<wgrant> :)
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/main/source/: 6392 pkgs in 35s
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/universe/source/: 39628 pkgs in 3min 37s
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 8s
<cjwatson> 15:53 <cjwatson> then second run:
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/main/source/: 6392 pkgs in 0s
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/universe/source/: 39628 pkgs in 6s
<cjwatson> (both with the new apt-ftparchive, but the first with no sources cache)
<cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 0s
<cjwatson> So does exactly what it says on the tin, I think.
<cjwatson> I wonder where I should put ubuntu-archive-publishing config, re https://code.launchpad.net/~cjwatson/launchpad/configurable-germinate-base/+merge/194340 .  Maybe I should just have a file where we look things up by os.environ["LPCONFIG"]
<wgrant> cjwatson: That or a config file named by LPCONFIG sounds reasonable.
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/preload-publisher-indexes/+merge/222768 should be ready for re-review now.
<wgrant> cjwatson: Thanks.
<wgrant> cjwatson: We'll want to test variously sized archives on DF to check for any weird postgres decisions on the huge queries, but that's nice and easy with NMAF (no pool required).
<cjwatson> no pool required?
<wgrant> Testing a-f at scale can only be done on pepo, as it depends on the pool being intact.
<cjwatson> oh, right, you mean we can let NMAF put the pool in place
<wgrant> No, we don't need a pool.
<cjwatson> I'm obviously not awake.  I understand now.
<wgrant> Well considering you were off IRC for two hours.
<cjwatson> Andreas had a didn't-want-to-sleep night.
<wgrant> :(
<cjwatson> Also, <3 caffeine
<wgrant> Heh
<cjwatson> OK, that a-f smoke-test on DF would have been more effective if I'd changed something.
<wgrant> Clean suites are quick to publish, at least.
<cjwatson> Oh, it did change.  Good.
<cjwatson> Just looking at the wrong publish-distro run.
<wgrant> Ah
<cjwatson> And see your previous comments about needing a pool.  Well, didn't explode.
<wgrant> It won't explode.
<wgrant> But the indices'll be pretty empty.
<cjwatson> Yeah, just memtest86+ in main.  Shrug.
<cjwatson> === modified file 'finalize.d/20-germinate' (properties changed: +x to -x)
<cjwatson> thanks, bzr
<cjwatson> Anyway, we can deploy once lpqateam catches up.  Shall I request that?
<wgrant> Sounds like a good plan.
<wgrant> Have you done one lately?
<wgrant> webops will ask for an RT
<wgrant> But otherwise as usual.
<cjwatson> Not lately, but I'd seen you switching to RTing things.
<cjwatson> Eh, what the heck did generate-extra-overrides just do on labbu
<cjwatson> Oh, maybe it's talking to production for its series information
<wgrant> Ah, heh, probably.
<cjwatson> Wonder what to do about that.  I could just cowboy -l dogfood in cron.germinate on labbu for now ...
<cjwatson> wgrant: bug 1328355 was applied live?
<_mup_> Bug #1328355: Distribution.guessPublishedSourcePackageName times out frequently <easy> <qa-ok> <timeout> <Launchpad itself:Fix Released by wgrant> <https://launchpad.net/bugs/1328355>
<wgrant> cjwatson: It was.
<cjwatson> Grr, api.dogfood.launchpad.net
<cjwatson> I guess we should SRU python-launchpadlib
<wgrant> I need to move the stagings at some point as well.
<wgrant> Luckily -l https://api.dogfood.launchpad.net/ works
<wgrant> er.
<wgrant> -l https://api.dogfood.paddev.net/
<cjwatson> Yeah, I was just thinking and checking that
<cjwatson> OK, cowboyed to death now.  Getting rid of lp-query-distro will kill one of the cowboys.
<wgrant> :)
<cjwatson> I'll probably shove the endpoint URI into the config infrastructure I'm writing for bug 1248869.
<_mup_> Bug #1248869: Stop the publisher from talking to people.canonical.com <soyuz-publish> <ubuntu-archive-publishing:In Progress by cjwatson> <https://launchpad.net/bugs/1248869>
<wgrant> Yup, makes sense.
<wgrant> You have the ndt etc. under control?
<wgrant> I might wander off.
<cjwatson> Yeah, go ahead.
<wgrant> Thanks.
<cjwatson> wgrant: I self-reviewed a testfix to preload-publisher-indexes to unblock buildbot.  Hopefully it looks right to you.
<lifeless> stub: is there a way in pytz to get a tzinfo object given a utc offset?
<wgrant> cjwatson: Looks good, thanks.
<cjwatson> I'm trying some test runs with it now.
<cjwatson> "/srv/launchpad.net/codelines/current/scripts/publish-distro.py -vv -d ubuntu --ppa --ms -A", hacked to publish only a single PPA, taking time between "Step C'" and "Step D", second run to ensure hot cache
<cjwatson> cjwatson/ppa, canonical-kernel-team/ppa, launchpad/ppa, kubuntu-ninjas/ppa
<cjwatson> Any other suggestions?
<wgrant> Those are pretty good options.
<cjwatson> -A publishes all series, but the more the merrier.
<cjwatson> (Actually I'm currently getting control data from r17041.)
<cjwatson> cjwatson/ppa                    (2.835) 1.260           (1.412) 1.375
<cjwatson> canonical-kernel-team/ppa       (33.896) 23.570         (9.972) 10.152
<cjwatson> launchpad/ppa                   (9.644) 3.739           (3.332) 3.367
<cjwatson> kubuntu-ninjas/ppa              (69.837) 43.198         (19.578) 20.031
<cjwatson> wgrant: ^- owner-name/ppa-name (r17041 first run) r17041 second run (r17043 first run) r17043 second run
<cjwatson> archives compare functionally identical (aside from compression timestamps and suchlike)
<cjwatson> looks pretty good to me
<wgrant> cjwatson: Still very slow, but quite an improvement. Is that just index generation time?
<cjwatson> I suspect the remaining time would bear some profiling, since it certainly shouldn't be maxing out on I/O there, but still
<wgrant> ie. C' -> D time?
<cjwatson> Right, just step C'
<cjwatson> Maybe lacking indexes for some of the rest?
<wgrant> All the queries you touched yesterday have sensible indices, but yeah, it's probably worth investigating now
<wgrant> I'll have a poke when you're done with labbu.
<cjwatson> I think I'm done now; that much is enough for me to say qa-ok I think
<wgrant> Agreed.
 * wgrant investigates.
<cjwatson> Released my vim lock on lib/lp/soyuz/scripts/publishdistro.py which remains cowboyed
<cjwatson> Feel free to mangle/revert
<wgrant> Thanks.
<wgrant> 335 queries
<wgrant> Hm
<wgrant> I might hack it to only do one suite.
<wgrant> Unless you have logs from the original run?
<wgrant> kubuntu-ninjas trusty binary-amd64 took 2.2s in my latest test run.
<wgrant> cjwatson: Did you keep your logs?
<wgrant> 6.1s with the old code.
<cjwatson> er, not in a file but I think I have scrollback
<cjwatson> Bah, not enough of it, sorry.
<wgrant> Nevermind, quick enough to rerun :)
<wgrant> I guess I should disable compression and see how much that shaves off
<wgrant> It was the limiting factor in my heavy caching experiments years ago.
<wgrant> bzip2 is only 5-10% of the time
<cjwatson> The {Plain,Gzip,Bzip2}TempFile scheme could be fixed to not do tempfile.mkstemp when it's just going to discard the result (in the gzip and bzip2 cases)
<cjwatson> I expect that's epsilon though
<wgrant> Yeah
<cjwatson> Oh, just the bzip2 case, not gzip
<wgrant> Will be interesting to see how much this improves things on prod
<cjwatson> Still, stupid code layout
<wgrant> We have network latency and auth etc. there.
<cjwatson> Yeah
<wgrant> So the many->notmany queries will be a much bigger win.
<wgrant> Plus wildcherry is a potato.
<cjwatson> kubuntu-ninjas trusty binary-amd64 took 23s on prod on 2014-05-13
<cjwatson> dunno how close to the same set of packages that is
<wgrant> Roughly 46000 additional queries in step C' across all suites
#launchpad-dev 2014-06-12
<wgrant> Er, 23000
<wgrant> 4000 queries to 10 for trusty amd64
<cjwatson> Score
<cjwatson> Tempting to get this deployed first thing tomorrow and find out.
<wgrant> The actual query time is negligible, so we're really down to Python optimisation now if we want to go further.
<wgrant> And at that point you might as well skip the Storm objects, since they could easily be 30% of the remaining overhead.
<wgrant> The SSDs should also help a lot with the cold cache case, but if not we can pull some tricks with index-only scans.
<cjwatson> Yeah, I guess my 17043 runs weren't really cold cache to start with
<cjwatson> cron.ppa's two publish-distro runs combined averaged 656 seconds in the 99 runs in today's log file
<wgrant> cprov: https://code.launchpad.net/~wgrant/launchpad/bug-1083709/+merge/222895
<cprov> wgrant: on it
<cprov> wgrant: now, isDriver(IPerson) works for IHasDriver and IHasDrivers ?
<wgrant> cprov: Right. isOneOfDrivers always did, but there's no reason for isDriver to not also consider .drivers.
<cprov> wgrant: looks sensible, I don't get the old isOneOfDrivers() ...
<wgrant> cprov: Thanks
<wgrant> cprov, cjwatson: I'm tagging possibly-RTM-relevant bugs "phone-rtm"
<cprov> wgrant: okay
<cprov> wgrant: will we consider the ppa publisher scale a problem for rtm ?
<wgrant> cprov: No, but it'
<wgrant> s mostly fixed now anyway.
<wgrant> RTM isn't going to use any significantly sized PPAs.
<cprov> right, the derived-distros is a way of avoiding huge PPAs
<cprov> but it kind of hold us a little more to a-f
<wgrant> No more than Ubuntu does.
<cprov> true
<stub> lifeless: pytz.FixedOffset(minutes), which never seemed to make it into the README
<lifeless> stub: AHHA
<lifeless> stub: <3
<cjwatson> cprov: We identified PPA publishing performance as an issue when figuring out what to do for RTM, but it wasn't the reason we decided to use a derived distribution instead.  The reason for that was that PPAs are insufficiently isolated from their parent distribution: they still use the parent in builds, and chroots aren't archive-specific.
<cjwatson> The model works better if we split out an entire new series.
<cprov> cjwatson: I see, thanks for contextualising it.
<lifeless> cprov: any chance, if I file a bug, that adding specs via the API would be done soon ?
<lifeless> cprov: it looks like bugs are adding via the /bugs collection, but there is no /specifications top level collection, so the createSpecification method isn't exposed
<lifeless> wgrant: ^ if you happen to be in an awake state before cprov answers :))
<cprov> lifeless: I can certainly look at it and figure out how much effort would be involved
<lifeless> cprov: that would rock :)
<cprov> lifeless: cool, *bug* me and I will discuss it later with wgrant
<lifeless> cprov: wicked - thanks! https://bugs.launchpad.net/launchpad/+bug/1329424
<_mup_> Bug #1329424: cannot create specification via API <Launchpad itself:New> <https://launchpad.net/bugs/1329424>
#launchpad-dev 2014-06-13
<lifeless> wgrant: oh yeah - hi and see ^
<wgrant> lifeless: Hi
<StevenK> I thought createSpec did work via the API
<StevenK> I have so forgotten how slow bzr grep is
<lifeless> wgrant: hai
<wgrant> cprov: https://code.launchpad.net/~wgrant/launchpad/bug-736005-begins/+merge/223035
<cprov> wgrant: sure
<cprov> wgrant: r=me
<wgrant> cprov: Thanks
<cprov> wgrant: dummy dummy test removed, huh ?
<wgrant> cprov: The test just verifies that view.sequence works
<wgrant> But I removed view.sequence
<cprov> wgrant: it's funny how code evolves and things tests like that remains, because they were always the same value by definition. There was never a need to test it, specially involving all that expensive setup.
<wgrant> Mhm.
<wgrant> dogfood's also currently running some schema updates to make suggestions about 100x cheaper.
<wgrant> Will see how that's looking tomorrow.
<cprov> cool
<wgrant> Oh and of course it finishes just as I'm about to hit suspend and sleep.
<Ursinha> jtv: Holland payback time :)
<Ursinha> netherlands, even :)
<jtv> Ursinha-afk: yup, I wasn't expecting that!
#launchpad-dev 2015-06-08
<wgrant> blr: How's it going?
<blr> wgrant: bit of a conundrum actually, zope testbrowser is apparently unable to set the disabled attribute on a control (set by js when the bzr setbranch form is hidden)
<wgrant> blr: zope.testbrowser doesn't execute JavaScript.
<blr> yep
<wgrant> JS stuff should be testing using YUI3 unit tests.
<wgrant> blr: Can you have a look at the buildbot failures soon to unblock landings?
<blr> wgrant: sure
<blr> wgrant: I suppose it doesn't matter, there's not much value in adding a new doctest.
<wgrant> blr: Right, adjust existing doctests if need be, but avoid adding new bits.
<wgrant> And don't attempt to test JS; it won't work.
<blr> would be useful to look at moving new browser tests to phantomjs perhaps.
<blr> wgrant: fixed the doctest, will grab some lunch and look at the smoketest failures
<blr> wgrant: would it be better to drop requester from the repr, or ammend the permissions?
<wgrant> blr: I'd amend the permissions, I think.
<wgrant> blr: Pretty much everything can see Person anyway, so this is a one-liner in security.cfg
<blr> wgrant: done, thanks. How do I go about poking buildbot?
<wgrant> blr: bzr lp-land --testfix
<wgrant> blr: Have you fixed all three failures?
<blr> wgrant: yes, one doctest and the others were both resolved with permissions
<wgrant> Yep
<blr> wgrant: oh, need a new MP?
<wgrant> blr: yep
<blr> wgrant: https://code.launchpad.net/~blr/launchpad/dont-fear-the-repr/+merge/261338
<wgrant> blr: lgtm
<blr> wgrant: cheers (I'll leave you in peace to _actually_ be not here today)
<wgrant> Feel free to ask questions. I'm around, just not around.
<blr> wgrant: thanks :)
<blr> wgrant: buildbot seems rather unhappy, unable to connect to db.
<wgrant> blr: That's a normal flaky test.
<blr> wgrant: ah good, had me worried.
#launchpad-dev 2015-06-09
<wgrant> blr: What're you working on atm? Did you sort out the big branch's test failures?
<blr> wgrant: I did yep, I'm looking at https://bugs.launchpad.net/launchpad/+bug/676769 atm
<mup> Bug #676769: resubmitting a merge proposal should reuse the old commit message <code-review> <confusing-ui> <lp-code> <Launchpad itself:Triaged by blr> <https://launchpad.net/bugs/676769>
<wgrant> blr: That should be quick, but don't put too much effort into it -- the resubmit workflow needs to be entirely redesigned.
<blr> wgrant: sure. Anything I should be thinking about today prior to the stakeholder meeting?
<wgrant> blr: Not really, I don't think.
<wgrant> blr: Is the big branch ready for review?
<blr> wgrant: yes if you could have a look again thanks.
<blr> wgrant: oh one omission that I fogot to ask you about regarding the licensing for the git icons - does the entire text of the creative commons license need to be in LICENSE?
<blr> forgot even
<blr> or is it sufficient to state that the images are licensed CCv3 and provide a link
<wgrant> blr: It needs to include the entire text if we're including the icon.
<wgrant> Where's the icon used atm?
<blr> push instructions
<blr> wgrant: updated the license
<wgrant> blr: Did you explore splitting the page into a bzr half and a git half? You could then hide the non-default half behind an expander or something by default, making the page less awkward for projects that don't care about the other VCS.
<wgrant> I've just selected Bazaar, so why is the next thing on the page a Git command?
<blr> wgrant: err that's weird
<blr> you aren't seeing the push instructions update on changing the radio?
<wgrant> Hrm, I have JS issues, apparently.
<wgrant> Recreated the container over the weekend, must have broken something.
<blr> you should see the form toggling from the bzr|git radio
<blr> Didn't consider the expander, do you think that would work better?
<blr> would usually consider using an accordion when there are more than 2 sections, but here it seemed fairly binary
<wgrant> Oh, my container was too well secured.
<blr> heh
<wgrant> blr: That's working much better. We probably still want to provide the ability to set the other side's options, though.
<wgrant> (splitting the page into sections by VCS would make that easier, and also improve the non-JS case)
<blr> wgrant: oh, I suppose in that case we really will need a split.
<blr> frankly, I didn't consider the case of needing to set both. :|
<wgrant> It's not common, but it isn't very difficult, and it makes the path for migration much less unclear.
<blr> that's true. So I suppose we need a +setbranch/git and +setbranch/bzr
<wgrant> I want to set up my new Git trunk so I can test that my CI systems build against it, but I can't do that without changing my project's main VCS away from Bazaar :(
<blr> right
<wgrant> I think they can be on the same page.
<wgrant> Hidden like now.
<wgrant> But there's a $something that'll show the non-default one.
<blr> ok a link stating [Configure non-default (git|bzr)] ... something like that?
<blr> could we use the horizontal space?
<blr> would be a bit of a break from convention for LP forms, but I'd imagine most people have the resolution (would be nice to see those stats, did we ever get GA access?)
<beuno> where default is just something expressed in the UI?  does it have other consequences?
<beuno> also, hi!
<wgrant> Evening beuno.
<wgrant> beuno: It just selects which VCS is preferred when showing listings, push/clone instructions, etc. at the moment.
 * beuno nods
<wgrant> In the UI it's not referred to as the "default VCS", just the "VCS", but the other one can be used as well if it exists, if you click the relevant link.
<blr> hi beuno
<wgrant> eg. LP's default VCS is Bazaar, but because it has a Git repo too there's a Git link on https://code.launchpad.net/launchpad
<wgrant> Most projects won't be doing that long-term.
<wgrant> But it's important for migration and some weird projects that are really several codebases in one project.
<wgrant> (eg. lots of PES things)
<beuno> and I guess the rationale for splitting them into parallel universes is because you can't merge between them
<wgrant> And they're very difficult (and usually not interesting) to list together.
<wgrant> Due to the repo vs. branch differences.
<beuno> ah yes
<beuno> that thing
<wgrant> Having them as separate worlds is fine for most cases and adequate for all, but we need to make the other world accessible.
 * beuno nods
<blr> wgrant: any thoughts on how to best approach the layout?
<blr> are there are existing UI patterns for that sort of thing (i.e. 1. Configure this, but 2. optionally configure this) in LP? Would be best to follow existing convention, but I can't think of anything
<wgrant> blr: It need not be obvious. Just a widget somewhere that says something about "$VCS settings" and causes the rest of the widgets to appear. You already have something sort of along those lines in the description of the default VCS widget, where it mentions the other one is still available.
<wgrant> That could potentially be replaced with a single link somewhere that says "let me poke the other one too"
<wgrant> I don't know of anything similar.
<wgrant> It's very rare that we have two alternate worlds that sometimes need to exist in parallel.
<blr> yep, nothing springs to mind. Ok, I'll re-work it, and if it seems cludgy we can try a different layout I guess.
<blr> having 2 distinct forms will make the doctests happier at least.
<wgrant> blr: They're probably still the same form, but I guess see how it turns out.
<cjwatson> I've got various mostly UI branches up for review from the tail end of last week, in case they got missed
<cjwatson> Currently working on merge detection
<wgrant> Whoops, forgot about those, sorry.
<cjwatson> Except I need like four times as much coffee to do the turnip side of merge detection. :-P
<wgrant> cjwatson: Just an API call that takes a commit ID and a collection of other commit IDs and calls merge_base on each of them?
<cjwatson> That bit's easy, but I think it's also useful to determine the commit at which it was merged into the left-hand history, analogous to bzr
<wgrant> hahaha left-hand history
<cjwatson> I'm forever being annoyed at other sites for making it hard to find when something was merged
<wgrant> I've a feeling we're not in Bazaar any more.
<cjwatson> Yeah I know but it's actually useful :P
<wgrant> But https://git.launchpad.net/launchpad/log/?h=db-devel makes SO MUCH SENSE.
<wgrant> Who needs topology :)
<wgrant> cjwatson: Do you have a strategy for determining the merge point?
<wgrant> You'd basically need to rewrite merge_base in Python.
<wgrant> I think.
<cjwatson> We can use merge_base to speed up the common case of unmerged MPs.
<wgrant> True.
<cjwatson> So first, exclude all MPs where merge_base(source, target) != source
<cjwatson> But then it's graph theory in Python
<cjwatson> Hence coffee
<wgrant> At least it should only have to be done once per MP, I suppose.
<wgrant> Hm.
<cjwatson> I think a lot of that work can be amortised across all MPs
<wgrant> You can't actually terminate it early, even.
<wgrant> Oh, unless you use GIT_SORT_TOPOLOGICAL, I suppose.
<wgrant> Then I think the merge point is just the last one you see.
<cjwatson> If I can't do it reasonably I'll just leave merged_revision_id empty
<cjwatson> or null or whatever it is
<wgrant> I guess we could possibly also store the last point we've closed up to.
<wgrant> Then you can .hide(previous_commit) to limit the search sapce.
<wgrant> But not much point if it performs adequately anyway.
<cjwatson> Going for a walk to think about the algorithm.  Whatever I come up with I'll test on everything in LP's bzr-personal repo to see how it performs.
<cjwatson> If it can do that it can probably take more or less anything.
<cjwatson> Looks like the optimisation of running merge_base on everything first to test if it's merged at all isn't actually an optimisation.  merge_base isn't that quick on large histories, and git_merge_base_many isn't exported.  So we'd have to roll our own logic there even if we left aside the mergepoint detection.
<cjwatson> I have an algorithm which roughly works based on merge_base for the first pass, but it took nearly two minutes to run over all the LP bzr-personal branches.
<cjwatson> Also, repo.walk(oid, GIT_SORT_TOPOLOGICAL) takes time equivalent to walking the entire history before it returns any results at all.
<wgrant> Yep
<cjwatson> That's a little under two seconds for LP.
<cjwatson> I think it may be best to roll our own rev walker for this; we can order topologically without having to walk the whole history first.
<wgrant> How?
<cjwatson> Depth-first traversal until we hit a node not all of whose children we've seen yet, at which point we push that onto a queue and look at it later.
<cjwatson> Memory requirements should be no worse than libgit2's topological sort must already be doing.
<wgrant> Children or parents?
<wgrant> Traverse from which end?
<cjwatson> Seems like a bug that libgit2's toposort is behaving this way, though.  If you haven't asked for time or reverse as well, there's no reason it needs to walk the whole graph first.
<cjwatson> Oh, right, we don't know the children yet
<cjwatson> Hm, OK, that is actually hard.
<wgrant> Yup :)
<wgrant> libgit2 isn't wrong here.
<cjwatson> It's also possible that it will be more efficient to just enumerate all the left-hand IDs up front.
<cjwatson> Anyway, meeting first ...
<cjwatson> (I have tests for this algorithm which pass, so shouldn't be too hard to refactor once I work something better out)
<cjwatson> Fetching a complete list of left-hand parents for LP is about six times as fast as starting a topological walk and fetching the first commit from it
<bigjools> wgrant: hey there - is there any published documentation on handling of PPA keys and how they are used / secured? Basically, something that convinces a layman of the security.
<wgrant> bigjools: You wrote it :P
<wgrant> There's no documentation that I know of, though.
<bigjools> wgrant: I may have written it but now I am a customer ;)
<bigjools> looking for something official
#launchpad-dev 2015-06-10
<blr> wgrant: that seems to be working better with the expanders (less code too), although I seem to have introduced a validation regression - will get to that tomorrow.
<wgrant> blr: Great.
<wgrant> Let me know if you run into any trouble.
<blr> thanks - I'll be in town for an hour and a bit in the morning to pickup a monitor, but back before lunch.
<blr> looking forward to no longer hunching over my laptop :P
<wgrant> You've been single-monitor all this time?
<wgrant> My condolences.
<cjwatson> I'm not *totally* sure that my dual-monitor setup is exactly ergonomic, with the RH one being above and to the right of my laptop
<cjwatson> But it's better than being single-monitor ...
<blr> made sure the panel is vesa mountable - will be good to have two eventually.
<cjwatson> My T450s is on the right continent now, so hopefully running 1920x1080 + 1600x1200 won't be too hideously confusing
<wgrant> Yay
<cjwatson> wgrant: You're right (git-merged-revno), it's much simpler to put merged_revision in the model, thanks.
<wgrant> cjwatson: I'd call it something displayish if it were expected, but merged_revision is fine internally.
<wgrant> ...
<wgrant> s/expected/exported/
<cjwatson> Yeah, I didn't export it
<wgrant> Bah
<wgrant> Goddammit, branchrevision
<cjwatson> ?
<wgrant> You can't have been that cold.
<wgrant> Just a failed scan despite a few pushes during the day.
<wgrant> cjwatson: Have you heard anything about git support in mojo?
<cjwatson> Nope
<wgrant> Tempted to switch something like turnip git-ward now that things are generally functional.
<cjwatson> But mojo is definitely an issue there.
<cjwatson> I guess we could run an import.
<wgrant> Yeah, as long as we don't use gpgsig the import will be fine.
<wgrant> We should dogfood at least *something* ASAP now it's reasonable.
<wgrant> And the lack of CI on turnip makes it an easy target :)
<wgrant> Also it'
<cjwatson> I suspect the firewall will need a tweak
<wgrant> s a git server hosted on bzr.
<cjwatson> For self-imports
<wgrant> Not just a firewall tweak, but yeah.
<wgrant> Anything under .launchpad.net is blacklisted.
<cjwatson> I wanted to finish merge detection before we started dogfooding.
<cjwatson> Otherwise it'll just be annoying
<wgrant> True.
<wgrant> I was thinking that distro MPs didn't do it, so it couldn't be too annoying.
<wgrant> But it's distro *branches* that don't.
<cjwatson> Yeah, there are XXX comments about that
<wgrant> (also I assumed something based on UDD being reasonable, which is itself unreasonable)
<cjwatson> Much easier to have sensible logic without branch lifecycles.
<wgrant> Quite.
<wgrant> Deletable branches are cool.
<cjwatson> wgrant: One reason I included target_default on the form for now is that otherwise if you retarget a repository to personal and back then the target defaultness goes away and you don't know why and have no UI way to restore it.
<cjwatson> But I agree that the permission limit there is problematic.
<cjwatson> Perhaps it should simply not be possible to retarget a target default.
<wgrant> cjwatson: The lack of UI way to restore it will go away once the product config page is reworked.
<cjwatson> Yeah.  Retargeting a target default repository is a slightly weird thing to do anyway, though, and has lots of intersecting permissions.
<wgrant> Yep.
<wgrant> It's inconsistent with the owner-target default checkbox.
<wgrant> But the permissions for that are much less intractable.
<cjwatson> Exactly.
<cjwatson> That's just edit plus checking that there's no conflict.
<wgrant> Yep
<cjwatson> I'll work on something to make the target uneditable with an explanatory message in this case.
<cjwatson> wgrant: What's the modern equivalent of the onKeyPress thing you called out in https://code.launchpad.net/~cjwatson/launchpad/git-repository-ui-edit-target/+merge/261232 ?
<cjwatson> I can't find anything in LP that's using anything different
<blr> cjwatson: I'm not certain, but perhaps he was suggesting adding an event handler to the JS rather than using the widget event?
<cjwatson> blr: I'm still at the cargo-culting level with LP JS :-/
<cjwatson> Or JS in general really
<cjwatson> Are you thinking of something to do with YUI.Event?
<cjwatson> Oh, there are some .on('keypress', blah) calls in our JS
<cjwatson> OK, that I can work with, looks straightforward enough
<blr> cjwatson: you might want to clarify with wgrant, but at a guess that's what he meant.
<cjwatson> Yeah, it makes sense now that I've found it
<blr> cool :)
<cjwatson> Ideally I'd figure out how to make the JS live with the widget, mind
<cjwatson> But I guess it's already separate
#launchpad-dev 2015-06-11
<blr> wgrant: is there something magical about fill-slot="extra_info"?
<blr> the template has source/target/prereq branch and description, however they're not enumerated in the template - they're set as initial_values in the view, I just don't understand how they're rendered
<blr> hmm nvm, I think I see
<wgrant> blr: Did you work it out?
<blr> wgrant: half way! Have the resub form rendering the new field, and have updated the model/interface, just need to repopulate the text widget
<wgrant> blr: It probably just uses the generic-edit template which renders all nominated fields in order, and sets the values from the view's initial_values property.
#launchpad-dev 2015-06-12
<wgrant> blr: ui-project-setbranch has conflicts; can you merge devel and resolve them?
<wgrant> And would it be worth cherrypicking the semi-unrelated menu changes into a separate branch, to get the "Configure translations" -> "Configure Translations" bits out of the big diff?
<blr> ugh, yep sure
<blr> wgrant: can do sure, if it is obscuring the primary intention of the branch.
<wgrant> blr: I think it should be pretty easy to put in a separate branch, and it would shrink the 2200-line diff by quite a bit...
<wgrant> Just a bzr merge -i from the old branch into the new plus a couple of fixups where they overlap should work.
<wgrant> Then we can review and land the new branch before the old, and the diff will be a whooooole lot more sensible.
<blr> wgrant: sounds reasonable
<wgrant> blr: New JS is much better, thanks.
<blr> yep I think that should work well enough.
<blr> will let you know when I have the 2 branches
<wgrant> I think there should only be like one file that needs the two sets of changes disentangled.
<blr> wgrant: wow, managed to blow up more tests than anticipated. Also update the BMP factories. Ready for you now if you have a moment: https://code.launchpad.net/~blr/launchpad/recycle-commit-message/+merge/261793
<wgrant> blr: Explosive tests are fun.
<blr> wgrant: is there a reason there's no argument for 'description' on resubmit? (in the interface)
<wgrant> blr: Someone probably forgot to add it there when the field was added.
<wgrant> Probably an idea to fix that too.
<blr> wgrant: yep, I did - just checking I wasn't missing something :)
<blr> as much as I like the keyboard on the lenovo, it is such an improvement using a proper mechanical keyboard...
<wgrant> As laptop keyboards go, the T4[45]0s have by far the best I've used.
<wgrant> But yes, a proper desktop mechanical keyboard is hard to beat.
<blr> had a look at the new macbook the other day - impressive hardware, but there's even less travel on the keys now than previous models... it would be intolerable I think heh
<wgrant> The one-port-to-rule-them-all model?
<wgrant> Yeah, certainly shiny and expensive.
<blr> hah yeah
<wgrant> But the keyboard is way worse than an MBA
<wgrant> And that's saying something...
<blr> wgrant: hmm I didn't receive a pqm notification for that last lp-land, and I don't see anything in the queue..
<wgrant> blr: buildbot had spuriously failed.
<blr> wgrant: ah that old chestnut.
<wgrant> lp:~person-name-100028/ubuntu/+source/unique-from-factory-py-line3422-100026/+git/gitrepository-100029
<wgrant> The factory creates the best URLs.
<StevenK> wgrant: Not enough -deactivatedaccount-deactivatedaccount in the person portion
<wgrant> StevenK: I'll have you know that no account has more than 17 of them.
<wgrant> https://launchpad.net/~deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount-deactivatedaccount for reference
<wgrant> Which isn't actually deactivated.
<wgrant> Also I guess that technically has 18, since I missed the first one, as it doesn't start with -
<StevenK> Hahaha
<wgrant> cjwatson: Oh, good catch.
<wgrant> Thanks.
<cjwatson> Some day I'll find a problem with one of your MPs that isn't entirely trivial :-)
<wgrant> I'm normally sufficiently pedantic about things like that :/
<cjwatson> Almost finished syncing everything over, should be back in the saddle properly this afternoon
<wgrant> Yay
<cjwatson> Trying to sort out my git-detect-merges branch, which is proving a bit awkward because passing a mock hosting client through the event infrastructure would be nightmarish, so I'm having to bite the bullet and sort out a fixture
<cjwatson> But despite being on PyPI now using turnip proper as the fixture is rather difficult - I don't want LP deployments to have to have a usable version of pygit2, and we don't have test_requires in LP at the moment
<wgrant> Considered ZopeUtilityFixture?
<cjwatson> So I'm doing a quick Twisted emulation of the relevant bits of the API in the short term
<cjwatson> Hm, yeah, that might be easier if I turned GitHostingClient into a utility, it's true
<cjwatson> Easier than writing something that emulates the other end of the API and then having a fixture to override the endpoint URL, which is what I'd been doing
<wgrant> Yep.
<wgrant> There are also various libraries around that mock out requests responses.
<wgrant> I've gone through three so far for webhooks stuff
<cjwatson> And GHC really ought to be a utility.
<wgrant> And they all mostly work.
<wgrant> But they're a bit weird.
<cjwatson> I have the guts of the Twisted version, but it means converting things over to AsynchronousDeferredRunTests and then drinking heavily.
<cjwatson> So maybe that's a blind alley.
<wgrant> You got the order wrong, but sure.
<cjwatson> I think order is immaterial, they can be done concurrently
<wgrant> Indeed.
<lifeless> cjwatson: whats the drinking heavily for? Is there framework stuff we can do to make it less drenched?
<lifeless> cjwatson: also, the glasgow haskell compiler should be a utility?
<cjwatson> GitHostingClient :-)
<cjwatson> y'all still advertise AsynchronousDeferredRunTest as "use at your own peril"; I'm happy to modify tests that are already using it but for something new where all I need is for LP to use a mock version of a utility, I think wgrant is right that ZopeUtilityFixture is simpler
<cjwatson> ... and indeed it is.  /me pushes
<lifeless> cjwatson: oh, we should remove that terror warning
<lifeless> I'll consult with jml
#launchpad-dev 2016-06-15
<mwhudson> does anyone have a thing for downloading all the debs from a ppa?
<cjwatson> mwhudson: debmirror should be able to do that
#launchpad-dev 2016-06-17
<xnox>  W: Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/us.ports.ubuntu.com_ubuntu-ports_dists_yakkety_InRelease
<xnox> Date: Fri, 17 Jun 2016  8:06:11 UTC
<xnox> (bad)
<xnox> Date: Fri, 17 Jun 2016 03:15:36 UTC
<xnox> (good)
#launchpad-dev 2018-06-12
<guysoft42> hey all, how do I add code blocks or any syntax to launchpad bugs? Anything I search on google or in launchpad leads me to other bugs and not howtos
#launchpad-dev 2020-06-08
<SpecialK|Canon> jelmer: nice loggerhead MPs!
<cjwatson> Oh right, I didn't notice those somehow
<cjwatson> jelmer: How did you make python3 work without porting away from simpletal, since simpletal is py2-only at present?
<cjwatson> jelmer: Or wait, I see that simpletal 5 is py3, I guess I missed that because they don't have a bilingual version
<tomwardill> no YUI!
 * tomwardill mutters about ahead-of-time COW filesystems and ownership
<tomwardill> (everything fails if it's owned by root)
<tomwardill> none of the directories are consistent
<tomwardill> there's about 4 different checkouts in the image afaics
<SpecialK|Canon> :(
<tomwardill> ah, nope, it's just a documentation / set of code 1 / set of code 2 mismatch
<tomwardill> "buildbotâ112:116:Buildbot system user,,,:/var/lib/buildbot:/bin/false"
<tomwardill> yeah, a shell of /bin/false would do it
<tomwardill> you're not getting a shell to execute commands in that case!
<tomwardill> IT'S DOING A THING
<tomwardill> I can tell, all my CPU just went away somewhere
<SpecialK|Canon> :D
<tomwardill> blergh, lxc-start-ephemeral has been deprecated in favour of lxc-copy and no longer appears to work
<cjwatson> lxc-copy --ephemeral should do it, give or take
<tomwardill> lxc-start-ephemeral works, unless you give it a command to run,t hen it fails
<tomwardill> lxc-copy fails with what looks like an apparmour type failure (my host is a container, so may be related)
<tomwardill> "overlayfs cannot be stacked on top of zfs"
<tomwardill> "default/containers/buildbot-worker on / type zfs (rw,xattr,posixacl)"
<tomwardill> well.
<tomwardill> poop
<cjwatson> lxd vm to the rescue?
<tomwardill> yeah, guess so
<cjwatson> Probably simplest ...
<tomwardill> sigh... so close
#launchpad-dev 2020-06-09
 * tomwardill mutters at lxc and testr
<tomwardill> getting there, it'll at least start a lxc now, but testr fails to find any tests
<tomwardill> cjwatson: any idea how to debug testr failing to produce a list of tests?
<cjwatson> Do you have a .testr.conf ?
<cjwatson> I'd drop down a level at a time - next level down is probably "sudo /usr/local/bin/lp-setup-lxc-test <name of container> --list" I think?
<tomwardill> yep, that seems to work
<tomwardill> (I've hacked it to use lxc-copy and lxc-attach rather than lxc-start-ephemeral)
<tomwardill> which I suspect is the problem, but I can't work out why not
<tomwardill> testr appears to be executing the command successfully, so I assume it's either not receiving or failing to parse the output somehow
 * tomwardill -> lunch
<tomwardill> okay, can someone `apt-get install python-testrepository` and then do `testr list-tests` in a LP checkout and see if it produces anything?
<tomwardill> testr doesn'
<tomwardill> doesn't appear to work in xenial
<tomwardill> or at least, doesn't list any tests
<SpecialK|Canon> Well, /bin/sh: 1: xvfb-run: not found
 * SpecialK|Canon spins up a container
<tomwardill> oh, yes
<tomwardill> you will need to do that in a launchpad/rocketfuel container
<tomwardill> pappacena or cjwatson might be in a positiont o do that faster :)
 * pappacena doing
<cjwatson> also empty output, let's see if I can remember how this works
<pappacena> empty for me too https://www.irccloud.com/pastebin/2PM4CxJJ/
<tomwardill> the docs imply that should produce a list, but don't actually state it
<tomwardill> (or indeed how/where/what form the list would be)
<cjwatson> You do get one on stdout if you run the underlying command
<cjwatson> Let me poke a bit
<tomwardill> yeah, that's the symptoms I'm getting
<tomwardill> I want to say something like 'subunit V1 vs v2', but I've got nothing to back that up with
<cjwatson> It is possible
<cjwatson> Could try | subunit-1to2 or some such
<cjwatson> Or actually
<cjwatson> --subunit-v2 maybe
<tomwardill> cjwatson: aha!
<tomwardill> that did it
<cjwatson> testrepository imports subunit.v2 if it exists
<cjwatson> and changes behaviour
<cjwatson> and I don't think it existed on precise
<tomwardill> yep, that would do it
<tomwardill> right, lets see if I can make everything else work now
 * tomwardill undoes all the hacks
<cjwatson> We'll need to be a little careful because the buildbot master will still be precise (for the time being) and thus won't understand v2
<cjwatson> For test subunit output
<cjwatson> Feeding the output through subunit-2to1 for the time being will probably deal with that
<tomwardill> right
<tomwardill> that's the config I've got locally atm (precise master, xenial slave), so should be able to experiment with it
<cjwatson> Ultimately moving to subunit v2 will probably be a good thing though; should fix some encapsulation bugs
<tomwardill> s/slave/worker, sorry. Buildbot still has old references.
<cjwatson> subunit v2 was invented exactly because v1 got out of sync really easily
<cjwatson> And I introduced subunit v2 support to zope.testrunner when I did a bunch of work on it in 2018
<cjwatson> Mainly to massively simplify the LP fork
<tomwardill> ah, nice
<cjwatson> So it *should* work, but it's basically untried in an LP context
<tomwardill> well, my cpu is definitely doing something
<cjwatson> Because we couldn't really use it until we upgraded buildbot :)
<tomwardill> yep, it's running tests, with terrible output and weird formatting
<cjwatson> testrepository should hopefully be ingesting it all
<tomwardill> looks like it's failing to start postgres correctly
<tomwardill> but that's progress at least
<tomwardill> it also never finishes the step, so that's a thing.
 * tomwardill will look at that tomorrow, slightly early EOD for bicycle club
<cjwatson> You know I think I'll take TypeError: 'object() takes no parameters' as a signal that I should EOD and try again tomorrow
<SpecialK|Canon> lol, have a good evening :)
#launchpad-dev 2020-06-10
<tomwardill> I need someone who's better at bash parsing than I am:
<tomwardill> lxc-attach -n "$container_name" -- env -u LANG USER=buildbot $PWD/bin/with-xvfb $PWD/bin/test -vvv --shuffle --subunit "$@" | subunit-1to2
<tomwardill> how do I make that pipe be part of the subcommand?
<tomwardill> currently it's attempting to pipe the output of lxc-attach into it
<cjwatson> Does lxc-attach produce some extra stuff as well as the output of the underlying command then?
<cjwatson> But also, that seems backwards
<cjwatson> If you want v2 output you could just use --subunit-v2.  I thought you needed to produce v2 for testrepository and then use subunit-2to1 to convert it into something the master understands, though
<SpecialK|Canon> tomwardill: quote the bit after the --?
<cjwatson> It gets a bit awful with "$@" though
<cjwatson> Better to avoid that if we can
<cjwatson> BTW the other thing I noticed is that lpbuildbot/bzrbuildbot/subunittest.py is probably going to need to gain v2 support
<cjwatson> Oh, except that's only when we upgrade the master, never mind
<tomwardill> yeah, that's my backup plan for if the subunit-2to1 doesn't work
<tomwardill> (yes, that previous one was backwards)
<cjwatson> If lxc-attach is producing extra output, can you get it to shut up by adding the -q option?
<cjwatson> I think we want lp-setup-lxc-test to be producing a clean subunit stream without any extra guff anyway
<tomwardill> it's more that my output looks something like this: https://usercontent.irccloud-cdn.com/file/h1zsmSsw/image.png
<cjwatson> Right, I get that, I'm just wondering why you need to avoid the output of lxc-attach being passed to subunit-2to1
<cjwatson> It could only be a problem if lxc-attach is doing more than just passing through stdout from its inferior command
<tomwardill> I don't know tbh, I've got myself confused
 * tomwardill starts at the beginning again
 * cjwatson makes an LXC container to test.  Been ages since I had one of these ...
<tomwardill> I could probably just ship you my worker lxdvm
<cjwatson> Heh, I think lxc-create will still be faster
<tomwardill> I suspect this might end in having to upgrade lpbuildbot/bzrbuildbot/subunittest.py with v2
<tomwardill> (as you suggested)
<cjwatson> Tricky because praseodymium doesn't have the python-subunit version needed for that
<cjwatson> I think it should be avoidable in this pass
<cjwatson> (We can backport python-subunit if we have to, but let's try to avoid that work just now)
<tomwardill> cjwatson: my current lp-setup-lxc-test https://pastebin.canonical.com/p/Q2ZDHtyf9G/
<tomwardill> based on a xenial worker/test-lxc
<cjwatson> OK
<tomwardill> oh hey, got postgres to behave
<tomwardill> now just the subunit stuff to solve
<tomwardill> oh no, it's only partially behaving
<tomwardill> boo
<tomwardill> nope, postgres still refusing to start when run via buildbot. Works fine by hand
 * tomwardill stares at it
#launchpad-dev 2020-06-11
<cjwatson> tomwardill: lxc-attach seems to pass output straight through without modification, so I think you can just do lxc-attach -n "$container_name" -- env blah | subunit-2to1, much as you had above
<tomwardill> ah, right :)
<tomwardill> will give that a try once I've worked out why postgres is refusing to start
<tomwardill> thanks :)
<cjwatson> Though consider error handling
<cjwatson> As in, what happens if lxc-attach exits non-zero
<cjwatson> Pipes tend to lose that unless you take special care
<cjwatson> You still want to stop the container, but should preserve the exit code
<tomwardill> right
<SpecialK|Canon> someone with more familiarity with traversal want to review https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/385489 ?
<SpecialK|Canon> (because hi)
<tomwardill> cjwatson: before I break out pdb agian, any idea what might be causing: https://pastebin.canonical.com/p/djCTpmdCwF/
<cjwatson> tomwardill: bad working directory maybe?
<tomwardill> `pwd` agrees with my current directory
<cjwatson> or permissions?
<cjwatson> open_for_writing swallows any IOError that isn't ENOENT!
<cjwatson> quality
<cjwatson> if you add an else: raise into open_for_writing (which really ought to be there anyway), you might get a better error message
<tomwardill> hmm, I appear to have a bunch of files that are owned by root
<tomwardill> I suspect that is not ideal
<cjwatson> Oh, your lxc-attach arrangements don't seem to switch user
<cjwatson> add '$PWD/utilities/run-as buildbot' just before 'env', maybe?
<tomwardill> yeah, this is from pre-that step I think
<tomwardill> trying to work out which step is doing it
<cjwatson> Might be leftovers from a previous run?
<tomwardill> yeah, think so
<tomwardill> poking
<cjwatson> And yeah, lxc-start-ephemeral -u ... meant "switch to this user", and part of the reason I added utilities/run-as was that at least at the time there was nothing else with quite exactly the right semantics
<cjwatson> (lxc-attach -u and lxc exec --user both take uids rather than usernames; setpriv(1) didn't exist yet)
<tomwardill> the good news is that I'm just about at the point where I can hack master.cfg and it will repeatedly get to the same state so I can debug it
<tomwardill> ... I should pick a faster subsection of the test suite to test this
 * tomwardill twiddles thumbs a bit more
<tomwardill> run-as is giving me a permissions error on chdir to the build directory, but only when run via buildbot
<tomwardill> wtf
<cjwatson> User namespaces can cause much confusion sometimes, maybe ...
<tomwardill> yeah, something weird going on
<tomwardill> works fine run from a terminal
 * tomwardill sighs at the amount of shell/environment/namespace learning I don't know
<tomwardill> unsure how I'm getting permission denied changing to the directory that is cwd
<StevenK> That is clearly perms
<StevenK> You either aren't the owner, or in the group, or there's no +x
<tomwardill> drwxr-xr-x 20 buildbot buildbot 4.0K Jun 11 10:55 build
<StevenK> All the way up to / ?
<tomwardill> hmm, no, buildbot ownership stops at /var/lib
<StevenK> I'd expect that, but hopefully everything has +x
<tomwardill> looks like it
<tomwardill> I can be in that directory quite happily in a shell
<tomwardill> oh, wait
<tomwardill> maybe I can't in this situation
<tomwardill> wtf
<tomwardill> root@lptests-xenial_tfbWfo:/var/lib/buildbot/lp-devel-xenial/build# su buildbot
<tomwardill> Cannot execute /bin/bash: Permission denied
<cjwatson> buildbot outside and inside the container might not be the same thing
<cjwatson> I would get the most minimal possible reproducer you can manage and strace it
<cjwatson> and also make sure to be looking at permissions by id (ls -nl) inside the container
<tomwardill> so, yeah
<tomwardill> `/` was 0700
<tomwardill> ... that's a thing
 * tomwardill gets lunch, leaves it for future tom to worry about
<ilasc> and in this context of twom dealing with complex issues, I come along and ask theÂ rudimentary question: in LP how do we split a large MP in severalÂ smallerÂ MPs ? Do I just create separate git branches and open MPs for each new branch or is there something I canÂ do at the level of the large MP that I'm not yet awareÂ of?Â 
<StevenK> Years and years ago one of my friends did 'chmod 644 .*' as root in a top-level directory and then wondered why no one could log in
<ilasc> :)
<cjwatson> ilasc: Separate branches and open MPs for each.  Are you familiar enough with the git-level operations here?
<cjwatson> (Also, prerequisites in MPs may be useful, depending on how you lay out the branch structure)
<ilasc> thanks cjwatson, hmmm good question :) just to make sure I start on the right path, I assume I start creating the smaller new git branches from master ?
<ilasc> indeed figured prerequisites in MPs will be necessary
<cjwatson> Normally from master, yes.
<cjwatson> The "splitting commits" section of "man git rebase" may be useful.
<cjwatson> If the bits you need to split up are separate enough in their respective files, you can often manage it with "git add -p" or whatever equivalent exists in your IDE.  Failing that I sometimes resort to just dumping out the overall patch and editing it down to the bits I want before applying it, but editing patches by hand certainly isn't for everyone
<ilasc> great, ok, thanks Colin! it sounds like our approaches are similar in this case, I always go for editing patches by hand :)
<cjwatson> Oh, I'm glad I'm not the only person who puts up with that
<cjwatson> I do need it slightly less since I found tools to let me do line-by-line rather than hunk-by-hunk changes to the git index
<ilasc>  :)
<cjwatson> Also keep a git ref to the original thing around, then you can't lose it
<ilasc> good idea :)
<SpecialK|Canon> `git add --patch`'s editor option is <3
<cjwatson> I prefer vimagit since I discovered it last year sometime, but same sort of idea
<tomwardill> cjwatson: any idea wher eI need the umask change?
<cjwatson> I'm not quite sure, it was just a hunch as to how you might end up with mysterious mode 700
<cjwatson> Is the base container like this or just the ephemeral copy?
<tomwardill> good question, sec
<cjwatson> If the former, look in your build pipeline, if the latter, start from lp-setup-lxc-test and trace down
<cjwatson> (probably)
<tomwardill> yeah, it's the latter
<cjwatson> Not actually what I expected
<tomwardill> which makes sense, as lp-setup-lxc-test is the only bit I've actually changed
<cjwatson> Though it probably should have been since you reported different behaviour when running from a terminal
<tomwardill> yeah
<tomwardill> a hack fix would be to just chmod / ;)
<cjwatson> buildbot's buildslave runs with umask 077 by default unless you say --umask=022
<cjwatson> Maybe relevant?
<cjwatson> But you could also just umask 022 at the top of lp-setup-lxc-test ...
<cjwatson> I suspect that'll do it
<cjwatson> I could be wrong here, because I thought our buildbot worker config already did umask 022, but it's been some time since I looked at that and maybe it got lost somewhere along the way
<tomwardill> I'll have a look and give that a try
<cjwatson> puppet modules/launchpad/templates/buildbot.tac.erb has it
<cjwatson> Hm.  Did you write buildbot.tac or whatever the modern equivalent is for the workers yourself?  Or where did you get it from?
<tomwardill> I didn't write it
<tomwardill> came from lpsetup I think
<cjwatson> It might be a good idea to get sluagh:/srv/buildbot/lpbuildbot/buildbot.tac and compare
<cjwatson> lpsetup's might be wrong
<tomwardill> and it has an interesting thing:
<tomwardill> `umask = None`
<cjwatson> That might be from lpbuildbot demo/slave/buildbot.tac
<cjwatson> Which I'm not certain is in sync
<tomwardill> yeah, that makes sense
<tomwardill> asked for the real one
<tomwardill> well, it gets further, now to see if postgres works
<tomwardill> it's running tests!
<tomwardill> weeee
<tomwardill> now just subunit to work out
<ilasc> +!
<ilasc> +1
<ilasc> ... can't type :P
<tomwardill> now, how do I make it stop
 * tomwardill reboots the worker
<tomwardill> okay, might need to teach the test step about subunit 2
<cjwatson> --subunit-v2 | subunit-2to1 you mean?  or something else?
<tomwardill> piping the lxc-attach output through subunit-2to1 just reproduces the same problem of testr not understanding the ouput
<tomwardill> and trying to pipe the testr output through it still results in weird stdout in the logs and the step not understanding how many tests have run
<cjwatson> Ah, hm
<cjwatson> Maybe testr adds too much extra stuff
<tomwardill> hmm, or maybe I've done something wrong somewhere
<tomwardill> as `testr run --parallel --concurrency=2 --subunit --full-results '|' subunit-2to1` looks a bit weird, given the escaping around the pipe
<cjwatson> Where did you put the subunit-2to1 in that case?
<tomwardill> in the master.cfg
<cjwatson> What's the diff?
<tomwardill>             command=['testr', 'run', '--parallel', '--concurrency=2', '--subunit', '--full-results', '|', 'subunit-2to1']))
<cjwatson> Oh
<cjwatson> Well, yes
<cjwatson> That's an argv
<cjwatson> More or less
<cjwatson> It's not passed to a shell, so doesn't understand |
<tomwardill> which makes sense
 * cjwatson looks at buildbot.steps.shell
<cjwatson> So ... there's no fiddly quoting required for the arguments themselves there
<cjwatson> You *could* just try:
<cjwatson> command=['sh', '-c', 'testr run --parallel --concurrency=20 --subunit --full-results | subunit-2to1']
<cjwatson> Definitely a workaround, but ought to help
<tomwardill> the docs spec that you can give command as a single string
<tomwardill> and it does basically that
<tomwardill> (although that's in the latest docs)
 * tomwardill tries
<cjwatson> I am a bit suspicious 'cos I can't find what implements that, but maybe
<tomwardill> running
<cjwatson> But the sh -c trick should definitely work if that doesn't
<tomwardill> https://usercontent.irccloud-cdn.com/file/7xneaob9/image.png
<tomwardill> success!
<cjwatson> Progress!
<tomwardill> the stdout is good too
<tomwardill> okay, so I think that's all the problems worked through
<tomwardill> now I just need to document what they were, work out patches and file an RT to try this...
<tomwardill> concurrency 5 is making my computer VERY LOUD
<cjwatson> Nice
<cjwatson> Out of interest, does this fix the "unknown worker (bug in our subunit output?)" thing that we currently get?  Looked like it might from your image ...
<tomwardill> it seems to...
<tomwardill> we have a list of workers too!
<cjwatson> Ooh, does this let us download independent subunit streams from each worker?
<tomwardill> ooh, which tells you which worker ran which tests
<cjwatson> EXCELLENT
<tomwardill> I think the only 'stream' we get is the list of tests
<cjwatson> That will make debugging certain kinds of test isolation bugs so much easier
<tomwardill> if we upgrade from precise to xenial, do we need to rebuild the xenial LXC that we already have?
<cjwatson> Well, even separate lists of tests for each worker is a lot better than nothing
<cjwatson> I have no idea
<cjwatson> Hopefully not
<tomwardill> indeed, as I don't really want to have to try and maintain this script :)
<cjwatson> As long as you have something working locally, I think it's OK to debug it into existence a little bit on production if necessary
<tomwardill> hmm, getting some 'App server startup timed out' failures, but that may well be due to the load on the VM/machine
<cjwatson> Yeah, likely
<tomwardill> it's at 350% cpu usage and has eaten all the ram allocated to it
<cjwatson> om nom nom
<tomwardill> they're not on the same tests as the ones I had in the last run, so points towards that at least
<tomwardill> wish I'd left this machine in the basement now
<cjwatson> Heh
<cjwatson> This is great though, super-happy to see these improvements
<tomwardill> getting this out and working, then transcribing over to LXD will be super nice
<cjwatson> And hopefully LXD won't be too difficult after this
<cjwatson> Yeah
<tomwardill> and cleaning up/sorting lpsetup along the way
<cjwatson> I think I've decided I don't have enough brain to review https://code.launchpad.net/~pappacena/turnip/+git/turnip/+merge/385158 today.  I've reviewed the Launchpad bits that need to precede that ...
<tomwardill> fixed container cleanup and exit code return too
<SpecialK|Canon> nice
<tomwardill> okay, will work out a plan and extract / update the files required tomorrow morning
<tomwardill> but it's looking good/feasible now
<cjwatson> Non-lcy01 bionic image builders aren't working.  I've (belatedly) deployed staging equivalents to test this.  lgw01 is failing due to a glance API difference, bos02 is possibly something else but I haven't worked it out yet.
<cjwatson> wgrant: ^- could I have a quick review of https://code.launchpad.net/~cjwatson/canonical-is-charms/gss-glance-v2-private/+merge/385608 ?
<cjwatson> Looks like bos02 is probably the same thing after all.
 * cjwatson cowboys on lgw01 bionic staging to test
<cjwatson> Looks like that fixes it on lgw01, indeed
<wgrant> cjwatson: Ah, fun
#launchpad-dev 2020-06-12
<SpecialK|Canon> https://code.launchpad.net/~doismellburning/launchpad/+git/launchpad/+merge/385644 - my og_title assert is failing and it's not yet clear to me if my template code is wrong or I should be poking makeTemplateView to inject more info - would appreciate feedback please
<SpecialK|Canon> (both specific and general!)
<cjwatson> SpecialK|Canon: What's the error message?
<SpecialK|Canon> cjwatson: `testtools.matchers._impl.MismatchError: 'http' not in None`
<SpecialK|Canon> presumably because `request/URL` is None
<cjwatson> I normally write more like content.find('meta', property='og.title'), but your syntax should work too
<cjwatson> request/URL is used in og:url rather than og:title though
<SpecialK|Canon> cjwatson: property is a builtin in some modern Pythons though
<SpecialK|Canon> cjwatson: hence I went for the dict
<cjwatson> Ah yeah, fair point
<SpecialK|Canon> (otherwise I would have)
<cjwatson> "modern" :)
<SpecialK|Canon> bah sorry, og_url not og_title
<SpecialK|Canon> https://pastebin.canonical.com/p/y7S2VGN6YQ/ in full
<SpecialK|Canon> https://pastebin.ubuntu.com/p/DFWsYspMcf/ even
<SpecialK|Canon> very few things declare recommended_canonical_url, fine (I do mean to add tests for some of those later)
<cjwatson> Mkay, having a look
<cjwatson> I think soupmatchers would make this clearer FWIW
<cjwatson> I always like composite tests to test as much as they can at once, because it means you get better error messages
<SpecialK|Canon> Hm that does look interesting
<cjwatson> request/URL should evaluate to http://launchpad.test here, since that's what makeTemplateView sets up ...
<SpecialK|Canon> That's what I assumed; I'm struggling to find much in the way of docs about `request` though
 * cjwatson sets a breakpoint on recommended_canonical_url
<cjwatson> It's a LaunchpadTestRequest in this case
<SpecialK|Canon> https://zope.readthedocs.io/en/latest/zopebook/AppendixC.html#tales-overview helpfully tells me it's a thing, but not much more
<SpecialK|Canon> Ah
<SpecialK|Canon> ...yes in hindsight it is, isn't it
<cjwatson> Which is a thin layer over https://zopepublisher.readthedocs.io/en/latest/browser_api.html#zope.publisher.browser.TestRequest
<cjwatson> As for template language docs, I generally point at https://pagetemplates.readthedocs.io/en/latest/ instead of that old Zope 2 doc
<SpecialK|Canon> Digging a little more I suspect I may instead want view/url or context/url based on other usages, but I've not yet found anything definitive
<SpecialK|Canon> Ah thanks
<cjwatson> Ah
<cjwatson> SpecialK|Canon: alternate expressions after | only get evaluated if the previous expressions *fail*, not just return None
<SpecialK|Canon> Well, https://pagetemplates.readthedocs.io/en/latest/history/TutorialPart2.html?highlight=view%2Furl#inserting-text uses request/URL, but I'm still not clear on the exact different semantics
<cjwatson> SpecialK|Canon: Try "python: view.recommended_canonical_url or request.URL" instead
<SpecialK|Canon> cjwatson: ...that'll do it, thanks
<SpecialK|Canon> hurrah, cheers :)
<tomwardill> hopefully make css generation repeatable enough: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/385678
<tomwardill> note, this disables compressed output for now, it makes the file ~80K bigger, but it's a single file that is then cached
<tomwardill> but means SASS leaves the comments in, so I can work out which file is which!
<tomwardill> I'll re-enable compressed output when this bug is confirmed fixed
<cjwatson> Will you be able to compare between qastaging/staging/dogfood or something?
<cjwatson> r=me anyway
<tomwardill> that's the hope, yes
<tomwardill> we have enough environments that build this that I can see if we get consistent results from it
<tomwardill> I kind of dislike that we have rules that are dependant on the ordering of the filesystem
<tomwardill> (apparently)
<cjwatson> Yes, we clearly shouldn't
<tomwardill> hopefully this will fix it for now, a better fix would probably be to create a SCSS _index file and use that rather than the jsbuild mechanisms
<cjwatson> Yeah
<cjwatson> Bah, another buildbot failure due to me
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/385680 - quick testfix review?
<tomwardill> looking
<tomwardill> cjwatson: +1
<cjwatson> cheers
<cjwatson> I think that's about all the useful stuff I'm going to get done today.  Have a good weekend
<SpecialK|Canon> cjwatson: you too!
<SpecialK|Canon> https://code.launchpad.net/~doismellburning/launchpad/+git/launchpad/+merge/385644 if anyone's still around...?
<tomwardill> mind blanked, what's the env var I need to use lp-shell with a local instance?
<tomwardill> disables the SSL validation
<tomwardill> LP_DISABLE_SSL_CERTIFICATE_VALIDATION=
<tomwardill> that would do it
<SpecialK|Canon> so https://dev.launchpad.net/Running - presumably "Point your usual web browser at https://launchpad.test, and accept the local self-signed certificate." predates "it is advisable to try Launchpad in an LXC/LXD container"
<cjwatson> Probably, but I don't get the connection
<SpecialK|Canon> is there anything more modern somewhere else?
<cjwatson> What is your objection?
<SpecialK|Canon> cjwatson: well the latter means that doing the former won't achieve much
<cjwatson> Not so
<SpecialK|Canon> cjwatson: because I'm running LP in a container
<cjwatson> Works fine
<cjwatson> You do have to follow the link to https://dev.launchpad.net/Running/RemoteAccess
<SpecialK|Canon> and my host is not aware of launchpad.test
<SpecialK|Canon> Right
<cjwatson> It will be if you do the stuff in Running/RemoteAccess :)
<cjwatson> Running may just need to be reordered a bit
<SpecialK|Canon> "Works fine" *if* you have read some as-yet-unmentioned docs is...a stretch ;P
<cjwatson> Yeah, I hadn't noticed that RemoteAccess is actually mentioned further down the page
<SpecialK|Canon> But yes, thanks for the pointer, I'll rejig that a bit
<cjwatson> A few people have been through the docs without mentioning that :)
 * tomwardill always reads the entire paper before answering questions ;)
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/385683 add gitrepository.id to the API
<cjwatson> Good catch that it's misordered though
<tomwardill> (one of my previous statements is a complete lie)
<tomwardill> I think at least myself and pappacena were given both links at the same time, so it's less obvious
<cjwatson> Could be
<cjwatson> https://dev.launchpad.net/Running/LXD also emphasises it a bit at the end
<cjwatson> Anyway, not here :)
 * tomwardill -> also not here, before dog eats anything else he's not meant to because I haven't fed him yet
<SpecialK|Canon> Cheers :)
<SpecialK|Canon> Have good weekends!
