[00:03] <wgrant> lifeless: No.
[00:03] <wgrant> lifeless: I'll do it.
[00:03] <wgrant> It is rather intimidating.
[00:17] <LPCIBot> Project windmill-db-devel build #234: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/234/
[00:32]  * wgrant cries at r12970
[00:35] <lifeless> \o/ 12976
[00:35] <wgrant> !!!
[00:37] <LPCIBot> Project windmill-devel build #26: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/windmill-devel/26/
[00:55] <wgrant> lifeless: Just by looking at the diff I can see it is bad.
[00:55] <lifeless> wgrant: 12970
[00:55] <lifeless> ?
[00:55] <wgrant> Yes.
[00:55] <wgrant> It affects the API.
[00:55] <wgrant> When this was to most definitely not affect the API yet.
[00:56] <wgrant> It will break security unembargoing.
[00:56] <wgrant> (since the security team doesn't have upload privileges)
[00:57] <lifeless> they don't ?
[00:58] <wgrant> Hee hee hee.
[00:59] <wgrant> lifeless: I invite you to look at canonical.launchpad.security.AppendArchive
[01:00] <lifeless> wgrant: blink
[01:00] <lifeless> so how does anyone upload to Ubuntu ?
[01:01] <wgrant> lifeless: AppendArchive isn't used for uploads.
[01:01] <lifeless> class AppendArchive(AuthorizationBase):
[01:01] <lifeless>     """Restrict appending (upload and copy) operations on archives.
[01:01] <wgrant> It's used for protecting syncSource, basically.
[01:01] <lifeless> that seems misleading
[01:01] <wgrant> Yes.
[01:01] <lifeless> so
[01:01] <wgrant> It's also used for PPAs, I guess.
[01:01] <wgrant> But not primaries.
[01:01] <wgrant> And:
[01:02] <wgrant> wgrant@lamuella:/tmp/ubuntu-archive-tools$ ./edit_acl.py query --person ubuntu-security
[01:02] <wgrant> == All rights for ubuntu-security ==
[01:02] <wgrant> Archive Upload Rights for canonical-partner-dev: archive 'partner', component 'partner'
[01:02] <wgrant> There is no ArchivePermission granting them access.
[01:03] <lifeless> is this a defect or oversight
[01:03] <wgrant> Merely one of the stack of hacks that is security-in-soyuz.
[01:03] <lifeless> I mean, it seems to me that only folk that can upload should be able to unembargo
[01:03] <lifeless> otherwise there is a backdoor-to-uploading
[01:03] <wgrant> Sort of. ubuntu-security is only meant to be able to upload to -security.
[01:03] <wgrant> But that can't be done yet.
[01:04] <wgrant> And syncSource has to be extremely restricted.
[01:04] <wgrant> Because it is a hack.
[01:04] <wgrant> It is full of hacks.
[01:04] <wgrant> And it cannot be widely used yet.
[01:04] <lifeless> so, is this unembargoing untested?
[01:04] <wgrant> (it doesn't respect the queue, announcements, etc.)
[01:04] <wgrant> lifeless: Unembargoing is normal package copying by this special set of users that probably aren't represented in sampledata.
[01:05] <wgrant> But let's see.
[01:07] <wgrant> Headdesk.
[01:08] <wgrant> It only checks that launchpad.Append is sufficient to call syncSource (with somebody who already has upload privileges), and then checks that adding someone else to ubuntu-security gives them launchpad.Append.
[01:08] <lifeless> so the test is invalid
[01:09] <lifeless> because the control is incomplete
[01:09] <lifeless> ?
[01:09] <wgrant> Yes, and because security-in-soyuz is mostly P3As + a stack of several horrible special cases.
[01:10] <wgrant> (which will hopefully be eliminated as part of the second stage of DDs)
[01:10] <wgrant> (but not yet)
[01:12] <wgrant> Anyway, let me make myself a security person on dogfood and actually try it.
[01:19] <lifeless> http://www.lifehacker.com.au/2011/05/the-cognitive-cost-of-doing-things/
[01:27] <lifeless> flacoste: http://www.google.com/support/analyticshelp/bin/answer.py?hl=en&answer=1205784&topic=1120718
[01:30] <wgrant> virtinst 0.500.1-2ubuntu6.1 in lucid (Signer is not permitted to upload to the component 'main'.)
[01:30]  * wgrant rolls it back.
[01:32] <magcius> jam, did you take a look at the new CSS stuff I added?
[02:15] <wgrant> StevenK: Deleted delayed copies yet?
[02:15] <wgrant> All of the 70ish syncSource timeouts are from the unoptimised delayed copy path :(
[02:19] <spm> lifeless: wrt the page speed via GA, last I used that in anger it gave accurate percentages; but lousy actual numbers of X's doing Y. ie treat as strong guideline vs any sort of "X many Users are..."
[02:20] <lifeless> spm: thanks
[02:21] <StevenK> wgrant: Ah
[02:22] <wgrant> lifeless: Time to whitelist Distribution:+bugs and Distribution:EntryResource:searchTasks?
[02:22] <lifeless> wgrant: the poor queries are > 20 seconds
[02:22] <wgrant> lifeless: We've only had frequent complaints about these over the last month or so.
[02:22] <lifeless> yes
[02:23] <wgrant> lifeless: I suspect they used to work eventually.
[02:23] <wgrant> After the cache got reasonably warm.
[02:23] <wgrant> As it stands, common single-tag searches fail consistently.
[02:23] <lifeless> since we added the optimised index which is causing pathological tag lookups
[02:23] <lifeless> its not coldcache
[02:23] <wgrant> Ah. Wonderful.
[02:23] <lifeless> all of bugtags,bugs,bugtasks sit in hot cache
[02:24] <lifeless> either the vm cache or the pg shared region
[02:25] <lifeless> fun 1 /  121  RootObject:+login
[02:26] <lifeless> StevenK:  271 OperationalError: FATAL: Ident authentication failed for user "sync_packages" FATAL: Ident authentication failed for user "sync_packages"
[02:26] <lifeless> StevenK: do you need bugs filed about this? [there are two issues - the auth, and the OOPS prefix not being configured.
[02:27] <StevenK> lifeless: Sounds like a plan -- please assign them to Gavin, since he worked on it and it should be fresh in his mind.
[02:27] <wgrant> The first one needs a LOSA nudge, not a bug.
[02:27] <lifeless> both are config
[02:27] <wgrant> The second needs a trivial prod-config branch, which is just as fast as filing a bug..
[02:27] <lifeless> right
[02:27] <lifeless> been two days though
[02:27] <lifeless> allenap: ^
[02:28] <lifeless> well, > 2, but as top oops
[02:28] <StevenK> Sigh. TALES, please DIAF.
[02:29] <spm> it's problematic that ident faileds are getting into prod. these should be being added to the "funky rollout notes" at least. :-(
[02:29] <wgrant> lifeless: Hm, that RootObject:+login is a bit bd. The OpenID assoc takes 20s :/
[02:30] <StevenK> It raises LocationError, but it doesn't tell me which template causes it.
[02:31]  * StevenK works it out using the test
[02:32] <wgrant> That is definitely the biggest problem I have with TALES most of the time.
[02:32] <StevenK> lifeless: So, Gavin's work is based on much earlier work by jelmer that was written, but never used. I guess he assumed it was already in the prod configs, but I can't say for sure.
[02:33] <StevenK> spm: ^ you too
[02:49] <StevenK> Rargh, so the test and ZCML all point me to a view and template that don't make use of parent_series
[02:50] <wgrant> StevenK: Which view is erroring?
[02:50] <StevenK> (distroseries, '+index')
[02:50] <wgrant> It's not using SourcePackage.packaging at all?
[02:51] <mwhudson> coming from nowhere in particular, but why does launchpadlib use oauth signing if all requests are over https?
[02:53] <StevenK> wgrant: I don't think that's relevant, given the LocationError is (None, 'displayname')
[02:53] <mwhudson> oh hang on, it doesn't
[02:53] <wgrant> mwhudson: Doesn't it?
[02:54] <mwhudson>         oauth_request.sign_request(
[02:54] <mwhudson>             OAuthSignatureMethod_PLAINTEXT(),
[02:55] <mwhudson> my lplib might be hilariously out of date, mind
[03:03] <mwhudson> ok, it was, but it still seems OAuthSignatureMethod_PLAINTEXT is used
[03:03] <wgrant> Yeah.
[03:03]  * mwhudson tries to remember what oauthnonce is for then
[03:03] <wgrant> PLAINTEXT doesn't mean unsigned.
[03:04] <wgrant> I don't understand how it works, though.
[03:04] <mwhudson> http://tools.ietf.org/html/rfc5849#section-3.4.4 says It does not
[03:04] <mwhudson>    utilize the signature base string or the "oauth_timestamp" and
[03:04] <mwhudson>    "oauth_nonce" parameters.
[03:04] <wgrant> That indeed makes sense.
[03:05] <mwhudson> i'm not sure it's what lib/contrib/oauth.py does though
[03:05] <mwhudson> whee
[03:05] <wgrant> Yeah :/
[03:05] <lifeless> noone is
[03:09] <mwhudson> hooray
[03:09] <StevenK> wgrant: distroseries-index.pt was a red herring, the ZCML points out the page includes distroseries-portlet-derivation
[03:09] <wgrant> StevenK: Ah.
[03:19] <LPCIBot> Project windmill-devel build #27: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/27/
[03:24] <lifeless> stub: morning
[03:24] <stub> Indeed! It really is!
[03:24] <lifeless> stub: I've put https://bugs.launchpad.net/launchpad/+bug/534203 in the dba bugqueue
[03:24] <_mup_> Bug #534203: Timeouts on POFile:+filter (filter by person) <lp-translations> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/534203 >
[03:24] <lifeless> stub: yeah, congrats!
[03:26] <stub> Oh yay. Translations. The *other* place we have large amounts of asymmetric data yielding crappy plans.
[03:26] <lifeless> if translationmessage.potemplate was usable
[03:26] <lifeless> I think it would be fine
[03:28] <stub> So a lot of these queries have already been optimized at some point by jtv & danilo, trying to get certain cases to work. I don't know which queries they are though.
[03:28] <stub> But we should jtv in on translations queries when he is available.
[03:28] <lifeless> this bug was filed by jtv in fact
[03:28] <stub> c/should/should bring
[03:28] <lifeless> I wanted to ask him about the potemplate oclumn
[03:28] <lifeless> yes, I agree
[03:32] <stub> lifeless: About it being inconsistently populated?
[03:33]  * stub wonders if the data model changes have been completed, and all the unwanted columns now dropped.
[03:34] <lifeless> stub: yeah, and/or repopulating it and using it to make the query fast
[03:34] <stub> lifeless: We need a modern OOPS code in that bug btw. I think the linked ones are all from last year still.
[03:35] <lifeless> stub: hmm, I refreshed it yesterday :)
[03:35] <stub> ok... not obvious in the web ui that the description is fresh.
[03:35] <stub> oic
[03:45] <stub> Just looking at one of the example queries in the desc, but TranslationMessage.variant no longer exists.
[03:50] <lifeless> stub: yeah, I kept most of jtv's analysis because I wasn't sure how much was still relevant or not
[03:50] <lifeless> speak of the devil
[03:51] <jtv> uh-oh
[03:51] <StevenK> Haha
[03:51] <jtv> what is this about?
[03:51] <jtv> am I liable?
[03:51] <stub> lifeless: how long before we make decisions on moving the translations to Cassandra or not? Is that a 6 month or multiyear change do you think?
[03:51] <stub> jtv: Analyzing https://bugs.launchpad.net/launchpad/+bug/534203
[03:51] <_mup_> Bug #534203: Timeouts on POFile:+filter (filter by person) <lp-translations> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/534203 >
[03:51] <lifeless> stub: making a decision we could do now; scheduling the work ...
[03:52] <jtv> Ah
[03:52] <lifeless> stub: I have some thoughts on restructuring the app stack so that we can carve off parts of the system
[03:52] <lifeless> stub: I want to finish a current analysis I'm on, but then if you're up for the weekly call I can run it past you
[03:53] <stub> So, if it stays in PG, partitioning the data on language makes sense. But that is laborious and time consuming to set up, especially if we are avoiding large outages.
[03:54] <stub> And will make all-language queries slower, but I don't think we have any of them.
[03:55] <stub> Another alternative is language specific partial indexes.
[03:56] <stub> Probably tm__english__sumitter__idx ON TranslationMessage(submitter) WHERE language=1; for all hundred odd languages. We could combine too for lesser languages, but I suspect better just to create one index per language and put up with the noise.
[03:57] <jtv> Would it make sense to cluster on language?  That wasn't an option in the past, but is now.
[03:58] <stub> jtv: The queries I'm looking at, yes. But I don't have a comprehensive overview on the queries.
[03:58] <lifeless> jtv: what is translationmessage.potemplate
[03:58] <jtv> lifeless: that means "this message is not shared, but rather diverged for this specific template."
[03:59] <lifeless> jtv: so if its null many different templates may use the message?
[03:59] <jtv> Yes.
[03:59] <lifeless> jtv: and if its not null the list of templates is 1
[03:59] <jtv> It should be mostly null.
[03:59] <lifeless> jtv: is it going to be deleted?
[03:59] <jtv> No.
[03:59] <lifeless> ok
[04:00] <stub> The schema migration is all done now btw?
[04:00] <lifeless> so this page - is it showing /who/ contributed to the translation, or /what translations/ the person did?
[04:00] <stub> I think it was completed.
[04:00] <jtv> stub: you mean for translations message sharing?  We still have a few obsolete database columns but I don't think we use them now.
[04:00] <stub> jtv: Do you know if there are bugs open on dropping them?
[04:00] <jtv> lifeless: pofile:+filter shows a particular person's contributions.
[04:01] <jtv> stub: some, I think
[04:01] <LPCIBot> Project windmill-devel build #28: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/28/
[04:01] <lifeless> jtv: the tti.sequence component in the sort seems unuseful to me
[04:01] <lifeless> jtv: or do I misunderstand the model
[04:02] <jtv> lifeless: sequence expresses the order in which the translatable strings occur in the template.  Even to someone who doesn't know or care about the text of the template, it conveys context that may be useful.
[04:02] <jtv> I'm loading up the bug so I can be more specific.
[04:03] <stub> bwahaha... there is a blessed SQL 'hack' that makes that final pastbin work in 1% of the time
[04:04] <lifeless> jtv: so we're looking at a users contributions to the one project, is the sequence an ordinal in the file?
[04:04] <lifeless> jtv: if it is, the the date sort is useless
[04:04] <lifeless> jtv: if it isn't, then wouldn't most-recent-first be simpler?
[04:04] <jtv> lifeless: both sort criteria are needed.
[04:04] <lifeless> why?
[04:05] <jtv> A single user may have submitted multiple translations for the same message.
[04:05] <lifeless> and we show them all? even rejected/old/stale ones?
[04:05] <jtv> Yes.  With indication of status.
[04:05] <stub> compare https://pastebin.canonical.com/47090/ with https://pastebin.canonical.com/47136/ . The latter is identical except for the OFFSET 0 in the subquery, which is an unofficial query hint that lots of people use. Now I know why.
[04:05] <lifeless> grah
[04:05] <jtv> This is useful in evaluating a translator's work.
[04:06] <lifeless> jtv: it slowly degrades over time though
[04:06] <stub> c/query hint/planner hint
[04:06] <jtv> It's also useful for a translator to get insight into review status.
[04:06] <lifeless> stub: !!
[04:07] <lifeless> stub: that may be hot
[04:07] <stub> lifeless: it is. I get 20 s with the original query.
[04:07] <lifeless> stub: can you do that explain on the standalone replica ?
[04:07] <lifeless> which should be cold
[04:08] <stub> cold, 5ms
[04:08] <stub> This is silly
[04:08]  * stub double checks his query
[04:08] <lifeless> the 84756 lookups in the related table are the issue in the first place
[04:09] <stub> lifeless: standalone is being build. The queries are strangely fast when the tables exist and have 0 rows.
[04:09] <lifeless> hhhaaaa
[04:09] <lifeless> ok
[04:10] <jtv> I suspect the difference is mainly due to the planner selecting by language first and by user after, which the optimization barrier inverts.
[04:10] <jtv> I'd expect more selectivity from selecting by translator first and then by language.
[04:10] <lifeless> doing subselect vs join also did that
[04:11] <lifeless> plans are in the bug IIRC
[04:12] <lifeless> stub: on qastaging
[04:12] <lifeless>  explain select count(*) from bug where latest_patch_uploaded is not null;
[04:12] <lifeless>                             QUERY PLAN
[04:12] <lifeless> ------------------------------------------------------------------
[04:12] <lifeless>  Aggregate  (cost=644851.46..644851.47 rows=1 width=0)
[04:12] <lifeless>    ->  Seq Scan on bug  (cost=0.00..644777.55 rows=29562 width=0)
[04:12] <lifeless>          Filter: (latest_patch_uploaded IS NOT NULL)
[04:12] <lifeless>     "bug__latest_patch_uploaded__idx" btree (latest_patch_uploaded)
[04:13] <lifeless> stub: its indexed, and a small % of rows have that set:
[04:13] <lifeless>  select count(*) from bug where latest_patch_uploaded is not null;
[04:13] <lifeless>  count
[04:13] <lifeless> -------
[04:13] <lifeless>  29562
[04:13] <lifeless> stub: so why would it do a table scan ?
[04:14] <stub> small = 5-10%
[04:15] <lifeless> yeah, we have 800K bugs, 30K is < 5%
[04:16] <lifeless> with seqscan off:
[04:16] <lifeless>  Aggregate  (cost=10000644851.46..10000644851.47 rows=1 width=0)
[04:16] <lifeless>    ->  Seq Scan on bug  (cost=10000000000.00..10000644777.55 rows=29562 width=0)
[04:16] <lifeless>          Filter: (latest_patch_uploaded IS NOT NULL)
[04:16] <lifeless> it doesn't think the index is valid?
[04:16] <lifeless> stub: can you reproduce, and/or see if prod has the same issue?
[04:16] <stub> lifeless: possibly because the bug table is pretty much pinned in RAM
[04:17] <stub> but yeah, on prod seq scan for pulling 4% of the bugs
[04:18] <stub> lifeless: probably fixable by making it partial - WHERE latest_patch_uploaded IS NOT NULL
[04:19] <lifeless> explain select id from bug where latest_patch_uploaded is not null order by latest_patch_uploaded ; grabs the index
[04:19] <lifeless> stub: could you add a test index like that to qastaging ? I'm working on the +patches perf bug
[04:19] <stub> lifeless: And does explain analyze show it takes longer real time?
[04:19] <lifeless> checking
[04:21] <stub> Building the index atm, so your checks will suck.
[04:21] <lifeless> http://pastebin.com/FBRt1HPr
[04:22] <lifeless> its using your new index :>
[04:22] <lifeless> can you run those two to compare on prod ?
[04:23] <stub> So that is quite interesting. See that PG correctly picked the plan that starts returning rows soonest? The 'order by' query runs slightly faster to completion, bug doesn't start returning rows for 240ms
[04:24] <stub> which is rather stupid when this is a COUNT()
[04:24] <stub> But the estimated time for the second query is higher than the first, so understandable. And the differences probably statistically insignificant.
[04:25] <lifeless> stub: right, I couldn't compare counts to counts, so changed it to get th eid
[04:25] <lifeless>                ->  BitmapAnd  (cost=43791.26..43791.26 rows=23966 width=0) (actual time=16801.315..16801.315 rows=0 loops=1)
[04:25] <lifeless>                      ->  Bitmap Index Scan on bug__latest_patch_uploaded__idx2  (cost=0.00..397.06 rows=29562 width=0) (actual time=15.446..15.446 rows=29562 loops=1)
[04:26] <lifeless>                      ->  Bitmap Index Scan on bug_duplicateof_idx  (cost=0.00..43382.43 rows=577826 width=0) (actual time=16773.721..16773.721 rows=578095 loops=1)
[04:26] <stub> partial index built and table analyzed btw. Look for '_idx2' in your plans
[04:26] <lifeless> yeah
[04:30] <lifeless> stub: could you add a similar one for duplicateof ?
[04:30] <lifeless> (where duplicateof is null)
[04:30] <lifeless> I don't know that that will be sufficient
[04:30] <lifeless> or...
[04:31] <lifeless> perhaps index id where duplicateof is null and latest_patch_uploaded is not null
[04:32] <lifeless> I wonder if we need to denorm the openness or not of a bug into the bug
[04:32] <lifeless> this is the bug I'm looking at https://bugs.launchpad.net/launchpad/+bug/618392
[04:32] <_mup_> Bug #618392: Distribution:+patches slow 10% of requests timing out <lp-bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618392 >
[04:35] <stub> We have some indexes elsewhere on id WHERE [partial clause] to speed up joins and ordering - may well work here too.
[04:35] <stub> duplicateof index built and table analyzed
[04:37] <lifeless> nope, unused
[04:37] <stub> and the id one
[04:38] <lifeless> ah, with just duplicates by id
[04:38] <lifeless> this is what its doing:
[04:38] <lifeless>                ->  BitmapAnd  (cost=43791.26..43791.26 rows=23966 width=0) (actual time=20175.537..20175.537 rows=0 loops=1)
[04:38] <lifeless>                      ->  Bitmap Index Scan on bug__latest_patch_uploaded__idx2  (cost=0.00..397.06 rows=29562 width=0) (actual time=16.256..16.256 rows=29562 loops=1)
[04:39] <lifeless>                      ->  Bitmap Index Scan on bug_duplicateof_idx  (cost=0.00..43382.43 rows=577826 width=0) (actual time=20146.004..20146.004 rows=578095 loops=1)
[04:39] <lifeless>                            Index Cond: (duplicateof IS NULL)
[04:39] <lifeless> the expensive  bit is pulling all the bits out from the nondupes
[04:39] <lifeless> could you drop all these indices
[04:39] <lifeless> and try
[04:39] <lifeless> (id) where duplicateof is null and latest_patch_uploaded is not null
[04:40] <lifeless> OTOH this may be index bloat I guess
[04:41] <stub> whoops... I'd created duplicateof index IS NOT NULL, not IS NULL
[04:41] <lifeless> :>
[04:41] <lifeless> ok, so lets try fixing that first
[04:41] <lifeless> general purpose things being better, all other stuff equal
[04:43] <lifeless> booyah
[04:43] <lifeless> thats fast
[04:43] <stub> You now have two indexes - on id - WHERE duplicateof IS NULL , and WHERE duplicateof IS NULL AND latest_patch_uploaded IS NOT NULL
[04:43] <stub> which one gets used?
[04:44] <lifeless>          ->  Index Scan using bug__new_patches__idx on bug  (cost=0.00..1013500.47 rows=23045 width=4) (actual time=0.086..93.803 rows=28607 loops=1)
[04:44] <lifeless> 500ms for the batch, 300ms for the count
[04:45] <lifeless> do you think this is reasonable to add live ?
[04:45] <stub> sure
[04:45] <lifeless> \o/
[04:45] <lifeless> -65-1.sql ?
[04:45] <stub> first, I'll drop the more complex index. Lets see if it is nearly as fast with just the id WHERE duplicateof IS NULL
[04:45] <lifeless> ok
[04:46] <stub> That index will be useful for other stuff too
[04:46] <lifeless> yeah
[04:46] <stub> lifeless: ok. try now.
[04:46] <stub> yeah - 65-1
[04:46] <lifeless> Time: 3089.849 ms
[04:47] <lifeless> faster than it was without it I think,
[04:47] <lifeless>  Aggregate  (cost=7085778.93..7085778.94 rows=1 width=0) (actual time=2905.470..2905.471 rows=1 loops=1)
[04:47] <stub> ok. crappy time. Wonder if I should create both indexes or just the fast one for this case?
[04:47] <stub> or is the time close?
[04:47] <lifeless> 2.5 second difference
[04:48] <stub> right. crappy time :)
[04:48] <lifeless> 500ms vs 3000ms
[04:48] <lifeless> I suggest just the tuned one
[04:48] <lifeless> less chance of destablising some other query
[04:49] <stub> Should be impossible to destabalize - if it is possible to use the partial index, it will be faster because it is smaller. Assuming no memory pressure on the indexes because index size is insignificant compared to total RAM.
[04:49] <stub> But from change management pov, yeah.
[04:49] <LPCIBot> Project windmill-db-devel build #235: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/235/
[04:50] <lifeless> stub: https://bugs.launchpad.net/launchpad/+bug/735977 appears to be a destablisation result to me
[04:50] <_mup_> Bug #735977: MaloneApplication:+bugs timeouts searching for tags across all bugs <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/735977 >
[04:50] <lifeless> stub: so my paranoia is heightened
[04:50] <stub> I'll do that now and repack bug_fti while I'm at it.
[04:50] <lifeless> \o/
[04:50] <lifeless> paste me the sql
[04:50] <lifeless> and I'll make it into a branch
[04:51] <stub> I'll do the -1 patch if I may - I do the patch first then translate it to live statements to run on the slaves.
[04:52] <stub> less chance of screwups
[04:53] <lifeless> I'm always happy for someone else to do work
[04:53] <lifeless> once you've done this lets do the catchup call
[04:53] <LPCIBot> Project devel build #689: FAILURE in 5 hr 7 min: https://lpci.wedontsleep.org/job/devel/689/
[04:58] <stub> I should flag this patch as fixing Bug #618392 yes?
[04:58] <_mup_> Bug #618392: Distribution:+patches slow 10% of requests timing out <lp-bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618392 >
[04:58] <lifeless> yes
[05:07] <stub> prod_1 done
[05:08] <lifeless> 361 queries/external actions issued in 2.65 seconds
[05:08] <lifeless> \o/
[05:08] <stub> prod_2 complete
[05:09] <lifeless> wiiin
[05:09] <stub> prod_3 taking its time
[05:09]  * stub checks for long running transactions - probably the backups at this time
[05:11] <stub> yer... prod_3 blocked until the backups stop or I kill 'em
[05:11] <lifeless> thats master right ?
[05:11] <stub> yes
[05:12] <lifeless> up to you
[05:12] <lifeless> are the backups offsite?
[05:12] <spm> s/u/a/ <== stub
[05:19] <stub> spm: eh?
[05:20] <stub> The reapers are killing archivepublisher jobs for about a day now.
[05:20] <spm> stub "... or I kill 'em" ergo stub ==> stab. s/u/a/ :-)
[05:21] <stub> lifeless: I think we can wait another 3 hours until they complete.
[05:22] <lifeless> sure
[05:22] <lifeless> so, call?
[05:22] <stub> lifeless: sure
[05:22]  * stub kills his u1 sync
[05:24] <stub> So I installed wonderwall as a workaround for some throttling issues, and it seems to be helping my system a lot. Lets see if this call is stable - Skypeout was crappy yesterday before that install.
[05:27] <lifeless> I'm here
[05:34] <lifeless> stub: I'm here when your network comes back :>
[05:34] <lifeless> stub: I can hear you mostly fine
[05:34] <stub> Sre. I'm watching the network monitor. I'm seeing a sine wave
[05:34] <stub> Like there is some sort of really crappy traffic shaping going on, or I'm only getting
[05:34] <lifeless> of traffic or signal ?
[05:47] <lifeless> stub: brewaking up?
[05:47] <lifeless> stub: hows the graph?
[05:47] <stub> still broken
[05:47] <stub> think it is isp conjestion
[05:54] <LPCIBot> Project windmill-devel build #29: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/29/
[06:08] <lifeless> sine wave?
[07:12] <lifeless> stub: another 'fun' one for you - https://bugs.launchpad.net/launchpad/+bug/777601
[07:12] <_mup_> Bug #777601: BugComment:EntryResource timeouts <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/777601 >
[07:24] <stub> lifeless: so that was qa staging yesterday. Was something happening there at 11:18:48 UTC like a database update?
[07:44] <lifeless> stub: huh, that was prod
[07:45] <stub> really?
[07:46] <lifeless> OOPS-1950DQ231
[07:46] <lifeless> DQ
[07:46] <stub> oic. 'last seen' was qastaging
[07:46] <lifeless> so (IIRC) wampee
[07:46] <stub> I saw QASTAGING in big type :)
[07:47] <lifeless> :>
[07:48] <stub> I'm not aware of any activity that would explain that. I'll need to dive into the logs.
[07:49] <stub> But first a shower. I'm all sticky.
[07:49] <StevenK> TMI, stub
[09:02] <allenap> lifeless: Okay, I'll follow that up (the sync_packages thing). Nothing is creating jobs that use the sync_packages user yet so it's good someone saw this now before we do start creating jobs. Thanks.
[09:03] <bigjools> morning dudes
[09:05] <mrevell> hello!
[09:07] <StevenK> allenap: O hai! Can you look at http://pastebin.ubuntu.com/603553/ ? It's the last bit for the mega-branch you've been reviewing.
[09:07] <StevenK> allenap: This cleans up the UI tests to work
[09:09] <allenap> StevenK: Sure.
[09:20] <bigjools> wgrant: did you see the MP I assigned you?
[09:24] <allenap> StevenK: _create_child_and_parent() is defined several times. If they're all the same perhaps you could factor them out?
[09:24] <allenap> StevenK: Otherwise it looks good, r=me.
[09:25] <StevenK> allenap: Thanks!
[09:25] <StevenK> allenap: I was not amused at all of the duplication of derived_series = ...
[09:25] <allenap> No :)
[09:32] <LPCIBot> Project windmill-devel build #30: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/30/
[09:45] <wgrant> bigjools: Ah, sorry missed it in the middle of lots of other review emails that I didn't care about.
[09:45] <wgrant> Looking now.
[09:45] <bigjools> wgrant: heh, I unsubscribed from the review list a while ago
[09:46] <wgrant> bigjools: Hmm, that simplifies the code a little bit, doesn't it...
[09:46] <bigjools> a tad
[09:47] <bigjools> dunno what that guadalinux stuff was in there for
[09:47] <wgrant> For DDs with binary copies, without considering at all how it would work.
[09:48] <wgrant> StevenK/anyone: https://code.launchpad.net/~julian-edwards/launchpad/slow-build-hunt-bug-777183/+merge/59968 would like to be mentored.
[09:48] <lifeless> allenap: cool thanks
[09:50] <adeuring> good morning
[10:04]  * stub wonders why launchpadlib is pulling credentials from kwallet when he is using Unity
[10:04] <bigjools> the irony, I am using KDE and it pulls them from gnome-keyring
[10:04] <LPCIBot> Project devel build #690: STILL FAILING in 5 hr 11 min: https://lpci.wedontsleep.org/job/devel/690/
[10:05] <bigjools> lifeless: around?
[10:05] <stub> I wonder if it remembers somewhere where you first used it?
[10:05] <bigjools> IIRC it depends on the ordering of importing dependent modules, whichever works first it uses
[10:06] <bigjools> you can make a .keyringrc as well
[10:06] <stub> I'm sure there must be an environment variable to sniff for a heuristic.
[10:06] <bigjools> just the config I think
[10:07] <bigjools> StevenK: still there?
[10:08] <bigjools> ok, allenap?
[10:09] <lifeless> bigjools: vaguely
[10:09]  * stub wonders what a .keyringrc file looks like
[10:10] <bigjools> lifeless: if you have time, can you mentor wgrant's review of my branch please?
[10:10] <bigjools> https://code.launchpad.net/~julian-edwards/launchpad/slow-build-hunt-bug-777183/+merge/59968
[10:10] <bigjools> 81 lines, mostly removals
[10:16] <bigjools> stub: the file's actually called keyringrc.cfg
[10:17] <bigjools> you can use it to override the backend
[10:17] <bigjools> I think it's supposed to live in ~ which is somewhat annoying
[11:03] <bigjools> allenap: ooo I had no idea PG had a notification mechanism
[11:03] <bigjools> stub: apparently you didn't like it though
[11:05] <stub> bigjools: Unity? I worked out how to turn off the scrollbars that were driving me batty when they popped up and am now starting to appreciate it. KDE was nice, particularly plasmoids built in and working and some decent ones shipped, but still a little busy for me or something.
[11:06] <bigjools> stub: I was talking about http://www.postgresql.org/docs/8.4/interactive/sql-notify.html
[11:06] <bigjools> :)
[11:06] <stub> oh... notification. Its fine. The only reason I don't like it is we really shouldn't be spending time reimplementing rabbitmq badly.
[11:06] <bigjools> maybe
[11:07] <bigjools> but considering we already did that anyway ...
[11:07] <bigjools> Job *cough*
[11:07] <stub> bigjools: I can't quite recall if it works outside transaction boundaries - if LISTEN is stuck inside a transaction, we could be shooting ourselves in the foot with long running transactions.
[11:08] <bigjools> I was thinking more of doing it in a twistd process, we have a bit more control over transactions
[11:08] <stub> Because if we do something wrong once, we should keep making the same mistake until everyone believes it is the right thing to do.
[11:08] <bigjools> heh
[11:08] <stub> Or am I thinking of politics?
[11:21] <lifeless> if you want queues
[11:21] <lifeless> land my rabbitmq branch
[11:21] <lifeless> :)
[11:26] <bigjools> lifeless: well we keep adding polling jobs everywhere, it's about time ...
[11:27] <lifeless> I have a branch, tests all pass locally, something in ec2 hates it and I haven't had time to debug
[11:30] <rvba> wgrant: bigjools: Just pushed the modifications to the branch William marked as bad earlier today. Could you have a look? (http://bazaar.launchpad.net/~rvb/launchpad/change-perm-sync/revision/12950)
[11:31] <rvba> I changed the MP's status back to WIP but the partial diff does not get updated.
[11:32] <bigjools> rvba: you need a new branch + MP
[11:33] <rvba> ok
[11:33] <bigjools> rvba: I would add a comment explaining why lp.append is needed for the security updates
[11:33] <LPCIBot> Project windmill-db-devel build #236: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/236/
[11:34] <bigjools> it's a bit of a hack unfortunately
[11:34] <wgrant> bigjools, rvba: I think we will be able to remove the hack this year, but until then it needs to stay (and probably have a test)
[11:35] <bigjools> yeah he has a test
[11:35] <bigjools> rvba: I would move that test somewhere else though, let me think
[11:35] <wgrant> s-i-s seems to be basically P3As + a stack of hacks that all need to be genericised for DDs to work properly.
[11:36] <lifeless> 'special cases'
[11:37] <bigjools> rvba: lib/lp/soyuz/scripts/tests/test_copypackage.py
[11:37] <wgrant> lifeless: Have you *seen* them?
[11:37] <lifeless> some
[11:37] <rvba> bigjools: ok
[11:37] <lifeless> just noting that they may me lacking the 'elegant solution to a non trivial problem' aspect of 'hack'
[11:38] <bigjools> rvba: I'd be tempted to also move the syncing tests out of test_distroseries
[11:38] <bigjools> that's not the first place I'd look for them
[11:38] <bigjools> but the waters are already muddied by this feature being half in lp.registry and half in lp.soyuz
[11:39] <wgrant> It should be entirely in lp.soyuz, or the trees should be merged :/
[11:39] <wgrant> Why was it in lp.registry to start with?
[11:39] <bigjools> wgrant: noodles put the DSD stuff there
[11:39] <wgrant> Ah, right.
[11:39] <wgrant> That seems so long ago now :/
[11:39] <lifeless> wgrant: are you around tomorrow?
[11:39] <bigjools> about a year :)
[11:39] <wgrant> lifeless: Unless they've instantiated a new public holiday while I wasn't looking.
[11:40] <bigjools> that happened here!
[11:40] <stub> We only get that on military coups.
[11:40] <bigjools> The Royal Wedding could be considered a military coup
[11:41] <lifeless> wgrant: I'd like to run some concepts I'm tossing round - starting to socialise - past you; I'd use bigjools but ETIMEZONE
[11:41] <rvba> bigjools: right, I'll leave one to check for UI notifications and see if some of the other tests can be moved to test_copypackages.
[11:41] <lifeless> wgrant: so skype/mumble at some convenient point of th day for you
[11:41] <bigjools> rvba: thanks
[11:41] <bigjools> lifeless: I can be around later today my time if you want
[11:42] <lifeless> bigjools: I'm doing big picture sketching and looking to understand more of the bits of soyuz that glue together
[11:43] <bigjools> lifeless: you definitely want me then :)
[11:43] <wgrant> lifeless: Sure, whenever.
[11:43] <lifeless> bigjools: I'd be delighted, but I plan to sleep till i wake tomorrow, after the 5:45 start this morning
[11:43] <lifeless> bigjools: I don't want to interrupt your evening unnecessarily
[11:43] <wgrant> Sleep till you wake.
[11:43] <wgrant> What a novel concept.
[11:44] <rvba> bigjools: btw, I'm not sure I know why exactly "lp.append is needed for the security updates" ... it's a hack all right ... but why is it there?
[11:44] <bigjools> lifeless: nae prob
[11:44] <lifeless> wgrant: theres a book on it
[11:44] <bigjools> rvba: it was the only reasonable way for the security team to get permission to copy to -security at the time
[11:44] <lifeless> bigjools: if you are around in my morning and want to chat, I'd be delighted; from experience (this will be the 4th/5th time I've run it up the flagpole) its about a 45-60 minute topic
[11:44] <bigjools> heh
[11:45] <lifeless> bigjools: failing that, we can make a time next week if you like
[11:45] <bigjools> I'll pop in to irc tonight
[11:46] <wgrant> rvba: Basically, the security team needs to be able to copy any package into the Security pocket, even if they lack upload privileges. And *only* the security team can be permitted to use syncSource into the primary archive, since it bypasses the queue and announcements.
[11:46] <lifeless> bigjools: to give you context, with performance a Solved Problem, I'm starting to look longer term at our architecture
[11:46] <lifeless> bigjools: at how we evolve the system, what platform we want to be on, and how we can break out of the monolithic situation we're in safely
[11:46] <rvba> wgrant: bigjools ok. Thanks, I'll had a comment then.
[11:46] <bigjools> lifeless: excellent, I have a lot of ideas and semi-plans about that
[11:46] <wgrant> lifeless: I am relieved to hear that you are looking at this.
[11:46] <rvba> s/had/add/
[11:47] <wgrant> rvba: Thanks!
[11:47] <bigjools> a lot of our architecture was designed around more machines being in the pipeline
[11:47] <bigjools> rvba: thank you
[11:47] <lifeless> bigjools: its nearly 11pm and I've worked straight through except for dinner with Lynne; I guarantee 0% retention of anything you say now :>
[11:48] <wgrant> rvba: We'll want to redo what you did in a few months, I suspect. But we can't get ahead of ourselves :(
[11:48] <bigjools> lifeless: I hear you
[11:48] <bigjools> wgrant: baby steps
[11:48] <lifeless> anyhow, I'll be active here again in 9ish hours.
[11:48] <lifeless> night all
[11:48] <bigjools> nn!
[11:48] <wgrant> Night lifeless.
[11:49] <wgrant> lifeless: I'd like to be around for this specialised deploy, so please don't try it before I'm around.
[12:01] <bigjools> wgrant: just sent you email so you can sanity check the steps I think are necessary for syncs via the queue
[12:04] <wgrant> bigjools: Looks sane.
[12:04] <wgrant> bigjools: We'll need to shuffle the upload policy stuff around a bit... but it should be pretty much OK.
[12:04] <wgrant> I think +queue and the queue tool should mostly work, as long as PackageUpload.displayname works.
[12:04] <bigjools> yeah, I want to do the policy stuff in such a way that it's easy to convert for generic suites
[12:04] <stub> Speaking of holidays, apparently it is coronation day and I'm not supposed to be here. I guess I'll be slacker than usual tomorrow.
[12:05] <wgrant> And acceptFromQueue does the right thing,
[12:05] <wgrant> But if not, easy enough to fix.
[12:05] <bigjools> stub: how will we notice?  /me hides
[12:06] <bigjools> wgrant: right, the change will be in acceptFromQueue
[12:06] <bigjools> although I expect we'll need to amend both for display purposes so it says it's a sync
[12:07] <wgrant> Yeah.
[12:07] <wgrant> but it's not going to be as hard as I have long feared.
[12:07] <wgrant> Mostly because of that convenient job.
[12:14] <wgrant> StevenK: Did you get anywhere with the overrides?
[12:15] <bigjools> 20 bugs away from #777777
[12:19] <wgrant> bigjools: Have you had any more thoughts about how to arrange the generic copier underpinnings?
[12:20] <bigjools> no
[12:20] <bigjools> too busy thinking about this project :)
[12:20] <wgrant> Yeah.
[12:20] <bigjools> now, where's the damn table for copy jobs
[12:21] <wgrant> Is it DistributionJob or ArchiveJob or similar?
[12:21] <wgrant> lib/lp/soyuz/model/packagecopyjob.py:class PackageCopyJob(DistributionJobDerived):
[12:22] <bigjools> Bug #776283
[12:22] <_mup_> Bug #776283: PackageCopyJob needs separate archive and series columns for query purposes <derivation> <Launchpad itself:Triaged> < https://launchpad.net/bugs/776283 >
[12:22] <bigjools> it uses DistributionJob
[12:45] <wgrant> gmb: Well done!
[12:45] <gmb> wgrant: Yeah. Trigger happy, premature filing.
[12:46] <gmb> The failure state of "clever" is "arsehole"
[12:46] <wgrant> Yeah.
[12:46] <wgrant> Still, 4 bugs to go!
[12:46] <gmb> I'll leave it to mpt.
[12:46] <gmb> And stick to being doggedly successful instead of smart.
[12:49] <wgrant> That sounds like a plan.
[12:49] <wgrant> Any Yellow people want to handle the bug in #launchpad?
[12:49] <wgrant> YUI doesn't seem to like + in an id selector.
[12:55] <wgrant> bigjools: Are you going to separate the two different facets of upload checking?
[12:56] <wgrant> Some of the checks are uploader policy (eg. signature checks), while others are distroseries acceptance policy.
[12:56] <wgrant> They are currently conflated, but that doesn't really work.
[13:21] <wgrant> Is unmuting a bug meant to give me the option to unsubscribe the team?
[13:21] <wgrant> That seems sort of the opposite action...
[13:33] <bigjools> wgrant: yes
[13:34] <bigjools> uploader policy is fixed for the most part
[13:40] <wgrant> bigjools: Right. Or it's controlled by an option on process-upload.
[13:40] <wgrant> Not series-specific.
[13:40] <bigjools> exactly
[13:40] <wgrant> So it must be split.
[13:40] <bigjools> that's the plan
[13:41] <wgrant> Excellent.
[13:41] <bigjools> anyone know if you can change the billing date for AWS?
[13:41] <wgrant> You mean so it appears more than a day before expenses are due?
[13:41] <wgrant> That would be lovely.
[13:41] <wgrant> But I don't think so.
[13:41] <bigjools> they bill just at the exact wrong time for me
[13:46] <henninge> danilos: could you comment on bug 196679, though?
[13:46] <_mup_> Bug #196679: ValueError on +language-packs <lp-translations> <oops> <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/196679 >
[13:47] <henninge> danilos: I mean, do you have a hint for me if this still needs to be done?
[13:47] <danilos> henninge, I thought jtv and you got the hang of it (I remember seeing his comment about how it's still there somewhere)
[13:47] <henninge> danilos: in a mail but it sounded much like guess work on his part.
[13:47] <henninge> Just wondering if you had more insight.
[13:48] <danilos> henninge, let me take a look
[13:49] <wgrant> bigjools: Have you seen the repeated archivepublisher transaction killings over the past couple of days?
[13:49] <LPCIBot> Project windmill-db-devel build #237: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/237/
[13:49] <bigjools> wgrant: no, I'm not paying any attention to oopses
[13:50] <bigjools> and, BTW, ****ing keyboard drivers
[13:50] <wgrant> bigjools: Not OOPSes, but pgkillidle spam. The first was on the 3rd, and there have been ~8 since.
[13:50] <wgrant> I can't find anything in the logs.
[13:50] <wgrant> Don't even know which host it's from.
[13:50] <bigjools> wgrant: I suspect it's to do with the timeout getting lowered
[13:51] <bigjools> if that happened yet
[13:51] <wgrant> Extremely unlikely.
[13:51] <wgrant> These transactions had been idle for half an hour.
[13:51] <bigjools> that it happened?
[13:51] <wgrant> It's been 9s for a week now.
[13:51] <bigjools> not that timeout
[13:51] <bigjools> there's a script timeout IIRC
[13:51] <wgrant> Hmm.
[13:51] <danilos> henninge, it seems it should be able to reproduce the bug with more than 15 language packs (shortlist default), and the base pack not being in the list that shortlist returns, as jtv noted as well
[13:51] <bigjools> I vaguely remember stub saying something about lowering it and panicking
[13:51] <bigjools> and then forgot about it
[13:52] <wgrant> Heh
[13:52] <henninge> danilos: I see, I'll try that. thanks
[13:53] <bigjools> wgrant: so I'd check with stub to see if he changed anything
[13:54] <stub> eh?
[13:54] <stub> wgrant: I'll grab the hosts for you - they should be in the pg logs
[13:54] <wgrant> stub: Oh, that would be great, thanks.
[13:54] <danilos> henninge, I think the right fix would be to use a larger shortlist value there (find what's the most we had for a distro release)
[13:55] <wgrant> stub: Then we can hopefully identify the process from the ps logs.
[13:58] <jcsackett>           https://dev.launchpad.net/ | On call reviewer: allenap |
[13:58] <allenap> Oops.
[13:58] <allenap> :)
[13:58] <jcsackett> crap, i didn't even hit enter.
[14:00] <deryck> Morning, all.
[14:01] <deryck> henninge, ping for standup
[14:03] <bigjools> wgrant, I am going to start the mawson DB upgrade later today, you don't need it do you?
[14:04] <LPCIBot> Project windmill-devel build #31: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/31/
[14:05] <wgrant> bigjools: I need to do some minor QA on it tomorrow.
[14:05] <wgrant> bigjools: So probably best to run it over the weekend...
[14:05] <wgrant> Or I can start it when I'm done tomorrow.
[14:05] <wgrant> That's probably best.
[14:06] <bigjools> wgrant: what QA?
[14:06] <wgrant> r12979
[14:06] <bigjools> I need DF *done* before end of Sunday so I can have it ready for monday
[14:06] <wgrant> buildd-manager path change.
[14:06] <wgrant> Ah, true.
[14:06] <bigjools> you can QA that on staging
[14:06] <wgrant> I guess we can do this on staging.
[14:06] <wgrant> Yeah.
[14:06] <wgrant> OK, blow it away!
[14:06] <bigjools> cheers :)
[14:07] <wgrant> I guess we also need to clean the librarian out, but can leave most of the archive.
[14:07] <bigjools> librarian purging is part of my monkey-following script
[14:07] <wgrant> Great.
[14:11] <bigjools> wgrant: for this policy checker I am thinking to sort of copy the uploadpolicy one but makeit  a) use DB objects, b) only have distro policy checks.  Then add a PackageUpload.checkPolicy(policy) or something.
[14:12] <wgrant> bigjools: Can't it basically be encapsulated in DistroSeries or Archive?
[14:12] <wgrant> There's only one or two methods that need to be exposed.
[14:12] <bigjools> no
[14:12] <bigjools> well
[14:12] <bigjools> I'd rather it was a totally separate module
[14:12] <bigjools> then it can be re-used in more ways
[14:13] <wgrant> Do we need to replace anything other than autoApprove and autoApproveNew?
[14:14] <bigjools> at the moment, no, but I expect it to grow
[14:14] <wgrant> Right.
[14:14] <bigjools> rejectPPAUploads, for example
[14:14] <bigjools> maybe announcelist
[14:14] <wgrant> Most of that is archiveuploader-specific.
[14:14] <wgrant> Except for announcelist, which is just wrong.
[14:15] <bigjools> it should be distro-specific
[14:15] <bigjools> s/it/they/
[14:20] <bigjools> now,  the hardest part has just arrived.  What do I call it and where should it live :)
[14:20] <wgrant> It is similar to archivedependencies, so it might belong alongside it in adapters/
[14:21] <bigjools> adapters is a horrible name but there's nowhere else better that I can see
[14:22] <bigjools> I mean, packagecopier is in scripts/  ffs
[14:24] <wgrant> Hmm, I argued with you about packagecopier a while ago and you said it belonged in scripts/ :)
[14:26] <bigjools> did I? :)
[14:26] <deryck> henninge, we don't hear you any more
[14:26] <henninge> deryck: I still hear you
[14:53] <bigjools> wgrant: bugger, test failure from the getBuildByArch change
[14:54] <wgrant> bigjools: Ew, which?
[14:59] <bigjools> publishing.txt and testDuplicatedBinaryUploadGetsRejected
[15:03] <wgrant> Hmm. The second one is a braindead test.
[15:03] <wgrant> Copying to the same distro and archive and expecting a new build to be created.
[15:03] <wgrant> Nonsensical.
[15:04] <bigjools> not look at it yet, just going in now
[15:04] <wgrant> publishing.txt seems to be the same.
[15:05] <benji> abentley: will you be able to QA bug 747844 today?  I have a revision behind it that I'd like to get released.
[15:05] <_mup_> Bug #747844: process-mail oopses with TypeError due to non US-ASCII chars in the email address <oops> <process-mail> <qa-needstesting> <Launchpad itself:Fix Committed by abentley> < https://launchpad.net/bugs/747844 >
[15:05] <wgrant> It copies from one series in a PPA to another.
[15:05] <wgrant> ANd expects builds to be created.
[15:05] <bigjools> wgrant: huh, the description is insane
[15:05] <abentley> benji: yes.  I am QAing it right now.
[15:05] <bigjools> "source gets copied to another suite in the same archive..... The binary rebuild"
[15:05] <bigjools> !
[15:05] <benji> abentley: great! thanks
[15:06] <bigjools> wgrant: that's completely bong, wtf
[15:06] <wgrant> bigjools: Yeah, so it asserts that the duplicated build will fail to upload.
[15:06] <wgrant> bigjools: It's a good check.
[15:06] <wgrant> But it relies on a bug: that the build is created at all.
[15:06] <wgrant> Need to create the build manually.
[15:06] <benji> rvba: will you be able to QA bug 775529 today?  I have a revision behind it that I'd like to get released.
[15:06] <_mup_> Bug #775529: Synching to a released series should put packages in -updates. <derivation> <qa-needstesting> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/775529 >
[15:06] <wgrant> And publishing.txt needs to be reworded to use a different archive.
[15:06] <wgrant> bigjools: DF is still usable?
[15:06] <wgrant> I'll QA that thing now.
[15:07] <bigjools> wgrant: yes, still waiting for the dump
[15:07] <rvba> benji: I would need DF to be up for that ... bigjools?
[15:07] <rvba> bigjools: can you update DF?
[15:08] <bigjools> it's up, but we can defer QA since it won't block rollout
[15:08] <bigjools> mark it untestable for now
[15:08] <rvba> benji: bigjools will do.
[15:08] <benji> awesome, thanks
[15:08] <wgrant> benji: You need to deploy at least 12980 due to a bad rev, and 12979 has an unusual rollout requirement on LPS -- could you make sure that's done if you request one before I'm around again?
[15:10] <benji> wgrant: I'm doing a lazy release.  I'm just lining things up so my revision will be ready when you do your deploy.
[15:10] <wgrant> benji: Ah, OK. That makes things easier.
[15:11] <benji> for the both of us ;)
[15:13] <LPCIBot> Yippie, build fixed!
[15:13] <LPCIBot> Project devel build #691: FIXED in 5 hr 8 min: https://lpci.wedontsleep.org/job/devel/691/
[15:16] <abentley> losa: please run process-mail for qastaging
[15:17] <hloeung> abentley: hi, give me a sec...
[15:17] <abentley> hloeung: thanks.
[15:19] <wgrant> Ah, I was wondering why DF was so slow.
[15:20] <wgrant> Driven into iowait by a network file copy. Yay.
[15:21] <hloeung> abentley: done... see https://pastebin.canonical.com/47159/
[15:22] <wgrant> bigjools: I'm giving up for now, will do it on staging tomorrow. Do with mawson what you will.
[15:23] <abentley> hloeung: Hmm.  That doesn't look right, and it hasn't generated the outgoing mail I expected, either.
[15:23] <abentley> hloeung: that was definitely qastaging, not staging?
[15:23] <hloeung> abentley: yep, qastaging
[15:25] <bigjools> wgrant: ok
[15:26] <abentley> hloeung: could you try running it on staging, please?
[15:27] <hloeung> abentley: process-mail runs every 10 minutes on both staging and qastaging
[15:27] <abentley> hloeung: Oh.  I was under the impression that no scripts run on qastaging.
[15:30] <abentley> hloeung: I sent mail to 5977@bugs.staging.launchpad.net more than 20 minutes ago, and it still hasn't appeared in the staging mail folder.
[15:38] <bigjools> wgrant: publishing.txt is also on crack
[15:38] <bigjools> it's copying intra-archive and expecting rebuilds
[15:43] <jtv> henninge, mrevell: one very small note about the sharing announcement: where it says "That sharing also works between different versions of a project or package:" it may be helpful to say "already" instead of "also" (to set the right expectation, i.e. that this is not the part that has changed most recently).
[15:44] <mrevell> jtv, Thanks, that's a good suggestion.
[15:45] <jtv> Also, "series" is one of those things that are obvious to us devs but I'm worried about how clear they are to others.  I'd prefer to say "release series" a lot of the time to avoid leaving those "not in the know" behind.
[15:46] <jtv> mrevell, in case you were already off typing: ^  :)
[15:47] <mrevell> ah, thanks jtv, I did wonder about that but then I kinda thought that we don't have much of a better alternative.
[15:47] <jtv> Release series.
[15:48] <jtv> When we talk internally, that's needlessly verbose.  But every time we omit it, a potential user may go "what are these people talking about?  All I want is to host my project!"
[15:49] <jtv> And consequently, strangle a kitten.
[15:50] <jtv> mrevell: also, the the dreaded repeated word across line line breaks: "opt"
[15:51] <mrevell> ah
[15:53] <jtv> (It is a dark day for travelers worldwide.  All the sudokus I picked up on this trip were so easy that they might as well have said "each little square should have a number in it, but some are empty.  Can you see where they are?")
[15:54] <jcsackett> allenap: can i get you to take a look at https://code.launchpad.net/~jcsackett/launchpad/spam-button-ui/+merge/60072
[15:55] <allenap> jcsackett: Sure.
[15:58] <jtv> mrevell: moving on, the paragraph starting with "Previously, Ubuntu took translation templates" introduces two consecutive notions with the word "this" (which makes it a bit unnatural to read) and uses the word "allowed" to describe an undesirable effect (which may confuse readers who lack confidence with the subject matter).  Especially because it also uses the word "diverge" in a way that's hard to avoid, but doesn't sit quite right with the 
[15:59] <jcsackett> allenap: i believe the diff appears pretty big, but a goodly junk of that is boilerplate for javascript tests and the fact that i renamed a test file.
[15:59] <allenap> jcsackett: No worries. It's such a cool feature I'd read it twice.
[15:59] <jcsackett> allenap: :-)
[16:00] <mrevell> thanks jtv, this is very helpful.
[16:00] <jtv> mrevell: how about moving the cause ahead of the consequence there?  "New upstream translations made after that initial import had no way [...].  Thus Ubuntu often ended up with different translations as translation work upstream and in Ubuntu continued independently over Ubuntu's 6-month release cycle."
[16:01] <mrevell> that's great, much clearer jtv
[16:02] <jtv> mrevell: explaining things is important, and any improvement is multiplied by the number of readers.  That's why I always appreciated your documentation work so much.
[16:03] <mrevell> Ah, thanks man. Looks like I'll have to buy you a beer in Dublin :) Although, I seem to remember that buying beer in Dublin requires a hefty bank loan first.
[16:03] <jtv> Well let's pool our mortgages and try the Guinness.  :)
[16:04] <sinzui> jcsackett: do you have time to mumble?
[16:04]  * bigjools will be putting some beverages in the car
[16:04] <jcsackett> sinzui: i do. one moment.
[16:05]  * jcsackett fires up mumble and prays
[16:05] <jtv> bigjools: bringing your own lukewarm English beer to Dublin?  How does that make things any better?
[16:06] <bigjools> jtv: you, as a Dutch man, have no right at all to talk about beer
[16:06] <jtv> (Apart from the slight reduction in lukewarm English beer the United Kingdom will experience)
[16:06] <jtv> Ohhh yes.  Bring it on!
[16:06] <jcsackett> sinzui: i am here, but it seems that you cannot hear me.
[16:06] <mrevell> haha
[16:06] <bigjools> if you want to drink beer served at sub-zero temperatures, be my guest!  But I prefer mine to actually taste of something ;)
[16:07] <jtv> bigjools: I appreciate that English cooking gave you a taste for cat's urine, but proper beer is best served at 4 degrees centigrade.
[16:07] <bigjools> jtv: you lie.  like a cheap carpet.
[16:07] <bigjools> the only reason it's served that cold is so you can tell it from urine my friend
[16:08] <jtv> bigjools: if you wanted to hit me where it hurt, you would have said "is someone trying to teach me about beer from the United States of America?"
[16:08] <jtv> Wouldn't have hurt _me_, but at least it would have hurt _someone_ who deserved it.
[16:09] <bigjools> jtv: anyone who drinks beer at 4C deserves it :)
[16:09] <sinzui> what's to know? American is like making love in canoe.
[16:09] <sinzui> We do have real breweries now, but that does not mean you will find drinkable that beer in an established
[16:10] <bigjools> I did find some great beer in the US once
[16:10] <jtv> bigjools: thank you, I'm glad you don't begrudge me my delicious cool beer.
[16:10] <bigjools> it had been imported from England :D
[16:10] <jtv> Except the Americans probably knew to refrigerate it, and it was a whole new experience of pain-free beer-drinking.
[16:11] <huwshimi> Actually the states is where a lot of the micro brewers say modern beer is at
[16:11] <bigjools> jtv: get back in your box, troll!
[16:11] <huwshimi> They influence a lot of beer these days
[16:11] <huwshimi> particularly a lot of pale ales
[16:11] <jtv> Face it, mainland Europe figured out how to make lager taste good while the Brits mumble out ale and "proper beer" into their warm, smelly pilsner ripoffs.
[16:12] <jtv> Ahhh this feels good.
[16:12] <jtv> Even American coffee can make you irritable if you drink enough of it.
[16:13] <huwshimi> Something like the Sierra Nevada Pale Ale is amazing and has influenced a bunch of amazing beers in Aus
[16:13] <huwshimi> I used to hate on American beer, but you can find crap beer anyway
[16:13] <jtv> I'll see if I can get some.  :)
[16:13] <huwshimi> jtv: You should
[16:14] <jtv> mrevell: I can't escape the impression that the blog post is trying to go in multiple directions at once.  Is it highlighting the new sharing, or more the whole new data flow that emerges from it?  The bit about branch imports is at the same time useful and a bit of a side track.
[16:18] <jtv> mrevell: it may help to separate the two aspects of the story.  "What has just changed?  How does it compare to existing sharing features?  What do these and other recent improvements add up to in terms of data flow?"
[16:19] <abentley> benji: qa-ok
[16:19] <benji> abentley: thanks!
[16:20] <jtv> mrevell: Not very different from what you do now, except I think it will make it clearer what each individual part of the text is driving at.
[16:20] <mrevell> jtv sorry otp
[16:20] <jtv> OK
[16:22] <rvba> allenap: could you take a look at https://code.launchpad.net/~rvb/launchpad/change-perm-sync2/+merge/60077 ? (I think you are a good candidate to review this, because you've heard the whole story about this branch ;))
[16:23] <rvba> bigjools: ^^
[16:28] <allenap> rvba: Sure.
[16:29] <allenap> rvba: I'm doing another review right now, but I can do your's once I've finished.
[16:29] <rvba> allenap: great, thanks.
[16:44] <jtv> bigjools: when we discussed creating distroseries indexes from the regular publisher cron, you mentioned that the manual verification steps were a bit of a holdover from when this was all a much bigger deal.  But since this goes into the new python-based publish-ftpmaster (another big change), shall I just leave the notification of release managers in place for the time being?  I can make it say something like "please verify manually."
[16:44] <bigjools> TDD = Tedious Driven Development with our test layers
[16:44]  * jtv adds TLA
[16:44] <bigjools> which is turning into RDD, Rage Driven Development
[16:45] <jtv> bigjools: hang on, I was already uploading TDD
[16:45] <bigjools> jtv: you could ask them if they would like to keep it!
[16:45] <bigjools> I honestly don't know
[16:45] <jtv> bigjools: what I mean is, it'd be good to keep their existing procedure for at least one more cycle, but more to ensure that my changes didn't screw things up.
[16:49] <bigjools> jtv: well sending them an email is not existing procedure, so I'd check about that since it might not be necessary.
[16:50] <jtv> The email is a relatively superficial change, I hope--from "run this and then verify" to "wait until the system tells you this has been run and then verify."
[16:51] <jtv> But as long as there's a manual step triggered by completion of an automated step, I think the notification makes sense.
[16:51] <jtv> I'll ask cjwatson though.
[16:52] <bigjools> ok cool
[16:52] <jtv> cjwatson, are you here?
[16:54] <cjwatson> jtv: we'll likely do the manual checks whether you want us to or not :-)
[16:55] <jtv> Good to hear.  :-)
[16:55] <jtv> The question is:
[16:55] <jtv> now that we're automating the manual "publish-distro.py" runs to create indexes for a new series,
[16:55] <jtv> do you want Launchpad to notify the release managers when that has been done?
[16:56] <cjwatson> oh, we don't need e-mail about that, no
[16:56] <cjwatson> we are already thoroughly used to monitoring publisher activity
[16:56] <jtv> Well, that certainly makes things simpler!
[16:56] <cjwatson> e-mail> or any other notification
[16:57] <cjwatson> there are plenty of existing Ubuntu processes that involve waiting for a publisher cycle to complete
[16:57] <jtv> Okay, then I'll rip out the email and you'll just have to assume that the regular publish-ftpmaster run will, the first time around, just initialize the indexes and quit.
[16:57] <jtv> (By the way, those publish-ftpmaster runs will be a bit faster now because of changes in process architecture and cache management)
[16:57] <cjwatson> IOW we no longer need to stop the publisher and run publish-distro by hand?
[16:58] <jtv> Definitely no longer the latter; let me just verify the former.
[16:58] <cjwatson> (the first publish-distro run already created indexes, it's just that it had to be given careful flags to do so)
[16:58] <cjwatson> well, and that we had to run initialise-from-parent beforehand, which I understand is going away
[16:59] <jtv> The first publish-distro run already created indexes, but that's currently a manual run.  We're talking about moving that into the regular cron job.
[17:01] <jtv> Yes, initialise-from-parent will be going away AIUI but I'm still fuzzy on the details there so can't make promises on whether you'll still need to stop the cron jobs.
[17:01] <jtv> But I'm at least taking steps 12-14 out of that window.
[17:02] <cjwatson> you're not taking out step 13, because that's not in Launchpad's gift to remove. :-)
[17:02] <cjwatson> but ack on 12 and 14
[17:02] <cjwatson> that's what I meant by careful flags (-A)
[17:02] <jtv> I am taking step 13 out of the disabled-cron window, but not out of the process.  Steps 12 & 14 will become automated.
[17:02] <cjwatson> having the publisher notice that the suite hasn't been published yet and Just Doing It is all that needs to happen to remove 12 and 14.
[17:03] <jtv> Exactly; that's what I'm working on.
[17:03] <bigjools> everything's frozen anyway so no harm by JFDI
[17:04] <cjwatson> i.e. bug 55211
[17:04] <_mup_> Bug #55211: shouldn't require careful publisher run when cloning new released distrorelease <lp-soyuz> <new-release-cycle-process> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/55211 >
[17:04] <bigjools> 5 digit bug!
[17:04] <mrevell> Hey jtv, I'll run up another draft of the announcement
[17:04]  * cjwatson closed two five-digit bugs just in the last week or so
[17:05] <cjwatson> they're less rare than people seem to think ... :-)
[17:05] <jtv> ack mrevell
[17:05] <bigjools> we're probably unwittingly closing a bunch with derived distros
[17:05] <jtv> I'll mark that one as in progress then.
[17:09] <jtv> bigjools: if I make the initial-run check check for Frozen status, then the change won't affect previously released distroseries.
[17:10] <bigjools> sounds very sane
[17:14] <jtv> bigjools: it probably doesn't make much sense to have a feature flag for this then, either.  The work will simply come with the new publish-ftpmaster.
[17:14] <bigjools> yup
[17:16] <jtv> yuuuup
[17:16] <bigjools> phrasing!
[17:17] <allenap> rvba: I'm not going to get to your branch until later this evening. Is that okay?
[17:17] <jtv> It's not a competition, bigjools!
[17:19] <rvba> allenap: of course, no worries.
[17:20] <allenap> rvba: Thanks, sorry about that.
[17:22] <bigjools> I am executing a remarkably satisfying sql statement on mawson
[17:23] <bigjools> DROP DATABASE launchpad_dogfood
[17:24] <cjwatson> jtv: as long as it's clear that the Frozen status doesn't apply only to just-opened series
[17:24] <jtv> cjwatson: otp, brb
[17:36] <poolie> bigjools, :)
[17:37] <poolie> should i have a bug for even cleanup type landings, like https://code.launchpad.net/~mbp/launchpad/mail-cleanup/+merge/58867 ?
[17:37] <bigjools> poolie: it's getting the last laugh though, the production restore is failing!
[17:49] <LPCIBot> Project windmill-devel build #32: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/32/
[17:55] <jcsackett> allenap, rvba: i can take rvba's MP.
[17:56] <jcsackett> unless there is a particular reason you should do it, allenap.
[18:31] <LPCIBot> Project windmill-devel build #33: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/33/
[18:41] <jcsackett> allenap: saw your notes on my MP; i concur on the on and failure success handlers; meant to connect them up and clearly forgot.
[18:42] <jcsackett> regarding the permission checks tho, i was under the impression that check_permission did cache permission stuff on it's own. i take it i'm laboring under false info?
[18:51] <LPCIBot> Project windmill-db-devel build #238: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/238/
[18:52] <benji> jcsackett or allenap: hi guys, I have a very small branch for you to review: https://code.edge.launchpad.net/~benji/launchpad/bug-777766/+merge/60100
[18:52] <jcsackett> benji: i can look at it in a bit, but i'm currently looking over a 1400+ diff. :-P
[18:53] <benji> cool, thanks
[18:53] <jcsackett> np. just wanted you to know of the wait. :-)
[19:20] <sinzui> Why do we keep seeing edge URL in reviews. That env is deprecated, soon to be obsolete?
[19:53] <jcsackett> benji: i am looking at your branch now.
[19:53] <benji> sinzui: I used to have bookmarklets with edge in them, and prompted by you I just found "edge" in one of my MP-related scripts which was the source of the "edge" above
[19:54] <sinzui> ah
[19:59] <allenap> jcsackett: Ah, I didn't see your comment about taking rvba's branch. I'm back now so I can do it, but thanks for the offer.
[19:59] <jcsackett> allenap: it's done. :-)
[19:59] <allenap> jcsackett: Oh, awesome :)
[20:01] <jcsackett> benji: only one question on your branch. does the crazy character mash that is base64 show up in a url now? or is this just in ids? from the test, i'm assuming the latter.
[20:01] <benji> jcsackett: yeah, it's just in IDs; user's wont see it
[20:01] <jcsackett> benji: good stuff. :-) r=me, then.
[20:02] <benji> I just used the URL base 64 because instead of + and / it uses - and _ (so it's ID-safe (except for the stuping paddin characters (which I remove after the fact))).
[20:16]  * lifeless stretches
[20:16] <lifeless> wgrant: if its specialised, is it on unusual deployment notes ?
[20:20] <LPCIBot> Project windmill-devel build #34: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/34/
[20:22] <abentley> lifeless: is it true that testtools doesn't provide a matcher for regular expressions?
[20:28] <lifeless> I'm not aware of such a matcher; it would be great to have one
[20:29] <lifeless> matchers were an experiment, one I think that has panned out well, but we haven't gone and migrated all the custom asserts etc into matchers
[20:30] <lifeless> I was thinking yesterday that its about time we did that
[20:31] <abentley> lifeless: Ah, I see.  It seemed like an odd oversight.
[20:35] <lifeless> abentley: we started with just a couple - equals and doctestmatches
[20:35] <lifeless> and have grown from there as people found them useful and nice to work with
[20:35] <lifeless> flacoste: what are these wiki edits all about
[20:36] <lifeless> e.g. ... +  * r9242  [r=jtv][ui=none][bug=297458] For wgrant: export bug nominations.
[20:36] <flacoste> lifeless: testplans -> Trash
[20:36] <lifeless> flacoste: but they are being edited not deleted
[20:36] <flacoste> lifeless: nah, i'm using Delete Page
[20:36] <flacoste> not sure how the notification looks liek
[20:36] <lifeless> -weird-
[20:36] <lifeless> I can see its gone now I try the page url itself
[20:37] <lifeless> let me forward you the notification so you can go WTF
[20:37] <flacoste> lifeless: i don't nee dmore WTF moments :-)
[20:37] <lifeless> sent :>
[20:40] <flacoste> lifeless: Moin crack :-)
[20:41] <lifeless> yeah
[20:43] <lifeless> you can understand my confusion
[20:47] <flacoste> it sure is confusing
[20:49] <LPCIBot> Project db-devel build #518: FAILURE in 5 hr 8 min: https://lpci.wedontsleep.org/job/db-devel/518/
[21:01] <LPCIBot> Project windmill-devel build #35: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/35/
[21:02] <lifeless> sinzui: can you spare 45m to an hour sometime today to talk with me ?
[21:03] <sinzui> I can
[21:03] <lifeless> sinzui: I've started looking beyond the immediate exigencies of performance... great
[21:04] <lifeless> bigjools is apparently popping back into irc at some point his evening; given tz constraints I want to chat with him when he does that; we could either wait for that to have happened, or have a call that might be interrupted
[21:13] <lifeless> I need a keyboard with catguards
[21:13] <lifeless> I'm thinking raised 2cm ridges at each end
[21:20] <deryck> jcsackett, you might like to review my current js fix.  It's stuff you and I chatted about before.
[21:20] <deryck> jcsackett, if you have the time, of course.
[21:20] <lifeless> deryck: I'd like to chat with you similarly
[21:20] <lifeless> deryck: but I imagine you are right on EOD as you run to UK times for some bizarre reason?
[21:20] <deryck> lifeless, about js?
[21:21] <lifeless> deryck: about our architecture
[21:21] <deryck> ah
[21:21] <deryck> lifeless, I've got about 1/2 hour left.  but I have some CHR-like duties I've yet to do.
[21:21] <benji> jcsackett: I have another small review if you have another few minutes: https://code.launchpad.net/~benji/launchpad/bug-771239/+merge/60113
[21:21] <deryck> lifeless, but I don't mind chatting.  Or we can chat first thing for you tomorrow?
[21:21] <lifeless> I would (even though tomorrow is sat) but we have to go pick up a kitten
[21:22] <jtv> allenap: you're not really still here, are you?
[21:22] <lifeless> 1/2 hour can probably touch the high notes
[21:22] <lifeless> deryck: so, lets do it!
[21:23] <deryck> lifeless, ok, let's do.  firing up mumble.
[21:23] <lifeless> deryck: d you have skype or voip?
[21:23] <lifeless> mumble is crappy from here
[21:23] <deryck> ah
[21:23] <deryck> I don't know if I ever installed skype.  let me see.
[21:23] <lifeless> the canonical voip is pretty nice if you have that
[21:24] <lifeless> if you haven't got skype we can try mumble, we might get lucky
[21:24] <deryck> lifeless, I haven't setup voip yet.  doing skype install now.  should be quick.
[21:24] <lifeless> kk
[21:24] <deryck> I need to get voip up.
[21:25] <deryck> I'll warn you its loud here.  Working from my wife's shop and it's after school time. :-)
[21:25] <lifeless> thats ok
[21:25] <deryck> you say that now ;)
[21:25]  * deryck is not bothered by noise, but late day calls are fun
[21:27] <benji> allenap: I have a small review if you have a few minutes: https://code.launchpad.net/~benji/launchpad/bug-771239/+merge/60113
[21:28] <maxb> lifeless: If you have a moment between all the calls, could you revisit the question of giving me access to the lpreview-body packaging branch as broached in bug 766149? Thanks.
[21:28] <_mup_> Bug #766149: bzr-lpreview-body from launchpad ppa blocks python 2.6 install on natty by using site-packages <packaging> <Launchpad review body plugin:Confirmed> <bzr (Ubuntu):Invalid> < https://launchpad.net/bugs/766149 >
[21:28] <lifeless> maxb: I have no issue with that.
[21:28] <lifeless> flacoste: ^
[21:29] <flacoste> maxb: me neither
[21:29] <jtv> jcsackett: are you available for a review?  It's large, but more than half the overall diff is deleted lines.  https://code.launchpad.net/~jtv/launchpad/bug-777941/+merge/60116
[21:39] <maxb> Great, in that case, could you migrate ownership of https://code.launchpad.net/~launchpad/+recipe/lpreview-body-daily and https://code.launchpad.net/~launchpad/lpreview-body/packaging to ~launchpad-committers ?
[21:45] <LPCIBot> Project windmill-devel build #36: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/36/
[21:48] <lifeless> sinzui: so with that in mind, what time suits you for a call
[21:54] <lifeless> bah
[21:54] <lifeless> I have to get that battery sorted
[21:55] <lifeless> maxb: what were those urls
 Great, in that case, could you migrate ownership of https://code.launchpad.net/~launchpad/+recipe/lpreview-body-daily and https://code.launchpad.net/~launchpad/lpreview-body/packaging to ~launchpad-committers ?
[21:55] <maxb> (thanks)
[21:56] <lifeless> apparently not
[21:56] <poolie> hello lifeless
[21:56] <lifeless> I get 'undefined' on the picker widget
[21:56] <lifeless> hi poolie
[21:57] <lifeless> sinzui: if you replied I missed it
[21:57] <poolie> i'm trying to ec2 test some branches
[21:57] <poolie> they seem to be sinking without trace
[21:57] <poolie> i suppose while trying to send mail or something
[21:58] <lifeless> if the failure is too big it can hit gmails size limits
[21:58] <sinzui> lifeless: sorry. I was distracted. we can talk now or in 4 hours
[21:59] <lifeless> sinzui: now sounds good to me
[22:00] <poolie> is there any way to ask it to just leave the instance running serving the file?
[22:00] <sinzui> lifeless: skype?
[22:00] <lifeless> poolie: I don't think so
[22:04] <poolie> :/
[22:05] <flacoste> poolie: patch welcome :-)
[22:05] <flacoste> poolie: there is an option not to detach from the instance
[22:05] <flacoste> you could tail the output
[22:06] <poolie> tail the output of what?
[22:07] <poolie> the hotel wireless drops out occasionally
[22:07] <poolie> i guess i could start an ec2 machine, ssh there and run screen, then run things from that
[22:11] <salgado> poolie, you can use ec2 demo to start the instance; that way you have everything ready to run the test suite when you ssh into the instance
[22:11] <poolie> am i supposed to normally be able to ssh in to the test versions?
[22:11] <poolie> they seem to have a server but it doesn't accept my key
[22:13] <salgado> poolie, I think your key is associated to the ubuntu user there
[22:13] <poolie> oh ok
[22:14] <poolie> no, apparently not?
[22:15] <salgado> I'm pretty sure your key is there, just need to find out under which user
[22:16] <poolie> oh ec2test@
[22:17] <jcsackett> deryck and benji: sorry for not replying, i seem to be having IRC connection issues today. i have looked at your branches, and they are a-ok. :-)
[22:17] <jcsackett> jtv: i am looking at yours now.
[22:23] <poolie> ok, so ssh'd in, the log shows
[22:23] <poolie> Running command: /usr/bin/xvfb-run --error-file=/var/tmp/xvfb-errors.log --server-args='-screen 0 1024x768x24' /var/launchpad/test/bin/test -vv --subunit -vvv
[22:23] <poolie> and nothing more
[22:23] <poolie> it does seem to be using cpu
[22:27] <LPCIBot> Project windmill-devel build #37: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/37/
[22:45] <mwhudson> hmm, i seem to be getting launchpad bug mail again
[22:46] <jcsackett> jtv: r=me. i have two suggestions in the MP, but they don't block landing--just things to consider.
[22:46] <mwhudson> i guess this is to do with the better subscription story stuff being enabled for ~launchpad?
[22:46] <mwhudson> ah no, it's ~launchpad loosing its contact address i think
[22:48] <mwhudson> luckily the better subscription story lets me turn off the firehose
[22:52] <jelmer> ooh, the new bug subscriptions are nice :)
[23:04] <LPCIBot> Project windmill-db-devel build #239: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/239/