=== Ursinha is now known as Ursinha-afk [03:15] spm: Hi. I'm trying to QA my fix for bug #592573. The differences between the right panels on https://launchpad.net/builders and https://edge.launchpad.net/builders show that there are ~146 builds in some sort of limbo. Do you have a moment to do a bit of DB poking to work out what they are and why? [03:15] <_mup_> Bug #592573: BuilderSet.getBuildQueueSizes doesn't consider non-binary builds [03:19] wgrant: hrm. curious. sure. I guess the ones you have via the bug is a good starter? [03:27] spm: SELECT id, builder, lastscore, job, job_type, processor FROM buildqueue WHERE virtualized=true; [03:27] At the moment that query *should* be empty. [03:28] But it will probably return around 61 rows [03:28] (23461 rows) ho ho ho ho [03:28] Oh, oops. Forgot a join. [03:28] heh, np [03:28] SELECT id, builder, lastscore, job, job_type, processor FROM buildqueue JOIN job ON job.id = buildqueue.job WHERE virtualized=true AND job.status = 0; [03:29] you did get the '61', just missed the 23,400 as well. so.. not too shabby? [03:29] SELECT buildqueue.id, builder, lastscore, job, job_type, processor FROM buildqueue JOIN job ON job.id = buildqueue.job WHERE virtualized=true AND job.status = 0; [03:29] yarp. 61 rows [03:30] one looks suspiciously old. very low job# compared to the rest [03:30] There should be nothing sensitive there (I excluded logtail). Can you pastebin, please? [03:31] http://paste.ubuntu.com/466237/ [03:31] Oh wow :( [03:31] hrm? in what sense? [03:31] Can you 'SELECT MAX(id) FROM buildqueue;' just to see how old those are? [03:32] 3708959 eek [03:32] Ah, so nothing new. Good. [03:32] Just from the early days of the move to the job system, I suspect. [03:33] (those are all queued builds that are being ignored, so are somehow corrupt) [03:35] ahh I see. is this something that should be cleaned up? or can be happily ignored? [03:36] We need to clean it up. I guess I'll talk to Julian about it. [03:37] oki, ta [03:37] SELECT * FROM buildpackagejob WHERE job=2691238; [03:38] I suspect it's the BuildPackageJobs that are missing. [03:38] id | job | build [03:38] --------+---------+--------- [03:38] 495749 | 2691238 | 1684155 [03:38] Damn. [03:45] spm: ola amigo. como estas [03:45] spm: or, should I say - ¿como estas? [03:46] mtaylor: alas, my spanish is limited to that picked up via overhearing Dora the Explorer. So beyond Hola and Gracios(sp?) you've lost me :-) [03:46] spm: ¿como estas? means "what's up?" - although I honestly don't speak much myself [03:47] heh [03:47] but I'm in cozumel, mx, so I'm trying to make an effort past "uno mas magarita, por favor" [03:48] spm: One last one: [03:48] SELECT buildqueue.id, builder, lastscore, buildqueue.job, job_type, processor, build FROM buildqueue JOIN job ON job.id = buildqueue.job JOIN buildpackagejob ON buildpackagejob.job = job.id WHERE virtualized=true AND job.status = 0; [03:48] Then I can examine the builds myself. [03:48] mtaylor: ha! [03:49] * mtaylor cringes that you're implementing queues in the database... then decides to shut his mouth before he's asked to fix it [03:49] wgrant: http://paste.ubuntu.com/466245/ [03:49] what do you mean *implementing*?!? implemented. :-) [03:50] *shudder* :) [03:50] mtaylor: Before my time :( [03:50] This is five years old :( [03:50] I believe plans are to move to something like rabbit, or whatever. but nfi. [03:50] wgrant: it's ok - in my days as a mysql consultant, I saw _many_ _MANY_ thing implemented inside of an RDBMS that didn't belong there [03:51] but my favorite things people mis-use dbs for are queues and email [03:52] free tip #1 from formerly over-paid db consultant - if your app needs some sort of message system that's similar to email ... USE EMAIL SERVERS .. don't half-way implement private broken email in database tables [03:52] spm: :) [03:52] spm: Ah, it's not a bug after all. Most of those are builds that were cancelled by manual SQL to mark them superseded. [03:52] ah [03:53] Not all, but most. I'll work out how to clean them up. [03:53] Thanks. [03:53] mtaylor: that sounds suspiciously like heresy. email for email!?!? I mean, srsly!?!? [03:54] wgrant: np [03:55] spm: I know - right? you should have seen the look on the client's face the first time I suggested that as the fix for their performance isues [03:55] issues [03:55] "um, hai! you've implemented email in php ... try learning the internet" [03:55] heh [03:56] NIH. Alive and well. [03:56] yup. also "I don't want to learn anything" [04:58] morning [05:19] * wgrant is still looking for someone to land three branches. [05:29] here's two fun lines to have next to each other: [05:29] from canonical.launchpad.layers import WebServiceLayer [05:29] from canonical.testing.layers import FunctionalLayer [05:43] mwhudson: can you help wgrant out? [05:43] pretty please? [05:46] ah right [05:46] wgrant: url me up [05:47] mwhudson: lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts, lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/really-publish-ppa-ddebs [05:48] * mwhudson grrs at not being able to give branch urls to ec2 land [05:49] Can't you? [05:49] I thought you could now. [05:49] well i didn't work for these [05:55] Does anyone happen to know if BFB works for Hardy yet? [05:59] wgrant: ok, three instances started [06:00] wgrant: did it not work for hardy at some point? [06:01] mwhudson: Oh, right, I remember now. [06:01] Its bzr was old, so it didn't work with 2a. [06:03] Thanks for sending those off. [06:23] mwhudson: Ah, no, it is really broken like this: http://launchpadlibrarian.net/52193804/buildlog.txt.gz [06:23] bzr-builder: Depends: python-debian but it is not going to be installed [06:24] wgrant: weee [06:26] * wgrant builds it the old way :( [08:03] wgrant: hi [08:39] lifeless: Hi. [08:40] mtaylor, re: Tarmac needing commit message set... The reasoning is because Tarmac doesn't know what to set the commit message to when it commits if you don't specify it. [08:40] mtaylor, I realize that this is rather awkward. Merge queues will fix that. [08:40] * rockstar keeps rolling the squeaky wheel [08:40] what advice do we normally get to people who don't receive the account confirmation mail? [08:41] check spam [08:41] SSO does hate some domains, though. [08:43] or they hate us? [08:43] is there any escape? [08:44] Well, OK, one of my email aliases that is handled by a Canonical machine does not get LP or SSO email. [08:44] it becomes a losa ping [08:44] Everything else works. [08:44] So there's something not quite right about some email setup somewhere. [08:44] wgrant: orly? have you filed an rt? [08:45] lifeless: The address isn't important to me, so I never really bothered. [08:48] lifeless: What was that ping before about? [08:48] buildds [08:48] seen the thread on deployment? [08:48] Yes. [08:48] They don't care if the connection dies. [08:48] can you confirm or deny slave behaviour ? [08:49] You can kill buildd-manager at any time, and it will be fine. [08:49] \o/ [08:49] elmo: ^ [08:49] wgrant: what other things can go wrong [08:49] buildd-manager and the protocol are pretty stateless. [08:49] (apart from DB build state) [08:49] sure [08:50] wgrant: is there anything that can be done to make the publisher runs able to be interrupted without causing havoc ? [08:50] lifeless: Um. [08:50] They shouldn't be tooo bad at the moment. [08:50] any manual recovery == too bad [08:50] For PPAs it would be really bad. [08:50] For primary it should just about work. [08:51] (primary has atomic dists/ update; PPAs do not) === almaisan-away is now known as al-maisan [08:54] Phase A (file publishing) is fine, since it doesn't matter if we publish something again on the next run. [08:55] B (domination) is all in one transaction, and doesn't touch the filesystem at all. [08:55] But C and D touch indices, so will leave the archive inconsistent. [08:56] Hm, actually, it's slightly worse. [08:56] If we get through A fine, commit, then get terminated, there'll be no dirty pockets to force index regeneration when everything is rerun. [08:57] 2pc needed ? [08:57] That would mean a half-hour transaction. [08:57] ugh [08:57] (alternatively we could make the publisher less goddamn slow) [08:57] or start domination later ? [08:58] So, there are two problems: [08:58] there are two sorts of people in the world .... [08:58] 1) We rely on state internal to the publisher process to work out whether we need to regenerate indices. [08:58] 2) Termination during index generation will leave the archive inconsistent for at least a few minutes. [08:59] seems like they are both addressed by making indice regeneration into a separate task [08:59] pipeline-like [08:59] Applying the atomic dists/ update system to all archives would solve #2 simply. [09:00] For #1... we may need to store dirty pockets in the DB. [09:00] also less code variation - ++ [09:03] Perhaps. [09:03] But the way it's done now is not exactly... acceptable. [09:03] GUten morgen. [09:06] wgrant: are bugs filed to fix it? [09:07] lifeless: It's cron.publish-ftpmaster. Everyone is terrified. [09:07] is it risk ? [09:09] It's evil and does some things that nobody understands any more. [09:09] Well, I guess that's just dsync. [09:13] Morning bigjools. [09:13] You may be, er, interested to know that we have somewhere around 146 inconsistent builds on production. [09:13] They have BuildQueues and Jobs, but are not actually pending. [09:15] sigh [09:15] givebacks? [09:15] Some are SUPERSEDED, others are FULLYBUILT. [09:15] did I just catch lifeless looking at cron.publish-ftpmaster? [09:15] The former can be explained by LOSAs manually cancelling builds, I suppose. [09:16] I expect so - doing it wrong. Are they package builds? [09:16] binary that is [09:16] They're all BPBs, yes. [09:16] are they rebuilds? [09:16] http://paste.ubuntu.com/466245/ are the virtualized ones. Although there are a couple of legitimate builds that snuck in there. [09:17] how was this discovered? [09:17] The ones with score == 0 are the buggy ones. [09:17] I was QAing my getBuildQueueSizes change. [09:17] Compare the build queue sizes on edge and production. [09:17] sigh [09:18] lastscore=0 indicates a retry [09:18] Ah, true. I guess it would be NULL if they'd just finished naturally. [09:19] They're fortunately all fairly old. [09:19] I need to clean up the rebuild anyway [09:20] There are 80 or so non-virt builds which I didn't query for, since the non-virt build queues are large. [09:20] how old? [09:21] Well, we're up to BQ 3700000 or so. [09:21] Most of them are around 3330000 [09:21] did you get any dates? [09:22] They should be in Job, but I didn't query for them, no. [09:22] I decided I'd bothered people enough. [09:23] it's so much harder to query across jobs now :( [09:24] http://paste.ubuntu.com/466245/ has the query [09:24] I have some pre-potted ones [09:24] Just throw a couple of extra columns like BPB.status and Job.date_created in, I suppose. [09:25] argh, I can't even do this on staging any more [09:25] No perms? [09:25] no, we wipe the queues when restoring [09:26] to stop staging collecting build from production [09:26] Oh, true. [09:26] So, the few builds on there from before 3300000 are all SUPERSEDED. [09:26] Then there are a couple around 3300000 [09:26] And the rest around 3330000 [09:27] I haven't checked the statuses for the last two categories, though [09:32] wgrant: there's one (!) like that on dogfood [09:33] bigjools: What's its status? [09:33] Build status, not Job status. [09:37] * bigjools rides SQL wild horses [09:38] bigjools: asking about things that will make deployment hard [09:38] bigjools: easy deployment is a really important thing for iterating on performance at faster than monthly cycles. [09:38] lifeless: do you know how it will make it hard? [09:39] bigjools: no idea, wgrant mentioned it was all [09:39] other than being a PoS bash script, I don't see the problem [09:39] bigjools: for the publisher specifically I'm told that interrupting it causes damaged PPA archives until its run again, and in some cases until the PPA has a new build and the publisher runs again. [09:40] yes it's possible, but easy to fix [09:40] bigjools: If we die during !primary index generation, the indices will be inconsistent until the next run. [09:40] And if we die before index generation, there'll be no dirty pockets, so no index regeneration will occur. [09:40] we need to write to a tmp dir like ubuntu does [09:40] That's what I suggested. [09:41] But it's not trivial to port, given how it's done now. [09:41] and remove the stupid partial commits [09:41] We can't do that until we publish everything really quickly. [09:41] bigjools: the impact of that on the deployment story is that we have to be careful about when we start deployments, shutting down the task well in advance, which adds latency and that adds perceived downtime. [09:41] I think a half-hour transaction over the whole primary archive would make stub cry a bit. [09:42] none of this is intractable, but the more steps that are needed, the more coordination, the slower the process is. [09:42] yeah, whoever wrote the publisher knew nothing about recoverability [09:43] yup [09:43] bigjools: Alternatively, we could write indices atomically like the primary archive, and store dirty pockets somewhere persistent. [09:43] Everything else should work. [09:43] wgrant: everything you do to make the system more resilient, j'adore [09:43] me likey atomic [09:44] Or we make the publisher complete in a few seconds, and do it all in one transaction. [09:44] But I can't get it much below two minutes. [09:44] can you make it pipeline / incremental ? [09:45] The thing that takes all the time (after optimisation) is serialising the indices. [09:45] does the db know the indices ? [09:45] pipelining is possible but has ramifications [09:46] could you, just commit, then do the indices as a read-back from the db ? [09:46] (in principle) [09:48] lifeless: you mean lazy generation? [09:48] not sure I follow [09:49] I'm fixing the conflict, btw [09:50] is there any meaningful difference between IStore(self) and Store.of(self)? [09:50] IStore uses c.l.webapp.adapter.get_store [09:51] I think the second reads more easily [09:51] Does the former respect the master/slave policy? [09:51] bigjools: I mean doing the transaction commit as soon as possible [09:51] bigjools: and generating the indices from the result of the commit, not within the commit [09:52] lifeless: The problem is that if we commit before doing indices, we don't know that we need to regenerate them later. [09:52] lifeless: what wgrant said [09:52] wgrant: that seems fixable [09:52] hehe - take a look at the publisher :) [09:52] lifeless: Right, by storing dirty pockets in the DB, which fixes it all. [09:52] But is very ugly. [09:53] compared to 30 minute db transactions with uninterruptable cron scripts that prevent rollouts for 30% of the day. [09:53] not ugly at all. [09:53] Shhhhh. [09:54] I'm very keen to see things here improved. [09:54] The general principle of 'do small amounts of work, often' and 'delay till outside of transactions things that don't need to be in the transaction' are very dear to me. [09:56] wgrant, the former goes straight to the master, I think. [09:56] jml: Ah. [10:04] wgrant: does this look sane to display the affected builds? http://pastebin.ubuntu.com/466361/ [10:06] wgrant: your branches were 1 of 3, did you get the emails? [10:12] mwhudson: Yeah, saw that. Thanks. [10:12] bigjools: I would have said buildfarmjob.status NOT IN (0, 6), but looks OK. [10:13] wgrant: using NOT IN makes queries slow [10:13] bigjools: Ah, I guess so. [10:13] bigjools: Also, grab buildqueue.virtualized. [10:13] why? [10:14] Why not? [10:14] Might as well get all the categorisation information. [10:16] I only omitted it from the initial query because I was restricting to virt jobs. [10:23] * wgrant grumbles about the incomplete Registry<->Soyuz split. [10:41] mwhudson: Tests fixed. Can you please rerun, if you're still around? [10:42] I guess not. [10:43] Can someone else please re-EC2 lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts? [10:49] wgrant: problematic queue rows blitzed, how's it looking? [10:49] bigjools: So none of them were actually pending? [10:50] some where but I left those alone ;) [10:50] sigh [10:50] some *were* [10:50] That looks OK. [10:50] The numbers match now. [10:50] \\o/ [10:50] I will be really glad when we get the model rework done, and such inconsistency becomes impossible. [10:50] yarp [10:51] Because we will be able to avoid storing the status in four or five places... [10:51] although notice that all of those were from march/april [10:51] Yeah. [10:52] wgrant: https://edge.launchpad.net/~oem-archive/+archive/budapest/+build/1880257 [10:52] can you see that? [10:53] bigjools: No. Is that the one that failed to upload this morning? [10:54] yes [10:54] I considered going hunting for the upload log, but decided the search space was slightly too big. [10:54] only it says it's built properly so it looks like it dispatched twice :/ [10:54] What was the upload error? [10:54] PM [10:54] Or is it FULLYBUILT now? [10:54] ? [10:55] oH. [10:55] Right. [10:55] it's built [10:55] So it was built four times. [10:55] Succeeded the first and last. [10:55] But somehow was retried after the first. [10:55] Yay. [10:55] I thought we'd weeded out all of that :( [11:05] wgrant: I suspect double clicking on UI buttons [11:05] bigjools: But... transactions. [11:05] wgrant: that already causes havok on copy packages [11:05] For copies, sure. [11:05] wgrant: they don't help if the db constraints don't catch the problem [11:05] bigjools: But a retry resets the BFJ status. [11:06] They both have to update the same row. [11:06] I have a suspicion something is double-forwarding every now and then [11:06] or something [11:06] to the same thing [11:06] see the bug I filed about getting two bugs [11:06] bigjools: I really hope that Postgres doesn't accept that. [11:06] wgrant: unless there's a constraint, it will [11:06] lifeless: really? ugh [11:08] elmo: https://bugs.edge.launchpad.net/soyuz/+bug/607397 [11:08] <_mup_> Bug #607397: buildds need to survive the buildd master being upgraded [11:08] elmo: can you please describe there the build farm issue you related to me - or perhaps its no longer an issue ? [11:09] bigjools: With SERIALIZABLE as the isolation level, a concurrent update like that is prevented. [11:09] I just checked. [11:09] I wonder what Storm uses, though. [11:10] elmo: nvm [11:10] It default to serializable, as I'd hoped. [11:12] Ahh. [11:12] But LP overrides it to READ COMMITTED, and that allows it. [11:12] Damn. [11:13] bwah [11:13] all of lp is read committed ? [11:13] I think so. Need to check Postgres logs harder. [11:13] that's.... not good [11:13] * wgrant looks harder. [11:14] At least the appserver transactions immediately set it to READ COMMITTED, or so the postgresql logs show. [11:15] bah [11:15] wgrant: you broke our builds [11:15] :P [11:15] lifeless: It was too quick to be my fault :( [11:15] Build Reason: [11:15] * wgrant blames the build system. [11:15] Build Source Stamp: [branch bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel] HEAD [11:15] Blamelist: William Grant [11:15] BUILD FAILED: failed shell_6 compile [11:15] Oh yes, I got the email. [11:15] But the test suite doesn't take an hour. [11:16] So either I broke things really badly, or the build system is broken as usual. [11:16] Or someone turned on parallelisation while I wasn't looking. [11:16] no [11:17] not done yet [11:18] bigjools: Ah, wait, there's a UNIQUE on buildpackagejob.build. So you can't queue the build twice regardless of how screwed LP's default transaction level may or may not be. [11:19] that's the kind of constraint I like [11:19] So we have more hunting to do. [11:19] grar [11:19] Anything in the logs yet, or must we go librarian diving? [11:20] hang on [11:22] Hmm. The librarian GC is rather aggressive :/ [11:22] mpt: hai [11:22] mpt: lunch is @ 1 [11:22] mpt: are you planning to starve me? [11:22] It will immediately kill anything unreferenced and with no expiry date set. [11:23] lifeless, clearly, by "1" I meant "2" [11:23] awesome [11:23] Ursinha-afk: how does the oops <-> bug stuff work ? [11:24] wgrant: are you working on 78 SELECT COUNT(*) FROM Archive, BinaryPackageBuild, BuildFarmJob, PackageBuild WHERE distro_arch_se ... tus=$INT AND Archive.purpose IN ($INT,$INT) AND Archive.id = PackageBuild.archive AND ($INT=$INT): [11:24] ? [11:24] jml: Um, canonical.uuid has been gone for more than a hundred devel revisions... [11:24] lifeless: I don't really know how to fix it. [11:25] There's nothing obviously wrong with it that I can see. [11:25] http://paste.ubuntu.com/465800/ is the EXPLAIN ANALYZE of the other slow query, which is just about the same. [11:26] right [11:26] so dropping the count * separate query will save 5 seconds [11:26] sorry 7.7 [11:26] But that's lazr.restful. [11:26] Not sure we can do much about that. [11:26] And the query shouldn't take long at all anyway. [11:26] sure we can [11:26] is there a bug for it ? [11:26] The slow queries? [11:26] wgrant, I'm just the messenger. [11:26] the problem [11:26] I filed one a month or two ago. [11:26] whats the number [11:27] * wgrant is hunting. [11:27] If bug search was a little better... [11:27] hush [11:27] Bug #590708 [11:27] <_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out

[11:28] jml: Well, since I have no recent shipit, I can do nothing. [11:29] bigjools: hi - https://bugs.edge.launchpad.net/soyuz/+bug/590708 [11:29] <_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out

[11:30] * wgrant just commented with the paste. [11:30] lifeless: I wonder if we should test kernel delayed copies and acceptance from +queue before taking the timeout down permanently. Those are done infrequently, take ages, and it's pretty bad if they stop working. [11:34] wgrant: can you add a test plan for testing them on staging ? [11:34] wgrant: staging is at 12 seconds. [11:34] lifeless: Um, I'm not sure if testing on staging is valid. [11:35] wgrant: why wouldn't it be ? [11:35] lifeless: It sucks performance-wise. [11:36] right [11:36] so if it works on staging, we're set for prod. [11:36] True. [11:45] bigjools: Any luck? It'd be nice to get onto it before librarian-gc deletes the evidence. [11:45] wgrant: no, best start diving [11:50] bigjools: I guess you could just look for any recent restricted 'uploader.log's... [11:51] urh [11:51]