[00:05] spm: hi [00:06] spm: I'm sure you have a bunch of firefighting to do [ubunet cron, *cough*] [00:06] spm: but I wonder, as its 1am here, if you can do me a solid first [00:06] lifeless: yo; sup? [00:06] spm: which is, to deploy lp:~lifeless/launchpad/malone so I can see if it does make dup checking faster [00:06] on staging [00:06] oh sure, app server I assume? [00:07] yeah [00:10] yoikes that's a big patch. it seemed to have a merge fail; but it scrolled out of my buffer. trying again to a file [00:10] spm: it should be tiny [00:10] spm: https://code.edge.launchpad.net/~lifeless/launchpad/malone/+merge/30904, [00:10] bzr merge --preview lp:~lifeless/launchpad/malone ? [00:11] just take the latest version then of it? [00:11] spm: https://code.edge.launchpad.net/~lifeless/launchpad/malone/+merge/30904/+preview-diff/+files/preview.diff perhaps [00:11] that works [00:11] spm: its probably other devel stuff naffing staging up, or something [00:11] nod [00:12] damn. 4/4 hunks failed [00:12] which ones [00:12] oh, all [00:12] :-) [00:12] erm [00:12] 1am, you said earlier. noted. :-) [00:12] on which files [00:12] there is much more than 4 hunks in there [00:13] http://paste.ubuntu.com/469077/ [00:13] yeah - that was just the most obvious [00:13] spm: can you revert; then pastebin bzr missing lp:~lifeless/launchpad/malone ? [00:14] preview - i always preview/dry-run first [00:14] sure sure [00:15] lifeless: http://paste.ubuntu.com/469078/ [00:16] odd [00:16] 11189 should be included [00:16] can you grep slow_ lib/canonical/database/nl_search.py ? [00:16] 0 results found [00:17] ok [00:17] merge -c 11199 lp:~lifeless/launchpad/malone ; [00:18] merge --force -c -1 lp:~lifeless/launchpad/malone [00:18] should behave better [00:21] oki [00:21] its in place ? [00:21] is that bzr, svn or git merge? [00:21] not yet [00:21] bzr [00:21] 1am. trolling is too easy. sigh. [00:21] 121 [00:22] qed [00:22] 1st went in easy [00:22] vidi vici veni [00:22] on a preview; live doing now... [00:23] Civil Engingeering Students Association (@ Qld Uni) CESA: We Came. We Swore. We Concreted. [00:24] coolio, both applied; restarting [00:25] lifeless: restarted with new shiny ponies [00:26] hmm, you sure ? [00:27] yup :-) [00:28] this is the app server right? not code*? [00:28] start time: Mon Jul 26 00:24:52 2010 (BST natch) [00:30] booyah [00:30] wgrant: try this page https://bugs.staging.launchpad.net/ubuntu/+filebug [00:30] with e.g. eclipse get unmet dependency error [00:31] its hitting disk a lot still [00:31] but once off disk its hell snappy for me [00:32] mwhudson: ^ [00:32] spm: what do you think [00:33] Hm, not too bad. [00:33] whimper. can't login to staging I think... [00:33] But the results... I don't know. [00:33] No I lie. I'm in. wooo! [00:33] wgrant: they are missing the prefilter step that was doing millions of comparisons to filter out terms; will reinstate that in a bit [00:34] wgrant: other than that its ranked precisely the same way [00:34] [terribly] [00:34] seemed snappy to me [00:34] k [00:34] bombs away, I say [00:34] lifeless: Isn't it returning a subset of the original results, though? [00:35] yes [00:35] but ranking is strictly additive given the same terms as the rank function [00:36] so 3 terms >> 1 term anyhow [00:36] and we're not [yet] using heat or anything [00:37] night [00:38] Right. [00:38] I guess you try the one-missing query, and then jump to two-missing, three-missing etc. queries if there aren't enough results. [00:38] s/you/you could/ [00:52] spm: Some more corrupt builds appear to have sprung up :( [00:52] If you have a moment, could you run 'SELECT buildqueue.id, builder, lastscore, buildqueue.job, job_type, processor, build FROM buildqueue JOIN job ON job.id = buildqueue.job JOIN buildpackagejob ON buildpackagejob.job = job.id WHERE buildqueue.virtualized = false AND buildqueue.processor = 1 AND job.status = 0' for me? [00:53] There are somehow 13 corrupt builds in the i386 non-virt queue, for a total of 40 seconds of buildd time, and they don't appear to actually be in the queue. [00:54] *blink*. 0 rows? [00:54] Bah. [00:55] SELECT buildqueue.id, builder, lastscore, buildqueue.job, job_type, processor FROM buildqueue WHERE buildqueue.virtualized = false AND buildqueue.processor = 1; [00:55] They must be even more broken than the last lot. [00:56] we don't break by halves round here [00:56] id | builder | lastscore | job | job_type | processor [00:56] ---------+---------+-----------+---------+----------+----------- [00:56] 3233211 | | 12510 | 1941857 | 1 | 1 [00:56] (1 row) [00:56] wgrant: ^^ that last score seems... curious [00:56] Just means it's in a private archive. [00:56] is that like the 'score' we'd override a build? ahh kk. [00:56] So there's something odd going on here, and that's not it. === Ursinha-afk is now known as Ursinha [00:58] (https://edge.launchpad.net/builders -- see the i386 non-virt build queue. that query should have returned everything in it. :() [00:59] looks [00:59] Hmm. What if you say 'virtualised IS NULL' instead of 'virtualized = false'? [00:59] non-virt the offical builders yes/no? [00:59] Official, yes. [00:59] non-virt *is* the offical builders yes/no? yes, ta [01:00] 13 rows :-) [01:00] jobtype=4, processor=1, lastscore=1000 [01:01] so curious that the = false gets something at all. [01:01] recognising that sql has 3 binary states - on, off, null :-/ [01:02] It is somewhat odd, since it's not appearing in the queue, and is not building. But that's not the main issue. [01:02] Thanks for your assistance. [01:03] np [01:04] Odd that they're not being dispatched. But they're translations jobs, so who knows what evil they are up to. [01:06] I know people who would know what evil they're up to..... [01:10] There are another 18 properly virtual jobs floating around inappropriately, but it's just about impossible to track them down until the queue is empty. [01:10] Filed bug #609904, but that doesn't explain why the builds exist in the first place. [01:10] <_mup_> Bug #609904: BuilderSet.getBuildQueueSizes needs to consider virtualized=NULL as false [01:12] jtv: Hi. There are 13 pending translation template build farm jobs, and they're not being dispatched. Do you know why that would be? [01:27] wgrant: hi. No idea... it's only TTBJs? [01:30] jtv: There are some other non-dispatching builds around. But they're corrupt -- I'm not sure if the TTBJs are. [01:31] wgrant: corrupt in what way? [01:31] jtv: The status of the PackageBuild is wrong. [01:32] But you only have a single status, so that's not the case. [01:32] wgrant: it's not one of the recent changes and us not having a Build? [01:32] Maybe the bad build status is a result of the same thing that's causing these jobs not to be dispatched? [01:32] jtv: Unlikely. I saw some TTBJs building not even 24 hours ago, so it's not a general problem. [01:32] I wonder if the branchjob is missing or something. [01:33] A branchjob and a job are all we have, no? [01:33] And a buildqueue. [01:33] Ah right [01:33] that too [01:33] The buildqueue and job are present. [01:33] Not sure about the branchjob. [01:33] I don't trust the recent branch deletion changes, so I'll blame them for now :) [01:33] Deleting a branch might cause this... [01:34] The recent branch changes would cause both the BJ and the BQ to be cleaned up. [01:34] How old are these broken jobs? [01:36] The old behaviour (still running on production afaik) would cause an incomplete cleanup. [01:36] SELECT buildqueue.id, builder, lastscore, buildqueue.job, buildqueue.job_type, processor, job.date_created, job.status, branch, branchjob.job_type, json_data FROM buildqueue JOIN job ON job.id = buildqueue.job LEFT JOIN branchjob ON branchjob.job = job.id WHERE buildqueue.virtualized IS NULL AND buildqueue.processor = 1; [01:36] spm: ^^ [01:36] yup [01:37] wgrant: no rows [01:37] hrm that json data for those is interesting; all over the place. [01:37] Those two statements conflict.... [01:37] I ran mine on staging. [01:37] Ah. [01:38] staging has buildqueue cleared. [01:38] Because it has a buildfarm now. [01:38] jtv: https://pastebin.canonical.com/35029/ I think I can post that publicly, but can you confirm or deny? [01:38] There are no private TTBJs yet, so it /should/ be safe... but... [01:39] yeah, juts wanted to be doubly sure [01:39] the sus ones are all public [01:40] wgrant: http://paste.ubuntu.com/469098/ [01:41] Hmm. [01:41] So they all have BranchJobs, are not deleted, and all very fresh. [01:43] I don't know what's going on, unless it's trying to dispatch them but failing. [01:45] And maybe re-dispatching them? [01:51] wgrant, spm: it's suspicious that these are all so recent... might it be a dispatch priority issue? [01:52] jtv: recent as in the last 24ish hours? [01:52] spm: yes [01:53] Did we CP anything that could have done this? [01:53] jtv: and btw, good morning! you're up early? :-) [01:53] spm: amsterdam [01:53] 'up early' still rings true :-) [01:53] up late. :) [01:53] heh [01:54] met up with team members & an ex team member today... a good portion of the Prague attendance went to Holland next it seems. [01:54] jtv: no CP's to the build side since the 9th, so unlikely? [01:54] ahh cool! [01:55] Some for holidays, others for the GNU meeting, yet more for Guadec. [01:56] No CPs... then either it's operational or it's something weird on edge. [01:56] The branches I checked out weren't deleted, so I doubt it's related to branch deletion. [01:58] jtv: Yeah, it's a bit odd. I guess someone needs to check buildd-manager logs at some point :/ [01:58] edge? edge gets updated regularly - so CP? [01:58] :( [01:59] try again; so CP isn't as unlikely an option- it's just the normal state of affairs. [01:59] Hmm. I guess it's possible it's priority. [01:59] spm: yes, it's just that edge is less likely to have an effect on the build farm [01:59] But that would imply that the queue hasn't been empty for 24 hours. [01:59] ...and yet we have builders sitting idle. [02:00] I guess I'll just hope they disappear over the day. [02:02] Maybe something's bottlenecking on uploads? [02:04] wgrant: is the builder marked as idle while its uploads are being processed? [02:04] jtv: No. [02:04] it'd be nice to know what dispatch queries are going out right now... [02:05] It's normal at the moment for lots of builders to be idle, because buildd-manager sucks. [02:05] but... buildd-manager is far worse than normal. [02:05] It's redispatching the same set over and over. [02:05] Every minute or so. [02:05] lamont alluded to this yesterday, saying there was a rogue build causing it to abort most cycles. [02:06] I'll talk to Julian tonight, I suppose. [02:06] (if you click on a few of the current virtual builds, you'll see they started <30s ago. [02:06] Then if you refresh in another 30 seconds, they'll still be <30s ago. [02:06] Anyway, uni. [02:06] Probably not much we can do now. [02:10] If the builds keep getting re-dispatched, that's a familiar symptom of something... [02:10] ...slaves breaking protocol? [02:10] as you say, uni. [02:10] build farm busy [02:12] rockstar: would a busy build farm do that? That'd be a pretty major and obvious bug. [02:12] (IIRC a job gets marked as in-progress _and committed_ before actual processing starts) [02:13] jtv, well, that was supposed to go to another channel, but yes, it seems the build farm is busy right now. [02:13] rockstar: from what wgrant has been finding out, it appears that some jobs are being eternally re-dispatched [02:13] jtv, oh wait, you thought I was following your context. [02:13] Yes, I'm weird that way [02:13] :-) [02:14] jtv, no, I haven't been reading the backchat. [02:14] rockstar: hey [02:14] rockstar: you home again? [02:14] rockstar: and back to normal? [02:14] jtv... weird... surely not [02:14] preposterous [02:15] thumper, well, "normal" is objective, but yeah, I'm home. [02:15] rockstar: objective or subjective? [02:15] pedantry rules [02:16] thumper: he means that he's normally objective. [02:23] heya rockstar === Ursinha is now known as Ursinha-afk [04:30] buildd-manager is still screwed :( [04:30] spm: No obvious errors in the log? [04:30] orsum. looking. [04:31] It should try to dispatch lots of builds, then encounter some error. Every <1min, probably. [04:32] exceptions.AssertionError: amd64 build of eclipse 3.4.1-0ubuntu2 in ubuntu karmic RELEASE (1213758) can not be built for pocket RELEASE: invalid pocket due to the series status of karmic. ?? [04:33] not every minute, but surely not far off. [04:33] That would do it. [04:34] That's... interesting. [04:34] it's a rather impressive stack trace too [04:35] been going for ages too [04:35] Bug #575165 [04:35] <_mup_> Bug #575165: Buildd-manager erroneously checks COPY archives for release pocket upload permissions when dispatching

[04:36] But that archive is disabled. [04:36] So the build shouldn't be active :/ [04:38] :( [04:42] I guess we could just manually suspend it. [04:42] Why it is not already suspended I do not know. [04:48] thumper: Is the porting of SPRBs from BuildBase to PackageBuild going to happen this cycle? [04:48] wgrant: I believe so [04:48] Excellent. [04:48] wgrant: abentley has it mostly done I think [04:48] he was stuck though on where the builds get set to fully built [04:49] at least that is where it was on my Friday [04:49] he may have solved that now [04:49] Yeah, there's the wonderfully-named 'findBuild' method which actually sets it to fully built... [04:50] WTF? [04:51] that does not sound like a mutating function [04:51] No. [04:51] fix plz? [04:51] Heh. Maybe once the build farm code has settled down and we've removed the four excess tables and several additional classes... [05:04] wow, recipe builds really don't have much priority do they? [05:06] What is their priority? [05:06] I forget. [05:06] Ah, 900. [05:06] Yeah, normal PPA builds are 2510 or so. [05:07] I need to convince everyone that build scores are stupid and need to die. [05:07] i'm just experiencing the "been expected to start in 1 hour for several hours" thing [05:08] Well, that's not due to the score. [05:08] That's due to everything being fucked. [05:08] i see [05:08] The ETA takes the score into account. [05:17] wgrant: you have a +1 from me to kill scores [05:17] but I don't hold much weight there [05:17] what we need is a way to prioritise some builds [05:17] and scores are an approximation for that [05:19] if team==losa then do-losa-builds-first elseif team==ubuntu-security then do security-builds else hahahaha fi ? [05:21] spm: something like that [05:21] thumper: A pretty ineffective approximation. [05:21] mwhudson: I'm deleting most of doc/branch-merge-proposals.txt [05:21] mwhudson: I'm rewriting it to not use sample data [05:21] mwhudson: and in my rewrite I realised it is a big pile of turd [05:21] There seems to be a bit of that going around lately. [05:22] sample data should be avoided in all tests [05:22] whereever possible [05:22] which is almost everywhere [05:22] Yeah. [05:24] thumper: hooray? [05:24] mwhudson: I'm actually writing documentation [05:24] the level of executability in the doc is very small [05:25] I'm trying to focus on the documentation being meaningful [05:36] thumper: !?!? bit excessive isn't it? *meaningful*!?!?! srsly? [05:36] yeah [05:36] what was I thinking [05:37] haha [05:37] * thumper replaces the doctest with "Read the freaking code bitch" [05:38] damn. that's almost too easy to take out of context. I shall pass. this time. [05:38] spm: heh [05:48] lifeless: hi, back in NZ? [05:49] nope [05:49] lifeless: EU still?? [05:49] I fly out from here in 12 hours [05:49] GNU Hackers Meeting was on [05:49] how'd that go? [05:52] pretty good [05:52] I need to write it up but [05:52] thumper: did you try the staging search perf ? [05:52] no [05:52] but I read your writeup [05:52] and commented on the merge proposal [05:53] yeah [05:53] its because of conflation [05:53] thumper: https://bugs.staging.launchpad.net/ubuntu/+filebug [05:53] sadly it failed an obscure teset [05:53] so I need to update some more tests with the same 'it sucks, sorry, fast-first' dialogue. === almaisan-away is now known as al-maisan [05:56] * thumper nods [05:56] thumper: I'm interested in perf for you, on that url. [05:57] 'folk in nz are like folk in china' [05:57] what do you want me to search on? [05:57] whatever you like [05:58] lifeless: "bzr transfers too much data" gave me a response in about 5 seconds, but none of them really relevant [05:59] thumper: item 0 for me is bug 82305 [05:59] <_mup_> Bug #82305: push and pull on bound branches use too much network [05:59] which is pretty relevant [05:59] well... yes [05:59] remember that this is in-ubuntu rather than in-bzr context - its not searching the bzr bug db [05:59] I suppose [06:00] https://bugs.edge.launchpad.net/ubuntu/+filebug [06:00] same search [06:00] doesn't return any bzr bugs [06:00] * thumper has to go collect people [06:01] ciao [06:04] and well production simply won't answer [08:15] good morning [08:23] stub: hi [08:28] Happy Asaha Bucha Day. [08:29] oh nice [08:29] you're on leave for it ? [08:30] Yer. Just ticking off a few things from my todo list first though. [08:30] cool [08:30] well I'm travelling tonight [08:30] I've done an audit of the timeout bugs in malone [08:31] and there are a few which would really benefit from you casting an eye over some query<->index stuff [08:31] if you had time tomorrow to just have an eyeball - just the malone stuff tagged 'timeout' [08:31] Ok. [08:32] theres things like a union plan which takes 11 minutes to run ! [09:01] Morning === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [09:58] * flacoste is away: Gone away for now [10:09] * thumper takes a look at the merge conflict [10:11] thumper, thanks [10:12] thumper, I was about to, since I last touched branch.txt [10:12] jml: it seems to be that we just need to delete that block again [10:12] sound right? [10:12] thumper, haven't got that far yet [10:12] thumper, still pulling db-devel :) [10:12] jml: but in the branch that you merged into devel you just deleted chunks of branch.txt? [10:13] thumper, pretty much. [10:13] (note that it doesn't use branch sampledata any more) [10:13] thumper, yeah, delete it. [10:14] thumper, all of that has been moved to unit tests [10:14] jml: the difference on db-devel is that someone reformatted the lines to the right length [10:15] * thumper deletes [10:18] thumper, thanks. [10:18] thumper, btw, I'm still progressively working through the code tests getting rid of their dependency on branch sampledata [10:18] jml: you should see my merge proposal that is up [10:18] thumper, I did! [10:18] I tried to focus on useful documentation [10:19] rather than a test [10:19] thumper, cool. [10:19] thumper, I'll review that in a moment. [10:19] * thumper leaves the office to drink a cuppa [10:20] adeuring: what does having sec proxies returned from the lpfactory do for us ? [10:21] lifeless: I think the tests should work with the same objects as we use in real code. [10:21] and we should be able to detect for example permission problems [10:21] adeuring: thanks [10:24] jml: yes it is still running, just not under a special layer [10:24] thumper, ahh, thanks. [10:24] gah, my email client is getting my emails :( [10:33] jml: Hope you're feeling better this week because I added you to the MP for the buildd-manager branch last week, it might make you feel sick again. [10:35] bigjools, thanks :) [10:35] jml: I do feel bad and apologise profusely. However, not *too* bad since I had to suffer when changing the damn thing :) [10:36] some of the tests are bloody awful, so I hope you have suggestions on how to make them look better. [10:46] bigjools: can you describe the awfulness ? [10:47] bigjools: like - overly coupled, too sensitive, not sensitive enough, visually long, hidden setup, ? [10:47] lifeless: hideous setup [10:47] see lib/lp/buildmaster/tests/test_manager.py, look for TestSlaveScanner.setUp() [10:48] it stubs out half the code its testing [10:48] and then selectively re-enables it in some tests! [10:48] and it probably falls foul of all those other things you listed [10:50] so, if different tests need different amounts stubbed [10:50] I think I'd tackle that by: [10:50] moving the setup to a helper function [10:51] delete the setUp() on the test class [10:51] at the top of each test call the helper with parameters appropriate to the test being run [10:51] the tests need re-writing, they're rubbish [10:51] it will still be hideous, but less so. [10:51] and it will be a cheap change to make [10:51] sort of - I'd need to work out exactly what each test is depending on [10:52] I would not be surprised if it's masking bugs [11:08] bigjools: Evening. [11:08] Are you aware of the rogue karmic rebuild which is killing everything? [11:09] (or was over the weekend; not sure if it's been fixed in the last few hours) [11:14] wgrant: sigh, thought it had disappeared [11:14] how the *hell* is it even there [11:18] bigjools: Yeah, it should surely have been suspended... [11:21] I wonder if someone pocket-copied it [11:21] bigjools: Within a copy archive? [11:21] Unlikely. [11:21] -> stuff [11:21] how do you know it's a copy archive build? [11:22] I thought it was build 1213758 [11:22] how do you know that? [11:22] I can see it in the logs, but...you? [11:22]