[00:20] thumper: ping [00:20] Bug 667554 is blocking deploys [00:20] <_mup_> Bug #667554: State change and reviewer request emails go out for work in progress

[00:21] thumper: (This is the 'qa promptly...-please-' plea) [00:21] thumper: I would nag wallyworld, but he isn't on irc just now :) [00:23] lifeless: I think we may need to roll back cesium. [00:23] It's knocked out half the non-virt builders. [00:24] 'NoneType' object has no attribute 'processor' [00:24] Wheee [00:25] There's a Twisted callback being called after a failure, when it shouldn't. [00:25] And the builder's cached behaviour is incorrect at the that point. [00:26] A quick workaround may be to remove the behaviour cache. [00:26] But can we just rollback to whatever it was before? [00:26] lifeless: ^ [00:33] I don't have the time nor the log access to debug this much further :/ [00:56] ok [00:57] wgrant: 11811 be ok ? [00:59] soyuz-production-rev-9886 is more likely. that's a softlink change. [00:59] lifeless: No. 11808 is probably the culprit. [00:59] 11801 was the initial landing, it was rolled back in 11805, then brought back in 11808. [00:59] I'd go with spm's suggestion. [00:59] 11793 then [01:00] any other revno isn't so much a rollback, as a rollout [01:00] spm: ah, please do that. [01:00] spm: we should also take cesium back out of nodowntime [01:00] oki [01:00] spm: until this is addressed [01:00] lifeless: where is the imap folder for the qastaging email box? [01:00] perhaps we need a 'lock' facility in the deployment manager [01:00] thumper: its the staging one [01:01] lifeless: how do I tell the difference? [01:01] thumper: right now you don't. I've filed an RT asking for something [01:01] lifeless: ok [01:01] thumper: OTOH there is almost no mail from staging/qastaging. So you can just empty the mbox [01:01] then do your thing [01:01] lifeless: what bzr hosting do we have for qastaging? [01:01] thumper: EPARSE [01:02] lifeless: like pushing branches to qastaging? [01:02] its all there [01:02] lp: won't work until the bzr bug I filed is fixed. [01:02] ok [01:02] but the full paths will [01:03] I should probably do the bzr one myelf [01:05] spm: Once that's rolled back and b-m restarted, the disabled buildds need to be undisabled. [01:05] figures [01:05] spm: thanks [01:05] it's rolled back, fwiw. that was fairly easy. just making sure all teh required services are restarted [01:06] yeah, thats the idea :) [01:06] low TTR [01:09] test_ppa_displayname_inline_edit failing on db-devel builder [01:15] :( [01:19] mwhudson: have you tried mumble recently? [01:19] mwhudson: it has started working for me fine [01:21] thumper: no [01:21] mwhudson: in the end I deleted my ~/.config/Mumble dir and started again [01:22] mwhudson: dropped the caching to 10ms [01:22] mwhudson: and it is working fine now [01:22] thumper: huh, cool [01:22] i should try it again then [01:22] I'm not entirely sure what else has changed, but I've been using it for standups this week [01:22] and for talking with flacoste [01:29] * StevenK peers at the build page. Where's my build log? [01:30] StevenK: Which build is borked? [01:30] wgrant: https://launchpad.net/ubuntu/+source/linux/2.6.37-2.10/+build/2032158 for example [01:31] StevenK: Looks like it was killed by the failure counter. [01:31] Retry. [01:31] Bonus points if you unkill all the builders. [01:32] I set it to OK and remove the failure message? [01:34] Indeedily. [01:35] Whee, adare promptly failed itself again [01:36] That's not unexpected. [01:36] ppc is sort of screwed at the moment. [01:37] At least it has a proper error this time. [01:40] I prefered the other error, it at least pointed to a code error in the buildd-manager :-) [01:41] allspice picked itself back up, though [01:43] wgrant: Have you filed a bug about this, or do I need to? [01:43] StevenK: I haven't. Do you have the traceback and related log entries? [01:44] Not to hand [01:44] http://paste.ubuntu.com/526017/ [01:47] wgrant: bug 671242 [01:47] <_mup_> Bug #671242: New buildd-manager disabling everything in sight [01:47] Good title! [01:48] StevenK: Thanks. [01:49] StevenK: is there a probe failure message immediately before each traceback? [01:49] I see there's one 9 seconds later, but that's probably from the next scan. [01:53] wgrant: After [01:54] :( [01:54] Traceback, and then the probe failure message immediately after [01:56] StevenK: Is there a "builder BLAH failure count: X" line before the traceback? [01:58] My theory is that assessFailureCounts knocks the build off the builder, then rescueIfLost fires in the same transaction, using the cached BFJB. [02:01] wgrant: That's attached to the failed probe messages, isn't it? [02:01] 2010-11-05 00:25:45+0000 [QueryProtocol,client] Builder lakoocha failed a probe, count: 4 [02:02] StevenK: The 'failed a probe' message occurs when a failure is detected with no build assigned. [02:03] 'builder BLAH failure count: X, job 'BLAH' failure count: Y' is used when there is a build assigned. [02:04] wgrant: I can see that in the logs for rothera, but it's a timeout error, not the processor none traceback [02:04] Intriguing. [02:06] Anyway, I guess we'll have to wait until Monday evening. [02:06] wgrant: I wonder if there's an addErrBack for rescueIfLost [02:08] StevenK: rescueIfLost isn't the problem here. [02:08] Its caller is. [02:09] It is being called with a dirty transaction with a builder with an outdated BFJB cache. [02:09] There are enough commits and aborts around that that shouldn't happen. [02:09] But there must be a missed case. [02:10] But I am not exactly a Twisted expert. [02:10] Neither [02:11] Maybe I should acquire a book. [02:13] StevenK: I hope there is lots of lovely logging for the new buildd manager [02:13] Not enough :( [02:13] But it's not as bad as it ws. [02:14] well add some dear liza... [02:14] has the buildd manager been rolled back? [02:14] wgrant: http://www.amazon.com/Twisted-Network-Programming-Essentials-Fettig/dp/0596100329/ref=sr_1_7?s=books&ie=UTF8&qid=1288923189&sr=1-7 ? [02:14] thumper: Yes [02:15] StevenK: can I talk to you in about 30 minutes about PPAs? [02:16] thumper: Sure [02:16] ta [02:17] thumper: As I see it, we have a bit of a conflict. Code likes to delete things, while the rest of Launchpad does not delete anything ever. [02:17] wgrant: sure it does... [02:17] Oh? [02:17] blueprints will be deletable soon [02:18] :-) [02:18] I hope you mean the code. [02:18] we have lots of garbo scripts that delete things [02:18] and no, I don't mean the code [02:18] They delete records of inconsequential happenings. [02:18] hahaha [02:18] Not OMGCRITICAL stuff like who uploaded some malware to Ubuntu. [02:19] * spm prints " They delete records of inconsequential happenings." to preserve for all time [02:37] Hard / Soft Page ID [02:37] 278 / 7 Archive:EntryResource:getBuildSummariesForSourceIds [02:37] 207 / 55 Person:+commentedbugs [02:37] 94 / 0 MailingListApplication:MailingListAPIView [02:37] 90 / 375 BugTask:+index [02:37] 83 / 269 CodeImportSchedulerApplication:CodeImportSchedulerAPI [02:37] 34 / 2190 Archive:+index [02:37] 32 / 138 Archive:+packages [02:37] 19 / 323 Distribution:+bugs [02:37] 18 / 14 ProjectGroup:+milestones [02:37] 11 / 82 MaloneApplication:+bugs [02:39] lifeless: Are they timeouts? [02:41] Hm, we only hold three of the top ten, and two of those are probably the same issue. Not bad. [02:42] StevenK: mumble? [02:43] thumper: Sec [02:54] Yippie, build fixed! [02:54] Project db-devel build (120): FIXED in 3 hr 9 min: https://hudson.wedontsleep.org/job/db-devel/120/ [02:54] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11862 [02:54] included. [03:04] lifeless: Interestingly, the deployment page says bad, but the bug is marked 'qa-untestable' [03:17] thumper: Are you okay to mentor my review of https://code.edge.launchpad.net/~wallyworld/launchpad/person-mergequeue-listview/+merge/39933 or would you prefer someone els [03:17] s/els$/else?/ [03:21] I can take a look [03:57] StevenK: which bug [04:00] spm: can you please add a feature flag to qastaging for qa needs? [04:00] hard_timeout team:bzr 15 5000 [04:03] lifeless: thatshould be applied [04:23] spm: can you drop that to 2000 ? [04:24] done [04:24] great [04:24] please remove the rule [04:25] done [04:29] thanks [04:37] Is PQM happy at the moment, and likely to remain so? It threw out https://code.edge.launchpad.net/~wgrant/launchpad/destroy-distroseries-lucilleconfig/+merge/38648 yesterday. [04:37] wgrant: we can't actually tell [04:37] ... [04:37] there is a bug [04:37] Hwhat? [04:38] the testfix state is not well exposed [04:38] last build of devel workd [04:38] last build of db-devel failed [04:38] Yay. [04:38] devel and db-devel are both building atm [04:38] WindmillTestClientException: {u'suite_name': u'lp.soyuz.windmill.tests.test_ppainlineedit', u'result': False, u'starttime': u'2010-10-5T6:12:5.320Z', u'params': {u'xpath': u"//h1[@id='displayname']/span[1]", u'validator': u"Celso's default PPA"}, u'debug': u"Text 'Celso's default PPA' was not found in the provided node. Found instead: PPA for Celso Providelo", u'output': None, u'endtime': u'2010-10-5T6:12:5.321Z', u'method': u' [04:38] u'validator': u"Celso's default PPA"}, u'debug': u"Text 'Celso's default PPA' was not found in the provided node. Found instead: PPA for Celso Providelo", u'output': None, u'endtime': u'2010-10-5T6:12:5.321Z', [04:38] u'method': u'asserts.assertText'} [04:39] looks like a genuine fail [04:39] StevenK: ^ [04:39] Hudson's still happy, though :/ [04:40] So not very legit. [05:05] That's ... odd [05:36] lifeless: was it you that pointed out mod_pagespeed? [05:41] no [05:41] I was the one that went eeeeek omg nooo [05:41] [just on the probably complexity of the heuristics it must have] [05:42] let alone the potential to mess with our API implementation [05:42] SpamapS: ^ [05:42] so I filed an ITP w/ debian to package it.. [05:42] and have started looking into it [05:43] 45 dependencies.. I think 3 of them are already packaged [05:43] they build a .deb, but they don't use dpkg-buildpackage to do it [05:43] -lol- [05:43] I'm very unsurprised. [05:43] Google project yeah? [05:43] the first thing it does, is sync all of its 3rd party dependencies from their upstream repositories at specific revisions. [05:44] yes, Google seems to be against releasing and stablizing things. ;) [05:44] well [05:44] I have to wonder, are we fighting the good fight, or holding back progress? [05:44] its a different definition of release ;) [05:45] TODO: describe advanced compilation options such as compiling against system headers. [05:45] SpamapS: *blink* [05:45] am I the only one who thinks thats *BASIC* compilation? [05:45] I hear ya [05:46] I can see how the conversation went at Google that led to gyp and gclient (the things that make this "just work") [05:47] "Its got 45 dependencies, it takes a long time to get them all onto one system." [05:47] "I can solve that!" [05:47] "Ok, solved. We can now compile all 45 dependencies with one command. Who wants to go play ping pong?" [05:47] they have a specific diagnostic story [05:47] if you look at go it becomes very clear [05:47] Well... [05:47] they -want- a single binary with no shared libraries [05:48] Have you seen Chromium? [05:48] It is the Java mentality cubed. [05:48] Yes, this uses the chromium tools [05:48] the reason they want this is so that when there is a fault, they know all the deps precisely with no room for skew [05:48] and for complete isolation between concurrently installed things on the same machine [05:49] Yes, but it makes everyone else sick. [05:49] I don't completely buy the value of this - apport does a pretty tremendous job [05:49] ridiculous.. they're including libjpeg, libpng, zlib [05:49] things that have been on every system for a long, long time. [05:50] yes [05:50] its not because they have to [05:50] its because they want to [05:50] the numeric analysis is something like: [05:50] They *think* they want to. [05:50] - if we get 10billion downloads [05:51] - we'll save 5 hours diagnosis due to folk with zlib1e which had a buffer overflow [05:51] wgrant: they want to. No thinking involved. [05:51] They're thinking in faster moving terms than most people actually move. Its valid for desktop software.. people see "there are updates" and they tend to update. Not so much for the server side. [05:51] no [05:51] its not about how fast things move [05:51] its about reproducability and support overhead [05:52] Which is funny, thats why we do what we do. ;) [05:52] I got into this fairly deeply @ LCA last year [05:52] But they're doing it by assembling a mini-distro around their product. [05:52] none of it attributable, of course [05:52] which is actually totally awesome.. if you, the user, trust them to keep all those deps in top shape. [05:55] lifeless: we didn't really solve this at UDS.. too many distractions.. its the same old problem. [05:55] SpamapS: Does the solution actually need discussion? [05:56] I don't think there's any option but to declare their strategy as being utterly crackful, ignore it, and port their build system to use real system libraries. [05:58] wgrant: 45 is a drop in the bucket. Hudson has over 1000 java libraries that aren't packaged. This keeps coming up... upstreams don't build on top of distro's, they build on top of libraries. [05:59] SpamapS: But Hudson at least doesn't bundle modified copies of its libraries, right? [05:59] modified, no, selected versions, yes absolutely. [06:00] to me, these upstreams have done a lot of our work for us and we throw it away because we only want to have one version of everything to support. [06:03] Surely Hudson isn't 1,000? 200, 300, sure, but 1k? [06:03] Doing a Launchpad and specifying a known-good set of unpackaged versions is one thing. [06:04] But bundling libraries... uh... kill them. [06:06] Didn't dpkg used to statically link against zlib in the deep dark ages? [06:24] lifeless: Can you remind me of a discussion we had at UDS? You mentioned using btrfs on the Hudson build slaves -- that one I remember, but there was another suggestion [06:57] StevenK: talk to JamesPage about it.. 1000 was the number of libs hudson needed to download from maven to build *after* it cross referenced with everything debian had packaged. [06:58] SpamapS: well I've argued (successfully I think) that we need N libraries concurrently for java and python [06:59] StevenK: errr I didn't suggest btrfs on the build slaves. [06:59] StevenK: I suggested an RT ticket asking for bigger images. [06:59] StevenK: our you can roll your own [07:01] lifeless: Actually, they're 20GiB, just the / is only 1.5GiB [07:02] StevenK: yes, but thats the partition you need bigger to install java etc, right ? [07:02] Currently it fits with about 50MiB to spare [07:02] EMEEP [07:03] Current devel slave has 120MiB, that will drop to roughly 50 over the course of the build [07:05] lifeless: Too close, and I should repack? [07:06] dunno about you but my sysadmin senses are screaming [07:07] * StevenK ponders an EBS store for persistant branches [07:16] steven@hudson:/tmp$ ls -1 | wc -l [07:16] 3589 [07:16] * StevenK sobs [07:54] lp.soyuz.windmill.tests.test_ppainlineedit fails on db-devel again ... How odd [07:55] Huh. [07:55] I don't recall that failing within the last few months. [07:56] And has now failed twice in 2 builds, but not in devel [07:58] And not in Hudson. [08:04] Indeed [08:04] Sounds like a reason to switch to me [08:04] :-P [08:12] hi wgrant, you still around? [08:14] bac: Hi. [08:14] wgrant: you know much about the inner workings of merge proposals? [08:14] bac: Not a huge amount, but possibly enough. [08:14] wgrant: i've noticed rollback branches don't get their status updated to 'merged' [08:15] and they clutter up +activereviews until cleaned up [08:15] Do you have a current example? [08:15] https://code.launchpad.net/~bac/launchpad/rollback-11801/+merge/39422 [08:15] bac: uhm [08:15] wgrant: unfortunately i spoiled it by updating the status manually [08:15] bac: that means one of two things [08:15] bac: it wasn't actually landed previous [08:16] bac: or the scanner glitched at the same time [08:16] https://code.launchpad.net/~mars/launchpad/rollback-11586-and-11588/+merge/36212 [08:16] bac: launchpad itself doesn't know about the rollback stuff [08:16] lifeless: well, there is another [08:16] Hm. [08:16] i know the first actually landed [08:16] i haven't checked on the second [08:16] https://code.launchpad.net/~bac/launchpad/rollback-11801 [08:16] It's being scanned. [08:16] Which is a little odd. [08:16] ah, so it is [08:16] for 10 days now [08:17] lifeless: Interestingly, the deployment page says bad for r11825, but the bug is marked 'qa-untestable' [08:17] see under scanner glitch [08:17] score one for lifeless [08:17] Indeed. [08:17] StevenK: https://bugs.edge.launchpad.net/qa-tagger/+bug/670792 [08:17] <_mup_> Bug #670792: handling of bad-commit- tags needs improving [08:17] These non-edge URLs are strange and foreign :( [08:17] Haha [08:17] Blame lifeless [08:18] wgrant: can i do anything to kick the scanner for that branch? [08:18] StevenK: its bad because of the bad-commit-REVNO tag (which is how we mark that 11825 was faulty) [08:18] bac: Not sure. [08:18] I don't really know how the scanner works these days. [08:19] wgrant: if it is a one-off i'm not concerned. but i saw two that appeared to be in the same situation [08:19] That is indeed concerning. [08:20] lifeless: i see your comment on the MP [08:20] lifeless: i didn't just now land it, i did it the day of the problem [08:20] just updated the status of the MP today [08:21] grah [08:21] ok [08:21] I wouldn't do that [08:21] what would you do? [08:21] because the metadata will be faulty about the rev it was merged into [08:21] I would look to see why it hadn't updated and get that fixed ;) [08:21] mark the MP rejected? [08:23] no [08:23] so, when you look at the branch and its scanning [08:23] thats an issue [08:23] look into that [08:23] lifeless: i did not see the branch was scanning [08:23] when its fixed, lp will mark the mp appropriately. [08:23] bac: right, I know :). I'm saying what I would have done was to look around for clues. [08:24] one of the places I'd look would be the branch page. [08:26] lifeless: so would resetting the MP back to 'approved' be inappropriate now? [08:27] bac: if the branch is still scanning, setting it back to approved is probably best. [08:27] OTOH [08:28] and then trying to get that scanner process kicked... [08:28] the main thing I'm concerned about is figuring out what happened, ensuring there is a bug that when fixed will prevent it happening again. [08:30] lifeless: ok, i'll change the MP back to 'approved' and open a bug referencing that branch but not request anything be done about the stuck scanner. thanks. [08:30] I'm not sure that it will be marked merged once it does scan, since the rev won't be new in devel. But I guess we'll see. [08:31] wgrant, lifeless: fwiw the other example i cited is a rollback branch that never landed so it was a bad example. [08:31] what pages does the recipe ui stuff turn up on ? [08:32] lifeless: Every branch page. [08:32] time to see if the flag really works [08:32] lifeless: There's a "Related source package recipes" section just below the "Branch merges" section. [08:46] ok [08:46] er === jtv1 is now known as jtv [08:48] good morning [08:54] hi adeuring [08:54] hi bac! [09:00] Morning [09:10] hi mrevell [09:11] hey bac, welcome back, heh [09:21] hello [09:22] jml: your scopes rules are present in production; both patches are in stable and qa'd - we just need to qa up to them and they will go live. [09:25] lifeless: sweet. [09:25] lifeless: I guess I need to populate the team, delete the edge rule and do comms [09:25] anything else? [09:28] yes. Synced with the revs being deployed of course. [09:29] we probably want to do a beta program in-app too [09:29] a-la gmail labs [09:30] lifeless: "here are our beta features, sign up to them?" [09:30] yeah [09:31] lifeless: I think that would be an excellent thing. [09:31] Are we going to have a lot of features for which that would be relevant, though? [09:31] wgrant: we have 3 concurrent at the moment [09:32] wgrant: I'm guessing so. [09:32] Do we? [09:32] wgrant: recipes, merge queues, subscribe-to-search [09:33] Hm, true. [09:34] oh and derived-series ui [09:35] lifeless: https://code.edge.launchpad.net/~lifeless/launchpad/databasefixture/+merge/38694 – what's the actual state of that MP? [09:35] jml: in principle landable [09:36] it bounced with some glitches that I think I've fixed. [09:36] if you want to give it a spin, please do. [09:37] lifeless: ok. I probably won't today. [09:48] * jml away for a bit === Guest26427 is now known as jelmer [11:06] Morning, all. === matsubara-afk is now known as matsubara [11:30] wgrant: Argh, it looks like your branch is cursed or something. [11:33] jelmer: Yeah, buildbot seems to have a failure that Hudson does not. [11:37] Still, this is just two failures so far. One of my previous batch took 10 attempts. So not that bad. [12:09] good morning [12:15] hi mars [13:03] how is it possible that merely importing a TestCase subclass that defines 'layer = None' causes the Zope testrunner to barf? === mrevell is now known as mrevell-lunch [13:09] mars: importing a TestCase makes it discoverabl [13:09] e [13:10] jml, that appears so. Now I'm trying to figure out how other people actually manage to use the subclass without it blowing up on import [13:15] 'bin/py -m pdb bin/test lp.testing.tests' at least gives me a debugger [13:24] do we have a way to mass add people to a team? [13:25] jml: why are you creating a new team for recipes instead of making them available to all members of the beta team as we have until now? [13:28] jml: I don't know if it exposes adding people to teams, but a launchpadlib script suggests itself to me [13:28] abentley: it seems a good way to try having a beta programme specific to a feature [13:28] benji: yeah, that would work. lots of roundtrips, but it would work & probably be faster than manual data entry. [13:29] jml: I don't think we should take the feature away from people who already had the option of using it, even if they hadn't used it yet. [13:29] jml: So please make the beta team a member of the recipe team :-) [13:33] the real spanner-in-the-works is that we don't yet have a way in-app of inviting people to join the beta [13:35] jml, that means we have a chance to do it well [13:35] jml, btw, turns out I had to put a 'del YUIUnitTestCase' at the end of the file, so the testrunner would not discover the class. [13:36] mars: is that what other modules do? [13:36] jml: That is a serious deficiency. People can join the team if they want, I suppose, but the requirement to do so will be new and surprising. [13:36] abentley: well, only surprising if they don't read the blog. [13:37] jml: Who reads the blog? [13:37] abentley: I think Francis does [13:37]