[00:14] wgrant, https://edge.launchpad.net/builders [00:17] StevenK: ^^ [00:19] fuuuuuuuuuu [00:19] That was meant to be fixed. [00:20] And we can't rollback after tomorrow. :/ [00:20] spm: Around? [00:20] we can cherrypick rollback [00:20] he doesn't appear to be [00:20] what do you need? [00:20] We have no StevenK this week, AIUI. [00:21] The new buildd-manager is still horribly broken. Can you see if there's anything interesting in its log? [00:21] yes, all sorts of crfap [00:24] Well, that's relatively more opaque than I would have hoped. [00:24] are we running the new buildmanager then? [00:24] We are. [00:25] wgrant: are we looking for anything in particular in the buildd manager log? [00:25] pjdc: I have a section of it. It's not very helpful. [00:25] any idea where the log is on devpad? [00:25] wgrant: do you know which machine the buildd manager runs on? [00:26] thumper: cesium [00:26] I have bits of the log already. [00:26] thumper: looks like they land in devpad:/x/launchpad.net-logs/production/cesium [00:26] pjdc: yep, there [00:27] Twisted errors make me sad. [00:28] I wonder if the fermium connection error at 2010-11-17 00:08:13 is the root. [00:28] Oh, no. [00:28] It was a few minutes before that. [00:28] Sad. [00:29] Can you see where it started? [00:29] pjdc: There have been no known network glitches this morning? [00:30] wgrant: not to my knowledge. [00:30] :( [00:31] 010-11-16 00:01:03+0000 [-] communication failed (User timeout caused connection failure.) [00:31] 2010-11-16 00:01:03+0000 [-] failure (None) [00:31] 2010-11-16 00:01:03+0000 [-] communication failed (User timeout caused connection failure.) [00:31] 2010-11-16 00:01:03+0000 [-] failure (None) [00:31] 2010-11-16 00:01:03+0000 [-] communication failed (User timeout caused connection failure.) [00:31] hmm, i could check if fermium was airlocked around the [00:31] 2010-11-16 00:01:03+0000 [-] failure (None) [00:31] 2010-11-16 00:01:03+0000 [-] communication failed (User timeout caused connection failure.) [00:31] 2010-11-16 00:01:03+0000 [-] failure (None) [00:31] seeing things like that [00:31] hmm... [00:31] perhaps a pastebin would be better [00:31] thumper: Isn't that >24 hours ago? [00:31] wgrant: yep, I'm looking for failures [00:34] hmm... [00:34] buildd-manager.log-20101116 has up to 2010-11-16 00:01:05+0000 [00:35] Yeah, they're named sort of wrongly. [00:35] buildd-manager.log starts at 2010-11-17 00:00:45+0000 [00:35] They're named after the day they're rotated. [00:35] and no log file in the middle [00:35] There's no -20101117? [00:35] nope [00:35] Um. [00:36] pjdc: can you see a -20101117 file on cesium? [00:36] thumper: looking [00:36] thumper: yes, there's a buildd-manager.log-20101117 [00:36] pjdc: can you get that to devpad plz? [00:37] according to the graph, we started loosing builders approx 12 hours ago [00:38] Does LPS say when cesium was updated? [00:38] thumper: landed in /tmp [00:39] pjdc: ta [00:40] Scanning failed with: <-- look suspect [00:40] That's fairly normal. [00:40] Unhandled error in Deferred: [00:40] That's not. [00:40] that's probably not [00:41] 2010-11-16 23:51:34+0000 [-] builder promethium failure count: 5, job 'amd64 build of widelands 1:15bzr5723-ppa1~natty1 in ubuntu natty RELEASE' failure count: 1 [00:41] 2010-11-16 23:51:34+0000 [-] Scanning failed with: User timeout caused connection failure. [00:41] There are hundreds of those. [00:41] And that's not the first. [00:42] Failure: twisted.internet.defer.CancelledError: [00:42] ?? [00:42] no [00:42] it isn't the first [00:42] I'm looking for suspect lines [00:43] is there a log entry for disabling a buider? [00:44] There used to be. [00:44] I never got around to reviewing the full 6000 line diff of this branch, though :/ [00:44] * wgrant looks. [00:45] Miserable failure when trying to examine failure counts: :-) [00:45] .... wow. [00:45] except: [00:45] GRGWOGIFJEWF [00:46] Anyway, what was the miserable failure? [00:47] thumper: You'll be pleased to know that not all failBuilder callsites log. [00:47] wgrant: I'm grepping [00:47] wgrant: I can kinda tell [00:47] I suppose we should do a launchpad status thingy [00:47] But we should be able to tell from the logs. [00:48] Since only two of the callsites are unobvious, and one has enough logging around it that we should be able to work it out. [00:48] wgrant: can you see devpad? [00:48] thumper: Not for another few weeks :( [00:48] just wondering, I'll copy the logfile to people.c.c [00:49] Thanks. [00:50] hmm... [00:50] I'm getting refused ssh [00:50] That's excellent. [00:52] hmm... something screwy is going on... [00:52] ok, it's on its way up [00:53] wgrant: I'm not entirely sure what to look for [00:53] thumper: Thanks. [00:53] * wgrant examines. [00:54] It's tempting to add more explicit logging, restart it, and hope it breaks again. [00:54] test rollout of 11926 on 2010-11-16 [00:55] that's from LPS [00:55] I was hoping for higher granularity. But I guess the log should help with that. [00:55] bug https://bugs.launchpad.net/soyuz/+bug/671242 [00:55] <_mup_> Bug #671242: New buildd-manager disabling everything in sight [00:55] 10:03, by the look of things. [00:56] thumper: Right, 11888 was deployed, but that broke with that bug. [00:56] 11926 fixed that. [00:56] But we now have another one. [00:56] this is a different problem? [00:57] Yes. [00:57] I believe. === thumper changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: buildd-manager is disabling things again | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting [00:57] I think this problem may be partially described in that bug, but it's not the one that was identified and fixed. [01:03] So, we have major problems at 14:41:45, 15:03:20 and 16:31:22, 23:39:11, and possibly 18:11:22, 19:52:18, [01:03] 23:39:11 is the big one which took out everything. [01:04] Each major failure starts with a single scan failure, than a huge number 9 seconds later. [01:04] elmo: ping? [01:05] He left a while ago. [01:05] :( [01:05] I think we are losaless [01:05] pjdc should be around, though? [01:05] pjdc: what do you know about LP deployment? [01:06] If you want to revert, we don't need a full deployment. It's just a symlink change and restart of buildd-manager. [01:06] thumper: not much. i assisted elmo with an emergency cowboy about a year ago, but that's about it. [01:06] :( [01:06] lifeless: are you really there? [01:06] pjdc: on cesium there was a rollout yesterday [01:06] pjdc: I'm hoping that they kept the old code around [01:07] thumper: Last time that was the case. [01:07] pjdc: as the new code is disabling all the builders [01:07] thumper: A LOSA just flipped the symlink back to the 10.10 rollout. [01:07] soyuz-production-rev-9886 [01:07] wgrant: if the buildd-manager is restarted, will it recheck the disabled buiders? [01:07] thumper: if i'm looking in the right place, there are two tree, 11926 and 9886 [01:07] thumper: yes [01:07] thumper: 9886 looks pretty old [01:08] pjdc: 11926 is the broken one [01:08] pjdc: 9886 will be from db-devel [01:08] so... probably the last rollout [01:08] Wed 13th of Oct [01:08] that's the date on rev 9886 of db-devel [01:09] thumper: No, we'll have to flip a flag on each to get them back. [01:09] 9886 is what we reverted to after 11888 failed. It's the last rollout. [01:09] pjdc: can you do that? [01:10] wgrant: how do we re-enable the builders? [01:11] thumper: so that'd be change the symlink, restart the service? [01:11] pjdc: AFAIK [01:11] thumper: i take it things can't get worse at this point? [01:11] thumper: again? nooo [01:11] this bodes badly for the rollout tomorrow. [01:11] like really. [01:11] thumper: 9886 is fine, its the last db-stable deploy [01:11] wgrant: you got log files etc - whats up ? [01:11] it was stable for hours - did this happen recently? could it be librarian changes? [hope not] [01:11] . [01:11] Yeah, cesium is as broken as it can be. [01:12] lifeless: I have the log change. [01:12] s/change// [01:12] whats causing this [01:12] before we change stuff [01:12] It was mostly OK for 4 hours. [01:12] After 13 hours it just completely melted down. [01:13] Still looking to see if I can get anything useful from the logs. [01:13] can we just restart it in the meantime and toggle the builds back on ? [01:13] thumper: The 'Builder OK' flag on Builder:+edit does it. Otherwise there might be a script around. [01:13] or will it kill them immediately ? [01:13] lifeless: I guess we could try that. [01:13] we need to figure out what happens tomorrow @ rollout time [01:14] which is what, 8 hours away [01:14] lifeless: right now it is a release blocker IMO [01:14] I'm starting an incident report [01:14] lifeless: I think we probably cherrypick 11808, 11815 and 11926 off cesium. [01:14] wgrant: 11926 itself is a problem ? [01:15] lifeless: No, but 11808 probably won't revert unless we revert 11926 first. [01:15] Er. [01:15] Not 11926. [01:15] That other one. [01:15] sorry, I left my mind reader in asia [01:16] The fix for the issue that caused us to roll cesium back from 11888. [01:16] 11898 [01:17] So 11808, 11815 and 11898. [01:19] Do we know if enablement has pulled any buildds today? [01:19] cody-somerville: ^ [01:19] wgrant: it's been quiet since the 12th, as far as i can tell [01:20] It doesn't explain everything (since non-virt buildds had the same error), but it might be something. [01:20] wgrant: whats the error [01:20] lifeless: 2010-11-16 16:31:13+0000 [-] Scanning failed with: User timeout caused connection failure. [01:20] 2010-11-16 16:31:13+0000 [-] Traceback (most recent call last): [01:20] 2010-11-16 16:31:13+0000 [-] Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. [01:21] lifeless: In most of the major failures in the log, there is one of those followed by dozens 9 seconds later. [01:22] lifeless: do you object to rolling back the buildd-manager code on cesium? [01:22] Perhaps we should try enabling things and see if they stay alive for long. [01:23] does the buildd-manager still do blocking things? [01:23] thumper: I want to be sure we understand it [01:23] Only when downloading files from slaves, I believe. [01:23] mwhudson: I believe it is twisted now [01:23] mwhudson: ^^ [01:23] pjdc: can you please: [01:23] wgrant: I thought that jelmer fixed that [01:24] thumper: fully? it's been somewhat twisted for a long time [01:24] * thumper needs food [01:24] - restart the builddmanager [01:24] thumper: jelmer fixed it so it uploads the downloaded files asynchronously. [01:24] - reenable a couple of fast buildds [01:24] - see what happens over a few minutes [01:24] thumper: A branch is coming to download them async, too, but it's not done yet. [01:24] thumper: go eat, nothing will change radically while you eat [01:24] i'm not too familiar with the buildd pool. can someone suggest candidates? [01:25] * pjdc picks 3 amd64 official builders [01:25] pjdc: A fairly random pick of the various categories: roseapple, allspice, doubah, samarium [01:26] Couple of new non-virt, and an old and new virt. [01:26] works for me [01:26] restarting buildd-manager [01:27] started, doing the buildds now [01:28] Hm. [01:28] Maybe we should turn the logging up. [01:28] (lib/lp/buildmaster/manager.py, s/logging.INFO/logging.DEBUG/) [01:28] re-enabled those four, plus yellow and crested since i had the tabs all ready [01:29] Great. [01:29] There are some odd five minute gaps in the log, and it would be nice to know if it actually does anything in them. [01:42] pjdc: the queue for amd64 is empty though [01:42] not sure if that'll show much [01:43] thumper: looks like doubah's done the business though, showing as disabled again [01:43] The builders were failed regardless of whether there was anything to build or not. [01:43] Oh, already? [01:43] oh ok [01:44] wgrant: does the buildd manager *read* from the librarian ? [01:44] lifeless: I don't think so. [01:44] I can't think why it would. [01:44] do builders ? [01:44] Yes. [01:44] what code path do they use to get their urls ? [01:45] Ahh. cesium provides them, I believe. [01:45] But it doesn't use the restricted librarian. [01:45] even for security builds etc? [01:45] No -- private build files are retrieved from the archive. [01:46] Since the builders can't have restricted librarian access. [01:46] (well, I guess they could now) [01:46] just to be sure [01:46] pjdc: are we seeing access denied for 91.189.89.189 or 91.189.89.188 from the builders that fail (or from cesium for that matter) [01:46] wgrant: what time was the first builder disabled ? [01:47] lifeless: Can't tell. But the first major incident was probably 14:41:45. 23:39:11 was the really big one. [01:47] lifeless: cesium can't connect to those IPs on 80 and 443 [01:48] oh [01:48] is this the restricted librarian [01:48] pjdc: ah, but is it *trying* [01:48] aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa [01:48] god damn it [01:48] lifeless: i'll check [01:48] elmo: there are two code paths for it in lp. internal stuff like the merge proposal diff code will use lp.internal still [01:48] cesium will be wanting to upload to it. Is that also going to be broken? [01:49] 188 IN PTR wildcard-restricted-launchpadlibrarian-net.banana.canonical.com. [01:49] elmo: I really really wouldn't expect this to be connected, but Just In Case. [01:49] wgrant: no, uploads have not altered *at all* [01:49] That's what I thought. [01:49] wgrant: are those times UTC? [01:49] lifeless: Yes. [01:50] ok, so 9 hours apart. [01:50] There were some in between. [01:50] (range of) unless we're seeing two different things [01:50] then its not the publicrestricted librarian work [01:50] When did that happen? [01:50] lifeless: a few rejects (5 total) for cesium going to both IPs on 80 and 443 [01:50] wgrant: and really? sed to change log levels ? [01:51] lifeless: Yup :D [01:51] pjdc: thats -very- interesting [01:51] wgrant: I can has bug, and fix. [01:51] elmo: did you just remove the publicrestricted feature flag? [01:52] lifeless: sorry, false alarm. those were my tests attempts. [01:52] elmo: from https://launchpad.net/+feature-rules [01:52] lifeless: I haven't touched anything [01:52] interesting [01:52] cause the setting is gone ;) [01:53] Hmm. [01:53] It's possible that translations jobs might read from the librarian. [01:53] I don't know them well. [01:53] * wgrant looks. [01:53] But they wouldn't be private. [01:55] pjdc: did you? [01:55] nb we really need that audit log [01:55] lifeless: sorry, did i what? [01:56] sounds like a no to me [01:57] ok, one thing at a time [01:57] uhm [01:57] We should probably have an 'assert not libraryfilealias.restricted' in BuilderSlave.cacheFile. [01:57] pjdc: please reenable doubah again. [01:57] But I doubt that's the problem here. [01:58] lifeless: ok [01:58] wgrant: or enable it [01:58] * thumper has to go get kids [01:58] lifeless: Hm> [01:58] pjdc: if this fails, its not the publicrestricted librarian. [01:58] lifeless: doubah re-enabled [01:58] https://launchpad.net/ubuntu/+source/fglrx-installer/2:8.780-0ubuntu3/+build/2049941 <-- roseapple worked (for one build) - did we know that? [01:58] Yeah, most things work for a while. [01:58] ok [01:58] It's not related to what the builder is doing at the time. [01:59] It may be related to what others are doing, but who knows. [01:59] wgrant: how do we know that? [01:59] lifeless: Because it affects dozens of builders at a time, whether they're idle or building recipes or building binaries. [01:59] doubah's gone. [01:59] wgrant: what if the timeout or some such is pseudo global, and one hung builder breaks all the ones open for the time window involved [02:00] lifeless: Exactly. [02:00] 12:59:04 < wgrant> It may be related to what others are doing, but who knows. [02:00] So, doubah is dead with a TCP timeout. [02:00] wgrant: in which case its related to what *one* does [02:00] I wish we had a traceback. [02:00] It would be sorta helpful to know what timed out. [02:04] lifeless: buildd-manager won't have cached the old FF? [02:05] wgrant: it wasn't restarted when the problem happened [02:05] True. [02:05] Can we disable everything, enable doubah, and see what happens? [02:06] wgrant: it was 3 hours ago now that the ff was turned on (and apparently off again) [02:06] Oh, so ages after the world exploded. I see. [02:07] wgrant: yeah, I'm convinced we're clear [02:07] lifeless: Chex turned the feature flag off after i complained that private codebrowse wasn't working [02:07] mwhudson: did that make it work? [02:08] (which seems entirely unrelated to me, but after he did it, private codebrowse started working again) [02:08] wtf [02:08] yes [02:08] codebrowse uses the librarian? [02:08] i'm betting some kind of coincidence [02:08] pjdc: can you please turn the flag on again - its listed under sql queries etc on LPS. [02:09] lifeless: that's a long page. what am i looking for exactly? [02:10] publicrestrictedlibrarian default 0 on [02:11] that doesn't mean much to me. is that a command? [02:11] its a line you put in https://launchpad.net/+feature-rules [02:11] https://dev.launchpad.net/LEP/FeatureFlags has plenty of docs - see the bottom o the page in partiuclar [02:12] ah, ok. so bung it in at the end, hit "Change"? [02:12] yes [02:12] done [02:13] mwhudson: still working ? [02:13] lifeless: will check [02:13] 'Server denied check_authentication' is what you saw? [02:13] lifeless: yes [02:14] zomg [02:14] pjdc: and remove it? [02:14] lifeless: it works now [02:14] oh [02:14] pjdc: don't remove it [02:14] lifeless: oh, it failed for you? [02:14] lifeless: ok, doing nothing :) [02:15] mwhudson: failed once [02:15] mwhudson: worked on second url [02:15] I think its stubs openid change [02:15] lifeless: random [02:15] mwhudson: so coincidence that that was the first private branch url you tried since 11926 was deploy. [02:16] pjdc: thanks [02:16] ok [02:16] so back to the buildd [02:16] pjdc: did doubah do day carooba? [02:16] So, I'd like to see this happen: [02:17] - Disable all builders. [02:17] - Shut down buildd-manager. [02:17] lifeless: you lost me at "do [02:17] - Change log leve. [02:17] " [02:17] - Enable doubah. [02:17] - Start buildd-manager [02:17] pjdc: you reenabled doubah [02:17] pjdc: did it die again? [02:17] It did. [02:18] ok [02:18] pjdc: could you do what wgrant just described [02:18] disable all builders incluing those currently building? or just the idle ones? [02:19] All, ideally. [02:19] all [02:19] we're wondering if doubah is broken [02:19] the alls have it [02:19] and a bug is making all the others get nuked when it goes *if* they happen to be lined up with it in the polling period [02:19] Well, I'm mostly hoping we can get a minimal case to fail. [02:19] I doubt there's anything wrong with doubah. [02:19] ok. I'm thinking that. [02:20] what is doubah - virt i386? [02:20] Since I picked four semi-randomly from a pool of 60. [02:20] Yeah. [02:20] A fairly beefy one, too. [02:22] all disabled, stopping the buildd-manager [02:22] wgrant: how is the log level changed? [02:23] pjdc: Aherm. [02:23] pjdc: s/logging.INFO/logging.DEBUG/ in lib/lp/buildmaster/manager.py. [02:23] wgrant: you *are* going to fix that. [02:24] lifeless: It is Twisted evil. [02:24] Which I don't know awfully well. [02:24] hasn't stopped you in the past [02:25] True. [02:25] wgrant: like so? http://paste.ubuntu.com/533321/ [02:25] pjdc: Yup. [02:25] wgrant: and you have help [02:25] Flip doubah back on, and start b-m up. [02:25]