[05:27] good morning, looks like the PPA publishers are down and some ppc64el builders are stuck [05:33] ricotz: Publisher fixed. ppc64el is slow, not actually stuck. [05:34] wgrant, thanks -- hmm, some builders are "cleaning" for hours afaics [05:36] I guess I picked the worst one ;) https://launchpad.net/builders/bos02-ppc64el-006/+history === cpaelzer__ is now known as cpaelzer [06:03] Hey. package I uploaded compiled successfully but still shown as pending for about 14 hours already. should I be worried ? === morphis5 is now known as morphis [08:30] rfried: Link? [08:31] cjwatson: hehe, released already (about 10 minutes after I've written the message) [08:31] perhaps there's a bot listening in the channel [08:32] rfried: A bit before you joined there was a note about a publisher outage being resolved, so it probably took about that long to work through its backlog [08:34] :) [10:02] hiho, 18/20 ppc builders stuck in cleaning again, could one give them a bump to resolve? [10:06] I can try but it's unlikely to achieve very much [10:06] really [10:06] May help slightly *shrug* [10:06] is there a wider issue due to e.g. the cloud movement? [10:06] In fact none of them appear to have been cleaning for long enough for my script to notice [10:07] hmm, I just happened to notice after lunch that the ppc64 instances didn't build (all others complete) [10:07] and then I saw the 18/20 [10:07] I have no data how long things are stuck [10:07] cjwatson: what is the duration your script usually watches for? [10:07] 10 minutes [10:08] well, hten lets wait at least 10 minutes [10:08] So either somebody else already did a reset, or it's not as bad as it looks [10:08] would be a odd coincidence, but ok for me [10:08] Not particularly [10:08] Quite a few networking errors in logs, anyway [10:09] cjwatson: I see two nodes disabled now [10:09] Or at least veeery slow port detaches [10:09] is going disabled/active part of the reset that your script does? [10:09] OK, we don't need to stream-of-consciousness this [10:09] We'll look into it [10:09] I already see 5 building again, so you are right the script or someone else might have bumped it already [10:10] * cpaelzer feels bad to bother cjwatson about this so often [10:35] I believe that bos02's ppc64el instances are mostly not booting properly [10:36] Laney: The issue seems very different from yours. [10:36] We can investigate it next week, hopefully. [10:36] You mean #120004? [10:36] bos01 and bos02 seem broken to me in different way [10:36] s [10:37] Yes, we're seeing instances fail to leave BUILD mostly. [10:37] bos01 isn't broken, the images are just missing some properties. [10:37] bos02 has a probable neutron issue on some ppc64el compute nodes [10:37] Interesting distinction. :P [10:38] The images are user data, not the cloud :) [10:40] Heh. [10:41] Anyway. How unlucky if that means bos02 broke for both of us differently. [10:57] cjwatson: wgrant: I still see most of the ppc64el builders hang :-/ [10:58] cjwatson: so your script still didn't reach 10 minutes on any of them? [10:58] or is it possible that the rest itself might be hanging/slow ? [10:59] I again saw some of the builders flip to disabled state for a short time [10:59] cpaelzer: As we've said, there's a known issue with the ppc64el builders that are causing things to be slow. [10:59] It's not going to be addressed this week. [10:59] And we're aware of the user-visible properties of the failure. [10:59] ok, thanks wgrant and cjwatson [11:00] If you have a particular build that has been waiting a long time, we can bump it [11:00] it seems I just wanted to do too much at once, will postpone the tasks depending on those for a while ... [11:00] not thanks, no single prop bumping for me in this case [11:00] I can wait as all others do [11:39] just to confirm - the 2235+990 ppc64el queue size on http://autopkgtest.ubuntu.com/running probably is the same "known to be slow" issue right? [11:41] That's a separate issue [11:41] But that one's hopefully worked around just a few seconds ago [11:44] oh nice [11:58] Soon [15:03] hi. apparently there used ot be a http redirection from http(s)://bazaar.launchpad.net to https://launchpad.net which is not there presently (it now returns a 404) [15:19] tomreyn: It's there for http://bazaar.launchpad.net/. Are you sure it was ever there for https? AFAIK no configuration there has changed recently. [15:29] cjwatson: hmm i was looking at google's cache, which says it can't tell (due to robots.txt i assume), and i also looked at http://archive.fo/https://bazaar.launchpad.net/ but this seems to have only cached the HTTP (not HTTPS) redirection, too. so you're probably right,. [15:32] what prompted me was this in #ubuntu-hardened (no idea why): [15:32] is it random outage or was it shutted down? https://bazaar.launchpad.net/ [15:32] "The resource could not be found" [15:42] It would probably be good to sort out a redirection from HTTPS as well at some point, but it's not a regression