[05:27] <ricotz> good morning, looks like the PPA publishers are down and some ppc64el builders are stuck
[05:33] <wgrant> ricotz: Publisher fixed. ppc64el is slow, not actually stuck.
[05:34] <ricotz> wgrant, thanks -- hmm, some builders are "cleaning" for hours afaics
[05:36] <ricotz> I guess I picked the worst one ;) https://launchpad.net/builders/bos02-ppc64el-006/+history
[06:03] <rfried> Hey. package I uploaded compiled successfully but still shown as pending for about 14 hours already. should I be worried ?
[08:30] <cjwatson> rfried: Link?
[08:31] <rfried> cjwatson: hehe, released already (about 10 minutes after I've written the message)
[08:31] <rfried> perhaps there's a bot listening in the channel
[08:32] <cjwatson> rfried: A bit before you joined there was a note about a publisher outage being resolved, so it probably took about that long to work through its backlog
[08:34] <rfried> :)
[10:02] <cpaelzer> hiho, 18/20 ppc builders stuck in cleaning again, could one give them a bump to resolve?
[10:06] <cjwatson> I can try but it's unlikely to achieve very much
[10:06] <cpaelzer> really
[10:06] <cjwatson> May help slightly *shrug*
[10:06] <cpaelzer> is there a wider issue due to e.g. the cloud movement?
[10:06] <cjwatson> In fact none of them appear to have been cleaning for long enough for my script to notice
[10:07] <cpaelzer> hmm, I just happened to notice after lunch that the ppc64 instances didn't build (all others complete)
[10:07] <cpaelzer> and then I saw the 18/20
[10:07] <cpaelzer> I have no data how long things are stuck
[10:07] <cpaelzer> cjwatson: what is the duration your script usually watches for?
[10:07] <cjwatson> 10 minutes
[10:08] <cpaelzer> well, hten lets wait at least 10 minutes
[10:08] <cjwatson> So either somebody else already did a reset, or it's not as bad as it looks
[10:08] <cpaelzer> would be a odd coincidence, but ok for me
[10:08] <cjwatson> Not particularly
[10:08] <cjwatson> Quite a few networking errors in logs, anyway
[10:09] <cpaelzer> cjwatson: I see two nodes disabled now
[10:09] <cjwatson> Or at least veeery slow port detaches
[10:09] <cpaelzer> is going disabled/active part of the reset that your script does?
[10:09] <cjwatson> OK, we don't need to stream-of-consciousness this
[10:09] <cjwatson> We'll look into it
[10:09] <cpaelzer> I already see 5 building again, so you are right the script or someone else might have bumped it already
[10:10]  * cpaelzer feels bad to bother cjwatson about this so often
[10:35] <Laney> I believe that bos02's ppc64el instances are mostly not booting properly
[10:36] <wgrant> Laney: The issue seems very different from yours.
[10:36] <wgrant> We can investigate it next week, hopefully.
[10:36] <Laney> You mean #120004?
[10:36] <Laney> bos01 and bos02 seem broken to me in different way
[10:36] <Laney> s
[10:37] <wgrant> Yes, we're seeing instances fail to leave BUILD mostly.
[10:37] <wgrant> bos01 isn't broken, the images are just missing some properties.
[10:37] <wgrant> bos02 has a probable neutron issue on some ppc64el compute nodes
[10:37] <Laney> Interesting distinction. :P
[10:38] <wgrant> The images are user data, not the cloud :)
[10:40] <Laney> Heh.
[10:41] <Laney> Anyway. How unlucky if that means bos02 broke for both of us differently.
[10:57] <cpaelzer> cjwatson: wgrant: I still see most of the ppc64el builders hang :-/
[10:58] <cpaelzer> cjwatson: so your script still didn't reach 10 minutes on any of them?
[10:58] <cpaelzer> or is it possible that the rest itself might be hanging/slow ?
[10:59] <cpaelzer> I again saw some of the builders flip to disabled state for a short time
[10:59] <wgrant> cpaelzer: As we've said, there's a known issue with the ppc64el builders that are causing things to be slow.
[10:59] <wgrant> It's not going to be addressed this week.
[10:59] <wgrant> And we're aware of the user-visible properties of the failure.
[10:59] <cpaelzer> ok, thanks wgrant and cjwatson
[11:00] <wgrant> If you have a particular build that has been waiting a long time, we can bump it
[11:00] <cpaelzer> it seems I just wanted to do too much at once, will postpone the tasks depending on those for a while ...
[11:00] <cpaelzer> not thanks, no single prop bumping for me in this case
[11:00] <cpaelzer> I can wait as all others do
[11:39] <cpaelzer> just to confirm - the 2235+990 ppc64el queue size on http://autopkgtest.ubuntu.com/running probably is the same "known to be slow" issue right?
[11:41] <wgrant> That's a separate issue
[11:41] <wgrant> But that one's hopefully worked around just a few seconds ago
[11:44] <cpaelzer> oh nice
[11:58] <Laney> Soon
[15:03] <tomreyn> hi. apparently there used ot be a http redirection from http(s)://bazaar.launchpad.net to https://launchpad.net which is not there presently (it now returns a 404)
[15:19] <cjwatson> tomreyn: It's there for http://bazaar.launchpad.net/.  Are you sure it was ever there for https?  AFAIK no configuration there has changed recently.
[15:29] <tomreyn> cjwatson: hmm i was looking at google's cache, which says it can't tell (due to robots.txt i assume), and i also looked at http://archive.fo/https://bazaar.launchpad.net/ but this seems to have only cached the HTTP (not HTTPS) redirection, too. so you're probably right,.
[15:32] <tomreyn> what prompted me was this in #ubuntu-hardened (no idea why):
 is it random outage or was it shutted down? https://bazaar.launchpad.net/
 "The resource could not be found"
[15:42] <cjwatson> It would probably be good to sort out a redirection from HTTPS as well at some point, but it's not a regression