| ricotz | good morning, looks like the PPA publishers are down and some ppc64el builders are stuck | 05:27 |
|---|---|---|
| wgrant | ricotz: Publisher fixed. ppc64el is slow, not actually stuck. | 05:33 |
| ricotz | wgrant, thanks -- hmm, some builders are "cleaning" for hours afaics | 05:34 |
| ricotz | I guess I picked the worst one ;) https://launchpad.net/builders/bos02-ppc64el-006/+history | 05:36 |
| === cpaelzer__ is now known as cpaelzer | ||
| rfried | Hey. package I uploaded compiled successfully but still shown as pending for about 14 hours already. should I be worried ? | 06:03 |
| === morphis5 is now known as morphis | ||
| cjwatson | rfried: Link? | 08:30 |
| rfried | cjwatson: hehe, released already (about 10 minutes after I've written the message) | 08:31 |
| rfried | perhaps there's a bot listening in the channel | 08:31 |
| cjwatson | rfried: A bit before you joined there was a note about a publisher outage being resolved, so it probably took about that long to work through its backlog | 08:32 |
| rfried | :) | 08:34 |
| cpaelzer | hiho, 18/20 ppc builders stuck in cleaning again, could one give them a bump to resolve? | 10:02 |
| cjwatson | I can try but it's unlikely to achieve very much | 10:06 |
| cpaelzer | really | 10:06 |
| cjwatson | May help slightly *shrug* | 10:06 |
| cpaelzer | is there a wider issue due to e.g. the cloud movement? | 10:06 |
| cjwatson | In fact none of them appear to have been cleaning for long enough for my script to notice | 10:06 |
| cpaelzer | hmm, I just happened to notice after lunch that the ppc64 instances didn't build (all others complete) | 10:07 |
| cpaelzer | and then I saw the 18/20 | 10:07 |
| cpaelzer | I have no data how long things are stuck | 10:07 |
| cpaelzer | cjwatson: what is the duration your script usually watches for? | 10:07 |
| cjwatson | 10 minutes | 10:07 |
| cpaelzer | well, hten lets wait at least 10 minutes | 10:08 |
| cjwatson | So either somebody else already did a reset, or it's not as bad as it looks | 10:08 |
| cpaelzer | would be a odd coincidence, but ok for me | 10:08 |
| cjwatson | Not particularly | 10:08 |
| cjwatson | Quite a few networking errors in logs, anyway | 10:08 |
| cpaelzer | cjwatson: I see two nodes disabled now | 10:09 |
| cjwatson | Or at least veeery slow port detaches | 10:09 |
| cpaelzer | is going disabled/active part of the reset that your script does? | 10:09 |
| cjwatson | OK, we don't need to stream-of-consciousness this | 10:09 |
| cjwatson | We'll look into it | 10:09 |
| cpaelzer | I already see 5 building again, so you are right the script or someone else might have bumped it already | 10:09 |
| * cpaelzer feels bad to bother cjwatson about this so often | 10:10 | |
| Laney | I believe that bos02's ppc64el instances are mostly not booting properly | 10:35 |
| wgrant | Laney: The issue seems very different from yours. | 10:36 |
| wgrant | We can investigate it next week, hopefully. | 10:36 |
| Laney | You mean #120004? | 10:36 |
| Laney | bos01 and bos02 seem broken to me in different way | 10:36 |
| Laney | s | 10:36 |
| wgrant | Yes, we're seeing instances fail to leave BUILD mostly. | 10:37 |
| wgrant | bos01 isn't broken, the images are just missing some properties. | 10:37 |
| wgrant | bos02 has a probable neutron issue on some ppc64el compute nodes | 10:37 |
| Laney | Interesting distinction. :P | 10:37 |
| wgrant | The images are user data, not the cloud :) | 10:38 |
| Laney | Heh. | 10:40 |
| Laney | Anyway. How unlucky if that means bos02 broke for both of us differently. | 10:41 |
| cpaelzer | cjwatson: wgrant: I still see most of the ppc64el builders hang :-/ | 10:57 |
| cpaelzer | cjwatson: so your script still didn't reach 10 minutes on any of them? | 10:58 |
| cpaelzer | or is it possible that the rest itself might be hanging/slow ? | 10:58 |
| cpaelzer | I again saw some of the builders flip to disabled state for a short time | 10:59 |
| wgrant | cpaelzer: As we've said, there's a known issue with the ppc64el builders that are causing things to be slow. | 10:59 |
| wgrant | It's not going to be addressed this week. | 10:59 |
| wgrant | And we're aware of the user-visible properties of the failure. | 10:59 |
| cpaelzer | ok, thanks wgrant and cjwatson | 10:59 |
| wgrant | If you have a particular build that has been waiting a long time, we can bump it | 11:00 |
| cpaelzer | it seems I just wanted to do too much at once, will postpone the tasks depending on those for a while ... | 11:00 |
| cpaelzer | not thanks, no single prop bumping for me in this case | 11:00 |
| cpaelzer | I can wait as all others do | 11:00 |
| cpaelzer | just to confirm - the 2235+990 ppc64el queue size on http://autopkgtest.ubuntu.com/running probably is the same "known to be slow" issue right? | 11:39 |
| wgrant | That's a separate issue | 11:41 |
| wgrant | But that one's hopefully worked around just a few seconds ago | 11:41 |
| cpaelzer | oh nice | 11:44 |
| Laney | Soon | 11:58 |
| tomreyn | hi. apparently there used ot be a http redirection from http(s)://bazaar.launchpad.net to https://launchpad.net which is not there presently (it now returns a 404) | 15:03 |
| cjwatson | tomreyn: It's there for http://bazaar.launchpad.net/. Are you sure it was ever there for https? AFAIK no configuration there has changed recently. | 15:19 |
| tomreyn | cjwatson: hmm i was looking at google's cache, which says it can't tell (due to robots.txt i assume), and i also looked at http://archive.fo/https://bazaar.launchpad.net/ but this seems to have only cached the HTTP (not HTTPS) redirection, too. so you're probably right,. | 15:29 |
| tomreyn | what prompted me was this in #ubuntu-hardened (no idea why): | 15:32 |
| tomreyn | <vidal72[m]> is it random outage or was it shutted down? https://bazaar.launchpad.net/ | 15:32 |
| tomreyn | <vidal72[m]> "The resource could not be found" | 15:32 |
| cjwatson | It would probably be good to sort out a redirection from HTTPS as well at some point, but it's not a regression | 15:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!