[00:03] <tumbleweed> cjwatson: and the primary archive?
[00:04] <cjwatson> tumbleweed: There were probably-similar-cause problems with the arm64, ppc64el, and some powerpc non-virtualised builders earlier today, but those have been resolved.
[00:04] <cjwatson> tumbleweed: I'm in the middle of retrying all the builds I can see.
[00:05] <tumbleweed> cjwatson: that sounds about right, thanks
[00:05] <cjwatson> Something to do with an ntp upgrade, we think.
[00:06] <tumbleweed> lol
[00:06] <cjwatson> I know, right
[10:06] <HeOS_> Hi to all!
[10:06] <HeOS_> What about staging?
[10:07] <HeOS_> He is not working. :(
[10:53] <wgrant> HeOS_: Should be working now.
[10:53] <HeOS_> wgrant: yeah, thanks!
[14:33] <HeOS> Is staging not working again?
[14:55] <HeOS> Could anyone help with staging?
[14:57] <cjwatson> Moment
[15:13] <HeOS> cjwatson, okay.
[15:23] <wgrant> HeOS: What is your API script trying to do?
[15:23] <wgrant> It's making an awful lot of requests.
[15:25] <cjwatson> Nothing anomalous in monitoring, it's not in swapdeath or anything
[15:26] <cjwatson> We should probably arrange for nagios to actually check the staging appservers.
[15:26] <wgrant> It's probably just the 2.7 hang bug that I've hit the test suite a couple of times.
[15:26] <wgrant> only shows up in reasonably concurrent situations, which we aren't exposed to on the single-threaded prod appservers.
[15:27] <cjwatson> We shouldn't have to wait for user reports of a hang.  (OTOH staging is down for restores for fairly long periods of time ...)
[15:29] <HeOS> wgrant, my script get a bugs and create a new series for this bugs.
[15:29] <cjwatson> I guess atemoya is unlikely to be able to cope with running more appservers.
[15:30] <cjwatson> It's got about 5GB free at the moment, but that'd be eaten up quickly enough ...
[15:34] <wgrant> We should probably attempt to track down the hang, but it's a bit late for that for me, unless you want to try gdbing it.
[15:35] <cjwatson> I'm inclined to restart the appserver and leave this on our backlog with some notes.  I have other things to do this afternoon :)
[15:36] <wgrant> Indeed.
[15:36] <wgrant> Hopefully it will happen again at a more reasonable time forme.
[15:37] <wgrant> It's probably the same issue that occasionally hangs buildbot and my local test suite, but only since 2.7
[15:43] <HeOS> Does my script load on staging?
[15:44] <cjwatson> HeOS: Staging isn't totally reliable in various ways.  You might find you spend less time waiting for us to deal with hangs if you slow your script down a bit.
[15:44] <cjwatson> HeOS: But surely whatever testing you're doing can be done in smaller-scale ways.
[15:45] <wgrant> The script could also be optimised a little. More than half of the requests are for duplicated series and milestones -- you'll find things are a few times faster if you cache those objects locally rather than re-requesting them every time.
[15:45] <cjwatson> HeOS: Anyway, it's back up now.
[15:47] <HeOS> I use a staging for debug and development. :)
[15:48] <HeOS> At this time my script should connect to staging and get data. I can't work without this.
[15:53] <cjwatson> Then I suggest you work on making it more efficient to reduce the probability of failures.
[16:08] <dobey> deploy a local instance of launchpad.dev for testing
[16:08] <dobey> if you really want to abuse it
[20:59] <HeOS> I don't touch a staging, but he is unavailable again.
[21:30] <cjwatson> HeOS: The last incomplete request was just before 18:00 and was from your IP address.
[21:30] <cjwatson> Sorry, just before 16:00.
[21:32] <cjwatson> I've asked for another appserver restart.
[21:50] <cjwatson> It's up again now.
=== MasterPiece| is now known as MasterPiece