[09:01] <rogpeppe> mornin' all, and Happy New Year!
[09:04] <axw> hey rogpeppe, happy new year :)
[09:04] <rogpeppe> axw: hiya
[09:04] <rogpeppe> axw: hope you've had a good break
[09:04] <axw> rogpeppe: yeah it was pretty relaxing, and you?
[09:05] <rogpeppe> axw: yeah, good thanks.
[10:03] <natefinch> fwereade, rogpeppe: I guess no meeting this morning?
[10:04] <rogpeppe> natefinch: i dunno, i'd assumed there would be one, just to get us off to a start
[10:05] <natefinch> rogpeppe: maybe most people are taking these two days off too?
[10:05] <rogpeppe> natefinch: that's entirely possible
[10:06] <fwereade> rogpeppe, would you kjoin us quickly please?
[10:07] <rogpeppe> fwereade: where?
[10:07] <fwereade> rogpeppe, https://plus.google.com/hangouts/_/calendar/bWFyay5yYW1tLWNocmlzdGVuc2VuQGNhbm9uaWNhbC5jb20.8sj9smn017584lljvp63djdnn8
[14:10] <rogpeppe> feeling stupid trying to upgrade my system - i click the "Upgrade" button and nothing at all happens. any clues?
[14:13] <rogpeppe> hmm, now running do-release-upgrade directly, with crossed fingers
[14:30] <sinzui> rogpeppe, make sure ot have go installed from upstream or a PPA. trusty has golang 1.2. I haven't had a successful test run using it.
[14:38] <rogpeppe> lunch
[14:47] <rogpeppe> sinzui: interesting - i've been using 1.2 for a while and it seems to work ok
[14:47] <rogpeppe> sinzui: we'll see how it goes - am currently downloading saucy; hopefully will succeed in upgrading to trusty later on today
[15:17] <rogpeppe> hmm, do-release-upgrade has now hung up
[15:17] <rogpeppe> last thing it printed was "Setting up adduser (3.113+nmu3ubuntu2) ..."
[15:17] <rogpeppe> it seems to be running dpkg
[15:17] <rogpeppe> but dpkg is stripped so difficult to find out what it's *actually* trying to do
[15:21] <rogpeppe> hmm, looks like it *might* be blocked trying to read from stdin
[15:24] <rogpeppe> oh, darn, i broke it
[15:24]  * rogpeppe wishes that do-release-upgrade could continue where it left off
[15:35] <rogpeppe> ha ha
[15:35] <rogpeppe> diff: \/etc\/cups\/cups\-files\.conf: No such file or directory
[15:35] <rogpeppe> diff: \/etc\/cups\/cups\-files\.conf\.dpkg\-new: No such file or directory
[15:37] <rogpeppe> rebooting
[17:02] <rogpeppe> well, i now seem to be running Trusty Tahr. that was actually quite painless.
[17:09] <sinzui> rogpeppe, The only problem I have had with trusty are reports that X/Mir crashed when I didn't see it crash
[17:10] <rogpeppe> sinzui: i'm hoping that by running it now, my bug reports might actually be useful to someone before the release...
[17:11] <rogpeppe> sinzui: only problem i've seen so far is:
[17:11] <rogpeppe> "bzr 2.6.0 is too new for pipeline 1.4"
[17:13] <sinzui> rogpeppe, we see spurious failures in CI regarding "unable to connect to environment" most often with azure and canonistack. The revision that just landed has never passed on canonistack, and I cannot tell if cloud or juju is at fault. Do you see anything insightful in this log: http://162.213.35.54:8080/job/canonistack-upgrade/452/consoleFull#-2115410258bc9712e0-ba8d-4476-87e1-593fdc4da857
[17:15] <sinzui> abentley, ^ can rogpeppe use/test the patch to bug 1209131 to get pipelines working again
[17:15] <_mup_> Bug #1209131: Pipeline reports it is too old for newly released bzr 2.6 <bzr-pipeline:New> <https://launchpad.net/bugs/1209131>
[17:22] <rogpeppe> sinzui: nothing jumps out at me
[17:23] <rogpeppe> sinzui: what is wait_for_agent_update actually doing?
[17:23] <rogpeppe> sinzui: and this line is weird - do you know where it's coming from?
 unknown: 1, 0, 2
[17:25] <sinzui> rogpeppe, wait is sleeping, call status, parse for agent state
[17:26] <rogpeppe> sinzui: so it succeeds n times, then fails?
[17:30] <sinzui> rogpeppe, I am not sure about wht the numbers mean. Each is the status of unit + the state-server
[17:31] <rogpeppe> sinzui: i'm trying to see what kind of failure we're observing here
[17:31] <rogpeppe> sinzui: is the problem that we're making some change, then can't connect, or is it that we're suddenly unable to connect for some unknown reason?
[17:32] <sinzui> ah, well maybe I can do better than read the test is doing. I think I can get the log from the upgrade
[17:33] <sinzui> rogpeppe, This is the all machines log of the failed upgrade test: http://162.213.35.54:8080/job/canonistack-upgrade/452/artifact/artifacts/all-machines-test-release-canonistack.log.gz
[17:37] <rogpeppe> sinzui: so is this error happening just after an environment has been upgraded?
[17:37] <sinzui> rogpeppe, yes
[17:37] <rogpeppe> sinzui: i'm not that surprised then
[17:38] <rogpeppe> sinzui: when you upgrade an environment, the API server will restart, dropping any existing connections
[17:38] <sinzui> rogpeppe, CI just passed r2179. The rev is blessed.
[17:38]  * rogpeppe needs a translation of that :-)
[17:39] <rogpeppe> sinzui: what does "blessed" mean in this context?
[17:40] <sinzui> rogpeppe, CI passed r2179 after 5 attempts. We could release it as 1.17.1. The log I am showing you is from the fourth failed attempt
[17:40] <sinzui> bless == release candidate
[17:40] <rogpeppe> sinzui: i see, so 1/5 tests passed...
[17:40] <sinzui> yep.
[17:41] <rogpeppe> sinzui: i suspect it might pass more often if you waited 20 seconds after upgrading before trying to connect again
[17:41] <sinzui> rogpeppe, okay. thank you. I will try that.
[17:42] <rogpeppe> sinzui: another possibility is to treat "connection is shut down" as a soft error, and retry again without failing the CI
[17:43] <sinzui> rogpeppe, I agree. If you expect the API to drop all connections and let natural error recovery to take over, then the test is too strict
[17:44] <rogpeppe> sinzui: there is perhaps an argument that the command line should be more resilient about what happens when an environment is upgraded, but it's not an easy problem to solve well.
[17:45] <sinzui> I am always will to retry. Since the test is attempting to behave as a devop, it should retry
[19:06]  * rogpeppe is done for the day