[02:29] <thumper> coffee time...
[02:45] <hazmat> thumper, wallyworld_, waigani  if any of you are up for user support.. there's a guy having trouble getting started with charm authoring #juju
[02:46]  * wallyworld_ hasn't written any charms :-(
[02:46] <waigani> waigani: sorry hazmat I don't think I'd be much help yet - still a newbie :/
[02:47] <hazmat> waigani, no worries.. we all start somewhere
[02:47] <hazmat> waigani, and welcome aboard
[02:47] <wallyworld_> maybe davecheney is around?
[02:47] <waigani> hazmat: thanks :)
[02:53] <davecheney> ping, can anyone review this CL ? https://codereview.appspot.com/69980044/
[02:53] <davecheney> is the landing bot working at the moment ?
[02:54] <waigani> davecheney:  wallyworld_ has been wrestling with it, he would be the one to ask
[02:55] <wallyworld_> davecheney: sadly not working, i've been landing stuff manually
[02:55] <wallyworld_> i can do your review and land
[02:56] <davecheney> wallyworld_: ta
[02:56] <wallyworld_> davecheney: there's a guy in #juju talking to hazmat about a charm problem, are you able to guess what the issue might be?
[02:57] <davecheney> i'm in that channel
[02:57] <davecheney> i've asked the guy for more information
[02:57] <hazmat> davecheney, thanks
[02:58]  * hazmat returns to customer work...
[03:01] <wallyworld_> davecheney: does azure or joyent or maas support the new arches from your branch?
[03:02] <davecheney> wallyworld_: maas yes
[03:02] <davecheney> joyent and azure, no
[03:02] <davecheney> this CL is a clone of thumper s
[03:02] <davecheney> s/arm64/ppc64/g
[03:02] <davecheney> basically
[03:02] <wallyworld_> do we need to update the maas provider?
[03:02] <wallyworld_> ah ok
[03:02] <davecheney> wallyworld_: in a perfect world yes
[03:02] <davecheney> but we don't have ppc64 maas machines to test on
[03:03] <wallyworld_> we really need to fix the "amd64", "i386", "arm", "arm64 ", "ppc64" thing
[03:03] <davecheney> so lets cross that bridge when we get to it
[03:03] <wallyworld_> ok
[03:03] <davecheney> wallyworld_: i agree, too many constants
[03:05] <wallyworld_> it's also a little disconcerting or something that no tests needed to be updated
[03:06] <davecheney> indeed
[03:06] <davecheney> i have no idea how thumper got away with it
[03:06] <wallyworld_> i'll file a bug
[03:06] <wallyworld_> we can fix the consts and tests all in one go
[03:08] <wallyworld_> davecheney: actually I think I modified a lot of that code at some point. i don't think we ever had tests for which arches various providers supported
[03:10] <waigani> wallyworld_: Reset in upgradejuju_test sets the agent-version. This throws  an "agent-version cannot be changed" error with my new code.
[03:10] <waigani> show agent-version be allowed to be changed?
[03:11] <waigani> *should
[03:11] <wallyworld_> yes
[03:11] <waigani> wallyworld_: okay, that check was brought over from apiserver
[03:11] <wallyworld_> it's how an upgrade is triggered
[03:11] <wallyworld_> let me check
[03:12] <waigani> wallyworld_: https://codereview.appspot.com/63790045/diff/40001/state/apiserver/client/client.go#newcode791
[03:12] <waigani> I brought that agent-version check over into state.
[03:12] <waigani> so I can just delete it?
[03:14] <wallyworld_> waigani: the check from the apiserver client is to stop people changing the agent version manually without going via upgrade
[03:14] <waigani> wallyworld_: right, so it only makes sense in apiserver?
[03:14] <wallyworld_> the SetEnvironAgentVersion() methof is used to change the agent version in state for ugrades
[03:15] <wallyworld_> only makes sense in client.SetEnvironment()
[03:15] <waigani> wallyworld_: yep. no problem, I'll get it out of state.
[03:15] <wallyworld_> but, we want to ensure only thing that can change agent version in state is the proper upgrade path
[03:16] <waigani> okay ...
[03:17] <wallyworld_> the check in client.EnvironmentSet() is all we do now i think
[03:18] <thumper> oh ffs
[03:18]  * thumper stabs something
[03:19] <waigani> wallyworld_: I'll make a todo and come back to that
[03:19] <wallyworld_> waigani: if needed. i'm not 100% across the finer detail
[03:20] <waigani> sure, we will revisit it at a later date
[03:23] <wallyworld_> thumper: i'm not sure i fully agree with our current upgrade logic. now, if you juju upgrade and don't specify anything, it gets the current agent version and increments the minor version number. i would have thought we would use the version of the CLI used to do the upgrade to determine the desired version to which to upgrade
[03:23] <thumper> wallyworld_: sounds reasonable
[03:23] <thumper> perhaps we do the following:
[03:23] <wallyworld_> if you use --upload-tools or --version, that sets things exlicitly
[03:23] <thumper> * if nothing is specified, use version.Current
[03:24] <thumper> * if version.Current == the version deployed, info, and call it success
[03:24] <thumper> if version.Current < deployed version, error ??
[03:24] <wallyworld_> effectively we do all that and more now
[03:25]  * thumper is trying to work out how to mock something out...
[03:25] <wallyworld_> excet for the case where nothing exlicit is secified
[03:25] <wallyworld_> a function?
[04:19] <wallyworld_> davecheney: can you merge latest trunk into your branch and resolve conflicts? i've got trunk r2375 (tip) and when I merge your branch to get conflicts cause "arm64" changes have already landed
[04:20] <wallyworld_> I wanted to run tests and then land it for you
[04:26] <thumper> davecheney: is there a signal to kill go test with to figure out where it has hung?
[04:27] <thumper> axw: ^^ do you know?
[04:27] <axw> thumper: umm, I think there is... I forget which
[04:27]  * axw looks
[04:30] <axw> thumper: according to a rather old post (2009), killing with SIGSEGV will do
[04:30] <davecheney> thumper: axw SIGQUIT
[04:30] <thumper> hmm
[04:30] <davecheney> or ^\
[04:31] <thumper> davecheney: is the a ctrl+ key for that?
[04:31] <axw> ah, SIGQUIT works for Go? ok
[04:31] <davecheney> the go command ignores that signal and passed it through to the child process
[04:31] <davecheney> SIGQUIT
[04:31] <davecheney> kill -QUIT
[04:31] <davecheney> cntl+\
[04:31] <davecheney> ^ all da same
[04:31] <thumper> ta
[04:31] <thumper> that worked
[04:31]  * thumper goes to figure out what he did wrong
[04:32] <davecheney> wallyworld_: thanks for landig that commit
[04:32] <davecheney> much appreacited
[04:33] <wallyworld_> davecheney: i haven't yet - i merged into my local trunk tip and got conflicts cause the "arm64" bit has already landed
[04:33] <wallyworld_> once I can merge and run the tests, then I can land
[04:34] <davecheney> wallyworld_: the bot said it was landed
[04:35] <wallyworld_> oh?
[04:35] <wallyworld_> is the bot running
[04:36] <wallyworld_> it wasn't last night
[04:36] <thumper> ah found it
[04:37] <wallyworld_> i thought it was still broken
[04:37]  * thumper stabs it in the face
[04:37] <wallyworld_> hence my offer to land the branch for you
[04:37] <davecheney> whelp, that is just weird
[04:37] <thumper> yep, that was it
[04:38] <wallyworld_> davecheney: i just checked juju-core - your branch isn't landed from what i can see
[04:39] <wallyworld_> mp still says aroved
[04:39] <wallyworld_> approved
[04:39] <wallyworld_> recent revisions on https://code.launchpad.net/~go-bot/juju-core/trunk doesn't list it
[04:40] <davecheney> ok
[04:40] <davecheney> maybe I was mistaken
[04:40] <wallyworld_> davecheney: i did approve the mp in preparation for testing and landing though
[04:40] <wallyworld_> maybe thats what you saw
[04:41] <wallyworld_> so the conflict i see is because your branch added "arm64" and "ppc" but "arm64" is already added
[04:41] <wallyworld_> hence conflict
[04:42] <davecheney> yeah, thumpers branch is a prereq
[04:42] <davecheney> if you can land that one first
[04:42] <davecheney> i'll redo mine
[04:44] <wallyworld_> davecheney: thumpers is already landed
[04:58] <davecheney> wallyworld_: ok, i'll redo my branch
[04:58] <wallyworld_> ok, i'll look at it after the school run
[05:28] <arosales> wallyworld_: any ideas on this bug https://bugs.launchpad.net/juju-core/+bug/1285803
[05:28] <_mup_> Bug #1285803: [Joyent Provider] metadata mismatch when testing again Joyent Public Cloud <joyent-provider> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1285803>
[06:21] <wallyworld_> arosales: i'll take a look and see what I can find
[06:24] <arosales> wallyworld_: thanks.  I had utlemming and curtis take a look and it didn't look related to the streams set up so I thought I would check with you.
[06:24] <wallyworld_> sure, i've got a meeting now but will get to it asap
[06:24] <arosales> wallyworld_: thanks
[06:24] <wallyworld_> np
[09:14] <dimitern> jamespage, hey, axw recently added rsyslog-gnutls as a required package and juju-local should have it in its list of dependencies
[09:15] <jamespage> dimitern, *$!£$%
[09:15] <jamespage> its in universe
[09:15] <jamespage> I'll look in a bit
[09:16] <dimitern> jamespage, cheers!
[09:45] <hazmat> jamespage, its in universe because of gnutls?
[09:45] <jamespage> hazmat, not sure - looking atm
[09:46] <jamespage> hazmat, nah - I don't think this is hard
[09:46] <jamespage> rsyslog source is in main
[09:46] <jamespage> just not the -gnutls binary package
[09:46] <mgz> and gnutls is in main
[09:47] <rogpeppe> cjohnston: ping
[09:47] <mgz> so, it doesn't seem *that* outlandish to promote the rsyslog-gnutls package to main
[09:47] <jamespage> mgz, indeed
[09:47] <jamespage> binary re-promotions are not normally that bad
[09:48] <hazmat> cool
[09:51] <jamespage> dimitern, hazmat: is there a bug I can reference please?
[09:53] <hazmat> jamespage, https://bugs.launchpad.net/juju-core/+bug/1285550
[09:53] <_mup_> Bug #1285550: juju-local must install rsyslog-gnutls <local-provider> <juju-core:In Progress by sinzui> <https://launchpad.net/bugs/1285550>
[10:02] <natefinch> dimitern, mgz, jam, rogpeppe: morning
[10:02] <mgz> mornin' nate, we'll set you up
[10:02] <natefinch> mgz: thanks
[10:04] <jam> morning nate: https://plus.google.com/hangouts/_/76cpjntt2l1bgmremihhnd1pko
[10:04] <jam> natefinch: ^^
[11:02] <jamespage> dimitern, hazmat: added and in trusty
[11:02] <dimitern> jamespage, thanks
[11:15] <jamespage> hazmat, urgh - just hit what might be a bug
[11:15] <jamespage> maas environment, doing some weird stuff but...
[11:15]  * hazmat waits for it
[11:15] <jamespage> the machine agent is installing LXC
[11:16] <jamespage> at the same time as hooks for the deployed service are trying to install update stuff
[11:16] <jamespage> resulting in #bag
[11:16] <jamespage> bang rather
[11:16] <jamespage> hazmat, lxc gets installed late now
[11:16] <hazmat> jamespage, hmm.. that shouldn't happen.. lxc should be getting installed if a container is being deployed..
[11:17] <hazmat> jamespage, this with 1.16.6
[11:17] <hazmat> ?
[11:17] <jamespage> hazmat, no 1.17.4
[11:17] <jamespage> it was OK on 1.16.6
[11:17] <jamespage> I'm using a mix of lxc containers and stuff running directly on host
[11:18] <hazmat> jamespage, ah.. so you do have container scheduled but its executing/being created at the same time as hooks, runtime pkg install needs to be serialized with hook execution to avoid blow up.
[11:18] <jamespage> hazmat, yah
[11:18] <jamespage> hazmat, its actually just the install install of lxc that's conflicting
[11:19] <jamespage> hazmat, I'll raise a bug
[11:58] <cjohnston> rogpeppe: pong
[12:03] <rogpeppe> cjohnston: apparently canonistack is in a bad way currently - do you think it might be possible to reproduce the bug (#1284183) on some other provider?
[12:03] <_mup_> Bug #1284183: jujuclient.EnvError: <Env Error - Details:  {   u'Error': u'watcher was stopped', u'RequestId': 9, u'Response': {   }} <api> <status> <juju-core:Triaged> <https://launchpad.net/bugs/1284183>
[12:03] <cjohnston> rogpeppe: I can reproduce in HP
[12:06] <rogpeppe> cjohnston: would the repro instructions be the same for HP ?
[12:06] <cjohnston> rogpeppe: skip the sshuttle part...
[12:07] <rogpeppe> cjohnston: thanks
[12:07] <cjohnston> and we pinned jujuclient yesterday, so you will need a slightly older code base
[12:07] <cjohnston> let me figure out which revno
[12:07] <cjohnston> rogpeppe: -r 311 should give it to you...
[12:09] <rogpeppe> cjohnston: thanks
[12:20] <jam> cjohnston: I'm trying to test your reproduction for bug #1284183, but it would seem I need a bit of config
[12:20] <_mup_> Bug #1284183: jujuclient.EnvError: <Env Error - Details:  {   u'Error': u'watcher was stopped', u'RequestId': 9, u'Response': {   }} <api> <status> <juju-core:Triaged> <https://launchpad.net/bugs/1284183>
[12:20] <jam> cjohnston: ERROR: Missing required environment variables:
[12:20] <jam>   CI_LAUNCHPAD_USER
[12:20] <jam>   CI_OAUTH_TOKEN
[12:20] <jam>   OS_USERNAME
[12:20] <jam>   CI_OAUTH_TOKEN_SECRET
[12:20] <jam>   CI_OAUTH_CONSUMER_KEY
[12:20] <jam> Please ensure the novarc file has been sourced.
[12:21] <cjohnston> jam: ./create_lp_creds.py
[12:21] <jam> cjohnston: how much access does it need?
[12:21] <jam> change anything ?
[12:21] <jam> (a bit scary to run your random scripts with full access to my LP account :)
[12:21] <cjohnston> jam: we use it for uploading to a pa
[12:21] <cjohnston> ppa
[12:22] <cjohnston> you probably won't need much access since I doubt you are going to be actually playing with the system itself
[12:22] <cjohnston> You may even be able to put in dummy info, I'm not sure.
[12:22] <cjohnston> ev: ^
[12:22] <cjohnston> OS_USERNAME would need to be valid tho
[12:23] <jam> cjohnston: it also needs locally installed cheetah ?
[12:23] <cjohnston> ack
[12:23] <cjohnston> yes
[12:23] <mattyw> rogpeppe, do you have a moment - just need you to double check a test for me
[12:24] <rogpeppe> mattyw: np
[12:24] <jam> cjohnston: apparently it also wants the rest of OS_AUTH_* stuff (is it running nova/swift directly)? but it doesn't preflight check them
[12:24] <ev> jam: you can get away with feeding it largely bogus Launchpad credentials as the bug you'll hit happens during deployment - it doesn't validate the LP data until you try to do something with the instances.
[12:24] <mattyw> rogpeppe, I made those changes we discussed with william on friday - including moving that error to the juju-core/errors package. now I get this error: http://paste.ubuntu.com/7038224/
[12:25] <mattyw> I'm guessing that's expected and I just need to add errors to that list to make it pass - but thought I should check
[12:25] <rogpeppe> mattyw: which package is failing there?
[12:26] <mattyw> rogpeppe, juju-core/cmd
[12:26] <rogpeppe> mattyw: ok, yeah, just add errors there
[12:27] <jam> ev, cjohnston: you also need "import yaml" is this just "python-yaml" ?
[12:27] <cjohnston> yes
[12:27] <mattyw> rogpeppe, thanks very much
[12:27] <rogpeppe> mattyw: np
[12:28] <jam> cjohnston: and now websocket
[12:33] <jam> cjohnston: well, python-websocket-client, but right now it has been running and seems to be happy on HP clound
[12:33] <jam> cloud
[12:33] <jam> cjohnston:  Service 'ci-juju-gui' does not need any more units added.
[12:33] <cjohnston> that should be on 0
[12:33] <jam> I do see:     agent-state-info: '(error: cannot get started instance: failed to get details
[12:33] <jam>       for serverId: 3681019
[12:33] <jam>       caused by: Maximum number of attempts (3) reached sending request to https://az-1.region-a.geo-1.compute.hpcloudsvc.com/v1.1/17031369947864/servers/3681019)'
[12:34] <jam> in "juju status" though the script hasn't finished yte
[12:38] <jam> cjohnston: I do see: "Connection Timeout: disconnecting client after 300.0 seconds" which IIRC is the message Bazaar gives when you've connected to Launchpad for more than 5 minutes without any actual requests (internally the bzr client *should* reconnect after that)
[12:38] <cjohnston> ack
[12:39] <jam> cjohnston: presumably we're stuck at " Waiting for units before adding relations" but that won't ever complete because of failures in provisioning due to rate limiting
[12:39] <cjohnston> jam: which limiting
[13:44] <jam> cjohnston: sorry, was out for lunch. HP cloud rate limiting how many requests you can send to Openstack (describe instances, start instance, etc)
[13:45] <cjohnston> hrm. interesting
[14:19] <ev> hazmat: shouldn't conn.connect take arguments in jujuclient? http://paste.ubuntu.com/7038687/
[14:19] <ev> seems like that was always the case for websocket-client: https://github.com/liris/websocket-client/blame/master/websocket.py#L422
[14:26] <sinzui> Hi natefinch I would like you to review my branch that changed the deps in the makefile.  https://codereview.appspot.com/71540043
[14:26] <natefinch> sinzui: sure
[14:31] <natefinch> sinzui: uh... looks good to me.  I'm going to trust that your regexes work correctly :)
[14:33] <sinzui> natefinch, yeah, they were nasty to write. well it was the finding which packages are available that was hard. the regex was easy
[14:34] <natefinch> sinzui: yeah, that's generally my experience with regexes too
[15:06] <rogpeppe> cjohnston: i think i might have a solution to the issue
[15:10] <hazmat> ev.. ugh
[15:11] <hazmat> k
[15:11] <hazmat> ev.. got a new toy to show..
[15:14] <ev> yay!
[15:14] <ev> what is it?
[15:22] <hazmat> ev, introspection
[15:24] <hazmat> ev, okay back to the fubar
[15:24] <ev> oooh, nice!
[15:57] <natefinch> mgz: not rushing you, just planning next steps - how's the bot looking?
[16:00] <mgz> natefinch: I'm deploying some things now
[16:01] <natefinch> mgz: cool
[16:13] <sinzui> mgz, you are replacing the lander?
[16:14] <mgz> mgz: our current one got screwed, I'm just putting it back up as a stopgap
[16:14] <natefinch> sinzui: yeah, the last one blew up
[16:14] <mgz> *sinzui
[16:14] <mgz> I don't need to be telling myself...
[16:15] <sinzui> I saw. jcsackett confirmed that the charmworld lander is fully operational. I can offer you that jenkin's based lander if canonistack has issues
[16:16] <sinzui> mgz, the juju qa team can always add a developer branch to Juju CI. We can add a separate job that does the proper merge too.
[16:17] <sinzui> mgz, also, I shutdown all the Juju-CI instances in canonistack and stopped canonistack testing. I hope nova is happy that we left
[16:17] <mgz> sinzui: ooo
[16:18] <mgz> sinzui: I think for now I'll try this, as I want to do some experimenting with how our tests get run to deal with flakiness/isolation
[16:18] <mgz> but moving to jenkins sounds like a good plan
[16:19] <sinzui> mgz okay, since we are moving to git, I can help setup a jenkins instance when you are ready
[16:19] <rogpeppe> natefinch: i think you need to push the changes to https://codereview.appspot.com/69600043/
[16:21] <natefinch> rogpeppe: they were pushed but not proposed. I'm proposing
[16:44] <voidspace> https://codereview.appspot.com/71490045
[16:45] <rogpeppe> voidspace, wwitzel3: code.google.com/p/go.tools/cmd/oracle
[16:45] <rogpeppe> oops
[16:45] <rogpeppe> voidspace, wwitzel3: http://godoc.org/github.com/fzipp/pythia
[16:46] <voidspace> thanks
[17:13] <rogpeppe> voidspace: lp:~rogpeppe/juju-core/511-instancepoller-aggregate
[18:06] <mgz> soooo.... the bot was up briefly, but is no longer
[18:06] <mgz> I shall redo later
[18:06] <natefinch> mgz: dang.  Good luck.
[20:22] <thumper> o/ natefinch
[20:31] <waigani> thumper: https://btrfs.wiki.kernel.org/index.php/FAQ#Why_does_df_show_incorrect_free_space_for_my_RAID_volume.3F
[20:35] <thumper> o/ wallyworld_
[20:37] <wallyworld_> hi
[20:38]  * wallyworld_ is inhaling caffeine
[20:56] <waigani> morning all :)
[21:29] <cjohnston> hazmat: any idea about: http://paste.ubuntu.com/7040757/ ?
[21:39] <davecheney> thumper: ping
[21:39] <davecheney> is now a good time for a chat ?
[21:40] <thumper> I have waigani here, but other than that, yes
[21:40] <davecheney> this is about the gccgo stuff
[21:40] <davecheney> i don't think it's private
[21:40] <davecheney> i'd like to strike while the iron is hot
[21:41] <thumper> kk
[21:41]  * thumper just did "juju add-unit -n50 ubuntu"
[21:41] <thumper> on the local provider
[21:41] <thumper> cpu approaching 100%
[21:41] <davecheney> nice computer you had there
[21:41] <davecheney> shame if anything happened to it
[21:42] <thumper> I'm testing btrfs cloning
[21:42] <davecheney> thumper: just emailed you a url
[21:42] <davecheney> see if that works
[21:42] <thumper> almost all up
[21:42] <davecheney> otherwise send me a hangout invite
[22:01] <hazmat> cjohnston, yeah.. a foobar compounded by a ci branch with merge from ev that didn't fix the issue
[22:02] <cjohnston> hazmat: I see.. any workaround?
[22:05] <hazmat> cjohnston, yes.. in 3hrs.. post call and post customer check-in
[22:09] <stokachu> hazmat: do you know when/if the decision has been made about the manual provider?
[22:09] <hazmat> stokachu, no word.. the thread stopped, probably gotten taken to team hangouts..
[22:09] <stokachu> ok ill bring it up tomorrow during cross team
[22:09] <stokachu> we rely on it as well
[22:10] <hazmat> stokachu, yeah.. that's a good spot for it.
[22:11] <cjohnston> hazmat: ack
[22:53] <bigjools> thumper: is there a bug anywhere about juju's LXCs in maas not having DNS entries?
[23:02] <hazmat> thumper, do you know if axw's revno 2373 completes the automatic upgrade for rsyslog to tls when upgrading juju?
[23:03] <hazmat> sinzui, are there any automated upgrade tests for stable (1.16) to dev (1.17)?
[23:03] <sinzui> hazmat, yes, for each environment
[23:03] <hazmat> sinzui, awesome
[23:34] <wallyworld_> hazmat: yes, the rsyslog upgrade will do the tls stuff
[23:36] <hazmat> wallyworld_, thanks
[23:36] <wallyworld_> np