[00:36] <davechen1y> thumper: cherylj
[00:36] <davechen1y> machine-0: 2016-02-09 00:36:41 INFO juju.apiserver apiserver.go:302 [9] user-admin@local API connection terminated after 7.391830549s, active connections: 5
[00:36] <davechen1y> machine-0: 2016-02-09 00:36:41 INFO juju.apiserver apiserver.go:302 [A] user-admin@local API connection terminated after 2.046902769s, active connections: 4
[00:37] <davechen1y> machine-0: 2016-02-09 00:36:41 INFO juju.apiserver apiserver.go:302 [C] user-admin@local API connection terminated after 91.680242ms, active connections: 3
[00:37] <davechen1y> ^ this is what I can do
[00:41] <thumper> davechen1y: active connections is new current count?
[00:43] <davechen1y> yup
[00:43] <davechen1y> cherylj: 2016-02-08 22:30:16 ERROR Command '('juju', '--debug', 'deploy', '-m', 'maas-1_9-deploy-trusty-amd64', 'local:trusty/dummy-sink')' returned non-zero exit status 1
[00:43] <davechen1y> where can I find the source for this charm ? assuming the charm matters
[00:44] <cherylj> davechen1y: https://private-fileshare.canonical.com/~cherylj/dummy-charms/
[00:44] <cherylj> There's a tar file there, and some copied commands that the CI test uses
[00:44] <davechen1y> ta
[00:45] <davechen1y> is this environement a ha environment ?
[00:45] <cherylj> but I imagine the charm doesn't matter
[00:45] <cherylj> davechen1y: probably not, but you can double check by looking at the test run
[00:45] <davechen1y> yeah, i deployed some of my usual favorites
[00:45] <davechen1y> i'm bloody sick of the tools versino checker
[00:45] <davechen1y> is that on the top of a list for someone to fix ?
[00:46] <davechen1y> checking every 3 seconds is stupid
[00:46] <davechen1y> 15 minutes would be sufficient
[00:46] <cherylj> davechen1y: no, there are other more serious failures that everyone's working on
[00:46] <cherylj> unfortunatley
[00:46] <cherylj> unfortunately, even.   Because that one is darn annoying
[00:46] <cherylj> davechen1y: no, the env is not HA
[00:47] <davechen1y> juju debug-log pauses if you run it after calling enable-ha
[00:47] <davechen1y> brilliant
[00:48] <davechen1y> so does juju ssh
[00:48] <davechen1y> that's wonderful
[00:48] <davechen1y> juju enable-ha
[00:48] <davechen1y> puts your environment into catatonia until that process finishes
[00:49] <cherylj> doesn't surprise me.  everything dies / pauses when you enable-ha
[00:54] <davechen1y> cherylj: so what's happened is
[00:55] <davechen1y> the addresses of the additional mongo servers have been added to the list of state servers
[00:55] <davechen1y> but those additional mongos are up
[00:55] <davechen1y> sorry, not up
[00:55] <davechen1y> they are still going through the cloud=init dance
[00:55] <davechen1y> so you have a 2/3 chance that the apiserver running on machine-0 will try to connect to those
[00:56] <davechen1y> 2016-02-09 00:56:04 DEBUG juju.worker.peergrouper desired.go:116 machine "0" is already voting
[00:56] <davechen1y> 2016-02-09 00:56:04 DEBUG juju.worker.peergrouper desired.go:123 machine "2" is not ready (has status: true)
[00:56] <davechen1y> 2016-02-09 00:56:04 DEBUG juju.worker.peergrouper desired.go:123 machine "3" is not ready (has status: true)
[00:56] <davechen1y> yet the api server still tries to dial it :)
[00:56] <davechen1y> 2016-02-09 00:55:49 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp 10.251.20.20:37017: getsockopt: connection refused
[00:56] <davechen1y> 2016-02-09 00:55:49 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp 10.241.59.50:37017: getsockopt: connection refused
[00:56] <davechen1y> 2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp 10.251.20.20:37017: getsockopt: connection refused
[00:56] <davechen1y> 2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp 10.241.59.50:37017: getsockopt: connection refused
[00:56] <davechen1y> 2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp 10.251.20.20:37017: getsockopt: connection refused
[00:56] <davechen1y> 2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp 10.241.59.50:37017: getsockopt: connection refused
[00:57] <cherylj> davechen1y: but the two bugs that you're looking at aren't using ha
[00:58] <davechen1y> right
[00:58] <davechen1y> i won't get distracted
[00:58] <davechen1y> i was just using enable-ha to try to get the apiserver to explode
[00:58] <davechen1y> and enter it's restart behaviour
[00:59] <cherylj> davechen1y: ah, I was confused
[01:00] <davechen1y> i shouldn't go poking around in juju
[01:00] <davechen1y> there be dragons
[01:01] <cherylj> ain't that the truth.
[01:03] <davechen1y> and it's failed
[01:03] <davechen1y> you'll love this
[01:03] <davechen1y> Attempt 63 to download tools from https://10.251.11.185:17070/tools/2.0-alpha2.1-precise-amd64...
[01:03] <davechen1y> + curl -sSfw tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s  --noproxy * --insecure -o /var/lib/juju/tools/2.0-alpha2.1-precise-amd64/tools.tar.gz https://10.251.11.185:17070/tools/2.0-alpha2.1-precise-amd64
[01:03] <davechen1y> curl: (7) couldn't connect to host
[01:03] <davechen1y> tools from https://10.251.11.185:17070/tools/2.0-alpha2.1-precise-amd64 downloaded: HTTP 000; time 0.001s; size 0 bytes; speed 0.000 bytes/s + echo Download failed, retrying in 15s
[01:03] <davechen1y> Download failed, retrying in 15s
[01:03] <davechen1y> machine-2 is trying to bootstrap
[01:03] <davechen1y> it needs tools from machine-0
[01:03] <davechen1y> machine-0's agent is trying to connect to the replica set
[01:04] <davechen1y> the replica set is down because it's trying to ensure ha
[01:04] <davechen1y> so, no tools
[01:04] <davechen1y> no bootstrap
[01:04] <davechen1y> no ha
[01:04] <davechen1y> no tools
[01:04] <davechen1y> etc
[01:05] <davechen1y> hmm
[01:05] <davechen1y> this is even weirder
[01:06] <davechen1y> this is another case of machine-0 not listening on port 17017
[01:06] <davechen1y> 17070
[01:06] <davechen1y> I need to look into this
[01:06] <davechen1y> if that port is not open, no api server
[01:39] <davechen1y> 2016-02-09 01:36:02 INFO juju.apiserver apiserver.go:302 [1] machine-0 API connection terminated after 3m35.591602712s, active connections: 0
[01:39] <davechen1y> 2016-02-09 01:36:02 INFO juju.apiserver apiserver.go:325 closed listening socket "[::]:17070" with final error: <nil>
[01:39] <davechen1y> gets more and more interesting
[01:40] <davechen1y> the api server shuts down, but never starts back up again
[01:46] <davechen1y> OH MY GODS
[01:46] <davechen1y> the bit where we send data to bash via ssh
[01:46] <davechen1y> after this line
[01:46] <davechen1y> 2016-02-09 01:43:36 DEBUG juju.utils.ssh ssh.go:249 using OpenSSH ssh client
[01:46] <davechen1y> we're sending ONE CHARACTER AT A TIME
[01:46] <davechen1y> axw: ping
[01:47] <axw> davechen1y: pong
[01:47] <axw> say whaaat
[01:48] <axw> davechen1y: where's hte code that's doing that?
[01:48] <davechen1y> https://bugs.launchpad.net/juju-core/+bug/1543388
[01:48] <mup> Bug #1543388: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>
[01:48] <davechen1y> i was watching the bootstrap
[01:48] <davechen1y> and i'm like why is top using 48% cpu
[01:48] <davechen1y> so I straced it
[01:48] <davechen1y> read(0, "e", 1)                         = 1
[01:48] <davechen1y> read(0, "H", 1)                         = 1
[01:48] <davechen1y> read(0, "t", 1)                         = 1
[01:48] <davechen1y> read(0, "D", 1)                         = 1
[01:48] <davechen1y> read(0, "N", 1)                         = 1
[01:48] <davechen1y> read(0, "y", 1)                         = 1
[01:48] <davechen1y> read(0, "q", 1)                         = 1
[01:48] <davechen1y> read(0, "5", 1)                         = 1
[01:48] <davechen1y> read(0, "g", 1)                         = 1
[01:48] <davechen1y> read(0, "H", 1)                         = 1
[01:48] <davechen1y> read(0, "/", 1)                         = 1
[01:48] <davechen1y> read(0, "S", 1)                         = 1
[01:48] <davechen1y> read(0, "o", 1)                         = 1
[01:48] <davechen1y> read(0, "5", 1)                         = 1
[01:48] <davechen1y> read(0, "t", 1)                         = 1
[01:49] <davechen1y> read(0, "6", 1)                         = 1
[01:49] <davechen1y> read(0, "C", 1)                         = 1
[01:49] <axw> :/
[01:49] <mup> Bug #1543388 opened: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>
[01:55] <mup> Bug #1543388 changed: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>
[01:58] <mup> Bug #1543388 opened: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>
[02:04] <davechen1y> can tomb.Kill block ?
[02:06] <natefinch> davechen1y: it does acquire a lock, so, in theory, yes.
[02:06] <natefinch> davechen1y: but otherwise, it just closes the dying channel
[02:06]  * thumper takes a deep breath and dives into importing units
[02:13]  * natefinch looks up the time formatting date for the 1000th time
[02:16] <davechen1y> natefinch: https://github.com/juju/juju/blob/master/apiserver/apiserver.go#L96
[02:16] <davechen1y> so here's the thing
[02:16] <davechen1y> the only way processCertChanges can exit is if someone called tomb.Kill
[02:16] <davechen1y> and hte only thing that can is cl.Close
[02:16] <davechen1y> so this code calls tomb.Kill twice, then tomb.Done for good measure ...
[02:16] <davechen1y> seems like overkill
[02:16] <cherylj> wallyworld: can you take a look at my enable-ha change:  http://reviews.vapour.ws/r/3782/
[02:17] <wallyworld> sure
[02:17] <cherylj> thanks!
[02:17] <wallyworld> cherylj: did axw ping you about possibly making a non blocking channel send?
[02:17] <natefinch> davechen1y: yep... also, it doubly bad, because someone might see cl.tomb.Kill(cl.processCertChanges()) and think they don't need those other lines, but that line won't ever kill the tomb
[02:18] <cherylj> wallyworld: no, not yet
[02:18] <wallyworld> was maybe an alternative to increasing the channel buffer size
[02:19] <wallyworld> but the buffer size increase might be acceptable for now until manifolds are done for those workers
[02:19] <cherylj> okay, can take a look later.   I'm going to be afk for ~40 mins or so, but I'll be back
[02:19] <wallyworld> ok
[02:20] <axw> cherylj: wallyworld: wasn't going to bother, since we need to change it again. non-blocking send wouldn't work here anyway, it's not just a notify chan
[02:20] <wallyworld> ok, just wanted to double check, ty :-)
[02:20] <natefinch> channels with buffers that are not 0 or 1 are pretty suspect, in my experience
[02:21] <axw> unless it's directly next to the things populating the channel, I tend to agree
[02:24] <natefinch> this sounds like one of those cases where you could have an arbitrary number of sends on the channel before any reads from the channel, in which case any buffer size could be insufficient
[02:28] <wallyworld> not arbitrary - is equal to the number of allowed controllers (7) plus a known number of local addresses
[02:28] <natefinch> rick_h___: noted, about the name=file for resources.
[02:28] <wallyworld> it's a short term fix until workers migrated to dep engine
[02:29] <natefinch> wallyworld: ahh, the fact that it's a short term fix makes a big difference.
[02:29] <wallyworld> for 2.0, sadly dep engine not going into 1.25
[02:30] <wallyworld> so not sure what to do there
[02:30] <natefinch> wallyworld: well, we're not going to support 1.25 for very long, right? ;)
[02:30] <wallyworld> 2 years :-(
[02:30] <davechen1y> natefinch: wahhahahahahaha
[02:30] <natefinch> wallyworld: I know, I was joking
[02:30] <wallyworld> but 1.25 is blocked right now, so need to get 1.25.4 out
[02:30] <davechen1y> o/ gallows humor
[02:31] <natefinch> wallyworld: just make the channel buffer 128... larger powers of 2 are always better
[02:31] <wallyworld> ok
[02:31] <wallyworld> i'll add a comment
[02:32] <natefinch> wallyworld: definitely comment why the 10 is 10.
[02:32] <davechen1y> axw: 2016-02-09 02:27:09 DEBUG juju.utils.ssh ssh.go:249 using OpenSSH ssh client
[02:32] <davechen1y> what happens after this line
[02:32] <davechen1y> somethign in bootstrap
[02:33] <davechen1y> but it's mute until the other side starts to output things
[02:33] <axw> davechen1y: umm. could be one of a few things, that debug message gets printed whenever an ssh client is created
[02:33] <axw> davechen1y: first we ssh to each of the possible controller addresses
[02:34] <axw> davechen1y: then (if you're uploading tools), copy tools across via ssh
[02:34] <axw> davechen1y: then run the cloud-config rendered as a bash script
[02:34] <axw> I think that's it
[02:35] <axw> davechen1y: AFAICR, we just open "ssh" with the script as a bytes.Buffer piped to the ssh process's stdin
[02:35] <axw> davechen1y: could be that ssh is in an interactive mode? looking for escape codes?
[02:35] <davechen1y> 13:34 < axw> davechen1y: then (if you're uploading tools), copy tools across via ssh
[02:35] <davechen1y> ^ it'll be this
[02:36] <davechen1y> i'm spelunking in the code now
[02:36] <davechen1y> short version is the openssh impl doens't buffer stdout/stdin
[02:36] <davechen1y> or something
[02:38] <axw> davechen1y: actually, gross as it is, we just add the contents of the tools to the bash script (base64 encoded or something)
[02:38] <axw> so the 2nd and 3rd steps are just one
[02:43] <natefinch> rick_h___: you around?
[03:00] <davechen1y> axw: that's fine
[03:00] <davechen1y> i knew we bas64'd them
[03:00] <davechen1y> the problem is something is unbuffered there and it's sending one character at at time over ssh
[03:00] <davechen1y> which is going to turn each byte into about 400
[03:00] <davechen1y> mabye 200
[03:00] <davechen1y> but it's a lot
[03:00] <davechen1y> and the cpu on both sides is non trivial
[03:05] <axw> davechen1y: yep. I had a look, nothing obvious. what were you stracing exactly? bash on the remote side? ssh on the client?
[03:07] <davechen1y> remote side
[03:07] <davechen1y> bash is hitting 50% cpu
[03:07] <davechen1y> results are in that ticket
[03:09] <axw> davechen1y: ok, will take another look later
[03:20] <davechen1y> umm, https://github.com/juju/juju/blob/master/api/apiclient.go#L554
[03:20] <davechen1y> natefinch: axw https://github.com/juju/juju/blob/master/api/apiclient.go#L554
[03:21] <davechen1y> this construction is unsafe
[03:23] <axw> davechen1y: you mean because two calls could race?
[03:24] <axw> davechen1y: if so yeah.. pretty sure it's always one thing's responsibility to close though.
[03:24] <axw> so in theory, but not in practice (unless we're doing something dumb, which I wouldn't rule out)
[03:26] <davechen1y> this would have to be hit way harder than we could
[03:26] <davechen1y> but it's entirely possible to hit this
[03:26] <davechen1y> https://bugs.launchpad.net/juju-core/+bug/1543404
[03:26] <mup> Bug #1543404: unsafe double channel close idiom <juju-core:New> <https://launchpad.net/bugs/1543404>
[03:27] <natefinch> that code in apiclient doesn't look like it's intended to be threadsafe, so hopefully we're not trying to use it from multiple goroutines
[03:28] <natefinch> lol panics
[03:29] <natefinch> https://github.com/juju/juju/blob/master/api/apiclient.go#L435
[03:34] <davechen1y> sooo, http://paste.ubuntu.com/14999670/
[03:34] <davechen1y> no matter what logging I add, i cannot get line 71 to output something
[03:35] <davechen1y> all I can think of is somehow tools are being cached
[03:35] <davechen1y> and i'm not pushing up what I think i'm pushig up
[03:36] <natefinch> davechen1y: log.Infof, not logger?
[03:38] <davechen1y> OH FOR FUCKS SAKE
[03:38] <davechen1y> thanks
[03:38]  * davechen1y wonders what log was in this scope ...
[03:39] <natefinch> heh, np.  One of those things your brain just can't see if you were the one that wrote it.
[03:43] <davechen1y> sooo, amazong just gave me a machien without a public ip
[03:43] <davechen1y> has that ever happened to anyone ?
[03:44] <davechen1y> it has no public ip or public dns
[03:46] <cherylj> natefinch, wallyworld, so should I do 10 or 128?
[03:46] <cherylj> :)
[03:46] <mup> Bug #1543404 opened: unsafe double channel close idiom <juju-core:New> <https://launchpad.net/bugs/1543404>
[03:46] <wallyworld> 128 according to nate
[03:46] <natefinch> cherylj: I was joking
[03:46] <cherylj> ha, ok :)
[03:47] <natefinch> cherylj: I do fear that 10 will just error out less often... but *shrug* Seems better than 1 :/
[03:47] <cherylj> natefinch: in practice, I see it firing twice
[03:47] <cherylj> if that helps :)
[03:47] <wallyworld> 10 has some science behnd it
[03:48] <cherylj> no, not science, it's http://i.imgur.com/24Jw4gM.gif
[03:48] <wallyworld> lol, educate dguess then
[03:48] <davechen1y> cherylj: 1 should be enough
[03:48] <wallyworld> based on knowledge of the system
[03:48] <cherylj> hehe.  I love that gif
[03:49] <cherylj> davechen1y: I've seen that it's not
[03:49] <davechen1y> if you just want to hand off the value between producer and consumer without either blocking
[03:49] <cherylj> it was 1 before
[03:49] <cherylj> and that was the problem
[03:49] <davechen1y> cherylj: shit
[03:49] <davechen1y> that's more serious
[03:49] <cherylj> it was sending twice
[03:49] <davechen1y> if it was already buffered
[03:49] <cherylj> yeah
[03:49] <davechen1y> what about making the recieve side timeout
[03:50] <cherylj> I'm not sure I see how we would do that.   The receiver is blocked waiting on a lock that the sender is holding
[03:50] <natefinch> davechen1y: it's a clusterfuck that is getting rewritten soonish
[03:50] <natefinch> davechen1y: thus, the bigger buffer is just a stopgap
[03:50] <cherylj> and a band aid for 1.25 :)
[03:50] <davechen1y> whee 1.25 cluserfuck, keeps for 2 years even under adverse conditions
[03:53] <cherylj> but, we should explore the other option menn0 suggested for 1.25 - where there's some other synchronization between certupdater and apiserver
[04:06] <thumper> natefinch: still around?
[04:07] <thumper> wanting to verify that the assign units collection is a transitory collection
[04:07] <thumper> meaning that once all units have been assigned to machines, the length of that collection should be zero
[04:12] <davechen1y> thumper: cherylj http://reviews.vapour.ws/r/3784/
[04:13] <axw> thumper: that is correct.
[04:13] <thumper> axw: ta
[04:16] <mup> Bug #1543408 opened: WatchControllerStatusChanges needs unit tests <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1543408>
[04:18] <natefinch> thumper: yes it should be zero
[04:19] <natefinch> thumper: when we assign a unit to a machine we also remove that unit from the unit assignment collection
[04:19] <thumper> saw that, just wanted to confirm
[04:19] <thumper> ta
[04:39] <axw> wallyworld: :/  I've been working on credentials support for the clouds
[04:40] <wallyworld> oh, damn
[04:40] <wallyworld> sorry, i thought you were doing --config
[04:40] <axw> wallyworld: I did, then th other. never mind. I've done other clouds as well
[04:40] <wallyworld> i started adding joyent support and noticed we needed to do a bit of work
[04:42] <anastasiamac_> wallyworld: axw: PTAL http://reviews.vapour.ws/r/3787/
[04:43] <wallyworld> axw: so mine just does maas and joyent. with maas, i made the maas-server come from the cloud endpoint attribute in clouds.yaml
[04:43] <axw> wallyworld: yeah, I did the same in my branch. reviewing now
[04:43] <wallyworld> axw: i'm just resolving a conflict and pushing
[04:44] <axw> anastasiamac_: not sure why you would move controllerserver to jujuclient. it's not a client-side thing.
[04:49] <wallyworld> axw: that's my fault, i misread a question and thought we were renaming controllerserver to just controller at the top lovel to hold server side controller stuff
[04:49] <wallyworld> i don't like the name controllerserver
[04:50] <axw> nor do I
[04:50] <axw> environmentserver made some sense, controllerserver does not
[04:50] <wallyworld> it was the best i could think of at the time :-/
[04:50] <wallyworld> i don't really like environmentserver either
[04:51] <axw> wallyworld: not suggesting we go back, but there was some connection to the two words before
[04:51] <wallyworld> axw: atm we have controller and controllerserver at the top level. we choose just one i think
[04:51] <axw> a controller's a controller, adding server to the end doesn't change anything
[04:52] <axw> wallyworld: I think go with "controller". the things that *were* in controller have been moved to jujuclient
[04:52] <wallyworld> oh, ok, we'll fix that
[04:52] <wallyworld> i like controller also
[04:53] <wallyworld> not sure if it should stay a top level package, but ok for now
[05:00] <axw> wallyworld: reviewed
[05:00] <wallyworld> ty, noticed your config one, looking at that
[05:00] <axw> wallyworld: I'll do azure, cloudsigma, and vsphere now
[05:00] <wallyworld> ok, i promise i won't :-)
[05:01] <axw> :)
[05:01] <wallyworld> axw: with the manta url - that's going away as soon as storage is gone
[05:01] <axw> wallyworld: yep, as per comment
[05:02] <wallyworld> fair enough, i'll leave in comment till then, i was hoping storage would be gone this week or next
[05:43] <axw> wallyworld: replied to comment about Apply
[05:43] <wallyworld> ok
[05:45] <wallyworld> axw: fair point, i think, seems like the tests need updating which i'll look at. are you happy with the modified todo for the private key stuff?
[05:45] <axw> wallyworld: didn't read yet, one sec
[05:47] <axw> wallyworld: yep. you probably wouldn't want to enter it interactively, but we can read the file during interactive entry of the filename, and add the value
[05:47] <axw> wallyworld: then your credentials.yaml is protected from changes on disk.
[05:47] <axw> maybe not obvious though
[05:47] <axw> not sure, we can leave it for now
[05:47] <axw> covered the 99% case I think
[05:47] <wallyworld> i think the key on disk will be pretty static
[05:47] <wallyworld> yeah, all uses use file path afaik
[05:57] <wallyworld> axw: that should be good to go now
[05:58] <axw> wallyworld: thanks, shipit
[05:58] <wallyworld> tyvm
[05:59] <axw> wallyworld: just testing azure, should be ready to propose the rest very shortly
[05:59] <wallyworld> awesome
[05:59] <axw> wallyworld: there's still an issue with azure, another case where we need to be able to specify multiple endpoints
[06:00] <axw> wallyworld: in azure there's separate endpoints for storage and everything else, and they're not necessarily derivable
[06:00] <wallyworld> damn
[06:00] <axw> wallyworld: pretty sure we're going to have to extend the clouds.yaml format
[06:00] <wallyworld> seems so
[06:01] <wallyworld> do we *need* azure storage long term?
[06:01] <axw> wallyworld: yes. for volume support, and also some more basic operations like specifying where the VM image should live
[06:02] <wallyworld> storage-endpoint then i guess
[06:02] <wallyworld> axw: are the storage endpoints well know like the auth ones?
[06:02] <wallyworld> can we add them to publoc cloud.yaml
[06:03] <axw> wallyworld: yes, for azure public cloud. for azure stack you'd specify your own
[06:03] <wallyworld> axw: well seems like we should just update our public cloud yaml and cloud metadata struct them
[06:04] <axw> wallyworld: you mean with a new storage-endpoint field?
[06:04] <wallyworld> yeah
[06:05] <wallyworld> if it's not derivable
[06:05] <axw> wallyworld: I'm on the fence as to whether it should be specific to storage, rather than having a flexible map of <service>:<service-endpoint>. storage-endpoint is probably fine though
[06:06] <wallyworld> given it's optional, it keeps the default yaml nice and simple
[06:06] <wallyworld> for other clouds that don't need it
[06:06] <wallyworld> we can always get feedback and tweak
[06:06] <axw> wallyworld: ok, sounds fine
[06:07] <cherylj> can someone help me figure out why go test isn't actually testing anything in a particular directory?  http://paste.ubuntu.com/15000293/
[06:07] <axw> wallyworld: added a card to Next
[06:07] <wallyworld> no suite registered?
[06:08] <cherylj> the test ran in the merge bot, and failed, obviously
[06:08] <cherylj> but not when I do it locally
[06:08] <wallyworld> sigh, hate that
[06:08] <axw> cherylj: have you changed anything? changed a test file from package to package_test perhaps?
[06:08] <cherylj> axw: no, I didn't change any test files
[06:09] <anastasiamac_> wallyworld: axw: updated move \o/ PTAL?
[06:09] <cherylj> looks like even on master I see the same issue
[06:09] <cherylj> I change a test to fail, and it happily thinks there's nothing to test
[06:11] <axw> anastasiamac_: one sec
[06:11]  * anastasiamac_ waiting :D
[06:12] <cherylj> ugh, the suite_test.go was for peergrouper_test, and nothing else was
[06:12] <anastasiamac_> cherylj: m blind but where is package_test.go?
[06:12] <cherylj> wonder how long it's been like that
[06:12] <anastasiamac_> in worker/peergrouper...
[06:12] <cherylj> anastasiamac_: guess they're using suite_test.go, rather than package_test
[06:13] <anastasiamac_> could be the problem?..
[06:13] <axw> anastasiamac_: LGTM, thanks
[06:13] <anastasiamac_> axw: \o/
[06:13] <anastasiamac_> wow - 2 in one day!!
[06:15] <axw> wallyworld: http://reviews.vapour.ws/r/3790/ -- here's the rest
[06:15] <wallyworld> ta, looking
[06:27] <wallyworld> axw: reviewed, you may want to rebase first as my branch is almost landed and it will probably conflict in fallback public clouds yaml
[06:27] <axw> wallyworld: thanks, yep, will do
[06:27] <wallyworld> bbiab, school pickup
[06:44] <cherylj> wallyworld, axw can one of you review the test changes I had to make?  http://reviews.vapour.ws/r/3782/
[06:45] <cherylj> I'm going to add in the unit tests for the change and am tracking that work via bug 1543408
[06:45] <mup> Bug #1543408: WatchControllerStatusChanges needs unit tests <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1543408>
[06:45] <axw> cherylj: just the last diff?
[06:45] <axw> or all of it?
[06:45] <cherylj> axw: yes, the last diff
[06:45] <cherylj> I copied the mock watcher that was there and modified it to be a strings watcher
[06:46] <cherylj> had to pull suite_test.go into the peergrouper package.  I'll see about making the tests all external when I write the tests for the functional change.
[06:46] <cherylj> it wasn't a no-op this time, so I skipped it
[06:49] <axw> cherylj: ignoring the error from ControllerInfo in state watcher seems a bit alarming. why not just error out there?
[06:49] <axw> (sorry, couldn't help myself and looked at the rest)
[06:50] <cherylj> axw: if we're getting an error there, we'll get that error elsewhere and things will get restarted
[06:50] <cherylj> that was my thinking anyway
[06:50] <axw> cherylj: so it's not harmful to error out there as well right?
[06:50] <cherylj> there's not really a way to return an error there, from what I saw.
[06:50] <axw> oh. /me looks again
[06:50] <blahdeblah> cherylj: have you moved to Australia, or do you just hate sleep? :-)
[06:50] <cherylj> I miss sleep
[06:51] <cherylj> we used to be buddies
[06:51] <cherylj> now he's all emaciated because I don't feed him
[06:51] <blahdeblah> That's no way to treat your buddies. :-)
[06:51] <axw> cherylj: right, there's not, sorry.
[06:51] <axw> cherylj: LGTM
[06:52] <cherylj> thanks, axw!
[06:53] <cherylj> and now, SLEEEEEP
[06:53] <blahdeblah> be nice to your buddies!
[06:54] <blahdeblah> cherylj: BTW, thanks for some of those recent bug fixes.
[06:55] <davechen1y> cherylj: i have a strong suspcion that bootstrap is not making it to waitForInitalisatin
[06:55] <davechen1y> I think it's bombing out _WAY_ earlier
[06:56] <anastasiamac_> davechen1y: shh... cherylj is sleeping \o/
[07:15] <axw> the new bootstrap syntax has already infected my brain. switching back and forth between 1.25 and 2.0 is going to be fun
[07:19] <wallyworld> anastasiamac_: looks like the controller.yaml file isn't quite right - it looks like it is storing model information as well as controllers
[07:19] <wallyworld> it also doesn't have the local. prefix
[07:19] <wallyworld> for the controller name
[07:27] <axw> davechen1y: it appears that piping stuff to bash causes bash to read a character at a time
[07:27] <axw> davechen1y: as in, piping commands
[07:44] <davechen1y> oh wow
[07:44] <davechen1y> that's rubbish
[07:53] <axw> wallyworld: would you like me to look at updating "switch", or are you doing that?
[08:12] <davechen1y> only in juju would our rpc layer have a field called "Code", that was a string ...
[08:14] <anastasiamac_> axw: RB is not picking this up.. Could you PTAL on github? https://github.com/juju/juju/pull/4347
[08:19] <anastasiamac_> wallyworld: so... according to the spec, "local" is a prefix that is added to model name at bootstrap.
[08:19] <anastasiamac_> wallyworld: to me, this reads as a separte PR unrelated to my controllers.yaml work
[08:20] <anastasiamac_> wallyworld: model name will be transformed before the file is written, hence, my work will just pick it automatically
[08:20] <anastasiamac_> wallyworld: ping me when u r back \o/
[08:20] <anastasiamac_> wallyworld: why there is model info in the file now, is probably an "oversight" :D i'll look
[08:21] <axw> anastasiamac_: done
[08:21] <anastasiamac_> axw: awesome \o/
[08:22] <anastasiamac_> axw: would we want to differntiate btw "controllers.lock", "models.lock", "accounts.lock" for different files?
[08:23] <axw> anastasiamac_: probably not. models and accounts are both related to controllers
[08:24] <anastasiamac_> axw: k. thnx
[08:24] <axw> anastasiamac_: meaning: removing a controller should remove related info
[08:24] <axw> anastasiamac_: so you can't not lock the controllers file if you're modifying models file
[08:25] <axw> and vice versa
[08:25] <anastasiamac_> axw: to what m observing, we lock a dir not a file...
[08:25] <axw> anastasiamac_: for now, yes
[09:10] <dimitern> frobware, voidspace, dooferlad, I've updated the PR to make the diff a bit easier to follow and included a fix after live testing on maas:
[09:10] <dimitern> frobware, voidspace, dooferlad, so please have a look when you can http://reviews.vapour.ws/r/3773/
[09:21] <voidspace> dimitern: ok
[09:44] <axw> davechen1y: would appreciate if you could test https://github.com/juju/juju/pull/4349 tomorrow, and see if it resolves the issue for you. fairly sure it does, but need to get on with other things
[09:48] <dimitern> axw, I think this should be applied to AddScripts as well - specifically in maas now we push a whole lot of python code to set up the bridge script
[09:49] <dimitern> ..or perhaps that includes all runcmd / bootcmd scripts already
[09:49] <axw> dimitern: is that going to pipe commands to bash? I don't think so
[09:49] <axw> dimitern: yeah, this is the entire cloud-config script
[09:51] <dimitern> axw, right, I've just noticed in maas we do a similar thing for the bridge script - put it in /tmp, trap exit and run it
[10:15] <kjackal> Hey everyone, does any one know if/how is it possible to access relation/conversation data from within an action?
[10:31] <voidspace> jam: dooferlad: dimitern: sorry, screwed my network temporarily
[10:45] <dimitern> jam, sorry I was too quick - what were you about to say?
[10:46] <jam> dimitern: just wanted to check if anyone has heard from alexisb today. Usually I have a 1:1 with her last night, but she missed it, and didn't reply to my email, which is unlike her
[10:47] <dimitern> jam, nope - but according to the calendar she was off yesterday
[10:47] <jam> dimitern: sigh. I had turned of "Juju Team Calendar" when I was at the Cape Town sprint, so that's why I didn't see it.
[10:48] <jam> dimitern: thanks for noticing and checking. team calendar is back on.
[10:53] <mup> Bug #1543517 opened: status command tests fail when you have dns spoofing <juju-core:New> <https://launchpad.net/bugs/1543517>
[10:54] <dimitern> :) np
[10:57] <voidspace> dimitern: when a user specifies multiple spaces for deployment constraints are we still only using the first?
[10:57] <voidspace> dimitern: apiserver/provisioner/provisioninginfo.go line 224
[10:57] <voidspace> dimitern: we should fix that soon I think
[11:02] <mup> Bug #1543517 changed: status command tests fail when you have dns spoofing <juju-core:New> <https://launchpad.net/bugs/1543517>
[11:05] <mup> Bug #1543517 opened: status command tests fail when you have dns spoofing <juju-core:New> <https://launchpad.net/bugs/1543517>
[11:08] <voidspace> dimitern: LGTM on your PR
[11:08] <voidspace> dimitern: the basic approach seems sound and I have no specific suggestions for it
[11:08] <voidspace> dimitern: there's a lot of context on the code this touches I'm missing, so you may want someone else to look at it too
[11:13] <dimitern> voidspace, sure, the more sets of eyes the better
[11:13] <dimitern> voidspace, cheers
[11:14] <dimitern> voidspace, yeah, that's due to aws
[11:14] <dimitern> voidspace, (ignoring all but the first space)
[11:14] <dimitern> voidspace, but in maas we actually use the spaces constraints for a machine directly
[11:40] <wallyworld> anastasiamac_: hey. yes it will be a separate PR, just wanted to let you know that it needed looking at as part of the overall work
[11:56] <wallyworld> voidspace: you found the problem, jeez what a strange issue
[12:06] <anastasiamac_> wallyworld: \o/ added card
[12:06] <wallyworld> ty
[12:41] <voidspace> wallyworld: yeah, took most of the day
[12:42] <wallyworld> voidspace: you may be interested in https://bugs.launchpad.net/bugs/1539428
[12:42] <mup> Bug #1539428: cmd/juju/status: status filtering performs IP resolution of patterns <status> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1539428>
[12:42] <wallyworld> related to your issue
[12:42] <voidspace> wallyworld: indeed
[12:45] <voidspace> wallyworld: so getting rid of the resolution would be a double win.
[12:45] <wallyworld> yes :-)
[12:46] <wallyworld> it will have to be fixed for 2.0 i think
[12:51] <dimitern> dooferlad, replied to your mail
[12:53] <dimitern> dooferlad, let me know if that helps
[13:13] <voidspace> frobware:  dimitern: trivial one for you http://reviews.vapour.ws/r/3796/
[13:16] <dimitern> voidspace, looking
[13:27] <dimitern> voidspace, reviewed
[13:31] <dimitern> dooferlad, ping
[14:02] <dooferlad> dimitern, voidspace: is the MAAS meeting happening? I am the only person in there...
[14:05] <dimitern> dooferlad, sorry got distracted - is it still going?
[14:05] <dooferlad> dimitern: just starting
[14:05] <dimitern> dooferlad, omw
[14:10] <voidspace> omw too
[14:48] <dimitern> dooferlad, did you had a chance to look at my PR #4331 ?
[14:49] <dooferlad> dimitern: I thought voidspace had
[14:49] <dooferlad> dimitern: happy to if it needs more eyes
[14:49] <dimitern> dooferlad, I'd appreciate it, thanks
[14:49] <dimitern> dooferlad, as I'd like to get it in today, if possible and have another one almost ready to propose
[14:52] <voidspace> dooferlad: I did review it and it looks sound to me, but I have a lot of missing context so I think another pair of eyes would be useful
[16:06] <dooferlad> dimitern: LGTM. I would appreciate some card/bug links, but I didn't require them because it is in old code.
[16:08] <dimitern> dooferlad, thank you
[16:12] <mup> Bug #1497301 opened: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>
[16:14] <cherylj> hey perrito666, looks like a couple bugs on the mongo3 branch need some love:  bug 1534620 and bug 1497301
[16:14] <mup> Bug #1534620: TestMongo26UpgradeStep fails on windows because of dos paths <ci> <regression> <test-failure> <windows> <juju-core:Incomplete> <juju-core update-mongodb3:Triaged> <https://launchpad.net/bugs/1534620>
[16:14] <mup> Bug #1497301: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>
[16:15] <mup> Bug #1497301 changed: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>
[16:24] <cherylj> hey dimitern, how are things going with maas-spaces?
[16:24] <mup> Bug #1497301 opened: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>
[16:25] <dimitern> cherylj, hey
[16:26] <dimitern> cherylj, we're tracking master daily and wait for some CI feedback from the changes we pushed in maas-spaces so far (4 days we had no CI run of maas-spaces)
[16:26] <dimitern> cherylj, we're also ~2d way from having a hopefully blessed maas-spaces
[16:26] <voidspace> dimitern: ping
[16:26] <dimitern> voidspace, pong
[16:27] <voidspace> dimitern: the card I'm working on is "provider/maas: Addresses() to set SpaceProviderId not SpaceName"
[16:27] <voidspace> dimitern: there is no provider/maas Addresses method
[16:27] <voidspace> dimitern: and in fact "SpaceName" doesn't appear at all in provider/maas/environ.go
[16:28] <dimitern> voidspace, it's in instance.go or interfaces.go IIRC
[16:28] <voidspace> dimitern: ah yes it is
[16:29] <voidspace> dimitern: better use of grep just found it in instance.go
[16:29] <voidspace> dimitern: sorry for the noise :-)
[16:29] <alexisb> dimitern, are there still changes the team needs to push to the maas spaces branch for a CI bless?
[16:29] <voidspace> regex for the win
[16:29] <voidspace> alexisb: yes
[16:29] <voidspace> alexisb: proper handling of controller space
[16:29] <alexisb> voidspace, and we are looking at 2 days for those fixes?
[16:29] <voidspace> alexisb: the failures related to "Default Space" actually highlighted an important problem which we've nearly fixed
[16:30] <voidspace> alexisb: yes
[16:30] <alexisb> ok
[16:30] <voidspace> we shouldn't have been using the default space at all and we *must* know the space to deploy controllers too
[16:30] <alexisb> voidspace, dimitern would a CI run on maas-spaces now be useful to validate landed fixes?
[16:30] <voidspace> alexisb: yes
[16:30] <alexisb> voidspace, thank you
[16:30] <dimitern> alexisb, yes, absolutely - we didn't have one for 4 days
[16:30] <voidspace> alexisb: space discovery problems should be completely fixed and we haven't had a full run since those fixes landed
[16:30] <alexisb> we need to stay in daily contact regarding progress on this branch
[16:31] <alexisb> voidspace, ack
[16:31] <dimitern> alexisb, we're hopefully down to 2-3 failures now compared to master, but still need to get a CI run to be sure
[16:31] <alexisb> sinzui, mgz, can we please get a run on the maas spaces branch
[16:31] <sinzui> alexisb: it is running
[16:31] <voidspace> alexisb: FYI  the three yellow cards in our "In Progress" kanban lane are tracking progress
[16:31] <voidspace> alexisb: https://canonical.leankit.com/Boards/View/101652562#workflow-view
[16:32] <voidspace> sinzui: thanks
[16:32] <alexisb> voidspace, awesome thank you
[16:51] <mup> Bug #1543660 opened: juju needs to fix the yaml parser <customer> <juju-core:New> <https://launchpad.net/bugs/1543660>
[17:08] <dooferlad> dimitern, voidspace: EOD for me. Still watching email.
[17:10] <cherylj> hey tych0, looks like CI on your branch had more build failures:  http://reports.vapour.ws/releases/3586  (non-trusty build failures)
[17:11] <dimitern> dooferlad, cheers, but I need to go soon as well
[17:12] <natefinch> katco: can you double check with rick_h__ that we're dropping comment for now?
[17:12] <rick_h__> natefinch: yes
[17:12] <katco> natefinch: doh, sorry i did. we are.
[17:12] <natefinch> oh good :)
[17:13] <katco> natefinch: sorry, forgot to include that in the email. ericsnow fyi ^^^
[17:28] <natefinch> ericsnow: did you have something you needed reviewed?
[17:29] <ericsnow> natefinch: http://reviews.vapour.ws/r/3778/
[19:09] <natefinch> ericsnow: how much of cmd/juju/charmcmd/store.go is copied from elsewhere?
[19:10] <ericsnow> natefinch: nearly all of it
[19:10] <natefinch> ericsnow: I thought so.  Can you put in comments as to that effect?  Then I don't have to try so hard to figure out why it's doing all this stuff.
[19:11] <natefinch> ericsnow: like directly in the function bodies that have been copied
[19:12] <ericsnow> natefinch: sure
[19:12] <ericsnow> natefinch: oh, and internal tests are making my life miserable
[19:13] <natefinch> ericsnow: how are internal tests making your life miserable?
[19:13] <ericsnow> natefinch: import cycles, and fixing them is a ton of work :(
[19:13] <natefinch> ericsnow: ahh boo.... well, import cycles are a valid reason to use external tests
[19:14] <ericsnow> natefinch: that doen't help me now though
[19:14] <natefinch> ericsnow: sorry :/
[19:15] <natefinch> ericsnow: is this because of existing internal tests?  If so, I'd definitely like to hear details at some point, so we can try to avoid this in the future.
[19:30] <natefinch> ericsnow, katco: trivial rename in charm metadata: https://github.com/juju/charm/pull/190
[19:31] <katco> natefinch: +1 from me!
[19:32] <ericsnow> natefinch: same here
[20:00] <natefinch> ericsnow, katco: I have the s/comment/description patch for juju ready, but it's dependent on my other patch that's up for review, and I can't make rbt show the right diff.... so I'll just leave it until the other one lands.  It's another trivial diff anyway.
[20:01] <katco> natefinch: ok that sounds fine
[20:01] <natefinch> katco: so, should I grab the --debug card?  or should we point the upgrade-charm card?
[20:02] <katco> natefinch: grab the --debug card since that's the last user-facing bit
[20:06] <natefinch> katco: ok
[20:21] <katco> natefinch: is this something specific to resources? https://github.com/juju/juju/commit/2910afceacca26dc1d880cedaa5d7560b249ebbe#diff-5c78256455800f538886c2beef98045aR66
[20:21] <katco> natefinch: doesn't seem to be in master at any point
[20:22] <mup> Bug #1543770 opened: Juju 2.0alpha1.1 does not assign a proper IP address for LXC containers <dhcp> <lxc> <maas> <network> <juju-core:New> <https://launchpad.net/bugs/1543770>
[20:22] <natefinch> katco: you mean the arguments struct?
[20:22] <katco> natefinch: specifically "toMachineSpec"
[20:23] <natefinch> katco: it is there on the left side of the diff... that's what juju deploy foo --to 3  gets turned into
[20:23] <natefinch> katco: possible that it's been changed in master?
[20:23] <katco> natefinch: it has; trying to resolve the diff
[20:24] <katco> natefinch: i was mistaken, just found it's lineage in master
[20:24] <katco> *its
[20:24] <natefinch> katco: certainly, it doesn't have anything to do with resources.. I was just refactoring the args into a struct.
[20:24] <katco> natefinch: thx that makes the merge much easier :)
[20:24] <natefinch> katco: cool
[20:40] <natefinch> is there a function to get the unit number from the unitID?
[20:41] <natefinch> like, I can certainly write one, but I'd rather reusing someone else's
[20:42] <menn0_> thumper: when you have a chance, could you look at my replies to your comments on http://reviews.vapour.ws/r/3745/ ?
[21:02] <aznashwan> any charm-tools people I could bother with a couple of reviews?
[21:02] <thumper> menn0: ok
[21:03] <natefinch> rick_h__, ericsnow, katco:  should we support juju resources foo/0 --debug, and only show detailed information about that particular unit's resources?
[21:03] <natefinch> (currently the spec only talks about juju resources foo --debug)
[21:04] <rick_h__> natefinch: yes, I think it should be able to go into per unit as well
[21:04] <natefinch> rick_h__: ok. luckily, that's not much more work
[21:04] <rick_h__> :)
[21:04] <thumper> menn0: I'm tempted to call yagni and just use user tags
[21:05] <natefinch> rick_h__: what should we show if the unit hasn't called resource-get for a resource?
[21:05] <natefinch> revision "-" or something?
[21:05] <rick_h__> natefinch: otp atm
[21:09] <natefinch> thumper, menn0: If yagni is an option, it's almost always the right choice.  "Just in case" just makes your code confusing when it can really only be one thing right now. (IMO)
[21:09] <menn0> thumper: i'm fine with that. The facade can always be bumped later.
[21:11] <menn0> natefinch: I generally agree with that too. In this case the cost difference between either approach is nominal so it wasn't clear which way to go.
[21:12] <natefinch> menn0: yeah. I generally prefer more specific types, so they convey the right information.  Seeing an API that takes a names.Tag, but really only one type of tag ever gets passed in, can easily make someone write the wrong code, trying to support a bunch of arguments that they don't really even need to worry about.
[21:13] <menn0> natefinch: yep, agreed.
[21:13] <menn0> natefinch: I'm going with the more specific type.
[21:42] <tych0> cherylj: ok, i'll take a look now. was distracted by some other stuff this morning
[21:56] <tych0> cherylj: https://github.com/juju/juju/pull/4355
[21:57] <tych0> looks like somehow that commit got dropped in all the shuffling. anyway, I think that should fix at least the problems you're seeing now.
[21:57] <tych0> (assuming the i18n stuff was the only issue, that's what i saw in my monte carlo sampling :)
[22:38] <menn0> thumper: can you pls take another look at https://github.com/juju/juju/pull/4303 ?
[22:39] <menn0> thumper: also http://reviews.vapour.ws/r/3746/ pls (tiny)
[23:29] <davecheney> here is a simple review https://github.com/juju/juju/pull/4357
[23:29] <davecheney> anyone ?
[23:31] <mup> Bug #1543839 opened: juju/service: test fixture panics if SetUpSuite fails <juju-core:New> <https://launchpad.net/bugs/1543839>
[23:34] <mup> Bug #1543839 changed: juju/service: test fixture panics if SetUpSuite fails <juju-core:New> <https://launchpad.net/bugs/1543839>
[23:40] <mup> Bug #1543839 opened: juju/service: test fixture panics if SetUpSuite fails <juju-core:New> <https://launchpad.net/bugs/1543839>