davechen1ythumper: cherylj00:36
davechen1ymachine-0: 2016-02-09 00:36:41 INFO juju.apiserver apiserver.go:302 [9] user-admin@local API connection terminated after 7.391830549s, active connections: 500:36
davechen1ymachine-0: 2016-02-09 00:36:41 INFO juju.apiserver apiserver.go:302 [A] user-admin@local API connection terminated after 2.046902769s, active connections: 400:36
davechen1ymachine-0: 2016-02-09 00:36:41 INFO juju.apiserver apiserver.go:302 [C] user-admin@local API connection terminated after 91.680242ms, active connections: 300:37
davechen1y^ this is what I can do00:37
thumperdavechen1y: active connections is new current count?00:41
davechen1ycherylj: 2016-02-08 22:30:16 ERROR Command '('juju', '--debug', 'deploy', '-m', 'maas-1_9-deploy-trusty-amd64', 'local:trusty/dummy-sink')' returned non-zero exit status 100:43
davechen1ywhere can I find the source for this charm ? assuming the charm matters00:43
cheryljdavechen1y: https://private-fileshare.canonical.com/~cherylj/dummy-charms/00:44
cheryljThere's a tar file there, and some copied commands that the CI test uses00:44
davechen1yis this environement a ha environment ?00:45
cheryljbut I imagine the charm doesn't matter00:45
cheryljdavechen1y: probably not, but you can double check by looking at the test run00:45
davechen1yyeah, i deployed some of my usual favorites00:45
davechen1yi'm bloody sick of the tools versino checker00:45
davechen1yis that on the top of a list for someone to fix ?00:45
davechen1ychecking every 3 seconds is stupid00:46
davechen1y15 minutes would be sufficient00:46
cheryljdavechen1y: no, there are other more serious failures that everyone's working on00:46
cheryljunfortunately, even.   Because that one is darn annoying00:46
cheryljdavechen1y: no, the env is not HA00:46
davechen1yjuju debug-log pauses if you run it after calling enable-ha00:47
davechen1yso does juju ssh00:48
davechen1ythat's wonderful00:48
davechen1yjuju enable-ha00:48
davechen1yputs your environment into catatonia until that process finishes00:48
cheryljdoesn't surprise me.  everything dies / pauses when you enable-ha00:49
davechen1ycherylj: so what's happened is00:54
davechen1ythe addresses of the additional mongo servers have been added to the list of state servers00:55
davechen1ybut those additional mongos are up00:55
davechen1ysorry, not up00:55
davechen1ythey are still going through the cloud=init dance00:55
davechen1yso you have a 2/3 chance that the apiserver running on machine-0 will try to connect to those00:55
davechen1y2016-02-09 00:56:04 DEBUG juju.worker.peergrouper desired.go:116 machine "0" is already voting00:56
davechen1y2016-02-09 00:56:04 DEBUG juju.worker.peergrouper desired.go:123 machine "2" is not ready (has status: true)00:56
davechen1y2016-02-09 00:56:04 DEBUG juju.worker.peergrouper desired.go:123 machine "3" is not ready (has status: true)00:56
davechen1yyet the api server still tries to dial it :)00:56
davechen1y2016-02-09 00:55:49 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp getsockopt: connection refused00:56
davechen1y2016-02-09 00:55:49 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp getsockopt: connection refused00:56
davechen1y2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp getsockopt: connection refused00:56
davechen1y2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp getsockopt: connection refused00:56
davechen1y2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp getsockopt: connection refused00:56
davechen1y2016-02-09 00:55:50 DEBUG juju.mongo open.go:117 connection failed, will retry: dial tcp getsockopt: connection refused00:56
cheryljdavechen1y: but the two bugs that you're looking at aren't using ha00:57
davechen1yi won't get distracted00:58
davechen1yi was just using enable-ha to try to get the apiserver to explode00:58
davechen1yand enter it's restart behaviour00:58
cheryljdavechen1y: ah, I was confused00:59
davechen1yi shouldn't go poking around in juju01:00
davechen1ythere be dragons01:00
cheryljain't that the truth.01:01
davechen1yand it's failed01:03
davechen1yyou'll love this01:03
davechen1yAttempt 63 to download tools from
davechen1y+ curl -sSfw tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s  --noproxy * --insecure -o /var/lib/juju/tools/2.0-alpha2.1-precise-amd64/tools.tar.gz
davechen1ycurl: (7) couldn't connect to host01:03
davechen1ytools from downloaded: HTTP 000; time 0.001s; size 0 bytes; speed 0.000 bytes/s + echo Download failed, retrying in 15s01:03
davechen1yDownload failed, retrying in 15s01:03
davechen1ymachine-2 is trying to bootstrap01:03
davechen1yit needs tools from machine-001:03
davechen1ymachine-0's agent is trying to connect to the replica set01:03
davechen1ythe replica set is down because it's trying to ensure ha01:04
davechen1yso, no tools01:04
davechen1yno bootstrap01:04
davechen1yno ha01:04
davechen1yno tools01:04
davechen1ythis is even weirder01:05
davechen1ythis is another case of machine-0 not listening on port 1701701:06
davechen1yI need to look into this01:06
davechen1yif that port is not open, no api server01:06
davechen1y2016-02-09 01:36:02 INFO juju.apiserver apiserver.go:302 [1] machine-0 API connection terminated after 3m35.591602712s, active connections: 001:39
davechen1y2016-02-09 01:36:02 INFO juju.apiserver apiserver.go:325 closed listening socket "[::]:17070" with final error: <nil>01:39
davechen1ygets more and more interesting01:39
davechen1ythe api server shuts down, but never starts back up again01:40
davechen1yOH MY GODS01:46
davechen1ythe bit where we send data to bash via ssh01:46
davechen1yafter this line01:46
davechen1y2016-02-09 01:43:36 DEBUG juju.utils.ssh ssh.go:249 using OpenSSH ssh client01:46
davechen1ywe're sending ONE CHARACTER AT A TIME01:46
davechen1yaxw: ping01:46
axwdavechen1y: pong01:47
axwsay whaaat01:47
axwdavechen1y: where's hte code that's doing that?01:48
mupBug #1543388: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>01:48
davechen1yi was watching the bootstrap01:48
davechen1yand i'm like why is top using 48% cpu01:48
davechen1yso I straced it01:48
davechen1yread(0, "e", 1)                         = 101:48
davechen1yread(0, "H", 1)                         = 101:48
davechen1yread(0, "t", 1)                         = 101:48
davechen1yread(0, "D", 1)                         = 101:48
davechen1yread(0, "N", 1)                         = 101:48
davechen1yread(0, "y", 1)                         = 101:48
davechen1yread(0, "q", 1)                         = 101:48
davechen1yread(0, "5", 1)                         = 101:48
davechen1yread(0, "g", 1)                         = 101:48
davechen1yread(0, "H", 1)                         = 101:48
davechen1yread(0, "/", 1)                         = 101:48
davechen1yread(0, "S", 1)                         = 101:48
davechen1yread(0, "o", 1)                         = 101:48
davechen1yread(0, "5", 1)                         = 101:48
davechen1yread(0, "t", 1)                         = 101:48
davechen1yread(0, "6", 1)                         = 101:49
davechen1yread(0, "C", 1)                         = 101:49
mupBug #1543388 opened: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>01:49
mupBug #1543388 changed: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>01:55
mupBug #1543388 opened: bootstrapping talks to the remote machine one character at a time <juju-core:New> <https://launchpad.net/bugs/1543388>01:58
davechen1ycan tomb.Kill block ?02:04
=== natefinch-afk is now known as natefinch
natefinchdavechen1y: it does acquire a lock, so, in theory, yes.02:06
natefinchdavechen1y: but otherwise, it just closes the dying channel02:06
* thumper takes a deep breath and dives into importing units02:06
* natefinch looks up the time formatting date for the 1000th time02:13
davechen1ynatefinch: https://github.com/juju/juju/blob/master/apiserver/apiserver.go#L9602:16
davechen1yso here's the thing02:16
davechen1ythe only way processCertChanges can exit is if someone called tomb.Kill02:16
davechen1yand hte only thing that can is cl.Close02:16
davechen1yso this code calls tomb.Kill twice, then tomb.Done for good measure ...02:16
davechen1yseems like overkill02:16
cheryljwallyworld: can you take a look at my enable-ha change:  http://reviews.vapour.ws/r/3782/02:16
wallyworldcherylj: did axw ping you about possibly making a non blocking channel send?02:17
natefinchdavechen1y: yep... also, it doubly bad, because someone might see cl.tomb.Kill(cl.processCertChanges()) and think they don't need those other lines, but that line won't ever kill the tomb02:17
cheryljwallyworld: no, not yet02:18
wallyworldwas maybe an alternative to increasing the channel buffer size02:18
wallyworldbut the buffer size increase might be acceptable for now until manifolds are done for those workers02:19
cheryljokay, can take a look later.   I'm going to be afk for ~40 mins or so, but I'll be back02:19
axwcherylj: wallyworld: wasn't going to bother, since we need to change it again. non-blocking send wouldn't work here anyway, it's not just a notify chan02:20
wallyworldok, just wanted to double check, ty :-)02:20
natefinchchannels with buffers that are not 0 or 1 are pretty suspect, in my experience02:20
axwunless it's directly next to the things populating the channel, I tend to agree02:21
natefinchthis sounds like one of those cases where you could have an arbitrary number of sends on the channel before any reads from the channel, in which case any buffer size could be insufficient02:24
wallyworldnot arbitrary - is equal to the number of allowed controllers (7) plus a known number of local addresses02:28
natefinchrick_h___: noted, about the name=file for resources.02:28
wallyworldit's a short term fix until workers migrated to dep engine02:28
natefinchwallyworld: ahh, the fact that it's a short term fix makes a big difference.02:29
wallyworldfor 2.0, sadly dep engine not going into 1.2502:29
wallyworldso not sure what to do there02:30
natefinchwallyworld: well, we're not going to support 1.25 for very long, right? ;)02:30
wallyworld2 years :-(02:30
davechen1ynatefinch: wahhahahahahaha02:30
natefinchwallyworld: I know, I was joking02:30
wallyworldbut 1.25 is blocked right now, so need to get 1.25.4 out02:30
davechen1yo/ gallows humor02:30
natefinchwallyworld: just make the channel buffer 128... larger powers of 2 are always better02:31
wallyworldi'll add a comment02:31
natefinchwallyworld: definitely comment why the 10 is 10.02:32
davechen1yaxw: 2016-02-09 02:27:09 DEBUG juju.utils.ssh ssh.go:249 using OpenSSH ssh client02:32
davechen1ywhat happens after this line02:32
davechen1ysomethign in bootstrap02:32
davechen1ybut it's mute until the other side starts to output things02:33
axwdavechen1y: umm. could be one of a few things, that debug message gets printed whenever an ssh client is created02:33
axwdavechen1y: first we ssh to each of the possible controller addresses02:33
axwdavechen1y: then (if you're uploading tools), copy tools across via ssh02:34
axwdavechen1y: then run the cloud-config rendered as a bash script02:34
axwI think that's it02:34
axwdavechen1y: AFAICR, we just open "ssh" with the script as a bytes.Buffer piped to the ssh process's stdin02:35
axwdavechen1y: could be that ssh is in an interactive mode? looking for escape codes?02:35
davechen1y13:34 < axw> davechen1y: then (if you're uploading tools), copy tools across via ssh02:35
davechen1y^ it'll be this02:35
davechen1yi'm spelunking in the code now02:36
davechen1yshort version is the openssh impl doens't buffer stdout/stdin02:36
davechen1yor something02:36
axwdavechen1y: actually, gross as it is, we just add the contents of the tools to the bash script (base64 encoded or something)02:38
axwso the 2nd and 3rd steps are just one02:38
natefinchrick_h___: you around?02:43
davechen1yaxw: that's fine03:00
davechen1yi knew we bas64'd them03:00
davechen1ythe problem is something is unbuffered there and it's sending one character at at time over ssh03:00
davechen1ywhich is going to turn each byte into about 40003:00
davechen1ymabye 20003:00
davechen1ybut it's a lot03:00
davechen1yand the cpu on both sides is non trivial03:00
axwdavechen1y: yep. I had a look, nothing obvious. what were you stracing exactly? bash on the remote side? ssh on the client?03:05
davechen1yremote side03:07
davechen1ybash is hitting 50% cpu03:07
davechen1yresults are in that ticket03:07
axwdavechen1y: ok, will take another look later03:09
davechen1yumm, https://github.com/juju/juju/blob/master/api/apiclient.go#L55403:20
davechen1ynatefinch: axw https://github.com/juju/juju/blob/master/api/apiclient.go#L55403:20
davechen1ythis construction is unsafe03:21
axwdavechen1y: you mean because two calls could race?03:23
axwdavechen1y: if so yeah.. pretty sure it's always one thing's responsibility to close though.03:24
axwso in theory, but not in practice (unless we're doing something dumb, which I wouldn't rule out)03:24
davechen1ythis would have to be hit way harder than we could03:26
davechen1ybut it's entirely possible to hit this03:26
mupBug #1543404: unsafe double channel close idiom <juju-core:New> <https://launchpad.net/bugs/1543404>03:26
natefinchthat code in apiclient doesn't look like it's intended to be threadsafe, so hopefully we're not trying to use it from multiple goroutines03:27
natefinchlol panics03:28
davechen1ysooo, http://paste.ubuntu.com/14999670/03:34
davechen1yno matter what logging I add, i cannot get line 71 to output something03:34
davechen1yall I can think of is somehow tools are being cached03:35
davechen1yand i'm not pushing up what I think i'm pushig up03:35
natefinchdavechen1y: log.Infof, not logger?03:36
davechen1yOH FOR FUCKS SAKE03:38
* davechen1y wonders what log was in this scope ...03:38
natefinchheh, np.  One of those things your brain just can't see if you were the one that wrote it.03:39
davechen1ysooo, amazong just gave me a machien without a public ip03:43
davechen1yhas that ever happened to anyone ?03:43
davechen1yit has no public ip or public dns03:44
cheryljnatefinch, wallyworld, so should I do 10 or 128?03:46
mupBug #1543404 opened: unsafe double channel close idiom <juju-core:New> <https://launchpad.net/bugs/1543404>03:46
wallyworld128 according to nate03:46
natefinchcherylj: I was joking03:46
cheryljha, ok :)03:46
natefinchcherylj: I do fear that 10 will just error out less often... but *shrug* Seems better than 1 :/03:47
cheryljnatefinch: in practice, I see it firing twice03:47
cheryljif that helps :)03:47
wallyworld10 has some science behnd it03:47
cheryljno, not science, it's http://i.imgur.com/24Jw4gM.gif03:48
wallyworldlol, educate dguess then03:48
davechen1ycherylj: 1 should be enough03:48
wallyworldbased on knowledge of the system03:48
cheryljhehe.  I love that gif03:48
cheryljdavechen1y: I've seen that it's not03:49
davechen1yif you just want to hand off the value between producer and consumer without either blocking03:49
cheryljit was 1 before03:49
cheryljand that was the problem03:49
davechen1ycherylj: shit03:49
davechen1ythat's more serious03:49
cheryljit was sending twice03:49
davechen1yif it was already buffered03:49
davechen1ywhat about making the recieve side timeout03:49
cheryljI'm not sure I see how we would do that.   The receiver is blocked waiting on a lock that the sender is holding03:50
natefinchdavechen1y: it's a clusterfuck that is getting rewritten soonish03:50
natefinchdavechen1y: thus, the bigger buffer is just a stopgap03:50
cheryljand a band aid for 1.25 :)03:50
davechen1ywhee 1.25 cluserfuck, keeps for 2 years even under adverse conditions03:50
cheryljbut, we should explore the other option menn0 suggested for 1.25 - where there's some other synchronization between certupdater and apiserver03:53
thumpernatefinch: still around?04:06
thumperwanting to verify that the assign units collection is a transitory collection04:07
thumpermeaning that once all units have been assigned to machines, the length of that collection should be zero04:07
davechen1ythumper: cherylj http://reviews.vapour.ws/r/3784/04:12
axwthumper: that is correct.04:13
thumperaxw: ta04:13
mupBug #1543408 opened: WatchControllerStatusChanges needs unit tests <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1543408>04:16
natefinchthumper: yes it should be zero04:18
natefinchthumper: when we assign a unit to a machine we also remove that unit from the unit assignment collection04:19
thumpersaw that, just wanted to confirm04:19
axwwallyworld: :/  I've been working on credentials support for the clouds04:39
wallyworldoh, damn04:40
wallyworldsorry, i thought you were doing --config04:40
axwwallyworld: I did, then th other. never mind. I've done other clouds as well04:40
wallyworldi started adding joyent support and noticed we needed to do a bit of work04:40
anastasiamac_wallyworld: axw: PTAL http://reviews.vapour.ws/r/3787/04:42
wallyworldaxw: so mine just does maas and joyent. with maas, i made the maas-server come from the cloud endpoint attribute in clouds.yaml04:43
axwwallyworld: yeah, I did the same in my branch. reviewing now04:43
wallyworldaxw: i'm just resolving a conflict and pushing04:43
axwanastasiamac_: not sure why you would move controllerserver to jujuclient. it's not a client-side thing.04:44
wallyworldaxw: that's my fault, i misread a question and thought we were renaming controllerserver to just controller at the top lovel to hold server side controller stuff04:49
wallyworldi don't like the name controllerserver04:49
axwnor do I04:50
axwenvironmentserver made some sense, controllerserver does not04:50
wallyworldit was the best i could think of at the time :-/04:50
wallyworldi don't really like environmentserver either04:50
axwwallyworld: not suggesting we go back, but there was some connection to the two words before04:51
wallyworldaxw: atm we have controller and controllerserver at the top level. we choose just one i think04:51
axwa controller's a controller, adding server to the end doesn't change anything04:51
axwwallyworld: I think go with "controller". the things that *were* in controller have been moved to jujuclient04:52
wallyworldoh, ok, we'll fix that04:52
wallyworldi like controller also04:52
wallyworldnot sure if it should stay a top level package, but ok for now04:53
axwwallyworld: reviewed05:00
wallyworldty, noticed your config one, looking at that05:00
axwwallyworld: I'll do azure, cloudsigma, and vsphere now05:00
wallyworldok, i promise i won't :-)05:00
wallyworldaxw: with the manta url - that's going away as soon as storage is gone05:01
axwwallyworld: yep, as per comment05:01
wallyworldfair enough, i'll leave in comment till then, i was hoping storage would be gone this week or next05:02
axwwallyworld: replied to comment about Apply05:43
wallyworldaxw: fair point, i think, seems like the tests need updating which i'll look at. are you happy with the modified todo for the private key stuff?05:45
axwwallyworld: didn't read yet, one sec05:45
axwwallyworld: yep. you probably wouldn't want to enter it interactively, but we can read the file during interactive entry of the filename, and add the value05:47
axwwallyworld: then your credentials.yaml is protected from changes on disk.05:47
axwmaybe not obvious though05:47
axwnot sure, we can leave it for now05:47
axwcovered the 99% case I think05:47
wallyworldi think the key on disk will be pretty static05:47
wallyworldyeah, all uses use file path afaik05:47
wallyworldaxw: that should be good to go now05:57
axwwallyworld: thanks, shipit05:58
axwwallyworld: just testing azure, should be ready to propose the rest very shortly05:59
axwwallyworld: there's still an issue with azure, another case where we need to be able to specify multiple endpoints05:59
axwwallyworld: in azure there's separate endpoints for storage and everything else, and they're not necessarily derivable06:00
axwwallyworld: pretty sure we're going to have to extend the clouds.yaml format06:00
wallyworldseems so06:00
wallyworlddo we *need* azure storage long term?06:01
axwwallyworld: yes. for volume support, and also some more basic operations like specifying where the VM image should live06:01
wallyworldstorage-endpoint then i guess06:02
wallyworldaxw: are the storage endpoints well know like the auth ones?06:02
wallyworldcan we add them to publoc cloud.yaml06:02
axwwallyworld: yes, for azure public cloud. for azure stack you'd specify your own06:03
wallyworldaxw: well seems like we should just update our public cloud yaml and cloud metadata struct them06:03
axwwallyworld: you mean with a new storage-endpoint field?06:04
wallyworldif it's not derivable06:05
axwwallyworld: I'm on the fence as to whether it should be specific to storage, rather than having a flexible map of <service>:<service-endpoint>. storage-endpoint is probably fine though06:05
wallyworldgiven it's optional, it keeps the default yaml nice and simple06:06
wallyworldfor other clouds that don't need it06:06
wallyworldwe can always get feedback and tweak06:06
axwwallyworld: ok, sounds fine06:06
cheryljcan someone help me figure out why go test isn't actually testing anything in a particular directory?  http://paste.ubuntu.com/15000293/06:07
axwwallyworld: added a card to Next06:07
wallyworldno suite registered?06:07
cheryljthe test ran in the merge bot, and failed, obviously06:08
cheryljbut not when I do it locally06:08
wallyworldsigh, hate that06:08
axwcherylj: have you changed anything? changed a test file from package to package_test perhaps?06:08
cheryljaxw: no, I didn't change any test files06:08
anastasiamac_wallyworld: axw: updated move \o/ PTAL?06:09
cheryljlooks like even on master I see the same issue06:09
cheryljI change a test to fail, and it happily thinks there's nothing to test06:09
axwanastasiamac_: one sec06:11
* anastasiamac_ waiting :D06:11
cheryljugh, the suite_test.go was for peergrouper_test, and nothing else was06:12
anastasiamac_cherylj: m blind but where is package_test.go?06:12
cheryljwonder how long it's been like that06:12
anastasiamac_in worker/peergrouper...06:12
cheryljanastasiamac_: guess they're using suite_test.go, rather than package_test06:12
anastasiamac_could be the problem?..06:13
axwanastasiamac_: LGTM, thanks06:13
anastasiamac_axw: \o/06:13
anastasiamac_wow - 2 in one day!!06:13
axwwallyworld: http://reviews.vapour.ws/r/3790/ -- here's the rest06:15
wallyworldta, looking06:15
wallyworldaxw: reviewed, you may want to rebase first as my branch is almost landed and it will probably conflict in fallback public clouds yaml06:27
axwwallyworld: thanks, yep, will do06:27
wallyworldbbiab, school pickup06:27
cheryljwallyworld, axw can one of you review the test changes I had to make?  http://reviews.vapour.ws/r/3782/06:44
cheryljI'm going to add in the unit tests for the change and am tracking that work via bug 154340806:45
mupBug #1543408: WatchControllerStatusChanges needs unit tests <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1543408>06:45
axwcherylj: just the last diff?06:45
axwor all of it?06:45
cheryljaxw: yes, the last diff06:45
cheryljI copied the mock watcher that was there and modified it to be a strings watcher06:45
cheryljhad to pull suite_test.go into the peergrouper package.  I'll see about making the tests all external when I write the tests for the functional change.06:46
cheryljit wasn't a no-op this time, so I skipped it06:46
axwcherylj: ignoring the error from ControllerInfo in state watcher seems a bit alarming. why not just error out there?06:49
axw(sorry, couldn't help myself and looked at the rest)06:49
cheryljaxw: if we're getting an error there, we'll get that error elsewhere and things will get restarted06:50
cheryljthat was my thinking anyway06:50
axwcherylj: so it's not harmful to error out there as well right?06:50
cheryljthere's not really a way to return an error there, from what I saw.06:50
axwoh. /me looks again06:50
blahdeblahcherylj: have you moved to Australia, or do you just hate sleep? :-)06:50
cheryljI miss sleep06:50
cheryljwe used to be buddies06:51
cheryljnow he's all emaciated because I don't feed him06:51
blahdeblahThat's no way to treat your buddies. :-)06:51
axwcherylj: right, there's not, sorry.06:51
axwcherylj: LGTM06:51
cheryljthanks, axw!06:52
cheryljand now, SLEEEEEP06:53
blahdeblahbe nice to your buddies!06:53
blahdeblahcherylj: BTW, thanks for some of those recent bug fixes.06:54
davechen1ycherylj: i have a strong suspcion that bootstrap is not making it to waitForInitalisatin06:55
davechen1yI think it's bombing out _WAY_ earlier06:55
anastasiamac_davechen1y: shh... cherylj is sleeping \o/06:56
axwthe new bootstrap syntax has already infected my brain. switching back and forth between 1.25 and 2.0 is going to be fun07:15
wallyworldanastasiamac_: looks like the controller.yaml file isn't quite right - it looks like it is storing model information as well as controllers07:19
wallyworldit also doesn't have the local. prefix07:19
wallyworldfor the controller name07:19
axwdavechen1y: it appears that piping stuff to bash causes bash to read a character at a time07:27
axwdavechen1y: as in, piping commands07:27
davechen1yoh wow07:44
davechen1ythat's rubbish07:44
axwwallyworld: would you like me to look at updating "switch", or are you doing that?07:53
davechen1yonly in juju would our rpc layer have a field called "Code", that was a string ...08:12
anastasiamac_axw: RB is not picking this up.. Could you PTAL on github? https://github.com/juju/juju/pull/434708:14
anastasiamac_wallyworld: so... according to the spec, "local" is a prefix that is added to model name at bootstrap.08:19
anastasiamac_wallyworld: to me, this reads as a separte PR unrelated to my controllers.yaml work08:19
anastasiamac_wallyworld: model name will be transformed before the file is written, hence, my work will just pick it automatically08:20
anastasiamac_wallyworld: ping me when u r back \o/08:20
anastasiamac_wallyworld: why there is model info in the file now, is probably an "oversight" :D i'll look08:20
axwanastasiamac_: done08:21
anastasiamac_axw: awesome \o/08:21
anastasiamac_axw: would we want to differntiate btw "controllers.lock", "models.lock", "accounts.lock" for different files?08:22
axwanastasiamac_: probably not. models and accounts are both related to controllers08:23
anastasiamac_axw: k. thnx08:24
axwanastasiamac_: meaning: removing a controller should remove related info08:24
axwanastasiamac_: so you can't not lock the controllers file if you're modifying models file08:24
axwand vice versa08:25
anastasiamac_axw: to what m observing, we lock a dir not a file...08:25
axwanastasiamac_: for now, yes08:25
dimiternfrobware, voidspace, dooferlad, I've updated the PR to make the diff a bit easier to follow and included a fix after live testing on maas:09:10
dimiternfrobware, voidspace, dooferlad, so please have a look when you can http://reviews.vapour.ws/r/3773/09:10
voidspacedimitern: ok09:21
axwdavechen1y: would appreciate if you could test https://github.com/juju/juju/pull/4349 tomorrow, and see if it resolves the issue for you. fairly sure it does, but need to get on with other things09:44
dimiternaxw, I think this should be applied to AddScripts as well - specifically in maas now we push a whole lot of python code to set up the bridge script09:48
dimitern..or perhaps that includes all runcmd / bootcmd scripts already09:49
axwdimitern: is that going to pipe commands to bash? I don't think so09:49
axwdimitern: yeah, this is the entire cloud-config script09:49
dimiternaxw, right, I've just noticed in maas we do a similar thing for the bridge script - put it in /tmp, trap exit and run it09:51
kjackalHey everyone, does any one know if/how is it possible to access relation/conversation data from within an action?10:15
voidspacejam: dooferlad: dimitern: sorry, screwed my network temporarily10:31
dimiternjam, sorry I was too quick - what were you about to say?10:45
jamdimitern: just wanted to check if anyone has heard from alexisb today. Usually I have a 1:1 with her last night, but she missed it, and didn't reply to my email, which is unlike her10:46
dimiternjam, nope - but according to the calendar she was off yesterday10:47
jamdimitern: sigh. I had turned of "Juju Team Calendar" when I was at the Cape Town sprint, so that's why I didn't see it.10:47
jamdimitern: thanks for noticing and checking. team calendar is back on.10:48
mupBug #1543517 opened: status command tests fail when you have dns spoofing <juju-core:New> <https://launchpad.net/bugs/1543517>10:53
dimitern:) np10:54
voidspacedimitern: when a user specifies multiple spaces for deployment constraints are we still only using the first?10:57
voidspacedimitern: apiserver/provisioner/provisioninginfo.go line 22410:57
voidspacedimitern: we should fix that soon I think10:57
mupBug #1543517 changed: status command tests fail when you have dns spoofing <juju-core:New> <https://launchpad.net/bugs/1543517>11:02
mupBug #1543517 opened: status command tests fail when you have dns spoofing <juju-core:New> <https://launchpad.net/bugs/1543517>11:05
voidspacedimitern: LGTM on your PR11:08
voidspacedimitern: the basic approach seems sound and I have no specific suggestions for it11:08
voidspacedimitern: there's a lot of context on the code this touches I'm missing, so you may want someone else to look at it too11:08
dimiternvoidspace, sure, the more sets of eyes the better11:13
dimiternvoidspace, cheers11:13
dimiternvoidspace, yeah, that's due to aws11:14
dimiternvoidspace, (ignoring all but the first space)11:14
dimiternvoidspace, but in maas we actually use the spaces constraints for a machine directly11:14
wallyworldanastasiamac_: hey. yes it will be a separate PR, just wanted to let you know that it needed looking at as part of the overall work11:40
wallyworldvoidspace: you found the problem, jeez what a strange issue11:56
anastasiamac_wallyworld: \o/ added card12:06
voidspacewallyworld: yeah, took most of the day12:41
wallyworldvoidspace: you may be interested in https://bugs.launchpad.net/bugs/153942812:42
mupBug #1539428: cmd/juju/status: status filtering performs IP resolution of patterns <status> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1539428>12:42
wallyworldrelated to your issue12:42
voidspacewallyworld: indeed12:42
voidspacewallyworld: so getting rid of the resolution would be a double win.12:45
wallyworldyes :-)12:45
wallyworldit will have to be fixed for 2.0 i think12:46
dimiterndooferlad, replied to your mail12:51
dimiterndooferlad, let me know if that helps12:53
voidspacefrobware:  dimitern: trivial one for you http://reviews.vapour.ws/r/3796/13:13
dimiternvoidspace, looking13:16
dimiternvoidspace, reviewed13:27
dimiterndooferlad, ping13:31
dooferladdimitern, voidspace: is the MAAS meeting happening? I am the only person in there...14:02
dimiterndooferlad, sorry got distracted - is it still going?14:05
dooferladdimitern: just starting14:05
dimiterndooferlad, omw14:05
voidspaceomw too14:10
dimiterndooferlad, did you had a chance to look at my PR #4331 ?14:48
dooferladdimitern: I thought voidspace had14:49
dooferladdimitern: happy to if it needs more eyes14:49
dimiterndooferlad, I'd appreciate it, thanks14:49
dimiterndooferlad, as I'd like to get it in today, if possible and have another one almost ready to propose14:49
voidspacedooferlad: I did review it and it looks sound to me, but I have a lot of missing context so I think another pair of eyes would be useful14:52
dooferladdimitern: LGTM. I would appreciate some card/bug links, but I didn't require them because it is in old code.16:06
dimiterndooferlad, thank you16:08
mupBug #1497301 opened: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>16:12
cheryljhey perrito666, looks like a couple bugs on the mongo3 branch need some love:  bug 1534620 and bug 149730116:14
mupBug #1534620: TestMongo26UpgradeStep fails on windows because of dos paths <ci> <regression> <test-failure> <windows> <juju-core:Incomplete> <juju-core update-mongodb3:Triaged> <https://launchpad.net/bugs/1534620>16:14
mupBug #1497301: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>16:14
mupBug #1497301 changed: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>16:15
cheryljhey dimitern, how are things going with maas-spaces?16:24
mupBug #1497301 opened: mongodb3  SASL authentication failure <ci> <mongodb> <regression> <unit-tests> <juju-core:Triaged> <juju-core upgrade-mongodb3:Triaged> <https://launchpad.net/bugs/1497301>16:24
dimiterncherylj, hey16:25
dimiterncherylj, we're tracking master daily and wait for some CI feedback from the changes we pushed in maas-spaces so far (4 days we had no CI run of maas-spaces)16:26
dimiterncherylj, we're also ~2d way from having a hopefully blessed maas-spaces16:26
voidspacedimitern: ping16:26
dimiternvoidspace, pong16:26
voidspacedimitern: the card I'm working on is "provider/maas: Addresses() to set SpaceProviderId not SpaceName"16:27
voidspacedimitern: there is no provider/maas Addresses method16:27
voidspacedimitern: and in fact "SpaceName" doesn't appear at all in provider/maas/environ.go16:27
dimiternvoidspace, it's in instance.go or interfaces.go IIRC16:28
voidspacedimitern: ah yes it is16:28
voidspacedimitern: better use of grep just found it in instance.go16:29
voidspacedimitern: sorry for the noise :-)16:29
alexisbdimitern, are there still changes the team needs to push to the maas spaces branch for a CI bless?16:29
voidspaceregex for the win16:29
voidspacealexisb: yes16:29
voidspacealexisb: proper handling of controller space16:29
alexisbvoidspace, and we are looking at 2 days for those fixes?16:29
voidspacealexisb: the failures related to "Default Space" actually highlighted an important problem which we've nearly fixed16:29
voidspacealexisb: yes16:30
voidspacewe shouldn't have been using the default space at all and we *must* know the space to deploy controllers too16:30
alexisbvoidspace, dimitern would a CI run on maas-spaces now be useful to validate landed fixes?16:30
voidspacealexisb: yes16:30
alexisbvoidspace, thank you16:30
dimiternalexisb, yes, absolutely - we didn't have one for 4 days16:30
voidspacealexisb: space discovery problems should be completely fixed and we haven't had a full run since those fixes landed16:30
alexisbwe need to stay in daily contact regarding progress on this branch16:30
alexisbvoidspace, ack16:31
dimiternalexisb, we're hopefully down to 2-3 failures now compared to master, but still need to get a CI run to be sure16:31
alexisbsinzui, mgz, can we please get a run on the maas spaces branch16:31
sinzuialexisb: it is running16:31
voidspacealexisb: FYI  the three yellow cards in our "In Progress" kanban lane are tracking progress16:31
voidspacealexisb: https://canonical.leankit.com/Boards/View/101652562#workflow-view16:31
voidspacesinzui: thanks16:32
alexisbvoidspace, awesome thank you16:32
mupBug #1543660 opened: juju needs to fix the yaml parser <customer> <juju-core:New> <https://launchpad.net/bugs/1543660>16:51
dooferladdimitern, voidspace: EOD for me. Still watching email.17:08
cheryljhey tych0, looks like CI on your branch had more build failures:  http://reports.vapour.ws/releases/3586  (non-trusty build failures)17:10
dimiterndooferlad, cheers, but I need to go soon as well17:11
natefinchkatco: can you double check with rick_h__ that we're dropping comment for now?17:12
rick_h__natefinch: yes17:12
katconatefinch: doh, sorry i did. we are.17:12
natefinchoh good :)17:12
katconatefinch: sorry, forgot to include that in the email. ericsnow fyi ^^^17:13
natefinchericsnow: did you have something you needed reviewed?17:28
ericsnownatefinch: http://reviews.vapour.ws/r/3778/17:29
natefinchericsnow: how much of cmd/juju/charmcmd/store.go is copied from elsewhere?19:09
ericsnownatefinch: nearly all of it19:10
natefinchericsnow: I thought so.  Can you put in comments as to that effect?  Then I don't have to try so hard to figure out why it's doing all this stuff.19:10
natefinchericsnow: like directly in the function bodies that have been copied19:11
ericsnownatefinch: sure19:12
ericsnownatefinch: oh, and internal tests are making my life miserable19:12
natefinchericsnow: how are internal tests making your life miserable?19:13
ericsnownatefinch: import cycles, and fixing them is a ton of work :(19:13
natefinchericsnow: ahh boo.... well, import cycles are a valid reason to use external tests19:13
ericsnownatefinch: that doen't help me now though19:14
natefinchericsnow: sorry :/19:14
natefinchericsnow: is this because of existing internal tests?  If so, I'd definitely like to hear details at some point, so we can try to avoid this in the future.19:15
natefinchericsnow, katco: trivial rename in charm metadata: https://github.com/juju/charm/pull/19019:30
katconatefinch: +1 from me!19:31
ericsnownatefinch: same here19:32
natefinchericsnow, katco: I have the s/comment/description patch for juju ready, but it's dependent on my other patch that's up for review, and I can't make rbt show the right diff.... so I'll just leave it until the other one lands.  It's another trivial diff anyway.20:00
katconatefinch: ok that sounds fine20:01
natefinchkatco: so, should I grab the --debug card?  or should we point the upgrade-charm card?20:01
katconatefinch: grab the --debug card since that's the last user-facing bit20:02
natefinchkatco: ok20:06
katconatefinch: is this something specific to resources? https://github.com/juju/juju/commit/2910afceacca26dc1d880cedaa5d7560b249ebbe#diff-5c78256455800f538886c2beef98045aR6620:21
katconatefinch: doesn't seem to be in master at any point20:21
mupBug #1543770 opened: Juju 2.0alpha1.1 does not assign a proper IP address for LXC containers <dhcp> <lxc> <maas> <network> <juju-core:New> <https://launchpad.net/bugs/1543770>20:22
natefinchkatco: you mean the arguments struct?20:22
katconatefinch: specifically "toMachineSpec"20:22
natefinchkatco: it is there on the left side of the diff... that's what juju deploy foo --to 3  gets turned into20:23
natefinchkatco: possible that it's been changed in master?20:23
katconatefinch: it has; trying to resolve the diff20:23
katconatefinch: i was mistaken, just found it's lineage in master20:24
natefinchkatco: certainly, it doesn't have anything to do with resources.. I was just refactoring the args into a struct.20:24
katconatefinch: thx that makes the merge much easier :)20:24
natefinchkatco: cool20:24
natefinchis there a function to get the unit number from the unitID?20:40
natefinchlike, I can certainly write one, but I'd rather reusing someone else's20:41
menn0_thumper: when you have a chance, could you look at my replies to your comments on http://reviews.vapour.ws/r/3745/ ?20:42
=== menn0_ is now known as menn0
aznashwanany charm-tools people I could bother with a couple of reviews?21:02
thumpermenn0: ok21:02
natefinchrick_h__, ericsnow, katco:  should we support juju resources foo/0 --debug, and only show detailed information about that particular unit's resources?21:03
natefinch(currently the spec only talks about juju resources foo --debug)21:03
rick_h__natefinch: yes, I think it should be able to go into per unit as well21:04
natefinchrick_h__: ok. luckily, that's not much more work21:04
thumpermenn0: I'm tempted to call yagni and just use user tags21:04
natefinchrick_h__: what should we show if the unit hasn't called resource-get for a resource?21:05
natefinchrevision "-" or something?21:05
rick_h__natefinch: otp atm21:05
natefinchthumper, menn0: If yagni is an option, it's almost always the right choice.  "Just in case" just makes your code confusing when it can really only be one thing right now. (IMO)21:09
menn0thumper: i'm fine with that. The facade can always be bumped later.21:09
menn0natefinch: I generally agree with that too. In this case the cost difference between either approach is nominal so it wasn't clear which way to go.21:11
natefinchmenn0: yeah. I generally prefer more specific types, so they convey the right information.  Seeing an API that takes a names.Tag, but really only one type of tag ever gets passed in, can easily make someone write the wrong code, trying to support a bunch of arguments that they don't really even need to worry about.21:12
menn0natefinch: yep, agreed.21:13
menn0natefinch: I'm going with the more specific type.21:13
=== natefinch is now known as natefinch-afk
tych0cherylj: ok, i'll take a look now. was distracted by some other stuff this morning21:42
tych0cherylj: https://github.com/juju/juju/pull/435521:56
tych0looks like somehow that commit got dropped in all the shuffling. anyway, I think that should fix at least the problems you're seeing now.21:57
tych0(assuming the i18n stuff was the only issue, that's what i saw in my monte carlo sampling :)21:57
menn0thumper: can you pls take another look at https://github.com/juju/juju/pull/4303 ?22:38
menn0thumper: also http://reviews.vapour.ws/r/3746/ pls (tiny)22:39
davecheneyhere is a simple review https://github.com/juju/juju/pull/435723:29
davecheneyanyone ?23:29
mupBug #1543839 opened: juju/service: test fixture panics if SetUpSuite fails <juju-core:New> <https://launchpad.net/bugs/1543839>23:31
mupBug #1543839 changed: juju/service: test fixture panics if SetUpSuite fails <juju-core:New> <https://launchpad.net/bugs/1543839>23:34
mupBug #1543839 opened: juju/service: test fixture panics if SetUpSuite fails <juju-core:New> <https://launchpad.net/bugs/1543839>23:40

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!