/srv/irclogs.ubuntu.com/2017/02/09/#juju-dev.txt

axwredir: ping00:30
rediraxw00:30
axwredir: I zoned out, what did you want to talk to me about?00:30
axwredir: was it about the ec2 region?00:30
rediryesa00:30
rediraxw: https://hangouts.google.com/hangouts/_/canonical.com/axw-redir00:31
perrito666menn0: priv00:39
rediranastasiamac: https://github.com/juju/juju/pull/694801:00
anastasiamacredir: \o/01:03
axwmenn0 thumper: when you migrate, you're expected to have a cloud definition in the target controller already right?01:30
thumperaxw: yes01:45
axwthumper: ok, thanks01:45
thumperaxw: the user initiating the migration has to have access to both controllers01:45
* axw nods01:45
axwthumper: there's a bug about not being able to change cloud endpoints. one current workaround (for 2.1) would be to bootstrap a new controller with the updated endpoints, and migrate to it01:46
thumperaxw: what do you mean?01:47
axwthumper: https://bugs.launchpad.net/juju/+bug/166259901:47
mupBug #1662599: Change maas api and password controller in model already created <juju:Incomplete> <https://launchpad.net/bugs/1662599>01:47
axwthumper: we currently lack an API/command to update a cloud definition within a controller01:47
thumperah... right01:48
axwnot sure if migration would succeed if the source controller's environs can't be opened though?01:48
thumperwe don't do model renaming on moving yet01:48
thumperthe model config for the maas endpoint is set when the model is created right?01:49
axwthumper: it's not in model config any more01:49
thumperhmm..01:50
thumperI recall now01:50
thumperaxw: side issue, any idea how to ask the go runtime who is holding references to an object?01:50
axwthumper: anyway doesn't matter, that person is on 2.0 and probably can't use the feature flag01:50
axwthumper: heap profile for in-use objects will show you that01:51
thumperhmm...01:52
axwthumper: go tool pprof -inuse_objects01:52
thumperI thought the heap profile was sampling01:52
axwthumper: yeah, no good if you want every single ref01:53
thumperyeah...01:53
thumperveebers: ping01:54
* redir eods01:54
veebersthumper: hey, I'm just heading out the door for a little bit01:54
axwthumper: what in particular do you want to track?01:54
thumperveebers: ack01:54
thumperaxw: I have some *state.State objects that have been closed but not GCed01:54
thumperwondering why...01:55
thumperperhaps just not busy enough01:55
thumperalthough01:55
thumperI cheekily added a runtime.GC to see01:55
thumperbut they aren't being cleaned up01:55
thumpercould be a real problem01:55
babbageclunkthumper: I guess as long as all the goroutines for them are stopped they probably aren't taking up much memory01:56
thumperwell... that is a question isn't it01:57
thumperI'm not convinced they are stopped01:57
babbageclunkuhoh01:57
thumperhence the continued increase memory and db load01:57
axwthumper: if we use https://golang.org/pkg/runtime/pprof/#NewProfile and https://golang.org/pkg/runtime/pprof/#Profile.Add, should be able to track what's creating the State objects at least. or do you already know that from the state pool modifications you made?02:00
thumperaxw: I've just added a state tracker02:00
thumper:)02:00
thumperwhich captures the stack for all new state instances02:01
axwthumper: ok. I think if you use the runtime/pprof things you can use "go tool pprof" to generate pretty pictures. but whatever works02:02
thumperhmm...02:02
thumperI was just about to propose for review02:02
thumperI'll take a quick look first02:02
thumpernot sure this will do what I want unfortunately02:03
* thumper goes to make coffee02:15
axwwallyworld: I just pushed a change to https://github.com/juju/juju/pull/6941 so that DetectCredentials returns a fully formed credential for localhost02:52
axwwallyworld: can you PTAL?02:52
wallyworldsure02:52
axwwallyworld: the reason for the change is so to make cred loaded by autoload-credentials immediately usable02:52
wallyworldi was wondering about that02:52
wallyworldbut thought i was missing something02:53
axwwallyworld: it was intentional but misguided. the idea was that the partial cred would be finalized against a specific cloud later, which would give you a chance to be prompted for the fingerprint/password02:54
axwwallyworld: but there's an interactive auth type for that now, and the partial cert breaks autoload02:54
wallyworldright, makes sense02:54
wallyworldaxw: i think there's a slight ambiguity. "If you're not on the LXD host, you can use juju add-credential localhost..."  <--- maybe "If you intend to interact with the LXd cloud from a host other than the one running LXD, you need to run from the LXD host 'juju add-credential ....'02:58
wallyworldor something like that02:58
wallyworldjust to be really clear what needs to be run on what machine02:58
axwwallyworld: you talking about the docs link?02:59
wallyworldyeah02:59
wallyworldaxw: i'd also leave out the lxd trust reference in the juju message.... i think we want to guide people to using juju tooling?03:00
wallyworldso maybe just mention the need to export the credential from the LXD host to the machine from where the user wants to gain access03:01
wallyworldand provide the link to the docs to show them how03:01
axwwallyworld: ok. I've updated the docs issue03:03
axwwallyworld: is this better for the command output? http://paste.ubuntu.com/23958386/03:05
wallyworldaxw: thanks, i added a small change "..... you must add credentials to that other machine manually."03:05
axwwallyworld: sounds good03:05
wallyworldaxw: yeah, reads much nicer. but i think "add-credential lxd" might work instead of "localhost"? not sure. but "localhost" could be confusing as they are adding a credential for a cloud "other there" not on localhost03:06
wallyworldif "lxd"works, they can name the credential after the host03:07
* axw tests03:07
wallyworldthe host running lxd03:07
axwwallyworld: I think it depends on the cloud name, which will be localhost03:07
wallyworldi think our code supports both lxd or localhost for the lxd provider?03:07
wallyworldie we use add-credential google or add-credential aws03:08
axwwallyworld: for bootstrapping yes, but I don't think so for add-model.. will see03:08
axwwallyworld: those are the names of the clouds, not the providers03:08
wallyworldright, but i *think* we special case lxd, but as you say, maybe only for bootstrap03:09
wallyworldi think we need to tweak add-model if it isn't suported there03:09
axwwallyworld: the UX for creds is still a bit rough I think03:09
wallyworldyeah03:09
wallyworldbut so is the lxd/localhost handling03:10
wallyworldaxw: we can land as is but i do think we need a followup to avoid falsely add creds for a cloud on another machine and using the cloud name "localhost"03:12
axwwallyworld: it's pretty low priority I think. this is an edge case03:13
wallyworldthat's IMHO. i would prefer "lxd" as we say things like we are running a "maas" cloud or an "lxd" cloud. or at least i do03:13
axwso yes, but probably not today03:13
axwwallyworld: it would probably be reasonable to fall back to the provider name when looking for creds03:14
wallyworldok. if we ship with it as "localhost", so long as both then still work if a changeis made. maybe we can fix before 2.1 final03:14
wallyworldyeah, i think so03:14
wallyworldaxw: so are you intending to followup with the docs guys to get the new doc link up before the rc goes out tomorrow? might be worth an email to nick or peter?03:17
axwwallyworld: will do03:17
wallyworldta03:17
axwwallyworld: not much point actually, it's not going to go into the stable docs until the final release is it?03:19
wallyworldfinal release is within a week - can we add it but not link to it from anywhere. then the only way you see it is via the juju cli output03:20
axwwallyworld: it's not going to go on its own page, it'll be on the LXD cloud page03:22
axwwallyworld: which is linked from the main page, and has stable vs. devel in the URL03:22
axwwallyworld: it's an edge case, is it really worth all this energy? we can mention it in the release notes and link to the issue or devel docs if it makes it in there03:23
wallyworldhmmmm. can we land in the devel docs03:23
wallyworldok03:23
thumperWTF? why is 2.1 version updated?03:29
thumperwe haven't done the rc1 release yet03:29
anastasiamacthumper: according to sinzui's mention in release call: "versions must be numbers and dots to get into proposed and released"03:30
sinzuithumper: that is true.03:30
* thumper isn't happy03:31
thumperit isn't a .0 release03:31
thumperit is an rc release03:31
sinzuithumper: some juju that read streams with letters in the version fall over03:31
sinzuithumper: I think you can recall that03:31
thumpersinzui: pre 1.2003:31
sinzuithumper: so to prevent breakage, there are controls that will abort if you put an "rc" in them03:32
sinzuithumper: I also brought this up with the release of 2.0 and no onw made a fuse (they should have)03:32
sinzuithumper: by putting the rc in *proposed* stream, we are commited to burn numbers. We can put rc1 in the devel stream...but we cannot promote it to proposed03:33
* sinzui wished we could decouple the client's build from the inteded version03:35
sinzuithumper: consider this if we rename juju to 2.1-rc1 again, and publish the agents to devel. The client will automatically look for agent in devel. The client package could be manually copied to the proposed ppa. The snapped client can also go to the candidate. The only problem is testers that need to adjust streams in scripts will need to set agent-stream=devel03:44
thumperick03:46
thumperwe need to fix this03:46
* thumper out03:46
wallyworldaxw: this PR is a lot of noise (fake auth field name as driveby), but fixes keymanager facade permissions https://github.com/juju/juju/pull/695104:41
axwwallyworld: ok, looking04:41
wallyworldta04:41
axwwallyworld: next time can you please separate the drive by into a separate commit, so it can be reviewed with unfocused eyes :)04:43
wallyworldyeah, fiar point04:44
wallyworldsorry04:44
wallyworldthe changes if noe are in apiserver/keymanager04:44
wallyworld*of note04:44
blahdeblahHi folks - is there going to be a similar fix for 1.25.x as will be in 2.0.3 for the memory leaking?  Just discovered that on of our main production environments on 1.25.6 has been getting OOM killed as well.04:58
axwwallyworld: left a bunch of comments, will look at tests after you respond05:02
axwblahdeblah: I don't think the one that affected 2.0.3 applies to 1.25.x05:03
wallyworldsure ty05:03
axwblahdeblah: we're up to 1.25.9 now, which does include some leak fixes05:03
axw(I hear there are still issues in 1.25.9, but it'd be good to rule out hte bugs that have already been fixed)05:05
blahdeblahaxw: In our production deployments, we've found post-1.25.6 releases less stable, and so froze them there.05:05
axwblahdeblah: I see05:05
blahdeblahI upgraded two small Canonistack envs to 1.25.10 (in proposed) late last week/early this week, in response to a request in lp:1587644, and they're still experiencing similar symptoms, although without the obvious memory leak.05:07
blahdeblahBut still need jujud-machine-0 restarts every day or two05:08
axwblahdeblah: ok, well I don't know when a new 1.25 will come. we're all focused on getting 2.1 out the door right now05:09
blahdeblahyeah - fair enough05:09
axwblahdeblah: apart from dealing with fallout from the odd prodstack explosion :o05:09
blahdeblah:-)05:09
blahdeblahI might poke around for any 1.25 bugs which match the symptoms05:09
babbageclunkmenn0: around? min reviewing this for me? https://github.com/juju/juju/pull/695205:15
babbageclunkmenn0: It's short!05:15
axwwallyworld: still reading responses, check my comment about controller tag again please05:20
wallyworldaxw: i'm missing something i think - it looks ok to me. the controller tag is only used in the mothod to check for superuser05:22
wallyworldthe model is passed in separately05:22
axwwallyworld: the comment was next to the call to common.HasPermission05:22
axwwallyworld: which does not take a model05:22
wallyworldoh, i am dyslexic05:23
wallyworldshould be ok now05:24
babbageclunkor axw? https://github.com/juju/juju/pull/695205:35
axwbabbageclunk: looking05:35
axwbabbageclunk: you have run the QA for all those clouds right?05:36
axwor do we have CI tests set up for this?05:36
axwbabbageclunk: LGTM05:40
wallyworldaxw: yeah, i don't think it's possible to have a model owner that's not an admin, hence no test was added. mayber it is with some convoluted test setup, i'll have a look05:42
wallyworldmight be doable with fake authenticator05:42
axwwallyworld: I can imagine that someone might want to create a model and then hand it off, but IMO that would be transfer of ownership. or we should call it "creator" rather than owner05:43
wallyworldyeah05:44
wallyworldpeople have asked for that05:44
axwwallyworld: I just replied again. while writing I realised that checkCanRead also will need to be updated if we're going to stick with the existing behaviour around system identity key05:45
wallyworldyeah, i'm going to review the logic again05:46
axwwallyworld: I added an RC1 section to the release notes, with a blurb about LXD creds. would you please have a read when you have a chance?05:49
wallyworldsure ty05:49
babbageclunkaxw: Sorry, bathtime06:19
axwbabbageclunk: oh, all clean now? ;)06:20
babbageclunkaxw: I've done a couple (lxd and ec2)06:20
babbageclunkaxw: yes thanks!06:20
axwbabbageclunk: do you know if CI is testing this already? we should make sure it's tested on all the clouds06:20
babbageclunkaxw: I think veebers has CI for all clouds06:20
axwbabbageclunk: ok cool06:20
babbageclunkaxw: I'll confirm with him tomorrow06:21
wallyworldaxw: release notes look ok, they can tidy formatting. i've pushed changes also. got to rush to school pickuo, bbiab06:22
axwwallyworld: thanks. looking06:22
=== mpontillo_ is now known as mpontillo
=== seyeongkim_ is now known as seyeongkim
wallyworldaxw: thanks for review. i didn't see the harm in users reading the public bit of the system key?06:45
axwwallyworld: that's the existing behaviour. I don't have a sproblem with us changing it in this case, but I want to know that it's intentional and tested06:46
jamwallyworld: thumper: I'm trying to reconcile 2.1 vs develop06:46
jamit looks like thumper added a signature to stateForRequestAuthenticade to return a "releaser" funct06:46
jambut wallyworld added a modelRestHandler.stateAuthFunc06:46
wallyworldaxw: ok, best to stick with existing behaviour. i was being strict about that for write06:47
jamwallyworld: but modelRestHandler.stateAuthFunc is never called06:47
jamit has a hard-conded stateForRequestAuthenticated in ServeGet06:47
axwwallyworld: SGTM06:47
wallyworldjam: i'll have a look, give me a few minutes06:48
jamwallyworld: k06:48
jamI'm trying to get my fixes in 2.1 into 2.2, and bringing some of Tim's along for the ride.06:48
jamwallyworld: ping me when you're available so I can go through it with you06:50
wallyworldok06:50
axwwallyworld jam: I just noticed that the version was already bumped to 2.1.0. did we already cut a release? stuff landed today won't make it?06:55
wallyworldaxw: they did that in preparation06:55
jamaxw: i don't know. I just saw the version bump myself.06:56
jamwallyworld: but wouldn't it be 2.1rc1 ?06:56
wallyworldthe rc will have 2.1.0 as thge version06:56
wallyworldi would have thought so06:56
wallyworldbu tthey want to take the binary and release unchanged as final06:56
wallyworldi don't agree with that myself06:56
wallyworldwhat if we need a rc206:56
axwthat sounds crack06:56
wallyworldyep06:57
wallyworldjam: free to talk now07:01
jamwallyworld: HO or IRC?07:01
wallyworldHO easier, standup one?]07:02
=== Dmitrii-Sh_ is now known as Dmitrii-Sh
anastasiamacwallyworld: axw: jam: earlier conv with sinzui re:version https://pastebin.canonical.com/178735/07:22
anastasiamacwallyworld: u were in the release call, if u disagreed, it'd b more constructive to have do so there07:22
wallyworldanastasiamac: i tried to but the call ended07:23
anastasiamacwallyworld: call ppl back :D07:23
wallyworldhonestly, the proposal is crack, john will send an email to get it fixed07:23
anastasiamacjam: axw: for reference not a new issue. we hit it at every release07:23
wallyworldanastasiamac: not any more07:24
wallyworldthe only juju versions that had issues were < 1.2007:24
jamneeding to set agent-streams=devel seems better than not being able to make an rc2, or have a 2.1.0 final that can identify itself as different.07:25
wallyworldwe don't support those any more07:25
jamwallyworld: well, we'd have to actually test that 1.25 doesn't crap itself with 'rc' in the proposed stream07:25
wallyworldit doesn't07:25
wallyworldthat was fixed for 1.2007:25
wallyworldor 1.19 even07:26
anastasiamacwallyworld: jam: great discussion to have with release team \o/07:26
wallyworldwe shouldn't need to - the policy is clear07:26
anastasiamacwho r coming online *shortly*07:26
wallyworldthis change was done on a whime07:26
anastasiamacwallyworld: obviously not if they made a request and approved it07:26
wallyworldas can be seen from all the wtf07:26
wallyworldit wasn't approved07:26
wallyworldnot by us07:27
wallyworldand there was nothing in writing etc07:27
anastasiamacit was done on request from sinzui directly07:27
wallyworldif it were approved tim, andrew, john and i would not be going etf etf07:27
wallyworld*wtf07:27
anastasiamacnothing inwritng, except in releas call minutes... did u need it in blood?07:27
wallyworldfor such a major change to policy, it needs proper discussion, not a btw at the end of the release call07:28
anastasiamacit was not at the end07:28
anastasiamacanyway, nothing was released yet and now is ur chance to fix07:29
wallyworldyep, that is good07:29
wallyworldjam will follow up07:29
anastasiamacobviously, communication failure since 1.2007:29
wallyworlddon't buy that07:29
anastasiamacwhat do u ever?07:29
wallyworldaxw: i'm having trouble booting lxd. i've cleared ~/.config/lxc and there's nothing in credentials.yaml..... but at end of bootstrap07:32
wallyworld2017-02-09 07:27:51 INFO juju.cmd supercommand.go:63 running jujud [2.1.0.1 gc go1.6.3]07:32
wallyworld2017-02-09 07:27:51 ERROR cmd supercommand.go:458 new environ: Get https://10.132.22.1:8443/1.0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "root@wallyworld")07:32
wallyworldthis is on 2.1 branch07:33
wallyworldi'll dig in a bit after soccer, not sure if you've seen that issue07:33
axwwallyworld: ehhh? I have not seen that, no07:34
wallyworldyeah weird07:35
wallyworldi did apt remove and then apt install lxd yesterday07:35
wallyworldto test a snap issue07:35
wallyworldmaybe time for lxd init again07:36
axwwallyworld: oh, you probably need to remove any credentials you have stored for lxd in credentials.yaml then07:38
wallyworldaxw: yeah i checked, none there07:38
axwhm, dunno then07:38
wallyworldyeah, me either07:38
=== frankban|afk is now known as frankban
axwwallyworld: did you see that I reviewed your PR?08:36
axwah, soccer08:37
=== frankban is now known as frankban|afk
=== frankban|afk is now known as frankban
=== admcleod_ is now known as admcleod
balloonsredir, not sure what the thought is on pushing PR's against 2.1 and develop. But here's one if you don't mind having a look: https://github.com/juju/juju/pull/6956 and https://github.com/juju/juju/pull/695719:39
redirballoons: looking19:39
redirballoons: I don't know about 2.119:42
redirdevelop LGTM though.19:42
balloonsredir, as in, you don't know if they want things to land or ?19:42
redirballoons: correct19:42
balloonsahh. I want it to land in 2.1 to help us toward that bless :-)19:43
redirOK.19:43
redirshipit19:43
redirjust saying I don't know, but some blesses would be greatly celebrated I'm sure:)19:44
balloonsredir, lol.. :-) Ok,19:46
redirLooks great to me balloons19:47
=== frankban is now known as frankban|afk
=== frankban|afk is now known as frankban
rediranother easy review; https://github.com/juju/juju/pull/695821:40
rediranyone ^^21:41
redirjuju-dev ^^21:41
babbageclunkredir: yay, easy review! Looking!21:55
babbageclunkredir: this doesn't sound that easy21:56
redirhehe it's small21:58
babbageclunkredir: LGTM'd22:01
redirbabbageclunk: tx22:01
babbageclunkredir: also I'm OCR for me, but not for your timezone22:18
* redir blinks22:19
redirthanks for the review in my timezone:)22:19
babbageclunkredir: :)22:21
alexisbanastasiamac, ping22:30
anastasiamacalexisb: pong22:31
thumperbabbageclunk: https://github.com/juju/juju/pull/695923:10
babbageclunkthumper: reviewed - lgtm23:20
babbageclunkspecial OCR deal today - any PR that's more deletions than additions is auto-approved!23:21
babbageclunk(offer not valid where void)23:21
babbageclunkI mean, offer void where prohibited23:21
thumper:)23:26
axwthumper: there is quite a lot of awesome for such a small amount of code with custom pprof profiles :)23:28
thumperaxw: yeah23:29
thumperwhen I realised that we needed to hook into close and not use a finalizer23:29
thumperit made sense to go with pprof23:29
thumperso nice to look at too23:29
thumperand less things to hook up23:29
thumperthanks for pointing that out to me23:29
thumperit's good23:29
axwthumper: does the tool just work? i.e. can you use go tool pprof --svg ?23:29
axwthumper: no worries23:29
thumperhaven't tried that actually23:29
redirbabbageclunk: this should look familiar: https://github.com/juju/juju/pull/696023:31
=== frankban is now known as frankban|afk
babbageclunkredir: looking23:37
babbageclunkredir: net negative lines! Approved!23:38
babbageclunkredir: Also a legitimate change.23:38
axwanastasiamac thumper: standup?23:46
babbageclunkthumper/axw: review this? https://github.com/juju/juju/pull/696123:57
babbageclunkunfortunately adds lines.23:57
axwbabbageclunk: will do after standup, maybe after taking kids to school23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!