[00:30] <axw> redir: ping
[00:30] <redir> axw
[00:30] <axw> redir: I zoned out, what did you want to talk to me about?
[00:30] <axw> redir: was it about the ec2 region?
[00:30] <redir> yesa
[00:31] <redir> axw: https://hangouts.google.com/hangouts/_/canonical.com/axw-redir
[00:39] <perrito666> menn0: priv
[01:00] <redir> anastasiamac: https://github.com/juju/juju/pull/6948
[01:03] <anastasiamac> redir: \o/
[01:30] <axw> menn0 thumper: when you migrate, you're expected to have a cloud definition in the target controller already right?
[01:45] <thumper> axw: yes
[01:45] <axw> thumper: ok, thanks
[01:45] <thumper> axw: the user initiating the migration has to have access to both controllers
[01:45]  * axw nods
[01:46] <axw> thumper: there's a bug about not being able to change cloud endpoints. one current workaround (for 2.1) would be to bootstrap a new controller with the updated endpoints, and migrate to it
[01:47] <thumper> axw: what do you mean?
[01:47] <axw> thumper: https://bugs.launchpad.net/juju/+bug/1662599
[01:47] <mup> Bug #1662599: Change maas api and password controller in model already created <juju:Incomplete> <https://launchpad.net/bugs/1662599>
[01:47] <axw> thumper: we currently lack an API/command to update a cloud definition within a controller
[01:48] <thumper> ah... right
[01:48] <axw> not sure if migration would succeed if the source controller's environs can't be opened though?
[01:48] <thumper> we don't do model renaming on moving yet
[01:49] <thumper> the model config for the maas endpoint is set when the model is created right?
[01:49] <axw> thumper: it's not in model config any more
[01:50] <thumper> hmm..
[01:50] <thumper> I recall now
[01:50] <thumper> axw: side issue, any idea how to ask the go runtime who is holding references to an object?
[01:50] <axw> thumper: anyway doesn't matter, that person is on 2.0 and probably can't use the feature flag
[01:51] <axw> thumper: heap profile for in-use objects will show you that
[01:52] <thumper> hmm...
[01:52] <axw> thumper: go tool pprof -inuse_objects
[01:52] <thumper> I thought the heap profile was sampling
[01:53] <axw> thumper: yeah, no good if you want every single ref
[01:53] <thumper> yeah...
[01:54] <thumper> veebers: ping
[01:54]  * redir eods
[01:54] <veebers> thumper: hey, I'm just heading out the door for a little bit
[01:54] <axw> thumper: what in particular do you want to track?
[01:54] <thumper> veebers: ack
[01:54] <thumper> axw: I have some *state.State objects that have been closed but not GCed
[01:55] <thumper> wondering why...
[01:55] <thumper> perhaps just not busy enough
[01:55] <thumper> although
[01:55] <thumper> I cheekily added a runtime.GC to see
[01:55] <thumper> but they aren't being cleaned up
[01:55] <thumper> could be a real problem
[01:56] <babbageclunk> thumper: I guess as long as all the goroutines for them are stopped they probably aren't taking up much memory
[01:57] <thumper> well... that is a question isn't it
[01:57] <thumper> I'm not convinced they are stopped
[01:57] <babbageclunk> uhoh
[01:57] <thumper> hence the continued increase memory and db load
[02:00] <axw> thumper: if we use https://golang.org/pkg/runtime/pprof/#NewProfile and https://golang.org/pkg/runtime/pprof/#Profile.Add, should be able to track what's creating the State objects at least. or do you already know that from the state pool modifications you made?
[02:00] <thumper> axw: I've just added a state tracker
[02:00] <thumper> :)
[02:01] <thumper> which captures the stack for all new state instances
[02:02] <axw> thumper: ok. I think if you use the runtime/pprof things you can use "go tool pprof" to generate pretty pictures. but whatever works
[02:02] <thumper> hmm...
[02:02] <thumper> I was just about to propose for review
[02:02] <thumper> I'll take a quick look first
[02:03] <thumper> not sure this will do what I want unfortunately
[02:15]  * thumper goes to make coffee
[02:52] <axw> wallyworld: I just pushed a change to https://github.com/juju/juju/pull/6941 so that DetectCredentials returns a fully formed credential for localhost
[02:52] <axw> wallyworld: can you PTAL?
[02:52] <wallyworld> sure
[02:52] <axw> wallyworld: the reason for the change is so to make cred loaded by autoload-credentials immediately usable
[02:52] <wallyworld> i was wondering about that
[02:53] <wallyworld> but thought i was missing something
[02:54] <axw> wallyworld: it was intentional but misguided. the idea was that the partial cred would be finalized against a specific cloud later, which would give you a chance to be prompted for the fingerprint/password
[02:54] <axw> wallyworld: but there's an interactive auth type for that now, and the partial cert breaks autoload
[02:54] <wallyworld> right, makes sense
[02:58] <wallyworld> axw: i think there's a slight ambiguity. "If you're not on the LXD host, you can use juju add-credential localhost..."  <--- maybe "If you intend to interact with the LXd cloud from a host other than the one running LXD, you need to run from the LXD host 'juju add-credential ....'
[02:58] <wallyworld> or something like that
[02:58] <wallyworld> just to be really clear what needs to be run on what machine
[02:59] <axw> wallyworld: you talking about the docs link?
[02:59] <wallyworld> yeah
[03:00] <wallyworld> axw: i'd also leave out the lxd trust reference in the juju message.... i think we want to guide people to using juju tooling?
[03:01] <wallyworld> so maybe just mention the need to export the credential from the LXD host to the machine from where the user wants to gain access
[03:01] <wallyworld> and provide the link to the docs to show them how
[03:03] <axw> wallyworld: ok. I've updated the docs issue
[03:05] <axw> wallyworld: is this better for the command output? http://paste.ubuntu.com/23958386/
[03:05] <wallyworld> axw: thanks, i added a small change "..... you must add credentials to that other machine manually."
[03:05] <axw> wallyworld: sounds good
[03:06] <wallyworld> axw: yeah, reads much nicer. but i think "add-credential lxd" might work instead of "localhost"? not sure. but "localhost" could be confusing as they are adding a credential for a cloud "other there" not on localhost
[03:07] <wallyworld> if "lxd"works, they can name the credential after the host
[03:07]  * axw tests
[03:07] <wallyworld> the host running lxd
[03:07] <axw> wallyworld: I think it depends on the cloud name, which will be localhost
[03:07] <wallyworld> i think our code supports both lxd or localhost for the lxd provider?
[03:08] <wallyworld> ie we use add-credential google or add-credential aws
[03:08] <axw> wallyworld: for bootstrapping yes, but I don't think so for add-model.. will see
[03:08] <axw> wallyworld: those are the names of the clouds, not the providers
[03:09] <wallyworld> right, but i *think* we special case lxd, but as you say, maybe only for bootstrap
[03:09] <wallyworld> i think we need to tweak add-model if it isn't suported there
[03:09] <axw> wallyworld: the UX for creds is still a bit rough I think
[03:09] <wallyworld> yeah
[03:10] <wallyworld> but so is the lxd/localhost handling
[03:12] <wallyworld> axw: we can land as is but i do think we need a followup to avoid falsely add creds for a cloud on another machine and using the cloud name "localhost"
[03:13] <axw> wallyworld: it's pretty low priority I think. this is an edge case
[03:13] <wallyworld> that's IMHO. i would prefer "lxd" as we say things like we are running a "maas" cloud or an "lxd" cloud. or at least i do
[03:13] <axw> so yes, but probably not today
[03:14] <axw> wallyworld: it would probably be reasonable to fall back to the provider name when looking for creds
[03:14] <wallyworld> ok. if we ship with it as "localhost", so long as both then still work if a changeis made. maybe we can fix before 2.1 final
[03:14] <wallyworld> yeah, i think so
[03:17] <wallyworld> axw: so are you intending to followup with the docs guys to get the new doc link up before the rc goes out tomorrow? might be worth an email to nick or peter?
[03:17] <axw> wallyworld: will do
[03:17] <wallyworld> ta
[03:19] <axw> wallyworld: not much point actually, it's not going to go into the stable docs until the final release is it?
[03:20] <wallyworld> final release is within a week - can we add it but not link to it from anywhere. then the only way you see it is via the juju cli output
[03:22] <axw> wallyworld: it's not going to go on its own page, it'll be on the LXD cloud page
[03:22] <axw> wallyworld: which is linked from the main page, and has stable vs. devel in the URL
[03:23] <axw> wallyworld: it's an edge case, is it really worth all this energy? we can mention it in the release notes and link to the issue or devel docs if it makes it in there
[03:23] <wallyworld> hmmmm. can we land in the devel docs
[03:23] <wallyworld> ok
[03:29] <thumper> WTF? why is 2.1 version updated?
[03:29] <thumper> we haven't done the rc1 release yet
[03:30] <anastasiamac> thumper: according to sinzui's mention in release call: "versions must be numbers and dots to get into proposed and released"
[03:30] <sinzui> thumper: that is true.
[03:31]  * thumper isn't happy
[03:31] <thumper> it isn't a .0 release
[03:31] <thumper> it is an rc release
[03:31] <sinzui> thumper: some juju that read streams with letters in the version fall over
[03:31] <sinzui> thumper: I think you can recall that
[03:31] <thumper> sinzui: pre 1.20
[03:32] <sinzui> thumper: so to prevent breakage, there are controls that will abort if you put an "rc" in them
[03:32] <sinzui> thumper: I also brought this up with the release of 2.0 and no onw made a fuse (they should have)
[03:33] <sinzui> thumper: by putting the rc in *proposed* stream, we are commited to burn numbers. We can put rc1 in the devel stream...but we cannot promote it to proposed
[03:35]  * sinzui wished we could decouple the client's build from the inteded version
[03:44] <sinzui> thumper: consider this if we rename juju to 2.1-rc1 again, and publish the agents to devel. The client will automatically look for agent in devel. The client package could be manually copied to the proposed ppa. The snapped client can also go to the candidate. The only problem is testers that need to adjust streams in scripts will need to set agent-stream=devel
[03:46] <thumper> ick
[03:46] <thumper> we need to fix this
[03:46]  * thumper out
[04:41] <wallyworld> axw: this PR is a lot of noise (fake auth field name as driveby), but fixes keymanager facade permissions https://github.com/juju/juju/pull/6951
[04:41] <axw> wallyworld: ok, looking
[04:41] <wallyworld> ta
[04:43] <axw> wallyworld: next time can you please separate the drive by into a separate commit, so it can be reviewed with unfocused eyes :)
[04:44] <wallyworld> yeah, fiar point
[04:44] <wallyworld> sorry
[04:44] <wallyworld> the changes if noe are in apiserver/keymanager
[04:44] <wallyworld> *of note
[04:58] <blahdeblah> Hi folks - is there going to be a similar fix for 1.25.x as will be in 2.0.3 for the memory leaking?  Just discovered that on of our main production environments on 1.25.6 has been getting OOM killed as well.
[05:02] <axw> wallyworld: left a bunch of comments, will look at tests after you respond
[05:03] <axw> blahdeblah: I don't think the one that affected 2.0.3 applies to 1.25.x
[05:03] <wallyworld> sure ty
[05:03] <axw> blahdeblah: we're up to 1.25.9 now, which does include some leak fixes
[05:05] <axw> (I hear there are still issues in 1.25.9, but it'd be good to rule out hte bugs that have already been fixed)
[05:05] <blahdeblah> axw: In our production deployments, we've found post-1.25.6 releases less stable, and so froze them there.
[05:05] <axw> blahdeblah: I see
[05:07] <blahdeblah> I upgraded two small Canonistack envs to 1.25.10 (in proposed) late last week/early this week, in response to a request in lp:1587644, and they're still experiencing similar symptoms, although without the obvious memory leak.
[05:08] <blahdeblah> But still need jujud-machine-0 restarts every day or two
[05:09] <axw> blahdeblah: ok, well I don't know when a new 1.25 will come. we're all focused on getting 2.1 out the door right now
[05:09] <blahdeblah> yeah - fair enough
[05:09] <axw> blahdeblah: apart from dealing with fallout from the odd prodstack explosion :o
[05:09] <blahdeblah> :-)
[05:09] <blahdeblah> I might poke around for any 1.25 bugs which match the symptoms
[05:15] <babbageclunk> menn0: around? min reviewing this for me? https://github.com/juju/juju/pull/6952
[05:15] <babbageclunk> menn0: It's short!
[05:20] <axw> wallyworld: still reading responses, check my comment about controller tag again please
[05:22] <wallyworld> axw: i'm missing something i think - it looks ok to me. the controller tag is only used in the mothod to check for superuser
[05:22] <wallyworld> the model is passed in separately
[05:22] <axw> wallyworld: the comment was next to the call to common.HasPermission
[05:22] <axw> wallyworld: which does not take a model
[05:23] <wallyworld> oh, i am dyslexic
[05:24] <wallyworld> should be ok now
[05:35] <babbageclunk> or axw? https://github.com/juju/juju/pull/6952
[05:35] <axw> babbageclunk: looking
[05:36] <axw> babbageclunk: you have run the QA for all those clouds right?
[05:36] <axw> or do we have CI tests set up for this?
[05:40] <axw> babbageclunk: LGTM
[05:42] <wallyworld> axw: yeah, i don't think it's possible to have a model owner that's not an admin, hence no test was added. mayber it is with some convoluted test setup, i'll have a look
[05:42] <wallyworld> might be doable with fake authenticator
[05:43] <axw> wallyworld: I can imagine that someone might want to create a model and then hand it off, but IMO that would be transfer of ownership. or we should call it "creator" rather than owner
[05:44] <wallyworld> yeah
[05:44] <wallyworld> people have asked for that
[05:45] <axw> wallyworld: I just replied again. while writing I realised that checkCanRead also will need to be updated if we're going to stick with the existing behaviour around system identity key
[05:46] <wallyworld> yeah, i'm going to review the logic again
[05:49] <axw> wallyworld: I added an RC1 section to the release notes, with a blurb about LXD creds. would you please have a read when you have a chance?
[05:49] <wallyworld> sure ty
[06:19] <babbageclunk> axw: Sorry, bathtime
[06:20] <axw> babbageclunk: oh, all clean now? ;)
[06:20] <babbageclunk> axw: I've done a couple (lxd and ec2)
[06:20] <babbageclunk> axw: yes thanks!
[06:20] <axw> babbageclunk: do you know if CI is testing this already? we should make sure it's tested on all the clouds
[06:20] <babbageclunk> axw: I think veebers has CI for all clouds
[06:20] <axw> babbageclunk: ok cool
[06:21] <babbageclunk> axw: I'll confirm with him tomorrow
[06:22] <wallyworld> axw: release notes look ok, they can tidy formatting. i've pushed changes also. got to rush to school pickuo, bbiab
[06:22] <axw> wallyworld: thanks. looking
=== mpontillo_ is now known as mpontillo
=== seyeongkim_ is now known as seyeongkim
[06:45] <wallyworld> axw: thanks for review. i didn't see the harm in users reading the public bit of the system key?
[06:46] <axw> wallyworld: that's the existing behaviour. I don't have a sproblem with us changing it in this case, but I want to know that it's intentional and tested
[06:46] <jam> wallyworld: thumper: I'm trying to reconcile 2.1 vs develop
[06:46] <jam> it looks like thumper added a signature to stateForRequestAuthenticade to return a "releaser" funct
[06:46] <jam> but wallyworld added a modelRestHandler.stateAuthFunc
[06:47] <wallyworld> axw: ok, best to stick with existing behaviour. i was being strict about that for write
[06:47] <jam> wallyworld: but modelRestHandler.stateAuthFunc is never called
[06:47] <jam> it has a hard-conded stateForRequestAuthenticated in ServeGet
[06:47] <axw> wallyworld: SGTM
[06:48] <wallyworld> jam: i'll have a look, give me a few minutes
[06:48] <jam> wallyworld: k
[06:48] <jam> I'm trying to get my fixes in 2.1 into 2.2, and bringing some of Tim's along for the ride.
[06:50] <jam> wallyworld: ping me when you're available so I can go through it with you
[06:50] <wallyworld> ok
[06:55] <axw> wallyworld jam: I just noticed that the version was already bumped to 2.1.0. did we already cut a release? stuff landed today won't make it?
[06:55] <wallyworld> axw: they did that in preparation
[06:56] <jam> axw: i don't know. I just saw the version bump myself.
[06:56] <jam> wallyworld: but wouldn't it be 2.1rc1 ?
[06:56] <wallyworld> the rc will have 2.1.0 as thge version
[06:56] <wallyworld> i would have thought so
[06:56] <wallyworld> bu tthey want to take the binary and release unchanged as final
[06:56] <wallyworld> i don't agree with that myself
[06:56] <wallyworld> what if we need a rc2
[06:56] <axw> that sounds crack
[06:57] <wallyworld> yep
[07:01] <wallyworld> jam: free to talk now
[07:01] <jam> wallyworld: HO or IRC?
[07:02] <wallyworld> HO easier, standup one?]
=== Dmitrii-Sh_ is now known as Dmitrii-Sh
[07:22] <anastasiamac> wallyworld: axw: jam: earlier conv with sinzui re:version https://pastebin.canonical.com/178735/
[07:22] <anastasiamac> wallyworld: u were in the release call, if u disagreed, it'd b more constructive to have do so there
[07:23] <wallyworld> anastasiamac: i tried to but the call ended
[07:23] <anastasiamac> wallyworld: call ppl back :D
[07:23] <wallyworld> honestly, the proposal is crack, john will send an email to get it fixed
[07:23] <anastasiamac> jam: axw: for reference not a new issue. we hit it at every release
[07:24] <wallyworld> anastasiamac: not any more
[07:24] <wallyworld> the only juju versions that had issues were < 1.20
[07:25] <jam> needing to set agent-streams=devel seems better than not being able to make an rc2, or have a 2.1.0 final that can identify itself as different.
[07:25] <wallyworld> we don't support those any more
[07:25] <jam> wallyworld: well, we'd have to actually test that 1.25 doesn't crap itself with 'rc' in the proposed stream
[07:25] <wallyworld> it doesn't
[07:25] <wallyworld> that was fixed for 1.20
[07:26] <wallyworld> or 1.19 even
[07:26] <anastasiamac> wallyworld: jam: great discussion to have with release team \o/
[07:26] <wallyworld> we shouldn't need to - the policy is clear
[07:26] <anastasiamac> who r coming online *shortly*
[07:26] <wallyworld> this change was done on a whime
[07:26] <anastasiamac> wallyworld: obviously not if they made a request and approved it
[07:26] <wallyworld> as can be seen from all the wtf
[07:26] <wallyworld> it wasn't approved
[07:27] <wallyworld> not by us
[07:27] <wallyworld> and there was nothing in writing etc
[07:27] <anastasiamac> it was done on request from sinzui directly
[07:27] <wallyworld> if it were approved tim, andrew, john and i would not be going etf etf
[07:27] <wallyworld> *wtf
[07:27] <anastasiamac> nothing inwritng, except in releas call minutes... did u need it in blood?
[07:28] <wallyworld> for such a major change to policy, it needs proper discussion, not a btw at the end of the release call
[07:28] <anastasiamac> it was not at the end
[07:29] <anastasiamac> anyway, nothing was released yet and now is ur chance to fix
[07:29] <wallyworld> yep, that is good
[07:29] <wallyworld> jam will follow up
[07:29] <anastasiamac> obviously, communication failure since 1.20
[07:29] <wallyworld> don't buy that
[07:29] <anastasiamac> what do u ever?
[07:32] <wallyworld> axw: i'm having trouble booting lxd. i've cleared ~/.config/lxc and there's nothing in credentials.yaml..... but at end of bootstrap
[07:32] <wallyworld> 2017-02-09 07:27:51 INFO juju.cmd supercommand.go:63 running jujud [2.1.0.1 gc go1.6.3]
[07:32] <wallyworld> 2017-02-09 07:27:51 ERROR cmd supercommand.go:458 new environ: Get https://10.132.22.1:8443/1.0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "root@wallyworld")
[07:33] <wallyworld> this is on 2.1 branch
[07:33] <wallyworld> i'll dig in a bit after soccer, not sure if you've seen that issue
[07:34] <axw> wallyworld: ehhh? I have not seen that, no
[07:35] <wallyworld> yeah weird
[07:35] <wallyworld> i did apt remove and then apt install lxd yesterday
[07:35] <wallyworld> to test a snap issue
[07:36] <wallyworld> maybe time for lxd init again
[07:38] <axw> wallyworld: oh, you probably need to remove any credentials you have stored for lxd in credentials.yaml then
[07:38] <wallyworld> axw: yeah i checked, none there
[07:38] <axw> hm, dunno then
[07:38] <wallyworld> yeah, me either
=== frankban|afk is now known as frankban
[08:36] <axw> wallyworld: did you see that I reviewed your PR?
[08:37] <axw> ah, soccer
=== frankban is now known as frankban|afk
=== frankban|afk is now known as frankban
=== admcleod_ is now known as admcleod
[19:39] <balloons> redir, not sure what the thought is on pushing PR's against 2.1 and develop. But here's one if you don't mind having a look: https://github.com/juju/juju/pull/6956 and https://github.com/juju/juju/pull/6957
[19:39] <redir> balloons: looking
[19:42] <redir> balloons: I don't know about 2.1
[19:42] <redir> develop LGTM though.
[19:42] <balloons> redir, as in, you don't know if they want things to land or ?
[19:42] <redir> balloons: correct
[19:43] <balloons> ahh. I want it to land in 2.1 to help us toward that bless :-)
[19:43] <redir> OK.
[19:43] <redir> shipit
[19:44] <redir> just saying I don't know, but some blesses would be greatly celebrated I'm sure:)
[19:46] <balloons> redir, lol.. :-) Ok,
[19:47] <redir> Looks great to me balloons
=== frankban is now known as frankban|afk
=== frankban|afk is now known as frankban
[21:40] <redir> another easy review; https://github.com/juju/juju/pull/6958
[21:41] <redir> anyone ^^
[21:41] <redir> juju-dev ^^
[21:55] <babbageclunk> redir: yay, easy review! Looking!
[21:56] <babbageclunk> redir: this doesn't sound that easy
[21:58] <redir> hehe it's small
[22:01] <babbageclunk> redir: LGTM'd
[22:01] <redir> babbageclunk: tx
[22:18] <babbageclunk> redir: also I'm OCR for me, but not for your timezone
[22:19]  * redir blinks
[22:19] <redir> thanks for the review in my timezone:)
[22:21] <babbageclunk> redir: :)
[22:30] <alexisb> anastasiamac, ping
[22:31] <anastasiamac> alexisb: pong
[23:10] <thumper> babbageclunk: https://github.com/juju/juju/pull/6959
[23:20] <babbageclunk> thumper: reviewed - lgtm
[23:21] <babbageclunk> special OCR deal today - any PR that's more deletions than additions is auto-approved!
[23:21] <babbageclunk> (offer not valid where void)
[23:21] <babbageclunk> I mean, offer void where prohibited
[23:26] <thumper> :)
[23:28] <axw> thumper: there is quite a lot of awesome for such a small amount of code with custom pprof profiles :)
[23:29] <thumper> axw: yeah
[23:29] <thumper> when I realised that we needed to hook into close and not use a finalizer
[23:29] <thumper> it made sense to go with pprof
[23:29] <thumper> so nice to look at too
[23:29] <thumper> and less things to hook up
[23:29] <thumper> thanks for pointing that out to me
[23:29] <thumper> it's good
[23:29] <axw> thumper: does the tool just work? i.e. can you use go tool pprof --svg ?
[23:29] <axw> thumper: no worries
[23:29] <thumper> haven't tried that actually
[23:31] <redir> babbageclunk: this should look familiar: https://github.com/juju/juju/pull/6960
=== frankban is now known as frankban|afk
[23:37] <babbageclunk> redir: looking
[23:38] <babbageclunk> redir: net negative lines! Approved!
[23:38] <babbageclunk> redir: Also a legitimate change.
[23:46] <axw> anastasiamac thumper: standup?
[23:57] <babbageclunk> thumper/axw: review this? https://github.com/juju/juju/pull/6961
[23:57] <babbageclunk> unfortunately adds lines.
[23:59] <axw> babbageclunk: will do after standup, maybe after taking kids to school