[00:04] <evhan> After adding storage to a running k8s charm, is it necessary to perform some manual action, (e.g. cycle the units)? I have a set of "storages" in pending after having added storage, but can't see any corresponding activity and I'm unsure how to make it go.
[00:11] <wallyworld> evhan: how did you add the storage? juju attach-storage isn't supported for k8s units as all units need to be homogeneous. did you use kubectl patch on a specific pod?
[00:13] <evhan> No, introduced to metadata.yaml and upgrad-charm'ed.
[00:13] <evhan> Sorry, should have mentioned that.
[00:14] <evhan> I suppose with the CLI tools it would normally be `juju add-storage` followed by `attach-storage`, in which case I would have gotten the message that it was unsupported.
[00:14] <wallyworld> in metadata it declares it can support storage but you need a --storage argument to allocate storage against the declaration. did you use --storage when upgrading?
[00:18] <evhan> Yeah, sorry, the command was: `juju upgrade-charm --path . fluent-bit-k8s --storage varlog=20M`
[00:26] <wallyworld> evhan: it may well be that's not something we've proprtly tested and there's a bug to fix, we'll have to look at it
[00:27] <evhan> OK, I'll try to collect some more info. Playing around a bit more I think I may have taken the side door and avoided all the "you can't do that" signs.
[04:38] <tlm> wallyworld, babbageclunk: what is the best method to work out the current controller hostname/ip:port in workers ?
[04:41] <babbageclunk> tlm: hmm, not totally sure - maybe from the agent worker? Although that's machine level - would need to check. wallyworld?
[04:41] <wallyworld> the api port is a controller config setting
[04:43] <wallyworld> there's typically no hostname, only ip. there's a controller nodes collection, and also machines. you get the machine if from the controller tag
[04:44] <tlm> yeah ip is fine
[04:44] <tlm> what do we report when the controller is inside k8s
[04:44] <tlm> api server really wants a service in that case but I am tossing up if I should try and do that kung fu
[04:47] <hpidcock> tlm: if we know the controller is in the same cluster, maybe we use the service directly, if the controller is external, we can create a service and publish endpoints
[04:47] <wallyworld> we record the service ip address in state
[04:48] <wallyworld> for the controller, there's a cloudService collection
[04:48] <tlm> hpidcock: yep thats fine but still need to get the ip
[04:48] <wallyworld> i can't recall off hand what the doc id is for the controller
[04:48] <wallyworld> we use cloudService for the service address for all deployed apps
[04:48] <tlm> does the data already exist in the machine manifolds ?
[04:49] <tlm> just need ip and a bool of current controller if it's inside of k8s
[04:49] <wallyworld> you need to query state in the manifold start func where the statepool / state object is passed in
[04:49] <wallyworld> use have the controller tag, so can get the key of the controllerService doc to query
[04:49] <tlm> ok will take a look. Going to hard code an ip in and get this finished then swing back to it
[04:50] <wallyworld> righto
[04:50] <wallyworld> kelvinliu might remember off the top of his head what the cloudService doc id for the controller is
[04:50] <wallyworld> to allow the service ip to get retrieved
[04:53] <kelvinliu> wallyworld: cloudService containers the svc info but not container info
[04:53] <wallyworld> that's what we want
[04:53] <wallyworld> the service ip address
[04:53] <tlm> are we talking kube service ?
[04:53] <wallyworld> of the controller
[04:54] <wallyworld> we talk via the service ip always
[04:54] <tlm> if so do we make a service even if the controller is not in k8's ?
[04:54] <kelvinliu> right, cloudService .Addresses()
[04:54] <wallyworld> what's the doc key though?I can't recall offhand
[04:56] <hpidcock> wallyworld: when your free next https://github.com/juju/juju/pull/11214 no rush, I'm working on the next PR and will assume this is "done done"
[04:56] <kelvinliu> https://github.com/juju/juju/blob/develop/state/state.go#L1135
[04:56] <kelvinliu> it's controller UUID for k8s controller
[04:57] <wallyworld> ah yeah, thanks
[04:58] <tlm> thats k8's specific
[04:58] <tlm> doesn't apply when we are not running the controller in k8's
[04:59] <wallyworld> right, but non k8s deployments do not write anything to the cloudService collection
[04:59] <wallyworld> the cloudContainer collection is also k8s specific
[05:00] <tlm> no what hpidcock and I are thinking is, for every controller in juju we make a service in each k8's cluster we have. This service either points to the internal controller or an external one. We can then just piggy back off of this known point for things like admission/sidecars etc etc
[05:00] <wallyworld> and this is a k8s only worker right?
[05:01] <tlm> yep
[05:02] <wallyworld> depending on where the controller is running, i think we'll need to get it's ip address from either the machine collection or the cloudService collection
[05:04] <tlm> we'll only need ip for outside of k8s. Maybe we HO this at some stage to make sure it's a good idea and not cup holders
[05:08] <wallyworld> i'm free whenever
[05:08] <tlm> ok joining :)
[05:09] <tlm> hpidcock: want to join ?
[05:09] <hpidcock> sure why not
[05:56] <kelvinliu> https://github.com/juju/juju/pull/11226 merge 2.7 to develop, anyone free to take a look? thanks!
[06:05] <hpidcock> kelvinliu: thanks for getting that all merged. only one issue
[06:06] <kelvinliu> thx, just found the problem, fixed
[06:54] <hpidcock> wallyworld: only reason to move up the merge of that PR is an intermittent test failure fix for ActionSuite.TestFindActionTagsByLegacyId
[06:55] <wallyworld> ok, i'll look
[06:56] <hpidcock> merge on https://github.com/juju/juju/pull/11224 failed due to it
[07:46] <wallyworld> hpidcock: didn't quite get done reviewing, have to EOD, will finish tomorrow
[08:56] <nammn_de> manadart: I opened a initial move-to-space apiserver patch. I do have some unclear things which I mentioned in the patch as well as in the network cli spec. Could you take a look please? https://github.com/juju/juju/pull/11213
[08:57] <nammn_de> manadart achilleasa: this is the command part patch of the remove-command.  I have linked the spec as well https://github.com/juju/juju/pull/11183 If someone wants to take a look at the spec instead/with
[08:59] <manadart> nammn_de: I'll take a look when I can.
[09:10] <achilleasa> jam: did you have a chance to take a look at my replies in 11204? Wdyt about renaming the call to "CommitHookChanges"?
[09:11] <jam> achilleasa: offhand that sounds pretty good. I'm in a bit of a crunch right now, but I'll try to get to it this afternoon
[13:01] <stickupkid> jam do you have the rights to merge this? https://github.com/go-goose/goose/pull/77
[13:01] <stickupkid> or rick_h
[13:06] <hml> stickupkid: I think I do
[13:07] <hml> stickupkid: merged
[13:07] <stickupkid> hml, tyvm
[13:08] <stickupkid> yay unblocked myself
[13:08] <stickupkid> win win
[14:28] <manadart> nammn_de: I think the sections should be "Setting units to track a branch", "Changing application config on a branch" etc, rather than "Units are made to track the branch", "Changes are made to the branched application" and so on.
[14:29] <nammn_de> manadart: rgr
[14:35] <nammn_de> manadart: Need some input on `6. Aborting a branch`. After trying out it seems that we can only abort a branch when no units are following it. The error message indicates that resetting the settings is working as well. Did not work for me
[14:47] <manadart> nammn_de: Not sure what you mean. If units tracking the branch are set to track "master", then we should be able to abort it.
[14:48] <manadart> If the branch has no config changes, then committing the branch is an effective abort.
[14:50] <nammn_de> manadart: ahh making them track master. Somehow this slipped me :D. Will add that to the post.
[14:50] <nammn_de> But the abort message returns "ERROR branch is in progress. Either reset values on tracking units or remove them to abort." Which indicate if I reset the values I can run `juju abort <branch>` again, which is not the case
[14:52] <manadart> nammn_de: I would have to read the code.
[14:52] <nammn_de> manadart: ahh I am just meaning this independant of code. Just from an user perspective
[14:53] <nammn_de> was more of a concern
[14:53] <hml> quick backport review pls: https://github.com/juju/juju/pull/11228
[14:55] <hml> another quick backport review pls: https://github.com/juju/juju/pull/11229
[14:56] <hml> a third is coming once 11229 lands
[14:56] <nammn_de> just wanted to point out, that aborting `juju abort` a branch seems to not work that easily.  Tried different possibilites. https://paste.ubuntu.com/p/Rsf5XDPrWZ/
[14:58] <manadart> hml: Approved the first one.
[14:59] <manadart> nammn_de: I remember now. Since we are going to be doing charm upgrade rollout using branches, you have to see them through.
[15:00] <manadart> nammn_de: Have you read the spec? I'll dig it up.
[15:01] <nammn_de> manadart: No only the doc. If you have the spec I can read it up.
[15:01] <nammn_de> My point is mostly that the returning error message is confusing . As it indicates that resetting helps
[15:01] <hml> manadart:  ty
[15:08] <stickupkid> manadart, you got 5-10 minutes to pair this openstack port changes, want to make sure I'm getting it right?
[15:11] <manadart> stickupkid: Yep.
[15:20] <nammn_de> hml: you worked on this before afaict. Does this patch make sense to you? https://github.com/juju/juju/pull/11230
[15:22] <hml> nammn_de: looking
[15:27] <hml> nammn_de:  approved
[15:29] <nammn_de> hml: thanks!
[15:32] <rick_h> achilleasa:  hml was just going to ping that anything we can backport into 2.7.3 that we want folks to have charm-wise would be great. I saw hml had a couple of them going so <3
[16:17] <achilleasa> stickupkid: so... remember that interesting test with leadership?
[16:18] <stickupkid> achilleasa, yeah
[16:18] <stickupkid> better be good
[16:18] <hml> merge conflicts fixed for pr, rebase review please: https://github.com/juju/juju/pull/11229
[16:18] <stickupkid> looking for a book type story out of this achilleasa
[16:18] <achilleasa> looks like LeadershipClaimer hits the one in state/leadership.go but LeadershipChecker doesn't :D
[16:18] <stickupkid> HA
[16:18] <achilleasa> all I need to do is find out why this is set up like that...
[16:19] <stickupkid> that'll be why it couldn't find it then
[16:19] <achilleasa> I verified that the checks use the same model UUID
[16:19] <stickupkid> BECAUSE IT'S CRAP
[16:19] <stickupkid> haha
[16:19] <freyes> hello, I was taking a look to the function AllSpaces() which should return the spaces in a model, but I'm not seeing any filtering by model uuid, am I missing some context set somewhere? - https://github.com/juju/juju/blob/2.7/state/spaces.go#L288
[16:19] <achilleasa> if achilleasRunsTests { leadershipChecker = makeRandomChecker() }
[16:19] <stickupkid> achilleasa, hahaha yeah
[16:22] <achilleasa> freyes: you would normally run that on a state instance obtained for a particular model; it will filter by uuid behind the scenes for you
[16:23] <freyes> achilleasa, ok, so I should go back on call stack to figure out where the state object is being built, right?
[16:24] <freyes> achilleasa, what happens if there is state instance defined? should be failing with error?
[16:24] <achilleasa> freyes: if you are debugging an API call you will normally find this call in the facade code (apiserver/facade/{agent, client})
[16:24] <freyes> *there is no state
[16:24] <freyes> achilleasa, I'm looking into the upgrades steps
[16:25] <freyes> if you want more context, I'm troubleshooting this error: https://bugs.launchpad.net/juju/+bug/1863777
[16:25] <mup> Bug #1863777: getting machine upgrade ops: space with name:  not found <sts> <upgrade-juju> <juju:New> <https://launchpad.net/bugs/1863777>
[16:29] <achilleasa> freyes: hmmm... I guess this is the one you are looking for: https://github.com/juju/juju/blob/develop/upgrades/steps_27.go#L28-L33
[16:30] <achilleasa> and https://github.com/juju/juju/blob/develop/upgrades/steps_27.go#L42-L48 (most probably)
[16:32] <freyes> right, it's where I started from, I should probably keep reading that code and probably study it with a debugger, it will probably be easier
[16:33] <freyes> achilleasa, appreciated, and congratulations on your book :-)
[16:33] <achilleasa> freyes: try searching in the machines collection. I think the error might be refering to the addresses array in the machine docs
[16:33] <achilleasa> freyes: thanks ;-)
[16:34] <freyes> achilleasa, ah, interesting what you just mentioned .... it was recently mentioned to me they manually added new nics to the controller machines, so probably the ip addresser registered them and it's expected to have that space
[16:42] <manadart> freyes: What has happened here is that the space name has been supplied with machine addresses by MAAS, but reload-spaces has not consumed MAAS' space topology into the collection.
[16:43] <freyes> manadart, but probably because maas is not aware that the controllers machines have nics in the maas-mgmt space since they were hotplugged
[16:44] <freyes> https://bugs.launchpad.net/juju/+bug/1863777/comments/1
[16:44] <mup> Bug #1863777: getting machine upgrade ops: space with name:  not found <sts> <upgrade-juju> <juju:New> <https://launchpad.net/bugs/1863777>
[16:45] <freyes> so this apparent inconsistent state is the one triggering the issue
[16:45] <manadart> freyes: I think MAAS and Juju are aware, given that addresses in the machines collection are decorated with the name. But without running juju reload-spaces on the (controller) model, it is not in Juju's spaces collection, so when we try to look up an ID for it, it is not found.
[16:47] <freyes> I see, I think I have what's needed to come up with a reproducer, I will post my progress in the bug
[16:47] <freyes> thanks, manadart
[16:48] <manadart> Sure thing freyes.
[16:51] <achilleasa> stickupkid: so the checker is from the apiserver package....
[16:52] <stickupkid> achilleasa, is it the raft one?
[16:54] <achilleasa> stickupkid: I can't figure out where it gets initialized. So I took a shortcut by patching the suite state and creating a new uniter API.... looks like it gets the state ones now...
[16:54] <stickupkid> ha
[16:55] <achilleasa> yeah... you normally get back a "uniter state" instance from these calls which seems to be a shim but I can't really tell...
[17:00] <stickupkid> hml, done
[17:00] <hml> stickupkid: ty!
[17:01] <hml> stickupkid: turns out my 3 backports have overlap enough for them needing to be done in serial.  :-(
[22:27] <timClicks> question for the audience - which juju plugins do you use? (did you know that plugins were a thing?)
[23:54] <evhan> As of yesterday. :)