[00:52] <wallyworld> vino: did you have a minute, i just looked at juju run pr, quick hangout?
[00:57] <vino_> yes
[00:58] <vino_> wallyworld: yes
[00:59] <wallyworld> vino: see you in standup
[01:00] <wallyworld> anastasiamac: small PR if you have a chance - needed for 2.4 beta 1 https://github.com/juju/juju/pull/8613
[01:03] <anastasiamac> wallyworld: nws, m about to pr up the comment u've asked and will need a review too :D
[01:18] <anastasiamac> babbageclunk: veebers: could u please TAL https://github.com/juju/juju/pull/8614 does it make sense to u coming in cold into the context?
[01:21] <babbageclunk> anastasiamac: will take a look soon
[01:22] <wallyworld> anastasiamac: will do just let me know the PR
[01:27] <anastasiamac> wallyworld: https://github.com/juju/juju/pull/8614 :)
[01:27] <wallyworld> looking
[01:28] <wallyworld> anastasiamac: lgtm, ty
[01:37]  * anastasiamac does a fast-landing dance \o/
[01:45] <babbageclunk> oops, sorry anastasiamac
[01:45] <babbageclunk> Oh, should have looked first before holding off!
[01:46] <anastasiamac> babbageclunk: u r sorry i dance? :D
[01:47] <babbageclunk> No, I just should have paused what I was doing!
[01:48] <anastasiamac> babbageclunk: nws, i just needed someone to tell me that that i made sense :D wallyworld is good at keeping my english straight :D PR is heading for landing now
[01:49] <anastasiamac> babbageclunk: no no, it was not worth interuppting anyone... was just a review ;D
[01:53] <babbageclunk> Well I'll do better next time.
[02:06] <veebers> sorry anastasiamac was at lunch :-\
[02:06] <anastasiamac> veebers: all good!
[02:13]  * thumper goes to make a coffee
[02:23] <veebers> wallyworld: when you have a spare moment if you could ack the test addition I made since your approval: https://github.com/juju/juju/pull/8599
[02:23] <wallyworld> sure, sorry i totally forgot
[02:23] <veebers> nw
[02:25] <wallyworld> veebers: just a minor tweak
[02:27] <veebers> gah, my illusion of perfection is broken ;-) I'll fix it up and merge
[02:35] <wallyworld> anastasiamac: thanks for review, i missed those other errors, i've updated PR with a new commit
[02:46] <veebers> wallyworld: remind me the command that shows charm hooks run?
[02:46] <veebers> hah, that's almost not english :-)
[02:47] <wallyworld> juju show-status-log
[02:48] <veebers> awesome thanks
[02:49] <veebers> wallyworld: is install hook retried by default perchance?
[02:49] <wallyworld> veebers: yah, all hooks are retried i think 3 times
[02:49] <veebers> wallyworld: ah right cheers, I'll update the test charm to take that into account :-)
[02:50] <wallyworld> yeah sorry should have mentioned that
[02:50] <veebers> nw, no biggie
[02:53] <wallyworld> anastasiamac: i just noticed you already approved but given the changes i made for errors are non-trivial, would appreciate another look; they are all in the 2nd commit
[02:53] <anastasiamac> wallyworld: k, looking now :D
[02:53] <wallyworld> awesome, ty
[02:59] <wallyworld> yay, thanks for review
[03:00] <thumper> babbageclunk: got a minute, I'm a little confused
[03:00] <babbageclunk> thumper: sure
[03:00] <thumper> babbageclunk: jumping in the HO
[03:01] <anastasiamac> wallyworld: that was really good !!! loved how u consolidated all these errors. once i saw it, it totally made sense
[03:02] <wallyworld> yeah, we do tend to scatter shite everywhere
[03:02] <wallyworld> was good to clean it up
[03:02] <wallyworld> thanks for noticing in the PR
[03:02] <wallyworld> was late when i did it :-)
[03:02] <anastasiamac> \o/
[03:31] <veebers> It appears that juju is attempting to re-try the install hook at least 8 times. Am I doing something odd in my charm to trigger this?
[03:31] <veebers> all my charm is doing is in the install hook increment the attempt count (file stored in tmp) and setting: "status-set blocked "Install hook failed on purpose." then exit 1
[03:35] <veebers> wallyworld: oh you;re back :-) I was just complaining: It appears that juju is attempting to re-try the install hook at least 8 times. Am I doing something odd in my charm to trigger this?
[03:35] <veebers> all my charm is doing is in the install hook increment the attempt count (file stored in tmp) and setting: "status-set blocked "Install hook failed on purpose." then exit 1
[03:35] <wallyworld> yeah, using vpn drops irc :-(
[03:36] <wallyworld> i don't think you're doing anything amiss
[03:36] <wallyworld> i think juju reties 3 times
[03:36] <wallyworld> maybe just exit 1 and see what happens though that shouldn't make a difference
[03:37] <thumper> wallyworld: it retries more than 3 times
[03:37] <wallyworld> really? :-(
[03:37] <wallyworld> i wonder why
[03:37] <wallyworld> even 3 seems too many
[03:37] <wallyworld> i had recalled that we settled on 3 but obviously i'm wrong
[03:38] <veebers> it's tried 10 times now ^_^
[03:38] <wallyworld> i'm sure in my testing it didn't do that
[03:38] <wallyworld> it retried what i could have sworn was 3 times and then gave up
[03:38] <wallyworld> and left the unit in error state
[03:39] <veebers> wallyworld: if your testing was just exit 1 always you might not have seen it, but because I'm waiting to make sure it stops trying to ensure my resolve is the thing that fixed it I see it happen
[03:39] <wallyworld> but could be wrong there too
[03:39] <wallyworld> if you always exit 1 then resolve will be the thing that moves it on
[03:40]  * thumper thrashes his computer somewhat
[03:42] <veebers> wallyworld: right, I can resolve --no-retry to move it on, but I can't ensure that resolve will re-run the hook when requested (although it seems perhaps that's not needed as it's *always* retrying)
[03:43] <thumper> babbageclunk: got 5 minutes?
[03:44] <thumper> babbageclunk: I'm invoking the live demo demons
[03:44] <thumper> babbageclunk: demonstrating as yet untried code
[03:44] <babbageclunk> sure
[03:44] <wallyworld> veebers: yeah, the code will need to be read to see what's going on. i can't recall off hand
[03:44] <wallyworld> --no-retry would be a good start
[03:45] <veebers> wallyworld: ack. I'll proceed with a functional test for no-retry. If I get nosey / have time I might try figure out what the code is doing there too
[03:45] <wallyworld> sgtm
[03:46]  * thumper jumps back in HO
[03:56] <veebers> wallyworld: I'm confused, my juju 2.3.5 resolve works as expected, passing --no-retry skips the install hook and goes onto the start hook
[04:15] <thumper> babbageclunk: I realise there was one more space where I haven't hooked it up :)
[04:15] <babbageclunk> thumper: I mean, I guess that had to have been the problem
[04:16] <thumper> :)
[04:18] <babbageclunk> thumper: so it's working now with quick status changes?
[04:18] <thumper> shtill threading
[04:34] <thumper> babbageclunk: got it using the new code, but all coming back as unknown not alive
[04:34] <thumper> so all machine showing down
[04:34]  * thumper sighs
[04:34] <babbageclunk> doh
[04:35] <thumper> added more logging
[04:35] <thumper> ah FFS
[04:36]  * thumper knows
[04:46] <thumper> babbageclunk: I think I have it this time...
[04:46] <babbageclunk> by George, I think he's got it!
[04:49] <thumper> nope
[04:49] <thumper> got logic backwards
[04:49] <thumper> trying again
[04:49] <thumper> this is the culmination of around two years of effort
[04:49] <thumper> I'll be celebrating when this is in
[04:50] <wallyworld> thumper: sent email about upgrade test for 2.3.6 - short answer NFI, can't reproduce
[04:51] <thumper> BOOM!!!
[04:51] <thumper> working
[04:51] <thumper> wallyworld: ok
[04:51] <thumper> babbageclunk: wanna see it before I run to get Maia
[04:51] <thumper> ?
[04:53] <thumper> too late
[04:53] <thumper> gotta go
[04:54] <thumper> I'll be back on later tonight to polish and put up for review
[04:54] <thumper> wallyworld: this is new presence hooked up in status
[04:54] <wallyworld> oh joy
[04:54] <thumper> with user updatable controller feature flags
[04:54] <wallyworld> faaark, nice
[04:54] <thumper> wallyworld: jump on a HO
[04:54] <thumper> I want to show someone
[04:54] <wallyworld> righto
[04:54] <wallyworld> lol
[04:54] <wallyworld> vino_: go fmt is still sad. you'll need to do a quick fix for run_test.go
[04:54] <vino_> that same line 95 ?
[04:55] <thumper> wallyworld: in our 1:1
[04:55] <vino_> ok let me look
[05:01] <wallyworld> vino_: there's about 4 errors
[05:02] <babbageclunk> thumper: oops, sorry
[05:06] <vino_> i cudnt get this githook fix in my setup.
[05:06] <vino_> huh..
[05:08] <anastasiamac> thumper: wallyworld: PTAL licencing for 2.3 - https://github.com/juju/juju/pull/8615
[05:09] <wallyworld> ok
[05:09] <wallyworld> jeez, 31 files
[05:09] <anastasiamac> wallyworld: but mostly tests :D
[05:10] <anastasiamac> 31? m seeing 25...
[05:14] <wallyworld> maybe i can't count
[05:21] <veebers> wallyworld: you're 'resolve --retry' fix, did that go in for 2.3.5 or 2.3.6?
[05:21] <wallyworld> 2.3.6
[05:23] <veebers> wallyworld: so with 2.3.5 we should see that 'resolve --no-retry' does in fact try the errored hook? I'm not seeing that, but I"ll double down now and confirm that
[05:24] <wallyworld> yeah, i think 2.3.5 has the issue but hmmm, not sure now
[05:30] <vino_> wallyworld: run_test.go - 4 errors correct ?
[05:31] <wallyworld> sounds right yeah, that's whay my go fmt said i think
[05:31] <vino_> line: 105, 110, 119
[05:31] <wallyworld> simething like that, don't have the putput anymore
[05:31] <vino_> no worries.
[05:38] <vino_> wallyworld: done. plz verify
[05:40] <wallyworld> looking
[05:42] <wallyworld> vino_: i thought there were four errors, https://pastebin.ubuntu.com/p/b3DM8qm4m3/
[05:47] <kelvinliu_> wallyworld: i still got the same problem. so after I run `juju deploy ./mysql`, I got this message immediately, after unit status changed to `allocating`.
[05:48] <vino_> can u see that ?
[05:48] <wallyworld> which message?
[05:48] <vino_> next to @line104
[05:49] <wallyworld> lookin again at pr
[05:50] <wallyworld> vino_: ah, look ok. the pr showed 2 changes together
[05:50] <wallyworld> sorry, looks good, will approve
[05:50] <vino_> yea.
[05:50] <vino_> but i shd get this githook and few other things fixed sooner..
[05:50] <vino_> no sorry required :)
[05:50] <wallyworld> yeah, see if you can get it working
[05:51] <wallyworld> also, our landing bot shoulld not accept prs with these errors
[05:51] <wallyworld> we need to fix that
[05:51] <wallyworld> kelvinliu_: which message are you seeing?
[05:51] <kelvinliu_> wallyworld: some logs are here https://pastebin.ubuntu.com/p/D9sjnXvY4K/
[05:55] <wallyworld> kelvinliu_: i can't see anything obvious. does juju status show an incremented version number to indicate the upgrade went ok? ie 2.4-beta1.2 or something like that
[05:55] <wallyworld> you could try creating a new model and trying again
[05:57] <kelvinliu_> wallyworld: the controller upgrade went ok, and tried it a few times, now I got 4 testcaas models
[05:58] <wallyworld> you should see in the operator logs that the charm reactive code is calling set-pod-spec
[05:58] <wallyworld> this triggers juju to create the unit
[05:58] <wallyworld> which then creates the mysql pod
[05:59] <kelvinliu_> wallyworld: no, it's not there
[05:59] <wallyworld> ok, let me run up a k8s cluster and see what my logging says
[06:01] <kelvinliu_> wallyworld: so after `juju-unit  executing    running start hook` then `workload   unknown`, then `juju-unit  idle`
[06:03] <wallyworld> i can't recall the exact order off hand, i'll see when my k8s cluster starts
[06:03] <kelvinliu_> wallyworld: thx
[06:04] <kelvinliu_> wallyworld: or do u have a couple of minutes for a screen share session?
[06:05] <wallyworld> kelvinliu_: i won't be much help until i run up my k8s bundle to compare
[06:05] <kelvinliu_> wallyworld: ok, sure, we will wait for the cluster up and running first
[06:26] <wallyworld> kelvinliu_: i just started a k8s cluster and deployed mysql without any problem
[06:26] <wallyworld> did you want to do a hangout?
[06:27] <kelvinliu_> wallyworld: yes, plz
[06:27] <wallyworld> see you in standup
[06:27] <kelvinliu_> yup
[06:48] <vino_> have a quick question.. only snap install, will install the service files in /var/lib/juju/init ?
[07:01] <vino_> i dont see in /var/lib/juju/tool
[07:23] <wallyworld> vino_: snap install of juju is just the client
[07:23] <wallyworld> when you bootstrap, it pulls down the agent binary and creates the systemd files on the vm
[07:24] <wallyworld> so you need to bootstrap and ssh into the controller machine to see what was done
[07:25] <kelvinliu_> wallyworld: just raised a bug for the remove-relation failure on apps running k8s -> https://bugs.launchpad.net/juju/+bug/1764649 and also created a card on caas trello board plz let me know if any questions. thx
[07:25] <mup> Bug #1764649: juju caas remove-relation does not work <juju:New for wallyworld> <https://launchpad.net/bugs/1764649>
[07:32] <wallyworld> great ty
[07:40] <vino_> wallyworld: sorry i was a bit away for my tea.
[07:41] <vino_> yes i did that.
[07:41] <vino_> and i am there
[07:43] <wallyworld> vino_: great ok, so the service files - you should see a symlink from files /etc/systemd/system to their actual location under /var/lib/juju/init
[07:43] <vino_> :) yes.
[07:45] <vino_> i did ssh to instance but was looking in another terminal didnt realize :p
[07:45] <wallyworld> ah no worries
[08:12] <thumper> wallyworld: still around?
[08:13] <wallyworld> thumper: sorta
[08:13] <wallyworld> soccer soon
[08:13] <thumper> wallyworld: how soon?
[08:13] <thumper> just getting PR ready
[08:13] <wallyworld> 15
[08:14] <wallyworld> maybe 0
[08:14] <wallyworld> 20
[08:15] <thumper> long enough
[08:15] <wallyworld> ok
[08:19] <thumper> poo, need to remove some critical logging...
[08:21] <thumper> wallyworld: https://github.com/juju/juju/pull/8617
[08:21] <thumper> wallyworld: did you want to get on a hangout to talk through?
[08:21] <thumper> smaller than the penultimate branch
[08:22] <wallyworld> thumper: can do
[08:22]  * thumper jumps in HO
[08:53] <thumper> phew...
[08:53]  * thumper waits for the merge
[08:58] <thumper> manadart: ping
[09:15] <manadart> thumper: Pong. On a call with jam.
[09:15] <thumper> manadart: hey, was just wondering if you had an ETA on removing the horrible lxd patches we have
[09:16] <manadart> thumper: Not yet, but I can peel off this intermittent failure I am working on and round up such an estimate.
[09:17] <thumper> heh... no, we're good
[09:17] <thumper> what is the intermittent failures?
[09:20] <manadart> https://bugs.launchpad.net/juju/+bug/1753418
[09:20] <mup> Bug #1753418: intermittent failure in kvmProvisionerSuite.TestKVMProvisionerObservesConfigChanges <intermittent-failure> <test-failure> <juju:In Progress by manadart> <https://launchpad.net/bugs/1753418>
[09:41] <thumper> night all
[09:42] <manadart> G'night.
[13:48] <hml> g’morning o/
[13:49] <rick_h_> morning hml
[20:27] <thumper> morning
[20:30] <hml> morning thumper
[20:38] <veebers> Morning o/
[21:31] <thumper> wallyworld: so... no power nor internet at home now
[21:32] <thumper> my laptop battery has several hours I guess
[21:32] <thumper> and tethered to my phone
[21:32] <thumper> they are replacing a damanged pole outside my house, so have turned off power, but we never got notified
[21:32] <thumper> from 9 till 3
[21:33] <wallyworld> thumper: oh that's what happended
[21:33] <thumper> yeah, I was a little surprised, but you should see the kids
[21:34] <thumper> they are like, I can't charge my phone?
[21:34] <thumper> can't watch netflix?
[21:34] <thumper> what am I going to do?
[21:35] <hml> ha!
[21:36] <wallyworld> lol
[21:36] <wallyworld> thumper: so no release call :-) i thought i might be good to chat about that upgrade issue even thoigh nix is away
[21:38] <thumper> wallyworld: let's do a voice only hangout
[21:39] <thumper> to save my data
[21:39] <wallyworld> ok
[22:07] <veebers> wallyworld: I'm scratching my head on this one, re: resolve command: every build I've tried it works as expected, --no-retry skips the failing install hook and continues.
[22:08] <veebers> I tried 2.3.5 (snap installed), I tried the commit before your one that fixed the issue
[22:08] <babbageclunk> ha, if you're benchmarking something where you're appending to logs, make sure part of the thing isn't copying all of the logs so your bechmark gets progressively slower.
[22:09] <wallyworld> veebers: i don't have an explanation for that off hand, will need to investigate
[22:19] <wallyworld> thumper: we can fill out much of the 2.4 release notes but the enable-ha/remove-machine and ha space sections are todo (john/joe) as are model owner changes (you). i think we may need a day to get these sorted
[22:19] <thumper> yeah
[22:20] <wallyworld> so we maybe should take a view tomorrow after forcing folks to fill out notes today :-)
[22:20] <wallyworld> we can do 2.4 and 2.3.6 a day apart
[22:20] <wallyworld> different people do each
[22:35] <vino> hey Wallyworld: morning! fixed the format issues u have mentioned.
[22:35] <wallyworld> great ty
[22:35] <wallyworld> looking at pr
[22:36] <wallyworld> lgtm
[23:47] <wallyworld> babbageclunk: ready!
[23:48] <wallyworld> still in same hangout
[23:49] <babbageclunk> ok