[01:54] <thumper> https://github.com/juju/juju/pull/11874 for the stopping and starting of unit workers
[01:55] <thumper> wallyworld, kelvinliu, hpidcock, tlm: ^^^
[01:56] <hpidcock> thumper: looking
[02:04] <hpidcock> wallyworld: might need to look at it too
[02:09] <tlm> looking
[02:12] <wallyworld> looking too
[02:17] <kelvinliu> looking 2
[02:18] <thumper> I feel blessed
[02:33] <wallyworld> thumper: i had a question/concern about the behaviour of the start/stop handler not waiting for a response
[02:52] <thumper> ok, will look later, brain is in the next thing just now
[13:28] <achilleasa> can someone take a look at https://github.com/juju/juju/pull/11877?
[16:11] <achilleasa> hml: can you take a look when you have some time? ^^^
[17:03] <mcayland> hi everyone - i've been trying to upgrade to a newer version of juju to run the latest octavia plugin
[17:04] <mcayland> i've done the model upgrade but it seems like the juju agents themselves don't get upgraded
[17:04] <mcayland> i can see lots of messages like "no matching agent binaries available" in the log files
[17:04] <mcayland> can anyone provide any pointers to help fix this?
[17:08] <hml> achilleasa:  sure
[17:18] <petevg> mcayland: Did you upgrade the controller model first, then upgrade the models housing the applications? And have you taken a look at https://juju.is/docs/help-model-upgrades? Let me know if you have any specific questions, but that page might help you get started troubleshooting.
[17:21] <mcayland> petevg: yes indeed, i followed that page quite closely
[17:22] <mcayland> petevg: the other thing i had to do was remove the old juju apt packages and switch to a snap version instead
[17:24] <mcayland> i did a 2 stage upgrade: first upgrade to the latest deb packages which got me to juju 2.6
[17:24] <mcayland> this looks like it is reflected correctly in the agent.conf files
[17:26] <mcayland> then i installed a new snap juju 2.7.8 package - "juju status" looks fine
[17:27] <mcayland> "juju upgrade-model" gives me "ERROR some agents have not upgraded to the current model version 2.7.8"
[17:27] <mcayland> and lists most of the agents
[17:34] <mcayland> petevg: i managed to run "juju upgrade-model --agent-stream 2.7.8 --debug"
[17:35] <mcayland> but 2.7.8 isn't listed in the "available agent binaries" list. 2.7.7 is there, so is 2.8.1. i guess that's why i'm failing...
[17:37] <mcayland> oooh there is an "--ignore-agent-versions" switch. let me try that...
[17:38] <mcayland> "started upgrade to 2.8.2" i think that's promising
[17:50] <petevg> mcayland: Fingers cross, but I think that you're on the right track. There aren't deb based binaries for some of the latest juju releases, which is probably the issue you were running into. Getting everything onto the snap should fix it ...
[17:50] <petevg> hml: do you have any other advice for mcayland? (Starting w/ a deb based controller, upgrading to a snap based one.)
[17:51] <hml> petevg: looking
[17:55] <mcayland> petevg: i can see that some of the agents have now upgraded, lots of activity
[17:56] <mcayland> petevg/hml: things have started to settle down now. looks like 3 services have errors...
[17:57] <hml> mcayland:  the controller is running 2.7.8 yes? and now model has upgraded to 2.7.8?
[17:58] <mcayland> hml: there were no 2.7.8 binaries present and --ignore-agent-versions seems to have updated all the agents to 2.8.2
[17:58] <hml> mcayland: what’s the controller running?
[18:00] <mcayland> hml: looks like controller is also at 2.8.2
[18:01] <hml> mcayland: what version of the snap?
[18:01] <hml> mcayland: what snap channel for the juju snap, to be more specific
[18:02] <mcayland> hml: i went for the basic "sudo snap install juju --classic"
[18:04] <hml> mcayland: so you should have been upgrading to 2.8.1, afaik with that snap.  might be why you couldn’t get to 2.7.8.
[18:04] <hml> mcayland: juju version shows 2.8.1?
[18:05] <mcayland> hml: yes it does
[18:05] <mcayland> i'm just looking to see why designate failed
[18:06] <mcayland> "TypeError: sequence item 0: expected str instance, NoneType found"
[18:06] <hml> mcayland: looks like using the —ignore-agent-versions upgraded you to a release that is in process right now, but not yet official 2.8.2
[18:06] <mcayland> return ';'.join(self.relation.client_ips())
[18:06] <hml> 2.8.2 is the latest/candidate channel right now
[18:07] <mcayland> hml: hmm okay - i hope there's nothing too unstable there(!)
[18:07] <hml> mcayland: its in final test right now, if that’s good, will be released
[18:08] <mcayland> hml: great, thanks! any thoughts on the designate-bind error above? it looks like it's missing some config?
[18:08] <hml> mcayland: can you give me more context around the TypeError, looks like a python error, so would be related to the charm, not juju itself
[18:09] <mcayland> hml: that's the tail end of juju unit log
[18:09] <mcayland> self.relation.client_ips() makes me think it's trying to update some config
[18:10] <hml> mcayland:  that’s from the charm itself, and data it gets for relations to other charms
[18:11] <mcayland> hml: i can pastebin the whole backtrace if it helps?
[18:12] <hml> mcayland: trying to find someone who knows more about the designate charm.  i’m on juju itself.  :-)
[18:15] <mcayland> hml: https://pastebin.ubuntu.com/p/54BKYY8hrs/
[18:17] <hml> mcayland: looks like the issue is at ln 30\
[18:20] <beisner> hi mcayland, hml - I may be able to help with the designate charm issue.
[18:20] <hml> beisner:  the pastebin is just above
[18:20] <beisner> First, I need to determine if the charm itself is at the latest stable release.
[18:20] <hml> beisner:  ty
[18:21] <beisner> mcayland ^
[18:23] <mcayland> beisner: thanks! juju status tells me it is jujucharms 29 ubuntu
[18:29] <beisner> ok thx, mcayland.  what all has been tried here?  ie. `juju resolved <unit-in-error>` etc?  Are there any other units in an error state?
[18:30] <mcayland> beisner: it's basically a cluster i've upgraded via 2.6 to 2.8.2
[18:30] <mcayland> i've not used "juju resolved" at all
[18:31] <mcayland> after the upgrade there are 3 units not running: nova-cloud-controller, designate-bind and octavia-dashboard
[18:31] <beisner> mcayland: mind sharing a `juju status` output?  (take care to review/sanitize for private info)
[18:32] <beisner> https://pastebin.ubuntu.com/
[18:35] <mcayland> beisner: https://pastebin.ubuntu.com/p/qyZRXVqx8n/
[18:41] <beisner> thx mcayland, looking
[18:44] <mcayland> beisner: also thanks to everyone here :)  just trying to figure out why nova-cloud-controller is unhappy...
[18:45] <pmatulis> vault will need unsealing for sure
[18:47] <pmatulis> not sure why it's sealed though. was this a working cloud?
[18:48] <mcayland> pmatulis: the vault was sealed before i tried to do the upgrade, it's a private cloud that has been moved to a new home
[18:55] <beisner> mcayland: can you tell me more about that move?  ie. Any network, infra, storage changes with that?
[18:56] <mcayland> beisner: new storage and compute resources was added, but that was after the move and before i upgraded juju
[18:57] <beisner> mcayland: I see, nova-compute/5 and 7 most likely.  Was everything functioning normally after the move, and after the scale-out of compute?
[18:58] <beisner> ps. still looking at logs and at a local deployment.
[18:59] <mcayland> beisner: yeah afaict it was all fine - the juju upgraded i needed because i was getting a lbaas config error which i traced to needing a later octavia charm
[19:00] <mcayland> fwiw that error has now gone, but i should probably figure out why nova-cloud-controller isn't running though :/
[19:00] <mcayland> i've also tried to unseal the vault and it fails. something tells me another admin has done some upgrades since i was last looking after it...
[19:05] <beisner> mcayland: ok so designate-bind is active/ok?
[19:07] <pmatulis> there is both a mysql unit and a percona-cluster unit, that's odd. can we have the relations as well? bottom part of 'juju status --relations'
[19:09] <beisner> hmm, yeah, good catch pmatulis.  they're both the percona-cluster charm, at differing revs.
[19:10] <mcayland> beisner: no, that's broken too - see the pastebin above at https://pastebin.ubuntu.com/p/54BKYY8hrs/
[19:11] <beisner> mcayland: oh... you were talking about the lbaas error being gone.  got it.
[19:12] <mcayland> pmatulis: https://pastebin.ubuntu.com/p/SFMKk9WSnX/
[19:12] <mcayland> beisner: yeah :)  well i guess that at least made the upgrade worthwhile... ;)
[19:13] <beisner> ok, so one db for vault; one db for openstack.  that's fine.
[19:13] <beisner> db cluster, that is.
[19:16] <mcayland> beisner: and here's the output of the nova-cloud-controller log: https://pastebin.ubuntu.com/p/q2VSPtdRrQ/
[19:17] <mcayland> Generating template context for amqp  - followed by - update-status ERROR no relation id specified
[19:32] <beisner> mcayland: so re: designate-bind, the only thing I can suggest is to try `juju resolved designate-bind/0` which will retry the last operation.  I'd be curious if that brings it to an active/ready state or not.
[19:32] <mcayland> beisner: okay let me see...
[19:33] <beisner> mcayland: In this topology, it doesn't look like vault can play a role in the success of your nova-c-c, but barbican may not be happy until you work out the vault bits.
[19:34] <beisner> mcayland: on nova-c-c, you could either do a somewhat blind attempt at `resolved` there too;  or go into the unit and attempt to start each of the services; or start with inspecting each of the payload logs on the system unit to diagnose their startup issues.
[19:35] <mcayland> beisner: i don't think the juju resolved for designate-bind has helped :/
[19:38] <mcayland> "Failed to start apache2.service: Unit apache2.service is masked."
[19:40] <beisner> mcayland: what is the current unit error state on designate-bind after the `resolved` run?
[19:41] <mcayland> beisner: it still says error with "hook failed: "install"
[19:43] <mcayland> how do i get the default relations?
[19:43] <mcayland> there are no manual "juju add-relation" commands in https://jaas.ai/nova-cloud-controller
[19:44] <mcayland> so presumably it has some defaults?
[19:46] <beisner> mcayland: the "reference" minimal example is @ https://jaas.ai/openstack-base/bundle/ (see bundle.yaml).  bear in mind that is all the latest on Ubuntu 20.04 with OpenStack Ussuri.   BTW, what versions are you running?  I see it's Bionic, but I'm not sure which version of OpenStack.
[19:47] <mcayland> beisner: it's bionic-stein
[19:48] <beisner> ack
[19:48] <beisner> mcayland: anyway, if you're wondering about relations, my suggestion is to do a stare & compare against that minimal bundle.
[19:49] <beisner> mcayland: Beyond that, again, good old-fashioned system service diags on the host is next up in order to see what's not going right on the n-c-c unit.
[19:50] <pmatulis> one big difference will be that the new bundle will contain mysql-router and mysql-innodb-cluster, instead of just 'mysql'. the placement charm will also be new
[19:52] <mcayland> beisner: what about the "ERROR no relation id specified" message under "Generating template context for amqp"?
[19:52] <mcayland> i can see the amqp relation there...
[19:53] <pmatulis> i guess this is the bundle to compare with:
[19:53] <pmatulis> https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-stein/bundle.yaml
[19:57] <mcayland> pmatulis: thanks - i think this is going to have to wait until tomorrow now, it's getting late...
[19:58] <mcayland> beisner, hml, pmatulis: thanks for all the suggestions. i guess this is going to take some time to figure out :(
[20:07] <beisner> you're welcome, mcayland - feel free to drop into freenode channel #openstack-charms as well.