wallyworldhpidcock: a small PR for the security review, no rush https://github.com/juju/juju/pull/1198100:22
hpidcockwallyworld: ok looking now00:23
hpidcockno rush == rush00:23
wallyworldfake news00:23
hpidcockwallyworld: added a comment, but overall looks ok00:36
wallyworldkelvinliu: not sure if you'll have any time, here's an azure pr for initial spaces support https://github.com/juju/juju/pull/1198306:18
wallyworldkelvinliu: fyi, the core logic to use spaces info to get subnets is copied from the openstack provider. i'll get joe to look also next week when he's back. in the meantime we can test on site06:21
kelvinliuwallyworld: lgtm, ty06:34
wallyworldawesome ty06:34
tychicusWhat is the best way to troubleshoot "Incomplete relation: vault"16:17
tychicusThe OSD log shows DEBUG juju-log secrets-storage:78: Deferring OSD preparation as vault not ready16:17
tychicusI've trying removing and readding the relation16:18
tychicusjuju remove-relation ceph-osd:secrets-storage vault:secrets16:18
tychicuseverything else can communicate with vault correctly16:18
tychicusvault status shows: Unit is ready (active: true, mlock: disabled)16:20
thedactychicus: Hi. Can we first try the action refresh-secrets. `juju run-action --wait vault/0 refresh-secrets`16:27
tychicusdone, that triggered "Deferring OSD preparation as vault not ready" on the OSD's16:28
thedacOk, let me check on what the code is looking for. That might give us a hint. One sec16:30
tychicusI believe it is checking vault_kv here https://opendev.org/openstack/charm-ceph-osd/src/branch/master/hooks/ceph_hooks.py#L49416:35
thedacRight, which is checking the relation data for secrets-storage here: https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/vaultlocker.py#L3916:36
thedacSo our next step is to see what is set on the relation and see if anything is missing.16:39
thedac`juju run --unit ceph-osd/0 -- "relation-ids secrets-storage"`16:39
thedac`juju run --unit ceph-osd/0 -- "relation-get -r secrets-storage:<ID from above command> - vault/0"`16:39
thedacNote: You may need to change the unit numbers from above and please sanitize the output to your satisfaction before pasting.16:39
tychicusceph-osd/0_role_id: '"e8e9709d-3686-cd80-a3b7-f3fec2517a8e"'16:41
tychicusceph-osd/0_token: '"s.6oyksjfe9pE55nQEODREBrIx"'16:41
tychicusceph-osd/1_role_id: '"9804cb6a-1cb6-99a5-f7e1-ac19f5fe0273"'16:41
tychicusceph-osd/1_token: '"s.COcndyxMYZ0b8o7XZKeTYQzG"'16:41
tychicusceph-osd/2_role_id: '"8ad4d5c7-2e92-2a2a-1aef-f298187b162b"'16:41
tychicusceph-osd/2_token: '"s.lyIyWDVnkh72WnmhUPA8gB8k"'16:41
tychicusceph-osd/3_role_id: '"9c641dc6-0da9-3d20-efc7-647ed05796d5"'16:41
tychicusceph-osd/3_token: '"s.sOyYRReEL6CgSKXDnWzwr0gn"'16:41
tychicusceph-osd/4_role_id: '"a79dacf5-d21c-882d-8362-ac280bd7e42d"'16:41
tychicusceph-osd/4_token: '"s.aZzNwScAezAeBsXMrcvE6gM3"'16:41
tychicusceph-osd/5_role_id: '"d1596430-0a95-d317-464d-c7708560ca84"'16:41
tychicusceph-osd/5_token: '"s.jCuF9G1niDiBQcxBBPauxOIo"'16:41
thedacOk, one sec16:41
thedacAt first glance we seem to be missing the vault_url setting. Unless that was a paste error.16:43
tychicusthe vault_url is missing16:44
thedacOk, that is our culprit. Let me look at the vault side and see how that could happen.16:44
tychicusif I run the same command for nova-compute/0 it does return the vault_url16:45
vultaireHey, got a question re: juju 2.6.x upgrading (https://discourse.juju.is/t/pre-juju-2-6-5-upgrade-steps-for-large-log-collections/1633)16:52
vultaireThere's a sentence which says "Please stop the controllers when running this script."16:52
vultaireWhat precisely does this mean?  Does this mean all systemd juju-related services except juju-db?  (I presume juju-db would need to be left running since that's where mongo is running from...?)16:53
vultaireor am I wrong thee?16:53
vultaireI don't want to assume and make a mistake; sounds like the consequences for such are less than fun.16:54
thedactychicus:  OK, I am guessing this is our problem: https://github.com/openstack/charm-vault/blob/master/src/reactive/vault_handlers.py#L536 and https://github.com/openstack-charmers/charm-interface-vault-kv/blob/master/provides.py#L41 If vault and ceph-osd do not have a common spaces binding for secrets-storage it never publishes the vault_url.16:54
thedactychicus: My team desperately needs to make that clearer. Sorry to make you jump through hoops. But I am pretty confident that is the problem.16:55
jamvultaire, it would be stop the juju-machine-X services for the controller machines.17:00
tychicusso my current settings are vault: endpoint-bindings: secrets: os-internal-api17:00
tychicusso my current settings are vault: endpoint-bindings: secrets: os-public-api17:00
vultaire@jam: thanks for the clarification, perfect.17:00
tychicussorry first entry was in error17:01
jamvultaire, note that stopping the controllers effectively stops everything else, since they can't talk to the controller, but you don't have to explicitly stop them.17:01
tychicusand ceph-osd: endpoint-bindings: secrets-storage: os-internal-api17:02
jam@vultaire, also note pjdc's comment that you can also just drop the collection entirely which is likely to be faster.17:02
jamJuju will recreate it during controller startup17:02
tychicusupdating vault to vault: endpoint-bindings: secrets: os-internal-api17:02
thedactychicus: That would do it. Those need to match.17:02
tychicusshould resolve the issue17:02
vultaireack - I'll simply stop the jujud-machine units.  Also, the script was updated to include pjdc's suggestion17:02
vultaireso no worries there17:02
jamah, good.17:02
tychicusthedac: after updating the binding, what would need to happen to update the vault_url?17:05
tychicusor can the vault url not be changed once it is set?17:06
thedacIf the bindings are updated, a refresh-secrets action run should fix things. Juju just recently added the feature to update a binding. So depending on the version a re-deploy on the ceph side may be necessary.17:08
thedacNo problem17:10

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!