[09:03] <stickupkid> manadart, should I be able to upgrade a controller whilst doing a upgrade-series of a machine?
[09:04] <stickupkid> seems pretty risky
[09:09] <manadart> stickupkid: Hmm probably not.
[09:16] <stickupkid> manadart, the sticky situation is that it makes it harder to iterate on the feature... i.e. fix something on the controller side and run upgrade-controller
[09:16] <stickupkid> manadart, maybe force can tell it, it's fine and yolo it
[09:17] <manadart> stickupkid: Yeah, OK.
[09:39] <icey> hey, anybody seen something like https://pastebin.ubuntu.com/p/HTSFzH68mJ/ recently?
[09:39] <icey> I can't deploy things with spaces on maas suddenly :)
[09:47] <manadart> icey: Juju version?
[09:48] <icey> manadart: as noted in the pastebin: version: 2.8.2
[09:48] <icey> manadart: client is 2.8.1-groovy-amd64
[09:49] <manadart> icey: OTP at the mo', but definitely want to look at this with you. 10 mins?
[09:49] <icey> manadart: 10 minutes if brief, otherwise I'd rather lunch first :)
[09:50] <icey> manadart: it's managed to hang around since mid last week, doubt it'll stop happening this moment :)
[09:50] <manadart> icey: OK, go ahead. Ping me when you have a little time.
[09:50] <icey> +1
[09:50] <stickupkid> manadart, CR - https://github.com/juju/juju/pull/11929
[10:03] <stickupkid> manadart, another ho when you've got 5...
[10:09] <manadart> stickupkid: In daily.
[10:20] <stickupkid> manadart, I know why, it's because most of upgrade stuff is on client/client facade, which we shouldn't touch
[10:20] <stickupkid> manadart, I'll make a new facade
[10:21] <stickupkid> manadart, thinking about using modelmanager actually... ValidateModelUpgrade
[10:33] <icey> manadart: ok - I'm now well fed :-D
[10:35] <manadart> icey: Righto, this will be faster: https://meet.google.com/pxd-zjad-bgh
[14:46] <manadart> icey: Ping.
[14:46] <icey> manadart: pong
[14:47] <manadart> icey: I did some playing around here, and it seems that if you delete the documents from `toolsmetadata`, juju will re-fetch the agent binary from streams.
[14:47] <icey> oooh nifty
[14:48] <manadart> icey: The only downside is the the old one will still be in the blobstore as an orphan.
[14:48] <manadart> If you cared, you could locate and delete it to save a few MBs.
[14:48] <icey> manadart: so... db.delete('toolsmetadata') or something?
[14:49] <icey> manadart: hahaha yeah, no
[14:49] <manadart> `db.toolsmetadata.deleteMany({})`
[14:51] <icey> { "acknowledged" : true, "deletedCount" : 4 }
[14:51] <icey> :-D
[14:52] <manadart> icey: Ah bugger. It still won't upgrade the controller. One sec.
[14:52] <icey> manadart: I see that
=== disposable3 is now known as disposable2
=== davdunc_ is now known as davdunc
[15:11] <manadart> icey: Looks like the only thing for it would be to jump on the controllers and curl/wget it from `https://streams.canonical.com/juju/tools/agent/2.8.2/juju-2.8.2-ubuntu-amd64.tgz`
[15:11] <icey> manadart: that's fine :)
[15:11] <icey> where should I put that after untarring it?
[15:13] <manadart> /var/lib/juju/tools/2.8.2-{series}-amd64 should have the jujud/jujuc currently running.
[15:15] <manadart> That *should* do it - the controllers have the correct binary, and new machines will cause the controller to get the new (correct) one when not found it toolsmetadata.
[15:15] <manadart> *in toolsmetadata.
[15:16] <manadart> I have to EoD. Let us know how you get on.
[15:16] <icey> manadart: pushing the binaries around now, hopefully done and working shortly :)
[15:17] <icey> geez manadart - is there an easy way to replace these binaries? it's annoyed that the binary is running :-P
[15:39] <icey> manadart: after a slightly big hammer to upgrade the controllers: "0/lxd/0   pending                           pending        focal          starting"
[15:42] <stickupkid> hml, going to review your PR now, been fighting with error messages
[15:51] <icey> drat manadart - 0/lxd/0   down                                             pending               focal          host machine "0" has no available device in space(s) "ceph-access-space", "ceph-replica-space", "public-space"
[15:51] <icey> I'll look again in the AM
[16:05] <hml> stickupkid: added comments to: https://github.com/juju/juju/pull/11928
[16:26] <stickupkid> hml, fixed
[16:26] <hml> stickupkid:  ack
[16:28] <hml> stickupkid: approved
[16:34] <stickupkid> hml, ta