[06:47] <icey> I have a machine in my MAAS that Juju cannot deploy to; when it requests a machine, I can see in the maas.log that a machine was requested but Juju reports "failed to acquire node: unexpected: ServerError: 404 Not Found (Not Found)" ; I'm not sure where to look next
[08:34] <bjarne_l> Hi, i got the new 2.5.0~alpha1-7104-g18ae9e407-0ubuntu1~18.04.1 when i updated today, but I don't understand which commit it's based on? the tag 2.5.0-alpha1 is from a65328f744d16e840074633a980239d250d2c90a on 2018-05-24, which cant be the one, can it?
[08:36] <TJ-> bjarne_l: the commit ID is in the version: 18ae9e407
[08:37] <TJ-> bjarne_l: If I recall my git-foo correctly, the version format is <tag>-<qty-commits-since-then>-<commit-id>
[19:03] <roaksoax> icey: that's a strange error, seems like it didn't find the machine. Maybe incorrect constraints?
[19:04] <roaksoax> bradm: i guess i didn't tag it correctly :)
[19:04] <roaksoax> err
[19:04] <roaksoax> bjarne_l: ^^
[19:04] <roaksoax> bjarne_l: i guess i didn't tag it correctly
[19:11] <icey> roaksoax I'm trying to use a fairly simple bundle: https://pastebin.ubuntu.com/p/QJ325FMcWn/
[19:11] <icey> if I remove spaces (do something like deploy the ubuntu charm), it works fine
[19:11] <icey> but the moment spaces are in, it gets weird
[19:11] <icey> I tried declaring the spaces in the machine constraints as well but that didn't work
[19:11] <icey> (may have done it wrong)
[19:12] <roaksoax> icey: because spaces are using to constraint the machine
[19:12] <roaksoax> icey: it probably means your MAAS doesn't have spaces
[19:12] <icey> it does
[19:12] <icey> I made them myself
[19:13] <icey> https://screenshots.firefox.com/wDRMmBk7pix7JNR1/10.0.4.2 roaksoax
[19:13] <roaksoax> icey: and the vlan where the machine is, in the right space ?
[19:14] <roaksoax> icey: or, another way to look at it, is the machine in any of those two subnets ?
[19:14] <icey> roaksoax there's 3 machines, they each have 2 nics on one switch (internal), and one on another (public)
[19:14] <icey> the machines are all configured like: (screenshot incoming)
[19:15] <icey> https://screenshots.firefox.com/FMpdyukQWiAMvtCX/10.0.4.2 roaksoax
[19:16] <roaksoax> icey: ok. So do this. Grab the constraints that juju is using when requesting a machine from maas
[19:16] <icey> (I'm hoping that you'll just say, "Oh! That needs to be configured like this!" and it'll work)
[19:16] <icey> [('agent_name', ['eb790a32-2962-4c3f-8825-159b612fd63b']), ('interfaces', ['magpie:space=-1']), ('tags', ['nvme']), ('zone', ['default'])]
[19:16] <icey> (already have the maas logs open :-P)
[19:17] <roaksoax> icey: space=-1
[19:17] <roaksoax> icey: that seems to be the issue
[19:18] <icey> I agree; something is amis :-P
[19:18] <icey> for reference, I'm using maas from the stable PPA on Bionic (2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1))
[19:18] <roaksoax> icey: the log with constraints, is that from juju or maas ?
[19:19] <icey> that log line is from maas
[19:19] <icey> it is what juju sent to maas for it
[19:19] <roaksoax> icey: confirm that's what juju requests
[19:19] <roaksoax> icey: because if that's the case, then there's either a bug in juju, in the bundle or similar
[19:19] <icey> well, I have a clean controller up on this maas
[19:20] <icey> how can I confirm what Juju is sending, without just looking at the maas logs?
[19:20] <roaksoax> icey: because juding by that log, what I read is that juju is telling maas interface=magpie:space=-1 instead of interface=magpie:space=internal
[19:20] <icey> (looking with log levels turned up, I haven't noticed raw requests)
[19:21] <roaksoax> icey: i dont have an environment setup atm, bu juju debug-log should show you the request it makes to maas and which constraints it uses
[19:26] <icey> machine-0: 19:25:15 TRACE maas request f9: POST http://10.0.4.2:5240/MAAS/api/2.0/machines/?op=allocate, params=agent_name=63e56b3b-db02-4ad2-8115-d76916426f8e&interfaces=magpie%3Aspace%3D-1&tags=nvme&zone=defau
[19:26] <icey> lt
[19:26] <icey> machine-0: 19:25:15 TRACE maas response f9: error: "ServerError: 404 Not Found (Not Found)"
[19:26] <roaksoax> icey: -1
[19:26] <roaksoax> icey: that seems juju to me
[19:26] <icey> Juju is sending the interfaces=magpie:-1 but
[19:27] <icey> then, it seems like there are two options: I'm failing to translate my maas spaces for juju correctly, or juju is confusing my spaces
[19:28] <icey> do I need to run `add-space` for juju to know about the spaces?!
[19:28] <icey> mhm nope: https://pastebin.ubuntu.com/p/sbsYYnVPCH/
[19:29] <icey> Juju knows about the spaces, and has the correct subnets
[19:29] <roaksoax> icey: i actually would have thought juju would automatically get those
[19:29] <icey> roaksoax it did
[19:29] <roaksoax> icey: uhmmm
[19:29] <roaksoax> icey: i would file a bug with juju folks
[19:29] <icey> roaksoax yea
[19:29] <roaksoax> i dont know the juju codebase unfortunately to know
[19:29] <icey> they're going to love me this week; there's an issue with the lxd container networking on maas as well :-D
[19:30] <icey> where it doesn't make the bridge: https://pastebin.ubuntu.com/p/Gf3qkQ6SzS/
[19:30] <icey> woot :-P
[19:30] <roaksoax> icey: i think that may be a cloud-init issue actually
[19:31] <icey> oh really
[19:31] <roaksoax> icey: which should be resolved with cloud-init from -proposed
[19:33] <icey> roaksoax interesting; I'm going to try one more thing to see if it resolves this weirdness; as it'll let me write a more helpful bug if it works (I have a nasty theory that if the units directly on a machine are in the undefined space, juju forgets about spaces when placing containers)
[19:33] <roaksoax> ugh, probably
[19:34] <icey> wow really weird
[19:34] <icey>  failed to start machine 2 (failed to acquire node: No available machine matches constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])] (resolved to "interfaces=magpie:space=4 tags=nvme zone=default")), retrying in 10s (9 more attempts)
[19:34] <icey> the other machines I'm asking for work
[19:34] <icey> well
[19:34] <icey> probably because I don't have 4 machines in those spaces :-P
[19:34] <icey> super weird
[19:36] <roaksoax> icey: so are you selectin space=4 ?
[19:36] <roaksoax> it seems to me that space=<the name of space>
[19:36] <icey> yes
[19:36] <icey> not selecting space=4
[19:36] <icey> but space=internal/storage
[19:36] <roaksoax> icey: so juju is sending an ID instead of a space name ?
[19:36] <icey> I'm trying to get stuff deploying with this split network (10G internal, 1G public)
[19:37] <icey> well, it sends the space ID: Aug  3 19:35:20 bifrost maas.api: [info] Request from user chris to acquire a machine with constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])]
[19:39] <icey> it looks like it's aliasing the (undefined) space as -1
[19:39] <roaksoax> icey: yeah, that's the 'id' of the space in MAAS
[19:40] <icey> roaksoax except apparently -1 isn't correct, as maas gives a 404 on it :-/
[19:40] <icey> HA, juju doesn't think that the undefined space is -1
[19:40] <icey> https://pastebin.ubuntu.com/p/R8rgDqX7GD/
[19:40] <roaksoax> icey:
[19:40] <roaksoax> ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=1 dry_run=True | pastebinit -f diff
[19:41] <roaksoax> http://paste.ubuntu.com/p/CtTJzbrDrx/
[19:41] <roaksoax> icey: it works for me, e.g. i do get a machine in that space
[19:41] <icey> right, space=1
[19:41] <icey> juju is sending space=-1
[19:41] <icey> there _is_ no space=-1
[19:41] <icey> my space IDs are (apparently) 5, 8, and 11
[19:42] <roaksoax> ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=-1 dry_run=True
[19:42] <roaksoax> Not Found
[19:42] <roaksoax> icey: yeah
[19:43] <icey> well, maas acts reasonably with it :-P
[19:43] <roaksoax> icey: but seems juju should be requesting a machine with a space
[19:43] <icey> well, the last machine has no space requirements
[19:43] <icey> I'm actually using a named space binding, maybe I should make it the default binding ("": undefined)
[19:44] <roaksoax> icey: but are you specifically requesting a machine "without space" ?
[19:44] <icey> roaksoax no, I'm specifying that it should be in the undefined space
[19:44] <icey> I suppose I could try removing that binding
[19:44] <roaksoax> icey: ok, hold on, 'the undefined space' doesn't exist :). It means there's no space defined
[19:44] <icey> yeah yeah
[19:45] <roaksoax> icey: but that said, juju translates that into '-1'
[19:45] <icey> https://screenshots.firefox.com/bRt3jP2rdzwNdRON/10.0.4.2
[19:45] <icey> undefined space :-P
[19:45] <icey> you think it would be better to actually give that an official space?
[19:45] <icey> frankly, I'd rather that be the default and everything else require specification
[19:47] <roaksoax> icey: so this is what I see
[19:47] <roaksoax> ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=undefined dry_run=True | pastebinit -f diff
[19:47] <roaksoax> http://paste.ubuntu.com/p/xwqt9g6Q48/
[19:47] <roaksoax> icey: so if you specify space=undefined, it does give you a machine
[19:47] <roaksoax> icey: where as if you specify space=-1, it doesn't
[19:47] <icey> yeah; unfortunately juju is sending -1
[19:49] <icey> I think giving _every_ network a space is going to get it working, but it's annoying that it has to be explicit :-/
[19:50] <roaksoax> icey:
[19:50] <roaksoax> ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=2 dry_run=True | pastebinit
[19:50] <roaksoax> http://paste.ubuntu.com/p/QpDMBxNqkz/
[19:50] <roaksoax> ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=testing dry_run=True | pastebinit
[19:50] <roaksoax> http://paste.ubuntu.com/p/hWD5KY2dYP/
[19:50] <roaksoax> icey: so i changed from undefined to another space called testing
[19:50] <roaksoax> icey: and sending 'ids' it now works
[19:50] <roaksoax> so the bug is in maas
[19:50] <roaksoax> where if you send -1, it won't work
[19:51] <icey> well, isn't that fun :-P
[19:51] <icey> I suppose I'll target MAAS with my bug as well :-P
[19:51] <roaksoax> *but* i need to understand  whether it is intended to allow request space=undefined
[19:51] <icey> enjoy roaksoax https://bugs.launchpad.net/maas/+bug/1785314
[19:51] <roaksoax> icey: because if may be that we want not_space
[19:52] <icey> roaksoax we may decide that any space use means all spaces must be explicit, but if we do, we should document that :-P
[19:52] <icey> I'm off to bed :-P
[19:53] <icey> have a good weekend!
[19:53] <roaksoax> icey: y too
[19:53] <roaksoax> u too