[06:47] I have a machine in my MAAS that Juju cannot deploy to; when it requests a machine, I can see in the maas.log that a machine was requested but Juju reports "failed to acquire node: unexpected: ServerError: 404 Not Found (Not Found)" ; I'm not sure where to look next [08:34] Hi, i got the new 2.5.0~alpha1-7104-g18ae9e407-0ubuntu1~18.04.1 when i updated today, but I don't understand which commit it's based on? the tag 2.5.0-alpha1 is from a65328f744d16e840074633a980239d250d2c90a on 2018-05-24, which cant be the one, can it? [08:36] bjarne_l: the commit ID is in the version: 18ae9e407 [08:37] bjarne_l: If I recall my git-foo correctly, the version format is -- [19:03] icey: that's a strange error, seems like it didn't find the machine. Maybe incorrect constraints? [19:04] bradm: i guess i didn't tag it correctly :) [19:04] err [19:04] bjarne_l: ^^ [19:04] bjarne_l: i guess i didn't tag it correctly [19:11] roaksoax I'm trying to use a fairly simple bundle: https://pastebin.ubuntu.com/p/QJ325FMcWn/ [19:11] if I remove spaces (do something like deploy the ubuntu charm), it works fine [19:11] but the moment spaces are in, it gets weird [19:11] I tried declaring the spaces in the machine constraints as well but that didn't work [19:11] (may have done it wrong) [19:12] icey: because spaces are using to constraint the machine [19:12] icey: it probably means your MAAS doesn't have spaces [19:12] it does [19:12] I made them myself [19:13] https://screenshots.firefox.com/wDRMmBk7pix7JNR1/10.0.4.2 roaksoax [19:13] icey: and the vlan where the machine is, in the right space ? [19:14] icey: or, another way to look at it, is the machine in any of those two subnets ? [19:14] roaksoax there's 3 machines, they each have 2 nics on one switch (internal), and one on another (public) [19:14] the machines are all configured like: (screenshot incoming) [19:15] https://screenshots.firefox.com/FMpdyukQWiAMvtCX/10.0.4.2 roaksoax [19:16] icey: ok. So do this. Grab the constraints that juju is using when requesting a machine from maas [19:16] (I'm hoping that you'll just say, "Oh! That needs to be configured like this!" and it'll work) [19:16] [('agent_name', ['eb790a32-2962-4c3f-8825-159b612fd63b']), ('interfaces', ['magpie:space=-1']), ('tags', ['nvme']), ('zone', ['default'])] [19:16] (already have the maas logs open :-P) [19:17] icey: space=-1 [19:17] icey: that seems to be the issue [19:18] I agree; something is amis :-P [19:18] for reference, I'm using maas from the stable PPA on Bionic (2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1)) [19:18] icey: the log with constraints, is that from juju or maas ? [19:19] that log line is from maas [19:19] it is what juju sent to maas for it [19:19] icey: confirm that's what juju requests [19:19] icey: because if that's the case, then there's either a bug in juju, in the bundle or similar [19:19] well, I have a clean controller up on this maas [19:20] how can I confirm what Juju is sending, without just looking at the maas logs? [19:20] icey: because juding by that log, what I read is that juju is telling maas interface=magpie:space=-1 instead of interface=magpie:space=internal [19:20] (looking with log levels turned up, I haven't noticed raw requests) [19:21] icey: i dont have an environment setup atm, bu juju debug-log should show you the request it makes to maas and which constraints it uses [19:26] machine-0: 19:25:15 TRACE maas request f9: POST http://10.0.4.2:5240/MAAS/api/2.0/machines/?op=allocate, params=agent_name=63e56b3b-db02-4ad2-8115-d76916426f8e&interfaces=magpie%3Aspace%3D-1&tags=nvme&zone=defau [19:26] lt [19:26] machine-0: 19:25:15 TRACE maas response f9: error: "ServerError: 404 Not Found (Not Found)" [19:26] icey: -1 [19:26] icey: that seems juju to me [19:26] Juju is sending the interfaces=magpie:-1 but [19:27] then, it seems like there are two options: I'm failing to translate my maas spaces for juju correctly, or juju is confusing my spaces [19:28] do I need to run `add-space` for juju to know about the spaces?! [19:28] mhm nope: https://pastebin.ubuntu.com/p/sbsYYnVPCH/ [19:29] Juju knows about the spaces, and has the correct subnets [19:29] icey: i actually would have thought juju would automatically get those [19:29] roaksoax it did [19:29] icey: uhmmm [19:29] icey: i would file a bug with juju folks [19:29] roaksoax yea [19:29] i dont know the juju codebase unfortunately to know [19:29] they're going to love me this week; there's an issue with the lxd container networking on maas as well :-D [19:30] where it doesn't make the bridge: https://pastebin.ubuntu.com/p/Gf3qkQ6SzS/ [19:30] woot :-P [19:30] icey: i think that may be a cloud-init issue actually [19:31] oh really [19:31] icey: which should be resolved with cloud-init from -proposed [19:33] roaksoax interesting; I'm going to try one more thing to see if it resolves this weirdness; as it'll let me write a more helpful bug if it works (I have a nasty theory that if the units directly on a machine are in the undefined space, juju forgets about spaces when placing containers) [19:33] ugh, probably [19:34] wow really weird [19:34] failed to start machine 2 (failed to acquire node: No available machine matches constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])] (resolved to "interfaces=magpie:space=4 tags=nvme zone=default")), retrying in 10s (9 more attempts) [19:34] the other machines I'm asking for work [19:34] well [19:34] probably because I don't have 4 machines in those spaces :-P [19:34] super weird [19:36] icey: so are you selectin space=4 ? [19:36] it seems to me that space= [19:36] yes [19:36] not selecting space=4 [19:36] but space=internal/storage [19:36] icey: so juju is sending an ID instead of a space name ? [19:36] I'm trying to get stuff deploying with this split network (10G internal, 1G public) [19:37] well, it sends the space ID: Aug 3 19:35:20 bifrost maas.api: [info] Request from user chris to acquire a machine with constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])] [19:39] it looks like it's aliasing the (undefined) space as -1 [19:39] icey: yeah, that's the 'id' of the space in MAAS [19:40] roaksoax except apparently -1 isn't correct, as maas gives a 404 on it :-/ [19:40] HA, juju doesn't think that the undefined space is -1 [19:40] https://pastebin.ubuntu.com/p/R8rgDqX7GD/ [19:40] icey: [19:40] ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=1 dry_run=True | pastebinit -f diff [19:41] http://paste.ubuntu.com/p/CtTJzbrDrx/ [19:41] icey: it works for me, e.g. i do get a machine in that space [19:41] right, space=1 [19:41] juju is sending space=-1 [19:41] there _is_ no space=-1 [19:41] my space IDs are (apparently) 5, 8, and 11 [19:42] ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=-1 dry_run=True [19:42] Not Found [19:42] icey: yeah [19:43] well, maas acts reasonably with it :-P [19:43] icey: but seems juju should be requesting a machine with a space [19:43] well, the last machine has no space requirements [19:43] I'm actually using a named space binding, maybe I should make it the default binding ("": undefined) [19:44] icey: but are you specifically requesting a machine "without space" ? [19:44] roaksoax no, I'm specifying that it should be in the undefined space [19:44] I suppose I could try removing that binding [19:44] icey: ok, hold on, 'the undefined space' doesn't exist :). It means there's no space defined [19:44] yeah yeah [19:45] icey: but that said, juju translates that into '-1' [19:45] https://screenshots.firefox.com/bRt3jP2rdzwNdRON/10.0.4.2 [19:45] undefined space :-P [19:45] you think it would be better to actually give that an official space? [19:45] frankly, I'd rather that be the default and everything else require specification [19:47] icey: so this is what I see [19:47] ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=undefined dry_run=True | pastebinit -f diff [19:47] http://paste.ubuntu.com/p/xwqt9g6Q48/ [19:47] icey: so if you specify space=undefined, it does give you a machine [19:47] icey: where as if you specify space=-1, it doesn't [19:47] yeah; unfortunately juju is sending -1 [19:49] I think giving _every_ network a space is going to get it working, but it's annoying that it has to be explicit :-/ [19:50] icey: [19:50] ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=2 dry_run=True | pastebinit [19:50] http://paste.ubuntu.com/p/QpDMBxNqkz/ [19:50] ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=testing dry_run=True | pastebinit [19:50] http://paste.ubuntu.com/p/hWD5KY2dYP/ [19:50] icey: so i changed from undefined to another space called testing [19:50] icey: and sending 'ids' it now works [19:50] so the bug is in maas [19:50] where if you send -1, it won't work [19:51] well, isn't that fun :-P [19:51] I suppose I'll target MAAS with my bug as well :-P [19:51] *but* i need to understand whether it is intended to allow request space=undefined [19:51] enjoy roaksoax https://bugs.launchpad.net/maas/+bug/1785314 [19:51] icey: because if may be that we want not_space [19:52] roaksoax we may decide that any space use means all spaces must be explicit, but if we do, we should document that :-P [19:52] I'm off to bed :-P [19:53] have a good weekend! [19:53] icey: y too [19:53] u too