/srv/irclogs.ubuntu.com/2018/08/03/#maas.txt

iceyI have a machine in my MAAS that Juju cannot deploy to; when it requests a machine, I can see in the maas.log that a machine was requested but Juju reports "failed to acquire node: unexpected: ServerError: 404 Not Found (Not Found)" ; I'm not sure where to look next06:47
bjarne_lHi, i got the new 2.5.0~alpha1-7104-g18ae9e407-0ubuntu1~18.04.1 when i updated today, but I don't understand which commit it's based on? the tag 2.5.0-alpha1 is from a65328f744d16e840074633a980239d250d2c90a on 2018-05-24, which cant be the one, can it?08:34
TJ-bjarne_l: the commit ID is in the version: 18ae9e40708:36
TJ-bjarne_l: If I recall my git-foo correctly, the version format is <tag>-<qty-commits-since-then>-<commit-id>08:37
roaksoaxicey: that's a strange error, seems like it didn't find the machine. Maybe incorrect constraints?19:03
roaksoaxbradm: i guess i didn't tag it correctly :)19:04
roaksoaxerr19:04
roaksoaxbjarne_l: ^^19:04
roaksoaxbjarne_l: i guess i didn't tag it correctly19:04
iceyroaksoax I'm trying to use a fairly simple bundle: https://pastebin.ubuntu.com/p/QJ325FMcWn/19:11
iceyif I remove spaces (do something like deploy the ubuntu charm), it works fine19:11
iceybut the moment spaces are in, it gets weird19:11
iceyI tried declaring the spaces in the machine constraints as well but that didn't work19:11
icey(may have done it wrong)19:11
roaksoaxicey: because spaces are using to constraint the machine19:12
roaksoaxicey: it probably means your MAAS doesn't have spaces19:12
iceyit does19:12
iceyI made them myself19:12
iceyhttps://screenshots.firefox.com/wDRMmBk7pix7JNR1/10.0.4.2 roaksoax19:13
roaksoaxicey: and the vlan where the machine is, in the right space ?19:13
roaksoaxicey: or, another way to look at it, is the machine in any of those two subnets ?19:14
iceyroaksoax there's 3 machines, they each have 2 nics on one switch (internal), and one on another (public)19:14
iceythe machines are all configured like: (screenshot incoming)19:14
iceyhttps://screenshots.firefox.com/FMpdyukQWiAMvtCX/10.0.4.2 roaksoax19:15
roaksoaxicey: ok. So do this. Grab the constraints that juju is using when requesting a machine from maas19:16
icey(I'm hoping that you'll just say, "Oh! That needs to be configured like this!" and it'll work)19:16
icey[('agent_name', ['eb790a32-2962-4c3f-8825-159b612fd63b']), ('interfaces', ['magpie:space=-1']), ('tags', ['nvme']), ('zone', ['default'])]19:16
icey(already have the maas logs open :-P)19:16
roaksoaxicey: space=-119:17
roaksoaxicey: that seems to be the issue19:17
iceyI agree; something is amis :-P19:18
iceyfor reference, I'm using maas from the stable PPA on Bionic (2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1))19:18
roaksoaxicey: the log with constraints, is that from juju or maas ?19:18
iceythat log line is from maas19:19
iceyit is what juju sent to maas for it19:19
roaksoaxicey: confirm that's what juju requests19:19
roaksoaxicey: because if that's the case, then there's either a bug in juju, in the bundle or similar19:19
iceywell, I have a clean controller up on this maas19:19
iceyhow can I confirm what Juju is sending, without just looking at the maas logs?19:20
roaksoaxicey: because juding by that log, what I read is that juju is telling maas interface=magpie:space=-1 instead of interface=magpie:space=internal19:20
icey(looking with log levels turned up, I haven't noticed raw requests)19:20
roaksoaxicey: i dont have an environment setup atm, bu juju debug-log should show you the request it makes to maas and which constraints it uses19:21
iceymachine-0: 19:25:15 TRACE maas request f9: POST http://10.0.4.2:5240/MAAS/api/2.0/machines/?op=allocate, params=agent_name=63e56b3b-db02-4ad2-8115-d76916426f8e&interfaces=magpie%3Aspace%3D-1&tags=nvme&zone=defau19:26
iceylt19:26
iceymachine-0: 19:25:15 TRACE maas response f9: error: "ServerError: 404 Not Found (Not Found)"19:26
roaksoaxicey: -119:26
roaksoaxicey: that seems juju to me19:26
iceyJuju is sending the interfaces=magpie:-1 but19:26
iceythen, it seems like there are two options: I'm failing to translate my maas spaces for juju correctly, or juju is confusing my spaces19:27
iceydo I need to run `add-space` for juju to know about the spaces?!19:28
iceymhm nope: https://pastebin.ubuntu.com/p/sbsYYnVPCH/19:28
iceyJuju knows about the spaces, and has the correct subnets19:29
roaksoaxicey: i actually would have thought juju would automatically get those19:29
iceyroaksoax it did19:29
roaksoaxicey: uhmmm19:29
roaksoaxicey: i would file a bug with juju folks19:29
iceyroaksoax yea19:29
roaksoaxi dont know the juju codebase unfortunately to know19:29
iceythey're going to love me this week; there's an issue with the lxd container networking on maas as well :-D19:29
iceywhere it doesn't make the bridge: https://pastebin.ubuntu.com/p/Gf3qkQ6SzS/19:30
iceywoot :-P19:30
roaksoaxicey: i think that may be a cloud-init issue actually19:30
iceyoh really19:31
roaksoaxicey: which should be resolved with cloud-init from -proposed19:31
iceyroaksoax interesting; I'm going to try one more thing to see if it resolves this weirdness; as it'll let me write a more helpful bug if it works (I have a nasty theory that if the units directly on a machine are in the undefined space, juju forgets about spaces when placing containers)19:33
roaksoaxugh, probably19:33
iceywow really weird19:34
icey failed to start machine 2 (failed to acquire node: No available machine matches constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])] (resolved to "interfaces=magpie:space=4 tags=nvme zone=default")), retrying in 10s (9 more attempts)19:34
iceythe other machines I'm asking for work19:34
iceywell19:34
iceyprobably because I don't have 4 machines in those spaces :-P19:34
iceysuper weird19:34
roaksoaxicey: so are you selectin space=4 ?19:36
roaksoaxit seems to me that space=<the name of space>19:36
iceyyes19:36
iceynot selecting space=419:36
iceybut space=internal/storage19:36
roaksoaxicey: so juju is sending an ID instead of a space name ?19:36
iceyI'm trying to get stuff deploying with this split network (10G internal, 1G public)19:36
iceywell, it sends the space ID: Aug  3 19:35:20 bifrost maas.api: [info] Request from user chris to acquire a machine with constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])]19:37
iceyit looks like it's aliasing the (undefined) space as -119:39
roaksoaxicey: yeah, that's the 'id' of the space in MAAS19:39
iceyroaksoax except apparently -1 isn't correct, as maas gives a 404 on it :-/19:40
iceyHA, juju doesn't think that the undefined space is -119:40
iceyhttps://pastebin.ubuntu.com/p/R8rgDqX7GD/19:40
roaksoaxicey:19:40
roaksoaxubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=1 dry_run=True | pastebinit -f diff19:40
roaksoaxhttp://paste.ubuntu.com/p/CtTJzbrDrx/19:41
roaksoaxicey: it works for me, e.g. i do get a machine in that space19:41
iceyright, space=119:41
iceyjuju is sending space=-119:41
iceythere _is_ no space=-119:41
iceymy space IDs are (apparently) 5, 8, and 1119:41
roaksoaxubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=-1 dry_run=True19:42
roaksoaxNot Found19:42
roaksoaxicey: yeah19:42
iceywell, maas acts reasonably with it :-P19:43
roaksoaxicey: but seems juju should be requesting a machine with a space19:43
iceywell, the last machine has no space requirements19:43
iceyI'm actually using a named space binding, maybe I should make it the default binding ("": undefined)19:43
roaksoaxicey: but are you specifically requesting a machine "without space" ?19:44
iceyroaksoax no, I'm specifying that it should be in the undefined space19:44
iceyI suppose I could try removing that binding19:44
roaksoaxicey: ok, hold on, 'the undefined space' doesn't exist :). It means there's no space defined19:44
iceyyeah yeah19:44
roaksoaxicey: but that said, juju translates that into '-1'19:45
iceyhttps://screenshots.firefox.com/bRt3jP2rdzwNdRON/10.0.4.219:45
iceyundefined space :-P19:45
iceyyou think it would be better to actually give that an official space?19:45
iceyfrankly, I'd rather that be the default and everything else require specification19:45
roaksoaxicey: so this is what I see19:47
roaksoaxubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=undefined dry_run=True | pastebinit -f diff19:47
roaksoaxhttp://paste.ubuntu.com/p/xwqt9g6Q48/19:47
roaksoaxicey: so if you specify space=undefined, it does give you a machine19:47
roaksoaxicey: where as if you specify space=-1, it doesn't19:47
iceyyeah; unfortunately juju is sending -119:47
iceyI think giving _every_ network a space is going to get it working, but it's annoying that it has to be explicit :-/19:49
roaksoaxicey:19:50
roaksoaxubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=2 dry_run=True | pastebinit19:50
roaksoaxhttp://paste.ubuntu.com/p/QpDMBxNqkz/19:50
roaksoaxubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=testing dry_run=True | pastebinit19:50
roaksoaxhttp://paste.ubuntu.com/p/hWD5KY2dYP/19:50
roaksoaxicey: so i changed from undefined to another space called testing19:50
roaksoaxicey: and sending 'ids' it now works19:50
roaksoaxso the bug is in maas19:50
roaksoaxwhere if you send -1, it won't work19:50
iceywell, isn't that fun :-P19:51
iceyI suppose I'll target MAAS with my bug as well :-P19:51
roaksoax*but* i need to understand  whether it is intended to allow request space=undefined19:51
iceyenjoy roaksoax https://bugs.launchpad.net/maas/+bug/178531419:51
roaksoaxicey: because if may be that we want not_space19:51
iceyroaksoax we may decide that any space use means all spaces must be explicit, but if we do, we should document that :-P19:52
iceyI'm off to bed :-P19:52
iceyhave a good weekend!19:53
roaksoaxicey: y too19:53
roaksoaxu too19:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!