| icey | I have a machine in my MAAS that Juju cannot deploy to; when it requests a machine, I can see in the maas.log that a machine was requested but Juju reports "failed to acquire node: unexpected: ServerError: 404 Not Found (Not Found)" ; I'm not sure where to look next | 06:47 |
|---|---|---|
| bjarne_l | Hi, i got the new 2.5.0~alpha1-7104-g18ae9e407-0ubuntu1~18.04.1 when i updated today, but I don't understand which commit it's based on? the tag 2.5.0-alpha1 is from a65328f744d16e840074633a980239d250d2c90a on 2018-05-24, which cant be the one, can it? | 08:34 |
| TJ- | bjarne_l: the commit ID is in the version: 18ae9e407 | 08:36 |
| TJ- | bjarne_l: If I recall my git-foo correctly, the version format is <tag>-<qty-commits-since-then>-<commit-id> | 08:37 |
| roaksoax | icey: that's a strange error, seems like it didn't find the machine. Maybe incorrect constraints? | 19:03 |
| roaksoax | bradm: i guess i didn't tag it correctly :) | 19:04 |
| roaksoax | err | 19:04 |
| roaksoax | bjarne_l: ^^ | 19:04 |
| roaksoax | bjarne_l: i guess i didn't tag it correctly | 19:04 |
| icey | roaksoax I'm trying to use a fairly simple bundle: https://pastebin.ubuntu.com/p/QJ325FMcWn/ | 19:11 |
| icey | if I remove spaces (do something like deploy the ubuntu charm), it works fine | 19:11 |
| icey | but the moment spaces are in, it gets weird | 19:11 |
| icey | I tried declaring the spaces in the machine constraints as well but that didn't work | 19:11 |
| icey | (may have done it wrong) | 19:11 |
| roaksoax | icey: because spaces are using to constraint the machine | 19:12 |
| roaksoax | icey: it probably means your MAAS doesn't have spaces | 19:12 |
| icey | it does | 19:12 |
| icey | I made them myself | 19:12 |
| icey | https://screenshots.firefox.com/wDRMmBk7pix7JNR1/10.0.4.2 roaksoax | 19:13 |
| roaksoax | icey: and the vlan where the machine is, in the right space ? | 19:13 |
| roaksoax | icey: or, another way to look at it, is the machine in any of those two subnets ? | 19:14 |
| icey | roaksoax there's 3 machines, they each have 2 nics on one switch (internal), and one on another (public) | 19:14 |
| icey | the machines are all configured like: (screenshot incoming) | 19:14 |
| icey | https://screenshots.firefox.com/FMpdyukQWiAMvtCX/10.0.4.2 roaksoax | 19:15 |
| roaksoax | icey: ok. So do this. Grab the constraints that juju is using when requesting a machine from maas | 19:16 |
| icey | (I'm hoping that you'll just say, "Oh! That needs to be configured like this!" and it'll work) | 19:16 |
| icey | [('agent_name', ['eb790a32-2962-4c3f-8825-159b612fd63b']), ('interfaces', ['magpie:space=-1']), ('tags', ['nvme']), ('zone', ['default'])] | 19:16 |
| icey | (already have the maas logs open :-P) | 19:16 |
| roaksoax | icey: space=-1 | 19:17 |
| roaksoax | icey: that seems to be the issue | 19:17 |
| icey | I agree; something is amis :-P | 19:18 |
| icey | for reference, I'm using maas from the stable PPA on Bionic (2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1)) | 19:18 |
| roaksoax | icey: the log with constraints, is that from juju or maas ? | 19:18 |
| icey | that log line is from maas | 19:19 |
| icey | it is what juju sent to maas for it | 19:19 |
| roaksoax | icey: confirm that's what juju requests | 19:19 |
| roaksoax | icey: because if that's the case, then there's either a bug in juju, in the bundle or similar | 19:19 |
| icey | well, I have a clean controller up on this maas | 19:19 |
| icey | how can I confirm what Juju is sending, without just looking at the maas logs? | 19:20 |
| roaksoax | icey: because juding by that log, what I read is that juju is telling maas interface=magpie:space=-1 instead of interface=magpie:space=internal | 19:20 |
| icey | (looking with log levels turned up, I haven't noticed raw requests) | 19:20 |
| roaksoax | icey: i dont have an environment setup atm, bu juju debug-log should show you the request it makes to maas and which constraints it uses | 19:21 |
| icey | machine-0: 19:25:15 TRACE maas request f9: POST http://10.0.4.2:5240/MAAS/api/2.0/machines/?op=allocate, params=agent_name=63e56b3b-db02-4ad2-8115-d76916426f8e&interfaces=magpie%3Aspace%3D-1&tags=nvme&zone=defau | 19:26 |
| icey | lt | 19:26 |
| icey | machine-0: 19:25:15 TRACE maas response f9: error: "ServerError: 404 Not Found (Not Found)" | 19:26 |
| roaksoax | icey: -1 | 19:26 |
| roaksoax | icey: that seems juju to me | 19:26 |
| icey | Juju is sending the interfaces=magpie:-1 but | 19:26 |
| icey | then, it seems like there are two options: I'm failing to translate my maas spaces for juju correctly, or juju is confusing my spaces | 19:27 |
| icey | do I need to run `add-space` for juju to know about the spaces?! | 19:28 |
| icey | mhm nope: https://pastebin.ubuntu.com/p/sbsYYnVPCH/ | 19:28 |
| icey | Juju knows about the spaces, and has the correct subnets | 19:29 |
| roaksoax | icey: i actually would have thought juju would automatically get those | 19:29 |
| icey | roaksoax it did | 19:29 |
| roaksoax | icey: uhmmm | 19:29 |
| roaksoax | icey: i would file a bug with juju folks | 19:29 |
| icey | roaksoax yea | 19:29 |
| roaksoax | i dont know the juju codebase unfortunately to know | 19:29 |
| icey | they're going to love me this week; there's an issue with the lxd container networking on maas as well :-D | 19:29 |
| icey | where it doesn't make the bridge: https://pastebin.ubuntu.com/p/Gf3qkQ6SzS/ | 19:30 |
| icey | woot :-P | 19:30 |
| roaksoax | icey: i think that may be a cloud-init issue actually | 19:30 |
| icey | oh really | 19:31 |
| roaksoax | icey: which should be resolved with cloud-init from -proposed | 19:31 |
| icey | roaksoax interesting; I'm going to try one more thing to see if it resolves this weirdness; as it'll let me write a more helpful bug if it works (I have a nasty theory that if the units directly on a machine are in the undefined space, juju forgets about spaces when placing containers) | 19:33 |
| roaksoax | ugh, probably | 19:33 |
| icey | wow really weird | 19:34 |
| icey | failed to start machine 2 (failed to acquire node: No available machine matches constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])] (resolved to "interfaces=magpie:space=4 tags=nvme zone=default")), retrying in 10s (9 more attempts) | 19:34 |
| icey | the other machines I'm asking for work | 19:34 |
| icey | well | 19:34 |
| icey | probably because I don't have 4 machines in those spaces :-P | 19:34 |
| icey | super weird | 19:34 |
| roaksoax | icey: so are you selectin space=4 ? | 19:36 |
| roaksoax | it seems to me that space=<the name of space> | 19:36 |
| icey | yes | 19:36 |
| icey | not selecting space=4 | 19:36 |
| icey | but space=internal/storage | 19:36 |
| roaksoax | icey: so juju is sending an ID instead of a space name ? | 19:36 |
| icey | I'm trying to get stuff deploying with this split network (10G internal, 1G public) | 19:36 |
| icey | well, it sends the space ID: Aug 3 19:35:20 bifrost maas.api: [info] Request from user chris to acquire a machine with constraints: [('agent_name', ['24afd75e-dad2-4a9d-82f9-d320144e8ddc']), ('interfaces', ['magpie:space=4']), ('tags', ['nvme']), ('zone', ['default'])] | 19:37 |
| icey | it looks like it's aliasing the (undefined) space as -1 | 19:39 |
| roaksoax | icey: yeah, that's the 'id' of the space in MAAS | 19:39 |
| icey | roaksoax except apparently -1 isn't correct, as maas gives a 404 on it :-/ | 19:40 |
| icey | HA, juju doesn't think that the undefined space is -1 | 19:40 |
| icey | https://pastebin.ubuntu.com/p/R8rgDqX7GD/ | 19:40 |
| roaksoax | icey: | 19:40 |
| roaksoax | ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=1 dry_run=True | pastebinit -f diff | 19:40 |
| roaksoax | http://paste.ubuntu.com/p/CtTJzbrDrx/ | 19:41 |
| roaksoax | icey: it works for me, e.g. i do get a machine in that space | 19:41 |
| icey | right, space=1 | 19:41 |
| icey | juju is sending space=-1 | 19:41 |
| icey | there _is_ no space=-1 | 19:41 |
| icey | my space IDs are (apparently) 5, 8, and 11 | 19:41 |
| roaksoax | ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=-1 dry_run=True | 19:42 |
| roaksoax | Not Found | 19:42 |
| roaksoax | icey: yeah | 19:42 |
| icey | well, maas acts reasonably with it :-P | 19:43 |
| roaksoax | icey: but seems juju should be requesting a machine with a space | 19:43 |
| icey | well, the last machine has no space requirements | 19:43 |
| icey | I'm actually using a named space binding, maybe I should make it the default binding ("": undefined) | 19:43 |
| roaksoax | icey: but are you specifically requesting a machine "without space" ? | 19:44 |
| icey | roaksoax no, I'm specifying that it should be in the undefined space | 19:44 |
| icey | I suppose I could try removing that binding | 19:44 |
| roaksoax | icey: ok, hold on, 'the undefined space' doesn't exist :). It means there's no space defined | 19:44 |
| icey | yeah yeah | 19:44 |
| roaksoax | icey: but that said, juju translates that into '-1' | 19:45 |
| icey | https://screenshots.firefox.com/bRt3jP2rdzwNdRON/10.0.4.2 | 19:45 |
| icey | undefined space :-P | 19:45 |
| icey | you think it would be better to actually give that an official space? | 19:45 |
| icey | frankly, I'd rather that be the default and everything else require specification | 19:45 |
| roaksoax | icey: so this is what I see | 19:47 |
| roaksoax | ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=undefined dry_run=True | pastebinit -f diff | 19:47 |
| roaksoax | http://paste.ubuntu.com/p/xwqt9g6Q48/ | 19:47 |
| roaksoax | icey: so if you specify space=undefined, it does give you a machine | 19:47 |
| roaksoax | icey: where as if you specify space=-1, it doesn't | 19:47 |
| icey | yeah; unfortunately juju is sending -1 | 19:47 |
| icey | I think giving _every_ network a space is going to get it working, but it's annoying that it has to be explicit :-/ | 19:49 |
| roaksoax | icey: | 19:50 |
| roaksoax | ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=2 dry_run=True | pastebinit | 19:50 |
| roaksoax | http://paste.ubuntu.com/p/QpDMBxNqkz/ | 19:50 |
| roaksoax | ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=testing dry_run=True | pastebinit | 19:50 |
| roaksoax | http://paste.ubuntu.com/p/hWD5KY2dYP/ | 19:50 |
| roaksoax | icey: so i changed from undefined to another space called testing | 19:50 |
| roaksoax | icey: and sending 'ids' it now works | 19:50 |
| roaksoax | so the bug is in maas | 19:50 |
| roaksoax | where if you send -1, it won't work | 19:50 |
| icey | well, isn't that fun :-P | 19:51 |
| icey | I suppose I'll target MAAS with my bug as well :-P | 19:51 |
| roaksoax | *but* i need to understand whether it is intended to allow request space=undefined | 19:51 |
| icey | enjoy roaksoax https://bugs.launchpad.net/maas/+bug/1785314 | 19:51 |
| roaksoax | icey: because if may be that we want not_space | 19:51 |
| icey | roaksoax we may decide that any space use means all spaces must be explicit, but if we do, we should document that :-P | 19:52 |
| icey | I'm off to bed :-P | 19:52 |
| icey | have a good weekend! | 19:53 |
| roaksoax | icey: y too | 19:53 |
| roaksoax | u too | 19:53 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!