=== Salty is now known as Guest45701 === disposable3 is now known as disposable2 === mimizone_ is now known as mimizone === marlinc_ is now known as marlinc [15:18] I’m apparantly missing something regarding where to access the API. Is it provided on the same port as the WebUI (I’d assume so)? /api/version/ and /api/2.0/ both return “No Such Resource" [15:19] whelp. Just figured it out, /MAAS/api/version/ [15:21] that’s a tad unexpected. Had the impression /MAAS/ and /API/ would be respective UI and API roots. [15:21] The docs do not mention the MAAS_URL is the prefix. [15:34] benlake, please provide the URL to the doc you followed [15:35] https://docs.ubuntu.com/maas/2.1/en/api [15:36] benlake, ok, interesting. the intended page to get started with the CLI/API is: [15:36] https://docs.ubuntu.com/maas/2.1/en/manage-cli [15:37] err, I wanted to get started with the API. [15:37] we try to direct people at this top-level page: [15:37] https://docs.ubuntu.com/maas/2.1/en/intro-management [15:37] but i can see how you can end up directly on the API page [15:38] API Documentation is a giant top level button :D [15:38] right [15:38] I’m not using the CLI tool, I was looking for, and found, the API docs :P [15:39] which are great. Only trip was seeing all the /api roots paths, and not realizing that was in addition to whatever the configured MAAS_URL is [15:41] the API docs are generated from the maas code. i don't have any influence there [15:41] roaksoax, possible to add some clarity? ^^^ [15:42] benlake: cli is just a reflection of the API [15:42] benlake: it is auto generated [15:42] benlake: MAAS_URL is a config on /etc/maas/rackd.conf [15:42] I’m confused by what you’re attempting to convey [15:43] why is the CLI relevant to wanting to access the API directly and discoverying how? [15:43] and yes, I realize where the MAAS_URL is, it was just not apparent it needed to be the prefix to the paths referenced by the API docs. [15:45] the service provided on 5240 could provide both /api/ and /maas/ [15:45] nothing is wrong. I was just commenting that I tripped over /api needing to be /maas/api [15:46] benlake, in passing, for any general documentation issues you can open them here: [15:46] https://github.com/CanonicalLtd/maas-docs/issues/new [15:47] yeah, the threshold for making paperwork is tricky. [15:48] for a minor trip up that could easily be an expectation of mine simply not being the case; I leaned toward saying it here and seeing if it’s worth the issue. [15:48] thanks for he ref. [15:51] benlake, your comment is valid IMO. and IRC is fine to discuss. i really meant for any future issues [15:56] if I make interfaces changes to a controller, is there a way to get MAAS up to speed? Are the fabric references tied to interface names internally? [16:40] are PPAs added via MAAS available through the proxy? [16:41] * benlake is guessing not since Package Repositories says “…subsequently deployed” [17:02] Bug #1694759 opened: RSD Pod refresh shows ComposedNodeState is "Failed" [17:17] Bug #1694761 opened: Invalid Pod Resources error shows in API but not UI [17:26] Bug #1694761 changed: Invalid Pod Resources error shows in API but not UI [17:31] Hmm, are the MAAS NTP settings used in the PXE boot images? I’m seeing ntp.ubuntu.com used during an attempted Release operation. [17:32] I’m also seeing massing ureadahead errors, but I think they’ve been there for a while. [17:32] s/massing/massive/ - by massive I mean like 29k+ entries in syslog [17:32] Bug #1694761 opened: Invalid Pod Resources error shows in API but not UI [17:33] Bug #1694767 opened: RSD composition not setting local disk tags [17:48] benlake: the ureadahead errors are being looked at, non-maas issue though [17:48] benlake: that's a bug in ureadahead [17:48] thanks for the confirmation [17:49] benlake: as far as the proxy, it is a cashing proxy , so your ppas would still use the default proxy [17:49] benlake: on NTP, the commissioning environment will configure ntp [17:49] neat @ proxy [17:49] Re: ntp, not seeing that happen [17:50] benlake: go to /etc/ and grep for the expected config [17:50] i'd need to check [17:50] benlake: unless it is systemd timesync that's doing those queries [17:51] yeah, its systemd [17:52] I’m seeing these things, which are likely not related at all, because a Release is failing (multiple tries) [17:52] I’ve deployed/released this node a few times over the last few days, and now it’s hung [17:53] are there any telltales in the logs that could confirm for me the booted envronment is/will attempt to erase didks? [17:53] also disks [17:56] benlake: did the machine enter disk erasing mode and started erasing ? [17:56] yes, then timed out [17:56] benlake: you should see progress in the events log [17:57] benlake: timeout when ? [17:57] while waiting for disk erase to complete [17:57] it never timed out before... [17:57] benlake: do you have an example of how it did ? [17:57] I popped into KVM to see what was up and I caught the tail end of “erasing disks complete” and then it powered off [17:58] it just timed out again, so maybe. what do you mean example? [17:58] “Node changed status - From 'Disk erasing' to 'Failed disk erasing'" [18:00] benlake: do you have a full log ? [18:00] sure [18:00] benlake: also /var/log/maas/rsyslog///messages [18:01] would you only the latest release attempt and for me to remove ureadahead errors? [18:01] * benlake notices a lot of snapd nonsense attemting to access the internet [18:02] that’s annoying. [18:02] benlake: the ureadahead is unrelated though [18:03] right, I was just going to remove it from the log so you don’t have to see it... [18:03] it’s 29k entries of noise [18:10] roaksoax: this is allt he runs sans ureadahead http://paste.ubuntu.com/24728711/plain/ [18:10] I just noticed this, which is interesting: cloud-init[3156]: request to http://10.128.1.130:5240/MAAS/metadata//2012-03-01/ failed. sleeping 32.: HTTP Error 409: CONFLICT [18:17] according to the logs, it says it erased the disks. MAAS state for this node is “failed disk erasing”. I guess I’ll mark it as Broken, Fixed, and attempt to Deploy it [18:21] benlake: so, i need a full node event log [18:21] benlake: and the syslog [18:21] to know what may be wrong [18:21] all I did was grep -v ureadahead, you want all of that? [18:21] benlake: i think wiith what I have wrt to cloud-init is enough [18:22] the only thing I stripped was ureadahead [18:22] if that is unrelated, you have everything [18:23] benlake: my thinking is that this could be the culprit: May 31 17:55:52 electron systemd[1]: Failed to start Apply the settings specified in cloud-config. [18:23] May 31 17:55:52 electron systemd[1]: cloud-config.service: Unit entered failed state. [18:23] May 31 17:55:52 electron systemd[1]: cloud-config.service: Failed with result 'exit-code'. [18:23] benlake: do you have a node event log ? You can go to the UI and grab it from there [18:23] specially the "full" [18:23] or you can grab it from the API on the 'events' endpoint [18:23] lemme see [18:24] oh, you want the MAAS events, got it [18:26] ugh, between timesyncd and snapd, this thing sits around timing out for 5 minutes. [18:27] benlake: http://pastebin.ubuntu.com/24728895/ [18:27] that's the issue [18:28] I saw that before pasting and was about to mention it [18:30] here is the entry in MASS UI, https://screencast.com/t/9ccpHGoc [18:31] is something wrong with that? [18:34] I went through the events exercise anyhow: http://paste.ubuntu.com/24728917/ [18:37] hmm, it seems cloud-init might not know to use a proxy [18:39] is it a project expectation deploying nodes have internet access? [18:39] * benlake is just wondering [18:45] benlake: no. MAAS can work on completely offline environments [18:45] yeah, I see it can, I’ve been using it that way [18:45] benlake: but my guess of what's happening here is that sudo add-apt-repository ppa: is failing behind a proxy [18:46] just digging in the logs due to this PPA thing seems to show a lot of internet access attempts on these ephemeral images [18:46] roaksoax: aye, that seems to be the case. It not being configured to use one [18:46] https://askubuntu.com/questions/724224/cant-add-ppa-on-ubuntu-15-10-user-or-team-does-not-exist [18:46] old, but likely the same issue [18:48] benlake: the other thing you could do, is to add the ppa as a URL [18:49] I think I tried that, but the UI said it was invalid [18:49] prevented me from saving it. let me mess with it again [18:53] benlake: do it as 'add repository' [18:55] gotcha, does this look correct? https://screencast.com/t/lJy1PUpE49i [19:18] benlake, you want to add a PPA right? [19:18] yes, but see the discussion above [19:25] well, i know i added a PPA before so i'm confused. are you using the default proxy? [19:31] yes [19:31] are you sure your node didn’t have internet access when being deployed? [19:32] cloud-init attempts to use add-apt-repository when booted into the ephemeral environment and does not honor any proxy [19:34] oh. right, of course it did [19:36] enabling a PPA requires internet access [19:37] not sure MAAS devs expect that to be required. [19:37] if maas is supposed to be "offline ready" then i guess that's a bug. but i'm not sure how you can enable a PPA w/o internet [19:38] if they do, another update to the docs would be a good idea :) [19:38] can you file one? [19:38] if the env vars htto_proxy and https_proxy are set, it’ll use it [19:38] only if roaksoax confirms [19:38] *http_proxy [19:45] roaksoax: adding as non-PPA worked, thanks! Also the URL in my screenshot was completely wrong. I failed on that one. [19:55] :) [19:56] benlake: glad to hear it works now [19:56] benlake, you were able to add a PPA as a non-PPA? 'splain? [19:57] https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack [19:57] green link, “Technical details about this PPA" [19:57] shows the direct repo URL [19:58] can use that with the add repo option instead of add ppa. basically the PPA helpers just lookup that URL and fetch the repo’s public key for you [20:28] seems I have another auto IP address selection issue. The NTP server being set for deployed nodes is not the desired/functional IP [20:28] I like the cascade of NTP servers, region, rack ,etc. [20:29] but am I going to need to use the “force external NTP server” option here? [20:51] dang, but that would force the region controller to use itself as an NTP server. [20:57] benlake, thanks (for the PPA stuff) [22:55] benlake, what URL did you end up using? you said you made a mistake originally? [22:56] bad: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack [22:56] good: http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu