[15:18] <benlake> I’m apparantly missing something regarding where to access the API. Is it provided on the same port as the WebUI (I’d assume so)? /api/version/ and /api/2.0/ both return “No Such Resource"
[15:19] <benlake> whelp. Just figured it out, /MAAS/api/version/
[15:21] <benlake> that’s a tad unexpected. Had the impression /MAAS/ and /API/ would be respective UI and API roots.
[15:21] <benlake> The docs do not mention the MAAS_URL is the prefix.
[15:34] <pmatulis> benlake, please provide the URL to the doc you followed
[15:35] <benlake> https://docs.ubuntu.com/maas/2.1/en/api
[15:36] <pmatulis> benlake, ok, interesting. the intended page to get started with the CLI/API is:
[15:36] <pmatulis> https://docs.ubuntu.com/maas/2.1/en/manage-cli
[15:37] <benlake> err, I wanted to get started with the API.
[15:37] <pmatulis> we try to direct people at this top-level page:
[15:37] <pmatulis> https://docs.ubuntu.com/maas/2.1/en/intro-management
[15:37] <pmatulis> but i can see how you can end up directly on the API page
[15:38] <benlake> API Documentation is a giant top level button :D
[15:38] <pmatulis> right
[15:38] <benlake> I’m not using the CLI tool, I was looking for, and found, the API docs :P
[15:39] <benlake> which are great. Only trip was seeing all the /api roots paths, and not realizing that was in addition to whatever the configured MAAS_URL is
[15:41] <pmatulis> the API docs are generated from the maas code. i don't have any influence there
[15:41] <pmatulis> roaksoax, possible to add some clarity? ^^^
[15:42] <roaksoax> benlake: cli is just a reflection of the API
[15:42] <roaksoax> benlake: it is auto generated
[15:42] <roaksoax> benlake: MAAS_URL is a config on /etc/maas/rackd.conf
[15:42] <benlake> I’m confused by what you’re attempting to convey
[15:43] <benlake> why is the CLI relevant to wanting to access the API directly and discoverying how?
[15:43] <benlake> and yes, I realize where the MAAS_URL is, it was just not apparent it needed to be the prefix to the paths referenced by the API docs.
[15:45] <benlake> the service provided on 5240 could provide both /api/ and /maas/
[15:45] <benlake> nothing is wrong. I was just commenting that I tripped over /api needing to be /maas/api
[15:46] <pmatulis> benlake, in passing, for any general documentation issues you can open them here:
[15:46] <pmatulis> https://github.com/CanonicalLtd/maas-docs/issues/new
[15:47] <benlake> yeah, the threshold for making paperwork is tricky.
[15:48] <benlake> for a minor trip up that could easily be an expectation of mine simply not being the case; I leaned toward saying it here and seeing if it’s worth the issue.
[15:48] <benlake> thanks for he ref.
[15:51] <pmatulis> benlake, your comment is valid IMO. and IRC is fine to discuss. i really meant for any future issues
[15:56] <benlake> if I make interfaces changes to a controller, is there a way to get MAAS up to speed? Are the fabric references tied to interface names internally?
[16:40] <benlake> are PPAs added via MAAS available through the proxy?
[16:41]  * benlake is guessing not since Package Repositories says “…subsequently deployed”
[17:02] <mup> Bug #1694759 opened: RSD Pod refresh shows ComposedNodeState is "Failed" <MAAS:Confirmed for newell-jensen> <MAAS RSD :Confirmed for newell-jensen> <https://launchpad.net/bugs/1694759>
[17:17] <mup> Bug #1694761 opened: Invalid Pod Resources error shows in API but not UI <MAAS:Confirmed for blake-rouse> <https://launchpad.net/bugs/1694761>
[17:26] <mup> Bug #1694761 changed: Invalid Pod Resources error shows in API but not UI <MAAS:Confirmed for blake-rouse> <https://launchpad.net/bugs/1694761>
[17:31] <benlake> Hmm, are the MAAS NTP settings used in the PXE boot images? I’m seeing ntp.ubuntu.com used during an attempted Release operation.
[17:32] <benlake> I’m also seeing massing ureadahead errors, but I think they’ve been there for a while.
[17:32] <benlake> s/massing/massive/ - by massive I mean like 29k+ entries in syslog
[17:32] <mup> Bug #1694761 opened: Invalid Pod Resources error shows in API but not UI <MAAS:Confirmed for blake-rouse> <https://launchpad.net/bugs/1694761>
[17:33] <mup> Bug #1694767 opened: RSD composition not setting local disk tags <MAAS:Confirmed for newell-jensen> <MAAS RSD :Confirmed for newell-jensen> <https://launchpad.net/bugs/1694767>
[17:48] <roaksoax> benlake: the ureadahead errors are being looked at, non-maas issue though
[17:48] <roaksoax> benlake: that's a bug in ureadahead
[17:48] <benlake> thanks for the confirmation
[17:49] <roaksoax> benlake: as far as the proxy, it is a cashing proxy , so your ppas would still use the default proxy
[17:49] <roaksoax> benlake: on NTP, the commissioning environment will configure ntp
[17:49] <benlake> neat @ proxy
[17:49] <benlake> Re: ntp, not seeing that happen
[17:50] <roaksoax> benlake: go to /etc/ and grep for the expected config
[17:50] <roaksoax> i'd need to check
[17:50] <roaksoax> benlake: unless it is systemd timesync that's doing those queries
[17:51] <benlake> yeah, its systemd
[17:52] <benlake> I’m seeing these things, which are likely not related at all, because a Release is failing (multiple tries)
[17:52] <benlake> I’ve deployed/released this node a few times over the last few days, and now it’s hung
[17:53] <benlake> are there any telltales in the logs that could confirm for me the booted envronment is/will attempt to erase didks?
[17:53] <benlake> also disks
[17:56] <roaksoax> benlake: did the machine enter disk erasing mode and started erasing ?
[17:56] <benlake> yes, then timed out
[17:56] <roaksoax> benlake: you should see progress in the events log
[17:57] <roaksoax> benlake: timeout when ?
[17:57] <benlake> while waiting for disk erase to complete
[17:57] <benlake> it never timed out before...
[17:57] <roaksoax> benlake: do you have an example of how it did ?
[17:57] <benlake> I popped into KVM to see what was up and I caught the tail end of “erasing disks complete” and then it powered off
[17:58] <benlake> it just timed out again, so maybe. what do you mean example?
[17:58] <benlake> “Node changed status - From 'Disk erasing' to 'Failed disk erasing'"
[18:00] <roaksoax> benlake: do you have a full log ?
[18:00] <benlake> sure
[18:00] <roaksoax> benlake: also /var/log/maas/rsyslog/<machine-name>/<date>/messages
[18:01] <benlake> would you only the latest release attempt and for me to remove ureadahead errors?
[18:01]  * benlake notices a lot of snapd nonsense attemting to access the internet
[18:02] <benlake> that’s annoying.
[18:02] <roaksoax> benlake: the ureadahead is unrelated though
[18:03] <benlake> right, I was just going to remove it from the log so you don’t have to see it...
[18:03] <benlake> it’s 29k entries of noise
[18:10] <benlake> roaksoax: this is allt he runs sans ureadahead http://paste.ubuntu.com/24728711/plain/
[18:10] <benlake> I just noticed this, which is interesting: cloud-init[3156]: request to http://10.128.1.130:5240/MAAS/metadata//2012-03-01/ failed. sleeping 32.: HTTP Error 409: CONFLICT
[18:17] <benlake> according to the logs, it says it erased the disks. MAAS state for this node is “failed disk erasing”. I guess I’ll mark it as Broken, Fixed, and attempt to Deploy it
[18:21] <roaksoax> benlake: so, i need a full node event log
[18:21] <roaksoax> benlake: and the syslog
[18:21] <roaksoax> to know what may be wrong
[18:21] <benlake> all I did was grep -v ureadahead, you want all of that?
[18:21] <roaksoax> benlake: i think wiith what I have wrt to cloud-init is enough
[18:22] <benlake> the only thing I stripped was ureadahead
[18:22] <benlake> if that is unrelated, you have everything
[18:23] <roaksoax> benlake: my thinking is that this could be the culprit: May 31 17:55:52 electron systemd[1]: Failed to start Apply the settings specified in cloud-config.
[18:23] <roaksoax> May 31 17:55:52 electron systemd[1]: cloud-config.service: Unit entered failed state.
[18:23] <roaksoax> May 31 17:55:52 electron systemd[1]: cloud-config.service: Failed with result 'exit-code'.
[18:23] <roaksoax> benlake: do you have a node event log ? You can go to the UI and grab it from there
[18:23] <roaksoax> specially the "full"
[18:23] <roaksoax> or you can grab it from the API on the 'events' endpoint
[18:23] <benlake> lemme see
[18:24] <benlake> oh, you want the MAAS events, got it
[18:26] <benlake> ugh, between timesyncd and snapd, this thing sits around timing out for 5 minutes.
[18:27] <roaksoax> benlake: http://pastebin.ubuntu.com/24728895/
[18:27] <roaksoax> that's the issue
[18:28] <benlake> I saw that before pasting and was about to mention it
[18:30] <benlake> here is the entry in MASS UI, https://screencast.com/t/9ccpHGoc
[18:31] <benlake> is something wrong with that?
[18:34] <benlake> I went through the events exercise anyhow: http://paste.ubuntu.com/24728917/
[18:37] <benlake> hmm, it seems cloud-init might not know to use a proxy
[18:39] <benlake> is it a project expectation deploying nodes have internet access?
[18:39]  * benlake is just wondering
[18:45] <roaksoax> benlake: no. MAAS can work on completely offline environments
[18:45] <benlake> yeah, I see it can, I’ve been using it that way
[18:45] <roaksoax> benlake: but my guess of what's happening here is that sudo add-apt-repository ppa:<etc> is failing behind a proxy
[18:46] <benlake> just digging in the logs due to this PPA thing seems to show a lot of internet access attempts on these ephemeral images
[18:46] <benlake> roaksoax: aye, that seems to be the case. It not being configured to use one
[18:46] <benlake> https://askubuntu.com/questions/724224/cant-add-ppa-on-ubuntu-15-10-user-or-team-does-not-exist
[18:46] <benlake> old, but likely the same issue
[18:48] <roaksoax> benlake: the other thing you could do, is to add the ppa as a URL
[18:49] <benlake> I think I tried that, but the UI said it was invalid
[18:49] <benlake> prevented me from saving it. let me mess with it again
[18:53] <roaksoax> benlake: do it as 'add repository'
[18:55] <benlake> gotcha, does this look correct? https://screencast.com/t/lJy1PUpE49i
[19:18] <pmatulis> benlake, you want to add a PPA right?
[19:18] <benlake> yes, but see the discussion above
[19:25] <pmatulis> well, i know i added a PPA before so i'm confused. are you using the default proxy?
[19:31] <benlake> yes
[19:31] <benlake> are you sure your node didn’t have internet access when being deployed?
[19:32] <benlake> cloud-init attempts to use add-apt-repository when booted into the ephemeral environment and does not honor any proxy
[19:34] <pmatulis> oh. right, of course it did
[19:36] <pmatulis> enabling a PPA requires internet access
[19:37] <benlake> not sure MAAS devs expect that to be required.
[19:37] <pmatulis> if maas is supposed to be "offline ready" then i guess that's a bug. but i'm not sure how you can enable a PPA w/o internet
[19:38] <benlake> if they do, another update to the docs would be a good idea :)
[19:38] <pmatulis> can you file one?
[19:38] <benlake> if the env vars htto_proxy and https_proxy are set, it’ll use it
[19:38] <benlake> only if roaksoax confirms
[19:38] <benlake> *http_proxy
[19:45] <benlake> roaksoax: adding as non-PPA worked, thanks! Also the URL in my screenshot was completely wrong. I failed on that one.
[19:55] <roaksoax> :)
[19:56] <roaksoax> benlake: glad to hear it works now
[19:56] <pmatulis> benlake, you were able to add a PPA as a non-PPA? 'splain?
[19:57] <benlake> https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
[19:57] <benlake> green link, “Technical details about this PPA"
[19:57] <benlake> shows the direct repo URL
[19:58] <benlake> can use that with the add repo option instead of add ppa. basically the PPA helpers just lookup that URL and fetch the repo’s public key for you
[20:28] <benlake> seems I have another auto IP address selection issue. The NTP server being set for deployed nodes is not the desired/functional IP
[20:28] <benlake> I like the cascade of NTP servers, region, rack ,etc.
[20:29] <benlake> but am I going to need to use the “force external NTP server” option here?
[20:51] <benlake> dang, but that would force the region controller to use itself as an NTP server.
[20:57] <pmatulis> benlake, thanks (for the PPA stuff)
[22:55] <pmatulis> benlake, what URL did you end up using? you said you made a mistake originally?
[22:56] <benlake> bad: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
[22:56] <benlake> good: http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu