[07:18] <oz_> hey guys does anyone know on maas 1.9.3 can i tag multiple vlans on nodes added in maas itself? i was not able to change the part in the networking config on nodes.
[07:57] <mup> Bug #1604702 opened: Unable to add a 16384-bit ssh key to my account from web UI <MAAS:New> <https://launchpad.net/bugs/1604702>
[08:00] <mup> Bug #1604702 changed: Unable to add a 16384-bit ssh key to my account from web UI <MAAS:New> <https://launchpad.net/bugs/1604702>
[08:06] <mup> Bug #1604702 opened: Unable to add a 16384-bit ssh key to my account from web UI <MAAS:New> <https://launchpad.net/bugs/1604702>
[09:25] <siva> Hi. I have taken 3 physical servers. On one server I installed ubuntu 15.10(Wily). And then installed MAAS. Configured the MAAS Cluster. Now I am trying to register my hardware with MAAS. MAAS done with PXE boot of its nodes and assigned IPs . When i was trying to ssh into the MAAS nodes, it was giving Permission denied (public key issue). In MAAS UI I added public key of MAAS installed server. And finally it was showing commission statu
[09:26] <siva> Cloud-init v.0.7.5 started running and finished . I am using MAAS Version 1.9.3+bzr4577-0ubuntu1 (wily1).   Finally , the MAAS nodes have stuck with the info -> Cloud-init v.0.7.5 finished at wed, 20 jul 2016 13:01:51  +0000 Datasource  DataSource ConfigDriveNet [net, ver=2] [Source=/dev/sdb1]. UP 43.10sec.
[09:27] <siva> I observed the Maas log files under /var/log/maas. I didn’t get any error messages. But I didn’t understand why commission is failing.
[09:27] <siva> MAAS log info : http://paste.openstack.org/show/537405/
[09:28] <siva> Could you please provide me the solution why MAAS commission failed.
[10:06] <mup> Bug #1604738 opened: Dynamic subnet range disappears after edit and is replaced with "No IP ranges have been reserved for this subnet." <MAAS:New> <https://launchpad.net/bugs/1604738>
[12:57] <acovrig> what debugging steps can I take, deployments are failing and I’m not sure why
[13:15] <kiko> acovrig, you'll need to tell us more about the failure first :)
[13:16] <kiko> siva, hey there
[13:17] <acovrig> kiko: that’s what I’m asking, all I can tell is it says “Failed deployment”, how can I learn more about the failure to troubleshoot
[13:18] <kiko> acovrig, what are you deploying?
[13:23] <acovrig> ubuntu 16.04
[13:23] <roaksoax> acovrig: are you using MAAS 2.0 ?
[13:23] <roaksoax> acovrig: get the installation log from the WebUI (at the bottom of the page)
[13:24] <roaksoax> acovrig: and if you are using 2.0, look in: /var/log/maas/rsyslog/<machine-name>/../messages
[13:25] <acovrig> roaksoax: the last thing I get is Node installation - ‘cloudinit’ running modules for final then Node changed status - From ‘Deploying’ to ‘Failed deployment’
[13:27] <acovrig> I see a few “timed out waiting for reply from <ip> (ntp)” which would explain it trying to hit a status point of 2012…
[13:28] <roaksoax> acovrig: my guess is that it deployed successfully, but on the reboot, it either didn't boot from the disk, or it failed to reach maas after the the first reboot
[13:28] <roaksoax> acovrig: can you pastebin the cloud-init outout + the install log from the wbeui ?
[13:32] <acovrig> roaksoax: install log from the webui, where would the cloud-init output be? https://gist.github.com/acovrig/446fc9b616f72979c7995a5132fb607f
[13:39] <roaksoax> acovrig: yeah, if it is not there, then that means cloud-init/curtin didn't send it back
[13:39] <roaksoax> acovrig: also, can you pb the rsyslog ?
[13:40] <acovrig> roaksoax: sure, currenlty I have the BIOS set to PXE boot first, should I not (I.E. have it boot from the HDD after deployment)?
[13:41] <roaksoax> acovrig: no you don't, giving it a second look it seems something failed before the machine was actually rebooted
[13:41] <roaksoax> Node changed status - From 'Deploying' to 'Failed deployment'	Tue, 19 Jul. 2016 15:16:52
[13:41] <roaksoax> Node installation - 'cloudinit' running modules for final	Tue, 19 Jul. 2016 15:16:52
[13:41] <roaksoax> acovrig: look at the timestamp
[13:41] <roaksoax> acovrig: so the rsyslog should show why was that ?
[13:42] <acovrig> roaksoax: OK, gist updated (https://gist.github.com/acovrig/446fc9b616f72979c7995a5132fb607f#file-rsyslog)
[13:43] <acovrig> roaksoax: interesting: the rsyslog only goes to 13:53, not to 15:16 (time)
[13:45] <acovrig> roaksoax: sudo grep 15:16 /var/log/maas/rsyslog/HOST-DELL-*/*/messages shows nothing O.o
 I see a few “timed out waiting for reply from <ip> (ntp)” which would explain it trying to hit a status point of 2012…
[13:47] <kiko> roaksoax, acovrig: ^^^ is this not relevant?
[13:47] <kiko> could the times be skewed and failing for that reason?
[13:47] <acovrig> yea, I watched the physical console a few times and saw it getting a 401 from my MAAS server because of it; what would cause it to not be able to reach ntp?
[13:48] <kiko> acovrig, IIRC the node is trying to reach ntp.ubuntu.com
[13:48] <kiko> is that reachable?
[13:50] <acovrig> kiko: odd, I’m deploying to 2 nodes now to see if it was just a timing issue; I SSH’d into one of them and it can ping the MAAS server (router for the nodes) but not ping google (8.8.8.8), but it can DNS dig google.com...
[13:51] <kiko> acovrig, well, typically dig will work, as MAAS is the DNS server
[13:51] <kiko> acovrig, it looks like routing (or NAT) is broken?
[13:52] <acovrig> kiko: I have 3 FW rules on MAAS: -A FORWARD -i eno1.8 -o eno1 -m state --state RELATED,ESTABLISHED -j ACCEPT
[13:52] <roaksoax> kiko: the NTP shouldn't be an issue release
[13:52] <acovrig> -A FORWARD -i eno1 -o eno1.8 -j ACCEPT
[13:52] <roaksoax> acovrig: what do you have under /var/log/maas/rsyslog/ ?
[13:52] <acovrig> and -A POSTROUTING -o eno1.8 -j MASQUERADE
[13:52] <roaksoax> acovrig: is it just completely empty ?
[13:53] <kiko> acovrig, the MASQUERADE entry looks wrong
[13:53] <kiko> acovrig, shouldn't that be -o eno1?
[13:53] <kiko> oh hmm
[13:53] <acovrig> roaksoax: no, I have my 2 hosts/date/messages and maas-enlisting-node/date/{messages,ip}
[13:54] <kiko> roaksoax, right, but if there is no upstream connectivity, could things be failing?
[13:54] <acovrig> kiko: the MAAS server is on VLAN 8 (tagged) to get to the nodes (untagged)
[13:54] <roaksoax> kiko: yeah it could, but NTP shouldn't be a cause of failure
[13:55] <acovrig> don’t the nodes use a date to connect back to the MAAS server?
[13:55] <kiko> acovrig, eno1.8 is external, I understand now
[13:55] <kiko> acovrig, do the packets appear correctly masqueraded?
[13:55] <kiko> acovrig, if you plug a laptop into the untagged vlan, can it get out?
[13:56] <acovrig> kiko: good question, pending test
[13:57] <kiko> roaksoax, sure, but NTP failing could be an indication that networking in general is failing :)
[14:01] <acovrig> kiko, roaksoax: a windows 7 client can’t access icanhazip.com from chrome, so routing must be broken
[14:02] <kiko> acovrig, right. fix the networking first and then come back to me. :)
[14:02] <acovrig> I updated the gist (https://gist.github.com/acovrig/446fc9b616f72979c7995a5132fb607f) with my iptables and ifconfig; not sure what’s wrong...
[14:03] <acovrig> sysctl net.ipv4.ip_forward is on...
[14:03] <kiko> acovrig, check the input and output chains too, otherwise add logging and use tcpdump/wireshark to see what's happening
[14:11] <acovrig> kiko: OK, not sure what was wrong, but fixed iptables and networking works, just started a deployment, we’ll see what happens.
[14:12] <kiko> acovrig, list out the rules you have now (iptables -L FORWARD -n, perhaps INPUT and OUTPUT too) and put it in a paste?
[14:12] <acovrig> roaksoax: FYI: I couldn’t get maas to accept an SSH key yesterday and was able to install 2.0.0~beta3+bzr4941-0ubuntu1 and now it accepts my key(s).
[14:12] <kiko> acovrig, cool
[14:15] <acovrig> kiko: looking at diff, I got my ifs flipped; I guess the guide I looked at used eth1 as internet and eth0 as lan and I didn’t notice :/
[14:15] <kiko> aha
[14:15] <kiko> thanks
[14:20]  * acovrig emits a squeel of excitement: “Deployed” :D 
[14:22] <acovrig> now to get juju and openstack to work :)
[14:42] <kiko> acovrig, it worked? w00t!
[14:44] <acovrig> kiko: yup, now I just need to find a good guide to get juju+openstack to work as a failover cluster for VMs
[14:44] <roaksoax> acovrig: this should do : https://jujucharms.com/openstack-base/
[14:45] <acovrig> roaksoax: ooh, thanks
[17:11] <mup> Bug #1604901 opened: No way to retrieve complete event log through the cli <oil> <MAAS:New> <https://launchpad.net/bugs/1604901>
[17:17] <kiko> acovrig, all a-ok?
[17:18] <acovrig> kiko: with MAAS, yes, I’m still working though juju/openstack ‘solutions’
[17:18] <kiko> okay
[17:35] <G_Dog1985> do i need two server to run maas and openstack or can it be don by on mass
[17:35] <G_Dog1985> done*
[19:59] <mup> Bug #1604938 opened: node failed deployment "Exception: I/O operation on closed file." <oil> <MAAS:New> <https://launchpad.net/bugs/1604938>
[21:12] <mup> Bug #1604962 opened: node set to "failed deployment" for no visible reason <oil> <MAAS:New> <https://launchpad.net/bugs/1604962>
[22:06] <mup> Bug #1604987 opened: Event log should always include a reason why a node was marked Failed Deployment <oil> <MAAS:Triaged> <MAAS 2.0:Triaged> <MAAS trunk:Triaged> <https://launchpad.net/bugs/1604987>
[22:54] <mup> Bug #1546760 changed: Re-attempt power on for initial PXE failure during node deployment to alleviate PXE failures <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1546760>
[22:54] <mup> Bug #1605008 opened: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605008>
[23:00] <mup> Bug #1605008 changed: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605008>
[23:00] <mup> Bug #1546760 opened: Re-attempt power on for initial PXE failure during node deployment to alleviate PXE failures <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1546760>
[23:06] <mup> Bug #1546760 changed: Re-attempt power on for initial PXE failure during node deployment to alleviate PXE failures <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1546760>
[23:18] <mup> Bug #1605008 opened: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605008>
[23:24] <mup> Bug #1605008 changed: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:Invalid> <https://launchpad.net/bugs/1605008>