[00:39] <mup> Bug #1274432 changed: MAAS does not make me a sandwich <MAAS:Won't Fix> <https://launchpad.net/bugs/1274432>
[01:00] <mup> Bug #1544795 opened: MAAS CLI should preserve the URL the user enters <MAAS:New> <https://launchpad.net/bugs/1544795>
[09:12] <haasn> Hmm. I'm trying out a new networking setup, where maas handles DHCP - and I plugged a new node into this subnet and booted it successfully, but after it ends with “cloud-init successful” etc.; it just does nothing
[09:12] <haasn> the node doesn't show up, the machine stays on
[09:18] <haasn> Ah
[09:18] <haasn> I tried changing the NTP IP published via DHCP to the one of the maas internal address, now I seem to get further - it's stuck in a loop because network is unreacahable
[09:18] <haasn> Because it's only plugged into a private network atm
[09:19] <haasn> (I want to test this configuration, where the MAAS nodes have no public interface at all)
[09:23] <haasn> Ah, maas-proxy configuration was missing an allow line for the new internal subnet I added
[09:24] <haasn> It should really automatically configure itself to allow all the subnets the cluster controller is in
[09:24] <haasn> (Or perhaps the one it's managing)
[09:26] <haasn> I'm still not sure why exactly it's failing. Says it's trying to contact 169.254.169.254, I don't recognize that IP
[09:28] <dweaver> haasn, that is the metadata service IP, like on AWS.  MAAS provides a metadata service on IP 169.254.169.254 for an image to contact using cloud-init.  The IP should be redirected to MAAS server in iptables.
[09:28] <haasn> Can't resolve DNS either, even though dig @cluster-controller ubuntu.com works
[09:29] <haasn> dweaver: Okay. It might be that the iptables setting in the node is wrong then. How can I log into it for debugging? I don't have any login details for this image
[09:30] <dweaver> haasn, so, if the image isn't contacting cloud-init, then it didn't get an SSH key and you won't be able to log in.
[09:30] <haasn> The last thing I see is “BEGIN SSH HOST KEY KEYS”, followed by some keys, followed by “END SSH HOST KEY KEYS” and “cloud-init v. 0.7.5 finished”
[09:31] <haasn> and then cc_final_message.py[WARNING]: Used fallback datasource
[09:31] <haasn> And now it's doing nothing
[09:31] <haasn> No reboot, no new entry in maas
[09:31] <dweaver> haasn, Yes, that's the clue - used fallback data source.
[09:31] <dweaver> It should have used the MAAS data source, so it couldn't get to 169.254.169.254 and obtain data.
[09:32] <dweaver> haasn, If you are using MAAS server as the router gateway, then it should "just work".
[09:33] <dweaver> haasn, sounds like something on the network config is not working for you.
[09:33] <haasn> dweaver: MAAS server is not a gateway
[09:33] <haasn> The nodes cannot reach the internet at all
[09:33] <haasn> They are on a private subnet with just the maas controller
[09:34] <haasn> And the maas controller has a DNS, HTTP and APT proxy configured
[09:35] <dweaver> haasn, so you haven't set a default route at all??
[09:35] <haasn> dweaver: Correct
[09:35] <dweaver> haasn, Try setting the default gateway to the MAAS server then
[09:35] <dweaver> Then packets destined for 169.254.169.254 should get sent to the MAAS server and iptables rewrite them
[09:36] <haasn> Oh, the iptables is on the MAAS server, not the node
[09:36] <dweaver> Otherwise they have nowhere to go.  Yes
[09:37] <haasn> Is this done so that the discovery images don't need to know about the IP of the maas region controller?
[09:37] <haasn> Wouldn't it be easier to pass the IP as a boot-time kernel parameter?
[09:39] <haasn> dweaver: Works now, thanks!
[09:39] <dweaver> haasn, MAAS is architected to be like a cloud provider, and use cloud-init on boot, which uses a metadata server for the data, like AWS, Openstack, Azure, GCE, etc.
[09:39] <haasn> Yeah, but I mean can't the metadata server pull the IP it's contacting from a kernel parameter
[09:39] <haasn> and then maas could provide its own IP as the kernel parameter when PXE booting
[09:39] <haasn> Then you wouldn't need the iptables “hack”
[09:40] <haasn> s/the metadataserver/cloud-init/
[09:42] <dweaver> haasn, Well, it could be done many, many different ways.
[09:43] <haasn> It does seem like some MAAS stuff is already pulling stuff from kernel parameters, e.g. during the commissioning process there's some URL to the maas server in there
[09:43] <dweaver> The hack is however a standardised method of providing data to a booting image over the network and allows cloud-init to always use the same method.
[09:46] <haasn> fair enough
[09:49] <haasn> I wonder if it would be possible to have the DNS zone setting on a per-interface basis, so I can configure a different DNS zone for every subnet
[10:14] <binoy> Is there any package available for maas to do the api calls
[11:34] <Razva> hey folks! my server has two NICs: eno1 and enp0s25. I don't know why but enp0s25 is not detected in any way
[11:34] <Razva> it's a fresh Ubuntu 15.10 install
[11:36] <Razva> 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05)
[11:36] <Razva> 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
[11:37] <Razva> 2: enp0s25: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
[11:37] <Razva> 3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
[12:14] <haasn> How do tags work? I'm trying to create a tag for virtual machines that looks like definition='contains(product, "KVM")', but I don't see this tag being applied to any nodes
[12:15] <haasn> Here is an example lshw output: https://0x0.st/XEl.txt
[12:15] <haasn> It says lshw:product: KVM ()
[12:21] <haasn> also tried /product, //node[class="system"]/product
[12:21] <haasn> I am using tag rebuild followed by tag list to check, and there are 0 nodes every time
[12:21] <Razva> I swear to God that I don't see any Networks in the menu: http://i.imgur.com/wKDBvvX.png
[12:21] <haasn> s/tag list/tag nodes/
[12:22] <haasn> Razva: JavaScript blocked or something?
[12:23] <Razva> nope...
[12:25] <haasn> I copy/pasted the QEMU example from http://en.community.dell.com/techcenter/os-applications/w/wiki/7432.using-tags-with-maas-and-juju-in-ubuntu-server-14-04-lts and changed ‘QEMU’ to ‘Red Hat’ (since that's what my VMs show up as) and it still does not work
[12:30] <haasn> Is the tag mechanism broken? Even examples _straight from the documentation_ simply do not work
[12:37] <haasn> I tried verifying my tag using the XPath evaluator at http://www.utilities-online.info/xpath/
[12:38] <haasn> That one required feeding it //lshw:node[@class='system']/lshw:product for it to be evaluated correctly
[12:38] <haasn> (The lshw: prefixes are notably absent from the MAAS docs)
[12:38] <haasn> But even with the lshw: prefixes it does not work inside the actual maas
[12:40] <haasn> Has anybody tested the tag mechanism at all?
[12:41] <haasn> Is there a single working example on the internet? MAAS v1.9
[12:42] <haasn> I don't think this documentation has been touched since v1.3, judging by the history
[12:51] <haasn> I extracted the lshw xml manually with lshw -xml > /tmp/lshw.xml and ran xmlstarlet sel -T -t -v 'contains(//node[@class="system"]/product, "KVM")' /tmp/lshw.xml and it returns ‘true’. So...
[12:51] <haasn> MAAS is clearly bugged, other XPath utilities find this just fine
[12:52]  * haasn opens a bug report
[13:10] <mup> Bug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>
[13:17] <mup> Bug #1544962 changed: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>
[13:20] <mup> Bug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>
[13:40] <Razva> folks really, I cannot boot from MAAS
[14:17] <roaksoax> haasn: on your bug, please attach the lshw output from MAAS, that will help instead of gathering the one from the system
[14:17] <roaksoax> redelmann: hi there. That doesn't really provide us much explanation. Can you expand?
[14:18] <redelmann> roaksoax, ??
[14:19] <roaksoax> redelmann: argh! my bad
[14:19] <roaksoax> Razva: Hi there! that doesn't really provide us with much explanation. Can you please expand?
[14:19] <redelmann> roaksoax, dont worry, btw i fix maas-proxy, after reinstall it start working.
[14:22] <roaksoax> redelmann: cool!
[14:22] <redelmann> roaksoax, the only change was switching public and private interfaces ( eth0 <--> eth1 )
[14:23] <Razva> yeeeeeeey roaksoax is on!
[14:24] <Razva> roaksoax ok so here it is. I'm making a fresh Ubuntu Cloud install on a server with two nics: eno1 (internet) and enp0s25 (lan)
[14:25] <Razva> question 1: do I need to setup the LAN IP manually, before installing MAAS, or should I add it through the GUI after installing MAAS?
[14:28] <roaksoax> Razva when you say the LAN IP, what do you mean?
[14:28] <roaksoax> Razva: typically, you'd configure the server on the networks you want it to be, before you install MAAS
[14:28] <Razva> 192.168.etc
[14:28] <roaksoax> Razva: after yoiu install MAAS, you configure DNS/DHCP from MAAS' WebUI to provide services on the network you want (aka. on a Cluster Interface)
[14:29] <roaksoax> redelmann: glad you made it work!
[14:29] <redelmann> (Y)
[14:30] <Razva> roaksoax can we chat on #ubuntu-server so jamespage is "on the loop"?
[14:30] <jamespage> I'm here
[14:30] <jamespage> ish
[14:30] <Razva> great
[14:31] <Razva> so, I'm going to install 15.10 and setup the "internet" nic, then login and setup the "lan" nic. all good 'til here?
[14:31] <Razva> btw thanks for your effort guys, I really really appreciate this!
[14:33] <roaksoax> Razva: right, so if you are using MAAS as the gateway, then you'd also need to setup NAT so the machines on the LAN can have internet
[14:33] <roaksoax> Razva: that being said, you should configure DHCP/DNS on the Cluster Interface connected to the LAN
[14:34] <Razva> got it. but before that I suppose I need to install Ubuntu Server "normally", by setting up the "net" nic, right?
[14:35] <Razva> I did SEVEN installs today, so I want to verify each step, just in case :D
[14:35] <roaksoax> Razva: right, so 1. install ubuntu server. 2. configure eth0 -> internet eth1 -> lan 3. install MAAS
[14:35] <Razva> is there any wasy way to rename a nic? they have terrible names... :|
[14:35] <roaksoax> Razva: you can manually rename the nic's if that so you wish
[14:36] <roaksoax> Razva: that is for the MAAS server
[14:37] <Razva> I was talking about the os nics, but whatever, I've learned the by heart :)
[14:43] <Razva> while it installs, let me state what I've learned about NAT. basically this is the solution: sudo iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eno1 -j MASQUERADE
[14:44] <Razva> where eno1 is the "net nic" and 1.0/24 is the LAN IP, which I'll set on the "lan nic"
[14:44] <Razva> is that correct?
[14:52] <Razva> roaksoax you've said "lan 3" that would be...the nat...?
[15:06] <roaksoax> Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan" 3. "Install MAAS"
[15:06] <roaksoax> Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan". 3. "Install MAAS"
[15:06] <Razva> ah ok :))
[15:11] <Razva> roaksoax looks good? http://pastebin.com/6V7Evui5
[15:15] <roaksoax> Razva: i think it does.. I'd need to look up iptables since haven't done them in a while :)
[15:16] <Razva> roger
[15:16] <Razva> how can I verify that the nat is ok?
[15:19] <haasn> I have a conceptual question with MAAS
[15:19] <Razva> roaksoax: http://pastebin.com/LU9FX5GP < full network setup, full network status, full iptables status
[15:19] <haasn> How do I map MAAS machines to physical locations, in case something is broken?
[15:19] <haasn> There is a “locator” functionality that can be used via IPMI to make the host's lights flash, but MAAS doesn't have a front-end for it (though I could write a wrapper script)
[15:20] <haasn> How do other deployments solve this problem?
[15:20] <haasn> e.g. say a drive fails on host whispered-news.maas. How do I find this host to replace the drive?
[15:20] <Razva> haasn sorry I'm a newb, but here's my funny idea: stickers! :)
[15:20] <haasn> Razva: So you mean, when MAAS generates its names, I attach a sticker of the generated name to the host?
[15:21] <Razva> YES
[15:21] <haasn> Then I will have an O(n) effort of finding the server given only the sticker name
[15:21] <haasn> What I *could* also do is, before booting the servers for the first time, add their MAC addresses to the maas DHCP with pre-defined names like host0 - host20
[15:22] <roaksoax> haasn: you can assign them to zones.. zones is just descriptive though
[15:22] <haasn> That way maas would give them names I could directly resolve to locations
[15:22] <Razva> OR you could write a script that'll play the Mario Soundtrack if/when something fails. THAT would be epic! :D
[15:22] <haasn> roaksoax: So like, “zone 2-1” to mean “first host in the second rack”? One zone per node?
[15:22] <haasn> Or just one zone per rack and then use stickers for the rest?
[15:22] <haasn> That latter approach could actually work really well
[15:23] <Razva> haasn https://www.autoitscript.com/forum/topic/40848-beep-music-mario-bros-theme/ < there you have it!
[15:24] <roaksoax> haasn: you can use zones whichever way you'd like
[15:24] <roaksoax> haasn: you can have zone-floor1 zone-floor2
[15:24] <roaksoax> for example
[15:24] <Razva> roaksoax can you please take a look on the pastebin (http://pastebin.com/LU9FX5GP)? I swear I'll not touch anything until somebody gives me the green light! :)
[15:25] <haasn> roaksoax: That's true, but I'm trying to minimize effort involved here - that's why I'm asking; to figure out which of the infinitely many solutions has the lowest effort
[15:25] <haasn> I'm mostly interested in how _real_world_ mass setups handle this
[15:25] <roaksoax> Razva: that seems to be ok
[15:26] <Razva> roaksoax: woohooo, next, install MAAS, as stated at http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot (steps #2 and #3)?
[15:26] <roaksoax> haasn: in real world people would use zones to place machines in different places or to group them by a comon identifier
[15:26] <roaksoax> Razva: that's correct
[15:26] <Razva> fingers crossed
[15:27] <Razva> should I apt-get upgrade or leave it like it is?
[15:27] <haasn> roaksoax: How do people map host names to physical server locations in the real world?
[15:28] <Razva> haasn that was a question I addressed 3 weeks ago. after a long discussion the answer was "as you wish".
[15:28] <haasn> Got any examples? I'm looking for inspiration
[15:28] <Razva> I've mapped them like this: town-job.continent.domain.tld
[15:28] <Razva> a controller in Maidenhead (europe) would be mh-controller1.eu.mydomain.com
[15:29] <Razva> some just use IDs, because a machine can/will change it's function in time
[15:29] <Razva> for example this controller can becone, in time, a compute or storage
[15:29] <haasn> Razva: How do you get MAAS to name them mh-controller1 instead of surprised-change or w/e?
[15:29] <haasn> Hard-code the MACs into its DHCPd?
[15:30] <haasn> Maybe I should start by describing my problem
[15:30] <Razva> haasn no idea, never worked with MAAS, this is my first try (I've worked with Proxmox, OpenVZ and OpenStack)
[15:30] <haasn> I now have 10 identical machines named only “ashamed-ducks”, “creative-winter”, “fine-jar” etc.
[15:30] <haasn> Right now the _only_ way I can distinguish them realistically is by looking at the MAC address, because right now that follows a clear pattern (they are test VMs, and VM 10 has :10 at the end of its MAC)
[15:31] <Razva> haasn are these your own machines or rented in some DC?
[15:31] <haasn> Razva: My own. (They're actually VM instances, but I plan to move to physical machines)
[15:31] <roaksoax> haasn: that depends per organization
[15:31] <haasn> Razva: And, well, that's why I'm asking this question in #maas specifically - I want to know how real-world *MAAS* deployments solve their hostnames
[15:31] <roaksoax> haasn: people in diferent organizations have different ways of doing that
[15:31] <haasn> Not how you could theoretically lay them out given full control :)
[15:32] <mup> Bug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>
[15:32] <haasn> I could maintain my own list of MAC -> server slot locations in an internal documentation wiki
[15:33] <haasn> (Or maintain this association as zones)
[15:34] <Razva> haasn belive it or not, I would just stick them. really.
[15:35] <haasn> Razva: Yeah, and put them into zones based on the rack
[15:35] <haasn> That's what I'll most likely end up doing
[15:35] <mup> Bug #1545035 changed: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>
[15:39] <Razva> haasn 9 years ago (virtualization was kinda "non-invented") I had ~2000 servers. we've tried a lot of software solutions, even wrote our own, but in the end...stickers fixed it :)
[15:39] <Razva> or you can try to use some fancy software, define your locations -> levels -> racks, and basically build a virtual representation of your DC
[15:39] <Razva> but in the end it will be very time consuming so...we dropped it
[15:44] <mup> Bug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>
[15:46] <Razva> roaksoax: http://i.imgur.com/QNvGXyz.jpg < looks ok?
[15:50] <Razva> roaksoax I swear to God that PXE doesn't works :|
[16:17] <roaksoax> Razva: you sure machines are pxe booting ? do the logs say anything?
[16:46] <Razva> roaksoax can you please ket me know what log files should I tailf?
[16:47] <Razva> DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25
[16:47] <Razva> DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25
[16:48] <Razva> is this a server trying to boot?
[16:48] <roaksoax> Razva: tail -f /var/log/maas/clusterd.log
[16:48] <roaksoax> Razva: that should tell you whether a node is trying to PXE or not
[16:50] <Razva> 2016-02-12 17:51:32+0200 [ClusterClient,client] Cluster '4fb34068-97f2-4cd9-a7dd-444475a71de7' registered (via mh-controller1:pid=1292).
[16:51] <Razva> but the machine didn't boot...
[16:51] <roaksoax> Razva: ps faux | grep dhcpd ?
[16:52] <roaksoax> Razva: do you have console logs? Are you sure your machine is trying to get a dhcp address from MAAS ?
[16:53] <Razva> dhcpd     1782  0.0  0.0  32908 13264 ?        Ss   17:54   0:00 dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid -cf /var/lib/maas/dhcpd.conf -lf /var/lib/maas/dhcp/dhcpd.leases enp0s25
[16:54] <Razva> no other server online on the network, just MAAS and the "client"
[16:57] <roaksoax> Razva: right, so that seems that dhcpd is providing on enp0s25
[16:57] <roaksoax> Razva: and your machines don't see DHCP offers
[16:57] <roaksoax> or your client
[16:57] <roaksoax> Razva: do you have stp enabled ?
[16:57] <Razva> no idea what is stp
[16:58] <roaksoax> https://maas.ubuntu.com/docs/install.html#configure-switches-on-the-network
[16:58] <roaksoax> Razva: ^^
[16:59] <Razva> no idea, it's a manged/shared switch with a vlan
[16:59] <Razva> Feb 12 18:41:27 mh-controller1 dhcpd[1782]: DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25
[16:59] <Razva> Feb 12 18:41:28 mh-controller1 dhcpd[1782]: DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25
[16:59] <Razva> doesn't this means that "somebody" is asking for offer?
[17:22] <roaksoax> Razva: apparently
[17:32] <Razva> roaksoax Portfast is enabled on the switch
[17:39] <Razva> enp0s25   Link encap:Ethernet  HWaddr 0c:c4:7a:0d:76:af
[17:39] <Razva> wait...so...the server is offering...dhcp to itself?
[19:29] <mup> Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
[19:35] <mup> Bug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
[19:38] <mup> Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
[19:41] <mup> Bug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
[19:44] <mup> Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
[20:14] <haasn> Razva: But if you have 2000 servers, each with a sticker on them, and you just know you have to find host X; do you then go through 2000 hosts one by one until you find X?
[20:35] <Razva> no, because each room has it's own "number", room is split into zones, and each zone is split into racks :)
[20:36] <Razva> you can have like L1R3Z6R2
[20:36] <Razva> Level 1 Room 3 Zone 6 Rack 2
[20:37] <Razva> if you have random names generated by MAAS, just make an XLS where host15r38 = L1R3Z6R2 and there you have it :)
[20:40] <haasn> XLS?
[21:54] <Razva> any idea why the heck all the nodes are shutting down after booting from PXE?