[00:39] Bug #1274432 changed: MAAS does not make me a sandwich [01:00] Bug #1544795 opened: MAAS CLI should preserve the URL the user enters === med_ is now known as Guest76507 [09:12] Hmm. I'm trying out a new networking setup, where maas handles DHCP - and I plugged a new node into this subnet and booted it successfully, but after it ends with “cloud-init successful” etc.; it just does nothing [09:12] the node doesn't show up, the machine stays on [09:18] Ah [09:18] I tried changing the NTP IP published via DHCP to the one of the maas internal address, now I seem to get further - it's stuck in a loop because network is unreacahable [09:18] Because it's only plugged into a private network atm [09:19] (I want to test this configuration, where the MAAS nodes have no public interface at all) [09:23] Ah, maas-proxy configuration was missing an allow line for the new internal subnet I added [09:24] It should really automatically configure itself to allow all the subnets the cluster controller is in [09:24] (Or perhaps the one it's managing) [09:26] I'm still not sure why exactly it's failing. Says it's trying to contact 169.254.169.254, I don't recognize that IP [09:28] haasn, that is the metadata service IP, like on AWS. MAAS provides a metadata service on IP 169.254.169.254 for an image to contact using cloud-init. The IP should be redirected to MAAS server in iptables. [09:28] Can't resolve DNS either, even though dig @cluster-controller ubuntu.com works [09:29] dweaver: Okay. It might be that the iptables setting in the node is wrong then. How can I log into it for debugging? I don't have any login details for this image [09:30] haasn, so, if the image isn't contacting cloud-init, then it didn't get an SSH key and you won't be able to log in. [09:30] The last thing I see is “BEGIN SSH HOST KEY KEYS”, followed by some keys, followed by “END SSH HOST KEY KEYS” and “cloud-init v. 0.7.5 finished” [09:31] and then cc_final_message.py[WARNING]: Used fallback datasource [09:31] And now it's doing nothing [09:31] No reboot, no new entry in maas [09:31] haasn, Yes, that's the clue - used fallback data source. [09:31] It should have used the MAAS data source, so it couldn't get to 169.254.169.254 and obtain data. [09:32] haasn, If you are using MAAS server as the router gateway, then it should "just work". [09:33] haasn, sounds like something on the network config is not working for you. [09:33] dweaver: MAAS server is not a gateway [09:33] The nodes cannot reach the internet at all [09:33] They are on a private subnet with just the maas controller [09:34] And the maas controller has a DNS, HTTP and APT proxy configured [09:35] haasn, so you haven't set a default route at all?? [09:35] dweaver: Correct [09:35] haasn, Try setting the default gateway to the MAAS server then [09:35] Then packets destined for 169.254.169.254 should get sent to the MAAS server and iptables rewrite them [09:36] Oh, the iptables is on the MAAS server, not the node [09:36] Otherwise they have nowhere to go. Yes [09:37] Is this done so that the discovery images don't need to know about the IP of the maas region controller? [09:37] Wouldn't it be easier to pass the IP as a boot-time kernel parameter? [09:39] dweaver: Works now, thanks! [09:39] haasn, MAAS is architected to be like a cloud provider, and use cloud-init on boot, which uses a metadata server for the data, like AWS, Openstack, Azure, GCE, etc. [09:39] Yeah, but I mean can't the metadata server pull the IP it's contacting from a kernel parameter [09:39] and then maas could provide its own IP as the kernel parameter when PXE booting [09:39] Then you wouldn't need the iptables “hack” [09:40] s/the metadataserver/cloud-init/ [09:42] haasn, Well, it could be done many, many different ways. [09:43] It does seem like some MAAS stuff is already pulling stuff from kernel parameters, e.g. during the commissioning process there's some URL to the maas server in there [09:43] The hack is however a standardised method of providing data to a booting image over the network and allows cloud-init to always use the same method. [09:46] fair enough [09:49] I wonder if it would be possible to have the DNS zone setting on a per-interface basis, so I can configure a different DNS zone for every subnet [10:14] Is there any package available for maas to do the api calls [11:34] hey folks! my server has two NICs: eno1 and enp0s25. I don't know why but enp0s25 is not detected in any way [11:34] it's a fresh Ubuntu 15.10 install [11:36] 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05) [11:36] 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection [11:37] 2: enp0s25: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 [11:37] 3: eno1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 [12:14] How do tags work? I'm trying to create a tag for virtual machines that looks like definition='contains(product, "KVM")', but I don't see this tag being applied to any nodes [12:15] Here is an example lshw output: https://0x0.st/XEl.txt [12:15] It says lshw:product: KVM () [12:21] also tried /product, //node[class="system"]/product [12:21] I am using tag rebuild followed by tag list to check, and there are 0 nodes every time [12:21] I swear to God that I don't see any Networks in the menu: http://i.imgur.com/wKDBvvX.png [12:21] s/tag list/tag nodes/ [12:22] Razva: JavaScript blocked or something? [12:23] nope... [12:25] I copy/pasted the QEMU example from http://en.community.dell.com/techcenter/os-applications/w/wiki/7432.using-tags-with-maas-and-juju-in-ubuntu-server-14-04-lts and changed ‘QEMU’ to ‘Red Hat’ (since that's what my VMs show up as) and it still does not work [12:30] Is the tag mechanism broken? Even examples _straight from the documentation_ simply do not work [12:37] I tried verifying my tag using the XPath evaluator at http://www.utilities-online.info/xpath/ [12:38] That one required feeding it //lshw:node[@class='system']/lshw:product for it to be evaluated correctly [12:38] (The lshw: prefixes are notably absent from the MAAS docs) [12:38] But even with the lshw: prefixes it does not work inside the actual maas [12:40] Has anybody tested the tag mechanism at all? [12:41] Is there a single working example on the internet? MAAS v1.9 [12:42] I don't think this documentation has been touched since v1.3, judging by the history [12:51] I extracted the lshw xml manually with lshw -xml > /tmp/lshw.xml and ran xmlstarlet sel -T -t -v 'contains(//node[@class="system"]/product, "KVM")' /tmp/lshw.xml and it returns ‘true’. So... [12:51] MAAS is clearly bugged, other XPath utilities find this just fine [12:52] * haasn opens a bug report [13:10] Bug #1544962 opened: MAAS tags don't find any nodes [13:17] Bug #1544962 changed: MAAS tags don't find any nodes [13:20] Bug #1544962 opened: MAAS tags don't find any nodes [13:40] folks really, I cannot boot from MAAS [14:17] haasn: on your bug, please attach the lshw output from MAAS, that will help instead of gathering the one from the system [14:17] redelmann: hi there. That doesn't really provide us much explanation. Can you expand? [14:18] roaksoax, ?? [14:19] redelmann: argh! my bad [14:19] Razva: Hi there! that doesn't really provide us with much explanation. Can you please expand? [14:19] roaksoax, dont worry, btw i fix maas-proxy, after reinstall it start working. [14:22] redelmann: cool! [14:22] roaksoax, the only change was switching public and private interfaces ( eth0 <--> eth1 ) [14:23] yeeeeeeey roaksoax is on! [14:24] roaksoax ok so here it is. I'm making a fresh Ubuntu Cloud install on a server with two nics: eno1 (internet) and enp0s25 (lan) [14:25] question 1: do I need to setup the LAN IP manually, before installing MAAS, or should I add it through the GUI after installing MAAS? [14:28] Razva when you say the LAN IP, what do you mean? [14:28] Razva: typically, you'd configure the server on the networks you want it to be, before you install MAAS [14:28] 192.168.etc [14:28] Razva: after yoiu install MAAS, you configure DNS/DHCP from MAAS' WebUI to provide services on the network you want (aka. on a Cluster Interface) [14:29] redelmann: glad you made it work! [14:29] (Y) [14:30] roaksoax can we chat on #ubuntu-server so jamespage is "on the loop"? [14:30] I'm here [14:30] ish [14:30] great [14:31] so, I'm going to install 15.10 and setup the "internet" nic, then login and setup the "lan" nic. all good 'til here? [14:31] btw thanks for your effort guys, I really really appreciate this! [14:33] Razva: right, so if you are using MAAS as the gateway, then you'd also need to setup NAT so the machines on the LAN can have internet [14:33] Razva: that being said, you should configure DHCP/DNS on the Cluster Interface connected to the LAN [14:34] got it. but before that I suppose I need to install Ubuntu Server "normally", by setting up the "net" nic, right? [14:35] I did SEVEN installs today, so I want to verify each step, just in case :D [14:35] Razva: right, so 1. install ubuntu server. 2. configure eth0 -> internet eth1 -> lan 3. install MAAS [14:35] is there any wasy way to rename a nic? they have terrible names... :| [14:35] Razva: you can manually rename the nic's if that so you wish [14:36] Razva: that is for the MAAS server [14:37] I was talking about the os nics, but whatever, I've learned the by heart :) [14:43] while it installs, let me state what I've learned about NAT. basically this is the solution: sudo iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eno1 -j MASQUERADE [14:44] where eno1 is the "net nic" and 1.0/24 is the LAN IP, which I'll set on the "lan nic" [14:44] is that correct? [14:52] roaksoax you've said "lan 3" that would be...the nat...? [15:06] Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan" 3. "Install MAAS" [15:06] Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan". 3. "Install MAAS" [15:06] ah ok :)) [15:11] roaksoax looks good? http://pastebin.com/6V7Evui5 [15:15] Razva: i think it does.. I'd need to look up iptables since haven't done them in a while :) [15:16] roger [15:16] how can I verify that the nat is ok? [15:19] I have a conceptual question with MAAS [15:19] roaksoax: http://pastebin.com/LU9FX5GP < full network setup, full network status, full iptables status [15:19] How do I map MAAS machines to physical locations, in case something is broken? [15:19] There is a “locator” functionality that can be used via IPMI to make the host's lights flash, but MAAS doesn't have a front-end for it (though I could write a wrapper script) [15:20] How do other deployments solve this problem? [15:20] e.g. say a drive fails on host whispered-news.maas. How do I find this host to replace the drive? [15:20] haasn sorry I'm a newb, but here's my funny idea: stickers! :) [15:20] Razva: So you mean, when MAAS generates its names, I attach a sticker of the generated name to the host? [15:21] YES [15:21] Then I will have an O(n) effort of finding the server given only the sticker name [15:21] What I *could* also do is, before booting the servers for the first time, add their MAC addresses to the maas DHCP with pre-defined names like host0 - host20 [15:22] haasn: you can assign them to zones.. zones is just descriptive though [15:22] That way maas would give them names I could directly resolve to locations [15:22] OR you could write a script that'll play the Mario Soundtrack if/when something fails. THAT would be epic! :D [15:22] roaksoax: So like, “zone 2-1” to mean “first host in the second rack”? One zone per node? [15:22] Or just one zone per rack and then use stickers for the rest? [15:22] That latter approach could actually work really well [15:23] haasn https://www.autoitscript.com/forum/topic/40848-beep-music-mario-bros-theme/ < there you have it! [15:24] haasn: you can use zones whichever way you'd like [15:24] haasn: you can have zone-floor1 zone-floor2 [15:24] for example [15:24] roaksoax can you please take a look on the pastebin (http://pastebin.com/LU9FX5GP)? I swear I'll not touch anything until somebody gives me the green light! :) [15:25] roaksoax: That's true, but I'm trying to minimize effort involved here - that's why I'm asking; to figure out which of the infinitely many solutions has the lowest effort [15:25] I'm mostly interested in how _real_world_ mass setups handle this [15:25] Razva: that seems to be ok [15:26] roaksoax: woohooo, next, install MAAS, as stated at http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot (steps #2 and #3)? [15:26] haasn: in real world people would use zones to place machines in different places or to group them by a comon identifier [15:26] Razva: that's correct [15:26] fingers crossed [15:27] should I apt-get upgrade or leave it like it is? [15:27] roaksoax: How do people map host names to physical server locations in the real world? [15:28] haasn that was a question I addressed 3 weeks ago. after a long discussion the answer was "as you wish". [15:28] Got any examples? I'm looking for inspiration [15:28] I've mapped them like this: town-job.continent.domain.tld [15:28] a controller in Maidenhead (europe) would be mh-controller1.eu.mydomain.com [15:29] some just use IDs, because a machine can/will change it's function in time [15:29] for example this controller can becone, in time, a compute or storage [15:29] Razva: How do you get MAAS to name them mh-controller1 instead of surprised-change or w/e? [15:29] Hard-code the MACs into its DHCPd? [15:30] Maybe I should start by describing my problem [15:30] haasn no idea, never worked with MAAS, this is my first try (I've worked with Proxmox, OpenVZ and OpenStack) [15:30] I now have 10 identical machines named only “ashamed-ducks”, “creative-winter”, “fine-jar” etc. [15:30] Right now the _only_ way I can distinguish them realistically is by looking at the MAC address, because right now that follows a clear pattern (they are test VMs, and VM 10 has :10 at the end of its MAC) [15:31] haasn are these your own machines or rented in some DC? [15:31] Razva: My own. (They're actually VM instances, but I plan to move to physical machines) [15:31] haasn: that depends per organization [15:31] Razva: And, well, that's why I'm asking this question in #maas specifically - I want to know how real-world *MAAS* deployments solve their hostnames [15:31] haasn: people in diferent organizations have different ways of doing that [15:31] Not how you could theoretically lay them out given full control :) [15:32] Bug #1545035 opened: maas-cluserd cant bind to tftp port [15:32] I could maintain my own list of MAC -> server slot locations in an internal documentation wiki [15:33] (Or maintain this association as zones) [15:34] haasn belive it or not, I would just stick them. really. [15:35] Razva: Yeah, and put them into zones based on the rack [15:35] That's what I'll most likely end up doing [15:35] Bug #1545035 changed: maas-cluserd cant bind to tftp port [15:39] haasn 9 years ago (virtualization was kinda "non-invented") I had ~2000 servers. we've tried a lot of software solutions, even wrote our own, but in the end...stickers fixed it :) [15:39] or you can try to use some fancy software, define your locations -> levels -> racks, and basically build a virtual representation of your DC [15:39] but in the end it will be very time consuming so...we dropped it [15:44] Bug #1545035 opened: maas-cluserd cant bind to tftp port [15:46] roaksoax: http://i.imgur.com/QNvGXyz.jpg < looks ok? [15:50] roaksoax I swear to God that PXE doesn't works :| [16:17] Razva: you sure machines are pxe booting ? do the logs say anything? [16:46] roaksoax can you please ket me know what log files should I tailf? [16:47] DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25 [16:47] DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25 [16:48] is this a server trying to boot? [16:48] Razva: tail -f /var/log/maas/clusterd.log [16:48] Razva: that should tell you whether a node is trying to PXE or not [16:50] 2016-02-12 17:51:32+0200 [ClusterClient,client] Cluster '4fb34068-97f2-4cd9-a7dd-444475a71de7' registered (via mh-controller1:pid=1292). [16:51] but the machine didn't boot... [16:51] Razva: ps faux | grep dhcpd ? [16:52] Razva: do you have console logs? Are you sure your machine is trying to get a dhcp address from MAAS ? [16:53] dhcpd 1782 0.0 0.0 32908 13264 ? Ss 17:54 0:00 dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid -cf /var/lib/maas/dhcpd.conf -lf /var/lib/maas/dhcp/dhcpd.leases enp0s25 [16:54] no other server online on the network, just MAAS and the "client" [16:57] Razva: right, so that seems that dhcpd is providing on enp0s25 [16:57] Razva: and your machines don't see DHCP offers [16:57] or your client [16:57] Razva: do you have stp enabled ? [16:57] no idea what is stp [16:58] https://maas.ubuntu.com/docs/install.html#configure-switches-on-the-network [16:58] Razva: ^^ [16:59] no idea, it's a manged/shared switch with a vlan [16:59] Feb 12 18:41:27 mh-controller1 dhcpd[1782]: DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25 [16:59] Feb 12 18:41:28 mh-controller1 dhcpd[1782]: DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25 [16:59] doesn't this means that "somebody" is asking for offer? [17:22] Razva: apparently [17:32] roaksoax Portfast is enabled on the switch [17:39] enp0s25 Link encap:Ethernet HWaddr 0c:c4:7a:0d:76:af [17:39] wait...so...the server is offering...dhcp to itself? === redelmann is now known as rudi|brb === rudi|brb is now known as redelmann [19:29] Bug #1545119 opened: Documentation missing for devices cli command [19:35] Bug #1545119 changed: Documentation missing for devices cli command [19:38] Bug #1545119 opened: Documentation missing for devices cli command [19:41] Bug #1545119 changed: Documentation missing for devices cli command [19:44] Bug #1545119 opened: Documentation missing for devices cli command [20:14] Razva: But if you have 2000 servers, each with a sticker on them, and you just know you have to find host X; do you then go through 2000 hosts one by one until you find X? [20:35] no, because each room has it's own "number", room is split into zones, and each zone is split into racks :) [20:36] you can have like L1R3Z6R2 [20:36] Level 1 Room 3 Zone 6 Rack 2 [20:37] if you have random names generated by MAAS, just make an XLS where host15r38 = L1R3Z6R2 and there you have it :) [20:40] XLS? [21:54] any idea why the heck all the nodes are shutting down after booting from PXE?