mupBug #1274432 changed: MAAS does not make me a sandwich <MAAS:Won't Fix> <https://launchpad.net/bugs/1274432>00:39
mupBug #1544795 opened: MAAS CLI should preserve the URL the user enters <MAAS:New> <https://launchpad.net/bugs/1544795>01:00
=== med_ is now known as Guest76507
haasnHmm. I'm trying out a new networking setup, where maas handles DHCP - and I plugged a new node into this subnet and booted it successfully, but after it ends with “cloud-init successful” etc.; it just does nothing09:12
haasnthe node doesn't show up, the machine stays on09:12
haasnI tried changing the NTP IP published via DHCP to the one of the maas internal address, now I seem to get further - it's stuck in a loop because network is unreacahable09:18
haasnBecause it's only plugged into a private network atm09:18
haasn(I want to test this configuration, where the MAAS nodes have no public interface at all)09:19
haasnAh, maas-proxy configuration was missing an allow line for the new internal subnet I added09:23
haasnIt should really automatically configure itself to allow all the subnets the cluster controller is in09:24
haasn(Or perhaps the one it's managing)09:24
haasnI'm still not sure why exactly it's failing. Says it's trying to contact, I don't recognize that IP09:26
dweaverhaasn, that is the metadata service IP, like on AWS.  MAAS provides a metadata service on IP for an image to contact using cloud-init.  The IP should be redirected to MAAS server in iptables.09:28
haasnCan't resolve DNS either, even though dig @cluster-controller ubuntu.com works09:28
haasndweaver: Okay. It might be that the iptables setting in the node is wrong then. How can I log into it for debugging? I don't have any login details for this image09:29
dweaverhaasn, so, if the image isn't contacting cloud-init, then it didn't get an SSH key and you won't be able to log in.09:30
haasnThe last thing I see is “BEGIN SSH HOST KEY KEYS”, followed by some keys, followed by “END SSH HOST KEY KEYS” and “cloud-init v. 0.7.5 finished”09:30
haasnand then cc_final_message.py[WARNING]: Used fallback datasource09:31
haasnAnd now it's doing nothing09:31
haasnNo reboot, no new entry in maas09:31
dweaverhaasn, Yes, that's the clue - used fallback data source.09:31
dweaverIt should have used the MAAS data source, so it couldn't get to and obtain data.09:31
dweaverhaasn, If you are using MAAS server as the router gateway, then it should "just work".09:32
dweaverhaasn, sounds like something on the network config is not working for you.09:33
haasndweaver: MAAS server is not a gateway09:33
haasnThe nodes cannot reach the internet at all09:33
haasnThey are on a private subnet with just the maas controller09:33
haasnAnd the maas controller has a DNS, HTTP and APT proxy configured09:34
dweaverhaasn, so you haven't set a default route at all??09:35
haasndweaver: Correct09:35
dweaverhaasn, Try setting the default gateway to the MAAS server then09:35
dweaverThen packets destined for should get sent to the MAAS server and iptables rewrite them09:35
haasnOh, the iptables is on the MAAS server, not the node09:36
dweaverOtherwise they have nowhere to go.  Yes09:36
haasnIs this done so that the discovery images don't need to know about the IP of the maas region controller?09:37
haasnWouldn't it be easier to pass the IP as a boot-time kernel parameter?09:37
haasndweaver: Works now, thanks!09:39
dweaverhaasn, MAAS is architected to be like a cloud provider, and use cloud-init on boot, which uses a metadata server for the data, like AWS, Openstack, Azure, GCE, etc.09:39
haasnYeah, but I mean can't the metadata server pull the IP it's contacting from a kernel parameter09:39
haasnand then maas could provide its own IP as the kernel parameter when PXE booting09:39
haasnThen you wouldn't need the iptables “hack”09:39
haasns/the metadataserver/cloud-init/09:40
dweaverhaasn, Well, it could be done many, many different ways.09:42
haasnIt does seem like some MAAS stuff is already pulling stuff from kernel parameters, e.g. during the commissioning process there's some URL to the maas server in there09:43
dweaverThe hack is however a standardised method of providing data to a booting image over the network and allows cloud-init to always use the same method.09:43
haasnfair enough09:46
haasnI wonder if it would be possible to have the DNS zone setting on a per-interface basis, so I can configure a different DNS zone for every subnet09:49
binoyIs there any package available for maas to do the api calls10:14
Razvahey folks! my server has two NICs: eno1 and enp0s25. I don't know why but enp0s25 is not detected in any way11:34
Razvait's a fresh Ubuntu 15.10 install11:34
Razva00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05)11:36
Razva04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection11:36
Razva2: enp0s25: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 100011:37
Razva3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 100011:37
haasnHow do tags work? I'm trying to create a tag for virtual machines that looks like definition='contains(product, "KVM")', but I don't see this tag being applied to any nodes12:14
haasnHere is an example lshw output: https://0x0.st/XEl.txt12:15
haasnIt says lshw:product: KVM ()12:15
haasnalso tried /product, //node[class="system"]/product12:21
haasnI am using tag rebuild followed by tag list to check, and there are 0 nodes every time12:21
RazvaI swear to God that I don't see any Networks in the menu: http://i.imgur.com/wKDBvvX.png12:21
haasns/tag list/tag nodes/12:21
haasnRazva: JavaScript blocked or something?12:22
haasnI copy/pasted the QEMU example from http://en.community.dell.com/techcenter/os-applications/w/wiki/7432.using-tags-with-maas-and-juju-in-ubuntu-server-14-04-lts and changed ‘QEMU’ to ‘Red Hat’ (since that's what my VMs show up as) and it still does not work12:25
haasnIs the tag mechanism broken? Even examples _straight from the documentation_ simply do not work12:30
haasnI tried verifying my tag using the XPath evaluator at http://www.utilities-online.info/xpath/12:37
haasnThat one required feeding it //lshw:node[@class='system']/lshw:product for it to be evaluated correctly12:38
haasn(The lshw: prefixes are notably absent from the MAAS docs)12:38
haasnBut even with the lshw: prefixes it does not work inside the actual maas12:38
haasnHas anybody tested the tag mechanism at all?12:40
haasnIs there a single working example on the internet? MAAS v1.912:41
haasnI don't think this documentation has been touched since v1.3, judging by the history12:42
haasnI extracted the lshw xml manually with lshw -xml > /tmp/lshw.xml and ran xmlstarlet sel -T -t -v 'contains(//node[@class="system"]/product, "KVM")' /tmp/lshw.xml and it returns ‘true’. So...12:51
haasnMAAS is clearly bugged, other XPath utilities find this just fine12:51
* haasn opens a bug report12:52
mupBug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>13:10
mupBug #1544962 changed: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>13:17
mupBug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>13:20
Razvafolks really, I cannot boot from MAAS13:40
roaksoaxhaasn: on your bug, please attach the lshw output from MAAS, that will help instead of gathering the one from the system14:17
roaksoaxredelmann: hi there. That doesn't really provide us much explanation. Can you expand?14:17
redelmannroaksoax, ??14:18
roaksoaxredelmann: argh! my bad14:19
roaksoaxRazva: Hi there! that doesn't really provide us with much explanation. Can you please expand?14:19
redelmannroaksoax, dont worry, btw i fix maas-proxy, after reinstall it start working.14:19
roaksoaxredelmann: cool!14:22
redelmannroaksoax, the only change was switching public and private interfaces ( eth0 <--> eth1 )14:22
Razvayeeeeeeey roaksoax is on!14:23
Razvaroaksoax ok so here it is. I'm making a fresh Ubuntu Cloud install on a server with two nics: eno1 (internet) and enp0s25 (lan)14:24
Razvaquestion 1: do I need to setup the LAN IP manually, before installing MAAS, or should I add it through the GUI after installing MAAS?14:25
roaksoaxRazva when you say the LAN IP, what do you mean?14:28
roaksoaxRazva: typically, you'd configure the server on the networks you want it to be, before you install MAAS14:28
roaksoaxRazva: after yoiu install MAAS, you configure DNS/DHCP from MAAS' WebUI to provide services on the network you want (aka. on a Cluster Interface)14:28
roaksoaxredelmann: glad you made it work!14:29
Razvaroaksoax can we chat on #ubuntu-server so jamespage is "on the loop"?14:30
jamespageI'm here14:30
Razvaso, I'm going to install 15.10 and setup the "internet" nic, then login and setup the "lan" nic. all good 'til here?14:31
Razvabtw thanks for your effort guys, I really really appreciate this!14:31
roaksoaxRazva: right, so if you are using MAAS as the gateway, then you'd also need to setup NAT so the machines on the LAN can have internet14:33
roaksoaxRazva: that being said, you should configure DHCP/DNS on the Cluster Interface connected to the LAN14:33
Razvagot it. but before that I suppose I need to install Ubuntu Server "normally", by setting up the "net" nic, right?14:34
RazvaI did SEVEN installs today, so I want to verify each step, just in case :D14:35
roaksoaxRazva: right, so 1. install ubuntu server. 2. configure eth0 -> internet eth1 -> lan 3. install MAAS14:35
Razvais there any wasy way to rename a nic? they have terrible names... :|14:35
roaksoaxRazva: you can manually rename the nic's if that so you wish14:35
roaksoaxRazva: that is for the MAAS server14:36
RazvaI was talking about the os nics, but whatever, I've learned the by heart :)14:37
Razvawhile it installs, let me state what I've learned about NAT. basically this is the solution: sudo iptables -t nat -A POSTROUTING -s -o eno1 -j MASQUERADE14:43
Razvawhere eno1 is the "net nic" and 1.0/24 is the LAN IP, which I'll set on the "lan nic"14:44
Razvais that correct?14:44
Razvaroaksoax you've said "lan 3" that would be...the nat...?14:52
roaksoaxRazva: Sorry, I menat 2. "configure eth0>internet, eth1>lan" 3. "Install MAAS"15:06
roaksoaxRazva: Sorry, I menat 2. "configure eth0>internet, eth1>lan". 3. "Install MAAS"15:06
Razvaah ok :))15:06
Razvaroaksoax looks good? http://pastebin.com/6V7Evui515:11
roaksoaxRazva: i think it does.. I'd need to look up iptables since haven't done them in a while :)15:15
Razvahow can I verify that the nat is ok?15:16
haasnI have a conceptual question with MAAS15:19
Razvaroaksoax: http://pastebin.com/LU9FX5GP < full network setup, full network status, full iptables status15:19
haasnHow do I map MAAS machines to physical locations, in case something is broken?15:19
haasnThere is a “locator” functionality that can be used via IPMI to make the host's lights flash, but MAAS doesn't have a front-end for it (though I could write a wrapper script)15:19
haasnHow do other deployments solve this problem?15:20
haasne.g. say a drive fails on host whispered-news.maas. How do I find this host to replace the drive?15:20
Razvahaasn sorry I'm a newb, but here's my funny idea: stickers! :)15:20
haasnRazva: So you mean, when MAAS generates its names, I attach a sticker of the generated name to the host?15:20
haasnThen I will have an O(n) effort of finding the server given only the sticker name15:21
haasnWhat I *could* also do is, before booting the servers for the first time, add their MAC addresses to the maas DHCP with pre-defined names like host0 - host2015:21
roaksoaxhaasn: you can assign them to zones.. zones is just descriptive though15:22
haasnThat way maas would give them names I could directly resolve to locations15:22
RazvaOR you could write a script that'll play the Mario Soundtrack if/when something fails. THAT would be epic! :D15:22
haasnroaksoax: So like, “zone 2-1” to mean “first host in the second rack”? One zone per node?15:22
haasnOr just one zone per rack and then use stickers for the rest?15:22
haasnThat latter approach could actually work really well15:22
Razvahaasn https://www.autoitscript.com/forum/topic/40848-beep-music-mario-bros-theme/ < there you have it!15:23
roaksoaxhaasn: you can use zones whichever way you'd like15:24
roaksoaxhaasn: you can have zone-floor1 zone-floor215:24
roaksoaxfor example15:24
Razvaroaksoax can you please take a look on the pastebin (http://pastebin.com/LU9FX5GP)? I swear I'll not touch anything until somebody gives me the green light! :)15:24
haasnroaksoax: That's true, but I'm trying to minimize effort involved here - that's why I'm asking; to figure out which of the infinitely many solutions has the lowest effort15:25
haasnI'm mostly interested in how _real_world_ mass setups handle this15:25
roaksoaxRazva: that seems to be ok15:25
Razvaroaksoax: woohooo, next, install MAAS, as stated at http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot (steps #2 and #3)?15:26
roaksoaxhaasn: in real world people would use zones to place machines in different places or to group them by a comon identifier15:26
roaksoaxRazva: that's correct15:26
Razvafingers crossed15:26
Razvashould I apt-get upgrade or leave it like it is?15:27
haasnroaksoax: How do people map host names to physical server locations in the real world?15:27
Razvahaasn that was a question I addressed 3 weeks ago. after a long discussion the answer was "as you wish".15:28
haasnGot any examples? I'm looking for inspiration15:28
RazvaI've mapped them like this: town-job.continent.domain.tld15:28
Razvaa controller in Maidenhead (europe) would be mh-controller1.eu.mydomain.com15:28
Razvasome just use IDs, because a machine can/will change it's function in time15:29
Razvafor example this controller can becone, in time, a compute or storage15:29
haasnRazva: How do you get MAAS to name them mh-controller1 instead of surprised-change or w/e?15:29
haasnHard-code the MACs into its DHCPd?15:29
haasnMaybe I should start by describing my problem15:30
Razvahaasn no idea, never worked with MAAS, this is my first try (I've worked with Proxmox, OpenVZ and OpenStack)15:30
haasnI now have 10 identical machines named only “ashamed-ducks”, “creative-winter”, “fine-jar” etc.15:30
haasnRight now the _only_ way I can distinguish them realistically is by looking at the MAC address, because right now that follows a clear pattern (they are test VMs, and VM 10 has :10 at the end of its MAC)15:30
Razvahaasn are these your own machines or rented in some DC?15:31
haasnRazva: My own. (They're actually VM instances, but I plan to move to physical machines)15:31
roaksoaxhaasn: that depends per organization15:31
haasnRazva: And, well, that's why I'm asking this question in #maas specifically - I want to know how real-world *MAAS* deployments solve their hostnames15:31
roaksoaxhaasn: people in diferent organizations have different ways of doing that15:31
haasnNot how you could theoretically lay them out given full control :)15:31
mupBug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>15:32
haasnI could maintain my own list of MAC -> server slot locations in an internal documentation wiki15:32
haasn(Or maintain this association as zones)15:33
Razvahaasn belive it or not, I would just stick them. really.15:34
haasnRazva: Yeah, and put them into zones based on the rack15:35
haasnThat's what I'll most likely end up doing15:35
mupBug #1545035 changed: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>15:35
Razvahaasn 9 years ago (virtualization was kinda "non-invented") I had ~2000 servers. we've tried a lot of software solutions, even wrote our own, but in the end...stickers fixed it :)15:39
Razvaor you can try to use some fancy software, define your locations -> levels -> racks, and basically build a virtual representation of your DC15:39
Razvabut in the end it will be very time consuming so...we dropped it15:39
mupBug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>15:44
Razvaroaksoax: http://i.imgur.com/QNvGXyz.jpg < looks ok?15:46
Razvaroaksoax I swear to God that PXE doesn't works :|15:50
roaksoaxRazva: you sure machines are pxe booting ? do the logs say anything?16:17
Razvaroaksoax can you please ket me know what log files should I tailf?16:46
RazvaDHCPOFFER on to 0c:c4:7a:0d:76:af via enp0s2516:47
RazvaDHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s2516:47
Razvais this a server trying to boot?16:48
roaksoaxRazva: tail -f /var/log/maas/clusterd.log16:48
roaksoaxRazva: that should tell you whether a node is trying to PXE or not16:48
Razva2016-02-12 17:51:32+0200 [ClusterClient,client] Cluster '4fb34068-97f2-4cd9-a7dd-444475a71de7' registered (via mh-controller1:pid=1292).16:50
Razvabut the machine didn't boot...16:51
roaksoaxRazva: ps faux | grep dhcpd ?16:51
roaksoaxRazva: do you have console logs? Are you sure your machine is trying to get a dhcp address from MAAS ?16:52
Razvadhcpd     1782  0.0  0.0  32908 13264 ?        Ss   17:54   0:00 dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid -cf /var/lib/maas/dhcpd.conf -lf /var/lib/maas/dhcp/dhcpd.leases enp0s2516:53
Razvano other server online on the network, just MAAS and the "client"16:54
roaksoaxRazva: right, so that seems that dhcpd is providing on enp0s2516:57
roaksoaxRazva: and your machines don't see DHCP offers16:57
roaksoaxor your client16:57
roaksoaxRazva: do you have stp enabled ?16:57
Razvano idea what is stp16:57
roaksoaxRazva: ^^16:58
Razvano idea, it's a manged/shared switch with a vlan16:59
RazvaFeb 12 18:41:27 mh-controller1 dhcpd[1782]: DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s2516:59
RazvaFeb 12 18:41:28 mh-controller1 dhcpd[1782]: DHCPOFFER on to 0c:c4:7a:0d:76:af via enp0s2516:59
Razvadoesn't this means that "somebody" is asking for offer?16:59
roaksoaxRazva: apparently17:22
Razvaroaksoax Portfast is enabled on the switch17:32
Razvaenp0s25   Link encap:Ethernet  HWaddr 0c:c4:7a:0d:76:af17:39
Razvawait...so...the server is offering...dhcp to itself?17:39
=== redelmann is now known as rudi|brb
=== rudi|brb is now known as redelmann
mupBug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>19:29
mupBug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>19:35
mupBug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>19:38
mupBug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>19:41
mupBug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>19:44
haasnRazva: But if you have 2000 servers, each with a sticker on them, and you just know you have to find host X; do you then go through 2000 hosts one by one until you find X?20:14
Razvano, because each room has it's own "number", room is split into zones, and each zone is split into racks :)20:35
Razvayou can have like L1R3Z6R220:36
RazvaLevel 1 Room 3 Zone 6 Rack 220:36
Razvaif you have random names generated by MAAS, just make an XLS where host15r38 = L1R3Z6R2 and there you have it :)20:37
Razvaany idea why the heck all the nodes are shutting down after booting from PXE?21:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!