| mup | Bug #1274432 changed: MAAS does not make me a sandwich <MAAS:Won't Fix> <https://launchpad.net/bugs/1274432> | 00:39 |
|---|---|---|
| mup | Bug #1544795 opened: MAAS CLI should preserve the URL the user enters <MAAS:New> <https://launchpad.net/bugs/1544795> | 01:00 |
| === med_ is now known as Guest76507 | ||
| haasn | Hmm. I'm trying out a new networking setup, where maas handles DHCP - and I plugged a new node into this subnet and booted it successfully, but after it ends with “cloud-init successful” etc.; it just does nothing | 09:12 |
| haasn | the node doesn't show up, the machine stays on | 09:12 |
| haasn | Ah | 09:18 |
| haasn | I tried changing the NTP IP published via DHCP to the one of the maas internal address, now I seem to get further - it's stuck in a loop because network is unreacahable | 09:18 |
| haasn | Because it's only plugged into a private network atm | 09:18 |
| haasn | (I want to test this configuration, where the MAAS nodes have no public interface at all) | 09:19 |
| haasn | Ah, maas-proxy configuration was missing an allow line for the new internal subnet I added | 09:23 |
| haasn | It should really automatically configure itself to allow all the subnets the cluster controller is in | 09:24 |
| haasn | (Or perhaps the one it's managing) | 09:24 |
| haasn | I'm still not sure why exactly it's failing. Says it's trying to contact 169.254.169.254, I don't recognize that IP | 09:26 |
| dweaver | haasn, that is the metadata service IP, like on AWS. MAAS provides a metadata service on IP 169.254.169.254 for an image to contact using cloud-init. The IP should be redirected to MAAS server in iptables. | 09:28 |
| haasn | Can't resolve DNS either, even though dig @cluster-controller ubuntu.com works | 09:28 |
| haasn | dweaver: Okay. It might be that the iptables setting in the node is wrong then. How can I log into it for debugging? I don't have any login details for this image | 09:29 |
| dweaver | haasn, so, if the image isn't contacting cloud-init, then it didn't get an SSH key and you won't be able to log in. | 09:30 |
| haasn | The last thing I see is “BEGIN SSH HOST KEY KEYS”, followed by some keys, followed by “END SSH HOST KEY KEYS” and “cloud-init v. 0.7.5 finished” | 09:30 |
| haasn | and then cc_final_message.py[WARNING]: Used fallback datasource | 09:31 |
| haasn | And now it's doing nothing | 09:31 |
| haasn | No reboot, no new entry in maas | 09:31 |
| dweaver | haasn, Yes, that's the clue - used fallback data source. | 09:31 |
| dweaver | It should have used the MAAS data source, so it couldn't get to 169.254.169.254 and obtain data. | 09:31 |
| dweaver | haasn, If you are using MAAS server as the router gateway, then it should "just work". | 09:32 |
| dweaver | haasn, sounds like something on the network config is not working for you. | 09:33 |
| haasn | dweaver: MAAS server is not a gateway | 09:33 |
| haasn | The nodes cannot reach the internet at all | 09:33 |
| haasn | They are on a private subnet with just the maas controller | 09:33 |
| haasn | And the maas controller has a DNS, HTTP and APT proxy configured | 09:34 |
| dweaver | haasn, so you haven't set a default route at all?? | 09:35 |
| haasn | dweaver: Correct | 09:35 |
| dweaver | haasn, Try setting the default gateway to the MAAS server then | 09:35 |
| dweaver | Then packets destined for 169.254.169.254 should get sent to the MAAS server and iptables rewrite them | 09:35 |
| haasn | Oh, the iptables is on the MAAS server, not the node | 09:36 |
| dweaver | Otherwise they have nowhere to go. Yes | 09:36 |
| haasn | Is this done so that the discovery images don't need to know about the IP of the maas region controller? | 09:37 |
| haasn | Wouldn't it be easier to pass the IP as a boot-time kernel parameter? | 09:37 |
| haasn | dweaver: Works now, thanks! | 09:39 |
| dweaver | haasn, MAAS is architected to be like a cloud provider, and use cloud-init on boot, which uses a metadata server for the data, like AWS, Openstack, Azure, GCE, etc. | 09:39 |
| haasn | Yeah, but I mean can't the metadata server pull the IP it's contacting from a kernel parameter | 09:39 |
| haasn | and then maas could provide its own IP as the kernel parameter when PXE booting | 09:39 |
| haasn | Then you wouldn't need the iptables “hack” | 09:39 |
| haasn | s/the metadataserver/cloud-init/ | 09:40 |
| dweaver | haasn, Well, it could be done many, many different ways. | 09:42 |
| haasn | It does seem like some MAAS stuff is already pulling stuff from kernel parameters, e.g. during the commissioning process there's some URL to the maas server in there | 09:43 |
| dweaver | The hack is however a standardised method of providing data to a booting image over the network and allows cloud-init to always use the same method. | 09:43 |
| haasn | fair enough | 09:46 |
| haasn | I wonder if it would be possible to have the DNS zone setting on a per-interface basis, so I can configure a different DNS zone for every subnet | 09:49 |
| binoy | Is there any package available for maas to do the api calls | 10:14 |
| Razva | hey folks! my server has two NICs: eno1 and enp0s25. I don't know why but enp0s25 is not detected in any way | 11:34 |
| Razva | it's a fresh Ubuntu 15.10 install | 11:34 |
| Razva | 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05) | 11:36 |
| Razva | 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection | 11:36 |
| Razva | 2: enp0s25: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 | 11:37 |
| Razva | 3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 | 11:37 |
| haasn | How do tags work? I'm trying to create a tag for virtual machines that looks like definition='contains(product, "KVM")', but I don't see this tag being applied to any nodes | 12:14 |
| haasn | Here is an example lshw output: https://0x0.st/XEl.txt | 12:15 |
| haasn | It says lshw:product: KVM () | 12:15 |
| haasn | also tried /product, //node[class="system"]/product | 12:21 |
| haasn | I am using tag rebuild followed by tag list to check, and there are 0 nodes every time | 12:21 |
| Razva | I swear to God that I don't see any Networks in the menu: http://i.imgur.com/wKDBvvX.png | 12:21 |
| haasn | s/tag list/tag nodes/ | 12:21 |
| haasn | Razva: JavaScript blocked or something? | 12:22 |
| Razva | nope... | 12:23 |
| haasn | I copy/pasted the QEMU example from http://en.community.dell.com/techcenter/os-applications/w/wiki/7432.using-tags-with-maas-and-juju-in-ubuntu-server-14-04-lts and changed ‘QEMU’ to ‘Red Hat’ (since that's what my VMs show up as) and it still does not work | 12:25 |
| haasn | Is the tag mechanism broken? Even examples _straight from the documentation_ simply do not work | 12:30 |
| haasn | I tried verifying my tag using the XPath evaluator at http://www.utilities-online.info/xpath/ | 12:37 |
| haasn | That one required feeding it //lshw:node[@class='system']/lshw:product for it to be evaluated correctly | 12:38 |
| haasn | (The lshw: prefixes are notably absent from the MAAS docs) | 12:38 |
| haasn | But even with the lshw: prefixes it does not work inside the actual maas | 12:38 |
| haasn | Has anybody tested the tag mechanism at all? | 12:40 |
| haasn | Is there a single working example on the internet? MAAS v1.9 | 12:41 |
| haasn | I don't think this documentation has been touched since v1.3, judging by the history | 12:42 |
| haasn | I extracted the lshw xml manually with lshw -xml > /tmp/lshw.xml and ran xmlstarlet sel -T -t -v 'contains(//node[@class="system"]/product, "KVM")' /tmp/lshw.xml and it returns ‘true’. So... | 12:51 |
| haasn | MAAS is clearly bugged, other XPath utilities find this just fine | 12:51 |
| * haasn opens a bug report | 12:52 | |
| mup | Bug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962> | 13:10 |
| mup | Bug #1544962 changed: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962> | 13:17 |
| mup | Bug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962> | 13:20 |
| Razva | folks really, I cannot boot from MAAS | 13:40 |
| roaksoax | haasn: on your bug, please attach the lshw output from MAAS, that will help instead of gathering the one from the system | 14:17 |
| roaksoax | redelmann: hi there. That doesn't really provide us much explanation. Can you expand? | 14:17 |
| redelmann | roaksoax, ?? | 14:18 |
| roaksoax | redelmann: argh! my bad | 14:19 |
| roaksoax | Razva: Hi there! that doesn't really provide us with much explanation. Can you please expand? | 14:19 |
| redelmann | roaksoax, dont worry, btw i fix maas-proxy, after reinstall it start working. | 14:19 |
| roaksoax | redelmann: cool! | 14:22 |
| redelmann | roaksoax, the only change was switching public and private interfaces ( eth0 <--> eth1 ) | 14:22 |
| Razva | yeeeeeeey roaksoax is on! | 14:23 |
| Razva | roaksoax ok so here it is. I'm making a fresh Ubuntu Cloud install on a server with two nics: eno1 (internet) and enp0s25 (lan) | 14:24 |
| Razva | question 1: do I need to setup the LAN IP manually, before installing MAAS, or should I add it through the GUI after installing MAAS? | 14:25 |
| roaksoax | Razva when you say the LAN IP, what do you mean? | 14:28 |
| roaksoax | Razva: typically, you'd configure the server on the networks you want it to be, before you install MAAS | 14:28 |
| Razva | 192.168.etc | 14:28 |
| roaksoax | Razva: after yoiu install MAAS, you configure DNS/DHCP from MAAS' WebUI to provide services on the network you want (aka. on a Cluster Interface) | 14:28 |
| roaksoax | redelmann: glad you made it work! | 14:29 |
| redelmann | (Y) | 14:29 |
| Razva | roaksoax can we chat on #ubuntu-server so jamespage is "on the loop"? | 14:30 |
| jamespage | I'm here | 14:30 |
| jamespage | ish | 14:30 |
| Razva | great | 14:30 |
| Razva | so, I'm going to install 15.10 and setup the "internet" nic, then login and setup the "lan" nic. all good 'til here? | 14:31 |
| Razva | btw thanks for your effort guys, I really really appreciate this! | 14:31 |
| roaksoax | Razva: right, so if you are using MAAS as the gateway, then you'd also need to setup NAT so the machines on the LAN can have internet | 14:33 |
| roaksoax | Razva: that being said, you should configure DHCP/DNS on the Cluster Interface connected to the LAN | 14:33 |
| Razva | got it. but before that I suppose I need to install Ubuntu Server "normally", by setting up the "net" nic, right? | 14:34 |
| Razva | I did SEVEN installs today, so I want to verify each step, just in case :D | 14:35 |
| roaksoax | Razva: right, so 1. install ubuntu server. 2. configure eth0 -> internet eth1 -> lan 3. install MAAS | 14:35 |
| Razva | is there any wasy way to rename a nic? they have terrible names... :| | 14:35 |
| roaksoax | Razva: you can manually rename the nic's if that so you wish | 14:35 |
| roaksoax | Razva: that is for the MAAS server | 14:36 |
| Razva | I was talking about the os nics, but whatever, I've learned the by heart :) | 14:37 |
| Razva | while it installs, let me state what I've learned about NAT. basically this is the solution: sudo iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eno1 -j MASQUERADE | 14:43 |
| Razva | where eno1 is the "net nic" and 1.0/24 is the LAN IP, which I'll set on the "lan nic" | 14:44 |
| Razva | is that correct? | 14:44 |
| Razva | roaksoax you've said "lan 3" that would be...the nat...? | 14:52 |
| roaksoax | Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan" 3. "Install MAAS" | 15:06 |
| roaksoax | Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan". 3. "Install MAAS" | 15:06 |
| Razva | ah ok :)) | 15:06 |
| Razva | roaksoax looks good? http://pastebin.com/6V7Evui5 | 15:11 |
| roaksoax | Razva: i think it does.. I'd need to look up iptables since haven't done them in a while :) | 15:15 |
| Razva | roger | 15:16 |
| Razva | how can I verify that the nat is ok? | 15:16 |
| haasn | I have a conceptual question with MAAS | 15:19 |
| Razva | roaksoax: http://pastebin.com/LU9FX5GP < full network setup, full network status, full iptables status | 15:19 |
| haasn | How do I map MAAS machines to physical locations, in case something is broken? | 15:19 |
| haasn | There is a “locator” functionality that can be used via IPMI to make the host's lights flash, but MAAS doesn't have a front-end for it (though I could write a wrapper script) | 15:19 |
| haasn | How do other deployments solve this problem? | 15:20 |
| haasn | e.g. say a drive fails on host whispered-news.maas. How do I find this host to replace the drive? | 15:20 |
| Razva | haasn sorry I'm a newb, but here's my funny idea: stickers! :) | 15:20 |
| haasn | Razva: So you mean, when MAAS generates its names, I attach a sticker of the generated name to the host? | 15:20 |
| Razva | YES | 15:21 |
| haasn | Then I will have an O(n) effort of finding the server given only the sticker name | 15:21 |
| haasn | What I *could* also do is, before booting the servers for the first time, add their MAC addresses to the maas DHCP with pre-defined names like host0 - host20 | 15:21 |
| roaksoax | haasn: you can assign them to zones.. zones is just descriptive though | 15:22 |
| haasn | That way maas would give them names I could directly resolve to locations | 15:22 |
| Razva | OR you could write a script that'll play the Mario Soundtrack if/when something fails. THAT would be epic! :D | 15:22 |
| haasn | roaksoax: So like, “zone 2-1” to mean “first host in the second rack”? One zone per node? | 15:22 |
| haasn | Or just one zone per rack and then use stickers for the rest? | 15:22 |
| haasn | That latter approach could actually work really well | 15:22 |
| Razva | haasn https://www.autoitscript.com/forum/topic/40848-beep-music-mario-bros-theme/ < there you have it! | 15:23 |
| roaksoax | haasn: you can use zones whichever way you'd like | 15:24 |
| roaksoax | haasn: you can have zone-floor1 zone-floor2 | 15:24 |
| roaksoax | for example | 15:24 |
| Razva | roaksoax can you please take a look on the pastebin (http://pastebin.com/LU9FX5GP)? I swear I'll not touch anything until somebody gives me the green light! :) | 15:24 |
| haasn | roaksoax: That's true, but I'm trying to minimize effort involved here - that's why I'm asking; to figure out which of the infinitely many solutions has the lowest effort | 15:25 |
| haasn | I'm mostly interested in how _real_world_ mass setups handle this | 15:25 |
| roaksoax | Razva: that seems to be ok | 15:25 |
| Razva | roaksoax: woohooo, next, install MAAS, as stated at http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot (steps #2 and #3)? | 15:26 |
| roaksoax | haasn: in real world people would use zones to place machines in different places or to group them by a comon identifier | 15:26 |
| roaksoax | Razva: that's correct | 15:26 |
| Razva | fingers crossed | 15:26 |
| Razva | should I apt-get upgrade or leave it like it is? | 15:27 |
| haasn | roaksoax: How do people map host names to physical server locations in the real world? | 15:27 |
| Razva | haasn that was a question I addressed 3 weeks ago. after a long discussion the answer was "as you wish". | 15:28 |
| haasn | Got any examples? I'm looking for inspiration | 15:28 |
| Razva | I've mapped them like this: town-job.continent.domain.tld | 15:28 |
| Razva | a controller in Maidenhead (europe) would be mh-controller1.eu.mydomain.com | 15:28 |
| Razva | some just use IDs, because a machine can/will change it's function in time | 15:29 |
| Razva | for example this controller can becone, in time, a compute or storage | 15:29 |
| haasn | Razva: How do you get MAAS to name them mh-controller1 instead of surprised-change or w/e? | 15:29 |
| haasn | Hard-code the MACs into its DHCPd? | 15:29 |
| haasn | Maybe I should start by describing my problem | 15:30 |
| Razva | haasn no idea, never worked with MAAS, this is my first try (I've worked with Proxmox, OpenVZ and OpenStack) | 15:30 |
| haasn | I now have 10 identical machines named only “ashamed-ducks”, “creative-winter”, “fine-jar” etc. | 15:30 |
| haasn | Right now the _only_ way I can distinguish them realistically is by looking at the MAC address, because right now that follows a clear pattern (they are test VMs, and VM 10 has :10 at the end of its MAC) | 15:30 |
| Razva | haasn are these your own machines or rented in some DC? | 15:31 |
| haasn | Razva: My own. (They're actually VM instances, but I plan to move to physical machines) | 15:31 |
| roaksoax | haasn: that depends per organization | 15:31 |
| haasn | Razva: And, well, that's why I'm asking this question in #maas specifically - I want to know how real-world *MAAS* deployments solve their hostnames | 15:31 |
| roaksoax | haasn: people in diferent organizations have different ways of doing that | 15:31 |
| haasn | Not how you could theoretically lay them out given full control :) | 15:31 |
| mup | Bug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035> | 15:32 |
| haasn | I could maintain my own list of MAC -> server slot locations in an internal documentation wiki | 15:32 |
| haasn | (Or maintain this association as zones) | 15:33 |
| Razva | haasn belive it or not, I would just stick them. really. | 15:34 |
| haasn | Razva: Yeah, and put them into zones based on the rack | 15:35 |
| haasn | That's what I'll most likely end up doing | 15:35 |
| mup | Bug #1545035 changed: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035> | 15:35 |
| Razva | haasn 9 years ago (virtualization was kinda "non-invented") I had ~2000 servers. we've tried a lot of software solutions, even wrote our own, but in the end...stickers fixed it :) | 15:39 |
| Razva | or you can try to use some fancy software, define your locations -> levels -> racks, and basically build a virtual representation of your DC | 15:39 |
| Razva | but in the end it will be very time consuming so...we dropped it | 15:39 |
| mup | Bug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035> | 15:44 |
| Razva | roaksoax: http://i.imgur.com/QNvGXyz.jpg < looks ok? | 15:46 |
| Razva | roaksoax I swear to God that PXE doesn't works :| | 15:50 |
| roaksoax | Razva: you sure machines are pxe booting ? do the logs say anything? | 16:17 |
| Razva | roaksoax can you please ket me know what log files should I tailf? | 16:46 |
| Razva | DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25 | 16:47 |
| Razva | DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25 | 16:47 |
| Razva | is this a server trying to boot? | 16:48 |
| roaksoax | Razva: tail -f /var/log/maas/clusterd.log | 16:48 |
| roaksoax | Razva: that should tell you whether a node is trying to PXE or not | 16:48 |
| Razva | 2016-02-12 17:51:32+0200 [ClusterClient,client] Cluster '4fb34068-97f2-4cd9-a7dd-444475a71de7' registered (via mh-controller1:pid=1292). | 16:50 |
| Razva | but the machine didn't boot... | 16:51 |
| roaksoax | Razva: ps faux | grep dhcpd ? | 16:51 |
| roaksoax | Razva: do you have console logs? Are you sure your machine is trying to get a dhcp address from MAAS ? | 16:52 |
| Razva | dhcpd 1782 0.0 0.0 32908 13264 ? Ss 17:54 0:00 dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid -cf /var/lib/maas/dhcpd.conf -lf /var/lib/maas/dhcp/dhcpd.leases enp0s25 | 16:53 |
| Razva | no other server online on the network, just MAAS and the "client" | 16:54 |
| roaksoax | Razva: right, so that seems that dhcpd is providing on enp0s25 | 16:57 |
| roaksoax | Razva: and your machines don't see DHCP offers | 16:57 |
| roaksoax | or your client | 16:57 |
| roaksoax | Razva: do you have stp enabled ? | 16:57 |
| Razva | no idea what is stp | 16:57 |
| roaksoax | https://maas.ubuntu.com/docs/install.html#configure-switches-on-the-network | 16:58 |
| roaksoax | Razva: ^^ | 16:58 |
| Razva | no idea, it's a manged/shared switch with a vlan | 16:59 |
| Razva | Feb 12 18:41:27 mh-controller1 dhcpd[1782]: DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25 | 16:59 |
| Razva | Feb 12 18:41:28 mh-controller1 dhcpd[1782]: DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25 | 16:59 |
| Razva | doesn't this means that "somebody" is asking for offer? | 16:59 |
| roaksoax | Razva: apparently | 17:22 |
| Razva | roaksoax Portfast is enabled on the switch | 17:32 |
| Razva | enp0s25 Link encap:Ethernet HWaddr 0c:c4:7a:0d:76:af | 17:39 |
| Razva | wait...so...the server is offering...dhcp to itself? | 17:39 |
| === redelmann is now known as rudi|brb | ||
| === rudi|brb is now known as redelmann | ||
| mup | Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119> | 19:29 |
| mup | Bug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119> | 19:35 |
| mup | Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119> | 19:38 |
| mup | Bug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119> | 19:41 |
| mup | Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119> | 19:44 |
| haasn | Razva: But if you have 2000 servers, each with a sticker on them, and you just know you have to find host X; do you then go through 2000 hosts one by one until you find X? | 20:14 |
| Razva | no, because each room has it's own "number", room is split into zones, and each zone is split into racks :) | 20:35 |
| Razva | you can have like L1R3Z6R2 | 20:36 |
| Razva | Level 1 Room 3 Zone 6 Rack 2 | 20:36 |
| Razva | if you have random names generated by MAAS, just make an XLS where host15r38 = L1R3Z6R2 and there you have it :) | 20:37 |
| haasn | XLS? | 20:40 |
| Razva | any idea why the heck all the nodes are shutting down after booting from PXE? | 21:54 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
