| === jam1 is now known as jam | ||
| Budgie^Smore | anyone around fam with maas-enlist error codes? I am getting return code 8 | 05:01 |
|---|---|---|
| kklimonda | @pmatulis: better docs for networking side (fabrics, spaces, subnets, vlans) and how it all fits together; better docs on image customization - both with curtin (late and early commands), as well as preparing own images; actually, going back to networking, I think I've configured bonding using some random blog post; I haven't even touched more advanced partitioning, but at this point I'm assuming it's also going to be underdocumented. | 06:31 |
| cnf | morning | 10:56 |
| cnf | anyone around that can help me underatand the network requirements etc for maas? | 10:56 |
| brendand | cnf, do you have any specific questions? | 10:58 |
| cnf | brendand: well, maybe, i'm trying to understand what is needed to get maas working | 10:59 |
| cnf | i have machines that are connected with a 2 x 10G LAG | 10:59 |
| cnf | and i'm not sure how to get PXE on that, and what management connectivity is needed to machines after pxe boot etc | 10:59 |
| cnf | a point to documentation that explains this would help, for now | 11:00 |
| brendand | cnf, try reading https://docs.ubuntu.com/maas/2.0/en/ thoroughly | 11:01 |
| cnf | yes, i have that open | 11:02 |
| cnf | have had for a while | 11:02 |
| brendand | cnf, for the nodes, the pxe boot config is really hw dependent | 11:02 |
| brendand | cnf, so you'll have to consult the machines documentation for that | 11:03 |
| brendand | cnf, and the power management is again machine dependent, so it really depends what kind of hw it is | 11:03 |
| cnf | power management? | 11:06 |
| brendand | cnf, if you make the machines pxe boot then maas might auto-detect the power parameters | 11:06 |
| brendand | cnf, so start there | 11:06 |
| cnf | i don't understand, where does power management come in? | 11:07 |
| brendand | cnf, so maas can turn the machine on and off | 11:07 |
| cnf | but that doesn't explain how to do the pxe booting on my network | 11:07 |
| cnf | or what connectivity is required on a network level | 11:08 |
| brendand | cnf, one machine needs to function as the rack/region controller. other ones can be nodes. they must all be on the same vlan | 11:09 |
| brendand | cnf, maas does assume good familiarity with configuring a network | 11:09 |
| cnf | yes, but i don't know what is required | 11:10 |
| cnf | and I can't find the requirements documented | 11:12 |
| cnf | hmz | 11:55 |
| DesktopMan | cnf: once you install and configure the rack controller it supplies and pxe. there isn't much of a network requirement for that part, chuck them all on a switch and you're good | 12:03 |
| cnf | DesktopMan: well, i'm not good | 12:03 |
| cnf | it's a switch that's already in use for a LOT of other things | 12:03 |
| DesktopMan | the rack controller also does nat for you so the nodes get wan access through it | 12:03 |
| DesktopMan | then you should pxe boot on a vlan | 12:03 |
| DesktopMan | to isolate it | 12:03 |
| cnf | and how do i do that on a LAG link? | 12:03 |
| DesktopMan | server bios, under boot settings | 12:04 |
| DesktopMan | choose the nic for pxe there, and you can also set a vlan (if it's supported) | 12:04 |
| cnf | yes, but no LACP, afaik | 12:04 |
| cnf | anyway, i'm looking into getting that sorted on the juniper side | 12:04 |
| cnf | DesktopMan: so MAAS needs the servers to be in an untagged vlan, a single one for pxe and management? | 12:05 |
| DesktopMan | no, you can set the dhcp / pxe server for a vlan if you want | 12:05 |
| cnf | so how do you configure the vlan in the os once it is booted? | 12:05 |
| cnf | well, s/do you/does maas/ ? | 12:05 |
| DesktopMan | I've only used the maas web interface for that, but you go to subnets and click the vlan you want pxe on | 12:06 |
| DesktopMan | then configure the dhcp server for that vlan | 12:06 |
| cnf | DesktopMan: right, that's on the management side | 12:06 |
| cnf | that makes sense | 12:06 |
| cnf | on the metal side, maas needs to first pxe boot, and then login to the installed OS to change config, right? | 12:07 |
| DesktopMan | after your machines have booted from pxe you commision them (get them ready for use), then deploy them. after deployment (os install) you can configure the rest of the network interfaces from maas | 12:08 |
| DesktopMan | or you can log in and do it yourself manually | 12:08 |
| DesktopMan | when you commision you can select to allow ssh access, if you do it installs the ssh keys for the current maas user. | 12:09 |
| cnf | well, to commission them, maas needs network connectivity to them, right? | 12:10 |
| DesktopMan | yes, but it uses the pxe network for that | 12:11 |
| cnf | but that means that to the metal, the MAAS vlan needs to be untagged | 12:11 |
| cnf | k, so my biggest problem is getting pxe to work on a LACP bond | 12:12 |
| DesktopMan | yeah that part I'm not sure about. is the lacp bond hardware configured? | 12:13 |
| DesktopMan | or is it done at the OS level | 12:13 |
| cnf | well, on the metal on an OS level | 12:14 |
| cnf | so during boot, there is no LACP yet | 12:14 |
| cnf | but the switch does expect it | 12:14 |
| cnf | we found something that might work on the qfabrics | 12:16 |
| cnf | where you can force one of the members to be up, even without lacp | 12:16 |
| cnf | going to test if that works | 12:16 |
| pmatulis | kklimonda, thanks for the feedback | 13:43 |
| cnf | does maas work on a V6 network? | 14:05 |
| roaksoax | cnf: yes | 14:07 |
| roaksoax | it does | 14:07 |
| roaksoax | latest version of MAAS for better support though | 14:07 |
| pmatulis | kklimonda, any more specific info can be put here: https://github.com/CanonicalLtd/maas-docs/issues/new | 14:08 |
| cnf | roaksoax: hmm, docs seem to indicate you still need ipv4 present, though? | 14:23 |
| cnf | https://docs.ubuntu.com/maas/2.1/en/installconfig-network-ipv6 | 14:23 |
| roaksoax | cnf: not really, no | 14:25 |
| mup | Bug #1408820 changed: [docs] configure.html#preseed has out of date link <doc> <trivial> <MAAS:Invalid> <https://launchpad.net/bugs/1408820> | 16:03 |
| mup | Bug #1666852 changed: apt IGN messages timeout very high causes failed commissioning and deployment <MAAS:Invalid> <https://launchpad.net/bugs/1666852> | 16:03 |
| mup | Bug #1668315 opened: [2.1+] Missing documentation about preseeds in new documentation <doc> <MAAS:Triaged> <https://launchpad.net/bugs/1668315> | 16:03 |
| cnf | hmz | 16:03 |
| cnf | damn thing won't pxe boot | 16:07 |
| cnf | how do people use MAAS on machines on a LAG? | 16:13 |
| kklimonda | my machines have separate management interface that I use to PXE them | 16:14 |
| kklimonda | anyway, is it LAG or LACP? | 16:16 |
| cnf | LACP | 16:18 |
| cnf | 2 x 10G | 16:18 |
| kklimonda | ok, that's tricky - some switches allow you to configure LACP interface to bring the link up as a normal interface until LACP control packet is received | 16:20 |
| kklimonda | but IIRC that's not part of a standard | 16:20 |
| cnf | trying to find how to do that on a juniper | 16:21 |
| cnf | I was hoping http://www.juniper.net/techpubs/en_US/junos10.3/topics/reference/configuration-statement/force-up-edit-interfaces-ex-series.html would help, but it doesn't seem to | 16:22 |
| kklimonda | hmm, afaik force-up should do the trick | 16:22 |
| cnf | doesn't seem to | 16:23 |
| kklimonda | it can be configured only on a single interface from bundle afair, are you sure you are configuring the correct one? | 16:23 |
| cnf | it was set on 1 interface, and both of them are set to PXE boot on the DL380 | 16:27 |
| kklimonda | those are 10gig interfaces, are you sure your machine is booting from 10gig and not 1gig only? just checking, I've seen servers which had to have 10gig PXE enabled in BIOS | 16:28 |
| cnf | yep | 16:29 |
| cnf | checked the bios, made sure the 10G ones are at the top | 16:29 |
| cnf | made sure it where the ipv4 ones, as well | 16:29 |
| kklimonda | I'm testing MAAS 2.2 beta (from ppa:maas/next) and it's throwing a lot of >>django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key"<< in logs, any idea if that has been reported? | 16:35 |
| capncrunch4me | is there a way in MAAS to have it load BTRFS instead of ext4 when a machine is deployed? | 16:35 |
| kklimonda | perhaps a custom image? | 16:36 |
| capncrunch4me | it strikes me as a custom storage layout | 16:36 |
| cnf | hmm, dammit :/ | 16:36 |
| kklimonda | good point | 16:37 |
| kklimonda | cnf: you don't have a separate 1gbe interface? | 16:37 |
| cnf | plenty of interfaces | 16:37 |
| kklimonda | none connected? ;) | 16:38 |
| cnf | not really switches to connect them to, though | 16:38 |
| cnf | everything is 10G LACP here | 16:38 |
| cnf | i could add a switch, but then that switch would not have the right vlan | 16:38 |
| cnf | and then i'd have 1G for a few machines, at best | 16:39 |
| cnf | hmm, all searches do indeed point to force-up | 16:47 |
| cnf | but that isn't working, it seems | 16:47 |
| kklimonda | what switch model do you have? | 16:47 |
| cnf | juniper QFX3500 | 16:48 |
| cnf | so if I enable dhcp in MAAS, do i need to start the service manually on the linux machine it is on? | 16:54 |
| roaksoax | cnf: No. MAAS will configure / start dhcp. We dont use the "systems" daemon. MAAS runs its own isc-dhcp daemon | 16:55 |
| cnf | roaksoax: ok, but i should see a dhcpd process, right? | 16:55 |
| cnf | or is it called something else? | 16:56 |
| cnf | yeah, maas didn't start the dhcpd it seems... | 17:01 |
| cnf | wow | 17:01 |
| kklimonda | roaksoax: is it running actual isc-dhcp-server, or does it have its own DHCP implementation? | 17:01 |
| kklimonda | roaksoax: at least on 2.2 port 69 is boung to by some python3 process | 17:01 |
| kklimonda | ok, my bad | 17:02 |
| mimizone | hi. what's the recommended method replacing preseed scripts in maas 2.1? I need to modify the rt_tables/ip routes during deployment. | 17:02 |
| mup | Bug #1668329 opened: [2.2, 2.1] ssh key comment contains unicode character - node deployment fails <MAAS:Triaged> <https://launchpad.net/bugs/1668329> | 17:03 |
| roaksoax | kklimonda: it is running isc-dhcp | 17:04 |
| roaksoax | kklimonda: as maas-dhcp | 17:04 |
| cnf | dhcpd 2364 0.0 0.1 35848 13520 ? Ss 18:00 0:00 dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid -cf /var/lib/maas/dhcpd.conf -lf /var/lib/maas/dhcp/dhcpd.leases ens192 | 17:04 |
| cnf | for me | 17:04 |
| roaksoax | that's the one | 17:04 |
| cnf | which wasn't started when I enabled dhcp in the UI | 17:04 |
| kklimonda | yeah, sorry - I've done off-by-one mistake - 67/68 is dhcp, 69 is tftp | 17:05 |
| kklimonda | should MAAS be creating DNS entries for machines' BMC IP addresses? | 17:10 |
| roaksoax | kklimonda: not for BMC's | 17:11 |
| brendand | roaksoax, we had troubles before with bmc addresses being used for maas - in the ci we create an ip range to cover the bmc addresses | 17:14 |
| roaksoax | brendand: was that the source of the issue really ? | 17:14 |
| brendand | roaksoax, creating the range was the solution, not the cause | 17:15 |
| brendand | roaksoax, just wondering if it's relevant to what kklimonda said | 17:16 |
| roaksoax | brendand: in our case because we do a bad design thing to share the subnet of the bmc's | 17:18 |
| kklimonda | it's not a bad design if it works! | 17:18 |
| kklimonda | @brendand: could this be the cause of MAAS throwing IntegrityErrors? | 17:20 |
| brendand | roaksoax, i thought the bmcs were on a different subnet? 10.245.143.0/21 as opposed to 10.245.136.0/21? | 17:20 |
| * roaksoax is distracted, otp | 17:22 | |
| mimizone | anyone on my question regarding modifying the /etc/ip/rt_tables and ip routes during deployment? where can I customized provisioning scripts? It seems preseed is not the recommended way anymore in maas 2.1 | 17:42 |
| cnf | ok, i see dhcp requests coming in | 17:59 |
| cnf | but maas isn't responding | 17:59 |
| mimizone | is there still the concept of curtin user_data/custom scripts in maas 2.1? anyone has a pointer to some documentation by any chance? | 18:02 |
| cnf | hmm, and ilo4 support seems broken | 18:13 |
| cnf | it runs the wrong IPMI command | 18:15 |
| stormmore | so I have been trying to use maas-enlist but it doesn't appear to be working for 2.1, but I am not getting any useful errors to be able to share :-/ | 18:19 |
| stormmore | when I hit enter it hangs for a second and then the prompt returns with zero other output | 18:20 |
| mimizone | any familiar with the /etc/maas/preseeds/curtinXXXXX files? I would like to add a few customization in there. | 19:13 |
| pmatulis | mimizone, this will soon be documented better. lemme know if this is sufficient for your use case. https://goo.gl/tqZiKB | 19:25 |
| mimizone | pmatulis: thanks. I'll check it out and come back to you in a bit | 19:26 |
| pmatulis | mimizone, thanks | 19:27 |
| mimizone | pmatulis: it is very similar to the document for 1.9 basically, right? | 19:28 |
| pmatulis | mimizone, yeah | 19:28 |
| pmatulis | mimizone, it would be great if you could add any deficiencies as a docs bug - https://github.com/CanonicalLtd/maas-docs/issues/new | 19:29 |
| mimizone | pmatulis: sure. I'll see what I can express clearly that would be useful to all. | 19:30 |
| pmatulis | perfect | 19:31 |
| mimizone | pmatulis: now I am hacking the curtin_userdata file until I get my commands right. but so far, I break the deployment. | 19:31 |
| mimizone | I am trying to run a "sed" command on the /etc/network/interfaces to fix a bug in the file that was generated. | 19:32 |
| mimizone | would you know anything in how I can add a command properly in the curtin_userdata? | 19:32 |
| mimizone | here what I do so far | 19:32 |
| mimizone | network_commands: | 19:32 |
| mimizone | maas_fix_00: ["bash","-c", "sed s/'|| truepost-up route add'/'|| true\npost-up route add'/g", $OUTPUT_INTERFACES] | 19:32 |
| stormmore | hmmm I am trying to use maas-enlist to enlist some virtual nodes into maas but I keep getting failures | 19:34 |
| stormmore | I keep getting return code 8 but I don't know what that means and there isn't any other visible error, any suggestions on what logs might provide some insight into what is going on? | 19:36 |
| stormmore | it is weird, even pxe booting the machine is failing to enlist :-/ | 19:57 |
| roaksoax | stormmore: how is it failing ? | 20:08 |
| roaksoax | stormmore: try /var/log/maas/rsyslog/maas-enlist* | 20:08 |
| stormmore | I am having a hard time seeing why it is failing, scrolls off screen way too fast. Trying to get in didn't work before the system shutdown :-/ | 20:10 |
| stormmore | I don't see that log roaksoax :-/ | 20:14 |
| roaksoax | stormmore: ok, try this, is /etc/maas/rackd.conf point to an IP address or localhost ? | 20:15 |
| stormmore | on the maas server? | 20:15 |
| stormmore | if so, then yeah it is pointing to the right IP | 20:16 |
| roaksoax | stormmore: any error logs in /var/log/maas/*.log ? | 20:17 |
| stormmore | roaksoax, https://gist.github.com/cm-graham/62ed001b9158f8dddbc1a521cce8abc4 is what I am seeing in rackd.log | 20:23 |
| Siva_ | Hi | 20:23 |
| stormmore | am I seeing what I think is a dns issue? | 20:23 |
| Siva_ | I installed a VM to be managed by MAAS | 20:23 |
| roaksoax | stormmore: could be. seems MAAS is trying to contact the region and it is failing | 20:24 |
| Siva_ | After adding the 'Power address' like qemu+ssh://you@<host-where-vm-is>/system | 20:24 |
| stormmore | starting to look like it :-/ need to update the nameservers on the maas server :-/ | 20:25 |
| Siva_ | When I click the 'check now' for Power.. i get the following error | 20:26 |
| Siva_ | No rack controllers can access the BMC of node: juju-controller | 20:26 |
| Siva_ | maas log shows this : Feb 27 19:27:31 localhost maas.power: [error] juju-controller: Power state could not be queried: Failed to login to virsh console | 20:26 |
| Siva_ | Any help on this is much appreciated | 20:26 |
| roaksoax | Siva_: seems like the rack controller cannot resolve <juju-controller> | 20:26 |
| Siva_ | @roaksoax, what does that mean? Not clear | 20:27 |
| roaksoax | Siva_: that the rack controller in maas cannnot resolve a node called <juju-controller> | 20:27 |
| roaksoax | Siva_: yseems you are doing qemu+ssh://you@juju-controller/system | 20:28 |
| Siva_ | I my case, I explicitly sepcify the ipaddress as qemu+ssh://contrail@10.84.33.1/system | 20:28 |
| roaksoax | Siva_: ah gotcha now | 20:28 |
| roaksoax | Siva_: can you ping 10.84.33.1 from where the rack controller is running ? | 20:28 |
| Siva_ | yes | 20:28 |
| roaksoax | Siva_: seems that no rack controller can contact that IP address | 20:28 |
| Siva_ | I am able to ping it | 20:29 |
| roaksoax | bladernr: ^^ | 20:29 |
| roaksoax | err | 20:29 |
| roaksoax | blake_r: ^^ | 20:29 |
| roaksoax | blake_r: seems that the rack cannot reach a BMC that's pingable | 20:30 |
| Siva_ | the ip address 10.84.33.1 is the bastion host ipaddress where the vm is launched | 20:30 |
| stormmore | can you ssh to the IP? | 20:31 |
| Siva_ | @stormmore, yes I am able ssh into that from my desktop | 20:31 |
| roaksoax | Siva_: but from the MAAS server ? | 20:32 |
| Siva_ | yes, from the maas server. It ask for a password ... when entered it goes through | 20:33 |
| Siva_ | @roaksoax, does the maas log error give you any clue? | 20:35 |
| Siva_ | @roaksoax, does the maas log error pasted above give you any clue? | 20:35 |
| stormmore | wondering if it is cause it asks for a password or the maas server isn't using the right key | 20:36 |
| Siva_ | @stormore, the ipaddress is the management ip address of the bastion host | 20:38 |
| Siva_ | How do I verify if maas server is using the correct key? | 20:38 |
| blake_r | Siva_: how is the SSH handled? | 20:40 |
| blake_r | Siva_: using ssh keys or password? | 20:40 |
| blake_r | Siva_: it its by SSH key then you need to add the SSH key to the maas user | 20:40 |
| blake_r | Siva_: if the private is under the MAAS user then it cannot authenticate | 20:41 |
| blake_r | Siva_: using password is much easier to setup | 20:41 |
| Siva_ | @blake_r, I am using the same 'username' and 'password' here in the 'Power address' field here as I would when 'ssh'ing into the bastion host from my desktop | 20:43 |
| Siva_ | Does that answer your question? | 20:43 |
| blake_r | Siva_: did you place the password in the Power pass field? | 20:43 |
| Siva_ | @blake_r, nope | 20:44 |
| blake_r | Siva_: the password must go into that field | 20:44 |
| Siva_ | @blake_r, I am trying it now | 20:45 |
| stormmore | looks like blake_r is taking you down the path that I was thinking... if you are using user/pass then you need to provide both for it to work | 20:47 |
| Siva_ | @blake_r, that worked | 20:53 |
| blake_r | Siva_: great! | 20:53 |
| Siva_ | Pls help me understand, why do we need this optional field. I remember setting this up around 6 months back without the password field | 20:53 |
| Siva_ | What should be done to make this work without the password field | 20:53 |
| Siva_ | @blake_r, thanks for your help. Kindly let me know your input on the above question | 20:54 |
| blake_r | Siva_: if you don't want a password field, then you need to create a ssh key in the "maas" user account | 20:55 |
| blake_r | Siva_: then push the public key to the virsh server | 20:55 |
| blake_r | Siva_: or copy your private ssh key to the maas user | 20:55 |
| blake_r | Siva_: all MAAS services run as the "maas" user, it needs access to the SSH key to authenticate | 20:56 |
| stormmore | Siva_, basically what blake_r said. it is worth noting that those are the common options (password or ssh key) and if you want passwordless that leaves ssh keys | 20:59 |
| Siva_ | @blake_r, I do have the maas key and they maas key is present in the bastion host (where the vm is launched) | 21:06 |
| Siva_ | so passwordless should have worked | 21:06 |
| blake_r | Siva_: you need to make sure | 21:06 |
| blake_r | Siva_: you can try "sudo -u maas -H virsh -c {your-connection-string} list --all" | 21:07 |
| Siva_ | yes, I tried that and it worked | 21:07 |
| Siva_ | so passwordless should have worked. That's why it is confusing for me | 21:07 |
| Siva_ | @blake_r, from which machine, do you want me to issue the command 'sudo -u maas -H virsh -c {your-connection-string} list --all' | 21:09 |
| Siva_ | I tried from the maas container | 21:09 |
| blake_r | which every machine is running the rack controller | 21:19 |
| roaksoax | Siva_: /win 3 | 21:43 |
| roaksoax | err | 21:43 |
| roaksoax | sry | 21:43 |
| stormmore | still no idea why I can't get this to enlist :-/ | 21:49 |
| stormmore | all I do know is I keep getting return code 8 but that doesn't tell me anything about what is going on | 21:52 |
| roaksoax | stormmore: please pastebin your logs /var/log/maas/*.log | 21:52 |
| roaksoax | stormmore: if you can get a console log, that would also be helpful | 21:52 |
| stormmore | do you mean /var/log/syslog when you say console?? | 21:53 |
| roaksoax | stormmore: console as in the server outputs logs to the serial console | 21:54 |
| stormmore | roaksoax, that might be easier said that done | 21:55 |
| stormmore | roaksoax, regiond.log - http://paste.ubuntu.com/24080653/ | 21:59 |
| stormmore | sorry that is the rackd.log | 21:59 |
| mup | Bug #1660860 changed: [1.9] mdadm error message during 14.04 deploy <MAAS:Invalid> <MAAS 1.9:New> <https://launchpad.net/bugs/1660860> | 22:01 |
| stormmore | roaksoax, here is the correct regiond.log - http://paste.ubuntu.com/24080662/ | 22:02 |
| stormmore | roaksoax, I suspect dns being the problem | 22:02 |
| stormmore | roaksoax, I suspect that I am running into a race condition since the maas server is using a different dns to get it setup but the node is using the same which isn't pointing back to the maas server | 22:03 |
| roaksoax | stormmore: this is the issue: | 22:07 |
| roaksoax | 2017-02-27 21:51:06 provisioningserver.rackdservices.tftp: [critical] TFTP back-end failed. | 22:07 |
| roaksoax | Traceback (most recent call last): | 22:07 |
| roaksoax | Failure: twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=13020:cmd=GetBootConfig:ask=18]') | 22:07 |
| roaksoax | stormmore: tail -f /var/log/maas/*.log and capture the moment that the machine is pxe booting and share the output of logs with us | 22:08 |
| roaksoax | stormmore: if you can also provide a console log of the PXE process it would be good as well | 22:08 |
| roaksoax | stormmore: /win | 22:09 |
| roaksoax | err | 22:09 |
| stormmore | roaksoax, that is part of my problem, I don't see enough of the console log to determine the issue or have a way of getting the console log off the machine :-/ I do know that maas system is unable to look itself up because it is using a dns server that doesn't have the record | 22:10 |
| roaksoax | stormmore: but ht eproblem seems to be in the pxe boot | 22:16 |
| roaksoax | stormmore: at least from the logs above | 22:16 |
| roaksoax | stormmore: and it doesn't seem that you are pointin rackd.conf to a dns record | 22:16 |
| stormmore | roaksoax, well pxe seems to be partially working, it is when it goes to enlist the machine that it doesn't work | 22:17 |
| stormmore | I am waiting for my freshly installed maas server to finish syncing an image so that I can try again | 22:17 |
| stormmore | roaksoax, it is worth noting that manually trying to enlist from a freshly installed system doesn't work either :-/ | 22:18 |
| mup | Bug #1660860 opened: [1.9] mdadm error message during 14.04 deploy <MAAS:Invalid> <MAAS 1.9:New> <https://launchpad.net/bugs/1660860> | 22:19 |
| roaksoax | stormmore: you mean, you try to add a machine via the webui or API and it also fails ? | 22:20 |
| stormmore | roaksoax, http://paste.ubuntu.com/24080751/ | 22:21 |
| stormmore | roaksoax, using maas-enlist | 22:21 |
| stormmore | roaksoax, that is literally the totality of output from running that command :-/ | 22:23 |
| mup | Bug #1660860 changed: [1.9] mdadm error message during 14.04 deploy <MAAS:Invalid> <MAAS 1.9:New> <https://launchpad.net/bugs/1660860> | 22:25 |
| stormmore | roaksoax, http://paste.ubuntu.com/24080847/ is all the maas logs during a pxe boot attempt | 22:39 |
| mimizone | pmatulis: are you still around? I am trying to find the right naming convention for the files in /etc/maas/preseeds/ | 22:41 |
| mimizone | pmatulis: so far curtin_ubuntu_amad64_generic_nodename doesn't seem to work. | 22:41 |
| mimizone | amd64 | 22:41 |
| Siva_ | @blake_r, the command sudo -u maas -H virsh -c {your-connection-string} list --all works form the maas container but passwordless acess is not working | 22:42 |
| Siva_ | any clue? | 22:42 |
| stormmore | Siva_, whats the output from ssh-add -l? | 22:42 |
| Siva_ | @stormore, ubuntu@xenial-maas:~$ ssh-add -l Could not open a connection to your authentication agent. | 22:44 |
| stormmore | that will be your problem | 22:44 |
| Siva_ | @stormmore, not clear, what am I missing here? | 22:44 |
| Siva_ | @stormmore, from which machine should I run that command? | 22:45 |
| stormmore | Siva_, that tells me that you don't have the key loaded and there for passwordless login won't work | 22:45 |
| stormmore | Siva_, maas controller | 22:45 |
| stormmore | actually it should probably be sudo -u maas ssh-add -l unless you are already logged in as the maas user | 22:46 |
| roaksoax | stormmore: seems the the machine pxe booted into the image | 22:47 |
| pmatulis | mimizone, i'm sorry i haven't looked at that stuff in a long time. i don't know | 22:47 |
| roaksoax | stormmore: and the image is trying to access the metadata, which seems to be successful | 22:48 |
| roaksoax | stormmore: 2017-02-27 22:36:48 twisted.python.log: [info] ::ffff:10.255.255.4 - - [27/Feb/2017:22:36:48 +0000] "GET /MAAS/metadata/enlist/2012-03-01/user-data HTTP/1.1" 200 8272 "-" "python-requests/2.9.1" | 22:48 |
| stormmore | roaksoax, yeah just doesn't enlist! | 22:48 |
| Siva_ | @stormmore, ubuntu@xenial-maas:~$ sudo -u maas ssh-add -l Could not open a connection to your authentication agent. | 22:48 |
| Siva_ | same error | 22:48 |
| mimizone | pmatulis: first of all, I forgot the release number... but I am digging into it. Do you know if the code is in maas basically? | 22:48 |
| stormmore | roaksoax, and trying to load into the image that is already installed, installing maas-enlist, and trying doesn't help me figure out why it isn't enlisting | 22:48 |
| pmatulis | mimizone, it should work, yes | 22:49 |
| pmatulis | mimizone, if you could include a reproducer in a docs bug then i would have something to go by | 22:49 |
| roaksoax | stormmore: so have you ssh'd into the image itself ? | 22:49 |
| stormmore | Siva_, well that will explain why it will ask for a password using the command roaksoax gave you, you don't have the key loaded into an agent thus it doesn't know it has a key to use | 22:49 |
| stormmore | roaksoax, correct, I have an image on the machine that I am trying to enlist | 22:50 |
| Siva_ | @stormore, the following command works though..... sudo -u maas -H virsh -c {your-connection-string} list --all | 22:50 |
| Siva_ | @stormore, the following command works though..... sudo -u maas -H virsh -c {your-connection-string} list --all.... it does not ask for a password | 22:51 |
| stormmore | ok tryin sudo -u maas -H virst -c ssh-add -l | 22:52 |
| stormmore | argh! | 22:52 |
| stormmore | ok tryin sudo -u maas -H virsh -c ssh-add -l | 22:52 |
| stormmore | Siva_, the problem when it asks you for a password is caused by the fact that the command you are trying to run doesn't know where the private key is | 22:53 |
| roaksoax | stormmore: so the enlistment process is: | 22:56 |
| roaksoax | 1. machine PXE boots | 22:56 |
| roaksoax | 2. loads the image from MAAS' iscsi | 22:56 |
| roaksoax | 3. image (cloud-init) access maas metadata server | 22:56 |
| roaksoax | 3. cloud-init runs metadata, and contacts maas to "create" a "new" machine | 22:56 |
| roaksoax | stormmore: from the last log, I see that 1, 2 and 3 went ahead successfully | 22:56 |
| roaksoax | 4 is what may be failing | 22:57 |
| stormmore | yeah and then nothing to show that it is even trying to enlist | 22:57 |
| stormmore | (on a side note, I really wish the return codes from maas-enlist were documented cause that might give some indication) | 22:58 |
| mimizone | pmatulis: sorry to bug, but what is the the typical value of "release" in the preseed.py? it's not 16.04. Would it be xenial, or the entire name Ubuntu 16.04 LTS "Xenial Xerus"? | 23:00 |
| mimizone | it is the value release=Config.objects.get_config('commissioning_distro_series')) | 23:00 |
| mimizone | or release=node.get_distro_series()) | 23:00 |
| roaksoax | stormmore: is there no output in the console ? | 23:01 |
| stormmore | roaksoax, not that I can get to a place that I can cut and paste :-/ | 23:01 |
| pmatulis | roaksoax, can you answer mimizone's query? | 23:02 |
| roaksoax | stormmore: and you said nothing here either right ? /var/log/maas/rsyslog/maas-enlisting-node/ | 23:02 |
| roaksoax | mimizone: can you pint me to where you are looking exactly ? | 23:03 |
| mimizone | roaksoax: in maasserver/preseed.py | 23:03 |
| mimizone | trying to find out how the preseed file is picked, based on the release/node name etc... | 23:04 |
| roaksoax | mimizone: what do you mean ? | 23:04 |
| mimizone | see the function get_preseed(node) | 23:04 |
| mimizone | my understanding is that the preseed file is picked using that naming "{prefix}_{osystem}_{node_arch}_{node_subarch}_{release}_{node_name}" | 23:05 |
| roaksoax | mimizone: release should be like 'xenial', 'trusty' etc | 23:05 |
| stormmore | roaksoax, ok we might be making a little progress. found those /var/logs/maas/rsyslogs | 23:05 |
| mimizone | prefix = curtin during the installation deployment | 23:05 |
| mimizone | roaksoax: ok thanks. let me try. | 23:06 |
| roaksoax | mimizone: curtin_userdata_ubuntu_amd64_generic_trusty_node01 | 23:07 |
| roaksoax | mimizone: it should be something along those lines | 23:07 |
| mimizone | still need the userdata? | 23:07 |
| roaksoax | mimizone: for ubuntu you can avoid specifying ubuntu | 23:07 |
| stormmore | roaksoax, those logs were all messy so I have deleted them and trying again to get a clean run | 23:08 |
| stormmore | now it didn't recreate the log :-/ | 23:08 |
| mimizone | looks like curtin_ubuntu_amd64_generic_xenial_nodename does the trick. It crasehd my deployment = good sign since my curting yaml file is most likely invalid :) | 23:09 |
| roaksoax | mimizone: hehe, just cp curtin_userdata to yours and modify accordingly | 23:09 |
| roaksoax | stormmore: restart rsyslog | 23:09 |
| roaksoax | ? | 23:09 |
| stormmore | roaksoax, hmmm knew there was something I forgot to do ;-) | 23:10 |
| stormmore | roaksoax, I think I found the problem http://paste.ubuntu.com/24081111/ | 23:14 |
| stormmore | roaksoax, specifically curl: (22) The requested URL returned error: 400 BAD REQUEST | 23:14 |
| stormmore | roaksoax, think I am on the right track? | 23:20 |
| pmatulis | mimizone, sweet, you got punted ahead :) | 23:21 |
| mimizone | pmatulis: :) there must be more to it now, because just copying the curtin_userdata to the other config for specifically that node breaks the deployment | 23:24 |
| pmatulis | mimizone, like i said, i haven't looked at that stuff in donkey's years | 23:26 |
| * pmatulis needs to make some ravioli for some small people | 23:27 | |
| mimizone | seeing stuff like this in the maas logs NameError: name 'curtin_preseed' is not defined at line 4 column 5 in file /etc/maas/preseeds/curtin_ubuntu_amd64_generic_xenial_osv16ocp3b | 23:27 |
| mimizone | trying again with a 2 liner config file instead of an entire copy of the curtin_userdata... | 23:28 |
| mimizone | how can I make the same GET /MAAS/metadata/latest/by-id/agqa6n/?op=get_preseed from the command line to see what's returned? | 23:32 |
| mimizone | because whatever it sends stops the deployment... | 23:33 |
| mimizone | ok got it | 23:35 |
| mimizone | curl -X GET http://172.30.72.130/MAAS/metadata/latest/by-id/agqa6n/?op=get_preseed | 23:35 |
| stormmore | OK I am going to take a break for a bit and try again when I get home | 23:39 |
| mimizone | What I am noticing is that if I remove the specific configuration file for my node, the metdata/getpreseed call returns something completely different from the curtin_userdata | 23:45 |
| mimizone | it returns all the necessary stuff for apt, datasource, syslog etc... | 23:46 |
| mimizone | otherwise it returns my file, that misses all that info. which I assume explains the deployment aborts the process | 23:47 |
| mimizone | got it. | 23:48 |
| mimizone | the file to copy is the /etc/maas/preseeds/curtin | 23:49 |
| mimizone | not the curtin_userdata | 23:49 |
| mimizone | basically I need to add {{preseed_data}} at the top of my own file | 23:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!