| Lord_Set | Are you able to comission MAAS nodes under one cluster controller and transfer ownership of them to another? | 00:05 |
|---|---|---|
| bigjools | Lord_Set: no | 00:05 |
| Lord_Set | :( alright | 00:06 |
| Lord_Set | Thanks for talking to Dustin by the way. We are going to setup up a phone meeting with him on Tuesday with the CEO and lead developer and myself (lead network and systems engineer) to talk about ubuntu, maas and juju. | 00:08 |
| bigjools | excellent | 00:08 |
| bradm | anyone played with neutron much? got openstack deployed with maas and juju and having weird neutron issues :-/ | 00:12 |
| Lord_Set | Which issues? | 00:14 |
| Lord_Set | I used Neutron with the Cisco Nexus API | 00:15 |
| Lord_Set | err use | 00:15 |
| bradm | I'm seeing mtu issues when trying to boot juju on top of it, if I have the mtu on the neutron gateway set to 1500, instances won't boot - but if I drop the instance MTUs down to under 1500, say 1456, it works fine | 00:15 |
| Lord_Set | Which network infrastructure do you have? | 00:15 |
| bradm | its all Cisco kit - we've moved most if if not all of it to Cisco Nexus stuff | 00:16 |
| bradm | the Nexus switches are pretty new to us, this is the first thing we're using them for | 00:17 |
| Lord_Set | Alright. Was just trying to get an idea. I have never seen that issue before though I have some ideas of what may be causing it. Though are you using GRE or Vlans? | 00:18 |
| bradm | GRE, which is why we're playing with mtu | 00:19 |
| Lord_Set | I haven't had much luck with GRE and honestly not a fan of it | 00:19 |
| bradm | if I tweak the neutron dhcp agent to hand out mtu of 1456, things seem to work better, but it'd be much preferred if we didn't have to do it | 00:19 |
| Lord_Set | Especially when using the Nexus switches | 00:19 |
| bradm | I was of the understanding that you needed a mtu of > 1500 on the physical boxes if you wanted to leave the instances VMs on 1500, due to the overhead of GRE | 00:20 |
| Lord_Set | Correct | 00:20 |
| Lord_Set | GRE has a pretty beefy overhead and if you don't have good nics to offload you WILL have performance hits and issues | 00:20 |
| bradm | at this point we're not even seeing it working, let alone looking at performance | 00:21 |
| Lord_Set | What was your reasoning for using GRE? | 00:21 |
| bradm | hm, but if setting the neutron gateway to 1546 doesn't fix it, I wonder if that means something else in the chain isn't set right | 00:22 |
| bradm | Lord_Set: wasn't really my choice, its a project I've been finishing off from someone else | 00:22 |
| Lord_Set | Without seeing a network map and your configs and neurton configs it is hard to speculate. | 00:23 |
| bradm | yeah, fair enough, these things are pretty complex | 00:24 |
| bradm | but at least it sounds like my base reasoning about the mtu is right, I just need to figure out where else in the chain it might be breaking | 00:24 |
| Lord_Set | I am super close to being a CCIE... Network engineering is my cup of tea with Cisco especially. I am sure I could help if I could get more info. | 00:25 |
| bradm | what sort of info would you need? | 00:25 |
| Lord_Set | Copies of the switch/router configs, a network map and your neutron config | 00:26 |
| bradm | the physical ports aren't real interesting, they've just got a description and a switchport access vlan ### | 00:27 |
| Lord_Set | Is this a test, development or production network? | 00:28 |
| bradm | its going to be production once we get things going | 00:28 |
| bradm | but I can mess around with settings right now | 00:28 |
| Lord_Set | Alright, well give this a try... make a /etc/neutron/dnsmasq-neutron.conf | 00:29 |
| Lord_Set | put a single line in there | 00:29 |
| Lord_Set | dhcp-option-force=26,1400 | 00:29 |
| bradm | yup, that works | 00:29 |
| bradm | well, I have 1456, but that makes things move forward | 00:29 |
| === CyberJacob is now known as CyberJacob|Away | ||
| Lord_Set | Try 1400 specifically. | 00:30 |
| bradm | oh, ok | 00:30 |
| Lord_Set | It is what Cisco and Openstack are recommending for GRE it looks like now. | 00:30 |
| bradm | ok, restarted neutron-dhcp-agent, and trying a bootstrap again | 00:30 |
| Lord_Set | It seems higher mtu sizes are causing packet fragmentation | 00:31 |
| Lord_Set | you will have to restart the neutron-server | 00:31 |
| Lord_Set | service neturon-server restart | 00:31 |
| bradm | oh, the dhcp-agent has been sufficent in the past to make things work - maybe thats why it hasn't worked fully | 00:32 |
| Lord_Set | There are other services that are dependent of neutron-server that have to be restarted as well | 00:32 |
| bradm | curious, there's no neutron-server service on ubuntu | 00:34 |
| bradm | there's neutron-dhcp-agent, neutron-l3-agent, neutron-metadata-agent, and neutron-plugin-openvswitch-agent | 00:34 |
| bradm | I'll just restart them all :) | 00:35 |
| Lord_Set | Odd, what version are you running? | 00:35 |
| bradm | 2013.2.1-0ubuntu1~cloud0, which is havana | 00:36 |
| bradm | it seems about the same as with 1456, I'll keep trying it out | 00:43 |
| Lord_Set | Are you able to check port statistics or use wireshark and see if it is fragmenting the packets even though the MTU is lowered? | 00:50 |
| bradm | let me see.. | 00:53 |
| bradm | I have access to the switch, I'm just not a cisco type | 00:53 |
| Lord_Set | sh int INTERFACE | 00:56 |
| Lord_Set | you should look for underruns, overruns, and other RX errors | 00:58 |
| bradm | none of those | 00:58 |
| Lord_Set | That is good | 00:58 |
| bradm | there are some ''giants'' | 00:58 |
| Lord_Set | Is this on a 1 or 10 gig interface? | 00:59 |
| bradm | and these are actually some of the few ports that aren't on the nexus switch yet | 00:59 |
| bradm | only 1gig interfaces | 00:59 |
| Lord_Set | Ok | 00:59 |
| Lord_Set | Well the giants are frames larger than 1518 bytes | 01:01 |
| bradm | ah, which could be from GRE I'm guessing | 01:02 |
| Lord_Set | Which model switch is that connected to currently? | 01:02 |
| bradm | a 3750 | 01:02 |
| bradm | more specifically, a WS-C3750X-48P according to the config | 01:03 |
| Lord_Set | What does "show system mtu" show you? | 01:03 |
| bradm | hah, everything is 1500 | 01:04 |
| Lord_Set | Even Jumbo frames? | 01:04 |
| bradm | yup | 01:04 |
| Lord_Set | well then type "system mtu jumbo 9000" | 01:05 |
| bradm | http://pastebin.ubuntu.com/6997387/ | 01:05 |
| Lord_Set | for that to take effect you will have to reload the switch | 01:05 |
| bradm | this switch does have some other stuff on it, I'd have to sort out a time to reload it | 01:05 |
| Lord_Set | remember to copy run start or wr mem after typing that | 01:05 |
| Lord_Set | Alright | 01:05 |
| Lord_Set | That is very likely your issue | 01:06 |
| bradm | I'm thinking it'll be easier for me to get these ports moved over to the nexus switch | 01:06 |
| Lord_Set | Most likely | 01:06 |
| bradm | I'm just about as far away from these switches as is physically possible right now :-/ | 01:06 |
| Lord_Set | heh | 01:06 |
| Lord_Set | I know the feeling | 01:06 |
| bradm | as in, they're on the other side of the world | 01:06 |
| Lord_Set | if you end up being able to reload the switch you should also set the 10/100 mtu to 1546 | 01:07 |
| Lord_Set | system mtu 1546 | 01:07 |
| bradm | but that sure looks like it would explain why 1546 doesn't work on the neutron gateway | 01:07 |
| Lord_Set | Yep | 01:07 |
| bradm | how do you see mtu on the nexus switches? could that be a problem too? | 01:08 |
| Lord_Set | It is possible | 01:08 |
| bradm | ? is my favourite cisco command. :) | 01:09 |
| Lord_Set | lol | 01:09 |
| Lord_Set | It is pretty magical | 01:09 |
| Lord_Set | http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/112080-config-mtu-nexus.html | 01:10 |
| Lord_Set | That will work for other model nexus switches as well | 01:10 |
| bradm | haha, I'd just found that page via google and was reading | 01:10 |
| bradm | it looks like everything is 1500 there as well, thats going to be the issue | 01:11 |
| bradm | sweet, thats great, thanks for all your help, I can work on fixing all that up | 01:11 |
| Lord_Set | No problem! Glad to help | 01:12 |
| bradm | I really need to learn more Cisco side of things, I'm more of a linux sysadmin | 01:17 |
| Lord_Set | Cisco is a whole world in itself... especially with the Nexus line of switches and can be pretty complex when you start tying storage networking into the mix. | 01:18 |
| Lord_Set | With FCOE and FC in the same switch. | 01:18 |
| bradm | yeah, definately, and things start getting hazy when you are talking Neutron too, with all the SDN | 01:20 |
| Lord_Set | Yep | 01:20 |
| Lord_Set | If you can convince whoever you need to I would strongly suggest rebuilding your openstack cluster to vlan based instead of GRE | 01:21 |
| Lord_Set | a lot less headaches and super easy with the nexus api to automate everything | 01:21 |
| bradm | sounds like a good idea, I'll have a look into it | 01:22 |
| bradm | the fact we're using juju charms to deploy openstack might be a factor too, since I think they only do openvswitch right now - no reason that couldn't (nor indeed, shouldn't) change though, just more work. | 01:25 |
| Lord_Set | http://blogs.cisco.com/datacenter/availability-of-the-nexus-1000v-cloud-networking-platform-for-openstack-unveiled-at-cisco-live/ | 01:26 |
| bradm | https://wiki.openstack.org/wiki/Cisco-neutron | 01:27 |
| bradm | the charm says it only supports openvswitch though | 01:30 |
| Lord_Set | Interesting. | 01:30 |
| === edu-afk is now known as edamato | ||
| === mwhudson is now known as zz_mwhudson | ||
| === zz_mwhudson is now known as mwhudson | ||
| svetmy5 | Hi! Gettng "TFTP prefix: unable to locate configuration file" on PXE boot of a node from maas. Please suggest the fix? | 02:46 |
| bigjools | what state is the node in, in maas? | 02:47 |
| bigjools | if it's still "Ready" you can't boot the node yet, maas needs to start it | 02:47 |
| === edamato is now known as edu-afk | ||
| Lord_Set | Does MAAS support reporting to a log server? | 02:55 |
| bigjools | the appserver logs are via Apache | 02:57 |
| bigjools | as in, the threads are Apache threads so whatever Apaches does for logging | 02:57 |
| bigjools | the celery daemons and pserv are standalone logs, so I don't think compatible with a log server | 02:58 |
| Lord_Set | I was more looking at having the logging report to a log server such as Solarwinds | 02:58 |
| Lord_Set | Specifically maas.log and pserv.log | 02:58 |
| bigjools | yeah it would be a good idea to use syslog and ship a configuration to have separated logs | 02:59 |
| bigjools | I don't know anything about Solarwinds | 02:59 |
| Lord_Set | http://www.solarwinds.com/kiwi-syslog-server.aspx | 03:00 |
| bigjools | ok so it's syslog compatible | 03:06 |
| Lord_Set | Awesome | 03:06 |
| svetmy5_ | Hi, if i missed this could you please repost, i'm getting tftp prefix: configuration file not found from on booting from maas? | 03:08 |
| bigjools | svetmy5_: [12:47:00] <bigjools> what state is the node in, in maas? | 03:14 |
| bigjools | [12:47:22] <bigjools> if it's still "Ready" you can't boot the node yet, maas needs to start it | 03:14 |
| svetmy5_ | It doesn't get discovered by maas, i tried adding by mac address, so it's in commissioning state. | 03:15 |
| bigjools | svetmy5_: can you pastebin the exact log please | 03:16 |
| Lord_Set | Any reason why the maas.log would be empty? | 03:25 |
| bigjools | Lord_Set: no errors to report I guess - what's in the apache logs? | 03:26 |
| Lord_Set | Let me check | 03:27 |
| Lord_Set | http://pastebin.ubuntu.com/6997820/ | 03:29 |
| bigjools | well... | 03:32 |
| bigjools | that's a new one on me | 03:32 |
| bigjools | jtv: any ideas? --^ | 03:32 |
| Lord_Set | rofl just my luck | 03:32 |
| * jtv looks | 03:32 | |
| bigjools | reboot? :) | 03:33 |
| Lord_Set | Already tried that. | 03:33 |
| bigjools | or restart Apache | 03:33 |
| Lord_Set | Done that too | 03:33 |
| Lord_Set | and every other associated process | 03:33 |
| Lord_Set | err service | 03:33 |
| jtv | Looks as if either a stale lock hangs around, or we get stuck during startup. | 03:35 |
| jtv | You may need to delete the lock file in /run/lock. | 03:35 |
| jtv | We build the filename as '/run/lock/' + __name__, which I hope doesn't mean that the lock file will be called __main__. | 03:36 |
| Lord_Set | I will look now | 03:36 |
| === mwhudson is now known as zz_mwhudson | ||
| Lord_Set | root@EZ-GAR-MAAS-01:/var/log/apache2# ls /run/lock | 03:37 |
| Lord_Set | apache2 EZ-GAR-MAAS-01.Dummy-1-3830 EZ-GAR-MAAS-01.MainThread-3691 | 03:37 |
| Lord_Set | EZ-GAR-MAAS-01.Dummy-1-1822 EZ-GAR-MAAS-01.Dummy-1-3831 maasserver.start_up.lock | 03:37 |
| jtv | That's the one. | 03:37 |
| jtv | start_up is the module name. | 03:37 |
| Lord_Set | Ok. I will delete it. | 03:37 |
| jtv | Once you're absolutely sure that you've killed the processes. | 03:37 |
| jtv | This wasn't supposed to happen; we do register an atexit for cleaning up the lock. But there's always a way to break it I suppose. | 03:38 |
| bigjools | stop Apache first | 03:38 |
| jtv | Yes, good idea. :) | 03:38 |
| Lord_Set | Well Apache started a lot faster that time | 03:39 |
| Lord_Set | And that fixed my PXE issue as well generating the config file | 03:41 |
| Lord_Set | lol and another odd one | 03:42 |
| Lord_Set | FQDN | 03:42 |
| Lord_Set | ;; connection timed out; no servers could be reached.master | 03:42 |
| bigjools | great - is svetmy5_ your colleague? | 03:42 |
| Lord_Set | Nope | 03:43 |
| Lord_Set | err yes | 03:43 |
| Lord_Set | Didn't realize that was him. He changed his name. | 03:43 |
| Lord_Set | He is our lead developer | 03:44 |
| jtv | I think I have an idea what that FQDN is... | 03:45 |
| jtv | "dig" prints its errors to stdout, not stderr. | 03:45 |
| Lord_Set | Ahh ok | 03:45 |
| Lord_Set | I just found it amusing | 03:45 |
| jtv | That it is, in a way. I guess it must be the enlistment userdata calling "dig." | 03:46 |
| jtv | I'll file a bug. | 03:46 |
| jtv | Filed as bug 1284964 | 03:52 |
| ubot5 | bug 1284964 in MAAS "Bad FQDN: ";; connection timed out; no servers could be reached.master"" [High,Triaged] https://launchpad.net/bugs/1284964 | 03:52 |
| Lord_Set | Cool, thanks | 03:57 |
| Lord_Set | If I specify the nameserver in the preseed instead of the string value should that fix it? | 03:59 |
| jtv | AFAICT this happens in the enlistment preseed... I don't know if we ever re-generate the hostname. | 04:02 |
| jtv | But if the nameserver isn't responding, then the hostname is going to be problematic one way or another. | 04:03 |
| jtv | Although yes, this does look as if it's ignoring any nameserver selection, doesn't it? | 04:04 |
| Lord_Set | It does even though it is specified everywhere. I have also double checked all the config files just to make sure. | 04:04 |
| Lord_Set | Though I have queried the local nameserver built into maas and it responds | 04:05 |
| Lord_Set | I was thinking of setting the nameserver in the preseed_master | 04:06 |
| Lord_Set | d-i netcfg/get_nameservers string 10.10.10.50 | 04:06 |
| bigjools | DHCP sets it, no need | 04:06 |
| jtv | May have been a transient thing: maybe the nameserver was just restarting in response to a config change, or upgrading... | 04:06 |
| bigjools | jtv: we should be using reload, not restart, for that anyway | 04:07 |
| bigjools | for this very reason | 04:07 |
| jtv | Yup. There may be some finicky details though about what can be done with a reload and what will require a restart. | 04:07 |
| bigjools | jtv: not that I know of! reload just tells it to re-read confug | 04:08 |
| jtv | Yes, what I mean is that some daemons have some settings that they can't change with a mere reload. | 04:08 |
| Lord_Set | There are a few things I have had to consistently change in configs to get some stuff working. For example the filename being specified in the dhcpd.conf as pxelinux.0 | 04:09 |
| Lord_Set | Uncommenting the tftp lines in the pserv.yaml as well | 04:10 |
| bigjools | jtv: true, but I think named is designed so reloading and/or using rndc covers everything, as it's a sufficiently high-profile daemon | 04:10 |
| bigjools | Lord_Set: you should not have to do any of that, I am surprised. It works OOTB for me | 04:11 |
| Lord_Set | Odd | 04:11 |
| Lord_Set | From a fresh install of ubuntu server and then installing the MAAS and Juju packages or doing a install from the media and choosing to make the server a MAAS server? | 04:11 |
| bigjools | installation of packages - but the latter does the same thing AFAIK | 04:12 |
| bigjools | you need new packages I expect | 04:13 |
| Lord_Set | I just rebooted the server | 04:13 |
| bigjools | if you install from 12.04 media, then ... earrghhhh | 04:13 |
| Lord_Set | Been installing from 13.10 media | 04:13 |
| bigjools | phew | 04:13 |
| Lord_Set | I will recheck versions once it is back up. | 04:14 |
| Lord_Set | There aren't any maas or juju updates available to me so I am guessing I am current. | 04:29 |
| Lord_Set | FYI jtv and bigjools, setting the nameserver in the preseed_master worked | 04:36 |
| bigjools | huh | 04:37 |
| bigjools | submit a patch! | 04:37 |
| Lord_Set | heh, I had to manually add the nameserver | 04:37 |
| Lord_Set | Not much of a patch | 04:37 |
| jtv | So you bypassed the one offered by DHCP? | 04:38 |
| bigjools | maybe the installer ignores it | 04:39 |
| Lord_Set | DHCP wasn't passing it's configured nameserver to the preseed for some reason | 04:39 |
| bigjools | it doesn't go to the preseed, it ends up in the dhcp client, which is supposed to set the resolv.conf, or dnsmasq config if dnsmasq being used | 04:40 |
| Lord_Set | in the preseed_master what is the purpose of the d-i netcfg/get_nameservers string then? | 04:40 |
| Lord_Set | Also, which log file would have ipmi errors in it? | 04:41 |
| Lord_Set | Is MAAS using ipmitool in the background or doing custom calls via XML? | 04:42 |
| bigjools | I don't know what d-i is using to set the nameserver, that setting might be an override | 04:42 |
| bigjools | it spawns ipmitool | 04:42 |
| Lord_Set | Ok | 04:42 |
| bigjools | all from a template you can customise | 04:43 |
| Lord_Set | I have odd issues with it and getting it to work with our mass amounts of HP DL360 and 380 G5 servers. | 04:43 |
| Lord_Set | They have ILO version 2 which supports IPMI version 2 I believe. | 04:44 |
| Lord_Set | https://bugs.launchpad.net/maas/+bug/1086162 | 04:48 |
| ubot5 | Ubuntu bug 1086162 in MAAS "IPMI based power management default to IPMI 1.5 based authentication" [High,Triaged] | 04:48 |
| Lord_Set | Seems to be the same bug | 04:48 |
| bigjools | someone is fixing that as we speak! | 04:48 |
| Lord_Set | :) | 04:49 |
| jtv | In fact I approved the MP earlier. | 04:51 |
| jtv | Meanwhile, bigjools, you might like https://code.launchpad.net/~jtv/maas/use-mock-matchers/+merge/208272 | 04:51 |
| bigjools | jtv: \o/ | 04:52 |
| jtv | It's only a start. I count 90 more instances of '\.assert_' after this. :( | 04:52 |
| bigjools | jtv: I'll just fix up my branch and land it | 04:53 |
| jtv | Great, thanks. | 04:54 |
| jtv | Really have to go at these things, or you get the worst of both worlds. | 04:54 |
| bigjools | jtv: zoiks I completely forgot the docstrings on two classes | 04:54 |
| jtv | Meanwhile, there's a whole bunch of new CSS lint. No idea where it came from. :( | 04:54 |
| jtv | I thought you left the docstrings out deliberately. | 04:55 |
| jtv | It slowed me down a bit in reading, but not very much. | 04:55 |
| jtv | Although obviously these things do add up. | 04:55 |
| bigjools | jtv: no, sorry, I fucked up | 04:56 |
| jtv | Tsk. | 04:56 |
| Lord_Set | Have you run any tests to see about the max number of servers able to reliably commission and install at once? | 05:12 |
| Lord_Set | http://pastebin.ubuntu.com/6998135/ | 05:26 |
| Lord_Set | Another interesting error. Takes place after deleting a node, powering the server on and letting it preseed again. | 05:26 |
| bradm | Lord_Set: that's generally what I see if I let the server try and enlist again | 05:36 |
| Lord_Set | It looks like once you delete a node it isn't removing the hostname from the database for some reason | 05:37 |
| Lord_Set | ugh, working through ipmi issues for the lose | 05:49 |
| Lord_Set | bradm, http://pastebin.ubuntu.com/6998208/ | 05:56 |
| Lord_Set | Looks like it is an issue with the dhcp host map | 05:56 |
| bradm | Lord_Set: huh, sure does | 06:05 |
| Lord_Set | bigjools, are you still around? | 06:32 |
| Lord_Set | jtv welcome back | 07:06 |
| Lord_Set | Have you seen any ilo/ipmi issues where MAAS won't properly create it's own maas account in the ilo/ipmi? It sure thinks it does but doesn't generate any errors in any logs about it. | 07:07 |
| jtv | Doesn't ring a bell, no. | 07:08 |
| Lord_Set | On one server it isn't even pulling the ipmi information during the enlist... just leaves it blank | 07:09 |
| === CyberJacob|Away is now known as CyberJacob | ||
| Lord_Set | Ipmipower seems super picky... | 07:20 |
| Lord_Set | Hopefully that team plans on working on it's issues soon | 07:21 |
| bigjools | Lord_Set: I am back | 07:23 |
| bigjools | so let me look at your questions in order | 07:23 |
| bigjools | no, we've not done that concurrency testing. We suspect it will be limited to bandwidth required for tftp/broadcast traffic on a LAN segment | 07:24 |
| bigjools | umm I've never seen that node deletion error and I do A LOT of deletion + re-enlist cycles. Weird. | 07:25 |
| bigjools | Lord_Set: I've seen plenty of problems with creating ilo accounts. We have a bunch of fixes landing/about to land in trunk for that, it's much more reliable now. | 07:26 |
| Lord_Set | Any way I can get my hands on those fixes now? ;) | 07:27 |
| bigjools | and yes ipmi is a PITA and quite flakey sometimes | 07:27 |
| bigjools | sure, use the daily PPA :) | 07:27 |
| Lord_Set | Because this is being a thorn in my side! | 07:27 |
| Lord_Set | I will do so for sure | 07:27 |
| bigjools | what hardware were you using again? | 07:27 |
| Lord_Set | All HP DL360 and DL380 G5s | 07:27 |
| bigjools | proliants? | 07:27 |
| Lord_Set | Yep | 07:27 |
| bigjools | I think there's a kernel bug when inserting the ipmi module | 07:28 |
| Lord_Set | They use iLo 2.0 which supports IPMI 2.0 when updated to their latest firmware. | 07:28 |
| bigjools | go into the enlist_userdata preseed and uncomment the IPMI_SI_PARAMS setting | 07:28 |
| Lord_Set | Alright | 07:28 |
| bigjools | I also saw some reports of the latest ilo firmware being rather buggy | 07:28 |
| Lord_Set | It is a lot better than having IPMI 1.0 | 07:29 |
| bigjools | my proliants that I test with are 1.2 and seem ok, someone else on 1.3 is having trouble | 07:29 |
| Lord_Set | They skipped 1.5 | 07:29 |
| Lord_Set | I have a lot of Dell 1950 G3, 2950 G3 and R805s that I will test with tomorrow. | 07:30 |
| Lord_Set | All with DRAC | 07:30 |
| bigjools | k | 07:30 |
| bigjools | would be nice feedback for us! | 07:30 |
| Lord_Set | glad to help | 07:30 |
| Lord_Set | your project is fueling the backbone of what this startup is doing... well we plan to do once we get further into development | 07:31 |
| bigjools | sweet | 07:40 |
| bigjools | Lord_Set: this could be your duplicate hostname bug https://bugs.launchpad.net/ubuntu/+source/maas-enlist/+bug/1081660 | 07:42 |
| ubot5 | Ubuntu bug 1081660 in maas-enlist (Ubuntu) "If maas-enlist fails to reach a DNS server, the node will be named ";; connection timed out; no servers could be reached"" [Critical,Triaged] | 07:42 |
| * bigjools EODs | 07:42 | |
| Lord_Set | Looks like it | 07:43 |
| Lord_Set | Or really close. I will have to go through the bug report to be sure. | 07:43 |
| rvba | Lord_Set: basically if you have a node which has that name because the enlisting nodes can't reach a DNS server, if you try to enlist a second node (while still having the DNS problem) you'll have the {'hostname': [u'Node with t | 07:46 |
| rvba | his Hostname already exists.']} error | 07:46 |
| Lord_Set | Yeah I found the issue was becase of the preseed dns problem. Once I manually set the preseed nameserver the issue disapeered. | 07:47 |
| rvba | rbasak: Hi there, I've got a question for you about uvtool: how can I get uvtool to use a particular version of Ubuntu, for instance Trusty beta1? | 07:47 |
| Lord_Set | PXE booting with 10g nics kicks ass especially when installing to 6 drive raid 10 SSD arrays | 07:58 |
| Lord_Set | bigjools, seen this one before? Happened after updating to daily ppa | 08:12 |
| Lord_Set | http://pastebin.ubuntu.com/6998610/ | 08:12 |
| bigjools | Lord_Set: are you using trusty? | 08:22 |
| bigjools | daily only works on trusty | 08:22 |
| * bigjools away | 08:22 | |
| Lord_Set | That would explain it... Will upgrade to trusty now. | 08:23 |
| === CyberJacob is now known as CyberJacob|Away | ||
| Lord_Set | How do you reset a MAAS password in trusty? | 09:33 |
| Lord_Set | I don't see the command under maas anymore | 09:34 |
| jpds | Can someone tell me how I would manually fetch a file like pxelinux.cfg/01-00-25-90-7e-9e-20 ? | 10:02 |
| jpds | I need to inspect what it contains. | 10:05 |
| rbasak | rvba: off the top of my head: uvt-simplestreams-sync --source http://cloud-images.ubuntu.com/daily release=trusty label=beta1 arch=... | 10:32 |
| rbasak | rvba: then: uvt-kvm create ... release=trusty label=beta1 arch=... | 10:32 |
| rbasak | rvba: "uvt-simplestreams-query" can be used with the same filter arguments to show you what you have. If you have arguments that narrow this list down to one, then uvt-kvm should be able to use it. | 10:32 |
| rvba | rbasak: thanks. So label=beta1 is just a simplestreams filter right? | 10:42 |
| rbasak | rvba: right | 10:42 |
| rvba | arges: that means we can use that for the ephemerals too. | 10:42 |
| rvba | arges: oops, sorry. | 10:42 |
| rvba | I meant allenap. | 10:42 |
| jpds | rvba: Any idea about me? | 10:43 |
| rbasak | rvba: oh, one error. You don't need --source. Beta images are in the "released" stream, not the "daily" stream, and the former is the default. | 10:43 |
| rvba | rbasak: okay | 10:44 |
| rbasak | http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.json shows you what you can filter on. "ftype=tar.gz" is automatically added by uvtool, so it gets the right download type. | 10:44 |
| rvba | jpds: I'll get back to you in a sec… | 10:44 |
| rvba | rbasak: thanks, that's very useful. | 10:47 |
| jpds | This is also rather interesting: http://pastebin.ubuntu.com/6999107/ | 10:47 |
| rbasak | Err, "ftype=disk1.img" is automatically added by uvtool. That's the right download type :) | 10:53 |
| rbasak | np! | 10:53 |
| rvba | jpds: If you're using a recent version of MAAS, the easiest way to see that is to raise the log level in /etc/maas/maas_local_settings.py to "DEBUG". Then /var/log/maas/maas.log will contain the content of all the requests that go through. | 10:56 |
| jpds | rvba: So, /etc/maas/maas_local_settings.py; DEBUG = True. | 11:04 |
| jpds | Oh, log: level: DEBUG. | 11:04 |
| rvba | jpds: yeah, LOGGING_LEVEL = 'DEBUG' | 11:05 |
| rvba | Then you need to restart apache2. | 11:05 |
| jpds | rvba: Well, bizarrely, all maas.log shows is an OAuthUnauthorized error. | 11:14 |
| jpds | rvba: Right, it's logging things to every file, but maas.log. | 11:16 |
| rvba | o_O | 11:16 |
| jpds | But it's certainly running. | 11:18 |
| rvba | jpds: what version of MAAS are you using. | 11:19 |
| rvba | ? | 11:19 |
| jpds | rvba: cloud-archive:tools. | 11:19 |
| rvba | Ah, that's why you're not seeing the content of the requests. | 11:19 |
| jpds | Nooo. :( | 11:21 |
| rvba | jpds: Let me think about another way… | 11:25 |
| rvba | jpds: something you can do is thiss: | 11:29 |
| rvba | this* | 11:29 |
| rvba | Look in /var/log/apache2/access.log for requests like: | 11:31 |
| rvba | http://localhost/MAAS/api/1.0/pxeconfig/?cluster_uuid=0c6ea249-fa34-45e1-bbee-706695cafb41&local=192.168.21.5&mac=00-e0-81-d1-b1-47&r | 11:31 |
| rvba | (But with your MAC address, of course) | 11:31 |
| rvba | Then use wget to reproduce the exact same request. | 11:31 |
| rvba | It won't give you the complete template, but it will contain all the elements used by the provisioning server to produce the template. | 11:31 |
| rvba | Basically the tftp server queries the MAAS server to get all the info it needs to produce pxelinux.cfg/01-00-25-90-7e-9e-20. By doing what I described you recreate the request made by the tftp server to the MAAS server. | 11:32 |
| jpds | Oh, pxeconfig instead of pxelinux. | 11:33 |
| jpds | That's why I couldn't find anything in the apache logs. | 11:34 |
| Lord_Set | I really need to sleep but keeping running into bugs and issues :( | 11:37 |
| Lord_Set | Any of the MAAS team available? | 12:08 |
| jpds | rvba: Yeah, we need the actually pxelinux input... | 12:52 |
| jpds | Anyway I can fake the TFTP request? | 12:52 |
| rvba | jpds: I've never done it so you'll have do some research about that but I don't see why you couldn't. | 12:55 |
| Lord_Set | rvba you are a MAAS dev correct? | 13:01 |
| jpds | rvba: http://pastebin.ubuntu.com/6999651/ | 13:07 |
| jpds | Well, clearly I have a different remote= in the Apache logs. | 13:09 |
| jpds | Lord_Set: It's generally better if you ASK the question. | 13:10 |
| Lord_Set | I am having issues with the enlistment detecting ILO/IPMI... I am currently running trusty 14.04 with the daily ppa as well. | 13:16 |
| larry_ | hello | 13:35 |
| jamespage | Lord_Set, it would be so much easier if all server manufacturer did the same thing | 13:41 |
| Lord_Set | James I concur | 13:42 |
| jamespage | Lord_Set, is maas not able to detect/configure ipmi during enlistment | 13:42 |
| jamespage | ? | 13:42 |
| Lord_Set | Correct | 13:42 |
| Lord_Set | It was doing it fine with 13.10 | 13:42 |
| jamespage | Lord_Set, does it set any data at all? | 13:42 |
| Lord_Set | Nope | 13:42 |
| jamespage | urgh - that's bad | 13:42 |
| Lord_Set | Yeah :( | 13:43 |
| Lord_Set | I am going to have to delve deep into it later with bigjools and jtk when they are back around. Or whatever devs are available. | 13:43 |
| jamespage | Lord_Set, please raise a bug with details about what versions you are using and what type of server you are using | 13:43 |
| jamespage | nice thing about bugs are they are async :-) | 13:44 |
| Lord_Set | Yeah I will. | 13:44 |
| Lord_Set | Though as a bonus MAAS without doing anything detects my broadcom 10g dual port nics. | 13:46 |
| rvba | Lord_Set: Yes I am. I'd be happy to help you but I'm pretty busy this afternoon. If you encounter problems, please file bugs and we'll get back to you asap. | 13:57 |
| rvba | rbasak: any idea what I'm doing wrong? http://paste.ubuntu.com/6999893/ | 14:05 |
| rbasak | rvba: have you synced those things? Sorry if I wasn't clear - query queries what is synced locally. To query the remote end, use sstream-query (simplestreams package) | 14:06 |
| rvba | rbasak: ah! That's it, I thought it was querying the remote end. Thanks! | 14:07 |
| rbasak | Sorry (again) for the lack of documentation. I was going to hide in a corner and work on that this afternoon actually. | 14:08 |
| rvba | Cool :). | 14:08 |
| rvba | Hum, gpg error now… http://paste.ubuntu.com/6999909/ | 14:12 |
| rvba | rbasak: sorry to bother you again but any idea if I need to fiddle with gpg key before using sstream-query? ^ | 14:16 |
| rvba | rbasak: ah! nm, I got it. | 14:16 |
| === CyberJacob|Away is now known as CyberJacob | ||
| === cmagina_ is now known as cmagina | ||
| === zz_mwhudson is now known as mwhudson | ||
| === smoser` is now known as smoser | ||
| === CyberJacob is now known as CyberJacob|Away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!