[00:05] <Lord_Set> Are you able to comission MAAS nodes under one cluster controller and transfer ownership of them to another?
[00:05] <bigjools> Lord_Set: no
[00:06] <Lord_Set> :( alright
[00:08] <Lord_Set> Thanks for talking to Dustin by the way. We are going to setup up a phone meeting with him on Tuesday with the CEO and lead developer and myself (lead network and systems engineer) to talk about ubuntu, maas and juju.
[00:08] <bigjools> excellent
[00:12] <bradm> anyone played with neutron much?  got openstack deployed with maas and juju and having weird neutron issues :-/
[00:14] <Lord_Set> Which issues?
[00:15] <Lord_Set> I used Neutron with the Cisco Nexus API
[00:15] <Lord_Set> err use
[00:15] <bradm> I'm seeing mtu issues when trying to boot juju on top of it, if I have the mtu on the neutron gateway set to 1500, instances won't boot - but if I drop the instance MTUs down to under 1500, say 1456, it works fine
[00:15] <Lord_Set> Which network infrastructure do you have?
[00:16] <bradm> its all Cisco kit - we've moved most if if not all of it to Cisco Nexus stuff
[00:17] <bradm> the Nexus switches are pretty new to us, this is the first thing we're using them for
[00:18] <Lord_Set> Alright. Was just trying to get an idea. I have never seen that issue before though I have some ideas of what may be causing it. Though are you using GRE or Vlans?
[00:19] <bradm> GRE, which is why we're playing with mtu
[00:19] <Lord_Set> I haven't had much luck with GRE and honestly not a fan of it
[00:19] <bradm> if I tweak the neutron dhcp agent to hand out mtu of 1456, things seem to work better, but it'd be much preferred if we didn't have to do it
[00:19] <Lord_Set> Especially when using the Nexus switches
[00:20] <bradm> I was of the understanding that you needed a mtu of > 1500 on the physical boxes if you wanted to leave the instances VMs on 1500, due to the overhead of GRE
[00:20] <Lord_Set> Correct
[00:20] <Lord_Set> GRE has a pretty beefy overhead and if you don't have good nics to offload you WILL have performance hits and issues
[00:21] <bradm> at this point we're not even seeing it working, let alone looking at performance
[00:21] <Lord_Set> What was your reasoning for using GRE?
[00:22] <bradm> hm, but if setting the neutron gateway to 1546 doesn't fix it, I wonder if that means something else in the chain isn't set right
[00:22] <bradm> Lord_Set: wasn't really my choice, its a project I've been finishing off from someone else
[00:23] <Lord_Set> Without seeing a network map and your configs and neurton configs it is hard to speculate.
[00:24] <bradm> yeah, fair enough, these things are pretty complex
[00:24] <bradm> but at least it sounds like my base reasoning about the mtu is right, I just need to figure out where else in the chain it might be breaking
[00:25] <Lord_Set> I am super close to being a CCIE... Network engineering is my cup of tea with Cisco especially. I am sure I could help if I could get more info.
[00:25] <bradm> what sort of info would you need?
[00:26] <Lord_Set> Copies of the switch/router configs, a network map and your neutron config
[00:27] <bradm> the physical ports aren't real interesting, they've just got a description and a switchport access vlan ###
[00:28] <Lord_Set> Is this a test, development or production network?
[00:28] <bradm> its going to be production once we get things going
[00:28] <bradm> but I can mess around with settings right now
[00:29] <Lord_Set> Alright, well give this a try... make a /etc/neutron/dnsmasq-neutron.conf
[00:29] <Lord_Set> put a single line in there
[00:29] <Lord_Set> dhcp-option-force=26,1400
[00:29] <bradm> yup, that works
[00:29] <bradm> well, I have 1456, but that makes things move forward
=== CyberJacob is now known as CyberJacob|Away
[00:30] <Lord_Set> Try 1400 specifically.
[00:30] <bradm> oh, ok
[00:30] <Lord_Set> It is what Cisco and Openstack are recommending for GRE it looks like now.
[00:30] <bradm> ok, restarted neutron-dhcp-agent, and trying a bootstrap again
[00:31] <Lord_Set> It seems higher mtu sizes are causing packet fragmentation
[00:31] <Lord_Set> you will have to restart the neutron-server
[00:31] <Lord_Set> service neturon-server restart
[00:32] <bradm> oh, the dhcp-agent has been sufficent in the past to make things work - maybe thats why it hasn't worked fully
[00:32] <Lord_Set> There are other services that are dependent of neutron-server that have to be restarted as well
[00:34] <bradm> curious, there's no neutron-server service on ubuntu
[00:34] <bradm> there's neutron-dhcp-agent, neutron-l3-agent, neutron-metadata-agent, and neutron-plugin-openvswitch-agent
[00:35] <bradm> I'll just restart them all :)
[00:35] <Lord_Set> Odd, what version are you running?
[00:36] <bradm> 2013.2.1-0ubuntu1~cloud0, which is havana
[00:43] <bradm> it seems about the same as with 1456, I'll keep trying it out
[00:50] <Lord_Set> Are you able to check port statistics or use wireshark and see if it is fragmenting the packets even though the MTU is lowered?
[00:53] <bradm> let me see..
[00:53] <bradm> I have access to the switch, I'm just not a cisco type
[00:56] <Lord_Set> sh int INTERFACE
[00:58] <Lord_Set> you should look for underruns, overruns, and other RX errors
[00:58] <bradm> none of those
[00:58] <Lord_Set> That is good
[00:58] <bradm> there are some ''giants''
[00:59] <Lord_Set> Is this on a 1 or 10 gig interface?
[00:59] <bradm> and these are actually some of the few ports that aren't on the nexus switch yet
[00:59] <bradm> only 1gig interfaces
[00:59] <Lord_Set> Ok
[01:01] <Lord_Set> Well the giants are frames larger than 1518 bytes
[01:02] <bradm> ah, which could be from GRE I'm guessing
[01:02] <Lord_Set> Which model switch is that connected to currently?
[01:02] <bradm> a 3750
[01:03] <bradm> more specifically, a WS-C3750X-48P according to the config
[01:03] <Lord_Set> What does "show system mtu" show you?
[01:04] <bradm> hah, everything is 1500
[01:04] <Lord_Set> Even Jumbo frames?
[01:04] <bradm> yup
[01:05] <Lord_Set> well then type "system mtu jumbo 9000"
[01:05] <bradm> http://pastebin.ubuntu.com/6997387/
[01:05] <Lord_Set> for that to take effect you will have to reload the switch
[01:05] <bradm> this switch does have some other stuff on it, I'd have to sort out a time to reload it
[01:05] <Lord_Set> remember to copy run start or wr mem after typing that
[01:05] <Lord_Set> Alright
[01:06] <Lord_Set> That is very likely your issue
[01:06] <bradm> I'm thinking it'll be easier for me to get these ports moved over to the nexus switch
[01:06] <Lord_Set> Most likely
[01:06] <bradm> I'm just about as far away from these switches as is physically possible right now :-/
[01:06] <Lord_Set> heh
[01:06] <Lord_Set> I know the feeling
[01:06] <bradm> as in, they're on the other side of the world
[01:07] <Lord_Set> if you end up being able to reload the switch you should also set the 10/100 mtu to 1546
[01:07] <Lord_Set> system mtu 1546
[01:07] <bradm> but that sure looks like it would explain why 1546 doesn't work on the neutron gateway
[01:07] <Lord_Set> Yep
[01:08] <bradm> how do you see mtu on the nexus switches?  could that be a problem too?
[01:08] <Lord_Set> It is possible
[01:09] <bradm> ? is my favourite cisco command. :)
[01:09] <Lord_Set> lol
[01:09] <Lord_Set> It is pretty magical
[01:10] <Lord_Set> http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/112080-config-mtu-nexus.html
[01:10] <Lord_Set> That will work for other model nexus switches as well
[01:10] <bradm> haha, I'd just found that page via google and was reading
[01:11] <bradm> it looks like everything is 1500 there as well, thats going to be the issue
[01:11] <bradm> sweet, thats great, thanks for all your help, I can work on fixing all that up
[01:12] <Lord_Set> No problem! Glad to help
[01:17] <bradm> I really need to learn more Cisco side of things, I'm more of a linux sysadmin
[01:18] <Lord_Set> Cisco is a whole world in itself... especially with the Nexus line of switches and can be pretty complex when you start tying storage networking into the mix.
[01:18] <Lord_Set> With FCOE and FC in the same switch.
[01:20] <bradm> yeah, definately, and things start getting hazy when you are talking Neutron too, with all the SDN
[01:20] <Lord_Set> Yep
[01:21] <Lord_Set> If you can convince whoever you need to I would strongly suggest rebuilding your openstack cluster to vlan based instead of GRE
[01:21] <Lord_Set> a lot less headaches and super easy with the nexus api to automate everything
[01:22] <bradm> sounds like a good idea, I'll have a look into it
[01:25] <bradm> the fact we're using juju charms to deploy openstack might be a factor too, since I think they only do openvswitch right now - no reason that couldn't (nor indeed, shouldn't) change though, just more work.
[01:26] <Lord_Set> http://blogs.cisco.com/datacenter/availability-of-the-nexus-1000v-cloud-networking-platform-for-openstack-unveiled-at-cisco-live/
[01:27] <bradm> https://wiki.openstack.org/wiki/Cisco-neutron
[01:30] <bradm> the charm says it only supports openvswitch though
[01:30] <Lord_Set> Interesting.
=== edu-afk is now known as edamato
=== mwhudson is now known as zz_mwhudson
=== zz_mwhudson is now known as mwhudson
[02:46] <svetmy5> Hi! Gettng "TFTP prefix: unable to locate configuration file" on PXE boot of a node from maas. Please suggest the fix?
[02:47] <bigjools> what state is the node in, in maas?
[02:47] <bigjools> if it's still "Ready" you can't boot the node yet, maas needs to start it
=== edamato is now known as edu-afk
[02:55] <Lord_Set> Does MAAS support reporting to a log server?
[02:57] <bigjools> the appserver logs are via Apache
[02:57] <bigjools> as in, the threads are Apache threads so whatever Apaches does for logging
[02:58] <bigjools> the celery daemons and pserv are standalone logs, so I don't think compatible with a log server
[02:58] <Lord_Set> I was more looking at having the logging report to a log server such as Solarwinds
[02:58] <Lord_Set> Specifically maas.log and pserv.log
[02:59] <bigjools> yeah it would be a good idea to use syslog and ship a configuration to have separated logs
[02:59] <bigjools> I don't know anything about Solarwinds
[03:00] <Lord_Set> http://www.solarwinds.com/kiwi-syslog-server.aspx
[03:06] <bigjools> ok so it's syslog compatible
[03:06] <Lord_Set> Awesome
[03:08] <svetmy5_> Hi, if i missed this could you please repost, i'm getting tftp prefix: configuration file not found from on booting from maas?
[03:14] <bigjools> svetmy5_: [12:47:00] <bigjools> what state is the node in, in maas?
[03:14] <bigjools> [12:47:22] <bigjools> if it's still "Ready" you can't boot the node yet, maas needs to start it
[03:15] <svetmy5_> It doesn't get discovered by maas, i tried adding by mac address, so it's in commissioning state.
[03:16] <bigjools> svetmy5_: can you pastebin the exact log please
[03:25] <Lord_Set> Any reason why the maas.log would be empty?
[03:26] <bigjools> Lord_Set: no errors to report I guess - what's in the apache logs?
[03:27] <Lord_Set> Let me check
[03:29] <Lord_Set> http://pastebin.ubuntu.com/6997820/
[03:32] <bigjools> well...
[03:32] <bigjools> that's a new one on me
[03:32] <bigjools> jtv: any ideas? --^
[03:32] <Lord_Set> rofl just my luck
[03:32]  * jtv looks
[03:33] <bigjools> reboot? :)
[03:33] <Lord_Set> Already tried that.
[03:33] <bigjools> or restart Apache
[03:33] <Lord_Set> Done that too
[03:33] <Lord_Set> and every other associated process
[03:33] <Lord_Set> err service
[03:35] <jtv> Looks as if either a stale lock hangs around, or we get stuck during startup.
[03:35] <jtv> You may need to delete the lock file in /run/lock.
[03:36] <jtv> We build the filename as '/run/lock/' + __name__, which I hope doesn't mean that the lock file will be called __main__.
[03:36] <Lord_Set> I will look now
=== mwhudson is now known as zz_mwhudson
[03:37] <Lord_Set> root@EZ-GAR-MAAS-01:/var/log/apache2# ls /run/lock
[03:37] <Lord_Set> apache2                      EZ-GAR-MAAS-01.Dummy-1-3830     EZ-GAR-MAAS-01.MainThread-3691
[03:37] <Lord_Set> EZ-GAR-MAAS-01.Dummy-1-1822  EZ-GAR-MAAS-01.Dummy-1-3831     maasserver.start_up.lock
[03:37] <jtv> That's the one.
[03:37] <jtv> start_up is the module name.
[03:37] <Lord_Set> Ok. I will delete it.
[03:37] <jtv> Once you're absolutely sure that you've killed the processes.
[03:38] <jtv> This wasn't supposed to happen; we do register an atexit for cleaning up the lock.  But there's always a way to break it I suppose.
[03:38] <bigjools> stop Apache first
[03:38] <jtv> Yes, good idea.  :)
[03:39] <Lord_Set> Well Apache started a lot faster that time
[03:41] <Lord_Set> And that fixed my PXE issue as well generating the config file
[03:42] <Lord_Set> lol and another odd one
[03:42] <Lord_Set> FQDN
[03:42] <Lord_Set> ;; connection timed out; no servers could be reached.master
[03:42] <bigjools> great - is svetmy5_ your colleague?
[03:43] <Lord_Set> Nope
[03:43] <Lord_Set> err yes
[03:43] <Lord_Set> Didn't realize that was him. He changed his name.
[03:44] <Lord_Set> He is our lead developer
[03:45] <jtv> I think I have an idea what that FQDN is...
[03:45] <jtv> "dig" prints its errors to stdout, not stderr.
[03:45] <Lord_Set> Ahh ok
[03:45] <Lord_Set> I just found it amusing
[03:46] <jtv> That it is, in a way.  I guess it must be the enlistment userdata calling "dig."
[03:46] <jtv> I'll file a bug.
[03:52] <jtv> Filed as bug 1284964
[03:52] <ubot5> bug 1284964 in MAAS "Bad FQDN: ";; connection timed out; no servers could be reached.master"" [High,Triaged] https://launchpad.net/bugs/1284964
[03:57] <Lord_Set> Cool, thanks
[03:59] <Lord_Set> If I specify the nameserver in the preseed instead of the string value should that fix it?
[04:02] <jtv> AFAICT this happens in the enlistment preseed...  I don't know if we ever re-generate the hostname.
[04:03] <jtv> But if the nameserver isn't responding, then the hostname is going to be problematic one way or another.
[04:04] <jtv> Although yes, this does look as if it's ignoring any nameserver selection, doesn't it?
[04:04] <Lord_Set> It does even though it is specified everywhere. I have also double checked all the config files just to make sure.
[04:05] <Lord_Set> Though I have queried the local nameserver built into maas and it responds
[04:06] <Lord_Set> I was thinking of setting the nameserver in the preseed_master
[04:06] <Lord_Set> d-i     netcfg/get_nameservers  string 10.10.10.50
[04:06] <bigjools> DHCP sets it, no need
[04:06] <jtv> May have been a transient thing: maybe the nameserver was just restarting in response to a config change, or upgrading...
[04:07] <bigjools> jtv: we should be using reload, not restart, for that anyway
[04:07] <bigjools> for this very reason
[04:07] <jtv> Yup.  There may be some finicky details though about what can be done with a reload and what will require a restart.
[04:08] <bigjools> jtv: not that I know of! reload just tells it to re-read confug
[04:08] <jtv> Yes, what I mean is that some daemons have some settings that they can't change with a mere reload.
[04:09] <Lord_Set> There are a few things I have had to consistently change in configs to get some stuff working. For example the filename being specified in the dhcpd.conf as pxelinux.0
[04:10] <Lord_Set> Uncommenting the tftp lines in the pserv.yaml as well
[04:10] <bigjools> jtv: true, but I think named is designed so reloading and/or using rndc covers everything, as it's a sufficiently high-profile daemon
[04:11] <bigjools> Lord_Set: you should not have to do any of that, I am surprised.  It works OOTB for me
[04:11] <Lord_Set> Odd
[04:11] <Lord_Set> From a fresh install of ubuntu server and then installing the MAAS and Juju packages or doing a install from the media and choosing to make the server a MAAS server?
[04:12] <bigjools> installation of packages - but the latter does the same thing AFAIK
[04:13] <bigjools> you need new packages I expect
[04:13] <Lord_Set> I just rebooted the server
[04:13] <bigjools> if you install from 12.04 media, then ... earrghhhh
[04:13] <Lord_Set> Been installing from 13.10 media
[04:13] <bigjools> phew
[04:14] <Lord_Set> I will recheck versions once it is back up.
[04:29] <Lord_Set> There aren't any maas or juju updates available to me so I am guessing I am current.
[04:36] <Lord_Set> FYI jtv and bigjools, setting the nameserver in the preseed_master worked
[04:37] <bigjools> huh
[04:37] <bigjools> submit a patch!
[04:37] <Lord_Set> heh, I had to manually add the nameserver
[04:37] <Lord_Set> Not much of a patch
[04:38] <jtv> So you bypassed the one offered by DHCP?
[04:39] <bigjools> maybe the installer ignores it
[04:39] <Lord_Set> DHCP wasn't passing it's configured nameserver to the preseed for some reason
[04:40] <bigjools> it doesn't go to the preseed, it ends up in the dhcp client, which is supposed to set the resolv.conf, or dnsmasq config if dnsmasq being used
[04:40] <Lord_Set> in the preseed_master what is the purpose of the d-i     netcfg/get_nameservers  string then?
[04:41] <Lord_Set> Also, which log file would have ipmi errors in it?
[04:42] <Lord_Set> Is MAAS using ipmitool in the background or doing custom calls via XML?
[04:42] <bigjools> I don't know what d-i is using to set the nameserver, that setting might be an override
[04:42] <bigjools> it spawns ipmitool
[04:42] <Lord_Set> Ok
[04:43] <bigjools> all from a template you can customise
[04:43] <Lord_Set> I have odd issues with it and getting it to work with our mass amounts of HP DL360 and 380 G5 servers.
[04:44] <Lord_Set> They have ILO version 2 which supports IPMI version 2 I believe.
[04:48] <Lord_Set> https://bugs.launchpad.net/maas/+bug/1086162
[04:48] <ubot5> Ubuntu bug 1086162 in MAAS "IPMI based power management default to IPMI 1.5 based authentication" [High,Triaged]
[04:48] <Lord_Set> Seems to be the same bug
[04:48] <bigjools> someone is fixing that as we speak!
[04:49] <Lord_Set> :)
[04:51] <jtv> In fact I approved the MP earlier.
[04:51] <jtv> Meanwhile, bigjools, you might like https://code.launchpad.net/~jtv/maas/use-mock-matchers/+merge/208272
[04:52] <bigjools> jtv: \o/
[04:52] <jtv> It's only a start.  I count 90 more instances of '\.assert_' after this.  :(
[04:53] <bigjools> jtv: I'll just fix up my branch and land it
[04:54] <jtv> Great, thanks.
[04:54] <jtv> Really have to go at these things, or you get the worst of both worlds.
[04:54] <bigjools> jtv: zoiks I completely forgot the docstrings on two classes
[04:54] <jtv> Meanwhile, there's a whole bunch of new CSS lint.  No idea where it came from.  :(
[04:55] <jtv> I thought you left the docstrings out deliberately.
[04:55] <jtv> It slowed me down a bit in reading, but not very much.
[04:55] <jtv> Although obviously these things do add up.
[04:56] <bigjools> jtv: no, sorry, I fucked up
[04:56] <jtv> Tsk.
[05:12] <Lord_Set> Have you run any tests to see about the max number of servers able to reliably commission and install at once?
[05:26] <Lord_Set> http://pastebin.ubuntu.com/6998135/
[05:26] <Lord_Set> Another interesting error. Takes place after deleting a node, powering the server on and letting it preseed again.
[05:36] <bradm> Lord_Set: that's generally what I see if I let the server try and enlist again
[05:37] <Lord_Set> It looks like once you delete a node it isn't removing the hostname from the database for some reason
[05:49] <Lord_Set> ugh, working through ipmi issues for the lose
[05:56] <Lord_Set> bradm, http://pastebin.ubuntu.com/6998208/
[05:56] <Lord_Set> Looks like it is an issue with the dhcp host map
[06:05] <bradm> Lord_Set: huh, sure does
[06:32] <Lord_Set> bigjools, are you still around?
[07:06] <Lord_Set> jtv welcome back
[07:07] <Lord_Set> Have you seen any ilo/ipmi issues where MAAS won't properly create it's own maas account in the ilo/ipmi? It sure thinks it does but doesn't generate any errors in any logs about it.
[07:08] <jtv> Doesn't ring a bell, no.
[07:09] <Lord_Set> On one server it isn't even pulling the ipmi information during the enlist... just leaves it blank
=== CyberJacob|Away is now known as CyberJacob
[07:20] <Lord_Set> Ipmipower seems super picky...
[07:21] <Lord_Set> Hopefully that team plans on working on it's issues soon
[07:23] <bigjools> Lord_Set: I am back
[07:23] <bigjools> so let me look at your questions in order
[07:24] <bigjools> no, we've not done that concurrency testing.  We suspect it will be limited to bandwidth required for tftp/broadcast traffic on a LAN segment
[07:25] <bigjools> umm I've never seen that node deletion error and I do A LOT of deletion + re-enlist cycles.  Weird.
[07:26] <bigjools> Lord_Set: I've seen plenty of problems with creating ilo accounts.  We have a bunch of fixes landing/about to land in trunk for that, it's much more reliable now.
[07:27] <Lord_Set> Any way I can get my hands on those fixes now? ;)
[07:27] <bigjools> and yes ipmi is a PITA and quite flakey sometimes
[07:27] <bigjools> sure, use the daily PPA :)
[07:27] <Lord_Set> Because this is being a thorn in my side!
[07:27] <Lord_Set> I will do so for sure
[07:27] <bigjools> what hardware were you using again?
[07:27] <Lord_Set> All HP DL360 and DL380 G5s
[07:27] <bigjools> proliants?
[07:27] <Lord_Set> Yep
[07:28] <bigjools> I think there's a kernel bug when inserting the ipmi module
[07:28] <Lord_Set> They use iLo 2.0 which supports IPMI 2.0 when updated to their latest firmware.
[07:28] <bigjools> go into the enlist_userdata preseed and uncomment the IPMI_SI_PARAMS setting
[07:28] <Lord_Set> Alright
[07:28] <bigjools> I also saw some reports of the latest ilo firmware being rather buggy
[07:29] <Lord_Set> It is a lot better than having IPMI 1.0
[07:29] <bigjools> my proliants that I test with are 1.2 and seem ok, someone else on 1.3 is having trouble
[07:29] <Lord_Set> They skipped 1.5
[07:30] <Lord_Set> I have a lot of Dell 1950 G3, 2950 G3 and R805s that I will test with tomorrow.
[07:30] <Lord_Set> All with DRAC
[07:30] <bigjools> k
[07:30] <bigjools> would be nice feedback for us!
[07:30] <Lord_Set> glad to help
[07:31] <Lord_Set> your project is fueling the backbone of what this startup is doing... well we plan to do once we get further into development
[07:40] <bigjools> sweet
[07:42] <bigjools> Lord_Set: this could be your duplicate hostname bug https://bugs.launchpad.net/ubuntu/+source/maas-enlist/+bug/1081660
[07:42] <ubot5> Ubuntu bug 1081660 in maas-enlist (Ubuntu) "If maas-enlist fails to reach a DNS server, the node will be named ";; connection timed out; no servers could be reached"" [Critical,Triaged]
[07:42]  * bigjools EODs
[07:43] <Lord_Set> Looks like it
[07:43] <Lord_Set> Or really close. I will have to go through the bug report to be sure.
[07:46] <rvba> Lord_Set: basically if you have a node which has that name because the enlisting nodes can't reach a DNS server, if you try to enlist a second node (while still having the DNS problem) you'll have the {'hostname': [u'Node with t
[07:46] <rvba> his Hostname already exists.']} error
[07:47] <Lord_Set> Yeah I found the issue was becase of the preseed dns problem. Once I manually set the preseed nameserver the issue disapeered.
[07:47] <rvba> rbasak: Hi there, I've got a question for you about uvtool: how can I get uvtool to use a particular version of Ubuntu, for instance Trusty beta1?
[07:58] <Lord_Set> PXE booting with 10g nics kicks ass especially when installing to 6 drive raid 10 SSD arrays
[08:12] <Lord_Set> bigjools, seen this one before? Happened after updating to daily ppa
[08:12] <Lord_Set> http://pastebin.ubuntu.com/6998610/
[08:22] <bigjools> Lord_Set: are you using trusty?
[08:22] <bigjools> daily only works on trusty
[08:22]  * bigjools away
[08:23] <Lord_Set> That would explain it... Will upgrade to trusty now.
=== CyberJacob is now known as CyberJacob|Away
[09:33] <Lord_Set> How do you reset a MAAS password in trusty?
[09:34] <Lord_Set> I don't see the command under maas anymore
[10:02] <jpds> Can someone tell me how I would manually fetch a file like pxelinux.cfg/01-00-25-90-7e-9e-20 ?
[10:05] <jpds> I need to inspect what it contains.
[10:32] <rbasak> rvba: off the top of my head: uvt-simplestreams-sync --source http://cloud-images.ubuntu.com/daily release=trusty label=beta1 arch=...
[10:32] <rbasak> rvba: then: uvt-kvm create ... release=trusty label=beta1 arch=...
[10:32] <rbasak> rvba: "uvt-simplestreams-query" can be used with the same filter arguments to show you what you have. If you have arguments that narrow this list down to one, then uvt-kvm should be able to use it.
[10:42] <rvba> rbasak: thanks.  So label=beta1 is just a simplestreams filter right?
[10:42] <rbasak> rvba: right
[10:42] <rvba> arges: that means we can use that for the ephemerals too.
[10:42] <rvba> arges: oops, sorry.
[10:42] <rvba> I meant allenap.
[10:43] <jpds> rvba: Any idea about me?
[10:43] <rbasak> rvba: oh, one error. You don't need --source. Beta images are in the "released" stream, not the "daily" stream, and the former is the default.
[10:44] <rvba> rbasak: okay
[10:44] <rbasak> http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.json shows you what you can filter on. "ftype=tar.gz" is automatically added by uvtool, so it gets the right download type.
[10:44] <rvba> jpds: I'll get back to you in a sec…
[10:47] <rvba> rbasak: thanks, that's very useful.
[10:47] <jpds> This is also rather interesting: http://pastebin.ubuntu.com/6999107/
[10:53] <rbasak> Err, "ftype=disk1.img" is automatically added by uvtool. That's the right download type :)
[10:53] <rbasak> np!
[10:56] <rvba> jpds: If you're using a recent version of MAAS, the easiest way to see that is to raise the log level in /etc/maas/maas_local_settings.py to "DEBUG".  Then /var/log/maas/maas.log will contain the content of all the requests that go through.
[11:04] <jpds> rvba: So, /etc/maas/maas_local_settings.py; DEBUG = True.
[11:04] <jpds> Oh, log: level: DEBUG.
[11:05] <rvba> jpds: yeah, LOGGING_LEVEL = 'DEBUG'
[11:05] <rvba> Then you need to restart apache2.
[11:14] <jpds> rvba: Well, bizarrely, all maas.log shows is an OAuthUnauthorized error.
[11:16] <jpds> rvba: Right, it's logging things to every file, but maas.log.
[11:16] <rvba> o_O
[11:18] <jpds> But it's certainly running.
[11:19] <rvba> jpds: what version of MAAS are you using.
[11:19] <rvba> ?
[11:19] <jpds> rvba: cloud-archive:tools.
[11:19] <rvba> Ah, that's why you're not seeing the content of the requests.
[11:21] <jpds> Nooo. :(
[11:25] <rvba> jpds: Let me think about another way…
[11:29] <rvba> jpds: something you can do is thiss:
[11:29] <rvba> this*
[11:31] <rvba> Look in /var/log/apache2/access.log for requests like:
[11:31] <rvba> http://localhost/MAAS/api/1.0/pxeconfig/?cluster_uuid=0c6ea249-fa34-45e1-bbee-706695cafb41&local=192.168.21.5&mac=00-e0-81-d1-b1-47&r
[11:31] <rvba> (But with your MAC address, of course)
[11:31] <rvba> Then use wget to reproduce the exact same request.
[11:31] <rvba> It won't give you the complete template, but it will contain all the elements used by the provisioning server to produce the template.
[11:32] <rvba> Basically the tftp server queries the MAAS server to get all the info it needs to produce pxelinux.cfg/01-00-25-90-7e-9e-20.  By doing what I described you recreate the request made by the tftp server to the MAAS server.
[11:33] <jpds> Oh, pxeconfig instead of pxelinux.
[11:34] <jpds> That's why I couldn't find anything in the apache logs.
[11:37] <Lord_Set> I really need to sleep but keeping running into bugs and issues :(
[12:08] <Lord_Set> Any of the MAAS team available?
[12:52] <jpds> rvba: Yeah, we need the actually pxelinux input...
[12:52] <jpds> Anyway I can fake the TFTP request?
[12:55] <rvba> jpds: I've never done it so you'll have do some research about that but I don't see why you couldn't.
[13:01] <Lord_Set> rvba you are a MAAS dev correct?
[13:07] <jpds> rvba: http://pastebin.ubuntu.com/6999651/
[13:09] <jpds> Well, clearly I have a different remote= in the Apache logs.
[13:10] <jpds> Lord_Set: It's generally better if you ASK the question.
[13:16] <Lord_Set> I am having issues with the enlistment detecting ILO/IPMI... I am currently running trusty 14.04 with the daily ppa as well.
[13:35] <larry_> hello
[13:41] <jamespage> Lord_Set, it would be so much easier if all server manufacturer did the same thing
[13:42] <Lord_Set> James I concur
[13:42] <jamespage> Lord_Set, is maas not able to detect/configure ipmi during enlistment
[13:42] <jamespage> ?
[13:42] <Lord_Set> Correct
[13:42] <Lord_Set> It was doing it fine with 13.10
[13:42] <jamespage> Lord_Set, does it set any data at all?
[13:42] <Lord_Set> Nope
[13:42] <jamespage> urgh - that's bad
[13:43] <Lord_Set> Yeah :(
[13:43] <Lord_Set> I am going to have to delve deep into it later with bigjools and jtk when they are back around. Or whatever devs are available.
[13:43] <jamespage> Lord_Set, please raise a bug with details about what versions you are using and what type of server you are using
[13:44] <jamespage> nice thing about bugs are they are async :-)
[13:44] <Lord_Set> Yeah I will.
[13:46] <Lord_Set> Though as a bonus MAAS without doing anything detects my broadcom 10g dual port nics.
[13:57] <rvba> Lord_Set: Yes I am.  I'd be happy to help you but I'm pretty busy this afternoon.  If you encounter problems, please file bugs and we'll get back to you asap.
[14:05] <rvba> rbasak: any idea what I'm doing wrong? http://paste.ubuntu.com/6999893/
[14:06] <rbasak> rvba: have you synced those things? Sorry if I wasn't clear - query queries what is synced locally. To query the remote end, use sstream-query (simplestreams package)
[14:07] <rvba> rbasak: ah!  That's it, I thought it was querying the remote end.  Thanks!
[14:08] <rbasak> Sorry (again) for the lack of documentation. I was going to hide in a corner and work on that this afternoon actually.
[14:08] <rvba> Cool :).
[14:12] <rvba> Hum, gpg error now… http://paste.ubuntu.com/6999909/
[14:16] <rvba> rbasak: sorry to bother you again but any idea if I need to fiddle with gpg key before using sstream-query? ^
[14:16] <rvba> rbasak: ah! nm, I got it.
=== CyberJacob|Away is now known as CyberJacob
=== cmagina_ is now known as cmagina
=== zz_mwhudson is now known as mwhudson
=== smoser` is now known as smoser
=== CyberJacob is now known as CyberJacob|Away