#maas 2013-02-04
<roaksoax> rvba: howdy! the "fix" is merely to not list raring on etc/maas/import_pxe_files thogh i expected they would be available by now
<roaksoax> smoser ETA on raring ephemetal?
<rvba> roaksoax: Hi, thanks for your reply.
<rvba> It's a bit weird to add support for raring before the images are available :).
<roaksoax> rvba: it was requested :)
<rvba> Okay :).
<rvba> But now the automated testing is broken :/
<roaksoax> they want to test raring deployment
<roaksoax> rvba: just disable raring by default on import_pxe_files forbthe time being
<rvba> roaksoax: you mean revert your change?
<roaksoax> rvba: not completely
<roaksoax> rvba only etc/import_pxe_files
<roaksoax> rvba or in import_ephemerals only list precise and quantal
<roaksoax> i think the latter is better
<roaksoax> or i can do it :)
<rvba> roaksoax: if you don't mind, that would be great ;)
<roaksoax> rvba: this should do it: http://paste.ubuntu.com/1608483/
<rvba> roaksoax: if the images are going to be available presently, it might not be worth itâ¦
<roaksoax> rvba: yeah I asked smoser last week, he said that he was hoping to have them soon that's why I didn't disable it by default
<rvba> Ok, I could also fix etc/import_pxe_files in the integration test for now.
<roaksoax> yeah that too :)
<roaksoax> rvba: btw... http://pastebin.ubuntu.com/1608495/
<rvba> roaksoax: that's a weird one, I'm going to ask allenap to have a look at this one ;)
<roaksoax> cool :)
<roaksoax> thanks
<roaksoax> rvba: that seems to be quantal archive error
<roaksoax> rvba: so no owrroes as the SRU should cover that
<roaksoax> rvba: or maybe is my machine which is broken
<roaksoax> weird
<roaksoax> since I upgraded ot the SRU version
<roaksoax> anyway
<roaksoax> wXgs3Ra4yWsNY3aRSw:S5EWzC8uc6mJxFvvcp:YLGfy3ZSZZMUuepwdeDPwBRgLznPf4gR
<roaksoax> err
<roaksoax> rvba: how can I change the power parameters of a node from the cli?
<roaksoax> 22 10.98.191.18
<roaksoax> err
<allenap> roaksoax: maas-cli(8) has an example of setting power params; search for power_parameters. There's no documentation of the parameter names themselves though, afaict :-/
<roaksoax> allenap: yeah i got it :)! though the problem is that I only want to update 1 field, but it clears out everything else
<allenap> roaksoax: Mmm, sorry :(
<roaksoax> allenap: heh no worries :) I'll work around it :)
<PeterVink> Question: When setting up PXE boot with only option 66 in the DHCP (as my router only supports 66, not 67), will it work to PXE boot the systems for MAAS?
<PeterVink> Question: When setting up PXE boot with only option 66 in the DHCP (as my router only supports 66, not 67), will it work to PXE boot the systems for MAAS?
<PeterVink> Anybody?
<smoser> rvba, could you review https://code.launchpad.net/~smoser/maas/iscsi-root-equal/+merge/146016 for me ?
<smoser> or, because it is 'approve', can i just set it so
<smoser> i just forget what the rules are
<rvba> smoser: you've got a +1 vote so you can mark it approved and it will get landed. (also, I created a package with that branch and had a run in the lab)
<rvba> lgtm btw ;)
<smoser> rvba, gracias. i'll mark approved.
<rvba> smoser: will you need to backport it to 1.2 ?
<smoser> rvba, i'm sure its trivial to do.
<smoser> i'd consider it a low priority bug fix.
<smoser> it is arguably security releated, in that a user of a instance in maas could "take over" a node as it is right now.
<rvba> smoser: it is trivial to do.  Then you can self-approve that 1.2 branch as it will be a simple backport.
<smoser> rvba, ok. i'll do that. thanks for your help.
<smoser> rvba, is there a "correct" way to do the 1.2 cherry pick ?
<smoser> ( cd ../trunk.iscsi-root-equal && bzr log -r -1 -p ) | patch -p0
<smoser> is easy enough, but clealry would not maintain any bzr knowledge of what was cherry picked
<rvba> smoser: the simplest way is to use bzr to do the work for you.
<smoser> this is where you tell me how to do that.
<rvba> smoser: cd maas-1.2 && bzr merge -c <revision_id> ../maas && bzr commit -m "Backport of revision <revision_id>"
<smoser> ah. '-c' .  i tried '-r' and it wasn't so happy.
#maas 2013-02-05
<shang__> bigjools: ping, curious if the new maas has been backported to 12.04 yet?
<bigjools> shang__: you'll have to ask roaksoax, he's sorting that out
<shang__> bigjools: thanks
<shang__> roaksoax: thoughts? :)
<dholbach> hiya
<dholbach> can somebody reply to https://code.launchpad.net/~racb/ubuntu/precise/squid-deb-proxy/cloud-archive/+merge/142088 or let me know if bug 1087145 can be closed?
<ubot5> bug 1087145 in squid-deb-proxy "maas proxy prevent nodes access cloud archive " [Undecided,New] https://launchpad.net/bugs/1087145
<dholbach> hey rbasak
<dholbach> I asked this earlier - maybe you can help:
<dholbach> can somebody reply to https://code.launchpad.net/~racb/ubuntu/precise/squid-deb-proxy/cloud-archive/+merge/142088 or let me know if bug 1087145 can be closed?
<ubot5> bug 1087145 in squid-deb-proxy "maas proxy prevent nodes access cloud archive " [Undecided,New] https://launchpad.net/bugs/1087145
<dholbach> (trying to get the sponsoring queue cleaned up)
 * rbasak looks
<rbasak> dholbach: we've had the discussion on conffile updates in SRUs now, and my understanding is that we can do it here if needed. Daviey thought an SRU to Precise was appropriate for this which is why I prepared the merge. But I don't have permissions to create that sort of bug task. Daviey, do you think we still need this?
<rbasak> ^^
<dholbach> rbasak, did you see mvo's response in the merge proposal?
<rbasak> Yes
<dholbach> ok
<dholbach> thanks :)
<rbasak> The issue is that MAAS users in Precise are presumably affected, hence the desire for an SRU
<dholbach> sure
<roaksoax> dholbach: cold u please assing the bug to me :)
<roaksoax> shang__: soon after we fix few moew issues
<dholbach> done
<roaksoax> dholbach: thanks ;)
<roaksoax> rvba: howdy
<roaksoax> how can we delete all nodes from maas (without accessing the shell)
<allenap> roaksoax: I shall attempt to help! Do you mean from the web UI?
<roaksoax> allenap: from anywhere (just looking for a user friendly)
<allenap> roaksoax: Right, so "without accessing the shell" means "no hackery", so maas-cli is okay. Let me see...
<roaksoax> bigjools: ping
<bigjools> roaksoax: yo
#maas 2013-02-06
<roaksoax> bigjools: yo! so we have some serious DNS issues
<smoser> bigjools, i'm not able to reproduce roaksoax's problem locally
<smoser> bigjools, see what you think about my comment on https://bugs.launchpad.net/maas/+bug/1116700
<ubot5> Launchpad bug 1116700 in MAAS "MAAS 1.2 no longer auto generates a MAC based hostname" [Critical,Incomplete]
<EntropyWorks> once I add node and it MAC I chose the Power type IPMI. filled in the parameters and saved. should the node reboot right?  It never sees to do that. If I manually force it to boot off the network it grabs the pxelinux.0 but doesn't seem to have a pxelinux.cfg/default
<bigjools> EntropyWorks: maas is responsible for starting the node, if you start it manually it won't work
<bigjools> because the tftp server won't realise it's getting booted as the node is not in the right state
<jtv> Code review needed!   https://code.launchpad.net/~jtv/maas/bug-1116700/+merge/146835
<rvba> jtv: I'll have a look presently.
<jtv> Thanks.
<jtv> I see you have replaced the word "momentarily" in your lexicon.  :)
<rvba> Well, I still use it, just to piss Julian off.
<rvba> :)
<jtv> :)
<jtv> I had a call with another tall man called Julian earlier.
<rvba> I hope you managed to piss him off too.
<jtv> I almost got his name wrong because Edwards somehow claims the mental monopoly on that slot in my brain.
<jtv> So yes, I almost pissed him off.
<jtv> I should add "British expat" to round out the similarities.
<smoser> rvba, https://code.launchpad.net/~smoser/maas/lp1103716/+merge/146863 if you'd like
<rvba> smoser: sure, I'll have a look in a sec
<smoser> allenap, wanted to folow up a bit on fast path installer...
<allenap> smoser: What's up?
<smoser> how are we doing there.. do we/i/roaksoax need to do anything?
<allenap> smoser: We've not done anything on it, so, no, you don't need to do anything I don't think, we do. Is there a deadline?
<smoser> i dont have a deaadline, no
<smoser> but should we set one ?
<smoser> feature freeze is march 7.
<smoser> lets say i'd like to have necessary bits into maas by feb 21 ?
<allenap> smoser: That sounds sane to me. I'll email Julian, and CC you, as this is his decision.
<GTFr0> I'm having a problem with MAAS, where nodes stay in the "commissioning" state.  I think it maybe a NIC issue (not sure), but is there any way of verifying what NIC drivers are in the PXE booted kernel?
<GTFr0> (I originally through the issue was oath-related time skew, but I updated the ephemeral disk image to run ntpdate on boot, and that didn't seem to help)
<GTFr0> the kicker is that I can't find any indication in any logs on the MAAS server that any of the commissioned nodes are connecting
<smoser> GTFr0, are you able to see console logs ?
<smoser> if you enlisted correctly, then that is the same initramfs that is used for commissioning.
<smoser> (ie, same modules and dthe like)
<smoser> but it is quite possible that you're just missing nodes
<smoser> er... missing modules
<smoser> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1115710
<ubot5> Launchpad bug 1115710 in MAAS "Mellanox mlx4_en network driver is not automatically loaded" [Critical,Triaged]
<smoser> GTFr0, http://paste.ubuntu.com/1617332/ is how you can get yourself a new initramfs
<GTFr0> smoser: I added the nodes manually on the MAAS server
<smoser> then i suspect you are right.
<smoser> about the netowkr drivers
<smoser> if you can see console (serial consoel or vga) you can probably verify that
<GTFr0> hrm, not using Mellanox ethernet cards.  Oddly enough, these are Proliant Gen8 servers with i350 ethernet onboard
<GTFr0> afaik, i350 uses the standard igb driver, and if I do the standard Ubuntu server install, it sees the ethernet interfaces just fine
<GTFr0> and I don't think it's a DHCP issue, because the nodes will boot PXE
<GTFr0> smoser: when they boot PXE, I can see the console, but all I see is a ubuntu login prompt and am not able to login to get dmesg or other diagnostic info
<smoser> ah.
<smoser> GTFr0, ok. so if you pxeboot in commissioning and get to a ubuntu login prompt
<smoser> then you're definitely getting networking, as that is coming over iscsi
<smoser> you can "backdoor" the ephemeral image so you can login
<smoser> see info on how to do that at https://lists.launchpad.net/maas-devel/msg00817.html
<smoser> and https://code.launchpad.net/~matsubara/maas/ephemeral-img-debugging-doc/+merge/143927
<matsubara> thanks smoser, that ping reminded me I need to chase evilnickveitch for a review :-)
<evilnickveitch> oops! I'll do it tomorrow matsubara !
<matsubara> evilnickveitch, thanks!
<michaela_> what exactly is a node?
<EntropyWorks> bigjools: that makes sense. but it never sends the reboot command via IPMI. so I'm stuck now with 4 nodes waiting saying Commissioning.
<EntropyWorks> Nice to see the Mellanox mlx4_en is going to be added. that has been a major headache
<smoser> EntropyWorks, did you ever test that for me ?
<smoser> EntropyWorks, https://code.launchpad.net/~smoser/ubuntu/raring/kmod/lp-1115710/+merge/146760 .  put that mlx4.conf file in /etc/modprobe.d and then un-do any of the hacks you'vae done to get it loaded
<smoser> then reboot and make sure networking works as expected
<smoser> if we dont get that fix in, then you'll have the modules in your initramfs but you'll still not have networking because the _en wont get loaded.
<EntropyWorks> smoser: haven't tried that but in precise and quanta the initrd.gz didn't actually contain the drivers so just adding that to /etc/modprobe.d wouldn't have done it.
<smoser> understood
<smoser> i'm asking you to test an installed system, really.
<smoser> just to see if it comes up right if you dont have your custom udev rule or entry in /etc/modules (or some other manual way)
<EntropyWorks> my fix was this for straight pxe installing. http://goo.gl/PfOZd  now I'm looking at MAAS but haven't had much luck getting it to work.
<smoser> EntropyWorks, right.
<smoser> bu tafter you install
<smoser> then what?
<EntropyWorks> smoser: ok let me see about doing that. just grabbing your branch now bzr branch lp:~smoser/ubuntu/raring/kmod/lp-1115710
<smoser> all you really need is that mlx4.conf file into /etc/modprobe.d
<smoser> that should be enough
<smoser> EntropyWorks, and *thank you* for your help
<EntropyWorks> I will also rem  mlx4_en from /etc/modules which is something I believe i did in my late_command.sh
<EntropyWorks> these machine take about 3 min to reboot. so here we go
<smoser> right.
<EntropyWorks> humm. mlx4_core 000:05:00.0: comand 0xc failed: fw status 0x40
<smoser> EntropyWorks, where do you see that ?
<EntropyWorks> on the console
<smoser> dmesg ? are the modules loaded? (lsmod | grep mlx4)
<EntropyWorks> mlx4_core loaded but nothing else
<EntropyWorks> I have a meeting so be back after that and will chat again
<smoser> ok. can you try one other thing for me ?
<smoser> ok. thank you.
<EntropyWorks> smoser: back, what would you want next to be done?
<smoser> hooray
<smoser> ok
<smoser> so you rebooted and mlx4_en was not loaded, right?
<EntropyWorks> correct
<smoser> so lets replace that mlx4.conf with
<smoser> # mlx4_core should load mlx4_en (LP: #1115710).
<smoser> install mlx4_core /sbin/modprobe --ignore-install mlx4_core; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib
<ubot5> Launchpad bug 1115710 in MAAS "Mellanox mlx4_en network driver is not automatically loaded" [Critical,Triaged] https://launchpad.net/bugs/1115710
<smoser> and then reboot and see how we fare.
<EntropyWorks> so just that whole line.  and remove the softdep line?
<smoser> remove the soft dep
<smoser> and, yeah, one full line
<EntropyWorks> ok rebooting again
<EntropyWorks> nope. btw I wish I could see more than just this on the serial console http://paste.ubuntu.com/1618052/ . It hangs there until I get a login prompt
<EntropyWorks> still not luck loading the mlx4_en
<EntropyWorks> and in desg [    8.031717] mlx4_core 0000:05:00.0: command 0xc failed: fw status = 0x40
<EntropyWorks> no mention of mlx4_en in desg
<EntropyWorks> s/desg/dmesg/
<smoser> EntropyWorks, hm..
<smoser> so to see more on the console there, i tihnk you'll need to append console=ttyS0
<smoser> and that should function
<smoser> but then you wont see the stuff on the vga console
<smoser> can you try now rebooting after removing ml4x.conf entirely?
<smoser> err... first can you just try running 'modprobe mlx4_core'
<smoser> and seeing if networking _en gets loaded
<EntropyWorks> let me unload mlx4_core and mlx4_en again ( did that by hand just a moment ago )
<smoser> i just dont get it.
<smoser> with either of those mlx4.conf in place, i get both  modules loaded in my vm.
<smoser> (granted i dont have the hardware, but ... )
<EntropyWorks> doing that loaded both and the mlx4_ib and some other ib_mad ib_core
<EntropyWorks> but doesn't seem to do the trick when rebooting
<smoser> right.
<smoser> and if we remove that, we *do* get mlx4_core loaded, right?
<smoser> EntropyWorks, ^ . lets try
<smoser> removing the mlx4.conf file entirely
<smoser> and rebooting
<smoser> see if we still see the message in dmesg
<EntropyWorks> Ok removing that conf completely and will reboot
<smoser> EntropyWorks, i really appreciate your time
<smoser> i'm gonna have to run for a while.
<smoser> if you can just fill me in here.
<EntropyWorks> np, I'll be back tomorrow just /msg me and I will get it
<EntropyWorks> this mlx4 issue has been a problem for a long time for me.
<EntropyWorks> now if I could get MAAS to reboot a node... LOL
<EntropyWorks> rebooted and mlx4_core was loaded but not the others
<EntropyWorks> smoser: without /etc/modprobe.d/mlx4.conf only mlx4_core loaded
<EntropyWorks> humm...
<EntropyWorks> The directory containing the ephemeral images/info is missing (u'/var/lib/maas/ephemeral/quantal/ephemeral/amd64').  Make sure to run the script 'maas-import-pxe-files'.
<EntropyWorks> but I do have a dir in there /var/lib/maas/ephemeral/quantal/ephemeral/amd64/20121017/ with disk.img, info, initrd.gz, linux, tgt.conf
<EntropyWorks> and have run maas-import-pxe-files
<EntropyWorks> what gives
<bigjools> it takes a while for the re-scan of the files to happen
<bigjools> the message should disappear within a few minutes
<EntropyWorks> ok, I'll let it sit for a but and watch the logs. but why won't MAAS actually send the IPMI to the node to reboot? I have HP servers with iLO3 which is on a network reachable from the MAAS box.
<bigjools> EntropyWorks: maas doesn't do "reboot", it does power on and power off
<bigjools> power on when allocated and power off when deallocating
<EntropyWorks> ok so I need to power off the machine so it can power it on then...
<bigjools> EntropyWorks: what are you trying to achieve, at a higher level?
<EntropyWorks> higher lever. get a rack of HP ProLiant SL390s into MAAS
<bigjools> why do you need to reboot outside of the normal commissioning/allocation cycle?
<EntropyWorks> well I just installed MAAS on one machine in the rack. I then went to add a node.  but it never gets past Commissioning
<smoser> EntropyWorks, yeah, that message doesn't like to go away some times.
<smoser> EntropyWorks, and without any mlx4.conf, do we still see the "command 0xc failed" message in dmesg?
<EntropyWorks> I know the machine work and could see each other because I had my own PXE setup that with a patch initrd.gz for the mlx4
<EntropyWorks> smoser: I think you do [    7.955801] mlx4_core 0000:05:00.0: command 0xc failed: fw status = 0x40
<smoser> ok. so thats just a red herring then.
<smoser> but our changes to that ocnfig file seem not to be doing anything.
<smoser> just to be sure, you're p utting that file into '/etc/modprobe.d/mlx4.conf'
<smoser> right?
<EntropyWorks> bigjools: I need MAAS to actually do the commissioning/allocation cycle, it doesn't want to add any of the nodes.  2 nodes in this MAAS, 2 nodes offline
<bigjools> EntropyWorks: oh is this part of ther trouble you're having with the mlx4?
<EntropyWorks> bigjools: nope different problem
<bigjools> smoser: we probably ought to put together a faq "why is my node not commissioning?"
<EntropyWorks> smoser: when the file /etc/modprobe.d/mlx4.conf contains "install mlx4_core /sbin/modprobe --ignore-install mlx4_core; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib"  it does not load the modules at boot. it will load the modules when "modprob mlx4_core"
<EntropyWorks> by hand
<smoser> right.
<smoser> i just dont understand that.
<EntropyWorks> the way I've been making life work is adding mlx4_en to /etc/modules
<smoser> hm..
<smoser> maybe your initramfs has the 'modules' entry in it?
<smoser> and the driver is then getting loaded in initramfs when it doesn't have that file
<smoser> (i'm kind of grasping at this point, but i really dont understand it otherwise)
<smoser> i just did this:
<smoser> sudo modprobe "pci:v000015B3d00001010svfsdfbcfscfif"
<smoser> and i get the mlx4_en loaded
<smoser> (that matches one of the aliases)
<EntropyWorks> so I could open my initrd.img-3.5.0-23-generic and poke inside I guess
<EntropyWorks> but that should be stock.
<smoser> EntropyWorks, well, if you update-initramfs
<smoser> well, since you had that /etc/modules entry
<smoser> it could/might get pulled into there
<smoser> on an update-initramfs
<EntropyWorks> I have not run that command yet
<smoser> right
<smoser> but it happens on kenrle install and such
<EntropyWorks> humm. the kernel is installed before I add to /etc/modules
<EntropyWorks> I'm betting
<smoser> well, yes, but it gets updated lots of times.
<smoser> do this:
<EntropyWorks> yep I do it in my late_command.sh
<smoser>  lsinitramfs /boot/initrd.... | grep mlx
<smoser> and i suspect/hope we see it there.
<smoser> my theory is that if its there, then when modprobe comes up in root, it doesn't find it.
<smoser> well, its already been loaded
<smoser> so nothing tries to load it
<EntropyWorks> it is there. both mlx4_core.ko and mlx4_en.ko
<smoser> right. but you do not have the mlx4.conf in there.
<smoser> so, put mlx4.conf in place
<EntropyWorks> correct
<smoser> then 'update-initramfs -u -k $(uname -r)'
<smoser> and reboot
<smoser> and again, thank you
<smoser> i have to run
<smoser> will check in later.
<EntropyWorks> so I should put the file back in /etc/modprobe.d/mlx4.conf and then just run the command 'update-initramfs -u -k $(uname -r)'
<EntropyWorks> bigjools: having some info for why a node is not Commissioning would be wonderful
<bigjools> EntropyWorks: yes!  It's a complicated area, a whole lot of stuff can go wrong :(
<EntropyWorks> doing a tcpdump I  know MAAS is sending something to the IP of the IPMI device. but what ever it is doesn't power on the server.
<bigjools> was the bmc detected at enlistment?
<bigjools> or are you configuring manually?
<EntropyWorks> bigjools: using ipmitool -I lanplus -H 10.X.X.X -U Administrator -P foobar chassis power on
<EntropyWorks> does turn the server on.
<bigjools> ok compare with the power script that maas is using
<EntropyWorks> where should I look for that :-)
<bigjools> provisioningserver/power/templates IIRC
<EntropyWorks> humm. I wonder if MAAS is expecting IPMI to be on the same interface as the machines network card. my IPMI is in iLO which is an out of bandwidth daughter card. which is on a different netwok than the machines NIC.  the iLO has an IP address already and doesn't need one from MAAS
<bigjools> no it doesn't expect that
<bigjools> it just stores IP address
<EntropyWorks> ok cool
<bigjools> I expect it's a v1 / v2 problem
#maas 2013-02-07
<EntropyWorks> it looks like your using bmc-config
<EntropyWorks> which I don't know if I can get it to work right with iLO3
<bigjools> that's roaksoax's work :)
<bigjools> he would know better than I
<EntropyWorks> bigjools: humm /usr/share/maas/preseeds/enlist_userdata looks to be the place. but if I'm reading this it wants to modify the IPMI setting not just power off and on a machine
<bigjools> that's the detection stuff
<bigjools> not the power template
<bigjools> you'll have an ipmi.template somewhere
<EntropyWorks> ah
<EntropyWorks> bigjools: I think I got it! without modify the template
<EntropyWorks> modified /etc/freeipmi//freeipmi.conf and added driver-type LAN_2_0
<bigjools> nice
<EntropyWorks> I think the IPMI config in the webgui should have a choice since someone may be in a mixed env
<bigjools> well it is on a per-node basis
<bigjools> we used to have more choice for ipmi IIRC but roaksoax changed it
<EntropyWorks> yes and that is why it should be in the webgui so you could provision a driver-type LAN and then add another node and do driver-type LAN_2_0
<EntropyWorks> with out login into the MAAS server and modifying that /etc/freeipmi/freeipmi.conf to override the default of LAN
<EntropyWorks> the underlying tool is ipmipower
<EntropyWorks> great... note to self do not delete the last MAC address for a node. now I can't select it to remove it...
<bigjools> oops
<bigjools> please file a bug
<bigjools> (you could add a dummy mac)
<EntropyWorks> from the webgui I can't I only see the "(name)"  which isn't a link
<EntropyWorks> when a mac is there its "93:73:71:73:c1:47(name)"
<EntropyWorks> and you have to click the mac address to delete or modify the node
<bigjools> damn
<smoser> EntropyWorks, so you still around ?
<smoser> yeah, what i was saying was to just try putting that file into place and update-initramfs
<smoser> and then reboot
<EntropyWorks> yes
<EntropyWorks> ok I think that worked. I'm rebooting that machine again just to check
<EntropyWorks> bigjools: the link is there in the HTML  but nothing for it to underline as a link.
<bigjools> EntropyWorks: ok thanks
<EntropyWorks> https://bugs.launchpad.net/maas/+bug/1117875
<ubot5> Launchpad bug 1117875 in MAAS "Removed MAC leaves node but can't edit or remove" [Undecided,New]
<smoser> EntropyWorks, ok. this makes sense now.
<smoser> at least i think.
<smoser> after adding that rule (and removing entry in /etc/modules) we need to run 'update-initramfs'.
<smoser> otherwise, the module would get loaded in the initramfs
<smoser> and then the modprobe.d rules that we were applying only are put into affect when that module (mlx4_core) is installed.
<smoser> and since it was already getting installed in the initramfs, the changes were not doing anything.
<smoser> so...
<EntropyWorks> $ lsinitramfs /boot/initrd.img-3.5.0-23-generic | grep mlx4
<EntropyWorks> etc/modprobe.d/mlx4.conf
<EntropyWorks> lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/mellanox/mlx4
<EntropyWorks> lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
<EntropyWorks> lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
<smoser> i'd really apprecate it if you would put the soft dep one in place
<smoser> then update-initramfs
<EntropyWorks> ok
<smoser> then reboot
<smoser> and, cross your fingers, please
<smoser> :)
<EntropyWorks> humm
<EntropyWorks> $ update-initramfs -u -k $(uname -r)
<EntropyWorks> update-initramfs: Generating /boot/initrd.img-3.5.0-23-generic
<EntropyWorks> FATAL: modprobe: softdep dependency loop encountered inserting mlx4_core
<EntropyWorks> still want me to reboot?
<EntropyWorks> it appears that it did not add mlx4_en to the image...
<EntropyWorks> rebooting anyway...
<smoser> hm..
<smoser> shoot. i remember that.
<smoser> that is odd.
<EntropyWorks> only the mlx4_core was loaded
<EntropyWorks> so that didn't work
<bigjools> EntropyWorks: which version of maas are you using?  I cannot reproduce your bug
<smoser> right. EntropyWorks so lets try the whole update-initramfs process with the 'modprobe'.
<smoser> er... with the 'install version
<smoser> of mlx4.conf
<EntropyWorks> bigjools: 0.1+bzr1269+dfsg-0ubuntu1 i'm running on quantal 12.10
<bigjools> EntropyWorks: from the stable PPA?
<bigjools> oh is that the version in quantal
<EntropyWorks> yea its the default in quantal I'm not using another PPA except to mess around with salt
<bigjools> for me, it's the names that are linkified on the node list
<bigjools> not the macs, so I can click on it happily
<EntropyWorks> weird
<bigjools> can you try the PPA version?  it fixes a LOT of bugs and is about to get SRUed
<EntropyWorks> I only installed MAAS yesterday on a clean system too
<bigjools> https://launchpad.net/~maas-maintainers/+archive/stable/+packages
<EntropyWorks> smoser: modifying the mlx4.conf again and initramfs  for you and reboot
<EntropyWorks> bigjools: nice! this one has the @%#$$T@# IPMI settings that was causing me to scream at my monitor
<bigjools> EntropyWorks: \o/
<bigjools> I wondered why you were having trouble!
<bigjools> keep forgetting all the old versions differences
<EntropyWorks> *head to desk*
<smoser> EntropyWorks, sorry for your pain.
<smoser> did we get our modules loaded ?
<smoser> i'm about to push this and ask for it to be loaded into raring.
<EntropyWorks> smoser: well if it wasn't for that you wouldn't have me to trouble shoot the mlx4 issues so... i don't mind since that is one that really pissed me off for a while
<smoser> yeah, fwiw, people generally seem to think taht the driver itself is just being stupid here.
<smoser> that it isn't automatically loading stuff itself.
<EntropyWorks> that worked.
<smoser> woowhoo!
<EntropyWorks> now just have to fix the initrd.gz that will get used for MAAS too :-)
<EntropyWorks> $ lsmod | grep mlx4
<EntropyWorks> mlx4_en                63080  0
<EntropyWorks> mlx4_core             173435  1 mlx4_en
<smoser> oh
<smoser> and one more thing, EntropyWorks
<smoser> you have functional network based on that, right ?
<smoser> (just a little thing like "does it actually give you network" :)
<EntropyWorks> yes networking works.
<EntropyWorks> I wanted to see what happens if I unload the both and reload mlx4_en
<EntropyWorks> that added more modules then before
<EntropyWorks> mlx4_ib                57930  0
<EntropyWorks> mlx4_en                63080  0
<EntropyWorks> mlx4_core             173435  2 mlx4_ib,mlx4_en
<EntropyWorks> ib_mad                 47134  3 mlx4_ib,ib_cm,ib_sa
<EntropyWorks> ib_core                82363  7 mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
<EntropyWorks> what i did was
<EntropyWorks> modprobe -r mlx4_en
<EntropyWorks> modprobe -r mlx4_core
<EntropyWorks> modprobe mlx4_en
<EntropyWorks> well past time to leave the office. smoser catch you tomorrow if you need more mlx4 debugging
<smoser> thanks again.
<smoser> EntropyWorks, right. i tested that.
<smoser> seems generally functional.
<smoser> but thanks for thinking
<EntropyWorks> what ports on the MAAS server need to be open by the way? would be nice to have a ufw config in /etc/applications.d/
<EntropyWorks> next problem I have to solve is what image has the /etc/apt/sources.list or where I need to add the http_proxy info for MAAS to work in my env
<sam_one> morning kool kats
<EntropyWorks> anyone have a quicker way to add machines then doing it via the webgui?
<EntropyWorks> the nodes now have gotten Allocated but since they appear to not have working networking I'm suck.  I can't delete the node from the gui either
<EntropyWorks> lovely
<EntropyWorks> btw this command just fails: sudo maas createsuperuser
<EntropyWorks> http://paste.ubuntu.com/1622243/
<bigjools> EntropyWorks: eummmm that's really odd, I've never seen anything like that. Something is screwed in your system.
<bigjools> you can add machines via the command line or API
<EntropyWorks> bigjools: I was able to remove them via the maas-cli
<bigjools> great
<EntropyWorks> the createsuperuser thing however I have no clue.
<bigjools> me neither
<bigjools> my only guess is that somehow your PPA install clashes with the quantal package
<bigjools> can you file a bug please and supply as much info as you can about your installation (versions etc)
<EntropyWorks> maybe. it was happening before in the quantal packages too
<EntropyWorks> sure
<bigjools> oh weird
<bigjools> matsubara, hi
<EntropyWorks> bigjools: humm yea I'm think something just is wrong with my server. not sure what. going to just reload from scratch and try again.
<bigjools> EntropyWorks: try LC_ALL=C sudo maas createsuperuser
<EntropyWorks> nope
<bigjools> you're missing locale info
<bigjools> is it even set?
<EntropyWorks> wonder if my python is screwed someplace
<bigjools> echo $LC_ALL
<EntropyWorks> not set
<bigjools> ok C doesn't work
<bigjools> set it to a valid locale
<bigjools> en_US for example
<bigjools> bugger
<bigjools> you need sudo -E as well
<bigjools> preserves env
<EntropyWorks> yea, but no good either
<EntropyWorks> something messed up with my install. check this out
<EntropyWorks> $ maas-cli
<EntropyWorks> Error: no arguments given.
<EntropyWorks> Run /usr/lib/python2.7/dist-packages/maascli/__main__.py --help for usage details.
<EntropyWorks> I'm got to do the sane thing and nuke this machine and try again :-)
<bigjools> so, LC_ALL=en_US sudo -E maas createsuperuser
<bigjools> doesn't work?
<EntropyWorks> ah! yes that prompted me!
<EntropyWorks> yea I see when I run locale
<EntropyWorks> LC_ALL=
<EntropyWorks> and  LANGUAGE=
<EntropyWorks> so thats causing me the issue i guess
<EntropyWorks> just reset it to en_US.UTF-8 logged out then back in and tada command works
<bigjools> cool
#maas 2013-02-08
<EntropyWorks> which is the better doc to follow? http://maas.ubuntu.com/docs/quantal/install.html  or https://maas.readthedocs.org/en/latest/_build/
<bigjools> the former
<EntropyWorks> thanks
<bigjools> no problem, I am interested in your progress
<EntropyWorks> can I get away without have the squashfs image for quantal? since I had to set IMPORT_SQUASHFS=0 to get 'sudo maas-import-pxe-files' to complete with out error
<bigjools> Yes, it should not be getting downloaded, it doesn't work
<EntropyWorks> since I updated my initrd.gz to have the mlx4 drivers looks like i have some proxy issues since the commissioning needs to apt-get some items. is there a template that I could change so I could use my own internal mirror?
<bigjools> if you're using the latest PPA package it has support to set a mirror or a proxy in the config page
<roaksoax> robbiew:
<roaksoax> err
<roaksoax> sorry
<roaksoax> rvba:
<roaksoax> ping
<EntropyWorks> bigjools: so the issue wasn't the proxy. looked closer it is bind / named  requiring dnssec-validation.
<EntropyWorks> bigjools: my setup also required me to setup a forwarder for dns.
<EntropyWorks> bigjools: dang, dns works now but that didn't work for me. the first Commissioning stage doesn't seem to use the proxy settings
<EntropyWorks> I need to find the initrd.gz and most likely tweek it
<EntropyWorks> i would like to change which apt mirror its using since I have my own local mirror
<EntropyWorks> ok ran 'juju bootstrap' and machines rebooted and 'Allocated to admin' but of course no mlx4_en driver so no networking... fustrating
<EntropyWorks> what image is this booting?
<EntropyWorks> I have tried to modify these but that didn't work
<EntropyWorks>  /var/lib/maas/ephemeral/quantal/ephemeral/amd64/20121017/ initrd.gz
<EntropyWorks>  /var/lib/maas/tftp/amd64/generic/quantal/install/initrd.gz
<EntropyWorks>  /var/lib/maas/tftp/amd64/generic/quantal/commissioning/initrd.gz
<EntropyWorks> then tried the ephemeral disk.img too but still nada
#maas 2013-02-10
<bigjools> smoser, roaksoax: can you confirm the solution mentioned here? https://bugs.launchpad.net/maas/+bug/1119594
<ubot5> Launchpad bug 1119594 in MAAS "Node install fails when sda has pre-existing md raid arrays" [Undecided,New]
#maas 2014-02-03
<tomixxx> hi, can i add the maas-server itself as a node=?
<tomixxx> hi, what can i do that the nodes in my maas-environment download their things from the maas-server?
<tomixxx> i cannot resolve any package like archive.ubuntu.com
<tomixxx> the problem is, that juju does not work either, in case package resolving does not work...
<tomixxx> hi, i have the following network configuraiton: http://pastebin.ubuntu.com/6867969/
<tomixxx> my maas-server is connected to its nodes via eth0 but my nodes cannot download any package
<tomixxx> eth1 has access to internet
<tomixxx> so my question is, do i need to bridge both interfaces in some way?
<21WACCDQH> tomixxx: http://askubuntu.com/questions/320121/simple-port-forwarding
<roaksoax> bigjools: howyd!! crochet has just been accep[ted in universe, so should be in the archives in the next few hours
<bigjools> roaksoax: hurray!
<bigjools> I presume there's a MIR for it too?
<roaksoax> bigjools: indeed, will be filling a MIR tomorrow
<bigjools> coolio
<roaksoax> cool
<roaksoax> anyuways, i'm done for the day
<roaksoax> ttyl!
#maas 2014-02-04
<tomixxx> @bigjools, are u online?
<tomixxx> bigjools: are u online?
<tomixxx3> hi, when i try to assign multiple charms to the same machine, i get an error in agent-state-info. it seems that the machine cannot download some things from http://cloud-images.ubuntu.com. Please have a look on the output of the terminal when i call "juju status": http://pastebin.ubuntu.com/6873041/ As u can see in the output, i have tried it now several times but every time the lxc container failed to create
<allenap> tomixxx3: Try asking again on #juju.
<allenap> tomixxx3: Ah, you already have.
<ticking> Hey, is there a recommended Maas Version in combination with a consistent Dokumentation somewhere?
<tomixxx> the maas-server itself cannot be an additional node, right?
<roadmr> hello! I added some nodes to my maas but they weren't added in /etc/bind/maas, is there a way to force rewriting of the zone.master file to reflect currently known nodes?
<roadmr> why is my /etc/bind/maas/zone/master out of sync with the nodes I actually have in my maas cluster? :(
<roaksoax> roadmr: did you change the name of your maas cluster zone?
<roadmr> roaksoax: I added a zone; I had "default" and I added a "broken" zone (currently has no nodes)
<roadmr> but to be clear, "default" is still there
<roaksoax> roadmr: right, but in the cluster controller config, you can change the name of the zone
<roaksoax> by default it says 'master'
<roadmr> roaksoax: oh! no, that hasn't changed
<roaksoax> roadmr: go ahead and change it
<roaksoax> that should fix things
<roadmr> Can't rename DNS zone to master2; nodes are in use.
<roadmr> interestingly, 5 out of my 9 nodes *are* in the zone.master file, but the rest aren't :(
<roadmr> roaksoax: is there another way to force resync of the dns zone file? even if on the command line. I just don't want to have to add them manually :( (lazy bum that I am)
<roaksoax> roadmr: try restarting the cluster celery process, check rabbitmq queues to see if they are filled with messages?
<roaksoax> roadmr: check that all the maas related processes are up and running
<roadmr> roaksoax: hmm, so I rebooted one of the missing nodes and when that one came up, the other missing ones also appeared in the zone files. Weird!
<roaksoax> roadmr: indeed
#maas 2014-02-05
<jtv> rvba: does Django give Network.clean() any way to know whether & which field validations may have failed?
<jtv> Or do I need to either re-compute or cache the knowledge inside the model object?
<rvba> jtv: not sure about that.  I would think so because model validation is modelled after form validation.
<jtv> I think I can just generate a double error in this case â one for the netmask field, and one global one.  Not ideal, but...
<rvba> jtv: are you landing the validation branches?
<jtv> Yes, but there were some conflicts to be resolved.
<rvba> Ah, okay.
<jtv> Hence the Django problems: even if all your fields fail validation, Django still wants you to validate your model.  Even if all of _that_ fails, it still wants you to validate uniqueness.
<rvba> That's weird.
<rvba> Are you sure that's how it works?
<rvba> Doesn't make sense.
<jtv> Pretty sure â from observation _and_ from reading the code.  :)
<jtv> The documentation doesn't mention the whole issue AFAICS.
<jtv> Have a look at /usr/lib/python2.7/dist-packages/django/db/models/base.py
<jtv> Look for full_clean.
<jtv> And be warned that exception.update_error_dict(errors) means âexception, add your errors to my errors dict,â not the other way around.
<rvba> yep
<rvba> Okay, you're right, apart from the fact that only fields that didn't fail the previous validation go through uniqueness validation.
<jtv> Ah, that's what "exclude" is for!  Thanks.
<tomixxx3> hi, if i want to add virtual nodes to my maas-environment, how much effort is this compared to add a single physical machine which is quite simple
<jtv> tomixxx3: it's doable, but maas won't actually _create_ the VMs for you.
<jtv> So you'll have to take care of that part yourself.
<tomixxx3> jtv, ok i udnerstand
<tomixxx3> jtv: would be nice if maas would do this :-)
<jtv> Yes, and believe me it has come up!
<jtv> But it's not really a priority since the way we see it, it's mostly something you'd want for debugging.
<jtv> The point of MAAS is bare metal, after all.  :)
<tomixxx3> jtv: at the moment, i have still the problem that my nodes cannot access the internet
<jtv> tomixxx3: was it not an option to set up routing with your server as a bridge?
<tomixxx3> jtv: yes, but i dont know exactly how to achieve this
 * jtv searches...
<tomixxx3> jtv: if eth0 is the private network and eth1 connects the server to the internet, can i put the route-statements to eth0?
<tomixxx3> jtv: and what aobut the "router ip" in the cluster interface settings?
<tomixxx3> jtv: i guess it is not sufficient to simply add the IP of eth1 ?
<jtv> The router IP is not really a setting... just let MAAS discover it during commissioning.
<tomixxx3> jtv: ok
<jtv> Hmm...  A search came up with a video guide.  Not the ideal format for this stuff, but it may help: http://www.irongeek.com/i.php?page=videos/ethernet-bridge-ubuntu-linux
<jtv> I think many people talking about setting up a bridge mean a bridged ethernet interface for virtual machines...
<jtv> tomixxx3: if you need NAT, it's probably a bit more complicated.
<tomixxx3> jtv: but when i bridge eth0 and eth1, does my prviate network resp. the dhcp server of my private network NOT interfere the internet-network?
<tomixxx3> jtv: i mean the university network?
<jtv> It depends a bit: did you use a network range that is unique within the university network?
<jtv> If you hook up routing where multiple machines could have the same IP address, then that would cause trouble of course.  And it just plain wouldn't work very well.
<tomixxx3> 10.0.0.100 to 10.0.0.200
<jtv> Is that range still free on the university network?
<tomixxx3> dunno
<tomixxx3> jtv: assuming, i bridge eth0 and eth1: should i use dhcp then or static configuraiton in "interfaces" file
<jtv> Important to know.  If you have a range that the admins give you to use as part of the uni network, then it's relatively simple â you can just hook up the networks.
<tomixxx3> jtv: because eth0 is configured static now, and eth1 is configured dhcp
<jtv> eth0 should remain static, but the nodes should use dhcp.
<jtv> That's because if your server also becomes the gateway to the internet, it had better stay at one fixed address.  :)
<tomixxx3> jtv: so the bridge looks like: auto br0
<tomixxx3> iface br0 inet dhcp
<tomixxx3>   bridge_ports eth0 eth1
<jtv> I must admit, I no longer have any knowledge of how this part works.
<jtv> It's been a long, long time.
<tomixxx3> jtv: or: i need simpley to add the line "bridge_ports eth0" to eth1 ?
<jtv> If you ask the admins "hey can I use 10.0.0.100 thru 10.0.0.200" (actually it works in powers of two, so probably not exactly that range) and they turn a light grey, breathe in sharply, and start to swear at you, then you can't just bridge and you'll need NAT (or a different range).
<tomixxx3> hmm
<tomixxx3> and with NAT, i can NEVER destroy my uni network?
<jtv> Never say never!  If you work hard at it, I'm sure there's a way.  It's not my position to make promises.
<jtv> (In case any lawyers are listening...)
<tomixxx3> jtv: i mean, when i route dhcp-broadcasting out of my prviate network, does this not harm the network?
<jtv> The main thing is never to pass DHCP requests between your network and the university's, and to make sure that the address range you expose externally is available for your use.
<jtv> You should not route _that_ traffic.
<jtv> I'm not sure (did I mention it was a while ago?) but I don't think you'd usually want to route broadcast traffic anyway.
<tomixxx3> jtv: so i should avaoid that broadcast-stuff is send?
<tomixxx3> jtv: some kind of package filter?
<jtv> Yes.
<tomixxx3> jtv: damn, it get quite complex then.....
<jtv> Yes, networking is hard.  :)
<tomixxx3> jtv: my vision was that maas takes care of this things :D
<jtv> I guess that's why software-defined networking is becoming more popular.
<jtv> I bet it'd be easier with an off-the-shelf router, but that costs you a router as opposed to a network card.
<jtv> Maybe ufw ("uncomplicated firewall") can make this easier.
<tomixxx3> jtv: ok, if we go back the roots: i have my 3 nodes and they are connected to the university network. maas is not managing dhcp and dns. the nodes has accees to the internet but the problem is, that they boot from another dhcp server, namely from that of the university
<tomixxx3> jtv: is there not a simple "trick" to let nodes boot from the maas-server images?
<tomixxx3> jtv: ahh
<jtv> ahh?
<jtv> There is a way to do that, but it requires a configuration tweak to the main dhcp server.
<tomixxx3> jtv: ok, i thought the following: i boot from the maas-dhcp server, as soon as the nodes are deployed to maas in state "ready", i turn off the dhcp settings of maas server, nodes get now IPs from university and has access to internet
<jtv> The MAAS documentation has something about this, but it will depend on the type of DHCP server.
<jtv> Ooo clever, but also slightly nasty.  Two problems with it that I can see:
<jtv> 1. Other machines belonging to innocent PhD students may request addresses just while you're doing this.
<jtv> 2. When a machine has an address from DHCP, it will need to talk to the DHCP server periodically to renew it.
<jtv> Ah, here's the documentation I was looking for.  Not a lot, because it all depends: http://maas.ubuntu.com/docs/configure.html#manual-dhcp-configuration
<jtv> Actually I'm not even sure that's complete...  I thought there had to be a setting to make PXE get the images from the MAAS server, too.
<tomixxx3> jtv: about 1.) i don't get it. all 3 nodes in my cloud get unique IP addresses as far as i know
<tomixxx3> about 2.) so the machine is not able to recognize that the dhcp server changed?
<jtv> True... but then it'd probably change its IP address.
<jtv> (Also, you need this while booting the machine, not just while commissioning it)
<jtv> A node in a MAAS is not supposed to change its IP address... that's an assumption we've used in several places.
<tomixxx3> jtv: and if i let the node get the same IP like from the other dhcp server? ^^
<jtv> If you can make it do that, then I think that would work.
<tomixxx3> jtv: puhh
<jtv> Not very simple either.  :/
<tomixxx3> jtv: because i can only define a range and not concrete IP addresses?
<tomixxx3> jtv: i will ask my admin if i can have a free range for my nodes, so that i can bridge the network interfaces. but assume i have bridged them: is then guruanteed that my nodes will access my prviate dhcp server and not the university- dhcp server again?
<tomixxx3> jtv: but at least, i guess, i'am right if i say: AT LEAST the JUJU bootstrap node need a internet-connection?
<jtv> tomixxx3: I don't _think_ it does, if your juju version is recent enough...  Then I think the machine where you run Juju needs internet.
<jtv> IIRC it will upload its tools to MAAS, which has a file store.
<jtv> Of course any machine where you want to issue juju commands to the environment will have to be able to access it.
<tomixxx3> jtv: the problem is, i want to deploy multiple juju charms on a single node, so i need lxc-containers. but creating those lxc containers fails because the node answers" cannot get file http://cloud-images.ubuntu.com/..."
<jtv> Ah!  I'm not sure but I think that may be lxc needing access to the archive.
<jtv> What should work is to make it use your MAAS server as an http proxy.
<tomixxx3> Yeah, sounds cool
<jtv> Should also give you the benefit of caching!
<jtv> Although... university network in the heart of Europe... probably not that important to you.  :)
<tomixxx3> jtv: yeah, good network here :D
<tomixxx3> jtv: but, nevertheless, i take it if it is for free ;)
<jtv> :)
<tomixxx3> jtv: do u know how i can configure juju so that it uses the maas server as a proxy?
<jtv> No, sorry, haven't tried that.
<tomixxx3> kk
<tomixxx3> jtv: referring this article: http://askubuntu.com/questions/310153/how-do-i-get-maas-and-juju-to-recognize-proxy-settings-in-a-deployment is "my.proxy" the IP of the maas server?
<jtv> Sounds like yes... let me read that.
<tomixxx3> jtv: k, ty
<jtv> tomixxx3: that looks like exactly what you need... but Gavin says he wrote an updated answer that makes it easier.
<jtv> "In the settings page (the cog near the top right)" etc.
<jtv> He uses http://example.com/ as the example, but your MAAS server should go in its place.
<jtv> The format is: http://<host>:<port>
 * jtv looks up the port
<jtv> I think it was port 3128... you can try that out by telnetting to it:
<jtv> telnet <maas-server-address> 3128
<jtv> Yes, that would be port 3128 by default.
<tomixxx3> it says "connected to 10.0.0.9"
<tomixxx3> jtv: ok
<jtv> Yes it won't tell you much, but if it's listening there, good chance it's the proxy.
<jtv> (Fun tip: many TCP-based internet protocols are text-based so you can try them out manually through telnet.)
<tomixxx3> hehe
<tomixxx3> do i have to reboot now sth, after clicked "save"?
<jtv> errr
<jtv> It could be that you need to re-commission the nodes with that setting in place.
<tomixxx3> oh , hopefully, it fixes the lxc-creation problem then xD
<jtv> Let's hope so!
<tomixxx3> ok, i will do it... as u said: let's hope the best
<tomixxx3> cause it tried to deploy a juju charm and i got the same error
<jtv> Yeah I think that method would take hold while installing the node.
<tomixxx3> kk at least while booting the nodes will become able to resolve archive.ubuntu.com etc. so at least one thing is "fixed" ;)
<jtv> Progress!  :)
<tomixxx3> yeah
<tomixxx3> ahh the node cannot resolve again "http://archive.ubuntu.com/..
<jtv> :(
<tomixxx3> there is still a chance for Juju to succeed ^^
<tomixxx3> btw. "stopping landscape ..." is the only task always "FAIL"
<jtv> May be worth having a look in Landscape's logs then.
<tomixxx3> k, i get the following now: "bad archive mirror" when i deploy juju bootstrap node
<tomixxx3> the http-proxy: which IP address is it? the one of the interface connected to the nodes or the the IP of the interface connected to uni-network?
<jtv> It's the one on the nodes' side.
<tomixxx3> ok :(
<jtv> Because this tells the nodes: "here's where your http proxy is."
<jtv> I'm afraid I'll have to leave it at that for tonight!
<jtv> TschÃ¼Ã!
<tomixxx3> k, ty for help and hints!
<jtv> np
<tomixxx3> nooooooo, now i always get this bad archive mirror when i try to isntall the bootstrap juju node.... even if i remove the http proxy....
<tomixxx3> matsubara: hi, are u online?
<matsubara> tomixxx3, hi, I'm online. How can I help? (I'm a bit busy so expect some delay in my answers)
<tomixxx3> matsubara: ty! i'am a little bit depressed: i have set the option "http proxy" in the maas dashport to be the maasserver+port 3128. then i tried to add nodes again. They become ready. however, when i try to setup the juju bootstrap node, i get a "bad arhcive mirror" now while booting.
<tomixxx3> matsubara: and the problem is: even if i remove the http proxy in the dashboard, from now, i always get this "bad archive mirror" excpetion when a node in the cloud boots
<tomixxx3> matsubara: i tried to set the proxy, because i need juju to create lxc-containers and they need internet-connection but the nodes have no internet-connection, so my hope was that the maas-server could be the proxy...
<tomixxx3> matsubara: i dont know if u remember my old problem: private network (3 nodes) + university network to get internet-connection
<matsubara> tomixxx3, so, let me see if I understand, you get the 3 nodes in the Ready state, then run juju bootstrap and juju returns with a "bad archive mirror" error? Or is that in the node console/screen? How did you setup that proxy? Is it a regular squid proxy or a squid-deb-proxy? (given the port it looks like it's a squid proxy)
<tomixxx3> matsubara: yes, 2 nodes in ready state, 1 node is maas-server, the node which become the juju bootstrap server get stuck with a "bad archive mirror" in node console screen.
<tomixxx3> matsubara: i set the proxy via the maas-dashboard : settings -> "Proxy for HTTP and HTTPS traffic" -> http://10.0.0.9:3128
<tomixxx3> matsubara: in order words: the address of the proxy was the address of the maas-server
<matsubara> tomixxx3, and how did you configure that proxy running on 10.0.0.9:3128?
<tomixxx3> matsubara: i did not configure it at all...
<tomixxx3> matsubara: i tried telnet 10.0.0.9 3128 and it was able to connect...
<matsubara> tomixxx3, what do you get as the responde from that telnet? If you point your browser to  http://10.0.0.9:3128 do you get a squid error message?
<tomixxx3> matsubara: i get "the requested URL could not be tretrieved"
<matsubara> tomixxx3, can you paste /var/log/squid/access.log ?
<tomixxx3> kk
<tomixxx3> squid3?
<matsubara> yep, sorry
<matsubara> and error.log too
<tomixxx3> np
<tomixxx3> http://pastebin.ubuntu.com/6880076
<matsubara> but I think you'd need to configure squid properly. I don't think MAAS, by default configures the standard proxy for you, just the squid-deb-proxy, which uses port 8000, IIRC
<tomixxx3> matsubara: and even, if i remove the http proxy, remove the nodes from maas, and re-commission all nodes, i get this "bad archive mirror" in console output of the nodes
<tomixxx3> matsubara: i never got that before!
<tomixxx3> matsubara: sorry, i mean, i get this message in the console output of the juju bootstrap node
<tomixxx3> matsubara: i have not tried so far the other nodes
<matsubara> tomixxx3, sorry, don't know what could be wrong, maybe this might help: http://askubuntu.com/questions/295999/maas-juju-still-get-bad-archive-mirror
<tomixxx3> matsubara: i have already written through this article
<tomixxx3> matsubara: i always had "manage dhcp and dns"
<matsubara> tomixxx3, does your maas server route traffic to the internet for your nodes? Maybe your nodes are trying to reach the external world through the MAAS server but that's not working?
#maas 2014-02-06
<jtv> bigjools: I'm thinking to address the issue by accepting zero as a vlan tag, but normalising it to None.
<bigjools> jtv: evil, but.... Django
<jtv> In a word.
<jtv> bigjools: by the way, have you seen those over-frequent DHCP config rewrites (and dhcpd restarts) lately?  Or did they go away?
<bigjools> jtv: I have not checked, I can do so in a  bit
<jtv> I'd be grateful if you could.
<bigjools> will build a new package for my hp
<jtv> And by the way, if you have time for a review... https://code.launchpad.net/~jtv/maas/zero-vlan-tag-is-not-unique/+merge/205073
<bigjools> jtv: I don't suppose I can persuade you to go and edit the last migration that set this up?
<jtv> You can, if you want to.
<bigjools> Having multiple migrations for the same new model is ugly as hell
<jtv> I went with the safer option...  If you want me to play dangerous with the schema, sure.  :)
<bigjools> meh
<jtv> bigjools: updated.
<bigjools> jtv: cheers
<jtv> bigjools: one thing I didn't get around to bringing up... vlan tags: hex or dec?
<bigjools> jtv: either I guess, prefix with 0x if you want hex
<jtv> If you want to get fancy, I say: separate card!
<bigjools> jtv: your signal fix has not worked
<bigjools> still getting lease write jobs after probes
<jtv> Arrrrrgh.
<jtv> O!
<jtv> Wait...
<jtv> No.  For a moment I was hopeful that I'd simply forgotten to remove something.
<jtv> bigjools: can you verify that you're running a version where src/maasserver/dhcp_connect.py connects dhcp_post_change_NodeGroupInterface to a _field change_ on NodeGroupInterface, not to a save?
<bigjools> jtv: I was running the daily build
<bigjools> 1.4+bzr1853+dfsg+1891+231~ppa0~ubuntu14.04.1
<bigjools> so r1891
<bigjools> I powered down the machine now so can't check code easily
<jtv> From trunk, right, not 1.4?
<bigjools> yup
<jtv> Ohdearohdearohdear
<jtv> Hmm... was there really never a test for "does nothing if fields don't actually change"?
 * jtv writes one
<bigjools> :)
<jtv> Nope, nope, that passes.
<jtv> So something else is triggering that signal.
 * jtv throws some more hate at this use of signals
<jtv> bigjools: I do see that leases are uploaded every minute as per the celerybeat schedule...
<bigjools> jtv: yes and then it writes dhcp and restarts server
<jtv> Could you try it with a longer unconditional-dhcp-lease-upload schedule time?
<bigjools> can't today will come back tomorrow
<jtv> OK.
<jtv> It's just that this may be a side effect of the leases upload.
<jtv> Which happens once a minute.
<bigjools> jtv: I am offski.  Speak tomorrow.
<jtv> nn bigjools
<rvba> jtv: What I'm seeing is that the *DNS* config still gets rewritten every minute.
<jtv> Different!
<jtv> Julian said he saw the DHCP probe followed by the DHCP config rewrite.
<jtv> He was running from package, so he wasn't looking at the "make run" output which combines all logs.
<rvba> I don't see that.  I see the DHCP probe followed by the DNS config rewrite.
<jtv> In the same log?
<rvba> I'm also running from a package.
<jtv> I'm just making guesses about what he might have seen...
<rvba> jtv: hang on, I forgot to look at one log fileâ¦
<rvba> No trace of the DHCP rewrite.
<rvba> (I'm using the daily package.)
<jtv> So... progress.
<rvba> jtv: Something else (but related): have a look at the apache log: http://paste.ubuntu.com/6884158/.  As you can see, we first query ".../interfaces?op=list" and then we get redirected to "/interfaces/?op=list".  If we could issue the right request from the start (with the '/' at the end), we would avoid one redirect every minute.  This is a detail of course but I thought it was worth mentioning.
<jtv> Yes...  you can always file a tech-debt bug.  :)
<rvba> heh :)
<rvba> jtv: src/maasserver/dns_connect.py says saving a NodegroupInterface triggers a full DNS rewriteâ¦
<jtv> Yes, DNS rewrites are more easily triggered.  I was just looking for a DHCP rewrite...
<rvba> bigjools: can you double check that you're still seeing the DHCP config being rewritten every minute?
<rvba> bigjools: I just tested the daily package and I don't see that.  I see the *DNS* config is still being rewritten every minute.
<bigjools> rvba: no I can't, sorry. But I was using the daily and it was definitely doing it on the cluster celery.
<bigjools> probe Â» config write Â» restart dhcp
<rvba> I don't see that.  And I've got a DHCP server configured and running.
<bigjools> ok I'll check again tomorrow
<bigjools> too late for me now
<rvba> Okay.
<bigjools> eating with plate on my lap and the laptop on the arm of the chair here
<jtv> A recipe for success...
<rvba> jtv: bug filed: https://bugs.launchpad.net/maas/+bug/1276985
<ubot5> Launchpad bug 1276985 in MAAS "The "Edit cluster interface" page shows an undocumented "Foreign dhcp ip" field." [Medium,Triaged]
<rvba> bigjools: ^ (with screenshot ;))
<jtv> Thank you ubot5
<bigjools> rvba: grar, docs
<bigjools> rvba: why do you keep talking about editing it?  You can't edit it.
<bigjools> rvba: oh I see the page
<bigjools> my bad.  WTF.  That didn't used to be there.
<rvba> bigjools: :)
 * jtv is glad to see that the screenshot was of some use.
<rvba> jtv: time for a tiny review? https://code.launchpad.net/~rvb/maas/fix-none-vlan/+merge/205119
<jtv> Sure
<rvba> Thanks.
<tomixxx3> hi, how can i completely remove maas? i have executed sudo "apt-get purge maas ; sudo apt-get autoremove" and rebooted the server, but after reboot, i am still able to open the MAAS dashboard...
<tomixxx3> ok, i have it already
<tomixxx3> hi, after executing "$ maas-cli maas node-groups import-boot-images" i get 'HttpResponse' object has no attribute '_is_string'
<tomixxx3> does anyone know what's the problem here?
<gmb> tomixxx3: Can you try running `maas-cli refresh` and see if that fixes it?
<tomixxx3> gmb: hmm, now i have already started sudo maas-import-pxe-files
<tomixxx3> gmb: should i abort that?
<tomixxx3> gmb: furthermore, i have executed sudo add-apt-repository cloud-archive:tools and sudo apt-get-update before installing MAAS but it seems i always get version 1.0...
<gmb> tomixxx3: No, just let it run
<gmb> tomixxx3: What does `apt-cache policy maas` return?
<tomixxx3> gmb: ah, it says 1.4+bzr16934+
<tomixxx3> gmb: but i see not the very same output and screens like described here: http://maas.ubuntu.com/docs/install.html#post-install
<gmb> tomixxx3: Okay, what *do* you see?
<tomixxx3> gmb: slightly different screens and output most of the time ;)
<tomixxx3> gmb: for example, i cannot enter the IP address at start, i need to call dpgk-reconfigure... afterwards
<tomixxx3> gmb: on start i get a (wrong) IP assigned
<tomixxx3> gmb: furthermore, "--password" is missing in the createadmin statement
<tomixxx3> last but not least, i cannot execute maas-cli maas node-gruops import-boot-images
<gmb> tomixxx3: Okay, one thing at a time.
<gmb> tomixxx3: Have you created an admin yet?
<tomixxx3> gmb, i have done this installation for mulitple times now
<tomixxx3> gmb: i can create an admin, but i have to add "--password=xxxx"
<tomixxx3> thats not the problem
<gmb> tomixxx3: So which of the things that you listed as a problem is actually the problem? :)
<tomixxx3> gmb: none of them
<tomixxx3> gmb: ah, when i set the http proxy in the maas dashboard, save, and delete the proxy again, it seems that the proxy does not appear even if it is not displayed anymore
<tomixxx3> gmb: but that's a looooooooong story
<tomixxx3> gmb: but the reason why i reinstall maas at the moment
<tomixxx3> another thing: nodes are not able to download packages
<tomixxx3> but maybo this is because of my tricky architecture
<tomixxx3> -appear, +disappear
<tomixxx3> but i know, we don't live in a perfect world ;)
<gmb> tomixxx3: I'm not sure that this is something that we can solve easily over IRC â partly because you'll have to explain the whole architecture to me. Can you file a bug for the proxy issue please (https://bugs.launchpad.net/maas/+filebug) with instructions for reproducing the problem?
<gmb> tomixxx3: Usually with nodes not being able to download packages that's a routing issue
<gmb> tomixxx3: (Although they should just use the proxy if they can)
<tomixxx3> in short the architecute is: 2 nodes + 1 maas server connected via switch. additionally, a 2nd network interface in the maas-server connects the server to an external (university) network which is more or less a black box to me ;)
<tomixxx3> but i need the connection to the university in order to get internet-access - at least - at the maas-server
<gmb> tomixxx3: Okay, that's not all that tricky; it's just that the nodes on the private / MAAS-managed network need to be able to get out to the internet *via* the MAAS server.
<tomixxx3> yeah... i have not achieved this milestone yet
<tomixxx3> however, i have tried a lot of things, i guess
<gmb> tomixxx3: Two things you can do (assuming DHCP is set up correctly in MAAS; it should be from what you've said):
<tomixxx3> yep, dhcp works
<gmb> tomixxx3: Okay, so, gimme a sec here...
<tomixxx3> k
<tomixxx3> gmb: altough, the problem is, also my university has a dhcp proxy which wants to give all my beautiful nodes some beuatiful IP :-)
<tomixxx3> therefore, i have two network interfaces
<gmb> tomixxx3: Ah, okay, so you're *not* having to route all traffic out through the MAAS server then?
<gmb> The nodes have two NICs each?
<gmb> And one of those NICs on each node is connected to the university network
<gmb> The other to MAAS
<tomixxx3> gmb: yes, they have fixed IPs from uni
<tomixxx3> gmb: but the problem is, if they get Ip from university and dhcp from university, they try to download images from university instead from maas server
<gmb> tomixxx3: So. There's no need to plug them in to the university network *if* we configure the MAAS server to act as a gateway.
<gmb> That should solve both that problem.
<tomixxx3> gmb: that's right. therefore i have 2 network interfaces at the server
<tomixxx3> gmb: the private network works together with the private maas dhcp
<tomixxx3> gmb: only problem left: how do the nodes are able to download their packages?
<gmb> tomixxx3: Well, once the MAAS server is configured properly as a getaway (i.e. IPTables is set up, IP forwarding is turned on and so on) the nodes will be able to connect to the outside world through the MAAS server â essentially as though it were just a firewall.
<gmb> Have you touched IPTables on the MAAS server already?
<gmb> s/getaway/gateway/
<tomixxx3> i have alredy tried sth with iptables but it did not work
<tomixxx3> but maybe i just misconfigured sth
<tomixxx3> i dont know well about iptables
<tomixxx3> i would appreciate if u could offer me some iptables statements just to put into "interfaces" file ;)
<tomixxx3> well, i can tell what i ve tried: "post-up iptables -t nat -A POSTROUTING -o eth1 -j Masquuerade"
<tomixxx3> i put this statement to eth0, which is connected to the nodes
<tomixxx3> and eth1 is the interface card which connects me to the university network and to the internet
<gmb> tomixxx3: Okay, just working on a script for you to try.
<tomixxx3> gmb: kk, thank you, i really appreciate this!
<gmb> tomixxx3: Try this script: http://paste.ubuntu.com/6885719/
<gmb> (on the MAAS server, run it as root)
<tomixxx3> kk, i do not have to modify the scrip in some way?
<tomixxx3> eth0 = private network, eth1 = university network
<gmb> tomixxx3: I've already put those values in.
<tomixxx3> gmb: kk, thank you very much!
<gmb> tomixxx3: No worries.
<tomixxx3> gmb: i have a question:
<gmb> tomixxx3: Sure.
<tomixxx3> gmb: the dhcp packages from my maas-server will not travel to university-network?
<tomixxx3> gmb: i dont want do destroy university network
<gmb> tomixxx3: If you've set up DHCP correctly it will *only* work on the private network.
<tomixxx3> gmb: even if lets say the range is 10.0.0.100 to 10.0.0.200 in my private network but also at the universtiy this range is used=
<tomixxx3> ?
<gmb> tomixxx3: Well, I'd suggest using a different IP range â 192.168.1.0/24 for example â just to avoid confusion.
<gmb> Otherwise you're going to have sad times if you do get it wrong.
<gmb> Because you won't be able to tell which interface was on which network.
<tomixxx3> gmb: so i have to ask a network admin, if the ip range 10.0.0.100 to 200 is free?
<tomixxx3> just to clairfy
<gmb> tomixxx3: No.
<gmb> tomixxx3: It's a private network
<gmb> You can use whatever range you like
<gmb> tomixxx3: And it won't ever leak out into the outside world.
<gmb> tomixxx3: Because the MAAS server is NAT'ing all the packets coming through it from the nodes.
<tomixxx3> and the server is only able to send dhcp pacakgeS WITHIN the private network
<gmb> Correct.
<tomixxx3> but it would be different if i would BRIDGE both interfaces, right?
<gmb> YES.
<gmb> DO NOT DO THAT.
<tomixxx3> KK :-)
<gmb> OR BAD THINGS (POTENTIALLY) HAPPEN
<gmb> NAT on university networks is _much_ safer :)
<bjf> i'm still new enough that i don't understand the stages a system goes through in maas. after i have a working MAAS server up and running and i power cycle a different node, that node netboots from MAAS and is "discovered" correct?
<bjf> after "discovery" the node is powered down. next, i tell MAAS to "Commission" the node. the node is powered back up, netbooted from maas; maas runs some things on it and then powers it down again, correct?
<bjf> i have a node whose status is "Commissioning" and is still powered up and has been this way since last week
<bigjools> bjf: your understanding is correct. Either the power detection is not working or it didn't PXE boot.
<bjf> bigjools, ok, thanks
<bjf> bigjools, it's beta HW so could be several reasons why it's "stuck"
<bigjools> bjf: you can check the power detection worked by going into the edit node page and see if the parameters are correct.  which power type are you using?
<bigjools> ah ok :)
<bigjools> bjf: the other possibility is that the commissioning scripts are hung
<bigjools> which is always possible on new hardware
<bjf> bigjools, i'm using CDU
#maas 2014-02-07
<bigjools> morning ticking
<bigjools> errr
<bigjools> morning jtv
<jtv> Hi bigjools
<jtv> autocomplete?
<bigjools> no just incompetence
<bigjools> jtv1: why do we have a nodegroupinterface created in nodegroup.new()?
<jtv> Unpleasant, isn't it?  Hysterical raisins.
<bigjools> it is responsible for the interface that has no name
<bigjools> I removed it and am running tests, there's a ton of failures
<jtv> I hate it too.  Complicates lots of tests, and the ones it simplifies, rely on the implicitness of it all.
<bigjools> blech
<jtv> It would be very costly indeed to change.
<jtv> Sheer number of tests that need updating...
<jtv> Updating tests on autopilot would risk a lot of false negatives I think.
<bigjools> yes
<bigjools> but this empty interface is no use to man nor beast
<jtv> Is it really empty though?  I thought NodeGroup.new() took parameters that it passed on to the NGI?
<bigjools> but not the name
<bigjools> so it's useless
<bigjools> BTW:
<bigjools>   File "/home/ubuntu/trunk/src/maasserver/preseed.py", line 99, in get_curtin_installer_url
<bigjools>     node.architecture + "/" + series +
<bigjools> TypeError: coercing to Unicode: need string or buffer, NoneType found
<bigjools> not sure why I am getting those all of a sudden
<jtv> I made changes there, though as far as I could tell, the None was always a risk there.
<jtv> pick_cluster_controller_address() must be returning None.  Which can happen.
<jtv> Just not in any case where the old code didn't also return None.
<bigjools> yegads is that how deep this insanity goes...
<jtv> Ahhhh, maybe the nodegroupinterface has a null IP address?
<jtv> Ohhhh, what does NodeGroupInterface.get_network() do in that case?
<jtv> Sorry, .network
<jtv> I wonder why it uses the broadcast address.  But no matter.
<jtv> What may be happening is that your node is in the network defined by a nodegroupinterface whose ip field is None.
<jtv> The ip field is not allowed to be null...  so not that.
<jtv> The only cases where pick_cluster_controller_address can return None AFAICS are: NodeGroupInterface.ip is None, or the NodeGroup has no NodeGroupInterfaces.
<jtv> Whu!?  NodeGroupInterface.network has some details that puzzle me.
<jtv> Implicit conversions to bool, construction of an IPNetwork from another IPNetwork...
<jtv> IPNetwork("%s/%s" % (IPNetwork("%s/%s" % (broadcast, netmask)), netmask) â why?
<jtv> Ah, missed one:
<jtv> IPNetwork("%s/%s" % (IPNetwork("%s/%s" % (broadcast, netmask)).network, netmask)
<jtv> Looks like the intent is to strip the host bits off the network address.
<bigjools> doesn't IPNetwork have functions to do that?
<jtv> Yes.
<bigjools> also should we remodel NGI to use a Network FK?
<jtv> I've got a branch here that simplifies that, and provides a wrapper for the error-prone IPNetwork constructor.
<jtv> You're right that a foreign key makes sense...  Unfortunately the situation with the broadcast address gets a bit convoluted.
<jtv> I don't see why we store the broadcast address at all.
<jtv> You see what I mean?  Weird to have a FK to Network, _and_ a broadcast IP.
<jtv> Technically just the IP address might be enough â call it ip plus FK.
<bigjools> jtv: we are going to need a FK at some point because we need to track which NIC the network lives on
<bigjools> oh no we don't
<bigjools> sorry mixing up my ngi and node interfaces
<bigjools> jtv: anyway broadcast as a separate field becomes irrelevant
<jtv> It'd be nice to ditch it...
<gmb> jtv, rvba, allenap: Anyone know why bin/test.maas would be hanging after:
<gmb> nosetests maasserver.tests.test_power_parameters --exclude=provisioningserver --exclude=maastesting --exclude=maascli --verbosity=3
<gmb> nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
<gmb> nose.config: INFO: Excluding tests matching ['provisioningserver', 'maastesting', 'maascli']
<gmb> ?
<jtv> gmb: database in use?
<jtv> e.g. if you have a "make harness" session open in another shell.
<gmb> jtv: AAAAH.
<gmb> jtv: Yep, thanks.
<jtv> \o/
<jtv> rvba: from this log you pasted, http://paste.ubuntu.com/6890145/ â is there a corresponding GET on interfaces/eth0/ before the PUT?
<jtv> If there is, there may be a client-side race condition.
<rvba> I don't remember one.
<rvba> (I don't have the full log)
<jtv> But if there isn't, and we just do a PUT that updates a few attributes and leaves the others unchanged, then it would have to happen in a transaction.
<jtv> I'll look at maas-test code again.
<rvba> jtv: let's talk about this in a hangout if you don't mind.
<gmb> Interesting, test tools doesn't appear to have a facility to say "always show the details of the test run, even on success." Well, isn't that wizard?
 * gmb hacks.
<roaksoax> bigjools: around?
#maas 2014-02-09
<bigjools> roaksoax: on a Saturday morning?  No, I wasn't :)
<bigjools> hey lifeless good to see you still watching the maas bug mail :)
<lifeless> bigjools: :)
#maas 2015-02-02
<maashelp> install ubuntu 14.10 configure interfaces configure dns apt update apt upgrade apt install maas-region-controller dpkg-reconfigure maas-region-controller ----> http://192.168.10.1/MAAS/ create new root account login  install ubuntu 14.10 configure interfaces configure dns apt update apt upgrade apt install maas-cluster-controller dpkg-reconfigure maas-cluster-controller ----> 192.168.10.1 maas login <maas_user> http://192.168.10.1/MAAS
<maashelp> it was suppose to be a list...
<maashelp> install ubuntu 14.10 /n configure interfaces /n configure dns /n apt update /n apt upgrade /n apt install maas-region-controller /n dpkg-reconfigure maas-region-controller ----> http://192.168.10.1/MAAS/ /n  create new root account /n login /n  install ubuntu 14.10 /n configure interfaces /n configure dns /n apt update /n apt upgrade /n apt install maas-cluster-controller /n dpkg-reconfigure maas-cluster-controller ----> 192.168.10.1 /
<maashelp> ***Region Controller*** install ubuntu 14.10 ---->> configure interfaces ---->> configure dns ---->> apt update ---->> apt upgrade ---->> apt install maas-region-controller ---->> dpkg-reconfigure maas-region-controller ----> http://192.168.10.1/MAAS/ ---->>  create new root account ---->> login ---->> ***Cluster Controller*** install ubuntu 14.10 ---->> configure interfaces ---->> configure dns ---->> apt update ---->> apt upgrade ---
<maashelp> ntroller
<maashelp> dpkg-reconfigure maas-cluster-controller ----> 192.168.10.1 ---->> maas login <maas_user> http://192.168.10.1/MAAS/ <api_key> ---->>  check MaaS web interface... no clusters appear in the clusters tab. ---->>  I've been troubleshooting this for 4 days now and its getting rough.... ---->> does anyone know what makes these two communicate properly? ---->>
<roaksoax_> rbanffy: ping
<rbanffy> roaksoax, icmp_seq=1 ttl=64 time=0.024 ms
<saltlake> I found this link :http://askubuntu.com/questions/144531/how-do-i-install-openstack :
<malikeye> is 14.04 not supported? the wiki says 12.04... but I found the PPA for the project and I'm using that... having issues
#maas 2015-02-03
<htrmeira> Hey guys. I have a doubt about the commissioning phase, does anyone knows where I can find a doc explaining what happens at this phase?
<kiko> htrmeira, the docs online at maas.ubuntu.com should be pretty clear, but feel free to ask and we'll explain as well
<htrmeira> kiko: the doubt is that when commissioning the machine boots and then shuts down, but no changes happens. Is it normal? Thank you.
<kiko> htrmeira, yes, it's normal. commissioning is the step where we query the hardware profile and record it. the deploy action is what actually causes an installation to happen
<htrmeira> kikp: Ok. Thank you very much. =)
<kiko> htrmeira, what version of maas are you using?
<htrmeira> 1.7
<htrmeira> kiko: 1.7
<kiko> htrmeira, 1.7.1?
<htrmeira> kiko: 1.7.0~beta8+bzr3272-0ubuntu1.2
<kiko> htrmeira, if you can upgrade to 1.7.1 (which is in the testing PPA now) you'll have a nicer experience
<htrmeira> Cool. Do I have to change the repository to do this upgrade?
<htrmeira> kiko: Is there an easy way to add a modified ubuntu image?
<kiko> you'll need to set up your own streams server
<kiko> let me see if there are docs
<htrmeira> kiko: thank you very much. if there isn't, maybe i can contribute with this later.
<htrmeira> i'll have to take a break here, but anything i'll be here later. thanks.
<kiko> htrmeira, http://maas.ubuntu.com/docs/sstreams-mirror.html
<kiko> htrmeira, https://bugs.launchpad.net/maas/+bug/1414658
<ubot5> Launchpad bug 1414658 in MAAS "Need a way to provide a simplestream for "custom images"" [Wishlist,Triaged]
<htrmeira> Thank you, kiko. =D
<kiko> allenap, j^2: I'm going to let you continue without me for the hour and then I'm back
<j^2> allenap: hi!
<allenap> j^2: Hello!
<kiko> allenap, ping?
<malikeye> just re-built on 12.04 and I'm getting this error: (2)No such file or directory: mod_wsgi (pid=2229): Unable to change working directory to '/home/maas'
<malikeye> followed the docs exactly
<kiko> malikeye, re-built 1.7.1?
<kiko> or is this maas 1.2?
<kiko> if it's 1.7.1 you may be running into a too-old apache2 version problem
<kiko> or missing wsgi package
<malikeye> it's 1.5.2
<kiko> hmm
<kiko> I don't know if anybody runs something like that
<kiko> why 1.5.2 specifically?
<malikeye> weird... yeah, why am I running that version, lol
<malikeye> should be on 1.7 I would guess
<kiko> malikeye, 1.7.1 from the testing ppa?
<kiko> and you are the first person I know of testing that on precise
<kiko> so you will definitely be breaking the ice
<malikeye> I'm on 12.04 using add-apt-repository cloud-archive:tools
<kiko> but that's often a nice feeling. :)
<malikeye> I used that PPA last night on 14.04 and I couldn't get the APM stuff working, so I couldn't deploy anything
<malikeye> cloud-archive:tools didn't work with 14.04 so I had to use testing
<malikeye> what's the best config now? there is so many different versions of things.... 14.04 with testing?
<malikeye> looks like devlopment trunk says latest... I'll try that again
<kiko> malikeye, 14.04 + testing PPA is the bomb
<kiko> it's what everybody's on :)
<malikeye> cool
<malikeye> I couldn't get the power management stuff working... this ultimately will be for UCS, but I couldn't even get WOL working
<kiko> 1.7.1 has WAY better hardware support
<malikeye> would get an error... I forget what it was
<kiko> it will be SRU'd to trusty this month
<malikeye> have you worked with UCS?
<malikeye> never used the API stuff with it, and the docs are hard to find
<malikeye> figure if I can at least get WOL working, then I can tackle that after
<kiko> malikeye, it should work out of the box, and newell is here to help if you have problems that we haven't addressed in 1.7.1
<kiko> newell, (am I correct about it working OOTB?)
<newell> UCSM is ootb yes
<malikeye> cool
<kiko> malikeye, what timezone are you in?
<malikeye> EST
<kiko> malikeye, okay, newell is PST so you have reasonable overlap
<newell> malikeye, yeah let me know if you have any questions
<malikeye> newell: nice, thanks
<kiko> newell, just saw your MP
<kiko> newell, do we explicitly run the templates with "sh" or "bash"?
<kiko> just asking because it'd be much nicer if we could use #!/whatever in the template file
<newell> kiko, I just copied what ther templates are using...let me look
<kiko> and make them execable
<kiko> I think you can't right now though
<kiko> I need to split
<kiko> ttyt!
<newell> ack
<malikeye> awesome, just triggered a blade startup from maas... very cool
<malikeye> not sure what I did last night, but this is much better
<newell> malikeye, glad to hear it :)
<malikeye> so, I'm not seeing where to actually deploy here... the host is green... is this strictly for landscape / juju? or should I be able to deploy with it?
<malikeye> the docs don't really say how to.... guess I should read more ;)
<malikeye> pretty cool stuff... looks like I'm hosed up at the openstack-installer though.... it bootstrapped the blade, but juju is just chilling waiting for something to happen
<malikeye> looks like it might be trying to deploy an LXC container, but I don't see one configured in maas
<hazmat> https://bugs.launchpad.net/maas/+bug/1417793
<ubot5> Launchpad bug 1417793 in MAAS "MAAS selection of node for juju constraints does not choose "least-capable" match" [Undecided,New]
<hazmat> ctlaugh on #juju was having some probs
<hazmat> with that
 * hazmat looks for any maas developers hanging out on #juju
#maas 2015-02-04
<SimplySeth> how does one query the api using curl ?
<SimplySeth> I tried curl -u "username:password"  "http://machine/MAAS/api/1.0/nodes/?op=list and got"  "Unrecognised signature"
<malikeye> anybody deploy openstack with landscape / maas? I can get the OS deployed on the first node, but juju just sits there waiting for something
<malikeye> I have two network on the juju / landscape VM, and all of the hosts have 2 NIC's in them
<malikeye> do I need to define the networks in maas? the instructions aren't really clear
<malikeye> ahh, it wants to lookup the host with a hostname... gotta flip the DNS on I think
<malikeye> now it's stuck on deploying landscape... one wall at a time :)
<malikeye> so I have two nics on the landscape/juju VM, and I have both networks defined in maas... do I need to put that second nic on the network? do I need to put the maas on that second network too? or does it just manage the nics on the target nodes with that second defined network?
<malikeye> seems to be doing better with the second nic on the nodes attached at the maas level
<malikeye> meh.... gotta have 5 machines.... only got 4 :(
<johng> hi.  looking for help with MAAS 1.7.  Seems to be some internal communications issue after initial image download
<johng> socket.error: [Errno 104] Connection reset by peer
<kiko> johng, can you give me more detail?
<johng> in a proxy environment, all in one on a KVM vm.  All setup works, set proxy via UI, can see image downloads happen.  after image downloads, i get tjhis error in the logs and cluster is out of sync
<johng> see images are there on the systems..  download works fine
<johng> 2015-02-04 12:07:25-0500 [-] Logged OOPS id OOPS-d8256ede274f6493daf4df634b69b74c: error: [Errno 104] Connection reset by peer
<kiko> johng, can you show me a screenshot of your /images page?
<kiko> what do you mean when you say "download works fine"?
<johng> I can see heavy activity tomy proxy with iftop
<johng> and images screen gets to 2/2
<kiko> I see
<kiko> johng, can you show me a screenshot of your /images page?
<johng> not sure how to get it to you via this freenode web..
<johng> it shows the 1404 amd64 image is there
<johng> no custom images
<johng> 1404 and amd64 are all that is selected
<SimplySeth> what is the proper way to query "http://myIP/MAAS/api/1.0/nodes/?op=list" ? a regular curl -X GET doesn't work.
<kiko> SimplySeth, when you say "doesn't work" what do you mean?
<johng> there is a python stack with the error as well...
<SimplySeth> Kiko ... the response is :Unrecognised signature: GET
<SimplySeth> kiko:  works in chrome just fine though
<kiko> SimplySeth, are you on 1.7 or 1.5?
<saltlake>  experts, am using ubunt openstack-installer to install openstack, I keep getting this complaint that there are not 5 nodes that have more than one disk. I have given each node 2 disks each 20GB partions.  Please advise
<kiko> I think the issue is that nodegroups is the new black
<kiko> no, that's not it, you're right
<kiko> SimplySeth, I will tell you in a sec
<SimplySeth> kiko: 1.7 ... I think is what comes with 14.04LTS
<kiko> SimplySeth, no, 14.04 currently has 1.5.4
<SimplySeth> kiko: then it's 1.5
<kiko> ok
<saltlake> really been stuck at this for more than a day and I don't know how to even login to the nodes since I don't have a username and password on the node that is commissioned by maas.
<saltlake> Once maas commissions a node how can anyone login to it to for debug purposes.. (I need to see if the VM has 2 disks or not) pleae help
<roadmr> saltlake: did you set up an ssh key in maas? if so, you can simply ssh to nodes, that key would give you access
<kiko> SimplySeth, I think the issue may be the lack of a cookie.  /nodes/ does not allow anonymous requests
<saltlake> raodmr: I will try that right away. But if you can it is possible to check this VM xml to see if I have the 2 disks setup right ? http://pastebin.com/RbeMVGTH
<SimplySeth> kiko: okay
<saltlake> grep sdb5. I don't why openstack would not see this this
<kiko> SimplySeth, I can reproduce by using a private browser window and going to the URL
<kiko> Unrecognised signature: GET list
<saltlake> and why openstack-install repeatedly complains of the node not having more than one disk
<kiko> SimplySeth, I think an ansible controller for maas would be killer!
<roadmr> saltlake: hm, per this, this node has hda, vda and vdb disks, right? I don't know if this will confuse maas but it certainly confuses me :)
<SimplySeth> kiko: that's why I'm trying to learn the secret sauce to querying the API
<saltlake> roadmr: I changed vda and vbd to hdb and hdc resp and tried as well (since /dev/sdb5 is a physical parition .. not formatted)
<saltlake> roadmr: I changed bustype from virtio to idea as well for those 2 disks vba and vdc but that failed commmisioning. Do you have a sample xmls for a node you created for openstack ?
<saltlake> xmls-> xml
<roadmr> saltlake: sorry, I've only ever used bare metal with maas :/ no idea about virsh nodes
<saltlake> Oh no!!
<saltlake> ok raodmr: I will try ssh ing to see if that tells me anything
<roadmr> saltlake: did you find out where to add the key?
<saltlake> I had added the key in the maas preference page
<saltlake> but I rebooted the server and it won't come back up..
<roadmr> saltlake: go to your account details, you'll find an "SSH Keys" section there with a button, then put your public key there
<saltlake> then ssh to the IP address for that node
<roadmr> saltlake: yes, once the node is up you can ssh directly to its ip
<roadmr> saltlake: (or even to its blahblah.maas hostname if you're in the maas node itself)
<kiko> SimplySeth, let me research and I'll post an answer to your askubuntu question
<kiko> just a sec
<SimplySeth> kiko: I would greatly appreciate it.
<kiko> SimplySeth, meanwhile, see SimplySeth, not answering your ansew
<kiko> err
<kiko> SimplySeth, meanwhile, see http://blog.allenap.me/2013/06/workaround-for-uploading-files-to-maas.html
<SimplySeth> kiko: cool thanks
<kiko> SimplySeth, note the use of the python client api
<SimplySeth> kiko: I might have go directly to python instead of playing with BASH
<kiko> SimplySeth, the issue is that the API uses OAuth
<kiko> SimplySeth, I will still get you a more complete answer
<SimplySeth> kiko: thanks
<johng> my python error with the connection resets:
<johng> 2015-02-04 12:38:31-0500 [-] Unhandled Error
<johng> File "/usr/lib/python2.7/dist-packages/provisioningserver/boot/uefi.py", line 134, in download_grubnet
<kiko> johng, can you put the whole traceback in a pastebin?
<johng> sure - how do I do that?
<johng> googled it
<johng> see if this works
<johng> http://pastebin.com/Snh4nBAi
<kiko> johng, what version of maas?
<johng> ii  maas           1.7.1+bzr334 all          MAAS server all-in-one metapackag
<johng> same for dhcp and dns there as well
<kiko> interesting
<kiko> johng, it seems to me that the problem is that the UEFI download is not being done through the proxy
<kiko> johng, can you check to see if my hunch is correct?
<kiko> yes, it seems so
<kiko> shucks
<johng> so my LTS image downloads, but boot does not?  there is only one place to set proxy
<kiko> yes
<kiko> it seems to be a bug in maas
<johng> is it an easy edit somewhere that I can fix it?  or shoudl O gi use 1.5?
<kiko> I am not sure 1.5 would work tbh
<johng> very much apprecite you identifying it.  much much appreciate if you can provide workaround.  :-0
<kiko> johng, can you confirm first that the proxy is not getting the request for the grubnetx64.efi.signed file?
<johng> I don't have access to the proxy.  What can I get fro yuo from the box?
<kiko> johng, what port is the proxy listening on?
<johng> 8080
<magicninja> guys i need some help i got maas going fine and it runs through an install from pxe boot but once done i just come to a black screen but power is still on and i can even send control alt delete and reboot the box why cant i see anything?
<kiko> magicninja, this is at the end of installation, when the node would normally power off? is this maas 1.5 or 1.7?
<kiko> johng, I guess a packet capture of anything going through your default route and going to port 80 would be useful
<magicninja> maas 1.7 i just installed the whole thing yesterday and it does shut down after the install but when it boots back up to my hdd it goes to a black screen
<kiko> johng, because that would indicate we are trying to fetch resources directly
<magicninja> thx kiko
<kiko> magicninja, at which point does it go to a black screen?
<johng> ok.  I'm set up for that. be back in a few
<johng> thanks
<kiko> thanks
<magicninja> after install is done it powers off then back up it boots my newly installed hdd but then just goes to a black screen
<kiko> magicninja, does it for instance go through grub? do you see kernel messages?
<magicninja> but the computer is still on
<magicninja> i dont see grub like normal
<magicninja> it doesnt show up it just goes straight to black screen
<kiko> magicninja, that's weird. perhaps the grub install failed?
<magicninja> but i can still press control alt delete and it will reboot
<kiko> right
<malikeye> landscape isn't liking the hosts in maas.... all nodes had 2 x nics and 2 scsi disks, but landscape still complains that they don't have 2 disks
<kiko> malikeye, Beret might be able to help you, but I think he'd EOD now
<magicninja> i tried reinstalling it from a rescue usb but no dice there either i can get to grub then but after i say boot ubuntu in grub... black screen
<malikeye> lsblk certainly shows 2 disks
<magicninja> reinstalling grub that is
<magicninja> i have another problem too if i try to do this install from a virtual box vm it boots into the image pxeboot goes fine but about halfway through starts saying ifconfig failed giving up after 60 seconds
<magicninja> any ideas there im all ears
<magicninja> physical machines have first problem vms just wont go past that spot they drop me to initramfs
<magicninja> just ran through the automated install process again finishing up now
<magicninja> doubt there will be any change though
<kiko> magicninja, the ipconfig issue is a typical networking problem related to your bridge
<kiko> it's very common
<magicninja> but it does let me pxeboot
<magicninja> it starts the image and everything just halfway through says that
<magicninja> do i have to do this on like a nat virtual network to make those work
<magicninja> i dont know why it wouldnt work though
<magicninja> i mean how i have it now with paravirtualized devices is how im running my server
<magicninja> the maas server is a vm and i just plug a box into our network and it pxeboots right away
<magicninja> with hardware it even goes through the whole process
<kiko> magicninja, right, it's because dhcp is not very robust to packet droppage
<magicninja> with my hardware issue could it be because the drive in the box im installing is 3TB
<kiko> so if the bridge for some reason drops or delays a packet
<kiko> then the ipconfig will fail
<magicninja> ah
<magicninja> so do it on a nat'ed virtual network and that should make the vms work?
<kiko> magicninja, another situation where this happens is when the node has multiple interfaces
<magicninja> oh and what is the best idrac etc to use for maas
<magicninja> yeah node doesnt have multiple interfaces
<kiko> magicninja, any IPMI-compliant BMC should be fine
<magicninja> its a vm lol
<kiko> a VM with multiple NICs
<magicninja> nope just one nic
<magicninja> i made sure after scouring the internet
<kiko> okay
<magicninja> ill let you know what my hardware box does in just a minute
<magicninja> im not totally sure if this would cause a problem but...
<magicninja> the hardware machine i am testing with has intel chipset
<magicninja> but on the frontend when i click on the node i had it marked as amd64 after noticing this after my last attempt i changed it started it again and am almost done with the fresh install
<magicninja> sure is nice to have a chat room to talk this over with someone who knows about t
<magicninja> this stuff
<magicninja> ill try and stick around and give what support i can
<magicninja> i gotta admit with the architecture marked differently the install process sure did change
<kiko> SimplySeth, http://askubuntu.com/questions/581444/how-to-query-maas-api-with-curl/581688#581688
<magicninja> after a node installs it *should* come to a login prompt right?
<kiko> magicninja, yes, but you should have uploaded an SSH key at this point which will allow you to ssh in, as no user is enabled to log in with a password by default
<SimplySeth> kiko: thank you
<magicninja> yeah i have one out there
<kiko> SimplySeth, let me know if you get stuck; I tried hard to find a good way to do it with curl but it seemed like a bit too much work
<magicninja> i just dont see anything on the screen so i cant tell if i have a video problem or something is wrong wrong
<kiko> magicninja, my hypothesis is that the bootloader isn't getting installed
<kiko> magicninja, that would lead to the machine hanging at a blank screen
<magicninja> when it ran through the process it said it was installing it
<magicninja> ill pop in a live cd and reinstall it to the sda drive and reboot and let you know what happens
<magicninja> and even after this latest install i still get the same thing
<kiko> magicninja, yeah, boot into a usb stick and chroot into the installed partition
<kiko> magicninja, a grub-install should work
<SimplySeth> Kiko .. I'm perfectly comfy with Python .. thx
<kiko> magicninja, having said that, AIUI for a 3TB frive it needs to be a GPT partition and UEFI boot
<kiko> SimplySeth, okay -- if there are issues with the API let me know, because I don't know how many users we have that don't use the `maas` commandline client
<magicninja> if its not gpt would that cause a problem
<kiko> magicninja, I think 2TB+ drives require GPT to be bootable
<saltlake> roadmr: I tried ssh to the client machine but it isn't possible to do so !!
<saltlake> The ssh key is for the maas node not the the other client nodes that are commissioned
<roadmr> saltlake: oh! maybe the key didn't get copied if deployment failed :(
<saltlake> the commissioning worked fine
<roadmr> saltlake: hm, all deployed nodes should have that key added to them
<magicninja> apparently its not up in any way cause i cant ping or anything after it goes to black screen but keyboard still functions
<roadmr> saltlake: yes, the key won't be added until they're actually started or provisioned, I think
<kiko> saltlake, did you ssh as the ubuntu user?
<kiko> roadmr is correct, the node needs to be deployed
<saltlake> I can't "deployment" to happen since the installer can't see 2 disks on the VM client
<kiko> if you have only commissioned then you won't be able to connect
<roadmr> saltlake: you could "start" one node to then SSH into it and verify the disk configuration, maybe
<kiko> +1
<johng> tcpdump done.  lots of files donwloaded, until we get to that grubx64.efi.signed file...  proxy send about 4k then sends a reset wiht TSval and TSecr in the packets...
<saltlake> roadmr: How do  Issh to it ? I don't have the username and password since it is pxe booted and commissioned from MAAS
<roadmr> saltlake: in the maas web UI, go to the node's page, there's a "start" button there
<saltlake> Ahh like that I was doing a virsh start node
<roadmr> saltlake: use that to start the node, it should boot up and install the operating system, it will be left in a "ready" state I think
<roadmr> saltlake: yes, once maas tells you the node is ready (or deployed, perhaps), then, from the maas server's terminal, you should be able to ssh ubuntu@NODE_IP
<roadmr> saltlake: it shouldn't ask for a password then, assuming you have the keys set up correctly
<saltlake> ok me trying.. it is just starting up.
<saltlake> ubuntu
<saltlake> sorry ignore.. wrong window
<roadmr> :D
<kiko> johng, interesting -- seems to me like your proxy may be causing the issue (it might for instance not like the filetype or extension)
<kiko> or content
<saltlake> radmr: after the node is started should it reboot ? (I am seeing the virtual window) Also should I levae network as the first boot option after the VM is commissioned ?
<kiko> saltlake, the node will not reboot once started; it will be left running
<kiko> saltlake, the boot option is irrelevant as we instruct the node to chain boot to the disk
<magicninja> kiko are you some kinda dev or something?
<saltlake> roadmr: kiko: thank you.
<saltlake> roadmr: it booted up and I see this :  cat /proc/partitions
<saltlake> major minor  #blocks  name
<saltlake>  253        0   20971520 vda
<saltlake>  253       16   20971520 vdb
<saltlake>    8        0   31457280 sda
<saltlake>    8        1   31456256 sda1
<roadmr> saltlake: I'd say it has 3 disks,right?
<kiko> magicninja, product owner
<saltlake> roadmr: Yes it does.. so my xml is correct
<saltlake> roadmr: However the ubuntu installer compains "Atleast 5 nodes need more than 1 disk) and I have 8 nodes setup exaclty like this!!
<saltlake> ubuntu installer - > openstack-install
<roadmr> saltlake: which installer are you using? I wonder if the vda nomenclature confuses it somehow
<roadmr> saltlake: maybe it expects the disks to be sd* or hd* but I don't know the installer, where can I find it?
<saltlake> roadmr: I actually put hdb, hdc in the xml but the VM shows it thus!!
<saltlake> roadmr: http://pastebin.com/TM5EAzZR
<roadmr> saltlake: in the xml you showed me, the target names are vda and vdb; if you want them to show as hda/hdb in the VM, you have to change the <target dev='vda' to say dev='hda' for instance>
<roadmr> saltlake: ohh I see you changed it then
<roadmr> saltlake: after you changed the xml, did you undefine the VMs? otherwise they'll continue to use the old configuration.
<saltlake> roadmr: No I only edited and restarted
<roadmr> saltlake: I think you have to virsh undefine the nodes for the new config to take effect
<roadmr> saltlake: hehe I also haven't worked a lot with virsh (I suck, really), but my sequence is: virsh define blah.xml; virsh start vm-name; work work; virsh destroy vm-name; virsh undefine vm-name
<roadmr> saltlake: if I just start/destroy/start the vm, it doesn't reread the xml, uses the one I had when I defined it
<saltlake> roadmr: I will try that and give u the status.. .. the installer is fromthis : sudo apt-get install openstack; sudo openstack-install.. It goes through 20 min and then fails on the drive issue
<saltlake> ok I will be back after undefining everything in a little bit
<roadmr> saltlake: yo could undefine just one, then redefine it, and restart it with maas, just to ensure it changes as you expect
<roadmr> saltlake: once you're sure, do the others
<saltlake> roadmr: I just the dumped the xml for that client an I see the tag "backingstore" for those disks.. is that normal ?
<roadmr> saltlake: it should be OK, just specifies which storage mechanism to use
<saltlake> ok will try
<kiko> johng, can you check with them?
<johng> I tried a diff proxy, got same error
<magicninja> where is the log written about an install of node
<magicninja> var log something?
<johng> I'm diggin into packet capture now to see exactly why I got a reset
<kiko> magicninja, on the maas side or on the node side?
<saltlake> roadmr: I just undefined the VM, redined it. restarted the node from the MAAS server. sshd to it and I still see the same thing for /proc/partitions.
<roadmr> saltlake: that's weird..
<saltlake> roadmr: i did not reprovision it from MAAS.. maybe that is requierd.
<roadmr> saltlake: did you destroy the vm? it should not show if you do 'virsh list'
<saltlake> gosh i did not .. sorry ..
<roadmr> :D
<magicninja> ok so i ran grub-install /dev/sda and now rebooting
<kiko> magicninja, is it a GPT partition table btw?
<magicninja> oh you know what i didnt think to check while i was in there
<kiko> heh
<magicninja> fdisk -l should show me right?
<magicninja> i mean i figure that it is otherwise i dont know how it could have installed there in the first place
<kiko> yep
<magicninja> ok going back in to check before i boot it
<kiko> it's mostly curiosity
<magicninja> im curious now too
<magicninja> and only takes a sec to go back and check
<johng> kiko - I see some double slashes in the url being requested
<kiko> johng, that shouldn't matter -- you should be able to wget it from another machine (not behind a proxy) to test.
<johng> trying now
<kiko> johng, double-slashes are generally folded in URLs
<magicninja> yeah my /dev/sda is gpt
<saltlake> roadmr: Its the same after the destroy/recommission  - restart http://pastebin.com/Mfmm08cS
<roadmr> saltlake: damn :) can you pastebin the complete .xml file?
<saltlake> Yes It on that linkk with the /proc/paritions below the xml
<saltlake> roadmr: Oh it didn't put the whole file 1 sec
<roadmr> saltlake:  :)
<kiko> magicninja, okay, so maas got that right at least
<saltlake> roadmr: http://pastebin.com/xgN4rUqf
<kiko> magicninja, does it contain a UEFI partition?
<johng> kiko.  get network error on my local pc, as well as wget from the maas box
<magicninja> no just EFI i believe
<johng> http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/uefi/grub2-amd64/current/grubnetx64.efi.signed is the url
<magicninja> thats prob why but how do i manipulate that
<kiko> magicninja, it should work
<kiko> johng, 2015-02-04 17:17:08 (376 KB/s) - âgrubnetx64.efi.signedâ saved [1001336/1001336]
<kiko> works from here behind my proxy
<kiko> kiko@gasolinux:~$ wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/uefi/grub2-amd64/current/grubnetx64.efi.signed
<kiko> --2015-02-04 17:17:05--  http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/uefi/grub2-amd64/current/grubnetx64.efi.signed
<kiko> Connecting to 192.168.99.4:3128... connected.
<kiko> Proxy request sent, awaiting response... 200 OK
<kiko> [...]
<kiko> johng, looks like a local issue. can you check with the proxy gods if they can unblock the issue?
<roadmr> saltlake: something continues to be very weird, you're declaring hda, hdb, hdc in your .xml, but the partitions file shows vda, vdb, sda, which is completely different.
<kiko> johng, I' m surprised you are getting a network error on your local box, though, unless you are behind the same proxy.
<kiko> if you are, then you've nailed the problem :)
<saltlake> roadmr: I don't get it either.. I think I should deleter the physical location of the client1 img with ha
<saltlake> sda!!
<saltlake> roadmr: Does having the virtio bustype matter vs ide /
<saltlake> roadmr: I will actually define a completely new name and stop using client1 and use it to see if there is any diff
<roadmr> saltlake: sounds good. From what I just read, you should always try to use virtio
<kiko> totally
<saltlake> if you don't mind sharing the source of the link.. it is not so apparent from virt-install --help
<johng> thanks for helping kiko.  It was failing with nework error in chrome.  after you tried i tried IE and it works.  now it downloads in chrome as well..  go figure.  no MAAS for me.  guess I'll go staight juju rather than fight the peoxy-people..
<johng> wget still fails...
<johng> Proxy request sent, awaiting response... 206 Partial Content
<kiko> johng, how will you go straight juju without installing the nodes?
<kiko> that's not a failure
<kiko> partial content means that you already have part of the file locally
<johng> Read error at byte 842669/1001336 (Connection reset by peer). Retrying.
<johng> get lots of those..
<johng> that's wget.
<kiko> johng, and the proxy is mandatory, IOW, you can't issue requests directly?
<magicninja> i just dont get it i dont know what more to check even after installing grub still no dice
<johng> its in a lab.
<kiko> johng, is it always at that location?
<kiko> magicninja, I am a bit unclear as to what the boot process looks like on your node -- is it attempting to PXE-boot first, and then going blank?
<johng> need openstack on trusty.  hoping I can just juju deploy stuff without MAAS.  I have cobbler there..
<johng> new to juju and MAAS if you could not tell)
<magicninja> i can make it do that but as i am through the install process even if i pxe boot it just boots my hdd
<kiko> magicninja, that's correct
<magicninja> it connects then mass tells it to boot the hard drive
<magicninja> but then just blank screen
<kiko> johng, you can't juju deploy without a provider
<magicninja> but keyboard still works and box still on
<magicninja> just no screen
<magicninja> does not respond to ping either
<kiko> magicninja, hmm, to rule out anything else, does a USB install on that system works?
<kiko> work
<johng> kiko I saw there was a manual provider but have not researched...  will that not allow juju without MAAS for openstack deploy?
<magicninja> thats what im trying now but im pretty sure it will
<johng> am I going to have to build openstack from scratch?  (uck)
<magicninja> inside the usb install when i hit partitioning it looks like maas set up three partitions one 3TB ext4 and one swap for 2.1GB and i think i did one called biosgrub 1MB
<kiko> magicninja, that seems correct, it's a UEFI install IOW
<kiko> magicninja, is your BIOS set up to boot in EFI mode btw?
<magicninja> hmmm not sure
<kiko> johng, I think you are likely to suffer more than if you just get this file to come through :-)
<magicninja> before i reinstal let me check
<kiko> johng, you can set up a local mirror and tell maas to use that
<kiko> magicninja, that can make a difference
<johng> kiko - am I able to put the file there manually and avoid that last part of the download?
<magicninja> in bios i have a pci rom priority it was set to efi compatible but the only other option is legacy rom
<kiko> johng, I'm not entirely sure of the answer to that :)
<kiko> magicninja, pci rom? not boot?
<kiko> magicninja, https://wiki.archlinux.org/index.php/beginners%27_guide#Troubleshooting_boot_problems
<johng> debmirror is on my list to learn..  is that what your referring to?
<kiko> johng, well, I was actually referring to a simplestreams mirror, but I need help from a dev to give you a more complete answer
<kiko> johng, well.. you could definitely hack the source and avoid the need to download that file :)
<kiko> johng, look at uefi.py in provisioningserver/boot
<kiko> just comment out the block from line 206 to 213
<kiko> and put the file in the right place
<kiko> i.e.
<kiko> var/lib/maas/boot-reources/snapshot*/grubx64.efi
<kiko> that should be enough -- it's the last file of the boot resources you need to download
<saltlake> roadmr: same stuff with a new client that never existed so far.. I am confused.. Do you happen to have a virt-install command line you could share on how you might set up a disk .. is it because the source directory /dev/sdc5/ /dev/sdb5 are logical volumes on a physical disk ?
<johng> wow, I guess I have tried a lot.  I have about 40 snapshots in there
<kiko> johng, we download those once a day IIRC
<johng> All mine are dated today.  :-)
<malikeye> you actually need 5 nodes for OS itself... the landscape / juju stuff sits on it's own node... so I'm using a huge blade to host landscape / juju
<kiko> johng, gar!
<saltlake> malikeye: Can you share how you created the VM nodes ?
<magicninja> well after reinstalling grub i can get into grub now but when i boot ubuntu it just goes to same ol black screen
<kiko> johng, I need to split, but if you still have issues, could you email maas-devel and we'll look at it overnight?
<kiko> thanks
<magicninja> i just know this is something stupid
<malikeye> I didn't use VM's... I have UCS blades... I would've liked for landscape / juju to live in a VM though
<malikeye> you can't choose where it gets deployed I don't think
<kiko> magicninja, did you see the URL I posted to you?
<magicninja> yeah havent gone through it yet though
<johng> kiko cluster howing sync after a reboot..
<kiko> magicninja, https://wiki.archlinux.org/index.php/Kernel_mode_setting#Disabling_modesetting
<kiko> johng, synced! cool :)
<kiko> I need to split, catch you all tomorrow
 * kiko waves
<johng> do you know if the process will go back directories and find the grub file, or do I need to keep it in the latest one somhow?
<saltlake> malikeye: I have a large server and am working with VM ndose for everything. I don't care for performance.
<johng> later kiko!  thanks a bunch
<kiko> johng, the latter
<malikeye> I want the controllers to be VM and the nodes to be raw KVM
<saltlake> malikeye: I really hope to have a basic openstack install done else I get fired.. but if I figure it out I will share it on a blog for sure
<johng> k
<malikeye> these are 32-cre 256GB blades... so a whole blade for landscape / juju is a waste
<malikeye> saltlake: just make sure you have 5 nova nodes, 2 disks on each node, and 2 nics on each node
<malikeye> there is no way around that, unless you do the all-in-one
<malikeye> hmm, this is weird... maas only sees half the ram on these blades
<saltlake2> Now I am quite certain that this is an issue with openstack-install and that it is unable to see 2 disks on the VM nodes and its complaint about "Atleast 5 nodes should have more than 1 disk) is unfounded. I have to be NOT THE first person who is facing this.
<malikeye> you need 5 nodes dedicated to nova with 2 disks
<malikeye> then you need a node for landscape / juju... you can run maas on that node I believe
<malikeye> if not on that same node, another for maas
<malikeye> == 7
<saltlake2> I have 8 VM nodes with 2 disks each. But the openstack-install has that gripe
<malikeye> are you using virtio?
<saltlake2> 1 of those -> landscape. But there are 7 more
<roadmr> saltlake2: you never told me where to get openstack-install :) I can try to see how it checks for number of disks, see if your configuration is confusing it in any way
<malikeye> I think it needs to be scsi disks.... it doesn't like virtio
<roadmr> saltlake2: what confuses *me* is why you get those very weird disk names. I've never used virt-install, I have a hand-hacked .xml file :/
<saltlake2> malikeye: I have not eplored the scsi disk thing at ALL!!
<saltlake2> that could make sense
<saltlake2> roadmr: sorry. this is how I got openstack-install  http://www.ubuntu.com/download/cloud/install-ubuntu-openstack
<saltlake2> point 4
<saltlake2> malikeye: I will try to ee if making those disks scsi helps.. I hope those sources can be logical partitions.
<malikeye> it will bootstrap them and overwrite everything
<malikeye> I read a bug somewhere that virtio was an issue, so hopefully that will fix it for you
<roadmr> saltlake2: oh boy... it'll take me a while to look at openstack-install, but I'll get around to it
<saltlake2> roadmr: You've been most helpful.. thank you So much!!
<malikeye> I can't see how any "regular" person would deploy this stuff.... so many logs to look through for strict requirements for the environment
<saltlake2> malikeye: I will explore the scsi ooption.. I hope I don't need real physical disks .. but can get away with using something virtul!!
<malikeye> it should work on virtual... but the storage device that's getting reported is virtio, and it probably either doesn't look for that, or blocks that as a viable storage taret
<malikeye> target
<malikeye> the landscape openstack install hangs at 1% with no errors in any of the logs I can see
<malikeye> hmmm, second time around it's progrssing
<malikeye> get this error while installing the neutron gateway : dpkg status database is locked by another process
<malikeye> not to bitch, but so much for "your own private cloud in minutes", lol
<magicninja> man if this would just work it would be great but at this rate i might be spinning up a fai server
<magicninja> i keep getting dropped to initramfs after one of my vms boots to maas
<magicninja> says alert! /dev/disk/blahblahblah does not exist dropping to a shell
<magicninja> only on my virtualbox vms though
<magicninja> still cant get my hardware box to boot up and not black screen
<malikeye> landscape apparently doesn't like my network setup... I have the nodes, maas, and landscape on one VLAN on nic across the environment, and then a /24 on the second nic across the environment
<malikeye> this really should work
<magicninja> anyone else have any ideas about getting this black screen to go away make it all the way through and just cant get passed this spot
<saltlake2> malikeye: Please explain somewhere on how u setup another virtual netwrok to make landsacpe happy.. I will face the same issue
<malikeye> saltlake2: I haven't gotten it to work yet, but I have 2 VLAN's... one for infra and one for instances
<malikeye> everything is on both network... VLAN1 is for landscape / maas
<malikeye> which works fine... when I get to the landscape part, I select VLAN2 as the "public" network, as I really have to, it forces me to, because at least one node needs to be on that network, and the NICs are all tagged to that network anyway
<saltlake2> malikeye: I have one private network for all the client nodes is and is a 192.168.122.XX and the MAAS controller is on 192.168.122.100 so all are on the private network. I don't know how to do the second network or how it should be setup ? I could connect it to the bridge on the host
<saltlake2> Would that be acceptable
<saltlake2> ?
<malikeye> not sure... I'm rebuilding at this point... putting maas and the openstack-installer on one node... hoping I can target a vsphere guest as the landscape / juju host
<malikeye> trying to follow the directions as close as possible... but having to use the VLAN's, which is what is screwing things up I believe
<malikeye> landscape -> maas -> UCS is working fine
<saltlake2> which instructions are you following to setup this up ?
<malikeye> saltlake2: http://www.ubuntu.com/download/cloud/install-ubuntu-openstack
<saltlake2> malikeye: thanks.. I really hope I figure it out !!
<saltlake2> malikeye: I am following the same link :-D
<saltlake2> malikeye: But the darn link does not talk about the requirements of the second NIC interface.. i think if one is the provate I will put tha on the local network of the office
<malikeye> exactly
<saltlake2> cool.. BTW I am new to VMs .. KVMss .. openstack but bloody determined to get this done.. the only thing I am familiar with is ceph storage!!
<saltlake2> I found a way to get iscsi on the virsh xml
<malikeye> meh, WOL only works on OS's vmware tools on them, and there is no vmware integration for power management
<malikeye> wow, vmware announced there own openstack distribution today
<malikeye> another thing that happens, is that there is no intuitive way to "clean up".... if you need to tear down and start over again, the leases get stuck..... and the leases get sent all around, so just nuking them from /var doesn't work "clean" I don't think
<malikeye> fresh maas install, and it doesn't probe them memory correctly
<malikeye> looks like it fails at the landscape-client install
<malikeye> I can't even tell which machine is doing what at this point... maas/landscape/juju are on one VM, which deployed another box, which deploys a bunch of containers, that try to deploy another box... it's very confusing, lol
<bdx> try watching the logs.......
<malikeye> which logs? there are tons of them
<bdx> in .cloud-install/commands.log
<malikeye> none that I can find that show any errors
<malikeye> I get past that point... this is landscape running the show now
<malikeye> I triaged all the way to that with logs ;)
<malikeye> but once it started getting distributed, I'm lost now
<malikeye> sorry, my topology isn't what I just said it was... I have maas and cloud-install on one VM, which works fine... then landscape/juju get deployed to a blade, which is where all the LXC's get started, which I can get to fine... it's when I go to deploy the actual OS cluster that it fails
<bdx> I go to deploy the actual OS   â gnuoy
<bdx>                   â cluster
<bdx> what is the actual os deployment?
<bdx> os cluster?
<malikeye> it's openstack... it's deploying 14.04
<malikeye> yep
<malikeye> it's gotta be something with my networking
#maas 2015-02-05
<malikeye> j #ubuntu-solutions
<smattan> hello I'm having some issues with my MaaS design and need some help if availible.  Has anyone been able to successfully deploy CentOS via MaaS?
<smattan> I am also looking for Windows solutions. as currently the only clue I had at wiki.cloudbase.it/maas doesn't seem to work anymore
<SimplySeth> would there be any objection to adding 'python-maas-client' and 'python-maas-provisioningserver' to PyPi  so that it can be used on all platforms ?
<smattan> hello I'm having some issues with my MaaS design and need some help if availible.  Has anyone been able to successfully deploy CentOS via MaaS?
<kiko> good morning
<kiko> smattan, we haven't published docs on it, so I would assume nobody apart from blake_r has :)
<saltlake> roadmr, malikeye: I tried everything I could read about adding a scsi controller to the guest and adding the blcok device.. but whatever I do the guest will not see the additional disks in lshw -class disk
<roadmr> saltlake: I downloaded the openstack installer but it's really complex, I haven't yet figured out how it verifies compliance of the nodes :/
<saltlake> roadmr: That is awfully sweet of u to do that.. but I am still stuck at my stupid VM unable to see more than its boot disk. I added a scsi controller and disks from the fedora page documentation .. and I am just unable to proceed!! :-(
<saltlake> http://docs.fedoraproject.org/en-US/Fedora/18/html/Virtualization_Administration_Guide/sect-Managing_storage_controllers_in_a_guest.html
<saltlake> roadmr: I FINALLY figured it out!!
<roadmr> saltlake: \o/ yay! what'd you do?
<saltlake> arggghhh : JOY!!
<saltlake> http://pastebin.com/vcE7JetA
<roadmr> cool :D
<saltlake> roadmr: http://pastebin.com/MuzueVEy
<roadmr> ohh I see, interesting way to define them as scsi
<saltlake> roadmr: Yes without that lshs -c disk was not showing them as disk..
<saltlake> HOWEVER : the maas webpage still does not show the "DISK (GB) to be a total of all 3 disks. I am hoping it picks up only the boot disk and that openstack installer still is able to finally work now!!
<roadmr> saltlake: did you commission the node? it will only gather hardware information after commissioning
<saltlake> roadmr: definitley I commissioend it and started it
<roadmr> saltlake: oh! ok then it should... but that information is gathered from lshw, so if lshw shows them, I'm pretty sure maas will be happy
<saltlake> roadmr: I then sshd to it like u told me to y'day and that's how I get the llshw -c disk info output
<saltlake> roadmr: this is the best I have so will go ahead and hope openstack installer is able to proceed !!
<roadmr> saltlake: hope it works :)
<saltlake> roadmr: thanks!! again thanks for responding and helping me out
<roadmr> saltlake: you did most of the work :)
<kiko> saltlake, neat :)
<saltlake> roadmr, kiko: thanks, it helps alot to ask and discuss here when I am all by myself headbanging in my little cave!!
<mbruzek> whit juju quickstart bundle:~hazmat/kubernetes/kubernetes
<mbruzek> wrong room
<malikeye> saltlake: I believe it only picks up the boot disk
<malikeye> well, it only reports that in maas
<SimplySeth> kiko: I'm in the process of asking permission to add the maas api python module to PyPi so it can be 'pip installed' from any platform
<saltlake> malikeye: cool but I have not got there yet but am hoping openstack installer still sees that!!
<saltlake> malikeye: thanks..
<malikeye> you'll find out when you get to the landscape part, if that's what you are goin
<malikeye> doing
<kiko> SimplySeth, that's cool, but do keep in mind it's part of a larger source tree -- I dunno how trees like that are handled
<malikeye> I'm giving up on the bundled package.... gonna try the manual setup
<malikeye> the automated one just fails and I can't figure out why... so step by step will at least let me narrow it down
<SimplySeth> kiko: I copied the module dir to my mac and it's running fine.
<SimplySeth> kiko: it just required 'pip install oauth'
<kiko> SimplySeth, I am wondering how we will handle updates to it
<SimplySeth> kiko: ahhh good point
<johng> hi all.  SSH keys that I entered in MAAS gui show in new VM instances on bootup, but ssh attempts fail with permission denied (publickey)
<johng> is there a better way to get teh kys in the instances working?  Or should I set a password somewhere in a kickstart or seed?
<serverascode> johng: which user are you trying to log in as?
<johng> not root.  using a regular user account $
<serverascode> I think by default it creates an ubuntu user, but I just started using maas yesterday lol
<johng> I put three keys from 3 diff machines in the MAAS UI preferences..
<johng> all machines fail with same message
<serverascode> ssh ubuntu@ip works for me with maas by default
<johng> doh!  your awesome!   thanks
<johng> I was thinking my user need to match the one at the end of the pub key..
<johng> guess not
<roadmr> johng: indeed the key you specify will be added to each node's "ubuntu" user's authorized_keys
<roadmr> johng: you could add a Host block to your .ssh/config specifying User ubuntu for the IP addresses of your maas cluster, this would avoid mechanically typing ubuntu@ every single time :)
<kiko> roadmr, +1 :)
<roadmr> TBH I don't mind typing ubuntu@ as I rarely need to log into raw nodes... but it's useful for debugging/learning
<saltlake> malikeye: I am now at another roadblock. I had all the client nodes on a private virtual network. I need the second nic so I added these lines to the VMs xml:    <interface type='bridge'>
<saltlake>       <source bridge='br0'/>
<saltlake>       <target dev='vnet2'/>
<saltlake>       <model type='virtio'/>
<saltlake>     </interface>
<saltlake> where br0 is a bridge to the eth0 on the host machine.
<saltlake> should this work to create 2 Nics per VM node ?
<saltlake> I am certain my second nic is configured properly!! but starting the VMAASServer with the 2 network interfaces .. I can longer reach the MAAS apache website.. what is going on ?
<saltlake> if anyone can pastebin their VM nodes xml with 2 NICs configured it would be awwesome ..
<roadmr> saltlake: sorry, as usual :( I've only ever used one nic in virsh :(
<serverascode> hi all, can I specify that a node gets a particular IP address using MaaS?
<saltlake> roadmr: No problem :-) iTs a journey no issues.. I tried a few things .. I added a second interface to the MAAS controller eth1 from the MAAS controller page. THen I went back to the VMAAS virsh xml and updated it to have a second interface. After that I can atleast start the MAAS Controller and i still get the MAAS webpage working.
<saltlake> roadmr: However ifconfig -a on the maasserver node does not assign the P address I specified on the cluster interface page setting.. (LIke when I set address 192.68.122.100 for eth0) ifconfig -a shows the same. that does not apply to eth1
<roadmr> hmm..
<serverascode> hi all...does "allocated to $user" mean the node is done, or should it have a status of deployed or something else? I'm having a hard time figuring out what the status should be
#maas 2015-02-06
<blake_r> serverascode: its means you own the machine
<blake_r> serverascode: you need to start the machine for it to deploy and install ubuntu
<blake_r> serverascode: once you click start you will see the status change to deploying and then deployed once finished
<malikeye> saltlake: I gave up on the openstack-install... I'm building everything from the OS docs now
<saltlake> malikeye: Oh no!! which os docs ? Are on redhat or ubuntu ? I am guessing ubuntu ??
<malikeye> http://docs.openstack.org/juno/install-guide/install/apt/content/
<malikeye> installing Juno.... then I'll go through the process of upgrading to Kilo
<saltlake> maikeye: U are on baremetal right ?
<saltlake> No vms right ?
<saltlake> malikeye: If I make no progress by midday with openstack-install I follow ur footsteps!!
<malikeye> my controller and neutron nodes will be VM's, but the nova nodes are bare metal, yep
<saltlake> malikeye: I just need a basic openstack so I can get to the storage aspect of things tp plug in the ceph cluster.. have no idea yet but hoe to get the openstack install done soon.
<kiko> malikeye, what happened to cause you to give up?
<saltlake> kiko, malikeye: I have progressed to my next problem with openstack-install: "There must be at least one machine, available in the MAAS cluster, that has both multiple disk and two or more network interfaces connected to the networks that are defined in MAAS. " reported in landscape. Now I know this error is valid. But I need a suggestion on the best way to fix.
<saltlake> I created 2 networks and registered them on the cluster. 1. isolated at 192.168.254.0 and made that a managed network etc. 2. maas-eth0 that is created by default. The nodes come up connected on maas-eth0 but what do I need to do so they also connect to isolated nw ? They all have 2 mac addresses etc but do not get assigned an IP address on "isolated"
<kiko> saltlake, that's normal, actually -- maas doesn't set the nodes to automatically configure any interface beyond the first (boot) interface
<kiko> saltlake, so if that's where you're getting stuck then there's something else amiss
<kiko> saltlake, now, does maas show two networks in your networks tab, with multiple interfaces attached to each?
<saltlake> kiko: yes it does show 2 networks.. I am trying to paste an image someplace
<saltlake> kiko: ON the maas page under networks: I see 2 networks: isolated and maas-etho wuth the network ip addresses. But landscape complains that there needs to be a node connected on 2 pr more networks and on "isolated" I see "attached nodes" to be 0
<kiko> dpb1, ping?
<saltlake> kiko: On each node I see 2 mac addresses from the VM interfaces info where one mac is not on any network.
<kiko> saltlake, have you set both interfaces as managed by maas?
<kiko> in the clusters page?
<saltlake> Kiko: Yes I set the "isolated" to be also managed with DNS and DHCP and gave it an IP address etc and dynamic range etc.. but when I login to the maas-controller VM node.. and do an ifconfig -a
<saltlake> I do not see the second ethernet interface get an IP address like eth0 on maas-eth0.
<saltlake> (I hope I am being clear)
<dpb1> saltlake: I'm looking for some instructions on this part, sec
<saltlake> http://pastebin.com/BkVzXz2J :
<dpb1> saltlake: some reference: http://askubuntu.com/questions/580427/openstack-deployment-with-maas-juju-and-autopilot
<dpb1> saltlake: any neutron gateway capable node in your system will need to show up in maas on both networks.  sometimes you have to manually associate the mac address to the network in the maas network tab (the interface is a bit tedious there)
<kiko> saltlake, as I said, the second interface won't get an IP address, that much is fine.
<saltlake> dpb1: So when I click on the "isolated network" It does list under "Connected network interface cards" and lists ALL the mac addresses
<saltlake> even the one on the second interface of the VM nodes.
<dpb1> saltlake: also, we have found that your second network (the one that you don't want MAAS to manage) cannot have DHCP/DNS running when you go to use the cloud installer.  getting it set up with those settings is fine, but you will need to turn it off before proceeding.
<dpb1> saltlake: and they all show up in maas-eth0, right?
<saltlake> dpbq: Ok.. I am still reading on the link u sent to see how to manually add those nodes in on the second interface. maas-eth0 is no problem they all get an IP address and show connected.
<saltlake> dpb1: BTW the isolated interface is a dummy. I don't know what it is going to be used for.. anything I really need to be aware of ?
<saltlake> actually I know I need to know ALLLOOTTT more :-D
<dpb1> saltlake: you don't need it, it's just a way some people like to configure things.  We actually test all the time with just one network in maas, as long as one machine has two network cards connected to that network, it should be fine.
<saltlake> dpb1: thanks .. thanks alot for helping me out!!
<saltlake> kiko: thanks ..
<saltlake> this channel is super helpful
<dpb1> saltlake: if you are making dummy networks (that are not physically separated), it could be leading to your issues.
<dpb1> saltlake: http://www.ubuntu.com/download/cloud/install-ubuntu-openstack  <-- those instructions actually talk about the single network case.
<saltlake> dpb1: I have everything on VMs on 1 server for familiary with Openstack.. with maas-eth0 things can get to the local network but the second nic is all virtual
<saltlake> which is an isolated virtual network with the maas controller and the other VM nodes on it!!
<saltlake> I really need to take some time to understand the architecture of all these components and how they talk to each other.
<dpb1> saltlake: ok... reading up on another couple things
<dpb1> sec
<saltlake> dpb1: I added a machine to the "isolated" network and I running openstack-install again and hope to not hit the 4th checkbox in point 6 on this link  http://www.ubuntu.com/download/cloud/install-ubuntu-openstack
<dpb1> saltlake: and, we don't really test the landscape openstack autopilot on KVM, since this is meant to be a bare metal installer.  You might instead want to try the single machine option of the openstack-installer?  There are some random issues you will run into with KVMs and MAAS.  Certainly there are people using it, but it's not a first-class citizen.  Just a warning.
<dpb1> saltlake: you are of course free to try things out as you please. :)
<nashville> well just figured i would drop by and thank everyone for their help yesterday
<kiko> nashville, hey, did everything work out ok?
<nashville> i cant get the screen to work but i have gotten everything installing and setup as i want
<nashville> just have to ssh to the boxes
<nashville> monitors are a no go
<kiko> nashville, likely an intel vt issue -- did you experiment with changing the kernel params?
<nashville> not sure if that means something but i would be interested to know if others can plug a monitor into the back of the hardware they install and it come up
<nashville> i did
<nashville> but still no dice
<saltlake> dpb1: That is not good..for me :-( since I don't have the hardware at my disposal till I showcare how to connecte up a ceph cluster to it .. .. Do you know of another installer that might be more friendly .. wt=ith KVMs
<nashville> i am having super problems with doing it in vm too
<nashville> but if its hardware its great
<nashville> exactly what i need
<nashville> our company is about to implement maas for our project :)
<roadmr> saltlake: if what you want is an openstack install, and not maas itself, you could perhaps try devstack (http://docs.openstack.org/developer/devstack/guides/single-machine.html), or if you have maas working and you're able to start nodes, but the openstack-install is still complaining, you could try a manual openstack install with juju
<kiko> nashville, what are you building?
<nashville> i like juju for what its worth
<nashville> just a system to auto install all our machines we ship lots of hardware boxes out which all need ubuntu
<nashville> and fai was a hassle
<nashville> so i decided to try something new and went with maas
<nashville> happy i did
<kiko> nashville, cool, keep me posted on any issues you run into
<nashville> so once they come up cfengine will see them install our software and after about thirty minutes they will be ready to ship
<nashville> with our custom partition setup and our ssh keys
<nashville> would have been a nightmare to do all that in fai
<nashville> kiko you helped me out a lot yesterday though at the time it prob didnt seem like that
<dpb1> saltlake: you used the 'openstack-installer' to get to this point right?
<nashville> i know my name is different but im sure you remember me crying yesterday
<dpb1> saltlake: openstack-install, I think is the command
<kiko> I realized
<nashville> seriously big thanks kiko i have never gotten support like that from anyone anywhere usually im just on my own
<nashville> so sincerely thank you
<kiko> you're welcome
<kiko> if you need more help just ask! we work hard so maas should Just Work, but there's still quite a few rough edges so we put the effort into helping out
<roadmr> saltlake: this question has some information on how to deploy openstack "manually" with juju: http://askubuntu.com/questions/78314/how-do-i-deploy-an-openstack-cloud-with-juju
<nashville> you know im more than happy to do any testing or anything that would help out the community
<nashville> maybe ill just sit in here and give what help i can now that i am through it and have a feel for everything
<kiko> nashville, if you have the time, just blog about how you got your setup working and that's already help enough
<nashville> i have been extensively writing about it on fb i guess i could just transfer that to a blog
<nashville> dont think my private fb posts count lol
<nashville> who are the people supplying support in here anyway?
<nashville> you guys devs on this project is that how you know so much about it
<nashville> all my info came from people who did what you just recommended and then you got me over the hump
<dpb1> saltlake: the reason I asked, is just openstack-install has a single machine option, it may be better suited to what you are attempting to do.
<saltlake> dpb1: Yes I will see if this last run of openstack-install also fails with something or the other.. I will quit and try single machine option. Then if that fails with follow roadmrs suggestion!! HA!!
<saltlake> roadmr: thanks.. looks like others have given up with openstal-install !! I am another oneof those maybe but will findout and will live to blog about it !!
<roadmr> saltlake: cool :) good luck heh
<saltlake> roadmr: :-D thanks
<malikeye> one of these would be nice : http://www.tranquilpcshop.co.uk/ubuntu-orange-box/
<saltlake> malikeye: looks like a powere generator!!
<saltlake> But in this case probably a power hogger
<roadmr> saltlake: the PSU is only 320W, so it's lower-power than the average high-end gaming PC
<saltlake> dpb1: atleast one thing got done: i.e as per page http://www.ubuntu.com/download/cloud/install-ubuntu-openstack  Point 6 got 4 green checkmarks
<dpb1> saltlake: great, as I said, techincally the pieces are there and vms will work.  It's just not something we test.  that is what the single machine install is designed to do, but would be cool to see your results. :)
<saltlake> dpb1: will def share results.
<dpb1> great. :)
<saltlake> dpb1: You were right.. landscape autopilot failed later. As soon as I tried to configure the stack, it fails to recognise atleast one machine connected to a "public interface" be it maas-eth0 or "isolated".
<dpb1> saltlake: :(
<saltlake> dpb1: I will now try the single machine install and see where it goes
<dpb1> saltlake: great, #ubuntu-solutions on freenode has people who can help with it (they wrote it).
<saltlake> dpb1: thanks for that tip
<malikeye> meh... when I add the br-ext, I lose my gateway, even though I'm forcing the eth0 gateway to route correctly
<saltlake> malikeye : I think I faced the same problem. I worked around that one by creating the network first on the MAAS controller and then adding that network interface to the Maas VM instance.. its weird
<nashville> just wondering does this look correct to you guys
<nashville>     500 10000 30720 ext4                            \                   $primary{ } $bootable{ }                \                   method{ format } format{ }              \                   use_filesystem{ } filesystem{ ext4 }    \                   mountpoint{ / }
<nashville> well formatting sucks but you get the idea?
<nashville> i am basically wanting to write all the parts to a certain size and leave the last one as /foo and have it consume the rest of the space
<saltlake> malikeye: I am going to use your link now.. the single machine install with openstack-install is over 2 hrs and not proceeding..
<stokachu> saltlake: takes awhile to complete, can you pastebin your ~/.cloud-install/commands.log?
<stokachu> saltlake: also jump on #ubuntu-solutions so we don't pollute this channel
<saltlake> stokachu: Sure I moved.
<xipasduarte> Hello, can anyone help me out? I would like to know how does one update Maas to a newer version (same version of ubuntu) and if this is wise. I am using ubuntu 14.04 with maas 1.5
<newell> xipasduarte, you can do an upgrade
<newell> sudo apt-get update && sudo apt-get upgrade
<newell> that should pull the new version of MAAS for you
<xipasduarte> I did and the version stayed the same
<xipasduarte> which I found a bit strange
<newell> did you see the new version being installed?
<xipasduarte> no, there never was an indication that the maas packages would be upgraded
<newell> what does apt-cache policy maas give you?
<xipasduarte> Installed: 1.5.4+bzr2294-0ubuntu1.3
<newell> ppa:maas-maintainers/stable
<newell> try adding htat
<newell> that will give you the new 1.7.1 release
<xipasduarte> I used the create Maas option on the ubuntu install process, might it be that the repositories were not added?
<xipasduarte> yes, that did it
<xipasduarte> thank you newell
<newell> np xipasduarte
#maas 2015-02-07
<heartones> Hi every one, can any one give me an idea, if it possible to install MaaS/Juju on two servers or I need more nodes for this
#maas 2015-02-08
<heartones> does any one experience bmc Failed to query node's BMC â Node could not be queried node...?
#maas 2016-02-08
<haasn> Hmm, it seems MAAS uses `ipmipower` to interact with IPMI 2.0 clients, but `ipmipower` does not seem to work on my server (always answers âinvalid passwordâ)
<haasn> ipmitool on the other hand works fine
<haasn> I wonder if it would be possible to get this changed
<haasn> Seems like ipmitool works and FreeIPMI does not
<haasn> Hmm, it seems like there used to be an ipmi.template in use but it was changed to a ânativeâ ipmi controller written in python in some version that I can't quite discern
<haasn> The bazaar web interface is virtually unusable for trying to figure out when this change happened, anybody got a clue? I'm trying to pull the âlastâ version of ipmi.template out of the history so I can rewrite it using ipmitool instead of freeipmi
<haasn> Ah, I can do a bzr checkout then convert that to git format using bzr fast-export | git fast-import in order to get it in a usable format :)
<haasn> incidentally, I tried setting up a RAID6 with 4 disks using the new storage config stuff in 1.9 and it flat-out doesn't work at all: https://0x0.st/XWQ.txt
<haasn> I tried it again and it (inexplicably) got farther, this time erroring out in GRUB
<haasn> but that seems to be a grub issue
<haasn> https://0x0.st/XWg.txt head /dev/zero > /dev/disk ?
<haasn> Isn't that horrendeously slow compared to dd if=/dev/zero of=/dev/disk bs=1M ?
<mup> Bug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>
<mup> Bug #1542982 changed: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>
<mup> Bug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>
<mup> Bug #1542982 changed: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>
<mup> Bug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>
<haasn> Hmm. I'm at the point where I've run into so many issues since switching from 1.8 to 1.9 that I just want to downgrade to see if the universe still makes sense at all, what ppa can I add to give me the 1.8 versions? The maas/stable one only contains 1.7 and 1.9 (and a few others)
<haasn> (turns out upgrading other stuff solved at least some of the most important issues)
<BlackDex> Hello there. I'm using maas 1.9, and i need to have two different vlan's on a single bond. is this possible?
<BlackDex> Ah, i think i have found it. Adding the vlan via the maas cli, and after that i'm able to use that in the GUI ;)
<mup> Bug #1541640 changed: MaaS region controller 1.9 fails on Ubuntu 15.10 <MAAS:Invalid> <https://launchpad.net/bugs/1541640>
<mup> Bug #1542761 changed: MAAS will not import images and there is no error as to why <MAAS:Invalid> <https://launchpad.net/bugs/1542761>
<jamespage> blake_r, hey - time for a bit of maas 1.9 network tuning?
<jamespage> I need to figure out how to set MTU on MAAS configured interfaces
<blake_r> jamespage: only one interface or the whole vlan?
<jamespage> blake_r, well mtu applies at the physical link level - I'm not actually using vlan's here
<blake_r> jamespage: yes but if you set it on the vlan that all the interfaces are connected to, then when you deploy those nodes, any that have a connection to that vlan will have the mtu set to that value
<blake_r> jamespage: that way you don't have to go to every node and set the mtu in MAAS
<jamespage> blake_r, ah right ok
<blake_r> jamespage: so if that is what you want
<blake_r> jamespage: just do this "maas my-maas-session vlan update 1 mtu=1500"
<jamespage> blake_r, I absolitely want everything connected to have the same mtu
<blake_r> jamespage: 1 being the vlan id
<blake_r> jamespage: not the vid
<blake_r> jamespage: use "maas my-maas-session vlans read" to get the list of vlans"
<jamespage> blake_r, hmm - not liking that command
<blake_r> jamespage: the update?
<jamespage> vlans read
<jamespage> blake_r, using 1.9.0 from the stable PPA
<blake_r> jamespage: do "maas refresh" first
<blake_r> jamespage: that way the maas client has the updated API interface
<jamespage> blake_r, gotcha
<jamespage> can I apply mtu to the untagged vlan?
<blake_r> jamespage: yes its jsut like another vlan in MAAS
<blake_r> jamespage: you will see it in the vlans listing
<jamespage> blake_r, {"__all__": ["Cannot modify the default VLAN for a fabric."]}
<jamespage> eek
<blake_r> jamespage: ah crap
<blake_r> jamespage: that should be allowed for the mtu value, crap
<blake_r> jamespage: file a bug please, will get that fixed in 1.9.1
<blake_r> jamespage: no you have 2 options
<blake_r> jamespage: now*
<jamespage> blake_r, will do
<blake_r> jamespage: either set it on each interface
<blake_r> jamespage: or I can show you how to do it manually using the maas shell
<jamespage> either is good with me - its only 9 interfaces in total
<blake_r> jamespage: then lets just do it over the API
<blake_r> jamespage: "maas my-maas-session interface update {node-system-id} {interface-id} mtu=1500"
<jamespage> blake_r, ok - trying that out now
<jamespage> maas applied to all interfaces on all nodes...
<blake_r> jamespage: ??
<jamespage> blake_r, testung now
<jamespage> blake_r, http://paste.ubuntu.com/14993910/
<blake_r> jamespage: looks good
<jamespage> blake_r, 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master juju-br0 state UP group default qlen 1000
<jamespage> looking good!
<blake_r> jamespage: cool
<blake_r> jamespage: please file that bug
<jamespage> blake_r, promise I will!
<blake_r> jamespage: thanks
<jamespage> blake_r, https://bugs.launchpad.net/maas/+bug/1543195
<mup> Bug #1543195 opened: unable to set mtu on default VLAN <api> <networking> <MAAS:Triaged> <MAAS 1.9:Triaged> <https://launchpad.net/bugs/1543195>
<jamespage> blake_r, hey - so I think xfs support for disks formats is on the roadmap right?
<jamespage> apologies for the questions - so close to 0 post deploy hacks...
<roaksoax> jamespage: xfs will be added yes
<jamespage> roaksoax, awesome - thanks for confirming
<roaksoax> jamespage: in fact, i think we can even get it to 1.9
<jamespage> \o/
<mup> Bug #1531843 changed: [Xenial 1.10] IPMI query fails <1.10> <ipmi> <python3> <xenial> <MAAS:Incomplete> <https://launchpad.net/bugs/1531843>
<haasn> the power status for one of my machines is stuck on âpower errorâ, with the last machine event being: Failed to query node's BMC - Power state could not be queried: [Errno 8] Exec format error
<haasn> I have fixed the problem in question and verified it works by manually opening a python instance and running os.execl on the file in question
<haasn> (/usr/sbin/ipmipower)
<haasn> But for some reason maas doesn't seem to get this. There are no machine events since then, despite me hammering the âcheck nowâ button. I don't see any ipmipower processes being spawned either. I tried rebooting the machine, restarting apache2 etc. but maas just doesn't seem to want to retry
<roaksoax> haasn: what maas version are you using? What hardware are you using? Are you sure the power credentials in MAAS are the correct ones?
<roaksoax> haasn: is there anything more on the logs (pastebin)
<haasn> roaksoax: maas v1.9, hardware of the machine in question is a sun fire x4200, I am sure the power credentials in MAAS are the correct ones because I can copy/paste them from the webUI to my own ipmipower invocation and turn the machine on and off
<haasn> e.g.
<haasn> su - maas
<haasn> /usr/sbin/ipmipower -D LAN_2_0 -h X.Y.Z.Z -u maas -p <snip> --stat  # this works fine
<roaksoax> haasn: is anything shown on the logs ?
<haasn> Hmm, seems there's a new entry as of 20:18: âpassword invalidâ
<haasn> why it took half an hour for it to switch from âexec format errorâ to âpassword invalidâ I do not know
<haasn> This seems like it might be executing the wrong version of ipmipower. I replaced ipmipower by my own version which contains a patch for the âpassword invalidâ bug on X4200 servers and moved it to /usr/sbin/ipmipower and the first one to /usr/sbin/ipmipower-orig
<haasn> Is it possible it's executing /usr/sbin/ipmipower-orig somehow?
<roaksoax> haasn: not at all
<haasn> I can't find the exact location of the command it's executing in src/provisioningserver/drivers/power/ipmi.py anywhere
<haasn> Okay, some update: It *is* spawning new ipmipower processes, they just exit so ridiculously fast that I couldn't see them in `watch`. With -n0.1 I see them popping up for a split second
<haasn> Maybe it's time to install auditd and log all process spawns so I can figure out what it's executing? Unless there's a simpler way
<haasn> Why it can't include the command it executed in the log files is beyond my understanding
<haasn> Well, then again, so is âwhy can't it log every failure to the log files instead of only every 30 minutes?â
<roaksoax> uhmm
<roaksoax> haasn: you could try hacking ipmi.py to log out what IPMI command is being send
<roaksoax> in MAAS' logs
<koaps> heya
<koaps> does anyone know what happened to MAAS 1.8.3? I don't see it available in stable anymore
<roaksoax> koaps: it has been superseeded by 1.9
<koaps> ya, but shouldn't it still be avail to get if I use the version name? I see 1.7.6 and 1.7.3 there
<haasn> Heh, auditd didn't want to work but I found a beautifully simple way to accomplish what I wanted: while true; do pgrep -a ipmipower; sleep 0.1; done
<haasn> this gave me a nice log of (virtually) all executed ipmipower commands
<haasn> Okay, I found the error and solved it
<haasn> The problem was that it was adding -W opensesspriv, and the -W flag overrides previous flags - so my âpatchedâ version of ipmipower (which was really just ipmipower-orig -W sun20 "$@") didn't really do anything. I changed it to ipmipower-orig "$@" -W sun20 and that way my sun20 workaround flag overrides the previous opensesspriv work-around flag
<haasn> Now it works
<haasn> And I can control the power status of all my servers just fine
<haasn> maas desperately needs some mechanism for adding your own IPMI workaround flags to nodes
<roaksoax> koaps: i wonder if the PPA removed it, since we have not made the removal of the version from the PPA itself
<roaksoax> haasn: you can always file a bug :)
<haasn> roaksoax: there is already a bug on the issue, I commented on it
<haasn> But I don't really want to be spending the next few months manually running ipmipower commands until it gets fixed
<roaksoax> haasn: well, that may seem like an issue with the firmware rather than MAAS itself
<roaksoax> haasn: since the firmware requires non-standard paramters
<haasn> roaksoax: Yes okay but that doesn't really help admins who want to provision servers with MAAS. FreeIPMI contains these work-arounds for a reason, not exposing the ability to use them to the admins in charge of running the machines when it would literally be nothing more than one input box and one line of code for appending it to the command line is just being annoying for the sake of annoying, IMO
<roaksoax> haasn: right, but in situations like this, providing ability to do work around is not always something that will improve MAAS' robustness
<roaksoax> haasn: as this is an isolated issue
<mup> Bug #1543286 opened: Exception: 'Node' object is not iterable <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543286>
<mup> Bug #1543301 opened: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301>
<mup> Bug #1543301 changed: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301>
<mup> Bug #1543301 opened: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301>
<haasn> This is weird. Often, when I try deploying to a VM, I end up in the weird situation where the deploy process seems to be complete but the machine then just gets stuck in an offline state after it was supposed to reboot
<haasn> if I ssh into the machine, I see The system is going down for reboot NOW!
<haasn> But.. it never reboots. It just powers off
<haasn> And MAAS also still thinks it's turned on until I manually hit âcheck nowâ
<haasn> Power control is via virsh
<haasn> What could possibly be going wrong here? Evidently the machine thinks its rebooting, it just.. doesn't
<haasn> Hmm, seems to be an inherent part of KVM; âsudo rebootâ has the same issue for some reason
<fluxcore> I use plenty of KVM stuff outside of maas/openstack and don't have any such problems...
<haasn> I've used dozens of KVM VMs on this same vm host, and it's running like 5 other VMs right now that all work fine
<haasn> as in I can reboot them
<haasn> but the ones used by maas have this inexplicable failure
<roaksoax> haasn: Is this during deployment ?
<roaksoax> haasn: also, maas doesn't query the VM's/BMC's every second. WE do it ever few minutes to ensure BMC's dont go crazy
<roaksoax> haasn: i deploy about ~100 to ~200 vm's every few weeks for testing, and have not seen such behavior. However, I've seen a few occurrences where they all fail to PXE at once because there are too many VM's on the same host
<roaksoax> haasn: but that's how the host's bridge was configured
<haasn> roaksoax: Yes, this is during deployment
<haasn> I just reproduced it again: clicked the machine, hit âdeployâ in the web UI, chose ubuntu 14.04 lts and default kernel
<haasn> it started deploying up until the point where rebooted the machine
<haasn> and now it's just offline
<haasn> I'll try removing the machine and re-adding/commissioning it in kvm. Maybe something got messed up because I used an xml dump to batch-create a bunch of VMs
<haasn> destroyed the VM and recreated it and it seems to be working now
<haasn> must be some weird failure to do with the way it was created, idk
#maas 2016-02-09
<haasn> Hmm.. some update on the IPMI issue I had: Turns out when I said it was working.. well, not fully. It works for --stat, but it doesn't work for turning the power on or off
<haasn> But the odd thing is: For power on/off it doesn't even seem to be launching ipmipower at all
<haasn> I don't see any processes created the way I see processes created every time I click âcheck nowâ
<haasn> Ah, it's going through ipmi-chassis-config - of course
<mup> Bug #1524853 changed: DHCP lease allocation does not match deployed node's IP address <MAAS:Expired> <https://launchpad.net/bugs/1524853>
<haasn> What's the easiest way to customize the images usable in MAAS, preferably in a uniform way? For example, I want to make a change to the default /etc/cloud/cloud.cfg to change the way new systems are deployed
<haasn> But I don't really want to maintain my own custom image - I'd rather this change be âuniversalâ, if that makes sense
<haasn> Hmm. I guess I want to be modifying preseeds/curtin_userdata?
<haasn> No, that doesn't seem right. Curtin only deals with the installation process - but I want to customize things *after* the installation is successful (i.e. from within the new system, before cloud-init runs)
<mup> Bug #1519810 changed: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>
<mup> Bug #1519810 opened: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>
<mup> Bug #1519810 changed: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>
<haasn> I think part of what I want to accomplish is done by adding late_commands to curtin_userdata. I tried adding two commands (to run `apt-get update` and `apt-get upgrade` automatically on newly deployed nodes) but now I get some weird behavior: when deploying, the log eventually changes from âdeployedâ to âfailed deployingâ but the node itself still gets displayed as âdeployingâ
<haasn> Seems like there's some sort of weird inconsistency in the MAAS database itself here, where the log clearly thinks the status got changed but the UI disagrees
<haasn> Without the lines I added it works fine
<haasn> The change I made is adding the two apt-get lines here: https://0x0.st/Xt8.txt
<haasn> Anybody know what's wrong with them? It's not like I can see a log output because the UI doesn't even think installation failed (yet)
<haasn> Including just the update line seems to work fine, it's the upgrade line that's causing problems
<haasn> Maybe because it takes too long?
<haasn> The command does seem to be running âjust fineâ
<haasn> Ah, there's some errors in the install log still visible after the failed deployment
<haasn> I think I need to run the upgrade from inside the newly booted system
<haasn> rather than curtin in-target
<haasn> seems like I would want it to go in user_data, but I can't set this via the web UI nor do I want to manually set it for every single node, wouldn't it be possible to have some kind of global âdefault user_dataâ?
<roaksoax> haasn: why not use the daily images instead? and if you see failed deployment after update/ugrade then something is failing
<roaksoax> road/win 4
<mup> Bug #1543707 opened: MAAS 1.9+ should not allow whitespace characters in space names <MAAS:New> <https://launchpad.net/bugs/1543707>
<haasn> roaksoax: daily images? I am using them, at least as far as I'm aware
<haasn> âlast update 09 feb 2016â
<haasn> but deploying a machine and logging into them still requires like 300 package upgrades
<haasn> there is some mention on the bug tracker of this https://bugs.launchpad.net/maas/+bug/1492531
<haasn> looks like it's not using daily images by default and afaik there's no way to change that without changing the source?
<haasn> The milestone is set as MAAS v1.9.0, perplexingly enough
<haasn> 1.9.0 was released and I'm using itb ut the bug still seems to be open?
<haasn> Ah, the base URL is a setting in the settings menu
<haasn> Unrelated: It seems maas and all of its deployed nodes all automatically live in Etc/Utc timezone, which does make debugging logs etc. slightly more difficult than it needs to be. Is this intentional or could I go ahead and change them all to a german timezone if all my servers are going to be in germany?
<roaksoax> haasn: the bug refers to using the images from the 'releases' stream, not from the 'daily' stream. Did you change the stream of images in the settings page?
<haasn> roaksoax: I did as soon as I was made aware that you could change the stream of images on the settings page. (I was looking for the setting in the conf files to no avail)
<roaksoax> haasn: oh! i think that may  be becuase there's an issue with the image publishing right now, which may not be publishing newer images
<roaksoax> haasn: but will have a look in a sec
<haasn> roaksoax: I'm confused. I already solved my problem, didn't I?
<roaksoax> haasn: right, if you are using daily images then you should be ok, becuase the releases images are out of date
<roaksoax> haasn: releases are more in line with current release than latest packages
<roaksoax> haasn: but if you using the dailies, and it still has tons of packages to update
<roaksoax> then there's two options
<roaksoax> 1. new images not being published
<roaksoax> 2. or something else is going on
<haasn> I dunno, I haven't tested the dailies yet. (It's still synchronizing them)
<haasn> I'm going to strongly assume the problem will be fixed after switching to the dailies
<roaksoax> haasn: ah ok! then you should be good to go with daily
<haasn> Hmm. Is it normal to have ridiculously high IO usage on postgresql after syncing images?
<haasn> Like, the machine is nearly unresponsive
<haasn> `service restart postgresql` seems to have fixed it?
<roaksoax> haasn: for HA, the images are stored in the DB
<roaksoax> haasn: and tehn synced to the clusters
<haasn> Ah, after waiting a bit the io usage is going up again
<haasn> roaksoax: Should I get a dedicated disk for postgresql just to avoid this clogging up the system?
<haasn> Or what's the intended design here?
<haasn> It's running on a RAID10 of 15000rpm server drives
<haasn> And has 10+ GB of RAM left unused
<haasn> Maybe if I could get postgresql to keep more in memory and write it slower to the disk that would be fine
<roaksoax> yeah that should be fine
<roaksoax> haasn: we don't have intended design for postgres
<roaksoax> haasn: but with MAAS 2.0, the intention is to have the region controller separated from the cluster controllers  *if* we want to support HA
<roaksoax> haasn: so we can have Postgresql in HA, Multiple Region Controllers, Multiple cluster controllers
<haasn> Will it still be possible to have the region controller and cluster controller on the same machine for small setups?
<haasn> I'm using it to organize a max of 10-20 machines
<roaksoax> haasn: yes you will
<haasn> roaksoax: I had a chat with #postgresql and it seems ubuntu's defaults for checkpoint_segments is too low
<haasn> postgresql is getting overloaded with the large writes and writing checkpoints constantly, hence the high load
<haasn> Not sure if it would somehow be possible to tune this parameter when installing maas but I have no idea if there's a favorable interaction here
<haasn> (same for checkpoint_completion_target and checkpoint_timeout)
<roaksoax> haasn: maybe it would be good to change the defaults altogether
<haasn> roaksoax: Deployed from the daily image now, only like 5 packages needed upgrading so it seems to be working. Thanks for the suggestion!
<roaksoax> haasn: no prob! we will have better UX for that in 2.1 hopefully
<haasn> roaksoax: I would still love an easier way to set up a list of scripts to be run on fresh installs
<haasn> for example use cases I may want all new nodes to install a puppet client or set up some ssh certificates
<haasn> My ideal UX for this would be a list of shell files I can dump into a postinstall/ directory inside /etc/maas/preseeds
<roaksoax> haasn: right, there's definitely a lot of things we have to improve overall
<roaksoax> haasn: but we are getting there
<haasn> just having shell files like XX_setup_puppet.sh that gets run after a new install would be great
<haasn> and not that hard to implement
<haasn> Btw, how much âsecurityâ is in the design of MAAS? Should I ensure that all communications between the cluster controller and nodes it's setting up happen on a private network? What about API keys, is it safe to access the maas API over the internet using an API key, or does that mean anybody else could steal my API key and do stuff?
<haasn> i.e. does the API key function like a private/public system to ensure integrity and identity or is it just a replicatable token
<roaksoax> haasn: region/cluster communication is not *yet* happening securely, but we are in the process of getting that in the roadmap for 2.1 or 2.2
<mup> Bug #1543828 opened: Install of 1.10 fails due to simplejson dependency <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1543828>
#maas 2016-02-10
<mup> Bug #1543917 opened: python3-seamicroclient missing library prevents cluster from starting <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543917>
<mup> Bug #1543917 changed: python3-seamicroclient missing library prevents cluster from starting <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543917>
<mup> Bug #1543917 opened: python3-seamicroclient missing library prevents cluster from starting <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543917>
<mup> Bug #1543917 changed: python3-seamicroclient missing library prevents cluster from starting <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543917>
<mup> Bug #1543917 opened: python3-seamicroclient missing library prevents cluster from starting <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543917>
<haasn> It would be great if nodes could get a âdescriptionâ field that you can optionally fill in when deploying. (same goes fro the API)
<haasn> That way services like juju or users of the web UI could have a quick way of providing very small but potentially very helpful annotations to the list of nodes
<mup> Bug #1543968 opened: MAAS 1.9.0 allows non-unique space names and shows no space id in a subnet <MAAS:New> <https://launchpad.net/bugs/1543968>
<mup> Bug #1543975 opened: nodes and interfaces api/command line reference spaces by name and not id <MAAS:New> <https://launchpad.net/bugs/1543975>
<mup> Bug #1274432 opened: MAAS does not make me a sandwich <MAAS:Fix Committed> <https://launchpad.net/bugs/1274432>
<mup> Bug #1544143 opened: start operation returns success even if series not available <MAAS:New> <https://launchpad.net/bugs/1544143>
<mup> Bug #1544211 opened: maas power type driver for ravello systems <MAAS:New> <https://launchpad.net/bugs/1544211>
<mup> Bug #1544258 opened: cannot refresh cli unless all endpoints are available <amd64> <apport-bug> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544258>
<mup> Bug #1544258 opened: cannot refresh cli unless all endpoints are available <amd64> <apport-bug> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544258>
<mup> Bug #1544258 changed: cannot refresh cli unless all endpoints are available <amd64> <apport-bug> <xenial> <MAAS:Triaged> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544258>
<mup> Bug #1544258 opened: cannot refresh cli unless all endpoints are available <amd64> <apport-bug> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544258>
<mup> Bug #1544308 opened: MAAS migrations import maasserver <MAAS:Triaged> <https://launchpad.net/bugs/1544308>
<redelmann> Hi there!
<redelmann> im trying to install a fresh maas 1.9 server
<redelmann> after i set everithing in server
<redelmann> and start comissioning a node
<redelmann> i get an error with apt proxy
<redelmann> inside maas server /var/log/maas/proxy/access.log
<redelmann> i found a lot of "1455064516.921      0 10.10.10.151 TCP_MISS/503...."
<redelmann> it's a know bug?
<redelmann> or there is somthing wrong with my settings?
<roaksoax> redelmann: TCP_MISS means it tried accessing the proxy for whatever it was downloading at the time, but it wasn't found on the cache
<roaksoax> redelmann: so it will have the proxy download it aain
<redelmann> roaksoax, ok, im trying to get node logs
<redelmann> roaksoax, in screen i can see that it faild make some apt update or install
<redelmann> roaksoax, if i disable "enable the use of APT and HTTP/HTTPS"
<redelmann> roaksoax, node comissioning finish ok
<roaksoax> redelmann: have you set an upstream DNS ? That might be it
<redelmann> roaksoax, DNS on maas server system?
<redelmann> roaksoax, actually i have internet on maas server, so i think DNS are OK
<roaksoax> redelmann: the machines that you deploy are, by default, pointed to the MAAS server
<roaksoax> redelmann: where we run DNS
<roaksoax> redelmann: you can tell MAAS that you have other DNS' on the network
<roaksoax> so they can resolve
<roaksoax> upstream
<redelmann> roaksoax, LOL, you was rigth, i forget to set DNS upstream
<redelmann> roaksoax, i will try it
<redelmann> roaksoax, thank you very much
<roaksoax> redelmann: no probe!
<redelmann> roaksoax, it didnt help
<redelmann> roaksoax, i copy /etc/apt/apt.conf.d/95cloud-init-proxy to another machine
<redelmann> roaksoax, and im having some error: Err http://archive.ubuntu.com trusty-updates/universe amd64 Packages
<redelmann>   503  Service Unavailable
<roaksoax> redelmann: is thison a differnet network?
<redelmann> roaksoax, same network
<redelmann> roaksoax, also i can navigate into http://maas-server-ip:8000
<redelmann> roaksoax, and it show to me some squid page
<redelmann> roaksoax, im gonna google some default/safe squid proxy
<redelmann> roaksoax, and override maas squid proxy for testing
<roaksoax> sure!
#maas 2016-02-11
<mup> Bug #1544385 opened: Deploy of Virtual Machine fail using MAAS <architecture-ppc64le> <bugnameltc-136402> <severity-high> <targetmilestone-inin14044> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544385>
<mup> Bug #1544385 changed: Deploy of Virtual Machine fail using MAAS <architecture-ppc64le> <bugnameltc-136402> <severity-high> <targetmilestone-inin14044> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544385>
<mup> Bug #1544385 opened: Deploy of Virtual Machine fail using MAAS <architecture-ppc64le> <bugnameltc-136402> <severity-high> <targetmilestone-inin14044> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544385>
<mup> Bug #1274432 changed: MAAS does not make me a sandwich <MAAS:Opinion> <https://launchpad.net/bugs/1274432>
<mup> Bug #1274432 opened: MAAS does not make me a sandwich <MAAS:Opinion> <https://launchpad.net/bugs/1274432>
<mup> Bug #1274432 changed: MAAS does not make me a sandwich <MAAS:Opinion> <https://launchpad.net/bugs/1274432>
<haasn> When trying to commission a particular machine, I get kernel stack traces constantly during boot
<haasn> (and it seems âstuckâ)
<haasn> But it worked in the past, not sure what has changed
<haasn> Is there any way I can set like a âfallbackâ or âbackupâ commissioning image that uses an older version or something?
<mup> Bug #1274432 opened: MAAS does not make me a sandwich <MAAS:In Progress by mpontillo> <https://launchpad.net/bugs/1274432>
<haasn> I don't understand this error: âCluster interface only has a dynamic rangeâ. I get this when I leave a node's interface on âAuto assignâ. It's attached to an interface that the maas cluster controller is managing DHCP+DNS on, so huh?
<haasn> Oh, I think I get it: I need to configure a separate âstaticâ range
<mpontillo> haasn: yes; in MAAS 1.9 you need a static range to be defined on the cluster interface in order for MAAS to know which IP address pool it is allowed to automatically allocate from
<haasn> âMust be in the same network as the dynamic rangeâ is a bit misleading. I took this to mean that the static range must be a subnet. But in reality, they cannot overlap
<mpontillo> haasn: correct; the dynamic range is for addresses offered by DHCP. the static range (which must be in the same CIDR) is for addresses that MAAS assigns to deployed nodes.
<haasn> I'm just suggesting the documentation be rewritten to make this a bit clearer
<mpontillo> haasn: you can set the node to DHCP if you prefer it to use a single pool. when you use AUTO, we'll write the IP address to /etc/network/interfaces when the node deploys, so there is no dependency on DHCP
<haasn> mpontillo: I'm actually missing a third option: Use the static IP detected when commissioning
<mpontillo> haasn: that's a good point; it's funny that you mention that, because I was going to re-work some of the networking documentation today ;-)
<haasn> For one of my clusters I'm using a flat networking setup with an external DHCP server, I'd like it to hard-code the IPs assigned to the node by that DHCP
<haasn> I can set it to âDHCPâ but then it still needs DHCP on every boot :p
<haasn> (Then again, I guess it always does for PXE)
<mpontillo> haasn: yes, unmanaged networks are a little more difficult for MAAS to support, but it can be done. but you may be able to use static assignment in this case (although it's true that we may not auto-fill the field with the address we saw during commissioning)
<mpontillo> haasn: because in MAAS's view, that was an ephemeral address we saw when the node DHCP'd; it would be dangerous to allow the user to use it statically, when we aren't sure if the DHCP server has reassigned it
<haasn> I guess it's not a big deal
<haasn> but I would like to be able to change the default setting for a particular subnet
<haasn> i.e. default âAuto assignâ or default âDHCPâ
<haasn> (In the interface settings for the cluster)
<mpontillo> haasn: that's not a bad idea (you might consider filing a "Wishlist" bug), but I would be more interested in understanding why MAAS can't handle your DHCP services. MAAS works better with that end-to-end integration, in my experience
<haasn> mpontillo: Because this cluster is managing a few âspareâ hosts that are in a public subnet including many other hosts, most of which have advanced DHCP requirements (e.g. unusual PXE setups, hardcoded MAC addresses everywhere, etc.)
<haasn> So I really can't change the fact that an external DHCP is managing this subnet, and I can't exactly have two DHCP servers running in the same broadcast domain
<mpontillo> haasn: right. if you were able to include arbitrary configuration snippets into the DHCP server, would that allow you to use MAAS for DHCP?
<haasn> mpontillo: Unlikely. I don't see a point either, the external DHCP server is running off ~600 lines of dhcpd.conf, redundant power supplies, etc.  Moving a big part of the server farm infrastructure into some the MAAS controller (which is just a virtual host) is not something that I feel comfortable doing in any situation
<mpontillo> haasn: understood. thanks.
<mpontillo> haasn: can I ask what software that DHCP server is running? if you had a way to integrate it more tightly with MAAS, so that MAAS could get a more complete view of the network, would you be willing to consider doing that?
<haasn> mpontillo: ISC dhcpd
<haasn> I don't plan on using MAAS heavily in this subnet
<haasn> (I do plan on using MAAS heavily in a different subnet)
<haasn> But for the purposes of this subnet MAAS is just an easy âget me some spare hardware so I can test some stuff nowâ button
<haasn> So I sort of want to keep it isolated
<haasn> The setup right now is that ISC handles PXE for the nodes (hard-coded via MAC addresses) and forwards them to MAAS via next-server
<mpontillo> haasn: thanks. for "unmanaged" networks such as these, the behavior actually changed between MAAS 1.8 and MAAS 1.9. In MAAS 1.8, when we deploy a node, we leave it configured to DHCP. In MAAS 1.9, we write the IP address to /e/n/i for reliability
<mpontillo> haasn: this has the advantage that the nodes can still boot without DHCP, but the disadvantage that you must have a static range defined (unless you choose DHCP)
<mpontillo> haasn: it sounds like that's the basic issue you were hitting
<haasn> ah
<haasn> mpontillo: I did run into one issue when migrating to v1.9 where it would auto-assign a new deployment the IP .1, which was the default gateway of the subnet (and that was not a fun IP clash to run into :p)
<haasn> Even though I had no subnet range configured iirc
<haasn> (Or maybe I did, for the purpose of having it statically assign these)
<mpontillo> haasn: ouch. while that does sounds like a bug to me, I can see how MAAS might be confused in that case. (if it doesn't have any information on that subnet, it would think that all the IP addresses in that subnet are free for the taking)
<mpontillo> haasn: I think we need to improve MAAS's network discovery in order to prevent situations like this.
<mpontillo> haasn: like DHCP, we should never assign an address that we can tell is already in use
<haasn> mpontillo: It could do an effort to try an ARP on the IP before assigning
<haasn> And maybe also a ping
<mpontillo> haasn: agreed. the subtle issue regarding ARP is that there is no requirement that the MAAS server actually be on the same L2 network as the host being deployed (though admittedly it's difficult to convince MAAS to allow this, at the moment)
<haasn> mpontillo: I guess the idea is to âtry many, fail if any failâ
<haasn> Servers might be ignoring ping, too
<mpontillo> haasn: indeed.
<Lyncos1> Hi,
<Lyncos1>  I'm trying to figure out how to make a custom Ubuntu/Debian image for maas.. but cannot find the relevant documentation... the only thing I find is scripts to do Redhat or Centos stuff
<haasn> mpontillo: Could also display a warning if the reverse DNS entry for the IP doesn't match the detected hostname for a node
<haasn> Incidentally, I still can't figure out why maas refuses to use the existing hostnames of machines with two attached NICs
<haasn> Only one of them gets an IP (the other is ignored)
<haasn> And that IP has an associated hostname
<roaksoax> haasn: maas only creates a dns entry for the IP used for PXE booting
<haasn> roaksoax: Then my behavior seems to be a bug
<haasn> Because that's the only interface that's allow to PXE boot to begin with (because the other doesn't even have a DHCP entry)
<haasn> But as soon as I take a second NIC, and plug it into the same network
<haasn> It ignores the existing hostname and generates a random one
<haasn> (Or even into a different network. Even one that is not internet-facing)
<haasn> Although, perplexingly, it works fine on another machine - which has 4 NICs, two of which are plugged into different network
<haasn> Hmm. Maybe, just maybe, it's complaining about the interface ordering? The affected machine has the PXE/boot/connected interface as eth1 and the other one (no PXE, only internal IP) as eth0
<haasn> Actually I think the one with 4 NICs only works because it got detected when only one was plugged in
<haasn> The one that doesn't work got detected afterwards
<haasn> Anyway it wouldn't be as big of a deal if I could figure out a way to change a node's name
<haasn> It seems to be set in stone once detected
<haasn> I could just manually rename it from âtight-breadâ to âwsl44â and that way it would actually make sense next to wsl42, wsl43 and wsl45
<haasn> Can I do this via the API?
<haasn> Also has the wrong value in /etc/hostname on a deployed system :/
<haasn> (Which actually causes some issues, because now e.g. sudo complains about not being able to resolve the host - which of course it can't, as it's a made up name rather than the perfectly fine host name that the machine should already have)
<roaksoax> haasn: that's probably because the second interface is DHCP'ing from MAAS' dynamic range
<roaksoax> haasn: and that is given a different DNS name
<haasn> roaksoax: The problem persists even if it's plugged into the same network (with no maas DHCP)
<mup> Bug #1544385 changed: Deploy of Virtual Machine fail using MAAS <architecture-ppc64le> <bugnameltc-136402> <severity-high> <targetmilestone-inin14044> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544385>
<mup> Bug #1544385 opened: Deploy of Virtual Machine fail using MAAS <architecture-ppc64le> <bugnameltc-136402> <severity-high> <targetmilestone-inin14044> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544385>
<mup> Bug #1544385 changed: Deploy of Virtual Machine fail using MAAS <architecture-ppc64le> <bugnameltc-136402> <severity-high> <targetmilestone-inin14044> <maas (Ubuntu):New> <https://launchpad.net/bugs/1544385>
<mup> Bug #1544743 opened: changing fabrics and other entries in network/storage is a bit flaky <landscape> <MAAS:New> <https://launchpad.net/bugs/1544743>
<mup> Bug #1544743 changed: changing fabrics and other entries in network/storage is a bit flaky <landscape> <MAAS:New> <https://launchpad.net/bugs/1544743>
<mup> Bug #1544743 opened: changing fabrics and other entries in network/storage is a bit flaky <landscape> <MAAS:New> <https://launchpad.net/bugs/1544743>
<mup> Bug #1544757 opened: Maas 1.9 Power Drivers for HP Moonshot are no longer stable <hyperscale> <MAAS:New for newell-jensen> <MAAS 1.10:New> <MAAS 1.9:New> <https://launchpad.net/bugs/1544757>
#maas 2016-02-12
<mup> Bug #1274432 changed: MAAS does not make me a sandwich <MAAS:Won't Fix> <https://launchpad.net/bugs/1274432>
<mup> Bug #1544795 opened: MAAS CLI should preserve the URL the user enters <MAAS:New> <https://launchpad.net/bugs/1544795>
<haasn> Hmm. I'm trying out a new networking setup, where maas handles DHCP - and I plugged a new node into this subnet and booted it successfully, but after it ends with âcloud-init successfulâ etc.; it just does nothing
<haasn> the node doesn't show up, the machine stays on
<haasn> Ah
<haasn> I tried changing the NTP IP published via DHCP to the one of the maas internal address, now I seem to get further - it's stuck in a loop because network is unreacahable
<haasn> Because it's only plugged into a private network atm
<haasn> (I want to test this configuration, where the MAAS nodes have no public interface at all)
<haasn> Ah, maas-proxy configuration was missing an allow line for the new internal subnet I added
<haasn> It should really automatically configure itself to allow all the subnets the cluster controller is in
<haasn> (Or perhaps the one it's managing)
<haasn> I'm still not sure why exactly it's failing. Says it's trying to contact 169.254.169.254, I don't recognize that IP
<dweaver> haasn, that is the metadata service IP, like on AWS.  MAAS provides a metadata service on IP 169.254.169.254 for an image to contact using cloud-init.  The IP should be redirected to MAAS server in iptables.
<haasn> Can't resolve DNS either, even though dig @cluster-controller ubuntu.com works
<haasn> dweaver: Okay. It might be that the iptables setting in the node is wrong then. How can I log into it for debugging? I don't have any login details for this image
<dweaver> haasn, so, if the image isn't contacting cloud-init, then it didn't get an SSH key and you won't be able to log in.
<haasn> The last thing I see is âBEGIN SSH HOST KEY KEYSâ, followed by some keys, followed by âEND SSH HOST KEY KEYSâ and âcloud-init v. 0.7.5 finishedâ
<haasn> and then cc_final_message.py[WARNING]: Used fallback datasource
<haasn> And now it's doing nothing
<haasn> No reboot, no new entry in maas
<dweaver> haasn, Yes, that's the clue - used fallback data source.
<dweaver> It should have used the MAAS data source, so it couldn't get to 169.254.169.254 and obtain data.
<dweaver> haasn, If you are using MAAS server as the router gateway, then it should "just work".
<dweaver> haasn, sounds like something on the network config is not working for you.
<haasn> dweaver: MAAS server is not a gateway
<haasn> The nodes cannot reach the internet at all
<haasn> They are on a private subnet with just the maas controller
<haasn> And the maas controller has a DNS, HTTP and APT proxy configured
<dweaver> haasn, so you haven't set a default route at all??
<haasn> dweaver: Correct
<dweaver> haasn, Try setting the default gateway to the MAAS server then
<dweaver> Then packets destined for 169.254.169.254 should get sent to the MAAS server and iptables rewrite them
<haasn> Oh, the iptables is on the MAAS server, not the node
<dweaver> Otherwise they have nowhere to go.  Yes
<haasn> Is this done so that the discovery images don't need to know about the IP of the maas region controller?
<haasn> Wouldn't it be easier to pass the IP as a boot-time kernel parameter?
<haasn> dweaver: Works now, thanks!
<dweaver> haasn, MAAS is architected to be like a cloud provider, and use cloud-init on boot, which uses a metadata server for the data, like AWS, Openstack, Azure, GCE, etc.
<haasn> Yeah, but I mean can't the metadata server pull the IP it's contacting from a kernel parameter
<haasn> and then maas could provide its own IP as the kernel parameter when PXE booting
<haasn> Then you wouldn't need the iptables âhackâ
<haasn> s/the metadataserver/cloud-init/
<dweaver> haasn, Well, it could be done many, many different ways.
<haasn> It does seem like some MAAS stuff is already pulling stuff from kernel parameters, e.g. during the commissioning process there's some URL to the maas server in there
<dweaver> The hack is however a standardised method of providing data to a booting image over the network and allows cloud-init to always use the same method.
<haasn> fair enough
<haasn> I wonder if it would be possible to have the DNS zone setting on a per-interface basis, so I can configure a different DNS zone for every subnet
<binoy> Is there any package available for maas to do the api calls
<Razva> hey folks! my server has two NICs: eno1 and enp0s25. I don't know why but enp0s25 is not detected in any way
<Razva> it's a fresh Ubuntu 15.10 install
<Razva> 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05)
<Razva> 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
<Razva> 2: enp0s25: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
<Razva> 3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
<haasn> How do tags work? I'm trying to create a tag for virtual machines that looks like definition='contains(product, "KVM")', but I don't see this tag being applied to any nodes
<haasn> Here is an example lshw output: https://0x0.st/XEl.txt
<haasn> It says lshw:product: KVM ()
<haasn> also tried /product, //node[class="system"]/product
<haasn> I am using tag rebuild followed by tag list to check, and there are 0 nodes every time
<Razva> I swear to God that I don't see any Networks in the menu: http://i.imgur.com/wKDBvvX.png
<haasn> s/tag list/tag nodes/
<haasn> Razva: JavaScript blocked or something?
<Razva> nope...
<haasn> I copy/pasted the QEMU example from http://en.community.dell.com/techcenter/os-applications/w/wiki/7432.using-tags-with-maas-and-juju-in-ubuntu-server-14-04-lts and changed âQEMUâ to âRed Hatâ (since that's what my VMs show up as) and it still does not work
<haasn> Is the tag mechanism broken? Even examples _straight from the documentation_ simply do not work
<haasn> I tried verifying my tag using the XPath evaluator at http://www.utilities-online.info/xpath/
<haasn> That one required feeding it //lshw:node[@class='system']/lshw:product for it to be evaluated correctly
<haasn> (The lshw: prefixes are notably absent from the MAAS docs)
<haasn> But even with the lshw: prefixes it does not work inside the actual maas
<haasn> Has anybody tested the tag mechanism at all?
<haasn> Is there a single working example on the internet? MAAS v1.9
<haasn> I don't think this documentation has been touched since v1.3, judging by the history
<haasn> I extracted the lshw xml manually with lshw -xml > /tmp/lshw.xml and ran xmlstarlet sel -T -t -v 'contains(//node[@class="system"]/product, "KVM")' /tmp/lshw.xml and it returns âtrueâ. So...
<haasn> MAAS is clearly bugged, other XPath utilities find this just fine
 * haasn opens a bug report
<mup> Bug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>
<mup> Bug #1544962 changed: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>
<mup> Bug #1544962 opened: MAAS tags don't find any nodes <MAAS:New> <https://launchpad.net/bugs/1544962>
<Razva> folks really, I cannot boot from MAAS
<roaksoax> haasn: on your bug, please attach the lshw output from MAAS, that will help instead of gathering the one from the system
<roaksoax> redelmann: hi there. That doesn't really provide us much explanation. Can you expand?
<redelmann> roaksoax, ??
<roaksoax> redelmann: argh! my bad
<roaksoax> Razva: Hi there! that doesn't really provide us with much explanation. Can you please expand?
<redelmann> roaksoax, dont worry, btw i fix maas-proxy, after reinstall it start working.
<roaksoax> redelmann: cool!
<redelmann> roaksoax, the only change was switching public and private interfaces ( eth0 <--> eth1 )
<Razva> yeeeeeeey roaksoax is on!
<Razva> roaksoax ok so here it is. I'm making a fresh Ubuntu Cloud install on a server with two nics: eno1 (internet) and enp0s25 (lan)
<Razva> question 1: do I need to setup the LAN IP manually, before installing MAAS, or should I add it through the GUI after installing MAAS?
<roaksoax> Razva when you say the LAN IP, what do you mean?
<roaksoax> Razva: typically, you'd configure the server on the networks you want it to be, before you install MAAS
<Razva> 192.168.etc
<roaksoax> Razva: after yoiu install MAAS, you configure DNS/DHCP from MAAS' WebUI to provide services on the network you want (aka. on a Cluster Interface)
<roaksoax> redelmann: glad you made it work!
<redelmann> (Y)
<Razva> roaksoax can we chat on #ubuntu-server so jamespage is "on the loop"?
<jamespage> I'm here
<jamespage> ish
<Razva> great
<Razva> so, I'm going to install 15.10 and setup the "internet" nic, then login and setup the "lan" nic. all good 'til here?
<Razva> btw thanks for your effort guys, I really really appreciate this!
<roaksoax> Razva: right, so if you are using MAAS as the gateway, then you'd also need to setup NAT so the machines on the LAN can have internet
<roaksoax> Razva: that being said, you should configure DHCP/DNS on the Cluster Interface connected to the LAN
<Razva> got it. but before that I suppose I need to install Ubuntu Server "normally", by setting up the "net" nic, right?
<Razva> I did SEVEN installs today, so I want to verify each step, just in case :D
<roaksoax> Razva: right, so 1. install ubuntu server. 2. configure eth0 -> internet eth1 -> lan 3. install MAAS
<Razva> is there any wasy way to rename a nic? they have terrible names... :|
<roaksoax> Razva: you can manually rename the nic's if that so you wish
<roaksoax> Razva: that is for the MAAS server
<Razva> I was talking about the os nics, but whatever, I've learned the by heart :)
<Razva> while it installs, let me state what I've learned about NAT. basically this is the solution: sudo iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eno1 -j MASQUERADE
<Razva> where eno1 is the "net nic" and 1.0/24 is the LAN IP, which I'll set on the "lan nic"
<Razva> is that correct?
<Razva> roaksoax you've said "lan 3" that would be...the nat...?
<roaksoax> Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan" 3. "Install MAAS"
<roaksoax> Razva: Sorry, I menat 2. "configure eth0>internet, eth1>lan". 3. "Install MAAS"
<Razva> ah ok :))
<Razva> roaksoax looks good? http://pastebin.com/6V7Evui5
<roaksoax> Razva: i think it does.. I'd need to look up iptables since haven't done them in a while :)
<Razva> roger
<Razva> how can I verify that the nat is ok?
<haasn> I have a conceptual question with MAAS
<Razva> roaksoax: http://pastebin.com/LU9FX5GP < full network setup, full network status, full iptables status
<haasn> How do I map MAAS machines to physical locations, in case something is broken?
<haasn> There is a âlocatorâ functionality that can be used via IPMI to make the host's lights flash, but MAAS doesn't have a front-end for it (though I could write a wrapper script)
<haasn> How do other deployments solve this problem?
<haasn> e.g. say a drive fails on host whispered-news.maas. How do I find this host to replace the drive?
<Razva> haasn sorry I'm a newb, but here's my funny idea: stickers! :)
<haasn> Razva: So you mean, when MAAS generates its names, I attach a sticker of the generated name to the host?
<Razva> YES
<haasn> Then I will have an O(n) effort of finding the server given only the sticker name
<haasn> What I *could* also do is, before booting the servers for the first time, add their MAC addresses to the maas DHCP with pre-defined names like host0 - host20
<roaksoax> haasn: you can assign them to zones.. zones is just descriptive though
<haasn> That way maas would give them names I could directly resolve to locations
<Razva> OR you could write a script that'll play the Mario Soundtrack if/when something fails. THAT would be epic! :D
<haasn> roaksoax: So like, âzone 2-1â to mean âfirst host in the second rackâ? One zone per node?
<haasn> Or just one zone per rack and then use stickers for the rest?
<haasn> That latter approach could actually work really well
<Razva> haasn https://www.autoitscript.com/forum/topic/40848-beep-music-mario-bros-theme/ < there you have it!
<roaksoax> haasn: you can use zones whichever way you'd like
<roaksoax> haasn: you can have zone-floor1 zone-floor2
<roaksoax> for example
<Razva> roaksoax can you please take a look on the pastebin (http://pastebin.com/LU9FX5GP)? I swear I'll not touch anything until somebody gives me the green light! :)
<haasn> roaksoax: That's true, but I'm trying to minimize effort involved here - that's why I'm asking; to figure out which of the infinitely many solutions has the lowest effort
<haasn> I'm mostly interested in how _real_world_ mass setups handle this
<roaksoax> Razva: that seems to be ok
<Razva> roaksoax: woohooo, next, install MAAS, as stated at http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot (steps #2 and #3)?
<roaksoax> haasn: in real world people would use zones to place machines in different places or to group them by a comon identifier
<roaksoax> Razva: that's correct
<Razva> fingers crossed
<Razva> should I apt-get upgrade or leave it like it is?
<haasn> roaksoax: How do people map host names to physical server locations in the real world?
<Razva> haasn that was a question I addressed 3 weeks ago. after a long discussion the answer was "as you wish".
<haasn> Got any examples? I'm looking for inspiration
<Razva> I've mapped them like this: town-job.continent.domain.tld
<Razva> a controller in Maidenhead (europe) would be mh-controller1.eu.mydomain.com
<Razva> some just use IDs, because a machine can/will change it's function in time
<Razva> for example this controller can becone, in time, a compute or storage
<haasn> Razva: How do you get MAAS to name them mh-controller1 instead of surprised-change or w/e?
<haasn> Hard-code the MACs into its DHCPd?
<haasn> Maybe I should start by describing my problem
<Razva> haasn no idea, never worked with MAAS, this is my first try (I've worked with Proxmox, OpenVZ and OpenStack)
<haasn> I now have 10 identical machines named only âashamed-ducksâ, âcreative-winterâ, âfine-jarâ etc.
<haasn> Right now the _only_ way I can distinguish them realistically is by looking at the MAC address, because right now that follows a clear pattern (they are test VMs, and VM 10 has :10 at the end of its MAC)
<Razva> haasn are these your own machines or rented in some DC?
<haasn> Razva: My own. (They're actually VM instances, but I plan to move to physical machines)
<roaksoax> haasn: that depends per organization
<haasn> Razva: And, well, that's why I'm asking this question in #maas specifically - I want to know how real-world *MAAS* deployments solve their hostnames
<roaksoax> haasn: people in diferent organizations have different ways of doing that
<haasn> Not how you could theoretically lay them out given full control :)
<mup> Bug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>
<haasn> I could maintain my own list of MAC -> server slot locations in an internal documentation wiki
<haasn> (Or maintain this association as zones)
<Razva> haasn belive it or not, I would just stick them. really.
<haasn> Razva: Yeah, and put them into zones based on the rack
<haasn> That's what I'll most likely end up doing
<mup> Bug #1545035 changed: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>
<Razva> haasn 9 years ago (virtualization was kinda "non-invented") I had ~2000 servers. we've tried a lot of software solutions, even wrote our own, but in the end...stickers fixed it :)
<Razva> or you can try to use some fancy software, define your locations -> levels -> racks, and basically build a virtual representation of your DC
<Razva> but in the end it will be very time consuming so...we dropped it
<mup> Bug #1545035 opened: maas-cluserd cant bind to tftp port <MAAS:New> <https://launchpad.net/bugs/1545035>
<Razva> roaksoax: http://i.imgur.com/QNvGXyz.jpg < looks ok?
<Razva> roaksoax I swear to God that PXE doesn't works :|
<roaksoax> Razva: you sure machines are pxe booting ? do the logs say anything?
<Razva> roaksoax can you please ket me know what log files should I tailf?
<Razva> DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25
<Razva> DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25
<Razva> is this a server trying to boot?
<roaksoax> Razva: tail -f /var/log/maas/clusterd.log
<roaksoax> Razva: that should tell you whether a node is trying to PXE or not
<Razva> 2016-02-12 17:51:32+0200 [ClusterClient,client] Cluster '4fb34068-97f2-4cd9-a7dd-444475a71de7' registered (via mh-controller1:pid=1292).
<Razva> but the machine didn't boot...
<roaksoax> Razva: ps faux | grep dhcpd ?
<roaksoax> Razva: do you have console logs? Are you sure your machine is trying to get a dhcp address from MAAS ?
<Razva> dhcpd     1782  0.0  0.0  32908 13264 ?        Ss   17:54   0:00 dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid -cf /var/lib/maas/dhcpd.conf -lf /var/lib/maas/dhcp/dhcpd.leases enp0s25
<Razva> no other server online on the network, just MAAS and the "client"
<roaksoax> Razva: right, so that seems that dhcpd is providing on enp0s25
<roaksoax> Razva: and your machines don't see DHCP offers
<roaksoax> or your client
<roaksoax> Razva: do you have stp enabled ?
<Razva> no idea what is stp
<roaksoax> https://maas.ubuntu.com/docs/install.html#configure-switches-on-the-network
<roaksoax> Razva: ^^
<Razva> no idea, it's a manged/shared switch with a vlan
<Razva> Feb 12 18:41:27 mh-controller1 dhcpd[1782]: DHCPDISCOVER from 0c:c4:7a:0d:76:af via enp0s25
<Razva> Feb 12 18:41:28 mh-controller1 dhcpd[1782]: DHCPOFFER on 192.168.1.11 to 0c:c4:7a:0d:76:af via enp0s25
<Razva> doesn't this means that "somebody" is asking for offer?
<roaksoax> Razva: apparently
<Razva> roaksoax Portfast is enabled on the switch
<Razva> enp0s25   Link encap:Ethernet  HWaddr 0c:c4:7a:0d:76:af
<Razva> wait...so...the server is offering...dhcp to itself?
<mup> Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
<mup> Bug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
<mup> Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
<mup> Bug #1545119 changed: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
<mup> Bug #1545119 opened: Documentation missing for devices cli command <doc> <MAAS:New> <https://launchpad.net/bugs/1545119>
<haasn> Razva: But if you have 2000 servers, each with a sticker on them, and you just know you have to find host X; do you then go through 2000 hosts one by one until you find X?
<Razva> no, because each room has it's own "number", room is split into zones, and each zone is split into racks :)
<Razva> you can have like L1R3Z6R2
<Razva> Level 1 Room 3 Zone 6 Rack 2
<Razva> if you have random names generated by MAAS, just make an XLS where host15r38 = L1R3Z6R2 and there you have it :)
<haasn> XLS?
<Razva> any idea why the heck all the nodes are shutting down after booting from PXE?
#maas 2016-02-13
<dbainbri> I have been attempting to build a maas install that does not rely on access to the internet to function, thus i need to store the boot images locally. i did a wget -r from the maas.ubutnu.com releases to copy down the files structure and put that on my maas server's apache html area, so that i can access it from the web and have updated the boot-sources accordingly. unfortunately MAAS is not seeing these images. I did see a streams 
<stormmore> anyone used interface names like eth0:0 in the MAAS UI before?
<stormmore> or does anyone know how to clear the MAAS db back to install state?
<haasn> The Wake-on-lan doesn't specify the host interface to use so etherwake just defaults to eth0, which causes problems for me: It should have been eth1 (the identifier of the management network interface). Would it be possible to make the logic a bit smarter here, in that it could figure out automatically which subnet to send out the WOL packet on? It would definitely be doable
<haasn> (Or just send them on all, which is the easier and more robust solution :p)
<haasn> I guess I'll just hard-code the interface in the ether_wake.template
<roaksoax> haasn: wake on lan will be deprecated on 2.0
<haasn> roaksoax: Meh
<haasn> roaksoax: Will there at least be a ânoneâ option then?
<stormmore> Oh that is a shame to hear, wake on lan might be the only option for some people :-/
<stormmore> anyone know if there is a link to a public roadmap for MAAS?
<haasn> If the template system continues to exist then I suppose you could just continue to have the template in your config folder, mind
<Razva> any idea why all the servers that I connect to MAAS got shut down, and I have no option to start them...?
<haasn> Can I edit the scripts that MAAS runs (via cloud-init) to set up SSH keys etc.?
#maas 2016-02-14
<Razva> good mornin' folks. why in the world is MAAS running...two DHCP servers...?
<Razva> should I use 14.04 LTS for Ubuntu Cloud Images? or 15.10?
<haasn> Razva: ps aux | grep dhcp | curl -sF "file=@-" https://0x0.st
<Razva> mornin' haasn
<Razva> https://0x0.st/X0i.txt
<Razva> haasn at this point I've reinstalled Ubuntu
<Razva> just a sec to setup networking
<haasn> well that machine is clearly running 0 DHCP servers :p
<Razva> yup, right now yes
<Razva> can you please take a look at https://github.com/Razva/newbadmin/blob/master/firewall/nat.sh ?
<Razva> it's a script one of the good folks here provided for NAT
<Razva> I've slighly modified it
<Razva> looks ok?
<Razva> (for NAT)
<haasn> Razva: seems reasonable, not sure if the MASQUERADE line would benefit from an extra -i "$INTIF" or not but if it works it works
<Razva> any idea how to retain this after reboot...? :|
<haasn> Razva: iptables-save and iptables-restore can be used to save the current settings to a file (and do the opposite). Most distros provide some sort of init script for automatically doing that
<Razva> got it
<Razva> haasn: https://0x0.st/X0K.txt < that ps aux you've requested
<Razva> seems that only MAAS is running, right?
<haasn> seems to only be running one
<haasn> on interface enp0s25 (this is your private interface, I take it?)
<Razva> weird, last time it was i*something-dhcp-server running also... :|
<Razva> exactly, it's the lan
<Razva> now, let's try to PXE boot
<haasn> Razva: when stuff like that happens it's a good idea to look at the -f option, in this case /var/lib/maas/dhcpd.conf
<haasn> If that's /etc/dhcp/dhcpd.conf or something like that then it's probably some âdefaultâ system-wide dhcpd service running accidentally
<Razva> do you recommend 14 LTS or 15, for Ubuntu Cloud?
<haasn> I have no idea. I use 14 LTS because I haven't had a reason to change
<Razva> does Liberty works with 14?
<Razva> yup, it successfully booted, no idea why last time it basically started two DHCP servers...it was a fresh install...
<Razva> haasn except "Allow SSH access and prevent machine from powering off", should I check the other two boxes?
<Razva> Retain network & Retain storage?
<haasn> What those checkboxes mean is that if you're commissioning a node that you've already comissioned in the past, any storage/network configuration you made manually will be preserved instead of reset back to the default
<haasn> If this is your first time commissioning that node they do nothing
<Razva> aaand if I didn't ever commisioned them...I suppose that I should just "SSH and prevent machine from powering off", right?
<haasn> I don't see any reason to set that unless you're trying to debug something that went wrong
<Razva> yay, ok
<Razva> the only "wrong" thing until now is that MAAS is seeing "power on" even if the machine is 100% powered off :|
<Razva> any idea why>
<haasn> Razva: the power status is not checked all the time but polled at regular intervals (every few minutes), you can click âcheck nowâ to force an update
<Razva> haasn did it 3 times, they still point as being on
<Razva>  Checking power -> Power on
<haasn> Razva: What power type?
<Razva> even tried to "shutdown" them from MAAS
<Razva> IPMI. they are Dell servers
<haasn> Razva: try asking ipmipower -D LAN_2_0 -h HOST -u maas -p PASS --stat what it thinks of your host
<Razva> 	Power State	OFF
<Razva> haasn any idea what's the pass? :))
<haasn> Razva: you can check in the maas GUI
<haasn> It should have created a new account/password when enlisting
<haasn> If ipmipower works but MAAS does not, seems like a bug to me
<Razva> got it, just a sec to see
<Razva> HOST = IP or host.maas ?
<haasn> It should be the same hostname as in the maas power configuration
<Razva> paltry-cave < who in the world is generating these names? :)
<Razva> 95.154.222.13: connection timeout
<Razva> interesting.
<haasn> Razva: you could also try logging in via SSH
<Razva> I'm power cycling both machines, see what's the result
<Razva> long live iDRAC :)
<Razva> the machines are not even commisioned, so I cannot really SSH in...
<Razva> oook both are not booting via PXE
<Razva> ssh key bla bla...
<Razva> both are waiting here: http://screencast.com/t/LvFwpbHIaBnK
<Razva> ^ is that "normal"?
<Razva> aaaand they seem to be stuck at that point :|
<haasn> Razva: SSH to the service processor (IPMI interface), not the host OS
<Razva> SSH seems to be working, but I cannot login
<haasn> Razva: with the maas username/password?
<haasn> what about root?
<Razva> haasn yup, with the maas u/p, I cannot login
<Razva> I'll try to login with root now
<Razva> 95.154.222.13: connection timeout < from ipmipower
<Razva> root@95.154.222.13's password: < ssh works
<Razva> /admin1-> < root worked
<Razva> any hints now? :D
<haasn> Razva: set the root un/pw in maas and be done with it? :p
<haasn> oh, connection timeout from ipmipower with root p/w too?
<haasn> Razva: try ipmitool -I lanplus -U root -P ... -H ... user list
<haasn> If ssh works but ipmipower does not, then you might need to specify a firmware-specific workaround flag (via -W)
<haasn> e.g. I need -W sun20 for some of my old sun servers
<Razva> haasn exactly
<Razva> haasn but I cannot even reset IPMI
<Razva> ipmitool -H 95.154.222.13 mc reset warm
<Razva> Password:
<Razva> Error: Unable to establish LAN session
<haasn> did you try the ipmitool command I suggested?
<Razva> nope, just a sec
<Razva> ipmitool -I lanplus -U root -H -H 95.154.222.13 user list
<Razva> Password:
<Razva> Address lookup for -H failed
<Razva> Could not open socket
<Razva> (same with minus the extra -H)
<Razva> ipmitool -I lanplus -U root -H 95.154.222.13 user list
<Razva> Password:
<Razva> Error: Unable to establish IPMI v2 / RMCP+ session
<Razva> but I can SSH login...
<Razva> sh root@95.154.222.13
<Razva> root@95.154.222.13's password:
<Razva> /admin1->
<haasn> Razva: maybe something here will help? https://www.gnu.org/software/freeipmi/freeipmi-bugs-issues-and-workarounds.txt
<haasn> also see man ipmipower (scroll down to workarounds)
<Razva> I've contacted the data-center...maybe they can do a power-cycle/reset...
<haasn> doubt it will help
<Razva> its their server, so I should be able to list users...at least.
<Razva> hey guys. I have no idea why but maas refuses to interpret the right status from IPMI...
<Razva> the server is powered down, I can see that via DRAC and console..
<Razva> but still maas reports the server as being online. any hints...?
<redelmann> hi, im having some trubles with unmanaged networks
<redelmann> im trying to let my local dhcp and dns to manage this network
<redelmann> but for some reason /etc/resolv.conf is setting maas ip address as DNS
<redelmann> i double check my local dhcp setting and everything seems fine
<redelmann> so, why maas is putting herself as DNS of an unmanaged network?
<Razva> redelmann no idea, but that's what maas does (at least on my setup)
<redelmann> Razva, ok, thank. It little strange, i was supposing that unmanaged network is UNmanaged :P
<redelmann> Razva, i think juju bootstrap fail in my setup because of this
<Razva> redelmann I have a very weird issue. MAAS reports power on, even if power is off (I can see in iDRAC that power is off, and console shows No power)
<redelmann> Razva, it's that for me?
<Razva> redelmann yes
<Razva> maybe you have any ideas...
<redelmann> Razva, noup.
<redelmann> Razva, im playing with maas for 2 weeks to now
<redelmann> Razva, so im not the most indicated person
<Razva> haha, I'm playing for ehm...2 days :D
<Razva> I've successfully setup 3 machines, everything PXE boots, but this single darn machine doesn't wants to boot....
<Razva> or whatever...boots but doesn't wants to report power as it should
<redelmann> Razva, maybe hardware problem?
<Razva> no idea...
<Razva> do you use IPMI or...?
<redelmann> IPMI and virsh
<redelmann> i'am mixing virtual nodes with intel hardware
<Razva> I have only intel / Dell hardware, but no idea why the heck I cannot control it...
<redelmann> im just testing maas for a full implementation
<Razva> well...I didn't had any issues with it until now, only problem was that - at first install - Ubuntu installed a separate DHCP server as well. no idea why, but a clean install solved everything
<Razva> Power on < I'm gonna kill somebody.
<redelmann> Razva, you are installing MAAS from apt?
<Razva> the server is off god damnit...
<redelmann> Razva, try with "check now" link :P
<redelmann> Razva, or manually power on the server
<Razva> yeah if I manually power on it works, but this is not the way to do it...
<redelmann> Razva, after manually power on, swith to off with maas
<Razva> how...?
<Razva> I cannot manually set power off in maas
<redelmann> Razva, i know but maybe yoy can "sync" what maas think about node and rearl node state
<Razva> please do me a favor
<redelmann> Razva, take action -> power off?
<Razva> ipmipower -D LAN_2_0 -h SOME-IPMI-IP -u root -p THE-ROOT-PASSWORD --stat
<Razva> does it works for you?
<redelmann> Razva, im in home rigth know.
<redelmann> Razva, i dont know why cant ssh to cluster
<redelmann> Razva, give me a second
<redelmann> Razva, but usually thats the command i use to check the whole cluster power status without problem
<redelmann> Razva, i never use -D, only -h ip
<redelmann> Razva, did you try with --debug?
<Razva> well...what command do you use, usually?
<Razva> (ipmipower_powercmd.c, _retry_packets, 1214): host = 95.154.222.13; p = 1; Sending retry, retry count=24
<Razva> redelmann are you deploying OpenStack or just MaaS?
<redelmann> Razva, just MAAS
<redelmann> Razva, ipmipower -h ip -u root -p THE-ROOT-PASSWORD --stat
<redelmann> Razva, i let ipmipower to decide ipmi version
<Razva> 95.154.222.13: connection timeout
<Razva> but the interesting thing is that I can ssh to it
<Razva> /admin1->
<redelmann> Razva, ipmipower --version?
<redelmann> Razva, some old ipmipower utilitis had a bug with -p options i think
<Razva> redelmann via SSH?
<redelmann> Razva, no, just ipmipower --version in machine you are trying to connect
<redelmann> Razva, maas server i suppose
<Razva> ipmipower - 1.1.5
<Razva> you?
<redelmann> Razva, look at this: https://lists.gnu.org/archive/html/freeipmi-users/2005-11/msg00018.html
<redelmann> Razva, its old
<redelmann> Razva, forget about it, ipmipower 0.1.3 its the problem
<redelmann> Razva, ipmipower - 1.1.5
<roaksoax> Razva: do you have logs from /var/log/maas/ , it is probably telling why it cannot connect  to IPMI
<roaksoax> if that's the case
<roaksoax> redelmann: maas doens't really support external DHCP/DNS anymore. But eventhen, MAAS is the defautl DNS server unless you set it differently for each subnet
<redelmann> Razva, did you try ipmiping <IP>
<redelmann> roaksoax, http://paste.ubuntu.com/15060537/
<redelmann> roaksoax, i set DNS on my maas-eth0
<roaksoax> redelmann: what version of MAAs are you using?
<redelmann> roaksoax, it's that no enough?
<roaksoax> 1.9
<redelmann> roaksoax, 1.9
<redelmann> roaksoax, it's testing purpose
<roaksoax> redelmann: right, so when you deploy, is the interface of the machine configured to do what? DHCP ?
<redelmann> roaksoax, exactly
<roaksoax> redelmann: right, so DHCP should be setting the DNS server right?
<redelmann> roaksoax, rigth
<roaksoax> redelmann: also, can you get the curtin config ?
<roaksoax> redelmann:  maas <session> node get-curtin-config <system-id>
<redelmann> roaksoax, ohh THANK, i was losing my head
<redelmann> roaksoax, http://paste.ubuntu.com/15060721/
<redelmann> roaksoax, /etc/resolv.conf contain: 192.168.100.1 (my lan DNS) and 192.168.100.10 (my mass server lan ip)
<redelmann> roaksoax, i was expecting to have only 192.168.100.1
<roaksoax> redelmann: Yeah I see. is the NIC on your system connected to 'maas-eth0' subnet ?
<roaksoax> redelmann: i think the problem here is that since MAAS is not configure eth0, it is ont telling /etc/network/interfaces to set the DNS of maas-eth0
<roaksoax> redelmann: eitherway DHC{P should be setting it
<roaksoax> and it doesn't seem like it is ?
<redelmann> roaksoax, maas server and node have a NIC on 'maas-eth0'
<roaksoax> redelmann: right, but MAAS is not managing DHCP/DNS< and my *guess* is that because of that, DNS server of maas-eth0 is not being set
<roaksoax> redelmann: on the config
<redelmann> roaksoax, that's rigth
<redelmann> right
<roaksoax> redelmann: so, yif you could file a bug, I'll discuss it with my guys and see if we can fix that
<roaksoax> cause I think we porbably should
<roaksoax> redelmann: please attach both, the maas-eth0 subnet and the curtin config
<roaksoax> to the bug report
<redelmann> roaksoax, great, between my poor English and my mass-newby it's hard to explain. i will do my best
<redelmann> roaksoax, ok
<roaksoax> redelmann: don't worry, just file it, and I'll fix the wording if need by
<roaksoax> be*
<redelmann> roaksoax, https://bugs.launchpad.net/maas/+bug/1545500
<redelmann> roaksoax, hope you can understand the whole idea/problem :P
<redelmann> roaksoax, thank for you
<redelmann> roaksoax, thank for your time
<mup> Bug #1545500 opened: MAAS unmanaged network set wrong DNS information <MAAS:New> <https://launchpad.net/bugs/1545500>
<haasn> Hopefully somebody reads this: If you're thinking of managing maas hosts with juju, use ansible instead! My life is so much happier and it's easier to set up, maintain and interact with :)
<stormmore> somehow I donât see juju bootstrap being that difficult a command to use
#maas 2017-02-06
<mup> Bug #1662073 opened: 'regiond' tries to get 'bootloaders' that are not selected <MAAS:New> <https://launchpad.net/bugs/1662073>
<mup> Bug #1662111 opened: when deploying 'virsh' type of machine(KVM), it repeats deploying constantly <MAAS:New> <https://launchpad.net/bugs/1662111>
<mup> Bug #1661880 changed: specify static ip address when taking ownership of device <maas-at-home> <MAAS:New> <https://launchpad.net/bugs/1661880>
<mup> Bug #1662185 opened: device discovery runs correctly, at first, but not subsequently <maas-at-home> <MAAS:New> <https://launchpad.net/bugs/1662185>
<mup> Bug #1662274 opened: When maas gui tells you to install some packages, it should offer to do so for you <maas-at-home> <MAAS:New> <https://launchpad.net/bugs/1662274>
<mup> Bug #1662274 changed: When maas gui tells you to install some packages, it should offer to do so for you <maas-at-home> <MAAS:New> <https://launchpad.net/bugs/1662274>
<mup> Bug #1662274 opened: When maas gui tells you to install some packages, it should offer to do so for you <maas-at-home> <MAAS:New> <https://launchpad.net/bugs/1662274>
<mmxe> Hi
<mmxe> Does anyone have any idea what's going on with the error in the log I posted over the weekend?
<mmxe> The error occurs during commissioning process. My commisioning fails and the only error I see is that ssh keys weren't installed for user ubuntu
<mmxe> I've added ssh keys to my MAAS user account.
<pmatulis> mmxe, best file a bug. include all logs. not just a snippet
<roaksoax> mmxe: it seems the amchine is trying to enlist, not commission
<roaksoax> mmxe: the issue doens't seem to be the ssh keys
<roaksoax> mmxe: but cloud-init dying for some reason
<roaksoax> mmxe: what version of MAAS are you using ?
<pmatulis> he said 2.1.3 earlier
<roaksoax> ah!
<roaksoax> mmxe: make sure you have the latest images as well
<mmxe> I have the images that MAAS downloaded automatically.
<mmxe> On a side note, it would be nice to see the MAC addresses associated with DHCP leases in the lease list so it's easier to fill out power control MACs and IP.
<mmxe> It's sort of a convoluted process to pair a MAC with a machine name.
<roaksoax> mmxe: you dont need MAC's for BMC's
<mmxe> I select IPMI as my "Power type" and the last field is "Power MAC"
<mmxe> It's not necessary to populate it?
<roaksoax> mmxe: no it is not
<mup> Bug #1657491 opened: Several IPs assigned to the same iface in the DB due to 'free' leases <sts> <MAAS:In Progress by mpontillo> <https://launchpad.net/bugs/1657491>
<mup> Bug #1662343 opened: [2.1.3] Commissioning doesn't pick up new storage devices <oil> <MAAS:New> <https://launchpad.net/bugs/1662343>
<mup> Bug #1662343 changed: [2.1.3] Commissioning doesn't pick up new storage devices <oil> <MAAS:New> <https://launchpad.net/bugs/1662343>
<mup> Bug #1662343 opened: [2.1.3] Commissioning doesn't pick up new storage devices <oil> <MAAS:New> <https://launchpad.net/bugs/1662343>
<catbus1> roaksoax: Hi, I tried to add maas cloud to conjure-up, but it errors out with trackback: unable to find: <home folder>/.local/share/juju/accounts.yaml. Is there something I need to do prior to setting this up in conjure-up?
<catbus1> or 'cattle' not found in juju's bootstrap-config.yaml
<mup> Bug #1662361 opened: `machine deploy` parameter expectations and error output are not consistent <uosci> <MAAS:New> <https://launchpad.net/bugs/1662361>
#maas 2017-02-07
<mup> Bug #1662404 opened: "No rack controllers can access the BMC of node: <node name>" when using OpenStack nova power driver  <MAAS:New> <https://launchpad.net/bugs/1662404>
<mup> Bug #1647863 changed: DHCP snippet at node level is not added to dhcpd.conf <MAAS:Expired> <https://launchpad.net/bugs/1647863>
<brandor5> Hello everyone: I'm starting to play with maas/juju and I'm having an issue accessing systems that get deployed... when I use maas to deploy a system i'm not able to ssh to it... I get "Permission denied (public key)"... I've double checked that my maas account has my pub key and my ssh-agent has the private key loaded... where would i start to debug this?
<brandor5> anyone?
<roaksoax> brandor5: does the machine show 'deployed' ?
<roaksoax> brandor5: in MAAS ?
<roaksoax> brandor5: has juju not failed or shows the machine as pending ?
<brandor5> roaksoax: yep, everything looks good
<brandor5> roaksox: i'm having hte problem even without juju... just using maas to deploy a machine
<roaksoax> brandor5: so ssh ubuntu@ip would reject your connection ?
<brandor5> exactly
<brandor5> i'm thinking I need to add the backdoor account to an image and then retry to see what's happening
<roaksoax> brandor5: check rsyslog first - /var/log/maas/rsyslog/<machine-name>/<date>/messages -> and pastebin that
<brandor5> ok
<brandor5> @roaksoax: just an update... I wanted to clear out all the keys/config in maas and now things are working
<brandor5> appreciate the help :)
<brandor5> and i just found out the real issue, a coworker had provided the wrong ssh key to maas for systems
<roaksoax> brandor5: ah!
<roaksoax> glad it is working for you now :)
<mup> Bug #1602468 changed: [1.9] loading initial data for maasserver: maas-region-admin reports duplicate key error during package install <cdo-qa> <cdo-qa-blocker> <MAAS:Invalid> <MAAS 1.9:Invalid> <https://launchpad.net/bugs/1602468>
<Guest32187> Hi everyone!  I know when a server restarts, by default it will be recommissioned/provisioned by MAAS.  Is there a way to only have MAAS provision hosts a single time, then have them boot locally after that?
<yinonby> Hello, I'm trying for the first time to deploy basic openstack using juju and maas
<yinonby> I'm following this guide: https://jujucharms.com/openstack-base/
<yinonby> Some things are not clear to me
<yinonby> 1. should I use maas to deploy the bare metals with an operating system (ubuntu) prior to starting this guide, or do juju charms already install the operating system using maas?
#maas 2017-02-08
<KenErwin> Does anyone know if there's a way to have MAAS provision baremetal servers, then have them boot from local disk instead of PXE?
<KenErwin> We have a requirement that some of our servers can be restarted without being re-provisioned :(
<pmatulis> KenErwin, all machines should remain in PXE mode
<KenErwin> For us though, we have about 800 MS SQL servers that we would be provisioning.  Once they've been provisioned, they'll need to keep their state
<KenErwin> :(
<KenErwin> so for restarts etc. it can't re-commission them.  Also the company wants to make sure that even if MAAS goes down, they could restart safely
<KenErwin> is that impossible with MAAS?
<blahdeblah> KenErwin: If a machine is provisioned, when it boots from PXE, MAAS will detect that and tell it to boot from local disk
<blahdeblah> And if MAAS is down, it should fall back to local disk after PXE fails
<blahdeblah> i.e. The scenario you've described should be fine with leaving servers on PXE, assuming people don't go in and change their deployed status in the console.
<blahdeblah> KenErwin: You could protect against that by having a local script change the boot order after deployment is complete, if you want to be sure, though.
<KenErwin> that's great news!  Thank you very much!
<blahdeblah> KenErwin: In my test env, I just leave the boxes in PXE mode all the time, but I can understand why you might want to change that for production database servers. :-)
<KenErwin> :D ty again.  Yep, the DBA team might kill me
<mup> Bug #1661869 changed: maas install fails inside of a 16.04 lxd container due to avahi problems <maas-at-home> <MAAS:Invalid> <avahi (Ubuntu):Confirmed for lathiat> <lxd (Ubuntu):Invalid> <https://launchpad.net/bugs/1661869>
<David_Orange> Hello MAAS experts, i am trying to work with maas 2.1.3+bzr5573 APIs and I have an issue with Authentication.
<David_Orange> I am using a https://maas.ubuntu.com/docs2.0/api_authentication.html slightly modified version(http://paste.ubuntu.com/23953461/)
<David_Orange> but i got an error: 'Unrecognised signature: method=GET op=list'. Any help would be welcome, Thanks
<brendand> David_Orange, i wrote a blog post about using the API, the code i use is a little simpler - https://maastery.wordpress.com/2016/10/17/maas-rest-python-requests/
<brendand> David_Orange, feedback would be appreciated, it might get upgraded to official documentation if people like it
<David_Orange> brendand: thanks. I check that and give you my feedback
<brendand> David_Orange, as for your error, i think that endpoint may be out of date unfortunately. our documentation needs a bit more tidying
<David_Orange> found a few errors in request.get call, it would suggest:
<David_Orange> brendand:
<David_Orange> url = 'http://mymaas.com:5240/MAAS/api/2.0/machines/'
<David_Orange> response = requests.get(url, headers=get_auth_headers(url, apikey))
<David_Orange> missing url parameter unget_auth_headers and a finish ')'
<David_Orange> brendand: got the same error " signature: method=GET op=list'
<David_Orange> brendand: you suggest that it is a problem of sync between client and server ?
<brendand> David_Orange, can you paste the whole code you're running, obviously without your api key (you can replace that with <apikey>)
<brendand> http://paste.ubuntu.com/
<David_Orange> brendand: http://paste.ubuntu.com/23953623/
<brendand> David_Orange, delete line 13 first of all
<brendand> but it probably doesn't really affect anything
<brendand> the main problem is that /nodes/?op=list is not available anymore
<brendand> if you want to get a list of machines, just use '/machines/' as the uri
<David_Orange> great :)
<David_Orange> brendand: you found it
<David_Orange> brendand: it was due to the old uri
<David_Orange> brendand: if i can +1 you code to the doc, let me know :)
<David_Orange> brendand: do you know if there is any swagger description of the maas api ?
<brendand> David_Orange, no, there isn't but the api description is there in the docs
<David_Orange> brendand: ok, thanks for your help
 * brendand files a bug to get that fixed
<mup> Bug #1662852 opened: Machines configured with non-MAAS IPMI credentials do not commission <MAAS:New> <https://launchpad.net/bugs/1662852>
<mup> Bug #1662854 opened: API authentication docs page refers to old API endpoint <MAAS:New> <https://launchpad.net/bugs/1662854>
<mup> Bug #1662946 opened: MTU is not set correctly on VLAN if raw device is unconfigured <MAAS:New> <https://launchpad.net/bugs/1662946>
<mup> Bug #1662948 opened: MTU is not set correctly on VLAN if raw device is unconfigured <MAAS:New> <https://launchpad.net/bugs/1662948>
<Fl1nt_> Hi guys!
<Fl1nt_> Ok, quick question, is there any canonical guys over there?
<pmatulis> Fl1nt_, this is a technical channel. do you have a technical question? if so, just ask it
<Fl1nt_> pmatulis: sort of, not really a technical one, but a pretty important one for our current project
<Fl1nt_> On another topic, regarding a technical question, is there a plan to provide maas as snaps?
<roaksoax> Fl1nt_: there is, we are working on them. We are pretty close
<Fl1nt_> roaksoax: Ok, very good news :D lovin it
<Fl1nt_> I've another request but this one need to be talked privately as it's on the name of my company and can't yet be publicly exposed, that why I was looking for a canonical dev/representative.
<pmatulis> Fl1nt_, perhaps i can find someone in the canonical support organization?
<Fl1nt_> pmatulis: That would be cool, many thanks
<Fl1nt_> Thanks a lot to both of you pmatulis and roaksoax, I've got to go, see you soon!
<mup> Bug #1663008 opened: [2.2] There's no way to select what interfaces MAAS binds to <MAAS:Triaged> <https://launchpad.net/bugs/1663008>
<mup> Bug #1663008 changed: [2.2] There's no way to select what interfaces MAAS binds to <MAAS:Triaged> <https://launchpad.net/bugs/1663008>
<mup> Bug #1663008 opened: [2.2] There's no way to select what interfaces MAAS binds to <MAAS:Triaged> <https://launchpad.net/bugs/1663008>
<iwi> hi there, could anyone share some info where does maas serve tftp from? I need to server a file over tftp but i'm struggling where should i put it :/
#maas 2017-02-09
<errr> what is the difference between a node and a machine?
<errr> this project seems very incomplete.. is there a better library to use for the maas api? https://github.com/maas/python-libmaas
<BlackDex> errr: the difference is that a machine is a physical machine bare-metal (or maybe even a virtual machine), and a node could be an container like lxd for instance
<brendand> BlackDex, errr - 'Nodes' also encompasses controllers and non-managable 'devices'
<mmxe> I'm trying to log on to a node to investigate why commissioning fails.
<mmxe> I followed the instructions in Troubleshooting to add a password to "ubuntu" user account
<mmxe> https://docs.ubuntu.com/maas/2.1/en/troubleshoot-faq
<mmxe> The script provided references a path that doesn't exit, and I'm not sure how to edit it
<mmxe> This line here: imgs=$(echo /var/lib/maas/boot-resources/*/*/*/*/*/*/root-image)
<mmxe> I checked a random path in /var/lib/maas/boot-resources/*/*/*/*/*/*/
<mmxe> there's no file called "root-image"
<mmxe> There's only boot-inited, boot-kernel, and squashfs
<mmxe> Can anyone tell me how to lot on to the ephemeral image of the node and access the /var/log/cloud-init.log,  /var/log/boot.log, and /var/log/cloud-init-output.log files
<mmxe> MAAS is reporting "Failed commissioning" on every node I try to bring up.
<mmxe> I'm watching VGA output, and it seems to be functioning mostly correctly. Nodes can access network, boot by PXE, MAAS applies updates after the initial install
<mup> Bug #1662073 changed: 'regiond' tries to get 'bootloaders' that are not selected <MAAS:Invalid> <https://launchpad.net/bugs/1662073>
<mup> Bug #1663276 opened: test_stop_rescue_mode_logs_and_raises_errors breaks when previous_status is DEPLOYED <tests> <MAAS:Triaged> <https://launchpad.net/bugs/1663276>
<mup> Bug #1663290 opened: boot-source-selections endpoint does not document "os" parameter <MAAS:New> <https://launchpad.net/bugs/1663290>
<ms2i> I asked for some help as mmxe a couple hours ago. I got disconnected and can't see if there's any reponse.
<ms2i> Thanks
<errr> BlackDex: thanks
<errr> ms2i: there was no response
<ms2i> Ok, thanks
<ms2i> Does anyone know how to get on the mailing list? All the links I can find go to launchpad, and the last message there says the list moved, but it's impossible to see where.
<pmatulis> ms2i, so you cannot commission any node?
<ms2i> Right, I can't commission any node. I have 2 HP DL360 G7
<ms2i> Also Quanta Computers Windmill platform
<mup> Bug #1663324 opened: Maas Region Controller fails to install using external postgres via debconf <maas (Ubuntu):New> <https://launchpad.net/bugs/1663324>
<brendand> ms2i, which list? maas-devel?
<brendand> ms2i, https://lists.ubuntu.com/mailman/listinfo/maas-devel
<brendand> roaksoax, update topic ^
<brendand> i can't
<roaksoax> ms2i: why can't you commission any node ?
<roaksoax> ms2i: what are the symbols ?
<ms2i> I'm not sure what you mean by "symbols". The ephemeral image installs, then updates with apt, then shuts down and MAAS reports "failed commissioning" I'm trying to figure out how to log on to the ephemeral image so I can look at cloud-init logs
<pmatulis> ms2i, and this is true for all nodes? you haven't been able to commission a single node?
<pmatulis> ms2i, what is the underlying machine type for a failing node?
<roaksoax> menn0: do you have a log
<roaksoax> err
<roaksoax> sroy
<menn0> roaksoax: ?
<mup> Bug #1663387 opened: MAAS should print its version string to its logs on startup <support> <MAAS:Triaged> <https://launchpad.net/bugs/1663387>
<mup> Bug #1554636 changed: maas serving old image to nodes <landscape> <MAAS:Fix Released by ltrager> <MAAS 1.9:Triaged> <MAAS 2.0:Fix Committed> <https://launchpad.net/bugs/1554636>
<Damnit__> I'm trying to install MAAS and i am stuck
<Damnit__> is MASS free?
<pmatulis> Damnit__, where are you stuck? what instructions are you following?
<Damnit__> i am lost when my terminal asks for maas-rack install-shared-secret
<Damnit__> I see the file
<Damnit__> pmatulis:
<Damnit__> what terminal command do i enter to get this information?
#maas 2017-02-10
<pmatulis> Damnit__, what instructions are you following? in general i mean
<mup> Bug #1663517 opened: test_calls_decompose_machine leaving junk in reactor <tests> <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1663517>
<zeestrat> Hey, how can I change the hostname of a region/rack controller via the GUI or CLI?
<brendand> zeestrat, there might be a ui bug in changing the hostname..
<zeestrat> brendand: I'm not sure if I understand. Could you elaborate?
<brendand> zeestrat, the list of controllers is at MAAS/#/nodes?tab=controllers
<brendand> if you click on one, then click on the hostname field, it doesn't let you edit it. i'm not sure if that is intentional
<brendand> roaksoax, should we be able to update the controller hostname?
<zeestrat> Yes, I noticed that. So I was hoping there was a way to do so with the CLI.
<brendand> zeestrat, i can't actually see a way at the moment, the region-controllers update command has no hostname parameter that i can see
<zeestrat> brendand: I couldn't find anythere there either. Shal I wait for a response from roaksoax before creating a bug?
<brendand> zeestrat, you could file a bug along the lines of 'controller hostname field not editable'. if it's not *meant* to be editable it should not be a text field, i think
<mup> Bug #1663569 opened: [2.2] Prefer using <fabric-name>-<vlan-name> for shared network statement in dhcpd.conf <MAAS:New> <https://launchpad.net/bugs/1663569>
<mup> Bug #1663569 changed: [2.2] Prefer using <fabric-name>-<vlan-name> for shared network statement in dhcpd.conf <MAAS:New> <https://launchpad.net/bugs/1663569>
<mup> Bug #1663569 opened: [2.2] Prefer using <fabric-name>-<vlan-name> for shared network statement in dhcpd.conf <MAAS:New> <https://launchpad.net/bugs/1663569>
<roaksoax> admcleod: no you cant
<roaksoax> err
<roaksoax> admcleod: sorry
<roaksoax> zeestrat: no you cant
<roaksoax> zeestrat: i responded on the bug
<roaksoax> zeestrat: but basically, MAAS does not have a running agent on the controllers to change the hostname of the underlying system
<zeestrat> roaksoax: Changing the hostname of the system is not a problem, however changing that (i.e. /etc/hostname) does not seem to update it in MAAS.
<zeestrat> Basically we had a change of naming policy and would like this change to also reflect in MAAS on our controllers running in production.
<roaksoax> zeestrat: does hostname -f reflect your new hostname ?
<roaksoax> zeestrat: so you want ot change the underlying hostname but it doesn't reflect in MAAS
<roaksoax> zeestrat: i'd recommend you re-open the bug and rename it to "changing controller's underlying machine hostname does not reflect in MAAS"
<roaksoax> zeestrat: actually
<roaksoax> zeestrat: i just tested
<roaksoax> zeestrat: and it changed the hostname for me
<roaksoax> zeestrat: i did: sudo hostname xyz
<roaksoax> zeestrat: updated it in /etc/hosts
<roaksoax> zeestrat: sudo service maas-rackd restart
<roaksoax> zeestrat: and boom, new hostname
<zeestrat> Oh, then I must have done something wrong. Let me try again.
<roaksoax> zeestrat: try restarting maas-rackd
<zeestrat> Will do. Get back to you in a few min.
<roaksoax> tested again
<brendand> roaksoax, also i think the controller hostname should be text and not an uneditable text-field. it's a bit confusing atm
<roaksoax> brendand: maybe it is the browser, it doesn't show as an uneditable text field to me
<brendand> roaksoax, when you click on it what happens?
<roaksoax> nothing
<brendand> roaksoax, looking at the angular code it does appear to be a non-editable text field
<roaksoax> brendand: maybe because I'm now looking at latest trunk :)
<zeestrat> roaksoax: You were absolutely right. Dumb mistake on my part. Forgot about an old node with the same name so of course MAAS complained because of the name collision.
<roaksoax> zeestrat: no worries, it is not something obvious anyway :)
<zeestrat> Thanks for bearing with me :)
<roaksoax> zeestrat: thanks for using maas :)
<zeestrat> roaksoax: With the hours we have saved we owe y'all a beer or two! Just give a ping if you come by Norway by any chance.
<zeestrat> roaksoax: While I have you, do you know if there's a doc that describes which connections/ports are used between region and rack controllers? I saw some discussions in https://bugs.launchpad.net/maas/+bug/1352923 but nothing up to date.
<pmatulis> zeestrat, afaik tcp ports 5240 and 5250-5259
<roaksoax> zeestrat: 5250-5259 is for internal communication between region/rack
<roaksoax> zeestrat: for user access, 5240
<zeestrat> How about if you have region controller in HA with a VIP (keepalived as per your guide)? Do the rack controllers only talk to the VIP set in the rackd.conf, or also directly to one of the region controllers?
<mup> Bug #1663612 opened: [2.2, UI] During commissioning, UI shows 'Error:Node must be connected to a network.' <MAAS:New> <https://launchpad.net/bugs/1663612>
<mup> Bug #1663614 opened: MAAS creates a websocket connection per tab <performance> <MAAS:Triaged> <https://launchpad.net/bugs/1663614>
<MaWe_> Hello there, I've a strange error while commisioning: "update clock skew to -7324" and "http error 401"
<MaWe_> after several unique messages (within minutes) the systen shuts down.
<MaWe_> MaaS node list shows "Failed commisioning"
<MaWe_> 3 nodes, 3 errors :-(
<MaWe_> any suggestions?
<roaksoax> MaWe_: what version of MAAS are you using ?
<roaksoax> MaWe_: seems like the clocks on your systems are not in sync
<roaksoax> MaWe_: but cloud-init and ntp should handle that
<MaWe_> Version: 2.1.3+bzr5573-0ubuntu1~16.04.1
<MaWe_> maas-server was out of sync, did a 'service ntp restart' and it worked.
<MaWe_> how can i test if cloud-init and ntp are ok?
<roaksoax> MaWe_: go to the MAAS UI, Under the nodes tab, go to Controllers
<MaWe_> nameservers are configured and should work
<roaksoax> access your controller and see the status of the services
<roaksoax> make sure that ntp shows gree
<mup> Bug #1663643 opened: Pod compose handles timeouts badly <pods> <MAAS:Triaged by blake-rouse> <https://launchpad.net/bugs/1663643>
<MaWe_> ntp is green
<roaksoax> MaWe_: and the issue keeps happening ? cloud-init should just fix the clock-skew
<MaWe_> it keeps happening :-(
<MaWe_> sorry.. have to leave now, will reboot the machine and try it on monday. -- thanks for your help
<mup> Bug #1663651 opened: [2.2, rev5710] MAAS has 'test' action, but it has nothing in it. <MAAS:New> <https://launchpad.net/bugs/1663651>
<mup> Bug #1663652 opened: Maas with HA and remote rsyslog not work <MAAS:New> <https://launchpad.net/bugs/1663652>
<bladernr> roaksoax, am I correct in thinking you can apply kernel parameters now based on tags? (e.g. tag=noacpi could apply the acpi=off parameter to the kernel)?
<bladernr> roaksoax, and if so, does that only apply to the ephemeral, or is it also set when the deployed kernel is installed?
<roaksoax> bladernr: you were always able to apply parameters based on tags
<roaksoax> bladernr: it will apply them to deployed machines too
<bladernr> ahhh, ok.  What I meant, I guess was that I just learned that it was possible (so new to me).  Thanks for confirming!!
<mup> Bug #1663686 opened: [2.2] After machine is set 'Ready' (after commissioning), messages fail to post <MAAS:Triaged> <https://launchpad.net/bugs/1663686>
#maas 2017-02-12
<iwassa> hey there
#maas 2018-02-05
<Phill93> Hi
<Phill93> is it possible to clean the tests for a machine?
<anderbw> Hi there, I have a dhcp related issue, anyone willing to assist me in finding out the issue? tl;dr: dhcpd doesn't send any dhcpoffer anymore. and all clis that earlier got an addr keeps sending a dhcpreq too. Main confusion stems from it working initially, then suddenly not.
<anderbw> nvm, confused about the terminology used in maas for a sec. all good.
<roaksoax> anderbw: :)
<mup> Bug #1747458 opened: No user visible error/warning when dhcpd crashes and is restarted <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1747458>
<jac_cplane> hi - is there a way to make maas an authoritative dns for a subnet after i've already deployed when it prev was not authoritative
<mup> Bug #1747459 opened: MAAS never notices that ntp crashes <cdo-qa> <maas-ha-testing> <MAAS:Incomplete> <https://launchpad.net/bugs/1747459>
<roaksoax> jac_cplane: dns in maas is not really bound to a subnet
<roaksoax> you can make a domain authoritative though
<roaksoax> right now, via the API only thoguh
<jac_cplane> @roaksoax
<jac_cplane> thanks.  can you point me to the api
<jac_cplane> i think we figured it  out - this seems to work  :  maas maas domain update 2 authoritative=True
<roaksoax> jac_cplane: should be something like  maas admin domain update <domain_id> authoritative=True
<roaksoax> indeed
<jac_cplane> yes - thanks!
<bradm> any one know how to debug why a post to /MAAS/metadata is getting a 500 error?  happening while doing a commission on some nodes, specifically during the testing of disks
#maas 2018-02-06
<mup> Bug #1747548 opened: [2.3] 500 error from maas metadata while running hardware tests <MAAS:Incomplete> <https://launchpad.net/bugs/1747548>
<mup> Bug #1728308 changed: [2.3] Comissioning didn't not remove disks that no longer exist <MAAS:Expired> <https://launchpad.net/bugs/1728308>
<mup> Bug #1747685 opened: MAAS UI could warn about missing EFI partition <MAAS:New> <https://launchpad.net/bugs/1747685>
<mup> Bug #1747685 changed: MAAS UI could warn about missing EFI partition <MAAS:New> <https://launchpad.net/bugs/1747685>
<mup> Bug #1747688 opened: [API, wishlist] not possible to get bios_boot_mode from MAAS CLI <internal> <MAAS:Incomplete> <https://launchpad.net/bugs/1747688>
<mup> Bug #1747764 opened: rack controller HA fails during a network partition <cdo-qa> <maas-ha-testing> <MAAS:New> <https://launchpad.net/bugs/1747764>
#maas 2018-02-07
<mup> Bug #1747928 opened: When a known server in Deploying state boots to the enlisting environment, it should not just power off. <cdo-qa> <cdo-qa-blocker> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1747928>
<mup> Bug #1747928 changed: When a known server in Deploying state boots to the enlisting environment, it should not just power off. <cdo-qa> <cdo-qa-blocker> <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1747928>
<mup> Bug #1747930 opened: MAAS should recognize servers that are Powered Off in Deploying state, and do something about it. <cdo-qa> <cdo-qa-blocker> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1747930>
<mup> Bug #1747928 opened: When a known server in Deploying state boots to the enlisting environment, it should not just power off. <cdo-qa> <cdo-qa-blocker> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1747928>
<mup> Bug #1747930 changed: MAAS should recognize servers that are Powered Off in Deploying state, and do something about it. <cdo-qa> <cdo-qa-blocker> <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1747930>
<mup> Bug #1747930 opened: MAAS should recognize servers that are Powered Off in Deploying state, and do something about it. <cdo-qa> <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1747930>
<mup> Bug #1747928 changed: When a known server in Deploying state boots to the enlisting environment, it should not just power off. <cdo-qa> <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1747928>
<mup> Bug #1747930 changed: MAAS should recognize servers that are Powered Off in Deploying state, and do something about it. <cdo-qa> <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1747930>
<mup> Bug #1747928 opened: When a known server in Deploying state boots to the enlisting environment, it should not just power off. <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1747928>
<mup> Bug #1748010 opened: "Unconfigured" interface in MAAS becomes DHCP interface on deployed node <MAAS:New> <https://launchpad.net/bugs/1748010>
<mup> Bug #1748031 opened: [2.4, devel]  Dashboard/Discovery page: builtins.IndexError: list index out of range <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1748031>
<mup> Bug #1748031 changed: [2.4, devel]  Dashboard/Discovery page: builtins.IndexError: list index out of range <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1748031>
<mup> Bug #1748031 opened: [2.4, devel]  Dashboard/Discovery page: builtins.IndexError: list index out of range <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1748031>
<mup> Bug #1748034 opened: [2.4, devel] Node failed to be commissioned, because of the following error: ScriptSet matching query does not exist. <hardware-testing> <MAAS:Triaged> <https://launchpad.net/bugs/1748034>
<mup> Bug #1748034 changed: [2.4, devel] Node failed to be commissioned, because of the following error: ScriptSet matching query does not exist. <hardware-testing> <MAAS:Triaged> <https://launchpad.net/bugs/1748034>
<mup> Bug #1748034 opened: [2.4, devel] Node failed to be commissioned, because of the following error: ScriptSet matching query does not exist. <hardware-testing> <MAAS:Triaged> <https://launchpad.net/bugs/1748034>
<mup> Bug #1748031 changed: [2.4, devel]  Dashboard/Discovery page: builtins.IndexError: list index out of range <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1748031>
<mup> Bug #1655447 opened: IEEE indexes in netaddr package may not match data in ieee-data package <MAAS:Triaged by mpontillo> <MAAS 2.3:New> <python-netaddr (Ubuntu):Confirmed> <https://launchpad.net/bugs/1655447>
#maas 2018-02-08
<mup> Bug #1748051 opened: [2.4, devel]  <MAAS:New> <https://launchpad.net/bugs/1748051>
<mup> Bug #1748052 opened: [2.4, devel] ] Unable to write to plugin cache /usr/lib/python3/dist-packages/twisted/plugins/dropin.cache: error number 13 <MAAS:Triaged> <https://launchpad.net/bugs/1748052>
<mup> Bug #1748055 opened: [2.4, devel] While commissioning/testing <MAAS:Triaged> <https://launchpad.net/bugs/1748055>
<tasker> my maas-proxy cache is consuming some disk space. is there an official way to clear out this cache or is "rm -r /var/spool/maas-proxy/*" an acceptable solution?
<tosaraja> Is maas itself providing the cloud-init content when deploying a host? I'm trying to deploy a 17.10 host here and I'm getting "404 Not found http://old-releases.ubuntu.com/ubuntu artful-security Release". Well, artful isn't an old-release, so why is it trying to pick it from there?
<mup> Bug #1748187 opened: Only 16.04 Xenial available for commissioning <MAAS:New> <https://launchpad.net/bugs/1748187>
<mup> Bug #1748187 changed: Only 16.04 Xenial available for commissioning <MAAS:Won't Fix> <https://launchpad.net/bugs/1748187>
<mup> Bug #1589140 changed: No WOL option in latest MAAS version for 16.04 <MAAS:Won't Fix> <https://launchpad.net/bugs/1589140>
<ejat> hi .. any update on bug 1673724
<ejat> bug #1673724
<roaksoax> ejat: seems fixed to me
<kiko> ejat, are you seeing it on a modern version of MAAS?
<niedbalski> roaksoax, https://bugs.launchpad.net/maas/+bug/1743144 is affecting HPe machines , not sure why that repository is added by default.
<niedbalski> roaksoax, https://pastebin.ubuntu.com/26542719/ fyi
<andrew-ii> I need to deploy Debian 9, but it seems like `ifenslave` is not included by default. Is there a time in the curtin setup that I can make it install the package before it configures the NICs?
<andrew-ii> (That is, Debian 9's default ISO doesn't include it. Perhaps I need to make a new Debian cloud installer image from scratch?)
<roaksoax> niedbalski: you can disable the config to install third party drivers on the settings
<roaksoax> niedbalski: or you can remove that from the config in /etc/maas/drivers.yaml
<niedbalski> roaksoax, yeah, but shouldn't be this disabled per series? as this isn't really available for xenial
<roaksoax> niedbalski: not relaly not. We have no way of knowing it is on the repository or not
<roaksoax> niedbalski: but since there's options to disable/enable this or remove the use of the driver altogether
<roaksoax> i think there's ways to not be affected
<roaksoax> niedbalski: in fact, you could even change the repository where you get the drivers from
<niedbalski> roaksoax, well, a mention in the documentation is worth then, it took me a few to discover it as by default third party drivers are enabled.
<roaksoax> niedbalski: uhmm seems this section as removed from the docs
<niedbalski> roaksoax, looks like
<niedbalski> roaksoax, now I am hitting 1730456 :-)
<roaksoax> niedbalski: what's your rackd.conf and your regiond.conf ?
<niedbalski> roaksoax, https://pastebin.ubuntu.com/26543275/
<roaksoax> niedbalski: i bet that 10.10.1.7 is not the ip the machines can reach MAAS at
<niedbalski> roaksoax, https://pastebin.ubuntu.com/26543312/ .. yes, that's not the address the machines are reaching (192.168.100.0/24)
<roaksoax> niedbalski: so you have to options, leave rackd.conf as localhost and update regiond.conf correctly
<roaksoax> niedbalski: or mofidy rackd.conf
<niedbalski> roaksoax, do you see any evident correlation with the error that I just posted? I wonder if you know something I don't :-)
<roaksoax> niedbalski: that one could be a clockskew thing
<niedbalski> roaksoax, clock is aligned in both maas/deployed node
<niedbalski> anyways, I am ntp syncing and modifying regiond accordingly.
<roaksoax> yeah seems cloud-init is doing the right thing by fixing the clock skew
<roaksoax> niedbalski: did you fix rackd.conf, restarted it and retried ?
<niedbalski> roaksoax, probando doctor
<niedbalski> roaksoax, https://pastebin.ubuntu.com/26543384/
<roaksoax> niedbalski: release the machine, or abort it and try ?
<niedbalski> roaksoax, just did it.
<roaksoax> niedbalski: it could be due to the clock skew that it cannot authorize
<roaksoax> niedbalski: hwat about other mcahines ?
<niedbalski> roaksoax, yes, is there something like a cache for tokens or similar?
<roaksoax> no, we dont cache tokes
<roaksoax> no, we dont cache tokens
<roaksoax> each tim e a new token gets re-generated
<roaksoax> niedbalski: that said, is this a commissioning or enlistment ?
<roaksoax> niedbalski: err or deployment ?
<niedbalski> roaksoax, thats ok, other machines (non hp) works ok, and this started to happen after I disabled the third party drivers
<niedbalski> roaksoax, commissioning
<roaksoax> niedbalski: the other one could be that its ready user data from the disk instead of the pxe process
<niedbalski> roaksoax, the mac address remains the same, i deleted/created the machine with another name/uuid just in case.
<roaksoax> niedbalski: yeah lets try that and see what happens
<roaksoax> that's a strange error though
<niedbalski> roaksoax, should I wipe out the disks before?
<roaksoax> niedbalski: if you could that'd be good
<niedbalski> roaksoax, might be the old userdata is being read from the disk
<roaksoax> indeed
<niedbalski> roaksoax, have you seen something like that?
<roaksoax> niedbalski: nope, i personally havent, althjough we did fix a bug long time ago that required a new cloud-init so that it wouldn't read form disk
<roaksoax> niedbalski: are your images the latest ?
<niedbalski> roaksoax, I think yes, they are in sync with images.maas.io
<roaksoax> i wonder if cloud-init could have regressed and no longer listens to this option we send it kernel params
<niedbalski> roaksoax, let me see, I will wipeout the disk arrays
<niedbalski> roaksoax, which option?
<roaksoax> we send an option on the kernel command line to alwys use the maas datasource
<niedbalski> roaksoax, commission with any non-available ppa, (commission fails), remove the ppa and retry commissioning
<niedbalski> simplest reproducer.
<xygnal> roaksoax this memory bug is killing our performance.  our clients are u happy with the delays. how do we step up the level of toubleshooting on 1744765
<roaksoax> xygnal: at this point, the only thing i can't think of is that's related to you running on top of vmware
<roaksoax> xygnal: we have confirmed we have larger maas' (e.g. way many more machines in a single maas), running on hardware that dont exhibit these issues
<roaksoax> niedbalski: aha! you weren't giving me enough info. So it is clear now. cloud-init fails to configure the archive, it tels that to maas and the machine gets marked as faled commissioning, the oauth keys expire and you see that in the logs :)
<roaksoax> xygnal: on the same version that is
<roaksoax> xygnal: so you could run a test by dumping your db, and importing it in a cleanly installed maas of the same version
<roaksoax> xygnal: on different underlying hardware
<niedbalski> roaksoax, https://pastebin.ubuntu.com/26543565/
<niedbalski> roaksoax, ok, so the first failure was ok, then I removed the archive (disabled third party drivers completely) and re-commissioned
<roaksoax> niedbalski: yeah that makes sense
<niedbalski> roaksoax, ok, failed the commissioning with the same error
<niedbalski> i wiped out the disk in case it was using disk source
<roaksoax> niedbalski: can you show all the cloud-init-output log ?
<roaksoax> niedbalski: and all cloud-init.log too
<niedbalski> roaksoax, oops :-)
<niedbalski> roaksoax, q: does the oauth tokens gets expired in case of failure during any of the cloud-init stages?
<roaksoax> niedbalski: yes, so if cloud-init sayas "hey maas i failed to configure this thing you told me to" maas sees the failure message from cloud-init, and marks it failed commissioning
<roaksoax> niedbalski: and then you would see those errors that you cant access the metadata
<niedbalski> roaksoax, gotcha, let me retry and upload the full cloud-init logs for you to look.
<niedbalski> roaksoax, ok, now all went through .. i removed the maas squid cache, the problem was due to a hash mismatch while running the apt update phase on cloud-init, not sure how the proxy ended up that way.
<niedbalski> roaksoax, deploying now, lets see :-)
<roaksoax> cool
<xygnal> roaksoax how much memory on these bare metal region controllers you test on?
#maas 2018-02-09
<tasker> my maas server is running out of disk space. I don't want to turn off the  maas-proxy service, but I do want to clean it up. is there an "official" way to do this? or can I simply "rm -r /var/spool/maas-proxy/*"?
<xygnal> roaksoax: please give me more detail about your larger unimpacted environment where this issue is not seen.  what kind of hardware capacity?  what sysctl settings on the OS?
<roaksoax> xygnal: i don't have the details i hand as wasn't given that information. But 32GB of ram seems plenty to me. I have a feeling that this could be related to it running on vmware
<roaksoax> that said
<roaksoax> how UI intensive are you ?
<xygnal> we are very UI intensive.  we have as many as probably 4-6 people refreshing at the same time impatiently, beacuse refresh takes a long time.
<xygnal> like i have implied before, if the first scan of nodes is slow. any further scans and reloads-of-scans just caues memory to go craaaaazy until its killed
<xygnal> and that source reason of the first scan being SLOW is likely why we see it trigger so often
<roaksoax> xygnal: so the devices list has bene testing with 8000+ nodes
<roaksoax> the machine listing with 2000+ "fake nodes"
<xygnal> and what data do these fake nodes have in them, that a real node would have, such as all of the logs of its commissions?
<xygnal> are those not, normal, regular things to have in a node history? a real node hstory?
<roaksoax> xygnal: the only difference is power paramaters
<roaksoax> but everything else is filled
<roaksoax> e.g. commissioning data, events, hardware testing data, etc
<xygnal> ah if they are not checking power then i would expect them to simply breeze by at high speeds
<xygnal> no hardware, no slowdowns
<xygnal> that doest seem very effective testing to me :/
<roaksoax> xygnal: so why dont you try to start a couple workers ?
<roaksoax> xygnal: or at least an extra worker ?
<roaksoax> xygnal: in 2.3 that's all done in systemd
<xygnal> we can do that? i thought 4 was the limit
<roaksoax> xygnal: we dont support more owrkers on 2.3, but its been done in te past afaik
<roaksoax> xygnal: 2.4 will introduce dynamic worker up to 8 workers
<xygnal> i'll look into adding a worker in systemd
<xygnal> would that actally  cause any difference scans or just UI response?
<roaksoax> xygnal: it should spread the load more, we dont pin specific workers to specific services
<roaksoax> at least not on 2.3
<roaksoax> 2.4 will have some worker separation between what each worker does
<roaksoax> xygnal: also, i would be interested in knowing what data is being setn over the websocket
<xygnal> I provided screen shots of that in the bug report
<xygnal> but mike confirmed we cannot export the logs for web socket
<roaksoax> xygnal: yea we can't but I mean, see how big the data being sent is
<xygnal> it didnt look that big to me
<xygnal> look at the attached screen shots :)
<roaksoax> xygnal: have the bug link in hand ?
<xygnal> if you want network traces, or database dumps, core dumps from memory kills, just tell us what to gather to get you deeper
<roaksoax> what i'm more interested in knowing is what data is loading over the websocket and how big it is
<roaksoax> for example, it could be loading data for the 500 machines
<roaksoax> instead of loading data only for the machines that you can see
<roaksoax> although that should have gotten fixed
<xygnal> https://bugs.launchpad.net/maas/+bug/1744765
<roaksoax> maybe the hardware testing is loading more dat
<xygnal> it looks to be grabbing them 50 at a time
<xygnal> from what i saw in the websocket calls
<roaksoax> xygnal: https://i355451027.restricted.launchpadlibrarian.net/355451027/Screen%20Shot%202018-02-01%20at%202.42.35%20PM.png?token=PqMK4FCvf7Dfg9cp83g88PzFwD0K4hMd
<roaksoax> xygnal: in thta screenshot, the above has a length of 75489
<xygnal> what is *in* that request
<xygnal> what is the payload to make it so big
<xygnal> or rather, what CAN it be that would make it so big
<roaksoax> xygnal: i would like to see the expanded output
<roaksoax> to know for sure
<roaksoax> xygnal: but in the one that's already expanded
<xygnal> not sure how to do that since it was not letting me export
<roaksoax> xygnal: you can see it seems to be for various machines
<xygnal> web sockets does not support export to file
<roaksoax> xygnal: right, that's fine screenshots are fine
<xygnal> will see if i can copy it, i thought i had trouble getting it to LET me
<roaksoax> but for example, in that 75k length one
<xygnal> what you want a 10 page screen shot?
<xygnal> ;)
<roaksoax> xygnal: the data will be organized per machine, so first things first would be to see for how many machines its showing that data
<xygnal> it looked like it was 50 machines at a time, in those requests
<xygnal> when iwas expanding and digging around
<roaksoax> e.g. if it is showing for 590... even though the UI is only rendering 10, then that seems like a bug
<roaksoax> xygnal: and tin the 75k one, per ecah machine, what data is being sent
<roaksoax> so i would need to know those two things
<xygnal> I will see what I can  get
<roaksoax> thanks
<xygnal> btw, when thsi happens, we dont see all of the twistd3 processes going nuts at the same time.
<xygnal> it's usually one or two processes that just grow grow grow in cpu and memory
<xygnal> so i dont think threading is going to do much
<xygnal> if you think our commission logs could be part of the problem, is there a quick database query you could propose to see just how much of that data we have?
<roaksoax> i dont think the commissioning logs are the problem actually, since I believe you applied a fix in the websockets to not load the whole file
<roaksoax> but rather, if virtual scrolling is working as expected, it should not be loading the data from the 500 machines
<roaksoax> only from the ones you see rendered
<roaksoax> xygnal: https://stackoverflow.com/questions/29953531/how-to-save-websocket-frames-in-chrome
<xygnal> nice find :)
<xygnal> we did apply a fix that as proposed, but we backed it out in prod after it did not have affect.
<xygnal> fyi
<roaksoax> yeah that was only for machine details
<roaksoax> not really for the listing
<xygnal> roaksoax: any change increased threads may cause problems connecting to rack controllers? none of mine can connect now.
<mup> Bug #1748538 opened: [2.4] Updating the boot source can cause duplicate entries in the boot source cache <MAAS:Triaged> <https://launchpad.net/bugs/1748538>
<xygnal> roaksoax:  soemthing caused all the rackd's to hang, so i restarted their service..  I tried that code in the inspector console, it fails. syntax is not right.  not sure how to write proper syntax to do this.
<mup> Bug #1748542 opened: [2.4, API] Pods create do not document the parameters needed per type <doc> <pod> <trivial> <MAAS:Triaged> <https://launchpad.net/bugs/1748542>
<xygnal> roaksoax:  bug updated with requested WS traces
#maas 2018-02-10
<mup> Bug #1748569 opened: [2.3, trunk] rackd makes redudant RPC calls to the region sitting idle <performance> <MAAS:In Progress by blake-rouse> <https://launchpad.net/bugs/1748569>
<mup> Bug #1748570 opened: [2.3, trunk] MAAS power types should be cached by the regiond and not asked all the time <performance> <MAAS:Triaged by blake-rouse> <https://launchpad.net/bugs/1748570>
<mup> Bug #1736767 changed: Manual power won't survive commissioning/deploying <MAAS:Expired> <https://launchpad.net/bugs/1736767>
#maas 2018-02-11
<mup> Bug #1748712 opened: Upgrade to 2.4.0a1 failed <MAAS:New> <https://launchpad.net/bugs/1748712>
<mup> Bug #1748732 opened: Deleting a controller should not block on it being the primary rack controller <MAAS:New> <https://launchpad.net/bugs/1748732>
<Troy_> Hi, I'm wondering if it's possible with MAAS to pool together 6 servers and create VMs from that combined pool of resources?
#maas 2020-02-03
<mup> Bug #1861477 changed: [2.7] Regression: Commissioning data does contain block device information <cdo-qa> <cdo-release-blocker> <foundations-engine> <MAAS:Invalid by adam-collard> <https://launchpad.net/bugs/1861477>
<mup> Bug # changed: 1811107, 1822222, 1832231, 1858817, 1858820, 1859032, 1859210, 1859426, 1859481, 1859622, 1859623, 1859630, 1859849
<mup> Bug # changed: 1850850, 1851264, 1859087, 1859626
<mup> Bug # opened: 1850850, 1851264, 1859087, 1859626
<mup> Bug # changed: 1850850, 1851264, 1859087, 1859626
#maas 2020-02-04
<sfeole> o/    hey all,   I was wondering if anyone knows when maas-images will produce the the latest Bionic point release images for bionic 18.04.4 ?
<sfeole> the first release candidate is here: http://iso.qa.ubuntu.com/qatracker/milestones/410/builds
<mup> Bug #1818703 changed: regiond stops listening on the API port 5240 until regiond is restarted (listening sockets are lost) after database failover <cpe-onsite> <MAAS:Triaged by ack> <https://launchpad.net/bugs/1818703>
#maas 2020-02-05
<mup> Bug #1861952 opened: [2.7.0~rc2] cannot create passwords with '+' character <MAAS:New> <https://launchpad.net/bugs/1861952>
<mup> Bug #1861952 changed: [2.7.0~rc2] cannot create passwords with '+' character <MAAS:New> <https://launchpad.net/bugs/1861952>
<mup> Bug #1861952 opened: [2.7.0~rc2] cannot create passwords with '+' character <MAAS:New> <https://launchpad.net/bugs/1861952>
<mup> Bug #1862057 opened: cannot dnsresource delete id <delete> <dnsresource> <id> <MAAS:New> <https://launchpad.net/bugs/1862057>
<mup> Bug #1862057 changed: cannot dnsresource delete id <delete> <dnsresource> <id> <MAAS:New> <https://launchpad.net/bugs/1862057>
<mup> Bug #1862057 opened: cannot dnsresource delete id <delete> <dnsresource> <id> <MAAS:New> <https://launchpad.net/bugs/1862057>
#maas 2020-02-06
<mup> Bug #1862102 opened: [2.7.0~rc3] Redfish power configuration vanishes after commissioning <MAAS:New> <https://launchpad.net/bugs/1862102>
<mup> Bug #1862102 changed: [2.7.0~rc3] Redfish power configuration vanishes after commissioning <MAAS:New> <https://launchpad.net/bugs/1862102>
<mup> Bug #1862102 opened: [2.7.0~rc3] Redfish power configuration vanishes after commissioning <MAAS:New> <https://launchpad.net/bugs/1862102>
<mup> Bug #1862103 opened: [2.7.0~rc3] Cannot create KVM pod with bonded NICs <MAAS:New> <https://launchpad.net/bugs/1862103>
<mup> Bug #1862103 changed: [2.7.0~rc3] Cannot create KVM pod with bonded NICs <MAAS:New> <https://launchpad.net/bugs/1862103>
<mup> Bug #1862103 opened: [2.7.0~rc3] Cannot create KVM pod with bonded NICs <MAAS:New> <https://launchpad.net/bugs/1862103>
<mup> Bug #1862103 changed: [2.7.0~rc3] Cannot create KVM pod with bonded NICs <MAAS:New> <https://launchpad.net/bugs/1862103>
<mup> Bug #1862103 opened: [2.7.0~rc3] Cannot create KVM pod with bonded NICs <MAAS:New> <https://launchpad.net/bugs/1862103>
<mup> Bug #1862107 opened: Rack controller DNS does not forward queries to regional controllers <MAAS:New> <https://launchpad.net/bugs/1862107>
<mup> Bug #1861543 changed: Nodes list page does not show node action errors <ui> <MAAS:Fix Released by caleb-ellis> <https://launchpad.net/bugs/1861543>
<mup> Bug #1861543 opened: Nodes list page does not show node action errors <ui> <MAAS:Fix Released by caleb-ellis> <https://launchpad.net/bugs/1861543>
<mup> Bug #1861543 changed: Nodes list page does not show node action errors <ui> <MAAS:Fix Released by caleb-ellis> <https://launchpad.net/bugs/1861543>
#maas 2020-02-07
<mup> Bug #1862345 opened: Logging out results in a "Method Not Allowed" error <ui> <MAAS:New> <https://launchpad.net/bugs/1862345>
<mup> Bug #1862345 changed: Logging out results in a "Method Not Allowed" error <ui> <MAAS:Confirmed for caleb-ellis> <maas-ui:New> <https://launchpad.net/bugs/1862345>
<mup> Bug #1862345 opened: Logging out results in a "Method Not Allowed" error <ui> <MAAS:Confirmed for caleb-ellis> <maas-ui:New> <https://launchpad.net/bugs/1862345>
<mup> Bug #1862379 opened: DB migrations is in a bad state <MAAS:Triaged> <MAAS 2.7:Triaged> <https://launchpad.net/bugs/1862379>
<mup> Bug #1862345 changed: Logging out results in a "Method Not Allowed" error <ui> <MAAS:Fix Released by caleb-ellis> <maas-ui:Fix Released> <https://launchpad.net/bugs/1862345>
<mup> Bug #1862345 opened: Logging out results in a "Method Not Allowed" error <ui> <MAAS:Fix Released by caleb-ellis> <maas-ui:Fix Released> <https://launchpad.net/bugs/1862345>
<mup> Bug #1862345 changed: Logging out results in a "Method Not Allowed" error <ui> <MAAS:Fix Released by caleb-ellis> <maas-ui:Fix Released> <https://launchpad.net/bugs/1862345>
#maas 2020-02-09
<mup> Bug #1862514 opened: hpdsa driver no loner available <MAAS:New> <https://launchpad.net/bugs/1862514>
<mup> Bug #1862514 changed: hpdsa driver no loner available <MAAS:New> <https://launchpad.net/bugs/1862514>
<mup> Bug #1862514 opened: hpdsa driver no loner available <MAAS:New> <https://launchpad.net/bugs/1862514>
<mup> Bug #1862514 changed: hpdsa driver no loner available <MAAS:New> <https://launchpad.net/bugs/1862514>
<mup> Bug #1862514 opened: hpdsa driver no loner available <MAAS:New> <https://launchpad.net/bugs/1862514>
