[00:04] <mup> Bug #1595753 opened: [beta8] HMC power driver regression -- Not able to connect via SSH. <MAAS:In Progress by newell-jensen> <https://launchpad.net/bugs/1595753>
[00:11] <valeech> mpontillo: bug opened: https://bugs.launchpad.net/maas/+bug/1595755
[00:19] <mup> Bug #1595755 opened: MTU does not get set on bonded interfaces <MAAS:New> <https://launchpad.net/bugs/1595755>
[00:22] <mpontillo> valeech: thanks.
[00:31] <mup> Bug #1595755 changed: [2.0b8] MTU does not get set on bonded interfaces <MAAS:New> <https://launchpad.net/bugs/1595755>
[00:34] <mup> Bug #1595755 opened: [2.0b8] MTU does not get set on bonded interfaces <MAAS:New> <https://launchpad.net/bugs/1595755>
[15:48] <mup> Bug #1595996 opened: Create bond interface in the official documentation <MAAS:New> <https://launchpad.net/bugs/1595996>
[15:54] <mup> Bug #1595996 changed: Create bond interface in the official documentation <MAAS:New> <https://launchpad.net/bugs/1595996>
[15:57] <mup> Bug #1595996 opened: Create bond interface in the official documentation <MAAS:New> <https://launchpad.net/bugs/1595996>
[16:18] <mup> Bug #1596018 opened: (doc) missing instructions on how to enable SSL support <MAAS:New> <https://launchpad.net/bugs/1596018>
[17:37] <newell_> blakeh: ping
[17:54] <mup> Bug #1596046 opened: maas 2.0 pxeboot fails on power 8 <MAAS:New> <https://launchpad.net/bugs/1596046>
[18:08] <newell_> blakeh: you around?
[18:09] <blakeh> here
[18:09] <newell_> Have you downloaded images for your MAAS 2.0 server?
[18:09] <blakeh> yes
[18:10] <blakeh> ppc and amd
[18:10] <newell_> blakeh: and you have DHCP configured for this MAAS server?
[18:10] <newell_> blakeh: from the bug report it seems like you are deploying a 2nd MAAS server on one of the MAAS 1.9 nodes that you deployed?
[18:10] <blakeh> newell - yes
[18:10] <blakeh> newell - i have the first one shut donw
[18:11] <newell_> Or is this an isolated MAAS 2.0 installation (from MAAS 1.9)
[18:11] <newell_> ?
[18:11] <blakeh> newell  -- i have a 14.04  with maas 1.9 that is currently powered off on same subnet-
[18:12] <blakeh> newell - to prevent dhcp problems
[18:12] <newell_> blakeh: okay, just wanted to make sure
[18:12] <blakeh> newell -- deploying an x86 node works fine
[18:12] <newell_> when you try and boot your power nodes with MAAS 1.9 is it serving pxelinux.0 as well?
[18:13] <blakeh> newell - yes- no issue
[18:13] <newell_> blakeh: okay power should be getting served something other than pxelinux.0
[18:13] <newell_> so that is concerning
[18:15] <blakeh> newell - i think pxelinux.0 is correct for ppc64el--- ppc64 is served by boot ppc64.bin
[18:15] <blakeh> newell -- lookin in /var/lib/maas/dhcpd.conf seems to indicate this is correct - on both 1.9 and 2.0
[18:16] <newell_> blakeh: ah maybe I am thinking powerVM
[18:16] <newell_> what type of power is this?
[18:16] <newell_> powerNM
[18:16] <newell_> NV*
[18:16] <blakeh> yes nv
[18:16] <newell_> okay that is why
[18:16] <newell_> thanks for verifying
[18:17] <blakeh> power 822L
[18:17] <newell_> are you using trusty images or xenial?
[18:17] <newell_> or something else?
[18:17] <blakeh> did put in a bug
[18:17] <blakeh> xenial
[18:17] <newell_> let me ping the powerNV guy and see if he has tried this with MAAS 2.0
[18:18] <blakeh> sure -- Bug #1596046 has some logs attached
[18:18]  * newell_ looking now
[18:22] <newell_> blakeh: so this is simply manually turning the power on for the machine correct?
[18:22] <newell_> blakeh: have you tried entering the power parameters and trying to commission the node directly?
[18:22] <blakeh> yes -- power on and do a pxe boot
[18:22] <blakeh> have not tried entering manually yet
[18:23] <newell_> blakeh: try that for now
[18:24] <blakeh> ok - will take a few minutes
[18:46] <wililupy> crazy question here...Is there a way to get MAAS's DNS server to sync with a regular bind9 DNS server?
[18:48] <blakeh> newell:  so at this point when i select commission for the power 8 node it does power it on an bring it up to petitboot, but does not load a new os on sda - hangs on commissioning- i expect it to time out and error.
[18:51] <newell_> blakeh: so it was able to find the pxelinux.cfg unlike before?
[18:52] <blakeh> no -- it was unable to install an os -- no pxe boot  ---
[18:52] <newell_> blakeh: no OS is installed during commissioning
[18:53] <blakeh> perhaps i picked the wrong action >  i selected commission- should it be acquire?
[18:53] <newell_> commissioning is what you want
[18:53] <newell_> do you have console log?
[18:53] <blakeh> correct-- i am trying again-- net info was not correct last time
[18:53] <newell_> ack
[18:54] <newell_> wililupy: not sure.  mpontillo any idea if we can get MAAS DNS server to sync with regular bind9 DNS server?
[18:55] <mpontillo> wililupy: newell_: not sure what is meant by "sync", but I understand that a common configuration is to delegate a subdomain to MAAS
[18:56] <wililupy> Reason for asking is that I don't want to have to continue to have to manually add the servers provisioned by maas manually in my DNS server.
[18:56] <mpontillo> MAAS only partially takes control of the bind9 configuration. it's likely that you could configure bind9 however you like, and MAAS will mostly stay out of your way
[18:57] <mpontillo> wililupy: what I do for the MAAS I test with is I tell my local dnsmasq that my MAAS DNS server is authoritative for the .maas (or whatever) domain
[18:57] <mpontillo> wililupy: in short, yes, it should be possible. alternatively you could just rsync the zone files MAAS generates to your other bind9 server
[18:57] <mpontillo> wililupy: there are any number of ways you could tackle this problem
[18:58] <wililupy> mpontillo: Gotcha. That makes sense. I could add a subdomain in my DNS server such as maas.domain.net and have it be the authority for that.
[18:58] <wililupy> it being MAAS
[18:58] <mpontillo> wililupy: yeah, just update the domain in MAAS to be whatever your chosen subdomain is
[18:59] <blakeh> newell -- so it loaded the os on the hard drive- ub 14.04 instead of installing the commissioning os - u 16.04
[19:01] <bdx> hey whats up all? Is there a reason why a space can't be assigned to a fabric with no subnets defined?
[20:42] <sjl> how does one log into a failed centos deploy?  the iDRAC console shows login with 'localhost login' and my normal ssh key doesn't apply since the node instance apparently did not successfully get on the network
[21:15] <pacavaca_> 3
[21:15] <pacavaca_> oops, accidentally pressed enter
[21:18] <pacavaca_> Is there anybody around to answer some questions?
[21:45] <mpontillo> sjl: did the machine boot into CentOS? IIRC, since MAAS don't do advanced network configuration on the CentOS images, you must use DHCP. not sure if that helps.
[21:46] <mpontillo> sjl: normally I would say that you may able to modify the cloud-init user data (see /etc/maas) to configure a password, but it sounds like your deployed nodes aren't able to reach the MAAS metadata server anyway
[21:47] <mpontillo> sjl: did you try an Ubuntu deployment? if so, any issues?
[21:47] <mpontillo> pacavaca_: just ask your question; if someone knows the answer they will jump in
[21:47] <sjl> mpontillo: the problem is intermittent with about 20% failure rate with centos,  I haven't seen any errors with ubuntu but haven't tried that much to be sure
[21:48] <mpontillo> sjl: so it works one out of every ~5 times -- on the same machine, or different machines? it would be good to isolate why it fails, when it does...
[21:48] <sjl> mponntillo: yes the machine did boot to centos, so it was able to communicate to maas at some stage
[21:49] <mpontillo> sjl: if you are using MAAS 2.0 you might check /var/log/maas/rsyslog/<node> to see if there are any clues. cloud-init should configure the MAAS server as an rsyslog endpoint
[21:49] <sjl> mpontillo: same machine
[21:49] <mpontillo> sjl: though I don't think that rsyslog feature persists after deployment is done
[21:51] <sjl> i'm using the standard centos66 from http://images.maas.io/ephemeral-v2/daily/  Is there a preseed user account ('root' or 'centos') with a static password to get me in?
[21:52] <mpontillo> sjl: no, there are no static passwords. if cloud-init is failing to get the machine's configuration, I doubt that an additional cloud-init preseed with a static password would help you
[21:53] <pacavaca_> maas2: on the controller machine I've changed some network configuration. Basically I broke a bridge and gave the IP back to the physical NIC. Now how can I tell maas to re-detect the network configuration on controller? And actually, why can't I run DHCP on a bridge?
[21:54] <mpontillo> pacavaca_: if you are on MAAS 2.0 it should auto-detect within ~30 seconds
[21:54] <mpontillo> pacavaca_: DHCP on a bridge should work fine. which version of MAAS 2.0 are you running? if you didn't use the PPA, you probably have beta 3. if that is the case, you want to get the latest beta from ppa:maas/next
[21:55] <sjl> how about a backdoor method into centos66?  I've tried without success: https://maas.ubuntu.com/docs/troubleshooting.html?highlight=backdoor#debugging-ephemeral-image
[21:55] <pacavaca_> mpontillo: hmm. it doesn't seem to happen http://d.pr/i/17W8y I'm on beta8 (from ppa). Should I look into some logs? I haven't found a way to enable DHCP on a bridge through UI..
[21:55] <mpontillo> pacavaca_: though I am not sure what you mean by "broke a bridge and gave the IP back to the physical NIC"; if you have a bridge configured, the IP does need to be on the bridge.
[21:56] <pacavaca_> mpontillo: it was. but now I removed that bridge
[21:58] <pacavaca_> Uh oh. Seems it has resolved. I did not remove a bridge but just brought it down and brought the NIC back up and it didn't work. But now I did 'brctl delbr br0' and MAAS detected the new config right away.
[21:58] <pacavaca_> So, false alarm. Thank you for the attention mpontillo!
[21:59] <mpontillo> pacavaca_: no problem, glad it's working ;-)
[22:00] <mpontillo> pacavaca_: DHCP is enabled per-VLAN in MAAS 2.0, not per interface. this change was made so that we can support HA. you'll find the DHCP on/off options on the VLAN details page under Networks. (a but buried, I admit.)
[22:04] <pacavaca_> mpontillo: yeah, just found it.. seems my bridge got destroyed for nothing :)
[22:53] <bdx> general note: when enlisting virsh instances, ensure you don't configure them to '--autostart' :-)
[22:57] <pacavaca_> mpontillo: how does MAAS figure out when node is deployed? I just tried using my custom image and the machine got re-imaged correctly, I can log in and use it. However, MAAS still shows it as deploying. Should there be some maas agent running on machine or what's the mechanism?
[23:01] <mpontillo> pacavaca_: we consider MAAS to be agentless. but cloud-init acts as our agent, and I *think* (but would have to verify) that we can tell the machine is deployed due to the notification from cloud-init
[23:06] <pacavaca_> mpontillo: ok, thanks. Is there any way I can verify cloud-init is functionalble? Because this is a custom image I've created in some hacky way and things may be broken.
[23:07] <mpontillo> pacavaca_: well, I confirmed that the cloud-init request for user-data is what hands the node off to deployed state. does your custom image include cloud-init?
[23:08] <pacavaca_> It should, but how do I verify that?
[23:09] <bdx> :-)
[23:09] <bdx> pacavaca_: why/how should it?
[23:10] <mpontillo> pacavaca_: well, besides the obvious, I might do a packet capture from the rack controller during deployment and compare what happens for a supported image to yours.
[23:10] <pacavaca_> I initially created the image using curtinator and I assume it installed cloud-init. But then I did some modifications to the contents of tarball, so It might broke something
[23:10] <pacavaca_> that's to bdx :)
[23:14] <mpontillo> pacavaca_: okay. well, I would suggest testing with the official Ubuntu image to see if you can isolate the problem. I can't dig too deeply into custom images since that's something Canonical sells support for, as it can be a bit involved to get it right, as you see =)
[23:16] <pacavaca_> mpontillo: sure, I understand that. Just looking for some hints to simplify debugging. Ok, I will try using the official image just to see if it actually works.
[23:34] <f1gjam> hey guys, I am trying to install openstack via autopilot, but juju fails to boot strap a node - here is the log file http://paste.ubuntu.com/17779796/
[23:35] <f1gjam> any ideas?
[23:43] <bdx> flgjam: ask in #ubuntu-solutions
[23:49] <bdx> things are looking sweet you guys!
[23:49] <bdx> just gave the experimental3 a go
[23:49] <bdx> beautiful
[23:49] <bdx> http://imghub.org/image/nKE0
[23:50] <bdx> http://imghub.org/image/nccF
[23:50] <bdx> http://imghub.org/image/n9ZB
[23:51] <bdx> http://imghub.org/image/nuMw
[23:52] <bdx> that is a beautiful thing
[23:52] <bdx> thanks!
[23:52] <bdx> you guys are killlling it!