[01:02] Bug #1621285 opened: Server VLAN's on a Rack Details page not loading. [06:53] Bug #1621344 opened: [2.0] Multiple failed trusty deployments -- rename failed, Stale file handle

=== frankban|afk is now known as frankban [10:34] Bug #1621407 opened: [2.1] Add node listing filters for Network (NIC) and Storage (devices) tags [10:34] Bug #1621409 opened: On save tags bug storage/interfaces [11:19] Bug #1621344 changed: Error reading package list on ephemeral environemt

[11:37] Bug #1621344 opened: Error reading package list on ephemeral environemt

[11:40] Bug #1621344 changed: Error reading package list on ephemeral environemt

[11:55] Bug #1621344 opened: Error reading package list on ephemeral environemt

[12:16] Bug #1621446 opened: [Vanilla] Design QA bugs [12:43] Bug #1621344 changed: Error reading package list on ephemeral environemt

[14:15] roaksoax, how do you think we should advise sujeet to store firmware blobs that commissioning scripts should access? [14:16] roaksoax, I actually think the object store is a nice solution to that problem [14:16] roaksoax, did it have major issues? [14:23] Hello all [14:23] anyone has recently installed maas over vsphere node? [14:23] kiko: yes it did have major issues and juju didn't need it anymore. The object store was there for juju really. We didn't removed it in 2.0 because juju still needed it. [14:24] kiko: we wont remove it in 2.0 series [14:24] 2.x series [14:24] roaksoax, what were the major issues? [14:24] thanks roaksoax, [14:24] kiko: so if there's a use case to keep it, we can keep it [14:24] I am in a much preliminary issue, adding nodes to maas [14:24] roaksoax, we should think about it more carefully, yeah [14:24] pxe works but I can not found the proper power type [14:25] nirlevy, I think PCdude has an installation like that, currently with some problems [14:25] i'll need to dig through the buglist to find the issues it has [14:25] nirlevy, there's a VMware power type? [14:25] kiko: nothing blocking though, it is still usable [14:25] roaksoax, thanks [14:25] roaksoax, I remember there was an auth issue IIRC [14:25] but that's all I remember [14:26] can someone please explain the methodology behind virtual node booting, [14:27] I am confused, in one hand I have virtual nodes over my esxi host [14:27] on the other hand maas node should include libvirt to manage them, is it correct? [14:29] nirlevy, no, MAAS uses libvirt for KVM nodes [14:31] nirlevy, I'm curious -- does the version of MAAS you are using not have a VMware power type? what version are you using? [14:31] nirlevy, virtual nodes boot into an ephemeral environment via PXE [14:31] nirlevy, what we need the power controller for is simply to turn the nodes on and off [14:32] I am using ubuntu 14.04 and apt-get installation which is version 1.9 [14:32] for maas [14:32] roaksoax, the 1.9 SRU went through then? [14:33] nirlevy, and is there not a VMware power type? [14:33] in the dropdown? [14:34] it do have VMWare in the drop down menu [14:36] after pxe boot nodes status is "new" [14:36] when I commissioning them, the operation fails [14:36] nirlevy, correct, you go to the node then, choose that power type and set the VMware details correctly [14:37] the VMware setup is rather manual [14:37] let me try it now. [14:39] the status of the nodes after PXE boot is at login, I do not know the user and password (is there is any default?) those are the same details I need for the VMWare option [14:42] nirlevy, the user and password are your VSphere credentials [14:42] nirlevy, the nodes are New because they have just enlisted; once you have the power type set up you can kick off a commissioning process [14:43] Thanks, Trying [14:43] nirlevy, if you get failures check your VSphere access logs [14:44] Bug #1618543 opened: freeipmi lacks IPv6 support

[14:45] poor freeipmi [14:51] failed to log in even in the vsphere console.. [14:51] same credentials as my vsphere. [14:54] to be exact same credentials as my vsphere client on host running ESXI [14:57] shall I try my MAAS web credentials? [15:01] Did not work either. [15:08] Bug #1621507 opened: ipconfig lacks ipv6 support

[15:14] kiko, any ideas? [15:18] maas maas nodes accept-all - does not work either, command seem to be wrongly parsed [15:58] nirlevy, MAAS web credentials? [15:58] nirlevy, maybe you're confused [15:58] nirlevy, the power parameters are what MAAS uses to contact vSphere to tell it to turn a VM on [15:59] nirlevy, so they are really the credentials on the VMWare side [15:59] mpontillo, do you have any advice for nirlevy? [16:01] Hi [16:02] I usually see in the internet that to install MAAS we need at least 5 computers [16:02] but the new docs say the minimum is 2 [16:02] is that true? [16:04] kiko: nirlevy: In a call now, will read the scrollback shortly [16:04] mpontillo, I'll copy you on an email reply to him in fact, so just look out for that. it has a bit more detail [16:04] vfontanella, actually, to install OpenStack you have a minimum requirement [16:04] vfontanella, MAAS itself can run on 1 machine [16:05] vfontanella, but of course, you use MAAS to control other machines, so.. it's only really useful if you have a bunch to manage [16:05] I see [16:05] that means if I have a couple xenservers and esxi, I can install in one server and manage them [16:06] vfontanella: depends on what you want to do; I run it on a laptop ;-) (but I'm just testing it mainly) [16:06] I mean manage the VMs on it [16:07] kiko: ok thanks, please loop roaksoax in as well; I may not be able to respond quickly since I have a lot to do by the end of next week [16:07] I tried to install it in a vbox but I had problems to power on the vms [16:07] mpontillo, I think he's missing python-pyvmomi, actually? [16:08] mpontillo, I can't really believe we ship MAAS half-broken for VMWare if that's true [16:08] kiko: could be the certificate issue too [16:08] kiko: I'll take a quick look [16:08] mpontillo, thanks [16:09] mpontillo, that package needs to be installed on the rack controller, right? [16:09] kiko: yes, though the most recent version in Xenial is missing some code necessary to work around an issue with certificate validation that many people run into [16:10] kiko: so some people have had more success installing the latest version from pip [16:13] kiko: nirlevy: see my comments on this bug https://bugs.launchpad.net/maas/+bug/1593469 [16:13] mpontillo, and the latest 1.9.4 doesn't contain your workaround? [16:13] kiko: ah, correct, I think the workaround is only in 2.0 [16:13] kiko: but the version in trusty didn't have this issue ;-) [16:14] mpontillo, this user is on trusty.. [16:14] (what a tangled web this is.) [16:16] kiko: we dont ship maas broken [16:16] kiko: if python-pyvmomi is not ther emaas should be notifying the issues [16:17] kiko: there's no mpython-pyvmomi in trusty [16:17] we alwasy shipped it in a PPA [16:17] nirlevy: have you tried the "Add Hardware > Chassis" option? that usually works best with vmware instances. It's best to name all the nodes you want to enlist in MAAS with a common prefix, and use the prefix filter to ensure MAAS only configures the VMs you want to use. [16:56] kiko: any idea if someone is online that could help with my problem? === frankban is now known as frankban|afk [16:57] PCdude, let me try and help again. where are you stuck, remind me of the whole story [16:59] kiko: thats cool [17:00] kiko: so just a quick review. I have 1 controller and 5 nodes. 3 of them have 2 NIC's. 2 of those three have both NIC's connected to the private network. The last one has one connection to the public network and one to the private network [17:00] PCdude, what hardware is this? [17:10] kiko: hello [17:10] kiko: I'm bootstrapping juju agin [17:10] and I keep getting this error: http://pastebin.com/PLganGgV [17:10] neith, okay, let me know how it goes [17:10] kiko: yesterday worked like a charm [17:11] kiko: today cant bootstrap juju same error always [17:11] neith, hmm, odd. I'm otp so I'll be slow in replying [17:23] kiko: I was disconnected for a moment, so dont know if I missed something, but anyway thanks for looking at it [17:25] PCdude, what hardware is this? [17:28] so the nodes and controller are running in esxi, but the host is 16GB of corsair memory, asus H87i-plus motherboard, processor high-end of two generations back. I dont know the exact number but I could look it up [17:28] kiko: is that what u mean with the hardware? [17:32] PCdude, aha, so it's a vmware-based setup. okay [17:32] PCdude, did you use the autopilot vmdk we provide? [17:33] kiko: yes its vmware based. [17:34] kiko: no I did not use that, coz its a single machine solution. I wanted to try it all out as a copy of what I wanted to set up with real machines later on [17:35] kiko: but it should still work right? I mean, just using VMware ESXI could become problematic with WOL and that kind of stuff, but not for the nodes itself as VM [17:35] PCdude, it should work, yes, but there are many pitfalls you can encounter (as you see) [17:35] PCdude, why WoL as opposed to our native VMWare support? [17:38] kiko: well there is support in MAAS to connect directly with some API to vmware, but that is only possible with an more advanced version of ESXI. the lowest is free and that is what I use. I cant afford the more expensive ones. The problem is that ESXI does not allow API calls in my version. So I cant use the MAAS option, which would be great if I [17:38] could use it [17:38] PCdude, ah, indeed, I see the challeng [17:38] e [17:39] kiko: the next problem was that WOL does not even work with ESXI. So I installed a package on my main PC and listen to the WOL magic package. I respond to that by turning the machine on the GUI [17:39] kiko: not ideal , but for testing not bad I think [17:39] kiko: I can live with it, since when I move over to real machines WOL works and openstack can operate on its own [17:41] kiko: do the main network on wich nodes are pxe booting requires to have a GW to internet? [17:41] kiko: seems like juju wants to directly connect to the net from bootstrapped nodes [17:41] thanks mmcc [17:42] hi maas folks [17:42] PCdude, mmcc is one of the engineers responsible for openstack-install [17:42] kiko: great! :) [17:42] mmcc, PCdude has a vmware-based deployment of MAAS that is failing on the openstack-install phase [17:42] hey mmcc [17:42] hi PCdude [17:43] so how is it failing? If there's a lot of discussion in the backscroll, I'd be glad to read it if someone could paste it up for me. [17:43] http://askubuntu.com/questions/821804/openstack-with-landscape-install-fails [17:43] mmcc: most of the info is there [17:44] mmcc, and neith has a deployment that is failing like this: http://pastebin.com/PLganGgV [17:44] ok, looking now. thanks [17:44] mmcc, neith's issue I can only find reference to here: https://github.com/Ubuntu-Solutions-Engineering/openstack-installer/issues/870 [17:44] mmcc, and it seems like it's pretty reproducible [17:47] mmc should bootstrapped node be able to resolv maas server by hostname? [17:47] as the opposite? [17:53] mmcc: I have the systems still in the state of the moment after they crash, so I can look at any tiny file that can be of relevance for the troubleshooting [17:54] PCdude: so it looks like you're hitting a timeout built into the juju-deployer tool that openstack-installer uses internally. Are the systems in your MAAS slow to spin up? I don't know how using VMWare in there affects speed [17:57] mmcc: good point, tbh I dont know what I small since I have only experience with this machine. IT takes about 2-4 minutes to deploy a node from MAAS. the nodes have 1gb and 1 core. the controller has 4gb and 2 cores. an important thing to add is that some days ago I was facing another issue and with some help of kiko we find out that the juju deplo [17:57] yer was not installed by the openstack-installer, so I installed it manually. I have no idea if that is important, just saying to be complete [17:57] mmcc: I tried with more RAM on the nodes, but did not make an difference (I tried 2 and 4 GB) [17:57] mmcc, ah, yes, there was that point: juju-deployer was not installed when they installed openstack-install. is that expected? [17:59] kiko: yeah, juju-deployer is a dep of the 'openstack-landscape' package, which gets installed after running openstack-installer if you choose that option [18:00] if that wasn't happening, then something is odd with the packaging [18:00] mmcc: I choose that option, but it was not installed. by installing it manually, I got past the point it was complaining about in the past [18:01] PCdude: so is this VMs on one machine that are registered in MAAS as separate 'machines'? [18:02] I'm surprised that it didn't install juju-deployer. [18:02] yes, they are. So to be clear there is one host, which has 6 VM's on it. 5 of them are nodes and 1 is the controller. On the controller is MAAS installed and sees the other nodes (the other VM's) as seperate machines [18:03] neith: do you still have the systems sitting around in the same state? I'm curious if you can interact with juju successfully outside of the installer. for example, "JUJU_HOME=~/.cloud-install/juju juju status" [18:03] setting JUJU_HOME is necessary because we use a separate directory to avoid affecting an existing environment [18:04] mmcc: The machine has 16GB of corsair memory and ASUS h87i-plus motherboard and a high end processor (dont know the number exactly ) [18:06] ok [18:06] PCdude: if you try the same "JUJU_HOME=~/.cloud-install/juju juju status" command, what does it tell you? [18:07] it's possible that has errors that juju-deployer doesn't relay back to the user, so it just waits until a timeout [18:09] mmcc: It can't even reach this state now. I'm trying to replay openstal-install [18:09] pastebin? [18:09] mmcc: before I run openstack-install -u and it gets: Error: an inet prefix is expected rather than "False". [18:10] mmcc: http://pastebin.com/raw/AZY2ZHiV [18:10] neith: you can ignore that error. unfortunately the uninstall script is loud if it's tearing down an incomplete install [18:12] mmcc: Alright, running it again [18:14] PCdude: so it is trying to spin up one machine with four LXC containers on it. the containers are stuck in a 'pending' state [18:15] the next thing I'd do is to dig around on that machine, via "JUJU_HOME=~/.cloud-install/juju juju ssh 0" [18:16] and then once I'm on machine 0, look at the lxc containers on there and see if there are any errors that look obvious [18:17] mmcc: well I have one controller and one nodes is used for MAAS. I have not seen one machine ever startup to do that. So I cant login to that extra machine and try those commands [18:18] PCdude: your juju status says that 'node1.maas' was brought up as a juju machine [18:19] mmcc: ah ok, yes that is correct. I will issue the command on that machine. I thought there should spin up another one than node1 [18:20] PCdude: no, the landscape install puts its services into LXC containers on a single machine [18:23] mmcc: I have never used LXC before, so what command should I use to check that? [18:23] mmcc: one note is that version 1.0.8 is installed of lxc, while 2.0.3 is the recommended version on trusty (source:apt-cache) [18:23] sudo lxc-ls --fancy [18:24] PCdude: openstack-installer is glue over MAAS, Juju, and LXC. You can see the bundle that we deploy here: https://jujucharms.com/u/landscape/landscape-dense-maas - I think the way forward for you is to verify that the underlying tech works in your environment before trying again with openstack-installer. e.g. make sure that you can bring up lxc containers inside your VMs, then try deploying a juju bundle that uses lxc containers onto [18:24] your MAAS. [18:25] http://pastebin.com/raw/Lh1uTDQt [18:25] many issues with running openstack-installer end up being better answered by the communities around those component technologies [18:25] mmcc: that is the output of "sudo lxc-ls --fancy" [18:25] mmcc: ah ok, I am really learning stuff here :) [18:26] so it sounds like there's an issue bringing up lxc containers with juju on your environment. I'd suggest asking in #juju for expert advice on where to look for that [18:26] I've dug in there in the past, but I'd probably have to ask there myself, it's been long enough [18:27] mmcc: kiko : sorry for my previous questions, I cant trust the network guys, they messed up the cables again. Though I understand its hard to manage nodes with 8 physical interfaces without mistakes [18:27] neith: so you're in good shape now? that's good to hear [18:28] mmcc: I'am wondering if we could improve maas by checking if nodes using the same subnets are really able to talk to each other [18:28] mmcc: good I will head over to the JUJU channel [18:29] avoiding loops and cross connections [18:35] neith, that does sound useful to me, but I'm not a maas dev. [18:37] if no one else sees it here, I'd file a feature request on https://bugs.launchpad.net/maas/+filebug [18:37] neith, mmcc: what's what mpontillo is already working on for release 2.1 [18:37] neith, a bit miffed that that was the problem in the end -- the failure mode is really obscure [18:39] kiko: mmcc well, there happens to be a know issue with bringing up instances. somehwere along the line apt-get update and apt-get upgrade fail. Disabling this option would make the installer to succeed [18:40] PCdude, why is the upgrade failing though? [18:40] PCdude, network flakiness? [18:41] kiko: dont know yet, still busy in JUJU channel [18:42] kiko: mmcc https://lists.ubuntu.com/archives/juju/2016-September/007845.html [18:43] also good to know for other problems in the future with this version I think [18:44] this is for Xenial, but maybe for trusty too lets see [19:14] PCdude, I'm thinking you are continue to be in pain in your current setup [19:14] PCdude, maybe you should pivot into getting some hardware, or using KVM instead of VMware? [19:16] kiko: Well, I have made a pretty precise plan in what I want to do with openstack. Though, I have some blind spots on understanding some parts. Thats why wanted to try and really reproduce the acutal usage situation. Since the test fase on ESXI will eventually gives me a clue on what hardware I will need. [19:17] kiko: I have tried KVM, but I ran in some issues too with that. I cant remember what it was. [19:17] PCdude, I think it fails for a few reasons, potentially the first that WoL isn't all that reliable [19:19] kiko: ah ok, yeah some other people told me that too. I was thinking about maybe buying some serious equipment (second handed of course) to do the trick. [19:20] PCdude, my overall guidance would be to prefer hardware if possible, and if not, to use KVM, which will be less magical than VMware [19:22] kiko: I think checking if MAAS nodes can talk to *each other* requires software running on the deployed node; that is actually out of scope. the network discovery for 2.1 gives MAAS better information about which nodes have been seen on subnets directly attached to MAAS servers. [19:22] kiko: good point and I agree for sure. yeah I tried KVM probably an hour after I find out about openstack. So maybe there is part of the problem what I had with KVM. [19:27] mpontillo, yes, in 2.1 you get the first step of that, which is can the nodes talk to the networks MAAS knows about [19:27] mpontillo, I assume it's actually "knows about", not necessarily "manages"? [19:28] kiko: when it comes to interactions with juju, spaces are presumed to be the model of which networks can talk to each other in MAAS. that is something that needs to be defined by the MAAS admin though; MAAS does not yet try to determine that automatically, even in 2.1 [19:31] do you guys run maas on ubuntu 16.04 ? [19:31] mpontillo, that's okay, I'm just saying that if the user has his networks listed in MAAS, and the cluster controllers are attached to them, we can check if the nodes talk to them [19:31] kv, sure! [19:32] any tips or tricks? I am having quite a few issues with it [19:32] kiko: yes, that much is certainly true. [19:34] kv, first, upgrade to 2.0 final. :-) [19:34] kv, second, tell us all about your issues [19:34] mpontillo, that is already worlds better than where we are today [19:36] kilo any good docs for 16.04 ? [19:37] kilo, i had lots of issue with the dhcpd [19:38] kv, http://maas.io/docs covers 2.0 [19:38] kv, what exact version are you running? [20:08] Bug #1621610 opened: [2.0,2.1] MAAS allow's machine to transition to 'commissioning' even if no images are present [20:16] kiko: mmcc it turns out NOT to be JUJU haha [20:17] as said before the LXD containers do not get an IP address and this should relate to either MAAS in failing of giving it or LXD by not providing it right [20:17] PCdude, it rarely is, unfortunately. most problems are in the MAAS domain :-) [20:18] kiko: haha ok, stokachu suggested that maybe MAAS ran out of IP addresses [20:19] both static and dynamic now have 100 addresses each for DHCP requests (50 before that) [20:21] kiko: How can I diagnose the MAAS DHCP settings in a CLI window? [20:40] kiko: /etc/dhcp/dhcpd.conf is pretty empty [20:43] PCdude, I think they are in /etc/maas or /var/lib/maas [20:46] kiko: ah yes of course that makes sense [20:56] Sep 8 13:51:53 sj36 maas.dhcp: [ERROR] Could not rewrite DHCPv4 server configuration (for network interfaces enp1s0f0): Command `maas-rack atomic-write --filename /var/lib/maas/dhcpd.conf --mode 0644` returned non-zero exit status 1:#012None [20:56] kilo any idea why I keep getting that error? [21:12] well good night everyone :) [21:23] kv: what's the permissions under /var/lib/maas/ ? [21:26] kv, does /var/lib/maas exist? :) [21:26] PCdude, happy you've made progress. we should do better at showing you when you're running out of IPs.. [21:28] kilo drwxr-xr-x 5 maas maas 4.0K Sep 8 14:19 maas [21:41] Bug #1621615 opened: network not configured when ipv6 netbooted into cloud-init

[21:41] Bug #1621635 opened: [2.0, 2.1, UX] MAAS doesn't warn the user that MAAS it not managing DHCP when attempting to commission [21:50] Bug #1621647 opened: MAAS 2.0 SSL verification error when adding UCSM chassis [22:08] Bug #1621647 changed: MAAS 2.0 SSL verification error when adding UCSM chassis [22:11] Bug #1621647 opened: MAAS 2.0 SSL verification error when adding UCSM chassis [22:40] oh boy...777 on /var/lib/maas and /var/lib/maas/dhcp still no luck [22:44] everything seems fast...and promising but dhcp ... [22:45] kv: what happens if you run that command manually ? [22:45] maas-rack atomic-write --filename /var/lib/maas/dhcpd.conf --mode 0644 [22:46] kv: also, how did you enable DHCP ? [22:46] kv: are you running under a vm? lxc container ? may there be something preventing thigs to work ? [22:46] like apparmosr ? [22:46] i ran that command from root and it did not do anything [22:47] i enabled using the ui [22:47] i also tried from cli [22:52] i think i found the issue [22:53] it was related to sudo on maas account [22:54] kv: interesting, there's no sudoers file ? [22:54] maas should have installed the sudoers file [22:55] i think it did but my puppet module removed it :) [22:56] ah! [22:57] i added 'maas ALL=(ALL) NOPASSWD: ALL' and it is happy now [22:59] kv: cool [23:01] thanks roaksoax [23:09] dhcp is working...tftp got timeout when booting from pxe...new host was not registered in maas ui. [23:09] any idea roaksoax ? [23:11] kv: logs I can look at ? [23:12] kv: rackd.log or a console log [23:13] pxe-e32: tftp open timeout [23:13] i don't see anything in rackd.log [23:15] weird tftpd isn't installed by default? [23:19] kv: maas run its own tft server [23:19] kv: check /var/log/maas/rackd.log [23:20] kv: can the machine reach the tftp address it is told to ? [23:21] bingo [23:21] looking good