[04:17] <mup> Bug #1709095 changed: Commissioning fails when cloud init can't find a data source <MAAS:Expired> <https://launchpad.net/bugs/1709095>
[04:26] <mup> Bug #1709095 opened: Commissioning fails when cloud init can't find a data source <MAAS:Expired> <https://launchpad.net/bugs/1709095>
[04:35] <mup> Bug #1709095 changed: Commissioning fails when cloud init can't find a data source <MAAS:Expired> <https://launchpad.net/bugs/1709095>
[08:36] <mup> Bug #1733285 opened: Unable to create pod with MAAS 2.3-rc2 on 17.10 <s390x> <MAAS:New> <https://launchpad.net/bugs/1733285>
[08:48] <mup> Bug #1733285 changed: Unable to create pod with MAAS 2.3-rc2 on 17.10 <s390x> <MAAS:New> <https://launchpad.net/bugs/1733285>
[08:51] <mup> Bug #1733285 opened: Unable to create pod with MAAS 2.3-rc2 on 17.10 <s390x> <MAAS:New> <https://launchpad.net/bugs/1733285>
[15:33] <xygnal> roaksoax: is the centos6 images in the maas repo currently broken?  curtain fails trying to set grub root on our device.
[15:34] <xygnal> have a traceback but it doenst explain a whole lot
[15:37] <xygnal> it need sto set root to sdm, as the RAID1 is showing as final disk not first disk (We have jbods too)
[15:45] <roaksoax> xygnal: not to my knowledge. we've not made any changes other than getting the latest centos6 images
[15:45] <roaksoax> xygnal: also, we dont support efi on centos6 if that's what you were using
[15:46] <roaksoax> since at the time of enablement,what was needed for efi wasn't available in centos6 iirc
[15:52] <xygnal> roaksoax: we are running BIOS only on all of our stuff.  I was just asking about EFI in case it might helkp our boot situation.
[15:52] <xygnal> roaksoax: we are using th strait vanilla image you host for centos7
[15:52] <xygnal> er centos6*
[15:52] <xygnal> typo
[15:52] <xygnal> and it fails with that
[15:52] <xygnal> its not installing on /dev/sda which is the only reason I can imagine its having a problem
[15:52] <[Kid]> is there a way to use hostnames with juju and MAAS to deploy certain machine numbers to certain nodes by hostname?
[15:52] <xygnal> it shows root and boot selection for /dev/sdm but it fails with that
[15:53] <[Kid]> hopefully that makes sense
[15:55] <xygnal> roaksoax: i am booting  centos6 iso now to see what order the disks select in. could be taht /dev/sdm is not the same once in the OS?
[15:55] <roaksoax> xygnal: for centos maas picks the first disk the os finds and that's where it installs
[15:55] <xygnal> hm but no that should still be in the ubuntu install image at that point
[15:56] <xygnal> roaksoax: we have code in our curtin preseed file that uses the boot device
[15:56] <xygnal> roaksoax: you helped us to code that actually to work around it
[15:56] <roaksoax> [Kid]: I think juju /may/ have a way to use a constraints based on machine name. It used to have, I haven't used juju in a while
[15:56] <xygnal> but it seems to only be working in centos7
[15:57] <[Kid]> according to the guys in #juju tags are the only way
[15:57] <roaksoax> xygnal: strange. Both centos6/7 images process it exactly the same
[15:57] <[Kid]> not a huge deal
[15:57] <roaksoax> xygnal: the only thing I could imagine is a different version of software inside centos may be the cause ?
[15:57] <roaksoax> xygnal: can you file a bug ?
[16:02] <xygnal> roaksoax: hold on.  I just disccovered he is using centos not custom, and our curtin code is only in custom
[16:02] <xygnal> so need to look at this first and make sure its being used correctly
[16:06] <roaksoax> xygnal: ack, ah that's it then
[16:07] <roaksoax> xygnal: it needs to be custom
[16:21] <xygnal> roaksoax: just tried my custom stuff in the centos file, still fails.  I have debug output for you.
[16:22] <xygnal> roaksoax: pasted in private chat
[16:22] <xygnal> roaksoax: does ubuntu curtin not support the same options as custom?
[16:22] <xygnal> er
[16:22] <xygnal> sorry
[16:22] <xygnal> why do i keep using ubuntu?
[16:22] <xygnal> brain not working
[16:23] <xygnal> Can curtin_userdata_centos  support the same options as curtin_userdata_custom
[16:24] <xygnal> i never used the ubuntu image - just brain tired. i meant the centos image vs the custom image. we have been using centos and centos preseed file to try to replicat the same results
[16:39] <xygnal> roaksoax: export the image, imported it as custom image
[16:39] <xygnal> tried again
[16:39] <xygnal> same failure, same error
[16:52] <roaksoax> xygnal: images under "centos" are not sent the storage config, so curtin doesn't do the partition
[16:52] <roaksoax> xygnal: custom images do get it
[16:52] <roaksoax> xygnal: that's probably why things fail
[16:53] <xygnal> roaksoax: well i just did custom and it failed exactly the same way
[16:53] <xygnal> so thats not it
[16:57] <xygnal> roaksoax: do you want to see the full trace output on that like i did for the previous one?
[16:58] <xygnal> at this point im considering writing a manual endpoint to turn jbod on/off before/after deployment manually
[16:58] <xygnal> because its the only way MAAS can cope with centos6 on our hardware
[17:40] <roaksoax> xygnal: does centos6 work with jbods ?
[17:40] <roaksoax> with your jbods ?
[17:48] <correct> if I don't get the paid version of MAAS, this means I can't provision Windows systems?
[18:05] <kiko> correct, you can but you need to roll your own images, essentially
[18:05] <kiko> the image build service is commercial
[18:05] <kiko> so perhaps s/correct/partially correct/
[18:20] <rick_h> howdy folks, I've got a long running maas on a single laptop that I've upgraded today to rc-2 and I'm getting the "One rack controller is not yet connected to the region. Visit the rack controllers page for more information."
[18:20] <rick_h> I've restarted the machine. As it's a single laptop it's both region/rack controller and I'm not up on what the new UX is trying to tell me here. Any hints on what's been changing?
[18:26] <rick_h> checking logs: https://pastebin.canonical.com/203665/
[18:28] <roaksoax> rick_h: the rack controller cannot connect to the region for some reason
[18:29] <roaksoax> rick_h: pb both regiond.log and rackd.log
[18:29] <rick_h> k, checking
[18:30] <correct> kiko: lol.
[18:31] <rick_h> roaksoax: oh hmm, it's trying to connect ipv6
[18:31] <correct> kiko: I'm rollin'em already with cobbler.  Excited to test out MAAS.
[18:32] <correct> kiko: so.. when say roll my own, typycally, I have cobbler use ghost to deploy the image... with MAAS does it work in a similar fashion?
[18:33] <rick_h> roaksoax: so I've got the 10.0.0.1 in both rackd and regiond.conf files. Logs show: Region not available: User timeout caused connection failure. (While requesting RPC info at b'http://[::ffff:10.0.0.1]/MAAS/rpc/').
[18:33] <correct> as I am using a ghost image, does MAAS offer some other ways of deploying my custom image?
[18:33] <rick_h> roaksoax: is it pulling from something other than the maas_url in each?
[18:36] <roaksoax> rick_h: is maas_url pointing to 10.0.0.1/MAAS or 10.0.0.1:5240/MAAS ?
[18:37] <rick_h> roaksoax: in the rackd it's sans-port, in the regiond it's with port
[18:37] <roaksoax> rick_h: use :5240, maas opens port on 5240, not on other ports
[18:42] <kiko> correct, no, MAAS uses curtin -- see https://readthedocs.org/projects/curtin/
[18:42] <kiko> correct, sorry, I'm OTP so very slow today
[18:42] <kiko> shucks
[18:42] <kiko> correct, http://curtin.readthedocs.io/en/latest/ is what I really meant
[18:46] <xygnal> darn
[18:46] <xygnal> TJ's suspicion dint pan out
[18:46] <xygnal> after extracting a failed/stalled boot
[18:46] <xygnal> the checksums on the initrd and kernel match :(
[18:46] <xygnal> why is it stalling?!
[18:50] <correct> kiko,.. while installing MAAS, I got an error regarding psql
[18:51] <correct> I'm gussing server have not started
[19:04] <xygnal> you are are correct
[19:04] <correct> It appears there is a problem with the bootstrap install?
[19:05] <xygnal> you are are correct
[19:05] <correct> xygnal: umm... is there a post installation script for the db?
[19:05] <xygnal> i'm not answering your questions i'm just saying your name ;)
[19:05] <correct> lol
[19:06] <xygnal> i dont know what your problem is, i am in here as i have my own problems
[19:07] <correct> basically..I'm installing MAAS.  I am at a section that configures postgressdb for MAAS
[19:07] <correct> it erors out.
[19:07] <correct> as it can't connect to DB
[19:07] <roaksoax> rick_h: /win 5
[19:07] <roaksoax> err
[19:07] <rick_h> wheee
[19:09] <correct> I get this https://snag.gy/osnOIl.jpg
[19:11] <rick_h> roaksoax: hmm, seems because it's long running I'm having fun with old e/n/i vs new interface names/etc and that's causing me to have fun. Thanks for helping me poke at the logs/etc
[19:13] <correct> lol.. this is not working.  i'm not sure what to do here.
[19:15] <correct> This question may not have been addressed: https://askubuntu.com/questions/965303/unable-to-install-maas-directly-using-dvd
[19:16] <xygnal> no fun -o
[19:16] <xygnal> =p
[19:17] <heyya> If I skip the DB setup.. is can't it be configured later?
[19:18] <heyya> or do I need to do a layered install?
[19:18] <heyya> *can the postgre db be configured after ?  if so, how so?
[19:19] <xygnal> if i had to take a guess heyya i would say, it needs to populate the database during install
[19:19] <heyya> xygnal, yeah.. but this install has been mostly unattended
[19:20] <xygnal> possible to spin up a database in advance, even if you dont keep it?
[19:20] <heyya> I saw a posting mention that I need to layer the install
[19:20] <heyya> like install ubuntu server with ssh, then maas controller, etc
[19:21] <heyya> ubuntu server with postgresql
[19:21] <heyya> I need to look at installer script
[19:22] <heyya> the installer is searching for a DB that does not exist
[19:22] <xygnal> TJ-: wireshark can decode the TFTP files! still trying to get a client-end capture though
[19:22] <heyya> crap.. just don't wanna do this again
[19:22] <xygnal> i thoguht it spun up the db for you
[19:23] <heyya> xygnal: that's what I thought, did you see the posting ? ---> https://askubuntu.com/questions/965303/unable-to-install-maas-directly-using-dvd
[19:24] <TJ-> xygnal: you still at that!?
[19:26] <xygnal> TJ-: well that was friday so i only just picked it up agian today.
[19:26] <xygnal> TJ-: I was able to confirm the whole TFTP image goes throguh, but without the client capture i can't verify the sum
[19:26] <TJ-> xygnal: seems like a month ago to me
[19:27] <xygnal> TJ-: it's new infra so we dont have spans setup yet. tough spot.
[19:27] <TJ-> xygnal: right; did you try with bootstrapping via iPXE
[19:27] <xygnal> TJ-: no, woudln't help us since MAAS doesn support it yet
[19:28] <xygnal> or are you implkying ipxe client can tell me if the checksum was good?
[19:28] <xygnal> wouldn't it still be force to use old PXE protocol because of server?
[19:28] <xygnal> you wanted me to spin up a whole second pxe server to test in there and that seems a bit overkill
[19:29] <TJ-> xygnal: As I said last week - if you booted the hardware from an iPXE CD-ROM/USB, so it then tries to do a PXE boot, and it works all 50 times (or however many you test for), it'd indicate the problem could be in the system firmware's PXE implementation
[19:30] <TJ-> xygnal: but if 1 or more of those tests fail you know there's an intermittent network issue
[19:33] <xygnal> TJ-: only happens in networks with switches that have ACI
[19:33] <xygnal> TJ-: same hardware. so we suspect not firmware.
[19:34] <heyya> isn't that ACL ?
[19:36] <xygnal> nope
[19:36] <TJ-> xygnal: So you're sure the issue is the network?
[19:36] <xygnal> ACI. cisco switch technology.
[19:36] <xygnal> TJ-: 99% certain, but i just found a cisco ACI bug release saying that it causes UDP/TFTP traffic to fail
[19:36] <xygnal> so crossing my fingers we aren't at taht ACI patch level
[19:37] <TJ-> xygnal: so, even if you prove a packet goes astray/is corrupted, you still won't know why or where it happens
[19:37] <TJ-> xygnal: oh really? I bet you are!
[19:37] <TJ-> xygnal: what's the bug reference?
[19:37] <heyya> Do you put all the MAAS components on one node?
[19:38] <TJ-> xygnal: 'fail' or 'drop packets' - the former suggests the connection fails.
[19:38] <xygnal> CSCaz88813
[19:38] <xygnal> uz*
[19:38] <xygnal> not az
[19:39] <xygnal> "bridge packets dropped on network"
[19:41] <xygnal> find it?
[19:47] <TJ-> xygnal: that does look like it, but if it is down to a specific byte sequence, you'd expect the boot to fail the same way every time, surely? (Because its always the same data and presumably packet headers ) - Or does the server get a different IP each time/some times ?
[19:47] <xygnal> xygnal: yes it does get different IPs.  There are commissions and deployments.
[19:48] <xygnal> they fail so re-trying grabs a new IP
[19:48] <xygnal> the commissions at the least
[19:48] <TJ-> xygnal: if it is that bug you can test it easily. Get the server booted successfully one-time, then run a UDP stream to it and compare what's sent to what's received. That way you have full control on the server-side
[19:49] <TJ-> xygnal: in other words; no need to try debugging TFTP, just debug a continuous UDP stream
[19:51] <TJ-> xygnal: you can use netcat (nc) either side, e.g: "cat /some/large/text/file | nc -4u server $testport" ->>> "nc -4ul $testport | tee /tmp/received/file"
[20:00] <xygnal> TJ-: so a netcat test?
[20:01] <xygnal> damn you internet
[20:21] <TJ-> xygnal: I think we both lost connection! did you see my netcat example?
[20:21] <TJ-> 19:51:36    TJ- | xygnal: you can use netcat (nc) either side, e.g: "cat /some/large/text/file | nc -4u server $testport" ->>> "nc -4ul $testport | tee /tmp/received/file"
[20:22] <roaksoax> heyya: not from the cd. The bug is fixed but dind't make it into the CD
[20:22] <roaksoax> heyya: also, you can install directly after ytou install from cd
[20:22] <xygnal> TJ-: we are at 2.3 fw level so, oh well
[20:22] <xygnal> can't be the cisco bug
[20:23] <TJ-> xygnal: no harm doing the UDP streaming test though.
[20:23] <xygnal> oh yes I have to in order to prove the network that its not my problem
[20:23] <xygnal> thats next
[20:49] <mup> Bug #1733411 opened: MAAS should create static leases for ephemeral boots when possible <MAAS:Triaged> <https://launchpad.net/bugs/1733411>
[21:19] <mup> Bug #1733420 opened: virsh power type check error <MAAS:Incomplete> <https://launchpad.net/bugs/1733420>
[21:38] <heyya> roaksoax: I just did a fresh install of ubuntu server with dns,postgres and ssh.  Server is up
[21:38] <heyya> roaksoax: apt-cache has the package, but I was reading I should add ppa:mass/stable
[21:41] <heyya> anyway.. will install use the ppa
[22:14] <roaksoax> heyya: the ppa wont gfive you anything different from the archives
[22:14] <roaksoax> heyya: unless you want something newer
[22:14] <roaksoax> heyya: maas 2.3 goes out officially tomorrow and 2.2 will be replaced in PPA
[22:14] <roaksoax> so if you want the latest ppa is probably the best
[22:15] <heyya> sweet.. did it.. FYI MAAS is SOOOO nice
[22:15] <roaksoax> glad to hear that :)
[22:15] <heyya> very mature from what I can see
[22:16] <heyya> I have dhcp funning already.. I guess I can configure my dhcp to use the pxe from MAAS
[22:18] <roaksoax> yu can, yes
[22:33] <heyya> not sure how I want to set it up yet.  With MAAS, I'll have 2 pxe servers running.
[22:33] <heyya> I guess I'll have to disable the other pxe while I'm testing
[22:34] <heyya> or restict it.
[22:34] <heyya> hmm.. ok.. I'll lock it down to a network
[23:11] <heyya> What are DHCP snippets in Maas?  does that indicate the network for serving dhcp?
[23:27] <heyya> When i dig @MAAS i get WARNING: recursion requested but not available
[23:53] <mup> Bug #1733442 opened: Cannot delete Virsh Pod <MAAS:In Progress by newell-jensen> <https://launchpad.net/bugs/1733442>
[23:56] <mup> Bug #1733442 changed: Cannot delete Virsh Pod <MAAS:In Progress by newell-jensen> <https://launchpad.net/bugs/1733442>