/srv/irclogs.ubuntu.com/2016/12/12/#juju.txt

=== BlackDex_ is now known as BlackDex
=== frankban|afk is now known as frankban
kjackalGood morning Juju world!08:30
=== jamespag` is now known as jamespage
=== petevg_afk is now known as petevg
Teranethi everyone do we have any JUJU people here ?17:08
bdxTeranet: how's it going?17:12
Teranetgood except my JUJU is not doing what is should do17:14
bdxTeranet: what does `juju --version` return?17:15
Teranetjuju bootstrap     localhost  fails17:15
Teranet2.0.2-xenial-amd6417:15
TeranetErrror is : ERROR failed to bootstrap model: waited for 20m0s without being able to connect: ssh: connect to host17:15
bdxTeranet: ok, have you verified that lxd is configured correctly on your machine?17:15
Teranetit get's an IP assigned but the container when I see it don't get the IP17:16
bdxTeranet: can you launch a lxd instnace manually?17:16
bdxTeranet: `lxc launch ubuntu:16.04 ubuntu-test`17:16
TeranetLXD is running correctly and the the bridge is in place17:16
Teranetok let me try17:16
Teranetsysadmin@sf2-maas00:~$ lxc launch ubuntu:16.04 ubuntu-test17:17
TeranetCreating ubuntu-test17:17
TeranetStarting ubuntu-test17:17
Teranet 17:17
Teranetlooks like it just did17:17
bdxTeranet: ok, can you try this command for bootstrapping lxd local `juju bootstrap lxd lxd-dev`17:18
bdxTeranet: you can remove that test container with `lxd delete ubuntu-test --force`17:18
Teranetok will run it give it a sec17:20
Teranetok same result :17:20
Teranetsysadmin@sf2-maas00:~$ juju bootstrap lxd lxd-dev17:20
TeranetCreating Juju controller "lxd-dev" on lxd/localhost17:20
TeranetLooking for packaged Juju agent version 2.0.2 for amd6417:20
TeranetTo configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md17:20
TeranetLaunching controller instance(s) on lxd/localhost...17:20
Teranet - juju-141282-0 (arch=amd64)17:20
TeranetFetching Juju GUI 2.2.517:20
TeranetWaiting for address17:20
TeranetAttempting to connect to 10.5.100.53:2217:20
Teranetit's like not assigning the IP to the VM17:20
Teranet12: vethNL6O8E@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 100017:21
Teranet    link/ether fe:88:68:61:5d:72 brd ff:ff:ff:ff:ff:ff link-netnsid 017:21
Teranet    inet6 fe80::fc88:68ff:fe61:5d72/64 scope link17:21
Teranet       valid_lft forever preferred_lft forever17:21
Teranet14: vethMBJXX9@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 100017:21
Teranet    link/ether fe:f1:f5:1e:81:dc brd ff:ff:ff:ff:ff:ff link-netnsid 117:21
Teranet    inet6 fe80::fcf1:f5ff:fe1e:81dc/64 scope link17:21
Teranet       valid_lft forever preferred_lft forever17:21
TeranetI do see IPv6 but no IPv4 Ip's assigned17:21
bdxTeranet: did your ubuntu-test container get an ip?17:21
bdxTeranet: it sounds like you might need to reconfigure your lxd-bridge17:22
bdx`lxc list | grep ubuntu-test`17:22
Teranetnope17:23
bdxahh, thats the issue17:23
Teranethold on17:23
Teranetsysadmin@sf2-maas00:/etc/dhcp$ lxc list | grep ubuntu-test17:24
Teranet| ubuntu-test   | RUNNING | 10.5.100.68 (eth0) |      | PERSISTENT | 0         |17:24
bdxahh, ook17:24
Teranetthe ubuntu test yes17:24
Teranetstupid eth017:24
Teranetthat shit has to be em1 grrr17:24
Teranetmy Server run's em1 and em2 as interfaces not eth017:25
bdxTeranet: are you using lxdbr0 for your lxd bridge?17:25
Teranetlet me verify17:25
Teranet4: lxdbr0:17:25
bdxTeranet: `cat /etc/default/lxd-bridge`17:26
bdxthat should produce something similar to -> http://paste.ubuntu.com/23619976/17:27
bdxfor your ip space17:27
bdxthere are some critical configs in that file for the lxd-bridge17:27
Teranetit did http://paste.ubuntu.com/23619977/17:28
TeranetIp's been assigned but SSH times out :-(17:28
bdxoooh17:28
bdxTeranet: its just being timely I think17:28
bdxTeranet: is this running on a laptop with spinning disk?17:29
Teranetnope17:29
Teranetrun's on a big DELL Server with SSD's17:29
bdxwell, ok then17:29
bdxha17:29
bdxTeranet: has your bootstrap finished?17:30
Teranetnope it's hanging17:30
Teranethttp://paste.ubuntu.com/23619986/17:30
bdxTeranet: if/when you bootstrap again, it might be helpful to pass the --debug flag (my bad for not getting you that with the command initially)17:30
Teranetlike this17:30
Teranetis ok let me redo it17:31
bdxTeranet: I see the problem17:32
bdxyour gateway17:33
Teranetwhy so ?17:33
bdx## IPv4 network (e.g. 10.0.8.0/24)17:33
bdxLXD_IPV4_NETWORK="10.5.100.50/24"17:33
Teranetsysadmin@sf2-maas00:/etc/dhcp$ ip r17:34
Teranetdefault via 10.5.100.1 dev em1 onlink17:34
Teranet10.5.100.0/24 dev em1  proto kernel  scope link  src 10.5.100.25017:34
Teranet10.5.100.0/24 dev lxdbr0  proto kernel  scope link  src 10.5.100.5017:34
rick_hy\17:34
Teranetthere should be no issue17:34
bdxTeranet: Your not natting ivpv417:35
bdxehh17:35
bdxipv417:35
bdxTeranet: `sudo dpkg-reconfigure -p medium lxd`17:35
Teranetcorrect why should I it's the same network17:36
jrwrenyou'd need something to bridge those two interfaces.17:36
bdxTeranet because the container will not know how to talk to the internet17:36
jrwrenthose are not the same network (unless there is other config). Those are 2 networks that happen to use the same address space.17:36
Teranetcorrect on purpose17:37
bdxjrwren: nice catch17:37
Teranetbecause I don't want LXD to use a different network17:37
bdxTeranet: then you need to create the bridge on the interface17:37
Teranetwhich is17:38
jrwrenTeranet: in that case, you'll need to create your own bridge and add em1 to it and tell lxd to use it instead of lxdbr017:38
Teranet4: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 100017:38
Teranet    link/ether fe:55:75:16:41:9a brd ff:ff:ff:ff:ff:ff17:38
Teranet    inet 10.5.100.50/24 scope global lxdbr017:38
Teranet       valid_lft forever preferred_lft forever17:38
Teranet    inet6 fe80::84ba:caff:feba:ff95/64 scope link17:38
Teranet       valid_lft forever preferred_lft forever17:38
jrwrenTeranet: http://jrwren.wrenfam.com/blog/2015/11/10/converting-eth0-to-br0-and-getting-all-your-lxc-or-lxd-onto-your-lan/17:38
Teranethttp://paste.ubuntu.com/23620012/17:40
Teranettake a look17:40
bdxTeranet: em1 needs to be in  'manual' mode17:42
Teranetok let me fix that17:42
Teranetdone now reboot again ?17:43
bdxTeranet: did you add lxdbr0 to em1 similar to jrwren's blog post?17:46
jrwreni never reboot.17:46
Teranetlxd changes on network's won't go into effect without a hard reboot17:47
jrwrenifup containerbr should be enough17:47
jrwrenThat has not been my experience.17:47
Teranetthat's a fact17:47
jrwreni've no wish to argue.17:47
bdxTeranet: you can use brctl, and ifup/ifdown | ifconfig to accomplish these tasks w/o restarting17:48
Teranetbdx not under newer ubuntu setup's just fyi17:49
bdxTeranet: really ?17:49
Teranetyes it's an open issue with Kernel 4 since they pushed it out and the fix is in the works17:50
bdxTeranet: `sudo ifdown em1 && sudo ifup em1` doesn't do the trick?17:50
Teranetsome stupid register is not proper set which very often lets the interface restart hang or just emulate the restart but actually not doing a thing17:50
bdxcrazy17:50
jrwrenterrible! What kernel version? I want to be sure to avoid it.17:51
Teranetyes it's a known bug in the 4.4 series17:51
Teranetit's still in the latest 4.4.0-3517:51
Teranetwhich I use right now17:51
Teranetok box is back up17:52
Teranetstupid Dell init takes way to long :-(17:52
Teranetbut wait till you work on Supermicro boxes :-(17:53
Teranet10min reboot or longer even17:53
jrwrensomehow I'm on 4.4.0-45-generic, Wed Oct 19 14:12:37 UTC 2016 x86_64. Are you on x86_64?17:54
TeranetLinux sf2-maas00 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux17:54
jrwrenoh, -35 v. -53. a bit of discalulia17:55
jrwrendiscalculia17:55
TeranetLOL must had patched over night the box17:55
Teranetsorry we have over 15000 Server here17:56
Teranetoften I get lost whats where17:56
Teranetstill looks to hang at the same crap ;-(17:58
bdxTeranet: so now you need to unconfigure the lxd bridge from lxd's perspective17:59
=== frankban is now known as frankban|afk
bdxTeranet: `sudo dpkg-reconfigure -p medium lxd`17:59
TeranetI think I like to run this without a stupid bridge mode at all if possible17:59
jrwrenbridge is the only way to not have LXD use a different network. You can use the defaults and use routing and NAT, or you can use bridge, but I don't know of a way to use networking with LXD without one of those two things.18:01
Teraneti think I might than have to yell at ubuntu again18:02
TeranetLOL18:02
bdxTeranet: all it takes is -> http://paste.ubuntu.com/23620112/18:03
Teranetok let me try with NAT18:04
Teranetmight solve the crap18:04
bdxTeranet: ^ is direct bridging18:04
Teraneteven I don't like the NAT in a Corp network18:04
bdxTeranet: ^ isn't nat18:04
TeranetI do have to because we have around 300 networks and within those I must be able to reach each individual LXD when needed18:05
Teranetso if network is not proper we have a bigger issue18:05
jrwrenbdx: if you bridge em1 onto lxdbr0 then the lxd-net service will provide DHCP service out that em1 interface. If you are plugged onto a LAN, you might make your neighbors angry.18:06
Teranetwe use a dedicated Management LAN and IPAM I reserved those blocks18:07
Teranetso they won't collide18:07
TeranetMAAS DHCP is 180-199 right now18:07
Teranetand LDX only 51-6918:08
Teranetso they won't collide in this test setup18:08
bdxjrwren: not if lxd's networking is disabled18:08
bdxjrwren: http://paste.ubuntu.com/23620112/ works for me when I have external dhcp and I want to extend that to my containers on my host18:09
bdxwhich is what I feel like Teranet is trying to do18:09
bdxTeranet: is that correct?18:09
jrwrenbdx: ah, nice.18:09
Teranetlet me check18:10
TeranetI just adjusted to your settings bdx lets give me a sec18:12
TeranetFYI my MAAS Cluster setup uses the em1 on all boxes as Management and em2 is the public TRUNK with 30 Vlan's later on which is the public facing network ranges so to speak18:13
bdxTeranet: I must warn you, if this is a node in your MAAS, you are best off bootstrapping Juju to your maas, then letting maas configure the networks, and deploy containers ontop of them with juju using the maas provider18:19
Teranethow would that be different ?18:19
Teranetshouldn't that be the same procedure ?18:19
bdxTeranet: trying to manually mangle MAAS provisioned interfaces will only bring you grief .... does this make sense?18:19
bdxTeranet: MASS has built in container networking18:20
bdxTeranet: If you bootstrap your MAAS, you can then `juju deploy ubuntu --to lxd:<machine-#>`18:20
TeranetI have MAAS running and this is the Cluster Node on which I deploy18:20
bdxTeranet: or even better, `juju deploy ubuntu --to lxd:<machine-#> --constraints "spaces=<my-mass-network>,<my-other-maas-network>`18:21
Teranetdbx ??? now you lost me there18:22
bdxTeranet: sf2-maas0018:22
Teranetas far as I recall no matter if MAAS or not you deloy the first juju boot strap and all on the Cluster Master Node18:23
bdxis sf2-maas00 a node in  your maas?18:23
Teranetsf2-maas00 is the Master of the Cluster18:23
Teranetno it's the Regional Controller of the MAAS18:23
bdxahhh, so your combined region/rack controller, ok18:23
bdxTeranet: have you any machines checked into your maas?18:23
TeranetI have 5 NODES ready18:24
Teranetall same specs18:24
TeranetSupermicro  24 core  32GB ram :-(  and 500GB SSD for starters18:24
bdxTeranet: ok, do you have network spaces/subnets created, and associated with your node interfaces?18:24
bdxin your maas18:24
Teranetall are in em1 Management network 10.5.100.0/2418:25
Teraneteven their IPMI  is18:25
Teranetthe idea is to have OpenStack at the end doing al lthe work18:26
bdxTeranet: have you created network spaces/subnets in the context of MAAS and configured your nodes interfaces to be attached to the networks?18:26
bdxTeranet: I've been deploying openstack on maas for a few years now18:26
Teranetcould you skip stupid juju to get Openstack up ?18:27
bdxTeranet: so without any networks configured in maas other than the mgmt net, containers deploy by default to the mgmt network18:27
Teranetyes that's what they should for now18:27
bdxTeranet: do you have juju bootstrapped to your maas?18:28
Teranet... not 100% sure if I did or not18:28
Teranethow can I check ?18:28
bdxTeranet: you would have to `juju bootstrap maas my-maas`18:28
bdxhttps://jujucharms.com/docs/2.0/clouds-maas18:29
TeranetI just had juju installed and try to get this piece working18:29
Teranetso I think I do not have it tied to the MAAs yet18:29
bdxTeranet: so your missing the juju <-> MAAS communication link18:29
bdxTeranet: once you get juju bootstrapped to your maas, deploying containers and openstack itself becomes very easy18:30
Teranetright but brings me back to juju bootshit to fail right from the  get go18:30
bdxTeranet: work on bootstrapping your maas18:30
bdxTeranet: bootstrapping your maas has nothing to do with containers or lxd bridge18:30
bdxyou just need to do this https://jujucharms.com/docs/2.0/clouds-maas18:31
bdxonce you have your maas bootstrapped, you can deploy containers to your nodes, and the networking bit gets handled by juju/maas, and you don't have to do any of it manually, its quite nice18:31
bdxTeranet: once you have your maas bootstrapped you can just `juju deploy ubuntu --to lxd:<machine-#>` and your container should be reachable from your maas management network18:33
Teranetok now I go this : maas-prod          0                 maas        Metal As A Service18:33
Teranet 18:33
Teranetso I added the stuff18:34
bdxso now18:34
bdx`juju bootstrap maas-prod maas-prod-controller`18:35
Teranetok question so maas-prod I get but maas-prod-controller is than sf2-maas00 ?18:36
bdx'maas-prod-controller' will be the name of your juju controller18:36
Teranetoh crap how can I check that again ?18:36
bdxthe juju controller will be selected from your available machines in your maas18:37
bdxand then will be named `maas-prod-controller`18:37
bdxyour juju controller will be created upon bootstrap18:37
Teranethttp://paste.ubuntu.com/23620258/   ???18:38
bdxahhh18:39
bdxone last step18:39
bdxjuju add-credential maas-prod18:40
Teranetok now it ask for credential name ??18:40
bdxmaas-admin18:41
bdx(arbitrary - I think)18:41
Teranetso the admin user of maas ?18:41
bdxehh ... yes18:41
bdxthat might be best for consistency .... in case you end up with multiple maas users18:42
Teranetok .... building now18:44
bdxooh, maas is bootstrapping now?18:45
Teranetso far :18:45
Teranetsysadmin@sf2-maas00:~$ juju bootstrap maas-prod maas-prod-controller18:45
TeranetCreating Juju controller "maas-prod-controller" on maas-prod18:45
TeranetLooking for packaged Juju agent version 2.0.2 for amd6418:45
TeranetLaunching controller instance(s) on maas-prod...18:45
Teranet - crcrbk (arch=amd64 mem=32G cores=24)18:45
bdxexcellent18:45
TeranetFetching Juju GUI 2.2.518:45
Teranetstill fetching18:46
bdxTeranet: can you see that one of your nodes has started to deploy in maas?18:47
Teranetlet me check18:47
Teranetyes 1 NODE turned on and is deploying now18:48
Teranetok somehow I will need to write this shit down later on LOL18:48
Teranetbecause our ubuntu docu is not working at all again :-(18:49
Teranetdo figure LOL18:49
bdxTeranet: I compiled a ~200 page doc for my last company that details deploying openstack on maas with juju18:50
bdxlet me see if I can dig it up for you :-)18:51
Teranetthis I have to put back onto ubuntu.com manuals LOL18:51
bdxTeranet: you can get a default vanilla deploy very easily just using the juju/maas docs I think, but you will want to start creating your own docs for this type of project for sure18:52
Teranetthink wrong :-) that's why I am here because someone F'd up the docu18:52
TeranetLOL18:52
bdxTeranet: can you point out to me where the doc is failing you?18:53
Teranetso when I did install after MAAS the apt install juju18:54
Teranetonce that's in it directs you to a full bunch of BS18:54
Teranetit should direct you for OS setup with juju just like you gave me the striped down version of commands18:55
bdxAhh, yes installing maas, has a few extra gotchas that need be addressed ... did you reference the MAAS 2.0 docs https://maas.ubuntu.com/docs/install.html?18:55
Teranetcorrect that one is out of date as well18:57
Teranetso we have it on the list to rewrite it18:57
Teranetbecause with 16 its simpler now18:57
bdxahh, ok18:57
Teranetgood my workshop for this is in 3 weeks LOL18:58
Teranetnot today18:58
Teranetgosh still building wtf :-)19:02
Teranetok crap now what :  ERROR failed to bootstrap model: bootstrap instance started but did not change to Deployed state: instance "crcrbk" is started but not deployed19:05
Teranet 19:05
Teranetdebug again ?19:05
rick_hTeranet: are the nodes all setup to boot from lan first and disk second?19:08
rick_hTeranet: when I've had that I've not had the boot order in the bios setup right and so it couldn't deliver the image, reboot, and get the node on disk going19:09
Teranetyes network first than USB if attached and than RAID SSD19:11
Teranetand IPMI controlled19:11
bdxTeranet: thats your issue19:11
bdxTeranet: you have a preconfigured hardware raid?19:11
Teranetyes19:12
TeranetRAID is set and the boxes are all in READY status19:12
bdxTeranet: can you configure your raid via maas please19:12
Teranetit already is19:12
bdxTeranet: oh19:13
bdxrick_h: what is the status of raid support in maas?19:13
bdxTeranet: I would advise you to get this setup in the most default/vanilla way initially19:14
Teranetvgroot-lvroot19:14
Teranet479.0 GB19:14
Teranetext419:14
bdxok19:14
bdxTeranet: I ran into a few issues when setting up raid via maas, it had to do with how I was creating vgs and mounting filesystems19:15
bdxTeranet: can you make the raid a secondary goal19:15
bdxand kill it for now, for the sake of just getting up and running with minimal issues19:15
Teranetso should I blow the hole MAAS box away and re add it or what ?19:16
bdxTeranet: yea, if you could just reprovision the nodes to not use raid pls19:17
Teranet?? it's a HARDWARE Raid just FYI19:17
Teranetso the box don't see it19:17
bdxTeranet, yea, thats also an issue19:18
bdxTeranet: if its a hardware raid, then you didn't configure it via maas19:18
bdxTeranet: get rid of the raids, let maas manage your disks19:18
bdxTeranet: you can use maas to create a raid of those disks for your /19:19
TeranetI think you don't get it19:19
bdxlater, after you get everything standing vanilla19:19
TeranetI won't change it because thats how it's default19:19
TeranetRAID 1 is a must set and I won't change it19:19
bdxTeranet: maas expects to manage your hardware19:20
bdxTeranet: if you try to interfere, you will experience difficulties19:20
bdxTeranet: you can still have those disks in raid119:21
bdxTeranet: you would just need to do it via maas, not your hba controller19:21
Teranetand I won't let mass software control my hardware raid19:22
bdxTeranet: maas doesn't control your raid19:23
bdxTeranet: maas just configures it upon deployment19:23
TeranetMaas don't see it as a Raid and that's how it has to be19:24
bdxTeranet: Ok, I had the same thing going on .... I ended up switching my raid controllers into hba mode, bc it was just easier to let maas do it, but if you insist on using hardware raid, you might want to run the specs of your setup by the maas team so they can verify MAAS will work as intended on your hadware configuration19:26
Teranetthey are and I verified them myself19:27
Teranetso that ain't the issue19:27
bdxTeranet: conceptually, what you are thinking should work, I'm just trying eliminate extra things that might cause issues while getting you up and running initially19:27
bdxTeranet: one thing you can do to troubleshoot, is to make sure the nodes deploy via maas, w/o juju19:28
Teranetthey do as I said all 5 nodes are in RAEADY mode19:28
bdxe.g. just deploy the node from the maas gui, and make sure they deploy successfully19:28
Teranetso they are deployed via MAAS alone no problem with IPMI19:29
bdxif that checks out, then juju will be able to deploy them first19:29
bdxok, then do you release them before bootstrapping with juju?19:29
bdxTeranet: if they deploy successfully via MAAS standalone, then juju bootstrap should work for sure .... so strange its not for you19:31
Teranetno offense here bdx but this is not how it should work19:31
bdxTeranet: how so?19:31
TeranetI am certain some code has changed in juju again and now the BS don't work again19:31
TeranetI saw some guys pushed code changes in without approval19:32
Teranetworse comes to worse I need to roll back the code and ban those guys from doing changes to our code again19:32
cory_fukjackal, kwmonroe, petevg: New version of BT has been released, with matrix support, xunit reporting, and a few other fixes.19:38
petevgcory_fu: sweet!19:38
kjackalawesome, thanks cory_fu19:38
cory_fukjackal: Thank tvansteenburgh.  He merged and released it.  :)19:39
kjackalthen ... thanks tvansteenburgh!19:39
* tvansteenburgh waves19:39
=== mskalka is now known as mskalka_
=== mskalka_ is now known as mskalka
cory_futvansteenburgh: Man, you are on *top* of the PRs today.  :)20:37
tvansteenburghi try cory_fu20:37
tvansteenburghyou do get special treatment though20:37
cory_fu:)20:38
=== mskalka_ is now known as mskalka

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!