[02:20] I have an LXD container that seems to be live, but juju shows it as pending. [02:21] I'm a bit at a loss for why. The logs don't seem to show anything wrong, I think. [02:22] `juju ssh 0` followed by `sudo lxc list` shows it has IPV4 addresses, but it doesn't show anything before with `juju status` [02:24] maas has the new container's two IP addresses shown like the machine's lxc command, but juju doesn't seem to be aware [02:25] andrew-ii: does `juju status --format=yaml` show error? [02:26] I don't think so. Just 'juju-status: current: pending' and 'machine-status: current: running' [02:28] I feel like maybe it's related to maas rack HA, since a few days ago I added a second rack controller (but didn't work on it until today) [02:29] This is with a juju 2.1.2 controller (fresh) [02:31] Oh and maas 2.1.3 [02:32] andrew-ii: m not sure what's going on.. and there is nothing in Juju or MAAS logs? [02:32] Not that I saw... I'll check again, though. [02:33] andrew-ii: and u r not using kvm on maas? [02:33] No, should only be lxd, I think [02:36] andrew-ii: file a bug, include ur bootstrap/deploy commands, logs and --format=yaml status output :) any other info like MAAS setup, network topology, etc as u see fit [02:36] Alrighty. I was assuming it was something screwy with how I set it up [02:37] Should I try to rebuild the juju controller from an earlier version? [02:38] if u can confirm if it works with earlier version would b awesome \o/ u could also try later Juju version - 2.2: alpha1 or daily snap... [02:38] it mayb a problem with just 2.1.x [02:38] Alrighty. Lemme gather the logs and such [02:44] Err. Dumb questions asked poorly: how do I use root with `juju scp -- -r root@0:/var/log/juju ./machine-0-juju-logs`? [04:43] andrew-ii: juju run --unit /0 'cp -a /var/log/juju /home/ubuntu/ridiculous && chown -R ubuntu /home/ubuntu/ridiculous' [04:43] andrew-ii: juju scp -- -r cwr/0:ridiculous /tmp [04:43] it's not pretty andrew-ii, and i can't belive juju scp doesn't work in root context, but it is what it is. [04:46] oh, i said "cwr" in that scp command.. i meant "", as in, whatever unit you want to scp from. i just happen to test this in an env with cwr/0 deployed. [05:32] i just learned about the existence of foreman. does anyone have experience what is better? juju or foreman [07:34] ybaumy: I'd compare it more to MAAS. Comes from Red Hat and has been around for a while. Works well with Puppet [08:04] Good morning Juju world === frankban|afk is now known as frankban [09:34] Hello there. Is it possible to have an bootstrap node in LXD for usage with MAAS? [09:35] so i have a maas node, and i create an LXD container on that same node to be the bootstrap node? [09:40] BlackDex: i used kvm [09:41] i do that also [09:41] but i wonderd if i can skip the v in kvm ;) [09:41] because of performance [09:44] maybe i can do it by first creating a LXD container, and adding that to MAAS [09:45] giving it the correct tags and tell juju to use that [09:47] i don't think you can add lxd to maas? [09:53] yea you can :) [09:53] MAAS Supports LXD :) [10:24] kwmonroe: thanks! The command makes sense and works fine [11:35] What hook gets called if I 'juju attach' a resource to an application? [11:36] I'm guessing config-changed ? [11:38] nope, upgrade-charm according to docs. [12:06] stub: correct, upgrade-charm. [12:06] jamespage: where is the ganglia charm repo? [12:07] the charm points to https://code.launchpad.net/~charmers/charms/trusty/ganglia/trunk but that hasn't been touched in a long time [12:56] tvansteenburgh: https://github.com/ganglia-charms [14:07] Another DQAP: Can I deploy onto a controller? [14:08] Mostly because `juju deploy openvpn` is just so convenient... [14:09] Granted, I suspect it adds a ton of crazy to the model, but I've seen references to people finagling odd setups and figured I'd ask. [14:12] morning [14:13] balloons: poke? [14:13] andrew-ii: you can certainly try, there shouldn't be much collision, but it's something you'll want to test first [14:13] andrew-ii: juju deploy --to 0 -m controller openvpen [14:13] cnf, howdy [14:13] ohai! \o [14:13] balloons: so i'm back it :P [14:14] did you see https://bugs.launchpad.net/juju/+bug/1674759 ? [14:14] Bug #1674759: juju upgrade-juju doesn't honor proxy settings [14:15] anastasiamac: also poke :P [14:17] cnf, looks like it would be worth trying the unstable 2.2 and see if it things are better [14:17] balloons: cn i upgrade to 2.1.2 with a 2.2 client? [14:18] or are you/they suggesting upgrading the controller to 2.2? [14:18] it's a PoC, so i don't mind overly much [14:19] cnf, you could use upgrade to go backwards, since there wouldn't be anything newer if you did try 2.2-alpah1 [14:19] But yea, it would be a new controller [14:19] $ ./juju upgrade-juju [14:19] no prepackaged tools available, using local agent binary 2.2-alpha1.1 [14:19] ERROR no matching tools available [14:21] updated the issue as well [14:22] balloons: i'm tempted to just delete the controller, and bootstrap a new one [14:22] but i will probably run into this again, then [14:24] cnf, just changing the client won't change things indeed. You'd have to bootstrap a new controller [14:25] hmm, so my only option is to bootstrap a new controller [14:25] hmm [14:26] as far as verifying if the newer juju fixes the bug yes [14:26] Or bootstrapping a newer controller and doing model migration [14:26] hmm [14:27] i might as well throw everything away, and start new [14:27] lxd might be useful here as well [14:27] balloons: how so? [14:28] if you lack the physical machines [14:28] can i deploy a controller to MAAS using LXD? [14:31] I think actually the easier way to do it is to add a vm using virsh on the maas controller for the juju controller. That's a better way to double dip I think [14:32] that's what i have now [14:32] the juju controller is a KVM [14:32] ahh, awesome [14:32] so can you simply add another kvm? [14:32] on an ESXi vm... [14:33] it would only have to live long enough to migrate your workload. In the interest of checking the bug, you migrate, then attempt to upgrade [14:33] well, "simply" [14:34] but i have nothing deployed atm [14:34] so i might as well just trash this one, and bootstrap again [14:34] ack [14:34] it just has me worried i'll run into it again [14:35] hmm [14:36] also, if i do this, i can no longer help debug the problem [14:37] anyone know when anastasiamac comes online? [14:40] cnf, about 7 hours from now [14:40] cnf: she's in NZ timezone [14:40] hmm, that's a bummer :P [14:40] cnf, presumably you could recreate easily enough [15:20] cory_fu: i don't know enough puppet to grok the syntax on https://issues.apache.org/jira/browse/BIGTOP-2708, but i *think* we're ok because we used role-based bigtop (and not component-based, which is what this is patching). [15:20] having said that, i shall deploy zeppelin and see what happens :) [15:34] hmm, crap, i used the wrong user for the controller again [15:34] can I cswitch the maas suer the juju controller uses? [15:52] hmm, why is juju status --output=yaml empty? [15:52] juju status shows me plenty [15:55] hmz [15:56] wtf [15:59] ugh, i really don't know how to debug this stuff... [16:00] jamespage: what was the right incantation? [16:04] cnf: --format [16:04] cnf: --format=yaml [16:05] oh [16:05] i'm blind :P [16:06] so now to find why this isn't working, still :( [16:07] jamespage: http://termbin.com/vz8q see anything obvious? [16:11] juju seems to just break shit, and then stop doing stuff [16:11] wth :( [16:11] maas brings up the machine with all the right IP's [16:11] juju changes the IP's around to bridge interfaces, except doesn't do it right [16:11] and then fails [16:12] i think :( [16:12] can anyone help with this? [16:12] jam: any chance you can caste your eyes over cnf's problem above? [16:12] appears to be some sort of network-space device binding allocation lxd type problem [16:15] seems every time i file a big, and get around it, something else pops up [16:21] jamespage: btw, that person has not called me yet ^^; [16:21] dno if you made the link to the mails etc, but we met at cfgmgmtcamp [16:22] cnf: yep I gotcha [16:28] jamespage: nice that you are here. maybe you can help. is the following right to do. remember i set the root password for the mysql charm already. now i want .. juju add-machine -n2 ; juju add-unit mysql --to machine1 ; juju add-unit hacluster --to machine2 ; juju add-relation mysql/0 hacluster/0 ; juju add-relation mysql/1 hacluster/0 ??? [16:29] so jam is the person i need for this, right? [16:30] or is it juju deploy hacluster --to machine2 i guess [16:35] hmm [16:35] i'm at a loss here [16:38] lazyPower did you read the backlog in channel? [16:39] Budgie^Smore: i did see that you encountered some failure scenario(s) in AWS [16:41] cnf: i feel you [16:41] lazyPower yup, I need to resize the instances anyway so am going to do a clean install rather than salvage this on but wondered if your team would like any failure data before I destroyed the cluster [16:42] cnf: i'm pretty sure jam is in UTC+. you may have more luck pasting http://termbin.com/vz8q in #openstack-charms. [16:42] kwmonroe: i think it's a juju issue, but no harm, i guess [16:44] cnf: i'm not familiar enough with network space binding to offer much help -- hopefully #openstack-charms has the people with the right know-how for ya. [16:44] i hope so [16:44] i'm getting quite dispirited... [16:45] Budgie^Smore: oh heck yeah [16:45] Budgie^Smore: can you get us a juju-crashdump of the model? [16:45] Budgie^Smore: snap install juju-crashdump --edge && juju-crashdump. I can get you a secure upload point again [16:45] need a sec tho i'm in an openstack meeting [16:45] yeah I need a min or two to get my laptop up and running, etc. [16:46] make that a few more than than that since I need more coffee [16:53] ok I haven't used snap before and getting an error about requiring classic or confinement override [16:53] kwmonroe: I'm already a bit stumped on what the problem is with cnf's deployment [16:55] jamespage: bit more reference http://termbin.com/ep44 and http://termbin.com/6o0h [16:55] ybaumy: kinda - adding units is nearly as you describe [16:55] maas brings it up fine, then juju tries to make the bridges [16:56] you'll want todo something like juju add-unit --to lxd: mysql [16:56] with regards to hacluster [16:56] ybaumy: juju deploy hacluster hacluster-mysql [16:56] juju add-relation hacluster-mysql mysql [16:57] lazyPower I added --classic to that snap command [16:57] ybaumy: hacluster is a subordinate charm so to make units of it, you relate it to a principle charm like percona-cluster [16:57] or glance or cinder or nova-cloud-controller [16:58] ok [16:58] gonna make a backup before i try that [16:59] jamespage: Do y'all have a reference bundle for OpenStack HA laying around besides https://launchpadlibrarian.net/298175262/bundle.yaml? [16:59] If not, then that might help you out, ybaumy. [17:01] Budgie^Smore: ah good plan, i forgot that flag [17:01] Budgie^Smore: sorry, this is what happens when i triple-task.... :\ [17:02] zeestrat: i dont want to start over at that point .. [17:02] no worries, being able to figure out issues like that is makes me good at what I do [17:02] zeestrat: i know of that template you pasted it before for me [17:03] zeestrat: but i have a setup and dont want to start over every 2 days [17:03] zeestrat: i already scripted alot to fit the current environment i dont know if the scripts will work with that new setup then [17:05] ybaumy: My bad. Memory is not what it used to be! [17:08] getting a bunch of runtime/cgo: pthread_create failed: Resource temporarily unavailable [17:08] zeestrat: yesterday i got a deadline for the POC. and i now have one month to show them that ubuntu/juju/maas is the platform we want to use. else i have to start with redhat or suse [17:08] getting a bunch of "runtime/cgo: pthread_create failed: Resource temporarily unavailable:" errors running that command [17:08] so i need some support [17:12] Budgie^Smore: thats emitting from snapd [17:12] Budgie^Smore: so long as the process hasn't returned its still tarballing up the relevant bits of the model and preparing a dump package [17:13] Budgie^Smore: i encourage you to extract it and take a quick peek before washing hands, this app dumps the full state of a "crashed model" for post analysis, this may be something you want to have in your toolkit as you're building a pretty heady k8s model right? [17:14] lazyPower, where did you want this ran? I assumed on the client that was connecting to the client but I am thinking now you meant a different system since I got told it needs to run as root but root isn't connected to the controller [17:15] cory_fu: ^ needs to run as root? huh? [17:15] Budgie^Smore: hang on cc'ing a stakeholder on the project [17:16] Budgie^Smore: but no, it should be run as your user on your client workstation, that much is correct. [17:16] lazyPower oh I was intending on holding onto a copy to look at myself too as yes I want to at least try and avoid this scenario but I need to upgrade the underlying infra anyway and always like to start from a known good state [17:16] lazyPower: Trying to get caught up on the backscroll, but what run as root? [17:16] no one that has an idea what is going on here http://termbin.com/vz8q ? [17:17] cory_fu: ah, this was returned from juju-crashdump for Budgie^Smore [17:17] cory_fu: i was instructing budgie to create a crashdump archive of the model before destroying it so we could do some post-analysis of what went sideways during model deltas. [17:17] and apparently its complaining it needs to be run as root? is this new behavior? I've not seen this before. [17:17] lazyPower: Hrm. crashdump shouldn't require root on the local machine, and anything it does on the remote machine should be done with `juju run` so it should have root (I think) [17:18] lazyPower, cory_fu here is the ful output from juju-crashdump: http://paste.ubuntu.com/24229662/ [17:18] lutostag: ^ [17:18] cnf: alot of waiting for machine... what does maas say? [17:18] ybaumy: all machines up [17:18] it's only "waiting" on lxd containers [17:18] oh that bubbled up from snapd again. thats so weird. [17:18] juju is fucking up the networking [17:18] Budgie^Smore: did you get a crashdump package in $PWD? [17:19] Yeah, I have no idea what those messages are about [17:19] ybaumy: http://termbin.com/ep44 and http://termbin.com/6o0h [17:19] lazyPower don't see any files created :-/ [17:19] maas makes the net devices, and adds the right IPs [17:19] :| thats no bueno, it should have left a crashdump-$datestamp package in $PWD.... ok lets get lutostag in on this one to help if he's around [17:20] juju then tries to create bridges, and move the ip's [17:20] for some reason it fails [17:20] thanks cory_fu for taking a look. [17:20] and then complains it can't find ips [17:20] cnf: hmm that looks like a mess. i wouldnt know where to start too sorry [17:21] which vlan is the public ip [17:21] for the lxd containers [17:21] lazyPower, lutostag for the record here is the command I used to install juju-crashdump: $ sudo snap install juju-crashdump --edge --classic [17:21] you get BridgeName:br-enp3s0f0.4013}] devices on host "machine-0" for container "machine-0-lxd-7" with delay=0, acquiring lock "machine-lock" to-bridge="enp3s0f0.4013" --activate --bridge-prefix=br- --reconfigure-delay=0 /etc/network/interfaces <<'EOF' [17:21] and then a few blank lines [17:22] and then find host bridge for space(s) "space-public" for container "0/lxd/7"), retrying in 10s (3 more attempts) [17:22] lazyPower: lutostag has much more understanding of crashdump. I just use it from time to time. ;) [17:22] and the same thing again [17:22] >,< [17:22] cory_fu: i still associate you as a stakeholder since you pimp it so much :) [17:22] and by extension, i now pimp it just as much [17:22] lol [17:22] you are using maas dhcp right on that vlan? [17:23] well apparently you are pimping an STI right now :P [17:23] O_O [17:23] ybaumy: it's not dhcp, it's maas assigned [17:23] Budgie^Smore: if you snap list, do you have 1.0.0 of juju-crashdump installed? [17:23] cnf: bridges not appearing right? [17:24] cnf: beisner just highlighted this problem to me - https://bugs.launchpad.net/juju/+bug/1672327 [17:24] Bug #1672327: Too long names for bridges [17:24] lazyPower hey that was the cleanest of the jokes that flashed through my head... anyway http://paste.ubuntu.com/24229699/ [17:25] jamespage: well, i'll be... [17:25] cnf: those where not my words... [17:25] damn ivoks is not here... [17:25] why do I have to run into every fucking weird bug with this? [17:25] Dmitrii-Sh: re bug https://bugs.launchpad.net/juju/+bug/1672327 [17:25] Bug #1672327: Too long names for bridges [17:26] Dmitrii-Sh: did you and ivoks figure out a workaround for that? [17:26] cnf, welcome to the club, just ask lazyPower how many weird issues I have hit [17:26] jamespage: isnt it possible to use udev for that? [17:26] jamespage: as a workaround we just renamed the interfaces to something really short [17:26] Dmitrii-Sh: renamed them where? [17:26] in maas? [17:26] jamespage: e.g. instead of en1s0 use e0 [17:26] cnf: yes [17:27] hmm, ok [17:27] i'm fine with using eth0 and eth1, ffs [17:27] cnf: we all where :-) [17:27] cnf: the problem is that UAPI kernel headers have a byte limit of 15 [17:27] cnf: same with libc [17:27] stupid ass retarded piece of shit systemd! [17:27] cnf: :^) [17:28] ok [17:28] jamespage: well, i'd say both maas and juju should guard against this [17:28] instead of just silently failing [17:28] don't disagree [17:28] yeah, thats all the same here too Budgie^Smore not sure why its tanking :( this bums me out [17:28] cnf: links to the kernel code and libc are in the comment if needed [17:28] ok, so destroying the setup [17:28] changing network names [17:28] and deploying again [17:29] cnf: y [17:29] lazyPower sorry :-/ the sarcastic side of me has a few choice "jokes" for the devs right now ;-) [17:29] lazyPower hey at least this one isn't your fault :) [17:29] Budgie^Smore: do what you gotta do man ;) i can filter appropriately [17:29] jamespage: asked a maas question over in #maas [17:29] ikr! [17:29] for a change, i dont feel totally responsible [17:30] Dmitrii-Sh: how exactly did we do the renames for the interfaces? [17:30] lazyPower nope I won't got to that dark place, I am reforming ops guy :P [17:30] i dont know much about that but cant you just modify udev network rules every first boot to rename interfaces [17:30] thats what i would do [17:31] ybaumy: modify them how? [17:32] ybaumy: this is all automated deploys [17:32] ybaumy: that's pretty much exactly what maas will do on deployment [17:32] jamespage: in short: I don't have a script for it yet. I just renamed them via maas gui. Changed interface names to 2-byte names and then redeployed. Any VLAN interfaces are updated automatically by maas [17:32] Dmitrii-Sh: indeed [17:32] jamespage: ah ok [17:32] i'm collecting a LONG lust of maas and juju bugs :/ [17:33] cnf: hearing you [17:33] jamespage: i would have modified cloud-init but thats probably what maas does too [17:34] Dmitrii-Sh: do you know how to get back interfaces you deleted in MAAS? [17:34] Dmitrii-Sh: assuming you don't know the MAC addresses anymore :P [17:35] cnf: I suspect if you recommission in the machine, they will get re-discovered - but I'd defer to those with superior MAAS knowledge to me [17:35] k [17:36] cnf: better to recommission them, yes. That should boot an ephemeral ubuntu image via PXE get hw data again and bail out [17:36] so br-bond0.4011 is fine, at least [17:41] so anyone know how i can change the user a juju controller uses? [17:42] to talk to maas, that is [17:44] you have to switch the credential being used... I'd have to dig docs [17:45] cnf: ~/.local/share/juju there is a credentials file [17:45] juju add-credential with the same id for the credential will update it [17:45] ybaumy: yes, which isn't used after bootstrap [17:45] cnf: last time i just edited that [17:46] or like Budgie^Smore says [17:47] as far as I can see, that doesn't get used after bootstrap [17:48] it should be used anytime juju needs to get cloud resources [17:48] it isn;t [17:48] i thought so too [17:48] try it [17:48] bootstrap with one user [17:48] then change local credentials [17:48] and then deploy something [17:48] they'll be added to maas with the original credentials [17:49] hmm [17:49] then you have to use the command [17:49] sorry then [17:49] cant try right now. backing up everything [17:51] cnf: I think you should be able todo it with 'juju update-credential' [17:51] jjuju update-credential yeah [17:51] jamespage: that works, indeed [17:51] jamespage: doesn't change what is already running, though :P [17:51] but it is something, thanks [17:51] cnf: in terms of how that's allocated in maas [17:51] no I'd not expect it todo that [17:51] ok cnf, that is the part that I don't get [17:51] ok, while those HP machines are booting, i'm going for a shower, and look for food [17:52] Budgie^Smore: what? [17:52] cnf: I'm also concerned you might island existing allocated ressources. [17:52] cnf: you cant change the owner of a added ressource in maas once its added i guess [17:52] cnf when you say change what is already running [17:52] jamespage: yeah, and it's the controller :P [17:53] lets see if we can get openstack working, i'll worry about ACL's and accounts later [17:53] bbiab [17:53] jamespage: thanks so far [17:53] cnf I probably missed it but what are you trying to accomplish after you change the credentials juju uses? [17:54] Budgie^Smore: juju should use a juju user in talking to maas [17:54] i was loged in with my admin user [17:56] cnf, ok I think there maybe a misunderstanding in the difference of cloud credential and juju user here [17:59] a juju user is just a user of a cloud but not the cloud user [17:59] right? === frankban is now known as frankban|afk [18:00] cnf, they are 2 separate ACL systems, reason being clouds (including MaaS) auth APIs are different and the juju user is for interfacing with juju [18:01] cnf, for example for AWS it isn't even a username and password you pass to authenticate, it is 2 keys which are associated with a user account on Amazon's end [18:03] petevg, ping you around? [18:04] stroke: i am. What's up? [18:04] sfeole: silly autocorrect munger your name, though. [18:04] petevg, hey, i wanted to use some exit handlers in libjuju, one that i looked at was atexit, But i don't think I can properly utilize that handler with asyncio [18:05] sfeole: there are usually asyncio equivalents for that sort of thing. [18:05] petevg, ahh there are [18:05] petevg, i'll take a look then [18:06] petevg, i want to simply destroy a model upon exit [18:06] sfeole: cool. Ping me if you want to bounce anything off of me. [18:06] petevg, sounds good [18:12] Budgie^Smore: ybaumy so i talk to juju, not to the cloud [18:12] Budgie^Smore: ybaumy so i should not HAVE cloud credentials, juju should [18:15] cnf what I haven't tried is if you can talk to juju without being able to talk to the cloud (or MaaS) but I have tried logging in as multiple users to juju from the same machine so that both juju accounts had access to my cloud credentials [18:15] Budgie^Smore: well, if you have several users talking to maas, that gets a mess [18:15] because each need their own cloud credentials? [18:15] then you need to mirror ACLs [18:15] that will become a MESS very very fast... [18:15] cnf and in the juju world, models are "owned" by users and you have to give other users access the model [18:16] Budgie^Smore: right [18:16] Budgie^Smore: but then do the other users also need access to the same underlying cloud resources? [18:16] and what if they don't match? [18:16] so i expect the juju controller to use one user / api key / whstever [18:16] and deal with juju user itself [18:17] cnf oh I kinda get that but from a security stand point you want all your ops people to have their own account on both the cloud and juju. this is more important in public clouds though where you already have to set people up there separate from juju [18:17] Budgie^Smore: no, no you really don't [18:17] Budgie^Smore: because ACLs on both will NOT match up [18:17] they won't have the same mechanisms [18:18] ( i mean, you want them to have a cloud account to deal with cloud issues, but not to talk to the cloud through juju) [18:18] cnf and there is why you see why they are separate in juju to start with [18:18] Budgie^Smore: what? [18:19] a juju user should NOT! need a cloud account to talk to juju and do stuff [18:19] no you want them to use their cloud credentials through juju especially since otherwise you wouldn't be able to tell actions apart [18:19] no, no you really don't... [18:19] that will be a mess [18:19] it just can't work [18:20] cnf how are you going to get instances spun up with out clound access? [18:20] Budgie^Smore: JUJU should have cloud access [18:20] not the juju user [18:20] this is how you set up vnf controllers... [18:20] cnf how are you going to tell apart who did what through juju to the cloud? [18:21] it just doesn't work if you need credentials all the way down the line... [18:21] Budgie^Smore: in juju..., [18:21] that is why you have a juju controller... [18:23] Budgie^Smore: say i make a budgie/default model, which you have access to [18:23] you need a budgie user on the cloud [18:23] cnf but that is not how it works nor would any infosec team I know sign off on it working like that. yes you can use juju to determine some of that but infosec onces to be able to see it at every level and shared access at any level is frowned upon significantly [18:23] then i give bob access to budgie/default [18:23] now the bob user on the cloud needs access to your resources on the cloud [18:23] but the cloud has no granularity, so he has access to ALL your resources? [18:23] Budgie^Smore: uhm, sure infosec would sign off on it, why would they not? [18:24] that's nonsense [18:24] that's how orchestration works [18:24] needing credentials on every single step along the way is unmaintainable [18:24] and as such a security nightmare [18:25]