=== thumper is now known as thumper-afk === thumper-afk is now known as thumper === ubuntu is now known as Guest65049 === Guest65049 is now known as bdx === axino` is now known as axino [08:39] good morning Juju world! [10:56] I'm trying to upgrade juju 1.25.6 -> 1.25.10 but it seems to fail right at the beginning and not even starting the upgrade. http://paste.ubuntu.com/24220847/ [10:57] Any idea what's going on here? At machine #0 I can't see anything that even registers as a upgrade attempt [10:59] if one of the machines fails during bundle deployment, can I tell juju to retry this one machine and all units that depend on it? [11:10] kklimonda: this is a rather charm/specific issue. The charms should stay in waiting/blocked state and react to other charms states [11:11] kklimonda: so as soon as you resolve the issue with the machine/charm the rest of the bundle should react to that [11:12] but what if MAAS has failed to deploy the machine? [11:12] I can't retry it, as juju plumbing would be missing [11:16] juhu juju [11:17] how do i change the cpu overcommitment values in nova.conf .. it says that its maintained by juju [11:17] i want a 20:1 ratio [11:19] i know thats high but its just for test instances [11:22] Hi, I deployed a bundle with maas; but one of the machines got broken (stuck with no access to this machine). So I marked this machine as broken. Now how can I proceed the deployment? [11:25] Any advice? Is it possible to make it to allocate a new machine and proceed the remaining application deployment? [11:28] jianghuaw_ kklimonda: you can try a juju retry-provisioning . If that fails you can deploy the failing application in another machine and re-add the relations [11:28] if any [11:29] kjackal: thanks, does it work for both machines and lxc containers? [11:29] looks like what I'm looking for [11:30] kjackal, thanks. [11:30] kklimonda: yes works for any provider. The bundle is just a "script" [11:32] ybaumy: I do not see anyhting in https://jujucharms.com/nova-compute/266 . You could ask at #openstack-charms [12:29] hmm, ok [12:33] jamespage: this is the result from your bundle http://termbin.com/dy1u did i do something silly? you where going to test it, right? [12:34] cnf: I think what I said was I don't have anywhere to test this right now :( [12:34] cnf: juju status --format=yaml might tell us more [12:34] ah, ok :P [12:35] message: '{"hostname": ["Node with this Hostname already exists."]}' o,O [12:36] that's on an lxd container [12:36] also [12:36] current: provisioning error [12:36] message: 'unable to setup network: host machine "0" has no available device [12:36] in space(s) "space-openstack-mgmt"' [12:36] but it does? [12:38] http://termbin.com/va82 [12:42] oh, nm [12:42] it seems MAAS is being weird again? [12:42] wtf? [12:43] jamespage: ok, back to #maas it seems omO [12:52] or is it [12:52] bah! i don't know [12:53] this is frustrating [12:53] is there a good way to make an interface that's local to a specific charm rather than a global interface on interfaces.juju.solutions? [12:57] icey: yeah, just put it in INTERFACE_PATH (and set that environment variable) [12:58] marcoceppi: ideally I'd love to have it in tree for this charm [12:58] icey: why? [12:58] kind of defeats the purpose of interop [12:58] marcoceppi: it's an existing interface; I'm porting a charm (old, bash + python) into layers + reactive. The interface is very explicitly only for this application [12:59] icey: why even have an interface at all [12:59] if nothing will ever connect to it [12:59] marcoceppi: there are things that connect to it, but nothing else will provide it [12:59] I suppose I can make a full interface and host it [13:00] icey: well the interface on interfaces.juju.solutions include the provides and requires parts [13:00] marcoceppi: yeah [13:00] icey: yeah, I would go the full monty if other things consume (or provide) it [13:00] the other things consuming it already have written their code to do that ;-) [13:01] icey: but new applications can now use the layer you produce :) [13:01] indeed marcoceppi [13:05] stokachu: https://github.com/conjure-up/conjure-up/issues/750 any thoughts? [13:07] marcoceppi: what does 'sudo apt-get update' give you? does it work? [13:08] stokachu: no, 403 as well [13:08] hmm [13:08] so we rely on either that ppa or the snap version of lxd [13:09] marcoceppi: are you behind a proxy or anything? [13:09] at home [13:09] maybe that needs a refresh [13:11] oddly enough, works elsewhere [13:11] nvm [13:11] got it? [13:12] actually, nope [13:13] okay, I think I figured it out [13:13] what was it? [13:14] bad apt proxy config [13:14] well, apt proxy config, which was fine, but not for ppas [13:15] ah ok [13:16] now I have to fight with snapd [13:23] marcoceppi: apt-cacher-ng ? [13:23] squid-deb-proxy [14:38] hmz ok [14:38] lets try this again :( [14:39] it seems juju breaks my network config on maas nodes [14:39] and then complains about it [14:40] :( [14:41] it tries to move an ip to a bridge [14:41] and then fails? [14:42] can anyone help debug this? [14:43] i'm stuck at http://termbin.com/3z4n [14:48] hmm, this is really frustrating [14:55] jamespage: are you available? [14:57] hmm, how do i upgrade a controller? [14:57] 2.1.2 has something about one of the errors [15:03] >,< [15:04] ugh, wtf [15:04] upgrade instructions don't work [15:06] cnf: you did juju switch controller and juju upgrade-juju? [15:06] rick_h: yes, upgrade-juju doesn't work [15:07] what does it say? [15:07] $ juju upgrade-juju [15:07] no prepackaged tools available, using local agent binary 2.1.2.1 [15:07] ERROR no matching tools available [15:07] cnf: hmm, what did you deploy from? [15:08] 2.1.2-sierra-amd64 [15:08] it is what i used to bootstrap [15:08] well, i used 2.1.1 to bootstrap [15:10] cnf: hmm, so it couldn't find a 2.1.2 for the architecture and tried to use yours but you're on osx which of course can't be uploaded. [15:10] right [15:10] cnf: what is this installed on? [15:11] an ubuntu vm [15:11] cnf: the controllers? [15:11] not sure why it wouldn't be able to upgrade an ubuntu vm to 2.1.2. What version is it at now? juju show-controller xxxxx [15:12] 2.1.1 [15:12] on a xenial VM? [15:12] ubuntu xenial that is [15:12] yes [15:13] i'm trying to upgrade because https://jujucharms.com/docs/devel/reference-release-notes says "[juju] Handle 'Node with this Hostname already exists' errors when provisioning containers. LP:#1670873" [15:13] Bug #1670873: juju fails when requesting an IP for a container when retrying after lxd forkstart

[15:13] and my deploy is failing, and this is one of the errors in my logs [15:13] don't know if this will fix it, but hey [15:13] cnf: k, that sounds good. I'm not sure why the upgrade wouldn't find the 2.1.2 agents though. [15:14] uhu [15:14] cnf: so the controller is a VM, how did you get it setup? With the manual provider? I thought I saw something about MAAS earlier? [15:14] juju model-config shows the right proxies are set [15:14] rick_h: juju bootstrap [15:14] it's a KVM running on the MAAS controller, and added as a MAAS machine [15:14] cnf: on the vm itself using localhost? [15:15] so it is a VM, but it looks like a maas machine to juju [15:15] cnf: ah ok, so yea using the maas provider bits [15:15] yes [15:15] so i use my laptop to run the juju command against a maas / juju setup in the lab [15:15] balloons: have a sec? can you think of why the controller wouldn't find agents for 2.1.2 on xenial/maas? ^ [15:15] cnf: makes sense [15:16] rick_h, what does the log show? [15:16] which log? [15:16] bootstrap -- try running it with --debug too [15:16] the obvious answer is if the maas doesn't have outside internet access [15:17] http://termbin.com/p59f [15:17] balloons: it's already bootstrapped and running [15:17] balloons: the bootstrap was done 2 weeks or so ago [15:17] balloons: so he's trying to juju upgrade-juju to go from 2.1.1 to 2.1.2 [15:22] what's the controller version? [15:22] Are we sure it thinks it's 2.1.1? [15:23] balloons: using show-controller that's what cnf says? [15:23] and it was installed 2 or 3 weeks ago :P [15:24] agent-version: 2.1.1 [15:25] ack [15:27] cnf: can you run the 'juju upgrade-juju' with --debug on it and see if it outputs something helpful on where it's going and we can verify it can reach it? [15:28] rick_h: i did, and pasted the output above [15:28] http://termbin.com/p59f [15:28] oh, sorry I missed it [15:30] it's weird because I don't see it doing anything beyond starting the initial search [15:31] yeah [15:31] and juju model-config shows the right proxies are set [15:32] cnf, can you bootstrap a new 2.1.2 controller? Just to make sure you can see streams? [15:32] balloons: streams? [15:32] btw, "juju status" says "upgrade available: 2.1.2" === joedborg_ is now known as joedborg [15:36] balloons: and bootstrapping a new controller would be about an hour work [15:36] i have nothing ready to take it, atm [15:36] cnf, no worries. Don't want to try that then [15:37] i'll put that on the last resort list :P [15:37] it's possible, but i'd rather go for the easer debugging first, if we can [15:38] cnf, sync-tools may also be an ok test [15:38] if you run with juju upgrade-juju --dry-run or juju upgrade-juju --dry-run --agent-version 2.1.2 [15:38] what happens? [15:39] $ juju upgrade-juju --dry-run --agent-version 2.1.2 [15:39] upgrade to this version by running [15:39] juju upgrade-juju --agent-version="2.1.2" [15:39] :P [15:40] juju sync-tools --public --debug --version 2.1 --local-dir=. --dry-run --stream=released [15:41] http://termbin.com/qx52 [15:42] k, so it can see agents just fine [15:45] balloons: 16:44:55 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "file:///Users/cnf/tools/streams/v1/index2.json" because of missing information: "content-download" data not found [15:45] is that normal? [15:46] cnf, do you have local streams? Where did that come from? [15:46] balloons: i took away the --dry-run from that last command [15:50] A model can't reasonably share machines, right? Like, two models can't coexist? [15:52] hmm [15:53] cnf, juju show-controller [15:54] balloons: https://bpaste.net/show/60fec9e338f8 [15:55] andrew-ii: no, the little watchers running on there would probably get pretty confused [15:56] rick_h: Thanks - I was pretty sure it was nonsensical, but I didn't find the verbage (obvious as it may be) [16:00] balloons: i'm both glad and worried this stumps you as well :P [16:01] glad because it means i wasn't doing obviously stupid stuff, and worried because debugging it is going to be a pita [16:02] cnf, do you know the history of the controller? How was it created, and what's happened to it along the way? [16:02] balloons: yeah, it's all me [16:02] i bootstrapped it 2 or 3 weeks ago [16:03] from this very machine [16:03] using the 2.1.1 client right? [16:04] cnf, juju model-defaults [16:04] yes [16:04] http://termbin.com/hoqz [16:05] model-config has proxy overrides [16:06] cnf, how about model-config then as well :-) [16:06] http://termbin.com/detm [16:07] o/ juju world [16:09] cnf, juju model-config logging-config=juju.apiserver=trace [16:09] juju model-config -m controller logging-config=juju=trace [16:10] ok, and then juju upgrade-juju again? [16:11] yea, with debug. I don't think in this case it will show anything more [16:11] However, we're not getting a good return on finding the agents from streams. [16:11] Have you upgraded controllers before? [16:11] it doesn't show anything more [16:12] no [16:12] balloons: i'm brand baby new to juju / maas [16:12] There may be an issue with it not respecting proxy on upgrade [16:12] (been quite a frustrating experience so far :( ) [16:12] It's clear I think that it's not seeing streams. It should return something [16:12] It's possible to bootstrap another controller and migrate your workload, or do use sync-tools and manually push the agent to the controller to upgrade it [16:13] i did sync-tools, so i have them all locally now [16:13] where should they be on the controller? [16:13] (though it sounds like a bug of sorts) [16:14] cnf, yes, you found a bug indeed I believe. I'll have to repo it, but I'm thinking that's it [16:15] $ pwd [16:15] ubuntu@juju-controller:/var/lib/juju/tools$ ls [16:15] 2.1.1-xenial-amd64 machine-0 [16:15] btw [16:15] that is on the controller [16:16] right, so we'll need to make a local stream, then tell the controller about it, and upgrade using it [16:16] ok [16:18] ok, so you ran synctools and have them all in a local folder? [16:18] lazyPower, you about today? do you remember us talking about elasticsearch integration? [16:19] balloons: http://termbin.com/mq29 [16:21] Budgie^Smore: in what context? [16:21] http://termbin.com/rdvo is cleaner :P [16:23] lazyPower, from memory (so don't quote me on this, slept a few times since then), you were already testing it as part of the CDK bundle [16:23] Budgie^Smore: when we first released the CDK bundle we had elastic beats + elasticsearch + kibana as part of the bundle [16:23] so the integration points already exist [16:24] so add charm, relate and deploy? [16:24] That should be the basics, yeah [16:24] Budgie^Smore: give me a bit i'm ina meeting [16:25] Budgie^Smore: but you should be able to deploy beats-core and add the beats=>kube relations, configure the beats and that should be the basics of the operation though. [16:25] there's a new elasticsearch charm incoming from bdx that targets ES 5.x [16:25] might be worth tracking that work as well [16:25] no worries, about to head to the office [16:30] cnf, sorry just a moment [16:30] sure [16:44] cnf, which model are you in? [16:44] controller [16:45] or i should say admin/controller i guess [16:49] cnf, so my idea isn't possible to avoid you bootstrapping another controller [16:49] :( [16:50] I beat on it a bit, but apart from manually placing the tools and editing the db, it's not going to happen [16:50] do you know what the bug is? [16:50] i don't fancy doing this again next time :/ [16:50] cnf, I'd encourage you to post to the list to get feedback on others in locked down maas environments on how they manage things [16:50] the collective knowledge is better than me [16:51] i'm not quite sure what to post [16:51] besides "it doesn't work" [16:51] cnf, well you shouldn't be doing any of this. upgrade-juju should just work [16:51] yes, it should :P [16:51] but if i don't know why it doesn't, i don't trust it will work next time [16:52] cnf, can you ssh into the controller [16:52] yes [16:52] we actually never proved the controller has a good proxy [16:52] juju ssh -m controller 0 [16:52] then try grabbing from streams.canonical.com [16:53]

Index of /

[16:53] cnf, when you bootstrap, the agent is uploaded by the client to the controller. The deployed machines then get the agent from the controller. So it's possible the controller doesn't have access at all [16:53] curl can get to it just fine [16:54] unless juju just ignores the set proxies [16:54] via proxy yes? [16:54] yes [16:54] are we sure the proxy set in juju is correct? [16:54] uhm, no? [16:54] i don't know what "correct" is besides "juju model-config" [16:55] it's set in /etc/apt/apt.conf.d/95-juju-proxy-settings [16:55] and it's set in the env [16:55] so i don't know what else should be set [16:56] unless the tool that does the downloading from the stream ignores this? [16:57] ahh, we did confirm juju status shows upgrade availible yea? [16:58] so the proxy works, the values set in model-config match your expectations, and juju can kind of see the upgrade, since it tells you it exists [16:58] yes [16:58] controller dsmaas-controller dsmaas 2.1.1 upgrade available: 2.1.2 [16:59] right [17:02] cnf, did you bootstrap with those proxy settings? [17:02] yes [17:03] did you show me juju model-config -m controller? [17:03] whom is conjure-up targetted at? [17:04] (for example I seem to be missing the point of deploying OpenStack with one command, given how complicated software it is, and how much planning ahead is required) [17:04] is this for demos and lab? [17:05] stokachu, ^^ [17:05] kklimonda: getting the initial base infrastructure stack deployed successfully is one thing, maintaining it over time is another [17:06] balloons: yes, but i can show it again :P [17:06] cnf, ty :-) [17:06] balloons: http://termbin.com/69cn [17:07] kklimonda: the ability to spin up openstack, or any other complex software stack with a single command is really the polish on the block [17:07] Anyways, if you wouldn't mind posting to the mailing list about your issues, that would be lovely. It would also be useful for you to ask how folks best handle upgrades in these situations, though, it should work. [17:07] bdx: yes - but that's my point. what does it bring to the table over juju? [17:08] it just seems to be another layer of indirection [17:08] kklimonda: its a layer of useability [17:09] i don't know the mailing list [17:09] i'm generally not a fan of mailing lists [17:10] cnf, ahh. Well, you certainly don't have to post. I can file a bug about it, but it may be useful for you to do so, so you can track it: https://bugs.launchpad.net/juju/+filebug [17:11] what do i call it? [17:11] juju upgrade-juju fails ? [17:11] kklimonda: I don't want the users of my charms maintaining yaml configs all over the place, its easier for me, and my users, if I create spells for theses infrastructure stacks so as the deploys can be interactive and intuitive [17:11] it's so generic :/ [17:11] cnf, juju upgrade-juju doesn't honor proxy settings [17:12] kklimonda: especially for openstack .... your config.yaml for an openstack bundle can end up being 1000+ lines [17:12] I wonder if I can repo quickly actually [17:12] bdx: been trying to deploy openstack with juju for 3 weeks, little polish to that :( [17:12] balloons: are you sure it's the proxy ? [17:12] cnf, did you try conjure-up, heh? [17:12] bdx: yes, but that configuration (and the decisions behind it) still have to be made [17:12] cnf, your log indicates you don't get anything back from the version check [17:13] cnf, it should return, nothing to upgrade, or XXXX found. You get nothing and it drops to trying a locally built one [17:13] cnf: I have successfully deployed openstack with juju in a myriad of different ways, let me know if you need some insight, I would be glad to give you some pointers if needed [17:14] bdx: i can't get anything sensible out of juju so far [17:14] kklimonda: As another side of the tale, we find the extra layer of abstraction to give little value so we stick to Juju. [17:14] can't even upgrade it, it seems [17:15] kklimonda: conjure-up also allows for different types of provisioning automation not available via vanilla juju [17:15] kklimonda: e.g. lxd-profiles [17:16] balloons: https://bugs.launchpad.net/juju/+bug/1674759 [17:16] Bug #1674759: juju upgrade-juju doesn't honor proxy settings [17:16] sorry if it's a bit susinct, i'm tired and hungry atm [17:16] cnf, I know we didn't want to bootstrap a controller, but I would encourage you to try upgrading with https://jujucharms.com/docs/2.1/models-migrate. It's likely how you should manage models in production anyway [17:16] ie, bootstrap a 2.1.2 controller, then migrate your models. Finally, teardown the 2.1.1 controller [17:16] balloons: well, that assumes you have spare machines to do this :P [17:17] cnf, yea, that's the blessing and curse [17:17] i'll have to do that twice [17:17] once away, once back [17:17] why twice? [17:17] and bootstrapping a new machine takes 20 minutes on its own [17:17] balloons: once to a machine, then back to the vm [17:17] kklimonda: look at the example of the kubernetes spell for lxd provider .... conjure-up lends to some really cool extended functionality where you can use pre/post scripts to modify things outside of your juju environment [17:18] conjure-up looks like even more magic ontop of juju [17:18] i can't get juju to behave sanely, i don't want even more magic personally [17:18] cnf, kklimonda: https://github.com/conjure-up/spells/blob/master/canonical-kubernetes/steps/00_pre-deploy [17:19] if allows you to do cool and important things that you guys aren't really looking at or taking into account [17:19] bdx: that looks ugly, what's that for? [17:20] ty for the bug report cnf [17:20] cnf: that is a conjure-up pre-deploy script - it runs prior to conjure-up deploying your juju stuff to configure the lxd profile https://github.com/conjure-up/spells/blob/master/canonical-kubernetes/steps/lxd-profile.yaml#L6 [17:20] o,O [17:21] cnf: things like this (customizing lxd profiles) are a huge hassle as well as a blocker for people trying to deploy things to containers that need special modifications === tinwood_ is now known as tinwood [17:22] and how do i install conjure-up? [17:23] cnf, kklimonda: if you aren't wise to all of things you would need to do to lxd to make it support deploying kubernetes ... these things are encapsulated in the conjure-up workflow/spells [17:23] i don't care about lxd, much [17:23] cnf: are you familiar with snaps? [17:23] no [17:23] cnf: you are trying to ride the horse without knowing well the saddle [17:24] ... [17:24] bdx: I mean, I understand what you're saying but someone will have to understand all this once the deployment is done [17:25] conjure-up is available on both Ubuntu Trusty 14.04 LTS and Ubuntu Xenial 16.04 LTS [17:25] no ubuntu here [17:25] work is underway to port it to MacOS, but that's still pending. [17:25] cnf: you are going to have a tough time trying to run all/any of this from non Ubuntu Xenial [17:25] bdx: so no thank you [17:26] cnf: conjure-up is delivered as a snap - which isn't a thing on non-ubuntu systems [17:26] bdx: thats a lie [17:26] :-0 [17:26] bdx: snap is supported on centos, debian, arch, sles [17:26] oooh [17:26] thanks LP! [17:26] bdx: <3 happy to alley oop some knowledge [17:26] cnf: srry [17:26] so you don't know your own tools... [17:27] either way, as long as juju isn't reliably, no amount of magic on top of it will make me trust it [17:28] cnf: I'm just a community member, I don't work for canonical .... I try to know the tools as well as possible, I didn't know snap was supported across other os's srry [17:29] kklimonda: have to know what? [17:29] you don't deploy / run openstack without knowing the details [17:29] cnf: what details? [17:30] yeah, you don't run openstack, i take it [17:30] balloons: where is the mailing list, btw? [17:30] yeah ... I have been since it was released to the public ... before there was automation [17:31] bdx: for example, even if conjure spell manages apparmor profiles on host machines for the user, that doesn't mean this is removing any burden - you still have to understand what has been changed, and why, or you'll have a bad day later. [17:31] cnf, https://lists.ubuntu.com/mailman/listinfo/juju [17:31] bdx: in production, with people using it? [17:31] cnf: yea man [17:31] deployed via juju [17:32] and you suggest people deploy openstack without knowing about how it works? what network is used for that [17:32] or what hardware is holding ceph data etc [17:32] cnf: that is the admins responsibility to track that information [17:32] balloons: oh, you need to join it? [17:32] o,O [17:32] cnf: juju/maas really help there too [17:32] cnf: our goal is not to remove the knowledge requirement for management over time, its to remove the knowledge barrier to get started. We still advcoate you should read teh book at least once, we want to abstract away that requirement though. [17:32] cnf, well juju@lists.ubuntu.com if you just want to send a mail. But replies posted only to the list obviously won't reach you [17:32] bdx: for the past 3 weeks, juju / maas has been gettin in my way [17:32] not helping me [17:34] cnf: sorry to hear that, let me know if you need some guidance [17:34] to be quite honest, if i can't get something working soon, my advice here will be to NOT use juju / canonical for openstack deployment [17:34] bdx: feel free to start with https://bugs.launchpad.net/juju/+bug/1674759 [17:34] Bug #1674759: juju upgrade-juju doesn't honor proxy settings [17:35] kklimonda: so if you are deploying kubernetes to lxd, you are probably just looking for a POC anyway right? [17:36] bdx: well, conjure-up seems like a pretty cool POC/lab deployment tool [17:36] I don't deny that :) [17:36] balloons: thanks for the help so far, i'm calling it a day [17:36] i'm tired, frustrated and hungry [17:36] balloons: not a good state to deal with this :P [17:36] cnf, you are most welcome. Sorry to hear about your troubles :-( [17:38] kklimonda: I can't speak for others, or for how the tool is intended to be used, but I find it most useful for getting my initial stack deployed, then I switch over to pure juju for the rest of the lifecycle ops [17:42] cnf: did you try `juju upgrade-juju --build-agent` [17:43] bdx: seems like a legit bug in the upgrade command not able to go through the proxy that's set [17:43]