=== CyberJacob is now known as zz_CyberJacob === beisner- is now known as beisner === zz_CyberJacob is now known as CyberJacob === CyberJacob is now known as Guest50303 === bradm_ is now known as bradm [07:02] Good morning Juju world! === Guest50303 is now known as CyberJacob [09:21] I have a user attempting to run juju on MacOS Sierra [09:21] the juju client is crashing and cannot run anything [09:40] icey-travel: I do not know how to help you. But do we have a ticket to track this issue? Have others seen this problem or are you the first one? [10:08] kjackal: it's this bug: https://bugs.launchpad.net/juju/+bug/1633495 and is fixed comitted, marcoceppi says that it will release with 2.0.1 [10:08] Bug #1633495: Panic MacOS Sierra [10:09] icey-travel: thank you [11:20] can anyone help SimonKLB in #juju-dev with a question on charmhelpers? [11:21] we can take it here or in pm, whatever works [11:41] i've created a merge request for a possible fix to the issue, please see https://code.launchpad.net/~simonklb/charm-helpers/include-empty-config-options/+merge/309105 === mbeierl2 is now known as mbeierl === shawniverson is now known as Guest21243 === tris- is now known as tris [12:47] Hi, is it possible to make bootstrap command more verbose? [12:49] deanman: Have you tried --debug flag? [12:51] anrah_: Thanks that works, i remembered there was such a command on pre-2.0 but just couldn't find it on newer documentation. === mhall119_ is now known as mhall119 === Tristit1a is now known as Tristitia [13:27] lazyPower: ping, can you give SimonKLB any tips on testing across a relation broken/back again setup please? [13:27] lazyPower: how do you all test that in a way that it's triggered, wait the right amount of time, etc. [13:28] rick_h_ : so to ensure i'm understanding the scenario - We're looking for how to test removing a relationship, and making amulet wait until we're certain the relationship has been fully broken? [13:29] lazyPower: want to take it in pm or do you want me to spam here? :) [13:29] SimonKLB spam away my friend [13:29] SimonKLB: feel free to spam here [13:29] lazyPower: <3 ty [13:29] this will be useful to others i'm sure [13:29] so, in amulet there is a relation function that i've been using to check whether or not the relation exist [13:29] it throws an exception when the relation is gone [13:30] you know SimonKLB, i've tested the daylights out of the inverse... adding relations. I dont think i've removed a single relation with amulet code [13:30] but, i suspect we can do this like so [13:31] * lazyPower double checks the amulet api docs before banging out a response [13:31] yea, the problem seem to be adding a relation too quickly after removal [13:31] for reference, thats over at: https://pythonhosted.org/amulet/ [13:31] well, this seems implementation specific, but you can wait_for_messages() [13:32] assuming your charm will emit a status message that its either waiting on said relationship (Required) or that its handled some operation on that relationship (optional) [13:32] yea that is actually what i did first, but in that case the message showed up too quickly and the relation was still dangling, adding the relation again would simply fail with "this relation already exist" [13:33] thats when i started using the relation function in amulet [13:33] https://github.com/juju/amulet/blob/master/amulet/sentry.py#L282 [13:33] using this exception: https://github.com/juju/amulet/blob/master/amulet/sentry.py#L316 [13:33] this worked until recently [13:33] Ah and its a nice and generic exception too [13:33] yea i checked the message haha [13:33] so it requires a fancy diaper [13:34] so im not sure what changed, but the problem in this case is not that its not removed yet, but rather that the *-joined hook is not firing again after readding the relation [13:34] hmm, question [13:35] are you absolutely certain the states are being removed that execute in that context? [13:35] and its not a case of state pollution keeping your code from re-executing? [13:36] i recently encountered that, and i wasn't removing the .connected and .available states i set during the joined on the -departed phase of the relatinoship. [13:37] the interface hasnt changed as far as i know, the available state is being removed on broken/departed [13:37] the problem is that the joined hook is not being fired and because of that the available state is never activated again [13:38] doing stuff slowly by hand works fine [13:39] SimonKLB : I think at this point its prudent we file a bug [13:39] we dont have the plumbing in amulet to properly introspect this that i can see [13:39] yea i talked to rick_h_ quickly and as he said this is a huge async operation [13:39] so things could be still left dangling on the remote end etc [13:40] it really is, it looks like we've only really accounted for adding relations and then probing that relationship dictionary of values [13:40] we haven't added support for truly knowing if the relationship is gone, and how to identify when that operation is complete [13:40] and then we have the added complexity of why your code isn't re-executing [13:41] if there was a way to truly know when the relation is gone and cleaned up in juju i think the problems would go away [13:41] well, you have that nice and generic exception [13:41] it would be neat if the add/remove calls would queue up [13:41] but that seems like a hack [13:41] yea i meant in juju itself [13:41] not in amulet [13:41] yeah, similar to the d.setup() routine [13:41] oh [13:41] i was going the oppostie direction and saying that lies in the test tooling :P [13:42] haha, it might be possible for amulet to check it, but im not sure there is any way to probe juju itself for the relation status [13:42] man i need more coffee, i'm typoing at increased rates [13:42] sure there is [13:42] juju has this fancy relationship data structure these days in status output [13:42] we could simply parse that to determine if the relationship has been removed before/after a .realtionship() operation [13:42] how well does it reflect the "real" status? [13:42] at least i think we could, thats my over simplified analysis [13:43] for example executing relation-list could give you a false image of what is really the case [13:44] or atleast that seem to be the case right now [13:44] SimonKLB - lets start by filing a bug against amulet with what you're wanting to do, so we have a captured feature request [13:44] alright, ill do that [13:45] but hmm.. [13:45] SimonKLB - from there, we'll investigate and file bugs against the other projects [13:45] its not really appropriate to say that amulet isnt providing real information about the relation since its actually correct in that the relation is gone from the clients perspective [13:46] there is not a problem adding the relation again after the amulet function says that the relation is gone [13:46] its the fact that the joined hook is not being fired again if you add the relation too quickly that is the issue [13:46] and that's not really amulet's fault, imo [13:46] yeah but if we dont have a bug to start somewhere, we're not likely to fix the problem. [13:47] yea but could it be better for me to fill the bug in some other project? [13:47] I wont be upset if the bug doesn't' get filed against amulet if thats what you're asking ;) [13:48] haha, alright, ill write something to start off with there [13:48] and good shout on testing this SimonKLB [13:48] i have a few cases i can think of we should be testing removals, that we arent. so we'll hit this pretty soon as well i'm sure. [13:49] that, and this https://github.com/juju-solutions/layer-basic/issues/79 is basically the only things left for me to send our charm for review :) [13:49] so im trying to sort it out as good as i can [13:59] balloons: good morning! [13:59] good morning [13:59] balloons: I've been on leave for a month, and I'm back and ... want to totally do pure snapped juju with snapped lxd, anyone doing this yet? [14:00] SimonKLB we'll do what we can to help :) [14:00] Hey guys. Is there an alternative to PPAs for hosting the .deb pacakges that are deployed by the charms? That I can use as default [14:00] jcastro, welcome back. Did you try the snap on fedora? [14:00] lazyPower: created the issue here https://github.com/juju/amulet/issues/154 [14:01] which one, lxd or juju? [14:01] jcastro, we need a LXD interface for the juju snap [14:01] jcastro, juju [14:01] balloons: oh ok, so it won't work right now [14:01] jcastro, it will (should( in devmode [14:02] is a lxd interface milestoned? [14:04] jcastro, no [14:04] jcastro, but others would also like one [14:04] make it so \o/ [14:04] the onus in on the consumer to create it [14:05] jcastro.... build the interface! ;) [14:06] * jcastro tries to find a place to hide [14:07] I have a k8s video to make today [14:07] balloons: is us-east-2 looking good? I'm debating doing it someplace new and shiny. [14:07] jcastro, I was playing it it in the other day [14:07] OH > VA right? :-) [14:08] that totally depends on who is peering closest to my ISP. [14:08] heh. [14:10] us-central DC's ftw [14:10] hey jcastro [14:11] i have a ton of assets i created for use with my slides that match the branding of jujucharms.com - where would you like me to offload these for others to use? in teh talks repo? [14:11] yeah, I'm not sure we have an assets directory there though [14:11] might need to make one? [14:11] let me push these to dropbox and you tell me if they're worth having in there [14:12] either someone has been handling PRs while I was gone, or no one has been submitting things, heh [14:12] a little of column a, little of column b [14:12] Baqar: I wonder if resources would help you here [14:12] Baqar: is it like workload debs of the service or is it something like security updates for the OS or something? [14:16] I'm running into some issue with local VM bootstrapped with "localhost". Controller gets bootstrapped just fine and i can see the LXD instance running but when trying to deploy a service the new machine never get to start. I'm suspicious of proxy settings, without proxy same workflow works just fine. Any hints how to see what's happening on subsequent LXDs not booting? [14:17] For some reason also juju debug-log won't show any output === wolverin_ is now known as wolverineav [15:23] jcastro thanks. Looking at that. [15:24] deanman on the hypervisor search for relevant logs in /var/log/juju/ === kwmonroe_ is now known as kwmonroe [15:27] jcastro - late on this one, but here's the dropbox link i promised earlier https://www.dropbox.com/sh/bc7ihzr6hfztfwy/AAAB9Vfqsql9VesKpJqEi46xa?dl=0 [15:42] cory_fu - when you get a chance, it would be good to get your opinions on https://code.launchpad.net/%7Esimonklb/charm-helpers/include-empty-config-options/+merge/309105 [15:42] marcoceppi cc'd ^ [15:42] i'm about to land a hack-in-place work around to this issue, but here's the root that simon has identified [16:00] lazyPower cory_fu this bug is basically asking for the opposite, that unset config values should be removed from hookenv.config() https://bugs.launchpad.net/charm-helpers/+bug/1630706 [16:00] Bug #1630706: Config options remain in hookenv.config() even when unset [16:01] before you merge you should make sure that it won't be a problem [16:02] cmars should probably get in on the discussion :) [16:03] yeah, its a peristence cache issue [16:03] caching, oh you gloriously misunderstood thing [16:03] * lazyPower hugs it === Guest57444 is now known as rcj === rcj is now known as Guest61926 === Guest61926 is now known as rcj === zerick_ is now known as zerick [16:20] SimonKLB, LGTM. i'd typically use `if config.get('key'):` anyway, so this works fine. thanks! [16:56] PR on juju-deployer https://code.launchpad.net/~johnsca/juju-deployer/series-fix/+merge/309143 to address failures like http://8.19.32.215:8000/test/ca4e3cae9a7a40b3ba04fc368cae1664 where it tries to deploy with the wrong series [16:57] kwmonroe, kjackal, petevg, marcoceppi: ^ [17:08] cory_fu: +1 for working code that does what it's supposed to do. I think that chunk-o-if-statements is a great candidate for moving into a utils module in python-libjuju at some point -- it would be nice to have one and only one place that implements charm store url parsing in Python ... === frankban_ is now known as frankban|afk [17:09] petevg: Very much agreed. It would be great to have deployer phased out by libjuju [17:10] +1 :-) === ahasenac` is now known as ahasenack === ahasenack is now known as Guest91762 [17:33] Baqar: You mean to ssh into the controller and check the logs there? [17:34] hi guys! I'm using juju 2 here. I deployed the landscape-client charm and added a relation between it and some services that are running on xenial machines. But it is failing to add a relation on services running on trusty machines: ERROR cannot add relation "landscape-client:container stoqwiki:juju-info": principal and subordinate applications' series must [17:34] match [17:35] what can I do to workaround that? I really wanted to add the landscape-client to those trusty machines [17:42] hackedbellini: you'll need to add it a second time with the trusty series [17:42] hackedbellini: and relate that on it's own [17:44] rick_h_: you mean I have to duplicate the charm? [17:45] hackedbellini - correct, you'll need to deploy it with --series=trusty [17:45] hackedbellini - on the gui under the charm config, you can select the series the subordinate should be if its a multi-series subordinate [17:46] a subordinate charm deployed into a model can only occupy a single series boundary, if you wish to have subordinates related to both, you will have to duplicate the charm one for each series you wish to deploy. [17:47] lazyPower: hrm ok. That is really sad since I have to duplicate every change I do to the subordinate charms, but I'll accept the solution for now [17:47] also, why can't I remove a subordinate relation? [17:49] hackedbellini - you can, the GUI angrily denies you though. [17:49] oh, nevermind. I couldn't from the gui but I could make it work on a terminal [17:49] hackedbellini - if you remove the relation on the CLI it works as expected. [17:55] Anyone using juju 2 behind proxy? It seems that I'm able to bootstrap just fine but deployment of services fail as well as several other juju commands like juju debug-log and ssh [17:57] deanman - yep, there's model proxy configurations you need to set [17:57] deanman juju model-config and juju model-defaults should help you [17:57] deanman - also be aware, that there are some charms that haven't been properly tested behind a proxy, and some of those charms may require bugs to be filed to ensure the charm author is aware of network limited environments and that they are using commands that may/may-not require proxy flags if defined. [17:57] lazyPower: I'm passing proxy configuration during bootstrap command with --config and i'm setting all three options, http/https/noproxy [17:58] deanman - sure, did that land in your model-defaults onfigurations? [17:58] Well it must have cause controller was able to update/upgrade and download any required software [17:59] cory_fu: MP is +1 from me too, though i'm curious your thoughts on machine series constraints. if i define charm: cs:ubuntu; to: 1 and define machine: 1: series: trusty, should there be some glue that adds appropriate series data if the "to" machine defined it? i'm not sure if you'll have machine data at Charm.from_service() -- just curious if you think that's what *should* happen. [17:59] deanman but we're interested in the units, not the model-controller at this point. Can you humor me and check model-config / model-defaults to see if your http/https/noproxy is configured there? [18:00] lazyPower: Sure, let me check this [18:00] kwmonroe: My inclination is to let it break. If you did that on the CLI, I'm fairly certain that Juju would reject the deploy command because you didn't give the series (even though it could infer it from the machine, as you say) [18:01] fwiw deanman, i do this on my juju2 stuffs behind a proxy. as lazyPower mentioned, bootstrap --config sets the controller config, --model-default sets model config: [18:01] juju bootstrap --config http-proxy=$http_proxy --config https-proxy=$https_proxy --config no-proxy=$no_proxy --model-default http-proxy=$http_proxy --model-default https-proxy=$https_proxy --model-default no-proxy=$no_proxy [18:01] ^^ in that case, i have my proxies defined as envvars [18:02] kwmonroe: well it must be that then, it wasn't clear to me that --config would only pass these to the controller configuration only [18:03] ack cory_fu. works for me. i much prefer the explicit cs:series/charm syntax in bundles anyway. [18:06] lazyPower: Are the charmbox docs up to date wrt juju 2 and lxd provider? it's linking ~/.juju instead of .local/share/juju [18:07] aisrael - MASTER is known to be out of date as its still targeting the 1.25 release [18:07] aisrael if you swap to devel it should be more current [18:07] aisrael mbruzek started cleaning that up last week, we should have some new docker image tags this week/next week, as 2.0 went GA and we're lagging behind on getting that work complete [18:08] lazyPower: gotcha, thanks [18:09] kwmonroe: Out of curiosity and while i'm trying to test your suggestions, with only --config for the controller shouldn't it respond to juju debug-log afterwards? [18:14] deanman: i'm not 100%, but i think all models need to send their log data to the controller model for 'debug-log' to pick it up. so if you had a model that wasn't capable of reaching the controller, that might cause debug-log problems. you could verify that debug-log is at least working on the controller model with "juju debug-log -m controller" [18:19] kwmonroe: I think my confusion came from https://jujucharms.com/docs/2.0/models-config. Is the following sentence right? "These values can also be passed to a new controller for use with the default model it creates. To do this, use the --config argument with bootstrap:" [18:23] deanman - i think that was teh behavior at one time, but looks to be no longer the case. [18:23] rick_h_ - can you confirm/deny the alleged? [18:23] * rick_h_ reads backlog [18:24] i'm checking on that too.. bootstrap is taking it's sweet time.. [18:24] lazyPower: no worries, as long as we correct it for future newbies like me ;-) [18:25] deanman: hmm, so I would expect the proxy config set on bootstrap to carry through and be inherited to all models [18:25] deanman: you can validate that with the model-config command I believe. [18:25] deanman: it shows all values and if they're inherited or not [18:25] deanman: what would be good would be the output of those hanging commands with the --debug flag [18:25] deanman: to see where it's getting and hanging [18:26] deanman: we have a bunch of use behind proxies and some tests so I'm fairly confident that it should work, but as you know, the config can get tricky because of the various ways it can get setup [18:26] Ok, so I'm running again my juju2 setup with proxy, again using only --config to pass proxy, bootstrap completes just fine [18:26] model-config indeed reports that these proxy settings are inherited in model [18:26] deanman: k, and can you run "juju model-config" from the default model? [18:26] deanman: k [18:26] deanman: and do some commands like 'juju status' work? === ubuntu is now known as Guest22446 [18:27] * lazyPower scratches head [18:27] juju status works fine, juju debug-log just shows nothing [18:27] deanman: k, can you run debug-log with the --debug [18:27] ? [18:28] it is stuck right after API hostnames unchanged - not resolving [18:28] juju ssh 0 does not work either, it gives an "ERROR machine 0 not found" [18:28] deanman: k, juju switch controller [18:28] deanman: and then try it [18:29] deanman: in juju2 each model is independent and so the machine number needs to exist in the currently active model [18:29] after switching debug-log works fine [18:29] and ssh [18:29] deanman: k, so this is a case that to do controller-level operations you need to be on the controller [18:29] deanman: the default model doesn't have anything worth talking about or to [18:29] ooooohhhhh...... [18:30] aha! moment [18:30] deanman: <3 let me know if you run into any other issues. Thanks for picking up and learning the new stuff in juju2. We hope it works well for you. [18:30] lazyPower: kwmonroe ^ fyi [18:31] another question. My lxd deployment is using the .localdomain domain on the network. How can I change it to something else (say .foobar)? [18:31] I couldn't find the configuration for that [18:32] rick_h_ sorry about that, i clearly misunderstood what was happening. [18:32] rick_h_: ok switching back to default model, model-config shows proxy settings just fine, juju deploy wordpress executes, but new lxd never gets created. How can i debug that? [18:34] rick_h_: basically the machine status from pending goes to down and then everything hangs, charm is never deployed [18:34] deanman: check juju status --format=yaml [18:34] deanman: guessing something isn't right with juju talking to the lxd endpoint? [18:34] deanman: might have to check debug-log and the lxd logs to see if there's something up there. [18:34] hackedbellini: sorry, there's not a custom domain flag in juju for that. [18:35] fwiw deanman rick_h_, using --config during bootstrap alone will not carry forward to new models.. so the docs are correct in the --config will take care of the 'default' model, but just be aware you'll need --model-default if you want those proxy settings on other models. http://paste.ubuntu.com/23375527/ [18:35] kwmonroe: :( that sucks [18:36] rick_h_: "agent is not communicating with the server" Could it be that proxy is not forwarded after all to model as kwmonroe mentions ? [18:36] deanman: what are you bootstrapped against? [18:36] deanman: is this maas, or some cloud or? [18:36] rick_h_: localhost [18:37] deanman: what's your no_proxy setting? does it include the lxd network? [18:37] deanman: so you need a proxy on localhost? So the thing here would be that lxd is running on localhost or some address. [18:37] deanman: so the question is, does this need to be not proxied? [18:38] kwmonroe: 127.0.0.1,localhost,10.0.3.1. Last one is the range i configure to lxdbr0 [18:40] rick_h_: I fail to understand your question, my setup is networking behind corporate network, using mac as host and a xenial as guess which has proxy configuration. To make juju localhost setup work don't i have to also pass proxy to model so that every lxd created (host->guest->lxd) is able to retrieve stuff from net? [18:42] kwmonroe: Is that no_proxy config of mine right? I have a pre-configured lxdbr0 file which i provision before bootstrapping so the range is chosen by me. [18:43] deanman: yea, I don't run the proxy much so I'm not sure tbh. I'm guessing there's something in there that's got juju trying to proxy vs looking for the local lxd api endpoint, but I'm probably guessing wrong there [18:43] deanman: best thing is to hit the logs and see [18:43] deanman: ssh to the host and check the main host lxd logs and then the debug-log or go look at the /var/log/juju/xxxx.log on the machine 0 [18:44] rick_h_: that's the source of my problem, debug-log does not give any output on default model [18:44] deanman: right, but don't look there. [18:44] deanman: the controller handles everything so the logs on there are the important ones [18:44] rick_h_: you mean switch to controller ? [18:44] ok let me have a look, maybe i could find something more usefull for debugging conversation === X-Istence is now known as x58 [18:52] rick_h_: should i pastebinit ? [18:53] deanman: sure [18:57] deanman: fwiw, i "no_proxy" the *entire* lxdbr0 subnet (see my noproxy config in the earlier pastebin: http://paste.ubuntu.com/23375527/). just guessing here, but what you have probably allows containers to talk to the controller (10.0.3.1 is the controller), but perhaps it's not enough for the controller to talk back to the container (which would be a 10.0.3.x) [19:00] deanman: i do this in my env with: export no_proxy=`echo localhost 127.0.0.1 10.245.65.108 10.0.3.{1..255} | sed 's/ /,/g'` <-- note the 10.245.65.108 is my own machines eth0 address, as i don't want communication to/from myself and the controller/containers proxied either. [19:01] kwmonroe: that could be it, sorry for some reason pastebinit fails with some locale problem. I will try your configuration [19:02] deanman: wouldn't it be great if we could just trust the internet, get onto ipv6, and get rid of proxies all together? [19:02] (hint: yes) [19:03] kwmonroe: if only then i would be able to focus on how to present juju2 on colleagues instead of fighting this :-) [19:09] :) === Guest91762 is now known as ahasenack === ahasenack is now known as Guest95404 [19:30] I'm looking for some docs on setting the architecture when bootstrapping into a maas cluster [19:31] I have juju cli on s390 and would like to bootstrap into an x86 maas.. but hitting "ERROR failed to bootstrap model: cannot start bootstrap instance: no matching tools available" [19:32] vmorris - i've hit that trying to bootstrap x architecture [19:32] specifically i was trying to bootstrap ARM from AMD64 [19:32] lazyPower yea, that sounds the same.. were you able to work around it? [19:33] vmorris - to make things less obvious, i wasn't able to build tools for that arch locally and use --build-tools. [19:33] not at present, i discovered we dont support 32 bit arch arm architecture, which is what i have available to me [19:33] so i promptly abandoned the effort, but i see that might have been premature [19:33] vmorris - i would encourage you to file a bug, and then post on the public mailing list so we can raise awareness of the situation. There might be a clean workable fix that i'm not aware of. [19:34] lazyPower: thanks, i'll consider it. If I don't make any progress in the next hour or two I will likely do so [19:35] vmorris sorry i didn't have better news, but i can validate your findings for you [19:35] as i too have encountered that [19:36] yes, thanks for that confirmation.. i figured it must be my architecture mismatch, but I expected there would be a way to work around it -- as you mentioned there likely is [19:40] kwmonroe: i tried setting a full range of ips for no_proxy as per your setup. I could confirm with model-config but still i get the "agent is not communicating with the server". For some reason pipeing into pastebinit does not give a full URL with my logs so i can share more data. [19:43] lazyPower: wouldn't I just run --constraints arch=amd64 during bootstrap? (i'm waiting for a failed bootstrap to exit to try) [19:44] thats worth a try, but i was unable to get that to work [19:44] ok === alexisb is now known as alexisb-afk [19:53] kwmonroe: http://pastebin.com/g2M4ZZN3. A lot of "connection refused". This is my model-config http://pastebin.com/2GatAEb3 === Guest21243 is now known as spammy [20:02] Also "machine-0: 20:00:11 INFO juju.tools.lxdclient no image for ubuntu-trusty found in https://streams.canonical.com/juj u/images/releases/" and "machine-0: 20:01:21 ERROR juju.provisioner cannot start instance for machine "1": image not imported!" seem related to my problem. [20:30] lazyPower: yeah this works 'like a charm': $ juju bootstrap --constraints arch=amd64 maascloud maascloud-1 [20:31] thats *awesome* vmorris [20:31] now i feel the need to go back and beat on my arch problem with a larger hammer [20:31] good luck :D [20:46] juju status [20:52] deanman : Failed to connect to controller. are you sure its bootstrapped? (this is a pun ;) and a bad one at that) [20:57] lazyPower: :-D === CyberJacob is now known as zz_CyberJacob === mup is now known as 21WAAGPR0