=== cmars` is now known as cmars [06:10] /msg NickServ identify 561107 [06:14] junaidali: time to change your password :-) [06:14] blahdeblah, lol yeah [06:14] a whitespace, ruined me [06:14] :D [06:14] You might want to consider making it 3 or 4 times longer, while you're at it. :-) [06:15] junaidali haha [06:15] and add a few random characters instead of just all numbers heh === frankban|afk is now known as frankban [07:37] Good morning Juju world [07:57] I'm having an issue bootstrapping on lxd with juju 2, the end of the log is at http://pastebin.ubuntu.com/23377823/ [08:12] nevermind, looks like https://bugs.launchpad.net/juju/+bug/1547268 is it, and referenced iptables change fixed it [08:12] Bug #1547268: Can't bootstrap environment after latest lxd upgrade <2.0-count> [08:41] Is there a chat history of this channel persisted somewhere? [08:48] deanman: https://irclogs.ubuntu.com/2016/10/24/ [10:08] Hey everyone, I have this strange behavior. When i apt-get install amulet it also brings in the python-jujuclient package which is not "bundletester friendly". This fails: Command (juju api-endpoints -e localhost-localhost:admin/default) [10:09] Have you seen this before? [10:10] bugg@tom-laptop2:~$ sudo apt-get install amulet [10:10] Reading package lists... Done [10:10] Building dependency tree [10:10] Reading state information... Done [10:10] E: Unable to locate package amulet [10:10] nope :) [10:11] magicaltrout: problem solved! [10:11] okay... well if I add the ppa lets see what dependencies it brings in [10:11] yeah i can vouch for you [10:12] magicaltrout: I probably have some left over library from juju 1.25 [10:12] i'm sure i used to pip install amulet [10:25] magicaltrout: Got it, thanks! [10:57] magicaltrout: to it seems the apt repo has slightly older versions of jujuclient than pip [10:57] I had to apt-get install the jujuclient and then pip upgrade it === Guest95404 is now known as ahasenack === ahasenack is now known as Guest59736 [11:35] Any lightweight xenial charm to suggest for quickly debugging environmental issues? [12:09] Which juju provider do you use for amulet testing? LXD, AWS, or OpenStack [12:09] ? === mhall119_ is now known as mhall119 [12:50] i've got a maas deployment here, and i was able to bootstrap a juju controller, but when i go to deploy charms it hangs [12:50] agent status stays on 'allocating' [12:50] and message is 'waiting for machine' [12:54] i'm not seeing much in the juju --debug output [12:54] where else can i look for what is hanging up? [12:55] machine-0.log on the controller [12:55] may be something like a constraint on the machine that means nothing in the maas matches [12:56] eg constraint on maas name N, and the controller already took that machine [13:05] mgz: hmm, okay i'm pouring through it now [13:08] vmorris: try juju status --format=tabular to get some additional machine provisioning feedback [13:08] okay [13:09] things look okay in the machine-0.log until ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down [13:10] that's also actually fine [13:11] and it seems the maas provider logging at debug isn't great, I only have 'juju.provider.maas environ.go:1035 started instance "..."' [13:11] and not any more details than that before [13:14] tabular format didn't help with additional info === ubuntu is now known as Guest74698 [13:28] yeah nothing's really popping out here [13:28] vmorris: sorry, meant format=yaml [13:28] rich_h_ mgz: I'm going to pastebin the machine-0 log ERROR messages, if that's interesting [13:29] ah okay, let me try this [13:29] * rick_h_ drinks more coffee [13:30] nope, nothing more interesting there, machine-status: pending [13:36] mgz rick_h_: please see https://gist.github.com/anonymous/a69b42d950075f63876f443701bc37d9 [13:36] vmorris: if you can repro easily, it's probably worth setting provider.maas logging to trace [13:36] it runs for a bit, then gets in an error loop that ran overnight, so the latter 80% of that doc isn't useful [13:36] mgz ah okay, i can try this [13:37] as in, `export JUJU_LOGGING_CONFIG="=DEBUG; juju.provider.maas=TRACE"` [13:37] then bootstrap, then deploy [13:38] thanks, okay - - i'll try this morning [13:39] though... that's a lot of worker restarting [13:39] more than normal [13:40] implies the local api server is just very unhappy for some reason, probably worth looking for mongo issues in syslog as well [13:43] rick_h_: does juju not try to look in ~/.aws/credentials? [13:44] jcastro: hmm, looking at docs it looks in env vars [13:44] vmorris: You could also try $juju debug-log -m controller -l INFO --reply . At least that helped me dig something from the logs... [13:44] jcastro: not sure about which file. I'm looking [13:44] rick_h_: I'm going to file a wishlist if not [13:44] --replay* [13:44] jcastro: +1 [13:44] rick_h_: that's the default config from the AWS tool, I figure we could reuse it [13:45] jcastro: 1. On Linux, $HOME/.aws/credentials and $HOME/.aws/config [13:45] jcastro: juju autoload-credentials will slurp it up [13:45] jcastro: is in the doc at least [13:46] ah, of course, the snapped juju doesn't have access to that right? [13:46] that's probably why [13:46] jcastro: ah, might be [14:56] rick_h_ mgz: re-bootstrapped with debugging enabled, trying to deploy again and still not finding anything jumping out at me [14:57] first failure that i think might be interesting follows DEBUG juju.apiserver request_notifier.go:140 -> [6] machine-0 95.606348ms {"request-id":39,"response":"'body redacted'"} Singular[""].Claim [14:57] ERROR juju.api monitor.go:59 health ping timed out after 30s [14:57] then ERROR juju.rpc server.go:510 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45560: write: broken pipe [15:01] would an http/https/ftp_proxy setup in the config.yaml during bootstrap of the juju controller cause this? [15:13] can juju storage be defined in a bundle? [15:13] you can't define config stuff in a bundle, so i'd be amazed if you could define storage in a bundle [15:13] magicaltrout: of course you can : [15:13] :) [15:14] magicaltrout: https://jujucharms.com/u/canonical-storage/ceph-with-dash defines config for the ceph-osd charm [15:14] magicaltrout: what config stuff can you not define? [15:14] clearly i've been drinking [15:14] * magicaltrout checks the links [15:14] magicaltrout: anything good? :P [15:14] rick_h_: can we define storage to attach in the bundle? [15:15] rick_h_: debug-log just hangs, ideas? [15:15] icey-travel: so I thought that we could, but I think the bundle can only use already created/defined pools [15:15] ah you can set-config, i was under the impression you couldn't store config options within a bundle definition [15:15] jcastro: switch to the controller [15:15] that would override the underlying charm config defaults [15:15] rick_h_: generally speaking, the bundle is uused for demos on AWS / GCE / OpenStack [15:16] icey-travel: so I think you'll need to script the creation of the pools, but the bundle can then use those pools. Let me check the code/docs quick [15:19] icey-travel: magicaltrout looks like I lied on the storage end. Not seeing anything to support it :( [15:19] rick_h_: no worries, it seemed like a risky thin to add anyways [15:25] question: with maas, once the juju controller is bootstrapped, does the controller perform power up and down of machines directly or does it still use maas to perform this function? [15:26] vmorris: juju uses maas to handle that [15:27] Hey guys! If I want to upgrade a machine from trusty to xenial, will juju/lxd realize that the machine has changed series? And after that, can I force the charm deployed in it to change series to xenial? [15:27] okay, that's what i thought.. can't understand why juju would be happy to power on the controller machine and get it up, but trying to add a machine to juju then doesn't seem to work [15:27] hackedbellini negative [15:27] hackedbellini - series boundaries are charm-boundaries as well unless the charm is multi-series, and in that case, i dont know but i doubt that will work... [15:27] lazyPower: negative for both questions [15:27] ? [15:28] i'm positive the juju agent will get confused [15:28] i'm less clear about how lxd will handle it [15:29] lazyPower: hrm, I see. So what happens if I want to upgrade the container from trusty to xenial? I should avoid that and instead create a new machine and migrate stuff to it? [15:29] correct, the recommended way is a re-deploy and data migration [15:29] less snowflakes, more cattle [15:29] lazyPower: ok, thanks for the info! :) [15:30] rick_h_: what do you mean "switch to the controller" [15:30] like, the admin model? [15:31] jcastro: juju switch controller [15:31] oh [15:31] that machine is running fine [15:32] oh I see what you mean [15:32] now debug-log works, <3 [15:32] http://paste.ubuntu.com/23379383/ [15:33] jcastro: ? what are you doing? [15:34] trying launch canonical-kubernetes in us-east-2 [15:34] ...hacking [15:34] jcastro: hmm, do you have a VPC aws account? [15:35] jcastro: oh hmmm....sec [15:35] yes, at least I think I do [15:35] I wonder if there's anything extra I'm supposed to do to use a new region? [15:36] jcastro: oh lordy [15:36] jcastro: yea, looking, there was a branch for the instance types stuff...I'm trying to see if that's the issue [15:36] rick_h_, jcastro: https://bugs.launchpad.net/juju/+bug/1636551 [15:36] Bug #1636551: instance-type not being recognized [15:37] bdx: is that the new instance type? [15:37] bdx: e.g. last weekish they added some new ones around that haven't they? [15:38] jcastro: bdx I wonder if this is going to effect it: https://github.com/juju/juju/commit/ffc98ca2a4d12e46c969efc2b03aca6c8568243b [15:38] there was a call to do an update-clouds for AWS stuff, but I thought that was region not instance type [15:38] rick_h_: I'm just trying to deploy a m4.2xlarge in us-east-1 [15:39] bdx: right, but didn't aws create new instances there lately? [15:40] rick_h_: oh shoot, this might be user error -> https://bugs.launchpad.net/juju/+bug/1636307/comments/10 [15:40] Bug #1636307: cannot deploy to network space [15:41] I was previously having an issue with juju not recognizing multiple constraints passed under a single constraints flag, so I've been using another '--constraint' flag for each constraint [15:44] rick_h_: dimitern's fix resolved the bugs I've been experiencing [15:44] bdx: <3 [15:45] yeah this doesn't look good... [15:45] machine-0: 11:45:19 ERROR juju.rpc error writing response: write tcp 127.0.0.1:17070->127.0.0.1:42472: write: broken pipe [15:46] http://paste.ubuntu.com/23379436/ [15:47] this is following a clean bootstrap, no other actions performed [15:57] ah i'm onto something i think [15:57] my juju controller has the first interface address for the maas-controller in it's dns resolution [15:58] this is not a reachable address... i can control this in the maas configuration right? [16:14] rick_h_: works fine on us-east-1 btw [16:15] jcastro: yea, will have to talk to axw about that. Kind of defeats the purpose of update-clouds and the instance info [16:48] Hello Team, Getting Bootstrap error Please advise me on this error link http://pastebin.ubuntu.com/23379728/ [16:49] rick_h_: does anything launch for you in us-east-2? [16:49] or anyone for that matter? [16:49] Prabakaran: can your system launch lxd containers already or is this a brand new setup? [16:52] jcastro: after beta release of juju i am installing it againg [16:52] juju beta was working .. i was able to create lxds [16:52] I would just confirm that your system can launch lxd containers, just to ensure that that works [16:53] jcastro: it works [17:00] jcastro: otp, will test in a bit. === zz_CyberJacob is now known as CyberJacob [17:28] Hi kwmonroe, still trying to sort my proxy issue and why i can't deploy a charm. Does the following give any hint to you? It seems that it cannot download an image but during bootstrapping it did manage to download the same image and boot an LXD. [17:28] http://paste.ubuntu.com/23379874/ [17:34] deanman: does 'sudo lxc list' show running container(s)? [17:38] kwmonroe: It does, the controller LXD in running state [17:39] kwmonroe: http://pastebin.com/Un0vXqni [17:42] Do i have to explicitly also configure LXC proxy ? [17:49] i don't think so deanman... at least, i didn't. i checked my machine-0 logs and i don't see anything like "finding agent binaries...". i'll re-bootstrap and see if i can see anything related to image fetching [17:50] kwmonroe: This small excerpt is from bootstraping operation. http://paste.ubuntu.com/23379981/ As you can see it can find the image just fine. Only when deploying charm it complains about not finding it. [17:51] kwmonroe: Thank you for your time and support [17:53] np deanman -- it's really bizarre that you clearly found 'ubuntu-xenial' for the controller. doesn't make sense why it wouldn't find it again for a subsequent unit. [17:53] oh hey.. deanman, my re-bootstrap says this: [17:53] 17:50:46 INFO juju.environs.tools tools.go:101 finding agent binaries in stream "released" [17:53] kwmonroe: from log format most probably isn't the same code? [17:53] why does yours say 'devel'? [17:55] kwmonroe: noticed that as well, but i'm simply using an upgraded 16.04 and followed instructions on docs. [17:55] kwmonroe: How can i change to devel branch ? [17:55] rick_h_: do you happen to know where 'deve' [17:55] shoot... rick_h_, do you happen to know where 'devel' comes from in line 2 of deanman's paste? http://paste.ubuntu.com/23379874/ [17:55] kwmonroe: looking [17:56] kwmonroe: bootstrapping with a dev release? [17:56] kwmonroe: like the dev PPA or maybe from source [17:56] deanman: what does 'juju version' say? [17:57] 2.0.0-xenial-amd64 [17:58] I did use sudo add-apt-repository ppa:juju/stable but used ansible to provision that but shouldn't make a difference unless ansible screwed up?? [18:01] if it is of any help I'm using vagrant/ansible with bento/ubuntu-16.04 image and i then i do perform an apt upgrade. [18:04] deanman: 2.0.0-x-y should be the GA version.. i'm pretty sure my juju-2.0 comes from ppa:juju/stable too. what does 'grep stream ~/.local/share/juju/bootstrap-config.yaml' show? [18:04] i see this, i bet you see 'devel': [18:05] $ grep stream ~/.local/share/juju/bootstrap-config.yaml [18:05] agent-stream: released [18:05] image-stream: released [18:06] actually deanman, this is probably a better way to get that info: [18:06] $ juju model-config | grep stream [18:06] agent-stream default released [18:06] image-stream default released [18:06] http://pastebin.com/VGsZvMcp [18:06] your bet was lost :-) [18:06] shoot [18:07] i see the same output as yours when using the latter command [18:11] still i am getting the same bootstrap issue. Could somebody help me on this pastebin.ubuntu.com/23379728/ [18:11] I do have a ppa_juju_stable_xenial.list with "deb http://ppa.launchpad.net/juju/stable/ubuntu xenial main" on my sources.list.d [18:13] yeah deanman, i think you've got the right juju. otherwise, 'version' would have said 2.x-beta or something not '2.0.0' [18:14] deanman: and you're sure 'juju model-config' shows your proxy stuff set, right? [18:16] kwmonroe: http://pastebin.com/YM9GHtrC [18:17] hmm [18:19] even output of $juju model-config -m default has controller in front of the proxy settings compared to yours where it has model http://pastebin.com/9eczDGTu === frankban is now known as frankban|afk [18:20] kwmonroe: noticed that ? mine has controller in front and your pastebin has model instead. [18:21] deanman: i hadn't noticed that, but more concerning is that your agent-version is 2.0.0.1. now i'm back to wondering where your juju came from :) [18:25] apt-cache show juju -> http://pastebin.com/T1dd3f71 [18:25] answers your question! [18:28] deanman: what's the output from apt-cache show juju-2.0? [18:30] http://pastebin.com/y7DJcM7m [18:32] welp deanman, i'm stumped. i dunno why our agent versions don't match, nor why this shows a devel stream: http://paste.ubuntu.com/23379874/. i don't even know if those are the cause of the un-deployable charms anyway :/ [18:34] at least i could try revert to stable and start from there again [18:35] deanman: you could try 'sudo apt remove juju-2.0 juju-core --purge' and start over... or perhaps someone in the #juju-dev room will have other ideas.. and there's always the file-a-bug option ;) -- https://bugs.launchpad.net/juju/+filebug [18:37] well i started already with a fresh vm and added stable ppa and doing apt install of juju zfsutils-linux [18:37] i bet that works. i'm good at betting. [18:38] kwmonroe: haha, ok let's see, if apt-cache reports again devel then i have to talk to the maintainer i guess ? [18:38] yours say stable on apt-cache right? [19:06] alright deanman, so i think you'll be back in business if you bootstrap with --config and *NOT* development=true [19:07] and remember, if you want the proxies to propogate to future models, use --config and --model-default for http_ https_ and no_proxy [19:07] kwmonroe: Double checking here: juju bootstrap localhost lxd-test --config transmit-vendor-metrics=false --config http-proxy=$http_proxy --config https-proxy=$http_proxy --config no-proxy=$no_proxy --model-default http-proxy=$http_proxy --model-default https-proxy=$http_proxy --model-default no-proxy=$no_proxy --debug [19:07] bingo deanman [19:07] i bet that works [19:11] well not betting on it, still a couple issues which not related to --development ? the "controller" in front of proxy entries on model-config -m default [19:14] don't bet on any advice from kwmonroe [19:14] i have first hand experience of that! [19:15] :) [19:16] deanman: i wouldn't worry about the controller vs model bit. my model-config output was from a test where i did *not* bootstrap with --model-default values [19:17] when you include those, the proxy prefix switches to controller (unless override explicitly with juju add-model --config ) [19:18] deanman: the bit that i'm not clear on is why there are no devel images. tbh, i don't know the glue between streams and agent versions, so i dunno if that's a bug or not. [19:19] ok i'm testing, let's hope that this is it :-) [19:19] if nothing else, at least you got to run a lot of apt commands today. that's always neat. [19:20] lol [19:21] learned how to pastebinit too, i can safely put it on my resume i guess [19:21] ;-) [19:22] thats a quality tool everyone should know [19:23] magicaltrout: mine was broken though, adding even more to my overall agony [19:23] lol [19:23] not good [19:23] hehe [19:25] on a completely random topic..... I never realised Global Entry was only $100... i'm sorely tempted to sign up [19:25] haha magicaltrout. as if you'd pass the background check. we know where #brexit started. [19:25] this is true [19:26] never said i'd get accepted :P [19:29] Last trip was pulled over when i answered that i did have a croissant on my bag, could globar entry have saved me the embarassament? [19:31] not a croissant! [21:05] cmars, mbruzek, lazyPower: concerning an all-encompassing ssl/tls layer -> https://gist.github.com/jamesbeedy/c94cd6e8c7cb4246818aeff7b9adf5ad [21:15] bdx: that looks pretty incomplete. I left some comments with code of what I was thinking. [21:19] mbruzek: yea ...just trying to get a solid idea/base direction [21:19] thx [21:19] bdx yeah I am willing to change the current tls relation to accommodate the other types [21:20] I am not familiar with lets encrypt. [21:21] mbruzek: I've been trying to make use of LE more .... the primary drawback is that LE runs a verification on your fqdn<->publici [21:21] oh [21:22] mbruzek: making its use in private network space .... ehh [21:22] So you can't just "tell" LE what your ip-address, and fqdn is? It verifies that? [21:22] mbruzek: yea [21:23] bdx - welcome to what i've been saying ;) LE is great for freely encrypting public facing sites that wouldn't otherwise have been tls wrapped [21:23] however its not a silver bullet, so the direction you're taking in that sketch is the right path i feel [21:23] bdx: interesting. so what if LE could not contact the system? say it was behind a firewall ? [21:23] but as mbruzek said its incomplete,a nd there's other concerns that aren't represented there. [21:23] mbruzek: it fails hard and loud [21:23] bdx OK [21:23] such as this is tied to easyrsa/le layers, its not an abstracted front end to swap out any backend. [21:24] bdx I have my work plate full at the moment but I could take a look at this with you some time [21:24] bdx - for example i just learned that gandi (the dns registrar) has a tls key cli tool [21:24] which allows for on-site requesting of a tls certificate [21:25] mbruzek: awesome [21:26] wow, thats awesome ... even more reason to make the tls/ssl portion lean towards pluggability/extensiblity [21:27] sorry :) i feel like i kind of inserted myself in this convo [21:27] but i've been having shower thoughts about our tls story and how to make that more robust [21:27] mbruzek, lazyPower: I'm trying to accomodate the use case here [21:27] bdx: some people have told me to check out cfssl [21:27] we're missing 2 critical components. 1) rekey the infrastructure, 2) revoke certificates. [21:28] we wan't to deploy ssl/tls all the time [21:28] so I feel the layer has to have some kind of standardization [21:28] like, include this layer and you get the best ssl/tls depending on what you environment allows right [21:28] it seems like it should be an optional relation, and simply stated, if it has the relation, it should use whatever the tls abstraction provides. [21:29] exactly [21:29] and thats dependent on what its related to [21:29] yes, I see [21:29] so consider the following [21:29] isntead of having a 1 size fits all CA charm [21:30] we have flavors of CA, easyrsa, letsencrypt, and we can colocate these places in lxd. they dont need to run API's or anything fancy. The fact juju proxies those requests and executes locally is kind of nice compared to say running CFSSL internally for an API that you then have to secure and harden [21:32] bdx int he case of lets encrypt, there's still the routability issue when its colocated in lxd... [21:32] theres a few strategies to combat that, such a using a frontend like traefik which knows LE and can conscript certificates as a reverse proxy for domains that dont have TLS but declare they want it. [21:33] or kube-lego [21:33] or any of the other LE wrappers, but i'm not certain how well they perform under load and so forth. so i still have a ton of discovery to do there. if you've got any experience with any of them it would be great to riff on that [21:33] lazyPower: if I'm picking up what you are laying down, "just have these different key/crt providers laying around in my environment, and write my charms to take advantage of whichever" ? [21:33] if so, done [21:34] essentially, yeah [21:34] lazyPower: I think the layer abstraction you mentioned is important though, and is what I'm trying to get at here [21:35] bdx - yeah, i agree, it needs to be a standard set of relations that every CA provider implements [21:35] and its the way we funnel behavior to an end result that we care about [21:35] eg: Just give me keys and get the foo out of the way [21:37] at the end of the day, thats all we really care about... that we have certificates that came from the intended source. If thats an internal EasyRSA provider, or an external CA like LE, or the new thing that hasn't been popularized yet. [21:37] totally, because otherwise it turns into piles of jimcrackery in every charm ..... if the layer negated this it would be great, bc then you could just include the layer and get tls the best way no matter what platform/provider/routing [21:37] yep [21:37] agree [21:37] i think mbruzek has some solid starts there with what he did in the new easyrsa charm [21:38] and we could crib from that to set that standard, and then start making our way through the other layers and folding in some kind of "conformance test" to use the kubernetes term [21:38] agreed [21:38] ok, end of lazy showerthought, i hope this was as helpful to you as it has been to me while trying to demystify TLS [21:39] extremely [21:39] mbruzek, lazyPower: thx [21:39] <3 [21:41] my outline captures a view of the target use case though, right? [21:41] if user specifies it then obviously use it [21:41] bdx yeah, its showing an approach [21:41] i think whats more valuable is the intent, that it shouldn't care where the cert data is coming from [21:41] it should just return cert data [21:42] or block until it has some [21:42] else, if they want tls option A-Z (LE), then opt for that secondly [21:42] as a last resort, get an internally signed cert or you can't continue [21:42] right === hml_ is now known as hml