[00:33] so the remaining issue, is that I don't understand how to determine if the charm knew better than to bounce both nodes at the same time or not [00:51] so the remaining issue, is that I don't understand how to determine if the charm knew better than to bounce both nodes at the same time or not === disposable3 is now known as disposable2 [15:02] afternoon [15:02] can someone explain the k8s loadbalancer to me? :) [15:03] cause i look at the microbots example and that uses the xip.io dns service but doesn't seem to use the loadbalancer, but I'm failing to see how services are routed through it [15:04] when i say "explain" i just mean point me to some docs that reflect what is implemented here :) [15:12] oooh thats only apiserver distribution on ingress [15:12] i should read better [15:14] magicaltrout: i'm listening but i'm not sure what you're asking :P [15:14] hahah [15:14] i'm trying to rather badly it seems, figure out external routing for k8s services :) [15:15] i thought the load balancer provided it but that seems to be api call balancing [15:15] magicaltrout: have you read https://kubernetes.io/docs/concepts/services-networking/service/ [15:15] I'm using iptables to forward traffic to a worker node ip (because my worker nodes are on an ip range that conjure-up selected, that isn't routable in my network) [15:16] magicaltrout: the LoadBalancer can do it if your cloud supports it [15:16] interesting [15:16] Dwellr: juju config flannel and set the cidr [15:16] Dwellr: we're adding support to conjure-up to allow you to configure subordinate charms [15:17] stokachu: I don't really have a cidr.. [15:17] my confusion lies for example, with the microbots setup, you get a dns endpoint that points to the ip of a worker [15:17] Dwellr: maybe Cynerva has some additional best practices around that [15:17] stokachu: Im running conjure up inside virtualbox, the virtualbox vm has a bridged adapter onto my house lan, and I'm using iptables to forward from the bridged adapter to the worker node.. [15:18] stokachu: that's really only relevant to the cluster network [15:18] stokachu: that lets me hit say, microbot, from other pcs on my house lan (provided I pass the fake host header so ingress routes the request etc) .. but I'm very much still learning =) if there's a better/cleaner way.. I'm all ears [15:18] ah ok [15:18] magicaltrout: right, so you it only works if you can route to the node ip [15:19] Dwellr: i think iptables is the way to go [15:19] yeah tvansteenburgh but also, if you stuck a real DNS entry against the node ip, if you tore it down and the service ended up on a different worker [15:20] you'd have a stale DNS entry for your users, right? [15:20] that's why you don't want to point dns directly at a node ip [15:20] you point at a Service ip [15:20] which *can* be provided by a LoadBalancer, but it can also be configured manually [15:21] haha, minds melting, i've lived in mesos land for too long [15:21] Service type can take an externalIP [15:21] magicaltrout: read that doc page from beginning to end - i bet you'll understand fine after that [15:21] hehe yeah i'm running through it thanks tvansteenburgh [15:29] is there anyway for me to see what juju run-action kubernetes-worker/0 microbot replicas=x does? It's great that it's automated etc, but I want to peek at what it did =) [15:30] https://github.com/kubernetes/kubernetes/blob/master/cluster/juju/layers/kubernetes-worker/templates/microbot-example.yaml [15:34] thanks =) more than handy =) [15:37] and.. what's the simplest way to get my worker nodes to pull from a custom docker repo? I figure I can run docker back on my vm host, and have the workers pull from that repo, (and mebbe enable docker tls so then I can push to the vm from outside too) [15:38] * Dwellr trying to get toward 'kube in a box' ;p where I can spin up the vm, and develop and deploy apps to it from outside the vm [15:56] why is the charm snap throwing this exception about one of my local python packages https://paste.ubuntu.com/25486709/ [16:22] Dwellr: there's a registry action on the worker nodes [16:23] whats a registry action, and how does that relate to my current understanding of a docker repository that I push my images to ? [16:25] Dwellr: an action is a Juju thing - think of it as a script that you can run on a unit [16:26] like, microbot is an action [16:26] okie, so what does a registry action do ? [16:27] Dwellr: https://jujucharms.com/u/containers/kubernetes-worker/52 [16:27] search page for registry [16:29] thanks tvansteenburgh that page was indeed very useful ;) [16:29] oooh.. thanks =) [16:30] skay: That is an issue with the classic snap and the reason we worked on moving to strictly confining the snap. In fact, we *just* released the strictly confined snap a little while ago, so if you `snap refresh charm` and try again, it should be fixed [16:30] Dwellr: if you'd rather use Helm, there's a chart for a private registry here: https://github.com/madeden/charts/tree/master/docker-registry [16:31] I'm still learning.. using conjure up as a way to spin up a slightly more complex kube deployment than minikube will, so I can play with what it means to deploy and run apps on kube etc [16:31] Dwellr: cool, no prob :) [16:32] so I think the juju action for the registry will work [16:32] I'm slowly building up a vagrantfile that builds my kube in a box =) [16:32] cory_fu: sweet! worked [16:33] cory_fu: (actually, I unstalled it and installed it from scratch. at first it didn't find any update) [16:33] so far it installs conjure-up, uses it to install kube to localhost (interestingly using 'conjure-up kubernetes-core localhost' produces no output in vagrant until the command completes, which takes like 10mins or so).. [16:34] Dwellr: is this from vagrant ssh into the machine and running conjure-up? [16:34] and then it copies the kube config, rewrites it to update the server address to the external bridged ip for the vbox vm, adds insecure-skip-tls-verify, and copies the config to the vagrant shared dir. [16:34] skay: Odd about the refresh. Maybe it auto-refreshed before you tried it. Either way, glad it's working! [16:35] Dwellr: you may be able to get around the no output from conjure-up using `ssh -t` option [16:35] stokachu: I mean when I have config.vm.provision : shell, privileged: false, :inline => <<-EOT [16:35] conjure-up kubernetes-core localhost [16:35] EOT [16:36] Dwellr: that sounds like a neat setup - you should write a blog post about it [16:36] Dwellr: stil could be ssh option [16:36] that, when I run `vagrant up` and it runs the provisioning steps.. there's no output from the command until it completes [16:36] Dwellr: it's inline but i think vagrant has to ssh first [16:36] Hmm.. wonder if passing -t is possible in the provision flags [16:37] Dwellr: try config.ssh.pty= true [16:37] in config.vm.provider [16:37] http://blog.medyagh.com/2014/07/fix-vagrant-error-sudo-sorry-you-must.html [16:37] the vagrant file then goes on to setup ip forwarding for all nodeports (and 80 & 443) to the first worker ip, and 6443 to the master node ip from the bridged interface [16:37] stokachu: I'll try that in a mo =) [16:38] cool [16:39] please publish this awesomesauce [16:39] Dwellr ^ [16:39] hmm the docs for config.ssh.pty say that "When pty is enabled, it is important to note that command output will not be streamed to the UI. Instead, the output will be delievered in full to the UI once the command has completed." [16:39] Dwellr: ah that's opposite what i was going for [16:39] and thats kinda what I'm already seeing with the conjure-up command ;p [16:39] yea hmm [16:40] tvansteenburgh: it will be published =) .. I'm doing this as a side project to create a dev env in a box for gameontext.org [16:40] Dwellr: though it's odd b/c ive used this option in jenkins ci runs [16:40] to see the output as it happens [16:40] i ran into the same issue until adding that [16:41] I can always try =) [16:41] sure :) [16:43] was also figuring I could pipe it thru tee or sommat [16:44] ah yea that may be another way [16:50] Is this right? The only way to export a model to import as a bundle is via the gui? https://jujucharms.com/docs/2.2/charms-bundles [16:51] xarses_: i think that's right yeah [16:51] =( [16:51] I don't have gui's attached to most of my controllers [16:51] xarses_: just run `juju gui` and you'll have one :) [16:52] you can do that to get the initial bundle and then just edit by hand from there [16:52] i think that's what most ppl do [16:52] ya, we don't use the gui around here [16:52] neither do i [16:55] wow, all the icons are missing [16:55] from the gui? [16:55] ya, it's showing the missing image icon [16:56] i blame hurricane Irma [16:57] xarses_: which charms? [16:57] openstack*, percona, ceph, nagios [16:59] xarses_: can you see this https://api.jujucharms.com/charmstore/v5/ceph/icon.svg [17:00] yes, but shouldn't it download these things and keep them with the charm? [17:01] also, we don't use charms directly from the store [17:01] * tvansteenburgh looks around for a gui dev [17:01] uh, export button doesn't work [17:03] * hatch waves to tvansteenburgh [17:04] hatch: xarses_ is having some trouble with his gui [17:04] missing icons, export button not working [17:04] rotfl [17:04] nice one hatch [17:04] lol [17:04] sorry wrong key [17:04] hatch: xarses_ is having some trouble with his gui [17:04] got that [17:04] export [17:04] and icons [17:04] yep [17:05] ok xarses can you click the ? in the top right and head for the keyboard shortcuts to find out what version of the GUI you're using [17:05] also, browser, and version of Juju [17:07] oops xarses_ ^ [17:14] bdx: What do you think of the soft-launched layer-index repo vs the old micro-site? [17:15] cory_fu: I've been waiting on this for a while now .... super excited [17:15] cory_fu: it will make it easier for me to run a private registry for use in my charm ci/cd [17:17] bdx: Excellent. I know the PR workflow adds a slight barrier to entry to getting new layers registered, but I think it's still reasonable. [17:19] totally, reasonable for sure. It also gives a central place users can subscribe to and more easily track the addition of new layers [17:19] like, I want to know when new interfaces/layers are added to the registry [17:19] this make it easy, you just follow the repo [17:33] just like to say [17:33] the new status button in juju gui [17:33] is epic [17:38] hatch how does the keyboard shortcuts page show version? [17:39] juju version is 2.2.2, browser is chrome 60.0.3 [17:42] cli from `juju gui` implies its 2.4.2 [17:42] > GUI 2.4.2 for model [18:07] xarses_: can you run juju upgrade-gui ? That's quite old. [18:07] xarses_: it should be 2.9.2 [18:08] hmm, 2.4.2 is Feb release [18:15] ya, thats about when the controller was originally deployed [18:15] does it not upgrade with the controller? [18:16] well, export works now [18:16] icons are still broken [18:27] hmm.. tried doing conjure-up kubernetes-core localhost > mylogfile.txt and the log file stays at 0 bytes until conjure up is done too [18:29] I'm guessing because they are all local charms [18:29] xarses_: k, icons are pulled based on the charm urls so they might get blocked. [18:29] xarses_: ok, if they're all local charms they're supposed to be pulled from the controller itself. [18:29] xarses_: can you open the browser debug tool and see if you can get an error for the icon urls? I imagine they're getting a 404 or something [18:29] xarses_: and file a bug on github.com/juju/juju-gui with the details ? [18:30] xarses_: and no, the gui is out of sync with juju itself. There's usually a release every month or so. Since it's not effecting the actual workings of Juju but UX fixes/etc it's on the side. [18:32] ahh, here is the bug, its trying to resolve the admin user and my model name for icons on api,jujucharms.com/charmstore/... [18:33] xarses_: :( [18:33] lemme sanitize [18:34] yep, raises a similar 404 message [18:34] https://api.jujucharms.com/charmstore/v5/~my-local-admin-user/my-local-model-name/meta/any?include=bundle-metadata&include=bundle-machine-count&include=charm-config&include=charm-metadata&include=charm-metrics&include=common-info&include=extra-info&include=id-revision&include=manifest&include=owner&include=published&include=resources&include=revision-info&include=stats&include=supported-series&include=tags [18:36] xarses_: yea, that's a bug-a-boo to fix. Apologies on that. [18:38] https://gist.github.com/xarses/4b1bea1bcdf8012737dee5f4f57ee7f3 the charms in the model for reference [18:58] Dwellr: boo :( [19:09] Dwellr: I found that I had to capture stderr in my pipe to catch most of the information during bootstrap, pass --debug flags and pipe to tee to get a useful looking log (mind you this was just with juju bootstrap) [19:12] rick_h: file a bug on LP? [19:13] I'm having partial success doing conjure-up & then tail -f --pid=$! ¬ [19:14] I'm having partial success doing conjure-up & then tail -f --pid=$! ~/.cache/conjure-up/conjure-up.log [19:14] Dwellr: nice [19:14] Oh I wonder [19:14] gets me a log that updates as far as 'Executing script ... steps/00_deploy-done' [19:14] but then it stops while juju status is showing the master/worker etc are still init'ing [19:15] Yea those logs are in the spell steps dir [19:15] Like 00-deploy_done.err [19:16] cory_fu: any idea why the output from headless install only prints out when it's completed? [19:17] This is from running it via vagrant up and even in our Jenkins runs it would do that [19:17] Dwellr: did you try that try setting yet? [19:17] TTY [19:17] not yet.. apparently it's pretty strongly recommended against.. so leaving that as last resort [19:17] Ik [19:17] Ok [19:18] stokachu: No idea. I don't use vagrant often. [19:18] Dwellr: the only other thing I can think of is that we use terminal colors for the output [19:19] so does apt-get update/install and that outputs ok during vagrant [19:19] Ah ok [19:21] rick_h: https://bugs.launchpad.net/juju-gui/+bug/1716037 [19:21] Bug #1716037: gui fails to render local:charm icons [19:22] Dwellr: could vagrant only be printing a certain fd? [19:26] well.. it wouldn't explain why conjure-up kubernetes-core localhost > mylog.log doesn't write anything into mylog.log until the process finishes.. [19:26] that's nothing to do with vagrant [19:30] Dwellr: does `2>&1` help ? [19:33] stokachu: Maybe we could try adding flush=True to print() and cprint() in send_msg: https://github.com/conjure-up/conjure-up/blob/master/conjureup/utils.py#L248-L257 [19:33] cory_fu: good idea [19:33] stokachu: (flush=True docs: https://docs.python.org/3/library/functions.html#print) [19:34] could I patch that in locally after installing conjure-up ? [19:34] AFAICT, cprint() passes through args to print(), so it should work on either [19:34] (happy to test if possible) [19:34] Dwellr: so you would need to checkout the source [19:34] Yeah, I don't think you could patch the code in the snap [19:34] Dwellr: if you clone https://github.com/conjure-up/conjure-up [19:34] Dwellr: then run `make sysdeps; make dev` [19:34] heh.. was hoping being python the source would be on my system somewhere [19:34] then you can . conjure-dev/bin/activate [19:34] and try it there [19:35] Dwellr: It is, but bundled into the snap filesystem. The confinement of snaps would make it challenging to modify [19:36] yeah.. I know nothing re snaps =) [19:37] Dwellr: Well, they essentially are a way to package, confine, and apply security to an application to make it safer and easier to manage. [19:39] I think you'd have to be root to unpack and modify the bundled file system image and even then I think the signing of the snap might prevent it [19:39] root simple, signing not. [19:39] just paste your private key here in the channel, and i'll resign it ;p [19:39] lol [19:40] dont forget cc# and security code [19:40] You could manage it, but in the end it's just easier to use the source from GitHub and either run the dev venv as stokachu suggested, or build the snap from that and install with --dangerous [19:40] I'll need your visa number, 4 digit pin, and 3 digit verification code [19:40] lol [19:40] rofl.. jinx [19:40] :) [19:40] Dwellr: yea the source is quicker, especially if you dont want to build a new snap [19:40] lemme clone my vagrant file, and have it stop before it does all the badness =) [19:41] stokachu: I wonder if we could test using the output redirection that Dwellr mentioned. [19:41] * Dwellr rather glad all this is running on his new dumpster-server ;p [19:42] sold as 'would not power on', I bought it just for the case (3u chenbro with 8x hotswap sata) .. turns out it powers on just fine (after adding some ram) .. it's a twin xeon e5530 server, (now) with 24gb per cpu, [19:43] Nice [19:44] stokachu: I'm seeing the same thing with redirection to a file [19:44] total price (excluding ram/drives).. just under 50$ [19:45] Dwellr, stokachu: Yep, adding flush=True works [19:46] cool =) how long before that can make it into a proper build? [19:46] stokachu: I noticed that redirecting the output to a file still generates colored output. I think "elif syss.__stdin__.isatty():" should actually be testing stdout, no? [19:52] Dwellr: I think we could get it in --edge today. I'm about to put up a PR now [19:53] is that a flag for snap install ? [19:54] Dwellr: yes, it looks at the edge channel of the snap to get pre-release stuff. [19:54] okie.. I can add that to the vagrant script quickly enough later then to test. [19:55] ergh, rick_h two new problems can't logout from the store now that I linked my user, and since I linked my user there isn't a button to switch back to the models, you just have to stay on the user ui or go hack the url [19:56] actually, 3 the linking the user didn't go to the proper redirect and authorize oauth page, it just selected an account [19:56] xarses_: k, will check it out. I just got the latest release yesterday and haven't tried the account page thing. [19:56] huh? [19:56] linking the user? [19:57] * rick_h is lost on what xarses_ is up to [19:57] yes linking the juju user in the gui to the charmstore [19:57] k, /me is bootstrapping to see i can see what you see [19:57] cory_fu: nice [19:58] Dwellr: so basically when it's ready you would run `sudo snap install conjure-up --classic --edge` [19:58] or if already installed run `sudo snap refresh conjure-up --egde` [19:58] it should have raised an oauth authorize page, it just accepted the context I was in, instead of the oauth page making me confirm the app [19:59] xarses_: it's an openid vs oauth thing. It should open in a new window you can close once you auth (and think it says you can close it now) [20:00] and its now logged in to the wrong user, and I don't see a unlink button [20:01] stokachu: https://github.com/conjure-up/conjure-up/pull/1148 [20:02] cory_fu: thanks, ill merge and push to edge soon as it's done with ci [20:02] stokachu: +1 [20:04] xarses_: ok, so I just tried to replicate it and I logged into the charmstore and it shows at the top my two models on my controller (controller and default) and the list of charms and bundles from my charmstore account. I then clicked on the default model, and clicked manage to get back to the GUI [20:05] xarses_: it shows my username as "admin" on my profile page. I'm not sure what "wrong name" so any screenshots/etc would be helpful in looking at what went off the rails for you. [20:05] xarses_: bugs are tracked on GH at github.com/juju-juju-gui/issues [20:05] ya working on those [20:07] rick_h: for the oauth, I expected a dialog like https://drive.google.com/open?id=0B_Hn-o0wEGNfZHdLZmt4VTdYV1E [20:08] I got https://drive.google.com/open?id=0B_Hn-o0wEGNfUVR6aVNGOGNVOUE as the first page [20:09] xarses_: hmm, I think because the GUI doesn't request any of the account details it doesn't have that dialog [20:09] xarses_: you only have to give approval if the application asks for the details so it can restore it locally for any reason [20:12] I'd still expect the ask, its now linked to the wrong account because it just used the first cookie it found [20:12] oauth is supposed to be explicit in this regard, the oauth host isn't supposed to allow the app if the user doesn't allow it [20:14] with regards to the other issue https://drive.google.com/open?id=0B_Hn-o0wEGNfQ0Y0cmVwRUhEbDg [20:14] xarses_: as for logging back out you're right there's not a box there. The work-around for the moment would be to wipe the cookies and local storage on the browser for the api.jujucharms.com domain from the browser :( [20:14] xarses_: understand, it's not oauth atm but good feedback. [20:15] rick_h: I haven't even authorized that account for jujucharms.com either [20:15] xarses_: right so from there if you click on the model name you can click on "manage" [20:15] xarses_: right, you're butting up against how ubuntu sso works [20:15] and that its super broken for multiple accounts? [20:16] xarses_: notice nothing through ubuntu sso has that setup. I think you can wipe https://login.ubuntu.com/+applications and remove the web login? I'm not sure [20:16] Dwellr: it's building now, you can see the status here https://code.launchpad.net/~conjure-up/+snap/conjure-up [20:17] xarses_: so the user/etc is the local juju account. The charmstore login is kind of on the side and doesn't effect the juju "admin" account you logged into the GUI in any way [20:17] looks like it'll be at least 40 minutes before it starts [20:17] usually takes about 5 minutes to build [20:17] rick_h: ya, I get that but I can't log it out now, as its on the wrong usso account [20:18] xarses_: right, so that's why I said that it's a valid bug to not have a logout of charmstore button there, so you'd have to remove the localstorage/cookies from your browser atm apologies [20:22] k [20:22] oh, those expand? thats totally not obvious [20:23] xarses_: +1, the team's working on an updated profile page that groups things into more logical buckets and reworks that UX [20:23] the grouping is fine [20:23] xarses_: I hate the expandy-two-click stuff and it's not middle click to open in new tab friendly heh [20:23] just the layout didn't imply that they could expand [20:23] +2 to the middle click [20:24] I have to have a 3-5 button mouse [20:24] Idk how people live with out them [20:24] lol [20:24] xarses_: I get your point though after tinkering with this stuff. It just uses your current sso account vs allowing you to pick when you login [20:26] oh, you don't get to "pick" with the oauth ask from openstack.org either, it just gives you a chance to logout and log back in with the account you want it to use [20:26] xarses_: right [20:26] lp and oauth askers seem to be cool with my two accounts [20:26] these usso ones just pick the first one [20:27] which was fine for the force portal [20:27] xarses_: I've got a bug filed on the lack of the logout button but yea, I think there's some limitation on working with the login.ubuntu.com stuff. Will see [20:27] but is annoying me with the charm store [20:28] gotcha, sorry I didn't grasp what you meant at first. Because I have 2-fa it always stops up for me to 2fa but if you didn't have that it'll just dump right in. [21:05] ya'll trying the Ceph PV stuff in the CDK readme [21:06] and i get timeout expired waiting for volumes to attach/mount for pod "default"/"web". list of unattached/unmounted volume [21:06] not having used Ceph before, what am i missing or where should I prod? [21:14] magicaltrout: what's the output of kubectl get pv [21:15] and kubectl get pvc [21:20] magicaltrout: i gotta run, feel free to post on the ML or file an issue on the cdk repo [21:26] no worries ex-tim [22:52] magicaltrout: i don't have experience with ceph either, but if i were you, i'd swap that for something lighter, like hdfs. [22:59] hmmm kwmonroe its like deja-vu [23:00] you're right though, i did [23:00] /tmp