[00:12] simplest review ever, anyone? https://github.com/juju/juju/pull/8913 :D [00:42] wallyworld: I'm pretty sure I want to pull in and import k8s.io/kubernetes/pkg/credentialprovider as it provides DockerConfigEntry and DockerConfigJson, as simple as adding to dependencies.tsv and godeps-ing right? [00:44] veebers: yep [00:44] veebers: run go get first [00:44] then run godeps [00:44] wallyworld: ack, cheers [00:54] what blatantly obvious and silly thing am i doing wrong/missing here: https://paste.ubuntu.com/p/zvz74BWp68/ [00:58] * veebers waits for go get, 550M later . . . [01:09] never mind. it was very obvious and silly [02:23] veebers, wallyworld: we need to fix our snap beta channel [02:23] thumper: oh? [02:23] veebers: run snap info juju [02:23] beta is behind candidate and stable [02:24] the way the channels are they should be increasing levels of risk [02:24] so for us, I think beta should match candidate [02:24] until we are ready to announce a beta for 2.5 [02:25] thumper: there is a snap (snapcraft?) command you can run to point beta -> candidate, I can't recall it right now, let me thinkj [02:25] veebers: we should have it automated... [02:26] the interesting bit is how to know when we swith away from having match candidate [02:26] and on to it's own thing [02:26] thumper: aye, was just going to ask how we might consider making that decision in an automated fashion [02:28] thumper: so in this case candidate and beta will be the same, when we're posed to release 2.4.1, candidate should be frozen and beta continue to follow 2.4 branch tip [02:28] yeah... [02:28] until we are ready for a 2.5 beta [02:29] so ... it is a bit awkward [02:29] right, although, perhaps beta should follow edge (2.5)? [02:30] sorry, follow 2.5 branch tip (which just happens to be develop at thsi point) [02:33] thumper: I *think* for now that "snapcraft close juju beta" will make beta follow candidate [02:40] wallyworld: I might be being dumb, a service has 1 or many units right? So EnsureService will be called first or at least always called, and EnsureUnit called if extra units are added? (i.e. so ensuring secrets are created belongs in EnsureService, which they can then be consumed in EnsureUnit . . .) [02:43] veebers: i was under impression that we no longer have services but applications? if u mean applications, i believe they can have 0+ units [02:44] anastasiamac: lol my bad you're right, should have said app. The first line in EnsureService is "creating/updating application . . ." ^_^ [02:46] veebers: \o/ we are working on renaming all service* reference but obviously non-user-visible occurences have low priority... fwiw, it very rare and kind of strange to have an app with 0units.. however we do not prevent it [02:47] wallyworld: what is externalcontrollerupdater worker? [02:48] wallyworld: it is failing intermittently on merge [02:48] veebers: not EnsureUnit, that is an old method I have removed [02:48] i am facing that issue [02:48] vino: yeah, I was looking at your merge [02:48] externalcontrollerupdater worker failure. [02:48] veebers: when a unit is added, juju increases the replica count on the deployment controller [02:48] wallyworld: lies, I can see it here in my editor [02:48] vino: just ask for merge again [02:48] veebers: you may not have latest code [02:49] or my PR got rejected again [02:49] i am trying to build again [02:49] nothing calls that method [02:49] wallyworld: I'm sure I pulled yesterday just before working on this. I'll check [02:49] or veebers added it himself :D [02:49] wallyworld: ok cool, I'll ignore it for now [02:49] hah ^_^ [02:49] veebers: even if it is there, nothing calls it [02:49] :D [02:50] ack, thats something I should have checked as well (is this called) [02:50] thumper: that worker watches for changes to the api addresses of another controller in cmr scenario [02:50] anastasiamac: I should create a charm called "IDoWhateverIWant" that will have 0 units [02:51] veebers: i like it but our charms are not camelcased... then again u do indeed do ^^ [02:51] hah ^_^ [02:53] thumper, wallyworld, vino: Huh, I wonder if the charmstore-url config additions introduced that, but surely I would have seen that on the test runs on my PR? [03:01] veebers: after my last commit i was able to see things successfully built. havent touched that part of code. [03:01] wallyworld: git question, if you and me have been ratting around in k8s.go, should I: commit what I have, git fetch develop, git rebase develop? [03:06] thumper: just going thru ur review comments. quick question. the exportbundle.go inside cmd/juju/bundle/ [03:06] is that corret place to be in or it should be inside cmd/juju/model/ folder [03:07] vino: I don't really care [03:07] either is fine by me [03:07] developer's choice [03:08] ok. [03:08] i am thinking of moving inside the cmd/juju/model folder [03:09] as it is related to model [03:26] wallyworld: pr for 2.3 https://github.com/juju/juju/pull/8914 to uninstall rather than bounce a worker for bug 1781096 PTAL? [03:26] Bug #1781096: "model-upgrader" manifold worker returned unexpected error: model has been removed [03:26] ok [03:26] ta [03:29] anastasiamac: errors.IsNotFound(err) doesn't apply for facade errors right? we need to use a params IsCodeNotFound() helper? [03:29] k [03:41] wallyworld: thnx! updated [03:42] looking [03:44] anastasiamac: done, lgtm, i did request an extra test, see what you think [03:45] thumper: reading thru ur comments just reminded me that I missed a unit test for version chk failure in the Pr just got merged. I have initiated to add the same in bundleSuite and when reverting the changes its lost. apologise. I am adding unit tests for client facade will add here in apiserver facade too. [03:45] * vino going for lunch. [03:49] vino: that's fine. I figured you'd add one when you start the implementation [03:51] wallyworld: thumper: https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/608 [03:52] if you guys have any idea .. [03:54] ybaumy: i asked the k8s guys and they said your setup was vms on a mac and those vms could not write to your disks. if that's the issue, that not something we can really solve as it is a config/setup issue on your end [03:55] you need to configure your vms to get full access to the underlying hardware [03:55] i don't own a mac so can't really help much for that [03:57] hmm i dont understand.. could not write to my disks... but the ubuntu host for for juju localhost is already writing to that disk isnt it? and everything else is just a virtual layer if i understand that? [03:58] if there was a general problem i would see filesystem device errors on my ubuntu vm running in vmware? [03:58] but it aint so [03:59] ybaumy: the k8s guys said that the error logs we full of errors saying something like "write permission denied" when the k8s nodes tried access the disks [03:59] i have no idea how to fix that myself as i don't run a mac [04:00] wallyworld: will test this today on vmware vcloud director which is pretty much vpshere [04:00] it seems very weird [04:00] this is running on intel [04:00] basic esx [04:01] ybaumy: this is outside my areas of expertise [04:01] what is the disk layout of these machines? [04:02] its pretty much the normal vmware vsphere esx environment .. on top we have vcloud director running. i will setup a vm there with ubuntu 1604 and then make juju localhost as i have here [04:03] vcloud director is a overlay for vsphere. its thought to be for service providers [04:03] the disks are vmdk's or ovf [04:06] as far as osx setup goes.. i still believe there is something fishy with the apparmor/lxc when i look at dmesg on boot [04:06] i will get back to that tonight [04:07] have to shower now and get ready for work [04:07] later [04:17] wallyworld: did u want to check the added test or heppy for me to land as-is? [04:17] nah, i trust you [04:19] i'll frame it \o/ [04:21] Hmm, adding a dep import has borked my test with a "panic: /tmp/go-build353638742/b001/provider.test flag redefined: log_dir" error, as per https://pastebin.canonical.com/p/tYVSMXYNFV/ any thoughts on how to fix it? [04:27] wallyworld, could you take a look? ^ [04:27] veebers: soon, otp [04:27] ack [04:59] veebers: do you have a diff? [04:59] did you make any source code changes in juju [05:00] wallyworld: to "fix" it, remove the "k8s.io/kubernetes/pkg/credentialprovider" import (and remove use of credentialProvider so it compiles) :-) I'll paste diff now for further context [05:01] wallyworld: http://paste.ubuntu.com/p/rJ3HDVrWQD/ [05:03] wallyworld, would u take a look this small PR https://github.com/juju/juju/pull/8909 ? thanks [05:03] kelvin: jump in HO? [05:04] yup [05:19] veebers: what's the line in dependemcies.tsv? [05:20] wallyworld: oh, I haven't added anything there yet :-\ I just did a go get [05:20] ok, np [05:20] i can see the error now [05:37] veebers: we need to flatten the dependnecies - multiple k8s packages import the same package. we don't have the tooling for that yet [05:38] wallyworld: ok, should I just 'vendorise' the structs that I need from it? Shouldn't be much at all [05:38] veebers: for now, i think that's all we can go yeah [05:38] add a TODO [05:38] wallyworld: it's isolated as we use the structs to Marshal a json []byte, so don't pass them to anything external [05:39] can do [05:39] yeah, i check that also and came to same conclusion [05:40] vino: what's your question? [05:42] wallyworld: i am trying to initialize the facade.Context which can be used to be passed in NewFacade. I use mockState and other auth params to initalise the ctx. [05:42] aparanetly it looks for *state.State [05:42] no don't do that [05:42] i am not sure if it is correct way [05:42] you had this solved earlier [05:42] oh ok. [05:42] yes. [05:43] there needs to be 2 New methods [05:43] one that takes context [05:43] yes. correct. [05:43] and another that takes the interfaces for the bits needed [05:43] the tests will use the one that takes the interfaces [05:43] and pass in the mocks [05:44] But for this api verification i am trying to call NewFacedeV1(ctx) [05:45] wallyworld: I resolved the case which u mentioned earlier. But is this feasible using mocktest ? [05:45] you can't do that from tests [05:45] aha. [05:45] ok. [05:45] there should also be a New for V1 that takes the correct params [05:45] not ctx [05:45] i do see in places like facades/client/client_test.go they do initialze. but they do use JUJUConnSuite [05:46] ok. Got it. adding a function as u mention is a good solution. [05:46] ok thanks wallyworld. Thats clear for me. [05:47] vino: no worries, all these New functions get confusing [05:48] Its ok wallyworld as and when it work on those areas its becoming clear. ty. [05:48] great :-) [06:44] I have maas in a little lab setup; I can deploy to it via maas, but when I try to bootstrap the juju controller (targeting a machine that works fine otherwise) the bootstrap bails with "ERROR failed to bootstrap model: bootstrap instance started but did not change to Deployed state: instance "tk36nk" is started but not deployed" - do I just need to [06:44] increase timeouts or is there possibly something else going wrong? [06:44] looks like it bailed at a hair over 20 minutes [07:13] kelvin: trivial PR to fix cory's service port error https://github.com/juju/juju/pull/8917 [07:14] icey: there's a bootstrap-timeout parameter you can try, juju bootstrap --config bootstrap-timeout=1200 <--- time in seconds [07:15] wallyworld yeah, I got it to bootstrap on a higher timeout ; thanks! [07:15] great! [07:15] wallyworld, looks awesome! [07:15] gr8 ty, will land and cory will be happy === frankban|afk is now known as frankban [07:59] icey: I am having something like that too will try that [08:04] icey: on your lab setup: is it routable between the real internet and (assuming) separate workers-net? [08:06] los__ my MAAS vlan can access the internet just fine, yeah [08:28] wallyworld, got this PR for fixing https://bugs.launchpad.net/juju/+bug/1777841 and https://bugs.launchpad.net/juju/+bug/1780154 -> https://github.com/juju/juju/pull/8909, would u take a look tmr? thanks. [08:28] Bug #1777841: unit-remains in goal state after being flagged for removal [08:28] Bug #1780154: goal-state error on remote relation [08:28] sure [08:28] might try tonight if i get a chance [08:28] just finishing a PR [08:29] wallyworld, not urgent, take ur time, thanks! [09:15] manadart: yo, have you got 5 minutes? [09:17] stickupkid: Sure. Stand-up HO. [10:19] manadart: CR please https://github.com/juju/juju/pull/8918 [10:21] stickupkid: Will look in a mo' [10:21] manadart: perfect, thanks :) [10:35] stickupkid: Approved it. [10:35] manadart: spot on - thanks [10:51] Hey all: seeing "node post-installation failure - 'cloudinit' running modules for config" ...idea on how to find details? (this message masks lots of things from network probs and actual bugs it seems)? [10:53] Box-stock install of latest (non-beta) Juju on MaaS, and 1) no gui was started because 2) juju status shows machine "pending" and Agent "allocating" and App is "waiting" [11:03] Nothing in /var/log/cloud-init.log or -output.log looks fail [11:17] jam: Looked at the pruning PRs. Approved both, but some comments on the txn one. [11:39] This is weirder: my juju bootstrap "failed" silently, but, the controller is working, and, I remove-application'd juju-gui and a deploy now is firing up another node and provisioning it...! [13:00] los_: the GUI is built in and not something you deploy or remove [13:00] los_: I'm not sure what you're adding there? [13:04] Looks like I was able to reproduce this bug: https://bugs.launchpad.net/juju/+bug/1642868 [13:05] Bug #1642868: remove-application does not remove failed application [13:09] But the solution from the duplicate works, my bad! $ juju resolve --no-retry haproxy/0 [13:10] rick_h: I'm doing a stock bootstrap, but, I get cloud-init errors and there is no juju-gui.... but my juju controller still works (I can deploy and hit things) [13:10] I ssh onto boxes and ping the world, nslookup, etc. and (after installing lynx) they can hit http, [13:10] I found out that it seems that full networking / routing to the world is now required for the MaaS worker machines [13:14] rick_h_: I'm trying the good ol' mariadb+mediawiki demo out and can't get that to work. Its been since 1.2x for me on Juju so lots changed [13:59] manadart: this is the lxd default region PR https://github.com/juju/juju/pull/8920 [14:13] Is there a reason why, when I build a charm using "charm build" I get a different default output folder if I set the path to my layer (builds/) vs if I simply use the name of the layer (trusty/) ? [16:15] stickupkid: Approved it. [17:02] Hey all. @thumper: Made progress since last... had to turn on full-routing between the real net and the maas net. Bootstrap does not hang; but I get cloud-init fail, with no good info in the /var/log/cloud*log [17:24] thumper: the weird thing is juju-gui is down, but, the controller is up (can deploy, etc.) [17:32] los_: what do you mean by the gui is down? [17:32] rick_h: I can't connect to it [17:32] los_: does the command 'juju gui' not give you the address info and that'll load up for you? [17:33] rick_h: I do $ juju gui ... get that URL, and no connection [17:33] los_: what's it say? is it on a different network interface that you didn't see/have access to? === valeech_ is now known as valeech [17:33] los_: but you can speak to the controller with the juju client? [17:33] los_: e.g. juju status/etc? [17:33] rick_h: when I ssh to the box(es) I can do all the network things...ping, nslookup, (and after install lynx) hit external web pages etc [17:33] los_: try using the --debug flag on the juju status command and see if the IP address you're connecting to matches the GUI url [17:33] rick_h: Yes, and even deploy NEW things [17:33] los_: right, that's outbound traffic [17:34] los_: right, so compare the addresses your juju client is using to the GUI url please [17:35] rick_h: The IP / port are both the same [17:35] los_: hmmm, I find that darn funny that it won't connect then. [17:37] rick_h: I am embarassed that I'm starting to think its browser->maas_controller->maas problem... but juju is on the maas_controller...so routing [17:38] los_: oh, you're issuing the juju commands from MAAS on the MAAS network while your local browser isn't on that network? [17:38] los_: I use sshuttle to help with that in my MAAS. [17:38] los_: https://github.com/sshuttle/sshuttle [17:39] rick_h: That's a proxy type thing? I'll check it out. I'm firing up ngrok to see if I'm nuts. I (thx!) used maas/juju a while ago when it was "MaaS must be on private network, and maas-controller will proxy..." is that still a true setup idea? [17:39] los_: there's an apt package for it and so my MAAS network is a 10.0.0.0 subnet and I do something like sshuttle -r rick@mymaas.address 10.0.0.0/24 [17:40] los_: so MAAS can proxy traffic outbound but won't do anything inbound. [17:40] los_: and honestly it's less maas proxying as you can setup the MAAS node as a forwarder [17:41] los_: when you go to load the GUI you need to get inbound via the maas and to the address of the juju controller to get at the http server where you get the web pages from [17:41] los_: so yea, sshuttle does a tunnel for you over SSH so that you can get at the private IPs from your local laptop/desktop [17:44] rick_h: OHhhhhhhhh I don't need the iptables shenanigans then!? [17:46] los_: you might need the IP tables bit to allow the MAAS nodes to tunnel out to the internet through the MAAS server. Normally you just need to set up IPV4 forwarding on the MAAS server and set that as the gateway for the MAAS managed nodes. [17:46] los_: but no, to get inbound from your laptop you can skip iptables stuff if you use the sshuttle tunnel. Then you only need inbound ssh on the maas server [17:47] rick_h: Got you, do the iptables if you had some provisioning situation that needed it... I worked this infini-timeout on the bootstrap with @thumper yesterday and had no luck until I had routing going (can't contact juju controller via API?) [17:48] los_: well so that's what I mean. When the MAAS node comes up cloud-init tells it to go get the Juju software it needs to run from streams.canonical.com and such [17:48] los_: if the MAAS nodes can't get out to get them, well then it'll be stuck [17:49] los_: and then the charm will need to be fetched from api.jujucharms.com (charmstore) so you can put it on the MAAS node and execute it [17:49] rick_h: right but I was confused cuz I saw (on the console of the maas worker machine) gazoodles of packages being updated all happy etc. [17:50] los_: right, part of what's run is apt update and apt upgrade to make sure that the latest version of cloud-init and such are available (bugs fixed/etc) [17:51] rick_h: Wowwwww this is transcendental: sshuttle is the grandson of ol' SunOS Term, and then of kermit...its self-running via python! Awesome! [17:51] rick_h: (Having old guy moment...there's been 1M hours dedicated to getting around dumb networking via these kinds of tools....so cool!) [17:52] los_: lol yea it's a darn handy tool to have in your pocket [17:52] ssh is so ubiquitous at this point [17:53] rick_h: yes yes, I just hope it works #include "crypto_worries.h" [17:54] los_: definitely use the apt package if you can so that it's pre-built for you [17:54] los_: apt get install sshuttle [17:54] rick_h: right, ok, my mind is blown: I run this on my (client, this, laptop, desktop workstation) and point it at a user with -r on the maas controller? [17:55] rick_h: Or on the maas controller pointed to the juju controller? [17:55] Aha, ... on the maas-controller! [17:56] los_: right, you're saying "if I try to hit anything in the 10.x subnet use this ssh connection to the MAAS server and go through it" [17:56] los_: so now you can use the juju client on your laptop, the browser on there, etc. [17:57] rick_h: Mind...pakow! [17:57] :) [17:57] hopefully helps you get along with things [17:59] rick_h: oh wow man,...yep, restarting from scratch (bootstrap) ...I've got a demo for our local LUG on Fri...and I want some time to write a SetiAtHome charm (with the new-to-me reactive model, solving all my RandoStartUpSequenceEventProbs that drove me off of charms a while ago...they got REALLY complicated with lockfiles for each component [17:59] SetiAtHome would be like a "dummy load" in radio [18:00] rick_h: thanks again, for-real! [18:00] los_: no problem, good luck on your project! [18:00] I've taken a lot of notes, down to error messages being confusing etc. and then I'll look at whether they are fileable [18:01] rick_h: Thanks...after that...maybe ngrok, and then after the demo, my real goal: Blender Flamenco [18:01] always happy to get feedback [18:10] rick_h: I think Juju's time is now: hybrid public/private cloud, CloudFlare's Argo in front... always preferring the private cloud (larger) made out of HA MaaS out-of-warranty servers, no RAID, 1xPSU, not "reliable" but the controllers (MaaS/Juju) are old-skool in-warranty/RAID/2xPSU/HA'd... [18:10] 1) if private cloud dies (or connectivity) flops to public cloud and you scale [18:11] 2) if public, or major fraction of networking (like VZ outage) goes, pivot to private and spin another replica on another (out of affected area) public cloud [18:11] If the workload is batchable (e.g. video rendering) then everything is restartable... [18:15] los_: very cool yea we definitely think that flexibility and consistent experience across clouds is important [18:17] rick_h: Its a very neat time, lots to learn and can get done. This Blender Flamenco thing is to produce a local render for a small co of friends of mine--they don't have the bandwidth to upload to pay services, let alone....the pay! [18:41] rick_h: ok, maximum whiney mode ON: juju gui is up (thx!) but I get a red warning bubble "Unable to list models: read tcp...connection reset by peer" [18:42] rick_h: (on my workstation) $ sshuttle -r user@maas-on-my-same-network othernetwork.0/24 [18:56] rick_h: A clue: more pop-ups of "unable to retrieve model list: unknown version (4) of interface "ModelManager" [20:03] los_: hmm, that sounds like the GUI doesn't know/talk to the API version. might have to ask hatch and company [20:03] rick_h: Yeah...I will check into that. Many the googles etc. its all working great so thanks rick! [20:04] los_: <3 that's good to hear [20:12] * hatch waves at rick_h_ [20:12] los_: hatch here is the GUI expert and might have an idea on the "unable to retrieve model list: unknown version (4) of interface "ModelManager" [20:14] los_ what version of the GUI/Juju are you using? [20:42] Morning all o/ [20:44] morning veebers [20:44] How are things going hml ? [20:45] veebers: pretty good - you? [20:49] hml: heh, bit of a broken nights sleep and now the fire is being a prick to get going ^_^ All good otherwise [21:01] There, fire sorted. I used a little known technique called: "swearing at it a bunch" [21:13] rick_h_, thumper was going to as re: snap tracks updating but thought I would try out the disource discussion [22:55] kelvin: when you grabbed the caas mysql charm for testing, it was as easy as cloning it and running "charm build ." or so, right? [23:15] veebers, yes [23:21] anastasiamac: https://pastebin.canonical.com/p/wR46sTqpXJ/ [23:37] kelvin, wallyworld I need to "ln -s" microk8s.kubectl to, um, $JUJU_DATA right, then I can add-k8s with config? [23:38] veebers, microk8s.config | juju add-k8s microk8s [23:39] is ur JUJU_DATA in a different place? [23:39] what kelvib said is all you need [23:40] wallyworld, got one question, does charm-upgrade work for caas? [23:42] awesome, thanks kelvin, wallyworld [23:45] veebers, np [23:46] wallyworld, kelvin: sorry to pester, do I need to do anything different to get the unit logs? i.e. I see message: "Hook failed: "install"", it's the changes I made, but want to see the stack trace [23:46] kelvin: it is supposed to yeah [23:46] juju debug-log -m doesn't show anything [23:46] veebers: kubectrl -n log -f [23:48] wallyworld: awesome, that's the business. I can see how I've borked up now ^_^ [23:48] :-) [23:50] wallyworld, ic, thx