[00:01] veebers: that happens and we just create watcher. don't know why we get dropped [00:02] wallyworld: ok. I haven't yet reproed the error you saw in your email, gonna leave things running, go to lunch and see if anything pops up [00:05] veebers: the error happens immediately on deploy for me [01:13] wallyworld: do you do anything more than just: juju deploy cs:~veebers/caas-mariadb ? [01:14] you need storage [01:14] either static pvs or dynaimic [01:14] or you can comment out the storage bit to test wothout [01:15] edit metadata.yaml [01:15] wallyworld: ah, I commented out storage. So I need to setup storage somehow to repro? [01:15] to repro what? [01:15] wallyworld: the error you see in the logs [01:15] which one? [01:15] there were a couple [01:16] wallyworld: or was your comment about neededing storage just regarding deploying that charm [01:16] yeah, was just about deploying [01:16] wallyworld: There was only one in your email, let me grab it real quick [01:16] sorry, haven't got it paged in [01:16] wallyworld: unexpected error: resource name may not be empty [01:16] wallyworld: I can deploy that charm (and others) and see no errors at all :-| [01:17] not even this one? ERROR juju.worker.dependency "caas-unit-provisioner" manifold worker returned [01:17] unexpected error: resource name may not be empty [01:17] (except for WARNING juju.workers.caasunitprovisioner k8s event watcher closed, restarting which I see every 6-8 minutes) [01:17] wallyworld: nope, none. [01:17] hmmm [01:18] i'll see if i can reproduce [01:18] juju debug-log -m controller --replay | grep ERROR comes back clean [01:18] wallyworld: oh, which k8s cluster where you using? That shouldn't matter though [01:18] k8s deployed on top of aws [01:22] wallyworld, veebers would you take a look this PR when u got time? https://github.com/juju/juju/pull/8997/files thanks [01:23] kelvinliu: the jenkins 2.3 and 2.4 builds work? [01:25] can do [01:25] kelvinliu: the re-run of the unit tests for your test run passed this time around, so the failure seems like an intermittent test not something you're build changes have introed [01:26] wallyworld, i think the build was working fine. [01:26] veebers, awesome! [01:37] kelvinliu: i left a few comments [01:45] kelvinliu: once you've fixed the operator image job, do a 'resume build' on the latest 2.3 commit to get a full run through [01:45] veebers, sure [01:46] wallyworld, thanks. looking now [02:01] sweet, my branch merged into develop [02:01] all that 2.4 goodness [02:02] veebers: seems like the merge job for juju/clock is also not talking to github [02:02] thumper: aye, they will all need redeployed [02:03] ok... are you sorting that? [02:03] thumper: I am, I pinged you the PR for the changes, but I'm just going to go ahead and re-deploy it now anyway as that's how I roll [02:03] * thumper nods, looking now [02:04] I'm def going to template/use a project those jobs on Friday, those 30-ish files could be 2 files [02:04] thumper: try again now on the clock PR [02:05] veebers: build noticed [02:05] sweet [02:05] I'll wait for that to complete [02:05] (just to be sure) [02:05] then merge [02:12] veebers, http://localhost:18080/view/ci-run/job/ci-run/1004/console 2.3 buildjob is passing [02:56] kelvinliu: more dep changes just landed in develop [02:58] wallyworld, yes, I saw thumper 's comments. I m fixing an 2.3 build issue now, but it's not related dep change. [02:58] np [03:42] thumper, wallyworld veebers I just sync the toml file from dev branch dependencies.tsv, please a take a look again when u got time, I m going for lunch now, be back soon. [03:46] I trust that the build will fail should we be missing things [03:57] hey wpk, nice to see you around [05:12] kelvinliu: one thing we will need to do is figure out how to allow people to switch branches, eg git checkout 2.4 will leave the vendor dir in place. and that will mess up the 2.4 build [05:13] maybe a make target to move it out of the way / rename. we don't want to have to donload the whole lot again when switch back to devleop [05:15] kelvinliu: veebers: wallyworld: with ^^ in mind, if m working on 2.3 (or 2.4) do i now neeed to run 'make dep' instead of 'godeps -u dependencies.tsv' locally? [05:16] anastasiamac: 2.3 and 2.4 will retain use of godeps and you can use via the make target if you want as the makefile is checked in code [05:17] the issue is how to move the vendor dir out of the way [05:17] wallyworld: nice. thnx [05:17] as that will mess with the 2.3 and 2.4 builds [05:19] wallyworld, the dependencies are cached on local, so rm -rf vendor then run make dep will not be too slow [05:19] wallyworld: i see. m guessing ur saw rog's suggestion in the doc on how to deal with it [05:21] anastasiamac, if u work on 2.3 or 2.4, u can just follow previous workflow, either use make dep/godeps or godeps -u dependencies.tsv [05:21] kelvinliu: yep, got it :) [05:21] anastasiamac, all should be working fine [05:21] i haven't seen the doc update yet [05:22] anastasiamac, actually 2.3 2.4 is still using godeps. [05:23] kelvinliu: when u say "still" do u mean, they will keep use it? :D [05:24] anastasiamac, yes. unless we want to migrate 2.3 and 2.4 to dep [05:25] kelvinliu: ack [05:28] jam: thumper: so rog has suggested a bash script to move vendor dirs from upstream repos away before doing a build. http://paste.ubuntu.com/p/fVBJXMnbdf/ But that seems rather error prone to me and is working against the grain. I think we are better off just embracing upstream tooling rather than coming up with a workaround to compound the issues with using a homegrown solution. Agree? [05:30] wallyworld: so that is 'for everything in $GOPATH hide vendor/ when building and then restore wehn we're done' ? [05:31] everyhting in dependencies,tsv [05:31] every upstream in there [05:31] you may have other stuff in gopath [05:31] wallyworld: and this is *instead* of using dep ? [05:31] yeah [05:32] i thik he's trying to find a workaround to not switch to it [05:32] i haven't tried the swrript, i only just noticed it as a comment on the googel doc [05:36] i can see though ways thing could mess up, eg you moved the vendor dirs, then updated dependencies.tsv; you can end up with your upstream repos messed up if stuff is removed from dependencies.tsv etc. just more moving parts to go wrong [05:36] veebers, i think the branch check condition issue has been solved. http://localhost:18080/view/ci-run/job/ci-run/1014/ http://localhost:18080/view/ci-run/job/ci-run/1010/ [05:36] kelvinliu: ok will check shortly, just truing to get this streams stuff sorted [05:37] veebers, thx [05:39] kelvinliu: did you see my question about how we deal with switch branches? [05:40] wallyworld, yes, I did. sorry, I relied to anastasiamac only [05:40] wallyworld, if u work on 2.3 or 2.4, u can just follow previous workflow, either use make dep/godeps or godeps -u dependencies.tsv [05:41] kelvinliu: oh sorry, you included me, i missed it [05:41] what i meant was not that [05:41] ah, typo, make godeps but no make dep [05:41] but when you git checkout 2.4 branch, the vendir dir will interfere with the build [05:41] so it needs to be moved out the way [05:42] when you say thinsg are cacheed loally, where is that? [05:42] wallyworld, in this case, we either gitignore vendor in 2.3/2.4 or rm -rf vendor. the dependencies are cached on local, so rm -rf vendor then run make dep will not be too slow [05:42] will it really be fast to rm -rf vendor and then later run dep ensure again? [05:42] gitignore vedor will not do anything [05:43] as the dir will still be there right? [05:43] it will just be untracked in git [05:43] wallyworld, ll $GOPATH/pkg/dep [05:44] so if that is the case and dep ensure is fast, we need a make target to remove vendor or something [05:44] and that will need to be in the 2.3 and 2.4 makefiles [05:44] wallyworld, vendor is just files copied from this cache dir but not git repos. so dep cache the git repos in there to do version ensuring [05:44] and we'll need to document that workflow [05:45] ok, we were fearful that dep ensure would dowmload everything from upstream [05:45] after the vendor dir was removed when swiching to 2.4 and back to develop [05:48] wallyworld, ic. i am thinking currently we need to run make godeps to ensure the correct revisions after switch branches because 2.3 and develop branches could have different dependencies.tsv. [05:49] thumper, wallyworld would you mind giving this a quick eyeball and a +/- 1? This is an update to the scripts jerff uses, and an update to them: http://paste.ubuntu.com/p/vmPJvP2nMx/ [05:50] kelvinliu: indeed we will and that is what we do now each time. but when switch from develop we will *also* need to move the vendor dir [05:51] wallyworld, if we have vendor gitignored on 2.3, 2.4 and develop branches. it will solve this problem? [05:51] how? [05:51] gitignore doesn't remove the dir when switch branches does it? [05:52] if there's a vendor dir there, it will still be there after git checkout 2.4 [05:52] wallyworld, if it's ignored, we don't need remove it [05:52] what stops go build from using it? [05:53] surely not gitignore? [05:53] wallyworld, ah, u r right.. [05:53] wallyworld, unless we commit the vendor, but it would be too big for us [05:54] right, hence the suggestion to adjust the make targets for godeps in 2.3 and 2.4 make files to rm -rf vendor [05:54] wallyworld, well, we can just rm -rf vendor in make godeps target [05:54] that way the workflow will be the same across all branches [05:54] so we'll need a 2.3 and 2.4 PR [05:55] the assuption is that dep ensure a second time ia very fast [05:55] assuming stuff is cached like you say [05:55] wallyworld, yes, it's fast [05:55] wallyworld, thumper if you have no issues with it I'll push and get hatch to try again (it's late for him and I need to get dinner sorted for the boy :-)) [05:56] veebers: looks ok at first glance [05:56] actually, this diff now as I sorted out 2 small issues http://paste.ubuntu.com/p/XvMm34HKVF/ [05:56] wallyworld: sweet, I'll push it up and try it, if it fails I'll revert it and try again later (tonight or tomorrow) [05:57] sgtm, no harm if there's an issue [05:57] veebers: you happy that we are good with the dep stuff? [05:57] wallyworld, well, it's fast if we think ~10s is not too slow [05:58] that's reasonable, it takes godeps a few seconds sometimes [05:58] is that on an SSD? [05:58] could I get a review plz on https://github.com/juju/juju/pull/9035 - tiny check to ensure local charms have hooks at deploy [05:58] wallyworld, yes. half of the time was for dep to figure out the revision to copy to [05:59] i can like with 10s personally [06:00] wallyworld: thumper: this PR (9035) will b heading into 2.3 and 2.4 if we can/like [06:00] wallyworld: yep, it looks like kelvinliu has the ci-run bits sorted [06:01] ok, gr8 ty [06:01] kelvinliu: since it's EOD soon let's land tomorrow so we can get discourse post etc sorted out [06:01] and not rush [06:03] wallyworld, yup [06:07] anastasiamac: done, ty, IS wil be happy [06:12] wallyworld: re: error and suggestion to build charm... we rarely make suggestions in errors coming from api (which this one is)... and digging in deploy command will be to snow-flaky... so to compromise, we could rename this error to say "poorly built charm:.." instead of "invalid charm:..." [06:12] what do u think? [06:14] invalid charm is probaly ok, cause that's what it is [06:14] ack [06:16] i was tossing and turning btw one and the other all day... at the end went with "invalid" coz it felt moer definitive, final almost.. whereas "poorly built" implied "hey, u've tried to built it but did such a bad job of it, we have to b polite here" [06:16] inalid works for me [06:17] +1 [06:34] wallyworld, thumper: after a couple iterations need to make this change: http://paste.ubuntu.com/p/rsKTJh5XMF/ [06:45] wallyworld, thumper FYI the fixes worked, the gui release/streams is sorted [07:21] veebers: thank you [08:17] Hi Everyone. I asked this question over on #maas but was redirected to ask it here. I hope someone can help. [08:18] It's about maas with juju lxds and maas's builtin proxy feature [08:18] I have setup maas and juju. I have multiple VLANS and all of them have a gateway address on the maas controller. I have enabled builtin proxy but when I juju deploy an lxd container the proxy settings are not configured and i have to manually enter the lxd and create/add the maas proxy settings into apt.d/proxy.conf file [08:19] i have been told that possible juju is not setting the inherited maas proxy setting sent down by maas. Any help would be appreciated. [08:25] jam: ^^^^ ? i can't recalloff hand the supported behaviour [08:26] naturalblue: juju has model-config for apt-proxy and http-proxy. You need to set them for Juju to set it on the containers it creates. We don't auto-populate the values from maas. [08:32] jam: ah i see. What happens if you have some lxds that need 1 proxy because they only have some bindings and others that have different bindings. Is it possible to set and apt-proxy setting to a specific lxd on deployment [08:35] naturalblue: we don't support per-machine/per-container proxy settings atm. [08:38] jam: so if i have different containers that have different bindings I will need to make manual changes to apt/apt.d/juju-proxy.conf to reflect this. It's not so bad [08:39] jam: i last thing. Is there a way to see the network settings for all units in a model. I wish to see if all machines/units/lxds have 1 common binding that i could use in the model-config and circumvent having to change multiple apt config files [08:51] naturalblue: juju status --format=yaml I believe lists bindings for applications. But that would be newly introduced in 2.4, IIRC. [08:52] i am on 2.4 so i will give it a go [08:53] jam: thats a cool command unfortunately it only shows the public-addresses and none of the other. Still handy to have though. Thanks [09:50] jam: I added the proxy settings into the model-config and redeployed the lxd units. it passed down the https and ftp into the 95-juju-proxy-settings file but for some reason didn't pass the http setting. [09:55] naturalblue: can you give the output of "juju model-config" for me ? [09:56] ok [09:56] will i paste it here or pastebin [09:56] jam: will i paste it here or pastebin [09:56] pastebin is good [09:57] col [09:58] jam: https://pastebin.com/PbpmtTuK [09:58] naturalblue: you didn't set apt-http-proxy or http-proxy [10:00] theres something not right there. i did set them and when i ran the command earlier it showed them all being set. [10:01] that paste also shows that https is not set but it was and has actually been passed down to the lxd containers [10:02] i set them using the command "juju model-default apt-http-proxy=http://172.30.100.1:8000". i did this for apt-http/https as well as juju-http/https [10:02] for some reason all the config settings for http & https have gone but the ftp has stayed [10:04] i must have ran acommand somewhere that set them back to the default rather than the one i wanted. i am still not sure the difference between juju model-config, juju model-default and juju model-defaults [10:11] it would seem that although there are 2 commands juju model-default and juju model-defaults they are both identical in what they do [10:13] jam: https://pastebin.com/rTcSL9PT This shows all the settings correct (juju moel-defaults comand) but the other command (juju model-config) only shows the ftp settings [10:23] jam: externalreality approved it yesterday, but I've made some additions. Mind taking another look at https://github.com/juju/juju/pull/9029 ? [11:32] manadart: will do [11:32] naturalblue: you'll need to "juju model-config --reset apt-https-proxy" etc. The value from model-defaults is snapshotted when the model is created, and only used again if you reset the value [11:36] jam: Yeah thanks. I worked it out and removed all the excess proxys using juju model-default --reset juju-http-proxy,juju-ftp-proxy etc [11:37] then i redepoyed the model. at first i removed the http-proxy etc instead of the juju-http-proxy but found everything just got stuck on initialising agent, so tried again with http-proxy instead of juju-http-proxy and its working away now. i'll see how it goes [11:44] naturalblue: if you set 'juju-http-proxy' then juju will set a JUJU_HTTP_PROXY value for the charms to use, but it *won't* set HTTP_PROXY [11:45] naturalblue: its been a discussion since often charms actually need to enable and disable the proxy because NO_PROXY doesn't work the way you would like [11:45] (all of the C libs only support exact IP matches, not CIDR, which means you can't just say "don't proxy everything in the local datacenter") [11:46] cory_fu: Can we release a charms.reactive update? PostgreSQL charm is failing under trusty, fixed by https://github.com/juju-solutions/charms.reactive/commit/328ae1b2f69f07059b17766a9db0586c3243cc14 [11:47] I think I can bump the version number and push it to pypi if we are good [11:51] I can release just up to my commit if the series upgrade stuff needs more baking [12:43] jam: Thanks for the help. Everyday's a schoolday :) It's working now and the deploymets have worked for me. Now to debug the openstack services themselves. [12:43] Thanks again [12:44] naturalblue: happy to help. I'm glad you got it working. [12:51] stub: +1, let's do it [12:52] cory_fu: ta. HEAD or just up to that commit? [12:53] stub: Up to HEAD, I think. I don't think the upgrade series flags will affect anyone who doesn't want to start testing them. [12:54] stub: Also, you'll want to update the VERSION file and changelog [12:54] cory_fu: Shall I call that 'preliminary support for operating system series upgrade'? [12:54] (in the changelog) [12:54] Yeah, that sounds good [12:55] Technically that will be a new feature, so 0.7.0. Or do I get to call this 1.0.0 ? [12:56] stub: :) I'm not really sure why we've been holding off on calling it 1.0.0, other than potentially reserving that for breaking changes. I'm not averse to making the switch. [12:57] stub: Oh, I think layer:basic might be range-pinned to <1.0.0 [12:57] stub: Nope, it's charms.reactive>=0.1.0,<2.0.0 [12:57] stub: GTG if you want to pull that trigger. :) [12:57] k [13:00] stub: Speaking of layer:basic, did you see my reply to the issue you opened about the import stuff. Would you like to discuss that further? I'm still leaning toward it being better than not but if it's causing problems or seems like really bad form, I'd like to try to resolve it [13:01] cory_fu: I was going to think further and do some tests. [13:01] stub: Ok, sounds good. Let me know what you find. [13:01] cory_fu: I can come up with some rationalizations, but they might just be me not liking stuffing everything into charms.layer in the first place. [13:02] cory_fu: I submitted an issue on layer:status, which looks like it is relying on naming convensions, with a way of avoiding that need. [13:03] stub: Fair enough. It suits my mental organizational scheme better but it was always intended as a convention and not a hard rule. [13:03] I honestly thought that more people would use external libraries rather than every single layer embedding a small lib. [13:03] But I've never done that myself, either, so... [13:04] With layers and interfaces requiring separate repos, there is already too much to juggle. [13:04] Yeah [13:04] And if we could stuff them together in shared repos, you don't need external libraries because they can share code already [13:05] So when interface:pgsql can point to a subdir of the pg charm source, I stop needing an external library to share my helpers like the ConnectionString class. [13:20] https://github.com/juju-solutions/charms.reactive/pull/187 [13:20] cory_fu: ^^ (tests pass locally, but I'll see what travis says too) [13:22] * stub twiddles formatting [13:36] stickupkid: I’m not having issues with localhost and develop: https://pastebin.canonical.com/p/cbHKgszbz2/ [13:36] stickupkid: which version of the lxd snap are you using? [13:36] hi all [13:37] I got strange error: bootstrap.go:538 failed to bootstrap model: cannot start bootstrap instance: unexpected: ServerError: 400 Bad Request [13:37] any idea? [13:37] hml: 3.3 as well, i'll restart :p [13:37] I am using MAAS + juju in virtualmachine [13:38] juju command: juju bootstrap maas-cloud maas-cloud-controller --bootstrap-series=bionic [13:39] hml: thanks for this, was about to go down a rabbit hole [13:40] stickupkid: did a restart work? [13:40] hml: i'll do it in a bit [13:41] stickupkid: my lxd config has been up and running for a long time, haven’t re-setup since then [13:42] hml: k, thaknk [13:42] s/thaknk/thanks/ [13:46] what I am doing wrong? [13:48] Ce faci? [13:49] drcode: try with --debug to see what it's trying to do. That feels like a cloud api call that got told no for some reason [13:49] drcode: so we would need more info on what cloud/etc you're doing [13:50] can I past is here? [13:56] drcode: I'd suggest using a pastebin like https://paste.ubuntu.com/ [14:09] thank you [14:11] https://paste.ubuntu.com/p/G2pZZwn6Rd/ [14:22] drcode: is ur maas install configured to have bionic images? [14:24] manadart: you got 5 minutes, in about 30 minutes, to pick your brains before end of day for you? [14:29] stickupkid: Sure, just say when. [14:36] hml: worked after restart [14:36] I will check it , is it recommanded to use bionic? [14:36] stickupkid: nice [14:37] do I need to setup NAT in MAAS , or juju can use MAAS proxy? [14:39] drcode: u r asking for it according to ur msg: [23:38:39] juju command: juju bootstrap maas-cloud maas-cloud-controller --bootstrap-series=bionic [14:39] ok [14:40] I think I had problem with NAT [14:40] Can I tell juju to use MAAS proxy? [14:46] hml: I was able to clear the test Neutron model networks in the end; see what you think. [14:46] manadart: ack, i’ll take a look [14:47] manadart: now? HO? [14:47] stickupkid: Ja. [20:07] rick_h_: when you get a chance, i think i have the pr comments resolved if you could pls take a 2nd look. ty! https://github.com/juju/juju/pull/9015/commits/462a644b24ab064b4ae9172e328867db428d2d96 [20:07] hml: cool ty looking [20:09] rick_h_: it helped once i stopped the square peg round hole problem :-) [20:09] hml: good to hear [20:10] hml: do we still test if the cert is parse-able? [20:10] hml: or am I missing that in there? [20:10] rick_h_: yes… - it’s checked when the filename is given [20:11] rick_h_: pollster now allows a VerifyCertFile to be provided [20:11] ah doh, yea right after it's read. missed it [20:11] rick_h_: not intuitive just reading the code. ;-) [20:12] hml: it's not bad, I just missed it. [20:40] Morning all o/ [20:42] veebers: morning [20:47] hey hml o/ hows things this morning (also known as afternoon in some places ;-)) [20:47] veebers: good. and u? [20:48] hml: all good, it's a bit cool but have the fire going [21:51] fyi, https://bugs.launchpad.net/juju/+bug/1786099 in 2.4.2 proposed [21:51] I'm aware and looking at it [21:51] it is a bit icky... [21:51] let's just say there is an easy and wrong fix [22:12] kelvinliu__: did you happen to push your ci-run changes up? [23:00] where's wally? [23:03] veebers, yes [23:03] veebers, i did [23:03] kelvinliu__: sorry as in merge it to master? [23:18] veebers, ah, im going to merge if u r happy [23:23] kelvinliu__: yes please [23:23] I'm going to be doing updates to ci-run so don't want to overwrite your changes [23:23] veebers, do it now, thx [23:24] veebers, done. [23:24] mean, cheers [23:41] thumper: want a teady bear still? [23:48] wallyworld: if thumper does not have ur attention, can i? [23:49] guess so [23:49] i you insist [23:49] yes, m draging u into a corner.. standup? [23:49] ok