/srv/irclogs.ubuntu.com/2016/08/05/#juju-dev.txt

mupBug #1610012 opened: can't migrate a model off a controller twice <migration> <juju-core:New for menno.smits> <https://launchpad.net/bugs/1610012>00:02
anastasiamacmenn0: thumper: could we do a quick ho re:migration?00:10
anastasiamacm happy to talk to one of u, if getting time with both is difficult00:11
* thumper is about to go walk the dog00:11
thumperwallyworld: blocker tag removed00:11
menn0anastasiamac: can it wait until later today please? i'm trying to get some of these bugs fixed00:11
wallyworldhuzah00:11
anastasiamacmenn0: of course \o/ ping when u have a chance00:13
menn0anastasiamac: will do00:13
wallyworldthumper: here's the muxtex bit that needs changing const prefix = "@/var/lib/juju/mutex-"00:18
wallyworld*mutex00:19
wallyworldbug 160496700:20
mupBug #1604967: Apparmor denies bind to abstract unix sockets such as @/var/lib/juju/mutex-/store-lock <snapd-interface> <Snappy:Triaged by jdstrand> <https://launchpad.net/bugs/1604967>00:20
wallyworldif we change the path, i don't think we'll need the apparmor change00:20
wallyworldwill need to test00:20
thumperwhat abstract sockets does snappy apparmor allow?00:31
thumperwallyworld: ?00:31
thumperany?00:31
thumperwallyworld: looks like a bug on snappy not juju00:32
wallyworldthumper: not a snappy bug, but a profile issue which can be solved by using path that doesn'nt need an apparmor tweak00:35
wallyworldi'll test00:35
wallyworldwe got advice from the snappy folds IIRC that's what we needed to do00:35
menn0wallyworld or thumper: http://reviews.vapour.ws/r/5378/00:50
menn0or anastasiamac or axw: ^^^ :)00:50
axwmenn0: looking00:50
axwwallyworld: thanks for hitting merge. doesn't matter now, but I was going to merge the end of the pipeline since it includes all the other commits00:51
wallyworldoops00:51
axwwallyworld: all good :)  I'll do it with the other one00:51
axwmenn0: LGTM00:56
menn0axw: thank you00:56
axwmenn0: can you please review this trivial one: http://reviews.vapour.ws/r/5379/. I had added a CloudSpec API to provisioner, but it's not needed now after your changes to use environ-tracker01:28
menn0axw: looking01:31
menn0axw: done01:33
axwmenn0: thanks01:33
niedbalskidooferlad, ping02:07
niedbalskidooferlad, dimitern https://bugs.launchpad.net/juju-core/+bug/161003702:15
mupBug #1610037: Juju2 beta14, missing network stanzas. <sts-needs-review> <juju-core:New> <https://launchpad.net/bugs/1610037>02:15
mupBug #1610037 opened: Juju2 beta14, missing network stanzas. <sts-needs-review> <juju-core:New> <https://launchpad.net/bugs/1610037>02:17
menn0yes!02:28
* menn0 quashes migration bugs02:29
hatchIf I've found, what I believe to be a regression would you prefer I re-open an old bug or create a new one?02:30
anastasiamacniedbalski: dooferlad is on paternity leave :) dimitern will be online in about 4+ hrs \o/02:31
anastasiamachatch: dpends on what bug u r planning to re-open..02:31
hatchanastasiamac: https://bugs.launchpad.net/juju-core/+bug/156658902:32
mupBug #1566589: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released by tycho-s> <https://launchpad.net/bugs/1566589>02:32
hatchThere doesn't appear to be a way to specify a custom lxd network interface name02:32
anastasiamachatch: re-open please \o/02:32
hatcheven when setting the default profile to use something else it still tries to use lxcbro02:33
hatchallllrighty, I'm just running one last test here then I'll re-open02:33
hatchthanks02:33
anastasiamachatch: please advice why u r re-opening in the bug comments :)02:34
hatchoh definitely - I'm also trying to use juju to bootstrap lxd's while running in an lxd itself02:34
hatchso it's a little funky network setup wise as it is02:35
anastasiamachatch: \o/ the more info, the better to diagnose :)02:35
=== natefinch-afk is now known as natefinch
mupBug #1566589 opened: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1566589>03:20
thumperaxw: do both volumes and filesystems have settings?03:34
thumperaxw: or was it just storage?03:34
axwthumper: neither. they both refer to "storage pools", and pools are defined in settings03:35
thumperaxw: through settings on the state.Storage type?03:35
thumperhmm...03:36
axwthumper: storage/poolmanager takes a settings manager, which state implements03:36
thumperaxw: I think what we really need to do is to get to the state where we have all the storage bits at least mostly handled, then create an environment where all these moving parts are created, then dump it and check the output03:37
thumperuntil then, I'm kinda guessing03:37
axwwallyworld thumper: have you seen this before? not sure if it's something I've caused... http://juju-ci.vapour.ws:8080/job/github-merge-juju/8636/artifact/artifacts/windows-out.log/*view*/03:38
* thumper looks03:39
wallyworldaxw: that's due to other recent work to remove supported ciphers i believe03:39
thumperaxw: that is an intermittent failure03:40
thumperbut all the extra log spam is new03:40
axwok, thanks03:40
natefinchug, did I leave log spam in?  Sorry.03:40
natefinchoh oh... I think I know what that is.. shit.  I edited the stdlib... I should probably put that bad03:41
natefinchback03:41
menn0wallyworld: do you have time to give thumper and I (and anyone else) a quick snappy intro?03:49
menn0wallyworld: I'm pretty much ready to send this test build out03:49
wallyworldmenn0: sorry, missed ping, was out having 1:1 with anastasia05:27
wallyworldmenn0: i need to do some work to tools upload (which i am doing now). if you can wait till monday....05:28
menn0wallyworld: all good... I figured out most of it myself05:30
menn0wallyworld: installing snapcraft from source was the hardest part05:30
wallyworldi had to do that too today - had to bring in some python deps05:31
wallyworldmenn0: the tools work i am doing is do stuff "just works" for snaps without upload-tools05:31
wallyworldand also cleans stuff up a bit - there's a bit of cruft there, and i reckon i can see a suspect ofr that bootstrap version issue recently05:32
menn0wallyworld: ofr?05:34
wallyworld*for05:35
wallyworldyou know i can't type :-)05:35
wallyworldaxw: i added a few things, plus a bit of info on the other config work06:20
wallyworldi'll do another pass a bit later06:20
axwwallyworld: thanks. should I add in reacting to credential changes to in-scope work?06:21
wallyworldaxw: i did that already, maybe needs a few more words06:22
axwwallyworld: ah I see. no that's fine06:22
wallyworldaxw: i also added the update clouds worker06:23
fwereade_menn0, what PR did you want me to look at? can't seem to get to reviews.vapour.ws06:48
fwereade_menn0, and if you have a moment to look at https://github.com/juju/juju/pull/5932 before the weekend, that would be awesome06:48
fwereade_menn0, (also, I'm dumb, but I couldn't find the code that prevents us from migrating mid-charm-upgrade: can you point me to it?)06:49
fwereade_thumper, if you're around you too could address my last 2 messages :)06:50
thumpermeetingology: we might not have it yet07:04
meetingologythumper: Error: "we" is not a valid command.07:04
thumperfwereade_: see above mse07:04
thumpermsg07:04
thumperbah humbug07:04
=== frankban|afk is now known as frankban
fwereade__wallyworld, long shot: do you recall what the bits starting at about state/charm_test.go:250 were testing? am confused by the non-txn ops09:09
wallyworldfwereade__: not sure off hand without a bit of thought. i don't recognise the code, it looks like it was moved from somewhere else09:13
mupBug #1610169 opened: invalid lxd config after update to 2.0-beta14-0ubuntu1~16.04.2~juju1 <juju-core:New> <https://launchpad.net/bugs/1610169>10:36
babbageclunkfwereade__, dimitern: a state.LinkLayerDevice can have multiple addresses, but a network.InterfaceInfo (and a params.NetworkConfig) only has one. Do s11:12
babbageclunkOops11:12
babbageclunkDo you think I should produce multiple NetworkConfigs for a device with multiple addresses? Or pick one somehow?11:13
dimiternbabbageclunk: multiple11:14
babbageclunkdimitern: Yeah, I was leaning towards that. Cool, thanks.11:15
dimiternbabbageclunk: more verbose representation is used in network/ (at least for now)11:15
dimiternbabbageclunk: there are tests (see networkingcommon) to convert between those IIRC11:16
dimiternbabbageclunk: in case you're wondering why - network.InterfaceInfo came to be as a way to have the full config per NIC needed to render /e/n/i11:23
babbageclunkdimitern: Yeah, I guessed that from the fields :)11:24
dimitern:)11:25
dimiternhatch: unlikely, but are you around? re bug 156658911:37
mupBug #1566589: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>11:37
dimiternrick_h_: ^^ closed that one and unassigned you from it, FYI11:38
mupBug #1566589 changed: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>11:42
fwereade__babbageclunk, I've just thought of a problem11:46
babbageclunkfwereade__: oh good11:46
fwereade__babbageclunk, we really shouldn't delete the machine document while it's got outstanding resources11:47
fwereade__babbageclunk, that might be the only thing keeping the model alive11:47
fwereade__babbageclunk, not the end of the world11:47
fwereade__babbageclunk, but it means that the provisioner should now be explicit about not *removing* machines, but on handing responsibility to something else once the instance has gone away and unblocked it11:48
mupBug #1566589 opened: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>11:48
fwereade__babbageclunk, and then the "I've finished with machine X" messages can trigger the actual *remove*11:48
fwereade__babbageclunk, one upside there is that we never have parallel responsibility for the machine, so whatever's responsible can write status without worrying11:49
fwereade__babbageclunk, how badly have I wrecked your day..?11:50
babbageclunkfwereade__: So we end up splitting machine.Remove into two parts - one that does everything except removing addresses, link layer devices and the actual remove, and the other part that really removes everything.11:51
rick_h_dimitern: ty11:51
fwereade__babbageclunk, well, I'm wondering if it's more like `Provisioner.MarkForGC([machine-3-lxd-3])` or whatever name we pick11:52
fwereade__babbageclunk, where that txn checks the machine is dead and creates the removal doc for the attention of the watcher11:52
babbageclunkfwereade__: like, moderately? :) I just got finished doing it the other way. Haven't had a chance to think through the implications yet - might not actually be much, except that I probably no longer need to move various methods off Machine if we'll still have one at the end.11:52
babbageclunkfwereade__: yeah, ok - that makes sense.11:53
fwereade__babbageclunk, it definitely hits the fiddly-txn-ops side :( but I think the removals+watch remain the same?11:53
babbageclunkfwereade__: yeah, that part is the same. And it's cleaner this way.11:54
babbageclunkfwereade__: ok, thanks for the headsup - I'll start switching over to that after lunch, should get something up for review soonish.11:55
fwereade__babbageclunk, just added some replies on http://reviews.vapour.ws/r/5366/ fwiw11:56
fwereade__babbageclunk, thanks :)11:56
mupBug #1566589 changed: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>11:57
babbageclunkfwereade__: those review comments are enlightening, thanks!12:01
* babbageclunk lunches12:01
fwereade__does anyone know why we might rewrite the revisions of charmstore charms "to handle the revision differences between unpublished and published charms in the store"?12:01
fwereade__rogpeppe, rick_h_ ^^12:01
fwereade__babbageclunk, cool, yw :D12:02
rogpeppefwereade__: what's the context?12:02
fwereade__rogpeppe, migrations12:02
rogpeppefwereade__: model migration?12:03
fwereade__rogpeppe, apparently we might have charm archives with revisions that don't match their url, and that worries me12:03
fwereade__rogpeppe, yeah12:03
fwereade__rogpeppe, and the comment mentions the charm store -- and that's all I know :)12:03
rogpeppefwereade__: where does that remark come from?12:03
fwereade__rogpeppe, apiserver/charms.go:26212:04
fwereade__rogpeppe, I think menn0's been gone for a while, it's nbd, I will mail him if it doesn't resonate with you12:04
rogpeppefwereade__: i'm looking at the code. gimme a minute or two.12:05
fwereade__rogpeppe, cheers :)12:05
rogpeppefwereade__: i don't think revisions in charm archives are a thing any more12:06
rogpeppefwereade__: i don't think the code should ever be looking at them12:06
rogpeppefwereade__: it was always a bad idea12:06
fwereade__rogpeppe, strongly agree12:06
fwereade__rogpeppe, ok, I had also sorta thought that; I will continue to poke away at it in that light12:08
rogpeppefwereade__: it looks like api.Client.UploadCharm always attaches a revision12:11
rogpeppefwereade__: so that logic is pretty much irrelevant AFAICS12:11
fwereade__rogpeppe, well... that would explain it, I suppose12:11
rogpeppefwereade__: i'd suggest changing it so that it fails if there's no revision form field specified12:12
fwereade__rogpeppe, yeah, that sounds sane to me12:12
rogpeppefwereade__: given that the charm package is still "unstable", i'd really like to remove the whole notion of revision from charm.Charm12:12
fwereade__rogpeppe, +1eMAXINT12:13
rogpeppefwereade__: :)12:13
rogpeppefwereade__: it shouldn't be hard to change in the charm package itself :)12:13
fwereade__rogpeppe, I'm pretty sure it'd be worth it despite the downstream pain12:15
rogpeppefwereade__: agreed12:15
rogpeppefwereade__: feel free to go for it12:15
fwereade__...you know, we really *don't* use .Revision() very much at all12:18
fwereade__I can certainly excise its use from juju/juju12:18
fwereade__rogpeppe, do you have a picture of how it'd impact other dependencies? how much stuff would I plunge into unbuildable catastrophe if core were suddenly using a new Charm interface?12:20
rogpeppefwereade__: the only real impact would be on the semantics of local repos12:22
rogpeppefwereade__: but we don't support local repos like that any more anyway really12:23
rogpeppefwereade__: and i bet no-one relies on the current semantics, which are bizarre12:23
fwereade__rogpeppe, yeah12:23
fwereade__rogpeppe, what about name?12:23
rogpeppefwereade__: i'd love to get rid of Name too12:24
rogpeppefwereade__: the charm store never uses it, or Revision12:24
fwereade__rogpeppe, ...and unless we *actually get rid of* Name+Revision we have this opportunity for mismatch everywhere we go12:24
rogpeppefwereade__: exactly12:24
rogpeppefwereade__: putting the name and revision in the content is a silly idea12:25
fwereade__rogpeppe, no argument here12:25
rogpeppefwereade__: i wish we'd persuaded gustavo back in the day...12:25
fwereade__:)12:25
anastasiamacdimitern: thank you for looking at this lxdbr0 bug earlier \o/ do u think that this is related? https://bugs.launchpad.net/juju-core/+bug/161016912:28
mupBug #1610169: invalid lxd config after update to 2.0-beta14-0ubuntu1~16.04.2~juju1 <juju-core:New> <https://launchpad.net/bugs/1610169>12:28
* dimitern managed wade through maas'es bind9 config and fix streams.canonical.com to 127.0.0.1 \o/ 12:31
sinzuirick_h: bug 1610243 is a blocker. The azure provider is broken. Juju cannot bootstrap13:16
mupBug #1610243: Azure provider storage account not found <azure-provider> <blocker> <bootstrap> <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610243>13:16
rick_h_sinzui: rgr looking13:17
mupBug #1610238 opened: UnitSuite.TestWithDeadUnit timed out waiting for agent to finish <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1610238>13:18
mupBug #1610239 opened: Race in src/gopkg.in/mgo.v2 <ci> <intermittent-failure> <race-condition> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1610239>13:18
mupBug #1610243 opened: Azure provider storage account not found <azure-provider> <blocker> <bootstrap> <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610243>13:18
tvansteenburghanyone know what would cause only 2 facades to be reported (UserManager and ModelManager) when logging in to the api?13:20
tvansteenburgh(juju2)13:20
tvansteenburghwell actually i'm getting more than 2, but the Client facade is missing and i'm not sure why that would be the case13:28
mupBug #1609994 changed: Race in github.com/juju/loggo global <ci> <race-condition> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1609994>13:36
tvansteenburghLogin response: {'server-version': '2.0-beta12', 'facades': [{'name': 'AllModelWatcher', 'versions': [2]}, {'name': 'Cloud', 'versions': [1]}, {'name': 'Controller', 'versions': [3]}, {'name': 'MigrationTarget', 'versions': [1]}, {'name': 'ModelManager', 'versions': [2]}, {'name': 'UserManager', 'versions': [1]}], 'server-tag': 'model-d63ff9c4-3d46-464b-8c98-9459afbef958', 'servers': [[{'type': 'ipv4', 'scope': 'local-cloud', 'por13:37
tvansteenburghhmm, do i need to login to a specific model endpoint to get the Client facade perhaps?13:38
natefinchtvansteenburgh: yeah, I think that's a new thing - they split off the controller API from the model API13:44
mupBug #1609994 opened: Race in github.com/juju/loggo global <ci> <race-condition> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1609994>13:45
mupBug #1609994 changed: Race in github.com/juju/loggo global <ci> <race-condition> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1609994>13:48
mupBug #1610254 opened: model-migration: Mongo db is not in an expected state <ci> <model-migration> <mongodb> <juju-core:Triaged> <https://launchpad.net/bugs/1610254>13:48
mupBug #1610255 opened: Cannot start bootstrap instance: DB is locked <bootstrap> <ci> <lxd-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610255>13:48
mupBug #1610254 changed: model-migration: Mongo db is not in an expected state <ci> <model-migration> <mongodb> <juju-core:Triaged> <https://launchpad.net/bugs/1610254>13:54
mupBug #1610255 changed: Cannot start bootstrap instance: DB is locked <bootstrap> <ci> <lxd-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610255>13:54
rick_h_natefinch: standup ping take 414:01
mupBug #1610254 opened: model-migration: Mongo db is not in an expected state <ci> <model-migration> <mongodb> <juju-core:Triaged> <https://launchpad.net/bugs/1610254>14:06
mupBug #1610255 opened: Cannot start bootstrap instance: DB is locked <bootstrap> <ci> <lxd-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610255>14:06
mupBug #1610260 opened: AWS Error fetching security groups EOF/timeout <bootstrap> <ec2-provider> <intermittent-failure> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1610260>14:06
niedbalskidimitern, ping14:11
niedbalskidimitern, re: 1610037, check: https://pastebin.canonical.com/162396/14:13
dimiternniedbalski: otp, will get back to you shortly14:15
dimiternniedbalski: I might be missing something, but I still can't see /e/n/i from the container in your paste14:22
dimiternniedbalski: ah, sorry - line 28714:22
dimiternniedbalski: there do those lines come from, e.g.: 'post-up ifup bond0:1' ?14:26
niedbalskidimitern, that's the host deployed by maas.14:28
dimiternniedbalski: so maas/curtin rendered that?14:29
niedbalskidimitern, yep; here is the config on the maas-ui, fyi: http://pasteboard.co/4O4UwMdWX.png14:29
hatchdimitern: thanks for the response on the bug. I noticed the beta14 email quite late and was going to confirm it in the morning. You beat me to it :)14:29
dimiternniedbalski: can you also paste /var/log/cloud-init-output.log and /var/log/juju/machine-0.log please?14:30
dimiternhatch: ;) no worries14:31
hatchdimitern: this weekend I'll resume my testing on running Juju in an LXD where the nested lxd's also get an ip on the network14:31
* hatch crosses fingers14:31
niedbalskidimitern, sure.14:32
dimiternhatch: is this on maas?14:32
dimiternhatch: or lxd provider?14:32
hatchdimitern: nope, just a raw Xenial box: Metal > LXD > Juju > LXD14:32
hatchI had the both LXD's in that diagram getting the ip's but Juju wouldn't use them14:33
hatchso hoping it will now14:33
dimiternhatch: I *think* you might be out of luck, but if you give me a few more details about what you want to test I can perhaps save you some time if it won't ever work ;)14:33
dimiternhazmat: so a plain xenial, you ssh into it, apt install juju-2, bootstrap lxd lxd, and then what?14:34
dimiternsorry hazmat ;) hatch: ^^14:34
hatchdimitern: so I've supplied a bridge to the outer LXD's so that they get a real IP following jrwren's blog post. That all worked well. Then inside that LXD I created yet another bridge pointing to the parent containers lxd and then IT was receiving ip's from DHCP14:35
hatchhowever jujud kept using the 10. ip instead of the br0 ip14:35
dimiternhatch: so now you can customize which bridge to use for lxd provider before bootstrapping, but changing the one on the default LXD profile (i.e. apt install lxd, configure, then bootstrap)14:36
dimiternhatch: once bootstrapped though, juju will insist on rendering a default /etc/default/lxd-bridge with lxdbr0 using another subnet than the one the controller container is on14:37
natefinchrick_h_: fwereade__ was a big help in finding an way forward that we can do incrementally.  Basically we'll tack jsonschema on the side of what we already have in environschema, and we can incrementally move things over to use that, rather than needing to do a big bang change.14:38
rick_h_natefinch: <3 ty fwereade__ and sounds like a solid pla14:38
rick_h_plan14:38
dimiternhatch: to make it work, you'll need to manually go to each nested container and change the bridge config14:38
hatchdimitern: that's terrible :)14:39
hatchdimitern: so my goal is to run Juju in an LXD and have each LXD it creates to get a real IP on the network so that I can actually access them14:39
hatcham I out of luck?14:39
dimiternhatch: you can do some iptables magic on the controller LXD container to let that happen14:40
rick_h_hatch: so you need juju to support remote lxd servers so juju can be in a lxd and talk to other lxd on the root machine but that's not there atm14:40
hatchrick_h_: yeah that was my first idea, which, as you just mentioned doesn't work :D14:40
babbageclunkfwereade__: Is it legit to run multiple machine.Remove() transactions in response to a single CompleteMachineRemoval(ids) call, or should I build up a big transaction from each of the machines remove ops?14:40
hatchdimitern: I had it working using `redir` but that was manual and painful :D14:41
babbageclunkfwereade__: s/from each/with each/14:41
hatchI was hoping for an easy DHCP ip :)14:41
dimiternhatch: assuming the above setup, on machine-0 (LXD controller container) you'll have 1 IP from the bridge you configured on the host machine (e.g. 10.42.0.12)14:41
dimiternhatch: then on that same machine, if you enable IP forwarding (sudo sysctl -w net.ipv4.ip_forward=1 && echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf), and add this rule:14:43
dimiternhatch: sudo iptables -t nat -A POSTROUTING -s 10.33.0.0/24 ! -d 10.33.0.0/42 -j MASQUERADE14:45
dimiternhatch: 10.33.0.0/24 is the LXD network inside the controller LXD container, while 10.42.0.0/24 is the one on its host14:46
hatchinteresting14:46
dimiternhatch: this will allow hosted containers on the controller LXD to access outside with NAT, without being on the same network as the controller14:47
dimiternalso - no need to touch anything else on each nested LXD machine14:47
hatchdimitern: so how would I access those lxd's from my desktop?14:48
dimiternhatch: but I'm trying it now to double check I'm not misaken14:48
dimiternhatch: ah :)14:48
hatchdimitern: using what I currently have I can use redir on the host lxd pointing to the child lxd and it 'works'14:48
hatchno fancy iptables stuff necessary14:49
hatchbut that's a manual step14:49
hatchIF I could get the nested lxd's a dhcp ip I'd be golden14:49
hatchwhich, I can do, but Juju doesn't use it :)14:49
hatchit uses the internal 10. ip which it can't access after I break it ;)14:49
dimiternhatch: you can only get dhcp for containers on the lxd provider anyway14:49
dimiternonly on maas the ux is better atm14:50
hatchdimitern: ok so what I'll do over the weekend is outline in detail what I'm trying to do and the steps I've taken and where I'm hitting roadblocks and then maybe we can work towards making the experience better so that this workflow is a Juju reality14:51
dimiternhatch: that'd be great! thanks, I'll be happy to try to replicate your setup and chat about improving the UX!14:51
hatchgreat thanks14:52
dimiternnp ;)14:52
dimiternniedbalski: ping (re logs?)14:53
katcofwereade__: here's the fix if you're interested: http://reviews.vapour.ws/r/5384/14:58
katconatefinch: could use a review when you get 'round to it ^^14:58
natefinchkatco: will do15:09
natefinchrick_h_: what was it you needed me to review?15:09
rick_h_natefinch: /me looks at board15:10
rick_h_natefinch: oh, to fill in for OCR a bit since Michael is out today15:10
rick_h_natefinch: so I guess a quick run of things that might have come in overnight from the other folks15:10
rick_h_natefinch: fwereade__'s branches we're going to ask the migration folks to review15:10
natefinchrick_h_: ok, sure. Wanted to make sure there wasn't something in particular15:10
rick_h_natefinch: but you're free to poke at it if you'd like to be a +2 on it15:11
natefinchrick_h_: cool15:11
rick_h_natefinch: looks like katco's branch just hit the review lane15:11
rick_h_natefinch: so that would be <3 to help quickly turn that around15:11
natefinchwill do15:11
fwereade__katco, I have unhelpful comments on http://reviews.vapour.ws/r/5384/ I'm afraid15:20
fwereade__katco, well, hopefully not unhelpful15:20
katcofwereade__: lol k tal15:20
fwereade__katco, I think the bad scenario is where we do have a misbehaving cloud, and *every* machine we try to provision goes through the same backoff process, uninterruptibly, and after N minutes reports its status and has the provisioner move on to start the process with the next15:22
katcofwereade__: yeah that is problematic. really not sure if i should try and tackle that here, or in a follow-up15:22
katcofwereade__: i mean i guess in a sense this would be "breaking" things by fixing things15:23
fwereade__katco, yeah, this is a fragile-ecosystem sort of concern15:23
katcofwereade__: if i were to set status on every retry (as we discussed, sorry), is that enough? or would the user be just as frustrated...15:24
fwereade__katco, that is a good mitigation, for sure -- can we extract timings? if we say when we will, and do what we say, that is at least *something*15:25
fwereade__katco, even if juju is not doing the smartest thing, you can see what it *is* doing15:25
katcofwereade__: yeah, they're defined explicitly in i think provisioner.go15:25
fwereade__katco, which is a lot better than a silent magic box15:25
fwereade__katco, yeah that sounds right15:26
katcofwereade__: so, do that, land it, and then take a more comprehensive look at retries, and specifically how scheduling works here?15:26
fwereade__katco, what's the total delay per machine that we'll actually have before we move on?15:26
katcofwereade__: 3 retries of 10 seconds, so 30s15:27
fwereade__katco, ah, that's not so bad15:27
fwereade__katco, mm, I wonder if it's be easy to make jupsen cut off access to provider endpoints... mattyw?15:28
mattywfwereade__, not currently, but it could be added quite easily I think15:30
mattywfwereade__, you can cut off the controller from all traffic, but that's not advisable15:31
fwereade__katco, so, ok, I think I am comfortable that what we currently have is a sane and conservative strategy, but (1) the drawbacks, and the havoc that will be wreaked if we tweak the strategy, *must* be documented in shades of terror and woe; and (2) I think it would be a good idea to cut off provider access and tell the controller to deploy a bunch of machines, and see how it actually feels in practice15:35
katcofwereade__: ok, i'll have to figure out how to do that15:36
fwereade__katco, do take a look at jupsen15:36
fwereade__katco, I bet you could hack it quickly and situationally with ease15:37
katcofwereade__: i.e. implement it in jupsen?15:37
fwereade__katco, it's just the first thing that springs to mind for inducing netsplits in juju15:37
fwereade__katco, I barely remember the internals though so I'm not sure15:38
katcofwereade__: responded to a few of your comments; require feedback15:42
natefinchkatco: I'm reviewing your branch now BTW16:04
katconatefinch: cool16:05
natefinchkatco: do we not care about the task dying in the middle anymore?  We used to check task.catacomb.Dying()16:15
katconatefinch: oh jees... i had that in there, but i've refactored this thing like 3 times. it must have fallen out. the select statement should still be in there16:17
natefinchkatco: heh, yep, I've done the same thing.16:18
natefinchI wish that kind of thing were easier to test for.... there should be a test that kills it in the middle and ensures it behaves correctly... but that's really tricky.16:19
katconatefinch: if the code were structured to be unit testable, it wouldn't be so hard16:19
katconatefinch: but as it sits, yeah. it would be some kind of weirdly shaped full-stack test16:20
dimiternfwereade__: are you still there btw?16:23
fwereade__dimitern, o/, sort of16:28
fwereade__katco, I think I've fed back16:28
katcofwereade__: ta16:28
natefinchkatco: published some comments16:30
katconatefinch: ta16:30
fwereade__katco, natefinch: IIRC menn0? was just doing some work to extract the environ dependency which is one of the big ones that drags heaps of sludge with it16:31
dimiternfwereade__: am I right to think most workers expect apicaller resource, which in turn depends on the upgrade gate to be lifted?16:31
fwereade__dimitern, ummmmmmm I forget the exact upgrade dance16:31
fwereade__dimitern, there's a couple of layers remember16:31
fwereade__dimitern, we need to complete state upgrades before letting the apiserver at it16:31
dimiternfwereade__: I'm looking at bug 1607749 and so far it seems we have a catch 22 where the proxyupdater can't start because the upgrader haven't opened the gate yet, but it NEEDS a proxy to do that16:32
mupBug #1607749: juju bootstrap fails with MAAS trunk and juju (beta12 behind a proxy too) <bootstrap> <maas-provider> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1607749>16:32
fwereade__dimitern, but then most other upgrades will be happening through the api16:32
fwereade__dimitern, I don't understand what's triggering an upgrade there16:35
mupBug #1610319 opened: Juju fails to download charm <juju-core:New> <https://launchpad.net/bugs/1610319>16:37
fwereade__dimitern, if you upgrade juju and change proxy settings at the same time I can see how that could happen16:37
dimiternfwereade__: what I think happens is this: 1) juju client cannot login due to "upgrade in progress" (ok and expected), 2) upgrader starts first, before even proxyupdater (bad), 3) upgrader tries to hit streams.c.c, fails (with set http_proxy= in the exec-start.sh job for jujud-machine-0 works!), 4) proxyupdater stops with dependency "apicaller" not present16:37
* perrito666 reads a test that breaks so many principles of good testing that makes him cry16:37
dimiternfwereade__: this is during bootstrap16:38
fwereade__dimitern, why are we bootstrapping with tools that immediately want to replace themselves?16:38
dimiternfwereade__: now that's interesting - I'm not passing --upload-tools, but building from source (tag juju-2.0-beta12)16:40
mupBug #1610319 changed: Juju fails to download charm <juju-core:New> <https://launchpad.net/bugs/1610319>16:40
dimiternfwereade__: and I can see in the logs current version is juju-2.0-beta12.1, want version juju-2.0-beta1216:40
fwereade__dimitern, I *think* the upgrade-on-bootstrap is the problem16:41
fwereade__dimitern, *but* I would be super supportive of something that cached proxy settings in agent config and set them up as early as possible16:42
fwereade__dimitern, leave the updating to the worker, and make it update the cache too16:42
dimiternfwereade__: --auto-upgrade is not passed to juju bootstrap, so I'm not sure why the upgrade is triggered16:43
fwereade__dimitern, I bet there's some subtle version mismatch16:43
dimiternfwereade__: what you suggest makes perfect sense though16:43
fwereade__dimitern, honestly they should probably go into agent config at cloudconfig time16:44
dimiternfwereade__: yeah16:44
fwereade__dimitern, I don't like putting things in agent config unless I *have* to, but this makes a better case than most16:44
mupBug #1610319 opened: Juju fails to download charm <juju-core:New> <https://launchpad.net/bugs/1610319>16:46
dimiternwell beta15 apparently works better than beta12 in that same scenario :/16:48
=== frankban is now known as frankban|afk
rick_h_katco: ping, assigned a bug your way that might be fixed with your last chunk of work17:16
katcorick_h_: cool, ta17:16
rick_h_katco: can you see if you can replicate with the given steps and trunk and if so it sounds like it might be related to the revision issue you were chasing17:16
katcorick_h_: k17:16
* rick_h_ hopes it's an easy "fix already comitted" :)17:16
katcothat would be nice17:16
* rick_h_ goes to grab lunchables17:18
natefinchsinzui: reviewboard is borken... getting 500s who's in charge of that now?  G17:49
sinzuinatefinch: no one17:52
sinzuinatefinch: I can take a look17:52
sinzuinatefinch: is there a specific url that gives a 500?17:53
* sinzui visits app17:53
natefinchsinzui: looks like any specific review url17:53
natefinchsinzui: or.. not17:53
sinzuinatefinch: I am not seeing errors17:53
natefinchsinzui: http://reviews.vapour.ws/r/5355/17:53
sinzuinatefinch: ty17:54
sinzuijust that one url, the neighbors are fine17:54
sinzuinatefinch: there are seveal errors like this in the log https://pastebin.canonical.com/162540/ all are for /r/5355/. I see me, you and axw.18:01
natefinchlol python18:02
sinzuinatefinch: I think the data for this review is bad. I will see what I can do18:02
natefinchsinzui: there's a way to delete it and recreate it... but I think it requires admin rights to really really delete18:03
sinzuinatefinch: yeah. I am hunting for that power, or a command line util on the host18:03
katcosinzui: natefinch: ericsnow still works for canonical, just saying :)18:07
natefinchkatco: lol good point18:07
katcoi don't think it's *that* much of an imposition to ask for instructions 1x. but we should write them down18:08
alexisbperrito666, ping18:15
perrito666alexisb: pong18:15
sinzuinatefinch: do you know who created it? I have a list of reviews, but no id to match it to the url18:15
natefinchsinzui: andrew18:16
* natefinch is pinging ericsnow18:17
ericsnownatefinch: what's up?18:18
katcoericsnow: hey! o/18:19
perrito666wow, instant summoning18:19
ericsnow:)18:19
katcowwitzel3: summon? for old times sake?18:19
natefinchericsnow: I think we need to perma-delete a review so it can be recreated18:19
* perrito666 burns incense in a monument for ericsnow to pray for the fixing of review board18:19
natefinchericsnow: but I'm not sure any of us are admins, and IIRC one needs to be an admin to do it18:20
natefinchericsnow: start of the story, this review returns a 500: http://reviews.vapour.ws/r/5355/18:20
ericsnownatefinch: yep18:20
ericsnownatefinch: looks like katco still is an admin18:21
natefinchkatco is the new ericsnow.  It is known.18:21
katcoi quit18:21
natefinchlol18:21
katcogood working with you folks18:21
perrito666please make me admin too, I know my way around django18:22
perrito666and that gives us some redundancy18:22
natefinchperrito666 is the new ericsnow18:22
ericsnow(also admin: alexisb, cmars, frobware, fwereade__, thumper, jam, sinzui, and wallyworld)18:22
katcorick_h_: https://bugs.launchpad.net/juju-core/+bug/1610319/comments/518:23
mupBug #1610319: Juju fails to download charm <juju-core:Triaged by cox-katherine-e> <https://launchpad.net/bugs/1610319>18:23
alexisbericsnow, what are we admin to?18:24
natefinchericsnow: nice.  Good.  Sorry, I didn't realize.  That's a good list.  I agree with putting perrito666 on there... this really should be dev's job to maintain.18:24
natefinchalexisb: reviewboard18:24
alexisbaaah yeah ok18:25
ericsnowalexisb: reviewboard (click on your username and you'll see an "Admin" link)18:25
* katco goes for lunch18:25
ericsnownatefinch: all good now?18:25
natefinchericsnow: is there a trick to permadeleting a review?  I don't have the admin UI, so I don't know how to do it.  Does the author have to delete it first?18:26
ericsnownatefinch: it's a third option in the "Close" menu: "Delete Permanently"18:27
rick_h_  katco <3 ty much18:27
natefinchericsnow: cool, thanks.18:27
ericsnownatefinch: take care!18:27
natefinchericsnow: I release you from the summons.  Thanks for the help.18:27
* natefinch makes a page in the wiki for reviewboard tips18:29
alexisbthank you natefinch18:29
fwereade__that was a truly spectacular CL, by the way; I think it's the only time RB's been broken by sheer awesomeness18:32
alexisb"broken by sheer awesomeness"18:33
alexisb:)18:33
perrito666I would love to open that issue in rb18:35
natefinchperrito666: are you an admin on rb now?18:35
perrito666"rb breaks when excessive awesomeness present in patch"18:35
perrito666natefinch: checking18:36
perrito666natefinch: nope, still a mere mortal18:37
mupBug #1604955 changed: TestUpdateStatusTicker can fail with timeout <ci> <intermittent-failure> <test-failure> <juju-core:Fix Released by fwereade> <https://launchpad.net/bugs/1604955>18:40
mupBug #1608105 changed: LXD no longer activates all interfaces on initial deploy when using MAAS2rc3 and JUJU Beta13 <juju-core:Fix Released> <https://launchpad.net/bugs/1608105>18:40
rick_h_katco: can you add a link to your PR in your aws storage card please?18:41
natefinchalexisb, sinzui: can one of you make perrito666 and me admins on reviewboard?18:46
natefinchI don't think there's any reason not to have a ton of admins18:46
alexisbnatefinch, yep one se18:47
alexisbc18:47
sinzuinatefinch: Fix (-ish) rb fellover parsing the first change set. I unlinked it http://reviews.vapour.ws/r/5355/18:47
sinzuinatefinch: I think I can18:48
alexisbsinzui, I got it18:48
alexisbperrito666, you should have all powers now18:48
alexisbnatefinch, doing yours next18:48
natefinchalexisb: cool, thanks18:48
alexisbok natefinch make sure you are all powerful18:49
sinzuinatefinch: My suspision that a comment couldn't be parsed was wrong. I am surprised that the changesets are parse as markdown.18:50
natefinchsinzui: lol, it parses the PR/commit messages as markdown?  that's horrible18:50
natefinchsinzui: I mean, it still shouldn't *break*18:50
sinzuinatefinch: I think it parsed the diff as markdown!18:51
natefinchlol18:51
sinzuinatefinch: the comments were fine.18:51
sinzuiI understand the schema now so I wont be slow the next time this happens18:52
natefinchwell, good18:52
natefinchalexisb: thanks18:53
perrito666natefinch: alexisb http://stream1.gifsoup.com/view/572272/he-man-o.gif18:54
alexisbperrito666, I loved he-man growing up, I so wanted to be she-ra18:56
natefinchnice nice... yeah, he-man was one of the few licensed toys I had growing up.  I think my mother still has some of them in the attic somewhere.18:56
natefinchalexisb: are you an admin on https://github.com/juju/environschema/ ?  I need a v2 branch there18:58
rick_h_natefinch: I can help with that18:58
natefinchrick_h_: thanks18:59
rick_h_natefinch: https://github.com/juju/environschema/tree/v2 when you're sure it's stable18:59
rick_h_natefinch: let's make sure to go back and make it the default branch of the project18:59
natefinchrick_h_: yep19:00
rick_h_and with that I'm going to run to get the boy, have a good weekend folks19:00
natefinchrick_h_: see ya19:00
perrito666yeah, he man was strategically broadcast at the time kids get out of school and get together to drink tea with cookies19:09
mupBug #1610319 changed: Juju fails to download charm <juju-core:Fix Committed by cox-katherine-e> <https://launchpad.net/bugs/1610319>19:41
mupBug #1610397 opened: juju2, maas2, cloud deployment failure when two domains are used. <kanban-cross-team> <landscape> <juju-core:Triaged by rharding> <MAAS:New> <nova-cloud-controller (Juju Charms Collection):New> <https://launchpad.net/bugs/1610397>19:41
mupBug #1610397 changed: juju2, maas2, cloud deployment failure when two domains are used. <kanban-cross-team> <landscape> <juju-core:Triaged by rharding> <MAAS:New> <nova-cloud-controller (Juju Charms Collection):New> <https://launchpad.net/bugs/1610397>19:44
natefinchperrito666: I need another brain19:46
perrito666natefinch: have you tried amazon?19:49
mupBug #1610397 opened: juju2, maas2, cloud deployment failure when two domains are used. <kanban-cross-team> <landscape> <juju-core:Triaged by rharding> <MAAS:New> <nova-cloud-controller (Juju Charms Collection):New> <https://launchpad.net/bugs/1610397>19:50
natefinchperrito666: got time for a hangout?19:51
perrito666natefinch: sure, gimme a sec19:51
natefinchperrito666: when you're ready: gah, dammit19:53
natefinchlol, wrong copy and paste19:53
natefinchhttps://hangouts.google.com/hangouts/_/canonical.com/moonstone?authuser=119:53
mbruzekCan I get someone to triage this bug: https://bugs.launchpad.net/juju-core/+bug/1609893 I know it is not a high priority probem but I am able to reproduce it again today.20:53
mupBug #1609893: juju status returns ERROR not logged in <usability> <juju-core:Triaged> <https://launchpad.net/bugs/1609893>20:53
=== natefinch is now known as natefinch-afk
mupBug #1610450 opened: feature request: suspend models <juju-core:New> <https://launchpad.net/bugs/1610450>21:29
perrito666aghh really blocked again?22:40
perrito666alexisb: is it really that blocking?22:41

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!