[00:02] Bug #1610012 opened: can't migrate a model off a controller twice

[00:10] menn0: thumper: could we do a quick ho re:migration? [00:11] m happy to talk to one of u, if getting time with both is difficult [00:11] * thumper is about to go walk the dog [00:11] wallyworld: blocker tag removed [00:11] anastasiamac: can it wait until later today please? i'm trying to get some of these bugs fixed [00:11] huzah [00:13] menn0: of course \o/ ping when u have a chance [00:13] anastasiamac: will do [00:18] thumper: here's the muxtex bit that needs changing const prefix = "@/var/lib/juju/mutex-" [00:19] *mutex [00:20] bug 1604967 [00:20] Bug #1604967: Apparmor denies bind to abstract unix sockets such as @/var/lib/juju/mutex-/store-lock [00:20] if we change the path, i don't think we'll need the apparmor change [00:20] will need to test [00:31] what abstract sockets does snappy apparmor allow? [00:31] wallyworld: ? [00:31] any? [00:32] wallyworld: looks like a bug on snappy not juju [00:35] thumper: not a snappy bug, but a profile issue which can be solved by using path that doesn'nt need an apparmor tweak [00:35] i'll test [00:35] we got advice from the snappy folds IIRC that's what we needed to do [00:50] wallyworld or thumper: http://reviews.vapour.ws/r/5378/ [00:50] or anastasiamac or axw: ^^^ :) [00:50] menn0: looking [00:51] wallyworld: thanks for hitting merge. doesn't matter now, but I was going to merge the end of the pipeline since it includes all the other commits [00:51] oops [00:51] wallyworld: all good :) I'll do it with the other one [00:56] menn0: LGTM [00:56] axw: thank you [01:28] menn0: can you please review this trivial one: http://reviews.vapour.ws/r/5379/. I had added a CloudSpec API to provisioner, but it's not needed now after your changes to use environ-tracker [01:31] axw: looking [01:33] axw: done [01:33] menn0: thanks [02:07] dooferlad, ping [02:15] dooferlad, dimitern https://bugs.launchpad.net/juju-core/+bug/1610037 [02:15] Bug #1610037: Juju2 beta14, missing network stanzas.

[02:17] Bug #1610037 opened: Juju2 beta14, missing network stanzas.

[02:28] yes! [02:29] * menn0 quashes migration bugs [02:30] If I've found, what I believe to be a regression would you prefer I re-open an old bug or create a new one? [02:31] niedbalski: dooferlad is on paternity leave :) dimitern will be online in about 4+ hrs \o/ [02:31] hatch: dpends on what bug u r planning to re-open.. [02:32] anastasiamac: https://bugs.launchpad.net/juju-core/+bug/1566589 [02:32] Bug #1566589: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface [02:32] There doesn't appear to be a way to specify a custom lxd network interface name [02:32] hatch: re-open please \o/ [02:33] even when setting the default profile to use something else it still tries to use lxcbro [02:33] allllrighty, I'm just running one last test here then I'll re-open [02:33] thanks [02:34] hatch: please advice why u r re-opening in the bug comments :) [02:34] oh definitely - I'm also trying to use juju to bootstrap lxd's while running in an lxd itself [02:35] so it's a little funky network setup wise as it is [02:35] hatch: \o/ the more info, the better to diagnose :) === natefinch-afk is now known as natefinch [03:20] Bug #1566589 opened: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface [03:34] axw: do both volumes and filesystems have settings? [03:34] axw: or was it just storage? [03:35] thumper: neither. they both refer to "storage pools", and pools are defined in settings [03:35] axw: through settings on the state.Storage type? [03:36] hmm... [03:36] thumper: storage/poolmanager takes a settings manager, which state implements [03:37] axw: I think what we really need to do is to get to the state where we have all the storage bits at least mostly handled, then create an environment where all these moving parts are created, then dump it and check the output [03:37] until then, I'm kinda guessing [03:38] wallyworld thumper: have you seen this before? not sure if it's something I've caused... http://juju-ci.vapour.ws:8080/job/github-merge-juju/8636/artifact/artifacts/windows-out.log/*view*/ [03:39] * thumper looks [03:39] axw: that's due to other recent work to remove supported ciphers i believe [03:40] axw: that is an intermittent failure [03:40] but all the extra log spam is new [03:40] ok, thanks [03:40] ug, did I leave log spam in? Sorry. [03:41] oh oh... I think I know what that is.. shit. I edited the stdlib... I should probably put that bad [03:41] back [03:49] wallyworld: do you have time to give thumper and I (and anyone else) a quick snappy intro? [03:49] wallyworld: I'm pretty much ready to send this test build out [05:27] menn0: sorry, missed ping, was out having 1:1 with anastasia [05:28] menn0: i need to do some work to tools upload (which i am doing now). if you can wait till monday.... [05:30] wallyworld: all good... I figured out most of it myself [05:30] wallyworld: installing snapcraft from source was the hardest part [05:31] i had to do that too today - had to bring in some python deps [05:31] menn0: the tools work i am doing is do stuff "just works" for snaps without upload-tools [05:32] and also cleans stuff up a bit - there's a bit of cruft there, and i reckon i can see a suspect ofr that bootstrap version issue recently [05:34] wallyworld: ofr? [05:35] *for [05:35] you know i can't type :-) [06:20] axw: i added a few things, plus a bit of info on the other config work [06:20] i'll do another pass a bit later [06:21] wallyworld: thanks. should I add in reacting to credential changes to in-scope work? [06:22] axw: i did that already, maybe needs a few more words [06:22] wallyworld: ah I see. no that's fine [06:23] axw: i also added the update clouds worker [06:48] menn0, what PR did you want me to look at? can't seem to get to reviews.vapour.ws [06:48] menn0, and if you have a moment to look at https://github.com/juju/juju/pull/5932 before the weekend, that would be awesome [06:49] menn0, (also, I'm dumb, but I couldn't find the code that prevents us from migrating mid-charm-upgrade: can you point me to it?) [06:50] thumper, if you're around you too could address my last 2 messages :) [07:04] meetingology: we might not have it yet [07:04] thumper: Error: "we" is not a valid command. [07:04] fwereade_: see above mse [07:04] msg [07:04] bah humbug === frankban|afk is now known as frankban [09:09] wallyworld, long shot: do you recall what the bits starting at about state/charm_test.go:250 were testing? am confused by the non-txn ops [09:13] fwereade__: not sure off hand without a bit of thought. i don't recognise the code, it looks like it was moved from somewhere else [10:36] Bug #1610169 opened: invalid lxd config after update to 2.0-beta14-0ubuntu1~16.04.2~juju1 [11:12] fwereade__, dimitern: a state.LinkLayerDevice can have multiple addresses, but a network.InterfaceInfo (and a params.NetworkConfig) only has one. Do s [11:12] Oops [11:13] Do you think I should produce multiple NetworkConfigs for a device with multiple addresses? Or pick one somehow? [11:14] babbageclunk: multiple [11:15] dimitern: Yeah, I was leaning towards that. Cool, thanks. [11:15] babbageclunk: more verbose representation is used in network/ (at least for now) [11:16] babbageclunk: there are tests (see networkingcommon) to convert between those IIRC [11:23] babbageclunk: in case you're wondering why - network.InterfaceInfo came to be as a way to have the full config per NIC needed to render /e/n/i [11:24] dimitern: Yeah, I guessed that from the fields :) [11:25] :) [11:37] hatch: unlikely, but are you around? re bug 1566589 [11:37] Bug #1566589: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface [11:38] rick_h_: ^^ closed that one and unassigned you from it, FYI [11:42] Bug #1566589 changed: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface [11:46] babbageclunk, I've just thought of a problem [11:46] fwereade__: oh good [11:47] babbageclunk, we really shouldn't delete the machine document while it's got outstanding resources [11:47] babbageclunk, that might be the only thing keeping the model alive [11:47] babbageclunk, not the end of the world [11:48] babbageclunk, but it means that the provisioner should now be explicit about not *removing* machines, but on handing responsibility to something else once the instance has gone away and unblocked it [11:48] Bug #1566589 opened: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface [11:48] babbageclunk, and then the "I've finished with machine X" messages can trigger the actual *remove* [11:49] babbageclunk, one upside there is that we never have parallel responsibility for the machine, so whatever's responsible can write status without worrying [11:50] babbageclunk, how badly have I wrecked your day..? [11:51] fwereade__: So we end up splitting machine.Remove into two parts - one that does everything except removing addresses, link layer devices and the actual remove, and the other part that really removes everything. [11:51] dimitern: ty [11:52] babbageclunk, well, I'm wondering if it's more like `Provisioner.MarkForGC([machine-3-lxd-3])` or whatever name we pick [11:52] babbageclunk, where that txn checks the machine is dead and creates the removal doc for the attention of the watcher [11:52] fwereade__: like, moderately? :) I just got finished doing it the other way. Haven't had a chance to think through the implications yet - might not actually be much, except that I probably no longer need to move various methods off Machine if we'll still have one at the end. [11:53] fwereade__: yeah, ok - that makes sense. [11:53] babbageclunk, it definitely hits the fiddly-txn-ops side :( but I think the removals+watch remain the same? [11:54] fwereade__: yeah, that part is the same. And it's cleaner this way. [11:55] fwereade__: ok, thanks for the headsup - I'll start switching over to that after lunch, should get something up for review soonish. [11:56] babbageclunk, just added some replies on http://reviews.vapour.ws/r/5366/ fwiw [11:56] babbageclunk, thanks :) [11:57] Bug #1566589 changed: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface [12:01] fwereade__: those review comments are enlightening, thanks! [12:01] * babbageclunk lunches [12:01] does anyone know why we might rewrite the revisions of charmstore charms "to handle the revision differences between unpublished and published charms in the store"? [12:01] rogpeppe, rick_h_ ^^ [12:02] babbageclunk, cool, yw :D [12:02] fwereade__: what's the context? [12:02] rogpeppe, migrations [12:03] fwereade__: model migration? [12:03] rogpeppe, apparently we might have charm archives with revisions that don't match their url, and that worries me [12:03] rogpeppe, yeah [12:03] rogpeppe, and the comment mentions the charm store -- and that's all I know :) [12:03] fwereade__: where does that remark come from? [12:04] rogpeppe, apiserver/charms.go:262 [12:04] rogpeppe, I think menn0's been gone for a while, it's nbd, I will mail him if it doesn't resonate with you [12:05] fwereade__: i'm looking at the code. gimme a minute or two. [12:05] rogpeppe, cheers :) [12:06] fwereade__: i don't think revisions in charm archives are a thing any more [12:06] fwereade__: i don't think the code should ever be looking at them [12:06] fwereade__: it was always a bad idea [12:06] rogpeppe, strongly agree [12:08] rogpeppe, ok, I had also sorta thought that; I will continue to poke away at it in that light [12:11] fwereade__: it looks like api.Client.UploadCharm always attaches a revision [12:11] fwereade__: so that logic is pretty much irrelevant AFAICS [12:11] rogpeppe, well... that would explain it, I suppose [12:12] fwereade__: i'd suggest changing it so that it fails if there's no revision form field specified [12:12] rogpeppe, yeah, that sounds sane to me [12:12] fwereade__: given that the charm package is still "unstable", i'd really like to remove the whole notion of revision from charm.Charm [12:13] rogpeppe, +1eMAXINT [12:13] fwereade__: :) [12:13] fwereade__: it shouldn't be hard to change in the charm package itself :) [12:15] rogpeppe, I'm pretty sure it'd be worth it despite the downstream pain [12:15] fwereade__: agreed [12:15] fwereade__: feel free to go for it [12:18] ...you know, we really *don't* use .Revision() very much at all [12:18] I can certainly excise its use from juju/juju [12:20] rogpeppe, do you have a picture of how it'd impact other dependencies? how much stuff would I plunge into unbuildable catastrophe if core were suddenly using a new Charm interface? [12:22] fwereade__: the only real impact would be on the semantics of local repos [12:23] fwereade__: but we don't support local repos like that any more anyway really [12:23] fwereade__: and i bet no-one relies on the current semantics, which are bizarre [12:23] rogpeppe, yeah [12:23] rogpeppe, what about name? [12:24] fwereade__: i'd love to get rid of Name too [12:24] fwereade__: the charm store never uses it, or Revision [12:24] rogpeppe, ...and unless we *actually get rid of* Name+Revision we have this opportunity for mismatch everywhere we go [12:24] fwereade__: exactly [12:25] fwereade__: putting the name and revision in the content is a silly idea [12:25] rogpeppe, no argument here [12:25] fwereade__: i wish we'd persuaded gustavo back in the day... [12:25] :) [12:28] dimitern: thank you for looking at this lxdbr0 bug earlier \o/ do u think that this is related? https://bugs.launchpad.net/juju-core/+bug/1610169 [12:28] Bug #1610169: invalid lxd config after update to 2.0-beta14-0ubuntu1~16.04.2~juju1 [12:31] * dimitern managed wade through maas'es bind9 config and fix streams.canonical.com to 127.0.0.1 \o/ [13:16] rick_h: bug 1610243 is a blocker. The azure provider is broken. Juju cannot bootstrap [13:16] Bug #1610243: Azure provider storage account not found

[13:17] sinzui: rgr looking [13:18] Bug #1610238 opened: UnitSuite.TestWithDeadUnit timed out waiting for agent to finish

[13:18] Bug #1610239 opened: Race in src/gopkg.in/mgo.v2

[13:18] Bug #1610243 opened: Azure provider storage account not found

[13:20] anyone know what would cause only 2 facades to be reported (UserManager and ModelManager) when logging in to the api? [13:20] (juju2) [13:28] well actually i'm getting more than 2, but the Client facade is missing and i'm not sure why that would be the case [13:36] Bug #1609994 changed: Race in github.com/juju/loggo global

[13:37] Login response: {'server-version': '2.0-beta12', 'facades': [{'name': 'AllModelWatcher', 'versions': [2]}, {'name': 'Cloud', 'versions': [1]}, {'name': 'Controller', 'versions': [3]}, {'name': 'MigrationTarget', 'versions': [1]}, {'name': 'ModelManager', 'versions': [2]}, {'name': 'UserManager', 'versions': [1]}], 'server-tag': 'model-d63ff9c4-3d46-464b-8c98-9459afbef958', 'servers': [[{'type': 'ipv4', 'scope': 'local-cloud', 'por [13:38] hmm, do i need to login to a specific model endpoint to get the Client facade perhaps? [13:44] tvansteenburgh: yeah, I think that's a new thing - they split off the controller API from the model API [13:45] Bug #1609994 opened: Race in github.com/juju/loggo global

[13:48] Bug #1609994 changed: Race in github.com/juju/loggo global

[13:48] Bug #1610254 opened: model-migration: Mongo db is not in an expected state

[13:48] Bug #1610255 opened: Cannot start bootstrap instance: DB is locked

[13:54] Bug #1610254 changed: model-migration: Mongo db is not in an expected state

[13:54] Bug #1610255 changed: Cannot start bootstrap instance: DB is locked

[14:01] natefinch: standup ping take 4 [14:06] Bug #1610254 opened: model-migration: Mongo db is not in an expected state

[14:06] Bug #1610255 opened: Cannot start bootstrap instance: DB is locked

[14:06] Bug #1610260 opened: AWS Error fetching security groups EOF/timeout

[14:11] dimitern, ping [14:13] dimitern, re: 1610037, check: https://pastebin.canonical.com/162396/ [14:15] niedbalski: otp, will get back to you shortly [14:22] niedbalski: I might be missing something, but I still can't see /e/n/i from the container in your paste [14:22] niedbalski: ah, sorry - line 287 [14:26] niedbalski: there do those lines come from, e.g.: 'post-up ifup bond0:1' ? [14:28] dimitern, that's the host deployed by maas. [14:29] niedbalski: so maas/curtin rendered that? [14:29] dimitern, yep; here is the config on the maas-ui, fyi: http://pasteboard.co/4O4UwMdWX.png [14:29] dimitern: thanks for the response on the bug. I noticed the beta14 email quite late and was going to confirm it in the morning. You beat me to it :) [14:30] niedbalski: can you also paste /var/log/cloud-init-output.log and /var/log/juju/machine-0.log please? [14:31] hatch: ;) no worries [14:31] dimitern: this weekend I'll resume my testing on running Juju in an LXD where the nested lxd's also get an ip on the network [14:31] * hatch crosses fingers [14:32] dimitern, sure. [14:32] hatch: is this on maas? [14:32] hatch: or lxd provider? [14:32] dimitern: nope, just a raw Xenial box: Metal > LXD > Juju > LXD [14:33] I had the both LXD's in that diagram getting the ip's but Juju wouldn't use them [14:33] so hoping it will now [14:33] hatch: I *think* you might be out of luck, but if you give me a few more details about what you want to test I can perhaps save you some time if it won't ever work ;) [14:34] hazmat: so a plain xenial, you ssh into it, apt install juju-2, bootstrap lxd lxd, and then what? [14:34] sorry hazmat ;) hatch: ^^ [14:35] dimitern: so I've supplied a bridge to the outer LXD's so that they get a real IP following jrwren's blog post. That all worked well. Then inside that LXD I created yet another bridge pointing to the parent containers lxd and then IT was receiving ip's from DHCP [14:35] however jujud kept using the 10. ip instead of the br0 ip [14:36] hatch: so now you can customize which bridge to use for lxd provider before bootstrapping, but changing the one on the default LXD profile (i.e. apt install lxd, configure, then bootstrap) [14:37] hatch: once bootstrapped though, juju will insist on rendering a default /etc/default/lxd-bridge with lxdbr0 using another subnet than the one the controller container is on [14:38] rick_h_: fwereade__ was a big help in finding an way forward that we can do incrementally. Basically we'll tack jsonschema on the side of what we already have in environschema, and we can incrementally move things over to use that, rather than needing to do a big bang change. [14:38] natefinch: <3 ty fwereade__ and sounds like a solid pla [14:38] plan [14:38] hatch: to make it work, you'll need to manually go to each nested container and change the bridge config [14:39] dimitern: that's terrible :) [14:39] dimitern: so my goal is to run Juju in an LXD and have each LXD it creates to get a real IP on the network so that I can actually access them [14:39] am I out of luck? [14:40] hatch: you can do some iptables magic on the controller LXD container to let that happen [14:40] hatch: so you need juju to support remote lxd servers so juju can be in a lxd and talk to other lxd on the root machine but that's not there atm [14:40] rick_h_: yeah that was my first idea, which, as you just mentioned doesn't work :D [14:40] fwereade__: Is it legit to run multiple machine.Remove() transactions in response to a single CompleteMachineRemoval(ids) call, or should I build up a big transaction from each of the machines remove ops? [14:41] dimitern: I had it working using `redir` but that was manual and painful :D [14:41] fwereade__: s/from each/with each/ [14:41] I was hoping for an easy DHCP ip :) [14:41] hatch: assuming the above setup, on machine-0 (LXD controller container) you'll have 1 IP from the bridge you configured on the host machine (e.g. 10.42.0.12) [14:43] hatch: then on that same machine, if you enable IP forwarding (sudo sysctl -w net.ipv4.ip_forward=1 && echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf), and add this rule: [14:45] hatch: sudo iptables -t nat -A POSTROUTING -s 10.33.0.0/24 ! -d 10.33.0.0/42 -j MASQUERADE [14:46] hatch: 10.33.0.0/24 is the LXD network inside the controller LXD container, while 10.42.0.0/24 is the one on its host [14:46] interesting [14:47] hatch: this will allow hosted containers on the controller LXD to access outside with NAT, without being on the same network as the controller [14:47] also - no need to touch anything else on each nested LXD machine [14:48] dimitern: so how would I access those lxd's from my desktop? [14:48] hatch: but I'm trying it now to double check I'm not misaken [14:48] hatch: ah :) [14:48] dimitern: using what I currently have I can use redir on the host lxd pointing to the child lxd and it 'works' [14:49] no fancy iptables stuff necessary [14:49] but that's a manual step [14:49] IF I could get the nested lxd's a dhcp ip I'd be golden [14:49] which, I can do, but Juju doesn't use it :) [14:49] it uses the internal 10. ip which it can't access after I break it ;) [14:49] hatch: you can only get dhcp for containers on the lxd provider anyway [14:50] only on maas the ux is better atm [14:51] dimitern: ok so what I'll do over the weekend is outline in detail what I'm trying to do and the steps I've taken and where I'm hitting roadblocks and then maybe we can work towards making the experience better so that this workflow is a Juju reality [14:51] hatch: that'd be great! thanks, I'll be happy to try to replicate your setup and chat about improving the UX! [14:52] great thanks [14:52] np ;) [14:53] niedbalski: ping (re logs?) [14:58] fwereade__: here's the fix if you're interested: http://reviews.vapour.ws/r/5384/ [14:58] natefinch: could use a review when you get 'round to it ^^ [15:09] katco: will do [15:09] rick_h_: what was it you needed me to review? [15:10] natefinch: /me looks at board [15:10] natefinch: oh, to fill in for OCR a bit since Michael is out today [15:10] natefinch: so I guess a quick run of things that might have come in overnight from the other folks [15:10] natefinch: fwereade__'s branches we're going to ask the migration folks to review [15:10] rick_h_: ok, sure. Wanted to make sure there wasn't something in particular [15:11] natefinch: but you're free to poke at it if you'd like to be a +2 on it [15:11] rick_h_: cool [15:11] natefinch: looks like katco's branch just hit the review lane [15:11] natefinch: so that would be <3 to help quickly turn that around [15:11] will do [15:20] katco, I have unhelpful comments on http://reviews.vapour.ws/r/5384/ I'm afraid [15:20] katco, well, hopefully not unhelpful [15:20] fwereade__: lol k tal [15:22]