[00:02] <mup> Bug #1610012 opened: can't migrate a model off a controller twice <migration> <juju-core:New for menno.smits> <https://launchpad.net/bugs/1610012>
[00:10] <anastasiamac> menn0: thumper: could we do a quick ho re:migration?
[00:11] <anastasiamac> m happy to talk to one of u, if getting time with both is difficult
[00:11]  * thumper is about to go walk the dog
[00:11] <thumper> wallyworld: blocker tag removed
[00:11] <menn0> anastasiamac: can it wait until later today please? i'm trying to get some of these bugs fixed
[00:11] <wallyworld> huzah
[00:13] <anastasiamac> menn0: of course \o/ ping when u have a chance
[00:13] <menn0> anastasiamac: will do
[00:18] <wallyworld> thumper: here's the muxtex bit that needs changing const prefix = "@/var/lib/juju/mutex-"
[00:19] <wallyworld> *mutex
[00:20] <wallyworld> bug 1604967
[00:20] <mup> Bug #1604967: Apparmor denies bind to abstract unix sockets such as @/var/lib/juju/mutex-/store-lock <snapd-interface> <Snappy:Triaged by jdstrand> <https://launchpad.net/bugs/1604967>
[00:20] <wallyworld> if we change the path, i don't think we'll need the apparmor change
[00:20] <wallyworld> will need to test
[00:31] <thumper> what abstract sockets does snappy apparmor allow?
[00:31] <thumper> wallyworld: ?
[00:31] <thumper> any?
[00:32] <thumper> wallyworld: looks like a bug on snappy not juju
[00:35] <wallyworld> thumper: not a snappy bug, but a profile issue which can be solved by using path that doesn'nt need an apparmor tweak
[00:35] <wallyworld> i'll test
[00:35] <wallyworld> we got advice from the snappy folds IIRC that's what we needed to do
[00:50] <menn0> wallyworld or thumper: http://reviews.vapour.ws/r/5378/
[00:50] <menn0> or anastasiamac or axw: ^^^ :)
[00:50] <axw> menn0: looking
[00:51] <axw> wallyworld: thanks for hitting merge. doesn't matter now, but I was going to merge the end of the pipeline since it includes all the other commits
[00:51] <wallyworld> oops
[00:51] <axw> wallyworld: all good :)  I'll do it with the other one
[00:56] <axw> menn0: LGTM
[00:56] <menn0> axw: thank you
[01:28] <axw> menn0: can you please review this trivial one: http://reviews.vapour.ws/r/5379/. I had added a CloudSpec API to provisioner, but it's not needed now after your changes to use environ-tracker
[01:31] <menn0> axw: looking
[01:33] <menn0> axw: done
[01:33] <axw> menn0: thanks
[02:07] <niedbalski> dooferlad, ping
[02:15] <niedbalski> dooferlad, dimitern https://bugs.launchpad.net/juju-core/+bug/1610037
[02:15] <mup> Bug #1610037: Juju2 beta14, missing network stanzas. <sts-needs-review> <juju-core:New> <https://launchpad.net/bugs/1610037>
[02:17] <mup> Bug #1610037 opened: Juju2 beta14, missing network stanzas. <sts-needs-review> <juju-core:New> <https://launchpad.net/bugs/1610037>
[02:28] <menn0> yes!
[02:29]  * menn0 quashes migration bugs
[02:30] <hatch> If I've found, what I believe to be a regression would you prefer I re-open an old bug or create a new one?
[02:31] <anastasiamac> niedbalski: dooferlad is on paternity leave :) dimitern will be online in about 4+ hrs \o/
[02:31] <anastasiamac> hatch: dpends on what bug u r planning to re-open..
[02:32] <hatch> anastasiamac: https://bugs.launchpad.net/juju-core/+bug/1566589
[02:32] <mup> Bug #1566589: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released by tycho-s> <https://launchpad.net/bugs/1566589>
[02:32] <hatch> There doesn't appear to be a way to specify a custom lxd network interface name
[02:32] <anastasiamac> hatch: re-open please \o/
[02:33] <hatch> even when setting the default profile to use something else it still tries to use lxcbro
[02:33] <hatch> allllrighty, I'm just running one last test here then I'll re-open
[02:33] <hatch> thanks
[02:34] <anastasiamac> hatch: please advice why u r re-opening in the bug comments :)
[02:34] <hatch> oh definitely - I'm also trying to use juju to bootstrap lxd's while running in an lxd itself
[02:35] <hatch> so it's a little funky network setup wise as it is
[02:35] <anastasiamac> hatch: \o/ the more info, the better to diagnose :)
[03:20] <mup> Bug #1566589 opened: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1566589>
[03:34] <thumper> axw: do both volumes and filesystems have settings?
[03:34] <thumper> axw: or was it just storage?
[03:35] <axw> thumper: neither. they both refer to "storage pools", and pools are defined in settings
[03:35] <thumper> axw: through settings on the state.Storage type?
[03:36] <thumper> hmm...
[03:36] <axw> thumper: storage/poolmanager takes a settings manager, which state implements
[03:37] <thumper> axw: I think what we really need to do is to get to the state where we have all the storage bits at least mostly handled, then create an environment where all these moving parts are created, then dump it and check the output
[03:37] <thumper> until then, I'm kinda guessing
[03:38] <axw> wallyworld thumper: have you seen this before? not sure if it's something I've caused... http://juju-ci.vapour.ws:8080/job/github-merge-juju/8636/artifact/artifacts/windows-out.log/*view*/
[03:39]  * thumper looks
[03:39] <wallyworld> axw: that's due to other recent work to remove supported ciphers i believe
[03:40] <thumper> axw: that is an intermittent failure
[03:40] <thumper> but all the extra log spam is new
[03:40] <axw> ok, thanks
[03:40] <natefinch> ug, did I leave log spam in?  Sorry.
[03:41] <natefinch> oh oh... I think I know what that is.. shit.  I edited the stdlib... I should probably put that bad
[03:41] <natefinch> back
[03:49] <menn0> wallyworld: do you have time to give thumper and I (and anyone else) a quick snappy intro?
[03:49] <menn0> wallyworld: I'm pretty much ready to send this test build out
[05:27] <wallyworld> menn0: sorry, missed ping, was out having 1:1 with anastasia
[05:28] <wallyworld> menn0: i need to do some work to tools upload (which i am doing now). if you can wait till monday....
[05:30] <menn0> wallyworld: all good... I figured out most of it myself
[05:30] <menn0> wallyworld: installing snapcraft from source was the hardest part
[05:31] <wallyworld> i had to do that too today - had to bring in some python deps
[05:31] <wallyworld> menn0: the tools work i am doing is do stuff "just works" for snaps without upload-tools
[05:32] <wallyworld> and also cleans stuff up a bit - there's a bit of cruft there, and i reckon i can see a suspect ofr that bootstrap version issue recently
[05:34] <menn0> wallyworld: ofr?
[05:35] <wallyworld> *for
[05:35] <wallyworld> you know i can't type :-)
[06:20] <wallyworld> axw: i added a few things, plus a bit of info on the other config work
[06:20] <wallyworld> i'll do another pass a bit later
[06:21] <axw> wallyworld: thanks. should I add in reacting to credential changes to in-scope work?
[06:22] <wallyworld> axw: i did that already, maybe needs a few more words
[06:22] <axw> wallyworld: ah I see. no that's fine
[06:23] <wallyworld> axw: i also added the update clouds worker
[06:48] <fwereade_> menn0, what PR did you want me to look at? can't seem to get to reviews.vapour.ws
[06:48] <fwereade_> menn0, and if you have a moment to look at https://github.com/juju/juju/pull/5932 before the weekend, that would be awesome
[06:49] <fwereade_> menn0, (also, I'm dumb, but I couldn't find the code that prevents us from migrating mid-charm-upgrade: can you point me to it?)
[06:50] <fwereade_> thumper, if you're around you too could address my last 2 messages :)
[07:04] <thumper> meetingology: we might not have it yet
[07:04] <meetingology> thumper: Error: "we" is not a valid command.
[07:04] <thumper> fwereade_: see above mse
[07:04] <thumper> msg
[07:04] <thumper> bah humbug
[09:09] <fwereade__> wallyworld, long shot: do you recall what the bits starting at about state/charm_test.go:250 were testing? am confused by the non-txn ops
[09:13] <wallyworld> fwereade__: not sure off hand without a bit of thought. i don't recognise the code, it looks like it was moved from somewhere else
[10:36] <mup> Bug #1610169 opened: invalid lxd config after update to 2.0-beta14-0ubuntu1~16.04.2~juju1 <juju-core:New> <https://launchpad.net/bugs/1610169>
[11:12] <babbageclunk> fwereade__, dimitern: a state.LinkLayerDevice can have multiple addresses, but a network.InterfaceInfo (and a params.NetworkConfig) only has one. Do s
[11:12] <babbageclunk> Oops
[11:13] <babbageclunk> Do you think I should produce multiple NetworkConfigs for a device with multiple addresses? Or pick one somehow?
[11:14] <dimitern> babbageclunk: multiple
[11:15] <babbageclunk> dimitern: Yeah, I was leaning towards that. Cool, thanks.
[11:15] <dimitern> babbageclunk: more verbose representation is used in network/ (at least for now)
[11:16] <dimitern> babbageclunk: there are tests (see networkingcommon) to convert between those IIRC
[11:23] <dimitern> babbageclunk: in case you're wondering why - network.InterfaceInfo came to be as a way to have the full config per NIC needed to render /e/n/i
[11:24] <babbageclunk> dimitern: Yeah, I guessed that from the fields :)
[11:25] <dimitern> :)
[11:37] <dimitern> hatch: unlikely, but are you around? re bug 1566589
[11:37] <mup> Bug #1566589: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>
[11:38] <dimitern> rick_h_: ^^ closed that one and unassigned you from it, FYI
[11:42] <mup> Bug #1566589 changed: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>
[11:46] <fwereade__> babbageclunk, I've just thought of a problem
[11:46] <babbageclunk> fwereade__: oh good
[11:47] <fwereade__> babbageclunk, we really shouldn't delete the machine document while it's got outstanding resources
[11:47] <fwereade__> babbageclunk, that might be the only thing keeping the model alive
[11:47] <fwereade__> babbageclunk, not the end of the world
[11:48] <fwereade__> babbageclunk, but it means that the provisioner should now be explicit about not *removing* machines, but on handing responsibility to something else once the instance has gone away and unblocked it
[11:48] <mup> Bug #1566589 opened: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>
[11:48] <fwereade__> babbageclunk, and then the "I've finished with machine X" messages can trigger the actual *remove*
[11:49] <fwereade__> babbageclunk, one upside there is that we never have parallel responsibility for the machine, so whatever's responsible can write status without worrying
[11:50] <fwereade__> babbageclunk, how badly have I wrecked your day..?
[11:51] <babbageclunk> fwereade__: So we end up splitting machine.Remove into two parts - one that does everything except removing addresses, link layer devices and the actual remove, and the other part that really removes everything.
[11:51] <rick_h_> dimitern: ty
[11:52] <fwereade__> babbageclunk, well, I'm wondering if it's more like `Provisioner.MarkForGC([machine-3-lxd-3])` or whatever name we pick
[11:52] <fwereade__> babbageclunk, where that txn checks the machine is dead and creates the removal doc for the attention of the watcher
[11:52] <babbageclunk> fwereade__: like, moderately? :) I just got finished doing it the other way. Haven't had a chance to think through the implications yet - might not actually be much, except that I probably no longer need to move various methods off Machine if we'll still have one at the end.
[11:53] <babbageclunk> fwereade__: yeah, ok - that makes sense.
[11:53] <fwereade__> babbageclunk, it definitely hits the fiddly-txn-ops side :( but I think the removals+watch remain the same?
[11:54] <babbageclunk> fwereade__: yeah, that part is the same. And it's cleaner this way.
[11:55] <babbageclunk> fwereade__: ok, thanks for the headsup - I'll start switching over to that after lunch, should get something up for review soonish.
[11:56] <fwereade__> babbageclunk, just added some replies on http://reviews.vapour.ws/r/5366/ fwiw
[11:56] <fwereade__> babbageclunk, thanks :)
[11:57] <mup> Bug #1566589 changed: ERROR cannot find network interface "lxcbr0": route ip+net: no such network interface <juju-core:Fix Released> <https://launchpad.net/bugs/1566589>
[12:01] <babbageclunk> fwereade__: those review comments are enlightening, thanks!
[12:01]  * babbageclunk lunches
[12:01] <fwereade__> does anyone know why we might rewrite the revisions of charmstore charms "to handle the revision differences between unpublished and published charms in the store"?
[12:01] <fwereade__> rogpeppe, rick_h_ ^^
[12:02] <fwereade__> babbageclunk, cool, yw :D
[12:02] <rogpeppe> fwereade__: what's the context?
[12:02] <fwereade__> rogpeppe, migrations
[12:03] <rogpeppe> fwereade__: model migration?
[12:03] <fwereade__> rogpeppe, apparently we might have charm archives with revisions that don't match their url, and that worries me
[12:03] <fwereade__> rogpeppe, yeah
[12:03] <fwereade__> rogpeppe, and the comment mentions the charm store -- and that's all I know :)
[12:03] <rogpeppe> fwereade__: where does that remark come from?
[12:04] <fwereade__> rogpeppe, apiserver/charms.go:262
[12:04] <fwereade__> rogpeppe, I think menn0's been gone for a while, it's nbd, I will mail him if it doesn't resonate with you
[12:05] <rogpeppe> fwereade__: i'm looking at the code. gimme a minute or two.
[12:05] <fwereade__> rogpeppe, cheers :)
[12:06] <rogpeppe> fwereade__: i don't think revisions in charm archives are a thing any more
[12:06] <rogpeppe> fwereade__: i don't think the code should ever be looking at them
[12:06] <rogpeppe> fwereade__: it was always a bad idea
[12:06] <fwereade__> rogpeppe, strongly agree
[12:08] <fwereade__> rogpeppe, ok, I had also sorta thought that; I will continue to poke away at it in that light
[12:11] <rogpeppe> fwereade__: it looks like api.Client.UploadCharm always attaches a revision
[12:11] <rogpeppe> fwereade__: so that logic is pretty much irrelevant AFAICS
[12:11] <fwereade__> rogpeppe, well... that would explain it, I suppose
[12:12] <rogpeppe> fwereade__: i'd suggest changing it so that it fails if there's no revision form field specified
[12:12] <fwereade__> rogpeppe, yeah, that sounds sane to me
[12:12] <rogpeppe> fwereade__: given that the charm package is still "unstable", i'd really like to remove the whole notion of revision from charm.Charm
[12:13] <fwereade__> rogpeppe, +1eMAXINT
[12:13] <rogpeppe> fwereade__: :)
[12:13] <rogpeppe> fwereade__: it shouldn't be hard to change in the charm package itself :)
[12:15] <fwereade__> rogpeppe, I'm pretty sure it'd be worth it despite the downstream pain
[12:15] <rogpeppe> fwereade__: agreed
[12:15] <rogpeppe> fwereade__: feel free to go for it
[12:18] <fwereade__> ...you know, we really *don't* use .Revision() very much at all
[12:18] <fwereade__> I can certainly excise its use from juju/juju
[12:20] <fwereade__> rogpeppe, do you have a picture of how it'd impact other dependencies? how much stuff would I plunge into unbuildable catastrophe if core were suddenly using a new Charm interface?
[12:22] <rogpeppe> fwereade__: the only real impact would be on the semantics of local repos
[12:23] <rogpeppe> fwereade__: but we don't support local repos like that any more anyway really
[12:23] <rogpeppe> fwereade__: and i bet no-one relies on the current semantics, which are bizarre
[12:23] <fwereade__> rogpeppe, yeah
[12:23] <fwereade__> rogpeppe, what about name?
[12:24] <rogpeppe> fwereade__: i'd love to get rid of Name too
[12:24] <rogpeppe> fwereade__: the charm store never uses it, or Revision
[12:24] <fwereade__> rogpeppe, ...and unless we *actually get rid of* Name+Revision we have this opportunity for mismatch everywhere we go
[12:24] <rogpeppe> fwereade__: exactly
[12:25] <rogpeppe> fwereade__: putting the name and revision in the content is a silly idea
[12:25] <fwereade__> rogpeppe, no argument here
[12:25] <rogpeppe> fwereade__: i wish we'd persuaded gustavo back in the day...
[12:25] <fwereade__> :)
[12:28] <anastasiamac> dimitern: thank you for looking at this lxdbr0 bug earlier \o/ do u think that this is related? https://bugs.launchpad.net/juju-core/+bug/1610169
[12:28] <mup> Bug #1610169: invalid lxd config after update to 2.0-beta14-0ubuntu1~16.04.2~juju1 <juju-core:New> <https://launchpad.net/bugs/1610169>
[12:31]  * dimitern managed wade through maas'es bind9 config and fix streams.canonical.com to 127.0.0.1 \o/  
[13:16] <sinzui> rick_h: bug 1610243 is a blocker. The azure provider is broken. Juju cannot bootstrap
[13:16] <mup> Bug #1610243: Azure provider storage account not found <azure-provider> <blocker> <bootstrap> <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610243>
[13:17] <rick_h_> sinzui: rgr looking
[13:18] <mup> Bug #1610238 opened: UnitSuite.TestWithDeadUnit timed out waiting for agent to finish <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1610238>
[13:18] <mup> Bug #1610239 opened: Race in src/gopkg.in/mgo.v2 <ci> <intermittent-failure> <race-condition> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1610239>
[13:18] <mup> Bug #1610243 opened: Azure provider storage account not found <azure-provider> <blocker> <bootstrap> <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610243>
[13:20] <tvansteenburgh> anyone know what would cause only 2 facades to be reported (UserManager and ModelManager) when logging in to the api?
[13:20] <tvansteenburgh> (juju2)
[13:28] <tvansteenburgh> well actually i'm getting more than 2, but the Client facade is missing and i'm not sure why that would be the case
[13:36] <mup> Bug #1609994 changed: Race in github.com/juju/loggo global <ci> <race-condition> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1609994>
[13:37] <tvansteenburgh> Login response: {'server-version': '2.0-beta12', 'facades': [{'name': 'AllModelWatcher', 'versions': [2]}, {'name': 'Cloud', 'versions': [1]}, {'name': 'Controller', 'versions': [3]}, {'name': 'MigrationTarget', 'versions': [1]}, {'name': 'ModelManager', 'versions': [2]}, {'name': 'UserManager', 'versions': [1]}], 'server-tag': 'model-d63ff9c4-3d46-464b-8c98-9459afbef958', 'servers': [[{'type': 'ipv4', 'scope': 'local-cloud', 'por
[13:38] <tvansteenburgh> hmm, do i need to login to a specific model endpoint to get the Client facade perhaps?
[13:44] <natefinch> tvansteenburgh: yeah, I think that's a new thing - they split off the controller API from the model API
[13:45] <mup> Bug #1609994 opened: Race in github.com/juju/loggo global <ci> <race-condition> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1609994>
[13:48] <mup> Bug #1609994 changed: Race in github.com/juju/loggo global <ci> <race-condition> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1609994>
[13:48] <mup> Bug #1610254 opened: model-migration: Mongo db is not in an expected state <ci> <model-migration> <mongodb> <juju-core:Triaged> <https://launchpad.net/bugs/1610254>
[13:48] <mup> Bug #1610255 opened: Cannot start bootstrap instance: DB is locked <bootstrap> <ci> <lxd-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610255>
[13:54] <mup> Bug #1610254 changed: model-migration: Mongo db is not in an expected state <ci> <model-migration> <mongodb> <juju-core:Triaged> <https://launchpad.net/bugs/1610254>
[13:54] <mup> Bug #1610255 changed: Cannot start bootstrap instance: DB is locked <bootstrap> <ci> <lxd-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610255>
[14:01] <rick_h_> natefinch: standup ping take 4
[14:06] <mup> Bug #1610254 opened: model-migration: Mongo db is not in an expected state <ci> <model-migration> <mongodb> <juju-core:Triaged> <https://launchpad.net/bugs/1610254>
[14:06] <mup> Bug #1610255 opened: Cannot start bootstrap instance: DB is locked <bootstrap> <ci> <lxd-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1610255>
[14:06] <mup> Bug #1610260 opened: AWS Error fetching security groups EOF/timeout <bootstrap> <ec2-provider> <intermittent-failure> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1610260>
[14:11] <niedbalski> dimitern, ping
[14:13] <niedbalski> dimitern, re: 1610037, check: https://pastebin.canonical.com/162396/
[14:15] <dimitern> niedbalski: otp, will get back to you shortly
[14:22] <dimitern> niedbalski: I might be missing something, but I still can't see /e/n/i from the container in your paste
[14:22] <dimitern> niedbalski: ah, sorry - line 287
[14:26] <dimitern> niedbalski: there do those lines come from, e.g.: 'post-up ifup bond0:1' ?
[14:28] <niedbalski> dimitern, that's the host deployed by maas.
[14:29] <dimitern> niedbalski: so maas/curtin rendered that?
[14:29] <niedbalski> dimitern, yep; here is the config on the maas-ui, fyi: http://pasteboard.co/4O4UwMdWX.png
[14:29] <hatch> dimitern: thanks for the response on the bug. I noticed the beta14 email quite late and was going to confirm it in the morning. You beat me to it :)
[14:30] <dimitern> niedbalski: can you also paste /var/log/cloud-init-output.log and /var/log/juju/machine-0.log please?
[14:31] <dimitern> hatch: ;) no worries
[14:31] <hatch> dimitern: this weekend I'll resume my testing on running Juju in an LXD where the nested lxd's also get an ip on the network
[14:31]  * hatch crosses fingers
[14:32] <niedbalski> dimitern, sure.
[14:32] <dimitern> hatch: is this on maas?
[14:32] <dimitern> hatch: or lxd provider?
[14:32] <hatch> dimitern: nope, just a raw Xenial box: Metal > LXD > Juju > LXD
[14:33] <hatch> I had the both LXD's in that diagram getting the ip's but Juju wouldn't use them
[14:33] <hatch> so hoping it will now
[14:33] <dimitern> hatch: I *think* you might be out of luck, but if you give me a few more details about what you want to test I can perhaps save you some time if it won't ever work ;)
[14:34] <dimitern> hazmat: so a plain xenial, you ssh into it, apt install juju-2, bootstrap lxd lxd, and then what?
[14:34] <dimitern> sorry hazmat ;) hatch: ^^
[14:35] <hatch> dimitern: so I've supplied a bridge to the outer LXD's so that they get a real IP following jrwren's blog post. That all worked well. Then inside that LXD I created yet another bridge pointing to the parent containers lxd and then IT was receiving ip's from DHCP
[14:35] <hatch> however jujud kept using the 10. ip instead of the br0 ip
[14:36] <dimitern> hatch: so now you can customize which bridge to use for lxd provider before bootstrapping, but changing the one on the default LXD profile (i.e. apt install lxd, configure, then bootstrap)
[14:37] <dimitern> hatch: once bootstrapped though, juju will insist on rendering a default /etc/default/lxd-bridge with lxdbr0 using another subnet than the one the controller container is on
[14:38] <natefinch> rick_h_: fwereade__ was a big help in finding an way forward that we can do incrementally.  Basically we'll tack jsonschema on the side of what we already have in environschema, and we can incrementally move things over to use that, rather than needing to do a big bang change.
[14:38] <rick_h_> natefinch: <3 ty fwereade__ and sounds like a solid pla
[14:38] <rick_h_> plan
[14:38] <dimitern> hatch: to make it work, you'll need to manually go to each nested container and change the bridge config
[14:39] <hatch> dimitern: that's terrible :)
[14:39] <hatch> dimitern: so my goal is to run Juju in an LXD and have each LXD it creates to get a real IP on the network so that I can actually access them
[14:39] <hatch> am I out of luck?
[14:40] <dimitern> hatch: you can do some iptables magic on the controller LXD container to let that happen
[14:40] <rick_h_> hatch: so you need juju to support remote lxd servers so juju can be in a lxd and talk to other lxd on the root machine but that's not there atm
[14:40] <hatch> rick_h_: yeah that was my first idea, which, as you just mentioned doesn't work :D
[14:40] <babbageclunk> fwereade__: Is it legit to run multiple machine.Remove() transactions in response to a single CompleteMachineRemoval(ids) call, or should I build up a big transaction from each of the machines remove ops?
[14:41] <hatch> dimitern: I had it working using `redir` but that was manual and painful :D
[14:41] <babbageclunk> fwereade__: s/from each/with each/
[14:41] <hatch> I was hoping for an easy DHCP ip :)
[14:41] <dimitern> hatch: assuming the above setup, on machine-0 (LXD controller container) you'll have 1 IP from the bridge you configured on the host machine (e.g. 10.42.0.12)
[14:43] <dimitern> hatch: then on that same machine, if you enable IP forwarding (sudo sysctl -w net.ipv4.ip_forward=1 && echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf), and add this rule:
[14:45] <dimitern> hatch: sudo iptables -t nat -A POSTROUTING -s 10.33.0.0/24 ! -d 10.33.0.0/42 -j MASQUERADE
[14:46] <dimitern> hatch: 10.33.0.0/24 is the LXD network inside the controller LXD container, while 10.42.0.0/24 is the one on its host
[14:46] <hatch> interesting
[14:47] <dimitern> hatch: this will allow hosted containers on the controller LXD to access outside with NAT, without being on the same network as the controller
[14:47] <dimitern> also - no need to touch anything else on each nested LXD machine
[14:48] <hatch> dimitern: so how would I access those lxd's from my desktop?
[14:48] <dimitern> hatch: but I'm trying it now to double check I'm not misaken
[14:48] <dimitern> hatch: ah :)
[14:48] <hatch> dimitern: using what I currently have I can use redir on the host lxd pointing to the child lxd and it 'works'
[14:49] <hatch> no fancy iptables stuff necessary
[14:49] <hatch> but that's a manual step
[14:49] <hatch> IF I could get the nested lxd's a dhcp ip I'd be golden
[14:49] <hatch> which, I can do, but Juju doesn't use it :)
[14:49] <hatch> it uses the internal 10. ip which it can't access after I break it ;)
[14:49] <dimitern> hatch: you can only get dhcp for containers on the lxd provider anyway
[14:50] <dimitern> only on maas the ux is better atm
[14:51] <hatch> dimitern: ok so what I'll do over the weekend is outline in detail what I'm trying to do and the steps I've taken and where I'm hitting roadblocks and then maybe we can work towards making the experience better so that this workflow is a Juju reality
[14:51] <dimitern> hatch: that'd be great! thanks, I'll be happy to try to replicate your setup and chat about improving the UX!
[14:52] <hatch> great thanks
[14:52] <dimitern> np ;)
[14:53] <dimitern> niedbalski: ping (re logs?)
[14:58] <katco> fwereade__: here's the fix if you're interested: http://reviews.vapour.ws/r/5384/
[14:58] <katco> natefinch: could use a review when you get 'round to it ^^
[15:09] <natefinch> katco: will do
[15:09] <natefinch> rick_h_: what was it you needed me to review?
[15:10] <rick_h_> natefinch: /me looks at board
[15:10] <rick_h_> natefinch: oh, to fill in for OCR a bit since Michael is out today
[15:10] <rick_h_> natefinch: so I guess a quick run of things that might have come in overnight from the other folks
[15:10] <rick_h_> natefinch: fwereade__'s branches we're going to ask the migration folks to review
[15:10] <natefinch> rick_h_: ok, sure. Wanted to make sure there wasn't something in particular
[15:11] <rick_h_> natefinch: but you're free to poke at it if you'd like to be a +2 on it
[15:11] <natefinch> rick_h_: cool
[15:11] <rick_h_> natefinch: looks like katco's branch just hit the review lane
[15:11] <rick_h_> natefinch: so that would be <3 to help quickly turn that around
[15:11] <natefinch> will do
[15:20] <fwereade__> katco, I have unhelpful comments on http://reviews.vapour.ws/r/5384/ I'm afraid
[15:20] <fwereade__> katco, well, hopefully not unhelpful
[15:20] <katco> fwereade__: lol k tal
[15:22] <fwereade__> katco, I think the bad scenario is where we do have a misbehaving cloud, and *every* machine we try to provision goes through the same backoff process, uninterruptibly, and after N minutes reports its status and has the provisioner move on to start the process with the next
[15:22] <katco> fwereade__: yeah that is problematic. really not sure if i should try and tackle that here, or in a follow-up
[15:23] <katco> fwereade__: i mean i guess in a sense this would be "breaking" things by fixing things
[15:23] <fwereade__> katco, yeah, this is a fragile-ecosystem sort of concern
[15:24] <katco> fwereade__: if i were to set status on every retry (as we discussed, sorry), is that enough? or would the user be just as frustrated...
[15:25] <fwereade__> katco, that is a good mitigation, for sure -- can we extract timings? if we say when we will, and do what we say, that is at least *something*
[15:25] <fwereade__> katco, even if juju is not doing the smartest thing, you can see what it *is* doing
[15:25] <katco> fwereade__: yeah, they're defined explicitly in i think provisioner.go
[15:25] <fwereade__> katco, which is a lot better than a silent magic box
[15:26] <fwereade__> katco, yeah that sounds right
[15:26] <katco> fwereade__: so, do that, land it, and then take a more comprehensive look at retries, and specifically how scheduling works here?
[15:26] <fwereade__> katco, what's the total delay per machine that we'll actually have before we move on?
[15:27] <katco> fwereade__: 3 retries of 10 seconds, so 30s
[15:27] <fwereade__> katco, ah, that's not so bad
[15:28] <fwereade__> katco, mm, I wonder if it's be easy to make jupsen cut off access to provider endpoints... mattyw?
[15:30] <mattyw> fwereade__, not currently, but it could be added quite easily I think
[15:31] <mattyw> fwereade__, you can cut off the controller from all traffic, but that's not advisable
[15:35] <fwereade__> katco, so, ok, I think I am comfortable that what we currently have is a sane and conservative strategy, but (1) the drawbacks, and the havoc that will be wreaked if we tweak the strategy, *must* be documented in shades of terror and woe; and (2) I think it would be a good idea to cut off provider access and tell the controller to deploy a bunch of machines, and see how it actually feels in practice
[15:36] <katco> fwereade__: ok, i'll have to figure out how to do that
[15:36] <fwereade__> katco, do take a look at jupsen
[15:37] <fwereade__> katco, I bet you could hack it quickly and situationally with ease
[15:37] <katco> fwereade__: i.e. implement it in jupsen?
[15:37] <fwereade__> katco, it's just the first thing that springs to mind for inducing netsplits in juju
[15:38] <fwereade__> katco, I barely remember the internals though so I'm not sure
[15:42] <katco> fwereade__: responded to a few of your comments; require feedback
[16:04] <natefinch> katco: I'm reviewing your branch now BTW
[16:05] <katco> natefinch: cool
[16:15] <natefinch> katco: do we not care about the task dying in the middle anymore?  We used to check task.catacomb.Dying()
[16:17] <katco> natefinch: oh jees... i had that in there, but i've refactored this thing like 3 times. it must have fallen out. the select statement should still be in there
[16:18] <natefinch> katco: heh, yep, I've done the same thing.
[16:19] <natefinch> I wish that kind of thing were easier to test for.... there should be a test that kills it in the middle and ensures it behaves correctly... but that's really tricky.
[16:19] <katco> natefinch: if the code were structured to be unit testable, it wouldn't be so hard
[16:20] <katco> natefinch: but as it sits, yeah. it would be some kind of weirdly shaped full-stack test
[16:23] <dimitern> fwereade__: are you still there btw?
[16:28] <fwereade__> dimitern, o/, sort of
[16:28] <fwereade__> katco, I think I've fed back
[16:28] <katco> fwereade__: ta
[16:30] <natefinch> katco: published some comments
[16:30] <katco> natefinch: ta
[16:31] <fwereade__> katco, natefinch: IIRC menn0? was just doing some work to extract the environ dependency which is one of the big ones that drags heaps of sludge with it
[16:31] <dimitern> fwereade__: am I right to think most workers expect apicaller resource, which in turn depends on the upgrade gate to be lifted?
[16:31] <fwereade__> dimitern, ummmmmmm I forget the exact upgrade dance
[16:31] <fwereade__> dimitern, there's a couple of layers remember
[16:31] <fwereade__> dimitern, we need to complete state upgrades before letting the apiserver at it
[16:32] <dimitern> fwereade__: I'm looking at bug 1607749 and so far it seems we have a catch 22 where the proxyupdater can't start because the upgrader haven't opened the gate yet, but it NEEDS a proxy to do that
[16:32] <mup> Bug #1607749: juju bootstrap fails with MAAS trunk and juju (beta12 behind a proxy too) <bootstrap> <maas-provider> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1607749>
[16:32] <fwereade__> dimitern, but then most other upgrades will be happening through the api
[16:35] <fwereade__> dimitern, I don't understand what's triggering an upgrade there
[16:37] <mup> Bug #1610319 opened: Juju fails to download charm <juju-core:New> <https://launchpad.net/bugs/1610319>
[16:37] <fwereade__> dimitern, if you upgrade juju and change proxy settings at the same time I can see how that could happen
[16:37] <dimitern> fwereade__: what I think happens is this: 1) juju client cannot login due to "upgrade in progress" (ok and expected), 2) upgrader starts first, before even proxyupdater (bad), 3) upgrader tries to hit streams.c.c, fails (with set http_proxy= in the exec-start.sh job for jujud-machine-0 works!), 4) proxyupdater stops with dependency "apicaller" not present
[16:37]  * perrito666 reads a test that breaks so many principles of good testing that makes him cry
[16:38] <dimitern> fwereade__: this is during bootstrap
[16:38] <fwereade__> dimitern, why are we bootstrapping with tools that immediately want to replace themselves?
[16:40] <dimitern> fwereade__: now that's interesting - I'm not passing --upload-tools, but building from source (tag juju-2.0-beta12)
[16:40] <mup> Bug #1610319 changed: Juju fails to download charm <juju-core:New> <https://launchpad.net/bugs/1610319>
[16:40] <dimitern> fwereade__: and I can see in the logs current version is juju-2.0-beta12.1, want version juju-2.0-beta12
[16:41] <fwereade__> dimitern, I *think* the upgrade-on-bootstrap is the problem
[16:42] <fwereade__> dimitern, *but* I would be super supportive of something that cached proxy settings in agent config and set them up as early as possible
[16:42] <fwereade__> dimitern, leave the updating to the worker, and make it update the cache too
[16:43] <dimitern> fwereade__: --auto-upgrade is not passed to juju bootstrap, so I'm not sure why the upgrade is triggered
[16:43] <fwereade__> dimitern, I bet there's some subtle version mismatch
[16:43] <dimitern> fwereade__: what you suggest makes perfect sense though
[16:44] <fwereade__> dimitern, honestly they should probably go into agent config at cloudconfig time
[16:44] <dimitern> fwereade__: yeah
[16:44] <fwereade__> dimitern, I don't like putting things in agent config unless I *have* to, but this makes a better case than most
[16:46] <mup> Bug #1610319 opened: Juju fails to download charm <juju-core:New> <https://launchpad.net/bugs/1610319>
[16:48] <dimitern> well beta15 apparently works better than beta12 in that same scenario :/
[17:16] <rick_h_> katco: ping, assigned a bug your way that might be fixed with your last chunk of work
[17:16] <katco> rick_h_: cool, ta
[17:16] <rick_h_> katco: can you see if you can replicate with the given steps and trunk and if so it sounds like it might be related to the revision issue you were chasing
[17:16] <katco> rick_h_: k
[17:16]  * rick_h_ hopes it's an easy "fix already comitted" :)
[17:16] <katco> that would be nice
[17:18]  * rick_h_ goes to grab lunchables
[17:49] <natefinch> sinzui: reviewboard is borken... getting 500s who's in charge of that now?  G
[17:52] <sinzui> natefinch: no one
[17:52] <sinzui> natefinch: I can take a look
[17:53] <sinzui> natefinch: is there a specific url that gives a 500?
[17:53]  * sinzui visits app
[17:53] <natefinch> sinzui: looks like any specific review url
[17:53] <natefinch> sinzui: or.. not
[17:53] <sinzui> natefinch: I am not seeing errors
[17:53] <natefinch> sinzui: http://reviews.vapour.ws/r/5355/
[17:54] <sinzui> natefinch: ty
[17:54] <sinzui> just that one url, the neighbors are fine
[18:01] <sinzui> natefinch: there are seveal errors like this in the log https://pastebin.canonical.com/162540/ all are for /r/5355/. I see me, you and axw.
[18:02] <natefinch> lol python
[18:02] <sinzui> natefinch: I think the data for this review is bad. I will see what I can do
[18:03] <natefinch> sinzui: there's a way to delete it and recreate it... but I think it requires admin rights to really really delete
[18:03] <sinzui> natefinch: yeah. I am hunting for that power, or a command line util on the host
[18:07] <katco> sinzui: natefinch: ericsnow still works for canonical, just saying :)
[18:07] <natefinch> katco: lol good point
[18:08] <katco> i don't think it's *that* much of an imposition to ask for instructions 1x. but we should write them down
[18:15] <alexisb> perrito666, ping
[18:15] <perrito666> alexisb: pong
[18:15] <sinzui> natefinch: do you know who created it? I have a list of reviews, but no id to match it to the url
[18:16] <natefinch> sinzui: andrew
[18:17]  * natefinch is pinging ericsnow
[18:18] <ericsnow> natefinch: what's up?
[18:19] <katco> ericsnow: hey! o/
[18:19] <perrito666> wow, instant summoning
[18:19] <ericsnow> :)
[18:19] <katco> wwitzel3: summon? for old times sake?
[18:19] <natefinch> ericsnow: I think we need to perma-delete a review so it can be recreated
[18:19]  * perrito666 burns incense in a monument for ericsnow to pray for the fixing of review board
[18:20] <natefinch> ericsnow: but I'm not sure any of us are admins, and IIRC one needs to be an admin to do it
[18:20] <natefinch> ericsnow: start of the story, this review returns a 500: http://reviews.vapour.ws/r/5355/
[18:20] <ericsnow> natefinch: yep
[18:21] <ericsnow> natefinch: looks like katco still is an admin
[18:21] <natefinch> katco is the new ericsnow.  It is known.
[18:21] <katco> i quit
[18:21] <natefinch> lol
[18:21] <katco> good working with you folks
[18:22] <perrito666> please make me admin too, I know my way around django
[18:22] <perrito666> and that gives us some redundancy
[18:22] <natefinch> perrito666 is the new ericsnow
[18:22] <ericsnow> (also admin: alexisb, cmars, frobware, fwereade__, thumper, jam, sinzui, and wallyworld)
[18:23] <katco> rick_h_: https://bugs.launchpad.net/juju-core/+bug/1610319/comments/5
[18:23] <mup> Bug #1610319: Juju fails to download charm <juju-core:Triaged by cox-katherine-e> <https://launchpad.net/bugs/1610319>
[18:24] <alexisb> ericsnow, what are we admin to?
[18:24] <natefinch> ericsnow: nice.  Good.  Sorry, I didn't realize.  That's a good list.  I agree with putting perrito666 on there... this really should be dev's job to maintain.
[18:24] <natefinch> alexisb: reviewboard
[18:25] <alexisb> aaah yeah ok
[18:25] <ericsnow> alexisb: reviewboard (click on your username and you'll see an "Admin" link)
[18:25]  * katco goes for lunch
[18:25] <ericsnow> natefinch: all good now?
[18:26] <natefinch> ericsnow: is there a trick to permadeleting a review?  I don't have the admin UI, so I don't know how to do it.  Does the author have to delete it first?
[18:27] <ericsnow> natefinch: it's a third option in the "Close" menu: "Delete Permanently"
[18:27] <rick_h_>   katco <3 ty much
[18:27] <natefinch> ericsnow: cool, thanks.
[18:27] <ericsnow> natefinch: take care!
[18:27] <natefinch> ericsnow: I release you from the summons.  Thanks for the help.
[18:29]  * natefinch makes a page in the wiki for reviewboard tips
[18:29] <alexisb> thank you natefinch
[18:32] <fwereade__> that was a truly spectacular CL, by the way; I think it's the only time RB's been broken by sheer awesomeness
[18:33] <alexisb> "broken by sheer awesomeness"
[18:33] <alexisb> :)
[18:35] <perrito666> I would love to open that issue in rb
[18:35] <natefinch> perrito666: are you an admin on rb now?
[18:35] <perrito666> "rb breaks when excessive awesomeness present in patch"
[18:36] <perrito666> natefinch: checking
[18:37] <perrito666> natefinch: nope, still a mere mortal
[18:40] <mup> Bug #1604955 changed: TestUpdateStatusTicker can fail with timeout <ci> <intermittent-failure> <test-failure> <juju-core:Fix Released by fwereade> <https://launchpad.net/bugs/1604955>
[18:40] <mup> Bug #1608105 changed: LXD no longer activates all interfaces on initial deploy when using MAAS2rc3 and JUJU Beta13 <juju-core:Fix Released> <https://launchpad.net/bugs/1608105>
[18:41] <rick_h_> katco: can you add a link to your PR in your aws storage card please?
[18:46] <natefinch> alexisb, sinzui: can one of you make perrito666 and me admins on reviewboard?
[18:46] <natefinch> I don't think there's any reason not to have a ton of admins
[18:47] <alexisb> natefinch, yep one se
[18:47] <alexisb> c
[18:47] <sinzui> natefinch: Fix (-ish) rb fellover parsing the first change set. I unlinked it http://reviews.vapour.ws/r/5355/
[18:48] <sinzui> natefinch: I think I can
[18:48] <alexisb> sinzui, I got it
[18:48] <alexisb> perrito666, you should have all powers now
[18:48] <alexisb> natefinch, doing yours next
[18:48] <natefinch> alexisb: cool, thanks
[18:49] <alexisb> ok natefinch make sure you are all powerful
[18:50] <sinzui> natefinch: My suspision that a comment couldn't be parsed was wrong. I am surprised that the changesets are parse as markdown.
[18:50] <natefinch> sinzui: lol, it parses the PR/commit messages as markdown?  that's horrible
[18:50] <natefinch> sinzui: I mean, it still shouldn't *break*
[18:51] <sinzui> natefinch: I think it parsed the diff as markdown!
[18:51] <natefinch> lol
[18:51] <sinzui> natefinch: the comments were fine.
[18:52] <sinzui> I understand the schema now so I wont be slow the next time this happens
[18:52] <natefinch> well, good
[18:53] <natefinch> alexisb: thanks
[18:54] <perrito666> natefinch: alexisb http://stream1.gifsoup.com/view/572272/he-man-o.gif
[18:56] <alexisb> perrito666, I loved he-man growing up, I so wanted to be she-ra
[18:56] <natefinch> nice nice... yeah, he-man was one of the few licensed toys I had growing up.  I think my mother still has some of them in the attic somewhere.
[18:58] <natefinch> alexisb: are you an admin on https://github.com/juju/environschema/ ?  I need a v2 branch there
[18:58] <rick_h_> natefinch: I can help with that
[18:59] <natefinch> rick_h_: thanks
[18:59] <rick_h_> natefinch: https://github.com/juju/environschema/tree/v2 when you're sure it's stable
[18:59] <rick_h_> natefinch: let's make sure to go back and make it the default branch of the project
[19:00] <natefinch> rick_h_: yep
[19:00] <rick_h_> and with that I'm going to run to get the boy, have a good weekend folks
[19:00] <natefinch> rick_h_: see ya
[19:09] <perrito666> yeah, he man was strategically broadcast at the time kids get out of school and get together to drink tea with cookies
[19:41] <mup> Bug #1610319 changed: Juju fails to download charm <juju-core:Fix Committed by cox-katherine-e> <https://launchpad.net/bugs/1610319>
[19:41] <mup> Bug #1610397 opened: juju2, maas2, cloud deployment failure when two domains are used. <kanban-cross-team> <landscape> <juju-core:Triaged by rharding> <MAAS:New> <nova-cloud-controller (Juju Charms Collection):New> <https://launchpad.net/bugs/1610397>
[19:44] <mup> Bug #1610397 changed: juju2, maas2, cloud deployment failure when two domains are used. <kanban-cross-team> <landscape> <juju-core:Triaged by rharding> <MAAS:New> <nova-cloud-controller (Juju Charms Collection):New> <https://launchpad.net/bugs/1610397>
[19:46] <natefinch> perrito666: I need another brain
[19:49] <perrito666> natefinch: have you tried amazon?
[19:50] <mup> Bug #1610397 opened: juju2, maas2, cloud deployment failure when two domains are used. <kanban-cross-team> <landscape> <juju-core:Triaged by rharding> <MAAS:New> <nova-cloud-controller (Juju Charms Collection):New> <https://launchpad.net/bugs/1610397>
[19:51] <natefinch> perrito666: got time for a hangout?
[19:51] <perrito666> natefinch: sure, gimme a sec
[19:53] <natefinch> perrito666: when you're ready: gah, dammit
[19:53] <natefinch> lol, wrong copy and paste
[19:53] <natefinch> https://hangouts.google.com/hangouts/_/canonical.com/moonstone?authuser=1
[20:53] <mbruzek> Can I get someone to triage this bug: https://bugs.launchpad.net/juju-core/+bug/1609893 I know it is not a high priority probem but I am able to reproduce it again today.
[20:53] <mup> Bug #1609893: juju status returns ERROR not logged in <usability> <juju-core:Triaged> <https://launchpad.net/bugs/1609893>
[21:29] <mup> Bug #1610450 opened: feature request: suspend models <juju-core:New> <https://launchpad.net/bugs/1610450>
[22:40] <perrito666> aghh really blocked again?
[22:41] <perrito666> alexisb: is it really that blocking?