[00:13] wallyworld, axw: review please? https://github.com/juju/juju/pull/7010 [00:14] ok [00:16] wallyworld: thanks - ended up being a bit stupid [00:16] a one liner [00:17] babbageclunk: lgtm [00:17] wallyworld: yup - the better fix is lots of lines across 3 packages [00:18] wallyworld: thanks [00:18] for 2.1 this is fine IMO [00:30] babbageclunk: is that change safe for concurrent AddCharmWithAuthorization calls? will two calls to repo.Get end up getting the same bundle, and one will remove it from underneath the other? [01:19] wallyworld, axw: during cloud-init, the proxy settings are only used to write out to ~ubuntu/.juju-proxy [01:19] menn0: yeah, that's what i feared [01:19] wallyworld, axw: which is dumb anyway [01:19] menn0: no, we do also set them for the cloud-init scripts [01:19] wallyworld, axw: the proxy settings aren't used in any other way [01:19] menn0: see `exportedProxyEnv := w.icfg.ProxySettings.AsScriptEnvironment()` [01:20] menn0: in cloudconfig/userdata_unix.go [01:20] err, userdatacfg_unix.go [01:20] axw: you're right. I missed that the proxy settings were executed [01:22] axw, wallyworld: so the one thing we should fix is that the proxy settings should probably be written out to the global profile (which affects all users) instead of just the ubuntu user [01:22] axw, wallyworld: do you agree? [01:22] menn0: seems reasonable [01:22] i think so [01:22] i wonder why we restricted to ~ubuntu in the first place [01:22] axw, wallyworld: which is fine except maybe for the manual provider [01:23] axw: ugh, you're right. Ok, I'll have to go for the more invasive way. [01:23] axw: thanks though! Luckily the merge failed. [01:24] menn0: if you manually provision, you're handing over the reins. I think it's fine to do that. ideally we would be able to undo on agent uninstall tho [01:24] babbageclunk: nps [01:24] axw: yep... which should be doable [01:24] wallyworld, axw: it looks like we don't bother with the proxy settings on windows [01:25] * menn0 files some bugs [02:22] wallyworld babbageclunk: I'm going to be out both standups tomorrow and thursday (dentist, then school assembly) [02:22] no worries [02:30] wallyworld: seems you were right, LXD talks to the source directly [02:30] axw: it was a guess rather than being right :-) [02:30] wallyworld: but the client does need to connect to the source and the LXD as well [02:31] wallyworld: so I think it's that initial bit where it's trying to get the info that's going to be the problem [02:31] right, and that's something we can fix at our end [02:31] albeit a bit hackish perhaps [02:40] well... bollocks, and more bollocks [02:41] somehow in my massive pile of changes, I seem to have broken some of the macaroon stuff [02:41] seems that what used to end up having a successful post to /auth/discharge, now gets: [02:41] cannot get discharge from \"https://localhost:35994/auth\": cannot acquire discharge: cannot http POST to \"https://localhost:35994/auth/discharge\": Post https://localhost:35994/auth/discharge: x509: certificate signed by unknown authority [02:42] since this is a test... the tlsConfig should be set to use the CA cert... I think [02:42] but this is in the http bakery code... [02:42] so NIF [02:42] NFI [02:42] * thumper diggs through the code changes to see if I have inadvertantly deleted something important [02:44] hmm... [02:44] * thumper thinks some... [02:45] hmm.... I have a cunning plan [02:45] hopefully not too cunning... [02:46] gah... [02:47] I think this is another "localhost" != "127.0.0.1" issue with the bakery apiHost [02:53] * thumper stretches his brain [03:22] this may or may not work... [03:22] I'm hoping it will [03:22] it'll save a lot of pain [03:24] yay, that fixed the macaroon issue [03:24] phew [03:24] now for the others... [03:24] I may be able to revert some of these other shitty changes I have done [03:24] reduce the diff size [03:24] hazaah [03:38] * thumper afk for a bit -- kid pickup === thumper is now known as thumper-afk [04:22] * thumper-afk headdesks [04:22] oh === thumper-afk is now known as thumper [04:22] FFS [04:22] * thumper takes a deep breath [04:23] was trying to work out why the mutex values I added weren' [04:23] weren't stopping concurrent writes [04:23] then I noticed the reciever wasn't a pointer [04:23] so every method call got its own mutex [04:23] awesome [04:25] nice :( what area r u in? [04:28] rpc/jsoncodec [04:28] it was my doing though, adding the mutex [04:28] because gorilla/websocket has checks for concurrent writes / reads [04:53] thumper: still around? [04:54] kinda [04:54] whazzup [04:54] ? [04:54] remember the "mad-hack" you guided me through when we had missing txn-revnos? [04:54] ah... yeah [04:55] It's just happened to me for the 3rd time in a different environment, with the same charm. [04:55] Wondering if it's worth logging a bug [04:55] (given that it's 1.25.x) [04:56] really? that is kinda batshit crazy [04:56] IKR! [04:57] have you been doing any juju upgrades? [04:57] this is 1.25.6 and has been for quite a while [04:57] hmm... [04:58] is it an environment that is resistent to upgrading? [04:58] Well, between 1.25.7 and 1.25.9, we were quite resistent to upgrading... ;-) [04:59] I don't have a problem with upgrading to 1.25.10, but is that going to fix anything? [05:00] not sure... [05:00] I'd have to go look at the 1.25.6 codebase [05:01] blahdeblah: file a bug with as much info as you have, and I'll take a look to diagnose tomorrow [05:01] it is possible that 1.25.10 will fix, but it probably makes sense to look [05:01] if I can't find a good answer, at least we can write that on the bug [05:01] thumper: I kept detailed notes about what we did and last time just stepped through and fixed it [05:01] assign to me too [05:02] So I'm happy to go through again and do the same; or would you prefer I left it until we can dive in further? [05:10] blahdeblah: no, don't wait to fix [05:10] I'll just be reading code [05:11] * thumper thrashes his computer some more with a full test run [05:11] babbageclunk, menn0: I found a way around the whole "localhost" != "127.0.0.1" issue [05:11] much cleaner [05:11] thank god [05:15] thumper: OK - thanks [05:33] jam: if you get a chance, i'd appreciate a look at the IngressNetworksForRelation() API implementation in https://github.com/juju/juju/pull/7012. For now I just get state.AllIPAddresses() and filter multicast and loopback. I also filter out IPv6 and AWS didn't like it when updating the security groups [05:38] wallyworld: looking [05:38] thanks [05:40] hey y'all, got a groovey review: https://github.com/juju/juju/pull/7013 [05:40] although babbageclunk did say he'd take it [05:41] given that it is huge [05:41] not thaaaaaat big [05:42] * thumper is done [05:43] laters folks [05:54] Bug #1666396 opened: Missing txn-revnos in mongodb leads to missing status updates [06:53] anastasiamac: would you please review https://github.com/juju/juju/pull/7014? [06:53] axw: in a sec but will :D [07:01] axw: "the proxy should reject attempts to connect to its own lxdbr0 address" [07:01] axw: I'm not sure that should be "the proxy should" [07:02] jam: it should :) what isn't clear? [07:02] jam: the idea is to show that no_proxy is working [07:02] axw: so you said the host's IP address on lxdbr0 is 10.250.243.1, and then you set the proxy as 10.250.243.1 if the proxy is rejecting attempts to connect to that address [07:03] jam: I did also test without no_proxy to show that it *does* fail [07:03] then nothing can talk to the proxy [07:03] jam: specifically, it should reject attempts to connect to 10.250.243.1 *through* the proxy [07:03] I'll reword [07:03] axw: k, that could be spelled a bit clearer [07:04] cause I'm pretty sure you're connecting *to* the proxy via that IP :) [07:16] axw: ugh, just spent 10 minutes searching for the right version, cause we have "github.com/juju/juju/utils/proxy" *and* "github.cqm/juju/utils/proxy" [07:16] jam: heh yeah :/ [07:17] axw: we should probably update that code to support CIDR notation [07:18] I see a fair bit of stuff that plays games with the strings instead of using net.SplitHostPort, etc. [07:19] axw: hasPort() => strings.LastIndex(':') > strings.LastIndex(']') [07:19] jam: would be nice, but I don't think $no_proxy supports CIDR normally? [07:19] seems ... fishy [07:20] jam: it does a bit. I think it was lifted from net/http [07:20] axw: yeah, looking at docs it seems it is intended to support domain suffixes instead of IP prefxies [07:20] prefixes [07:38] axw: your PR relies on having env vars for proxies to use in the CLI command. and uses model config for the agent related stuff. IIANM. Is that an issue that these 2 sources could be out of sync? [07:57] axw, jam: the fishy string monkeying in the net/http proxy handling code has been fixed in recent versions of the Go stdlib [07:58] menn0: sounds like we should update our fork [07:59] wallyworld: I'm somewhat ok with that. "What are the PROXY settings *of my laptop* is not the same as what are the PROXY settings for my cloud" [07:59] jam: our fork in gh/juju/juju/utils/proxy does the right thing I believe [07:59] menn0: hasPort still does strings.Find [07:59] sorry strings.LastIndex [08:00] jam: ok right. well in Go 1.6 it's much worse :) [08:00] jam: yeah, that does make sense, but it can lead to confusion for local lxd cloud where everything runs on same host [08:00] jam: the code looks like current master [08:00] (Go master) [08:00] I believe [08:01] menn0: net.http.ResetCachedEnvironment() [08:01] seems to be a thing [08:02] jam: only in an export_test.go [08:02] yeah [08:02] I just tracked that down [08:02] blah [08:02] hence all the trickery [08:02] menn0: one of those "if its useful in tests, it would *probably* be useful in production for other people" [08:03] jam: indeed :) [08:03] nobody would ever change proxy settings on the fly, right? [08:10] wallyworld: that's existing behaviour, didn't want to muck with it right now [08:10] no worries, it does make sense that that's what it needs to be [08:10] wallyworld: i.e. we previously would have honoured $http_proxy in the client, but not used http-proxy from --config in the client [08:11] we should make sure doc is clear [08:11] and maybe print a message if there are differences when using local provider or something? maybe that's overkill [08:20] jam: thanks for the review [08:21] wallyworld: we could print out what proxy settings are detected, might be useful. but not in this PR === jamespag` is now known as jamespage [11:51] does anyone know if tim replicated https://bugs.launchpad.net/juju/+bug/1634178 ? [11:51] Bug #1634178: login doesn't prompt password [11:57] jam: thanks for review, i've replied to comments etc. i'll take another look, but I could not see subnets being populated from the provider at all - the provider (the aws one I looked at) seemed to populate only the ipadresses collection [11:58] wallyworld: the provider isn't populating subnets, that's the discussion I had with you [11:58] wallyworld: the ipaddresses is only being populated in 2.1 by the JUju machiner watching the local ip devices [11:58] jam: oh, i thought we said maas and aws did (via their NetworkInterface implementation [11:58] wallyworld: but you're missing a bunch of the information, like all the subnets that will be used in the next add-unit [11:59] wallyworld: they aren't? [11:59] aws and MAAS should be [11:59] not that i saw - i traced the code and it all seemed to point to ipaddresses being poulated [11:59] i'll need to look again [11:59] i started using AllSubnets() and it never seemed to get populatred [11:59] wallyworld: all of the "juju deploy --bind space" stuff requires subnets to have values [11:59] hence i looked at the code and saw ipaddresses [12:00] ok, i'll take another look [12:00] i may have misread something [12:09] jam: i'm looking at the discover spaces worker. that seems to want to update the subnets collection, right? [12:10] wallyworld: yeah [12:10] jam: but aws provider does not support spaces discovery [12:10] so maybe it's only maas that will have subnets collection populated [12:11] wallyworld: so for AWS, you do manual "juju add-space" and "juju add-subnet" if you want to use "juju deploy --bind", because we don't know what subnet is in what space if you don't tell us. [12:11] jam: right, so that screws up the firewaller and cmr [12:11] we want cmr to work without adding spaces [12:11] without requiring spaces [12:12] wallyworld: so update the discovery mechanic to record subnets without requiring spaces [12:12] wallyworld: its still *far* better [12:12] else you run into "what machines have I actually deployed so far" [12:12] instead of "what subnets could be in use as soon as I add-unit" [12:12] right, i didn't didn;t think i needed to do that as i thought aws provider populated its stuff [12:12] i'm just seeing a lot of this code for the first time [12:12] not sure what's there already [12:12] wallyworld: to be fair, *it should have* [12:13] wallyworld: (I only saw concrete code a few months ago as well) [12:13] and i'm getting my head across the model etc [12:13] wallyworld: well, I'm pretty solid on how things "should" be, and loosely aware of how things *are*. :) [12:13] ok, so i'll rework the PR to use subnets; the fallabck to 0.0.0.0/0 will work, and then i'll need to add stuff to the providers [12:14] everything is awesome [12:14] "how things are" is interesting, a fair bit to do still [12:14] it seems [12:17] indeed [12:17] jam: so would it be reasonable to modify the discover spaces worker to say that is the provder doesn't support spaces discovery, just ask it for all subnets (if it supports that which i think aws does) and whack those in the subnets collection? my fear is that subnets appear to have a space name attribute and that if we record subnets with a blank space name, things will screw up later [12:21] wallyworld: we're currently treating "" as the "unknown" space elsewher, I think its fine [12:24] jam: ok, i'll see how it goes. i'll look to land this PR after changing t subnets; do a followup which adds the subnet listener to the worker so that as subnets are added async the ingress rules are updated; start fixing the proivders starting with aws to populate subnets info. i wasn't budgeting to that that last bit, nor updating the other providers; hopefully it won't all blow out too much [13:40] does anyone have strong feelings about removing colors from vanilla loggo and putting them into a separate package under utils? [13:41] we've had problems with all our services producing unparsable logs because of the default behaviour === rogpeppe1 is now known as rogpeppe [13:41] and i don't think it's right that a low level logging package should be pulling a bunch of special dependencies for that purpose [13:43] as long as you dont break juju default behavior [13:43] without colors, loggo has no external dependencies. with them, it depends on 6 large external repositories [13:43] perrito666: we'll change juju to use the colour writer [13:44] perrito666: its behaviour won't change until the loggo dep is updated in juju [13:44] rogpeppe: wel without colors I would not use juju logger :p so they make sense to me [13:44] rogpeppe: I dont have objections then [13:44] jam, wallyworld: ^ ? [13:45] rogpeppe: I would feel better if you enlisted the opinion of thumper since he implemented the colour fancines [13:45] * perrito666 wrote color in brit english just for rogpeppe [13:46] perrito666: ok, i'll post a mail to the list [13:46] rogpeppe: yeah, best tto ask thumper; i do see you have a point though [13:46] perrito666: i never see thumper in my time zone [13:47] rogpeppe: yes, he seems to be in the exact oposite, would you like for me to ping him so he answers your mail promptly? [13:47] perrito666: please [13:47] rogpeppe: noted [13:47] perrito666: ta [13:47] perrito666: i'm sure he'll see my mail to the list anyway [13:47] rogpeppe: certainly just making sure he answers asap [14:02] perrito666: i've sent a message to the list [14:02] rogpeppe: tx, I added an item to my todo to talk to thumper as soon as he says good mornig [14:03] perrito666: thanks! [15:16] jam: ping me if you are back around === mup_ is now known as mup [18:55] all right, i forget, if i'm going to propose a PR into juju, what should I clone off of / propose merging with? master? develop? staging? whaaa? [18:55] cmars: hard question [18:55] lol [18:56] I believe it would be safe to branch staging and propose into develop [18:56] perrito666, ok, thanks! [18:57] cmars: although you might want to ask someone else too, I can never recall if staging is up to date with 2.1 [18:57] sinzui: ? [18:58] perrito666: cmars, for the last 10 days, staging and master have been getting develops blesses. staging is testing the bless develop got a few hours ago. [18:59] perrito666: cmars staging/master is NEVER a stable branch. 2.1 and 2.0 are their own histories... [18:59] which makes you wonder about the intent to always release off og master [19:01] hmm [19:01] i'll start with staging and cherrypick it wherever it needs to go === frankban is now known as frankban|afk [19:57] morning! [19:58] babbageclunk: morning [20:10] perrito666: feel like a fairly easy review? https://github.com/juju/charmrepo/pull/110 [20:13] well.... shit [20:17] thumper: ah, one of those mornings [20:17] thumper: btw, there is a rogpeppe mail that is pretty much just for you [20:17] babbageclunk: if you can hold until I return from the supermarket ill gladly review that [20:19] also if any of you is very familiar with how juju handled tools in 1.25 there is someone needing help in the private channel :) [20:19] perrito666: dude, if you don't want to just say so. ;) [20:19] thumper: feel like a fairly easy review? https://github.com/juju/charmrepo/pull/110 [20:20] perrito666: saw it [20:20] babbageclunk: how about a call, you can explain you change, and I need a teddy bear [20:26] ok [20:26] thumper: ^ [20:26] babbageclunk: https://hangouts.google.com/hangouts/_/canonical.com/lol-wat [20:43] back === jog_ is now known as jog [21:53] menn0: re bug 1666660 - normally we remove the primary document (the model) before any satellite documents. I assume that will be the focus of your fix? [21:53] Bug #1666660: list-models sometimes errors after a migration (before model has been removed) === frankban|afk is now known as frankban [22:06] wallyworld: I'm not working on that at the moment and haven't got a specific solution in mind [22:07] menn0: no worries, was just curious based on the description [22:07] wallyworld: looking at removeAllModelDocs, it removes the model doc last [22:07] wallyworld: which will be the problem [22:07] yep [22:07] wallyworld: at the point it's called the model is marked Dead [22:08] wallyworld: I suspect that model listing and lookup should avoid Dead models [22:08] we've had similar issues in the past with other entity->subordinate object relationships [22:08] yay mgo txns and not being atomic [22:09] "txns" [22:09] wallyworld: understood [22:09] wallyworld: in this case I think it makes sense for removeAllModelDocs to remove the model last [22:10] wallyworld: in case it fails, leaving the model doc behind makes it possible to retry [22:10] wallyworld: I think GetModel should ignore Dead models === mup_ is now known as mup === mup_ is now known as mup === frankban is now known as frankban|afk === thumper is now known as thumper-dogwalk [23:13] thumper-dogwalk: Did https://bugs.launchpad.net/juju-core/+bug/1666396 make sense? Please let me know if you need any further information. [23:13] Bug #1666396: Missing txn-revnos in mongodb leads to missing status updates