mupBug #1613487 opened: Fails to build tools when no matching tools found. <bootstrap> <ci> <regression> <juju-core:New> <https://launchpad.net/bugs/1613487>00:42
mupBug #1613491 opened: juju service names limited to 66 characters <juju-core:New> <https://launchpad.net/bugs/1613491>00:57
menn0wallyworld: i've spent a bit of time looking into this environs/sync issue and nothing jumps out01:05
menn0wallyworld: I don't see any cases of a gc.C being stored inappropriately or even goroutines being used01:05
wallyworldyeah, i had a brief look earlier and am in the same boat01:05
wallyworldwe really need there to be more info logged01:06
menn0wallyworld: ha! I had it running with dave's stress script while I was having lunch and managed to have it fail like that once out of ~100 times01:07
menn0wallyworld: so it is somewhat reproducible01:07
wallyworldand yet CI fails more regularly01:08
wallyworldhard to know where to target though with no errors01:08
menn0wallyworld: i'm going to try running it again with -check.vv01:09
menn0wallyworld: ok it failed quite quickly that time01:11
menn0just checking the output01:11
menn0wallyworld: got it01:12
* menn0 pastebins01:12
menn0wallyworld: here's the 3 failures I just got: http://paste.ubuntu.com/23060118/01:14
wallyworldmenn0: intersting, wtf isn't that deterministic01:15
wallyworldi can look into it01:15
menn0wallyworld: I see it01:17
menn0wallyworld: GetMockBundleTools is patched in during SetupTest and it keeps a reference to the gc.C that was passed to SetUpTest01:18
menn0wallyworld: when it fails during the test that gc.C is no longer the current one01:18
menn0(i.e. the one that gets passed to the test)01:19
menn0that's what causes these tests that both pass and fail01:19
menn0wallyworld: moving the GetMockBundleTools patching into the tests themselves makes them fail consistently (due to typed nil)01:22
axwwallyworld: if you don't mind, PTAL at http://reviews.vapour.ws/r/5441/01:23
wallyworldmenn0: ah, i hate that multiple gc.C thing01:25
wallyworldmenn0: i have a branch i need to start, i can fix in that01:25
menn0wallyworld: I'll push what i've done so far01:26
wallyworldmenn0: ok, np. i haven't had any time so far today to dive in, keep getting pulled left and right01:26
menn0wallyworld: understood. this one got my attention because a) it's odd and 2) it's bitten me several times in the last day01:27
wallyworldyeah, it will be fixed... today!01:27
wallyworldaxw: lgtm. we'll need to address the @local suffic at some point because I think it was going to be removed01:29
axwwallyworld: okey dokey. I'd love it if we just had one canonical form always01:29
wallyworldagreed, i think that's where the push is coming from, need to touch base on that again01:30
menn0wallyworld: pushing now. all the tests pass again except one and I'm not sure if the test is wrong or the implementation.01:31
menn0wallyworld: the tests were only passing by luck before it seems.01:31
menn0(due to the typed nil comparison01:31
wallyworldmenn0: ok, ta, will merge into mine01:31
menn0wallyworld: https://github.com/mjs/juju/tree/environ-sync-tests01:32
=== menn0 is now known as menn0-afk
=== natefinch-afk is now known as natefinch
natefinchcould really use a review: http://reviews.vapour.ws/r/5440/  axw or anyone else02:13
axwwallyworld: FYI, possibly not going to have these changes done today. I'm going to have to change a bunch of code in state, API, etc. to use credential tags instead of names02:18
wallyworldnot surprised02:18
axwso probably won't get to LXD till tomorrow02:18
natefinchwallyworld: the auto-upload-tools thing still seems dangerous to me. What if you have 2.0 deployed, have a client/server built locally, but 2.2 is available in streams.  What does juju upgrade-juju do?02:20
wallyworldupgrade to 2.202:21
wallyworldwell, actually,i need to check the urgrade rules02:21
wallyworldif you have a, then you have built it and so are expected to know what you are doing02:22
natefinchwallyworld: expected to know what you're doing is a lot different than understand exactly what fine details of the code are going to do02:22
wallyworldi can't recall off hand how we select the allowed upgarde version when searching02:22
natefinchwallyworld: it seems like any time both streams and your local are valid upgrade paths, there'02:22
wallyworldupgrades have always been like that02:22
natefinchthere's no way to know which one the user intends to use02:23
wallyworldit searches for packaged tools first02:23
wallyworldif it finds them, it will use them in preference02:23
wallyworldso whatever upgrade rules we have for selecting will be used02:23
wallyworldi think we look for major.minor02:24
wallyworldit only uploads automatically if no packaged tools are there, and only if what you have is different and only if jujud matches the client being used02:24
natefinchI guess my point is - I want to make sure we never automatically upgrade someone from a released version to a development version with some explicit flag or confirmation02:25
wallyworldright now, there's not confirmation - it will use a jujud mactching juju02:31
natefinchthat seems undesirable02:31
wallyworldso if juju is 2.1-beta1 it will upgrade a 2.0 (if the upgrade rules select such a version)02:32
wallyworldi'l need to check that02:32
natefinchI just don't want someone to get screwed who happens to build juju from source to test out the latest and greatest... forgets for a week, then upgrades their production server and boom, now they're on devel02:39
mupBug #1611097 changed: cannot add/destroy a model if done too fast <oil> <oil-2.0> <juju-core:Invalid> <https://launchpad.net/bugs/1611097>02:40
wallyworldmenn0-afk: when you are back, here's a PR to fix that test and a CI issue http://reviews.vapour.ws/r/5442/02:46
=== menn0-afk is now known as menn0
menn0wallyworld: I am back.... forgot to update nick02:47
wallyworldmenn0: ah, i see a mistake, a wrong test suite name, fixing02:48
wallyworldhmm, scrap that, it can stay as is02:49
anastasiamacaxw: wallyworld: bug 1582021 was addressed, right?02:54
mupBug #1582021: Juju loses track of current controller when destroying model <2.0> <bitesize> <destroy-model> <juju-release-support> <usability> <juju-core:Triaged> <https://launchpad.net/bugs/1582021>02:54
wallyworldthink so?02:54
axwdon't recall exactly02:55
anastasiamaci'll mark as fixed committed \o/ would have been sooo very lovely to have a functional test that would just tell us :) mayb even have a link to a passing ci test in the bug :)02:55
anastasiamacveebers: ^^02:56
menn0wallyworld: done!03:04
menn0wallyworld, thumper: do you think I moved the notification about an available upgrade into the new NOTES status column too?03:08
thumperaxw: why do we store the charm url on the storage instance when you can get it from the owner?03:08
thumpermenn0: +103:09
menn0thumper: that's what I was thinking too03:09
thumpermenn0: showing the most relevant info03:09
thumpermigration > upgrade available > nothing03:09
wallyworldmenn0: yeah, good idea03:09
axwthumper: it's not necessarily the same. it's the charm URL that was in use when the storage instance was created, not the current one03:10
menn0thumper: yep that's exactly what I was thinking03:10
thumperaxw: but why do we care?03:10
axwthumper: I forget why we have it at all. we don't use it AFAIK03:10
axwwallyworld: do you recall?03:10
thumperwe could just drop it, yeah?03:11
axwthumper: if we haven't needed it thus far, probably safe to drop it03:12
wallyworldthumper: axw: i can't even see where it's used03:12
thumperaxw: you ok if I do that as part of the migration branch I'm doing?03:12
axwthumper: SGTM03:12
thumperok, cool03:12
* thumper is looking how to add storage to the model to test exports03:13
wallyworldmenn0: thumper: it seems to me that in the multi-model world, stuff is not getting removed when a model is destroyed. it never used to matter in the single model case. but now for example, i can see where we are creating model settings and constraints in modelSetupOps but i can't see where this data is ever removed via calls to removeConstraintsOps etc. I can see where constraints and what not are removed for applications, units,03:36
wallyworldmachines, but not models. am i on crack?03:36
menn0wallyworld: I didn't write this stuff but RemoveAllModelDocs catches everything03:37
menn0wallyworld: very brute force03:37
* wallyworld looks at that method03:37
menn0wallyworld: not sure when/how it gets called when models are destroyed normally03:37
menn0wallyworld: until yesterday it was missing user perms but I fixed that03:38
menn0wallyworld: looks like the undertaker worker ends up calling it, so everything should be getting removed03:39
wallyworldmenn0: awesome thanks03:39
* natefinch tries for like the millionth time to use custom streams with the CI tests03:42
mupBug #1474291 changed: juju called unexpected config-change hooks after read tcp i/o timeout <bug-squad> <docs> <hooks> <openstack> <uosci> <juju-core:Invalid> <https://launchpad.net/bugs/1474291>03:49
mupBug #1613516 opened: AWS instance types should be detected <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1613516>03:49
natefinchit's really a shame we can't just upload tools to the controller and tell it what series they're for.03:50
thumperOMG, state.AddApplication is 230 lines...03:50
natefinchat my first job out of college, the main entry function was 1000 lines long03:52
thumperover 100 lines of that is arg validation03:52
thumpernatefinch: I worked at a bank where the CreateInstrument function was 10000 lines03:52
thumperit was horrible03:52
thumperand noone wanted to fix it03:53
natefinchthat's beyond horrible03:53
thumperbecause "what if it broke"03:53
natefinchyes.. what IF it broke? :/03:53
thumperno unit tests, just overnight regression tests03:53
natefinchmy last job, we barely had tests.  I'm sure it was like 5% coverage.... and many developers openly criticized the idea of writing tests.03:54
natefinchit's a damn miracle anything worked.  We did have like 8 full time QA people doing manual tests, I guess.03:56
menn0wallyworld: so can you show me your clouds.yaml?03:56
wallyworldonly if you show me yours :-D03:56
wallyworldi'll pastrebin03:56
wallyworldmenn0: http://paste.ubuntu.com/23060299/03:57
thumperaxw: able to save me some time and point me at the simplest test code that adds a storage instance to the db?03:57
menn0wallyworld: I see the problem. I had my config at the wrong level... no "config" section.03:58
thumperaxw: factory.MakeApplication?03:58
menn0wallyworld: I wonder if juju add-cloud should complain if you mess this up?03:58
menn0(like I did)03:58
veeberswallyworld, axw: re: a test for 1582021 as mentioned by anastasiamac, does this sound reasonable for a testplan? test would bootstrap, add a new model, switch to it, (ensure status works) destroy that model then ensure that it could both "list-models" and "status" successfully?03:58
wallyworldveebers: if you just destroy a model and don't switch to another one, you will get an error04:00
wallyworldit will be interesting to repro the race condiiton04:00
veeberswallyworld: is that the error that is shown in the bug? Perhaps I'm not sure what the test behaviour should be04:05
wallyworldit depends if the bug has been fixed. the bug refers to a not found which is the original symptom. if the current model is cleared out, then there will be a different error refering to no current model selected04:07
anastasiamacwallyworld: veebers is writting the test to detemrine whether the bug has been fixed. he just needs to confirm if his scenario will veriffy the fix04:08
wallyworldright, understood. as i said, there's 2 aspects. the resetting of current model and the race04:08
wallyworldthe error wrt the resetting of current model will depend on if the bug has been fixed. and you need to switch models if you want to avoid an error of either kind04:09
wallyworldif the bug has been fixed, you get the not found, if not you'll get the no current model as stated above04:10
wallyworldassuming you don't switch models04:10
anastasiamacwallyworld: the bug specifically deals with no-explicit-switch situation.. it's covered in bug description04:11
anastasiamacwallyworld: mark is not happy with "not found"... did u read the bug? :D04:11
wallyworldright, but the test also needs to cover the switch case04:11
veeberswallyworld: Ok, so I want to confirm the mention bug is fixed, so I destroy and don't switch, doing a list-models or status should give me . . .something? better asked what should the expected result be04:11
wallyworldand as i said, if the bug has been fixed (ie current model reset), you will not get the not found04:11
wallyworldbut you will get a different error04:12
wallyworlduntil you switch to a different model04:12
wallyworldi thought i explain it! you will get a different error04:12
wallyworldnot a not found error but an error saying there is not current model04:13
wallyworldand if you switch everything will be ok again04:13
veebersok cool, I'll run with that. Cheers04:14
wallyworldif you get stuck, let me know; sorry if i was unclear04:14
axwthumper: sorry, was having lunch. if you just add a unit whose charm has non-optional storage, it'll add a storage instance automatically04:15
thumperaxw: is there an example of that anywhere?04:16
thumperdo we have a testing charm that has non-optional storage?04:16
axwthumper: state/storage_test.go, StorageStateSuiteBase.setupSingleStorage (actually that one's non-optional, but that code sets up the storage constraints)04:17
axwthumper: err sorry, that one's *optional*04:17
thumperthat'll do...04:18
thumperI'll crib from that04:18
veeberswallyworld: fyi I get the error message "..no model in focus" which seems like the bugfix is in place for that part at least04:24
wallyworldveebers: awesome, that means that it is fixed, yeah04:24
wallyworldso now the test should switch04:24
wallyworldand confirm that everything is ok04:24
wallyworldi would have thought list-models could still work though04:24
wallyworldeven with no model in focus04:24
wallyworldif it doesn't that's a bug IMO04:25
wallyworldhow do you know what to switch to04:25
wallyworldif you can't list models04:25
anastasiamacveebers: would be awesome if commented on the bug with your findings. we can change status of the bug based on ur test \o/04:27
veeberswallyworld: 'models' and 'list-models' seem to work, although switching to a model doesn't seem to work, although I could be doing something off in my test code. I'll re-pro manually04:29
wallyworldlist-models works even without a current model? cool04:29
wallyworldi think we set current model with a *04:29
wallyworldso list-models should not have the * without a current model selected04:30
natefinchwallyworld: I'm trying to use my own streams to run a test.  if I bootstrap with --metadata-source ... where is that stored?  I don't see that config val in get-model-config or get-controller config04:38
wallyworldit's not stored04:38
wallyworldwhat streams? tools or images?04:39
natefinchwallyworld: tools04:39
wallyworldtools are stored in blobstore04:39
wallyworldmetadata-source is only used to tell bootstrap where to find tools to upload and store04:40
natefinchwallyworld: I'm trying to test something on windows, so I need a secondary machine to be able to get the windows binary that I have built. how do I do that?04:41
wallyworldit's not trivial04:42
wallyworldone way is to use agent-metadata-url pointing to an apache http server04:42
natefinchwell, so I have tools & streams generated and sitting in S304:42
wallyworldif it's a public bucket you could use that url as agent-metadata-url04:43
natefinchwallyworld: ok, I thought metadata-source would have done that, but I can set metadata-url, that's fine04:43
natefincher agent-metadata-url04:43
wallyworldmetadataa-source -s just fpr bootstrap04:44
wallyworldto tell it where to find a local (as in dir) source of tools/image metadata04:44
wallyworldgenerated using the plugins04:44
wallyworldwhich is then sent to controller suring bootstrap and stored in state04:44
natefinchahh... then maybe it should complain if I give it a URL :/04:45
wallyworldfile:// is a url04:45
wallyworldwhich it uses04:45
wallyworldi guess it could complain if not file://04:45
wallyworldor a dir04:45
natefinchbut --metadata-source https://s3.amazonaws.com/24b99da448314945835933a0c8edffb3 just fails silently04:45
natefinchnot fails... is ignored04:45
wallyworldyeah, there's not been much polish on dev only tools04:45
wallyworldi didn't think it failed, that must be a new bug04:46
natefinchbootstrap doesn't fail, which is kind of the problem04:47
menn0thumper or wallyworld or anastasiamac: MM status in tabular output http://reviews.vapour.ws/r/5444/04:48
wallyworldmenn0: i asked for one more test04:55
menn0wallyworld: I thought of that one but was being lazy. done now :)_05:14
wallyworldwe've all been there05:15
wallyworldyou have a shipit anyway05:15
mupBug #1613537 opened: metadata-source doesn't sanity check the value you give it <juju-core:New> <https://launchpad.net/bugs/1613537>05:19
wallyworldaxw: if you had a moment at any time, this is a mechanical change to the model defaults api to accommodate cloud/region. not used yet but will be when reed's stuff gets further along http://reviews.vapour.ws/r/5445/05:44
axwwallyworld: ok, in a little while05:46
wallyworldsure, no hurry05:46
axwwallyworld: one fundamental thing: we have cloud tags now, should be using that instead of just a cloud name05:48
wallyworldah, yeah05:48
wallyworldaxw: updated for when you get to look later05:58
wallyworldthans axw06:09
mupBug #1551959 changed: juju-tools 1.25.0 with juju-agent 1.25.3 has networking issues <juju-core:Won't Fix> <https://launchpad.net/bugs/1551959>06:19
anastasiamacwallyworld: axw: just to confirm - i believe this bug has been fixed with new model naming convention that includes user name, non? :D bug 157635906:22
mupBug #1576359: Cannot talk to a new model with a reused name <juju-release-support> <switch> <usability> <juju-core:Triaged> <https://launchpad.net/bugs/1576359>06:22
wallyworldnot sure off hand06:23
wallyworldi wasn't familar with that bug06:23
anastasiamacaxw:  ? :D06:23
axwanastasiamac: not sure that that one is fixed06:23
axwanastasiamac: I don't think I did anything specifically that would have fixed that06:23
anastasiamacaxw: wallyworld: k. thnx06:24
axwwallyworld: https://github.com/juju/names/pull/72  - sorry, third time's the charm perhaps?06:31
mupBug #1100560 changed: Support method of removing a subordinate charm <canonical-is> <canonical-webops-juju> <remove-relation> <subordinate> <pyjuju:Won't Fix> <juju-core:Invalid> <https://launchpad.net/bugs/1100560>06:31
mupBug #1432759 changed: Transient error on status while running deployments via quickstart <api> <quickstart> <race-condition> <status> <juju-core:Invalid> <https://launchpad.net/bugs/1432759>06:31
wallyworldaxw: just gotta race out to school pickup, will look when i get back06:31
wallyworldam running really late06:31
=== rogpeppe is now known as rogpeppe1
mikenanastasiamac: Hey there. Just to be clear, bug 1613491 is not critical for me - we're working around it easily enough.06:52
mupBug #1613491: juju service names limited to 66 characters <juju-core:Triaged> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1613491>06:52
anastasiamacmiken: awesome \o/ thank you - i've updated the bug06:53
mupBug # changed: 862422, 1228355, 1301255, 1340756, 146822307:01
=== frankban|afk is now known as frankban
mupBug # changed: 1217814, 1283866, 1398589, 1401358, 1403460, 1417149, 142099607:28
axwwallyworld rogpeppe: this PR should address concerns about the cloud API, and also updates other bits to use the new cloud credential tag: http://reviews.vapour.ws/r/5447/07:29
rogpeppeaxw: i'll swap ya :) http://reviews.vapour.ws/r/5439/07:29
axwrogpeppe: sure, looking07:29
axwrogpeppe: hmm, I think it was intentional to leave them there, so old clients can get a useful error message07:31
rogpeppeaxw: won't they still get the same error message?>07:31
rogpeppeaxw: i think i tried to leave the behaviour unchanged. perhaps i didn't though...07:32
axwrogpeppe: maybe? I just saw a test that removed a check for the error message, so assumed it didn't any more07:32
* axw looks deeper07:32
rogpeppeaxw: ah, yes, maybe i removed that test when i shouldn't have07:32
wallyworldthe idea is that older 1.25 clients should get a nice message07:32
wallyworldnot sure of the details though07:33
rogpeppewallyworld: yes, they still should, but yeah, i removed some of the infrastructure that supported the test07:33
wallyworldif it works with the proposed changes, then that's ok07:33
wallyworld+1 to deleting code if it makes sense :-)07:34
rogpeppewallyworld: i think we could keep the test but make it manually call Login with version 2; no need for OpenWithVersion, I think.07:35
wallyworldsgtm, keep as little as possible07:35
mupBug #1494936 changed: imageSuite.TestDownloadEnvironmentPath <bug-squad> <ci> <intermittent-failure> <panic> <simplestreams> <unit-tests> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1494936>07:40
mupBug #1494936 opened: imageSuite.TestDownloadEnvironmentPath <bug-squad> <ci> <intermittent-failure> <panic> <simplestreams> <unit-tests> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1494936>07:49
mupBug #1494936 changed: imageSuite.TestDownloadEnvironmentPath <bug-squad> <ci> <intermittent-failure> <panic> <simplestreams> <unit-tests> <wily> <juju-core:Invalid> <https://launchpad.net/bugs/1494936>07:58
rogpeppeaxw: reviewed08:09
axwrogpeppe: ta08:13
axwrogpeppe: I'd prefer to leave mention of user/cred syntax in the help docs until we have a means to share08:26
rogpeppeaxw: "a means to share" ?08:27
axwrogpeppe: share credentials - ACLs08:27
axwrogpeppe: i.e. you may be able to use the syntax user/cred, but it won't work unless user == you08:28
rogpeppeaxw: the problem is that historically that means we'll forget to add it to the help docs08:28
axwrogpeppe: I'd rather that than confuse people by having them think they can do something they can't08:28
rogpeppeaxw: i'd add it to the help docs and add a caveat that currently the user must be the same as the model user08:28
rogpeppeaxw: or, tbh, since it'll just fail, that's probably fine08:29
rogpeppeaxw: it'll fail with "permission denied" presumably08:29
rogpeppeaxw: which is exactly how it would fail if the permissions hadn't been shared08:29
axwrogpeppe: "not found" at the moment I think08:29
rogpeppeaxw: even if the other user's creds actually exist?08:30
rogpeppeaxw: oh yes, because that's what we always do for perm denied i guess08:30
axwrogpeppe: yeah. generally we do that so you can't sniff out info08:30
rogpeppeaxw: so, i think it's fine to document the syntacx08:30
rogpeppeaxw: then when we implement sharing, people will find a use for it08:31
axwrogpeppe: ok08:32
rogpeppeaxw: also, describing it in the help docs is often a good indicator as to whether it's a decent feature or not08:32
rogpeppeaxw: i.e. can it be described nicely08:32
rogpeppeaxw: tell me about CreateLocalLoginMacaroon08:54
axwrogpeppe: it's how you get a macaroon that you can log into a controller as a local user08:56
axwrogpeppe: what do you want to know about it in particular...?08:56
rogpeppeaxw: i'm just wondering if it should be part of the controller-only API or not08:57
axwrogpeppe: yes, it is a controller level thing08:57
rogpeppeaxw: i'm getting a test failure where login fails08:58
rogpeppeaxw: because it's using the model-level connection (that it's using to log in) but then uses that to create a local login macaroon08:58
rogpeppeaxw: and that fails because UserManager is controller-only08:59
axwrogpeppe: what's the test?08:59
rogpeppeaxw: featuretests.cmdLoginSuite.TestLoginCommand08:59
rogpeppeaxw: it seems like a legit failure to me08:59
axwrogpeppe: ah, it's actually being used in change-user-password09:00
axwwhich should be controller-only..09:00
rogpeppeaxw: it is09:01
rogpeppeaxw: changePasswordCommand embeds ControllerCommandBase09:01
rogpeppeaxw: hmm, so is login actually09:01
rogpeppeaxw: so now i'm confused :)09:02
axwrogpeppe: that should mean they're getting controller API connections09:02
rogpeppeaxw: yup09:02
rogpeppeaxw: so it should work, but i'm getting09:02
rogpeppe[LOG] 0:00.242 ERROR cmd failed to create a temporary credential: permission denied (unauthorized access)09:02
axwrogpeppe: sorry, nothing's jumping out from the code... login is passing "" for the model name to get the API connection09:04
rogpeppeaxw: yeah, there's something odd going on. thanks.09:05
rogpeppeaxw: now i just have to work out how to get the featuretests to show trace-level debug logs of the api server09:06
axwrogpeppe: loggo.GetLogger("juju.apiserver").SetLogLevel(loggo.TRACE)09:07
axwI think09:07
rogpeppeaxw: i'll try that09:07
rogpeppeaxw: still not seeing any api server traffic (i also tried GetLogger with "" and "<root>")09:11
rogpeppeaxw: ISTR working out a way to do this before, but it's gone from my memory now unfortunately09:12
babbageclunkrogpeppe: I've never managed to set the root logger level using GetLogger("<root>") - try using ConfigureLoggers instead.09:20
rogpeppebabbageclunk: ha, thanks09:20
rogpeppebabbageclunk: the loggo code thinks that it should be ""09:21
babbageclunkrogpeppe: ah, that might be what I got wrong09:22
rogpeppebabbageclunk: but i suspect that all the logging is being lost09:22
rogpeppebabbageclunk: GetLogger should really have a comment09:22
rogpeppebabbageclunk: or better, all loggers should start with "." so you wouldn't need a special case for <root>; "." would always be the root logger.09:22
rogpeppebabbageclunk: too late for that now though09:23
babbageclunkdimitern: do you know if Will's out all week?09:26
dimiternbabbageclunk: I don't know - yesterday was a public holiday in Malta IIRC09:27
babbageclunkdimitern: Ah, ok.09:27
babbageclunkdimitern: Although he might have piggybacked a holiday on that.09:28
babbageclunkdimitern: Do you have any more comments other than the model tag checking stuff in http://reviews.vapour.ws/r/5366/?09:29
dimiternbabbageclunk: let me have a look..09:29
babbageclunkdimitern: Thanks! I made the other changes you suggested.09:30
babbageclunkdimitern: I've finished (for some value of finished :) the api client as well.09:31
rogpeppeaxw: ok, i just about succeeded. the problem is that juju/cmd.Log.Start tries to make the logs go to ctx.Stderr; it tries to replace the default loggo writer, but there isn't one. we should make it so you can just see the logs when desired, i think. maybe add another writer or something.09:32
macgreagoirwallyworld axw: I have that AddController change merged, much thanks.09:34
dimiternbabbageclunk: machineundertaker will be running once per model, right?09:35
axwmacgreagoir: thank you :)09:35
dimiternbabbageclunk: i.e. it will be configured inside cmd/jujud/agent/model/manifolds.go ?09:37
dimiternlike the space importer, etc.09:37
babbageclunkdimitern: I think so? Where's environ-tracker?09:38
babbageclunkdimitern: Yes, in cmd/jujud/agent/model/manifolds.go09:38
dimiternbabbageclunk: yeah, it's there as well09:39
babbageclunkdimitern: I think you're right that the model tag is unnecessary.09:39
dimiternbabbageclunk: yeah, that's what I'm getting at :)09:39
rogpeppeaxw: ah! i've discovered what the problem is. CreateLocalLoginMacaroon isn't considered a read-only call09:39
babbageclunkdimitern: But I'm reluctant to pull it out and then have Will explain a sensible reason why it's needed and then have to put it back in.09:39
dimiternbabbageclunk: as long as you're a model manager, there's no need to check or take a model tag (uuid) to work on - you'll only every work on one model anyway09:40
axwrogpeppe: why is that a problem?09:40
dimiternbabbageclunk: I'm interested to learn Will's reasoning for having ConnectedModel() ?09:40
rogpeppeaxw: because in that test i'm guessing the user is read-only09:40
rogpeppeaxw: (i haven't checked)09:41
babbageclunkdimitern: But you mentioned in your review that there were some other things you were thinking about as well?09:41
rogpeppeaxw: but in general, i think it's probably OK for even read-only users to be able to create a login macaroon09:41
babbageclunkdimitern: ConnectedModel is only needed for the model checking.09:41
axwrogpeppe: yeah probably. it's not writing anything to the model09:41
rogpeppeaxw: yup09:41
rogpeppeaxw: i haven't yet worked out why this has changed in my branch though09:42
dimiternbabbageclunk: so AuthModelManager() being true means you've logged in as a machine agent with jobmanagermodel09:43
dimiternbabbageclunk: it's true authorizer.GetAuthTag() does not give you the model uuid, just that machine tag you logged in as09:44
dimiternbabbageclunk: but then again, the model manager engine runs per model09:44
dimiternbabbageclunk: in startModelWorkers (cmd/jujud/agent/machine.go)09:46
babbageclunkdimitern: ok (reading)09:47
dimiternbabbageclunk: so, if checking whether the model uuid matches was an issue, it should've been done for at least some of the other workers+facades part of the model manager manifold09:47
dimiternunless I'm missing something..09:48
babbageclunkdimitern: Yeah, in which case we'd have needed ConnectedModel ages ago.09:48
babbageclunkOk, I'm convinced. It's easy to rip it out, and I'll keep the commit around so that it should be easy to put it back in if Will convinces us the other way.09:49
dimiternbabbageclunk: only if we need to test impossible cases :)09:49
dimiternbabbageclunk: but Will might have another good reason09:50
dimiternbabbageclunk: i.e. being explicit about which uuid you should allow in the facade09:50
dimiternbabbageclunk: but if that's the case, then it's much easier to take modelTag as argument to machineundertaker.NewAPI(), store it and check it internally09:50
babbageclunkdimitern: I think it might have been something like that. Hang on, I'll find the conversation in the logs.09:51
dimiternbabbageclunk: I'm happy to be proven wrong :)09:51
dimiternbabbageclunk: perhaps the intent is to call AllMachineRemovals() with multiple model uuids as args, say if we're tearing down more than 1 model09:53
dimiternbabbageclunk: but this means we're designing the facade with the intent to make it global, not per-model, and to allow 1 instance of machineundertaker to run for all models - then it makes sense09:54
babbageclunkdimitern: No, I don't think so. As you say, we'll be connected to one model.09:54
dimiternbabbageclunk: I'll add some comments09:55
dimiternif we had JobManageController, implying "can manage all hosted models", then fine..09:57
babbageclunkdimitern: For some more background (but nothing conclusive as far as I can tell) search for ConnectedModel in here: https://irclogs.ubuntu.com/2016/08/09/%23juju-dev.txt09:58
dimiternbabbageclunk: cheers, will do09:58
babbageclunkdimitern: But yeah, if you've got any other suggestions for changes, I'll make those before ripping out the model checking so that it should still be easy to put it back.09:59
dimiternbabbageclunk: do you have a link to Will's previous review about AllMachineRemovals() taking a list of model tags?10:01
babbageclunkdimitern: It's in comments on katco's review - not much detail I'm afraid.10:02
dimiternah ok10:04
babbageclunkfwereade: Hey! Speak of the devil!10:10
fwereadebabbageclunk, hey, sorry10:11
fwereadebabbageclunk, was signed into canonical but not here for some reason10:11
fwereadebabbageclunk, what can I do for you?10:11
babbageclunkdimitern and I are trying to understand why AllMachineRemovals needs to take the model uuid?10:13
babbageclunkfwereade: (as seen in http://reviews.vapour.ws/r/5366/)10:13
fwereadebabbageclunk, the api method?10:18
babbageclunkfwereade: yeah10:19
rogpeppeaxw: ok, i've found out what's happening. prior to my change, all the controller-only facades were accessible to read-only users, but now the read-only layer always applies10:19
fwereadebabbageclunk, because depending on the connection state is generally unhelpful, and silently ties the implementation down so it can't be moved to other contexts10:19
rogpeppeaxw: i have a problem with apiserver.admin.doLogin - it's too big and hard to understand for such a security-crucial method10:20
babbageclunkdimitern: ^^10:20
fwereadebabbageclunk, we have multi-model controllers -- even if all our connections right now have an implicit model, that's not fundamental to the problem space10:20
fwereadebabbageclunk, when you're talking to a controller directly, for example, there are going to be at least some things you can rationally do to all-your-known-models10:21
fwereadebabbageclunk, and so just making sure we name our targets from the beginning makes it trivial to extend that code out, differing only in auth concerns, as we grow10:22
fwereadebabbageclunk, sane?10:22
babbageclunkfwereade: yeah, that makes sense to me. dimitern?10:22
dimiternbabbageclunk, fwereade: catching up now10:23
fwereadebabbageclunk, note, tangentially, that controllers make an additional api connection per hosted model, and this is kinda silly -- the fact that implicit-model is baked in all over the place makes it necessary for the moment, but it's not a Good Thing, and writing our code *without* assuming that context will help us if/when we actually address that10:24
dimiternfwereade: so you're saying AllMachineRemovals(modelTags ...) is a good method to have on a facade that's only going to work against a single model?10:24
fwereadedimitern, yes10:24
fwereadedimitern, this shoudl not even be a question in your mind10:24
fwereadedimitern, because I don't want it to be a question in consumers' minds either10:25
fwereadedimitern, facade methods accept bulk args, full stop :)10:25
dimiternfwereade: I'm trying to see why - if in the future that same facade is used by a controllerManager worker of sorts, that can deal with multiple hosted models, that's fine10:25
dimiternfwereade: but which model you're working on seems to align well with the SRP10:26
fwereadedimitern, well, no, because you can't share the implementation without a rewrriate that takes implicit state into account10:26
fwereadedimitern, it's really just another special case of "explicit is better than implicit"10:27
dimiternfwereade: are we going to change all the other modelmanager manifold workers to use that approach?10:27
fwereadedimitern, which ones are not?10:27
fwereadedimitern, this is not even remotely in question10:27
dimiternfwereade: I see how this is good, but it irked me as inconsistent w.r.t current code examples10:27
fwereadedimitern, it has been How You Write An API from the very beginning10:27
dimiternfwereade: well, not so much for models10:27
dimiternfwereade: but I guess why not10:28
fwereadedimitern, we didn't even have models10:28
fwereadedimitern, name your targets10:28
dimiternfwereade: well, different models are likely to have different cloud creds10:28
fwereadedimitern, and?10:28
dimiternfwereade: it seems to me we need a new JobManageController or something10:29
fwereadedimitern, we just need a JobFlag in the machine agent, I think10:29
dimiternfwereade: I'm not that familiar with the flags yet10:30
dimiternfwereade: but fair enough10:30
fwereadedimitern, the whole flag implementation is in jujud/agent/engine, it's pretty comprehensible10:30
dimiternfwereade: how about machineundertaker.NewAPI() taking a modelUUID / tag to work on10:30
fwereadedimitern, the two workers that implement engine.Flag are pretty trivial10:30
fwereadedimitern, is that apiserver-side?10:31
fwereadedimitern, then, no10:31
dimiternfwereade: yeah10:31
dimiternfwereade: isn't that more explicit than going through the authorizer each time?10:31
fwereadedimitern, what you are saying is "I have a special case that I can imagine a different implementation, please may I throw consistency to the winds and take on secret technical debt?"10:32
fwereadedimitern, just accept your Entities like every other method should10:32
dimiternfwereade: I'm just saying we can change the NewAPI constructor if we need later, keep the AllMachineRemovals() as-is (taking model tags), and have a slightly simpler initial implementation10:33
fwereadedimitern, api-side, sure, create the type with a single ModelUUID and call .AllMachineRemovals() -- but we want the data we send over the wire to make sense in isolation10:33
fwereadedimitern, how do we change the NewAPI constructor? isn't that essentially constrained by the facade package?10:34
fwereadedimitern, as it is we have the implicit model known because we have the *State, and yes, unpicking that will be tedious10:34
dimiternfwereade: it is, but it's not visible to api clients, and can change without breaking the wire format of the facede10:35
rogpeppe[11:21:24] <fwereade> babbageclunk, when you're talking to a controller directly, for example, there are going to be at least some things you can rationally do to all-your-known-models10:35
rogpeppefwereade: FWIW controller-only APIs exist in a separate space10:35
rogpeppefwereade: so it will of necessity be a different facade in that case10:35
dimiternfwereade: ok, I just wanted to clarify your reasons for asking to add ConnectedModel()10:35
fwereaderogpeppe, ofc10:35
dimiternfwereade: I don't mind it otherwise10:36
fwereaderogpeppe, facades should basically be thin and just contain auth handling, and hand over to shared backend implementations10:36
rogpeppefwereade: so (assuming I've understood the gist correctly) I have no objection to a connection on a model-only API not taking the model tag as parameters10:36
fwereaderogpeppe, right, but I do10:37
rogpeppefwereade: i don't get it10:37
rogpeppefwereade: we don't pass the model tag when, say, adding a service10:37
fwereaderogpeppe, right, and that is also unhelpful10:40
rogpeppefwereade: i really don't understand. what's the purpose in passing redundant information over the API other than to add more possible error paths and code bloat generally?10:40
fwereaderogpeppe, because it expands the scope of places you need to know and understand to comprehend the meaning and effect of a given api call10:42
fwereaderogpeppe, tell me more about how extra magic reduces error paths10:42
rogpeppefwereade: i don't consider denormalisation "less magic". it's just more code.10:42
rogpeppefwereade: understanding that an API connection is per-model is fundamental currently10:43
rogpeppefwereade: i think it actually makes the code *harder* to understand10:44
fwereaderogpeppe, except for the connection ones, which are a bit more interesting in how they interact with one-or-maybe-many models10:44
rogpeppefwereade: because someone that starts to understand it will think "but why is this necessary?" and there's no good answer10:44
rogpeppefwereade: "the connection ones"?10:44
fwereaderogpeppe, s/connection/controller/ sorry10:45
rogpeppefwereade: sure, the controller ones are indeed not model-specific, and that's also fundamental currently.10:45
rogpeppefwereade: and to work on the API you need to have a good understanding of that10:46
rogpeppefwereade: and adding unnecessary parameters just confuses matters10:46
fwereaderogpeppe, it's not fundamental at all, it's a consequence of trying to enforce conn<->model in the first place10:46
rogpeppefwereade: we already have hideous amounts of code bloat and it seems like you *like* that10:47
rogpeppefwereade: the conn<->model thing is fundamental to our architecture10:47
fwereaderogpeppe, I am surprised that you would pick such a fight on the battleground of "the one approach that actually gets us reused code in apiserver"10:47
fwereaderogpeppe, no, it's just a lazy mistake10:47
rogpeppefwereade: it's actually not10:47
rogpeppefwereade: anyway, as i've said far too many times before, bulk calls are an expensive red herring10:51
rogpeppefwereade: not that that specifically applies in this case other than the requirement to watch several models at once.10:51
rick_h_dimitern: ping for sync?11:03
dimiternrick_h_: oops, omw11:03
rick_h_fwereade: ping when you get a sec please. Would like to sync up in https://hangouts.google.com/hangouts/_/canonical.com/rick?authuser=111:21
mupBug #1613672 opened: Nothing prevents ModelManager.DestroyModels from destroying the controller model <juju-core:New> <https://launchpad.net/bugs/1613672>11:47
mupBug #1613672 changed: Nothing prevents ModelManager.DestroyModels from destroying the controller model <juju-core:New> <https://launchpad.net/bugs/1613672>11:53
rogpeppeit's always nice when you start to create a new bug and find that not only is there an existing bug report but the fix went in a couple of hours ago :) https://bugs.launchpad.net/juju-core/+bug/159335011:54
mupBug #1593350: Juju register allows users to register alternatively named controllers of the same UUID <juju-core:Fix Committed by macgreagoir> <https://launchpad.net/bugs/1593350>11:54
mupBug #1613672 opened: Nothing prevents ModelManager.DestroyModels from destroying the controller model <juju-core:New> <https://launchpad.net/bugs/1613672>11:59
mupBug #1613688 opened: adding a controller user gives user read-write access to all models <juju-core:New> <https://launchpad.net/bugs/1613688>12:17
wallyworlddimitern: please has review of small test fix? http://reviews.vapour.ws/r/5448/13:00
wallyworldor perrito666 ? ^^^^13:19
perrito666wallyworld: checking13:20
perrito666wallyworld: shipit13:21
perrito666now go to sleep13:22
anastasiamacwallyworld: perrito666 sadi it ^^13:24
* rick_h_ goes for a coffee run13:27
voidspacefrobware: ping14:00
frobwarevoidspace: pong14:00
voidspacefrobware: what's the best ppa for maas 1.9?14:00
voidspacefrobware: for trusty14:00
rick_h_natefinch: dimitern ping for standup14:00
frobwarevoidspace: pretty sure that ppa:maas/stable14:01
voidspacefrobware: sounds good - thanks14:01
perrito666dimitern: around?14:23
dimiternperrito666: o/14:24
natefinchmgz: was it juju.environs.simplestreams that I needed at TRACE?  Don't want to run an 8 minute bootstrap to figure it out :/14:36
mgznatefinch: yeah, but let me just double check14:40
mgznatefinch: yeah, that's what I did last time, probably doesn't hurt to do juju.environs as a whole, as some things have likely changed since then14:43
natefinchahh yes, the trickle down effect14:45
mgznatefinch: remember to do --show-log or --log-file global options as well14:45
natefinchthere's tools-metadata-url and agent-metadata-url.... what's the difference?14:45
mgztools- is the old name14:46
mgzthey end up at the same place14:46
voidspacefrobware: does maas 1.9 have vlans?14:48
frobwarevoidspace: yes14:48
frobwarevoidspace: but much easier to set them up in /e/n/i before installation of maas14:48
voidspacefrobware: ah yes, they need to be in /e/n/i of the rack controller14:49
voidspaceI forgot, thanks14:49
voidspaceI've already installed maas unfortunately14:49
voidspaceI'll add it and restart maas and see if it picks it up14:49
frobwarevoidspace: it won't :(14:50
voidspacefrobware: bl**dy fecking bunghole14:51
voidspacefrobware: thanks14:51
katco[12:35]       <katco> rick_h_: ok15:15
katco            <rick_h_> katco: ty15:15
katco[12:38]         <mup> Bug #1612744 opened: MachinerSuite.TestMachinerMachineAssignedUnits agent should be terminated <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1612744>15:15
katco                <mup> Bug #1612745 opened: rackerSuite.TestGainLeadership number of ClaimLeadership is wrong <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1612745>15:15
mupBug #1612744: MachinerSuite.TestMachinerMachineAssignedUnits agent should be terminated <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1612744>15:15
mupBug #1612745: rackerSuite.TestGainLeadership number of ClaimLeadership is wrong <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1612745>15:15
katco                <mup> Bug #1612747 opened: DebugHooksServerSuite.TestRunHook signal: killed <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1612747>15:15
mupBug #1612747: DebugHooksServerSuite.TestRunHook signal: killed <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1612747>15:15
katco[12:46]             * rick_h_ gets some lunch with that done15:15
katco[13:09]    <kwmonroe> ejat: i use juju/azure15:15
katco[13:10]        <ejat> kwmonroe: okie .. its working now .. thanks15:15
katco[14:11]       <redir> bbiab15:15
katco[14:19]             * rick_h_ runs to get my truck and the boy from summer camp. have a good weekend folks15:15
katco[14:30]         <mup> Bug #1612775 opened: resource get fails, resource already staged <juju-core:New> <https://launchpad.net/bugs/1612775>15:15
katco[15:02]  <perrito666> it would be extremely nice of us to name the same thing in the same way across facades15:15
mupBug #1612775: resource get fails, resource already staged <juju-core:Triaged> <https://launchpad.net/bugs/1612775>15:15
katco              <redir> perrito666: novel idea15:15
katco[15:03]  <perrito666> authorizer and state are called something different across all facades15:15
katco[15:04]  <perrito666> and just found one without authorizer15:15
katco[15:13]   <natefinch> sinzui: are you still having problems with that windows deploy CI test?  I seem to be able to deploy a windows charm just fine on azure15:15
katco             <sinzui> natefinch: I think not. I am still wating for one to complete to say all is well15:15
katco             <sinzui> natefinch: once juju broke the reate limit. azure locked our for many hours15:15
katco          <natefinch> sinzui: great :)15:15
katco[15:14]   <natefinch> oops15:15
katco[15:19]      <sinzui> natefinch: I see some azure successes. most failed though.I suspect the failure were at the tail end of the lockout. I will retest.15:15
katco[15:27]       <katco> sinzui: hey does qa have any plans to release the qa tools as a snap?15:16
katco             <sinzui> katco: we expect to do dailies15:16
katco              <katco> sinzui: daily releases of the qa tools?15:16
rick_h_katco: ummm, howdy?15:16
katco[15:28]      <sinzui> katco: oh, no. no plans to snape jujuc-ci-tools or juju-release-tools15:16
katco              <katco> sinzui: sorry, jujuc-ci-tools15:16
perrito666gotta love emacs15:16
katco              <katco> sinzui: having just gone through the setup process, it would be awesome to just download a snap15:16
katco[15:29]      <sinzui> maybe balloons will want try it. Since snap doesn't work with windows, centos, or osx, we are dedicated to working with a common solution.15:16
katco[15:30]       <katco> sinzui: ah good point15:16
katco                <mup> Bug #1612793 opened: bootstrap azure/southcentralus is failing in 2.0-betaX <juju-core:New> <https://launchpad.net/bugs/1612793>15:16
mupBug #1612793: bootstrap azure/southcentralus is failing in 2.0-betaX <azure-provider> <juju-core:Triaged by axwalk> <https://launchpad.net/bugs/1612793>15:16
rick_h_perrito666: heh, is this the equiv of a emacs core-dump?15:16
katco[15:39]  <perrito666> ... obtained int = 215:16
katco         <perrito666> ... expected int = 215:16
katco         <perrito666> evidently15:16
katco[15:44]       <katco> yay my recator compiles for the 1st time15:16
katco              <katco> refactor15:16
katco[16:39]         <mup> Bug #1612335 changed: Azure rate limit leads to catastrophic failure for subscription <azure-provider> <bootstrap> <ci> <destroy-controller> <kill-controller> <regression> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1612335>15:16
mupBug #1612335: Azure rate limit leads to catastrophic failure for subscription <azure-provider> <bootstrap> <ci> <destroy-controller> <kill-controller> <regression> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1612335>15:16
natefinchsomeone pasted the wrong thing into their IRC window15:16
katco                <mup> Bug #1612722 changed: Juju cannot destroy models/machine, all testing substrates are exhausted <blocker> <ci> <destroy-controller> <kill-controller> <regression> <juju-core:Fix Released by rharding> <https://launchpad.net/bugs/1612722>15:16
katco[16:48]         <mup> Bug #1612836 opened: Cannot bring up hosted  model machines in azure <azure-provider> <ci> <deploy> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1612836>15:16
mupBug #1612722: Juju cannot destroy models/machine, all testing substrates are exhausted <blocker> <ci> <destroy-controller> <kill-controller> <regression> <juju-core:Fix Released by rharding> <https://launchpad.net/bugs/1612722>15:16
mupBug #1612836: Cannot bring up hosted  model machines in azure <azure-provider> <ci> <deploy> <regression> <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1612836>15:16
katco[17:14]       <katco> rick_h_: ping, don't suppose you're on15:16
katco[17:34]  <perrito666> mmm, ppas dont work nicely with my deb proxy15:16
katco[20:15]       <redir> EoW. LALters15:16
katco              <redir> laters even15:17
natefinchlast time on juju-dev....15:17
* balloons hears natefinch in a cool annoucer's voice15:17
perrito666I read him with the bsg font15:17
sinzuikatco: Since the azure provider is broken by bug 1612836 we cannot see if the windows deployment is still broken15:17
mupBug #1612836: Cannot bring up hosted  model machines in azure <azure-provider> <ci> <deploy> <regression> <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1612836>15:17
perrito666sinzui: katco left the channel15:18
* perrito666 pastes the last 100 lines to show sinzui 15:18
natefinchshe must have hit the wrong thing in emacs15:18
perrito666it all started with a /names so she most likely hit the combination for paste instead of enter15:19
* redir has flashback and blinks15:20
mgzit wasn't even a flood kick...15:22
perrito666katco: are you ok?15:22
katcowow i have no clue what happened15:22
perrito666need an exsorcism?15:22
perrito666lol, I have no clue how to write that in english I just noticed15:23
katcoi don't even know what it ended up posting to the channel... did i spam everyone or something?15:23
rick_h_katco: :) a bit15:23
perrito666katco: it was a nice recap of friday15:23
rick_h_katco: we had a rerun episode of #juju-dev, not a bad one though.15:23
katcodid it send the buffer from friday?15:23
perrito666katco: you did15:24
perrito666katco: where you trying to send the buffer from a different day?15:24
katcoi was trying to ask mgz and sinzui a question lol... let me try to paste this error message again15:24
* perrito666 cringes15:24
katcomgz: sinzui: have you seen this error before? value *errors.Err = &errors.Err{message:"", cause:(*errors.Err)(0xc82395c5f0), previous:(*errors.Err)(0xc82395c640), file:"github.com/juju/juju/cmd/juju/application/deploy.go", line:691} ("the provided bundle has the following errors:\ncharm path in application \"dummy\" does not exist: dummy")15:25
katco(sorry for the weird spam earlier all)15:25
katcomgz: sinzui: trying to figure out if this is a spurious error, i.e. a timing issue in CI where the dummy path doesn't exist before the bundle is deployed, or...?15:26
mgzkatco: new to me. looks like more local charm/deploy syntax confusion?15:26
sinzuikatco: no. do you have juju-ci-tools/repository checked out and do you gave JUJU_REPOSITORY seto the path so that the path to the charm is correct15:27
katcomgz: hm, ok. i didn't get this locally when running through the test suite. i'll try the ci suite next15:27
dimiternrick_h_: whew.. bug 1603473 updated15:27
mupBug #1603473: Relation fails as units/machines are on different subnet on a multi NIC setup, juju 2.0 beta11 <11> <2.0> <beta> <juju> <juju-core:Incomplete by dimitern> <https://launchpad.net/bugs/1603473>15:27
katcosinzui: this is from an official ci run, not me running ci locally15:27
rick_h_dimitern: ty, let's see how that goes15:28
dimiternrick_h_: can you have a look and perhaps comment, if needed?15:28
rick_h_dimitern: /me goes to look15:28
sinzuikatco: good, my thoughts remain the same.15:28
mgzkatco: github-merge-juju/8788 ? looks like a real issue with your branch I think15:28
dimiternrick_h_: cheers!15:28
natefinchmgz: thanks for the help with the logging... that helped get me past my problem (the code was expecting to be pointed one directory deeper than I realized.... if only we had documentation for this sort of thing.15:29
mgznatefinch: ace, glad it helped15:29
katcomgz: i was looking at 8785, but i'm sure that one is the same15:29
mupBug #1613476 changed: juju cli commands should be logged by juju <oil> <oil-2.0> <juju-core:Invalid> <https://launchpad.net/bugs/1613476>15:29
natefinchmgz: it's not *working* but at least I get better output and fixed that one problem15:32
admcleodbabbageclunk: hello15:33
babbageclunkadmcleod: hey!15:33
* natefinch lunches15:33
admcleodbabbageclunk: hows it going?15:33
=== natefinch is now known as natefinch-afk
babbageclunkadmcleod: yeah, good thanks - how are you? Still in Spain?15:33
admcleodbabbageclunk: not bad, yeah im still in spain but im in barcelona now...15:34
* rick_h_ grabs lunchables15:35
admcleodbabbageclunk: i think i said i was going to live in valencia. im not sure now :)15:36
babbageclunkadmcleod: who can be sure in this mad world?15:37
beisnerhi all.  ? re: manual provider.  by default, lxc units get placed behind nat on the bridge, meaning lxc units are in a bit of a bubble in terms of directly communicating with lxc units on another manual unit.  is there a juju-ism to specify transparent bridge?15:37
admcleodbabbageclunk: i just dont know. which is a great segue into my question about juju lxc networking with manual provider15:37
admcleodbabbageclunk: which is also beisers question15:37
* beisner yields the floor :)15:37
katcomgz: sinzui: thanks, i can reproduce locally15:40
admcleodbeisner: was it something i said?15:45
beisneradmcleod, ha!15:46
rick_h_admcleod: beisner i'm not aware of one atm. file a bug and i'll see if the network experts have any thoughts. it might requure some juju-run-fu15:48
rick_h_admcleod: beisner more complicated setups require the provider worm (such as maas has gotten) so the manual provider hasn't gotten complex network love15:50
rick_h_admcleod: beisner there's a todo for using the fan on public clouds for container coms, worth adding manual to that list it seems.15:51
admcleodrick_h_: ah right, thanks15:54
beisnerrick_h_, thanks, we'll push on with a few ideas and raise a bug if it looks like something outside the expected behavior.15:55
frobwarevoidspace: ping - do you still have problems at the commissioning stage with my scripts?16:03
frobwareso it turns out my first snap is never going to work... :(16:21
* frobware one square to back...16:22
mupBug #1613782 opened: windows cannot bootstrap trusty <bootstrap> <ci> <jujuqa> <regression> <windows> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613782>16:26
* frobware has exactly this problem, but for accessing .maascli.db: https://www.mail-archive.com/snapcraft@lists.ubuntu.com/msg00345.html16:26
=== natefinch-afk is now known as natefinch
natefinchanyone know what this means? sinzui, mgz: 2016-08-16 15:25:49 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "https://s3.amazonaws.com/24b99da448314945835933a0c8edffb3/tools/streams/v1/index2.json" because of missing information: "content-download" data not found16:56
mupBug #1613804 opened: mongod sigabrt <intermittent-failure> <test-failure> <windows> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613804>16:56
mgzabentley: ^any ideas on natefinch's message?17:04
natefinchI see  "datatype": "content-download" under index": {  "com.ubuntu.juju:released:tools": {17:06
abentleynatefinch: I can tell you that LXD and MAAS use "content-download" and AWS uses "image-id".17:06
natefinchabentley: what does azure use?  Because that's where I'm trying to deploy17:07
natefinchnote this is streams for tools, not images17:07
abentleynatefinch: Oh, sorry.  For tools, everything uses content-download.17:08
abentleynatefinch: I see that this index has only the released streams.  Are you testing a beta?17:11
natefinchabentley: I'm testing my own code17:11
sinzuinatefinch: abentley is suggesting that you need to decale the agents to be devel17:12
natefinchman, I really wish I could just juju upload-tools win2012hvr=/home/nate/bin/jujud.exe17:13
abentleynatefinch: What sinzui said.  You'll need devel streams or to set agent-stream=released for this to work.17:13
sinzuinatefinch: I think the call to juju metdata generate-agents needs an arg to make agent stanzas17:13
* sinzui though juju refused to honour agent-stream=released when the version has letters in it17:13
* natefinch is gonna have to document the crap outta this17:17
sinzuinatefinch: I think the command is17:17
sinzuiuju metadata generate-tools -d <workingdir> --stream17:17
sinzuinatefinch: and we can detect a letter in the version to switch from released to devel. I can fix that now17:17
mupBug #1613823 opened: Google App Engine IP is ephemeral by default  <juju-core:New> <https://launchpad.net/bugs/1613823>17:17
mupBug #1613824 opened: cmdSubnetSuite teardown fails: unexpected message <ci> <intermittent-failure> <jujuqa> <unit-tests> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613824>17:17
natefinchsinzui: I was just using the streams-from-local script17:18
sinzuinatefinch: yeah, that is the script I ma going to change17:18
natefinchsinzui: ok.  yes, it would be nice if what it output worked by default with the ci scripts that it is right next to.17:18
sinzuinatefinch: the script is checking for alpha or beta in the version and setting devel. What version is yoru juju?17:24
natefinchsinzui: beta.. whatever master is17:24
sinzuinatefinch: in your stream directory, do you have a sir named "tools/evel"17:25
sinzuinatefinch: in your stream directory, do you have a sir named "tools/devel"17:26
natefinchsinzui: no, just released.  I'm currently rerunning with agent-stream=released17:27
sinzuinatefinch: is your system juju 1.25.6?17:28
sinzuinatefinch: well that wouldn't matter becuase the script runs jujud --version17:28
sinzuinatefinch: found the problem.17:29
mupBug #1613829 opened: TargetDiskBlobAlreadyExists on deploy (Azure) <azure-provider> <ci> <deploy> <intermittent-failure> <jujuqa> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613829>17:32
mgzrick_h_: got a slot for a pre-imp call some point in the next few hours?17:34
rick_h_mgz: sure thing, getting off the phone shortly17:35
rick_h_mgz: free when you are17:35
dimiternanybody up for a short review ? http://reviews.vapour.ws/r/5449/ - fixes bug 161262417:41
mupBug #1612624: Bootstrap fail on MAAS if ipv6 is disabled <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1612624>17:41
mupBug #1613829 changed: TargetDiskBlobAlreadyExists on deploy (Azure) <azure-provider> <ci> <deploy> <intermittent-failure> <jujuqa> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613829>17:41
* dimitern eod-s17:42
natefinchsinzui: does --agent-stream work?  It seems like it's being ignored: http://pastebin.ubuntu.com/23062301/17:45
sinzuinatefinch: that is what I muttered about above. wallyworld told me he added a rule that a devel version would never use  released streams. in CI tests, we create devel streams17:46
natefinchsinzui: I guess I can just edit my tools to s/released/devel17:46
mupBug #1613829 opened: TargetDiskBlobAlreadyExists on deploy (Azure) <azure-provider> <ci> <deploy> <intermittent-failure> <jujuqa> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613829>17:47
sinzuinatefinch: you. can. my change to parse the "juju version" does pass the riight args, but juju then flips out because it seems to alway require a released stream too17:48
=== redelmann_ is now known as redelmann
sinzuinatefinch: generate-tools refuse to make devel streams :( you only need to change "released" ot "devel" in index2.json to move forward. I am going to explore this approach for the script17:58
natefinchsinzui: this is like some greek tragedy17:58
natefinchsinzui: I can edit easily enough, will retry once I fix it in s317:59
natefinchgah, this stuff is a wall of text18:08
mupBug #1613838 opened: TestBootstrap on arm/ppc64el/s390 cannot build agent binary <arm64> <bootstrap> <ci> <jujuqa> <ppc64el> <regression> <s390x> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613838>18:18
mupBug #1613838 changed: TestBootstrap on arm/ppc64el/s390 cannot build agent binary <arm64> <bootstrap> <ci> <jujuqa> <ppc64el> <regression> <s390x> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613838>18:21
natefinchsinzui: do you understand what's wrong here? ^  I don't know what the streams logging is supposed to look like, so it's hard for me to understand what's wrong18:22
natefinchit looks like one of the products is com.ubuntu.juju:win2012r2:amd64  and one of the candidates is the same18:23
natefinchunfortunately someone printed out with %v not %#v so it's hard to map the textual output to the fields in the struct that it's printing18:24
sinzuinatefinch: streams/v1/com.ubuntu.juju-devel-tools.json is listing agents we didn't even include in streams. this is crazy18:25
natefinchwhat's weird is that this log line is after the code has supposedly stripped out all non-matching candidates, yet clearly there are still non-matching ones, like com.ubuntu.juju:12.04:amd6418:26
natefinchand in fact, the next line just chooses the first matching one, which means it is choosing com.ubuntu.juju:12.04:amd6418:27
natefinchnot that it tells us that...18:27
natefinchin the logs18:27
mupBug #1613838 opened: TestBootstrap on arm/ppc64el/s390 cannot build agent binary <arm64> <bootstrap> <ci> <jujuqa> <ppc64el> <regression> <s390x> <juju-core:Triaged by rharding> <https://launchpad.net/bugs/1613838>18:33
mupBug #1613839 opened: bundle deployment does not accept "1.0" as valid float <juju-core:New> <https://launchpad.net/bugs/1613839>18:33
sinzuinatefinch: generate-tools is completely busted. It never includes the agents we asked to be added. this proabbly because 1.x is not allowed to know about 2.x. I amd going to switch system jujus and see if 2.x can make streams for 2.x18:33
natefinchsinzui: k18:34
bleepbloopI did a reboot of my juju state nodes and when they restarted I lost the ability to run `juju --debug status` and now get an error `health ping failed: connection is shut down` and in my `juju debug-log` I am getting ` cannot publish API server addresses: cannot set API addresses: state changing too quickly; try again soon`. I installed the version that came18:36
bleepbloopwith xenial which is 2.0 beta12, what is weird is everything else seems to still be working, it is just freaking out when juju status is run18:36
bleepbloopHopefully this is the right place to ask since the issue is with the beta version18:36
mupBug #1613839 changed: bundle deployment does not accept "1.0" as valid float <juju-core:New> <https://launchpad.net/bugs/1613839>18:39
natefinchbleepbloop: we encountered something like this.. I think it's been fixed since that beta18:44
natefinchbleepbloop: hmm.. the one I'm thinking of was fixed before beta 1218:47
mupBug #1613839 opened: bundle deployment does not accept "1.0" as valid float <juju-core:New> <https://launchpad.net/bugs/1613839>18:48
bleepbloop@natefinch Hmm okay, I'm also seeing a `pinger failed: E11000 duplicate key error collection: presence.presence.pings`, could that be related to the `health ping failed` error?18:51
meetingologybleepbloop: Error: "natefinch" is not a valid command.18:51
bleepbloopOops meant natefinch:, been using slack too much :P18:51
natefinchbleepbloop: I think it's very similar to the other bug that we fixed, maybe this is another instance of it nearby in the code.  It's an error from juju's mongo driver.  Basically, mongo is changing stuff out from under us because ¯\_(ツ)_/¯ it's mongo18:53
mupBug #1613842 opened: [Feature] Juju Resources should also support fingerprint checking <juju-core:New> <https://launchpad.net/bugs/1613842>18:57
natefinchbleepbloop: the pinger thing I'm guessing is a different problem... but not 100% sure.18:57
bleepbloopHmm okay, is there a good way to workaround it for now? I just tried to upgrade to beta15 to see if that would help and ended up with a "ERROR unknown object type "ModelConfig" (not implemented)" so that seems to not be the way to go :/18:58
natefinchsinzui: we don't officially support upgrading from beta to beta, do we?18:59
sinzuinatefinch: no we don't19:00
natefinchbleepbloop: I understand this is the version that comes with Xenial... it's unfortunately still in beta due to, well, a bunch of reasons.19:00
natefinchbleepbloop: rebooting may help.  It generally only happens then the machine is under load19:03
natefinchbleepbloop: it may help to simply not reboot and leave it alone for a while, too.  Upgrade sometimes can work and sometimes doesn't... seems like you're hitting one of the "it doesn't"19:03
natefinchbleepbloop: I filed a bug for it - https://bugs.launchpad.net/juju-core/+bug/1613855  It'll get adressed soon, but that won't help fix your current environment, unfortunately.19:08
mupBug #1613855: SetAPIHostPorts runs afoul of "state changing too quickly" <juju-core:New> <https://launchpad.net/bugs/1613855>19:08
bleepbloopnatefinch: Okay, no problem I rolled back from the ppa, it seems to have not made things any worse so thats good at least, would it help if I  included the full logs for the error?19:14
mupBug #1613855 opened: SetAPIHostPorts runs afoul of "state changing too quickly" <juju-core:New> <https://launchpad.net/bugs/1613855>19:18
bleepbloopAlso just found some more fun `panic: runtime error: index out of range` seems to be happening on all of the nodes (servers are running in HA)19:19
natefinchbleepbloop: ew19:20
natefinchbleepbloop:  you may be in a bad state if you tried to upgrade and it failed halfway through.  Still, I wouldn't expect to see a panic like that.19:22
bleepbloopnatefinch: I don't think the panics are due to my upgrade attempt, the panics all happened 2-3 hours ago where all I'd done is rebooted the machines19:23
natefinchahh hmm19:23
natefinchcan you pastebin me one of them?19:23
bleepbloopsure no problem, one second19:25
bleepbloopnatefinch: http://pastebin.com/UKeD4msg it happens 3 times in the included log19:36
bleepbloopnatefinch: if you search for "panic: runtime error" you'll find it19:37
mupBug #1613858 opened: generate-tools ignores 2.x agents and creates bad path <jujuqa> <metadata> <regression> <streams> <juju-core:Triaged> <https://launchpad.net/bugs/1613858>19:42
natefinchbleepbloop: looks like you stumbled on another bug19:44
natefinchbleepbloop: yet another notch on your belt - https://bugs.launchpad.net/juju-core/+bug/1613866  Sorry you're running into these problems.  Not really sure why we haven't seen this one more often... maybe something special in the timing in your system.19:51
mupBug #1613866: server panic during juju status if no status history for a unit <juju-core:New> <https://launchpad.net/bugs/1613866>19:51
mupBug #1613864 opened: Missing "juju-2"/"juju2" command. <landscape> <juju-core:New> <https://launchpad.net/bugs/1613864>19:51
mupBug #1613866 opened: server panic during juju status if no status history for a unit <juju-core:New> <https://launchpad.net/bugs/1613866>19:51
natefinchrick_h_: this whole streams debacle is just crazy19:53
mupBug #1613864 changed: Missing "juju-2"/"juju2" command. <landscape> <juju-core:New> <https://launchpad.net/bugs/1613864>19:54
mupBug #1613866 changed: server panic during juju status if no status history for a unit <juju-core:New> <https://launchpad.net/bugs/1613866>19:54
natefinchrick_h_: it's very obvious that we need much better UX around setting custom streams URLs if we actually expect anyone to ever do it, not to mention much better facilities for debugging problems with it.  And that's not to mention the fact that we have a showstopped bug with it.19:55
mupBug #1613864 opened: Missing "juju-2"/"juju2" command. <landscape> <juju-core:New> <https://launchpad.net/bugs/1613864>19:57
mupBug #1613866 opened: server panic during juju status if no status history for a unit <juju-core:New> <https://launchpad.net/bugs/1613866>19:57
mupBug #1613864 changed: Missing "juju-2"/"juju2" command. <landscape> <juju-core:New> <https://launchpad.net/bugs/1613864>20:03
mupBug #1613866 changed: server panic during juju status if no status history for a unit <juju-core:New> <https://launchpad.net/bugs/1613866>20:03
natefinchthumper: Let me know what you think of my assessment of this bug: https://launchpad.net/bugs/161385520:04
mupBug #1613855: SetAPIHostPorts runs afoul of "state changing too quickly" <juju-core:New> <https://launchpad.net/bugs/1613855>20:04
natefinch(sorry to jump on you :)20:04
natefinchmorning :D20:05
thumpernatefinch: yep, agree with your analysis20:05
thumperneeds to be fixed asap20:05
thumpernatefinch: I've marked it critical20:07
rick_h_natefinch: don't disagree. It's why wallyworld's spent time and things trying to find ways to work w/o streams20:08
mupBug #1613864 opened: Missing "juju-2"/"juju2" command. <landscape> <juju-core:New> <https://launchpad.net/bugs/1613864>20:12
mupBug #1613866 opened: server panic during juju status if no status history for a unit <juju-core:New> <https://launchpad.net/bugs/1613866>20:12
* natefinch writes his own streams data by hand20:22
bleepbloopSo as a last ditch effort and to curb my own curiousity, are there more up to date instructions with exactly how to connect to the mongodb database directly?20:24
rick_h_natefinch: what happened with kicking off a CI run, pausing it, and investigating that? I know you were trying to do that the other day and didn't hear why that fell over?20:24
rick_h_bleepbloop: hmm, someone had a tool for helping with that. katco?20:24
katcorick_h_: a tool for pausing ci?20:25
natefinchrick_h_: there was some problem where the new code was not available yet.  but maybe that's no longer the case.  sinzui?20:25
rick_h_katco: no, for connecting to mongodb20:25
rick_h_katco: maybe I was wrong it wasn't you. /me is trying to search email for it20:25
katcorick_h_: oh, right... i sent a script out to the email list20:25
katconatefinch: rick_h_: check juju-dev20:25
rick_h_katco: right, that's what I was thinking of. bleepbloop is asking about that and maybe the list email would be a useful start for bleepbloop20:25
katcobleepbloop: ok, one sec20:26
rick_h_katco: july 27th, found the email20:26
katcobleepbloop: https://lists.ubuntu.com/archives/juju-dev/2016-July/005772.html20:26
katcobleepbloop: give that a try. fairly easy to understand20:26
rick_h_ty katco20:27
katcorick_h_: hth20:27
natefinchsinzui: is CI able to run that windows test I've been trying to run, and if so, could we run it with --keep-env so I can investigate it?20:29
=== mup_ is now known as mup
cmarsare there docs, on how to test with clocks ?20:41
katcowhat's the testing suite that allows me to create temporary files/directories?20:42
natefinchkatco: c.Mkdir()20:42
natefinchkatco: not a suite20:42
katconatefinch: ta20:43
=== natefinch is now known as natefinch-afk
mupBug #1587644 changed: jujud and mongo cpu/ram usage spike <canonical-bootstack> <canonical-is> <juju-core:Fix Released> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1587644>21:04
mupBug #1612653 changed: lxd fails in beta15 when installing via snap <juju-core:Triaged> <https://launchpad.net/bugs/1612653>21:04
mupBug #1613882 opened: stateSuite.SetUpTest mongo is unreachable <ci> <intermittent-failure> <regression> <unit-tests> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1613882>21:04
bleepbloopThank you all for your help! Last question, how well does 1.25 work on xenial? Are there any compatibility issues or should it be fully compatible?21:15
perrito666bleepbloop: I don't think 1.25 works on xenial21:17
rick_h_natefinch-afk: this has your name on it, can you update/create a card/etc please? https://bugs.launchpad.net/juju-core/+bug/155227421:18
mupBug #1552274: juju list-credentials inconsistencies between format output <2.0-count> <bitesize> <conjure> <juju-release-support> <rc1> <usability> <juju-core:In Progress by natefinch> <https://launchpad.net/bugs/1552274>21:18
bleepbloopI see there are xenial packages for 1.25, is there anything specific that doesn't work?21:25
perrito666bleepbloop: it was an educated guess, I am not sure the right mongo package is built for xenial21:30
perrito666but then again, I might be wrong21:30
perrito666rick_h_: do you know?21:30
rick_h_bleepbloop: perrito666 nothing that I'm aware of? I mean it works?21:33
perrito666rick_h_: it means it works21:33
mupBug #1613888 opened: Cannot destroy application (contention) shortly after unblocking it <ci> <intermittent-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1613888>21:49
* rick_h_ runs for the night21:59
* redir hopes rick_h_ catches the night.22:26
mupBug #1613892 opened: juju "show" flags are inconsistent  <juju-core:New> <https://launchpad.net/bugs/1613892>22:28
perrito666Will be 5 late to standup23:09
axwwallyworld: this needs another review if you would later: http://reviews.vapour.ws/r/5447/23:36
perrito666wallyworld: you just dropped23:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!