rediri'll be around for a while00:07
menn0thumper: no right now I'm not. I'm working on this mgo dup key issue00:07
menn0thumper: I can help if needed though00:07
redirthumper: menn0 I'll be around for another 90 minutes or so00:08
redirso no huge rush00:08
menn0redir: should I have a quick look at it with you now while you're still around?00:08
redirmenn0: if you are at a good stopping point sure. If not let me know when you get to one.00:09
menn0redir: now is good00:09
menn0redir: hangout or IRC?00:10
menn0ok, let's just use the tanzanite-standup one00:10
menn0redir: ^00:10
wallyworldaxw: do you have a moment for a quick chat?01:00
axwwallyworld: sure01:00
wallyworldstandup wil do01:00
axwwallyworld: very hard to hear you01:03
wallyworldaxw: i'll reconnect01:03
thumperredir: is that race sorted?01:45
mupBug #1604919 changed: juju-status stuck in pending on win2012hvr2 deployment <oil> <oil-2.0> <juju-core:New> <https://launchpad.net/bugs/1604919>01:48
redirmenn0: https://github.com/reedobrien/mgo/tree/fix-race-in-stats02:11
redirmenn0: I can make a PR if you want. But if you think it needs more polish... go ahead.02:12
=== natefinch-afk is now known as natefinch
redirmenn0: same results on the tests with or without the patch.02:12
redirthumper: ^^02:14
thumpermenn0: http://reviews.vapour.ws/r/5284/diff/#02:16
thumpermenn0: I think you fixed the original, this is 1.25 now02:16
thumperaxw: oh, you are oncall reviewer02:16
thumperaxw: you could look above if you like02:17
mupBug #1458585 changed: SSHGoCryptoCommandSuite.TestCommand fails <blocker> <ci> <go1.6> <intermittent-failure> <regression> <test-failure> <wily> <xenial> <juju-core:Invalid> <juju-core 1.23:Won't Fix> <juju-core 1.25:In Progress by thumper> <https://launchpad.net/bugs/1458585>02:18
axwthumper: sure, looking02:20
axwthumper: LGTM02:21
thumperaxw: ta02:21
thumperaxw,wallyworld: isn't status history updated outside of transactions?02:24
thumperso raw access?02:25
axwthumper: I think so?02:25
wallyworldthumper: i thought so02:25
thumperallcollections doesn't mark it as rawAccess02:25
thumperI'm looking at the bug about big transactions to clean up02:25
* thumper looks02:26
wallyworldmaybe the collection is accessed directly without txn ops02:26
perrito666it is accessed without txn ops02:26
perrito666or was last time I looked02:26
thumperhmm... we probably want to mark it as so otherwise removing the items uses transactions :)02:27
thumperI'll do it02:27
menn0thumper: sorry, was lunching... looking02:31
menn0thumper: looks good, although I don't think I fixed this in master (at least I don't remember doing it)02:33
thumperI compared the patch with the utils package02:34
thumpermenn0: axw has reviewed anyway02:34
natefinchIs add-cloud intended to work with the static clouds, for example, ec2?  (it does, I just am wondering if it's *supposed to*)02:36
wallyworldnatefinch: it is supposed to allow you to add your own config02:38
wallyworldthumper: so getRawCollection() - does that not work if collection is not marked as raw?02:38
wallyworldthst sorta sucks a bit02:38
thumperwallyworld: it does02:38
natefinchwallyworld: ok, I had assumed it was only for things like manual, openstack, and maas, where you can actually create your own cloud02:39
thumperwallyworld: but we have code that iterate over the all collections02:39
thumperand do different things for raw02:39
wallyworldah ok02:39
wallyworldnatefinch: private cloud set up is one use case for sure02:39
natefinchwallyworld: just wondering, since I'm working on the interactive add-cloud02:40
wallyworlddo the private cloud first, and then the public cloud config stuff can be done later - i'm prerry sure the doc references it02:41
wallyworldone issue is that we need to enumerate valid config items to make user input sensible02:41
wallyworldand there's not really a clear path to do that yet02:41
natefinchwallyworld:  yeah... right now I'm just hard-coding it per cloud.02:43
wallyworldthe valid config items?02:44
natefinchwallyworld: what values to ask the user for (which map to config values)02:44
wallyworldwe should be able to use the config schema definition02:44
wallyworldjust as is done when parsing config.yaml02:44
wallyworldbut use it to get at the valid item names and types02:45
natefinchI'm not sure I can use the schema to make a user-friendly UI.02:46
wallyworldreally? it has the attribute names, their types, whether they are mandatory etc02:47
natefinchwhat might be a good compromise is to put the interactive add-cloud code inside each provider, so it lives with the code that knows what values to ask about.02:47
natefinchwallyworld: the key word there is user-friendly02:48
wallyworldhow else would it be done if not using the schema?02:48
natefinchwallyworld: by having a human write sentences that humans can understand02:48
wallyworldhuh? for name - value input?02:48
natefinchwallyworld: this is for the interactive add-cloud.  I'm not going to just prompt them with the yaml field names.02:49
wallyworldthat's what is used now in config.yaml files, and on the CLI as --config args02:50
natefinchyes, and it's awful, and that's why we're writing an interactive mode.02:50
wallyworldwhat's wrong with allowing them to enter a value for "http-proxy" say02:50
wallyworldthat is entirely readable02:51
wallyworldyou'd use tab completion to help too02:51
wallyworldand print the description of each field also02:51
natefinchbecause if we write it by hand, it'll be clean, crisp, and tasteful. If we auto-generate it, it'll feel auto-generated, it won't be able to do intelligent verification (like checking that a URL is a valid URL)...02:55
wallyworldwhy can't we check a valid url? of course we can02:57
natefinchthe schema doesn't say it's a url, it just says it's a string.... I'm sure we check it later on sometime, but it's one of those things where you'll enter 10 fields, then it'll go off and run through a verification, and spit back that field #7 is incorrect02:58
wallyworldwhether it is done off a schema or we hard code prompts, the UX is the same02:58
natefinchI guarantee you, the UX will not be the same if it's hand coded.02:59
wallyworldthe schema does need to say it's a url, yes. we can do that02:59
natefinchThat's just one example02:59
wallyworldwhy won't it be the same?02:59
wallyworldit's printing a prompt to enter a value02:59
wallyworldin either case02:59
natefinchyes, but the prompt is quite likely not to be exactly the same as the description of the yaml field03:00
wallyworldso we fix our metadata03:00
natefinchalso, there's intelligence that is outside the schema... like the fact that openstack username and password only apply if the auth-mode is user-pass03:00
wallyworldhow do we handle the case where the config schema gains a new value?03:00
wallyworldwe have already talked about the need to make the schema support grouping and conditional fields03:01
natefinchmaking the schema more complicated is not the answer03:01
natefinchit's ok to write code03:01
wallyworldit's not ok to create something that is hard to maintain03:02
wallyworldand prone to skew03:02
wallyworldschemas are the common solution for user data entry. our gui uses json schema for example03:02
natefinchwe'll spend days, weeks, adding gobs of complexity to the schema just to avoid the simplest of imperative code.03:04
wallyworldit's not gobs of complexity03:05
natefinchevery time we have to do an if/else... that's significant added complexity and cognitive overhead to the schema... when it's completely trivial in imperative code.03:05
natefinchhow do you signify that openstack user and password only apply if auth-mode is userpass?03:06
wallyworldusing a directive in the schema - that's what schemas are for03:06
natefinchhow do you do it?03:06
* thumper taps his fingers while the state tests run03:07
thumperperhaps I should go and clean the coffee machine03:07
wallyworldi haven't got the definitive refence handy03:07
wallyworldbut we are talking about bog standard schema features03:08
thumperoh ffs03:09
thumperstate tests take over 10 minutes03:09
natefinchthat's jsonschema... the provider config schema is our own custom thing, isn't it? gopkg.in/juju/environschema.v103:09
wallyworldif we hard code it, who will take the lead on ensuring any config additon or change to the schema will be reflected in add-cloud?  that sort of double handing and non-DRY leads to bugs03:09
thumpertime go test -check.v -test.timeout=1200s -cover03:10
wallyworldyes it is our own, but trivial to enhance03:10
thumperlets see what that shows us03:10
thumpernow I'm definitely going to clean the coffee machine03:10
natefinchwallyworld: I was actually thinking that the interactive code for each provider should live in the provider. Then it lives right next to the configuration, and so it's easy to check that any changes to the config have corresponding changes in the interactive code.03:11
wallyworldbut a lot of the base config is defined outisde of the provider in environs/config03:11
natefinchsure, if and when we decide to automate those parts, the common interactive bits for those parts can live next to the common config definitions.  The other nice thing about just writing code is that it's easy to pick a sensible subset of things to ask the user, rather than giving them 50 prompts, when 48 of those are optional.03:14
wallyworldthe optional schema attributes would not be prompted for - the code can use the is-optional property of the schema to drive that03:16
natefinchwe're getting down a rat hole.  My initial statement is still the same.  We'll have to add a ton of complexity to our schemas (and I *do not* believe that adding conditional dependencies to our schema is a small amount of work), all to save a very small and easily maintained bit of code.  And every time we have some other custom field that works in some custom way, we'll have to extend our schema in new and interesting ways and it'll becomes a03:20
natefinchmaintenance burden of its own.  Also to avoid writing self-contained, very simple imperative code that lives near the configuration values it is intended to fill.03:20
wallyworldi disagree, as do every other project that uses a schemma to drive user input03:21
thumpercoverage: 85.3% of statements03:21
thumperok  github.com/juju/juju/state617.074s03:21
natefinchwallyworld: you can generate a functional UI, you can't generate a *good* one.  And I bet I can write all the UIs for all our private clouds in less time than I could write the update to our schema to just support openstack.03:24
wallyworldsure, but at the cost of maintability. it's always easy to bang out bespoke code03:25
wallyworldopenstack and aws both have different credential types03:26
thumperaxw: http://reviews.vapour.ws/r/5285/03:26
natefinchit's not complicated code.  Print "What auth mode would you like (options..):    if authmode == openstack.userpass {  Print "What username?"  }  etc.03:27
wallyworldand that can be trivially driven by a schema03:28
wallyworldto produce the exact same output03:28
wallyworldbut will be maintainable03:29
natefinchI already find our usage of schemas to be hard to understand.... adding more complexity is just going to make that worse.03:30
natefinchto be fair, I think that has less to do with the schema itself and more our usage of it03:32
wallyworldseems like you are conflating a schema driven approach with not understanding our schema03:32
natefinchwell, no.03:34
axwthumper: reviewed03:34
natefinchwallyworld: I'm trying to avoid writing a framework, and just write what we need.  Do the simplest thing that could possibly work.  If I'm some newbie juju dev, and I add a config value to some provider, and it needs some custom logic in the prompt, it's trivial to just write that code.  If instead I have to go try to figure out how to bend the schema to work, or, more likely, add logic to the schema, it's going to take me an order of magnitude03:41
natefinchlonger.  We haven't even started and we've already mentioned two additions we would have to make for the schema.03:41
wallyworldwe can disagree :-)03:43
natefinchsure, but we gotta pick one :)03:44
axwnatefinch: the reason for using a schema is because you may not have acccess to the provider code at all times03:44
axwnatefinch: e.g. the GUI, or an older juju client03:45
natefinchwell, the older client wouldn't have the new provider code anyway, so it wouldn't know what to do with the information even if it could request it.03:50
axwnatefinch: why not? the client doesn't talk to the cloud directly, except for when it's bootstrapping or destroying the controller03:51
axwnatefinch: sorry, you're specifically looking at bootstrap aren't you. credentials are bigger than just bootstrap03:53
natefinchaxw: currently looking at add-cloud03:53
axwnatefinch: it should be possible to add credentials for a cloud after bootstrap also03:53
thumperaxw: removed the retry, and cleaned up03:55
thumperaxw: the undertaker will restart if there is an error03:55
thumperand the other uses are in migration03:55
thumperand I can work with menn0 to make sure those are retried if necessary03:56
axwthumper: thanks03:59
axwthumper: LGTM with comment fix03:59
thumperaxw: ta03:59
* thumper waits for test run to complete03:59
thumperPASS: allwatcher_internal_test.go:940: allWatcherStateSuite.TestStateWatcherTwoModels305.009s04:01
thumperthis test seems excessively long04:01
natefinchaxw, was that 6 minutes for one test?04:01
axwnatefinch: I guess you mean thumper04:02
thumpernatefinch: yes04:02
natefinchaxw: oops, sorry, yes04:02
thumper11.5 minutes for the whole package04:02
natefinchthumper: good lord04:02
thumper5 minutes for that one04:03
natefincher yes, 5, bad math penalty.04:03
* anastasiamac impressed with thumper's machine - my machine panics as off late while running all tests in state package 04:05
mupBug #1605050 opened: Controller doesn't use tools uploaded with juju bootstrap --upload-tools <juju-core:New> <https://launchpad.net/bugs/1605050>04:12
thumperanastasiamac: just provide the timeout option04:15
thumpermine takes > 10 minutes too04:15
wallyworldthumper: can you recall - what is the error if you try and deploy to a dying model04:15
thumperI don't recall04:16
anastasiamacthumper: \o/ yep, test.timeout is super-useful :D04:16
anastasiamacthumper: tyvm04:16
* thumper is looking at this very slow test04:29
thumpermenn0: got a minute?04:33
menn0thumper: give me 30s04:33
thumperI wonder if this fix is safe04:36
thumper305s -> 6s04:36
menn0thumper: ok, what's up?04:37
thumper1:1 ?04:37
menn0thumper: ok back again04:49
mupBug #1605057 opened: allWatcherStateSuite.TestStateWatcherTwoModels takes > 5 mintues <tech-debt> <unit-tests> <juju-core:In Progress by thumper> <https://launchpad.net/bugs/1605057>05:12
thumpertest win of the day05:16
thumpermenn0: http://reviews.vapour.ws/r/5286/05:17
menn0thumper: looking05:17
mupBug #1605057 changed: allWatcherStateSuite.TestStateWatcherTwoModels takes > 5 mintues <tech-debt> <unit-tests> <juju-core:In Progress by thumper> <https://launchpad.net/bugs/1605057>05:21
thumpermenn0: if you have a txn.Op with an assertion but no Insert/Remove etc, and it is the only op, what happens if the assertion fails?05:22
thumperI mean as a []txn.Op05:22
thumperbut it is the only one05:22
menn0thumper: then an ErrAborted is returned (I guess)05:24
wallyworldaxw: here's a fix for that deleted model bug http://reviews.vapour.ws/r/528705:24
menn0thumper: and we should be careful with this kind of thing. remember it seems to encourage mgo/txn to create runaway txn-queue fields05:24
mupBug #1605057 opened: allWatcherStateSuite.TestStateWatcherTwoModels takes > 5 mintues <tech-debt> <unit-tests> <juju-core:In Progress by thumper> <https://launchpad.net/bugs/1605057>05:24
thumpermenn0: ack, I'll do something else05:25
menn0thumper: review done. looks good. just a couple of minor suggestions05:25
axwwallyworld: reviewed05:34
wallyworldaxw: i was using the "Word of God", but yours is better :-)05:38
axwwallyworld: lol05:39
wallyworldaxw: also, william has indicated in the past that he hates a generic notfound code used everywhere05:39
wallyworldbut i can ask again05:39
* thumper is done for the day05:40
thumperlaters folks05:40
=== akhavr1 is now known as akhavr
=== frankban|afk is now known as frankban
mupBug #1605096 opened: Juju 2.0 Resource Error - sha mismatch error while re-deploying the charm <juju-core:New> <https://launchpad.net/bugs/1605096>07:34
babbageclunkmenn0: Nice spotting on the upload tools problem!08:10
wallyworldfwereade: hey, i have a small PR which uses a new bespoke error code for a model which is no longer there (ie model not found). andrew looked but wanted your +1 on using the new error code. i'm sure we've talked in the past and not just using generic notfound everywhere is preferred. could you take a look? http://reviews.vapour.ws/r/5287/08:18
wallyworldi need to land this for beta13 so if you +1 could you hit $$JFDI$$ for me as i need to head to soccer for a bit08:19
menn0babbageclunk: just finishing en email to you with a few bits08:33
fwereadewallyworld, LGTM, I left a trivial issue open but I'm happy to land it08:33
babbageclunkmenn0: Cool cool08:33
menn0babbageclunk: sent!08:34
* menn0 has to run for now. will check in again later08:34
babbageclunkmenn0: kthxbye08:37
mgzha... we're not actually using the uploaded tools? well, that explains some of the oddness.08:37
=== akhavr1 is now known as akhavr
=== akhavr1 is now known as akhavr
=== akhavr1 is now known as akhavr
* fwereade is trying to arrange for the split water main outside the house to be dealt with, might not make the meeting09:57
babbageclunkOh no, I totally missed the meeting! I got the 10-minute reminder but was wrestling with a patch and forgot about it by 11. :(10:34
menn0babbageclunk: back again for a bit10:57
babbageclunkmenn0: oh hai10:57
babbageclunkmenn0: Replied to your message.10:57
menn0babbageclunk: just reading that now10:58
menn0babbageclunk: your modification LGTM. it's a little more correct that way isn't it.10:59
menn0babbageclunk: does all look well when you run this under your test?11:01
wallyworldfwereade: thanks for +1, back from soccer, will land after eating11:03
babbageclunkmenn0: So, for the logging were you suggesting a sync.Once living at global scope, and then a call to Do from Query.Apply?11:23
menn0babbageclunk: or newSession or something? you don't want it too early, certainly not at import time, because logging may not be set up properly yet.11:25
babbageclunkmenn0: Ok, makes sense - and the key thing is that the once lives at package scope so that it doesn't log once per session.11:26
menn0babbageclunk: yep exactly11:26
stokachubabbageclunk: menn0: 10 runs in, no errors yet11:38
babbageclunkstokachu: Awesome!11:38
menn0stokachu: ok great.11:38
menn0stokachu: please give it a bit more testing if you can11:38
stokachuim going to let them run about 4 hours11:38
menn0That should do it :)11:39
stokachuill post my findings before lunch11:39
babbageclunkmenn0: Do we set mgo's logger in juju? I can't see a SetLogger call anywhere.11:42
menn0babbageclunk: I don't think you want to use mgo's logging infrastructure in this case11:42
menn0import loggo, create a logger with loggo.GetLogger and use that11:42
menn0that'll feed straight into juju's logging system11:43
babbageclunkmenn0: Ok, sounds good.11:43
=== akhavr1 is now known as akhavr
babbageclunkmenn0: Probably doesn't matter, but: logging at info level? Questionable but means we can be sure it we'd see it without the reporter having to change the log level.12:06
babbageclunkmenn0: https://github.com/babbageclunk/mgo/commit/17abfbbc91fbb6a37e682946a8d423b1f7c4633d12:08
menn0babbageclunk: I thought the default log level was WARNING, not sure though12:08
babbageclunkmenn0: In Juju?12:08
menn0It starts at DEBUG and then at some point during agent startup the model's logging config is applied12:09
menn0if there hasn't been an explicit one specified a default is used12:09
* menn0 checks12:09
menn0babbageclunk: TBH this message is almost certainly going to appear before the model's logging config is applied so may it's a moot point12:10
babbageclunkOk, I'll do it as debug and make sure I can see it in a bootstrap.12:10
menn0babbageclunk: sounds good12:14
menn0babbageclunk: i'm off. past bed time here12:14
babbageclunkCan someone who knows about Juju logging help me? I've added the logging I was discussing with menn0 ^^, but I can't see it in logs after bootstrapping.12:55
fwereadebabbageclunk, am pretty sure that `juju set-model-config logging-config=INFO` will set the level globally12:56
fwereadebabbageclunk, also take note of maybe wanting to log some things in the controller model and some in the hosted model12:57
fwereadebabbageclunk, s/log some things/see some logs/12:57
babbageclunkfwereade: So I should check machine-0.log on the controller and in (say) default?12:58
babbageclunkfwereade: I've changed the place where I log to be a warning but it still doesn't seem to show up either on the controller or a machine that I add.12:59
fwereadebabbageclunk, huh12:59
babbageclunkfwereade: Do we log to mongo?12:59
fwereadebabbageclunk, we do; I am brainfarting on exactly where the code lives13:00
babbageclunkI'm logging in a sync.Once in session.Dial. If the log writer calls Dial, then maybe that first message gets dropped on the floor (since the writer's being set up) and then the message never gets logged again (because Once)?13:01
fwereadebabbageclunk, hmm, I am a little loath to suggest it but I would be inclined to use the mgo logging stuff there13:01
mupBug #1605241 opened: juju2beta12 lxd instances not starting <conjure> <juju-core:New> <https://launchpad.net/bugs/1605241>13:02
fwereadebabbageclunk, and use mgo.SetLogger to give it something that near-enough matches what it expects13:02
babbageclunkfwereade: Where would the logging go? Oh, whereever the logger writes to.13:02
fwereadebabbageclunk, which can itself be wrapping a loggo.Logger13:02
* perrito666 wrong window13:03
babbageclunkfwereade: Wouldn't that mean my patch would need to patch something within juju/juju?13:03
fwereadebabbageclunk, as a patch to mgo alone, I think it should use mgo's existing logging facilities13:04
fwereadebabbageclunk, if we wanted to make use of it in juju, we can/should separately register a logger for mgo so the output goes where we want it13:04
babbageclunkfwereade: But there's nowhere in juju/juju that already calls mgo.SetLogger13:05
fwereadebabbageclunk, this is true -- what's the high level goal here? to get something in juju's logs that tells us that mgo did the right thing?13:05
fwereadebabbageclunk, if so, I think we should be using the existing mgo logging13:06
fwereadebabbageclunk, if not, sorry I misunderstood :)13:06
babbageclunkfwereade: Just so we can tell that the juju running has the patch without torture testing (like stokachu's doing now)13:06
babbageclunkfwereade: but the existing mgo logging isn't hooked up, so I'd need to make a separate change to do that.13:07
fwereadebabbageclunk, and I'm not sure that would be super-helpful either... we probably don't want to risk logging db ops to the db13:07
babbageclunkfwereade: yeah - I think that's kind of the problem I'm hitting.13:08
fwereadebabbageclunk, so, how about exposing `mgo.HasAppliedE11000Patch() bool` and logging that explicitly in controller setup?13:08
fwereadebabbageclunk, which can just Once the patch and return true?13:09
fwereadethat's weird though, logging shouldn't be what triggers it13:09
babbageclunkfwereade: Would I have to do that with reflection in the Juju code? Otherwise no-one would be able to build without applying the patch.13:09
babbageclunk(Which might be a good thing.)13:10
fwereadebabbageclunk, ha. dammit again :)13:10
fwereadebabbageclunk, although, yeah13:10
fwereadebabbageclunk, that may very well be a good thing13:10
fwereadebabbageclunk, I am not comfortable developing against arbitrarily-varying dependencies13:11
babbageclunkfwereade: probably means we'd need better (some) tooling to apply/unapply patches.13:11
fwereadebabbageclunk, it would, and we were just discussing in the tech board how we probably didn't want to devote effort to it because we wanted this situation to be rare and not encourage it13:12
babbageclunkfwereade: crazy idea - in the once, start a goroutine that waits 2 seconds and then logs the patch message.13:12
* babbageclunk felt bad typing that.13:13
* fwereade peers disapprovingly over half-moon spectacles13:13
* babbageclunk is just going to try it for shits and giggles.13:13
mupBug #1605241 changed: juju2beta12 lxd instances not starting <conjure> <juju-core:New> <https://launchpad.net/bugs/1605241>13:14
* fwereade -- if such a thing lands -- will try to arrange matters such that he gets to hunt you for sport ;p13:15
fwereadebabbageclunk, but, yeah, see if it makes a difference13:15
babbageclunkWould at least confirm that that's the problem.13:16
fwereadebabbageclunk, I am generally uncomfortable with our using logging as this sort of data channel, *and* with adding a loggo dependency to mgo...13:16
fwereadebabbageclunk, ...but I don't have a better answer offhand, I must admit13:18
babbageclunkfwereade: This logging is only intended to be in the patch - it'll never go upstream, and will be gone when the patch can go.13:18
babbageclunkfwereade: but I know what you mean.13:18
fwereadebabbageclunk, there's nothing so permanent as a temporary fix13:19
fwereadebabbageclunk, how will we know when we're using a fixed upstream build? ;p13:19
fwereadebabbageclunk, unless that also logs...13:19
mupBug #1605241 opened: juju2beta12 lxd instances not starting <conjure> <juju-core:New> <https://launchpad.net/bugs/1605241>13:20
babbageclunkThe problem with the patch is there's no place we can look to determine whether a binary includes it - if the binary is tagged with a hash we can at least look in dependencies.tsv and follow the chain.13:21
fwereadebabbageclunk, (can we abuse the flag package to accept `-mgo.ensure-e11000-flag` or did we isolate it well enough that we can't? ...not that that's any better really anyway)13:23
fwereadebabbageclunk, wondering about env vars... they're probably the least worst globals I know...13:25
* fwereade actually has no idea13:27
babbageclunkHmm. Am I looking in the wrong place?13:28
babbageclunkIs everything in machine-0.log on the controller machine?13:28
fwereadebabbageclunk, I most often just use juju `debug-log --replay` to get it locally so long as the system's generally functioning sanely13:30
fwereadebabbageclunk, with appropriate `-m whatever`13:30
babbageclunkok - that's what I'm doing (with -m controller)13:30
babbageclunkbums, I was sure that would work.13:31
fwereadebabbageclunk, hey13:32
fwereadebabbageclunk, can you just make it write out a file somewhere..? say, in the standard temp dir, whenever it starts?13:32
babbageclunkYeah, that seems extremely obvious now.13:33
babbageclunkI can definitely do that.13:33
fwereadebabbageclunk, cool13:33
babbageclunkfwereade: Thanks!13:33
fwereadebabbageclunk, a pleasure :)13:33
* babbageclunk just wasn't expecting this to be so hard!13:34
babbageclunkI guess that's all of software in a nutshell.13:34
fwereadebabbageclunk, never a truer word was spoken13:34
fwereadebabbageclunk, that still does have the what-will-a-fixed-upstream-build-do problem, though13:37
babbageclunkI was thinking about putting the time in it, but I guess that's really just the mtime of the file. So an empty file it is!13:37
babbageclunkfwereade: Well, it won't write the file, but we'll be able to see that the PR was merged into the commit. It's just normal change-management stuff, isn't it?13:39
babbageclunkfwereade: (I'm not sure I understand the problem.)13:39
fwereadebabbageclunk, yeah, if it's just an internal check that's fine -- but if we expect anyone in the field to need to check it, surely we should always write it forever?13:44
fwereadebabbageclunk, i.e. if it's just for us, to have confidence that it was applied in build X, we can trigger the write on an env var and just make that one of the things we do until we merge the new version13:45
fwereadebabbageclunk, if it's not just for us it's 1000x harder13:45
babbageclunkfwereade: I think it's just for us.13:46
babbageclunkfwereade: why an env var?13:47
babbageclunkfwereade: (as opposed to unconditionally)13:47
babbageclunkfwereade: Ha ha - what if trying to log from inside dial was triggering a panic due to infinite recursion and then loggo was suppressing that?13:49
fwereadebabbageclunk, just because it's nicer not to dump the file wherever we run in production13:53
fwereadebabbageclunk, (that would be engagingly horrifying)13:53
babbageclunkfwereade: yeah, makes sense - is it hard to set the env var so that it's set before the controller agent gets bootstrapped? Or would we check the env var each time, and only once.Do the writing when the env var was true13:56
fwereadebabbageclunk, I was imagining that patched-mgo could have an init() that checks the var, writes the file if true, and that's all13:59
babbageclunkfwereade: ok, but how would we set the env var on the bootstrapped machine in time for that? I guess we could set the var and bounce jujud.14:00
rick_h_katco: ping for standup14:01
babbageclunkfwereade: ha ha, I'm lame - all my theorising was for naught. The real problem was that I was logging in Dial, which we never call, and not in DialWithInfo.14:22
fwereadebabbageclunk, haha, bad luck14:23
babbageclunkSo just logging with loggo works fine.14:23
fwereadebabbageclunk, ok, please decorate it with suitably apocalyptic warnings about the terrible practices and I'm happy14:23
babbageclunkwilco! (At least it's not the asynchronous logging with a sleep version. I really only suggested that to push the overton window across. :)14:24
natefinchfwereade: I can send an email about what we're doing with interactive commands... gotta run, will be back in an hour and a half, which I know is getting late for you.14:25
=== natefinch is now known as natefinch-afk
fwereadenatefinch-afk, I might stop then but will try to be back later14:29
frobwaredear lazy-web, anybody know off-hand if Python 3's standard library has YAML support (reading/writing) baked in?14:36
rick_h_frobware: everyone in python land uses http://pyyaml.org/14:38
frobwarerick_h_: is that part of the standard library?14:39
rick_h_frobware: no, it's not14:39
frobwarerick_h_: asking because the script (bridge) has no 3rd party dependencies14:40
frobwarerick_h_: if we introduce dependencies we just need to ensure they get to the node being deployed before the script is run14:40
frobwarerick_h_: I just added thar as a "risk" to the doc14:41
mgzfrobware: can you just output json?14:49
mgzthat's also yaml, and is parsable by python stdlib14:50
mgzwe should really only use yaml for the config files we expect users to edit a lot14:51
frobwaremgz: need to parse networkd's (YAML) config, and rewrite.14:54
rick_h_frobware: where does this script come from?14:55
mgzfrobware: ah, it's not our yaml? suck14:56
frobwarerick_h_: juju - we add it to cloud-init as text which gets dropped into a directory and then we have additional "rules" in our cloud-init to go look for that file and run it.14:56
mgzfrobware: wait, don't we always have pyyaml on cloud images?14:57
rick_h_frobware: and it's python because that's what it needs to be for cloud-init?14:57
mgzfrobware: it's a cloud-init dependency14:57
mgzso, we can't run cloud-init if it's not installed14:57
frobwarerick_h_: it was the easiest way for it to get to the node being deployed. It's essentially a char* - for those old enough. :)14:58
frobwarerick_h_: https://github.com/frobware/juju/blob/1.20/provider/maas/environ.go#L65915:00
frobwarerick_h_: and this time for real: https://github.com/juju/juju/blob/master/provider/maas/environ.go#L120015:01
alexisbbabbageclunk, I am going to change location sbefore we meet15:20
alexisbwill be back online in a few minutes, but may be a little later than half past15:20
babbageclunkhi alexisb - sorry, was afk when you sent that message.15:56
fwereadehas anyone else seen cmd/jujud/agent failures with "model is not prepared", and/or deadlocks involving the dummy provider?15:57
rick_h_frobware: so I just bootstrapped, created a machine with add-machine and from there ran python3 import yaml16:02
rick_h_frobware: and it's there16:03
rick_h_frobware: so I don't know if it's the image, the cloud-init, or whatever setup but pyyaml is ootb on a machine in xenial16:03
frobwarerick_h_: great & thanks16:04
frobwarerick_h_: the only downside given our current logic is that we look for python2 first16:05
frobwarerick_h_: do you still have the environment running? can you install python2 and try the same import?16:06
=== natefinch-afk is now known as natefinch
rick_h_frobware: sure thing16:07
natefinchfwereade: I'm here, but feel free to not be here, since it's dinner time over there16:08
rick_h_frobware: no luck there, no yaml ootb in python216:09
frobwarerick_h_: so the issue there is that there's no guarantee somebody doesn't customise their MAAS image and we get python2 by our current search order.16:09
rick_h_frobware: you have to install python-yaml to get it in py216:10
rick_h_frobware: so can we just change the search order?16:10
rick_h_frobware: I mean py3 is the default now?16:10
frobwarerick_h_: see the comment: https://github.com/juju/juju/blob/master/provider/maas/environ.go#L120016:10
rick_h_frobware: so I see/process that. I guess if we know the script works in py3, and if we find it (they modified so py3 is there in trusty for instance) what's the risk of things going boom? maybe the version of py3?16:12
frobwarerick_h_: I would say less so than Feb/March - it was all new at that point.16:13
frobwarerick_h_: and the unit tests test both python2 and 3 -- assuming you have both installed (on the SUT)16:13
babbageclunka quiet stokachu is a happy stokachu?16:13
rick_h_frobware: ok, just pushing/questioning :)16:13
stokachubabbageclunk: so far so good!16:13
* babbageclunk dances16:13
stokachubabbageclunk: ive been beating the hell out of it with menno's patched binary16:14
stokachuno issues yet16:14
frobwarerick_h_: yep, the issue we have is... unless you go and test all this manually we can't just suck it and see against a CI / test rig.16:14
frobwarerick_h_: which is true for a lot of what we do16:14
babbageclunkstokachu: (ever seen Archer? I want to say phrasing)16:14
babbageclunkstokachu: great!16:14
stokachubabbageclunk: love archer16:15
rick_h_frobware: understand, ok...so where does this leave you?16:15
rick_h_frobware: you get yaml in py3, but not other stuff. But this is all about using the new networking in yakkety stuff which is yaml?16:15
rick_h_frobware: or is this for something else?16:15
frobwarerick_h_: no, you're right this is yakkety (so python3) so maybe ok16:15
rick_h_frobware: so if they monkey with the maas image we'll still know to use py3 because it's got yakkety only networkd stuff16:15
rick_h_frobware: so triggering off the OS vs the python search tree seems like a potential path forward?16:16
frobwarerick_h_: the monkeying comes from me taking a default Y image and makeing my curtin/custom_pressed have python2 for nefarious reasons16:16
rick_h_frobware: so you mean to not just add py2, but to remove py3?16:17
rick_h_I mean, just because py2 is there doesn't mean we have ot use it as long as py3 is there as well.16:17
frobwarerick_h_: where "me" is some customer in the wild who knows he wants python2 from the get-go16:17
frobwarerick_h_: yep, we get first switch on series16:17
frobwarerick_h_: well, that's likely to break stuff. the issue is really only in adding py2.16:18
rick_h_frobware: right, but we can be resilient to adding py2 because we can be smarter than just looking at python available16:18
frobwarerick_h_: but I don't think we should sweat this. we can first switch on series.16:18
rick_h_if we want to marshall/unmarshall yaml it's because networkd gave it to us and we know to use py3 in that case16:18
frobwarerick_h_: yep16:18
rick_h_frobware: rgr16:18
rick_h_sounds like a plan16:19
frobwarerick_h_:  I'm always wary of the sands shifting underneath my feet. :-D16:19
* frobware is out of here before there's a python 4...16:20
mupBug #1605313 opened: juju 2.0-beta12 ERROR unable to contact api server after 61 attempts: upgrade in progress (upgrade in progress) <sts> <juju-core:New> <https://launchpad.net/bugs/1605313>16:32
redirbrb reboot17:11
mupBug #1605335 opened: Bootstrap agent initialization timeout too small <juju-core:New> <https://launchpad.net/bugs/1605335>17:41
=== frankban is now known as frankban|afk
mupBug #1605313 changed: juju 2.0-beta12 ERROR unable to contact api server after 61 attempts: upgrade in progress (upgrade in progress) <sts> <juju-core:New> <https://launchpad.net/bugs/1605313>19:36
mupBug #1605383 opened: Tab complete causes traceback when trying to tab-complete juju commands <juju-core:New> <https://launchpad.net/bugs/1605383>20:06
rick_h_natefinch: ty for the email, while that shakes out can we work on talking with techboard/etc folks on path forward but in the meantime pick up the windows bug in the todo side for now please?20:28
rick_h_natefinch: and hold onto the add-cloud while the path forward shakes out20:28
natefinchrick_h_: sounds good20:28
rick_h_natefinch: cool ty20:29
=== natefinch is now known as natefinch-afk
redirbrb stepping away for a minute21:39
=== redir is now known as redir_afk
anastasiamacnatefinch-afk: could u plz review this one? http://reviews.vapour.ws/r/5291/21:43
anastasiamacanyone else is welcome to PTAL :)21:43
mupBug #1604514 changed: Race in github.com/joyent/gosdc/localservices/cloudapi <ci> <joyent-provider> <race-condition> <juju-core:Fix Released by menno.smits> <juju-core 1.25:Fix Released by menno.smits> <https://launchpad.net/bugs/1604514>21:45
mupBug #1604559 changed: LockingSuite.TestTestLockingFunctionDetectsDisobeyedLock did not obey <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1604559>21:45
mupBug #1605057 changed: allWatcherStateSuite.TestStateWatcherTwoModels takes > 5 mintues <tech-debt> <unit-tests> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1605057>21:45
mupBug #1589635 changed: github.com/juju/juju/state fails on TestMachinePrincipalUnits with an unexpected name <ci> <regression> <test-failure> <unit-tests> <juju-core:Fix Released> <https://launchpad.net/bugs/1589635>21:54
=== frankban|afk is now known as frankban
=== ses is now known as Guest33830
=== redir_afk is now known as redir
thumpermenn0: thoughts on this failure? http://juju-ci.vapour.ws:8080/job/github-merge-juju/8503/artifact/artifacts/windows-out.log23:34
mgzthumper: just looks like bug 160030123:44
mupBug #1600301: cmd/jujud/agent MachineSuite.TestHostedModelWorkers fails because compute-provisioner never really starts <ci> <intermittent-failure> <regression> <unit-tests> <juju-core:In Progress by dooferlad> <https://launchpad.net/bugs/1600301>23:44
menn0thumper: so I've seen this kind of thing in that package quite a bit recently23:58
menn0as I said in the call there seems to be a few different failure modes23:59
menn0I'm guessing they're all the same root cause23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!