[00:32] <hml> veebers: ping
[00:44] <veebers> hml: pong o/ how's things?
[00:44] <hml> veebers: good - getting cold
[00:45] <hml> veebers: is there a god way to force a model migration failure? - i’m working on your bug for show-model
[00:46] <veebers> hml: babbageclunk would have a good idea, I think attempting to migrate before units are idle would do it (i.e. do the commands really quick)
[00:46] <hml> veebers: at least with the tests, i’m not sure any error status will be more helpful.  :-/  e.g. "machine sanity check failed, 2 errors found"
[00:47] <veebers> hml: it's starting to get warmer here, although we've had really nice warm days then it descends into cold days
[00:47] <hml> veebers: of course, he’s the one who knew how to fix this .
[00:47] <hml> veebers: the fun days - hot, cold, hot
[00:47] <veebers> aye ^_^
[00:49] <veebers> hml: I love that Christmas (and thus Christmas break) is over the middle of summer here :-)
[00:49] <hml> veebers: that’s just plain weird
[00:49] <veebers> ^_^
[00:49] <hml> veebers: that said, the family heads south for christmas
[00:50] <veebers> hml: hah, flying in arrow formation I hope :-)
[00:50] <hml> veebers: ha
[01:22] <veebers> Hmm, with lxd, now when publishing an image from an existing container it takes ages and create tmp files > 100GB. I think something is wrong :-\
[01:22] <babbageclunk> hml: sorry, was grabbing lunch - unfortunately the units not being ready won't fail in the right way now.
[01:31] <hml> babbageclunk: i’m thinking this is what you had in mind?  https://github.com/hmlanigan/juju/commit/06db34cb637594cbe6e80db4ecb1f22778b2988f
[01:32] <hml> babbageclunk: is there a good way to force a model migration abort?
[01:32] <hml> not having any luck at it
[01:36] <babbageclunk> hml: not that I know of - it generally indicates a failure of prechecking. You could simulate it with a wrench?
[01:36] <hml> babbageclunk: wrench?
[01:37] <babbageclunk> hml: we've got a mechanism for throwing errors if a wrench file is present...
[01:37] <babbageclunk> hml: for testing hard-to-reach scenarios...
[01:37] <babbageclunk> hml: hang on, finding it
[01:37] <hml> babbageclunk: haven’t run into that yet.
[01:38] <hml> babbageclunk: the wrench part :-)
[01:38] <babbageclunk> hml: oh, there's one in the migrationmaster already...
[01:39] <babbageclunk> hml: see the call to wrench.IsActive?
[01:39] <hml> babbageclunk: yes
[01:41] <babbageclunk> basically you could throw an error from transferModel if wrench.IsActive("migrationmaster", "die-in-export")
[01:42] <hml> babbageclunk: okay - and wrench active would be a line in  /var/lib/juju/wrench/machine-agent?
[01:44] <babbageclunk> I think it would be wrench/migrationmaster - add die-in-export to that file
[01:44] <hml> babbageclunk: ah, ty
[01:46] <babbageclunk> hml: no worries
[01:46] <hml> babbageclunk: the change was what you had in mind?
[01:46] <babbageclunk> hml: yup
[02:00]  * thumper sighs
[02:01] <thumper> we have found two new buts that we really need to fix for 2.3
[02:01] <thumper> heh
[02:01] <thumper> buts
[02:01] <thumper> gugs
[02:01] <thumper> bugs
[02:01] <thumper> those things
[02:01] <thumper> babbageclunk: got 10 minutes?
[02:01] <thumper> babbageclunk: I'd like to talk a few things through
[02:05]  * thumper steps away
[02:15] <babbageclunk> thumper: sorry, looking again!
[02:15] <babbageclunk> thumper: grab me when you come back
[02:21] <hml> babbageclunk: veebers: at least the last error won’t be overwritten by the abort msg with model migration.  we’ll see how useful the errors are.  :-)
[02:23] <axw> wallyworld: looking at your PR shortly, I've just put up https://github.com/juju/juju/pull/8087 if you have time to look; moves state pool to worker/state
[02:24] <wallyworld> axw: will do, just getting a quick bite after finishing interview
[02:24] <axw> sure, no great rush
[02:25] <veebers> hml: excellent :-) It should be very helpful when migrations fail (especially in tests so we can categorise them)
[02:30] <jam> axw: so we would still try a specific zone, but doesn't your patch mean we'll at least fall back to another one?
[02:31] <jam> I suppose if the issue is that pods will give you one when they otherwise would want to fallback to explicit hardware
[02:31] <jam> but apparently the underlying bug was actually that MAAS would ignore a 'tag' constraint which is a better way to target real hardware
[02:32] <blahdeblah> thumper: Did you get a note from the meeting about checking whether juju create-backup excludes previous backups?
[02:32] <blahdeblah> thumper: Also, has any thought been given to leaving the backups in the filesystem instead of mongodb?
[02:36] <hml> wallyworld: babbageclunk: i have a quick review if you’re around: https://github.com/juju/juju/pull/8088
[02:36] <babbageclunk> yup
[02:36] <wallyworld> ok
[02:41] <wallyworld> hml: the migration message won't actually sat that the migration is aborted though will it? it will show some arbitary error text but the user won't know what the final outcome is. maybe it's a recoverable, temporary error, who knows
[02:41] <hml> wallyworld: i left as info, since it was calling setInfoStatus…
[02:42] <wallyworld> you mean the log message?
[02:42] <hml> wallyworld: yes
[02:42] <wallyworld> what written to logs is a warning though since something failed
[02:42] <wallyworld> the setInfoStatus() is just an api name
[02:43] <axw> jam: in the cases where it would *fail* because of going to the zone, then sure. for this bug, I was only thinking about the case where we can get a machine in either zone, but we really want the one in the default zone
[02:43] <wallyworld> is there a way to track the last migration error in the worker and when aborting, preprent the error with "aborted: " or something
[02:43] <hml> wallyworld: in this case, goes to info
[02:43] <hml> wallyworld: i’ll have to look
[02:43] <wallyworld> setStatusInfo(0 goes to the model status right? not the logs
[02:43] <hml> it goes to both
[02:44] <wallyworld> ok, but the final logged message should be a warning - sys admins grep for warnings and knowing something has failed is important
[02:45] <hml> okay, can be changed
[02:46] <hml> there doesn’t appear to be a last failure.
[02:47] <hml> would have to change for every abort, instead of in abort
[02:47] <hml> and some places not easily known -
[02:50] <wallyworld> i haven't looked at code - can we record last failure as an attribute on the migration op. there's ony one place where the final abort status is set? or no?
[02:51] <hml> there is only one place abort status is set -
[02:52] <wallyworld> that's good - we just need to check for last error and prepend that with "migration aborted: " or whatever
[02:52] <wallyworld> that way it is 100% clear that the thing has stopped and why
[02:52] <hml> where is the last error?
[02:53] <wallyworld> yo need to record it!
[02:53] <wallyworld> create a varioable to hold it
[02:54] <wallyworld> have a heper method somehwere used to update status with anerror, and overwrite a lastError each time. or something
[02:54] <hml> sorry i’m being dense here.
[02:54] <hml> there’s one i can add on to
[02:54] <wallyworld> could be me oversimplifying
[02:55] <wallyworld> i only have a little brain
[02:55] <hml> i was thinking the worker in this case was at a highlevel
[02:55] <hml> but it’s in mibrationmater
[02:58] <wallyworld> yeah, i think migration master entity should be able to track errors encountered doing the job
[03:07] <wallyworld> babbageclunk: you loving simple streams? :-D
[03:09] <wallyworld> axw: just started looking - why was srv.run() put in a go routine? maybe it will be clear when i read more of the PR?
[03:10] <wallyworld> just curious if it was due to a drive by or part of the PR
[03:10] <axw> wallyworld: apiserver.Server is a worker, I'm just making it conform to our usual patterns
[03:10] <wallyworld> ok, ta
[03:10] <axw> wallyworld: it's not a big deal, just tidying up
[03:10] <wallyworld> no worries, sgtm
[03:13] <wallyworld> axw: and that means you were able to pull expireLocalLoginInteractions() out of a separate go rountine
[03:15] <hml> wallyworld: changes done
[03:15] <babbageclunk> wallyworld: I am not
[03:15] <wallyworld> hml: awesome tyvm, looking
[03:16] <wallyworld> hml: much nicer, thank you!
[03:17] <wallyworld> babbageclunk: did we need a HO?
[03:17] <babbageclunk> wallyworld: probably couldn't hurt
[03:17] <wallyworld> righto
[03:20]  * hml headed to the airport to pick up my dad
[03:20] <wallyworld> hml: see you later
[03:25] <axw> wallyworld: yeah, the idea was to stop using the waitgroup for two different things (tracking sub-workers, and outstanding connections/requests)
[03:30] <babbageclunk> thumper: did you come back?
[04:03] <wallyworld> axw: sorry about delay, had meeting, lgtm
[04:15] <axw> wallyworld: no worries. I've left a bunch of comments on your PR. I'd like it to be split up a bit (see comments), and worker responsibilities separated
[04:15] <wallyworld> ok looking
[04:17] <axw> wallyworld: re renaming StateTracker to StatePoolTracker or whatever, I'd really rather not. I'm considering StatePool just an implementation detail, as the entry point to "state". I want to replace it with a different type which manages all the state workers, as we've talked about recently
[04:17] <wallyworld> ok
[04:24] <babbageclunk> wallyworld: if we find the same binary version in multiple streams it doesn't matter, does it? They'll be the same.
[04:24] <wallyworld> they are today but not guaranteed
[04:25] <wallyworld> alsthough they will always be the same in practice
[04:25] <babbageclunk> We won't find 2.3-rc1 in both released and proposed.
[04:25] <wallyworld> correct
[04:25] <wallyworld> rc1 should only be in proposed
[04:25] <wallyworld> sorry, i think i misunderstood the first time
[04:25] <babbageclunk> No, that was a slightly different question.
[04:47] <thumper> wallyworld: https://github.com/juju/juju/pull/8089
[04:47] <wallyworld> ok
[04:47]  * thumper heads out for more kid duty
[04:47] <thumper> bbl
[05:06] <thumper> wallyworld: I'm not sure I agree with you on the select bits
[05:06] <wallyworld> thumper: why have 2 select blocks which creates bloat when we just need one?
[05:06] <thumper> wallyworld: because it is explicit and clear
[05:07] <thumper> you aren't allowed to send to a nil channel
[05:07] <thumper> not even allowed to try
[05:07] <wallyworld> sending to a nil channel as a no-op is fairly standard
[05:07] <thumper> no
[05:07] <wallyworld> oh wait
[05:07] <wallyworld> receiving
[05:07] <wallyworld> sorry
[05:07] <thumper> pulling off a nil channel
[05:07] <wallyworld> yeah, ignor eme
[05:07] <thumper> ok
[05:07] <thumper> I'll look at the Timeout bit
[05:07] <wallyworld> ok
[05:08] <thumper> I think in order to keep the patch small, we shouldn't rename Timeout in this branch
[05:08] <thumper> in case we need to back port it
[05:09] <thumper> I'm ok in principle with renaming Timeout
[05:09] <thumper> although Timeout is fairly self explanitory
[05:11] <wallyworld> except that there's 2 and it's a bit ambiguous, but the doc helps and agree about minimising change
[06:02] <wallyworld> axw: why do you want separate NewIAASModel() and NewCAASModel() when the code for both is 95% the same? there's just a couple of different checks which should disappear at some point eg the IAAS restriction on new model being in the same cloud as the controller we have talked about removing
[06:03] <wallyworld> given model type is eassentially just a model attribute, i'm not sure what such a far reaching change buys us
[06:04] <wallyworld> there > 30 usages of NewModel()
[06:08] <axw> wallyworld: because it keeps the callers simpler, not having to pass in irrelevant things like storage providers and constraints. so each time you add something IAAS specific you don't have to worry about CAAS code, and vice versa
[06:08] <axw> wallyworld: all the existing ones are for IAAS models, right?
[06:08] <wallyworld> we already support adding a caas model
[06:09] <wallyworld> most of the test cases are iaas
[06:09] <wallyworld> this NewModel() code with caas type has been there for a while
[06:09] <wallyworld> i think
[06:10] <wallyworld> i can change, will just add some noise to the pr
[06:10] <wallyworld> i'm starting a new pr
[06:10] <wallyworld> for the below the water line stuff
[06:10] <axw> wallyworld: leave that one then
[06:10] <axw> it can be refactored later
[06:11] <wallyworld> axw: i can easily do in a followup
[06:12] <axw> wallyworld: my biggest beef is with responsibility of making/maintaining connections, and head was getting too full to dive into the state innardy bits
[06:12] <axw> the rest is small stuff
[06:12] <wallyworld> yup fair point
[06:12] <axw> except ModelActive
[06:12] <wallyworld> i've changed but still dislike having to make 2 separate mgo queries
[06:13] <wallyworld> you don't have a model object at the call site - just a uuid
[06:13] <wallyworld> and the ony thing that calls ModelActive() is there
[06:13] <axw> wallyworld: but you can change that, by using a state pool -no?
[06:13] <wallyworld> no, this is before the agent is started
[06:13] <wallyworld> when it is figuring out what manifolds to use
[06:14] <axw> wallyworld: it's in the model worker manager? that's not before any agents have started...
[06:15] <axw> it's run outside of the machine manifold, which is part of the problem
[06:15] <axw> it needn't be though
[06:15] <wallyworld> ok, i'll look closer
[06:15] <axw> wallyworld: I can move that into the machine manifold if you like, while you're doing other things?
[06:16] <axw> chasing down a test failure atm, which I'll need to fix first
[06:16] <wallyworld> ok, that would be grand. i can rebase on top of that. i've got about 3 branches on the go
[06:16] <wallyworld> got the skeleton operator agent going
[06:17] <wallyworld> with a bit of common stuff extracted from uniter
[06:17] <wallyworld> small steps
[06:22] <axw> wallyworld: I'm going to merge develop into the feature branch, OK?
[06:22] <wallyworld> \o/
[06:22] <axw> there's some agent fixes I don't want to work around
[06:23] <wallyworld> yup
[06:23] <wallyworld> i wish git had pipelines like bzr
[06:23] <wallyworld> would make working on several dependent branches so much easier
[07:02] <wallyworld> axw: you still working on the state pool tracker pr or can that land now the feature branch has been updated from develop?
[07:03] <axw> wallyworld: I'm trying to figure out why a test failed
[07:03] <wallyworld> righto
[07:03] <axw> wallyworld: I've been unable to reproduce
[07:03] <wallyworld> joy
[07:04] <axw> might just land and see if it pops up again, might be blue moon type of thing
=== frankban|afk is now known as frankban
[08:57] <babbageclunk> wallyworld: can you take a look at another WIP and let me know whether you're happier with that approach?
[08:58] <axw> wallyworld: took a bit longer than expected, here's my PR to move modelworkermanager to the dependency engine: https://github.com/juju/juju/pull/8093
[08:58] <babbageclunk> wallyworld: oops https://github.com/juju/juju/pull/8092
[08:58] <axw> pipped to the post!
[09:01] <wallyworld> babbageclunk: looking
[09:08] <babbageclunk> axw: I don't know, you got the link there first
[09:08] <axw> :)
[09:09] <wallyworld> babbageclunk: looks pretty good
[09:10] <babbageclunk> wallyworld: cool cool, just testing
[09:10] <wallyworld> babbageclunk: i'm not sure there's more to do to handle searching across datasources? as the current behaviour might be sufficient?
[09:11] <wallyworld> axw: ty, i'll look after dinner
[09:11] <babbageclunk> wallyworld: maybe? It'll still stop on the first datasource that has an index though
[09:11] <wallyworld> yeah
[09:12] <wallyworld> if datasource A only has develp and B has released
[09:12] <wallyworld> it needs to consider all
[09:12] <babbageclunk> right
[09:12] <wallyworld> i think the branch as is can land though
[09:12] <wallyworld> and another done to do the other bit
[09:12] <babbageclunk> I need to fix unit tests
[09:13] <babbageclunk> ok, will do that
[09:13] <wallyworld> jeez, it's late for you
[09:13] <babbageclunk> had to break to feed and bathe kids
[09:14] <wallyworld> see how you go, i might have to pick it up tomorrow
[09:14] <wallyworld> i need to go afk for a short while for dinner
[09:14] <babbageclunk> ok
[09:41] <wallyworld> axw: awesome, ty. i will have some conflicts but I'll deal :-)
[09:41] <axw> wallyworld: cool
[09:42] <wallyworld> axw: i'll have 2 PRs tomorrow. one for the state/infra stuff, one for the worker and manifold stuff. and then I'll weave in the facade stuff. and after that I'll propose the operator skeleton. so I guess that's 4 :-)
[09:43] <axw> wallyworld: sounds good. I'll try and move the remaining machine workers to manifolds in the morning, and we can figure out what CAAS things I can do in 1:!
[09:43] <axw> 1:1 even
[09:44] <wallyworld> yay, more caas stuff
[09:47] <babbageclunk> git question because I'm feeling stupid - if I want to see all the diff-stats for the accumulated commits in my branch, how do I do it?
[09:48] <babbageclunk> I've tried git diff --stat upstream/develop..my-branch but it includes changes on develop that I haven't rebased into my branch yet.
[09:49] <babbageclunk> bah, just rebased so that I didn't need to worry about it.
=== salmankhan1 is now known as salmankhan
[10:25] <axw> balloons: looks like jenkins has died in the ass again - https://github.com/juju/juju/pull/8093
[13:18] <babbageclunk> wallyworld: ping? unlikely
[13:25] <jam> babbageclunk: go to bed
[13:25] <jam> :)
[13:25] <jam> babbageclunk: but if there is something I can help with
[13:25] <jam> axw: balloons said it was out of disk the other day
[13:26] <babbageclunk> jam: I'm thiiiis close to stopping! Thanks, I think I worked it out.
[13:32] <wpk> jam: it's not even 3am, leave him alone!
[13:45] <babbageclunk> wpk :)
[13:48] <babbageclunk> is mergebot down?
[13:50] <babbageclunk> balloons: is there a problem with mergebot? This PR isn't getting built: https://github.com/juju/juju/pull/8092
[13:50]  * babbageclunk goes to bed
[13:58] <balloons> I can look. Mean
[13:58] <balloons> Every morning this week
[13:59] <balloons> I need to share the doc with you all on how to troubleshoot
[14:02] <balloons> Axw, so perhaps we need to clean and reboot the vsphere daily to keep things going?
[14:10] <balloons> looks like cloud instance died
[14:18] <balloons> restored, jobs running again
[16:14] <balloons> wpk, does this look better? https://github.com/juju/juju/pull/8086
=== frankban is now known as frankban|afk
[18:18] <hml> axino: ping
[18:53] <bdx> https://bugs.launchpad.net/juju/+bug/1732764
[18:53] <mup> Bug #1732764: series + spaces + artful = fail <juju:New> <https://launchpad.net/bugs/1732764>
[19:22] <hml> balloons: is that doc for troubleshooting the merge bot around?  :-)  github-check-merge-juju-pipline thinks it has a config error.
[19:23] <balloons> bdx, xenial works and artful doesn't?
[19:26] <balloons> hml, needs updated and it was shared with you at one point :-) https://docs.google.com/document/d/1TN4SG8QXNbXpFn_9QTpPgI7XxD85uzdlttGB0QRlm2I/edit#
[19:27] <hml> balloons: i have to remember that far back?  ;-)
[19:28] <bdx> ballons: yeah
[19:28] <balloons> bdx, ty. We've actually been testing this over the last couple weeks
[19:28] <bdx> the only combo that breaks it is "series + spaces + artful + beta3"
[19:28] <bdx> ballons np
[19:29] <balloons> hml, ohh the old job was restoed
[19:29] <balloons> just needs to be removed from jenkins again
[19:29] <bdx> Im making an official redis snap right now, so it wont matter to me anymore - I was chasing the 4.x in artful lol
[19:30] <bdx> good to get these things fixed anyway though
[19:30] <balloons> bdx, ack. And indeed
[19:32] <balloons> hml, anyways notice the actual bot ran fine on your pr
[19:33] <hml> balloons: i did, wasn’t sure what the pipeline thing was
[21:58] <hml> wallyworld: i looked at the bug for a panic in the storageprovisioner, it’s easy to see the cause, but i’m wondering why the pointer is nil at that point.  who’s been in that area?
[21:58] <hml> wallyworld: just to make sure there isn’t a bad root case
[21:58] <hml> or cause even
[22:08] <balloons> wallyworld, babbageclunk, please test sigfile in the edge snap
[22:10] <balloons> and let me know; since we're going to ship that in rc1 if it's good
[22:16] <balloons> or if you don't say anything :p
[23:31] <axw> balloons: I guess we could try, but it didn't take long for it to start misbehaving after I restarted it last time