[00:08] <anastasiamac> veebers: sinzui: balloons: ^^
[00:25] <veebers> anastasiamac, stokachu: I believe that only either sinzui or anastasiamac can do that due to having macs?
[00:33] <anastasiamac> veebers: my mac is dual boot but does not boot into macOS since Oct ;( sinzui maybe the only one \o/
[00:33] <stokachu> ive got a mac now too so i can do a build if need be
[00:34] <stokachu> anastasiamac: just have sinzui sync up with me if he doesn't think he'll get to a mac build for the beta
[00:34] <anastasiamac> stokachu: \o/ as long as u don't bootstrap controller on mac - we don't support it
[00:35] <stokachu> anastasiamac: just localhost right?
[00:35] <stokachu> anastasiamac: im going to blacklist localhost in conjure-up on a mac anyway
[00:35] <anastasiamac> stokachu: AFAIK we only support clients on Mac... sounds good :D
[00:36] <stokachu> hmm
[00:49] <thumper> rick_h: let me know if you come back and have a few minutes
[00:54] <thumper> rick_h: I think we have this one solved
[00:59] <rick_h> thumper: what's up?
[00:59] <thumper> rick_h: looking at that dashboard, memory seems to be holding steady
[00:59] <thumper> rick_h: also, I'd like you to run something for me on the apiserver
[00:59] <thumper> rick_h: juju-statepool-report
[01:00] <rick_h> k, sec
[01:03] <rick_h> thumper: https://pastebin.canonical.com/183289/
[01:04] <thumper> perfect
[01:04] <thumper> rick_h: fixed that leak
[01:04] <thumper> hazaah
[01:04] <rick_h> thumper: yea, looks like it
[01:04] <rick_h> thumper: ty!
[01:04] <thumper> happy?
[01:04]  * rick_h will find something else to be displeased about :P
[01:04] <rick_h> thumper: very much so, I'll let this run for a while more and then get screenshots/etc to let folks konw we've tested it
[01:04] <thumper> sure
[01:05] <rick_h> thumper: I'd also like to chat with your QA folks to see how we add some sort of regular check like this
[01:05] <rick_h> thumper: it's pretty scriptable at this point with the setup here, but I assume we'll want to do this somewhere maybe on a cloud or something
[01:05] <rick_h> actually, I should run this on gce or aws and see if it has the same issue w/o the fix
[01:05]  * thumper nods
[01:05] <thumper> may well do
[01:21] <menn0> thumper, wallyworld, axw, jam: due to kids stuff, tech board may be tricky for me today. i'll try to make it if I can.
[01:22] <wallyworld> ok
[01:36] <thumper> menn0: ack
[01:37] <menn0> wallyworld, axw: here's the big apiserver cleanup: https://github.com/juju/juju/pull/7133
[01:37] <menn0> no more global facade registry
[01:37] <wallyworld> menn0: will look after standup
[01:37] <menn0> wallyworld: no rush
[01:43] <axw> menn0: looks awesome at a first glance. have to go shortly, or I'd be all over it
[01:43] <menn0> axw: no rush I have to pick up kids anyway
[01:50] <axw> thumper: your PR is against staging
[01:50] <thumper> bollocks
[01:50]  * thumper fixes
[01:51] <thumper> changed
[01:54] <thumper> $ juju bootstrap --agent-version 2.1.0 dev leak-test-2.1
[01:54] <thumper> error: requested agent version major.minor mismatch
[01:54] <thumper> wat?
[01:54] <thumper> why can't my 2.2-beta1 local juju deploy a 2.1 controller?
[01:56] <thumper> that's dumb
[01:59] <thumper> ugh...
[01:59] <thumper> we do that because out tool checking code is shite
[02:00] <thumper> what a POS
[02:19] <wallyworld> thumper: from memory the policy was to require the bootstrap client and agent versions to match down to the minor version number
[02:20] <wallyworld> it never used to be like that a long time ago, but due to potential incompatibilities a policy decision was made
[02:37] <jam> thumper: I've been trying to draw out your ping vs pong and ReadDeadline stuff, I think we might have a small problem with timing
[02:37] <wallyworld> menn0: love the pr
[02:38] <menn0> wallyworld: \o/
[02:38] <jam> thumper: as for minor version stuff, we did it because we broke bootstrap at least once between minor versions and we never wrote code to say "if you're this version controller, use *this* bootstrap code"
[02:38] <thumper> jam: working on interactive stuff
[02:38] <jam> if we change the arguments to "bootstrap-state" for example (which is what we did)
[02:39] <jam> thumper: specifically, you only increment 'readdeadline' when you get a Pong
[02:39] <thumper> yeah
[02:39] <jam> so if you have a Ping and an immediate Pong, then if your next Ping happens and its pong is delayed longer than the first
[02:39] <jam> you might hit timeout
[02:39] <jam> It's easier if it is drawn out, but I have to get the kid ready for school.
[02:39] <menn0> bdx: just remembered I still owed you an answer. sorry.
[02:39] <thumper> jam: yeah, I'm hitting something in my testing, and adding logging to work it out
[02:40] <jam> thumper: also, we need to update Write deadline everytime we do any writes, not just with the Pinger
[02:40] <thumper> but yes, I'm hitting an unexpected close
[02:40] <wallyworld> menn0: thumper: jam: i may well miss tech board, i forgot i need to drive my son to see a surgeon
[02:40] <jam> thumper: because Deadline is for *everything* on the socket
[02:40] <menn0> bdx: the distinction between machine and unit agent is determined by the jujud command line
[02:40] <thumper> jam: but these only write control messages
[02:40] <jam> thumper: where are we writing things like the actual log messages themselves?
[02:40] <thumper> we only read
[02:40] <jam> or is the ping initiated only on the reader side
[02:40] <thumper> yes
[02:41] <menn0> bdx: there are further distinctions if the jujud is acting as a machine agent. the agent finds out its role(s) over the API (ultimately from Juju's MongoDB) and runs the workers required for each role.
[02:41] <jam> thumper: k. I think we might need to set some sort of Read delay whenever we send a Ping, possibly smaller than we set it when we get a Pong
[02:42] <jam> there should be a "how often do we Ping" and a "how long can a Pong" take. and getting a Pong would probably set the Deadline to Now()+time-to-next-ping+time-for-response
[02:42] <thumper> jam: I cribbed this from all the online docs and examples around this ahndling
[02:42] <jam> and sending a Ping would set it to Now()+time-for-response
[02:43] <jam> say it might take 10s for a Pong to respond, but it could respond in 1ms.
[02:43] <jam> and our ping interval is 20s
[02:44] <thumper> jam: yeah, I was wondering about that delay, so that is why I'm doing some interactive testing with additional logging
[02:44] <jam> 0s, Deadline=20s, Ping, 0.1s Pong, Deadline=20.1s, 20s Ping, 20.1s Timeout, 22s Pong
[02:44] <jam> thumper: ^^ check my work, but I think that's the failure case I came up with
[02:45] <thumper> I think the key is to make sure the ping frequency is sufficiently smaller than the pong delay added
[02:47] <thumper> perhaps we set pong delay to 90s and ping frequence to 60s
[02:47] <thumper> we ping every 60 seconds
[02:47] <thumper> and that gives a 30s window for pong to come back
[02:47] <thumper> even though we expect a subsecond pong
[02:47] <thumper> each pong pushes the read window out 90s
[03:08] <jam> thumper: so I think they are functionally equivalent, I'm just not sure which is easier for people to understand
[03:32] <jam> thumper: menn0: tech board?
[04:48] <babbageclunk> jam: when you get a chance can you take another look at https://github.com/juju/juju/pull/7126?
[04:49] <babbageclunk> jam: I'll do separate PRs to store the networks in state and tag subnets with the vpc-id in ec2.
[05:18] <anastasiamac> babbageclunk: axw: wallyworld: PTAL https://github.com/juju/names/pull/78
[05:41] <wallyworld> anastasiamac: lgtm, ty!
[06:27] <axw> wallyworld: just got back, need to eat then I'm free to chat
[06:27] <wallyworld> axw: rightio. i'm about to go get lachie from the train, but will be home in say 20 or 25 minutes
[06:27] <axw> ok, sounds good
[06:45] <babbageclunk> wallyworld, axw: I'm here too!
[06:52] <wallyworld> babbageclunk: awesome, time for a chat in the standup HO?
[06:52] <wallyworld> axw: ^^?
[06:53] <axw> wallyworld: sure, brt
[07:14] <anastasiamac> babbageclunk: axw: wallyworld: dependencies update PTAL : https://github.com/juju/juju/pull/7135
[07:16] <wallyworld> anastasiamac: lgtm
[11:09] <jam> babbageclunk: reviewed
[12:58] <jam> rick_h: did you restart your testing? I tried logging into Grafana again, but admin/testings didn't work
[12:58] <rick_h> jam: yes, remove the s
[12:58] <rick_h> jam: just sent an email on the new run starting
[12:59] <rick_h> jam: there's a race in the grafana charm that if you set the password too close to the deploy (before it's up) the password isn't set right so you end up changing it again (add an s to the end heh)
[12:59] <jam> rick_h: ah, it doesn't respect config at boot, only if it gets set once the app is actually up?
[12:59] <jam> sucky
[13:00] <rick_h> jam: rgr, seems that way.
[14:57] <rogpeppe> here's a refactoring of the juju login command. would really appreciate a quick review as this is critical for us. https://github.com/juju/juju/pull/7136
[14:57] <rogpeppe> jam: any chance you might be able to take a look?
[20:05] <niedbalski> thumper, anastasiamac wallyworld https://bugs.launchpad.net/juju-core/+bug/1675154 , any idea about this one?
[20:05] <mup> Bug #1675154: Upgrade not possible from 1.25.6 to 1.25.10  <sts> <juju-core:New> <https://launchpad.net/bugs/1675154>
[20:07] <mup> Bug #1675154 opened: Upgrade not possible from 1.25.6 to 1.25.10  <sts> <juju-core:New> <https://launchpad.net/bugs/1675154>
[20:08] <thumper> niedbalski: not at this stage, but I need to step away to eat breakfast
[20:49] <mattyw_> thumper, let me know when you're back from breakfast, I have reviews for you if you have time
[20:49] <thumper> mattyw_: here
[20:50] <mattyw_> thumper, so I *think* descriptions might be right now? https://github.com/juju/description/pull/5
[20:50]  * thumper looks
[20:52] <mattyw_> thumper, and after that we should talk about my core pull request - I think I know why that test was passing (when it shouldn't have) but we should talk about it
[20:52] <thumper> ok
[21:05] <thumper> mattyw_: review done
[21:07] <mattyw_> thumper, awesome thanks, I'll fix it up now and hassle you again. Would you have time to discuss https://github.com/juju/juju/pull/7128 in the meantime or would you rather I get descriptions/5 out of the way first?
[21:07] <thumper> we can discuss now
[21:09] <thumper> mattyw_: noticed why it was passing
[21:09] <mattyw_> thumper, the reason the test was passing without me having done the serialisation code I think is because it all comes down to s.importModel. That calls state.Export and state.Import. But at no point does that hit yaml
[21:10] <thumper> because the import/export tests don't serialize/deserialize the model
[21:10] <thumper> right
[21:10] <mattyw_> awesome
[21:10] <thumper> it assumes that the model package is tested
[21:10] <thumper> as it should
[21:10] <mattyw_> so if I get the description package sorted is the rest of the stuff in that pr the right approach?
[21:10] <thumper> mattyw_: so when the description PR is done, we are good
[21:11] <thumper> yep
[21:11] <mattyw_> awesome, thanks very much for the help. I'll fix it up now
[21:13] <mup> Bug #1675154 changed: Upgrade not possible from 1.25.6 to 1.25.10  <sts> <juju-core:Invalid> <https://launchpad.net/bugs/1675154>
[21:20] <mattyw_> thumper, ptal https://github.com/juju/description/pull/5/files
[21:43] <mattyw_> thumper, sorry to hassle you - but are you able to take another look?
[21:44] <thumper> mattyw_: sucked into calls now
[21:44] <thumper> I have it up
[21:45] <mattyw_> thumper, ok - I'll go afk for a bit if you could take a look at that and https://github.com/juju/juju/pull/7128/files when you can. I'll keep checking back every so often incase there are things to do
[21:47] <babbageclunk> wallyworld: you about? (I mean, you probably shouldn't be, but if you are...)
[21:47] <wallyworld> babbageclunk: in release call
[21:47] <wallyworld> will be done soon
[21:47] <babbageclunk> wallyworld: ok thanks
[22:07] <wallyworld> babbageclunk: free now
[22:09] <babbageclunk> wallyworld: hey - so jam approved my PR, but in this comment (https://github.com/juju/juju/pull/7126#pullrequestreview-28347395) he suggested deploying to an imported subnet.
[22:09] <wallyworld> looking
[22:10] <babbageclunk> wallyworld: I'd like to do that test, but I'm not sure how to - should I use some kind of placement directive?
[22:10] <wallyworld> babbageclunk: there is a placement directive for subnets, yes
[22:10] <wallyworld> --to subnet=<blah>
[22:10] <wallyworld> i think
[22:10] <wallyworld> where blah is a subnet id or cidr
[22:11] <babbageclunk> wallyworld: I don't think that's implemented in GCE though. It's probably not that much work to add...
[22:11] <wallyworld> babbageclunk: in that case my view is we land what you have
[22:11] <babbageclunk> ok, doing that
[22:12] <wallyworld> babbageclunk: we can add a card to the board to remind us to come back and check
[22:13] <babbageclunk> ok - I'll do that
[22:14] <wallyworld> hml: did you end up using the standalone mysql charm or your testing? i assume that's all gone ok?
[22:14] <hml> wallyworld: yes, i used the standalone mysql - and everything worked.
[22:14] <wallyworld> yay
[22:14] <hml> wallyworld: added the tests, one final sanity check and i’ll do the PR
[22:15] <wallyworld> let;s mak sure a bug is filed against the percona cluster charm
[22:15] <hml> wallyworld: okay, will file - anything specific i should say beyond the obvious
[22:16] <wallyworld> not really
[22:30] <babbageclunk> wallyworld: should I move on to storing network for the subnet in state? or something else
[22:30] <wallyworld> babbageclunk: would be good to round out the current gce work, agree?
[22:31] <babbageclunk> wallyworld: yeah, I think so
[22:31] <wallyworld> should hopefully be fairly straightforward
[22:31] <babbageclunk> ok, doing that.
[22:45] <thumper> mattyw_: description PR merged
[22:47] <babbageclunk> wallyworld: hmm - should the network name be part of the subnet's document ID?
[22:47] <babbageclunk> wallyworld: It should, right? We could have the same cidr in different networks.
[22:48] <wallyworld> babbageclunk: it depends what's needed to make the id unique and what asserts are used
[22:48] <wallyworld> what do we do for ec2?
[22:48] <babbageclunk> wallyworld: the id at the moment is just the CIDR.
[22:48] <wallyworld> right, that sounds wrong
[22:49] <babbageclunk> at the moment we don't capture vpc-id in ec2
[22:49] <wallyworld> or maas?
[22:50] <babbageclunk> No, jam was saying maas doesn't have that concept. (Although what are fabrics in maas?)
[22:51] <babbageclunk> oh, no - fabrics aren't the same as different networks. https://docs.ubuntu.com/maas/2.1/en/intro-concepts#fabrics
[22:51] <mattyw_> thumper, many thanks, how's the core one now? https://github.com/juju/juju/pull/7128
[22:52] <thumper> mattyw_: need to update description hash for dependencies
[22:53] <thumper> mattyw_: rest looks good
[22:53] <wallyworld> babbageclunk: it looks like the state model is wrong - it assumes cidr is unique
[22:54] <wallyworld> babbageclunk: which may be true for maas
[22:54] <wallyworld> but you are sying is not true for gce?
[22:55] <mattyw_> thumper, I think the old hash was ok - but I've updated it now anyway
[22:56] <babbageclunk> wallyworld: I don't think it's true for GCE no, hang on, checking docs
[22:56] <thumper> mattyw_: well, you should always refer to the mainline hash :)
[22:56] <mattyw_> true
[22:56] <thumper> mattyw_: LGTM
[22:57] <mattyw_> thumper, many thanks
[22:58] <babbageclunk> wallyworld: https://cloud.google.com/compute/docs/networking#ip_ranges
[22:58] <babbageclunk> wallyworld: ranges must be unique and non-overlapping within a network
[22:58] <wallyworld> babbageclunk: hence cidr as doc id is ok then, right?
[22:59] <babbageclunk> wallyworld: but couldn't we have subnets from different networks?
[23:00] <babbageclunk> Or should I not worry about that for now?
[23:00] <wallyworld> maybe, but i'm not familiar enough with the juju network data model. they did the current model for a reason i assume
[23:01] <wallyworld> i'd just stick with the convention that's been done for other existing providers
[23:01] <wallyworld> for now
[23:01] <babbageclunk> wallyworld: I'm talking about in state - I don't think it's anything to do with providers at that point, is it?
[23:02] <wallyworld> right, i'm talking about in state too - but the providers set the data to go into state
[23:02] <wallyworld> so what's in state needs to be able to model what the providers provider
[23:02] <babbageclunk> ha
[23:03] <babbageclunk> Ok, so I won't worry about trying to incorporate the network id into the doc id for now, we can do that later. More important to capture it for now.
[23:03] <babbageclunk> wallyworld: ^
[23:04] <wallyworld> babbageclunk: i'm still not sure of why you think we need to do that just for gce?
[23:05] <babbageclunk> I don't think we do jsut for GCE.
[23:06] <wallyworld> ok, so i don't think we should be second guessing the network model at this point. let's implement for gce something consistent with what's already been done. the model needs to be lookeed at holistically perhaps
[23:06] <babbageclunk> ok
[23:06] <wallyworld> as part of the ongoing work to implement spaces etc
[23:35] <anastasiamac> hml: ping
[23:35] <hml> anastasiamac: ack
[23:35] <anastasiamac> hml: could u do a quick hangout?
[23:36] <anastasiamac> hml: we could hog ur team's standup one ;D
[23:36] <hml> anastasiamac: sure
[23:37] <anastasiamac> hml: k. m there :)