[01:07] <axw> wallyworld: did I miss anything this morning?
[01:07] <wallyworld> axw: not really, i can fill you in if you wanted a qick HO
[01:07] <axw> wallyworld: ok, see you at standup
[01:07] <wallyworld> with site news etc
[02:03] <menn0> babbageclunk: ping?
[02:03] <babbageclunk> menn0: hey
[02:04] <menn0> babbageclunk: so I think your hacked mgoprune takes care of a case which mgopurge doesn't handle (as you found)
[02:04] <babbageclunk> menn0: ok
[02:05] <babbageclunk> menn0: I mean, it just reports them, doesn't do any cleanup
[02:05] <menn0> babbageclunk: you were seeing Insert ops in the txns collection which didn't have a matching entry in stash - is that right?
[02:06] <babbageclunk> menn0: they had a stash record, but that stash didn't have the txnid_nonce in txn-queue
[02:06] <menn0> hmmm ok
[02:07] <babbageclunk> menn0: It was raising the error from flusher.go:475
[02:08] <menn0> babbageclunk: the existing PurgeMissing function goes through the stash and looks for txn-queue entries there which don't have a matching txns doc
[02:08] <menn0> babbageclunk: but this is the other way around
[02:09] <babbageclunk> menn0: right
[02:09] <menn0> babbageclunk: I guess we should extend mgopurge to handle this
[02:11] <babbageclunk> menn0: yeah, that was my thinking
[02:11] <menn0> babbageclunk: how did you fix the issue?
[02:13] <babbageclunk> menn0: The bad transactions were all leadership lease updates (I think), so blahdeblah thought it made more sense to remove them.
[02:13] <babbageclunk> menn0: so we removed the txn id from the txn-queue of the *other* record and then removed the txn record.
[02:14] <babbageclunk> i.e. all of the txns were an update to one record and an insert.
[02:15] <babbageclunk> menn0: I thought the other way to fix it was to insert the txn id into the start of the stash record that was missing it. That's probably the more general fix.
[02:16] <babbageclunk> (Although we didn't try it, so I guess we can't be sure it's actually rignt.)
[02:17] <menn0> babbageclunk: that might work but it seems riskier... I wonder if it's safer to just remove the badness
[02:18] <babbageclunk> menn0: that was the thinking in this case, although it might not only happen with this specific type of (pretty disposable) transaction
[02:19] <menn0> babbageclunk: true... i'll have a play around
[02:21] <blahdeblah> babbageclunk: blame me, why don't you? :-P
[02:21] <babbageclunk> menn0: maybe the fact that we've only seen it with leadership-related transactions means it's something to do with the way we're handling them? Or maybe it's just that we have lots of them, so that's where it turns up.
[02:21] <menn0> I think it's the latter
[02:22] <menn0> babbageclunk: but it would be great to understand why these problems at all
[02:22] <menn0> happen at all
[02:22] <menn0> babbageclunk: we've never really managed to figure that out... it's either mongodb screwing up or a bug in mgo
[02:22] <babbageclunk> blahdeblah: :) I was just trying to lend the decision some weight by adding your name to it!
[02:38] <blahdeblah> I'm not fat; my chest just slipped a bit
[04:22] <axw> wallyworld: I've found some nice low hanging fruit for reducing the number of active sessions. every agent (machine & unit) has a session open for the lifetime of the agent's connection, for logging to the db
[04:23] <wallyworld> yeah, i think it's 2 per agent?
[04:23] <axw> wallyworld: I'm thinking we might want to apply jam's idea of aggregating writes into a bulk insertion for presence here as well
[04:23] <axw> wallyworld: just 1 AFAICS
[04:23] <wallyworld> sgtm
[04:34] <babbageclunk> wallyworld, axw: seen the message about mongo replication in #juju@irc.canonical.com? Any ideas?
[04:35] <axw> babbageclunk: I'm afraid not
[04:35] <axw> babbageclunk: I mean I've seen it now, I don't have any good ideas
[04:37] <jam> axw: wallyworld: "reducing the number of active sessions", is that closing the session while its active, or doing all of the rights by Cloning the one session per agent?
[04:38] <jam> axw: wallyworld: I'd also say aggregating presence should certainly be 2.2.1 not delay 2.2rc3/2.2.0
[04:38] <wallyworld> no, it won't delay
[04:38] <wallyworld> it will be 2.2.1
[04:38] <jam> I'm off today for my wife's birthday, so I won't be around much
[04:38] <wallyworld> have fun
[04:39] <axw> jam: if we aggregate, then either Copy() for the lifetime of the aggregator, or Copy() just before doing a bulk insert, and close straight after. I'm not sure which is best yet
[04:39] <wallyworld> site looks ok atm
[04:39] <jam> axw: I'd copy before insert, given if we bulk updating, we shouldn't have a lot of those active.
[04:39] <jam> axw: I was trying to find a nice number for the frequency, given its a 30s window, it feels like we could batch all the way up to 1s intervals
[04:40]  * jam goes to take the dog out
[04:40] <axw> jam: that sounds reasonable. it could be made adaptive too, i.e. insert more frequently if the incoming rate is slower
[04:40]  * axw waves
[04:41] <jam> axw: the other option is that we just aggregate whatever we get until the last update finishes
[04:42] <jam> so we only have 1 presence-based write at any time
[04:42] <jam> and just splat down whatever updates we've gotten since then
[04:42]  * axw nods
[05:03] <anastasiamac> menn0: standup?
[07:18] <axw> wallyworld: do you know if it's ok for us to merge things like https://github.com/juju/docs/pull/1910 into master? or should we wait for evilnick or someone else on the docs team to merge?
[07:18] <axw> not sure if there's some special process, don't want to bork things
[07:34] <wallyworld> axw: yeah, not sure. for something like that would be nice to merge
[07:35] <axw> wallyworld: ok. burton, I'll merge and we can ask for forgiveness... :)
[07:36] <burton-aus> axw wallyworld GREAT!
[08:40] <axw> wallyworld: no rush since it won't be landing until >2.2, but https://github.com/juju/juju/pull/7496 is a prereq for a follow-up db logger session reduction PR
[08:54] <wpk>  netpln--help
[09:05] <wallyworld> axw: just getting dinner, will look after
[09:10] <wpk> wallyworld: btw, I got your message, thank you, I'm still looking for a hotel in Brisbane (but there are rooms available, I'll book something in the next few days)
[09:28] <wallyworld> wpk: great, let me know if you need any help etc. i've booked you an extra 2 nights a the venue and then monday night will be in Brisbane
[09:42] <wpk> Is there a way to check if goyaml.Unmarshal processed everything?
[09:42] <wpk> I see an error returned if there's a field of wrong type, but what about field that is in yaml but not in structure?
[10:37] <mup> Bug #1697664 opened: Juju 2.X using MAAS, juju status hangs after juju controller reboot <juju-core:New> <https://launchpad.net/bugs/1697664>
[11:50] <wpk> anyone familiar with yaml.v2 ?
[12:05] <stub> wpk: Fields in the yaml and not in the structure get dropped. I think if you don't want that, you need to unmarshal to a mapping and handle it yourself.
[12:05] <wpk> stub: I added an UnmarshalStrict method and PRed
[12:08] <wpk> Other thing is that yamllint says that a list marshaled by goyaml is badly indented
[23:15] <babbageclunk> What's the best way to generate a lot of status updates for a model? Can I do it with juju run?
[23:16] <babbageclunk> Or do we have a charm that will do it?
[23:16] <babbageclunk> Asking for a friend.
[23:16] <babbageclunk> wallyworld, menn0: ^
[23:17] <wallyworld> babbageclunk: you mean the hook execution?
[23:18] <babbageclunk> wallyworld: Oh, will any hook execution generate updates? So I can just use peer-xplod?
[23:19] <wallyworld> babbageclunk: there needs to be an actual implementation of that hook to then write values
[23:19] <wallyworld> and juju only runs the hook once every 5 minutes
[23:20] <wallyworld> or are you talking about unit status history log?
[23:20] <babbageclunk> wallyworld: Hmm, that'll take too long. I guess I should make a charm that just sits in a loop updating status?
[23:20] <wallyworld> rather than unit/machine status values
[23:21] <babbageclunk> wallyworld: For the pruner it doesn't matter - just want to generate megs of updates in status history, right?
[23:21] <wallyworld> oh, i thought you were talking about status values
[23:21] <wallyworld> "status updates"
[23:22] <wallyworld> you can always hack a charm as you suggest
[23:22] <wallyworld> or use juju run
 I don't understand the difference between status values ands status updates.
[23:23] <babbageclunk> Oh, you mean the update status hook.
[23:23] <wallyworld> sorry, my brain was differentiating between what goes into the status collection vs status history collection
[23:24] <wallyworld> it's all very confusing, similar terminology
[23:24] <babbageclunk> Gotcha, sorry. Yeah, I want to fill up the history so that I can QA my pruning work.
[23:24] <wallyworld> you want to test history pruning
[23:24] <babbageclunk> I'm not really asking for a friend, that was a joke. :)
[23:24] <wallyworld> lol
[23:25] <wallyworld> i'd just do a juju run inside a bash loop?
[23:25] <wallyworld> you can execute the status-set hook tool inside juju run
[23:25] <babbageclunk> Ah, awesome - playing with that now - thanks!
[23:37] <menn0> babbageclunk: easy review please: https://github.com/juju/txn/pull/37
[23:38] <babbageclunk> menn0: lookign
[23:38] <babbageclunk> ng
[23:40] <babbageclunk> menn0: Approved
[23:42] <menn0> babbageclunk: cheers