| axw | wallyworld: did I miss anything this morning? | 01:07 |
|---|---|---|
| wallyworld | axw: not really, i can fill you in if you wanted a qick HO | 01:07 |
| axw | wallyworld: ok, see you at standup | 01:07 |
| wallyworld | with site news etc | 01:07 |
| menn0 | babbageclunk: ping? | 02:03 |
| babbageclunk | menn0: hey | 02:03 |
| menn0 | babbageclunk: so I think your hacked mgoprune takes care of a case which mgopurge doesn't handle (as you found) | 02:04 |
| babbageclunk | menn0: ok | 02:04 |
| babbageclunk | menn0: I mean, it just reports them, doesn't do any cleanup | 02:05 |
| menn0 | babbageclunk: you were seeing Insert ops in the txns collection which didn't have a matching entry in stash - is that right? | 02:05 |
| babbageclunk | menn0: they had a stash record, but that stash didn't have the txnid_nonce in txn-queue | 02:06 |
| menn0 | hmmm ok | 02:06 |
| babbageclunk | menn0: It was raising the error from flusher.go:475 | 02:07 |
| menn0 | babbageclunk: the existing PurgeMissing function goes through the stash and looks for txn-queue entries there which don't have a matching txns doc | 02:08 |
| menn0 | babbageclunk: but this is the other way around | 02:08 |
| babbageclunk | menn0: right | 02:09 |
| menn0 | babbageclunk: I guess we should extend mgopurge to handle this | 02:09 |
| babbageclunk | menn0: yeah, that was my thinking | 02:11 |
| menn0 | babbageclunk: how did you fix the issue? | 02:11 |
| babbageclunk | menn0: The bad transactions were all leadership lease updates (I think), so blahdeblah thought it made more sense to remove them. | 02:13 |
| babbageclunk | menn0: so we removed the txn id from the txn-queue of the *other* record and then removed the txn record. | 02:13 |
| babbageclunk | i.e. all of the txns were an update to one record and an insert. | 02:14 |
| babbageclunk | menn0: I thought the other way to fix it was to insert the txn id into the start of the stash record that was missing it. That's probably the more general fix. | 02:15 |
| babbageclunk | (Although we didn't try it, so I guess we can't be sure it's actually rignt.) | 02:16 |
| menn0 | babbageclunk: that might work but it seems riskier... I wonder if it's safer to just remove the badness | 02:17 |
| babbageclunk | menn0: that was the thinking in this case, although it might not only happen with this specific type of (pretty disposable) transaction | 02:18 |
| menn0 | babbageclunk: true... i'll have a play around | 02:19 |
| blahdeblah | babbageclunk: blame me, why don't you? :-P | 02:21 |
| babbageclunk | menn0: maybe the fact that we've only seen it with leadership-related transactions means it's something to do with the way we're handling them? Or maybe it's just that we have lots of them, so that's where it turns up. | 02:21 |
| menn0 | I think it's the latter | 02:21 |
| menn0 | babbageclunk: but it would be great to understand why these problems at all | 02:22 |
| menn0 | happen at all | 02:22 |
| menn0 | babbageclunk: we've never really managed to figure that out... it's either mongodb screwing up or a bug in mgo | 02:22 |
| babbageclunk | blahdeblah: :) I was just trying to lend the decision some weight by adding your name to it! | 02:22 |
| === akhavr1 is now known as akhavr | ||
| blahdeblah | I'm not fat; my chest just slipped a bit | 02:38 |
| axw | wallyworld: I've found some nice low hanging fruit for reducing the number of active sessions. every agent (machine & unit) has a session open for the lifetime of the agent's connection, for logging to the db | 04:22 |
| wallyworld | yeah, i think it's 2 per agent? | 04:23 |
| axw | wallyworld: I'm thinking we might want to apply jam's idea of aggregating writes into a bulk insertion for presence here as well | 04:23 |
| axw | wallyworld: just 1 AFAICS | 04:23 |
| wallyworld | sgtm | 04:23 |
| babbageclunk | wallyworld, axw: seen the message about mongo replication in #juju@irc.canonical.com? Any ideas? | 04:34 |
| axw | babbageclunk: I'm afraid not | 04:35 |
| axw | babbageclunk: I mean I've seen it now, I don't have any good ideas | 04:35 |
| jam | axw: wallyworld: "reducing the number of active sessions", is that closing the session while its active, or doing all of the rights by Cloning the one session per agent? | 04:37 |
| jam | axw: wallyworld: I'd also say aggregating presence should certainly be 2.2.1 not delay 2.2rc3/2.2.0 | 04:38 |
| wallyworld | no, it won't delay | 04:38 |
| wallyworld | it will be 2.2.1 | 04:38 |
| jam | I'm off today for my wife's birthday, so I won't be around much | 04:38 |
| wallyworld | have fun | 04:38 |
| axw | jam: if we aggregate, then either Copy() for the lifetime of the aggregator, or Copy() just before doing a bulk insert, and close straight after. I'm not sure which is best yet | 04:39 |
| wallyworld | site looks ok atm | 04:39 |
| jam | axw: I'd copy before insert, given if we bulk updating, we shouldn't have a lot of those active. | 04:39 |
| jam | axw: I was trying to find a nice number for the frequency, given its a 30s window, it feels like we could batch all the way up to 1s intervals | 04:39 |
| * jam goes to take the dog out | 04:40 | |
| axw | jam: that sounds reasonable. it could be made adaptive too, i.e. insert more frequently if the incoming rate is slower | 04:40 |
| * axw waves | 04:40 | |
| jam | axw: the other option is that we just aggregate whatever we get until the last update finishes | 04:41 |
| jam | so we only have 1 presence-based write at any time | 04:42 |
| jam | and just splat down whatever updates we've gotten since then | 04:42 |
| * axw nods | 04:42 | |
| anastasiamac | menn0: standup? | 05:03 |
| === frankban|afk is now known as frankban | ||
| axw | wallyworld: do you know if it's ok for us to merge things like https://github.com/juju/docs/pull/1910 into master? or should we wait for evilnick or someone else on the docs team to merge? | 07:18 |
| axw | not sure if there's some special process, don't want to bork things | 07:18 |
| wallyworld | axw: yeah, not sure. for something like that would be nice to merge | 07:34 |
| axw | wallyworld: ok. burton, I'll merge and we can ask for forgiveness... :) | 07:35 |
| burton-aus | axw wallyworld GREAT! | 07:36 |
| axw | wallyworld: no rush since it won't be landing until >2.2, but https://github.com/juju/juju/pull/7496 is a prereq for a follow-up db logger session reduction PR | 08:40 |
| wpk | netpln--help | 08:54 |
| wallyworld | axw: just getting dinner, will look after | 09:05 |
| wpk | wallyworld: btw, I got your message, thank you, I'm still looking for a hotel in Brisbane (but there are rooms available, I'll book something in the next few days) | 09:10 |
| wallyworld | wpk: great, let me know if you need any help etc. i've booked you an extra 2 nights a the venue and then monday night will be in Brisbane | 09:28 |
| wpk | Is there a way to check if goyaml.Unmarshal processed everything? | 09:42 |
| wpk | I see an error returned if there's a field of wrong type, but what about field that is in yaml but not in structure? | 09:42 |
| mup | Bug #1697664 opened: Juju 2.X using MAAS, juju status hangs after juju controller reboot <juju-core:New> <https://launchpad.net/bugs/1697664> | 10:37 |
| wpk | anyone familiar with yaml.v2 ? | 11:50 |
| stub | wpk: Fields in the yaml and not in the structure get dropped. I think if you don't want that, you need to unmarshal to a mapping and handle it yourself. | 12:05 |
| wpk | stub: I added an UnmarshalStrict method and PRed | 12:05 |
| wpk | Other thing is that yamllint says that a list marshaled by goyaml is badly indented | 12:08 |
| === frankban is now known as frankban|afk | ||
| babbageclunk | What's the best way to generate a lot of status updates for a model? Can I do it with juju run? | 23:15 |
| babbageclunk | Or do we have a charm that will do it? | 23:16 |
| babbageclunk | Asking for a friend. | 23:16 |
| babbageclunk | wallyworld, menn0: ^ | 23:16 |
| wallyworld | babbageclunk: you mean the hook execution? | 23:17 |
| babbageclunk | wallyworld: Oh, will any hook execution generate updates? So I can just use peer-xplod? | 23:18 |
| wallyworld | babbageclunk: there needs to be an actual implementation of that hook to then write values | 23:19 |
| wallyworld | and juju only runs the hook once every 5 minutes | 23:19 |
| wallyworld | or are you talking about unit status history log? | 23:20 |
| babbageclunk | wallyworld: Hmm, that'll take too long. I guess I should make a charm that just sits in a loop updating status? | 23:20 |
| wallyworld | rather than unit/machine status values | 23:20 |
| babbageclunk | wallyworld: For the pruner it doesn't matter - just want to generate megs of updates in status history, right? | 23:21 |
| wallyworld | oh, i thought you were talking about status values | 23:21 |
| wallyworld | "status updates" | 23:21 |
| wallyworld | you can always hack a charm as you suggest | 23:22 |
| wallyworld | or use juju run | 23:22 |
| babbageclunk | <squints> I don't understand the difference between status values ands status updates. | 23:22 |
| babbageclunk | Oh, you mean the update status hook. | 23:23 |
| wallyworld | sorry, my brain was differentiating between what goes into the status collection vs status history collection | 23:23 |
| wallyworld | it's all very confusing, similar terminology | 23:24 |
| babbageclunk | Gotcha, sorry. Yeah, I want to fill up the history so that I can QA my pruning work. | 23:24 |
| wallyworld | you want to test history pruning | 23:24 |
| babbageclunk | I'm not really asking for a friend, that was a joke. :) | 23:24 |
| wallyworld | lol | 23:24 |
| wallyworld | i'd just do a juju run inside a bash loop? | 23:25 |
| wallyworld | you can execute the status-set hook tool inside juju run | 23:25 |
| babbageclunk | Ah, awesome - playing with that now - thanks! | 23:25 |
| menn0 | babbageclunk: easy review please: https://github.com/juju/txn/pull/37 | 23:37 |
| babbageclunk | menn0: lookign | 23:38 |
| babbageclunk | ng | 23:38 |
| babbageclunk | menn0: Approved | 23:40 |
| menn0 | babbageclunk: cheers | 23:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!