[04:24] axw: I've added tests, could you take a quick look please: https://github.com/juju/juju/pull/289/files [04:26] looking [04:38] waigani: reviewed [04:38] axw: thanks - I'll mock out ssh.Client === Guest78621 is now known as wallyworld [04:43] axw: hiya, just wanted a quick update on the azure/manual bootstrap issues [04:44] wallyworld: hey! well, I was OCR this morning, so I only got back to it a little while ago [04:44] so far, not really had any insights [04:45] wallyworld: I had an idea about how to speed up bootstrap on azure (start out on ephemeral disk, then migrate over to OS disk at end of bootstrap), but that will probably only paper over the issue [04:45] axw: azure has slower i/o in general so perhaps that might be implicated somehow [04:45] but there seems to be a mongo/juju startup issue doesn't there [04:45] yeah, I'm pretty sure it's related to slow disk, but I don't know how to work around it at the moment... [04:46] seems so, yes [04:46] axw: i can't recall the code - but we added direct=true, simplified things also; I wonder why it's started happening again [04:46] wallyworld: not sure ify ou saw my comment on the bug from late friday. I found some options relating to background indexing that we might want to use, but it doesn't help in this case [04:46] didn't see that yet [04:46] wallyworld: yeah, that didn't seem to help [04:47] if i recall correctly, we seem to dial mongo ok, but then fail when we try to use it [04:47] maybe the dial is returning ok but the db is still initialising? [04:48] wallyworld: yeah, I think the read is just blocked on the backend [04:48] axw: so, maybe be enhance dial to somehow poll for db readiness, whatever that involves [04:49] wallyworld: yep, that is what I am investigating now. [04:50] axw: anyways, need to go grab breakfast, good luck, thank you, and keep me in the loop :-) [04:50] wallyworld: have not looked at manual bootstrap again yet, deferring till this is fixed as it only affects master [04:50] ok [04:50] sure, will do [04:50] enjoy [04:50] will do [06:37] jam, alexisb, hi [06:38] hi dimitern [06:38] did you see your invite? [06:41] alexisb, I did - not sure if I responded, but I'll be there [06:41] dimitern, cool, I will ping you when we start ahangount [06:41] dimitern, other qs for me or jame? [06:41] alexisb, thanks [07:30] jam, I don't have the g+ link - where to join? [07:31] jam1, g+ link? [07:32] dimitern: working on it [07:32] jam1, thanks :) [07:55] axw: ping [07:56] voidspace: pong [07:59] axw: hey, hi [07:59] axw: about closing port 37017 in an upgrade step [07:59] axw: for the azure provider I discussed with you the masking rules changing - causing the firewaller to close the port [08:00] axw: did that only apply to azure, or would it be true of the other providers too? [08:01] if any provider's Ports method returns 37017, the firewaller will close it [08:01] any port that wasn't opened via OpenPorts [08:01] axw, hey [08:01] dimitern: thanks for coming. We'll likely change how we do the meeting. We'll switch to using the conference call for audio [08:01] dimitern: heya [08:01] so that you can hear with the good microphone [08:02] and leave a G+ to do video [08:02] but just mute it [08:02] dimitern: you can call the conference line ok, right? [08:02] axw, catching up with emails, I've seen your question about SetAPIHostPorts and will respond soon, sorry I haven't seen it earlier [08:02] dimitern: no rush, thanks [08:02] jam1, oh really? I had issues with these audio conf calls before [08:03] jam1, I used them before (joined), but the audio quality was very poor [08:03] dimitern: you were having trouble hearing people in the room, so we were hoping to fix those, maybe we could set up a trial run in 30 minutes when we have a break? [08:03] jam1, sgtm [08:03] voidspace: (sorry, forgot to prefix; here's me doing it in case you're waiting for your IRC client to poke you) [08:04] axw: ah, I didn't see - thanks [08:04] jam1, and thanks (if it was you) for scheduling this morning's call for 10:30 when we have our usual 1:1 so I couldn't miss it :) [08:04] dimitern: that was just serendipity, but I'm glad it worked [08:04] axw: I got distracted by a blog post that jml linked to about the difference between concrete and abstract dependencies [08:04] or something like that [08:05] okey dokey :) [08:05] axw: cool - so I need to check the Ports methods on the other providers and check they're not masking StatePort [08:05] jam1, so will there be other calls before the standup? [08:05] thanks [08:05] voidspace: right [08:06] great [08:07] hrm [08:07] voidspace: that was my understanding though, but now I'm confused - openstack doesn't mask the port at all [08:11] dimitern: you mean 1:1's? not today, because I'm sprinting. [08:11] we'll try to test the conference call in 20 mins, though [08:11] jam1, no, I mean other meetings like the last one [08:11] jam1, alright [08:12] dimitern: your next meeting with Mark S is tomorrow morning [08:12] we might ask you to chat during the day [08:12] jam1, no worries, just let me know [08:12] dimitern: will do [08:19] axw: ah [08:19] axw: so we probably need an explicit upgrade close [08:19] axw: or would you prefer further investigation? [08:20] voidspace: I'd prefer the latter. It'd be nice if we can just make use of the existing logic to reconcile ports [08:21] right [08:42] dimitern: so I think we're skipping the conference call, the room just got really noisy during the break [08:42] but likely tomorrow we will just call your phone directly [08:42] rather than doing a conference hangout. [08:42] If we need the hangout, I did get the information [08:43] jam1, my phone? can't we use hangouts somehow, so I can use my headphones? [08:44] well, I suppose I can connect headphones + mic to my phone as well [09:11] axw: voidspace: so I didn't follow the whole conversation yet, but my thoughts were certainly "lets stop creating new things that are broken" rather than worrying about "upgrade" just yet. [09:11] jam1: I've done that bit [09:11] if its confusing, I'd much rather have those split out and land the first part [09:12] jam1: Will gave me an LGTM on that PR, on the condition that I also look at upgrade [09:12] jam1: can't land anything due to critical bugs however [09:12] voidspace: sure [09:12] jam1: but yes, upgrade/port closed as separate PR [09:13] dimitern: so the /etc/network/interfaces stuff needs to be a critical bug for our team [09:13] because it means if you do "juju bootstrap local" and restart your machine, networking doesn't come up [09:13] voidspace: sgtm [09:13] cool [09:13] dimitern: I don't think thumper actually created a bug, but I'm not sure. [09:14] jam1, I'll file a bug and work on it today then [09:15] dimitern: you're welcome to delegate to voidspace or TheMue [09:15] jam1, even better :) [09:28] dimitern: TheMue is gone today, so it is between you and voidspace [09:29] jam1, righto [09:29] voidspace, jam1, I couldn't find a bug for that so far, so I'll file a new critical bug for it soon [09:29] dimitern: there is one [09:29] I'll see if I can dig it up [09:30] dimitern: well, the title and diagnosis don't match, but I'm pretty sure it's the cause of https://bugs.launchpad.net/juju-core/+bug/1349635 [09:30] axw, awesome, thanks! [09:31] nps [09:36] * rogpeppe wonders what's the likelihood of getting something landed in juju-core this week [09:38] wallyworld: azure's apt mirrors were busted before; I told IS, and they fixed it. I've just run CI for azure again and it worked, but there's been no changes to support that [09:38] I guess it's still a bit touchy [09:38] voidspace, are you available to work on bug 1349635 today? [09:38] wallyworld: I've not come up with any useful things to improve, so at this stage I think we should just increase the socket timeout [09:39] axw: remind me, what is the timeout now? [09:39] 20 secs? [09:39] dimitern: ok, after lunch :-) [09:39] voidspace, sure :) [09:39] wallyworld: 21s [09:40] which is 2 x the heartbeat interval [09:40] mup, bug 1349635 [09:40] dimitern: Bug #1349635: Networker shouldn't touch /etc/network/interfaces in a local environment

[09:40] wallyworld: I don't think the heartbeats require a lock though [09:40] wallyworld: db reads will not return if something is holding the lock [09:40] axw: so perhaps at startup of mongo, there's no heartbeat yet so the socket can die [09:41] wallyworld: I'm pretty sure it's just that the response is taking longer than 21s, due to disk being super slow [09:41] voidspace, I've assigned you to the bug and added a kanban card for it as well in the moonstone lane [09:41] dimitern: thankds [09:41] dimitern: I don't like tickets with repro instructions that end "You now have a broken network configuration."... [09:41] axw: yes, that's sort of what I was implying. but once mongo spins up, the heartbeat should keep the socket alive [09:42] axw: so is it worth increasing the timeout just for bootstrap [09:42] voidspace, :) it's actually easier to reproduce, without breaking the network config (at least on my machine it didn't). I'll add comment to the bug [09:42] heh [09:42] as we don't want it too long in practise since we want to know early when mongo has died [09:42] wallyworld: yeah, that's kinda what I was thinking too [09:43] wallyworld: I think if we increase it to 60s for bootstrap we should be safe [09:43] yep, cause that's why it was reduced from 60s or whatever. or was it 10mins [09:43] I think dial timeout was 10 mins [09:43] and socket timeout = dial timeout if you don't specify otherwise [09:44] yeah, and 10 mins is waaaay toooo looong [09:44] so we're only guessing here that's what's causing the bootstrap failures, but it seems plausible [09:44] axw: i'm still unclear why dial seemed to be returning ok though [09:44] wallyworld: dial doesn't need a lock [09:44] and then the subsequent operation failed [09:45] so there was nothing holding the lock? but rather it failed because the lock wasn't created yet? [09:46] wallyworld: I mean, dial would not be affected by something holding a lock [09:46] wallyworld: each db in mongo has a single write lock [09:46] sure, what was holding the lock to prevent the op from running and hence cause the socket to timeout? [09:46] adding an admin user will require that lock [09:47] I don't know. I guess something to do with replica set initiation [09:47] it's pretty difficult to get the information given the time window for failure [09:47] hmmm, so given repliaset can take, what 2 minutes to start, the timeout needs to be at least that long [09:48] wallyworld: that's how long it takes to start, which is what we wait for. I don't know what it's doing afterwards [09:48] i really wish we could block bootstrap until mongo startup was finished [09:49] well I can set the socket timeout to 1 hour, and that'd be like waiting forever ;) [09:50] axw: i wish we could poll every X seconds and each time log "waiting for mongo to start" [09:50] any idea why I can't go get gopkg.in/natefinch/npipe.v2 ? http://paste.ubuntu.com/7950527/ [09:50] dimitern: it's Windows-only [09:51] axw, ah, right - ok, godeps seems happy now [09:51] wallyworld: given that I'm EOD soon, and off for the next three days, I'd have to defer that to someone else [09:51] I can increase the timeout now though [09:52] axw: that would be great, thanks [09:52] enjoy your "holiday" [09:53] wallyworld: thanks :) still sore from pulling up carpet all yesterday... filthy stuff [09:53] can imagine [10:18] morning [10:20] mornin' [10:45] mgz: when you have a moment, can you please review https://github.com/juju/juju/pull/463 [10:48] axw: looks good, do we want a dumb test that checks the timeout is set to the longer value? [10:54] voidspace, the hangout acts funny again, but I think we managed to talk about what was needed, right? [10:55] voidspac_, (in case you missed that) the hangout acts funny again, but I think we managed to talk about what was needed, right? [10:55] dimitern, do you have a couple of minutes for a quick question? [10:55] dimitern: I think so, yes [10:56] mattyw, sure, go ahead [10:56] dimitern: my interenet connection is a bit rubbish I'm afraid [10:56] dimitern, would a hangout be ok? [10:56] dimitern: generally ok, but I have to keep restarting the router [10:56] voidspac_, oh, bugger [10:56] mattyw, certainly - just send me a link [10:56] dimitern: I think we covered everything though, when I need you I will hassle you :-) [10:56] dimitern, https://plus.google.com/hangouts/_/gywto2yelf2av2i4h7epsfy32ma?pqs=1&authuser=0&hl=en [10:57] voidspac_, no worries at all :) [11:09] dimitern, thanks [11:10] mgz: not sure how I'd test it. you can't query an mgo.Session for its timeouts [11:10] open to suggestions [11:11] axw: I was thinking of something dumb like just mocking out all the bits that do anything and asserting that the timeout set function is given the bigger value [11:12] or mock DialWithInfo and assert on the opts [11:12] DialWithInfo doesn't use that option, it's not set until after we get a session [11:12] hmm [11:12] I'll work something out [11:12] your DialWithInfo [11:13] ah [11:13] yeah, I could do that [11:15] it's less to validate the current change, and more so something breaks if we change the logic [11:22] mgz: I've added a test that checks the options passed to agent.InitializeState [11:22] axw: landit [11:24] orrite, bbl [12:52] dimitern, do you have a moment for another question? - should be quick [12:53] mattyw, sure, what do you need? [12:53] morning all [12:54] dimitern, it appears that the unit struct returned from st.AddUnit() doesn't fill in the charmURL properly, I was wondering if this was expected? I added this http://paste.ubuntu.com/7951605/ to state/unit_test.go SetUpTest [12:55] mattyw, I'm not 100% sure, but I think this is the expected behavior, as the uniter is supposed to set the charm url after deployment [12:56] ping mgz [12:57] dimitern, is that the best place to get the charm url from? the metrics need to know which unit they came from so I'm just using unit for that, but they also need to know the charmurl so I thought they should probably get it from there [12:57] bac: hey [12:58] mattyw, the best place to get the charm url is from the unit's service document, as it has to be set once the service gets created (or updated when upgraded) [12:58] dimitern, ok thanks [12:58] nps [12:58] mgz: are you using jenkins-github-lander in your ci? if so, you should update to the newest version. i fixed a bug last week that you'd want. [12:59] bac: thanks, I will [13:02] mgz: increasing this timeout certainly seems to have made the tests unhappy [13:02] :/ [13:02] bac: done [13:02] axw: :( [13:02] can we test-override back small? [13:03] or do we think making it longer also made real-world scenarios bad? [13:04] mgz: I dunno, I'd rather not paper over it. I found a bunch of tests that don't close mgo sessions and such, going to backport it to 1.20 now [13:04] not sure if it'll help any, but it won't hurt [13:04] urg [13:04] you also have a branhc up for that kind of thing, stack on our other fixes needed? [13:05] mgz: yeah that's for master, need to backport it [13:06] axw: sounds good [13:19] anyone with good http-fu know what's going on at line 1503 here? http://golang.org/src/pkg/net/http/server.go#L1503 [13:25] rogpeppe, according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html, "The asterisk "*" means that the request does not apply to a particular resource, but to the server itself, and is only allowed when the method used does not necessarily apply to a resource." [13:26] dimitern: ta! [13:26] dimitern: i should have tried "asterisk" as a search term... [13:27] rogpeppe, yep, I had the same issue initially [13:30] dimitern: voidspace: [13:30] just checking in with you guys [13:30] to see how its going [13:30] mgz: did you get my email about cloudsigma? [13:31] jam1: hi [13:31] jam1: doing ok [13:31] voidspace: did you end up picking up the /etc/network/interfaces bug? [13:31] jam1: working on critical bug 1349635 (delegated by dimitern !) [13:32] jam1: yep [13:32] bug #1349635 [13:32] voidspace: I cancelled our 1:1 today since you're not really working for me right now, unless you'd like to meet [13:32] bah, mup, where are you… :0 [13:32] jam1: think we're just going to have to special case local provider [13:32] jam1, I managed to almost catch up with emails (few more to reply to), decided to do the changes only in the high level networking spec, not the low level one (yet) [13:32] mup: bug #1349635 [13:32] natefinch: I saw [13:32] jam1: Bug #1349635: Networker shouldn't touch /etc/network/interfaces in a local environment

[13:32] natefinch: fine [13:32] natefinch, :) [13:32] dimitern: i think high level first works well, as I think that is what we're focused on discussing in the short term [13:33] jam1: yup, will reply, can do some of the reviewing [13:33] jam1, yeah, as the basic design is likely to change in the coming days, let's first have it discussed and approved, then consolidate the changes in the low level doc [13:47] mgz: landed the timeout increase. having trouble getting 1.20 tests to run on my machine in general, giving up for tonight. see you in a few days [13:47] wallyworld: ^^ [13:47] axw: thanks, I'll try to keep an eye out on th reliability [13:47] axw: thank you for all the work today [13:47] enjoy the break [13:47] cheers [13:48] * axw logs off [13:49] axw: o/ have fun [13:58] updated https://github.com/juju/juju/pull/448 w/ davecheney's comments [14:30] dimitern: ping [14:31] voidspace, hey [14:36] wwitzel3: what was that tool we were using at the sprint to manage containers? [14:40] how can I send ctrl-a beginning-of-line to bash in a debug-hooks tmux session? [14:40] ericsnow: qemu? [14:45] ericsnow: virt-manager [14:45] wwitzel3: thanks [14:47] sinzui: ping/morning [14:47] hi perrito666 [14:48] sinzui: can I borrow a few minutes of your time? [14:48] yes [14:48] sinzui: I am trying to fix the issue with ha timeouting [14:48] now I see the invocation for the test is: test_recovery.py --ha --charm-prefix=local:precise/ /mnt/jenkinshome/jobs/functional-ha-recovery/workspace/extracted-bin/usr/lib/juju-1.21-alpha1/bin test-function-hp-2 [14:49] can I omit the --charm one? [14:49] I do not have a local charm repo [14:49] or in any case what should I ahve on said repo? [14:50] you need the dummy charms... [14:50] perrito666, bzr branch lp:juju-ci-tools/repository [14:51] export JUJU_REPOSITORY=/repository [14:52] perrito666, The test needs to deploy charms and it is specifically looking for the dummy source and sink charms [14:52] excelente branching, tx [14:59] does anyone know what an .lbox file does? contents: "propose -for=lp:goamz -cr" [15:01] katco: lbox is a go cli for reviewboard + lp [15:01] katco: lets you create merge proposal from the command line [15:01] *s [15:03] whit: ty... so a file checked into a repository root would, what, let reviewboard know how to do merges? [15:03] * katco is having difficulty finding any docs on lbox as well. [15:03] katco: uh… I’m not super familiar with the actual mechanics. [15:04] katco: let me fetch a link to the code [15:04] whit: that would be great ty [15:05] katco: some narrative background: http://blog.labix.org/2011/11/17/launchpad-rietveld-happycodereviews [15:05] whit: thank you so much! [15:06] katco: code: https://code.launchpad.net/~niemeyer/lbox/trunk [15:07] katco: I’m sort of an lbox n00b. I’m sure someone here could give you a better intro [15:07] such as the author, niemeyer [15:12] natefinch, axw last commit may have fixed bug 1350911. I am going to rerun the test to verify it is repeatable. [15:19] can I get a LGTM / LBTM on this? https://github.com/juju/juju/pull/448 -- it's for a "high" bugfix [15:22] sinzui: sounds like I may have picked the right bug to assign to myself then ;) [15:29] perrito666, ericsnow bug 1351030 and bug 1351019 might be fixable with axw's last merge into 1.20. [15:30] that would be a blast :) [15:30] axw last commit is? [15:30] sinzui: what's the hash? [15:30] :) [15:31] ericsnow, 2c4db889f1260c8965a88171aeba14f8b21468c2 [15:31] * perrito666 pulls [15:32] * perrito666 tries [15:36] sinzui: could we try to confirm if bug 1350983 is resolved by the bootstrap timeout change? [15:37] mgz, I think we should make a patch and pr with fixes-1350983 in the message [15:37] CI may love it [15:38] sinzui: I will try that. it;s a little complicated by mongo test flakiness, that axw also but some fixes for on trunk, but hasn't gone through yet (as it wasn't actually resolving a blocking bug) [15:38] :( [15:38] so I may put that in first with a jfdi for the test stability first [15:39] sinzui: running tests with axw's fix, in the mean time, lunch [15:47] i don't really understand the remapping but ctrl-a, ctrl-b sends ctrl-a to bash [15:47] natefinch, the bug may still be present. my manual run panicked at UniterSuite.TearDownTest. CI saw the failure and has started retesting [15:49] sinzui: ack. I'll run the tests locally and see what's up === seelaman` is now known as seelaman [16:10] sinzui: no luck with current master so I guess axw's patch did not fix lp:1351030 [16:12] perrito666, :( [16:12] natefinch, I am marking the bug you selected from 1.20 is fixed. It is passing more than 50% of the time, which was the pld rate [16:13] old rate [16:13] woo hoo === Ursinha is now known as Ursinha-afk [16:20] natefinch, ping [16:20] can you and perrito666 cover the cloudbase interlock tomorrow? [16:20] as I will be busy with other stuff [16:20] natefinch, ^^ [16:21] alexisb: I presume I do if you throw me a bit more context :p [16:21] perrito666, let me add you to the invite [16:22] really it is just a time for gabriel to ask questions and for both sides to ensure there is noone waiting on anything [16:22] ah certainly [16:22] book me in [16:23] alrighty I added you and I will leave it on the calendar [16:23] thanks perrito666 ! [16:26] perrito666: thanks [16:27] perrito666: it's no big deal, I'll be there too. [16:28] natefinch: I usually can answer questions, the ocassions when I cannot I just forward people to you or ask myself in turn [16:55] meh, apparently lenovo decided that thinkpad users no longer need 16G of ram :( man Ill never find a decent computer to replace mine [16:57] perrito666: the XPS15 is a nice machine... [16:57] wight? [16:59] perrito666: fwiw http://blog.kate.cox2.name/2014/06/i-was-recently-in-market-for-new-laptop.html [16:59] katco: sweet thank you :) [16:59] perrito666: a hair under 2 KG (just weighed it) That's with the smaller batter [16:59] y [17:00] natefinch: tx for the metric weight I know that hurt a bit inside [17:00] god i wish we used the metric system and 24h format time [17:00] it just makes so much more sense [17:00] yep [17:00] katco: the RAM column is factory default or max possible? [17:01] perrito666: our scale was actually already set to grams... it's what I use for cooking most of the time because doing things in 1/8ths of ounces is dumb [17:01] perrito666: factory default [17:01] natefinch: uh, you do precise cooking? [17:01] I go "by the eye" [17:03] perrito666: I don't even like measuring things by volume, except for liquids and stuff like sugar that is non-compressible and yet fills the available volume. [17:04] natefinch: well I usually do baking by the volume, most recipes are expressed in cups and stuff like that which can be found on measuring jars [17:04] * natefinch has a scale that goes down to a tenth of a gram [17:04] natefinch: nothing that needs to be cooked with that level of precision is legal hre [17:04] yep, and a cup of flour can vary by about 30% depending on how compact it is [17:04] * natefinch has watched way too much Good Eats [17:05] natefinch: no wonder you took us to the most complicataed restaurant ever! ;) [17:05] katco: mac book air shouldn't be on your list. The webcan doesn't work at all :( [17:05] jrwren: lol i did not know that. [17:05] apparently I want an X230 with x240 microprocessor [17:05] jrwren: i suppose it would have gotten reviewed had i landed on that [17:06] katco: Now you know. Other than the lack of webcam, it makes a nice Ubuntu laptop. [17:06] jrwren: well hopefully i won't be in the market anytime soon :) [17:09] honestly I could do with some of the providers saying "this laptop idling consumes this much" bc the battery load itself says nothing [17:09] for what I know the machine could take more energy than iron man [17:10] perrito666: doesn't that say more about the software than the battery? [17:10] katco: true, but some level of benchmark would be useful [17:11] system76 makes machines which runs like a charm with ubunutu, so they could say how much an idling ubuntu runs on those things [17:11] this looks wonderful https://system76.com/laptops/model/galu1 [17:11] perrito666: the only problem i have with that is that i would never feel like i was comparing apples to apples across brands [17:12] katco: well of course, it would be apples vs lenovos vs s76s :p [17:12] perrito666: rofl [17:12] (golf clap) [17:27] I think on my nextr trip I am getting a galago ultra pro, thanks katco [17:34] at what point does /usr/lib/juju/bin/mongod actually get installed? [17:34] comes with the tools [17:35] we have a hard-coded path to /usr/lib/juju/bin/mongod in mongo/mongo.go [17:36] when I run tests it does not exist [17:36] so what in the test suite is supposed to install these tools? [17:37] (and uninstall them when done?) === Ursinha-afk is now known as Ursinha [18:01] perrito666: if i'm there, you can handle my gazelle pro a bit to see if you're satisfied with the build quality === Ursinha is now known as Ursinha-afk [18:08] katco: my wife says that I could also stop treating my computers as if they where anvils [18:08] perrito666: haha [18:09] my most common issue is dropping the laptop on the bag while the bag is on the floor [18:09] therefore hitting the floor [18:10] perrito666: i do that sometimes >.< === Ursinha-afk is now known as Ursinha [18:33] mm sinzui seems to have fled === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha === menn0_ is now known as menn0 [21:39] https://github.com/juju/juju/pull/415 is ready for review... again >_> *pokes fwereade* [21:45] is anyone else playing with ha? [22:23] pop cultural quiz: do know the common expression as "spanner in the works" or "monkey wrench in the works" [22:23] (the reason for this will become clear soon enough) [22:24]