[00:05] <wgrant> blr: Hi
[00:06] <blr> hello!
[00:06] <wgrant> blr: Colin can't do his Monday evening next week, so how does your Wednesday morning work for a meeting?
[00:06] <blr> sure
[00:21] <cjwatson> I think that works here.  Kirsten's sometimes out on Tuesdays which leaves me looking after the children, but if we're talking about something like 23:00 London then that would normally be OK, I think.
[00:22] <cjwatson> Of course the difference shifts radically between winter and summer ...
[00:23] <cjwatson> 2300 London is, what, 1000 for wgrant and 1200 for blr?
[00:23] <wgrant> I think so.
[00:23] <wgrant> I can easily do 3-4h earlier, but I suspect that makes it clashier for you.
[00:24] <cjwatson> Yeah, that's kind of pessimal here
[00:24] <blr> 11 for me currently I think
[00:24] <wgrant> 90 minutes ago, so midday for you.
[00:24] <cjwatson> We should probably replan in summer rather than trying to find something that has 2h of tolerance with no practical experience
[00:24] <wgrant> Definitely.
[00:24] <cjwatson> Silly planet
[00:25] <cjwatson> Well, silly DST
[00:26] <blr> timezones are such a nuisance.
[00:26] <wgrant> We should all move to Queensland.
[00:26]  * blr plays the NZ gigabit fibre trump card
[00:26] <wgrant> Yeah yeah, you and your marginally less insane government.
[00:27] <blr> _marginally_
[00:28] <StevenK> Gigabit fibre with a silly name
[00:30] <blr> well cjwatson and wgrant, I can try to be flexible around times, tend to be up fairly early
[00:31] <blr> and happy to hop on a hangout late if that also works on the other side
[00:31] <cjwatson> I think there's few enough of us that we can have a bit of slippage
[00:31] <wgrant> We'll have more in APAC soon to swing it our way :P
[00:31] <cjwatson> Yeah, if anything I'm more likely to find it occasionally convenient to be later
[00:32] <cjwatson> Chances of having the children in bed before 10 < chances of Australia having a sane government
[00:32] <blr> hahah
[00:32] <wgrant> Heh
[00:33] <cjwatson> But we can see how it goes
[00:34] <cjwatson> Night
[00:34] <wgrant> Night
[00:34] <blr> g'night
[00:46] <wgrant> Hrm
[00:47] <wgrant> So the librarian leak is not technically a leak at all. The HTTPConnection cycles are collectable, it's just that the GC doesn't run in time at a particular load level.
[00:47] <wgrant> Locally, 200rps is fine but 100rps eventually dies
[03:41] <thomi> ohai
[03:46] <blr> o/
[03:47] <wgrant> Morning.
[03:47] <wgrant> thomi: Your fix is on qastaging now, btw. Please check that it does what it says and doesn't break anything obvious, then change the bug tag from qa-needstesting to qa-ok.
[03:47] <wgrant> http://lpqateam.canonical.com/qa-reports/deployment-stable.html will then be happy and green
[03:48] <thomi> wgrant: ok, stupid question... where is qastaging?
[03:48] <wgrant> thomi: The aptly named qastaging.launchpad.net
[03:48] <wgrant> It's a snapshot of the prod DB that is updated when I get around to it, automatically running the latest tested code.
[03:49] <thomi> ahh
[03:49] <thomi> latest _untested_ code?
[03:49] <wgrant> Tested, but not manually QAed.
[03:49] <thomi> gotchya
[03:50] <wgrant> We land code to devel, http://lpbuildbot.canonical.com/waterfall runs the test suite over any new revisions.
[03:50] <wgrant> Assuming all the tests pass, a bot pulls devel into stable.
[03:50] <wgrant> qastaging polls stable every couple of minutes and updates whenever it changes.
[03:50] <wgrant> deployment-stable.html lists anything that's on qastaging but not on production.
[03:51] <thomi> cool
[03:51] <thomi> wgrant: I'm getting timeouts on qastaging. I sure hope that's not due to my code...
[03:52] <wgrant> thomi: No, qastaging's just on a much smaller DB server
[03:52] <wgrant> Only 32GiB of RAM
[03:52] <thomi> ok
[03:52] <wgrant> It'll take a few refreshes for the bugs homepage to load, if it ever does.
[03:52] <thomi> that makes me a little bit sad on the inside
[03:52] <wgrant> Yeah
[03:52] <thomi> I have like 8GiB or RAM sitting around, I'll mail them to you :D
[03:52] <wgrant> But qastaging's resourcing can never quite be a valid match for prod, simply because it has very little load.
[03:53] <wgrant> So even if it had 128GiB of RAM like prod, there'd be less contention so it would be implausibly fast.
[03:53] <blr> wgrant: would you have a moment to re-review the verbose-diff branch again next week, or did you have some concerns around unhandled edge cases still?
[03:54] <wgrant> blr: I think it's probably good, but I need to torture test it.
[03:55] <wgrant> I'll get to it on Monday unless the sky keeps falling :)
[03:55] <wgrant> Thanks for working through the bzrlib etc. changes. I think we have a good solution now.
[03:55] <blr> ok, it would just potentially be forgotten amoungst all the git work, so thought I would mention it :)
[03:56] <wgrant> I had planned to do it before you got back, but production burning down has had me unfortunately distracted.
[03:56] <blr> hah yes
[03:57] <blr> and what was the deal with the spam thing?
[03:57] <wgrant> Just a few incompetent spammers from Egypt trying to find gaps.
[03:57] <wgrant> They really like advertising Arabs Got Talent.
[03:57] <blr> launchpad seems like an unlikely vector heh weird
[03:57] <wgrant> Though my Arabic isn't very good, so I may be misinterpreting.
[03:58] <wgrant> It's a particularly odd vector for non-English spam, since the vast majority of the content is English and anything else is very detectable.
[03:59] <blr> they should contribtute to translations as penance
[03:59] <wgrant> Heh
[04:30] <thomi> right - I'm off for the day.
[04:31] <thomi> Will be at LCA next week, so wgrant gets a week's respite from my annoying questions.
[04:31] <thomi> see y'all in the future!
[04:41] <blr> yep, about to pass out myself. wgrant the routing in cornice/pyramid is a refreshing change from django :)
[05:33] <stub> wgrant: The explicit close branch should sort the need for garbage collection under load, assuming requests.Session.close() actually closes the sockets and doesn't leave that for the garbage collector...
[05:34] <stub> wgrant: the http_connection is set (if it doesn't already exist) for all actions afaics, not just on retries.
[05:35] <stub> wgrant: But maybe we are more confident with just bumping up the fd limit and not worrying about switching to an untested-by-us python-swiftclient at this stage?
[05:42] <stub> We would need over 100 *errors* per second to trigger the issue, wouldn't we? I guess a pentest or something might generate that many 404s.
[05:49] <wgrant> stub: the explicit close branch doesn't reliably fix it locally. I'm not sure why.
[05:49] <wgrant> Where do you get the 100 errors per second number?
[05:49] <wgrant> It mostly depends on how often gc.collect() is invoked.
 07:47:36> So the librarian leak is not technically a leak at all. The HTTPConnection cycles are collectable, it's just that the GC doesn't run in time at a particular load level.
 07:47:49> Locally, 200rps is fine but 100rps eventually dies
[05:50] <stub> Oh, right
[05:50] <stub> load is load
[05:51] <stub> If the close() branch does nothing, I guess we want to drop it (at least for now) to avoid unnecessary changes.
[05:52] <stub> I'd rather not sabotage this Swift cutover unnecessarily
[05:54] <stub> With the 404's going back to the pool, as they were supposed too, the other errors should be so infrequent that it won't matter if gc is slow.
[05:54] <wgrant> Right, that's my theory.
[05:55] <stub> Cool. Then we are done if reality agrees, apart from landing a one line patch.
[05:55] <wgrant> I wonder if it's worth looking at the new swiftclient to see if we can use its internal multithreading support rather than creating hundreds of them.
[05:55] <wgrant> Anyway, this seems to work for now.
[05:56] <stub> We should only ever create 10 of them, really
[05:58] <stub> I didn't know enough about twisted internals to be more clever. eg. are deferToThread threads reused forever or a limited number of times? If the latter, threading.locals will leak.
[16:55] <cjwatson> wgrant: \o/ working split-out txpkgupload
[16:56] <cjwatson> passes all tests, successful ftp and sftp uploads
[16:56] <cjwatson> authenticating against local LP authserver
[16:57] <cjwatson> shall I release lazr.sshserver 0.1 and proceed with removing that from the LP tree?
[16:57] <cjwatson> lp:txpkgupload exists if you want to give it a once-over.
[17:02] <cjwatson> I expect I'll tweak things a bit as I attempt to split it out of LP and think about deployment.
[19:14] <cjwatson> The only thing I've noticed so far is that I probably want to make twistd --logfile DTRT rather than requiring YAML log configuration.
[19:16] <cjwatson> I think I have roughly suitable puppet branches for deploymgr config and for switching over globally.  Need to think about the best deployment ordering at some point when it isn't 7pm on a Friday.
[22:42] <wgrant> cjwatson: Oh, excellent.
[22:43] <wgrant> cjwatson: We should probably upgrade txlongpoll while we're looking at that sort of stuff. Prod's currently using the pre-YAML version.
[22:44] <wgrant> What tweaking do you envisage as you split it out of LP?
[22:44] <wgrant> I'd get that working before releasing lazr.sshserver, just because there's no real reason to release beforehand and it shouldn't take long.
[23:51] <cjwatson> wgrant: I haven't noticed anything other than making the logging a bit more graceful so far (the mechanics for this changed when switching from TAC to plugins and there were a few ways I could have done it), but I haven't played with it very much yet as I only got it working pretty close to EOD.
[23:51] <cjwatson> pre-YAML> useful to know, that means I can't look to it for advice ;-)
[23:52] <cjwatson> I need to figure out where to put configs for different installations.  Possibly just different command-line options from the init script.
[23:54] <wgrant> Yeah.
[23:55] <wgrant> txlongpoll is also packaged for use by MAAS, but I suspect it doesn't have its own initscript.
[23:55] <cjwatson> -rw-r--r-- root/root       404 2012-03-14 16:14 ./etc/init/txlongpoll.conf
[23:56] <cjwatson> exec /usr/bin/twistd -n --pidfile=/run/txlongpoll.pid --logfile=/var/log/txlongpoll.log txlongpoll --config-file=/etc/txlongpoll.yaml
[23:57] <cjwatson> I don't think it's worthwhile packaging this though
[23:57] <wgrant> No, just mentioning it for possible inspiration in terms of config handling.
[23:58] <cjwatson> Mm.  I based txpkgupload's plugin pretty directly on it.
[23:58] <cjwatson> But I should check back for how their logging works, since I see --logfile there.
[23:59] <cjwatson> Should be able to release to download-cache and PyPI on Monday, anyway.
[23:59] <wgrant> Yeah, I'll look over them both on my Monday, but from what I've seen they're all good.