| wgrant | blr: Hi | 00:05 |
|---|---|---|
| blr | hello! | 00:06 |
| wgrant | blr: Colin can't do his Monday evening next week, so how does your Wednesday morning work for a meeting? | 00:06 |
| blr | sure | 00:06 |
| cjwatson | I think that works here. Kirsten's sometimes out on Tuesdays which leaves me looking after the children, but if we're talking about something like 23:00 London then that would normally be OK, I think. | 00:21 |
| cjwatson | Of course the difference shifts radically between winter and summer ... | 00:22 |
| cjwatson | 2300 London is, what, 1000 for wgrant and 1200 for blr? | 00:23 |
| wgrant | I think so. | 00:23 |
| wgrant | I can easily do 3-4h earlier, but I suspect that makes it clashier for you. | 00:23 |
| cjwatson | Yeah, that's kind of pessimal here | 00:24 |
| blr | 11 for me currently I think | 00:24 |
| wgrant | 90 minutes ago, so midday for you. | 00:24 |
| cjwatson | We should probably replan in summer rather than trying to find something that has 2h of tolerance with no practical experience | 00:24 |
| wgrant | Definitely. | 00:24 |
| cjwatson | Silly planet | 00:24 |
| cjwatson | Well, silly DST | 00:25 |
| blr | timezones are such a nuisance. | 00:26 |
| wgrant | We should all move to Queensland. | 00:26 |
| * blr plays the NZ gigabit fibre trump card | 00:26 | |
| wgrant | Yeah yeah, you and your marginally less insane government. | 00:26 |
| blr | _marginally_ | 00:27 |
| StevenK | Gigabit fibre with a silly name | 00:28 |
| blr | well cjwatson and wgrant, I can try to be flexible around times, tend to be up fairly early | 00:30 |
| blr | and happy to hop on a hangout late if that also works on the other side | 00:31 |
| cjwatson | I think there's few enough of us that we can have a bit of slippage | 00:31 |
| wgrant | We'll have more in APAC soon to swing it our way :P | 00:31 |
| cjwatson | Yeah, if anything I'm more likely to find it occasionally convenient to be later | 00:31 |
| cjwatson | Chances of having the children in bed before 10 < chances of Australia having a sane government | 00:32 |
| blr | hahah | 00:32 |
| wgrant | Heh | 00:32 |
| cjwatson | But we can see how it goes | 00:33 |
| cjwatson | Night | 00:34 |
| wgrant | Night | 00:34 |
| blr | g'night | 00:34 |
| wgrant | Hrm | 00:46 |
| wgrant | So the librarian leak is not technically a leak at all. The HTTPConnection cycles are collectable, it's just that the GC doesn't run in time at a particular load level. | 00:47 |
| wgrant | Locally, 200rps is fine but 100rps eventually dies | 00:47 |
| thomi | ohai | 03:41 |
| blr | o/ | 03:46 |
| wgrant | Morning. | 03:47 |
| wgrant | thomi: Your fix is on qastaging now, btw. Please check that it does what it says and doesn't break anything obvious, then change the bug tag from qa-needstesting to qa-ok. | 03:47 |
| wgrant | http://lpqateam.canonical.com/qa-reports/deployment-stable.html will then be happy and green | 03:47 |
| thomi | wgrant: ok, stupid question... where is qastaging? | 03:48 |
| wgrant | thomi: The aptly named qastaging.launchpad.net | 03:48 |
| wgrant | It's a snapshot of the prod DB that is updated when I get around to it, automatically running the latest tested code. | 03:48 |
| thomi | ahh | 03:49 |
| thomi | latest _untested_ code? | 03:49 |
| wgrant | Tested, but not manually QAed. | 03:49 |
| thomi | gotchya | 03:49 |
| wgrant | We land code to devel, http://lpbuildbot.canonical.com/waterfall runs the test suite over any new revisions. | 03:50 |
| wgrant | Assuming all the tests pass, a bot pulls devel into stable. | 03:50 |
| wgrant | qastaging polls stable every couple of minutes and updates whenever it changes. | 03:50 |
| wgrant | deployment-stable.html lists anything that's on qastaging but not on production. | 03:50 |
| thomi | cool | 03:51 |
| thomi | wgrant: I'm getting timeouts on qastaging. I sure hope that's not due to my code... | 03:51 |
| wgrant | thomi: No, qastaging's just on a much smaller DB server | 03:52 |
| wgrant | Only 32GiB of RAM | 03:52 |
| thomi | ok | 03:52 |
| wgrant | It'll take a few refreshes for the bugs homepage to load, if it ever does. | 03:52 |
| thomi | that makes me a little bit sad on the inside | 03:52 |
| wgrant | Yeah | 03:52 |
| thomi | I have like 8GiB or RAM sitting around, I'll mail them to you :D | 03:52 |
| wgrant | But qastaging's resourcing can never quite be a valid match for prod, simply because it has very little load. | 03:52 |
| wgrant | So even if it had 128GiB of RAM like prod, there'd be less contention so it would be implausibly fast. | 03:53 |
| blr | wgrant: would you have a moment to re-review the verbose-diff branch again next week, or did you have some concerns around unhandled edge cases still? | 03:53 |
| wgrant | blr: I think it's probably good, but I need to torture test it. | 03:54 |
| wgrant | I'll get to it on Monday unless the sky keeps falling :) | 03:55 |
| wgrant | Thanks for working through the bzrlib etc. changes. I think we have a good solution now. | 03:55 |
| blr | ok, it would just potentially be forgotten amoungst all the git work, so thought I would mention it :) | 03:55 |
| wgrant | I had planned to do it before you got back, but production burning down has had me unfortunately distracted. | 03:56 |
| blr | hah yes | 03:56 |
| blr | and what was the deal with the spam thing? | 03:57 |
| wgrant | Just a few incompetent spammers from Egypt trying to find gaps. | 03:57 |
| wgrant | They really like advertising Arabs Got Talent. | 03:57 |
| blr | launchpad seems like an unlikely vector heh weird | 03:57 |
| wgrant | Though my Arabic isn't very good, so I may be misinterpreting. | 03:57 |
| wgrant | It's a particularly odd vector for non-English spam, since the vast majority of the content is English and anything else is very detectable. | 03:58 |
| blr | they should contribtute to translations as penance | 03:59 |
| wgrant | Heh | 03:59 |
| thomi | right - I'm off for the day. | 04:30 |
| thomi | Will be at LCA next week, so wgrant gets a week's respite from my annoying questions. | 04:31 |
| thomi | see y'all in the future! | 04:31 |
| blr | yep, about to pass out myself. wgrant the routing in cornice/pyramid is a refreshing change from django :) | 04:41 |
| stub | wgrant: The explicit close branch should sort the need for garbage collection under load, assuming requests.Session.close() actually closes the sockets and doesn't leave that for the garbage collector... | 05:33 |
| stub | wgrant: the http_connection is set (if it doesn't already exist) for all actions afaics, not just on retries. | 05:34 |
| stub | wgrant: But maybe we are more confident with just bumping up the fd limit and not worrying about switching to an untested-by-us python-swiftclient at this stage? | 05:35 |
| stub | We would need over 100 *errors* per second to trigger the issue, wouldn't we? I guess a pentest or something might generate that many 404s. | 05:42 |
| wgrant | stub: the explicit close branch doesn't reliably fix it locally. I'm not sure why. | 05:49 |
| wgrant | Where do you get the 100 errors per second number? | 05:49 |
| wgrant | It mostly depends on how often gc.collect() is invoked. | 05:49 |
| stub | <wgrant> 07:47:36> So the librarian leak is not technically a leak at all. The HTTPConnection cycles are collectable, it's just that the GC doesn't run in time at a particular load level. | 05:50 |
| stub | <wgrant> 07:47:49> Locally, 200rps is fine but 100rps eventually dies | 05:50 |
| stub | Oh, right | 05:50 |
| stub | load is load | 05:50 |
| stub | If the close() branch does nothing, I guess we want to drop it (at least for now) to avoid unnecessary changes. | 05:51 |
| stub | I'd rather not sabotage this Swift cutover unnecessarily | 05:52 |
| stub | With the 404's going back to the pool, as they were supposed too, the other errors should be so infrequent that it won't matter if gc is slow. | 05:54 |
| wgrant | Right, that's my theory. | 05:54 |
| stub | Cool. Then we are done if reality agrees, apart from landing a one line patch. | 05:55 |
| wgrant | I wonder if it's worth looking at the new swiftclient to see if we can use its internal multithreading support rather than creating hundreds of them. | 05:55 |
| wgrant | Anyway, this seems to work for now. | 05:55 |
| stub | We should only ever create 10 of them, really | 05:56 |
| stub | I didn't know enough about twisted internals to be more clever. eg. are deferToThread threads reused forever or a limited number of times? If the latter, threading.locals will leak. | 05:58 |
| cjwatson | wgrant: \o/ working split-out txpkgupload | 16:55 |
| cjwatson | passes all tests, successful ftp and sftp uploads | 16:56 |
| cjwatson | authenticating against local LP authserver | 16:56 |
| cjwatson | shall I release lazr.sshserver 0.1 and proceed with removing that from the LP tree? | 16:57 |
| cjwatson | lp:txpkgupload exists if you want to give it a once-over. | 16:57 |
| cjwatson | I expect I'll tweak things a bit as I attempt to split it out of LP and think about deployment. | 17:02 |
| cjwatson | The only thing I've noticed so far is that I probably want to make twistd --logfile DTRT rather than requiring YAML log configuration. | 19:14 |
| cjwatson | I think I have roughly suitable puppet branches for deploymgr config and for switching over globally. Need to think about the best deployment ordering at some point when it isn't 7pm on a Friday. | 19:16 |
| wgrant | cjwatson: Oh, excellent. | 22:42 |
| wgrant | cjwatson: We should probably upgrade txlongpoll while we're looking at that sort of stuff. Prod's currently using the pre-YAML version. | 22:43 |
| wgrant | What tweaking do you envisage as you split it out of LP? | 22:44 |
| wgrant | I'd get that working before releasing lazr.sshserver, just because there's no real reason to release beforehand and it shouldn't take long. | 22:44 |
| cjwatson | wgrant: I haven't noticed anything other than making the logging a bit more graceful so far (the mechanics for this changed when switching from TAC to plugins and there were a few ways I could have done it), but I haven't played with it very much yet as I only got it working pretty close to EOD. | 23:51 |
| cjwatson | pre-YAML> useful to know, that means I can't look to it for advice ;-) | 23:51 |
| cjwatson | I need to figure out where to put configs for different installations. Possibly just different command-line options from the init script. | 23:52 |
| wgrant | Yeah. | 23:54 |
| wgrant | txlongpoll is also packaged for use by MAAS, but I suspect it doesn't have its own initscript. | 23:55 |
| cjwatson | -rw-r--r-- root/root 404 2012-03-14 16:14 ./etc/init/txlongpoll.conf | 23:55 |
| cjwatson | exec /usr/bin/twistd -n --pidfile=/run/txlongpoll.pid --logfile=/var/log/txlongpoll.log txlongpoll --config-file=/etc/txlongpoll.yaml | 23:56 |
| cjwatson | I don't think it's worthwhile packaging this though | 23:57 |
| wgrant | No, just mentioning it for possible inspiration in terms of config handling. | 23:57 |
| cjwatson | Mm. I based txpkgupload's plugin pretty directly on it. | 23:58 |
| cjwatson | But I should check back for how their logging works, since I see --logfile there. | 23:58 |
| cjwatson | Should be able to release to download-cache and PyPI on Monday, anyway. | 23:59 |
| wgrant | Yeah, I'll look over them both on my Monday, but from what I've seen they're all good. | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!