[07:18] landing the last of the sass branches [07:19] wgrant: if you get a mo for a hopefully simple DB review: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/384129 [07:36] tomwardill: Reviewed. [07:37] ta [08:04] blergh, actual test failures for master [08:04] looking now [08:12] Hm, I thought I'd cleaned up the buildbot failures earlier [08:12] Oh a landing broke it, nvm [08:12] yeah, at least one of them is reproducible locally [08:13] ... slightly odd landing to break it, though. [08:13] lib/lp/app/javascript/picker/tests/test_personpicker.html [08:13] JS failure somehow, so not entirely weird [08:14] although it does mean I have to work out how to debug these tests [08:15] You can usually run them in your normal browser just by opening the listed HTML file [08:15] yeah, I've got that [08:18] "pickerPatcherPersonPickerTests" [08:18] ... picked a pickled pepper? [08:54] well, found 2 other bugs, but not fixed that one yet [08:59] found it [09:00] \o/ [09:07] https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/384954 fix css imports in tests [09:07] Making decent progress with the OCI registry upload status UI, but having to fix a few bits of model along the way first [09:08] slightly surprised that change only broke two tests, but there we go. [09:09] r=me [09:11] cjwatson: just pushed a change to flip the order of 'base' and 'components' (and fix the comment), as in the old method the components css was appended to the list [09:11] so base was never actually the last import, despite what the comment said [09:11] I wondered about that. Also fine. [09:12] Oh also [09:12] thanks, landing [09:12] Can you fix the node-sass version in that comment? [09:12] .. not landing [09:12] yes! [09:12] done [09:12] Thanks, go ahead [09:13] * tomwardill -> coffee and then DB patch [10:09] \\ø/ build success [10:11] Will probably be worth having a manual look over qastaging once it updates [10:11] Just to make sure there are no obvious oddities from the uplift [10:12] wgrant: the ocifile date_last_used shouldn't contend too much, I can't see us building enough oci images often enough to hit lock contention on that table [10:12] and the garbage job collects every 7 days, so it shouldn't bloat with 'dead' images either [10:12] cjwatson: yeah, I'll be doing that [10:12] Oh, that's not what I mean by bloat. [10:12] I mean PostgreSQL relation tuple bloat. [10:13] * tomwardill does not know what that means, I'm afraid [10:14] Oh right, Mr. MongoDB Background :) [10:14] it's hard to have bloat when you have no data [10:17] tomwardill: PostgreSQL's MVCC implementation means that a row's data on disk is actually immutable. The physical manifestations of a row are often referred to as "tuples", and contain the actual row data, plus metadata like the transaction IDs where the tuple was created and deleted. An update to a row actually marks the old tuple as deleted and creates a new one, and updates any indexes to point at [10:17] both. VACUUM is the process that runs (usually automatically) to find old tuples that aren't visible to any current transactions, and properly deletes them. [10:18] This works nicely, except when you have long-running transactions like, say, a backup. Since the backup runs in a transaction that references an old snapshot of the DB, tuples that were deleted during the backup cannot be properly deleted until the backup completes. [10:19] This doesn't just cause the table to grow in size - it can also slow things down significantly, as the index will reference all the tuples and queries will have to fetch them all to work out which one is alive. [10:19] This has mostly been a problem on tables like Builder and BuildQueue, where existing rows get updated frequently and automatically by buildd-manager. [10:20] aah, right [10:20] It is also seen occasionally when an appserver hangs in the middle of a transaction for a couple of days. The whole app can start to slow down due to all the dead tuples lying around. [10:22] that makes sense [10:22] I don't think we're going to be updating that table with that level of frequency [10:23] (pending an OCI based team decided they need minutely rebuilds or something) [10:25] But what if there are a million users building things based on ubuntu:18.04! [10:25] (but yes, unlikely to be a problem at the start, and we can always only update it if it's at least an hour old and SKIP LOCKED, or something like that, down the line if we need to) [10:25] But something to think about in future. [10:25] yes, and not something I knew to consider, so thanks for the explanations :) [10:39] https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/384952 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/384957 are a couple of fixes to various bits of the OCI upload arrangements. They're independent, but I'd like to get at least one of them reviewed please so that I can linearise work based on both of them [10:40] looking now [10:53] qastaging is showing a 500 error again [10:58] Right, but this is a normal one with a proper error page [10:59] qastaging doesn't update itself in a nodowntime kind of way, so it routinely does this in the middle of updates [10:59] However - atemoya.canonical.com::qastaging-logs/qastaging-update.log doesn't show it as being in the middle of an update right now [10:59] Oh, hm, the sudoers fix apparently didn't quite take [11:00] -> #launchpad-ops