[00:01] wgrant: http://pastebin.ubuntu.com/1199747/ [00:03] StevenK: Have you confirmed that it now fails with the diff reverted? [00:04] wgrant: Just did. [00:05] IllegalTarget: Product-name-100217 doesn't allow Private bugs. [00:05] Sounds good, then [00:10] StevenK: Hm, why are you using multiple products? [00:36] wgrant: Sorry, was noming. Yeah, let me remove that bit. [00:42] wgrant: http://pastebin.ubuntu.com/1199798/ [00:43] StevenK: You could also remove the makeCommercialSubscription bit if you use makeProduct(bug_sharing_policy=BugSharingPolicy.PUBLIC_OR_PROPRIETARY) [00:43] Does that also allow USERDATA? [00:44] Yes [00:44] Which invalidates the test, does it not? [00:44] You'd still have to transition it to BugSharingPolicy.PROPRIETARY after creating the bug [00:44] It just saves you from having to manually create the commercialsubscription [00:45] I think it's moot, it would be the same amount of lines [00:45] But I can if you want [00:46] Perhaps [00:46] Anyway, it looks fine now [00:50] wgrant: The diff is updated if you want to +1 [00:51] You can actually kill of two more lines by not setting the product or bug owners, but I think you might kill me too [00:53] wgrant: And use login_celebrity, I guess [00:54] StevenK: person_logged_in(product.owner) [00:54] Due to the APG it'll even have access to the bug [00:56] wgrant: owner killed [00:56] That sounds painful [00:56] Haha [00:58] wgrant: Thanks, tossing at ec2. [01:00] Bloody hell, buildbot, finish already [01:04] wallyworld, wgrant: Some cards can probably move out of QA-Landing [01:04] probably [01:07] buildbot only finished 30 seconds ago [01:08] wgrant: Heh, it was saying 56 minutes when I looked 10 minutes ago. [01:08] 6 seconds after you mentioned QA-Landing, in fact... [01:32] bigjools: 1920093641 [01:32] bigjools: 1.9B [01:32] lifeless: 433453453458798 [01:33] bigjools: rows. [01:33] sweet [01:33] can we scale? Yes we can! [01:33] of course, it is 68GB on disk. [01:34] + indices [01:34] If we're talking about branchrevision, there's like 100GB of indices [01:35] Ah, no, only about 70GB now. We pruned a few [01:36] lifeless: is it web scale? [01:37] bigjools: http://www.mongodb-is-web-scale.com/ [01:37] haha [01:38] lifeless: "If there's a problem writing your data, you're fucked. Does that sound like a good design to you?" [01:39] bigjools: "If that's what they need to do to get those kickass benchmarks, then it's a great design." [01:39] lifeless: "Does /dev/null support sharding?" [01:39] I can't continue, laughing too hard [01:40] bigjools: you hadn't seen this ? [01:40] I have [01:40] but it remains hilarious [01:42] indeed! [01:43] bigjools: http://www.xtranormal.com/watch/11762072/native-html5 [01:45] ORDER BY Branch.date_last_modified DESC, Branch.target_suffix ASC, Branch.lifecycle_status DESC, Branch.owner_name ASC, Branch.name ASC [01:45] would you like some tie-breakers? [01:55] lifeless: Have you thought about getting eg. bug search totals from bugsummary? [01:56] I think that would rock [01:56] only works for some cases though [01:56] could be used to set upper bounds on bad estimates. [01:56] (can't be used to set lower bounds with our search algebra) [01:57] Yeah [01:57] Most of the common searches should work [01:57] personally I just want to nuke all the counts [01:57] Obviously not fti [01:57] or make them round - 10's 100's 1000's [01:58] put that behind a FF for a bit [01:58] see what the reaction is [01:58] if its good, change the code base to estimate everywhere. [02:00] bigjools: this is hilarious - http://www.xtranormal.com/watch/7023615/watch_movie [02:00] must...resist.... [02:00] too much to do [02:01] lifeless: i didn't think the native-html5 one was very funny [02:01] it was quite odd, though, so that's something i guess [02:01] lifeless: Well, EXPLAIN-based estimates will often be waaaaaaay off [02:01] mwhudson: the html5 one was not as good as I thought it would be [02:02] mwhudson: the one I just linked is a follow on from the mongodb one [02:02] wgrant: shrug :> [02:02] ahh [02:02] wgrant: I guess we need some care [02:02] It should be better now that we have bugtaskflat, I guess [02:02] But before bugtaskflat it would have been hopeless [02:04] lifeless: ok, watch_movie is pretty good [02:04] 3 mins in [02:06] OOPS report is pretty today :) [02:06] much better [02:06] not sure I'd call it pretty [02:07] Well [02:07] Prettier than it's been for a while [02:07] yes [02:07] massive improvement [02:07] just there is more to do [03:42] wgrant: https://dogfood.launchpad.net/ubuntu/+source/unity/+changelog [03:53] grah [03:53] Non-sql time: 5595 ms [03:53] if you're fixing timeouts [03:53] https://bugs.launchpad.net/~lifeless/?advanced=1 [03:55] bug 1049452 if anyone wants to have a poke [03:55] <_mup_> Bug #1049452: Person:+bugs timeout on advanced search form < https://launchpad.net/bugs/1049452 > [03:56] wgrant: StevenK: How do we release launchpad-buildd ? [03:57] lifeless: dch -i [03:57] lifeless: Then wait for IS to get us builders back [03:57] also, annoying - why is https://bugs.launchpad.net/python/+bug/96878 not FR for python, upstream has it as such. [03:57] Then RT [03:57] <_mup_> Bug #96878: Launchpad session cookie is accessible from Javascript

< https://launchpad.net/bugs/96878 > [03:58] lifeless: Resolution: accepted [04:02] lifeless: tsk tsk, duping your own timeout bugs [04:04] well, blame my short memory [04:09] lifeless: I was hoping the timeout was on +reportedbugs.. [04:09] wgrant: the search works fine... [04:09] wgrant: [04:09] 57 queries/external actions issued in 1.68 seconds [04:09] AJAX Log ↓ [04:09] not the fastest thing under the sun, but working >> not working. [04:09] Can't /setup/ the query though. [04:09] The form DIAF. [04:10] I bet its listing milestones for every project on Launchpad, more or less. [04:10] I made that pretty fast in most cases [04:10] But maybe you're terrible :( [04:10] and doing so super inefficiently. Probably a list of storm objects [04:10] with in [04:11] milestone in milestones: <- trigger the sqlbase __eq__ which is dog slow. [04:11] + quadratic in a list. [04:11] Not that slow [04:11] -bet you- [04:11] yes that slow [04:11] we've had 20s timeouts from that before. [04:11] shortly after new bug subscriptions, if you recall [04:12] Perhaps [04:34] wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/dsp-changelog-timeout/+merge/123880 [04:35] StevenK: Have you recowboyed that to check that the query count is now sanE? [04:37] No, let me that do [04:39] My dyslexia seems strong today [04:39] At least I didn't run make start stop [04:41] Heh [04:41] Huh [04:42] wgrant: 198 -> 51 [04:42] Limit (cost=10000003268.61..10000003269.44 rows=1 width=0) (actual time=122.637..122.637 rows=0 loops=1) [04:42] The main +specworkload query apparently doesn't please postgres very much [04:43] enable_seqscan=0 applies a 10 billion cost penalty [04:43] But that plan still wins [04:44] wgrant: No review? [04:45] StevenK: That wasn't the only extract_bug_numbers callsite? [04:46] (it was) [04:46] Ah [04:46] There's one in stringformatter.py itself [04:46] It's called in stringformatter itself [04:46] Yeah [04:46] I can inline it and stop exporting it if you wish [04:46] Nah [04:46] Probably not worth it [04:47] r=me, thanks [04:47] We still have a few VPC queries, but that's possibly just removedby so I don't really care [04:48] Yeah, we could make the pre-loading smarter by including some of the data from spph [04:49] Not as big an issue as the bugs :) [04:49] Indeed, so it's fine for now [04:49] The page is still slow on DF, but a fair bit of it is probably template time [04:50] And Unity has a TON of SPRs [04:50] Tell me when you're done and I'll revert DF [04:50] I'm done [04:55] Tempted to close 1000257 [04:55] bug #1000257 [04:55] <_mup_> Bug #1000257: OOPS sending email notification for bug watch < https://launchpad.net/bugs/1000257 > [04:56] Indeed [04:56] I can't see how it happened, but it did [04:59] StevenK: ndt [04:59] Probably [05:24] wgrant: http://pastebin.ubuntu.com/1199974/ [05:59] StevenK: https://devpad.canonical.com/~lpqateam/ppr/lpnet/ [06:21] wallyworld: Nice easy one for you: https://code.launchpad.net/~stevenk/launchpad/destroy-plus-daily-builds/+merge/123888 [06:31] wallyworld seems to be missing. [07:25] Been butting my head against pgbouncer. [07:26] stub: oh? [07:26] Unfortunately, root cause of my problems seems to be that KILL and RESUME just affect if opening new outgoing connections work or not. [07:26] stub: btw your part line is ... fun [07:26] '18:52 -!- stub [~stub@canonical/launchpad/stub] has left #launchpad-dev ["PART #launchpad :PART #postgresql :PART #slony :PART #storm :PART #zope3-dev :ISON cr3 jkakar daniels intellectronica daf czajkowski static SteveA fabbione Ng [07:26] thumper bigjools debonzi BradB bac sinzui_is_awake jdub allenap Znarl chrism andrewv elmo lifeless `a"] [07:26] ' [07:26] So the timouts for how long until I check if a backend is up, how long I wait for a backend connection, how long a client connection keeps retrying for a backend, all kick in [07:27] I have no idea what an ISON is [07:27] me neither [07:29] Anyway, unless I or someone wade into the C code, we might need two pgbouncers to do the updated fdt and actually lower downtime rather than increase it [07:29] stub: sure; how would that work ? [07:30] I can reduce the timeouts during the window, but to a minimum of 1 second. And that makes a huge difference if we are aiming at <5s [07:31] lifeless: Instead of KILL and RESUME on individual databases, we have separate pgbouncer instances for master and slave connections and shut them down/start them up [07:31] But it means a lot more bespoke production crap [07:31] init scripts etc. [07:32] And certainly not generalizable for outside of canonical. [07:36] Oh [07:36] I thought we had an alternative to pause that rejected new connections [07:36] But indeed, such a thing does not exist [07:37] wgrant: We have KILL and PAUSE, and we need KILL in current launchpad because we are running in session mode, not transactional mode [07:37] Right, but we have nothing like PAUSE that doesn't hang [07:37] WHich is what we really need [07:37] If we were running in transactional mode, yeah. [07:38] Even in session mode [07:38] But for current LP, the existing connections would never die unless we signalled all the clients 'oi, go reconnect now' [07:38] Right, but KILL serves that purpose already [07:38] We can kill all existing connections easily [07:38] Yes, so for now KILL does what we need [07:38] We just can't reject new ones claenly [07:39] KILL rejects new connections [07:40] If we run it constantly, yes [07:40] The documentation is fuzzy there [07:40] But it's a momentary thing [07:40] It drops all current and pending clients at the moment it's run [07:40] If we PAUSEd and ran kill every 10ms, it would work [07:41] On my system, KILL keeps it dead until RESUMEd [07:41] which is why I say the docs are wrong [07:41] I can still connect, it just hangs [07:42] On 1.5.1 [07:42] If I then KILL again, the hung connection drops, as expected [07:42] $ psql -p 6432 launchpad_dev_master [07:42] psql: ERROR: pgbouncer cannot connect to server [07:43] That's very interesting [07:43] Do you have a tiny bouncer->server timeout or something? [07:43] 2012-09-12 14:43:01.020 5838 LOG C-0x1e0d518: launchpad_dev_master/stub@unix:6432 login failed: db=launchpad_dev_master user=stub [07:43] 2012-09-12 14:43:01.020 5838 LOG S-0x1e2aec8: launchpad_dev_master/stub@unix:5432 closing because: pause mode (age=0) [07:44] wgrant: oh, you are experiencing the same problem my test case picked up [07:44] wgrant: After a KILL, the pools don't know that the db is down so keep trying to open connections, and failing because the db is down [07:44] wgrant: So it keeps retrying for 15 seconds until the 'db is down' flag is set [07:45] wgrant: And then, after a RESUME, that pool is still stuffed as it only tries to reconnect to a dead db every N seconds. [07:46] Ah, there we go, it does eventually time out [07:46] And now it is dead properly [07:46] So what we really need is a way to force it to poll and time out quickly [08:00] good morning [08:01] transactional mode still woudn't be fast enough, would it ? [08:02] I don't think so [08:11] Ah [08:11] It's pretty easy to make it to do what we want [08:11] Like [08:11] Really easy [08:13] share [08:13] Will have patch in a sec [08:15] Particularly if I don't try to run an amd64 binary on i386 === almaisan-away is now known as al-maisan [08:30] wgrant: I can reduce the timeouts, but it still can add about 2 seconds to the time according to my test case [08:31] wgrant: Patch welcome though, I hate relearning C everytime I need to do this. [08:32] It all works, except that changing the size of the PgDatabase struct makes it segfault [08:32] Just adding the new unused flag [08:32] Segfault [08:32] But if I reuse one of the existing flags it all works [08:32] Will look further after dinner [08:33] I can't see a hardcoded size anywhere obvious [08:34] jelmer: when you get a chance can you follow on the Question you were on https://answers.launchpad.net/launchpad/+question/206501 so I can see if it can be closed. [08:34] cheers [09:02] (it turns out their makefile deps suck, so a make clean fixed it) [09:04] stub: http://pastebin.ubuntu.com/1200173/ [09:04] You currently need a DISABLE/KILL/RESUME then ENABLE sequence [09:05] As disable doesn't actually kill, and kill implicitly pauses [09:07] nice [09:09] Cool [09:10] That on github? [09:10] I hate tracking down the official repo on GitHub [09:10] Maybe there's a link somewhere... [09:10] * wgrant googles [09:11] I could argue that KILL should imply DISABLE [09:11] Indeed [09:11] Alternatively KILL could just kill [09:11] and you're expected to PAUSE first [09:12] yeah, I can take the patch to the mailing list [09:12] https://github.com/markokr/pgbouncer-dev looks like it [09:12] Unless they use git://git.postgresql.org/git/pgbouncer.git directly [09:13] They do, but the github account is owned by the primary committer [09:13] So it seems relevant [09:13] * wgrant has forked it [09:19] I always forget how obtuse git is [10:58] stub: I've revised it a bit, so it now rejects only at client connect time and is not so terrible. https://github.com/wgrant/pgbouncer-dev/compare/disable [10:58] I'll create a pull request if you don't object soon [11:00] wgrant: Cool. I was going to build a package here to test my test case [11:00] Great [11:00] (running through the full process before I go and hack up full-update.py) [11:00] You still need the DISABLE/KILL/RESUME, do stuff, ENABLE sequence, but I'll mention improving that on the MP [11:01] If you leave it paused too long it'll still lose the backend connections and hang once you reenable [11:01] But if you resume immediately after killing you can keep it disabled forever [11:03] Also note that DISABLE only forbids new client connections. An existing client connection can still grab a server connection if it hasn't already. I could forbid in find_server as well, but I'm not sure if it's worth it [11:03] Given you need to KILL anyway [11:19] So htf do I convert https://github.com/wgrant/pgbouncer-dev/compare/disable to a diff? [11:21] stub: You're beyond my GitHub knowledge at this point, I'm afraid [11:21] Let me poke around [11:22] I guess I'm locked in ;) [11:23] http://pastebin.ubuntu.com/1200362/ [11:26] stick ".patch" on the end of that URL [11:26] Ah [11:26] I tried .diff [11:26] But not .patch [11:27] Hm [11:27] .diff works now too [11:27] I must have been on a different URL [11:27] Because hacking urls is discoverable UI [11:27] on commits, .diff and .patch give you different things [11:27] Yeah, .patch gives headers and commit message and stuff [11:29] Still seems noise to strip [11:29] oh, guess diff ignores it [11:30] no, noise to strip. === al-maisan is now known as almaisan-away [11:31] oh no, I'm on 1.5.2, patch is trunk [11:31] There shouldn't be conflicts outside NEWS [11:32] yeah [11:32] I just need more coffee [11:42] built the package, manual tests look good [11:48] And my updated tests like it too. [11:48] wgrant: Looks good. [11:48] Great [11:49] My only significant concern is that an evil client could connect but not cause a server connection to be established. [11:49] Then you DISABLE, and wait for connections to disappear from the backend server [11:49] But if you don't KILL, there could still be someone lurking in pgbouncer [11:49] waiting to strike and ruin your day [11:50] (to reproduce, open up a psql session but don't do anything. Call DISABLE in another session, and see that you can still talk in your original session) === gary_poster|away is now known as gary_poster [12:07] stub: Hopefully this will be a little less messy (and faster!) than sudoing around multiple instances [12:07] Oh, much. [12:08] We will need to KILL anyway after a timeout [12:08] To catch evil transactions. [12:08] Yeah [12:09] Definitely [12:09] But for the master we will just want to DISABLE and then KILL immediately [12:09] http://paste.ubuntu.com/1200439/ is the test case for the always-slave process, and it is passing with the new pgbouncer [12:10] Oh, an actual LP test case [12:10] Fancy [12:13] stub: https://code.launchpad.net/~stevenk/launchpad/buildfarmjob-index-redux/+merge/123854 would love a review [12:13] stub: Looks good [12:15] got the failwhale [12:15] not even an oops [12:15] 502? [12:15] Huh [12:15] Haven't seen many of those since we switched back to the rightful DB servers [12:16] StevenK: Didn't we do this already? [12:17] stub: Almost. The previous index did not have status, this one does. The two of them will work in concert for Builder:+history [12:18] yer [12:18] stub: Builder:+history with the index we added to prod makes it really fast. Builder:+history also allows filtering by status, which is slow. [12:18] Well, slow for uncommon statuses [12:19] Right. [12:19] It's noticibly slower. Like 4 seconds versus 0.4 [12:21] It it ready for prod or just qastaging? [12:21] Oh, I'm building this pgbouncer package for precise. I can just copy it to lucid and it will be fine, right? [12:22] I think it should be fine, but I admit to not actually testing it. [12:22] stub: I'm happy to play around on DF tomorrow if you're more comfortable with me having tested it. [12:23] Just removing it requires a FDT update. Apart from that no real worries applying it to prod. [12:23] stub: I'm not sure if the same binary will work [12:23] We have a ~lucid1 for 1.5.1 atm... === almaisan-away is now known as al-maisan [12:24] So reupload for lucid or make a recipe to work around the limitation [12:24] Exactly [12:25] StevenK: Its on qastaging [12:25] Let's have a look [12:25] 0.8 hot for main [12:26] 1.3 for FTBFS hot [12:26] SUPERSEDED is more interesting [12:26] 0.9 for depwait hot [12:27] superseded is 2.2 cold, 0.29 hot [12:27] Excellent. [12:27] To production we go. [12:28] How about I land it first? [12:28] :-) [12:31] stub: It's landed as r15944, you can add it to prod at your leisure === abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: abentley | Critical bugs: ∞ === abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: ∞ [13:19] deryck: I'm the OCR. Could you please review https://code.launchpad.net/~abentley/launchpad/spec-creation-info-type/+merge/123821 ? [13:19] abentley, sure [13:19] deryck: Thanks. [13:19] np! === cjohnston_ is now known as cjohnston [14:27] rick_h_: btw, my enum is lp.registry.enums.PUBLIC_PROPRIETARY_INFORMATION_TYPES [14:28] abentley: cool, thanks. Will look at updating that today. [14:41] adeuring, can you comment on bug 839779 and bug 450480 about how to fix them? [14:41] <_mup_> Bug #839779: HWDB doesn't create HWDeviceClass anymore < https://launchpad.net/bugs/839779 > [14:41] <_mup_> Bug #450480: Get PCI and USB vendor/product names from the "PCI ID Repository" project and from linux-usb.org respectively < https://launchpad.net/bugs/450480 > [14:42] * adeuring is looking [14:50] sinzui: I am a bit lost ATM regarding [14:50] #839779 [14:50] <_mup_> Bug #839779: HWDB doesn't create HWDeviceClass anymore < https://launchpad.net/bugs/839779 > [14:50] I guess that newer HWDB reports do not contain any data about devices classes anymore. [14:51] but I need to check [14:52] sinzui: regarding bug 450480: WHat do you miss in the bug report? [14:52] <_mup_> Bug #450480: Get PCI and USB vendor/product names from the "PCI ID Repository" project and from linux-usb.org respectively < https://launchpad.net/bugs/450480 > [14:52] adeuring, why a cronscript? Shouldn't the data be processed correctly when it is uploaded? [14:53] or put another way? I do not understand why hal has that data, but udev does not? [14:54] sinzui: ask the udev developers ;) The data is probably not important enough [14:55] okay. So want a process that gathers the secondary data to provide consistent, historic reporting? [14:55] sinzui: the script that processed the HWDB report can of course also read the two URLs directly [14:55] and use the data. [14:55] but we should also update existing "bad" records. [14:56] okay. and calling out to external site is blocked for most servers [14:57] sinzui: that's a problem indeed... But we might be able to read the data manually form time to time [14:57] understood. The data does not change often, which is why you want to pull it daily or weekly [14:58] sinzui: right. [14:58] okay. I think I know enough to fix this [14:58] abentley, your code looks great. well tested and solid. r=me. [14:58] deryck: Thanks. [14:58] sinzui: BTW, we wil always have the situation that a report contains a devcie where vendor/product names are yet knwon, even in the two databases mentioned in the bug report. [14:59] I don't know how often these DBs are updated, but we must always expect some lag [14:59] when for example some small vendor sells a new USB mouse [14:59] * sinzui nods [15:00]