[00:12] wallyworld: wgrant, I am back [00:12] * wallyworld opens mumble [00:13] win - 1567 OOPS-1983AS138 Bug:EntryResource [00:19] wgrant: guess what [00:19] lifeless: What? [00:20] http://pastebin.com/ZpWvaAgr [00:20] lifeless: Heh. Yay [00:20] thats why list() was around official bug tags before [00:21] not because it wasn't a list [00:21] can you suggeset any improvements ? [00:22] lifeless: Not really :/ [00:22] are we coming out of rc now ? [00:23] We can't QA on prod any more. But I guess so. [00:23] prod ? [00:23] Well, prod worked well as a QA environment yesterday... ahem. [00:23] *blink* [00:24] lifeless: It found a bug that would have really screwed us over on Thursday. [00:24] Because we wouldn't have been able to roll back. [00:24] right [00:25] but we don't have a branch with that rolled back and without db changes [00:25] No. [00:25] and we aren't aware of any damage in rev 13168 [00:25] that we haven't already agreed to [00:26] Correct. [00:26] so, unless we have some more testing we want to do [00:26] we should un-rc [00:26] Indeed. [00:26] But we said that yesterday too. [00:26] And then the world melted. [00:26] yes [00:26] unknown unknowns, they suck. [00:27] still [00:27] I've very glad I have been conservative about moving the freeze time closer to the downtime [00:37] Yes... [00:38] Once we have a faster test suite and no stupid feature deadlines, we can improve. [00:46] Project windmill-db-devel build #371: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-db-devel/371/ [00:50] allenap: are you still trying to get rabbit stable ? [00:52] lifeless: I don't think so. [00:52] StevenK might know. [00:52] He is not, no. === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:212 - 0:[######=_]:256 === StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugs:212 - 0:[######=_]:256 [01:59] sinzui: I've picked bug 307620 off that list [01:59] <_mup_> Bug #307620: Confusing message "There is a no package named 'xserver-xorg' in Debian" when filing a bug

< https://launchpad.net/bugs/307620 > [01:59] lifeless: ah. Hysterical raisins. [01:59] It's so lightly tested it's scary [02:01] StevenK: How do you propose to fix that? [02:02] lifeless, abentley: Do we know why PQM is silent about that? [02:02] wgrant: By doing what sinzui suggested in comment 18 [02:02] wgrant: I do not. [02:02] StevenK: Ah, heh. [02:02] wgrant: You think that's a good idea or a bad one? [02:02] StevenK: Mmm. We will have a proper solution to this soon. [02:02] StevenK: Ensemble needs it. [02:03] (hi lifeless) [02:03] StevenK: https://code.launchpad.net/~wgrant/launchpad/fix-sprite-rebuild-rules/+merge/63791 [02:03] But Ensemble is only concerned about SPNs [02:04] StevenK: It's a similar issue. [02:04] But true. [02:04] wgrant: r=me, awesome work [02:04] Thankyou sir. [02:07] Project windmill-devel build #179: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-devel/179/ [02:10] Hmm. The OOPSes are creeping back up. [02:10] wgrant: it blows an exception through to cron [02:10] comment 18 ? [02:25] The Internet hasn't ended yet? [02:25] What with World IPv6 Day and all that. [02:25] Doesn't seem to have [02:27] Tomboy needs bzr integration. [02:51] Project windmill-devel build #180: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/180/ [02:53] Can I set a distribution that I've just created to use LP as a bug tracker? [02:54] StevenK: As a mortal? Probably not. [02:55] Although +edit may work... [02:55] Some distro stuff is only accessible to admins. [02:55] Yeah, it's on +edit still. [02:55] wgrant: In a test case [02:56] StevenK: Set official_malone = True === Ursula is now known as Ursinha [02:58] Grar sampledata. [03:10] wgrant: Bah, setting offical_malone to True gives me the same message [03:10] StevenK: Which message? [03:10] wgrant: u"Distribution-774925 doesn't use Launchpad as its bug tracker." [03:10] grep may help. [03:15] wgrant: Spelling correctly would also help. :-( [03:25] StevenK: Heh. [03:35] Project windmill-devel build #181: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/181/ [03:43] Good day gentlebeings. === jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, jtv | Critical bugs:212 - 0:[######=_]:256 [03:44] Greeting jtv. [03:44] greeting back === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, jtv | Critical bugs:214 - 0:[######=_]:256 [03:55] (yes, its going the wrong way) [03:56] The graphs are quite depressing. [04:18] jml: I would like to catch up with you this week; my tonight is doable for me if it works for you. [04:35] * wgrant stabs SQLObject [04:39] oh === almaisan-away is now known as al-maisan [04:39] Oh? [04:39] what provoked the stabbing ? [04:40] The old cls.select thingy doesn't seem to behave just like Storm, particularly around using SQL(..., params=...) in orderby. [04:41] thats a SQLObject result set, it is a little different [04:41] Yeah. [04:42] I'm notting stabbing SQLObject for being buggy and inconsistent -- merely for existing at all. [04:42] Er. [04:42] s/notting/not/ [04:58] poolie: in canonical wikis 'Bug:1234' will link to lp automatically [04:59] that's good to know [05:00] i guess you realized my edit was fixing a typo, not just inserting the pad bit [05:03] Changing person vocab sorting is fuuuuun. [05:03] wgrant: The type of fun that is spelt with no n, and an extra c, k, e and d? [05:03] Something like that. [05:04] Like Help in CS is spelt with no p and an extra l [05:08] jtv: Are you free to review one of my branches? [05:08] StevenK: yes, just finished the last one. [05:09] StevenK: the also-affects one? [05:10] jtv: Yes. [05:14] jtv: I'm in the middle of allenap's MP. [05:16] rvb's MP looks okay to me, but I'm comfortable about reviewing a JS-only branch [05:16] Er. Not comfortable [05:16] * StevenK goes to inject tea [05:18] poolie: I didn't realise that,no. [05:49] Project windmill-devel build #182: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/182/ [05:59] Tests normally run as the 'launchpad' DB user, right? [06:04] Yes, I think so. [06:04] jtv: No news is good news? [06:33] StevenK: sorry for the delays — lots of distractions so threw in a spot of lunch as well [06:40] jtv: do you know anything about the storage-and-performance of arrays vs related tables in pg ? [06:41] jtv: I'm wondering if we should make subscriptions into an array (or duplicate them into one) on (perhaps just) private bugs [06:42] jtv: specifically, every time we access a private bug, we want to check the condition (private is false or has-subscription) [06:42] lifeless: I know very little except what's obvious. This sounds like a good idea; I've been quietly wishing for arrays in several places but not daring to speak out. [06:43] However front-end library support is not a given. [06:43] Generalized array parsing is an annoying little job that I decided not to tackle in libpqxx because it was coming in libpq, and AFAIK it never came. [06:43] ah [06:43] so I believe we do array processing in some (limited) places already [06:44] Nice [06:44] storm etc support is a good question though [06:44] this may be done via 'raw' sql today. [06:44] I did crib a bit of Storm support from somewhere and put it in the codebase IIRC… let me dig it up [06:44] no need, I'm not hacking on this just now [06:44] ok [06:45] was curious about the performance implications more than higher level stuff [06:45] I think you can index and search arrays for specific elements. [06:46] But duplicating into arrays as you say would give us the best of both, at the cost of writing a bit more data. [06:46] we're massively read heavy [06:46] There you go. [06:46] other than crazy things like bug heat, I've no issue with us writing more to read more efficiently [06:46] well [06:46] once we get the DB server disk upgrades done. [06:46] Although (advocate o/t devil) we've also got scaling headroom for reads that we don't have for writes [06:46] until then we need to be juidicious about it ;) [06:49] Project windmill-db-devel build #372: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/windmill-db-devel/372/ [06:50] * lifeless makes a 800K row temp table [06:51] oh -wow- [06:51] we have a slightly brain dead privacy check for assignee [06:51] The new one? [06:51] yes [06:52] well, https://code.launchpad.net/~mbp/launchpad/790902-mail-librarian/+merge/63816 makes it a uuid [06:52] goes from bugtask -> bug -> bugtask.assignee -> teamparticipation [06:52] the lp typo test to make the initial patch seems fairly consistent at about 45m for me [06:52] this is accurate but fiee. [06:56] so, denorming 3 fields onto bugtask and drop the assignee visibility -> 2 second table scan query. [06:57] still an OOM slower than bugsummary [06:58] poolie: is that friction, familiarity, not knowing where to find things? [06:58] StevenK, jtv, could you have a look at https://code.launchpad.net/~mbp/launchpad/790902-mail-librarian/+merge/63816 for me please? [06:58] /1/1 [06:58] poolie: I'm just reviewing StevenK but shouldn't take long, so I can take it after that [06:58] erk, damn. I hadn't done that all day! [06:58] about 5m getting the environment up to date, eg updating lp-sourcedeps, make schema etc [06:59] ah, recently i've had a run of things where there is redundant code that makes the fix more complicated [06:59] like the "decoy" mail sending code in the ppa notification stuff === al-maisan is now known as almaisan-away [07:03] perhaps none of them are truly typo-level fixes [07:40] StevenK: review done — I'm recommending some changes though [07:43] poolie: looking at yours now… why the trans.begin() in incoming.py? [07:44] jtv: structured() is smart enough to not escape things that are themselves an instance of structured() -- since I wrote that bit [07:44] jtv: rather than the other [07:44] StevenK: ahhh! But then where does that displayname get escaped? [07:44] it seemed to me only incoming.py really has an opinion about when things hould be committed [07:44] jtv: So, there are two displayname, both are inside a structured() of some form [07:46] StevenK: You fail. [07:46] 20 + binary_tracking = structured( [07:46] 21 + ' Launchpad does not track binary package names ' [07:46] 22 + 'in %s.' % distribution.displayname) [07:46] s/ %/,/ [07:47] poolie: I've bumped my head into this myself… turns out a transaction is started automatically whenever we access the database while not in a transaction and not in autocommit mode. So for the general case (and I don't know whether that includes this of course) all that begin() really does for us is increase the length of the running transaction. [07:47] wgrant: see my review and our ongoing discussion. [07:47] wgrant: Sigh, so I do. [07:48] poolie: I may be misunderstanding you though because I'm trying to carry two conversations at once. :) [07:48] ok [07:48] i didn't add that line, i just moved it [07:49] your logic makes sense [07:49] That's a fine answer AFAIC. [07:49] does "start if not already started" actually take much time? [07:49] If you don't feel comfortable changing it, leaving it as-is is valid. [07:49] No, that's really really quick. [07:49] that's good to know for the future [07:49] later i may try to clean up incoming.py more [07:49] Good call keeping that separate. [08:05] jtv: I have another one, if you're free. [08:06] StevenK: that wasn't enough to put you off? :) Sure, I'm almost done with Martin's. [08:07] jtv: This is one is 1: Much smaller, and 2: Needs more thought, but this is quick solution to the problem. [08:08] StevenK: being talked at from 5 directions; will be with you shortly. [08:10] poolie: you're done [08:12] StevenK: wait… the vocab currently only returns descriptions, and you're changing it not to return descriptions? [08:14] jtv: How useful do you think it is for people to pick out a source or binary package purely on their description only? [08:14] It's utterly utterly pointless [08:14] I don't think I've ever knowingly read a package description. What's in them, in practice? [08:15] (Just out of interest) [08:15] libc6's description is: Embedded GNU C Library: Shared libraries [08:15] hmm, 150ms stats portlets appear possible [08:15] StevenK: That's the summar. [08:15] StevenK: Not the description. [08:16] Oh, in which case the picker will show "Contains the standard libraries that are used by nearly all programs on" [08:16] Right. [08:16] jtv: ^ [08:16] Wow, that _does_ sound very helpful. [08:16] And that's one of the most useful descriptions. [08:16] (irony there) [08:17] This package contains debugging files used to investigate problems with [08:17] That's another example. [08:17] StevenK: I can see how you'd want to think about the proper solution some more… is it still worth setting a title though, or testing for it? [08:17] You can just leave out the third arg to SimpleTerm() [08:18] …and cut it out of the doctest. [08:18] If you're going to replace it very soon with something more sensible, it makes sense to keep it. Or maybe something in the picker requires it? Otherwise, I'd just drop it. [08:19] jtv: The picker does not seem to use the term, only the title, so we need to return it twice [08:20] hmm, where is the stubster [08:20] I see [08:23] StevenK: approved [08:25] jtv: Thanks! [08:27] rvba: I see you have a review waiting. I may get to that soon. [08:27] (There's one more in the queue before you) [08:27] jtv: great, thanks (it's a js fix). === StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv | Critical bugs:214 - 0:[######=_]:256 [08:46] rvba: that's a nasty little bug you're tackling there. As they say, only two things are hard in software engineering: naming things and cache invalidation. [08:46] And off-by-one errors. [08:48] good morning [08:51] hi adeuring! [08:51] need refreshment, brb === jtv is now known as jtv-afk [08:52] thanks jtv [08:52] it would be cool if lp did ipv6 day next year [08:52] Hahahah [08:53] indeed, but... [08:53] in all your spare time :) [08:53] It's got nothing to do with LP :) [08:54] And then someone in China pops up and says "here's a version of IP that just uses 64-bit addresses, and we NAT it into a hole in the IPv4 space, and we already talked to the equipment manufacturers" and then all that effort is going to be wasted. :) [08:55] I think there's like two significant Australian ISPs that provide native v6 :/ [08:55] i'm all right jack :) [08:56] wgrant: meaning it's mostly a canonical-network thing? [08:56] poolie: Entirely. [08:56] Well. Unless you call the frontends LP. [08:56] Which I guess you almost could. [09:00] poolie: I thought we chatted about this on the phone [09:00] poolie: when I closed that bug [09:00] oh about ipv6? [09:00] yes, we did [09:01] stub: hola! [09:01] Yo [09:01] stub: I want to add 3 columns to bugsummary :> [09:02] Oh joy :) [09:02] https://bugs.launchpad.net/launchpad/+bug/793848/comments/4 [09:02] <_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts < https://launchpad.net/bugs/793848 > [09:03] lifeless: You are quadrupling the number of rows. Not sure if the distro queries are going to survive with their tag counts. [09:04] stub: https://bugs.launchpad.net/launchpad/+bug/793848/comments/5 [09:05] :P [09:05] <_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts < https://launchpad.net/bugs/793848 > [09:05] stub: its about 50ms slower on the tag counting case === jtv-afk is now known as jtv [09:07] lifeless: Will it be exploded by tags and milestones, or can we add CHECK (has_patch IS NULL != tag IS NULL) etc. [09:07] Hello devers [09:07] stub: its more dimensions [09:07] stub: so more cross products [09:08] Right. So we have to be careful, as every addition could put us over the top with this model [09:08] we do: [09:08] select count(*) from bugsummary2; [09:08] count [09:08] -------- [09:08] 758533 [09:09] \dt+ bugsummary2; [09:09] List of relations [09:09] Schema | Name | Type | Owner | Size | Description [09:09] -----------+-------------+-------+-------+-------+------------- [09:09] pg_temp_1 | bugsummary2 | table | ro | 54 MB | [09:09] Did you check the existing queries for distribution and distroseries use cases? [09:09] distribution yes, haven't checked distroseries [09:09] I think we will be ok too, this time.... [09:09] I'd expect distribution to be worst, and its 200ms [09:11] stub: we could make the booleans and importance nullable and have two separate summaries - one for use in this query, one for the tags one [09:12] but I don't think the complexity is worth it - this seems pretty straight forward and the performance (hot) is still extremely shiny [09:13] lifeless: If we took that approach, I'd just stick them in a separate table. [09:16] yeah [09:21] so, whats the right way forward to apply this change === almaisan-away is now known as al-maisan [09:23] lifeless: The usual. db patch to add the new columns and extend all the triggers, test on staging... [09:23] lifeless: Probably one for me. [09:23] stub: would you? [09:24] although it will likely be cut & paste work with the existing code as a template. [09:24] sure [09:24] yah [09:24] I have a test script in that bug, which is an adapted version of the original patch (but no triggers) [09:25] oh btw [09:25] stub: were you aware that copying a template db locks the source ? [09:25] exclusive-locks [09:25] anyhow, I put a spinlock-with-random-backoff into the db layer when I found this some months ago [09:25] Your template db can't be in use, so not surprising. [09:25] I'm thinking of making a test-run-specific template [09:26] You get an error if there are any open connections to the template, or at least you used to. [09:26] so launchpad_ftest_template -> launchpad_ftest_template_$PID -> launchpad_ftest_$PID [09:26] do you see any problem with such a double-indirection ? [09:27] This is to avoid contention I guess? No problem except twice as many databases to clean up on a cancelled run. We should probably automatically do that somehow... might need to maintain shared state in some well known location, such as a dedicated PG database for lp test run metadata. [09:28] lifeless: Is db setup time worth optimizing yet though? Last I checked, teardown and setup was only 10-15 minutes or so of our total test run time. [09:28] stub: parallel testing contention... [09:29] stub: I have no solid figures, but I'd like to be at least theoretically interaction-free. [09:29] Yer, it will be better. Just wondering if there is other low hanging fruit that should be tackled first. [09:30] stub: possibly. Feel free to try parallel testing anytime you like :) [09:30] StevenK: which reminds me, does jenkins parallel test devel nowadays ? [09:30] There used to be some backoff time required *anyway*, because it took a short while after you closed your connection before PG had cleaned up the backend and the connection was really dead. [09:31] I think that is better now under 8.4, where it just blocks until the db is available. Might be possible to pull out the retry code (?) [09:31] definitely isn't possible :) [09:32] running 8 test threads at once, they all blew up on template copying until I added the random backoff [09:32] I mean after a separate template per thread. [09:32] oh right [09:32] up [09:32] uhm [09:32] I guess if it isn't needed, the retry will never happen so no need to pull it out yet. [09:33] if we're using the same helper to copy from l-f-t to l-f-t-$pid and l-f-t-$pid to l-f-$pid then it needs to be in for the first copy [09:33] right [09:34] mark had the idea of a background task that created fresh databases for tests to consume, but I never chased it once we saw db teardown/setup was not as significant as we thought. [09:34] *blink* [09:34] that would create IO and CPU contention [09:34] unless you had less test runners than the machines concurrency could support [09:35] grr 14492 tests [09:35] 14491 passed [09:35] this was singlethreaded tests, so building the db would chew io while the test was chewing cpu [09:35] whats the bet I failed to push [09:35] and most tests don't need new dbs, so it would be idle a lot of the time anyway. [09:36] yeah [09:36] I wouldn't pursue the idea :) [09:37] lifeless: What do you mean? [09:37] StevenK: there was a jenkins job doing parallel test runs [09:38] There still is [09:38] what branch does it run [09:39] http://bazaar.launchpad.net/~lifeless/launchpad/librarian [09:39] StevenK: can we switch that to running stable ? [09:40] and sorry for breaking trunk, I had not pushed :( [09:40] Done [09:42] StevenK: thanks! [09:59] lifeless: Although extending BugSummary will fix counts, I think there will still be problems elsewhere discovering bugs that have been fixed upstream. If people are interested in how many, they are also interested in a set of links. Is this search also problematic? Perhaps being able to flag a bugtask as 'fixed upstream' will make both the counts and search acceptable. And if the counts still are not acceptable, make the triggers much easier. [10:00] stub: I'd be fine with denormalising 'fixed_upstream' onto bugtask [10:01] garbo task or checkwatches could do that [10:01] stub: or a trigger could [10:01] yer. [10:02] stub: I did an experiment with a wider bugtask [10:02] thanks for the kind words jtv [10:02] stub: it can get down to 2 seconds if we can drop the join onto bug [10:02] but checkwatches is the only thing that updates the relevant rows I think (?), so there might be simplest. [10:02] https://bugs.launchpad.net/launchpad/+bug/793848/comments/1 [10:02] <_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts < https://launchpad.net/bugs/793848 > [10:02] poolie: you know me — you earned them. :) [10:03] stub: there are 2 cases to fixed upstream, a) bugwatch, b) a product task [10:03] stub: b) is changed by users. [10:05] k [10:06] lifeless: so just briefly i'm merging those two mail->librarian things into just one, that uses a uuid [10:07] poolie: cool, I saw. Looks good. [10:08] stub: comment 3 gets a 2 second query with reflexive-join to find fixed_upstream [10:09] stub: by denormalising duplicateof, latest_patch_uploaded and private onto bugtask === al-maisan is now known as almaisan-away [10:09] thanks [10:09] stub: but we have to join to bug to get assignee [10:09] stub: so we're still looking at 4.5 seconds minimum [10:09] stub: -> 3.5 seconds too slow === almaisan-away is now known as al-maisan [10:10] Unless we use triggers to mush together bug and bugtask into one flat BugSearch table... [10:11] maybe too fat for sanity... but I guess fat for a reason. [10:11] stub: I doubt it would get down to the 100ms that bugsummary does [10:11] right, but useful for stuff besides counts [10:11] stub: the thing about the portlets is that they want multiple criteria shown at once [10:11] or is this only a problem with counts and I'm fixing a problem we don't have? [10:12] stub: indeed, and I'm totally happy to have a bugsearch table too, but its a different scenario [10:12] stub: so these counts link to searches [10:12] and bug search is also slow [10:12] but the one portlet shows 9 different searches summarised [10:13] so an individual search has rather more headroom than the portlet === al-maisan is now known as almaisan-away [10:13] Right, so we will need bugsummary for this sort of things in every case, unless we can make bug search super fast (10x more than we are currently aiming at) [10:14] Or something similar... we already know BugSummary isn't going to scale to too many more dimensions [10:14] But there are a finite number of possibilities that I can't recall of the top of my head. [10:15] Hey, from Python I try to load a pot from LP (staging). The code should work and I should have the right permission, nevertheless I get HTTP Error 401: Unauthorized (Unknown access token). If I paste the URL in my browser I DO get the correct response. Any ideas? [10:16] your python code hasn't logged in via either oauth or openid [10:16] Or anonymously. [10:22] It does launchpadlib.login_with() [10:22] How can I check if it's done right === almaisan-away is now known as al-maisan [10:25] lifeless, this is the output: http://pastebin.com/kZ9wr6BL [10:45] jtv: could you please have a look at this MP: https://code.launchpad.net/~adeuring/launchpad/bug-735979/+merge/63832 ? [10:45] RawChid: can you paste the code too? [10:45] adeuring: ok, but that's the last one for today. :) === jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:214 - 0:[######=_]:256 [10:46] jtv: thanks! [10:47] RawChid: one issue is that the url you're printing is on lpnet, but you say you're trying to load from staging [10:47] i wonder if you are only authenticated on one of them? [10:49] poolie, I just doscovered I can load the pot from production without a problem. [10:49] You're right! I authenticate to staging, but the URL is production. Thanks a lot [10:49] you're very welcome [10:52] adeuring: hi [10:52] adeuring: you know about ++profile++ right ? [10:52] hi lifeless [10:52] lifeless: argh, yeah, right! [10:52] thanks for the reminder [10:52]