[00:00] AAArrgggghhh!!!! [00:00] FFS [00:00] factory.makeBugTarget returns a bugtarget which has a bug with two! targets [00:00] conjoined master? [00:01] * thumper shrugs [00:01] I don't even know what that means [00:03] thumper: sinzui and i just worked on that, let me see if i can understand it for you [00:03] if you have a bug targeted on the development focus for a pillar, then you have 2 bug tasks [00:03] one on the pillar, one on the series [00:03] and this is handled specially. [00:03] lifeless: a bare "makeBugTask" should give the simplest possible, surely [00:03] I don't know why we do this. [00:04] thumper: if the target is the default series for the pillar, making a bug on that target will implicitly create one on the pillar [00:05] lifeless: a simple makeBugTask should not target any series [00:05] thumper: sure [00:06] lifeless: So, are you looking at the bug search issue? It is just about critical. [00:06] ok, it seems that it has created a bug with two different product targets [00:07] leonardr: what enlightenment can you offer? [00:07] thumper: ok, thats different [00:07] lifeless: if you just call makeBugTask() with no arguments, the target is a product, and a product has no need for a prerequisite bugtask, so i'm not sure what's going on [00:07] er, thumper -^ [00:08] I know [00:08] make bug makes a default task [00:08] it has to [00:08] makeBugTask makes a task [00:08] so when you go makeBugTask with no params [00:08] you get two tasks [00:08] wgrant: so oopses were still nominal yesterday [00:08] one from the makeBug call [00:08] wgrant: are you seeing other data ? [00:08] and one added in the makeBugTask [00:09] * thumper fixorates [00:10] thumper: your fix may conflict with the branch i'm trying to land now, though it should be reconcilable [00:10] leonardr: what did you do about it? [00:10] thumper: i changed something else in makeBugTask [00:10] let me see... [00:10] which branch is it? [00:11] lp:~leonardr/launchpad/bug-106338 [00:12] lifeless: bzr bug searches time out, and when they don't they take >10s. [00:12] lifeless: Didn't this change only get deployed like 7 hours ago? [00:13] hmm... [00:13] thumper: we made some other changes to makeBugTask [00:13] wgrant: 12582 was the change [00:14] leonardr: I see that [00:14] leonardr: I'm going to change my approach in my test to not need it [00:14] ok. fwiw, i think your proposed change is complementary to ours [00:15] Right, and I see the LPS change for that a little over 7 hours ago. [00:15] lifeless: So I don't think we've seen any OOPS reports for that yet. [00:19] wgrant: https://bugs.launchpad.net/ubuntu/+bugs?field.searchtext=sshd&orderby=-importance&search=Search&field.status:list=NEW&field.status:list=INCOMPLETE_WITH_RESPONSE&field.status:list=INCOMPLETE_WITHOUT_RESPONSE&field.status:list=CONFIRMED&field.status:list=TRIAGED&field.status:list=INPROGRESS&field.status:list=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package= [00:21] wgrant: what I mean is 'searches on ubuntu seem ok' [00:27] If wanted to add another static class to this: would it be or ? [00:29] The latter [00:29] tal is for when you need things evaluated [00:30] mmm [00:30] it probably won't combine right [00:30] you probably need to fix scope_css_class [00:35] lifeless: Can you run select count(id) from SourcePackageRelease; on qastaging? I'd like to get a feel for the size of the table [00:36] lifeless: How do I figure out what object that tal statement is using? There are two classes that define scope_css_class. [00:37] lifeless: I guess that's kind of a general question about these templates [00:37] huwshimi: bug 735196 may interest you [00:37] specifically the templates in question are lib/lp/answers/templates/questions-index.pt and lib/lp/blueprints/templates/specifications-index.pt [00:37] <_mup_> Bug #735196: Text pops out of announcment box < https://launchpad.net/bugs/735196 > [01:35] :( [01:36] T-20 [01:36] Heh, yes. [01:37] It is a painful wait. [01:37] But also not today a terribly interesting one. [01:37] wgrant: can you tell me the command to run to tar up an egg. in the past i've just done it manually but i'm sure there's an easier way. [01:37] setup.py egg_info ... [01:37] wallyworld: It varies. [01:37] or setup.py sdist [01:37] wallyworld: Often python setup.py egg_info sdist [01:37] or bze export [01:37] Right. [01:38] All of what lifeless says is valid. [01:38] Which egg? [01:38] thanks. i'll try one of those. i have proposed a change to lazr-js and want to use it locally while i'm waiting to get it approved [01:39] setup.py egg_info -b-somesuffix sdist [01:39] wgrant: egg_info does an sdist AFAICT [01:39] -b adds a suffix to the version. [01:39] Hm. [01:40] * wallyworld goes to impersonate a chicken [01:41] * wgrant deletes code. [01:41] holy cow [01:41] that with clause I was working on yesterday -rocks- [01:42] cold on qastaging: Total runtime: 151.508 ms [01:42] (20 rows) [01:42] Time: 193.425 ms [01:42] lifeless: POFile:+translate? [01:42] yeah [01:42] Wasn't the old one several repitions of >1s? [01:42] *repetitions [01:42] so the old one was 1.1s cold, 300ms hot [01:42] this is [01:42] Ah. [01:42] 190ms hot, 9ms cold [01:43] I doubt that, somehow. [01:43] stub tested on prob, 70ms cold [01:43] 190ms cold 9ms hot. doh. [01:43] Heh. [01:44] we'll probably find it throws some corner case out and becomes pathological [01:45] cats are awesome [01:45] Occasionally. [01:46] Mine likes to sit next to my computer. This is OK except when the side of the case is off, and tail and CPU fan intersect. [01:47] lol [01:47] Unhappy cat. [01:47] What is yours doing? [01:47] right now, curled up in the basement of its climber, snoring gently [01:48] Heh. [01:50] * wgrant deletes lots of NullBugTask-related hacks. [01:58] wgrant: https://pastebin.canonical.com/44718/ may interest you (re search perf) [02:02] dropping the private-bug query -> 1 second [02:03] but, 10% of the estimated cost [02:04] lifeless: :( [02:04] It's still got a really awful plan, though. [02:04] mawson goes the other way and is sub-second. [02:05] * 307 Time Outs [02:05] So close :( [02:06] hard + soft = 1800 [02:06] 300 down from yesterdays report [02:06] And we only had <8 hours of fixedness in that report. [02:08] so I think we're good to move ahead [02:08] To 12s? Or less? [02:08] 12 [02:08] :( [02:08] I'm worried about search [02:08] Yeah. [02:09] crazy eddie plans [02:09] Gnrgh. [02:09] jcsackett: hows cve:+index treating you? [02:09] Distribution:+bugtarget-portlet-bugfilters-stats will need fixing soon [02:09] bugtask:+index is looking pretty happy, as such things go [02:11] Bugs' ZCML is crap. [02:14] s/[^ ]* // [02:17] +editstatus is protected from rendering on a NullBugTask in three or four ways. [02:18] win [02:19] Why would a task listing ever hit a nullbugtask? :/ [02:19] well [02:20] what was that bug I was looking at yesterday [02:20] the one with memcache in it ? it wasn't redirecting [02:20] I know why it sometimes doesn't redirect. [02:20] That doesn't explain how one would have *ever* shown up in a listing. [02:20] oh [02:20] thats because the current context has to show [02:21] Hm? [02:21] in the past anyhow === matsubara-afk is now known as matsubara [02:34] Our tests are slow. [02:36] lifeless: Did you miss my ping? [02:37] StevenK: presumably [02:37] whats uo ? [02:37] ah [02:37] yes, i did it [02:38] 822218 [02:38] Ouch, thanks [02:38] lifeless: your with_ statement seems to have a bug in __contains__ [02:39] lifeless: possibly worth making a test for [02:39] lifeless: I think the fastest way to fix the job on loganberry is to run a memcached on it until the job is done. But I think you might want to investigate why we have one memcached per appserver anyway. [02:40] wgrant: http://pastebin.ubuntu.com/580404/ [02:40] StevenK: Looks good. [02:41] wgrant: Shouldn't you be the OCR, anyway? [02:41] StevenK: A reasonable point. [02:43] lp.bugs.tests.test_bugtask_search.TestNoPreloadBugtaskTargets* is SLOW. [02:43] How slow? [02:43] thumper: what happens? [02:43] StevenK: Many minutes. [02:43] StevenK: we have a cluster [02:43] lifeless: the with statement isn't used [02:43] wgrant: Kill it! [02:44] StevenK: and deterministic routing to them [02:44] lifeless: and you get: ProgrammingError: relation "blocked" does not exist [02:44] LINE 1: ....id != 14 AND (Specification.id in (select id from blocked)) [02:44] StevenK: so the fastest way is to make sure the memcaches are configured correctly and have gsa open firewalls if needed. [02:44] thumper: thanks [02:44] thumper: probably shallow [02:44] lifeless: probably [02:45] lifeless: So the way forward is an RT? [02:45] StevenK: in either scenario yes [02:46] RT's make me sad. [02:51] wgrant: If I'm reading these numbers right, we have processed over 4,500 SPRs with 6 failures. [02:52] StevenK: Excellent. [02:52] But since memcache doesn't work, we keep looking at the 6 failures. [02:52] Right. [02:59] heh [02:59] doing bzr pull --overwrite lp:~lifeless/storm/with-without-datetime in my lp tree -- bad [02:59] Haha [03:06] thumper: lp:~lifeless/storm/with-without-datetime contains the fix [03:07] thumper: I'll let you make a new egg from the branch, because you'll need that to include it in your branch [03:07] lifeless: I don't need it [03:07] just bumped into it [03:07] ah [03:08] ok [03:08] I might bump the egg shortly, thanks for letting me know [03:16] wgrant, lifeless: Back of envolope maths: We're looking at ~ 2 months [03:16] Really? [03:17] How many need to be processed? [03:17] That's almost twice my prediction. [03:17] My maths was assuming all of them, which I've just realised is dead wong [03:18] so, you can make this faster [03:19] change garbo so that rather than a 15 minute per loop limit [03:19] This runs last, dying only on the hour? [03:19] it sets a 50 minute budget for all loops [03:19] and per loop allows $remaining_budget as a limit [03:20] lifeless: No, let me fix my maths first [03:21] thumper: the ajax love for blueprints is very nice [03:21] StevenK: fixing this would be a good idea anyway [03:22] mwhudson: yay, it is noticed :) [03:22] the 15 minute per loop is bong because if 5 loops take max time we'll exceed one hour [03:22] * StevenK looks for the query lifeless wrote [03:22] thumper: you should be coming to UDS, i bet all the ubuntu devs would buy you beer [03:23] thumper: I think you should blog about the blueprint changes and link to the screencast [03:23] Let's not advertise blueprints.… [03:23] That may involve jcastro and dholbach flying down to hug you [03:23] StevenK: I did blog about it [03:24] wgrant: its good to tell people about things we're improving [03:26] lifeless: select changelog is null, count(*) from sourcepackagerelease group by changelog is null; on qas, plz [03:26] f | 270845 [03:26] t | 551373 [03:27] So we are missing 551k changelogs? [03:27] on qas [03:27] Right [03:27] On prod it is likely 547k [03:28] Still 48 days, give or take [03:29] so, do the change I propose to garbo (and put this job at the end) [03:29] its a general win and will give you 3-4 times the throughput [03:29] 14:19:52 < lifeless> and per loop allows $remaining_budget as a limit [03:29] How is that going to work? [03:29] That will allow even the first loop to exhaust the full window. [03:29] lifeless: Yes, I was just about to look at that [03:29] Preventing the subsequent ones from running at sll. [03:30] wgrant: So, garbo-hourly has a budget of 50 minutes. Allow the loops to have a budget of whatever is left of the 50, rather than 15 minutes. [03:30] StevenK: Right. So loop 1 has 50 minutes to run it, and does so. [03:31] wgrant: Can haz review of https://code.launchpad.net/~stevenk/launchpad/dsd-base_source_pub-archive/+merge/53357 [03:31] Goodbye loops >=2 [03:31] lifeless: wgrant has a good point [03:31] StevenK: I shouldn't, but since I'm mentored... [03:31] lifeless: Could you mentor StevenK's? [03:31] wgrant: Well, then I'll ask lifeless for a review, then [03:32] wgrant: You don't need to feel obligated to review [03:32] StevenK: I am happy to, but I think it's not really appropriate in this case. But this case is pretty trivial, so meh. [03:34] Anyone want to review https://code.launchpad.net/~wgrant/launchpad/unuse-nullbugtask/+merge/53363? [03:35] wgrant: it would, but only one of them ever uses its full budget atm - the spr changelog one [03:35] https://code.launchpad.net/~lifeless/launchpad/bug-734642/+merge/53365 [03:36] lifeless: That is false, read the logs [03:36] StevenK: what else is using its full budget ? [03:38] 2011-03-15 02:13:16 INFO Running OAuthNoncePruner [03:38] 2011-03-15 02:28:08 DEBUG Done. 1 items in 1 iterations, 891.136 seconds, average size 1.000000 (0.00112216283655/s) [03:38] That isn't every run, but it does happen. [03:38] that looks like a bug [03:38] table contention perhaps [03:38] It is table contention [03:39] not at 15 minutes its not [03:39] It blocks on two transactions and sleeps [03:39] But I don't want to paste 6 lines to the channel [03:43] thumper: https://code.launchpad.net/~wgrant/launchpad/unuse-nullbugtask/+merge/53363? [03:43] * thumper looks [03:44] StevenK: the backoff algorithm needs fixing I guess [03:45] StevenK: done [03:45] * thumper graduates StevenK [03:45] StevenK: I'll send an email to the list [03:46] Project windmill build #49: STILL FAILING in 1 hr 7 min: https://hudson.wedontsleep.org/job/windmill/49/ [03:47] thumper: Thanks. Could you now mentor https://code.launchpad.net/~lifeless/launchpad/bug-734642/+merge/53365? [03:47] thumper: !!!! [03:47] (congrats StevenK) [03:49] thumper: Let me edit ReviewerSchedule [03:52] yes, congrats [03:52] wgrant i filed bug 735260 for the problem i mentioned before [03:52] <_mup_> Bug #735260: multiple timeouts in bug search < https://launchpad.net/bugs/735260 > [03:52] it seems still broken [03:52] poolie: You probably want to convince lifeless that it has regressed. [03:52] I am convinced. [03:53] yes, the tradeoffs have altered [03:53] ubuntu bug search is faster than it was [03:53] and we have more of those [03:54] seriously [03:54] yes, seriously === matsubara is now known as matsubara-afk [03:55] we had 300 oopses from ~ 8 hours of the new search query [03:55] (total timeouts) [03:55] ah, i wasn't questioning your assertion, just expressing frustration at not being able to search [03:55] thats down day on day from monday and from the weekend [03:55] good thing i have email i guess [03:58] https://lpstats.canonical.com/graphs/OopsLpnetHourly/ [03:59] there is certainly a spike [03:59] i'm glad it's progressing [04:00] thumper: https://code.launchpad.net/~lifeless/launchpad/bug-734642/+merge/53365 [04:00] thumper: oh you have already - thanks [04:13] lifeless: Did you want to reping about the LPS feature flags? [04:14] poolie: try searching at bugs.launchpad.net/bugs [04:14] poolie: its performing fine - 5 seconds to query sshd [04:14] StevenK: econtext [04:15] lifeless: Removal of the recipe beta feature flag [04:15] lifeless, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1900G239 [04:15] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1900J237 [04:15] is not 'fine' [04:16] poolie: thats very odd [04:16] because I tried right before commenting [04:16] poolie: what query did you use? [04:16] Also right after commenting? [04:16] Master vs. slave, maybe? [04:16] wgrant: possibly [04:16] 98 queries/external actions issued in 4.67 seconds [04:17] i typed 'shortreadverror' in the search field there, both with 'bzr' in the project field and without [04:17] 29 queries/external actions issued in 6.53 seconds [04:17] bug 413430 bug 402857 bug 673439 [04:18] <_mup_> Bug #413430: ShortReadvError on index file < https://launchpad.net/bugs/413430 > [04:18] <_mup_> Bug #402857: ShortReadvError in pack file: bzr check should check pack file hashes and recover from problems < https://launchpad.net/bugs/402857 > [04:18] just curious, why did you untag it 'oops' and 'regression'? [04:18] we use oops and timeout tags as partitions [04:18] 'oops' means 'non-timeout oops'? [04:18] ok [04:19] search performance is being a problem but we haven't figured out what is up yet [04:19] it may well be a coincidence [04:19] particularly given that right now the same searches are working for me an dbreaking for you [04:19] this used to work reliably, and now it doesn't --> regression, is it not? [04:19] poolie: it didn't work reliably [04:19] For !Ubuntu it did... [04:20] wgrant: poolie is searching in ubuntu still - the bug he filed was an ubuntu bug search [04:20] for me, it's gone from working >95% of the time to working <10% of the time [04:20] no, it wasn't [04:20] Wasn't it in the bzr product? [04:20] wgrant, yes [04:20] https://bugs.launchpad.net/ubuntu/+bugs?field ... [04:20] poolie: ^ thats the url in your bug description, which I have not altered [04:22] the other two oopses you've reported here I have asked for a sync of, to look at [04:22] ok [04:22] it is also failing in plain bzr searches, i'm pretty sure [04:22] It is, I saw some earlier. [04:22] i'm confused, i thought i posted several oopses in the description [04:23] for instance https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1900J237 [04:23] you pasted that here, its syncing at the moment [04:23] after my suggestion to search on the root context [04:24] the J oops is on bzr [04:25] 9 second count cold; 2 second count hot [04:25] i'm getting a 'please try again' page talking to lp [04:25] in fact https://bugs.launchpad.net/bzr/+bugs?field.searchtext=winsshd&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package= is pe [04:25] rsistently timing out [04:25] anyhow [04:25] does 'regression' mean 'used to work and now doesn't' or is it more subtle? [04:25] -> Index Scan using bug_fti on bug (cost=0.00..2306.35 rows=42 width=4) (actual time=43.873..1929.329 rows=7 loops=1) [04:25] Index Cond: ((fti)::tsvector @@ '''shortreadverror'''::tsquery) [04:25] i don't care what tags it has, i just want to understand what they mean [04:25] Filter: ((duplicateof IS NULL) AND ((NOT private) OR (SubPlan 1))) [04:26] poolie: regression isn't strictly defined; I may add it back to the bug once we have a handle on whats up [04:26] poolie: regressions and timeouts and oopses all fall in the same priority bucket (and escalated bugs) [04:27] anyhow, to be clear [04:27] searching just in bzr was working and now isn't [04:27] i might have accidentally been in the package context for one search that i pasted into the bug, but it's also failed several times in the project [04:28] sure [04:28] so, we're looking at it [04:28] thanks [04:28] have been all day in fact [04:30] i'll let you get on with it [04:30] just wanted to be sure the bug was accurate [04:33] poolie: regression is really about code changes that break things horribly. There is some evidence that this isn't a code change issue per se [04:33] *even though* a code change was landed [04:37] StevenK: Want to use your new superpowers on https://code.launchpad.net/~wgrant/launchpad/delete-nullbugtask/+merge/53367? [04:37] Ugh, forgot prereq. [04:40] StevenK: https://code.launchpad.net/~wgrant/launchpad/delete-nullbugtask/+merge/53369 [04:42] wgrant: Done. [04:42] It's going to hard to get out of the habit of code* everything [04:42] Thanks. [04:43] wgrant: I fully support your plan to kill NullBugTask, and wish to subscribe to your newsletter. [04:43] StevenK: you should also set the MP status, probably. [04:43] Also done [04:43] You overwrote mine :( [04:43] But thanks. [05:13] StevenK: I need some help with Q/A for generating the DSDJs. [05:13] I need to turn soyuz.derived_series_jobs.enabled on, which I'm not privileged to do, and then I need to make sure that everything that will create SPPHs still works which I don't know how to do. [05:14] Turn it on *where* [05:14] poolie: mostly fixed [05:19] StevenK: on qastaging, I guess—depends on where we can do whatever generates SPPHs. [05:19] gina can't run on qastaging, you can upload and copy, though [05:20] Does that mean one place will get exercised but another won't? [05:20] Obviously. [05:20] gina can run on DF, but carefully [05:21] So if I can get someone to enable the flag on qastaging, I can test uploading to PPAs and copying between PPAs there? [05:21] Indeed [05:22] lifeless: could you enable soyuz.derived_series_jobs.enabled for me on qastaging? [05:24] jtv: I don't have the admin bit [05:24] jtv: stub does; stub^ [05:25] How do I do that? Is this a feature flag or something? [05:26] visit qastaging.launchpad.net/+feature-rules [05:26] It's a boolean feature flag. Set it default true. [05:26] add a line like [05:26] soyuz.derived_series_jobs.enabled default 0 on [05:26] (flag, scope, sequence, value) [05:29] poolie: all sorted now AFAWCT [05:30] It's still not great, but it's <2s now. [05:30] super [05:35] That ff is on now. [05:35] On qastaging [05:35] jtv: ^^^ [05:35] thanks stub [05:55] StevenK: how do I make dput upload to qastaging? [05:55] Ugh, I can never remember. [05:55] Erm. [05:55] Does asuka run poppy? [05:56] I didn't think it did. [05:56] You can do recipes and copies there, but not uploads AFAIK. [05:58] Doesn't look like it does. [05:58]