[00:02] bah, gdb not installed [00:03] rofl [00:03] wgrant: probably, but i suspect that it will be very boring, just twisted sitting in an select loop with no active sockets [00:04] a clean shutdown *should* tell us about un-gc'd deferreds [00:04] also [00:04] triggering a gc from gdb should do that [00:05] mwhudson: There should be lots of them, though... [00:05] yeah [00:05] lsof output too then [00:06] Aha. [00:06] Got it. [00:06] mbarnett: Can you check if there's a running PPA builder reset trigger? [00:06] I'm not sure what it's called. [00:07] vm_resume_command is the config key. [00:07] lifeless: flacoste suggested i should also land the flags ui into lp devel [00:07] since at the moment it seems to now be only in db-devel [00:07] but its precondition db changes should now be in devel [00:07] poolie: yes, definitely [00:08] wgrant: i don't see one off hand [00:08] Hm, now it won't respond to SIGTERM. [00:08] * wgrant is trying a few things locally. [00:10] mbarnett: There really should be one running. [00:10] wgrant: a hung one ? [00:10] lifeless: Ideally. [00:10] It looks like it's an ssh process. [00:11] ./ftpmaster/launchpad-lazr.conf:vm_resume_command: ssh -i /home/lp_buildd/.ssh/ppa-reset-builder ppa@%(vm_host)s [00:11] It will hang if that does, but recovers if it dies, AFAICT. [00:12] here is everything running as the lp_buildd user: http://pastebin.ubuntu.com/479127/ [00:13] Mmm. Zombies. [00:14] So, it will hang until the resume trigger exits after a communication failure, bypassing even the normal timeout. But I don't see why it would continue to hang if there's no process running. [00:15] hah, i think lamont came along and kicked the hung process. [00:16] Ah, yes, everything's running again. [00:16] yeah, now we are going [00:16] So that was probably it. [00:17] lamont> or a child ssh process killed. /me did that, seems happier now [00:17] Phew. [00:18] lifeless: ping [00:18] okay, i see the ssh process he killed [00:18] good to know that will get things moving agian [00:19] lifeless: I've got the bzr failure down to two test files [00:19] COOL [00:19] bah [00:19] lifeless: is it worthwhile trying to eliminate individual tests? [00:19] Cool [00:19] thumper: it may well be [00:19] lifeless: ok [00:23] well, all builders are now actually doing stuff, so that is good [00:27] mbarnett: Bug #618955 [00:27] Bug #618955? [00:27] :( [00:29] thanks for filing that. I have subscribed us losas [00:31] lifeless: gc.collect likely to screw things up? [00:31] thumper: I wouldn't expect it too [00:32] lp.translations.scripts.tests.test_message_sharing_migration.TestSharingMigrationPerformance [00:32] lifeless: that test makes the other one fail [00:32] lifeless: I'm just seeing if it is either test in that test case, or a specific one [00:32] progress! [00:34] lifeless: either of the two tests in that testcase cause the failure [00:51] lifeless: calling TestSharingMigrationPerformance._listLoadedObjects triggers the error [00:51] which uses gc.get_objects [00:52] I'm guessing that is it [00:52] zomg [00:52] yes, a gc held reference to a lazy object is equivant to a bound alias [00:52] don't-do-that [00:52] :) [00:53] advice needed on how to fix this test [00:54] hmm [00:54] easiest not to call that damn thing at all [00:55] its a high cost function [00:55] perhaps installing a Storm Tracer [00:56] its also a poor surrogate for what I think they want to test [00:56] I don't really understand their test [00:56] so [00:56] I think what they want to say is [00:57] 'if there are 15 pomsgids in memory before you call this function, there are only those 15 in memory afterwards' [00:57] and the implication is that no pomsgids were loaded in the interim [00:57] this only works if there is a non-weak cache installed (we have one) in the storm layer, and nothing resets it or otherwise invalidates it. [00:58] try this [00:58] do a gc.collect() at the top of _listLoadedObjects [00:58] this doesn't invalidate the test because: [00:58] - gc collect can happen anytime [00:58] this should fix the problem because: [00:59] - we don't have any references to the scope replacer [00:59] trying [00:59] but also please file a tech debt bug - this is a very roundabout way to say that something didn't happen - its a surrogate for something we can measure directly with only a little more effort [00:59] nope [00:59] still fails [00:59] darn [01:00] oh [01:00] by bad [01:00] we have code for storm tracers [01:00] so that isn't working because [01:00] if it will only take a little more to fix, I could do that now [01:00] there is a module in bzrlib that hasn't triggered its lazy load. [01:01] and we go past it twice, or something [01:01] though I am a little surprised it didn't work. Does it blow up in that function, or in your test code? [01:02] we have StormStatementRecorder [01:02] lifeless: it blows up in the following test [01:02] I would cjec that there are no Selects that return a [01:02] *check* [01:02] that seems much more targeted, to me. [01:02] hmm... [01:02] the statement recorder can't check that [01:03] it records the raw sql [01:03] SELECT.*POSMSGSETID\..*FROM -> boom [01:05] I'm recording the statements now, with a pdb set [01:05] to look [01:05] well [01:05] we don't expect to see that, because the test 'passes' :) [01:07] Ursinha: timeout bugs get high + triaged :) [01:07] zero-oops-policy [01:07] just fyi [01:07] you don't see POSMSGSETID.* in the SQL [01:07] good [01:07] it gets expanded to all columns [01:07] storm expands the * [01:07] right [01:07] and it qualifies them [01:07] posmsgsetid.foo [01:07] ah, I see what you mean now [01:08] is what you'd see when pomsgsetid's are looked up [01:08] lifeless, cool, should I triage them as such when filing them? [01:08] Ursinha: where there is as clear a policy as the zero-oops-one, I absolutely think so [01:08] cool [01:08] Ursinha: where its going to be team-determined, then no - or at least, I don't in those cases except [01:09] lifeless, what's the difference [01:09] where I am putting my TA hat on and asking as a favour-to-the-TA to do something [01:09] Ursinha: well a non-oops non-timeout bug doesn't have a predetermined importance [01:09] is any in python 2.5? [01:09] lifeless, ah, I was only talking about timeout bugs :) [01:10] Ursinha: :) [01:10] Ursinha: I was giving a general answer :) [01:10] sorry for de confusion [01:10] lifeless, no problem, thanks for the information :) [01:29] lifeless: can you check out this? http://pastebin.ubuntu.com/479160/ [01:29] lifeless: do you think it is testing enough of the same thing? [01:29] +1 [01:30] I certainly do [01:30] should be faster too [01:30] lifeless: I'll break it into a separate branch [01:32] lifeless: and no bad interaction [01:33] lifeless: about performance tuesday, is the looking up of branch subscriptions because of the checking for launchpad.View ? [01:33] lifeless: if so, have you seen how the branchlistings avoid this particular death by sql? [01:35] * thumper files a bug on rosetta [01:36] mwhudson: I don't know [01:36] mwhudson: I'm trying to get a test that is precisely the same [01:36] ok [01:37] mwhudson: for now, I have one that shows 'one more query as you add a private bug' [01:37] let me paste [01:38] so, I can fix one cause of too many queries [01:38] but I know I'll be missing the particular one :( [01:38] http://pastebin.com/wPNqcuzn is my test [01:39] mwhudson: AIUI you retrieve too much and then do a custom security check [01:39] lifeless: 'you'? [01:39] 'branhc listings' [01:39] that's the launchpad default usually, it's not what branch listings do [01:40] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1689EB3170 [01:40] is the oops I'm trying to duplicate [01:40] branch listings do a query to find the branches the user can see and then pre-populate the permission cache [01:40] 3286439196420launchpad-main-slaveSELECT BugSubscription.bug, BugSubscription.date_created ... [01:40] blah [01:40] so the security adapter doesn't do any queries [01:40] 328 6439 19 6420 launchpad-main-slaveSELECT BugSubscription.bug, BugSubscription.date_created, BugSubscription.id, [01:40] is what I'm trying to make my test trigger [01:40] but I'm having trouble making it do it [01:41] possibly due to caching :) [01:41] lifeless: also, another trick [01:41] lifeless: do you know about LP_DEBUG_SQL_EXTRA ? [01:41] do [01:41] what doth that do? [01:41] make run LP_DEBUG_SQL=1 [01:41] gives a metric shit-load of output [01:41] prints all queries as the output [01:41] extra gives a stacktrace too [01:41] LP_DEBUG_SQL_EXTRA gives tracebacks as well as the sql [01:42] ok [01:42] very very handy [01:42] as thumper says,. it's mounds of output [01:42] in optimisation work [01:42] but can be handy to figure out where that query is coming from [01:42] I had 70k lines of output for the branch index page [01:42] I should capture that for these performance tests [01:42] toss it into kcachegrind [01:43] lifeless: https://code.edge.launchpad.net/~thumper/launchpad/fix-TestSharingMigrationPerformance/+merge/32828 [01:43] lifeless: if you make the mark on that, I'll land the fix [01:43] lifeless: and that was pretty much the only thing blocking bzr 2.2 landing === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [01:49] mwhudson: so the problem with reproducing the query is that I don't know how to do it *at all*, not just in-a-test, other than 'visit this page that oopses' :) [01:50] lifeless: challenging [01:53] even sprinking store.invalidate()'s doesn't trigger it [01:53] ah well, so I'll fix the problem I *have* demonstrated. [01:56] it would be nice to be able to say [01:56] 'now make run should give me this test environment' [01:56] or something [01:57] barry had some terrible hacks for that iirc [01:58] sync_to_demo() [01:58] ? [02:23] do windmill tests hate everyone, or just me ? [02:26] lifeless: It's everyone, and life itself [02:31] does this mean anything to anyone ? [02:31] ====================================================================== [02:31] ERROR: lp.registry.windmill.tests.test_person_picker.TesPersonPickerWidget.test_person_picker_widget [02:31] ---------------------------------------------------------------------- [02:31] _StringException: Text attachment: garbage [02:31] ------------ [02:31] [] [02:31] ------------ [02:33] I see that most times I run that. [02:33] So I just sort of ignore it. [02:33] While giving it a look of extreme disapproval. [02:33] blocking landing is asad [02:34] Oh, it did that in EC2? [02:34] yes [02:34] :/ [02:34] yes [02:37] >< [02:37] my registry /participants fix works here [02:37] but there is an extra query on prod [02:37] sorry [02:37] ec2 [02:38] lifeless: for reasons i have (shamefully) never really dug into, i find that when i have a genuine failure in ec2, i get a windmill failure too [02:38] when i fix the genuine failure, the branch lands ok [02:39] maybe this is just luck or maybe something else [02:39] sigh [02:39] a new SELECT Person.account, Person.creation_comment, Person.creation_rationale, Person.datecreated, Person.defaultmembershipperiod, Person.defaultrenewalperiod, Person.displayname, Person.hide_email_addresses, Person.homepage_content, Person.icon, Person.logo, Person.mailing_list_auto_subscribe_policy, Person.merged, Person.mugshot, Person.name, Person.personal_standing, Person.personal_standing_reason, Person.registrant, Person [02:39] mwhudson: interesting [02:40] I've sent cachedproperty off on its own with the registry branch fix [02:40] while I try to figure out wtf I see an extra query even with devel merged in [02:40] maybe something landed in the last few hours [02:57] oh wow [02:57] we trigger a DB lookup on every person lookup via this code path [02:57] shudder [03:08] yay storm bugs [03:08] object cache coherency, can you imagine it? [03:12] it sounds fairly unlikely :-) [03:15] har har har [03:15] thats twice in 1 month [03:15] https://bugs.edge.launchpad.net/storm/+bug/619017 [03:15] I'm not trusting the layer much now, as a result [03:16] I'd rather storm didn't have an object cache. [03:16] it would make it much simpler [03:16] and more focused [03:19] mwhudson: or perhaps you were being sardonic ? [03:20] lifeless: a touch, yes [03:21] ha! [03:34] poolie: ping [03:35] mwhudson: know the simplest solution to this? http://pastebin.ubuntu.com/479196/ [03:36] thumper: :( [03:36] mwhudson: could trim the input on _show_lline [03:36] mwhudson: but we'd still have the | [03:37] thumper: let me try to remember what the test is actually testing [03:37] mwhudson: most of the TestLoggingUIFactory tests fail [03:37] with similar errors [03:37] I'm guessing the base class was updated in 2.2 [03:38] thumper: yeah, looks like progress reporting changes in bzr [03:38] thumper: possibly needs a code fix, as that's going to make the code import logs look rather lame i suspect [03:38] :-| [03:40] I wonder if the _render_bar was renamed [03:40] i'm just updating my bzr branch [03:40] (i think it's fairly ancient) [03:41] * thumper runs away again [03:44] thumper: got a branch i can run tests in? [03:45] I wonder if overriding _avail_width to always return None in LoggingTextProgressView will help [03:45] poolie: [03:45] ^ [03:49] alternatively def _render_line(self): return self._render_bar() will probably do it [03:55] mwhudson: so [03:55] File "/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/webapp/authorization.py", line 207, in checkPermission [03:55] principal.account) [03:56] is that the code path you're talking about w.r.t. checking permissions doing db access ? [03:56] probably [03:56] oh [03:56] is the next call down in canonical.launchpad.security? [03:57] lifeless: ^ [03:58] checkAccountAuthenticated [03:58] yes [03:58] then yes [03:58] so the *menu* is generating this [03:58] :/ [03:58] enabled_with_permission ? [03:58] what's the context object being checked? [03:59] dunno [03:59] but probably milestone/product [03:59] mm [04:00] lifeless: can you pastebin the whole traceback? [04:00] sure [04:01] http://pastebin.com/EkXrtUK6 [04:03] mwhudson: yeah, I do [04:04] lifeless: wee, it's checking if the user can review the page [04:04] er [04:04] lifeless: wee, it's checking if the user can review the product [04:04] mwhudson: lp:~thumper/launchpad/new-bzr [04:04] ReviewByRegistryExpertsOrAdmins [04:04] mwhudson: how did you figure that out ? [04:04] although I'm about to run out yet again [04:05] lifeless: # [04:05] File "/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/security.py", line 232, in checkAuthenticated [04:05] # [04:05] return user.in_admin or user.in_registry_experts [04:05] -> look at line 232 of security.py [04:05] hi tech haxorring! [04:05] oh [04:05] I figured that would be generic. serves me right [04:05] it's crummy, but it's a once per page check [04:06] mwhudson: found it [04:06] thumper: oh? [04:06] mwhudson: we set _render_bar to return '' [04:06] mwhudson: the base _render_line method does bar_string = self._render_bar() first [04:06] then [04:07] if (task_part or trans) and not bar_string: [04:07] bar_string = '| ' [04:07] further down [04:07] mwhudson: then why do I see two of them ? :) [04:07] aah [04:07] :( [04:07] lifeless: both sides of the or execute a query i guess [04:07] * lifeless facepalms [04:07] lifeless: i guess i should have said "fixed number of queries per page" [04:07] lifeless: ORMs r grate [04:08] thumper: i guess overriding _render_line() is one fix [04:09] mwhudson: yeah [04:10] I'll look at it again when I get back [04:11] mwhudson: want to have a rotation in 6 months or so, to work on a separated mapper approach ? :) [04:12] lifeless: heh [04:13] lifeless: we should rewrite launchpad in haskell and use meta-programming to compute the set of needed queries before we actually start executing code for the page!! [04:13] mwhudson: funny you should say that [04:14] we do need something along those lines (the needed queries, not haskell) [04:14] lifeless: given the pypy background, perhaps not so much [04:14] mwhudson: :) [04:16] it some sense, it would be fairly easy to do the page rendering twice, once to collect the queries and then again for real [04:16] you could even do the first bit ahead of time [04:16] the fun part is conditionals, of course [04:17] fsvo fun? [04:18] you _could_ abstractly interpret the code repeatedly, taking a different path and giving a different answer for each branch [04:18] pypy does exactly this [04:19] see [04:19] while this sounds extremely promising [04:19] I'm actually ok with just putting a cap on the queries [04:19] :) [04:19] and failing tests if the cap is exceeded [04:30] hi thumper, mwhudson [04:32] mwhudson: all ok now? [04:33] how can I tell if an Interface is exported via APIS ? [04:34] if it has export* decorators on it [04:35] ok [04:35] so BugTaskSearchParams isn't exported [04:35] \o/ I can mess with it [04:43] poolie: i think so, although it's really thumper who's worrying about this [05:00] can anyone explain what a 'non conjoined task' is ? [05:05] lifeless: A task that isn't conjoined. Do you know what a conjoined task is? [05:09] no [05:09] lifeless: When you're done with wgrant, do you have time to mumble? [05:09] sure [05:09] I think I've *finally* found where this private check is kicking in [05:10] StevenK: but skype please [05:10] it works [05:10] lifeless: If you have a task for a project and its development series, the project task is hidden, and becomes a conjoined slave to the development series task. [05:10] (that's then you see "Status tracked in SomeSeries") [05:10] ok [05:10] StevenK: rbtcollins [05:10] Setting the status on the series task sets it on the project task too. [05:10] * StevenK grumbles how lifeless hates freedom, and hides [05:11] StevenK: do you have skype? [05:12] lifeless: Yes [05:12] and do you want to talk [05:12] lifeless: Yes -- and I cope with mumble's lag every stand-up, so ... :-) [05:12] lifeless: So I'm only teasing [05:22] arch_tag in [%s] [05:22] where the %s is [05:22] sqlvalues(tags) [05:22] or something [05:22] you need , [05:23] 's [05:23] so I guess ",".join(sqlvalues(*arch_tags)) [05:23] include = "architecturetag IN (%s)" % sqlvalues( [05:23] ','.join(self.arches)) [05:23] only [05:23] include = "AND whatyou haveabove [05:24] sqlvalues(list(self.arches))? [05:25] wgrant: I guess [05:25] wgrant: but why the list ? [05:25] wgrant: But I can't just blat that into a string with %s? [05:25] lifeless: so i am thinking of just using a textarea for editing the rules [05:25] its a tuple already [05:25] at least to start with [05:25] poolie: +1 [05:25] since it's a very nerd-oriented thing [05:26] lifeless: Ah, not already a resultset? [05:26] s/already/just/ [05:26] wgrant: Damn it, it's my code. :-) It's a tuple [05:26] poolie: hi [05:26] Just sqlvalues(self.arches) should work, then. [05:26] wgrant: does that return a iterable of strings ? [05:26] hi thumper [05:26] poolie: I was discussing the issue we have with our code import bzr logger [05:26] poolie: and that things changed [05:26] poolie: http://pastebin.ubuntu.com/479196/ [05:27] poolie: is an example of the failure [05:27] poolie: just working out what to change in our logger [05:27] huh [05:27] was thinking overriding _render_line [05:27] i would guess this is actually a uifactory problem [05:27] and that these tests want to test TestTextUI [05:27] but they don't explicitly set it, they just assume it's the default [05:27]