[00:00] how does one specify a new dependency to buildout ? [00:03] lifeless: setup.py [00:03] mwhudson: I'm looking in lazr.batchnavigator [00:03] mwhudson: I have a test-only dependency, or should I not worry about the subtlety [00:04] setup.py *might* support test_requires? [00:04] * lifeless goes with damn-the-torpedoes [00:08] thumper: wgrant mumble? [00:08] sure [00:08] Sure. [00:09] wallyworld: you around? [00:09] He's on mumble. [00:09] yep [00:10] thumper: sinzui: i can't hear you guys. looks like my audio is foobar again. will have to restart :-( [00:10] wallyworld: ok [00:11] wgrant: does mumble need stabbing? [00:11] sinzui: Thumper and I are talking OK... [00:12] sinzui: I can hear him [00:12] sinzui: and you [00:12] maybe I fell off the earth [00:12] I feel like that actually [00:13] sinzui: we can hear you [00:13] sinzui: it is just you [00:13] fab [00:16] mumble gave up on it's own [00:23] thumper: sinzui: sorry, my audio has totally died for some reason :-( [00:24] wallyworld: okay. mumble booted me from the conversation twice [00:25] sinzui: it's not mumble. i have no output at all. speakers/headphone all silent :-( [01:01] lifeless, shall we stick with https://bugs.launchpad.net/launchpad/+bug/525758 [01:01] <_mup_> Bug #525758: A project's branch privacy policy is not visible on the branch settings page < https://launchpad.net/bugs/525758 > [01:01] cyes [01:01] yes [01:06] poolie: mail sent; I hope I captured everything - if I didn't, please correct me ;) [01:07] very nice, thanks [01:10] ok, https://bugs.launchpad.net/launchpad/+bug/750871 [01:10] <_mup_> Bug #750871: want an api and ui to set branch privacy policy < https://launchpad.net/bugs/750871 > [01:10] lifeless, it looks like you could create a Branch through the api with private=true [01:11] which would avoid a race of making them private after they're created [01:11] in that case it would be enough to just have bzr call it [01:11] (arguably a bit messy to need to do this as well as sshing in, but not too bad) [01:11] do you think that won't work? [01:12] you still need a policy setup to permit privacy at all [01:13] + having to use bzr push --use-existing-dir [01:14] wgrant: ugh [01:14] wgrant: we broke 'redirect to a real context' [01:15] https://bugs.launchpad.net/launchpad/+bugs/5011 [01:15] well, if bzr called the api it would hopefully be smart enough to expect a directory there [01:15] lifeless: We didn't. [01:15] wgrant: it was already broken ? [01:15] lifeless: The container is '+bug' [01:15] lifeless: Not '+bugs' [01:15] ah, so the default policy, even if you have an entitlement to private branches, is to disallow them? [01:15] * lifeless fails [01:15] poolie: right [01:16] poolie: noone can has private branch until a policy is setup permitting a group they are in to have them [01:17] poolie: I think 750871 is a dupe of 525758 isn't it ? [01:18] ah, no its not [01:20] they're arguably the same, but one is "show it" and the other is "let me change it" [01:20] i was going to file one more for "change it per branch" [01:20] well, that already works, just only one way [01:21] huwshimi, it seems like pressing enter in the tags field no longer saves the changes [01:21] a search for 'branch privacy' is pretty interesting [01:21] poolie: Can you file a bug please? [01:22] i will [01:22] oh also https://bugs.launchpad.net/launchpad/+bug/252172 [01:22] <_mup_> Bug #252172: Changing a branch's status/privacy isn't obvious < https://launchpad.net/bugs/252172 > [01:22] lifeless, is bug 444498 a dupe of bug 347772? [01:22] <_mup_> Bug #444498: branch privacy for packages (of projects with privacy enabled) < https://launchpad.net/bugs/444498 > [01:22] <_mup_> Bug #347772: Privacy policy support for package branches < https://launchpad.net/bugs/347772 > [01:22] poolie: no [01:23] but I just edited it before you asked anyhow :) [01:23] * wgrant hunts for someone to review https://code.launchpad.net/~wgrant/launchpad/bug-750640/+merge/56276 [01:23] poolie: 444498 is about extending privacy to package branches when the upstream has a commercial ticket [01:24] 347772 is about permitting distro contributors to do private branches for embargoed changes (BFB into primary archive will need that) [01:24] wgrant: is + fields.append('Source', source) right ? [01:25] wgrant: if source is None, will that DTRT [01:26] lifeless: Yes, see eg. the handling of Essential. [01:26] huwshimi, https://bugs.launchpad.net/launchpad/+bug/750880 [01:26] <_mup_> Bug #750880: pressing enter in bug tags field doesn't save changes < https://launchpad.net/bugs/750880 > [01:26] wgrant: its almost worth doing a Matcher there [01:26] wgrant: but your helper is awkward, a nicer spelling is included in my review [01:27] Thanks. [01:27] huwshimi: hi [01:27] poolie: Does that happen when you hit the save button as well? [01:27] lifeless: Hello. [01:28] huwshimi: I don't recall any feedback from you on my suggestion about your easy-ui bugs & priority [01:28] lifeless: Treating it as a dict is wrong, but I guess it's OK for tests. [01:28] huwshimi: so I thought I'd poll you for some [01:28] huwshimi, there is no save button [01:28] as i suspected it's due to a problem in tag suggestion [01:28] wgrant: your test treats it as one :) [01:28] or so it seems [01:28] poolie: OK then this looks like a new issue [01:28] wgrant: so either match on a vector (and definitely do a matcher), or be much more pithy :> [01:29] i think i'ts fallout from the previous fix [01:29] hm so https://bugs.launchpad.net/launchpad/+bug/120306 suggests there is already a ui for this [01:29] poolie: tag edits are saving for me [01:29] it may be to do with using a new tag? [01:29] poolie: yes there is, as I said. But public->private is deliberately not supported [01:30] then you can't toggle it, can you? :) [01:30] poolie: no :> [01:30] well [01:30] you can toggle once [01:30] anyhow [01:30] I think that we need to revisit it [01:31] ok, so i'll file a new one, mentioning it may be complex [01:31] but I don't think its a proximate cause of pain [01:31] blah [01:31] the words are not coming today. I mean 'changing that wouldn't make a lot of difference' [01:32] lifeless: I'm not sure it's appropriate for us to elevate these bugs. If we did we would have to raise up a few hundred bugs to high. [01:33] huwshimi: but you think they are very important for the user experience, right? Or did I misread your email ? [01:33] lifeless: The idea is not that people should have to do these bugs, but a way of signifying that these are some good bugs to look at if people want to do some UI work. [01:34] huwshimi: I think this is a flawed way to analyze the problem [01:35] huwshimi: You made a compelling argument that fixing these bugs is both relatively cheap and will have a large impact on usability and product perception. [01:35] lifeless, it's exactly the button these users are wanting to push [01:35] i agree it would be better if they'd never gotten in to this state [01:35] anyhow, https://bugs.launchpad.net/launchpad/+bug/750889 [01:35] poolie: but they want to push it because another critical step has been missed. [01:35] <_mup_> Bug #750889: need to be able to make public branches private < https://launchpad.net/bugs/750889 > [01:36] lifeless: In as much that fixing any UI bugs is a good thing. [01:36] huwshimi: well, the UI isn't a special case for users: a bug is a bug [01:37] hard to use is hard to use [01:37] ok now to fix the actual branch === lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | 12:40 -!- Topic for #launchpad-dev: Performance Tuesday! | devel in RC until r12735 releasable | https://dev.launchpad.net/ [01:39] huwshimi: If I understand correctly, you are postulating a disconnect between 'this is important (to the UI)' and 'this is important' [01:39] huwshimi: is that correct? [01:40] lifeless: I don't think the bugs I've tagged are necessary any more important than other UI bugs (in some cases exactly the opposite). All I am doing is identifying bugs that are cheap to fix. [01:41] huwshimi: now I'm really confused [01:41] lifeless: One sec. [01:41] lifeless: You've screwed up the topic a bit [01:42] huwshimi: you said in your email that these were high leverage bugs, holding three properties: easy to fix, only need ui changes, make a big difference to users [01:42] bwah === lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews [01:42] lifeless, shall we have a bug for 'bzr push --private'? [01:42] poolie: I think that would be good [01:42] istm that's a bit less important than getting the policy set right [01:42] but may be worth filing anyhow [01:51] lunch I think before tackling the inmemory xmlrpc mock for the stacking changes [02:28] huwshimi: the ajax log stuff is awesome [02:29] I've having a few wtf moments [02:29] (at latencies) [02:30] lifeless: Yeah it's nice to see that stuff all the time. We're actually under for quite a few queries, and most of the rest are are not a long way over [02:31] huwshimi: I had some come in at 24 seconds this morning >< [02:31] lifeless: Oh really!@ [02:33] yeah [02:33] I may have hit browser concurrent connection limit or something [02:33] I opened a few tabs [02:48] wgrant: why isn't it a regression ? [02:49] lifeless: What regressed/ [02:50] wgrant: our hiding of merge proposals [02:50] wgrant: when we added a feature (prereqs) we backslid on disclosure [02:51] Mmmmm. Maybe barely. [02:51] We're not really disclosing stuff we shouldn't, though. [02:52] So it's more that prereqs are broken when the prereq is private but the proposed branch is not. [02:52] I think that makes it a broken feature, not a regression. [03:07] disclosure is defined as being viral [03:07] thats not working here [03:07] thats a regression [03:09] Viral to what extent? [03:10] A private bug does not make a branch private. [03:10] A private branch does not make a bug private. [03:10] A private prereq should possibly not make an MP private. [03:10] a branch link is invisible if either the branch or the bug is invisible [03:10] a mp is invisible is any of the items on it are invisible [03:10] Except a bug or blueprint or [03:11] one step removed [03:11] mmm, viral is the wrong word [03:11] I would love to see 4 / 0 None fixed [03:13] Hmm, it's the librarian. [03:14] Erm. [03:14] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1920MPJ4 [03:14] Reads the same file 376 times. [03:16] wgrant: I actually meant the 'None' part :) [03:26] lifeless: Do you know of a bug for late evaluation of attachment LFCs on BugTask:+index? [03:26] no [03:26] Have you looked at that at all? [03:27] shouldn't be happening [03:27] perhaps fallout from librarian code changes you did? [03:27] Which? [03:27] It only affects attachments that appear in comments. [03:27] removing the feature flag [03:27] oh [03:28] No, that's unrelated. [03:28] hadn't seen that at all [03:28] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1920A1682 for example. [03:28] 3s selecting attachment LFCs. [03:28] Fixing now. [03:28] win [03:28] Rendered in 10.9s for me just now :/ [03:28] why does it need the LFC's at all ? [03:28] (with ?comments=all) [03:28] It shows the file size. [03:29] whatnow [03:29] Xorg.0.log (23.1 KiB, text/plain) [03:29] in the comment ? [03:29] This is in the comments, not the portlet. [03:29] hmm... [03:29] 178 queries/external actions issued in 5.27 seconds [03:29] lifeless: ?comments=all [03:29] we really should show the real url in the oops report [03:30] 521 queries/external actions issued in 10.93 seconds wheeeee [03:30] lifeless: The query string is there if you look hard, but yes. [03:30] lifeless: Exactly. [03:30] wgrant: gets truncated sometimes [03:30] wgrant: so, are you going to eager load, or make the attachment the same as the portlet [03:31] lifeless: It should be really cheap to eager load. [03:31] cool [03:31] I mean, LFC isn't that fat. [03:32] Hmm, except the hashes. But it's still <100kb of extra data. [03:33] sure [03:33] Bug search really, really sucks. [03:39] This will possibly fix half of the BugTask:+index timeouts. [03:39] I hope. [03:40] Mm, although I don't know what the OOPS report shows. [03:40] Since it shows more entries for BugTask:+index in "top timeouts" than there were timeouts. [03:42] rotfl [03:50] lifeless: how can I tell if a repository is stacked or just pretending? [03:50] lifeless: how can I open the repository to find the number of revisions? [03:51] I have a branch that appears to be stacked locally using my new branch id alias [03:51] thumper: I'm not sure what "just pretending" could mean [03:51] but I want to confirm for myself that it is doing what it says [03:51] spiv: I mean is saying it is stacked, but the branch repository has all the revisions [03:51] Either a repository has at least one fallback repository, or it doesn't… [03:51] can it do that? [03:51] does it confirm that the fallback repository is valid? [03:52] Probably you can have the state where the stacked repo has all the revisions (e.g. stacking on an empty repo) [03:52] But I don't know why you'd care? [03:52] humour me [03:52] bzrlib always opens all the fallbacks [03:52] Unless you explicitly do Branch.open(ignore_fallbacks=True) [03:53] I'm guessing “can be opened” is what you mean by “is valid”? [03:54] open(base, _unsupported=False, possible_transports=None) [03:54] no ignore_fallbacks [03:54] repository? or bzrdir? [03:55] *Branch* [03:55] thumper: you can look at len(branch.repository.all_revision_ids()) [03:55] The fallback location is defined in the branch.conf [03:55] (iirc) [03:55] If you open a repository directly, without going via a branch, it will have no fallbacks configured. [03:55] thumper: it opens the fallbacks at Repository.open() / Branch.open() [03:56] thumper: it will fail eagerly [03:56] lifeless: not at Repository.open [03:56] thumper: this may change in future but hasn't yet [03:56] spiv: yes, true, I'm handwaving [03:56] because I hadn't read your prose yet :P [03:57] thumper: (BzrDir.open_branch also accepts ignore_fallbacks=…) [03:57] http://pastebin.ubuntu.com/589515/ [03:57] seems to be working [03:57] info -v tells me what I care about [03:57] and that is the 3 revisions bit [03:57] thumper: but that said, it sounds like you just want to open the branch, and if it can be opened then the repository and its fallbacks must exist. [03:58] There may be a good reason why you want to check whether the revisions exist in the branch's repo or the stacked-on repo, but it's not obvious to me why you should need to check that. [04:01] thumper: I would expect info -v in that context would be counting revisions in the fallbacks as well as that direct location. [04:05] * spiv → lunch [04:11] spiv: no, it doesn't [04:11] spiv: which is good [04:12] interesting [04:13] bzr info -v http://bazaar.launchpad.dev/~thumper/wikkid/soundex doesn't show the count of revisions in the repository [04:22] lifeless: I'm experiencing delays in Q/A for db-devel r10383. What are the chances of that one becoming a blocker for this rollout? [04:23] jtv: 0 [04:23] jtv: The merge has already occurred. [04:23] r10381 [04:24] so no? thanks, that's good to know [04:25] Right. [04:25] It is very scary and needs to be QAd at some point, but it's not 11.04-criticasl. [04:27] Well the interesting part is that besides the big change, I'd also want to see that the minor changes to the original script are still safe. [04:28] jtv: FTR I mailed the -dev list on monday about this [04:28] jtv: I'd like to know if I was unclear etc so I can communicate better in future [04:28] jtv: I meant that both are very scary. [04:28] jtv: That script hasn't been changed significantly in a Long Time. [04:28] Even though these changes look pretty safe. [04:30] lifeless: I see it now—I suspect thunderbird has been hiding it from me. [04:32] wgrant: it's extra-scary even because I know so little of what to expect from the scripts that it invokes. Did you see the run-parts directories in cronscripts/publishing/distro-parts? [04:33] bbiab [04:54] wow the new +authorize-token is quite slick [04:54] well done leonard, or whoever [04:54] The system-wide one? Yeah. [04:55] (Pdb) from lp.bugs.model.bug import Bug [04:55] (Pdb) Bug.get(16) [04:55] [04:55] (Pdb) print self.context.bugID [04:55] 16 [04:55] (Pdb) print self.context.bug [04:55] None [04:55] What? [04:56] Hmm. Maybe I have a slave object, I guess, but... [04:56] ... and attempting to check self.context._store crashed pdb. Yay. [04:56] win [05:03] wgrant: this is a new bugtask ? [05:04] lifeless: Yes. I create it, test the query count, add attachments, test the query count. The query count test seems to be removing it from the store, which I guess shouldn't surprise me too much. [05:04] its invalidating the world [05:04] grab it again from the set [05:05] (at a guess) [05:05] psatebin ? [05:09] lifeless: Retrieved, still broken :/ [05:10] wgrant: ok, thats new then [05:10] I've not seen *that* before [05:10] time to edit z3batching [05:10] if I'm not back in an hour, send in a search party [05:10] Heh [05:11] * StevenK goes to tell the people in the next apartment where they can store the masonry drill they've been using since 9am. [05:12] It's in the room right next to me, so I have a splitting headache [05:12] lifeless: http://pastebin.ubuntu.com/589533/ [05:13] makeBugAttachment works if you drop line 59 of that diff. [05:13] :-((( [05:13] WTF? [05:14] running a test I get a failure that looks hard to explain [05:14] so I run the test with -D [05:14] it stops at: -> result.addSuccess(test) [05:14] (Pdb) c [05:14] so I continue [05:14] Ran 1 tests with 0 failures and 0 errors in 11.213 seconds. [05:14] stupid POS [05:16] wgrant: that makes no sense at all [05:16] lifeless: The test, or the problem? [05:17] wgrant: the problem [05:17] wgrant: there is no store.reset etc etc in that matcher [05:18] nor a store.invalidate [05:18] A store.invalidate won't remove the objects, either... it should just mark everything AutoReload, right? [05:19] store.invalidate is pretty shallow yes [05:19] store.reset is harsher, IIRC [05:20] Right, I think reset does some really bad stuff. [05:20] I guess I'll just create a fresh task for now :/ [05:23] wgrant: I'd really like to know whats up [05:23] wgrant: have you tried to bisect? [05:27] lifeless: Bisect the matcher? [05:27] I probably should. [05:30] wgrant: yes [05:35] can I get someone else to run TestBranchChangedErrorHandling ? [05:35] I have it failing on devel [05:35] unless I add -D [05:35] in which case it kinda passes === almaisan-away is now known as al-maisan [05:44] thumper: I'm pretty sure 'bzr info -v' of a branch does count revisions from the stacked-on repository. What makes you think otherwise? [05:45] spiv: it shows the full history count for the branch, but the repository says 3 revision [05:45] spiv: since the output says that, that is why I think that way [05:46] thumper: not in quick testing locally [05:46] thumper: and not according to my reading of the code [05:46] spiv: is for me [05:47] spiv: http://pastebin.ubuntu.com/589515/ [05:48] It's probably affected by running over a smart server. [05:48] Probably that's a bug. [05:49] but very handy [05:51] You're probably the first person to think so :) [06:07] WTF :/ [06:11] wgrant: ENODETAIL [06:14] Set up a browser? Fine. Browse to a question? Fine. Browse to a bug? You can't use makeBugAttachment any more. [06:15] Somehow browsing to BugTask:+index manages to break the subscribers, I think. [06:15] lifeless: ^^ [06:15] makeBugAttachment doesn't work at all, even on a new bug. [06:18] (this affects anything usin setupBrowserForUser, not just the matcher. [06:19] wgrant: I thought it wasn't the matcher ;) [06:20] So did I, but I didn't have evidence. [06:29] wgrant: Pity me, I'm debugging the packagecloner [06:30] StevenK: pity the fool ? === al-maisan is now known as almaisan-away [06:48] Oh, LaunchBag. [06:49] Fucking LaunchBag, please die. [06:50] wgrant: Then remove it? [06:53] I have some misgivings about the output of *canonical*_url depending on what you've traversed through lately. [06:55] wgrant: Perhaps it wants to make sure you take your shoes off so you don't dirty the carpets. [06:56] Hah [06:57] wgrant: rotfl [06:57] Sigh, I thought the build was linked from BPPH [06:58] StevenK: You have to go through BPR. [06:58] lifeless: No. [06:58] Ah, I see it. [06:58] Unless you are rolling around to squash LaunchBag flat, in which case it might be acceptable. [07:17] sigh at domain name spam [07:38] pop quiz [07:38] nvm [07:52] lifeless: is there no way to resolve a qa-bad other than roll back the entire revision and then land a fixed re-run of the branch? [07:55] jtv: you can just land a fix [07:56] jtv: have it claim to rollback the bad revision [07:57] lifeless: thanks. I think https://dev.launchpad.net/QAForContinuousRollouts sort of skips over the exact meaning of --rollback; I thought it had bzr implications. [07:57] jtv: feel free to tweak it; its purely a workflow tag for us [07:58] if the commit says '[rollback=1234] fix 1234', then its pretty clearly not a rollback [07:58] lifeless: so it just pairs the revisions to mark the fact that the newer revision fixes a qa-bad on the older one? [07:58] IWBN to have qatagger have a unbreak=1234 that does the same as rollback but doesn't confuse people [07:58] jtv: yes [07:59] OK… I guess https://dev.launchpad.net/QAProcessContinuousRollouts needs updating as well. [07:59] lifeless, oh was your complaint about domain spam by any chance inspired by pda.lv? [07:59] i just hit that [08:00] 'Internet copyright of "canonical" and Bdjikln company' [08:00] Dear President & CEO, .... - addressed to info@ and delivered to me [08:02] So CEO is Russian for "occupant"? === almaisan-away is now known as al-maisan [08:39] No reviewers today? [08:40] * StevenK pushes wgrant forward. [08:41] a volunteer! [08:41] * jtv pins suspiciously plastic-y medal and branch on wgrant: https://code.launchpad.net/~jtv/launchpad/db-bug-751054/+merge/56298 [08:42] * wgrant is looking. [08:45] jtv: Done. [08:45] StevenK: Want to mentor? [08:45] thanks [08:45] only fair :) === al-maisan is now known as almaisan-away [08:47] wgrant: Done === almaisan-away is now known as al-maisan [08:49] jtv: Well, wgrant is on the schedule as OCR today, which is why I picked on him [08:49] wgrant: I'm not sure what you mean about me saying that I reverted to the devel version… what I said was "I really just copied the devel version of the script into a db-devel branch." What's missing? [08:50] It's not actually exactly reverting, since the devel version had changed in the meantime. [08:51] Mmm [08:56] good morning [09:42] lifeless: Thanks. [09:45] Bah, real test failure. [09:55] Odd... [09:55] I can't reproduce it, but it happens on ec2. [09:55] wgrant? [09:56] Anyway, I should go. === gmb changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | On call reviewer: gmb | https://code.launchpad.net/launchpad-project/+activereviews === al-maisan is now known as almaisan-away [10:59] gmb: got a branch for you to review if it's convenient! https://code.launchpad.net/~jtv/launchpad/db-bug-244328/+merge/56310 [10:59] jtv: Sure. [10:59] Splendid, thanks [11:00] * jtv runs out for some simple, urgently-needed nutrients [11:04] are we in RC mode? [11:04] Just testfix, I think. [11:12] ffs. [11:15] jtv: r=me [11:16] not that it matters! [11:34] gmb: thanks === almaisan-away is now known as al-maisan [11:54] StevenK, wgrant: do you chaps know anything about the pending merge on df current? [11:56] jtv: You can drop it. [11:56] \o/ thanks [11:56] That's weeks old, though, and not actually applied any more... [11:58] jml: Um, that makes you feel *better*? [11:58] jml: I can see why, but it also scares me more. [11:58] Because something has changed. [11:59] Although what exactly I cannot tell, since the assertion message suggests that the text matches the regexp fine... [11:59] wgrant: it's intermittent, and it's probably because of code. [11:59] rather than environment [11:59] wgrant: grar, every time I see a test change for the publisher I want to shout loudly when the file is left in lp/soyuz [11:59] and it's only that test rather than other tests in the file [11:59] bigjools: That orig.tar.gz? Yeah. [12:00] wgrant: no, lib/lp/soyuz/tests/test_publish_archive_indexes.py and friends [12:00] should be lp/archivepublisher [12:00] Oh. [12:00] Yeah. [12:00] I guess I should have moved it. [12:00] but the code itself is in Soyuz. [12:00] So maybe not. [12:00] wgrant: I mean, I feel terrible that we tolerate having such things in our test suite, but believe me that's a small fraction of the terrible I feel, and I'm trying to take my wins where I can. [12:00] along with most of the code in the soyuz model .... [12:01] bigjools: Shall I move it before I land the branch, or keep it with the code it tests? [12:02] wgrant: rs=me if you want to move it [12:02] I think we should keep it in Soyuz until we split the classes, though :/ [12:02] maybe [12:02] bigjools: can I just run publish-ftpmaster on df or will that open a hole that the Old Ones may use to break through into our world? [12:02] (LPCONFIG=dogfood) [12:03] jtv: the cron.publish-ftpmaster script can run unaltered [12:03] jtv: Please back up dists/ first, though. [12:03] (And of course I'm just naming one of the more likely failure modes as an example; please let me know of other risks as well) [12:03] nothing will cross from the Other Side [12:03] Phew. [12:03] wgrant: dists/ is probably already fucked [12:03] bigjools: Some of it is, some of it is not. [12:04] It's reasonably easy to judge, and useful for comparisons when trying to unfuck the publisher :) [12:04] bigjools: that's /srv/launchpad.net/ubuntu-archive/ubuntu/dists/ ? [12:05] jtv: yes [12:05] right ho [12:05] wgrant: unfuck the publisher? You have a spare year? :) [12:05] deryck: have you seen bug 751187? [12:05] <_mup_> Bug #751187: Upstream translations not being imported for some Ubuntu packages < https://launchpad.net/bugs/751187 > [12:06] deryck: not sure if it's relevant to the work you guys are doing [12:06] jml, I haven't seen. I'll look now. [12:07] deryck: thanks. [12:08] deryck: there's the possibility that it's a simple outdated timestamp. [12:08] It may pay to have a look at the import queue entries. [12:41] deryck: hi. my laptop is acting up. no sound, flaky network :-( did you get my recent email? i just want to make sure you don't waste time on the windmill stuff since i've got it sorted [12:42] wallyworld_, no, but my mail setup is flaky. mail not getting to me after upgrade of server. [12:42] I'm looking into my mail problem now. [12:43] deryck: cool. i got rejection messages. my sound is totally borked for "no" reason. very frustrating. [12:45] my busted mail is very frustrating. [12:45] wgrant: I don't suppose you should be here at this time, but if you are: I got publish-ftpmaster to run in 27 minutes or so, but bigjools tells me that's a good thing. Perhaps you could take a look? [12:46] jtv: That is plausible if no release pocket was dirty. [12:46] Let's see. [12:46] deryck: did you upgrade the kernel recently? that's what did my sound in i think. or did you upgrade to natty? [12:46] wgrant: the output is in /srv/launchpad.net/codelines/current/jtv7.log [12:47] I was about to ask. [12:47] Thanks. [12:48] 2011-04-05 11:10:57 DEBUG Publishing The Maverick Meerkat-RELEASE [12:48] 2011-04-05 11:10:57 DEBUG Attempting to publish pending sources. [12:48] 2011-04-05 11:11:00 DEBUG Added /srv/launchpad.net/ubuntu-archive/ubuntu/pool/main/h/hello/hello_2.6-1lp5.dsc from library [12:48] It really finished in only 27 minutes? [12:49] Dirty maverick-release should have put you through at least 3-4 hours of pain... [12:49] wallyworld_, yeah, I have a server I share with a friend and he upgraded over night. [12:49] wgrant: I ripped out all the explicit gc and the cache invalidations. [12:49] jtv: Ah, you can close one of my bugs, then. [12:49] No longer necessary with the generational cache. [12:49] jtv: But that wasn't the bit that made it take hours. [12:50] What was? [12:50] (I had a branch that did the same thing, but it didn't definitely improve performance) [12:50] Queries when generating override and file lists. [12:51] Huh. [12:51] Only took 7 minutes to generate binary file lists. [12:51] WTF? [12:51] This is reason to be extremely suspicious about its results. [12:51] Mind you, the various calls to publish-distro etc. are all in one process now, so we may be getting better cache re-use. [12:52] (When combined with not continually invalidating or flushing the cache, of course) [12:52] But by all means, please be suspicious about the results! That's what I really need here. [12:52] ... oh. [12:52] You deleted publishingtunableloop. [12:52] *That* is the important bit. [12:52] Yes [12:53] Because it did explicit gc and cache invalidation. [12:53] As long as we're not running into RAM usage issues (due to materialising a huge resultset in one hit), that is fine and should be a massive benefit. [12:53] And essentially nothing else. [12:54] The issue was that it did binary file lists in 16ish batches of 10000 each, each of which took 8-9 minutes for the DB to calculate due to the compound sort key. [12:54] Yeah… I figured (and documented in the MP) that if there are still problems of this nature, we're probably better off taking a fresh, post-StupidCache, post-bash look at them. [12:54] Actually, I think I did test this once before. [12:54] By hacking tunableloop to always use a batch size of 200000. [12:55] It pains me to admit it, but the loop tuner was never the right tool for this particular job. [12:55] But it wasn't quite so wildly successful, possibly because of the remaining cache crap. [12:56] Bear in mind by the way that we no longer have the infinite-caching problem. [12:56] RIght. [12:56] So if we still have memory problems in this script, it's going to show up as excessive database waits rather than as swap or OOM. [12:57] But I'd be very much interested in your appraisal of the data that this run produced. [13:00] launchpad@mawson:/srv/launchpad.net/ubuntu-archive/ubuntu$ ls -l dists/maverick/main/source/Sources [13:00] -rw-r--r-- 1 launchpad launchpad 3763543 Apr 5 11:31 dists/maverick/main/source/Sources [13:00] That shouldn't exist any more, should it? [13:01] wgrant: that's done by one of the run-parts scripts, which I'm not running in this case. [13:01] Basically Ubuntu-specific. [13:01] jtv: Ah! Well, apart from that it looks pretty reasonable. I will throw more at it tomorrow, when I am meant to be working. [13:01] Well done on destroying PublishingTunableLoop. [13:01] Probably won't help productive by more than a couple of minutes, but helps DF massively... [13:02] (But actually glad you're spotting these things, because it makes it much more likely that you'd have spotted any noticeable oversights) [13:02] Thanks very much for looking into this. [13:02] s/productive/production/, grar. [13:02] Next time let's run domination in parallel. :-) [13:03] Hmm? [13:03] You can run judgeSuperseded in parallel with index generation if you really want to, but that's about it... [13:03] Just guessing: maybe domination is embarrassingly parallelizable across architectures. [13:03] Indeed. [13:04] But it should also be blindingly fast now. [13:04] And it takes a fair bit of the time, actually. [13:04] Maybe on mawson. [13:04] On prod each pocket should be no more than 1.5s or so. [13:04] On this run, I see about 4 minutes going into it. That's a sizable percentage of overall runtime. [13:05] Not that parallel runs would necessarily help mawson. :) [13:05] I think parallel runs on mawson might make the chassis itself glow red hot. [13:05] jtv: You can see it speed up significantly as the caches get hotter. [13:05] Kewl! [13:06] StevenK: rofl [13:06] amd64 takes 59s, armel 15s, i386 13s, and it continues. [13:06] yeah nice [13:06] Source still takes 90s, but it's more like 2s on prod (was 150s until I fixed it six months ago) [13:07] wildcherry has fewer issues with caches, given that it doesn't have 4GiB and all of the LP services. [13:08] rvba: Just to let you know, I am working on your review. [13:08] wgrant: by the way you'll notice complaints about stuff not being signed. You probably saw this coming, but that's because gpg signing also is in a run-parts script now. [13:08] henninge: great. [13:08] rvba: Here is as far as I have come and there are a few things that need attention. [13:08] jtv: Yeah, I'll walk through everything, get more stuff publishable, populate partner, add fake keys, enable the rest of run-parts, etc. tomorrow and see what breaks. [13:08] rvba: if you want you can start on those. http://paste.ubuntu.com/589661/ [13:09] Then I may change my qa-hellno to qa-ok. [13:10] wgrant: I'm so glad I don't know how to do all of that. I backed up before my run using cp -r /srv/launchpad.net/ubuntu-archive/ubuntu/dists /srv/launchpad.net/dists.bak [13:10] wgrant: You mean it isn't qa-overmydeadbody ? [13:10] jtv, wgrant: I'm not entirely convinced about those complaints being to do with lack of signing [13:10] wgrant: do you want me to put that old dists back in place? [13:10] since it's never been an issue on DF before [13:10] Invalid archive signature? It could be reading the wrong file or something I guess === al-maisan is now known as almaisan-away [13:12] bigjools: you're right… files seem to be missing [13:12] jtv: which equates to a further path issue [13:12] Yes, and it's not the partner path [13:13] well, in fact the original path issue :) [13:14] Hmm maybe not… In the example I'm looking at, the diff, dsc, and orig tarball are all there but the deb isn't. [13:14] Where should that deb have come from? (If it's the librarian, then I can think of a reason) [13:21] bigjools: I didn't see those errors. I'm trying not to look at it in depth yet. [13:22] bigjools: But if it's "Invalid archive signature" on a deb, it could be from the failed librarian transfers when I ran the careful publisher. [13:22] I forget the full list, but fennec was one of them. [13:22] On most archs its debs failed to transfer properly from production. [13:22] Is that one of the ones with errors? [13:24] wgrant: fennect looks to be behind a sizable portion of the errors, yes [13:24] As for partner, there are no fresh uploads and probably nothing existing on disk to index. I will sync and upload stuff tomorrow to test it out. [13:24] jtv: Aha, great. [13:24] So it's probably nothing to worry about, but I'll check it out. [13:24] About half the pages of errors are completely full of fennec-related lines [13:25] Then there's python-htmltmpl [13:25] and a bit of microcode.ctl [13:25] I don't recall those, but possibly because they sound arch-indep. [13:25] So there would be around 1/5 of the errors. [13:27] I think python-htmltmpl is again repeated lots of times (per arch I guess) and that seems to be most of the rest. [13:27] I may manually regrab the files tomorrow and see what happens. [13:27] As well as -C'ing partner. [13:27] Hopefully that won't take too long. [13:30] One thing that bothers me a lot is the carrying-over of old directories. For all I know we might not notice if, for instance, we stop generating a file and keep copying an old copy instead. [13:31] jtv: Yeah, but we have no choice :/ dists isn't all generated by the publisher. [13:31] Custom uploads end up in there too. [13:31] Custom uploads are bad in a few ways. [13:31] And good in not many :( [13:32] Well, here's hoping that we're making some real progress in manageability and transparency of the process. [13:33] Oh, we have another buildbot failure in progress. :( [13:35] What has combusted now? [13:36] Dunno summary won't load [13:36] That's why I asked :/ [13:36] It's just hanging for you too? [13:37] I don't see any failures on the waterfall [13:37] Oh drat I'm mis-reading it. [13:37] I see nothing in stdio. [13:37] jtv: Ah, good. [13:38] Sorry for the noise. [13:39] Project windmill build #139: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/windmill/139/ [13:39] Not helping, Jenkins. [13:50] wgrant: my fix for the partner archive path is in db-devel now, so if you update df's codebase tomorrow it should be in there. [13:50] jtv: Great, thanks. [13:50] jml, can you send me a ping at my work email, see if I got mail sorted out now? [13:50] deryck: sure. [13:50] I may merge db-stable into devel again tomorrow, since there are no DB changes. [13:50] I'll clean up my pending merge first. [13:51] jtv: Which branch is that? [13:51] jtv: It's fairly critical to sensible running the script. [13:51] The gc/cache cleanup [13:51] i mean where is it on LP, so I can merge it again tomorrow. [13:51] Alternatively, I could just commit it. [13:51] hurray, mail! Thanks, jml! [13:51] wgrant: Oh, I was about to give you the pointer. :) [13:52] wgrant: lp:~jtv/launchpad/db-bug-244328 [13:52] Feel free to attach your own kill-kill-kill bug. === jcsackett_ is now known as jcsackett [14:20] henninge: just pushed the corrections from the first batch of comments you made (Thanks already for the careful review :)). [14:20] wgrant: hopefully jtv answered all your questions, I was at lunch [14:20] bigjools: well… "is there a God" was a bit of a sticky point [14:22] rvba: I'll have a look. I just sent off the complete review. [14:22] henninge: great, I'll look into it. [14:22] bigjools: I think so. [14:22] jtv: contentious but ultimately easy to answer :) [14:23] bigjools: we've been over this. You don't count. [14:25] Project windmill build #140: STILL FAILING in 46 min: https://lpci.wedontsleep.org/job/windmill/140/ [14:29] henninge: a small question about your comment of my use of @@. I though this was the only way to access an attribute on the default _view_ attached to this object. But since it's not used in the code base, I'd like to know how else this can be done ... [14:35] rvba: oh, this is an attribute of the view? [14:35] henninge: yes [14:35] ah yes ;) [14:35] rvba: view/attribute [14:36] henninge: but the view available in the page is attached to another object [14:36] henninge: let me explain: [14:36] hm [14:36] I need to access an attribute of the view attached to an object inside a loop [14:37] the current view is attached to the context [14:37] I need to access another view [14:37] rvba: I understand [14:37] rvba: now the notation makes sense, too. [14:40] rvba: now I wonder, too, why it is not used anywhere else [14:40] * henninge greps again [14:40] indeed ... [14:41] if you create a small template fragment that is used inside the loop then it's not necessary to use this @@ because you can make the association (view/object) for this fragment only [14:42] but if all you need is to get an attribute from a view, adding a 3 lines template fragment looks overkill. [14:44] rvba: I think the usual approach would be to push this into the model [14:45] rvba: In this case, DistroSeriesDifference should have parent_source_package and source_package attributes. [14:46] rvba: and you'd use "difference/parent_source_package/fmt:url" in the template. [14:46] henninge: I confess I don't really like putting things like this into the model [14:47] henninge: but if it's the way it's done in the rest of the code I'll do it [14:47] rvba: that's my interpretation, yes [14:47] henninge: what does this /fmt:url do exactly? [14:48] rvba: it gets the canonical_url for the object, the same thing the function does. [14:48] henninge: ok, I get it. [14:48] so you wouldn't need that for tal:condition, but for tal:attributes [14:49] ok [14:49] rvba: to me it sounds better to not have the different view depend on each other too much. [14:49] s/view/views/ [14:50] henninge: good point [14:50] rvba: I'd say it makes the code more fragile. [14:50] we have a pretty close relationship between view and template in Launchpad. [14:51] rvba: Ha! [14:51] pub = IResultSet(pubs).one() [14:52] Database code does not belong in view code, that much I know. ;-) [14:52] rvba: so this is a strong indicator that these should be pushed into the model. [14:52] henninge: that's right. You win :) [14:52] :-D [14:52] Launchpad wins! === jtv is now known as jtv-afk [14:53] henninge: all right ... I'll go though the rest of your remarks and get back to you when it's done [14:54] henninge: you should explain why "Database code does not belong in view code" so that rvba learns something :) [14:54] bigjools: Hey, I am only going through the motions here ... :-P [14:55] I think it's pretty standard isn't it? Views are for fetching data from models and getting it ready for the templates. [14:55] the main reason is security [14:56] rvba: it's also because you should only handle security proxied objects in view code. [14:56] there we go :) [14:56] rvba: yeah, it's actually the main reason [14:56] ;-) [14:56] ok, makes sense [14:57] rvba: plain database queries will give you naked objects. [14:57] can't have that in public ... [14:57] right [14:58] henninge: thanks again, I'm learning a lot thanks to this review process I reckon :) [15:00] rvba: generally if view code needs to get a database object, it does a query via the context, or uses a secured utility [15:04] * jml mumbles something about sorting & views. [15:17] allenap: "field.summary": u"Even The Register likes it." - just made me lose a mouthful of coffee [15:17] bigjools: Hehe :) [15:23] allenap: you have a typo in lib/lp/registry/help/distribution-add-series.html [15:24] bigjools: Dang. [15:24] "its version is used instread" [15:24] bigjools: Nothing wrong with that ;) [15:24] That's how they say it round here. [15:24] if you're dyslexic [15:25] bigjools: I'll fix it now. [15:25] allenap: I could make more comments about round ther e:) [15:25] but they would apply to round here too [16:13] anyone happen to know if logging something as error in our cronscripts automatically creates an OOPs? [16:15] jcsackett: it is not automatic [16:15] jcsackett: an text can be logged as an error. We want an exception. [16:16] sinzui: right, i was double checking. b/c once i was able to get tests for the hardwardb stuff running, the call site I saw that corresponded to the OOPs error msg didn't cause an exception. :-( === almaisan-away is now known as al-maisan === salgado is now known as salgado-lunch === al-maisan is now known as almaisan-away [16:19] jcsackett: look for globalErrorUtility or a subclass. it will call handling() [16:19] sinzui: ah, okay. [16:23] deryck: I've got the form showing: http://people.canonical.com/~abentley/translations-usage.png Just want to confirm this is really what you want before I go implement saving. [16:24] * deryck looks [16:24] abentley, yeah, looks good to me. [16:25] deryck: alrighty then. [16:43] hey deryck, is this type of markup (found in several bugs pt files) still valid? i don't see that file being built anymore. [16:43] http://paste.ubuntu.com/589741/ [16:45] bac, we shouldn't be doing that anymore. [16:45] deryck: thanks for the confirmation [16:45] bac, np === matsubara is now known as matsubara-lunch [16:54] deryck: would you mind looking at the top of soyuz/templates/archive-copy-packages.pt? i assume it is wrong too but don't know why it was ever there. [16:55] ok, looking.... [16:56] bac, yeah, it's wrong too [16:56] not needed. [16:56] anything in lib/lp/APPNAME/javascript gets pulled in automatically. [16:56] so anything loading specific js files from icing/build is wrong [16:57] bac, yup. [16:57] bac, it's loading files twice in devmode but no harm to production. [16:58] deryck: well those files don't exist in devmode, so it is oopsing trying to load them [16:58] bac, ah, right. [16:58] which confuses people, e.g. me [16:59] gotcha === deryck is now known as deryck[lunch] [17:18] gmb: can you have a look at https://code.launchpad.net/~bac/launchpad/remove-old-devmode-js/+merge/56400 [17:19] * bac lunches === jcsackett_ is now known as jcsackett [17:37] bac: Sure [17:43] Project windmill build #141: STILL FAILING in 1 hr 15 min: https://lpci.wedontsleep.org/job/windmill/141/ [17:43] bac: r=me === Ursinha is now known as Ursinha-lunch [17:54] rvba: still around? [17:54] henninge: yes [17:55] why did you have to use structured().escapedtext? [17:56] I mean, I thought just structured() would have been enough? [17:56] I did that as first but then I get the __repr__ version of the object in my page ... and it's not pretty [17:58] s/as first/at first/ [17:58] Night!# === salgado-lunch is now known as salgado [18:02] flacoste: you have disappeared from our secret squirrel IRC server! [18:02] jml: really!?! [18:02] flacoste: yep. flacoste has quit (Connection reset by peer) [18:03] night all [18:09] rvba: really? that's interesting. [18:10] rvba: you mean this gives you a _repr_ of the structured object? [18:10] 871 +

[18:10] henninge: yes [18:11] I wonder if wgrant has already been changing things ... [18:11] I saw quite a few .escapedtext in the code ... [18:11] * henninge checks places he knows === deryck[lunch] is now known as deryck [18:13] hm, those that I had don't go straight to TAL. [18:14] rvba: see, now I learned something, too ... ;-) [18:14] henninge: :) [18:16] henninge: the proper way might be to do this: [18:16] [18:16] not much difference though [18:17] but the code is full of these [18:17] henninge: I'll have to get going is a few [18:18] s/is/in/ [18:19] rvba: I am not done yet, I am sorry. We will have to do another short round tomorrow morning. [18:19] I'll finish my review of your changes tonight, so you can have them in the morning. [18:19] henninge: sounds perfect [18:20] henninge: see you tomorrow [18:20] rvba: see you! ;) === matsubara-lunch is now known as matsubara === Ursinha-lunch is now known as Ursinha === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [19:05] hi deryck, sinzui: we've got a few places in our JS that uses Y.fail. they seem to be untested and calls to it actually fail. do either of you have any experience with it? [19:07] bac: it's a test module method, no? We shouldn't be using that in regular code. [19:07] deryck: that is my impression too. but i see it in many places [19:07] deryck: should we just be throwing an error? [19:08] bac: yeah, I think we either want to use Y.error directly or the showError mechanism we have. [19:10] deryck: it is in five modules, including lp.js and client.js [19:10] hmmmm, ok [19:10] deryck: how would I get the URL for a translation_group from its name in Javascript? [19:11] deryck: I figure translation_groups.getByName, but I don't know how to get translation_groups. [19:12] deryck: and i just discovered it is part of 'test', so you have to include 'test' in production code, which is vile. [19:13] bac: yeah, agreed. so the client module looks like it's trying to log the fail. I think Y.error or even a simple Y.log('some msg', 'error') is enough there. [19:13] and it points to none of those paths ever being exercised... [19:13] bac: and I think that ^^ could be the basic pattern. I'm certain we don't want Y.fail in js code. [19:13] right === Ursinha is now known as Ursinha-afk [19:14] ok === Ursinha-afk is now known as Ursinha [19:14] abentley: are they exported in the api? Can you get them from an api get? [19:14] deryck: yes, they're exported in the API. I don't know how to get them from an API get. [19:18] I'm not sure either. I'm thinking and looking at code. [19:20] deryck: it seems like there should be a generic answer to "how do I get the URI for a collection", and then the rest would fall into place. [19:20] yeah [19:24] abentley: yeah, so it's the same as an entry in the client, right. Y.lp.client.Collection. but like you say you need the URI. I really don't know how you get that. trying to find an example somewhere. [19:26] abentley: where are you getting the name from? [19:27] deryck: I'm getting it from the form data. [19:28] deryck: from the form I posted a screenshot of earlier. [19:28] right [19:28] deryck: Unless I'm doing it wrong? (I'd love to be doing it wrong!) [19:29] abentley: are you doing an XHR post of the form on submit? Or trying to handle each piece with the API? [19:30] deryck: the latter. [19:30] abentley: can you not make it easier by doing a single form post with the data? [19:31] deryck: It's not something I'd thought of 'till now. [19:31] deryck: how would that work? [19:32] abentley: do an XHR post to the page that normally handles this step. on success flash green and go to next part of page. report if it errors. [19:34] abentley: we may not do this anywhere else. I can't find an example. But you would just use Y.io directly. [19:34] deryck: could you give me more detail or an example of the first piece? [19:34] deryck: That is, making the XHR post. [19:36] abentley: typing up something.... [19:42] abentley: something like this: http://pastebin.ubuntu.com/589828/ [19:43] henninge: still around? [19:43] rvba: yes and no ;) [19:44] henninge: I see ... just a quick question then if you don't mind [19:44] rvba: go ahead [19:45] henninge: in your review you advise me to move back getParentPackageSetsNames into the views ... but my understanding of our previsou co [19:45] sorry ... [19:45] previous conversation was that you wanted these methods in the model to avoid the clumsy @@ inside the template [19:46] rvba: sorry, there must have been some misunderstanding, then. [19:46] henninge: I understood that the main point was security. [19:47] henninge: but still, I'll have to use @@ to be able to call a method from a view (but not the "main" view) associated with an object in the template. [19:47] rvba: yes, that's why (parent_)source_package_release belong in the model. [19:48] deryck: thanks. I think I can hack that. [19:48] rvba: no, you don't ;) [19:48] abentley: cool. hope it helps. [19:48] henninge: then I think I missed something [19:49] rvba: which view is the one that the template belongs to? [19:50] rvba: ah here, DistroSeriesMissingPackages [19:50] henninge: yes [19:50] that's the view that needs the formatting code. [19:50] henninge: but I'll need to call a view that is acting on the difference, not the distroseries [19:51] that's the part I don't understand properly I guess ... how to call a method, not on the main view of the template ... but on the view associated with the many differences displayed on the page [19:51] rvba: aha, difference is from a tal:repeat [19:51] henninge: right [19:52] henninge: there is my problem :-) [19:52] henninge: hence the @@ [19:52] rvba: yes, you have too options to avoid the @@ [19:53] rvba: option 1: construct the list in TAL [19:53] henninge: yes, I can use ','.join directly inside the TAL [19:54] * rvba greps [19:54] nah, avoid python in TAL at all costs. [19:54] right [19:54] rvba: you could do a tal repeat over difference.source_package_release [19:55] and use tal:condition on the "last" variable (I forget the details of tal:repat here) to add a comma or not. [19:56] rvba: option 2: construct it in the view [19:56] henninge: got it ... quite ugly though. [19:56] yes, I agree [19:56] (I'm talking about the repeat) [19:56] I know [19:57] rvba: but then you will not be able to loop over differences/batch but will have to create your own list. [19:57] which works ok [19:58] still looks a little bit of a hack to me though :-) [19:58] but I'm glad to use any method that works [19:59] rvba: currently differences is the content of view/cached_differences. [19:59] ye [19:59] s [20:00] rvba: you can instead create a propety that iterates over cached_differences internally and yields dicts of information [20:00] I think I get the idea: create a simple list wrapper with the differences and the formatted packagesets [20:00] right [20:01] yup [20:01] * henninge goes back to do other stuff [20:01] henninge: thanks for your time, it's late for the two of us ... I'll sleep over this and see if the morning brings a new idea. [20:02] rvba: great idea [20:02] ;-) [20:02] * henninge cannot sleep yet, though [20:02] * rvba meither [20:02] s/meither/neither/ [20:14] morning [20:14] deryck: hi [20:15] hi lifeless [20:16] deryck: bug heat decay [20:16] ah yes [20:16] we talked about that at the thunderdome, no? [20:16] deryck: stub has identified *something* causing rapid index bloat in bug/bugtask [20:16] deryck: we talked about the rendering stuff - threshold and the curve fitting function [20:17] ok, gotcha. new issues now. the joys of heat! :-) [20:17] deryck: stub is still gathering data, but it may be the bug heat decay (because every row in bug (and now bugtask)) is being rewritten at high frequency [20:18] I'm being opportunistic, since you're still here ;) and asking about how important the decay aspect is [20:18] like, if we removed the 'time since last alteration' component from the heat function, do we think folk would care? [20:19] lifeless: it's very important actually. the decay helps bugs not sit in the top 10 hot bugs too long. bugs can gain heat quickly and they need to decay quickly or it gets odd looking on that bugs home page. [20:19] lifeless: the decay happens via the offline job or cron, though. So it could be run less frequently. [20:19] deryck: did we try it without the decay at any point ? [20:20] deryck: I ask because hot bugs are often lightning rods anyway [20:20] deryck: which means they get meaningless comments that have the effect of nullifying the decay [20:20] lifeless: we tried various forms of heat. we never tried without the decay alone. [20:21] lifeless: decay helps mitigate the lightening rod. comments don't add heat, either. [20:22] lifeless: so I think the decay bit runs in garbo. perhaps we could go daily instead of hourly? I don't recall how frequently it runs actually. === salgado is now known as salgado-brb [20:24] deryck: the frequency the task runs isn't the issue [20:24] deryck: its the rewriting of the entire table over the course of a week [20:24] deryck: *if* this is the problem, its baked into the use of a decay function at all. [20:24] deryck: it may not be [20:24] lifeless: ah, ok. [20:24] deryck: like I say, I'm gathering data. [20:25] lifeless: and, to be clear, there isn't a decay funtion per se. There is one calculation function. part of that function handles decay [20:26] deryck: yep, I'm on that [20:26] ok, cool [20:26] update_bug_date_last_updated updates bug_last_updated, which is the key thing for the decay, on every IObjectModified event [20:27] which includes [20:27] for="lp.bugs.interfaces.bugmessage.IBugMessage lazr.lifecycle.interfaces.IObjectCreatedEvent" [20:27] handler="lp.bugs.subscribers.buglastupdated.update_bug_date_last_updated"/> [20:27] deryck: so commenting on a bug freshens its heat [20:27] deryck: that might not have been the intent, but it is the implementation :) [20:28] lifeless: I don't think that's entirely correct. update_bug_date_last_updated has nothing to do with heat, IIRC. [20:28] in trusted.sql [20:29] days_since_last_update = (datetime.utcnow() - date_last_updated).days [20:29] lifeless: sorry to seem dumb, I'm in the middle of natty upgrade and get really look [20:29] total_heat = int(total_heat * (0.99 ** days_since_last_update)) [20:30] lifeless: right. that heat function *uses* date_last_updating but the event that causes date_last_updated to be updated doesn't cause the heat function to run. [20:30] deryck: right [20:31] deryck: but the reason we have to recalculate the heat on every bug [20:31] deryck: is because date_last_updated is part of the function [20:31] [and date_created] === Ursinha is now known as Ursinha-afk [20:32] deryck: I was analysing the 'commenting on a bug does not affect its heat' - but it does [20:32] deryck: because it resets date_last_updated [20:33] deryck: anyhow, will leave you to your upgrade :) stub is gathering concrete data to pin down the cause of bloat [20:33] lifeless: well indirectly. commenting affects date_last_updated. And that factors into heat. But commenting does not trigger a recalculation of heat. Changing those dates does not trigger a recalculation of heat. [20:33] lifeless: ok, fair enough :-) Good luck with it. [20:34] deryck: thanks [20:34] * sinzui just discovered multitouch is now enabled on his macbook [20:35] sinzui: really? how well is it working? [20:36] three fingers: resize/move via love handles. four fingers: opens the dash [20:38] contacts synced for the first time in 4 months this morning too. The test will be my next update though. My primary computer has only synced twice in 18 months [20:39] deryck: I see the confusion I was causing; I was saying that commenting on a bug stops the 'if days_since_last_update > 0:' condition firing [20:39] ah ha [20:40] deryck: any bug commented on less than 1 day before the background task to update heat, will have an unscaled heat [20:40] lifeless: I see what you mean now [20:40] deryck: and also that bugs which are in the top 10 will likely get a continual stream of comments - they are lightning rods [20:41] right [20:41] deryck: all they need is one comment every week or so to stop them dropping out [20:41] right === almaisan-away is now known as al-maisan [20:41] sinzui: nifty. [20:41] deryck: which is why I'm wondering if there is any /practical/ impact of dropping the date based scaling [20:42] oh, hell [20:42] * sinzui gets children [20:42] deryck: IFF stub finds that it is the heat routine causing issues [20:42] lifeless: right. given that, maybe there's no harm. [20:42] lifeless: we could try and see. [20:42] yeah [20:42] there is no rush, not till we know the bloat cause [20:43] thanks for the time! [20:44] np! Thanks for bringing it up. === salgado-brb is now known as salgado === maxb_ is now known as maxb === Ursinha-afk is now known as Ursinha [21:05] Later on, everyone. [21:17] wgrant: would you like the honours ? === al-maisan is now known as almaisan-away [21:52] Project windmill build #142: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/windmill/142/ [22:10] * mwhudson_ raised an eyebrow at https://launchpad.net/c++ [22:11] -lol- [22:11] terrible project name === mwhudson_ is now known as mwhudson [22:12] yeah [22:12] jml: hooray for the upgrade to twisted 11 being so easy! [22:12] 933, in _get_nodes [22:12] found[idx] = cache[idx] [22:12] RuntimeError: maximum recursion depth exceeded === Ursinha is now known as Ursinha-bbl === s1aden is now known as sladen [22:36] jcsackett: [22:36] ping [22:37] * sinzui is on stupid medication today === salgado is now known as salgado-afk [22:50] sinzui: pong. [22:50] sorry, didn't hear the notification. [22:51] jcsackett: mumble? [22:59] * wallyworld sad. coffee machine broken :-( === matsubara is now known as matsubara-afk === a1 is now known as al [23:37] gmb: ping [23:38] gmb: lp:~gmb/launchpad/bug-1-timeout - this will cause timeouts on bug message collections via the API I think [23:42] hi lifeless [23:46] gmb: s/timeouts/bad-data/ - it needs rolling back, I've explained in the bug [23:46] hi poolie [23:46] lifeless: Which honours? [23:46] wgrant: deleting shipit [23:47] Woot. [23:47] I wasn't sufficiently up to date on email. [23:47] But yay. [23:47] Is production sufficiently gone? [23:48] see chat with chex in -ops an hour back [23:49] So that looks like a yes. [23:49] And if not, then the unused shipit appservers will break, oh no. [23:49] This is very good news. [23:50] We can finish purging Account. [23:50] And then delete AccountPassword and eventually Account itself! [23:50] And we will be sensible again! [23:51] VPC 4 eva [23:51] Yes. [23:52] lifeless: Have you rolled back gmb's thing? [23:52] not yet [23:52] I have to pop out for a bit [23:52] if you want to do it yourself, that would be cool [23:52] I can't see any good reason for him to be collecting /all/ bug messages for bug 1 in his code [23:52] <_mup_> Bug #1: Microsoft has a majority market share so I suspect the change is actually entirely different [23:53] Project windmill build #143: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/windmill/143/ [23:53] change needed is entirely different, I mean [23:53] OK, I'll check it and roll it back. [23:53] thanks [23:54] have we seen an OOPS for this in the oops reports? [23:55] -> bbiab [23:55] mwhudson: hah! [23:56] jml: ? [23:56] mwhudson: it took way longer than you might think, because of two separate spurious test failures during ec2 and a testfix mode when I went to land today [23:56] mwhudson: but the patch was small :) [23:57] jml: well, ok, so "modulo usual launchpad development craptitude"? [23:57] mwhudson: yeah, modulo that, it was a piece of cake [23:57] mwhudson: I wish every Python library cared as much about backwards compatibility. [23:58] anyway, off to bed [23:58] g'night.