[00:02] thumper, rockstar, stub, mwhudson, stevek: reviewers meeting [00:05] something in the bugs page seems to make firefox go insane :( [00:05] js-wise === Ursinha is now known as Ursinha-afk [01:00] jml: another recipe bug for you - https://bugs.launchpad.net/launchpad-code/+bug/687623 [01:00] <_mup_> Bug #687623: No ability to manually trigger a build 'like the daily build system' [01:08] thumper: spiv: would appreciate your thoughts here: we have ~ 10 requests against BRANCH:%2Bbranch/tsumego20k/1.0 but https://code.launchpad.net/~branch says no code? [01:09] spm: %2B is +, not ~ [01:09] Ah [01:09] wrong group. that's a start. [01:10] https://code.launchpad.net/tsumego20k looks more like it [01:11] But there's no 1.0 series. [01:11] There's a 'stable' series (with a 1.0 release). [01:12] hrm. and their branch is only 472K as well. [01:12] So, requests for that branch are presumably wrong, and ought to give "branch not found". [01:12] lp:tsumego20k as in [01:12] they probably mean /stable rather than /1.0? [01:13] well. it gets worse. that, and another branch are eating memory. just got a swap alert. I'm going to have to big stick shortly. [01:13] Oh and for added fun the stable branch appears to have a broken stacked-on target. [01:14] would that be causing the pain that codehost is experiencing? [01:14] I wouldn't *think* so [01:14] ah. they have another: BRANCH:~goadict/tsumego20k/release-1.0 [01:15] But it could be... it's hard to be sure. [01:16] mwhudson: so gist is we have one, maybe 2 or 3 branches that are spinning doing "something"; and eating memory and CPU [01:16] I don't know of any open bugs that would cause this sort of spinning now that codehosting is running bzr 2.2 [01:16] ah ok [01:16] Er, 2.2.2 [01:17] spm: have you whacked them with that gdb thing? [01:17] no, wasn't game to just yet. [01:17] haven't done any whacking - just watching/gathering. [01:17] k [01:17] https://pastebin.canonical.com/40699/ [01:18] https://pastebin.canonical.com/40700/ for a process listing [01:19] the db-devel ones, while alarming to have a few, don't seem to be problematic. [01:19] Huh. [01:20] Some say "+branch", some say "%2Bbranch". I wonder why. [01:20] Probably a red herring for this issue. [01:20] the cpu/load spike seems to be coming down tho. so I'm strongly tempted to just let things resolve on their own. [01:20] well... load. not cpu. that's still flat out. [01:23] that's a lot of processes [01:23] i wonder which user 2978124 is [01:23] probably launchpad-pqm [01:23] how do we find out? select * from person? [01:24] y [01:24] too obvious that. if even a sysadmin can guess correctly. /me files a big for greater obscurity. [01:24] bug, too. [01:25] launchpad-buildbot [01:25] ah ok, that makes some kind of sense too [01:29] so something baout those branches tho is driving codehost to 100% cpu for ages, which is bad. :-( [01:29] worth a pygdb on one? [01:30] haha. on codehost itself, and it's easier to copy the branch from my local pc up. [01:32] hee hee [01:32] i guess you could run bzr get /srv/.../00/0x/xx/xx if you coudl be bothered [01:32] yeah, i reckon pygdb one [01:32] the user on the other end must be bored already by now [01:35] or afk [01:39] oh that's a bad sign... the backtrace.py is not showing a whole lotta activity... [01:39] * spm lets it be for a bit longer [01:40] :/ [02:06] mwhudson: I gave up. it never progressed beyond 'Thread 1' as output. trying a gcore. [02:06] spm: :( [02:06] spm: maybe just attach with gdb and run 'bt' [02:07] the process will only be single threaded [02:07] sure [02:07] mwhudson: not necessarily (but probably) [02:08] err. pebkac? $ gdb 23558 ; (gdb) bt ==> No stack. [02:08] mwhudson: the insert_stream verb in the server spawns a thread as that was easiest... [02:08] spm: gdb -p 23558 ? [02:08] bah [02:08] that's better. ta [02:09] spiv: whee [02:09] oh wow. it looks almost like recursion hell [02:09] pls to save me from rtfm - the command to disable gdb paging? [02:10] set pagination off [02:10] i guess that might explain why pygdb didn't work so well [02:11] though it prints the stack as it goes [02:11] holy crap. I'm copping 50k/s of gdb output. and it's not stopping..... [02:11] spm: be careful what you wish for! [02:12] reckon [02:12] glad I don't have "infinite" buffers any more, or this'd suck [02:12] well this sounds like fin [02:12] fun [02:13] https://pastebin.canonical.com/40701/ for 3 pages of so far [02:14] whisky tango foxtrot!! [02:15] https://pastebin.canonical.com/40702/ next 4 pages (i think...) [02:15] i call busticated. [02:15] Huh, this is an lp-serve process? [02:15] codehost? [02:16] yes [02:16] spiv: yes [02:16] Ah, I see, it veers off into Twisted via lp.codehosting.vfs.transport.SynchronousAdapter [02:17] So this is a bit disheartening :( [02:17] spiv: so it's your fault? [02:17] * spm establishes random pin-the-blame-game early [02:17] spm: well, I thought bzr 2.2.2 fixed this bug. :/ [02:18] :-( [02:18] Or rather, fixed one of the two issues that combined to cause it. [02:18] oh, it's stringifying a traceback inside twisted's failure code [02:18] ah luverly, so we've maybe just discovered #3? [02:18] It appears to be a error-handler-falls-over-and-recurses-into-the-error-handler death sprial. [02:18] mwhudson: right [02:18] spiv: so stabby time, or more info would be useful? [02:19] * spiv digs up the issue he thought he fixed. [02:19] spm: hmm. The traceback is probably enough. Can you give me a minute to ponder? [02:19] sure. it's not wonderful, but getting this fixed is worth the pain [02:20] spm: can you paste some more traceback actually? [02:20] easily [02:20] well.. easily but boringly. just sayin' [02:20] just a couple more pages [02:21] Argh, this really looks like the bug I fixed! [02:21] i'm not sure if it's infinite recursion or just stupidly deep recursion [02:21] repr of NotBranchError [02:21] https://pastebin.canonical.com/40703/ [02:22] mwhudson: I don't believe there's a difference outside academia there :-) [02:22] * spiv hmms [02:23] Oh, *blah* [02:23] * spiv thinks out loud [02:23] NotBranchError guards against open_repository failing with unexpected exceptions [02:23] you think "*blah*"?? really!?!?! [02:23] spiv: let me guess, it's partly twisted's fault? [02:23] mwhudson: (yes) [02:23] But open_repository is calling off into the lp VFS stuff [02:23] Which calls into Twisted stuff [02:24] anything traceback that has __getstate__ in it instantly makes me want to cry [02:24] and punch glyph in the face in about 2001 [02:24] * spm hands mwhudson a tissue box, and starts playing a small violin [02:24] Which thanks to the crumminess of Twisted's Failure class stringifies errors and tracebacks [02:25] Which happens inside the lp codehosting VFS stuff, so my NotBranchError guard doesn't matter: Twisted tries to stringify a NotBranchError (whose bzrdir fails badly on open_repository) [02:26] So off we go into the recursion. [02:26] === Top 10 Time Out Counts by Page ID === [02:26] Hard / Soft Page ID [02:26] 146 / 5425 Archive:+index [02:26] 62 / 400 Distribution:+bugs [02:26] 61 / 233 BugTask:+index [02:26] 59 / 2 Person:EntryResource:retractTeamMembership [02:26] 38 / 419 Distribution:+bugtarget-portlet-bugfilters-stats [02:26] 20 / 28 DistroArchSeries:+index [02:26] 11 / 16 DistroSeries:+queue [02:26] 11 / 2 Cve:+index [02:26] 10 / 95 ProjectGroupSet:CollectionResource:#project_groups [02:26] 9 / 6 ProjectGroup:+milestones [02:26] Roughly: repr(NotBranchError) -> open_repository -> codehosting VFS -> Twisted gunk -> repr(NotBranchError) [02:26] The fix for the top one should be on qastaging in a few minutes, hopefully. [02:27] mwhudson: does that sound plausible? [02:27] spiv: yes [02:27] mwhudson: ok, now what can we do about it? :( [02:27] spiv: make Failure throw away the traceback entirely? [02:27] mwhudson: I *think* perhaps this isn't something bzrlib can sensibly fix. [02:28] perhaps with a monkey patch? [02:28] spiv: how can we reproduce this locally? [02:28] mwhudson: IIRC the condition is something like a lookup on +branch/something/something causes a strange VFS error [02:29] spiv: i don't quite understand why NotBranchError is getting raised in the smart server [02:29] oh no, maybe that makes sense [02:29] mwhudson: Stacking, maybe? [02:29] Because the codehosting VFS is returning that from a transport operation, or something odd like that? [02:29] thumper may remember the details [02:29] wgrant: stacking is mostly a client side thing though [02:29] I'll see if I can dig out the relevant conversation from my logs [02:30] spiv: is NotBranchError raised by BzrDir.open or something maybe? [02:30] It's definitely an issue relating to how the server-side vfs translates stuff, maybe in weird cases like a dev focus stacked on itself, or something. [02:31] IIRC it's at the point of trying to open the bzrdir that in LP would have the auto-stacking config [02:31] IIRC +branch might be involved? Or something? [02:31] mwhudson: yeah, probably [02:31] +branch shouldn't be involved unless someone has been editing .bzr/branch/branch.conf by hand [02:32] i'm about to break for lunch - need anything more before i run? [02:32] noone would be silly enough to do that :O [02:32] spm: I think it's probably ok to kill before you run, if you want [02:33] spiv: okidoki [02:33] spm: yeah, as spiv says [02:33] mwhudson: see logs for this channel for Oct 11 [02:33] mwhudson: I think that's when I chatted with thumper about this initially [02:33] just reviewing now [02:34] ugh. looks like the owner is kicking the branch/whatever operation(s) off again [02:35] * spm has applied the sledgehammer :-( [02:38] * mwhudson pokes around with sftp [02:38] mwhudson: so in thumper's situation (which involved a patch he may have abandoned, so there may some red herrings here) [02:40] i can't see anything unusual :( [02:40] https://code.launchpad.net/~goadict/tsumego20k/trunk [02:40] mwhudson: he encountered this problem on an open_branch HPSS call to ~user/proj (trying 'bzr revno ~user/proj/no-such-branch', I think) [02:40] isn't a real branch [02:40] hang on, let me check what it _is_ [02:41] spiv: hm [02:41] mwhudson: default_stack_on = /~goadict/tsumego20k/trunk [02:41] there's a bzrdir there, but no branch [02:41] spiv: yeah, which is what one would expect [02:41] mwhudson: in sftp://bazaar.launchpad.net/~goadict/tsumego20k/.bzr/control.conf [02:41] spiv: right [02:42] That is a branch AFAICT over SFTP? [02:42] and not a stacked branch. [02:42] spiv: parse error [02:43] mwhudson: lftp bazaar.launchpad.net:~goadict/tsumego20k/trunk/.bzr> cat branch/format [02:43] Bazaar Branch Format 7 (needs bzr 1.6) [02:43] Why do you say that ~goadict/tsumego20k/trunk isn't a real branch? [02:43] spiv: by mistake, sorry [02:43] Ah. [02:44] spiv: https://code.launchpad.net/~goadict/tsumego20k/stable [02:44] spiv: ^ that one is just a bzrdir [02:44] lftp mwhudson@bazaar.launchpad.net:/~goadict/tsumego20k/stable/.bzr> ls [02:44] -rw-r--r-- 1 1001 1001 141 Dec 09 02:21 README [02:44] -rw-r--r-- 1 1001 1001 35 Dec 09 02:21 branch-format [02:44] drwxr-xr-x 2 1001 1001 4096 Dec 09 02:21 branch-lock [02:45] the gdb trace is talking about ~goadict/tsumego20k/release-1.0/.bzr/repository though [02:45] so maybe this stable branch is a red herring [02:46] I don't think it is. [02:47] My recollection is that accessing non-existent paths triggered this codehosting vfs issue. [02:48] But maybe I'm wrong :/ [02:48] yeah, maybe that's it [02:48] it's striking that all the problem processes are accessing paths that shouldn't work at all, like [02:48] BRANCH:%2Bbranch/tsumego20k/1.0 [02:48] codehost [02:49] the thing is of course that maybe the user is getting confused and flailing around creating and deleting things in the webapp [02:49] mwhudson: also, the gdb trace includes stuff like: [02:49] >> [02:49] spiv: that's not so surprising really [02:50] the synchronous launchpad transport wraps the asynchronous one [02:52] mwhudson: argh, the data is changing from under us :) [02:52] mwhudson: now ~goadict/tsumego20k/stable has a repo [02:56] A bit odd: _cache={'//~goadict/tsumego20k/release-1.0': ('BRANCH_TRANSPORT', {'writable': True, 'trailing_path': '.bzr/repository/format', 'id': 410572}, )} [02:59] writeable=True suggests that branch existed in the db at the time [03:07] wow, i was a bit confused there -- was looking at launchpad.net not launchpad.dev :) [03:08] of course launchpad.dev is impossible to use [03:09] because of 1 miiiiiiiiiiiiiillion js files [03:09] devmode off [03:09] problem solved. [03:09] GET /+icing/rev10399/yui/datatype/lang/datatype-date-format_it-IT.js [03:09] GET /+icing/rev10399/yui/datatype/lang/datatype-date-format_it.js [03:09] Yes. [03:09] COME ON! [03:09] And half of them OOPS. [03:09] So /var/tmp/lperr fills up. [03:09] mostly 404ing [03:10] where is devmode set? configs/development/launchpad.conf? [03:10] That. [03:11] $ du -sh /var/tmp/lperr/2010-12-09/ [03:11] 4.6M /var/tmp/lperr/2010-12-09/ [03:11] Yeah. [03:15] spiv: well i don't know what's going on :/ [03:17] mwhudson: hmm [03:19] spiv: i guess it might be interesting to see the bottom 100 lines or so of the traceback [03:19] mwhudson: yeah [03:20] mwhudson: also translateVirtualPath maybe should have a catch-all errback [03:21] spiv: sounds plausible, without looking at the code or thinking terribly hard [03:21] mwhudson: to avoid unexpected errors leaking out and confusing callers that expect only transport-related errors. [03:21] I don't see any obvious routes for those to happen [03:21] And I've just spent the last 30 min or so looking for one :) [03:23] bleh, they're still trying to get the code, or update or whatever. have sent the team members emails asking them to stop. [03:25] spm: it would be good to know exactly what they're trying to do [03:25] mwhudson: spiv: would a copy of the raw branch(s) off crowberry be of any use to you guys in debugging? seeing as they're able to kill it time and again. [03:25] mwhudson: sure - do we have a formal bug yet? I'll send that to them and ask them to pls update with as much info as possible. [03:25] spm: dunno, not sure [03:25] spm: more use to mwhudson probably [03:25] spm: how big are they? [03:25] My local lp dev environment isn't quite functional [03:26] spm: i haven't filed a bug [03:26] spiv: you're one up on me. don't have one at all. :-) [03:26] mwhudson: I was going to file "Weird shit happening with codehost" but felt it was lacking [03:26] spm: "Error handling infinite recursion death spiral of doom" maybe? [03:27] The "of doom" is a sure way to add some drama to a lacklustre title ;) [03:27] I like my subject better, nothing personal [03:27] "Weird shit of doom happening with codehost" then :P [03:27] but yours is more technically accurate. I'll go with that [03:27] hahaha [03:28] (A slow, painful doom...) [03:28] weird-shit-of-doom sounds like a good bug tag [03:28] +1 [03:29] mwhudson: the only one that works is tiny. http://bazaar.launchpad.net/~goadict/tsumego20k/trunk/files [03:29] spm: for added laughs the user has been actively changing stuff while we've been looking [03:30] Replacing an empty bzrdir with one with a repo etc. [03:30] well. trying to. I keep killing their processes off as it hurts [03:34] shame. codehost has dropped to #2 spot in firefox quick find for "filebug" [03:34] Does Soyuz win? [03:35] ha. no. most of my bugs are generically filed against launchpad these days. never quite sure where they go. [03:35] from mid jan 'launchpad' [03:35] one bug tracker. \o/ [03:36] We'll see how that goes. [03:37] wgrant: more successfully than lifeless' holiday, I hope... ;) [03:37] Indeed. [03:37] lifeless: I'd suggest you visit some remote island with minimal internet access [03:38] spiv 1, lifeless 0 [03:38] lifeless: but you already moved to NZ ;) [03:38] spiv 3, lifeless 0 [03:38] * lifeless is on wow atm [03:38] lifeless: that's more like it :) [03:38] so, I claim success. [03:38] spiv 3, lifeless -1 [03:38] oi! [03:39] Some people are so judgemental! [03:39] spm 1, spiv 0, lifeless -1 [03:39] s/ w+!/sysadmins!/ [03:39] wow [03:39] found the stone core [03:39] someone has to be. it's a burden that we take upon ourselves to spare the lesser folks aka developers [03:39] first quest in there... [03:40] 115K xp [03:43] https://bugs.edge.launchpad.net/launchpad-code/+bug/687653 [03:43] <_mup_> Bug #687653: codehost error handling infinite recursion death spiral of doom === Ursinha-afk is now known as Ursinha [04:18] lifeless: https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages is promising. 4s. [04:19] Ah, 2s with hot cache. [04:19] Not too bad. [04:20] OOPS-1804QS12 [05:17] wgrant: [05:17] SQL time: 16441 ms [05:17] Non-sql time: 4324 ms [05:17] Total time: 20765 ms [05:17] Statement Count: 50 [05:18] wgrant: also there are two repeats of a 1000ms query [05:20] Hmm. [05:20] A COUNT()? [05:20] 2203410171017SQL-launchpad-main-slave(SELECT SourcePackagePublishingHistory.archive, SourcePackagePublishingHistory.component, SourcePackagePublishingHistory.datecreated, SourcePackagePublishingHistory.datemadepending, SourcePackagePublishingHistory.datepublished, SourcePackagePublishingHistory.dateremoved, [05:20] ... [05:20] SourcePackagePublishingHistory.id, DistroArchSeries.architecturetag) EXCEPT (SELECT SourcePackagePublishingHistory.archive, [05:20] Oh, that one. [05:38] Project db-devel build (206): FAILURE in 13 min: https://hudson.wedontsleep.org/job/db-devel/206/ === almaisan-away is now known as al-maisan [08:59] morning all [08:59] Morning bigjools. [09:00] We has a regression. [09:00] Which I already had a one-line branch to fix... [09:00] Without knowing about it. [09:00] wgrant: do tell [09:00] bigjools: Bug #687662 [09:00] <_mup_> Bug #687662: Upload processor attempts to verify hashes against expired files [09:01] how did that happen? [09:01] Hmm? [09:01] oh, that change for getFile [09:01] Yeah. [09:01] do you think more pairs of eyes would have caught that? [09:02] Who knows. [09:03] wgrant: what affect elsewhere will your fix cause? [09:04] bigjools: It'll fix the +files/foo.orig.tar.gz 404. It could also impact other A.getFileByName callsites, but I doubt any rely on this behaviour, what with expired LFAs being useless and everything. [09:06] wgrant: right [09:06] wgrant: whack up an MP and let's get it landed [09:06] bigjools: MP is already there. [09:06] Oh. Branch is not linked. [09:07] yeah [09:07] Fix-ed. [09:07] woo JS regression, the bug page says "remove assignee" when there isn't one [09:07] wgrant: you've got all pt_br [09:08] bigjools: WFM [09:08] s/got/gone/ [09:09] wgrant: the test should not be expanding the doctest, would you mind making a unit test? [09:10] test_archive.py [09:11] Hello [09:13] bigjools: k [09:17] bigjools: Should I take out the whole doctest while I'm at it? It's short enough. [09:17] wgrant: be my guest [09:18] the test style in test_archive should be obvious [09:23] Yeah, I've already added a few to it. [09:26] good morning [09:28] Hallo adeuring! [09:28] moin henninge [09:32] So, I am trying to update sample data that is used in tests. [09:32] Uhoh. [09:33] I know, I should rewrite the test to not use sample data, but ... [09:33] ... I want to finish this branch today. ;-) [09:34] Anyway, I ran the migration script locally that updates the data the way I need it. [09:34] You need to run it on launchpad_ftest_playground. [09:34] Did "make newsampledata" and updated current.sql and current-dev.sql. [09:34] Then make newsampledata. [09:34] oh [09:35] hm [09:35] what do I get then? [09:36] newsampledata.sql and newsampledata-dev.sql, or something like that. [09:36] Then you clobber current(-dev).sql with those. [09:52] Interesting. I just checked the (changed) current(-dev).sql that I have and what I did only changed current-dev.sql but not current.sql. [09:52] Right, you ran it on launchpad_dev, not launchpad_ftest_playground. [09:52] I did not know that. [09:52] You may need to run with LPCONFIG=test-playground [09:52] How do I ... [09:52] .. thanks ;) [09:53] Oh yeah, thanks for running those three branches last night. [09:53] They all passed and landed first time! [09:53] This hasn't happened in many months. [09:55] :-D [09:59] Looking good, let's see if my tests passes now ... [10:10] Hooray! ;) [10:10] Excellent. [10:22] wgrant: I had a thought [10:22] your change to getFileByName will revert us back to the old situation [10:22] Not entirely. [10:22] well pretty much, as soon as a file expires you can re-upload it [10:22] Right. We cannot prevent that. [10:23] Because somebody put the hashes in the LFC. [10:23] we need to [10:23] which was stupid [10:23] Yes. [10:23] It will stop the primary archive case. [10:23] And that is the one we really care about. [10:23] we care about people screwing up the PPA publisher too [10:23] They can't do that. [10:23] Since the files will have expired, the old source can't be copied any morew. [10:25] I'll have to think about it [10:25] my manflu is blocking rational thought [10:26] My fix makes us a quite a bit better than we were before. And it's as good as we can get without changing the data model. [10:26] I really wish people would think about what they're doing before they start irrevocably deleting data. [10:27] hahaha [10:28] LP has a great history of not deleting anything [10:28] Yes. [10:28] The librarian has a girth that rivals Ron Jeremy [10:29] Sure, and we need to delete things from it. [10:29] But there are better ways. [10:29] Like not deleting the hashes. [10:29] * bigjools is filing a bug about that right now [10:29] * wgrant makes a note to check it in a decade. [10:30] https://bugs.edge.launchpad.net/launchpad-foundations/+bug/687752 [10:30] <_mup_> Bug #687752: It's impossible to check a librarian file's hash once it's expired [10:30] * bigjools notices that gary_poster is not a Foundations bug contact :) [10:31]