| bac | thumper, rockstar, stub, mwhudson, stevek: reviewers meeting | 00:02 |
|---|---|---|
| mwhudson | something in the bugs page seems to make firefox go insane :( | 00:05 |
| mwhudson | js-wise | 00:05 |
| === Ursinha is now known as Ursinha-afk | ||
| lifeless | jml: another recipe bug for you - https://bugs.launchpad.net/launchpad-code/+bug/687623 | 01:00 |
| _mup_ | Bug #687623: No ability to manually trigger a build 'like the daily build system' <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/687623> | 01:00 |
| spm | thumper: spiv: would appreciate your thoughts here: we have ~ 10 requests against BRANCH:%2Bbranch/tsumego20k/1.0 but https://code.launchpad.net/~branch says no code? | 01:08 |
| spiv | spm: %2B is +, not ~ | 01:09 |
| spm | Ah | 01:09 |
| spm | wrong group. that's a start. | 01:09 |
| spm | https://code.launchpad.net/tsumego20k looks more like it | 01:10 |
| spiv | But there's no 1.0 series. | 01:11 |
| spiv | There's a 'stable' series (with a 1.0 release). | 01:11 |
| spm | hrm. and their branch is only 472K as well. | 01:12 |
| spiv | So, requests for that branch are presumably wrong, and ought to give "branch not found". | 01:12 |
| spm | lp:tsumego20k as in | 01:12 |
| spiv | they probably mean /stable rather than /1.0? | 01:12 |
| spm | well. it gets worse. that, and another branch are eating memory. just got a swap alert. I'm going to have to big stick shortly. | 01:13 |
| spiv | Oh and for added fun the stable branch appears to have a broken stacked-on target. | 01:13 |
| spm | would that be causing the pain that codehost is experiencing? | 01:14 |
| spiv | I wouldn't *think* so | 01:14 |
| spm | ah. they have another: BRANCH:~goadict/tsumego20k/release-1.0 | 01:14 |
| spiv | But it could be... it's hard to be sure. | 01:15 |
| spm | mwhudson: so gist is we have one, maybe 2 or 3 branches that are spinning doing "something"; and eating memory and CPU | 01:16 |
| spiv | I don't know of any open bugs that would cause this sort of spinning now that codehosting is running bzr 2.2 | 01:16 |
| mwhudson | ah ok | 01:16 |
| spiv | Er, 2.2.2 | 01:16 |
| mwhudson | spm: have you whacked them with that gdb thing? | 01:17 |
| spm | no, wasn't game to just yet. | 01:17 |
| spm | haven't done any whacking - just watching/gathering. | 01:17 |
| mwhudson | k | 01:17 |
| spm | https://pastebin.canonical.com/40699/ | 01:17 |
| spm | https://pastebin.canonical.com/40700/ for a process listing | 01:18 |
| spm | the db-devel ones, while alarming to have a few, don't seem to be problematic. | 01:19 |
| spiv | Huh. | 01:19 |
| spiv | Some say "+branch", some say "%2Bbranch". I wonder why. | 01:20 |
| spiv | Probably a red herring for this issue. | 01:20 |
| spm | the cpu/load spike seems to be coming down tho. so I'm strongly tempted to just let things resolve on their own. | 01:20 |
| spm | well... load. not cpu. that's still flat out. | 01:20 |
| mwhudson | that's a lot of processes | 01:23 |
| mwhudson | i wonder which user 2978124 is | 01:23 |
| mwhudson | probably launchpad-pqm | 01:23 |
| spm | how do we find out? select * from person? | 01:23 |
| mwhudson | y | 01:24 |
| spm | too obvious that. if even a sysadmin can guess correctly. /me files a big for greater obscurity. | 01:24 |
| spm | bug, too. | 01:24 |
| spm | launchpad-buildbot | 01:25 |
| mwhudson | ah ok, that makes some kind of sense too | 01:25 |
| spm | so something baout those branches tho is driving codehost to 100% cpu for ages, which is bad. :-( | 01:29 |
| spm | worth a pygdb on one? | 01:29 |
| spm | haha. on codehost itself, and it's easier to copy the branch from my local pc up. | 01:30 |
| mwhudson | hee hee | 01:32 |
| mwhudson | i guess you could run bzr get /srv/.../00/0x/xx/xx if you coudl be bothered | 01:32 |
| mwhudson | yeah, i reckon pygdb one | 01:32 |
| mwhudson | the user on the other end must be bored already by now | 01:32 |
| spm | or afk | 01:35 |
| spm | oh that's a bad sign... the backtrace.py is not showing a whole lotta activity... | 01:39 |
| * spm lets it be for a bit longer | 01:39 | |
| mwhudson | :/ | 01:40 |
| spm | mwhudson: I gave up. it never progressed beyond 'Thread 1' as output. trying a gcore. | 02:06 |
| mwhudson | spm: :( | 02:06 |
| mwhudson | spm: maybe just attach with gdb and run 'bt' | 02:06 |
| mwhudson | the process will only be single threaded | 02:07 |
| spm | sure | 02:07 |
| spiv | mwhudson: not necessarily (but probably) | 02:07 |
| spm | err. pebkac? $ gdb 23558 ; (gdb) bt ==> No stack. | 02:08 |
| spiv | mwhudson: the insert_stream verb in the server spawns a thread as that was easiest... | 02:08 |
| spiv | spm: gdb -p 23558 ? | 02:08 |
| spm | bah | 02:08 |
| spm | that's better. ta | 02:08 |
| mwhudson | spiv: whee | 02:09 |
| spm | oh wow. it looks almost like recursion hell | 02:09 |
| spm | pls to save me from rtfm - the command to disable gdb paging? | 02:09 |
| spm | set pagination off | 02:10 |
| mwhudson | i guess that might explain why pygdb didn't work so well | 02:10 |
| mwhudson | though it prints the stack as it goes | 02:11 |
| spm | holy crap. I'm copping 50k/s of gdb output. and it's not stopping..... | 02:11 |
| spiv | spm: be careful what you wish for! | 02:11 |
| spm | reckon | 02:12 |
| spm | glad I don't have "infinite" buffers any more, or this'd suck | 02:12 |
| mwhudson | well this sounds like fin | 02:12 |
| mwhudson | fun | 02:12 |
| spm | https://pastebin.canonical.com/40701/ for 3 pages of so far | 02:13 |
| mwhudson | whisky tango foxtrot!! | 02:14 |
| spm | https://pastebin.canonical.com/40702/ next 4 pages (i think...) | 02:15 |
| spm | i call busticated. | 02:15 |
| spiv | Huh, this is an lp-serve process? | 02:15 |
| spm | codehost? | 02:15 |
| spm | yes | 02:16 |
| mwhudson | spiv: yes | 02:16 |
| spiv | Ah, I see, it veers off into Twisted via lp.codehosting.vfs.transport.SynchronousAdapter | 02:16 |
| spiv | So this is a bit disheartening :( | 02:17 |
| spm | spiv: so it's your fault? | 02:17 |
| * spm establishes random pin-the-blame-game early | 02:17 | |
| spiv | spm: well, I thought bzr 2.2.2 fixed this bug. :/ | 02:17 |
| spm | :-( | 02:18 |
| spiv | Or rather, fixed one of the two issues that combined to cause it. | 02:18 |
| mwhudson | oh, it's stringifying a traceback inside twisted's failure code | 02:18 |
| spm | ah luverly, so we've maybe just discovered #3? | 02:18 |
| spiv | It appears to be a error-handler-falls-over-and-recurses-into-the-error-handler death sprial. | 02:18 |
| spiv | mwhudson: right | 02:18 |
| spm | spiv: so stabby time, or more info would be useful? | 02:18 |
| * spiv digs up the issue he thought he fixed. | 02:19 | |
| spiv | spm: hmm. The traceback is probably enough. Can you give me a minute to ponder? | 02:19 |
| spm | sure. it's not wonderful, but getting this fixed is worth the pain | 02:19 |
| mwhudson | spm: can you paste some more traceback actually? | 02:20 |
| spm | easily | 02:20 |
| spm | well.. easily but boringly. just sayin' | 02:20 |
| mwhudson | just a couple more pages | 02:20 |
| spiv | Argh, this really looks like the bug I fixed! | 02:21 |
| mwhudson | i'm not sure if it's infinite recursion or just stupidly deep recursion | 02:21 |
| spiv | repr of NotBranchError | 02:21 |
| spm | https://pastebin.canonical.com/40703/ | 02:21 |
| spm | mwhudson: I don't believe there's a difference outside academia there :-) | 02:22 |
| * spiv hmms | 02:22 | |
| spiv | Oh, *blah* | 02:23 |
| * spiv thinks out loud | 02:23 | |
| spiv | NotBranchError guards against open_repository failing with unexpected exceptions | 02:23 |
| spm | you think "*blah*"?? really!?!?! | 02:23 |
| mwhudson | spiv: let me guess, it's partly twisted's fault? | 02:23 |
| spiv | mwhudson: (yes) | 02:23 |
| spiv | But open_repository is calling off into the lp VFS stuff | 02:23 |
| spiv | Which calls into Twisted stuff | 02:23 |
| mwhudson | anything traceback that has __getstate__ in it instantly makes me want to cry | 02:24 |
| mwhudson | and punch glyph in the face in about 2001 | 02:24 |
| * spm hands mwhudson a tissue box, and starts playing a small violin | 02:24 | |
| spiv | Which thanks to the crumminess of Twisted's Failure class stringifies errors and tracebacks | 02:24 |
| spiv | Which happens inside the lp codehosting VFS stuff, so my NotBranchError guard doesn't matter: Twisted tries to stringify a NotBranchError (whose bzrdir fails badly on open_repository) | 02:25 |
| spiv | So off we go into the recursion. | 02:26 |
| lifeless | === Top 10 Time Out Counts by Page ID === | 02:26 |
| lifeless | Hard / Soft Page ID | 02:26 |
| lifeless | 146 / 5425 Archive:+index | 02:26 |
| lifeless | 62 / 400 Distribution:+bugs | 02:26 |
| lifeless | 61 / 233 BugTask:+index | 02:26 |
| lifeless | 59 / 2 Person:EntryResource:retractTeamMembership | 02:26 |
| lifeless | 38 / 419 Distribution:+bugtarget-portlet-bugfilters-stats | 02:26 |
| lifeless | 20 / 28 DistroArchSeries:+index | 02:26 |
| lifeless | 11 / 16 DistroSeries:+queue | 02:26 |
| lifeless | 11 / 2 Cve:+index | 02:26 |
| lifeless | 10 / 95 ProjectGroupSet:CollectionResource:#project_groups | 02:26 |
| lifeless | 9 / 6 ProjectGroup:+milestones | 02:26 |
| spiv | Roughly: repr(NotBranchError) -> open_repository -> codehosting VFS -> Twisted gunk -> repr(NotBranchError) | 02:26 |
| wgrant | The fix for the top one should be on qastaging in a few minutes, hopefully. | 02:26 |
| spiv | mwhudson: does that sound plausible? | 02:27 |
| mwhudson | spiv: yes | 02:27 |
| spiv | mwhudson: ok, now what can we do about it? :( | 02:27 |
| mwhudson | spiv: make Failure throw away the traceback entirely? | 02:27 |
| spiv | mwhudson: I *think* perhaps this isn't something bzrlib can sensibly fix. | 02:27 |
| mwhudson | perhaps with a monkey patch? | 02:28 |
| mwhudson | spiv: how can we reproduce this locally? | 02:28 |
| spiv | mwhudson: IIRC the condition is something like a lookup on +branch/something/something causes a strange VFS error | 02:28 |
| mwhudson | spiv: i don't quite understand why NotBranchError is getting raised in the smart server | 02:29 |
| mwhudson | oh no, maybe that makes sense | 02:29 |
| wgrant | mwhudson: Stacking, maybe? | 02:29 |
| spiv | Because the codehosting VFS is returning that from a transport operation, or something odd like that? | 02:29 |
| spiv | thumper may remember the details | 02:29 |
| mwhudson | wgrant: stacking is mostly a client side thing though | 02:29 |
| spiv | I'll see if I can dig out the relevant conversation from my logs | 02:29 |
| mwhudson | spiv: is NotBranchError raised by BzrDir.open or something maybe? | 02:30 |
| spiv | It's definitely an issue relating to how the server-side vfs translates stuff, maybe in weird cases like a dev focus stacked on itself, or something. | 02:30 |
| spiv | IIRC it's at the point of trying to open the bzrdir that in LP would have the auto-stacking config | 02:31 |
| spiv | IIRC +branch might be involved? Or something? | 02:31 |
| spiv | mwhudson: yeah, probably | 02:31 |
| mwhudson | +branch shouldn't be involved unless someone has been editing .bzr/branch/branch.conf by hand | 02:31 |
| spm | i'm about to break for lunch - need anything more before i run? | 02:32 |
| lifeless | noone would be silly enough to do that :O | 02:32 |
| spiv | spm: I think it's probably ok to kill before you run, if you want | 02:32 |
| spm | spiv: okidoki | 02:33 |
| mwhudson | spm: yeah, as spiv says | 02:33 |
| spiv | mwhudson: see logs for this channel for Oct 11 | 02:33 |
| spiv | mwhudson: I think that's when I chatted with thumper about this initially | 02:33 |
| spiv | just reviewing now | 02:33 |
| spm | ugh. looks like the owner is kicking the branch/whatever operation(s) off again | 02:34 |
| * spm has applied the sledgehammer :-( | 02:35 | |
| * mwhudson pokes around with sftp | 02:38 | |
| spiv | mwhudson: so in thumper's situation (which involved a patch he may have abandoned, so there may some red herrings here) | 02:38 |
| mwhudson | i can't see anything unusual :( | 02:40 |
| mwhudson | https://code.launchpad.net/~goadict/tsumego20k/trunk | 02:40 |
| spiv | mwhudson: he encountered this problem on an open_branch HPSS call to ~user/proj (trying 'bzr revno ~user/proj/no-such-branch', I think) | 02:40 |
| mwhudson | isn't a real branch | 02:40 |
| mwhudson | hang on, let me check what it _is_ | 02:40 |
| mwhudson | spiv: hm | 02:41 |
| spiv | mwhudson: default_stack_on = /~goadict/tsumego20k/trunk | 02:41 |
| mwhudson | there's a bzrdir there, but no branch | 02:41 |
| mwhudson | spiv: yeah, which is what one would expect | 02:41 |
| spiv | mwhudson: in sftp://bazaar.launchpad.net/~goadict/tsumego20k/.bzr/control.conf | 02:41 |
| mwhudson | spiv: right | 02:41 |
| spiv | That is a branch AFAICT over SFTP? | 02:42 |
| spiv | and not a stacked branch. | 02:42 |
| mwhudson | spiv: parse error | 02:42 |
| spiv | mwhudson: lftp bazaar.launchpad.net:~goadict/tsumego20k/trunk/.bzr> cat branch/format | 02:43 |
| spiv | Bazaar Branch Format 7 (needs bzr 1.6) | 02:43 |
| spiv | Why do you say that ~goadict/tsumego20k/trunk isn't a real branch? | 02:43 |
| mwhudson | spiv: by mistake, sorry | 02:43 |
| spiv | Ah. | 02:43 |
| mwhudson | spiv: https://code.launchpad.net/~goadict/tsumego20k/stable | 02:44 |
| mwhudson | spiv: ^ that one is just a bzrdir | 02:44 |
| mwhudson | lftp mwhudson@bazaar.launchpad.net:/~goadict/tsumego20k/stable/.bzr> ls | 02:44 |
| mwhudson | -rw-r--r-- 1 1001 1001 141 Dec 09 02:21 README | 02:44 |
| mwhudson | -rw-r--r-- 1 1001 1001 35 Dec 09 02:21 branch-format | 02:44 |
| mwhudson | drwxr-xr-x 2 1001 1001 4096 Dec 09 02:21 branch-lock | 02:44 |
| mwhudson | the gdb trace is talking about ~goadict/tsumego20k/release-1.0/.bzr/repository though | 02:45 |
| mwhudson | so maybe this stable branch is a red herring | 02:45 |
| spiv | I don't think it is. | 02:46 |
| spiv | My recollection is that accessing non-existent paths triggered this codehosting vfs issue. | 02:47 |
| spiv | But maybe I'm wrong :/ | 02:48 |
| mwhudson | yeah, maybe that's it | 02:48 |
| mwhudson | it's striking that all the problem processes are accessing paths that shouldn't work at all, like | 02:48 |
| mwhudson | BRANCH:%2Bbranch/tsumego20k/1.0 | 02:48 |
| mwhudson | codehost | 02:48 |
| mwhudson | the thing is of course that maybe the user is getting confused and flailing around creating and deleting things in the webapp | 02:49 |
| spiv | mwhudson: also, the gdb trace includes stuff like: | 02:49 |
| spiv | <Deferred at 0xa99a878 current result: <twisted.python.failure.Failure <class 'bzrlib.errors.NoSuchFile'>>> | 02:49 |
| mwhudson | spiv: that's not so surprising really | 02:49 |
| mwhudson | the synchronous launchpad transport wraps the asynchronous one | 02:50 |
| spiv | mwhudson: argh, the data is changing from under us :) | 02:52 |
| spiv | mwhudson: now ~goadict/tsumego20k/stable has a repo | 02:52 |
| spiv | A bit odd: _cache={'//~goadict/tsumego20k/release-1.0': ('BRANCH_TRANSPORT', {'writable': True, 'trailing_path': '.bzr/repository/format', 'id': 410572}, <float at remote 0x34d4cd0>)} | 02:56 |
| mwhudson | writeable=True suggests that branch existed in the db at the time | 02:59 |
| mwhudson | wow, i was a bit confused there -- was looking at launchpad.net not launchpad.dev :) | 03:07 |
| mwhudson | of course launchpad.dev is impossible to use | 03:08 |
| mwhudson | because of 1 miiiiiiiiiiiiiillion js files | 03:09 |
| wgrant | devmode off | 03:09 |
| wgrant | problem solved. | 03:09 |
| mwhudson | GET /+icing/rev10399/yui/datatype/lang/datatype-date-format_it-IT.js | 03:09 |
| mwhudson | GET /+icing/rev10399/yui/datatype/lang/datatype-date-format_it.js | 03:09 |
| wgrant | Yes. | 03:09 |
| mwhudson | COME ON! | 03:09 |
| wgrant | And half of them OOPS. | 03:09 |
| wgrant | So /var/tmp/lperr fills up. | 03:09 |
| mwhudson | mostly 404ing | 03:09 |
| mwhudson | where is devmode set? configs/development/launchpad.conf? | 03:10 |
| wgrant | That. | 03:10 |
| mwhudson | $ du -sh /var/tmp/lperr/2010-12-09/ | 03:11 |
| mwhudson | 4.6M/var/tmp/lperr/2010-12-09/ | 03:11 |
| wgrant | Yeah. | 03:11 |
| mwhudson | spiv: well i don't know what's going on :/ | 03:15 |
| spiv | mwhudson: hmm | 03:17 |
| mwhudson | spiv: i guess it might be interesting to see the bottom 100 lines or so of the traceback | 03:19 |
| spiv | mwhudson: yeah | 03:19 |
| spiv | mwhudson: also translateVirtualPath maybe should have a catch-all errback | 03:20 |
| mwhudson | spiv: sounds plausible, without looking at the code or thinking terribly hard | 03:21 |
| spiv | mwhudson: to avoid unexpected errors leaking out and confusing callers that expect only transport-related errors. | 03:21 |
| spiv | I don't see any obvious routes for those to happen | 03:21 |
| spiv | And I've just spent the last 30 min or so looking for one :) | 03:21 |
| spm | bleh, they're still trying to get the code, or update or whatever. have sent the team members emails asking them to stop. | 03:23 |
| mwhudson | spm: it would be good to know exactly what they're trying to do | 03:25 |
| spm | mwhudson: spiv: would a copy of the raw branch(s) off crowberry be of any use to you guys in debugging? seeing as they're able to kill it time and again. | 03:25 |
| spm | mwhudson: sure - do we have a formal bug yet? I'll send that to them and ask them to pls update with as much info as possible. | 03:25 |
| mwhudson | spm: dunno, not sure | 03:25 |
| spiv | spm: more use to mwhudson probably | 03:25 |
| mwhudson | spm: how big are they? | 03:25 |
| spiv | My local lp dev environment isn't quite functional | 03:25 |
| mwhudson | spm: i haven't filed a bug | 03:26 |
| spm | spiv: you're one up on me. don't have one at all. :-) | 03:26 |
| spm | mwhudson: I was going to file "Weird shit happening with codehost" but felt it was lacking | 03:26 |
| spiv | spm: "Error handling infinite recursion death spiral of doom" maybe? | 03:26 |
| spiv | The "of doom" is a sure way to add some drama to a lacklustre title ;) | 03:27 |
| spm | I like my subject better, nothing personal | 03:27 |
| spiv | "Weird shit of doom happening with codehost" then :P | 03:27 |
| spm | but yours is more technically accurate. I'll go with that | 03:27 |
| spm | hahaha | 03:27 |
| spiv | (A slow, painful doom...) | 03:28 |
| mwhudson | weird-shit-of-doom sounds like a good bug tag | 03:28 |
| spm | +1 | 03:28 |
| spm | mwhudson: the only one that works is tiny. http://bazaar.launchpad.net/~goadict/tsumego20k/trunk/files | 03:29 |
| spiv | spm: for added laughs the user has been actively changing stuff while we've been looking | 03:29 |
| spiv | Replacing an empty bzrdir with one with a repo etc. | 03:30 |
| spm | well. trying to. I keep killing their processes off as it hurts | 03:30 |
| spm | shame. codehost has dropped to #2 spot in firefox quick find for "filebug" | 03:34 |
| wgrant | Does Soyuz win? | 03:34 |
| spm | ha. no. most of my bugs are generically filed against launchpad these days. never quite sure where they go. | 03:35 |
| lifeless | from mid jan 'launchpad' | 03:35 |
| lifeless | one bug tracker. \o/ | 03:35 |
| wgrant | We'll see how that goes. | 03:36 |
| spiv | wgrant: more successfully than lifeless' holiday, I hope... ;) | 03:37 |
| wgrant | Indeed. | 03:37 |
| spiv | lifeless: I'd suggest you visit some remote island with minimal internet access | 03:37 |
| spm | spiv 1, lifeless 0 | 03:38 |
| spiv | lifeless: but you already moved to NZ ;) | 03:38 |
| spm | spiv 3, lifeless 0 | 03:38 |
| * lifeless is on wow atm | 03:38 | |
| spiv | lifeless: that's more like it :) | 03:38 |
| lifeless | so, I claim success. | 03:38 |
| spm | spiv 3, lifeless -1 | 03:38 |
| lifeless | oi! | 03:38 |
| spiv | Some people are so judgemental! | 03:39 |
| spm | spm 1, spiv 0, lifeless -1 | 03:39 |
| lifeless | s/ w+!/sysadmins!/ | 03:39 |
| lifeless | wow | 03:39 |
| lifeless | found the stone core | 03:39 |
| spm | someone has to be. it's a burden that we take upon ourselves to spare the lesser folks aka developers | 03:39 |
| lifeless | first quest in there... | 03:39 |
| lifeless | 115K xp | 03:40 |
| spm | https://bugs.edge.launchpad.net/launchpad-code/+bug/687653 | 03:43 |
| _mup_ | Bug #687653: codehost error handling infinite recursion death spiral of doom <canonical-losa-lp> <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/687653> | 03:43 |
| === Ursinha-afk is now known as Ursinha | ||
| wgrant | lifeless: https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages is promising. 4s. | 04:18 |
| wgrant | Ah, 2s with hot cache. | 04:19 |
| wgrant | Not too bad. | 04:19 |
| wgrant | OOPS-1804QS12 | 04:20 |
| lifeless | wgrant: | 05:17 |
| lifeless | SQL time: 16441 ms | 05:17 |
| lifeless | Non-sql time: 4324 ms | 05:17 |
| lifeless | Total time: 20765 ms | 05:17 |
| lifeless | Statement Count: 50 | 05:17 |
| lifeless | wgrant: also there are two repeats of a 1000ms query | 05:18 |
| wgrant | Hmm. | 05:20 |
| wgrant | A COUNT()? | 05:20 |
| lifeless | 2203410171017SQL-launchpad-main-slave(SELECT SourcePackagePublishingHistory.archive, SourcePackagePublishingHistory.component, SourcePackagePublishingHistory.datecreated, SourcePackagePublishingHistory.datemadepending, SourcePackagePublishingHistory.datepublished, SourcePackagePublishingHistory.dateremoved, | 05:20 |
| lifeless | ... | 05:20 |
| lifeless | SourcePackagePublishingHistory.id, DistroArchSeries.architecturetag) EXCEPT (SELECT SourcePackagePublishingHistory.archive, | 05:20 |
| wgrant | Oh, that one. | 05:20 |
| LPCIBot | Project db-devel build (206): FAILURE in 13 min: https://hudson.wedontsleep.org/job/db-devel/206/ | 05:38 |
| === almaisan-away is now known as al-maisan | ||
| bigjools | morning all | 08:59 |
| wgrant | Morning bigjools. | 08:59 |
| wgrant | We has a regression. | 09:00 |
| wgrant | Which I already had a one-line branch to fix... | 09:00 |
| wgrant | Without knowing about it. | 09:00 |
| bigjools | wgrant: do tell | 09:00 |
| wgrant | bigjools: Bug #687662 | 09:00 |
| _mup_ | Bug #687662: Upload processor attempts to verify hashes against expired files <Soyuz:New> <https://launchpad.net/bugs/687662> | 09:00 |
| bigjools | how did that happen? | 09:01 |
| wgrant | Hmm? | 09:01 |
| bigjools | oh, that change for getFile | 09:01 |
| wgrant | Yeah. | 09:01 |
| bigjools | do you think more pairs of eyes would have caught that? | 09:01 |
| wgrant | Who knows. | 09:02 |
| bigjools | wgrant: what affect elsewhere will your fix cause? | 09:03 |
| wgrant | bigjools: It'll fix the +files/foo.orig.tar.gz 404. It could also impact other A.getFileByName callsites, but I doubt any rely on this behaviour, what with expired LFAs being useless and everything. | 09:04 |
| bigjools | wgrant: right | 09:06 |
| bigjools | wgrant: whack up an MP and let's get it landed | 09:06 |
| wgrant | bigjools: MP is already there. | 09:06 |
| wgrant | Oh. Branch is not linked. | 09:06 |
| bigjools | yeah | 09:07 |
| wgrant | Fix-ed. | 09:07 |
| bigjools | woo JS regression, the bug page says "remove assignee" when there isn't one | 09:07 |
| bigjools | wgrant: you've got all pt_br | 09:07 |
| wgrant | bigjools: WFM | 09:08 |
| bigjools | s/got/gone/ | 09:08 |
| bigjools | wgrant: the test should not be expanding the doctest, would you mind making a unit test? | 09:09 |
| bigjools | test_archive.py | 09:10 |
| mrevell | Hello | 09:11 |
| wgrant | bigjools: k | 09:13 |
| wgrant | bigjools: Should I take out the whole doctest while I'm at it? It's short enough. | 09:17 |
| bigjools | wgrant: be my guest | 09:17 |
| bigjools | the test style in test_archive should be obvious | 09:18 |
| wgrant | Yeah, I've already added a few to it. | 09:23 |
| adeuring | good morning | 09:26 |
| henninge | Hallo adeuring! | 09:28 |
| adeuring | moin henninge | 09:28 |
| henninge | So, I am trying to update sample data that is used in tests. | 09:32 |
| wgrant | Uhoh. | 09:32 |
| henninge | I know, I should rewrite the test to not use sample data, but ... | 09:33 |
| henninge | ... I want to finish this branch today. ;-) | 09:33 |
| henninge | Anyway, I ran the migration script locally that updates the data the way I need it. | 09:34 |
| wgrant | You need to run it on launchpad_ftest_playground. | 09:34 |
| henninge | Did "make newsampledata" and updated current.sql and current-dev.sql. | 09:34 |
| wgrant | Then make newsampledata. | 09:34 |
| henninge | oh | 09:34 |
| henninge | hm | 09:35 |
| henninge | what do I get then? | 09:35 |
| wgrant | newsampledata.sql and newsampledata-dev.sql, or something like that. | 09:36 |
| wgrant | Then you clobber current(-dev).sql with those. | 09:36 |
| henninge | Interesting. I just checked the (changed) current(-dev).sql that I have and what I did only changed current-dev.sql but not current.sql. | 09:52 |
| wgrant | Right, you ran it on launchpad_dev, not launchpad_ftest_playground. | 09:52 |
| henninge | I did not know that. | 09:52 |
| wgrant | You may need to run with LPCONFIG=test-playground | 09:52 |
| henninge | How do I ... | 09:52 |
| henninge | .. thanks ;) | 09:52 |
| wgrant | Oh yeah, thanks for running those three branches last night. | 09:53 |
| wgrant | They all passed and landed first time! | 09:53 |
| wgrant | This hasn't happened in many months. | 09:53 |
| henninge | :-D | 09:55 |
| henninge | Looking good, let's see if my tests passes now ... | 09:59 |
| henninge | Hooray! ;) | 10:10 |
| wgrant | Excellent. | 10:10 |
| bigjools | wgrant: I had a thought | 10:22 |
| bigjools | your change to getFileByName will revert us back to the old situation | 10:22 |
| wgrant | Not entirely. | 10:22 |
| bigjools | well pretty much, as soon as a file expires you can re-upload it | 10:22 |
| wgrant | Right. We cannot prevent that. | 10:22 |
| wgrant | Because somebody put the hashes in the LFC. | 10:23 |
| bigjools | we need to | 10:23 |
| bigjools | which was stupid | 10:23 |
| wgrant | Yes. | 10:23 |
| wgrant | It will stop the primary archive case. | 10:23 |
| wgrant | And that is the one we really care about. | 10:23 |
| bigjools | we care about people screwing up the PPA publisher too | 10:23 |
| wgrant | They can't do that. | 10:23 |
| wgrant | Since the files will have expired, the old source can't be copied any morew. | 10:23 |
| bigjools | I'll have to think about it | 10:25 |
| bigjools | my manflu is blocking rational thought | 10:25 |
| wgrant | My fix makes us a quite a bit better than we were before. And it's as good as we can get without changing the data model. | 10:26 |
| wgrant | I really wish people would think about what they're doing before they start irrevocably deleting data. | 10:26 |
| bigjools | hahaha | 10:27 |
| bigjools | LP has a great history of not deleting anything | 10:28 |
| wgrant | Yes. | 10:28 |
| bigjools | The librarian has a girth that rivals Ron Jeremy | 10:28 |
| wgrant | Sure, and we need to delete things from it. | 10:29 |
| wgrant | But there are better ways. | 10:29 |
| wgrant | Like not deleting the hashes. | 10:29 |
| * bigjools is filing a bug about that right now | 10:29 | |
| * wgrant makes a note to check it in a decade. | 10:29 | |
| bigjools | https://bugs.edge.launchpad.net/launchpad-foundations/+bug/687752 | 10:30 |
| _mup_ | Bug #687752: It's impossible to check a librarian file's hash once it's expired <Launchpad Foundations:New> <https://launchpad.net/bugs/687752> | 10:30 |
| * bigjools notices that gary_poster is not a Foundations bug contact :) | 10:30 | |
| wgrant | Does he read the ML, perhaps? | 10:31 |
| wgrant | Oh I hate doctests. | 10:38 |
| lifeless | wgrant: so your fix looks good? | 10:58 |
| bigjools | wgrant: I guessed it wouldn't be long before the "no ui" whingers started on the ppa stats bug | 10:58 |
| wgrant | lifeless: For which? | 10:59 |
| lifeless | Archive:+index | 10:59 |
| wgrant | lifeless: It seems to behave OK on qastaging now. | 10:59 |
| lifeless | cool | 11:00 |
| wgrant | Not great. | 11:00 |
| bigjools | lifeless: you do a really bad impression of someone who's on holiday :) | 11:00 |
| wgrant | But I will look at that more when I don't have to pester too many people. | 11:00 |
| lifeless | bigjools: I spent all day playing WoW :) | 11:00 |
| bigjools | yes I know someone else who took holiday to do that :) | 11:00 |
| lifeless | bigjools: the reason I take big blocks of holiday is it takes me ages to spin down and forget about work. | 11:00 |
| lifeless | bigjools: 2 weeks in is about where I actually start to feel like I'm on leave | 11:00 |
| bigjools | maybe disconnecting from IRC would help? :) | 11:00 |
| lifeless | bigjools: I have many non work interests I wish to be on IRC for | 11:01 |
| * wgrant searches for the banhammer. | 11:01 | |
| wgrant | bigjools: New tests are up, this time without conflicts. | 11:09 |
| bigjools | bonus | 11:09 |
| jml | lifeless: last time I went on leave, I also stayed on IRC but deliberately left many work-related channels. | 11:14 |
| jml | lifeless: I found that helped | 11:14 |
| * jml isn't really here | 11:15 | |
| henninge | Does anybody have an idea what this is about? | 11:45 |
| henninge | http://paste.ubuntu.com/541413/ | 11:45 |
| maxb | traceback header with no traceback? thats a bit odd | 11:48 |
| wgrant | maxb: Is there a librarian running? | 11:50 |
| wgrant | Er, henninge ^^ | 11:50 |
| henninge | wgrant: oh, you mean an old process? | 11:51 |
| henninge | yes :( | 11:51 |
| wgrant | Hmm. | 11:51 |
| * henninge slaps head | 11:51 | |
| wgrant | Looks like the new librarian fixture needs work. | 11:51 |
| henninge | yes, possibly. But I just remembered that I had this issue before after forcibly terminating a test run. | 11:52 |
| henninge | but I agree that something like this should be detected | 11:53 |
| henninge | Tests are running happily now :) | 11:54 |
| deryck | Morning, all. | 12:08 |
| === Ursinha is now known as Ursinha-brb | ||
| gary_poster | bigjools: I get those emails somehow or other | 14:15 |
| bigjools | gary_poster: maybe you're subscribed to lp bug | 14:16 |
| bigjools | s | 14:16 |
| gary_poster | maybe so | 14:16 |
| === matsubara is now known as matsubara-lunch | ||
| benji | mthaddon: I'm working on the script to run the LP tests in the new tarmac-based world and would like to write up how to set up the environment it needs; do you know of any server setup documentation for the current buildbot machines? | 15:31 |
| mthaddon | benji: otp | 15:31 |
| benji | mthaddon: thanks, no rush | 15:31 |
| === matsubara-lunch is now known as matsubara | ||
| EdwinGrubbs | allenap: hi | 15:47 |
| allenap | EdwinGrubbs: Hi there. | 15:56 |
| allenap | EdwinGrubbs: I wrote down a kind of rationale for BugSubscriptionInfo: http://pastebin.ubuntu.com/541503/ | 15:57 |
| * EdwinGrubbs looks | 15:57 | |
| EdwinGrubbs | allenap: by "snapshot value factory", do you mean that getDirectSubscriptions() might return an existing value? | 15:59 |
| EdwinGrubbs | allenap: what do you think the difference is between this solution and pre_iter_hook? Are you planning on adding more stuff to the BugSubscriptionInfo, which I just don't get to see yet? | 16:00 |
| allenap | EdwinGrubbs: BugSubscriptionInfo is designed to work with a bug snapshot, so that (Zope) subscribers can find out deltas in subscriptions without having to issue many queries. | 16:02 |
| EdwinGrubbs | allenap: do you think that there would be a case in the future that you would need to batch the results of getDirectSubscriptions? | 16:03 |
| allenap | EdwinGrubbs: Some big queries are being hit 7 or more times in a single page request, when they only need to be executed twice. | 16:03 |
| allenap | EdwinGrubbs: I doubt we'll ever have that many subscribers. | 16:03 |
| allenap | direct subscribers. | 16:03 |
| allenap | EdwinGrubbs: What would the pre_iter_hook do in this case? | 16:05 |
| === beuno is now known as beuno-lunch | ||
| EdwinGrubbs | allenap: oops, I meant that pre_iter_hook would eliminate the need for BugSubscriberSet, not BugSubscriptionInfo. pre_iter_hook would take the place of the self.subscribers call made before returning BugScriptionSet.sorted. | 16:08 |
| === salgado is now known as salgado-lunch | ||
| EdwinGrubbs | btw, those are two sentences, although it looks like the period is just separating a class and an attribute. | 16:10 |
| allenap | EdwinGrubbs: Okay :) Sorry for being slow, my other half is badgering me for things at the same time. | 16:14 |
| EdwinGrubbs | no problem | 16:15 |
| allenap | EdwinGrubbs: A pre-iter hook when fetching the subscriptions would work, so it would mean 2 queries when fetching subscriptions. Might not be a bad thing though, because the person/subscriber is almost always referenced. | 16:21 |
| allenap | EdwinGrubbs: That would make the implementation of BugSubscriptionSet and StructuralSubscriptionSet a bit simpler, but only by the same amount that BugSubscriptionInfo gets more complex. | 16:22 |
| allenap | EdwinGrubbs: I don't think it removes the usefulness of BugSubscriberSet, because that's useful for its sorted property. | 16:23 |
| EdwinGrubbs | allenap: sorry, the standup got delayed to just now. | 16:32 |
| allenap | No worries :) | 16:33 |
| EdwinGrubbs | allenap: the potential benefit of pre_iter_hook that may not apply here is that it only runs the main query and the pre_iter_hook query when it is iterated over. The BugSubscriptionSet runs the first query immediately to turn it into a frozenset, and then it runs the second query if the BugSubscriptionSet.sorted is used. | 16:35 |
| EdwinGrubbs | allenap: I'm wondering what the benefit is to putting the results in a set and then putting them into a sorted list. Do you use the set directly someplace else? | 16:36 |
| allenap | EdwinGrubbs: It's necessary to identify different subsets of subscribers. There's the obvious ones like direct and duplicate subscribers, but then there are duplicate-only subscribers (which I've added in a later branch), also notified subscribers, indirect subscribers. Many of these are remixes of the simpler subscription/subscriber sets. | 16:39 |
| allenap | EdwinGrubbs: I could do these as UNION/EXCEPT queries, but the complexity shoots up. | 16:40 |
| allenap | The reason it shoots up is because there are BugSubscription records which are not compatible with StructuralSubscription records, and because some of the queries involve joining through bugtask to target (i.e. product or productseries or ...). | 16:42 |
| EdwinGrubbs | allenap: ok, I don't think I have any more questions about the sets. It just was bothering me to have an unusual implementation solve such a similar problem. | 16:51 |
| EdwinGrubbs | allenap: I have a branch in pqm that effectively implements the same thing as load_people(). I'll not that in the mp with a few other small items. Thanks for explaining everything to me. | 16:52 |
| EdwinGrubbs | s/not/note/ | 16:53 |
| allenap | EdwinGrubbs: Cool. I agree it's weird. I've tried something like this refactor before and a lot of subtle issues seem to crop up, so I started from the premise of having a sets-based API so that I could understand myself what was going on as I went along. | 16:54 |
| gary_poster | jml, ping | 16:56 |
| jml | gary_poster: hi | 16:57 |
| gary_poster | hey. Could matsubara and I pick your brain on mumble for a few minutes about some subunit/ec2 remote.py details? | 16:58 |
| gary_poster | sorry, jml ^^^ | 16:58 |
| jml | gary_poster: sure. gimme a few seconds first. | 16:59 |
| gary_poster | of course, thanks | 16:59 |
| benji | mthaddon: when you get a minute; I have the script ready that will be the driver for the tests in the new tarmac-based SMM world; I think you're the one I should talk to about getting it installed | 17:05 |
| mthaddon | benji: I'm past EOD I'm afraid - can you add the details to the RT? | 17:06 |
| === al-maisan is now known as almaisan-away | ||
| benji | mthaddon: gladly; I haven't done anything with RTs yet, how do I do that? (is there something I should read on the RT system?) | 17:07 |
| === salgado-lunch is now known as salgado | ||
| mthaddon | benji: I've added you as a cc to the ticket, and have sent you a reply - you should be able to just reply to the email | 17:10 |
| benji | mthaddon: great, thanks | 17:10 |
| === deryck is now known as deryck[lunch] | ||
| * jml gone | 17:17 | |
| === beuno-lunch is now known as beuno | ||
| === benji is now known as benji-lunch | ||
| === deryck[lunch] is now known as deryck | ||
| lifeless | gary_poster: jml has run, can I help ? | 19:03 |
| === benji-lunch is now known as benji | ||
| gary_poster | lifeless, he was able to get in touch with us before he ran. thank you very much! | 19:27 |
| lifeless | ok, cool | 19:31 |
| lifeless | if you're doing subunit stuff, it would be cool to wire up SubunitStream ( the table ) to the API for uploads. | 19:32 |
| lifeless | I'd just love ec2test to add a stream to the branch it tests. | 19:33 |
| === Ursinha is now known as Ursinha-brb | ||
| === EdwinGrubbs is now known as Edwin-afk2 | ||
| LPCIBot | Yippie, build fixed! | 20:16 |
| LPCIBot | Project db-devel build (207): FIXED in 3 hr 35 min: https://hudson.wedontsleep.org/job/db-devel/207/ | 20:16 |
| === matsubara is now known as matsubara-afk | ||
| === salgado is now known as salgado-afk | ||
| james_w | deryck, hi, I'm interested in what you say about not liking testing pages. You say that you have talked about this previously. Was that on a public mailing list? | 20:41 |
| deryck | james_w, yes, several threads (2-3 maybe?) on launchpad-dev. about ui, ajax, and testing all this. | 20:41 |
| james_w | deryck, ah, specifically where js gets involved? | 20:42 |
| deryck | james_w, mostly about that, yes. though I do have some criticisms of our page tests in an old proposal thread of mine | 20:42 |
| deryck | but that's probably less about testing web pages and more about bad tests generally | 20:43 |
| deryck | :-) | 20:43 |
| james_w | down with bad tests! | 20:43 |
| james_w | deryck, have you had a look at soupmatchers that is now available in the LP testing infrastructure thanks to rockstar? | 20:43 |
| deryck | james_w, I haven't yet, no. though rockstar did rave to me about it. | 20:44 |
| james_w | deryck, cool. I'd be interested in what you make of it if you find time to have a play with it | 20:44 |
| deryck | james_w, ok, cool. I will definitely look at it and let you know | 20:45 |
| james_w | it's static html only, but I think it could be a lot better than the current approach of extracting text from elements and printing it etc. | 20:45 |
| LPCIBot | Project devel build (285): FAILURE in 3 hr 12 min: https://hudson.wedontsleep.org/job/devel/285/ | 21:43 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=jml][ui=none][no-qa] Update testtools to r151. | 21:43 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][no-qa] Use the stdlib multiprocess module to | 21:43 |
| LPCIBot | run WADL generation in parallel subprocesses. | 21:43 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=638924] Fixed timeout on ubuntu milestone | 21:43 |
| LPCIBot | +index page. | 21:43 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=gmb][ui=none][bug=677511] Fix "invited members" link created by | 21:43 |
| LPCIBot | javascript. | 21:43 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=193870] The bug subscriptions portlet for | 21:43 |
| LPCIBot | StructuralSubscriptionTargets has been updated to improve readability. | 21:43 |
| === Ursinha-brb is now known as Ursinha | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!