wgrant | Right. | 00:00 |
---|---|---|
lifeless | StevenK: its not part of lp-setup ? yes, I'd like to see it | 00:09 |
StevenK | lifeless: It is, it's shipped as an example. | 00:35 |
StevenK | lifeless: http://pastebin.ubuntu.com/1255107/ | 00:36 |
lifeless | ok | 00:37 |
lifeless | can you pastebin the lxc-start-ephemeral script you have ? | 00:37 |
lifeless | StevenK: and what version of testtools and subunit and testrepository do you have (outside the lxc containers) | 00:38 |
=== matsubara-afk is now known as matsubara | ||
StevenK | lifeless: Versions http://pastebin.ubuntu.com/1255115/ | 00:38 |
lifeless | wgrant: heat cam eback, the very next year | 00:39 |
StevenK | lifeless: lxc-start-ephemeral is from precise, lxc 0.7.5-3ubuntu63 | 00:40 |
lifeless | ugh | 00:44 |
lifeless | its got an exit 0 in there. That will cause bad signal, if ssh exits non-zero, we want that to propogate. | 00:44 |
lifeless | Someone has been 'fixing' it I bet. | 00:44 |
lifeless | wgrant: ^ | 00:44 |
lifeless | that said | 00:45 |
lifeless | StevenK: Stopping lxc goes to | 00:45 |
lifeless | &2 | 00:45 |
StevenK | I wonder if the lxc from precise is not right and I need a different version? | 00:45 |
lifeless | StevenK: we need to figure out why its ending up in stdout | 00:45 |
lifeless | StevenK: I wonder, perhaps jenkins is insane | 00:46 |
StevenK | wallyworld: How did you build the 0.19.7 tarball? | 00:47 |
wallyworld | python setup sdist | 00:48 |
wallyworld | setup.py even | 00:48 |
StevenK | wallyworld: So did I, but I have 3 more files than you | 00:48 |
wallyworld | in the dist directory? | 00:48 |
StevenK | No, in the tarball | 00:48 |
wallyworld | which ones? | 00:48 |
StevenK | +/setup.cfg | 00:49 |
StevenK | +/src/lazr.restful.egg-info/not-zip-safe | 00:49 |
StevenK | +/src/lazr.restful.egg-info/PKG-INFO | 00:49 |
wgrant | lifeless: Heat came back? | 00:49 |
wallyworld | i should have had the pkg-info in mine | 00:49 |
wallyworld | not sure about the others | 00:49 |
wallyworld | i don't know why there's a difference | 00:50 |
lifeless | wgrant: see the bug bryceh filed | 00:50 |
wgrant | lifeless: Ah yeah, that | 00:50 |
wgrant | lifeless: That's why I'm interested in locks :) | 00:50 |
lifeless | indeed | 00:50 |
wgrant | lifeless: However | 00:51 |
wgrant | lifeless: Note the time there. | 00:51 |
wgrant | lifeless: I don't think this particular case is mostly DB locks, but I don't know for sure which is why I wanted to get reports rolling :) | 00:52 |
wgrant | I suspect this is just Python | 00:52 |
wgrant | As the statement timeout should mean the query times out in well under 23 seconds... | 00:52 |
lifeless | wgrant: query times are not checked during lock waits | 00:53 |
lifeless | or weren't, its possible that 9.1 fixed that. | 00:53 |
wgrant | lifeless: Are so. | 00:53 |
wgrant | In 9.1 they definitely are | 00:53 |
lifeless | ok, so thats nice. | 00:53 |
wgrant | I'm pretty sure they were in 8.4 too | 00:53 |
lifeless | 24 seconds is slow for python, and there is no data to be returning there. | 00:54 |
wgrant | But I don't remember for sure | 00:54 |
wgrant | By "Python" I mean "someone else probably had a lock" | 00:54 |
wgrant | Because as you say, there is no data. | 00:54 |
wgrant | It may be interesting to check postgres logs for the killed transaction. | 00:54 |
wgrant | See when it actually happened. | 00:54 |
wgrant | We log errors, right? | 00:54 |
lifeless | pretty sure | 00:55 |
lifeless | StevenK: jml: jelmer: so the testrepository you have is old, the testing cabal builds are failing. | 00:55 |
StevenK | lifeless: Ah | 00:56 |
lifeless | thats not the cause of the issue AFAICT, but its a nuisance. | 00:56 |
StevenK | wallyworld: How did you add the 0.19.7 milestone when https://launchpad.net/lazr.restful/trunk times out? | 00:56 |
wallyworld | StevenK: url hack | 00:56 |
wgrant | Huh, that times out? | 00:56 |
* wgrant guesses workitems | 00:56 | |
wgrant | No | 00:56 |
wgrant | Maintenance is not allowed to URL hack as the first step | 00:57 |
wgrant | Fix timeout instead :) | 00:57 |
StevenK | wgrant: OOPS-539a88c269cd05876823126d6dbb9e50 | 00:57 |
wallyworld | wgrant: i'm not going to fix a timeout to delay adding a release | 00:57 |
wallyworld | add release first | 00:57 |
wgrant | Yeah, it's the workitems regression :( | 00:57 |
wgrant | 23 150ms queries | 00:57 |
StevenK | wallyworld: What's the URL then? | 00:58 |
wgrant | +addmilestone | 00:58 |
wgrant | IIRC | 00:58 |
wallyworld | hmm. +addmilestone | 00:58 |
wallyworld | i think | 00:58 |
lifeless | so, working through this | 00:58 |
lifeless | lxc-start-ephemeral, assuming my copy == stevenk's copy, as he did not pastebinit :), outputs 'Stopping lxc' to &2, stderr | 00:59 |
lifeless | testrepository, which spawns the lp-setup-lxc-test processes uses stdout=subprocess.PIPE, stdin=subprocess.PIPE) | 00:59 |
wgrant | lifeless, StevenK: It's odd that we don't see this problem in the DC | 00:59 |
wgrant | Unless lp-setup-lxc-test does it | 00:59 |
wgrant | We use a puppeted version of that in the DC | 01:00 |
lifeless | so we'd expect stderr to be the jenkins provided stderr fd inherited from testrepository | 01:00 |
StevenK | echo "Stopping lxc" >&2 | 01:00 |
lifeless | however | 01:01 |
lifeless | we see all the ephemeral chatter in this file | 01:01 |
lifeless | Setting up ephemeral container... | 01:01 |
lifeless | Starting up the container... | 01:01 |
lifeless | ah, thats on stdout | 01:01 |
StevenK | That's from lp-setup-lxc-build I think | 01:02 |
wgrant | http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/70/steps/shell_9/logs/stdio | 01:03 |
wgrant | red == stderr | 01:03 |
wgrant | black == stdout | 01:03 |
lifeless | StevenK: its from lxc-start-ephemeral | 01:04 |
lifeless | StevenK: see 'setup_container' | 01:05 |
StevenK | Ah, indeed | 01:05 |
StevenK | wallyworld, lifeless: Can haz pypi rights for lazr.restful? | 01:06 |
wallyworld | sure | 01:06 |
lifeless | so in http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/70/steps/shell_9/logs/stdio, Stopping lxc is on stderr | 01:06 |
lifeless | as I pasted above, testrepository doesn't capture stderr at all, it does the default for subprocess | 01:06 |
wallyworld | StevenK: user name is SteveK? | 01:07 |
wallyworld | StevenK even | 01:07 |
wallyworld | ? | 01:07 |
StevenK | wallyworld: Yeah, stevenk | 01:07 |
wallyworld | StevenK: should be done | 01:08 |
wgrant | Hmmm | 01:09 |
wgrant | Postgres really really seems to hate BitmapOr | 01:09 |
wgrant | Even when the alternative is two seqscans. | 01:09 |
lifeless | a quick re-check of subprocess doesn't given any obvious bugs | 01:10 |
lifeless | StevenK: can I login on the executor, will need root access | 01:12 |
StevenK | lifeless: ubuntu@184.73.75.214 | 01:13 |
StevenK | I've fed it your lifeless-64 ssh key | 01:13 |
lifeless | waaaay but its busay | 01:14 |
lifeless | or | 01:14 |
lifeless | StevenK: is port 22 open in the sec group ? | 01:14 |
StevenK | I'm not sure if it's open to 0/0 | 01:15 |
lifeless | ssh: connect to host 184.73.75.214 port 22: Connection timed out | 01:15 |
StevenK | Right | 01:15 |
StevenK | lifeless: Should I add just your IP or a netblock? | 01:16 |
lifeless | shrug | 01:16 |
StevenK | lifeless: You should be good | 01:17 |
* StevenK drops lazr.restful-0.19.[2-5].tar.gz from the download-cache | 01:21 | |
StevenK | lifeless: Ah, nice. You're right for the mo? | 01:22 |
lifeless | jenkins, you mad bro | 01:23 |
lifeless | testr 12536 jenkins 1w FIFO 0,8 0t0 65754 pipe | 01:23 |
lifeless | testr 12536 jenkins 2w FIFO 0,8 0t0 65754 pipe | 01:23 |
StevenK | Oh | 01:24 |
lifeless | testr 12536 jenkins 0r FIFO 0,8 0t0 65753 pipe | 01:25 |
lifeless | testr 12536 jenkins 1w FIFO 0,8 0t0 65754 pipe | 01:26 |
lifeless | testr 12536 jenkins 2w FIFO 0,8 0t0 65754 pipe | 01:26 |
lifeless | testr 12536 jenkins 3u REG 202,1 6975862 32201 /home/jenkins/launchpad/lp-branches/workspace/devel/.testrepository/tmp2CwhL_ | 01:26 |
lifeless | testr 12536 jenkins 5r FIFO 0,8 0t0 69881 pipe | 01:26 |
lifeless | thats the full set | 01:26 |
lifeless | testr's in, out, err, the temp file its stream its inputs too, 4 would have been the stdin for the child process, which is closes, and 5 is where its reading hte child output from | 01:26 |
StevenK | Hah, so it's Jenkins bug/misfeature | 01:26 |
lifeless | well | 01:27 |
lifeless | we have correlation | 01:27 |
lifeless | not causation | 01:27 |
lifeless | StevenK: so, we run sudo /usr/local/bin/lp-setup-lxc-test lptests /home/jenkins/.ssh/id_rsa $IDOPTION $LISTOPT ? | 01:28 |
lifeless | what does the DC run ? | 01:28 |
StevenK | sudo /usr/local/sbin/lp-setup-lxc-test I think | 01:29 |
StevenK | I think the only difference is the DC script has stuff hardcoded | 01:29 |
lifeless | fd's for lp setup look ok | 01:29 |
lifeless | lp-setup- 12540 root 0r FIFO 0,8 0t0 69880 pipe | 01:29 |
lifeless | lp-setup- 12540 root 1w FIFO 0,8 0t0 69881 pipe | 01:29 |
lifeless | lp-setup- 12540 root 2w FIFO 0,8 0t0 65754 pipe | 01:29 |
lifeless | err is as we expect, out is the pipe | 01:30 |
lifeless | lxc-start 12541 root 0r FIFO 0,8 0t0 69880 pipe | 01:30 |
lifeless | lxc-start 12541 root 1w FIFO 0,8 0t0 69881 pipe | 01:30 |
lifeless | lxc-start 12541 root 2w FIFO 0,8 0t0 65754 pipe | 01:30 |
lifeless | that looks fine too | 01:30 |
lifeless | and finally ssh | 01:31 |
lifeless | ssh 13831 root 0r FIFO 0,8 0t0 69880 pipe | 01:31 |
lifeless | ssh 13831 root 1w FIFO 0,8 0t0 69881 pipe | 01:31 |
lifeless | ssh 13831 root 2w FIFO 0,8 0t0 65754 pipe | 01:31 |
lifeless | so, theres nothing down through to the ssh layer that can explain stderr output ending up on stdout for testr | 01:31 |
lifeless | other stderr output *does* end up in the jenkins console | 01:32 |
lifeless | the added address to .ssh/known-hosts specifically | 01:32 |
lifeless | https://lpci.wedontsleep.org/job/devel/1102/console | 01:32 |
lifeless | Warning: Permanently added '10.0.3.39' (RSA) to the list of known hosts. | 01:32 |
StevenK | wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/preview-diff-none-type/+merge/126601 | 01:33 |
wallyworld | StevenK: perhaps the assertVoteReference could be _assertVoteReference? | 01:35 |
wallyworld | otherwise looks good | 01:35 |
lifeless | StevenK: kill the build? run again as concurrency 4 ? the stream written so far is intact. | 01:36 |
LPCIBot | Project devel build #1102: ABORTED in 2 hr 23 min: https://lpci.wedontsleep.org/job/devel/1102/ | 01:36 |
StevenK | lifeless: #1103 away | 01:37 |
StevenK | wallyworld: http://pastebin.ubuntu.com/1255162/ | 01:38 |
wallyworld | +1 | 01:39 |
StevenK | Bleh, my oops got pruned again | 01:44 |
StevenK | Clearly, I need to log into neem and make copies | 01:45 |
wgrant | lifeless: See also OOPSes like https://oops.canonical.com/oops/?oopsid=OOPS-c182f7f94210ede212ae5b835ef993c7 | 01:58 |
wgrant | lifeless: 12s query that couldn't be blocked by any normal lock | 01:58 |
wgrant | We have a general problem, and I suspect that Bryce's bug is mostly that general problem. | 01:58 |
lifeless | wgrant: sounds plausible | 02:15 |
lifeless | this is mystifying | 02:24 |
StevenK | What is? | 02:27 |
lifeless | I was having trouble finding any evidence of a problem | 02:27 |
lifeless | xvfb-run 8494 jenkins 0r FIFO 0,8 0t0 11112539 pipe | 02:27 |
lifeless | xvfb-run 8494 jenkins 1w FIFO 0,8 0t0 11112540 pipe | 02:27 |
lifeless | xvfb-run 8494 jenkins 2w FIFO 0,8 0t0 11112540 pipe | 02:27 |
StevenK | Ah | 02:27 |
lifeless | but - this is such evidence | 02:27 |
lifeless | xvfb-run has stderr and stdout linked | 02:28 |
lifeless | so we can corrupt a stream within the subprocess | 02:28 |
lifeless | across the ssh barrier | 02:28 |
StevenK | lifeless: You think this is jenkins level or lower? | 02:28 |
StevenK | I'm not sure if Jenkins does something like exec 2>&1 that it will hit processes it spawns | 02:29 |
lifeless | well, I'm still only at correlation stage | 02:30 |
lifeless | jenkins definitely is doing 2>&1 | 02:30 |
lifeless | but that shouldn't have any way of affecting children of children if they make their own pipes | 02:31 |
lifeless | which testr does for stdout | 02:31 |
lifeless | ssh normally preserves stderr as a separate fd | 02:33 |
lifeless | e.g. | 02:33 |
lifeless | ssh localhost 'echo true >&2' 2>/dev/null | 02:33 |
lifeless | ssh localhost 'echo true >&2' | 02:33 |
lifeless | compare the output | 02:34 |
lifeless | it knows whats stdout and stderr on the slave, and faithfully lines that up on the invoking side | 02:34 |
lifeless | so WTF would xvfb-run be seeing stderr=stdout | 02:34 |
lifeless | ssh -n -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/jenkins/.ssh/id_rsa jenkins@10.0.3.132 -- env -u LANG xvfb-run --error-file=/var/tmp/xvfb-errors.log --server-args='-screen 0 1024x768x24' -a /home/jenkins/launchpad/lp-branches/workspace/devel/bin/test -vvv --shuffle --subunit --load-list /home/jenkins/launchpad/lp-branches/workspace/devel/temp/tmp0bW4iD | 02:35 |
lifeless | is the command being run | 02:35 |
lifeless | ssh -n localhost 'env echo true >&2' 2>/dev/null | 02:36 |
lifeless | still good | 02:36 |
lifeless | I'll file a bug on the silly exit 0 / exit 1 stuff | 02:39 |
lifeless | https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1059943 | 02:43 |
_mup_ | Bug #1059943: lxc-start-ephemeral masks process exit code <lxc (Ubuntu):New> < https://launchpad.net/bugs/1059943 > | 02:43 |
wgrant | lifeless: I already filed and fixed that in quantal | 02:44 |
lifeless | wgrant: oh, can you dupe it appropriately please then ? | 02:45 |
wgrant | And I have a precise backport branch. | 02:45 |
lifeless | DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1 | 02:45 |
lifeless | ^ xvfb-run | 02:45 |
lifeless | StevenK: so, smoking gun. | 02:49 |
lifeless | StevenK: this may not explain all the corruption we're encountering, but I think it covers a pretty good chunk thereof | 02:50 |
lifeless | StevenK: are you running lucid or precise in the lxc containers? | 02:50 |
lifeless | StevenK: https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1059947 | 02:50 |
_mup_ | Bug #1059947: xvfb-run munges stdout and stderr together <xorg-server (Ubuntu):New> < https://launchpad.net/bugs/1059947 > | 02:50 |
lifeless | StevenK: it doesn't explain HTF 'Stopping lxc' got from lxc-start-ephemeral &2 into &1 | 02:52 |
wgrant | blink | 03:03 |
wgrant | Ah | 03:04 |
lifeless | wgrant: ? | 03:05 |
wgrant | MilestoneTag.specifications had me confused for a while | 03:05 |
wgrant | Why it was intersecting all the resulting specs | 03:05 |
wgrant | But it makes some sense now | 03:06 |
LPCIBot | Project devel build #1103: STILL FAILING in 1 hr 32 min: https://lpci.wedontsleep.org/job/devel/1103/ | 03:09 |
StevenK | lifeless: Whatever lp-setup does, which I guess is lucid? | 03:13 |
wgrant | Right, inside the containers is lucid | 03:14 |
StevenK | xvfb-run bug? :-( | 03:14 |
lifeless | StevenK: contributing factor at most | 03:15 |
lifeless | I've just edited the bug out of the root container and started another run | 03:15 |
StevenK | So buildbot doesn't xvfb-run, then? | 03:16 |
lifeless | timing bugs are terrible | 03:17 |
wgrant | buildbot does | 03:17 |
wgrant | But isn't the problematic stderr coming from *outside* xvfb-run? | 03:17 |
lifeless | wgrant: one case | 03:18 |
lifeless | which as I said | 03:18 |
lifeless | 15:52 < lifeless> StevenK: it doesn't explain HTF 'Stopping lxc' got from lxc-start-ephemeral &2 into &1 | 03:18 |
wgrant | Ah, right | 03:18 |
lifeless | but | 03:19 |
lifeless | it may be confused signals | 03:19 |
lifeless | see | 03:19 |
lifeless | https://lpci.wedontsleep.org/job/devel/1103/console | 03:19 |
lifeless | where we have test data in the jenkins console | 03:19 |
lifeless | that happens when tests are mangled at a lower layer | 03:19 |
lifeless | and subunit's parser decides its seeing junk rather than tests | 03:19 |
lifeless | e.g. the subprocess race/rhing | 03:20 |
lifeless | which we need to get a reproducable test case for | 03:20 |
lifeless | _StringException: lost connection during test 'lp/registry/javascript/tests/test_milestone_creation' | 03:20 |
lifeless | the end of https://lpci.wedontsleep.org/job/devel/1103/console | 03:20 |
lifeless | shows | 03:20 |
lifeless | test: lp.codehosting.pukill: 186: No such process | 03:20 |
lifeless | Stopping lxc | 03:20 |
lifeless | ller.tests.test_scheduler.TestPullerWireProtocol.test_methodDispatchWithArguments | 03:20 |
lifeless | which is, I think, testr's stdout intermingled with the shared stderr, and - *this is fine*. | 03:20 |
lifeless | Its expected given jenkins mangling of the two onto one stream | 03:21 |
lifeless | StevenK: you probably want two forked packages; lxc (for the lxc-start-ephemeral fix from wgrant) and a xvfb-run with fixed stderr handling | 03:22 |
wgrant | https://code.launchpad.net/~wgrant/ubuntu/precise/lxc/bug-1050351 | 03:23 |
lifeless | StevenK: so we'll see if thats any better, and if it isn't we'll see about making a reproducable test case. | 03:36 |
lifeless | for now, I'm going to context switch a bit | 03:37 |
wgrant | StevenK: Thoughts on bug #810716 and bug #690834? | 03:48 |
_mup_ | Bug #810716: Translations section builds have highest priority in copy builds <buildfarm> <Launchpad itself:Triaged> < https://launchpad.net/bugs/810716 > | 03:48 |
_mup_ | Bug #690834: Language packages in P3As are also scored 0 <lp-soyuz> <oem-services> <soyuz-build> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/690834 > | 03:48 |
wgrant | For context, if section == 'translations' then the score short-circuits to 0 | 03:48 |
wgrant | Ignoring any archive offset | 03:48 |
StevenK | Ywah | 03:48 |
wgrant | I suspect we just want to apply the archive offset to the 0 | 03:48 |
StevenK | I'll be hitting that code to | 03:48 |
wgrant | So they'll sit at the bottom of the archive | 03:48 |
wgrant | Oh? | 03:49 |
wgrant | Oh | 03:49 |
wgrant | I see you took the card | 03:49 |
wgrant | I was just collecting those three bugs to fix | 03:49 |
StevenK | Oh, you can do it if you want | 03:49 |
wgrant | If you've been busy with jenkins and haven't actually started... | 03:49 |
StevenK | I've created a branch and found the code, so 'not really' | 03:50 |
wgrant | Ah, well I just wrote a test, so I'm probably slightly ahead. | 03:50 |
StevenK | Nothing of consequence, so feel free to steal the card and I'll find another bug | 03:50 |
StevenK | Or re-package lxc | 03:50 |
wgrant | Thanks | 03:50 |
wgrant | Heh | 03:50 |
StevenK | Haha | 03:53 |
StevenK | I can see the edit page in bug 1059324, but the reporter can't | 03:53 |
_mup_ | Bug #1059324: Can't edit branch import <403> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1059324 > | 03:54 |
wgrant | StevenK: ~vcs-imports can see +edit-import, mortals can't. | 03:54 |
wgrant | It's likely that the link should just be hidden. | 03:54 |
wgrant | It may just be a matter of adding one line of decorator. | 03:55 |
StevenK | Right | 03:55 |
StevenK | @with_permission or so? | 03:55 |
wgrant | @enabled_with_permission, IIRC | 03:55 |
StevenK | Bleh, the qa-tagger hates multi-task bugs? :-( | 03:59 |
wgrant | Should work, normally. | 04:00 |
wgrant | What's the issue? | 04:00 |
StevenK | It stated orphaned for my rev | 04:01 |
StevenK | Maybe it insists on default_bugtask, but that would be silly | 04:01 |
wgrant | Hm | 04:01 |
wgrant | That's odd | 04:01 |
StevenK | wgrant: Hmm, lib/lp/code/browser/branch.py edit_import checks perms already. Is it checking the wrong thing? | 04:03 |
wgrant | StevenK: That's a very good point | 04:04 |
wgrant | Although I wouldn't trust that function to not be insane | 04:05 |
wgrant | Given that it sets enabled = True a line earlier | 04:05 |
StevenK | Haha, I wonder if that's the fix | 04:06 |
wgrant | It's just a normal local | 04:06 |
wgrant | So I doubt it | 04:06 |
StevenK | 6 Unauthorized: (<CodeImport for ~rudloff/chaton-cms/0.1>, 'updateFromData', 'launchpad.Edit') | 04:09 |
StevenK | That seems more like he can view the page and the submit is blowing up | 04:10 |
wgrant | Right, that means the user doesn't hold launchpad.Edit on the CodeImport | 04:10 |
wgrant | What's the view permission? | 04:10 |
StevenK | launchpad.Edit on IBranch, looks like | 04:12 |
wgrant | Ah | 04:13 |
wgrant | That doesn't explain why there's a link, though | 04:13 |
wgrant | Perhaps there is no link | 04:13 |
wgrant | And the user URL-hacked. | 04:13 |
wgrant | Try to reproduce locally or on qastaging? | 04:13 |
StevenK | How did he POST, then? | 04:13 |
wgrant | The view is Branch:+edit-import, so the permission is wrong | 04:13 |
wgrant | It'll load for the branch owner | 04:14 |
StevenK | We have a few OOPSes, so we should be able to work it out | 04:14 |
wgrant | But the edit_import menu item correctly checks against branch.codeimport | 04:14 |
StevenK | OOPS-1d2dccea10b7b7c5f4bc11cf0eaa9ebf | 04:14 |
wgrant | The OOPS won't say | 04:14 |
wgrant | It'll show the referer as +edit-import either way | 04:14 |
StevenK | Right, so the ZCML is wrong | 04:15 |
wgrant | And can't really be right. | 04:15 |
StevenK | Not if we use for="...", right | 04:15 |
StevenK | wgrant: I'm not clear how to create a branch import | 04:19 |
wgrant | +new-import | 04:21 |
wgrant | http://code.qastaging.launchpad.net/launchpad/+new-import | 04:21 |
StevenK | wgrant: Right, so no link. URL hacking as the branch owner shows the form and submitting gives Not allowed here | 04:25 |
StevenK | Maybe we want to redirect away in initialize() | 04:26 |
StevenK | "You are not permitted to edit the import for this branch" etc | 04:26 |
wgrant | Or raise a Forbidden or so | 04:27 |
wgrant | No point being pretty if you have to URL-hack to get there. | 04:27 |
StevenK | Indeed | 04:27 |
StevenK | It already does for anonymous, indeed | 04:28 |
wgrant | If you waste more than two lines of code on this, you're probably doing it wrong. | 04:28 |
wgrant | StevenK: I think I'll take this opportunity to finally eliminate time-based scoring. | 04:30 |
wgrant | Any objections? | 04:30 |
* wgrant murders. | 04:31 | |
StevenK | wgrant: We have time based scoring? | 04:32 |
StevenK | If it will complete quickly, score it higher? | 04:33 |
wgrant | No | 04:33 |
wgrant | If it's been in the queue for 5 minutes, give it a score bump of 5 | 04:33 |
wgrant | an hour is a bump of 20 | 04:33 |
StevenK | Oh, right | 04:33 |
wgrant | This ignores the fact that the tiebreaker is id, which is age anyway | 04:33 |
wgrant | And we don't run queue-builder any more | 04:33 |
StevenK | wgrant: Yes, murder it. | 04:33 |
wgrant | So the score is never calculated except when the build is 0 seconds old. | 04:33 |
wgrant | Thanks | 04:33 |
StevenK | Can we destroy queue-builder too? | 04:34 |
wgrant | I already got rid of most of it a while ago | 04:34 |
StevenK | wgrant: I'm at -4 with this branch with some cleanups and no test | 04:34 |
wgrant | :) | 04:34 |
wgrant | We have a lot of crap in our code. | 04:34 |
StevenK | Not sure if raise Forbidden is okay | 04:35 |
wgrant | Unauthorized may be more common, I think | 04:35 |
wgrant | But I forget | 04:35 |
lifeless | StevenK: ok, so the xvfb-run isn't the root cause, and whether it contributed is an open q | 04:38 |
lifeless | StevenK: tomorrow I shall poke further, the artifacts gathered should be sufficient | 04:39 |
StevenK | Ah | 04:39 |
lifeless | https://lpci.wedontsleep.org/job/devel/1104/console | 04:39 |
lifeless | shows subunit | 04:39 |
lifeless | so its escaped the parser again | 04:39 |
StevenK | lifeless: So I can kill the executor ec2 when the current one fails? | 04:40 |
StevenK | wgrant: Do you think a test is worth it? | 04:41 |
wgrant | StevenK: If you've deleted enough to still leave you net-negative :) | 04:42 |
LPCIBot | Project devel build #1104: STILL FAILING in 1 hr 28 min: https://lpci.wedontsleep.org/job/devel/1104/ | 04:44 |
wgrant | StevenK, wallyworld_: https://code.launchpad.net/~wgrant/launchpad/bug-1058310/+merge/127414 https://code.launchpad.net/~wgrant/launchpad/bug-1056617/+merge/127412 | 04:53 |
czajkowski | StevenK: wgrant kicking ass on bugs I see :) very nice | 04:56 |
wallyworld_ | wgrant: r=me on 2nd one | 04:59 |
wgrant | Today should take us below 300 criticals :) | 05:00 |
wgrant | wallyworld_: Thanks | 05:00 |
StevenK | wgrant: And r=me on the first | 05:00 |
wgrant | Did we implement ParallelReviewing while I wasn't watching? | 05:00 |
czajkowski | wallyworld_: ah and you | 05:00 |
czajkowski | tab complete and me are not working at this hour | 05:01 |
wallyworld_ | czajkowski: well, today i'm removing an old feature flag for dynamic bug listings which was released a while back. not sure if that's critical, but it needs to be done to keep the code clean and maintainable | 05:01 |
wallyworld_ | tedious work | 05:02 |
czajkowski | which is needed | 05:02 |
wgrant | That's probably our biggest single bit of tech debt :) | 05:02 |
=== matsubara is now known as matsubara-afk | ||
wallyworld_ | i'm almost finished. a few more doc tests to go | 05:02 |
wallyworld_ | maybe a bit optimistic to say "almost". "substantially" is better | 05:03 |
* wgrant plays buildbot roulette | 05:03 | |
czajkowski | ah hat reminds me I need to find the bug on signing the CoC to make it easier to see if we can get some sort of check box implemented :D | 05:03 |
wgrant | Heh | 05:03 |
wgrant | StevenK: You have a couple of ancient branches on https://code.launchpad.net/launchpad/+activereviews | 05:10 |
StevenK | No I don't. :-) | 05:11 |
wgrant | Thanks. | 05:11 |
StevenK | 5 files changed, 24 insertions(+), 32 deletions(-) | 05:11 |
StevenK | I think that's far enough | 05:11 |
wallyworld_ | StevenK: wgrant: have you noticed that the branch badges on the dynamic bug listing are no longer anchors linking to the associated branch? | 05:25 |
StevenK | wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/no-edit-import-for-you/+merge/127417 | 05:25 |
wallyworld_ | StevenK: r=me | 05:34 |
StevenK | wgrant: I wonder if bug 1022334 can be closed due to YUI 3.5.1 | 06:19 |
wgrant | StevenK: No | 06:20 |
wgrant | StevenK: The bug *listing* back behaviour may be fixed, but the bug reporting one probably isn't | 06:20 |
StevenK | I thought zope handled at this stuff, TBH | 06:21 |
wgrant | Our JavaScript is not Zope | 06:21 |
StevenK | Oh, it's a JS bug? | 06:21 |
wgrant | Well, +filebug is very JSey, so it's probably not unrelated. | 06:22 |
StevenK | Yeah | 06:22 |
wgrant | Or it could be that the view is too custom and doesn't let initial_values through properly. | 06:22 |
StevenK | wgrant: In https://oops.canonical.com/oops/?oopsid=OOPS-91e85c5a76936c13c1993df1985491e5 I guess the branch is None? | 06:29 |
wgrant | StevenK: Potentially because the requesting user can't see it | 06:30 |
wgrant | But it looks like it. | 06:30 |
wgrant | Yeah | 06:31 |
wgrant | StevenK: Yeah, it happens whenever the development focus is an import that isn't visible. | 06:32 |
StevenK | Heh, right | 06:32 |
StevenK | wgrant: So we should skip that bit? | 06:32 |
wgrant | That may be appropriate. | 06:32 |
StevenK | wgrant: It seems I broke buildbot. But it doesn't look to be my fault. | 06:33 |
wgrant | It just has a personal vendetta against you | 06:33 |
wgrant | Ah, yeah | 06:33 |
wgrant | We see that one sometimes :( | 06:33 |
StevenK | wgrant: QA! :-P | 06:35 |
wgrant | Did that like 90 seconds ago | 06:35 |
StevenK | lib/lp/code/templates/product-branch-summary.pt makes me sad | 06:39 |
StevenK | wgrant: I should force away? | 06:57 |
wgrant | StevenK: Yep | 06:57 |
adeuring | good morning | 07:02 |
wgrant | StevenK: tsk tsk | 07:02 |
StevenK | wgrant: Hm? | 07:03 |
wgrant | StevenK: You're not very good at deleting views :) | 07:04 |
StevenK | I was supposed to delete one? | 07:04 |
wgrant | You deleted +daily-builds | 07:04 |
StevenK | wgrant: Sure, but? | 07:05 |
wgrant | You missed 300 lines of code that was used only by it :) | 07:05 |
StevenK | Have you binned them, or shall I have at it? | 07:05 |
wgrant | They're miiiiiiine | 07:05 |
* StevenK makes a note to talk to wgrant's mother about sharing | 07:06 | |
wgrant | We already finished sharing last month :) | 07:06 |
bigjools | s/sharing/closing legs/ | 07:06 |
wgrant | No need for any more. | 07:06 |
wgrant | Ah, make that 400 lines. | 07:06 |
StevenK | wgrant: Meh, I'm -3500 and you're -1235 :-P | 07:07 |
wgrant | I won't be dictated by fact-checkers. | 07:09 |
wgrant | Also, http://webnumbr.com/launchpad-critical-bugs | 07:09 |
wgrant | We're 3 bugs away from where we were at the start of December. | 07:10 |
wgrant | Getting there. | 07:10 |
wgrant | StevenK: Your new test seems to be bad | 07:35 |
wgrant | If you're in AppServerLayer, you're probably trying to catch the exception over HTTP | 07:36 |
wgrant | Which isn't likely to be a gigantic success. | 07:36 |
=== mmrazik is now known as mmrazik|otp | ||
StevenK | Hmmm, I thought it was passing locally | 07:40 |
StevenK | Let me roll it back | 07:41 |
wgrant | stub: Huh | 07:48 |
wgrant | "specification_product_name_uniq" UNIQUE CONSTRAINT, btree (name, product) | 07:48 |
wgrant | Why is it that way around? :/ | 07:48 |
stub | lookup by name? | 07:48 |
wgrant | Possibly, but it seems pretty odd | 07:49 |
wgrant | I prefer the theory that someone tried to make it as useless as possible. | 07:49 |
stub | it is the only suitable lookup by name, which would be for traversal | 07:49 |
wgrant | Right, but the URL is /product/+spec/foo | 07:49 |
wgrant | I don't know much that looks up by unqualified name. | 07:49 |
stub | yeah, so it can use that index. | 07:50 |
wgrant | Right, but the distribution index is around the right way :) | 07:50 |
wgrant | The product one is just inverted for probably no reason | 07:50 |
stub | it would have been inverted to look up by unqualified name. I see that is probably pointless, yes. | 07:50 |
wgrant | I'm not sure productseries and distroseries indices are useful, but I guess we might as well | 07:51 |
wgrant | The table isn't exactly high-write... | 07:51 |
* wgrant adds three | 07:51 | |
wgrant | I didn't even consider that product might have been unindexed | 07:52 |
wgrant | Since that's basically the primary thing we've queried on for 7 years... | 07:52 |
wgrant | w | 07:52 |
wgrant | stub: I've added new indices (including UNIQUE (product, name) so we can kill off the other one), though the diff is taking its time to update | 08:03 |
wgrant | It's possible we want one on just name, but I don't think so | 08:03 |
stub | yup | 08:05 |
stub | unqualified doesn't make sense | 08:05 |
wgrant | It certainly doesn't make sense. | 08:05 |
wgrant | But it still might happen | 08:05 |
wgrant | Because a lot of blueprints is a little behind its sense quota. | 08:06 |
wgrant | But let's go without | 08:06 |
=== frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban | Firefighting: - | Critical bugs: ~300 | ||
wgrant | stub: If that looks good I'll land it | 08:10 |
wgrant | frankban: Hi. I'm trying to clean up the review queue, and you have three branches on https://code.launchpad.net/launchpad/+activereviews. I think they're all for stuff that's now in lp:lpsetup, so they can be rejected from LP, right? | 08:11 |
stub | wgrant: yeah, that looks good. It might make more sense for these to be partial indexes, but I don't think we know enough yet to be sure. | 08:12 |
frankban | wgrant: yes | 08:12 |
wgrant | stub: Given the data volumes and rates we're looking at, it won't matter for another few years :) | 08:13 |
wgrant | frankban: Thanks | 08:17 |
wgrant | frankban: Can you set https://code.launchpad.net/~frankban/launchpad/setuplxc-remove-sleep/+merge/96422 from Approved to Rejected to get it off the list? It's proposed to a branch that I don't have review privs on. | 08:18 |
frankban | wgrant: thank you! and done | 08:21 |
wgrant | Great, thanks. | 08:22 |
=== mmrazik|otp is now known as mmrazik | ||
lifeless | stub: should francis be the 2nd db reviewer going forward, or someone else ? | 08:45 |
czajkowski | morning folks | 08:54 |
stub | lifeless: Depends if Francis is comfortable giving the go ahead for patches. One feature of the reviews is to avoid hard to recover failures and dataloss. | 08:57 |
lifeless | stub: Indeed. | 08:59 |
lifeless | gnight | 09:09 |
wgrant | Night lifeless. | 09:09 |
=== mmrazik is now known as mmrazik|lunch | ||
=== mmrazik|lunch is now known as mmrazik | ||
=== matsubara-afk is now known as matsubara | ||
adeuring | frankban: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/product-sharing-sec-adapter/+merge/127473 ? | 12:36 |
frankban | adeuring: sure | 12:38 |
adeuring | thanks! | 12:38 |
wgrant | adeuring: A couple of notes: | 12:38 |
wgrant | Product.information_type exists on production now | 12:38 |
wgrant | Though not on qastaging; you might be able to convince ops or stub to fix that | 12:39 |
wgrant | And also, LimitedView is probably not what you want, unless you've discussed that with Curtis/Robert already | 12:39 |
adeuring | wgrant: makes sense | 12:39 |
wgrant | View is what makes sense here | 12:39 |
adeuring | wgrant: i discussed this with sinzui for Ispecification | 12:39 |
adeuring | there were some pitfalls with lp.View | 12:40 |
wgrant | adeuring: LimitedView is intended to give people enough to know that something exists, but basically nothing else. | 12:40 |
sinzui | correct | 12:40 |
wgrant | Oh, sinzui's here | 12:40 |
czajkowski | sinzui: would you be free later on for a call re maintenance, have stand up now. | 12:40 |
sinzui | LimitedView reveals the spec name, displayname, and project, .public, and .information_type. They do not have icons that need to be revealed | 12:41 |
sinzui | yes | 12:41 |
sinzui | yes czajkowski | 12:41 |
wgrant | It's not clear that LimitedView is necessary for products at this point | 12:42 |
wgrant | But even if it is, it would probably expose name, displayname, owner, and that's about it | 12:43 |
czajkowski | sinzui: thanks gew Q's need to be handed over and want to go through them first | 12:43 |
sinzui | wgrant, subscribing someone to a bug, blueprint, or branch will require limited view to the product and possibly series | 12:43 |
wgrant | sinzui: I think those are likely to give full View to the product. | 12:44 |
wgrant | But that may be yet to be established. | 12:44 |
sinzui | wgrant, adeuring, without limited view, you cannot traverse to get to the artefact. | 12:44 |
wgrant | Possession of View implies possession of LimitedView by default | 12:44 |
sinzui | wgrant, We are not here to establish that, Deryck's squad are. We know that traversal requires access to .name to get the a subscribed private arttfact | 12:45 |
wgrant | Sure | 12:45 |
wgrant | But that doesn't mean that 90% of the attributes that were zope.Public should be launchpad.LimitedView now | 12:46 |
wgrant | 89% should be launchpad.View, 1% should be launchpad.LimitedView | 12:46 |
wgrant | Otherwise LimitedView is meaningless. | 12:46 |
sinzui | wgrant, I agree. I think 7 attributes are about the right number | 12:46 |
wgrant | Right :) | 12:46 |
sinzui | limitedview is intended to convey enough information to allow someone to identify the thing they are looking at. It deters spying and deceit when a private something is in public relationship | 12:48 |
sinzui | adeuring, OEM projects are only willing to reveal their name, displayname, icon, logo, .public, .information_type, and maybe .active to contractors that they subscribe to bugs and branches | 12:51 |
adeuring | sinzui: ok; just working replacining lp.LimitedView with lp.View for the current branch | 12:52 |
sinzui | okay. | 12:52 |
frankban | adeuring: should I wait for you to push these changes? | 12:57 |
adeuring | frankban: make sense, I think | 12:57 |
frankban | adeuring: ok | 12:57 |
adeuring | frankban: i renamed the permission to lp.view now. I'll add lp.limitedview in a later branch for the properties that need it. Testing both permissions would let the diff size explode ;) | 13:01 |
frankban | adeuring: ok | 13:01 |
deryck | abentley, adeuring -- always have troubles getting into the hangout the first time, still trying.... | 13:32 |
abentley | deryck: I'm confused about where we stand wrt NULLs in Product.information_type. | 14:30 |
deryck | abentley, we need a garbo job to fill in values for existing products. nulls are currently allowed. | 14:31 |
abentley | deryck: I think we don't have a card for the garbo job. And we also need to fix the sampledata. | 14:42 |
deryck | abentley, I lumped the garbo job into your current card back when I created it. And I can add a card now for sampledata. | 14:43 |
deryck | abentley, and I don't mind if you split off that garbo stuff into a new card, too. | 14:43 |
abentley | deryck: I read my card as stuff to do "after information_type + garbo job for current values" | 14:44 |
deryck | abentley, yeah, I probably phrased it badly. :) | 14:45 |
abentley | deryck: I see what you meant now. | 14:45 |
abentley | deryck: Do you know whether sampledata updates can land in stable? ISTM there would be no value to landing them in db-stable. | 14:58 |
deryck | abentley, I believe they can indeed land in stable. | 14:58 |
stub | Nothing except db patches lands on db-devel/db-stable now | 15:23 |
stub | Occasionally some tests to go with db patch, but that is really rare. | 15:23 |
abentley | stub: Thanks. | 15:25 |
abentley | stub: But if a db patch requires sampledata changes, we should still land both as one branch, right? | 15:26 |
stub | yeah | 15:27 |
stub | The bare minimum required to keep the tests passing and the build working, or if your db patch contains stored procedures tests to prove the code works. | 15:28 |
deryck | abentley, what was the feature flag you used for the new project UI? | 15:30 |
abentley | deryck: 'disclosure.private_projects.enabled' | 15:31 |
deryck | abentley, thanks | 15:31 |
xnox | what is initial LOC credit for somebody who never committed to lp? | 15:53 |
xnox | -100, 0, 100 ? | 15:54 |
sinzui | 0 | 15:54 |
adeuring | frankban: if you are still around, could you do another review? https://code.launchpad.net/~adeuring/launchpad/correct-permission-check-for-iproduct/+merge/127518 | 15:55 |
frankban | adeuring: on a call now, but I will take a look in 30 or so... | 15:56 |
adeuring | frankban: thanks! (this MP is a bit smaller, BTW) | 15:57 |
deryck | abentley, adeuring, just to confirm, we do still need PROJECTGROUP/+newproject updated to match projects/+new | 16:01 |
deryck | so I'll leave the card on the board, obviously. | 16:01 |
abentley | deryck: roger. | 16:01 |
adeuring | ack | 16:01 |
=== frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300 | ||
deryck | abentley, adeuring -- qastaging is updated for Product.information_type now. | 16:44 |
adeuring | deryck: cool | 16:47 |
xnox | sinzui: ok, thanks. | 16:50 |
sinzui | xnox, do you need help finding cruft to remove? | 16:50 |
xnox | sinzui: well there is a link to bug reports that are trianged with the cruft-removal-bug tag ;-) so I'm good. | 16:51 |
sinzui | okay. Ping me if you need help. | 16:51 |
xnox | trying to pick something easy, as I haven't done lp development yet, but there are a couple bugs that hinder my productivity ;-) =))))) | 16:51 |
xnox | but are "low" priority for now. | 16:52 |
abentley | deryck: There's no OCR. Could you please review https://code.launchpad.net/~abentley/launchpad/model-product-info-type/+merge/127558 ? | 18:07 |
deryck | abentley, I will shortly. Just grabbing me a bite to eat. | 18:32 |
abentley | deryck: Actually, I missed a spot. Fixing now. | 18:33 |
abentley | deryck: ready for review again: https://code.launchpad.net/~abentley/launchpad/model-product-info-type/+merge/127558 | 19:15 |
deryck | abentley, ack | 19:21 |
abentley | deryck: Garbo job was really easy with your suggestion: https://code.launchpad.net/~abentley/launchpad/product-info-type-garbo/+merge/127579 | 19:31 |
deryck | oh nice | 19:33 |
deryck | abentley, r=me for both. | 19:36 |
abentley | deryck: thanks. | 19:37 |
deryck | np | 19:38 |
abentley | deryck: How do we decide whether EMBARGOED/PROPRIETARY are reasonable information_types for a given Product? Do we look for a commercial subscription or something? | 19:43 |
deryck | abentley, yes, you have to have a commercial subscription to use them. | 19:43 |
abentley | deryck: | 19:47 |
abentley | deryck: I'm going to restrict it in the model for now, and we can add db constraints afterwards. | 19:47 |
deryck | abentley, sounds good. | 19:47 |
* deryck goes AFK for a little bit | 19:52 | |
StevenK | lifeless: Moar Jenkins? | 22:30 |
wallyworld_ | wgrant: StevenK: sinzui: you seen bug 1055766? | 22:46 |
_mup_ | Bug #1055766: grep -R doesn't automatically search amazon <apport-bug> <i386> <oneiric> <Ayatana Design:Invalid> <command-not-found (Ubuntu):Invalid> <gnome-terminal (Ubuntu):Invalid> < https://launchpad.net/bugs/1055766 > | 22:46 |
StevenK | wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/no-edit-import-for-you-redux/+merge/127613 | 23:29 |
=== Ursinha is now known as Jorjao | ||
wallyworld_ | StevenK: the test comment should be something like "Unauthorised users are forbidden....." since it's not just the branch owner who is not allowed | 23:31 |
=== Jorjao is now known as Ursinha | ||
StevenK | wallyworld_: http://pastebin.ubuntu.com/1257063/ | 23:32 |
wgrant | StevenK: Given lifeless is probably terribly busy, might be better to throw broken Jenkinses at me sometimes | 23:32 |
lifeless | hi | 23:32 |
lifeless | I'm going to be poking it | 23:32 |
wallyworld_ | r=me | 23:32 |
lifeless | StevenK: can you setup an executor ? | 23:32 |
StevenK | lifeless: I thought you only needed the build artifacts, but sure. | 23:33 |
lifeless | StevenK: reproducability is key, until I have a core recipe its an open question what will be needed | 23:34 |
lifeless | running just http://paste.ubuntu.com/1257072/ should give us a fail single-threaded. | 23:39 |
lifeless | StevenK: so what I want is an executor, and jenkins told not to run anything | 23:39 |
lifeless | StevenK: run one build, to seed it and recor da failure | 23:40 |
lifeless | and then I'll poke around as jenkins and reproduce etc | 23:40 |
lifeless | StevenK: so I want ssh, root access etc | 23:40 |
StevenK | lifeless: Said executor is up and in the middle of lp-setup | 23:40 |
StevenK | lifeless: Right, so I should add it to Jenkins and kick off a devel build too? | 23:41 |
sinzui | StevenK, I pulled the latests critical data. The chart is updated | 23:41 |
lifeless | StevenK: please, and we can configure jenkins to only do manual builds | 23:42 |
StevenK | lifeless: I already have | 23:42 |
lifeless | coolio | 23:42 |
* StevenK waits for Firefox to choke down sinzui's JS | 23:43 | |
lifeless | StevenK: will you prep a fixed xvfb for the lp PPA | 23:44 |
sinzui | Not the js, it is 5 megs of bug data | 23:44 |
StevenK | I can, but as you say, it's no smoking gun | 23:44 |
lifeless | StevenK: it a necessary fix for reliability | 23:44 |
StevenK | For lucid, I guess | 23:45 |
lifeless | StevenK: any stderr IO from anything in the test suite or under it can corrupt a stream w/out that fixed. | 23:45 |
lifeless | StevenK: for precise too | 23:45 |
StevenK | I'm not sure if it's inside the container or outside that is the important one | 23:45 |
lifeless | inside | 23:45 |
lifeless | xvfb isn't present outside the container | 23:45 |
* StevenK stares at the xorg source package | 23:49 | |
lifeless | xorg-server ? | 23:49 |
StevenK | Bleh | 23:50 |
StevenK | lifeless: http://pastebin.ubuntu.com/1257092/ | 23:55 |
* StevenK stabs Contact this team's admins | 23:56 | |
lifeless | StevenK: looks good to me | 23:57 |
StevenK | lifeless: If you have no complaints about the version, I'll upload it | 23:57 |
lifeless | StevenK: seems fine to me | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!