| === bschaefer_ is now known as bschaefer | ||
| === cpaelzer_ is now known as cpaelzer | ||
| pitti | Good morning | 05:03 |
|---|---|---|
| dholbach | good morning | 06:36 |
| dpm | seb128, wgrant, morning. Eventually, the manual .po file uploads for evolution got imported: https://translations.launchpad.net/ubuntu/wily/+source/evolution/+imports | 06:45 |
| seb128 | dpm, hey, good to know | 06:46 |
| dpm | there is a caveat, though | 06:46 |
| dpm | after long waiting and seeing that the .po files did not get imported automatically after the template had, I had to explicitly choose the template they belonged to | 06:47 |
| dpm | then they got imported | 06:47 |
| dpm | perhaps it was me being impatient and they would have been auto-imported, but after a few hours waiting, I wasn't sure | 06:47 |
| TJ- | Has anyone seen this package-hold for aptdaemon. I'm not making sense of it since there's nothing in /var/lib/dpkg/status that shows the version dependency mentioned: "python3-aptdaemon : Depends: gir1.2-packagekitglib-1.0 (< 0.9) but 0.9.5-0ubuntu1 is to be installed" | 08:26 |
| TJ- | Ahhh bug 1496292 | 08:40 |
| ubottu | bug 1496292 in aptdaemon (Ubuntu) "/usr/sbin/aptd:AttributeError:/usr/sbin/aptd@39:main:__init__:__init__" [High,Fix released] https://launchpad.net/bugs/1496292 | 08:40 |
| seb128 | TJ-, you used wily-proposed and got a buggy version that got deleted later, you might want to sudo apt-get install python3-aptdaemon/wily | 08:45 |
| TJ- | Yeah, I figured it out once I found the older changelog entry | 08:50 |
| tjaalton | is there a way to narrow down the updates that hit wily around Sep 10th, because something triggers X crashes since then | 08:58 |
| TJ- | tjaalton: Other that /var/log/apt/history* ? | 09:01 |
| seb128 | tjaalton, wily-changes list? | 09:01 |
| tjaalton | TJ-: not mine, trying to figure out distro-wide what happened when | 09:01 |
| seb128 | tjaalton, which ones? do you have a bt? | 09:01 |
| tjaalton | seb128: well kinda, gives some sort of an idea | 09:02 |
| tjaalton | at least I can rule out some X stuff.. | 09:02 |
| tjaalton | dupes of https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1237904 | 09:02 |
| ubottu | Launchpad bug 1237904 in xorg-server (Ubuntu) "Xorg crashed with SIGABRT in OsAbort()" [Medium,Confirmed] | 09:02 |
| tjaalton | 1495049 and up | 09:03 |
| tjaalton | which was Sep 14th actually | 09:03 |
| seb128 | tjaalton, the first issue on e.u.c? | 09:04 |
| seb128 | that seems to be gdm trying to start a xserver when there is already one active | 09:04 |
| seb128 | they all have gdm options in their command line | 09:04 |
| tjaalton | ah, that would be a good hint | 09:05 |
| seb128 | I pinged darkxst several time about it | 09:05 |
| seb128 | installing gdm on my test laptop also bricks it | 09:05 |
| seb128 | e.g no dm is starting | 09:05 |
| seb128 | so I think it's gdm screwed up | 09:06 |
| seb128 | tjaalton, the xorg logs all have | 09:07 |
| seb128 | [ 451.402] _XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed | 09:07 |
| seb128 | [ 451.402] _XSERVTransMakeAllCOTSServerListeners: server already running | 09:07 |
| seb128 | [ 451.403] | 09:07 |
| tjaalton | good | 09:07 |
| seb128 | btw the xorg hook seems grumpy | 09:07 |
| seb128 | https://launchpadlibrarian.net/221676859/HookError_source_xorg_server.txt | 09:07 |
| seb128 | File "/usr/lib/python3/dist-packages/problem_report.py", line 627, in __setitem__ | 09:07 |
| seb128 | assert k.replace('.', '').replace('-', '').replace('_', '').isalnum() | 09:07 |
| seb128 | AssertionError | 09:07 |
| seb128 | tseliot, ^ | 09:08 |
| tseliot | seb128: yay, something else failed. I'll look into that | 09:16 |
| seb128 | tseliot, thanks | 09:16 |
| tjaalton | gdm didn't change since Aug 7th | 09:20 |
| tjaalton | but it's some other change that triggers the bug then | 09:20 |
| === dupondje_ is now known as dupondje | ||
| TJ- | tjaalton: looking at the DpkgLog attached to 1495049 shows the dates/packages recently updated; nothing entirely obvious, but libc-bin, dbus, systemd might be suspect | 09:33 |
| darkxst | seb128, I still havent been able to reproduce on either of my systems | 09:37 |
| seb128 | darkxst, seeing the number of report you must have some users who can? | 09:37 |
| darkxst | seb128, I will ask around | 09:43 |
| seb128 | thanks | 09:44 |
| darkxst | seb128, do you have the e.u.c link handy? | 09:45 |
| seb128 | darkxst, https://errors.ubuntu.com/problem/27941445d596fb71be692b9008c0847d2ce6c428 | 09:47 |
| darkxst | emailed to QA team | 09:50 |
| tjaalton | thanks | 09:53 |
| TJ- | The common issue from the Xorg logs is the first session starts on VT7, fails, and the replacement starts on VT2 | 09:54 |
| seb128 | darkxst, tjaalton, ^ | 09:56 |
| darkxst | TJ-, gdm should start on vt7 | 09:56 |
| darkxst | the user session should start on vt2 | 09:56 |
| TJ- | darkxst: but should the process on VT7 remain running? | 09:59 |
| darkxst | TJ-, yes | 10:01 |
| darkxst | and actually gdm should be running on wayland | 10:04 |
| darkxst | for the FOSS drivers atleast | 10:04 |
| darkxst | maybe it isnt since we don't seed xwayland yet | 10:04 |
| === hikiko is now known as hikiko|ln | ||
| TJ- | I'd suspect the first X server on VT7 owns/uses "/tmp/.X11-unix/" sockets and the 2nd server on VT2 tries to use the same socket | 10:46 |
| === hikiko|ln is now known as hikiko | ||
| cjwatson | should be a different name under /tmp/.X11-unix/ for each server - X0, X1, etc. | 10:54 |
| TJ- | cjwatson: that's what I'd expect; unfortunately that info isn't available in the bug report logs. "_XSERVTransMakeAllCOTSServerListeners: server already running" certainly suggests something has the socket the user session wants | 10:59 |
| seb128 | TJ-, would the stacktrace help? if you tag the bug apport-request-retrace you can get a next one from the next duplicate (rather than having it cleaned by the retracer) | 11:14 |
| TJ- | seb128: I was just wondering how we could do that; I'm not convinced the stacktrace in the master report is the same. I've been picking dupes at random and those after 1495049 all look different, most obviously ProcCmdline | 11:15 |
| seb128 | no, the master is not the same | 11:15 |
| seb128 | e.u.c launchpad batch OsAbort() errors it seems | 11:15 |
| seb128 | the master was a VT_ACTIVATE fail | 11:16 |
| TJ- | added the tag | 11:16 |
| seb128 | but the recent ones are all gdm ones | 11:16 |
| seb128 | so I expect a new retrace is going to give us a new bt | 11:16 |
| seb128 | showing the gdm issue | 11:16 |
| TJ- | I use KDE so I'm not familiar with the way gdm is running 2 X servers for (presumably the greeter) and the user session, but the ProcCmdline entries of all the dupes I've looked at are referring to the gdm owned X on VT7 | 11:18 |
| === _salem is now known as salem_ | ||
| psychic_ | quit | 12:38 |
| === salem_ is now known as tiagosh | ||
| ogra_ | grrr ... why do pipes not work in the initrd when i use libc tools but work when using the busybox commands | 13:16 |
| * ogra_ wonders if there is a trick | 13:18 | |
| infinity | ogra_: One tool printing to stderr and the other to stdout? | 13:28 |
| ogra_ | hmm | 13:28 |
| infinity | (Just a guess) | 13:28 |
| ogra_ | infinity, hah, no ... but seems - means something else | 13:31 |
| ogra_ | "/bin/wget -O - http://192.168.2.74:1500/ubuntu-15.04-snappy-armhf-rpi2.img.xz | /usr/bin/xzcat | dd of=/dev/mmcblk0" works fine | 13:32 |
| ogra_ | "/bin/wget -O - http://192.168.2.74:1500/ubuntu-15.04-snappy-armhf-rpi2.img.xz | /usr/bin/xzcat - | dd of=/dev/mmcblk0" doesnt | 13:32 |
| pitti | ah, busybox wget not accepting - ? | 13:32 |
| pitti | uh, odd | 13:32 |
| ogra_ | no, xzacat not accepting - | 13:32 |
| ogra_ | *xzcat | 13:32 |
| ogra_ | using the libc dd in that line gets even more weird :) | 13:33 |
| ogra_ | but yay ... my initrd installer constriction works | 13:33 |
| davmor2 | ogra_: lets just stick with because it hates you ;) | 13:33 |
| * ogra_ wraps more duct tape around it | 13:33 | |
| infinity | An initrd that grabs an unathenticated http URL and blats it to disk seems super secure. | 13:33 |
| ogra_ | yeah, awesome, aint it ? :) | 13:34 |
| infinity | Also, that should be armhf+rpi2 ... | 13:34 |
| davmor2 | infinity: shhhhh you're not meant to notice that | 13:34 |
| ogra_ | it is the same machine the serial cable is attached to ... for image tests | 13:34 |
| ogra_ | infinity, it so should ... | 13:34 |
| ogra_ | well, effectively armhf+raspi2 | 13:35 |
| infinity | Or that, yes. | 13:35 |
| ogra_ | but that name was already used when i antered the snappy team :/ | 13:35 |
| ogra_ | like the BBB image is called armhf-beagle or some such weird thing | 13:36 |
| * ogra_ is curious how long that dd will actually take ... | 13:37 | |
| ogra_ | (really bad that busybox dd doesnt know the blocksize arg) | 13:38 |
| seb128 | wgrant, why did you reject https://code.launchpad.net/~cjohnston/launchpad/1260760-unset-source/+merge/229462? (I guess it was wrong bug would have been nice to comment about what was the issue) | 13:43 |
| seb128 | or asked differently, what launchpad dev to I need to pay a beer to to get https://bugs.launchpad.net/launchpad/+bug/1260760 fixed? | 13:43 |
| ubottu | Launchpad bug 1260760 in Launchpad itself "editing an Ubuntu (gtk+3.0) bug unsets the source" [Undecided,Triaged] | 13:43 |
| ogra_ | wheee ! | 13:59 |
| ogra_ | it boots (and only took ~20min for dd) | 13:59 |
| dholbach | barry, do the two python sessions need a blueprint in LP for tracking work items and stuff? | 14:57 |
| dholbach | (just noticed them in summit) | 14:57 |
| barry | dholbach: maybe couldn't hurt although we all know what needs to be done. mostly i want to make sure everyone's aligned with the plan | 14:59 |
| dholbach | ok, wfm | 15:00 |
| barry | brb | 15:00 |
| === fginther` is now known as fginther | ||
| infinity | rbasak: I have a weird juju testsuite regression in wily that's about to hold up the release, who do I talk to to debug this ASAP? | 16:17 |
| rbasak | infinity: sinzui. I've asked him to join. | 16:17 |
| rbasak | infinity: BTW, I'm quite bothered by https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1490110. It breaks upgrades for anyone with lxc and juju-local installed. | 16:19 |
| ubottu | Launchpad bug 1490110 in lxc (Ubuntu) "package lxc 1.1.3-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Medium,Confirmed] | 16:19 |
| rbasak | Which is Every Charm Developer I think. | 16:19 |
| cjwatson | juju bootstrap totally works here on current wily, notwithstanding what the test suite is saying, but then I don't understand why it's failing | 16:20 |
| sinzui | hi infinity can I help with a juju issue | 16:20 |
| infinity | sinzui: Hey. What do you make of https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-wily/wily/amd64/j/juju-core/20151021_155639@/log.gz ? | 16:20 |
| rbasak | I can tell you what future-{local,manual}-provider do. | 16:20 |
| rbasak | They fake a future release to make sure that the development release isn't broken as soon as it opens. | 16:21 |
| rbasak | panic: osVersion reported an error: Could not determine series | 16:21 |
| rbasak | That may be related. | 16:21 |
| rbasak | The update to distro-info-data may be related. | 16:21 |
| cjwatson | oh, I didn't notice that that was future- | 16:21 |
| infinity | So, it's faking a future release incorrectly? :P | 16:21 |
| rbasak | Hmm, maybe. | 16:22 |
| rbasak | Do you happen to have a shell at the point the test breaks? Otherwise I can do it. | 16:22 |
| cjwatson | I suspect that it is faking something insufficiently far in the future | 16:22 |
| cjwatson | or maybe is confused by the last line having LTS in it? | 16:22 |
| rbasak | It seems likely to me that this is a bug in my faking-the-future code. | 16:23 |
| sinzui | Juju has a very stupid policy to hard coding known releases and treating them as everything that will ever exist. The reading of ubuntu.csv by juju ti to get a more current list. | 16:24 |
| * rbasak is busy reproducing | 16:24 | |
| sinzui | cjwatson: infinity rbasak: this is the bug I think you are seeing https://bugs.launchpad.net/juju-core/+bug/1465317 | 16:25 |
| ubottu | Launchpad bug 1465317 in juju-core "Wily osx win: panic: osVersion reported an error: Could not determine series" [High,In progress] | 16:25 |
| infinity | rbasak: Okay, so I think this bears investigation, as the test is clearly doing an oops of some sort, but now that I understand the scope, I'm also going to ignore the failure. Thanks for pointing out the future bit. | 16:25 |
| cjwatson | the hacked up file seems to have one more field than it should | 16:26 |
| cjwatson | but I think it's always been that way, so WTF | 16:27 |
| cjwatson | echo "${version},${codename},${series},${created},${release},${eol},${eol}" >> /usr/share/distro-info/ubuntu.csv | 16:27 |
| cjwatson | looks like that's one more ,${eol} than there should be | 16:27 |
| rbasak | I think that might have been intentional | 16:28 |
| sinzui | infinity: cjwatson what has changed between this builds? http://autopkgtest.ubuntu.com/packages/j/juju-core/wily/amd64/ to get the failure? If nothing changed, then I think a race condition is in the code where the wrong table of versions was read | 16:28 |
| rbasak | version,codename,series,created,release,eol,eol-server | 16:28 |
| rbasak | cjwatson: so two eol fields | 16:28 |
| cjwatson | sinzui: new distro-info-data | 16:28 |
| cjwatson | rbasak: oh, also, it's emitting dates that aren't dates | 16:28 |
| cjwatson | rbasak: (ah, I forgot eol-server. right) | 16:28 |
| cjwatson | 2015-26-19,2015-26-20,2015-26-22,2015-26-22 | 16:28 |
| sinzui | ah. bugger. I really hate jujus design to fail with the unknonwn | 16:29 |
| cjwatson | haha, I see the date problem | 16:29 |
| cjwatson | created=`date -d '-2 days' +%Y-%M-%d` | 16:29 |
| cjwatson | release=`date -d '-1 days' +%Y-%M-%d` | 16:29 |
| cjwatson | eol=`date -d '+1 days' +%Y-%M-%d` | 16:29 |
| cjwatson | %M should be %m | 16:29 |
| cjwatson | don't know if that's the problem here | 16:29 |
| cjwatson | it's probably not, but should be fixed! | 16:30 |
| rbasak | Oops. Sorry. Thanks! | 16:31 |
| rbasak | I might as well try that fix first and see if the error goes away. | 16:32 |
| cjwatson | we think that's not the problem because the historical passes are at times that ought to have guaranteed an invalid month | 16:33 |
| sinzui | rbasak: should I plan to rebase 1.24.7 on 1.24.6-0ubuntu3 packaging :) | 16:36 |
| rbasak | sinzui: :) | 16:36 |
| rbasak | sinzui: yes in theory, but a fix to the test should be trivial to merge. | 16:36 |
| sinzui | rbasak: I have automated more of the certifification for Juju. It get less painful each time | 16:37 |
| Odd_Bloke | Laney: Do we have a bug where we're tracking getting the UI needed to handle bug 1463072? | 16:38 |
| ubottu | bug 1463072 in gnome-terminal (Ubuntu) "highlighting on left mouse double click ends at :" [Medium,Won't fix] https://launchpad.net/bugs/1463072 | 16:38 |
| rbasak | It's not /usr/share/distro-info/ubuntu.csv I don't think. Either /etc/lsb-release or /etc/os-release. | 16:42 |
| rbasak | I think it's os-release. | 16:44 |
| Laney | Odd_Bloke: I don't know if there is a bug, but I have asking upstream on my to do list for the beginning ish of the cycle (no promises given the URL case which is the reason most people want this has a different way to achieve the same thing) | 16:49 |
| * ogra_ wonders if thats related to always having the quotes of a quoted string selected too on double click | 16:50 | |
| infinity | ogra_: That was fixed. | 16:50 |
| ogra_ | (it used to only select the bits between quotes in earlier versions) | 16:50 |
| ogra_ | infinity, oh, in wily ? | 16:50 |
| infinity | ogra_: Yeah. | 16:50 |
| ogra_ | i guess i found my first reason to upgarde then :) | 16:50 |
| infinity | ogra_: It wasn't that the behaviour changed, it was that some tools (like wget) switched from using ' and " to using unicode pretty quotes. | 16:51 |
| ogra_ | yeah | 16:51 |
| infinity | ogra_: And VTE/gnome-terminal didn't recognize those as boundaries. | 16:51 |
| infinity | ogra_: But, yeah, that's fixed. | 16:51 |
| ogra_ | dpkg-buildpackage too ... that is where it annoys me most | 16:52 |
| Odd_Bloke | Laney: I'm probably not going to chase you any more, as I've just switched to Terminator which does what I want. :p | 16:52 |
| rbasak | sinzui: I think the future faking is working correctly, apart from the date month/minute bug that cjwatson pointed out that doesn't seem to be the problem here. | 16:53 |
| rbasak | sinzui: it's something to do with how Juju is parsing /etc/os-release I think. If I revert os-release (but not the others, causing everything to be mismatched incidentally) then Juju works. | 16:54 |
| rbasak | sinzui: however the faking is just making it look like there's a "Y" release and the machine is on it. One difference is that it claims that Y is LTS. | 16:54 |
| sinzui | rbasak: understood. I think the issue is the bug I posted earlier. Juju specifies exact version instead of minimum versions. My bug is about stopping the hard coding so that all of us can upgrade. | 16:56 |
| rbasak | sinzui: I think it's over to upstream to look further now please. I would also note that release dates faked aren't coherent (Y released before X) so if that's the problem then please let me know and we can make the faking smarter. | 16:56 |
| rbasak | sinzui: I'm not convinced it's the same bug but I can see how it could be. | 16:56 |
| rbasak | Though it seems unlikely that Juju is parsing dates as it didn't barf on impossible dates :) | 16:57 |
| Laney | rbasak: Is is that you put "LTS" in VERSION_ID? | 16:59 |
| Laney | At least deleting that seems to make juju version work here | 16:59 |
| rbasak | Hmm. Good point. | 17:00 |
| * rbasak tries | 17:00 | |
| rbasak | Yes, that's it. Thanks Laney. Sorry sinzui. | 17:00 |
| infinity | Ahh, yes, VERSION_ID is always a number, LTS doesn't belong there. | 17:01 |
| rbasak | Yeah I have two variables, $version which includes a potential LTS and $version_ that doesn't. VERSION_ID should be using the latter. | 17:02 |
| * rbasak 's code pre-dates os-version, but never mind. He'll fix it. | 17:02 | |
| rbasak | infinity: do you want an upload if this fixes the issue? | 17:03 |
| rbasak | I can fix the date thing as well. | 17:03 |
| infinity | rbasak: It's not on any images, so sure. | 17:04 |
| infinity | rbasak: It's always nice to have tests passing. | 17:04 |
| rbasak | OK. Just testing my fix from scratch now. | 17:05 |
| Laney | What if they test that you cry when slapped in the face? | 17:05 |
| === robru is now known as robru|sick | ||
| rbasak | Juju fix uploaded. | 17:11 |
| * rbasak EODS | 17:11 | |
| infinity | rbasak: Ta. | 17:12 |
| hallyn_ | so bug 1490110 is looking weirder and weirder. I guess actually do-release-upgrade is crashing, then apport crashes while trying to report on it bc do-release-upgrade's python version was upgraded (so /proc/self/exe is deleted) | 17:21 |
| ubottu | bug 1490110 in lxc (Ubuntu) "package lxc 1.1.3-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Critical,Confirmed] https://launchpad.net/bugs/1490110 | 17:21 |
| * hallyn_ retries the whole shebang again with hopes of gdb attaching | 17:22 | |
| === inaddy is now known as tinoco | ||
| === funkyHat_ is now known as funkyHat | ||
| slangasek | pitti: so there are some duplicates of LP: #1504897 coming in... e.g. LP: #1508533 | 18:33 |
| ubottu | Launchpad bug 1504897 in nfs-utils (Ubuntu) "package nfs-common 1:1.2.8-9ubuntu10 failed to install/upgrade: подпроцесс установлен сценарий post-installation возвратил код ошибки 100" [Undecided,Incomplete] https://launchpad.net/bugs/1504897 | 18:33 |
| ubottu | Launchpad bug 1508533 in nfs-utils (Ubuntu) "package nfs-common 1:1.2.8-9ubuntu10 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Undecided,New] https://launchpad.net/bugs/1508533 | 18:33 |
| hallyn_ | ok so when lxc is built for wily, the pkg builder adds a versioned dependency on dbus, a version which is > what's in vivid. so do-release-upgrade from vivid fails bc dbus is too old. | 19:02 |
| hallyn_ | should lxc add a pre-depends on dbus for this, or should this be being handled by whatever is adding the depends? | 19:02 |
| hallyn_ | (this is still bug 1490110) | 19:02 |
| ubottu | bug 1490110 in lxc (Ubuntu) "package lxc 1.1.3-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Critical,Confirmed] https://launchpad.net/bugs/1490110 | 19:02 |
| cjwatson | a pre-depends will only make any difference at all if you're expecting the new dbus to be unpacked and configured *when your preinst runs* | 19:06 |
| hallyn_ | cjwatson: i dunno. when i pre-install the newer version of dbus, it still fails with | 19:15 |
| hallyn_ | invoke-rc.d: unknown initscript, /etc/init.d/lxc not found. | 19:15 |
| hallyn_ | which makes no sense since there is no /etc/init.d/lxc in the old pkg either | 19:15 |
| === 7F1AAQCDM is now known as ben___ | ||
| rbasak | hallyn_: http://paste.ubuntu.com/12887946/ line 131 maybe? | 19:38 |
| hallyn_ | rbasak: that looks fine though | 19:40 |
| hallyn_ | i mean it should be updated, but at the moment it's fine bc the upstart job is shipped | 19:41 |
| hallyn_ | question is why is invoke-rc.d insisting on using the sysvinit job? | 19:41 |
| hallyn_ | rbasak: also you're supposed to be eod :) | 19:41 |
| rbasak | So it calls invoke-rc.d even though there is no init.d script because it finds the upstart script | 19:41 |
| rbasak | But upstart isn't present, so invoke-rc.d will do whatever it does when upstart isn't present. | 19:42 |
| rbasak | That logic makes no sense. | 19:42 |
| rbasak | I don't really understand how it is supposed to interact with systemd though. | 19:42 |
| hallyn_ | rbasak: 'invoke-rc.d lxc start' talks to systemd just fine | 19:43 |
| hallyn_ | at least, when i do it by hand | 19:43 |
| hallyn_ | obviously not during this upgrade :) | 19:43 |
| hallyn_ | but yeah, maybe this is just a kick in the pants to change this | 19:43 |
| rbasak | hallyn_: could it be related to dh_systemd_start handling happening a little bit later in the generated postinst? | 19:43 |
| rbasak | I can't see how though | 19:44 |
| hallyn_ | hm. | 19:44 |
| hallyn_ | btw i was thinking that mustve come from lxc.postinst but it does not, so that's just the autogenerated postinst code.... nothing lxc can do about it afaics | 19:44 |
| rbasak | My hypothesis is that it's either a bug in dh_installinit/dh_systemd* or the way you're calling it in the package. Maybe if you drop the upstart job (rm_conffile etc) the problem will go away. | 19:45 |
| hallyn_ | can't do that, there are still upstart users | 19:46 |
| hallyn_ | like phone | 19:46 |
| rbasak | Ah, good point. | 19:46 |
| hallyn_ | mbiebl_: pitti: hey, one of you good fellows around? | 19:46 |
| hallyn_ | we're looking at bug 1490110 . the lxc package ships upstart and systemd jobs, not sysvinit. upgrade from vivid to wily fails on invoke-rc.d lxc start insisting on there being a /etc/init.d/lxc, and we can't tell why. | 19:47 |
| ubottu | bug 1490110 in lxc (Ubuntu) "package lxc 1.1.3-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Critical,Confirmed] https://launchpad.net/bugs/1490110 | 19:47 |
| TJ- | hallyn_: I saw something very similar with a user recently, using mongodb, and tracked it down to the postinst script incorrectly trying to use update-rc.d when there's an /etc/init.d/.legacy-bootordering | 19:49 |
| TJ- | hallyn_: Hazy recall now, but I *think* the control flow was going via invoke-rc.d or similar | 19:50 |
| hallyn_ | TJ-: what should postinst be doing instea dof update-rc.d? | 19:53 |
| TJ- | hallyn_: just gone back through my logs - terrible memory! It was actually showing up via (manually reproduced doing) "dpkg --configure libpam-systemd" | 20:01 |
| TJ- | hallyn_: "the root cause is a call to "update-rc.d systemd-logind defaults" " | 20:01 |
| TJ- | hallyn_: if you're seeing the same/similar issue, lines 15-19 of the 14.04 libpam-systemd.postinst were the crux: http://paste.ubuntu.com/12739482/ | 20:08 |
| hallyn_ | TJ-: the 'defaults' line, or also line 19? | 20:09 |
| TJ- | hallyn_: I'm trying hard to recall! I *think* I originally though line 16, but then we did a "set -x" trace and it showed it was line 19 (invoke-rc.d) | 20:10 |
| hallyn_ | lxc is using invoke-rc.d to start some other things (mainly apparmor and dnsmasq) | 20:12 |
| hallyn_ | what did you replace that with? | 20:12 |
| TJ- | hallyn_: Just found the log (the user changed nicknames): "dpkg-divert --local --rename --add /usr/sbin/invoke-rc.d" and "ln -s /bin/true /usr/sbin/invoke-rc.d" and then undid it afterwards. | 20:13 |
| rbasak | Wow! | 20:13 |
| rbasak | That seems a bit...extreme :) | 20:14 |
| hallyn_ | TJ-: will try that, thanks much :) | 20:14 |
| TJ- | rbasak: I was trying to get the user out of a broken "apt-get -f install" situation | 20:14 |
| hallyn_ | although, hm, i don't want invoke-rc.d t onot run | 20:14 |
| sarnold | heh, I'd recommend asking one of the greybeards before going that far :) | 20:15 |
| TJ- | hallyn_: right, I think it depends on which services require it, but as I said that was the root cause for the user I was helping | 20:15 |
| TJ- | I seem to recall I found the work-around recommended on the -devel mailing list from some time back, or else in a launchpad bug comment | 20:15 |
| TJ- | Not sure how sane this was, but I found myself saying: "...the reason that invoke-rc.d is failing | 20:17 |
| TJ- | since it can't find the 'upstart' version of initctl, it tries to use sysv-init, and there is no sysv-init script for systemd-logind | 20:17 |
| hallyn_ | should this count as a bug against sysv-rc? | 20:17 |
| doko | kees, is harderning-wrapper still maintained, or is it obsolete now? | 20:30 |
| achiang | mdeslaur: hey, have there been any 14.04 security updates that might break LXC? trying to start a container and getting: lxc-start: conf.c: setup_pts: 1772 Permission denied - mount failed '/dev/pts/ptmx'->'/dev/ptmx | 20:31 |
| * achiang decides to try and boot an older kernel to see if that is related | 20:32 | |
| pitti | slangasek: right, and bug 1502536 is essentially the same; I haven't yet managed to reproduce it | 20:33 |
| ubottu | bug 1502536 in ofono (Ubuntu Wily) "package ofono 1.17.bzr6904+15.10.20150928.1-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [High,Incomplete] https://launchpad.net/bugs/1502536 | 20:33 |
| pitti | slangasek: but vila attached the apt-clone tarball yesterday; I guess this can somehow be used to reconstruct the original system | 20:34 |
| rbasak | pitti: that looks like the same as https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1490110 that hallyn is working on | 20:34 |
| ubottu | Launchpad bug 1490110 in lxc (Ubuntu) "package lxc 1.1.3-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Critical,Confirmed] | 20:34 |
| === tiagosh is now known as _salem | ||
| pitti | hallyn_: hello; this was meant to be fixed in vivid's invoke-rc.d and service already, hmm | 20:34 |
| pitti | hallyn_: actually sounds like the same issue I was just telling slangasek | 20:35 |
| pitti | rbasak: right, these are by and large all the same | 20:35 |
| rbasak | pitti: we currently have a reproducer | 20:35 |
| pitti | rbasak: oh, you do? I tried umpteen scenarios for ofono and nfs-common | 20:35 |
| rbasak | pitti: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1490110/comments/20 | 20:35 |
| ubottu | Launchpad bug 1490110 in lxc (Ubuntu) "package lxc 1.1.3-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 100" [Critical,Confirmed] | 20:35 |
| rbasak | On Vivid install juju-local + lxc then do-release-upgrade to Wily. | 20:36 |
| pitti | rbasak: note that for fixing upgrades it's enough to have the fix in -updates, so we don't need to respin for that | 20:37 |
| pitti | just a 0-day SRU | 20:37 |
| rbasak | ack | 20:37 |
| pitti | rbasak: odd; from a vivid cloud image I tried installing nfs-common, ofono, then dist-upgrade and what not; but I"ll try with this | 20:37 |
| rbasak | I used uvt-kvm | 20:37 |
| sarnold | achiang: looks a bit like https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781 | 20:37 |
| ubottu | Launchpad bug 1504781 in lxc (Ubuntu Trusty) "lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66: AppArmor denies /dev/ptmx mounting" [Undecided,Fix released] | 20:37 |
| rbasak | I have the cloud image I used. I haven't tried multiple times so I don't know if there's something non-deterministic involved. | 20:37 |
| achiang | mdeslaur: hm, yes. booting with 3.13.0-65-generic #106-Ubuntu allows my vagrant-lxc container to boot again | 20:38 |
| achiang | and with 3.13.0-66-generic it is broken | 20:38 |
| sarnold | achiang: see also https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1507959 -- it may be worth unduping those two? | 20:38 |
| ubottu | Launchpad bug 1504781 in lxc (Ubuntu Trusty) "duplicate for #1507959 lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66: AppArmor denies /dev/ptmx mounting" [Undecided,Fix released] | 20:38 |
| achiang | sarnold: reading through the first one 1504781 certainly describes my symptoms | 20:40 |
| * rbasak goes to bed | 20:41 | |
| achiang | running apt-get upgrade to get lxc 1.0.7-0ubuntu0.9 and will try with kernel -66 again | 20:43 |
| achiang | brb | 20:43 |
| achiang | Yeah, that fixed it. Thanks sarnold | 20:44 |
| hallyn_ | pitti: hi, so you have a handle on that bug? | 21:00 |
| pitti | hallyn_: I duplicated/summarized it, but I don't have a working reproducer yet | 21:00 |
| pitti | obviously it's some mis-handling in invoke-rc.d | 21:01 |
| pitti | in vivid we added some handling for packages which don't have init.d scripts, but apparently there's some case where this doesn't work | 21:01 |
| hallyn_ | hm. as rbasak said uvt-kvm makes it very easy, or you could do it with lxc-create -t | 21:01 |
| hallyn_ | uh, lxc-create -t ubuntu-cloud | 21:01 |
| pitti | hallyn_: I'm running dist-upgrades in qemu | 21:01 |
| pitti | so far I installed lxc and juju-local, and now dist-upgrading | 21:02 |
| pitti | which still works fine | 21:02 |
| hallyn_ | so it's something about the clou dimages then | 21:02 |
| pitti | but so far indeed I didn't use do-release-upgrade, I'll try that next | 21:02 |
| hallyn_ | (which is what uvt-kvm uses) | 21:02 |
| hallyn_ | oh! | 21:02 |
| hallyn_ | ok | 21:02 |
| pitti | hallyn_: no, I also got reports from desktops | 21:02 |
| pitti | like vila's | 21:02 |
| hallyn_ | ok | 21:02 |
| pitti | I can't imagine how apt full-upgrade vs. do-release-upgrade would make any difference, but who knows | 21:03 |
| pitti | hallyn_: do you have a way to reproduce it? | 21:05 |
| pitti | so far every reporter couldn't reproduce it after the dist-upgrade, any further dpkg-reconfigure, apt-get install -f, apt-get install --reinstall, sh -x invoke-rc.d etc. went fine | 21:06 |
| pitti | ah! do-release-upgrade indeed! | 21:09 |
| hallyn_ | indeed :) | 21:21 |
| hallyn_ | does it set some weird configuration? | 21:21 |
| pitti | imported/invoke-rc.d | 21:22 |
| pitti | SRSLY? | 21:22 |
| pitti | our dist upgrade tarballs have their own version? | 21:22 |
| hallyn_ | <blink> | 21:22 |
| pitti | and lo and behold, diff -u /usr/sbin/invoke-rc.d imported/invoke-rc.d | 21:22 |
| pitti | all the systemd support is missing | 21:23 |
| Laney | umm? | 21:23 |
| pitti | mvoooooooooooooooooooooooo | 21:23 |
| hallyn_ | phew | 21:23 |
| Laney | because it wants to apply this diff to it? | 21:24 |
| pitti | Laney: no, I ran diff between the version in wily.tar.gz (the release upgrader tarball) and the real invoke-rc.d from sysv-rc | 21:25 |
| pitti | I updated the bug | 21:25 |
| pitti | now, I don't have the slightest idea how this upgrade tarball is constructed | 21:25 |
| pitti | bdmurray: ^ do you happen to know? or is that still all mvo's domain? | 21:26 |
| Laney | pitti: I mean it has this copy because it has DistUpgrader/imported/invoke-rc.diff applied or what? | 21:27 |
| Laney | (in the u-r-u source package) | 21:27 |
| pitti | Laney: perhaps, yes | 21:27 |
| Laney | does it just come right from here to the tarball? | 21:27 |
| pitti | Laney: yes, I just unpacked it and looked at imported/invoke-rc.d | 21:28 |
| pitti | do-r-u leaves it behind in /tmp/ | 21:28 |
| bdmurray | pitti: DistUpgrade/build-tarball.sh creates it | 21:29 |
| cjwatson | and it's spat out as part of the ubuntu-release-upgrader package build and gets into LP via a raw-upgrader (or some such) dpkg-distaddfile | 21:30 |
| cjwatson | raw-dist-upgrader, that's the one | 21:30 |
| pitti | cjwatson: ah, so uploading u-r-u should do it? | 21:30 |
| Laney | So yes, can fix the one in the package | 21:30 |
| cjwatson | pitti: right | 21:30 |
| cjwatson | (disclaimer: have not been following the rest of the discussion at all) | 21:31 |
| pitti | cjwatson: do you know, would that also work when SRUing this? i. e. upload to wily-updates? | 21:31 |
| pitti | I don't think we want to rebuild images for this | 21:31 |
| pitti | (or maybe it's necessary, but checking options here) | 21:31 |
| cjwatson | I believe so but would have to check | 21:31 |
| bdmurray | pitti: yes, it would we just need to change the meta-release file url to point to -updates | 21:31 |
| pitti | cjwatson: rest of the discussion is not that important; the essence is, the current wily.tar.gz upgrade tarball is buggy | 21:31 |
| bdmurray | pitti: I already have an SRU pending for bug 1508539 | 21:32 |
| ubottu | bug 1508539 in ubuntu-release-upgrader (Ubuntu Wily) "KernelRemoval information not updated for 15.10" [High,In progress] https://launchpad.net/bugs/1508539 | 21:32 |
| cjwatson | oh that's it | 21:32 |
| cjwatson | it gets the URL from http://changelogs.ubuntu.com/meta-release* | 21:32 |
| cjwatson | (the various versions of that, there's an index at the top level) | 21:32 |
| cjwatson | I think that's updated by hand but it can be made to point to -updates | 21:32 |
| cjwatson | and indeed does for some series | 21:33 |
| bdmurray | cjwatson: that's what I was saying | 21:33 |
| pitti | bdmurray: ah good; mind if I reject your upload and add the invoke-rc.d update and reupload? | 21:33 |
| cjwatson | bdmurray: right, indeed, just wanted to fill in the details | 21:33 |
| bdmurray | pitti: no, that sounds find. What bug is this fixing? | 21:33 |
| pitti | bdmurray: seems you didn't yet push the release tag to bzr anyway :) | 21:33 |
| cjwatson | since I think this is poorly understood by a lot of devs | 21:33 |
| pitti | bdmurray: bug 1504897 | 21:33 |
| ubottu | bug 1504897 in ubuntu-release-upgrader (Ubuntu X-series) "packages with only upstart+systemd without sysvinit fail to upgrade with do-release-upgrade: upgrade tarballs ship obsolete invoke-rc.d" [High,In progress] https://launchpad.net/bugs/1504897 | 21:33 |
| pitti | bdmurray: i. e. various packages failing to upgrade with "no such init.d script" and exit status 100 | 21:34 |
| bdmurray | pitti: so bug 1508247 then? | 21:35 |
| ubottu | bug 1508247 in lxc (Ubuntu) "Upgrade failed due to lxc" [Undecided,New] https://launchpad.net/bugs/1508247 | 21:35 |
| pitti | bdmurray: yep, duplicated; similar to bug 1490110 (also already duped) | 21:35 |
| ubottu | bug 1504897 in ubuntu-release-upgrader (Ubuntu X-series) "duplicate for #1490110 packages with only upstart+systemd without sysvinit fail to upgrade with do-release-upgrade: upgrade tarballs ship obsolete invoke-rc.d" [High,In progress] https://launchpad.net/bugs/1504897 | 21:35 |
| bdmurray | pitti: okay, I'll keep an eye out for those | 21:36 |
| Laney | can we just take this diff into the normal invoke-rc.d in future? | 21:37 |
| Laney | feel like this trap might come up again | 21:37 |
| pitti | seems reasonably fine, yes | 21:38 |
| pitti | for X we should really do that | 21:38 |
| * Laney writes a note for friday | 21:38 | |
| * pitti taps foot for this looong pre-build.sh to finsih | 21:42 | |
| pitti | ok, uploaded | 21:47 |
| pitti | can we test this somehow while it's in -proposed? | 21:48 |
| Laney | I guess re-target http://changelogs.ubuntu.com/meta-release-development ? | 21:48 |
| pitti | like temporarily pointing ^ to -proposed ? | 21:49 |
| Laney | bdmurray: ^? | 21:49 |
| pitti | I wonder if there's some secret option/env var to use a local file | 21:49 |
| pitti | arges: ^ as you currently seem to do SRUs, would you mind looking at ubuntu-release-upgrader in wily? (I uploaded it, so don't want to self-accept) | 21:50 |
| bdmurray | usually meta-release-proposed is used to point at proposed | 21:50 |
| pitti | arges: this should probably become a 0-day SRU tomorrow | 21:50 |
| pitti | arges: (sorry for the shameless abuse!) | 21:51 |
| cjwatson | I think you can pass it a local configuration file too | 21:51 |
| Laney | It's ok, I'm reviewing it | 21:51 |
| TJ- | pitti: I did something with it some years ago; edited "/etc/update-manager/meta-release" | 21:51 |
| cjwatson | stash it in /etc/update-manager/meta-release IIRC | 21:51 |
| cjwatson | snap | 21:52 |
| Laney | I've released 0-day SRUs before | 21:52 |
| pitti | TJ-, cjwatson: awesome, thanks | 21:52 |
| Laney | so don't feel too naughty accepting it with thise intention :) | 21:52 |
| Laney | cjwatson: ah, feel free | 21:52 |
| pitti | Laney: ack, thanks; you have more context now, too | 21:52 |
| cjwatson | I'm not reviewing it | 21:52 |
| Laney | oh | 21:52 |
| cjwatson | my snap was to TJ- | 21:52 |
| Laney | fair | 21:52 |
| pitti | so once it's accepted/built, we can test the upgrade | 21:52 |
| arges | pitti: will look | 21:53 |
| pitti | arges: seems Laney is on it, so nevermind; thanks anyway! | 21:53 |
| arges | pitti: oh I don't usually look at dev releases | 21:53 |
| arges | pitti: ok ok | 21:53 |
| pitti | arges: it's that gray area between dev and stable now :) | 21:53 |
| lifeless | dable? stev? | 21:53 |
| pitti | lifeless: almost-solid? | 21:54 |
| lifeless | hmm, what do you call jellies that are nearly set ? | 21:55 |
| pitti | cjwatson: while we can soak in your wisdom: how does http://changelogs.ubuntu.com/meta-release{,-development} get updated once this gets released? is that "vi" by someone with perms, or does that live in some bzr? | 21:55 |
| TJ- | lifeless: Ready to eat | 21:55 |
| pitti | (or "emacs" of course -- this ain't no flamebait) | 21:56 |
| bdmurray | pitti: it lives in bzr but someone needs to put on the server | 21:57 |
| bdmurray | pitti: ~ubuntu-core-dev/meta-release/ubuntu/ | 21:57 |
| pitti | bdmurray: ah, is that an RT? | 21:59 |
| bdmurray | pitti: no, I have access | 21:59 |
| pitti | bdmurray: ah, ok | 21:59 |
| pitti | hallyn_, rbasak, Laney, bdmurray, cjwatson: thanks for your help -- nice teamwork! | 22:00 |
| cjwatson | as do I, Michael, and Adam | 22:00 |
| Laney | pitti: accepted, thanks | 22:01 |
| Laney | modulo hotel wifi (done now) | 22:01 |
| pitti | Laney: ok,thanks; it's getting late, so unless someone beats me to it I'll test it first thing in the morning | 22:02 |
| Laney | I'm guessing you might beat us to being awake/online :) | 22:03 |
| bdmurray | pitti: I can test it what should I have installed? | 22:03 |
| Laney | (but if not, will do) | 22:03 |
| pitti | bdmurray: I added a test case to bug 1504897 description | 22:03 |
| ubottu | bug 1504897 in ubuntu-release-upgrader (Ubuntu X-series) "packages with only upstart+systemd without sysvinit fail to upgrade with do-release-upgrade: upgrade tarballs ship obsolete invoke-rc.d" [High,Fix committed] https://launchpad.net/bugs/1504897 | 22:03 |
| bdmurray | pitti: cool, thanks. Its also possible to just manually download the dist-upgrader tarball, extract it and run ./wily | 22:04 |
| pitti | bdmurray: that'd be great, thanks; you also have sru-release and ~ubuntu-core-dev/meta-release/ubuntu/ powers, so we can land this by tomorrow | 22:05 |
| pitti | bdmurray: ah, ./wily instead of do-release-upgrade -d? | 22:05 |
| pitti | I don't see the spethial tarball on https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:15.10.14/+build/8156705 | 22:06 |
| bdmurray | pitti: do-release-upgrade -d does the checking of the metarelease file and stuff. I'm saying you can http://archive.ubuntu.com/ubuntu/dists/wily-proposed/main/dist-upgrader-all/current/ download wily.tar.gz and unpack it and then run it | 22:06 |
| pitti | so that doesn't work through dpkg-distaddfile, but some other side channel I suppose | 22:06 |
| pitti | bdmurray: ah, cool; so that's essentially what do-r-u does | 22:07 |
| bdmurray | pitti: yes | 22:07 |
| * pitti waves good night then, cu tomorrow | 22:08 | |
| Laney | night pitti | 22:08 |
| pitti | "today" already :) | 22:08 |
| sarnold | night pitti, see you in two or three hours? :) | 22:08 |
| cjwatson | pitti: it really is dpkg-distaddfile, you just don't see it in the UI | 22:08 |
| cjwatson | pitti: but follow the link to the build log and you'll see it being added there | 22:09 |
| hallyn_ | pitti: thank you! | 22:09 |
| cjwatson | (the modelling of custom uploads in LP's database is ... inadequate, which is why the web UI doesn't display them) | 22:10 |
| * Laney showers, hopefully this is published shortly and I can check it | 22:11 | |
| bdmurray | Laney: I'm testing it already fwiw | 22:42 |
| Laney | bdmurray: me too, but your internets are probably more internet than my internet | 22:45 |
| sarnold | hehehe | 22:46 |
| israeldahl | Hi all, can anyone point me to any documentation on building a custom PowerPC ISO. | 22:53 |
| israeldahl | I have already built for x86 | 22:53 |
| israeldahl | 32 bit and 64bit are not hard to do (for me) using deboostrap and chroot... but PowerPC is harder does anyone know the secret? | 22:54 |
| Laney | bdmurray: OK, finished(!) and I don't see those messages now | 23:02 |
| bdmurray | Laney: neither do I | 23:04 |
| israeldahl | Does anyone know where documentation to build a Power PC chroot is? | 23:05 |
| israeldahl | I have PPC and x86. I would like to do it on x86 (if possible) but would be willing to use PPC if I have to | 23:06 |
| israeldahl | Does debootstrap work with chroot on PPC? | 23:06 |
| sarnold | israeldahl: wouldn't debootstrap on the ppc work? | 23:06 |
| sarnold | heh | 23:06 |
| cjwatson | Native debootstrap always works. Our own build processes rely on it. | 23:07 |
| israeldahl | thanks cjwatson | 23:07 |
| Laney | bdmurray: unless you want to take care of it I'll sru-release in the morning | 23:07 |
| * Laney sleeps now, night | 23:07 | |
| israeldahl | cjwatson is it possible to use qemu or something to do it on x86? | 23:07 |
| cjwatson | If you were doing it on x86 then in practice you would need to grab an existing powerpc image and use qemu-system-powerpc. It would be slow. | 23:07 |
| cjwatson | qemu-system-ppc, sorry | 23:08 |
| cjwatson | In theory you can probably start the process with on-the-fly per-process emulation using qemu-user-static, but in practice you would run into trouble somewhere along the line with that approach. | 23:08 |
| israeldahl | cjwatson you mean I would need to loop mount the iso and chroot in? | 23:08 |
| cjwatson | If you have a powerpc system then the path of least resistance is definitely to use that. | 23:09 |
| cjwatson | qemu-system-ppc != chroot. Anyway, go native if you can. | 23:09 |
| israeldahl | cjwatson thanks I will try it. I have scripts for building on x86, so everything should be pretty much the same (except kernel names) but the process functions the same, correct? | 23:10 |
| cjwatson | Then you don't need qemu and everything's much simpler and quite probably faster as well, unless your powerpc system is exceptionally slow. | 23:10 |
| cjwatson | Right. | 23:10 |
| cjwatson | Well, and the top-level ISO bits are a bit different. | 23:10 |
| israeldahl | cjwatson does isolinux (this is precise based) work the same for PPC? | 23:11 |
| cjwatson | It's not in a shape that you can run directly, but the stuff in "bzr branch lp:~ubuntu-cdimage/debian-cd/ubuntu debian-cd" (directory tools/boot/trusty/ etc.) at least shows you the options we use | 23:11 |
| cjwatson | isolinux is x86-specific, and does not work in any way, shape, or form for powerpc | 23:11 |
| israeldahl | cool thank you I will totally check it out! | 23:11 |
| israeldahl | ok so I need yaboot and all that | 23:11 |
| cjwatson | right, or in theory grub but our current images are yaboot-based | 23:12 |
| israeldahl | Great cjwatson you have been very helpful! I may pop back in after a few days if I need more insight | 23:13 |
| israeldahl | you rock :) | 23:13 |
| cjwatson | np | 23:14 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!