/srv/irclogs.ubuntu.com/2014/05/02/#ubuntu-devel.txt

* infinity grumps at rebooting to a dnsmasq eating 100% CPU and no name resolution.00:12
infinitystgraber: ^-- Do you know what's up with that?00:12
mwhudsonthis has been happening to me quite a lot too00:13
mwhudsonwell not sure about the eating cpu part00:13
mwhudsonbut resuming laptop and having no dns has been frequent00:13
mwhudson(until killall dnsmasq, which seems to make resolvconf just put the remove nameservers in /etc/resolv.conf)00:13
infinityI killed dnsmasq, unplugged network, replugged network, and got a fresh dnsmasq that wasn't broken.00:14
stgraberinfinity: yes, new dnsmasq is broken00:15
stgraberbug 131339300:15
ubottubug 1313393 in dnsmasq (Ubuntu) "dnsmasq lockup at 100% cpu" [Undecided,Confirmed] https://launchpad.net/bugs/131339300:15
stgraberSimon Kelley appears to be on it00:16
Ryan_Lanebarry: ImportError: No module named _sysconfigdata_nd <-- virtualenv package in 14.04 is broken00:16
Ryan_Laneany plans on this being fixed? it's not great having to use pip to install a working virtualenv00:16
Ryan_Lanehm. this may be related to something being locked in config management00:19
Ryan_Laneindeed. user error. ignore me00:22
=== rww is now known as rww`
=== _salem is now known as salem_
=== salem_ is now known as _salem
xnoxrsalveti: http://people.canonical.com/~xnox/gcc-i686-linux-android_4_amd64.deb02:26
xnoxrsalveti: but i can't build it in the archive, getting ice from compiler whilst bootstrapping...02:26
* sarnold rubs his eyes .. i686 .. amd64 .. linux .. android ..02:27
sarnoldsurely it must just be a bad dream02:27
xnoxdon't worry it fails bootstrap with ubuntu toolchain, builds fine with aosp provided official c compiler..... wait trusting trust02:50
=== rww` is now known as rww
pittiGood morning05:05
pittislangasek: poll> I didn't see an option to convert to local timezones (but I didn't create an account)05:08
pittislangasek: yes, these are meant to be London times as I wrote; I left out the ones which definitively don't overlap between EU and US (unless you want to meet in the middle of the night)05:08
pittihallyn: the scipts and sourcing of /etc/default/cgmanager are a  bit awkward, but that can all be translated pretty much 1:105:10
pittihallyn: as a guide, I recommend reading man systemd.unit and systemd.service (they explain the available options in an unit file), and perhaps even more look at existing ones05:10
pittihallyn: e. g. for cgmanager you don't need to specify any dependencies, the default basic.target (usual stuff mounted etc.) is fine05:11
pittihallyn: as for the cgm_extra_mounts= bits, I recommend putting those into config files; systemd units are not conffiles, and upstart jobs really really should have never been05:12
slangasekpitti: morning - yeah, I figured out that was the case, so filled it out accordingly :)05:26
infinityIt hurt my brain a bit to convert to London times, and I have a clock that does it and everything.05:27
pittislangasek: I saw, thanks05:29
pittixnox: you already uploaded gvfs for plist in 1.20.1-1ubuntu2 :)06:07
pittixnox: anyway, gvfs' tests are broken with the new libgphoto2, I need to update them (same as with umockdev)06:07
infinitypitti: BTW, don't know if you've been following backscroll in #debian-devel, but seems when you boot with systemd, /run/shm is no longer a thing.  Might be nice to JFDI a fix on that, so I can stop arguing with Tollef about it.06:43
Unit193Yes, /run/shm doesn't exist for me.06:44
pittiinfinity: eek; yes, we'll need that06:53
pittiat least a symlink to /dev/shm/06:53
pittiinfinity: what was Tollef's argument against that?06:54
pittiinfinity, Unit193: mind filing a bug (LP is ok)? I have my head full of debugging state for autopkgtest mess, and that's my #1 prio ATM06:54
infinitypitti: Debian's wrong, everyone else uses /dev/shm, systemd is always right, we should just drop /run/shm and fix anything that references it instead of having a compat migration plan.06:58
infinitypitti: Basically.06:58
infinitypitti: I might have disagreed a fair bit on the conclusion.06:58
=== work_alkisg is now known as alkisg
geserdoko__: Hi, is merging python-babel somewhere on your TODO list? Could you also take a look at bug #1299442 as I assume that it's fixed in Debian (run into this bug during a dist-upgrade to trusty). Or is it OK if I prepare the merge and possible SRU?07:00
ubottubug 1299442 in python-babel (Ubuntu) "UnknownLocaleError: unknown locale 'en'" [Undecided,Confirmed] https://launchpad.net/bugs/129944207:00
pittiinfinity: argh, what was that site to grep all Debian source packages again?07:01
pitti*tock* *tock* memory, start working!07:01
geserhttp://sources.debian.net/ ?07:03
infinitypitti: http://codesearch.debian.net/07:03
pittiaah, that was it, tahnks07:04
pittiinfinity, Unit193: nevermind, already reported: debian bug 67475507:05
ubottuDebian bug 674755 in systemd "systemd: tmpfs inconsistencies (/run/lock, /run/shm, /tmp)" [Normal,Open] http://bugs.debian.org/67475507:05
pittiso indeed most of codesearch's results seem to be quirks for Debian using /run/shm, but still, having a compat symlink doesn't hurt07:07
infinitypitti: I think at least a compat symlink for wheezy->jessie migrations and 14.04->16.04 migrations probably makes sense.  Discussing dropping it after, if literally everyone else uses /dev/shm and no one can come up with an argument for why we decided it might be nice to have it on another FS (or referenced from run) is a fine conversation to have later.07:10
infinitypitti: Then again, if there *is* a valid reason for it to be on its own FS, inverting the mountpoint and link would be nice.07:11
pittiinfinity: it has always been its own tmpfs, it's just whether it was mounted on /dev/shm/ or /run/shm/07:11
infinity(For the record, comments in code aside, glibc DTRT in either case, as long as both paths exists, and one is a link to the other)07:11
pittiright, eglibc has /run/shm/ as an alternate fallback for the Debian case07:12
infinitypitti: Err, right, sorry, it's its own FS, but maybe there we sane arguments for it not loving under udev's tmpfs.07:12
infinitypitti: And maybe those arguments are long buried in the past.07:12
pittiinfinity: I think it was mostly from the time when some Debian people said "but we don't want udev"07:12
infinitypitti: That glibc code you're reading is actually the kfreebsd patch.07:12
infinitypitti: The linux path Just Works if the symlink is in place.07:13
infinity(The linux code is horribly awful after that point, and I might sent a patch upstream tomorrow to tear out about 50 lines of nonsense, but at leas that bit works...)07:14
infinitys/sent/send/07:14
slangasekinfinity: wasn't /run/shm a change that rleigh inflicted on us, deviating from upstream?07:14
infinityslangasek: Yeah, but does anyone know why? :P07:14
slangasekinfinity: because he thought it was prettier07:14
infinityIf there are no valid arguments, I'm all for reverting the mountpoint back to /dev/ (as systemd does), but I still think a release cycle of grace period for a compat symlink is sane.07:15
slangasek"shared memory is not a device"07:15
pittilet's rename /etc/ to /config because it's prettier :)07:15
pittiinfinity: yes, agreed07:15
slangasekinfinity: but you might want to scan all your chroots before trying to switch this back, because I'm not sure the migration *to* /run/shm was ever finished correctly07:16
infinityIn fact, my buildds all have the inverted setup (mount in /dev, symlink in /run) because launchpad-buildd has a bug where the inverse explodes, and I'm too lazy to fix it. :P07:16
slangasekI didn't mean just buildd chroots fwiw07:16
infinityslangasek: Yeah, how this will work for chroot upgrades is a touchy thing.  And, in fact, that's touched on in the initscripts postinst comments of doom.07:17
infinityslangasek: For real systems, it's likely cake.  Reboot into systemd, it mount /dev/shm, does a check on the state of /run/shm, and if it's a directory, wipes it and replaces with symlink.07:18
infinityWhich, handily, would DTRT if you rebooted back into sysvinit or upstart, I think, as they'd just mount the inverted way on the next boot.  I think.07:18
infinityBut anything with a more manual chroot mount setup is touchier.07:18
infinityErr, /run/shm wouldn't be a directory, obviously, it's in run.07:19
infinityBut you know what I mean.07:19
infinitysystemd would just create symlink and done.07:19
infinity(on a real system)07:19
infinityI'm obviously too tired to discuss this sanely. :P07:21
pittiinfinity: for "real" systems there's no migration issue really, as both /dev/ and /run/ are (dev)tmpfses07:35
Saviqpitti, hey, question about autopkgtest - what's the recommended approach when you have mocks (built during dpkg-buildpackage, but not packaged) that you need for autopkgtest, is it better to package them in a foo-tests package or so, or build them during the test run itself?07:35
pittiand for chroots that should just do [ -e /run/shm ] || ln -s /dev/shm /run/shm07:35
infinitypitti: Yeah, that was my "I'm obviously too tired" comment.07:35
pittiSaviq: if it's cheap to build (which I assume), use "needs-build" and use the built mocks07:36
pittiSaviq: err, "build-needed"07:36
Saviqpitti, huh, that's contrary to DEP-8 :)07:36
pittiSaviq: if that builds too much, your debian/tests/foo can also do something like make -C tests/mocks/ to only build what you actually need for the test07:36
Saviqpitti, yeah, the latter was what I was thinking of07:37
pittiSaviq: DEP-8 makes no prescription for that07:37
infinityAnd if it's really expensive, ship foo-test.07:37
infinityWhich is what I plan to do with glibc, once it can actually run the testsuite out of tree (getting there!)07:37
pittiSaviq: we use packaged installed tests for packages like glib where building everything again would be too expensive07:37
Saviqpitti, I'm about unity8 here, a full build took ~12 mins on LP07:38
Saviqpitti, that considered cheap?07:39
pittiSaviq: but hopefully you don't need to build the full unity for just the mocks?07:39
Saviqpitti, sure, it would take even less for the mocks07:39
pittiSaviq: I take it it will take much longer on armhf07:39
pittiSaviq: if you can do a partial build, that's definitively preferred07:39
Saviqpitti, ~21 mins07:39
Saviqfor a full build07:39
pitti-tests packages are by and large clutter07:39
Saviqyeah, that's what I thought07:40
SaviqI think building only the mocks will only be like ⅓ of the time07:40
Saviqok, going that route then07:40
Saviqpitti, thanks! could I ask you for a review once I have the autopkgtest MP'd?07:41
pittiSaviq: sure!07:41
Saviqgreat, thanks07:41
zygapitti: excellent write-up and progress on the systemd sprint, it was very fun to read (brings back memories from fun and productive sprints)07:58
pittizyga: thanks :)08:05
ogra_pitti, heh, was that sprint photo taken by a painter ?08:17
pittiogra_: heh, that was a fun one; we couldn't find a place where to put the mobile phone so that it would see us all08:18
pittiogra_: so we used a laptop and cheese; apparently that laptop's  webcam was crappy, or he still had some "oil painting" effect enabled..08:18
ogra_lol08:18
=== superm1_ is now known as superm1
=== infinity3 is now known as infinity
=== Mirv_ is now known as Mirv
=== mnepton is now known as mneptok
=== Ursinha_ is now known as Ursinha
=== halfie_ is now known as halfie
=== dbarth_ is now known as dbarth
=== warp10_ is now known as warp10
=== davmor2_ is now known as davmor2
=== Adri2000_ is now known as Adri2000
=== vrruiz_ is now known as rvr
ari-tczewxnox: you have updated one B-D in package libzrtpcpp (2.0.0-3build2) (http://launchpadlibrarian.net/155442202/libzrtpcpp_2.0.0-3build1_2.0.0-3build2.diff.gz), is it still necessary, or can be dropped?09:17
cjwatsonLogan_: I've added you now, thanks09:18
MacSlowWhat's the command-du-jour to get from cozy 14.04 to utopic... the usual suspects don't work (yet)?09:30
geserMacSlow: sensible-editor /etc/apt/sources.list09:36
MacSlowgeser, I feared that answer :)09:36
pittiMacSlow: sudo sed -i 's/trusty/utopic' /etc/apt/sources.list :)09:36
pittiMacSlow: sorry, sudo sed -i 's/trusty/utopic/' /etc/apt/sources.list09:36
mvoMacSlow: or wait a tiny bit and I upload a new release upgrader09:37
MacSlowpitti, that's even more terrifying ;)09:37
pittiMacSlow: but noninteractive09:37
MacSlowmvo, ah... yeah, that sounds safer :)09:37
mvoMacSlow: tiny bit in relative terms, but I hope to get it ready by end-of-today09:38
mvoMacSlow: :)09:38
pittiuntil then: schroot FTW :)09:38
MacSlowpitti, geser: I'm so used to dog-fooding our (ui-)tools, I don't do things manually anymore09:39
MacSlowmvo, the time-resolution-granularity of days is enough of a hint for me09:40
mvoMacSlow: great, I had hoped for that09:40
pittiev_: oh, nice! https://www.oasis-open.org/news/pr/iso-and-iec-approve-oasis-amqp-advanced-message-queuing-protocol09:43
ev_wow, that's one of those things where my first reaction was, "wait, it wasn't already a standard?"09:44
ev_pitti: how's the dep8 rearchitecture coming along?09:45
ev_are you enjoying working with swift and rabbit?09:45
=== ev_ is now known as ev
pittiev: was a bit stalled while Antonio was reviewing my patches so far, and I was working on the current utopic autopkgtest bugs09:45
pittiev: but he pulled all my patches now, so I can contine (however, utopic autopkgtest regressions will still keep me a bit busy)09:46
* ev nods09:46
pittiev: yes, I do; it's quite lovely, AMQP is so much easier than having to endure the jenkins pain09:46
evyeah, tell me about it09:46
evthanks to doanac we've killed Jenkins in the airline09:47
eventirely a queue-based, microservice architecture now09:47
zygais it normal for gnome-control-center to show 13.10 in the 'details' tab (system info?)09:53
zygaon 14.1009:53
zygaoh, nice!09:54
LaneyDon't know about normal, but it is known that that's wrong09:55
LaneyIt needs manually fixing and nobody's done it yet09:56
pittishows 14.10 here09:56
zygapitti: unity or gnome?09:56
pittizyga: oh, unity09:56
zygapitti: yeah, I though the fork might be responsible09:56
zygapitti: probably shows 13.10 in 14.04 as wel09:56
zygawell09:56
pittiodd, why would that not just use /etc/os-release..09:57
seb128pitti, because it's a logo/image not text09:57
zygaohhh09:57
zygawtf? :)09:57
seb128zyga, use unity ;-)09:57
pittiWTF indeed -- a logo for a version number?09:57
seb128pitti, the ubuntu circle on top of it is in the same image09:58
zygaseb128: I ususally do09:58
zygaseb128: I wanted to see what gnome looks like lately09:58
seb128pitti, but patches are welcome if you want to change that to be dynamic and align a logo for the circle and some text with the right styling bellow09:59
seb128it's just that nobody got to do that09:59
seb128(since trusty we generate the logo at buildtime at least)09:59
zygaoh10:00
pittiseb128: right, the ubuntu version seems to do just that?10:01
=== vrruiz_ is now known as rvr
seb128pitti, yes, darkxst_ said he was going to backport those changes to g-c-c10:04
=== alkisg is now known as work_alkisg
=== work_alkisg is now known as alkisg
darkxst_seb128, pitti yes I will try look at fixing that this weekend10:09
=== gusnan_ is now known as gusnan
xnoxpitti: something is even more weird with adt though, e.g. dbus-test-runner migrated despite failing its test, and it's build-deps & build-essential are not installed whilst executing the test => but needed as it does compile tests.10:17
zygaxnox: who's the best person to talk to about CI loop failures?10:18
zygaxnox: I had strange failures that block packages that work out okay in debian10:18
zygaohhh10:23
xnoxari-tczew: given the version number, it was simply done to force that rebuild into dep-wait & rebuild only after that one is rebuild. it's save to drop that change. Note however, i had to make a change to libzrtpcpp in unicorn and it is a valid one http://launchpadlibrarian.net/174142130/libzrtpcpp_2.0.0-3build2_2.0.0-3ubuntu1.diff.gz that is needed in debian as well.10:26
ari-tczewxnox: are you going to merge that one? if not, I can do that. there are CVE fixes.10:28
xnoxzyga: ci loop failures => ask on #ubuntu-release to start the conversation.10:30
xnoxari-tczew: i really should, and forward patch to debian to get it back into sync10:30
ari-tczewxnox: ok so I'm leaving that one10:31
zygaxnox: thanks10:33
=== MacSlow is now known as MacSlow|lunch
pittixnox: jibel has a fix for the premature promotion, but not rolled out yet10:41
=== alkisg is now known as work_alkisg
=== darkxst_ is now known as darkxst
darkxstpitti, any idea why modemmanager is calling upstart's initclt script when try to upgrade?10:57
darkxst(under systemd)10:57
darkxstdpkg bug perhaps?10:58
pittidarkxst: yes, it's debian bug 683084 -- we need to merge sysvinit11:02
ubottuDebian bug 683084 in sysv-rc "Make invoke-rc.d/update-rc.d systemd-aware" [Wishlist,Fixed] http://bugs.debian.org/68308411:02
pittidarkxst: that's next on my list after I fix the "tar: unexpected EOF" breakage in autopkgtest's qemu runner11:02
pitti(as that's hampering production)11:02
darkxstpitti, ok11:17
xnoxari-tczew: merge uploaded & change forwarded https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=74665511:31
ubottuDebian bug 746655 in libzrtpcpp-dev "libzrtpcpp-dev: Missing dependency on libssl-dev" [Normal,Open]11:31
ari-tczewok11:34
mdeslaur@pilot in11:37
=== udevbot changed the topic of #ubuntu-devel to: Trusty Final released! | Archive: Open | Devel of Ubuntu (not support or app devel) | build failures -> http://qa.ubuntuwire.com/ftbfs/ | #ubuntu for support and discussion of lucid -> trusty | #ubuntu-app-devel for app development on Ubuntu http://wiki.ubuntu.com/UbuntuDevelopment | See #ubuntu-bugs for http://bit.ly/lv8soi | Patch Pilots: mdeslaur
=== _salem is now known as salem_
=== MacSlow|lunch is now known as MacSlow
=== bfiller_afk is now known as bfiller
=== pete-woods is now known as pete-woods-lunch
shadeslayerpitti: why am I getting emails from jenkins about failiures in https://jenkins.qa.ubuntu.com/job/utopic-adt-python-qt4/4/? :P13:07
=== tedg` is now known as tedg
pittishadeslayer: because you did the last upload13:09
shadeslayerI ... I did?13:09
pittishadeslayer: I restarted the test btw; it's the same bug I'm trying to work around for two days now (argh old qemu)13:10
shadeslayeraha, the silly encoding issue in python qt413:10
pittishadeslayer: accordingto https://launchpad.net/ubuntu/+source/python-qt4/4.10.4+dfsg-1ubuntu1 ?13:10
shadeslayerpitti: right :)13:10
pittishadeslayer: nevertheless, not your fault (jibel and I also get all failure mails, I'm checking them all)13:10
shadeslayercheers! :)13:11
shadeslayerbdmurray: got your email about kubuntu-driver-manager crashing, there have been 2 crashes, and both lack a stacktrace13:21
hallynpitti: the upstart jobs are not conffiles.  the cgm_extra_mounts in the upstart job is the default, to be overridden by /etc/default/cgmanager, which is the conffile13:21
hallynpitti: yes cgmanager doesn't have manyd ependences, but other things depend on it (like lxc), and in particular the curious interaction is that between cgmanager and cgproxy...  but ok i'll force myself to address it at some point13:22
hallynpitti: thanks for the pointers to systemd.unit and systemd.service13:23
pittihallyn: yes, so I guess one should depend on the other then?13:23
pittihallyn: I hope we can get rid of this stopping of cgproxy in cgmanager.conf and replace that with proper dependencies?13:24
pittihallyn: they are conffiles, as they are in /etc/init/; but what I really meant is that we don't want to encourage users to edit them13:25
hallynpitti: not sure.  the issue is that in a container we want only cgroxy to run;  on a post-3.13 kernel host we want cgmanager only;  on older kernels we want both cgmanager and cgproxy.13:26
hallynso in upstart we want to just have cgmanager be the thing you have to know about and it starts the right thing13:26
hallynyes we don't want people to edit them13:26
pittihallyn: oh, so one doesn't depend on the other, they just have different startup conditions?13:26
hallynwe want people to override them with /etc/default/13:26
hallynno cgproxy does have to start after cgmanager, if they're both going to run13:27
pittiah, "start on started cgmanager"13:27
hallynpitti: but in any case, *that* is a work item for a bit later, the more urgent thing for right now is to straighten out the session cgroups in containers in utopic13:27
hallynpitti: i just got up, gonna have some coffee, then set up a clean environment ot test in again. will you be around for another hour or two?13:28
pittihallyn: I will, but today I'm still in firefighting mode, and thus I'm not in the mood of much else :/13:28
hallynpitti: ok, np, i'll not bug you then :)  shouldn't be too hard to find the root cause.  have fun!  er i mean, good luck!  er i mean, ...  \o13:29
cjwatsonIt's surprising how long a system can survive while not being able to exec any processes13:56
cjwatsonI set an i386 LXC container to unconfined on my amd64 system, and tried to start it; it promptly installed the qemu-x86_64 binfmt handler and none of my ordinary binaries were executable13:57
hallynyou know, i seem to recall seeing that a few years ago, but i forget why.13:57
cjwatsonI had chroots around, but couldn't use sudo from them because LD_LIBRARY_PATH isn't allowed on setuid executables13:57
hallynif only we had a binfmt namespace :)13:57
cjwatsonquite13:57
cjwatsonI was able to escape without rebooting though13:57
hallynthat might be worth a blog post with details13:58
cjwatsonexecute ssh from an i386 chroot (with LD_LIBRARY_PATH set appropriately) to connect to the container I'd just started, which of course was working fine13:58
cjwatsonthen sudo inside that and update-binfmts --disable qemu-x86_6413:58
cjwatsonit might be, although I just wasted half an hour on this instead of what I was supposed to be doing :)13:59
hallynyeah i have an insane fear of running containers unconfined13:59
cjwatsonwhat I was actually trying to do was to be able to mount -o proc none $chroot/proc (etc.) inside the container14:00
jpdsAn unconfined container is not a container.14:00
cjwatsonI don't care, in this case14:00
cjwatsonwell not much14:00
cjwatsonI mean, in this instance I care that it broke my binfmt namespace :-), but I don't care about the security of the container right now14:01
cjwatsonI guess I actually want lxc-container-default-with-nesting14:02
cjwatson... except that doesn't work either14:03
hallyncjwatson: right it's only for nesting lxc.  you want to generalize that one to allow mounting proc anywhere14:04
hallyni've considered having a menu-driven set of apparmor bits for the many things ppl might want to do...14:04
cjwatsondo you happen to have an example lying around?14:05
hallynthough -with-nesting and -with-mounting seem to usually do what ppl want14:05
cjwatsonwhat I'm specifically doing is running launchpad-buildd in a container14:05
hallyncjwatson: http://paste.ubuntu.com/7379730/  should work...14:06
cjwatsonwhich wants to run http://paste.ubuntu.com/7379736/14:06
hallyncjwatson: eh,14:07
cjwatsonI guess I need a few more bits; is it possible to just say "honestly I don't care for this container, let it mount anything anywhere"?14:07
hallyncjwatson: just add "mount,' to the profile14:07
cjwatsonok, great, thanks14:07
hallyncjwatson: http://paste.ubuntu.com/7379738/14:07
cjwatsonmount: proc is not a block device14:10
cjwatsonor for "sudo mount -o proc none /path/to/proc": "mount: special device none does not exist"14:10
cjwatsonhallyn: ^- that's with http://paste.ubuntu.com/7379780/ in /etc/apparmor.d/lxc/lxc-default-mount-anything, I ran "service apparmor reload", and http://paste.ubuntu.com/7379784/ in /var/lib/lxc/precise-lpdev/config - did I miss something?14:13
bdmurrayshadeslayer: I'm looking into it, thanks14:16
=== pete-woods-lunch is now known as pete-woods
shadeslayerbdmurray: cheers :)14:17
mvoMacSlow: I uploaded the utopic release upgrader now so once that hits the archive you will be able to upgrade - at last if the package startpar is installable then14:18
hallyncjwatson: hm, that looks ike it should work14:18
=== paulliu1 is now known as paulliu
MacSlowmvo, ok... thanks for the heads up14:19
bdmurraymvo: hi, have you seen bug 1310891?14:20
ubottubug 1310891 in ubuntu-release-upgrader (Ubuntu Trusty) "cached meta-release file should not be saved if it is html" [High,Triaged] https://launchpad.net/bugs/131089114:20
hallynstgraber: ^ any idea?14:20
hallyncjwatson: huh, worked for me14:23
hallyni did "stop lxc; start lxc" rather than apparmor reload,14:23
cjwatsonI'll try that14:24
cjwatsonhallyn: no, still the same thing - where could I look to debug this?14:26
hallyncjwatson: what does /proc/self/attr/current show in the container?14:27
hallynd'oh14:28
hallyncjwatson: mount -t proc, not mount -o proc :)14:28
cjwatsonoh whoops14:28
cjwatsonyeah, that's fine, sorry for the idiocy :)14:29
hallynphew :)  sanity restored14:29
cjwatson(and "lxc-container-default-mount-anything (enforce)", FWIW)14:29
mvobdmurray: good morning! let me check14:33
mvobdmurray: yeah, totally agreeed, let me work on a fix14:34
Logan_cjwatson: can you please do a full requeue on hdf5?14:43
cjwatsonLogan_: you mean on package-import?14:43
Logan_yeah14:43
cjwatsonLogan_: don't have access, try xnox14:43
cjwatsonogra_: http://people.canonical.com/~cjwatson/tmp/lp-livefs-building.png - glorious success may be distantly visible from here14:43
Logan_cjwatson: ah, apologies14:44
ogra_wheee !!!14:44
* ogra_ hugs cjwatson ... thats awesome !!!!14:44
Logan_xnox: wanna do a full requeue on hdf5 for me? :)14:44
cjwatsonstill a fair bit of UI coding to go but it's definitely getting there now14:44
ogra_yeah, looks great14:44
xnoxLogan_: no, package-importer is stopped at the moment.14:45
Logan_oh, really?14:45
xnoxLogan_: due to utopic importing.14:45
Logan_gotcha14:45
xnoxLogan_: why else we would have 800 outstanding jobs? =) http://package-import.ubuntu.com/status/14:45
Logan_because UDD is broken? :P14:46
xnoxLogan_: hm, actually it is running.14:46
xnoxLogan_: right, it was started on the 28th. I'd rather wait for it to process backlog.14:47
Logan_ok14:47
=== roadmr is now known as roadmr_afk
hallynstgraber: should liblxc1 be depending on (cgmanager | cgroup-lite)15:15
hallynbc i can't right now easily test with cgroup-lite...15:15
hallynalso i suppose i should look into getting rid of cgroup-lite now that cgroupfs-mount (based on cgroup-lite) is in debian15:15
stgraberhallyn: so I thought about doing that, but this would have had the side effect that anyone using libvirt or who already had LXC installed wouldn't get cgmanager15:19
hallynrecommends?15:22
hallynDepends: (cgroup-lite | cgmanager) \n Recommends: cgmanager ?15:22
pittimterry: I have to reopen bug 1315185; thanks for the initial fix!15:28
ubottubug 1315185 in deja-dup (Ubuntu) "pass in trusty, but fails ADT tests in utopic" [High,Confirmed] https://launchpad.net/bugs/131518515:28
pittimterry: sorry, it seems adt-britney is horribly buggy ATM :( (we need jibel's fix rolled out)15:28
=== salem_ is now known as _salem
mterrypitti, guh15:30
stgraberhallyn: that would probably do the trick, yes15:31
stgraberhallyn: I'll do the update to the git packaging and do a daily build15:32
hallyncool, thanks15:33
mdeslaur@pilot out15:33
=== udevbot changed the topic of #ubuntu-devel to: Trusty Final released! | Archive: Open | Devel of Ubuntu (not support or app devel) | build failures -> http://qa.ubuntuwire.com/ftbfs/ | #ubuntu for support and discussion of lucid -> trusty | #ubuntu-app-devel for app development on Ubuntu http://wiki.ubuntu.com/UbuntuDevelopment | See #ubuntu-bugs for http://bit.ly/lv8soi | Patch Pilots:
mterrypitti, do we have a nice wiki page explaining the best way to run adt tests locally?15:33
mterryThe proximate fix here is easy (add dbus-x11 as a test dep) but want to make sure that's all15:34
mterryhttps://wiki.ubuntu.com/QATeam/AutomatedTesting/AutoPackageTesting15:36
LaneyI've been using http://packaging.ubuntu.com/html/auto-pkg-test.html#executing-the-test prepare-testbed / run-adt-test15:37
Laneybut I found out earlier that this isn't always the same as what jenkins does15:37
pittiLaney: right, that changed since utopic, I need to get that updated15:38
Laneyyou mean the document or the behaviour of run-adt-test?15:38
pittimterry: ^15:38
pittiLaney: the document; run-adt-test is obsolete15:38
Laneyah15:38
pittimterry, Laney: /usr/share/doc/autopkgtest/README.running-tests.gz is now the "official" documentation how to run stuff15:39
Laneyshame, it was quite easy to use15:39
pittiLaney: yes, but run-adt-test had quite some limitations15:39
LaneyI haven't grasped running adt-run directly properly yet15:39
pittiLaney: but the new one isn't really that difficult either15:39
pittithe biggest difficulty is that we don't yet have utopic cloud images15:39
Laney/current exists now15:39
pitti(but that would affect both the old and the new system)15:39
pittiooh, does it?15:40
pittinice!15:40
Laneyyep!15:40
Laneyso what's the way to set up a fresh VM with the new way?15:40
pittiso, something like15:40
pittiadt-buildvm-ubuntu-cloud -v -o /my/dir/for/VMs15:40
=== roadmr_afk is now known as roadmr
pittiyou might need -r utopic if you are still on trusty15:41
pittiand optiionally -a i386 for selecting an arch15:41
pittiso15:41
pittiadt-buildvm-ubuntu-cloud -v -r utopic -a amd64 -o /my/dir/for/VMs15:41
pittiif you run apt-cacher-ng, this is also useful: -p http://10.0.2.2:314215:42
pittiand then15:42
pittiadt-run deja-dup --- qemu /my/dir/for/VMs/adt-utopic-amd64-cloud.img15:42
pittiLaney, mterry: ^ that's how jenkins does it now15:42
pittiLaney, mterry: but if your test doesn't need particular privileges, you can also test in a schroot (everyone has that, right?)15:43
pittiadt-run deja-dup --- schroot utopic15:43
mterrypitti, I got it -- I'm using chroot pointing at my pbuilder chroot15:43
pittiyou can also test a local package, and have that built first, see README.running-tests.gz15:44
hallynstgraber: so should libpam-systemd:amd64 be all i need to have logins be placed into cgroups?15:44
pittimterry: heh, that works, too15:44
LaneyI tried with schroot and I think it had the same build-needed problem15:44
Laney(can that be?)15:44
mterrypitti, hey...  so deja-dup has autopilot tests but I could never figure out how to get them working in adt...  I assume that's gotten better?15:44
pittimterry: nothing changed in that regard recently; but I wasn't aware that it isn't working15:45
xnoxpitti: why did the dependencies changed? as in why are "build-essential & build-deps" are not kept for jobs that declare "needs-building" ?15:45
hallynwell and libsystemd-login0:amd6415:45
pittimterry: have a look at shotwell: autopilot-sandbox-run autopilot_tests15:45
mterrypitti, uh..  I can't remember exact issue, but I think I just had some difficulty getting X and such to work in adt.  But if shotwell does it, I can steal their harness15:45
pittimterry: autopilot-sandbox-run does the gory details of what needs to happen with xvfb and dbus, so you mostly just need xvfb and dbus-x11 and autopilot-desktop15:46
pittimterry: autopilot-sandbox-run is the harness, and it has existed for a long time now; but I guess it isn't well-known15:46
pittimterry: and yes, it took us a whole afternoon to figure out, it's quite tricky with xvfb15:46
mterrypitti, No, I remember that.  I had problems running it for some reason.  Will re-examine since I'm futzing with it here15:47
pittixnox: because we don't use the null runner any more, and each test/build gets a fresh VM15:47
pittixnox: if you need the build depends in your test, you need to test-depend on @builddeps@15:47
xnox pitti ah @builddeps@ is what i am after then! and that includes build-essential?15:48
pittixnox: in fact, one of deja-dup or dbus-test-runner seemed to build the pacakge twice -- once with needs-build, and another time with "make' in the test15:48
pittixnox: yes15:48
xnoxpitti: right, indeed then needs-build can be dropped, and @builddeps@ be used.15:48
xnoxLaney: ^ now we know.15:48
LaneyI did know15:48
xnoxLaney: i didn't =( =)15:48
Laneythe thing I didn't know whas about the removal of build-deps from build-needed15:48
Laneybut that is actually quite clear in the spec15:48
pittixnox: ah, so it was dbus-test-runner apparently; right, should work without needs-build15:49
Laneyit only does make check, not make15:49
pittiyeah, just with -null we couldn't enforce that15:49
pitti(and -null had quite a few more problems and also required this whole set of additionla scripts around it, which is why we wanted to abandon it)15:50
LaneyI have a weird feeling it's testing the built tree though15:50
xnoxpitti: are adt VMs running in US timezone or UTC? or no useful clock at all?15:50
pittibtw, I think I fixed the weird "tar: unexpected EOF" thing now, so tests should not fail any more due to that15:50
pittixnox: let me check15:51
pittixnox: it copies the host's timezone15:51
pittialderamin: America/New_York15:51
pittialdebaran: Etc/UTC15:51
xnoxpitti: yeah, my job was on alderamin.15:52
pittialbali: Etc/UTC15:52
pittiwazn: Etc/UTC15:52
pittibut that "copy from host" behaviour hasn't changed15:52
pittimaybe the host timezones changed recently15:52
pittixnox: if you need a particular one, it's probably best to export TZ= in your test?15:53
pittihm, what's wrong with ubiquity!?15:54
pittioh, uninstallable on amd6415:54
pittimterry, Laney: oh, for completeness: jenkins runs against -proposed, so if you want to do that, run "adt-run <package> -U --apt-pocket=proposed --- ..."15:55
pittihm, dholbach already offline15:55
pittiI'll see to catch him next week to get http://packaging.ubuntu.com/html/auto-pkg-test.html updated15:55
LaneyI think that's auto deployed from the guide15:56
Laneyin bzr15:56
Laney(is it?)15:56
pittiso uninstallable libtimezonemap1 seems to break quite a lot ATM15:57
* pitti binNEWs it to put it out of its misery15:57
xnoxpitti: no, i want job-logs and console-logs have sanity =) when i view the adt job artifacts.15:58
xnoxpitti: which are comparable across different runners.15:58
xnoxpitti: there is no reason to use New-York timezone =) on build slaves.15:59
xnoxe.g. console ouput says - adt-run [2014-05-02 11:52:40]: version @devel@, yet jenkins job started at (02-May-2014 15:52:39) which is mildly hallarious =)16:00
pittixnox: heh, yes; I'll create an RT to have it changed16:02
xnoxLaney: i wonder if the test is now busted, and should actually just be "#!/bin/sh true" and just rely on needs-building, cause i bet the directory is not RW either.16:05
xnoxpitti: do you have some inside ways to inspect what http://d-jenkins.ubuntu-ci:8080/view/Utopic/view/AutoPkgTest/job/utopic-adt-dbus-test-runner/ARCH=amd64,label=adt/11/console is actually stuck doing?16:05
hallynstgraber: dangit, now with lxc-default-with-nesting profile and lxc.mount.auto=cgroup:mixed, and cgroup-lite on host but no cgmanager, logins give me:16:10
hallynAn error occurred while mounting /sys/fs/cgroup.16:10
hallynkeys:Press S to skip mounting or M for manual recovery16:10
hallynhitting 'S' proceeds just fine, all cgroups are mounted16:10
hallynso why is mountall doing that16:11
xnoxpitti: i am failing to parse http://d-jenkins.ubuntu-ci:8080/view/Utopic/view/AutoPkgTest/job/utopic-adt-bzr/ARCH=amd64,label=adt/1/ it appears fully passed to me....16:13
xnoxoh stderr output?16:13
hallynwell removing the line from /lib/init/fstab works, but since there is something mounted on /sys/fs/cgroup i would expect mountall to ignore it16:15
xnoxhallyn: how is /sys/fs/cgroup mounted? ans is it racing with mountall?16:17
xnoxhallyn: what are the options used? if all are the same, it shouldn't matter if mountall mounted it or somebody else did.16:18
hallynxnox: 77 72 0:36 / /sys/fs/cgroup ro,relatime - tmpfs cgroup_root ro,size=10240k,mode=75516:18
hallynit's mounted by lxc before init ever starts16:19
hallynthen under that mount are one (bind) mount per cgroup16:19
hallynso is mountall objecting bc it is called 'cgroup_root' instead of 'none'?16:19
xnoxhallyn: it's objecting to it being ro, and it remounts it rw.16:19
hallynbut its optiona16:20
xnoxhallyn: because raisons =)16:20
hallynl16:20
xnoxhallyn: it's optional, in a sense that it may be missing. however when it is not missing, it will try to remount it rw.16:20
xnoxslangasek: what was the right way to make mountall not do silly things to things that are already mounted ^16:20
xnoxhallyn: lxc could write out it ro into /etc/fstab inside container? or is that a no-no16:21
hallynas i recall there wasn't a right way, and i've always disagreed (and slangasek disagreed with me :) about that behavior :)16:21
hallynyeah i don't want it to have to mess with /lib/init/fstab - that is so 11.1016:21
hallynotoh,16:22
hallynthere's really no reason why *that* mount needs to be ro16:22
stgraberso why is /sys/fs/cgroup ro to begin wtih?16:22
hallynthe cgroupfs mounts, yes, but not that one16:22
hallynstgraber: 'an abundance of caution' by Christian i think16:22
hallynlemme rebuild and see if changing that fixes it16:22
stgraberright, just make it rw, problem solved, it's space restricted anyway and the user is root so nothing prevents them from remounting or from mounting another tmpfs on top of it16:22
pittixnox: dbus-test-runner> yes, I can ssh into the thing and look16:23
pitti     |-sh---su---sh---with-bustle---make---bash---bash---make---make---bash---make---bash---bash---test-libdbustes---gtester-+-lt-t+16:25
pitti     |                                                                                                                       `-{gma+16:25
pittihmm, not that helpful16:25
* xnox *chuckles* 16:26
xnoxi'm glad Laney is TIL now =)16:26
Laneyis it producing a test-suite.log?16:26
pittithere's nothing using noticeable CPU16:26
pittiLaney: apparently not16:26
xnoxi ponder if there is a run-away dbus process or some such16:26
xnoxtest-dbus-out-of-every-buildd-even-implemented-by-tedg16:27
xnoxs/even/ever/16:27
xnox=)16:27
pittixnox, Laney: http://paste.ubuntu.com/7380560/ in case that helps16:27
pittiyes, two zombie dbus-daemons16:27
tedgWe had some issues with that, but then we even added a "clean up the runaway processes process" :-/16:28
tedgPerhaps we just need to run it all under a session Upstart.16:28
xnoxtedg: well, or under a cgroup from cgmanager.16:29
hallyngah.  he actually remounts it ro twice16:29
xnoxtedg: and kill cgroup at the end?16:29
tedgxnox, That'd probably work.16:29
tedgIs there like a libcgmanager?16:30
pittiLaney, xnox: I really need to go AFK now; want me to kill that job and you run it locally in QEMU, or want me to just retry it?16:30
xnoxtedg: there is only libcgmanager.16:30
xnoxtedg: and dbus api.16:30
Laneypitti: that's already a retry, feel free to kill it (would be good if you could get test-libdbustest-test.log first though)16:30
xnoxpitti: i've killed it once, i can kill it again. but if we kill it again, britney/adt will request it again no?16:31
pittixnox: not automatically16:31
pitti./tmp/apt0-build/dbus-test-runner-14.04.1+14.04.20140320/tests/test-libdbustest-test.log16:31
pittithat's the only one16:31
Laneyanything useful in it?16:32
pittiLaney: could be: http://paste.ubuntu.com/7380593/16:32
Laneyinteresting16:32
pittiooh16:32
pittiit seems to remember its full build directory in the test run16:33
pittiso we might need to fully restore that in a new VM16:33
* pitti kills the VM for now16:33
hallynok patch sent for that16:35
hallynstgraber: would you expect that logind could move me into a cgroup without cgmanager (with just /sys/fs/cgroup/* mounted)?16:36
* pitti bids good night, have a nice weekend everyone!16:36
stgraberhallyn: yes16:37
stgraberhallyn: and it did last I tried16:37
hallyndesn't seem to be working for me.16:41
stgraberhallyn: I just tried on a machine here which never had cgroup-lite or lxc installed and I'm in the right spot of the systemd cgroup as expected16:44
hallynstgraber: without cgmanager installed on the host, you get int eh right cgroup in a container?16:47
stgraberhallyn: ah, not in the container, just checking on a host without cgmanager but with cgroup-lite installed16:47
stgraber(was just confirming my systemd change worked as expected and it does)16:48
hallynfor some reason in a container with cgroups mounte it's not workign.  (yes on the host it's workign for me)16:54
=== bfiller is now known as bfiller_afk
stgraberhallyn: anything vaguely useful in /var/log/upstart/systemd-logind.log?16:59
hallynyes, for anyone who asks, something went weird with qemu utopic - i don't see where 2.0.0+dfsg-2ubuntu1 ever made it to utopic, and so i had pushed 2.0.0~rc1+dfsg-0ubuntu4 after that...  i dunno...  the new 2.0.0+dfsg-2ubuntu2 should straighten it out16:59
hallynyeah...17:00
hallynNew session c1 of user ubuntu.17:00
hallyncgmanager: cgm_enter for controller=systemd, cgroup_path=user/1000.user/1.session/user/1000/c1.session, pid=476 failed: Escape request from different namespace requires a proxy17:00
hallynRemoved session c1.17:00
hallyn/proc/self/fd/9: 4: ulimit: error setting limit (Invalid argument)17:00
hallynbut cgmanager is not running17:00
hallynnor is cgproxy.  what on earth?17:01
stgraberhmm, well, it shouldn't ever hit that code unless cgm_connect succeeded...17:01
hallynoh maybe those are old17:01
hallynyeah the file doesn't exist on a new container17:02
hallynoooh!17:04
hallynthe problem is that lxc is not mounting the name=systemd cgroup17:04
stgraberthat'd explain it17:05
hallynok well maybe that also explains the issue with cgmanager - are you telling me taht17:05
hallynsystemd-logind takes your path in name=systemd and uses that as the base for your new cgroup?17:06
hallyn4:name=systemd:/user/1000.user/1.session/user/1000.user/1.session/user/1000.user/c2.session17:08
=== zequence_ is now known as zequence
slangasekxnox, hallyn: the "optional" refers to it being optional because the filesystem support may be missing in the kernel, and if the filesystem is supported it's not considered optional at all.  As to the specifics of why mountall thinks the pre-existing mount is insufficient, I'm not sure; it certainly could be the mount name, or the ro status, it definitely won't be anything to do with the other mount options since today we *fail* to remount ...17:23
slangasek... filesystems on boot that were mounted with wrong options :)17:23
hallynright, it was ro status17:24
hallynwe've updated lxc to not mount that ro, as it was rather silly17:24
hallynstgraber: ok, i think i've got it.  the problem is again that lxc is ignoring name=systemd (because it's not in /proc/cgroups presumably).17:26
hallynso init in container has 10:name=systemd:/user/1000.user/1.session17:26
hallynwhile all others are 9:perf_event:/lxc/u217:26
hallynok, so this is fixable.  phew.  and it wasn't cgmanager messing up.  phew.17:26
hallynbut it is src/lxc/cgmanager messing up17:28
hallynthe heck...  and heres a tiny memory leak in the code i have to change17:39
hallynstgraber: phew, with a patch to cgmanager and to lxc that's straightened out.18:27
=== blkperl_ is now known as blkperl
=== bfiller_afk is now known as bfiller
=== roadmr is now known as roadmr_afk
infinityLaney: I can't commit to the libtimzonemap bzr trunk, can you commit and tag my 0.4.3 upload?  Kthx.19:42
=== roadmr_afk is now known as roadmr
achiangstgraber: out of curiosity, what desktop environment does edubuntu use?20:43
infinityachiang: unity20:45
achianginfinity: thx20:45
stgraberachiang: we offer both Unity and gnome-flashback. Both are always installed and you may choose the default in the installer.20:47
achiangstgraber: thanks!20:47
infinityOr that. :)20:47
achiangstgraber: i'm proxying a question for someone else -- is it possible to use unity + LTSP if the end client doesn't have very powerful graphics?20:48
achiangs/possible/recommended/ ?20:48
stgraberachiang: it may be possible though my experience so far is that GL over the network is a pain and only a very small subset of Intel cards can manage it. The others will do software rendering on the server side and stream the bitmaps over the network which you almost certainly don't want.20:48
stgraberachiang: that's why even if you choose Unity as your desktop environment in Edubuntu, LTSP will always default to gnome-flashback20:49
achiangstgraber: got it, thanks!20:49
rdzhey all. i'm looking for the right channel to ask about writing upstart jobs. is this the place?20:49
infinityrdz: A good  place to start is http://upstart.ubuntu.com/cookbook/20:50
rdzinfinity, thanks.. i already checked that and i couldn't find an answer to my problem there20:51
argeshallyn: fyi i filed bug 1315530 about what we discussed. I think there may be qemu packaging fix in there as well21:04
ubottubug 1315530 in apport (Ubuntu) "apport-retrace doesn't work with qemu" [High,New] https://launchpad.net/bugs/131553021:04
tjaaltona friend of mine claims that lts->lts upgrade removed mysql & owncloud including the data, which I find hard to believe..21:12
tjaaltonupgrade to trusty21:12
tjaaltondo-release-upgrade, needed -d to work21:12
hallynarges: so does it also happen when you launch a vm by hand using /usr/bin/qemu-system-x86_64 ?21:13
argeshallyn: i can try that as well21:13
infinitytjaalton: Needing -d to work is a feature, we don't offer lts->lts upgrades until the first point release is out.21:15
infinitytjaalton: As for the removal of packages and all his data, that sounds pretty nasty.  A reproducer consisting of initial pre-upgrade package list would be nice.21:16
tjaaltoninfinity: yeah I'm investigating it, having root to the machine helps..21:18
tjaaltonthough it got rolled back to a previous snapshot already21:18
tjaaltonbut at least owncloud was not from an official source21:18
infinitytjaalton: Ahh, that doesn't help much then. :/21:18
dobeyhow would an *upgrade* remove *any* data?21:19
tjaaltonright..21:19
infinitytjaalton: If owncloud is being removed on upgrade due to conflicts or similar, and it's not our package, we can't fix if it has braindead maintainer scripts that wipe your data on removal.21:19
tjaaltonyeah I'll review those21:19
infinitydobey: If an upgrade removes a package, and the package wipes data on remove (instead of purge), then you'd get there.21:19
dobeyoh i guess if maintainer scripts are insane, yeah21:20
argeshallyn: same results21:20
dobeyinfinity: even on purge, removing apache isn't going to remove all my web sites i configured in /srv/21:20
infinitydobey: No, true.  But some packages assert stronger ownership over their data, and will purge certain bits of /var/lib (etc) on purge.  Which is sort of what "purge" means, so not entirely unheard of.21:21
infinitydobey: But a do-release-upgrade never purges, only removes, so that point is moot.21:21
dobeyright21:21
tjaaltonhmm not owncloud to blame, it only has a simple postinst to reload apache21:22
infinitytjaalton: There was a mention of mysql, though.  Is this owncloud data in a mysql DB?  And did removal of mysql-server do Very Bad Things?21:22
infinityIf so, that's a bug we *can* (and effin' should) fix.21:22
tjaaltonyeah it got migrated to the mysql backend aiui, the default one didn't perform21:23
infinityI'm a silly sort of person that likes to think people read the output of do-release-upgrade and panic when it says "I'm about to remove your database server", but reality isn't so pleasant.21:23
tjaaltonwell it's friday night and all.. and it wasn't really that critical a server21:25
dobeyhah yeah. it's like people complaining that ubuntuone got removed on upgrade. and i tell them "well you clicked the button to remove those packages, if it did."21:25
tjaaltonand he took a snapshot just in case21:25
infinitytjaalton: +1 for the pre-upgrade snapshot.21:25
infinityI look forward to a day when btrfs is trustworthy enough for us to do that sort of thing by default.21:25
infinity(I'm not entirely unconvinced that we shouldn't have just shipped LVM by default on all systems a decade ago, and used cow LVM snapshots for upgrades...)21:26
tjaaltonthis is a virtual machine, so it's a snapshot of the vm stage21:28
infinityYeah, VMs make this much simpler.21:28
infinityThey make everything else harder, but hey, at least you get some wins. :P21:29
tjaaltonso does d-r-u honour held packages?21:29
tjaaltonI'd guess not21:30
infinityPossibly?21:30
infinityIt's really just a thin wrapper around python-apt, by default it would respect dpkg holds, but it's entirely possible that's been overridden on purpose too.21:30
tjaaltonyeah guess it's easier to just look at the source :P21:31
=== timrc-afk is now known as timrc

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!