/srv/irclogs.ubuntu.com/2018/10/30/#ubuntu-devel.txt

=== Unit193 is now known as HeadlessHorseman
=== RAOF is now known as Guest50892
=== Guest50892 is now known as RAOF
infinitytumbleweed: Around?08:05
kklimondawhat's the procedure for building custom ubuntu core images that can be installed on servers, without having to go through snapcraft? I'm curious if ubuntu core is an alternative to other "container OS", but the documentation is lacking.08:12
kklimonda(or rather geared towards IoT deployments)\08:12
LtWorfkklimonda: i use debootstrap08:41
kklimondaLtWorf: I don't think you can build Ubuntu Core images using debootstrap, given how different they are from a normal ubuntu distribution08:47
LtWorfi think it's the same, just lacking systemd08:48
LtWorfand very minimal08:48
kklimondaare we both talking about https://www.ubuntu.com/core ?08:49
mwhudsonkklimonda: ubuntu-image is the thing that makes ubuntu core images08:49
mwhudsonkklimonda: what would you want to customize in your image?08:50
kklimondamwhudson: different kernel, specific packages, container runtimes, at this point nothing and everything - I'm just looking into various available options, thinking about future upgrades of the servers we have.08:55
kklimondaat this point I'm trying to build a model of how all the ubuntu core parts fit together08:56
mwhudsonkklimonda: well for the most part, adding packages is just a matter of installing more snaps, having them baked into the image is mostly an optimization08:56
mwhudsonthe kernel would be different though08:57
dokotsimonq2: is LP: #1660108 still an issue with GCC 8?09:01
ubottuLaunchpad bug 1660108 in gcc-6 (Debian) "Since 6.3.0-3ubuntu1 some acc autotests fail with "atomic_base.h:390:7: error: inlining failed in call to always_inline"" [Undecided,New] https://launchpad.net/bugs/166010809:01
=== RAOF is now known as Guest36026
kklimondamwhudson: different, as in unsupported and frowned upon, or just poorly documented?09:18
mwhudsonkklimonda: just different09:18
kklimondamwhudson: is core even meant to compete with project atomic, rancheros etc.?09:18
mwhudsonkklimonda: i certainly know nothing about how the kernel snap is maintaineed09:18
kklimondamhm09:18
LocutusOfBorgseb128, https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa/+packages11:13
LocutusOfBorgI uploaded the new poppler if you want to grab it :)11:14
LocutusOfBorgI syncd it with the latest debian fixes, so now the delta is about two patches and one line in rules file11:14
seb128LocutusOfBorg, thx, you need sponsoring or...?11:14
LocutusOfBorgseb128, I need somebody who does upload it :) if you want to do the transition, please go ahead :)11:21
LocutusOfBorgI think we need some symbols refresh11:22
seb128LocutusOfBorg, I can have a look, is D open to upload yet?11:25
dokono11:28
LocutusOfBorgdoko, your "no" made his connection drop!11:36
LocutusOfBorgseb128, no, it is not open yet, but we should try to push when it opens :)11:54
seb128LocutusOfBorg, right, I can do that, thx for the work!11:54
LocutusOfBorgwe can manually sync rdeps from debian, since they have lots of 0.68 fixes pushed already11:54
LocutusOfBorgand fix new failures together11:55
LocutusOfBorgI asked pochu to refresh symbols files and push the new version in experimental too11:55
seb128thx11:56
=== M_hc is now known as _hc
chiluk@apw, @cking, we've noticed a roughly 30% increased cpu usage of our web apps when moving from 4.17->4.18 *(mainline kernels), have you guys had any similar reports?15:16
udevbotError: "apw," is not a valid command.15:16
chilukapw, cking, we've noticed a roughly 30% increased cpu usage of our web apps when moving from 4.17->4.18 *(mainline kernels), have you guys had any similar reports?15:16
apwchiluk, i have not indeed sforshee ^ ?15:18
chilukbasically we discovered it when one particular cgroup cpu bound app started being throttled when we moved to 4.18...15:18
sforsheechiluk: first I've heard of anything like that15:18
chilukyeah a performance regression was expected 4.14->4.15 due to kpti/meltdown/spectre fun... but we're definitely hitting something 4.17->4.18... anyhow.. ping me if you guys hear anything..15:19
sforsheechiluk: the only thing that comes to mind recently that might have caused anything like that is l1tf mitigations15:22
chilukyeah I saw that.. I'll check llc hit % later today.15:22
chiluk<- somehow became a performance guy in the last few weeks.15:22
chiluksforshee:  wasn't l1tf pushed into 4.19?  did that get backported onto the stable trees?15:23
sforsheechiluk: I'd guess it got backported assuming the stable was maintained at that point. I don't remember exactly when all of that landed15:24
chilukyeah I'll go look it up.15:24
chilukthanks for the idea sforshee15:24
sforsheechiluk: https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html15:26
tyhickschiluk: L1TF first landed in a mainline release in 4.19 but they landed in -stable trees, as well15:35
chilukinteresting.. afaict whatever I'm hitting hit us during the 4.18 development cycle so it's unlikely to be l1tf.15:35
tyhickschiluk: additionally, you should only see a perf hit from L1TF mitigations if these web apps are running inside of VMs15:36
tyhickschiluk: the bare metal mitigation for L1TF was very simple and should have a negligible perf hit15:36
chilukyeah we're baremetal with cgroup containers in mesos... so yeah.. I knew there was a reason I ignored l1tf...15:37
chilukwell if you guys hear anything with the recent release of cosmic let me know, and I'll be sure to return the favor.15:39
didrocksxnox: hey, small question, do you know where is the ubiquity translation installed on disk for all languages? (I'm particularly interested in the minimal translation we ship for languages we don't fully ship on the iso; I guess it's only ubiquity ones?)16:10
xnoxdidrocks, ubiquity assembles it's own strings, a few stock strings, and d-i strings. and yes ships it itself.16:11
didrocksxnox: any particular files they are in? Like looking for "mhr" on disk only shows the slideshow files + some generic example-content and such16:12
didrocks(but ok, the main info I was interested in was "everything related/we ship for those are part of ubiquity, and nothing else")16:13
xnoxhmmmmm16:13
cjwatsonMost of it should be in /var/lib/dpkg/info/ubiquity.templates IIRC16:13
cjwatsonOr somewhere similar (and then loaded into the debconf db of course)16:14
xnoxdidrocks, yes, that!16:14
didrocksoh right, they are coming from debconf…16:14
* xnox just found it =)16:14
didrocksthanks cjwatson & xnox :)16:14
* xnox is slow =)16:14
xnoxit's 15M templates file16:14
didrocksunsurprisingly :p16:15
didrocksxnox: in subiquity, how do you do anything special with the multi-layer/union fs system? Like creating the multi-layers, rebuilding the dpkg database in the end (depending on the amount of layers you touched) and anything related? Also, any doc about those? :)16:34
xnoxdidrocks, there is nothing special about that.16:36
didrockshum, you have certainly a way to build upon the multi-layer system that you told was available at FOSDEM?16:36
didrocks(when we talked about the minimal installation)16:36
xnoxdidrocks, it's just perfectly stacked. as in bootstrap minimal squashfs, copy, mount overlay install more, unmount & copy aside, mount again (multi-lower), install more, unmount etc.16:37
xnoxdidrocks, let me point you at the code16:37
xnoxthus each layer has it's own dpkg db, cause it got update.16:37
xnoxthus each layer has it's own dpkg db, cause it got updated.16:37
didrocksah16:37
didrockshum16:37
didrocksbut we are going to have a lot of layers16:38
didrockslike for langpacks16:38
xnoxhttps://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/ubuntu-server/hooks/030-root-squashfs.binary16:38
didrocksso, it means a lot of combinations16:38
xnoxhttps://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/ubuntu-server/hooks/031-maas-squashfs.binary16:38
xnoxhttps://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/ubuntu-server/hooks/032-installer-squashfs.binary16:38
xnoxdidrocks, i did say, it's not gonna work to out of order slot in language packs....16:38
xnoxdidrocks, so when discussed with e.g. seb128 we did say, one would have to pre-install the 6 language packs into the installer layer (live-boot only), and install the only required language pack, in target, from the .deb from the pool.16:39
xnoxit does mean, that effectively all lang packs are shipped twice.16:39
didrocksnot really nice16:39
xnoxunless one can magically reconstruct a deb out of livefs16:39
didrockscan we somehow keep the pacakge names for a given layer and register them in dpkg without unpacking/running things?16:40
xnoxdidrocks, the layers that this works for is "minimal", "full", "installer".16:40
didrocksshould be quite safe for langpaks16:40
didrocksor just generate the now (12) top combinations16:41
xnoxdidrocks, locales regenerated?16:41
xnoxyou do call16:41
infinitylangpacks are going to have to be the top layer, so it can be peeled off before installation, and we'll have to just reinstall the deb.16:41
didrocksyeah16:41
xnox/usr/share/locales/install-language-pack "en" "" "$2" || true16:41
infinityAnything else is a bit too magic.16:41
xnoxpostinst16:41
didrocksthat's exactly where I'm going to head to16:42
didrockswasn't in my previous diagram, but as a top layer, for the languages we fully ship, doesn't sound that crazy16:42
xnoxinfinity, for old ddebs we did have magic to reconstruct a deb, out of the filesystem, no?16:42
didrocks(12 because 6*2: minimal + langpacks)16:42
didrocksfull*16:42
xnoxno, we will not do that16:42
infinityxnox: Reconstructing debs is a big ick.16:43
xnoxok16:43
infinityIt's a bit of a shame to ship them twice, but it's not really a big deal.16:43
xnoxminimal, full, installer (with ubiquity and langpacks in it); and then pool needs to have: ubiquity, langpacks again.16:43
xnox(because oem-config, and langpacks)16:43
infinityWe can maybe come up with some vaguely "elegant" way to stitch together dpkg status snippets, but I wouldn't want to consider that for a first cut.16:44
xnoxalso pool logic needs to change, not to exclude things that are already shipped.... and somehow maybe based that off the full squashfs, rather than the minimal one16:44
xnoxit would be nice if langpacks were innert; without maintainer scripts.16:44
didrocksor minimal, full, installer (with only trads for installer) and lang-minimal-desktop-en, lang-minimal-desktop-fr, … lang-full-desktop-en…16:44
xnoxand like have triggers to trigger things at most.16:44
didrocksthen -{lang} are for the 6 we fully ship16:44
didrocksand lang-full-desktop-en stack on top of lang-minimal-desktop-en,16:45
didrocksas infinity said, at the top of the whole layer stack16:45
infinityxnox: Changing langpacks to declarative with triggers would be easy, especially since we control all of them.16:45
infinityxnox: But you still have the status DB to contend with.16:45
xnoxinfinity, that's the rpm command to rebuilddb?! =)16:45
* xnox giggles16:46
infinitydidrocks: Err, why are you doing en and fr stacks?16:46
infinitydidrocks: That implies a user making a choice *at boot* before you stack the FSes.16:46
didrocksinfinity: ah, for the live session…16:46
infinitydidrocks: Surely, you just want all the langpacks running in the live env.16:46
xnoxinfinity, cause didrocks wants to ship each langpack 3 times, clearly ;-)16:47
didrockswell, I don't think we want to install them at boot time from the pool either…16:47
didrocksxnox: no, they would only ship once that way16:47
infinitydidrocks: No, you want installer+languages in the top stack.16:47
xnoxdidrocks, how would live session have them all?16:47
infinitydidrocks: And then langpack debs in the pool to install the user's selected language in the target.16:47
xnoxdidrocks, or you want to stack live session, with a broken dpkg database?16:47
xnoxdidrocks, ubiquity can install the langpacks during installation; the same way it currently removes redundand ones.16:48
xnoxdidrocks, and i think installing the one you want is quicker than removing 5 you don't.16:48
infinitydidrocks: I get what you're trying to do, but it's not going to work.16:48
didrocksyeah, but one of the goal would have been quicker installation time :p16:48
xnoxdidrocks, installing one lang pack is quick. removing 5 is not.16:48
didrocksbut hem, yeah, live session…16:48
infinityxnox: Weeeeell.16:48
didrocksxnox: not only the langpack, there are a lot of packages16:48
infinityxnox: Installing langpacks calls localegen, that's not quick.16:48
xnoxcan we preship that too? or not?16:49
didrocksand some of the issues we currently have in ubiquity is that it's not in sync with locale-chooser on what to installer for instance :p16:49
infinityThat said, we could change the langpacks to ship pre-compiled locales.16:49
xnoxif clean-install copy this, otherwise regen.16:49
infinityNot sure why we don't actually.16:49
xnoxdidrocks, when you say "not only the langpacks" what else do you mean?16:49
xnoxdidrocks, current minimal install is borked and we know that. but we will have minimal squashfs to copy from and desktop (full) squashfs to copy from.16:50
didrocksxnox: see the live seed, dictionaries, libreoffice-something-{lang}, thunderbird-*16:50
xnoxthen the "extra" packages to install is the langpacks + bootloader.16:50
infinityAnd some languages pull in input methods, etc.16:50
didrocksyep16:50
xnoxalthough imho we should preinstall bootloaders packages in minimal too, and and just pre-install both grub-pc & grub-efi16:50
infinityAnyhow, there's no "clean" solution to that right now.16:50
didrocksyeah, sounds like it16:51
infinityThe dpkg database being a single flat file makes it tricksy.16:51
infinityYes, we could write something to deal with that.16:51
xnoxthere is no db.d/ true16:51
infinityBut I don't want that to be a blocker or even a near-future goal for stacked installers.16:51
xnoxdidrocks, i think having minimal + desktop + installer squashfs is a win, despite the langpacks mess.16:52
xnox...16:52
xnoxinfinity, didrocks - why do we not preinstall 6 languages by default?!16:52
didrocksxnox: yeah, I wanted to go the extra mile and have something really nice, but doesn't sound like it's doable in the short term16:52
didrocksbecause live session?16:52
xnoxi mean for target16:52
didrockshum, it's quite large16:52
didrockswe are already taking a non negligeable amount of spaces for other things16:53
infinityxnox: langpacks aren't small.  It's kinda why we have them.16:53
didrocksotherwise, we would ship them in the initial debian binary package :p16:53
didrocksand yeah, especially considering input methods, dictionaries…16:53
xnoxen is 6M16:53
xnoxand we need en always, no?16:53
xnoxsorry 7M16:54
didrockscounting all extra packages?16:54
didrocksno, talking to Gunnahr, we don't16:54
didrocksand one of the goal is to stop installing it for everyone16:54
xnoxmaybe not, this is pack-en-base gnome-en-base16:54
didrocksand dictionaries…16:54
didrockslook at my seed reorg, it should make clear what we install for each language16:54
didrocksalso, there are some dep chains IIRC16:54
xnoxdidrocks, just make them snaps16:54
didrockslike if we install libreoffice-{lang} unconditionnally16:55
didrocksthis pulls back libreoffice16:55
didrockscontradicting minimal :p16:55
didrocksxnox: sure, note that I said "we are already taking a non negligeable amount of spaces for *other* things" ;)16:56
xnoxdidrocks, infinity - i think there are two options. Preinstall langpacks in minimal squashfs; stack desktop; live -> remove langpacks (existing codepaths really)16:56
xnoxor have clean minimal, clean desktop, and preinstall langpacks in live; and install langpacks from /pool to /target.16:57
didrocksI guess installing what's needed is what makes the most sense, as we are going to rerun locale-gen anyway in both cases?16:57
xnoxand preinstall in minimal squashfs, is actually the path of least resistance / least change, imho.16:58
didrockswhy not in the live one? sounds like we want to keep it on top if we install what needed afterwards16:58
didrocksremember that we can't install everything langpack-related in minimal, due to deps on libreoffice, thunderbird…16:59
didrocksor even, keeping a "default language" stack, maybe on top of everything? dunno yet16:59
didrocksanyway, at least, I guess my dream is broken :p but thanks for the info and code pointer xnox & infinity17:00
xnoxdidrocks, preinstall language-pack-base-[6 langs] in minimal; preinstall thunderbird-locale-[6 langs] in desktop; live has no new langpacks.17:01
didrocksxnox: why it it better than its own layer on top?17:01
xnoxdidrocks, then ubiquity removes un-needed langpacks as it did before.17:01
didrocksah, for removing17:01
xnoxdidrocks, because you don't need to ship a second copy in the pool, this way.17:01
didrockshum, unsure if it's better than installing17:01
didrockstrue17:01
didrocksbut 5 locale-gen calls potentially instead of one17:02
didrocksand more packages to remove17:02
xnoxdidrocks, and minimal, is slightly faster than now, as it only removes langpacks; and doesn't need to remove e.g. thunderbird packs.17:02
didrocksyep17:02
xnoxit should be just one locale-gen call.17:02
xnoxcause it should be trigger based, on remove; just like kernels....17:02
didrocksis it really a trigger, not posinst? I remember to have seen it running multiple times on upgrade17:02
didrocksok, need to think about it anyway, but good food for thoughts :)17:03
xnoxif it runs multiple times, we can fix that, we have triggers tech to make it a single call per transaction.17:03
xnoxminor bug17:03
didrocksI wonder how we could keep clear definition of what's needed langpack-wise17:03
didrocksand share that with check-local-support17:03
didrocks(as now, the definition of a complete locale is going to be split in 2 places)17:03
xnoxdidrocks, also need to check if delta-squashfs can be actually generated for all cases; and how big they are.17:04
xnoxdidrocks, by doing reverse-stacking; e.g. make the big-fat all in one squashfs; and generate delta squashfs with things removed.17:05
xnoxas in negate 5 langauges; times 6;17:05
didrockstimes 6, not times 2 ?17:05
didrocks(minimal and full)17:05
xnoxand again negate desktop down to minimal and negate 5 languages; times 617:06
didrocksbut it's a good idea :)17:06
xnoxno idea if it works17:06
rflemingsarnold: (responding to your message from last week) You say you're skeptical of userspace nfs ... I'd love to have it so I can mount to my NAS on demand through Deja Dup :)17:06
xnoxcause overlayfs squashfs for removals, i wasn't sure it was working right.17:06
rflemingsarnold: or through nautilus17:06
didrocksxnox: sounds like an interesting things to try out17:06
didrocksdoesn't impact us on doing the langpacks in minimal + full stack as decided17:07
sarnoldrfleming: and 'mount -t nfs -osoft,rw nas:/exports/home/fleming /home/rfleming/nas' doesn't do the trick?17:07
didrocksand replace the removal part with this if we see it works17:07
rflemingsarnold: it does... but I have to do it.  duplicity/deja-dup does backups whenever it feels like it.  I'm also using it for my laptop :)17:08
rflemingsarnold: maybe I'm just salty because Windows CIFS is better implemented in nautilus than NFS is :)17:09
xnoxdidrocks, "as decided" *eyebrow*17:09
xnoxdidrocks, deciding things without a feasible implementation?! interesting17:09
didrocksxnox: ? it's exactly what you wrote: "preinstall language-pack-base-[6 langs] in minimal; preinstall thunderbird-locale-[6 langs] in desktop; live has no new langpacks"17:10
didrocksthis is shipping "the langpacks in minimal and in full" stacks17:10
xnoxdidrocks, so far, this is all madness =)17:11
didrockswhy would that be different to what is currently in server?17:11
didrocks(with subiquity)17:11
xnoxdidrocks, server has no langpacks17:11
didrocksyou are that afraid by the size of each stack?17:11
xnoxno17:11
xnoxit's just genuinely not useful17:12
xnoxwe do locale, and keyboard setup.17:12
xnoxthat's it.17:12
didrocksyeah, but this is not "without a feasible implementation" then17:12
didrocks(I'm only talking about that part here, then a second step is to look at your negative langpacks squashfs delta idea)17:13
xnoxdidrocks, langpacks have never been stacked to-date. the only stacking we did was strict supersets, in order, not this tree of things.17:13
didrocksit's going still to be strict supersets?17:13
didrockslive17:13
didrocksfull desktpo17:13
didrocksminimal desktop17:13
xnoxstrict supersets should work, yes.17:13
didrockswith full & minimal having langpacks for the 6 shipped langs17:13
xnoxdidrocks, also, let's stop calling it live; but call it `installer`17:14
didrocksyeah, that's the part I set to "decided" :p17:14
xnoxcause that matches e.g. subiquity iso.17:14
didrocksagreed17:14
xnoxdidrocks, and installing langpacks from /pool is imho "cleaner"17:15
xnoxbut time/disk-space17:15
xnoxbut time/image-space17:15
didrocksyeah17:15
didrockslet's see once this "default stacks" are here if we can look at the negative delta and also their impact on disk size, it's just an install time optimization in the end as it's the "same" than removing debs, but pre-generated17:16
shadeslayerhey hey, anyone know why adding a library path (/usr/lib/aarch64-linux-gnu/mali-egl/) to /etc/ld.so.conf.d/01_mali.conf doesn't work?20:15
shadeslayerwhen I have lrwxrwxrwx 1 root root 13 Jun 30  2015 /usr/lib/aarch64-linux-gnu/mali-egl/libEGL.so.1 -> libEGL.so.1.420:16
shadeslayer( all the paths are valid fwiw )20:16
sarnoldnamei -l on that path look fine?20:17
sarnoldhow are you determining that it didn't work?20:17
ahasenackshadeslayer: did you run ldconfig after making that change?20:35
shadeslayerahasenack: yes20:40
shadeslayersarnold: I determined it didn't work because ldconfig -p still points libEGL.so.1 to /usr/lib/aarch64-linux-gnu/libEGL.so.120:40
shadeslayersarnold: https://paste.ubuntu.com/p/JtsSnc78V9/20:46
ahasenackshadeslayer: can you try ldd on a binary that is linked with that lib?20:49
ahasenackand do you have only one libEGL in the output of ldconfig -p?20:49
shadeslayerahasenack: yeah, ldd says         libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007fb6f8b000)21:03
shadeslayerahasenack: sarnold so if I export LD_LIBRARY_PATH to the extra path, then it works21:07
ahasenackshadeslayer: does your /etc/ld.so.conf have an include directive for /etc/ld.so.conf.d/*.conf ?21:08
shadeslayerahasenack: yes21:08
shadeslayerand I've added the 01_mali.conf in /etc/ld.so.conf.d/21:08
ahasenackcan you try adding the path directly to /etc/ld.so.conf instead of the file inside the .d directory?21:08
tjaaltonshadeslayer: /usr/lib/aarch64-linux-gnu/libEGL.so.1 is from libglvnd and I doubt you can override that other than by using a diversion21:09
shadeslayertjaalton: but I also have the exact same issue with GLESv2.so21:11
tjaaltonsame thing21:11
shadeslayerah damn21:11
shadeslayertjaalton: why is that the case?21:11
shadeslayertjaalton: I essentially worked around it by setting LD_LIBRARY_PATH and setting my mali-egl dir in front of everything21:11
tjaaltonthe ld.so.conf trick only worked when both were in a subdir21:11
shadeslayerI see21:11
tjaaltonso it works?21:12
shadeslayertjaalton: with the env variable21:12
tjaaltonok21:12
tjaaltonnvidia-340 migrated to diversions because of this21:12
tjaaltontoo bad only newer nvidia blobs support glvnd21:12
tjaaltonnot amdgpu-pro, not any of the arm stuff21:13
shadeslayertjaalton: I'm still confused as to *why* this is a issue :)21:20
tjaaltonshadeslayer: ldconfig, it used to work when mesa & blobs put their libs in a separate directory, and then that directory was added to a ld.so.conf.d snippet via alternatives21:24
tjaaltondepending on which one you chose, the correct set of libs was loaded21:24
shadeslayerand now it prefers everything in /usr/lib/arch-tripet over everything in a subdir?21:25
tjaaltonyes21:26
shadeslayerI see!21:26
shadeslayertjaalton: thanks for the info :)21:26
tjaaltonyw21:27
shadeslayerI find it incredibly frustrating though21:28
shadeslayeroh well21:28
tjaaltonwhich package installs stuff in /usr/lib/../mali-egl?21:31
shadeslayertjaalton: I'm making my own21:31
shadeslayertjaalton: it's essentially this https://github.com/netrunner-pine64/pine64-mali-x1121:31
shadeslayerbut for a new board on Ubuntu 18.0421:31
tjaaltonso make it use dpkg diversions21:32
shadeslayerack21:32
sarnoldshadeslayer,tjaalton, aha :) thanks for the information22:24

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!