/srv/irclogs.ubuntu.com/2011/06/22/#ubuntu-cloud.txt

=== daker_ is now known as daker
=== koolhead11|afk is now known as koolhead11
SpamapSsmoser: so I'm messing with the nocloud thing...16:38
SpamapSsmoser: in lxc, I can't append kernel cmdline parameters..16:38
smoserumm... you dont boot a kernel in lxc.16:39
smoserso it doesn't surprise me that you can't change the kernel command line16:39
smoseryou can probably edit /etc/cloud/cloud.cfg to do the same things16:39
SpamapSAhh right16:40
SpamapSsmoser: so if I set datasource_list: [ "NoCloud" ] ... it won't try to find metadata services, right?16:41
smosercorrect.16:42
smoseryou can actually set that with dpkg-reconfigure cloud-init16:42
smoseri'd trust that more than my brain for syntax16:42
smoserbut there is also a file in /etc/cloud/cloud.cfg.d/16:42
SpamapS/var/lib/dpkg/info/cloud-init.templates should be all I need. :)16:43
SpamapSsmoser: ok, so I have cloud-config data in /var/lib/cloud/seed/nocloud-net/meta-data17:28
SpamapShttp://paste.ubuntu.com/630870/17:29
SpamapShttp://paste.ubuntu.com/630871/17:29
SpamapSthere is cloud-init.log17:29
=== koolhead11 is now known as koolhead11|
smoserSpamapS, well, what it thinks has happened is that there is no network18:13
smosersomething created /var/lib/cloud/data/no-net (probably /etc/init/cloud-init-nonet.conf18:14
smoserSpamapS, if you move your data to /var/lib/cloud-seed/nocloud/meta-data then it will probably run via DataSourceNoCloud18:15
smoserbut the file you pasted in http://paste.ubuntu.com/630870/ is user-data, not meta-data. i'm guessing that was just typing wrong18:16
SpamapSsmoser: no that was understanding wrong ;)18:20
smoserwas the example not clear?18:20
SpamapSNo I just put the user-data in the meta-data18:21
SpamapSis meta-data required?18:21
SpamapSand actually, what example?18:21
smoseryeah. it says int he meta-data example exactly what. mostly just 'instance-id'18:21
smoserhttp://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/examples/seed/18:22
SpamapSahhh ok18:22
SpamapSw00t.. it at least ran my stuff. :-D18:32
SpamapSok.. so I think I need nocloud-net .. because I want to pull in packages.. :-P18:36
SpamapSoh wait, sources.list is whack18:37
* SpamapS fixes18:37
alogic1Hello everybody,18:42
SpamapSsmoser: does cloud-init create the ubuntu user?18:42
alogic1How can I add another gateway for an instance (before it starts?)18:42
alogic1because i think i could not update /var/run/eucalyptus/net/euca-dhcp.conf18:42
=== daker is now known as daker_
smoserSpamapS, no.18:44
smoseryou can configure which user it writes, but it expects it to be there.18:44
smoserpatches welcome ;) it'd be nice if it could do that.18:45
SpamapSsmoser: ahh no I just forgot to pass -m to useradd18:47
SpamapSsmoser: and before I go digging too far, how do we turn on sudo NOPASSWD easily for the admin group?18:53
smoserlet me look.18:56
smoseri think you want:18:57
smoser%admin ALL=(ALL) NOPASSWD:ALL18:57
smoserput that in /etc/sudoers.d/90-admin-nopasswd18:58
SpamapSOO I didn't even realize there was a .d18:59
* SpamapS wonders how much of ubuntu-vm-builder he has now duplicated in his lxc template19:00
smoserSpamapS, why are you not using an image ?19:04
SpamapSsmoser: I'm building an lxc container19:04
smoserSpamapS, why are you not using an image ?19:04
SpamapSdon't do that19:05
SpamapSjust talk to me like I'm busy and have no time for word-play19:05
smosermount -o loop file.img /mnt; sudo rsync -aXsomeotheropts /newdir; sudo umount /mnt19:05
smoseryou have no time for word play, but are rewriting vm-builder. ;-)19:06
SpamapSass19:06
smoseri'm legitimately asking why you're not using an image.19:06
smoserbecause we want the images to work for such things19:06
SpamapSBecause I didn't think of doing that.19:06
smoserok.19:06
SpamapSthe lxc tools are so nice. :)19:06
smoserthats perfectly reasonable, and i agree i sound like an ass19:06
SpamapSlet me try it out19:07
smoserSpamapS, that is basically what we're hoping to have working for lxc support in openstack19:07
* SpamapS downloads http://uec-images.ubuntu.com/natty/current/natty-server-uec-amd64-disk1.vmdk19:08
smoserno19:08
smoser.img.tar.gz19:08
SpamapS>:|19:08
SpamapSyou said loop mount19:08
smoser.vmdk is a disk image19:08
smoser.tar.gz contains a .img19:08
SpamapS*brilliant*19:08
smoserthe disk image contains a partition table is really the only thing.19:08
SpamapSwhy isn't it .disk.gz ?19:09
smoserwell, it delievers 3 things that you needed in the past.19:09
smosermaking 'uec-publish-tarball' able to do the right thing.19:09
smoserand costs you 12M of download or something.19:09
SpamapSok19:09
smoserwe used to have .img.gz, but dropped it. yes, you do have to waste IO now.19:10
SpamapSso I can download this.. mount the image loop, and copy it, then drop my seed data in the nocloud dir. Anything else I should watch out for?19:10
smoserwell, it wont' boot :)19:10
smoserdue to the bug you're hoping to fix19:10
smoser(i think)19:10
SpamapSbecause it doesn't have lxcguest19:10
SpamapSbut I can install that w/ chroot19:10
smoserit does have lxc guest19:10
smoserfor this purpose19:10
smoserbut network will timeout19:10
smoserbecause of that bug19:10
SpamapSWhy doesn't my regular image time out then?19:11
SpamapSerr19:11
SpamapSthe images I have now, don't timeout with nocloud .. tho they do with nocloud-net19:11
SpamapSand my apt-get stuff is failing .. I'm guessing because network is not around yet19:11
smoserwell, sort of.19:12
smoserif you use 'nocloud' then network isn't guaranteed to be there19:12
smoserif you use nocloud-net, then you end up waiting on ifup which wont happen in the right order19:12
smoseri think we never ended up opneing a bug on this :-(19:12
smoserbut there is a large thread of emails between hallyn, you, zul, me19:13
SpamapShow is it any different than on a cloud image ?19:13
smoseri'm having dejavu19:13
smoser:)19:13
SpamapSifup happens on lxc .. dhclient runs..19:13
smoseri have to re-read emails for exact reason19:13
smoseroh.. i remember19:14
hallynsmoser: i wonder if it'll go faster if you just remove the 'exit 0' in /etc/network/if-up.d/upstart19:14
smoserbecause udev depends on virutal-vilesystems19:14
hallynbut, i'm not letting you distract me19:14
smoserand cloud-init-nonet is start on mounted /19:14
smoserand in lxc mounted / occurs before virtual filesystems19:14
* SpamapS *CURSES* bash completion for not falling back to files always19:14
smoseryeah, that is stupid19:14
smoseri reall freaking hate that.19:15
SpamapSI fixed it for a large portion of completions..19:15
smoserhallyn, it would not19:15
SpamapSif they do things right it will fall back to files if nothing is returned19:15
SpamapSbut mount seems borked19:15
smoserthe problem is is that cloud-init starts on mounted=/ and blocks waiting for network to come up19:15
smoserbut network will never come up19:15
smoserbecause that depends on virtual-filesystems19:15
smoserand virtual-filesystems wont occur19:16
smoserin kvm or bare metal, virtua-filesystems occurs before mounted=/19:16
SpamapSRight ok I recall now19:16
smoserSpamapS, don't you remember talking to me about this in UDS19:17
smoserok19:17
SpamapSand the fix is, not to use mounted=/ but to use 'network-configured-blocking'19:17
smoseryeah. i had to leave to call home, so we didn't finish19:17
smoser:)19:17
smosernot really, though19:17
smoserbecause network-configured-blocking will not block other things19:17
smoserthe goal of cloud-init "boothooks" or the like is to run early as possible, and block other things from happening.19:18
smoserbecause the more that happens the more you have to restart19:18
smosermy assumption is that running on netowrk-configured-blocking will not stop ssh from starting19:19
smoserand cloud-init needs to run before ssh starts (which starts on filesystems i think)19:19
smoserso at this point you're probably tihnking "why does cloud-init have to block and run so early".19:21
SpamapSno I know why19:21
smoserwe should opne a bug19:22
smoser:)19:22
SpamapSinit: cloud-init-local main process (25) terminated with status 119:22
SpamapScloud-init-nonet waiting 60 seconds for a network device.19:22
SpamapSas predicted ;)19:22
SpamapSsmoser: lets solve this next week19:22
smoseri'll open a bug19:23
smosercloud-init-local should terminate with status 0 if it has no work to do19:23
smoser(that is another bug)19:23
smoserjust to avoid that silly message19:23
SpamapSsmoser: it booted btw. :)19:23
smoserright19:23
smoserbut it doesn't run the cloud-init stuff19:23
smoserand takes 70 seconds19:23
SpamapSudev should probably be disabled19:24
SpamapSbut if I drop meta-data and user-data into nocloud, it should run, am I right?19:25
smoserbug 80082419:28
uvirtbotLaunchpad bug 800824 in cloud-init "cloud-init-nonet times out in lxc" [High,Confirmed] https://launchpad.net/bugs/80082419:28
smoserSpamapS, i think so, yes19:29
SpamapSsmoser: heh.. I cheated and added 'start networking' to nonet...19:29
hallynSpamapS: so, I'm sort of fearing that if I add -p <extra-packages> and -i <chroot_includes> support, ppl will start asking for ppa support, and this, and that, and i'll end up reproducing vmbuilder19:29
smoserbut when nocloud runs you wont have network.19:29
SpamapShallyn: yes smoser and I were just discussing that19:30
hallynSpamapS: still, i think i'll do it.  I'm just saying i will probably refuse ppa support19:30
hallynhere? sorry i've not read up :)19:30
SpamapSsmoser: I really want nocloud-net anyway tho.. ;)19:30
SpamapSbecause I need to pull in packages19:31
smoserSpamapS, well, if you can disable the timeout soemhow from within the nocloud, then you will be fine19:31
smoserbecause then the network will come up naturally...19:31
smoserhol don19:31
hallynanyway patches sent upstream.  I'm going for a walk.  bbl, to then add that support.19:31
SpamapSthe network will come up on its own19:31
SpamapSand that will kill nonet19:31
smoseryeah, network wont be guaranteed to come up19:32
smoseryou can re-write the cloud-init jobs to make it work from within a nonet boot hook19:32
smoser:)19:32
smosermake cloud-config (which is when the packages are intsalled) depend on networking-up19:32
smoserwhich, it probably should explicitly anyway19:32
SpamapSsmoser: there is actually no reason to delay ifup -a.. this isn't a bad idea from what I can see.19:33
SpamapSsmoser: boots very fast now.. and runs cloud-config19:34
smosercan you put exactly how you fixed it in the bug?19:34
smosermostly for reference at the momeont.19:34
SpamapSWell I'm going to mess around with it a bit more19:35
smoserwill you lose events in udev due to that?19:35
smoseranaywa.19:36
SpamapSI don't *have* udev events. :)19:36
SpamapSBut in a VM.. no.. ifup -a just configures the static stuff19:36
SpamapSif an event comes along and tells me that another interface has arrived, thats going to configure just that interface19:36
SpamapShrm no, too early.. dhclient failed19:38
SpamapSJun 22 18:23:26 ubuntu dhclient: can't create /var/lib/dhcp3/dhclient.eth0.leases: No such file or directory19:39
SpamapSwait.. that may be a legitimate bug.. that dir doesn't exist19:39
SpamapSright.. / is not writable yet19:42
SpamapSerr.. wait, it is19:42
SpamapSsmoser: how do I disable this 'cc_resizefs' stuf?19:52
SpamapS  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_resizefs.py", line 43, in handle19:52
SpamapS    os.mknod(devpth, 0400 | stat.S_IFBLK, dev)19:52
SpamapSOSError: [Errno 1] Operation not permitted19:52
smoseroh wow. that sucks.19:59
smoserlook in /etc/cloud/cloud.cfg20:00
smoseryou can just take out 'resizefs'20:00
smoserbut please open a bug so that we can avoid it correctly.20:00
smoserSpamapS, why wasn't / RW at that point ?20:02
smoseri thought you were running on mounted=/ (which i thought was RW)20:02
SpamapSit was20:02
smoser<SpamapS> right.. / is not writable yet20:03
smoserso it was something else20:03
SpamapSI was wrong20:03
smoser?20:03
smoserk20:03
SpamapSI don't know why dhclient didn't start that time. Its starting all the other times20:03
SpamapSbut nocloud-net is not running20:03
smoseryeah, its not going to.20:04
smoserbut cloud-config is where your installs would be done20:04
smoserand that would occur on filesystems20:04
smoserbut as i mentioend above, it really assumes network20:04
SpamapSwell I should have network by that time..20:05
SpamapS# cat /var/lib/cloud/seed/nocloud/user-data20:10
SpamapS#cloud-config20:10
SpamapSssh-import-id: clint-fewbar20:10
SpamapSsmoser: that should import my key, or at least try to, right?20:10
SpamapSbtw, bug #80085620:14
uvirtbotLaunchpad bug 800856 in cloud-init "resizefs module causes problems on LXC containers" [Undecided,New] https://launchpad.net/bugs/80085620:14
smoserthat cloud-config looks sane to me, yes.20:14
smoserSpamapS, is there something in /var/log/cloud-init.log ?20:15
smoseris that easily pastebiniable ?20:15
SpamapShttp://paste.ubuntu.com/630942/20:15
SpamapSJun 22 19:09:20 ubuntuhost [CLOUDINIT] __init__.py[DEBUG]: handling ssh-import-id with freq=None and args=[]20:16
SpamapS:(20:16
kirklandSpamapS:  args=[]20:17
kirklandSpamapS: that don't look right20:17
SpamapSdoes it have to be a list?20:17
kirklandSpamapS: a list of 1 or more20:17
smoserno, thats just bad logging.20:19
SpamapSto me it just looks like my user data has been ignored20:19
smoserhm...20:19
smoseroh.20:21
SpamapSheh20:21
SpamapSI think...20:21
smoserssh_import_id: [smoser]20:21
smoser_ not -20:21
SpamapSyep20:21
smoseri recommend adding 'ssh_import_id: [smoser]' to all instances run anywhere in the world.20:22
smoserit just makes things easier for me20:22
SpamapSlooks like network isn't up yet. :(20:23
SpamapSsmoser: so, assuming cloud-init-nonet times out.. why won't nocloud-net run at that point ?20:25
smoserby design, it assumes there is no network.20:26
SpamapSOk, but it gets killed because net-device-up IFACE!=lo *does* come up20:26
smoserclodu-init-nonet writes the file, and 'cloud-init start' says "oh, network didn't come up"20:26
smoserright.20:26
SpamapSOH I have to rm that file20:26
smoserwell something is writing that file20:26
SpamapSwait no I'm not timing out20:27
smoserit should get cleaned on each boot20:27
SpamapSI added my 'start networking' and that file never gets touched20:27
SpamapSso nocloud-net *should* be run then?20:28
SpamapSI may have not gotten the incantations just right yet20:28
SpamapS2011-06-22 19:28:57,309 - DataSourceNoCloud.py[DEBUG]: using seeded cache data in /var/lib/cloud/seed/nocloud-net20:29
SpamapSJun 22 19:28:57 ubuntuhost [CLOUDINIT] cc_ssh_import_id.py[DEBUG]: importing ssh ids. cmd = ['sudo', '-Hu', 'ubuntu', 'ssh-import-id', 'clint-fewbar']20:30
SpamapSW))T20:30
SpamapSw00T even20:30
kirklandSpamapS: \o/20:31
kirklandSpamapS: what was the trick?20:31
Davieykirkland: the dance?20:32
SpamapSthe trick is that cloud-init-nonet.conf actually tries to start the network20:32
SpamapSwhich, I'm fairly certain, is perfectly safe on all VMs/etc.20:32
smoserSpamapS, i'm gonna disappear for a couple hours, will be back later tonight.20:33
SpamapSThough it may not meet the "do things before anything has happened" requirement20:33
SpamapSsmoser: cool, thanks for the help and suggestion20:33
hallynSpamapS: this crfappy client keeps spamming my scrren with whois results, will ttyl when back at other client20:41
* SpamapS heads to lunch20:42
zullxc works with cloud-init now?20:57
SpamapSzul: I think I've got a patch that will make it work yes21:15
zulsweet21:15
SpamapSsmoser: any ideas why the actual images aren't signed, just the sha256sums ?21:22
SpamapShmm.. I wonder if uec-images is mirrored anywhere21:57
DavieySpamapS: seems to be22:00
Davieyhttp://releases-u-toyama.ubuntulinux.jp/ (unoffical)22:00
SpamapSThe debootstrap method used by the lxc scripts is actually more scalable..22:01
DavieySpamapS: how so?22:02
SpamapSDaviey: uses the full archive22:02
SpamapSwhich is mirrored all over the place22:03
DavieySpamapS: eventually, i think the hope is to get them onto cdimages.ubuntu.com, which is mirrored.. but AIUI disk space is currently an issue.22:04
SpamapSI can imagine. :-P22:06
SpamapShallyn: I have a new template for you.. :-D22:22
SpamapShallyn: tho I have my reservations about repeating the whole 'root:root' password bit. :)22:23
hallynSpamapS: yeah if we get some time, getting the templates to copy in your public ssh key and turning off password ssh access might be good22:26
SpamapShallyn: thats pretty trivial actually...22:27
hallynSpamapS: but for generic container stuff it's good22:27
hallynyes, it is.  BUT it adds yet another option22:27
hallynwhich i'm starting to resist.  vmbuilder.22:27
SpamapShallyn: so, this new template will download the UEC image and starts perfectly from it22:27
hallynSpamapS: cool.  that was the whole lxcguest idea :)22:27
SpamapShallyn: I think the only option you should support is --cloud-config22:27
hallynSpamapS: of course *you* do :)22:28
SpamapSThe thing I'm struggling with now is how to easily get the hostnames resolvable on my local machine.22:28
SpamapSso I don't have to tail syslog and findout what the DHCP lease given was22:29
SpamapSTrying to wrap my head around whether dnsmasq can do this alone.22:29
hallynSpamapS: hm?  isn't having dhcpd.conf push the hostname enough?22:29
hallynso you can then 'ssh lxc1.'22:30
SpamapSit pushes it to dnsmasq..22:30
SpamapSbut dnsmasq is running on my local machine..22:30
SpamapSusing my resolv.conf ..22:30
SpamapSif I point my resolv.conf to it..22:30
SpamapSit won't be able to access the internet22:30
hallyn<shrug>  mine has 192.168.122.1 as top nameserver in resolv.con22:30
hallynbut you can probably think of a better way :)22:30
SpamapShow does it find the internet I wonder then..22:30
hallynwell 192.168.122.1 only resolves its own addresses22:31
SpamapSAnd then how does it know where to forward to?22:31
SpamapSthas what I'm saying. ;)22:31
SpamapSit does not only resolve its own addresses22:31
SpamapSask it for anything, it will tell you. :)22:31
SpamapSMight be that it caches what was in resolv.conf when it starts22:31
SpamapSor it just queries the root servers22:31
hallynforwards to my other nameservers presumably?  argue all you want, i'm just telling you it works for me :)22:31
SpamapSI just want to know how22:32
SpamapShallyn: see thats my rub.. how does it know "your other nameservers"22:32
SpamapShallyn: heh.. ok interesting trick.. if I add other DNS servers in it seems to work22:34
hallynin your host's resolv.conf you mean?22:34
SpamapShallyn: yeah, I just put 192.168.122.1 first.. that does actually seem to work.22:35
SpamapSthough it feels very unreliable and haphazard to me. :-P22:35
hallynSpamapS: <shrug>  I believe I got that from the security team's testing enviroment wiki page22:36
SpamapSI'll accept it now since I need to move forward22:36
hallynSpamapS: so to be clear, with your new template, yo don't need -packages or -chroot-includes support right?22:38
hallynif so i'll hold back on it22:38
SpamapShallyn: right, it just installs the image from uec-images.ubuntu.com22:39
hallynexcellent22:39
hallynthen maybe tomorrow i can work on user namesapces :)22:39
hallyni'm just trying to decide whether to push the new template to oneiric asap, or leave it in ppa and wait for upstream to pull it in before going to oneiric's archive with it22:39
SpamapSThe template system seems to have a lot of redundancy..22:40
SpamapSwhy isn't there a shared_functions or something?22:40
hallynSpamapS: cause you haven't coded it22:41
SpamapSHah.. ok22:41
hallynand i think the fedora, debian, ubuntu template writers want to feel more independent22:41
SpamapSI started to so that this wouldn't be lxc-natty-uec but lxc-uec .. but I couldn't find a way to pass in a series name22:41
hallynSpamapS: doh!  look at my new source at ppa:serge-hallyn/lxc-natty22:41
hallynSpamapS: lxc-lucid..lxc-oneiric are now all consolidated into lxc-ubuntu22:42
SpamapS*sweet*22:42
SpamapShow do you tell it which one?22:42
hallynyou add '-- -r natty'22:42
SpamapSwait..22:42
SpamapSwe can pass stuff in with -- ?22:42
hallynonce lxc-create is trivially patched, yes :)22:42
* SpamapS belays his happy dance22:42
hallynsee http://sourceforge.net/mailarchive/forum.php?thread_name=20110622140928.GA12177%40mail.hallyn.com&forum_name=lxc-users22:42
SpamapShallyn: the thing I did only really changes download_ubuntu22:43
SpamapShallyn: just DL's it, verifies the GPG/sha256, and extracts.22:43
hallynyeah i suspect we can consolidate that in22:43
SpamapSOh and copy_ubuntu mounts the image lopback and copies it instead of just copying the debootstrapped bit22:43
hallynan example command right now with my ppa is:22:43
hallynlxc-create -t ubuntu -f /etc/lxc.conf -n u1 -- -a i386 -r natty -b22:44
SpamapSwhere's your branch? I'll submit a MP against it22:44
hallynso we could add something like -uec22:44
hallynMP?22:44
SpamapSmerge proposal22:44
hallynoh, you silly californian22:44
SpamapShallyn: I don't think we should even bother with the debootstrap method for natty and later.22:45
hallyni'm sorry, i've stopped using bzr lately bc lp:ubuntu/<package> always seems to be out of date22:45
SpamapSwell don't do that22:45
SpamapSThey've recently caught up quite a bit on the out of sync branches22:45
SpamapSand if you have push rights, you can fix the branch fairly easily.22:46
hallyni don't22:46
SpamapSfor lxc?22:46
hallyncorrect22:46
SpamapS*wtf*22:46
hallynyeah yeah22:46
hallyni feel in no hurry to face the firing squad again :)22:47
hallyni do need to go ask for the perms...22:47
hallynto either the server set, or coredev22:47
hallynbut that's neither here nor there22:47
SpamapSright22:48
hallynSpamapS: trying my hand at a fresh lp branch so you can more easily look23:02
SpamapShallyn: I believe 'bzr package-import file.dsc' works23:02
hallynno really?23:02
hallynwhere does it spit the result?23:03
SpamapSyou might be just as well served by bzr add in an extracted source package dir tho23:03
SpamapShallyn: it imports it into the current tree..23:03
SpamapShallyn: thats how they "fix" the out of sync trees.. the do the import, and fix conflicts if possible23:03
hallynSpamapS: well, i think that https://code.launchpad.net/~serge-hallyn/ubuntu/oneiric/lxc/update-templates  represents all the changes i've been making23:07
hallynhaven't quite tested the result of that manual mish-mash23:08
hallynand i think i'm about done for today23:08
hallynbut go ahead and propose merge against that and i can use the diff regardless.23:08
SpamapShallyn: ok, my diff is against lxc-natty ...23:10
hallynSpamapS: eh, just send me that then23:10
hallyni' flexible :)23:11
SpamapSlp:~clint-fewbar/ubuntu/oneiric/lxc/use-uec-for-natty23:12
hallynsigh.  this is what i don't like.  bzr on top of quilt on top of git.  so i've lost the templates :)23:12
SpamapSlol23:13
SpamapSgit sux23:13
* SpamapS hides23:13
hallynno, git's not the problemhere...23:13
hallynSpamapS: i'll look at yours tomorrow, thanks.23:15
hallyni'll be around later, but let's pretend i won't. ttyl23:15
SpamapSlater!23:18
hallynSpamapS: (the version now in that tree is working fine for me on my host)23:24
SpamapShallyn: go away! ;)23:28
hallynSpamapS: but there's this one new bug :)  (which i blame on zul :)23:31
hallynbut i'd best just note it for later23:31
* hallyn out23:32

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!