[00:02] <pmatulis> anyone here get sssd working with sudo configured in ldap?
[00:27] <ipl31> Hello, so I had a qlogic 10 Gig card that worked oneiric but now seems to not be happy because of a firwmare version in precise. Did something change in the linux-firmware package?
[00:31] <sarnold> ipl31: were there any enlightening notes in the /usr/share/doc/<foo>/changelog* files?
[00:33] <ipl31> sarnold: unfortunately no
[00:33] <sarnold> :(
[00:34] <sarnold> ipl31: if you steal the firmware from the old version, does it work?
[00:34] <ipl31> good question I should try that
[00:35] <ipl31> although box is remote and I have not network access only remote KVM, but I will see if there is way I can get it on there
[00:35] <sarnold> oof
[00:35] <ipl31> yeah :)
[00:35] <sarnold> would it be easier to get a local card?
[00:35] <ipl31> yeah it might be, its 10G card so I can't buy one locally would need to have it shipped
[00:36] <ipl31> but I might end up doing that
[00:36] <sarnold> .. it doesn't sound cheap :) but still, that also sounds like a system that's best to leave alone as much as you can. :)
[00:36] <ipl31> actually system is not in service so I can do whatever I want
[00:36] <sarnold> woo :
[00:36] <sarnold> :)
[00:36] <ipl31> testing the upgrade to precise
[01:49] <stgraber> SpamapS: I'm testing on an ivy bridge laptop (i7) with a lot of RAM and a very fast SSD...
[02:06] <SpamapS> stgraber: ok, so not that dissimilar from my mac book air (i5, 4GB RAM, average SSD)
[02:23] <Guest2295> Hello all!!!
[02:23] <Guest2295> I am coming I think to the end of a very drawn out problem with reference to my primary system running ubuntu.
[02:49] <Tohuw> I need to install Magick-config, but it seems the only packages that provide it from the Ubuntu repos require X11. Can someone enlighten me as to why this might be, and if there's a suitable alternative if my only goal is to provide Magick-config for gem install rmagick?
[02:52] <sarnold> Tohuw: drat, I wondered if graphicsmagick-libmagick-dev-compat might be easier .. but no luck, it also eventually depends upon libx11-dev
[02:53] <sarnold> Tohuw: I believe I've read some mechanism to fake dpkg into thinking that some specific packages are already installed; I _hope_ that graphicsmagick or imagemagick would load libraries only as they need them..
[02:57] <Tohuw> sarnold: I could force the install of the package, but I'd rather not... based on my reading of Magick-config, it seems odd to require the X11 stack for it...
[02:57] <sarnold> Tohuw: it'd be a package-level dependency, the individual tool may not need it at all.
[03:05] <Tohuw> sarnold: I found a workaround via installing libgraphicsmagick1-dev, which installs an acceptably minimum number of X11 dev packages.
[03:05] <sarnold> Tohuw: oh! hooray for just the -dev packages. that'll save a bit.
[03:06] <Tohuw> Yes, considerably. x11-common, libx11-dev, x11proto-xext-dev. I can live with that.
[03:17] <Tohuw> For the record, the correct package ended up being 'graphicsmagick-libmagick-dev-compat' to resolve rmagick not installing on Ubuntu via gem install. This is the most minimal provider I could find.
[03:23] <Tohuw> I didn't catch that libgraphicsmagick1-dev provides GraphicsMagick-config, which is not quite the same. I also needed libmagickwand-dev.
[05:55] <basil60> I'd like some advice on possible malware on ubuntu11.04 server please?
[06:06] <basil60> I've received a number of complaints from my ISP that unsolicited emails have originated from IP address. I've scanned for rootkits , malware a,d almost every other conceivable problem on my PC. As soon as I turned my server back on, the complaints returned. I'm sour it's some system change that I made that has permitted this problem... I'd just like some advice on shutting it down.
[06:18] <basil60> Thanks guys...I might try again later
[08:04] <jellybean> Good morning all
[08:04] <jellybean> Is there a recommended procedure for installing Ubuntu Server to a compact flash drive?
[09:52] <DarkStar1> hi all.. Is there anyway I can increase the size of a directory? specifically one under the /var tree
[09:59] <chmac> DarkStar1: Directories don't have sizes exactly, unless you put more stuff into the directory, in which case it'll get "bigger"
[09:59] <chmac> DarkStar1: Or are you running out of space?
[10:00] <DarkStar1> chmac: I ran out of space. I was importing old mail archives into a the directory path /var/vmail
[10:00] <chmac> DarkStar1: Do you have space elsewhere on the disk?
[10:01] <DarkStar1> chmac: yeah. let me pastethe output of df -hl
[10:03] <DarkStar1> chmac: http://fpaste.org/05Cn/ so I'm guessing I need to increase the amount of space allocated to /dev
[10:03] <chmac> DarkStar1: OVH server right?
[10:03] <DarkStar1> yeah
[10:03] <DarkStar1> chmac: houw'd you know :)
[10:04] <chmac> They have this weirdness with rootfs and /dev/root, very strange. /dev/root doesn't actually exist.
[10:04] <DarkStar1> hhmm.
[10:05] <DarkStar1> mayhaps I can remount /var on dev2?
[10:05] <DarkStar1> on /dev
[10:05] <DarkStar1> ?
[10:11] <chmac> DarkStar1: There is no /dev
[10:11] <chmac> It's a virtual filesystem, your only real file system is /dev/md2
[10:12] <chmac> You could try symlinking your /var/vmail directory to /home/vmail or something like that.
[10:12] <chmac> ovh are pretty stupid in the way they lay out your disk, I always repartition them.
[10:14] <sri> hi
[10:14] <Guest18447> hi
[10:15] <DarkStar1> chmac: hhm……...
[10:15] <DarkStar1> good idea. Will have to trythat later. thanks
[10:16] <chmac> DarkStar1: Have you been using the machine for a while?
[10:16] <chmac> DarkStar1: You could also repartition it, that's one advantage of software raid, you can repartition on the fly.
[10:16] <chmac> DarkStar1: I wouldn't recommend it with a production system, but ovh provide 100G of ftp backup, so you can push a backup to be doubly safe, take one disk out of the raid array, repartition, bring it back into the array, resync, wait, repeat.
[10:17] <DarkStar1> chmac: for over a week now. I've been setting it up for a client and there a few sites hosted on it so
[10:17] <DarkStar1> far
[10:17] <chmac> DarkStar1: In that case you definitely want to repartition, all your /var/www/ and /var/logs/ will be on that 10G root partition, likewise /var/lib/mysql
[10:17] <DarkStar1> Oh.. I'll have to book sometime off to do that then
[10:17] <chmac> You can do the whole thing with the machine still online, because you have 2 disks.
[10:18] <DarkStar1> shit!!
[10:19] <DarkStar1> all that time setting up >:/
[10:19] <feisar> hi my question's about virsh and vmbuilder ubuntu but #ubuntu-virt is a little quiet: if I created a vm using vmbuilder kvm ubuntu but i did it from the wrong directory, can I just mv my .qcow2 then edit the VM using virsh # edit VM_NAME?
[10:19] <DarkStar1> chmac: That just poo pooe'd my tgi friday feeling now
[10:20] <chmac> DarkStar1: You will need to take down the machine in order to move data around onto new partitions, but it's not so hard
[10:21] <DarkStar1> chmac: what a stupid way to partition a machine
[10:21] <DarkStar1> Pissed off at them now
[10:23] <chmac> DarkStar1: You pay peanuts, you get monkeys!
[10:24] <chmac> DarkStar1: In my opinion, their SSD boxes are a better deal than their traditional disks, 120G is usually way more than we ever use on any of our web machines, and the SSDs perform much, much better.
[10:24] <DarkStar1> chmac: but let's be honest that kind of beetlejuice partition is spiteful
[10:24] <chmac> DarkStar1: It's just stupid, but so is much of what I see at OVH! :-)
[10:26] <DarkStar1> SSds aren't worth the price atm and I need space. Lots of it. the mail archve I was trying to migrate is 100Gb zipped up
[10:26] <chmac> DarkStar1: Then you want a spinning disk!
[10:30] <RoyK> [slightly offtopic] hm... I just upgraded a server install to Quantal and installed ubuntu-desktop on it, and lightdm doesnt't start automatically
[10:33] <AlexO> Hey, I opened the 3306 port (for mysql) but when I'm trying to connect with telenet on 3306, I get a connect refused, the strangest this is, that it's was working yesterday, but today It's not working anymore...
[10:33] <AlexO> any ideas?
[10:33] <AlexO> -this+thing*
[10:34] <Lachezar> AlexO: netstat -antup | grep 3306 shows?
[10:35] <AlexO> "tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      15568/mysqld"
[10:36] <Lachezar> AlexO: That seems OK. Are you sure you're telnet-ing into the right address?
[10:37] <AlexO> Lachezar: yep, I just checked it again
[10:38] <RoyK> AlexO: using ufw?
[10:38] <AlexO> RoyK: iptables
[10:41] <RoyK> AlexO: turn on logging in iptables, then, or just use ufw
[10:41]  * RoyK uses ufw for 95% of his work
[10:41] <DarkStar1> I only know a little iptables and 0 about ufw
[10:42] <AlexO> RoyK: what do you mean but turn on logigng in iptables? You mean open the 3306 port ?
[10:42] <AlexO> by*
[10:44] <AlexO> I'm runing out of batery, need to find a plug brb
[10:44] <AlexO> if you have any idea let me know my screen will still be there
[10:45] <DarkStar1> AlexO: he means you should enable iptables logging so that you can see infomation about the firewall
[12:32] <jamespage> rbasak, is the subarch support SRU required to support diff arm archs?
[12:32] <rbasak> jamespage: yes, but also any arch at all
[12:32] <rbasak> jamespage: any ARM arch at all I mean
[12:33] <jamespage> rbasak, ack
[12:35] <jamespage> rbasak, looking at the branch now
[12:35] <rbasak> thanks!
[12:36] <jamespage> rbasak, 02- patch is missing from the branch :-(
[12:37] <rbasak> jamespage: well that's embarrassing :-/
[12:37]  * rbasak digs it out
[12:37] <jamespage> rbasak, lol
[12:38] <rbasak> jamespage: wondering how best to recover from this
[12:39] <rbasak> jamespage: another upload with a bumped ubuntu revision?
[12:39] <jamespage> rbasak, just push a new version to the branch
[12:39] <rbasak> jamespage: roaksoax has already uploaded - it's in the unapproved queue
[12:39] <jamespage> rbasak, ah! I wish he'd commented.
[12:39] <jamespage> I think we can get that rejected
[12:40] <rbasak> jamespage: ok, so do I need to bump the version, or does it count as not published?
[12:40] <rbasak> I suppose nobody will have been able to install from it if it never actually entered precise-proposed?
[12:40] <jamespage> rbasak, no - its not been published anywhere yet!
[12:40] <rbasak> OK
[12:40] <jamespage> rbasak, oh - was the upload for quantal or precise?
[12:41] <rbasak> jamespage: both. roaksoax took the change upstream for quantal I think, and uploaded precise as-is
[12:41] <jamespage> rbasak, sorry - LP was confusing me
[12:42] <jamespage> rbasak, actually it looks OK in the queue
[12:42] <jamespage> http://launchpadlibrarian.net/118134341/maas-enlist_0.4-0ubuntu1.1_0.4-0ubuntu1.2.diff.gz
[12:42] <rbasak> jamespage: I'm confused. How? Or does that mean that roaksoax noticed and fixed it up? Also the pocket was wrong, which I know he fixed
[12:43] <jamespage> rbasak, I suspect roaksoax fixed you up :-)
[12:43] <rbasak> thanks roaksoax :)
[12:44]  * rbasak commits fixes to the branch anyway
[12:44] <rbasak> At least it should match now
[12:44] <rbasak> jamespage: sorry for the mess!
[12:44] <jamespage> rbasak, no problem
[12:44] <jamespage> rbasak, story of my day today
[12:44] <jamespage> looking at stuff thats already been done!
[13:31] <jamespage> zul: ^^ nova upgrade bug
[13:33] <zul> jamespage: not cool
[13:50] <selvodka> hello
[13:54] <Ulfr> I just switched my frontend webservers from CentOS to Ubuntu because of an issue with PCRE, but now I'm trying to troubleshoot a problem my servers have after about a day of uptime where sessions will stack up in HAProxy and I haven't the foggiest why. Can anyone help me get started with troubleshooting?
[13:57] <Ulfr> I use HAProxy to load balance to squid reverse proxies running apache2 on ubuntu 10.04 if that helps
[14:01] <Mupfi> hi there :-)
[14:03] <Mupfi> anybody here who is familiar with kerberos?
[14:03] <feisar> hi what's the fix for the 'waiting for network configuration' on every boot with 12.04?
[14:04] <Mupfi> i have problems with the krb5-kdc which woulden't start
[14:04] <Mupfi> krb5kdc: cannot initialize realm ALKAR.INTERN - see log file for details
[14:05] <Mupfi> and the logfile tell me "krb5kdc: No such file or directory - while initializing database for realm ALKAR.INTERN"
[14:48] <hallyn> smb: hi, bug 914788, it's marked fix released and nominated for oneiric.  but you seem to be talking about q new quantal patch?
[14:50] <smb> hallyn, Yes, since George made the patch for q but probably should have done a new report
[14:51] <jamespage> zul, I enjoyed that bug ^^
[14:51] <smb> Things broke again after Debian removed the alternates setting which created xen-default link
[14:51] <zul> i did as well :)
[14:51] <hallyn> smb: can you mark it Triaged again?
[14:51] <hallyn> I'll push the fix this afternoon then.  tuesday is freeze :)
[14:52] <smb> hallyn, not sure I can do that, but in theory we probably should make the main task triaged again and add a precise task marked fixed?
[14:53] <smb> hallyn, Yeah, seem you have to accept the nomination at least.
[14:54] <hallyn> smb: sigh, ok, let me set them how I THINK you mean them
[14:55] <smb> hallyn, I could do the rest, I just cannot accept nominations for anything I cannot upload
[14:55] <hallyn> smb: can you reload and tell me if that's ok?
[14:55] <hallyn> smb: and then you're saying i can just take that patch, pop it onto quantal package, and expect itto work?
[14:56] <smb> hallyn, I looks like I thought (probably need to decide to won't fix for O) and no not that simple
[14:57] <smb> hallyn, Not sure it helps you but I liked a branch
[14:57] <smb> hallyn, A bzr branch. Basically I had to pop back to the old patch and have pushed the modified ones on top
[14:57] <hallyn> oh, *linked* a branch :)
[14:57] <hallyn> i thought you were facebooking
[14:57] <smb> errr yeah... :-P no
[14:57] <smb> Just missing keys
[14:58] <hallyn> smb: so does the branch work?
[14:58] <smb> hallyn, I did the test-build from that
[14:58] <hallyn> smb: ok, thanks.  lemme look at it
[14:58] <SpamapS> smb: so, any progress?
[14:58] <SpamapS> smb: have not looked through my bugmail yet today
[14:59] <smb> SpamapS, Tsk, well it works for me which is not progress I guess. And I asked for a more detailed list of steps
[14:59] <smb> SpamapS, Not sure whether there is any special network setup involved or so
[15:01] <SpamapS> smb: let me try my reproduction steps one more time..
[15:06] <SpamapS> smb: hm, doing the basic steps doesn't cause it.. I'll try again with juju, maybe it is doing something special
[15:07] <smb> SpamapS, Maybe I prepare another kernel in parallel that logs a lot of stack traces (for every dev_hold and dev_put). Its ugly but if you are the only one to be lucky I would let you run it and post the results. :)
[15:09] <SpamapS> smb: for sure :)
[15:11] <SpamapS> smb: indeed, something juju is doing in its destroy-environment is causing the issue reliably.. will boil it down to a test case
[15:33] <hallyn> smb: i dunno, patch looks good to me (though i can't bzr import it bc of the way you moved the patch around in the queue)  I'll run the qa regression tests against it, then push the package from source
[15:35] <smb> hallyn, Grr and I was just using quilt and bzr locally. :-P But ok, just note patches, because there were two
[15:36] <hallyn> smb: yeah, bzr just doesn't deal well with any sort of funky quilt usage.  but I'm just doing 'bzr bd -S' out of your bzr tree, worked fine, so it'll have both your patches
[15:37] <smb> hallyn, Ok, cool. Yeah, I probably should just have all patches unapplied and then replaced the file before applying them again
[15:43] <smb> SpamapS, If you find a nice test case, please let me (err well the bug report) know. Otherwise the smb2 version is now up.
[15:47] <SpamapS> smb: ok, working on it now
[16:12] <DarkStar1> qq the cron task: 3 0 * * 1/3/5  will run this task 0003 every mon-wed-friday right?
[16:19] <DarkStar1> No one here?
[16:19] <kendosan> hello guys, when i put my bash command in startup, when i reboot everything runs , but the bashscript that runs gives me error ffmpeg not found,  i run a nohup on startup
[16:19] <kendosan> so im not sure what is wrong
[16:19] <kendosan> when i cd in directory and do nohup manualy everything is cool
[16:19] <Guest54412> hmm does anyone here have lamp installed
[16:19] <patdk-lap> kendosan, that doesn't tell you anything?
[16:20] <kendosan> hmm thats the thing im not sure what im doing :(
[16:21] <kendosan> for example on startup commandline nohup sh /var/www/bash/nohup.sh >/dev/null 2>&1 &
[16:22] <sarnold> kendosan: try without throwing away the error messages..
[16:22] <kendosan> er i will try
[16:22] <patdk-lap> kendosan, oviously your ENVIROMENT is different
[16:22] <patdk-lap> different path settings and other things
[16:23] <kendosan> patdk, what do you mean ? i run ubuntu server 12.04 64bit
[16:23] <TJ-> kendosan: Which 'nohup' is start-up using? The shell's version, or "/usr/bin/nohup" ? They have differences, especially if the start-up script isn't using a log-in shell
[16:23] <kendosan> just a second please :)
[16:23] <sarnold> TJ-: oh? I hadn't heard this :) thanks for the heads up.
[16:23] <kendosan> i put the nohup commandline here  /etc/rc.local
[16:24] <patdk-lap> so no login, so you can't depend on anything being right
[16:24] <patdk-lap> and have to make sure path and all are specified
[16:24] <kendosan> it executes but, the thing is it does not run, when i type it in manual way :O
[16:24] <kendosan> maybe im just confused with all this :D
[16:25] <TJ-> kendosan: Usually that's because the shell environment at start-up is minimal compared to a log-in shell. Best debugging is to insert, in your start-up shell, a debug line of the form "env > /tmp/start-up.log" so you can look at the env it runs with
[16:26] <kendosan> thanks i will try to do that
[16:26] <TJ-> kendosan: s/your start-up shell/your start-up shell script/'
[16:26] <hallyn> smb: my build of your libvirt tree oddly fails on an augeas lens test
[16:27] <smb> hallyn, there is lenses in libvirt???
[16:27] <hallyn> in the tests, yeah
[16:27] <hallyn> biam
[16:29] <smb> Weird I have created a source package from my tree and ran it though sbuild... my only problem there is building sometimes ending up in /build which then bails because of an overlayfs bug
[16:29] <smb> The second attempt with /build diverted to non-overlayed /home/... did succeed
[16:30] <DarkStar1> the cron task: 3 0 * * 1-5/2  will run this task every mon-wed-friday right?
[16:30] <DarkStar1> I'm new to this
[16:31] <DarkStar1> the aim is that it will run it at 0003h on those days.
[16:32] <SpamapS> smb: installing your kernel next
[16:33] <SpamapS> smb: the test case is still the same... just start and stop a container
[16:33] <DarkStar1> no one know?
[16:33] <sarnold> DarkStar1: that's how I read it as well
[16:33] <sarnold> DarkStar1: though I'd probably use 1,3,5
[16:33] <sarnold> no longer, no shorter, but more explicit
[16:33] <SpamapS> smb: note that the container I start/stop also does an apt-get upgrade.. I wonder if there's something weird in there
[16:33] <DarkStar1> sarnold: ok
[16:34] <DarkStar1> with the commas right? I'm new to task automation.
[16:35] <sarnold> DarkStar1: yes, with the commas
[16:35] <smb> SpamapS, Hm, at least a bit more usage of the net devices. And the container is just created with "lxc-create -t ubuntu -n name -- -r quantal" ?
[16:41] <SpamapS> smb: -t ubuntu-cloud
[16:41] <SpamapS> smb: and -r precise
[16:42] <SpamapS> smb: actually there's also some userdata fed in which does an apt-get upgrade and installs a few things...
[16:43] <smb> SpamapS, I guess I will better wait for your detailed description... ;-P
[16:44] <SpamapS> smb: you can use my juju branch which does it reliably..
[16:44] <SpamapS> smb: bzr branch lp:~clint-fewbar/juju/local-cloud-img
[16:45] <SpamapS> smb: from that dir, PYTHONPATH=$PWD PATH=$PWD/bin:$PATH bin/juju bootstrap
[16:45] <SpamapS> smb: and PYTHONPATH=$PWD PATH=$PWD/bin:$PATH bin/juju deploy wordpress
[16:46] <SpamapS> smb: then the same but 'juju destroy-environment'
[16:46] <smb> SpamapS, Probably you should write that down then *in* the bug report. My memory is very limited (especially when it is actually the weekend)
[16:47] <SpamapS> smb: I'm hoping to have a lower level reproducer
[16:48] <smb> SpamapS, Me too :)
[16:51] <SpamapS> smb: so far, unable to reproduce with your stacktrace happy kernel
[16:52] <SpamapS> smb: n/m, reproduced
[16:52] <SpamapS> smb: want syslog?
[16:52] <smb> SpamapS, Yes please in the bug report.
[16:53] <smb> SpamapS, Probably just a matter of timing as it now is a bit slower with the lots of output
[17:03] <hallyn> smb: even plain libvirt is faling to build.  hopefully i just messed up my localhost, lemme try on a clean instance
[17:05] <smb> hallyn, Yeah, hopefully. Well, in some way. But I could not explain how that test did not bother me otherwise...
[17:27] <frojnd> Hi there. I don't know why but when I try to change /etc/motd as root it won't update it. And when I reopen file with vim it's still last message
[17:27] <SpamapS> hallyn: did you come up with a plan for bisecting btw?
[17:28] <hallyn> SpamapS: bisecting what?
[17:28] <SpamapS> hallyn: I can reproduce 100% of the time on this hardware with the fairly lengthy steps I posted in the bug.. but have not yet boiled it down to anything less than a long juju deploy/destroy
[17:28] <sarnold> frojnd: see update-motd(5)
[17:29] <SpamapS> hallyn: netns bug
[17:29] <hallyn> SpamapS: so you can't reproduce with the test program attached to the bug?
[17:29] <SpamapS> hallyn: no, this is a different problem
[17:29] <SpamapS> hallyn: the simple fix of flushing the route cache addressed that
[17:30] <hallyn> are you sure there's no zombie tasks pinning the netns?
[17:30] <SpamapS> hallyn: to reproduce I have to boot up an ubuntu-cloud based container and then stop it
[17:30] <frojnd> This happens if I'm editing with nano too.. So I must do something wrong. Why after editing /etc/motd as root this new login message won't exist?
[17:30] <SpamapS> hallyn: no, how would I check?
[17:30] <hallyn> anyway no, i've been leaving that to smb to not reproduce effort
[17:30] <hallyn> ps -ef should show the defunct tasks...
[17:30] <SpamapS> nothing there
[17:30] <hallyn> SpamapS: what is the bug#?
[17:30] <SpamapS> bug #1021471
[17:31] <SpamapS> hallyn: I posted the steps to reproduce using my juju branch
[17:31] <hallyn> SpamapS: given that it conflicts with the bug which smb did fix, i'm not sure we can bisect now :)
[17:31] <hallyn> still i could try.  but if smb thinks he's about got it nailed...
[17:32] <smb> hallyn, No not that one. Just got data for it to look trhough
[17:32] <hallyn> ok.  lemme finis up the libvirt one and then i'll see if i can reproduce and join the party
[17:33] <smb> hallyn, Though if you want to run bisecting maybe from the other end (take our mainline kernels between 3.2 and 3.6 as a start)
[17:33] <smb> Usually it turns out to be one -rc1 and then the fun begins
[17:34] <smb> Err make that 3.5
[17:34] <hallyn> smb: happy to.  i was going to do linus' tree, what do you mean by 'our mainline'?  (i think i can find it from old emails, but if you have git url handy...)
[17:34] <hallyn> right :)
[17:34] <hallyn> 3.2..3.6 was going to be my range
[17:34] <hallyn> uh 3.5  wtf?
[17:35] <smb> http://kernel.ubuntu.com/~kernel-ppa/mainline/
[17:35] <hallyn> thanks.  ttyl
[17:36] <frojnd> Is there a manual or something how to change default ssh login prompt? Changing /etc/motd as root doesn't work unfortunatelly.
[17:36] <smb> Well, we know 3.5 is broken (3.6 apparently not) So you could go that way as well, but at least for that I know that the removed the route cache for one thing
[17:36] <hallyn> yeah, i meant 3.5, i don't knwo why i keep mistyping.  ttyl :)
[17:49] <SpamapS> hallyn: while you're still playing, I'm going to try 3.4.12-quantal
[17:49] <SpamapS> hallyn: as in, http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4.12-quantal/
[17:50] <hallyn> sounds good.
[18:09] <hallyn> smb: regression tests still running;  built fine on fresh instance;  intend to push this afternoon.
[18:10] <hallyn> zul: you have no libvirt changes to push to quantal right?
[18:10] <zul> nope
[18:10] <smb> hallyn, ok, phew
[18:26] <SpamapS> hallyn: I am unable to reproduce on 3.4.12 btw
[18:27] <hallyn> SpamapS: in your recipe, should i just 'apt-get install juju'?
[18:27] <SpamapS> hallyn: no
[18:27] <SpamapS> hallyn: it shows, bzr branch the specific branch I'm using
[18:27] <SpamapS> hallyn: it still happens with stock juju, but not as frequently for some reason
[18:27] <hallyn> you don't show building/installing it
[18:27] <SpamapS> bzr branch
[18:27] <SpamapS> just run it from there
[18:28] <hallyn> i did bzr branch, entered it, then you say run 'juju', but juju was not installed (and ./juju is a dir)
[18:28] <hallyn> oh hm
[18:29] <hallyn> so it should be coming from my new path.  but it's not happy with it.  why?
[18:29] <hallyn> (checking)
[18:29] <SpamapS> I set PATH :)
[18:30] <hallyn> SpamapS: well i've since apt-get installed juju, and id o see juju in my path, so i'm guessing there was a dependency of juju which wasn't installed.  (the error has scrolled off ths creen)
[18:30] <hallyn> proceeding
[18:31] <SpamapS> ahh right
[18:31] <SpamapS> python-txzookeeper or python-txaws or twisted or something like that :-P
[18:33] <SpamapS> hallyn: ok, so in the interest of narrowing your bisect ... 3.4.12 is fine.. what would be the other end of the spectrum to try? 3.5.5?
[18:38] <hallyn> SpamapS: not sure, hold on lemme check around which release the route cache was removed
[18:38] <hallyn> btw, first attempt on c1.xlarge did NOT reproduce.  trying again
[18:38] <SpamapS> hallyn: if it fails again, perhaps let me login and I'll see if there are assumptions I've left out
[18:39] <hallyn> heck, the commit "ipv4: reintroduce route cache garbage collector" sounds suspicious :)
[18:41] <SpamapS> hallyn: hah yeah
[18:41] <hallyn> SpamapS: 3.5-rc7 did not have commit 89aef8921bfbac22f00e04f8450f6e447db13e42.  3.6 did.  (that being the commit removing the routing cache)
[18:41] <SpamapS> hallyn: I haven't actually confirmed that 3.6 doesn't reproduce with my particular issue
[18:43] <SpamapS> hallyn: ok, 3.5.5 does reproduce
[18:43] <SpamapS> hallyn: also reproduces with the original reproducer, so no news there
[18:44] <SpamapS> [  191.483843] unregister_netdevice: waiting for lo to become free. Usage count = 2
[18:44] <SpamapS> thats 2, so route cache + "the other thing"
[18:44] <hallyn> SpamapS: hrmph, juju seems to have messed with my ptys
[18:44] <SpamapS> hallyn: ??
[18:45] <SpamapS> hallyn: ok, so I'll try 3.6-rc1 next
[18:46] <hallyn> i ran 'juju bootstrap' in byobu, exited the shell, and now defunct java tasks are pinning /dev/pts/1, and re-starting byobu hangs
[18:46] <hallyn> rebooting, not sure what state i just left things in... sigh
[18:47] <SpamapS> hallyn: oh thats interesting.. zookeeper should have daemonized.
[18:47] <SpamapS> hallyn: thats likely the culprit there
[18:49] <SpamapS> hallyn: so i'll do 3.6-rc1, and then 3.5-rc7
[18:52] <SpamapS> hallyn: yeah looks like we need to use start-stop-daemon to run ZK so it will let go of the terminal
[18:52] <SpamapS> hallyn: you should have just killed the java process
[18:57] <SpamapS> hallyn: AHA!
[18:57] <SpamapS> hallyn: 3.6 still has it
[18:58] <SpamapS> hallyn: well, 3.6-rc1
[18:58] <SpamapS> hallyn: so, the route cache issue is separate
[18:58] <SpamapS> [  253.892068] unregister_netdevice: waiting for lo to become free. Usage count = 1
[18:58] <SpamapS> thats on 3.6 rc1
[19:01] <smb> SpamapS, Unfortunately there are a few additions to the whole mess that are (or where) not even in upstream linux when I looked last
[19:02] <SpamapS> smb: well I think you were looking at the route cache issue and using stgraber's reproducer only
[19:02] <hallyn> SpamapS: interesting
[19:02] <SpamapS> trying 3.6-quantal now
[19:03] <smb> I was also following a few hints from upstream, but they pointed to a follow up on things that happened after removing the route cache. So things may have gone jojo
[19:04] <SpamapS> well if nothing else, its clear that this is a separate bug from the route cache issue
[19:04] <hallyn> SpamapS:  it might be worth opening a separate upstream kernel bugzilla
[19:04] <hallyn> except, of course, we probably need a reproducer outside of juju :)
[19:05] <SpamapS> yeah
[19:05] <SpamapS> I think its something in the timing
[19:06] <SpamapS> hallyn: lxc-stop returns immediately, right, you still need lxc-wait to know that it is stopped?
[19:08] <hallyn> SpamapS: i think so, though the python api one can wait iirc
[19:09] <hallyn> SpamapS: though lxc_stop "sort of" waits, by waiting for the monitor socket to close
[19:11] <SpamapS> ah ok
[19:12] <SpamapS> weird.. why do our cloud images have ntfs installed? :-P
[19:12] <Daviey> SpamapS: ntfs ebs volume ! :)
[19:13] <SpamapS> so much win
[19:13] <SpamapS> hallyn: ok, confirmed still in 3.6
[19:13] <SpamapS> hallyn: I'll try to boil this down to at least just lxc commands to reproduce
[19:16] <hallyn> SpamapS: gah, no, can't reproduce on amazon with juju.  i'll just try quickly running lxc testsuite and see ifthat helps
[19:16] <hallyn> smb: libvirt pushed, thanks.
[19:16] <smb> hallyn, At least something that worked today. :)
[19:16] <smb> thanks
[19:17] <hallyn> haha no kidding.  still no idea about psivaa's bug
[19:18] <hallyn> mdeslaur: if i said that after psivaa installs a new quantal desktop and installs kvm, then virt-manager, virt-manager fails to create/start VMs, but once you manually create one with virsh define and start that, then virt-manager succeeds
[19:18] <hallyn> mdeslaur: would that ring any bells for you?
[19:18] <mdeslaur> hallyn: no...what does virt-manager do?
[19:18] <hallyn> my only guess would be bad perms inside /var/lib/libvirt/images, but that doesn't really make sense either
[19:19] <hallyn> it manages to start kvm, but then kvm bombs with with -EPERM trying to actually use kvm
[19:19] <hallyn> (as seen in the /var/lib/libvirt/qemu/x.log)
[19:19] <hallyn> the bug # is 1057024 fwiw
[19:22] <mdeslaur> hallyn: trying on a fresh install now
[19:23] <hallyn> mdeslaur: thanks.  i've tried that 3 times no with no success, maybe i'm unconsciously doing something blatantly different...
[19:23] <mdeslaur> hallyn: with no success? you mean you couldn't reproduce his issue, or you couldn't get it to work?
[19:24] <hallyn> couldn't reproduce it
[19:24] <hallyn> worked for me every time
[19:29] <mdeslaur> hallyn: worked fine for me
[19:29] <mdeslaur> hallyn: not sure what's going wrong for him
[19:30] <mdeslaur> hallyn: hardware problem?
[19:31] <hallyn> mdeslaur: kvm does run fine by hand, or using 'virsh'.
[19:32] <hallyn> mdeslaur: but there are funky /dev/sda errors in syslog
[19:32] <hallyn> worse, psivaa has *two* machiens that do it
[19:32] <mdeslaur> hallyn: hrm
[19:32] <hallyn> so i started wondering if there is a corrupt archiv emirror...
[19:32] <hallyn> i dunno
[19:33] <mdeslaur> hallyn: I did a fresh install, dit "apt-get install virt-manager", did "apt-get install qemu-kvm", added myself to the libvirtd group, rebooted and everything worked
[19:33] <hallyn> mdeslaur: thanks for trying.  i'm out of ideas
[19:34] <hallyn> psivaa: any chance you can post dmesg from the other machine that also does this for you?  (to the bug)
[19:34] <hallyn> are they the same kind of laptop?
[19:37] <mdeslaur> hallyn: my iso was in my home directory, not sure that's relevant
[19:38] <hallyn> mdeslaur: hm, so was mine.
[19:40] <bgoliveira> Hello everyone. Could anyone can explain me why /etc/resolf.conf turns into a sym link to /run/resolvconf/resolv.conf ?
[19:41] <bgoliveira> I'm having to upgrate my servers from 11.04 to 12.04 ...and I just realised this because a normal user can not use the host comand.
[19:41] <SpamapS> hallyn: ok so here is how lxc-create is invoked..
[19:41] <SpamapS> 3742  execve("/usr/bin/lxc-create", ["lxc-create", "-n", "clint-local-ci-ubuntu-0", "-t", "ubuntu-cloud", "--", "--debug", "--hostid", "clint-local-ci-ubuntu-0", "-r", "precise", "--userdata", "/tmp/tmplp3Y7E"], [/* 13 vars */]) = 0
[19:41] <sarnold> bgoliveira: it's part of the 'resolvconf' package, which rewrites the resolv.conf file for different name servers based on dhcp results
[19:42] <SpamapS> hallyn: and lxc-start..
[19:42] <SpamapS> 3816  execve("/usr/bin/sudo", ["sudo", "lxc-start", "--daemon", "-n", "clint-local-ci-ubuntu-0", "-l", "DEBUG", "-o", "/home/clint/.juju/data/clint-local-ci/units/ubuntu-0/container.log"], [/* 21 vars */]) = 0
[19:42] <SpamapS> followed by..
[19:42] <SpamapS> 3820  execve("/usr/bin/sudo", ["sudo", "lxc-wait", "-n", "clint-local-ci-ubuntu-0", "-s", "RUNNING"], [/* 21 vars */]) = 0
[19:43] <psivaa> hallyn, dmesg attached to the bug
[19:43] <hallyn> psivaa: thanks
[19:44] <mdeslaur> hallyn: this is definitely odd: Could not access KVM kernel module: Permission denied
[19:44] <hallyn> SpamapS: hm, precise container, <shrug> maybe that makes a difference
[19:44] <psivaa> hallyn, one of them is Hp pavilion g6 and the other is Dell Inspiron
[19:44] <hallyn> mdeslaur: right.
[19:44] <SpamapS> hallyn: sure, I'll try a quantal container
[19:45] <hallyn> psivaa: and so where do you store the iso for virt-manager to pick up?
[19:46] <mdeslaur> psivaa: what's "getfacl /dev/kvm" show?
[19:46] <bgoliveira> sarnold for example, I can ping a website. But, ping only can resolv the name because it has a SUID bit active. Host doesn't have a SUID bit active ... any ideia?
[19:47] <Daviey> woot, sbeattie extended his ~ubuntu-server membership.. Great to have you. :)
[19:49] <mdeslaur> psivaa: ?
[19:50] <hallyn> mdeslaur: i fear it's way past EOD for psivaa
[19:50] <mdeslaur> hallyn: consolekit is supposed to set extended acls on the kvm device to your user
[19:50] <psivaa> mdeslaur, hallyn, ohh the iso's are stored on in desktop in one machine and home/myname/iso/ubuntu in anouter
[19:51] <mdeslaur> hallyn: maybe he's using some retro-grouch desktop environment
[19:51] <hallyn> mdeslaur: interesting
[19:51] <hallyn> i hate magic
[19:51] <mdeslaur> psivaa: can you do a "getfacl /dev/kvm"?
[19:52] <psivaa> mdeslaur, just added to the bug
[19:53] <hallyn> mdeslaur: but, /dev/kvm is group kvm, group read-write, and libvirt-qemu user is in default group kvm...
[19:53] <hallyn> so unless virt-manager manages to run kvm not in libvirt-qemu user, ...
[19:54] <mdeslaur> hallyn: he has group::---
[19:54] <mdeslaur> hallyn: incorrect permissions, kvm group doesn't have proper rights
[19:54] <mdeslaur> hallyn: (yeah, the extended acl isn't used)
[19:54] <psivaa> hallyn, mdeslaur on the other machine which now works after following th wiki from hallyn i have group::rw-
[19:55] <mdeslaur> psivaa: so something is breaking the permissions on that device
[19:56] <mdeslaur> psivaa: do you have /lib/udev/rules.d/40-qemu-kvm.rules ?
[19:57] <hallyn> SpamapS: say, ...  you're using juju, so you're not using lxcbr0?  are you using virbr0?
[19:57] <hallyn> well you're not specifying -f, so i guess you must be using lxcbr0
[19:57] <psivaa> mdeslaur, yes
[19:57] <SpamapS> hallyn: this branch does use lxcbr0
[19:58] <hallyn> SpamapS: sadly, i still can't reproduce.
[19:58] <SpamapS> hallyn: yeah I'm working out a series of lxc-* commands to do it
[19:58] <hallyn> doing a loop of while [ $c -lt 100 ]; do sudo lxc-start -n p1 -d & sudo lxc-wait -n p1 -s RUNNING; sudo lxc-stop -n p1; done
[19:58] <hallyn> SpamapS: can you pastebin your userdata file?
[19:59] <psivaa> mdeslaur, just to make sure, i did a fresh quantal install, i did apt-get install kvm and install virtual machine manager using software centre, addded the user to the group and logged out and in back
[19:59] <SpamapS> hallyn: sure
[19:59] <SpamapS> hallyn: thats part of what I'm working on :)
[19:59] <mdeslaur> psivaa: quite odd, I'm not sure how your permissions get like that
[20:00] <mdeslaur> psivaa: if you reboot, do they go back to normal?
[20:02] <psivaa> mdeslaur, i'm not sure which permissions do you mean here, but rebooting alone did not solve the issue, it was running the sequence given in https://wiki.ubuntu.com/SergeHallyn_libvirtnest and then reboot solved it
[20:02] <hallyn> mdeslaur: on the other machine of psivaa's, definately /dev/kvm was rw-rw---- when it was not working
[20:02] <mdeslaur> hallyn: so, two different problems then
[20:03] <Daviey> SpamapS: Hey, are we expecting another juju upload?
[20:03] <mdeslaur> hallyn: it definitely can't work if the kvm group doesn't have access
[20:03] <psivaa> mdeslaur, although i only tried once with the wiki sequence, i could try once more if you think its needed
[20:03] <hallyn> mdeslaur: yeah...  actually, psivaa, wasn't that other lapto (with bad kvm perms) one you had a bad upgrade on?
[20:03] <Daviey> SpamapS: specifically, i want bug 1061286 included :)
[20:03] <gholms> smoser: Any idea why the resizefs cloud-init module opts to create a new device node?
[20:04] <gholms> I *suspect* it's so it doesn't have to attempt to locate the one under /dev, but I'd like to be sure.
[20:05] <SpamapS> hallyn: http://paste.ubuntu.com/1262631/
[20:05] <SpamapS> Daviey: yes I am, and it would include that
[20:05] <SpamapS> Daviey: hoping on Tuesday or Wednesday
[20:05] <hallyn> psivaa: if you reboot the second machine, which doesn't have group write perms on /dev/kvm (as shown by getfacl), does /dev/kvm *then* have group write perms?
[20:06] <Daviey> SpamapS: do you care if i monkey that patch in now, with your upload superseeding it?
[20:06] <smoser> gholms, yes.
[20:06] <smoser> how else would it figure out the device name ?
[20:06] <SpamapS> Daviey: go ahead, we may not get our act together for this 0.6 release in time to squeeze it into quantal
[20:07] <smoser> so it just doesn't bother.
[20:07] <Daviey> SpamapS: ok, thanks
[20:07] <gholms> smoser: By looking it up in /proc/partitions?
[20:07] <gholms> smoser: The problem I'm running into here is that Fedora and friends mount /run with nodev.
[20:07] <gholms> Same with /tmp and a host of other things.  :-\
[20:10] <smoser> gholms, we could just make it in /dev/
[20:10] <smoser> that is likely to not be mounted without 'nodev'
[20:10] <smoser> :)
[20:10] <smoser> err.. wahtever . i meant that probably we can create device nodes in /dev/
[20:10] <smoser> just try that rather than /tmp
[20:11] <smoser> it seems wasteful and pointless to find do a stat on /, get exactly the data we need, then to go looking in some other location for a name that doesnt matter.
[20:13] <gholms> Heh
[20:13] <gholms> smoser: Would you be against a patch that just grabs the device numbers from /, looks up the matching blockdev in /proc/partitions, and uses that directly?
[20:14] <gholms> I mean, I get *why* it works the way it does.
[20:14] <gholms> It just... adds more variables that can break.
[20:15] <sbeattie> Daviey: I like to delude myself into believing I know something about servers.
[20:15] <SpamapS> sbeattie: you know nothing john snow
[20:15] <smoser> gholms, it is less variables.
[20:15]  * SpamapS apologizes.. the GoT withdrawals are getting intense
[20:16] <smoser> you're suggesting more variables (parsing /proc/partitions incorrectly, or /proc not being mounted)
[20:16] <SpamapS> hallyn: success!
[20:16] <sbeattie> SpamapS: no worries... Winter is coming. :)
[20:16] <SpamapS> hallyn: I have a simpler reproducer now
[20:16] <gholms> smoser: touché
[20:16] <smoser> just make the device in /dev/.tmp.cloudinit
[20:16]  * gholms hrms
[20:16] <gholms> Yeah, that's probably the best bet at this point.
[20:16] <Daviey> sbeattie: hah, get a grip.. you've probably done more server package uploads than me :)
[20:17] <SpamapS> hallyn: http://paste.ubuntu.com/1262645/
[20:17] <SpamapS> hallyn: now, here's the tough part.. it only seems to happen if you wait for the system to fully boot
[20:17] <SpamapS> hallyn: the way that script works, you have to press enter after cloud-init says "done booting"
[20:17] <SpamapS> hallyn: interrupting it earlier seems to not reproduce
[20:17] <SpamapS> hallyn: which should actually help narrow down what is causing the issue
[20:22] <SpamapS> hallyn: I need to take a break and get some lunch, but hopefully that helps
[20:22] <SpamapS> hallyn: I wonder if the proprietary WL driver has anything to do with this.
[20:25] <psivaa> hallyn, yes the bad upgrade one is the one with wrong permissions, to check if rebooting recovers the group permissions, i need to reboot *this machine,
[20:28] <Daviey> SpamapS: fancy reviewing http://pb.daviey.com/wTv8/ ?
[20:33] <psivaa> hallyn, mdeslaur rebooting has made the getfacl /dev/kvm -> group::rw- and i am able to create vm's now
[20:35] <mdeslaur> psivaa: interesting...maybe the qemu-kvm postinst isn't setting the device permissions as intended
[20:37] <hallyn> mdeslaur: in the past psivaa has shown /dev/kvm perms to b eright when still unable to create vms
[20:37] <mdeslaur> hallyn: ok
[20:38] <mdeslaur> hallyn: I just confirmed postinst seems to work too, so it's not that
[20:38] <mdeslaur> hallyn: I'm stumped now
[20:39] <psivaa> mdeslaur, im doing a fresh quantal install now ill let you know if rebooting alone gives the req'd permission, iirc it did not,
[20:39] <mdeslaur> psivaa: ok. I'm eod, but leave the info in the bug.
[20:40] <psivaa> mdeslaur, ack
[20:40] <Daviey> SpamapS: would also seem prudent to include http://pb.daviey.com/0DDk/ i guess
[20:48] <SpamapS> Daviey: indeed, that one fixes canonistack
[20:48] <Daviey> SpamapS: you don't make my life easier, you know. :P
[20:48] <SpamapS> Daviey: why not just make a snapshot from trunk?
[20:49] <Daviey> SpamapS: well it's getting that way... you know the CURRENT package FTBFS?
[20:49] <SpamapS> no
[20:49] <SpamapS> but I suspect its a race
[20:49] <SpamapS> had to retry the PPA builds a few times
[20:49] <SpamapS> anyway, I have not eaten yet, and I think if I delay any further my stomach will eat me
[20:49] <Daviey> ok
[20:51] <hallyn> SpamapS: can you use the laptop over wired and not load the proprietary wireless driver?
[21:03] <hallyn> SpamapS: nope, repro.sh still doesn't reproduce it here.  jinkeys!  lemme try that laptop over yonder
[21:20] <hallyn> SpamapS: nope, not there either
[21:44] <SpamapS> hallyn: this one has no wired conn.. but I can try w/ my pro
[21:46] <SpamapS> hallyn: I suppose I can try w/ everything cached and just shut down wireless too
[21:53] <hallyn> SpamapS: yup, should work.  of course it's still possible that it's just your fast ssd making it happen
[21:57] <SpamapS> hallyn: I've had it work quite a high percentage of the time on the spinning rust too
[21:58] <SpamapS> hallyn: ok I forgot to rmmod wl .. but it did happen while on wired LAN and eth1 (wl) was down
[22:02] <SpamapS> hallyn: ok, actually, no.. it was the route cache one
[22:02] <SpamapS> hallyn: so indeed, it may only be my air that reproduces with the route cache fix smb made
[22:32] <hallyn> utlemming: precise cloud images have empty /dev
[22:33] <utlemming> hallyn: I think that might affect oneiric and quantal too
[22:34] <utlemming> hallyn: I believe that it is excluded in the build since the initramfs populates it
[22:34] <SpamapS> hallyn: progress! it only happens when wl is involved
[22:35] <hallyn> utlemming: no, quantal has /dev populated
[22:35] <utlemming> hallyn: right, I just checked....
[22:35] <hallyn> utlemming: ok, i just wanted to make sure it wasn't intentional
[22:36] <utlemming> hallyn: nope...I'll check into what is happening
[22:36] <SpamapS> hallyn: oo, and rmmod on wl makes the problem go away
[22:36] <SpamapS> err no
[22:36] <SpamapS> ignore that
[22:37] <hallyn> utlemming: great, thanks
[22:40] <basil60> hi i've had complaints from isp about my home network sending unsolicited emails. I've run some "extensive" tests on my pc, and found nothing. I suspect it may come from my linux server (ubuntu 11.04). Any suggestions on how I may test it?
[22:41] <sarnold> basil60: if you've got a wordpress or phpbb or something similar installed, that's a reasonable possibility
[22:41] <sarnold> if your machine has been _rooted_, you may not be able to do much examining "from inside" the system. But if it is a simple / stupid exploit, it might be fixable from within...
[22:42] <basil60> not running Wordpress
[22:42] <basil60> I ran "rootkitcheck" yesterday - it found nothinh
[22:43] <sarnold> basil60: does 'netstat -anp' show any unexpected connections?
[22:45] <basil60> can i export that netstat command to a text file?
[22:45] <sarnold> netstat -anp > /tmp/file
[22:45] <sarnold> :)
[22:46] <basil60> netstat -anp >/tmp/file/netstat
[22:46] <sarnold> that'll fail unless you already have a /tmp/file directory
[22:46] <sarnold> netstat -anp > /tmp/netstat would work alright
[22:50] <basil60> dovecot appears to be running - wasn't even sure i had that turned on
[23:02] <SpamapS> smb`: so, good news, the problem is just in the wl drivers..
[23:02] <SpamapS> smb`: see bug report for more info
[23:03] <hallyn> SpamapS: ok, i wasn't sure which order the bug and irc comments went in :)  i thought after the lp comment you decided it was NOT jsut the drivers.
[23:03] <hallyn> so, phew ;)
[23:03] <SpamapS> it *is*
[23:03] <hallyn> and, nice reminder about proprietary drivers, i guess
[23:03] <SpamapS> just that you can't rmmod wl to clear it
[23:03] <SpamapS> I thought wl was supposed to be fully free'd at some point
[23:03] <hallyn> so, WOOT!  smb might have it fixed in quantal release then?

[23:04] <SpamapS> hallyn: yes, his last patch addresses the route cache problem that the reproducing C program shows perfectly
[23:04] <hallyn> SpamapS: right, but we're past kernel freeze
[23:04] <SpamapS> hallyn: ok, so worst case its in that 0-day SRU kernel that we always end up shipping anyway ;)
[23:05]  * SpamapS adds a "its past beer-thirty" disclaimer to that comment
[23:05] <hallyn> all i see there is 'beer', and i'm outta here :)
[23:05] <hallyn> tttyl