[00:54] <SpamapS> Daviey: I'm not sure I expect a spec from that one.. being a roundtable, its more to gather the interested parties in the room and do a checkup on where we're at
[01:03] <AaronMickDee> What's the easiest way to set up a virtual machine server with Ubuntu? I host all my stuff on my box, but a lot of the stuff hosted is for friends. I'd rather just give them a virtual machine rather than a user account. Any ideas?
[01:04] <sarnold> AaronMickDee: qemu/kvm is pretty awesome.
[01:14] <Daviey> SpamapS: I want a drafter that is the person responsible for driving the roundtable then :)
[06:34] <koolhead17> Daviey: around
[07:38] <brontosaurusrex> what would i use for spider/search engine for a small intranet site ?
[07:38] <brontosaurusrex> like a local google
[07:40] <xnox> brontosaurusrex: http://www.searchblox.com/ ?
[07:41] <xnox> brontosaurusrex: or well "Google Mini"
[07:42] <brontosaurusrex> $5000 per server?
[07:42] <brontosaurusrex> it must be free
[08:03] <Daviey> koolhead17: hey
[08:03] <chris|> brontosaurusrex, you could try yacy
[08:42] <brontosaurusrex> tested searchblox, its messy
[08:43] <brontosaurusrex> chris|, p2p part can be disabled i hope?
[09:01] <Japje> zow/w 157
[09:01] <Japje> meh
[09:17] <frojnd> Hi there. I just got an ubuntu 12.4 server to my hands and I'm seeking somekind of a handbook, or server guide. I don't find this https://help.ubuntu.com/12.04/serverguide/index.html guide pretty informative. Can you suggest me any good guides? Or wikis?
[09:25] <xnox> frojnd: what do you actually want to learn / know?
[09:28] <RoyK> !ask
[09:30] <frojnd> xnox_: everything :)
[09:30] <frojnd> Currently I'm adding users that will do some programming on the server
[09:30] <frojnd> I'd like to create a user that has admin access so I can disable root login
[09:32] <xnox_> frojnd: by default root is disabled on ubuntu. the first account has sudo rights. and it looks like you are after "linux CLI" instead of server tasks....
[09:32] <frojnd> I've put the user to sudo group (if it will want to do some maintainance work) like this: # usermod -aG sudo newuser
[09:32] <frojnd> xnox_: ok so I can safely read "debian handbook" for tasks like this
[09:33] <frojnd> xnox_: I got ssh access with only root
[09:33] <xnox_> frojnd: yes. debian handbook, debian-administrator website, http://ubuntu-manual.org/
[09:33] <RoyK> you can turn off root login over ssh in /etc/ssh/sshd_config
[09:34] <frojnd> The problem is that when I login with this newuser and try to do sudo something it gives me incorrect password attempt
[09:34] <frojnd> xnox_: tnx
[09:34] <frojnd> RoyK: but first I have to put some users into admin group I persume
[09:34] <frojnd> don't wanna lock myself out
[09:34] <RoyK> frojnd: sudo prompts for the *user's* password, not the root password
[09:34] <frojnd> RoyK: exactly
[09:34] <frojnd> and it tells me it's wrong
[09:34] <frojnd> let me check if I'm in sudo group
[09:35] <frojnd> with that user
[09:35]  * frojnd is apologizing for multiple lines - won't happen again
[09:36] <frojnd> Ok. So it was my wrong password :/ :)
[09:38] <frojnd> time to disable root login
[09:48] <Adri2000> zul: keystone and python-keystoneclient minor fixes proposed through merge proposal in LP, fyi.
[09:52] <maruq> hi guys
[09:52] <maruq> I'm trying to setup ntpd on 12.04 on ec2.
[09:52] <frojnd> Hm.. I have 2 hard drivs: sda and sdb. Both are 250GB. Now / boot swap etc (system) is innstalled on sda and I've assign 200GB from sdb for /home. But I think 250GB for / is too much?
[09:52] <maruq> as far as I can tell, it's install ntp package, set the servers, then enable independent_wallclock
[09:53] <maruq> 12.04 seems to have moved the /proc/sys/xen dir. any idea where I should be looking for indepenent_wallclock now?
[11:00] <edgy> Hi, I just want to understand where the recovery kernel argument in grub.cfg coming from or documented? or is it ubuntu specific?
[11:04] <rbasak> edgy: for grub 2, try /etc/grub.d and /etc/default/grub
[11:08] <edgy> rbasak: may be you didn't understand my question, isn't ubuntu has an entry that would allow you to go to recovery mode?
[11:12] <edgy> rbasak: : grub.cfg contains linux /boot/vmlinuz-3.5.0-16-generic root=/dev/mapper/vg00-lv_root ro recovery ... What's recovery here referes to?
[11:14] <rbasak> edgy: it signals to the initramfs that recovery is requested. See /usr/share/initramfs-tools/init
[11:16] <edgy> rbasak: but there is no kernel option called recovery, where is this documented? the init file you refer me to contains recovery=y, so what?
[11:17] <rbasak> edgy: userspace can make use of kernel options too. It's common for bootstrapping to pass things into userspace from the bootloader this way. Userspace bootstrap programs examine /proc/cmdline to see all arguments that the kernel received, and the kernel ignores arguments that it doesn't recognise.
[11:17] <rbasak> edgy: /usr/share/initramfs-tools/init parses /proc/cmdline and sets its variable recovery=y if recovery was on the cmdline
[11:18] <edgy> rbasak: ok so the recovery option is passed to which user space program?
[11:18] <maruq> hi guys. anyone know how I enable independent_wallclock in 12.04 on EC2? /proc/sys/xen doesn't seem to exist
[11:18] <rbasak> edgy: the initramfs init script, which comes from /usr/share/initramfs-tools/init
[11:20] <edgy> rbasak: and how the initramfs init script would handle it? the code there is not enough for me to understand
[11:21] <Daviey> maruq: try, sysctl -w xen.independent_wallclock=1
[11:21] <edgy> rbasak: it's just saying for x in $(cat /proc/cmdline); do recovery) recovery=y
[11:21] <rbasak> edgy: not sure, sorry. I recover my systems by hand so never used the recovery feature. I would read the code to find out
[11:21] <rbasak> edgy: so look for where else it uses the recovery variable
[11:21] <maruq> Daviey: just tried. I get 'error: "xen.independent_wallclock" is an unknown key' :(
[11:22] <edgy> rbasak: exactly that was the question I grepped the whole initramfs and no where else used this variable
[11:22] <edgy> rbasak: how do you do it manually?
[11:22] <Daviey> maruq: Ah, HVM instances won't have it.
[11:22] <Daviey> maruq: I think you already get it for free.  I assume you have ntpd running?
[11:23] <rbasak> Apart from on the "exec run-init" line?
[11:23] <rbasak> Looks like it gets upstart to fire a "recovery" event. The next place to look is in upstart's configuration for jobs that fire on that event.
[11:23] <rbasak> I use a recovery disc
[11:24] <maruq> Daviey: yeah, I installed ntp package, went with default settings & restarted the service
[11:24] <edgy> rbasak: I used to do it by passing init=/bin/bash which gives me a recovery shell, too
[11:24] <maruq> Daviey: 'less /sys/devices/system/clocksource/clocksource0/current_clocksource' says "xen"
[11:24] <rbasak> Usually if something's wrong it just drops down to a recovery shell anyway
[11:25] <rbasak> I haven't had to use init=/bin/sh in years
[11:25] <maruq> Daviey: Is there a way I can confirm that it's actually using the timeservers?
[11:25] <Daviey> maruq: Sorry, i don't know.  Maybe someone else will have a better answer
[11:27] <Daviey> maruq: syslog should contain ntpd info
[11:29] <maruq> Daviey: this is what I'm seeing in syslog: https://gist.github.com/f4f927d94e5dec938e06 is that correct?
[11:30] <edgy> rbasak: there seems to be a loop here? ls -l /lib/recovery-mode/recovery-mode/recovery-mode
[11:30] <edgy> lrwxrwxrwx 1 root root 18 Jun 22 15:54 /lib/recovery-mode/recovery-mode/recovery-mode -> /lib/recovery-mode
[11:31] <rbasak> Probably useful when working with chroots
[11:34] <edgy> rbasak: don't know
[11:36] <edgy> rbasak: ok thanks for you help
[11:57] <zapotah> hi
[11:58] <zapotah> after the latest updates something went wrong
[11:58] <zapotah> one of the bridge interfaces wont come up
[11:58] <zapotah> configuration is as follows
[11:58] <zapotah> two nics
[11:58] <zapotah> with lacp bond0
[11:58] <zapotah> vlan interface vlan40 with bridge to bond0.40
[11:59] <zapotah> a few other vlan interfaces as well
[11:59] <zapotah> but the bond0.40 wont come up
[12:01] <zapotah> vlan100 interface with bridge to bond0.100 comes up and cat /proc/net/vlan/config shows that everything is working fine with that one
[12:02] <zapotah> problem being here that atm four servers that have the vlan 40 attached wont start
[12:03] <zapotah> running xen on the server
[12:30] <smb> SpamapS, stgraber, So I got one more iteration to fix bug 1021471. Probably SpamapS could have a look at the latest (and now only) packages at people~smb/clonetst
[12:33] <Daviey> zul: what is the difference between nova-novncproxy and nova-vncproxy?
[12:36] <zul> Daviey:  im not 100% sure
[12:37] <Daviey> zul: it's a change after f3
[12:39] <zul> Daviey:  no difference in my opinon it still uess novnc
[12:39] <melmoth> Daviey, one is to use with horizon, the other with a java client.
[12:40] <melmoth> ohh, may be i m confusing with novnc..
[12:41] <melmoth> from http://docs.openstack.org/trunk/openstack-compute/admin/content/faq-about-vnc.html :
[12:41] <Daviey> zul / melmoth: There difference is the package namees. :).. It seems we switched package names after f3
[12:41] <melmoth> nova-vncproxy was removed from the nova source tree. The Essex analog for this process is nova-novncproxy, which is provided by an external project.
[12:42] <melmoth> so it looks like there has been some name changed from diable to essex.
[12:42] <zul> Daviey: bah sorry i didnt notice
[12:42] <Daviey> hmm
[12:42] <Daviey> we still build it from nova tho?
[12:43] <Daviey> zul: The thread on the OS mailing list.. I'm trying to work out if it's a trivial doc fix
[12:43] <melmoth> i must admit i was really confused finding the right package for the right kind of proxy (novnc for horizon and nova-vncproxy for the java client)
[12:44] <zul> Daviey: ok ill try to reproduce the upgrade issue though and double check things
[12:44] <Daviey> melmoth: yeah, i don't want to give poor advice
[12:44] <Daviey> zul: i did just reproduce it
[12:44] <Daviey> zul: it's not an upgrade issue, a plain install issue
[12:45] <zul> grrr
[12:45] <Daviey> zul: I am about to reply with, "In your apt-get install line, can you replace nova-vncproxy with nova-novncproxy.  This was a change made after Folsom-3, and the documentation hasn't been updated to reflect." - but i want to make sure that is accurate
[12:45] <Daviey> FWIW, it does /install/
[12:45] <Daviey> but is it what the user wants :)
[12:46] <zul> Daviey:  right
[12:46]  * Daviey adds some ubiquity 
[12:48] <zul> Daviey: right you are more awake :)
[12:50] <koolhead17> hey zul
[12:50] <zul> koolhead17: hi
[12:50] <koolhead17> Daviey: which doc you are talking about sir? Do i need to modify something :)
[12:53] <Daviey> koolhead17: yeah, possibly
[12:54] <Daviey> koolhead17: It's not your fault.
[12:56]  * xnox ponders if Daviey meant English noun "ubiquity" or the Ubiquity - Ubuntu Installer....
[12:56] <koolhead17> Daviey: just let me know what changes are needed will do. In a meanwhile with new quantum pkg in place we will push quantum doc too :)
[12:56] <koolhead17> in the same doc
[12:56] <Daviey> xnox: not the installer
[12:57]  * xnox is back in the installer then
[12:57] <Daviey> koolhead17: rocking !
[13:29] <jasonmsp> good morning all..  What is the meaning of  this first part of  a cronjob  "root [ -O /tmp/dir.cache]"  I'm used to seeing if statements as in "root if [-x " but i can't find any reference to this on a web search.
[13:45] <SpamapS> smb: downloading your latest fixed kernel now
[13:46] <smb> SpamapS, Great. :)
[13:49] <skrite> hey all, i am looking for an easy to configure mail server distro i can build and run in a vm for our company... any ideas?
[13:53] <SpamapS> skrite: http://www.turnkeylinux.org/search/luceneapi_node/mail%20type:appliance ... these would probably work
[13:53] <SpamapS> skrite: Zimbra in particular is pretty nice.
[13:54] <skrite> SpamapS: thanks
[14:45] <zul> Daviey: based on andrew's comment on the ec2 bug https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migrate_repo/versions/107_add_instance_id_mappings.py it doesnt happen when the table is created and migrated
[15:25] <koolhead17> i see Daviey live :)
[15:26] <SpamapS> smb: you going to be around for a bit longer? Just now about to test that kernel
[15:26] <chmac> denyhosts has started blocking localhost on a handful of machines in the last 24 hours. Any idea what's going on?
[15:26] <chmac> I'm not aware of anything that's changed on our setup.
[15:40] <SpamapS> chmac: sounds like a bug.. localhost should be whitelisted
[15:41] <chmac> SpamapS: Right, sounds odd, and what caused it to flare up now, on 3 servers, within 24 hours of each other, all with similar configs...
[15:41] <chmac> I'll double check the whitelisting.
[15:41] <chmac> I only noticed the issue because monit can't connect on localhost, so it thinks sshd is down, restarts it a few times, then times out. Meanwhile, it's running all the time.
[15:52] <moose> hey
[15:53] <moose> has anyone compared chroot to virtualbox?
[15:56] <moose> hello?
[15:56] <holstein> moose: hey.. you can just ask.. if anyone knows they will answer
[15:57] <moose> ok
[15:57] <holstein> i have only used Vbox, and its arguably "easier" i would say.. thought the chroot might be less overhead if thats what you want/need.. theres Vbox headless (text mode)
[15:58] <moose> what are the pros and cons of chroot vs [3~[3~[3~[3~[F[3L:/ok
[15:58] <moose> which one is more secure?
[15:59] <holstein> i wouldnt trust any casual answer on the IRC about that... i would say if you are experience, either can be as secure as you need
[16:01] <moose> how much resources does a virtual box need compared to chroot?
[16:01] <holstein> moose: i have only used Vbox, but i would just set both up and see..
[16:02] <moose> ok
[16:02] <moose> thank u
[16:04] <SpamapS> moose: chroot and virtualbox are like a bicycle and a car... pretty hard to compare
[16:04] <SpamapS> holstein: and don't be silly, chroot cannot be as secure as a VM.
[16:05] <moose> my server has only 500mb of m,
[16:05] <SpamapS> moose: heh, thats not a server then :)
[16:05] <SpamapS> you might be using it to serve things, but its not a server computer :)
[16:05] <moose> well, not a sver for virtual box maybe, but chroot might be ok
[16:05] <holstein> SpamapS: i still think it would depend on the use case and the skill level.. but like i said, i use Vbox..
[16:06] <SpamapS> smb: initial result seems positive!
[16:06] <SpamapS> hallyn: ^^ smb's newest kernel seems to address the issue
[16:07] <moose> is 500 RAM not enough for a server? min is only 125mb
[16:07] <SpamapS> crap
[16:07] <SpamapS> [  455.352556] unregister_netdevice: waiting for lo to become free. Usage count = 1
[16:07] <SpamapS> hallyn: spoke too soon
[16:08] <hallyn> what did his newest kernel do?
[16:08] <SpamapS> No idea he hasn't been sharing patches
[16:08] <hallyn> i wonder how hard it woudl be to 'just' yank the routing cache :)
[16:08] <SpamapS> just posting kernels which I try ;)
[16:08] <SpamapS> root     16118  0.0  0.0  27532  1032 ?        Ds   09:07   0:00 lxc-start --daemon -n clint-local-ci-u1-0 -l DEBUG -o /home/clint/.juju/data/clint-local-ci/units/u1-0/container.log
[16:08] <SpamapS> dead
[16:08] <SpamapS> smb: n/m, still broken
[16:08] <hallyn> SpamapS: i meant what approach did that kernel take to trying to fix it
[16:08] <SpamapS> hallyn: *no idea*
[16:09] <hallyn> ok
[16:09] <SpamapS> hallyn: you'd have to ask smb
[16:09] <SpamapS> [  577.079821] rtdbg: fib_disable_ip(lo/ffff880164917000/ffff880073e68000, -1)
[16:09] <SpamapS> seems to be printing that out a lot
[16:09] <hallyn> i did, implicitly :)  irc is a wonderful thing
[16:10] <moose> how do i supress [D[D[D[D[D[D[D[D[D[3~[3~[3~[3~[3~[3~[3~[D[D[D[D[D[D[D[D[3~[3~[3~[3~[3~[3~[3~[3~[F[A[A[A[B[B[B[B[B[B[3~[3~[3~[3~[3~
[16:10] <SpamapS> hallyn: could this be avoided at all by restarting the lxc-net service between lxc-stop/lxc-destroy and lxc-start ?
[16:11] <SpamapS> moose: use the right $TERM setting
[16:12] <hallyn> SpamapS: no
[16:12] <SpamapS> yeah just tried that
[16:12] <SpamapS> hallyn: anything we can put in the instances to clear out the route cache right before halt?
[16:12] <hallyn> SpamapS: what we could try,
[16:13] <SpamapS> at this point, I'm not really confident that we'll have a solution for 12.10's release
[16:13] <hallyn> what we could do is stash away /proc/self/ns/net for the open container
[16:13] <hallyn> or actually, mnt
[16:13] <hallyn> then use that after container shuts down to flush the route cache
[16:13] <hallyn> but really, smb's earlier kernel tried to do that from th ekernel, and it didn't work
[16:14] <hallyn> so my guess would be that your write to the 'flush' file would hang, waiting for a mutex to P
[16:14] <hallyn> i still think that despite assumptions we've made, it would be worth bisecting upstream kernel both for the commit breaking it, and teh commit fixing it
[16:15] <hallyn> noone has AFAIK *verified* that dropping the route cache fixed it
[16:15] <hallyn> if need be i'll spin up a bigmem.xlarge or whatever amazon instance and start the bisection, but i've been waiting to hear whether smb is doing that already
[16:15] <SpamapS> hallyn: right, there is a lot of change between here and there
[16:16] <SpamapS> hallyn: how does that work, bisecting on amazon?
[16:16] <hallyn> SpamapS: does an upstream kernel havethe needed supprot to boot inamazon?  i figure it would work the same way as installing a new kernel .deb
[16:17] <hallyn> i.e i was assuming EBS instances can boot whatever grub says to
[16:17] <SpamapS> hallyn: you'd have to ask smoser and utlemming.. I'm just not sure how it would work
[16:17] <hallyn> still going to wait for smb.  i dont' want to duplicate work
[16:17] <hallyn> meanwhile i have to piss off upstart :)
[16:17] <SpamapS> I admit that I usually put my hands on my ears and sing "Camp town ladies" whenever people talk about the kernel
[16:18] <sarnold> (that's an option? :)
[16:18] <hallyn> yes, but your punishment is dealing with php :)
[16:18] <sarnold> hallyn: ooh, that's cold.
[16:19] <smoser> hallyn, ebs (or instance store) boot with "pvgrub"
[16:19] <smoser> which reads /boot/grub/menu.lst
[16:20] <smoser> as far as upstream kernels having the right stuff, i suspect that current upstream can be configured to boot correctly.
[16:20] <hallyn> smoser: "boot with pvgrub" - where do i specify that?
[16:21] <utlemming> hallyn: that happens automatically with AWS
[16:22] <utlemming> pvgrub is specific to Xen
[16:23] <hallyn> cool
[16:23] <hallyn> thx
[16:24] <smoser> it happens automatically with our images in pvgrub
[16:24] <smoser> you register with a kernel
[16:24] <smoser> our images are registered with the pvgrub "kernel"
[16:24] <smoser> (since 10.04)
[16:49] <RoyK> hm... apt-btrfs-snapshot looks like a jolly good idea, but does it integrate with grub to allow booting from the previous snapshot somehow?
[17:28] <xnox> RoyK: initramfs does not support booting of any subvolumes bug @
[17:28] <xnox> RoyK: so, no. You can change @ to achive the same I believe.
[17:29] <xnox> but I have not tried.
[17:29] <zapotah> i stated earlier that something was broken with the latest updates
[17:29] <zapotah> dont know if its the devs fault or whoevers
[17:30] <zapotah> but specifying in the /etc/network/interfaces a vlan40 interface and a line with bridge_ports bond0.40 does not create a sub interface bond0.40
[17:31] <zapotah> it created a bond0.100 interface which was the first specified
[17:31] <zapotah> in interface vlan100
[17:32] <zapotah> manually specifying bond0.100 bond0.40 etc with vlan-raw-device bond0 works
[17:32] <zapotah> after that stuff works like it used to
[17:33] <zapotah> dont know if the problem lies with ubuntu
[17:33] <zapotah> package vlan or ifenslave
[17:34] <zapotah> can anyone say if ubuntu has the lacp function built-in nowadays or is the ifenslave still required
[17:34] <zapotah> ?
[17:34] <zapotah> ifenslave package*
[17:50] <resno> any suggestions about sys admin blogs i should read?
[17:54] <howdy> Hello. Has anyone here created a dhcp server? If so, if I have a network such as [modem] -> dhcp server -> [switch] -> client .... and the client can talk to the dhcp server... what would be stopping the client from accessing the internet? I tried to open some ports through iptables but it doesnt seem to like me. Any thoughts?
[17:55] <sarnold> howdy: if you're assigning RFC 1918 non-routable addresses to your clients, you'll also need a network address translation firewall: http://en.wikipedia.org/wiki/Network_address_translation
[17:56] <patdk-lap> dhcp server has nothing to do with accessing internet
[17:57] <howdy> Well, I know it's to assign ips. But I guess I'm misunderstanding something.
[17:59] <howdy> sarnold: Thanks.
[18:12] <Poapfel> Everytime I try to restart my network I get this error message http://paste42.de/4195/ is there another way to restart the network?
[18:16] <Jeeves_> Poapfel: No. They broke networking restart while porting it to upstart
[18:17] <hallyn> zul: do you have any ideas on bug 1057024 ?  i don't...
[18:17] <hallyn> i can't reproduce it.  everythign looks kosher
[18:17] <hallyn> it sure looks like libvirt-qemu user simply can't access /dev/kvm.  but perms are correct
[18:18] <sarnold> hallyn: any AA rejects?
[18:19] <hallyn> sarnold: hm, i haven't asked for that info, as i don't see how it's possible :)  but i'm getting desparate enough...
[18:20] <zul> hallyn: i havent seen that before but this looks suspicious:
[18:20] <zul> http://pastebin.ubuntu.com/1260491/
[18:20] <hallyn> zul: i know!  but /dev/kvm is owned by kvm group, with group write perms
[18:20] <hallyn> psivaa: hi
[18:20] <psivaa> hallyn, hi
[18:21] <hallyn> psivaa: sarnold was just suggesting maybe there's apparmor perms problem.
[18:21] <hallyn> can you pastebin the tail end of /var/log/syslog?
[18:21] <zul> hallyn: im asking for the dmesg output as well
[18:21] <psivaa> sure, 1 sec
[18:21] <hallyn> zul: psivaa is the bug submitter
[18:21] <zul> psivaa:  oh hi
[18:22] <zul> hallyn: can you re-load the module
[18:22] <zul> and then restart libvirt
[18:22] <hallyn> zul: note *i* can't reproduce the bug
[18:23] <zul> hallyn: yeah googling doesnt have anything interesting
[18:23] <psivaa> hallyn, zul https://pastebin.canonical.com/75924/ is the syslog
[18:23] <hallyn> wait a sec
[18:24] <hallyn> psivaa: jinkeys, that requires 2-factor auth for me to read
[18:25] <psivaa> hallyn, ohh yea, what other form could i paste?
[18:25] <zul> psivaa:  dmesg please
[18:26] <hallyn> psivaa: oh hey.  do you by chance have vmware or virtualbox installed?
[18:26] <psivaa> hallyn, yes, i do have it installed
[18:26] <psivaa> zul, is canonical pastebin ok for you?
[18:26] <hallyn> gah!
[18:27] <zul> psivaa:  prefer not :)
[18:27] <hallyn> psivaa: can you unload its kernel modules?
[18:27] <psivaa> hallyn, not sure how to do that :)
[18:27] <hallyn> we may want to call this a linux kernel (kvm module) bug, but that'll be the problem
[18:27] <hallyn> psivaa: can you pastebin 'lsmod ' output?
[18:27] <hallyn> or just apt-get purge virtualbox, if you don't actually use it
[18:28] <hallyn> but for testing we should be able to just rmmod the module (once we know its name)
[18:28] <smb> hallyn, SpamapS, Not sharing patches is a lie. You should for a change read the bug report. It basically now does yank the route cache as it is supposed to
[18:28] <hallyn> smb: ?  who is not sharing patches?
[18:28] <smb> With the debug kernel you should actually see it in the logs
[18:28] <psivaa> zul, hallyn: https://pastebin.canonical.com/75926/ is the dmesg, sorry i dont have anyother means now :$
[18:29] <smb> hallyn, Though SpamapS was telling that I would not ;)
[18:29] <smb> though
[18:29] <smb> *thought
[18:29] <smb> gah
[18:29] <hallyn> smb: im' 99% sure he didn't mean whatever he said quite like that
[18:29] <Poapfel> Jeeves_: ?
[18:29] <Poapfel> so there is no way to restart it?
[18:29] <hallyn> smb: are you able to/have you been doing a bisect to determine where the bug was introduced or fixed?  should i be trying that?
[18:29] <SpamapS> smb: indeed I was a liar. :)
[18:30] <hallyn> (bisecting upstream, not ubuntu kernel)
[18:30] <psivaa> lsmod https://pastebin.canonical.com/75927/
[18:30] <smb> hallyn, I am 100% sure I am overstating the fact ;)
[18:30] <smb> The problem with net is that they a) replaced the whole route cache by something different between 3.5 and 3.6
[18:31] <smb> That makes bisecting a major pain in the rectum
[18:31] <SpamapS> smb: I'm still running the affected machine so, do you need any more dmesg's or such?
[18:31] <hallyn> smb: right, but i was thinking of bisecting 3.2 .. 3.5 to figure out where it was introduced
[18:31] <hallyn> smb: hey do you know offhand the name of the virtualbox kernel module?
[18:31] <smb> Plus they have some fixes in the pipe (linux-next) that also targeted a similar sounding issue
[18:31] <Jeeves_> Poapfel: You can ifdown, but that will probably disconnect you
[18:32] <Jeeves_> screen , ifdown ; ifup mght work
[18:32] <smb> hallyn, hm no. Some dkms thing...
[18:32] <hallyn> smb: yeah, finding the commit that fixes it is IMO less likely to work :)
[18:32] <hallyn> ok thx,
[18:34] <hallyn> psivaa: drat, i can't find the virtual box kernel module in your lsmod list.  Do you mind going through software center and removing virtualbox, to see if that fixes it?
[18:34] <smb> hallyn, seems there is 4 of them
[18:34] <smb> hallyn, vboxpci, vboxnetadp vboxdrv and vboxnetflt
[18:34] <hallyn> oddly those aren't apparently loaded though
[18:35] <smb> Neither here apparently. I even forgot I installed it once
[18:35] <psivaa> hallyn, i have purged virtualbox now but still its occurring
[18:35] <hallyn> psivaa: after a reboot?
[18:35] <smb> hallyn, No this time I am lying
[18:36] <smb> Or I cannot type the grep all the times I am trying
[18:36] <hallyn> smb: but if you'd like me to try the bisect (starting tonight) i'm happy to try
[18:36] <psivaa> hallyn, is that ok if we continue a little later or tomorrow :), i need to urgently go out
[18:36] <hallyn> psivaa: of course - thanks!
[18:36] <psivaa> hallyn, sorry aboutt that
[18:36] <smb> SpamapS, When things fail for you. Is it always a 1 reference left?
[18:36] <hallyn> talk to you tomorrow
[18:37] <SpamapS> smb: yes, though I think that number goes up with the number of containers stopped
[18:37] <smb> SpamapS, I believe it was 2 when the route cache is involved (probably a factor of two) but maybe there is another leak left now
[18:37] <Poapfel> my ipv6 connections breaks everytime a few minutes after a reboot, how I am suppose to fix this if I am not able to restart the network?
[18:38] <SpamapS> smb: agreed, I think it was 2 before as well
[18:39] <SpamapS> smb: with #24~smb1 it was 2
[18:40] <SpamapS> smb: and with #24~smb2 it was 1, and has been 1 ever since
[18:40] <smb> SpamapS, Right that also seemed to be consistent with the testcase for which I do not see a problem anymore.
[18:40] <hallyn> woot! progress :)
[18:41] <SpamapS> ok, so what else bumps the refcount?
[18:41] <smb> SpamapS, Could you please post a complete /var/log/syslog file to the bug report
[18:41] <SpamapS> smb: sure, doing that now
[18:41] <smb> SpamapS, Everything that uses the netdev
[18:41] <smb> SpamapS, Ok, then I can look at it tomorrow
[18:45] <SpamapS> smb: attached
[18:46] <smb> SpamapS, thanks!
[18:47] <smb> SpamapS, Oh and btw, with the test case and without patches the problem would go away after somethimes up to 5 minutes
[18:47] <smb> SpamapS, It would be a valuable info if that also happens for this case
[18:48] <SpamapS> smb: it has not gone away in 5 minutes for me.. but I may be screwing it up by doing another lxc-start (which then gets stuck in disk wait)
[18:49] <SpamapS> smb: I'd swear that this time, with your current patch, it took longer for the unregister_netdevice messages to pop up
[18:50] <smb> SpamapS, Yeah, that would add to the theory that we now hit another problem which was hidden by the previous fail.
[18:50] <delinquentme> hey all .. I've got a server process I'm running and I've launched it through a SSH connection .. I would like the process to persist after I've closed the terminal
[18:51] <smb> Though I need to carefully go through the log there and compare addresses
[18:51] <delinquentme> how can I do this?  ... i thought $ bundle exec trinidad & might do it
[18:51] <delinquentme> that is not the case
[18:51] <SpamapS> smb: are you reproducing locally still btw?
[18:52] <sarnold> delinquentme: investigate screen, tmux, and nohup. nohup is easiest IFF it works.\
[18:52] <SpamapS> delinquentme: if you just want it to keep running and you don't care much about being able to interact with it.. 'nohup programname &' does it
[18:52] <delinquentme> actually I think i got it :D
[18:52] <smb> SpamapS, Well for me the problem was sort of fixed. But I only use the test C code and not too many times. So no
[18:52] <SpamapS> smb: ok .. its fairly simple for me to reproduce with lxc-create/lxc-start/lxc-stop
[18:54] <smb> SpamapS, I guess I will have to try those steps on further debugging or see what would happen when modifying the test case a bit
[18:57] <SpamapS> smb: sounds like something during the actual machine boot bumps the refcount and then the tear-down is missing that
[18:59] <smb> SpamapS, Sounds like I should definitely see to be able to reproduce this locally and then enable the ugly warn_on on every get and put.
[19:00] <smb> SpamapS, Anyway what is interesting is that stgraber is less likely hitting that but would be doing the same thing...
[19:00] <SpamapS> smb: the dreaded race condition
[19:01] <SpamapS> stgraber: ^^ are you testing on a system w/ SSD or slow disk?
[19:01] <SpamapS> My SSD box definitely reproduces 100% while my older laptop sometimes doesn't.. I think.. its all me guessing
[19:02] <smb> The disk would be a less likely suspect but who knows.
[19:02] <SpamapS> well...
[19:02] <SpamapS> if things happen in a different order..
[19:02] <SpamapS> and there is a missing lock somewhere
[19:03] <SpamapS> would explain a counter skew
[19:03] <SpamapS> smb: I'm thinking the boot and/or shutdown of the container and its network configuration might be racing something else.
[19:04] <SpamapS> like, lxc-wait .. which I realize now is in the mix, I just haven't been considering it part of the equation
[19:05] <smb> SpamapS, I would not completely rule it out, its just one suspect that rather comes later to your mind. First with how those refcounts are handled and net involved you would think of rcu and cpu based races...
[19:11] <Aresby> I'm considering mv'ing production resources to ubuntu-server.  Been exploring what's "in the release", as well the whole PPA infrastructure.  General question re: kernel -- how "vanilla" upstream are the kernels u-s uses & makes available?  Are they typically UN-patched upstream releases? more in line with Opensuse-esque franken-kernels?  Or something in-between?
[19:11] <Aresby>   For my usecase, I'd *prefer* to be as close to upstream kernel + upstream Xen (virtualization is mission-critical here) as possible ...
[19:12] <patdk-lap> aresby, I don't know anything about suse kernels
[19:12] <patdk-lap> should be just like redhat kernels
[19:12] <patdk-lap> a kernel version is picked, and that version is stuck to, and security patchs are backported to it
[19:13] <patdk-lap> there are some custom stuff, but not much
[19:13] <patdk-lap> the custom stuff I know of, is attempting to be ported into the normal kernel
[19:13] <smb> Aresby, Quite upstream + stable patches. There were one or two patches but those are gone or are not really changing much.
[19:14] <smb> (or being separate drivers)
[19:15] <Aresby> patdk-lap: ok, thanks.  I'll continue to read-up.  How amenable to DIY-build/use of newer kernels is U-S?  I assume there's likley PPAs that already do this --- but, if i DIY, I'd like the process to be less filled with distro-specific "gotchas".
[19:15] <Aresby> smb: ARe thos patches Ubuntu-community generated/maintained?  From/thru Canonical?  or from Upstream itself?
[19:16] <patdk-lap> I normally just download the source, add my patch into the build
[19:16] <patdk-lap> and push it to a ppa :)
[19:16] <patdk-lap> simple
[19:17] <Aresby> patdk-lap: Heh, sure *that* part is.  It's the "Oh, btw, your custom-built kernel is gonna smoke your box" stuff that I'm worried abt ;-)
[19:17] <Aresby> plus the "and don't let the door hit you in the ass on the way out of #irc" Centos-isms ...
[19:18] <smb> Aresby, Community maintained in some sense I would say. Like aufs/overlayfs, some dm-raid45 one. Probably community in a wider sense.
[19:19] <smb> Aresby, If you look at our public git trees there is a ubuntu subdir which contains all those. Everything else is more or less like a current upstream stable kernel of the same version
[19:19] <Aresby> smb  ok.  tbh, I *am* banking on the assumption that I don't have to explain to folks in _here_ what 'enterprise' means, or why I might want a custom kernel build -- so "community" is far less of a risk.
[19:25] <Aresby> Thanks!
[19:28] <zapotah> so, anyone know if somethings changed with vlan or ifenslave or something with ubuntu networking for my described problem to pop up out of the blue with updates
[19:31] <zapotah> in /etc/network/interfaces thers an lacp bond interface over which ive created a few vlan bridges
[19:32] <zapotah> after some updates this past few weeks i encountered a problem today in which i updated the xen hosts and they were apparently unable to create bond0.x subinterfaces corresponding to the vlans
[19:33] <zapotah> had to manually specify the bond0.x subinterfaces for it to work again but i was wondering if theres been some fundamental change to the networking workings as of lately
[19:34] <zapotah> anyone?
[19:34] <zapotah> anyone at all?
[19:36] <zapotah> i lack the ability to troubleshoot the code and the inner workings of the linux networking stack so im asking if anyone would have any insight into this problem
[19:37] <zapotah> we have production systems running with the exact same configuration as the lab setup i updated today and it would be extremely embarassing for them to stop working because of an update
[19:39] <sarnold> zapotah: (if you're completely stuck here, consider also serverfault.com -- feels appropriate to me there, too, unless you've got a debian-specific bug of some sort..)
[19:47] <Aresby> Iiuc, launchpad is the primary (only?) jumping off point for finding newer-than-release PPAs.  Still getting my sea-legs:  starting with search@launchpad, how do I find a/the "most-likely-to-be-production-reliable" PPA/pkg for a server app?  Let's, e.g., say MySQL v5.6 ...
[19:52] <zapotah> sarnold: im not stuck because the way i solved the problem in the lab env propably works for the production env as well.
[19:52] <maswan> Aresby: production-reliable? then you need to know who runs it and trust them to ship [security] updates at a resonable pace.
[19:54] <patdk-lap> aresby, percona
[19:54] <zapotah> sarnold: my concern is that the problem is more deeply rooted and as I am unable to debug it further than the functionality and how im able to work around the bug im somewhat concerned as to how to make sure the problem wont fatally affect the production systems
[19:55] <Aresby> maswan: Well,  production-reliable-er ... Where I hail from (opensuse) there are different shades of repos -- from the 'official', to 'almost, or will be, official' to end-users' "hobby" repos.  I'm trying to get a sense for what's what @ubuntu.
[19:55] <Aresby>   patdk-lap That's not the question -- MySQL is an example.  I'm asking about ubuntu repos
[19:55] <patdk-lap> but if your looking for, production/stable/reliable, you are NOT looking for mysql 5.6
[19:55] <patdk-lap> aresby, heh?
[19:55] <sarnold> zapotah: makes sense :) It's way outside of my experience, and I wanted to make sure you weren't stuck with something bad ;)
[19:56] <patdk-lap> you aren't making sense, if it isn't in ubuntu main repo, it's not going be production/stable/reliable/secutity patched
[19:56] <patdk-lap> unless it's done via a 3rd party
[19:56] <Aresby> patdk-lap: Great.  Pick another app -- that's newer than what ubuntu-release ships.  The specific app is irrelevant to my question.
[19:56] <patdk-lap> and your completely on your own there
[19:56] <patdk-lap> yes, and your going have to either do it yourself, or find someone you trust/pay to do it
[19:57] <patdk-lap> it's the same way for rhel
[19:57] <patdk-lap> no difference
[19:57] <Aresby> So @ubuntu it's either in "main" or it's not dealt with in bugs/lists/community etc?
[19:58] <patdk-lap> main is handled
[19:58] <maswan> patdk-lap: eh, some of it can be decently supported. like pitti's postgresqls.
[19:58] <patdk-lap> anything else is handled by whoever handles it :)
[19:59] <maswan> but yeah, if you need something newer than currently released, you're usually better off waiting 6 months and jumping onto a non-LTS
[19:59] <patdk-lap> rhel is the same, they handle what is in their repo, what is outside it, they don't handle, epel, or any other repo you locate
[20:00] <Aresby> maswan: Assuming that "pitti's postgresqls" means a PPAC that's widely adopted/used, and/or built by someone s with good track-record, how does one go about find the "pitti-ish" repos?
[20:00] <patdk-lap> if you want newer than what is supported, it's not going be production/stable/... by ubuntu
[20:01] <zapotah> I know that the people on this channel are propably not financially invested in the development of ubuntu so theyre not bound to give and find answers to difficult problems but if canonical makes folks to believe that ubuntu is an enterprise ready platform i wish problems like this wouldnt rise no matter what
[20:01] <maswan> Aresby: By knowing and maybe having a chat with the maintainer.
[20:02] <patdk-lap> zapotah, I can't help you, I have never had your issue, my vlans and bonded interfaces work fine
[20:02] <Aresby> maswan: Not ideal, but fair/reasonable point.  Thanks.
[20:02] <zapotah> patdk-lap: any idea what couldve caused this kind of problem with a xen host?
[20:03] <zapotah> its work-aroundable sure
[20:03] <patdk-lap> heh xen?
[20:03] <patdk-lap> try simplifing the issue
[20:03] <maswan> Aresby: some might have stated policies. but in general you don't want to run production services on things outside the dist
[20:03] <patdk-lap> your throwing too many *issues* into it
[20:03] <zapotah> patdk-lap: how come?
[20:03] <patdk-lap> to hard to know *who* or *what* broke it
[20:04] <patdk-lap> xen has been gone from ubuntu for a long time now
[20:04] <Aresby> maswan: Sure.  Just a matter of "outside the dist" means different things @ different distros -- and I'm learning @ubuntu.
[20:04] <zapotah> patdk-lap: uhh how come? the latest xen has been provided by the main repo until a few months ago
[20:04] <patdk-lap> anything outside the dist, would be, not by default included :)
[20:05] <maswan> Aresby: there is a bit of difference between "main" and "universe" too, "main" is promised support from canonical, "universe" is "community supported". real security fixes go fine in universe, but getting bug fixes through kan be iffy at times depending on who is handling it etc.
[20:06] <zapotah> havent checked lately but xen 4.1.2 was provided by the main repo until just a little while ago
[20:06] <zapotah> i know ubuntu favors kvm nowadays
[20:06] <patdk-lap> I dropped all xen support back when it went out in 10.04
[20:07] <patdk-lap> but still, does the issue only happen when using xen?
[20:07] <patdk-lap> do you know?
[20:07] <zapotah> unfortunately i dont have a non-hypervisor host to make sure
[20:08] <Aresby> zapotah: xen 4.2 builds and functions cleanly on Ubuntu 12, fwiw.  One of the reasons I'm looking at switching TO ubuntu.  this was my starting point: https://help.ubuntu.com/community/Xen
[20:08] <zapotah> Aresby: i know ive tried
[20:11] <zapotah> Aresby: xenapi and ovmf support at build requires some modifying of the Makefiles and env variable¨s
[20:11] <maswan> Aresby: That said, if you are really needing something not in the dist, using a ppa is a good starting point though, and a good starting point if you need to build your own packages.
[20:12] <Aresby> maswan: Noted, thanks.
[20:12] <maswan> Aresby: you just want to be sure you're following upstream's security announces etc in case the ppa goes stale
[20:13] <Aresby> maswan: yep
[20:14] <Aresby> zapotah: there _were_ some mods required in my 1st runthroughs.  In general, it seems that Xen upstream's "most native" dev env is Debian.  Which, iiuc, should make Ubuntu easily doable.
[20:17] <Aresby> maswan: "really needing something" varies.  I'm certainly not a glutton for punishment, but it's oft-happened that an 'official repo' @distro has pkg-brokenness with little interest in, or urgency about, pushing fixes.  When that happens, I look for those "pitti-ish" repos, or DIY ...
[20:23] <zapotah> Aresby: i think it required the modifying of the static GCC44 variable and the nonfuncioning autodetection of the XML anc CURL configuration variables
[20:24] <zapotah> and with ovmf the version difference of GCC between debian and ubuntu
[20:26] <zapotah> but after dealing with those it compiled without problems
[20:26] <Aresby> zapotah: yep, reading Xen@Debian info atm ...
[20:27] <zapotah> ashamed i have to admit i didnt have time to actually try it out if it actually worked
[20:27] <zapotah> but i would guess it did
[20:30] <zapotah> libvirt has some problems compiling atm with xen 4.2 but i guess everyonell have to wait for them to patch that atm
[20:30] <zapotah> problem with libxl.h api
[20:31] <zapotah> tried to debug that but again it goes above my field of expertise
[20:33] <Aresby> zapotah libxl challenges seem to be vendor-agnostic; there's enough to go around.  My hope is getting closer to upstream -- @ both kernel & xen -- will simplify matters.