| MrHeavy | I'm having an issue with cloud-init+AWS on 12.04 where #include doesn't seem to be working the way I expect | 00:42 |
|---|---|---|
| MrHeavy | user-data.txt contains the #include directive I put there, but the resulting cloud-config.txt is empty | 00:43 |
| MrHeavy | No errors in cloud-init.log | 00:43 |
| MrHeavy | Any ideas on how I can get some kind of useful output? | 00:43 |
| Zhenjin | I am getting ic2 ic2-3: sendbytes: NAK bailout. messages, is this a serious issue or something i can ignore? | 00:57 |
| sarnold | Zhenjin: I don't know if it'll help you understand it better or not :) but the comment near that error in the kernel source code says "A slave NAKing the master means the slave didn't like something about the data it saw. For example, maybe the SMBus PEC was wrong." | 01:01 |
| sarnold | (at least, I assume you mean i2c, not ic2) | 01:01 |
| Zhenjin | Hmm, well it doesnt appear to cause any trouble besides showing that message every once in a while, does it cause damage behind the scenes that stops other things from working? | 01:06 |
| Zhenjin | and yea its i2c not ic2 | 01:07 |
| sarnold | Zhenjin: i2c is often used for things like temperature reporting.. maybe keep an eye on those numbers, some might not always make sense? | 01:07 |
| Zhenjin | check the temparatures? alright, from google i find that i need to download something for it sudo apt-get, havent got to work yet tough DNS issue, is there another way to do that? | 01:11 |
| sarnold | the 'sensors' program from 'lm-sensors' package is definitely the easiest way... I'd get your DNS working first. :) | 01:13 |
| Zhenjin | http://imgur.com/YQOwgPU Here is some info on what a tried for the dns | 01:31 |
| Zhenjin | maybe you can see what i did wrong :)? | 01:31 |
| sarnold | Zhenjin: you don't want 'search google.com' unless you're actually on google's network and want to refer to hosts in the google.com domain without specifying their FQDN :) | 01:33 |
| sarnold | Zhenjin: but that doesn't seem like it would give you the errors you've got | 01:33 |
| Zhenjin | should i delete the search google.com or change it to something else? | 01:36 |
| sarnold | I'd just delete it, I haven't used 'search' in 15 years and don't miss it much :) | 01:37 |
| Zhenjin | done, what would likely be the issue besides that? something in particular i should look for? | 01:38 |
| sarnold | Zhenjin: can you contact 8.8.8.8? | 01:38 |
| sarnold | does ping work? how about host www.google.com 8.8.8.8 ? | 01:39 |
| Zhenjin | zhenjin@ZhenServer:~$ ping 8.8.8.8 | 01:42 |
| Zhenjin | PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. | 01:42 |
| Zhenjin | after that it doesnt give zhenjin@ZhenServer:~$ anymore | 01:42 |
| Zhenjin | just empty chat | 01:42 |
| Zhenjin | is it just me being impratient? | 01:42 |
| sarnold | Zhenjin: no, you should see ping responses near immediately | 01:43 |
| sarnold | (just hit ^C to kill ping) | 01:44 |
| Zhenjin | --- 8.8.8.8 ping statistics --- | 01:44 |
| Zhenjin | 174 packets transmitted, 0 received, 100% packet loss, time 174384ms | 01:44 |
| sarnold | Zhenjin: there's your next project :) figure out why you can't do UDP or ICMP packets with 8.8.8.8 | 01:44 |
| sarnold | Zhenjin: try also 4.2.2.1, that's another publicly available DNS recursor | 01:45 |
| Zhenjin | replace it in resolv.config? | 01:45 |
| sarnold | Zhenjin: try pinging it first | 01:45 |
| Zhenjin | Nope, whats the ^C key? | 01:47 |
| sarnold | Zhenjin: it's a terminal interrupt key -- it sends a the SIGINT signal, which often kills a program | 01:48 |
| Zhenjin | which key on my keyboard? :p | 01:49 |
| === rubberneck1 is now known as rubberneck | ||
| sarnold | Zhenjin: hold down the control key while pressing the C key | 01:51 |
| Zhenjin | ty | 01:51 |
| Zhenjin | --- 4.2.2.1 ping statistics --- | 01:51 |
| Zhenjin | 303 packets transmitted, 0 received, 100% packet loss, time 304141ms | 01:51 |
| Zhenjin | so ill try changing it in the resolv.conf? | 01:51 |
| sarnold | darn. | 01:52 |
| sarnold | no, it probably wouldnt work either. | 01:52 |
| sarnold | do you control the 192.168.2.1 gateway? it might be the problem.. | 01:52 |
| Zhenjin | shall i send a screenshot of cmd ipconfig? | 01:53 |
| sarnold | sure | 01:53 |
| Patrickdk | on windows now? | 01:54 |
| sarnold | hehe, I assumed it was a typo :) | 01:54 |
| Patrickdk | unlikely :) | 01:55 |
| Zhenjin | http://imgur.com/Mb0P0ai | 01:55 |
| * sarnold owes Patrickdk another beer :) | 01:55 | |
| Patrickdk | hehe | 01:56 |
| Patrickdk | that was just too funny | 01:56 |
| sarnold | Patrickdk: I ohpe you're keeping track of how many beers I owe you :P | 01:56 |
| rubberneck | Zhenjin: you have a differnet gateway in your linux box | 01:56 |
| Patrickdk | that is a lot of gateways | 01:57 |
| Patrickdk | I wonder what one is being used | 01:57 |
| sarnold | Zhenjin: it looks like you're trying to use the windows box as a gateway; if that is correct, you'll need to configure it to perform Network Address Translation while forwarding packets | 01:57 |
| sarnold | (hey, what's the point of "default gateway" on each NIC??) | 01:58 |
| Patrickdk | and it has too much gateways :) | 01:58 |
| * Patrickdk wonders what route print, shows | 01:58 | |
| sarnold | I think windows forgets what "default gateway" actually means. | 01:58 |
| Patrickdk | na, it depends what is actually pluged in and working | 01:59 |
| Patrickdk | but if both are, that isn't really defined and windows warns about it | 02:00 |
| Zhenjin | so i need to put the gateway my cmd shows 192.168.2.254 into /etc/network/interfaces? | 02:00 |
| sarnold | Zhenjin: probably not | 02:00 |
| sarnold | I don't think that would work, unless you actually -do- get your internet from 192.168.2.254, rather than 25.96.34.38, which feels far more likely to me | 02:01 |
| Zhenjin | the hamachi 1p4 address? | 02:02 |
| sarnold | yes | 02:02 |
| Zhenjin | alright ill try it | 02:03 |
| appleguru | Any network gurus online? | 02:03 |
| sarnold | Zhenjin: good luck :) I'm off to dinner | 02:03 |
| appleguru | I have an ubuntu server box setup with 2 NICs. eth0 and eth1... | 02:04 |
| Zhenjin | alright, have fun eating | 02:04 |
| appleguru | I have eth0 with a 10.1.2.50 address, 255.255.255.0 subnet mask... | 02:05 |
| Zhenjin | nope hamachi ip4 didnt work | 02:05 |
| appleguru | And eth1 with a 10.1.75.20 address, 255.255.255.0 subnet mask | 02:05 |
| appleguru | if I plug my computer into either port, with appropriate similar settings... I can reach servers running in my Ubuntu box at either IP address | 02:06 |
| appleguru | Any idea why? | 02:06 |
| appleguru | (I'd expect to only be able to reach the box on the correct subnet for a given port, but that's not what I'm seeing) | 02:08 |
| === aNDREL0|2 is now known as aNDREL0 | ||
| === tim is now known as thumper | ||
| martisj | morning | 05:47 |
| martisj | how do i get a list of the versions that will be installed when updating php-apc | 05:47 |
| martisj | through apt-get | 05:47 |
| martisj | is it possible to see when a package was updated last? | 06:02 |
| ScottK | martisj: Use -V with apt-get to see the versions. | 06:08 |
| sarnold | martisj: /var/log/dpkg.log | 06:19 |
| babinlonston | Hi any one there to help me about ubuntu linux server backup | 06:42 |
| sarnold | babinlonston: I'm headed to bed, so just some quick pointers: duplicity and bacula | 06:42 |
| babinlonston | ok fine | 06:43 |
| sarnold | babinlonston: I use rsnapshot from one drive to another on my laptop, it's nice, but not off-site. off-site is on my todo list. | 06:43 |
| babinlonston | me to gone through rsnapshots its nice but , one question , only its possible to take backup by root user ? can i add a separate user as backup-user and can i get the privilege of root to take backups of configuration files and User's files is it possible ? | 06:45 |
| andol | babinlonston: rsnapshot can run as any user, even if some parts of the default config might assume the root user. | 06:46 |
| babinlonston | Will rsnapshot take the backup of files which have root ownership from sysadmin user ? | 06:48 |
| babinlonston | or did i need to add the sysadmin user to sudo group ? | 06:48 |
| andol | babinlonston: That really depends on how the rest of the ownership settings for a file looks like. | 06:49 |
| babinlonston | ok ill try | 06:49 |
| sarnold | babinlonston: if you want to run rsnapshot from a user's crontab and back up just that user's files, that's fine, but the configuration file may need some .. configuration :) | 06:49 |
| andol | babinlonston: I guess the most flexible thing to do would be to use acl:s and expcitly give read rights, and only read rights, for your backup user. | 06:49 |
| === smb` is now known as smb | ||
| babinlonston | good point u given ill try and let u know | 06:50 |
| === Ursinha-afk is now known as Ursinha | ||
| dreibaume | hi, aa-logprof always tells me "Log contains unknown mode apparmor=. ". anyone had this problem before? | 10:48 |
| jdstrand | dreibaume: can you file a bug at https://bugs.launchpad.net/ubuntu/+source/apparmor/+filebug (ideally using 'ubuntu-bug apparmor') | 10:53 |
| === Ursinha is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| roboto_ | on fresh install trying to figure out why after setting up Samba the folders in my network look like servers?? sorry if this is a crosspost, couldn't find a #ubuntu-network. Other computer with same config. works fine 99% of the time | 11:38 |
| reuf | hello anyone worked with hsphere? | 11:59 |
| rbasak | hallyn: if I use lxc-start-ephemeral, is there any way to see the root filesystem of the container from the host? I see eg. /var/lib/lxc/.../delta0, but this is only have of it; I want to see the whole thing as the guest sees it. This is so that the host can wait for /var/lib/cloud/instance/boot-finished without having to interpret the overlay in an overlayfs-specific way. If this isn't available right now, would you entertain a wishlist item to bind | 12:21 |
| rbasak | only half of it | 12:21 |
| rbasak | !anyone | reuf | 12:23 |
| ubottu | reuf: A high percentage of the first questions asked in this channel start with "Does anyone/anybody..." Why not ask your next question (the real one) and find out? See also !details, !gq, and !poll. | 12:23 |
| hallyn | rbasak: we could come up with other ways, but I would think that lxc-attach would be the simplest way - is there any reason that wouldn't work? | 12:57 |
| hallyn | sigh, not sure what's going on, but spamassassin doesn't seem to be doing as good a job as it used to in filtering binary-looking spam. i keep sa-learning it every morning, but the same amount keeps coming through | 13:01 |
| rbasak | hallyn: it's a bit awkward programatically. I have to come up with a shell rune to get what I want. I suppose I could keep calling stat(1) until I get what I want, but I'm currently calling lstat(2) which I feel gives me more control. But in this case, it would work, yes. It's nice to be able to get to the container filesystem from the host though, for example I can copy and examine stuff directly from my own environment, instead of having to work in | 13:01 |
| rbasak | hallyn: it's the text scraping I don't like, IYSWIM. | 13:02 |
| hallyn | rbasak: we can add a 'container.rootfs_mount(target)' api which would be only a few lines (based on the existing bdev stuff) | 13:02 |
| rbasak | hallyn: I already hit /var/lib/lxc/.../rootfs quite a lot - that's what I lose with epehemeral containers | 13:02 |
| hallyn | rbasak: then you could watch under target. The problem with that, | 13:03 |
| hallyn | is that users doing what you're doing - with containers which are using tmpfss mounted at boot - would get results they didn't expect | 13:03 |
| rbasak | The engineering perfection way would be to inotify on /var/lib/lxc/.../var/lib/cloud/instance/boot-finished, and that's awkward without having inotify-tools in the guest. | 13:03 |
| hallyn | rbasak: it's not engineering perfection for a few reasons: | 13:03 |
| hallyn | 1. fs may not support inotify, | 13:03 |
| rbasak | OK, so fall back to poll in that case | 13:04 |
| hallyn | 2. container's /var/lib/lxc may be mounted by container userspace | 13:04 |
| hallyn | so just mounting the container's root would not suffice | 13:04 |
| rbasak | The container's /var/lib/lxc? | 13:05 |
| hallyn | yes | 13:05 |
| hallyn | ok, the container's /var/lib/cloud | 13:05 |
| rbasak | I'm not nesting here. | 13:05 |
| rbasak | Oh OK | 13:05 |
| rbasak | Yeah it does require that the host has knowledge of what the guest is doing here. | 13:05 |
| hallyn | rbasak: is there any reason not to just look at /proc/$(container-init-pid)/root/var/lib/cloud/instance/boot-finished? | 13:05 |
| rbasak | It didn't occur to me to do that! | 13:06 |
| rbasak | I just wanted the host to get access to the guest's fs. If I can do it that way, then great! | 13:06 |
| hallyn | it's usually my go-to-way to inspect a running containre's rootfs | 13:06 |
| rbasak | How do I get the container-init-pid? | 13:07 |
| hallyn | lxc-info -p -n $name | 13:07 |
| rbasak | OK, thanks. It still feels a bit ugly having to scrape the output of that. Two requests for enhancement. 1) an option to print the pid only, without the field name; 2) a symlink to /proc/$(container-init-pid)/root available in /var/lib/lxc/.../something :-) | 13:09 |
| rbasak | But that'll do great for now. Thank you! | 13:09 |
| hallyn | lxc-info -p -n $name | awk -F: '{ print $2 }' :) | 13:09 |
| hallyn | but i agree | 13:09 |
| hallyn | rbasak: if you wanna open bugs for each of those, they both sound reasonable - but wishlist - items. | 13:10 |
| rbasak | It's fine from shell but feels particularly ugly in Python, even though I just need to do .split()[1] | 13:10 |
| rbasak | hallyn: OK - thanks. I understand they're wishlist. Wouldn't expect anythign else. | 13:10 |
| hallyn | \o | 13:10 |
| rbasak | hallyn: interesting. I was interpreting /var/lib/lxc/$name/delta0/var/lib/cloud/instance/boot-finished as a normal user. I can't do that via /proc. Though that's reasonable, I was using sudo to call all the lxc- commands, so I'll need to think about how to get to boot-finished now. If I have to call out to sudo and parse the output, I might as well write a shell snippet to do the test, and call sudo lxc-attach -- $snippet. | 13:19 |
| hallyn | rbasak: well the downside to lxc-attach is that the snippet then needs to exist inside the container's rootfs | 13:20 |
| rbasak | That's true, but here I just need a shell and stat(1) so I should be OK I think. | 13:20 |
| rbasak | What do you think about /var/lib/lxc/$name/rootfs not being 0700? | 13:21 |
| hallyn | rbasak: as I say, I also don't mind adding a 'container.mount_root(target)' api bit, I just fear people will misunderstand/misuse it | 13:21 |
| hallyn | rbasak: /var/lib/lxc/$name/rootfs being 0700 is not the problem | 13:21 |
| hallyn | the problem is it's not unconditionally the rootfs :) actually lxc doesn't make it 0700 | 13:22 |
| rbasak | It's not 0700. It's 0755. | 13:22 |
| rbasak | Right. I'm just observing that it's 0755 and wondering if it's an issue that unprivileged users can get there in the first place. | 13:22 |
| hallyn | Eh, if you wanna keep that stuff secret you can also use /usr/share/lxc/hooks/mountecryptfs :) | 13:23 |
| rbasak | :) | 13:23 |
| hallyn | stgraber: do you think a 'container.mount_rootfs(target)' would be a useful api extension, or lead us to trouble with people who expect the container's mountall to also have run? | 13:24 |
| stgraber | hallyn: hmm, I can think of a lot of corner cases and relatively few use cases for it, do you (or someone else) actually have a need for it? | 13:26 |
| hallyn | stgraber: rbasak wants to watch for a file to be created in the container that shows the container is booted | 13:26 |
| stgraber | hallyn: and he wants to double-mount the rootfs for that? won't that cause possible fs corruption? | 13:27 |
| rbasak | stgraber: I was proposing a bind mount. I don't mind how to achieve this, but I need to bring up an lxc container and then do stuff in it automatically (and then tear it down). I'm working on adt-virt-lxc. | 13:28 |
| rbasak | I was having problems with /tmp being cleaned after I was already using it. | 13:28 |
| stgraber | rbasak: /proc/<pid>/root/... | 13:28 |
| stgraber | rbasak: note that lxc-attach should also be working now (if your kernel is >= 3.8) | 13:29 |
| rbasak | Yeah, hallyn pointed me to that, and I'll use it now. Though awkwardly I need to be root for that, and I was calling sudo lxc-* as a normal user, so I can't do it without calling sudo, which means that I can't reach /proc/<pid>/root programmatically. | 13:29 |
| rbasak | Using lxc-attach heavily, thanks :) | 13:29 |
| hallyn | stgraber: not sure about fs corruption... i wouldn't think so. But with overlayfs I'm actually not sure, it just may :) | 13:33 |
| rbasak | hallyn: AIUI, a bind mount will be fine, but perhaps trying to mount everything again won't. There should be no problem doing a bind mount though, right? | 13:34 |
| hallyn | bind mount of what? You actually can't cleanly bind mount from the /proc/$pid/root, unfortunately. | 13:41 |
| hallyn | rbasak: at this point you might be better off setting up a rshared directory and doing an lxc-attach mount into there... not sure if that would work or not | 13:42 |
| hallyn | well, at least you could set up the rshared dir before starting up the container, use /var/lib/lxc/$container/fstab to mount it into the container, lxc-attach into the container to bin-dmount /var/lib/cloud into it, and then watch there. that *should* work | 13:43 |
| hallyn | but not sure you're willing to do that :) | 13:43 |
| hallyn | (and if you are, i'd have to play a bit to see about making it work) | 13:43 |
| hallyn | biab | 13:45 |
| rbasak | hallyn: bind mount of whatever you're mounting with container.mount_rootfs(target)! | 13:46 |
| rbasak | Anyway, I have a path forward now, so I'll get on with that. Thanks very much for the help and the discussion. | 13:47 |
| meh3 | hey guys, i have a little issue with my ipv6 on my server, im still learning about iptables and so on, if someone check this out http://paste.ubuntu.com/5911219/ | 13:55 |
| meh3 | is this blocking ipv6 connections? | 13:55 |
| === hatch_ is now known as hatch | ||
| andol | meh3: ip6tables is the command you are looking for. | 13:57 |
| rbasak | meh3: probably not, though I'm not certain. iptables happens at layer 3 and only for IPv4, AFAIK. If you want to block IPv6 by MAC address, then as andol says you want ip6tables. | 13:59 |
| hallyn | rbasak: stgraber: mind you other people are doing things like that to watch for container 'boot finished' state. I don't recall offhand how they do it. | 14:06 |
| stgraber | hallyn: one solution I discussed on the mailing list a while back was bind-mounting a socket over /dev/lxc and then writing to that | 14:06 |
| hallyn | yeah if rbasak actually has a specific use case for it this might be a good time to work on whatever support we need to solve it generally | 14:07 |
| hallyn | i guess all we'd need for that is mknod + an added entry to /var/lib/lxc/$c/fstab | 14:07 |
| rbasak | For the boot-finished case, how about the lxc package provides a tool for that? Or something added to lxc-wait? I realise this is specific to containers running cloud-init, but I think that's a common enough case. | 14:08 |
| hallyn | we can't "just add" something to lxc-wait, we'd need conventions respected by distro userspaces | 14:09 |
| hallyn | to use stgraber's suggestion, | 14:10 |
| hallyn | the template could be asked to create /var/lib/lxc/$c/bootsock, and add an entry to mount that onto container's /dev/lxc, | 14:10 |
| hallyn | then user is responsible for having userspace write 'booted' to /dev/lxc when done | 14:10 |
| hallyn | if we go that far then i suppose we could hack lxc-wait to watch that file | 14:11 |
| hallyn | s@file@file/sock@ | 14:11 |
| rbasak | That sounds good | 14:16 |
| hallyn | rbasak: the /proc/$pid/root one, i'm not sure that's actually reasonable after all. But I"ll keep it open as we think about it. | 14:37 |
| rbasak | hallyn: I don't follow. What aspect isn't reasonable? As a means of accessing the rootfs of a container from the host? Its permissions? Or something else? | 15:08 |
| rbasak | hallyn: oh, just seen the bug. You mean the symlink proposal? | 15:09 |
| hallyn | yeah | 15:09 |
| hallyn | it's sort of institutionalizing a hack. | 15:10 |
| rbasak | I see. Fair enough. As long as we can access it somehow. | 15:10 |
| rbasak | The hack could be replaced without changing the interface in the future, perhaps? | 15:10 |
| hallyn | i just wanna let it sink in for a bit :) please do keep prodding me on it from time to time (both this and the /dev/lxc boot completion detection) | 15:10 |
| rbasak | It is really useful to be able to get to the container fs. Not just for boot-finished, but for vim/less and other tools as well. | 15:10 |
| rbasak | Will do - thanks. | 15:11 |
| rbasak | I appreciate that we want to think about it. No problem - it's not blocking me. | 15:11 |
| rbasak | Don't want to introduce an interface that we later regret and get stuck with it. | 15:11 |
| hallyn | especially now that we're approaching 1.0 and won't feel as free to abuse the users :) | 15:11 |
| hallyn | cool - tty | 15:12 |
| rbasak | BTW, whatever the solution is, I'd really like it to be available (mounted, symlink, whatever) by default, or have a tool to make it available using just the container name. That way tools that use lxc can make use of it and all the user has to do is provide an lxc container name to clone (or start-ephemeral) from. Without having to arrange it in a special way. | 15:13 |
| rbasak | (without having the *user* from having to arrange it in a special way) | 15:14 |
| theazman | Hey, anyone here familiar with amanda backup? I am trying to restrict the program from reaching hosts connected via wifi, haven't found a way to do it in the program. Is there a way to restrict that program from reaching hosts via wifi, while still allowing users connected via wifi to reach the server? | 15:21 |
| hallyn | zul: pushing a libvirt package based on 1.1.0 hourly tarball to ppa:serge-hallyn/libvirt-mav, fwiw. Was quite trivial, just some patch wrangling to do. Might save you 30 minutes on 1.1.0 merge (when that's released in 1-2 weeks) | 15:32 |
| hallyn | hm, i didn't add the apparmor fix for audit_write. | 15:33 |
| hallyn | anyway hopefully it fixes the memleak. | 15:33 |
| zul | hallyn: ok ill have a look monday (tearing down a house today) | 15:49 |
| hallyn | KABOOM | 16:01 |
| Siebjee | Hi all, i'm wondering where ubuntu is storing its old installation if you have re-installed ubuntu but stated that you didn't want a format while leaving the old data intact | 16:20 |
| jsonperl | PatrickDK: tried some more stuff that seemed to make sense… I set open file limits to 999999 since it seemed maybe hanging connections + current connections might use up the 1024 | 16:21 |
| patdk-wk | I didn't break it | 16:21 |
| jsonperl | then I changed these sysctl settings http://pastebin.com/4CgcDgT8 | 16:21 |
| jsonperl | Then I STILL saw the issue with pretty low connection count last night SAD FACE | 16:22 |
| jsonperl | SysRq show blocked state output… Am I correct that nothing is blocked here? http://pastebin.com/NJ44MKrh | 16:30 |
| pagec | ubuntu 12.04 trying to install smbldap, i download and ran the script smbldap-config.pl and i get this error: "Can't exec @PERL_CMD@ at ./smbldap-config.pl line 1." Perl is installed, anyone know what to do to fix it? | 16:43 |
| jsonperl_ | Does anybody know what the * (asterisk) line is in "ss -s" output | 17:22 |
| === jsonperl_ is now known as jsonperl | ||
| === Guest91027 is now known as DWSR | ||
| jsonperl | Like what is the 1600 line up top?? http://pastebin.com/4tbVzG9v | 17:38 |
| === andreas__ is now known as ahasenack | ||
| RoyK | jsonperl: not sure, but I guess it's whatever's not listed in the other categories | 17:42 |
| patdk-wk | dunno, for unix sockets, I get 138 | 17:42 |
| patdk-wk | for ALL I get 223 | 17:42 |
| jsonperl | It's strage, it's always 0 for me except in my "I'm having a problem" snapshots | 17:42 |
| patdk-wk | but for * it lists 285 | 17:42 |
| jsonperl | hmm | 17:42 |
| patdk-wk | Total: 221 (kernel 285) | 17:43 |
| patdk-wk | that first line is more important than * | 17:43 |
| jsonperl | Yep, which is low(ish)… It just seems like a clue | 17:43 |
| jsonperl | since it's really high on problem servers | 17:44 |
| patdk-wk | this is from an ftp server | 17:44 |
| jsonperl | Gotca | 17:44 |
| patdk-wk | web server: Total: 218 (kernel 233) | 17:44 |
| patdk-wk | syslog server: Total: 289 (kernel 395) | 17:45 |
| sarnold | patdk-wk: he's got a fairly popular game server running | 17:45 |
| patdk-wk | I know | 17:45 |
| sarnold | okay :) hehe | 17:45 |
| patdk-wk | so should be likely kindof like a webserver | 17:45 |
| jsonperl | wow kernel 1600 in that paste | 17:45 |
| jsonperl | It is and isn't | 17:45 |
| jsonperl | connections are persistent | 17:45 |
| patdk-wk | well, long keepalive :) | 17:46 |
| jsonperl | No reverse proxy :( | 17:46 |
| patdk-wk | no idea how you got so many kernel sockets | 17:46 |
| patdk-wk | that sounds like an issue | 17:46 |
| jsonperl | It does | 17:46 |
| jsonperl | :D | 17:46 |
| patdk-wk | or maybe that is just all your threads | 17:46 |
| jsonperl | 294 threads from the games servers total… no more no less | 17:47 |
| patdk-wk | I don't have anything that does a lot of threads, other than apache | 17:47 |
| patdk-wk | and even there, it's not insane | 17:47 |
| jsonperl | Main thread, + 20 workers | 17:47 |
| jsonperl | 14 servers | 17:47 |
| patdk-wk | hmm | 17:47 |
| jsonperl | This looks suspicious though right?? | 17:48 |
| jsonperl | Now i wanna dig into the source of ss… i doubt i'd understand it | 17:49 |
| patdk-wk | kernel is slabstat.socks | 17:55 |
| patdk-wk | Total: 1141 (kernel 1265) (my desktop machine) | 17:57 |
| jsonperl | Hmm… | 17:57 |
| jsonperl | oh yea: printf("* %-9d %-9s %-9s\n", slabstat.socks, "-", "-"); | 17:57 |
| jsonperl | what the crap is slabstat | 17:57 |
| patdk-wk | using nfs? | 17:57 |
| jsonperl | negative | 17:57 |
| patdk-wk | slab is kernel memory allocator | 17:57 |
| patdk-wk | not sure what socks are for slab | 17:58 |
| sarnold | the kernel memory allocator tries to know the exact sizes of kernel memory objects, so it can keep tightly-packed ranges of memory available for use for those specific objects again | 17:59 |
| === Ursinha-afk is now known as Ursinha | ||
| jsonperl | btw, using jemalloc now | 18:00 |
| patdk-wk | that doesn't affect kernel | 18:00 |
| sarnold | when a new memory object is required, the kernel can re-use old objects that are the right size and perhaps even partially constructed already.. | 18:00 |
| patdk-wk | dump an output of /proc/slabinfo | 18:01 |
| patdk-wk | a broken one would be interesting | 18:01 |
| jsonperl | Servers are fine now | 18:01 |
| jsonperl | k, just cat the whole thing to a file? | 18:01 |
| sarnold | slabtop output might be more readable | 18:01 |
| patdk-wk | ya | 18:01 |
| patdk-wk | :) | 18:01 |
| jsonperl | ya slabtop? | 18:01 |
| jsonperl | aiight, added to my oh shit script | 18:02 |
| patdk-wk | not slabtop I hope | 18:03 |
| jsonperl | nope | 18:03 |
| patdk-wk | it's not very scriptable :) | 18:03 |
| jsonperl | that puppy looks interactive | 18:03 |
| patdk-wk | ya | 18:03 |
| patdk-wk | not sure I have ever had a slab issue | 18:04 |
| patdk-wk | I know I have had issues before the kernel added slab into it | 18:04 |
| jsonperl | I'm still teetering on upgrading to 3.8 kernel | 18:04 |
| jsonperl | i feel like that's kinda last ditch | 18:05 |
| sarnold | heh, pre-slab is -ancient- :) | 18:05 |
| jsonperl | this feels like a networking issues (settings or something) | 18:05 |
| patdk-wk | yes, my active kernel hacking was ancient | 18:05 |
| patdk-wk | I was big into hacking on 2.0 | 18:05 |
| patdk-wk | alittle less on 2.2 | 18:05 |
| patdk-wk | and pretty much died on 2.4 | 18:06 |
| sarnold | man things were easier in 2.0 :) | 18:07 |
| patdk-wk | I loved using that qnx scheduler back then | 18:07 |
| patdk-wk | still wish I could use it today | 18:07 |
| patdk-wk | not motivated enough to hack it in though | 18:08 |
| jsonperl | Patrick did you look at those sysctl settings i changed? | 18:08 |
| patdk-wk | yep | 18:08 |
| jsonperl | Seem aiight? | 18:08 |
| jsonperl | I feel like we might be hitting the open file limit still… the numbers just look about right | 18:09 |
| patdk-wk | did you add ulimit to the script startup? | 18:10 |
| jsonperl | added soft nofile 999999, hard nofile 999999 to a conf file in limits.d | 18:11 |
| patdk-wk | oh, I never touch that | 18:11 |
| jsonperl | should accomplish the same goal right? | 18:11 |
| patdk-wk | dunno, I don't know enough about limits.d | 18:12 |
| patdk-wk | I don't even have a limits.d | 18:12 |
| jsonperl | ulimit -a => open files (-n) 999999 | 18:12 |
| jsonperl | it's in security | 18:12 |
| sarnold | The limits.d stuff will only be applied if pam_limits is somewhere in the PAM stack used to start those processes... | 18:13 |
| jsonperl | So maybe that's the issue… it's still seeming to hit it | 18:13 |
| patdk-wk | using ubuntu startup scripts, I'll just add ulimit into /etc/default/x | 18:13 |
| patdk-wk | if you own startupscript, just add it right before you launch your app | 18:14 |
| sarnold | jsonperl: check lsof or fuser output, I strongly doubt you're hitting nearly-a-million in a single process... | 18:14 |
| patdk-wk | sarnold, he means hitting the default of 10254 | 18:14 |
| patdk-wk | 1024 | 18:14 |
| jsonperl | Not a milion… but certainly the default | 18:14 |
| sarnold | patdk-wk: ah, okay | 18:14 |
| sarnold | note upstart has nice limit stuff built-in too, no need to do the shell approach: http://upstart.ubuntu.com/cookbook/#limit | 18:15 |
| jsonperl | hmm i'll do that | 18:15 |
| jsonperl | i LOVE upstart btw | 18:15 |
| jsonperl | so awesome | 18:15 |
| sarnold | once I found the .override files, my opinion of upstart improved drastically :) | 18:15 |
| jsonperl | I tried to install it on debian a while back… what a fiasco | 18:15 |
| patdk-wk | oh? .override? | 18:15 |
| sarnold | it should be better now | 18:15 |
| patdk-wk | in my upstart, I added ulimit to pre-start | 18:16 |
| sarnold | patdk-wk: an easy way to keep tasks from starting, or changing their start conditions.. http://upstart.ubuntu.com/cookbook/#override-files | 18:16 |
| sarnold | way easier than managing the huge pile of sysv-init symlinks :) | 18:16 |
| patdk-wk | no fun, I liked, mv /etc/init/x /etc/init/.disabled/x | 18:17 |
| jsonperl | so i run the game as the deepworld user | 18:17 |
| jsonperl | if i "sudo su - deepworld -c 'ulimit -a'" and it report 999999, shouldn't that mean I'm good | 18:18 |
| sarnold | IF whatever mechanism you use to start the deepworld applications also runs through the PAM stack, and the pam.d/whatever file in question calls on pam_limits, yes | 18:19 |
| * patdk-wk notes sudo uses the pam stack | 18:19 | |
| jsonperl | Hmm, ok. I gotta go read about the PAM stack | 18:20 |
| patdk-wk | or well, su does | 18:20 |
| patdk-wk | easier test | 18:20 |
| patdk-wk | add ulimit -a to your startup script :) | 18:20 |
| jsonperl | haha | 18:20 |
| patdk-wk | and see what it says | 18:20 |
| jsonperl | ok good idea | 18:20 |
| sarnold | .. but upstart may not. start-stop-daemon or whatever you use to start the program from initscripts may not. cron will, but it may not include pam_limits ... | 18:20 |
| patdk-wk | guess when the packages upgrade, I'll roll over to .override files | 18:30 |
| jsonperl | KICK ASS: open files (-n) 1024 | 18:35 |
| jsonperl | Good idea patdk (about adding ulimit -a to startup script) | 18:35 |
| patdk-wk | there are bug reports about limit being broken in upstart for 12.04, can't tell if it was fixed for 12.04 or what | 18:42 |
| jsonperl | one way to find out :) | 18:43 |
| patdk-wk | hmm, next dovecot/postfix releases, I'll have to rework my init scripts some | 18:46 |
| jsonperl | didn't even run | 18:46 |
| jsonperl | init: Failed to spawn deepworld-game-5000 main process: unable to set "nofile" resource limit: Operation not permitted | 18:46 |
| jsonperl | ah no perms | 18:46 |
| patdk-wk | if this fixes it, you know what the real issue is? :) | 18:48 |
| jsonperl | Too many hanging connections | 18:50 |
| jsonperl | A noob trying to run an MMO? | 18:50 |
| patdk-wk | na | 18:50 |
| patdk-wk | programmers ignoring error codes when calling functions | 18:50 |
| jsonperl | How so | 18:51 |
| sarnold | argh bane of my exisitence | 18:51 |
| patdk-wk | it would be helpful if they loged the error when attempting to open a file, and failed | 18:51 |
| patdk-wk | your log would say, UNABLE TO OPEN FILE XXXX: .... | 18:51 |
| jsonperl | To be fair, i've had a hard time parsing my logs… we jam way too much garbage into syslog | 18:51 |
| jsonperl | A problem I aim to fix shortly | 18:52 |
| patdk-wk | I have a syslog server | 18:52 |
| patdk-wk | it collects all the logs from everything and shoves them into mysql | 18:52 |
| jsonperl | We do that to loggly | 18:52 |
| jsonperl | That would be nice to have our own though | 18:52 |
| patdk-wk | then I just have different things trigger email alerts, or browse the logs via webpage | 18:52 |
| jsonperl | Do you use any packages for that? | 18:53 |
| patdk-wk | well, a long long time ago :) | 18:53 |
| patdk-wk | there was php-syslog-ng | 18:53 |
| patdk-wk | now it went commercial to be named logzilla | 18:53 |
| jsonperl | upstart script run as root right? | 18:53 |
| jsonperl | by default | 18:53 |
| patdk-wk | I have been updating and maintaining it myself for a long time now | 18:53 |
| jsonperl | Gotcha | 18:53 |
| patdk-wk | should be yes | 18:54 |
| jsonperl | Publish it :) | 18:54 |
| patdk-wk | while I have no issue making my customizations public | 18:54 |
| patdk-wk | it has no *installer* | 18:54 |
| patdk-wk | so isn't much fun to setup | 18:54 |
| jsonperl | Wonder why it would have issues setting ulimits | 18:54 |
| patdk-wk | but as I don't really set it up :) | 18:54 |
| jsonperl | Ha, true | 18:54 |
| patdk-wk | setup does take some work | 18:54 |
| jsonperl | unable to set "nofile" resource limit: Operation not permitted, is that the bug you were talking about? | 18:54 |
| patdk-wk | don't think so | 18:55 |
| patdk-wk | I couldn't track down the bug specifically | 18:55 |
| patdk-wk | just saw people talking about it | 18:55 |
| patdk-wk | and it was reported | 18:55 |
| jsonperl | There's a couple nice services on the market for centralized syslogs… loggly is <i>pretty</i> good, as is papertrail | 18:55 |
| patdk-wk | but my search kept failing to locate it | 18:55 |
| jsonperl | With the amount of moving parts I'm managing nowadays, I'm happy to let other people do the lifting on stuff like that | 18:55 |
| patdk-wk | ya | 18:56 |
| patdk-wk | I'm doing a few gigs of logs a day | 18:56 |
| jsonperl | yep, same | 18:56 |
| jsonperl | well maybe 2 | 18:56 |
| jsonperl | jeez thats a lot of logs | 18:56 |
| patdk-wk | mine is all email traffic | 18:56 |
| jsonperl | wow | 18:56 |
| sarnold | wow :) | 18:57 |
| patdk-wk | ya, there wasn't too many solutions back in 2005 :) | 18:57 |
| patdk-wk | and it needed to be fast, for the day | 18:57 |
| patdk-wk | today, it's not hard to log that much | 18:57 |
| sarnold | logging it isn't hard | 18:58 |
| sarnold | doing something intelligent with the logs -is- hard :) | 18:58 |
| patdk-wk | well, | 18:58 |
| patdk-wk | logging it in a way, that was more useful to use than *grep* | 18:58 |
| jsonperl | It'd be nice to throw it at elasticsearch or some REALLY fast full text engine | 18:58 |
| patdk-wk | and for back then, attempting not to overflow diskspace | 18:58 |
| patdk-wk | well, the commerical one, supports sphinx | 18:59 |
| patdk-wk | I haven't added sphinx support in yet | 18:59 |
| jsonperl | yea, sphinx is good enough i spose | 18:59 |
| jsonperl | near realtime fulltext search on logs, that'd be cool | 18:59 |
| jsonperl | https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669 | 19:03 |
| uvirtbot | Launchpad bug 938669 in upstart "upstart jobs do not respect /etc/security/limits.conf" [Undecided,Invalid] | 19:03 |
| jsonperl | I'll drop the limits in cups.conf | 19:07 |
| sarnold | jsonperl: oof, yeah, I can see how that'd be confusing. but what that bug report 'wants' is fundementally not how things work :) hehe | 19:12 |
| jsonperl | yep, makes sense… just thought it interesting | 19:12 |
| === vmdsch is now known as Guest55222 | ||
| === Ursinha is now known as Ursinha-afk | ||
| jsonperl | patrickdk/sarnold: i just slapped ulimit -n 999999 in the upstart file | 19:43 |
| jsonperl | works like a chaaam, now we wait | 19:43 |
| sarnold | jsonperl: how very odd. I'd have xpected the limit command to Just Work.. | 19:44 |
| patdk-wk | :) | 19:44 |
| jsonperl | lolz | 19:44 |
| jsonperl | you forgot the TM | 19:44 |
| sarnold | hehe | 19:44 |
| jsonperl | upstart has its rough spots | 19:44 |
| jsonperl | but i like it a lot mostly | 19:45 |
| jsonperl | the devil you know... | 19:45 |
| patdk-wk | ya, upstart got a lot better for me when I started adjusting all the start/stop on commands | 19:45 |
| patdk-wk | postfix depends on dovecot being started (can't use lmtp/auth without it) | 19:46 |
| jsonperl | yep, the whole chained startup thing is great | 19:46 |
| jsonperl | Our architecture relies on it a lot | 19:46 |
| patdk-wk | oh wait, that machine is still 10.04 | 19:46 |
| patdk-wk | it probably doesn't have any limit support, let alone broken suppport :) | 19:46 |
| sarnold | haha | 19:47 |
| patdk-wk | all other machines have been upgraded | 19:47 |
| patdk-wk | but highly used mailservers are always last and most scary | 19:47 |
| sarnold | *nod* | 19:47 |
| patdk-wk | not scary cause it will break, just people get pissy it's down | 19:47 |
| jsonperl | sarnold: bout the "Just Working" I was getting permission issues, which didn't make sense, I punted | 19:47 |
| sarnold | jsonperl: were those user jobs? or system jobs? | 19:47 |
| sarnold | (with the knowledge that I might be butchering the terminology) | 19:48 |
| jsonperl | system, meaning run as root? | 19:48 |
| sarnold | I think so | 19:48 |
| jsonperl | then yep | 19:48 |
| sarnold | or at least started as root initially | 19:48 |
| jsonperl | I actualy sudo to the user in the script to run | 19:48 |
| jsonperl | su rather | 19:48 |
| jsonperl | the run as user stuff in upstart always seems to be more hassle than it's worth | 19:49 |
| sarnold | aha, did you uncomment the limits stuff in /etc/pam.d/su ? that might have been another acceptable approach | 19:49 |
| patdk-wk | still feel that is ugly way to do it | 19:50 |
| patdk-wk | most likely to get forgetten about | 19:50 |
| jsonperl | it's less files to touch for me for sure | 19:50 |
| sarnold | yeah | 19:50 |
| jsonperl | but na, i'm just doing it in the server scripts i think | 19:50 |
| jsonperl | it works | 19:50 |
| jsonperl | now to update a bajillion servers :/ | 19:51 |
| patdk-wk | I really don't touch that stuff, unless I'm making a shell server | 19:51 |
| === Rallias is now known as Gasseus | ||
| === Ursinha-afk is now known as Ursinha | ||
| === roasted_ is now known as roasted | ||
| sidnei | hallyn: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1205086 i think this might be up your alley | 20:04 |
| uvirtbot | Launchpad bug 1205086 in lxc "lxc-net dnsmasq --strict-order breaks dns for lxc non-recursive nameserver" [Undecided,New] | 20:04 |
| hallyn | or over my head :) | 20:11 |
| hallyn | stgraber: ^ is the strict-order there (for lxc dnsmasq0 for a reason? | 20:13 |
| === Gasseus is now known as Rallias | ||
| hallyn | sidnei: would it be possible for you to post some sample configs? | 20:33 |
| sidnei | hallyn: ok, let me fish out some snippets | 20:37 |
| hallyn | sidnei: thanks. | 20:37 |
| sidnei | hallyn: there, hope it helps | 20:43 |
| streulma | it's quiet here... | 21:29 |
| jsonperl | tis | 21:32 |
| RobHaz | Hello | 21:37 |
| RobHaz | What kinds of srvers can i have? | 21:37 |
| RobHaz | Im having now, ssh, and samba + webserver | 21:38 |
| RobHaz | what more can i have? | 21:38 |
| streulma | RobHaz: mailserver ;) | 21:39 |
| RobHaz | streulma: Is there a doc how to set ip up? | 21:40 |
| streulma | for Ubuntu Server 12.04 ? yes | 21:40 |
| streulma | RobHaz:https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204-postfix-dovecot-mysql/ | 21:41 |
| streulma | RobHaz: https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204-postfix-dovecot-mysql/ | 21:41 |
| === FUF_ is now known as FUF | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!