zzakmy issue was resolved on #postfix, thank you00:19
=== airtonix_ is now known as airtonix
allaireHi, anybody has some monit experience?02:15
gld1982ltdhi all. i need help getting sshfs working on boot. fstab entries are not working.02:16
=== arrrghhh is now known as arrrghhhAWAY
atpa8atrying to figure out the cloud thingie :P any help here?03:18
atpa8aright now i'm running a kvm on one server and a few virtual machines03:19
atpa8acan i use MAAS to do the same?03:24
atpa8aon one server that is03:24
atpa8aor is that an overkill?03:25
ketan985Hello Friends, I want to control server in such way that it kill process when it consumes more memory05:30
=== thumper is now known as thumper-afk
rbasakketan985: try looking into "ulimit" and "oom killer".07:06
=== rvba` is now known as rvba
=== smb` is now known as smb
ztuiophello there07:38
ztuiophello anyone there ?07:39
ztuiop /join ubuntu-gb07:40
zertyuianyone there ?08:29
zertyuiis it pĂ´ssible to create a root user with temporary access ?08:30
melmothzertyui, you can add any user on the sudo list temporary.08:34
melmothbut .... how can you be sure this user will not change things while he is root so he can be root again later on ?08:35
zertyuiyes if he would liek08:37
zertyuibut i m looking for a command creating users with timestamp access08:38
zertyuiso impossible ?08:45
maxbroot, by definition, can do anything. Including adding backdoors to regain root later08:49
zertyuiyes of course i m an idiot do not understand that08:54
zertyuimy question was how to actroy root access to a user with timestamp access ?08:55
zertyuiyou are logged in root on a system  i would like to a create a user08:56
zertyuiand give him a root08:56
zertyuijust for example for 2 days08:56
melmothadd him in the sudeor file, and remove him after 2 days.08:56
melmothif you wanna have this automatically done, write a script that does it for you. and use cron to launch it08:57
zertyuithen the user autamatically back to an normal user i would do that automatically08:57
zertyuiwithout doing it manually08:57
ttxDaviey: "agile granite foundations", wow :)08:59
rbasakYou can set user accounts to expire. See usermod(8).09:05
rbasaksudoers(5) expiry doesn't exist, AFAIK, but you can cron it.09:05
psivaajdstrand: Would like to know if there is any update on bug #1197484. ETA for any possible fix, etc?09:17
uvirtbotLaunchpad bug 1197484 in isc-dhcp "Connection requests to saucy server VMs from a hosts fail after fresh VM installs" [High,New] https://launchpad.net/bugs/119748409:17
=== Guest46086 is now known as jpds
jamespageyolanda, that nagios3 merge is confusing - I'm not entirely sure why the debdiffs's are so huge10:13
yolandalet me double check10:14
yolandanot sure now10:14
yolandajamespage, you mean the diffs between the two ubuntu versions?10:18
jamespageyolanda, yes - the debian commits are patches and packaging10:19
yolandaI'll recheck the process10:21
jamespageyolanda, even the diff between -3 and -4  in debian is massive10:21
jamespageyolanda, I don't think its what you have done10:21
jamespagealthough I did not expect to see the diff in debian/po10:21
yolandalet me paste the report10:21
yolandamaybe you see something10:21
jamespageyolanda, yeah - I know10:23
jamespagebut think about what change you actually made for 3ubuntu110:23
jamespagethat should be the only delta10:23
yolandalet me recheck taht10:24
jamespageyolanda, oh - you should also close out the merge bug as part of your changelog10:24
jamespageI was about to add that to the bug report but got sidetracked by this issue10:25
yolandaok, didn't know about it, so i should reference the LP bug in changelog?10:26
jamespageyolanda, yes10:28
jamespageyolanda, I think the po mods are a grab-merge bug10:28
jamespageyolanda, if I do the merge using ubuntu:nagios3 and lp:debian/sid/nagios3 I get what I would expect10:28
yolandaok, so i'll try with that approach10:29
yolandajamespage, is that better to rely on manual merges, not in grab_merge script?10:30
yolandaor maybe do the grab_merge and then check that for unexpected results?10:30
jamespageyolanda, yeah - thats what I end up doing10:31
yolandaok, then i'll fix that, and i also have to update the changelog for the others10:31
jamespageyolanda, great - thanks!10:33
yolandajamespage, much more cleaner debdiff doing a manual merge10:51
yolandai'll resend the patches10:52
yolandajamespage, generated diff between prev ubuntu version and this one is huge anyway, mostly same size, but diff between debian/ubuntu is clean now10:59
jamespageyolanda, when you attach patches please can you make sure that you tick the 'this is a patch to fix the problem' option11:27
jamespageit breaks the sponsorship tooling otherwise11:27
jamespageyolanda, for the nagios merge the bug in the changelog does not match the one in launchpad11:28
jamespageyolanda, fwiw you can just push the branch to launchpad/raise a MP instead of doing the debdiff's11:31
yolandajamespage, i remember don't having the permissions, i think11:34
jamespageyolanda, to mark patches as 'patches'?11:34
yolandano, for the MP11:35
jamespageyolanda, anyone can raise a merge proposal11:35
yolandaok, and you do the merge? i can't remember who recommended me to use the debdiff approach11:35
yolandanp, having lunch and i'll raise the mp11:36
jamespageyolanda, OK - for the quid3 merge:11:39
jamespagedpkg-source: info: local changes detected, the modified files are:11:39
jamespage squid3-3.3.4/src/cf.data.pre11:39
jamespagewhen I try to use the debdiff - can you take a look at that as well11:39
yolandajamespage: https://code.launchpad.net/~yolanda.robla/ubuntu/saucy/nagios3/debian_merge/+merge/17473611:57
jamespageyolanda, lots of conflicts12:00
yolandai know12:00
yolandalots of conflicts between prev and this ubuntu veresion12:00
yolandabut it's like that using the grab-merge and the manual merging also12:00
yolandai'll try resubmitting the mp, just a moment12:05
jamespageyolanda, nm - I already uploaded that one12:06
yolandais that ok?12:07
jamespageyolanda, I just fixed up the debdiff you uploaded a while back12:07
yolandaoh ok12:07
yolandai'll check the squid3 problem, not sure what happens12:07
zuljamespage:  i really need to beat out neutron today12:12
jamespagezul, beat away!12:14
jamespagemorning btw12:14
zulgood morning12:15
jamespageyolanda, https://jenkins.qa.ubuntu.com/job/saucy-adt-nagios3/12:17
yolandalet me try locally12:19
hxmi want to emulate a NAS server in my ubuntu server, is that possible?12:29
hxmexists any kind of software for do this?12:29
hxmwhat could be the query string for google, nas software gives too many results12:29
PiciWhat exactly are you trying to acheive?12:30
hxmcreate folders, rename files, that basic things12:30
hxmi guess a NAS is a webdav server, no?12:30
PiciSo just a file server? Pick a protocol.12:30
PiciNFS, SMB, whatever.12:31
hxmyes, i use smb, but some users asked for have a HTTP interface with admin control12:31
hxmi just wonder if that exists12:31
hxmor the user just should login as admin when map the server12:32
atpa8ai would think you need some CMS type of software12:40
=== cmagina-away is now known as cmagina
RoyKhxm: what do you need to administer over web?13:18
rbasakhxm: look into freenas. It's BSD based, but perhaps you can run it inside a VM on Ubuntu Server?13:19
RoyKfreenas is good, runs on zfs too IIRC, which is good indeed13:19
jdstrandpsivaa: still trying to reproduce13:22
psivaajdstrand: ok, i had an impression that you've seen it somewhere else as well, may be i misunderstood13:23
jdstrandpsivaa: I did, but I've yet to reproduce it13:24
psivaajdstrand: ok, understand13:24
ChocoboHmmm, any idea why my NFS mount would lock up periodically (to the point of needing a reboot)?13:36
RoyKChocobo: not sure, it could be anything. anything in the logs? dmesg?13:40
yolandajamespage, i tested nagios3 tests again locally, and run fine for me13:43
jamespageyolanda, bah13:43
yolandasome dependency with nagios3-cgi should be the problem? why it works locally with run-adt-test, and not on the test machine?13:45
atpa8acan MAAS be a good substitute for managing virtual machines?14:48
ChocoboRoyK: Not really... I can still mount other NFS exports on the same interfaces, but it just hangs when I try to mount a certain export.   It is strange because other nodes in the cluster all have the problematic export mounted14:51
ChocoboWhen I try to mount it there is tons of traffic (using tcpdump)  this is strange14:59
RoyKChocobo: same server as well?15:00
ChocoboRoyK: What do you mean same server?  yes, I can mount other exports from the same server.15:01
RoyKChocobo: if you mount with options soft,intr, than the connection should be interruptable15:01
RoyKotherwise, the default action for NFS is to hang while the server's unavailable15:02
yolandajamespage, about squid3, i'm finding an strange problem with the patches. I removed all .pc directory, retried again, applying all patches manually, etc...15:06
yolandawhen i do a bzr bd -S i have this error :bzr: ERROR: An error (1) occurred running quilt: The working tree was created by an older version of quilt. Please run 'quilt upgrade'.15:06
ChocoboRoyK: this is my fstab entry:  dedup-ib:/big_pool/os-grizzly /os-grizzly nfs rw,async,noatime,nolock,tcp,bg,intr,hard,_netdev,noauto 0 015:06
yolandarunnning quilt upgrade doesn't help, it complains about that the quilt metadata is already in version 2, nothing to do15:07
yolandapackaging with a debuild works, but not sure it that is ok15:07
RoyKChocobo: perhaps try soft instead of hard15:10
RoyKChocobo: it won't fix the issues, but may make it easier to debug15:10
RoyKbtw, I don't think noatime is a valid nfs flag15:10
ChocoboRoyK: Thanks, I will give it a shot.15:11
RoyKChocobo: btw, is this some dedup thing?15:14
ChocoboRoyK: it is a ZFS backend that has deduplication enabled, yes.15:15
RoyKlots of memory in the machine?15:19
RoyKin my experience, zfs dedup is *very* hungry for memory15:19
bitnumushey, my system clock keeps drifting15:22
bitnumuswhats the best solution to fix this15:22
bitnumusdoesnt ubuntu have a default cron to handle this ?15:22
RoyKbitnumus: ntp should keep your clock in sync15:23
RoyKbitnumus: is this a vm?15:23
bitnumusnot sure how the provider has it setup, its a VPS15:23
RoyKthen it's probably a vm15:24
RoyKcan you pastebin lshw output?15:24
bitnumuslshw ?15:24
patdk-wkwhy would ubuntu have a cron to handle clock? that is the worst idea ever15:24
bitnumuspatdk-wk, just what i've read15:24
RoyKbitnumus: apt-get install ntp15:25
bitnumusntp is installed15:25
bitnumusmaybe not running but15:25
bitnumusi looked at this a few days ago now, something about ntpdate15:25
RoyKbitnumus: yes, lshw, it should show on what hardware or hypervisor you're running15:25
bitnumusRoyK, that gives 'bad command'15:26
RoyKthen apt-get install it :)15:26
RoyKbitnumus: perhaps dmidecode will tell15:26
rbasakSome VPSes don't let you set the clock.15:26
RoyKbut lshw output is better15:27
rbasakI had one where the clock was out, the kernel wasn't available to user modification, and setting the clock resulted in an error. I had to get the hosting provider to fix it.15:27
bitnumusRoyK,  http://pastebin.com/z9aFjKGm15:28
RoyKrbasak: openvz or vserver based systems don't have individual clocks15:28
bitnumusso i've installed ntp, anything i need to do to initialise it ?15:29
RoyKbitnumus: not sure, but I guess vserver15:29
bitnumusdoes it need a reboot15:29
RoyKbitnumus: to manually set the time from a timeserver, use ntpdate pool.ntp.org15:29
bitnumusi dont want to manually do anything, i need it to keep up to date with next to 0 drift15:30
RoyKyou might need to stop ntp first because of an open socket15:30
bitnumuswill ntpd keep it in check ?15:30
bitnumusso no reboot or anything ?15:30
bitnumushow often should it update it ?15:30
RoyKbut if the clock is too far askew, ntpd might not catch up15:31
atpa8aor it will take some time to catch up15:31
RoyKso, service ntp stop ; ntpdate pool.ntp.org ; service ntp start15:31
bitnumusna, atm its about 1second out15:31
rbasakntpd adjusts the clock speed to match the time it's syncing. So it's not updated as such. Once  the clock stays in sync it should just appear to be in sync.15:31
bitnumusmaybe that was my issue before, it drifted to 264seconds15:31
RoyKthen it souldn't be needed to use ntpdate15:31
bitnumusok great stuff15:32
streulmawhat is the best way to update your server? apt-get upgrade, or apt-get upgrade --show-upgraded, or apt-get dist-upgrade, or aptitude dist-upgrade ?15:32
bitnumuscheers ^15:32
patdk-wkstreulma, depends on the goal15:32
RoyKstreulma: I just do apt-get update && apt-get -y dist-upgrade && apt-get -y autoremove15:32
patdk-wkupgrade everything, upgrade security patches only, ...15:32
patdk-wkdist-upgrade is what I use, and you need it to bring in new kernel security patches15:33
streulmaused command from RoyK15:34
patdk-wkroyk, maybe use virt-what next time, over lshw?15:35
RoyKpatdk-wk: virt-what?15:35
RoyKdidn't know that ;)15:35
patdk-wkI knew it existed, but couldn't remember the name15:36
patdk-wkand didn't know if it did openvz and them, but it does15:36
RoyKstreulma: can you try virt-what as patdk-wk suggested?15:36
patdk-wkbitnumus, you mean?15:36
RoyKuh, yes15:36
RoyKbitnumus: ?15:37
patdk-wkya, what is the word :)15:37
RoyKbitnumus: can you try virt-what as patdk-wk suggested?15:37
ChocoboRoyK: There is 512GB in that machine I believe.15:37
RoyKChocobo: should suffice for rather a large amount of diskspace ;)15:37
patdk-wkhate to see that reboot15:38
RoyKI've seen clock drift with xen15:38
patdk-wkyour ok running ntp on xen15:38
patdk-wkit won't keep the clock perfect though15:39
patdk-wkbut it will keep it close15:39
streulmaI've seen some time wibble on Xzn15:40
=== whaley is now known as aTribeCalled
weeb1eHi everyone16:20
weeb1eI just got access to two new sponsored servers running the latest ubuntu16:20
weeb1eI haven't used the past few ubuntu versions so some things have changed16:20
weeb1eFirst up, is there something special that needs to be done now to change my sudo password?16:21
weeb1epasswd does not seem to be working16:21
hachreWhat do you mean by 'sudo password'16:22
hachreif your user is in the etc sudoers file it can access root via it's own user password16:22
hachreif you wanna change the root password you can do 'sudo passwd'16:23
weeb1eI've tried about 4 times now, I use passwd, enter the current password, then enter my new password twice, it says it is changed, yet if I open a new SSH session only the old password works16:23
hachreare you changing the password for the user you also try to ssh in with?16:23
weeb1eSo I mean my own account password which is the only account on the box16:23
hachrenothing in that regard has changed, is this a brand new installation?16:24
weeb1eMy next question would be how to check if a root password is set at all, since I know ubuntu does not set one by default, so I would only want to change that if one is already set16:24
weeb1eBut I first need to figure out why my own password is not changing16:24
weeb1ehachre: Yes, I was told it was installed today16:24
hachreit's weird, passwd should go through the /etc/pam.d/system-auth component16:25
weeb1eI see now it is not the latest ubuntu even, they installed Ubuntu 12.04.2 LTS16:25
hachreI think16:25
hachreah yea16:25
weeb1eI hope 12.04 does not cause me grief16:25
hachrethats the latest LTS release16:25
rbasakweeb1e: "sudo getent shadow root" to see the root password hash. If it's "!" or "*" or something, then there's no root password set. That's generally the same across all distros.16:26
weeb1ethanks rbasak, any idea why passwd wouldn't be taking effect for me?16:26
weeb1eThere is a hash there so I assume a root password is set16:26
rbasakweeb1e: "passwd" sets the password you use to sudo with (your own user password). What you ssh in as should be the same password. I don't know of any reason that wouldn't work unless your provider is doing something? Is it a fresh install on real hardware, or some kind of VM?>16:26
weeb1erbasak: I know that, passwd says it works for my user but then my users password is not changed in any new SSH sessions16:27
weeb1eIt is a fresh install on hardware, I had to wait a few days for them to remove the VM and install an OS directly16:27
weeb1eVMs are useless for realtime software which requires minimal overhead and max performance16:28
rbasakweeb1e: how about an ssh user@localhost from the machine itself?16:28
RoyKweeb1e: were you running "passwd" as your own user, or as root?16:29
weeb1erbasak: Still only the old password works16:29
weeb1eRoyK: My own user16:29
rbasakweeb1e: "sudo getent shadow youruser" before and after changing the password. Does that get updated?16:29
RoyKcheck the file date of /etc/shadow16:29
=== aTribeCalled is now known as whaley
weeb1erbasak: Yes it changes16:32
weeb1eThis is very weird16:32
RoyKweeb1e: and then if you try to ssh youruser@localhost?16:32
weeb1eI am getting very confused, I have worked with plenty Ubuntu servers in the past and have never had an issue like this16:32
rbasakweeb1e: "grep password /etc/pam.d/sshd" - does that say "@include common-password" or something else?16:32
rbasakweeb1e: it certainly shouldn't do that on a default install.16:33
weeb1eRoyK: Hmm, that worked now from that same session16:33
RoyKweeb1e: but not from another machine?16:33
weeb1eAnd now it works in a new session16:33
weeb1eWhy the hell would it suddenly work on the 6th attempt :|16:33
RoyKok, possibly PEBKAC ;)16:33
weeb1eOh wait, I changed the root password16:34
weeb1eSo the only explaination is it is using the root password for my own account?16:34
weeb1eWhy would it be doing that16:34
weeb1erbasak: Yes, that is included16:34
weeb1eI guess the techie that installed these boxes did something odd16:35
rbasakweeb1e: the behaviour you're describing is certainly non-standard non-default.16:35
RoyKnormally, on ubuntu, root doesn't have a password. it means you can boot to single if you have physical access without a password, but then, if you have physical access, you can normally override most security16:35
RoyKrbasak: seems to me he just ran passwd as root, nothing more16:36
rbasakweeb1e: it might be worth comparing /etc/ssh/sshd_config and /etc/pam.d/* against a default system.16:36
weeb1eIf I had to remove the root password now, would my user still work with its own password?16:36
weeb1eI just don't want to lock myself out16:36
rbasakweeb1e: RoyK: yeah, perhaps I've misunderstood the details.16:36
weeb1eHow would I remove the root password?16:36
RoyKweeb1e: no need, really16:37
RoyKweeb1e: your system is only slightly more secure with a root password16:37
rbasakweeb1e: leave an ssh session running "sudo -i" so you have a root prompt. Change and test at will. If you leave the session open you can recover from problems using that.16:37
RoyKweeb1e: you may want to turn off root login in /etc/ssh/sshd_config, though16:38
weeb1eWell, ok I don't need to remove the root password16:38
weeb1eBut I don't want all accounts to use that password16:38
weeb1eWould removing "@include common-password" be enough to solve that?16:38
RoyKweeb1e: all accounts have their own passwords16:38
weeb1eRoyK: Like I said I can't login or sudo with my own accounts password16:39
weeb1eIt only started working when I set the root password to my own password16:39
=== jkyle_ is now known as jkyle
rbasakweeb1e: I'd avoid changing /etc/pam.d at all unless you're restoring defaults that have been changed. AIUI, the behaviour you want *is* default on Ubuntu16:40
RoyK@include common-password is there by default16:40
weeb1eOk well let me change my own password and see if it takes effect now16:40
rbasakThe beahviour I've heard you describe here (as far as I've understood what you've said) *is not* default on Ubuntu.16:40
RoyKweeb1e: well, now, after you have successfully changed your password, login and try sudo -i16:41
weeb1erbasak: Yeah that was my understanding too, I've used plenty ubuntu servers and never experienced this before16:42
weeb1eBut now after having set the root password, changing my own accounts password works correctly16:42
RoyKweeb1e: I guess what you experienced was just taht you changed the wrong password16:42
weeb1eI still don't understand why it was not before16:42
weeb1eRoyK: I tried using passwd without sudo at least 5 times16:42
weeb1eAnd it said it worked, yet a new ssh session only worked with the old password16:43
RoyKnever seen that - ever - since I installed slackware 2.1 back in 199416:43
weeb1eVery odd behaviour16:43
RoyKweeb1e: indeed - does ssh youruser@localhost work with the new one?16:43
weeb1eWell, I have a second box that should be identical to this one, lets see how the password changing goes there16:43
weeb1eIt does now, it didn't before16:44
RoyKweeb1e: try localhost first16:44
RoyKif there's a difference between ssh to localhost and from another machine, there may be a man-in-the-middle somewhere16:44
RoyKwhich is rather alarming16:44
weeb1eNo, there is no difference, both ssh to localhost and an external ssh session failed for the first bunch of attempts16:45
weeb1eThey only started working with the newly set password after I changed the root password16:45
RoyKweeb1e: do both work now?16:46
RoyKthen you probably changed the wrong user's password16:46
weeb1eBut I have a second machine to test now, and it does not have a root password set16:46
RoyKtry again16:46
weeb1eOh well, that machine worked as expected16:47
weeb1eAll things do point to me having changed the wrong password, but I am also very sure that I did not.. but oh well, thanks for the help anyway16:48
weeb1eSo much for the machines being identical, the second box has something seriously wrong16:53
weeb1eE: Package 'build-essential' has no installation candidate16:53
RoyKweeb1e: I'd reinstall that if I were you16:54
RoyKperhaps run rkhunter or chkrootkit on it first16:54
RoyKand check the repos used16:54
RoyKor just nuke it16:54
weeb1eRoyK: Reinstall the whole OS?16:55
weeb1eI would have to get my sponsor to send a technician to do it16:55
RoyKif something has been let in that can be logging passwords, then it's rather bad16:55
RoyKcan you compare /etc/apt/* between the two machines?16:55
weeb1e"if something has been let in that can be logging passwords"?16:55
weeb1eWhere did you get that from?16:56
RoyKuse rsync -r from a separate machine to transfer the contents16:56
RoyKweeb1e: I'm just paranoid, sometimes that's all it takes16:56
weeb1eHmm, I'll compare the contents16:56
weeb1ewhy apt/* not just apt/sources?16:57
weeb1ethe sources.list files are the same16:57
RoyKbecause sources.d is another source to sources :P16:57
patdk-wksomeone can easily setup apt to use a proxy server16:57
patdk-wkand then give you whatever they want16:58
patdk-wkchecking sources won't detect that16:58
weeb1eYeah well, they could, but this sponsor likely does not have the technical know how for that :P16:58
* patdk-wk hopes no one gets my proxy :)16:58
RoyKweeb1e: check the checksums (md5 or sha) of passwd and the modules used by pam16:58
RoyKweeb1e: it may be false alert, but you're seeing some rather interesting issues that *may* turn up to be nasty16:59
weeb1eI'd need to find another 12.04.2 ubuntu server to compare against17:00
weeb1eLet me check if I have a VM installed17:00
RoyKweeb1e: first: download rkhunter and/or chkrootkit from the source, not from the repos, and run it/them17:00
weeb1eRoyK: I understand your concern, since I have just gained access to these boxes I'd rather be safe than sorry17:00
RoyKpatdk-wk: do you know any other checks to run on such a system?17:00
patdk-wknot really, I just don't bother anymore17:01
patdk-wkrestore from template17:01
RoyKpatdk-wk: why not?17:01
patdk-wkI do tend to keep the old ones around for inspection, and find the issue17:02
patdk-wkbut normally, people breaking into servers leave craploads of helpful info around17:02
RoyKpatdk-wk: doesn't work too well for physical machines, though17:02
patdk-wkgood thing I don't have any :)17:02
patdk-wkbut it would work the same way17:03
patdk-wkjust take longer to do a restore17:03
patdk-wkI do it for laptops, and desktops17:03
patdk-wkafter I install, I backup to a template, that I restore on the other ones17:03
patdk-wkand use if someone gets infected17:03
patdk-wkthat is windows though17:04
weeb1eI only have physical machines, without any physical access :/17:04
weeb1eVMs have too much overhead17:04
RoyKweeb1e: huh?17:04
RoyKweeb1e: we run 150ish VMs on 8 VMware hosts at work, and it runs smoothly17:05
RoyKwould probably run well on 6, or it will, when we reorganize the two clusters into one17:06
patdk-wksince when do vm's have overhead?17:09
patdk-wkatleast if your using an ept server, so e54xx or higher cpu17:09
RoyKpatdk-wk: heh - back before they added virtualization extensions ;)17:10
patdk-wkno, that was painful17:10
RoyKvmware around 2001 was rather heavy17:10
patdk-wkept caused it so you didn't have overhead for memory page changes17:10
patdk-wkif your server is that old, to not support vt, I would suggest, you don't need a server :)17:11
patdk-wkbut if your server is <5years old or so, you probably have ept support17:12
patdk-wkso the vm will have an unmeasurable amount of vt overhead17:12
patdk-wkI will say, going from physical to vmware, caused me a 15% additional overhead17:13
patdk-wkthen I realized the old servers didn't have ept, removed it, and I am <5% overhead17:13
RoyKpatdk-wk: got a cluster?17:15
patdk-wk4 clusters17:15
RoyKmany hosts?17:15
patdk-wklarge windows, small windows, large ubuntu, small rhel17:15
patdk-wkphysical, from 3 to 617:16
RoyKwhy separate the vm's into different clusters based on OS?17:16
patdk-wkroyk, they aren't17:16
=== matsubara is now known as matsubara-lunch
patdk-wkthey are in different datacenters doing different things17:16
patdk-wklarge windows cluster has like 5 rhel on it17:17
patdk-wkbut it has 400 windows vm's17:17
RoyKhow many hosts?17:17
patdk-wkon 5 blades17:17
RoyKnot bad17:17
RoyKhow much memory in those?17:17
patdk-wkcurrently, 144, and we are pushing into 80% used again17:17
patdk-wkthose blades are getting upgraded next spring, so moving to 386 or more ram, but need faster cpu's, single core performance in windows is really hurting lately17:19
RoyKwe have two clusters, plus a separate box for patient data, running a single vm, separate box of historical issues, I guess, since some people didn't trust putting a large VM on other machines that were exposed to the internet17:19
RoyKperhaps going for virtual datacentre one day17:19
patdk-wkif it had patient data, it would be a hippa issue here, much easier to say your in regulations17:20
patdk-wkbut not sure what the laws are there17:20
RoyKI know others that are using the same cluster for mixed data17:21
patdk-wkguess it's hipaa17:21
RoyKI see, thanks17:21
RoyKThe Norwegian Data Protection Authority is the main actor at this, and they allows (at least certain) installations of patient data VMs along with open servers17:22
PiciRoyK: "Exposed to the internet", you mean web servers? or just something that was able to access the internet outbound?17:25
patdk-wkok, hipaa doesn't forbid it :)17:25
patdk-wkbut if you don't want to report loss to your *customers*17:25
patdk-wkthen it must be approved and encrypted17:25
RoyKPici: web servers or others that can be reached from the internet17:26
patdk-wkso making it encrypted, and being able to verify loss, is simple if it's dedicated17:26
RoyKIMHO nothing is really dedicated when the blade is in the same chassis as the other blades and VLAN control is at the VM level17:27
patdk-wkok, it's long after lunch time17:29
RoyKpatdk-wk: 400 VMs on five hosts with 192GB seems rather heavy, it's like 1.92GB per VM17:44
RoyK(with four, if one fails)17:45
patdk-wkwell, depends on mem dedup though17:45
RoyKis that really efficient?17:45
patdk-wkas 240 or so are cloned win7, they dedup good17:45
patdk-wkya, they have 4gigs of ram each, and normally only use 1gig of ram each17:45
patdk-wkvmware view, for client access yes17:46
patdk-wkdestroyed on each logout17:46
RoyKshould try that out17:46
RoyKwe have some 20k users, mostly students, but some 1800 employees17:47
patdk-wkhow many are logged in at any given time?17:47
RoyKlooking at the fileserver statistics, perhaps 2k17:47
RoyKat really high times17:47
zuljamespage:  ping enjoy: https://code.launchpad.net/~zulcss/neutron/rename/+merge/17483217:47
patdk-wkwell, that would be how many licenses you need then17:47
RoyKpossibly rather expensive :P17:48
patdk-wkwell like everything17:48
RoyKdo you use thin clients for this?17:48
patdk-wkyou do it yourself, or you pay for it17:48
patdk-wkroyk, heh?17:48
PiciIs this a single hospital?17:48
patdk-wkclients==customers, we have no control over them17:48
RoyKPici: hioa.no17:48
RoyKpatdk-wk: I meant, are you using thin clients or PCs for this thing? thin clients as in those that only knows RDP or whatever access protocol, but doesn't have much of an OS locally17:52
patdk-wklike I said, how should I know17:53
weeb1eRoyK: I host realtime sensitive software which is affected by the overhead and timeslicing of virtual machines17:53
patdk-wkthey are controlled by the customer, offsite, nothing to do with our company17:53
weeb1eSuch software includes a variety of resource intensive game servers as well as multimedia transcoding and processing17:53
patdk-wkya, realtime stuff is not vm friendly17:53
weeb1eMy services are realtime and latency sensitive, so VMs are really not an option17:54
TheSovyou would be surprised how fast vm's can work17:54
patdk-wkdepends though17:54
TheSovwe keep hosts here with just 1 vm on them17:54
patdk-wkbut if latency is the only issue, latency normally trumps all vm latency issues17:55
RoyKweeb1e: I see17:55
TheSovthe 1 vm is a very important and high speed guest the reason its virtual is due to portability17:55
patdk-wknetwork latency17:55
RoyKwe installed varnish on a dedicated blade some time back, 200% speed increase17:55
RoyKso in some applications, virtualization isn't the best approach17:56
TheSovwhat happens if your blade backplane fails?17:56
patdk-wkroyk, that sounds like an ept issue :)17:56
TheSovwhich has happened to me17:56
RoyKpatdk-wk: ept?17:56
patdk-wkthe memory paging virtualization support in newer cpu's17:56
patdk-wkotherwise every page table lookup, hits the hyperviser17:57
RoyKTheSov: it all goes down, obviously, and the important VMs are started on the secondary site17:57
patdk-wkand since varnish is memory happy, it will matter a lot17:57
TheSovRoyK, im just saying virtualization, as much as it has its drawbacks is worth it most of the time17:57
patdk-wkI was getting 50% slowdown on some vm's17:57
TheSovif not for just machine portability17:57
TheSovnot being hardware dependant is ****** awesome17:58
RoyKTheSov: I know, but the positive side of virtualization is rather huge compared to the drawbacks17:58
TheSovi think we are arguing on the same side lol17:58
RoyK150 VMs as pizzaboxes would fill four racks17:58
RoyKand consume a rather large amount of power17:58
patdk-wkoh, maybe you needed those old rlx blade I used to have :)17:59
patdk-wk2ghz with 20gig drive, 24 per 4u blades17:59
RoyKwe have three Dell bladecentres atm17:59
patdk-wkso happy to drop them off a cliff17:59
TheSovi have an entire rack of dell r714's with 12 core processors and 128 gigs of ram18:00
RoyKrecycling the older ones for the secondary site18:00
TheSovthey rock18:00
patdk-wksounds like amd18:00
TheSovyes they are18:00
patdk-wkI'm normally ram heavy18:01
patdk-wkbut the 100% flash san is helping to change that18:01
patdk-wkno need to cache as much stuff in ram18:01
RoyKpatdk-wk: what sort of SAN do you have?18:01
RoyKsomething like zfs?18:03
patdk-wkit's not18:04
patdk-wkit works a lot like zfs, but it's not zfs at all18:04
patdk-wkthey are using raid3d, so it's basically raid6 but without a dedicated spare, but random holes all over18:04
RoyKhave you tried to yank a disk and put it in a zfs-enabled box and tried zpool import?18:04
patdk-wkit wouldn't work18:05
patdk-wkit's not zfs, as it's raid3d :)18:05
patdk-wkeven if they did zfs ontop of it18:05
RoyKwhat's raid3d?18:05
patdk-wkgoogle it18:05
patdk-wkibm made it18:05
patdk-wkit solves the slow rebuild issue of using spares18:05
patdk-wkhard to explain without the picture18:06
RoyK<1s failover is nice18:07
RoyKDell tells EqualLogic customers to increase iSCSI timeout to 120 to avoid problems18:07
patdk-wkwell, it's active/active18:07
RoyKwhich doesn't work too well with internal timeouts in databases, exchange etc18:08
patdk-wkya, vmware says to use 180sec18:08
patdk-wkand it pushs that into windows18:08
patdk-wkbut not linux18:08
RoyKdoesn't work with exchange18:08
patdk-wkI have never failed over exchange yet18:09
RoyKexchange uses non-blocking I/O and fails after some seconds18:09
RoyKpatdk-wk: all SSD SAN?18:11
RoyKwhat interlink?18:12
TheSovdude, lefthand networks has an amazing virtual san appliance18:12
patdk-wkusing 8gb fc18:12
TheSovi use that in combo with freenas and RDM to produce a high speed full failover san solution that functions at high speed18:12
TheSovFC!?!? ok i get off the boat here18:12
RoyKpatdk-wk: guess you get rather good IOPS from that thing18:12
patdk-wkonly have 4 of the 8fc connected right now18:13
patdk-wkbut we can max out the 2 fc ports per host, easily18:13
patdk-wkwith 4k iops18:13
RoyK4kiops doesn't sound that impressive, though18:14
RoyKwhen a single SSD can deliver 10x+ of that18:14
patdk-wka single ssd can do >200k iops?18:15
patdk-wkI know I can do random write iops at full speed18:15
patdk-wkyou can't say that about zfs with dedup, very easily18:16
patdk-wkroyk, one thing I do like about them, and why their numbers do seem low18:17
patdk-wktheir numbers you will get, they are the best numbers under perfect optimization conditions18:17
patdk-wkand they are working on a cost scale too18:18
patdk-wkso one gen old hardware, to keep costs down18:18
=== matsubara-lunch is now known as matsubara
RoyKpatdk-wk: how much storage do you have in total (net) on those SSDs?18:28
patdk-wk11tb raw usable18:28
RoyKhow many SSDs?18:28
patdk-wkwe have 19tb of data on it18:28
RoyKseems like an awful lot18:28
patdk-wk48 256gb ssd's18:29
patdk-wkwe moved our 15tb of thin allocated data from our old san, to it, and used 4.5tb18:29
RoyKyou should e getting a wee more than 4kiops from that bunch18:29
RoyKeven spinning rust should give you 4kiops with that amount of drives18:29
patdk-wkwee more than 4k block size iops?18:30
patdk-wkI keep peeking out around 300-500k iops18:30
patdk-wkway over their specs18:30
RoyKthat's a lot18:30
patdk-wkcan easily get 100-150k for a single stream18:31
RoyK(and my excuses to the language police for saying a bad word)18:31
patdk-wkit must not count anymore, or bot the bot would yell :)18:31
RoyKpatdk-wk: want to ship this over? you don't need it, do you? :D18:31
patdk-wkI kind of like it18:33
patdk-wkwe are getting a 2x dedup ratio, and a 2.3x compression ratio on it18:33
patdk-wkthey join those numbers into one though, generally18:33
patdk-wkbut we pre-tested our data using a tool that will read your lun and spit out what it would use18:34
patdk-wkso you can estimate how much you need18:34
IdleOneIf you're going to make sarcastic comments in regards to the ops right after using language you clearly know is not acceptable, why do it at all?18:34
* patdk-wk failed to see any sarcastic comments made18:36
IdleOnebecause you are not aware of all the facts perhaps18:36
IdleOneAnyway, let us all try to behave according to the rules please.18:37
patdk-wkIdleOne, is something about this sarcastic? "<RoyK> (and my excuses to the language police for saying a bad word)"18:38
patdk-wkas that was the only thing said after the word18:38
patdk-wkin this channel atleast18:39
patdk-wkand everything you said, if it was in reply to an off-channel comment, not sure why you would bring it in here18:39
IdleOneyes, first of all we are not "language police" second of all if you are aware enough to apologise for doingsomething wrong then you should have been aware enough not to do it.18:39
patdk-wkthe second part is not true18:40
patdk-wkit's one thing to know you did something wrong, it's another thing to break your habbit18:40
IdleOnesure it is. There is no excuse for bad behaviour. When someone joins an Ubutu channel they know what behaviour is acceptable and expected.18:40
* patdk-wk notes almost all drug addicts18:41
IdleOneespecially someone who has been in ubuntu channels as long as RoyK has.18:41
IdleOneWe all mess up now and then I'll grant you that, but in light of recent history. I think the rules woyuld have been fresh in his mind.18:42
=== cmagina is now known as cmagina-away
=== cmagina-away is now known as cmagina
=== arrrghhhAWAY is now known as arrrghhh
=== cmagina is now known as cmagina-away
=== cmagina-away is now known as cmagina
cybervikingHI there21:31
cybervikingI installed apache 2.4 but now when i try to analyze a log file, I got "-bash: fork: Cannot allocate memory" and the ssh session close. Do you know why ? :)21:32
RoyKcyberviking: what ubuntu version?21:39
RoyKcyberviking: how much memory?21:39
cyberviking          total     used    free21:41
cybervikingMem:       2097152     287824    180932821:41
cyberviking-/+ buffers/cache:      20072    207708021:41
cybervikingtrying to analyze a 35mo fil via some grep21:41
RoyKpastebin ps axfv21:44
RoyK!pastebin | cyberviking21:44
ubottucyberviking: For posting multi-line texts into the channel, please use http://paste.ubuntu.com | To post !screenshots use http://imagebin.org/?page=add | !pastebinit to paste directly from command line | Make sure you give us the URL for your paste - see also the channel topic.21:44
RoyKcyberviking: cannot fork seems like a bunch of processess staggering21:45
RoyKcyberviking: pastebin output of uptime as well21:46
cybervikingthe command is not so impressive but it crash, just one to know how time Googlebot was there21:47
cybervikingcat /var/log/apache2/other_vhosts_access.log|grep "15/Jul"|grep -v "Googlebot"|wc -l21:47
cybervikinguptime :  23:47:18 up  3:11,  1 user,  load average: 0.00, 0.00, 0.0021:47
RoyKso probably no disk issues21:48
RoyKbut now swap?21:48
cybervikingforget the "-v" on grep above of course :p21:48
cybervikingit's a VPS21:49
cybervikingwith no swap21:49
cybervikingSwap:            0          0          021:49
cybervikingI can shutdown apache, execute this command and start apache again it works ^^. But I want to understand what the hell happen here.21:55
RoyKpastebin ps axfv21:56
cybervikingthe only difference is without apache -/+ buffers/cache:      11092    208606021:57
cybervikingand with apache : -/+ buffers/cache:      15912    208124021:57
RoyKshould be no difference21:58
cybervikingI know :s, but it's not :D21:58
=== arrrghhh is now known as arrrghhhAWAY
jsonperli've got a bit of a strange situation with memory (potentially a swap thing)22:14
jsonperli have a bunch of servers running with 16gb of ram available... they have a leak and when they get somewhere above 1GB, they get restarted22:14
jsonperlbut for some reason, freeing of that memory seems to make the whole machine spike in cpu usage, and slows everything WAY DOWN while it happens22:15
jsonperli was thinking maybe tuning the swappiness might be the solution, but does anyone have an idea what I should be looking for?22:15
Patrickdkwhy would you think this is a swap issue?22:29
Patrickdkwhere is a pastebin with any results that back this up?22:29
Patrickdkcause if you have 16gig ram, and you reboot them when they > 1gb ram, you have personal issues, not swap issues22:30
jsonperleach process has 1gb of ram22:32
jsonperlthere are 14 server instances running22:32
jsonperlim now running sysstat so I can get some stats next time i see the issue22:32
jsonperlits a custom ruby / c game server22:32
Patrickdkreally, all you need to do is run vmstat, and maybe free, and probably ps axl, when you are having the issue22:33
Patrickdkto tell if you have a swap issue or not22:33
jsonperlwhat would i want to look for?22:33
Patrickdksomething wrong22:33
freeflyingjamespage: is there any particular reason for openvswitch package not using upstart?22:34
jsonperldoes that symptom seem indicative of a swapping issue?22:34
Patrickdkjsonperl, the issue is unknown yet, as you have not described anything22:34
Patrickdkyou said a cpu spike, swap issues don't cause cpu spikes, they cause disk spikes22:34
Patrickdkso far, that is the only clue given22:35
jsonperlsure ok here22:35
Patrickdkwhat is nice, is to use something like munin, so you know what it *normally* looks like22:36
Patrickdkthen you can tell what changed22:36
jsonperlBasically all server activity drops to 022:36
Patrickdksysstat does it also, I just never used it22:36
jsonperli have charts of core usage22:36
jsonperli basially persist mpstat to db22:36
jsonperlit only starts happening once servers cycle... and release a lot of memory22:36
jsonperlI'll paste one somewhere and link22:36
Patrickdkmpstat only gives cpu info22:37
jsonperlCpu is clearly a problem here22:38
Patrickdkso cpu usage drops to bottom22:38
Patrickdkthat says cpu is NOT the issue22:38
Patrickdkso again, we have no idea22:38
jsonperlWhat are some potential reasons for that22:38
jsonperlheavy IO wait time?22:38
PatrickdkI could list you atleast a few million22:38
Patrickdkbut there is no point22:39
jsonperltheres not much running on the machine22:39
jsonperlpretty much just these servers22:39
Patrickdkthis is why you need to record all basic stats22:39
Patrickdkdisk i/o, memory, cpu22:39
Patrickdkall in reference to each other22:39
jsonperlsysstat is doing that for me now22:39
Patrickdkother things, if this is a vm22:39
Patrickdkit could not be anything to do with you22:39
jsonperlits not a vm22:40
jsonperlphysical machine22:40
jsonperlall mine22:40
jsonperlPatrickdk ok im collecting stats on the minute now22:53
jsonperlhopefully ill see something interesting22:53
jsonperlthis sucks22:53
jsonperlI run deepworld btw... fun game if you have a mac or ios device22:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!