=== hggdh is now known as WieWoWasWozu
cpaelzergood morning06:31
lordievaderGood morning07:13
Slashmanhello, I think that I'm hitting a bug with the latest openjdk, I would like to rollback to the previous version, is there any way to do that ?11:05
joelioSlashman: within a point release or new version11:17
joeliobut fundementally, yess11:17
=== andreas is now known as ahasenack
Slashmanjoelio: apt-cache policy only show me 8u151-b12-0ubuntu0.16.04.2 or 8u77-b03-3ubuntu311:19
joelioit may still be in your /var/cache/apt/archives dir11:19
joelioin which case you can dpkg -i it11:20
joelioI take it it's 8 series you need?11:20
Slashmanjoelio: no trace of it in /var/cache/apt/archives11:28
Slashmanjoelio: I used the oracle jdk instead, since any tar.gz can be downloaded from their website... it's a little sad that I have to switch to that to use a different version11:29
joeliothe oracle jdk is available as a package too11:32
joelioSlashman: also, you probably have lost that specific version do to a security update11:39
joelioit's... java at the end of the day ;)11:39
SlashmanI prefer to have several different java version to test it, but it doesn't seem to come from the jvm in the end11:41
SlashmanI have "fork: retry: Resource temporarily unavailable" with 40GB free ram and plenty of none of limits that I know of breached11:42
SlashmanI have "fork: retry: Resource temporarily unavailable" with 40GB free ram and none of limits that I know of breached11:42
Slashmanthe result is "java.lang.OutOfMemoryError: unable to create new native thread"11:43
Slashmanwith 40GB ram free and 100GB unused swap, it should be some kind of limit... but I don't see which11:44
joelioSlashman: you need to set in in the jvm options12:11
joelioas the heap is a value you set, depending on the resource needed12:11
joelioalso be aware that the garbage collection mode changes when you set it about 4GB12:12
joeliobut that will probably be your issue12:12
joelioSlashman: what's the application you're using in java (usually there is an /etc/default/{thing} that allows you to tune)12:15
joelioor in things like Elasticsearch there is a jvm.options file nowadays12:15
faekjarzHi! [17.10 server / netplan] I want one of my NICs configured but booting into link DOWN. Where do i find the information / documentation to acheive this?13:23
tomreynfaekjarz: try asking in #netplan if you can't get help here13:26
tomreyndocumentatio should be in 'man 5 netplan' and online at http://people.canonical.com/~mtrudel/netplan/13:27
tomreyn...according to https://wiki.ubuntu.com/Netplan13:27
faekjarztomreyn: aye, i did already, but my pesky impatience ;)13:28
tomreynthis is the first time i heard of it, i assume it's fairly new13:28
faekjarzyes, i've found ~mtrudel/netplan already but ctrl+f link doesn't highlight what i'm looking for. Wrong keyword?13:32
Slashmanjoelio: that's not  a jvm issue, the error happens even if I try to run "java -version"13:35
Slashmannot sizing I meant, it's production software, that's not a new service or anything13:35
Slashmanjoelio: you can see the error when trying to run "java -version" here: https://apaste.info/TIIb13:37
joelioSlashman: have you tuned the heap, otherwise you'll be running on the default.13:39
joelioerr, ifd java -versoon is broken then I don't know, sounds like it's fubar13:39
Slashmansome system limit are reached, that's how I interpret it13:40
Slashmanbut nproc, nfile, etc are far from their limit13:40
joelioumm, you said you'd used oracle version?13:42
joeliovm_info: OpenJDK 64-Bit Server VM (25.151-b12) for linux-amd64 JRE (1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12), built on Oct 27 2017 21:59:02 by "buildd" with gcc 5.4.0 2016060913:42
SlashmanI'm using openjdk atm13:43
Slashman1.8.0_151 build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b1213:43
joelioso what that output is literally from a java -version ?13:43
joelio(in the pastebin)13:43
joelioeh, you said it was failing13:43
joelio13:35 < Slashman> joelio: that's not  a jvm issue, the error happens even if I try to run "java -version"13:44
joeliohave you checked your heap space?13:44
joeliothis is a fairly common thing to do13:44
joelioi.e. increase heap allocated to a java application13:44
joeliohave to do it on stuff deployed here, as the 256Mb is pretty low13:44
SlashmanI'll save you some time, this is a corporate production server, the prod JVM are tuned, the server had no issue, we serve thousand of connections per JVM, I'm trying to understand why now we have an issue where we have JVM error and "java -version" doesn't work anymore, the problem happened this morning and we had to restart 3/7 JVM, after restarting just one, the problems goes away, for a time13:47
Slashmanso it works for a time and suddenly we see "java.lang.OutOfMemoryError: unable to create new native thread" in tomcat logs when there is available memory, both in the heap and on the os13:48
Slashmanonly solution is to restart tomcat at this point13:49
Slashmanso, since this problem is "new", my guess is that we have reached some system limit, that is not the amount of available memory13:50
Slashmanit may have been a bug in the new java version but I just confirmed that we hav the issue on the previous one too, so that's not it13:53
=== albech1 is now known as albech
joelioSlashman: memtest?13:56
joelioor this n multiple boxes?13:57
Slashmannot possible right now, but the server iDrac doesn't report any ECC issue, I'll have to try it13:57
Slashmanjoelio: it happens on 2 servers, you're right, doesn't seem related to hardware13:57
Slashmanmaybe it's ubuntu related, I have some debian servers without the issue with the same config, I'll compare the sysctl values...13:58
joelioyea, also check process list output to see what it's been instantiated with, just to make sure those values are being set13:59
Slashmanwhat do you mean by that?14:01
joeliothe ps -ef / auxx output for the java process14:07
joeliocheck how it's been intantiated14:08
joeliomaybe there is a subtle difference14:08
Slashmanoh, that's the exact same line exept pid number ofc14:12
=== JanC is now known as Guest54856
=== JanC_ is now known as JanC
drabikonia: sdeziel: fwiw, I got ldirectord working with 3 containers, 1 director, 2 real servers16:06
drabcouple things are broken in the default pkg so took me a while, but it otherwise works very well16:07
drabsystemd unit has a bad path and the pkg actually won't install at all16:07
drabas there's a race condition with the config file16:07
sdezieldrab: glad to hear that. I'd have to look into ldirectord as it's new to me16:09
drabthere's still something I don't understand as far as networking goes tho, several of the howtos seemed to say I had to set the director as the default gw for the real servers, but I didn't16:09
drabI guess it seems more lightweight than haproxy/nginx for just pure tcp/udp connections16:09
draband *a lot* simpler, which is all I need16:09
sdezieldrab: the default gw thing is related to the asymmetric routing we talked about the other day16:11
sdezieldrab: could you share your ldirectord config?16:13
sdezielhttp://www.linuxvirtualserver.org/docs/ha/ultramonkey.html shows that is can "masq"uerade real servers so that could be why you got away without changing the default gw16:14
drabwell, you'd think, yeah, but actually when you look at how things re set up I don't get it16:14
drabI don't do masq, I do "gate" which is direct routing16:14
drabthe cavia in most howtos, for the real server to respond , is that a non arping interface needs to get the VIP16:15
drabso often lo:0 gets the VIP/32 , that's how I've seen in in most howtos, and it makes sense16:15
drabbox won't arp for that (you need to tweak sysctl), and it will still accept connections for that ip since it sees it as local16:15
drabresponses will also originate from that ip since it's the obvious selection given that the request was received for it16:16
drabso it all seems to make sense and it works just fine, leaving me puzzled why I should be setting the gw to the director16:16
drabwhich several howtos I found mention16:16
drabaltho they are all at least 5-7yrs old16:17
sdezielyeah, everything in that space seems pretty dated documentation-wise16:17
sdezielisn't ldirectord for http/https backends only though?16:18
drabwell ldirectord actually really only check that backends are alive, it doesn't even do any of the switching etc16:19
drabthat's done in kernel by ip_vs, which you have to modprobe16:19
drabso technical you can just install ipvsadm and modprove ip_vs and you're done16:19
drabas far as balancing goes16:19
sdezielOK so only the health checks, right16:19
drabbut that won't give you any monitoring of the backends. the monitoring part seems to be http in many examples, but maybe not16:20
sdezielso yeah, much lighter than a user space proxy like HAproxy/nginx16:20
drabalso ldirectord is just one perl script... which well, it's perl, but it's a single script16:20
drabsdeziel: and it happens in kernel space16:20
drabin theory this could simply be plugegd into nagios/icinga/whatever monitoring system16:21
sdezielyeah, got that :)16:21
draba nagios event_handler could run ipvsadm and remove the backend or something16:21
drabit would be trivial to implement16:21
sdezielis your VIP actually movable between 2 or more boxes?16:21
drabI haven't tried that part yet, it's next on the list, testing one component at a time16:22
drabgonna give keepalived a shot16:22
sdezielkeepalived integrates nicely with IPVS16:23
sdezieland keepalived can run whatever health you want it to16:23
sdezielno need to mess with nagios handlers16:23
drabI'm actually pretty happy to have figured this one out, because even in the case of exposing containers and whatnot this is now really straightforward, no iptables or other stuff16:23
drabseems I can just do straight ip_vs and fiddle with ipvsadm and I'm done16:24
drabset that up on the baremetal maybe and redirect to whatever containers on it at will swapping things around in just a oneliner16:24
drabsdeziel: keepalived will take care of the VIP, not the realservers, for those you still need something else like ldirectord or nagios16:28
drabor whatever16:28
drabmy point was, ldirectord isn't technically needed to get the balancing part going, I thought it was16:28
sdezielhttp://manpages.ubuntu.com/manpages/xenial/man5/keepalived.conf.5.html see the LVS section16:29
drabyeah, looks like I was wrong, so maybe ipvs + keepalived is all that's needed to both manage the VIP on the directors and manage the real servers.16:36
drabthat's great, one less compoennt, thanks16:36
sdezielldirectord seems to be responsible of monitoring and tapping into ipvsadm whenever needed16:37
sdezielkeepalived on the other hand provides a wrapper on top of ipvsadm and also handle monitoring16:37
drabwhich keepalivedd seems to be capable o doing too, no? that's how I read that section16:38
sdezielso if you setup with keepalived it should give you all the features you need16:39
sdezieland you won't have a SPOF anymore16:40
drabyeah, that should be fine, what I'm most concerned about is the zfs snapshot part that comes after that16:53
drabI've redone the lxd hosts so that /var/lib/lxd is on zfs itself, that way I can send snapshots over to a backup host and have all containers setup ina  single swoop16:54
drabbut in that case they are going to have the same ips, so they need to be stopped until it's time16:55
drabnow that I have lvs I'm wondering if instead I should have the containers on diff ips/not synced and just sync the data DS16:56
drabhaven't thought that through quite yet16:56
sdezielwhat do you mean same IPs? your 2 lxd hosts?16:57
drabthe containers16:57
drabif I put them on zfs and send the snaps to the failover lxd server16:57
drabthen all configs will be the same including mac and ip16:57
drabso if they come up I have a conflict16:58
sdezielI'm assuming you'll "lxc copy" them, right?16:58
sdezielbut yeah, the same instance can/should be up only once16:59
drabI wasn't planning on it, I was planning on setting up zfs-backup-snapshot or something like that16:59
drabsince all lxd is zfs backed up16:59
drabthat way I don't have to make a difference betwee lxd or other data stored on zfs16:59
sdezielunless you use the PPA/backports, I think there is no easy way to adopt a zfs16:59
sdezielhence the suggestion of lxc copy17:00
drabhow do you mean? adopt a zfs, that is17:00
sdezielsay you zfs send/receive the container's FS, the receiving lxd host won't be able to start it as is17:00
drabwhy not?17:01
drabI create a DS which I mount on /var/lib/lxd and then the default storage pool for lxc is a LXD DS. so both containers data and lxd config gets moved over to the receiving host17:01
drabby simply snapshotting everything and sending it17:02
draball names and whtnot are consistent, the only diff is the ip of the lxd host and its hostname, that's about it17:02
drabam I missing something?17:02
sdezielI guess this would work if you flip everything at once17:04
sdezielbut if you want to do it per container that's where you will need a different solution17:04
drabthis also means that ldirectord/keepalived wouldn't have to change since ips would be the same, it's basically almost cloning the whole system, which is quite neat, the only issue is the turn on/off17:04
drabthat's what I'm debating, if I will corner myself... but at the same time this keeps it pretty simple and this is a charity, not a tech company17:05
drabI just want them to have a decent failover solution and data in a diff physical place17:05
sdezielzfs send/receive should cut it then17:07
drabare you using sanoid/syncoid by any chance? seems one of the common solutions to deal with that stuff17:08
sdezieldrab: sanoid+syncoid might assist you with that17:08
sdezielif you want to venture into new territories, you can take a look at DRBD (~RAID1 over the network). Pretty nice17:09
drabyeah there was a thread about that on the zfsonnix ML, zfs + drdb, I think it's more than they need17:12
drabin fact simply telling them "if something happens turn off and keep off this machine" is possibly a very good place to be for them17:13
=== CodeMouse92 is now known as CodeMouse92__
drabsdeziel: did you look at http://www.znapzend.org/ by any chance?18:38
drabor even the "official" https://github.com/zfsonlinux/zfs-auto-snapshot18:38
sdezieldrab: no first time I hear about znap18:42
drabI like there README quite a bit: https://github.com/mikalsande/znap18:44
draband it's all bash, not a perl guy18:44
drabonly cavia is , the author says he only uses fbsd so it's only really tested there18:45
=== jelly-home is now known as jelly
hallyncpaelzer: why is https://launchpad.net/~ubuntu-virt/+archive/ubuntu/virt-daily-upstream disabled? :)22:07
Pinkamena_DHow to find what is causing 'Device or resource busy'? I am trying to move /home so that I can overwrite it with something else. However, I get "device or Resource busy" . I am su to root and there is nothing in lsof | grep home  . my cwd is in /22:17
drabPinkamena_D: is home on a diff drive?22:18
draban mv command is saying resource busy?22:18
drabwhups, got disconnected22:21
drabdon't know if my msgs went through22:21
Pinkamena_Dno, did not get any >.>22:21
drabPinkamena_D: I was asking, did you log in with ur user and then su to root?22:21
Pinkamena_Dsorry about it22:21
drabcould you login directly with root and try again?22:22
Pinkamena_Dthats correct, root login is disabled22:22
drabif your user's home is in /home then you can see the problem22:22
Pinkamena_Dthere is no way to remove all handles so thats not an issue?22:23
drabthere's probably somewhere a ref too that that's giving you the error even tho your user is technically not opening any file22:23
drabatltho that's just my guess, I don't think I've personally ran into that before22:23
drabPinkamena_D: what you could do is to change your homedir temporarily22:25
drabso say mkdir /var/tmp/tempuser22:25
drabchange your homedir to that, logout, log back in, try again22:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!