/srv/irclogs.ubuntu.com/2017/01/06/#ubuntu-server.txt

=== lynxman_ is now known as lynxman
=== lordievader is now known as Guest68833
=== TariusWork is now known as Guest58358
=== devil is now known as Guest89704
=== Chrisfu is now known as Guest38701
=== RickyB98 is now known as Guest31885
=== IdleOne is now known as Guest79909
=== KaeltenAway is now known as Kaelten
=== maxb_ is now known as maxb
=== baggar11_ is now known as baggar11
=== JanC_ is now known as JanC
=== JanC_ is now known as JanC
hypermisti turnt off all programs running ( that i started myself) and 827mb of ram is going no where can someone help me find the area / command i should use06:49
cpaelzerhypermist: maybe just cache?07:20
hypermistfound out i think its flask with a mem leak cpaelzer07:22
cpaelzerhypermist: ok, usually I'd say check /proc/meminfo and then depending what this points to07:23
cpaelzercache: vmtouch07:23
cpaelzeranon: smem -tk -c "pid user command swap vss uss pss rss”07:23
cpaelzerkernel stuff: slabcache and such07:23
cpaelzerbut you found it so, I hope there is a fix/workaround07:23
cpaelzerif you want the anon per mapping instead of per app smem -m -tk -c "map count pids swap vss uss rss pss avgrss avgpss"07:24
=== Guest89704 is now known as devil_
hypermistwelp cpaelzer okay so its not flask07:32
hypermisti dont think07:32
cpaelzerhypermist: then lets think together, where is your mem going07:33
cpaelzerhypermist: I'd recommend to sync and drop caches and then check where /proc/meminfo is pointing to07:33
cpaelzer$ sync; echo 3 > /proc/sys/vm/drop_caches07:34
cpaelzerfeel free to pastebin the meminfo if you want to think together about it07:34
hypermistwhen running that command it didn't output anything07:34
hypermistis it meant to ?07:34
cpaelzerhypermist: the first is clearing dirty page cache, the second is dropping all clean caches07:36
cpaelzerhypermist: only then you can get an idea where memory is going to07:36
cpaelzerhypermist: in some hardcore cases you even need to force it to swap to shrink some internal structures, but that is more rare07:36
hypermisthttp://prntscr.com/ds4fra07:36
cpaelzerhypermist: free is what nobody should care about, cat /proc/meminfo is a better start07:37
cpaelzerand you don't need pictures; just do like cat /proc/meninfo | pastebinit07:37
hypermisthttp://pastebin.com/xiGwHSjp07:38
cpaelzerhypermist: almost all that is remaining is in cache - hmm the drop should have freed that07:40
cpaelzerbut never the less it is memory that can be reclaimed in high memory pressure - so not really missing07:41
cpaelzeryou could go to force it out by cranking up a test to force it to swap and then stop07:42
cpaelzerthat usually gets the rest out but is unessesary stress07:42
hypermistits a pain so i sort of want it fixed haha07:42
hypermistso then if i have to setup a cron i can do so :D07:42
cpaelzerI don't see anything really missing07:42
cpaelzerit is correct that Linux tries to keep memory as utilized as possible and only gives up under stress07:42
cpaelzerall the caching that is done is useful07:42
hypermistso it wont screw up if i say run multiple programs07:43
cpaelzerIf you are afraid that you can't get all the memory in such stress cases test it07:43
cpaelzerI'll create a test command - give me a sec07:43
hypermistokay haha07:43
farhad--why cost variable doesnt change in my code: http://paste.ubuntu.com/23751145/07:44
farhad--i have always problem with this. and i know that i dont know something. some link that get me out of this problem forever.07:46
farhad--oh, sorry.07:46
farhadsorry to noise here.07:46
cpaelzer?07:47
cpaelzerhypermist: stress-ng --vm 1 --vm-bytes 256M --vm-keep --vm-populate --timeout 10s --metrics-brief07:49
hypermistdo i wanna run that with sudo ?07:50
cpaelzerhypermist: you can rise the amount of memory that touches and see how high you can go07:50
cpaelzerhypermist: no sudo needed07:50
cpaelzerhypermist: at some level this will cause swapping07:50
hypermistdo i have to install something ?07:50
cpaelzerstress-ng is a package07:50
hypermistokay installing it07:50
cpaelzerhypermist: I'd recommend running dstat -tvin in another terminal07:50
cpaelzerhypermist: once you see swapping to occur you know you are around the spot you can go07:50
cpaelzerif you do WAY MORE the oom killer might kill something07:51
cpaelzerfarhad: you sure one of the switch statements is hit?07:52
cpaelzerfarhad: add log debuggin there to make sure07:52
hypermistwell i changed vm-bytes to 1024M07:54
hypermistand no swapping is occuring07:54
cpaelzerhypermist: that means you can go 1G without07:54
cpaelzerhypermist: your meminfo suggested you can go somewhere around 1.5-1.7 to hit it07:55
cpaelzerand that will then also be what your programs can consume07:55
farhadcpaelzer: thank you for the feedback. i got mistake to say my problem here. i solved it in proper channel.07:59
cpaelzerok farhad08:02
=== Guest68833 is now known as lordievader
hypermistcpaelzer, Value 1572864000 is out of range for vm-bytes, allowed: 4096 .. 107374182408:05
hypermist ;s08:05
hypermistwhen i try do 1500M08:05
cpaelzerhypermist: well 1G per thread seems to be the limit then, but you can easily do it still - the number behind --vm is the number how much of those08:09
cpaelzerhypermist: so I'd think --vm 2 --vm-bytes 750M should do as well08:09
hypermistokay08:17
hypermistnope cpaelzer still hasnt made the cache drop08:20
hypermistcpaelzer, is it bad that my cache doesnt drop or does it not matter08:28
=== ahasenack is now known as Guest99073
=== pcw is now known as Guest98580
BlackDexhello there, is it possible to set every value you can add with sysctl also in the kernel boot params? for instance if i want to set the kernel.pid_max to a different value?08:43
cpaelzerBlackDex: some are exposed by the kernel as kernel argruments, but I don't think all of them - isn't sysctl.conf early enough for you?08:46
BlackDexcpaelzer: I'm using maas and juju. Some juju charms allow sysctl change, but that only happens if the charm is started. I prefere it to be done before that. And currently i have no option of setting this with juju or maas before08:49
cpaelzerBlackDex: you can deploy a custom syctl via user-data by Maas which will be handed to cloud-init when instantiating, so after first startup it will have the new sysctl.conf and will read and handle it on every boot08:50
cpaelzerBlackDex: would that work for you?08:50
BlackDexthat means that i have to change/add curtin stuff?08:51
cpaelzerBlackDex: If I'm not mistaken that means you could on your maas add a preseed, so everything installed by it will get this added08:53
cpaelzerBlackDex: https://maas.ubuntu.com/docs/development/preseeds.html08:53
BlackDexhmm oke08:53
cpaelzerThat just is what came to my mind, I'm not objecting if anybody else comes up with a nice solution08:54
BlackDexIt's not a bad idea :). I'm just a bit against changing the preseeds deliverd by maas08:55
cpaelzerBlackDex: I didn't mean to change, but to add custom bits on top08:57
cpaelzerBlackDex: otherwise it is hard to maintain correctly IMO08:57
cpaelzerbut I never did - maybe that is hard/impossible/featurerequest to "add on top"08:58
BlackDexor something like, {tag}08:59
BlackDexthat would be a nice thing08:59
cpaelzeryes08:59
BlackDexso i can create a special tag, like for kernel-opts that can be matched with a preeseed08:59
cpaelzerI like the idea09:00
BlackDexwell, ill go add a feature request :)09:00
=== Guest98580 is now known as PaulW2U
=== IdleOne is now known as Guest81108
BlackDexdone :) https://bugs.launchpad.net/maas/+bug/165451509:32
ubottuLaunchpad bug 1654515 in MAAS "Feature Request: Custom post-deploy (cloud-init or preseed) scripts per node linked to tags" [Undecided,New]09:32
hypermistcpaelzer, does it matter if i dont clear cache ?09:54
cpaelzerhypermist: ?10:23
hypermistcpaelzer, well after running the command you told me to it never dropped its cache10:23
cpaelzerhypermist: but the command succeded?10:24
hypermistyes10:24
cpaelzerhypermist: can you sudo run this http://paste.ubuntu.com/23751524/10:26
cpaelzerhypermist: and paste the whole output it created into a pastebin?10:26
hypermistokay10:26
hypermistcpaelzer, http://paste.ubuntu.com/23751534/10:28
cpaelzerhypermist: you have all you wanted 1921780 of 2038636 free, that is one of the best ratios I've ever seen10:29
cpaelzercache down to 25 MB10:30
cpaelzerwhich is ok10:30
hypermistoh okay10:30
hypermistoh i see i didn't notice :D10:30
cpaelzerto free the last 100M you really have to just shut down :-P10:30
hypermist:P10:30
cpaelzerI've seen systems wasting more memory just on device driver structures10:30
cpaelzere.g. if you plug a few thousand disks on SAN and init them all10:31
=== jamespag` is now known as jamespage
hypermisthaha10:42
=== Guest99073 is now known as ahasenack
=== ahasenack is now known as Guest76687
=== Guest76687 is now known as ahasenack
=== freyes__ is now known as freyes
BlackDexI have a strange issue with LXD and networking. I have serveral bare-metal servers which all are accessable with an MTU of 9000, i can ping from every bare-metal host to every bare-metal host with a package size of 8972 (9000).13:32
BlackDexI have a 13 LXD containers running with some services insided them13:33
BlackDexthe interfaces are bridged to the physical interface of the host13:33
BlackDexwithin some LXD containers i can ping with with 8972 to the host and to other hosts.13:34
BlackDexbut in other i can't ping with a size larger then 1472 :(13:34
BlackDexevery lxd interface is located in the same bridge on the same physical interface13:35
BlackDexi also can ping from lxd to lxd with a large mtu13:37
BlackDexso i'm a bit puzzled13:37
ikoniathat sounds interesting and unusual13:38
ikoniaare the containers all running on same physical host13:39
BlackDexyes13:39
ikoniaare all the bridges mapped to the same physical network device ?13:39
BlackDexand from host to host no problem, lxd to same host no problem. lxd to lxd no problem, lxd to other host bad13:40
BlackDexikonia: Yes, brctl shows all on the same bridge13:40
BlackDexif i tcpdump on the bridge allocated to the lxd i can see the packages13:40
ikoniainteresting, the symptoms sound like classic MTU problems, but as you say, you get different responses from different containers that are all using the same physical device, so the mtu under the hood is the same13:41
BlackDexalso if i tcpdump on the bond or br-bond i see the packages13:42
BlackDexyea, and i see the same packages length according to tcpdump from the working and not working lxd's13:42
lordievaderWhat is between lxd to other host? Does the full chain support jumbo frames?13:43
BlackDexlordievader: Yes, as it does for other hosts and most lxd's but not some13:44
lordievaderAh, right.13:44
BlackDexeven restarted the host, restarted the lxd containers them selfs, no change13:46
ikoniaBlackDex: out of interest if you do a traceroute for a "good" host and one for a "bad" host, do you see it use the same virtual interfaces ?13:47
BlackDexyes it does13:50
ikoniavery odd behaviour13:50
BlackDexwait a second13:51
ikoniayou're not looking in the wrong place are you, eg: something sily like your host is running low on ram (extreme example I know) and it's forcing virutal devices to have "loss"13:51
BlackDexno it does :)13:51
ikoniaas a test if you set the mtu small, say 256 (nice round number) does everything work13:53
BlackDexikonia: for the ping? or the network as a whole?13:54
BlackDexbecause ping works nice on 147213:54
ikoniaBlackDex: on the interface13:54
BlackDexeven flood ping13:54
BlackDexi tried that, and that seems to be working normaly13:55
BlackDexbut that isn't the solution i think ;)13:56
ikoniano no, I don't think thats a fix, but if that works, it does look like it's matching the symptoms of mtu13:57
BlackDexyea it does. Because everything seems to work nice with a lower mtu size within the lxd container it self13:57
ikoniawhat's the MTU on the physical card13:58
BlackDex900013:58
ikoniaBlackDex: does any of your internal comms between LXD use the physical NIC14:01
ikoniaBlackDex: do you see where I'm going.....14:01
BlackDexcomms?14:02
ikoniaBlackDex: container 1 -> container 2 yes you use bridged virtual interface, but depending on the IP addressing that may actually have to go via the physical interface14:02
BlackDexno, all interfaces are on the same subnet14:03
BlackDexall the interfaces of that same subnet are on the same physical interface/bridge14:03
ikoniaso the odds of it flooding the physical nic as a pass through is slim14:04
BlackDexyea, and that doesn't explain why container 1 can ping and container to can't14:05
ikoniagood point14:05
ikoniaBlackDex: are these live boxes, or can you play around safely14:05
BlackDexkinda live, depends on what to do :)14:06
ikoniashame,14:06
ikoniaso my gut is telling me somehow this is capacity14:06
ikoniaI was wondering if you could shutdown 2 - 3 of the "working" hosts and see if one of the "broken" hosts then starts working14:06
ikonia/win 414:07
ikoniaoops14:07
BlackDex:p14:07
BlackDexum14:07
BlackDexi think i have just 2 hosts which i can shutdown which both work :)14:07
BlackDexso i can check for the broken one14:07
ikoniaBlackDex: worth a shot if it doesn't cause you too much pain14:08
ikoniamay help prove if it's capacity or not14:09
ikoniamy gut is saying the symptoms look capacity, but it doesn't loook like it from the config you're sharing14:09
BlackDexnope, that doesn't seem to be the case14:12
BlackDex:)14:12
BlackDex:(14:12
BlackDexi even stop/started the not working14:12
BlackDexand after starting the alrady working, the both still work14:12
BlackDexalso, no messages in dmesg, syslog or what so ever :(14:12
lordievaderEliminating the basics, the mtu setting is applied correctly? Could you show the output of 'ip link show|grep mtu'?14:16
lordievaderOn the host ;)14:16
BlackDexlordievader: Those are all set correctly, as some containers do work with large packages, and a large ping from the host also works :)14:17
BlackDexlordievader: http://pastebin.com/pqqx5D9F14:19
BlackDexnote that br-ens255f0 is indeed 1500, so that is correct14:19
lordievaderStill quite a few nics with mtu = 1500.14:19
BlackDex;)14:20
=== _ruben_ is now known as _ruben
ikoniaBlackDex: this is most odd14:36
BlackDexindeed14:36
BlackDexi'm currently rechecking my switches14:37
BlackDexhope i can find something there14:37
ikoniaI enjoy something a bit different, but this is not offering much info14:37
ikoniaBlackDex: there is a good idea, are they all going into the same switch ?14:37
ikoniaactually - ignore that14:37
ikoniaI've just realised how stupid that question is14:37
BlackDexyes, but i figured that there is an mlag/lacp link14:37
BlackDexmaybe there is something strange over there14:38
ikoniabut it's not using the physical card at all14:38
ikoniaso even if there was a switch problem on the wire, it's not hitting the interfdace, you've got problems between container 1 and container 214:38
BlackDexthere are no problems between container and container14:38
BlackDexonly from container x to host14:38
ikoniaahhh, then I missunderstood that then14:38
BlackDexand LACP could explain the error14:39
ikoniayes, possibly, I don't think so, but it's worth ruling out14:39
BlackDexbecause what if the contairs that DO work are going via switch Y, and the one that doesn't via switch X, and the host i'm pinging wants to go via Y14:40
ikoniawhen you say container X to host do you mean the host they are on or a host generic on the network14:40
BlackDexhost in generic14:46
ikoniaBlackDex: yeah, worth checking then14:46
BlackDexother bare-metal system14:46
ikoniaBlackDex: I didn't quite grasp where your comms where being dropped here14:46
BlackDexEureka!14:46
ikoniagot something ?14:47
BlackDexits working now14:47
ikoniawhat did you do ?14:47
BlackDexthe switch is a cumulus switch14:47
BlackDexand i saw that the peerlink between the switches was just 1500!14:47
BlackDexand probably the traffic went from switch x to y via the peerlink14:48
BlackDexa 40GB link on 1500 mtu :p14:48
BlackDexso i now changed them all to 9216 (because of bridging etc.. also according to cumulus docs)14:48
BlackDexnow the only interface on the switch with 1500 is the management link14:49
ikoniawow - so you're flooding it basically14:50
BlackDexwell the traffic went like this...       lxd 9000 > lxd-host 9000 > switch1-port 9000 > peerlink 1500 > switch2-port 9000 > other-host 900014:54
BlackDexand because of LACP the other LXD went like this14:54
lordievaderHahaha, yeah. That doesn't get you a mtu of 9000 ;)14:54
BlackDexlxd 9000 > lxd-host 9000 > switch1-port#2 9000 > switch1-port#3 9000 > other-host 900014:55
ikoniawinner14:55
ikoniagreat find14:55
BlackDexthanks you both for being a soundboard for me ;)14:55
BlackDexIt helped me to clear everything14:56
ikoniaalways nice to see something a bit different / interesting14:56
BlackDexindeed. Well i learned something again today :)14:56
BlackDexwhich can be usefull for other stuff in the daily work14:57
DammitJimguys, I am planning upgrading Ubuntu from 14.04 to 16.0415:21
DammitJimI tested this on a server that has mysql-server 5.6 installed15:22
DammitJimfor some reason, the ubuntu upgrade removed mysql-client 5.6 and mysql-server wasn't working.15:22
DammitJimI had to purge mysql-server and then install version 5.715:22
DammitJimis this a problem because I didn't originally install just: apt-get install mysql-server without specifying the version?15:23
BlackDexmysql-server is normally linked to the stable version15:24
BlackDexor atleast stable according to cannonical15:25
BlackDexhow did you install it before/15:25
BlackDex?15:25
DammitJimso, it sounds like the latest stable version of mysql-server on 14.04 is 5.615:25
DammitJimapt-get install mysql-server-5.615:25
BlackDexah, well that could be an issue15:26
DammitJimso, what are my alternatives at this point?15:26
BlackDexand during the install you are asked to remove old packages15:26
DammitJimto upgrade ubuntu to 16.0415:26
compdocit doesnt upgrade the packages too?15:27
BlackDexwell, i think the best is to check what apt-get install mysql-server does currently15:27
BlackDexuse apt-get -n for a dry run15:27
BlackDexum15:27
BlackDexi mean `apt-get --dry-run`15:27
BlackDexso that it doesn't do anything at all15:27
BlackDex`apt-get --dry-run install mysql-server`15:28
BlackDexwith any luck, it is linked to the current version15:28
DammitJimlet me see15:28
BlackDexcompdoc: if a package uses a specific version number, and that packages isn't available anymore in 16.04, it won't upgrade, it will leave it there15:29
BlackDexbut if you have installed the mysql-client (without version) it could give some issues maybe, shouldn't but could15:29
DammitJimmysql-server : Depends: mysql-server-5.5 but it is not going to be installed15:30
DammitJimwhat the heck?15:30
BlackDexhehe15:30
BlackDexbecause you have 5.6 installed15:30
DammitJimwhy is it talking about 5.5 when I Have 5.6 installed15:30
DammitJimah, crap!15:31
DammitJimso, there are packages for 5.5 and 5.615:31
BlackDexapt-cache policy mysql-server15:31
DammitJimand we just happen to use 5.615:31
DammitJimBlackDex, what are we looking for? I don't want to paste all the output of that command15:31
BlackDexwhat version is it telling overthere15:32
BlackDexapt-cache showpkg mysql-server | grep mysql-server-515:32
DammitJim5.515:32
BlackDex5.5 is what my 14.04 server tells also15:33
BlackDexwell, what should happen is the following15:33
DammitJimso, am I screwed, then?15:34
BlackDexif you upgrade to 16.0415:34
BlackDexand do not remove any package15:34
BlackDexor at least check what is wants to uninstall15:34
DammitJimlet me check the same command on 16.0415:34
BlackDexan do-release-upgrade doesn't remove packages unless you tell it to15:35
DammitJimthat returns 5.715:35
BlackDexso it will leave your 5.6 installed15:35
DammitJimoh really?15:35
BlackDexwhat  you can do is not purge the package15:35
BlackDexbut just uninstall it15:35
BlackDexpurge will remove the config etc..15:36
BlackDexso `apt-get uninstall mysql-server-5.6`15:36
BlackDexafter that, do an `apt-get install mysql-server`15:36
BlackDexand you should be fine15:36
DammitJiminteresting concept15:37
BlackDexdone that with other packages then mysql and no probs at all15:37
DammitJimhttp://askubuntu.com/questions/760724/16-04-upgrade-broke-mysql-server15:37
DammitJimthat's what worried me15:37
DammitJimI know, I should take those forums with a grain of salt15:38
BlackDexwell, that can happen, but the awnser is correct15:38
DammitJimwhat answer? The one where he says to remove all .cnf files?15:39
BlackDexthe most important for mysql is the config and the database15:41
BlackDexas long as you have both, there will be nothing to wurry about15:41
DammitJimyeah, the database was still there thank goodness15:42
BlackDexyou can even install mariadb instead of mysql :)15:42
BlackDexmaybe a good idea if possible, create a backup of the database files15:42
BlackDexif you want to do a copy/past be sure to shutdown the mysql server first. Else you need to do mysqldump :)15:43
compdocmy 16.04 upgraded mine to 5.715:43
BlackDexcompdoc: probably because you had mysql-server installed :)15:44
BlackDexand not mysql-server-5.615:44
compdocyup. and it wouldnt stay running because of some conf changes. but all fine now15:44
BlackDexso it just upgraded mysql-server :)15:44
DammitJimgoing to mariadb is a bigger conversation with the rest of the teams15:57
DammitJimcompdoc, so, you did have some issues?15:58
compdocit was strange. mysql would die in the night. They changed to layout of the /etc/mysql folder and didnt want to use my.cnf, so i just had to make a small change, but took me a few days to spot it16:02
rbasakcompdoc: we do use my.cnf, but we have to share the path with MariaDB. Hence the changes.16:02
rbasakBut 5.7 also obsoleted some old configuration directives. That's the biggest cause of pain on upgrade AFAICT. We have some automated changes on upgrade for the most common things, but we can't cover everything unfortunately.16:03
compdocits been solid since16:04
rbasakDammitJim: I think you should be able fix up your scenario after upgrade to 16.04. Take a backup first though just in case.16:06
rbasakDammitJim: for MySQL vs. MariaDB, keep in mind that MySQL in Ubuntu is in main, and MariaDB is in universe. Both get good security support. MySQL security updates in Ubuntu come from Canonical's security team. MariaDB security updates in Ubuntu come from Otto, the MariaDB maintainer in Debian and Ubuntu.16:07
theGoati am trying to turn op a syslog-ng listener with TLS on ubuntu with syslog-ng 3.5  everything appears to be compiled correctly and i am getting no errors when running syslog-ng.  could there be something that is blocking it from setting up the listener, and would there be logs some where that would tell me why?16:21
=== jerichowasahoax` is now known as jerichowasahoax
=== Isla_de_Muerte is now known as NwS
=== kees_ is now known as kees
=== swebb_ is now known as swebb

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!