[06:49] <hypermist> i turnt off all programs running ( that i started myself) and 827mb of ram is going no where can someone help me find the area / command i should use
[07:20] <cpaelzer> hypermist: maybe just cache?
[07:22] <hypermist> found out i think its flask with a mem leak cpaelzer
[07:23] <cpaelzer> hypermist: ok, usually I'd say check /proc/meminfo and then depending what this points to
[07:23] <cpaelzer> cache: vmtouch
[07:23] <cpaelzer> anon: smem -tk -c "pid user command swap vss uss pss rss”
[07:23] <cpaelzer> kernel stuff: slabcache and such
[07:23] <cpaelzer> but you found it so, I hope there is a fix/workaround
[07:24] <cpaelzer> if you want the anon per mapping instead of per app smem -m -tk -c "map count pids swap vss uss rss pss avgrss avgpss"
[07:32] <hypermist> welp cpaelzer okay so its not flask
[07:32] <hypermist> i dont think
[07:33] <cpaelzer> hypermist: then lets think together, where is your mem going
[07:33] <cpaelzer> hypermist: I'd recommend to sync and drop caches and then check where /proc/meminfo is pointing to
[07:34] <cpaelzer> $ sync; echo 3 > /proc/sys/vm/drop_caches
[07:34] <cpaelzer> feel free to pastebin the meminfo if you want to think together about it
[07:34] <hypermist> when running that command it didn't output anything
[07:34] <hypermist> is it meant to ?
[07:36] <cpaelzer> hypermist: the first is clearing dirty page cache, the second is dropping all clean caches
[07:36] <cpaelzer> hypermist: only then you can get an idea where memory is going to
[07:36] <cpaelzer> hypermist: in some hardcore cases you even need to force it to swap to shrink some internal structures, but that is more rare
[07:36] <hypermist> http://prntscr.com/ds4fra
[07:37] <cpaelzer> hypermist: free is what nobody should care about, cat /proc/meminfo is a better start
[07:37] <cpaelzer> and you don't need pictures; just do like cat /proc/meninfo | pastebinit
[07:38] <hypermist> http://pastebin.com/xiGwHSjp
[07:40] <cpaelzer> hypermist: almost all that is remaining is in cache - hmm the drop should have freed that
[07:41] <cpaelzer> but never the less it is memory that can be reclaimed in high memory pressure - so not really missing
[07:42] <cpaelzer> you could go to force it out by cranking up a test to force it to swap and then stop
[07:42] <cpaelzer> that usually gets the rest out but is unessesary stress
[07:42] <hypermist> its a pain so i sort of want it fixed haha
[07:42] <hypermist> so then if i have to setup a cron i can do so :D
[07:42] <cpaelzer> I don't see anything really missing
[07:42] <cpaelzer> it is correct that Linux tries to keep memory as utilized as possible and only gives up under stress
[07:42] <cpaelzer> all the caching that is done is useful
[07:43] <hypermist> so it wont screw up if i say run multiple programs
[07:43] <cpaelzer> If you are afraid that you can't get all the memory in such stress cases test it
[07:43] <cpaelzer> I'll create a test command - give me a sec
[07:43] <hypermist> okay haha
[07:44] <farhad--> why cost variable doesnt change in my code: http://paste.ubuntu.com/23751145/
[07:46] <farhad--> i have always problem with this. and i know that i dont know something. some link that get me out of this problem forever.
[07:46] <farhad--> oh, sorry.
[07:46] <farhad> sorry to noise here.
[07:47] <cpaelzer> ?
[07:49] <cpaelzer> hypermist: stress-ng --vm 1 --vm-bytes 256M --vm-keep --vm-populate --timeout 10s --metrics-brief
[07:50] <hypermist> do i wanna run that with sudo ?
[07:50] <cpaelzer> hypermist: you can rise the amount of memory that touches and see how high you can go
[07:50] <cpaelzer> hypermist: no sudo needed
[07:50] <cpaelzer> hypermist: at some level this will cause swapping
[07:50] <hypermist> do i have to install something ?
[07:50] <cpaelzer> stress-ng is a package
[07:50] <hypermist> okay installing it
[07:50] <cpaelzer> hypermist: I'd recommend running dstat -tvin in another terminal
[07:50] <cpaelzer> hypermist: once you see swapping to occur you know you are around the spot you can go
[07:51] <cpaelzer> if you do WAY MORE the oom killer might kill something
[07:52] <cpaelzer> farhad: you sure one of the switch statements is hit?
[07:52] <cpaelzer> farhad: add log debuggin there to make sure
[07:54] <hypermist> well i changed vm-bytes to 1024M
[07:54] <hypermist> and no swapping is occuring
[07:54] <cpaelzer> hypermist: that means you can go 1G without
[07:55] <cpaelzer> hypermist: your meminfo suggested you can go somewhere around 1.5-1.7 to hit it
[07:55] <cpaelzer> and that will then also be what your programs can consume
[07:59] <farhad> cpaelzer: thank you for the feedback. i got mistake to say my problem here. i solved it in proper channel.
[08:02] <cpaelzer> ok farhad
[08:05] <hypermist> cpaelzer, Value 1572864000 is out of range for vm-bytes, allowed: 4096 .. 1073741824
[08:05] <hypermist>  ;s
[08:05] <hypermist> when i try do 1500M
[08:09] <cpaelzer> hypermist: well 1G per thread seems to be the limit then, but you can easily do it still - the number behind --vm is the number how much of those
[08:09] <cpaelzer> hypermist: so I'd think --vm 2 --vm-bytes 750M should do as well
[08:17] <hypermist> okay
[08:20] <hypermist> nope cpaelzer still hasnt made the cache drop
[08:28] <hypermist> cpaelzer, is it bad that my cache doesnt drop or does it not matter
[08:43] <BlackDex> hello there, is it possible to set every value you can add with sysctl also in the kernel boot params? for instance if i want to set the kernel.pid_max to a different value?
[08:46] <cpaelzer> BlackDex: some are exposed by the kernel as kernel argruments, but I don't think all of them - isn't sysctl.conf early enough for you?
[08:49] <BlackDex> cpaelzer: I'm using maas and juju. Some juju charms allow sysctl change, but that only happens if the charm is started. I prefere it to be done before that. And currently i have no option of setting this with juju or maas before
[08:50] <cpaelzer> BlackDex: you can deploy a custom syctl via user-data by Maas which will be handed to cloud-init when instantiating, so after first startup it will have the new sysctl.conf and will read and handle it on every boot
[08:50] <cpaelzer> BlackDex: would that work for you?
[08:51] <BlackDex> that means that i have to change/add curtin stuff?
[08:53] <cpaelzer> BlackDex: If I'm not mistaken that means you could on your maas add a preseed, so everything installed by it will get this added
[08:53] <cpaelzer> BlackDex: https://maas.ubuntu.com/docs/development/preseeds.html
[08:53] <BlackDex> hmm oke
[08:54] <cpaelzer> That just is what came to my mind, I'm not objecting if anybody else comes up with a nice solution
[08:55] <BlackDex> It's not a bad idea :). I'm just a bit against changing the preseeds deliverd by maas
[08:57] <cpaelzer> BlackDex: I didn't mean to change, but to add custom bits on top
[08:57] <cpaelzer> BlackDex: otherwise it is hard to maintain correctly IMO
[08:58] <cpaelzer> but I never did - maybe that is hard/impossible/featurerequest to "add on top"
[08:59] <BlackDex> or something like, {tag}
[08:59] <BlackDex> that would be a nice thing
[08:59] <cpaelzer> yes
[08:59] <BlackDex> so i can create a special tag, like for kernel-opts that can be matched with a preeseed
[09:00] <cpaelzer> I like the idea
[09:00] <BlackDex> well, ill go add a feature request :)
[09:32] <BlackDex> done :) https://bugs.launchpad.net/maas/+bug/1654515
[09:54] <hypermist> cpaelzer, does it matter if i dont clear cache ?
[10:23] <cpaelzer> hypermist: ?
[10:23] <hypermist> cpaelzer, well after running the command you told me to it never dropped its cache
[10:24] <cpaelzer> hypermist: but the command succeded?
[10:24] <hypermist> yes
[10:26] <cpaelzer> hypermist: can you sudo run this http://paste.ubuntu.com/23751524/
[10:26] <cpaelzer> hypermist: and paste the whole output it created into a pastebin?
[10:26] <hypermist> okay
[10:28] <hypermist> cpaelzer, http://paste.ubuntu.com/23751534/
[10:29] <cpaelzer> hypermist: you have all you wanted 1921780 of 2038636 free, that is one of the best ratios I've ever seen
[10:30] <cpaelzer> cache down to 25 MB
[10:30] <cpaelzer> which is ok
[10:30] <hypermist> oh okay
[10:30] <hypermist> oh i see i didn't notice :D
[10:30] <cpaelzer> to free the last 100M you really have to just shut down :-P
[10:30] <hypermist> :P
[10:30] <cpaelzer> I've seen systems wasting more memory just on device driver structures
[10:31] <cpaelzer> e.g. if you plug a few thousand disks on SAN and init them all
[10:42] <hypermist> haha
[13:32] <BlackDex> I have a strange issue with LXD and networking. I have serveral bare-metal servers which all are accessable with an MTU of 9000, i can ping from every bare-metal host to every bare-metal host with a package size of 8972 (9000).
[13:33] <BlackDex> I have a 13 LXD containers running with some services insided them
[13:33] <BlackDex> the interfaces are bridged to the physical interface of the host
[13:34] <BlackDex> within some LXD containers i can ping with with 8972 to the host and to other hosts.
[13:34] <BlackDex> but in other i can't ping with a size larger then 1472 :(
[13:35] <BlackDex> every lxd interface is located in the same bridge on the same physical interface
[13:37] <BlackDex> i also can ping from lxd to lxd with a large mtu
[13:37] <BlackDex> so i'm a bit puzzled
[13:38] <ikonia> that sounds interesting and unusual
[13:39] <ikonia> are the containers all running on same physical host
[13:39] <BlackDex> yes
[13:39] <ikonia> are all the bridges mapped to the same physical network device ?
[13:40] <BlackDex> and from host to host no problem, lxd to same host no problem. lxd to lxd no problem, lxd to other host bad
[13:40] <BlackDex> ikonia: Yes, brctl shows all on the same bridge
[13:40] <BlackDex> if i tcpdump on the bridge allocated to the lxd i can see the packages
[13:41] <ikonia> interesting, the symptoms sound like classic MTU problems, but as you say, you get different responses from different containers that are all using the same physical device, so the mtu under the hood is the same
[13:42] <BlackDex> also if i tcpdump on the bond or br-bond i see the packages
[13:42] <BlackDex> yea, and i see the same packages length according to tcpdump from the working and not working lxd's
[13:43] <lordievader> What is between lxd to other host? Does the full chain support jumbo frames?
[13:44] <BlackDex> lordievader: Yes, as it does for other hosts and most lxd's but not some
[13:44] <lordievader> Ah, right.
[13:46] <BlackDex> even restarted the host, restarted the lxd containers them selfs, no change
[13:47] <ikonia> BlackDex: out of interest if you do a traceroute for a "good" host and one for a "bad" host, do you see it use the same virtual interfaces ?
[13:50] <BlackDex> yes it does
[13:50] <ikonia> very odd behaviour
[13:51] <BlackDex> wait a second
[13:51] <ikonia> you're not looking in the wrong place are you, eg: something sily like your host is running low on ram (extreme example I know) and it's forcing virutal devices to have "loss"
[13:51] <BlackDex> no it does :)
[13:53] <ikonia> as a test if you set the mtu small, say 256 (nice round number) does everything work
[13:54] <BlackDex> ikonia: for the ping? or the network as a whole?
[13:54] <BlackDex> because ping works nice on 1472
[13:54] <ikonia> BlackDex: on the interface
[13:54] <BlackDex> even flood ping
[13:55] <BlackDex> i tried that, and that seems to be working normaly
[13:56] <BlackDex> but that isn't the solution i think ;)
[13:57] <ikonia> no no, I don't think thats a fix, but if that works, it does look like it's matching the symptoms of mtu
[13:57] <BlackDex> yea it does. Because everything seems to work nice with a lower mtu size within the lxd container it self
[13:58] <ikonia> what's the MTU on the physical card
[13:58] <BlackDex> 9000
[14:01] <ikonia> BlackDex: does any of your internal comms between LXD use the physical NIC
[14:01] <ikonia> BlackDex: do you see where I'm going.....
[14:02] <BlackDex> comms?
[14:02] <ikonia> BlackDex: container 1 -> container 2 yes you use bridged virtual interface, but depending on the IP addressing that may actually have to go via the physical interface
[14:03] <BlackDex> no, all interfaces are on the same subnet
[14:03] <BlackDex> all the interfaces of that same subnet are on the same physical interface/bridge
[14:04] <ikonia> so the odds of it flooding the physical nic as a pass through is slim
[14:05] <BlackDex> yea, and that doesn't explain why container 1 can ping and container to can't
[14:05] <ikonia> good point
[14:05] <ikonia> BlackDex: are these live boxes, or can you play around safely
[14:06] <BlackDex> kinda live, depends on what to do :)
[14:06] <ikonia> shame,
[14:06] <ikonia> so my gut is telling me somehow this is capacity
[14:06] <ikonia> I was wondering if you could shutdown 2 - 3 of the "working" hosts and see if one of the "broken" hosts then starts working
[14:07] <ikonia> /win 4
[14:07] <ikonia> oops
[14:07] <BlackDex> :p
[14:07] <BlackDex> um
[14:07] <BlackDex> i think i have just 2 hosts which i can shutdown which both work :)
[14:07] <BlackDex> so i can check for the broken one
[14:08] <ikonia> BlackDex: worth a shot if it doesn't cause you too much pain
[14:09] <ikonia> may help prove if it's capacity or not
[14:09] <ikonia> my gut is saying the symptoms look capacity, but it doesn't loook like it from the config you're sharing
[14:12] <BlackDex> nope, that doesn't seem to be the case
[14:12] <BlackDex> :)
[14:12] <BlackDex> :(
[14:12] <BlackDex> i even stop/started the not working
[14:12] <BlackDex> and after starting the alrady working, the both still work
[14:12] <BlackDex> also, no messages in dmesg, syslog or what so ever :(
[14:16] <lordievader> Eliminating the basics, the mtu setting is applied correctly? Could you show the output of 'ip link show|grep mtu'?
[14:16] <lordievader> On the host ;)
[14:17] <BlackDex> lordievader: Those are all set correctly, as some containers do work with large packages, and a large ping from the host also works :)
[14:19] <BlackDex> lordievader: http://pastebin.com/pqqx5D9F
[14:19] <BlackDex> note that br-ens255f0 is indeed 1500, so that is correct
[14:19] <lordievader> Still quite a few nics with mtu = 1500.
[14:20] <BlackDex> ;)
[14:36] <ikonia> BlackDex: this is most odd
[14:36] <BlackDex> indeed
[14:37] <BlackDex> i'm currently rechecking my switches
[14:37] <BlackDex> hope i can find something there
[14:37] <ikonia> I enjoy something a bit different, but this is not offering much info
[14:37] <ikonia> BlackDex: there is a good idea, are they all going into the same switch ?
[14:37] <ikonia> actually - ignore that
[14:37] <ikonia> I've just realised how stupid that question is
[14:37] <BlackDex> yes, but i figured that there is an mlag/lacp link
[14:38] <BlackDex> maybe there is something strange over there
[14:38] <ikonia> but it's not using the physical card at all
[14:38] <ikonia> so even if there was a switch problem on the wire, it's not hitting the interfdace, you've got problems between container 1 and container 2
[14:38] <BlackDex> there are no problems between container and container
[14:38] <BlackDex> only from container x to host
[14:38] <ikonia> ahhh, then I missunderstood that then
[14:39] <BlackDex> and LACP could explain the error
[14:39] <ikonia> yes, possibly, I don't think so, but it's worth ruling out
[14:40] <BlackDex> because what if the contairs that DO work are going via switch Y, and the one that doesn't via switch X, and the host i'm pinging wants to go via Y
[14:40] <ikonia> when you say container X to host do you mean the host they are on or a host generic on the network
[14:46] <BlackDex> host in generic
[14:46] <ikonia> BlackDex: yeah, worth checking then
[14:46] <BlackDex> other bare-metal system
[14:46] <ikonia> BlackDex: I didn't quite grasp where your comms where being dropped here
[14:46] <BlackDex> Eureka!
[14:47] <ikonia> got something ?
[14:47] <BlackDex> its working now
[14:47] <ikonia> what did you do ?
[14:47] <BlackDex> the switch is a cumulus switch
[14:47] <BlackDex> and i saw that the peerlink between the switches was just 1500!
[14:48] <BlackDex> and probably the traffic went from switch x to y via the peerlink
[14:48] <BlackDex> a 40GB link on 1500 mtu :p
[14:48] <BlackDex> so i now changed them all to 9216 (because of bridging etc.. also according to cumulus docs)
[14:49] <BlackDex> now the only interface on the switch with 1500 is the management link
[14:50] <ikonia> wow - so you're flooding it basically
[14:54] <BlackDex> well the traffic went like this...       lxd 9000 > lxd-host 9000 > switch1-port 9000 > peerlink 1500 > switch2-port 9000 > other-host 9000
[14:54] <BlackDex> and because of LACP the other LXD went like this
[14:54] <lordievader> Hahaha, yeah. That doesn't get you a mtu of 9000 ;)
[14:55] <BlackDex> lxd 9000 > lxd-host 9000 > switch1-port#2 9000 > switch1-port#3 9000 > other-host 9000
[14:55] <ikonia> winner
[14:55] <ikonia> great find
[14:55] <BlackDex> thanks you both for being a soundboard for me ;)
[14:56] <BlackDex> It helped me to clear everything
[14:56] <ikonia> always nice to see something a bit different / interesting
[14:56] <BlackDex> indeed. Well i learned something again today :)
[14:57] <BlackDex> which can be usefull for other stuff in the daily work
[15:21] <DammitJim> guys, I am planning upgrading Ubuntu from 14.04 to 16.04
[15:22] <DammitJim> I tested this on a server that has mysql-server 5.6 installed
[15:22] <DammitJim> for some reason, the ubuntu upgrade removed mysql-client 5.6 and mysql-server wasn't working.
[15:22] <DammitJim> I had to purge mysql-server and then install version 5.7
[15:23] <DammitJim> is this a problem because I didn't originally install just: apt-get install mysql-server without specifying the version?
[15:24] <BlackDex> mysql-server is normally linked to the stable version
[15:25] <BlackDex> or atleast stable according to cannonical
[15:25] <BlackDex> how did you install it before/
[15:25] <BlackDex> ?
[15:25] <DammitJim> so, it sounds like the latest stable version of mysql-server on 14.04 is 5.6
[15:25] <DammitJim> apt-get install mysql-server-5.6
[15:26] <BlackDex> ah, well that could be an issue
[15:26] <DammitJim> so, what are my alternatives at this point?
[15:26] <BlackDex> and during the install you are asked to remove old packages
[15:26] <DammitJim> to upgrade ubuntu to 16.04
[15:27] <compdoc> it doesnt upgrade the packages too?
[15:27] <BlackDex> well, i think the best is to check what apt-get install mysql-server does currently
[15:27] <BlackDex> use apt-get -n for a dry run
[15:27] <BlackDex> um
[15:27] <BlackDex> i mean `apt-get --dry-run`
[15:27] <BlackDex> so that it doesn't do anything at all
[15:28] <BlackDex> `apt-get --dry-run install mysql-server`
[15:28] <BlackDex> with any luck, it is linked to the current version
[15:28] <DammitJim> let me see
[15:29] <BlackDex> compdoc: if a package uses a specific version number, and that packages isn't available anymore in 16.04, it won't upgrade, it will leave it there
[15:29] <BlackDex> but if you have installed the mysql-client (without version) it could give some issues maybe, shouldn't but could
[15:30] <DammitJim> mysql-server : Depends: mysql-server-5.5 but it is not going to be installed
[15:30] <DammitJim> what the heck?
[15:30] <BlackDex> hehe
[15:30] <BlackDex> because you have 5.6 installed
[15:30] <DammitJim> why is it talking about 5.5 when I Have 5.6 installed
[15:31] <DammitJim> ah, crap!
[15:31] <DammitJim> so, there are packages for 5.5 and 5.6
[15:31] <BlackDex> apt-cache policy mysql-server
[15:31] <DammitJim> and we just happen to use 5.6
[15:31] <DammitJim> BlackDex, what are we looking for? I don't want to paste all the output of that command
[15:32] <BlackDex> what version is it telling overthere
[15:32] <BlackDex> apt-cache showpkg mysql-server | grep mysql-server-5
[15:32] <DammitJim> 5.5
[15:33] <BlackDex> 5.5 is what my 14.04 server tells also
[15:33] <BlackDex> well, what should happen is the following
[15:34] <DammitJim> so, am I screwed, then?
[15:34] <BlackDex> if you upgrade to 16.04
[15:34] <BlackDex> and do not remove any package
[15:34] <BlackDex> or at least check what is wants to uninstall
[15:34] <DammitJim> let me check the same command on 16.04
[15:35] <BlackDex> an do-release-upgrade doesn't remove packages unless you tell it to
[15:35] <DammitJim> that returns 5.7
[15:35] <BlackDex> so it will leave your 5.6 installed
[15:35] <DammitJim> oh really?
[15:35] <BlackDex> what  you can do is not purge the package
[15:35] <BlackDex> but just uninstall it
[15:36] <BlackDex> purge will remove the config etc..
[15:36] <BlackDex> so `apt-get uninstall mysql-server-5.6`
[15:36] <BlackDex> after that, do an `apt-get install mysql-server`
[15:36] <BlackDex> and you should be fine
[15:37] <DammitJim> interesting concept
[15:37] <BlackDex> done that with other packages then mysql and no probs at all
[15:37] <DammitJim> http://askubuntu.com/questions/760724/16-04-upgrade-broke-mysql-server
[15:37] <DammitJim> that's what worried me
[15:38] <DammitJim> I know, I should take those forums with a grain of salt
[15:38] <BlackDex> well, that can happen, but the awnser is correct
[15:39] <DammitJim> what answer? The one where he says to remove all .cnf files?
[15:41] <BlackDex> the most important for mysql is the config and the database
[15:41] <BlackDex> as long as you have both, there will be nothing to wurry about
[15:42] <DammitJim> yeah, the database was still there thank goodness
[15:42] <BlackDex> you can even install mariadb instead of mysql :)
[15:42] <BlackDex> maybe a good idea if possible, create a backup of the database files
[15:43] <BlackDex> if you want to do a copy/past be sure to shutdown the mysql server first. Else you need to do mysqldump :)
[15:43] <compdoc> my 16.04 upgraded mine to 5.7
[15:44] <BlackDex> compdoc: probably because you had mysql-server installed :)
[15:44] <BlackDex> and not mysql-server-5.6
[15:44] <compdoc> yup. and it wouldnt stay running because of some conf changes. but all fine now
[15:44] <BlackDex> so it just upgraded mysql-server :)
[15:57] <DammitJim> going to mariadb is a bigger conversation with the rest of the teams
[15:58] <DammitJim> compdoc, so, you did have some issues?
[16:02] <compdoc> it was strange. mysql would die in the night. They changed to layout of the /etc/mysql folder and didnt want to use my.cnf, so i just had to make a small change, but took me a few days to spot it
[16:02] <rbasak> compdoc: we do use my.cnf, but we have to share the path with MariaDB. Hence the changes.
[16:03] <rbasak> But 5.7 also obsoleted some old configuration directives. That's the biggest cause of pain on upgrade AFAICT. We have some automated changes on upgrade for the most common things, but we can't cover everything unfortunately.
[16:04] <compdoc> its been solid since
[16:06] <rbasak> DammitJim: I think you should be able fix up your scenario after upgrade to 16.04. Take a backup first though just in case.
[16:07] <rbasak> DammitJim: for MySQL vs. MariaDB, keep in mind that MySQL in Ubuntu is in main, and MariaDB is in universe. Both get good security support. MySQL security updates in Ubuntu come from Canonical's security team. MariaDB security updates in Ubuntu come from Otto, the MariaDB maintainer in Debian and Ubuntu.
[16:21] <theGoat> i am trying to turn op a syslog-ng listener with TLS on ubuntu with syslog-ng 3.5  everything appears to be compiled correctly and i am getting no errors when running syslog-ng.  could there be something that is blocking it from setting up the listener, and would there be logs some where that would tell me why?