=== lotuspsychje_ is now known as lotuspsychje
lotuspsychjeask here and idle a bit Koopz volunteers might wake at US timezone here07:45
Koopzhmm... my workday might be over by then ;D07:46
lotuspsychjeask anyway, you never know07:46
mIk3_08Hi guys.... how was the status of Linux as a Server Operating System?07:48
Koopzi've got 2 users on my server which i want to move over to a new server. Do i just need to edit /etc/passwd, /etc/group and /etc/shadow, add their lines, restart and i'm done? (already copied their home directories over)07:50
Koopzor do i even need to reboot?07:50
ducassemIk3_08: do you have an ubuntu question?07:51
mIk3_08none so far ducasse07:52
Koopzokay i got a package related question here, i just tried installing "cacti" which somehow ended up in apt also installing apache2 which i didn't want and shouldn't really have happened either08:23
Koopzthis is actually my first time checking dependencies for packages so i may be misinterpreting this but does cacti actually depend on libapache2-mod-php?08:24
blackflowKoopz: it shouldn't. I hink apache is just a default dep for "I need PHP!". If you installed php(-fpm) first, it should not call in apache08:26
blackflowKoopz: Uh, sorry, I mean web server, like nginx. if you have it installed first, it wouldn't call in apache for php deps. but you do need php-fpm08:27
Koopzi got php-fpm and nginx installed, that's why i'm asking08:27
blackflowoh and it still pulls in apache?08:28
Koopzhere's the map i got08:28
Koopzif i understand this thing correctly it wouldn't need libapache2-mod-php if i had javascript-common installed? ಠ_ಠ08:29
blackflowit shouldn't pull in apache if you have nginx installed. could be a bug in packaging. wouldn't be a first.08:31
Koopzthere does seem to be some kind of hard linking between apache2 and cacti though. After i installed cacti i tried removing all apache2 packages afterwards and when i removed apache2-bin, cacti got uninstalled too08:33
blackflowthe dep list says libapache2-mod-php OR php (virtual). so something else in that whole deplist must be pulling it in08:35
Koopzoh? does the pipe indicate the first option of the "OR"?08:36
blackflowKoopz: ah, wait....  it's a recommended package08:36
blackflowapt install --no-install-recommends cacti08:37
blackflowit's a bug if you ask me, it should be removed from cacti's recommended list as it's already a dependency pulled in via php (as alternative) and php is a must08:37
jamespagesahid: I'm about to start on the networking- and neutron- packages for release08:37
blackflowKoopz: also it's a bug that apache and nginx can coexist, as both would be attempted to start by default, and fail as both can't be listening on port 8008:38
sahidjamespage: ack, i'm building and testing nova08:39
jamespagesahid: fwiw if its really is just a version delta I'm not build testing prior to upload08:39
blackflowKoopz: yeah I was wrong. something else is pulling it in08:40
sahidjamespage: ah ok ok08:43
Koopzi figured out why apache2 was installed08:51
Koopz"php" wasn't marked as installed since i directly installed php-fpm08:52
blackflowKoopz: ah, so it must be explicitly installed, I see08:54
Koopzyeah i just avoided doing that since the last time i installed "php" apache2 was installed too08:56
Koopzit's safe to "install" it after installing fpm though08:56
jamespagesahid: ok just did neutron-vpnaas - how are you getting on?09:04
jamespagesahid: I'm going to restart from the bottom of the list and work upwards :-)09:10
jamespagewell after a coffee09:10
jamespagesahid: up to placement09:37
jamespagesahid: ok just did openstack-trove09:41
jamespagesahid: that just leaves nova* and octavia*09:42
jamespagesahid: tinwood is cutting the nova-lxd release09:42
sahidjamespage: ok i had to be afk, i just pushed nova https://git.launchpad.net/~sahid-ferdjaoui/ubuntu/+source/nova/09:49
sahidand i'm taking care of octavia right now09:49
sahid(btw i still have tested ova)09:49
sahidjamespage: https://git.launchpad.net/~sahid-ferdjaoui/ubuntu/+source/octavia09:52
jamespagesahid: master branch of the nova repo has rc2?09:55
sahidjamespage: i looked at http://uca-tracker/stein_upstream_versions.html09:57
jamespagesahid: well release is out :-)09:57
jamespageI don't know how often that report updates09:58
jamespagesahid: gbp import-orig --uscan should pick the latest version09:58
jamespagein the stein series09:58
sahidjamespage: not sure to understand, did i made a mistake?10:00
jamespagesahid: how did you download the new tarballs?10:01
sahidjamespage: uscan --verbose --download-version "$version" --rename --timeout 6010:01
sahidgbp import-orig --no-interactive --merge-mode=replace ../${project}_${version}.orig.tar.gz10:01
jamespagesahid: "gbp import-orig --uscan" will do much the same in one command10:01
jamespagebut will always pick the most recent version from tarballs.openstack.org (in this case)10:02
jamespagesahid: the debian/watch file is typically pinned to a major version series so its safe10:02
jamespagei.e. you won't jump to train :-)10:02
sahidoh yes...10:04
sahidok let me retry that10:04
sahidjamespage: ^10:04
jamespagesahid: +110:04
sahidjamespage: https://code.launchpad.net/~sahid-ferdjaoui/ubuntu/+source/nova/+git/nova10:16
sahidsounds better?10:16
jamespagesahid: yep - processing now!10:19
jamespagesahid: do you want todo the same for octavia (and octavia-dashboard)10:19
sahidyes sure i'm working on ocatva right now, i will do octavia-dashboard then10:20
=== Wryhder is now known as Lucas_Gray
tinwoodjamespage, sahid, 19.0.0 nova-lxd is now tagged and pushed to gerrit10:25
tinwoodsorry for delay; had to check a few things first10:25
jamespagetinwood: thanks!10:25
jamespagesahid: ok nova uploaded10:32
sahidjamespage: ack10:32
jamespagesahid: you do octavia-* I'll deal with nova-lxd10:32
sahidi'm working on octavia-dashboard but i have a issue with sphinx10:32
sahidwhen i execute gbp buildpackage -S -sa10:32
jamespagesahid: try with -d10:33
sahidok, all good: https://code.launchpad.net/~sahid-ferdjaoui/ubuntu/+source/octavia/+git/octavia https://code.launchpad.net/~sahid-ferdjaoui/ubuntu/+source/octavia-dashboard/+git/octavia-dashboard10:34
jamespagesahid: great10:34
jamespagesahid: ok both uploaded along with nova10:43
jamespagesahid: I'm just finishing off manila-ui and then I think we're all done10:44
jamespagesahid: most are wedged in the disco upload queue pending review by a member of the release team10:44
jamespagesahid: as we're in final freeze any seeded packages get reviewed10:44
jamespagesahid: but we have an exception so should all be ok10:44
jamespagesahid: release team just accepted all uploads I think so we're good10:50
jamespagetime to build,backport and recheck10:50
jamespage(all automated :-))10:50
DK2i need to downgrade to 7.1.27-1+ubuntu16.04.1 from 7.1.28-1+ubuntu16.04.1 is there any possibilitys? in the repos i can only find 7.1.28 theres no older package anymore10:52
jamespagesahid: if you need to check the upload queue - https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=10:54
jamespagethat's for disco - you can url hack for other releases :-)10:54
sahidjamespage: ack thanks :)10:55
jamespagesahid: so we have four in queue still - no need to chase yet :-)10:55
tomreynDK2: why do you need to downgrade php to an earlier version,a nd where are these versions from anyways (not ubuntu)?11:29
blackflow!info php xenial11:30
ubottuphp (source: php-defaults (35ubuntu6.1)): server-side, HTML-embedded scripting language (default). In component main, is optional. Version 1:7.0+35ubuntu6.1 (xenial), package size 2 kB, installed size 11 kB11:30
blackflow!info php bionic11:31
ubottuphp (source: php-defaults (60ubuntu1)): server-side, HTML-embedded scripting language (default). In component main, is optional. Version 1:7.2+60ubuntu1 (bionic), package size 3 kB, installed size 12 kB11:31
tomreynppa:ondrej/php for xenial has 7.1.28-1+ubuntu16.04.1+deb.sury.org+311:32
tomreynhttps://www.php.net/ChangeLog-7.php#7.1.28 fixes two security vulnerabilities. you don'T want to downgrade to a non-patched version.11:34
=== Wryhder is now known as Lucas_Gray
fooCan someone confirm: 0 */3 * * * /home/dev/sky/db-backups/autopgsqlbackup.sh - this runs at 0, 3, 6, 9, 12, 15, 18, 21 - right?13:50
rypervenchefoo: Correct. 00:00, 03:00, 06:00, etc.13:53
foorypervenche: thank you13:54
yossarianukhi - I am trying to setup a KVM host using ubuntu 18.04 - I want to set up a bonded bridge with VLAN with netplan14:01
yossarianukare there any examples any where ?14:01
yossarianukI can't find one that has bond, bridge, and VLAN14:02
yossarianukI have tried to attempt it - however the vlan isn't working - it may be due to needing config on the switch - I just wanted to make sure my config was correct14:03
yossarianukYou can see it here -> https://pastebin.com/uYx3u1NA14:03
cyphermoxit's a little hard to read because it's set up for tests, but there's https://github.com/CanonicalLtd/netplan/blob/master/tests/integration/scenarios.py#L7514:04
yossarianukCould some one look at the config (url above) and let me know if it looks like sane config ?14:04
yossarianukI wasn't sure if I put vlan in the right place..14:05
cyphermoxyes that looks fine14:05
fooI want to be extra sure... https://paste.ofcode.org/4aCLsTBGcQSi9M5Z44wXn7 - OOM is killing stuff left and right, this is becoming a significant issue and I'm having trouble tackling it. Does anyone see anything telling in that paste?14:06
fooI think OOM can still kill off process X even though it's caused by Y, right?14:07
yossarianukcyphermox: thanks - and thanks for the example... I notice that in the example you posted the vlan was added to the bridge interface - do I need to do that ?14:07
cyphermoxyossarianuk: no; you set it up the way you like we were just trying to mix and match things complicated enough to make it a good test14:09
cyphermoxfoo: it can kill any random thing asking for memory at the time; it doesn't have to be the process hogging things14:09
sdezielfoo: there is a selection process on what to kill during OOM14:10
foocyphermox: thank you, thought that was the process.14:10
fooWhat's the best way to see what process is consuming the most amount of memory over time?14:10
Ussatman vmstat14:12
Ussathtop is good also14:12
fooUssat: I was using atop but didn't see anything meaningful in there14:13
foovmstat 1 shows me IF the system is swapping. I want to know the exact process sucking up most memory (if that's possible). I don't seem to see that specifically in the man page... but perhaps I missed it14:13
fooWhat's also strange... I haven't ever seen the system swap. Despite OOM killing stuff. Should I see some swappage?14:14
fooIt's almost as if the system isn't set up to use swap14:14
foo(although it does exist I believe)14:15
Ussatlook at htop14:15
UssatI dont know any one command that will show that14:15
foooh, actually. There is no swap. heh14:15
fooSwap:             0           0           014:15
foo(from free -m)14:15
foo... it's probably suggested to have 2GB swap on a system or such, right?14:15
foo... to at least not have OOM kill off stuff14:15
fooI mean, tha doesn't solve my core problem but I'll probably want to do that14:15
fooUssat: thank you14:15
fooDo you suggest htop over atop? atop may be older14:16
yossarianukyou could try this14:16
yossarianukps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }'14:16
fooMem[||||||||||||||||||||||||||||||||||||||||||||||1.66G/1.95G]   Load average: 0.48 0.43 0.4414:16
yossarianukthat show mem of each process and sorts them14:16
fooyossarianuk: uh, thank you. |head of that... gives me some definite insight. This is helpful14:17
fooyossarianuk: saving this nugget for future14:17
UssatNo kidding, saved here also14:18
fooI'm thinking a polling script changed and is threading and sucking up resources.14:19
fooI'm tempted to run yossarianuk's command every minute with timestamp and log to file... |head14:19
* foo does14:19
foowhile [ 1 ] ; do date; ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | head; echo -------------; sleep 60; done14:20
fooNot the most pleasant, but output that to a file... it'll to the trick14:20
Ussatquick, dirty but effective, and thats what counts14:22
Ussatyossarianuk, I know you did not just come u with that, thats impressive14:22
Ussatnice one14:22
fooHmm, I wonder what this is: 111.37 Mb /usr/bin/lxcfs /var/lib/lxcfs/14:24
foo... now if only I could force this issue and see it happen in real time. For now, I wait, and trust the logs.14:25
fooAlso, can we agree that swap is generally a good idea? I'm a bit rusty in my admin but IIRC that's something I want.14:25
cyphermoxfoo, swap won't save you if something is leaking memory or getting to consume all that is available anyway14:27
fooUssat: oh. thank you.14:27
foocyphermox: aka. swap might just be eaten up too, correct?14:27
fooTrue, but can we agree... having it available (after I resolve this issue) is generally a good idea?14:28
cyphermoxswap is just "extra memory" on disk, that can be used to free up some RAM when context switching; but it's not a cure-all14:28
foocyphermox: agreed14:28
cyphermoxit generally will help14:28
sdezielbut it provides a nice space to push pages that are not currently in use14:28
fooagreed. ok, I'll look into that post-resolving this issue.14:29
* foo waits on the sidelines with a fly swapper 14:29
fooCome on you memory hog, show yourself14:29
cyphermoxfoo: you could just create a swapfile14:29
foocyphermox: I could, but I do want this issue to show itself... in the odd chance it doesn't consume all swap. I have htop going and while [ 1 ] ; do date; ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | head; echo -------------; sleep 60; done >> /home/foo/mem-issue.txt14:30
fooactually, per htop, I can see my system currently at 1.72/2GB RAM consumption. It's "idling" there... meaning, just a little more requirement could cause a problem14:32
fooDo I understand that correctly? I know sometimes the system uses available ram for when it needs it and thus that's not an actual current utilization IIRC14:32
fooGreat, it just happened - OOM murdered a process. Time to see what the culprit is.14:41
* foo enables per-second logging for more accuracy, every minute not enough if something spikes within minute and gets killed off14:53
JamesBensongbkersey: I've temporarily paused it.  I needed to get this Openstack deployment out.  I was hoping to use it for that, but I guess next round.  But I might ping you/community about it.  I bought these cards for all of our servers, r610,r710,r910.  So need to get them working!14:55
fooAnyone see anything strange here as it relates to memory usage? https://paste.ofcode.org/FDKFEpQt2e2ErWVXNb5Qrw14:55
sdezielfoo: you can look at some diagnostic that OOM-killer sends to dmesg, maybe that will tell you more about the culprit14:55
foosdeziel: hmm, I was but didn't see anything obvious, will take another look - thank you14:55
gbkerseyJamesBenson: hopefully you got the cards cheap... :)  I think I paid ~ $30 for mine...  Expensive thing was the 10G switch module for our 5406zl14:56
JamesBensongbkersey: :-/ I think it was around $65 a pop... for 15 cards.14:57
gbkerseynot that bad....  I bought a stack of fully populated R610/R710 for $100/box couldn't pass up the deal....14:58
JamesBensonWe've been buying from servermonkey servers and parts.  RAM from memoryamerica (lifetime warrenty)14:59
gbkerseyI found that the Dell twinax cables that came with the boxes would not work with the HP switch - because the nvram in the SPF did not say it was an HP14:59
gbkerseyended up buying a bunch of clone HP cables off of ebay and those worked just find.14:59
JamesBensonWe have the dell cables and 10g switch atm.  But we will need more cables, I found some on Amazon that should work14:59
fooIn the event someone knows how to read OOM data better than me and can provide some pointers, here's the OOM / kernel info: https://paste.ofcode.org/ytez6sPUZXdQbUQGyY69WS - I wonder if I want to look for oom_score in output?15:00
foosdeziel: ^15:00
fooThank you!15:00
* foo skimms15:00
gbkerseyI bought the clones on ebay....  they were cheap.15:00
JamesBensonyeah, we can't buy from ebay... university15:00
JamesBensontoo much of a pita.15:00
fooI mean, I guess it's possible this box just needs more memory...15:00
fooI could upgrade it to 4GB RAM. Actually, probably makes sense to add 2GB swap before doing that15:01
sdezielfoo: the way I read it, postgresql asked for more memory but none was available so OOM-killer started to look where to force reclaim some, the gunicorn process (27094) was selected as the best candidate to kill to free some RAM15:02
blackflowyou mean the kernel dice rolled just at the PID to the chagrin of gunicorn :)15:04
foosdeziel: thank you! Now, to help understand what you're seeing... you're basically looking at Apr 12 07:39:42 server kernel: [9534277.048613] postgres invoked oom-killer and then Apr 12 07:39:42 server kernel: [9534277.049074] Out of memory: Kill process 27094 (gunicorn) score 84 or sacrifice child - right?15:04
sdezielfoo: there are many invocation of the oom-killer in that paste, I only checked the first15:04
sdezielfoo: yes15:04
foosdeziel: thank you. yeah, it looks like even sshd invoked oom killer.15:05
gbkerseyJamesBenson: just be careful that the eeprom in the twinax matches your switch vendor especially if the switch is HP - the cards complain about the cable not being certified but they still work just fine.15:05
sdezielfoo: this first kill seem to have freed ~170mb of RAM15:05
foosdeziel: ... which would lead me to believe just because postgres invoked oom-killer, it's not necessary the main culprit... it simply couldn't find more memory available15:05
sdezielfoo: the process that wakes oom-killer isn't necessary the culprit, it just happens to be one process needing for some more memory but the memory pressure is the result of every process taking some memory away from the kernel...15:06
foosdeziel: I assume Apr 12 07:39:42 server kernel: [9534277.051857] Killed process 27094 (gunicorn) total-vm:391752kB, anon-rss:169656kB, file-rss:1164kB, shmem-rss:0kB - and specifically: anon-rss:169656kB is what you're seeing there. Thank you, this is helpful for me to do this myself next time.15:06
sdezielfoo: correct15:06
sdezielfoo: IIRC, the meaning numbers/metrics are "*-rss"15:07
blackfloweh "culprit" ... how do you define one. postgres wanted more RAM, kernel killed gunicorn in response. postgres totally is the culprit for that oom.    the only way to properly control that is to resource-limit individual processes, but that's usually less than optimal usage of RAM15:08
foosdeziel: right right. The next question is: A) does this server simply need more memory? or B) are some of the python processes ( per https://paste.ofcode.org/FDKFEpQt2e2ErWVXNb5Qrw ) simply taking "too much" memory. Yup https://stackoverflow.com/questions/18845857/what-does-anon-rss-and-total-vm-mean15:08
blackflowfoo: python is notorious for not returning the RAM it's no longer using, back to the OS15:09
sdezielfoo: it depends. your paste doesn't show the PID so it's hard to know. I'd check if a given gunicorn process sees its memory bubbling over time15:09
blackflowwe have some uwsgi apps that, for some requests, need to spike up RAM usage several times more than average. so we configure uwsgi to kill a running process when rss is larger than a set threshold15:09
fooblackflow: THANK you. I do have control over python code and can see about it... it's possible there is a python library causing an issue here15:10
blackflow(which happens after the request is done, this "killing" is a graceful shutdown-and-restart of the process)15:10
sdezieluwsgi is also what I've used and I liked it's flexibility15:10
fooWhat's interesting is I see this: [URGENT] set vm.overcommit_memory=2 in /etc/sysctl.conf and run sysctl -p to reload it. This will disable memory overcommitment and avoid postgresql killed by OOM killer. - from "/postgresqltuner.pl" - which leads me to believe... I might be able to set something to prevent oom getting invoked by python. Do ya'll generally suggest this?15:10
blackflowfoo: no idea, you should analyze individual processes RAM usage and make decisions based on that15:10
sdezielfoo: that's a global flag so it would be less risky to do on a dedicated DB server which isn't the case of your box15:11
blackflowfoo: no. overcommit is okay if used wisely. what you need is to resource-limit individual processes, so that OOM can't kill random processes15:11
fooOne thing that's somewhat telling, per https://paste.ofcode.org/FDKFEpQt2e2ErWVXNb5Qrw - line 11... 45.53 Mb /home/dev/website.com/venv/bin/python3 /home/dev/website.com/venv/bin/gunicorn - that's a django-based app. The other gunicorn stuff is for another app... and that's all at ~150MB. Sure, it's a bigger app, but if I had more insight into which python libs were sucking up memory there... hmmm...15:12
blackflowbut then only to find out what is frequently needing more than allocated, then act accordingly (eg, by adding more RAM, or by optimizing that process' RAM usage)15:12
sdezielfoo: you can probably do something more fine grain with systemd tuning how much RAM is given to gunicorn15:12
foothat might be helpful.15:12
blackflowwhich is the resource limiting that I'm talking about15:12
foosdeziel / blackflow  - thank you, I value some understanding here, appreciate your explanations.15:12
gbkerseyJamesBenson: this is what I see on the server side with the twinax I'm using: Warning: Unqualified SFP+ module detected, Port 0 from OEM but the next line says - NIC Link is Up, 10000 Mbps full duplex, Flow control: none15:13
fooSounds like my options are A) resource control gunicorn or B) see if I can less RAM usage in gunicorn (so 3 processes aren't taking up 150MB) ... what I'm not sure of is how A) would affect the actual gunicorn process (eg. if it can't get all it's memory, would it force gunicorn to complain? would it "slow down" performance for that process?)15:13
blackflowfoo: just remember that setting these flat limits makes your RAM usage suboptimal. allowing one process to temporarily peak is not bad, as long as you have the peaks under control15:13
blackflowwhen they all start to peak at the same time, that's when you need more RAM .... or somehow bring down those peaks.15:14
fooaka. bring down those peaks = option A or B)15:14
blackflowfoo: limiting a process will result in an error for that process only, when it requests more RAM and there isn't any15:14
sdezielfoo: or load balance the incoming requests between more servers15:14
blackflowfoo: python is verbose about that: https://docs.python.org/3.6/library/exceptions.html#MemoryError15:15
fooblackflow: great, that's what I was not aware of - thank you for explaining15:15
foosdeziel: this is such a low profile and low traffic app... this all randomly start which is what I'm wondering about15:15
fooblackflow: I actually have been seeing MemoryErrors, too. There are several things happening at once. Postgres generally is what complains about MemoryError. I've been trying to figure out root cause of this for about 3 months now. It randomly happened once in Feb, once in March, then about a dozen times this month. Not much has changed that I'm aware of. If anything, we switched from ubuntu 14.04 to15:17
foo18.04 in Dec 201815:17
blackflowfoo: how many workers have you configured for the gunicorn app?15:18
fooblackflow: there's a few different processes. eg. gunicorn runs django for main site, then gunicorn runs for our own app (that has 3 different gunicorn instances). Here's one of the instances sky-admin which is taking up the most RAM: https://paste.ofcode.org/YsYRbcRnnsbXdrj6rZpd7f - 1 worker15:21
blackflowso there's no dynamic number of processes? something that, say, scales up with number of requests coming in?15:23
blackflowbottom line you're definitely out of RAM. Since that's a DO droplet, perhaps it'd be wise to upgrade it, and then run a thorough analysis of how much RAM each process peaks at without an error, and then decide how/what to limit and whether you'll want to downgrade the droplet again15:24
fooblackflow: thank you! that's what I'm leaning towards... and even before upgrading droplet, I think enabling 2GB SWAP probably makes sense (right now none is enabled). Agreed? This would allow me to troubleshoot this, do testing, without things getting killed in producted by OOM15:26
blackflowpartially, yes15:27
fooActually, we probably had 2GB enabled swap on the old system before the 14.04 > 18.04 upgrade...15:27
fooThat might have been why I never saw this15:27
sdezielfoo: you might want to look at zram/zswap15:27
UssatI never run without a swap.15:27
* foo checks https://linuxize.com/post/how-to-add-swap-space-on-ubuntu-18-04/15:27
Ussaton any of my systems15:27
fooUssat: yeah... I think this was an oversight on my part15:27
UssatNP, happens and easy to fix15:27
foosdeziel: haven't heard of that, different than normal swapping I assume? Hmm, thanks15:27
blackflowI wouldn't recommend zram or zswap. that's like applying bandaid to a gaping wound.15:28
sdezielfoo: I just learned that it's presumably being used by default in ChromeOS15:28
foosdeziel: oh, interesting - thans15:28
blackflowespecially zswap is not swap at all, but memory compression of unused pages ---- that still occupy memory.15:28
fooblackflow / Ussat - appreciate your vote, thank you15:28
sdezielbut yeah, my first recommendation would be to use a plain swap(file) first15:28
foosdeziel: appreciate it!15:29
fooIs there a generally "best practice" swap size to use? I generally use 2GB15:29
Ussatfoo, thats a HOTLY debated topic. Generally all my systems have between 2-415:30
sdezielfoo: there are various guidelines. hybernation requires >= RAM IIRC15:30
Ussatbut it depends on system use, memory etc15:30
Ussatis this a physical or vm ?15:30
sdezielfoo: 2G sounds OK to me. If you end up swapping that much you'll definitely notice the performance hit15:30
Ussatand ya that15:31
fooUssat: VM, digial ocean droplet15:31
fooCurrently at 2GB15:31
Ussatsome apps (looking at you oracle) require a big swap15:31
fooI can add 2GB swap... if I go over that, probably makes sense to increase droplet swap15:31
Ussat2G swap should be fine15:31
Ussatand ya if you swap all 2G you will notice it15:31
blackflowsomething like Munin to monitor and graph over time RAM, swap usage, and other things, is very recommended too15:32
Ussatmost of my work VM's have 2-415:32
* foo learns about Swappiness Value15:33
fooSounds like next step here... given the OOM and reason for doing this... is to set up system monitoring and watching how often swap is used with some pretty graph or such, agree?15:33
foooh, heh, I just read backlog ... blackflow is a step ahead of me (thank you)15:33
Ussatif it runs a java app you might also look ap heap size15:34
Ussatloot at15:34
fooUssat: negative15:34
* Ussat is NOT a fan of java apps15:34
UssatThey are the bane of my existance15:35
fooUssat, blackflow, sdeziel, cyphermox, yossarianuk - I suspect I'm good for a bit here. I am very grateful for your time/contribution to this, thank you! This may all come down to not having swap enabled post a deployment I did in December. Still curious what is causing this to happen so much this month, but at least now I can troubleshoot/investigate without being stressed on a tight timeframe and I15:36
fooknow swap can cover for a bit. Thank ya'll.15:36
* foo waves magic wand and grants you all access to use his nick in config and code / etc15:37
blackflowfoo: re monitoring, yes. I'm a big fan of Munin we use on all our servers. But there's Zabbix and others too.15:37
blackflowfoo: having swap enabled even when you don't have OOMs is wise. nowadays various bloatware products will have unused pages that can be swapped out and RAM left for apps to use. esp. Python15:38
fooblackflow: I used to use nagios back in the day. I heard good things about Zabbix and Munun... going to look into getting this set up now. My next question is... do I spin up another Digital Ocean Droplet to monitor... wait actually, I guess I could run zabbix/munun on the system itself15:38
Ussatwe still use nagios :)15:38
UssatI just finished building a nagios server for core-team here15:38
blackflowmunin is very light, it's a cron based master process with (mostly) perl based sensors, that creates static HTML+png pages and graphs, which yes can run locally, no need for a separate DO15:39
fooUssat: rad!15:39
Ussatblackboxsw, I am trying to convince them to change to munin15:39
Ussatbut "we have always...."15:39
sdezielmunin and nagios have different use cases unless munin now offers more than it did years ago15:40
Ussatit does15:40
blackflowUssat: munin is very nice. we use it even to send us alerts though that's a bit unoptimal because it'll keep mailing every 5 minutes until the alrt value is below the treshold.15:40
sdezielhowever crappy NRPE is, it's pretty handy15:40
blackflowand custom plugins are dead easy to write, you can have them in anything. shell, perl, python, C, java, whatever.15:40
UssatWe are a mostly IBM shop so we have Tivoli monitoring for most things15:41
sdezielNRPE checks are also trivial to write15:41
blackflowsdeziel: munin is primarily to graph things, but we use it for alerts too as it can do alerts on value tresholds.15:41
sdezielblackflow: sounds like what netdata does which I'm more familiar with15:42
UssatMost of my AIX stuff is monitored with Tivoli, and we are getting more and more linux into Tivoli monitoring15:42
blackflowsdeziel: I'm not familiar with netdata, sorry15:42
sdezielblackflow: worth checking IMHO: https://my-netdata.io/#demosites15:43
blackflowsdeziel: huh real time streaming of data... interesting. sometimes I need that, and munin is limited to cron based invocations15:44
sdezielblackflow: the way multiple sites are aggregated is pretty nice as well15:45
sdezielit's decentralized by default and your browser is the one building the aggregated view15:45
blackflowI see.15:47
foo" < blackflow> and custom plugins are dead easy to write," - nice, I liked this about nagios... I wrote a few back in my day (~10 years ago)15:47
Ussatfoo something you may look at for that situation is nmon for linux15:48
Ussatfull disclosure, I know the author15:48
fooI had someone recently suggest librenms.org - for discovery + monitoring. *shrug* Wasn't my call, but curious to see how it performs15:48
foohaha, just pulled up home pages of zabbix, munin, and grafana ... munin is the least pretty to look at. Which probably means it was built by techs who have solid tech and don't care about eye candy corporate/enterprise-y stuff... I could be mistaken, but fun thought15:50
UssatI love that program15:50
fooUssat: huh, thanks, nmon looks cool. /me saves15:52
Ussatits VERY extensive15:52
UssatIt was origionally written for AIX and has been continusely improved...Nigel ported it to Linux recently but its a GREAT tool15:53
UssatI install it by default on all my builds15:53
UssatNIgel is a performance specialist for IBM15:55
neilduganI have a boot on a zfs system.. recently I have been having a problem with doing a "apt dist-upgrade" ... I keep getting a "grub-probe: error: failed to get canonical path of `rpool/ROOT/ubuntu'." ... does anyone know how to fix this?16:01
blackflowneildugan: _boot_ or _root_ on ZFS?16:02
blackflowlike /boot too?16:02
neilduganblackflow, both16:03
neildugan"grub-probe /" is the returning the error16:03
blackflowlooks like an open issue   https://github.com/zfsonlinux/grub/issues/516:04
blackflowpersonally I'm still under impression that grub ZFS support is not yet there.16:04
blackflowI run /boot separate on ext4 but that's primarily due to ZFS rootpool being LUKS'd16:05
neilduganblackflow, I wonder what changed recently to make this happen, though that is secondary to getting things working again16:12
blackflowneildugan: wouldn't know, really.16:14
neilduganblackflow, I tried a "grub-probe -vv /" I got a new error .. '/boot/grub/device.map': No such file or directory16:16
neilduganblackflow, should I generate one?16:17
blackflowneildugan: did you look at that bug report? there are some suggestions with env vars16:17
neilduganblackflow, yes I have, I have been reading it16:18
neilduganblackflow, I have found one that mentions zpool not being in the path... but on my system it is... I am reading further16:26
neilduganblackflow, thanks for the link, there are many options I should find something to that will work.16:59
BrianBlazehey sarnold I have gotten the application running! So happy thanks for the link for mysql :)19:39
BrianBlazeit only took a day to make happen lol19:39
sarnoldBrianBlaze: thanks for reporting back, it's great to hear you're up :)19:41
BrianBlazeI am so thankful to be on the latest version as it fixes a lot of issues we had19:41
sarnoldheh, given the fact that they wouldn't let you install on the newest mysql, somehow I'm not too surprised..19:43
sarnoldeven when it may be way better than it used to be, it still suggests a certain programming style :)19:43

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!