=== holoturoide is now known as hltrd
=== hltrd is now known as holoturoide
=== holoturoide is now known as drrzmr
=== drrzmr is now known as eder
=== eder is now known as drrzmr
=== drrzmr is now known as holoturoide
GoopHow do I install phpmyadmin without Apache? I have a server that MUST run on nginx only.05:50
andolGoop: Likely the same you would install any other phpmyadmin application under Nginx, using php-fpm. Also, a quick google search suggest that there are plenty of tutorials availible.05:58
Maximxxx100Ok interesting problem, one of my servers that I own has been comprised. It's making thousands of connections to one ip and port per second from random ports on my server. Now nothing about it is showing up on netstat, and nothing I can do shows what process or why this is happening. I've shut down every system process that is not necessary, and used ufw to block every in/out connection except SSH. And I tried blocking the ip06:47
Maximxxx100with iptables/ufw with no luck. It's still making connections no matter what I do. did a port scan on the ip and the ONLY open port is the one my server is making thousands of connections per second to located in Vietnam. Thanks.06:47
bhuddahMaximxxx100: backup all data.07:02
Maximxxx100I have bhuddah, I've had this particular server for 6 years without any problems. I would like to save it without going Nuclear and wiping with a new install, but I cannot stop my server from contacting this ip at all.07:05
bhuddahMaximxxx100: you can never ever save a compromised machine. you absolutely need to wipe it. repair is not an option. never.07:05
Maximxxx100I dont know if it's compromised for sure, it would be nice If I could find more information about it before. I wonder how it would get compromised in the first place. I've had unattended upgrades working perfectly, using only ssh keys and good passwords. And only used it to host a few files and private services for me.07:09
Maximxxx100and I always used user accounts for all services, yet the connections are coming from root. darn...07:10
Maximxxx100How did the sneaky Vietnam guy hack his way in is what I want to know.07:10
awalendeIs it possible, that the "out of memory killer" kicks in not when there is not enough ram, but too many pagetables are created?08:19
awalendeMy qemu vm keeps kicking the bucket, even tho I have still a bunch of memory left on the host08:19
cpaelzerawalende: maybe memory of one special kind is depleted like lowmem in the 32 bit past08:30
cpaelzerawalende: the oom should have put some output in your dmesg that might help08:30
cpaelzeryou could pastebinit to think about it together08:30
cpaelzersoemtimes knowing /proc/meminfo, /proc/pagetypeinfo can also help08:31
cpaelzerdepends on your actual case08:31
cpaelzerawalende: in general https://linux-mm.org/OOM has some more details and also a script to collect more data (I haven't tested/used that script, so it might need some polishing)08:33
awalendekern log: https://paste.ubuntu.com/p/BpSQ6ysxj4/08:54
awalendememinfo: https://paste.ubuntu.com/p/shg9q62ZbW/08:55
awalendepagetype info: https://paste.ubuntu.com/p/KxTfPKjqDg/08:56
cpaelzerther kern log should have ~15 more lines above that starting with Mem-Info08:57
awalendelike this? https://paste.ubuntu.com/p/xXsv88fMHd/09:00
cpaelzerawalende: without spending too much time I unfortunately also see no clear reason09:15
cpaelzerawalende: the allog being only order=0 from GFP_HIGHUSER_MOVABLE should succeed if your pagetypeinfo matches09:15
cpaelzerunless the pagetype info is e.g. form long before/after the actual issue09:16
cpaelzerwhich means whatever was depleated before isn't anymore when taking the data09:16
cpaelzerawalende: oh here we go09:16
cpaelzerawalende: in the moment you fail the high memory has only free:33316kB09:17
cpaelzerbut there is min:33320kB09:17
cpaelzerand the  GFP_HIGHUSER_MOVABLE can not tap on that reserve09:17
cpaelzerit might be that your overall free mem is on other nodes and/or other zones09:18
awalendemeh, then I probably have to bash my monitoring since it was reporting 110gb free mem on crash :x09:20
awalendebut thank you for lookin into it cpaelzer! I'll try to get a bigger grip on different memory sections on my server09:28
awalendecpaelzer , I believe to have found the cause now. Our QEMU VM has NUMA support enabled. I believe the seperation of memory banks can cause memory chokes depending on the load. I guess thats what you meant with "the free mem is on other nodes"?11:02
cpaelzerawalende: yes11:03
cpaelzerawalende: https://libvirt.org/formatdomain.html#elementsNUMATuning11:05
awalendeWe probably want the "preferred" mode here.11:34
UssatAnyone here run an ansible master on Ubuntu ? I assume you use the official ansible PPA's as listed here:  sudo apt-add-repository --yes --update ppa:ansible/ansible13:00
sdezielarchive.ubuntu.com is terribly slow (~15kB/s) from multiple locations13:15
lotuspsychjesdeziel: known issue @ the ubuntu-mirrors guys13:15
sdeziellotuspsychje: thanks :)13:15
lotuspsychje<moon127> tobikoch: we're aware, we had a large spike in traffic ~90 mins ago.  No sign that is anything but legitimate traffic so far, but we're pushing our transit to capacity at this time.13:15
sdezielI'll enjoy the dial-up experience in the meantime ;)13:16
lotuspsychjesdeziel: :p try sudo apt update perhaps13:16
Ussatok...so a new build of ubuntu.....: Err:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease13:56
Ussat  403  Forbidden [IP: 80]13:56
Ussatgetting a ton of those......13:57
dlloydyeah getting timeouts and sporadic failures for the aws us-east mirrors as well14:04
UssatWell.......have a few builds to do today...this puts a crimp in plans14:04
JanCmaybe ask in #canonical-sysadmin14:05
Ussatthis is not a good start to my day14:07
rbasakUssat: 403 seems odd. Are you sure you don't have something transparently MITMing?14:10
UssatOK, I was wrong, transparent proxy on this network, talking to my network team now rbasak14:26
rbasakUssat: #canonical-sysadmin confirmed a known issue. Perhaps your MITM is transforming the known issue into a 403?14:37
Ussatrbasak, yup...working with my networking team now14:37
Ussatits a monday here :(14:38
Ussatis there a someplace I can see all the IP's used for updates at canonical and ubuntu ? or a range, I need them for my prox16:15
=== Erich is now known as Eickmeyer
tewardsarnold: i maaaaay have found a bug in using umt on a later release :/21:01
JanCUssat: doesn't seem like a good idea to hardcode that21:37
UssatYa its fixed21:38
JanCimagine if they would decide to dynamically add cloud instances or something like that, there is no way you could keep a list of "known download servers" up-to-date...21:39
UssatJanC, ya its been fixed here21:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!