| trippeh_ | seems it was just bad memory. didnt trip any ECC tho. | 01:44 |
|---|---|---|
| sarnold | o_O | 01:45 |
| === inteus_ is now known as inteus | ||
| cpaelzer | good morning | 04:18 |
| cpaelzer | rbasak: nacc: ok, we will see if next time things are similar | 04:19 |
| cpaelzer | I'm ok if the hickups this time were due to the re-import | 04:19 |
| lordievader | Good morning | 06:21 |
| === mundus2018 is now known as mundus | ||
| === giraffe is now known as Guest18867 | ||
| sruli | anyone tried running ubuntu-server from usb stick in production? i have HP DL320 8x256GB SSD RAID 10, does it make sense to run the OS from internal slot, will it effect the performance of kvm vm's on the main raid? | 10:41 |
| blackflow | sruli: I haven't tried, but in theory I don't see anything wrong with that, assuming you treat the USB installation as readonly root, and mount writable paths from faster media, like the "main raid" you mention. | 10:45 |
| sruli | blackflow: i understand, on the flip side if i install the os on main raid will that have any effect on performance of kvm? | 10:46 |
| blackflow | sruli: again depends where you mount writable paths, but it'd probably be insignificant, unless you have any high volume logging going on or something like that. | 10:47 |
| sruli | blackflow: not show how to mount root as read only and which path i need to make to main raid | 10:48 |
| blackflow | if I were you, I'd consider using ZFS and put VMs on dedicated zvols. | 10:49 |
| sruli | blackflow: i ran some bonnie++ test with the os install on main raid vs on usb, see the results here, https://www.saurymper.com/pages/bonnie-results-2018-05-31.html see Set 1 & 3 tests | 10:49 |
| sruli | blackflow: i will put vm's on seperate LV's, i want to stay with raid 10, i am not sure if zfs can surpass the performance of dedicated raid card (p420) | 10:51 |
| blackflow | sruli: those numbers don't mean much to me, it's a synthetic benchmark. | 10:51 |
| blackflow | sruli: it might, depending on the size of hot data and amount of RAM | 10:51 |
| blackflow | but you also get all the goodies of ZFS: checksumming, snapshots, easy backups, ARC/L2ARC and in case of write heavy applications, ZIL | 10:54 |
| sruli | blackflow: to use that i would need to delete the raid set and expose the drives directly to os, right? | 10:55 |
| blackflow | and compression. Just as an example, I only use ZFS, exclusively on the servers. All the hot data (actively accessed) fits in RAM so ZFS' MFU (Most Frequently Used) caches are reigning supreme, while at the same time all the data is checksummed, auto-healing raid6 (raidz2), and backups are a snapshot + zfs send|recv away. | 10:56 |
| blackflow | sruli: yes. ZFS works best if it uses drives directly. | 10:56 |
| blackflow | if you're not familiar with ZFS, I wouldn't recommend you going straight to production with it. I just meant to say consider it, toy with it, learn it, because in the long run you'll reap huge benefits. | 10:57 |
| sruli | blackflow: i have 32GB ram, i guess zfs raid 10 would be the fastest? | 10:57 |
| sruli | blackflow: been using zfs z2 on my workstation for 3-4 years, installed and forgot about it (its a storage pool, os is on ssd) | 10:58 |
| blackflow | depends on the workload. but in general I'd say yes, raid10 -- or striped mirror as it is in ZFS -- should be very fast. | 10:58 |
| sruli | blackflow: will try it and run a bonnie test, maybe someone will be able to decipher those tests and see which is best option | 10:59 |
| blackflow | bonnie wont help you there. ZFS MFU is built over time and the true performance benefits come from L2ARC and ZIL -- unless you're already using ssd/nvme for disks | 11:00 |
| sruli | blackflow: its a new install no existing data, using consumer ssd's (hp s700 pro) | 11:07 |
| blackflow | sruli: then it boils down to if hot data fits in RAM. if not, ZFS will not be performant (but will be everything else). | 11:11 |
| sruli | blackflow: all the files are small (except maybe for the actual os installs on the VM's) and a few 2-300mb files | 11:17 |
| blackflow | sruli: that doesn't mean anything. it all depends on how they're used, especially if frequently used blocks can fit in RAM / can be identified by ZFS MFU. | 11:18 |
| sruli | blackflow: how do i figure that out? | 11:18 |
| blackflow | sruli: you don't, in advance. You have to put it in production and see, or have eexperience from previous such deployments. | 11:41 |
| Buoy172 | I have ubuntu 16.04 with caddy. I just installed php7.2 on it. I created a test page /var/www/index.php and its contents: https://pastebin.com/V6pLVMuE | 11:57 |
| Buoy172 | The output in the browser is the same as the contents. | 11:57 |
| Buoy172 | Not just "Hello World" | 11:57 |
| Buoy172 | What is the problem? | 11:57 |
| blackflow | Buoy172: you haven't configured the web server to interpret the .php scripts with the PHP interpreter | 11:59 |
| sandstrom | On Ubuntu 18.04, how can I let dnsmasq know about the upstream (dhcp-provided) DNS server? | 12:00 |
| sandstrom | In NetworkManager there was a specific setting for dnsmasq, but NetworkManager doesn't seem to be used in 18.04. | 12:00 |
| blackflow | sandstrom: NM is used, on desktop, but abstracted with netplan. It's still full networking authority in case of dhcp setups via netplan. Which means, NM + dhcp should set up DNS servers from dhcp data. but, irrc, dnsmasq is not default, so you must've set that up? | 12:02 |
| blackflow | sandstrom: anyway, to config dnsmasq under NM, see this, see paragraphs for "dns" setting: https://developer.gnome.org/NetworkManager/stable/NetworkManager.conf.html | 12:03 |
| sandstrom | blackflow this is an ubuntu server (if that matters), yes I've setup dnsmasq manually. | 12:04 |
| blackflow | oh yes, the channel is -server :) well, then the networking authority is networkd, for dhcp setups. otherwise you need full network config via netplan itself. | 12:05 |
| blackflow | or ignore netplan and configure networkd directly. | 12:05 |
| sandstrom | blackflow: when systemd is the authority, is it still possible to have it 'tell' dnsmasq about the upstream dns server? | 12:05 |
| Buoy172 | How do I configure the web server to interpret php? | 12:06 |
| blackflow | sandstrom: not sure. by default it wants to use systemd-resolved, and personally I have very little experience with dnsmasq, I use bind (and mask out resolved completely). | 12:06 |
| sandstrom | I haven't found anything similar to what you quoted in the man pages for 'systemd.network' | 12:07 |
| sandstrom | blackflow I wouldn't mind using bind either. My problem is that I want to forward dns queries under one special domain (.example) to a local dns server (say, 127.0.0.2). | 12:08 |
| blackflow | Buoy172: for example https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-in-ubuntu-16-04 | 12:08 |
| sandstrom | This doesn't seem to be possible under systemd-resolved, so I'm thinking that I need to configure systemd-resolved to pass all traffic to dnsmasq/bind, and then have dnsmasq/bind resolve with upstream (dhcp provided DNS) for everything except those under the special domain. | 12:09 |
| Buoy172 | blackflow: I don't need instructions for nginx. I have caddy. | 12:09 |
| blackflow | Buoy172: then look up caddy documentation on how to set up php-fpm fastcgi. | 12:09 |
| blackflow | sandstrom: nah, just drop resolved completely | 12:10 |
| Buoy172 | I'm looking... :) | 12:10 |
| blackflow | sandstrom: that way you have full control over /etc/resolv.conf and can do full DNS configuration with bind. eg, make it authoritative for .example, and pass through everything else to an upstream resolver, OR query the roots directly (my preference) | 12:10 |
| sandstrom | blackflow Agree, that's what I want | 12:11 |
| sandstrom | but I don't know how to let bind know about the dhcp provided dns server | 12:11 |
| blackflow | ah, I see what you mean | 12:12 |
| sandstrom | (I've figured out how to take back control over /etc/resolv.conf from systemd-resolved (read in the man pages), but don't know how I'll have systemd-networkd inform bind (or dnsmasq) about the upstream DNS servers (received via DHCP). I guess I could hardcode 8.8.8.8 or similar, but would like to avoid that if possible | 12:14 |
| blackflow | well I'm not sure, I'd have to google it, but it'd probably involve dhclient or dhcpcd integrating with bind/named somehow. | 12:15 |
| blackflow | eg, NetworkManager can integrate with unbound via hook scripts, and same principle could be used for bind, but... that's network manager. | 12:16 |
| sandstrom | alright, those are good pointers, I'll investigate further! Thanks a lot for your help! | 12:17 |
| blackflow | dhclient has such hook scripts too iirc | 12:17 |
| Buoy172 | blackflow: now the url outputs "502 bad gateway". Do I need to install php-fpm? | 12:25 |
| blackflow | Buoy172: yes, if you're configuring the web server to use php-fpm via fastcgi. | 12:26 |
| blackflow | and oyu should, that's pretty much a standard setup nowadays. PHP under fastcgi with php-fpm as the process manager for it. | 12:26 |
| Buoy172 | I installed php with this command: "apt-get install php7.2" | 12:28 |
| blackflow | Buoy172: I believe that pulls in php7.2-fpm | 12:28 |
| Buoy172 | I looked up that "dpkg --get-selections | grep php" lists all the installed php packages | 12:29 |
| Buoy172 | and here they are: https://pastebin.com/tP4dEetQ | 12:29 |
| Buoy172 | Does this mean I don't have php-fpm? | 12:29 |
| blackflow | Buoy172: yes, install php7.2-fpm | 12:30 |
| blackflow | Buoy172: ah yeah, it pulled in the libapache thing, I see the dependency is that libapache thing, OR fpm, OR cgi. | 12:31 |
| blackflow | libapache thing = libapache2-mod-php7.2, aka the mod_php for apache | 12:31 |
| sandstrom | Buoy172 tried any of http://oonlab.com/code/2016/08/12/setup-caddy-on-ubuntu-to-work-with-php/ https://www.digitalocean.com/community/tutorials/how-to-host-a-website-with-caddy-on-ubuntu-16-04 https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-caddy-on-ubuntu-16-04 | 12:33 |
| Buoy172 | sandstrom: thanks, I'll check those links out | 12:33 |
| sandstrom | top hits on google | 12:34 |
| Buoy172 | all right, managed to setup php! | 12:36 |
| Buoy172 | Thanks guys. I might have some more questions later. Please bear with me, I'm a beginner ;) | 12:37 |
| lordievader | Nothing wrong with that 😉 | 12:44 |
| coreycb | frickler: would you be able to respond to rbasak's question on the dynamic-routing bug? | 14:17 |
| frickler | coreycb: oh, I had interpreted that at being directed towards thedac who did the Ubuntu testing. I'll add a comment | 15:31 |
| coreycb | frickler: well possibly, i thought you tagged it but anyone will do if they tested | 15:33 |
| coreycb | thanks frickler | 15:34 |
| nacc | rbasak: re: ntp merge, have a sec? | 16:22 |
| rbasak | nacc: yes | 16:23 |
| rbasak | nacc: here? Or HO/Meet? | 16:23 |
| nacc | rbasak: whatever you prefer | 16:24 |
| rbasak | nacc: I sent you a link | 16:25 |
| rbasak | (DM) | 16:25 |
| teward | ahasenack: any idea why an autopkgtest inside snapped LXD doesn't have any netowrking connection? (It's like it's not requesting an IP...) | 17:01 |
| SSMAdmin | Hi all. Anyone else had their 18.04 apache server suddenly stopp working? Error.log only says "Out of Memory", apache2ctl -S" complain about a missing or empty Letsencrypt certificate (which actually IS there and is NOT empty), and "sudo apache2 -X -f /etc/apache2/apache2.conf" complain about Config variable ${APACHE_RUN_DIR} is not defined. I myself have not made any changes that should create these errors and apache | 17:31 |
| SSMAdmin | stop working. So I wonder what could have happened, and how to fix it. Any help?" | 17:31 |
| dpb1 | yuck | 17:33 |
| dpb1 | how much memory does your system have | 17:33 |
| SSMAdmin | 4G | 17:34 |
| SSMAdmin | and there is nothing else running on the machine, exerpt the kid wathcing netflix | 17:35 |
| SSMAdmin | apache is serving nextcloud, but there is no traffic there, only private cloud (which is now not working as apache refuse to run) | 17:36 |
| SSMAdmin | To me it looks like some system update have messed up things, as the sound settings on the desktop was also suddenly changed/reset, so I had to go in desktop system tools area to configure sound output once more to get the sound back. Strange things have happened. And I know for sure the box is not hacked, and my 6 year old is not capable of messing with system settings | 17:42 |
| matjam | SSMAdmin: I didn't realise this channel existed | 17:46 |
| matjam | so, thanks | 17:46 |
| yeats | SSMAdmin: could be that the server ran out of RAM for non-apache reasons and apache was just collateral damage | 17:46 |
| matjam | SSMAdmin: I've seen out of memory stuff in apache when using something like PHP and you increase the upload limit to something silly like 2GB and you don't have a small VM | 17:47 |
| matjam | *big vm | 17:47 |
| matjam | SSMAdmin: the certificate thing is weird too | 17:47 |
| matjam | SSMAdmin: you should be able to change the systemd policy to restart apache in any condition | 17:47 |
| matjam | or switch to nginx because it's better | 17:48 |
| * matjam ducks | 17:48 | |
| SSMAdmin | I dont think the server ran out of RAM, as it is only apache that runs most of the time. Kid log in and watch some netflix before bedtime, but else there is nothing running | 17:51 |
| SSMAdmin | Beside, it has worked in this type of configuration for years, then I upgraded to 18.04 i April, and it has been running without error since...until these strange things happened yesterday | 17:54 |
| matjam | well basically you should go through the logs | 17:56 |
| matjam | dmesg, syslog etc | 17:56 |
| matjam | look at the times around when the problem happened | 17:56 |
| matjam | also the apache2ctl -S command might complain if you did not run it as root and the certificate permissions are set such that your user can't read the files that are referred to in the configuration in apache | 17:57 |
| matjam | and running the apache2 command directly would complain as there is an environment variable set in the startup script for the apache daemon | 17:57 |
| matjam | so I don't think either of those things are symptomatic | 17:58 |
| matjam | SSMAdmin: also, are you running the official apache package from the ubuntu repos or did you build from source? Are you using PHP? any other deails you can give about the system? | 17:58 |
| matjam | "sudo apt install inxi pastebinit" and then "sudo inxi -Fr -c0 -z | pastebinit" and sharing the URL that it prints out might be helpful for us to get some context. | 17:59 |
| matjam | SSMAdmin: did you get the last thing I said? | 18:02 |
| pavlos | SSMAdmin: matjam I'm here too | 18:03 |
| SSMAdmin | matjam https://paste.ubuntu.com/p/f2BxZdNnXx/ | 18:04 |
| matjam | :thumpsup: | 18:04 |
| matjam | SSMAdmin: and you just used the Apache package in the Ubuntu repo right? | 18:05 |
| SSMAdmin | yes | 18:05 |
| matjam | SSMAdmin: whats the apache serving? A php app? | 18:12 |
| matjam | SSMAdmin: is it open to the internet? | 18:12 |
| SSMAdmin | matjam apache is serving nextcloud, and it is open to the internet, it has been working since April when I installed 18.04 and nextcloud and configured the system. Errors and strange behavior occured yesterday | 18:13 |
| matjam | SSMAdmin: it may be someone was running an attack against your server and managed to trigger a bug to consume memory and DoS your server. | 18:19 |
| matjam | nextcloud is written in PHP, so the code is executed in the apache process | 18:19 |
| matjam | which means if there's a bug in nextcloud or in php, it would behave like that | 18:20 |
| matjam | I've seen that behavior before when I've had a server being actively attacked, and I was running an older version of the app or PHP | 18:20 |
| matjam | make sure you've got libapache2-mod-php7.2/bionic-updates,bionic-security 7.2.5-0ubuntu0.18.04.1 amd64 isntalled not the older libapache2-mod-php/bionic,bionic 1:7.2+60ubuntu1 all | 18:22 |
| matjam | make sure nextcloud is updated | 18:22 |
| matjam | if you still have problems with it, I'd look at using nginx + php-fpm which is the fastcgi runner for php apps, and that runs your PHP application in a separate process - so at worst it will take out the PHP environment but not nginx. You can also use the same model with apache - it also supports fastcgi | 18:24 |
| matjam | SSMAdmin: I'd check the apache logs and see what was hitting your server on the lead up to the crash, I would suspect that there's lots of weird requests. | 18:24 |
| SSMAdmin | matjam thankyou. That is a lot to wrap my head around, but I'll try. I am not linux-trained, so only have basic commandline skills | 18:32 |
| SSMAdmin | I will read up about your suggestions | 18:33 |
| matjam | SSMAdmin: yeah, you should be aware that there's lots of automated scripts out there trying to find their way into whatever they can, you'll probably find lots of random accesses in your apache logs, thats usually not people thats some automated script | 18:33 |
| matjam | and they try like, every exploit they can against the server, and this will sometimes crash things | 18:33 |
| SSMAdmin | yes, I saw huge logs, so there is a lot in it. what is the command in nano to get to the last line of the log? | 18:34 |
| matjam | uh, shift pg-down I think? | 18:34 |
| matjam | I usually use less to view logs though | 18:35 |
| matjam | no chance I will accidentially edit the file | 18:35 |
| SSMAdmin | shift pg-down did not work | 18:36 |
| SSMAdmin | how to use less? | 18:36 |
| matjam | man less will give you all the commands | 18:36 |
| matjam | "man less" | 18:36 |
| SSMAdmin | thanks | 18:36 |
| dlloyd | last line is > | 18:36 |
| matjam | I typically do shift-F and then ctl-c to get to the bottom lol | 18:37 |
| matjam | there ya go | 18:37 |
| matjam | update-alternatives --set editor vim :P | 18:37 |
| SSMAdmin | dlloyd > alone did not work, shift > or ctrl > did not work either | 18:38 |
| dlloyd | uh, sure you are in less? | 18:39 |
| dlloyd | G should do eof too | 18:40 |
| SSMAdmin | dlloyd, sorry, I thought you ment nano | 18:42 |
| SSMAdmin | matjam, looked in access.log. There is nothing there indicating some attack. Last lines indicate working apache serving nextcloud connections from known clients. Then traffic just stops, no more entries in access.log | 18:48 |
| matjam | then may just have been a bug in nextcloud | 18:49 |
| matjam | above advice still holds | 18:49 |
| SSMAdmin | ok, i try deactivating nextcloud, and activate 000-default.conf and see what happens | 18:50 |
| SSMAdmin | well, that did not help, apache not running | 18:55 |
| matjam | you probably have an invalid apache configuration, so become root with "sudo -i" and then run apachectl configtest | 18:57 |
| SSMAdmin | Syntax OK | 19:00 |
| SSMAdmin | apache2 -X -f /etc/apache2/apache2.conf | 19:03 |
| SSMAdmin | [Fri Jun 01 21:02:09.055322 2018] [core:warn] [pid 27409] AH00111: Config variable ${APACHE_RUN_DIR} is not defined | 19:03 |
| SSMAdmin | apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot | 19:03 |
| SSMAdmin | This is a strange thing, as I have not touched this setting, so why should it suddenly complain about that variable? | 19:04 |
| SSMAdmin | In /etc/apache2/envvars it says "export APACHE_RUN_DIR=/var/run/apache2$SUFFIX". I have checked that /var/run/apache2 dir actually exist | 19:07 |
| SSMAdmin | So apache is complaining about a non-existing directory that actually exist | 19:09 |
| SSMAdmin | Similar to it complaining about a non-existing or empty letsencrypt certificat, that actually exist, and is NOT empty, its content is as originally created by certbot | 19:10 |
| SSMAdmin | have to put kid to bed, be back later | 19:10 |
| matjam | you can't start apache like that | 19:12 |
| matjam | why do you keep doing that .. | 19:12 |
| nacc | SSMAdmin: you aren't using the service file, which probably does other things, like ensure envvars is sourced. | 19:15 |
| SSMAdmin | matjam, I dont "keep doing that". I run some command I have found on the internet, from posts discussing similar problems, in order to find out what is wrong. I dont usually run apache manually | 19:43 |
| SSMAdmin | nacc you are probably correct, but I am just a newbie, so I dont know. Could you please tell me how to do this correct then? | 19:44 |
| matjam | systemctl start apache | 19:48 |
| matjam | look at error_log | 19:48 |
| SSMAdmin | yes I know, and I have tried it, nothing happens, not even an error message. The only thing that then happens in error.log is a new entry of "Out of Memory" | 19:49 |
| matjam | sudo systemctl status apache | pastebinit | 19:49 |
| matjam | also journalctl -u apache will give you what systemctl knows | 19:50 |
| SSMAdmin | systemctl status apache | pastebinit | 19:51 |
| SSMAdmin | Unit apache.service could not be found. | 19:51 |
| SSMAdmin | Du prøver å sende et tomt dokument. Avslutter. | 19:51 |
| matjam | its probably apache2 | 19:52 |
| SSMAdmin | then | 19:52 |
| SSMAdmin | journalctl -u apache | 19:52 |
| SSMAdmin | -- Logs begin at Sun 2018-04-29 19:26:53 CEST, end at Fri 2018-06-01 21:51:11 CEST. -- | 19:52 |
| SSMAdmin | -- No entries -- | 19:52 |
| SSMAdmin | apache2 yes, sorry, I try again | 19:52 |
| sarnold | my favourite part is how the tools don't give you any feedback that the unit name is unknown | 19:52 |
| sarnold | "lol here's the non-existant logs in that time frame!" | 19:53 |
| matjam | yeah it is idiotic | 19:53 |
| matjam | silent on success, print something on failure, thats the unix way | 19:54 |
| matjam | so service name is apache2 | 19:54 |
| matjam | sorry | 19:54 |
| matjam | I use nginx | 19:54 |
| SSMAdmin | Status apache2: https://paste.ubuntu.com/p/N22kwvvr9K/ | 19:54 |
| matjam | SSMAdmin: cat /etc/apache2/mods-enabled/mpm_prefork.conf | pastebinit | 19:55 |
| matjam | also cat /etc/apache2/apache2.conf | pastebinit | 19:57 |
| SSMAdmin | journalctl -u apache2: https://paste.ubuntu.com/p/cDx5t8shFM/ | 19:57 |
| matjam | SSMAdmin: also cat /etc/php/7.2/apache2/php.ini | 20:00 |
| SSMAdmin | apache2.conf: https://paste.ubuntu.com/p/ckJRxPfT8W/ | 20:00 |
| matjam | | pastebinit | 20:00 |
| SSMAdmin | php.ini: http://paste.ubuntu.com/p/B8FQbbkP6G/ | 20:01 |
| matjam | also, cat /var/log/apache2/error.log | pastebinit | 20:01 |
| SSMAdmin | error.log: http://paste.ubuntu.com/p/CWFGdVdQnY/ | 20:03 |
| sdeziel | this also work: "pastebinit /var/log/apache2/error.log" | 20:03 |
| sdeziel | cat doesn't hurt but I find it cool that pastebinit can eat files directly | 20:04 |
| SSMAdmin | thanks sdeziel | 20:04 |
| sarnold | Out of memory | 20:04 |
| sarnold | uhhhh | 20:04 |
| matjam | oh thats nice | 20:04 |
| matjam | SSMAdmin: did you increase php memory limit to 256M? | 20:04 |
| matjam | memory_limit | 20:05 |
| matjam | I'm assuming when you were setting up the app | 20:05 |
| SSMAdmin | yes I may have done that when configuring nextcloud settings | 20:05 |
| sdeziel | I find it odd that no timestamp is prepended to those messages | 20:07 |
| matjam | can you do, find /etc/apache2 -ls | pastebinit | 20:07 |
| matjam | sdeziel: I think it's the php module | 20:07 |
| matjam | sdeziel: my theory is we need to increase memory available to php in apache2 configuration | 20:07 |
| matjam | just trying to confirm there is no limts being set elsewhere | 20:07 |
| matjam | https://httpd.apache.org/docs/2.4/mod/core.html#rlimitmem | 20:08 |
| sdeziel | matjam: wouldn't that be in /etc/php/7.?/apache2/ then? | 20:08 |
| matjam | SSMAdmin: also confirm that grep -ir RLimitMEM /etc/apache2 returns nothing | 20:08 |
| SSMAdmin | matjam http://paste.ubuntu.com/p/CbjD3qcpNv/ | 20:08 |
| matjam | sdeziel: the apache directives should be in the apache conf.avail etc dirs | 20:09 |
| SSMAdmin | grep -ir RLimitMEM /etc/apache2 returns nothing | 20:09 |
| SSMAdmin | grep: returns: Fila eller mappa finnes ikke | 20:09 |
| SSMAdmin | No file or directory (translated from norwegian) | 20:10 |
| matjam | ok | 20:10 |
| matjam | lol yeah | 20:10 |
| matjam | just "grep -ir RLimitMEM /etc/apache2" | 20:10 |
| matjam | should return nothing at all | 20:10 |
| SSMAdmin | correct, last command did not return anything at all | 20:11 |
| SSMAdmin | lol, I see now | 20:12 |
| SSMAdmin | grep -ir RLimitMEM /etc/apache2 RETURNS NOTHING :-) | 20:12 |
| matjam | lol | 20:12 |
| matjam | sorry I should have quoted it | 20:12 |
| matjam | so this won't be a fix but if it works then it will confirm my theory | 20:13 |
| matjam | add "RLimitMEM max" to /etc/apache2/mods-enabled/php7.2.conf | 20:13 |
| matjam | at the bottom is fine | 20:14 |
| matjam | and then try "systemctl start apache2" | 20:14 |
| matjam | and then systemctl status apache2 | 20:14 |
| matjam | to see that it's running or not | 20:14 |
| SSMAdmin | Nope, refuse to run | 20:16 |
| matjam | hmm | 20:16 |
| matjam | grep -r www-data /etc | pastebinit | 20:18 |
| SSMAdmin | error.log just got a new entry of "Out of Memory", thats all | 20:18 |
| matjam | yeah | 20:18 |
| matjam | also modify memory_limit in php.ini to 128M temporarily to see if it will start | 20:19 |
| matjam | its really odd | 20:19 |
| matjam | /etc/php/7.2/apache2/php.ini | 20:19 |
| matjam | oh before you do that | 20:20 |
| matjam | have you rebooted this machine at all? | 20:21 |
| matjam | like, since yesterday | 20:21 |
| SSMAdmin | grep www-data: http://paste.ubuntu.com/p/jhjkqVqCDq/ | 20:21 |
| SSMAdmin | yes, rebooted several times | 20:22 |
| matjam | pastebinit /etc/systemd/system/multi-user.target.wants/apache2.service | 20:23 |
| SSMAdmin | changing memory-limit to 128M did not work | 20:24 |
| matjam | yeah ok | 20:24 |
| matjam | so you upgraded from 16.04 to 18.04? | 20:25 |
| SSMAdmin | http://paste.ubuntu.com/p/HGNyWyy9Hm/ | 20:25 |
| SSMAdmin | no, I installed fresh 18.04 | 20:25 |
| matjam | ah ok | 20:25 |
| matjam | did you make any other modifications to any other system files? | 20:26 |
| matjam | like /etc/sysctl.conf etc | 20:26 |
| SSMAdmin | previous system was 16.04 | 20:26 |
| SSMAdmin | no | 20:26 |
| matjam | so | 20:26 |
| SSMAdmin | what about uninstalling apache2 and reinstall it? | 20:27 |
| matjam | can you "cat /etc/apache2/sites-enabled/* | pastebinit" | 20:27 |
| matjam | well its one thing to try | 20:27 |
| matjam | we're not there yet | 20:27 |
| matjam | its something specific to your configuration, as I have just installed apache 2 & php etc and its working fine | 20:28 |
| matjam | and like, its something that if it was broken, lots of people would have been having problems | 20:29 |
| matjam | well that looks fine | 20:30 |
| SSMAdmin | could it be a harddrive error, some sector damaged, that holds information apache2 needs? | 20:31 |
| matjam | you'd see something in dmesg | 20:32 |
| matjam | or /var/log/syslog | 20:32 |
| matjam | sense error stuff like that | 20:32 |
| matjam | unlikely though | 20:32 |
| matjam | pastebinit /etc/apache2/envvars | 20:33 |
| SSMAdmin | http://paste.ubuntu.com/p/X288S9rgpc/ | 20:34 |
| matjam | seems fine | 20:35 |
| matjam | weeeeeeird | 20:35 |
| matjam | ok so this will suck | 20:36 |
| matjam | sec | 20:36 |
| matjam | I need to test locally | 20:36 |
| SSMAdmin | ok | 20:36 |
| sdeziel | if you suspect a mod_php issue I'd a2dismod it as a test | 20:37 |
| matjam | good point | 20:37 |
| matjam | do that | 20:37 |
| SSMAdmin | ok, a2dismod what? | 20:37 |
| matjam | a2dismod php7.2 | 20:38 |
| matjam | then start the server again with systemctl start apache2 | 20:38 |
| SSMAdmin | apache2 running | 20:39 |
| matjam | systemctl status apache2 | 20:39 |
| SSMAdmin | yes, its running | 20:39 |
| matjam | if it's running then, stop it with systemctl stop apache2 | 20:39 |
| matjam | yeah ok it's php | 20:39 |
| SSMAdmin | done | 20:39 |
| matjam | a2enmod php7.2 | 20:39 |
| SSMAdmin | ok, then problem is back | 20:40 |
| matjam | SSMAdmin: cd /etc/apache2 | 20:43 |
| matjam | source envvars | 20:43 |
| matjam | apache2 -X | 20:43 |
| matjam | as root | 20:43 |
| matjam | does it print anything useful> | 20:45 |
| SSMAdmin | ok, nothing happend. Did not print anything useful. "systemctl status apache2" gave same error message as earlier | 20:45 |
| matjam | yeah it will | 20:46 |
| matjam | when you say, nothing happened | 20:46 |
| matjam | it just returns straight back to the prompt? | 20:46 |
| SSMAdmin | yes | 20:46 |
| matjam | ok so as root, in /etc/apache2 | 20:46 |
| SSMAdmin | as root, yes | 20:47 |
| matjam | strace -f -o /tmp/apache2.strace apache2 -X | 20:47 |
| SSMAdmin | in /etc/apache2 | 20:47 |
| matjam | then patebinit /tmp/apache2.strace | 20:47 |
| matjam | ugh | 20:47 |
| matjam | pastebinit /tmp/apache2.strace | 20:47 |
| SSMAdmin | http://paste.ubuntu.com/p/5hqsrZdWBP/ | 20:49 |
| matjam | well thats useful | 20:50 |
| matjam | how much swap on this system? | 20:50 |
| matjam | "free" will tell you | 20:52 |
| matjam | free | pastebinit | 20:52 |
| matjam | also df -h | pastebinit | 20:52 |
| SSMAdmin | Swap: 999420 0 999420 | 20:52 |
| SSMAdmin | http://paste.ubuntu.com/p/dGhMcGzsBD/ | 20:53 |
| SSMAdmin | https://paste.ubuntu.com/p/S33jpR2QgG/ | 20:54 |
| matjam | so, system has 4GB ram, <1GB swap | 20:55 |
| sdeziel | the reading pattern of /dev/urandom is weird, 8 bytes at the time | 20:55 |
| matjam | that *should* be ok | 20:55 |
| matjam | sdeziel: my apache2 does that | 20:55 |
| matjam | seems "normal" | 20:55 |
| matjam | so like | 20:56 |
| matjam | mmap(NULL, 93874405974016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) | 20:56 |
| matjam | that number 93874405974016 | 20:56 |
| matjam | is the number of bytes its trying to allocate | 20:56 |
| sarnold | what's 85 TB here or there? :) | 20:57 |
| nacc | i'm 99% sure that this a local configuration issue | 20:57 |
| matjam | yeah | 20:58 |
| matjam | SSMAdmin: can you a2dismod ssl | 20:58 |
| sdeziel | could it be RLimitMEM max? | 20:58 |
| matjam | SSMAdmin: then try to start apache with systemctl start apache2 | 20:58 |
| matjam | sdeziel: tried that | 20:59 |
| matjam | sdeziel: as in, it had no RlimitMEM statement, we added one | 20:59 |
| SSMAdmin | a2dismod ssl done, still same error | 20:59 |
| sdeziel | matjam: right but I'd remove it | 20:59 |
| SSMAdmin | apache2 will not start | 20:59 |
| matjam | sure | 20:59 |
| matjam | SSMAdmin: can you remove the RLimitMEM entry from the php conf we added before | 21:00 |
| matjam | oh hang on | 21:00 |
| matjam | SSMAdmin: can you "cat /etc/apache2/mods-enabled/* | pastebinit" | 21:01 |
| matjam | I like the modularity of the debian config for apache | 21:01 |
| matjam | but man, it scatters the config all over the place | 21:01 |
| SSMAdmin | matjam, removed RLimitMEM, still same error | 21:02 |
| matjam | yeah figured | 21:02 |
| sdeziel | this would provide a fuller view: "grep . /etc/apache2/*-enabled/* | pastebinit" | 21:02 |
| SSMAdmin | https://paste.ubuntu.com/p/QP36brRfSD/ | 21:03 |
| SSMAdmin | http://paste.ubuntu.com/p/dVNGbMcwn3/ | 21:04 |
| matjam | sdeziel: mr fancy | 21:04 |
| sdeziel | matjam: hehe :) | 21:04 |
| sdeziel | the split out apache2 config was one of the key factor in me adopting Ubuntu years ago. I have now moved to nginx when feasible though | 21:05 |
| matjam | yeah | 21:05 |
| matjam | I like using fastcgi | 21:05 |
| matjam | problems with php don't take out the entire webserver | 21:05 |
| matjam | and it performs better | 21:06 |
| matjam | I dunno man, this all looks boringly normal | 21:06 |
| matjam | other than the weird mmap for 85TB or fram | 21:06 |
| matjam | of ranm | 21:06 |
| matjam | SSMAdmin: can you please "dpkg --list | grep php | pastebinit" | 21:07 |
| matjam | if the output of that is boringly normal | 21:07 |
| matjam | then I'm officially throwing in the towel | 21:07 |
| SSMAdmin | https://paste.ubuntu.com/p/XfKBbwf5gM/ | 21:08 |
| nacc | are we not getting an error from apache2 at all? | 21:08 |
| matjam | no I don't think its getting that far | 21:08 |
| matjam | I think its loading modules and php prints to stderr and that goes to error.log | 21:09 |
| matjam | so on my system I don't have libapache2-mod-php, just libapache2-mod-php7.2 | 21:09 |
| nacc | well, you *should* have mod-php :) | 21:10 |
| nacc | it's a metapackage to track updates | 21:10 |
| nacc | *upgrades | 21:10 |
| matjam | ah | 21:11 |
| * matjam throws in the towel | 21:11 | |
| matjam | SSMAdmin: sorry dude, I'm at a loss | 21:11 |
| matjam | SSMAdmin: you've hit some weird bug in php | 21:11 |
| sdeziel | SSMAdmin: any idea of what changed since it all started? | 21:12 |
| nacc | why do you think it's in php? | 21:12 |
| sdeziel | nacc: a2dismod php7.2 makes the issue go away | 21:12 |
| nacc | ah, that was what i was missing | 21:12 |
| sdeziel | SSMAdmin: you could try to disable every PHP modules and selectively reintroduce them one by one | 21:13 |
| nacc | has a purge of php (or at least libapache2-mod-php*) and reinstall been tried? | 21:13 |
| SSMAdmin | matjam, thank you for trying and spending a lot of time on it. Ok...its a php bug, but then it must be something from the ubuntu repos, because I have not changed anything myself. I has been running without error since april when I installed 18.04, suddenly yesterday it stopped working | 21:13 |
| sdeziel | SSMAdmin: could you share /var/log/apt/history.log ? | 21:14 |
| nacc | SSMAdmin: any clues if a packge update happened? /var/log/apt/history.log | 21:14 |
| SSMAdmin | apt history: http://paste.ubuntu.com/p/vdXWmG4PHR/ | 21:18 |
| sdeziel | nothing of interest in there | 21:19 |
| SSMAdmin | I have no idea what updates have been made by the system automatically. I did a manual update earlier today, in order to see if that would fix anything, but it didnt | 21:19 |
| sdeziel | zcat /var/log/apt/history.log.1.gz | pastebinit | 21:19 |
| SSMAdmin | sdeziel, https://paste.ubuntu.com/p/FTwcz38XBj/ | 21:21 |
| sarnold | looks so *plain* | 21:23 |
| sarnold | nothing here makes sense | 21:23 |
| sarnold | my next thought is to ask perf to dump a stack trace when mmap fails and hope that somepart of the stack trace indicates what might be wrong | 21:24 |
| sarnold | http://www.brendangregg.com/perf.html#StackTraces | 21:24 |
| SSMAdmin | I suggested removing and then reinstalling apache2. matjam did not think that was needed. How about reinstalling php7.2? | 21:24 |
| sdeziel | removing symlinks from /etc/php/7.2/apache2/conf.d/ would be simple too | 21:25 |
| SSMAdmin | Ok, I have no clue about this, so I just follow your suggestions. How do I remove these symlinks, and what will happen? | 21:26 |
| sarnold | meh, i'm not real hopeful, and I'd rather know the *cause*, but if you go that route, I suggest ot use apt-get purge rather than apt-get remove | 21:26 |
| SSMAdmin | sarnold, ok thanks | 21:27 |
| sdeziel | the phpmyadmin that's in "rc" bugs me | 21:27 |
| sdeziel | I'd purge this one first (I never liked that piece of software ;) | 21:27 |
| SSMAdmin | how do I purge it? | 21:28 |
| sdeziel | dpkg -P phpmyadmin | 21:28 |
| SSMAdmin | done | 21:29 |
| sdeziel | (I don't remember if apt-get purge would work on a package in "rc") | 21:29 |
| sdeziel | apache2 still choking on startup? | 21:29 |
| SSMAdmin | how do I check if the purge did what it should? | 21:29 |
| sdeziel | "dpkg -l| grep phpmyadmin" => should return nothing | 21:30 |
| SSMAdmin | apache2 still choking | 21:30 |
| SSMAdmin | returned nothing, so good | 21:30 |
| sarnold | apt-get purge on 'upstart' on my laptop took it from rc to un | 21:31 |
| sdeziel | thx | 21:31 |
| sdeziel | that's weird to see "openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = 12" when mod_ssl isn't even enabled | 21:33 |
| SSMAdmin | So, if I should try to clean out php (both 5 and 7?) and then reinstall, what should I do then? And that means I must reinstall also extra modules needed by apache and nextcloud? | 21:33 |
| SSMAdmin | I have a2enmod ssl now | 21:34 |
| sdeziel | I just picked it as it was few lines before the huge malloc | 21:35 |
| sdeziel | s/malloc/mmap/ | 21:35 |
| SSMAdmin | ohh btw, it was php7.2 only that is creating the problem, not php5? We only disabled php7 | 21:35 |
| sdeziel | on a bionic box, only 7.2 should be there | 21:36 |
| sdeziel | https://paste.ubuntu.com/p/XfKBbwf5gM/ showed it was the case | 21:36 |
| SSMAdmin | ok | 21:37 |
| SSMAdmin | so....apt purge <what>? | 21:38 |
| SSMAdmin | php7? | 21:38 |
| sdeziel | SSMAdmin: each PHP package one at the time, I'd start with php-apcu | 21:43 |
| SSMAdmin | purge or just disable them? | 21:44 |
| sdeziel | purge would probably be easier to undo afterward | 21:44 |
| SSMAdmin | ok, will you guide me? | 21:45 |
| SSMAdmin | so I start with apt purge php-acpu | 21:45 |
| sdeziel | I'm afraid I'll have to leave soon | 21:45 |
| SSMAdmin | ok | 21:46 |
| sdeziel | but it's pretty simple, purge, test, rince and repeat | 21:46 |
| sdeziel | the above paste captures the list of php packages you had before starting to poke at it | 21:47 |
| SSMAdmin | so,example php-curl, will that also purge php7.2-curl? Or is there some significant difference in all these? Are there some I should NOT purge? | 21:47 |
| SSMAdmin | Question put another way: Why are there some of these modules without the *7.2 part in the name | 21:50 |
| sdeziel | those are virtual packages | 21:50 |
| sdeziel | they point to the current php7.X-mod version | 21:50 |
| sdeziel | on Xenial it would have been php7.0-mod, on Artful: php7.1-mod and so on | 21:50 |
| SSMAdmin | ok, but there are some of them that does not have a php7.x version of them | 21:50 |
| SSMAdmin | php-acpu as an example | 21:51 |
| sdeziel | I _think_ that's because they don't build from the PHP sources and are external components | 21:52 |
| SSMAdmin | and should I purge the virtual ones, or the 7.x ones? | 21:52 |
| sdeziel | SSMAdmin: purging the 7.2 one should also purge the non versioned one | 21:54 |
| SSMAdmin | thanks, I go ahead, take them one by one, purge, test, reinstall, purge next, test, reinstall? | 21:55 |
| sdeziel | nacc also suggested to purge and reinstall libapache2-mod-php* | 21:56 |
| sdeziel | and he knows PHP far better than I do :) | 21:57 |
| SSMAdmin | ok, "apt purge libapache2-mod-php*" will work, but how about reinstall? "apt install libapache2-mod.php*" will probably not work | 21:58 |
| SSMAdmin | apt install libapache2-mod-php | 21:58 |
| SSMAdmin | apt install libapache2-mod-php* | 21:58 |
| sdeziel | installing libapache2-mod-php only should pull the other | 21:59 |
| SSMAdmin | ok thank you. I try | 21:59 |
| SSMAdmin | purging libapache2-mod-php* made the apache2 able to start | 22:01 |
| nacc | SSMAdmin: yeah that seems accurate (as purge will remove the config which implies a2dismod php7.2) | 22:02 |
| SSMAdmin | but now some ssl protocol error on the website | 22:02 |
| sdeziel | SSMAdmin: wasn't SSL disabled earlier? | 22:03 |
| SSMAdmin | so better leave libapache2-mod-php installed, for bugtracking the other modules | 22:03 |
| sdeziel | but if you have SSL errors, that probably means apache2 starts now, isn't it? | 22:03 |
| SSMAdmin | yes, but I enabled it again (a2enmod ssl), I think | 22:04 |
| SSMAdmin | sdeziel, apache starts now, yes | 22:04 |
| sdeziel | ok is mod-php installed yet? | 22:05 |
| SSMAdmin | no, should I reinstall it? | 22:06 |
| SSMAdmin | I try | 22:06 |
| sdeziel | yes because we already knew that without php involved it was working OK | 22:06 |
| SSMAdmin | Well, now apache2 is able to start with libapache2-mod-php reinstalled | 22:09 |
| nacc | SSMAdmin: so i'd assume it was due to a config change you did with php before | 22:09 |
| SSMAdmin | so now this part is fixed, apache2 is able to run with php involved | 22:09 |
| nacc | SSMAdmin: maybe related to nextcloud, dunno | 22:09 |
| SSMAdmin | nacc, no, I have not touched the system since I installed it in April, it has been running without error until yesterday | 22:10 |
| SSMAdmin | but now there is still a ssl protocol error | 22:10 |
| SSMAdmin | I did "a2enmod ssl" to enable it, we disabled it earlier in this session | 22:10 |
| sdeziel | SSMAdmin: didn't you have a nextcloud vhost specifically for SSL? if yes, I'd enable it | 22:11 |
| SSMAdmin | still some problem | 22:11 |
| sdeziel | what's apache2 error.log like now? | 22:11 |
| SSMAdmin | sdeziel, ahh...there it is, I try enable the nextcloud.conf file | 22:11 |
| SSMAdmin | wait a second.. | 22:11 |
| * sdeziel is in holding pattern | 22:12 | |
| SSMAdmin | HOLY MOLY, everything is suddenly working | 22:13 |
| dpb1 | wow | 22:14 |
| dpb1 | epic debug session? | 22:14 |
| dpb1 | like, 2 hours | 22:14 |
| sdeziel | but the root cause remains unkown so we are none the wiser | 22:14 |
| SSMAdmin | so, then only thing to fix the problem was to purge and reinstall libapache2-mod-php | 22:14 |
| nacc | which implies some config fubar | 22:14 |
| nacc | i know you'll say you didn't do anything | 22:15 |
| nacc | but that's not what the operations imply :) | 22:15 |
| sdeziel | still would have been nice to find what was mmap'ing 85T | 22:16 |
| sdeziel | I know sarnold, I hear you ;) | 22:16 |
| SSMAdmin | nacc, well, I KNOW I have not touched the system, exerpt running "apt update" and "apt upgrade" manually on a few occations. I have not touched a single config file or installed any additonal modules after I got the system running in April | 22:16 |
| SSMAdmin | sdeziel, what was that mmap'ing 85T? | 22:20 |
| nacc | from your strace of apache2 earlier | 22:20 |
| SSMAdmin | opps, too late, he is gone | 22:20 |
| SSMAdmin | what does it mean? Someone trying to send 85T to the server? | 22:20 |
| nacc | SSMAdmin: no, your apache2 program was trying map 85T of virtual memory, which the kernel said no to. | 22:21 |
| SSMAdmin | ok, I don't understand "mapping of virtual memory" | 22:22 |
| SSMAdmin | maybe I don't need to know? :-) | 22:23 |
| SSMAdmin | anyway, thanks a lot everyone. Say thanks to sdeziel from me, and also to matjam that worked a long time to try to help me | 22:24 |
| _KaszpiR_ | SSMAdmin you should REALLY order pizza/beer for them :) | 22:25 |
| SSMAdmin | _KaszpiR_, I will :-) Is there a command for that? | 22:26 |
| SSMAdmin | pastepitsa&beer ? | 22:27 |
| nacc | !cookie | 22:28 |
| ubottu | Wow! You're such a great helper, you deserve a cookie! | 22:28 |
| nacc | is the closest we have :) | 22:28 |
| SSMAdmin | !fortunecookie | 22:28 |
| nacc | SSMAdmin: sorry, didn't see your question -- it's the program trying to allocate memory to do something | 22:28 |
| SSMAdmin | nacc, what could need 85T? | 22:29 |
| nacc | SSMAdmin: that's what we don't know | 22:30 |
| nacc | i guess if it happens again, rather than just purge | 22:30 |
| nacc | make a backup of /etc, then purge, then reinstall and diff old and new /etc | 22:30 |
| SSMAdmin | thanks, but that sounds like something more complicated. I don't know how to do the diff-stuff | 22:31 |
| _KaszpiR_ | I suggest asking on privmsg for details where and when to deliver pizza ;) | 22:32 |
| SSMAdmin | I am a linux newbie, have been a newbie for many years | 22:32 |
| SSMAdmin | _KaszpiR_, that sound a bit like pizza-gate, I am not into that sort of stuff :-) | 22:33 |
| _KaszpiR_ | hahahahahaah | 22:33 |
| nacc | SSMAdmin: something link `sudo cp -aR /etc /etc/.bak; sudo apt purge libapache2-mod-php*; sudo apt install libapache2-mod-php; diff -urpN /etc.bak /etc` and if that last command returns anything, something your local config changed | 22:34 |
| nacc | bah first line was a typo :) | 22:34 |
| nacc | sudo cp -aR /etc /etc.bak | 22:34 |
| SSMAdmin | nacc thanks a lot, I take notes | 22:34 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!