| === Lcawte is now known as Lcawte|Away | ||
| stevenroose | is it possible that certain RAID controllers are not supported by Debian? | 01:49 |
|---|---|---|
| TJ- | stevenroose: Yes, it is possible | 01:50 |
| stevenroose | like an Intel RAID controller? They have drivers for RHeL and SUSE | 01:50 |
| stevenroose | but with most hardware, the community often ports them | 01:50 |
| stevenroose | TJ-: is there a way I can check? | 01:50 |
| TJ- | stevenroose: look at the PCI ID of the device, and see which, if any, driver has that device ID listed | 01:51 |
| stevenroose | I feel sorry to ask you, but how do I check pci ids and where do I find the driver listings for Debian? | 01:53 |
| TJ- | stevenroose: "lspci -nn" and look at the [VVVV:PPPP] id (Vendor:Product), then do "grep -i 'VVVV.*PPPP' /lib/modules/$(uname -r)/*" ... any module that claims an alias for that ID will be reported | 01:55 |
| stevenroose | 1000 for intel, could that be right? | 01:56 |
| stevenroose | cendor id | 01:56 |
| stevenroose | I think I got one | 01:59 |
| stevenroose | megaraid_sas | 01:59 |
| stevenroose | nice, thanks | 01:59 |
| stevenroose | altough, that means the issue lies within the raid controller, a lot harder to fix :s | 02:00 |
| stevenroose | TJ-: lsst question on RAID, do some raids provide a way to configure them from within linux and if so, how? | 02:01 |
| stevenroose | currently, I should boot into the BIOS of the RAID controller, but it wont boot | 02:01 |
| pmatulis | stevenroose: the "issue" *may be* that the controller is not supported by the Ubuntu kernel you're running | 02:02 |
| TJ- | stevenroose: it depends on the RAID controller make/model. Some have F/OSS tools as well as the BIOS config, others have proprietary Linux tools | 02:02 |
| TJ- | stevenroose: 'won't boot' doesn't tell us a lot. what do you see specifically? how is the disk partitioned (MBR or GPT) | 02:03 |
| TJ- | stevenroose: I fixed an issue for a user earlier with hardware RAID with 4TB disk array, partitioned as GPT whic wouldn't boot. It needed a hybrid MBR with the boot partition in the MBR, so that the BOOT BIOS could find it | 02:04 |
| stevenroose | TJ-: well, I had a software raid on the drives using mdadm, and I wanted to move them to the controller | 02:05 |
| stevenroose | I did not repartition them or anything.. | 02:05 |
| stevenroose | (I'm fine with the data being lost though) | 02:05 |
| stevenroose | prepared for that | 02:05 |
| TJ- | stevenroose: that will rarely work, the mdadm metadata will not work with the metadata the RAID controller will use | 02:05 |
| stevenroose | what the bios boot says is "there aren;t any devices: | 02:06 |
| stevenroose | TJ-: yea I figured that, I just assumed the controller would be able to reformat everything according to how they needed it | 02:06 |
| stevenroose | again, I prepared for losing the data | 02:07 |
| TJ- | stevenroose: boot with a Live ISO image, then look at the raw data on the hardware RAID device, that will give you some idea of how the array is presented | 02:07 |
| TJ- | stevenroose: have you already created the array in the controller's BIOS setup? | 02:07 |
| stevenroose | TJ-: how does booting from a live image matter there? | 02:07 |
| stevenroose | I can boot the computer | 02:07 |
| stevenroose | no, the controllers bios won't start | 02:08 |
| stevenroose | if I press the key to enter it, it prints "there aren't any devices" shortly and exits | 02:08 |
| stevenroose | continuing the regular boot process | 02:08 |
| TJ- | stevenroose: if the device won't boot put is present, you can look at the raw data and figure out if there's an offset required to get to the partitions inside the mdadm volumes | 02:08 |
| TJ- | stevenroose: have you connected the drives to the correct ports on the controller? | 02:09 |
| stevenroose | yes, it has one plug that fans out into 4 cables | 02:09 |
| stevenroose | I connected 0-2 to my 3 drives | 02:09 |
| stevenroose | the first thing you said I did not understand, though :p | 02:09 |
| TJ- | stevenroose: what is the exact make/model of controller? | 02:09 |
| stevenroose | Intel RAID controller SRCSASLS4I | 02:10 |
| stevenroose | I bought it second hand though | 02:10 |
| stevenroose | so at boottime it says that the drives [it used to work with] are not present, asks to press C to reconfigure, enters BIOS, prints "there arent't any devices" and exits | 02:11 |
| stevenroose | (it does not load the graphical bios like on the manual, though; it just prints console" | 02:11 |
| stevenroose | weird thing is, that it seems to recognize the disks in some way, before the message comes, it very briefly prints a table with SATA and other data in it (too short to give it a good look), with 3 rows, and when I booted with the devices unplugged, the table was emty | 02:13 |
| stevenroose | TJ-: so yes, the partitioning might be at fault. what should normally be done when preparing a hard disk for usage with a raid controller? I have no experience with them | 02:14 |
| TJ- | stevenroose: at boot time you see "Press <CTRL><G> to run BIOS Console 2." ? | 02:15 |
| stevenroose | yes | 02:15 |
| stevenroose | I press it | 02:15 |
| stevenroose | then the table comes up shortly | 02:15 |
| TJ- | stevenroose: At that point you should have a Configure menu | 02:16 |
| stevenroose | then there is something that I can pause or skip (with CTRL + somethign), but it is over so quickly that I cannot interact, and then the screen refreshed with some specs on top and then the "there aren't any devices" message | 02:16 |
| stevenroose | would you be able to tell more if I filmed it? :D | 02:17 |
| TJ- | stevenroose: I'm just reading the hardware guide | 02:17 |
| stevenroose | any luck? | 02:19 |
| TJ- | stevenroose: you need to read the Software User Guide, it has extensive info on configuration | 02:20 |
| stevenroose | got a link perhaps? | 02:21 |
| stevenroose | I have a disk here, but I dont have a drisk drive anymore on any of my current machines | 02:21 |
| stevenroose | just to make sure: there are no common requirements regarding the partitioning of disks when connecting them for hardware raid? | 02:22 |
| TJ- | Looks like Chapter 8 applies you your device. http://download.intel.com/support/motherboards/server/sb/d29305023_raid_swug_r23_5.pdf | 02:24 |
| stevenroose | TJ-: do you know the answer to my last question? it might save me a lot of time :) | 02:28 |
| TJ- | stevenroose: the controller is responsible for writing metadata to the disks, it then presents a logical view to the BIOS and operating system, and they can do whatever they want on the logical disk | 02:30 |
| stevenroose | TJ-: so whatever partitioning they have when connected, the controller will just ingore the partitioning and overwrite it when the setup is confirmed? | 02:31 |
| TJ- | stevenroose: it'll look for its own existing metadata and if not found assume the disks are unallocated | 02:32 |
| stevenroose | the problem is that the use guide assumes everything is working | 02:35 |
| stevenroose | like here | 02:35 |
| stevenroose | http://imgur.com/5eabysZ | 02:35 |
| stevenroose | I don't get to the second part of the first step | 02:35 |
| TJ- | stevenroose: maybe that's a sign the controller is faulty | 02:40 |
| TJ- | stevenroose: if no controller selection appears, then its probably dead | 02:41 |
| stevenroose | TJ-: I'm trying to film the boot process for you to judge what you think, but my phone doesnt seem to want to boot anymore, give me a minute | 02:46 |
| === JanC_ is now known as JanC | ||
| samthewildone | can I use ubuntu server as a mining for bitcoin without actually installing a desktop environment ? | 03:48 |
| pmatulis | stevenroose: skip to a terminal and use lspci to determine if the controller is recognized | 03:50 |
| lordievader | Good morning. | 06:04 |
| === Lcawte|Away is now known as Lcawte | ||
| === cluelessperson_ is now known as cluelessperson | ||
| cluelessperson | hey guys, I'm trying to setup my swap partition. I extened the drive space for my VM, and I created 2.25GB for swap, and I'm tryiong to setup fstab to load it, but it doesn't seem to work | 09:23 |
| cluelessperson | swapon -s shows nothing. | 09:24 |
| lordievader | cluelessperson: Did you reboot after you added it to fstab (or did you run swapon -a)? | 09:27 |
| cluelessperson | lordievader, rebooted, twice | 09:27 |
| cluelessperson | swapon -s http://paste.ubuntu.com/12170224/ | 09:28 |
| lordievader | Could you pastebin your fstab? | 09:28 |
| cluelessperson | cat /etc/fstab http://paste.ubuntu.com/12170232/ | 09:28 |
| lordievader | The uuid is correct? | 09:29 |
| cluelessperson | lordievader, I copied it when I did the make swap command | 09:30 |
| lordievader | cluelessperson: What is the output of 'sudo lsblk -o name,uuid'? | 09:30 |
| cluelessperson | http://paste.ubuntu.com/12170242/ | 09:31 |
| lordievader | Is sda2 swap? | 09:32 |
| cluelessperson | lordievader, It should be, yes | 09:32 |
| lordievader | Then the uuid should be: 7d7df027-4b53-4744-9656-cca0d1739a4e | 09:32 |
| lordievader | Then run swapon -a again. | 09:34 |
| cluelessperson | lordievader, excellent! :D | 09:36 |
| cluelessperson | lordievader, thanks, sorry for the annoyance. I must have mis-copied or gotten mixed up | 10:03 |
| lordievader | No problem ;) | 10:03 |
| hackeron | hey, anyone has idea what is leaking memory on my ubuntu server and how to troubleshoot? < http://superuser.com/questions/793192/invisible-memory-leak-on-linux-ubuntu-server-not-disk-cache-buffers | 10:26 |
| bekks | Nothing is leaking memory. | 10:27 |
| bekks | As being told in the post, the memory management works different from how you think it does. | 10:28 |
| bekks | hackeron: Please see http://www.linuxatemyram.com/ as well. | 10:28 |
| hackeron | bekks: I did - read again | 10:29 |
| hackeron | bekks: look at the -/+ buffers/cache line in that post | 10:29 |
| bekks | I did - can you rephrase whats to be seen there? | 10:29 |
| hackeron | bekks: no you didn't, -/+ buffers/cache: 1771 220 -- so after accounting for cache/buffers, there is only 220MB free, 1.7GB is missing/leaked | 10:30 |
| bekks | So read and try to understand http://www.linuxatemyram.com/ | 10:30 |
| hackeron | bekks: I did | 10:31 |
| hackeron | bekks: read and try to understand my question | 10:31 |
| bekks | I did. Your question is explained on http://www.linuxatemyram.com/ | 10:31 |
| hackeron | bekks: no it is not - you are misunderstanding the question | 10:31 |
| bekks | So rephrase it please. | 10:31 |
| hackeron | bekks: memory is not used for disk caching | 10:31 |
| hackeron | bekks: what that page says does not apply | 10:32 |
| bekks | So you manually disabled disk caching in memory? | 10:32 |
| dasjoe | That superuser.com question is a year old | 10:32 |
| bekks | If you didnt, then the memory is caching. | 10:32 |
| hackeron | dasjoe: it is still an issue in 15.04 | 10:32 |
| dasjoe | hackeron: I don't see an issue, sorry | 10:32 |
| bekks | hackeron: It isnt, and it never was, after understanding http://www.linuxatemyram.com/ | 10:32 |
| hackeron | bekks: I did, read the question: sync; sync; sync; echo 3 > /proc/sys/vm/drop_caches | 10:32 |
| hackeron | bekks: I did drop caches explicitly, it is not a disk caching issue | 10:33 |
| dasjoe | "note cached and buffers are both low so this is not disk cache or buffers!", how is 1771 MB "low"? | 10:33 |
| bekks | Droping cache does not mean disabling them. | 10:33 |
| hackeron | dasjoe: you are misreading the output of free.... | 10:33 |
| hackeron | dasjoe: -/+ buffers/cache means without buffers or cache | 10:34 |
| hackeron | bekks: dropping cache would clear that 1.7GB if it was a disk caching issue | 10:34 |
| bekks | hackeron: It would write down the content but not remove the cache. | 10:34 |
| hackeron | bekks: ok, how do you disable disk cache just to prove to you it is nothing to do with disk cache? | 10:35 |
| bekks | There is no issue to be proven. | 10:35 |
| hackeron | bekks: look at the meminfo output -- look at the cached line | 10:35 |
| bekks | Your question clearly shows that you did not understand the output of free -m | 10:35 |
| hackeron | bekks: no it does not - read the link you posted | 10:36 |
| bekks | I know that link very well, and it perfectly explains the output of free -m | 10:36 |
| hackeron | bekks: To see how much ram is free to use for your applications, run free -m and look at the row that says "-/+ buffers/cache" in the column that says "free". That is your answer in megabytes: | 10:36 |
| hackeron | bekks: so the free memory is 220MB according to your link | 10:37 |
| hackeron | bekks: you really really do not - I'm sorry to say | 10:37 |
| hackeron | bekks: you are wrong, completely and you need to read your link before you say anything more because this is pretty embarassing for you | 10:38 |
| bekks | hackeron: I'm not sorry to say that you dont understand what "free" and "unused" means. | 10:38 |
| hackeron | bekks: ok, just read your link... | 10:38 |
| hackeron | bekks: free is 153MB -- free when you subtract cache and buffers is 220MB -- there is 1.7GB unaccounted for | 10:38 |
| bekks | hackeron: The only thing embarrassing here is your attitude getting angry with people who try to help you. You're just not worth to waste my time any further. | 10:38 |
| bekks | *plonk* | 10:39 |
| hackeron | bekks: you are not helping, you are wrong and refusing to accept it | 10:39 |
| hackeron | Anyone has idea what is leaking memory on my ubuntu server and how to troubleshoot? (Note this is not a disk cache/buffers issue, free memory is 153MB, after subtracting cache and buffers, it is 220MB, there is 1.7GB missing and unaccounted for) < http://superuser.com/questions/793192/invisible-memory-leak-on-linux-ubuntu-server-not-disk-cache-buffers | 10:40 |
| hackeron | updated the question to include this for people like bekks who really should know better: "Please note, I have read and understood http://www.linuxatemyram.com - this is not a disk cache/buffers that is using up the memory. Before subtracting cache/buffers, the free memory is 153MB, after subtracting it is 220MB, there is still 1.7GB missing and unaccounted for." | 10:43 |
| dinet | good day | 12:01 |
| dinet | I have a quite annoying problem. Have three vhosts in apache and after a while apache stops responding. I got the sites to work yesterday but now it broke again | 12:01 |
| dinet | the sites is stored in ~/www/sitename with a link in /var/wwww | 12:01 |
| stevenroose | pmatulis: if you have aminute to look into this, I made a question over at serverfault, https://serverfault.com/questions/716105/intel-raid-controller-cannot-open-bios-and-doesnt-recognize-hard-drives | 12:02 |
| Sling | dinet: define 'after a while' and 'stops responding', does it happen frequenty, after a set time? after doing something specific | 12:02 |
| Sling | also did you check the logs and if possible mod_status output? | 12:02 |
| dinet | Sling: or more like the connection freezez. the browser can stay on loading for ~30 min. This stint it froze after less than 24 hours | 12:04 |
| dinet | Sling: saw this in error.log: H00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting | 12:05 |
| Sling | dinet: what MPM are you using and what settings does it have? did you change any of them? | 12:05 |
| dinet | mpm? | 12:06 |
| Sling | you can see this in apache2ctl -V output | 12:06 |
| dinet | Server MPM: prefork | 12:06 |
| Sling | http://httpd.apache.org/docs/2.4/mod/prefork.html#how-it-works | 12:07 |
| Sling | that describes which settings control how many clients can be served simultaneously in prefork | 12:07 |
| Sling | I would recommend using event or worker though, prefork is the least performing way of running apache httpd | 12:07 |
| dinet | can I change in http.conf? | 12:08 |
| dinet | and that is how long ago I did these things.. | 12:09 |
| Sling | depends on which version you are running, in 2.4 you can switch MPM my loading the relevant mpm module | 12:09 |
| Sling | but in 2.2 its compiled in the server | 12:09 |
| dinet | Sling: standard install in ubuntu | 12:09 |
| Sling | its also in the -V output | 12:10 |
| dinet | 2.4 | 12:10 |
| Sling | the default ubuntu package doesn't use prefork I think, unless you use mod_php, are you using that? | 12:11 |
| dinet | yes I do | 12:11 |
| Sling | hm thats also a quite oldfashioned way of running apache+php :) | 12:12 |
| Sling | come to #httpd and I will trigger some links to read up | 12:12 |
| dinet | Sling: or how it now i started. did the install early this summer and used apt-get install apache2 php5 mysql | 12:13 |
| Sling | dinet: yeah because of some legacy non-threadsafe php things, it will default to the prefork mpm | 12:14 |
| Sling | which spawns a process per request so it doesn't break non-threadsafe php features | 12:14 |
| Sling | but spawning a process every time a request comes in is not very performance efficient :) | 12:14 |
| RoyK | Sling: what would you recommend instead of mod_php? fastcgi? | 13:02 |
| Tangurin | hi! | 13:12 |
| patdk-lap | I would never recommend mod_php :) | 13:14 |
| patdk-lap | fastcgi is the best option | 13:14 |
| patdk-lap | going with either direct php, or php-fpm, I perfer fpm myself | 13:14 |
| JanC | Sling: -prefork doesn't spawn a process on every request (that's why it's called PREfork) | 13:16 |
| patdk-lap | and it still doesn't do it per request, but per connection | 13:17 |
| JanC | and dinet: if you have enough RAM to do so, you can raise MaxRequestWorkers indeed | 13:17 |
| patdk-lap | and php is built in trusty using threadsafe, so apache doesn't need prefork | 13:17 |
| tompaw | Morning. | 13:31 |
| Sling | JanC: well, once the serverstart-number of processes are started, thats essentially what happens on a busy server | 14:09 |
| Sling | in any case, its not optimal when dealing with high traffic | 14:10 |
| RoyK | patdk-lap: is php threadsafe now? | 14:36 |
| patdk-lap | it has been for a long long time | 14:37 |
| patdk-lap | the problem is, most modules aren't, so no one used the threadsafe version | 14:37 |
| patdk-lap | but debian/ubuntu with apache 2.4, switched to threadsafe mod_php | 14:38 |
| xedniv | what is the best way to clone a real/metal server to a vm under kvm? and if it is to be done manually, is thereanything ready-made that lets you configure which /etc directories and files should be carried over gracefully? | 15:52 |
| xedniv | possibly with interdiffing/choosing which one gets applied | 15:52 |
| === Tangurin is now known as Tangen | ||
| catalase | sudo usermod -a -G catalase debian-transmission. usermod: user 'debian-transmission' does not exist | 16:22 |
| catalase | how can a user own those files but not exist. i can't move them, i can't even view the directory. | 16:22 |
| catalase | why can't i change the permissions | 16:24 |
| catalase | sudo chmod 777 /var/lib/transmission-daemon. chmod: cannot access ‘/var/lib/transmission-daemon’: No such file or directory | 16:24 |
| maxb | catalase: Well, the obvious first question is 'Since the command says it doesn't exist, what's the evidence that it does?' | 16:27 |
| catalase | because when i use filezilla to look at the filesystem i can see that it exists | 16:30 |
| catalase | http://i.imgur.com/RPUl747.png | 16:30 |
| catalase | sudo service transmission-daemon stop. transmission-daemon: unrecognized service | 16:33 |
| catalase | i cant edit the settings.json to change the umask to 0 | 16:33 |
| maxb | I don't know filezilla. I suggest you use 'ls -l /var/lib', not least because you can paste that to IRC | 16:39 |
| skittishtrigger | In terminal, do you use the same method as changing /home mount points for /var? | 16:40 |
| samthewildone | is there a way I can run a gpu miner without the xserver on the server ? | 16:40 |
| samthewildone | or I must install a desktop environment ? | 16:40 |
| maxb | skittishtrigger: You probably need to explain in a little more detail what you're trying to do, I don't quite understand what you're asking from what you've said so far | 16:42 |
| skittishtrigger | I want to change the mount point for /var. atm (transferring servers and the mountpoint for /var is too small in new one) | 16:42 |
| skittishtrigger | or would symslinking be better in this scenario? | 16:44 |
| maxb | It's a bit unusual to have /var be its own mount point at all, have you considered just not using so many separate partitions? | 16:44 |
| maxb | Changing mount points is easy, you just edit /etc/fstab. However the hard part is preparing the replacement thing you're going to mount instead, and copying data | 16:45 |
| skittishtrigger | symlinking the assets folders might be a better way to go them. changing that should not be such a big hassle I am thinking | 16:46 |
| Demon_Jester | Hey guys, I ssh'd into my server, and noticed that htop, and screenfetch are only showing 4gb of ram, I ran "sudo lshw -short -C memory" and it showed 8gb, is there another way to detect all 8gb just to make sure? | 16:47 |
| Demon_Jester | I mean to see why not all 8gb of ram is not being detected. | 16:47 |
| Demon_Jester | wow I cannot type.. lol | 16:47 |
| RoyK | Demon_Jester: pastebin output of "free" | 16:58 |
| Demon_Jester | http://pastebin.com/ZxB05Bdr | 16:59 |
| Demon_Jester | swap shows 8gb or atleast i think it does. | 16:59 |
| RoyK | what about lshw? | 16:59 |
| Demon_Jester | lshw also shows this.. hold on | 16:59 |
| RoyK | linux only sees 4 gigs there | 16:59 |
| Demon_Jester | http://pastebin.com/mJpVDegp | 17:00 |
| RoyK | wierd | 17:00 |
| RoyK | 32bit system? | 17:00 |
| Demon_Jester | shouldn't be its x86_64 | 17:01 |
| Demon_Jester | OS I mean. | 17:01 |
| RoyK | uname -a ? | 17:01 |
| Demon_Jester | 3.16.0-43-generic #58~14.04.1-Ubuntu SMP Mon Jun 22 10:21:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux | 17:02 |
| Demon_Jester | i opened it to put a plastic hdd bracket back in there, i may have touched one of the ram sticks without knowing it? | 17:02 |
| === Joel is now known as Guest31832 | ||
| RoyK | Demon_Jester: does BIOS see the memory? | 17:03 |
| Demon_Jester | hmm i dont have a monitor readily avail to hook up.. | 17:04 |
| Demon_Jester | give me a few minutes and see if i can hook this monitor up to it | 17:05 |
| Demon_Jester | ok this may seem crazy but I think I know why it was only showing 4gb, I have two other desktops (just the towers no monitors or anything) on the same electrical plug, and I had them one, too, maybe the main seerver wasn't getting enough power and had to use only half of the ram? I turned the desktops off and now its using all 8gb | 17:17 |
| RoyK | Demon_Jester: doesn't make sense | 17:21 |
| RoyK | Demon_Jester: memory doesn't use a lot of power | 17:22 |
| Demon_Jester | idk maybe just a glitch or something? | 17:22 |
| RoyK | reseat memory modules | 17:23 |
| RoyK | that's the first thing | 17:23 |
| Demon_Jester | ok | 17:23 |
| linocisco | hi | 17:23 |
| linocisco | i have edited static interface in /etc/network/interfaces. but /etc/init.d/networking restart could not activate eth0 up | 17:24 |
| RoyK | pastebin it | 17:24 |
| linocisco | RoyK, http://pastebin.ubuntu.com/12175474/ | 17:25 |
| RoyK | linocisco: should work | 17:33 |
| linocisco | RoyK, too simple to see that does not work. I have rebooted and still not working. only lo and virbr0 are working | 17:34 |
| RoyK | linocisco: does ifconfig -a show eth0 ? | 17:35 |
| linocisco | RoyK, nope. just em1, lo and virbr0 | 17:36 |
| RoyK | missing a driver, perhaps? | 17:36 |
| linocisco | RoyK, i think we should rename eth0 to em1 | 17:37 |
| linocisco | RoyK, how do you think? | 17:37 |
| RoyK | might be - try to configure em1 | 17:37 |
| linocisco | RoyK, doesn't work | 17:39 |
| RoyK | linocisco: what does lshw has to say? | 17:48 |
| linocisco | RoyK, i have rebooted again | 17:50 |
| linocisco | RoyK, so many lines in lshw output | 17:59 |
| RoyK | it's quite verbose, yes | 18:02 |
| hackeron_ | Anyone have any ideas how to troubleshoot this memory leak? < http://superuser.com/questions/793192/invisible-memory-leak-on-linux-ubuntu-server-not-disk-cache-buffers | 18:13 |
| TJ- | hackeron_: have you tried replacing the applications with a placebo - if the same issue occurs you've at least proved its kernel not application related ? | 18:25 |
| hackeron_ | TJ-: if it's application related, it should show the memory growing in top or ps -- if the server is just idling and not doing anything, then the kernel does not leak memory. I am not sure what form this placebo can take? | 18:26 |
| TJ- | hackeron_: Pull in the data over the network in the same way but dump it without any kind of processing or piping (so it doesn't get into the caches) | 18:28 |
| TJ- | hackeron_: The other thing I'd do is ptrace kmalloc() and kfree() (and there relations) to track kernel module's use of memory | 18:28 |
| hackeron_ | TJ-: hmm, I will try this - good idea! -- is there a guide how to track kernel module's use of memory? - This is what I haven't been able to find. I think possibly a bug with the ethernet driver as I'm pulling terabytes of data. | 18:29 |
| TJ- | hackeron_: I'm also wondering if memory fragmention could be an issue | 18:30 |
| TJ- | hackeron_: maybe this will get you further http://elinux.org/Kernel_dynamic_memory_analysis#Memory_accounting | 18:30 |
| TJ- | hackeron_: Yes, that would be the easy culprit to blame :) Do the NICs have any kind of offload capability, and if so, have you tried changing that (if it is offloading, disable it, or if it is disabled, enable it) - anything to provoke a real change in the stats you're collecting to give you a clue | 18:31 |
| TJ- | hackeron_: from that link I think this may be useful: | 18:32 |
| TJ- | Internal fragmentation | 18:32 |
| TJ- | For different reasons (alignment, overhead, etc) when we request 100 bytes with kmalloc(100) the slab allocator may really allocate 128 bytes (or 140 bytes, we can't really know). These extra 28 bytes can't be used, and therefore you are wasting them | 18:32 |
| hackeron_ | TJ-: not quite sure how to check/change - I will check the ethtool man page as I'm hoping that will do it? | 18:32 |
| hackeron_ | TJ-: interesting - how would I detect something like that? | 18:33 |
| TJ- | hackeron_: offloading, if available, is usually kernel-module options so 'modinfo' will reveal | 18:33 |
| TJ- | hackeron_: use the suggestions in that link to do memory accounting on kmem | 18:33 |
| hackeron_ | TJ-: ok, will read and try to make sense of it all :) - I'm new to this low level stuff | 18:35 |
| hackeron_ | TJ-: thank you! | 18:35 |
| TJ- | hackeron_: Good luck with it, I know how infuriating and time consuming it can be | 18:36 |
| hackeron_ | TJ-: definitely :( - been trying to figure this out for over a year - but now on 1 server it is quicker to reproduce so goign to give it another crack! | 18:37 |
| TJ- | hackeron_: have you tried rate-limiting the network input and seeing if that correlates to the growth in the missing mem? | 18:38 |
| hackeron_ | TJ-: so this kmalloc vs slab allocator is purely kernel side? - or can this "memory disappearing" issue happen transparently by something in the userspace? | 18:38 |
| TJ- | hackeron_: E.g. force the link to run at 100Mbps instead of 1Gbps | 18:38 |
| TJ- | hackeron_: it's all in-kernel, but obviously kernel allocates memory for processes too so you can monitor internal kernel use and per-process usage. | 18:39 |
| hackeron_ | TJ-: ok, but generally, no user space app should be able to trigger this memory disappearing? - just to rule out ffmpeg/python, etc | 18:40 |
| TJ- | hackeron_: I'd also strongly advise you to talk to the kernel's memory management (mm) team. The web-site is at http://www.linux-mm.org and they have a mailing list linux-mm@kvack.org | 18:41 |
| TJ- | hackeron_: userspace could trigger... requesting lots of blocks of memory could hit the fragmentation issue such that kernel is left with lots of small blocks that cannot satisfy requests | 18:42 |
| hackeron_ | TJ-: and even once all those processes are killed, the memory will remain unusable until reboot? | 18:43 |
| hackeron_ | TJ-: and thank you, will try all your suggestions and speaking to them | 18:43 |
| hackeron_ | TJ-: made a list of the 5 things to try and added to the question, heh, thank you again :) -- So once all the processes are killed - all the lots of small blocks remain in limbo and are never released by the kernel? | 18:59 |
| TJ- | hackeron_: I would have thought that once memory is being freed wholesale the fragments should be coalesced back into larger chunks... but maybe you've found a subtle bug | 19:00 |
| TJ- | hackeron_: the fact that swap starts to be used suggests fragmentation may be part of the issue, if not all of it | 19:00 |
| hackeron_ | TJ-: seems to be the case - I've let the server idle for a week, with essentially just sshd and syslog running and it was still missing 1.8GB of ram :/ | 19:00 |
| hackeron_ | TJ-: I tried disabling swap and the memory still grows transparently and stays unusable even after I've killed just about every userspace process | 19:01 |
| TJ- | that almost certainly rules out the applications, then. | 19:01 |
| hackeron_ | TJ-: I cannot see any memory use growing in top at all :/ - but I can in newrelic which just uses the -/+ buffers/cache line in free -m | 19:02 |
| hackeron_ | TJ-: hmm, ok, I will try ethernet stuff (offload capability, rate limiting), then try to replace app with placebo (just reading network traffic), then see if I can figure out the tracking of kernl module's use of memory and "Internal fragmentation" in that link, if that fails, will ask the linux mm team | 19:04 |
| TJ- | I'm going to guess and say the issue is likely related to mlock ... allocations that aren't subsequently unlocked and returned to the pool | 19:04 |
| TJ- | It's worth asking the mm team early; they may be able to give you specific instructions and tooling to test with, they may have ideas already on what to focus on | 19:05 |
| TJ- | they are the experts, after all | 19:05 |
| hackeron_ | TJ-: hmm, ok, good idea - will ask them | 19:09 |
| === cz2 is now known as NaN | ||
| === NaN is now known as cz2 | ||
| F1skr | Is it possible to get a ubuntu server root tar package somewhere? Similar to what is provided here: https://cloud-images.ubuntu.com/vivid/current/ but without the cloud-init stuff? | 21:49 |
| ObrienDave | like a regular server ISO? | 21:50 |
| F1skr | I just need a root filesystem which I can boot in a systemd-nspawn container | 21:51 |
| F1skr | I can remove the cloud-init services from the cloud version but it's a bit annoying to remove stuff just to get it to boot | 21:54 |
| eatingthenight | anyone have a good program they use for log aggregation from multiple servers | 23:01 |
| pmatulis | eatingthenight: what features do you need that the default (rsyslog) does not provide? | 23:17 |
| jelly | eatingthenight: both syslog-ng and rsyslog are nice enough | 23:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
