[10:08] cking, apw, hi, I currently have remote access to the school that had that disk issue. The exact symptoms are that disk writes start at e.g. 200 MB/sec on an HDD, and after some time, they become 2 MB/sec, without going back to 200 MB even if we leave it for hours (was suspecting temperatures). This happens on 4.4 and 4.8 (xenial), but not on 3.2 (precise) or 4.2 (trusty). I've seen this in another case a few months ago, but I had dismissed it [10:08] What I'd like to ask is, apart from `ubuntu-bug linux`, should I test or collect something else, since I won't frequently have remote access to the school? [10:08] My test case is dd if=/dev/zero of=test bs=1M count=1000 conv=fdatasync; then cp -a some-20gb-folder; then dd again => and there it's slow and never gets fast again [10:12] alkisg, so what happens if after that initial dd you sync and re-run that test? [10:13] cking: it degrades gradually, e.g. starts from 200, then 190, then 100... [10:14] It also happens on their SSD, it's not related to the media [10:14] alkisg, so I guess what is happening is that initially the writes fill up the cache and so it appears to be writing fast, but as the cache gets full you get band width limited to the speed of the block device [10:15] cking: echo 3 > /proc/sys/vm/drop_caches would tell me that, wouldn't it? [10:15] It's an amazing difference, 200 mb/sec to 2 mb/sec, which makes the system completely unusable [10:16] I don't think it's related to caching, I've never seen something similar except for those 2 cases... [10:16] And, it happens only with 4.4+, not with previous kernels [10:17] alkisg, so, lets factor out the cache and just do the dd with conv=direct and see what speed you get [10:17] cking: ok, let me instruct the school to reboot with 4.8, as now it's with a trusty live cd that works fine. I can also give you reverse vnc if you like. Reporting in 2 mins... [10:18] cking, note these are CD boots so with whatever is on the media [10:19] (of course I'm dd'ing to the disk, not to RAM/overlayfs...) [10:20] * cking finds a fungible machine to try this out too [10:20] I've tested about 100 installation so far, so this issue should be quite rare... [10:24] alkisg, so whats the failure rate, e.g how many times have you seen it not working efficiently? [10:25] cking: it 100% reproducable, what varies is that it might need 20 GB of writes before it starts to happen, or 80 GB of writes [10:25] alkisg, I mean, of all the machines it's been installed on, how many exhibit this issue? [10:26] I've only seen that issue in 2 machines. I've installed 16.04 in about 100 machines so far. [10:26] alkisg, if you can add the machine info to the bug report than that may give some hint on what kind of common H/W issue it is related to [10:27] cking: I've remote access to their xenial/4.8 installation now, freshly booted (it's not a live cd). I can give you reverse vnc if you like. Should I start with some test, or with ubuntu-bug linux? [10:28] reporting the bug 1st would be a good start [10:28] bug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] https://launchpad.net/bugs/1 [10:28] ty, doing so... [10:44] https://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/1698118 [10:44] Ubuntu bug 1698118 in linux-hwe (Ubuntu) "Slow disk writes after some uptime, only on certain hw and 4.4+ kernels" [Undecided,New] === JanC is now known as Guest53242 === JanC_ is now known as JanC === MAbeeTT_ is now known as MAbeeTT [15:23] good day [15:24] how do i patch a kernel? with an rt patch [15:24] ? [15:25] i donloaded the newest rt patch [15:25] and something that looked as the kernel from kernel .org [15:25] in a gz format [15:26] of course the thing that seemed to be the kernel isnt the kernel...it is just a source file.. [15:27] knowhow, if you want to apply a patch, that applies to a source tree [15:27] knowhow, if you want to apply it to the ubuntu kernel, you want to get the kernel for the series on which [15:27] i am not a linux expert...so here what i did [15:27] knowhow, you intend to patch, preferably by checking out the git repo for that series [15:28] i am clueless about what you just said [15:28] i know what git is if that makes a difference [15:28] knowhow, https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel [15:28] knowhow, you are trying to do something deeply hard, and deeply distro specific, with a very big patchset [15:29] knowhow, you would want to have a good knowledge of the platform before even considering attempting something like that [15:29] this is what i have to apply [15:29] https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/ [15:30] it is the 4.30.rc1 way down in the list [15:30] i am on ubuntu 16.04 am i am used with .deb files [15:30] so what do i download from where to apply this rt patch to? [15:31] beacause this: [15:31] https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/ [15:31] contains no kernel source code [15:31] it is just a file in a gz format [15:32] i donloaded them from graphical environment...from firefox that is [15:32] knowhow, depends what you are trying to achieve, if you want to apply it to an ubuntu kernel, you would want to pick the ubuntu source tree git respository for 16.04 and apply it there [15:32] knowhow, have you considered using the low-latency kernel? [15:32] of course there is almost no way it will apply to anything other than 4.9 which is not a kernel base ubuntu picked [15:32] i allready have it installed ..that came in a deb format [15:33] you have RT installed already in deb format ? [15:33] for a sound card it probably works....but i have a cnc on a parallel port [15:33] no not rt [15:33] low-latency but from what i hear it is not good enough [15:34] not good enough for what [15:34] and i am not going to compile linuxcnc against that...i need to build one from scratch apply rt patch to it and install it...so i can boot it with grub customizer when i need it [15:35] apw...i said in the rows above...but i will repeat....linuxcnc [15:35] knowhow, you may find using the low-latency kernel with the application running with the FIFO or RR real time scheduler with a high priority is probably more than good enough [15:35] i think you over estimate just how much information about you and your plan i have in my head [15:35] what? [15:35] but, if you want to build an rt version of an ubuntu kernel, you need to apply that patch to the ubuntu-kernel [15:36] apw are yoy some kind of inteligent AI? [15:36] you are runnig on your system, and then build it [15:36] am i talking to a real person here? [15:36] knowhow, apw is a real person for sure [15:36] ok lets start from begining [15:36] that wiki link tells you how to get the git repositories you need to do that [15:36] and you had better know how to apply a patch if you are going to succeed in this endevour [15:36] i have the linuxcnc source...the sitee with instructions how to instaall it [15:37] i reached the step where the new kernel has to be prepared...i cannot screw up the one allready installed...just in case something goes horribly wrong [15:37] knowhow, that site states "optionally with realtime extensions", so you may not need RT [15:38] i can boot from the one not touched [15:38] i need it for operating the real machine...i do not need it for simulations [15:39] "simple installation on ubuntu" if you have to build your own kernel and apply rt to make that happen it is falling far short of "simple" [15:39] it will cut itself instead of the metal piece....and guys you dont want skynet in real life do you?...so help a stranger here :) [15:40] i need to know what source do i download for the patch and from where? [15:40] as i say if what you want is an ubuntu kernel with RT you need to get the kernel source git repo for your series, you need to get the nearest RT patch set, and you need to apply it [15:40] i will give again the link for the patch [15:40] https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/ [15:41] and in the list is the newest 4.9.30.rc1 [15:42] right but ubuntu doesn't have a 4.9. kernel, so if you want to have an ubuntu base you will need to apply it to 4.8 or 4.10, or use a 4.4 base with the rt for that [15:42] this wwill make the kernel realtime...now i need to know which is the newest stable kernel for ubuntu that can accept this patch...and the commands to apply it safely...and then compile it [15:42] and after that installing it of course...or make it a deb file for later easy use [15:43] there seems to be 4.4 and 4.8 bases for RT so as long as you are running one of those [15:43] i would recomment you use the latest RT for one of those [15:43] right now i run 4.7...on ubuntu which gets updated...regularily [15:43] as 16.04 was xenial and the GA kernel was 4.4, that would seem to be a lodical choice [15:43] knowhow, then you arn't running a kernel we produce [15:44] and you wouldn't be getting updates for 4.7 either (from us) [15:44] yeah but linus....updated it and i just grabbed the newest deb there...generic...and the low latency [15:45] they both work fine...but where do i get the latest stable generic source so i can apply this patch to it? [15:46] i instaalled them both...and with grub customizer i can boot which one i want [15:46] where do you get your mainline .debs now ? [15:46] as they should be offering you some route to source in those [15:46] hhold on i probably have the links somewhere [15:46] as i am sure it is not linus who is producing .debs, he is a fedora man iirc [15:48] aah no..no no....this got to be a serrious joke....is the latest stable version 4.11 ?....i checked the site two months ago and it was 4.7 or something [15:48] anyway here s the link [15:48] https://www.kernel.org/ [15:49] and yo got a .deb form there ? [15:49] but the rt patch does not get updated that quickly so i am going to use the kernel coresponding its version [15:50] v4.8 was released in october last year [15:51] so if you are going to try and construct a kernel source at 4.9.30 then i would clone linus' tree, checkout v4.9 and apply the delta patch from kernel.org [15:53] apw....it is seems aliens or mr smith from matrix ate the link from which i .... [15:54] * cking notes one can build .debs from the kernel source using: make deb-pkg INSTALL_MOD_STRIP=1 [15:54] anyway ....i just got my eyes opened with a can opener [15:55] cking how do i do that ? by including ALL and imean ALL the dependencies that a ffresh install of ubuntu without internet conection? [15:56] i do not find the 4.9.30 version on here [15:56] https://www.kernel.org/ [15:56] knowhow, pick the URL for the patch for 4.9.32 and adjust it to say 30 [15:57] apw...won't that screw up shite up...the patch is intended for 4.9.30 man...i do not want to ruin my ubuntu installation [15:58] knowhow: use this link - https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.9.30.tar.xz [15:58] i do not know how to remove bad kernels yet [15:58] knowhow, no i am saying to make the url like sforshee has said [15:59] ok are you a wizard or something? how did you find that? [15:59] he took the URL which is published there for any version on the page, and changed the version to the one you wanted [15:59] right, copy the link on git.kernel.org for 4.9.32, then paste it and change the 32 to a 30 [16:00] but it doesnt show on the page [16:00] right click the link and select the "copy" option [16:01] knowhow: it seems linuxcnc has a custom distribution you can install, why aren't you just using that? [16:01] well...for now i just found the source for my patch and i am going blind for a month now :| [16:01] knowhow: http://linuxcnc.org/docs/2.7/html/getting-started/getting-linuxcnc.html#_installing_linuxcnc [16:02] or http://linuxcnc.org/docs/2.7/html/getting-started/getting-linuxcnc.html rather [16:02] sforshee....no no no my man....no bad advice there...i have a 8 core laptop ...and i want to friggin' use it [16:03] i've been recommended that but i do not want to fill my house with computers [16:03] heh that will take a few hours to build it i suspect [16:03] and is old anyway [16:04] apw...yeah but that is why 8 cores and 32 gigs of ram should make it just couple of hours...but after that i have it for life [16:04] no? [16:05] it is a dell precision m6600 whith the docking station [16:05] * apw tries to imagine a kernel lasting weeks ... heh [16:06] when i do not operate the cnc machine i use the low latency kernel to develop the open source sound card [16:06] on low spec pcs probably lasts a day or so... [16:07] knowhow: if you use that link and apply the rt patch that should get you the source tree you need, building is much much more difficult than that though if you've never done it before [16:08] i have allready downloaded it and the patch too...i am going to apply it and compile it tomorow..beacause now it is evening [16:08] sforshee, to be fair, if one slams in the nearest ubuntu config into that tree and then build with make deb-pkg INSTALL_MOD_STRIP=1 it may just spit out the required devs [16:08] *debs [16:09] cking: only if there are not config changes needed [16:09] sforshee, true, but one just answers the relevant questions on the config step and one is done [16:10] do i need anything else to know? [16:10] you may need several GB free to do the build [16:10] the steps should be apply patch....compile kernel....and make deb file for easy install in the future [16:11] knowhow, the make dep-pkg step will build and package the kernel in one go [16:11] *deb-pkg [16:11] cking...say what?....gigs???....what i just downloaded has 88 megs....and the patch has 300 kilos [16:12] cking i allready cpied that line in a notepad file :))) [16:13] several gigs more like 30GB [16:13] yeah, my last build was 28GB [16:13] ckiing....i have a terabyte of sda1 and another one sda2 that means two terabytes....but... [16:14] how in the name of actual f u k ....can a 88 megs and 300 kilos can take 30 gigs? [16:14] i dont know...i am just asking [16:15] as i mentioned...i have the space...but....nimoy would say that this is highly ilogical [16:15] how can 88 megs...ok 89....can become 30 GB? [16:16] knowhow: 88 megs is compressed, the source is much bigger after decompression, then all the files produced by building take up space [16:16] knowhow, and compiled code with all the intermediate object code uses up a load of space [16:16] i know it takes more space....but THAT much? [16:17] i compiled files for microcontrolllers....and they take less space after compilation [16:18] oh i forgot about the object code....but stil...THAT much?... [16:18] ..and there a many modules being built = lots of code [16:19] i played with c++ on win but....30 gigs...you mean my deb file is going to be 30 gigs? [16:19] :| [16:19] no [16:20] knowhow, no, the final deb files will be tens of MB, don't fret over the intermediate data being generated during a build, it's the final product you need to concern yourself about [16:21] ooh those are going to be like a buffer in the process of compilation and after it finshes it erases that...intermediate data yeah....that was the expression [16:21] i set the swap file to the size of the ram anyway for a good measure [16:21] :))) [16:22] that is 32 gigs [16:22] it will probably never be used [16:23] so do i put the gz patch in the same directory of the 88 megs file that i just downloaded? [16:23] the rt one that is? [16:23] cking [16:23] ? [16:25] well in a copy version of course just in case murphy shows up to see what i am up to?...as usual :)) [16:25] knowhow, you uncompress and untar the kernel source, cd into the kernel source then apply the patch using patch -p1 < name-of-patch-file [16:26] if you are in luck, the patch will apply cleanly and then you can build the kernel [16:26] but the gz patch has to be in the uncompressed 88 megs folder right?...well the bigger version after decompression [16:26] do i need to check for build-deps or something? [16:27] i allready checked for linuxcnc from the linux cnc source folder [16:27] so that is prepared [16:28] knowhow, to be clear: [16:28] yes? [16:28] uncompress the kernel source and untar it [16:28] ok [16:28] uncompress the patch, and to make things easy move it into the kernel source directory [16:28] uncompress=untar [16:28] what i do after that with the rt patch gz file? [16:29] with the uncompressed patch file, apply it using patch -p1 < name-of-patch-file (what ever the name of the file is) [16:30] if you are in luck it will apply cleanly [16:30] but where the gz file needs to be for that command to be successful?...in the linux kernel source folder?...the one i just untared? [16:30] yep, in the kernel source directory [16:31] i am not to good with cli paths so i use graphic mode to copy everything in one place [16:31] ok got it [16:33] knowhow, you need some cli know-how to apply patches, get the .config sorted and to build and install the kernel for sure [16:35] well after the patch is going applied well....that is the next order in line...and probably the site i got the instructions from will explain more on that part [16:36] but first in row....thank you apw....for the linux 4.30 kernel source file that coresponds with my patch [16:37] and then some serios thanks goes to you cking...for explaining some basic command and file ,folder placements that for the expert is trivial but for a newbie like me isnt [16:39] i copied the steps of placing everything where it needs to be too...in the notepad...so i do not have to ask again the same things later [16:40] lets hope it all works out OK :-) [16:40] so big thanks for apw....cking...and the rest of the forum for the courtesy shown [16:41] i hope so too...if not...i will sure come crying like a litle big baby back here again :))) [16:41] if it dosnt i am not ruining a good installed kernel anyway [16:42] thanks again....and i will come again for the updates about the issue [16:44] cool [23:38] Add new softlockup stressor, use with caution(!) ??? [23:53] how can i install Unstable updated to 4.12-rc5 on 17.04 ?????