/srv/irclogs.ubuntu.com/2017/06/15/#ubuntu-kernel.txt

alkisgcking, apw, hi, I currently have remote access to the school that had that disk issue. The exact symptoms are that disk writes start at e.g. 200 MB/sec on an HDD, and after some time, they become 2 MB/sec, without going back to 200 MB even if we leave it for hours (was suspecting temperatures). This happens on 4.4 and 4.8 (xenial), but not on 3.2 (precise) or 4.2 (trusty).  I've seen this in another case a few months ago, but I had dismissed it 10:08
alkisgWhat I'd like to ask is, apart from `ubuntu-bug linux`, should I test or collect something else, since I won't frequently have remote access to the school?10:08
alkisgMy test case is dd if=/dev/zero of=test bs=1M count=1000 conv=fdatasync; then cp -a some-20gb-folder; then dd again => and there it's slow and never gets fast again10:08
ckingalkisg, so what happens if after that initial dd you sync and re-run that test?10:12
alkisgcking: it degrades gradually, e.g. starts from 200, then 190, then 100...10:13
alkisgIt also happens on their SSD, it's not related to the media10:14
ckingalkisg, so I guess what is happening is that initially the writes fill up the cache and so it appears to be writing fast, but as the cache gets full you get band width limited to the speed of the block device 10:14
alkisgcking: echo 3 > /proc/sys/vm/drop_caches would tell me that, wouldn't it?10:15
alkisgIt's an amazing difference, 200 mb/sec to 2 mb/sec, which makes the system completely unusable10:15
alkisgI don't think it's related to caching, I've never seen something similar except for those 2 cases...10:16
alkisgAnd, it happens only with 4.4+, not with previous kernels10:16
ckingalkisg, so, lets factor out the cache and just do the dd with conv=direct and see what speed you get10:17
alkisgcking: ok, let me instruct the school to reboot with 4.8, as now it's with a trusty live cd that works fine. I can also give you reverse vnc if you like. Reporting in 2 mins...10:17
apwcking, note these are CD boots so with whatever is on the media10:18
alkisg(of course I'm dd'ing to the disk, not to RAM/overlayfs...)10:19
* cking finds a fungible machine to try this out too10:20
alkisgI've tested about 100 installation so far, so this issue should be quite rare...10:20
ckingalkisg, so whats the failure rate, e.g how many times have you seen it not working efficiently?10:24
alkisgcking: it 100% reproducable, what varies is that it might need 20 GB of writes before it starts to happen, or 80 GB of writes10:25
ckingalkisg, I mean, of all the machines it's been installed on, how many exhibit this issue?10:25
alkisgI've only seen that issue in 2 machines. I've installed 16.04 in about 100 machines so far.10:26
ckingalkisg, if you can add the machine info to the bug report than that may give some hint on what kind of common H/W issue it is related to10:26
alkisgcking: I've remote access to their xenial/4.8 installation now, freshly booted (it's not a live cd). I can give you reverse vnc if you like. Should I start with some test, or with ubuntu-bug linux?10:27
ckingreporting the bug 1st would be a good start10:28
ubot5bug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] https://launchpad.net/bugs/110:28
alkisgty, doing so...10:28
alkisghttps://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/169811810:44
ubot5Ubuntu bug 1698118 in linux-hwe (Ubuntu) "Slow disk writes after some uptime, only on certain hw and 4.4+ kernels" [Undecided,New]10:44
=== JanC is now known as Guest53242
=== JanC_ is now known as JanC
=== MAbeeTT_ is now known as MAbeeTT
knowhowgood day15:23
knowhowhow do i patch a kernel? with an rt patch15:24
knowhow?15:24
knowhowi donloaded the newest rt patch15:25
knowhowand something that looked as the kernel from kernel .org15:25
knowhowin a gz format15:25
knowhowof course the thing that seemed to be the kernel isnt the kernel...it is just a source file..15:26
apwknowhow, if you want to apply a patch, that applies to a source tree15:27
apwknowhow, if you want to apply it to the ubuntu kernel, you want to get the kernel for the series on which15:27
knowhowi am not a linux expert...so here what i did15:27
apwknowhow, you intend to patch, preferably by checking out the git repo for that series15:27
knowhowi am clueless about what you just said15:28
knowhowi know what git is if that makes a difference15:28
apwknowhow, https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel15:28
apwknowhow, you are trying to do something deeply hard, and deeply distro specific, with a very big patchset15:28
apwknowhow, you would want to have a good knowledge of the platform before even considering attempting something like that15:29
knowhowthis is what i have to apply15:29
knowhowhttps://www.kernel.org/pub/linux/kernel/projects/rt/4.9/15:29
knowhowit is the 4.30.rc1 way down in the list15:30
knowhowi am on ubuntu 16.04 am i am used with .deb files15:30
knowhowso what do i download from where to apply this rt patch to?15:30
knowhowbeacause this:15:31
knowhowhttps://www.kernel.org/pub/linux/kernel/v4.x/stable-review/15:31
knowhowcontains no kernel source code15:31
knowhowit is just a file in a gz format15:31
knowhowi donloaded them from graphical environment...from firefox that is15:32
apwknowhow, depends what you are trying to achieve, if you want to apply it to an ubuntu kernel, you would want to pick the ubuntu source tree git respository for 16.04 and apply it there15:32
ckingknowhow, have you considered using the low-latency kernel?15:32
apwof course there is almost no way it will apply to anything other than 4.9 which is not a kernel base ubuntu picked15:32
knowhowi allready have it installed ..that came in a deb format15:32
apwyou have RT installed already in deb format ?15:33
knowhowfor a sound card it probably works....but i have a cnc on a parallel port15:33
knowhowno not rt15:33
knowhowlow-latency but from what i hear it is not good enough15:33
apwnot good enough for what15:34
knowhowand i am not going to compile linuxcnc against that...i need to build one from scratch apply rt patch to it and install it...so i can boot it with grub customizer when i need it15:34
knowhowapw...i said in the rows above...but i will repeat....linuxcnc15:35
ckingknowhow, you may find using the low-latency kernel with the application running with the FIFO or RR real time scheduler with a high priority is probably more than good enough 15:35
apwi think you over estimate just how much information about you and your plan i have in my head15:35
knowhowwhat?15:35
apwbut, if you want to build an rt version of an ubuntu kernel, you need to apply that patch to the ubuntu-kernel15:35
knowhowapw are yoy some kind of inteligent AI?15:36
apwyou are runnig on your system, and then build it15:36
knowhowam i talking to a real person here?15:36
ckingknowhow, apw is a real person for sure15:36
knowhowok lets start from begining15:36
apwthat wiki link tells you how to get the git repositories you need to do that15:36
apwand you had better know how to apply a patch if you are going to succeed in this endevour15:36
knowhowi have the linuxcnc source...the sitee with instructions how to instaall it15:36
knowhowi reached the step where the new kernel has to be prepared...i cannot screw up the one allready installed...just in case something goes horribly wrong15:37
ckingknowhow, that site states "optionally with realtime extensions", so you may not need RT15:37
knowhowi can boot from the one not touched15:38
knowhowi need it for operating the real machine...i do not need it for simulations15:38
apw"simple installation on ubuntu" if you have to build your own kernel and apply rt to make that happen it is falling far short of "simple"15:39
knowhowit will cut itself instead of the metal piece....and guys you dont want skynet in real life do you?...so help a stranger here :)15:39
knowhowi need to know what source do i download for the patch and from where?15:40
apwas i say if what you want is an ubuntu kernel with RT you need to get the kernel source git repo for your series, you need to get the nearest RT patch set, and you need to apply it15:40
knowhowi will give again the link for the patch15:40
knowhowhttps://www.kernel.org/pub/linux/kernel/projects/rt/4.9/15:40
knowhowand in the list is the newest 4.9.30.rc115:41
apwright but ubuntu doesn't have a 4.9.<anything> kernel, so if you want to have an ubuntu base you will need to apply it to 4.8 or 4.10, or use a 4.4 base with the rt for that15:42
knowhowthis wwill make the kernel realtime...now i need to know which is the newest stable kernel for ubuntu that can accept this patch...and the commands to apply it safely...and then compile it15:42
knowhowand after that installing it of course...or make it a deb file for later easy use15:42
apwthere seems to be 4.4 and 4.8 bases for RT so as long as you are running one of those15:43
apwi would recomment you use the latest RT for one of those15:43
knowhowright now i run 4.7...on ubuntu which gets updated...regularily15:43
apwas 16.04 was xenial and the GA kernel was 4.4, that would seem to be a lodical choice15:43
apwknowhow, then you arn't running a kernel we produce15:43
apwand you wouldn't be getting updates for 4.7 either (from us)15:44
knowhowyeah but linus....updated it and i just grabbed the newest deb there...generic...and the low latency15:44
knowhowthey both work fine...but where do i get the latest stable generic source so i can apply this patch to it?15:45
knowhowi instaalled them both...and with grub customizer i can boot which one i want15:46
apwwhere do you get your mainline .debs now ?15:46
apwas they should be offering you some route to source in those15:46
knowhowhhold on i probably have the links somewhere15:46
apwas i am sure it is not linus who is producing .debs, he is a fedora man iirc15:46
knowhowaah no..no no....this got to be a serrious joke....is the latest stable version 4.11 ?....i checked the site two months ago and it was 4.7 or something15:48
knowhowanyway here s the link15:48
knowhowhttps://www.kernel.org/15:48
apwand yo got a .deb form there ?15:49
knowhowbut the rt patch does not get updated that quickly so i am going to use the kernel coresponding its version15:49
apwv4.8 was released in october last year15:50
apwso if you are going to try and construct a kernel source at 4.9.30 then i would clone linus' tree, checkout v4.9 and apply the delta patch from kernel.org15:51
knowhowapw....it is seems aliens  or mr smith from matrix ate the link from which i ....15:53
* cking notes one can build .debs from the kernel source using: make deb-pkg INSTALL_MOD_STRIP=115:54
knowhowanyway ....i just got my eyes opened with a can opener15:54
knowhowcking how do i do that ? by including ALL and imean ALL the dependencies that a ffresh install of ubuntu without internet conection?15:55
knowhowi do not find the 4.9.30 version on here15:56
knowhowhttps://www.kernel.org/15:56
apwknowhow, pick the URL for the patch for 4.9.32 and adjust it to say 3015:56
knowhowapw...won't that screw up shite up...the patch is intended for 4.9.30 man...i do not want to ruin my ubuntu installation15:57
sforsheeknowhow: use this link - https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.9.30.tar.xz15:58
knowhowi do not know how to remove bad kernels yet15:58
apwknowhow, no i am saying to make the url like sforshee has said15:58
knowhowok are you a wizard or something? how did you find that?15:59
apwhe took the URL which is published there for any version on the page, and changed the version to the one you wanted15:59
sforsheeright, copy the link on git.kernel.org for 4.9.32, then paste it and change the 32 to a 3015:59
knowhowbut it doesnt show on the page16:00
sforsheeright click the link and select the "copy" option16:00
sforsheeknowhow: it seems linuxcnc has a custom distribution you can install, why aren't you just using that?16:01
knowhowwell...for now i just found the source for my patch and i am going blind for a month now :|16:01
sforsheeknowhow: http://linuxcnc.org/docs/2.7/html/getting-started/getting-linuxcnc.html#_installing_linuxcnc16:01
sforsheeor http://linuxcnc.org/docs/2.7/html/getting-started/getting-linuxcnc.html rather16:02
knowhowsforshee....no no no my man....no bad advice there...i have a 8 core laptop ...and i want to friggin' use it16:02
knowhowi've been recommended that but i do not want to fill my house with computers16:03
apwheh that will take a few hours to build it i suspect16:03
knowhowand is old anyway16:03
knowhowapw...yeah but that is why 8 cores and 32 gigs of ram should make it just couple of hours...but after that i have it for life16:04
knowhowno?16:04
knowhowit is a dell precision m6600 whith the docking station16:05
* apw tries to imagine a kernel lasting weeks ... heh16:05
knowhowwhen i do not operate the cnc machine i use the low latency kernel to develop the  open source sound card16:06
knowhowon low spec pcs probably lasts a day or so...16:06
sforsheeknowhow: if you use that link and apply the rt patch that should get you the source tree you need, building is much much more difficult than that though if you've never done it before16:07
knowhowi have allready downloaded it and the patch too...i am going to apply it and compile it tomorow..beacause now it is evening16:08
ckingsforshee, to be fair, if one slams in the nearest ubuntu config into that tree and then build with  make deb-pkg INSTALL_MOD_STRIP=1 it may just spit out the required devs16:08
cking*debs16:08
sforsheecking: only if there are not config changes needed16:09
ckingsforshee, true, but one just answers the relevant questions on the config step and one is done16:09
knowhowdo i need anything else to know?16:10
ckingyou may need several GB free to do the build16:10
knowhowthe steps should be apply patch....compile kernel....and make deb file for easy install in the future16:10
ckingknowhow, the make dep-pkg step will build and package the kernel in one go16:11
cking*deb-pkg16:11
knowhowcking...say what?....gigs???....what i just downloaded has 88 megs....and the patch has 300 kilos16:11
knowhowcking i allready cpied that line in a notepad file :)))16:12
ivanseveral gigs more like 30GB16:13
ckingyeah, my last build was 28GB16:13
knowhowckiing....i have a terabyte of sda1 and another one sda2 that means two terabytes....but...16:13
knowhowhow in the name of actual f u k ....can a 88 megs and  300 kilos can take 30 gigs?16:14
knowhowi dont know...i am just asking16:14
knowhowas i mentioned...i have the space...but....nimoy would say that this is highly ilogical16:15
knowhowhow can 88 megs...ok 89....can become 30 GB?16:15
sforsheeknowhow: 88 megs is compressed, the source is much bigger after decompression, then all the files produced by building take up space16:16
ckingknowhow, and compiled code with all the intermediate object code uses up a load of space16:16
knowhowi know it takes more space....but THAT much?16:16
knowhowi compiled files for microcontrolllers....and they take less space after compilation16:17
knowhowoh i forgot about the object code....but stil...THAT much?...16:18
cking..and there a many modules being built = lots of code16:18
knowhowi played with c++ on win but....30 gigs...you mean my deb file is going to be 30 gigs?16:19
knowhow:|16:19
ivanno16:19
ckingknowhow, no, the final deb files will be tens of MB, don't fret over the intermediate data being generated during a build, it's the final product you need to concern yourself about16:20
knowhowooh those are going to be like a buffer in the process of compilation and after it finshes it erases that...intermediate data yeah....that was the expression16:21
knowhowi set the swap file to the size of the ram anyway for a good measure16:21
knowhow:)))16:21
knowhowthat is 32 gigs16:22
knowhowit will probably never be used16:22
knowhowso do i put the gz patch in the same directory of the 88 megs file that i just downloaded?16:23
knowhowthe rt one that is?16:23
knowhowcking16:23
knowhow?16:23
knowhowwell in a copy version of course just in case murphy shows up to see what i am up to?...as usual :))16:25
ckingknowhow, you uncompress and untar the kernel source, cd into the kernel source then apply the patch using patch -p1 < name-of-patch-file16:25
ckingif you are in luck, the patch will apply cleanly and then you can build the kernel16:26
knowhowbut the gz patch has to be in the uncompressed 88 megs folder right?...well the bigger version after decompression16:26
knowhowdo i need to check for build-deps or something?16:26
knowhowi allready checked for linuxcnc from the linux cnc source folder16:27
knowhowso that is prepared16:27
ckingknowhow, to be clear:16:28
knowhowyes?16:28
ckinguncompress the kernel source and untar it16:28
knowhowok16:28
ckinguncompress the patch, and to make things easy move it into the kernel source directory16:28
knowhowuncompress=untar16:28
knowhowwhat i do after that with the rt patch gz file?16:28
ckingwith the uncompressed patch file, apply it using patch -p1 < name-of-patch-file (what ever the name of the file is)16:29
ckingif you are in luck it will apply cleanly16:30
knowhowbut where the gz file needs to be for that command to be successful?...in the linux kernel source folder?...the one i just untared?16:30
ckingyep, in the kernel source directory16:30
knowhowi am not to good with cli paths so i use graphic mode to copy everything in one place 16:31
knowhowok got it16:31
ckingknowhow, you need some cli know-how to apply patches, get the .config sorted and to build and install the kernel for sure16:33
knowhowwell after the patch is going applied well....that is the next order in line...and probably the site i got the instructions from will explain more on that part16:35
knowhowbut first in row....thank you apw....for the linux 4.30 kernel source file that coresponds with my patch16:36
knowhowand then some serios thanks goes to you cking...for explaining some basic command and file ,folder placements that for the expert is trivial but for a newbie like me isnt16:37
knowhowi copied the steps of placing everything where it needs to be too...in the notepad...so i do not have to ask again the same things later16:39
ckinglets hope it all works out OK :-)16:40
knowhowso big thanks for apw....cking...and the rest of the forum for the courtesy shown16:40
knowhowi hope so too...if not...i will sure come crying like a litle big baby back here again :)))16:41
knowhowif it dosnt i am not ruining a good installed kernel anyway16:41
knowhowthanks again....and i will come again for the updates about the issue16:42
ckingcool16:44
immuAdd new softlockup stressor, use with caution(!) ??? 23:38
immuhow can i install Unstable updated to 4.12-rc5 on 17.04 ?????23:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!