| === cyberzeus_ is now known as cyberzeus | ||
| s10gopal | jsalisbury, it is also bad | 03:09 |
|---|---|---|
| s10gopal | jsalisbury, can you please teach me how to build kernel for test ? | 07:47 |
| s10gopal | apw, ^ | 07:48 |
| apw | s10gopal, this page has the broad methods: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel | 07:50 |
| apw | s10gopal, but it is differennt if you are building a mainline one, there you might well use make kpkg | 07:50 |
| apw | s10gopal, but also unless you have a huge box, it takes many hours for a build | 07:51 |
| apw | s10gopal, the one we build test kernels on has over 200 cores | 07:51 |
| apw | s/cores/cpus | 07:51 |
| TJ- | apw: that's more cores than an apple tree! | 07:51 |
| s10gopal | i have core i5 6th gen and 128gb ssd , how much time it is going to take ? | 07:51 |
| s10gopal | 2 physical core | 07:52 |
| apw | i would be supprised if that is not in the range of 4-6 hours | 07:52 |
| apw | that would count as "tiny" on the scale of a machine to build the kernel | 07:53 |
| TJ- | s10gopal: I believe jsalisbury was planning on building the bisected kernel packages for you, you just need to report back in the LP bug report each time as to whether the last test was successful or not | 07:53 |
| apw | which is why having j-salisbury help you | 07:53 |
| apw | should be speeding things up | 07:53 |
| s10gopal | i thought , if i build kernel myself i dont have to wait for him to come online | 07:53 |
| TJ- | s10gopal: if you use the LP bug report to leave comments you don't need to wait for him either, he'll build a new package and comment in the bug with a link to the download location when it is ready | 07:55 |
| apw | s10gopal, indeed you would not though i am not convinced you could get a kernel built before he comes online anyhow | 07:56 |
| s10gopal | apw, everyone can access that super computer or just developers can ? | 08:08 |
| apw | s10gopal, that is an internal system | 08:09 |
| lorddoskias | compiling an upstream kernel with gcc-5 (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609 results in thread_union type being erased from the resulting kernel image. When i try compiling with gcc-7 (Ubuntu 7.2.0-1ubuntu1~16.04) 7.2.0 i thread_union is there? | 09:56 |
| apw | lorddoskias, erased in what sense ? | 10:21 |
| lorddoskias | apw: in the sense that if i open the resulting vmlinux with gdb and write "ptype union thread_union" i get "no such type" | 10:22 |
| lorddoskias | and as a result the "crash" utility is not working correctly as well | 10:22 |
| apw | lorddoskias, that does sound like a bug (or limitation) in the older compiler | 10:28 |
| lorddoskias | well the older compiler is whatever is shipped as lts for ubuntu 16.04.04 | 10:28 |
| lorddoskias | so in fact it is the supported compiler | 10:28 |
| lorddoskias | so the 5.4 has retpoline support enabled, whereas the 7.2.0 doesn't - the 16.04 backport is from 2017 | 10:29 |
| apw | lorddoskias, indeed, debugging with older compilers might be hard, though i assume you can dump that thing out | 10:30 |
| apw | as you do know the type it contains | 10:30 |
| lorddoskias | well given that the oldest kernel supported by the upstream kernel is 4.8 i won't call 5.4 old | 10:31 |
| lorddoskias | true, there is gcc 8 release approaching but still | 10:31 |
| apw | it builds it just fine, right | 10:31 |
| lorddoskias | build wise it's ok | 10:32 |
| apw | and you know which of those three things are in there, in your context | 10:32 |
| apw | so you can just dump the address as which ever it is | 10:32 |
| apw | i assume it is a lack of representation of unions, though i can find no direct evidence of that being improved in later compilers in a quick search | 10:33 |
| lorddoskias | but ptype hpet_lock works. and hpet_lock is defined as: arch/x86/kernel/hpet.c:union hpet_lock { | 10:34 |
| lorddoskias | ptype union ftrace_op_code_union also works | 10:35 |
| lorddoskias | so it's not being applied to all unions | 10:35 |
| apw | lorddoskias, from your 7.2 print of the type, does it contain anything other than stack, ie are the other optional fields in there | 10:36 |
| apw | lorddoskias, i am wondering if it is because it is a union with only one member | 10:37 |
| lorddoskias | let me rebuild and see | 10:37 |
| apw | as the others you list all definatly have more than one member | 10:37 |
| apw | it is possible there is only one member, so the type is optimised away | 10:38 |
| lorddoskias | i think this is an invalid optimisation, no ? | 10:39 |
| lorddoskias | and in fact here we are talking about debug info | 10:39 |
| apw | depends how you look at it, the type is technically redundant if it only has one member, the inner type can be promoted | 10:40 |
| apw | so if the optimiser did that, then there would be no references to the type in the emitted code | 10:40 |
| lorddoskias | so how come the newer compiler with presumably more advanced optimisation logic omits this optimisation but the older one doesn't? | 10:40 |
| apw | and so emitting the type would be a waste of space in the debug info | 10:40 |
| lorddoskias | but if i specifically configured the kernel to produce debug info then presumably i don't care much about optimisations,esp of the debug info | 10:41 |
| lorddoskias | that sounds ludicrous | 10:41 |
| lorddoskias | type = union thread_union { | 10:41 |
| lorddoskias | struct task_struct task; | 10:41 |
| lorddoskias | unsigned long stack[2048]; | 10:41 |
| lorddoskias | } | 10:41 |
| apw | so not that then | 10:41 |
| lorddoskias | so you are not right, because there are 2 members | 10:41 |
| apw | indeed | 10:41 |
| apw | it was a theory, you tested it, it failed | 10:42 |
| apw | that is the nature of theories | 10:42 |
| lorddoskias | also i have : # CONFIG_DEBUG_INFO_REDUCED is not set | 10:42 |
| lorddoskias | so the compiler should really be even more verbose | 10:42 |
| apw | the compiler is still only going to include what it thinks it needs, and it is clearly wrong in this case | 10:43 |
| apw | as many files include headers with lots of types, which you don't use | 10:43 |
| apw | so including them all would be dumb, it obviously thinks it can remove that one, incorrectly | 10:43 |
| lorddoskias | also i've been compiling kernels with older version of the LTS compiler | 10:43 |
| apw | but also in this case, you really have a clear way to dump this | 10:43 |
| lorddoskias | ah, freenode, being free node | 10:46 |
| apw | but also in this case, you really have a clear way to dump this | 10:46 |
| apw | so i don't see a huge amount of value in actually caring | 10:46 |
| lorddoskias | apw: what do you mean by having clear way to dupm this | 10:46 |
| lorddoskias | apw: the amount of value caring is the fact that crash, the utility which is used to debug kernel crashes | 10:46 |
| apw | by definition with a full union of this nature, you as dumper needs | 10:47 |
| lorddoskias | relies on thread_union being defined in order for it to adjust internal offsets and be able to dump stacks | 10:47 |
| apw | to know which of the N types it contains | 10:47 |
| apw | so you can dump it yourself no? | 10:47 |
| lorddoskias | what do you mean by dump it? you keep insisting on this? | 10:47 |
| apw | it is either a thread_thing or it is the raw stack | 10:47 |
| apw | why do you care this type exists or not | 10:48 |
| apw | presumably to display the contents | 10:48 |
| lorddoskias | the structure is needed so tthat programatically the stack size can be deduced | 10:48 |
| lorddoskias | because crash adjusts its internal stack size and is able to dump tasks stack? | 10:48 |
| apw | the stack size is also defined by STACK_SIZE no ? | 10:48 |
| apw | s/STACK_SIZE/THREAD_SIZE | 10:48 |
| apw | and on all of the architectures we support stacks grown downwards | 10:49 |
| lorddoskias | it is defined based on THREAD_SIZE / target arch size of ulong | 10:49 |
| apw | well it is defined as unsigned long foo[THREAD_SIZE/sizeof(long)] so it is exactly | 10:50 |
| apw | THREAD_SIZE bytes long | 10:50 |
| apw | on the presumption that none of the other things are bigger than that of course | 10:50 |
| lorddoskias | huhz | 10:50 |
| lorddoskias | so you are going to try everything than admit that the compiler is broken and basically you (i mean ubuntu, not you personally ) are not interested in fixing this? | 10:50 |
| lorddoskias | is that your position? | 10:51 |
| apw | no i am trying to help you get past this blocker to you | 10:51 |
| apw | i am sure the compiler is broken, if the newer one does the right thigns on the same code | 10:51 |
| apw | if you file a bug someone might be able to fix it, but the reality is | 10:51 |
| lorddoskias | i can get past this blocker by hacking around crash but that's not the point, the point is that ubuntus LTS toolchain should be producing working debug info, no ? | 10:51 |
| apw | that fixing the compiler contains risk | 10:51 |
| apw | and doing that for something which only affects debugging information is going to | 10:52 |
| apw | be a non-obvious risk balance | 10:52 |
| lorddoskias | what if i prove an earlier version of the LTS compiler worked, and that the current one is a regression ? | 10:52 |
| lorddoskias | what then? | 10:52 |
| apw | not that that is my call as a kernel engineer | 10:52 |
| apw | then that is a more compelling case to fix it | 10:53 |
| apw | but overall it would be a decision for the compiler maintainer not myself, so i wouldn't | 10:53 |
| apw | read a whole heap into my oppinion. updating compilers always makes me afraid | 10:53 |
| apw | my only care was to see if i could help you get past it without needing to fix it | 10:54 |
| apw | as that is not going to take 0 time | 10:54 |
| apw | even if they don't won't fix the bug | 10:54 |
| apw | lorddoskias, do either way file a bug against gcc-5, just don't expect it fixed in short order | 10:55 |
| lorddoskias | apw: i have bitter experience with reporting bugs to ubuntu in that those bugs are rarely acted upon. for example 1605843 ... | 10:56 |
| apw | LP: #1605843 | 10:56 |
| ubot5` | Launchpad bug 1605843 in linux (Ubuntu) "Kernel crashes from time to time when using ftrace " [Medium,Confirmed] https://launchpad.net/bugs/1605843 | 10:56 |
| lorddoskias | where i've provided asm dumps of what's wrong and even the commit that's supposed to fix it so it would have been a matter of cherry picking it up | 10:56 |
| apw | a fix which if i am reading history right was applied and released in 4.4.0-67.88 | 10:58 |
| lorddoskias | dunno, i switched to the . releases of the lts so i'm currently suing: 4.13.0-37-generic | 10:59 |
| lorddoskias | so it works there, but at the time following multiple kernel releases the issue wasn't fixed | 10:59 |
| apw | it did get fixed, about a year back, though the bug does not appear to have been referenced | 11:00 |
| lorddoskias | it could have even been closed | 11:01 |
| apw | and i have done so, we have a very large volume of bugs, it is very easy to lose individual ones over time | 11:01 |
| lorddoskias | so what's the suggested way to downgrade a package ? | 11:03 |
| apw | i have tended to get them from the launchpad librarian and install them with dpkg | 11:05 |
| apw | https://launchpad.net/ubuntu/+source/gcc-5/+publishinghistory | 11:05 |
| lorddoskias | well apt-cache madison seems to be helpful | 11:06 |
| apw | that only shows you versions in the pool i assume | 11:07 |
| lorddoskias | huhz, but then this barfs about unmed dependencies huhz | 11:09 |
| dijuremo | apw: Arggg my coworker turned off the DELL T3610 where I was testing prime95 on the patched kernel 4.13.0-38.43+lp1759920 I am going to move it elsewhere and start again. As far as I can tell, he properly powered it off at around 3PM, so that means 8-ish hours without freezing. | 11:09 |
| === gpiccoli_ is now known as gpiccoli | ||
| s10gopal | how to extract .config from kernel? | 11:27 |
| apw | s10gopal, our configs are in /boot alongside the binary kernel | 11:29 |
| s10gopal | thx | 11:30 |
| s10gopal | !ping | 11:38 |
| ubot5` | pong! | 11:38 |
| s10gopal | how to fix it ? scripts/sign-file.c:25:30: fatal error: openssl/opensslv.h: No such file or directory | 12:00 |
| s10gopal | #include <openssl/opensslv.h> | 12:00 |
| apw | you need to build with the build-dependencies for the kernel installed | 12:00 |
| apw | normally you do that by building in a chroot with those installed into it | 12:00 |
| s10gopal | apw, can you please explain it ? | 12:01 |
| s10gopal | apw, i am using time make deb-pkg | 12:01 |
| cascardo | s10gopal: man debuild | 12:07 |
| bjf | s10gopal, https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel | 12:08 |
| s10gopal | thx | 12:10 |
| s10gopal | i am at CC [M] drivers/media/pci/ivtv/ivtvfb.o. how much time it will take to finish ? | 12:47 |
| apw | s10gopal, it builds in parallle, so hard to say, when i do a build it takes about 10m but they are scrolling past fast enough to be hard to read | 13:01 |
| === TJ_Remix is now known as TJ- | ||
| s10gopal | how to fix ./scripts/package/builddeb: line 33: dpkg-gencontrol: command not found make[1]: *** [deb-pkg] Error 127 make: *** [deb-pkg] Error 2 ? | 14:20 |
| cascardo | s10gopal: do you have dpkg-dev installed? | 14:22 |
| cascardo | s10gopal: apt install build-essential ; apt-get build-dep linux | 14:23 |
| s10gopal | cascardo, it was not , i need to do time make deb-pkg again ? | 14:23 |
| cascardo | s10gopal: if you are running make deb-pkg, you are doing it wrong to build an ubuntu kernel | 14:23 |
| s10gopal | cascardo, how to do it? | 14:24 |
| cascardo | if you want a deb package for a mainline kernel, then you might want to use make deb-pkg | 14:24 |
| cascardo | s10gopal: did you read the documentation pointed at you? | 14:24 |
| s10gopal | cascardo, yes , but not complete | 14:25 |
| s10gopal | in place of make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-custom i should use make -j4 `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-custom to use all my 4 core ? | 14:27 |
| s10gopal | cascardo, ^ | 14:28 |
| cascardo | s10gopal: what documentation did you get that from? | 14:31 |
| cascardo | certainly not from https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel | 14:31 |
| s10gopal | https://wiki.ubuntu.com/KernelTeam/GitKernelBuild , it is easy i can understand them | 14:31 |
| s10gopal | it is wrong? | 14:32 |
| cascardo | it depends on what you want to achieve | 14:32 |
| s10gopal | i want to find bad commit | 14:33 |
| cascardo | s10gopal: from an ubuntu kernel or a mainline kernel? | 14:33 |
| s10gopal | bisect v4.12 and v4.13-rc1 | 14:33 |
| cascardo | s10gopal: you only need to use make deb-pkg, then, if you want to install those kernels using deb packages | 14:34 |
| s10gopal | mainline , http://kernel.ubuntu.com/~kernel-ppa/mainline/ | 14:34 |
| cascardo | but, yes, probably easier to just use make deb-pkg in that case | 14:34 |
| s10gopal | i need to repeat the whole process again ? | 14:35 |
| s10gopal | time make deb-pkg ? and again it is going to take 100 min or it will be faster this time ? | 14:35 |
| cascardo | s10gopal: it's possible it's gonna just do the last steps, if you haven't run any clean command | 14:39 |
| s10gopal | and how to find the kernel which i made ? | 14:39 |
| cascardo | should be in the top dir | 14:40 |
| s10gopal | at $HOME ? | 14:42 |
| TJ- | deb-pkg should put the .deb files in ../ (parent directory) | 14:44 |
| TJ- | or if 'make bzImage' in ./arch/x86/boot/ | 14:45 |
| s10gopal | TJ-, a user on ##linux taught me how to build kernel from git , how i can find ##linux log ? | 14:45 |
| TJ- | s10gopal: I have no idea, where di you put it? | 14:46 |
| s10gopal | like #ubuntu logs are at http://irclogs.ubuntu.com/ | 14:47 |
| s10gopal | what is the difference between time make and make ? | 14:48 |
| TJ- | s10gopal: oh, thought you meant you'd kept a log locally of the commands. I don't know if ##linux logs, I think it's up to you to do it locally | 14:49 |
| s10gopal | thx | 14:50 |
| s10gopal | jsalisbury, i have learned how to make kernel , i should test kernel build by you or i can build and test ? | 14:51 |
| === himcesjf_ is now known as him-cesjf | ||
| dijuremo | May I pick you brain kernel gurus? I am looking at some RHEL VMs, the hypervisor is patched and I manually updated the microcode to the 20180312 release, but the VM still shows as vulnerable. Does QUEMU, etc need to be patched as well for the vulnerability to be mitigated? | 17:53 |
| apw | yes qemu needs to be updated to pass through the spec bits | 17:55 |
| dijuremo | --> runs to open RH support case.... | 17:55 |
| dijuremo | LOL | 17:55 |
| tyhicks | dijuremo: they may have patched everything but you might be using a model which doesn't support IBRS | 18:09 |
| tyhicks | dijuremo: to see if such a model is available, you can run 'virsh cpu-models x86_64 | grep IBRS' | 18:10 |
| tyhicks | (IBRS support comes as a separate, unique model so that live migration wasn't broken with the addition of IBRS support to the CPU model) | 18:11 |
| tyhicks | (you can also use <cpu mode='host-passthrough'> in your domain xml if you don't plan on migrating VMs to another machine) | 18:12 |
| dijuremo | # virsh -r cpu-models x86_64 | grep IBRS | 18:15 |
| dijuremo | Nehalem-IBRS | 18:15 |
| dijuremo | Westmere-IBRS | 18:15 |
| dijuremo | SandyBridge-IBRS | 18:15 |
| dijuremo | IvyBridge-IBRS | 18:15 |
| dijuremo | Haswell-noTSX-IBRS | 18:15 |
| dijuremo | Haswell-IBRS | 18:15 |
| dijuremo | Broadwell-noTSX-IBRS | 18:15 |
| dijuremo | Broadwell-IBRS | 18:15 |
| dijuremo | Skylake-Client-IBRS | 18:15 |
| dijuremo | So it seems it does support them, but I am not quite sure to force their use. | 18:16 |
| dijuremo | I do not manage the RHEV Manager machine and it is outdated, running 3.5 so that may be a problem. Perhaps they need to update that so that I can see Nehalem-IBRS since I only see the CPU types without -IBRS as available. | 18:16 |
| dijuremo | I am a proxy for this question, a friend asks related to the ibpb hangups that you guys fixed with the 4.13.0-38.43+lp1759920 patches in the Ubuntu kernel: which debian kernels have, or will have the fix, or maybe what to look for in the upstream kernel changelog? | 18:24 |
| tyhicks | the problematic patch that was sent around during the spectre embargo is "x86/mm: Only set IBPB when the new thread cannot ptrace current thread" | 18:32 |
| tyhicks | instead of using that patch, this upstream patch should be used: | 18:33 |
| tyhicks | https://git.kernel.org/linus/18bf3c3ea8ece8f03b6fc58508f2dfd23c7711c7 | 18:33 |
| dijuremo | tyhicks: Any idea how I can find out what the latest microcode version should be for a specific CPU, Intel(R) Xeon(R) CPU E5530? I tried expanding the intel-ucode folder on one of my RHEL servers, but after reboot it is not using IBPB/IBRS. dmesg reports: microcode: CPU0 sig=0x106a5, pf=0x1, revision=0x19 | 19:02 |
| tyhicks | dijuremo: you'll find that info here: https://newsroom.intel.com/wp-content/uploads/sites/11/2018/04/microcode-update-guidance.pdf | 19:03 |
| dijuremo | tyhicks: So I note there that they have the new microcode 0x1c but dmesg is reporting in this machine 0x19, likely old from the BIOS. Any idea where I may be able to find out how to force load the new microcode? | 19:08 |
| dijuremo | Oh I think I know, I need to rebuilt initrd, right/ | 19:10 |
| dijuremo | Nope... no luck... still showing revision=0x19 after reboot. | 19:14 |
| s10gopal | jsalisbury: can you please build a set of kernels so i can test them without waiting for you to come online ? | 19:28 |
| jsalisbury | s10gopal, The thing with a bisect is you need to know if the current test kernel is good or bad to determine which test kernel to build next. | 19:29 |
| s10gopal | suppose we are at 10 so build a kernel for good and another for bad | 19:30 |
| s10gopal | at commit 5 and at commit 15 | 19:30 |
| s10gopal | jsalisbury: on my laptop it takes 3.5 hour to make kernel from source , how i can speed it up ? | 19:31 |
| s10gopal | i am already using -j5 | 19:31 |
| s10gopal | -j4 | 19:31 |
| jsalisbury | s10gopal, I can build you the next two kernels. We just have to be careful when testing and reporting test results. It's easy to make a mistake. | 19:31 |
| jsalisbury | s10gopal, you could get a faster computer ;-) | 19:32 |
| jsalisbury | s10gopal, Seriously thought, if it takes that long, tuning isn't going to speed it up much faster | 19:32 |
| s10gopal | i am having core i5 6200u , 12gb ram and 128gb ssd | 19:33 |
| s10gopal | can you please teach me how to remove modules from kernel , then it will compile fast ? | 19:34 |
| jsalisbury | s10gopal, You would need to edit config options. | 19:35 |
| jsalisbury | s10gopal, figuring all that out would probably take more time than just doing the bisect | 19:35 |
| s10gopal | jsalisbury: can i make kernel on remote machine ? | 19:37 |
| jsalisbury | s10gopal, sure, if you have a remote machine. | 19:37 |
| s10gopal | it is possible to rent one online ? and how much it is going to cost ? | 19:38 |
| jsalisbury | s10gopal, That I have no idea. You could get an Amazon or Google cloud image, but I really don't think that would be worth it. | 19:39 |
| s10gopal | jsalisbury: it is good | 19:39 |
| s10gopal | 4.12.0-041200-generic #201804051118 | 19:39 |
| jsalisbury | s10gopal, this bisect is going to tell us which commit introduced the regression. We are then going to have to work with upstream to fix up that commit or revert it. | 19:40 |
| jsalisbury | s10gopal, I can build the next kernel now | 19:41 |
| s10gopal | jsalisbury: ok i will test it | 19:41 |
| tyhicks | dijuremo: I just took a quick look and I suspect that Intel didn't fully update their slides | 19:47 |
| tyhicks | dijuremo: all the 106A5 CPUIDs have been marked RED, indicating that they're not going to update the microcode, except for yours | 19:48 |
| tyhicks | dijuremo: also, your processor isn't listed as receiving a microcode update in their release notes: https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File?product=873 | 19:49 |
| tyhicks | dijuremo: I suspect that they forgot to change your processor's row to RED in the slides :/ | 19:49 |
| tyhicks | the 106a5 microcode in their latest bundle matches what your kernel is reporting: | 19:50 |
| tyhicks | $ iucode-tool -L microcode.dat | grep -i 106a5 | 19:50 |
| tyhicks | 01/129: sig 0x000106a5, pf mask 0x03, 2013-06-21, rev 0x0019, size 10240 | 19:50 |
| dijuremo | Wow, that is so screwed up... How is one going to figure that out when the slides show yellow and also a firmware 0x1c and their download page says This download is valid for the product(s) listed below. and includes Intel® Xeon® Processor E5530 (8M Cache, 2.40 GHz, 5.86 GT/s Intel® QPI) | 19:52 |
| ntd | i have a few grievances with current 4.15, should be worked out before bionic rtm :) | 20:04 |
| ntd | "rtm" | 20:04 |
| aaa_ | what is rtm? | 21:36 |
| TJ- | Release To Manufacturing | 21:37 |
| aaa_ | ty | 22:39 |
| aaa_ | :) | 22:39 |
| ntd | There are some confirmed and reproducible issues with current 4.15 that really should be ironed out before bionic release (potential userbase won't be able to boot) | 23:23 |
| ntd | anyone i can talk to about it? | 23:23 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!