/srv/irclogs.ubuntu.com/2015/07/29/#ubuntu-kernel.txt

tewardeveryones aware of the kernel regression which caused hell on Trusty and others right?02:57
teward(breaking Wine, introducing firefox explosions, etc.)02:57
infinityteward: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1479093 ?03:11
ubot5Ubuntu bug 1479093 in linux (Ubuntu Wily) "Segfault in ld-2.19.so while starting Steam after upgrade to 3.13.0-59.98 " [Medium,Confirmed]03:11
infinityteward: You say "trusty and others", have you confirmed it on non-3.13 kernels?03:12
tewardinfinity: well, Wine works on -58-generic03:13
tewardbreaks on -59-generic03:13
tewardand it's not that bug.03:13
tewardinfinity: i just reverted to -58-generic to see if Wine functions, and it does, going to -59-generic again to see if it crashes03:13
tewardcause it crashed once today03:13
tewardthen never came up again03:13
tewardwonder if that had the details03:13
infinityteward: What makes you say it's not this bug?03:13
tewardinfinity: https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1478844  <-- this is the one that people are seeing it on03:13
ubot5Ubuntu bug 1478844 in ubuntu-release-upgrader (Ubuntu) "Last Kernel Update crash wine and other app like Firefox" [Undecided,Confirmed]03:13
tewardinfinity: the one I'm aware of is ^ that03:13
tewardnot the bug you've specifically pointed at03:14
infinityteward: I'm sure it's the same bug.03:14
teward(on Trusty)03:14
tewardinfinity: ack03:14
tewardshould those be duped then?03:14
tewardinfinity: i also didn't mean 'and others'03:14
infinityLooking to be sure.03:14
tewardi'm tired after beating my head against landscape all day03:14
infinityYeah, looks like the same bug.03:14
tewardinfinity: i'm on the bug now want me to duep it?03:15
infinityteward: Already done.03:15
tewardack03:15
tewardinfinity: so this is considered a regression, then?03:16
tewardand a fairly major one if it breaks wine, firefox, and others03:16
tewardI can do without firefox.  can't do without wine at the moment :/03:16
infinityteward: Quite definitely, yes.  We're looking more deeply into it tomorrow.03:16
tewardinfinity: so it's a 'first thing in the morning' kind of bug then03:17
infinityteward: As soon as we can isolate the issue, we'll toss out a quick fix, I think.  It's clearly more widespread than just "some people can't play a video game or two".03:17
tewardinfinity: oh, no doubt03:17
tewardinfinity: any chance of a 'roll it back' update03:17
tewardalthough...03:17
tewardthat would kill a sec update03:17
infinityteward: Roll it back to what?03:17
teward:/03:17
infinityteward: Right.03:17
infinityteward: Bit of a catch-22.03:17
tewardright03:18
tewardhow major was the patched sec vuln, i didn't dig into the USN03:18
teward(i have too many DSA's to read for work >.<)03:18
infinityteward: Major enough that we did the update out-of-cadence to meet a CRD, rather than take our usual time fitting into our 3-week cycle.03:19
tewardmmm03:19
tewardum... CRD?  I'm still learning all the acronyms xD03:19
infinityteward: That said, root vulns for your home gaming machine and root vulns for your colo server are certainly two very different things to consider.  And given all the people this bug affects, it seems pretty isolated to "weird stuff desktop people do" so far, so upgrade the server, roll back the desktop, maybe.03:20
=== gerald is now known as Guest32557
tewardmmm03:20
infinityteward: CRD = coordinated release date.  When all the distros and upstreams agree on the date and time to make an embargoed security vulnerability public.03:21
tewardahh, OK03:21
tewardinfinity: i know i have a couple people at work who said "Hey, why's Firefox crashing whenever I do my standard work during the day through it", and three people I do at-a-discount Linux support for saying similar, so from my perspective it's a major regression if it breaks everything.  roll back desktop, upgrade server, seems to me the course of action I'm taking, although I try and *not* have downtime on my servers...03:22
teward... speaking of which my LDS server apparently is having downtime, which means i cant pick/choose updates as easily for my other servers.... what joy >.>03:23
tewardinfinity: thanks for keeping me in the loop, i assume there'll be additional discussions on this here, or that I can lurk here for status updates?03:27
teward(I was about to file a bug on this issue too, glad I saw your response xD)03:27
infinityteward: Discussion will happen... Somewhere.03:28
tewardack03:28
tewardi'mma just lurk here then anyways.03:29
tewardan extra channel doesn't put additional strain on me or my bouncer :p03:29
apwteward, are you in an awake part of your timezone, and if so are you able to reproduce any of the issues reported ?09:43
apw(ie if we have anything to test, are you able to help validate)09:44
infinityI think he's American.  I'09:52
infinitym not sure why I think that.09:52
infinityLP claims he lives in America/Eastern, so...09:53
apw"not yet" then :)09:56
=== Elimin8r is now known as Elimin8er
nuschI've kernel panic on two systems recently, the reaseon is grub menu wasn't populated with initrd entry and encrypted root cannot be mounted. It's caused by lack of initrd bo user isn't alerted anyway that upgraded failed. Shouldn't grub skip such entry or alert user during upgrade that something failed? Users without knowledge about booting process will reinstall system in that case - there is no easily availble solution by googling error messages11:13
apwnusch, well it is in theory at least valid to not have an initrd.  that said we rarely if ever do, so perhaps a warning is appropriate, but will an unsophisticated user cope with that11:17
infinityA grub entry without an initrd isn't a bug or a misconfiguration, not sure what we'd be warning about.11:18
infinityBut if you mean an entry that points at an initrd that doesn't exist on disk because it didn't get created properly, that's a more interesting case.11:19
apwnusch, i think the better question and better bug would relate to _why_ the initrd build failed11:27
apwand noone was told11:27
infinitySomeone is always told.11:29
infinityThe postinsts fail.11:29
tewardapw: i'm awake now11:29
tewardinfinity: LP is accurate with my timezone11:29
infinityAnd apt/dpkg exit non-011:29
infinityteward: Excellent.  Andy has a kernel for you to test.11:30
tewardapw: the issues I can replicate are the firefox crashes (quite a few of em), Wine being 100% broken, and if you point me at what some of the other test cases are I can start testing at lunch - the actual 'error' i'm not sure of since there was never a crash bug created unfortunately11:30
tewardinfinity: OK, i'm heading out the door in 3 minutes so i'll download/test/install once I get to work11:30
infinityteward:  http://people.canonical.com/~henrix/CVE-2015-2390-trusty/11:30
tewardcorrection i'll download it now, test/install at work xD11:30
tewardinfinity: back in a moment, i'll test those packages against what I have seen here (crashes, Wine not working, etc.)11:40
tewardinfinity: booted up with those kernels - wine, which was broken under -59-generic, is now working again, so that issue was resolved.  Haven't gotten to the Firefox tests yet, but I'll run it through its paces and try and replicate the "randomly crashes for no immediately obvious reason" issue that was introduced previously11:45
teward(and in the interim i temporarily told Grub the default entry to use is the -58-generic - the last packaging of the kernel which didn't torpedo Wine and Firefox)11:46
teward(and specifically boot-tested the packages in the link you provided)11:46
henrix\o/11:47
tewardVMware Workstation is complaining...11:47
henrixteward: thanks for testing that kernel11:47
tewardbut that's not atypical, every new kernel it needs modules rebuilt for virtual nics11:47
tewardhenrix: glad to help, provided I'm not compiling kernels from scratch xD11:47
tewardhenrix: besides, two of the tools I need today need Wine so bahh11:47
tewardit was either use -58-generic, or test :P11:49
henrixteward: heh, yeah.  good thing it was easily reproducible ;)11:52
tewardhenrix: definitely.  ESPECIALLY when it's a case of "The update makes Wine totally unusable and breaks 666% of everything"11:56
tewardbah, laggggy client >.<11:56
tewardhenrix: it also helps knowing my way around the terminal, most standard users wouldn't be able to `dpkg -i` and know what they're doing or which packages to download xD11:56
tewardbut yeah, that appears to work...11:56
nusch@apw I meant the case when initrd isn't generated on disk by any reason(e.g low disk space) - if there is a way to tell if it should be generated we can use the same path to deduce that is necessary to boot, and if we from that point deduce that is neccesary to boot but not exists -> system is unbootable -> should we then create grub menu entry or at least make this entry default?13:20
apwnusch, well the kernel install should fail in that case, becasue the postinst should fail, so you should know you are in a world of hurt13:20
TJ-nusch: technically, GRUB's "/etc/grub.d/20_linux_xen" is at fault. It creates the list of kernels in 'linux_list' by searching for valid /boot/vmlinu* entries but doesn't also confirm there's a valid initrd (e.g. a /boot/initrd.img* *and* it passes the grub_file_is_not_garbage test. Obviously update-initramfs should make a lot of noise if the initrd.img isn't valid13:27
tewardinfinity: poke - is the 8-12 hours for fixed kernel availability still on target (from the earlier statement in the bug)20:06
tewardand do I need to worry about a delta between the packages i tested from henrix for the issue and what's been pushed to builders20:07
infinityteward: Yes.20:08
infinityteward: And there should be no delta, but it might be nice if you could re-smoketest a binary or two.20:08
infinityWhich I'm about to do anyway.20:09
tewardsorry i've had my head in ESXi's command line all day >.<  by 're-smoketest' is that anything specific other than 'test to see if it fixes the issues' like i did earlier20:09
* teward yawns20:09
teward(fixing VMs should NOT require editing the hypervisor >.<)20:09
infinityYeah.20:09
infinityhttps://launchpad.net/ubuntu/+source/linux/3.13.0-61.10020:10
tewardok20:10
tewardlemme reboot to -58-generic, the last known-to-work one, then remove the downloaded-by-hand ones i installed, then install from proposed20:11
tewardTESTING TIME :P20:11
infinityIt's not quite in proposed yet on mirrors.  Still some disk grinding on my end.20:11
tewardmmm20:11
infinityWhich is why I pointed you at LP. :)20:11
tewardindeed20:11
tewardinfinity: it wouldn't be on the main archive mirrors?20:11
teward(i.e. archive.ubuntu.com)20:11
infinityteward: Considering it's still publishing to ftpmaster, no.  Not on archive yet.20:12
tewardok20:13
* infinity gets to testing.20:14
tewardinfinity: so, basically, download from LP, install-test, see if it breaks? :P20:14
infinityBasically.20:15
tewardthat's a fairly easy thing :p20:15
infinityOkay, new trusty and lts-trusty binaries seem to work for me.20:18
infinitybjf: You need/want any sign-off on this 3.13 fiasco, or shall I just release to updates/security as soon as it's all happy in proposed?20:22
tewardinfinity: gonna install and reboot, and i'll give you my test results20:23
teward(never huts to have more than one tester :))20:23
infinityteward: Indeed.20:24
infinityteward: Plus, you can reproduce the bug, I was just testing that the kernel wasn't obviously broken. :P20:24
bjfinfinity, i'm ready when you are ready20:24
tewardright20:24
tewardbjf: lets wait for this test - make sure this actually fixes the bug20:24
teward(granted the delta between henrix's package and this one is probably near-zero but testing is always a good thing)20:24
infinitybjf: Alrighty.  Doing the -signed dance, should be ready after that.20:25
tewardback in a moment, reboot time :)20:25
infinityteward: The delta between henrix's and the ones he uploaded to LP is 3 lines in the changelog. ;)20:25
* henrix drums fingers...20:26
tewardinfinity: +1 on it working and not torpedoing :)20:31
infinityGood, good.20:31
tewardinfinity: you never know, one off thing on the builder at the wrong moment and *BOOM*20:31
tewardcould torpedo an entire system20:31
tewardand then you have other problems :P20:31
infinityI wonder how many weeks/months it will take us to find all the duplicate bugs and dupe them against the original.20:31
tewardinfinity: bet you a dollar it'll take 10 months, and more bugs will come in from people who don't pull updates frequently20:32
tewardwho upgraded to -59-generic and didn't update since xD20:32
infinityhenrix: And it's all done (according to the LP DB, anyway, allow 6 to 8 weeks for publishing, etc)20:40
henrixinfinity: \o/20:40
henrixinfinity: thanks a lot20:40
infinityhenrix: Let's see if we can avoid this sort of thing for, like, a few months.20:40
infinityMaybe we should switch to a BSD kernel.20:41
henrixinfinity: well... let's avoid it for at least the next week, as i hope to be mostly offline :)20:42
henrixinfinity: about the BSD kernel... yeah, let's talk about it over a beer in Seattle :)20:42
infinityhenrix: Heh.20:43
tewardinfinity: US mirrors appear to have it, or at least the one I hit about an hour ago.  apt-cache policy shows it available :)22:28
tewardglad to see this was rapidly fixed :)22:28
tewardprobably helps to have people affected testing and not just complaining about it, huh :P22:28
infinityIt doesn't hurt to have testers.22:29
tewardanyways, thanks to you, and the kernel team, for such rapid fix-release :)22:30

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!