[02:57] <teward> everyones aware of the kernel regression which caused hell on Trusty and others right?
[02:57] <teward> (breaking Wine, introducing firefox explosions, etc.)
[03:11] <infinity> teward: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1479093 ?
[03:11] <ubot5> Ubuntu bug 1479093 in linux (Ubuntu Wily) "Segfault in ld-2.19.so while starting Steam after upgrade to 3.13.0-59.98 " [Medium,Confirmed]
[03:12] <infinity> teward: You say "trusty and others", have you confirmed it on non-3.13 kernels?
[03:13] <teward> infinity: well, Wine works on -58-generic
[03:13] <teward> breaks on -59-generic
[03:13] <teward> and it's not that bug.
[03:13] <teward> infinity: i just reverted to -58-generic to see if Wine functions, and it does, going to -59-generic again to see if it crashes
[03:13] <teward> cause it crashed once today
[03:13] <teward> then never came up again
[03:13] <teward> wonder if that had the details
[03:13] <infinity> teward: What makes you say it's not this bug?
[03:13] <teward> infinity: https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1478844  <-- this is the one that people are seeing it on
[03:13] <ubot5> Ubuntu bug 1478844 in ubuntu-release-upgrader (Ubuntu) "Last Kernel Update crash wine and other app like Firefox" [Undecided,Confirmed]
[03:13] <teward> infinity: the one I'm aware of is ^ that
[03:14] <teward> not the bug you've specifically pointed at
[03:14] <infinity> teward: I'm sure it's the same bug.
[03:14] <teward> (on Trusty)
[03:14] <teward> infinity: ack
[03:14] <teward> should those be duped then?
[03:14] <teward> infinity: i also didn't mean 'and others'
[03:14] <infinity> Looking to be sure.
[03:14] <teward> i'm tired after beating my head against landscape all day
[03:14] <infinity> Yeah, looks like the same bug.
[03:15] <teward> infinity: i'm on the bug now want me to duep it?
[03:15] <infinity> teward: Already done.
[03:15] <teward> ack
[03:16] <teward> infinity: so this is considered a regression, then?
[03:16] <teward> and a fairly major one if it breaks wine, firefox, and others
[03:16] <teward> I can do without firefox.  can't do without wine at the moment :/
[03:16] <infinity> teward: Quite definitely, yes.  We're looking more deeply into it tomorrow.
[03:17] <teward> infinity: so it's a 'first thing in the morning' kind of bug then
[03:17] <infinity> teward: As soon as we can isolate the issue, we'll toss out a quick fix, I think.  It's clearly more widespread than just "some people can't play a video game or two".
[03:17] <teward> infinity: oh, no doubt
[03:17] <teward> infinity: any chance of a 'roll it back' update
[03:17] <teward> although...
[03:17] <teward> that would kill a sec update
[03:17] <infinity> teward: Roll it back to what?
[03:17] <teward> :/
[03:17] <infinity> teward: Right.
[03:17] <infinity> teward: Bit of a catch-22.
[03:18] <teward> right
[03:18] <teward> how major was the patched sec vuln, i didn't dig into the USN
[03:18] <teward> (i have too many DSA's to read for work >.<)
[03:19] <infinity> teward: Major enough that we did the update out-of-cadence to meet a CRD, rather than take our usual time fitting into our 3-week cycle.
[03:19] <teward> mmm
[03:19] <teward> um... CRD?  I'm still learning all the acronyms xD
[03:20] <infinity> teward: That said, root vulns for your home gaming machine and root vulns for your colo server are certainly two very different things to consider.  And given all the people this bug affects, it seems pretty isolated to "weird stuff desktop people do" so far, so upgrade the server, roll back the desktop, maybe.
=== gerald is now known as Guest32557
[03:20] <teward> mmm
[03:21] <infinity> teward: CRD = coordinated release date.  When all the distros and upstreams agree on the date and time to make an embargoed security vulnerability public.
[03:21] <teward> ahh, OK
[03:22] <teward> infinity: i know i have a couple people at work who said "Hey, why's Firefox crashing whenever I do my standard work during the day through it", and three people I do at-a-discount Linux support for saying similar, so from my perspective it's a major regression if it breaks everything.  roll back desktop, upgrade server, seems to me the course of action I'm taking, although I try and *not* have downtime on my servers...
[03:23] <teward> ... speaking of which my LDS server apparently is having downtime, which means i cant pick/choose updates as easily for my other servers.... what joy >.>
[03:27] <teward> infinity: thanks for keeping me in the loop, i assume there'll be additional discussions on this here, or that I can lurk here for status updates?
[03:27] <teward> (I was about to file a bug on this issue too, glad I saw your response xD)
[03:28] <infinity> teward: Discussion will happen... Somewhere.
[03:28] <teward> ack
[03:29] <teward> i'mma just lurk here then anyways.
[03:29] <teward> an extra channel doesn't put additional strain on me or my bouncer :p
[09:43] <apw> teward, are you in an awake part of your timezone, and if so are you able to reproduce any of the issues reported ?
[09:44] <apw> (ie if we have anything to test, are you able to help validate)
[09:52] <infinity> I think he's American.  I'
[09:52] <infinity> m not sure why I think that.
[09:53] <infinity> LP claims he lives in America/Eastern, so...
[09:56] <apw> "not yet" then :)
=== Elimin8r is now known as Elimin8er
[11:13] <nusch> I've kernel panic on two systems recently, the reaseon is grub menu wasn't populated with initrd entry and encrypted root cannot be mounted. It's caused by lack of initrd bo user isn't alerted anyway that upgraded failed. Shouldn't grub skip such entry or alert user during upgrade that something failed? Users without knowledge about booting process will reinstall system in that case - there is no easily availble solution by googling error messages
[11:17] <apw> nusch, well it is in theory at least valid to not have an initrd.  that said we rarely if ever do, so perhaps a warning is appropriate, but will an unsophisticated user cope with that
[11:18] <infinity> A grub entry without an initrd isn't a bug or a misconfiguration, not sure what we'd be warning about.
[11:19] <infinity> But if you mean an entry that points at an initrd that doesn't exist on disk because it didn't get created properly, that's a more interesting case.
[11:27] <apw> nusch, i think the better question and better bug would relate to _why_ the initrd build failed
[11:27] <apw> and noone was told
[11:29] <infinity> Someone is always told.
[11:29] <infinity> The postinsts fail.
[11:29] <teward> apw: i'm awake now
[11:29] <teward> infinity: LP is accurate with my timezone
[11:29] <infinity> And apt/dpkg exit non-0
[11:30] <infinity> teward: Excellent.  Andy has a kernel for you to test.
[11:30] <teward> apw: the issues I can replicate are the firefox crashes (quite a few of em), Wine being 100% broken, and if you point me at what some of the other test cases are I can start testing at lunch - the actual 'error' i'm not sure of since there was never a crash bug created unfortunately
[11:30] <teward> infinity: OK, i'm heading out the door in 3 minutes so i'll download/test/install once I get to work
[11:30] <infinity> teward:  http://people.canonical.com/~henrix/CVE-2015-2390-trusty/
[11:30] <teward> correction i'll download it now, test/install at work xD
[11:40] <teward> infinity: back in a moment, i'll test those packages against what I have seen here (crashes, Wine not working, etc.)
[11:45] <teward> infinity: booted up with those kernels - wine, which was broken under -59-generic, is now working again, so that issue was resolved.  Haven't gotten to the Firefox tests yet, but I'll run it through its paces and try and replicate the "randomly crashes for no immediately obvious reason" issue that was introduced previously
[11:46] <teward> (and in the interim i temporarily told Grub the default entry to use is the -58-generic - the last packaging of the kernel which didn't torpedo Wine and Firefox)
[11:46] <teward> (and specifically boot-tested the packages in the link you provided)
[11:47] <henrix> \o/
[11:47] <teward> VMware Workstation is complaining...
[11:47] <henrix> teward: thanks for testing that kernel
[11:47] <teward> but that's not atypical, every new kernel it needs modules rebuilt for virtual nics
[11:47] <teward> henrix: glad to help, provided I'm not compiling kernels from scratch xD
[11:47] <teward> henrix: besides, two of the tools I need today need Wine so bahh
[11:49] <teward> it was either use -58-generic, or test :P
[11:52] <henrix> teward: heh, yeah.  good thing it was easily reproducible ;)
[11:56] <teward> henrix: definitely.  ESPECIALLY when it's a case of "The update makes Wine totally unusable and breaks 666% of everything"
[11:56] <teward> bah, laggggy client >.<
[11:56] <teward> henrix: it also helps knowing my way around the terminal, most standard users wouldn't be able to `dpkg -i` and know what they're doing or which packages to download xD
[11:56] <teward> but yeah, that appears to work...
[13:20] <nusch> @apw I meant the case when initrd isn't generated on disk by any reason(e.g low disk space) - if there is a way to tell if it should be generated we can use the same path to deduce that is necessary to boot, and if we from that point deduce that is neccesary to boot but not exists -> system is unbootable -> should we then create grub menu entry or at least make this entry default?
[13:20] <apw> nusch, well the kernel install should fail in that case, becasue the postinst should fail, so you should know you are in a world of hurt
[13:27] <TJ-> nusch: technically, GRUB's "/etc/grub.d/20_linux_xen" is at fault. It creates the list of kernels in 'linux_list' by searching for valid /boot/vmlinu* entries but doesn't also confirm there's a valid initrd (e.g. a /boot/initrd.img* *and* it passes the grub_file_is_not_garbage test. Obviously update-initramfs should make a lot of noise if the initrd.img isn't valid
[20:06] <teward> infinity: poke - is the 8-12 hours for fixed kernel availability still on target (from the earlier statement in the bug)
[20:07] <teward> and do I need to worry about a delta between the packages i tested from henrix for the issue and what's been pushed to builders
[20:08] <infinity> teward: Yes.
[20:08] <infinity> teward: And there should be no delta, but it might be nice if you could re-smoketest a binary or two.
[20:09] <infinity> Which I'm about to do anyway.
[20:09] <teward> sorry i've had my head in ESXi's command line all day >.<  by 're-smoketest' is that anything specific other than 'test to see if it fixes the issues' like i did earlier
[20:09]  * teward yawns
[20:09] <teward> (fixing VMs should NOT require editing the hypervisor >.<)
[20:09] <infinity> Yeah.
[20:10] <infinity> https://launchpad.net/ubuntu/+source/linux/3.13.0-61.100
[20:10] <teward> ok
[20:11] <teward> lemme reboot to -58-generic, the last known-to-work one, then remove the downloaded-by-hand ones i installed, then install from proposed
[20:11] <teward> TESTING TIME :P
[20:11] <infinity> It's not quite in proposed yet on mirrors.  Still some disk grinding on my end.
[20:11] <teward> mmm
[20:11] <infinity> Which is why I pointed you at LP. :)
[20:11] <teward> indeed
[20:11] <teward> infinity: it wouldn't be on the main archive mirrors?
[20:11] <teward> (i.e. archive.ubuntu.com)
[20:12] <infinity> teward: Considering it's still publishing to ftpmaster, no.  Not on archive yet.
[20:13] <teward> ok
[20:14]  * infinity gets to testing.
[20:14] <teward> infinity: so, basically, download from LP, install-test, see if it breaks? :P
[20:15] <infinity> Basically.
[20:15] <teward> that's a fairly easy thing :p
[20:18] <infinity> Okay, new trusty and lts-trusty binaries seem to work for me.
[20:22] <infinity> bjf: You need/want any sign-off on this 3.13 fiasco, or shall I just release to updates/security as soon as it's all happy in proposed?
[20:23] <teward> infinity: gonna install and reboot, and i'll give you my test results
[20:23] <teward> (never huts to have more than one tester :))
[20:24] <infinity> teward: Indeed.
[20:24] <infinity> teward: Plus, you can reproduce the bug, I was just testing that the kernel wasn't obviously broken. :P
[20:24] <bjf> infinity, i'm ready when you are ready
[20:24] <teward> right
[20:24] <teward> bjf: lets wait for this test - make sure this actually fixes the bug
[20:24] <teward> (granted the delta between henrix's package and this one is probably near-zero but testing is always a good thing)
[20:25] <infinity> bjf: Alrighty.  Doing the -signed dance, should be ready after that.
[20:25] <teward> back in a moment, reboot time :)
[20:25] <infinity> teward: The delta between henrix's and the ones he uploaded to LP is 3 lines in the changelog. ;)
[20:26]  * henrix drums fingers...
[20:31] <teward> infinity: +1 on it working and not torpedoing :)
[20:31] <infinity> Good, good.
[20:31] <teward> infinity: you never know, one off thing on the builder at the wrong moment and *BOOM*
[20:31] <teward> could torpedo an entire system
[20:31] <teward> and then you have other problems :P
[20:31] <infinity> I wonder how many weeks/months it will take us to find all the duplicate bugs and dupe them against the original.
[20:32] <teward> infinity: bet you a dollar it'll take 10 months, and more bugs will come in from people who don't pull updates frequently
[20:32] <teward> who upgraded to -59-generic and didn't update since xD
[20:40] <infinity> henrix: And it's all done (according to the LP DB, anyway, allow 6 to 8 weeks for publishing, etc)
[20:40] <henrix> infinity: \o/
[20:40] <henrix> infinity: thanks a lot
[20:40] <infinity> henrix: Let's see if we can avoid this sort of thing for, like, a few months.
[20:41] <infinity> Maybe we should switch to a BSD kernel.
[20:42] <henrix> infinity: well... let's avoid it for at least the next week, as i hope to be mostly offline :)
[20:42] <henrix> infinity: about the BSD kernel... yeah, let's talk about it over a beer in Seattle :)
[20:43] <infinity> henrix: Heh.
[22:28] <teward> infinity: US mirrors appear to have it, or at least the one I hit about an hour ago.  apt-cache policy shows it available :)
[22:28] <teward> glad to see this was rapidly fixed :)
[22:28] <teward> probably helps to have people affected testing and not just complaining about it, huh :P
[22:29] <infinity> It doesn't hurt to have testers.
[22:30] <teward> anyways, thanks to you, and the kernel team, for such rapid fix-release :)