[00:45] <zyga> bjf: thanks for the heads up
[08:45] <j4s0nmchr1st0s> apw will you assit in getting a high speed wire provisioned?
[08:46] <apw> heh
[08:47] <j4s0nmchr1st0s> I am surrounded by "intelligence" agents.
[08:47] <j4s0nmchr1st0s> They keep siphoning my ip lines.
[08:49] <j4s0nmchr1st0s> It is slowing down productivity.\
[08:50] <j4s0nmchr1st0s> apw all you have to do is put in an order
[08:54] <j4s0nmchr1st0s> pottery barn
[08:56] <j4s0nmchr1st0s> apw: order a cable line
[08:57] <j4s0nmchr1st0s> Have you ever felt like you were surrounded by chick tracts
[08:58] <j4s0nmchr1st0s> digital ocean
[08:59] <apw> i think you over estimate my influence, we're straying rather off topic 
[09:02] <j4s0nmchr1st0s> apw: then play mindcraft with myself
[09:02] <j4s0nmchr1st0s> shark tech is very dangerous
[09:03] <j4s0nmchr1st0s> sharks are pure survival instinct
[09:05] <j4s0nmchr1st0s> You are going to have to incept some fresh code apw
[09:09] <j4s0nmchr1st0s> apw: I don't ask you to influence just carry out the plans
[09:09] <j4s0nmchr1st0s> Influence already is.
[09:10] <j4s0nmchr1st0s> There is some way to make sharks eat each other.
[09:12] <apw> this really isn't the forum for this
[09:21] <j4s0nmchr1st0s> It is the best forum I see out there already made.
[09:21] <apw> no really, not
[09:22] <j4s0nmchr1st0s> The only alternative that was recently made is github
[09:24] <j4s0nmchr1st0s> How many years of martyr psychometrics did you hold out on?
[09:26] <apw> this isn't ubuntu kernel related, and really not on topic for this channel, please find a more appropriate one
[10:01] <infinity> apw: \o/
[12:39] <smb> jibel, oh btw, did apw already talk to you about dkms testing for dahdi-dkms being futile as long as the host cannot connect to the internet?
[12:40] <jibel> smb, not that I remember. We can probably use the proxy if it's a requirement
[12:41] <smb> jibel, Yeah, the problem is that the dkms build tries to download stuff from somewhere.  so the test fails even if it would not fail otherwise
[12:42] <jibel> smb, I understand, I'll setup a proxy to see if it helps.
[12:42] <smb> jibel, Ok, great. Thanks
[13:43] <apw> jibel, smb, thanks for looking into that ... and smb well remembered
[14:42] <infinity> smb: The dkms build tries to download from the internet?  Eww.
[14:43] <smb> infinity, it's all their fault :-P
[14:44] <smb> infinity, maybe it is not done anymore with the more recent Debian versions which we probably should get in sync once the dust settles
[14:44] <infinity> smb: Err, but what is it downloading?
[14:44] <infinity> smb: This sounds like a massive bug to me.
[14:44] <smb> infinity, could be some firmware things
[14:44] <infinity> smb: Unless it's a non-free thing, then it shouldn't be in universe.
[14:46] <smb> infinity, We can and probably should do a more insightful review. Right now I was just trying to get them actually compile
[14:47] <smb> infinity, which was when I found that the last sync from Debian was done about 2012
[14:48] <smb> There seem to be three related source packages all of which have different versions numbers for us while in Debian they seem to be in sync
[14:49] <infinity> smb: That sounds rather unfortunate.
[14:50] <smb> that would be the nicest way of putting it coming to my mind, yes
[14:58] <caribou> arges: around ?
[14:58] <arges> caribou: yes
[14:59] <caribou> arges: howdy. Got a question regarding the makedumpfile fix that you submitted for ppc64el (the vmlinuX mod)
[14:59] <caribou> arges: it has been tested correctly ?
[14:59] <caribou> arges: I mean, I have no way to test it before uploading to Debian
[14:59] <arges> caribou: yes, i verified on ppc64el hardware
[15:00] <arges> caribou: i can assist with testing too if needed
[15:00] <caribou> arges: ok, I'll trust you on this one
[15:00] <caribou> arges: well, if you have a minute, I send a .deb your way to test it
[15:00] <arges> caribou: yea i think that's the only delta left when I did the last deiban sync too
[15:00] <arges> so that would be great if we got that in debian properly
[15:01] <caribou> arges: working on it atm
[15:01] <caribou> arges: you worked on the makedumpfile merge yet ?
[15:01] <caribou> arges: for vivid I mean
[15:01]  * arges looks
[15:01] <caribou> 'cause I did too
[15:02] <arges> caribou: hmm. i uploaded 1.5.7-1ubuntu1 i think
[15:03] <caribou> arges: well, I'm about to upload 1.5.7-3 to debian .2 had a fix for ssh networked dump & .3 will be the ppc64el fix
[15:03] <arges> caribou: yup. 
[15:03] <caribou> arges: so hopefully both will be in sync
[15:03] <caribou> & no delta
[15:03] <arges> caribou: ok. is ther that fix for changing the runlevel
[15:03] <arges> at which kdump runs:?
[15:03] <caribou> arges: not yet
[15:04] <arges> ok
[15:04] <caribou> arges: but networked kernel crash dump is in there
[15:04] <caribou> arges: I'm hoping to have runlevel S kernel dump in during this cycle
[15:04] <caribou> arges: I now have a complete test program for testing makedumpfile (both local & remote)
[15:05] <arges> cool : )
[15:05] <caribou> arges: I need to investigate autopkgtest for this one
[15:05] <caribou> arges: so do you have time to test the .deb before I upload it ?
[15:06] <arges> caribou: i can make some time sure
[15:06] <caribou> arges: ok, let me build it
[15:07] <j4s0nmchr1st0s> apw why was 12.10 "killed"
[15:09] <caribou> arges: which release you want it built on ?
[15:10] <arges> caribou: umm, trusty would be fine
[15:10] <caribou> arges: k
[15:10] <ogra_> j4because its lifecycle has ended
[15:19] <arges> caribou: can you send a ppc64el build? or give me sources and I'll build them
[15:20] <caribou> arges: oh, right. Hold on, I'll put the source package up there
[15:22] <caribou> arges: ok, source .gz & .dsc are up there
[15:22] <arges> caribou: ok
[15:45] <apw> henrix, ok that build seems to have made it to the builder at least (-ckt)
[15:46] <henrix> apw: cool! so the builds work; hopefully the triagger will also work... :)
[15:47] <apw> henrix, its only building, not seen it publish yet :)
[15:47]  * apw goes run the triager ...
[15:54] <alexbligh1> Any idea where I'd even *start* looking for bug where an 8 core machine running 3.8.0-32-generic #47~precise1-Ubuntu appears to be nearly entirely idle (CPU wise), kvm running mostly, load average > 250,000, "rcu_sched detected stalls on CPUs/tasks:" appears in logs, hung "shortly afterwards" (per customer). Whole system running from RAM I believe.
[15:55]  * apw stuggles to remember what 3.8 was even
[15:56] <infinity> alexbligh1: Well, the first step would be to run a supported kernel. ;)
[15:56] <infinity> apw: saucy.
[15:56] <apw> well that was my first guess :)  saucy, ouch
[15:56] <alexbligh1> infinity, that's a Precise h/w enablement kernel. Yes I know it should be 3.8.0-44.66~precise1
[15:57] <infinity> alexbligh1: No, it should be 3.13. :/
[15:57] <apw> alexbligh1, yeah but those went off support some time ago
[15:57] <henrix> infinity: actually, 3.8 was raring; saucy was 3.11
[15:57] <infinity> https://lists.ubuntu.com/archives/ubuntu-announce/2014-July/000186.html
[15:58] <infinity> henrix: Err, yes.  I can't alphabet.
[15:58] <alexbligh1> infinity, hmmm - I thought those were still supported even though R (obviously) isn't. I'm obviously wrong. Well, we can fix that.
[15:58] <infinity> henrix: I meant "the one after quantal", which is, apparently "S" in my mind today.
[15:58] <henrix> heh
[15:59] <apw> alexbligh1, but the rcu_sched reports imply someone got into the kernel and got stuck there, so it woudl be something likely running on proc all the time, you ought to be able to find that in like an sysrq whatsit
[15:59] <apw> alexbligh1, but yeah when 3.13 came back to precise that signalled eol for the older releases, there is a nice piccy that ogasawara drew to help visualise this, somewhere in teh wiki
[15:59] <infinity> Yeah, though rcu_sched stalls are ALWAYS kernel bugs.
[15:59] <infinity> So an upgrade might just fix it.
[16:00] <infinity> Often exacerbated by userspace, but always kernel bugs.
[16:00] <infinity> alexbligh1: Any custom dkms drivers or anything, or just a stock kernel build?
[16:01] <alexbligh1> infinity, apw no custom drivers, stock kernel build. The stacktrace we got from customer was: http://pastebin.com/edLwdQDp - I'm not seeing anything fantastically unexpected there.
[16:02] <alexbligh1> infinity, I agree an update might fix it. I was hoping I might be able to say "we think this update will fix it because it fixes X".
[16:02] <apw> alexbligh1, https://wiki.ubuntu.com/Kernel/Support has the nice piccy in case you need something to shove at them
[16:03] <infinity> alexbligh1: Oh, with KVM in play, I'd give you good odds that an upgrade to linux-lts-trusty will fix it "Just cause".
[16:03] <infinity> alexbligh1: KVM has improved in leaps and bounds in that timeframe.
[16:03] <infinity> And Paul even fixed a few bugs in RCU. :P
[16:03] <infinity> Though this doesn't look like an RCU bug.
[16:04] <alexbligh1> infinity, yeah I think that's about the best I'm going to get :-)
[16:04] <alexbligh1> thx
[16:04] <apw> alexbligh1, that dump only shows 0 4 and 5, i wonder where the others are
[16:05] <alexbligh1> apw, yeah I thought that too. I have an output from top showing them not doing much. Very strange.
[16:05] <apw> either they cut the report short, or they didn't respond to the NMI which seems unlikley
[16:07] <alexbligh1> apw, let me go ask support to find out whether the logs got chopped. If the processors literally froze, didn't respond to NMI etc., that would presumably explain the load average.
[16:09] <apw> well the lack of RCU completing implies things are stuck and stuck things normally hold locks, which leads to eveyrthing else train-wrecking into those locks
[16:10] <alexbligh1> apw, yep. thanks.
[16:24] <arges> another approach (if you can easily reproduce) is to induce a crashdump on a softlockup and analyize it from there
[16:33] <alexbligh1> arges, appears the customer can (whether he tries or not) reproduce one a day or so across a number of machines, so if an upgrade doesn't fix it I'll try that.
[16:33] <arges> alexbligh1: fwiw, here's a write-up i did (since i was debugging a hang recently) http://dinosaursareforever.blogspot.com/2014/10/getting-kernel-crashdumps-for-hung.html if you don't already know how to set that up
[16:35] <alexbligh1> what is the correct package to depend on to bring in the newest 3.13 on precise? http://packages.ubuntu.com/search?keywords=linux-lts-trusty&searchon=names&suite=precise&section=all is deeply unhelpful
[16:36] <alexbligh1> arges, thanks
[16:38] <infinity> alexbligh1: linux-lts-trusty for kernel+headers, or linux-image-lts-trusty for just kernel.
[16:38] <infinity> alexbligh1: And linux-signed-lts-trusty if they're in a secure-boot environment.
[16:39] <infinity> Err, I missed a "generic" in there.
[16:40] <alexbligh1> infinity, ah, that was my mistake too :-)
[16:40] <infinity> alexbligh1: linux-generic-lts-trusty for kernel+headers, linux-image-generic-lts-trusty for just kernel, linux-signed-generic-lts-trusty for SB.
[16:41] <alexbligh1> infinity, yep, got it, just missing the 'generic'.
[17:26] <alexbligh1> What's the hope of persuading someone to take the tiny patch in LP #1337262 to kmod for vivid?
[19:05] <cmagina> is there a changelog diff between the trusty-proposed 40.68 and the 40.69 kernels. trying to find out what exactly the difference is
[20:40] <arges> cmagina: there was a regression .69 is a fix for .68 with no changes to the ABI
[20:42] <arges> this is essentially the change (minus changelog changes etc etc) : http://pastebin.ubuntu.com/9012495/
[20:48] <cmagina> arges: ah, thanks for the info.