| erichammond | tgardner: (Took a meeting break) Yes, I should be able to take care of what is needed, though RightScale keeps changing the best way to set up their integration software with Ubuntu on EC2. | 00:00 |
|---|---|---|
| kees | ogasawara: 9a7e849 boots ok | 00:26 |
| ogasawara | kees: cool, I expected it to pass looking at the commit | 00:27 |
| kees | I guess it's 9908ff736adf261e749b4887486a32ffa209304c causing the problem then? | 00:27 |
| ogasawara | kees: yah, that's what the bisect is pointing to | 00:28 |
| ogasawara | kees: k, I'm going to try revert just that commit and build a final test kernel | 00:29 |
| kees | ogasawara: sweet | 00:29 |
| tgardner-afk | ccheney, http://kernel.ubuntu.com/~rtg/mainline/4f570f995f68ef77aae7e5a441222f59232f2d0e-maverick/linux-image-2.6.32-020632rc3g4f570f9-generic_2.6.32-020632rc3g4f570f9_amd64.deb | 00:44 |
| === kamal-away is now known as kamal | ||
| ogasawara | kees: this is the latest 2.6.35-6.7 Maverick kernel with the 9908ff7 reverted - http://people.canonical.com/~ogasawara/lp597075/2.6.35-6.7-9908ff7/ | 00:59 |
| ccheney | tgardner-afk, thanks | 00:59 |
| ogasawara | kees: I'm still building the mainline kernel with the commit reverted | 00:59 |
| ccheney | tgardner-afk, looks good | 01:05 |
| ccheney | be back in about 30m | 01:05 |
| ogasawara | kees: http://people.canonical.com/~ogasawara/lp597075/2.6.35-rc3-9908ff7/ - that's the latest 2.6.35-rc3 mainline kernel with 9908ff7 reverted | 01:12 |
| kees | ogasawara: lp597075/2.6.35-6.7-9908ff7 did not boot :( | 01:16 |
| kees | trying the mainline next... | 01:16 |
| ogasawara | kees: ugh, was afraid of that | 01:16 |
| kees | ogasawara: :( lp597075/2.6.35-rc3-9908ff7 did not boot either. | 01:23 |
| ogasawara | boo | 01:23 |
| kees | ogasawara: this is really odd. are there maybe multiple problems? | 01:23 |
| ogasawara | kees: I'm thinking it's a commit outside of i915 that might be triggering the i915 issue | 01:23 |
| kees | oh, the bisection was targetting only changes in i915, but there were other commits between it? | 01:24 |
| ogasawara | kees: yep | 01:24 |
| kees | gotcha. craps. | 01:24 |
| kees | well, if you can show me the commnds you're running, I can take this over maybe. I have access to the builders. | 01:25 |
| * ccheney back | 01:25 | |
| ogasawara | kees: ooh, have access to tyler? | 01:25 |
| ccheney | tgardner-afk, ok yea your 4f57f9 kernel was good | 01:25 |
| ogasawara | kees: that's what I was using as the build box | 01:25 |
| kees | ogasawara: yup, I'm on tyler | 01:27 |
| ogasawara | kees: git clone git://kernel.ubuntu.com/ubuntu/kteam-tools.git | 01:28 |
| kees | done | 01:28 |
| ogasawara | kees: cp /home/ogasawara/kteam-tools/kteam-tools/mainline-build/mainline-build-one to /home/kees/kteam-tools/kteam-tools/mainline-build/mainline-build-one | 01:29 |
| ogasawara | kees: It's just a tweaked version of the script to use the tyler chroots | 01:30 |
| tgardner-afk | ogasawara, did you swipe the one I've been using on emerald? | 01:30 |
| kees | ogasawara: okay (removed extra /kteam-tools from path, but good now) | 01:31 |
| ogasawara | tgardner-afk: nope, saw you'd added a README to the repo and then tweaked the script to make it work for me on tyler | 01:31 |
| tgardner-afk | I'm gonna spend tomorrow making them work on any machine, not just zinc | 01:31 |
| ogasawara | kees: oops, copy and paste error | 01:31 |
| ogasawara | tgardner-afk: that'd be awesome, I found it really useful doing these mainline bisects for kees | 01:32 |
| tgardner-afk | ogasawara, yep, I've been doing 'em for 2 days now | 01:32 |
| ogasawara | kees: so I did a mkdir lp597075 | 01:33 |
| === tgardner-afk is now known as tgardner | ||
| ogasawara | kees: just to keep things separate | 01:33 |
| erichammond | tgardner: The currently published Maverick kernel on EC2 is 2.6.32-305.9. Do you know if the "scheduler fixes" you mentioned are available in that version or would we need to wait for the next release? | 01:33 |
| kees | ogasawara: okay | 01:33 |
| ogasawara | kees: then in lp597075 I cloned the mainline git tree, git clone --reference /usr3/ubuntu/linux-2.6.git/ /usr3/ubuntu/linux-2.6.git/ | 01:33 |
| kees | okay, done | 01:34 |
| ogasawara | kees: cd linux-2.6 | 01:34 |
| ogasawara | kees: then run the following to build `/home/kees/kteam-tools/mainline-build/mainline-build-one <sha1> maverick` | 01:35 |
| kees | which sha1 do I want to start with? | 01:35 |
| tgardner | erichammond, hmm, I have not backported the EC2 kernel from Maverick. All I have are the standard kernels. I may have mislead you thinking that a standard kernel would work, but thats only for UEC, not Amazon, right? | 01:35 |
| ogasawara | kees: ah, so in a separate linux-2.6 tree I kept track of the bisect | 01:36 |
| ogasawara | kees: git bisect start v2.6.35-rc2 v2.6.35-rc1 | 01:36 |
| erichammond | tgardner: We are only interested in Amazon EC2 kernels as this is where the application servers run. | 01:37 |
| kees | separate tree? ooooh. on tree doing the bisect, the other tree doing the builds? | 01:37 |
| ogasawara | kees: right | 01:37 |
| ogasawara | kees: I just liked being able to read the bisect output uninterrupted | 01:37 |
| kees | ogasawara: and now I just get to do a full-blown bisect without limits | 01:37 |
| tgardner | erichammond, right. I don't think jjohansen has a working Maverick kernel for EC2 yet, correct? | 01:37 |
| ogasawara | kees: I think I could replay it, but am too lazy to look it up | 01:37 |
| tgardner | ogasawara, you know it keeps a log? .git/BISECT_LOG | 01:38 |
| ogasawara | kees: yep :( looked like around ~1500 commits between rc1 and rc2 | 01:38 |
| ogasawara | tgardner: there is? | 01:38 |
| jjohansen | tgardner, erichammond: yes I am trying to get the EC2 kernel for Maverick up | 01:38 |
| kees | ogasawara: so the sha1 I give is the one git bisect spits out? | 01:38 |
| kees | i.e.: | 01:39 |
| kees | 6$ git bisect start v2.6.35-rc2 v2.6.35-rc1 | 01:39 |
| kees | Bisecting: 373 revisions left to test after this (roughly 9 steps) | 01:39 |
| ogasawara | kees: yep | 01:39 |
| kees | [b1413357d924792e2e332dcb6b712a7fb2a5fb25] fbdev: fix frame buffer devices menu | 01:39 |
| kees | ^^^ that? | 01:39 |
| erichammond | tgardner, jjohansen: I saw a note about that in the latest server-team meeting summary, but I was able to get the Maverick alpha-1 AMI running on EC2. | 01:39 |
| ogasawara | kees: yah, so use b1413.. | 01:39 |
| ogasawara | tgardner: well, I'll be damed. /me makes a note | 01:39 |
| jjohansen | erichammond: right | 01:39 |
| kees | ogasawara: does mainline-build-one correctly maximize CPU parallelization in the build | 01:39 |
| kees | ? | 01:39 |
| tgardner | erichammond, my understanding from jjohansen is that it still fails in some zones | 01:39 |
| ogasawara | kees: not sure about that, but it'd crank em out pretty fast | 01:39 |
| tgardner | ogasawara, kees: it does | 01:40 |
| jjohansen | tgardner, erichammond: specifically the pv-ops stuff fails in some zones, the full Xen has other bugs, and problems | 01:40 |
| kees | fatal: 'maverick' does not appear to be a git repository | 01:40 |
| kees | ogasawara: I seem to be missing something | 01:40 |
| ogasawara | kees: shoot, forgot to have you add the remote | 01:41 |
| kees | ah-ha! | 01:41 |
| erichammond | tgardner, jjohansen: For this Asterisk test, we only need the 64-bit kernel in us-east-1, but it sounds like it will need to wait anyway. | 01:41 |
| kees | ogasawara: git remote add maverick /usr3/ubuntu/ubuntu-maverick.git/ ? | 01:41 |
| ogasawara | kees: git remote add maverick git://kernel.ubuntu.com/ubuntu/ubuntu-maverick.git | 01:41 |
| jjohansen | erichammond: try me again tomorrow, I have another kernel I will be testing soon | 01:41 |
| tgardner | kees, for i in dapper hardy jaunty karmic lucid maverick; do git remote add $i git://kernel.ubuntu.com/ubuntu/ubuntu-$i.git; done | 01:42 |
| kees | oh, not the local one? | 01:42 |
| ogasawara | kees: you could probably use the local one too | 01:42 |
| ogasawara | kees: it's sync'd regularly | 01:42 |
| tgardner | kees, it doesn't make much difference after the first pull | 01:42 |
| kees | ogasawara, tgardner: cool, now it's rockin' :) | 01:43 |
| ogasawara | kees: sweet | 01:43 |
| tgardner | kees, tyler is a wuss, you oughta try emerald. | 01:44 |
| tgardner | or tangerine | 01:44 |
| kees | tgardner: I was doing my Yama builds on tangerine. woof. | 01:44 |
| tgardner | tangerine and emerald are basically the same machine | 01:44 |
| kees | I wonder how much ccache would help when doing a bisect | 01:45 |
| tgardner | kees, I found on emerald and tangerine it makes about a 10 second diff over a full 3 arch pass on Lucid | 01:46 |
| tgardner | so, not much | 01:46 |
| kees | holy cow, that's not so great. :P | 01:46 |
| ogasawara | kees: ok, I gotta bail for a bit. keep me posted how the bisect goes. | 01:48 |
| kees | ogasawara: cool; thanks! | 01:51 |
| ccheney | tgardner, another build failure showed up | 01:51 |
| tgardner | ccheney, copying now | 01:52 |
| ccheney | ok | 01:52 |
| ccheney | 150e6c6 ? | 01:52 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/150e6c67f4bf6ab51e62defc41bd19a2eefe5709-maverick/linux-image-2.6.32-020632rc3g150e6c6-generic_2.6.32-020632rc3g150e6c6_amd64.deb | 01:53 |
| ccheney | tgardner, good | 01:58 |
| ccheney | tgardner, er it works i mean :) | 02:00 |
| tgardner | ccheney, Bisecting: 2 revisions left to test after this | 02:00 |
| ccheney | great :) | 02:01 |
| jjohansen | back on later | 02:01 |
| === jjohansen is now known as jjohansen-afk | ||
| kees | ogasawara, tgardner: should I expect this thing to work on tangerine? it only seems to build on tyler. | 02:08 |
| kees | oh, er | 02:08 |
| kees | /home/kees/kteam-tools/mainline-build/mainline-build-one: line 128: dch: command not found | 02:08 |
| kees | heh | 02:08 |
| tgardner | see mine stuff in emerald:~rtg/kteam-tools/mainline-build | 02:09 |
| tgardner | kees, see my* | 02:09 |
| tgardner | kees, its in a transitional phase. I'm gonna fix it tomorrow so that it works everywhere | 02:10 |
| kees | tgardner: okay, cool | 02:10 |
| * kees runs away for dinner | 02:10 | |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/e00ef7997195e4f8e10593727a6286e2e2802159-maverick/linux-image-2.6.32-020632rc5ge00ef79-generic_2.6.32-020632rc5ge00ef79_amd64.deb | 02:20 |
| ccheney | testing | 02:20 |
| ccheney | tgardner, works | 02:27 |
| tgardner | ccheney, ack | 02:27 |
| ccheney | looks like it must be cc56f7de7f00d188c7c4da1e9861581853b9e92f, the other commit is supposedly ps3 specific | 02:28 |
| tgardner | ccheney, probably, but lets play it through | 02:29 |
| ccheney | yea | 02:30 |
| tgardner | ccheney, this one has a good vibe though. | 02:30 |
| ccheney | tgardner, there is a v2 of the patch, if i understand what is on lkml that was posted on 5/29 | 02:36 |
| ccheney | may be related to the bug we are seeing, if it is indeed that patch causing the problem | 02:37 |
| tgardner | ccheney, I'll have it in 5 minutes | 02:37 |
| ccheney | ok | 02:37 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/cc56f7de7f00d188c7c4da1e9861581853b9e92f-maverick/linux-image-2.6.32-020632rc5gcc56f7d-generic_2.6.32-020632rc5gcc56f7d_amd64.deb | 02:43 |
| ccheney | testing | 02:44 |
| benjamim | Anyone can tell me if the current maverick kernel present on the PPA (2.6.35-5) already has vga-switchero compiled as default ? | 02:47 |
| RAOF | Yes. | 02:47 |
| RAOF | It deos. | 02:47 |
| benjamim | thanks RAOF | 02:48 |
| benjamim | i'm trying to switch to my RADEON graphics with "echo DDIS > /sys/kernel/debug/vgaswitcheroo/switch" | 02:49 |
| benjamim | but i receive a blackscreen | 02:49 |
| benjamim | is the command right ? | 02:49 |
| RAOF | I forget. | 02:50 |
| RAOF | From what I understand, that's highly unlikely to work properly if X is running. | 02:51 |
| ccheney | tgardner, failed test, i am going to verify e00ef79 is actually good and can run instances to be sure its the cc56f7d | 02:51 |
| benjamim | yep, after type that command i restart X | 02:52 |
| benjamim | but the screen just go black | 02:52 |
| benjamim | Only the intel graphics are working | 02:52 |
| RAOF | benjamim: I'd probably grep the git log for the switcheroo commit to see what to do. | 02:53 |
| tgardner | ccheney, this next kernel is the absolute proof since that last commit is now skipped (if it doesn't fail) | 02:53 |
| ccheney | tgardner, ok | 02:54 |
| benjamim | RAOF, could you tell what is the file that the switchero log is stored ? | 02:55 |
| RAOF | benjamim: It's not - it's in the VCS history. You'd need to clone the kernel's git tree. Alternatively, go to git.kernel.org and use the search box in Linus' tree. | 02:56 |
| benjamim | oh, i did not know that, I'll take a look. | 02:58 |
| benjamim | thanks for help RAOF | 02:58 |
| ccheney | tgardner, yea e00ef79 ran an instance so its good, i saw a different type of failure but we think its a different unrelated bug | 02:59 |
| ccheney | the other bug is probably due to eucalyptus not doing error checking | 03:00 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/f21121cde6e617b90cd03ce083652ca543004dc2-maverick/linux-image-2.6.32-020632rc5gf21121c-generic_2.6.32-020632rc5gf21121c_amd64.deb | 03:08 |
| ccheney | testing | 03:08 |
| ccheney | tgardner, looks good so far | 03:12 |
| tgardner | ccheney, play it all the way, just like the others. | 03:13 |
| ccheney | tgardner, its still running, just giving early feedback :) | 03:15 |
| ccheney | takes around 15m to run the test 10 times | 03:16 |
| * ccheney has to go to the grocery store, bbia 1hr | 03:16 | |
| ccheney | so far passed the test 3/3 with 7 left to try | 03:17 |
| tgardner | ccheney, k, see you in a bit | 03:17 |
| tgardner | ccheney, back in the AM | 04:27 |
| ccheney | i got swallowed by walmart | 05:16 |
| ccheney | sorry for being gone so long :-\ | 05:16 |
| * ccheney needs to remember to look at the list his wife writes up before leaving to make sure it is detailed enough to buy | 05:17 | |
| jk-- | 1) "items" | 05:25 |
| === kamal is now known as kamal-away | ||
| smb | Morning | 08:14 |
| cking | Good morning smb! Welcome to another wonder filled day | 08:16 |
| smb | cking, Too true | 08:16 |
| smb | cking, Just experienced the wonders of democratic bit reading. :-P | 08:16 |
| LLStarks | apw or ogasawara: any chance that enable-multiple-ring-buffer.patch can get backported for maverick? | 08:16 |
| * cking wonders what "democratic bit reading" is | 08:17 | |
| smb | LLStarks, Not apw and ogasawara will not be around till later | 08:17 |
| LLStarks | okay | 08:17 |
| LLStarks | any other kernel managers that i should speak to? | 08:17 |
| smb | cking, You read 5 times and let the majority decide whether its 0 or 1 | 08:17 |
| cking | smb, not sure that's entirely a good idea. a dictatorship suits me better. as long as I'm in control ;-) | 08:18 |
| smb | LLStarks, Depends on the urgency and the patch. And as I still wait for my mail to sync I don't even know whether apw has written something about this patch | 08:19 |
| smb | cking, In my case democracy seems the better choice as that allows the EDID to be ok even with some monitor switch in between | 08:20 |
| LLStarks | it's for vaapi support on intel | 08:20 |
| cking | smb, I get your drift now | 08:20 |
| LLStarks | kinda stupid to not have it since libva is now in ubuntu archives | 08:20 |
| LLStarks | http://intellinuxgraphics.org/h264.html | 08:21 |
| smb | LLStarks, Have you sent the patch to kernel-team@lists.ubuntu.com with some short explanation? | 08:21 |
| LLStarks | i'm not on the mailing list | 08:21 |
| RAOF | That ended up missing 2.6.35, did it? It's been on the intel-gfx lists for some time - I kinda thought it made it into Linus' tree. | 08:22 |
| LLStarks | made 2.6.35 from what i've heard | 08:22 |
| smb | LLStarks, You could still send to it. Though it then take a bit of time for it to get moderated to the list | 08:22 |
| LLStarks | not sure though | 08:22 |
| RAOF | Then rejoice, for Maverick will have 2.6.35 | 08:22 |
| LLStarks | like i said, i'm not sure if it made the cut | 08:23 |
| LLStarks | ah. it did. http://permalink.gmane.org/gmane.comp.video.dri.devel/46855 | 08:24 |
| RAOF | git log says that its in 2.6.35-rc3, so it's in Maverick now. | 08:24 |
| LLStarks | hopefully everything else required falls into place. | 08:24 |
| RAOF | The easiest form of backporting! | 08:24 |
| smb | RAOF, Very true | 08:25 |
| smb | RAOF, Btw, just out of interest. There have been a few bugs about invalid EDIDs. Do you know whether they are all over adapter space or mainly intel? | 08:25 |
| RAOF | The EDID processing code is in the shared drm code. EDID processing recently got more strict about the checksum being accurate (ie: actually checking the checksum, rather than completely ignoring it). | 08:27 |
| smb | RAOF, Right the EDID code itself is. But the implementation of the bit banging functions are driver specific. In my case intel_i2c.c | 08:28 |
| smb | RAOF, And I seem to experience flakyness when the monitor cable is not directly connected to the laptop | 08:28 |
| smb | RAOF, But as of this mornings "democratic" approach it seems to work. With mostly a 4:1 vote for the bit being 0 | 08:29 |
| RAOF | Urgh :) | 08:29 |
| smb | Oh yeah. :) | 08:29 |
| smb | Especially as the i915 code has so many bit values defined that I am completely lost about their intention | 08:30 |
| RAOF | I'm not sure how I'd tell if a bad EDID on a bug was due to the monitor or the i2c interface. It's not something I've regularly asked reporters about. | 08:30 |
| RAOF | And there are plenty of monitors with bad EDIDs to go around. | 08:31 |
| smb | Yes, that might be specific to my case. The interesting part would be the number of messages. If the remainder is always the same, I guess the EDID /monitor is at fault | 08:32 |
| smb | If, like in my case, the remainder always changes, there is flakyness | 08:32 |
| RAOF | Yeah. I *have* noticed some changes in that sort of area for i915 but not radeon or nouveau. | 08:33 |
| smb | That would sort of match my experience that all radeon (I think I have not tried nvidia) laptops when connected to that same monitor switch would get the right EDID but not that i915 based netbook | 08:37 |
| smb | And as it drove me nuts (because I mainly work on that netbook and it would only allow the high mode after much cursing and tricking around) I played around with that yesterday and today | 08:38 |
| RAOF | I think it might be at least in part because Intel chips are entirely integrated, so they rely on third parties to actually wire stuff up. ATi and nVidia both either (a) are discrete cards, wired up by people whose job it is to wire up discrete cards, or (b) on motherboards wired up by AMD or nVidia. | 08:39 |
| smb | Maybe also because compared to radeon i2c code which only seems to set unset a bit in the registers, the intel implementation has dir bits and a mask and val bits and a mask and I think I really need to talk to somebody that can actually make sense of that. :) I'll try Eric, maybe he can tell me what all of this means | 08:42 |
| cking | smb, http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-hardy.git;a=commit;h=13f8acafe67361d7afda05c23a0408820e0cb468 | 11:12 |
| === ghostcube_ is now known as ghostcube | ||
| === ayan-afk is now known as ayan | ||
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/3b2e8e02ca5a31b6f8db8de05becb613d819622a-maverick/linux-image-2.6.35-6-server_2.6.35-6.7_amd64.deb | 13:35 |
| ccheney | tgardner, thanks, testing | 13:48 |
| ccheney | tgardner, seems to be working, doing full test and going to run instance to be sure | 13:58 |
| tgardner | ccheney, ok, I've been looking at the patch. most of the changes are just pointer guards, but there is one change that could affect the behavior. | 13:58 |
| ccheney | ok | 13:59 |
| ccheney | tgardner, the pad block corrupted message doesn't appear but it seems to not run the instance, i am going to try it again and see if it will work | 14:28 |
| ccheney | tgardner, its possible there is more than one bug, i am going to look into finding other ways to verify if the image was registered properly | 14:29 |
| === sconklin-gone is now known as sconklin | ||
| tgardner | ccheney, a different behavior then the previous successful runs? | 14:34 |
| ccheney | with this kernel it appeared to register the image properly but then when i tried to launch an instance using the image it did not work, both worked before, i may need to retest your last kernel from last night to make sure it was working for running instance, but i am pretty sure it was | 14:35 |
| ccheney | i slept since then so i have forgotten a bit :-\ | 14:35 |
| tgardner | ccheney, retry 2.6.35-5.6 to verify it still produces the PAD error, then I'll build a kernel based on that version but with the offending commit reverted. | 14:37 |
| ccheney | ok will do | 14:37 |
| ccheney | tgardner, it ran this time, i think its probably an issue with eucalyptus that caused it to not work the first attempt | 14:40 |
| tgardner | ccheney, does it pass the 10 loop test? | 14:40 |
| ccheney | i'll try a few more times to make sure it consistently is working, but i wouldn't be surprised if it was euca for that part | 14:40 |
| ccheney | tgardner, yea the 10 loop test passed, that just tests the registration for the pad bug | 14:41 |
| ccheney | this is still on 2.6.35-6 | 14:41 |
| ccheney | the step i do after testing the pad bug is gone is to run the instance which seemed to hang the first attempt but worked on the second | 14:41 |
| ccheney | will get back to you soon about it, talking to the guys about it | 14:41 |
| tgardner | ccheney, just to be clear, you're running the kernel with cc56f7de7f00d188c7c4da1e9861581853b9e92f reverted? | 14:42 |
| ccheney | i am running http://kernel.ubuntu.com/~rtg/mainline/3b2e8e02ca5a31b6f8db8de05becb613d819622a-maverick/linux-image-2.6.35-6-server_2.6.35-6.7_amd64.deb | 14:42 |
| tgardner | ccheney, k, I'll add that to the bug report | 14:43 |
| ccheney | tgardner, ok | 14:43 |
| ccheney | tgardner, yea it seems to fix it, it only failed the first time of running instance and i don't have a great track record of instances running on my box anyway due to other weird eucalyptus bugs | 14:48 |
| ccheney | so i would consider the test to be successful | 14:48 |
| ttx | tgardner: what are our options wrt A2 ? would you consider using that "current minus problematic patch" kernel for A2 ? | 14:48 |
| ttx | s/patch/commit/ | 14:49 |
| tgardner | ttx, um, I don't completely understand _why_ that patch causes a failure. As far as I can tell it merely implements more rigorous error checking, which the Java machine may be tripping over. I'm still of the opinion that you should release with a Lucid kernel 'cause I'm not yet willing to advise that we revert that commit for the master kernel. | 14:53 |
| ttx | tgardner: hm... | 14:54 |
| tgardner | ttx, Linus has been gone for 2 weeks, so there may well be fixes for this already in the pipeline. I'll bug the authors about it. | 14:56 |
| ttx | tgardner: I thought there would be more value in testing a 2.6.35 kernel in the A2 milestone than to revert to a Lucid one. | 14:56 |
| ttx | tgardner: there have been some patches proposed by the same author that seem to "fix" things, ccehney found them | 14:56 |
| tgardner | ttx, perhaps I read a different thread. Which one did you read? | 14:57 |
| ttx | No clue if they fix anything relevant for us though | 14:57 |
| ttx | might be worth generating a test kernel with them and see | 14:57 |
| * ttx looks at reference | 14:57 | |
| tgardner | ttx, I read this one http://marc.info/?l=linux-fsdevel&m=127488511324250&w=2 which was inconclusive | 14:58 |
| ttx | tgardner: there are a few others at http://marc.info/?a=119980668900012&r=1&w=2 but I have no clue if they are more conclusive | 14:59 |
| tgardner | ttx, which is why I thought I'd contact the authors, 'cause I also have no clue | 15:00 |
| ttx | tgardner: if you're more comfortable with shipping a Lucid kernel for Maverick A2 than the modified 2.6.35 one, we should go that way | 15:01 |
| ttx | I'm missing historical referenbce to see what most acceptable for an A2 | 15:01 |
| ttx | tgardner: maybe that's a subject for the release meeting in one hour | 15:02 |
| tgardner | ttx, what is the use of testing a Maverick kernel in UEC if we _already_ know it doesn't work. It seems to me that it'll preclude testing of other UEC features. | 15:03 |
| ttx | tgardner: agreed, by "modified 2.6.35 one" I was thinking the one that ccheney just tested | 15:04 |
| ttx | the issue being it affects more than just UEC, but all servers | 15:04 |
| tgardner | ttx, well, I'm not ready to recommend that ogasawara revert that commit. I assume that the A2 release _must_ use the kernel from the archive, correct? | 15:05 |
| ttx | tgardner: that's my understanding as well | 15:05 |
| ogasawara | tgardner: and I'd prefer not to have to spin a new kernel, I uploaded what I hoped to be our final A2 kernel last night | 15:05 |
| tgardner | ogasawara, yep, agreed | 15:06 |
| ogasawara | and it's cutting it close to do another upload | 15:06 |
| ttx | tgardner: if A2 uses the one from the archive, how can you make it "ship the Lucid kernel" ? | 15:07 |
| ttx | another solution is to just releasenote the fact that for UEC, you should revert to a Lucid kernel due to a yet-to-be-investigated bug | 15:08 |
| tgardner | ttx, I guess its a matter of seeding? The Lucid kernel _does_ live in the archive, just in a different releas. | 15:08 |
| ttx | tgardner: ah ok. | 15:08 |
| ccheney | i'm not sure if you can seed something not in your pocket | 15:09 |
| ccheney | but that would be a question to ask someone like cjwatson probably | 15:09 |
| ttx | ogasawara: I'm unsure we should just prevent non-UEC server use cases from testing the 2.6.35 kernel in the A2 milestone | 15:09 |
| ccheney | if we can't and we want to use the old kernel we can probably get it copied to maverick | 15:09 |
| ttx | documenting the workaround for the UEC users sounds like the best move here | 15:09 |
| ccheney | hmm actually if it has the same source package name then probably the only way to do it is by telling users to install (but i may be wrong) | 15:10 |
| * tgardner gets breakfast. biab. | 15:10 | |
| ttx | tgardner, ogasawara: i'll mention the issue in the release meeting since the release team should decide on what they prefer to have in A2 | 15:11 |
| ttx | (and will confirm what our options are, if we have more than one) | 15:11 |
| ogasawara | ttx: ack | 15:12 |
| ccheney | ttx, yea our only options might be to either respin or to document in the a2 release notes, but i don't know if they have some magic they can perform on the image outside of regular seed generation | 15:12 |
| cnd | is there a way to keep my system at 10.04, but somehow tell apt to upgrade my kernel from maverick? | 15:15 |
| cnd | so far I've just been manually installing kernels | 15:15 |
| ttx | ccheney: that's what I'll ask at the release meeting | 15:16 |
| tgardner | cnd, you could run the LTS backport. | 15:16 |
| cnd | tgardner: can you remind me whether that differs in any way from the real maverick kernel? | 15:17 |
| cnd | config options? | 15:17 |
| tgardner | cnd, should be identical. | 15:17 |
| cnd | tgardner: are they now produced at the same time as the maverick kernels, or do they lag? | 15:17 |
| tgardner | cnd, the only diff is that the backport is built with the Lucid toolchain | 15:18 |
| tgardner | cnd, obviously it lags because its dependent on maverick, but usually no more then a day or so | 15:18 |
| cnd | ok | 15:18 |
| cnd | thanks | 15:18 |
| tgardner | cnd, it depends somewhat on the crazy shit ogasawara has done to maverick. I'm working on the backport right, but apparmor is giving me some grief. | 15:19 |
| cnd | heh | 15:20 |
| ccheney | tgardner, anything else you would like me to test? whenever the other patch gets reviewed by Linus I can test it out as well | 15:31 |
| tgardner | ccheney, it'll likely take a few days. | 15:32 |
| ccheney | tgardner, no problem just let me know when you are ready | 15:32 |
| tgardner | ccheney, just to be clear, this PAD error is emitted by a node host kernel when it is attempting to unpack and load a KVM guest, right? | 15:42 |
| JFo | tgardner, is this bug on your radar? bug 597904 | 15:42 |
| ubot2 | Launchpad bug 597904 in linux (Ubuntu Maverick) (and 1 other project) "No video on beagleboard with 2.6.35 kernels. (affects: 1) (heat: 10)" [Critical,New] https://launchpad.net/bugs/597904 | 15:42 |
| tgardner | JFo, it should get on mpoirier's radar | 15:42 |
| JFo | k | 15:43 |
| JFo | mpoirier, you around? | 15:43 |
| mpoirier | sure am | 15:43 |
| ccheney | tgardner, its emitted when attempting to uec-publish-tarball on the walrus server | 15:43 |
| JFo | k, have you seen that bug I just pointed to tgardner? | 15:43 |
| mpoirier | hold on, looking at it... | 15:43 |
| JFo | k | 15:43 |
| ccheney | tgardner, the node doesn't do anything with it at all, the node is the system running the kvm instance that you start using the image off the walrus server | 15:43 |
| ccheney | tgardner, i hope that was clear, i can try to reword a bit better if it wasn't :) | 15:44 |
| tgardner | ccheney, clear as mud. explain in a language that upstream will understand | 15:44 |
| ccheney | tgardner, ok | 15:44 |
| mpoirier | JFo: I am aware of the issue - been present since I joined the company. | 15:44 |
| JFo | cool | 15:45 |
| JFo | just worried as I got mail indicating it was critical, etc. | 15:45 |
| ccheney | tgardner, so this bug shows up on the server running 'walrus' which is the equivalent of Amazon S3 if you know what that is? | 15:45 |
| ccheney | tgardner, reading my uecglossary page :) | 15:45 |
| mpoirier | JFo: they are all critical right now... | 15:45 |
| JFo | and they set milestone to Alpha2 and the kernel freeze is today | 15:45 |
| mpoirier | JFo wont' happen. | 15:45 |
| ccheney | tgardner, so its running on the box not running the node controller, so not the one running the images in VMs | 15:45 |
| ccheney | tgardner, when you start an instance (VM) the software provides the image you tell it to use via 'walrus' S3 to the node controller to use as the base image to run for the VM | 15:46 |
| mpoirier | JFo: we need to coordinate with the mobile team. | 15:46 |
| JFo | yeah, I agree | 15:46 |
| ccheney | tgardner, our bug is failing when attempting to register the image (make it available for use) to the 'walrus' S3 service | 15:46 |
| JFo | just wanted to verify that no one had given them the idea that it was possible :) | 15:47 |
| tgardner | ccheney, more fundamentally, this is the walrus host kernel attempting to decrypt a file? | 15:47 |
| ccheney | tgardner, to explain better than that i may need to defer to ttx | 15:47 |
| ccheney | tgardner, yes | 15:47 |
| JFo | thanks for looking mpoirier | 15:47 |
| ccheney | tgardner, so not a vm kernel, actual bare metal kernel | 15:47 |
| ttx | tgardner: in fact it's walrus calling a Java decryption routine | 15:47 |
| tgardner | ccheney, and your 10 cycle test simply runs that decryption 10 times? | 15:48 |
| mpoirier | JFo. I'll have a chat with tgardner on how to proceed with the workload and what to prioritize first. | 15:48 |
| ccheney | ttx, i couldn't find where bouncycastle or openjdk-6 is actually using sendfile but i may be lacking my grep skills | 15:48 |
| ttx | when used against large files, ultimately it fails with that pad block error | 15:48 |
| ccheney | tgardner, essentially, my script attempts to register the image which does various things unknown to me including the decryption | 15:48 |
| tgardner | ttx, ccheney: ok, that seems to be the fundamental issue. | 15:48 |
| ccheney | tgardner, and repeats that 10 times, yes | 15:48 |
| ccheney | ttx: i did notice even when it does work it mentions something about the file being too large in the euca logs | 15:49 |
| ccheney | ttx, i don't know if you saw that message before | 15:49 |
| ttx | I suspect the JVM does things that are no longer well supported by the kernel | 15:49 |
| ccheney | some cache error message i'll see if i can find it, might be in the bug report, will check | 15:50 |
| ttx | smoser pointed at java.nio.channels.FileChannel transferTo using sendfile | 15:50 |
| JFo | mpoirier, ok | 15:50 |
| ccheney | hmm the cache message seems not in the bug report, i'll add it in case its useful | 15:50 |
| ccheney | ttx, do you happen to know what package that is in? seems not in the main openjdk-6 source | 15:51 |
| * ttx looks into his mighty java-Contents file | 15:51 | |
| * ccheney doesn't know enough about java to know how to locate where it would be | 15:52 | |
| ttx | openjdk-6-jre-headless rt.jar java.nio.channels.FileChannel | 15:52 |
| ccheney | if its a bug in java we can get doko to look into it | 15:52 |
| smoser | it would seem unlikely to me that java would utilize and expose something that isn't standard. | 15:52 |
| ccheney | hmm weird | 15:52 |
| ccheney | i grepped over openjdk-6 source | 15:52 |
| tgardner | ttx, do you think you could develop a simple reproducer? Something that an upstream developer could use to stimulate the bug without having to have all of the UEC stuff? | 15:52 |
| ttx | ccheney: didn't the eucalyptoids provide us with that ? | 15:53 |
| smoser | its possible that its a java bug, but i would think, given 2 years of function and then breakage in linux and expectation of java to run cross unix, that its kernel regression. | 15:53 |
| ccheney | ttx, iirc someone said there is a huge reproducer in java attached to the bug | 15:53 |
| tgardner | smoser, its clearly kernel regression (or at least a change in behavior) | 15:53 |
| ttx | right, it's just unclear why | 15:53 |
| ccheney | yea 134MB test program | 15:53 |
| tgardner | ah, I see it. | 15:54 |
| ccheney | its either regression or undefined behavior that was relied on | 15:54 |
| tgardner | ccheney, If I can reproduce it, then I can likely narrow down the exact line in the commit that is causing this issue | 15:55 |
| ccheney | tgardner, yea, looking to see if i can find the sendfile code used | 15:55 |
| ccheney | umm we have patches in the FileChannel area | 15:56 |
| * ccheney hopes we didn't cause this bug ourselves | 15:56 | |
| ccheney | actually hmm no those aren't ubuntu patches but something else | 15:56 |
| * ccheney upstream source looks a bit odd | 15:56 | |
| ttx | ccheney: the bug is occurring on RedHat too | 15:57 |
| ttx | ccheney: so it's not "just us" | 15:57 |
| ccheney | oh ok | 15:57 |
| ttx | it's anyone with 2.6.34+ | 15:57 |
| tgardner | ttx, do you have a reference to the RedHat report? | 15:57 |
| ccheney | yea the patches are from icedtea, which i think is similar to ooo-build for java | 15:57 |
| ccheney | i still don't see a reference to sendfile though | 15:57 |
| ttx | it's a forum post at eucalyptus.com | 15:57 |
| ttx | http://open.eucalyptus.com/forum/decrypting-image-exception | 15:58 |
| ccheney | maybe it gets generated in some way | 15:58 |
| * ccheney is going to write up the preliminary information to the mailing list in a few minutes | 15:58 | |
| === virtuald_ is now known as virtuald | ||
| * ccheney still isn't on the list apparently, or hasn't gotten the response back yet | 16:00 | |
| ccheney | ah ha my openjdk-6 source wasn't fully unpacked it has a bunch of extra stuff plus a huge tarball in the dir | 16:01 |
| ccheney | no wonder i saw no reference to sendfile in my grep | 16:02 |
| ccheney | now i found it | 16:02 |
| tgardner | ogasawara, the only item for us in the release meeting is the 'pad block corrupted' issue. You're aware of the status changes, i.e., we've narrowed it down to a single kernel commit? | 16:02 |
| ccheney | jdk/src/solaris/native/sun/nio/ch/FileChannelImpl.c | 16:03 |
| ogasawara | tgardner: yep, got it all written up to paste | 16:03 |
| tgardner | ogasawara, cool. on top of things as always :) | 16:03 |
| ccheney | its definitely special cased for solaris vs linux | 16:04 |
| ccheney | so not fully standard across *nix anyway | 16:05 |
| ccheney | http://pastebin.com/z5juRcCB | 16:05 |
| === kamal-away is now known as kamal | ||
| === bjf[afk] is now known as bjf | ||
| cnd | apw: ogasawara: I want to grab the latest kernel ddeb, but the amd64 version is missing | 16:45 |
| cnd | I see that the previous kernel, 2.6.35-5, is also missing amd64 ddebs | 16:46 |
| ogasawara | cnd: Is it still building? | 16:46 |
| ogasawara | amd64 that is | 16:46 |
| tgardner | cnd, its still building | 16:46 |
| cnd | oh | 16:46 |
| tgardner | ccheney, https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/588861/comments/10 doesn't reproduce the bug. | 16:46 |
| ubot2 | Launchpad bug 588861 in linux (Ubuntu Maverick) (and 4 other projects) ""pad block corrupted" error when trying to register an image with 2.6.34 kernel (affects: 1) (heat: 12)" [High,In progress] | 16:46 |
| ccheney | tgardner, oh ok | 16:47 |
| cnd | ogasawara: tgardner: launchpad says it's been done building for 2 hours | 16:47 |
| ccheney | tgardner, i'll grab it and see if it happens with that for me | 16:47 |
| tgardner | ccheney, dang, it would be nice to a have a simple reproducer | 16:47 |
| tgardner | cnd, it takes awhile for the publisher to run, plus things may be starting to freeze for A2 | 16:48 |
| ccheney | tgardner, i'm writing an email to upstream about it so maybe they will know how to make a simple test case | 16:48 |
| cnd | tgardner: ok, I suppose I'll chill for a bit | 16:48 |
| ccheney | er upstream eucalyptus i mean not kernel | 16:48 |
| tgardner | ccheney, hmm, this is gonna be a kernel issue I think. | 16:48 |
| tgardner | unless the java engine is just borked | 16:49 |
| ccheney | tgardner, yea i don't know the code well enough to be able to tell | 16:50 |
| tgardner | ccheney, its in openjdk-6-jdk, right? | 16:50 |
| ccheney | tgardner, yes sendfile is in that, if you download it and unpack the source there is a large tarball with the actual source inside of that | 16:51 |
| tgardner | working on it.... | 16:51 |
| ccheney | so double packed debian source | 16:51 |
| ccheney | thats how i managed to overlook the sendfile call earlier | 16:51 |
| ccheney | tgardner, i attached a pastebin link of the file using it to the bug report earlier | 16:52 |
| ogasawara | jjohansen: just fyi, I went ahead and marked the two apparmor A2 work items you had as done (sync distro apparmor with upstream version, and update compatibility patch) | 17:01 |
| jjohansen | ogasawara: thanks, I meant to do that | 17:01 |
| ogasawara | jjohansen: was selfish really, I just wanted our burn down charts to look good for the release meeting :) | 17:01 |
| jjohansen | hehe :) | 17:02 |
| * ogasawara bails for a few hrs | 17:04 | |
| ogasawara | tgardner: ^^ deflect any fires should there be any | 17:05 |
| GrueMaster | JFo: Any suggestions on how to get apport data through a serial console on a system that doesn't have a released kernel? Re: bug 597904 | 17:08 |
| ubot2 | Launchpad bug 597904 in linux (Ubuntu Maverick) (and 1 other project) "No video on beagleboard with 2.6.35 kernels. (affects: 1) (heat: 10)" [Critical,New] https://launchpad.net/bugs/597904 | 17:08 |
| JFo | GrueMaster, not off the top of the head | 17:09 |
| GrueMaster | Then QUIT MARKING OMAP BUGS AS INVALID. | 17:10 |
| GrueMaster | thank you. | 17:10 |
| JFo | GrueMaster, before you start yelling... probably best to give an example where I did | 17:10 |
| GrueMaster | I just did. | 17:11 |
| JFo | GrueMaster, get your facts straight | 17:11 |
| JFo | I set it incomplete | 17:11 |
| JFo | because I asked for info | 17:12 |
| JFo | and for the record, and your general fund of knowledge | 17:12 |
| JFo | I don't take people yelling at me very well | 17:12 |
| JFo | especially when they arewrong | 17:12 |
| GrueMaster | System isn't booting. Best info we are able to gather is through minicom logs. | 17:12 |
| JFo | please refer to my boss pgraner if you have issues\ | 17:12 |
| GrueMaster | This isn't an x86 system that has multiple tons of ways to collect this info. | 17:13 |
| GrueMaster | And because we need to get it working, we are getting hand-built kernels prior to release (which apport considers invalid). | 17:13 |
| JFo | I know that, hence my question as to whether it was possible | 17:13 |
| GrueMaster | Nope. Not possilble. | 17:14 |
| JFo | like I said, get your facts straight | 17:14 |
| JFo | then fine, add a comment to the bug to that effect, if necessary and we can move on | 17:14 |
| JFo | see how easy that could have been? | 17:14 |
| * smb moves on to boc and we. See you next week | 17:21 | |
| GrueMaster | JFo: sigh. I owe you an apology. I'm letting my frustrations of trying to get this image working get the best of me. I shouldn't have taken it out on you like that. | 17:28 |
| JFo | apology accepted. If you need to vent to me about something like that in the future, may I suggest a private window? :) | 17:29 |
| * JFo steps away for lunch | 17:30 | |
| tgardner | mpoirier, you around? | 17:31 |
| mpoirier | I am. | 17:31 |
| mpoirier | whazup ? | 17:31 |
| tgardner | mpoirier, did Lucid omap3 have video? Is this a maverick regression? | 17:31 |
| kees | ogasawara, tgardner: this bisect script rocks. :) | 17:31 |
| ogra | tgardner, maverick regression | 17:32 |
| mpoirier | let's mumble... | 17:32 |
| tgardner | kees, If I can get clear of UEC I'll try and make it better | 17:32 |
| GrueMaster | tgardner: Yes, Lucid works fine on omap 3. In fact, I tried the latest 2.6.35 kernel on that image as well. | 17:32 |
| kees | tgardner: works great for me already. :) | 17:33 |
| kees | ogasawara: I seem to be zeroing in on the middle of a mess of i915 commits again... we'll see how it goes. | 17:33 |
| GrueMaster | I have just been informed that there is now a 2.6.35-6 release kernel (not a hand rolled one). I'm going to pull that and retry it just to verify. Might even get some apport data. | 17:34 |
| * JFo crosses his fingers | 17:37 | |
| sconklin | kees: I'm about to leave for the day, but if you end up sending email, I'm interested in anything having to do with i915 | 17:45 |
| kees | sconklin: okay, sure. are you seeing problems too? | 17:45 |
| sconklin | kees: I'mnot, but I don't know which problems, release, or hardware you're talking about - I'm just trying to keep up with i915 in general | 17:46 |
| kees | sconklin: okay, cool. I have a Q35 intel mainboard with onboard G35 (?) that since -rc2 hangs on boot. I seem to be the only person with this problem. :P | 17:53 |
| kees | (pci id 8086:29b3) | 17:53 |
| sconklin | kees: Hmm. Hang is not good. I'm not paying much attention at all to maverick recently, been working on lucid stable updates. Have you tried the simple things like disabling kms? | 17:54 |
| kees | sconklin: booting with "nomodeset" you mean? | 17:55 |
| sconklin | kees: yes | 17:55 |
| kees | yeah, that doesn't work. :) | 17:55 |
| kees | the only way I could get past the initramfs was to blacklist i915 _and_ intel_agp. | 17:55 |
| sconklin | kees: that's a new one on me | 17:55 |
| sconklin | sorry | 17:55 |
| kees | then once into root fs, they weren't blacklist any more (initramfs bug) and udev immediately loads them and hangs the system. | 17:55 |
| kees | and if I blacklist them on the rootfs, when X starts, it ignore the blacklist and loads them again. it's been a really fun few days. ;) | 17:56 |
| maks_ | AGP needs to be built in. | 17:58 |
| maks_ | we got heavily hit by having it modular afaik upstream is not properly testing modular agp | 17:58 |
| maks_ | * [alpha, amd64, i386, amd64, powerpc] Make all AGP driver built-in to | 17:59 |
| maks_ | workaround race-condition between DRM and AGP. | 17:59 |
| maks_ | it was directly asked by airlied. | 17:59 |
| sconklin | kees: ^^ | 18:01 |
| kees | oh, lovely | 18:02 |
| kees | ogasawara: ^^ | 18:03 |
| ccheney | tgardner, yea their test case works for me even on the broken kernel, so its not good enough test apparently | 18:03 |
| ccheney | tgardner, i'm going to try to modify it to use the same image i was previously using to see if that makes a difference | 18:03 |
| tgardner | kees, the race is a generic problem. we had a macro that would allow you to force a symbolic dependency, but ogasawara ripped it out of Maverick 'cause it wasn't getting used anywhere. | 18:04 |
| kees | tgardner: er, so, is my problem not due to this? /me isn't sure what to test next. | 18:04 |
| tgardner | kees, I'm not sure. Try changing the config to build-in AGP | 18:05 |
| kees | will do | 18:06 |
| GrueMaster | JFo: No joy here. | 18:10 |
| GrueMaster | *** Problem in linux-image-2.6.35-6-omap | 18:10 |
| GrueMaster | The problem cannot be reported: | 18:10 |
| GrueMaster | This is not a genuine Ubuntu package | 18:10 |
| tgardner | ccheney, I'm building an instrumented kernel that might give me enough information to indicate which part of that patch is causing problems,. | 18:10 |
| ccheney | ok | 18:10 |
| GrueMaster | I might be getting this due to bug 595949. | 18:12 |
| ubot2 | Launchpad bug 595949 in linux-meta-ti-omap (Ubuntu Maverick) (and 3 other projects) "linux-meta-ti-omap depends on the wrong binary kernel in maverick (affects: 1) (heat: 564)" [Low,In progress] https://launchpad.net/bugs/595949 | 18:12 |
| === sconklin is now known as sconklin-gone | ||
| tgardner | GrueMaster, this is omap3, right? | 18:14 |
| GrueMaster | yes | 18:14 |
| tgardner | GrueMaster, k, lemme have a look | 18:14 |
| GrueMaster | It may be possible that this is because the kernel was just released and not on my mirror yet. | 18:15 |
| tgardner | GrueMaster, AFAIK it hasn't completed building yet. ogasawara upload the meta package before amd64 or armel were complete | 18:16 |
| GrueMaster | I have the kernel. It was at https://edge.launchpad.net/ubuntu/+source/linux/2.6.35-6.7/+build/1811239 | 18:17 |
| GrueMaster | or am I missing something? | 18:17 |
| tgardner | GrueMaster, what meta-package are you using? It should be linux-omap or linux-image-omap | 18:17 |
| GrueMaster | This is a hand rolled image initially running 2.6.34-5. I downloaded linux-image-2.6.35-6-omap from the above link. | 18:18 |
| tgardner | GrueMaster, ok, so why do you think you're hitting bug # 595949 ? | 18:19 |
| GrueMaster | Has it been fixed yet? | 18:20 |
| GrueMaster | My image was created 6/22. | 18:20 |
| tgardner | GrueMaster, yes, but I just noticed that there might still be a bogus package in the archive. | 18:20 |
| GrueMaster | And I haven't seen anything kernel related with apt-get update. | 18:21 |
| GrueMaster | hmmm | 18:21 |
| tgardner | GrueMaster, lemme chat up some archive admins about this | 18:21 |
| tgardner | GrueMaster, 'dpkg -l|grep ti-omap' | 18:25 |
| GrueMaster | none. | 18:26 |
| tgardner | GrueMaster, then you don't have the meta package installed? | 18:26 |
| GrueMaster | as I said earlier, it is broken (or was on 6/22 when this image was created). | 18:26 |
| tgardner | GrueMaster, um, try 'dpkg -l|grep omap' | 18:27 |
| GrueMaster | ii linux-image-2.6.34-5-omap 2.6.34-5.14 Linux kernel image for version 2.6.34 on OMA | 18:27 |
| GrueMaster | ii linux-image-2.6.35-2-omap 2.6.35-2.3 Linux kernel image for version 2.6.35 on OMA | 18:27 |
| GrueMaster | ii linux-image-2.6.35-6-omap 2.6.35-6.7 Linux kernel image for version 2.6.35 on TI | 18:27 |
| tgardner | GrueMaster, what happens with 'sudo apt-get install linux-image-omap' ? | 18:28 |
| GrueMaster | "apt-cache search ti-omap" only lists the linux-headers for 2.6.33|34|35 and omap4 meta packages. | 18:29 |
| tgardner | its not ti-omap, just omap | 18:29 |
| GrueMaster | It wants to revert to 2.6.35-5-omap kernel. | 18:30 |
| psusi | anyone care to apply the proposed patch in bug #595489 | 18:32 |
| tgardner | GrueMaster, after you've done an update? 'apt-get update' ? | 18:32 |
| ubot2 | Launchpad bug 595489 in linux (Ubuntu) (and 1 other project) "lvm snapshot causes deadlock in 2.6.35 (affects: 1) (heat: 6)" [High,In progress] https://launchpad.net/bugs/595489 | 18:32 |
| GrueMaster | Just a sec. Updated mirror, now updating apt. | 18:33 |
| tgardner | psusi, it ain't gonna happen for A2, but I've milestoned it for A3 | 18:34 |
| psusi | doh | 18:35 |
| tgardner | psusi, is Eric gonna get it into .35 ? Linux won't be back until next week. | 18:35 |
| tgardner | Linus* | 18:36 |
| psusi | he fired off the patch to the mailing lists and I guess it's waiting for Linus to apply | 18:36 |
| psusi | but since kernel freeze starts today, I think we're going to have to apply it ourselves? | 18:36 |
| tgardner | psusi, ok, this issue won't get lost now that its milestoned. I would prefer to get the fix direct from Linus' tree | 18:37 |
| psusi | yes, but that won't happen because of the kernel freeze will it? | 18:37 |
| tgardner | psusi, we don't freeze the kernel until the official 2.6.35 is released | 18:37 |
| psusi | ohh... I thought I read that today begins kernel freeze... | 18:38 |
| tgardner | this is just a momentary freeze in order to get A2 out the door | 18:38 |
| psusi | ahhh | 18:38 |
| tgardner | so there time yet | 18:38 |
| tgardner | there is* | 18:38 |
| psusi | then hopefully it will get into .35 final... though Eric said there is another deadlock possible, but I have yet to hit it | 18:39 |
| psusi | he's working on tracking that down now | 18:39 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/ae0f36f0b964caf916c2ffc9f84b28c0f91c18a2-maverick/linux-image-2.6.35-6-server_2.6.35-6.7_amd64.deb | 18:40 |
| tgardner | ccheney, do a 'sudo dmesg -c' right before you start your test | 18:40 |
| kees | maks_, sconklin-gone, tgardner: building with intel_agp built-in did not fix it. the bisect has resulted, finally, in this one: f1befe71fa7a79ab733011b045639d8d809924ad | 18:41 |
| kees | I'm building a maverick kernel with that reverted now... | 18:41 |
| tgardner | kees, you have an older i945? | 18:41 |
| kees | yes | 18:41 |
| kees | G35, though, not G33 | 18:42 |
| tgardner | kees, you should ask Kyle about this. He did a bunch of i945 stuff a few years ago. | 18:42 |
| kees | tgardner: I figured I'd poke Anholt since I think he's in my timezone. (if I can find him) | 18:43 |
| * tgardner lunches for a bit | 18:43 | |
| GrueMaster | tgardner: Ok, I am officially giving up trying to get apport data on this omap bug. I have followed all the steps on https://help.ubuntu.com/community/ReportingBugs based on a headless system, but apport is now trying to collect data on my desktop system (different arch altogether). | 18:47 |
| maks_ | I'd like to know what the fix for #554569 is | 18:54 |
| maks_ | the linked rh and freedesktop bugs all point to different patches | 18:54 |
| maks_ | oh the freedesktop bug finaly points to something interesting cool. | 18:56 |
| * bjf will be back in a bit | 18:59 | |
| === bjf is now known as bjf[afk] | ||
| tgardner | ccheney, any update? | 19:03 |
| ccheney | tgardner, not yet, sorry was at lunch | 19:10 |
| ccheney | tgardner, no response from my email and still looking into how to do the test with my own image | 19:11 |
| tgardner | ccheney, well, just rerun the original UEC failure case with this latest kernel so I can get some output from that patch | 19:12 |
| ccheney | 2.6.35-6 ? | 19:12 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/ae0f36f0b964caf916c2ffc9f84b28c0f91c18a2-maverick/linux-image-2.6.35-6-server_2.6.35-6.7_amd64.deb | 19:13 |
| ccheney | oh nm i see you sent me a new one | 19:13 |
| ccheney | tgardner, appears broken, is that expected? | 19:19 |
| ccheney | i'll see how it does over all 10 tests but it already has one failure | 19:19 |
| tgardner | ccheney, yes, I expect it to be broken. I want the dmesg output | 19:20 |
| ccheney | oh ok | 19:20 |
| ccheney | [ 211.296957] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 should this happen? | 19:20 |
| ccheney | i'll get the whole log once its done | 19:20 |
| tgardner | ccheney, I only need the results of one pass. _Before_ you start it do 'sudo dmesg -c' to clear the kernel buffer | 19:21 |
| ccheney | oh oops, i need to redo it then | 19:21 |
| ccheney | ok as soon as it finishes the first pass i will get it to you | 19:22 |
| ccheney | all i saw was once for one pass: [ 365.268713] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 should this happen? | 19:23 |
| ccheney | that was the entire output of dmesg during the first pass of the run | 19:24 |
| tgardner | ccheney, yeah, its some debug, but I think its harmless | 19:24 |
| ccheney | oh that wasn't my question it was this "[ 365.268713] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 should this happen?" exactly | 19:24 |
| ccheney | :) | 19:24 |
| tgardner | ccheney, oh, yes. its the debug I put in. | 19:24 |
| ccheney | ok | 19:25 |
| ccheney | tgardner, so what does the special output mean? :) | 19:30 |
| tgardner | ccheney, hang on, I'm spinning one more | 19:30 |
| ccheney | ok | 19:30 |
| tgardner | ccheney, 10 minutes or so | 19:30 |
| ccheney | ok | 19:31 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/0a87a0c1b12f56bd556fd4506041966717c87fb1-maverick/linux-image-2.6.35-6-server_2.6.35-6.7_amd64.deb | 19:45 |
| tgardner | same drill as before | 19:45 |
| ccheney | ok will test | 19:46 |
| ccheney | tgardner, i may have to disappear for an extended time in the next week, if so contact kirkland for how to proceed, i don't want to go into details in public | 19:46 |
| ccheney | tgardner, most of the server team knows the details though if it does happen so they can fill you in | 19:47 |
| tgardner | ccheney, ack | 19:47 |
| ccheney | tgardner, will be a medical emergency if it happens | 19:47 |
| tgardner | ccheney, you haven't disappeared already, have you? | 20:01 |
| ccheney | no sorry | 20:04 |
| ccheney | had to talk my father in law to fill him in on my wife's status | 20:04 |
| tgardner | ccheney, I'm about EOD, but wanted to ponder the results of that last patch | 20:05 |
| ccheney | ok let me see if it finished running | 20:05 |
| ccheney | doh i ran off to tell him before i rebooted and started it | 20:05 |
| ccheney | it should take about 3-4m | 20:05 |
| ccheney | tgardner, [ 129.980324] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null) | 20:11 |
| ccheney | repeated 20 times | 20:11 |
| ccheney | most within a second, first was 50s before i'll pastebinit | 20:11 |
| ccheney | tgardner, http://pastebin.ubuntu.com/455139/ | 20:12 |
| tgardner | ccheney, ack | 20:12 |
| tgardner | ccheney, hmm, I wonder if I restore the original lines there if it'll make a diff. you gonna be around for another 20 minutes? | 20:13 |
| ccheney | yea | 20:14 |
| tgardner | k, gimme 15 minutes | 20:14 |
| ccheney | tgardner, i'll be around the rest of the day unless emergency happens | 20:14 |
| ccheney | i'll do my best to alert everyone to that case if time permits | 20:14 |
| jjohansen | -> Lunch | 20:18 |
| tgardner | ccheney, http://kernel.ubuntu.com/~rtg/mainline/afef312909fa10e603a05e030b2ee2feebde8d9f-maverick/linux-image-2.6.35-6-server_2.6.35-6.7_amd64.deb | 20:33 |
| ccheney | testing | 20:34 |
| tgardner | this should be the last one. | 20:34 |
| === bjf[afk] is now known as bjf | ||
| tgardner | ccheney, I left the debug in along with the restored code. It might just work. | 20:38 |
| ccheney | ok will see what happens | 20:38 |
| ccheney | tgardner, we got a bit of bad news, probably more to follow in the next week, but i'm still around today | 20:45 |
| tgardner | ccheney, I'm definitely outta here after this test. | 20:45 |
| ccheney | ok | 20:46 |
| ccheney | just one result from 1 loop | 20:46 |
| ccheney | [ 193.874005] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null) | 20:46 |
| tgardner | ccheney, did the PAD error show up? | 20:47 |
| ccheney | oh i forgot to check that, sorry :-\ | 20:47 |
| ccheney | no i think it is working | 20:47 |
| ccheney | i'll rerun | 20:47 |
| ccheney | i didn't clear my euca logs | 20:47 |
| ccheney | it looks like it worked though | 20:47 |
| ccheney | yea it worked, i checked the timestamp from my script vs the timestamp of the last error on the log | 20:48 |
| ccheney | haven't run an instance yet, but it seems to no longer show pad corrupted | 20:48 |
| ccheney | yea no more pad corrupted error | 20:48 |
| ccheney | what did you change? | 20:49 |
| tgardner | ccheney, cool. I'll send a note to upstream about it later tonight. Add your results to the LP report. | 20:49 |
| ccheney | ok | 20:49 |
| tgardner | ccheney, I restored the 2 lines of code that made any substantive difference in the behavior of the patch | 20:49 |
| ccheney | ok | 20:52 |
| tgardner | ccheney, upstream email sent. I'm outta here. | 21:15 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!