[00:02] two line paste [00:02] PandaBoard Revision: 001 [00:03] oops [00:03] PandaBoard Revision: 001 [00:03] well darn [00:03] GrueMaster / rsalveti http://paste.ubuntu.com/484228/ [00:04] nice. [00:04] question though, do you want it more human readable, or more parse friendly [00:04] now to get other arch/platforms to follow suit. [00:05] i.e. should i just report revision without referencing pandaboard? [00:05] no, platform/revision is great info. [00:06] okie dokie [00:06] * prpplague gets ready to send patch to L24.9 team [00:06] This way we can differentiate between systems (i.e. Blaze/Panda or Beagle/BeagleXM/Gumstix, etc). [00:09] I'm assuming this info is gathered through probing the system & using a lookup table of some sort? [00:11] argh, my system is trashing... very slow [00:11] opening the link [00:11] prpplague: cool :-) [00:12] GrueMaster: 3-bit value on three gpios [00:13] ok [00:13] prpplague: if we have a separator like : I guess it's ok for parsers [00:13] name : version [00:13] rsalveti: thats what i was thinking [00:13] or something like that [00:14] rsalveti: i was thinking [00:14] rsalveti: something like this [00:14] boardname - revision : revisionnumber [00:15] PandaBoard - revision : 001 [00:15] is it that too much? [00:15] just leave it [00:15] PandaBoard : 001 [00:17] hm [00:17] don't know, most of the time just the revision should be ok [00:17] * GrueMaster finds nothing wrong with the pastebin version. [00:17] like beagleboard: c4 [00:17] pandaboard: es1 [00:17] an example [00:17] yea [00:17] ok, do the KISS thing [00:18] :-) [00:18] always [00:20] rsalveti: Back to the image issues, I am really having doubts as to what could be breaking the system. I created a manifest of the A3 image and compared it with 20100809 (first failing image). Upstart & ureadahead are unchanged. [00:20] hm... [00:21] maybe a dependency between the upstart services [00:21] waiting for something that doesn't get started [00:21] debugging upstart would be the way to go [00:21] testing new theory. We now have a new oem-config apparently. Will flash 20100809 image and update oem-config on it. [00:22] Two issues that I have seen; latest images fail oem-config and updating from A3 fails to boot. I doubt there is any relation to the two. === XorA is now known as XorA|gone [00:24] too many variables and not enough fast test systems. [00:25] now I can see 20100826.1 [00:25] will also download it [00:25] * GrueMaster has spent a frustrating 2.5 weeks on this issue. [00:25] :-( [00:25] sd card performance is frustrating [00:25] Yes. Very. [00:25] Even on my desktop. [00:27] GrueMaster: 152K/s of download is also frustrating [00:28] yes. Which is why I have my own mirror. [00:28] but this is for the image files [00:29] I have a cache server around [00:29] 1Gb network throughout the house also helps [00:29] for normal stuff [00:29] for sure [00:29] but an sd card with 3, 4m/s doesn't help [00:29] haha [00:29] I mirror both. And retain dailys until final. [00:30] Not that arm dailys take much room given the pool churn. [00:32] yep, will start mirroring the image stuff [00:32] the problem is that my stupid internet provider limits the amount of gb per month [00:32] I can download only 80gb I guess [00:33] with a 10m/s link [00:33] so I avoid doing mirror and downloading only what's needed [00:33] ouch. Some in the US are doing the same. Fortunately, my ISP has been around since early '90s and has a fat pipe. [00:33] that's nice [00:34] with my previous provider I didn't have this issue, but doesn't reach my current city [00:34] so I had to change to the stupid one [00:35] I'm currently limited by DSL speeds. Which out here in the sticks is ~6.5M/768k [00:35] * GrueMaster wishes he could get FIOS. [00:36] mine is currently 10/1 m/s [00:36] nice [00:42] rsalveti, What blocks getting newest rootstock upstream into Ubuntu? [00:43] persia: a new release? :-) [00:43] persia: just 2 simple bug fix, still need to get some work at the ui to create a new release [00:43] You don't happen to know any upstream rootstock folk that might feel like rolling a release, do you? [00:44] persia: it works fine with our version, but with upstream you'll always get the initrd and vmlinuz outside the rootfs [00:44] Ah, makes sense. Best to get the FFe filed *before* beta-release, as otherwise getting it into maverick will be sticky. [00:44] that's why I said GrueMaster to try it [00:44] easier to run mkimage later [00:44] persia: can you buy me some time? :P [00:45] Unfortunately, time and tuits are the two things of which I currently have shortages :( [00:45] I'd love to do it, but have more important issues do solve atm [00:45] yup :-( [00:52] rsalveti: Running now (got sidetracked by lack of lunch). [00:52] GrueMaster: with latest minimal image it boots... [00:52] let me start the upstart debugging session [01:00] Looks like rootstock finished ok. [01:00] :-) [01:01] oops. All three of my microSD cards are preoccupied atm. hrm. [01:05] Gahhh. Brb. Need to start dinner (roast). [01:25] Anyone good at picking the right place to drop casts for qreal != double issues? Seems like the outstanding bit for koffice. [01:28] argh, nothing works today [01:31] Ok, roast is now...roasting. [01:31] Which part of nothing is most annoying at the moment? [01:32] haha, like boards not booting? :-) [01:33] I get that all the time. [01:33] Oh yeah. that one. This is why we need large volumes of consumer-grade hardware (although I think I once heard a story about a chicken and an egg...) [01:33] I just file bugs and let the engineers fix it. Oh, wait... [01:34] ... [01:34] :P [02:00] that's weird, I can't see 20100826.1 anymore [02:00] wtf [02:00] some times it works, some times it doesn't [02:00] persia: any clue? [02:01] transparent proxies should be removed, and everyone should use IPv6 with endpoint verification. [02:01] argh [02:02] and the weird thing is that my current wget is still working! [02:02] but I can't browse this link [02:02] clean cache, different browsers [02:02] just can't [02:02] Once you get a hit that works, it will be fine. But stale/erroneous cache data needs to wait for timeout to pass. [02:02] * persia forgets how to trick those offhand [02:02] yep [02:08] GrueMaster: weird, it hangs completely [02:08] the kernel, serial, everything [02:10] rsalveti, Alternate thought: tell me which file you seek, and I can probably give you a deep URL (which may not be cached) [02:11] persia: now I'm able to download it! haha [02:11] persia: for the fs I'm still looking for it [02:11] * persia fails at parsing [02:13] nevermind :-) [02:13] * rsalveti needs a lot more coffee [02:13] Or a good night's sleep... [02:15] well, that'd help, for sure [02:19] rsalveti: you work so late and hard, man [02:19] rsalveti: do you have chance to test the 2.6.35 on your es2? [02:19] cooloney: well... nothing better to do haha :-) [02:19] cooloney: not yet, too much problems during the day [02:20] cooloney: do you have a binary? [02:20] I can easily test [02:22] rsalveti: i got an update patch from sebjan, will try to build again. [02:22] rsalveti: if i am ready, i will post the url for you, thx [02:22] cooloney: cool, just ping me when you're done [02:32] hm, installed network-manager and rebooted, now it hangs [02:33] Excellent! Uninstall network-manager (from a cross-chroot mounted on something else), and see if that unhangs. [02:34] that's what I'm doing now [02:34] but got many other packages installed as a consequence, but let's se [02:34] see [02:35] That's OK. We can dig through dpkg.log, and find the responsible party. [02:53] GrueMaster: persia: seems to be dbus, will test more [02:54] That's what ogra was finding [02:54] yep, will try to identify why [02:54] Sorry, had to take a break. Odd that dbus is giving you guys grief. I had updated that a couple of days ago and it has worked fine. [02:57] yep, it's dbus :-) [02:57] initct dbus start -> hangs [02:57] *initctl [02:57] or some dependency from upstart [02:58] Excellent! Can you get a strace? [02:58] checking again [02:58] will try [03:00] Is it dbus or network-manager? Like I said, I updated dbus a couple of days ago while slowly churning through the 400+ package updates. Didn't hang until network-manager update. [03:01] network-manager [03:01] But oddly it doesn't appear to start anything after mounting root. [03:01] removed the dbus dependency and it started fine [03:01] not networkmanager got the hang [03:01] It's obviously dbus if "initct dbus start -> hangs" [03:02] Might only happen in combination with other stuff, but strace should tell us. [03:02] nops, remove the dbus trigger from networkmanager [03:02] then started by hand [03:02] and it hang [03:02] stack trace (rather than syscall trace) would be better, but hard to get for a hang. [03:02] so it's probably nm [03:02] * persia suspects two bugs, rather than one [03:02] let me start nm with strace or logs [03:02] could be related with dbus also, on how it's using it [03:03] but would say it's closer with nm [03:03] let me check [03:03] 1 sec [03:03] Please strace the initctl hang first: that's lower level. it7s not worth masking it in nm if that makes it hard to find the dbus part. [03:04] one thing at a time [03:04] I want to first find the binary that's causing this [03:04] then we can trace that [03:05] Honestly, strace the symptom. [03:05] if you set strace to follow, you'll track over *many* binaries. [03:05] then dpkg -S will tell you the package. [03:05] Saves time doing it twive. [03:06] but for that I'd have to trace upstart [03:06] and not initctl start dbus [03:06] because when dbus is started, upstart tells nm to start [03:07] and that hangs [03:07] hrm. Having power & serial console on XM is not fun. [03:08] Right. strace -f -p ... [03:08] Then run initctl, and wait for the hang. [03:08] then check the strace to find out what hung it. [03:08] then dpkg -S to find the offending binary. [03:13] rsalveti: Did you need to use a null modem with your XM? [03:13] GrueMaster: nops [03:14] usb-serial only [03:14] same for panda [03:14] Ok. [03:15] Oops. Forgot xloader & uboot. That would explain things. [03:19] GrueMaster: :-) [03:21] I only appear to be getting garbage on serial [03:22] Same xloader & uboot as on the daily for beagle, right? [03:22] speed? [03:22] GrueMaster: yep [03:23] Serial works fine on the beagle. [03:23] Not on XM [03:24] * GrueMaster was worried that the exterior renovations done with the dremel may have fouled serial usb adapter. [03:28] * persia wishes koffice would build faster. [03:28] sigh. Time to fix dinner. Back later. [03:29] GrueMaster, You still have spare buildd cycles about? [03:29] Ah, not now :) [03:35] persia: GrueMaster: interesting, the hang happens when NM tries to activate the usb0 interface [03:36] if I give ifconfig usb0 up before initializing NM everything goes well [03:36] but if I try to load it directly, it'll hang when trying to activate the usb0 [03:43] What's the last syscall before it hangs? [03:57] a sendmsg to a socket [03:57] could be dbus [04:00] Which PID does the sendmsg? [04:01] And is the specific socket referenced earlier in the strace so you can determine if it's dbus? [04:06] cooloney: can you share the binary of your 2.6.35 kernel that works with your es1? [04:07] persia: sometimes just ifconfig usb0 up and ifconfig usb0 down gets the problem [04:07] the problem is that NM gives up to probe information and then put it down again [04:07] and put it up to configure [04:07] and this sequence is giving the issue [04:07] nm was updated, so this new sequence could be affecting ups [04:08] Right. [04:09] I'd suggest first concentrating on the ifconfig bit. Once that's sorted, let's see if the other bugs are still present. [04:09] sure, that's what I'm doing [04:10] It's clearly unsafe coding, but ifconfig should never cause a hang anyway, and that's the bit talking to the kernel, which is making the system hang (rather than just e.g. NM) [04:10] sure [04:10] that's why I want to test with latest kernel [04:10] from ti [04:10] Is there a way to force a kerneloops during a hang? I get lost when it gets to that level. [04:10] depends on the kind of hang [04:11] problem with code generally can show the trace and so on [04:11] but if the problem happens while communicating with the device, than things can be harder [04:12] Not surprising. I get confused as soon as I get below syscalls (and usually can only solve hangs where strace is looping, or waiting indefinitely for a response that will never come), rather than on the other side, but I imagine there's a few more barriers as one approaches control lines on hardware. [04:35] rsalveti: it is quite slow for me upload my package, damn it. [04:37] it seems a kernel bug at the ethernet driver [04:37] will take a better look at the down code [04:38] it hangs when giving the ioctl to put the interface down [04:46] rsalveti: http://people.canonical.com/~roc/kernel/omap4-2.6.35/ [04:47] rsalveti: finally uploaded. it is cross compiled [04:47] cooloney: nice, does it work fine with your es1? [04:49] it boots up on my es1 and works fine with my maverick rootfs. [04:49] cool [04:49] but not tested it on daily image [04:49] let me try it [04:49] ok, thx [04:49] np, I'm using a minimal one atm [05:19] cooloney: do you have your usb working? [05:20] doesn't work for me [05:20] and the ethernet needs usb [05:20] :-( [05:23] rsalveti: too bad, doesn't boot at all? [05:23] it boots, but doesn't work with usb [05:23] that that was the test I wanted to run [05:23] but thanks anyway [05:24] oh, thx, for that. could you post your dmesg somewhere? [05:24] i will discuss this with sebjan [05:25] cooloney: http://paste.ubuntu.com/484321/ [05:29] OMAP4430 ES1.0, is it ES2.0? [05:29] I assume it is ES2.0, rsalveti [05:39] cooloney: es1 [05:40] cooloney: I wanted to test on the board I know we have issues atm [05:40] :-) [05:40] persia: yes, I have the dove ready for building. [05:41] cooloney: I can test your deb on my es2 if you want [05:41] just a sec [05:46] cooloney: usb works with es2 [05:46] will reply your email [05:47] let me test it with 1gb [05:48] rsalveti: great, man. thx for the testing in your mid night, i think [05:49] sebjan: told me he fixed some EHCI or USB configuration on ES2.0, so maybe ES1.0 got some issue. [05:49] cooloney: hah, yeah [05:49] cooloney: oh, cool [05:49] they were also working on other things yesterday [05:49] mostly display and sound issues [05:49] let me put that on the email [05:55] cool, man, thx a lot [05:55] * cooloney needs some food for his lunch [05:57] cooloney: sent [06:12] GrueMaster, Excellent, although I found somewhere else to run that. I'll let you know if I think I have another one solved. [06:12] (but it might be too late by then) [06:47] persia: If you are looking at ftbfs issues, can you check into ubiquity? I looked at the lp page, and it indicates a version of ubiquity-frontend-gtk-2.3.8 was built 7 hours ago, but there are no armel packages [06:47] https://launchpad.net/ubuntu/+source/ubiquity/2.3.8/+build/1935852 [07:06] GrueMaster: ogra: lag: if you want usb with ES2 uses cooloney kernel, at http://people.canonical.com/~roc/kernel/omap4-2.6.35/linux-image-2.6.35-903-omap4_2.6.35-903.8_armel.deb [07:06] with mine (2.6.34) it doesn't work [07:07] ogra: lag: the hang while booting ("dbus" hang): bug 625108 [07:07] Launchpad bug 625108 in linux-ti-omap4 (Ubuntu) "System hangs when you run ifconfig usb0 up and ifconfig usb0 down (affects: 1) (heat: 8)" [Undecided,New] https://launchpad.net/bugs/625108 [07:07] I'm still looking at it, but I can confirm that it doesn't happen with ES2 and cooloney's kernel [07:08] and the driver didn't get any major changes between 2.6.34 and 2.6.35, so it could be related with the board [07:09] ogra: regarding gdm, just now I got a "working" system that I can check this, and the only thing I saw is that doesn't get started [07:09] Aug 27 04:13:46 acorn init: gdm goal changed from stop to start [07:09] Aug 27 04:13:46 acorn init: gdm state changed from waiting to starting [07:09] but that's all [07:09] please continue the debugging and I can help looking into that tomorrow [07:10] rsalveti: Don't you sleep? [07:10] GrueMaster: ogra: another interesting thing is that the hostname is now "localhost" but I can still get the builder's name at the messages [07:10] GrueMaster: I try, but doesn't work that well with me :-) [07:10] That gets changed by oem-config. [07:10] I know, but if we want to avoid having the builder's name logged, we should fix that [07:11] Too much Brazilian coffee. [07:11] I need some. :P [07:11] and messages I say the syslog messages of the first boot [07:11] GrueMaster: haha, possible [07:11] I can try to get you some at uds :-) [07:11] or even before, it all depends on ti :-) [07:11] Sorry, can't make it. Wife hexed me into a cruise. [07:11] TI maybe. [07:12] ouch, true [07:12] too bad [07:13] When I mentioned UDS and conflict in the same sentence, she started speaking in tongues and I swear her head spun a few times. [07:13] lol [07:14] Currently testing the new oem-config fixes on babbage then off to bed. Seams to work again. I have hope that we might have a working image again soon. [07:15] "Starting PC Card Services..." We really need to clean up the image next release. [07:15] can't make oem-config to run here [07:15] because of gdm it seems [07:16] Which version of oem-config? 2.3.8 is latest. === amitk is now known as amitk-afk [07:16] hm, 2.3.7 [07:16] Just finished. Now to see if GDM starts. === hrw|gone is now known as hrw [07:16] Oop, removing packages. [07:17] cool, good luck [07:17] need to go now [07:17] see ya in some hours [07:17] yea, understand. [07:18] get some sleep. [07:27] Yea, I now have a booting image again. 20100826.1 + oem-config updates are running on babbage. Very encouraging. Night all. [08:23] Morning All, If you any of you are interested in having a look at omap4 TRM it is now in http://focus.ti.com/general/docs/wtbu/wtbudocumentcenter.tsp?templateId=6123&navigationId=12667 === amitk-afk is now known as amitk [08:39] 07:04:39> ogra: lag: the hang while booting ("dbus" hang): bug 625108 [08:39] Launchpad bug 625108 in linux-ti-omap4 (Ubuntu) "System hangs when you run ifconfig usb0 up and ifconfig usb0 down (affects: 1) (heat: 8)" [Undecided,New] https://launchpad.net/bugs/625108 [08:39] Doh [08:39] ogra: lag: the hang while booting ("dbus" hang): bug625108 <- I don't have working Panda [08:39] ES1.0 [08:39] :( [08:49] ogra: Is there anything wrong with this line? [08:49] sudo mkimage -A arm -T script -C none -n "Ubuntu boot script" -d boot.script boot.scr [08:59] looks ok to me [09:06] Weird [09:06] It's churning out boot.scr's with no header? [09:08] lag, why sudo ? [09:08] Does it make a difference? [09:08] no ideas [09:08] I'll try [09:08] never used sudo with mkimage [09:09] Didn't make a difference [09:09] Still no header [09:10] Ah [09:10] No [09:10] It's just that cat doesn't print it [09:10] Emacs displays it fine [09:13] and cat or less dont ? [09:15] cat doesn't [09:15] less does [09:16] what nonprintables do you have in it? [09:16] Maybe `TERM=dumb cat ...`? [09:16] '^E^YV΄^E^B^F^@Ubuntu boot script^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^A^M^@^@^@^@ [09:16] :) [09:16] It's okay [09:16] I can use less/emacs [09:16] Yeah, cat should show that with TERM=dumb [09:16] It's not a problem [09:16] So long as I know [09:18] ogra: When do you think the oem-* bug will be fixed? [09:19] lag, i just kicked off a build [09:19] it shoudl be fine in the next image [09:19] though the usb0 bug will hit us hard [09:19] Which usb0 bug? [09:19] do you read the channel backlog if you get up ? [09:20] That ifconfig up usb0; ifconfig down usb0; ifconfig up usb0 hangs the machine. [09:20] its a useful practice ;) [09:20] ogra: No [09:20] bug 625108 [09:20] Launchpad bug 625108 in linux-ti-omap4 (Ubuntu) "System hangs when you run ifconfig usb0 up and ifconfig usb0 down (affects: 1) (heat: 8)" [Undecided,New] https://launchpad.net/bugs/625108 [09:20] Well, I flick though [09:20] Usually it's dribble [09:20] (which you mentioned to us earlier, so you ought have seen it) [09:20] d [09:21] Does this happen with TI's new patches? [09:21] Hard to say. The reporter went to bed. [09:21] I'll give it a go [09:21] Try it with TI's new patches and find out :) [09:22] This is Panda ES1.0 [09:22] I was under the impression that all support for ES1.0 is to be halted? [09:22] Bug report is for ES1.0. I'm not sure it's been tested with ES2.0 [09:23] Perhaps, but let's make sure the bug is ES1.0 only before we ignore it :) [09:23] hmm, theer is no bug about the dbus hang ? [09:23] lag, tricky for beta we dont have es2 yet [09:24] The dbus hang is dbus running NM init once dbus<->NM comes up, which is NM hanging on configuring usb0, which is 625108 [09:24] lag, so i guess beta has to stay es1 ... also we dont have bootloader or kernel for es2 [09:24] rsalveti spent hours hunting it down specifically. [09:24] persia, hmm, i thought i saw initctrl mentioned above [09:24] yes, i see that [09:24] Right. initctl start dbus hangs the machine, by the process I just described. [09:24] ok, then its fine with me ... just not sure what to do ... [09:25] kernel bug. lag's on it. [09:25] apart from disabling usb0 :) [09:25] Let's give lag a couple hours to hunt before we use the big hammer :) [09:25] persia, lag doesnt have working usb, no ? [09:25] (on the es1) [09:26] Ugh. [09:26] * persia dreams of a day when mass commercial HW is available, and all these niggles go away [09:26] natty FTW :) [09:27] lag, so i guess beta has to stay es1 ... also we dont have bootloader or kernel for es2 - [09:27] lag, show me the package on ports.ubuntu.com please [09:27] *packages [09:33]