[00:00] Yes, I see that, though difficult at times. Like having a large, fast, buildserver which is useless in a armel farm [00:01] But, back to another qemu issue: [00:02] qemu crashes (consistenly on this machine) while compuing SSH2 keys [00:02] With lots of output or almost no output? [00:03] "Creating SSH2 RSA key; this may take some time ...qemu: uncaught target signal 11 (Segmentation fault) - core dumped [00:03] Segmentation fault" [00:03] dpkg: error processing openssh-server (--configure): [00:03] Get a stack trace (you can see what was being called from /var/lib/dpkg/info/openssh-server.postinst ) [00:04] I need to modify the rootstock (in order for it not delete the rootfs when failing) to inspect deeper [00:04] If the crash is in ssh-keygen, then it probably needs fixing, but I doubt this is the case. [00:04] More likely the crash is in the kernel, in which case it's qemu not handling something in the ideal manner. [00:05] FYI I see some chatter on google about uncaught target signal 11 against armel [00:05] That, or step through it more slowly, perhaps using qemu-debootstrap [00:05] so it's not restricted to ubuntu or rootstock [00:06] signal 11 just means "Segmentation Fault". [00:06] aha [00:06] If qemu was catching segfaults, we'd all be complaining about that because it would make it hard to debug things :) [00:07] So, quick process to reproduce: [00:07] 1) make a directory [00:07] 2) populate a cross-chroot in that directory with qemu-debootstrap [00:07] 3) chroot into the populated chroot [00:07] 4) apt-get install openssh-server [00:08] 5) After failure, inspect the postinst [00:08] 6) Install your favorite debugging tools, and find the crash. [00:10] you need apt-get in there as well, dont you? [00:10] Hrm. qemu-debootstrap lacks a manpage: it takes all the same arguments (with the same meanings) as debootstrap, so just use --arch=armel and otherwise follow the debootstrap model [00:10] apt-get where? [00:11] If you do 3) apt-get need to be inside the chroot to do 4) right? Or have I misunderstood? [00:11] baaah [00:11] forget it [00:11] Ah good. I was getting very confused :) [00:11] my bad, sorry [00:11] No worries. [00:15] Is there a way to create a rootfs where everything is downloaded, but not configured, and then later when the real target executes, let it do all the postinst steps? [00:16] Rootstock at least fires up qemu to run the installer (or post installer, not sure on its specific task) [00:18] Sure. You can separate --first-stage and --second-stage in debootstrap. [00:18] Take a look at the qemu-debootstrap code for a simple walkthrough of how one does this with qemu. [00:19] Replicating that with hardware ought be relatively easy. [00:19] Mind you, it's not necessarily safe to install lots of other stuff before running --second-stage, as the base system isn't configured yet. [00:20] I see debootstrap retrieves, extracts, unpacks, configures. It seems to be the unpacking a later is done emulated. Am I right? [00:20] *The unpacking stage and the preceeding steps are done emulated. (it's getting late) [00:21] I'll check qemu-debootstrap in all cases, thanks [00:21] I'm not finding the source you're quoting. [00:22] But debootstrap pulls the base system in a way that doesn't much care about dependencies, so it kinda handy to get a base. [00:22] It does all of retrieval, extraction, unpacking, and configuration. Based on the arguments, it can perform subsets of these actions. [00:23] unpacking is safe to do on a foreign system: it's just unrolling tarballs. configuration is *not* safe to do on a foreign system. [00:23] Oh? How come? [00:24] foreign being the target system which shall run ubuntu in the end? [00:24] Because configuration scripts can run arbitrary binaries to accompish their goals. Running these in a chroot with a foreign architecture will not work. [00:24] Mind you, if you do things like the qemu-static binfmt trick, it's not really that foreign anymore [00:25] (although qemu support isn't quite up to full hardware) [00:25] And you could have binfmt back to intel on the armel (if such exits) to bridge the oposite gap? [00:25] "foreign" meaning that the ISA for the code doesn't match the ISA for the machine on which the chroot is construted. [00:26] It does exist. it's supported by qemu-debootstrap. It's so incredibly buggy that nobody should ever use it. [00:27] the main issue is that qemu guest support for ia32 isn't well tested on armel hosts, and there don't seem to be any engineers wanting to try to fix discovered issues. [00:27] And foreign is defined by the machine running qemu-debootstrap [00:27] Sorta. [00:27] A chroot is "foreign" if the instruction set in the chroot is not supported by the host. [00:28] So an i386 chroot on an amd64 host is not foreign, but an amd64 chroot on an i386 host is foreign. [00:28] armel chroots are foreign on everything not armel [00:28] Since the configure step needs to be done "native" then you're basically tied to use all of debootstrap to make an image [00:29] (although it may be that in the future armel chroots will not be foreign for armhf hosts) [00:29] Well, you can reimplement debootstrap, but every image building tool I know starts with debootstrap. [00:30] I'm trying to figure out why rootstock is considered "not for production" while debootstrap seems to be [00:30] I'm not entirely comfortable with the passive voice in that sentence :) [00:31] I don't consider rootstock suitable for production for three reasons: [00:31] no offence, I'm truly curious. And since we're going to use ubuntu, I should know [00:31] 1) It uses qemu-static binfmt hacks, such that the resulting image a) has extra files and b) is subject to any bug in qemu [00:33] 2) It doesn't use the typical d-i based scripts for initial system configuration, which in practice may mean nothing more than a potential for bitrot, but may also be incomplete [00:34] Hrm. I remembered there being three, but the third one escapes me now [00:35] Most likely, it was solved (since a lot of the issues passing through my head have corresponding memories of someone telling me they fixed it) [00:35] I'll give you 3) openssh keygen & installation works fine on qemu-debootstrap, but fails on rootstock [00:36] I've tried twice now [00:37] OK, I've surely got a lot of very useful info here [00:37] Heh. Then 3) it's buggy :) [00:37] :D [00:37] I'm very grateful for the conversation! [00:37] Aha! rootstock still assumes versatile as an emulation host. [00:39] (and apparently hardcodes a lucid kernel) [00:39] Why should either affect qemu in respect of SSH keygen... [00:40] rootstock is running a full VM, while deboostrap is only running chroot/binfmt? [00:40] I believe that Ubuntu is currently shipping Linaro's QEMU, which I thought was no longer targeting versatile as primary. [00:41] Rootstock works in three stages [00:42] 1) debootstrap to populate a chroot [00:42] 2) debootstrap under qemu-static to configure the chroot [00:42] 3) fully-emulated environment to install random stuff [00:43] Mind you, I'm not that familiar with the code, but that's how it appears to me from a quick look. [00:43] sounds like I should move away from rootstock and integrate debootstrap instead [00:43] debootstrap itself doesn't do anything architecture related. [00:43] qemu-debootstrap (also not something I'd use for production) does the chroot/binfmt stuff. [00:44] * persia has fairly strong feelings about doing things natively, if this isn't already obvious [00:44] I understand that [00:45] What's the most powerful armel machine available? Because that is what you need to do such things natively, IMHO [00:46] Available? Possibly TI omap4 panda or nVidia Tegra. [00:46] All the hardware I've used it IO-bound. [00:46] Doing debootstrap on armel for armel is ok, right [00:46] Find something (anything) with non-USB SATA, or other reasonable IO path, and you'll win. [00:47] Even an OMAP3 should keep up. [00:47] For I/O, Marvell is the best, but not available to the general public afaik. [00:47] Native debootstrap is the basis for all official images. [00:47] Is this just for image builds or for package compiling? [00:47] I'm certainly happy with my Marvell IO performance, but it's not ARMv7a, so can't run native debootstrap for Ubuntu. [00:48] Image builds. [00:48] * GrueMaster too lazy/busy to read backscroll. [00:48] Ah. [00:48] Backscroll summary: rootstrap is buggy and was being used on a high-speed build server for production images. [00:48] An dual core ARMv7a with SATA is what I want I think [00:48] hahahah [00:49] sveinse, If you can find such a beast, then yes, that's precisely what you want. [00:49] Well, for cost/speed, definately omap3/4 based atm. Although the imx51 isn't bad in the Efika nettop. [00:49] rootstock? ouch. [00:49] Just be sure to check the internal wiring: some devices claim SATA but really just have a USB<->SATA bridge, and you could do that yourself. [00:49] persia, You work at reuters or something? That's the most compact summary I've ever read [00:49] sveinse, I haven't worked with reuters is well over a decade, but thanks :) [00:49] The efika is true sata. Freescale fixed it post babbage3. [00:50] GrueMaster, Really? Nice! [00:50] But only Cortex A8 (single core). [00:50] debootstrap is single-threaded anyway. [00:50] That's what I was told at UDS. [00:51] Yea, but having one core focus on one thing while the other core handles normal tasks does provide some improvement. [00:51] I heard the mx51 Efika's were currently at reduced prices, to clear inventory for the mx53s, so if single-core is enough, now is probably a good time. [00:51] $125 for the nettop. [00:51] Ah, you mean like handling the IO requests. Heh. Yeah. [00:51] Yep. That's about half what they were. [00:51] I thought I heard a rumor about a armel server blade (for energy efficiency). I have no idea from where, though [00:52] I've heard of that too. [00:52] sveinse, I've been hearing all sorts of rumors like that for the past couple years, often from folk who really ought to know. I have yet to see any retail products like that which support ARMv7a [00:53] (retail servers with ARMv5te are plentiful though) [00:53] https://www.genesi-usa.com/store/ [00:53] Oops. $129. My bad. [00:53] That's like USB OTG. Have you guys ever seen a product with USB UTG? Which /uses/ the OTG part? [00:54] Yes. [00:54] UTG? [00:54] sorry, typo [00:54] its 2am here soon [00:54] Ah. [00:54] Time for more coffee. :P [00:55] Point is OTG is well defined by the USB standards, but it's very hard to come by any product and/or cables that are wired correctly [00:55] You're not shopping in the right places :) Cables are plentiful in some shops. [00:55] It seems the industry and the standards dont line up on it [00:56] The main issue is really that it's hard to figure out what it ought do. [00:56] The product we're taking forward does include OTG. But all cables I've had in my hands are not correct [00:56] I have found a few OTG cables that do work. mini to usb host. I've had a keyboard plugged into my beagle that way and it worked fine. [00:57] In the limited context of using OTG on an Ubuntu system, the main issue is figuring out how to provide access to a useful filesystem when in gadget mode. [00:57] Mini USB seems to be better supported, but micro is not [00:57] Gadget mode is an entirely different thing. [00:57] There's some micro cables around, but yeah, they're lots harder to get. [00:57] I also found a micro to mini adapter. Just got it Saturday, but haven't teted it yet. [00:58] are you using the beagle xM? [00:59] I have one, yes. [00:59] sveinse, Do you *need* to be micro, or would mini work? [00:59] Too late in the design of the product. The tools for the enclosure are being built now [01:00] Oh well. Maybe find some online source of cables and point customers there. [01:00] But mini are deprecated by USB, that's what puzzles me [01:01] It is? That's annoying. I have *lots* of mini cables and ports around. [01:01] True. They are taller than micro, and the EU wanted a standard deployed across all cell phone markets. [01:01] That's why the sudden conversion to micro. [01:02] http://www.penguin.cz/~utx/hardware/USB_Mini/ [01:02] Gadget cables are plentiful. It is the Host cables that are hard to find. [01:03] Host cables being usbmicro to USB B (square) or USB A female receptacle? [01:03] Right. [01:04] * sveinse want 10k cables of usbmicro to USB A female receptacle [01:04] Would you like fries with that? :P [01:04] That quantity ought make it easier. [01:06] Hmmm. Just looked closely at my micro<>mini adapter. It is for plugging a micro cable into a mini device, not the other way. Useless. [01:06] sveinse: persia: rootstock uses qemu and debootstrap [01:06] but it copies the qemu-arm-static before doing the second phase of debootstrap [01:06] it can also run inside a full vm [01:07] and that's why it's using vexpress kernel [01:07] yup. I just hoped finding it off the shelf somewhere. Yet the 5 cable samples I've tested all wires the ID pin incorrectly, so it's not trivial [01:07] but if you run with root, it just uses user mode emulation [01:07] Which for some reason is crashing while computing ssh host keys [01:07] and the reason why rootstock is not suitable for production is that it's not the official tool to generate images [01:08] and not doing all the specific hacks used by live-rootfs to create the rootfs [01:08] rsalveti, Do you think it ought migrate to use the omap3 kernel? [01:08] it creates the user with it's own way, and set the images using it's on setup [01:09] I have two things when I start: A debian repo and a list of packages. They are to be installed into a image which shall be put into an sdcard in production. [01:10] And if qemu-bootstrap is also dodgy because of qemu, then the only solution is to let an armel machine build the image for production [01:10] persia: last time I checked omap3 support you needed to create a whole bin image [01:10] with x-loader/u-boot and stuff [01:11] the only real problem with versatile is the ram restriction [01:11] besides that it works better with qemu [01:11] Oh. I thought the kernel team dropped versatile because omap3 support had improved enough to make it more suitable with qemu. [01:12] persia: well, it's hard to push omap 3 specific patches upstream [01:12] because upstream wants to emulate it exactly as the hardware should behave [01:12] How far do we have to push? I thought Ubuntu was shipping a special linaro-patched qemu [01:12] and versatile is kind of a virtual machine for qemu, no one actually check if the kernel really works with real hardware [01:13] yes, but last time I checked it was still missing some bits, would be nice to check it again [01:13] rsalveti: My rootstock crashes when install openssh-server. It did not prior to it being upgraded from maverick to natty. Nor does my desktop machine (amd64) crash it. [01:13] Some people have real versatile hardware, but not many, and it'&s known that the real hardware *can't* run Ubuntu (not ARMv7a compliant) [01:13] sveinse: unfortunately I'd recommend you to run natively on an ARM board if you want a stable tool [01:13] rootstock supports running on arm currently [01:14] versatile express should be fine [01:15] don't know how well that's support in qemu [01:15] sveinse, What's your timeframe for getting the SD images complete? If it's in July sometime, we should have clear instructions available for you beforehand. If it's sooner, you may have some muddling to do. (And no, you don't need to tell me the answer, just apply your answer to my comments) [01:15] rsalveti: interestingly, this is the first time since the start of our project that it has crashed. It has been faithful since like nov -10. Daily builds, 10-20 builds per day [01:15] sveinse: this could be easy to fix, but then you'll end up having issues with mono [01:15] Sure, we're still in development, so I can wait [01:16] Hrm. My usb micro host cables were from Green House Japan, but they don't seem to be offering any now. [01:16] if you don't use mono, or not plan to use, then rootstock should work in most of the cases [01:16] It's not just mono: it's gotten buggier again because fewer folk are heavily using the qemu-static stuff. [01:16] no mono, just this little regression :) [01:17] sveinse: do you have any specific change at your rootstock? [01:17] And there's something else wonky: the openssh-server case works with qemu-debootstrap and not with rootstrap, which implies something rootstrap-specific as a bug. [01:17] sveinse: if not, can you paste me the logs? or cmd arguments you're using? [01:18] s/rootstrap/rootstock/ [01:18] sure, hold on (takes a while) [01:23] persia, does this mean you are working on docs for bootstrapping or a tool? (Since referring to july) [01:25] sveinse, Yes. Clear documentation of the Ubuntu image building infrastructure is something that's supposed to happen soon. [01:25] excellent [01:25] Presumably you could use this for your purposes. [01:25] Seems so. [01:25] Note that we're *not* attempting to generate documentation for full factory-automated installation at this point. [01:26] No, I wouldn't expect that, nor demand such a thing [01:26] I'd like it if we could provide that, but it's a hard problem still. [01:27] And there are many holes to fall into I've noticed. [01:27] I believe there's some work in progress to handle recovery partition creation, and presumably that could also be used to generate recovery media. [01:27] E.g. rootstock as discussed. Previously (well haven't resolved it yet), the missing --sysroot from ubuntu/linary armel gcc [01:28] But multicast distribution in a production environment is still several steps away (mostly because very few hobbyists have the infrastructure to test such things, and most people who build them like to sell their solutions) [01:29] Our strategy is to make an sd-image which is duplicated across all units. And then we run a firstboot to make the units unique based on information from other sources [01:30] I'd strongly recommend using oem-config as that "firstboot". If it doesn't work for you, let's fix that, rather than have lots of implementations. [01:30] Thanks, I'll look into it [01:31] oem-config basically wraps d-i components for personalisation of a preconfigured system. Does stuff like handle user creation, hostname assignment, timezone & locale information, etc. [01:31] nice [01:32] Since you're installing openssh-server, you may also be interested in looking at vm-builder: I'm not sure if all the work done there to ensure post-boot systems has migrated into oem-config, but you will want to have that set of hacks as well (stuff like regenerating SSH keys on first boot, etc.) [01:33] openssh-server is a development tool for now [01:33] Heh, OK. If you're doing something like a desktop, handheld, phone, etc. then oem-config should have all the right bits already included. [01:33] If you're doing a server, then I'm not sure it's as ready. [01:34] while I'm at it: Our guys are reporting stability problems between Qt (QWS) and Natty. Is this something which is familiar? [01:34] I haven't had time to dig into the issue, so I don't know first hand [01:35] I've had a few things crash on my Kubuntu netbook, but most stuff seems to work. [01:35] (that being natty/armel) [01:35] The key thing is to report the bugs: if you can make something crash, it ought be fixed. [01:36] Perhaps I should diff the qt4-x11 sources and the stock qt. Because we are on stock qt (due to support from Nokia) [01:37] Yes, I'll report any bugs I come across. Just wanted to hear if its known to be unstable [01:37] Works for me, but that's only one data point, and I don't claim to use that system that much. [01:38] but thanks, it does help [01:40] rsalveti: I have to sleep now, but I'll get hold of you if I'm able to reproduce the crash [01:40] Just checked: there's a heap of patches against Qt. You don't need to diff though: just `apt-get source qt4-x11` and look in debian/patches [01:40] Thanks [01:40] sveinse: sure [01:40] kubuntu_23_arm_memory_barriers.patch is one of the things you may require :) [01:40] persia, yes, there's lots [01:42] persia, thanks you for your insights and discussions. I've learned a lot. In fact I shall bookmark the irc logs from this evening [01:44] good night === prpplague^2 is now known as prpplague === beerlogger is now known as winelogger [05:58] i am not entirely sure what the omap4 package has done for me [06:24] so i repeated the process with 11.04 that i used for 10.10, and it is corrupt again. and barfing on package updates.i even used a new SD card === ericm-afk is now known as ericm|ubuntu [09:15] rsalveti, still awake? [09:39] * persia suspects it will be a couple hours yet === winelogger is now known as omglogger === omglogger is now known as transitlogger [12:52] sveinse: I'm away now :-) [12:52] ok, I'm not in a hurry === prpplague is now known as prpplague^2 [15:48] shit. my panda has rescue rootshell and getty running in same time on ttyO2 ;( [15:49] did you boot oem-config without splash enabled ? [15:49] no, old rootfs with new kernel/initrd [15:50] got "single" on the cmdline ? [15:50] panda login: General error mounting filesystems. [15:50] upstart usually only fires up the rescue shell if you tell it to [15:50] oh [15:50] fun is that / /home are properly mounted [15:50] yeah, you likely had an fsck on your display ;) [15:51] it will redirect that to the splash ... if you work remotely that indeed doesnt help [15:54] ok, upgrading my ubuntupandaa1 to oneiric [15:54] ogra_: Here are the logs and reports on the SD card sesize issues on 11.04 on PandaBoard http://groups.google.com/group/pandaboard/browse_thread/thread/76d19fab249a1ce9# [15:54] GrueMaster: ^^^ a follow up from our recent IRC discussion [15:55] Please feel free to study them and request more info on that thread... or log a trouble ticket if that is more appropraite [15:55] 466MB to download, 217MB extra used on disk. 955 packages to upgrade, 95 new to install, 3 to remove, 1 on hold [15:55] jayabharath, /var/log/jasper.log would be intresting ... and a proper bug report [15:56] Atleast a couple of folks included the jasper.log [15:56] do you want me to put a bug report into the system? [15:57] that would help, yep, and tell people to add their stuff there [15:57] ok [15:57] Seems like we are having a lot more problem reported with natty than with maverick [15:57] I have the same hw as magog96 (Transcend 16G Class 10 SDHC & Panda A1). Worked fine here. [15:58] ~hail apt-cacher-ng [15:58] it could also be that we now have a larger user base .. i.e. more testing and more possiblity of user errors ;) [15:58] Most of them appear to be A2/A3 pandas. [15:59] Which I don't have to test with (backorder). === Dr_Who is now known as tgall_out [15:59] GrueMaster: We can try to update your pandaboard with a newer version... should not be a issue. [16:00] have a nice rest of day [16:09] jayabharath: I have noticed sproadic issues with natty on my A1, but have not been able to reproduce them. During testing, I would see them once or twice a week when testing daily images. Most of the time, it was just a matter of oem-config restarting instead of completing, but there was never any signs of corruption. [16:09] And oem-config would complete on the second pass. [16:09] I wonder if it is something in x-loader/u-boot that ships with natty? [16:19] Okay, one thing I just figured out about the i.mx53 boards -- they overheat [16:19] fast [16:20] Those processors actually need a heatsink. [16:20] did you check for an intel logo on the SoC ? [16:21] heh [16:21] *snicker* [16:21] Intel processors don't overheat. They throttle back to almost off when they reach 75C. [16:22] They could have done a much better job on the thermals .. this thing needs a heatsink, and they should have installed one on every board [16:22] persia, NCommander, do you guys remember wheer the spice seed notes are (i know for sure both of you were in the room when we discussed them) ... infinity got that spec assigned but it seems to be empty [16:24] ogra_: they should be on the etherpad document [16:25] hey infinity [16:26] o/ [16:28] infinity, challenge your javascript interpreter a bit ... http://summit.ubuntu.com/uds-o/meeting/other-o-arm-seed-spices/ [16:28] WOOO infinity ! [16:29] hmm? [16:30] ogra_: That looks decidedly unfinished. Fun! I especially like "Talk to lamont + kamion to make sure design is workable" [16:30] lol [16:31] well, talk to persia, NCommander or me if you need explanation, we were in the BOF session and *might* remember one or the other bit [16:31] * infinity laughs. [16:31] Cause it was oh-so-long ago, right? :) [16:32] indeed ... and there were these beers at night [16:32] Basically we decided to implement it with metapackages [16:32] *hungarian* beers [16:32] persia was talking notes [16:32] Belgian beers too [16:32] some pretty strong ones, as I recall [16:32] pfft [16:33] belgian "beer" [16:33] infinity: It's been a whole _month_ ... [16:33] NCommander: Metapackages shouldn't be entirely necessary, but I'm happy to get back in on the discussion in a bit, if I'm expected to implement the results. ;) [16:33] * martyn starts the biervars with Oliver [16:33] infinity, just make it work properly ... [16:33] we dont care about how :P [16:34] Wait, what? We were in sessions between beer??? [16:34] GrueMaster, thats when you sat down ... remember ? [16:34] But I sat most of the time...oh, wait. [16:35] *chuckle* [19:14] * pmathews wonders if there is a channel for low power RF devices from TI [21:23] ogra_: Ubuntu uses x-loader-omap4 L24.9git20100901-0ubuntu5 or x-loader-panda? [21:24] hrw: x-loader-omap4-panda 1.5.0+git20110325+b6bbfe7-1ubuntu1 [21:26] At least that is what is currently in oneiric. [21:46] I have some problem using rootstock on ubuntu 11.04 to build an lucid image for gumstix board [21:47] I have some segfault for the locale and I also have an error about E: Sub-process /usr/bin/dpkg received a segmentation fault. [21:47] E: Second stage build in chroot failed ! [21:47] E: Please see the log to see what went wrong. [21:47] can someone can help me to resolv this ? [21:48] dumarjo: Why lucid? Why not just use our natty-headless image? [21:48] I can try [21:48] so I use -d natty ? [21:49] You are creating an image to run from SD, right? [21:49] yes [21:49] but i would to use qemu too [21:49] Just download the pre-installed image and flash to an SD card. [21:50] from the gumstix web site ? [21:50] http://cdimage.ubuntu.com/releases/11.04/release/ubuntu-11.04-preinstalled-headless-armel+omap.img.gz [21:50] This will run through a serial console. [21:51] GrueMaster, he is using a gumstix, might not work [21:51] I thought it was tested by linaro to work. [21:51] I don't have documentation on that [21:52] we should put that on the wiki if Linaro did indeed test it [21:53] is this image can be use with qemu ? [21:57] same error [21:57] I use this command line [21:57] sudo rootstock --fqdn WebNode --login gumstix --password gumstix --imagesize 2G --seed linux-image-omap,ubuntu-minimal --notarball -d natty [21:57] something wrong in taht ? [21:58] looks ok. [22:05] ok then. I wil have to tale a look close towmorrow... thanx for your helps [22:45] GrueMaster: then it should provide upgrade path - my natty panda had L24.9 one and dpkg errors out with it as it's version does not begin with digit [22:45] GrueMaster: natty -> oneiric upgrade catched that [22:48] ubuntu kernel for panda still had cyan fb ;(