| flaccid | smoser: http://dpaste.org/Se9H/ only a couple of things - Couldn't get a file descriptor referring to the console; error: unexpectedly disconnected from boot status daemon; FATAL: Could not load /lib/modules/2.6.32-305-ec2/modules.dep: No such file or directory (need to work out best way to install these on squeeze/debian); | 00:38 |
|---|---|---|
| flaccid | and Waiting for /dev to be fully populated...udevd-work[258]: kernel-provided name 'evtchn' and NAME= 'xen/evtchn' disagree, please use SYMLINK+= or change the kernel to provide the proper name | 00:38 |
| flaccid | and Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key | 00:38 |
| flaccid | most of the above is kernel related. smoser i just need to address/fix the above and all should be good. if you could also advise what the latest actual kernel is that would be great | 00:39 |
| smoser | "actual kernel" | 00:39 |
| smoser | all the ubuntu published items conform to https://wiki.ubuntu.com/UEC/Images/NamingConvention | 00:40 |
| flaccid | yes | 00:40 |
| smoser | (or close to it) | 00:40 |
| smoser | for released kernels, they'll have ubuntu-images/<...> in their name (per describe-images) | 00:40 |
| smoser | er.. for kernels it will be | 00:40 |
| flaccid | yeah so what is the latest kernel id ? | 00:40 |
| smoser | well, for us-east-1, and lucid (released) that would be: | 00:41 |
| smoser | 099720109477/ubuntu-kernels/ubuntu-lucid-amd64-linux-image-2.6.32-305-ec2-v-2.6.32-305.9-kernel | 00:41 |
| smoser | aki-0b4aa462 | 00:41 |
| flaccid | is there someone that lists these ? | 00:42 |
| smoser | i have considered putting kernels in the data at http://uec-images.ubuntu.com/query/ | 00:42 |
| smoser | but largely, we don't consider kernels aki to be a product as much as a side effect of the image | 00:42 |
| smoser | that said, http://uec-images.ubuntu.com/query/lucid/server/released.current.txt | 00:43 |
| smoser | will have a list of the latest ami | 00:43 |
| smoser | and their assicoated aki | 00:43 |
| flaccid | thanks that helps a lot | 00:43 |
| smoser | there is 'daily.current.txt' there also | 00:44 |
| flaccid | now back to the errors if you don't mind helping? | 00:44 |
| smoser | and maverick ... | 00:44 |
| smoser | so lots to chose from | 00:44 |
| smoser | :) | 00:44 |
| smoser | i'vve seen the "Couldn't get a file descriptor" error before, but dont really recall what that was about. | 00:45 |
| smoser | fwiw, jjohansen is the kernel team member for this ec2 | 00:45 |
| smoser | i'm looking at some logs | 00:49 |
| smoser | we see the "Could'nt get a file descriptor" on karmic currently | 00:49 |
| flaccid | cool thanks | 00:49 |
| smoser | and on lucid prior to alpha2 | 00:49 |
| smoser | err.. prior to alpha3 | 00:49 |
| flaccid | and would also like your advise on installing/enabling the kernel mods from the ubuntu package in debian and if this is an ok practice etc. | 00:50 |
| smoser | where did you get the ramdisk ? | 00:50 |
| smoser | or am i mistaken: | 00:50 |
| smoser | Begin: Loading essential drivers... ... | 00:50 |
| smoser | Done. | 00:50 |
| smoser | Begin: Running /scripts/init-premount ... | 00:50 |
| flaccid | its ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100114 1 | 00:50 |
| flaccid | ari-3f2d7c7a | 00:50 |
| smoser | yeah, so we dont use ramdisk anymore | 00:50 |
| flaccid | right so set to default | 00:51 |
| smoser | set to default ? | 00:51 |
| flaccid | i'm assuming default=none now | 00:51 |
| flaccid | i'll relaunch with that | 00:52 |
| smoser | yeah, it probably wont hurt or affect it. | 00:52 |
| smoser | that has been my experience. the ramdisk doesn't hurt anything. just doesn't help. | 00:52 |
| smoser | regarding installing ubuntu packages, are you asking from licensing perspective ? | 00:53 |
| smoser | those modules come from the packages... our ec2 kernels are normal kernle builds and have normal packages. | 00:53 |
| smoser | https://launchpad.net/ubuntu/+source/linux-ec2 | 00:54 |
| smoser | i dont know how well those would install onto a debian | 00:54 |
| flaccid | more from an operational point of view. this is to address FATAL: Could not load /lib/modules/2.6.32-305-ec2/modules.dep: No such file or directory | 00:54 |
| flaccid | so yes the question is regarding installing them on debian | 00:54 |
| flaccid | see because the amazon kernels are not working with debian, i am having to use the ubuntu kernels | 00:55 |
| smoser | yeah. so i dont know whether or not the packages there will "just work" (ie install) on debian or not. | 00:56 |
| smoser | but almost certainly a dpkg -x linux-image-*.deb would get you the /lib/modules/ tha tyou need | 00:56 |
| flaccid | well i would either have to use the lucid repos briefly and install via apt or manually install the package and its deps via dpkg | 00:56 |
| flaccid | well the linux-image-* pkgs have deps.. | 00:57 |
| flaccid | ok well no ramdisk fixed that | 00:59 |
| flaccid | zcat: invalid magic - another small one to address | 00:59 |
| flaccid | i mean no ramdisk fixed the console descriptor errors | 00:59 |
| flaccid | well it still defaulted to ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0 | 01:10 |
| flaccid | that must be tied to the kernel | 01:10 |
| flaccid | wb | 01:13 |
| flaccid | smoser: if i do a reboot and i have installed these modules, this should remove the FATAL: Could not load /lib/modules/2.6.32-305-ec2/modules.dep: No such file or directory message right? | 01:43 |
| smoser | well maybe | 01:44 |
| smoser | you may be getting that from the ramdisk | 01:44 |
| smoser | at least i'm fairly certain that ramdisks do 'modprobe' which will result in a message like that. | 01:44 |
| flaccid | but if the file is there, it should not return No such file or directory right? | 01:45 |
| smoser | are you using a ramdisk ? | 01:45 |
| flaccid | this kernel appears to default to ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0 | 01:45 |
| smoser | if you are using a ramdisk,and it was built with a differnet kernel then it won't have that /lib/modules/<kernel-ver> and it will try to modprobe | 01:46 |
| flaccid | aki-296f3e6c | 01:46 |
| smoser | kernels dont default to a ramdisk | 01:46 |
| smoser | this is something that sucks. | 01:46 |
| flaccid | oh but an image defaults to a ramdisk? | 01:47 |
| smoser | if you registered an image with a ramdisk and kernel, then you are not able (that I know of) to boot without a ramdisk or kernel | 01:47 |
| smoser | and, actually, it goes back step, since you can give ramdisk and kernel to the bundle command | 01:47 |
| flaccid | ok so ramdisk is bogus, no need to build image with a ramdisk value? | 01:47 |
| smoser | if you've done that, then there is no way to specify "None' | 01:47 |
| smoser | right | 01:47 |
| smoser | then, you can (if you wanted) specify --ramdisk <ari-abcdefg> | 01:48 |
| smoser | but there is no --ramdisk <none> | 01:48 |
| flaccid | sweet then i'll omit ramdisk | 01:48 |
| smoser | right. thats a superset of the functionality, but then, if you ever wanted a ramdisk, the user has to specify one, which just isn't very friendly | 01:48 |
| flaccid | if ec2-bundle-vol rcvs --kernel --ramdisk --generate-fstab will it simply ignore the no param values | 01:49 |
| smoser | just do not pass --ramdisk | 01:49 |
| smoser | or --kernel | 01:49 |
| smoser | on bundle or on register | 01:49 |
| smoser | fwiw, there are scripts 'uec-publish-image' that you could use that, in my opinion are more friendly than ec2-bundle, ec2-upload, ec2-register | 01:50 |
| flaccid | well i still want a default kernel | 01:50 |
| smoser | http://manpages.ubuntu.com/manpages/lucid/man1/uec-publish-image.1.html | 01:51 |
| smoser | right. | 01:51 |
| smoser | i *think* (you'd have to test this) | 01:51 |
| smoser | that you can bundle without anything | 01:51 |
| smoser | and then add --kernel when you register the image | 01:51 |
| smoser | which, in this case, would allow you to avoid re-bundling | 01:51 |
| flaccid | thats cool i'll test that | 01:52 |
| flaccid | hmm it doesn't specify params http://docs.amazonwebservices.com/AmazonEC2/dg/2007-01-03/CLTRG-register.html | 01:54 |
| smoser | see the date in that url | 02:23 |
| smoser | http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-RegisterImage.html | 02:24 |
| smoser | is what you want | 02:24 |
| flaccid | oh yeah thanks, eek on google! | 02:26 |
| flaccid | thanks muchly :) | 02:27 |
| flaccid | standby, i'll have some results soon | 02:27 |
| flaccid | 1 is building now | 02:27 |
| flaccid | the good thing i've noticed so far is that these AMIs i'm creating work fine with the lucid kernel and boot up really fast | 02:30 |
| flaccid | hmm that doc doesn't mention -K and -C options yet i am using them currently.. | 02:35 |
| flaccid | Required option '-K, --private-key KEY' missing (-h for usage) | 03:01 |
| flaccid | heh its not in their doc | 03:01 |
| flaccid | ah must be in http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-RegisterImage.html | 03:51 |
| flaccid | err http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?CLTRG-common-args-api.html | 03:51 |
| flaccid | Unable to read instance meta-data for ramdisk-id | 04:11 |
| flaccid | Unable to read instance meta-data for product-codes | 04:11 |
| flaccid | ^^ from ec2-bundle-vol. what should i do about this ? | 04:12 |
| uvirtbot | flaccid: Error: "^" is not a valid command. | 04:12 |
| flaccid | smoser: built and registered with no ramdisk, a launch with default assigns ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0 | 04:41 |
| flaccid | the default kernel being assigned is ubuntu-kernels-testing/ubuntu-lucid-i386-linux-image-2.6.32-306-ec2-v-2.6.32-306.11-kernel | 04:41 |
| smoser | those errors don't surprise me. | 04:42 |
| flaccid | i guess we'll see in console output if it is wanting 306 or 301 modules that will determine if the ramdisk is asking for it or not | 04:42 |
| flaccid | yes but i want to get rid of those errors :) | 04:42 |
| smoser | no, the "unable to read instance metadat" errors from bundle-vol | 04:42 |
| smoser | they dont surprise me | 04:43 |
| flaccid | yeah i want to fix those.. | 04:43 |
| smoser | bundle-vol is a yucky hack. you shouldn't rely on it. | 04:43 |
| flaccid | well thats what aws publishes.. | 04:43 |
| smoser | bundle-vol basically has to "un-boot" a system. | 04:43 |
| smoser | which is just simply going to require a bunch of distribution/OS specific HACKs | 04:44 |
| smoser | its much cleaner to use debootstrap and bundle-img | 04:45 |
| flaccid | well FATAL: Could not load /lib/modules/2.6.32-306-ec2/modules.dep: No such file or directory. so its the kernel that wants it. i'll extra the mods in this build | 04:45 |
| flaccid | where is bundle-img ? | 04:45 |
| flaccid | i am using debootstrap | 04:45 |
| smoser | ec2-ami-tools | 04:46 |
| flaccid | oh you mean ec2-bundle-img | 04:47 |
| smoser | yeah | 04:47 |
| smoser | can you give me output of describe-istances for the image you booted ? | 04:47 |
| smoser | i think you booted with a ramdisk. | 04:47 |
| smoser | or, you dont 'have those modules in the root filesystem | 04:47 |
| smoser | ec2-describe-instances i-abcdefg | 04:47 |
| flaccid | for some reason that returns Client.InvalidInstanceID.NotFound: The instance ID 'i-ba897dfe' does not exist | 04:49 |
| flaccid | yet its running | 04:49 |
| flaccid | i didn't include the modules in that build however | 04:50 |
| flaccid | rightscale dashboard is indicating use of ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0 | 04:50 |
| smoser | it maybe that rightscale is buggy | 04:51 |
| smoser | if you did not bundle with '--ramdisk' | 04:51 |
| smoser | and did not register with '--ramdisk' | 04:51 |
| smoser | then you should not have a kernel | 04:52 |
| smoser | err... you should not have a ramdisk | 04:52 |
| smoser | it is possible that rightscale is being nice, and trying to use that one, in which case, that really sucks. | 04:52 |
| smoser | but describe-instances certinaly should be able to give you output | 04:52 |
| smoser | what about describe-images of the ami | 04:52 |
| flaccid | that returns does not exist as well. no idea why | 04:54 |
| flaccid | hmm are these region specific? | 04:54 |
| flaccid | i think its only returning us-east when i need us-west | 04:55 |
| smoser | yes. all of it is region specific | 05:00 |
| smoser | each region is its own "cloud". the only data they share is your account id | 05:00 |
| flaccid | nps | 05:00 |
| flaccid | this is the image | 05:02 |
| flaccid | IMAGEami-b5b8e9f0041819229125/RightImage-Debian-squeeze-20100618033019-i386-v5.4.6041819229125availableprivatei386machineaki-3197c674instance-store | 05:02 |
| flaccid | this is the instance | 05:03 |
| flaccid | INSTANCEi-ba897dfeami-e7b6e7a2ec2-204-236-179-7.us-west-1.compute.amazonaws.comip-10-162-143-68.us-west-1.compute.internalrunningCJF: AWS US-West (testing)0m1.small2010-06-18T03:39:43+0000us-west-1baki-c18edf84ari-8f2978camonitoring-disabled204.236.179.710.162.143.68instance-store | 05:03 |
| flaccid | so i guess either rightscale is allocating ari-8f2978ca or ec2 (likely rightscale) | 05:04 |
| flaccid | can you help confirm that there is no such thing as a 'default' ramdisk when registering or launching? | 05:06 |
| flaccid | we seem to be using 'default' with a reserved ari then.. | 05:07 |
| smoser | yep | 05:07 |
| smoser | its rightscale is being "nice" | 05:08 |
| smoser | well, when you register or bundle and pass '--ramdisk' that gives you a "default ramdisk" | 05:08 |
| smoser | and if you do that, you can't ever run an instance without a ramdisk. | 05:08 |
| smoser | but, the describe-images info above shows that the image has no ramdisk associated with it. | 05:09 |
| smoser | which means either | 05:09 |
| smoser | a.) the user must provide one in run-instances | 05:09 |
| smoser | b.) there will not be one used | 05:09 |
| flaccid | that would of came up in the describe-images but right? i took this out of the build process | 05:09 |
| flaccid | yep we are on same level | 05:09 |
| smoser | right. it would be shown in describe-images output | 05:09 |
| flaccid | i will raise a bug for the rightscale dashboard issue | 05:09 |
| smoser | look at any other one, you'll see an ari | 05:09 |
| smoser | fwiw, that 'smoser0' is a debug ramdisk :) | 05:10 |
| smoser | (with some interesting characteristics, it can process user data) | 05:10 |
| flaccid | hehe, looks like we chose your ramdisk image for the default... :) | 05:10 |
| smoser | our choice to not have ramdisks has thrown errors much furrther than i thought it would have | 05:15 |
| flaccid | there is issues with no ramdisk? | 05:16 |
| flaccid | is there any way to remove a debootstrap target dir? i.e. you can't remove /proc and /dev | 05:19 |
| smoser | you have to unmount proc and dev first | 05:36 |
| smoser | then remove | 05:36 |
| smoser | there are no issues with no ramdisk other than tools that expected a ramdisk | 05:36 |
| smoser | UEC had fits (Eucalyptus) | 05:37 |
| smoser | righscale apparently had fits | 05:37 |
| smoser | the expectation of an ari was just hard coded many places | 05:37 |
| smoser | oh yeah, other places. | 05:37 |
| smoser | landscape | 05:37 |
| smoser | and the canonical "image store" (part of UEC) | 05:37 |
| flaccid | i see | 05:38 |
| flaccid | ok now launching with a new image | 05:39 |
| flaccid | smoser: extracting from the .deb for those modules doesn't give the desired files | 06:14 |
| smoser | no? | 06:14 |
| smoser | you're right.sorry. | 06:15 |
| flaccid | i'll cp them across from the worker which is lucid | 06:15 |
| smoser | just the wrong package | 06:16 |
| smoser | the linux-ec2 is a metapackage | 06:16 |
| smoser | that depends on the right linux-image-2.6.32-305-ec2 | 06:16 |
| flaccid | i'm extracting linux-image-2.6.32-305-ec2_2.6.32-305.9_i386.deb | 06:17 |
| flaccid | i tell ya what but it boots so fast | 06:17 |
| smoser | http://packages.ubuntu.com/hu/lucid/linux-image-2.6.32-305-ec2 | 06:17 |
| smoser | yeah, thats what you want | 06:17 |
| flaccid | hopefully cp -R /lib/modules/2.6.32-305-ec2 "$DEBOOTSTRAP_TARGET_DIR"/lib/modules/ is good enough.. | 06:22 |
| smoser | i'd think so | 06:28 |
| flaccid | i'm getting 1.2min startups | 06:29 |
| flaccid | 14.5mins to create and build an AMI | 06:30 |
| flaccid | smoser: only a few things left to address now, http://dpaste.org/5fAp/ | 07:00 |
| smoser | start up is all over the map | 07:01 |
| smoser | what do you think is left ? | 07:01 |
| smoser | it looks pretty good | 07:01 |
| smoser | what do you mjean by "1.2min startups" ? | 07:03 |
| smoser | total time to login/ssh from what | 07:03 |
| smoser | flaccid, | 07:04 |
| flaccid | i think thats the boot time, not sure how its configured | 07:04 |
| flaccid | need to address the ' WARNING: Currently emulating unsupported memory accesses' | 07:05 |
| flaccid | there is no /lib/tls on that instance | 07:05 |
| flaccid | also Waiting for /dev to be fully populated...udevd-work[86]: kernel-provided name 'evtchn' and NAME= 'xen/evtchn' disagree, please use SYMLINK+= or change the kernel to provide the proper name | 07:05 |
| flaccid | and the few under cannot unset powersave mode | 07:06 |
| flaccid | and Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key | 07:06 |
| flaccid | and maybe /build/buildd/linux-ec2-2.6.32/drivers/rtc/hctosys.c: unable to open rtc device (rtc0) | 07:06 |
| flaccid | also Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key | 07:06 |
| flaccid | i'll build a lenny now | 07:13 |
| flaccid | lenny works but not with rightlink | 07:37 |
| smoser | i'm going to bed. good luck. | 07:39 |
| flaccid | thanks; cya later | 07:39 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!