[12:17] <TeTeT> kim0: thanks for your support of my membership application :)
[12:17] <kim0> TeTeT: most welcome ;)
[12:21] <flaccid> membership eek
[12:21] <flaccid> debian images away!
[12:22] <flaccid> http://www.rightscale.com/library/multi_cloud_images/All?search%5Badvanced_search%5D=&search%5Bfilter_value%5D=debian&search%5Bfilter_type%5D=title&search%5Bprice%5D=&search%5Border%5D=date_desc&x=0&y=0
[15:57] <navanjr> TeTeT: me again! ... network and disk do not appear to be an issue with my performance.  I have been trying to find a way to run a GFlop test on my instance. no luck so far.
[15:59] <navanjr> i did notice something interesting in my cpuinfo...  model name	: QEMU Virtual CPU version 0.12.5
[15:59] <navanjr> yesterday you asked about QEMU
[16:23] <navanjr> what does it mean if I find this message in my libvirt logs... Could not initialize KVM, will disable KVM support
[16:24] <navanjr> My instances run, however slow.
[16:24] <TeTeT> hi navanjr , I think it means you're running with qemu instead of kvm. Are you sure that kvm is enabled in the BIOS?
[16:24] <navanjr> i thought i was sure...
[16:24] <kim0> navanjr: what's the output of kvm-ok on the NC
[16:25] <navanjr> nate@nc:/var/log/libvirt/qemu$ kvm-ok, INFO: Your CPU supports KVM extensions, INFO: /dev/kvm exists, KVM acceleration can be used
[16:25] <TeTeT> kim0: I had it happen once in an on site training that kvm-ok reported fine, but still qemu was used
[16:26] <kim0> :s
[16:30] <navanjr> the line right above... open /dev/kvm: Permission denied
[16:35] <TeTeT> navanjr: on the node, if you just start kvm on the command line, what's the output?
[16:36] <TeTeT> navanjr: if there's none, means you're good
[16:37] <navanjr> well, im not good. :)
[16:38] <navanjr> open /dev/kvm: Permission denied, Could not initialize KVM, will disable KVM support, pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin", Could not initialize SDL - exiting
[16:38] <navanjr> do i just need to chown of the /dev/kvm ?
[16:40] <navanjr> nate@nc:/dev$ ls -lh kvm,  crw-rw---- 1 root root 10, 232 2011-03-18 11:30 kvm
[16:41] <kim0> navanjr: it's root:kvm for me
[16:41] <kim0> on natty
[16:42] <navanjr> ok i changed it....
[16:42] <TeTeT> kim0, navanjr : root:kvm for me on Lucid too
[16:43] <navanjr> so when i "sudo kvm" on the node
[16:43] <navanjr> here is what i get...
[16:43] <navanjr> pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin"
[16:43] <navanjr> init kbd.
[16:43] <navanjr> Could not open SDL display
[16:44] <kim0> UEC shouldn't really use kvm's SDL interface
[16:44] <navanjr> any worries there?
[16:44] <TeTeT> navanjr: install kvm-pxe to let the romfile problem go away
[16:44] <TeTeT> navanjr: looks better than before I'd say
[16:44] <TeTeT> navanjr: maybe time to launch another instance and see if it's any faster
[16:44] <navanjr> cool, i really appreciate you help so far!
[16:45] <navanjr> ill report back
[16:50] <TeTeT> be back later
[17:07] <SpamapS> smoser: re the udev thing.. I notice now that udev has moved from 165 to 167 since you reported the Connection Refused bug. Are you still seeing problems in the automated tests?
[17:11] <navanjr> thanks guys!  my instances are now running in KVM.  I have some new issues now.  Now my console says there are errors in clould-init.  all my new test are without any user-data.
[17:12] <navanjr> last line on the console... landscape-client is not configured, please run landscape-config.
[17:12] <kim0> navanjr: r u using the default networking mode, managed-novlan
[17:12] <navanjr> managed-novlan
[17:12] <kim0> navanjr: that one is normal .. ignore it :)
[17:13] <navanjr> the instance is running... virsh list says so
[17:13] <navanjr> responds to ping
[17:13] <navanjr> but cant ssh
[17:13] <kim0> It's probably hung waiting for user-data
[17:13] <kim0> takes like 20 mins to time out !
[17:13] <navanjr> oh
[17:14] <navanjr> consuming user data failed!
[17:14] <navanjr> Traceback (most recent call last):
[17:14] <navanjr>   File "/usr/bin/cloud-init", line 90, in <module>
[17:14] <navanjr>     main()
[17:14] <kim0> I'm not really sure why the user-data service would have problems in uec
[17:17] <navanjr> here is my console output... (fwiw)  http://pastebin.com/QcJgwFgS
[17:26] <TeTeT> navanjr: got better performance now?
[17:27] <navanjr> sure Do!
[17:27] <navanjr> having some trouble with cloud-init
[17:27] <navanjr> http://pastebin.com/QcJgwFgS
[17:33] <TeTeT> navanjr: hmm, that's bad, can you try a more recent image, like the released maverick one and see if that works? If not it might mean that your front-end is not setup correct
[17:35] <navanjr> it was all working previously to the chmod
[17:36] <navanjr> and the apt-get install kvm-pxe
[17:36] <navanjr> on the NC
[17:36] <TeTeT> navanjr: maybe a temporary issue, launch another instance and check
[17:37] <TeTeT> navanjr: I believe that very old lucid images had a problem with timeouts and cloud-init, but it was fixed, so I asked for trying out maverick
[17:40] <navanjr> ok will do!  ill let you know. thanks, TeTeT
[18:00] <smoser> SpamapS, i believe we saw the bug in beta1 testing
[18:01] <SpamapS> smoser: hrm
[18:01] <smoser> http://jenkins.qa.ubuntu-uk.org/view/natty-ec2/job/natty_ec2_ap-southeast-1_all-zones/3/artifact/ap-southeast-1b/i386/m1.small/instance-store/i-c518d190/3b5d50de-fc22-4e17-88fd-2b3a74e8e63b-timeout.console.txt/*view*/
[18:01] <smoser> that is a beta1 log
[18:02] <SpamapS> smoser: its *really* a bitch to debug udev
[18:02] <smoser> now, beta1 was udev 166-0ubuntu6
[18:02] <smoser> SpamapS, thats why i'm glad i got you to do it
[18:03] <SpamapS> udevd[217]: bind failed: Address already in use
[18:03] <SpamapS> that, I think, is relaly the crux of the problem
[18:03] <SpamapS> does initramfs run a udev?
[18:04] <smoser> it might.
[18:04]  * SpamapS can answer that himself
[18:06] <smoser> SpamapS,
[18:06] <smoser> $ lsinitramfs /boot/initrd.img-$(uname -r) | grep udev
[18:07] <smoser> says the answer to your question is "yes"
[18:08] <smoser> from logs i can't be certain if those error messages are before or after transition to "the real init"
[18:10] <smoser> diff from 166-0ubuntu6 -> 167-0ubuntu1 is
[18:10] <smoser> 54 files changed, 509 insertions(+), 1003 deletions(-)
[18:13] <smoser> SpamapS, http://paste.ubuntu.com/590359/ is the "summary of changes"
[18:16] <SpamapS> interesting.. udevd writes to /dev/kmsg .. didn't know that but it makes sense
[18:16] <SpamapS> [    0.630017] udev[59]: starting version 167
[18:16] <SpamapS> that explains that
[18:18] <SpamapS> smoser: this one MIGHT be related:        udevd: initialize fds, for proper close() on exit
[18:18] <SpamapS> smoser: possible that the initramfs udevd wasn't closing its sockets properly leading to them being left open.. but I don't think AF_LOCAL sockets do that.
[18:19] <smoser> yeah, that entry stuck out a bit
[18:20] <SpamapS> smoser: do you guys run constant tests like these? 167 has been in for a few days now
[18:20]  * SpamapS hopes nobody picks up on the desperation in his tone :)
[18:20] <smoser> we're not to the point of running daily tests
[18:20] <smoser> we're getting there, thanks to jamespage
[18:22] <smoser> SpamapS, http://paste.ubuntu.com/590361/ is the message you were looking for
[18:22] <smoser> thats the upstream commit
[18:23] <SpamapS> *hm*
[18:24] <SpamapS> if thats the case then its very hard to trigger the race artificially.. I have to get udevd to start sooner
[18:29] <RoAkSoAx> smoser: been a long time since I last worked with something this complicated in bash, but finally remembered a few things and learned others. This is what I'm doing so far, but it is just testing as since I started working on it in the past couple hours. Does this look sane to you so far? http://pastebin.ubuntu.com/590364/
[18:34] <smoser> regions_array=(${regions#,})
[18:34] <smoser> regions has a leading , ?
[18:35] <RoAkSoAx> smoser: yes, kernels too and presumming that initrds too
[18:35] <smoser> ah, yeah it does.
[18:35] <smoser> you should fix that issue up aabove though.
[18:35] <RoAkSoAx> smoser: ok
[18:36] <smoser> just because it looks confusing
[18:36] <smoser> acutally, its probably better if you actually just create the array above
[18:37] <smoser> then you dont have to mess with splitting it back up
[18:37] <RoAkSoAx> smoser: ok cool
[18:38] <smoser> youll have to make a temporary output file
[18:38] <smoser> --output "${output}"
[18:38] <smoser> if you give them all the same they'll probably truncate each others
[18:38] <RoAkSoAx> ok cool
[18:38]  * RoAkSoAx takes notes
[18:39] <smoser> the one other thing, is that you're inside an arch loop there
[18:39] <smoser> so right now you'd be publishing all regions at the same time for i386
[18:39] <smoser> and then for amd64
[18:39] <RoAkSoAx> smoser:  yeah I'm looking into handling that differently
[18:39] <smoser> k
[18:39] <smoser> thanks , RoAkSoAx
[18:40] <RoAkSoAx> smoser: i was thinking on something like waiting for the PIDS so, remove the wait after the for, and put it right after the arch while
[18:41] <RoAkSoAx> and queue all the PIDs, and wait for the queue to be empty
[18:42] <smoser> eyah. you might be able to use something like gnu parallel for that.
[18:42] <smoser> ie, you could just collect all the things up that you wan to run and pass it off to something else.
[18:49] <RoAkSoAx> ok will take a look at that
[19:16] <SpamapS> smoser: I just got the "Connection refused" bug on a vm w/ udev 167
[19:17] <SpamapS> actually no!
[19:17] <SpamapS> it has 166
[19:17] <SpamapS> forgot to upgrade this one
[19:17] <SpamapS> damnit
[19:17] <smoser> well, good news for local recreate
[19:17] <smoser> you can launch kvm with -nographic
[19:17] <smoser> in a loop
[19:17] <SpamapS> yeah its happened once before
[19:17] <smoser> and just kill it after 30 seconds or whatever
[19:18] <SpamapS> well now this is interesting..
[19:18] <SpamapS> the udev from kmsg says version 167..
[19:18] <SpamapS> but dpkg shows 166
[19:19] <SpamapS> err.. udevd is 167 but dpkg -l is 166. Ok well now I've gone cross eyed
[21:18] <navanjr> whats the trick to publishing maverick image?
[21:19] <navanjr> nate@nateDesktop:~$ uec-publish-tarball $UEC_IMG.tar.gz $RELEASE-$TIMESTAMP
[21:19] <navanjr> Wed Apr  6 15:11:29 CDT 2011: [21:19] <navanjr> Warning: no ramdisk found, assuming '--ramdisk none'
[21:19] <navanjr> kernel : maverick-server-uec-amd64-vmlinuz-virtual
[21:19] <navanjr> ramdisk: none
[21:19] <navanjr> image  : maverick-server-uec-amd64.img
[21:19] <navanjr> Wed Apr  6 15:11:36 CDT 2011: [21:19] <navanjr> failed to register maverick-server-uec-amd64-vmlinuz-virtual.manifest.xml
[21:19] <navanjr> failed: euca-register --name maverick-server-uec-amd64-vmlinuz-virtual maverick-20110406151031/maverick-server-uec-amd64-vmlinuz-virtual.manifest.xml
[21:19] <navanjr> Image: Image registration failed because the manifest referenced is invalid or unavailable.failed to upload kernel