[00:55] <buuo> Hey any one can help a cloud noob with some questions? :P
[00:57] <flaccid> no
[00:57] <flaccid> how long is a piece of string?
[01:57] <zoopster> that wasn't nice flaccid
[01:57] <zoopster> buuo: ask your question
[01:58] <flaccid> :)
[15:29] <mathiaz> smoser: hi!
[15:29] <mathiaz> smoser: where are the new releases of Ubuntu EC2 images usually announced?
[15:29] <mathiaz> smoser: ie which blog/mailing list should I follow?
[15:29] <smoser> ubuntu-cloud and ubunt-server i think
[15:30] <mathiaz> smoser: great - thanks!
[15:30] <smoser> but programattically, you should just check at http://uec-images.ubuntu.com/query
[15:31] <mathiaz> smoser: is ubuntu-ec2@lists.ubuntu.com still active?
[15:32] <smoser> no. folded into -cloud
[15:32] <mathiaz> smoser: hm - nope
[15:36] <mathiaz> smoser: can the lucid images from http://uec-images.ubuntu.com/releases/10.04/release/ be used with kvm/libvirt?
[15:36] <smoser> not terribly easily.
[15:37] <smoser> https://code.launchpad.net/~smoser/+junk/boothooks is what i used in that timeframe to do local testing.
[15:39] <asac> smoser: so a script in user-data ... where does that log to?
[15:40] <smoser> asac, console
[15:40] <smoser> ie /dev/console, in ec2/uec that should be seeable in euca-get-console-output
[15:41] <asac> smoser: nice... thx. will try!!
[15:41] <smoser> asac, you can also very easily write it wherever you want
[15:41] <smoser> if its shell, just do
[15:41] <smoser> {
[15:41] <smoser>   my script here
[15:42] <smoser> } 2>&1 | tee some-other-file.txt
[15:42] <asac> good idea ... now i would have to figure how to make wiki show my #!/bin/sh if i place it at beginning of line ;)
[15:50] <asac> smoser: https://wiki.linaro.org/Platform/Android/Specs/LinaroAndroidBuildInfrastructure#Experimental%20Node%20Scripts ... is that about the right approach to do something like this?
[15:52] <smoser> asac, yeah, that should work.
[15:52] <smoser> 2 things, though
[15:52] <smoser> a.) "cloud-config" can do similar things (the update an package install)
[15:52] <smoser> b.) see https://help.ubuntu.com/community/CloudInit for multipart information on user data.
[15:53] <smoser> multipart becomes useful if you've got separate sections of code that want to add things to cloud-config or cloud-init . then you can keep them separated in input parts
[15:54] <asac> right
[15:54] <asac> looks interesting
[15:54] <asac> for now its prototyping ... more important for me is to remember the internal ip of the controller so i can copy the results and logs back ;)
[15:55] <asac> i thought about encoding that in user-data ... but guess
[15:55] <asac> i will read through that page and see how i can make things cleaner with cloud-init/config
[15:56] <asac> smoser: user-data is constrained to 16k, right?
[15:56] <smoser> yes, but it can be compressed.
[15:56] <asac> ok ... just wonder how to best ensure that i dont grow more than that ... but seems its better to put everything in bzr like i try now
[15:57] <asac> and just have the update, apt, etc. stuff and then run that
[15:57] <smoser> "runit" in http://bazaar.launchpad.net/%7Eawstrial-dev/awstrial/trunk/annotate/head%3A/awstrial/trial/ec2_helper.py is an example of appending parts
[15:58] <asac> smoser: i know its lame, but is it safe to just copy a script like i have there in the user data field when launching instances on the webUI?
[15:58] <asac> or better use ec2- commands with --user-data-file?
[15:58] <asac> (well better is wrong question ...)
[15:58] <smoser> yes, it should work.
[15:59] <asac> cool ;)
[15:59] <smoser> regarding 'safe' no one makes promises on sensitive data you put in there though.
[15:59] <smoser> asac, please let me know if pasting into that box does not work when running from command line does (for plain text)
[15:59] <smoser> i dont ever launch from that ui
[15:59] <asac> sure ... safe == script is properly run ;)
[15:59] <asac> smoser: i am still not sure how to copy stuff back ... :/
[16:00] <smoser> copy back ?
[16:00] <smoser> as in phone data home ?
[16:00] <asac> smoser: is hte idea to create a keypair for one run ... copy back to controller and then invalidate keypair?
[16:00] <asac> smoser: yeah
[16:00] <asac> like i want the logs and the image produced come back to my controller
[16:00] <asac> for persistence
[16:00] <smoser> right.
[16:00] <asac> but i dont know what best to do ... and how to ensure that that is not completely insecure
[16:01] <smoser> so it really just depends on how secure you want to be.
[16:01] <asac> like ... putting private id_rsa in user-data ;)
[16:01] <asac> what are the options?
[16:01] <smoser> well, no one makes promises on secure. but realistically, nothing outside your instance is going to get at that.
[16:02] <asac> yeah ... but what are the options?
[16:02] <smoser> so, you could:
[16:02] <smoser> - create a set of iam credentials, share that with the instance, let hte instance write to s3, then invalidate the credentials
[16:03] <smoser>   (you can even make a set of credentials that only work from a single IP address)
[16:04] <smoser> - share a secret in user data or in an snapshot/ebs-volume or s3 to the instance, then have it use that to post back to the controller
[16:04] <smoser> (ie, in a header of http post)
[16:05] <smoser> probably lots of other ways...
[16:05] <asac> yeah
[16:05] <asac> i think i like the idea to http post a notification back ... and then have the controller pull stuff over
[16:05] <asac> rather than a push from the instance
[16:06] <asac> but i dont want to muss the notification because then i guess the shutdown would have to be done by controller as well
[16:06] <asac> and i hate having logic to shut down stuff after a certain amount of time if nothing happened ... though maybe i hav eto do that anyway
[16:06] <asac> i will think a bit more about it ;)
[16:06] <asac> so you say i can create a bucket for each instance and create a special key for just that bucket?
[16:06] <asac> in s3?
[16:07] <asac> hmm. i think there was a constrained of number of buckets
[16:07] <asac> smoser: last thing for today ... in user data web field ... base64 encoded ... should i check that?
[16:08] <asac> or does it mean that i pasted base64 encoded stuff?
[16:08] <asac> (guess the latter)
[16:09] <asac> ok fired off without base64
[16:09] <smoser> there is a limit to the number of buckets you can have (i think)
[16:09] <smoser> but you may be able to limit to s3 bucket path
[16:10] <smoser> yeah, i would think it means you're pasting base64.
[16:10] <asac> ok ... will check that out... also look into messaging stuff etc.
[16:10] <smoser> i didn't know you coudl do that.
[16:10] <asac> let me see if my job runs ;)
[16:11] <smoser> asac, http://docs.amazonwebservices.com/IAM/latest/UserGuide/UsingWithS3.html
[16:11] <smoser> Example 5: Allow a partner to drop files into a specific portion of the corporate bucket
[16:11] <smoser> so you could have a bucketk "results"
[16:11] <asac> smoser: one question wrt to money ... if i shutdown after 5 minutes do i pay for full hour?
[16:11] <smoser> asac, yeah.
[16:11] <asac> damn ;)
[16:12] <smoser> the ceiling function kinda sucks.
[16:12] <asac> my script failed now i cannot shut it down
[16:12] <asac> guess i will try on micro next time ;)
[16:12] <smoser> there are ways to play faking user data post-first boot
[16:12] <asac> so the user data script is not run in /root/ home directory it seems
[16:12] <smoser> in maverick its fairly easy
[16:12] <asac> cannot find where it did the checkout :/
[16:13] <smoser> hm... i dont actually  know what the path woudl be.
[16:13] <asac> etting up bzrtools (2.2.0-2) ...
[16:13] <asac> Processing triggers for python-central ...
[16:13] <asac> You have not informed bzr of your Launchpad ID, and you must do this to
[16:13] <asac> write to Launchpad or access private data.  See "bzr help launchpad-login".
[16:13] <asac> Branched 2 revision(s).
[16:13] <smoser> nothing changes cwd so, whatever upstart would run something at
[16:13] <asac> /var/lib/cloud/data/scripts/part-000: 8: ./ec2-tools/node/setup-android-build-node: not found
[16:14] <smoser> https://groups.google.com/group/ec2ubuntu/browse_thread/thread/d4d51238a2afb55b/bca69f3e89c88d0b?lnk=gst&q=user+data#bca69f3e89c88d0b discusses how you can test user data after initial launch
[16:15] <asac> thx
[16:19] <asac> guess i will use a spot instance next time i try it ;)
[16:19] <smoser> asac, its only $0.10
[16:19] <asac> let me check
[16:20] <asac> smoser: 0,34
[16:20] <asac> smoser: problem is i need 64-bit for android
[16:20] <asac> so either micro with slow EBS ... or large :/
[16:20] <smoser> true.
[16:20] <asac> not really a big problem either ;) ... will just expense it
[16:21] <smoser> i do use spots a lot for playing. the problem with them for development is that they're slower to start
[16:21] <smoser> ie, total time till up and reachable will be probably 8 minutes rather than < 3.
[16:21] <asac> damn
[16:21] <asac> my auto java license accept didnt work :(((
[16:21] <asac> guess i have to set seen as well
[16:22] <asac> yeah
[16:22] <asac> smoser: do spot instances get killed earliest after 1h? e.g. if i pay in 1h rates?
[16:23] <smoser> you always pay in 1 hour in crements.
[16:24] <smoser> i was always under the impression that spot instances would be killed on 1 hour units. but i'm not sure.
[16:24] <smoser> ie, it would seem somewhat unfair for you got get a spot instance, and then them kill you after 10 minutes and charge you for 55
[16:24] <smoser> er... for 60
[17:59] <smoser> soren, you around ?
[19:48] <patrickw> Why don't UEC images have devpts in fstab?  Is it unnecessary because of the kernel?  I'm customizing a Centos image (that will use an Ubuntu kernel) and was wondering about this difference.
[20:12] <asac> smoser: http://paste.ubuntu.com/542014/ ... m2.xlarge with latest instance store 64-bit maverick :/
[20:12] <asac> doesnt boot :/
[20:12] <smoser> known issue
[20:13] <asac> smoser: oh ... so just for that instance type?
[20:13] <smoser> hold on. getting bug
[20:13] <asac> can i select a different kernel?
[20:15] <smoser> bug 651370
[20:15] <smoser> you can't specify a different kernel
[20:16] <smoser> asac, hopefully updates next week maybe, with that through -proposed
[20:16] <asac> hmm
[20:17] <asac> i can select kernels when creating an instance ... whats that?
[20:17] <smoser> its what you think.
[20:17] <smoser> sort of
[20:17] <smoser> :)
[20:17] <asac> why cant proposed kernels be made available there?
[20:17] <smoser> you can select a kernel, and prior to maverick, we would have published that kernel to ec2 that you could select
[20:17] <asac> ah
[20:17] <smoser> but with maverick the kernel is "pv-grub"
[20:17] <asac> maybe "proposed" AMIs ?
[20:17] <smoser> and pv-grub loads the kernel from inside the image
[20:18] <asac> oh
[20:18] <asac> ok
[20:18] <smoser> proposed amis woudl be somewhat useful for this sort of thing
[20:18] <asac> yeah
[20:18] <asac> definitly ;)
[20:18] <asac> i would use it!!
[20:18] <smoser> and right now, lucid is building with --proposed
[20:18] <smoser> because i wanted to test some things, but in general, the idea is that the dailies get turned into the rleeases
[20:18] <smoser> bit for bit
[20:19] <smoser> maybe i could start doing a weekly -proposed, which would never get promoted
[20:19] <smoser> but i tihnk in general we have too much selection up there (with the daily builds) and that confuses people.
[20:19] <asac> hmm
[20:19] <asac> selection where? you mean on uec-images? ... maybe
[20:20] <smoser> (ie, you wouldn't get or expect an ISO to have -proposed enabled or have been built from there)
[20:20] <smoser> as in selection, i really mean in number of "ubuntu" images on ec2
[20:21] <asac> so this bug ... is that juts for my instance type? or just more frequent for large memory instances?
[20:22] <asac> the reporter talksa bout t1.micro
[20:22] <asac> but my previous instances worked well
[20:22] <asac> like m1.large etc.
[20:23] <asac> just m2.xlarge is causing pain now
[20:24] <smoser> its really only the larger memory instances.
[20:24] <asac> damn ;)
[20:24] <asac> those i want to test now :-P
[20:24] <smoser> t1.micro is what the opener (me) opened it with
[20:24] <asac> ah
[20:24] <smoser> i just launch a t1.micro and run 'ubuntu-bug'
[20:25] <smoser> especially if you can't seem to get a instance type that you want to open to start :)
[20:25] <asac> alrighty
[20:25] <smoser> the issue is really with getting a certain processor
[20:25] <asac> so is lucid perfect?
[20:25] <asac> ;)
[20:25] <smoser> lucid has some issues too.
[20:26] <asac> booting big memroy?
[20:28] <smoser> lucid's big issue is https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/614853
[20:28] <smoser> there is a patch, but you can look at it (http://launchpadlibrarian.net/58956370/lp614853.patch) its not that pretty
[21:47] <soren> smoser: Yes.
[21:49] <smoser> soren, i'll probaly try to ping you monday
[21:49] <soren> smoser: I'm intrigued :)
[21:51] <smoser> soren, well, it was regarding the ec2 metadata swrevi ce
[21:51] <smoser> andthe fun that that that delivers
[21:52] <smoser> basically, i'm trying to make "full disk images" of the uec images, that ccan then just be booted under kvm.
[21:52] <smoser> in maverick i did this, but i did so via a floppy and bootinga partition image.  the floppy loaded the kernel from inside the image but passes some custom kernel parameters that disable the waiting for the ec2 metadata service
[21:52] <soren> smoser: How does that relate to metadata? The block mapping?
[21:53] <smoser> but if i just have one disk image, i'd like for its contents to boot correctly and not wait some stupid long time for the metadata service to come about.
[21:55] <smoser> i was wondering if you had any good ideas for how i could decide when to wait for it. but i can't use kernel parameters, because on uec, the grub.cfg is used for passing kernel params (so i can't just have that disable ec2 wiating)
[22:02] <soren> I remember having this discussion a couple of times. :)
[22:02] <soren> With you, even :)
[22:03] <soren> I think our conclusion was that there's no really good way to tell the difference.
[22:03] <soren> smoser: Ok, let's attack this from a slightly different angle.
[22:04] <smoser> soren, no, ther isn't.
[22:04] <soren> smoser: What if we try to minimise the impact of this wait time? Why do we wait for it to turn up?
[22:04] <smoser> and i said i should have waited till monday as i have to run righ tnow.
[22:04] <soren> Instead of just letting it wait in the background.
[22:04] <soren> smoser: Ah.
[22:04] <soren> :)
[22:04] <smoser> we wait for it to turn up because its possibly important
[22:04] <soren> Alright, Monday it is.
[22:04] <soren> smoser: Use cases?
[22:05] <smoser> really its only most important on first boot
[22:05] <smoser> but the idea is that the user can do things on first boot to an instance that will get done
[22:05] <smoser> such as setting mount points or importing ssh keys...
[22:06] <smoser> some of the stuff might be difficult to do "later"
[22:06] <smoser> anyway, i do have to run.
[22:07] <soren> Sure.
[22:07] <soren> I'd be happy to talk on Monday.
[22:07] <soren> Have fun.