[00:35] <flaccid> anyone alive that can help on disable_root and ec2-fetch-credentials ?
[00:35] <flaccid> i'm trying to work out this system they introduced in karmic
[00:36] <flaccid> i'm not sure why they do it this way instead of just sshd and pam
[00:37] <erichammond> flaccid: Are you wanting to enable ssh to root?
[00:37] <flaccid> heya Eric, yeah, I have raised an enhancement request so we can enable other users than root to login and also call scripts etc.
[00:38] <flaccid> for now, i just want to re-enable root so i can bundle this test instance which has been right-enabled
[00:38] <flaccid> then i can look at the more specific issue in the ticket raised indirectly by yourself
[00:40] <flaccid> interesting implementation canonical does in /usr/bin/ec2-fetch-credentials ..
[00:40] <erichammond> I posted a description on how to enable root towards the end of http://alestic.com/2009/04/ubuntu-ec2-sudo-ssh-rsync
[00:44] <flaccid> i don't think that is going to do the trick, but i will try
[00:46] <flaccid> wow it does. how does that work?
[00:46] <flaccid> i'm trying to work out what calls /usr/bin/ec2-fetch-credentials to check disable_root boolean
[00:49] <flaccid> hmmm bundle call from rs dash still silently failing
[00:53] <flaccid> i think i will have to call the ec2 api manually
[00:54] <erichammond> flaccid: /etc/init.d/ec2-init
[00:54] <erichammond> calls ec2-fetch-credentials
[00:54] <erichammond> Where do you see disable_root?
[00:57] <flaccid> grep disable_root /usr/bin/ec2-fetch-credentials
[01:04] <flaccid> would you be able to help me with a manual bundle then?
[01:05] <erichammond> flaccid: of a running instance?
[01:05] <erichammond> or building an AMI from scratch?
[01:05] <flaccid> yeah so ec2-bundle-vol on a running instance. need to know best practices for the params on a running instance w/ rightscale
[01:06] <flaccid> basically because i've never done it before, need to know what params i should be giving http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/CLTRG-ami-bundle-vol.html
[01:07] <erichammond> Here are my notes which should work on Karmic: http://alestic.com/2009/06/ec2-ami-bundle
[01:07] <flaccid> great, thats quite detailed, thanks eric
[01:08] <erichammond> I'm also building Karmic AMIs from scratch with vmbuilder but haven't published any notes yet.
[01:08] <flaccid> got any ideas why rs bundle call is silently failing after enabling root ?
[01:09] <erichammond> I don't think I've ever used the RightScale bundle button.
[01:09] <flaccid> im testing bundle of Ubuntu 9.10 Karmic	ami-1515f67c after right-enabling
[01:09] <flaccid> yeah, our bundle button is not reliable and i wouldn't mind working out why. but without interaction with eng I don't know what the api call it issues is. would be good if aws logged all api calls and responses..
[01:10] <erichammond> Are there any logs on the system?  Have any Ubuntu AMIs been able to be bundled that way before?
[01:10] <erichammond> Perhaps it's an incompatibility with commands between CentOS and Ubuntu.
[01:10] <flaccid> we only notify that the bundle call was initiated. i can bundle jaunty no problem and also done debian
[01:11] <flaccid> i'll show you the build toolkit i made
[01:12] <flaccid> erichammond: https://rightscale-services.s3.amazonaws.com:443/rightimage-build-toolkit/README
[01:13] <flaccid> erichammond: could you summarise the main roadblock you had, thus the ticket submitted?
[01:15] <flaccid> in regards to /bin/env: bad interpreter: No such file or directory. thats actually our bug. we should be using /usr/bin/env
[01:15] <flaccid> in my build-toolkit i create an extra symlink to work around this
[01:16] <erichammond> This shows up in /var/log/install: "touch: cannot touch `/var/lock/subsys/rightscale': No such file or directory"
[01:16] <flaccid> yeah i got that in jaunty as well. just need to touch it iirc
[01:16] <erichammond> and "/opt/rightscale/bin/ec2.sh: /opt/rightscale/bin/WaitFor: /bin/env: bad   interpreter: No such file or directory"
[01:16] <flaccid> yeah for that one, do, /bin/env: symbolic link to `/usr/bin/env'
[01:17] <erichammond> symlink sounds like a very temporary workaround which should be corrected quickly.
[01:17] <flaccid> problem is they may not update the current rightscale scripts due to focus on RightLink
[01:17] <flaccid> we can still raise bugs and try to get priority on them
[01:19] <erichammond> ok, I can try adding these and see how the build goes: mkdir /var/lock/subsys; ln -s /usr/bin/env /usr/env
[01:21] <flaccid> mkdir -p /var/lock/subsys && touch /var/lock/subsys/rightscale; ln -s /usr/bin/env /bin/env
[01:21] <flaccid> :)
[01:21] <erichammond> thanks
[01:21] <flaccid> nps
[01:21] <erichammond> It's ok for subsys/rightscale to be owned by root?
[01:21] <flaccid> yep
[01:22] <flaccid> generally nothing in there most of the time
[01:23] <flaccid> if i add the mkdir/touch to my build toolkit, that should cover full support for jaunty and karmic
[01:27] <flaccid> just going to launch an rs server w/ jaunty template and rightimage to verify the console output errors such as the /var/lock/subsys/rightscale
[01:41] <erichammond> flaccid: All these different ways of installing RightScale are starting to wear me down.  What you just listed is different from what Martin provided me which is different from the other 4 ways I used to do it.
[01:43] <flaccid> well, that certainly is a problem
[01:43] <erichammond> Hopefully, we'll end up with a single recommended approach out of all this.
[01:46] <flaccid> its going to be hard, particularly because of the focus on RightLink which will provide packages to simply install the RightLink agent
[01:46] <flaccid> have you seen http://support.rightscale.com/index.php?title=06-FAQs/FAQ_0103_-_How_do_I_make_any_Amazon_Machine_Image_%28AMI%29_capable_of_running_RightScripts%3F ? my build toolkit is basically a fork of that
[01:49] <erichammond> Here's the basic script from Martin which requires setting a number of envariables first: http://alestic-downloads.s3.amazonaws.com/rightscale_install
[01:49] <erichammond> I see overlaps with your code.
[01:49] <flaccid> yes that is correct
[01:52] <erichammond> flaccid: so which one is preferred?
[01:53] <flaccid> there is no preferred. its all up to you. there is no official way to do this
[01:54] <flaccid> the equiv of this script is https://rightscale-services.s3.amazonaws.com/rightimage-build-toolkit%2Finstall_rightscale.bash and i just polished it up and modularised some things into other scripts
[01:54] <erichammond> Is root ssh required to perform some RightScale functions?
[01:54] <flaccid> yes because ssh/root is used
[02:01] <erichammond> flaccid: Were you looking for this before? /etc/ec2-init/ec2-config.cfg
[02:02] <flaccid> ah yes
[02:02] <flaccid> this is bundled by canonical right?
[02:02] <erichammond> yes, that file is on the Canonical AMIs
[02:03] <flaccid> coolio
[02:03] <erichammond> and any AMIs which are built with vmbuilder, I presume.
[02:03] <erichammond> (I'll know shortly)
[02:04] <flaccid> okies
[02:04] <flaccid> well i am actually getting the output from our worker on running the bundling script
[02:04] <flaccid> looks like missing dep for the script
[02:09] <flaccid> [2009-12-28 02:05:51] [Ec2BundleWorker] : ERR: /home/ec2//lib/ec2/amitools/crypto.rb:13:in `require': no such file to load -- openssl (LoadError)\n
[02:09] <flaccid> need openssl rubygem
[02:10] <flaccid> apt-get install libopenssl-ruby issued
[02:12] <flaccid> bundling initiated
[02:12] <flaccid> rsync is going so it appears to be working without error so far
[02:15] <flaccid> tar is running. good so far..
[02:17] <flaccid> gzip running :)
[02:23] <flaccid> splitting bundle now
[02:25] <flaccid> erichammond: that was successful. let me see how it goes now under an rs server
[02:25] <rberger> If I launch a canonical US-West AMI in US West it doesn't have access to http://us.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates and apt-get update fails
[02:26] <rberger> Anyone know how to fix it or if I'm doing something wrong?
[02:27] <erichammond> rberger: You can switch to us-west-1.ec2.archive.ubuntu.com
[02:27] <erichammond> Or (as I do) switch to the RightScale Ubuntu mirrors.
[02:28] <rberger> erichammond: Thanks, is this a known bug?  What is the address of the rightscale ubuntu mirros (ps you are my hero :-)
[02:31] <erichammond> rberger: https://bugs.launchpad.net/ubuntu/karmic/+source/ec2-init/+bug/494185
[02:31] <uvirtbot> Launchpad bug 494185 in ec2-init "ec2-init selects us-east-1 mirror when running in us-west-1 region" [High,Fix committed]
[02:31] <erichammond> It's marked "fix released", but I didn't think that it was in the latest official AMI from Canonical.
[02:32] <erichammond> I also suspect it will not work for you if you rebundle an AMI in us-east-1 and then migrate it to us-west-1 or eu-west-1.
[02:33] <erichammond> rberger: If you have runurl installed (http://alestic.com/2009/08/runurl) then the following command switches to the RightScale Ubuntu apt repository mirrors: sudo runurl run.alestic.com/apt/rightscale
[02:34] <erichammond> If you want to only upgrade to the Ubuntu archives as of a particular date, RightScale keeps daily snapshots.  For, say, December 1: sudo runurl run.alestic.com/apt/rightscale 2009/12/01
[02:34] <flaccid> cool
[02:35] <erichammond> RightScale has 2 load balanced, failover mirrors in us-west-1 and 3 in us-east-1.
[02:36] <erichammond> When one of the EC2 availability zones failed a couple months back, my instances in a different availability zone could still upgrade because they failed over to the mirror in a zone which was working.
[02:36] <flaccid> erichammond: i have a RightImage karmic working. would just need some script mods for monitoring and scripting to work, but this is separate to the image of course
[02:37] <erichammond> flaccid: Glad to hear it.  I'm continuing on the path I was traveling with some mods from this session, but may switch over to your code if I have problems.
[02:38] <flaccid> yeah its really just a matter of dealing with each issue
[02:38] <flaccid> erichammond: those errors you saw, they were in console output right?
[02:39] <erichammond> flaccid: They were in /var/log/install if we're talking about the same thing
[02:39] <rberger> erichammond: I am using the ami canonical lists on their website ami-7d3c6d38. THanks again.
[02:39] <flaccid> erichammond: ok. i will check this
[02:40] <flaccid> erichammond: a launch and terminate shows no errors in console output. so i will now go over the /var/log/install on a fresh launch
[02:40] <erichammond> rberger: Though I pointed you to some code on run.alestic.com I should caution you that it is just my playground and should not be used for production systems.  Feel free to copy the code and host it on your own reliable locations.
[02:42] <rberger> erichammond: Will do. Working on gettiing my own base system on us-west for use with opscode chef. Hopefully it won't be too hard... All your tools and docs are a big help!
[02:42] <erichammond> rberger: Glad you find them useful.
[02:42] <flaccid> rberger: have you checked out rightscale/chef yet ?
[02:43] <rberger> flaccid: Does the rightscale/ chef stuff cost money on a instance-hour basis?
[02:43] <flaccid> rberger: no
[02:43] <flaccid> you can get a free developer account if you wanna try it out
[02:44] <flaccid> well from memory you get chef w/ free accounts
[02:44] <flaccid> personally, im not into chef at all. im not a ruby guy either
[02:44] <rberger> flaccid: But in production it cost per instance per hour? I'm more concerned about production as our base line system is over 10 nodes..
[02:45] <flaccid> rberger: rightscale is a management platform, your cloud accounts such as AWS are with them
[02:46] <erichammond> flaccid: Ok, my vmbuilder-built, RightScale-enabled, Karmic AMIs work when started with the EC2 API.  Now to tag them with the magic tags in RightScale and test them there.
[02:47] <rberger> flaccid: It seems that the righscale folks are doing great stuff but we can't afford to pay extra on a per node/hour basis.
[02:47] <flaccid> erichammond: ok so you are essentially saying you are up to testing, right ?
[02:47] <erichammond> flaccid: yep, testing in RightScale.
[02:47] <erichammond> flaccid: Is there a trivial RightScript I could use to test that RightScript functionality works?
[02:47] <erichammond> I wrote my own, but figured there should be something available in the public library.
[02:48] <flaccid> rberger: we don't charge anything per hour, only certain overages with premium accounts. a free dev account doesn't cost you anything, only AWS...
[02:48] <flaccid> erichammond: a hello world is a good one to use
[02:48] <flaccid> i don't think there is anything in the library unfortunately
[02:48] <flaccid> i just use a bash hello world
[02:48] <rberger> flaccid: I'll look into it again, thanks
[02:49] <flaccid> rberger: nps. im a support engineer with RightScale. so if you have any questions, i'm around..
[02:51] <flaccid> erichammond: ok so only error left i have is touch: cannot touch `/var/lock/subsys/rightscale': No such file or directory because i didn't do anything before bundling. can i ask where you did the mkdir and touch for this ?
[02:51] <flaccid> just before bundling or in a sequence ?
[02:52] <erichammond> That was needed in the rightscale install script I was running, so I did mkdir/touch before I ran it.
[02:53] <erichammond> I'm building the image from scratch in a subdirectory, so I had to include the root of the image directory as well.
[02:53] <flaccid> yeah so you just mkdir and touch before bundling so it exists in the image, correct?
[02:59] <flaccid> ok doing another bundle after touching this file
[02:59] <flaccid> i'll let you know if i can find any problems/limitations after i have updated the template to use the new image
[03:00] <erichammond> flaccid: yes
[03:05] <flaccid> danke
[03:06] <erichammond> flaccid: No luck: flaccid: touch: cannot touch `/var/lock/subsys/rightscale': No such file or directory
[03:06] <erichammond> This may be wiped out in the boot process.
[03:06] <flaccid> okies, i shall look into this now!
[03:06] <erichammond> It may need to be fixed by RightScale when running on Ubuntu.
[03:08] <erichammond> I think Ubuntu would expect it to be /var/lock/rightscale/xxx
[03:08] <erichammond> with RightScale creating the subdirectory
[03:08] <erichammond> At least that's how the other /var/lock users seem to be behaving on my system.
[03:09] <erichammond> /var/lock has the same permissions as /tmp (world writable, sticky bit)
[03:09] <flaccid> might have to add to /etc/rc.local or something
[03:09] <flaccid> let me just replicate the problem from this new bundle when its done
[03:10] <erichammond> It has to be done before /etc/init.d/rightscale runs at S90
[03:11] <flaccid> right
[03:11] <flaccid> i assume /etc/rc.local is practically the last thing after the rc seq yeah ?
[03:13] <erichammond> S99 I believe
[03:13] <flaccid> ah rightio
[03:13] <flaccid> i'll check it out
[03:13] <erichammond> I'm also getting what looks like a more serious error in RightScale startup.  Here's the whole /var/log/install - http://paste.ubuntu.com/347903/
[03:15] <flaccid> danke
[03:16] <erichammond> Adding: apt-get install libxml-simple-ruby
[03:16] <erichammond> but this is going to be a slow process if I have to keep building AMIs to test :-\
[03:16] <erichammond> Should I switch to your code/
[03:16] <erichammond> ?
[03:16] <flaccid> yes iirc i handled that in my build toolkit
[03:17] <flaccid> probably in the deps scripts
[03:17] <flaccid> oops, dep script.
[03:17] <flaccid> yeah certainly covered that when i hit it
[03:22] <erichammond> flaccid: Are these still the latest and greatest instructions? https://rightscale-services.s3.amazonaws.com:443/rightimage-build-toolkit/README
[03:23] <flaccid> erichammond: yes. the whole thing is authored by myself and any updated, i just save straight back to the files
[03:24] <erichammond> Hm, looks like I'll have to chroot to the image directory first.
[03:25] <flaccid> i should create a TODO, there is still some things to add like java jre
[03:25] <flaccid> not sure why you would need to do something like that on a running instance..
[03:25] <erichammond> I'm not bundling my running instance.  I'm building a fresh Karmic AMI from scratch using vmbuilder.
[03:25] <flaccid> ok, sounds good then
[03:26] <flaccid> keep in mind this build toolkit is for running instances bundling. i havnt done any scratch stuff
[03:26] <erichammond> Does your code handle setting disable_root ?
[03:28] <flaccid> negative. i have not updated anything for karmic yet
[03:28] <erichammond> ok
[03:28] <flaccid> but that atm, looks like the intention. i doubt eng are going to address this 'feature' anytime in the near future..
[03:29] <flaccid> it does need to be addressed and my fav OS is freebsd, so when ec2 upgrades xen, i need support for non-root
[03:43] <flaccid> ok so rc.local will get run before rightscale init
[03:50] <flaccid> ok i am re-bundling with the touch in /etc/rc.local
[03:53] <erichammond> flaccid: From what I see S90rightscale is run before S99rc.local
[03:54] <flaccid> lrwxrwxrwx 1 root root  18 2009-10-27 13:43 S99rc.local -> ../init.d/rc.local
[03:54] <flaccid> lrwxrwxrwx 1 root root  20 2009-12-27 23:43 S99rightscale -> ../init.d/rightscale
[03:54] <flaccid> its S99 for both, so rc comes before ri
[03:54] <erichammond> Ah, you have rightscale at a different level in your software.
[03:54] <flaccid> well at least this is how it is with karmic in runlevel 4 default
[03:55] <flaccid> erichammond: maybe. i don't recall changing this manually, so we would need to see how it is set
[03:55] <flaccid> can check that out in regression
[03:55] <erichammond> What problems does that missing directory/file cause?
[03:58] <flaccid> none that i have noticed
[03:58] <flaccid> i don't even know what its for
[04:02] <flaccid> relaunching with new bundle
=== emacsian1 is now known as emacsian
[04:11] <flaccid> erichammond: that did the trick. so its all good, except for the fact that its not official supported yet and thus repos support is a no go etc.
[04:13] <erichammond> The Ubuntu repositories hosted by RightScale are very stable.  In my experience, they have been more stable than the mirrors hosted by Canonical in EC2.
[04:13] <flaccid> and also the logging and monitoring scripts will require updates
[04:13] <erichammond> yeah, that makes sense.
[04:13] <flaccid> erichammond: thats the repos themselves. im referring to the userland implementation on the instance. we don't support new versions yet and i'm not sure if the later releases are mirrored.
[04:14] <flaccid> configuring software repos will return Failed to generate repository configuration: unsupported ubuntu release 9.10
[06:41] <erichammond> flaccid: Ok, it looks like basic RightScale integration is working on my custom vmbuilder Karmic AMIs.  I'm still using Martin's code with info from this discussion with you.
[06:41] <erichammond> At least basic RightScript stuff works, though I also see your latest errors, too (Failed to generate repository configuration: unsupported ubuntu release 9.10).
[06:41] <erichammond> Thanks for your help.
[06:41] <erichammond> I'm off to see Avatar 3D again...
[06:42] <flaccid> coolio
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond
=== erichammond1 is now known as erichammond