#ubuntu-cloud 2010-11-15
<mhall119> so I'm trying to get setup with one of Amazon's free micro instances
<mhall119> are there any official AMIs for Ubuntu 10.04 server?
<mhall119> or 10.10
<kiall> Heya - question re UEC networking, should an instance be able to access another instance's public IP address assuming there is an "allow everything" security group defined? I'm thinking its some sort of NAT issue ... but not sure if its intended or not :)
<jmgalloway> I have an ssh problem
<jmgalloway> I have a 5 node uec setup, and now cant ssh into two of the nodes
<jmgalloway> I keep getting a Received disconnect from nc1:  2: Too many authentication failures from clcuser
<jmgalloway> anyone know how to fix this?
<mhall119> okay, I'm feeling a bit like an idiot, I setup my first ubuntu ami instance on ec2, but I have no idea how to ssh into it
<mhall119> ah, user==ubuntu
<mhall119> I got it now
<jmgalloway> anyone here good with ssh?
<jmgalloway> I have a cloud setup and cant ssh into two of my physical machines
<jmgalloway> I'm getting an error as long as I try to ssh from my test machine
<jmgalloway> if I ssh into these two remote machines with any other machine, it works perfectly
<flaccid> jmgalloway: always good idea to mention the error
<kiall> Does anyone know if euca is *supposed* to support the idea of resolving instances's public DNS hostnames, to their private IPs when the lookup is done from another instance?  (The same way amazon EC2 do ...)
#ubuntu-cloud 2010-11-16
<smoser> kiall, it doesn't do that in default installations
<smoser> i believe that you can have it be the dns server, but that is not default install, and without it, it cannot answer dns requests
<kiall> no problem .. one other question - should instances be able to communicate with each other via their public IPs then?
<kiall> (I'm seeing some strange NAT "stuff" happening at the mo ;))
<kiall> meaning there is no way for 2 instances to talk without knowing their private IPs (not exactly ideal...)
<smoser> both public and private should work for instance-to-instance traffic
<kiall> thats what I thought .. but until 30 seconds ago, that simply didnt work .. and now is looking a bit wrong (aka "ping public-ip" .. "reaply from private ip", and an instance cant ping its own public IP).. The change that got me that far was removing the bridge on the CC priv net, and switching to a normal eth interface(
<Makere> I'm having trouble running instances
<Makere> they get stuck on pending
<Makere> can't get the bundled images to run at all
<Makere> let's put this otherway, has ANYONE managed to ANY images downloaded from the website and HOW?
<Makere> I'm losing all hope
<kiall> Makere, i scrapped the "store" images .. they simply didnt work!
<kiall> Downloaded the 10.10 uec image from uec-images.ubuntu.com and used the uec-publish-tarball command to push it up..
<kiall> worked like a charm
<kiall> (oh and - I had to increase the m1.small settings to get it to work with 64 bit images.. they dont work out of the box ... go figure  )
<Makere> kiall: well the uec-images on didn't work for me
<Makere> got stuck on pending :(
<Makere> no errors on any log, nc, nc
<Makere> cc
<Makere> kiall, you runnin lucid or maverick?
<kiall> maverick
<Makere> controllers on seperate pc's?
<Makere> one or more clusters?
<Makere> just [Brunning thevm's without ramdisk?
<kiall> a combined CLC/CC/SC/Walrus and a few nodes... (so 1 cluster)
<kiall> and no ramdisk for the 10.10 uec-images EMI
<Makere> I'm trying 2 clusters with 1 cloud controller and walrus and 2 cluster controllers/sotrage controllers
<kiall> Havent done a split setup yet .. but I hope your starting simple and going with 1 cluster and 1 NC first ;)
<Makere> so I guess the problem has to do something with splitting them :p
<kiall> ...and going with 1 clc, 1 cc, 1 nc*
<kiall> anyway .. I thing I found was, once I had tried a load of different things .. stop. wipe. reinstall with your new knowledge ;)
<kiall> 1 thing i found*
<kiall> Also .. I found the package install "buggy" and ended up using the CD install (hell - it even asked more UEC specific setup questions .. go figure)
<Makere> yea
<Makere> I did CD install
<Makere> problem is that we've done it multiple times now
<Makere> and this is not the first time we get the problem
<Makere> kiall: have you updated the packages?
<Makere> on the nodes or the "Controller"
<kiall> Makere, yea - apt-get update && apt-get dist-upgrade && reboot after the CD install .. then setup bridges etc, edit config, clean restart the euca services, and reboot .. - thats about all i needed to get a VM booted (there was more .. but VMs booted)
<Makere> setup bridges?
<Makere> :O
<Makere> no mention of that in the installation guide
<kiall> lol ... im sure it mentions it loads of times ;)
<Makere> https://help.ubuntu.com/community/UEC/CDInstall atleast this one doesn't
<kiall> http://www.google.com/url?sa=t&source=web&cd=1&ved=0CBcQFjAA&url=http%3A%2F%2Fcssoss.wordpress.com%2F2010%2F05%2F10%2Feucalyptus-beginner%25E2%2580%2599s-guide-%25E2%2580%2593-uec-edition-chapter-7-%25E2%2580%2593-network%25C2%25A0management%2F&rct=j&q=eucalyptus%20beginner%27s%20guide%20networking&ei=Np_iTKb0MIHtsgbz3snkCw&usg=AFQjCNGOWI5oW4sBUfT8-AcILsaCl8_hSQ&sig2=vbyB86w94lQLeUdxEo-9tw&cad=rja <-- was a good howto on the network setup that
<kiall> doesnt leave anything out
<kiall> wow... that URL wasnt exactly the one I was going for ;)
<kiall> but it should work ;P
<kiall> LOL .. it actually doesnt mention bridges ..â¥
<Makere> yea
<Makere> :)
<kiall> yea .. just go to chapter one of that link (and do it the CD install way instead from my exp) .. ;)
<Makere> so annoying >_>
<Makere> that guide looks like million times better
<Makere> thanks
<Makere> I wonder if I should start from the beginning
<Makere> zzz takes 2 hours to setup with 2 clusters
<Makere> I guess usb installation would be faster
<Makere> oh wow
<Makere> that ubuntu community guide is so shit compared to this
<kiall> can anyone with a UEC instance running check if they can ping that instances public IP from itself?
<kiall> (the euca community cloud cant, neither can my setup .. but I believe it should be able to)
<smoser> kiall, i'll check
<kiall> cool :) Lets see If i should quit my job or not ;)
<Makere> lol
<smoser> well, ping of 'public-ipv4' fails
<kiall> right so .. at best thats an inconsistency with EC2, and worst a bug :)
<kiall> I'll dig into iptables rules and see whats missing/wrong and file a bug
* You're now known as ubuntulog
<smoser> i tested in 10.04 host, not 10.10 it could have changed.
<smoser> but it is a bug if its an inconsistency with ec2
<kiall> I've tested with 10.10, but I can't guarantee my setup is correct
 * kiall can *never* remember what hairpin NAT iptables rules should look like  .. google time
<kiall> funny - it looks like the iptables rules are in place for hairpin NAT ...
<smoser> kiall, i did you a favor, can you do one for me ?
<kiall> sure
<smoser> ec2metadata --local-ipv4
<smoser> in an instance
<kiall> oo unavailable
<smoser> yeah.
<smoser> eucalyptus metadata service doesn't like a trailing slash
<smoser> http://169.254.169.254/2008-02-01/meta-data/local-ipv4/
<smoser> is what ec2metadata asks for
<kiall> Yea - without the / it works spot on
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-utils/+bug/676144
<uvirtbot> Launchpad bug 676144 in cloud-utils "ec2metadata should not add trailing / to requests" [Undecided,New]
* You're now known as ubuntulog_
* You're now known as ubuntulog
<jmgalloway> anyone here good with ssh?
<jmgalloway> I am getting an error when trying to connect to my walrus and cluster controller machines..
<kiall> smoser, got the missing rule (evenually) ;)
<smoser> oh. great. open bug and put it in there.
<kiall> "-t nat -A POSTROUTING -s $VNET_PRIV_NET/16 -d $INSTANCE_PRIV_IP -j MASQUERADE" seems to do the trick
<kiall> infact .. I wonder if one of the existing rules can be changed (now that I look at it and it works...)
<kiall> Yup - one of the existing MASQ rules has a ! $PRIV_NET which is the real issue it seems...
<kiall> smoser, worlds worst worded bug report ;)
<kiall> https://bugs.launchpad.net/eucalyptus/+bug/676167
<uvirtbot> Launchpad bug 676167 in eucalyptus "Hairpin NAT on CC disabled - EC2 inconsistency" [Undecided,New]
<smoser> kiall, fyi, if you wanted to a.) be nice and b.) push your fix in
<kiall> trying to figure out this hole bzr thing ;)
<kiall> whole*
<smoser> you could branch
<smoser> bzr branch lp:ubuntu/eucalyptus/natty
<smoser> wait
<smoser> bzr branch lp:ubuntu/natty/eucalyptus
<smoser> cd eucalyptus
<smoser> quilt new kiall-fix-hairpin
<smoser> quilt edit path/to/new/file
<smoser> quilt refresh
<smoser> dch -i
<smoser> # edit the new message
<smoser> bzr commit
<kiall> aha .. git is way easier ;)
<smoser> well, thats not really git versus bzr
<smoser> more bzr packaging (using quilt)
<smoser> but i will argue no further against git (as i tend to lean that way also)
<kiall> lol
<kiall> OMG @ dch -i .. what a timesaver
<kiall> comitted rev 174 .. I presume I should find a patch file here somewhere/
<kiall> ?
<smoser> kiall, oh. i guess i shoudl have said you'd have
<smoser> to bzr add debian/patches
<smoser> i hope
<smoser> you probalby didn't get hte patch added
<kiall> Nope :)
<kiall> add and commit again?
<kiall> or some form of bzr history rewrite ;)
<smoser> you can bzr uncommit
<smoser> that is probably what i would do
<smoser> uncommit
<smoser> then, add
<smoser> then 'debcommit'
<smoser> oh, and make sure your debian/changelog has '(LP: #BUGNUM)'
<smoser> that will tag the branch that you fixed that bug
<smoser> and hten when you push the branch to somewhere (like 'bzr push lp:~yournamehere/ubuntu/maverick/eucalyptus/fix-public-addr-ping'
<smoser> )
<smoser> that will get automatically linked to the bug
<smoser> then, you bother Daviey and tell him to not be lazy
<Daviey> o/
<kiall> Right so .. that should be pushed to lp:~kiall/ubuntu/natty/eucalyptus/fix-hairpin-nat ;)
<Daviey> kiall: What is the bug #?
<kiall> 676167
<Daviey> bug #676167
<uvirtbot> Launchpad bug 676167 in eucalyptus "Hairpin NAT on CC disabled - EC2 inconsistency" [Undecided,New] https://launchpad.net/bugs/676167
<kiall> BTW .. It reads like it was translated via google translate a few times .. I apologize ;)
<Daviey> kiall: interesting... is that a regression?
<kiall> smoser, tested 10.04, i tested 10.10 .. so I dont think so
<Daviey> kiall: ok, thanks!
<Daviey> kiall: looks good....
<Daviey> 2 comments...
<kiall> dooh
<kiall> 1 see one of them
<kiall> I see*
<Daviey> for maverick, as it is post release, the pocket needs to be "maverick-proposed" not "maverick" (top line of debian/changelog)
<Daviey> secondly, it would be awesome if the patch had DEP-3 "patch tagging"
<Daviey> "http://dep.debian.net/deps/dep3/"
<kiall> Right so .. First time doing anything other than a quick PPA ;)
<Daviey> You can get an idea of what patch tagging looks like by looking at the other patches
<Daviey> kiall: You've done great (and the hard part)!  I don't mind fixing them up before merging..
<Daviey> otherwise, awesome if you do it :)
<kiall> dont forget to remove the extra "network, slashnet, " I left in .. when re-doing after a clean clone ;)
<kiall> (thats the 1 I spotted)
<Daviey> kiall: the other thing, the version number for maverick-proposed should be "2.0+bzr1241-0ubuntu4.1"
<kiall> aha I'll blame that one on "dch -i" making me thing it knew better ;)
<Daviey> lol
<Daviey> kiall: seriously awesome work tho...  i'm over the moon you've contributed this
<kiall> lol .. 1 line patch .. cmon ;) (I work on some open source projects .. 1 lines aren't that hard to come by!)
<Daviey> kiall: we've traditionally not had great success getting contributions outside the usual suspects..
<Daviey> that is the main reason i'm pleased
<Daviey> OK... i need to go and do some family time..
<kiall> sure .. cya
 * kiall gets back to my real work after 2 days on/off wondering what I had configured wrong ;)
<Daviey> kiall: one more comment... can you add an impact statement to the bug....  why this is importiant it's fixed in stable versions?
<Daviey> kiall: needs to be fixed in natty first, then either you - or me should backport it to maverick and lucid
<kiall> sure ..
<Daviey> \o/
<Daviey> kiall: ok.. speak soon o/
<kiall> Impact statement added (At least I think I wrote an impact statement ;) .. maybe! ) .. right. home, then sleep. cya!
<kiall> oh god. reboot a server before I leave .. and ...
<kiall> # shutdown -r now
<kiall> -bash: /sbin/shutdown: Input/output error
#ubuntu-cloud 2010-11-17
<ernop> hey guys, got a question - i had an AMI in one region and used cloudyscripts "copy AMI to another region" thing to copy it.  but when I start the new instance, I can't connect - ssl times out.  Is there some step i'm missing?  I can start copies of it in the original region and connect no problem.
<ernop> (and I am running a modified 10.10 server AMI)
<SpamapS> ernop: my first question would be why are you using a modified AMI? (keep in mind, I'm a novice, but I only ever use cloud-config customized instances)
<Makere> kind of lost of where to start to get an image dl'd from http://uec-images.ubuntu.com/releases/ to run
<Makere> any ideas what I should check?
<ernop> SpamapS - by modified I just mean that I have installed a bunch of my own stuff on it.  Thing is, it worked in the original region.
<SpamapS> Makere: meaning downloaded to run in UEC (as in, eucalyptus) ?
<SpamapS> ernop: yeah, I think I'd do that by putting all of my stuff in a dir and tarring that dir up, or use a proper config management system like puppet/chef.
<Makere> SpamapS: yes
<Makere> and used the uec-publish-tarball
<Makere> to push it there
<Makere> fml
<ernop> SpamapS ok, good idea.
<Makere> So I tried creating a new uec image using this guide: http://cssoss.wordpress.com/2010/05/10/eucalyptus-beginner%E2%80%99s-guide-%E2%80%93-uec-edition-chapter-4-%E2%80%93-image%C2%A0management/
<Makere> the instances get stuck on pending, any troubleshooting tips?
<Makere> why the store images run
<Makere> but I can't get my own images to run
<Makere> nor images from uec-images.ubuntu.com
<Makere> ok
<Makere> the problem I had was that it doesn't work if I bundle on cloud controller
<smoser> Makere, i would doubt that its related to where you bundle
<smoser> remember that when you do a run-instances after uploading, the first time it will have to copy from walrus to node, change format of image (adding a partition table and such), before running
<smoser> that is a lot of IO and will be very slow for something large
<smoser> you can debug a bit more by figuring out where your instance is going to run (euca_conf --list-nodes) and then looking at the node. top will show you some things. likely there is something in /var/log/eucalyptus or /var/lib/libvirt
<Makere> smoser: well I waited for an half an hour, kvm doesn't run on the node
<Makere> node logs don't have any errors
<smoser> hm..
<Makere> I have nc log upped somewhere
<Makere> if you want I cna put it to pastebin
<Makere> but personally I think to myself as case closed, and will not compile on the cloud controller anymore
<Makere> bundle* not compile
<Makere> it worked charmed after bundling on cluster controller
<Makere> like a charm*
<Makere> sorry, a bit tired
<Makere> I have walrus and cloud controller on same machine
<Makere> and cluster controller + storage controller on other two
<Makere> plus nodes
<smoser> well, for what its worht, i have a cloud (10.04 right now, but i've used 10.10) with just 2 systems
<smoser> one is CC CLC and SC, one is just NC
<smoser> and i do everything from the CC/CLC/SC node.
<Makere> I think it might have something to do on cloud/walrus controller not having CC on same machine
<smoser> it won't make any difference, but if you find the bundle and upload process to be a bit obtuse, you can use 'uec-publish-tarball' or 'uec-publish-image'
<Makere> I did my own image with the publish-image
<Makere> same thing as with publish-tarball on official images
<smoser> ah. ok.
<smoser> just wanted to make you aware that is there.
<Makere> yea thanks
<smoser> once you've uploaded, i seriously can't see a difference in where the bundle was done
<smoser> after bundle, its already loaded into walrus, and the NC will do the get from walrus. no path there would be affected by where the 'bundle' command was run.
<Makere> http://open.eucalyptus.com/forum/i-still-can-only-run-instances-store-images same problem here
<Makere> found the answer from there
<Makere> although in the "answer" I got the guy seems to have run CLC CC SC and Walrus on same PC
<Makere> ("Ok, so I really don't know" by seigie) in the middle
<Makere> weird thing is that I/We reinstalled the cloud 4 times
<Makere> and everytime same problem
<Makere> with diff settings etc
<Makere> ah well, just wanted too put a heads up if anyone gets same kind of problem
<smoser> you get the errors like that in the logs ?
<Makere> don't have the logs here now
<smoser> well, the right thing to do is open an ubuntu bug
<Makere> yea
<smoser> so, go through the steps again
<smoser> writing down what you do and your setup (CC/CLC and such)
<smoser> see the failure
<smoser> then run 'ubuntu-bug eucalyptus' on the system that has those ERROR messages in the logs
<smoser> sorry to give youthe run-around, but in the end, that is what is most likely to get the problem solved for you.
<TritoLux_> hello there, anybody?
<TritoLux_> I am trying to setup a cloud test environment and I managed to get it to work, but it gives random network problems when multiple users run multiple instances. The Eucalyptus team told me that it is a UEC specific problem and I would need some help to troubleshoot. Is there anybody willing to help?
<TeTeT> TritoLux_: can't promise anything, have you tried connecting via public or private ip? And are the instances really up and running? You can check that with euca-get-console-output
<TritoLux_> hi there, yes everything seems to work fine, I can administer the instances via hybridfox with no problems, apparently.. but when a second user starts his own instance, then the other users loose the attached volumes from the SC and they even have to reboot their instance in orger to regain connectivity
<TritoLux_> after rebooting the instances, then nobody can detach the previously attached volumes and I have to reboot the cloud to restore functionality, until the same problem occurs again
<TritoLux_> thanks for replying btw
<TritoLux_> I have been troubleshooting this problem with the Eucalyptus team, but after a couple of months they are still puzzled and they said that it is UEC specific
<TeTeT> TritoLux_: sounds bad. I would recommend opening a bug about this in Launchpad against the eucalyptus package
<TritoLux_> the Eucalyptus team suspects AoE to be the problem
<TritoLux_> for instance, when I launch an instance, I get this error message in the livirt log: pci_add_option_rom: failed to find romfile "pxe-e1000.bin"
<TritoLux_> for instance, when I launch an instance, I get this error message in the libvirt log: pci_add_option_rom: failed to find romfile "pxe-e1000.bin"
<TritoLux_> I wonder if it is related
<TritoLux_> I also removed the AppArmor security layer in order to see if it was causing the problem, but pretty much nothing has changed
<TritoLux_> as a matter of fact, I was never able to have a stable cloud based on UEC
<TritoLux_> thanks for your responses TeTeT, I will eventually shoot a bug in Launchpad, but if anybody else has any advices in the meanwhile that would great. I can provide you with all logs and configs of course
<TeTeT> TritoLux_: I think the error is unrelated. it goes away when you add some kvm-pxe package
<TeTeT> TritoLux_: btw are you on 10.04 LTS or on 10.10?
<TritoLux_> I started with 10.04 LTS and I performed a dist-upgrade
<TritoLux_> it is currently running with all latest available packages
<TritoLux_> the environment is set with 2 machines, one running CLC, CC, SC, Walrus and the other one running the NC
<TritoLux_> both have dual NIC's, the CLC uses a public NIC and the CC is installed on a private NIC
<TritoLux_> eucalyptus.conf is set to run both priv and public traffic on the bridge configured on the non public interface
<TritoLux_> the nic's are configured for multiclustering, even if it is running only one at the moment
<TritoLux_> the eucalyptus guys had a look at the network config and they said that it is supposed to work fine
<TritoLux_> however, because both public NIC and private NIC are configured with a static address, then avahi autoregistration failed during UEC installation and it cost me some troubles to arrange a properly registered node. Now the node registration seems fine, but I wonder if the initial failure might have caused more problems
<TritoLux_> the network mode is MANAGED-NOVLAN
<TritoLux_> I have no error messages in the logs, until the above problem occurs
<TeTeT> TritoLux_: I register nodes from hand all the time and have not discovered your problem. But then I just do training and not some pre-production tests
<TritoLux_> if you try to install a cloud where the CC is running on a private NIC with a static IP then you will reproduce above registration problem
<TritoLux_> so to have a main IF on the public NIC and the nodes to be discovered on a private NIC, both with static IP's
<TritoLux_> the UEC installation process allows you to choose a static IP on the main NIC, but the same question is not asked for the secondary NIC and it assumes DHCP on that one
<TritoLux_> then autoregistration fails
<TritoLux_> the above problems are preventing us to start our cloud business at the moment and switching to Debian is currently being taken into consideration to be honest
<TritoLux_> basically, it seems that I cannot run more than a customer onto an apparently working cloud, otherwise the entire network gets messed up, as in connectivity getting lost, attached volumes becoming unaccessible, etc. rebooting the cloud seems the only solution to restore funzionality again.. for a single user, until a second one starts another instance.
<TritoLux_> I have tried to run more instances from a single user and it seems to work fine. The problems occur mainly or only when a different user starts to interact with the cloud.
<TeTeT> TritoLux_: not sure if the multi user scenario is well tested. I use it in the trainings, but this cloud only exists for a week
<TritoLux_> I think that multiuser is essential to a cloud, otherwise we miss the point of having a cloud. But I think that the multi-nic scenario is even less tested.
<TritoLux_> at least when you have both IF's with static IP addresses
<TritoLux_> my idea is to have just the CLC facing the public and the rest of the cloud interacting through a private network so that the public cannot even reach the other nodes
<TeTeT> TritoLux_: sound idea
<TritoLux_> eheh, it might be a good idea, but I wasn't able to get it to work in practice so far, unfortunately
<TeTeT> TritoLux_: I believe it's a scenario that should be supported
<TritoLux_> I am quite sure it is supported indeed, but it hasn't worked for me so far
<TritoLux_> and I dunno why
<TritoLux_> as we speak, two more boxes are being setup for a debian testing environment which will be configured in the same way, so to see if it is really an UEC problem or not.. thing is that Debian alred
<TeTeT> good luck with that
<TritoLux_> as we speak, two more boxes are being setup for a debian testing environment which will be configured in the same way, so to see if it is really an UEC problem or not.. thing is that Debian already runs Eucalyptus 2.0
<TeTeT> as does maverick
<TritoLux_> and UEC is running 1.6.2
<TritoLux_> Maverick is not the dist I have then
<TeTeT> TritoLux_: hmm, you really should try it then, I believe
<TeTeT> TritoLux_: it's not an LTS, so support will run out in 18 months time though
<TritoLux_> I am running the LTS distribution
<TeTeT> that's 10.04 LTS
<TritoLux_> yes
<TritoLux_> I thought that the LTS was more stable though, that's why my boss set it as a requirement
<TritoLux_> or at least better tested
<TeTeT> TritoLux_: it basically is, but with UEC it's still a pretty new product and big improvements are made with each release
<TritoLux_> that's understandable
<TeTeT> TritoLux_: the biggest step for 10.04 LTS in regards to UEC was IMO the neat installation via the CD
<TeTeT> TritoLux_: I know there have been made all kind of tests for stability, but I doubt the multi user and multi nic scenario is one of them
<TritoLux_> I think that the multiuser and multi nic scenario would be one of the most requested for production environments though
<TritoLux_> if I remember correctly, the initial tests we run when we approved UEC were run on a single NIC and everything went pretty much ok, but when we had to go in production environment, then things changed of course and more requirements were set.. but UEC didn't work well anymore
<TritoLux_> when do you think that Euca 2.0 will be available for LTS?
<TeTeT> TritoLux_: not sure, unless there's a strong business case probably never. I don't see anyone putting resources into a backport
<TritoLux_> auch, sorry what's the advantage of running LTS then?
<TeTeT> TritoLux_: security updates for 5 years on the server. It's not about getting the latest software
<TeTeT> TritoLux_: it's just my 2 cent, but I'd evaluate UEC on Maverick, upgrade that on a six month base and then go with the next LTS for long term production deployment
<TeTeT> TritoLux_: but not sure if that meets your project specs
<TritoLux_> mmh.. I thought it was both
<TritoLux_> that's a good suggestion indeed
<TritoLux_> thanks for your help, I will have to discuss this with my colleagues and better explain the scenario
<TritoLux_> there might have been misunderstandings on the LTS concepts
<TeTeT> TritoLux_: good luck, couldn't provide much though
<TritoLux_> I appreciate your response anyway, thanks for your effort nevertheless
<TritoLux_> it's not a simple issue anyway
<TeTeT> agreed
<TritoLux_> I'll go for a break now, speak you soon maybe
<TritoLux_> thanks again
<TeTeT> will be off soon, bye
<TritoLux_> bye bye
<smoser> erichammond, you still unable to find anyway to update ami pages, right?
<smoser> it sure seems like they're just in flux, but i opened a ticket and reopened several times.  iwas told that to edit a page i should "submit an ami" and mention that it is an update to an existing page in the body.
<kiall> anyone had any luck using the EC2 PHP SDK with euca/UEC?
<erichammond> smoser: I guess I've given up on Amazon's AMI pages since I haven't updated them in forever.
<smoser> i'm somewhat peeved at them over it.
<erichammond> smoser: I guess I should put in one more update that just says "this AMI has been superceded by the new recommended official Ubuntu AMIs..."
<smoser> i recently made an effort on them, to move out of the canonical account (the one that owns our images) to another
<smoser> so that people who updated those dont have to know credentials that could cause issues.
<smoser> but now, i can't even update the ones we have
<erichammond> :(
<smoser> i really think that they're probably just in transition. visually there are changes.
<smoser> but there is no external announcemnt of such a thing, and its been going on for the better part of a month
<kiall> humm, smoser think I just discovered a bug in the UEC API .. Might be normal tho! You know sec group names on describe instances come back as "group" while on run_instances come back as "<uec_username>_group" ..
<kiall> that the way EC2 does it? (I havent used their API)
<smoser> what is uec_username ?
<kiall> the username you use to login to the CLC web UI
<smoser> i think its probably a bug.
<kiall> so desc instances might return "default" while run will return "admin_default" when using the admin account ...
<kiall> crap .. time to go digging again ;)
<smoser> some things are different with admin
<kiall> Also .. the only way I've gotton euca to build is on a PPA with updates turned off .. is there some trick to built it after you do a "apt-get dist-upgrade"?
<kiall> Well, im not using the admin user .. just thought it was a good example ;)
<smoser> if it fails to build from source, then please open a bug
<smoser> and you're saying the API lists the user name ?
<kiall> yes - the group name returned (assuming group = "app" and username = "kiall") by describe is "app" while run instance returns "kiall_app"
<smoser> kiall, i'd have to look at it more, but i've never noticed differing output in the utility
<smoser> in euca2ools when working with ec2 verusus uec
<smoser> so i'd expect that its not different from what comes back from the api
<smoser> but i dont use security groups much
<kiall> I'll dig in a little more and look at the raw output rather than what the PHP EC2 SDK is giving me .. possibly a bug elsewhere
<kiall> Yup - The euca API is def handles group names differently ..
<TritoLux_> hello there, I would like to upgrade 10.04LTS to 10.10, which means also a major upgrade from Eucalyptus 1.6.2 to 2.0.. is there any particular step to be concerned of? any known problems? would the existing cloud keep all volumes, credentials and configurations? thanks in advance for your answers
<smoser> kiall, thanks. please do open a bug against eucalyptus
<kiall> just getting it together now
<smoser> again, using 'ubuntu-bug eucalyptus'
<kiall> humm?
<smoser> if you use 'ubuntu-bug eucalyptus' on the system, then it will put some info in the bug about what versions you're using
<smoser> it will also collect logs if you allow it
<smoser> thats why i suggested you do that on the machine with the Errors earlier (regariding your fialure to leave 'pending')
<smoser> i've got to run.
<kiall> Ah that wasnt me ;)
<smoser> i think it was
<kiall> Nope - last thing we talked about was hairpin NAT
<smoser> hm.. so it was. i confused you with Makere
<smoser> anyway.
<smoser> same deal goes for you :)
<smoser> use ubuntu-bug eucalyptus , that collects some system info
<kiall> I'll see if that works over SSH to the server .. dont have it installed locally..
<kiall> thanks ...
#ubuntu-cloud 2010-11-18
 * kiall really hopes I havent found another eucalyptus bug ;) Metadata sent in via EC2 PHP SDK ends up as jibberish :/
<kiall> <-- feels stupid ... needed tp base64_encode it ;)
<kiall> (But - at least EC2 told me .. unlike ueca :P)
<kiall> grr - UEC elastic IP "stuck" on an instances that no longer exists .. cant disassocate it :/
<Makere> how do I remove the store-images from cloud and disk?
<TritoLux_> Hello there, I upgraded my cloud from 10.04LTS to 10.10, nodes went fine, but the CLC is now extremely slow with high cpu usage, even when eucalyptus is shut down. It seems like if the kernel is not doing well. Did anybody experience the same here or is willing to point me to the right direction to solve this annoying problem please?
<TritoLux_> If I run top, then all processes seem at 0%, but cpu usage is always between 10% and 50% anyway. The CLI is quite unresponsive and I cannot even troubleshoot well.
<TritoLux_> ksoftirqd processes often reach 20%, it seems that the system is struggling
<TritoLux_> the ubuntu forums are down at the moment, so there's not much room for investigation out there at the moment. any help here would be appreciated
<kiall> TritoLux_, I havent updated from 10.04, but my 10.10 install is pretty responsive (unless i do loads of euca-get-consoleoutput's in a row .. where euca-cloud will chew 100%*Num of cores for a few mins!)...  Cant be of any real help tho .. sorry ;)
<TritoLux_> thanks kiall, my problem persists even if eucalyptus is not running actually
<kiall> Wait - the CLC is slow even when all euca services are stopped? It cant be UEC related then!
<kiall> dooh .. beat me to it ;)
<TritoLux_> I have no idea what is going wrong
<TritoLux_> yes kiall, I just found out that it is UEC unrelated
<TritoLux_> it took me ages to shut down eucalyptus though
<kiall> I presume you didnt do any HW changes at the same time? and you've shut down everything you can in case its another service?
<kiall> Beyond that .. it kinda has to be a kernel issue once you have everything off, and know the hardware's good..
<TritoLux_> I still trying to understand what other service could be, but apart from ksoftirqd all other processes seem to be sleeping
<TritoLux_> the HW is still the same, I performed an ssh upgrade
<TritoLux_> the server is quite fast and it was running well with  10.04
<kiall> I wonder if you had any 3rd party hardware drivers installed (RAID cards etc) that you may be using an open source alternative now?
<TritoLux_> I had not installed any driver manually to be honest, I have an Areca controller that was automatically recognized flawlessly with 10.04
<TritoLux_> and since the system starts, I guess that it was recognized on 10.10 as well
<TritoLux_> hey wait a sec..
<TritoLux_> it is apparently eucalyptus related
<kiall> Humm? I thought euca was off?
<TritoLux_> it just took more than 30 mins to shutdown the cloud
<TritoLux_> but now the system is responsive again
<TritoLux_> 99.9% idle
<kiall> Oh - wow. that cant be right
<kiall> anything in the logs between 30mins ago and nowÂ¿
<TritoLux_> I'll try to see the logs now.. as I finally can access them
<TritoLux_> I received a mail from the cloud saying: [FATAL] There is nothing to do here, since there are no nodes with any plugins.  Please refer to http://munin-monitoring.org/wiki/FAQ_no_graphs at /usr/share/munin/munin-html line 38
<kiall> I was wondering why munin was a dep ;) .. but that *probably* shouldnt cause any major issues... nothing else?
<TritoLux_> I'm checking
<kiall> `tail -n 50 /var/log/eucalyptus/* | less` and done ;)
<TritoLux_> I have many java cloud errors
<kiall> Not sure .. I dont get many errors showing up in mine ..
<TritoLux_> ERROR [TxHandle:New I/O server worker #1-7] javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
<TritoLux_> javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
<kiall> Cant connect the the database .. something must have mucked up during the upgrade
<TritoLux_> to be honest, at the end of the CLC-CC-SC-Wlarus machine upgrade, I received some weird errors, such as:
<TritoLux_> File descriptor 42 (/dev/pts/0) leaked on lvs invocation. Parent PID 3754: /bin/sh
<TritoLux_>   /dev/etherd/e0.4p1: read failed after 0 of 4096 at 0: Input/output error
<TritoLux_>   /dev/etherd/e0.5p1: read failed after 0 of 2048 at 0: Input/output error
<TritoLux_>   /dev/etherd/e0.6p1: read failed after 0 of 4096 at 0: Input/output error
<TritoLux_>   /dev/etherd/e0.6p2: open failed: No such device
<kiall> looks like it was shutting down some EBS volume exports, or trying to?
<TritoLux_> those are the only ones I saw, apart from a non fatal dhcpd3 failure
<TritoLux_> the reasons I wanted to upgrade to 10.10 is actually because I suspected some AoE issues
<kiall> humm ... anyway .. i've gotta run! good luck
<TritoLux_> thanks kiall
<kiall> (btw - its be 99% that failing to connect to the DB is your issue ...)
<kiall> (assuming that log repeats a few times a min ...)
<TritoLux_> yes, it's quite often
<TritoLux_> what should I look for in order to troubleshoot this DB issue?
<kiall> Honestly - Not sure .. still at the trial stage here :)
<kiall> i'd check /var/lib/eucalyptus/db/ and find out how hsql or whatever its called works!
<TritoLux_> ok thanks
<TritoLux_> kiall, dunno if you are still there.. about the slow response issue after my maverick upgrade, the funny part is that if I reboot the server I have a slow system, but if I manage to stop eucalyptus and then I start it manually, then the system is responding fine. As if there is some conflict during startup.
<TritoLux_> those db messages were only reported during the shut down process, which took ages, so the system was still looking for the db I guess
<TritoLux_> why eucalyptus is slow at startup and whyh ittakes ages to be stopped.. that I dunno yet
<TritoLux_> but it works fine if I manually start it again, really weird
<TritoLux_> did anybody else experience the above slow response at startup?
#ubuntu-cloud 2010-11-19
<jmgalloway> I am having trouble registering node controllers
<jmgalloway> I cant get them to list on the cloud controller after being discovered by the cluster controller
<SpamapS> smoser: you know what would be an awesome use of the landscape cloud stuff / cloud10 ? If I could, in a bug report, grant the user the rights to run a node of any AMI for 1 hour.
<SpamapS> Like "can you repeat that with a fresh install of lucid? Here's 1 lucid node .. go"
<smoser> SpamapS, i suggested this in the discussion about awstrial and was suprised that no one else found it a good idea.
<smoser> especially for "can you reproduce this on the development release"
<SpamapS> smoser: even a button on each bug that says "I want to try and repeat this bug" that takes you to an appropriate VM.
<smoser> as, i think that question is almost absurd for an end user... "Can you re-install your operating system and try this thing that maybe doesn't even boot for you"
<smoser> :)
<SpamapS> If we can't do it on cloud10, even just a hook into vmbuilder would be pretty cool.. ;)  vmbuilder --bug 559033
<uvirtbot> Launchpad bug 559033 in evolution "Evolution crashes on changing task priority" [Medium,Invalid] https://launchpad.net/bugs/559033
<smoser> SpamapS, yeah, i really think something like that woudlbe valuable.  A similar thing is doable with testdrive, but with ec2 you dont have to download anything
<smoser> how vmbuilder ?
<SpamapS> well you'd end up with a vm that starts up with the offending package installed. ;)
<SpamapS> (I just discovered vmbuilder 2 days ago btw.. I may not understand its full power ;)
<smoser> SpamapS, well as far as ease of use goes, i would recommend testdrive over vm-builder
<smoser> test drive - > launch instance -> apt-get intsall ...
<SpamapS> never used testdrive, but maybe its what I should be using instead of fighting with vmbuilder
<smoser> vm-builder -> some long command line set -> lots of image building , downloading -> boot instance.
<smoser> testdrive just boots the isos, but does so very easily
<smoser> if you want to "testdrive" a server install, i suggest the UEC images and booting them in kvm
<smoser> testdrive (maybe in natty now) has support for uec images as things to 'test drive'
<jmgalloway> does the uec node controller have to have vt extensions on the CPU?
<jmgalloway> anyone know why when I update max_cores, I dont see an increase in vm's I can launch on the cloud controller?
<Makere> so when I shutdown nodes/controllers, what happens to VM's?
<jmgalloway> their process stops execution
<Makere> jmgalloway: VT is needed
<jmgalloway> yeah, I was just hoping to use some older computers for nodes...
<jmgalloway> I still dont understand about the max_cores not updating
<Makere> can't help with that
<Makere> reboot/waiting solves alot
<jmgalloway> I tried rebooting the entire cloud, restarting nc, cc, and eucalyptus....doesnt update
<Makere> I'd look it up for you but I broke my right index finger and typing with one hand is a bit time consuming
<jmgalloway> I had it working before
<Makere> did you try clean start with cc?
<jmgalloway> seems like you just update the variable then restart the services and it works
<jmgalloway> hmm
<jmgalloway> let me try that
<jmgalloway> tried, didnt work
<jmgalloway> #MAX_MEM=2048
<jmgalloway> #MAX_CORES="8"
<jmgalloway> #MAX_DISK="100"
<Makere> you need to set max_cores on all NC's
<jmgalloway> that's the eucalyptus.conf of the node controllers
<Makere> and remove the #
<jmgalloway> lol! wow
<jmgalloway> thanks
<Makere> :)
<Makere> np
<jmgalloway> I was searching everywhere!
<Makere> lol
<Makere> # is for comment
<Makere> they've been commented out
<jmgalloway> yeah...lol
<jmgalloway> y
<Makere> what is the max_cores supposed to do?
<Makere> rather new to uec
<jmgalloway> if you change that variable on the node controllers...it will increase the number of vm's you can execute on the node controllers cpu
<jmgalloway> kvm allows for 8*#of physical cores
<Makere> so dual core can run 16 VM's?
<Makere> thanks, this seems to be something useful
<jmgalloway> yes but be careful..make sure you leave cpu power for the underlying os
<Makere> would be cooler if you could set that on cc instead of needing to set it on all nc's
<jmgalloway> I have dual core cpus..I run 4 vm's per physical core
<jmgalloway> I know
<jmgalloway> well you could just dd a hard drive and put it in a lot of node controllers...everything setup then
<jmgalloway> what kind of setup do you have for your cloud?
<jmgalloway> i have a build for adding node controllers for around $400 each.. quad core intel, 4GB ram, 320GB hds.. able to run 16 vm's
<Makere> doing a school project
<Makere> got around 60 comps for short period of time
<Makere> first settings up everythin with around 10 comps
<jmgalloway> 60 cool
<Makere> 2 cluster controllers
<Makere> one clc (that has walrus)
<Makere> CC's have storage controllers also
<Makere> might be adding more CC's
<jmgalloway> I have one clc, one walrus, one cc/sc, and two node controllers
<jmgalloway> I am only able to run 16 vm's right now
<Makere> kinda just have had only trouble with the cloud
<jmgalloway> the cloud?
<jmgalloway> cloud controller?
<Makere> mostly getting VM's running
<Makere> but it's all fine now
<jmgalloway> have you tried hybridfox?
<Makere> yes we use it
<jmgalloway> It seems sketchy...sometimes it works, sometimes it doesnt
<Makere> had trouble getting bundled images to run, and storage images are crap
<Makere> hybridfox works nice after you restart it couple times
<Makere> like if you get an error, close the tab and open new obe
<Makere> one
<Makere> the comps we got are C2D E6300 and 4Gigs RAM
<jmgalloway> yeah, I have problems with the keys and permissions using hybridfox.  It will launch vm's but wont let me connect to them even though I use the key it made for me
<jmgalloway> 60 of those?
<Makere> yes
<Makere> around 60
<Makere> +/-5
<jmgalloway> I have 6 total machines
<jmgalloway> I am writing a front end for my cloud...since everything I'm building is hosted on a private network(s) I have to give users that are not attached to my private network the ability to launch vm's
<jmgalloway> php is running on my cloud controller..communicates in the browser with the WAN users and spawns vm's with a gui interface.  it has to keep up with ports and update the iptables since I only have one WAN address
<Makere> sounds nice
<Makere> with that amount of machines, why don't you integrate CLC/CC/Walrus?
<Makere> then you could have second storage controller and 2 more nodes
<jmgalloway> I want it to be fully scalable
<jmgalloway> which it would be..but making them work with different physical machines is more interesting than combining them together.
<jmgalloway> does the sc and cc still have to be on the same box?
<Makere> trnope
<Makere> nope*
<Makere> atleast I think
<Makere> :D
<jmgalloway> I thought it did in euca 1.6...I'm using ubuntu 10.10 and euca 2.0
<Makere> well it can be
<jmgalloway> yeah, mine are combined
<Makere> as I say, rather new to this
<jmgalloway> sc/cc
<jmgalloway> open eucalyptus doesnt natively support windows images either
<jmgalloway> but it can be done
<jmgalloway> all of my vm's are linux, single core, 256mb ram
<Makere> you planning to run the cloud for a long time?
<jmgalloway> until I graduate
<jmgalloway> 2 years possible
<Makere> I think you should be able to put storage controller and cc to 2 diff machines
<Makere> but just reading in between lines
<Makere> of ubuntu instructions
<Makere> CC needs a storage controller
<Makere> but it can be on other pc afaik
<jmgalloway> the one thing I'm not sure about is how to make persistent storage that is mounted automatically for users when they launch vm's
<Makere> create storage volumes and script to launch vm?
<Makere> one that launches vm and mounts the volume
<Makere> or perhaps launches vm, creates volume and mounts it
<jmgalloway> I need to look into that
<jmgalloway> also, I need to look into quotas
<jmgalloway> so a single user cant spawn enough vm's to use all resources
<Makere> ye
<Makere> I'm in rather tight schedule so I try avoid completely custom solutions
<jmgalloway> In my setup, users will not be able to spawn vm's by command line
<jmgalloway> yeah
<jmgalloway> I have 2 years haha
<jmgalloway> or 1.5 years to go
<Makere> I think we have like 4 weeks left
<jmgalloway> oh cool.  is your cloud interface public?
<Makere> nope
<Makere> it's a school project
<jmgalloway> mine is...although I just threw together a front end web site
<Makere> we need to figure out if it's useful to teach cloud to students
<Makere> got like less than 2 months in total to get everything setup, demo'ed, written documents etc
<jmgalloway> sure it is, its compatible with aws
<Makere> really tight schedule
<jmgalloway> are you working in a group?
<Makere> group of 4
<jmgalloway> ah cool.  I'm doing this all alone
<Makere> not to sound cocky, but I seem to be the only one knowing wtf were doing
<jmgalloway> there's a lot to wrap your head around to administrate a cloud setup
<Makere> meaning I do almost everything
<Makere> and teach others lol
<Makere> naw the other guy gets around too
<Makere> but the other 2
<Makere> gah
<jmgalloway> my advisor is teaching a cloud class in the spring, and she's using this cloud in the class
<jmgalloway> I have to lock it down
<Makere> I would need to walk them thru by hand
<Makere> hehe
<Makere> remember to make lot's of security groups
<jmgalloway> well i really just need two security groups..one for me, and one for everyone else
<Makere> can you change the security groups between machines after launching them?
<Makere> atleast hybrid fox didn't seem able
<jmgalloway> I dont know.  I am going for a custom interface that gives the user a gui..so they click to launch vm's, not caring about how it launches or security groups..they will just have access to the vm
<Makere> heh ok
<Makere> I'm off to bed, night
<jmgalloway> php will generate the script to launch the vm, vm size, security groups, etc
<jmgalloway> later
<Makere> we got a blog @ silvercloud.wordpress.com if you want to see how our project is doing
<Makere> sorrt
<Makere> sorry
<Makere> silvercloudproject.wordpress.com
<Makere> (some blatant advertising :)
<jmgalloway> ok cool
#ubuntu-cloud 2010-11-20
<shahid_> Hi
<shahid_> I am going to deploy UEC on Production. Can any one guide me what server configuration should I buy for production
<shahid_> I want to run 512MB, 1GB, 2GB, 4GB and 8GB RAM's Virtual Machines with respective HDD. At first in production I am planning to run eucalyptus-cloud, eucalyptus-cc, eucalyptus-sc, and eucalyptus-walrus in one server and one node eucalyptus-nc in another. Can any one guide me, which server should I buy.
<shahid_> I mean server configuration RAM, HDD for these two servers
<Makere> as much ram as you can buy and cpu cores plus some big ass raid5 on NC
<Makere> then the controller some rather fast cpu and maybe 8gigs ram plus raid1/5 on storage
<Makere> :p
#ubuntu-cloud 2010-11-21
<mattwj2002> hi all
<mattwj2002> I got a question
<mattwj2002> what is a cloud and I am not talking about those white puffy things in the sky :P
<mattwj2002> is it basically a cluster computer?
<flaccid> check wikipedia - Cloud Computing
<mattwj2002> okay
<mattwj2002> hmmm
<mattwj2002> clear as mud
<mattwj2002> well how about this
<mattwj2002> I am trying to understand what UEC is
<mattwj2002> flaccid: are you here/
<mattwj2002> ?
<flaccid> mattwj2002: http://www.ubuntu.com/cloud/private
<mattwj2002> so you could add and remove hardware or possibly virtual hardware from the cloud to a virtual application?
<flaccid> sorry that doesn't make sense
<mattwj2002> no worries
<mattwj2002> okay
<mattwj2002> lets says you build a web server
<mattwj2002> but all of a sudden you find you need more power. If you had a cloud based web server, you could add more boxes to the cloud and have more processing power?
<mattwj2002> or remove if necessary?
<flaccid> sure
<flaccid> you can scale vertically or horizontally
<mattwj2002> nice :)
<mattwj2002> what I meant about virtually is if they are in vmware or something similar
<flaccid> yes, vmware does vcloud
<mattwj2002> you could scale back
<mattwj2002> nice
<flaccid> thats an example cloud
<mattwj2002> very cool *thumbs up*
<flaccid> so you should read about what a private, public and hybrid cloud is
<mattwj2002> :)
<mattwj2002> it is similar to a cluster though
<mattwj2002> but I see the differences now
<flaccid> see also, grid computing
<mattwj2002> okay
<mattwj2002> well actually soon I should go to bed
<mattwj2002> I am getting tired
<mattwj2002> :-s
<mattwj2002> good night
<mattwj2002> bye
<Sh4rma> hi
<Sh4rma> alive?
<Sh4rma> hi
<Sh4rma> anyone runs loadbalancer with apache or http server ?
