coolmariorocksi have a question04:19
coolmariorocksif i way ask in here04:19
pleia2coolmariorocks: you probably want to ask in #ubuntu (this channel is for classes)04:19
coolmariorocksok thanks pleia204:20
chinnappanevolution + exchange 2010 is not showing folder ? please help me ?05:11
chinnappanevolution + exchange 2010 is not showing folder ? please help me ?05:11
chinnappando you have any documentation for file server in linux?05:14
=== Guest5698 is now known as zumbi
=== daker_ is now known as daker
=== mike is now known as Guest40475
=== _LibertyZero is now known as LibertyZero
showkatwhen will start the cloud session12:57
=== med_out is now known as medberry
rwhcloud session starts at 16:00 UTC13:27
=== paulw is now known as Guest32626
=== Wordpad2 is now known as Amivit
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Cloud Days - Current Session: Getting started with Ensemble - Instructors: kim0
=== Hugo is now known as Guest19460
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/25/%23ubuntu-classroom.html following the conclusion of the session.16:00
kim0Hello, Good morning, good evening and good afternoon16:00
kim0Welcome to Ubuntu Cloud Days!16:01
kim0This is the second UCD ever16:01
kim0This event will be run for two days (today and tomorrow)16:01
kim0You can find more information regarding the event on https://wiki.ubuntu.com/UbuntuCloudDays/16:01
kim0It would be great to spread the news and let your friends join in16:02
kim0This is a great chance to get introduced to new ubuntu related server and cloud technologies16:02
kim0as well as a chance to connect to developers and active community members16:02
kim0Alright ..16:02
kim0Let's get started then16:03
kim0At any time you can "ask a question"16:03
kim0this is done by prepending your question with QUESTION: .. example ..  "QUESTION: what is xxx?"16:04
kim0a bot will pick up the question, and the instructor will answer it at a suitable time16:04
kim0So .. this session is for Ensemble16:05
kim0Take a moment to check out:  https://ensemble.ubuntu.com/16:05
kim0Ensemble is a cloud orchestration framework16:06
kim0Since cloud layers an API over compute resources16:06
kim0many compute resources such as servers, are more and more being regarded as disposable16:06
kim0people fire up servers, use them for an hour and destroy them16:06
kim0this is valid for both public and private clouds16:07
kim0as such, it would be pretty good to think at a higher level than a "server"16:07
kim0namely to think at the "service" level16:07
kim0this is one of the main concepts of Ensemble16:07
kim0Let's quickly discuss a few key concepts about Ensemble16:08
kim01- Ensemble focuses on the higher level concept of "Services" rather than "servers"16:08
kim0Examples of a "service" would be16:08
kim0- MySQL16:08
kim0- Memcached cluster16:08
kim0- Munin: as a monitoring service16:09
kim0- Bacula: as a backup service16:09
kim0and so on16:09
kim02- The second important concept is that Ensemble completely "encapsulates" those services16:09
kim0that is, if you have no idea how to get munin running16:09
kim0if you ask ensemble to deploy it, you would have it running a minute or two16:10
kim0and you can connect it (read: relate it) to other services16:10
kim0and it would start graphing performance metrics from all around your infrastructure16:10
kim0you do not need to know how to control munin, it is encapsulated16:10
kim03- The third important concept, is that with Ensemble services are "composable"16:11
kim0that is, services have well defined interfaces16:11
kim0such that you can connect/relate many services together .. to form a large infrastructure16:11
kim0you can replace infrastructure components with others .. such as replace mysql with pgsql if you so wish16:11
kim0and if both of them implement the same interface!16:11
kim0so ..16:12
kim0Ensemble enables layering a high level API over "services" and allows composing sophisticated infrastructures from that .. easily, consistently and without worrying about any details!16:13
kim0If you have any questions16:13
kim0now would be a good time to ask16:13
kim0remember to prepend any question with "QUESTION:"16:13
kim0I will now prepare the demo environment, that should clear up things a bit16:14
kim0For anyone wanting to follow along with the demo16:15
kim0Please ssh as user guest to the following machine16:15
kim0ssh guest@ec2-50-19-23-213.compute-1.amazonaws.com16:15
kim0password: guest16:15
kim0you will get a read-only view to a shared screen session16:16
kim0I will start the demo16:17
kim0I will be pasting commands and output text in this session as well, for archival purposes16:17
kim0The very first step we do is:16:18
kim0$ ensemble bootstrap16:18
kim02011-07-25 16:17:22,569 INFO Bootstrapping environment 'sample' (type: ec2)...16:18
kim02011-07-25 16:17:23,637 INFO 'bootstrap' command finished successfully16:18
kim0What the ensemble bootstrap does, is it starts a "management node" if you will16:18
kim0that is used to control our cloud deployment16:18
kim0let's check out the files available in the current directory16:19
kim0$ ls16:19
kim0byobu-classroom  drupal  mysql16:19
kim0byobu-classroom: setup scripts for the shared screen session you are see'ing .. This is not related to Ensemble16:19
kim0drupal: Ensemble drupal formula16:19
kim0mysql: Ensemble mysql formula16:19
kim0What is a formula you ask ?16:20
kim0A formula holds instructions for Ensemble on how to install and manage a service16:20
kim0that is .. the drupal formula, tells Ensemble how to install drupal, how to connect it to the database, how to create DB tables, how to configure a drupal website behind a load balancer ...etc16:21
kim0It is the experience of devops .. distilled .. into a "formula" that everyone can use16:21
kim0This is one of the great reasons "why use Ensemble" ..16:21
kim0Your deployment, not only becomes FAST, repeatable but also, you get the experience of the Ensemble community16:22
kim0all working for you .. without you even knowing about it (if you so choose)16:22
kim0alright ..16:22
kim0Let's deploy MySQL16:22
kim0jump to the screen session16:22
kim0The command to deploy a production mysql database is16:24
kim0$ ensemble deploy --repository=. mysql mydb16:24
kim0Let's break down this command and understand what it does16:24
kim0ensemble deploy → Asking Ensemble to deploy a service16:24
kim0--repository = . → Mentioning to Ensemble that the formulas are available in the current directory16:25
kim0mysql mydb → Deploy the formula "mysql" as a service called "mydb"16:25
kim0let's quickly paste the output of the command16:25
kim0$ ensemble deploy --repository=. mysql mydb16:25
kim02011-07-25 16:22:51,307 INFO Connecting to environment.16:25
kim02011-07-25 16:22:54,857 INFO Formula deployed as service: 'mydb'16:26
kim02011-07-25 16:22:54,859 INFO 'deploy' command finished successfully16:26
kim0So .. deploy .. finished successfully16:26
kim0similarly .. let's deploy the "drupal" formula .. as "mywebsite"16:26
kim0$ ensemble deploy --repository=. drupal mywebsite16:26
kim02011-07-25 16:23:04,117 INFO Connecting to environment.16:26
kim02011-07-25 16:23:05,167 INFO Formula deployed as service: 'mywebsite'16:26
kim02011-07-25 16:23:05,168 INFO 'deploy' command finished successfully16:26
kim0This should be very familiar16:26
kim0Let us check the status of our deployment16:27
kim0We use the "ensemble status" command for that16:27
kim0Here is the command and its output16:28
kim0$ ensemble status16:28
kim02011-07-25 16:27:37,395 INFO Connecting to environment.16:28
kim0  0: {dns-name: ec2-50-17-158-183.compute-1.amazonaws.com, instance-id: i-8dc16dec}16:28
kim0  1: {dns-name: ec2-184-72-129-61.compute-1.amazonaws.com, instance-id: i-35de7254}16:28
kim0  2: {dns-name: ec2-50-16-71-235.compute-1.amazonaws.com, instance-id: i-15de7274}16:28
kim0  mydb:16:28
kim0    formula: local:mysql-9816:28
kim0    relations: {}16:28
kim0    units:16:28
kim0      mydb/0:16:28
kim0        machine: 116:28
kim0        relations: {}16:28
kim0        state: started16:28
kim0  mywebsite:16:28
kim0    formula: local:drupal-916:28
kim0    relations: {}16:28
kim0    units:16:28
kim0      mywebsite/0:16:28
kim0        machine: 216:28
kim0        relations: {}16:28
kim0        state: started16:29
kim02011-07-25 16:27:38,635 INFO 'status' command finished successfully16:29
kim0Let's try to understand this output16:29
kim0In the "machines" section16:29
kim0We have 3 machines deployed16:29
kim00 1 and 216:29
kim00 is always the very first "bootstrap" node16:29
kim01 and 2 are the machines running mysql and drupal ..16:29
kim0Looking at the "services" section16:29
kim0we understand that we just deployed the service "mydb" .. remember this is the name we chose16:29
kim0the mydb service is running on machine "1"16:30
kim0and it is "started"16:30
kim0that is .. mysql has been installed and it is "ready" to be used16:30
kim0the same for drupal .. it is running on machine 2 and is started as well16:30
kim0It is interesting to note16:30
kim0that "relations: {}"16:30
kim0is empty16:31
kim0what this really means is16:31
kim0that the services deployed "mysql" and "drupal":16:31
kim0have not been "coupled" yet ..16:31
kim0i.e. mysql does not have the drupal database created yet ..etc16:31
kim0the magic of Ensemble and the very cool part .. is when you start connecting infrastrcuture pieces together16:32
kim0watching how all pieces jump together and a bigger system is created16:32
kim0let's connect those two components16:32
kim0The command to connect them (read: relate them) is16:33
kim0$ ensemble add-relation mydb:db mywebsite16:33
kim0We are adding a relation between mydb (our instance of mysql) and mywebsite (an instance of drupal)16:33
kim0It is extremely interesting what is happening at this instant16:34
kim0once this relation is established16:34
kim0both services start communicating and collaborating towards creating that bigger infrastructure16:34
kim0so .. mysql creates a database for drupal16:34
kim0it "sends over" the dabase details "username, password, DB name...etc" to the machine running drupal16:35
kim0drupal gets this configuration information16:35
kim0rewrites its configuration files to use this DB16:35
kim0creates its tables and configures the DB16:35
kim0the services have now been coupled!16:35
kim0Let's check the status16:36
kim0$ ensemble status16:36
kim02011-07-25 16:36:08,453 INFO Connecting to environment.16:36
kim0  0: {dns-name: ec2-50-17-158-183.compute-1.amazonaws.com, instance-id: i-8dc16dec}16:36
kim0  1: {dns-name: ec2-184-72-129-61.compute-1.amazonaws.com, instance-id: i-35de7254}16:36
kim0  2: {dns-name: ec2-50-16-71-235.compute-1.amazonaws.com, instance-id: i-15de7274}16:36
kim0  mydb:16:36
kim0    formula: local:mysql-9816:36
kim0    relations: {db: mywebsite}16:36
kim0    units:16:36
kim0      mydb/0:16:36
kim0        machine: 116:36
kim0        relations:16:36
kim0          db: {state: up}16:36
kim0        state: started16:36
kim0  mywebsite:16:36
kim0    formula: local:drupal-916:37
kim0    relations: {db: mydb}16:37
kim0    units:16:37
kim0      mywebsite/0:16:37
kim0        machine: 216:37
kim0        relations:16:37
kim0          db: {state: up}16:37
kim0        state: started16:37
kim02011-07-25 16:36:09,646 INFO 'status' command finished successfully16:37
kim0Notice how the "relations:" field now relates each component to the other16:37
kim0of course this could be a much larger system16:37
kim0i.e. there could be a load balancer front end service, a backup service, a monitoring service ...etc16:37
kim0But fundamentally .. it's the same16:37
kim0You deploy components .. connect them together and your good to go!16:38
kim0So .. our drupal instance is ready .. why not pay it a visit16:38
kim0Since drupal is running on machine 2 .. from the machines section .. this is the machin we need: ec2-50-16-71-235.compute-1.amazonaws.com16:39
kim0Go ahead and visit16:39
kim0Indeed drupal is there waiting for us! (woohoo) that was easy16:39
kim0Note how I might have deployed drupal without really knowing anything about how it needs to be deployed16:39
kim0and yet .. the deployment is done according to best practices of the Ensemble formula writers community16:40
kim0Awesome .. let's create a tiny first post16:40
kim0Alright .. we now have some content16:41
kim0Just refresh http://ec2-50-16-71-235.compute-1.amazonaws.com/ensemble/16:41
kim0Now .. here comes another (OMG this is awesome) moment16:42
kim0What about your blog (or whatever service) suddenly becomes popular16:42
kim0you're slashdotted16:42
kim0You want to scale out16:42
kim0sure this has to be complex right!16:42
kim0let's check out how we can get this done16:43
kim0This is what we need16:43
kim0$ ensemble add-unit mywebsite16:43
kim0Yes that's it .. we have scaled out16:43
kim0let's quickly understand this command16:43
kim0add-unit : Adds a service unit to "mywebsite"16:44
kim0remember mywebsite is that name of our instance of the drupal formula16:44
kim0A new ec2 instance is created16:44
kim0It is important to note .. that Enesmeble uses plain "vanilla" ubuntu images16:44
kim0everything is installed and configured on the fly16:44
kim0the new node is configured as type "mywebsite"16:44
kim0what is really awesome is16:45
kim0since this new node, is of type mywebsite .. it already "knows" how to hook up to the surrounding services!16:45
kim0In this case .. only mysql .. but could be much more sophisticated16:45
kim0This is the DRY: Don't Repeat Yourself .. concept16:45
kim0let's again quickly check out status16:46
kim0$ ensemble status16:46
kim02011-07-25 16:46:17,368 INFO Connecting to environment.16:46
kim0  0: {dns-name: ec2-50-17-158-183.compute-1.amazonaws.com, instance-id: i-8dc16dec}16:46
kim0  1: {dns-name: ec2-184-72-129-61.compute-1.amazonaws.com, instance-id: i-35de7254}16:46
kim0  2: {dns-name: ec2-50-16-71-235.compute-1.amazonaws.com, instance-id: i-15de7274}16:46
kim0  3: {dns-name: ec2-50-16-175-35.compute-1.amazonaws.com, instance-id: i-73a50912}16:46
kim0  mydb:16:46
kim0    formula: local:mysql-9816:46
kim0    relations: {db: mywebsite}16:46
kim0    units:16:46
kim0      mydb/0:16:46
kim0        machine: 116:47
kim0        relations:16:47
kim0          db: {state: up}16:47
kim0        state: started16:47
kim0  mywebsite:16:47
kim0    formula: local:drupal-916:47
kim0    relations: {db: mydb}16:47
kim0    units:16:47
kim0      mywebsite/0:16:47
kim0        machine: 216:47
kim0        relations:16:47
kim0          db: {state: up}16:47
kim0        state: started16:47
kim0      mywebsite/1:16:47
kim0        machine: 316:47
kim0        relations:16:47
kim0          db: {state: up}16:47
kim0        state: started16:47
kim02011-07-25 16:46:18,907 INFO 'status' command finished successfully16:47
kim0"mywebsite" now has two service unit instances mywebsite/0 and mywebsite/116:47
kim0the new node is running machine "3" which is ec2-50-16-175-35.compute-1.amazonaws.com16:47
kim0which means .. visiting http://ec2-50-16-175-35.compute-1.amazonaws.com/ensemble/ .. You should see the second drupal instance16:49
kim0of course if you'd like to further scale out .. you just keep add'ing more units .. that's all it takes16:49
kim0The mysql formula supports adding "slave" nodes16:49
kim0so you can scale your DB via adding more slave nodes16:50
ClassBotThere are 10 minutes remaining in the current session.16:50
kim0alright .. time flies when you're having fun16:50
kim0What is really cool is that formulas can be written in ANY language16:50
kim0so bash, php, python .. whatever you fancy!16:50
kim0I will vim open the drupal formula in the screen session16:51
kim0Let me take any questions quickly16:51
ClassBotrwh asked: is there already a formula repo, or is this a service that's planned for the future?16:51
kim0great question16:51
kim0right now .. You can see formulas over at https://code.launchpad.net/principia16:52
kim0however a more integrated version is coming very soon ..16:52
kim0where you'll be able to search and install formulas just like you do with ppas16:52
ClassBotTeTeT asked: how much effort is it to write these relations? Isn't this more complicated than configuring the services themselves, e.g. how many units do I need to have so the initial investment in Ensemble pays off16:53
kim0Great question as well ..16:53
kim0It is pretty simple to write those relations16:53
kim0I just opened the db-relation-changed script for my drupal formula16:54
kim0as you can see it's a pretty simple bash script16:54
kim0that gets the database configuration details from ensemble .. then simply uses "sed" to render a template configuration file16:54
kim0I really like the fact that I do not have to wrestle with learning a new DSL configuration language16:55
ClassBotThere are 5 minutes remaining in the current session.16:55
kim0I'll use the remaining minutes to let you know that you can find the Ensemble community at16:55
kim0all developers, formula writers and community members hang out there16:55
kim0our goal is to cover all of free software with Ensemble formulas16:56
kim0such that you're able to ensemble deploy whatever you fancy .. just like you apt-get install whatever you want today16:56
kim0Please join in .. and start writing and contributing formulas16:56
kim0it's very easy .. there is no special language to learn, and the community is extremely helpful16:57
kim0you can ask me (or others ) any questions in #ubuntu-ensemble (or #ubuntu-cloud) at any time16:57
kim0I hope this was useful and fun .. see you in a next session16:57
kim0Next session will be for cloud-init .. an Ubuntu originated cloud technology16:58
kim0the two sessions afterwards will be for Orchestra and its integration with Ensemble .. both great technologies being developed this cycle16:58
kim0and the final session will be for Eucalyptus v3 .. I hope you will enjoy the first day of UCD16:58
kim0Good bye16:58
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Cloud Days - Current Session: Introduction to Cloudinit - Instructors: koolhead17
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/25/%23ubuntu-classroom.html following the conclusion of the session.17:00
koolhead17hello everyone17:00
=== medberry is now known as med_out
koolhead17cloud-init is the Ubuntu cloud technology that enables a cloud instance to bootstrap itself and customize 17:01
koolhead17we can do many operations on our instance before it boots up17:02
koolhead17its like adding an extra layer with more contents17:04
koolhead17lets talk about an example17:04
koolhead17you want an instance to have say apache installed automatically every time it boots up17:05
koolhead17you can simply use17:05
koolhead17 - apache217:05
koolhead17and if you are using amazon ec2 web interface you can pass the parameter during launching the instance.17:05
koolhead17cloud-init works for openstack as well as eucalyptus17:06
koolhead17i will try to show you demo of the same if possible at the end17:06
koolhead17lets say you want to boot your instance with a specific timezone everytime the instance boots17:07
koolhead17you can simply define that using17:07
koolhead17timezone: US/Eastern17:07
koolhead17parameter in the file which you will be passing17:08
koolhead17now lets move to the example file we have http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt17:10
koolhead17line 7 apt_update: false17:11
koolhead17which means the parameter will be passed and at time instance launches  automatic update will not happen17:11
koolhead17you can change it to apt_update: true and pass it during booting instance to enable it17:12
koolhead17similarly to enable/disable we have "apt_upgrade"17:12
koolhead17in the next line we can see it mentions about adding of repository. you can add your custom repository as well.17:13
koolhead17doing this will save some bandwidth in data-centre like environment :)17:14
koolhead17i will skip some of the examples from there :D17:14
koolhead17you can even run commands17:16
koolhead17line 20517:16
koolhead17- echo us.archive.ubuntu.com > /etc/hosts17:16
koolhead17you can run commands like :17:17
koolhead17runcmd: - [ ls, -l, / ]17:17
koolhead17one of the feature which i find most exciting and am fighting with it is debconf_selections: |17:18
koolhead17byobu_by_default: system17:19
koolhead17enables byobu to all the uses by default once they login17:20
koolhead17the availability of cloud-init technology on all the cloud environment am working (openstack, eucalyptus, ec2)17:22
koolhead17you can find more info and detailed instruction at kim0 blog http://foss-boss.blogspot.com/search/label/cloud-init17:23
=== Hugo is now known as Guest86346
koolhead17cloud-init comes pre installed if you are using ec217:26
koolhead17in case of openstack you need to install the package at time of preparing your cloud image17:26
koolhead17you can use euca tools in case of eucalyptus and openstack17:28
koolhead17on ec2 you can use web interface as well as via command line17:29
koolhead17so lets recap what all we have discussed so far17:29
koolhead17Some of the things cloud-init configures are:17:30
koolhead17setting hostname17:30
koolhead17generate ssh private keys17:30
koolhead17*which i forgot covering earlier :(17:31
koolhead17adding ssh keys to user's .ssh/authorized_keys so they can log in17:31
koolhead17setting up ephemeral mount points17:31
koolhead17to execute a command17:32
koolhead17automatic package update and upgrade17:32
koolhead17timezone setup17:33
koolhead17package installation17:33
koolhead17like apache217:33
koolhead17you can also see https://help.ubuntu.com/community/CloudInit17:34
koolhead17you people can take break now17:35
koolhead17the next session is about Orchestra17:35
koolhead17and it will be presented by 2 members from the server engineering team17:35
koolhead17it would have been more interesting with the demo which am unable to do :(17:37
koolhead17 /msg classbot !q17:40
=== daker is now known as daker_
ClassBotGuest32626 asked: is cloud-init available for other linux distros?17:41
koolhead17Guest32626: it is available for Amazon's linux.17:42
koolhead17which is similar to fedora ..17:42
koolhead17it has been adopted by Amazon17:42
koolhead17and can easily be ported to other linux'es17:42
ClassBotGuest86346 asked: Is that possible to configure route table after retrieving metadata with cloud-init ?17:44
koolhead17yes its very much possible with the script :)17:45
koolhead17one more important thing17:46
koolhead17we all are available at #ubuntu-cloud , our official cloud support channel for ubuntu. join us and hangout with ys17:46
koolhead17and 1 more thing the mega session is coming nest17:47
koolhead17about Orchestra and Ensemble .. Two pillar technologies for Ubuntu server in 11.1017:47
koolhead17Good bye .. and that's all :)17:48
ClassBotThere are 10 minutes remaining in the current session.17:50
ClassBotThere are 5 minutes remaining in the current session.17:55
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Cloud Days - Current Session: Orchestra and Ensemble (part1) - Instructors: smoser
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/25/%23ubuntu-classroom.html following the conclusion of the session.18:00
smoserOK, lets get started18:00
smoserHi, I'm Scott Moser, an Ubuntu Developer working on the Ubuntu Server Team.18:01
smoserIf you're not familiar with the way classroom works, please see https://wiki.ubuntu.com/Classroom/ClassBot18:01
smoserhint: join #ubuntu-classroom-chat18:01
smoserMuch of the Server Team's focus this cycle has been on deployment.  That deployment really falls into 2 different categories18:02
smoser * ensemble: deploying and managing services on top of existing Ubuntu installs (or new cloud instances)18:02
smoser * orchestra: deploying Ubuntu onto "bare metal".18:02
smoserA few weeks ago, it was decided that we wanted to make Orchestra a "provider" for Ensemble.18:03
smoserWhat this means is that we wanted to allow Ensemble to deploy and manage "bare metal" machines the same way that it originally knew how to manage EC2 instances.  Andres [RoAkSoAx] will talk more about that in the next session.18:03
smoserLike anybody else, we don't have enough hardware, and even less hardware with remotely controllable power switches and fast networks.18:03
smoserIn order to get ourselves an environment that we could develop the "orchestra provider" for ensemble I put together "cobbler-devenv".18:03
smoserThat can be found at http://bazaar.launchpad.net/~smoser/+junk/cobbler-devenv/files/head:/cobbler-server ,18:04
smoseror via 'bzr branch lp:~smoser/+junk/cobbler-devenv'18:04
smosercobbler-devenv allows you to very easily set up a cobbler development environment using libvirt.  That environment:18:04
smoser * includes an Orchestra server and 3 "nodes"18:04
smoser * includes a dhcp server and dns server18:04
smoser * will not interfere with physical networks *or* other libvirt networks.18:04
smoserThe code there is currently focused on deploying cobbler and an ensemble provisioning environment, but it not much is really specific to that purpose.18:05
smoserIf you've not already done so, go ahead and open the cobber-server url above or branch it.  The HOWTO explains how to set all this up.  I'll largely walk through that here with some more explanation as to what is going on than is in that file.18:05
smoseranyone have any questions so far?18:06
smoserok then.18:06
smoser== Some configuration first ==18:06
smoseras prereqs, you'll need to18:07
smoser$ apt-get install genisoimage libvirt-bin qemu-kvm18:07
smoserIn  order to interact with libvirt, you have to be in the libvirtd group,  and in order to use kvm acceleration you have to be in 'kvm' group.  So:18:07
smoser $ sudo adduser $USER kvm18:07
smoser $ sudo adduser $USER libvirtd18:07
smoser$ sudo apt-get install python-libvirt18:08
smoserAlso, note that libvirt does not work when images are in a private home directory.  The images must be viewable to the libvirt user.18:08
smoserthis cost me a fair amount of time once trying to debug why my VMs were getting "permission denied" when they were clearly readable (but the path to the images was not)18:09
smoserthe first step in the HOWTO document is to build a cobbler server.  To do that, we utilize build-image like:18:09
smoser$ ./build-image -vv --preseed preseed.cfg oneiric amd64 8G18:09
smoser# oops, we have somewhat changed sections of my talk here, we're now in18:10
smoser== building the Orchestra server VM ==18:10
smoserplease feel free to pipe-in with questions if you have them18:10
smoserNote, the above command won't actually work right now. :-(18:10
smoserbug 815962 means that that doesn't currently work, and wont until the next upload of debian-installer.18:10
smoserit should be fixed in 24 hours or so, though18:11
smoserThat command will take quite a while to run, probably heavily based on network speed as it is doing a network install.  Locally, with my local mirror, a natty build just took 12 minutes.18:11
smoserYou can have it use a mirror by editing preseed.cfg.18:11
smoserIt wraps all the following:18:11
smoser * grab the current mini-iso for oneiric18:12
smoser * extract the kernel and ramdisk using isoinfo18:12
smoser * repack the ramdisk so it has 'preseed.cfg' inside it, and set up the 'late_command' in installer to do some custom configuration for us (see 'late_command.sh').18:12
smoser * after install is done, boot the system again to do some final config that 'late_command' layed down.18:12
smoserIt does this via kvm and the kvm user net, so you can build this entirely without libvirt or root access.18:12
smoserI'm particularly proud of not needing root for this.18:13
smoseror any network access other than to the archive.18:13
smoserThis basic setup could be used for automated building of virtual machines (as it is here)18:13
smoserThe result is that you now have a disk image that is ready to boot.  We've built the Orchestra virtual server that will be in charge of provisioning the nodes.18:13
smoser$ ls -lh natty-amd64.img18:14
smoser-rw-r--r-- 1 libvirt-qemu kvm 1.3G 2011-07-25 12:39 natty-amd64.img18:14
smoserNow we just we need to set up a libvirt network, and put that image on it.18:14
* smoser pauses a bit for questions18:15
smosersees that there are some and is looking18:15
ClassBotTeTeT asked: do we setup a virtual environment to boot bare metal servers and install them?18:16
smoserTeTeT, sorry to be unclear18:16
smoserthe goal of cobbler-devenv is to have a purely virtual environment that models a typical hardware setup18:16
smoserwe'll end up with a cobbler server vm, and 3 "node" vms attached to a network where the cobbler server will be able to turn on the nodes and control their pixee boot via tftp18:17
ClassBotalexm asked: is it necessary to have 8G for the server? i _just_ have 8G in total in my desktop18:17
smoseralexm, I used 8G, though it is a bit large.  as you can see above, the total space *used* will be much less.18:18
smoserqcow is a sparse format.  I would guess you can get buy with 4G, but with all the installed components in the server, much less is going to be really tight.18:18
ClassBotm_3 asked: so './build-image -vv --preseed preseed.cfg natty amd64 8G' should work, but oneiric won't?18:19
smoserbuild-image with 'natty' "should work"18:19
smoseri verified the install went fine, but ran into bug https://launchpad.net/bugs/80426718:19
smoserthat caused me to not be 100% tested that path today18:19
ClassBotTeTeT asked: so if it's for a virtual environment, this means a non cloud environment, as otherwise installing OS is a non-issue, at least with euca and openstack?18:20
smoserTeTeT, right. it is for a virtual environment, and "non-cloud"18:20
smoserthe initial reason I developed this was to ease the development of the "orchestra provider" for ensemble18:21
smoserthrough that provider, ensemble will be able to install "bare metal" systems.18:21
smoserwe're just creating a virtual network that would be like a physical network and sytems you would have access to, but its easier to work with the virtual.18:21
smoserthe primary goal of "bare metal provisioning" for ensemble, is actually to provision a cloud18:22
ClassBotkim0 asked: What would it take to install real physical boxes out of that dev-env18:22
smoserto install real machines off of the cobbler vm, you'd have to set bridging up differnetly than i have it, and have your dhcp server point next-server to the cobbler system18:23
smosermoving on a bit18:23
smoser== Setting up Libvirt resources ==18:24
smoserplease feel free to ask questions. if you say 'smoser' in #ubuntu-classroom-chat i'm more likely to see it.18:24
smoserNow, back at the top level directory of cobbler-devenv we have a 'settings.cfg' file [http://bazaar.launchpad.net/~smoser/+junk/cobbler-devenv/view/head:/settings.cfg]18:24
smoserThe goal is that this file defines all of our network settings.  It has sections for 'network', 'systems' (static systems like the Orchestra Server) and 'nodes'.18:24
smoserthe only static system we have is 'cobbler', but there could be more described there.18:25
smoserWe create the libvirt resources by running './setup.py' (which should probably be renamed to something that does not look like it came from python-distutils)18:25
smoserthat script interacts with libvirt via python bindings18:26
smoser$ ./setup.py libvirt-setup18:26
smoserThat will put some output to the screen indicating that it created a 'cobbler-devnet' network, a 'cobbler' domain, and 3 nodes named 'node01' - 'node03'.18:26
ClassBotskrewler asked: Is support for Chef in the roadmap?  Or is it possible to substitute puppet for another CM tool, like cfengine or Chef?18:27
smoserskrewler, well, there is no real CM tool involved here.  The initial goal was to get Ensemble up, but it would take very little changes to make the setup able to use chef, cfengine or puppet.18:28
smoserThose things wouldprimarily be configured through cobbler kickstart templates (preseed templates).18:28
smoseri'm not really interested in that, though, this was really just to get a test environment up for ensemble, but it definitely could be utilized to test out other managmeent bootstrapping and management.18:29
smoserso.... above, we created the cobbler-devnet and 3 nodes18:30
smoserThe libvirt xml is based on the libvirt-domain.tmpl and libvirt-network.tmpl files, which are parsed as Cheetah template files.18:30
smoserThe end result is that we have a 'cobbler-devnet' network at, and has statically configured dhcp entries for our cobbler server and 3 nodes, so that when they DHCP they'll get set IP addresses.18:30
smoserthe cobbler-devnet network looks something like:18:31
smosernotice how we have MAC addresses in the network setup that will match with our mac addresses in the nodes18:31
smosernow our network is setup, so lets put the cobbler server on it18:32
smoserWe build a qcow "delta" image off of the pristine server image we built above so we can easily start fresh.18:32
smoser$ virsh -c qemu:///system net-start cobbler-devnet18:32
smoserNetwork cobbler-devnet started18:32
smoser$ qemu-img create -f qcow2 -b cobbler-server/natty-amd64.img  cobbler-disk0.img18:32
smoser$ virsh -c qemu:///system start cobbler18:32
smoserDomain cobbler started18:33
smoserThat will take some time to boot, but after a few minutes you should be able to ssh to the cobbler system using its IP address:18:33
smoser $ ssh ubuntu@
smoser(the password is 'ubuntu', obviously you should change that)18:33
smoserWhile you're there, you can verify that 'cobbler' works by running:18:33
smoser $ sudo cobbler list18:34
smoserthat should show you that there were some images imported for network install of Ubuntu.18:34
smoserAt this point You can also get to the web_ui of cobbler at: and poke around there.18:34
smosergenerally, we've got a fully functional cobbler server just waiting for something to install!18:34
smoserThen, back on the host system we populate the cobbler server with the 3 nodes that we've created.18:34
smoser$ ./setup.py cobbler-setup18:35
smoserThat uses the cobbler xmlrpc api to set up our nodes.  Now, a 'cobbler list' will show our nodes.18:35
smoserIt also configures those nodes to be controllable by the "virsh" power control (that is like ipmi, but for virtual machines).  We've got one more thing to do though before that can happen.18:35
smoserOn the host system we need to run:18:35
smoser $ socat -d -d \18:35
smoser     TCP4-LISTEN:65001,bind=,range=,fork \18:35
smoser     UNIX-CONNECT:/var/run/libvirt/libvirt-sock18:35
smosersocat is a useful utility, and the above command tells it to listen for ip connections on port 65001 and forward those to the unix socket that libvirt listens on.18:35
smoserbasically this makes libvirtd listen on a tcp socket18:36
smoserBefore you go screaming how horrible that is (it would be)18:36
smosernotice that We've limited the host IP to the IP range of the guest network, and told it to only listen on the guest's interface, so it is mildly secure. Definitely much better than just listening on all interfaces.18:36
smoserOnce that is in place, you can turn the nodes on and off via the cobbler web_ui.18:37
smoserBasically, at this point, we have modeled a lab with IPMI power control of node systems from the cobbler system.18:38
smosernodes can be turned on and off, and their network boot controlled via the cobbler vm system.18:38
smoserI should have pointed out above, that our libvirt xml for the Node systems has them network booting.18:38
smoserIf you configure 'network-boot' for a node, and then start it, it should begin to install itself.18:38
smoserYou can try that out, and then (from the host system) watch the install with:18:38
smoser $ vncviewer $(virsh vncdisplay node01)18:39
smoserIt should actually walk through a fully automated install.18:39
smoserWell, thats basically all I have.18:40
smoser=== Summary ===18:40
smoserafter all of that, what we have is a well configured network with a single cobbler server that is ready to install some nodes.18:40
smoserThe nodes actually have functional static-dhcp addresses and can communicate with one another via hostnames (node01, cobbler, node02...)18:41
smoserIn the next session, Andreas will talk about how we can use ensemble to control the cobbler server and provision the nodes.18:42
smoserThat way, ensemble can control our bare metal servers just like it can request new EC2 nodes.18:42
smoser(here, we're just pretending that those VMs are real hardware, but ensemble doesn't actually know the difference)18:43
smoserkim0, you could have executed examples yesterday...18:44
smoserso, yeah, i hope you can tomorrow.18:44
smoserif you want to just play with cobbler some, this is a really nice way to see how it fits all together18:44
smoserwithout having 2 or 3 spare systems sitting around.18:44
smoseri know that that was a big barrier to entry for me.18:45
ClassBotkim0 asked: So Ensemble would request powering on the hardware and installing it, then orchestrating it .. Is that advantageous to having all boxes installed and "waiting" for Ensemble ?18:45
smoserw've not shortcutted that, but you could.18:45
smoserin the real world scenario, though, the provisioning of a node will occur once ensemble is done with it.18:46
smoserthat ensures that they're "clean".18:46
smosersave some of your questions for RoAkSoAx but i'm guessing that end to end on cable modem speed you could have a cobbler vm built, and then a node deployed on it via ensemble in 3 hours or so at this point.18:47
ClassBotkim0 asked: Is installing the cobbler server planned as a CD boot option18:48
smoserkim0, i'm not sure how it will be exposed, but yeah, the goal is to make that *very* easy.18:48
smoseralexm said: smoser: note that cache=unsafe in build-image in unsupported in maverick's qemu, i just changed it with writeback18:49
smoserThanks alexm . 'writeback' is the right value there.18:49
ClassBotThere are 10 minutes remaining in the current session.18:50
smoserin minutes before this session i tried to see if i could get this all to go inside a ec2 guest18:51
smoserit "should work", but something was going wrong.18:51
ClassBotThere are 5 minutes remaining in the current session.18:55
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Cloud Days - Current Session: Orchestra and Ensemble (part2) - Instructors: RoAkSoAx
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/25/%23ubuntu-classroom.html following the conclusion of the session.19:00
RoAkSoAxalright then lets continue with the presentation19:00
RoAkSoAxwrong channel19:02
RoAkSoAx5:01 <+RoAkSoAx> Hi all. My name is Andres Rodriguez, and I'm an Ubuntu Developer working on the Ubuntu Server Team as well.19:02
RoAkSoAx15:02 <+RoAkSoAx> As Scott already mentioned today, we have been working on getting Ensemble to work with Orchestra. We've been using smoser's devenv to achieve this result. Today I'm going to show you how this work can be tested as a  proof of concept, as this is still  work in progress.19:02
RoAkSoAx15:02 <+RoAkSoAx> But first, lets learn a little bit more about the idea behind Orchestra and Ensemble integration19:02
RoAkSoAx15:02 <+RoAkSoAx> The main idea behind this was to basically use Ensemble with Orchestra/Cobbler the same way it's been used with ec2. However, on ec2 we can request instances easily and add more and more, but in Orchestra/Cobbler we19:02
RoAkSoAx can't. This is a limitation, however the pproach taken in this case, was to simply pre-populate the Orchestra server with "systems" (in terms of Cobbler). A system mis a physical system that is somewhere in the network  and that cobbler can deploy. So, we have a list of available machine ready to be deployed via PXE.19:02
RoAkSoAx15:02 < JohnSGruber> RoAkSoAx: do you want to paste this in the classroom?19:02
RoAkSoAxSo we could say that we will have to do two things with ensemble 1. Bootstrap and 2. Deploy, in the same way we would do with ec2.19:03
RoAkSoAxBootstrapping is when we tell ensemble to start creating the whole environment. In this case, bootstrapping means starting a machine to be the zookeeper machine, which will interface between a client machine from where we are issuing commands, and the provider (Orchestra), to deploy machine and create the relations between them.19:04
RoAkSoAxThe process here was to simply select a "system" provided by Orchestra/Cobbler. This system will then be powered on by interfacing with the power management interface the hardware has configured  (IPMI, WoL, ect, virsh), and will turn it on. When this machine boots up, it will find a PXE server on the network (Cobbler) and will start the installation process. Once the machine has finished installing, it will use cloud-init to install ensemble19:04
RoAkSoAxIn case of the development environment, we use virsh as the power management interface19:04
RoAkSoAxAs smoser already explained, the cobbler devenv provides machines that are ready to be deployed via PXE19:05
RoAkSoAxwhen we bootstrap with ensemble, it simply tells cobbler to start a machine19:05
RoAkSoAxcobbler uses virsh to start it19:05
RoAkSoAxand when the machine starts it searches for a PXE server, and installs the OS19:05
RoAkSoAxSo, as mentioned, the bootstrap process will start a new machine that we are gonna call the zookeeper19:06
RoAkSoAxonce the zookeeper is up and running, we can start deploying machines19:06
RoAkSoAxm_3: will get to that in a min ;)19:07
RoAkSoAxSo, when deploying, Ensemble will tell the zookeeper to deploy a machine with an specific service. The zookeeper will talk to the orchestra server in the same way it did when bootstrapping and will deploy a machine. It will also use cloud-init to install everything necessary to deploy the service.19:07
RoAkSoAxNow, since obviously ec2 is different from Orchestra/Cobbler we needed to make some changes in the approch taken to make things work (such as provide the meta-data for cloud-init). We needed a few things:19:08
RoAkSoAx1. Provide methods in ensemble to interface with Cobbler using its API19:09
RoAkSoAx2. Provide a custom preseed to be used when deploying machines through ensemble.19:09
RoAkSoAx3. Provide a method to pass cloud-init meta-data, and be populated before first boot so that cloud-init can do its thing.19:09
RoAkSoAxSo, how did we achieve this19:09
RoAkSoAx1. As already explained, ensemble uses cobbler as a provider communicating with it via the cobbler API.19:09
RoAkSoAx2. Since ec2 instances a VM really quick, it was easy to pass all the necessary values through cloud-init, however, in our case, we needed to do somthing similar, and the conclusion was to do it via a modified preseed to deploy whatever it was needed the same way19:10
RoAkSoAx3. We figured out a method to pass the cloud-init meta-data through the preseed19:11
RoAkSoAxso basically the change sin Cobbler were to provide a custom preseed to deploy the OS19:11
RoAkSoAxthis preseed contains what we call a late_command19:12
RoAkSoAxthis late_command will execute a script that will generate the cloud-init meta-data so after first boot, cloud-init will do its thing19:12
RoAkSoAxso what we did is to generate the cloud-init meta-data with ensemble as it was always done, but, we had to figure out how to do it to the preseend19:13
=== gbl is now known as Guest14530
RoAkSoAxhere, we generated text that was later enconded in base64.19:14
RoAkSoAxThis text was basically a shell script containing the information to populate cloud-init's meta-data19:14
RoAkSoAxso the late command in reality was to decode the base64 code and then, wirte the script and execute it19:15
RoAkSoAxthis decoding and writing was done by the preseed, right after finishing installing the system and before booting19:15
RoAkSoAxso when the machine restarted, cloud-init would do its thing19:15
RoAkSoAxso that was done by making ensemble interface with cobbler, and once the late command was generated, ensemble told cobbler "This is your late command" and cobbler simply executed it19:16
RoAkSoAxonce the machine finished installing, we had a fully functional zookeeper (or service)19:16
RoAkSoAxso basically, we wated to achive the same as with ec2, but we just had to figure out how to do it with the preseed19:17
RoAkSoAxand now, it works in a very similar way19:17
RoAkSoAxso the only things to consider were to 1. start a machine. 2. deploy the machine using the preseed. 3. ensure to pass the late_command19:17
RoAkSoAxand this way we would simulate the way how instnaces and cloud-init data is passed to instances in the cloud19:18
RoAkSoAxother than that, ensemble works pretty much exactly the same as it would with ec219:18
RoAkSoAxbut using orchestra19:18
RoAkSoAxNow, another change that was done is that ensemble when working on ec219:18
RoAkSoAxit used S3 to store some information that was using by ensemble to identify machines and place the formula meta-data19:18
RoAkSoAxinstead, we used a WebDav service with the apache2 servcer installed by cobbler19:19
RoAkSoAxhere, instead of obtaining and storing files on S3, we use the Orchestra server as storage for ensemble19:19
RoAkSoAxbased on those considerations, pretty much had to ensure that the interaction between the cobbler API and ensemble provided results the way its done with ec219:20
RoAkSoAxso how can we really test this with the development environment19:20
RoAkSoAxbut before,19:20
RoAkSoAxm_3: does this answer your question?19:21
RoAkSoAxI'll move on19:21
RoAkSoAxWith smoser's cobbler devenv we can certainly simulate a physical deployment using ensemble19:22
RoAkSoAxthe good thing is that the devenv will setup everything necessary from the orchestra side of things19:22
RoAkSoAxbut, I'll give and overview of what will orchestra do very soon19:22
RoAkSoAx1st. We would need to install orchestra-server, which will isntall cobbler and cobbler-web19:22
RoAkSoAxwith that, we would need to configure the webdav so that we have storage up and running19:23
RoAkSoAx(remember, this is already done by the cobbler-devenv)19:23
RoAkSoAxhow did we do this:19:23
RoAkSoAx=== Setting up file storage ===19:23
RoAkSoAx1. Enable Webdav19:23
RoAkSoAxsudo a2enmod dav19:23
RoAkSoAxsudo a2enmod dav_fs19:23
RoAkSoAx2. Write config file (/etc/apache2/conf.d/dav.conf)19:23
RoAkSoAxAlias /webdav /var/lib/webdav19:23
RoAkSoAx<Directory /var/lib/webdav>19:23
RoAkSoAxOrder allow,deny19:23
RoAkSoAxallow from all19:23
RoAkSoAxDav On19:23
RoAkSoAx3. Create formulas directory:19:23
RoAkSoAxsudo mkdir -p /var/lib/webdav/formulas19:23
RoAkSoAxchown www-data:www-data /var/lib/webdav19:24
RoAkSoAxsudo service apache2 restart19:24
RoAkSoAxnow, we need to pre-populate cobbler with all the avialable systems and provide it with a power management interface to be able to start a physical mnachine19:24
RoAkSoAxas previously explained, cobbler devenv uses virsh to simulate this behaviour19:25
RoAkSoAxhoweve,r in cobbler, we needed to know two things19:25
RoAkSoAx1. How do we know when a system is available 2. How do we know when the system has already been used and no longer available19:25
RoAkSoAxfor this, we had to look into cobbler's management classes concepts19:25
RoAkSoAxin this case we are using two, foo-available and foo-acquired. As the name says, one will be used to identify when a system is available to be used by ensemble, and the other one when the system has already been acquired by ensemble and might be in the process of bootstrapping or deploying a service, or even installing the OS19:27
RoAkSoAxbut, in cobbler terms, how can we add management classes and systems?19:27
RoAkSoAx=== Setting up cobbler ===19:27
RoAkSoAx1. Add management classes19:27
RoAkSoAxsudo cobbler mgmtclass add --name=foo-available19:27
RoAkSoAxsudo cobbler mgmtclass add --name=foo-acquired19:27
RoAkSoAx2. Add systems19:27
RoAkSoAxsudo system add --name=XYZ --profile=XYZ --mgmt-classes=foo-available --mac-address=AA:BB:CC:DD:EE:FF19:27
RoAkSoAxBasically, a system is a definition for a physical machine using a OS profile, and what mangement class to use at first19:27
RoAkSoAxthe mprofile is no other than the OS that will be installed in that machine19:28
RoAkSoAxand the management class has already been explained19:28
=== med_out is now known as medberry
RoAkSoAxof course you will have to configure the power management interface accordingly19:28
RoAkSoAxbut in the cobbler-devenv has alreayd been done19:28
RoAkSoAxso basically, we now have a Orchestra/Cobbler server up and running and we have configured it with systems, mgmtclasses and the file store19:29
RoAkSoAxso it is time for us to install and configure ensemble to use our cobbler server19:29
RoAkSoAxin this case, we are going to use the cobbler-devenv19:29
RoAkSoAxhowever, you will notice that you can simply chnage it to be used by physical machines19:29
RoAkSoAxif you already have an orchestra server up and rynning and preloaded with systems19:30
RoAkSoAxso first, we need to obtain the branch of ensemble that has orchestra support19:30
RoAkSoAxNOTE: This branch contains code that is under development and is still buggy19:30
RoAkSoAx 1. Obtain the branch:19:30
RoAkSoAxbzr branch lp:~ensemble/ensemble/bootstrap-cobbler19:30
RoAkSoAxcd bootstrap-cobbler19:30
RoAkSoAxnow we need to create an environments.yaml file for ensemble19:31
RoAkSoAxwe do this as follows:19:31
RoAkSoAx 2. Create the environments file (~/.ensemble/environments.yaml)19:31
RoAkSoAx   orchestra:19:31
RoAkSoAx      type: orchestra19:31
RoAkSoAx      orchestra-server:
RoAkSoAx      orchestra-user: cobbler19:31
RoAkSoAx      orchestra-pass: cobbler19:32
RoAkSoAx      admin-secret: foooo19:32
RoAkSoAx      ensemble-branch: lp:~ensemble/ensemble/bootstrap-cobbler19:32
RoAkSoAx      acquired-mgmt-class: foo-acquired19:32
RoAkSoAx      acquired-mgmt-class: foo-available19:32
RoAkSoAxnote that I'm already using the values for the cobbler-devenv19:32
RoAkSoAxsuch as orchestra-server IP address19:32
RoAkSoAxuser/pass for cobbler19:32
RoAkSoAxthe branch we need19:32
RoAkSoAxand the management classes19:32
RoAkSoAxtypo in last line19:33
RoAkSoAxshould be:19:33
RoAkSoAx      available-mgmt-class: foo-available19:33
RoAkSoAxso once this is done, and we have setup the cobbler-devenv correctly19:33
RoAkSoAxwe can start bootstrapping the zookeeper and then deploying the machines19:33
RoAkSoAxso the first step, and from the branch we have obtained, we do the follwoing:19:34
RoAkSoAxPYTHONPATH=`pwd` ./bin/ensemble bootstrap19:34
RoAkSoAxthis will bootstrap the zookeeper19:34
RoAkSoAxit will take time for it to install and deploy the zookeeper running19:34
RoAkSoAxit would probbaly take esveral minutes19:34
RoAkSoAxso I will containue explaining what it needs to be done19:34
RoAkSoAxso, when the zookeeper is up and running and cloud init has done its thing, we need to workaround something given that we just came into an error in the code19:35
RoAkSoAxthatis being examined19:35
RoAkSoAxbut it is simple and doens't actually affect the code19:35
RoAkSoAxso we need to connect to the zookeeper machine (through ssh, or any ither method19:35
RoAkSoAxand sudo the following (in the zookeeper machine)19:35
RoAkSoAxsudo -i19:35
RoAkSoAxssh-keygen -t rsa19:36
RoAkSoAxthis will create public keys that are verified by the zookeeper before deploying machines19:36
RoAkSoAxhowever, note that this is a work around and will be fixed soon19:36
RoAkSoAxI'm just pointing you guys to it in case you want to test it after the session of today19:36
RoAkSoAxonce this is done19:36
RoAkSoAxwe can start deploying machine19:36
RoAkSoAxand we simply do the following:19:36
RoAkSoAxPYTHONPATH=`pwd` ./bin/ensemble deploy --repository=examples mysql19:37
RoAkSoAxthis will tell zookeeper to deploy a machine, whcih will tell cobbler to start a machine via virsh19:37
RoAkSoAxand once installed it will run late-command and populate cloud-init meta-data19:37
RoAkSoAxon first boot19:37
RoAkSoAxcloud-init will do its thing19:37
RoAkSoAxand baaam19:37
RoAkSoAxwe would have a mysql server working on a physical node19:37
RoAkSoAxand I believe that's all I have for you today19:38
RoAkSoAxI think i run over the session too fast :)19:38
RoAkSoAxanyone has any questions?19:38
RoAkSoAxm_3: well that's indeed a limitation we have in comparison to ec2 as in physical environments (and cobbler) we are relying in the power management interface to deploy machines19:41
ClassBotm_3 asked: does the cobbler instance provide a metadata server for cloud-init?19:42
ClassBotm_3 asked: reboots... how robust is everthing wrt reboots?  (In EC2-ensemble, we just typically throw instances away)19:42
RoAkSoAxm_3: now, as far as rebooting machines and keep things persistant, at the moment, we are not handling that19:43
RoAkSoAxm_3: but the first approach was to preseed all that information and use debconf populate those values19:43
RoAkSoAxm_3: and have upstart scripts initialize the services on reboot19:43
RoAkSoAxm_3: however, we discussed the possibility of actually not doing that through the preseed but rather, provide cloud-init with a command to write those persistant values so on reboot they can be used19:44
RoAkSoAxm_3: you're welcome19:45
RoAkSoAxanyone any more questions?19:45
RoAkSoAxalright I guess there's not19:47
RoAkSoAxthank you all19:47
ClassBotThere are 10 minutes remaining in the current session.19:50
ClassBotalexm asked: RoAkSoAx: will ensemble/orchestra be in ubuntu-server manual for oneiric? a quick start guide, for instance19:52
RoAkSoAxalexm: I surely hope so! I guess that will depend how far we can get with this in the development cycle, but I'm confident it would19:52
ClassBotThere are 5 minutes remaining in the current session.19:55
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Cloud Days - Current Session: Eucalyptus 3: cloud HA and ID management - Instructors: nurmi
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/25/%23ubuntu-classroom.html following the conclusion of the session.20:00
nurmiHello all, and thank you very much for attending this session!20:01
nurmiToday, we're going to be discussing some new features of Eucalyptus 320:02
nurmiWhile there are quite a few, two of the most substantial are implementations of high availability and user/group identity management20:02
nurmiWe'll start with a discussion of Eucalyptus HA, and then switch to ID management next20:03
nurmiEucalyptus is designed as a collection of services which, when stitched together, form a distributed system that provides infrastructure as a service20:03
nurmiRoughly, eucalyptus services are organized in a tree hierarchy20:04
nurmiAt the top of the tree, we have components (Cloud Controller, Walrus) that are directly accessed by users20:04
nurmiIn the middle, we have a Cluster Controller and Storage Controller which set up/manage virtual networks and storage (EBS) respectively20:05
=== ghost is now known as Guest64902
nurmiand and the bottom of the tree, we have Node Controllers which control and manage virtual machines20:05
nurmiIn a nutshell, this collection of services provide users the ability to provision and control virtual infrastructure components that, within eucalyptus, we refer to as 'artifacts'20:06
nurmiFor example, virtual machines, virtual networks, and cloud storage abstractions (EBS volumes and S3 buckets/objects)20:06
nurmiThe design of Eucalyptus HA creates a distinction between the cloud service itself (eucalyptus components), and the artifacts that are created/managed by the service20:07
nurmiThe reason for this distinction is that, while the term 'High Availability' is generally meaningful20:08
nurmiThe requirements of making something 'Highly Available' varies greatly, depending on what that 'something' is20:08
nurmiIn Eucalyptus 3, we have a new architecture that provides High Availability for the cloud service itself20:09
nurmiThe architecture additionally supports adding High Availability to eucalyptus artifacts, in the future20:09
nurmiSo, the core design of Eucalyptus HA is as follows20:10
nurmiEach Eucalyptus component can run in 'non-HA' mode, exactly as it does today20:10
nurmiThen, at runtime, each component service can be made highly available by adding an additional running version of the component, ideally on a separate physical system20:11
nurmiThis results in a basic 'Master/Slave' or 'Primary/Secondary' mode of operation, where the Eucalyptus HA deployment is resilient to (at least) a single point of failure (for example, machine failure)20:12
nurmiAt any point in time, when running in HA mode, a component is either in 'Primary' or 'Secondary' mode20:13
nurmiany component in 'Secondary' mode is running, but is inactive until it is made Primary20:13
nurmiNext, each component, and the system as a whole, is designed to keep 'ground truth' about artifacts as close to the artifacts as possible20:14
nurmiFor example, all canonical information about virtual machine instances is stored on the node controller that is managing that VM20:14
nurmiand all canonical information about virtual networks that are active is stored with the Cluster Controller that is managing that network20:15
nurmiWhen a eucalyptus component becomes active, then20:15
nurmiwhich happens when the component first arrives, when it is 'restarted' or, when it is promoted from Secondary to Primary20:16
nurmithe component 'learns' the current state of the system by discovering what it needs from ground truth20:16
nurmiother services that are 'far' from ground truth, then, learn about ground truth from nearer components20:16
nurmiI'll use the Cluster Controller to illustrate how this design works as an example20:17
nurmiWhen a cluster controller enters into a running eucalyptus deployment, there are typically many artifacts that are currently running20:18
nurmithe very first operation that a cluster controller performs is to poll both above (Cloud Controller) and below (Node Controllers)20:18
nurmiin order to learn about the current state of all artifacts20:18
nurmiIt then uses this information to dynamically (re)create all virtual networks that need to be present in order for the currently active artifacts to continue functioning20:19
nurmiSo, whether a cluster controller is by itself (non-HA mode) and just reboots, or if a Primary cluster controller has failed and the secondary is being promoted20:20
nurmithe operation is the same: learn about ground truth and re-create a functional environment20:20
nurmiAll other HA eucalyptus components operate in a similar fashion, semantically20:20
nurmiStorage controller uses iSCSI volumes as ground truth20:21
nurmiWalrus uses shared filesystem, or a pre-configured DRBD setup for buckets/objects20:22
nurmiFinally, while the design of the software permits a simple 'no single point of failure' setup with just additional physical machines20:22
nurmi(to support Primary/Secondary model)20:23
nurmiWe also support deployments that have redundancy in the network infrastructure20:23
nurmiThis way, 'no single point of failure' can be extended to include network failures, as well, without having to alter the software/software configuration.20:23
nurmiWe've put a lot of effort into the new architecture to provide service high availability first, and hope that others will find the architecture ready to start adding HA for specific artifacts in near future releases20:25
nurmiUtilizing live migration for VM HA, utilizing HA SAN techniques for in-use EBS volume access HA, etc.20:25
nurmiThis brings us to the end of the first part of our discussion, thank you very much!  I would like to ask if there are any questions about Eucalyptus HA ?20:26
nurmiOkay ; the second part here will be led by Ye Wen, who will be talking about the new user and group management functionality in Eucalyptus 320:27
nurmiShort break until we can get '+v' for Ye20:37
wenyeHello, everyone. I'm going to continue this topic by discussing another new feature in Eucalyptus 3: the user identity management.20:39
wenyeWe have a completely new design for managing user identities in Eucalyptus 3, based on the concept of Amazon AWS IAM (Identity and Access Management).20:40
wenyeIn another word, we provide the same API as Amazon AWS IAM. Your existing scripts working for Amazon should be compatible with your new Eucalyptus 3 cloud.20:41
wenyeAt the same time, we augment and extend IAM with some Eucalyptus-specific features, to meet the need of some customers.20:42
wenyeWith IAM, you essentially partition the access to your resources (i.e. the artifacts as Dan said earlier) into "accounts"20:43
wenyeEach account is a separate name space for user identities.20:44
wenyeAccount is also the unit for resource usage accounting.20:44
wenyeWithin an account, you can manage a set of users.20:44
wenyeUsers can also be organized into groups.20:45
wenyeNote that group is a concept for assigning access permissions to a set of users. So users can be in multiple groups.20:45
wenyeBut users can be only in one account.20:45
wenyePermissions can be assigned to users and groups to control their access to the system resources.20:46
wenyeAs in IAM, you write a policy file to grant permissions.20:46
wenyeWe have a few extensions to the IAM concepts. I talk about a few here.20:47
wenyeIn IAM, you can't specify EC2 resources. For example, you can only say "allow user A to launch instance", but you can't say "allow user A to launch instance using image X".20:48
wenyeWe introduce the EC2 resources, so that you can do such things. One good use is to restrict the VM types for some users can launch instance with.20:48
wenyeAnother extension is the introduction of VM expiration or lifetime.20:49
wenyeYou can use an Eucalyptus-specific policy condition to specify a VM's lifetime or when to expire.20:49
ClassBotThere are 10 minutes remaining in the current session.20:50
wenyeThe biggest extension probably is the introduction of resource quota.20:50
wenyeWe extend the IAM policy syntax to allow the specification of resource quota. We use a special "Effect" to do that.20:51
wenyeSo you can say "Effect: Limit" in a policy, which indicates the permission is a quota permission.20:51
wenyeAnd then you can use the policy "Resource" and "Condition" to specify what resource and how large of the quota.20:52
wenyeYou can assign quota to accounts and users. And if a user is restricted by multiple quota spec, the smallest is taken into effect.20:53
wenyeWe don't have much time left. I'll briefly talk about another Eucalyptus 3 feature that is related to the identity management.20:54
wenyeThat is we enable the LDAP/AD sync in Eucalyptus 3.20:54
ClassBotThere are 5 minutes remaining in the current session.20:55
wenyeTo do that, you can simply write a LIC (LDAP Integration Configuration) and upload to the system. The identities in the system will then be synced from the specified LDAP/AD service.20:55
wenyeThere is the question of how to map the structure of LDAP tree to the IAM account/group/user model. We leave that for offline discussion. You can send us email at wenye@eucalyptus.com for more information.20:56
wenyeI'll use the remaining 3 minutes for questions.20:56
wenyeThanks everyone for attending this class!20:59
nurmiThank you all, and we look forward everyone trying out Eucalyptus 3 and letting us know what you think!21:00
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/25/%23ubuntu-classroom.html21:00
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat ||
=== yofel_ is now known as yofel
=== skrewler_ is now known as skrewler
missgawkerall done23:37

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!