/srv/irclogs.ubuntu.com/2011/06/13/#ubuntu-ensemble.txt

=== almaisan-away is now known as al-maisan
* al-maisan -> office10:33
=== kim0_ is now known as kim0
kiranm'ensemble status' always shows me connection timeout for me. Pasted the output here: http://paste.ubuntu.com/625813/12:41
niemeyerGood morning everyone13:47
kim0o/13:53
al-maisanGood morning niemeyer :)14:34
niemeyeral-maisan, kim0: Hey there!14:38
_mup_ensemble/trunk r252 committed by gustavo@niemeyer.net15:01
_mup_Merge add-python-yaml branch by Clint [r=niemeyer] [trivial]15:01
_mup_This adds the missing python-yaml package dependency.15:01
=== al-maisan is now known as almaisan-away
SpamapSHey guys. How would you all feel about moving everything except 'ensemble' out of /usr/bin ?16:48
SpamapSOtherwise... W: ensemble: binary-without-manpage usr/bin/open-port16:49
niemeyerSpamapS: Hmm16:53
niemeyerSpamapS: I think we should provide these man pages eventually16:53
niemeyerSpamapS: /usr/bin feels to me like the write place for these tools16:53
SpamapSare any of those ever useful outside of hooks though?16:53
niemeyerSpamapS: But then, I don't claim to understand the FS organization fantastically16:53
SpamapSIf no, then they're really more suitable in /usr/lib16:53
niemeyerSpamapS: Today, they're not, but I hope all of these tools are runnable out-of-band in the future16:54
niemeyerSpamapS: So we can do actions based on events not initiated by Ensemble itself16:54
SpamapSAhh, thats enough to make me think they need man pages then.16:55
SpamapSThough it might be better to have one command which tells the local agent to do things.16:56
niemeyerSpamapS: That may be a good idea.. that said, I still like how e.g. "relation-get foo" is concise16:57
SpamapSYeah, my thinking is that we're just going to dump all over the global /usr/bin namespace.16:58
niemeyerUgh.. past lunch time.. biab17:02
SpamapShmm.. should -e / --environment be changed to a global option, instead of one for the commands?17:34
SpamapSWhats the exact policy on trivial changes to the trunk? The online help for remove-relation says that it adds a relation.19:27
niemeyerSpamapS: If it's truly trivial and you're sure that's what should be in place, you can "cowboy" the change in19:34
niemeyerSpamapS: Marking it as [trivial] in the commit line19:34
niemeyerSpamapS: Many times these will also be preceded by an "on-the-fly" review request19:35
niemeyerSpamapS: E..g just post the diff somewhere and invite a review19:35
niemeyer(here)19:35
niemeyerSpamapS: Your example sounds good for a trivial, FWIW19:35
jimbakerbcsaller, hazmat, niemeyer - standup?19:43
niemeyerjimbaker: I'd rather see some _action_ happening!19:43
jimbakerniemeyer, ok19:44
koolhead17hi all19:45
niemeyerkoolhead17: Hey there19:46
niemeyerkoolhead17: Noticed you picked a formula to work on.. that's pretty cool, thanks!19:47
niemeyerkoolhead17: How're things going there so far/19:47
niemeyer?19:47
koolhead17niemeyer: :)19:47
koolhead17hoping to get it done by tonight!!19:47
koolhead17kim0: ping19:47
_mup_ensemble/set-transitions r239 committed by bcsaller@gmail.com19:48
_mup_fixed issue with merge of watch establishment in unit agent19:48
SpamapSkoolhead17: You chose the phpmyadmin formula right?20:01
koolhead17SpamapS: yes20:01
SpamapSkoolhead17: cool, its going to need some help from the mysql formula, but I already started on it.20:02
koolhead17SpamapS: yes. we need apache2/php5 as well :P20:02
SpamapSkoolhead17: no, thats all in your formula in the install hook20:02
SpamapSkoolhead17: what you need is an admin user that can read/write to/from all dbs.20:03
koolhead17SpamapS: hmm. am installing phpmyadmin along with depandency to come up with list of depandent pkgs which will go as depandant hook20:03
koolhead17on one of natty running vm on virtualboc20:04
koolhead17x20:04
SpamapSkoolhead17: right, but you want to administer a mysql database for some other app, right? like, you want to admin the database for wordpress.20:04
SpamapSkoolhead17: otherwise its just phpmyadmin with its own database20:04
koolhead17SpamapS: i thought this is a formula 4 a standalone phpmyadmin configuration. please correct me if am wrong 20:05
SpamapSkoolhead17: Its a good start. However, people can do that without ensemble.20:05
koolhead17now to create DB for a cms say drupal/joomla do we need a web-interface?20:05
SpamapSkoolhead17: ensemble is interesting because it can relate these services together. :)20:06
koolhead17:D20:06
koolhead17we can do that via appropriate hooks while writing formula for jooma/drupal/wordpress20:06
koolhead17isnt it20:06
koolhead17:P20:06
koolhead17there we can have mysql formula as depandency20:07
SpamapSkoolhead17: so you'll be able to say 'ensemble add-relation phpmyadmin:db wiki-db:db-admin .. and that will give you access to all the databases on wiki-db.20:07
koolhead17SpamapS: but i thought phpmyadmin has more to do with checking web-interface based mysql db :)20:08
koolhead17would we really need a formula like that as you suggsted!! :D20:08
SpamapSkoolhead17: usually phpmyadmin is used to do queries on a database from some other application.20:09
SpamapSkoolhead17: I don't see it being all that useful with just an empty database.20:09
koolhead17SpamapS: hmm!! your right20:10
koolhead17SpamapS: let me come up with the basic formula with hooks to install phpmyadin and then in 2nd level/round i can work on this20:10
SpamapSkoolhead17: thats a great way to do it.20:10
koolhead17it will be easier 4 me to pick up ensemble in that case :P20:10
SpamapSkoolhead17: install hook is usually first, then you add the relations. :)20:10
koolhead17SpamapS: yes. writing down all the pkgs saperately while testing a install in virtualbox :)20:11
SpamapSkoolhead17: you shouldn't need to do much more than 'apt-get install phpmyadmin'20:12
SpamapSkoolhead17: you will probably want to say '--no-install-recommends'20:12
koolhead17SpamapS: hmm 20:12
SpamapSactually no20:14
SpamapSit only suggests mysql-server20:14
* koolhead17 is running the command20:14
SpamapSso you can just do 'apt-get -y install phpmyadmin' .. it will automatically pull in php, apache, mysql client, etc. etc.20:14
koolhead17hmm20:15
koolhead17including apache2 and all20:15
koolhead17SpamapS: it also asks two options during install20:16
koolhead17 Web server to reconfigure automatically: <-- apache220:16
koolhead17 20:16
koolhead17 Configure database for phpmyadmin with dbconfig-common?  20:16
koolhead17also we need to add "Include /etc/phpmyadmin/apache.conf" in   /etc/apache2/apache2.conf to get it to work :D20:18
SpamapSkoolhead17: during an install hook, it will skip those20:18
SpamapSkoolhead17: thats what you do during the install hook.20:18
koolhead17SpamapS: hmm20:18
SpamapSkoolhead17: read the wordpress and mysql examples, they handle it20:19
koolhead17SpamapS: http://fewbar.com/2011/06/so-what-is-ensemble-anyway/20:20
koolhead17this one20:20
SpamapSkoolhead17: thats my post, not an example. ;) I mean in /usr/share/doc/ensemble/examples20:26
koolhead17SpamapS: :P20:27
koolhead17k20:27
koolhead17SpamapS: phpmyadmin is not dependant on mysql-server20:27
SpamapSkoolhead17: right, it only 'suggests' mysql-server20:28
koolhead17we need to install mysql-server too for a new installation in order to  get phpmyadmin working :D20:28
SpamapSNo!!20:29
SpamapSThe mysql server can live on another machine.20:30
koolhead17SpamapS: hmm.20:30
koolhead17also is there an option in apt-get which results in skipping the question part, like skipping option for username/passwd20:30
koolhead17and all20:31
SpamapS-q20:31
SpamapS-yq actually20:31
koolhead17SpamapS: thanks. 20:32
SpamapSkoolhead17: if you read that post you linked to, it deploys mysql as its own host.. it then relates mediawiki to it. Your phpmyadmin formula must relate to the database server using hooks.20:32
koolhead17hmm20:32
SpamapSkoolhead17: have you looked at the example mysql and wordpress formulas yet?20:33
koolhead17SpamapS: yes20:33
koolhead17i also have drupal formula here20:34
koolhead17it will be similar20:34
SpamapSkoolhead17: there are already 2 other drupal formulas in progress...20:36
koolhead17okey20:36
SpamapSkoolhead17: ok I'm not sure if you understand the examples if you are confused by my statements. Only one formula ever needs to install mysql-server.20:36
SpamapSDOH.. I just now noticed my last 'ensemble shutdown' a week ago didn't work right and left 11 m1.smalls running for a week20:41
SpamapSDOH DOH DOH20:41
robbiewlol20:42
* SpamapS decides tonight he will hack out an LXC provider.. 20:42
* robbiew makes sure to review the next expense report from SpamapS20:42
SpamapSrobbiew: its 1 *meeeeelion* dollars20:42
SpamapSok it was actually 7 m1.smalls and 4 t1.micros20:44
SpamapSthat should only be like, $7520:44
koolhead17hello robbiew20:48
koolhead17SpamapS: db-relation-changed what is this hook for20:48
robbiewhey koolhead1720:48
koolhead17in the wordpress example? assigning DB?20:49
SpamapSkoolhead17: When you relate the wordpress service to the mysql service, that is the hook that gets run20:49
koolhead17SpamapS: cool20:50
SpamapSkoolhead17: if you haven't read this yet, you really should: https://ensemble.ubuntu.com/docs/write-formula.html20:51
koolhead17SpamapS: i have that page open, drupal example one. i think kim0 wrote it :)20:52
m_3SpamapS: you totally just gave me one of those 'oh crap' moments... but everything's safe and shutdown20:57
SpamapSm_3: I am 99% sure I just typed it fast before slamming the laptop shut.. it probably never completed20:58
niemeyerm_3!20:59
niemeyerm_3: Welcome!20:59
m_3Hi Ensemble team20:59
hazmatSpamapS, ouch!20:59
m_3nice to meet everyone20:59
m_3niemeyer: thanks!21:00
* koolhead17 is scared, the cats are making scary noise outside21:03
SpamapShazmat: Hey, I've had debug-log disconnect on me twice now21:06
hazmatSpamapS, disconnect? what's it say on the terminal when that happens?21:07
SpamapShttp://paste.ubuntu.com/626112/21:08
SpamapShazmat: ^^21:08
SpamapSsweet..21:08
* SpamapS just added a db-admin interface to mysql so services can be related with 'all privileges' ..21:09
hazmatSpamapS, ah.. yeah.. this is the session event stuff, i've got a branch of txzookeeper which will resolve (https://code.launchpad.net/~hazmat/txzookeeper/session-event-handling), but its pending some additional testing work for setting up zk test clusters21:10
SpamapSahh ok21:10
SpamapSsame thing that killed my munin nodes?21:11
hazmatSpamapS, yup, once that's merged that should resolve this class of issues21:11
hazmatits basically zk client letting the app know about transient network connect/disconnects21:11
koolhead17SpamapS: -yq  option doesn`t seem to work :(21:20
hazmatbcsaller, jimbaker if you one you have time, could you please check out niemeyer's debug-hooks-fixes? its pretty critical to having that feature work well, and needs an additional review21:24
bcsallersure21:24
niemeyerDanke!21:29
_mup_ensemble/set-transitions r240 committed by bcsaller@gmail.com21:33
_mup_better error message when passed non-key/value pair on cli21:33
pindongahi everyone! I have been playing around a little with ensemble 21:34
pindongaI'm trying to create a provider for lxc :)21:34
pindongaI got as far as breaking everything, and slowly starting to put the pieces back in place :p21:34
hazmatpindonga, cool :-)21:34
pindongaso I can now create a lxc container, start it, and am stuck when ensemble tries to ssh into it21:35
pindongabecause it didn't yet install the zookeeper stuff into the container21:35
pindongacan you guys point me to the code that installs the necessary packages into the bootstrap instance?21:35
hazmatpindonga, ah.. you mean the ssh succeeds but the client can't connect to via the tunnel because zk isn't running21:35
pindongahazmat, ensemble tries to connect via ssh to port 281xx 21:36
pindongabut it fails21:36
hazmatpindonga, its using cloud-init.. see ensemble/providers/ec2/launch.py for the existing stuff21:36
hazmatpindonga, you can seed  cloud-init by dropping a file on disk as well instead of the ec2 metadata service21:36
pindongaI believe ensemble should do something like apt-get or dpkg to install zookeeper into the bootstrap instance, right?21:36
niemeyerCoffee!21:38
hazmatpindonga, it should but at the moment its relying on the image to provide it.. installing it at runtime was overly time consuming (ie. java stack from scratch).... so it got yanked from the cloud-init install instructions to part of image building.. we're planning  on yanking that though and going back to cloud-init based installation of zk21:38
hazmatpindonga, its a pretty small change to reenable it.. effectively just adding a apt-get install zookeeper to the cloud init commands list21:39
pindongahazmat, k, will look for it21:39
pindongaso right now ensemble cannot bootstrap from base install, as I understand?21:39
hazmatpindonga, it can, its just not configured that way atm.. actually let me double check something21:41
* hazmat waits for ec2 bootup21:41
hazmatpindonga, so it can bootstrap a regular image.. we just pre-install for speed21:46
hazmatpindonga, so ensemble/providers/common.py has the standard packages and repos we use as constants, ensemble/providers/ec2/utils.py has format_cloud_ini (which should get refactored into the previous common.py since its now generic).. and then ensemble/providers/ec2/launch.py will setup additional things for installation based on bootstrap node vs. machine node.21:47
hazmatin terms of getting cloud-init setup with with an lxc-provider, the cloud-init needs to be preseeded on disk before starting up the instance, as an example with (assuming natty) http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/seed/README21:48
pindongathanks, will look into it21:49
hazmatpindonga, cool, you might want to talk to SpamapS if you want to collab as he mentioned some interest in working on an lxc-provider as well21:56
hazmatand keep the questions coming ;-)21:57
pindongahazmat, right now I'm just doing the minimum to get it up and running 21:57
pindongano intention to get production code until I understand how this works :)21:57
hazmatbcsaller, re the branch config-set.. was there a merge conflict with this branch, i'm seeing some oddities in a diff against trunk?22:03
bcsallerhazmat: the config-set that was merged? or the later branch I emailed you about?22:04
hazmatbcsaller, the emailed branch22:04
bcsallerhazmat: I had a divergent branch issue between this and my other laptop from when this one was being repaired. That said its a few revisions back and I thought dealt with 22:05
hazmatbcsaller, as an example there's a test_hook_knows_service that looks like it got yanked in the branch.. some of other oddities are just relocating code chunks22:05
bcsallerhmm22:05
bcsallerhazmat: ahh, I think I know what happened there, That was code that was deprecated after a review but by the time it was ready for merge jim's branch was already depending on some of that code 22:07
bcsallerso I added those parts back in  (though my branch already contained the diff to remove them from the previous review)22:08
hazmatbcsaller, ah.. ic.. okay.. i'm having a look at it now22:08
bcsallerappreciate it 22:08
hazmatbcsaller, could you pastebin errors from a test run?22:19
bcsallerhazmat: sure, are you not getting errors?22:19
hazmati get a pretty constant set of 4 error22:20
hazmatover 3 full test runs22:20
bcsallerand if you run those alone do you still see them?22:20
hazmatbcsaller, haven't tried yet.. i just want to make sure we have the same error set to verify them and ensure we have sane local setup deltas22:21
hazmatbcsaller, is that number inline with your seeing?22:21
bcsallerthat time I only got ensemble.agents.tests.test_unit.UnitAgentResolvedTest.test_hook_error_on_resolved_retry_remains_in_error_state22:22
bcsallerbut it might change if I run it again (which I am)22:22
hazmatbcsaller, yeah.. one of these tests is definitely bleeding into the others22:26
bcsallerhazmat: yeah, that time it was a different one22:26
hazmatbcsaller, one review comment on the change.. it would be better to just reconstruct the service state as needed rather than passing it as a new construct arg to scheduler, relhookcontext, etc22:32
hazmatie. the rel hook context has a client and can reconstruct the state on demand22:33
bcsallerhazmat: doesn't the service name then need to be parsed from the unit_name22:33
hazmatbcsaller, it does, but that's a pretty trivial22:36
bcsallerI can make those changes 22:37
hazmati think the service-state-manager could grow a utility method for it as well22:37
hazmatreturn the service state given a unit name22:37
bcsallermakes sense,  yes22:41
dakeryo22:43
niemeyerdaker: Yo22:43
niemeyerdaker: So,22:43
niemeyerdaker: "--password=" < that's not going to work if you split the arguments 22:44
niemeyerdaker: You can drop the "=" sign there22:44
niemeyerdaker: Then they can be in separate arguments22:44
niemeyerdaker: This won't do what you want as well:22:44
niemeyer"< %(ensemble_sql)s" % config)22:44
dakershould be splited ?22:44
niemeyerdaker: It's a bit trickier in this case22:45
niemeyerdaker: When you provide something like that to the shell, what's happening is that the file descriptor for stdin is being replaced to that of the open file with the given name.22:45
niemeyerdaker: Have you considered using bash for your script?22:45
niemeyerdaker: In case that's what you're most comfortable with?22:46
dakeri am comfortable with python, bash was suggested by kim022:46
niemeyerdaker: Aha, ok22:47
dakerniemeyer, i have an idea22:49
niemeyerdaker: You may want to use -e22:49
niemeyerdaker: Rather than stdin22:49
dakeri'll just use MySQLdb22:50
niemeyerdaker: Otherwise you'll have to open a file and extend your do() function to replace stdin22:50
niemeyerdaker: Ah, that works too! :)22:50
SpamapSniemeyer: so the idea of using relation id is actually good23:34
SpamapSniemeyer: thinking through it, using service name for database names is actually VERY limiting.23:34
niemeyerSpamapS: I'm not entirely sure yet, to be honest23:34
SpamapSniemeyer: otherwise we can't have two services use the same database.23:35
niemeyerSpamapS: Agreed, for that kind of use case a relation id would be great23:35
niemeyerSpamapS: The question is more around the lines of reestablishment23:35
SpamapSniemeyer: so I think we need to have the consuming services provide us with a clear hint. It may usually be service name.23:35
niemeyerSpamapS: It isn't clear that what we want to happen when one removes and readds a relation23:36
niemeyers/that what/what/23:36
SpamapSniemeyer: IMO, thats up to the formula to document.23:36
niemeyerSpamapS: I think we need to guide people here23:36
niemeyerSpamapS: So that formulas implement it somewhat consistently, and follow a principle of least surprise23:37
SpamapSniemeyer: I was thinking of mapping relation-id to the requested context like "demo-wiki" .. then if something else asks for demo-wiki, give them access.23:37
niemeyerSpamapS: I don't get that last idea23:38
SpamapSniemeyer: can add an optional setting of 'maxrelations' for each database context too.. so you can say 'maxrelations=1' if you want to have exclusive databases.23:38
SpamapSniemeyer: so the relation-id is just a uuid, so there'd be a table..   relation-id: 11004-33-55-5-666, data-name: 'wiki'23:39
SpamapSniemeyer: when we break the relation, we remove that row. When somebody asks for 'data-name=wiki' we give them access to the 'wiki' database.23:39
SpamapSerr23:39
SpamapSmeh I'm all out of order.23:39
SpamapSniemeyer: basically use the relation-id to map to the data we care about23:40
niemeyerSpamapS: I don't understand how that's better than what we have now?23:41
SpamapSniemeyer: the only change is that the consuming formulas specify the database name, rather than the consuming service's name..23:41
SpamapSniemeyer: so that you can say, have a backend hadoop job querying the same database as the frontend webservers.23:42
SpamapSright now you can't do that23:42
niemeyerSpamapS: We already have that.. that's essential what relations do23:42
niemeyerSpamapS: They have a name, for precisely that reason23:43
niemeyerSpamapS: and that's already possible today (sharing the db between multiple services)23:43
SpamapSright now, if I spawn two mediawiki services.. and relate them both to a mysql service, they will get independent databases...23:43
SpamapSI may want them to share a db23:44
SpamapSyeah, thats a formula problem. :)23:44
SpamapSniemeyer: yes I guess I'm saying we should do that sooner rather than later... the relation-id is a separate issue.23:46
niemeyerSpamapS: Do what sooner?23:46
SpamapSniemeyer: change the mysql formula to accept the database name rather than assume it.23:48
SpamapSniemeyer: and as far as the relation-id, we can use that to know which relation maps to which database name.23:48
niemeyerSpamapS: I'd like to see a use case we can work on, first23:48
SpamapSas long as its available in all hooks23:48
SpamapSniemeyer: the mysql use case isn't good enough?23:49
SpamapSduring broken, I have no idea what resources to clean up23:49
niemeyerSpamapS: Sorry, I may have missed it.. what's the use case again?23:49
SpamapSit was in the bug you duped IIRC23:50
niemeyerSpamapS: Ah, ok.. that's about being able to identify relation ids, so that re-adding a relation to the same set of formulas can reuse resources.23:50
niemeyerSpamapS: That's not the same as manually specifying database names23:51
SpamapSIts actually more than that. Iduring broken, right now, there is no way to tell which relation was broken other than a hack where you run 'relation-list'23:51
niemeyerSpamapS: Why do you want to tell which relation has been broken?23:51
niemeyerSpamapS: I mean.. why do you want an identifier in that location?23:52
SpamapSBecause I want to revoke the user privileges, delete temp tables, etc. etc. The documented reasons for the broken relation are to clean up.23:52
niemeyerSpamapS: Hmmm..23:52
niemeyerSpamapS: Removing tables is not a good idea in general, but I see your point otherwise23:53
SpamapSnotice I said *temp*. :)23:53
niemeyerCool, ok :-)23:53
SpamapSI happen to agree that purging should be primarily manual. :)23:53
niemeyerSpamapS: Yeah, I think for _that_ the relation id in that ticket should handle it23:53
SpamapSOr at least married with a backup service.23:53
niemeyerSpamapS: THe issue is what do you expect when you re-add a relation to the same couple of services?23:54
SpamapSRight, for that, we definitely need to set an example.23:57
SpamapSI like the way the mysql formula handles that. If somebody requests an existing database, go ahead and give them access.23:57
niemeyerSpamapS: I'm starting to think the mysql formula should create a database after the remote service name, as it is doing right now23:57
niemeyerSpamapS: and we need a relation id23:58
SpamapSand never allow shared access to a single db?23:58
niemeyerSpamapS: That's a different use case.. let's handle it separately please23:59
niemeyerSpamapS: We already handle shared access23:59
niemeyerSpamapS: All the service units from the same service will access the same database23:59
SpamapSBut I want two services to be able to share data through mysql.23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!