=== almaisan-away is now known as al-maisan [10:33] * al-maisan -> office === kim0_ is now known as kim0 [12:41] 'ensemble status' always shows me connection timeout for me. Pasted the output here: http://paste.ubuntu.com/625813/ [13:47] Good morning everyone [13:53] o/ [14:34] Good morning niemeyer :) [14:38] al-maisan, kim0: Hey there! [15:01] <_mup_> ensemble/trunk r252 committed by gustavo@niemeyer.net [15:01] <_mup_> Merge add-python-yaml branch by Clint [r=niemeyer] [trivial] [15:01] <_mup_> This adds the missing python-yaml package dependency. === al-maisan is now known as almaisan-away [16:48] Hey guys. How would you all feel about moving everything except 'ensemble' out of /usr/bin ? [16:49] Otherwise... W: ensemble: binary-without-manpage usr/bin/open-port [16:53] SpamapS: Hmm [16:53] SpamapS: I think we should provide these man pages eventually [16:53] SpamapS: /usr/bin feels to me like the write place for these tools [16:53] are any of those ever useful outside of hooks though? [16:53] SpamapS: But then, I don't claim to understand the FS organization fantastically [16:53] If no, then they're really more suitable in /usr/lib [16:54] SpamapS: Today, they're not, but I hope all of these tools are runnable out-of-band in the future [16:54] SpamapS: So we can do actions based on events not initiated by Ensemble itself [16:55] Ahh, thats enough to make me think they need man pages then. [16:56] Though it might be better to have one command which tells the local agent to do things. [16:57] SpamapS: That may be a good idea.. that said, I still like how e.g. "relation-get foo" is concise [16:58] Yeah, my thinking is that we're just going to dump all over the global /usr/bin namespace. [17:02] Ugh.. past lunch time.. biab [17:34] hmm.. should -e / --environment be changed to a global option, instead of one for the commands? [19:27] Whats the exact policy on trivial changes to the trunk? The online help for remove-relation says that it adds a relation. [19:34] SpamapS: If it's truly trivial and you're sure that's what should be in place, you can "cowboy" the change in [19:34] SpamapS: Marking it as [trivial] in the commit line [19:35] SpamapS: Many times these will also be preceded by an "on-the-fly" review request [19:35] SpamapS: E..g just post the diff somewhere and invite a review [19:35] (here) [19:35] SpamapS: Your example sounds good for a trivial, FWIW [19:43] bcsaller, hazmat, niemeyer - standup? [19:43] jimbaker: I'd rather see some _action_ happening! [19:44] niemeyer, ok [19:45] hi all [19:46] koolhead17: Hey there [19:47] koolhead17: Noticed you picked a formula to work on.. that's pretty cool, thanks! [19:47] koolhead17: How're things going there so far/ [19:47] ? [19:47] niemeyer: :) [19:47] hoping to get it done by tonight!! [19:47] kim0: ping [19:48] <_mup_> ensemble/set-transitions r239 committed by bcsaller@gmail.com [19:48] <_mup_> fixed issue with merge of watch establishment in unit agent [20:01] koolhead17: You chose the phpmyadmin formula right? [20:01] SpamapS: yes [20:02] koolhead17: cool, its going to need some help from the mysql formula, but I already started on it. [20:02] SpamapS: yes. we need apache2/php5 as well :P [20:02] koolhead17: no, thats all in your formula in the install hook [20:03] koolhead17: what you need is an admin user that can read/write to/from all dbs. [20:03] SpamapS: hmm. am installing phpmyadmin along with depandency to come up with list of depandent pkgs which will go as depandant hook [20:04] on one of natty running vm on virtualboc [20:04] x [20:04] koolhead17: right, but you want to administer a mysql database for some other app, right? like, you want to admin the database for wordpress. [20:04] koolhead17: otherwise its just phpmyadmin with its own database [20:05] SpamapS: i thought this is a formula 4 a standalone phpmyadmin configuration. please correct me if am wrong [20:05] koolhead17: Its a good start. However, people can do that without ensemble. [20:05] now to create DB for a cms say drupal/joomla do we need a web-interface? [20:06] koolhead17: ensemble is interesting because it can relate these services together. :) [20:06] :D [20:06] we can do that via appropriate hooks while writing formula for jooma/drupal/wordpress [20:06] isnt it [20:06] :P [20:07] there we can have mysql formula as depandency [20:07] koolhead17: so you'll be able to say 'ensemble add-relation phpmyadmin:db wiki-db:db-admin .. and that will give you access to all the databases on wiki-db. [20:08] SpamapS: but i thought phpmyadmin has more to do with checking web-interface based mysql db :) [20:08] would we really need a formula like that as you suggsted!! :D [20:09] koolhead17: usually phpmyadmin is used to do queries on a database from some other application. [20:09] koolhead17: I don't see it being all that useful with just an empty database. [20:10] SpamapS: hmm!! your right [20:10] SpamapS: let me come up with the basic formula with hooks to install phpmyadin and then in 2nd level/round i can work on this [20:10] koolhead17: thats a great way to do it. [20:10] it will be easier 4 me to pick up ensemble in that case :P [20:10] koolhead17: install hook is usually first, then you add the relations. :) [20:11] SpamapS: yes. writing down all the pkgs saperately while testing a install in virtualbox :) [20:12] koolhead17: you shouldn't need to do much more than 'apt-get install phpmyadmin' [20:12] koolhead17: you will probably want to say '--no-install-recommends' [20:12] SpamapS: hmm [20:14] actually no [20:14] it only suggests mysql-server [20:14] * koolhead17 is running the command [20:14] so you can just do 'apt-get -y install phpmyadmin' .. it will automatically pull in php, apache, mysql client, etc. etc. [20:15] hmm [20:15] including apache2 and all [20:16] SpamapS: it also asks two options during install [20:16] Web server to reconfigure automatically: <-- apache2 [20:16] [20:16] Configure database for phpmyadmin with dbconfig-common? [20:18] also we need to add "Include /etc/phpmyadmin/apache.conf" in /etc/apache2/apache2.conf to get it to work :D [20:18] koolhead17: during an install hook, it will skip those [20:18] koolhead17: thats what you do during the install hook. [20:18] SpamapS: hmm [20:19] koolhead17: read the wordpress and mysql examples, they handle it [20:20] SpamapS: http://fewbar.com/2011/06/so-what-is-ensemble-anyway/ [20:20] this one [20:26] koolhead17: thats my post, not an example. ;) I mean in /usr/share/doc/ensemble/examples [20:27] SpamapS: :P [20:27] k [20:27] SpamapS: phpmyadmin is not dependant on mysql-server [20:28] koolhead17: right, it only 'suggests' mysql-server [20:28] we need to install mysql-server too for a new installation in order to get phpmyadmin working :D [20:29] No!! [20:30] The mysql server can live on another machine. [20:30] SpamapS: hmm. [20:30] also is there an option in apt-get which results in skipping the question part, like skipping option for username/passwd [20:31] and all [20:31] -q [20:31] -yq actually [20:32] SpamapS: thanks. [20:32] koolhead17: if you read that post you linked to, it deploys mysql as its own host.. it then relates mediawiki to it. Your phpmyadmin formula must relate to the database server using hooks. [20:32] hmm [20:33] koolhead17: have you looked at the example mysql and wordpress formulas yet? [20:33] SpamapS: yes [20:34] i also have drupal formula here [20:34] it will be similar [20:36] koolhead17: there are already 2 other drupal formulas in progress... [20:36] okey [20:36] koolhead17: ok I'm not sure if you understand the examples if you are confused by my statements. Only one formula ever needs to install mysql-server. [20:41] DOH.. I just now noticed my last 'ensemble shutdown' a week ago didn't work right and left 11 m1.smalls running for a week [20:41] DOH DOH DOH [20:42] lol [20:42] * SpamapS decides tonight he will hack out an LXC provider.. [20:42] * robbiew makes sure to review the next expense report from SpamapS [20:42] robbiew: its 1 *meeeeelion* dollars [20:44] ok it was actually 7 m1.smalls and 4 t1.micros [20:44] that should only be like, $75 [20:48] hello robbiew [20:48] SpamapS: db-relation-changed what is this hook for [20:48] hey koolhead17 [20:49] in the wordpress example? assigning DB? [20:49] koolhead17: When you relate the wordpress service to the mysql service, that is the hook that gets run [20:50] SpamapS: cool [20:51] koolhead17: if you haven't read this yet, you really should: https://ensemble.ubuntu.com/docs/write-formula.html [20:52] SpamapS: i have that page open, drupal example one. i think kim0 wrote it :) [20:57] SpamapS: you totally just gave me one of those 'oh crap' moments... but everything's safe and shutdown [20:58] m_3: I am 99% sure I just typed it fast before slamming the laptop shut.. it probably never completed [20:59] m_3! [20:59] m_3: Welcome! [20:59] Hi Ensemble team [20:59] SpamapS, ouch! [20:59] nice to meet everyone [21:00] niemeyer: thanks! [21:03] * koolhead17 is scared, the cats are making scary noise outside [21:06] hazmat: Hey, I've had debug-log disconnect on me twice now [21:07] SpamapS, disconnect? what's it say on the terminal when that happens? [21:08] http://paste.ubuntu.com/626112/ [21:08] hazmat: ^^ [21:08] sweet.. [21:09] * SpamapS just added a db-admin interface to mysql so services can be related with 'all privileges' .. [21:10] SpamapS, ah.. yeah.. this is the session event stuff, i've got a branch of txzookeeper which will resolve (https://code.launchpad.net/~hazmat/txzookeeper/session-event-handling), but its pending some additional testing work for setting up zk test clusters [21:10] ahh ok [21:11] same thing that killed my munin nodes? [21:11] SpamapS, yup, once that's merged that should resolve this class of issues [21:11] its basically zk client letting the app know about transient network connect/disconnects [21:20] SpamapS: -yq option doesn`t seem to work :( [21:24] bcsaller, jimbaker if you one you have time, could you please check out niemeyer's debug-hooks-fixes? its pretty critical to having that feature work well, and needs an additional review [21:24] sure [21:29] Danke! [21:33] <_mup_> ensemble/set-transitions r240 committed by bcsaller@gmail.com [21:33] <_mup_> better error message when passed non-key/value pair on cli [21:34] hi everyone! I have been playing around a little with ensemble [21:34] I'm trying to create a provider for lxc :) [21:34] I got as far as breaking everything, and slowly starting to put the pieces back in place :p [21:34] pindonga, cool :-) [21:35] so I can now create a lxc container, start it, and am stuck when ensemble tries to ssh into it [21:35] because it didn't yet install the zookeeper stuff into the container [21:35] can you guys point me to the code that installs the necessary packages into the bootstrap instance? [21:35] pindonga, ah.. you mean the ssh succeeds but the client can't connect to via the tunnel because zk isn't running [21:36] hazmat, ensemble tries to connect via ssh to port 281xx [21:36] but it fails [21:36] pindonga, its using cloud-init.. see ensemble/providers/ec2/launch.py for the existing stuff [21:36] pindonga, you can seed cloud-init by dropping a file on disk as well instead of the ec2 metadata service [21:36] I believe ensemble should do something like apt-get or dpkg to install zookeeper into the bootstrap instance, right? [21:38] Coffee! [21:38] pindonga, it should but at the moment its relying on the image to provide it.. installing it at runtime was overly time consuming (ie. java stack from scratch).... so it got yanked from the cloud-init install instructions to part of image building.. we're planning on yanking that though and going back to cloud-init based installation of zk [21:39] pindonga, its a pretty small change to reenable it.. effectively just adding a apt-get install zookeeper to the cloud init commands list [21:39] hazmat, k, will look for it [21:39] so right now ensemble cannot bootstrap from base install, as I understand? [21:41] pindonga, it can, its just not configured that way atm.. actually let me double check something [21:41] * hazmat waits for ec2 bootup [21:46] pindonga, so it can bootstrap a regular image.. we just pre-install for speed [21:47] pindonga, so ensemble/providers/common.py has the standard packages and repos we use as constants, ensemble/providers/ec2/utils.py has format_cloud_ini (which should get refactored into the previous common.py since its now generic).. and then ensemble/providers/ec2/launch.py will setup additional things for installation based on bootstrap node vs. machine node. [21:48] in terms of getting cloud-init setup with with an lxc-provider, the cloud-init needs to be preseeded on disk before starting up the instance, as an example with (assuming natty) http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/seed/README [21:49] thanks, will look into it [21:56] pindonga, cool, you might want to talk to SpamapS if you want to collab as he mentioned some interest in working on an lxc-provider as well [21:57] and keep the questions coming ;-) [21:57] hazmat, right now I'm just doing the minimum to get it up and running [21:57] no intention to get production code until I understand how this works :) [22:03] bcsaller, re the branch config-set.. was there a merge conflict with this branch, i'm seeing some oddities in a diff against trunk? [22:04] hazmat: the config-set that was merged? or the later branch I emailed you about? [22:04] bcsaller, the emailed branch [22:05] hazmat: I had a divergent branch issue between this and my other laptop from when this one was being repaired. That said its a few revisions back and I thought dealt with [22:05] bcsaller, as an example there's a test_hook_knows_service that looks like it got yanked in the branch.. some of other oddities are just relocating code chunks [22:05] hmm [22:07] hazmat: ahh, I think I know what happened there, That was code that was deprecated after a review but by the time it was ready for merge jim's branch was already depending on some of that code [22:08] so I added those parts back in (though my branch already contained the diff to remove them from the previous review) [22:08] bcsaller, ah.. ic.. okay.. i'm having a look at it now [22:08] appreciate it [22:19] bcsaller, could you pastebin errors from a test run? [22:19] hazmat: sure, are you not getting errors? [22:20] i get a pretty constant set of 4 error [22:20] over 3 full test runs [22:20] and if you run those alone do you still see them? [22:21] bcsaller, haven't tried yet.. i just want to make sure we have the same error set to verify them and ensure we have sane local setup deltas [22:21] bcsaller, is that number inline with your seeing? [22:22] that time I only got ensemble.agents.tests.test_unit.UnitAgentResolvedTest.test_hook_error_on_resolved_retry_remains_in_error_state [22:22] but it might change if I run it again (which I am) [22:26] bcsaller, yeah.. one of these tests is definitely bleeding into the others [22:26] hazmat: yeah, that time it was a different one [22:32] bcsaller, one review comment on the change.. it would be better to just reconstruct the service state as needed rather than passing it as a new construct arg to scheduler, relhookcontext, etc [22:33] ie. the rel hook context has a client and can reconstruct the state on demand [22:33] hazmat: doesn't the service name then need to be parsed from the unit_name [22:36] bcsaller, it does, but that's a pretty trivial [22:37] I can make those changes [22:37] i think the service-state-manager could grow a utility method for it as well [22:37] return the service state given a unit name [22:41] makes sense, yes [22:43] yo [22:43] daker: Yo [22:43] daker: So, [22:44] daker: "--password=" < that's not going to work if you split the arguments [22:44] daker: You can drop the "=" sign there [22:44] daker: Then they can be in separate arguments [22:44] daker: This won't do what you want as well: [22:44] "< %(ensemble_sql)s" % config) [22:44] should be splited ? [22:45] daker: It's a bit trickier in this case [22:45] daker: When you provide something like that to the shell, what's happening is that the file descriptor for stdin is being replaced to that of the open file with the given name. [22:45] daker: Have you considered using bash for your script? [22:46] daker: In case that's what you're most comfortable with? [22:46] i am comfortable with python, bash was suggested by kim0 [22:47] daker: Aha, ok [22:49] niemeyer, i have an idea [22:49] daker: You may want to use -e [22:49] daker: Rather than stdin [22:50] i'll just use MySQLdb [22:50] daker: Otherwise you'll have to open a file and extend your do() function to replace stdin [22:50] daker: Ah, that works too! :) [23:34] niemeyer: so the idea of using relation id is actually good [23:34] niemeyer: thinking through it, using service name for database names is actually VERY limiting. [23:34] SpamapS: I'm not entirely sure yet, to be honest [23:35] niemeyer: otherwise we can't have two services use the same database. [23:35] SpamapS: Agreed, for that kind of use case a relation id would be great [23:35] SpamapS: The question is more around the lines of reestablishment [23:35] niemeyer: so I think we need to have the consuming services provide us with a clear hint. It may usually be service name. [23:36] SpamapS: It isn't clear that what we want to happen when one removes and readds a relation [23:36] s/that what/what/ [23:36] niemeyer: IMO, thats up to the formula to document. [23:36] SpamapS: I think we need to guide people here [23:37] SpamapS: So that formulas implement it somewhat consistently, and follow a principle of least surprise [23:37] niemeyer: I was thinking of mapping relation-id to the requested context like "demo-wiki" .. then if something else asks for demo-wiki, give them access. [23:38] SpamapS: I don't get that last idea [23:38] niemeyer: can add an optional setting of 'maxrelations' for each database context too.. so you can say 'maxrelations=1' if you want to have exclusive databases. [23:39] niemeyer: so the relation-id is just a uuid, so there'd be a table.. relation-id: 11004-33-55-5-666, data-name: 'wiki' [23:39] niemeyer: when we break the relation, we remove that row. When somebody asks for 'data-name=wiki' we give them access to the 'wiki' database. [23:39] err [23:39] meh I'm all out of order. [23:40] niemeyer: basically use the relation-id to map to the data we care about [23:41] SpamapS: I don't understand how that's better than what we have now? [23:41] niemeyer: the only change is that the consuming formulas specify the database name, rather than the consuming service's name.. [23:42] niemeyer: so that you can say, have a backend hadoop job querying the same database as the frontend webservers. [23:42] right now you can't do that [23:42] SpamapS: We already have that.. that's essential what relations do [23:43] SpamapS: They have a name, for precisely that reason [23:43] SpamapS: and that's already possible today (sharing the db between multiple services) [23:43] right now, if I spawn two mediawiki services.. and relate them both to a mysql service, they will get independent databases... [23:44] I may want them to share a db [23:44] yeah, thats a formula problem. :) [23:46] niemeyer: yes I guess I'm saying we should do that sooner rather than later... the relation-id is a separate issue. [23:46] SpamapS: Do what sooner? [23:48] niemeyer: change the mysql formula to accept the database name rather than assume it. [23:48] niemeyer: and as far as the relation-id, we can use that to know which relation maps to which database name. [23:48] SpamapS: I'd like to see a use case we can work on, first [23:48] as long as its available in all hooks [23:49] niemeyer: the mysql use case isn't good enough? [23:49] during broken, I have no idea what resources to clean up [23:49] SpamapS: Sorry, I may have missed it.. what's the use case again? [23:50] it was in the bug you duped IIRC [23:50] SpamapS: Ah, ok.. that's about being able to identify relation ids, so that re-adding a relation to the same set of formulas can reuse resources. [23:51] SpamapS: That's not the same as manually specifying database names [23:51] Its actually more than that. Iduring broken, right now, there is no way to tell which relation was broken other than a hack where you run 'relation-list' [23:51] SpamapS: Why do you want to tell which relation has been broken? [23:52] SpamapS: I mean.. why do you want an identifier in that location? [23:52] Because I want to revoke the user privileges, delete temp tables, etc. etc. The documented reasons for the broken relation are to clean up. [23:52] SpamapS: Hmmm.. [23:53] SpamapS: Removing tables is not a good idea in general, but I see your point otherwise [23:53] notice I said *temp*. :) [23:53] Cool, ok :-) [23:53] I happen to agree that purging should be primarily manual. :) [23:53] SpamapS: Yeah, I think for _that_ the relation id in that ticket should handle it [23:53] Or at least married with a backup service. [23:54] SpamapS: THe issue is what do you expect when you re-add a relation to the same couple of services? [23:57] Right, for that, we definitely need to set an example. [23:57] I like the way the mysql formula handles that. If somebody requests an existing database, go ahead and give them access. [23:57] SpamapS: I'm starting to think the mysql formula should create a database after the remote service name, as it is doing right now [23:58] SpamapS: and we need a relation id [23:58] and never allow shared access to a single db? [23:59] SpamapS: That's a different use case.. let's handle it separately please [23:59] SpamapS: We already handle shared access [23:59] SpamapS: All the service units from the same service will access the same database [23:59] But I want two services to be able to share data through mysql.