[00:00] excellent [00:01] hazmat: any ideas on running the test suite? seems like I'm missing something [00:19] SpamapS, you need to define a ZOOKEEPER_HOME env variable pointing a zookeeper installation.. i think most of the dev team are using trunk builds of zk.. should work with both.. just running ./test will run all the tests you give the path to an individual test file or a dotted python name [00:19] hazmat: the zookeeper part seems broken [00:20] cannot use the system jars [00:20] actually its for ZOOKEEPER_PATH [00:20] SpamapS, yeah.. i was looking at that and wondering [00:20] $ jar -tvf /usr/share/java/zookeeper-3.3.3.jar |grep QuorumPeerMain 5371 Wed May 25 20:42:58 PDT 2011 org/apache/zookeeper/server/quorum/QuorumPeerMain.class [00:20] i pulled that code into txzookeeper for the cluster tests [00:20] The class is in there.. [00:20] but java can't find it. Which is weird. [00:20] * SpamapS *hates* how java starts anything [00:21] SpamapS, you can override it to point to an existing zk installation i believe, but it will reset any state in that zk [00:21] hazmat: I want to enable the tests in the package build eventually [00:21] so .. no ;) [00:23] SpamapS, you can spec zookeeperd as a test dependency, and then just set the env variable to point to it when running the tests, the package setups the server [00:24] hmm.. nevermind.. that only works for txzookeeper [00:24] --+ [00:24] No I like spinning one up [00:24] I want to know why it won't work. :-/ [00:25] SpamapS, its constructing the cli args for java in ensemble/tests/common.py against what you'll find in the zookeeper front end shell script [00:26] Yeah I'm running the java command manually from some debug I added to that.. and its not working [00:26] Honestly, its not working with the zookeeperd package either [00:26] SpamapS, is there something already running in the bg ( a previous zk server perhaps)? [00:27] SpamapS, there should be some verbose logs from it [00:27] no, but that wouldn't cause a NoClassDefFoundError [00:27] ah.. right [00:27] hmm [00:27] java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied) [00:28] thats the system level version failing.. hmm [00:32] looks like that is a misleading message [00:32] the problem seems to be that it can't find log4j [00:33] which you find w/ your trunk build because its probably in the build/lib/*.jar glob [00:40] hazmat: good call on the caffeine.. triple americano has me cranking [00:43] hazmat: I think I'll have the test harness look for /etc/zookeeper/conf/environment if ZOOKEEPER_PATH is unset [00:43] If that exists, CLASSPATH is set perfectly, no need to glob [00:44] SpamapS, sounds good.. the test harness will generate the config, its really just fixing up where it picks up the libs i think [00:44] if ZOOKEEPER_PATH is undefined [00:44] hazmat: yeah, all I need is CLASSPATH .. the rest is spot on [00:51] hazmat: woot! [00:51] * SpamapS files a bug and MP [00:53] <_mup_> Bug #800462 was filed: Test suite cannot run with system installed Zookeeper < https://launchpad.net/bugs/800462 > [00:56] test_too_many_arguments_provided ... [FAIL] [00:56] does that one fail for everyone right now? [00:58] ahh thats because I didn't have my AWS creds set [01:02] SpamapS, the tests shouldn't need your ec2 creds set [01:03] hazmat: http://paste.ubuntu.com/630596/ [01:04] SpamapS, ic.. its a bug.. they should be setting up their creds in their test config [01:05] hazmat: I think those are new tests [01:05] SpamapS, yeah... it looks like the ec2 cred issue only effects one test [01:05] agreed, is there a standard test access key that EC2 provides? [01:11] SpamapS, no [01:11] SpamapS, if you apply this diff that problem is fixed https://pastebin.canonical.com/48844/ [01:11] oh, nice [01:11] Well I'm not testing those areas.. [01:11] I just wanted to see the entire suite pass [01:12] SpamapS, if you don't mind giving a +1 on that i'll commit it to trunk its pretty trivial [01:12] * SpamapS re-runs w/ the diff [01:13] hazmat: so far so good [01:18] wow.. the tests take quite a while to finish [01:19] hazmat: new fail [01:19] http://paste.ubuntu.com/630601/ [01:23] jimbaker, any thoughts on that traceback? [01:23] SpamapS, i don't see those failures in trunk [01:24] SpamapS, is it possible they where introduced on the branch? if you want to push it i can look at the diff [01:26] hazmat: https://code.launchpad.net/~clint-fewbar/ensemble/test-using-system-zookeeper/+merge/65428 [01:26] hazmat: does not have your diff [02:17] hazmat: feels like there can be some common abstract classes for all providers to extend.. [02:24] hazmat, re traceback - i have seen something similar before where the argparse was changed with 2.7. regardless, it's not a forgiving test with respect to any changes [03:20] === freeflyi1g is now known as freeflying [08:49] morning [09:02] kim0: howdy... [09:02] * SpamapS should be going to sleep [09:02] oh yeah :) [09:02] kim0: hopefully by this time tomorrow we'll have an LXC machine provider.. :) [09:02] Woohoo YES [09:03] * kim0 smells goodness [09:03] I have it starting/stopping lxc containers.. just need to figure out cloud-init [09:03] * SpamapS passes out [09:03] hahah [09:03] thank you man .. this is awesome [09:03] wasn't there some cloud-init option [09:04] to point it at the url to get the metadata from [09:05] SpamapS: check out https://help.ubuntu.com/community/UEC/Images/KVMKernelOptions .. this part ds=nocloud-net;s=http://tinyurl.com/sm- [09:06] if that's actually the problematic part :) === daker_ is now known as daker [10:25] <_mup_> Bug #800580 was filed: generating status as svg results in error < https://launchpad.net/bugs/800580 > [10:48] yo i have a problem [10:48] i am writing a formula for phpbb3 [10:53] the install file have an apt-get install phpbb3 [10:54] and while it installing phpbb3 needs some passwords to be entered [10:59] some like this http://min.us/mvnIO1O#1f and this http://min.us/mvnE3V9#1f [11:00] so the forumla will never finished and its status will be always null [11:10] daker: debconf questions can be presseded, I don't know what policy there would be on that for formulas though [11:40] daker: checkout the debconf variables of phpbb3 from http://forum.soft32.com/linux/Bug-583197-Setting-phpbb3-PL1-ftopict514576.html [11:40] like phpbb3/admin-pass-ask [11:41] daker: you can actually use this way to silence it .. example for mysql → http://www.rndguy.ca/2010/02/24/fully-automated-ubuntu-server-setups-using-preseed/ [11:41] kim0, ok [14:59] daker, preseed is probably the best option [15:00] hazmat, thanks === koolhead11|afk is now known as koolhead11 [16:31] kim0: thanks for the tip, reading now [16:31] cool [16:31] and good morning [16:32] hehe :) [16:35] jimbaker, i was looking over your last branch in review, it wasn't clear that gustavo's patch was applied/pushed.. could you clarify? i just want to make sure i'm reviewing the latest and greatest. [16:36] there seem to be some bugs in the ensemble.lib.schema validator.. when something is missing a required element, or has something wrong.. it gives a completely unrelated error [16:37] hi all [16:37] like "found lxc, was expecting ec2" when the problem is that my lxc section doesn't have an admin-secret [16:38] do i have to do some more changes in my.cnf apart from changing bind-address = 0.0.0.0 to allow remote connections? [16:38] SpamapS, hmmm.. is the error on the provider type? and has that been updated with lxc as a valid string? [16:38] hazmat: yes.. I'll put together a simple test case. [16:38] ERROR 1130 (HY000): Host '192.168.1.3' is not allowed to connect to this MySQL server [16:39] koolhead11: the users that are added by the example and principia mysql formulas should take care of that, as they allow connection from any IP [16:40] koolhead11: have you related the two services yet? [16:41] SpamapS, am trying to create same on my 2 VM before initiating the same on EC2, in order to write hooks correctly [16:42] koolhead11: I'm not sure what you mean by that, but, just use EC2. It doesn't work in plain VM's (yet) [16:43] SpamapS, i mean i am writing the procedure for same and testing on local VM. Once am confident with the deployment i will sync it up with the EC2 as a formula. :) [16:46] koolhead11: well then the problem is that on the mysql server, you haven't created a user, which the db-relation-joined hook should be doing. [16:46] SpamapS, yes fixed it GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'passwd' WITH GRANT OPTION; [16:47] i had to do this as well apart from changing bind-address option :) [16:47] SpamapS, yes it was issue with server. it worked. thanks. :D [16:49] koolhead11: why aren't you just *using ensemble* to do this? [16:49] koolhead11: the whole point is that you don't have to care about what mysql does.. its handled by the already written mysql formula. [16:51] SpamapS, yes. as of now i have not reached at the stage to put my formula on ensemble. i am writing initial hooks and testing it. once its done i will use mysql formula to do that [16:52] koolhead11: ok. I think you'll find its easier to just test your formula by putting it in ensemble. [16:53] koolhead11: hopefully when I have the LXC container done you won't even have to pay Amazon for the pleasure. :) [16:53] SpamapS, ooh awesome!! :d [17:14] This LXC stuff is pretty cool [17:14] Like having a VM that boots in 1s. [17:14] hallyn: ^5 [17:15] SpamapS: hang onto your hats, i've got the stuff lifeless wanted coded, and dlezcano will review lxc-clone today, things are only getting better [17:15] hallyn: actually i'd be interested in discussing libvirt and the lxc templating stuff with you if you have some time. [17:16] hallyn: I find the lxc tools to be very straightforward.. so I'm leaning toward avoiding libvirt for now... [17:16] heh.. I'm leaning toward leaning away from something :) [17:17] hallyn: anyway, the templates.. it would be useful if I could pass in some optional data to them.. like environment variables. But they seem to have their environment stripped. [17:17] SpamapS: I'm working on the templates right now [17:18] hallyn: ahh, making them arch-configurable ? [17:18] I've got it supporting arch, release, and an option to bind-mount in your $home [17:18] yeah [17:18] what else would you like? [17:18] hallyn: pre-seeds. :) [17:18] ah, yes. i was wanting something like that [17:19] you want just a list of packages to install? [17:19] lxc-attach would be very helpful, if we run the unit agents outside the container, so they execute hooks in the container namespace [17:19] and an optional preseed file to set debconf selections. [17:19] hazmat: yeah, that just needs the kernel bits to land in upstream kernel finally [17:19] hazmat: that sounds like madness. ;) [17:19] just let the thing work like a server! [17:19] hazmat: TBH i'm not that big a fan (security nightmare), but... [17:20] SpamapS: can you put up an email or wiki page showing some example usages? [17:20] SpamapS: i '-packages=x,y,z' i can picture, not sure about debconfs [17:20] hallyn: Right now I'm struggling to configure cloud-init for instance. [17:20] hmm. interesting, for us it would be a minor security boon with separation of user provided code vs. the ensemble framework code. [17:21] its minor at this point, having lxc working, would be all kinds of awesome by itself [17:21] hazmat: i don't udnerstand what you mean. what woudl lxc-attach give you that sshing into the container doesn't? [17:21] lxc-attach is useful if your container is broken, but... [17:23] hallyn, ensemble is based on the execution of user provided hooks in response to events, at the moment we just fork them from the agent that receives the events, we could ssh into the container to do so, but its easier manipulating the forked environment without [17:24] chopped at 'environment without' [17:25] hazmat: you could also get funky and use expect over the container's console :-) but that's even more work than ssh [17:25] hazmat: don't the agents just run things in response to ZK events? [17:25] hazmat: you can also just run an agent in the container which talks over a unix sock to one on the host, and have the agent in container do the fork+maniuplate [17:25] SpamapS, re work like server, for the lxc provider, that probably makes the most sense, (ie a machine agent and unit agent per container).. for the multi unit servers, there is some room for better separation [17:26] but anyway, i'm in no way blocking lxc-attach, so on with the task at hand :) [17:26] hallyn, indeed, that's also viable, lots of different options, lxc-attach just looked like the right hammer from my casual observation [17:27] SpamapS, yup that's pretty much all they do at the moment, respond to state change events [17:36] hazmat, sounds good about applying the patch [17:37] anything else you need for the review? [17:38] it may make more sense for me to get its prereq branch in trunk first [17:38] jimbaker, nope, that's it, i've already done a review on the prereq [17:38] hazmat, indeed, that's why it can go into trunk ;) [17:39] jimbaker, i think there where a few un-addressed review comments on it, that should get looked at, but else sounds good [17:40] hazmat, it was unclear to me what the ordering should be in these things [17:40] jimbaker, how so? === koolhead11 is now known as koolhead11| [17:40] hazmat, as in, do we fix post review #1 so that it can also be seen in review #2 [17:40] jimbaker, if its approved, it should go in.. and if its got a pre-req, it should wait for the pre-req to get merged. [17:41] hazmat, so that's the ordering i'm talking about with the new review process, not about any changes to our established merge process [17:42] hazmat, so given that you want the patch applied, seems like it would work better if i had done that first vs keeping it fixed for your review [17:42] jimbaker, lp will interleave commits with r1 and r2.. in this case the comments will have some significant change on the code.. and general rule of thumb is to incorporate feedback and push latest, but its not required for the second review imo unless addressing the first entails significant change to the proposal under review [17:42] hazmat, indeed there were no significant changes here [17:43] hazmat, just minor fixes or it would not been approved [17:45] jimbaker, i disagree, the tests are the same, but i think in this case i think it is significant impl change from what's extant, so it would be nice to review the whole.. the diff covers is 20% of the whole. [17:46] hazmat, ok [17:49] jimbaker, thanks [17:57] re bash completion .. I am wondering why no one has written a generic completer where we'd write a spec in xml or yaml, and have it work from there .. instead of writing all the logic like → http://bazaar.launchpad.net/~bcsaller/ensemble/bash-completion/view/head:/etc/bash_completion.d/ensemble [17:59] you'd generate it from the arg parser [17:59] but that was more work, argparse isn't very introspectable [18:03] <_mup_> ensemble/expose-watch-exposed-flag r250 committed by jim.baker@canonical.com [18:03] <_mup_> Addressed review comments [18:03] bcsaller: even after generating it .. there's no generic consumer to use it right [18:03] any way .. I wish to get bash completion soonish :) [18:03] great work on that branch [18:04] SpamapS: whenever you need someone to test your LXC magic .. ping me :) [18:04] bcsaller, it is introspectable i think, but does require mucking with private variables [18:05] * kim0 afk [18:05] the data isn't intended to be used that way and its not easy, yes its there though [18:06] yeah, argparse definitely needs some updating. this is the curse of getting included in the standard library [18:06] there should be a generic argparse -> bash_completion.d function [18:11] SpamapS: +1 LXC tester [18:13] bcsaller, but it should also extend argparse on the metadata for parameters such that appropriate hooks are available, so we can get completion on things like wordpress/0 [18:18] <_mup_> ensemble/expose-provision-service-hierarchy r299 committed by jim.baker@canonical.com [18:18] <_mup_> Simplified the state tracking watches using Gustavo's patch [18:19] hazmat, ^^^ [18:19] so it's been pushed up [18:19] <_mup_> ensemble/trunk r260 committed by jim.baker@canonical.com [18:19] <_mup_> merged expose-watch-exposed-flag [r=niemeyer,hazmat][f=788825] [18:19] <_mup_> Updates ServiceState.watch_exposed_flag such that the watch only [18:19] <_mup_> returns after processing current state; it is permanent; the client [18:19] <_mup_> connection is guarded; and it is stateful and reliable. [18:20] I should have something to test soon .. so far it doesn't run my cloud-init stuff yet, but I'm getting close [18:28] hallyn: so one other thing that would be useful for the templating is a way to easily specify arbitrary file placement. Kind of like a .install file in a package. [18:29] hallyn: otherwise I have to use sudo to copy files in... I'd much rather be able to say you just need sudo for lxc-* [18:29] SpamapS: live-build has a concept of chroot-includes/, maybe we can do something like that [18:30] hm, maverick containers no worky [18:30] hallyn: yeah that would actually solve most of my preseed desires too. :) [18:31] since then I can just dump cloud-init data in === daker is now known as daker_ [18:47] * hazmat lunches bbiab [18:54] m_3: About the NFS formula.. [18:54] m_3: I've been thinking a lot about shared filesystems [18:55] m_3: I'm thinking we can make it generic enough where formulas can just 'requires:' on an interface of 'shared-filesystem' .. and the provider will tell them how to mount it. [19:09] * bcsaller heads off to the structure conference [19:09] yay, it all works. ship it [19:32] SpamapS: re: NSF agree [19:33] SpamapS: was trying to decide on long-format relation name or short distributed-shared-filesystem-relation-changed is a mouthful [19:33] SpamapS: dsf for (g)luster et al [19:34] SpamapS: maybe just nfs-relation-changed for NFS [19:35] dang.. dyslexic today... s/nsf/nfs/g [19:40] haha [19:41] nfs is fine for the relation name.. and gluster for gluster.. [19:41] ensemble add-relation mediawiki:shared-filesystem nfs-server-1:nfs makes a lot of sense to me [19:41] SpamapS: yup [19:42] SpamapS: had actually thought they needed to be symmetric [19:42] SpamapS: i.e., mediawiki:shared-filesystem nfs-server-1:shared-filesystem [19:43] SpamapS: but good to know [19:45] m_3: no the idea is for them to describe what part of the service you're relating to [19:45] <_mup_> ensemble/hooks-with-formula-dir r260 committed by kapil.thangavelu@canonical.com [19:45] <_mup_> invoker execute hooks in the formula directory [19:45] gluster actually has an NFS capability.. you could conceivably have a relation called 'nfs' for it and a 'gluster' relation for it.. both with the same interface.. [19:46] SpamapS: understand [19:52] SpamapS, atm they do need to be symmetric [19:57] hazmat: the relation names, not the interfaces. [19:58] SpamapS, ah.. indeed, quite true [19:58] SpamapS, actually its the opposite the interfaces need symmetry.. the names are local to the formula [19:58] hazmat: which is exactly what I was saying. :) [19:59] * hazmat is feeling a little slow this afternoon ;-) [19:59] maybe some more that of coffee [19:59] hazmat: triple americano. :) [19:59] yum [20:00] with cinammon for extra fat burning. :) [20:00] and maybe some ginseng so I can learn how to spel [20:00] somebody's been reading the 4hr body [20:00] m_3: 8 weeks of slow carb has me 15 lbs. lighter and 10 inches less around [20:01] SpamapS: damn... dude! 10in is a monstrous change [20:01] m_3: well I was pregnant w/ a food baby. ;) [20:02] SpamapS: I was at 20lbs down but stalled with moving and job changes over the past couple of weeks [20:02] m_3: its 6 inches from around the belly, and 4 around the hips.. dropped from 34 to 32 in jeans [20:02] SpamapS: awesome [20:02] and XL -> L in shirts [20:03] size changes are all great until you run out of clothes [20:04] but that's the good direction :) [20:11] m_3: the XL's still fit, but look like hand-me-downs from my big brother [20:31] <_mup_> ensemble/hooks-with-formula-dir r261 committed by kapil.thangavelu@canonical.com [20:31] <_mup_> adjust invoker construction by the lifecycle, hooks are now executed in the unit directory. [20:35] so it looks like we should be able to use the official UEC images rather than the lxc OS templates [20:35] hallyn: so.. I think I'll create a template which simply pulls the images, rather than uses debootstrap.. [20:38] SpamapS: ok [20:38] SpamapS: also consider lxc-clone -s, which takes about 1 second [20:39] hallyn: the benefit there is, the cloud images will be identical to an LXC container. [20:39] hallyn: I don't have that command ;) [20:45] hallyn, with btrfs, could we take a snapshot of the root fs, and use as a base for the container [20:59] just wondering if we could avoid a debootstrap given that we may already have an effective base image [21:14] argh.. mocking is fragile [21:16] hazmat: schroot already does a lot of stuff like that with LVM and/or btrfs .. most anything schroot can do, lxc can do. :) [21:16] My new challenge with this LXC provider is actually getting reachable hostnames.. [21:17] dnsmasq has them.. [21:17] but how do I get ensemble to use dnsmasq w/o changing resolv.conf.. :-P [21:25] hazmat: you can do that with lvm. i was going to do btrfs support in lxc-clone, but btrfs (just a month or two ago) was too shoddy [21:25] as in, complete fs corruption in one snapshot [21:39] hallyn: ugh.. still? [21:39] SpamapS: i was shocked [21:59] <_mup_> txzookeeper/session-event-handling r49 committed by kapil.foss@gmail.com [21:59] <_mup_> replace some mock'd error behavior tests with real equivalents where possible, yank the one that wasn't (sync w/ error) [22:09] <_mup_> txzookeeper/session-event-handling r50 committed by kapil.foss@gmail.com [22:09] <_mup_> client usage while disconnected, will return a failed deferred instead of raising an exception [22:31] SpamapS: is the debian packaging in lp:txzookeeper trunk the one that you plan to upload? [22:32] ajmitch: no [22:32] ajmitch: its in the DPMT svn repository at the moment [22:32] ah ok [23:02] <_mup_> ensemble/debug-log-relation-settings-changes r270 committed by jim.baker@canonical.com [23:02] <_mup_> Invoker now only logs about a flush on the changes from a YAMLState flush when there are actual changes [23:04] <_mup_> ensemble/debug-log-relation-settings-changes r271 committed by jim.baker@canonical.com [23:04] <_mup_> Use slice idiom for returning changes from YAMLState [23:06] hazmat: hah, I just discovered where the ensemble AMI shortens the deploy... ;) [23:06] cd /usr/lib/ensemble/ensemble && sudo /usr/bin/bzr up [23:07] *doh* [23:09] <_mup_> ensemble/debug-log-relation-settings-changes r272 committed by jim.baker@canonical.com [23:09] <_mup_> Removed unused names/imports from touched files [23:47] hazmat: you know what would be cool? if providers could define the set of commands to make an AMI/image/etc. [23:47] hazmat: I see what you mean by having to install all the non-base packages every time.. [23:47] will have to get serge to show me lxc-clone next week [23:53] hazmat: ugh, more stuff that is in EC2 that seems it should be shared... [23:53] SpamapS, re provider set of commands. do you mean like a provider.create_image() api? .. [23:54] SpamapS, which part re shared? [23:54] hazmat: 99% of the zookeeper code embedded in ec2 [23:54] ah.. [23:54] Half of this effort has been moving things into common. :-P [23:55] SpamapS, if yeah.. all the cloud-init formatting should be shareable, and like 99% of the variable input to that [23:56] hazmat: I'm done w/ cloud-init .. at this point I'm failing because I can't connect to ZK .. ;) [23:56] 2011-06-22 22:49:14,652:1265(0x7f3fe78dc700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client [23:56] Haven't touched that from dummy yet [23:58] hazmat: its really.. really hard to follow what actually happens in the ec2 provider. :-P