[04:56] bcsaller, that first paragraph in the spec is really hard to parse [04:57] huh, I'll try and re-phrase it [05:00] bcsaller, why the extra name qual on a service.. ie. isn't wordpress.logging sufficient? [05:01] ah [05:01] nm [05:01] hazmat: service A colocates _charm_ L (logging), service B colocates charm _L_, is A.B the same service as B [05:01] B.L [05:01] but ok [05:01] I'll stop explaing [05:02] bcsaller, wouldnt i want to associate the same logging service to multiple co-located [05:02] or couldn't i [05:02] their lifecycle is tied to the parent in each case [05:03] it might be they all get their own logging-client interface and those talk to a common service via normal relationships [05:03] ie. i want to configure my rsyslogging params once for the environment in the logging service and associate it to each of the other services in the environment [05:03] not configure it once per other deployed service [05:03] I think I explained that above [05:03] but if its lacking tell me [05:04] hazmat: no it never happens on my m2.xlarge .. only when doing local on my slow laptop disk [05:04] SpamapS, good to know, i'm hoping i'll have a few branches in the queue for it, probably by end of day monday [05:05] hazmat: so wp.wp-logging relates to logging-master and sql.sql-logging relates to logging-master [05:05] hazmat: does seem like 3s is *awfully* low given the intended application of juju [05:06] bcsaller, hmm.. i think its better to just have --with-service [05:07] bcsaller, i don't want a hundred separate munin-node services that i have to configure.. i want a munin service, and a munin-node service, and i deploy other services with the munin-node service. [05:07] hazmat: I don't follow, --with is still there and it behaves as before, removed the optional service name from the cli [05:07] I think there are two use cases. [05:07] the units are getting pair-wise relationships, else its just another service relation [05:07] hazmat: thats a good point [05:08] one is a "policy service" deployed with multiple services with shared configuration and relatable *one time* to other things.. [05:08] the other is add-on component services, which are only deployed with one service [05:08] the second though, is a subset of the first, so the first would be the more ideal use case [05:09] SpamapS, they individual units of the colo will still be able to establish relations with their parent (colo) service to effect service specific changes [05:09] er. parent service unit [05:10] I have to agree with hazmat. Basically you want a generic "web machine" service (in puppet this would be a class) which provides all the usual instrumenting and tweaking that all of your web app nodes get [05:10] so you deploy web nodes --with web-machine [05:11] the dotted name notation can buy us that but wp.logging and mysql.logging have different lifecycles even w/o the name change [05:12] hrm.. I duno.. I think you can effectively consider each co-lo'd service unit its own entity that just happens to go inside those units.. not seeing where one needs to address them separately. [05:13] I see merits in both.. [05:13] bcsaller, looks like i have a lot of comments, i'll followup on list [05:14] hazmat: great, and maybe at the meeting on Friday :) [05:14] I'm going to sign off, looking forward to the email and optimistic about this new feature :) [05:48] bcsaller, awesome! [05:48] * hazmat wants some more optimisim [05:49] maybe its that beer in the fridge [05:49] I'm optimistic as well btw. :) [05:51] * SpamapS fires up another m2.xlarge to hopefully finish this ceph thing [06:00] SpamapS, re 3s, its actually 10s for a session to go awol [06:01] and expire [06:01] but the heartbeat fails after 3s right? [06:01] the pings happen every 3.3s, but its only at 6.6s that its a considered an issue, and 10s at which its fatal [06:01] well anyway, I've given up running the local provider [06:02] on my laptop anyway [06:02] its fan-frickin-tastic on this giant instance. :) [06:03] SpamapS, fair enough.. at the moment local provider is a perfect playground for failure testing ;-) [06:03] so it definitely will get better soon.. testing connection failures on ec2 was always a bit spotty, but suspend/resume clocks jumps are very nice [06:04] to be clear, I have not been seeing this with suspend/resume.. just loads of 8+ [06:04] SpamapS, but local is always limited to the class of machine for sure in terms of what it can run concurrently.. out of curiosity what where you running in the containers on the laptop when it failed? [06:04] ceph [06:04] SpamapS, how many units? [06:04] 3 [06:05] doing very little [06:05] hmm.. that's very odd [06:05] just the churn of dpkg/apt [06:05] the disk response time goes to hell [06:05] i've seen m_3 do a dozen, and i normally get to a half dozen pretty regularly (no ssd) [06:05] 2011-10-19 23:48:40,630:480(0x7fa28ae7f700):ZOO_WARN@zookeeper_interest@1461: Exceeded deadline by 3509ms [06:06] 2011-10-19 23:48:45,173:480(0x7fa28ae7f700):ZOO_WARN@zookeeper_interest@1461: Exceeded deadline by 1210ms [06:08] * SpamapS sets up an 'at' job to shut down his m2.xlarge "now + 3 hours" [06:47] hazmat: ok, just got it on my EC2 instance. :( [06:48] SpamapS, how many units? [06:48] adding two units to my ceph ring [06:48] 2011-10-20 06:47:37,873:272(0x7f217bb29700):ZOO_ERROR@handle_socket_error_msg@1621: Socket [192.168.122.1:46463] zk retcode=-112, errno=116(Stale NFS file handle): sessionId=0x133201298a60003 has expired. [06:48] load of 0.84 [06:48] what's odd is that unit addition is serial in the machine agent [06:48] each unit does have 4 peer relations [06:48] not sure if that plays a factor [06:48] that shouldn't matter, overall load and avail mem does [06:49] maybe its just because ceph is experimental [06:49] 16G of memory.. :) [06:49] haha [06:49] experimental service somehow dooms zk? ;) [06:49] it is a fairly complex charm [06:50] SpamapS, indeed.. but more seriously.. we'll definitly have fixes for this shortly, but the fact that your triggering it so easily with ceph .. [06:50] 3 separate daemons to manage for ceph [06:51] happens when the new units are setting up during install [06:51] 2011-10-20 06:47:31,529:272(0x7f217bb29700):ZOO_ERROR@handle_socket_error_msg@1528: Socket [192.168.122.1:46463] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 0ms) [06:51] 2011-10-20 06:47:31,523: hook.output@INFO: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode. [06:51] 2011-10-20 06:47:31,523: hook.output@INFO: [06:52] 2011-10-20 06:47:31,530: hook.output@INFO: Setting up libterm-readkey-perl (2.30-4build2) ... [06:53] actually, it happened before the 2nd unit was even added really [06:53] Oct 20 06:47:31 ip-10-82-145-26 dnsmasq-dhcp[8370]: DHCPDISCOVER(virbr0) 92:40:49:b0:a5:45 [06:53] I wonder.. [06:54] happens when the new containers first request dhcp [06:54] maybe this interrupts networking somehow [07:02] hazmat: indeed, regardless of what I'm doing to trigger it.. resiliency to this, and any other unknown weirdness would be the first order of business [07:02] seems sporadic [07:03] * hazmat wants gluster [07:04] an actual production quality clustered file system that works well in virtualized environments, and does data redundancy [07:10] I think ceph will actually basically destroy gluster. [07:11] Already have a few friends who say its impossible to get performance out of it. [07:11] gluster that is [07:13] that said.. juju should make benchmarking them side by side quite easy! [07:23] hazmat: go to sleep! [07:25] SpamapS, that's odd i've seen very good benchmarks of gluster [07:26] and i've used it in production b4 [07:26] hazmat: I used it briefly before leaving the last place. Had to fight with it to get basics working. [07:27] And then a few weeks after leaving, the guys were picking my brain about lockups and inconsistencies after they had tuned it too far toward the dark side. [07:27] It may have been the app and the way it was using the FS.. [07:27] but to get the app working right the most careful mode had to be used. [07:28] They switched to MogileFS and never looked back, since all they really needed was object storage [07:28] perhaps.. its got lots of tunables.. i've used it for wonky things like svn repo sharing to trac instances and large file storage for media transformation [07:28] s/distributed media [07:28] anyways time for bed... g'nite [07:28] This was millions of images a day churning (pics of home foreclosures, sadly) [07:29] 'nite! [07:29] we should have a gluster/ceph shootout once there are charms. :) [07:29] :-) [08:23] hazmat: hi kapil, just reading the network security and dynamic ports thread, i wonder if a sort of 'conflicts' solution like in dpkg would help with this for charms? E.g. two charms both utilize port 80, so they conflict and cannot be co-located ever. While auto detection of this conflict is nice to have, a manual field might be good enough to start [08:27] TeTeT: heh, thats basically what I think too [08:27] has worked this way quite well in Debian and Ubuntu [08:28] SpamapS: working late night? Or are you in europe nowadays? [08:28] very late [08:28] trying to be prepared for UDS and the surrounding madness. :) [08:29] Also trying to write a CEPH charm so I can remind myself of all the things complex charm writers need fixed. :) [08:30] SpamapS: he he, I still haven't gotten around to write a charm myself. now with 11.10 out I consider Landscape Dedicated Server as a test bed [08:32] the more eyeballs the merrier [08:37] anyway, time to go pass out [08:38] good night === fucrum is now known as fulcrum [13:11] bcsaller: I'm just writing a response to you and hazmat, and I realised I'm unclear about something -- is there a strict distinction between colo and non-colo charms, or are people free to colo whatever they want? [14:33] what happen if for some reason the zookeeper server dies, is it possible to rebuild it and let the unit services/instances know about its (new ip, etc)? [14:34] or do I need to redeploy it all again? [14:34] mrsipan: I believe presently you have to redeploy [14:35] fwereade: ^? [14:35] robbiew: thanks [14:35] mrsipan, robbiew is right [14:36] mrsipan, we're planning to have high availability by 12.04 but there's nothing there yet [14:45] fwereade, thanks, that make zookeeper a SPOF for juju [14:45] for now [14:45] mrsipan: exactly so [14:46] mrsipan: be reassured that the HA story is a big and important one :) [14:46] a **VERY** big and important one ;) [14:46] fwereade, cool, thanks [15:23] fwereade, people are free like birds, any one can be a co-located jail bird... some however are born in prison and destined for a life there (standalone: false) [15:26] hazmat: how poetic [15:27] bcsaller, predestination is rather harsh [15:28] I do like that we relaxed the ideas around what can co-located from the original run [15:33] hazmat: ping [15:33] robbiew, png [15:58] I saw a video demoing juju but now I can't find it. anyone know where one is? [16:00] kim0: ^? [16:01] fwereade, i think i'm going to change my mind on the doc stuff related to juju ssh. the command itself is not documented. really we need to incorporate more detailed docs on each subcommand in juju [16:02] fwereade, so that should be done in a separate branch/bug [16:02] jimbaker, sounds sensible to me [16:02] jimbaker, my approve still stands ;) [16:02] fwereade, perfect :) [16:10] <_mup_> juju/ssh-passthrough r410 committed by jim.baker@canonical.com [16:10] <_mup_> Fix doc typo [16:19] <_mup_> Bug #878948 was filed: need to be able to define a "virtual" or "external" service < https://launchpad.net/bugs/878948 > [17:02] anyone else seeing this problem when building the docs on trunk with make clean && make html? btw, it works fine with make singlehtml, which is normally how i look at the docs [17:02] http://paste.ubuntu.com/714369/ [17:15] <_mup_> juju/ssh-passthrough r411 committed by jim.baker@canonical.com [17:15] <_mup_> Merged trunk [17:23] <_mup_> juju/ssh-passthrough r412 committed by jim.baker@canonical.com [17:23] <_mup_> Addressed review points [17:24] <_mup_> juju/scp-command r412 committed by jim.baker@canonical.com [17:24] <_mup_> Merged upstream [17:42] SpamapS, relating two things at once to haproxy is not a race condition [17:42] SpamapS, the hooks execute serially on haproxy [17:42] at least it shouldn't be a race condition [17:44] looking at the haproxy charm, it looks fine [18:32] hazmat: it will produce a single service, only pointing to the last one related to [18:32] SpamapS, ah.. its not multi service aware for backing [18:33] one listen section [18:33] needs host header stuff .. which would likely have to come from backend service config options [18:33] which did not exist when I wrote it :) [18:37] hazmat: so, about the capistrano renderer.. [18:38] hazmat: if we're not going to accept that, how are we going to recommend people integrate with juju? [18:42] SpamapS, that would subume very easily into a misc command we could distribute if we had a REST interface [18:42] or that people could innovate on outside the core [18:44] I have misc/status2(gource|hostname).py in several branches already.. would like to be able to just write a simple python module as a plugin that does what they do. [18:45] mine currently poll juju status by execing it (I know they could import juju.control.status but I don't want to assume thats a public API) [19:14] SpamapS, the same problem exists for the capistrano renderer, it would need to poll to be accurate [19:14] Which is *fine* [19:15] I see where its not core to what juju wants to do.. [19:15] but neither is dot [19:15] but there are so many other one off consumers of status, it seems like just having a REST interface would solve this [19:15] but that doesn't solve distribution [19:15] Yeah REST to replace execing juju status. [19:15] sadly i don't think the python plugin stuff would fly (it does have a distribution model though) [19:15] you don't have to distribute it. you'd want a capfile on the same boxes you'd want to run juju status on [19:15] oh [19:16] distribution of the tool? [19:16] yup [19:16] So, yeah, bzr gets this right, and so does git [19:16] put something in your ~/.juju that makes it understand this [19:16] thats what I really want [19:17] and also the ability to put something in thepythonpath that is like a juju plugin so I can package it up [19:17] yeah.. i'd be happy with easy_install juju_gource && ./bin juju gource [19:17] err [19:17] we have this distro thing [19:17] ;-) [19:17] * hazmat sheds a tear [19:18] but yeah, it should ultimately be easy_installable so that we can spread it wider than Ubuntu [19:18] its just not going to fly, its non compatible with other impl [19:18] So, anyway, point being, if we're going to say "no more status renderers in juju" then we need to start a juju status renderer project. [19:19] and probably need to rip out dot and png [19:19] the question is if we add it, where do we draw the line? [19:19] we don't :) [19:19] we open our arms as wide as they'll go [19:19] and love everybody. :) [19:19] i mean all my friends are using fabric.. do we add fabric output.. [19:19] that's why i was thinking its pretty trivial to replace that with a status2capistrano misc thingy [19:20] your point that this is not something core to juju is well taken. Give me the appropriate integration point... right now its juju stats | mything [19:20] status rather [19:20] the other thing is, it would be nice if users of status could also watch the /status node introduced by bcsaller's statusd branch, instead of polling [19:21] jimbaker: yeah, thats the plan [19:21] bcsaller, good to hear! [19:21] juju status --watch ? [19:22] yeah [19:22] bcsaller, what would that do? [19:22] delay status output till its changed? [19:23] not clear how it interacts with --output, but ignoring that it just runs its collect/output when the topo changes [19:23] or terminal screen refresh [19:23] ideally it just waits until the change, then immediately returns. long poll. but i suppose it could be convenient if it resets itself again for people just wanting to watch it from a term [19:23] just stream it [19:24] blocking till change is key, otherwise you can just poll with $watch juju status [19:24] give it an additional root yaml tree element of status_snapshot: $unix_epoch [19:24] bcsaller, agreed. and SpamapS, i think that would be a common case. just want a one-shot option too [19:24] Tho yaml probably doesn't have a SAX like mode [19:24] SpamapS, negronjl so is piping status to an external command an integration problem? [19:25] SpamapS, its not a stream for sure.. [19:25] hazmat: Its fine with me, I like the unix way. I think we can probably write a single tool that is extensible with plugins for any output desired. [19:26] If the /status node comes into being, then just execing it with --wait-for-change over and over would be fine since it would block until a change [19:27] At this point, I see enough friction against the output renderer that it would be good to gave either a plugin system, api or unified tool [19:27] negronjl, +1 for a plug in system. i know this is something that m_3 really wants to see as well [19:28] negronjl: how about we stick our heads together in Orlando and produce a single tool that lives outside of juju? [19:28] negronjl: we can make it a Recommends: of juju in Ubuntu. :-D [19:28] +1 on the single tool. [19:29] Just dump said tool in charm-tools. [19:29] SpamapS, negronjl that sounds good, i'd be happy to lend a hand on that, and make it pluggable via pypi [19:30] or you could just crib from the cli-plugins.. its pretty trivial [19:30] I want to keep charm-tools to just the bare minimum for maintaining charm [19:30] there's something to be said for juju status supporting a variety of renderers including dot, capistrano, out of the box. having the ability to plugin new output formats into it will just make our necessary limited support there justifiable [19:30] and very functional [19:30] Ok with me. Im sure we can all come together and create something awesome :) [19:30] https://code.launchpad.net/~hazmat/juju/cli-plugins [19:31] hazmat: !! ? [19:31] A happy medium could be to leave Capistrano in but still create the tool for other formats for now. [19:31] this may already have a framework? [19:31] SpamapS, just using entry points into python packages for a plugin, its like a half-dozen lines of code [19:31] Once the tool is done, we move the different renderer to the tool [19:32] SpamapS, its been veto'd for juju [19:32] True, the work is done.. why not just accept it and make it clear that it will go away when there is an external alternative? [19:32] hazmat: of course it has. :-/ [19:33] hazmat, what has been vetoed? [19:34] python cli plugins for juju [19:34] re cli-plugins, short & sweet, nice. we still need to specific support for stuff like status that should support an output format plugin [19:34] but that's also short work too [19:34] jimbaker, there already is effectively a registry of outputers, that a plugin could register for [19:34] its just we dont have any internal distinction of internal api vs public api [19:35] so we'd have to do some versioning around the entry point [19:35] the goal is to have language neutral plugins around a rest interface at some point [19:35] hazmat, exactly, it's already there, so just need to support [19:35] Spamaps, I think that at this point we can move faster and more efficiently by moving new renderers and such outside of juju and into charm-tools. At least we could bypass the veto votes :) [19:36] anyway, +1 for plugins, i hope we can do them, since it's so simple [19:37] hazmat, is there an ETA on the REST interface or API? [19:37] negronjl, i started it but, its really a backburner, prod tagged issues and colo are the highest priority atm [19:38] specifically for myself getting the disconnected/expiration stuff going is priority atm [19:38] hazmat, thx [19:38] indeed, one of the things about the current command setup is that it is almost wants to do plugins, otherwise why not use a class structure instead of modules [19:38] negronjl, i'd definitely like to see it for 12.04 though [19:39] there's alot up in the air on resources and goals for 12.04 though [19:40] * hazmat wanders off to take a break, bbiab [19:43] negronjl: lets call it juju-plus ;) [19:44] jimbaker: true we coul djust abuse juju.control and write our own commands :-D [19:45] SpamapS, well, one *could* do that, but i think it's better to avoid some sort of monkey patching. just use a plugin system, that's all [19:46] Spamaps, my sentiments exactly. We can use existing commands, modules, interfaces, etc. and bypass some of the non-technical issues. [19:47] I have to run back to the kid stuff now.. but good talk everybody. :) [19:47] negronjl: lets chat about this next week for sure. [19:48] Spamaps: sure thing [20:22] * hazmat scores four touchpads in the mail [20:22] xmas shopping is done ;-)