/srv/irclogs.ubuntu.com/2011/10/20/#juju.txt

hazmatbcsaller, that first paragraph in the spec is really hard to parse04:56
bcsallerhuh, I'll try and re-phrase it04:57
hazmatbcsaller, why the extra name qual on a service.. ie. isn't  wordpress.logging sufficient?05:00
hazmatah05:01
hazmatnm05:01
bcsallerhazmat: service A colocates _charm_ L (logging), service B colocates charm _L_, is A.B the same service as B05:01
bcsallerB.L05:01
bcsallerbut ok05:01
bcsallerI'll stop explaing05:01
hazmatbcsaller, wouldnt i want to associate the same logging service to multiple co-located05:02
hazmator couldn't i05:02
bcsallertheir lifecycle is tied to the parent in each case05:02
bcsallerit might be they all get their own logging-client interface and those talk to a common service via normal relationships05:03
hazmatie. i want to configure my rsyslogging params once for the environment in the logging service and associate it to each of the other services in the environment05:03
hazmatnot configure it once per other deployed service05:03
bcsallerI think I explained that above05:03
bcsallerbut if its lacking tell me05:03
SpamapShazmat: no it never happens on my m2.xlarge .. only when doing local on my slow laptop disk05:04
hazmatSpamapS, good to know, i'm hoping i'll have a few branches in the queue for it, probably by end of day monday05:04
bcsallerhazmat: so wp.wp-logging relates to logging-master and sql.sql-logging relates to logging-master05:05
SpamapShazmat: does seem like 3s is *awfully* low given the intended application of juju05:05
hazmatbcsaller, hmm.. i think its better to just have --with-service05:06
hazmatbcsaller, i don't want a hundred separate munin-node services that i have to configure.. i want a munin service, and a munin-node service, and i deploy other services with the munin-node service.05:07
bcsallerhazmat: I don't follow, --with is still there and it behaves as before, removed the optional service name from the cli05:07
SpamapSI think there are two use cases.05:07
hazmatthe units are getting pair-wise relationships, else its just another service relation05:07
bcsallerhazmat: thats a good point05:07
SpamapSone is a "policy service" deployed with multiple services with shared configuration and relatable *one time* to other things..05:08
SpamapSthe other is add-on component services, which are only deployed with one service05:08
SpamapSthe second though, is a subset of the first, so the first would be the more ideal use case05:08
hazmatSpamapS, they individual units of the colo will still be able to establish relations with their parent (colo) service to effect service specific changes05:09
hazmater. parent service unit05:09
SpamapSI have to agree with hazmat. Basically you want a generic "web machine" service (in puppet this would be a class) which provides all the usual instrumenting and tweaking that all of your web app nodes get05:10
SpamapSso you deploy web nodes --with web-machine05:10
bcsallerthe dotted name notation can buy us that but wp.logging and mysql.logging have different lifecycles even w/o the name change05:11
SpamapShrm.. I duno.. I think you can effectively consider each co-lo'd service unit its own entity that just happens to go inside those units.. not seeing where one needs to address them separately.05:12
SpamapSI see merits in both..05:13
hazmatbcsaller, looks like i have a lot of comments, i'll followup on list05:13
bcsallerhazmat: great, and maybe at the meeting on Friday :)05:14
bcsallerI'm going to sign off, looking forward to the email and optimistic about this new feature :)05:14
hazmatbcsaller, awesome!05:48
* hazmat wants some more optimisim05:48
hazmatmaybe its that beer in the fridge05:49
SpamapSI'm optimistic as well btw. :)05:49
* SpamapS fires up another m2.xlarge to hopefully finish this ceph thing05:51
hazmatSpamapS, re 3s, its actually 10s for a session to go awol06:00
hazmatand expire06:01
SpamapSbut the heartbeat fails after 3s right?06:01
hazmatthe pings happen every 3.3s, but its only at 6.6s that its a considered an issue, and 10s at which its fatal06:01
SpamapSwell anyway, I've given up running the local provider06:01
SpamapSon my laptop anyway06:02
SpamapSits fan-frickin-tastic on this giant instance. :)06:02
hazmatSpamapS, fair enough.. at the moment local provider is a perfect playground for failure testing ;-)06:03
hazmatso it definitely will get better soon.. testing connection failures on ec2 was always a bit spotty, but suspend/resume clocks jumps are very nice06:03
SpamapSto be clear, I have not been seeing this with suspend/resume.. just loads of 8+06:04
hazmatSpamapS, but local is always limited to the class of machine for sure in terms of what it can run concurrently.. out of curiosity what where you running in the containers on the laptop when it failed?06:04
SpamapSceph06:04
hazmatSpamapS, how many units?06:04
SpamapS306:04
SpamapSdoing very little06:05
hazmathmm.. that's very odd06:05
SpamapSjust the churn of dpkg/apt06:05
SpamapSthe disk response time goes to hell06:05
hazmati've seen m_3 do a dozen, and i normally get to a half dozen pretty regularly (no ssd)06:05
SpamapS2011-10-19 23:48:40,630:480(0x7fa28ae7f700):ZOO_WARN@zookeeper_interest@1461: Exceeded deadline by 3509ms06:05
SpamapS2011-10-19 23:48:45,173:480(0x7fa28ae7f700):ZOO_WARN@zookeeper_interest@1461: Exceeded deadline by 1210ms06:06
* SpamapS sets up an 'at' job to shut down his m2.xlarge "now + 3 hours"06:08
SpamapShazmat: ok, just got it on my EC2 instance. :(06:47
hazmatSpamapS, how many units?06:48
SpamapSadding two units to my ceph ring06:48
SpamapS2011-10-20 06:47:37,873:272(0x7f217bb29700):ZOO_ERROR@handle_socket_error_msg@1621: Socket [192.168.122.1:46463] zk retcode=-112, errno=116(Stale NFS file handle): sessionId=0x133201298a60003 has expired.06:48
SpamapSload of 0.8406:48
hazmatwhat's odd is that unit addition is serial in the machine agent06:48
SpamapSeach unit does have 4 peer relations06:48
SpamapSnot sure if that plays a factor06:48
hazmatthat shouldn't matter, overall load and avail mem does06:48
hazmatmaybe its just because ceph is experimental06:49
SpamapS16G of memory.. :)06:49
SpamapShaha06:49
SpamapSexperimental service somehow dooms zk? ;)06:49
SpamapSit is a fairly complex charm06:49
hazmatSpamapS, indeed.. but more seriously.. we'll definitly have fixes for this shortly, but the fact that your triggering it so easily with ceph ..06:50
SpamapS3 separate daemons to manage for ceph06:50
SpamapShappens when the new units are setting up during install06:51
SpamapS2011-10-20 06:47:31,529:272(0x7f217bb29700):ZOO_ERROR@handle_socket_error_msg@1528: Socket [192.168.122.1:46463] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 0ms)06:51
SpamapS2011-10-20 06:47:31,523: hook.output@INFO: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode.06:51
SpamapS2011-10-20 06:47:31,523: hook.output@INFO:06:51
SpamapS2011-10-20 06:47:31,530: hook.output@INFO: Setting up libterm-readkey-perl (2.30-4build2) ...06:52
SpamapSactually, it happened before the 2nd unit was even added really06:53
SpamapSOct 20 06:47:31 ip-10-82-145-26 dnsmasq-dhcp[8370]: DHCPDISCOVER(virbr0) 92:40:49:b0:a5:4506:53
SpamapSI wonder..06:53
SpamapShappens when the new containers first request dhcp06:54
SpamapSmaybe this interrupts networking somehow06:54
SpamapShazmat: indeed, regardless of what I'm doing to trigger it.. resiliency to this, and any other unknown weirdness would be the first order of business07:02
SpamapSseems sporadic07:02
* hazmat wants gluster07:03
hazmatan actual production quality clustered file system that works well in virtualized environments, and does data redundancy07:04
SpamapSI think ceph will actually basically destroy gluster.07:10
SpamapSAlready have a few friends who say its impossible to get performance out of it.07:11
SpamapSgluster that is07:11
SpamapSthat said.. juju should make benchmarking them side by side quite easy!07:13
SpamapShazmat: go to sleep!07:23
hazmatSpamapS, that's odd i've seen very good benchmarks of gluster07:25
hazmatand i've used it in production b407:26
SpamapShazmat: I used it briefly before leaving the last place. Had to fight with it to get basics working.07:26
SpamapSAnd then a few weeks after leaving, the guys were picking my brain about lockups and inconsistencies after they had tuned it too far toward the dark side.07:27
SpamapSIt may have been the app and the way it was using the FS..07:27
SpamapSbut to get the app working right the most careful mode had to be used.07:27
SpamapSThey switched to MogileFS and never looked back, since all they really needed was object storage07:28
hazmatperhaps.. its got lots of tunables.. i've used it for wonky things like svn repo sharing to trac instances and large file storage for media transformation07:28
hazmats/distributed media07:28
hazmatanyways time for bed... g'nite07:28
SpamapSThis was millions of images a day churning (pics of home foreclosures, sadly)07:28
SpamapS'nite!07:29
SpamapSwe should have a gluster/ceph shootout once there are charms. :)07:29
hazmat:-)07:29
TeTeThazmat: hi kapil, just reading the network security and dynamic ports thread, i wonder if a sort of 'conflicts' solution like in dpkg would help with this for charms? E.g. two charms both utilize port 80, so they conflict and cannot be co-located ever. While auto detection of this conflict is nice to have, a manual field might be good enough to start08:23
SpamapSTeTeT: heh, thats basically what I think too08:27
SpamapShas worked this way quite well in Debian and Ubuntu08:27
TeTeTSpamapS: working late night? Or are you in europe nowadays?08:28
SpamapSvery late08:28
SpamapStrying to be prepared for UDS and the surrounding madness. :)08:28
SpamapSAlso trying to write a CEPH charm so I can remind myself of all the things complex charm writers need fixed. :)08:29
TeTeTSpamapS: he he, I still haven't gotten around to write a charm myself. now with 11.10 out I consider Landscape Dedicated Server as a test bed08:30
SpamapSthe more eyeballs the merrier08:32
SpamapSanyway, time to go pass out08:37
TeTeTgood night08:38
=== fucrum is now known as fulcrum
fwereadebcsaller: I'm just writing a response to you and hazmat, and I realised I'm unclear about something -- is there a strict distinction between colo and non-colo charms, or are people free to colo whatever they want?13:11
mrsipanwhat happen if  for some reason the zookeeper server dies, is it possible to rebuild it and let the unit services/instances  know  about its (new ip, etc)?14:33
mrsipanor do I need to redeploy it all again?14:34
robbiewmrsipan: I believe presently you have to redeploy14:34
robbiewfwereade: ^?14:35
mrsipanrobbiew: thanks14:35
fwereademrsipan, robbiew is right14:35
fwereademrsipan, we're planning to have high availability by 12.04 but there's nothing there yet14:36
mrsipanfwereade, thanks, that make zookeeper a SPOF for juju14:45
mrsipanfor now14:45
fwereademrsipan: exactly so14:45
fwereademrsipan: be reassured that the HA story is a big and important one :)14:46
robbiewa **VERY** big and important one ;)14:46
mrsipanfwereade, cool, thanks14:46
hazmatfwereade, people are free like birds, any one can be a co-located jail bird... some however are born in prison and destined for a life there (standalone: false)15:23
bcsallerhazmat: how poetic15:26
hazmatbcsaller, predestination is rather harsh15:27
bcsallerI do like that we relaxed the ideas around what can co-located from the original run15:28
robbiewhazmat: ping15:33
hazmatrobbiew, png15:33
backburnerI saw a video demoing juju but now I can't find it. anyone know where one is?15:58
robbiewkim0: ^?16:00
jimbakerfwereade, i think i'm going to change my mind on the doc stuff related to juju ssh. the command itself is not documented. really we need to incorporate more detailed docs on each subcommand in juju16:01
jimbakerfwereade, so that should be done in a separate branch/bug16:02
fwereadejimbaker, sounds sensible to me16:02
fwereadejimbaker, my approve still stands ;)16:02
jimbakerfwereade, perfect :)16:02
_mup_juju/ssh-passthrough r410 committed by jim.baker@canonical.com16:10
_mup_Fix doc typo16:10
_mup_Bug #878948 was filed: need to be able to define a "virtual" or "external" service <juju:New> < https://launchpad.net/bugs/878948 >16:19
jimbakeranyone else seeing this problem when building the docs on trunk with make clean && make html? btw, it works fine with make singlehtml, which is normally how i look at the docs17:02
jimbakerhttp://paste.ubuntu.com/714369/17:02
_mup_juju/ssh-passthrough r411 committed by jim.baker@canonical.com17:15
_mup_Merged trunk17:15
_mup_juju/ssh-passthrough r412 committed by jim.baker@canonical.com17:23
_mup_Addressed review points17:23
_mup_juju/scp-command r412 committed by jim.baker@canonical.com17:24
_mup_Merged upstream17:24
hazmatSpamapS, relating two things at once to haproxy is not a race condition17:42
hazmatSpamapS, the hooks execute serially on haproxy17:42
hazmatat least it shouldn't be a race condition17:42
hazmatlooking at the haproxy charm, it looks fine17:44
SpamapShazmat: it will produce a single service, only pointing to the last one related to18:32
hazmatSpamapS, ah.. its not multi service aware for backing18:32
SpamapSone listen section18:33
SpamapSneeds host header stuff .. which would likely have to come from backend service config options18:33
SpamapSwhich did not exist when I wrote it :)18:33
SpamapShazmat: so, about the capistrano renderer..18:37
SpamapShazmat: if we're not going to accept that, how are we going to recommend people integrate with juju?18:38
hazmatSpamapS, that would subume very easily into a misc command we could distribute if we had a REST interface18:42
hazmator that people could innovate on outside the core18:42
SpamapSI have misc/status2(gource|hostname).py in several branches already.. would like to be able to just write a simple python module as a plugin that does what they do.18:44
SpamapSmine currently poll juju status by execing it (I know they could import juju.control.status but I don't want to assume thats a public API)18:45
hazmatSpamapS, the same problem exists for the capistrano renderer, it would need to poll to be accurate19:14
SpamapSWhich is *fine*19:14
SpamapSI see where its not core to what juju wants to do..19:15
SpamapSbut neither is dot19:15
hazmatbut there are so many other one off consumers of status, it seems like just having a REST interface would solve this19:15
hazmatbut that doesn't solve distribution19:15
SpamapSYeah REST to replace execing juju status.19:15
hazmatsadly i don't think the python plugin stuff would fly (it does have a distribution model though)19:15
SpamapSyou don't have to distribute it. you'd want a capfile on the same boxes you'd want to run juju status on19:15
SpamapSoh19:15
SpamapSdistribution of the tool?19:16
hazmatyup19:16
SpamapSSo, yeah, bzr gets this right, and so does git19:16
SpamapSput something in your ~/.juju that makes it understand this19:16
SpamapSthats what I really want19:16
SpamapSand also the ability to put something in thepythonpath that is like a juju plugin so I can package it up19:17
hazmatyeah.. i'd be happy with easy_install juju_gource && ./bin juju gource19:17
SpamapSerr19:17
SpamapSwe have this distro thing19:17
hazmat;-)19:17
* hazmat sheds a tear19:17
SpamapSbut yeah, it should ultimately be easy_installable so that we can spread it wider than Ubuntu19:18
hazmatits just not going to fly, its non compatible with other impl19:18
SpamapSSo, anyway, point being, if we're going to say "no more status renderers in juju" then we need to start a juju status renderer project.19:18
SpamapSand probably need to rip out dot and png19:19
hazmatthe question is if we add it, where do we draw the line?19:19
SpamapSwe don't :)19:19
SpamapSwe open our arms as wide as they'll go19:19
SpamapSand love everybody. :)19:19
hazmati mean all my friends are using fabric.. do we add fabric output..19:19
hazmatthat's why i was thinking its pretty trivial to replace that with a status2capistrano misc thingy19:19
SpamapSyour point that this is not something core to juju is well taken. Give me the appropriate integration point... right now its juju stats | mything19:20
SpamapSstatus rather19:20
jimbakerthe other thing is, it would be nice if users of status could also watch the /status node introduced by bcsaller's statusd branch, instead of polling19:20
bcsallerjimbaker: yeah, thats the plan19:21
jimbakerbcsaller, good to hear!19:21
jimbakerjuju status --watch ?19:21
bcsalleryeah19:22
hazmatbcsaller, what would that do?19:22
hazmatdelay status output till its changed?19:22
bcsallernot clear how it interacts with --output, but ignoring that it just runs its collect/output when the topo changes19:23
hazmator terminal screen refresh19:23
jimbakerideally it just waits until the change, then immediately returns. long poll. but i suppose it could be convenient if it resets itself again for people just wanting to watch it from a term19:23
SpamapSjust stream it19:23
bcsallerblocking till change is key, otherwise you can just poll with $watch juju status19:24
SpamapSgive it an additional root yaml tree element of    status_snapshot: $unix_epoch19:24
jimbakerbcsaller, agreed. and SpamapS, i think that would be a common case. just want a one-shot option too19:24
SpamapSTho yaml probably doesn't have a SAX like mode19:24
hazmatSpamapS, negronjl  so is piping status to an external command an integration problem?19:24
hazmatSpamapS, its not a stream for sure..19:25
SpamapShazmat: Its fine with me, I like the unix way. I think we can probably write a single tool that is extensible with plugins for any output desired.19:25
SpamapSIf the /status node comes into being, then just execing it with --wait-for-change over and over would be fine since it would block until a change19:26
negronjlAt this point, I see enough friction against the output renderer that it would be good to gave either a plugin system, api or unified tool19:27
jimbakernegronjl, +1 for a plug in system. i know this is something that m_3 really wants to see as well19:27
SpamapSnegronjl: how about we stick our heads together in Orlando and produce a single tool that lives outside of juju?19:28
SpamapSnegronjl: we can make it a Recommends: of juju in Ubuntu. :-D19:28
negronjl+1 on the single tool.19:28
negronjlJust dump said tool in charm-tools.19:29
hazmatSpamapS, negronjl that sounds good, i'd be happy to lend a hand on that, and make it pluggable via pypi19:29
hazmator you could just crib from the cli-plugins.. its pretty trivial19:30
SpamapSI want to keep charm-tools to just the bare minimum for maintaining charm19:30
jimbakerthere's something to be said for juju status supporting a variety of renderers including dot, capistrano, out of the box. having the ability to plugin new output formats into it will just make our necessary limited support there justifiable19:30
hazmatand very functional19:30
negronjlOk with me.  Im sure we can all come together and create something awesome :)19:30
hazmathttps://code.launchpad.net/~hazmat/juju/cli-plugins19:30
SpamapShazmat: !! ?19:31
negronjlA happy medium could be to leave Capistrano in but still create the tool for other formats for now.19:31
SpamapSthis may already have a framework?19:31
hazmatSpamapS, just using entry points into python packages for a plugin, its like a half-dozen lines of code19:31
negronjlOnce the tool is done,  we move the different renderer to the tool19:31
hazmatSpamapS, its been veto'd for juju19:32
SpamapSTrue, the work is done.. why not just accept it and make it clear that it will go away when there is an external alternative?19:32
SpamapShazmat: of course it has. :-/19:32
negronjlhazmat, what has been vetoed?19:33
hazmatpython cli plugins for juju19:34
jimbakerre cli-plugins, short & sweet, nice. we still need to specific support for stuff like status that should support an output format plugin19:34
jimbakerbut that's also short work too19:34
hazmatjimbaker, there already is effectively a registry of outputers, that a plugin could register for19:34
hazmatits just we dont have any internal distinction of internal api vs public api19:34
hazmatso we'd have to do some versioning around the entry point19:35
hazmatthe goal is to have language neutral plugins around a rest interface at some point19:35
jimbakerhazmat, exactly, it's already there, so just need to support19:35
negronjlSpamaps, I think that at this point we can move faster and more efficiently by moving new renderers and such outside of juju and into charm-tools.  At least we could bypass the veto votes :)19:35
jimbakeranyway, +1 for plugins, i hope we can do them, since it's so simple19:36
negronjlhazmat, is there an ETA on the REST interface or API?19:37
hazmatnegronjl, i started it but, its really a backburner, prod tagged issues and colo are the highest priority atm19:37
hazmatspecifically for myself getting the disconnected/expiration stuff going is priority atm19:38
negronjlhazmat, thx19:38
jimbakerindeed, one of the things about the current command setup is that it is almost wants to do plugins, otherwise why not use a class structure instead of modules19:38
hazmatnegronjl, i'd definitely like to see it for 12.04 though19:38
hazmatthere's alot up in the air on resources and goals for 12.04 though19:39
* hazmat wanders off to take a break, bbiab19:40
SpamapSnegronjl: lets call it juju-plus ;)19:43
SpamapSjimbaker: true we coul djust abuse juju.control and write our own commands :-D19:44
jimbakerSpamapS, well, one *could* do that, but i think it's better to avoid some sort of monkey patching. just use a plugin system, that's all19:45
negronjlSpamaps, my sentiments exactly.  We can use existing commands, modules, interfaces, etc. and bypass some of the non-technical issues.19:46
SpamapSI have to run back to the kid stuff now.. but good talk everybody. :)19:47
SpamapSnegronjl: lets chat about this next week for sure.19:47
negronjlSpamaps: sure thing19:48
* hazmat scores four touchpads in the mail20:22
hazmatxmas shopping is done ;-)20:22

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!