| === koolhead17 is now known as koolhead17|home | ||
| poolie | anyone here? | 06:33 |
|---|---|---|
| TeTeT | no developers I fear | 06:49 |
| kim0 | Morning everyone | 09:11 |
| hazmat | good morning | 12:43 |
| kim0 | hazmat: morning :) | 12:46 |
| niemeyer | hazmat, kim0: Yos! | 14:39 |
| niemeyer | hazmat: Good extended weekend? | 14:40 |
| hazmat | niemeyer, indeed.. lots of play time | 14:45 |
| kim0 | hehe | 14:49 |
| kim0 | Love the Ensemble audition program | 14:50 |
| kim0 | wonder if we could have a miniature public version of it (like for a week, to write a formula) | 14:50 |
| kim0 | The aws trial thing was a big success, I suppose we could do something similar | 14:51 |
| niemeyer | kim0: Maybe.. let's see how the program goes | 14:59 |
| === crazed_ is now known as crazed | ||
| === crazed is now known as Guest21517 | ||
| === Guest21517 is now known as crazed | ||
| === crazed is now known as crazed_ | ||
| === crazed_ is now known as crazed | ||
| SpamapS | goood morning ensembladeros | 16:47 |
| kim0 | SpamapS: morning o/ | 16:50 |
| kim0 | ensembladeros .. mm can we pick something easier to say :) | 16:51 |
| SpamapS | Ensemblanators? | 16:52 |
| SpamapS | Ensemblasters | 16:53 |
| jimbaker | visions of the toy story ride at disney in my head right now | 16:54 |
| SpamapS | Ensemble Space Rangers | 17:10 |
| robbiew | bcsaller: thanks for stepping up for the demo at structure...I wasn't thrilled about flying to SF for a day :D | 17:17 |
| bcsaller | robbiew: no problem | 17:17 |
| bcsaller | I like talking with people about the project | 17:17 |
| robbiew | :D | 17:18 |
| robbiew | bcsaller: how'd the cloudcamp go last week? | 17:18 |
| robbiew | (or whatever it was called, heh) | 17:18 |
| bcsaller | at first I was worried that it was too high level with lots of exec types running around, but by the end I think I made some good contacts and generated some interest | 17:19 |
| bcsaller | no one there had a good answer for orchestration, some didn't even understand they'd hit this problem but it became very clear in talking with people that a sysadmin named Tim sitting in the back room wasn't a viable orchestration solution | 17:20 |
| kim0 | SpamapS: | 17:23 |
| kim0 | SpamapS: Ensemblasters actually sounds great :) | 17:23 |
| robbiew | bcsaller: :)...sweet | 17:23 |
| SpamapS | bcsaller: dude, so harsh.. Tim's just tryin to feed his family. ;) | 17:24 |
| robbiew | lol | 17:24 |
| bcsaller | tim shouldn't have to work so hard, we offer better tools | 17:24 |
| SpamapS | True, though without work as a distraction Tim might head back to the bottle.. ;) | 17:26 |
| bcsaller | Tim should go back to enchanting | 17:26 |
| SpamapS | bcsaller: I'm going to a similar event down in SD .. how would you suggest approaching it? | 17:27 |
| jimbaker | at pycon, one reaction i had is that "tim" might want to ssh into an indiv machine and fix it manually. wrong, wrong :) | 17:27 |
| jimbaker | still an attitude that needs to be addressed | 17:27 |
| koolhead17 | beep beep :) | 17:29 |
| kim0 | koolhead17: hey | 17:29 |
| SpamapS | Actually to that point jim, the current way to do it "the right way" is to modify the formula and upgrade the service.. | 17:29 |
| SpamapS | jimbaker: I think there's a need for a parallel ad-hoc command interface | 17:30 |
| koolhead17 | kim0: am heading for niemeyer docs as you suggested. | 17:30 |
| kim0 | koolhead17: cool! | 17:30 |
| bcsaller | SpamapS: I really try to vary to to the audience. I went in, listened to the talks and the questions people had about other things and adjusted what I had to say. This time I was able to focus a lot of talk around the idea "Elasticity is one of the key properties of cloud. As you become elastic the way you need to think about and model services changes..." and so on. It was buzzword compliant enough that I think even the non-t | 17:30 |
| SpamapS | jimbaker: also sometimes it is as simple as un-sticking a stuck service with a gentle kill -9 ;) | 17:30 |
| * niemeyer waves | 17:31 | |
| kim0 | koolhead17: It's Ensemble docs :) you can write there too | 17:31 |
| koolhead17 | hello niemeyer :) | 17:31 |
| SpamapS | bcsaller: what was the format like? "un-conference" where people suggest what they'd like to discuss? | 17:31 |
| jimbaker | SpamapS, agreed for development all of these things are good (so ensemble ssh and other tools we will write). clearly we would hope that ad hoc needs are not necessary outside of dev | 17:31 |
| koolhead17 | kim0: are we waiting for oneiric then we will put a community documentation page for ensemble? | 17:32 |
| kim0 | koolhead17: not really .. docs are here https://ensemble.ubuntu.com/docs/ | 17:32 |
| bcsaller | SpamapS: in this case yes, for example audience might collect a list of questions and then people that think they can answer one or more of those questions might join a panel to field questions from the audience | 17:32 |
| kim0 | koolhead17: I'm just used to the older place | 17:32 |
| koolhead17 | http://people.canonical.com/~niemeyer/ensemble/user-tutorial.html :P | 17:32 |
| jimbaker | SpamapS, i think the real issue was that "tim" might need to not only do some transient changes, but there would be a need to capture from production some permanent fixes too | 17:33 |
| koolhead17 | haha | 17:33 |
| SpamapS | jimbaker: for instance, sometimes you want to restart all of your app servers because you changed a DNS setting and they're using a long TTL.. | 17:33 |
| kim0 | koolhead17: both are almost identical | 17:33 |
| * koolhead17 waves to jimbaker | 17:33 | |
| koolhead17 | kim0: ok | 17:33 |
| bcsaller | I was a little nervous getting up in front of a room full of people when the people on either side tended to be a little more business focused, but it worked out fine | 17:33 |
| SpamapS | jimbaker: yeah, in that instance "Tim" needs to learn how to use his tools better. :) | 17:33 |
| * kim0 hugs Tim | 17:33 | |
| kim0 | hehe | 17:33 |
| jimbaker | koolhead17, hi | 17:33 |
| SpamapS | bcsaller: alright that sounds great. :) | 17:33 |
| bcsaller | SpamapS: the whole thing worked pretty well | 17:34 |
| SpamapS | bcsaller: interesting.. the SD one is co-located with an event called "The Business of Cloud Computing" that is free for end users, but costs $1895.00 for "service providers" | 17:40 |
| bcsaller | This was co-located with another conference as well. I think it's the pattern for cloud camps | 17:41 |
| bcsaller | different messaging between the two as well. MS, IBM and Salesforce are really pushing PaaS at the CTO level | 17:42 |
| bcsaller | but that wasn' | 17:42 |
| bcsaller | wasn't so much what people were there to talk about | 17:42 |
| robbiew | bcsaller: another one in Mountain View -> http://www.devopsdays.org/events/2011-mountainview/proposals/ | 17:43 |
| SpamapS | The attendees list of this business of cloud is quite aheavy w/ C level | 17:43 |
| robbiew | if you're interested ;) | 17:43 |
| SpamapS | CTO, CIO, etc. etc. | 17:43 |
| SpamapS | Devops days is *awesome* | 17:43 |
| robbiew | June 17-18th | 17:43 |
| robbiew | call for proposals deadline is tomorrow | 17:43 |
| SpamapS | the one last year only had ignite talks and panels | 17:44 |
| bcsaller | robbiew: I'll put a proposal in and see what happens | 17:45 |
| robbiew | sweet | 17:45 |
| robbiew | spreading the gospel!!! | 17:45 |
| bcsaller | yes, but I really need to start scheduling time to follow up with people after these things. | 17:46 |
| SpamapS | Definitely should bring them in here too. :) | 17:46 |
| robbiew | SpamapS: +1 | 17:47 |
| SpamapS | ooops.. left my 8 node mediawiki running for 24 hours | 17:47 |
| bcsaller | some of the people I talked with might be too high level for this type of chat though | 17:48 |
| bcsaller | SpamapS: Ensemble audition ftw :) | 17:48 |
| robbiew | anyone in the Boston area?...interested in showing off ensemble thursday at a CloudCamp? http://www.cloudcamp.org/boston/2011-06-02 | 17:49 |
| SpamapS | any idea when 'ensemble set' will land? | 17:49 |
| * robbiew quits soliciting for talks....for now | 17:50 | |
| robbiew | uuuahahahahaaa | 17:50 |
| bcsaller | SpamapS: re: set, its working, should be merging one 1/2 of it today and the rest will be in review, its almost all though the process | 17:51 |
| bcsaller | SpamapS: we were joking that it was had to come up with a simple user visible example. I started with trying to change the blog title in wordpress and ouch, that was a can of worms, it really involves a mysql update and it spiraled out of control from there | 17:52 |
| SpamapS | bcsaller: I'd think with a mysql update it should be *dead* simple | 17:54 |
| SpamapS | bcsaller: mediawiki has me writing PHP files all over the place. :-P | 17:54 |
| bcsaller | SpamapS: set wordress blog-title="foo". Right, so now wordpress has to identify its relation to mysql but config option hooks are not built around relation context, I think if you follow that thread you'll see it gets silly fast | 17:55 |
| bcsaller | writing a PHP file would be a single change with immediate results that don't cross relationship boundaries. might make more sense for this | 17:56 |
| * SpamapS exports the entire "OpenBSD" category from wikipedia to import into his mediawiki | 17:58 | |
| SpamapS | bcsaller: right, this is where I think one thing missing from my current understanding of the model is ordering | 17:59 |
| SpamapS | bcsaller: I think we may need to be able to be able to guarantee ordering of relations | 17:59 |
| SpamapS | there's no sense relating website to a loadbalancer until all the required things are related | 18:00 |
| bcsaller | SpamapS: that's related but not quite where I was heading. | 18:00 |
| SpamapS | and since add-relation doesn't block until the relation exists.. this can be a problem I think. | 18:00 |
| niemeyer | bcsaller: Glad to hear your talk went well | 18:01 |
| SpamapS | bcsaller: Right, I guess what I'm saying is, that setting requires mysql... so settings should error or block until relations are setup. At that point you'd at least know wordpress knows how to talk to mysql. | 18:01 |
| niemeyer | SpamapS: Ordering isn't entirely straightforward, as we discussed back in Cape TOwn | 18:01 |
| bcsaller | SpamapS: relation settings are not even readily available in the config-changed hook as that's not a relation hook | 18:01 |
| SpamapS | niemeyer: indeed.. the stacks may be the place to address that | 18:03 |
| SpamapS | I'm still trying to see if something solves my problem of not being able to relate to my slave database until after it has been related to the master | 18:04 |
| SpamapS | whoa.. I just discovered the resolved command | 18:09 |
| SpamapS | sweeeet | 18:09 |
| SpamapS | heh.. now that I'm importing a massive amount of data into my mediawiki.. I find myself wishing I had a graphing system setup. :) | 18:13 |
| * SpamapS looks into reconnoiter formula | 18:14 | |
| niemeyer | Woot :) | 18:16 |
| SpamapS | niemeyer: what would be cool would be if the EAP program shared a single ensemble environment... so we could all relate all of our services to one another. ;) | 18:18 |
| SpamapS | have one EAP large instance for mysql for everybody's mysql needs. ;) | 18:18 |
| niemeyer | SpamapS: I'm not entirely sure other people would agree with that :-) | 18:20 |
| SpamapS | So here's an interesting conundrum. I want all the machines that I spawn to direct their syslog service to one syslog machine... essentially a system issue, not a service issue directly... | 18:25 |
| SpamapS | i had one thought which is that formulas can have a "management" relation defined, which will run traditional config management type stuff like this.. and then a management formula that does all these tasks | 18:27 |
| SpamapS | the other option is to just make it easy to drop in puppet/chef/cfengine to do these types of things | 18:29 |
| bcsaller | its not yet clear to me how that would fit into the lifecycle or why the current lifecycle can't perform those tasks | 18:30 |
| niemeyer | SpamapS: I'm not sure I get how that's any different from the other relations? | 18:31 |
| niemeyer | SpamapS: Why not a "syslog" relation? | 18:31 |
| SpamapS | niemeyer: because I'd have to define it for *every* formula in principia, and it would be the exact same program | 18:32 |
| SpamapS | ergo, it is at a different level | 18:33 |
| SpamapS | Its basically a machine policy, not a service policy | 18:33 |
| SpamapS | so another option is to provide a machine analog for formulas | 18:34 |
| niemeyer | SpamapS: We have to think about it a bit more.. I think it is a service policy because syslog is a service, but I see your point regarding handling that comfortably. | 18:35 |
| SpamapS | yeah, the receiver is a service... | 18:35 |
| bcsaller | SpamapS: we did talk about something like that, and something else that was provider specific, but nothing is spec'd yet | 18:35 |
| niemeyer | SpamapS: There are class of things which are related to the machine that need further thinking | 18:36 |
| SpamapS | I think making it as narrow and flexible as possible would be ideal. The thing I reclal talking about was "ensemble deploy ... --policy-formula=X" which would deploy another formula on the machine as X | 18:36 |
| SpamapS | But I recall that the implementation of that would be fairly disruptive | 18:37 |
| niemeyer | SpamapS: Yeah, that feels pretty close to ringing a bell | 18:38 |
| SpamapS | Another one that might be a simpler stop-gap would be to allow specifying cloud-config data to add to the initial cloud-config | 18:39 |
| niemeyer | SpamapS: That sounds like something hard to get out off down the road | 18:42 |
| SpamapS | It would at least be easier to migrate away from than AMI's, which is the other way I could see people solving it | 18:42 |
| niemeyer | SpamapS: We have to spend some time thinking about these issues, collecting a few use cases, and then sit down to design it properly and start implementing the barebones functionality | 18:43 |
| niemeyer | bcsaller, jimbaker, hazmat: Standup? | 19:31 |
| bcsaller | yeah | 19:31 |
| hazmat | sounds good | 19:32 |
| jimbaker | niemeyer, sure | 19:32 |
| hazmat | hmm.. skype still segfaulting on me | 19:32 |
| hazmat | hanging out in #ensemble on mumble | 19:33 |
| niemeyer | I'm happy to go with Mumble | 19:33 |
| _mup_ | ensemble/expose-provision-service-hierarchy r252 committed by jim.baker@canonical.com | 19:33 |
| _mup_ | Test corner case that service has been removed between watch and the watch function execution | 19:33 |
| niemeyer | Not sure for how long we'll be able to count with Skype either way | 19:33 |
| * SpamapS deploys his munin formula.. | 20:32 | |
| niemeyer | Woah | 21:35 |
| niemeyer | SpamapS: How did it go? | 21:35 |
| SpamapS | niemeyer: working out kinks | 21:35 |
| SpamapS | niemeyer: its doing everything I told it to.. but munin isn't picking up my file in /etc/munin/munin-conf.d .. have to figure that out. | 21:36 |
| SpamapS | ah! the dreaded type-o in the bash script problem :) | 21:38 |
| niemeyer | :-) | 21:41 |
| niemeyer | SpamapS: We need to compile bash! | 21:41 |
| * SpamapS imagines niemeyer writing a Go parser for bash | 21:44 | |
| niemeyer | SpamapS: Oh man.. I don't want to get anywhere near that | 21:45 |
| robbiew | lol | 21:52 |
| _mup_ | ensemble/trunk r239 committed by gustavo@niemeyer.net | 22:06 |
| _mup_ | Include examples as documentation. | 22:06 |
| niemeyer | hazmat: Packages churning | 22:09 |
| hazmat | niemeyer, awesome, thanks | 22:10 |
| niemeyer | hazmat: np | 22:10 |
| SpamapS | http://ec2-50-17-114-201.compute-1.amazonaws.com/munin/ | 22:12 |
| niemeyer | \o/ | 22:19 |
| SpamapS | It should gain mysql stats too.. qps, cache hits, etc. | 22:20 |
| niemeyer | hazmat: pkgs are up for all Ubuntu releases | 22:22 |
| SpamapS | hrm I think I hit a weird bug in the agent | 22:28 |
| SpamapS | http://paste.ubuntu.com/615517/ | 22:28 |
| SpamapS | load got crazy high on the box, I think that actually may be have what caused the issue | 22:29 |
| SpamapS | it was streaming those stack traces | 22:29 |
| _mup_ | ensemble/expose-provision-service-hierarchy r253 committed by jim.baker@canonical.com | 22:30 |
| _mup_ | Don't ignore watch_exposed_flag problem, fix it | 22:30 |
| niemeyer | SpamapS: Yeah, that's a weird traceback | 22:31 |
| niemeyer | SpamapS: Do you have the top of it? | 22:32 |
| _mup_ | ensemble/expose-provision-service-hierarchy r254 committed by jim.baker@canonical.com | 22:32 |
| _mup_ | Merged trunk | 22:32 |
| SpamapS | niemeyer: should be in the agent log right? | 22:34 |
| niemeyer | SpamapS: Yeah | 22:35 |
| niemeyer | SpamapS: These two tracebacks are just a side effect of something else that happened earlier | 22:35 |
| SpamapS | niemeyer: I think the agent may have crashed | 22:36 |
| niemeyer | SpamapS: Yeah, the traceback certainly looks bad | 22:36 |
| SpamapS | the new log is very small and from about 2 minutes before the traceback I pasted | 22:36 |
| niemeyer | SpamapS: In a weird way.. that's twisted complaining about something strange | 22:36 |
| SpamapS | ahh I was running debug-log when it started... | 22:36 |
| niemeyer | SpamapS: Hmm | 22:36 |
| SpamapS | http://paste.ubuntu.com/615520/ | 22:38 |
| SpamapS | seems like it missed the top | 22:38 |
| niemeyer | SpamapS: There's actually some interesting info there that I missed earlier | 22:39 |
| niemeyer | hazmat: Would you mind to have a look at this when you have a moment? | 22:39 |
| niemeyer | hazmat: The first traceback paste is within exists_and_watch, which is breaking due to the deferred being called twice | 22:40 |
| hazmat | niemeyer, looking | 22:40 |
| niemeyer | hazmat: I suspect it may have something to do with the recent refactorings | 22:40 |
| hazmat | interesting | 22:40 |
| SpamapS | if you guys want to login to the box or anything let me know | 22:40 |
| SpamapS | relations don't seem to be working to the service anymore | 22:40 |
| hazmat | SpamapS, thanks | 22:40 |
| niemeyer | SpamapS: That's understandable, thanks | 22:40 |
| niemeyer | SpamapS: The watching within the agent is borked, so it'll not behave properly | 22:41 |
| SpamapS | can it be restarted or anything? | 22:42 |
| hazmat | niemeyer, not related to the refactoring just a new event type i think that wasnt in the txzookeeper event mapping for pretty names, which we got exercised by a log statement | 22:42 |
| niemeyer | SpamapS: Which agent was that | 22:42 |
| niemeyer | SpamapS: You can restart it either way, but may need some env setup | 22:42 |
| hazmat | SpamapS, it can be restarted by hand, but its tedious you have to setup the /proc/pid/env and launch with the same cmdline | 22:43 |
| SpamapS | ah | 22:43 |
| SpamapS | reboot? | 22:43 |
| niemeyer | We need to work on this | 22:43 |
| niemeyer | hazmat: new event type? | 22:43 |
| hazmat | niemeyer, its not a new type.. just one that wasn't previously mapped | 22:43 |
| niemeyer | hazmat: A deferred was called twice.. that sounds like wrong wired of deferred chaining | 22:43 |
| niemeyer | s/wired/wiring | 22:43 |
| hazmat | niemeyer, its caused by a SESSION_EVENT | 22:44 |
| hazmat | niemeyer, indeed that too | 22:44 |
| niemeyer | hazmat: How do you mean? | 22:44 |
| niemeyer | hazmat: File "txzookeeper/client.py", line 393, in callback | 22:44 |
| hazmat | ah.. actually no the logging is async | 22:44 |
| hazmat | File "txzookeeper/client.py", line 79, in type_name | 22:44 |
| hazmat | return self.type_name_map[self.type] | 22:44 |
| hazmat | exceptions.KeyError: -1 | 22:44 |
| niemeyer | hazmat: This is within exists_and_watch | 22:44 |
| SpamapS | Ok well ec2-50-17-114-201.compute-1.amazonaws.com is the hostname, I added keys for hazmat and niemeyer (from launchpad) | 22:44 |
| hazmat | ah.. there are two tracebacks posted | 22:44 |
| SpamapS | Still haven't grabbed that lunch.. ;) | 22:45 |
| * SpamapS runs out for it | 22:45 | |
| niemeyer | SpamapS: Enjoy, and thanks! | 22:45 |
| hazmat | niemeyer, i was referring to the second traceback | 22:45 |
| hazmat | just looking at the first | 22:45 |
| hazmat | hmmm.. the tailspin might have been a recursive error logging loop | 22:46 |
| niemeyer | hazmat: Thanks | 22:46 |
| niemeyer | hazmat: I'll leave that with you.. feeling very sleepy right now | 22:47 |
| hazmat | niemeyer, get some sleep.. and thanks for setting up the ppa, much nicer to demo now | 22:47 |
| niemeyer | hazmat: Will do, and np | 22:47 |
| niemeyer | Laters | 22:48 |
| _mup_ | txzookeeper/fix-event-type-name-mapping r36 committed by kapil.foss@gmail.com | 22:55 |
| _mup_ | add some missing events to the event type name mapping | 22:55 |
| hazmat | hmm.. this is a bug in the ensemble watch usage.. http://zookeeper-user.578899.n2.nabble.com/watcher-semantics-for-session-events-in-the-C-client-td6206081.html | 23:00 |
| hazmat | not problematic for this particular case, which is fixed by the above commit | 23:00 |
| hazmat | hmm. i guess its its not really an issue since all of our watches refetch current state | 23:00 |
| hazmat | but it is an additional event firing which they most don't account for as spurious to their watch intent | 23:01 |
| hazmat | niemeyer, jimbaker, bcsaller ^ | 23:01 |
| hazmat | so given our usage its fine, given that we check state | 23:04 |
| hazmat | hmm.. actually we don't do it correctly, since in this case we also reset the watch | 23:05 |
| hazmat | hmmm | 23:08 |
| hazmat | it looks like we ran into this already.. http://comments.gmane.org/gmane.comp.java.hadoop.zookeeper.user/1951 | 23:15 |
| hazmat | but we never addressed it in our usage afaics | 23:15 |
| hazmat | hmmm.. this feels like something we should set on the client | 23:21 |
| hazmat | bcsaller, jimbaker could i get a +1 on this trivial.. http://paste.ubuntu.com/615542/ | 23:25 |
| bcsaller | hazmat: that's just mapping constants? | 23:26 |
| hazmat | bcsaller, yup | 23:26 |
| bcsaller | +1 | 23:26 |
| _mup_ | txzookeeper/trunk r38 committed by kapil.foss@gmail.com | 23:29 |
| _mup_ | Add some missing constants to the client event type mapping. [trivial][r=bcsaller] | 23:29 |
| SpamapS | hazmat: any progress on that problem? I am going to tear down the box if you don't need it | 23:31 |
| hazmat | SpamapS, i committed a fix for the immediate cause (that change to txzookeeper), i'm still trying to figure out if we're handling the event that was received properly or not (the error was from printing the event), it looks like that machine got disconnected from zk for a little bit | 23:33 |
| hazmat | SpamapS, feel free to tear down the machine | 23:33 |
| SpamapS | hazmat: thanks :) | 23:33 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!