| niemeyer | Today was more difficult than usual.. it's funny how things aggregate around a similar time frame every once in a while | 00:21 |
|---|---|---|
| _mup_ | ensemble/expose-provision-service-hierarchy r265 committed by jim.baker@canonical.com | 03:30 |
| _mup_ | Merged upstream expose-watch-exposed-flag | 03:30 |
| === jml` is now known as jml | ||
| kim0 | I'm pretty sure ensemble-log and set -eux are not logging for me (by default?) in the install hook | 13:31 |
| TREllis | 20 | 13:40 |
| TREllis | bah | 13:40 |
| kim0 | a ha .. I think I stumbled over a serious bug | 13:53 |
| kim0 | Here are the steps to reproduce: | 13:53 |
| kim0 | - deploy a formula with an error in its install hook | 13:53 |
| kim0 | - state becomes, install_error | 13:54 |
| kim0 | - launch debug-hooks | 13:54 |
| kim0 | - launch resolved -r <SU> | 13:54 |
| kim0 | - Broken hook is debugged under tmux | 13:54 |
| kim0 | - Now "status" mentions state is "installed" not "started"! | 13:55 |
| kim0 | Now I get the following traceback http://paste.ubuntu.com/620867/ | 13:55 |
| kim0 | and issuing add-relation .. doesn't really help move it from state installed | 13:56 |
| kim0 | niemeyer: hazmat ^ | 13:56 |
| kim0 | It seems the state-machine is stuck | 13:57 |
| niemeyer | kim0: Yo | 14:01 |
| kim0 | hey :) | 14:01 |
| niemeyer | kim0: Intresting | 14:02 |
| kim0 | indeed | 14:02 |
| kim0 | any manual way to move it into started ? | 14:02 |
| kim0 | so I can continue writing the tutorial :) | 14:02 |
| niemeyer | kim0: Yeah, it looks like a real bug indeed | 14:02 |
| kim0 | cool .. guess I'll just recreate a fresh env | 14:02 |
| kim0 | One note | 14:03 |
| niemeyer | kim0: Let's wait for hazmat.. he's coded that recently, and will likely be able to spot/fix what's up in no time | 14:03 |
| kim0 | awesome | 14:03 |
| kim0 | I also think the current functionality of fixing a broken hook is seriously lacking | 14:03 |
| kim0 | - fix remotely .. retry | 14:04 |
| kim0 | - copy to local machine | 14:04 |
| kim0 | - upgrade | 14:04 |
| kim0 | I wish debug-hooks would just be connected to local hooks dir some how | 14:05 |
| niemeyer | kim0: Hmm.. that's a bit strange.. why? | 14:06 |
| niemeyer | kim0: Oh, for copying? | 14:06 |
| niemeyer | kim0: Remote => local? | 14:06 |
| kim0 | and then upgrading | 14:06 |
| niemeyer | Ok, interesting | 14:06 |
| niemeyer | ahasenack, sidnei, zaid_h, wrtp: Hi all! | 15:48 |
| sidnei | heya niemeyer | 15:49 |
| zaid_h | o/ | 15:49 |
| wrtp | hi niemeyer | 15:52 |
| rockstar | Is ensemble going to be put up on pypi? | 15:53 |
| niemeyer | rockstar: Hmm | 16:00 |
| niemeyer | rockstar: I hadn't thought of that | 16:01 |
| niemeyer | rockstar: I guess it's doable, but I'm not sure about what's the real benefit when compared to having the PPA up | 16:01 |
| niemeyer | rockstar: What's your thinking? | 16:01 |
| rockstar | niemeyer, well, you want people other than Ubuntu folk using Ensemble, right? | 16:02 |
| rockstar | It's kind of odd to say "you can use whatever language you want to write Ensemble formulas as long as you use our OS" :) | 16:03 |
| niemeyer | rockstar: That sounds good for sure, but our current focus is on Ubuntu obviously | 16:03 |
| rockstar | niemeyer, yeah, for deployment, it's a no-brainer. | 16:03 |
| niemeyer | rockstar: Ah, I see what you mean | 16:03 |
| rockstar | But for the ensemble controller, it'd be nice to use what you want. | 16:03 |
| niemeyer | rockstar: Yeah, that sounds nice indeed | 16:03 |
| rockstar | The reason I ask is that I have people at the local LUG asking if I could give a demo at next month's meeting, so I wanted to know about usage. | 16:04 |
| niemeyer | rockstar: Cool | 16:04 |
| rockstar | niemeyer, I'd be happy to submit patches for any bugs I find on other OSes. | 16:04 |
| niemeyer | rockstar: It shouldn't be hard, actually | 16:04 |
| niemeyer | rockstar: Ensemble is really just a Python library at this point, with a tiny executable wrapper | 16:05 |
| rockstar | niemeyer, it seems relatively ready for pypi, that's why I asked. | 16:05 |
| niemeyer | rockstar: The dependencies are likely the boring part, though | 16:05 |
| rockstar | It didn't seem like a technical barrier, so I wondered if there were principle issues. | 16:05 |
| niemeyer | rockstar: I mean, I'm not sure python-zookeeper exists there, for instance | 16:05 |
| rockstar | niemeyer, there's txzookeeper | 16:05 |
| rockstar | I don't know if that's the same thing. | 16:05 |
| niemeyer | rockstar: Yeah, but that's an easier one | 16:05 |
| niemeyer | rockstar: That's a Python-only wrapper for Twisted | 16:06 |
| niemeyer | rockstar: It still needs python-zookeeper on the backend | 16:06 |
| rockstar | Ah. | 16:06 |
| rockstar | I know txAWS is. | 16:06 |
| niemeyer | rockstar: That's helpful already, and it's "the real thing" | 16:08 |
| jimbaker` | rockstar, fwiw, i used to run the entire test suite on mac os - there was only one minor bug, around a small diff in how temp files are named | 16:23 |
| jimbaker` | i didn't really look into it, but i believe it was purely a problem in the test | 16:24 |
| kim0 | hazmat: Hi .. are you there | 16:27 |
| * niemeyer heads to lunch | 16:28 | |
| kim0 | hazmat: just pinging you to check the bug I had reported earlier today, ping me if you have any questions | 16:28 |
| hazmat | kim0, yeah | 16:43 |
| * hazmat pokes | 16:43 | |
| kim0 | cool | 16:43 |
| === deryck is now known as deryck[lunch] | ||
| _mup_ | ensemble/resolved-install-should-start r240 committed by kapil.thangavelu@canonical.com | 17:05 |
| _mup_ | on retry install (with or without hooks) use a success transition id to automatically transition to started. | 17:05 |
| hazmat | kim0, is there a bug reported for the issue you mentioned earlier? | 17:07 |
| _mup_ | Bug #794129 was filed: install hook error, does not transition to start after resolved <Ensemble:New> < https://launchpad.net/bugs/794129 > | 17:09 |
| hazmat | there is now ;-) | 17:10 |
| kim0 | hazmat: cool :) yeah I hadnt opened one | 17:12 |
| hazmat | kim0, cool, the fix is in the review queue now | 17:12 |
| kim0 | hazmat: oh that fast! awesome :) | 17:12 |
| hazmat | kim0, TDD ftw | 17:13 |
| kim0 | niemeyer: Just linked a rewrite of the formula writer's tutorial to bug 788255 | 17:14 |
| _mup_ | Bug #788255: Ensemble needs a tutorial for writing a formula <Ensemble:In Progress by kim0> < https://launchpad.net/bugs/788255 > | 17:14 |
| kim0 | would be great if you could review it soonish, since I think I'll need to refer to it soon in a blog post | 17:14 |
| kim0 | thanks a lot | 17:14 |
| * hazmat admires steve job's reality distortion field | 17:26 | |
| hazmat | the truth is in the cloud.. the cloud, it just works.. | 17:27 |
| * niemeyer waves | 17:32 | |
| niemeyer | hazmat: Yeah :-) | 17:32 |
| niemeyer | Cloud just got 30% more hyped | 17:33 |
| SpamapS | So.. plugins.. agents.. | 17:33 |
| hazmat | SpamapS, plugins.. cli or agent? | 17:34 |
| SpamapS | I've been thinking about how mcollective works. | 17:34 |
| SpamapS | And how easy that would be to add to ensemble. | 17:34 |
| hazmat | SpamapS, agent side plugins then | 17:34 |
| SpamapS | yes | 17:34 |
| SpamapS | ensemble agent-plugin --service=wki-db --plugin=mysql-cmd "FLUSH LOGS" | 17:35 |
| hazmat | SpamapS, that would be nice, but we'd have to some dynamic dispatch to allow ready overriding of behavior for fully pluggable.. else we can manage them as some sort of collection of objects with lifecycle, which work if their mostly independent from existing functionality and using a watch based protocol | 17:36 |
| SpamapS | I mean, we have an agent, with an event driven comm channel to it. We should be able to use it to run stuff. :) | 17:36 |
| SpamapS | hazmat: again, eep op glorp glok.. I am sorry that I don't understand the internals of ensemble better by now. How you do it is very interesting to me, but even more is, how feasible is it? | 17:37 |
| hazmat | SpamapS, its quite feasible.. we have most of the building blocks available, we could isolate the plugins to separate processes if we want. | 17:38 |
| SpamapS | We could just say nah we're not going to do that, but then we need to help people setup mcollective or parallel-ssh or something to do these things. | 17:38 |
| niemeyer | SpamapS: We have to look at a few different use cases to see what would be the right thing to do | 17:39 |
| niemeyer | SpamapS: Flushing logs, for instance, sounds like something we'd want in the agent itself default | 17:39 |
| niemeyer | s/default/by default/ | 17:39 |
| hazmat | niemeyer, then it would need to be formula driven.. the agent is agnostic | 17:39 |
| niemeyer | hazmat: Log management sounds like a pretty general problem | 17:40 |
| hazmat | oh.. i guess a generic log flush is feasible | 17:40 |
| SpamapS | niemeyer: flushing the mysql logs? | 17:40 |
| hazmat | niemeyer, indeed, definitely also puts us pretty close into policy land in the agents | 17:41 |
| niemeyer | SpamapS: Flushing the X logs | 17:41 |
| niemeyer | SpamapS: Where X is whatever is running in the machine | 17:41 |
| hazmat | where do the logs go, what log values do we relay. do we have an alert/event engine feeding on the logs.. etc | 17:41 |
| SpamapS | We can't possibly know everything that every formula will want to expose to users on an arbitrary basis. In cassandra, there's DB compaction, log flushing, offlining for upgrades.. | 17:41 |
| SpamapS | I do like the idea of being able to provide arbitrary hooks in formulas | 17:42 |
| niemeyer | SpamapS: DB compaction is formula business.. log flushing is a generic problem we can tackle on the agent.. offlining for upgrade is formula business to | 17:42 |
| niemeyer | s/to/too | 17:42 |
| SpamapS | niemeyer: DB compaction happens when the admin wants to do it for Cassandra... | 17:42 |
| SpamapS | how would formulas do that now? | 17:43 |
| niemeyer | SpamapS: Sounds like a setting | 17:43 |
| hazmat | SpamapS, cron job from setting | 17:43 |
| SpamapS | no no no no cron! | 17:43 |
| niemeyer | SpamapS: ensemble set run-compaction=true | 17:43 |
| hazmat | :-) | 17:43 |
| SpamapS | niemeyer: THAT I like | 17:43 |
| hazmat | yeah.. cassandra has lots of knobs that it wants to expose the admin, that need to be turned in the right order | 17:43 |
| SpamapS | hazmat: while standing on the right foot | 17:44 |
| hazmat | niemeyer, with the flag being cleared post compaction? | 17:44 |
| SpamapS | Cassandra and Jenkins are both going to need a lot of the same knobs for the JVM .. I've been wondering about shared code. | 17:44 |
| niemeyer | hazmat: Yeah, possibly | 17:45 |
| SpamapS | Ultimately I'd want feedback from that setting. | 17:45 |
| SpamapS | not necessarily from the 'set' command, but somehow I need to be able to know that it ran and failed/succeeded | 17:45 |
| hazmat | SpamapS, i think we need some notion of hook inheritance.. it more tightly couples the formula namespace and inter formula deps.. but i dislike pushing down every hook into all formulas | 17:45 |
| niemeyer | hazmat: It sounds like we can introduce a specific type of setting for that kind of flag | 17:45 |
| niemeyer | SpamapS: Agreed | 17:45 |
| niemeyer | SpamapS: What kind of feedback? | 17:45 |
| hazmat | niemeyer, perhaps.. alternatively we have some sort of service agent (not unit) that can operate on a wholistic service view | 17:46 |
| niemeyer | hazmat: Agreed, except s/sinheritance/composition/ | 17:46 |
| hazmat | ie. do round robin upgrades, rolling upgrades, clear the compact flag after its done on all units | 17:46 |
| niemeyer | hazmat: Hooks are scripts, it's trivial to bundle a package with things for well known activities | 17:46 |
| hazmat | niemeyer, indeed, it is composition | 17:46 |
| SpamapS | Yeah I was thinking that a simple ensemble-formula-helpers package would do nicely | 17:47 |
| SpamapS | think debhelper. :) | 17:47 |
| niemeyer | hazmat: Rather than clearing the flag, it feels like it should be a timestamp based setting | 17:47 |
| hazmat | its convience and reuse, one hook copy to edit, vs. 30 if your upgrading/changing a behavior | 17:48 |
| niemeyer | hazmat: Something resembling this: ensemble set ... compact=now | 17:48 |
| hazmat | niemeyer, sounds good re set | 17:48 |
| niemeyer | hazmat: So each unit can observe the setting, and tell if it should compact or not | 17:48 |
| niemeyer | hazmat: No races, etc | 17:48 |
| niemeyer | no need to reset too | 17:49 |
| SpamapS | like just have the formula's setting hook call /usr/share/ensemble/helpers/hooks/jvm_settings .. | 17:49 |
| hazmat | niemeyer, if we don't reset to reflect current state, then we have a temporal disconnect for a unit coming back online or new unit.. that wouldn't nesc. need compaction | 17:50 |
| niemeyer | SpamapS: Yeah! Man, it's neat that this would just work.. | 17:50 |
| niemeyer | SpamapS: I mean, in the sense that the helper will have the environment | 17:50 |
| SpamapS | yep | 17:50 |
| niemeyer | hazmat: sorry, I actually meant compact=`date` | 17:50 |
| SpamapS | keeping it simple and clear means more people will make use of them | 17:51 |
| niemeyer | hazmat: Not with that syntax, of course, but that's the idea | 17:51 |
| hazmat | niemeyer, cool, but even then you need coordination by something external to the units | 17:51 |
| niemeyer | hazmat: Hm? | 17:51 |
| hazmat | you don't want to compact on your cassandra cluster in parallel | 17:51 |
| hazmat | you do it piecemeal because your putting an additional load on your data store | 17:51 |
| niemeyer | hazmat: That's up to the formula to coordinate internally.. that's the kind of thing we have peer relations for | 17:51 |
| hazmat | they can either coordinate or be coordinate by an external actor | 17:51 |
| SpamapS | as long as hooks can change the settings they've been given, that should be fine | 17:52 |
| hazmat | niemeyer, true | 17:52 |
| niemeyer | hazmat: We can help, of course, but as far as the "how do I tell things should happen", the setting system should be a good start | 17:52 |
| _mup_ | ensemble/expose-provision-service-hierarchy r266 committed by jim.baker@canonical.com | 17:53 |
| _mup_ | Need more robust ending assertions that don't eventually fail with pending deferreds, but non edge case code now works properly with new stateful semantics for watch_exposed_flag | 17:53 |
| niemeyer | SpamapS: It doesn't have to change the _settings_, I think we shouldn't allow it to, but there are ways it can store and relay information (peer relations, etc) | 17:54 |
| SpamapS | One thing that mcollective has that this wouldn't give me, that sysadmins *love* is that the output of the agents can be structured and piped so admins can do their grep/sed/awk magic with the data returned. | 17:54 |
| niemeyer | SpamapS: The idea so far is to reserve the settings framework for human interaction | 17:54 |
| SpamapS | niemeyer: the run-compaction=1 thing would then run compaction on adding unit though, would it not? | 17:58 |
| niemeyer | SpamapS: Yeah, we should add an easier way as discussed with hazmat above.. we could have something equivalent to run-compaction=`date` | 17:59 |
| niemeyer | SpamapS: So units can compare the timestamp | 17:59 |
| niemeyer | SpamapS: and tell if they're supposed to compact or not | 17:59 |
| SpamapS | sounds like we need a "message to the unit" interface not a "set a setting" interface | 17:59 |
| niemeyer | SpamapS: This also solves a number of other issues.. e.g. which units have already compacted? What if I ask to compact again? etc | 17:59 |
| niemeyer | SpamapS: "set a setting" is effectively a "message to the unit" interface | 18:00 |
| niemeyer | SpamapS: With the advantage that messages are not lost | 18:00 |
| niemeyer | SpamapS: Nor duplicated, etc | 18:00 |
| SpamapS | that can make it more complex too since sometimes you mean "send it to the service" and sometimes you mean "send it to the units that exist now" | 18:01 |
| niemeyer | SpamapS: "the service" in this sense is an abstract concept | 18:01 |
| niemeyer | SpamapS: Communication will always take place with the underlying units | 18:01 |
| niemeyer | SpamapS: I get your point, though, I think.. one has to understand how it works | 18:02 |
| niemeyer | SpamapS: That said, this would likely be true either way.. | 18:02 |
| niemeyer | SpamapS: E.g. how to tell all units they have to compact, but they should do so one at a time, etc | 18:03 |
| niemeyer | SpamapS: Whatever system we invent for facilitating this will still need some understanding of the issue at hand by the formula author | 18:03 |
| SpamapS | definitely.. I'm sure it will remain as simple as the other bits of the formula too. :) | 18:04 |
| niemeyer | SpamapS: Yeah, hopefully we'll evolve to that :) | 18:08 |
| === deryck[lunch] is now known as deryck | ||
| niemeyer | We have a _gigantic_ review queue on Ensemble.. | 18:24 |
| niemeyer | That's _awesome_.. :-) | 18:24 |
| niemeyer | Just finishing some Go packaging work to upgrade to the latest weekly, and will do a full round. | 18:24 |
| _mup_ | ensemble/trunk-merge r223 committed by kapil.thangavelu@canonical.com | 18:42 |
| _mup_ | merge trunk | 18:42 |
| _mup_ | ensemble/auto-dependency-spec r224 committed by kapil.thangavelu@canonical.com | 18:45 |
| _mup_ | merge auto dep resolution spec | 18:45 |
| _mup_ | ensemble/trunk r241 committed by kapil.thangavelu@canonical.com | 18:53 |
| _mup_ | merge auto-dependency-resolution prototype [r=niemeyer][f=777816] | 18:53 |
| _mup_ | An auto resolution algorithm implementation for deployment of | 18:53 |
| _mup_ | dependent formulas and relations with constraints satisified from both | 18:53 |
| _mup_ | repository and environment. | 18:53 |
| _mup_ | ensemble/trunk r242 committed by kapil.thangavelu@canonical.com | 18:57 |
| _mup_ | merge ensemble-deploy-auto-resolve prototype [r=niemeyer][f=788735] | 18:57 |
| _mup_ | Integration of auto dependency resolution of formulas and relations | 18:57 |
| _mup_ | into the deploy cli. | 18:57 |
| _mup_ | ensemble/trunk r243 committed by kapil.thangavelu@canonical.com | 18:59 |
| _mup_ | remove auto dependency resolution functionality, pending repository work, left in history for future ref. | 18:59 |
| _mup_ | ensemble/expose-provision-service-hierarchy r267 committed by jim.baker@canonical.com | 19:03 |
| _mup_ | Ensure test_watche_services does not tear down until exposed service watches have been observed to raise StopWatcher | 19:03 |
| hazmat | niemeyer, bcsaller, jimbaker` standup? | 19:08 |
| niemeyer | Do we need one today? | 19:08 |
| jimbaker` | hazmat, sure | 19:08 |
| niemeyer | I'm a bit behind myself, and feel slow in the last couple of days | 19:08 |
| hazmat | niemeyer, i'm good without | 19:08 |
| jimbaker` | no need on my part - i'm just reworking the timing issues introduced by the new stateful api for watch_exposed_flag | 19:09 |
| hazmat | jimbaker`, bcsaller any blockers or other stuff you guys want to talk about today? | 19:09 |
| niemeyer | If there are no important topics to sync upon/figure out, I'd prefer to keep hacking/reviewing/etc | 19:09 |
| jimbaker` | was slow going this morning, but now i'm making good progress | 19:09 |
| jimbaker` | in fact, excellent progress, so would like to focus on it | 19:09 |
| hazmat | yeah.. me too | 19:09 |
| bcsaller | yeah, I just need to finish what I'm working on | 19:09 |
| hazmat | all right.. tomorrow then | 19:09 |
| niemeyer | Super | 19:10 |
| _mup_ | ensemble/expose-provision-service-hierarchy r268 committed by jim.baker@canonical.com | 19:17 |
| _mup_ | Looping on all tests except test_watch_service_changes_removed_services, which needs to take in account new semantics | 19:17 |
| _mup_ | ensemble/expose-provision-service-hierarchy r269 committed by jim.baker@canonical.com | 19:24 |
| _mup_ | Fixed last test for new semantics | 19:24 |
| _mup_ | ensemble/security-specification r240 committed by kapil.thangavelu@canonical.com | 20:21 |
| _mup_ | security overview and todo spec draft | 20:21 |
| niemeyer | Okay.. packages out of the way.. will step out for a moment and come back to that review queue | 20:22 |
| niemeyer | hazmat: What was the outcome of our debate about the db-relation-joined issue on mysql again? | 20:57 |
| niemeyer | hazmat: The one re. the fact it wasn't updating on re-joins | 20:57 |
| hazmat | niemeyer, that the formula hook should check if credentials are already set on its relation, and if not it should create them, even if the db exists | 20:58 |
| niemeyer | hazmat: Ah, with the relation id thing, ok | 20:59 |
| niemeyer | hazmat: Thanks | 20:59 |
| hazmat | niemeyer, np | 20:59 |
| _mup_ | ensemble/trunk r244 committed by gustavo@niemeyer.net | 21:10 |
| _mup_ | Merged fix-mysql-example-formula-publish-details-precreated-databases branch | 21:10 |
| _mup_ | [a=kim0][r=niemeyer] | 21:10 |
| _mup_ | This enables reusing the same example mysql formula multiple times | 21:10 |
| _mup_ | when deployed. | 21:10 |
| _mup_ | The proper way to fix this is actually to use a database named after the | 21:10 |
| _mup_ | relation id, and then create a new database on every new relation id we | 21:10 |
| _mup_ | get. The problem is that we currently don't offer such a relation id for | 21:10 |
| _mup_ | the hooks, so this will do for now. | 21:10 |
| SpamapS | re the formulas re-creating credentials.. the principia mysql formula touches a flag file when the relation is broken, and revokes the credentials at that time. When the flag is present, or database is missing, it creates new credentials. | 21:59 |
| SpamapS | It has an additional check if its a slave since the db will be present but the credentials will be wrong in the db and need updating. | 21:59 |
| _mup_ | ensemble/trunk r245 committed by gustavo@niemeyer.net | 22:34 |
| _mup_ | Merged micro-is-sluggish branch [r=hazmat] | 22:34 |
| _mup_ | This changes the default instance type on EC2 to m1.small. | 22:34 |
| _mup_ | ensemble/expose-provision-service-hierarchy r271 committed by jim.baker@canonical.com | 22:46 |
| _mup_ | Additional cleanup re removing unnecessary observers | 22:46 |
| _mup_ | ensemble/trunk r246 committed by gustavo@niemeyer.net | 22:50 |
| _mup_ | Merge write-formula-tutorial branch [a=kim0] [r=niemeyer] | 22:50 |
| _mup_ | Yes, a formula writing tutorial! | 22:50 |
| kim0 | _mup_: are you being sarcastic :D | 22:53 |
| niemeyer | kim0: Not at all.. :-) | 22:54 |
| niemeyer | kim0: Something important we were lacking | 22:55 |
| kim0 | hehe cool then :) | 22:55 |
| _mup_ | ensemble/config-set r240 committed by bcsaller@gmail.com | 23:37 |
| _mup_ | merge trunk | 23:37 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!