| _mup_ | ensemble/config-set r218 committed by bcsaller@gmail.com | 00:07 |
|---|---|---|
| _mup_ | options validation | 00:07 |
| _mup_ | ensemble/expose-provisioning r258 committed by jim.baker@canonical.com | 00:58 |
| _mup_ | Partial fix to remove more state in the provisioning agent in favor of just reading it from ZK | 00:58 |
| _mup_ | ensemble/config-set r219 committed by bcsaller@gmail.com | 02:16 |
| _mup_ | support optional config files | 02:16 |
| === Daviey_ is now known as Daviey | ||
| _mup_ | ensemble/expose-provisioning r259 committed by jim.baker@canonical.com | 03:30 |
| _mup_ | Handle scenario where services or service units disappear from topology while applying port changes | 03:30 |
| hazmat | jimbaker, i'm working on cleaning up the watches on trunk now, fwiw, the best guideline rule for poke_zk seems to be if you know a watch is being established in the background | 05:00 |
| jimbaker | hazmat, i was just playing with that | 05:01 |
| jimbaker | it seemed like poke_zk + sleep was effective | 05:01 |
| _mup_ | ensemble/expose-provisioning r260 committed by jim.baker@canonical.com | 05:02 |
| _mup_ | Do not keep port policy in memory, read it from ZK every time | 05:02 |
| hazmat | jimbaker, basically you can use one poke per other zk activity you know is happening in the background | 05:04 |
| jimbaker | hazmat, makes sense | 05:05 |
| hazmat | ie, a one per client get/get_children, or exists, etc. | 05:06 |
| _mup_ | ensemble/config-set r220 committed by bcsaller@gmail.com | 05:09 |
| _mup_ | add get_serialization_data | 05:09 |
| _mup_ | ensemble/watching-godot-redux r227 committed by kapil.thangavelu@canonical.com | 05:10 |
| _mup_ | all the watch state protocol apis now wait on their initial callback before returning. | 05:10 |
| _mup_ | ensemble/watching-godot-redux r228 committed by kapil.thangavelu@canonical.com | 05:14 |
| _mup_ | apply the wait on initial invocation behavior to the debug log watch | 05:14 |
| _mup_ | ensemble/watching-godot-redux r229 committed by kapil.thangavelu@canonical.com | 05:40 |
| _mup_ | make lifecycle start fire after initial watch callbacks | 05:40 |
| _mup_ | ensemble/watching-godot-redux r230 committed by kapil.thangavelu@canonical.com | 05:43 |
| _mup_ | excise any magic waiting for lifecycle background activity | 05:43 |
| _mup_ | ensemble/config-set r221 committed by bcsaller@gmail.com | 05:45 |
| _mup_ | ConfigOptions available off formula directory/bundle | 05:45 |
| _mup_ | ensemble/watching-godot-redux r231 committed by kapil.thangavelu@canonical.com | 05:51 |
| _mup_ | excise extraneous sleeps from lifecycle tests. | 05:51 |
| _mup_ | ensemble/watching-godot-redux r232 committed by kapil.thangavelu@canonical.com | 06:03 |
| _mup_ | fix the test to account for valid sequence variation across multiple join events/ | 06:03 |
| _mup_ | ensemble/watching-godot-redux r233 committed by kapil.thangavelu@canonical.com | 06:08 |
| _mup_ | excise the last sleep from the workflow tests | 06:08 |
| _mup_ | ensemble/watching-godot-redux r234 committed by kapil.thangavelu@canonical.com | 06:13 |
| _mup_ | excise last extraneous sleep from lifecycle tests. | 06:13 |
| _mup_ | ensemble/config-set r222 committed by bcsaller@gmail.com | 07:16 |
| _mup_ | ConfigOptions.load can also take a dict as source data allowing easier initialization in FormulaState | 07:16 |
| _mup_ | ensemble/config-set r223 committed by bcsaller@gmail.com | 10:38 |
| _mup_ | persistent watch on config state | 10:38 |
| _mup_ | ensemble/config-set r224 committed by bcsaller@gmail.com | 12:14 |
| _mup_ | typo | 12:14 |
| _mup_ | ensemble/trunk-2 r222 committed by bcsaller@gmail.com | 12:15 |
| _mup_ | merge trunk | 12:15 |
| hazmat | g'morning | 14:13 |
| niemeyer | hazmat: yo! | 14:16 |
| hazmat | niemeyer, fixed most of the watches last night.. feels good to remove like a 50 sleeps and artificial stops | 14:17 |
| niemeyer | hazmat: Wow, seriously? | 14:34 |
| hazmat | niemeyer, indeed | 14:34 |
| hazmat | its slow going though.. have to run about a hundred iterations per test to verify | 14:34 |
| niemeyer | hazmat: So most of that crap was actually bound to the initial invocation derailing the deferred chain, rather than real background activity? | 14:34 |
| hazmat | niemeyer, yup | 14:34 |
| niemeyer | Aw | 14:35 |
| hazmat | niemeyer, some are for legitimate background activity | 14:35 |
| hazmat | like the watch firing on state change | 14:35 |
| niemeyer | Right | 14:35 |
| hazmat | hmm | 14:55 |
| hazmat | niemeyer, there's an interesting new sequencing behavior with the new hook semantics.. re ordering.. currently we don't allow upgrades unless we're in a running state, if an upgrade flag is set when the unit is not running, we'll end up ignoring it, it looks like we need to check it again after we're in the running state. | 15:39 |
| hazmat | not really about sequencing, just that made it more apparent | 15:40 |
| niemeyer | hazmat: Ah, oops | 15:40 |
| niemeyer | hazmat: But won't the upgrade fail? | 15:41 |
| hazmat | niemeyer, the upgrade won't happen it exits early.. but even though we might then have an upgrade flag set, we won't have another watch change triggering the upgrade behavior again when we're in the started state | 15:42 |
| hazmat | ie. the unit would eventually be running with an upgrade flag set | 15:42 |
| hazmat | currently we clear the flag if we're not in the running state | 15:45 |
| niemeyer | hazmat: I don't think I get it still.. if we disallow the upgrade flag from being set when the unit is not running, how can the upgrade flag be set when the unit is not running? | 15:45 |
| hazmat | but its not clear that's the right behavior | 15:45 |
| hazmat | niemeyer, good point.. i was just considering from the agent side, where really it should be prepared for the unit to be in any state | 15:45 |
| niemeyer | hazmat: Admittedly, there's a race condition there, though | 15:46 |
| niemeyer | hazmat: Between checking the state and setting the flag | 15:46 |
| hazmat | niemeyer, and the flag being acted upon by the agent | 15:46 |
| hazmat | which is why the agent checks separately | 15:46 |
| hazmat | i'll go back to clearing it, its probably the only sane thing to do, and the user can request an upgrade again | 15:47 |
| hazmat | in future with protocol abstractions we could introduce other things to enable the flag persisting tiill a running state | 15:47 |
| niemeyer | hazmat: Hmm.. I don't think there's a race about that | 15:48 |
| hazmat | niemeyer,upgrade flag is set, unrelated hook fails, unit needs resolved | 15:48 |
| niemeyer | hazmat: Or maybe I don't see what you mean | 15:48 |
| niemeyer | hazmat: Ah, ok.. yes | 15:48 |
| niemeyer | Will get some lunch.. biab | 15:52 |
| _mup_ | ensemble/watching-godot-redux r235 committed by kapil.thangavelu@canonical.com | 15:59 |
| _mup_ | process upgrade in the correct sequence during unit lifecycle startup | 15:59 |
| _mup_ | ensemble/watching-godot-redux r236 committed by kapil.thangavelu@canonical.com | 16:00 |
| _mup_ | cleanup pep8 violations from trunk | 16:00 |
| _mup_ | ensemble/watching-godot-redux r237 committed by kapil.thangavelu@canonical.com | 16:12 |
| _mup_ | reintroduce some sleeps that where part of error checking (giving a chance for an error to occur). | 16:12 |
| hazmat | hmm.. changing the topology watches breaks a lot of test assumptions | 16:19 |
| _mup_ | ensemble/watching-godot-redux r238 committed by kapil.thangavelu@canonical.com | 16:39 |
| _mup_ | update machine manager watch machine states to wait till current state is processed. | 16:39 |
| _mup_ | ensemble/watching-godot-redux r239 committed by kapil.thangavelu@canonical.com | 16:47 |
| _mup_ | test confirming environment watch processes initial environment as part of watch creation method. | 16:47 |
| niemeyer | hazmat: You mean with the fixes for them to not go off alone? | 16:48 |
| hazmat | niemeyer, econtext | 16:48 |
| niemeyer | hazmat: Re. your previous message above | 16:48 |
| hazmat | niemeyer, i'm also implementing this for the topology watches, some of them are already structured so they require a change | 16:48 |
| hazmat | before they fire the callback | 16:48 |
| hazmat | i'm just writing tests that in verify in the same way as the other watches, the current state has been processed before the watch method is over. | 16:49 |
| hazmat | oh.. i was wrong about breaking assumptions | 16:49 |
| hazmat | their fine.. i had an unrelated issue | 16:50 |
| niemeyer | Ah, ok | 16:50 |
| _mup_ | ensemble/watching-godot-redux r240 committed by kapil.thangavelu@canonical.com | 16:52 |
| _mup_ | fix pep8 violations from trunk | 16:52 |
| _mup_ | ensemble/watching-godot-redux r241 committed by kapil.thangavelu@canonical.com | 16:58 |
| _mup_ | test current state processing test for machine.watch_service_units | 16:58 |
| _mup_ | ensemble/watching-godot-redux r242 committed by kapil.thangavelu@canonical.com | 17:03 |
| _mup_ | test verifying current state processing of service.watch_relation_states | 17:03 |
| niemeyer | "preferred_email_address_link":"https://api.launchpad.net/1.0/~user/+email/the@email.net" | 17:03 |
| niemeyer | Interesting way to present emails in the API | 17:03 |
| niemeyer | Object: <canonical.launchpad.systemhomes.WebServiceApplication object at 0x7adc110>, name: u'+me' | 17:07 |
| niemeyer | Interesting data to return in the body of an API request too :) | 17:08 |
| _mup_ | ensemble/watching-godot-redux r243 committed by kapil.thangavelu@canonical.com | 17:19 |
| _mup_ | test for watch upgrade proceses current state | 17:19 |
| jimbaker | hazmat, so for the provisioning agent test, i added superpoke, which by default, pokes zk 10 times, sleeps 0.1s, pokes zk 10 more times | 17:23 |
| hazmat | jimbaker, ugh. | 17:23 |
| jimbaker | there's probably a more minimal version as you illustrated w/ your godot-redux | 17:23 |
| jimbaker | but it works well now - i'm getting a reliable looped test at last | 17:24 |
| hazmat | jimbaker, in the current godot-redux every sleep is documented with why its there. | 17:24 |
| hazmat | jimbaker, cool | 17:24 |
| jimbaker | hazmat, it certainly sounds like the right way to do it, although it's not clear to me how this can be done w/o being whitebox about knowing what the watch setup is, which could be quite complex in the case i have | 17:25 |
| jimbaker | since the watches are definitely timing dependent | 17:25 |
| jimbaker | to know more, i have to look at the logs more closely to see what ordering they have | 17:25 |
| hazmat | jimbaker, i'll have to take a look | 17:26 |
| hazmat | i'll search for the superpokes | 17:26 |
| hazmat | but not this week | 17:26 |
| hazmat | it will be nice to have the existing watches and tests on trunk fixed | 17:26 |
| jimbaker | hazmat, yeah, i think it's ok to start with superpoke, then make it better | 17:26 |
| hazmat | there's alot of watch apis though | 17:26 |
| jimbaker | the other thing i needed to do was to stop InternalTopologyErrors from leaking through | 17:27 |
| hazmat | jimbaker, indeed, you have a good understanding now though of what the sleeps are for? | 17:27 |
| jimbaker | right now, i'm doing that in the provisioning agent itself, but it looks like they are confined in two methods, so we can push back into the topology itself | 17:28 |
| hazmat | hmmm.. internaltopologyerrors? shouldn't really ever be seen | 17:28 |
| jimbaker | hazmat, i think i have a good working idea | 17:28 |
| hazmat | jimbaker, cool | 17:28 |
| jimbaker | seen here in http://pastebin.ubuntu.com/604149/, note the "badness for wordpress" log line (obviously i don't expect it to be there for any length of time) | 17:29 |
| jimbaker | hazmat, that probably occurs about 20% of the time | 17:32 |
| jimbaker | much rarer (i have now run this looped 140+ times) is ServiceUnitState.get_assigned_machine_id surfacing InternalTopologyError | 17:33 |
| _mup_ | ensemble/watching-godot-redux r244 committed by kapil.thangavelu@canonical.com | 17:38 |
| _mup_ | current state processing tests for watch debug hook, watch resolved, and watch relation resolved. | 17:38 |
| _mup_ | ensemble/expose-provisioning r261 committed by jim.baker@canonical.com | 17:47 |
| _mup_ | Use 'superpoke' in port provisioning tests, which now loop 200+ times | 17:47 |
| _mup_ | ensemble/watching-godot-redux r245 committed by kapil.thangavelu@canonical.com | 17:48 |
| _mup_ | unit agent test resolved while stoppped, switch out the sleep for a poke, change the wait condition to a state change from a hook execution | 17:48 |
| niemeyer | <jimbaker> hazmat, so for the provisioning agent test, i added superpoke, which by default, pokes zk 10 times, sleeps 0.1s, pokes zk 10 more times | 17:53 |
| niemeyer | Double ugh :) | 17:53 |
| niemeyer | ... and it only works if you jump three times on the left foot, and two times on the right one, in that order! | 17:54 |
| _mup_ | ensemble/watching-godot-redux r246 committed by kapil.thangavelu@canonical.com | 18:00 |
| _mup_ | cleanup the unit agent resolved watch tests. | 18:00 |
| === deryck is now known as deryck[lunch] | ||
| jimbaker | niemeyer, i share your dislike... the name superpoke only helps suggest that it is not desirable | 18:01 |
| niemeyer | jimbaker: It's not just a dislike.. we can't have something like this. | 18:02 |
| jimbaker | niemeyer, exactly | 18:02 |
| jimbaker | so we need to fix it | 18:02 |
| jimbaker | one step at a time... possibly with a jump three times, i guess | 18:02 |
| niemeyer | jimbaker: Yeah.. let's start by removing it :-) | 18:02 |
| jimbaker | niemeyer, don't worry, i won't suggest a proposal until it's gone | 18:03 |
| hazmat | jimbaker, when your sleeping that long, its generally means you need some sort of observation api so you can sync on a known point | 18:03 |
| niemeyer | jimbaker: It makes as much sense to be repeatedly poking and sleeping as it makes sense to press an elevator button repeatedly | 18:03 |
| hazmat | jimbaker, like wait_on_state or wait_on_hook | 18:03 |
| jimbaker | hazmat, niemeyer agreed, agreed :) | 18:03 |
| jimbaker | niemeyer, the more interesting thing is that code only looks at ZK for its state | 18:05 |
| jimbaker | in doing so, i have apparently found at least two bugs in our state API | 18:05 |
| niemeyer | jimbaker: Please submit fixes for these bugs in an isolated branch | 18:05 |
| jimbaker | niemeyer, that's my intent :) | 18:06 |
| niemeyer | jimbaker: Thanks | 18:06 |
| _mup_ | Bug #778628 was filed: Watches should return only after processing current state <Ensemble:New for hazmat> < https://launchpad.net/bugs/778628 > | 18:07 |
| jimbaker | i will also need an added method. right now, it's part of ServiceStateManager in my branch, get_service_units_for_machine(self, machine_state) | 18:08 |
| jimbaker | i put it there so as to avoid circular import dependencies, and there seems to be some precedence for doing that in ensemble.state.service | 18:08 |
| hazmat | niemeyer, watch fix branch in review now fwiw | 18:09 |
| niemeyer | jimbaker: It should be machine_state.get_service_unit_states() | 18:09 |
| jimbaker | if that makes sense, i will also suggest a separate branch for that too | 18:09 |
| jimbaker | niemeyer, that would be ideal | 18:09 |
| niemeyer | jimbaker: So let's do it :-0 | 18:09 |
| niemeyer | :-) | 18:09 |
| jimbaker | niemeyer, so what do we do to avoid circular imports? | 18:09 |
| niemeyer | jimbaker: Hmm.. the usual thing we do with Python | 18:09 |
| jimbaker | then i can realize that ideal | 18:09 |
| hazmat | that's odd lp says the diff is 1300 lines.. | 18:10 |
| hazmat | lp is confused | 18:10 |
| jimbaker | niemeyer, please elaborate your thought here, because python gives us some tools, but not a clear path | 18:10 |
| niemeyer | jimbaker: How do you usually handle circular imports with Python? | 18:11 |
| jimbaker | niemeyer, i probably would defer the import | 18:11 |
| niemeyer | jimbaker: Bingo | 18:12 |
| niemeyer | jimbaker: Simply import it within the function | 18:12 |
| niemeyer | s/function/method | 18:12 |
| hazmat | there are a few places that do deferred imports in appropriate function in the code base.. jim i'm curious which example you though was precedence of moving the function? | 18:12 |
| jimbaker | hazmat, you ask me on the spot | 18:13 |
| jimbaker | there does seem to be somewhat more code in ensemble.state.service that knows about our types of state | 18:13 |
| hazmat | jimbaker, no worries, i've got some extra time now, if you could point me to one of branches using superpoke. | 18:13 |
| jimbaker | hazmat, sure you can see the current ugliness (don't let niemeyer look at, he will gag - lp:~jimbaker/ensemble/expose-provisioning) | 18:14 |
| hazmat | jimbaker, ;-) | 18:15 |
| jimbaker | i on the other hand celebrate a modicum of progress | 18:15 |
| hazmat | jimbaker, this needs an observer api on the agent, probably best to pull it from the agent into a separate object (proto protocol, ala formula upgrade) | 18:26 |
| hazmat | jimbaker, we need to be able to know as an observer when the ports have changed | 18:26 |
| hazmat | there's half-dozen sleep calls all intermixed in these tests, all waiting for background watch activity | 18:26 |
| hazmat | there's nothing wrong with them except the need for better sync points | 18:27 |
| hazmat | jimbaker, more concretely adding a set_port_observer, with a callback that's invoked when the ports are modified would allow for solving this | 18:28 |
| hazmat | if you ever find yourself using alot of sleeps, this is the solution | 18:29 |
| niemeyer | Dude! This thing is starting to work for real | 18:30 |
| niemeyer | Sorry.. this thing == launchpad integration, for those watching ;-) | 18:31 |
| hazmat | niemeyer, can you sniff all the emails of launchpad members yet? | 18:31 |
| niemeyer | hazmat: Yep, almost there! | 18:31 |
| niemeyer | hazmat: Hmmm.. it should actually work already | 18:31 |
| niemeyer | hazmat: Will just implement some handy methods and paste an example | 18:32 |
| hazmat | jimbaker, out of curiosity do you use emacs? | 18:43 |
| jimbaker | hazmat, yes i do emacs | 18:44 |
| jimbaker | but more and more naively every year | 18:44 |
| hazmat | jimbaker, me too re naively ;-), but flymake mode with pep8 integration is da bomb | 18:45 |
| jimbaker | i used to know my way around emacs lisp, then just dot files, for customization | 18:45 |
| jimbaker | now i just do the out-of-the-box experience | 18:45 |
| hazmat | jimbaker, so do my comments above make sense as way out of superpoke ? | 18:45 |
| jimbaker | hazmat, sorry, i just saw that line... | 18:46 |
| jimbaker | hazmat, that makes perfect sense as an integration point | 18:46 |
| jimbaker | so basically one place to sync on | 18:47 |
| niemeyer | hazmat: Yeah, I'm happy you've shown me how much flymake helps | 18:48 |
| niemeyer | hazmat: Probably one of the best productivity gains I got with Python in the last year | 18:48 |
| jimbaker | hazmat, the good thing about budapest is i will get the live demo - i know you pointed me to a dot file before for this setup, but i managed to lose the link along the way | 18:48 |
| hazmat | http://kapilt.com/files/emacs.tgz | 18:49 |
| hazmat | ;-) | 18:49 |
| hazmat | sounds good | 18:49 |
| jimbaker | before i started comprehensively logging irc, using bouncers, etc | 18:50 |
| jimbaker | hazmat, thanks, i will give it a whirl | 18:50 |
| hazmat | setting up the irc proxy was probably one of my best gains of the last year | 18:50 |
| jimbaker | hazmat, so i think we need to set up an observer api like you suggest on each of the state variables that is being monitored | 18:51 |
| jimbaker | eg, watched_services | 18:51 |
| jimbaker | because some of the tests don't actually look at the ports that end up getting setup, they just ensure the watch cascade happens properly | 18:52 |
| jimbaker | hazmat, re irc proxies, yeah i saw my fellow canonical employees doing some cool stuff | 18:53 |
| jimbaker | actually one goal for me at budapest is to learn effective byobu | 18:53 |
| === deryck[lunch] is now known as deryck | ||
| _mup_ | ensemble/config-set r226 committed by bcsaller@gmail.com | 19:42 |
| _mup_ | fix ensemble set validation with tests | 19:42 |
| _mup_ | additional testing | 19:42 |
| _mup_ | ensemble/config-set r227 committed by bcsaller@gmail.com | 19:45 |
| _mup_ | test for missing/bad service in ensemble set | 19:45 |
| _mup_ | ensemble/config-set r228 committed by bcsaller@gmail.com | 19:48 |
| _mup_ | checkin missing config.yaml | 19:48 |
| _mup_ | Bug #778685 was filed: ensemble must provide a `set` command for config options <Ensemble:New for bcsaller> < https://launchpad.net/bugs/778685 > | 19:50 |
| niemeyer | We have to sit down at UDS and talk through some of the repository stuff | 19:58 |
| jimbaker | cool, my laptop fits in its new protective neoprene sleeve in my bike's pannier bag | 20:28 |
| niemeyer | jimbaker: You're brave :-) | 20:59 |
| hazmat | niemeyer, sounds good, we also need to discuss monitoring | 22:02 |
| _mup_ | ensemble/auto-dependency-resolution r219 committed by kapil.thangavelu@canonical.com | 23:04 |
| _mup_ | new test-api base class, and resolver tests based on. | 23:04 |
| niemeyer | Hmmm | 23:19 |
| niemeyer | Food needed | 23:19 |
| _mup_ | ensemble/auto-dependency-resolution r220 committed by kapil.thangavelu@canonical.com | 23:22 |
| _mup_ | resolver works to resolve dependencies from formula repos and environment. | 23:22 |
| hazmat | it lives | 23:22 |
| hazmat | niemeyer, to answer the question from yesterday.. yes it does work ;-) | 23:22 |
| hazmat | the weather in budapest looks great | 23:23 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!