[04:46] any timeline plans for state rewrite? [05:10] bradleyayers: I believe its been on hold since Keybuk reduced his involvement so much after leaving Canonical [05:10] damn :( [05:10] that's really really a shame [05:11] bradleyayers: probably more likely that you will see a reworking of 'expect fork' into something new before the state rewrite [05:11] so is the project basically 100% james now? [05:11] bradleyayers: slangasek and I chip in here and there, and we've gotten patches from redhat and a few other community contributors [05:12] fair enough [05:12] canonical should replace keybuk :) [05:13] jodh has been pretty good :) [05:14] bradleyayers: the state problem has fairly straight forward work arounds now too [05:21] bradleyayers: did you have a particular problem in mind that needs the state rewrite? I've solved most of them with the 'wait-for-state' job we added in Ubuntu 11.10 [05:22] granted, the addition of a 'while' keyword would be way better. :) [06:01] well, i suppose i just don't understand wait-for-state, and havent found a good tutorial for it [06:02] i basically want "start on started rabbitmq and started postgresql" "stop on stopping rabbitmq or stopping postgresql", and for my job to be started when either one is restarted [06:03] (my job relies on both postgresql and rabbitmq, so if either is restarted, i want my job to stop gracefully (and block the restarting thing), then start when it comes back up) [06:03] wait-for-state seems a bit cryptic to me and i'm not sure how to apply it to my situation [06:04] SpamapS: ^ [06:25] bradleyayers: your job is broken, IMO, and should be fixed to start normally without either of them available yet. [06:25] bradleyayers: because if it needs rabbit and postgres, they might be on a different box, and you can't control the order of boxes w/ upstart [06:25] by box you mean computer right? [06:26] hmm i see your point [06:26] we went through it w/ openstack too [06:26] openstack will likely run on thousands of computers [06:26] so it starts up with 3 phass.. 1) sanity checks, 2) poll for resources, 3) running [06:27] s/phass/phases/ [06:27] yep [06:27] But yeah, until you have that baked in.. [06:27] you need sequencing. [06:27] so wait-for-state, you'd just put this in pre-start: [06:27] start wait-for-state WAIT_FOR=rabbitmq WAITER=mything [06:27] start wait-for-state WAIT_FOR=postgres WAITER=mything [06:28] well wait-for-state won't scale to more than one box right [06:28] and just start on runlevel [2345] and stop on stopping postgresql or rabbitmq [06:28] bradleyayers: right [06:28] so i'd be better just changing my process to have a proper startup sanity + poll for resources check [06:29] yeah [06:29] then my process will die if either resource becomes unavailable [06:29] and will be restarted, and my startup checks will just keep it hanging there until the resources come back up [06:30] i suppose what jumps out to me in that scenario is that i lose the ability to tell if my process is really running yet using just upstart's "status" [06:30] which i suppose is fine, if you're thinking in a multi box setup [06:30] well no, just poll for a very long time for the things you need, you shouldn't be respawning much :) [06:31] bradleyayers: the fine grained eventing is for early-boot plumbing. [06:31] right [06:32] so i've just been taking the wrong approach [06:32] we all have [06:32] because it was easy to "get it right" when it was all in a single thread [06:32] yeah [06:32] and you had 5 servers [06:32] now you have parallelism everywhere, and 100+ servers [06:33] ya [06:33] the scenario that makes me wonder is the massive powerfail [06:33] hah, yeah [06:33] I remember the livejournal outtage, how they had to bring things up carefully [06:33] but really you shouldn't need to 'bring things up carefully' right? [06:34] i guess if you have a bunch of things all trying to become master [06:34] dbs -> cache servers -> storage servers -> queue workers -> app servers -> proxies [06:34] leader election *should* always be autoatic [06:34] but if everything just sits around waiting for their dependencies to be alive, why would you need to be careful about what order you start things up? [06:35] well you wouldn't [06:35] but thats now how things are put together all the time [06:35] its getting better though [06:35] yeah [06:35] EC2 helps with that.. since crap goes down so often. :) [06:35] well i greatly appreciate the advice [06:35] more people are used to writing resilient services [06:36] bradleyayers: and I appreciate the kick in the pants to document wait-for-state better [06:36] i believe i've been nagging for a long time :P [06:37] when you document it, you should put a flow diagram in :D [06:37] and a pony [06:37] everybody wants a pony [06:37] perhaps as the background [06:38] the colour scheme of the diagram should match the pony also [06:45] if I ever own a horse [06:45] I will name it Idempotent [06:46] cause I want him to run the same way, over and over [06:48] haha [06:49] i'm not sure the wit would be appreciated by many :P