[04:46] <bradleyayers> any timeline plans for state rewrite?
[05:10] <SpamapS> bradleyayers: I believe its been on hold since Keybuk reduced his involvement so much after leaving Canonical
[05:10] <bradleyayers> damn :(
[05:10] <bradleyayers> that's really really a shame
[05:11] <SpamapS> bradleyayers: probably more likely that you will see a reworking of 'expect fork' into something new before the state rewrite
[05:11] <bradleyayers> so is the project basically 100% james now?
[05:11] <SpamapS> bradleyayers: slangasek and I chip in here and there, and we've gotten patches from redhat and a few other community contributors
[05:12] <bradleyayers> fair enough
[05:12] <bradleyayers> canonical should replace keybuk :)
[05:13] <SpamapS> jodh has been pretty good :)
[05:14] <SpamapS> bradleyayers: the state problem has fairly straight forward work arounds now too
[05:21] <SpamapS> bradleyayers: did you have a particular problem in mind that needs the state rewrite? I've solved most of them with the 'wait-for-state' job we added in Ubuntu 11.10
[05:22] <SpamapS> granted, the addition of a 'while' keyword would be way better. :)
[06:01] <bradleyayers> well, i suppose i just don't understand wait-for-state, and havent found a good tutorial for it
[06:02] <bradleyayers> i basically want "start on started rabbitmq and started postgresql" "stop on stopping rabbitmq or stopping postgresql", and for my job to be started when either one is restarted
[06:03] <bradleyayers> (my job relies on both postgresql and rabbitmq, so if either is restarted, i want my job to stop gracefully (and block the restarting thing), then start when it comes back up)
[06:03] <bradleyayers> wait-for-state seems a bit cryptic to me and i'm not sure how to apply it to my situation
[06:04] <bradleyayers> SpamapS: ^
[06:25] <SpamapS> bradleyayers: your job is broken, IMO, and should be fixed to start normally without either of them available yet.
[06:25] <SpamapS> bradleyayers: because if it needs rabbit and postgres, they might be on a different box, and you can't control the order of boxes w/ upstart
[06:25] <bradleyayers> by box you mean computer right?
[06:26] <bradleyayers> hmm i see your point
[06:26] <SpamapS> we went through it w/ openstack too
[06:26] <SpamapS> openstack will likely run on thousands of computers
[06:26] <SpamapS> so it starts up with 3 phass.. 1) sanity checks, 2) poll for resources, 3) running
[06:27] <SpamapS> s/phass/phases/
[06:27] <bradleyayers> yep
[06:27] <SpamapS> But yeah, until you have that baked in..
[06:27] <SpamapS> you need sequencing.
[06:27] <SpamapS> so wait-for-state, you'd just put this in pre-start:
[06:27] <SpamapS> start wait-for-state WAIT_FOR=rabbitmq WAITER=mything
[06:27] <SpamapS> start wait-for-state WAIT_FOR=postgres WAITER=mything
[06:28] <bradleyayers> well wait-for-state won't scale to more than one box right
[06:28] <SpamapS> and just start on runlevel [2345] and stop on stopping postgresql or rabbitmq
[06:28] <SpamapS> bradleyayers: right
[06:28] <bradleyayers> so i'd be better just changing my process to have a proper startup sanity + poll for resources check
[06:29] <SpamapS> yeah
[06:29] <bradleyayers> then my process will die if either resource becomes unavailable
[06:29] <bradleyayers> and will be restarted, and my startup checks will just keep it hanging there until the resources come back up
[06:30] <bradleyayers> i suppose what jumps out to me in that scenario is that i lose the ability to tell if my process is really running yet using just upstart's "status"
[06:30] <bradleyayers> which i suppose is fine, if you're thinking in a multi box setup
[06:30] <SpamapS> well no, just poll for a very long time for the things you need, you shouldn't be respawning much :)
[06:31] <SpamapS> bradleyayers: the fine grained eventing is for early-boot plumbing.
[06:31] <bradleyayers> right
[06:32] <bradleyayers> so i've just been taking the wrong approach
[06:32] <SpamapS> we all have
[06:32] <SpamapS> because it was easy to "get it right" when it was all in a single thread
[06:32] <bradleyayers> yeah
[06:32] <SpamapS> and you had 5 servers
[06:32] <SpamapS> now you have parallelism everywhere, and 100+ servers
[06:33] <bradleyayers> ya
[06:33] <SpamapS> the scenario that makes me wonder is the massive powerfail
[06:33] <bradleyayers> hah, yeah
[06:33] <SpamapS> I remember the livejournal outtage, how they had to bring things up carefully
[06:33] <bradleyayers> but really you shouldn't need to 'bring things up carefully' right?
[06:34] <bradleyayers> i guess if you have a bunch of things all trying to become master
[06:34] <SpamapS> dbs -> cache servers -> storage servers -> queue workers -> app servers -> proxies
[06:34] <SpamapS> leader election *should* always be autoatic
[06:34] <bradleyayers> but if everything just sits around waiting for their dependencies to be alive, why would you need to be careful about what order you start things up?
[06:35] <SpamapS> well you wouldn't
[06:35] <SpamapS> but thats now how things are put together all the time
[06:35] <SpamapS> its getting better though
[06:35] <bradleyayers> yeah
[06:35] <SpamapS> EC2 helps with that.. since crap goes down so often. :)
[06:35] <bradleyayers> well i greatly appreciate the advice
[06:35] <SpamapS> more people are used to writing resilient services
[06:36] <SpamapS> bradleyayers: and I appreciate the kick in the pants to document wait-for-state better
[06:36] <bradleyayers> i believe i've been nagging for a long time :P
[06:37] <bradleyayers> when you document it, you should put a flow diagram in :D
[06:37] <SpamapS> and a pony
[06:37] <SpamapS> everybody wants a pony
[06:37] <bradleyayers> perhaps as the background
[06:38] <bradleyayers> the colour scheme of the diagram should match the pony also
[06:45] <SpamapS> if I ever own a horse
[06:45] <SpamapS> I will name it Idempotent
[06:46] <SpamapS> cause I want him to run the same way, over and over
[06:48] <bradleyayers> haha
[06:49] <bradleyayers> i'm not sure the wit would be appreciated by many :P