| bradleyayers | any timeline plans for state rewrite? | 04:46 |
|---|---|---|
| SpamapS | bradleyayers: I believe its been on hold since Keybuk reduced his involvement so much after leaving Canonical | 05:10 |
| bradleyayers | damn :( | 05:10 |
| bradleyayers | that's really really a shame | 05:10 |
| SpamapS | bradleyayers: probably more likely that you will see a reworking of 'expect fork' into something new before the state rewrite | 05:11 |
| bradleyayers | so is the project basically 100% james now? | 05:11 |
| SpamapS | bradleyayers: slangasek and I chip in here and there, and we've gotten patches from redhat and a few other community contributors | 05:11 |
| bradleyayers | fair enough | 05:12 |
| bradleyayers | canonical should replace keybuk :) | 05:12 |
| SpamapS | jodh has been pretty good :) | 05:13 |
| SpamapS | bradleyayers: the state problem has fairly straight forward work arounds now too | 05:14 |
| SpamapS | bradleyayers: did you have a particular problem in mind that needs the state rewrite? I've solved most of them with the 'wait-for-state' job we added in Ubuntu 11.10 | 05:21 |
| SpamapS | granted, the addition of a 'while' keyword would be way better. :) | 05:22 |
| bradleyayers | well, i suppose i just don't understand wait-for-state, and havent found a good tutorial for it | 06:01 |
| bradleyayers | i basically want "start on started rabbitmq and started postgresql" "stop on stopping rabbitmq or stopping postgresql", and for my job to be started when either one is restarted | 06:02 |
| bradleyayers | (my job relies on both postgresql and rabbitmq, so if either is restarted, i want my job to stop gracefully (and block the restarting thing), then start when it comes back up) | 06:03 |
| bradleyayers | wait-for-state seems a bit cryptic to me and i'm not sure how to apply it to my situation | 06:03 |
| bradleyayers | SpamapS: ^ | 06:04 |
| SpamapS | bradleyayers: your job is broken, IMO, and should be fixed to start normally without either of them available yet. | 06:25 |
| SpamapS | bradleyayers: because if it needs rabbit and postgres, they might be on a different box, and you can't control the order of boxes w/ upstart | 06:25 |
| bradleyayers | by box you mean computer right? | 06:25 |
| bradleyayers | hmm i see your point | 06:26 |
| SpamapS | we went through it w/ openstack too | 06:26 |
| SpamapS | openstack will likely run on thousands of computers | 06:26 |
| SpamapS | so it starts up with 3 phass.. 1) sanity checks, 2) poll for resources, 3) running | 06:26 |
| SpamapS | s/phass/phases/ | 06:27 |
| bradleyayers | yep | 06:27 |
| SpamapS | But yeah, until you have that baked in.. | 06:27 |
| SpamapS | you need sequencing. | 06:27 |
| SpamapS | so wait-for-state, you'd just put this in pre-start: | 06:27 |
| SpamapS | start wait-for-state WAIT_FOR=rabbitmq WAITER=mything | 06:27 |
| SpamapS | start wait-for-state WAIT_FOR=postgres WAITER=mything | 06:27 |
| bradleyayers | well wait-for-state won't scale to more than one box right | 06:28 |
| SpamapS | and just start on runlevel [2345] and stop on stopping postgresql or rabbitmq | 06:28 |
| SpamapS | bradleyayers: right | 06:28 |
| bradleyayers | so i'd be better just changing my process to have a proper startup sanity + poll for resources check | 06:28 |
| SpamapS | yeah | 06:29 |
| bradleyayers | then my process will die if either resource becomes unavailable | 06:29 |
| bradleyayers | and will be restarted, and my startup checks will just keep it hanging there until the resources come back up | 06:29 |
| bradleyayers | i suppose what jumps out to me in that scenario is that i lose the ability to tell if my process is really running yet using just upstart's "status" | 06:30 |
| bradleyayers | which i suppose is fine, if you're thinking in a multi box setup | 06:30 |
| SpamapS | well no, just poll for a very long time for the things you need, you shouldn't be respawning much :) | 06:30 |
| SpamapS | bradleyayers: the fine grained eventing is for early-boot plumbing. | 06:31 |
| bradleyayers | right | 06:31 |
| bradleyayers | so i've just been taking the wrong approach | 06:32 |
| SpamapS | we all have | 06:32 |
| SpamapS | because it was easy to "get it right" when it was all in a single thread | 06:32 |
| bradleyayers | yeah | 06:32 |
| SpamapS | and you had 5 servers | 06:32 |
| SpamapS | now you have parallelism everywhere, and 100+ servers | 06:32 |
| bradleyayers | ya | 06:33 |
| SpamapS | the scenario that makes me wonder is the massive powerfail | 06:33 |
| bradleyayers | hah, yeah | 06:33 |
| SpamapS | I remember the livejournal outtage, how they had to bring things up carefully | 06:33 |
| bradleyayers | but really you shouldn't need to 'bring things up carefully' right? | 06:33 |
| bradleyayers | i guess if you have a bunch of things all trying to become master | 06:34 |
| SpamapS | dbs -> cache servers -> storage servers -> queue workers -> app servers -> proxies | 06:34 |
| SpamapS | leader election *should* always be autoatic | 06:34 |
| bradleyayers | but if everything just sits around waiting for their dependencies to be alive, why would you need to be careful about what order you start things up? | 06:34 |
| SpamapS | well you wouldn't | 06:35 |
| SpamapS | but thats now how things are put together all the time | 06:35 |
| SpamapS | its getting better though | 06:35 |
| bradleyayers | yeah | 06:35 |
| SpamapS | EC2 helps with that.. since crap goes down so often. :) | 06:35 |
| bradleyayers | well i greatly appreciate the advice | 06:35 |
| SpamapS | more people are used to writing resilient services | 06:35 |
| SpamapS | bradleyayers: and I appreciate the kick in the pants to document wait-for-state better | 06:36 |
| bradleyayers | i believe i've been nagging for a long time :P | 06:36 |
| bradleyayers | when you document it, you should put a flow diagram in :D | 06:37 |
| SpamapS | and a pony | 06:37 |
| SpamapS | everybody wants a pony | 06:37 |
| bradleyayers | perhaps as the background | 06:37 |
| bradleyayers | the colour scheme of the diagram should match the pony also | 06:38 |
| SpamapS | if I ever own a horse | 06:45 |
| SpamapS | I will name it Idempotent | 06:45 |
| SpamapS | cause I want him to run the same way, over and over | 06:46 |
| bradleyayers | haha | 06:48 |
| bradleyayers | i'm not sure the wit would be appreciated by many :P | 06:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!