[14:52] <ssarah> hei guys
[14:53] <ssarah> i want to ask, i wana use upstart instead of node forever. How can i make it that upstart only tries to restart a process a number of times before calling for support?
[14:58] <ssarah> anyone? =O
[17:06] <slangasek> ssarah: what do you mean by "calling for support"?
[17:07] <slangasek> you can use the "respawn limit" command to control how many times upstart will retry a failing process; but I'm not sure if that's what you're asking for
[18:01] <ssarah> slangasek: i mean, that after failing for several times, it does a custom command
[18:02] <ssarah> get it? :)
[18:02] <slangasek> ok
[18:02] <slangasek> upstart is certainly flexible enough that this should be possible, but I can't point to a ready-made example of someone doing this before
[18:03] <slangasek> in fact I think you probably would need to keep a counter, outside of upstart itself, unless the failures are immediate and could be done through 'respawn limit'
[18:12] <xnox> ssarah: i'd recommend use existing monitoring to monitor that service is up, when upstart gives up restarting it the status will change to "stop/waiting" and your monitoring system (nagios, check-mk, etc) can take appropriate action for automated recovery and/or further escalation.
[18:12] <xnox> ssarah: at least, in my previous sys-admin days that's what we did. Since nagios already does smart escalation notification and has appropriate means to encode shifts and rotas.
[18:13] <ssarah> xnox: what status changes to "stop/waiting"
[18:13] <ssarah> ?
[18:17] <xnox> ssarah: of your job. If you start it, it becomes "$ status your-job" will say start/running. And would stay like that, and respawned as needed.
[18:18] <xnox> ssarah: when it fails permamently, it will change status to "$ status your-job" will become "stop/waiting", and thus your nagios monitoring can be checking that.
[18:19] <ssarah> im sorry if this sounds too basic but that variable is something i can see where?
[18:30] <xnox> ssarah: type on the command line: $ sudo status tty2
[18:30] <xnox> ssarah: $ sudo stop tty2
[18:30] <xnox> ssarah: $ sudo status tty2
[18:30] <xnox> ssarah: $ sudo start tty2
[18:30] <xnox> ssarah: $ sudo status tty2
[18:30] <xnox> ssarah: $ cat /etc/init/tty2.conf
[18:31] <xnox> ssarah: for more about jobs, their states, what states they transition through and how to control/inspect/observe that see http://upstart.ubuntu.com/cookbook/
[18:31] <xnox> ssarah: for more control you can add your custom jobs, or e.g. monitor signals, events and notifications on the DBUS interface