[13:56] <robert_ancell> I'm having trouble with session upstart in Ubuntu. I've forked gnome-settings-daemon into unity-settings-daemon and copied the upstart scripts (http://paste.ubuntu.com/6885290/, http://paste.ubuntu.com/6885292/). However, if I try and run a session gnome-session doesn't start. If I disable the gnome-settings-daemon script it does. So it appears the gnome-settings-daemon script is confusing the startup somehow
[13:56] <robert_ancell> Any ideas?
[13:57] <robert_ancell> http://paste.ubuntu.com/6885304/ is the output of upstart on a failed case
[13:57] <robert_ancell> It doesn't mention gnome-settings-daemon at all
[14:03] <jodh> robert_ancell: so unity-settings-daemon starts, but gnome-session does not. How are you disabling gnome-settings-daemon? Also, is there anything relevant in ~/.cache/upstart/*.log ?
[14:04] <robert_ancell> jodh, I'm disabling by moving the .conf to .conf.disabled
[14:04] <robert_ancell> jodh, I looked in the logs but couldn't see anything obvious
[14:04] <robert_ancell> I can bundle them up
[14:07] <robert_ancell> http://people.canonical.com/~bob/upstart-logs.tgz
[14:08] <jodh> robert_ancell: unity7.conf specifies 'start on xsession SESSION=ubuntu and started gnome-settings-daemon' so if you've disabled gnome-settings-daemon that won't start (unless you've changed other .confs?)
[14:08] <robert_ancell> jodh, I have also changed that to do "or unity-settings-daemon"
[14:08] <robert_ancell> no other changes
[14:09] <robert_ancell> I notice there is a gnome-session log there, might be from a working boot though
[14:09] <robert_ancell> I'll clear the logs and retry
[14:09] <jodh> robert_ancell: can you share the whole config for unity7 as that is likely the issue
[14:10] <robert_ancell> http://paste.ubuntu.com/6885356/
[14:11] <robert_ancell> http://people.canonical.com/~bob/upstart-logs.tgz for the logs on a fresh run after clearing
[14:11] <robert_ancell> http://people.canonical.com/~bob/upstart-logs2.tgz
[14:11] <robert_ancell> I mean
[14:11] <jodh> robert_ancell: surely that won't work with gnome-settings-daemon though since gnome-settings-daemon specifies 'xsession SESSION!=ubuntu' but unity7 states 'xsession SESSION=ubuntu'
[14:12] <robert_ancell> we are replacing gnome-settings-daemon with unity-settings-daemon in Unity
[14:12] <robert_ancell> gnome-settings-daemon should only run for GNOME etc
[14:17] <jodh> robert_ancell: according to the logs, the unity7 job is stopping in pre-start and not starting compiz. Could that explain it?
[14:17] <robert_ancell> jodh, I think compiz is started by gnome-session currently
[14:18] <robert_ancell> so it's behaving correctly
[14:19] <robert_ancell> my upstart logs in a normal session confirm that (compiz is in the gnome-session.log, unity7.log says "GNOME Session is starting Compiz")
[14:21] <robert_ancell> fancy coming into the office? ;)
[14:26] <robert_ancell> jodh, Is there any more information I can collect?
[14:27] <xnox> robert_ancell: jodh: the typical way to resolve this is to introduce an arbitrary sync event.
[14:27] <xnox> robert_ancell: jodh: e.g. both "unity-settings-daemon" and "gnome-settings-daemon" should do "initctl emit settings-daemon"
[14:27] <Mariano9> hi guys
[14:27] <xnox> robert_ancell: then unity7 would start on "xsession SESSION=ubuntu and settings-daemon"
[14:28] <robert_ancell> ok, I can try that
[14:28] <xnox> robert_ancell: ditto e.g. gnome/other desktop environments can do something else.
[14:28] <Mariano9> is there a command to check if an upstart job is correctly formated? like init-chkconfig or some?
[14:28] <xnox> robert_ancell: that way one would also be free to use one of the alternative settings-daemons.
[14:29] <jodh> Mariano9: yes - I put it in the mail I sent you - init-checkconf /path/to/job.conf
[14:29] <xnox> robert_ancell: one would do "initctl emit settings-daemon" in e.g. post-start
[14:29] <Mariano9> hey jodh
[14:29] <Mariano9> cannot find the command
[14:29] <Mariano9> it isn't there, init-chkconfig
[14:29] <xnox> robert_ancell: this is similar to how "indicators-*" start events work, it's simply emitted when indicators should start loading. And e.g. unity, gtk3-old-style-applet and others emit it to indicate they are ready to receive indicators.
[14:30] <jodh> Mariano9: it's *init-checkconf* not chkconfig
[14:30] <Mariano9> sorry, init-chkconf
[14:31] <Mariano9> "init-checkconf /etc/init/newrelic_rs_dfw_1.conf, -bash: init-checkconf: command not found"
[14:31] <xnox> Mariano9: what version of upstart are you using?
[14:32] <jodh> Mariano9: run "initctl version" to establish that
[14:32] <Mariano9> 0.6.5
[14:33] <xnox> Mariano9: excellent =) so pre-historic. Well, we can attempt to valide that job for you by reading it, if you can pass it on to us. e.g. paste.ubuntu.com
[14:34] <Mariano9> haha, just a sec
[14:36] <robert_ancell> xnox, same problem after adding the settings-daemon event. Still works if disabling the gnome-settings-daemon config
[14:37] <Mariano9> https://gist.github.com/Mariano-gon/8845357
[14:37] <Mariano9> xnox: there you go
[14:37] <Mariano9> (notice that I've commented some lines as I'm trying to get it goint)
[14:37] <jodh> Mariano9: you have 2 execs - only the last will be honoured.
[14:37] <xnox> robert_ancell: hm. Is there a ppa i can try, or is just by co-installing gnome & ubuntu settings daemon on my desktop?
[14:37] <Mariano9> *going
[14:38] <seb128> xnox, you should come to the office to help us debug those g-s-d u-s-d issues ;-)
[14:38] <xnox> seb128: you want me to comute into the office _now_ ?! =)
[14:38] <seb128> xnox, tomorrow morning would work as well ;-)
[14:38] <seb128> strike should be over then?
[14:38] <robert_ancell> xnox, it's not in a PPA at the moment, but I could add it
[14:39] <jodh> Mariano9: assuming you are running ubuntu lucid, procenv won't be available unless you built it from source?
[14:39] <Mariano9> nope, centOS 6.x here
[14:39] <xnox> seb128: i use trains anyway, so i'm unaffected. Tube is mostly non-existent south of the river.
[14:39] <jodh> Mariano9: what output do you get if you run "sudo start newrelic"?
[14:39] <jodh> Mariano9: ah, ok :)
[14:40] <robert_ancell> xnox, if you could come in tomorrow that would be great, we don't need to solve this immediately
[14:40] <Mariano9> "newrelic_rs_dfw_1 start/running, process 9251"
[14:40] <Mariano9> upstart sees it as running normal, but it isn't
[14:40] <xnox> Mariano9: that gist has two exec lines, i think last one wins.
[14:40] <xnox> Mariano9: the problem is the "bundle exec" was that that strange thing for ruby?
[14:41] <Mariano9> xnox: yes, it's a ruby gem
[14:41] <xnox> Mariano9: that will not work nicely as bundle forks many times and relies on environment variables and HOME, neither of which are set from within upstart
[14:41] <xnox> Mariano9: so you need probably need to do:
[14:41] <xnox> script
[14:41] <jodh> Mariano9: comment out the last line (exec bundle) so that upstart runs procenv first if you have that installed, then diff /tmp/procenv-job.log with a procenv log you take in an environment it does start in.
[14:41] <xnox>     source /path/to/bundle.sh
[14:42] <jodh> Mariano9: as mentioned, it may well be env vars. I have no idea what newrelic is but does it need $HOME?
[14:42] <xnox>     bundle exec /path/to/newrelic_rs
[14:42] <xnox> end script
[14:42] <xnox> Mariano9: and you probably want "expect daemon"
[14:42] <xnox> Mariano9: how is your bundle installed? user level or system-wide/root ?
[14:43] <Mariano9> xnox: jodh ok, does it matter the order of the stanzas?
[14:43] <xnox> Mariano9: you can't have duplicates, and at the moment I see two "exec" stanzas.
[14:43] <jodh> Mariano9: no, but as mentioned, if you specify a particular stanza multiple times, only the last is honoured.
[14:43] <xnox> Mariano9: it's declarative, not a top-down execution.
[14:43] <jodh> xnox: we've already covered that - see above
[14:43] <xnox> ah, sorry =)
[14:43] <Mariano9> right, i meant different stanzas order
[14:44] <Mariano9> but, it's declarative, so it doesn't matter the order
[14:44] <xnox> Mariano9: order of different stanzas does not matter, so you can have "start on" at the very end, etc.
[14:44] <Mariano9> great, let's try those
[14:46] <xnox> Mariano9: you should source bundle environment, it usually drops a script in /etc/profile.d or the users ~/.bashrc and the like. Not sure where it would live on a CentOS system however.
[14:46] <xnox> Mariano9: without that, bundle is mostly non-operational.
[14:46] <xnox> robert_ancell: ok, i'll be in tomorrow.
[14:46] <xnox> robert_ancell: and we can debug it.
[14:46] <robert_ancell> xnox, thanks
[14:46] <seb128> xnox, thanks
[14:50] <Mariano9> xnox: i think it's the same folder as newrelic_rs
[14:51] <xnox> Mariano9: right, so try:
[14:51] <xnox> script
[14:51] <xnox>     source that-file
[14:51] <xnox>     bundle exec
[14:51] <xnox> end script
[14:51] <Mariano9> because if I cd into it and run bundle exec ./bin/newrelic_rs, it will start great
[14:51] <xnox> Mariano9: due to chdir stanza you should be in the right directory already.
[14:52] <xnox> Mariano9: right, but bundle probably installed a script somewhere for your shell to find and source whenever you enter that directory.
[14:52] <Mariano9> like a link?
[14:52] <xnox> Mariano9: so one needs that "magic" snippet in the same way.
[14:52] <xnox> Mariano9: it's been many years since I last used bundle =)
[14:55] <Mariano9> source stanza will give the right path to execute the bundle command right?
[14:55] <xnox> source is not a stanza, source is a sh built in command
[14:56] <xnox> things between script... end script are just normal shell lines executed.
[14:56] <xnox> and that would be replacing exec.
[14:56] <Mariano9> ok
[14:56] <Mariano9> so, again, bundle path is the same as newrelic_rs then
[14:56] <xnox> Mariano9: oh there appears to be an easy solution for bundler.
[14:57] <xnox> Mariano9: --deployement --binstubs options
[14:57] <xnox> Mariano9: http://askubuntu.com/questions/105761/how-to-get-a-bundler-enabled-rails-app-running-as-a-service-with-upstart
[14:58] <xnox> Mariano9: apart from changing start on/stop on to what you had, it should just work on centos.
[14:58] <Mariano9> let's see
[15:01] <Mariano9> two questions xnox: 1- what if the stop job hangs?
[15:02] <Mariano9> how do I actually stop the initctl job?
[15:02] <Mariano9> "init: Sending TERM signal to newrelic_rs_dfw_1 main process (11127)" and will stay there for ever
[15:04] <Mariano9> ctrl+c, now it is shown as stop/killed, process 11127
[15:05] <xnox> Mariano9: after starting a job establish if the reality matches upstart expectations.
[15:06] <xnox> Mariano9: which is: process id in `ps' output matches, initctl status.
[15:06] <xnox> Mariano9: if they don't, the expect stanzas are wrong. more recent upstart handles such situations better.
[15:06] <xnox> Mariano9: but in your case you probably will need to add "expect fork", or "expect daemon".
[15:07] <xnox> Mariano9: and to clear the state, you'd need to re-exec upstart -> which actually for such old upstart means reboot.
[15:07] <xnox> Mariano9: i hope you are not developing this job on a production server.
[15:08] <Mariano9> host reboot?
[15:10] <Mariano9> there's no process in ps from newrelic, I've added the expect daemon before executing the start
[15:10] <Mariano9> is there a way to upgrade this version?
[15:12] <Mariano9> in fact there is
[15:12] <Mariano9> would it be good to upgrade it?
[15:42] <xnox> Mariano9: i don't know. I wonder if we should explore pushing a more up to date upstart into EPEL repositories.
[15:56] <Mariano9> xnox: so in order to clear the state I need to reboot the server?
[15:59] <xnox> Mariano9: or rename the job ;-) then the state is clear for the "new job name".
[16:01] <Mariano9> xnox: right, eventually I'll have tons of jobs :) 
[16:02] <Mariano9> xnox: is there a way to reload configuration once a job script is edited?
[16:04] <xnox> Mariano9: the configuration is automatically reloaded, but it will only take affect on newly start instance. So e.g. after a proper $ stop foo, the next $ start foo will use new configuration.
[16:04] <Mariano9> ok
[16:04] <xnox> Mariano9: reload/restart is not enough, those keep on using the in-memory configuration.
[19:39] <mc_bluebeard> Is it advisable to have an exec command that "blocks"…ie, that does not daemonize or return, but is instead long-running?
[20:05] <JanC> if you can, sure
[23:04] <xnox> mc_bluebeard: you can do that, in that case "your command is ~ running in foreground" and then you shouldn't specify any "expect ..." stanza.