[01:16] <twb> Has anybody already written a script that can look at /etc/init and spit out a graphviz (or similar) dependency DAG ?
[01:29] <stgraber> twb: like initctl2dot (which we ship by default with Upstart)?
[01:29] <twb> I guess so; I'm still on Ubuntu 10.04 and I hadn't dpkg -L'd anyway
[01:30] <stgraber> ah, not sure if we already had it back then, but I know it's been around for a while now
[01:30] <twb> I'll just pull it from HEAD and see if it works on its own
[01:31] <stgraber> if you take the one in trunk, I ported it to python3 which I don't believe we had in 10.04, so I doubt that'll work
[01:31] <stgraber> taking an older revision should be fine though
[01:31] <twb> OK
[01:32] <twb> http://archive.ubuntu.com/ubuntu/pool/main/u/upstart/upstart_1.8-0ubuntu1.dsc should be old enough to be using python2 ?
[01:33] <stgraber> there's a pretty good chance 1.8 already had the python3 rewrite as my branch got merged in November 2012
[01:34] <stgraber> I'm sure that 1.5 is a safer bet
[01:34] <twb> Hm, I can't see an initctl2dot in that tarball
[01:34] <stgraber> twb: looked under scripts/ ?
[01:34] <twb> I did find -name \*dot
[01:34] <stgraber> ah, look for dot.py then
[01:35] <twb> Ah found it
[01:35] <twb> I could just run the script on another, newer host
[01:36] <twb> ba-bow: $ ssh spoon initctl show-config -e --err--> initctl: invalid command: show-config
[01:36] <stgraber> ah, looks like upstart under 10.04 is a bit old for that command...
[01:37] <stgraber> so probably easier to get a 12.04 machine with the same set of jobs and run initctl2dot on that one
[01:37] <twb> What this is all in aid of, is to work out why squid & an in-house daemon both fail to start sometimes when the server is rebooted
[01:38] <stgraber> sounds like a race condition between the network interfaces getting configured and the daemon starting
[01:38] <twb> Which I suspect is because they're trying to bind to an interface that isn't up or resolve a hostname before LDAP or DNS is up or something.
[01:38] <twb> Looking at the dependency graph seemed like the best first step :-/
[01:38] <stgraber> 10.04 wasn't great for that, some stuff were switch to upstart, some were still under sysvinit, so things may be a bit random... I did a fair amount of work in 12.04 to try and get all of that a bit saner (though unfortunately it ended up being way too big to backport to 10.04)
[01:39] <twb> Can you pastebin what initctl show-config -e output looks like?  I can probably mangle that by hand with some sed.  I don't have a newer ubuntu system handy
[01:39] <stgraber> are squid and your inhouse software using upstart jobs or are they good old sysvinit jobs?
[01:39] <twb> upstart
[01:39] <stgraber> ok, what are their start conditions?
[01:39] <twb> squid is start on runlevel [2345] or net-device-up IFACE!=lo; stop on runlevel [!2345]
[01:39] <twb> ppcadm is start on runlevel [2345]; stop on runlevel [^2345]
[01:40] <stgraber> ok, and do those require some special network interfaces to appear? (thinking of bond, bridges, vlan, tunnel devices, ... that kind of stuff)
[01:40] <twb> That first one looks buggy to me, looks to me like it assumes a single-NIC system
[01:41] <twb> Lemme pastebin the interfaces(5)
[01:41] <stgraber> output of initctl showconfig -e on 13.10: http://paste.ubuntu.com/6152669/
[01:41] <twb> http://paste.debian.net/45811/
[01:42] <twb> AFAIK we don't have bonding on anything anymore, at least one with no bonding is having startup issues
[01:43] <stgraber> ok and upstream, staff and prisoner are all aliases for real network interfaces setup through udev?
[01:43] <twb> Yeah they're renamed from eth0 style names
[01:44] <stgraber> my current guess is that those appear as standard ethX devices which then get renamed by udev, so you get a first net-device-up event with the wrong name before it vanishes to get renamed...
[01:44] <twb> Ah.
[01:44] <stgraber> twb: you could try to get both jobs to do "start on net-device-up IFACE=prisoner" or whatever interface they actually need
[01:45] <twb> So if I just slap in an "AND net-device-up IFACE=staff AND net-device-up IFACE=prisoner" ?
[01:45] <stgraber> yep, that'd work, upstart will just wait until it got all 3 events and then will start the service
[01:45] <twb> I don't really care if they don't start for two minutes or whatever after boot -- plan Z is to just add a great big "pre-start exec sleep 300"
[01:46] <stgraber> sounds like having upstart wait for the interfaces would be considerably cleaner than a huge sleep in pre-start ;)
[01:46] <twb> No argument there
[01:47] <twb> The problem is that atm after a power outage, upstart tries a few times then gives up, end result is they never actually start and then the inmates riot because they can't get their porn.
[01:49] <twb> I'll do some restarts with extra net-device-up
[02:44] <twb> Hm, syslog says squid is getting an ABRT.
[02:44] <twb> (Before I change the start on)
[04:32] <twb> Ah, the reason squid is crashing is because some fool thought it would be a good idea to make it use a whitelist helper that talks to postgres.  And postgres as at 10.04 is in sysvinit still.
[04:32] <twb> Fun :-/
[04:34] <twb> And I'm guessing "start on runlevel [2345]" is when that runlevel *begins*, not when it ends.
[05:42] <twb> And the ppcadm one is not starting because it needs slapd to be accepting connections before it starts.
[05:43] <twb> Which is also in sysvinit as at 10.04
[12:55] <SwashBuckla> Anyone written a rule for when `sudo service <service> restart` is invoked? http://upstart.ubuntu.com/cookbook/#restart
[12:55] <SwashBuckla> I'm writing an upstart job to daemonize a script, and am struggling to capture when `restart` is called -- what state is the process in?
[12:55] <SwashBuckla> is there an upstart condition? It's not pre-start or pre-stop
[13:14] <jodh> SwashBuckla: is the problem that your job is not able to restart the daemon script? Have you read http://upstart.ubuntu.com/cookbook/#expect ?
[13:21] <SwashBuckla> I can restart it just fine
[13:21] <SwashBuckla> that works
[13:21] <SwashBuckla> but I want to capture in the daemons .conf file WHEN that happens, do FOo
[13:21] <SwashBuckla> Foo*
[13:32] <jodh> SwashBuckla: there is no "restart" state - using the restart(8) command, the job is restarted whilst retaining its original config. Is this job a system-level one? If so, if you really want the job itself to know it has restarted, your best bet it probably to log something via a pre-start then check that value in the pre-start so you know how many times you've been restarted.
[13:35] <SwashBuckla> jodh: hmm. OK.
[13:35] <jodh> SwashBuckla: generally jobs shouldn't care if they have been restarted - they just need to handle that scenario
[13:35] <SwashBuckla> yeah I'll have to leave some breadcrumbs and look for them
[13:36] <SwashBuckla> I need to fire off an email in the event that a daemon was restarted, started, or stopped
[13:36] <SwashBuckla> as I care about the state of a daemon on a headless server
[13:37] <jodh> SwashBuckla: it would be better then to create a new "monitor" job that specifies "start on (started or stopped)" and does whatever you need.
[14:33] <PaulePanter> Hi. How do I analyze the startup process of my Ubuntu 12.04 server?
[14:33] <PaulePanter> Is there an equivalent to `systemd-analyze`?
[15:33] <jodh> PaulePanter: not atm. I suggest you apt-get install bootchart: http://www.bootchart.org/
[15:40] <PaulePanter> jodh: Thanks. I already tried it, but it is not conclusive yet as a mdraid resync runs in the background.
[15:40] <PaulePanter> A little off topic, is there a guide what I can disable on a server (where no graphics is needed for example).
[15:53] <PaulePanter> I am on Ubuntu 12.04. How do I easily find out if there are Upstart scripts in later releases for packages not shipping them?
[15:54] <PaulePanter> Collectd is installed and does not seem to have one.
[15:56] <PaulePanter> Mark Brooks published one here http://blog.loadbalancer.org/upstart-script-for-collectd/ and I wonder if Ubuntu already ships one so I can compare them.
[16:06] <jodh> PaulePanter: Some of this is indeed rather off-topic (you are probably better off asking some of this on #ubuntu-server in future). However, you can use http://packages.ubuntu.com to query package contents for any ubuntu release. Or use 'apt-file list <pkg>' on a newer version of Ubuntu (you'll need to 'apt-get install apt-file') to see what files a package ships.
[16:09] <PaulePanter> jodh: Thanks. (That method is not very user-friendly though. ;-) )
[16:10] <jodh> PaulePanter: ?
[16:15] <PaulePanter> jodh: Looking at http://packages.ubuntu.com/saucy/amd64/collectd-core/filelist and finding out that there is an upstart file, I would not be able to easily check the content of the file or copy it.
[16:15] <PaulePanter> In the end I would need to download the deb file and extract it, if I am not mistaken.
[16:16] <PaulePanter> Or are all package’s source files tracked in Launchpad in Bazaar?
[17:15] <jodh> PaulePanter: yes, you can access the files on bzr: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/collectd/saucy/files
[17:22] <PaulePanter> jodh: Awesome! Thanks. That method is easy!
[17:22] <PaulePanter> Though in this case I could not find the upstart job.
[17:22] <PaulePanter> http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/collectd/saucy/files/head:/debian/
[23:42] <glisignoli> Hello, upstart newbie here. I have a server in single user mode and want to disable the starting of an upstart service for a few restarts, then enable it again.
[23:44] <glisignoli> *upstart 0.6.5
[23:44] <glisignoli> :(