| twb | Has anybody already written a script that can look at /etc/init and spit out a graphviz (or similar) dependency DAG ? | 01:16 |
|---|---|---|
| stgraber | twb: like initctl2dot (which we ship by default with Upstart)? | 01:29 |
| twb | I guess so; I'm still on Ubuntu 10.04 and I hadn't dpkg -L'd anyway | 01:29 |
| stgraber | ah, not sure if we already had it back then, but I know it's been around for a while now | 01:30 |
| twb | I'll just pull it from HEAD and see if it works on its own | 01:30 |
| stgraber | if you take the one in trunk, I ported it to python3 which I don't believe we had in 10.04, so I doubt that'll work | 01:31 |
| stgraber | taking an older revision should be fine though | 01:31 |
| twb | OK | 01:31 |
| twb | http://archive.ubuntu.com/ubuntu/pool/main/u/upstart/upstart_1.8-0ubuntu1.dsc should be old enough to be using python2 ? | 01:32 |
| stgraber | there's a pretty good chance 1.8 already had the python3 rewrite as my branch got merged in November 2012 | 01:33 |
| stgraber | I'm sure that 1.5 is a safer bet | 01:34 |
| twb | Hm, I can't see an initctl2dot in that tarball | 01:34 |
| stgraber | twb: looked under scripts/ ? | 01:34 |
| twb | I did find -name \*dot | 01:34 |
| stgraber | ah, look for dot.py then | 01:34 |
| twb | Ah found it | 01:35 |
| twb | I could just run the script on another, newer host | 01:35 |
| twb | ba-bow: $ ssh spoon initctl show-config -e --err--> initctl: invalid command: show-config | 01:36 |
| stgraber | ah, looks like upstart under 10.04 is a bit old for that command... | 01:36 |
| stgraber | so probably easier to get a 12.04 machine with the same set of jobs and run initctl2dot on that one | 01:37 |
| twb | What this is all in aid of, is to work out why squid & an in-house daemon both fail to start sometimes when the server is rebooted | 01:37 |
| stgraber | sounds like a race condition between the network interfaces getting configured and the daemon starting | 01:38 |
| twb | Which I suspect is because they're trying to bind to an interface that isn't up or resolve a hostname before LDAP or DNS is up or something. | 01:38 |
| twb | Looking at the dependency graph seemed like the best first step :-/ | 01:38 |
| stgraber | 10.04 wasn't great for that, some stuff were switch to upstart, some were still under sysvinit, so things may be a bit random... I did a fair amount of work in 12.04 to try and get all of that a bit saner (though unfortunately it ended up being way too big to backport to 10.04) | 01:38 |
| twb | Can you pastebin what initctl show-config -e output looks like? I can probably mangle that by hand with some sed. I don't have a newer ubuntu system handy | 01:39 |
| stgraber | are squid and your inhouse software using upstart jobs or are they good old sysvinit jobs? | 01:39 |
| twb | upstart | 01:39 |
| stgraber | ok, what are their start conditions? | 01:39 |
| twb | squid is start on runlevel [2345] or net-device-up IFACE!=lo; stop on runlevel [!2345] | 01:39 |
| twb | ppcadm is start on runlevel [2345]; stop on runlevel [^2345] | 01:39 |
| stgraber | ok, and do those require some special network interfaces to appear? (thinking of bond, bridges, vlan, tunnel devices, ... that kind of stuff) | 01:40 |
| twb | That first one looks buggy to me, looks to me like it assumes a single-NIC system | 01:40 |
| twb | Lemme pastebin the interfaces(5) | 01:41 |
| stgraber | output of initctl showconfig -e on 13.10: http://paste.ubuntu.com/6152669/ | 01:41 |
| twb | http://paste.debian.net/45811/ | 01:41 |
| twb | AFAIK we don't have bonding on anything anymore, at least one with no bonding is having startup issues | 01:42 |
| stgraber | ok and upstream, staff and prisoner are all aliases for real network interfaces setup through udev? | 01:43 |
| twb | Yeah they're renamed from eth0 style names | 01:43 |
| stgraber | my current guess is that those appear as standard ethX devices which then get renamed by udev, so you get a first net-device-up event with the wrong name before it vanishes to get renamed... | 01:44 |
| twb | Ah. | 01:44 |
| stgraber | twb: you could try to get both jobs to do "start on net-device-up IFACE=prisoner" or whatever interface they actually need | 01:44 |
| twb | So if I just slap in an "AND net-device-up IFACE=staff AND net-device-up IFACE=prisoner" ? | 01:45 |
| stgraber | yep, that'd work, upstart will just wait until it got all 3 events and then will start the service | 01:45 |
| twb | I don't really care if they don't start for two minutes or whatever after boot -- plan Z is to just add a great big "pre-start exec sleep 300" | 01:45 |
| stgraber | sounds like having upstart wait for the interfaces would be considerably cleaner than a huge sleep in pre-start ;) | 01:46 |
| twb | No argument there | 01:46 |
| twb | The problem is that atm after a power outage, upstart tries a few times then gives up, end result is they never actually start and then the inmates riot because they can't get their porn. | 01:47 |
| twb | I'll do some restarts with extra net-device-up | 01:49 |
| twb | Hm, syslog says squid is getting an ABRT. | 02:44 |
| twb | (Before I change the start on) | 02:44 |
| twb | Ah, the reason squid is crashing is because some fool thought it would be a good idea to make it use a whitelist helper that talks to postgres. And postgres as at 10.04 is in sysvinit still. | 04:32 |
| twb | Fun :-/ | 04:32 |
| twb | And I'm guessing "start on runlevel [2345]" is when that runlevel *begins*, not when it ends. | 04:34 |
| twb | And the ppcadm one is not starting because it needs slapd to be accepting connections before it starts. | 05:42 |
| twb | Which is also in sysvinit as at 10.04 | 05:43 |
| SwashBuckla | Anyone written a rule for when `sudo service <service> restart` is invoked? http://upstart.ubuntu.com/cookbook/#restart | 12:55 |
| SwashBuckla | I'm writing an upstart job to daemonize a script, and am struggling to capture when `restart` is called -- what state is the process in? | 12:55 |
| SwashBuckla | is there an upstart condition? It's not pre-start or pre-stop | 12:55 |
| jodh | SwashBuckla: is the problem that your job is not able to restart the daemon script? Have you read http://upstart.ubuntu.com/cookbook/#expect ? | 13:14 |
| SwashBuckla | I can restart it just fine | 13:21 |
| SwashBuckla | that works | 13:21 |
| SwashBuckla | but I want to capture in the daemons .conf file WHEN that happens, do FOo | 13:21 |
| SwashBuckla | Foo* | 13:21 |
| jodh | SwashBuckla: there is no "restart" state - using the restart(8) command, the job is restarted whilst retaining its original config. Is this job a system-level one? If so, if you really want the job itself to know it has restarted, your best bet it probably to log something via a pre-start then check that value in the pre-start so you know how many times you've been restarted. | 13:32 |
| SwashBuckla | jodh: hmm. OK. | 13:35 |
| jodh | SwashBuckla: generally jobs shouldn't care if they have been restarted - they just need to handle that scenario | 13:35 |
| SwashBuckla | yeah I'll have to leave some breadcrumbs and look for them | 13:35 |
| SwashBuckla | I need to fire off an email in the event that a daemon was restarted, started, or stopped | 13:36 |
| SwashBuckla | as I care about the state of a daemon on a headless server | 13:36 |
| jodh | SwashBuckla: it would be better then to create a new "monitor" job that specifies "start on (started or stopped)" and does whatever you need. | 13:37 |
| PaulePanter | Hi. How do I analyze the startup process of my Ubuntu 12.04 server? | 14:33 |
| PaulePanter | Is there an equivalent to `systemd-analyze`? | 14:33 |
| jodh | PaulePanter: not atm. I suggest you apt-get install bootchart: http://www.bootchart.org/ | 15:33 |
| PaulePanter | jodh: Thanks. I already tried it, but it is not conclusive yet as a mdraid resync runs in the background. | 15:40 |
| PaulePanter | A little off topic, is there a guide what I can disable on a server (where no graphics is needed for example). | 15:40 |
| PaulePanter | I am on Ubuntu 12.04. How do I easily find out if there are Upstart scripts in later releases for packages not shipping them? | 15:53 |
| PaulePanter | Collectd is installed and does not seem to have one. | 15:54 |
| PaulePanter | Mark Brooks published one here http://blog.loadbalancer.org/upstart-script-for-collectd/ and I wonder if Ubuntu already ships one so I can compare them. | 15:56 |
| jodh | PaulePanter: Some of this is indeed rather off-topic (you are probably better off asking some of this on #ubuntu-server in future). However, you can use http://packages.ubuntu.com to query package contents for any ubuntu release. Or use 'apt-file list <pkg>' on a newer version of Ubuntu (you'll need to 'apt-get install apt-file') to see what files a package ships. | 16:06 |
| PaulePanter | jodh: Thanks. (That method is not very user-friendly though. ;-) ) | 16:09 |
| jodh | PaulePanter: ? | 16:10 |
| PaulePanter | jodh: Looking at http://packages.ubuntu.com/saucy/amd64/collectd-core/filelist and finding out that there is an upstart file, I would not be able to easily check the content of the file or copy it. | 16:15 |
| PaulePanter | In the end I would need to download the deb file and extract it, if I am not mistaken. | 16:15 |
| PaulePanter | Or are all package’s source files tracked in Launchpad in Bazaar? | 16:16 |
| jodh | PaulePanter: yes, you can access the files on bzr: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/collectd/saucy/files | 17:15 |
| PaulePanter | jodh: Awesome! Thanks. That method is easy! | 17:22 |
| PaulePanter | Though in this case I could not find the upstart job. | 17:22 |
| PaulePanter | http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/collectd/saucy/files/head:/debian/ | 17:22 |
| glisignoli | Hello, upstart newbie here. I have a server in single user mode and want to disable the starting of an upstart service for a few restarts, then enable it again. | 23:42 |
| glisignoli | *upstart 0.6.5 | 23:44 |
| glisignoli | :( | 23:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!