=== Md is now known as Guest18462 === Md_ is now known as Md [02:57] For lucid, is this the right way to do multiple instances? http://paste.debian.net/144039/ [03:44] How the hell do you get a list of all active instances of a job [03:44] twb: initctl list will print out all active instances [03:47] Ah, OK, and then grep my job name out of that [03:49] Yeah initctl list | sed -rn 's/^network-interface $(.*)$.*/\1/p' [03:50] initctl list | sed -rn 's/^network-interface $(.*)$.*/\1/p' | while read url do stop streamripper URL=$url done [10:15] Guys, seriously -- is "Job failed to start" the best diagnostics message I can get? [10:15] I'll tell you what, it does not suffice. [10:17] Also, can I get the instance identifier (i.e. `instance $X/$Y` I want a variable containing "$X/$Y", potentially named $INSTANCE? I would TIAS but it's impossible to diagnostice errors.) [10:18] As init(5) says, UPSTART_INSTANCE. [10:18] Anything in syslog? If not, make your scripts log something. [10:18] Nov 9 10:12:02 localhost init: gunicorn (julkort/julkort:app) pre-start process (32452) terminated with status 1 [10:18] It says [10:19] Which is a little better I agree, but I want the stderr output [10:19] pre-start script [10:19] exec >>/tmp/foo 2>&1 [10:19] … [10:20] lericson: http://upstart.ubuntu.com/cookbook/#debugging [10:23] lericson: Have you checked your job with init-checkconf (http://upstart.ubuntu.com/cookbook/#init-checkconf)? Also, you may be affected by "set -e". Read this snippet for details: http://upstart.ubuntu.com/cookbook/#debugging-a-script-which-appears-to-be-behaving-oddly [10:27] Ah, thanks -- seems my pre-start script wasn't to its liking, will look into later. [10:27] But now I get this weird behavior, inictl list says: [10:27] gunicorn (julkort/julkort:app) start/running [10:27] Yet the process is dead, and the syslog says "main process ended, respawning" twice with one second in between [10:27] I'd expect it to respawn it 10 times within 15 seconds? [10:28] Or does it somehow consider it running? [10:28] (It would seem to be the case given the list output.) [10:28] Add similar logging to the main process and see why it dies immediately. [10:29] jhunt: And I don't have init-checkconf on this server, no idea how to get it either. [10:29] lericson: sounds like you're running lucid? [10:29] ion: It dies immediately because I told it to, but I want to know why Upstart isn't doing as advertised (respawning is a major point) [10:29] jhunt: Yes, LTS [10:29] lericson: do this then - http://upstart.ubuntu.com/cookbook/#older-versions-of-upstart [10:31] Thank you! I give you two 5/5 in friendliness, expertise and professionalism-- feels like customer support. [10:32] lericson: I'd guess you hadn't told it to respawn. If you have a service you want to respawn, you need to add the "respawn" stanza to your .conf file (see "man 5 init" for details). [10:32] I like the idea of Upstart by the way, I was a Gentooist for the longest time but I think Upstart > OpenRC [10:32] I did, jhunt [10:32] Please paste your job definition and the relevant lines from syslog. [10:32] Relevant might be that the service daemonizes itself (I do have expect fork) [10:33] Are you sure the service forks exactly once? [10:33] lericson: when you say "daemonizes", do you mean that? If so, you need to specify "expect daemon" (2 forks) rather than "expect fork" (1 fork) [10:34] lericson: this is a known problem - if you don't know how many times your app forks and mis-specify it, Upstart is unable to "track" the pid so you can get odd behaviour. We intend to fix this issue for the next LTS (it's a difficult problem to address). [10:34] http://pb.lericson.se/p/SBiMze/ [10:34] I know, jhunt -- I haven't read gunicorn's source lately, so I don't know, but I would guess they don't daemonize properly. [10:35] But you're saying this is a symptom of it losing the PID? [10:36] lericson: quite possibly. I'd recommend running "strace -fFv -o /tmp/strace.log /usr/bin/gunicorn ..." and grepping for fork/clone calls in /tmp/strace.log. [10:36] Also by the way, my pre-start script seems to check out fine. [10:37] It may be simpler just to tell the main process not to daemonize and drop the “expect” stanza. [10:37] Well, interestingly http://gunicorn.org/faq.html#gunicorn-fails-to-start-with-upstart [10:38] lericson: also, your "stop on" looks wrong - there is no standard "shutdown" event on Lucid. You generally specify "stop on runlevel [016]" (ie stop on halt/single-user mode/reboot). [10:38] Oh, I ripped that off of mysql [10:39] No, I didn't. [10:39] I got it somewhere anyway [10:45] So now initctl stop blah just sits there, apparently doing nothing. [10:45] syslog is empty [10:46] I take it initctl status says ”stop/running” for the job? [10:46] ^C'd it and it says gunicorn (julkort/julkort:app) stop/killed, process 32572 [10:46] ah, stop/killed indeed. [10:46] Another symptom of lying to Upstart about the forking behavior of the main process. :-) Run workaround-upstart-snafu 32572. http://heh.fi/tmp/workaround-upstart-snafu [10:50] I'd recomment: (1) killing the pid of your gunicorn process manually, then (2) retrying the "stop". Then, (3) comment out "respawn" and "expect fork" and starting the job. If the reported pid is correct, gunicorn isn't forking at all and you can then just re-enable the respawn. [10:50] If the pid is wrong, stop the job again (kill + stop), and try adding "expect daemon". [10:51] ion: It forks until it gets the same pid and then dies? :| [10:51] yeah [10:51] The maximum number of forks an app will do is 2 (there is no benefit doing more) [10:51] jhunt: It's dead already, not even a zombie process [10:52] Out of curiosity, how does it track forks? [10:53] so does "gunicorn --daemon" coupled with "expect daemon" work for you? [10:53] lericson: it uses ptrace [10:53] jhunt: I don't know, I can't even get it to start again. [10:53] Haven't tried ion's workaround (no ruby) [10:54] lericson: We will be making some improvements to the tracking this cycle and may start using cgroups at a later stage. However, 99% of apps can be handled via ptrace. [10:55] When status says stop/killed with an inexistent process, Upstart is in a confused state, perpetually expecting to receive a SIGCHLD for a process with that PID. The workaround provides such a process. :-P [10:56] jhunt: cgroups, huh? Didn’t Keybuk come up with a proc connector implementation much superior to a cgroups implementation? There’s even a working prototype. [10:56] Can't I restart the daemon? >_> [10:56] ion: Keybuk and I discussed both options recently. The proc connector has issues. [10:57] ok [10:57] ion: specifically wrt containerized environments [10:57] Is there a fix for the cgroups issues? [10:58] lericson: unfortunately, if the pid being reported by upstart no longer exists, nominally you'll have to reboot to clear Upstarts knowledge of that job. If you're on a dev box, just copy your .conf file to a new name and work with that newly named job for the meantime. [10:59] (or use workaround-upstart-snafu) [11:02] So quite predictably, running a tool that is essentially a fork bomb fork bombed the machine [11:02] It’s not a fork bomb. [11:02] lericson: seems that gunicorn doesn't fork at all by default. [11:03] jhunt: No, but it wants to [11:03] lericson: hence, you don't need "expect" at all. [11:03] As linked above, their FAQ explicitly says "use --daemon" [11:04] lericson: have you checked your gunicorn config file (daemon=False|True)? [11:04] The script has no more than three processes at any time. [11:04] jhunt: I did, and the reason Upstart got confused is me [11:04] However I'm more focused on fixing the issue than placing blame :-) [11:05] lericson: so, any joy with copying the .conf file and modifying it? [11:06] I'm going with that approach yeah, thanks for the tip-- other question: $UPSTART_INSTANCE expands to $UPSTART_INSTANCE (or rather doesn't expand at all) in my exec line [11:08] Ah, mystery is solved; I had expect fork but Gunicorn does proper daemonization https://github.com/benoitc/gunicorn/blob/master/gunicorn/util.py#L280 [11:10] lericson: I thought we'd already come to that conclusion :) [11:11] so you should be able to run gunicorn without --daemon and without the "expect" stanza, or with --daemon and "expect daemon". If one/both of these work, I'd appreaciate it if you could let the gunicorn people know so they can update their faq (and maybe provide a "debian/gunicorn.upstart" in their distribution files) [11:13] jhunt: I'll be contributing this upstart script to gunicorn most likely, and yes it does work as intended with --daemon and expect daemon [11:13] Which seems pretty obvious now that I say it, but hey [11:14] lericson: great - thanks very much!! [11:15] BTW, OpenRC has this /etc/init.d/foo zap command which is extremely useful in situations like these as I am sure you're aware, is this avoided on a philosophical basis or just not implemented? [11:15] jhunt: I'll update the cookbook with a section on how to identify which stanzas you need for daemons of various types hopefully this week... [11:17] lericson: well, historically "both", but we do intend to resolve this issue in the current cycle. Since Upstart now supports user jobs, being able to kill a mis-specified user job is particularly important as user jobs can be used as a testing ground for system jobs. [11:28] ISTM a lot of things can make Startup get stuck, nginx has the same issue. [11:38] So the same freeze effect happens if the post-stop script fails? [11:39] Hey, nevermind [11:40] Turns out it doesn't freeze [11:44] lericson: it can be tricky to write a new .conf file, particularly when it isn't obvious what the daemon is actually doing. However, once Upstart knows how to establish the pid of your job, you're in good hands :) [11:48] (-: [11:49] It occurred to me that you can't really expand variables in the `exec` line, is this true? [11:50] Or rather they need to be isolated somehow? [11:58] lericson: expansion should work fine. Maybe you've somehow got some unicode dollar/underscores in the original version of your .conf file? [13:06] jhunt: It was in the env lines [13:13] lericson: aha! [13:40] So how does one restart a service with Upstart? [13:41] initctl restart [13:41] Ok, thanks. [13:41] Does this first stop the job and the start it again, or how does it work? [13:44] yes that's how it works [14:15] So, is there a way to add a reload event? I was thinking along the lines of initctl emit reload JOB=nginx [14:16] Hmmmm [14:33] lericson: restart stops and starts the job without reloading the job config [14:33] lericson: initctl reload will tell upstart to send SIGHUP to the tracked process [14:43] SpamapS: The daemon I'm writing the script cycles PID on HUP. [14:44] So I have a task without an exec or script section, only post-start and post-stop. It is meant to start a set of services on boot [14:45] The problem is that it does nothing. [14:45] In fact starting it just makes initctl stop === ion_ is now known as ion [15:00] lericson: yeah, you have to use stop/start for that [15:01] I actually don't like restart for most operations. :-/ [15:01] I think it's a mistake assuming all daemons reload with SIGHUP without changing, and that it is a mistake assuming that all daemons best restart via stop + start. [15:02] lericson: I think there's a missing pardigm [:start, :stop, :restart (stop, something else?, :start), :reload] [15:03] ^ would fit the way daemons actually need to work [15:04] on reload exec nginx -s reload [15:05] make it go! [15:07] ^ lericson ? [15:07] is that valid "on reload …………" [15:07] ? [15:10] Nope, but that's what I want. [15:12] ^ right :) [15:14] You can change the kill signal now... but I don't think you can change the reload signal. :-P [15:15] lericson: its not meant to do all service orchestration for you. Its meant to facilitate starting and stopping of services. You can write a script to do complex interactions like that. [15:16] I agree that assuming behaviors of tracked daemons causes as many problems as it solves. [15:16] SpamapS: HUP certainly wasn't designed historically to tell a process to RELOAD [15:23] indeed, it just became a convention of nearly every daemon ;) [15:24] since daemons, by definition, do not have a terminal, "the terminal hung up" isn't meaningful for them in its original context [15:31] that's a point I hadn't considered [15:31] but, what's stopping people inventing new signals ? [15:31] (theoretically?) [15:42] Standardization across platforms [15:59] sure, but upstart introduced an arbitrary signaling interface, did it not ? [15:59] or, "Eventing" rather === ion_ is now known as ion