/srv/irclogs.ubuntu.com/2012/09/05/#upstart.txt

rbasakHellok, I'm trying to debug a an upstart boot configuration issue (cloud-init-nonet). I'm supply --debug on the kernel commandline, and this works until syslog starts, at which point I no longer see upstart debugging on the console. But I'm trying to understand the timing of events between upstart and various other things outputting to console, so ordering is important for me, and just reading syslog is insufficient. How do I get upstart to not switch t11:33
rbasako syslog?11:33
rbasakOr must I configure syslog to dump upstart syslog to console?11:35
SpamapSrbasak: 13:29
SpamapSrbasak: oops.. interesting is what I meant to type. ;)13:29
rbasakSpamapS: I think reconfiguring rsyslog is the way forward, but I've failed at it so far13:29
rbasakUnfortunately the documentation seems to assume that I want to learn everything about it before I can perform a simple task :-/13:30
SpamapSrbasak: why exactly can't you just look at /var/log/syslog?13:30
rbasakI can, but that doesn't give the ordering of events between what's in /var/log/syslog and what else printed to the console13:30
rbasakAlso, I'm interested in what happens before rsyslog was running, which _was_ printed to the console13:30
rbasakRight now, I'm getting half the logs I want to /dev/console, but suddenly they stop and the upstart-specific stuff suddenly gets redirected to /var/log/syslog13:31
rbasak(this is a system to which I have a serial console connected to conserver and thus both live view and logging to a file)13:31
rbasakAnd it's a boot issue, so this makes most sense for me13:31
SpamapSrbasak: well it will mess a bit w/ the ordering, but you could just 'echo manual | sudo tee -a /etc/init/rsyslog.override'13:33
SpamapSrbasak: which will make rsyslog not start13:33
rbasakI can try that - I hope it won't cause the problem to change :-/13:34
SpamapSit really shouldn't13:34
SpamapSbut it might speed up or slow down a race13:34
SpamapSwhich would actually be great info13:35
SpamapSsince you'd find that there was a race :)13:35
rbasakIt's a race I'm trying to fix!13:35
rbasak(at least I think it's a race)13:36
SpamapSrbasak: what are the symptoms?13:36
rbasakcloud-nonet hangs and times out on no network configured, so cloud-init never runs. And then the network gets configured.13:37
rbasakI think something is fundamentally wrong with the cloud-init-nonet mechanism13:37
rbasak(which is horrible enough as it is)13:37
rbasakSpamapS: reproduced with rsyslog disabled, but I still don't see the upstart logs in console after some particular point :-/13:39
rbasakAnd now /var/log/syslog doesn't exist13:39
SpamapSrbasak: have you looked at dmesg ?13:40
rbasakNo. Looks like it's going into dmesg13:40
SpamapSyeah, upstart logs to kmsg for all that early boot hotness :)13:41
rbasakWhat is causing kmsg to stop logging to /dev/console halfway through the boot?13:41
SpamapSno clue13:42
SpamapScould be something that is 'console owner'13:42
pixie79hi all - any ideas how i get the correct pid for a process that appears to fork 6 times when starting?14:17
SpamapSpixie79: ugh. step 1: slap the author around a bit. ;)14:21
SpamapSpixie79: no but seriously.. *why* does it fork 6 times?14:21
pixie79SpamapS: who knows - but i have asked the developers that14:23
SpamapSpixie79: perhaps those are 6 different pids that all do different things?14:37
SpamapSpixie79: whats important is finding the single "control pid"14:37
SpamapSpixie79: entirely possible that the first or second fork is the "right one"14:38
pixie79SpamapS: nope, the first few disappear then we get left with two processes at the end14:39
pixie79one of whoms ppid is '1'14:40
pixie79any other ideas on how to follow the chain past two ?15:20
rbasakFound it. /etc/sysctl.d/10-console-messages.conf turns off logging to console when upstart runs the procps task15:25
xakais there plan to add support for dependencies like "when i start A and it depends on B and B is not running - start B first and then start A"? in current version i can achieve that by using pre-start hook, but that is quite dirty solution and i'd like to see rule like "depends on A, B, ..."16:41
SpamapSxaka: there is a long term desire to rewrite with knowledge of state, mentioned in this bug and others https://bugs.launchpad.net/upstart/+bug/44765417:07
SpamapSxaka: you can, for now, do it with one line in the pre-start (on Ubuntu.. others don't have wait-for-state jobs)  start wait-for-state WAIT_FOR=A WAITER=me17:09
SpamapSxaka: personally though, I believe that such situations should be avoidable and things should be started "when they can", not "only when they are needed"17:10
pixie79SpamapS: well for information i got the following back from the programs devs. The first process forks then execs the process which needs to run, this process will then fork, and then fork once more into two processes leaving me with the two final processes one parent and one child18:23
SpamapSpixie79: thats not really an answer to the question of "why are you doing it wrong" thats just "yes we are doing it wrong"18:24
pixie79yep lol - but unfortunately they are mostly a windows house and getting any part of a linux support is hard 18:25
SpamapSpixie79: ah, so they're just not willing/able to learn how to properly daemonize?18:29
pixie79SpamapS: yep prity much18:29
SpamapSpixie79: well it would probably easier if you just didn't daemonize then. :-/18:30
SpamapSpixie79: if thats not a possibility, then hopefully it writes a pid file somewhere?18:31
pixie79i dont think we get anywhere on not daemonizing - i will check on a pid file18:31
pixie79i didnt know you had the option of checking for one?18:32
pixie79SpamapS: nope no pid either - i can see the current init.d job run as follows18:38
pixie79qdadmin start >> /tmp/output 2>&1 18:38
pixie79if [ $? = 0 ]18:38
pixie79then success18:38
SpamapSpixie79: how does it stop the program?18:39
pixie79SpamapS: the same, just swop the word start for stop18:39
SpamapSpixie79: there is an option to use the pid file, but you lose the 'respawn' capability, and have to run some kind of poll loop to check if the service is up in the post-start18:39
SpamapSpixie79: ok, so you can just make the upstart job have a pre-start which does 'exec qdadmin start' and a post-stop of 'exec qdadmin stop'18:40
SpamapSpixie79: you won't get a pid to track, but you don't *really* need one.18:40
SpamapSpixie79: http://paste.ubuntu.com/1187651/18:41
SpamapSpixie79: for 11.10 and later you will get logs in /var/log/upstart/$nameofjob.log18:42
pixie79SpamapS: ok - so then if it fails over it needs to be manually restarted and montored18:42
SpamapSpixie79: if its important, it should be monitored anyway. respawn is just to handle the most simple cases of failure18:43
pixie79yes - i do have zabbix doing monitoring as well - but it is nice to have things automatically restart as well - save a lot of out of hours issues18:43
SpamapSpixie79: daemonizing properly (or running without daemonizing) would also probably save a lot of hours of issues. :)18:44
pixie79true 18:45
pixie79SpamapS: well that is interesting - it would appear that when you do: qdadmin start it outputs the pid to syslog18:50
JanCpixie79: how does "qdadmin stop" determine what process to stop?18:55
pixie79JanC: i have no idea18:55
pixie79qdadmin is a binary18:56
JanCI suppose the programme's devs know that; but maybe it already uses a PID file or something similar?18:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!