| rbasak | Hellok, I'm trying to debug a an upstart boot configuration issue (cloud-init-nonet). I'm supply --debug on the kernel commandline, and this works until syslog starts, at which point I no longer see upstart debugging on the console. But I'm trying to understand the timing of events between upstart and various other things outputting to console, so ordering is important for me, and just reading syslog is insufficient. How do I get upstart to not switch t | 11:33 |
|---|---|---|
| rbasak | o syslog? | 11:33 |
| rbasak | Or must I configure syslog to dump upstart syslog to console? | 11:35 |
| SpamapS | rbasak: | 13:29 |
| SpamapS | rbasak: oops.. interesting is what I meant to type. ;) | 13:29 |
| rbasak | SpamapS: I think reconfiguring rsyslog is the way forward, but I've failed at it so far | 13:29 |
| rbasak | Unfortunately the documentation seems to assume that I want to learn everything about it before I can perform a simple task :-/ | 13:30 |
| SpamapS | rbasak: why exactly can't you just look at /var/log/syslog? | 13:30 |
| rbasak | I can, but that doesn't give the ordering of events between what's in /var/log/syslog and what else printed to the console | 13:30 |
| rbasak | Also, I'm interested in what happens before rsyslog was running, which _was_ printed to the console | 13:30 |
| rbasak | Right now, I'm getting half the logs I want to /dev/console, but suddenly they stop and the upstart-specific stuff suddenly gets redirected to /var/log/syslog | 13:31 |
| rbasak | (this is a system to which I have a serial console connected to conserver and thus both live view and logging to a file) | 13:31 |
| rbasak | And it's a boot issue, so this makes most sense for me | 13:31 |
| SpamapS | rbasak: well it will mess a bit w/ the ordering, but you could just 'echo manual | sudo tee -a /etc/init/rsyslog.override' | 13:33 |
| SpamapS | rbasak: which will make rsyslog not start | 13:33 |
| rbasak | I can try that - I hope it won't cause the problem to change :-/ | 13:34 |
| SpamapS | it really shouldn't | 13:34 |
| SpamapS | but it might speed up or slow down a race | 13:34 |
| SpamapS | which would actually be great info | 13:35 |
| SpamapS | since you'd find that there was a race :) | 13:35 |
| rbasak | It's a race I'm trying to fix! | 13:35 |
| rbasak | (at least I think it's a race) | 13:36 |
| SpamapS | rbasak: what are the symptoms? | 13:36 |
| rbasak | cloud-nonet hangs and times out on no network configured, so cloud-init never runs. And then the network gets configured. | 13:37 |
| rbasak | I think something is fundamentally wrong with the cloud-init-nonet mechanism | 13:37 |
| rbasak | (which is horrible enough as it is) | 13:37 |
| rbasak | SpamapS: reproduced with rsyslog disabled, but I still don't see the upstart logs in console after some particular point :-/ | 13:39 |
| rbasak | And now /var/log/syslog doesn't exist | 13:39 |
| SpamapS | rbasak: have you looked at dmesg ? | 13:40 |
| rbasak | No. Looks like it's going into dmesg | 13:40 |
| SpamapS | yeah, upstart logs to kmsg for all that early boot hotness :) | 13:41 |
| rbasak | What is causing kmsg to stop logging to /dev/console halfway through the boot? | 13:41 |
| SpamapS | no clue | 13:42 |
| SpamapS | could be something that is 'console owner' | 13:42 |
| pixie79 | hi all - any ideas how i get the correct pid for a process that appears to fork 6 times when starting? | 14:17 |
| SpamapS | pixie79: ugh. step 1: slap the author around a bit. ;) | 14:21 |
| SpamapS | pixie79: no but seriously.. *why* does it fork 6 times? | 14:21 |
| pixie79 | SpamapS: who knows - but i have asked the developers that | 14:23 |
| SpamapS | pixie79: perhaps those are 6 different pids that all do different things? | 14:37 |
| SpamapS | pixie79: whats important is finding the single "control pid" | 14:37 |
| SpamapS | pixie79: entirely possible that the first or second fork is the "right one" | 14:38 |
| pixie79 | SpamapS: nope, the first few disappear then we get left with two processes at the end | 14:39 |
| pixie79 | one of whoms ppid is '1' | 14:40 |
| pixie79 | any other ideas on how to follow the chain past two ? | 15:20 |
| rbasak | Found it. /etc/sysctl.d/10-console-messages.conf turns off logging to console when upstart runs the procps task | 15:25 |
| xaka | is there plan to add support for dependencies like "when i start A and it depends on B and B is not running - start B first and then start A"? in current version i can achieve that by using pre-start hook, but that is quite dirty solution and i'd like to see rule like "depends on A, B, ..." | 16:41 |
| SpamapS | xaka: there is a long term desire to rewrite with knowledge of state, mentioned in this bug and others https://bugs.launchpad.net/upstart/+bug/447654 | 17:07 |
| SpamapS | xaka: you can, for now, do it with one line in the pre-start (on Ubuntu.. others don't have wait-for-state jobs) start wait-for-state WAIT_FOR=A WAITER=me | 17:09 |
| SpamapS | xaka: personally though, I believe that such situations should be avoidable and things should be started "when they can", not "only when they are needed" | 17:10 |
| pixie79 | SpamapS: well for information i got the following back from the programs devs. The first process forks then execs the process which needs to run, this process will then fork, and then fork once more into two processes leaving me with the two final processes one parent and one child | 18:23 |
| SpamapS | pixie79: thats not really an answer to the question of "why are you doing it wrong" thats just "yes we are doing it wrong" | 18:24 |
| pixie79 | yep lol - but unfortunately they are mostly a windows house and getting any part of a linux support is hard | 18:25 |
| SpamapS | pixie79: ah, so they're just not willing/able to learn how to properly daemonize? | 18:29 |
| pixie79 | SpamapS: yep prity much | 18:29 |
| SpamapS | pixie79: well it would probably easier if you just didn't daemonize then. :-/ | 18:30 |
| SpamapS | pixie79: if thats not a possibility, then hopefully it writes a pid file somewhere? | 18:31 |
| pixie79 | i dont think we get anywhere on not daemonizing - i will check on a pid file | 18:31 |
| pixie79 | i didnt know you had the option of checking for one? | 18:32 |
| pixie79 | SpamapS: nope no pid either - i can see the current init.d job run as follows | 18:38 |
| pixie79 | qdadmin start >> /tmp/output 2>&1 | 18:38 |
| pixie79 | if [ $? = 0 ] | 18:38 |
| pixie79 | then success | 18:38 |
| SpamapS | pixie79: how does it stop the program? | 18:39 |
| pixie79 | SpamapS: the same, just swop the word start for stop | 18:39 |
| SpamapS | pixie79: there is an option to use the pid file, but you lose the 'respawn' capability, and have to run some kind of poll loop to check if the service is up in the post-start | 18:39 |
| SpamapS | pixie79: ok, so you can just make the upstart job have a pre-start which does 'exec qdadmin start' and a post-stop of 'exec qdadmin stop' | 18:40 |
| SpamapS | pixie79: you won't get a pid to track, but you don't *really* need one. | 18:40 |
| SpamapS | pixie79: http://paste.ubuntu.com/1187651/ | 18:41 |
| SpamapS | pixie79: for 11.10 and later you will get logs in /var/log/upstart/$nameofjob.log | 18:42 |
| pixie79 | SpamapS: ok - so then if it fails over it needs to be manually restarted and montored | 18:42 |
| SpamapS | pixie79: if its important, it should be monitored anyway. respawn is just to handle the most simple cases of failure | 18:43 |
| pixie79 | yes - i do have zabbix doing monitoring as well - but it is nice to have things automatically restart as well - save a lot of out of hours issues | 18:43 |
| SpamapS | pixie79: daemonizing properly (or running without daemonizing) would also probably save a lot of hours of issues. :) | 18:44 |
| pixie79 | true | 18:45 |
| pixie79 | SpamapS: well that is interesting - it would appear that when you do: qdadmin start it outputs the pid to syslog | 18:50 |
| JanC | pixie79: how does "qdadmin stop" determine what process to stop? | 18:55 |
| pixie79 | JanC: i have no idea | 18:55 |
| pixie79 | qdadmin is a binary | 18:56 |
| JanC | I suppose the programme's devs know that; but maybe it already uses a PID file or something similar? | 18:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!