| === robbiew is now known as robbiew_ | ||
| oly | can anyone tell me what happens when you do stop jobname ? ie what events are sent | 12:05 |
|---|---|---|
| oly | current my job starts but does not stop so guessing it need to handle specific events | 12:06 |
| oly | is there a way to debug upstart scripts i am running stop scriptname and its locking the terminal | 13:11 |
| oly | i can press ctrl + c to cancel but how can i find why it never stops its probably my program but i want to modify it to work correctly ? | 13:12 |
| sadmac | oly: locking the terminal is rare. Are none of the other vts working? | 14:18 |
| oly | sorry vts ? i am new to upstart | 14:21 |
| sadmac | oly: other virtual terminals | 14:21 |
| sadmac | oly: ctrl+alt+f(2-6) | 14:22 |
| oly | yeah its not locking like that it just does not return to the prompt | 14:22 |
| sadmac | oly: ok, open another terminal | 14:22 |
| oly | yep | 14:22 |
| sadmac | oly: initctl list should give you something to go on | 14:23 |
| sadmac | oly: hopefully you'll get a pid from there. Then you can attach gdb to it and go nuts | 14:23 |
| oly | usm start/killed, process 22445 thats what it says | 14:24 |
| oly | and its still sat at the prompt in the other terminal trying to start | 14:24 |
| oly | there any examples around for a python daemon script | 14:25 |
| oly | as my program is a python web server basically | 14:26 |
| sadmac | oly: start/killed means it was planning on coming back up... | 14:30 |
| sadmac | oly: try kill -9 22445 | 14:30 |
| oly | says theres no such process | 14:31 |
| oly | which i have just confirmed with ps -A | grep 22445 | 14:32 |
| sadmac | oly: ok, that's a problem... | 14:34 |
| sadmac | Keybuk: any ideas? ^^ | 14:34 |
| oly | i guess its something i have done i have been trying things with the scripts to get it working | 14:35 |
| sadmac | oly: does your job have pre-stop/post-stop scripts? | 14:35 |
| oly | nope at least not now | 14:36 |
| JanC | oly: http://www.python.org/dev/peps/pep-3143/ might be the best place for info about writing a unix daemon in python | 14:36 |
| oly | okay i have a lot of that code, is there anything on the upstart side for a program like that ? | 14:37 |
| oly | or can some one recommend me a good exmple that should work with that sort of program | 14:37 |
| oly | once i know the upstart script is correct i can start checking my code | 14:38 |
| oly | at the moment prestart echo does not even print | 14:38 |
| oly | guessing thats to do with the usm start/killed message | 14:38 |
| oly | is there any way to clear that ? | 14:38 |
| JanC | http://pypi.python.org/pypi/python-daemon/ --> a library implementing correct daemon behaviour | 14:39 |
| sadmac | oly: if the task is dead and gone then upstart should be moving along | 14:41 |
| oly | those links are great think i used them to write it originally | 14:43 |
| oly | can i reset upstart so it rechecks some how ? | 14:43 |
| sadmac | oly: try doing stop usm again | 14:43 |
| sadmac | ugh. I need to get the code in front of me... | 14:44 |
| oly | usm stop/killed, process 22445 | 14:44 |
| oly | now says already stopped | 14:44 |
| oly | perhaps it just takes a very long time to stop | 14:45 |
| sadmac | oly: it should be stop/waiting | 14:48 |
| sadmac | the process is gone, but the state machine isn't progressing | 14:48 |
| === robbiew_ is now known as robbiew | ||
| sadmac | and I forget what's supposed to happen in the Killed state so I don't know what to look for | 14:48 |
| sadmac | oly: check ps for <defunct> entries | 14:49 |
| oly | none related only ones are for chromium | 14:49 |
| Keybuk | usually means that upstart has the wrong pid for the process | 14:50 |
| Keybuk | ie. expect fork when you meant expect daemon | 14:50 |
| sadmac | Keybuk: but it got to the running state. It didn't hang until he stopped it | 14:50 |
| Keybuk | was the pid right when it was in the running state? | 14:51 |
| oly | unknown and cant restart now | 14:53 |
| oly | actually will just check that | 14:53 |
| oly | yeah because its not cleared from upstart it never starts now | 14:54 |
| sadmac | upstart should really deal with these kinds of crap situations better, though I think proc connector will make most of that irrelevant | 14:54 |
| oly | thanks for the help, useful info anyway | 14:57 |
| Keybuk | also check that the ppid of your daemon is actually 1 | 14:58 |
| Keybuk | sometimes you end up watching a pid that has a different parent | 14:58 |
| Keybuk | if ppid != 1, then upstart will never know it died | 14:58 |
| Keybuk | sometimes Upstart follows the wrong fork, for example: | 14:58 |
| Keybuk | expect fork | 14:58 |
| Keybuk | script | 14:58 |
| Keybuk | OPTS=$(cat /etc/default/options) | 14:58 |
| Keybuk | exec my-daemon $OPTS | 14:58 |
| Keybuk | end script | 14:58 |
| Keybuk | in that case, upstart ends up with the PID of "cat" | 14:59 |
| Keybuk | and since cat is a child of the shell, the shell gets the SIGCHLD not upstart | 14:59 |
| Keybuk | so it fails | 14:59 |
| sadmac | its kind of astonishing how many ways this thing can break | 15:00 |
| oly | yeah well i guess i should have used ecpect daemon instead of fork from a comment further up | 15:00 |
| oly | currently my daemon is not running and nothing with the pid upstart has is running | 15:01 |
| Keybuk | daemon means "forks twice" | 15:02 |
| Keybuk | not once | 15:02 |
| sadmac | unfortunately posix doesn't seem to offer a version of kill that lets you load the arguments, so you can't send a manual CLD and get it going again | 15:03 |
| oly | hum could this be an issue my program has child daemons which are still running | 15:03 |
| sadmac | oly: no, upstart likely lost those | 15:04 |
| oly | i did not think upstart would care about those | 15:04 |
| sadmac | it won't | 15:05 |
| oly | killed them anyway | 15:05 |
| oly | also can upstart handle my app if i dont start it as a daemon ie so that it would lock the terminal | 15:06 |
| oly | as i can do this as a test by passing a prameter to the program | 15:07 |
| sadmac | oly: it could work | 15:07 |
| sadmac | oly: actually no, upstart would grab the initial process and fall over immediately | 15:08 |
| oly | okay | 15:08 |
| oly | well thanks for the help will try again a bit later i think | 15:11 |
| oly | if anyone happens to think of a way to clear upstarts current list other than rebooting let me know :) | 15:29 |
| sadmac | oly: kill -HUP 1, but that will make it forget /everything/ its managing | 15:30 |
| oly | okay that will do for now | 15:31 |
| oly | at least while testing | 15:31 |
| oly | reboot each time would be more of a problem | 15:31 |
| sadmac | exploiting bugs against bugs. isn't life grand? | 15:31 |
| oly | lol, thats a bug is it :p | 15:31 |
| oly | think upstart needs a new option to force remove a process from its list by id | 15:32 |
| sadmac | oly: kill -HUP re-executes init. Technically this should have no noticeable effect. | 15:32 |
| sadmac | oly: maybe. It'd be better if it just managed services better (and it will.) | 15:32 |
| oly | yeah that would be better, but some times it nice to have a fall back option in place | 15:36 |
| Keybuk | oly: Upstart should never lose the process | 16:30 |
| Keybuk | I'd rather spend time fixing those bugs | 16:30 |
| Keybuk | otherwise some idiot will come up with something that deliberately tells upstart to lose pids because of some other problem, etc. | 16:30 |
| oly | fair enough | 16:31 |
| oly | but it seems if i for example writes bad code i coudl always break it even if its not intentional | 16:32 |
| oly | unless it keeps following the path of my programs forking | 16:32 |
| Keybuk | right, that's the correct fix ;) | 16:33 |
| oly | ah okay :p | 16:33 |
| sadmac | oly: eventually upstart will track everything your program forks off until one of them calls setsid. | 16:33 |
| sadmac | oly: this has just taken some kernel infrastructure (now in place). | 16:33 |
| oly | aha that would be good | 16:33 |
| oly | what would happen if you branched of seperate programs from it ? | 16:34 |
| sadmac | oly: they would call setsid() | 16:34 |
| sadmac | oly: that's just good unix practice | 16:34 |
| oly | my program launches itself then two sub process as different userts which it communicates with using ipc | 16:34 |
| oly | aha fair enough | 16:34 |
| Keybuk | oly: ah | 16:42 |
| Keybuk | but in that case, you want upstart to know about all three pids | 16:43 |
| Keybuk | you might even want it to, if one of the pids dies, to deliberately send SIGTERM to the other two so that the process can be respawned | 16:43 |
| oly | maybe, although the main program currently manages the other processes so thats not really important | 16:46 |
| Keybuk | ah, but if it segfaults ... | 16:46 |
| oly | i guess, but for now i would just be happy to stop and start the main program with upstart :) | 16:47 |
| sadmac | Keybuk: to an extent thats useful. In reality to do that right the daemon would have to tell upstart which pids are what and what they mean. If an apache worker dies we ignore it. If the apache master dies we need to kill the workers. | 16:48 |
| oly | might as well create it for modern linux system as it requires a lot of modern software to function :p | 16:48 |
| Keybuk | sadmac: sure | 16:48 |
| sadmac | Keybuk: I'm guessing libupstart will return for some of this. A lot of apps will want to give little hints to upstart and won't want to deal with the whole apparatus of their native dbus API to do it. | 16:49 |
| sadmac | Keybuk: (naturally libupstart would be a wrapper around a few nih_dbus calls, not the old protocl) | 16:50 |
| Keybuk | that kind of thing, yes | 16:51 |
| JanC | I don't know the exact way apache works, but if apache master dies, won't you want the workers to finish the current request first (or does it need the master for that)? | 16:52 |
| Keybuk | JanC: that's more like ssh ;) | 16:52 |
| Keybuk | better question, would you want to restart the master while the workers are still there? | 16:53 |
| oly | i know i would certainly want my child processes to finsh | 16:53 |
| oly | as they have a queue of commands to pass through | 16:53 |
| Keybuk | there are enough interesting scenarios here that knowing about "workers" is a good idea | 16:54 |
| oly | my program is not that intrested in the child processes as long as they are running, because it just writes commands to a text file and the children act on them | 16:55 |
| oly | so they can carry on running after the parent has finished in fact if they did not the system would probably be a lot more unpredictable | 16:55 |
| oly | if any of this info is of any use :) | 16:56 |
| sadmac | oly: you're using a text file as a command spool? | 16:56 |
| oly | yeah basically | 16:56 |
| sadmac | oly: ...upstart aside you may want to re-examine your programming techniques | 16:57 |
| oly | it queue up the commands because some can take a lot of time so they must not block the original program | 16:57 |
| oly | why is there a better way than ipc type method | 16:57 |
| oly | this was the only way i could find as all 3 program run as different users so communication between them was a pain | 16:58 |
| Keybuk | nothing wrong with that style | 16:58 |
| Keybuk | it's just using the filesystem as shared memory | 16:58 |
| Keybuk | tried, tested and true UNIX way of doing things, that | 16:58 |
| sadmac | Keybuk: when we have...shared memory | 16:58 |
| Keybuk | sadmac: which is, in Linux, implemented as a filesystem | 16:58 |
| Keybuk | in which you create files | 16:59 |
| Keybuk | that you use as spools | 16:59 |
| sadmac | or for that matter, pipes and unshared memory to spool from | 16:59 |
| Keybuk | pipes are two-way only | 16:59 |
| Keybuk | oly said he has two workers and a master | 16:59 |
| Keybuk | so fifos don't work | 16:59 |
| oly | the master works as root which breaks of a web server into a restricted use account, and send info back to the program in the root account | 17:00 |
| oly | thats the type of thing i needed | 17:00 |
| sadmac | Keybuk: there is a way. It involves some descriptor passing (easier to do with threads but very possible with processes). | 17:00 |
| Keybuk | sure | 17:00 |
| sadmac | I was starting to write a web server that did this. | 17:00 |
| Keybuk | but why not just use a file? :p | 17:00 |
| sadmac | oly: are you using mmap? | 17:01 |
| oly | nope not come across mmap | 17:02 |
| sadmac | oly: ok you should at least be mmaping the file. | 17:02 |
| oly | okay whats that do then in quick terms :p | 17:03 |
| oly | i will research at some point | 17:03 |
| sadmac | oly: It lets you access the file as though it were ram | 17:05 |
| oly | sounds intresting | 17:05 |
| oly | probably would have saved me a load of work writting a parse by sounds of it :p | 17:06 |
| JanC | oly: http://docs.python.org/library/mmap.html | 17:06 |
| JanC | ;) | 17:06 |
| oly | hum, i can think of other uses for that in my program as well | 17:08 |
| oly | :) | 17:08 |
| oly | thanks for that | 17:08 |
| Keybuk | ugh | 17:12 |
| Keybuk | mmap is going to hurt them | 17:12 |
| Keybuk | my hobby: finding programs that "cleverly" mmap() files into memory, and truncating the files under them :p | 17:13 |
| === robbiew is now known as robbiew_ | ||
| === robbiew_ is now known as robbiew | ||
| === robbiew is now known as robbiew_ | ||
| === sadmac__ is now known as sadmac_home | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!