| === dahlSTROM is now known as dahlSTROM|sleep | ||
| biffhero | I feel like an idiot with upstart. | 02:09 |
|---|---|---|
| biffhero | status mythtv-backend ----> mythtv-backend stop/stopping | 02:11 |
| biffhero | How do I make that finish the job. | 02:11 |
| biffhero | I have no myth* processes running, there are no files in /var/run/mythtv/ , and I can't get it to restart. | 02:14 |
| biffhero | If I run the script in /etc/init/mythtv-backend.conf directly, I am OK. But I suspect there's a problem with PID tracking. | 02:14 |
| === dahlSTROM|sleep is now known as dahlSTROM | ||
| tgm4883 | biffhero, if you do a 'sudo service mythtv-backend start' it should list the PID it's tracking | 15:27 |
| biffhero | tgm4883: thanks, it wasn't showing that, but I'll 2xcheck next time I have a chance to shut down the backend. | 15:42 |
| biffhero | As a matter of fact, let me go try that right now. | 15:42 |
| tgm4883 | biffhero, when you do... | 15:42 |
| tgm4883 | also check what PID mythbackend is actually using 'ps aux | grep myth' | 15:42 |
| biffhero | That's my problem. There's no mythtv-backend running at all. | 15:43 |
| biffhero | Is there a way to see if anyone is watching anything on the frontend machine? | 15:43 |
| tgm4883 | biffhero, is the backend currently started? | 15:44 |
| biffhero | Yeah, I faked it with running the script from /etc/init/mythtv-backend.conf by hand. | 15:44 |
| tgm4883 | ah | 15:44 |
| biffhero | "exec /usr/bin/mythbackend --syslog local7 --user mythtv --daemon" | 15:44 |
| biffhero | Because if I hadn't done that, no TV. No TV, no happy wife. | 15:45 |
| tgm4883 | I don't know of a way from the backend to do it, but you could use the services API to check I think, sec | 15:45 |
| tgm4883 | biffhero, this would work 'http://frontendip:6547/Frontend/GetStatus' | 15:45 |
| biffhero | [514]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo service mythtv-backend status | 15:47 |
| biffhero | mythtv-backend start/stopping | 15:47 |
| tgm4883 | biffhero, no, 'sudo service mythtv-backend start' | 15:47 |
| tgm4883 | biffhero, so you start it with that, it should list the PID | 15:47 |
| biffhero | [514]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo service mythtv-backend start | 15:47 |
| biffhero | start: Job is already running: mythtv-backend | 15:47 |
| tgm4883 | hmm | 15:47 |
| biffhero | [515]rob@freezer:/mnt/local/sde1/rsnapshot$ !ps | 15:48 |
| biffhero | ps auxw | grep -i myth | 15:48 |
| biffhero | rob 6656 0.0 0.0 4428 836 pts/12 S+ 08:47 0:00 grep -i myth | 15:48 |
| biffhero | root 12361 0.0 0.3 55364 3136 pts/5 S+ Oct12 0:09 emacs -nw ./mythremctl.py | 15:48 |
| biffhero | root 14350 0.0 0.0 4288 128 pts/21 S+ Oct07 0:01 tail -f mythbackend.log | 15:48 |
| tgm4883 | what happens if you stop it? | 15:48 |
| biffhero | Hangs, I'll try it right now. | 15:48 |
| biffhero | Just sitting there. | 15:48 |
| tgm4883 | I seem to recall this when I was debugging the startup script. Can you pastebin the conf file? | 15:48 |
| biffhero | See the output of the "status" command. It believes that the backend has received a "start" command, and that it is in the "stopping" phase. | 15:49 |
| biffhero | sure. | 15:49 |
| biffhero | Now, I want to give you a heads up on the /etc/init/mythtv-backend.conf | 15:49 |
| biffhero | I thought that might be my problem, and I saw a bug in ubuntu about it. | 15:50 |
| biffhero | Soi I grabbed the new one. It's not much different, but I am definitely running the one from most recent, while my mythtv-backend wasn't upgraded. | 15:50 |
| tgm4883 | ok | 15:50 |
| biffhero | I used RCS to preserve the old version, so I can choose either one of them. | 15:50 |
| biffhero | Which one would you like me to pastebin? (both had the same symptoms) | 15:51 |
| tgm4883 | the new one, as it should in theory be correct | 15:51 |
| biffhero | ok | 15:51 |
| biffhero | http://paste.ubuntu.com/6330651/ | 15:51 |
| tgm4883 | hmm ok, that looks good | 15:52 |
| biffhero | I think the problem is that upstart doesn't have the equivalent of "/bin/bash -x /etc/init.d/mythtv-backend stop" where I can see exactly where it is hanging, go grab the .pid file, and move forward.... | 15:52 |
| tgm4883 | are you using packages or building from source? | 15:52 |
| biffhero | packages. | 15:52 |
| biffhero | http://paste.ubuntu.com/6330654/ | 15:53 |
| tgm4883 | Take a look in /var/log/upstart/mythtv-backend.log and see if it looks like anything is important | 15:53 |
| biffhero | Yep, I've been there. There is not an /var/log/upstart/mythtv-backend.log file. | 15:53 |
| biffhero | There are .1.gz, etc. It's like the logrotate killed it. | 15:53 |
| tgm4883 | ok | 15:54 |
| biffhero | There's a segfault at the end of the .1.gz file. | 15:54 |
| biffhero | zcat .1.gz | pastebinit ... http://paste.ubuntu.com/6330654/ | 15:55 |
| biffhero | I think we are safe to ignore all of the duplicate entry things, and move on down to the very end for the segfault. | 15:55 |
| tgm4883 | that is the same pastebin you gave me before | 15:55 |
| biffhero | Oh, my bad. | 15:55 |
| biffhero | http://paste.ubuntu.com/6330668/ | 15:55 |
| biffhero | yeah, that's it. | 15:55 |
| tgm4883 | I'm not going to worry too much about the segfault for now, that would be an upstream thing anyway | 15:57 |
| biffhero | Sure. | 15:57 |
| tgm4883 | so we'd want to open a bug with them | 15:57 |
| tgm4883 | ok, do this | 15:57 |
| tgm4883 | 'sudo service mythtv-backend stop'... then ctrl-c when it hangs | 15:58 |
| biffhero | I'm ok leaving that one alone until I can reproduce it when I have more than 22G free on my recording filesystem. Is there a way to change "keep this much free space" without re-running mythtv-setup ? I set it to 10G, but maybe that's not enough. | 15:58 |
| tgm4883 | edit the conf file | 15:58 |
| biffhero | ok. | 15:58 |
| tgm4883 | in the conf file, comment out line 18 "expect fork" | 15:58 |
| tgm4883 | save the conf file, then do | 15:58 |
| biffhero | Heh. that one "... stop" that I did before that was hanging, I just now hit Ctrl-c, and ... | 15:58 |
| biffhero | ^C[513]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo service mythtv-backend stop | 15:58 |
| biffhero | stop: Job has already been stopped: mythtv-backend | 15:58 |
| tgm4883 | 'sudo service mythtv-backend start' | 15:59 |
| tgm4883 | ok | 15:59 |
| biffhero | so it worked, I just needed to wait longer? | 15:59 |
| tgm4883 | yea, lets make that conf file change and start it again | 15:59 |
| biffhero | ok. | 15:59 |
| tgm4883 | biffhero, no, it sounds like it's tracking the wrong PID | 15:59 |
| biffhero | I couldn't figure out how to get it to spit out what PID it was tracking. | 15:59 |
| biffhero | Yeah, I saw that in a bug report, and that's there the "eval" came from, right? | 16:00 |
| tgm4883 | did you start it again after changing the conf file? | 16:00 |
| biffhero | I did last night, but not yet. I'm doing that right now. | 16:00 |
| tgm4883 | ok, yea lets remove the expect fork and see what happens | 16:01 |
| biffhero | just did it. | 16:01 |
| biffhero | the service is taking a long time to start | 16:01 |
| tgm4883 | but it's not complaining of being already started? | 16:01 |
| biffhero | that is correct. The ctrl-c of a few minutes ago was the first time that a stop seemed to report success. | 16:02 |
| tgm4883 | are you able to open another terminal and see if mythbackend is running? | 16:02 |
| biffhero | yeah, I'm tailing /var/log/mythtv/mythbackend.log in a different screen. "tail -F /var/log/mythtv/mythbackend.log" | 16:02 |
| biffhero | Nothing is happening there. | 16:02 |
| tgm4883 | ok | 16:02 |
| biffhero | mythbackend is not starting, according to "ps" | 16:03 |
| biffhero | http://paste.ubuntu.com/6330704/ | 16:03 |
| tgm4883 | what about in the upstart log for the job? | 16:03 |
| biffhero | wait, that's bad. | 16:03 |
| biffhero | http://paste.ubuntu.com/6330705/ there you go. | 16:04 |
| biffhero | nothing in /var/log/upstart | 16:04 |
| tgm4883 | /var/log/upstart/mythtv-backend.log ? | 16:04 |
| biffhero | Yes, that is correct. http://paste.ubuntu.com/6330712/ | 16:05 |
| tgm4883 | geez, wtf upstart | 16:05 |
| biffhero | yeah, it's better! (unless it isn't) Why isn't there an "-vvvv" option? | 16:05 |
| tgm4883 | superm1, ping | 16:05 |
| biffhero | the "sudo service mythtv-backend start" is still sitting. | 16:07 |
| tgm4883 | biffhero, go and and ctrl+c that | 16:07 |
| tgm4883 | as far as I can tell, upstart isn't even trying to start that job | 16:07 |
| biffhero | ok, done | 16:07 |
| biffhero | yes, I wouldn't be surprised if the mythtv ping isn't working, but how to tell? | 16:07 |
| tgm4883 | well that is checking mysql is running | 16:08 |
| biffhero | Yes, and when I run them, it is all good. | 16:08 |
| tgm4883 | humor me for a second, can you logout of root and go back to your regular user and starting it? | 16:08 |
| biffhero | root@freezer:/mnt/local/sdb1/shares/mythtv/recordings# /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping >/dev/null | 16:08 |
| biffhero | root@freezer:/mnt/local/sdb1/shares/mythtv/recordings# echo $? | 16:08 |
| biffhero | 0 | 16:08 |
| tgm4883 | you might have to issue the stop command first | 16:08 |
| biffhero | I thought I _was_ using my regular user to do the sudo. Wasn't I? Hang on. | 16:09 |
| biffhero | yep, I was. | 16:09 |
| tgm4883 | hmm ok | 16:09 |
| tgm4883 | so the above is from a different terminal where you are logged in as root? | 16:10 |
| biffhero | the mysqladmin was from root. I'll try that as a user, but I have been doing "sudo", so .... ? | 16:10 |
| tgm4883 | yea IDK, this just isn't making any sense. I would expect something to show up in the log files | 16:11 |
| biffhero | [513]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping >/dev/null | 16:11 |
| biffhero | [514]rob@freezer:/mnt/local/sde1/rsnapshot$ echo $? | 16:11 |
| biffhero | 0 | 16:11 |
| biffhero | Me too. | 16:11 |
| biffhero | I'll comment out that pre-start check, and see where that goes. | 16:11 |
| biffhero | I put the "expect fork" back in, removed the "pre-start script" | 16:12 |
| biffhero | [516]rob@freezer:/mnt/local/sde1/rsnapshot$ time sudo service mythtv-backend start | 16:12 |
| biffhero | hanging. | 16:12 |
| biffhero | commenting out the "expect fork", so both of them are out. | 16:13 |
| tgm4883 | ok | 16:13 |
| tgm4883 | biffhero, also remove the --daemon from the command to start mythbackend | 16:13 |
| biffhero | ok, you think that the --daemon is a problem? I'm getting hanging, moving on to the "remove --daemon" test now. | 16:14 |
| biffhero | wait, do we want "expect fork" in there or not? | 16:15 |
| tgm4883 | biffhero, no, it will put mythtv logs also in the upstart log too | 16:15 |
| tgm4883 | so hopefully we'll see *something* happen | 16:15 |
| biffhero | ok, no exec, no pre-start, no --daemon | 16:16 |
| tgm4883 | the --daemon also causes mythbackend to fork (which is why we have expect fork), so the way we've got it now in yours is that mythbackend isn't forking, upstart isn't expecting a fork | 16:16 |
| biffhero | here's an interesting one. | 16:16 |
| biffhero | before I re-run the start, I do a "time sudo service mythtv-backend stop" first, just for safety. The first one (right now) hangs. | 16:17 |
| biffhero | Ctrl-c, 38 second wait. | 16:17 |
| biffhero | run it again, it immediately says, "already been stopped" | 16:17 |
| tgm4883 | yes, always do the stop | 16:17 |
| biffhero | running start now. | 16:17 |
| biffhero | hanging, nothing showing up in /var/log/upstart | 16:18 |
| biffhero | nothing showing up in /var/log/mythtv/ | 16:18 |
| biffhero | http://paste.ubuntu.com/6330795/ it isn't starting | 16:21 |
| biffhero | Waited 6m30s, not started, ctrl-c, starting it by hand, in case there's anyone at home who is watching. | 16:24 |
| biffhero | Let me know if you find the -vvvv option to /sbin/initctl | 16:25 |
| superm1 | tgm4883: yeah? | 16:46 |
| superm1 | biffhero: just to double check. what version of ubuntu are you running and what version of mythtv? | 17:02 |
| tgm4883 | superm1, it's really odd, it's like upstart isn't even attempting to start the job | 17:03 |
| superm1 | and is this upstart behavior a "new" problem, or this has been from the get go a problem | 17:03 |
| tgm4883 | I'm not sure if there is a way to debug upstart easily | 17:03 |
| superm1 | it's possible to debug upstart - /var/log/upstart logs stuff | 17:03 |
| superm1 | jobs output will end up there | 17:03 |
| tgm4883 | superm1, yea, but there is nothing for the mythtv job | 17:04 |
| superm1 | at least on my 12.04 based ubuntu box I see tons of logs there | 17:04 |
| tgm4883 | so then where do you go | 17:04 |
| tgm4883 | superm1, that is the correct place for job logs | 17:04 |
| superm1 | /var/log/upstart/$job.log | 17:04 |
| tgm4883 | but his doesn't have any new stuff, only old stuff | 17:04 |
| superm1 | does mythtv-backend.log show anything? | 17:04 |
| superm1 | oh. ok | 17:04 |
| tgm4883 | nope | 17:04 |
| biffhero | no. | 17:04 |
| biffhero | getting versions. | 17:04 |
| biffhero | 13.04, 2:0.26.0+fixes.20121118.340b5d4-0ubuntu1 | 17:05 |
| biffhero | nothing shows up in /var/log/upstart/ | 17:05 |
| biffhero | I'm going to restart something else, see if it goes in there. | 17:05 |
| superm1 | oh, ok so quite newer | 17:06 |
| superm1 | has this ever worked? | 17:06 |
| superm1 | the fixes for --daemon and forking and such weren't commited until about 6 months ago. https://github.com/MythTV/packaging/commit/7e395a139566134b82a05f32382a18f38799c9fe | 17:07 |
| biffhero | Yes, it has been working for quite a while. | 17:07 |
| biffhero | Yesterday morning, it decided to die. I'm not sure why, but there's a segfault in a job at 6am, feels like a logrotate timer. | 17:08 |
| superm1 | just to make sure / isn't full right? | 17:08 |
| biffhero | 87G free. | 17:09 |
| superm1 | and does this same behavior happen on a fresh boot? | 17:09 |
| biffhero | There's _nothing_ in /var/log/upstart/ that's not a .x.gz file... | 17:09 |
| biffhero | no, that's where it works OK. | 17:09 |
| biffhero | I haven't been able to reboot this time. | 17:10 |
| superm1 | so it's only upon trying to stop and restart the job it's happening | 17:11 |
| superm1 | you "can't" reboot? as in /sbin/reboot isn't working? | 17:15 |
| biffhero | As in right now it is busy doing other things. | 17:17 |
| biffhero | so I'm confused as to why the upstart isn't doing the job, and trying to debug it. | 17:18 |
| superm1 | oh ok | 17:18 |
| superm1 | well we debugged this ages back when we saw similar problems with it not tracking pids properly and reboots taking a long time | 17:18 |
| superm1 | that link above on github is what we came up with that should properly execute and track the pid and what we keep in 0.26, 0.27, and 0.28 branches now | 17:19 |
| biffhero | I _can_ run it ok if I just pull the script out of the command, but I don't know how to make upstart tell me what it is doing. It seems like such a black box move away from "/bin/bash -x" | 17:19 |
| biffhero | yeah, I have those files, have moved through a few iterations of them. I think these are all upstart problems. | 17:19 |
| superm1 | http://upstart.ubuntu.com/wiki/Debugging you can turn on verbosity of upstart logging | 17:19 |
| superm1 | just add --verbose to your kernel command line and reboot when you can | 17:20 |
| biffhero | it requires a reboot! bah. what a piece of work. | 17:20 |
| biffhero | So can you go from --verbose to non-verbose without a reboot? | 17:21 |
| superm1 | upstart manages pid 1 (/sbin/init) | 17:22 |
| superm1 | that's what it increases verbosity of | 17:22 |
| superm1 | so you can't restart /sbin/init without rebooting | 17:22 |
| biffhero | and it won't take a signal to re-read the config file? | 17:22 |
| biffhero | I thought that's what "kill -HUP" was for. | 17:23 |
| superm1 | if you want to add more logging for the job like they show on that debugging examples there you can do that | 17:23 |
| superm1 | and it rereads jobs | 17:23 |
| superm1 | but the managing daemon itself i don't believe will | 17:23 |
| superm1 | In newer versions of Upstart, "initctl log-priority <priority>" will set the level of logging. See the initctl manpage for valid priority levels. The debug and message priority levels can be useful for debugging job definitions. | 17:23 |
| superm1 | not sure whether the 13.04 version has that feature or that came in 13.10 | 17:24 |
| biffhero | nice page. initctl list | less shows a lot of stuff in "stop/waiting" | 17:26 |
| biffhero | but the one that I just shut down on my own shows "stop/killed, process xxxx" | 17:27 |
| superm1 | some jobs are only intended to run and then stop immediately | 17:28 |
| superm1 | i think upstart calls them "tasks" | 17:28 |
| biffhero | The /waiting makes me think that they aren't finished yet. | 17:28 |
| superm1 | i believe the stop/ means they are presenting stopped. the /waiting means they are waiting for a signal to start | 17:30 |
| superm1 | similar that a bunch that say start/ are presently in the started state | 17:30 |
| biffhero | Sure. | 17:31 |
| biffhero | But by that logic, wouldn't start/waiting mean the same as stop/waiting ? | 17:31 |
| biffhero | start/started == stop/stopped | 17:32 |
| superm1 | oh the manual says "<goal>/<status>" http://upstart.ubuntu.com/cookbook/#stopped-job | 17:34 |
| superm1 | 10.1.6.18.1 | 17:34 |
| biffhero | ok | 17:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!