=== dahlSTROM is now known as dahlSTROM|sleep
[02:09] <biffhero> I feel like an idiot with upstart.
[02:11] <biffhero> status mythtv-backend ----> mythtv-backend stop/stopping
[02:11] <biffhero> How do I make that finish the job.
[02:14] <biffhero> I have no myth* processes running, there are no files in /var/run/mythtv/ , and I can't get it to restart.
[02:14] <biffhero> If I run the script in /etc/init/mythtv-backend.conf directly, I am OK.  But I suspect there's a problem with PID tracking.
=== dahlSTROM|sleep is now known as dahlSTROM
[15:27] <tgm4883> biffhero, if you do a 'sudo service mythtv-backend start' it should list the PID it's tracking
[15:42] <biffhero> tgm4883: thanks, it wasn't showing that, but I'll 2xcheck next time I have a chance to shut down the backend.
[15:42] <biffhero> As a matter of fact, let me go try that right now.
[15:42] <tgm4883> biffhero, when you do...
[15:42] <tgm4883> also check what PID mythbackend is actually using 'ps aux | grep myth'
[15:43] <biffhero> That's my problem.  There's no mythtv-backend running at all.
[15:43] <biffhero> Is there a way to see if anyone is watching anything on the frontend machine?
[15:44] <tgm4883> biffhero, is the backend currently started?
[15:44] <biffhero> Yeah, I faked it with running the script from /etc/init/mythtv-backend.conf by hand.
[15:44] <tgm4883> ah
[15:44] <biffhero> "exec /usr/bin/mythbackend --syslog local7 --user mythtv --daemon"
[15:45] <biffhero> Because if I hadn't done that, no TV.  No TV, no happy wife.
[15:45] <tgm4883> I don't know of a way from the backend to do it, but you could use the services API to check I think, sec
[15:45] <tgm4883> biffhero, this would work 'http://frontendip:6547/Frontend/GetStatus'
[15:47] <biffhero> [514]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo service mythtv-backend status
[15:47] <biffhero> mythtv-backend start/stopping
[15:47] <tgm4883> biffhero, no, 'sudo service mythtv-backend start'
[15:47] <tgm4883> biffhero, so you start it with that, it should list the PID
[15:47] <biffhero> [514]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo service mythtv-backend start
[15:47] <biffhero> start: Job is already running: mythtv-backend
[15:47] <tgm4883> hmm
[15:48] <biffhero> [515]rob@freezer:/mnt/local/sde1/rsnapshot$ !ps
[15:48] <biffhero> ps auxw | grep -i myth
[15:48] <biffhero> rob       6656  0.0  0.0   4428   836 pts/12   S+   08:47   0:00 grep -i myth
[15:48] <biffhero> root     12361  0.0  0.3  55364  3136 pts/5    S+   Oct12   0:09 emacs -nw ./mythremctl.py
[15:48] <biffhero> root     14350  0.0  0.0   4288   128 pts/21   S+   Oct07   0:01 tail -f mythbackend.log
[15:48] <tgm4883> what happens if you stop it?
[15:48] <biffhero> Hangs, I'll try it right now.
[15:48] <biffhero> Just sitting there.
[15:48] <tgm4883> I seem to recall this when I was debugging the startup script. Can you pastebin the conf file?
[15:49] <biffhero> See the output of the "status" command.  It believes that the backend has received a "start" command, and that it is in the "stopping" phase.
[15:49] <biffhero> sure.
[15:49] <biffhero> Now, I want to give you a heads up on the /etc/init/mythtv-backend.conf
[15:50] <biffhero> I thought that might be my problem, and I saw a bug in ubuntu about it.
[15:50] <biffhero> Soi I grabbed the new one.  It's not much different, but I am definitely running the one from most recent, while my mythtv-backend wasn't upgraded.
[15:50] <tgm4883> ok
[15:50] <biffhero> I used RCS to preserve the old version, so I can choose either one of them.
[15:51] <biffhero> Which one would you like me to pastebin?  (both had the same symptoms)
[15:51] <tgm4883> the new one, as it should in theory be correct
[15:51] <biffhero> ok
[15:51] <biffhero> http://paste.ubuntu.com/6330651/
[15:52] <tgm4883> hmm ok, that looks good
[15:52] <biffhero> I think the problem is that upstart doesn't have the equivalent of "/bin/bash -x /etc/init.d/mythtv-backend stop"  where I can see exactly where it is hanging, go grab the .pid file, and move forward....
[15:52] <tgm4883> are you using packages or building from source?
[15:52] <biffhero> packages.
[15:53] <biffhero> http://paste.ubuntu.com/6330654/
[15:53] <tgm4883> Take a look in /var/log/upstart/mythtv-backend.log and see if it looks like anything is important
[15:53] <biffhero> Yep, I've been there.  There is not an /var/log/upstart/mythtv-backend.log file.
[15:53] <biffhero> There are .1.gz, etc.  It's like the logrotate killed it.
[15:54] <tgm4883> ok
[15:54] <biffhero> There's a segfault at the end of the .1.gz file.
[15:55] <biffhero> zcat .1.gz | pastebinit ... http://paste.ubuntu.com/6330654/
[15:55] <biffhero> I think we are safe to ignore all of the duplicate entry things, and move on down to the very end for the segfault.
[15:55] <tgm4883> that is the same pastebin you gave me before
[15:55] <biffhero> Oh, my bad.
[15:55] <biffhero> http://paste.ubuntu.com/6330668/
[15:55] <biffhero> yeah, that's it.
[15:57] <tgm4883> I'm not going to worry too much about the segfault for now, that would be an upstream thing anyway
[15:57] <biffhero> Sure.
[15:57] <tgm4883> so we'd want to open a bug with them
[15:57] <tgm4883> ok, do this
[15:58] <tgm4883> 'sudo service mythtv-backend stop'... then ctrl-c when it hangs
[15:58] <biffhero> I'm ok leaving that one alone until I can reproduce it when I have more than 22G free on my recording filesystem.  Is there a way to change "keep this much free space" without re-running mythtv-setup ?  I set it to 10G, but maybe that's not enough.
[15:58] <tgm4883> edit the conf file
[15:58] <biffhero> ok.
[15:58] <tgm4883> in the conf file, comment out line 18 "expect fork"
[15:58] <tgm4883> save the conf file, then do
[15:58] <biffhero> Heh.  that one "... stop" that I did before that was hanging, I just now hit Ctrl-c, and ...
[15:58] <biffhero> ^C[513]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo service mythtv-backend stop
[15:58] <biffhero> stop: Job has already been stopped: mythtv-backend
[15:59] <tgm4883> 'sudo service mythtv-backend start'
[15:59] <tgm4883> ok
[15:59] <biffhero> so it worked, I just needed to wait longer?
[15:59] <tgm4883> yea, lets make that conf file change and start it again
[15:59] <biffhero> ok.
[15:59] <tgm4883> biffhero, no, it sounds like it's tracking the wrong PID
[15:59] <biffhero> I couldn't figure out how to get it to spit out what PID it was tracking.
[16:00] <biffhero> Yeah, I saw that in a bug report, and that's there the "eval" came from, right?
[16:00] <tgm4883> did you start it again after changing the conf file?
[16:00] <biffhero> I did last night, but not yet.  I'm doing that right now.
[16:01] <tgm4883> ok, yea lets remove the expect fork and see what happens
[16:01] <biffhero> just did it.
[16:01] <biffhero> the service is taking a long time to start
[16:01] <tgm4883> but it's not complaining of being already started?
[16:02] <biffhero> that is correct.  The ctrl-c of a few minutes ago was the first time that a stop seemed to report success.
[16:02] <tgm4883> are you able to open another terminal and see if mythbackend is running?
[16:02] <biffhero> yeah, I'm tailing /var/log/mythtv/mythbackend.log in a different screen.  "tail -F /var/log/mythtv/mythbackend.log"
[16:02] <biffhero> Nothing is happening there.
[16:02] <tgm4883> ok
[16:03] <biffhero> mythbackend is not starting, according to "ps"
[16:03] <biffhero> http://paste.ubuntu.com/6330704/
[16:03] <tgm4883> what about in the upstart log for the job?
[16:03] <biffhero> wait, that's bad.
[16:04] <biffhero> http://paste.ubuntu.com/6330705/ there you go.
[16:04] <biffhero> nothing in /var/log/upstart
[16:04] <tgm4883>  /var/log/upstart/mythtv-backend.log ?
[16:05] <biffhero> Yes, that is correct.  http://paste.ubuntu.com/6330712/
[16:05] <tgm4883> geez, wtf upstart
[16:05] <biffhero> yeah, it's better!  (unless it isn't)  Why isn't there an "-vvvv" option?
[16:05] <tgm4883> superm1, ping
[16:07] <biffhero> the "sudo service mythtv-backend start" is still sitting.
[16:07] <tgm4883> biffhero, go and and ctrl+c that
[16:07] <tgm4883> as far as I can tell, upstart isn't even trying to start that job
[16:07] <biffhero> ok, done
[16:07] <biffhero> yes, I wouldn't be surprised if the mythtv ping isn't working, but how to tell?
[16:08] <tgm4883> well that is checking mysql is running
[16:08] <biffhero> Yes, and when I run them, it is all good.
[16:08] <tgm4883> humor me for a second, can you logout of root and go back to your regular user and starting it?
[16:08] <biffhero> root@freezer:/mnt/local/sdb1/shares/mythtv/recordings# /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping >/dev/null
[16:08] <biffhero> root@freezer:/mnt/local/sdb1/shares/mythtv/recordings# echo $?
[16:08] <biffhero> 0
[16:08] <tgm4883> you might have to issue the stop command first
[16:09] <biffhero> I thought I _was_ using my regular user to do the sudo.  Wasn't I?  Hang on.
[16:09] <biffhero> yep, I was.
[16:09] <tgm4883> hmm ok
[16:10] <tgm4883> so the above is from a different terminal where you are logged in as root?
[16:10] <biffhero> the mysqladmin was from root.  I'll try that as a user, but I have been doing "sudo", so .... ?
[16:11] <tgm4883> yea IDK, this just isn't making any sense. I would expect something to show up in the log files
[16:11] <biffhero> [513]rob@freezer:/mnt/local/sde1/rsnapshot$ sudo /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping >/dev/null
[16:11] <biffhero> [514]rob@freezer:/mnt/local/sde1/rsnapshot$ echo $?
[16:11] <biffhero> 0
[16:11] <biffhero> Me too.
[16:11] <biffhero> I'll comment out that pre-start check, and see where that goes.
[16:12] <biffhero> I put the "expect fork" back in, removed the "pre-start script"
[16:12] <biffhero> [516]rob@freezer:/mnt/local/sde1/rsnapshot$ time sudo service mythtv-backend start
[16:12] <biffhero> hanging.
[16:13] <biffhero> commenting out the "expect fork", so both of them are out.
[16:13] <tgm4883> ok
[16:13] <tgm4883> biffhero, also remove the --daemon from the command to start mythbackend
[16:14] <biffhero> ok, you think that the --daemon is a problem?  I'm getting hanging, moving on to the "remove --daemon" test now.
[16:15] <biffhero> wait, do we want "expect fork" in there or not?
[16:15] <tgm4883> biffhero, no, it will put mythtv logs also in the upstart log too
[16:15] <tgm4883> so hopefully we'll see *something* happen
[16:16] <biffhero> ok, no exec, no pre-start, no --daemon
[16:16] <tgm4883> the --daemon also causes mythbackend to fork (which is why we have expect fork), so the way we've got it now in yours is that mythbackend isn't forking, upstart isn't expecting a fork
[16:16] <biffhero> here's an interesting one.
[16:17] <biffhero> before I re-run the start, I do a "time sudo service mythtv-backend stop" first, just for safety.  The first one (right now) hangs.
[16:17] <biffhero> Ctrl-c, 38 second wait.
[16:17] <biffhero> run it again, it immediately says, "already been stopped"
[16:17] <tgm4883> yes, always do the stop
[16:17] <biffhero> running start now.
[16:18] <biffhero> hanging, nothing showing up in /var/log/upstart
[16:18] <biffhero> nothing showing up in /var/log/mythtv/
[16:21] <biffhero> http://paste.ubuntu.com/6330795/ it isn't starting
[16:24] <biffhero> Waited 6m30s, not started, ctrl-c, starting it by hand, in case there's anyone at home who is watching.
[16:25] <biffhero> Let me know if you find the -vvvv option to /sbin/initctl
[16:46] <superm1> tgm4883: yeah?
[17:02] <superm1> biffhero: just to double check.  what version of ubuntu are you running and what version of mythtv?
[17:03] <tgm4883> superm1, it's really odd, it's like upstart isn't even attempting to start the job
[17:03] <superm1> and is this upstart behavior a "new" problem, or this has been from the get go a problem
[17:03] <tgm4883> I'm not sure if there is a way to debug upstart easily
[17:03] <superm1> it's possible to debug upstart  - /var/log/upstart logs stuff
[17:03] <superm1> jobs output will end up there
[17:04] <tgm4883> superm1, yea, but there is nothing for the mythtv job
[17:04] <superm1> at least on my 12.04 based ubuntu box I see tons of logs there
[17:04] <tgm4883> so then where do you go
[17:04] <tgm4883> superm1, that is the correct place for job logs
[17:04] <superm1>  /var/log/upstart/$job.log
[17:04] <tgm4883> but his doesn't have any new stuff, only old stuff
[17:04] <superm1> does mythtv-backend.log show anything?
[17:04] <superm1> oh.  ok
[17:04] <tgm4883> nope
[17:04] <biffhero> no.
[17:04] <biffhero> getting versions.
[17:05] <biffhero> 13.04, 2:0.26.0+fixes.20121118.340b5d4-0ubuntu1
[17:05] <biffhero> nothing shows up in /var/log/upstart/
[17:05] <biffhero> I'm going to restart something else, see if it goes in there.
[17:06] <superm1> oh, ok so quite newer
[17:06] <superm1> has this ever worked?
[17:07] <superm1> the fixes for --daemon and forking and such weren't commited until about 6 months ago.  https://github.com/MythTV/packaging/commit/7e395a139566134b82a05f32382a18f38799c9fe
[17:07] <biffhero> Yes, it has been working for quite a while.
[17:08] <biffhero> Yesterday morning, it decided to die.  I'm not sure why, but there's a segfault in a job at 6am, feels like a logrotate timer.
[17:08] <superm1> just to make sure / isn't full right?
[17:09] <biffhero> 87G free.
[17:09] <superm1> and does this same behavior happen on a fresh boot?
[17:09] <biffhero> There's _nothing_ in /var/log/upstart/ that's not a .x.gz file...
[17:09] <biffhero> no, that's where it works OK.
[17:10] <biffhero> I haven't been able to reboot this time.
[17:11] <superm1> so it's only upon trying to stop and restart the job it's happening
[17:15] <superm1> you "can't" reboot?  as in /sbin/reboot isn't working?
[17:17] <biffhero> As in right now it is busy doing other things.
[17:18] <biffhero> so I'm confused as to why the upstart isn't doing the job, and trying to debug it.
[17:18] <superm1> oh ok
[17:18] <superm1> well we debugged this ages back when we saw similar problems with it not tracking pids properly and reboots taking a long time
[17:19] <superm1> that link above on github is what we came up with that should properly execute and track the pid and what we keep in 0.26, 0.27, and 0.28 branches now
[17:19] <biffhero> I _can_ run it ok if I just pull the script out of the command, but I don't know how to make upstart tell me what it is doing.  It seems like such a black box move away from "/bin/bash -x"
[17:19] <biffhero> yeah, I have those files, have moved through a few iterations of them.  I think these are all upstart problems.
[17:19] <superm1> http://upstart.ubuntu.com/wiki/Debugging you can turn on verbosity of upstart logging
[17:20] <superm1> just add --verbose to your kernel command line and reboot when you can
[17:20] <biffhero> it requires a reboot!  bah.  what a piece of work.
[17:21] <biffhero> So can you go from --verbose to non-verbose without a reboot?
[17:22] <superm1> upstart manages pid 1 (/sbin/init)
[17:22] <superm1> that's what it increases verbosity of
[17:22] <superm1> so you can't restart /sbin/init without rebooting
[17:22] <biffhero> and it won't take a signal to re-read the config file?
[17:23] <biffhero> I thought that's what "kill -HUP" was for.
[17:23] <superm1> if you want to add more logging for the job like they show on that debugging examples there you can do that
[17:23] <superm1> and it rereads jobs
[17:23] <superm1> but the managing daemon itself i don't believe will
[17:23] <superm1> In newer versions of Upstart, "initctl log-priority <priority>" will set the level of logging. See the initctl manpage for valid priority levels. The debug and message priority levels can be useful for debugging job definitions.
[17:24] <superm1> not sure whether the 13.04 version has that feature or that came in 13.10
[17:26] <biffhero> nice page.  initctl list | less shows a lot of stuff in "stop/waiting"
[17:27] <biffhero> but the one that I just shut down on my own shows "stop/killed, process xxxx"
[17:28] <superm1> some jobs are only intended to run and then stop immediately
[17:28] <superm1> i think upstart calls them "tasks"
[17:28] <biffhero> The /waiting makes me think that they aren't finished yet.
[17:30] <superm1> i believe the stop/ means they are presenting stopped.  the /waiting means they are waiting for a signal to start
[17:30] <superm1> similar that a bunch that say start/ are presently in the started state
[17:31] <biffhero> Sure.
[17:31] <biffhero> But by that logic, wouldn't start/waiting mean the same as stop/waiting ?
[17:32] <biffhero> start/started == stop/stopped
[17:34] <superm1> oh the manual says "<goal>/<status>" http://upstart.ubuntu.com/cookbook/#stopped-job
[17:34] <superm1> 10.1.6.18.1
[17:42] <biffhero> ok