| rlarkin | Hi, looking for a good link to help trouble shoot an init script inside an LXD container | 20:42 |
|---|---|---|
| rlarkin | the service ( salt-master version 15.08 ) starts just fine directly, and other init scripts are working as expected. but salt-master always fails with 'job failed to start' . I can't figure out what is expected ( and not there ) by the init script. | 20:44 |
| rlarkin | I am reading the cookbook page... | 20:45 |
| AnrDaemon | First, enable logging of init script. That's "console log". Second, enable logging of your program, if it has one. | 20:47 |
| AnrDaemon | And third, pastebin your init script so far. Let's take a look. | 20:47 |
| rlarkin | k | 20:49 |
| rlarkin | http://pastebin.com/9d6er0RC | 20:51 |
| rlarkin | doing logging | 20:51 |
| AnrDaemon | Well, that' snot upstart script. | 20:51 |
| rlarkin | huh. I just assumed trusty was upstart | 20:52 |
| AnrDaemon | It is, but what you linked is an abomination. | 20:52 |
| rlarkin | lol, that's good to know | 20:52 |
| rlarkin | it's what came with the package | 20:52 |
| AnrDaemon | They are still living in the 2008… | 20:53 |
| rlarkin | wow | 20:53 |
| rlarkin | I was trying to figure out what might be missing from my container environment and not finding anything. but it sounds like what I really need is a new init script. | 20:54 |
| AnrDaemon | http://pastebin.com/v67FsCHt | 20:55 |
| AnrDaemon | That'a about all that you need. Likely. | 20:55 |
| AnrDaemon | If salt-master have a switch to run explicitly in foreground and not daemonize, use that. | 20:55 |
| rlarkin | it has -d (daemonize) and -l debug options | 20:56 |
| rlarkin | that's so simple. | 20:56 |
| AnrDaemon | Well, okay. Just leave -d out. Upstart does process control, so it needs to know what is running. | 20:56 |
| AnrDaemon | Yes, it is, in many cases. | 20:57 |
| rlarkin | thanks man | 20:58 |
| AnrDaemon | There's 3 types of daemons. 1. Very simple ones, no config at all. 2. Those required config in comandline. 3. Those with external config. | 20:58 |
| AnrDaemon | Upstart is good for 1 and 3, you just start daemon and it does everything it has to do. | 20:58 |
| rlarkin | this is 3 (in a default location) | 20:58 |
| AnrDaemon | #2, of course, needs some treatment more often than not :/ | 20:58 |
| AnrDaemon | And not always a trivial treatment, given you can't pass options from pre-start to start. | 20:59 |
| rlarkin | cool. I'm adding this to my deployment | 20:59 |
| AnrDaemon | Do note the script don't have any start or stop clauses. You'll have to figure these your yourself, once you have your daemon up and running. | 21:02 |
| rlarkin | sure, thanks | 21:02 |
| AnrDaemon | Also check the cookbot for "expect …" syntax, if upstart still losing the tail. | 21:02 |
| AnrDaemon | cookbook** | 21:02 |
| AnrDaemon | (Some services fork to drop privileges after initialization, no way around that, unless you start them with restricted privileges to begin with, which is not always possible, i.e. if they need ports in 0..1023 range.) | 21:04 |
| rlarkin | k. | 21:04 |
| rlarkin | salt-master is not starting with the simple init script either. so I think my container env is missing something that's not needed by all services | 21:05 |
| AnrDaemon | Look in logs. (add "console log" to the script, add -l debug or what not to the daemon parameters.) | 21:07 |
| AnrDaemon | Log is in /var/log/upstart/<servicename>* | 21:07 |
| AnrDaemon | "console log" is an overall good idea, though. | 21:08 |
| rlarkin | there's no 'salt-master' file in /var/log/upstart. | 21:09 |
| rlarkin | the return is instantaneous | 21:09 |
| rlarkin | root@salt01:/etc/init.d# service salt-master start | 21:09 |
| rlarkin | start: Job failed to start | 21:09 |
| AnrDaemon | 1. Delete "salt-master" from /etc/init.d. | 21:11 |
| rlarkin | ok | 21:11 |
| AnrDaemon | 1. Run init-checkconf /etc/init/salt-master.conf | 21:11 |
| AnrDaemon | It will at least tell you if it is syntactically correct. | 21:11 |
| rlarkin | File /etc/init/salt-master.conf: syntax ok | 21:12 |
| AnrDaemon | initctl start salt-master | 21:12 |
| rlarkin | initctl: Job failed to start | 21:12 |
| AnrDaemon | What's in logs? | 21:13 |
| rlarkin | there's still no salt-master in /var/log/upstart | 21:13 |
| AnrDaemon | Anything in system log? | 21:13 |
| rlarkin | so far, syslog has only dhcp messages. | 21:13 |
| AnrDaemon | Try something real simple? Like exec /bin/printf "Started ok" | 21:15 |
| rlarkin | ok. this one fails with an instanc 'Job failed to start': http://pastebin.com/D3ZuaXrd | 21:19 |
| rlarkin | but this one works just fine: http://pastebin.com/k70KwXfh | 21:20 |
| rlarkin | permissions are identical | 21:21 |
| rlarkin | my paste is wrong, old one from klipper. I changed it to /usr/bin/printf | 21:24 |
| rlarkin | minor detail | 21:24 |
| AnrDaemon | Again, for the … time. | 21:32 |
| AnrDaemon | Do not confuse init script and upstart job. They are located in different places and started by different mechanics. | 21:33 |
| AnrDaemon | Also sorry, it is /usr/bin/printf | 21:35 |
| AnrDaemon | Just tried this script myself: http://pastebin.com/e4MFSCYM | 21:36 |
| AnrDaemon | # initctl start xxx | 21:36 |
| AnrDaemon | xxx start/running, process 19415 | 21:36 |
| rlarkin | ok yeah, that works | 21:39 |
| AnrDaemon | So, if you now replace /usr/bin/printf with /usr/bin/salt-master -l debug etc. it should at least try to start it. | 21:42 |
| rlarkin | using initctl and not service, salt-master started after I commented out #limit nofile 100000 100000 | 21:44 |
| rlarkin | and when I remove the abominable script I can start salt-master with 'service' | 21:45 |
| AnrDaemon | 100k ? :D | 21:46 |
| AnrDaemon | Even MySQL don't need that many. | 21:46 |
| AnrDaemon | Happily working with 32k | 21:46 |
| rlarkin | oh, actually it works when /etc/init.d/salt-master still exists. I guess it's not even relevant here. Hi upstart, nice to meet you. | 21:47 |
| AnrDaemon | As long as it is not installed… Or not called directly… You're using | 21:48 |
| AnrDaemon | upstart. | 21:48 |
| AnrDaemon | I suggested to get rid of it to reduce confusion. | 21:48 |
| rlarkin | at 50000 it works | 21:48 |
| rlarkin | thanks so much AnrDaemon | 21:49 |
| AnrDaemon | Does it actually do need that many open files? | 21:49 |
| rlarkin | in a much bigger environment , yes. but then I probably wouldn't be using a container | 21:50 |
| AnrDaemon | Fair point. | 21:50 |
| rlarkin | I'm actually trying to stand up a dev test env, so pretty small. just a dozen or so clients ( not thousands ) | 21:50 |
| rlarkin | the default lxd env probably sets a cap | 21:51 |
| rlarkin | ok, thanks for you help. thanks to you I can go have some family time :) | 21:51 |
| AnrDaemon | :D | 21:56 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!