Article 12 of 15
When Services Don’t Start
It’s shocking, shocking I say, but even on a properly configured system, services sometimes fail to start.
The Three-Layer Diagnosis
When a service doesn’t start, you’re likely dealing with one of three problems:
- The init script: The script may have incorrect permissions, syntax errors, or missing references.
- The service program: The daemon itself may be misconfigured, missing dependencies, or encountering runtime errors.
- The dependencies: A required facility may not be available at boot time, even if the init script ran correctly.
A successful init script call doesn’t always mean the service is actually functional. The script ran but that doesn’t prove the program succeeded.
Reading Logs Effectively
Logs are your primary window into what went wrong.
On BcalcOS, rsyslog receives messages from services and other parts of the system and writes them to log files according to the rules in /etc/rsyslog.conf and any included configuration files under /etc/rsyslog.d/.
Common destinations include:
/var/log/syslog: general system messages, including most service activity/var/log/auth.log: authentication and privilege-related events/var/log/cron.log: messages from the cron scheduler/var/log/kern.log: kernel messages/var/log/mail.log: the file may not exist if no messages have been logged/var/log/ufw.log: firewall activity
You can easily see this by opening /etc/rsyslog.conf. The file is plain text and each line maps a category of messages to a destination file.
Checking Recent Activity
Open the main log and filter by service name:
grep cron /var/log/syslog
You can also watch the log live while attempting to start the service. Keep the log visible in one terminal:
tail -f /var/log/syslog
Then in a second terminal:
sudo /etc/init.d/cron start
Watch the log output as you attempt to start the service. Error messages will appear in real time.
Checking Dependencies Practically
Every init script with an LSB header declares what it needs:
### BEGIN INIT INFO
# Required-Start: $remote_fs $syslog $time
...
### END INIT INFO
If a service fails, did all the declared facilities actually start?
For example, if you’re troubleshooting cron and it declares Required-Start: $syslog, then rsyslog must be running before cron starts. Check both services:
/etc/init.d/rsyslog status
/etc/init.d/cron status
If rsyslog isn’t running, cron may still run, but its messages may not be recorded in the expected log files. Check both services’ status and the available logs to determine what’s happening.
You can also examine the dependency data directly:
cat /etc/init.d/.depend.start
This file, generated by insserv, records startup targets and dependency relationships. The TARGETS line lists services considered by the generated startup configuration. Later lines show specific dependencies:
single: killprocs
cron: rsyslog
When you see cron: rsyslog, it means the generated dependency data lists rsyslog as a prerequisite for cron. This helps explain startup ordering, but it does not prove that either service started successfully.
A Systematic Approach
-
Check the status first.
/etc/init.d/NAME statustells you whether the service thinks it’s running. -
Read the log output.
Look for errors, permission denials, missing files, or timeout messages. -
Verify dependencies are actually running.
When you check a service’s status, look for output indicating it’s active and running (often showing a process ID or an “active”/“running” message). Confirm that a required service is not inactive or stopped. -
Make one change at a time.
Changing multiple settings simultaneously makes it difficult to know what fixed the problem. -
Verify the change worked.
Re-check status and logs after each modification.
Common Failure Patterns
Here are some scenarios you may encounter:
| Symptom | Likely Cause | Check |
|---|---|---|
| Script runs but service doesn’t listen on expected port | Port conflict, firewall, misconfiguration | sudo ss -tlnp | grep PORT |
| Permission denied in log | File permissions, missing user/group | ls -l /path/to/config |
| Service exits immediately after starting | Missing configuration file | Look for config file paths in init script |
| Timeout during startup | Resource contention, slow dependency | Check /var/log/kern.log for kernel messages |
| Works when started manually but not at boot | Dependency timing issue | Compare manual vs. boot environment |
At boot, services start in sequence; when you start a service manually, everything else is already running. That’s why a service can work when started by hand but fail at boot: a dependency may not have finished starting yet. Check the service’s Required-Start field (in the init script) and confirm those services are actually running before yours attempts to start.
As a general rule, service configuration problems show up in /var/log/syslog, while resource or hardware trouble (memory exhaustion, disk errors, I/O failures) is more likely to appear in /var/log/kern.log.
Each of these has a diagnostic path rooted in the three-layer model: script, service, or dependency.
When the Init Script Itself Is Broken
If the init script has syntax errors, the shell will report them:
sudo /etc/init.d/name start
/etc/init.d/name: 23: Syntax error: "fi" unexpected
This usually means:
- A typo in the script
- Missing executable permissions
- An incorrect shebang line
The first line (shebang) must point to a valid shell interpreter. On BcalcOS, this is usually #!/bin/sh for portability or #!/bin/bash for bash-specific features.
Check permissions:
ls -l /etc/init.d/name
If the script lacks executable permission, investigate why before changing it. Init scripts installed by packages should normally have the correct permissions; an unexpected change may indicate a damaged or modified file.
chmod +x /etc/init.d/name
And verify the first line points to an actual shell interpreter:
head -1 /etc/init.d/name
Running the Service Directly
Sometimes the init script masks errors or interferes with troubleshooting. The most direct test is to run the daemon program itself.
For example, to test cron without the init script:
/usr/sbin/cron -f
This bypasses the init script entirely and shows you what the daemon is actually complaining about.
Before running a daemon directly, consult its documentation for flags to run in the foreground or debug mode. Stop the existing instance first if necessary to avoid attempting to launch a duplicate.
Finding Logs for Custom Services
Not all services use rsyslog. Some write directly to their own log files. Check:
- The service’s configuration file (often in
/etc/or/etc/service-name/) - Common locations like
/var/log/service-name.log - The init script itself, which may redirect output to a log file
Key Takeaway
Service failures fall into three categories: script problems, service problems, and dependency problems.
- Check status with
/etc/init.d/NAME status - Read logs at
/var/log/syslogand the topic-specific logs under/var/log/ - Verify dependencies are actually running
- Make one change at a time
- Verify after each change