Shell Script Debugging: Trace Failures, Preserve Exit Codes, and Fix Safely
A root-cause workflow for shell failures involving quoting, pipelines, unset variables, traps, temporary files, and partial work.
On this page
Key takeaways
- Start with the exact interpreter and shell mode; Bash-specific syntax may fail under sh.
- Use xtrace selectively, send diagnostics to stderr, and avoid exposing secrets in trace output.
- Check every pipeline stage and command substitution whose failure matters.
- Use traps and temporary directories to clean up without hiding the original exit status.
Reproduce with the intended shell
The shebang selects the interpreter when a script is executed directly. Running sh script.sh bypasses a Bash shebang and can expose incompatible syntax. Check the interpreter, environment variables, working directory, and input files before changing the script. A failure that appears only in CI often comes from a different shell, locale, PATH, permission, or current directory.
Run syntax checks before execution: bash -n script.sh checks Bash syntax without running commands. ShellCheck can identify many quoting, expansion, and portability hazards; treat findings as review prompts and preserve the script's intended behavior.
Trace without leaking data
set -x prints expanded commands, which is useful for control flow but can expose tokens and passwords. Enable it only around the problematic section and send output to a protected diagnostic stream. Prefer explicit messages to stderr that identify phase, safe identifiers, and exit status.
Quote expansions such as "$file" so spaces and wildcard characters remain part of one argument. Use arrays in Bash when building argument lists; a single string containing spaces is not a safe substitute for an argv array.
Preserve failures through pipelines
Many shells report the status of the last pipeline command. set -o pipefail makes a pipeline fail when a stage fails, but expected “no matches” from grep may need separate handling. Capture command substitution failures when they matter; assigning output and checking status should not depend on an implicit assumption about set -e, whose behavior has context-sensitive exceptions.
Prefer explicit control flow for critical operations:
if output=$(fetch_data); then
printf '%s\n' "$output"
else
printf 'fetch_data failed with status %s\n' "$?" >&2
exit 1
fi
Cleanup and atomic changes
Create temporary files in a private temporary directory and install a trap to remove them. When generating a configuration file, write to a temporary path, validate it, then rename it into place on the same filesystem. This avoids leaving a partially written target after failure.
Cleanup handlers must not accidentally replace the original command's exit status. Keep the failure code, perform best-effort cleanup, then exit with the captured status. Avoid broad || true unless failure is explicitly harmless and documented.
Fix root causes, not shell options
If a script fails because input is malformed, validate the input. If a race occurs, use atomic file creation or locking. If a pipeline reports success after a producer failure, handle pipeline status. If a child process survives shutdown, track and signal it. Changing permissions to 777, suppressing errors, or enabling every strict option without testing can make a symptom disappear while widening risk.
Build a minimal fixture, test success and failure cases, inspect the exit code and side effects, then run under the production interpreter. A reliable script is one whose failure behavior is as deliberate as its happy path.