Bash Basics for Security Work

Bash is the shell behind most Linux terminals. It also runs on macOS, inside many containers and in Git Bash on Windows, which is what this article used. For a security student, it is the fastest way to ask a log file a question, check a folder for changes, or run the same check across many files. You do not need to program to use it. You need a terminal, a text editor and about an hour.

This article starts with what the shell does, then covers the habits that make shell work reliable: pipes, redirection, variables, loops and exit codes. After that you will tighten a script with set -euo pipefail. Then you will do two small exercises. First, you will count failed SSH logins per IP address in a sample log. Second, you will write a short script that records a SHA-256 hash for every file in a folder.

Everything here runs on files you create yourself. Nothing in this article touches another system.

#What Bash is

Bash stands for Bourne Again SHell. A shell reads the lines you type, runs other programs and prints what they produce. Bash is one of several shells, alongside sh, zsh and fish. It is the default on many Linux distributions, and most scripts you find online assume it.

Two terms come up in every section. A command is a program name plus its arguments, such as grep -c Failed sample.log. A script is a text file that lists commands, which Bash reads from top to bottom. A script lets you repeat work without retyping it, and it lets someone else read exactly what you did.

The reference for every behavior in this article is the GNU Bash Reference Manual. When a result surprises you, check the manual before you trust a guess.

#Why it matters for security work

Most security data is text. Authentication logs, web server access logs, DNS query logs, scanner reports and hash lists all arrive as lines. A large part of daily work is asking those lines questions. How many times did this address fail to log in? Which files changed since last week? Which account appeared for the first time today?

A spreadsheet or a Python program can answer each of those questions. Bash often answers them faster, in one line, and each step stays visible. That visibility matters more than speed for a beginner. When a result looks wrong, you run the first part of the pipeline alone and see what it produced.

Shell commands also show up in installation guides, container entry points and automation jobs. Reading them is part of the work, even when you would rather write Python.

#How it works, step by step

#Pipes send one program's output into the next

A pipe, written as |, connects the standard output of the left command to the standard input of the right command. Most command-line tools read lines, do one job and print lines. The shell joins them.

echo "one two three" | wc -w

Output:

3

wc -w counts words. It needs no file name because the pipe feeds it the text from echo. Small programs joined by pipes are the core habit of shell work.

#Redirection decides where output goes

Redirection changes where a command reads from or writes to. > writes standard output to a file and replaces the file's contents. >> appends to the end. 2> sends standard error, the channel where programs print their complaints, to a file. 2>&1 sends standard error to the same place as standard output.

echo first > notes.txt
echo second >> notes.txt
cat notes.txt

Output:

first
second

The shell opens and empties the file before the command runs. That is why this command destroys the data it was supposed to sort:

sort d.txt > d.txt

After that line, d.txt is empty. On a test file, the result was zero bytes. Write to a new file, check the result, then replace the old one.

Standard error matters in security work because errors often hold the answer. A missing file or a permission denied message is evidence. Redirecting errors with 2>/dev/null makes the output quieter, but a missing file then looks the same as a file with no matches. Keep errors visible while you are learning.

ls missing-file 2>/dev/null
echo "status=$?"

Output:

status=2

The error text went to the discard file, so the terminal shows only the status code.

#Variables and quoting

A variable stores a value. Assign it with no spaces around the equals sign. Read it by putting a dollar sign in front of the name.

target="10.0.0.5"
echo "Checking $target"

Wrap variables in double quotes. An unquoted variable is split at spaces, and the pieces can be expanded as wildcards. The difference shows up as soon as a value contains a space:

name="a b"
printf "[%s]\n" $name
printf "[%s]\n" "$name"

Output:

[a]
[b]
[a b]

The first printf received two words, because the shell split the value. The second received one. The same thing happens with a file called my report.txt when you type cat $file without quotes.

Scripts receive arguments as numbered variables. $1 is the first argument, $2 the second. The expression "${1:-.}" means "use the first argument if it is set and not empty, otherwise use the current directory". The hashing script uses it.

#Loops repeat a command for each item

A for loop runs its body once for each word in a list. A test in square brackets, such as [ "$n" -eq 2 ], returns 0 when the condition is true and 1 when it is false. An if statement uses that status to choose a branch.

for n in 1 2 3; do
  if [ "$n" -eq 2 ]; then
    echo "stop at $n"
    break
  fi
  echo "ok $n"
done

Output:

ok 1
stop at 2

break leaves the loop early, so the third item never runs.

#Exit codes tell you whether a command succeeded

Every command finishes with an exit status, a number from 0 to 255. Zero means success. Anything else means failure, and each program defines its own numbers. GNU grep returns 0 when it selects at least one line, 1 when nothing matches, and 2 on an error. Read the status with $? on the next line.

grep -q Failed sample.log
echo "match=$?"
grep -q nomatch sample.log
echo "nomatch=$?"

Output:

match=0
nomatch=1

The -q option suppresses output and only sets the status, which makes grep usable in an if without printing anything.

A pipeline has one status, and by default it is the status of its last command. The next section changes that.

#Strict mode: set -euo pipefail

Bash keeps running after errors unless you tell it not to. Three options make a script fail early. They are documented under the set builtin in the manual.

• set -e stops the script when a command fails. Commands in an if test, and commands in an && or || list other than the last one, do not stop it, because Bash expects those to fail sometimes.
• set -u treats an unset variable as an error, so a typo in a variable name stops the run instead of quietly becoming an empty string.
• set -o pipefail makes a pipeline fail if any command in it fails, not only the last one.

The default behavior first:

bash -c 'false | true; echo "status=$?"'

Output:

status=0

The false failed, but true came last and succeeded, so the pipeline reported success. With pipefail:

bash -c 'set -o pipefail; false | true; echo "status=$?"'

Output:

status=1

Now set -e with a failing command:

bash -c 'set -e; echo one; false; echo two'

Output:

one

The script stopped before two. The bash -c call itself exited with status 1.

set -u catches a missing argument:

bash -c 'set -u; echo "$1"' _

Output:

_: line 1: $1: unbound variable

The combined form, set -euo pipefail, is what most people mean by strict mode. It has a trade-off. A search that finds nothing is a normal result, but grep reports it with status 1, so under strict mode the script stops. Test the case before you rely on it:

bash -c 'set -euo pipefail; grep "Nope" sample.log | wc -l; echo "after"'

Output:

0

The script printed the count and then stopped without printing after. If "no matches" is a valid result for your script, add || true after the pipeline, or test the status yourself.

#Hands-on: count failed logins per IP

You will build a small log file, then count how often each source address appears on failed password lines. The log is synthetic. Its addresses come from the private 10.0.0.0/8 range, and the lines follow the format of OpenSSH messages on a lab host. Real authentication logs contain real people's addresses, so treat them as private data and read only logs you are allowed to read.

#Create the sample log

Create a folder for this lab, enter it, and write the file. The quoted 'EOF' stops Bash from expanding anything inside the text.

cat > sample.log <<'EOF'
Jan 10 10:00:01 lab sshd[1]: Failed password for root from 10.0.0.5 port 5000 ssh2
Jan 10 10:00:03 lab sshd[2]: Failed password for admin from 10.0.0.5 port 5001 ssh2
Jan 10 10:01:00 lab sshd[3]: Accepted publickey for alice from 10.0.0.9 port 5002 ssh2
Jan 10 10:02:00 lab sshd[4]: Failed password for bob from 10.0.0.7 port 5003 ssh2
EOF

Check the line count with wc -l sample.log. It reports 4.

#Count the failures in one pipeline

grep 'Failed password' sample.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr

Output:

      2 10.0.0.5
      1 10.0.0.7

Read the pipeline one stage at a time. Run the first two commands alone to see what each stage contributes.

1. grep 'Failed password' sample.log keeps the three failed lines and drops the Accepted line. Confirm the count with grep -c 'Failed password' sample.log, which prints 3.
2. awk '{print $(NF-3)}' prints one field from each line. NF holds the number of fields in the current line. Each line ends with from ADDRESS port PORT ssh2, so the address is the fourth field from the end, which is $(NF-3). Confirm it on the first line:

sed -n '1p' sample.log | awk '{print NF, $(NF-3)}'

Output:

14 10.0.0.5

The line has 14 fields, and field 11 is the address.

3. sort puts identical addresses next to each other. uniq -c counts adjacent duplicates and prints the count in front. Without the sort, uniq would count only neighboring repeats, so sorting always comes first.
4. sort -nr sorts the counts numerically, largest first.

The awk field number belongs to this log format. If you point the same command at a different log, check the field position first with the sed line above. A number that worked once is not proof that it fits the next file.

The same result can come from one awk program, because awk arrays count for you:

grep 'Failed password' sample.log | awk '{c[$(NF-3)]++} END {for (ip in c) print c[ip], ip}'

Output:

2 10.0.0.5
1 10.0.0.7

awk does not promise an order for for (ip in c), so sort the result if order matters. The GNU Awk User's Guide covers arrays and the END rule.

#Hands-on: hash every file in a folder

A hash is a fixed-length fingerprint of a file's contents. If you record the SHA-256 hash of every file in a folder today, you can run the same command next week and see which files changed. A hash does not prove who made a change or whether a file is safe. It only shows whether the bytes changed since you recorded them.

#Write the script

Create hashall.sh in the same folder, using the same quoted heredoc pattern.

cat > hashall.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
for f in "${1:-.}"/*; do [ -f "$f" ] && sha256sum "$f"; done
EOF

Read the script line by line:

• The first line is the shebang. It tells the system to run the file with Bash, found through env.
• set -euo pipefail turns on strict mode, as covered earlier.
• "${1:-.}"/* is the folder you pass in, or the current folder when you pass nothing. The * is a glob. The shell replaces it with the names in that folder, sorted. Names that start with a dot are skipped by default, as the filename expansion section of the manual describes.
• [ -f "$f" ] && sha256sum "$f" hashes only regular files. sha256sum on a folder would print an error, so the test filters folders out.

Make the script executable and run it on the current folder:

chmod +x hashall.sh
./hashall.sh .

chmod +x adds the execute permission. The ./ tells the shell to run the file in this folder, because the current folder is not in PATH by default.

Output in a folder that held only hashall.sh and sample.log:

288797e3d2baa72b086c5fca9821282d06726cb44272647d41128ababf255c94 *./hashall.sh
2c086498f19a2a8ca7fe112417ad8113449bf640bceabd4795808bd00a996893 *./sample.log

Git Bash prints an asterisk before each name, which marks binary mode in GNU coreutils. Other systems may show a space instead. The hash itself stays the same. If your values differ, check that your files match the text byte for byte, including line endings.

Confirm one value by hand:

sha256sum sample.log

It prints the same hash for sample.log that the script printed.

#Watch a hash change

Append one line to the log, then hash it again.

echo "x" >> sample.log
sha256sum sample.log

In the test run the hash became 3084d4c5a7659a2f3636b6079636b081b7ab1b6fca6a5e691c3afdd160959687, where it had been 2c086498f19a2a8ca7fe112417ad8113449bf640bceabd4795808bd00a996893. The file grew by two bytes and the fingerprint changed with it. Restore the original file from your own copy before you continue.

#Common mistakes

Unquoted variables and paths. cat $file breaks on names with spaces. Quote by default, and remove the quotes only when you know the value cannot contain spaces or wildcards.

Overwriting an input file with >. sort d.txt > d.txt empties the file before sort reads it. Write to a new file, check the output, then move it into place.

Trusting a field number. $(NF-3) worked on the sample. Test the field position on a few real lines before you build a report on it.

Strict mode stopping on a valid "no match". As shown above, grep returning 1 ends a pipeline under pipefail. Decide whether no matches is an error for your script, then handle it on purpose.

A loop that reports the wrong status. The hashall.sh above hashes every regular file. When the last name in the folder is a subfolder, though, the script prints every hash and then exits with status 1. The status came from the final [ -f ] test, which failed. The explicit if form does the same job and exits with 0:

#!/usr/bin/env bash
set -euo pipefail
for f in "${1:-.}"/*; do
  if [ -f "$f" ]; then sha256sum "$f"; fi
done

In a test folder with a subfolder as the last entry, this version printed each file hash and exited with status 0. The && version also exits with status 1 on an empty folder, so the if form is the better default.

Running scripts you did not write. A script can delete or send files as easily as it hashes them. Read the whole file first, check every command it calls, and run it on a copy of your folder.

#How to go further

Read the sections of the GNU Bash Reference Manual on pipelines, redirections, shell parameter expansion and the set builtin. Those cover most of what you saw here. Use the exit status material as a reference when a script behaves in a way you did not expect.

Read the GNU grep manual next. Try grep -i, grep -v and grep -E against your sample log and note how each option changes the exit status.

The GNU Awk User's Guide covers fields, patterns and arrays. The one-line counter above is a small example of what awk can do. Once it feels natural, write a second pattern that counts accepted logins per user.

Keep your scripts short. A ten-line script you can explain line by line is more useful than a hundred-line script you copied and cannot change. When a script grows past a screen, split it into functions, or consider a language with better data structures for the job.

#Sources

• GNU Bash Reference Manual: https://www.gnu.org/software/bash/manual/bash.html
• GNU grep manual: https://www.gnu.org/software/grep/manual/grep.html
• GNU Awk User's Guide: https://www.gnu.org/software/gawk/manual/gawk.html