Advanced Bash Scripting
awk
Option F
Learn how to use the -F option in awk
to customize the field separator when processing structured text. By default, awk
splits records on whitespace, but -F lets you define any character or regular expression as the delimiter.
usage: awk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...]
1. Default Field Splitting
By default, awk
treats any whitespace as the field separator:
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.8G 0 1.8G 0% /dev
tmpfs 366M 9.7M 356M 3% /run
...
When your data uses a different delimiter—such as commas, colons, or pipes—you must override this behavior.
2. Changing the Field Separator with -F
To use a colon (:
) as the separator:
$ awk -F ":" '{ print $1 }' sizeV1.txt
Filesystem
udev
tmpfs
/dev/sda1
...
Print the second field:
$ awk -F ":" '{ print $2 }' sizeV1.txt
Size
1.8G
366M
...
Combine fields 1 and 5:
$ awk -F ":" '{ print $1, $5 }' sizeV1.txt
Filesystem Use%
udev 0%
tmpfs 3%
...
You can also pipe data into awk
:
$ cat sizeV1.txt | awk -F ":" '{ print $2 }'
3. Why Quote or Escape the Separator
Certain characters (for example, |
, &
, *
, <
, >
) have special meanings to the shell.
Warning
Always quote or escape the field separator to prevent shell interpretation.
Incorrect:
awk -F | '{ print $1 }' employees.txt
This fails because |
is seen as a pipe.
Correct:
awk -F "|" '{ print $1 }' employees.txt
# or
awk -F \| '{ print $1 }' employees.txt
4. Common Field Separators
Separator | Shell Meaning | Example Use Case |
---|---|---|
: | None | /etc/passwd |
, | None | CSV export |
| | Pipe (must escape) | Database output |
\t | Tab (escape \t) | TSV files |
= | Assignment (escape) | Key-value configs |
5. Example: Processing Database Output
Here’s a Bash script (db.sh
) that runs a PostgreSQL query inside Docker, trimming each field:
#!/usr/bin/env bash
docker_run() {
docker exec -i employees_db psql -U postgres -d employees -tAQ <<EOF
SELECT
trim(id_employee::text),
trim(first_name),
trim(last_name),
trim(area),
trim(job_title),
trim(email),
trim(salary::text)
FROM "employee";
EOF
}
docker_run > employees.txt
exit 0
View the output:
$ cat employees.txt
1|Kriti|Shreshtha|Finance|Financial Analyst|[email protected]|60000
...
Extract first names (field 2):
$ awk -F "|" '{ print $2 }' employees.txt
Kriti
Rajasekar
Debbie
...
Note
See the GNU Awk User’s Guide for more awk
options.
Next Steps
Next, we’ll explore the -v option to declare variables within awk
:
usage: awk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...]
References
Watch Video
Watch video content