Notes Home

CA644, System Software
Table of Contents

Shell, Part 2

CA644, System Software

Dr. Niall McMahon

2022-09-20

If you print these slides, think about using two pages per sheet although don't worry too much about it!

Credits

Dr. Niall McMahon
Drawing on previous work by:
Dr. Michael Scriney
Dr. Long Cheng
And sources credited in the references.
Autumn 2022.

In Part 2

  • Loop, e.g. for ...
  • Redirection, e.g. >, >>, <
  • Pipes. e.g. tr a-z A-Z < os.txt | sort
  • Arithmetic operations, string operations and parameters
  • Functions
  • Filters, e.g. grep, cut, and regular expressions (regex)
  • Some other commands, e.g. sed, seq
  • Subshells

Loops

Python Style

Syntax

$ for <i> in <list>; do
> echo $i # show the current item.
> done

Note that this is a command line example. After entering the first line, the prompt becomes > until the loop is closed by the final line, done.

Examples

$ for i in "abc" "de" "f"; do
> echo $i # Show the current item in the list defined.
> done

$ for i in {1..10}; do
> echo $i # Show the current item in the range defined.
> done

C Style

Syntax

$ for ((exp1; exp2; exp3)); do
> statements
> done

The syntax is the same for while/until loops.

Example


$ for ((i=1; i<=10; i++))
> do
> echo $i
> done

All of this can be entered on a single line, i.e.

$ for ((i=1; i<=10; i++)); do echo $i; done

There is no ; after the do.

Exercise

Remembering that all the arguments passed to a script are stored in $@ and that the first line of a Bash script must be #!/bin/sh.

  • Write a script called print_all_args.sh that prints out each argument passed on a separate line.

File Operations

There are many commands whose sole purpose is to deal with files.

For example:

  • $ head <filename> shows the first 10 lines from a file.
  • $ tail <filename> shows the last 10.
  • $ cat <filename> prints out the contents of a file.

Help and Manuals

Each of these operations can have one or more flags or options.

For example, cat -n prints out line numbers.

Using --help displays the help file associated with this command.

$ man cat displays the man (manual) pages for the cat command. Type q to exit the manual.

Conditional Program Execution

You can use && as an And operator, i.e.

$ command1 && command2

$ command2 will only execute if command1 finishes without an error.

|| is used for Or operation, i.e.

$ command1 || command2

In this case, command2 will execute only if command1 fails.

Redirecting Output

The output and input of scripts can be redirected to and from files.

$ ls ~ >> list.txt
Lists the contents of your home directory and appends the output to a file called list.txt

$ ls ~ > list.txt
This lists the contents of your home directory and overwrites the output of a file called list.txt

Pipes

Overview

A pipe takes the output from one program and passes it to another.

The syntax is:

$ command1 | command2

The standard output for command1 will be passed into command2 as an argument.

Example

Imagine we put one of the loop examples into a script called loop_example.sh. So one of the lines in the file contains a for statement and we want to see this line. There are simpler ways to do this including using grep directly on the file, but we could do the simple minded:

$ cat ./loop_example.sh | grep "for"

In this example, the output from cat ./loop_example.sh - the content of the file - is passed in to the grep command as its second argument. grep looks for a substring and displays lines that contain the substring. (More about grep later.)

The output will be:

for i in {1..10}; do

Arithmetic Operations

Incorrect Syntax

Shell cannot perform arithmetic directly; the following structures will not work:

$ echo 1 + 2
$ x = 5
$ y = $x + 1
$ echo $y
$ x = $x + $y
$ echo $z

Correct Syntax

let

Use the builtin command let,

$ let x=1+2
$ echo $x

This also works,

$ let "x = 1 + 2"
$ echo $x

expr

You can also use the command, expr,

$ expr 1 + 2
$ x=`expr 1 + 2`
$ echo $x

Note that the inverted commas here are backticks. The are usually at the top left of the keyboard.

Double Brackets (( ))

This also works,

$ x=$((1+2))
$ echo $x

As an exercise, try,

((x++))
echo $x

And repeat the operation. Syntax is critical. Watch the spaces!

Working with Dates

Try this script:

#!/bin/bash
begin_time=`date +%s`
sleep 5s
finish_time=$(date +%s)
echo "t1: $begin_time"
echo "t2: $finish_time"

The syntax date +%[s|S] formats the current date and time in seconds. This is the number of seconds that have passed since midnight on January 1st 1970, the Unix epoch.

More About Conditionals

AND, OR

Overview

  • if, else and elseif statements.
  • We can combine these with logical operators using && and ||.
  • && is an AND statement.
  • || is an OR statement.

Example

$ let i=20
$ if [ $(($i % 2)) -eq 0 ] && [ $(($i % 10)) -eq 0 ]; then
> echo "Divisible by 2 and 10"
> fi
Divisible by 2 and 10

Note the use of the modulo operator %.

What about 2 or 10?

String Operations

Initialising String Variables, Re-Cap

From last week, there are different ways to initialise a string variable. Everything inside the quote marks in the following examples is a string:

$ str1="shell string"
$ str2=shell-string
$ str3='shell string $string'
$ str4=`command`

You can find the string length using,

$ echo ${#str}.

Using String Variables

To re-cap on last week's examples and extend them a little,

$ url="http://www.computing.dcu.ie/"
$ name="mcmahonn"
$ str1=$url$name
$ str2="$url $name"
$ str3=$url": "$name
$ str4="$url: $name"
$ str5="${name}'s website: ${url}/${name}/index.html"

This last line combines the defined variables into a full URL.

Substrings

You can create substrings from strings, i.e. parts of the original string:

$ TEST_STRING='system software'
$ echo ${TEST_STRING: 0: 2}
sy

The syntax here is ${string: start position: number of characters}.

Similarly,

$echo {TEST_STRING#*syst}
em software

The syntax here is ${string#*chars}, where string is the original string and charsis the text after which you want to extract a substring. Finally,

$echo {TEST_STRING%em*}
syst

This prints the characters to the left of the substring em. The syntax here is ${string%chars*} .

Shortened If/Else Statements

If we have a situation that uses logic like this:

(This is pseudo-code, not actual code ...)

if some_value is not null
output is equal to some_value
else
output is equal to <default_value>

You can write this in a shell script as:

output=${some_value:-<default_value>}

Example

#!/bin/sh
name=${1:-User}
echo "Hello $name, how are you?"

Note the syntax. It must be exactly like this.

Remember, $1 is the value of the first argument passed into the script. If the script is called checkname.sh, you might type ./checkname.sh name.

Functions

Overview

A function is a piece of code that does one small thing well. We can create shell functions directly at the terminal prompt or as part of a script. The structure is:

$ function name() {
> #Statements go here, i.e. what it does
> return <value>
> }

Example

Define a function getsum(), which can calculate the sum of the input arguments. At the command prompt, type,

$ function getsum()

This will bring up a new indented prompt. Enter, one line at a time, pressing Enter after each line,


> {
> local sum=0
> for n in $@
> do
> ((sum+=n))
> done
> return $sum
> }

Now, back at the command prompt, type,

$ getsum 1 2 3 4
$ echo $?

The second line prints out the return from the function getsum.

You can define and call a function in exactly the same way both at the prompt or in a script. Exactly as above.

You can also enter a function as a single line expression, with ; separating each line. In this example, the single line version is:

function getsum(){local sum=0; for n in $@; do ((sum+=n)); done; return $sum;}

grep

Overview

grep is one of the better known and useful commmands.

You can use it to find matching text in a file. It returns all the lines in the file that contain the matching text.

The syntax for usage is,

grep <flags> <search pattern> <file>

$ grep "Hello" myfile.txt
Will search the file myfile.txt for all lines that contain the string "Hello" exactly as written.

$ grep -i "Hello" myfile.txt
Ignore case during search, i.e. HELLO is the same as hello and all other combinations of upper and/or lowercase.

$ grep -v "Hello" myfile.txt
Search for lines that DO NOT contain "Hello", written exactly like this.

$ grep -vi "hello" myfile.txt
Search for lines that DO NOT contain "hello", case insensitive.

Regex

grep can also match search patterns, or regular expressions (regex). Some regex patterns include,

  • '^h': all lines that start with the letter "h".
  • 'thank you$': all lines that end with the words "thank you".
  • [0-9]: all numbers.
  • [a-z]: all lowercase letters. (uppercase '[A-Z]')
  • '[A-C]': match either A, B, or C.
  • '[a-zA-Z0-9]{4}': match 4 lower case letters, upper case letters, or numbers.

Regex, AND and OR

grep regex searches can be linked using OR or AND operators, i.e. find text that matches this pattern OR that pattern, find text that matches this pattern AND that pattern.

OR:

$ grep -e 'pattern1|pattern2' filename
$ grep -e pattern1 -e pattern2 filename

AND:

$ grep -e 'pattern1.*pattern2' filename
$ grep -e 'pattern1' filename | grep -e 'pattern2'

The following script finds every line that contains "if" in the files in the current working directory,


#!/bin/sh
for i in $(ls -a); do
grep ^if $i
done

cat

The command cat -n will print out the contents of a file - in my example people.txt - with line numbers:

cat -n ./people.txt

You can pipe this into grep:

cat -n ./people.txt | grep -E "[0-9]+\sI"

Our grep is now "[0-9]+\sI"
?[0-9]+: find one one or more instances of a number.
\s: that is Followed by a space.
I: that is followed by an "I".
We have to use grep -E to run this. -E means Extended Regular Expression

cut

We can use the cut command to write selected bytes, characters, or fields from each line of a file to standard output.

If a text file people.txt looks something like this:

Name,DOB,Address
Frank,01.01.1970,123 Fictional Street
Carly,01.01.1980,123 Imaginary Crescent

Then the command,

$ cut -f1,3 -d, ./people.txt

Will produce,

Name,Address
Frank,123 Fictional Street
Carly,123 Imaginary Crescent

cut turns the output into columns based on the delimiter and returns the specified column(s), in this case, columns 1 and 3.

Similarly, if the delimeter was not , but :, you would type:

$ cut -f1,3 -d: ./people.txt

sed, the Stream Editor

Substitute

sed is the Stream EDitor. As data is being read, sed carries out on-the-fly operations. The commands below all affect the output from sed only, not the original files. You can modify the original files by using the -i flag.

For example, sed can replace strings in a file:

$ sed 's/oldstring/newstring' <filename>

Here the s/ stands for substitute. sed will replace only the first occurence of the string in each line.

However, we can add additional flags, for example:

$ sed 's/oldstring/newstring/g' <filename>

Here, g, or global, means that sed will substitute every occurence of oldstring with newstring, not only the first.

Delete

Similarly, sed can be used to delete strings in a file:

The following removes lines 2 to 5 in the file.

$ sed '2,5d' <filename>

The following removes all lines from line 3 onwards.

$ sed '3,$d' <filename> # to the end line

Add

sed can be used to append to the stream, for example:

$ sed '1 a # This is a new comment' <filename>

Adds the line '# This is a new comment' after the first line. The following:

$ sed '$ a # This is a new comment' <filename>

Adds '# This is a new comment at the end of the file' after the last line.

seq

seq is a useful command; it generates a sequence of numbers:

$ seq <first> <increment> <last>

The output depends on the number of arguments passed in.

Exercises

Exercise 1

Write a script that accepts an integer as an argument. The script then counts from 0 up to and including that number, printing out the number and whether it is odd or even.

The output with an argument of 2 will look like:

            0 ... even
            1 ... odd
            2 ... even 
            

You must use a loop as part of your implementation.

Exercise 2

The file users.txt contains a list of users first and second names separated by a space:

Arati Lee
Samuel Chen
Lisa Cortez
Frank Murphy

Write a script which saves the surnames - second names - only into a file called second_names.txt.