CA644, System Software
Table of Contents
CA644, System Software
Dr. Niall McMahon
2022-09-20
If you print these slides, think about using two pages per sheet although don't worry too much about it!
Dr. Niall McMahon
Drawing on previous work by:
Dr. Michael Scriney
Dr. Long Cheng
And sources credited in the references.
Autumn 2022.
for ...
>, >>, <
tr a-z A-Z < os.txt | sort
grep, cut, and regular expressions (regex)
sed, seq
$ for <i> in <list>; do
> echo $i # show the current item.
> done
Note that this is a command line example. After entering the first line, the prompt becomes >
until the loop is closed by the final line, done
.
$ for i in "abc" "de" "f"; do
> echo $i # Show the current item in the list defined.
> done
$ for i in {1..10}; do
> echo $i # Show the current item in the range defined.
> done
$ for ((exp1; exp2; exp3)); do
> statements
> done
The syntax is the same for while
/until
loops.
$ for ((i=1; i<=10; i++))
> do
> echo $i
> done
All of this can be entered on a single line, i.e.
$ for ((i=1; i<=10; i++)); do echo $i; done
There is no ;
after the do
.
Remembering that all the arguments passed to a script are stored in $@
and that the first line of a Bash script must be #!/bin/sh
.
print_all_args.sh
that prints out each argument passed on a separate line.
There are many commands whose sole purpose is to deal with files.
For example:
$ head <filename>
shows the first 10 lines from a file.
$ tail <filename>
shows the last 10.
$ cat <filename>
prints out the contents of a file.
Each of these operations can have one or more flags or options.
For example, cat -n
prints out line numbers.
Using --help
displays the help file associated with this command.
$ man cat
displays the man (manual) pages for the cat
command. Type q
to exit the manual.
You can use &&
as an And
operator, i.e.
$ command1 && command2
$ command2
will only execute if command1
finishes without an error.
||
is used for Or
operation, i.e.
$ command1 || command2
In this case, command2
will execute only if command1
fails.
The output and input of scripts can be redirected to and from files.
$ ls ~ >> list.txt
Lists the contents of your home directory and appends the output to a file called
list.txt
$ ls ~ > list.txt
This lists the contents of your home directory and overwrites the output of a file called
list.txt
A pipe takes the output from one program and passes it to another.
The syntax is:
$ command1 | command2
The standard output for command1
will be passed into command2
as an argument.
Imagine we put one of the loop examples into a script called loop_example.sh
. So one of the lines in the file contains a for
statement and we want to see this line. There are simpler ways to do this including using grep
directly on the file, but we could do the simple minded:
$ cat ./loop_example.sh | grep "for"
In this example, the output from cat ./loop_example.sh
- the content of the file - is passed in to the grep
command as its second argument. grep
looks for a substring and displays lines that contain the substring. (More about grep
later.)
The output will be:
for i in {1..10}; do
Shell cannot perform arithmetic directly; the following structures will not work:
$ echo 1 + 2
$ x = 5
$ y = $x + 1
$ echo $y
$ x = $x + $y
$ echo $z
Use the builtin command let
,
$ let x=1+2
$ echo $x
This also works,
$ let "x = 1 + 2"
$ echo $x
You can also use the command, expr
,
$ expr 1 + 2
$ x=`expr 1 + 2`
$ echo $x
Note that the inverted commas here are backticks. The are usually at the top left of the keyboard.
This also works,
$ x=$((1+2))
$ echo $x
As an exercise, try,
((x++))
echo $x
And repeat the operation. Syntax is critical. Watch the spaces!
Try this script:
#!/bin/bash
begin_time=`date +%s`
sleep 5s
finish_time=$(date +%s)
echo "t1: $begin_time"
echo "t2: $finish_time"
The syntax date +%[s|S]
formats the current date and time in seconds. This is the number of seconds that have passed since midnight on January 1st 1970, the Unix epoch.
if
, else
and elseif
statements.
&&
and ||
.
&&
is an AND statement.
||
is an OR statement.
$ let i=20
$ if [ $(($i % 2)) -eq 0 ] && [ $(($i % 10)) -eq 0 ]; then
> echo "Divisible by 2 and 10"
> fi
Divisible by 2 and 10
Note the use of the modulo operator %
.
What about 2 or 10?
From last week, there are different ways to initialise a string variable. Everything inside the quote marks in the following examples is a string:
$ str1="shell string"
$ str2=shell-string
$ str3='shell string $string'
$ str4=`command`
You can find the string length using,
$ echo ${#str}
.
To re-cap on last week's examples and extend them a little,
$ url="http://www.computing.dcu.ie/"
$ name="mcmahonn"
$ str1=$url$name
$ str2="$url $name"
$ str3=$url": "$name
$ str4="$url: $name"
$ str5="${name}'s website: ${url}/${name}/index.html"
This last line combines the defined variables into a full URL.
You can create substrings from strings, i.e. parts of the original string:
$ TEST_STRING='system software'
$ echo ${TEST_STRING: 0: 2}
sy
The syntax here is ${string: start position: number of characters}
.
Similarly,
$echo {TEST_STRING#*syst}
em software
The syntax here is ${string#*chars}
, where string
is the original string and chars
is the text after which you want to extract a substring. Finally,
$echo {TEST_STRING%em*}
syst
This prints the characters to the left of the substring em
. The syntax here is ${string%chars*}
.
If we have a situation that uses logic like this:
(This is pseudo-code, not actual code ...)
if some_value is not null
output is equal to some_value
else
output is equal to <default_value>
You can write this in a shell script as:
output=${some_value:-<default_value>}
#!/bin/sh
name=${1:-User}
echo "Hello $name, how are you?"
Note the syntax. It must be exactly like this.
Remember, $1 is the value of the first argument passed into the script. If the script is called checkname.sh
, you might type ./checkname.sh name
.
A function is a piece of code that does one small thing well. We can create shell functions directly at the terminal prompt or as part of a script. The structure is:
$ function name() {
> #Statements go here, i.e. what it does
> return <value>
> }
Define a function getsum()
, which can calculate the sum of the input arguments. At the command prompt, type,
$ function getsum()
This will bring up a new indented prompt. Enter, one line at a time, pressing Enter after each line,
> {
> local sum=0
> for n in $@
> do
> ((sum+=n))
> done
> return $sum
> }
Now, back at the command prompt, type,
$ getsum 1 2 3 4
$ echo $?
The second line prints out the return from the function getsum
.
You can define and call a function in exactly the same way both at the prompt or in a script. Exactly as above.
You can also enter a function as a single line expression, with ;
separating each line. In this example, the single line version is:
function getsum(){local sum=0; for n in $@; do ((sum+=n)); done; return $sum;}
grep
is one of the better known and useful commmands.
You can use it to find matching text in a file. It returns all the lines in the file that contain the matching text.
The syntax for usage is,
grep <flags> <search pattern> <file>
$ grep "Hello" myfile.txt
Will search the file myfile.txt for all lines that contain the string "Hello" exactly as written.
$ grep -i "Hello" myfile.txt
Ignore case during search, i.e. HELLO is the same as hello and all other combinations of upper and/or lowercase.
$ grep -v "Hello" myfile.txt
Search for lines that DO NOT contain "Hello", written exactly like this.
$ grep -vi "hello" myfile.txt
Search for lines that DO NOT contain "hello", case insensitive.
grep
can also match search patterns, or regular expressions (regex). Some regex patterns include,
'^h'
: all lines that start with the letter "h".
'thank you$'
: all lines that end with the words "thank you".
[0-9]
: all numbers.
[a-z]
: all lowercase letters. (uppercase '[A-Z]'
)
'[A-C]'
: match either A, B, or C.
'[a-zA-Z0-9]{4}'
: match 4 lower case letters, upper case letters, or numbers.
grep
regex searches can be linked using OR or AND operators, i.e. find text that matches this pattern OR that pattern, find text that matches this pattern AND that pattern.
OR:
$ grep -e 'pattern1|pattern2' filename
$ grep -e pattern1 -e pattern2 filename
AND:
$ grep -e 'pattern1.*pattern2' filename
$ grep -e 'pattern1' filename | grep -e 'pattern2'
The following script finds every line that contains "if" in the files in the current working directory,
#!/bin/sh
for i in $(ls -a); do
grep ^if $i
done
The command cat -n
will print out the contents of a file - in my example people.txt
- with line numbers:
cat -n ./people.txt
You can pipe this into grep:
cat -n ./people.txt | grep -E "[0-9]+\sI"
Our grep is now "[0-9]+\sI"
?[0-9]+
: find one one or more instances of a number.
\s
: that is Followed by a space.
I
: that is followed by an "I".
We have to use grep -E
to run this. -E
means Extended Regular Expression
We can use the cut
command to write selected bytes, characters, or fields from each line of a file to standard output.
If a text file people.txt
looks something like this:
Name,DOB,Address
Frank,01.01.1970,123 Fictional Street
Carly,01.01.1980,123 Imaginary Crescent
Then the command,
$ cut -f1,3 -d, ./people.txt
Will produce,
Name,Address
Frank,123 Fictional Street
Carly,123 Imaginary Crescent
cut
turns the output into columns based on the delimiter and returns the specified column(s), in this case, columns 1 and 3.
Similarly, if the delimeter was not ,
but :
, you would type:
$ cut -f1,3 -d: ./people.txt
sed
is the Stream EDitor. As data is being read, sed
carries out on-the-fly operations. The commands below all affect the output from sed
only, not the original files. You can modify the original files by using the -i
flag.
For example, sed
can replace strings in a file:
$ sed 's/oldstring/newstring' <filename>
Here the s/
stands for substitute. sed
will replace only the first occurence of the string in each line.
However, we can add additional flags, for example:
$ sed 's/oldstring/newstring/g' <filename>
Here, g
, or global, means that sed will substitute every occurence of oldstring with newstring, not only the first.
Similarly, sed
can be used to delete strings in a file:
The following removes lines 2 to 5 in the file.
$ sed '2,5d' <filename>
The following removes all lines from line 3 onwards.
$ sed '3,$d' <filename> # to the end line
sed
can be used to append to the stream, for example:
$ sed '1 a # This is a new comment' <filename>
Adds the line '# This is a new comment' after the first line. The following:
$ sed '$ a # This is a new comment' <filename>
Adds '# This is a new comment at the end of the file' after the last line.
seq
is a useful command; it generates a sequence of numbers:
$ seq <first> <increment> <last>
The output depends on the number of arguments passed in.
Write a script that accepts an integer as an argument. The script then counts from 0 up to and including that number, printing out the number and whether it is odd or even.
The output with an argument of 2 will look like:
0 ... even 1 ... odd 2 ... even
You must use a loop as part of your implementation.
The file users.txt
contains a list of users first and second names separated by a space:
Arati Lee
Samuel Chen
Lisa Cortez
Frank Murphy
Write a script which saves the surnames - second names - only into a file called second_names.txt
.
© Copyright 2022. Please contact Niall McMahon for more.