An Introduction To Shell Programming

v1.0.0 / 01 may 03 / greg goebel / public domain

* The UN*X operating system provides a flexible set of simple tools to allow you to perform a wide variety of system-management, text-processing, and general-purpose tasks. These simple tools can be used in very powerful ways by tying them together programmatically, using using "shell scripts" or "shell programs".

The UN*X "shell" itself is a user-interface program that accepts commands from the user and executes them. It can also accept the same commands written as a list in a file, along with various other statements that the shell can interpret to provide input, output, decision-making, looping, variable storage, option specification, and so on. This file is a shell program.

Shell programs are, like any other programming language, useful for some things but not for others. They are excellent for system-management tasks but not for general-purpose programming of any sophistication. Shell programs, though generally simple to write, are also tricky to debug and slow in operation.

There are three versions of the UN*X shell: the original "Bourne shell (sh)", the "C shell (csh)" that was derived from it, and the "Korn shell (ksh)" that is in predominant use.

This document focuses on the Bourne shell. The C shell is more powerful but has various limitations, and while the Korn shell is clean and more powerful than the other two shells, it is a superset of the Bourne shell: anything that runs on a Bourne shell runs on a Korn shell. Since the Bourne shell's capabilities are probably more than most people require, there's no reason to elaborate much beyond them in an introductory document, and the rest of the discussion will assume use of the Bourne shell unless otherwise stated.

[1] GETTING STARTED

* The first thing to do in understanding shell programs is to understand the elementary system commands that can be used in them. A list of fundamental UN*X system commands follows:

  ls         # Give a simple listing of files.

  ll         # Give a listing of files with file details.

  cp         # Copy files.

  mv         # Move or rename files.

  rm         # Remove files.  

  rm -r      # Remove entire directory subtree.

  cd         # Change directories.

  pwd        # Print working directory.

  cat        # Lists a file or files sequentially.

  more       # Displays a file a screenfull at a time.

  pg         # Variant on "more".

  mkdir      # Make a directory.

  rmdir      # Remove a directory.

* The shell also allows files to be defined in terms of "wildcard characters" that define a range of files. The "*" wildcard character substitutes for any string of characters, so:

   rm *.txt

   rm book?.txt

   rm *book?.txt

* Another shell capability is "input and output redirection". The shell accepts input by default from what is called "standard input", and generates output by default to what is called "standard output". These are normally defined as the keyboard and display, respectively, or what is referred to as the "console" in UN*X terms.

However, you can "redirect" standard input or output to a file or another program if needed. Consider the "sort" command. This command sorts a list of words into alphabetic order, so if you enter:

   sort

   PORKY

   ELMER

   FOGHORN

   DAFFY

   WILE

   BUGS

   BUGS

   DAFFY

   ELMER

   FOGHORN

   PORKY

   WILE

   sort < names.txt

   sort < names.txt > output.txt

   sort < names.txt >> output.txt

   sort < names.txt | tee output.txt

By the way, "sort" has some handy additional options:

   sort -u    # Eliminate redundant lines in output.

   sort -r    # Sort in reverse order.

   sort -n    # Sort numbers. 

   sort +1    # Skip first field in sorting.

   ls xyzzy 2> /dev/null

* The shell allows you to execute multiple commands sequentially on one line by chaining them with a ";":

   rm *.txt ; ls

   sort < bigfile.txt > output.txt &

You instruct the shell that the file contains commands by marking it as "executable" with the "chmod" command. Each file under UN*X has a set of "permission" bits, listed by an "ll" as:

   rwxrwxrwx

You can use "chmod" to set these permissions by specifying them as an octal code. For example:

   chmod 644 myfile.txt

   chmod +x mypgm

For example, suppose you want to be able to inspect the contents of a set of archive files stored in the directory "/users/group/archives". You could create a file named "ckarc" and store the the following command string in it:

  ls /users/group/archives | pg

The following sections describe these features in a quick outline fashion. Please remember that you can't really make much effective use of most of the features until you've learned about all of them, so if you get confused just keep on going and then come back for a second pass.

BACK_TO_TOP

[2] SHELL VARIABLES

* The first useful command to know about in building shell programs is "echo", which allows you to perform output from your shell program:

   echo "This is a test!"

The shell allows you to store values in variables. All you have to do to declare a variable is assign a value to it:

   shvar="This is a test!"

   echo $shvar

   ll $lastdir

   $shvar=""

   allfiles=*

   echo $allfiles

Another subtlety is in modifying the values of shell variables. Suppose you have a file name in a shell variable named "myfile" and want to copy that file to another with the same name, but with "2" tacked on to the end. You might think to try:

   mv $myfile $myfile2

   mv $myfile ${myfile}2

If you want to call other shell programs from a shell program and have them use the same shell variables as the calling program, you have to "export" them as follows:

   shvar="This is a test!"

   export shvar

   echo "Calling program two."

   shpgm2

   echo "Done!"

   echo $shvar

BACK_TO_TOP

[3] COMMAND SUBSTITUTION

The next step is to consider shell command substitution. Like any programming language, the shell does exactly what you tell it to do and so you have to be very specific when you tell it to do something.

As an example, consider the "fgrep" command, which searches a file for a string. Suppose you want to search a file named "source.txt" for the string "Coyote". You could do this with:

   fgrep Coyote source.txt

   fgrep Wile E. Coyote source.txt

   fgrep "Wile E. Coyote" source.txt

For example, if you executed:

   echo "$shvar"

   echo '$shvar'

* Having considered "double-quoting" and "single-quoting", let's now consider "back-quoting". This is a little tricky to explain. As a useful tool, consider the "expr" command, which allows you to do simple math from the command line:

   expr 2 + 4

   expr 3 \* 7 

   echo $shcmd

   echo "$shcmd"

   echo '$shcmd'

   echo `$shcmd`

BACK_TO_TOP

[4] COMMAND-LINE ARGUMENTS

* In general, shell programs operate in a "batch" mode, that is, without interaction from the user, and so most of their parameters are obtained on the command line.

Each argument on the command line can be seen inside the shell program as a shell variable of the form "$1", "$2", "$3", and so on, with "$1" corresponding to the first argument, "$2" the second, "$3" the third, and so on.

There is also a "special" argument variable, "$0", that gives the name of the shell program itself. Other special variables include "$#", which gives the number of arguments supplied, and "$*", which gives a string with all the arguments supplied.

Since the argument variables are in the range "$1" to "$9", so what happens if you have more than 9 arguments? No problem, you can use the "shift" command to move the arguments down through the argument list. That is, when you execute "shift" then the second argument becomes "$1", the third argument becomes "$2", and so on, and if you do a "shift" again the third argument becomes "$1"; and so on. You can also add a count to cause a multiple shift:

   shift 3

BACK_TO_TOP

[5] DECISION-MAKING & LOOP CONSTRUCTS

* Shell programs can perform conditional tests on their arguments and variables and execute different commands based on the results. For example:

   if [ "$1" = "hyena" ]

   then

     echo "Sorry, hyenas not allowed."

     exit

   elif [ "$1" = "jackal" ]

   then

     echo "Jackals not welcome."

     exit

   else

     echo "Welcome to Bongo Congo."

   fi 

   echo "Do you have anything to declare?"

There are a wide variety of such test conditions:

   [ "$shvar" = "fox" ]     String comparison, true if match.

   [ "$shvar" != "fox" ]    String comparison, true if no match.

   [ "$shvar" = "" ]        True if null variable.

   [ "$shvar" != "" ]       True if not null variable.



   [ "$nval" -eq 0 ]        Integer test; true if equal to 0.

   [ "$nval" -ge 0 ]        Integer test; true if greater than or equal to 0.

   [ "$nval" -gt 0 ]        Integer test; true if greater than 0.

   [ "$nval" -le 0 ]        Integer test; true if less than or equal to 0.

   [ "$nval" -lt 0 ]        Integer test; true if less than to 0.

   [ "$nval" -ne 0 ]        Integer test; true if not equal to 0.



   [ -d tmp ]               True if "tmp" is a directory.

   [ -f tmp ]               True if "tmp" is an ordinary file.

   [ -r tmp ]               True if "tmp" can be read.

   [ -s tmp ]               True if "tmp" is nonzero length.

   [ -w tmp ]               True if "tmp" can be written.

   [ -x tmp ]               True if "tmp" is executable.

   case "$1" 

   in

     "gorilla")  echo "Sorry, gorillas not allowed."

                 exit;;

     "hyena")    echo "Hyenas not welcome."

                 exit;;

     *)          echo "Welcome to Bongo Congo.";;

   esac

* The fundamental loop construct in the shell is based on the "for" command. For example:

   for nvar in 1 2 3 4 5

   do

     echo $nvar

   done

   for file in *

   do

     echo $file

   done

   for file 

   do

     echo $file

   done

   for file

   do

     if [ "$file" = punchout ]

     then 

       break

     else

       echo $file

     fi

   done

   then

     :

   else

   n=10

   while [ "$n" -ne 0 ]

   do

     echo $n

     n=`expr $n - 1`

   done

   n=10

   until [ "$n" -eq 0 ]

   do

   ...

[6] OTHER SHELL FEATURES

* There are other useful features available for writing shell programs. For example, you can comment your programs by preceding the comments with a "#":

   # This is an example shell program.

   cat /users/group/grouplog.txt | pg    # Read group log file.

This will prevent prevent confusion if you find copies of the same file that don't have the same comments, or try to modify the program later. Shell programs can be obscure, even by the standards of programming languages, and it is useful to provide a few hints.

* You can read standard input into a shell program using the "read" command. For example:

   echo "What is your name?"

   read myname

   echo myname

* If you have a command too long to fit on one line, you can use the line continuation character "\" to put it on more than one line:

   echo "This is a test of \

         the line continuation character."

  . mycmds

* If you want to trace the execution of a shell program, you can use the "-x" option with the shell:

   sh -x mypgm *

* One last comment on shell programs before proceeding: What happens if you have a shell program that just performs, say:

  cd /users/coyote

The reason is that the shell creates a new shell, or "subshell", to run the shell program, and when the shell program is finished, the subshell vanishes -- along with any changes made in that subshell's environment. It is easier, at least in this simple case, to define a command alias in your UN*X "login" shell rather than struggle with the problem in shell programs.

BACK_TO_TOP

[7] USEFUL TOOLS

* Before we go on to practical shell programs, let's consider a few more useful tools.

The "paste" utility takes a list of text files and concatenates them on a line-by-line basis. For example:

   paste names.txt phone.txt > data.txt

* The "head" and "tail" utilities list the first 10 or last 10 lines in a file respectively. You can specify the number of lines to be listed if you like:

   head -5 source.txt      # List first 5 lines.

   tail -5 source.txt      # List last 5 lines.

   tail +5 source.txt      # List all lines after line 5.

   tr '[A-Z]' '[a-z]' < file1.txt > file2.txt

   tr '[a-z]' '[A-Z]' < file1.txt > file2.txt

   tr -d '*'

* The "uniq" utility removes duplicate consecutive lines from a file. It has the syntax:

   uniq source.txt output.txt

* The "wc (word count)" utility tallies up the characters, words, and lines of text in a text file. You can also invoke it with the following options:

    wc -c        # Character count only.

    wc -w        # Word count only.

    wc -l        # Line count only.

   find / -name findtest.txt -print

There are a wide variety of selection criteria. If you just want to print out the directories in a search from your current directory, you can do so with:

   find . -type d -print

One of the things that makes "find" extremely useful is that not only can you perform searches, you can perform an action when a search has a match, using the "-exec" option. For example, if you want to get the headers of all the files on a match into a single file, you could do so as:

   find . -name log.txt -exec head >> ./log \;

BACK_TO_TOP

[8] REGULAR EXPRESSIONS

* An advanced set of tools allows you to perform searches on text strings in files and, in some cases, manipulate the strings found. These tools are known as "grep", "sed", and "awk" and are based on the concept of a "regular expression", which is a scheme by which specific text patterns can be specified by a set of special or "magic" characters.

The simplest regular expression is just the string you are searching for. For example:

   grep Taz *.txt

But using the magic characters provides much more flexibility. For example:

   grep ^Taz *.txt

   grep Taz$ *.txt

Now suppose you want to be able to match both "Taz" and "taz". You can do that with:

  [Tt]az

   group_[abcdef]

   group_[a-f]

   set[0123456789]

   set[0-9]

   unit_[^xyz]

Other magic characters provide a wildcard capability. The "." character can substitute for any single character, while the "*" substitutes for zero or more repetitions of the preceding regular expression. For example:

   _*$

   test\.txt

BACK_TO_TOP

[9] GREP, SED, & AWK

* Now that we understand regular expressions, we can consider "grep", "sed", and "awk" in more detail.

The name "grep" stands for "general regular expression processor" and as noted it searchs a file for matches to a regular expression like "^Taz" or "_*$". It has a few useful options as well. For example:

   grep -v <regular_expression> <file_list>

   grep -n      # List line numbers of matches.

   grep -i      # Ignore case.

   grep -l      # Only list file names for a match.

* The name "sed" stands for "stream editor" and it provides, in general, a search-and-replace capability. Its syntax for this task is as follows:

   sed 's/<regular_expression>/<replacement_string>/[g]' source.txt

For example, to replace the string "flack" with "flak", you would use "sed" as follows:

   sed 's/flack/flak/g' source.txt > output.txt

   sed 's/bozo/d'

   sed -f sedcmds.txt source.txt > output.txt

   sed '/^Target/q' source.txt > output.txt

   sed '/^Target/ r newtext.txt' source.txt > output.txt

* Finally, "awk" is a full-blown text processing language that looks something like a mad cross between "grep" and "C". In operation, "awk" takes each line of input and performs text processing on it. It recognizes the current line as "$0", with each word in the line recognized as "$1", "$2", "$3", and so on.

This means that:

   awk '{ print $0,$0 }' source.txt

   awk '/Taz/  { taz++ }; END { print taz }' source.txt

You can do very simple or very complicated things with "awk" once you know how it works. Its syntax is much like that of "C", though it is much less finicky to deal with. Details of "awk" are discussed in another Vectorsite document.

BACK_TO_TOP

[10] SHELL PROGRAM EXAMPLES

* The most elementary use of shell programs is to reduce complicated command strings to simpler commands and to provide handy utilities.

For example, I can never remember the options for compiling an ANSI C program, so I store them in a script program named "compile":

   cc $1.c -Aa -o $1

   date +"date: %A, %d %B %Y %H%M %Z"

   date: Friday, 24 November 1995 1340 MST

   for file

   do

     mv $file `echo $file | tr "[A-Z]" "[a-z]"`

   done

BACK_TO_TOP

[11] QUICK REFERENCE

* This final section provides a fast lookup reference for the materials in this document. It is a collection of thumbnail examples and rules that will be cryptic if you haven't read through the text.

* Useful commands:

   cat                  # Lists a file or files sequentially.

   cd                   # Change directories.

   chmod +x             # Set execute permissions.

   chmod 666            # Set universal read-write permissions.

   cp                   # Copy files.

   expr 2 + 2           # Add 2 + 2.

   fgrep                # Search for string match.

   grep                 # Search for string pattern matches.

   grep -v              # Search for no match.

   grep -n              # List line numbers of matches.

   grep -i              # Ignore case.

   grep -l              # Only list file names for a match.

   head -5 source.txt   # List first 5 lines.

   ll                   # Give a listing of files with file details.

   ls                   # Give a simple listing of files.

   mkdir                # Make a directory.

   more                 # Displays a file a screenfull at a time.

   mv                   # Move or rename files.

   paste f1 f2          # Paste files by columns.

   pg                   # Variant on "more".

   pwd                  # Print working directory.

   rm                   # Remove files.  

   rm -r                # Remove entire directory subtree.

   rmdir                # Remove a directory.

   sed 's/txt/TXT/g'    # Scan and replace text.

   sed 's/txt/d'        # Scan and delete text.

   sed '/txt/q'         # Scan and then quit.

   sort                 # Sort input.

   sort +1              # Skip first field in sorting. 

   sort -n              # Sort numbers. 

   sort -r              # Sort in reverse order.

   sort -u              # Eliminate redundant lines in output.

   tail -5 source.txt   # List last 5 lines.

   tail +5 source.txt   # List all lines after line 5.

   tr '[A-Z]' '[a-z]'   # Translate to lowercase.

   tr '[a-z]' '[A-Z]'   # Translate to uppercase.

   tr -d '_'            # Delete underscores.

   uniq                 # Find unique lines. 

   wc                   # Word count (characters, words, lines).

   wc -w                # Word count only.

   wc -l                # Line count.

   shvar="Test 1"       # Initialize a shell variable.

   echo $shvar          # Display a shell variable.

   export shvar         # Allow subshells to use shell variable.

   mv $f ${f}2          # Append "2" to file name in shell variable.

   $1, $2, $3, ...      # Command-line arguments.

   $0                   # Shell-program name.

   $#                   # Number of arguments.

   $*                   # Complete argument list.

   shift 2              # Shift argument variables by 2.

   read v               # Read input into variable "v".

   . mycmds             # Execute commands in file.

   if [ "$1" = "red" ]

   then

     echo "Illegal code."

     exit

   elif [ "$1" = "blue" ]

   then

     echo "Illegal code."

     exit

   else

     echo "Access granted."

   fi 



   [ "$shvar" = "red" ]     String comparison, true if match.

   [ "$shvar" != "red" ]    String comparison, true if no match.

   [ "$shvar" = "" ]        True if null variable.

   [ "$shvar" != "" ]       True if not null variable.



   [ "$nval" -eq 0 ]        Integer test; true if equal to 0.

   [ "$nval" -ge 0 ]        Integer test; true if greater than or equal to 0.

   [ "$nval" -gt 0 ]        Integer test; true if greater than 0.

   [ "$nval" -le 0 ]        Integer test; true if less than or equal to 0.

   [ "$nval" -lt 0 ]        Integer test; true if less than to 0.

   [ "$nval" -ne 0 ]        Integer test; true if not equal to 0.



   [ -d tmp ]               True if "tmp" is a directory.

   [ -f tmp ]               True if "tmp" is an ordinary file.

   [ -r tmp ]               True if "tmp" can be read.

   [ -s tmp ]               True if "tmp" is nonzero length.

   [ -w tmp ]               True if "tmp" can be written.

   [ -x tmp ]               True if "tmp" is executable.

   case "$1" 

   in

     "red")      echo "Illegal code."

                 exit;;

     "blue")     echo "Illegal code."

                 exit;;

     *)          echo "Access granted.";;

   esac

   for nvar in 1 2 3 4 5

   do

     echo $nvar

   done 



   for file            # Cycle through command-line arguments.

   do

     echo $file

   done



   while [ "$n" != "Joe" ]     # Or:   until [ "$n" = "Joe" ]

   do

     echo "What's your name?"

     read n

     echo $n

   done

BACK_TO_TOP

[12] REVISION HISTORY

* This document was originally written during the 1990s, but I yanked it in 2001 as it didn't seem to be attracting much attention. In the spring of 2003 I retrieved it and put it back up as I realized it offered some value and it made no sense just to keep it archived, gathering dust.

Unfortunately, by that time I had lost track of its revision history. For want of anything better to do, I simply gave the resurrected document the initial revcode of "v1.0.0". I believe it is unlikely that any earlier versions of this document are available on the Internet, but since I had switched from a two-digit revcode format ("v1.0") to a three-digit format ("v1.0.0") in the interim, any earlier copies will have a two-digit revcode.

* Revision history:

   v1.0.0 / gvg / 01 may 03

ARM Executable ELF file translator
fromelf [options] input_file

Options:
       --help         display this help screen
       --vsn          display version information
       --output file  the output file. (defaults to stdout for -text format)
       --nodebug      do not put debug areas in the output image
       --nolinkview   do not put sections in the output image

Binary Output Formats:
       --bin          Plain Binary
       --m32          Motorola 32 bit Hex
       --i32          Intel 32 bit Hex
       --vhx          Byte Oriented Hex format

       --base addr    Optionally set base address for m32,i32

Output Formats Requiring Debug Information
       --fieldoffsets Assembly Language Description of Structures/Classes
       --expandarrays Arrays inside and outside structures are expanded

Other Output Formats:
       --elf         ELF
       --text        Text Information

                Flags for Text Information
                -v          verbose
                -a          print data addresses (For images built with debug)
                -c          disassemble code
                -d          print contents of data section
                -e          print exception tables
                -g          print debug tables
                -r          print relocation information
                -s          print symbol table
                -t          print string table
                -y          print dynamic segment contents
                -z          print code and data size information

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with the Invariant Sections being “Funding Free Software”, the Front-Cover Texts being (a) (see below), and with the Back-Cover Texts being (b) (see below). A copy of the license is included in the section entitled “GNU Free Documentation License”.

(a) The FSF's Front-Cover Text is:

A GNU Manual

(b) The FSF's Back-Cover Text is:

You have freedom to copy and modify this GNU Manual, like GNU software. Copies published by the Free Software Foundation raise funds for GNU development.

Introduction

This manual documents how to use the GNU compilers, as well as their features and incompatibilities, and how to report bugs. It corresponds to the compilers (GCC) version 4.5.0. The internals of the GNU compilers, including how to port them to new targets and some information about how to write front ends for new languages, are documented in a separate manual. See Introduction.

• G++ and GCC: You can compile C or C++ programs.
• Standards: Language standards supported by GCC.
• Invoking GCC: Command options supported by `gcc'.
• C Implementation: How GCC implements the ISO C specification.
• C Extensions: GNU extensions to the C language family.
• C++ Implementation: How GCC implements the ISO C++ specification.
• C++ Extensions: GNU extensions to the C++ language.
• Objective-C: GNU Objective-C runtime features.
• Compatibility: Binary Compatibility
• Gcov: gcov---a test coverage program.
• Trouble: If you have trouble using GCC.
• Bugs: How, why and where to report bugs.
• Service: How to find suppliers of support for GCC.
• Contributing: How to contribute to testing and developing GCC.
• Funding: How to help assure funding for free software.
• GNU Project: The GNU Project and GNU/Linux.
• Copying: GNU General Public License says how you can copy and share GCC.
• GNU Free Documentation License: How you can copy and share this manual.
• Contributors: People who have contributed to GCC.
• Option Index: Index to command line options.
• Keyword Index: Index of concepts and symbol names.