Joe Chung
Monmouth U. Homepage

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== UNIX Shells and Shell Scripts ======

----

===== UNIX shells =====

The [[https://unix.stackexchange.com/questions/14934/why-was-the-word-shell-used-to-descibe-a-command-line-interface | "shell"]] sits between you and the operating system

    * a user interface
      * reads input
      * translates commands into actions to be taken by the system
    * acts as a command interpreter
      * If commands are in a shell script, interprets the shell script.
        * The [[https://redmonk.com/sogrady/2024/03/08/language-rankings-1-24/|"Shell"]] language can still be considered an important language in some domains, despite being about as old as C.

See http://www.faqs.org/faqs/unix-faq/shell/shell-differences/ for a history of Unix shells and a shell feature comparison.

Also see https://wikiless.tiekoetter.com/wiki/Unix_shell, the Wikipedia article.


==== The venerable Bourne shell (sh) ====

    * I/O control features - often used for scripts
    * Not well suited for interactive use
          * Lacks modern bells & whistles, e.g. command history, completion, aliases, shell prompt setting
    * Other shells based on Bourne are better suited for interactive users
          * ksh, bash, [[https://scriptingosx.com/2019/06/moving-to-zsh/|zsh]] are sh-compatible and far easier to work with interactively
    * Default shell prompt is $ (can't be customized)


==== The C shell (csh) ====

    * Nicer for interactive use than sh
    * Uses C-like syntax for scripting
          * Not sh-compatible
    * I/O more awkward than Bourne shell
    * Introduced concept of job control
    * Introduced command history
    * Introduced use of ~ symbol to indicate a home directory (user's or others')
    * Default prompt is %
    * //tcsh// is a more modern implementation that introduces more bells & whistles such as completion and scrollable command history


==== The Korn Shell (ksh) ====

    * Developed by David Korn at AT&T
    * sh-compatible
    * Retained many csh features that were good for interactive work
    * Became part of System V UNIX standard
      * Licensed by AT&T;
        * used to have to pay to use it
        * public domain ksh (pdksh) is free and used in Linux and other free versions of UNIX

==== The Bourne Again Shell (bash) ====

    * Developed as a sh-compatible, free (open source ) shell for the [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)|GNU project]]
    * The currently dominant Linux/UNIX shell
    * Default shell in macOS (was)
    * Features from csh and ksh

==== Other shells ====

    * [[https://wikiless.tiekoetter.com/wiki/Z_shell|Z shell]] (zsh)
    * [[https://wikiless.tiekoetter.com/wiki/Friendly_interactive_shell|friendly interactive shell]] (fish)
    * [[https://wikiless.tiekoetter.com/wiki/Almquist_shell|Almquist shell]] (ash/dash)
==== Shell heritage ====

Based on the Bourne Shell (sh):

    * Korn (ksh)
    * Bourne-Again Shell (bash)
    * Z Shell (zsh) 

Based on the C Shell (csh):

    * T-C shell (tcsh)
    * ksh, bash, zsh also incorporate useful features of csh 

==== Built-in Shell Commands ====

  * The shells have a number of built-in commands:
    * Executed directly by the shell
    * Don't have to call another program to be run
    * Different shells have different built-ins
    * There is no executable file in a //bin// directory for built-in commands.
    * Examples: ''alias, echo, jobs, fg, bg, cd, history, pwd''

  * The ''which'' command returns the full path of a command, if the command is an external program.

  $ which echo
  /bin/echo                # (output of which)

  * The ''type'' built-in command tells you whether a command is a built-in command or a different type.

  $ type -a echo
  echo is a shell builtin
  echo is /bin/echo        # (Note that echo is both a shell-builtin as well as an external command.)

==== Basic Shell Job Control ====

(See [[cs370:cs_370_-_introduction_unix_fundamentals#processes_and_jobs|previous discussion]])

Additional commands:

  * ''kill'' command kills the specified background process
    * ''kill -9'' kills the specified background process quickly and messily; should use as last resort 
  * ''pkill'' and ''pkill -9'' commands kill jobs matching a text pattern
    * ''pkill -u jchung'' kills all jobs belonging to user ''jchung'' and will probably log ''jchung'' off.
      * Useful only for the ''root'' user and ''jchung'' himself.
  * ''nice'' and ''renice''
    * affect the priority of processes
      * ''nice'' allows you to start jobs with lower system priority.
      * ''renice'' allows you to change a job's "niceness" (priority) while it's running.
        * However, only ''root'' can make a running process "not nice", that is, give it a higher than normal priority.

----
----

===== Input / Output Redirection & Piping In UNIX Shells =====

  * Note previous discussion of the [[cs_370_-_introduction_unix_fundamentals?#the_cat_utility|cat]] command.

  * I/O redirection and piping allow...
    * Output redirection to a file
    * Input redirection from a file
    * Piping
          * Output of one command becomes the input of a subsequent command
          * Enables the "[[cs_370_-_introduction_unix_fundamentals?#the_unix_philosophy_or_style|UNIX philosophy]]"

==== Standard file descriptors ====

    * See [[https://wikiless.tiekoetter.com/wiki/Standard_streams|Standard streams]].

    * ''stdin'' - Standard input to the program, file descriptor 0
          * Normally from the keyboard, but can redirect from a file or command 
          * Java uses ''stdin'' through the ''java.util.Scanner'' class.
          * C++ uses ''stdin'' through the ''cin'' standard input stream statement. 

    * ''stdout'' - Standard output from the program, file descriptor 1
          * Normal output from a program 
          * Java uses ''stdout'' through the ''System.out.print (and println)'' method.
          * C++ uses ''stdout'' through the ''cout'' standard output stream statement.

    * ''stderr'' - Standard error output, file descriptor 2
          * Error messages or warnings of unusual situations 
          * Java uses ''stderr'' through the ''System.err.print (and println)'' method.
          * C++ uses ''stderr'' through the ''cerr'' standard output stream statement.

    * Stdin can be manual keyboard input or can be redirected from a file or other command. 
    * Both stdout & stderr normally sent to the terminal screen, but can ''redirect'' either or both to a file or pipe to another command 

==== Bourne Shell (sh)-compatible Input / Output Redirection Conventions ====

    * To redirect stdout to outfile 

            command > outfile

    * To append stdout to outfile 

            command >> outfile

    * To redirect input from infile 

            command < infile

    * To pipe standard output to another command 

            command1 | command2

    * To direct a command's stderr to file 

            command 2> file

    * To direct both stdout and stderr to file 

            command > file 2>&1
            
            or just
            
            command &> file

    * To append both stdout and stderr to file 

            command >> file 2>&1
            
            or just
            
            command &>> file

    * To pipe stdout and stderr to command2 

            command1 2>&1 | command2
            
            or just
            
            command |& command2

    * To redirect stdout and stderr to two separate files 
        * (the //"1"// in //"1>"// is optional)

            command 1> stdoutfile 2> stderrfile

    * To discard stderr 
      * (The /dev/null special file is a system "black hole" for output or errors.)

            command 2> /dev/null

      * To discard both stdout and stderr

            command > /dev/null 2>&1
            
            or just
            
            command &> /dev/null


===== Example =====

<code>
# In this example, we will redirect the standard output from a command
# to stdoutfile and the standard error from the same command to stderrfile.

# First, run the command normally to send stderr and stdout from ls to the screen:
ls -l /dev/sda1 /dev/foo

(This is stderr->) ls: cannot access /dev/foo: No such file or directory
(This is stdout->) brw-rw---- 1 root disk 8, 1 2010-08-21 16:32 /dev/sda1


# Next, run the command to redirect both stdout (1>) and stderr (2>) to files:
ls -l /dev/sda1 /dev/foo 1> stdoutfile 2> stderrfile

cat stdoutfile
brw-rw---- 1 root disk 8, 1 2010-08-21 16:32 /dev/sda1

cat stderrfile
ls: cannot access /dev/foo: No such file or directory

# Run first 'ls -l' command, but now discarding stderr:
ls -l /dev/sda1 /dev/foo 2> /dev/null
</code>

----

===== Shell Command Chaining =====

;   -    command separator

  # enter ~/cs370; display disk usage; return to previous dir
  cd ~/cs370; du -h; cd -
  
  # print 2 blank lines around output of fortune  
  echo; echo; fortune; echo; echo
 
&&  -  conditional chaining; run the following command only if previous command completes successfully
  
  # if string "jchung" is found in ~/.bashrc, cat ~/.bashrc  
  grep jchung ~/.bashrc && cat ~/.bashrc
 
<nowiki>||</nowiki>  -  conditional chaining; run the following command only if previous command did not complete successfully
  # if string "JCHUNG" not found in ~/.bashrc, print error message
  grep JCHUNG ~/.bashrc || echo "String not found."
 
( ) -  grouping - commands within parentheses are executed in a sub-shell (or child-shell) of the current shell

  (cd /; pwd)  # Execute commands in a sub-shell.
  /            # This is output from ''pwd'' in the sub-shell.
  
  pwd          # Execute ''pwd'' command in the current shell.
  /home/jchung # This is output from the current shell ''pwd'' command.

----

===== Quoting In The Shell =====

\  -  escape the following character (take it literally)
   
' '  - don't allow any special meaning to characters within single quotes
   
" "  -  allow variable and command substitution inside double quotes (does not disable $ and \ within the string)
   
<code>   
Command:   echo The \$SHELL env. var. contains \"$SHELL\".
Output:    The $SHELL env. var. contains "/bin/bash".

Command:   echo The '$SHELL' env. var. contains \"$SHELL\".
Output:    The $SHELL env. var. contains "/bin/bash".

Command:   touch \'  # Creates a file called '
Command:   rm \'     # Deletes the file called '
</code>
==== Shell command substitution ====

  * [[https://wikiless.tiekoetter.com/wiki/Command_substitution|Command substitution]] can be a useful technique, used for the "in-line" usage of a command's output.

  * In command substitution, the output of a command or pipeline of commands can be used to initialize variables
    * The output can also be used anywhere a command argument or variable might be used.

<code>
# Take the output of command and substitute it into the command line
`command` # Note the backward quotes (back-ticks).

# Run the uname command, which outputs Linux
uname
Linux

# Run the `uname` command in the shell; the shell
# attempts to run a nonexistent command called Linux.
`uname`
-bash: Linux: command not found

# Employing command substitution correctly, the output of 
# `uname` gets included in the output of echo.
echo "Hey, I'm using `uname`!"
Hey, I'm using Linux!
</code>

  * In ksh, bash and zsh, command substitution can also be done with //$(command)//
    * This makes command substitutions more natural to use as pseudo-variables.

<code>
# Use $() notation for command substitution.
echo "Hey, I'm using $(uname)!"
Hey, I'm using Linux!
  
# Edit a file that contains the date (MMDDYY format) embedded in its name;
# uses the 'date' command output; See the 'date' man page.
nano /tmp/activitylog-$(date +%m%d%y)

# Copy every .wav audio file on the system to /tmp, with (v)erbose cp output;
# uses output of the 'locate' command.
cp -v $(locate .wav) /tmp

# Play every .wav audio file on the system. (Control-C to stop)
aplay $(locate .wav)
</code>

----
----

===== Introduction to Shell Scripts =====

==== Before we begin, create ~/bin ====

  * **(Do in class)**

  * At the command line, create your own bin directory (''~/cs370/bin'' or ''~/se370/bin'').

  mkdir ~/cs370/bin   # or mkdir ~/se370/bin

  * Create a [[https://wikiless.tiekoetter.com/wiki/Symbolic_link|symbolic link]] (a kind of shortcut) called ''~/bin'' that points to your class directory's ''~/bin'' directory.

  ln -s  ~/cs370/bin  ~/bin

  * We want our shell scripts to go in ''~/bin'', our personal bin directory. 

  * Verify that ''~/bin'' is part of system's ''$PATH'' environment variable

  # See if ~/bin is in the $PATH environment variable (It should be).
  echo $PATH

==== Shell scripts ====

  * At their simplest, text files that contain lists of commands
    * The Bourne shell (sh) supported shell scripts and helped reduce the burden of repetitive typing of long commands for human operators.
  * Text files that contain the shell scripting programming language
    * Can vary in sophistication
    * Advances in csh, ksh and bash brought about a [[https://redmonk.com/sogrady/2024/03/08/language-rankings-1-24/|dynamic, interpreted, procedural programming language]] that is especially important in system programming, OS development, system administration, [[https://wikiless.tiekoetter.com/wiki/DevOps|DevOps]] and cloud ops.
    * Difference between shell scripting and other interpreted scripting languages like Perl, Python, Ruby
      * In shell scripting, your library of functions is the large collection of UNIX commands that you use together to perform tasks. 
  * Shell scripts are interpreted by the shell.
    * As Perl scripts are interpreted by the Perl interpreter, Python scripts are interpreted by the Python interpreter, etc.
    * The specific shell interpreter can be specified or just left to the default system shell to interpret.
      * The default system shell (''/bin/sh'') is typically bash or other shell that is Bourne shell compatible.
  * Shell scripts often are given a ''.sh'' file extension to distinguish them from other files.

  * Our first shell script: ''~/.bashrc''
    * Being a config file, it's not a true shell script, but everything in ''~/.bashrc'' is also found in shell scripts.
==== Simplest scripts ====

  * Contain commands that you would normally type at the command line
  * Not much different from aliases.
  * **(Do in class)** Create the ''~/bin/mycal.sh'' script.

<code>
# Change to your ~/bin directory
cd ~/bin

# Edit and save a new text file, mycal.sh
# See https://cssegit.monmouth.edu/jchung/csse370repo/-/blob/main/scripts/mycal 
# for the contents of mycal.sh.
nano mycal.sh

# Make mycal.sh executable (a runnable program)
chmod +x mycal.sh

# Run mycal.sh in the current directory (./)
./mycal.sh

# Run mycal.sh from your home directory. Since your ~/bin directory is in $PATH,
# you should be able to run your scripts from any directory.
cd
mycal.sh
</code>

==== Specify the shell interpreter ====

<code>
#!/bin/bash
#
# The above line means:
# /bin/bash will be used to interpret this shell script, if /bin/bash exists.
#
# Shell scripts should specify a shell interpreter on the first line.
#
# If the #!/bin/bash line is omitted from the script, either
#
#    1 - the script will run using the current interactive shell (bash)
# or
#    2 - the system shell (/bin/sh) will be used as the shell
#        command interpreter by default.
</code>

==== Shell scripts run in sub-shells ====

  * Any non-built-in command that you run in a shell will run in its own sub-shell.
    * See the note on //"() - grouping"// from [[cs_370_-_unix_shells_and_shell_scripting#shell_command_chaining|command chaining]].
  * **(Do in class)** This can present problems for some shell scripts, like the following:

<code>
#!/bin/bash

# mkcd.sh: Create a dir and change to it immediately.

# Usage: mkcd.sh <dirname>

mkdir -p $1 # $1 is the script's first command line argument 
cd $1
</code>

  * The //mkcd// script takes a directory name as a command line argument ($1), creates the directory and changes (cd) to the directory.
    * That's fine, except the //cd// part of the script will take place in a sub-shell. So, after running the //mkcd.sh// script, we are still in the same directory we started in.
    * The solution is to //source// the //mkcd.sh// script.
      * The //source// shell built-in command runs scripts inside the current shell, not in a sub-shell.

<code>
# Create ~/cs370/projects/1 directory and cd to it.
source mkcd.sh ~/cs370/projects/1
</code>

  * An alias added to ''~/.bashrc'' makes this command more natural to use.
    * Make sure you've created the ''mkcd.sh'' script in your ''~/bin'' directory.

<code>
alias mkcd='source ~/bin/mkcd.sh'
</code>

----
----