Joe Chung
Monmouth U. Homepage

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Introduction, UNIX Fundamentals ======

----

===== Accessing Your UNIX Account =====

  * HH 310 (Linux/Mac Lab), HH 305 (Mac Lab)
  * Remote access via Internet or LAN
    * UNIX servers (Linux)
      * rockhopper.monmouth.edu
      * plato.monmouth.edu
    * UNIX lab machines (Linux)
    * Terminal access via SSH
      * [[http://www.chiark.greenend.org.uk/~sgtatham/putty/ | PuTTY]] telnet/ssh client for Windows
    * Graphical desktop access
      * See [[https://cssegit.monmouth.edu/cshelp/csseremote/-/wikis/MUCSremote | MUCSremote]] for remote Linux desktop access

----

===== A Bit of UNIX History =====

  * https://wikiless.tiekoetter.com/wiki/History_of_Unix
    * Also see http://www.levenez.com/unix/ for a constantly updated UNIX history banner.

==== UNIX "standards": SysV and BSD ====

  * AT&T's Bell Labs creates UNIX
    * Evolves into "System V" UNIX 
  * Univ. of California-Berkeley takes AT&T's UNIX and develops BSD (Berkeley Standard Distribution)
  * Today, proprietary versions of UNIX and Linux contain some of both UNIX "standards"
  * Some differences in utilities
    * ''lpr'' (BSD) and ''lp'' (SysV) for printing
    * different options for utilities like ''ps''
    * Some UNIX variants include both kinds of utilities

----

===== Some UNIX Features =====

  * Multi-user
    * Allows multiple users to access a system at the same time 

  * Allows processes and peripherals to talk to each other, even on different machines
  * pipes and sockets
    * work between processes
    * work between machines on a network
      * standard output from a process on machine A is piped to standard input of a process on machine B 
    * Also see https://wikiless.tiekoetter.com/wiki/Unix_philosophy

  * Large number of standard command-line utilities
    * for text editing and text processing, compiling, file processing
    * the focus of much of this course because using these utilities together is the core of the UNIX philosophy
      * even in these days of advanced or dumbed-down GUIs and GUI applications 

  * An "open" system
    * allows programmers to easily access OS features via system calls 

  * A portable OS
    * Relatively easy to compile and run on a wide variety of platforms because it's mostly written in C
      * A contributing reason for so many proprietary variants of UNIX that were developed: Solaris, Irix, HP-UX, AIX, Tru64, Mac OS X...
      * A contributing reason for free UNIX variants that have been available on so many different platforms: Intel, PowerPC, Alpha, MIPS, ARM (Android), Playstation ...

  * Everything is (or should be) a file, for example ...
    * System processes are listed as pseudo files in /proc
    * System devices are accessed as special files in /dev 

  * Everything is (or should be) text.
    * The core UNIX utilities were written to handle text streams allowing them to easily pass data between them.

----

===== UNIX File System Hierachy =====

  * No drive letters, e.g. C:, D:, etc.

  * File systems on hard drive partitions are mounted and appear as directories under the root "/" directory
    * Also goes for remote file systems...
    * ... and removable media drives, e.g. floppy drives, CD-ROM, USB pen drives

<code>
            $ ls /

            bin/        df.txt   initrd/        media/  root/     sys/       var/
            boot/       etc/     lib/           mnt/    sbin/     tftpboot/
            denyhosts/  export/  linux-images/  opt/    selinux/  tmp/
            dev/        home/    lost+found/    proc/   srv/      usr/



            $ df
            Filesystem           1K-blocks      Used Available Use% Mounted on
            /dev/sda1               522028    404048    117980  78% /
            none                   1557628        48   1557580   1% /dev/shm
            /dev/sda2             10241088   8681048   1560040  85% /usr
            /dev/sda3               819232    639592    179640  79% /var
            /dev/sda5             21823584  20355644   1467940  94% /linux-images
            ...
            ...
            netapp:/home/cslab   286703264 129809216 156894048  46% /export/home
</code>

----

===== File System Organization =====

  * varies between different variants of UNIX, but usually...
  * Files and directories associated with applications and programs are usually under //usr//
    * //bin// directories contain executable programs and scripts
    * //lib// directories contain libraries that programs link with
    * //include// directories contain program header files that correspond with the libraries in lib
    * //man// directories contain manual pages used with the man command
    * //doc// directories contain package documentation
      * /usr/share/doc is the usual location for documentation
    * //share// directories contain other application data such as fonts, pixmaps
    * //var// directories contain application generated data such as database files, log files
    * //tmp// directories contain temporary, transient directories and files
      * /tmp (//tmp// in the root directory) is usually readable/writeable for every user and process 
    * //etc// directories contain application and system configuration files
      * /etc (etc in the root directory) contains most system and application configuration files.
        * the configuration files, in the UNIX tradition, are almost always plain text files
          *...and many have names that end with .conf

                     $ find /etc -name '*.conf' | less

    * Another set of bin, lib, include, man, doc, share, var, etc ... directories is under /usr/local

    * Nice picture and explanation at https://www.blackmoreops.com/2015/02/14/linux-file-system-hierarchy

----

===== Program organization =====

  * A large enough program is split into the various directories mentioned above
    * //whereis// will usually tell you how the program is split up

           $ whereis geany
           geany: /usr/bin/geany /usr/include/geany /usr/share/geany /usr/share/man/man1/geany.1.gz

----

===== The UNIX Philosophy or Style =====

(See also https://wikiless.tiekoetter.com/wiki/Unix_philosophy)

(See also [[https://web.archive.org/web/20191018190805/http://www.faqs.org:80/docs/artu/ch01s06.html|http://www.faqs.org/docs/artu/ch01s06.html]])

==== Avoid large, monolithic programs ====

  * A program should do one thing and do it well
    * Small programs should be combined to accomplish more complex tasks
    * Use "pipes," which allow output of one process to be used as input ("piped") to another process
      * ...without the need for intermediate temporary files

  * Examples

<code>
# upside down cal
cal | tac
  
# number of processes that are associated with me
ps aux | grep jchung | wc -l
  
# ??????
wget -q -O - https://www.gutenberg.org/cache/epub/11231/pg11231.txt | sed 's/ /\n/g' | sed 's/[^A-Za-z0-9]*$//g' | sed 's/^[^A-Za-z0-9]*//g' | tr A-Z a-z | sort | uniq -c | sort -rn
</code>

==== Don't reinvent the wheel ====

  * The UNIX approach to problem solving
    * If you can solve the problem by combining multiple existing utilities using pipes, do it; otherwise
    * ...ask people on the network if they know how to solve it. If they do, great; otherwise
    * ...if you could solve the problem with the aid of some other hand-written utilities, write the utilities yourself and add them to the UNIX repertoire.
      * Each utility you write should do one thing and do it well so that it can be reused to solve other problems. 
    * If more utilities won't do the trick, then write your own program to solve the problem.

==== Existing utilities ====

  * [[https://wikiless.tiekoetter.com/wiki/List_of_GNU_Core_Utilities_commands|The GNU project's list of core Unix utilities]]
  * [[https://wikiless.tiekoetter.com/wiki/Category:Unix_SUS2008_utilities|IEEE Single UNIX Specification list of standard Unix utilities]]
  * [[https://rentes.github.io/unix/utilities/2015/07/27/moreutils-package/|moreutils]]
  * [[https://www.unixgame.io/unix50|The Unix Game: Solve puzzles using Unix pipes]]

----

===== UNIX Shell Introduction =====

  * The "command line" or "command line interpreter"

  * Relationship between the shell and "Terminals"
    * The Terminal is not the shell.
      * The shell runs in the Terminal to interpret and execute commands and display output.
      * The shell can run invisibly outside of any Terminal.


==== Environment variables ====

  * Environment variables are everywhere in UNIX.
    * Used by running processes and in shells
    * Run ''env | less'' # (or ''set | less'') 
      * Look for value of the ''SHELL'' environment variable
        * ''env | grep SHELL''
        * ''echo $SHELL''
          * In the shell, all variables are accessed by prepending the ''$''.

==== Shell config files ====

  * system-wide shell config: ''/etc/profile''
  * user-specific shell config
    * ''~/.profile'' (for ksh and bash)
    * ''~/.bashrc'', ''~/.bash_profile'' (for bash)
      * ''~'' (the tilde character) is shorthand for ''$HOME''
        * See output of ''env | grep HOME'' or ''echo $HOME''
          * ... so ''~/.bash_profile'' for me is the same as ''/export/home/hawkdom2/jchung/.bash_profile''
          * ''ls ~'' or ''ls $HOME'' are equivalent

  * When you start an interactive shell, your shell reads its shell config file.
  * Shell configs set env vars, define functions, aliases, etc. and run programs.
  * The Shell prompt
    * ''PS1'' shell variable

==== Shell command history ====

  * scrolling
    * Use up and down arrow keys to scroll the command history.

  * searching
    * ''history'' command lists previously issued commands
    * ''history | grep cat'' searches for ''cat'' in the command history

  * shell history files
    * (Bash) Command history written to ''~/.bash_history'' when user exits
      * Number of commands kept in command history and command history file set w/ ''HISTSIZE'' and ''HISTFILESIZE'' environment variables, respectively.
        * Default value of ''HISTFILESIZE'' is 500
        * ''HISTSIZE'' and ''HISTFILESIZE'' are set in shell configuration file
        * **(Do in class)** Change ''HISTSIZE'' and ''HISTFILESIZE'' to 5000 to increase command line history.
          * This requires editing your ''~/.bashrc'' bash config file.
            * Always include a comment with your config file edits.
          * **Note:** This is a participation exercise.

  * Reissuing previous commands
    * previous commands can be issued again with "!" directive

<code>
# Display your command history
history
   
...
   
548  cat .bash_history
549  man bash
550  nano .bashrc
551  source .bashrc
   
!549       # issues 'man bash' command again
!man       # reissues last command starting w/ 'man'
!nan       # reissues the 'nano .bashrc' command
</code>
==== Shell command and filename completion ====

  * Type first few letters of a command or file name and hit Tab for completion.

==== Shell command aliases ====

  * typically set in shell profile so they are available for every interactive shell session
    * type ''alias'' to list aliases
       * setting an alias
         * ''alias commandalias='command(s) plus options'''

  * **(Do in class)** Add at least 3 useful aliases to your bash shell profile (''~/.bashrc'').
    * Include a comment.
    * **Note:** This is a participation exercise.

----

===== Intro to Basic UNIX Utilities =====

  * **Before proceeding**, see the [[cs370/cs_370_assignments#collection_setup | collection setup]] notes.
    * Create an ''examples/cat'' directory under your course directory (''cs370'' or ''se370'').

==== Getting help: man pages ====

  * Searching for apropriate man pages
    *  Use ''man -k'' (or the equivalent ''apropos'' command) 

<code>
man -k vim   # Search for all man pages containing keyword 'vim'
            
eview (1)            - easy Vim, edit a file with Vim and setup for modeless editing
evim (1)             - easy Vim, edit a file with Vim and setup for modeless editing
gvim (1)             - Vi IMproved, a programmers text editor
gvimdiff (1)         - edit two, three or four versions of a file with Vim and show differences
gvimtutor (1)        - the Vim tutor
rgvim (1)            - Vi IMproved, a programmers text editor
rvim (1)             - Vi IMproved, a programmers text editor
vim (1)              - Vi IMproved, a programmers text editor
vimdiff (1)          - edit two, three or four versions of a file with Vim and show differences
vimtutor (1)         - the Vim tutor
</code>

  * Navigating in man pages
      * 1G - move to top of man page, G - move to bottom of man page 
      * u, b or <Page Up> to page up
      * d, space or <Page Down> to page down
      * q to quit

  * Searching in man pages
      * Type '/' then the word or phrase you are searching for.
        * 'n' and 'p' to search for next and previous instances of search string, respectively

==== The cat utility ====

  * cat (short for //cat//enate) takes input from standard input or from a list of files and sends them to standard output
    * Access contents of a file w/ cat

<code>
cat filename
</code>

    * Creating a simple text file w/ cat 

<code>
cat > fileFromCat.txt # The '>' means that keyboard input is 'redirected' to fileFromCat.txt
The first line
second line
third line
EOF          # EOF is usually entered using Control-D
   
cat fileFromCat.txt
   
The first line
second line
third line
</code>

    * Concatenating a series of files w/ cat

<code>
# output contents of four files to screen
$ cat file1 file2 file3 file4

# output to a file, overwriting contents of allFilesTogether   
$ cat file1 file2 file3 file4 > allFilesTogether

# output to a file, appending to contents of allFilesTogether
$ cat file1 file2 file3 file4 >> allFilesTogether

# using a shell wildcard (*),
# cat the contents of all files whose names begin w/ 'file'; files are
# concatenated in sorted order, i.e. file1 file2 file3 file4
$ cat file*
</code>

==== Processes and jobs ====

  * Shell facilities for job control
    * suspend, bring job to foreground, put job in background
    * suspend with Control-Z    
      * Start the ''nano'' editor and then Control-T, Control-Z
      * Control-T, Control-Z suspends //nano//
        * The ''nano'' process is suspended
      * Type the ''fg'' command (''fg'' = foreground).
        * ''nano'' is brought to the foreground
      * Control-Z again
      * Type the ''bg'' command (''bg'' = background).
        * The ''nano'' process resumes running in the background.
      * Start ''man nano'' and Control-Z to suspend it.
      * Type the ''jobs'' command to list the two jobs currently running.
      * Resume ''nano'' with fg %1

  * ''ps'' command and jobs
    * The ''ps'' command displays process IDs (PIDs)
      * The PID and jobs number are not the same
    * ''ps aux'' or ''ps fax'' show all processes (Linux)
      * ''ps'' options depend on SysV or BSD implementation of ''ps''
        * On SysV-type systems, ''ps -ef'' displays all processes
        * On BSD-type systems, ''ps -aux'' displays all processes

  * The ''top'' command shows top processes on system in "real time"

==== File management ====

  * The ''./..'' shortcuts
    * ''.'' means current directory
    * ''..'' means parent directory
  * ''mkdir -p dirname''
    * ''"-p"'' option of ''mkdir'' creates entire subdirectory tree
    * Example: ''mkdir -p cs370/examples/cat''
  * ''rmdir'' vs. ''rm -r dirname''
    * ''rmdir'' is rarely used because it cannot remove non-empty directories
    * The more commonly used ''rm -r dirname'' must be used **with care**.
    * Example: ''rm -rf'' a_directory_name
      * Again, use ''rm -rf'' with [[https://www.youtube.com/watch?v=sVhxJglkQwI|great care]].
  * simple shell file name globbing
    * ''*'' - wildcard
      * ''ls abc*'': list files and directories that begin with "abc"
      * ''ls *abc*'': list files and directories that contain "abc"
    * ''[ ]'' - subset
      * ''ls [a-z]*'': list files and directories that begin with a lower case letter
  * file managers
    * ''mc'': a 2-pane console file manager for power users
    * various GUI file managers
==== Permissions and security ====

  * ''ls -l'' to view permissions
  * ''chmod'' to change permissions of files or directories
    * directories must be executable to be accessible
  * ''chown'' to change ownership
    * Only useful to ''root'' (system admin)
  * ''groups'' to see what groups you are in
  * ''chgrp'' to change group ownership of files or directories
  * ''setfacl'', ''getfacl'' to use access control lists
    * much more flexibility than standard ''chmod/chown/chgrp'' commands
  * Certain GUI file managers provide frontends to ''chmod/chown/chgrp/setfacl/getfacl''

==== Printing ====

  * ''lpr'' and ''lp'' to print from commandline
  * ''lpq'' and ''lpstat'' to view queued print jobs
  * ''lprm'' to remove your own print jobs from print queues
  * ''enscript'' to convert text to Postscript for prettier output

==== Using the mouse ====

  * If using a GUI desktop environment in UNIX, copying and pasting of text can be done between Terminals and other applications using common clipboard-like functionality.
    * Right-clicking in a GUI Terminal may bring up a context menu which includes "copy" and "paste" options.
    * Keyboard shortcuts to copy and paste in Terminals are usually ''Control-Shift-C'' and ''Control-Shift-V'', respectively.
  * UNIX desktops have an older copy-and-paste functionality that is commonly used with the mouse.
    * Select text by swiping with the left mouse button ...
      * or double click a word to select it ...
      * or triple click a line to select the entire line.
      * The selected text is automatically copied to a different clipboard area in memory.
    * Paste text by placing the mouse cursor at the insertion point and clicking the middle mouse button.

==== Secure Shell (ssh) ====

  * Ssh is the de facto method and protocol for logging into a console (terminal) session on a remote UNIX system.
  * The most common way to use ssh is to run

  ssh some_host_name  # Example: ssh plato, ssh plato.monmouth.edu

  * Your user account must exist on the remote host or you must specify another existing account when ssh'ing, e.g.,

  ssh s1100841@plato  # or ssh s1100841@plato.monmouth.edu (if off-campus)

  * **(Do in class)** Set up no-password, key authentication for ssh
    * This method uses [[https://wikiless.tiekoetter.com/wiki/Public-key_cryptography | public key encryption]] instead of passwords
    * The [[https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2 | set up of ssh public key authentication]] requires the following:
      * Generate a public/private key pair if you don't already have one.
        * command: ''ssh-keygen''
        * local files: ''~/.ssh/id_rsa, ~/.ssh/id_rsa.pub''
      * Copy the public key contents to ''authorized_keys'' on the remote UNIX system that you want to ssh to.
        * command: ''ssh-copy-id''
        * remote file: ''~/.ssh/authorized_keys''
      * **Note:** This is a participation exercise.

  * For maximum convenience, most users generate ssh keys without a passphrase.
    * But the best practice for ssh key authentication is to generate the key pair using a strong passphrase, and then use [[https://wikiless.tiekoetter.com/wiki/Ssh-agent | ssh-agent]] to type in the passphrase for us when ssh'ing to remote systems.

  * Remote file transfer programs that also use the ssh protocol are ''scp'' (secure copy), ''sftp'' (secure ftp) and ''rsync''.

----