cs370:cs_370_-_introduction_unix_fundamentals
Table of Contents
Introduction, UNIX Fundamentals
Accessing Your UNIX Account
- HH 310 (Linux/Mac Lab), HH 305 (Mac Lab)
- Remote access via Internet or LAN
- UNIX servers (Linux)
- rockhopper.monmouth.edu
- plato.monmouth.edu
- UNIX lab machines (Linux)
- Terminal access via SSH
- PuTTY telnet/ssh client for Windows
- Graphical desktop access
- See MUCSremote for remote Linux desktop access
A Bit of UNIX History
(Also see http://www.levenez.com/unix/ for a constantly updated UNIX history banner.)
UNIX "standards": SysV and BSD
- AT&T's Bell Labs creates UNIX
- Evolves into “System V” UNIX
- Univ. of California-Berkeley takes AT&T's UNIX and develops BSD (Berkeley Standard Distribution)
- Today, proprietary versions of UNIX and Linux contain some of both UNIX “standards”
- Some differences in utilities
- lpr (BSD) and lp (SysV) for printing
- different options for utilities like ps
- Some UNIX variants include both kinds of utilities
Some UNIX Features
- Multi-user
- Allows multiple users to access a system at the same time
- Allows processes and peripherals to talk to each other, even on different machines
- pipes and sockets
- work between processes
- work between machines on a network
- standard output from a process on machine A is piped to standard input of a process on machine B
- Large number of standard command-line utilities
- for text editing and text processing, compiling, file processing
- the focus of much of this course because using these utilities together is the core of the UNIX philosophy
- even in these days of advanced or dumbed-down GUIs and GUI applications
- An “open” system
- allows programmers to easily access OS features via system calls
- A portable OS
- Relatively easy to compile and run on a wide variety of platforms because it's mostly written in C
- A contributing reason for so many proprietary variants of UNIX that were developed: Solaris, Irix, HP-UX, AIX, Tru64, Mac OS X…
- A contributing reason for free UNIX variants that have been available on so many different platforms: Intel, PowerPC, Alpha, MIPS, ARM (Android), Playstation …
- Everything is (or should be) a file, for example …
- System processes are listed as pseudo files in /proc
- System devices are accessed as special files in /dev
- Everything is (or should be) text.
- The core UNIX utilities were written to handle text streams allowing them to easily pass data between them.
UNIX File System Hierachy
- No drive letters, e.g. C:, D:, etc.
- File systems on hard drive partitions are mounted and appear as directories under the root “/” directory
- Also goes for remote file systems…
- … and removable media drives, e.g. floppy drives, CD-ROM, USB pen drives
$ ls / bin/ df.txt initrd/ media/ root/ sys/ var/ boot/ etc/ lib/ mnt/ sbin/ tftpboot/ denyhosts/ export/ linux-images/ opt/ selinux/ tmp/ dev/ home/ lost+found/ proc/ srv/ usr/ $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 522028 404048 117980 78% / none 1557628 48 1557580 1% /dev/shm /dev/sda2 10241088 8681048 1560040 85% /usr /dev/sda3 819232 639592 179640 79% /var /dev/sda5 21823584 20355644 1467940 94% /linux-images ... ... netapp:/home/cslab 286703264 129809216 156894048 46% /export/home
File System Organization
- varies between different variants of UNIX, but usually…
- Files and directories associated with applications and programs are usually under usr
- bin directories contain executable programs and scripts
- lib directories contain libraries that programs link with
- include directories contain program header files that correspond with the libraries in lib
- man directories contain manual pages used with the man command
- doc directories contain package documentation
- /usr/share/doc is the usual location for documentation
- share directories contain other application data such as fonts, pixmaps
- var directories contain application generated data such as database files, log files
- tmp directories contain temporary, transient directories and files
- /tmp (tmp in the root directory) is usually readable/writeable for every user and process
- etc directories contain application and system configuration files
- /etc (etc in the root directory) contains most system and application configuration files.
- the configuration files, in the UNIX tradition, are almost always plain text files
- …and many have names that end with .conf
$ find /etc -name '*.conf' | less
- Another set of bin, lib, include, man, doc, share, var, etc … directories is under /usr/local
- Nice picture and explanation at https://www.blackmoreops.com/2015/02/14/linux-file-system-hierarchy
Program organization
- A large enough program is split into the various directories mentioned above
- whereis will usually tell you how the program is split up
$ whereis geany geany: /usr/bin/geany /usr/include/geany /usr/share/geany /usr/share/man/man1/geany.1.gz
The UNIX Philosophy or Style
(See also http://en.wikipedia.org/wiki/Unix_philosophy)
(See also http://www.faqs.org/docs/artu/ch01s06.html)
Avoid large, monolithic programs
- A program should do one thing and do it well
- Small programs should be combined to accomplish more complex tasks
- Use “pipes,” which allow output of one process to be used as input (“piped”) to another process
- …without the need for intermediate temporary files
- Examples
# upside down cal cal | tac # number of processes that are associated with me ps aux | grep jchung | wc -l # ?????? wget -q -O - https://www.gutenberg.org/cache/epub/11231/pg11231.txt | sed 's/ /\n/g' | sed 's/[^A-Za-z0-9]*$//g' | sed 's/^[^A-Za-z0-9]*//g' | tr A-Z a-z | sort | uniq -c | sort -rn
Don't reinvent the wheel
- The UNIX approach to problem solving
- If you can solve the problem by combining multiple existing utilities using pipes, do it; otherwise
- …ask people on the network if they know how to solve it. If they do, great; otherwise
- …if you could solve the problem with the aid of some other hand-written utilities, write the utilities yourself and add them to the UNIX repertoire.
- Each utility you write should do one thing and do it well so that it can be reused to solve other problems.
- If more utilities won't do the trick, then write your own program to solve the problem.
Existing utilities
UNIX Shell Introduction
- The “command line”
- Relationship between the shell and “Terminals”
- The Terminal is not the shell.
- The shell runs in the Terminal to interpret and execute commands and display output.
- The shell can run invisibly outside of any Terminal.
Environment variables
- Environment variables are everywhere in UNIX.
- Used by running processes and in shells
- Run
env | less
# (orset | less
)- Look for value of the
SHELL
environment variableenv | grep SHELL
echo $SHELL
- In the shell, all variables are accessed by prepending the
$
.
Shell config files
- system-wide shell config = /etc/profile
- user-specific shell config
- ~/.profile (for ksh and bash)
- ~/.bashrc, ~/.bash_profile (for bash)
- ~ is shorthand for $HOME
- env | grep HOME or echo $HOME
- … so ~/.bash_profile for me is the same as /export/home/hawkdom2/jchung/.bash_profile
- ls ~ or ls $HOME are equivalent
- When you start an interactive shell, your shell reads its shell config file.
- Shell configs set env vars, define functions, aliases, etc. and run programs.
- The Shell prompt
- PS1 shell variable
Shell command history
- scrolling
- Use up and down arrow keys to scroll the command history.
- searching
- history command lists previously issued commands
- history | grep cat searches for cat in the command history
- shell history files
- (Bash) Command history written to ~/.bash_history when user exits
- Number of commands kept in command history and command history file set w/ HISTSIZE and HISTFILESIZE env vars, respectively
- Default value of HISTFILESIZE is 500
- HISTSIZE and HISTFILESIZE set in shell configuration file
- (Do in class) Change HISTSIZE and HISTFILESIZE to 5000 to increase command line history.
- Reissuing previous commands
- previous commands can be issued again with “!” directive
# Display your command history history ... 548 cat .bash_history 549 man bash 550 nano .bashrc 551 source .bashrc !549 # issues 'man bash' command again !man # reissues last command starting w/ 'man' !nan # reissues the 'nano .bashrc' command
Shell command and filename completion
- Type first few letters of a command or file name and hit Tab for completion.
Shell command aliases
- typically set in shell profile so they are available for every interactive shell session
- type alias to list aliases
- setting an alias
- alias commandalias='command(s) plus options'
- (Do in class) Add some useful aliases to bash shell profile (~/.bashrc).
Intro to Basic UNIX Utilities
- Before proceeding, see the collection setup notes.
- Create an
examples/cat
directory under your course directory (cs370
orse370
).
Getting help: man pages
- Searching for apropriate man pages
- man -k (or the equivalent
apropos
command)
man -k vim # Search for all man pages containing keyword 'vim' eview (1) - easy Vim, edit a file with Vim and setup for modeless editing evim (1) - easy Vim, edit a file with Vim and setup for modeless editing gvim (1) - Vi IMproved, a programmers text editor gvimdiff (1) - edit two, three or four versions of a file with Vim and show differences gvimtutor (1) - the Vim tutor rgvim (1) - Vi IMproved, a programmers text editor rvim (1) - Vi IMproved, a programmers text editor vim (1) - Vi IMproved, a programmers text editor vimdiff (1) - edit two, three or four versions of a file with Vim and show differences vimtutor (1) - the Vim tutor
- Navigating in man pages
- 1G - move to top of man page, G - move to bottom of man page
- u, b or <Page Up> to page up
- d, space or <Page Down> to page down
- q to quit
- Searching in man pages
- Type '/' then the word or phrase you are searching for.
- 'n' and 'p' to search for next and previous instances of search string, respectively
The cat utility
- cat (short for catenate) takes input from standard input or from a list of files and sends them to standard output
- Access contents of a file w/ cat
cat filename
- Creating a simple text file w/ cat
cat > fileFromCat.txt # The '>' means that keyboard input is 'redirected' to fileFromCat.txt The first line second line third line EOF # EOF is usually entered using Control-D cat fileFromCat.txt The first line second line third line
- Concatenating a series of files w/ cat
# output contents of four files to screen $ cat file1 file2 file3 file4 # output to a file, overwriting contents of allFilesTogether $ cat file1 file2 file3 file4 > allFilesTogether # output to a file, appending to contents of allFilesTogether $ cat file1 file2 file3 file4 >> allFilesTogether # using a shell wildcard (*), # cat the contents of all files whose names begin w/ 'file'; files are # concatenated in sorted order, i.e. file1 file2 file3 file4 $ cat file*
Processes and jobs
- Shell facilities for job control
- suspend, bring job to foreground, put job in background
- suspend with Control-Z
- Start the nano editor and then Control-Z
- Control-Z suspends nano
- The nano process is suspended
- Type the fg command (fg = foreground).
- nano is brought to the foreground
- Control-Z again
- Type the bg command (bg = background).
- The nano process resumes running in the background.
- Start man nano and Control-Z to suspend it.
- Type the jobs command to list the two jobs currently running.
- Resume nano with fg %1
- ps command and jobs
- The ps command displays process IDs (PIDs)
- The PID and jobs number are not the same
- ps aux or ps fax show all processes (Linux)
- ps options depend on SysV or BSD implementation of ps
- On SysV-type systems, ps -ef displays all processes
- On BSD-type systems, ps -aux displays all processes
- The top command shows top processes on system in “real time”
File management
- The
./..
shortcuts.
means current directory..
means parent directory
- mkdir -p dirname
“-p”
option of mkdir creates entire subdirectory tree- Example:
mkdir -p cs370/examples/cat
- rmdir vs. rm -r dirname
- rmdir is rarely used because it cannot remove non-empty directories
- The more commonly used rm -r dirname must be used with care.
- Example: rm -rf a_directory_name
- Again, use
rm -rf
with great care.
- simple shell file name globbing
- * - wildcard
ls abc*
: list files and directories that begin with “abc”ls *abc*
: list files and directories that contain “abc”
- [ ] - subset
ls [a-z]*
: list files and directories that begin with a lower case letter
- file managers
- mc: a 2-pane console file manager for power users
- various GUI file managers
Permissions and security
- ls -l to view permissions
- chmod to change permissions of files or directories
- directories must be executable to be accessible
- chown to change ownership
- Only useful to root (system admin)
- groups to see what groups you are in
- chgrp to change group ownership of files or directories
- setfacl, getfacl to use access control lists
- much more flexibility than standard
chmod/chown/chgrp
commands
- Certain GUI file managers provide frontends to
chmod/chown/chgrp/setfacl/getfacl
Printing
- lpr and lp to print from commandline
- lpq and lpstat to view queued print jobs
- lprm to remove your own print jobs from print queues
- enscript to convert text to Postscript for prettier output
Using the mouse
- If using a GUI desktop environment in UNIX, copying and pasting of text can be done between Terminals and other applications using common clipboard-like functionality.
- Right-clicking in a GUI Terminal may bring up a context menu which includes “copy” and “paste” options.
- Keyboard shortcuts to copy and paste in Terminals are usually Control-Shift-C and Control-Shift-V, respectively.
- UNIX desktops have an older copy-and-paste functionality that is commonly used with the mouse.
- Select text by swiping with the left mouse button …
- or double click a word to select it …
- or triple click a line to select the entire line.
- The selected text is automatically copied to a different clipboard area in memory.
- Paste text by placing the mouse cursor at the insertion point and clicking the middle mouse button.
Secure Shell (ssh)
- Ssh is the de facto method and protocol for logging into a console (terminal) session on a remote UNIX system.
- The most common way to use ssh is to run
ssh some_host_name # Example: ssh plato, ssh plato.monmouth.edu
- Your user account must exist on the remote host or you must specify another existing account when ssh'ing, e.g.,
ssh s1100841@plato # or ssh s1100841@plato.monmouth.edu (if off-campus)
- Set up no-password, key authentication for ssh
- This method uses public key encryption instead of passwords
- The set up of ssh public key authentication requires the following:
- Generate a public/private key pair if you don't already have one.
- command: ssh-keygen
- local files: ~/.ssh/id_rsa, ~/.ssh/id_rsa.pub
- Copy the public key contents to authorized_keys on the remote UNIX system that you want to ssh to.
- command: ssh-copy-id
- remote file: ~/.ssh/authorized_keys
- For maximum convenience, most users generate ssh keys without a passphrase.
- But the best practice for ssh key authentication is to generate the key pair using a strong passphrase, and then use ssh-agent to type in the passphrase for us when ssh'ing to remote systems.
- Remote file transfer programs that also use the ssh protocol are scp (secure copy), sftp (secure ftp) and rsync.
cs370/cs_370_-_introduction_unix_fundamentals.txt · Last modified: 2023/05/30 02:16 by jchung