cs370:cs_370_-_introduction_unix_fundamentals
Table of Contents
Introduction, UNIX Fundamentals
Accessing Your UNIX Account
- HH 310 (Linux/Mac Lab), HH 305 (Mac Lab)
- Remote access via Internet or LAN
- UNIX servers (Linux)
- rockhopper.monmouth.edu
- plato.monmouth.edu
- UNIX lab machines (Linux)
- Terminal access via SSH
- PuTTY telnet/ssh client for Windows
- Graphical desktop access
- See MUCSremote for remote Linux desktop access
A Bit of UNIX History
-
- Also see http://www.levenez.com/unix/ for a constantly updated UNIX history banner.
UNIX "standards": SysV and BSD
- AT&T's Bell Labs creates UNIX
- Evolves into “System V” UNIX
- Univ. of California-Berkeley takes AT&T's UNIX and develops BSD (Berkeley Standard Distribution)
- Today, proprietary versions of UNIX and Linux contain some of both UNIX “standards”
- Some differences in utilities
lpr
(BSD) andlp
(SysV) for printing- different options for utilities like
ps
- Some UNIX variants include both kinds of utilities
Some UNIX Features
- Multi-user
- Allows multiple users to access a system at the same time
- Allows processes and peripherals to talk to each other, even on different machines
- pipes and sockets
- work between processes
- work between machines on a network
- standard output from a process on machine A is piped to standard input of a process on machine B
- Large number of standard command-line utilities
- for text editing and text processing, compiling, file processing
- the focus of much of this course because using these utilities together is the core of the UNIX philosophy
- even in these days of advanced or dumbed-down GUIs and GUI applications
- An “open” system
- allows programmers to easily access OS features via system calls
- A portable OS
- Relatively easy to compile and run on a wide variety of platforms because it's mostly written in C
- A contributing reason for so many proprietary variants of UNIX that were developed: Solaris, Irix, HP-UX, AIX, Tru64, Mac OS X…
- A contributing reason for free UNIX variants that have been available on so many different platforms: Intel, PowerPC, Alpha, MIPS, ARM (Android), Playstation …
- Everything is (or should be) a file, for example …
- System processes are listed as pseudo files in /proc
- System devices are accessed as special files in /dev
- Everything is (or should be) text.
- The core UNIX utilities were written to handle text streams allowing them to easily pass data between them.
UNIX File System Hierachy
- No drive letters, e.g. C:, D:, etc.
- File systems on hard drive partitions are mounted and appear as directories under the root “/” directory
- Also goes for remote file systems…
- … and removable media drives, e.g. floppy drives, CD-ROM, USB pen drives
$ ls / bin/ df.txt initrd/ media/ root/ sys/ var/ boot/ etc/ lib/ mnt/ sbin/ tftpboot/ denyhosts/ export/ linux-images/ opt/ selinux/ tmp/ dev/ home/ lost+found/ proc/ srv/ usr/ $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 522028 404048 117980 78% / none 1557628 48 1557580 1% /dev/shm /dev/sda2 10241088 8681048 1560040 85% /usr /dev/sda3 819232 639592 179640 79% /var /dev/sda5 21823584 20355644 1467940 94% /linux-images ... ... netapp:/home/cslab 286703264 129809216 156894048 46% /export/home
File System Organization
- varies between different variants of UNIX, but usually…
- Files and directories associated with applications and programs are usually under usr
- bin directories contain executable programs and scripts
- lib directories contain libraries that programs link with
- include directories contain program header files that correspond with the libraries in lib
- man directories contain manual pages used with the man command
- doc directories contain package documentation
- /usr/share/doc is the usual location for documentation
- share directories contain other application data such as fonts, pixmaps
- var directories contain application generated data such as database files, log files
- tmp directories contain temporary, transient directories and files
- /tmp (tmp in the root directory) is usually readable/writeable for every user and process
- etc directories contain application and system configuration files
- /etc (etc in the root directory) contains most system and application configuration files.
- the configuration files, in the UNIX tradition, are almost always plain text files
- …and many have names that end with .conf
$ find /etc -name '*.conf' | less
- Another set of bin, lib, include, man, doc, share, var, etc … directories is under /usr/local
- Nice picture and explanation at https://www.blackmoreops.com/2015/02/14/linux-file-system-hierarchy
Program organization
- A large enough program is split into the various directories mentioned above
- whereis will usually tell you how the program is split up
$ whereis geany geany: /usr/bin/geany /usr/include/geany /usr/share/geany /usr/share/man/man1/geany.1.gz
The UNIX Philosophy or Style
(See also https://wikiless.tiekoetter.com/wiki/Unix_philosophy)
(See also http://www.faqs.org/docs/artu/ch01s06.html)
Avoid large, monolithic programs
- A program should do one thing and do it well
- Small programs should be combined to accomplish more complex tasks
- Use “pipes,” which allow output of one process to be used as input (“piped”) to another process
- …without the need for intermediate temporary files
- Examples
# upside down cal cal | tac # number of processes that are associated with me ps aux | grep jchung | wc -l # ?????? wget -q -O - https://www.gutenberg.org/cache/epub/11231/pg11231.txt | sed 's/ /\n/g' | sed 's/[^A-Za-z0-9]*$//g' | sed 's/^[^A-Za-z0-9]*//g' | tr A-Z a-z | sort | uniq -c | sort -rn
Don't reinvent the wheel
- The UNIX approach to problem solving
- If you can solve the problem by combining multiple existing utilities using pipes, do it; otherwise
- …ask people on the network if they know how to solve it. If they do, great; otherwise
- …if you could solve the problem with the aid of some other hand-written utilities, write the utilities yourself and add them to the UNIX repertoire.
- Each utility you write should do one thing and do it well so that it can be reused to solve other problems.
- If more utilities won't do the trick, then write your own program to solve the problem.
Existing utilities
UNIX Shell Introduction
- The “command line” or “command line interpreter”
- Relationship between the shell and “Terminals”
- The Terminal is not the shell.
- The shell runs in the Terminal to interpret and execute commands and display output.
- The shell can run invisibly outside of any Terminal.
Environment variables
- Environment variables are everywhere in UNIX.
- Used by running processes and in shells
- Run
env | less
# (orset | less
)- Look for value of the
SHELL
environment variableenv | grep SHELL
echo $SHELL
- In the shell, all variables are accessed by prepending the
$
.
Shell config files
- system-wide shell config:
/etc/profile
- user-specific shell config
~/.profile
(for ksh and bash)~/.bashrc
,~/.bash_profile
(for bash)~
(the tilde character) is shorthand for$HOME
- See output of
env | grep HOME
orecho $HOME
- … so
~/.bash_profile
for me is the same as/export/home/hawkdom2/jchung/.bash_profile
ls ~
orls $HOME
are equivalent
- When you start an interactive shell, your shell reads its shell config file.
- Shell configs set env vars, define functions, aliases, etc. and run programs.
- The Shell prompt
PS1
shell variable
Shell command history
- scrolling
- Use up and down arrow keys to scroll the command history.
- searching
history
command lists previously issued commandshistory | grep cat
searches forcat
in the command history
- shell history files
- (Bash) Command history written to
~/.bash_history
when user exits- Number of commands kept in command history and command history file set w/
HISTSIZE
andHISTFILESIZE
environment variables, respectively.- Default value of
HISTFILESIZE
is 500 HISTSIZE
andHISTFILESIZE
are set in shell configuration file- (Do in class) Change
HISTSIZE
andHISTFILESIZE
to 5000 to increase command line history.- This requires editing your
~/.bashrc
bash config file.- Always include a comment with your config file edits.
- Note: This is a participation exercise.
- Reissuing previous commands
- previous commands can be issued again with “!” directive
# Display your command history history ... 548 cat .bash_history 549 man bash 550 nano .bashrc 551 source .bashrc !549 # issues 'man bash' command again !man # reissues last command starting w/ 'man' !nan # reissues the 'nano .bashrc' command
Shell command and filename completion
- Type first few letters of a command or file name and hit Tab for completion.
Shell command aliases
- typically set in shell profile so they are available for every interactive shell session
- type
alias
to list aliases- setting an alias
alias commandalias='command(s) plus options
'
- (Do in class) Add at least 3 useful aliases to your bash shell profile (
~/.bashrc
).- Include a comment.
- Note: This is a participation exercise.
Intro to Basic UNIX Utilities
- Before proceeding, see the collection setup notes.
- Create an
examples/cat
directory under your course directory (cs370
orse370
).
Getting help: man pages
- Searching for apropriate man pages
- Use
man -k
(or the equivalentapropos
command)
man -k vim # Search for all man pages containing keyword 'vim' eview (1) - easy Vim, edit a file with Vim and setup for modeless editing evim (1) - easy Vim, edit a file with Vim and setup for modeless editing gvim (1) - Vi IMproved, a programmers text editor gvimdiff (1) - edit two, three or four versions of a file with Vim and show differences gvimtutor (1) - the Vim tutor rgvim (1) - Vi IMproved, a programmers text editor rvim (1) - Vi IMproved, a programmers text editor vim (1) - Vi IMproved, a programmers text editor vimdiff (1) - edit two, three or four versions of a file with Vim and show differences vimtutor (1) - the Vim tutor
- Navigating in man pages
- 1G - move to top of man page, G - move to bottom of man page
- u, b or <Page Up> to page up
- d, space or <Page Down> to page down
- q to quit
- Searching in man pages
- Type '/' then the word or phrase you are searching for.
- 'n' and 'p' to search for next and previous instances of search string, respectively
The cat utility
- cat (short for catenate) takes input from standard input or from a list of files and sends them to standard output
- Access contents of a file w/ cat
cat filename
- Creating a simple text file w/ cat
cat > fileFromCat.txt # The '>' means that keyboard input is 'redirected' to fileFromCat.txt The first line second line third line EOF # EOF is usually entered using Control-D cat fileFromCat.txt The first line second line third line
- Concatenating a series of files w/ cat
# output contents of four files to screen $ cat file1 file2 file3 file4 # output to a file, overwriting contents of allFilesTogether $ cat file1 file2 file3 file4 > allFilesTogether # output to a file, appending to contents of allFilesTogether $ cat file1 file2 file3 file4 >> allFilesTogether # using a shell wildcard (*), # cat the contents of all files whose names begin w/ 'file'; files are # concatenated in sorted order, i.e. file1 file2 file3 file4 $ cat file*
Processes and jobs
- Shell facilities for job control
- suspend, bring job to foreground, put job in background
- suspend with Control-Z
- Start the
nano
editor and then Control-T, Control-Z - Control-T, Control-Z suspends nano
- The
nano
process is suspended
- Type the
fg
command (fg
= foreground).nano
is brought to the foreground
- Control-Z again
- Type the
bg
command (bg
= background).- The
nano
process resumes running in the background.
- Start
man nano
and Control-Z to suspend it. - Type the
jobs
command to list the two jobs currently running. - Resume
nano
with fg %1
ps
command and jobs- The
ps
command displays process IDs (PIDs)- The PID and jobs number are not the same
ps aux
orps fax
show all processes (Linux)ps
options depend on SysV or BSD implementation ofps
- On SysV-type systems,
ps -ef
displays all processes - On BSD-type systems,
ps -aux
displays all processes
- The
top
command shows top processes on system in “real time”
File management
- The
./..
shortcuts.
means current directory..
means parent directory
mkdir -p dirname
“-p”
option ofmkdir
creates entire subdirectory tree- Example:
mkdir -p cs370/examples/cat
rmdir
vs.rm -r dirname
rmdir
is rarely used because it cannot remove non-empty directories- The more commonly used
rm -r dirname
must be used with care. - Example:
rm -rf
a_directory_name- Again, use
rm -rf
with great care.
- simple shell file name globbing
*
- wildcardls abc*
: list files and directories that begin with “abc”ls *abc*
: list files and directories that contain “abc”
[ ]
- subsetls [a-z]*
: list files and directories that begin with a lower case letter
- file managers
mc
: a 2-pane console file manager for power users- various GUI file managers
Permissions and security
ls -l
to view permissionschmod
to change permissions of files or directories- directories must be executable to be accessible
chown
to change ownership- Only useful to
root
(system admin)
groups
to see what groups you are inchgrp
to change group ownership of files or directoriessetfacl
,getfacl
to use access control lists- much more flexibility than standard
chmod/chown/chgrp
commands
- Certain GUI file managers provide frontends to
chmod/chown/chgrp/setfacl/getfacl
Printing
lpr
andlp
to print from commandlinelpq
andlpstat
to view queued print jobslprm
to remove your own print jobs from print queuesenscript
to convert text to Postscript for prettier output
Using the mouse
- If using a GUI desktop environment in UNIX, copying and pasting of text can be done between Terminals and other applications using common clipboard-like functionality.
- Right-clicking in a GUI Terminal may bring up a context menu which includes “copy” and “paste” options.
- Keyboard shortcuts to copy and paste in Terminals are usually
Control-Shift-C
andControl-Shift-V
, respectively.
- UNIX desktops have an older copy-and-paste functionality that is commonly used with the mouse.
- Select text by swiping with the left mouse button …
- or double click a word to select it …
- or triple click a line to select the entire line.
- The selected text is automatically copied to a different clipboard area in memory.
- Paste text by placing the mouse cursor at the insertion point and clicking the middle mouse button.
Secure Shell (ssh)
- Ssh is the de facto method and protocol for logging into a console (terminal) session on a remote UNIX system.
- The most common way to use ssh is to run
ssh some_host_name # Example: ssh plato, ssh plato.monmouth.edu
- Your user account must exist on the remote host or you must specify another existing account when ssh'ing, e.g.,
ssh s1100841@plato # or ssh s1100841@plato.monmouth.edu (if off-campus)
- (Do in class) Set up no-password, key authentication for ssh
- This method uses public key encryption instead of passwords
- The set up of ssh public key authentication requires the following:
- Generate a public/private key pair if you don't already have one.
- command:
ssh-keygen
- local files:
~/.ssh/id_rsa, ~/.ssh/id_rsa.pub
- Copy the public key contents to
authorized_keys
on the remote UNIX system that you want to ssh to.- command:
ssh-copy-id
- remote file:
~/.ssh/authorized_keys
- Note: This is a participation exercise.
- For maximum convenience, most users generate ssh keys without a passphrase.
- But the best practice for ssh key authentication is to generate the key pair using a strong passphrase, and then use ssh-agent to type in the passphrase for us when ssh'ing to remote systems.
- Remote file transfer programs that also use the ssh protocol are
scp
(secure copy),sftp
(secure ftp) andrsync
.
cs370/cs_370_-_introduction_unix_fundamentals.txt · Last modified: 2024/05/22 21:26 by jchung