====== Introduction, UNIX Fundamentals ====== ---- ===== Accessing Your UNIX Account ===== * HH 310 (Linux/Mac Lab), HH 305 (Mac Lab) * Remote access via Internet or LAN * UNIX servers (Linux) * rockhopper.monmouth.edu * plato.monmouth.edu * UNIX lab machines (Linux) * Terminal access via SSH * [[http://www.chiark.greenend.org.uk/~sgtatham/putty/ | PuTTY]] telnet/ssh client for Windows * Graphical desktop access * See [[https://cssegit.monmouth.edu/cshelp/csseremote/-/wikis/MUCSremote | MUCSremote]] for remote Linux desktop access ---- ===== A Bit of UNIX History ===== (Also see http://www.levenez.com/unix/ for a constantly updated UNIX history banner.) ==== UNIX "standards": SysV and BSD ==== * AT&T's Bell Labs creates UNIX * Evolves into "System V" UNIX * Univ. of California-Berkeley takes AT&T's UNIX and develops BSD (Berkeley Standard Distribution) * Today, proprietary versions of UNIX and Linux contain some of both UNIX "standards" * Some differences in utilities * //lpr// (BSD) and //lp// (SysV) for printing * different options for utilities like //ps// * Some UNIX variants include both kinds of utilities ---- ===== Some UNIX Features ===== * Multi-user * Allows multiple users to access a system at the same time * Allows processes and peripherals to talk to each other, even on different machines * pipes and sockets * work between processes * work between machines on a network * standard output from a process on machine A is piped to standard input of a process on machine B * Also see http://en.wikipedia.org/wiki/Pipes_and_filters * Large number of standard command-line utilities * for text editing and text processing, compiling, file processing * the focus of much of this course because using these utilities together is the core of the UNIX philosophy * even in these days of advanced or dumbed-down GUIs and GUI applications * An "open" system * allows programmers to easily access OS features via system calls * A portable OS * Relatively easy to compile and run on a wide variety of platforms because it's mostly written in C * A contributing reason for so many proprietary variants of UNIX that were developed: Solaris, Irix, HP-UX, AIX, Tru64, Mac OS X... * A contributing reason for free UNIX variants that have been available on so many different platforms: Intel, PowerPC, Alpha, MIPS, ARM (Android), Playstation ... * Everything is (or should be) a file, for example ... * System processes are listed as pseudo files in /proc * System devices are accessed as special files in /dev * Everything is (or should be) text. * The core UNIX utilities were written to handle text streams allowing them to easily pass data between them. ---- ===== UNIX File System Hierachy ===== * No drive letters, e.g. C:, D:, etc. * File systems on hard drive partitions are mounted and appear as directories under the root "/" directory * Also goes for remote file systems... * ... and removable media drives, e.g. floppy drives, CD-ROM, USB pen drives $ ls / bin/ df.txt initrd/ media/ root/ sys/ var/ boot/ etc/ lib/ mnt/ sbin/ tftpboot/ denyhosts/ export/ linux-images/ opt/ selinux/ tmp/ dev/ home/ lost+found/ proc/ srv/ usr/ $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 522028 404048 117980 78% / none 1557628 48 1557580 1% /dev/shm /dev/sda2 10241088 8681048 1560040 85% /usr /dev/sda3 819232 639592 179640 79% /var /dev/sda5 21823584 20355644 1467940 94% /linux-images ... ... netapp:/home/cslab 286703264 129809216 156894048 46% /export/home ---- ===== File System Organization ===== * varies between different variants of UNIX, but usually... * Files and directories associated with applications and programs are usually under //usr// * //bin// directories contain executable programs and scripts * //lib// directories contain libraries that programs link with * //include// directories contain program header files that correspond with the libraries in lib * //man// directories contain manual pages used with the man command * //doc// directories contain package documentation * /usr/share/doc is the usual location for documentation * //share// directories contain other application data such as fonts, pixmaps * //var// directories contain application generated data such as database files, log files * //tmp// directories contain temporary, transient directories and files * /tmp (//tmp// in the root directory) is usually readable/writeable for every user and process * //etc// directories contain application and system configuration files * /etc (etc in the root directory) contains most system and application configuration files. * the configuration files, in the UNIX tradition, are almost always plain text files *...and many have names that end with .conf $ find /etc -name '*.conf' | less * Another set of bin, lib, include, man, doc, share, var, etc ... directories is under /usr/local * Nice picture and explanation at https://www.blackmoreops.com/2015/02/14/linux-file-system-hierarchy ---- ===== Program organization ===== * A large enough program is split into the various directories mentioned above * //whereis// will usually tell you how the program is split up $ whereis geany geany: /usr/bin/geany /usr/include/geany /usr/share/geany /usr/share/man/man1/geany.1.gz ---- ===== The UNIX Philosophy or Style ===== (See also http://en.wikipedia.org/wiki/Unix_philosophy) (See also [[https://web.archive.org/web/20191018190805/http://www.faqs.org:80/docs/artu/ch01s06.html|http://www.faqs.org/docs/artu/ch01s06.html]]) ==== Avoid large, monolithic programs ==== * A program should do one thing and do it well * Small programs should be combined to accomplish more complex tasks * Use "pipes," which allow output of one process to be used as input ("piped") to another process * ...without the need for intermediate temporary files * Examples # upside down cal cal | tac # number of processes that are associated with me ps aux | grep jchung | wc -l # ?????? wget -q -O - https://www.gutenberg.org/cache/epub/11231/pg11231.txt | sed 's/ /\n/g' | sed 's/[^A-Za-z0-9]*$//g' | sed 's/^[^A-Za-z0-9]*//g' | tr A-Z a-z | sort | uniq -c | sort -rn ==== Don't reinvent the wheel ==== * The UNIX approach to problem solving * If you can solve the problem by combining multiple existing utilities using pipes, do it; otherwise * ...ask people on the network if they know how to solve it. If they do, great; otherwise * ...if you could solve the problem with the aid of some other hand-written utilities, write the utilities yourself and add them to the UNIX repertoire. * Each utility you write should do one thing and do it well so that it can be reused to solve other problems. * If more utilities won't do the trick, then write your own program to solve the problem. ==== Existing utilities ==== * [[https://en.wikipedia.org/wiki/List_of_GNU_Core_Utilities_commands|The GNU project's list of core Unix utilities]] * [[https://en.wikipedia.org/wiki/Category:Unix_SUS2008_utilities|IEEE Single UNIX Specification list of standard Unix utilities]] * [[https://rentes.github.io/unix/utilities/2015/07/27/moreutils-package/|moreutils]] ---- ===== UNIX Shell Introduction ===== * The "command line" * Relationship between the shell and "Terminals" * The Terminal is not the shell. * The shell runs in the Terminal to interpret and execute commands and display output. * The shell can run invisibly outside of any Terminal. ==== Environment variables ==== * Environment variables are everywhere in UNIX. * Used by running processes and in shells * Run ''env | less'' # (or ''set | less'') * Look for value of the ''SHELL'' environment variable * ''env | grep SHELL'' * ''echo $SHELL'' * In the shell, all variables are accessed by prepending the ''$''. ==== Shell config files ==== * system-wide shell config = /etc/profile * user-specific shell config * ~/.profile (for ksh and bash) * ~/.bashrc, ~/.bash_profile (for bash) * ~ is shorthand for $HOME * //env | grep HOME// or //echo $HOME// * ... so ~/.bash_profile for me is the same as /export/home/hawkdom2/jchung/.bash_profile * //ls ~// or //ls $HOME// are equivalent * When you start an interactive shell, your shell reads its shell config file. * Shell configs set env vars, define functions, aliases, etc. and run programs. * The Shell prompt * //PS1// shell variable ==== Shell command history ==== * scrolling * Use up and down arrow keys to scroll the command history. * searching * //history// command lists previously issued commands * //history | grep// cat searches for //cat// in the command history * shell history files * (Bash) Command history written to ~/.bash_history when user exits * Number of commands kept in command history and command history file set w/ HISTSIZE and HISTFILESIZE env vars, respectively * Default value of HISTFILESIZE is 500 * HISTSIZE and HISTFILESIZE set in shell configuration file * **(Do in class)** Change HISTSIZE and HISTFILESIZE to 5000 to increase command line history. * Reissuing previous commands * previous commands can be issued again with "!" directive # Display your command history history ... 548 cat .bash_history 549 man bash 550 nano .bashrc 551 source .bashrc !549 # issues 'man bash' command again !man # reissues last command starting w/ 'man' !nan # reissues the 'nano .bashrc' command ==== Shell command and filename completion ==== * Type first few letters of a command or file name and hit Tab for completion. ==== Shell command aliases ==== * typically set in shell profile so they are available for every interactive shell session * type //alias// to list aliases * setting an alias * //alias commandalias='command(s) plus options'// * **(Do in class)** Add some useful aliases to bash shell profile (~/.bashrc). ---- ===== Intro to Basic UNIX Utilities ===== * **Before proceeding**, see the [[cs370/cs_370_assignments#collection_setup | collection setup]] notes. * Create an ''examples/cat'' directory under your course directory (''cs370'' or ''se370''). ==== Getting help: man pages ==== * Searching for apropriate man pages * man -k (or the equivalent ''apropos'' command) man -k vim # Search for all man pages containing keyword 'vim' eview (1) - easy Vim, edit a file with Vim and setup for modeless editing evim (1) - easy Vim, edit a file with Vim and setup for modeless editing gvim (1) - Vi IMproved, a programmers text editor gvimdiff (1) - edit two, three or four versions of a file with Vim and show differences gvimtutor (1) - the Vim tutor rgvim (1) - Vi IMproved, a programmers text editor rvim (1) - Vi IMproved, a programmers text editor vim (1) - Vi IMproved, a programmers text editor vimdiff (1) - edit two, three or four versions of a file with Vim and show differences vimtutor (1) - the Vim tutor * Navigating in man pages * 1G - move to top of man page, G - move to bottom of man page * u, b or to page up * d, space or to page down * q to quit * Searching in man pages * Type '/' then the word or phrase you are searching for. * 'n' and 'p' to search for next and previous instances of search string, respectively ==== The cat utility ==== * cat (short for //cat//enate) takes input from standard input or from a list of files and sends them to standard output * Access contents of a file w/ cat cat filename * Creating a simple text file w/ cat cat > fileFromCat.txt # The '>' means that keyboard input is 'redirected' to fileFromCat.txt The first line second line third line EOF # EOF is usually entered using Control-D cat fileFromCat.txt The first line second line third line * Concatenating a series of files w/ cat # output contents of four files to screen $ cat file1 file2 file3 file4 # output to a file, overwriting contents of allFilesTogether $ cat file1 file2 file3 file4 > allFilesTogether # output to a file, appending to contents of allFilesTogether $ cat file1 file2 file3 file4 >> allFilesTogether # using a shell wildcard (*), # cat the contents of all files whose names begin w/ 'file'; files are # concatenated in sorted order, i.e. file1 file2 file3 file4 $ cat file* ==== Processes and jobs ==== * Shell facilities for job control * suspend, bring job to foreground, put job in background * suspend with Control-Z * Start the //nano// editor and then Control-Z * Control-Z suspends //nano// * The //nano// process is suspended * Type the //fg// command (//fg// = foreground). * //nano// is brought to the foreground * Control-Z again * Type the //bg// command (//bg// = background). * The //nano// process resumes running in the background. * Start //man nano// and Control-Z to suspend it. * Type the //jobs// command to list the two jobs currently running. * Resume //nano// with fg %1 * //ps// command and jobs * The //ps// command displays process IDs (PIDs) * The PID and jobs number are not the same * //ps aux// or //ps fax// show all processes (Linux) * //ps// options depend on SysV or BSD implementation of //ps// * On SysV-type systems, //ps -ef// displays all processes * On BSD-type systems, //ps -aux// displays all processes * The //top// command shows top processes on system in "real time" ==== File management ==== * The ''./..'' shortcuts * ''.'' means current directory * ''..'' means parent directory * //mkdir -p dirname// * ''"-p"'' option of //mkdir// creates entire subdirectory tree * Example: ''mkdir -p cs370/examples/cat'' * //rmdir// vs. //rm -r dirname// * //rmdir// is rarely used because it cannot remove non-empty directories * The more commonly used //rm -r dirname// must be used **with care**. * Example: rm -rf a_directory_name * Again, use ''rm -rf'' with [[https://www.youtube.com/watch?v=sVhxJglkQwI|great care]]. * simple shell file name globbing * * - wildcard * ''ls abc*'': list files and directories that begin with "abc" * ''ls *abc*'': list files and directories that contain "abc" * [ ] - subset * ''ls [a-z]*'': list files and directories that begin with a lower case letter * file managers * mc: a 2-pane console file manager for power users * various GUI file managers ==== Permissions and security ==== * //ls -l// to view permissions * //chmod// to change permissions of files or directories * directories must be executable to be accessible * //chown// to change ownership * Only useful to //root// (system admin) * //groups// to see what groups you are in * //chgrp// to change group ownership of files or directories * //setfacl//, //getfacl// to use access control lists * much more flexibility than standard ''chmod/chown/chgrp'' commands * Certain GUI file managers provide frontends to ''chmod/chown/chgrp/setfacl/getfacl'' ==== Printing ==== * //lpr// and //lp// to print from commandline * //lpq// and //lpstat// to view queued print jobs * //lprm// to remove your own print jobs from print queues * //enscript// to convert text to Postscript for prettier output ==== Using the mouse ==== * If using a GUI desktop environment in UNIX, copying and pasting of text can be done between Terminals and other applications using common clipboard-like functionality. * Right-clicking in a GUI Terminal may bring up a context menu which includes "copy" and "paste" options. * Keyboard shortcuts to copy and paste in Terminals are usually //Control-Shift-C// and //Control-Shift-V//, respectively. * UNIX desktops have an older copy-and-paste functionality that is commonly used with the mouse. * Select text by swiping with the left mouse button ... * or double click a word to select it ... * or triple click a line to select the entire line. * The selected text is automatically copied to a different clipboard area in memory. * Paste text by placing the mouse cursor at the insertion point and clicking the middle mouse button. ==== Secure Shell (ssh) ==== * Ssh is the de facto method and protocol for logging into a console (terminal) session on a remote UNIX system. * The most common way to use ssh is to run ssh some_host_name # Example: ssh plato, ssh plato.monmouth.edu * Your user account must exist on the remote host or you must specify another existing account when ssh'ing, e.g., ssh s1100841@plato # or ssh s1100841@plato.monmouth.edu (if off-campus) * Set up no-password, key authentication for ssh * This method uses [[https://en.wikipedia.org/wiki/Public-key_cryptography | public key encryption]] instead of passwords * The [[https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2 | set up of ssh public key authentication]] requires the following: * Generate a public/private key pair if you don't already have one. * command: //ssh-keygen// * local files: //~/.ssh/id_rsa, ~/.ssh/id_rsa.pub// * Copy the public key contents to //authorized_keys// on the remote UNIX system that you want to ssh to. * command: //ssh-copy-id// * remote file: //~/.ssh/authorized_keys// * For maximum convenience, most users generate ssh keys without a passphrase. * But the best practice for ssh key authentication is to generate the key pair using a strong passphrase, and then use [[https://en.wikipedia.org/wiki/Ssh-agent | ssh-agent]] to type in the passphrase for us when ssh'ing to remote systems. * Remote file transfer programs that also use the ssh protocol are //scp// (secure copy), //sftp// (secure ftp) and //rsync//. ----