====== Some General SysAdmin Principles ====== ---- ===== Documentation ===== * Document your work. * Keep a journal. * Use at least a plain text file. * Date each entry. * If you have a boss that you send your journal to, spellcheck your journal. * The journal isn't just for your boss, it's for you. * It's your own FAQ. * It's your own collection of HOWTOs to help you and others reproduce work in the future on other systems. * When you make an entry in your journal, ask yourself, "Is this enough information to allow me and maybe others to reproduce this work that I just spent hours on? * It can get huge over time, which is why there may be a need for a better system of self-documentation than a flat text file. * Possibly something blog-like where you can add tags to each journal entry. * The journal has to be backed up and accessible from anywhere, at your fingertips. * This is why some admins keep logs in notebooks (paper notebooks), though this has several drawbacks. * If your journal is on a company server or your office workstation, and the network is down, and you're at home 30 miles away, and you need some vital information that's in the journal...now what? * Use comments in /etc/config files to document your changes. * Sign and date your comments for your future reference and for other admins. * In general, be well-organized. * whether that's the directory structure under your home directory or your written journal * Use technology in documentation. * Set up a web page or FAQ. * Use //video// or //digital photography//. * Sometimes, a picture is literally worth a thousand words. ---- ===== The hard stuff (security and backups) ===== * Take care of the hard stuff first. * The worst feelings for a SysAdmin (makes you fear getting fired): * The system got hacked. * Something important got deleted, and there's no backup. * Backups, security hardening, disaster recovery plan * How good is the backup? * Do you have a system and plan for doing bare-metal recovery? * Check if you can actually restore from the backup. * Run a file system integrity (FSI) checker. * Is your FSI database and configuration file on a read-only medium? * From read-only medium or NFS mount * Run a root kit scanner. * From a read-only medium or NFS mount * Be aware of recently discovered security problems and exploits and incidents. * Do you subscribe to newsgroups and mailing lists that might give you this information? * http://securityfocus.com/archive * Run logwatch (https://sourceforge.net/projects/logwatch/) or other program that alerts you to irregularities in your system logs. * Run an intrusion detection system. * Use a firewall. * IP tables on Linux * There are useful front-ends and canned firewall rules that you can use. * Whitelist incoming connections using tcpd (tcp wrapper). * Whitelist users who are allowed to ssh in. * Is it obvious what OS you are running, what servers you are running? * Make it less obvious. * telnet banners, /etc/issue*, info obtained from telnet to port 25 (SMTP) * Beware the "It won't happen here" or "Our users aren't that smart" mentality that leads to security problems. * Think in terms of redundancy. * Is there a backup of the backup? ---- ===== Efficiency Matters ===== * Proxies can reduce bandwidth usage for over-the-internet installs and general internet usage. * Avoid mass manual installations. * Try to use cloning as much as possible. ---- ===== Remote Access / Administration ===== * Unless every machine you administer is only on from 9am-5pm, system administration is not a 9-to-5 kind of job. * The ability to work remotely is essential and something that SysAdmins should demand and set up. * Enable Wake-on-LAN and/or timed power-on in BIOS'es of systems. * If you have input into the systems that are purchased, stress that these systems come with these types of features. * Learn to use serial consoles, remote KVM switches. * Enable more than one way to access a system remotely. * "# ifdown eth0" will kill network access to your system. * Have you enabled another way to access your system when that happens? * Learn to use the command line. * Yes, even on Windows. * Remote administration is best done at the command line. * Learn SSH port forwarding. * One of the few services/protocols that is trusted enough to allow through firewalls. ---- ===== Startup / Shutdown ===== * Do some checks before shutting down / restarting a system. * Is anyone logged on? * How long can the system afford to be shut down? * Is there a backup? Is a backup needed? * Does the system really require a reboot? * Is the boot loader properly configured? * Is there an alternate way to access the system in case the bootup fails? * See [[cs_471_-_general_sysadmin_principles#remote_access_administration | Remote Access / Administration]] ---- ===== Automation and Scripting ===== * Learn one scripting language well and use it consistently. * Using it consistently helps you to learn it well. * But not a language so obscure that only you use it and understand it. * Unless you really like typing, develop a set of aliases and short scripts that reduce typing. * Learn to use the command line (shell) history. * Increase the size of the command history. * Periodic processing (cron, at, etc.) * Processes must be non-interactive to be scheduled... * or learn //expect// or equivalent system. * Test cron script, as you would any other script you write. * Write scripts so that they are scalable. * Will still be useful as more systems are added to your admin stable. * Applies to any administration solution: Think in terms of scalability. ---- ===== Software Installation/Management, System Maintenance ===== * Workflow for software installation: * Look for official (e.g., Debian, Microsoft) source for software that you want to install. * In the case of Debian, software that can be installed using "apt-get, aptitude, ..." without adding third party sources/mirrors. * If official sources are not available, use a recommended unofficial source or mirror. * Be sure to get the source's or mirror's "keys" to ensure that the software that you get is "signed." * Installed packages should still be upgradeable/maintainable using official package management commands (apt-get, aptitude, yum, etc.) * If no recommended unofficial source or mirror exist, then get the source code, if available, and attempt to build and install it yourself. * Here, can either try to build your own installation packages or install using package "sandboxing" tools like stow/xstow. * Multiple package management schemes on the same system * Apple, Fink, and MacPorts on MacOSX situation * Require fiddling with $PATH and library search paths (Linux:/etc/ld.so.conf*) * Just because a package is "official" does not make it up-to-date (in terms of currency and bug/security fixes). * These are maintained by people with real lives, and the people are often volunteers, doing this on their own time. * For timely fixes, you might have to manually compile/install/maintain regularly instead of relying on the "official" package management system. * Automatic updates * Sounds nice, but an admin should know what's actually being installed or upgraded. * Software installs/updates should be an interactive activity. * But downloading (but not installing) upgrades overnight via scheduling should be fine. * Upgrade the OS/kernel/software carefully. * (or apply security patches, services packs, etc.) * Does it have to be done now? * Is the system being used? Are users logged on? * Can it wait until the end of the quarter/semester? * Have you backed up the system first? * Will you be able to back out to a previous system state if your upgrade is disastrous? * Have you tested the upgrade adequately on a test system(s)? * Have you tested the upgrade after you have applied it? * Only the high priority security fix patches or packages can be applied/installed in lieu of the whole bundle of upgrades/patches. * Install only software that is actually used. * Unnecessary software may contain vulnerabilities. * But depriving users of software they really need may motivate them to install it themselves, in their home directories. * Not all software comes nicely packaged. * such as commercial or "non-free" software * Sun's Java, icc (Intel compiler), eclipse IDE, netbeans IDE, vmware * May require compiling it yourself * and installing in /usr/local or /opt * Before installing, think about the possibility of uninstalling. * Compile and install to /usr/local means files scattered to /usr/local/bin, /usr/local/lib, /usr/local/share, /usr/local/etc, /usr/local/sbin. * Does software's Makefile have a working uninstall target? ---- ===== Users ===== * Will rarely describe their problems with a level of detail that is needed to solve their problems. * Asking for their password so that you can login as them to diagnose their problems, should be avoided. * If you must, then they must change their password afterward. * Can use //su - userid// or some other method to start a process as another user. * Ask for a screenshot. * Sometimes as easy as hitting the key * Or using their cell phones * "Eat your own dog food." * Use the same environment as the users, the same machines on the same networks as the users, so that you can catch problems they may come to you with. * Be a user. Use user tools. * Can't be a UNIX admin w/o first being a UNIX user (Also applies to Windows) * Do concern yourself with the usability of the OS environment for your users. * This will have the added benefit of reducing your support headaches. * Use sensible settings in default shell profiles in /etc/skel * without compromising security * no "." in $PATH, umask=077 * an informative shell prompt * Will the default user interface (window manager or desktop) be familiar and intuitive enough for CS 175 students and also minimize complaints from professors? * No drive letters and easy access to USB flash drives could frustrate many users, whether they are Linux newbies or not. * Systems such as autofs work for removable media, though setting them up are usually not easy tasks. * Balance against ease of administration. * Simple window managers like IceWM: faster startup, smaller memory footprint, easy to configure (for the admin), familiar-looking interface * Desktop environments like Gnome/KDE: slower startup, bigger memory footprint, easy to configure (for the user) * User training * A support web page with FAQs? * Occasional "UNIX for Dummies" seminars? * Don't assume that the users aren't smart enough or that they are not aware of published security exploits. * The "would never happen here" mentality ---- ===== Standardization vs. Diversity ===== * Standardize on one distro of Linux or use multiple distros? * If standardize, choose your standard distro wisely. * Consider the projected continuity of a distro. * Debian's continuity seems like a good bet. * less so with many Debian derivatives * Multiple distros also a good idea * redhat still the most widely used distro, and rpm the most widely used package format * Good to be familiar with rpm-based distros and redhat-like distros. * Multiple UNIX versions also a good idea * What if, by some miracle, SCO wins and Linux is lost? * FreeBSD waiting in the wings * Solaris x86? (future looks bleak) * AIX? (not free; bleh) * Mac OS X (see AIX) ---- ===== Advocacy ===== * Should a UNIX admin advocate the use of UNIX? * Up to a point * An admin should advocate the right tool for the job .. * .. and learn the right tool for the job, even if that happens to be Windows. ---- ===== Learning Administration ===== * Learn administration by doing administration. * More likely to learn if something goes wrong. ---- ===== Keeping up with the Joneses ===== * What technologies are being used "out there," and are we behind the times? * LDAP instead of NIS * cfengine instead of my own scripts * systemimager instead of my own scripts ---- ===== Ethics and Licenses ===== * A professor wants to use their single-user license for a language interpretor for an entire class of students .. * .. and wants you to install the interpretor on the server. * A student has 2 GB worth of mp3s in his home directory. * A student has 2 GB worth of legitimate research output in his home directory. * You have access to your boss' email spool file. ----