Table of Contents
Some General SysAdmin Principles
Documentation
- Document your work.
- Keep a journal.
- Use at least a plain text file.
- Date each entry.
- If you have a boss that you send your journal to, spellcheck your journal.
- The journal isn't just for your boss, it's for you.
- It's your own FAQ.
- It's your own collection of HOWTOs to help you and others reproduce work in the future on other systems.
- When you make an entry in your journal, ask yourself, “Is this enough information to allow me and maybe others to reproduce this work that I just spent hours on?
- It can get huge over time, which is why there may be a need for a better system of self-documentation than a flat text file.
- Possibly something blog-like where you can add tags to each journal entry.
- The journal has to be backed up and accessible from anywhere, at your fingertips.
- This is why some admins keep logs in notebooks (paper notebooks), though this has several drawbacks.
- If your journal is on a company server or your office workstation, and the network is down, and you're at home 30 miles away, and you need some vital information that's in the journal…now what?
- Use comments in /etc/config files to document your changes.
- Sign and date your comments for your future reference and for other admins.
- In general, be well-organized.
- whether that's the directory structure under your home directory or your written journal
- Use technology in documentation.
- Set up a web page or FAQ.
- Use video or digital photography.
- Sometimes, a picture is literally worth a thousand words.
The hard stuff (security and backups)
- Take care of the hard stuff first.
- The worst feelings for a SysAdmin (makes you fear getting fired):
- The system got hacked.
- Something important got deleted, and there's no backup.
- Backups, security hardening, disaster recovery plan
- How good is the backup?
- Do you have a system and plan for doing bare-metal recovery?
- Check if you can actually restore from the backup.
- Run a file system integrity (FSI) checker.
- Is your FSI database and configuration file on a read-only medium?
- From read-only medium or NFS mount
- Run a root kit scanner.
- From a read-only medium or NFS mount
- Be aware of recently discovered security problems and exploits and incidents.
- Do you subscribe to newsgroups and mailing lists that might give you this information?
- Run logwatch (https://sourceforge.net/projects/logwatch/) or other program that alerts you to irregularities in your system logs.
- Run an intrusion detection system.
- Use a firewall.
- IP tables on Linux
- There are useful front-ends and canned firewall rules that you can use.
- Whitelist incoming connections using tcpd (tcp wrapper).
- Whitelist users who are allowed to ssh in.
- Is it obvious what OS you are running, what servers you are running?
- Make it less obvious.
- telnet banners, /etc/issue*, info obtained from telnet to port 25 (SMTP)
- Beware the “It won't happen here” or “Our users aren't that smart” mentality that leads to security problems.
- Think in terms of redundancy.
- Is there a backup of the backup?
Efficiency Matters
- Proxies can reduce bandwidth usage for over-the-internet installs and general internet usage.
- Avoid mass manual installations.
- Try to use cloning as much as possible.
Remote Access / Administration
- Unless every machine you administer is only on from 9am-5pm, system administration is not a 9-to-5 kind of job.
- The ability to work remotely is essential and something that SysAdmins should demand and set up.
- Enable Wake-on-LAN and/or timed power-on in BIOS'es of systems.
- If you have input into the systems that are purchased, stress that these systems come with these types of features.
- Learn to use serial consoles, remote KVM switches.
- Enable more than one way to access a system remotely.
- ”# ifdown eth0” will kill network access to your system.
- Have you enabled another way to access your system when that happens?
- Learn to use the command line.
- Yes, even on Windows.
- Remote administration is best done at the command line.
- Learn SSH port forwarding.
- One of the few services/protocols that is trusted enough to allow through firewalls.
Startup / Shutdown
- Do some checks before shutting down / restarting a system.
- Is anyone logged on?
- How long can the system afford to be shut down?
- Is there a backup? Is a backup needed?
- Does the system really require a reboot?
- Is the boot loader properly configured?
- Is there an alternate way to access the system in case the bootup fails?
Automation and Scripting
- Learn one scripting language well and use it consistently.
- Using it consistently helps you to learn it well.
- But not a language so obscure that only you use it and understand it.
- Unless you really like typing, develop a set of aliases and short scripts that reduce typing.
- Learn to use the command line (shell) history.
- Increase the size of the command history.
- Periodic processing (cron, at, etc.)
- Processes must be non-interactive to be scheduled…
- or learn expect or equivalent system.
- Test cron script, as you would any other script you write.
- Write scripts so that they are scalable.
- Will still be useful as more systems are added to your admin stable.
- Applies to any administration solution: Think in terms of scalability.
Software Installation/Management, System Maintenance
- Workflow for software installation:
- Look for official (e.g., Debian, Microsoft) source for software that you want to install.
- In the case of Debian, software that can be installed using “apt-get, aptitude, …” without adding third party sources/mirrors.
- If official sources are not available, use a recommended unofficial source or mirror.
- Be sure to get the source's or mirror's “keys” to ensure that the software that you get is “signed.”
- Installed packages should still be upgradeable/maintainable using official package management commands (apt-get, aptitude, yum, etc.)
- If no recommended unofficial source or mirror exist, then get the source code, if available, and attempt to build and install it yourself.
- Here, can either try to build your own installation packages or install using package “sandboxing” tools like stow/xstow.
- Multiple package management schemes on the same system
- Apple, Fink, and MacPorts on MacOSX situation
- Require fiddling with $PATH and library search paths (Linux:/etc/ld.so.conf*)
- Just because a package is “official” does not make it up-to-date (in terms of currency and bug/security fixes).
- These are maintained by people with real lives, and the people are often volunteers, doing this on their own time.
- For timely fixes, you might have to manually compile/install/maintain regularly instead of relying on the “official” package management system.
- Automatic updates
- Sounds nice, but an admin should know what's actually being installed or upgraded.
- Software installs/updates should be an interactive activity.
- But downloading (but not installing) upgrades overnight via scheduling should be fine.
- Upgrade the OS/kernel/software carefully.
- (or apply security patches, services packs, etc.)
- Does it have to be done now?
- Is the system being used? Are users logged on?
- Can it wait until the end of the quarter/semester?
- Have you backed up the system first?
- Will you be able to back out to a previous system state if your upgrade is disastrous?
- Have you tested the upgrade adequately on a test system(s)?
- Have you tested the upgrade after you have applied it?
- Only the high priority security fix patches or packages can be applied/installed in lieu of the whole bundle of upgrades/patches.
- Install only software that is actually used.
- Unnecessary software may contain vulnerabilities.
- But depriving users of software they really need may motivate them to install it themselves, in their home directories.
- Not all software comes nicely packaged.
- such as commercial or “non-free” software
- Sun's Java, icc (Intel compiler), eclipse IDE, netbeans IDE, vmware
- May require compiling it yourself
- and installing in /usr/local or /opt
- Before installing, think about the possibility of uninstalling.
- Compile and install to /usr/local means files scattered to /usr/local/bin, /usr/local/lib, /usr/local/share, /usr/local/etc, /usr/local/sbin.
- Does software's Makefile have a working uninstall target?
Users
- Will rarely describe their problems with a level of detail that is needed to solve their problems.
- Asking for their password so that you can login as them to diagnose their problems, should be avoided.
- If you must, then they must change their password afterward.
- Can use su - userid or some other method to start a process as another user.
- Ask for a screenshot.
- Sometimes as easy as hitting the <Print Screen> key
- Or using their cell phones
- “Eat your own dog food.”
- Use the same environment as the users, the same machines on the same networks as the users, so that you can catch problems they may come to you with.
- Be a user. Use user tools.
- Can't be a UNIX admin w/o first being a UNIX user (Also applies to Windows)
- Do concern yourself with the usability of the OS environment for your users.
- This will have the added benefit of reducing your support headaches.
- Use sensible settings in default shell profiles in /etc/skel
- without compromising security
- no ”.” in $PATH, umask=077
- an informative shell prompt
- Will the default user interface (window manager or desktop) be familiar and intuitive enough for CS 175 students and also minimize complaints from professors?
- No drive letters and easy access to USB flash drives could frustrate many users, whether they are Linux newbies or not.
- Systems such as autofs work for removable media, though setting them up are usually not easy tasks.
- Balance against ease of administration.
- Simple window managers like IceWM: faster startup, smaller memory footprint, easy to configure (for the admin), familiar-looking interface
- Desktop environments like Gnome/KDE: slower startup, bigger memory footprint, easy to configure (for the user)
- User training
- A support web page with FAQs?
- Occasional “UNIX for Dummies” seminars?
- Don't assume that the users aren't smart enough or that they are not aware of published security exploits.
- The “would never happen here” mentality
Standardization vs. Diversity
- Standardize on one distro of Linux or use multiple distros?
- If standardize, choose your standard distro wisely.
- Consider the projected continuity of a distro.
- Debian's continuity seems like a good bet.
- less so with many Debian derivatives
- Multiple distros also a good idea
- redhat still the most widely used distro, and rpm the most widely used package format
- Good to be familiar with rpm-based distros and redhat-like distros.
- Multiple UNIX versions also a good idea
- What if, by some miracle, SCO wins and Linux is lost?
- FreeBSD waiting in the wings
- Solaris x86? (future looks bleak)
- AIX? (not free; bleh)
- Mac OS X (see AIX)
Advocacy
- Should a UNIX admin advocate the use of UNIX?
- Up to a point
- An admin should advocate the right tool for the job ..
- .. and learn the right tool for the job, even if that happens to be Windows.
Learning Administration
- Learn administration by doing administration.
- More likely to learn if something goes wrong.
Keeping up with the Joneses
- What technologies are being used “out there,” and are we behind the times?
- LDAP instead of NIS
- cfengine instead of my own scripts
- systemimager instead of my own scripts
Ethics and Licenses
- A professor wants to use their single-user license for a language interpretor for an entire class of students ..
- .. and wants you to install the interpretor on the server.
- A student has 2 GB worth of mp3s in his home directory.
- A student has 2 GB worth of legitimate research output in his home directory.
- You have access to your boss' email spool file.
