====== Security, Backups, Logging ====== ---- * See [[cs_471_-_general_sysadmin_principles#the_hard_stuff_security_and_backups|General SysAdmin Principles on Security and Backups]] ---- ===== Security ===== ==== Some Security Breach Examples We've Seen Over The Years ==== * (9/2003) SADMIND remote command execution on Solaris * http://archive.cert.uni-stuttgart.de/bugtraq/2003/09/msg00297.html * "This exploit targets a weakness in the default security settings of the sadmind RPC application. This application is installed and enabled by default on most versions of the Solaris operating system." * (3/2005) Linux kernel uselib() privilege elevation * http://isec.pl/vulnerabilities/isec-0021-uselib.txt and following related email exchange: Subject: another breakin Date: Fri, 18 Mar 2005 11:51:35 -0500 From: * * To: "Joseph Chung" Joe, I found that the bluehawk server had two processes named "uselib24" - which is apparently an exploit for the uselib() syscall. The webserver user was running it - so I need to figure out how they got the webserver to run a program to begin with. Date: Fri, 18 Mar 2005 13:49:35 -0500 From: Joe Chung To: * * Subject: Re: another breakin On Fri, Mar 18, 2005 at 11:51:35AM -0500, * * wrote: > Joe, I found that the bluehawk server had two processes > named "uselib24" - which is apparently an exploit for the uselib() I played around with the published exploit when it first came out (http://isec.pl/vulnerabilities/isec-0021-uselib.txt; just compiled it and ran it as jchung). If successful, it's supposed to give you a root shell. I was not able to get a root shell on rockhopper or the Linux lab machines with this exploit. According to a modified published exploit which is what uselib24 is (http://packetstorm.linuxsecurity.com/0501-exploits/uselib24.c), this'll only work on the 2.4 series of kernels. bluehawk was on 2.6.10-something, right? > syscall. The webserver user was running it - so I need to figure out > how they got the webserver to run a program to begin with. So it was running as the 'web' user, right? Could it have been someone local cracking passwords? bluehawk:/etc/passwd looks like a port from Irix. Have you seen https://www.redhat.com/archives/fedora-list/2005-March/msg00594.html yet? * (4/2004) university attacks used sniffed telnet traffic * [[https://web.archive.org/web/20060103103136/https://www.infoworld.com/article/04/04/14/HNuniattacks_1.html|http://www.infoworld.com/article/04/04/14/HNuniattacks_1.html]] * "Attackers gained access to the systems by cracking or sniffing passwords from insecure network traffic such as Telnet remote communications sessions or from password files on other compromised systems, according to the alert. ... Once logged on to a system, the attackers looked for systems that were not up to date with their operating system patches, then used known software exploits to elevate their privileges from user to administrator (or 'root') status." * (2/2007) Solaris 10 Telnet remote root login exploit * http://erratasec.blogspot.com/2007/02/trivial-remote-solaris-0day-disable.html * (9/2009) Microsoft Active Directory's 'reboot' and other accounts expose rockhopper. * These accounts had no password by default. * (4/2010) Hidden irc server on cslab, possibly through Oracle 8 WebDB. * Some key things to note: * privilege escalation * weak protocols * known vulnerabilities ---- ==== Security Exploit Types and Pathways ==== * User or administrator account gets compromised * by social engineering, dumpster diving, password guessing, network sniffing, etc. * rockhopper:/var/log/secure (//grep "invalid user" /var/log/secure//) * ... followed by privilege escalation using known vulnerabilities * Known vulnerabilities have often involved buffer overflows in SUID root programs. * ... followed by network sniffing or local capture of more account passwords using //"rootkits"// * ... which can open access to other systems on the network * A system service has a vulnerability which is exploited //remotely// to run arbitrary code on the system as a user or administrator. * Solaris SADMIND example above * web server and web application exploits * Physical access * Voting machines * Kiosks ---- ==== Prevention ==== * Use secure software. * ssh/scp/sftp instead of telnet/ftp and the old r-protocols (rsh, rcp, rlogin) * Apply security patches and security fixes. * Keep system up-to-date * Remove unneeded software and disable unneeded services. * SUID programs on servers that no one could possibly need * Services in /etc/inetd.conf or /etc/xinetd.d * Do you really need Apache? MySQL? Squid? * Stay informed. * Bugtraq, http://securityfocus.com * Vendor email security alerts * Try the published exploits yourself on your systems. * Restrict access * Physical access and lockdown * Allow connections only from some networks and hosts if possible * firewalls (iptables), tcpd * Security audit yourself. * Scan for open ports, guest and test accounts. * Check for easy passwords. ---- ==== Detection / Analysis ==== * File system integrity checkers * Tripwire, aide * Rootkit detectors * rkhunter * Intrusion detection systems * snort, wireshark * Honeypots, honeynets * Logwatch ---- ==== Reaction / Recovery ==== * Take system off the network * Look at logs * Check login times * Restore from trusted backups that (hopefully) predate the break-in ---- ==== Internet of Things (IoT) Security ==== * Connected devices can be particularly insecure. * They are not personal computers that you interact with all the time. * They often don't have traditional consoles (screens) or input devices. * They are meant to be "forgotten." * They are often shipped with default settings, including hostnames, accounts and passwords. ---- ---- ===== Backups ===== ==== Determining a Backup Strategy ==== * Part of overall "disaster recovery" plan * Relevant questions * How many drives and partitions do you have? * What does each partition hold? How unique is it? * How often do the contents change? * How likely is the disc to fail? * How much does it cost to replace the contents? * How much can your backup device hold? * How do you to recover if you can't boot from disc anymore? * How long does it take to back up/restore? * (How much time can you afford?) * What if you only need to restore some files? ---- ---- ===== Logging and Monitoring ===== * A critical part of the sysadmin's job is monitoring the system or systems or network. ==== Syslog ==== * Many UNIX systems use the //Syslog// facility for system logging. * //Syslog// allows you to enable messages by level and by facility. * Levels can be considered various levels of a problem (e.g. warning, error, emergency), whereas facilities are considered to be service areas (e.g. printing, email, network ). * //Syslog// also allows you to forward log entries to another machine for processing, in this way syslog functions as a distributed error manager. * The //Syslog// levels available are the following: LOG_EMERG A panic condition. This is normally broadcast to all users. LOG_ALERT A condition that should be corrected immediately, such as a corrupted system database. LOG_CRIT Critical conditions, e.g., hard device errors. LOG_ERR Errors. LOG_WARNING Warning messages. LOG_NOTICE Conditions that are not error conditions, but should possibly be handled specially. LOG_INFO Informational messages. LOG_DEBUG Messages that contain information normally of use only when debugging a program. ==== Facilities That Can Use Syslog ==== * Some of the facilities under which you can log messages include the following: LOG_KERN Messages generated by the kernel. These cannot be generated by any user processes. LOG_USER Messages generated by random user processes. This is the default facility identifier if none is specified. LOG_MAIL The mail system. LOG_DAEMON System daemons, such as telnetd, ftpd, rshd, etc. LOG_AUTH The authorization system: login, su, getty, etc. ftpd, and rshd also use LOG_AUTH. LOG_LPR The line printer spooling system: lpr, lpd, etc. LOG_LOCAL0 Reserved for local use. Similarly for LOG_LOCAL1 through LOG_LOCAL7. ==== Startup and Configuration of Syslog ==== * //Syslog// runs as a deamon named syslogd * This deamon is generally started up in the early stages of multi-user bootup. * The syslogd reads its configuration information whenever it receives the HUP signal. * The file generally is named /etc/syslog.conf . * Syslog uses a notation of facility.level * The keyword LOG_ can be ignored. * The second part of the line determines what SYSLOG does with the message. * That is based on what the starting character is; below is a summary: / A filename (beginning with a leading slash). The file will be opened in append mode. @ A hostname preceded by an at sign (``@''). Selected messages are forwarded to the syslogd on the named host. Letter A comma-separated list of users. Selected messages are written to those users if they are logged in. * An asterisk. Selected messages are written to all logged-in users. ---- ==== Other Important Files Where Information is Logged ==== * UNIX has a large number of files that hold logging information. * A few of the important ones... /var/log/messages The messages file holds information that prints to the console. These might include root logins and su attempts. /var/log/lastlog This file holds the most recent login time for each user in the system. On some systems this is a directory with each user stored as a single record. The directory hold a record for each person that has ever logged in. It can be used to track login records. The utmp and wtmp files The utmp file is where information such as the terminal line, login time, and command executing are stored for access by the who command. The wtmp keeps track of logins and logouts since reboot. The command last reads that file and processes the information. /var/adm/acct (Solaris) This is the system accounting file. If enabled, the accounting file records a record for every process listing the following information: + username who ran the command + name of the command + CPU time used + completion timestamp of the process + Flag indicating completion status Accounting information can be very useful in monitoring who is doing what on your system. In addition, by summarizing accounting to develop cummulative statistics you can often build a case for upgrading the system. /var/adm/sulog (Solaris) The sulog file holds records for everyone that has su'ed on the system. /var/log/lp-errs or /var/spool/lpd Many sites direct the SYSLOG output for the lpd (printing)system to this file. This file is then monitored for problems. ==== Handling Logs ==== * Logging is a critical responsibility of the system administrator * However, in large modern system the amount of information generated can overwhelm most system administrators. * Most system administrators develop scripts or tools to examine the log files and extract the important information. * Logwatch (man logwatch) can help. * [[https://cssegit.monmouth.edu:2443/jchung/cs471/blob/master/files/logwatch.rockhopper.sample|Sample logwatch report]] from 4/2018 * Logs can get large and fill up file systems * Utilities like //Logrotate// are used to archive and optionally compress log files when they reach a certain size or age. ---- ===== Lab Activity (DO NOT DO UNTIL INSTRUCTED) ===== ==== 1. The "wheel" group ==== * Create the //wheel// group to restrict certain administrative commands to a small group of users. * Apply to the ///bin/su// command on your VM. * Steps: * Add the //wheel// group to your system: sudo groupadd wheel * Add a normal user to the //wheel// group by editing the ///etc/group// file: sudo nano /etc/group or by running the usermod command: sudo usermod -G wheel username On belushi, the wheel group is defined as follows: wheel:x:1002:jchung * Change the owners and permissions of the ///bin/su// command: # See old owners, permissions ls -l /bin/su # Change owners, permissions sudo chgrp wheel /bin/su sudo chmod o-rx /bin/su sudo chmod u+s /bin/su # See new owners, permissions ls -l /bin/su * The new ownerships and permissions for ///bin/su// should look like the following: -rwsr-x--- 1 root wheel 40168 May 17 2017 /bin/su * Only members of the //wheel// will now be able to run //su// on your VM. ==== 2. Port scanning with nmap ==== * Install the //nmap// port scanning tool. * Look at its man page. * Scan your own host with //nmap//. * Scan another student's host with //nmap//. * Don't abuse //nmap//. ==== 3. Password checking with john ==== * On your Linux container, install the //john// package to audit password strength. Also install the package //wordlist//. You will look for passwords that are common dictionary words. * The //wordlist// package is a virtual package that refers to several possible wordlist packages. The one that you want to install is called //wamerican-huge//. * You need to run //john// as root, using the wordlist ///usr/share/dict/american-english-huge// to audit the encrypted passwords in /etc/shadow: sudo john -wordlist:/usr/share/dict/american-english-huge /etc/shadow * The process can take a long time. Any passwords that match a dictionary word will be displayed, along with the user that owns the password. ==== 4. Backups with rsnapshot ==== * Install and use [[http://rsnapshot.org|rsnapshot]] to create daily, weekly and monthly backups of the /etc, /home and /usr/local directories on your Linux container. * Rsnapshot is a simple utility. See its man page (//'man rsnapshot'//) for configuration syntax and cron job setup. * As root, edit the //rsnapshot// config file, ///etc/rsnapshot.conf//. * Note the file system location where backups will be stored (//snapshot_root//). * Note the file system directories that will be backed up (//"BACKUP POINTS / SCRIPTS"//). * The ///etc, /home and /usr/local// directories are already listed by default. * Change the //"BACKUP INTERVALS"// to the following to keep 7 daily, 4 weekly and 12 monthly backups: # modified by jchung, 4/2018 # retain hourly 6 retain daily 7 retain weekly 4 retain monthly 12 * Rsnapshot needs to use //cron// to perform scheduled backups. We will use root's cron table (crontab) to perform //rsnapshot// backups. Edit root's crontab by running //'sudo crontab -e'//, and add the following schedule to root's crontab: # m h dom mon dow command # rsnapshot jobs, jchung, 4/2018 0 0 * * * /usr/bin/rsnapshot daily 0 1 * * 0 /usr/bin/rsnapshot weekly 0 2 1 * * /usr/bin/rsnapshot monthly ==== 5. Install and configure logwatch ==== * Install and configure the [[https://www.digitalocean.com/community/tutorials/how-to-install-and-use-logwatch-log-analyzer-and-reporter-on-a-vps|logwatch log analyzer and reporter]] on your Linux VM. * You should have installed and tested the [[cs_471_-_email_networked_file_systems_system_file_sharing#the_postfix_mta|Postfix MTA]] already on your VM. * You will configure logwatch and Postfix to email you daily log reports to your MU email account. * By default, //logwatch// runs once per day and emails a report to root. See the ///etc/cron.daily/00logwatch// script that runs //logwatch// once per day. * Configure logwatch or Postfix to send the email to you (student_id@monmouth.edu). * Do **ONE** of the following: * 1 - Modify ///etc/cron.daily/00logwatch// to use your student email address. * 2 - Or add a [[https://www.ccsf.edu/Pub/UNIXhelp/mail/file_forward.html|.forward]] file to your normal user home directory that contains your student email address. * Verify that //logwatch// sends you an email report once per day. ==== 6. Check for and mitigate Meltdown/Spectre vulnerability ==== On your VMs, check for and try to mitigate vulnerability to the [[https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)|Meltdown/Spectre]] vulnerability. We will download and run a [[https://github.com/speed47/spectre-meltdown-checker|published vulnerability checking script]] (//spectre-meltdown-checker.sh//). The kernel version you are currently running on your VMs is //3.16.0-4-amd64//, per the output of //'uname -a'//. The //spectre-meltdown-checker.sh// script will report that your VM is vulnerable to the //Meltdown// exploit. To mitigate the //Meltdown// vulnerability, you will need to upgrade to a newer kernel. To upgrade to a new kernel, run the following commands to upgrade packages, including available kernel upgrades: sudo apt-get update sudo apt-get dist-upgrade One of the packages will be a newer version of the kernel (//3.16.0-5-amd64//). To use the new kernel, you will need to reboot with sudo reboot After reboot, verify that your VM is running kernel version //3.16.0-5-amd64//, and then run the //spectre-meltdown-checker.sh// script again to verify that your system is no longer vulnerable to //Meltdown//. ----