Joe Chung
Monmouth U. Homepage

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Security, Backups, Logging ======

----

  * See [[cs_471_-_general_sysadmin_principles#the_hard_stuff_security_and_backups|General SysAdmin Principles on Security and Backups]]

----

===== Security =====

==== Some Security Breach Examples We've Seen Over The Years ====

  * (9/2003) SADMIND remote command execution on Solaris
    * http://archive.cert.uni-stuttgart.de/bugtraq/2003/09/msg00297.html
    * "This exploit targets a weakness in the default security settings of the sadmind RPC application. This application is installed and enabled by default on most versions of the Solaris operating system."

  * (3/2005) Linux kernel uselib() privilege elevation
    * http://isec.pl/vulnerabilities/isec-0021-uselib.txt and following related email exchange:
<code>
    Subject: another breakin 
    Date: Fri, 18 Mar 2005 11:51:35 -0500
    From: * *
    To: "Joseph Chung" <jchung@monmouth.edu>

    Joe,  I found that the bluehawk server had two processes
    named "uselib24" - which is apparently an exploit for the uselib()
    syscall.  The webserver user was running it - so I need to figure out
    how they got the webserver to run a program to begin with.

       Date: Fri, 18 Mar 2005 13:49:35 -0500
       From: Joe Chung <jchung@monmouth.edu>
       To: * *
       Subject: Re: another breakin

       On Fri, Mar 18, 2005 at 11:51:35AM -0500, * * wrote:
       > Joe,  I found that the bluehawk server had two processes
       > named "uselib24" - which is apparently an exploit for the uselib()  

       I played around with the published exploit when it first came out  
       (http://isec.pl/vulnerabilities/isec-0021-uselib.txt; just compiled it
       and ran it as jchung). If successful, it's supposed to give you a    
       root shell. I was not able to get a root shell on rockhopper or the
       Linux lab machines with this exploit.

       According to a modified published exploit which is what uselib24 is
       (http://packetstorm.linuxsecurity.com/0501-exploits/uselib24.c),
       this'll only work on the 2.4 series of kernels. bluehawk was on
       2.6.10-something, right?

       > syscall.  The webserver user was running it - so I need to figure out
       > how they got the webserver to run a program to begin with.

       So it was running as the 'web' user, right? Could it have been someone
       local cracking passwords? bluehawk:/etc/passwd looks like a port from
       Irix.

       Have you seen
       https://www.redhat.com/archives/fedora-list/2005-March/msg00594.html
       yet?
</code>

  * (4/2004) university attacks used sniffed telnet traffic
    * [[https://web.archive.org/web/20060103103136/https://www.infoworld.com/article/04/04/14/HNuniattacks_1.html|http://www.infoworld.com/article/04/04/14/HNuniattacks_1.html]]
    * "Attackers gained access to the systems by cracking or sniffing passwords from insecure network traffic such as Telnet remote communications sessions or from password files on other compromised systems, according to the alert. ... Once logged on to a system, the attackers looked for systems that were not up to date with their operating system patches, then used known software exploits to elevate their privileges from user to administrator (or 'root') status." 

  * (2/2007) Solaris 10 Telnet remote root login exploit
    * http://erratasec.blogspot.com/2007/02/trivial-remote-solaris-0day-disable.html

  * (9/2009) Microsoft Active Directory's 'reboot' and other accounts expose rockhopper.
    * These accounts had no password by default.

  * (4/2010) Hidden irc server on cslab, possibly through Oracle 8 WebDB.

  * Some key things to note:
    * privilege escalation
    * weak protocols
    * known vulnerabilities

----

==== Security Exploit Types and Pathways ====

  * User or administrator account gets compromised
    * by social engineering, dumpster diving, password guessing, network sniffing, etc.
      * rockhopper:/var/log/secure (//grep "invalid user" /var/log/secure//)
    * ... followed by privilege escalation using known vulnerabilities
      * Known vulnerabilities have often involved buffer overflows in SUID root programs.
    * ... followed by network sniffing or local capture of more account passwords using //"rootkits"//
      * ... which can open access to other systems on the network

  * A system service has a vulnerability which is exploited //remotely// to run arbitrary code on the system as a user or administrator.
    * Solaris SADMIND example above
    * web server and web application exploits

  * Physical access
    * Voting machines
    * Kiosks

----

==== Prevention ====

    * Use secure software.
          * ssh/scp/sftp instead of telnet/ftp and the old r-protocols (rsh, rcp, rlogin)
          * Apply security patches and security fixes.
            * Keep system up-to-date
    * Remove unneeded software and disable unneeded services.
          * SUID programs on servers that no one could possibly need
          * Services in /etc/inetd.conf or /etc/xinetd.d
          * Do you really need Apache? MySQL? Squid? 
    * Stay informed.
          * Bugtraq, http://securityfocus.com
          * Vendor email security alerts
          * Try the published exploits yourself on your systems.
    * Restrict access
          * Physical access and lockdown
          * Allow connections only from some networks and hosts if possible
                * firewalls (iptables), tcpd 
    * Security audit yourself.
          * Scan for open ports, guest and test accounts.
    * Check for easy passwords.

----

==== Detection / Analysis ====

    * File system integrity checkers
          * Tripwire, aide 
    * Rootkit detectors
          * rkhunter
    * Intrusion detection systems
          * snort, wireshark
    * Honeypots, honeynets
    * Logwatch 

----

==== Reaction / Recovery ====

    * Take system off the network
    * Look at logs
      * Check login times 
    * Restore from trusted backups that (hopefully) predate the break-in 

----

==== Internet of Things (IoT) Security ====

  * Connected devices can be particularly insecure.
  * They are not personal computers that you interact with all the time.
    * They often don't have traditional consoles (screens) or input devices.
    * They are meant to be "forgotten."
  * They are often shipped with default settings, including hostnames, accounts and passwords.

----
----

===== Backups =====


==== Determining a Backup Strategy ====

   * Part of overall "disaster recovery" plan

   * Relevant questions
      * How many drives and partitions do you have?
      * What does each partition hold? How unique is it?
      * How often do the contents change?
      * How likely is the disc to fail?
      * How much does it cost to replace the contents?
      * How much can your backup device hold?                 
      * How do you to recover if you can't boot from disc anymore?
      * How long does it take to back up/restore?
         * (How much time can you afford?)
      * What if you only need to restore some files?

----
----

===== Logging and Monitoring =====

  * A critical part of the sysadmin's job is monitoring the system or systems or network.

==== Syslog ====

  * Many UNIX systems use the //Syslog// facility for system logging.
  * //Syslog// allows you to enable messages by level and by facility.
  * Levels can be considered various levels of a problem (e.g.  warning, error, emergency), whereas facilities are considered to be service areas (e.g. printing, email, network ).
  * //Syslog// also allows you to forward log entries to another machine for processing, in this way syslog functions as a distributed error manager.
    * The //Syslog// levels available are the following: 

<code>
       LOG_EMERG     A panic condition.  This is normally broadcast to all
                     users.

       LOG_ALERT     A condition that should be corrected immediately,
                     such as a corrupted system database.

       LOG_CRIT      Critical conditions, e.g., hard device errors.

       LOG_ERR       Errors.

       LOG_WARNING   Warning messages.

       LOG_NOTICE    Conditions that are not error conditions, but should
                     possibly be handled specially.

       LOG_INFO      Informational messages.

       LOG_DEBUG     Messages that contain information normally of
                     use only when debugging a program.  
</code>

==== Facilities That Can Use Syslog ====

  *  Some of the facilities under which you can log messages include the following: 

<code>
       LOG_KERN      Messages generated by the kernel.  These cannot be
                     generated by any user processes.

       LOG_USER      Messages generated by random user processes.  This is
                     the default facility identifier if none is specified.

       LOG_MAIL      The mail system.

       LOG_DAEMON    System daemons, such as telnetd, ftpd, rshd,
                     etc.

       LOG_AUTH      The authorization system:  login, su, getty,
                     etc.  ftpd, and rshd also use LOG_AUTH.

       LOG_LPR       The line printer spooling system: lpr, lpd, etc.

       LOG_LOCAL0    Reserved for local use. Similarly for LOG_LOCAL1 through
                     LOG_LOCAL7.
</code>

==== Startup and Configuration of Syslog ====

    * //Syslog// runs as a deamon named syslogd
    * This deamon is generally started up in the early stages of multi-user bootup.
    * The syslogd reads its configuration information whenever it receives the HUP signal.
    * The file generally is named /etc/syslog.conf . 
    * Syslog uses a notation of facility.level
       * The keyword LOG_ can be ignored.
       * The second part of the line determines what SYSLOG does with the message.
          * That is based on what the starting character is; below is a summary: 

<code>
       /  A filename (beginning with a leading slash).  The file will be opened
          in append mode.

       @  A hostname preceded by an at sign (``@'').  Selected messages are
          forwarded to the syslogd on the named host.

       Letter A comma-separated list of users.  Selected messages are written to
              those users if they are logged in.

       *  An asterisk.  Selected messages are written to all logged-in users.
</code>

----

==== Other Important Files Where Information is Logged ====

    * UNIX has a large number of files that hold logging information.
    * A few of the important ones... 

<code>
       /var/log/messages

         The messages file holds information that prints to the console.
         These might include root logins and su attempts.

       /var/log/lastlog

         This file holds the most recent login time for each user in the
         system. On some systems this is a directory with each user stored
         as a single record. The directory hold a record for each person
         that has ever logged in. It can be used to track login records.

       The utmp and wtmp files

         The utmp file is where information such as the terminal line, login
         time, and command executing are stored for access by the who
         command. The wtmp keeps track of logins and logouts since reboot.
         The command last reads that file and processes the information.

       /var/adm/acct (Solaris)

         This is the system accounting file. If enabled, the accounting file
         records a record for every process listing the following
         information:

           + username who ran the command
           + name of the command
           + CPU time used
           + completion timestamp of the process
           + Flag indicating completion status

         Accounting information can be very useful in monitoring who is
         doing what on your system. In addition, by summarizing accounting
         to develop cummulative statistics you can often build a case for
         upgrading the system.

       /var/adm/sulog (Solaris)

         The sulog file holds records for everyone that has su'ed on the
         system.

       /var/log/lp-errs or /var/spool/lpd

         Many sites direct the SYSLOG output for the lpd (printing)system
         to this file. This file is then monitored for problems.
</code>

==== Handling Logs ====

    * Logging is a critical responsibility of the system administrator
    * However, in large modern system the amount of information generated can overwhelm most system administrators.
    * Most system administrators develop scripts or tools to examine the log files and extract the important information.
       * Logwatch (man logwatch) can help.
         * [[https://cssegit.monmouth.edu:2443/jchung/cs471/blob/master/files/logwatch.rockhopper.sample|Sample logwatch report]] from 4/2018
    * Logs can get large and fill up file systems
       * Utilities like //Logrotate// are used to archive and optionally compress log files when they reach a certain size or age.

----

===== Lab Activity (DO NOT DO UNTIL INSTRUCTED) =====

==== 1. The "wheel" group ====

  * Create the //wheel// group to restrict certain administrative commands to a small group of users.
    * Apply to the ///bin/su// command on your VM.
      * Steps:
        * Add the //wheel// group to your system:

                  sudo groupadd wheel

        * Add a normal user to the //wheel// group by editing the ///etc/group// file:

                      sudo nano /etc/group
                  
                  or by running the usermod command:
                  
                      sudo usermod -G wheel username
                  
                  On belushi, the wheel group is defined as follows:
                  
                      wheel:x:1002:jchung

        * Change the owners and permissions of the ///bin/su// command:

                  # See old owners, permissions
                  ls -l /bin/su
                  
                  # Change owners, permissions
                  sudo chgrp wheel /bin/su
                  sudo chmod o-rx /bin/su
                  sudo chmod u+s /bin/su
                  
                  # See new owners, permissions
                  ls -l /bin/su

        * The new ownerships and permissions for ///bin/su// should look like the following:
<code>
                -rwsr-x--- 1 root wheel 40168 May 17  2017 /bin/su
</code>

        * Only members of the //wheel// will now be able to run //su// on your VM.

==== 2. Port scanning with nmap ====

  * Install the //nmap// port scanning tool.
  * Look at its man page.
  * Scan your own host with //nmap//.
  * Scan another student's host with //nmap//.
  * Don't abuse //nmap//.

==== 3. Password checking with john ====

  * On your Linux container, install the //john// package to audit password strength. Also install the package //wordlist//. You will look for passwords that are common dictionary words.
    * The //wordlist// package is a virtual package that refers to several possible wordlist packages. The one that you want to install is called //wamerican-huge//.
    * You need to run //john// as root, using the wordlist ///usr/share/dict/american-english-huge// to audit the encrypted passwords in /etc/shadow:

        sudo john -wordlist:/usr/share/dict/american-english-huge /etc/shadow

    * The process can take a long time. Any passwords that match a dictionary word will be displayed, along with the user that owns the password.


==== 4. Backups with rsnapshot ====

  * Install and use [[http://rsnapshot.org|rsnapshot]] to create daily, weekly and monthly backups of the /etc, /home and /usr/local directories on your Linux container.
  * Rsnapshot is a simple utility. See its man page (//'man rsnapshot'//) for configuration syntax and cron job setup.
  * As root, edit the //rsnapshot// config file, ///etc/rsnapshot.conf//.
    * Note the file system location where backups will be stored (//snapshot_root//).
    * Note the file system directories that will be backed up (//"BACKUP POINTS / SCRIPTS"//).
      * The ///etc, /home and /usr/local// directories are already listed by default.
    * Change the //"BACKUP INTERVALS"// to the following to keep 7 daily, 4 weekly and 12 monthly backups:

<code>
# modified by jchung, 4/2018
# retain                hourly  6
retain  daily   7
retain  weekly  4
retain  monthly 12
</code>

  * Rsnapshot needs to use //cron// to perform scheduled backups. We will use root's cron table (crontab) to perform //rsnapshot// backups. Edit root's crontab by running //'sudo crontab -e'//, and add the following schedule to root's crontab:

<code>
# m h  dom mon dow   command
# rsnapshot jobs, jchung, 4/2018
0 0 * * *       /usr/bin/rsnapshot daily
0 1 * * 0       /usr/bin/rsnapshot weekly
0 2 1 * *       /usr/bin/rsnapshot monthly
</code>


==== 5. Install and configure logwatch ====

  * Install and configure the [[https://www.digitalocean.com/community/tutorials/how-to-install-and-use-logwatch-log-analyzer-and-reporter-on-a-vps|logwatch log analyzer and reporter]] on your Linux VM.
  * You should have installed and tested the [[cs_471_-_email_networked_file_systems_system_file_sharing#the_postfix_mta|Postfix MTA]] already on your VM.
  * You will configure logwatch and Postfix to email you daily log reports to your MU email account.
  * By default, //logwatch// runs once per day and emails a report to root. See the ///etc/cron.daily/00logwatch// script that runs //logwatch// once per day.
  * Configure logwatch or Postfix to send the email to you (student_id@monmouth.edu).
    * Do **ONE** of the following:
      * 1 - Modify ///etc/cron.daily/00logwatch// to use your student email address.
      * 2 - Or add a [[https://www.ccsf.edu/Pub/UNIXhelp/mail/file_forward.html|.forward]] file to your normal user home directory that contains your student email address.
  * Verify that //logwatch// sends you an email report once per day.

==== 6. Check for and mitigate Meltdown/Spectre vulnerability ====

On your VMs, check for and try to mitigate vulnerability to the [[https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)|Meltdown/Spectre]] vulnerability.

We will download and run a [[https://github.com/speed47/spectre-meltdown-checker|published vulnerability checking script]] (//spectre-meltdown-checker.sh//).

The kernel version you are currently running on your VMs is //3.16.0-4-amd64//, per the output of //'uname -a'//. The //spectre-meltdown-checker.sh// script will report that your VM is vulnerable to the //Meltdown// exploit. To mitigate the //Meltdown// vulnerability, you will need to upgrade to a newer kernel.

To upgrade to a new kernel, run the following commands to upgrade packages, including available kernel upgrades:

<code>
sudo apt-get update
sudo apt-get dist-upgrade
</code>

One of the packages will be a newer version of the kernel (//3.16.0-5-amd64//). To use the new kernel, you will need to reboot with

<code>
sudo reboot
</code>

After reboot, verify that your VM is running kernel version //3.16.0-5-amd64//, and then run the //spectre-meltdown-checker.sh// script again to verify that your system is no longer vulnerable to //Meltdown//.

----