Samhain Services | The Basics

Standalone Configuration

Samhain (standalone)

Monitors local host
Uses local baseline database
Reports to (e.g.) syslog, log file, or email.

For larger installations, you may want to consider a Basic Client/Server or a Full installation.

Basic Client/Server Configuration

Yule Server

Part of the free (open-source) samhain software.
Receives and logs incoming reports from samhain clients
Keeps track of client status (active/inactive)
Serves baseline databases and runtime configuration files (stored server-side) to starting clients.

Samhain Client A

Monitors local host A
Retrieves runtime configuration and baseline database from Yule server
Reports via encrypted connection to Yule server (and optionally to e.g. syslog, log file, or email).

Samhain Client B

Monitors local host B
...
...

Samhain Client C

Monitors local host C
...
...

Full Client/Server Configuration

Database

MySQL, PostgreSQL, or Oracle.
Yule server stores reports from Samhain clients here
Status of reports is tracked.

Beltane II

Commercial Web based Management Console.
Easily view or search reports from Samhain clients.
Perform server-side updates of client baseline databases.
View status of Yule server and Samhain clients.

Yule Server

Part of the free (open-source) Samhain software to database.
Receives and logs incoming reports from Samhain clients
Keeps track of client status (active/inactive)
Serves baseline databases and runtime configuration files (stored server-side) to starting clients.

Samhain Client A

Monitors local host A
Retrieves runtime configuration and baseline database from Yule server
Reports via encrypted connection to Yule server (and optionally to e.g. syslog, log file, or email).

Samhain Client B

Monitors local host B
...
...

Samhain Client C

Monitors local host C
...
...

Supported Platforms

POSIX (e.g. Linux, *BSD, Solaris 2.x, AIX 5.x, HP-UX 11, and Mac OS X)
Windows 2000 / WindowsXP / Windows 7 with POSIX emulation (e.g. Cygwin). Please note that this is tested for the Samhain monitoring agent only, not for the server.

PCI DSS Compliance

The Payment Card Industry (PCI) Data Security Standard (DSS) mandates the use of file integrity monitoring software. Version 1.1 of the PCI DSS includes the following two requirements:

Sect. 11.5: "Deploy file integrity monitoring software to alert personnel to unauthorized modification of critical system or content files; and configure the software to perform critical file comparisons at least weekly."

Sect. 10.5.5: "Use file integrity monitoring and change detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert)."

While weekly checks can be done with any open-source file integrity checker, to the best of our knowledge Samhain is the only one that can perform incremental checks on growing logfiles (i.e. verify at each check that the data present at the preceding check have not been modified meanwhile), as required by Sect. 10.5.5 of the PCI DSS.

Centralized Management

Samhain can be used standalone on a single host, but its particular strength is centralized monitoring and management. The complete management of a samhain system can be done from one central location. To this end, several components are required.

A full samhain client/server system is built of the following components:

The samhain file/host integrity checker

This is the client/agent on the monitored host(s). It is designed to run as a daemon. This avoids repetitive warnings, because the daemon keeps a memory of file changes. However, if you prefer, it can also be invoked from cron.

The yule log server

Yule collects and logs reports from samhain clients on remote (or the local) hosts.
Yule allows samhain clients to download baseline databases and runtime configuration files (if stored on the server) at startup.
Yule keeps track of the status of clients, and can inform you if a client seems to be dead.
If you have edited a client's runtime configuration file on the server, yule can advise the client to reload the configuration file.

A relational database

The server will store reports from clients in this database. Oracle, MySql, or PostgreSQL are supported. While a database is in general not required, it is neccessary for the next component.

The Beltane II web-based console

Beltane II is a PHP application available as separate package.
Beltane II will pull reports from the database, and present them for review.
It then allows the user to update the client's baseline databases stored on the server, to reflect the file system changes reported by the client.
Additionally, Beltane II gives access to the client status as known by the server, and to the server interface for asking clients to reload their configuration.

File Integrity

Immediate notifications, reduced I/O load On Linux, Samhain 3.0+ can leverage the inotify mechanism to monitor file system events. This allows to receive immediate notifications about changes, and eliminates the need for frequent file system scans which may cause a high I/O load.
Checksum (TIGER192, SHA-1, SHA-256, or MD5), size, mode/permission, owner, group, timestamp (creation/modification/access), inode, number of hardlinks, major/minor device number (devices only), and linked path (symbolic links only) can be checked.
Samhain can also check more 'exotic' properties like: SELinux attributes (on Linux), POSIX ACLs (on systems supporting them), Linux ext2 file attributes (as set by chattr, e.g. the immutable flag), and the BSD file flags.
Who did it? On Linux, Samhain can leverage the Linux kernel audit system to determine which login user modified a file. (Please note that in general, i.e. on arbitrary POSIX operating systems, this is not possible to do. The required information is usually not saved by the operating system.)
Correct number of hardlinks for directories can be checked (this can reveal the presence of subdirectories hidden by kernel rootkits).
For small files (less than 9200 bytes after zlib compression), it is optionally possible to store the full file in the baseline database, such that it is possible to find out what has been changed.
Shell wildcard patterns (globbing) can be used to specify files and/or directories to check.
With the 'userdir' extension compiled in, paths can be specified relative to the home directories of all users within some range of UIDs (i.e. it's possible to say 'check ~/.profile for UIDs in the range N to M').
On Linux, prelink can be supported transparently (i.e. no spurious warnings after re-prelinking).

Scheduling of file checks

File checks can be performed at user-defined intervals.
Alternatively, a crontab-style schedule can be used to perform file checks at user-defined times.
It is possible to configure two different schedules, to check some files or directories more frequently then others.
File checks can be started anytime by sending a signal to the Samhain daemon.

Host Integrity Monitoring

Samhain is extensible by modules that can be compiled in at the users' discretion. The following list shows which modules are currently available. The documentation contains a HOWTO for creating additional modules.

Logfile monitoring/analysis

As of version 2.5.0, samhain optionally perform logfile monitoring/analysis. Currently supported formats are UNIX syslog, Apache (and compatible) access and error logs, Samba logfiles, and BSD-style process accounting logs.
Both whitelisting and blacklisting policies are supported. Perl-style regular expression can be used to match logfile entries.
Samhain supports checking for correlated events, for missing heartbeat messages, and automatic detection of bursts of repeated messages.

Windows registry check

On Windows/Cygwin, it is possible to check the integrity of individual keys (or hierarchies of keys) in the registry.

SUID/SGID files

Samhain can optionally check the filesystem for new SUID/SGID files.
This check can be scheduled independently from the regular file check.
Optionally, samhain can strip SUID/SGID permissions from new SUID/SGID files, or quarantine or delete them. By default, it will just report them.

Open ports

Samhain can optionally monitor which ports are open on the local host, and compare agains a list of allowed or required port/services.
On Linux and FreeBSD, also the program having the port open will be reported.

Process check

Samhain can optionally check for processes that are hidden (i.e. not listed in the output from 'ps'), or fake (i.e. listed by 'ps', but non-existent). Additionally, it is possible to check for the existence of (user-defined) required processes.

Mount check

Samhain can optionally monitor the presence and mount options of mounted filesystems.

Login/logoff events

Samhain can optionally monitor and report login/logoff events.
This check uses the system utmp file.

Logging Options

The verbosity and on/off status of each log facility can be configured individually.

Central log server. Messages are sent via encrypted TCP connections. Clients need to authenticate to the server.
Syslog.
Console (if daemon) / stderr.
Log file. To prevent unauthorized modifications of existing log records, the log file entries are signed.
E-mail (built-in mailer). E-mail reports are signed to prevent tampering.
RDBMS (currently MySQL, PostgreSQL, and Oracle are supported; support for unixODBC is untested).
Execute external program — this can be used to implement arbitrary additional logging facilities, or to perform active response to events.

Integration with other systems

Prelude

Samhain can be compiled against the libprelude library, which will enable it to function as a prelude sensor.

Nagios

A Perl plugin for Nagios (check_samhain.pl) is supplied as part of the samhain distribution.

Generic interfaces

Named pipe: samhain can write log messages to a named pipe.
Message queue: log messages can be provided on a message queue (SystemV IPC)
External programs can be executed from samhain. The log message is supplied on standard input.

Active response

Samhain can execute external programs upon user-defined events, and supply the corresponding log message to them. This can be used to:

Implement additional logging facilities, or
Provide active response (e.g. rebooting the machine, reconfiguring the firewall, ...).

Security

There is always a trade-off between security and convenience, and thus you may want to keep your file checking executable on disk and hope that an intruder will not tamper with it.
Samhain offers the following features to help protecting its integrity:

signed database and configuration file: Both the file signature database and the configuration file can be signed with GnuPG.
embedded password: For successful connection to the server, a password is required which is embedded into the executable itself (this password can be set exactly once after compiling the executable).
compiled-in key: Every executable built from source contains a unique random 64-bit key (unless this key is defined by the user at compile time). Logfile/e-mail messages generated by a different executable (with a different key) will not pass the verification routine with a 'known good' executable.
daemon mode: Samhain can run continuously as a daemon (background process), and any stop/restart process will leave a recognizable mark. Thus it is not possible to "slip in" a rogue executable as long as the daemon is running.
signed reports: eMail reports and messages written to the log file are signed. Messages sent to a central log server are additionally encrypted with Rijndael (AES).
stealth: what an intruder cannot see, s/he will not try to subvert. Samhain offers several options to hide itself.

Documentation

The manual in PDF and HTML format is available online on the Samhain development site. It includes explanations and examples for the setup, complete documentation of the format of the configuration file, and of the interface to external programs. There are also several HOWTOs for specific issues (e.g. the client/server setup).

This documentation is also included in the distribution tarball, along with man pages for the program and the configuration file, and sample configurations for Linux, FreeBSD, and Solaris.

Change Control Process Integration

Production machines have scheduled changes
...and those should not raise alerts

In most production environments, there are more or less frequent changes owing to a number of reasons, such as:

the installed software needs to be patched for security vulnerabilities.
new software packages have been requested by users, or
configuration files need to changed.

These changes can make the use of file integrity monitoring very awkward, especially if the environment is volatile or the number of monitored systems is large, unless by integration into the change control process one can avoid alerts for scheduled changes.

Samhain can integrate into your change control process

When we say »integrate into your change control process« we mean that Samhain provides the required feature set to meet the following list of criteria.

The whole procedure can be run in an automated way, i.e. it can be executed by scripts and without human intervention, once a list of affected files is available.
- It is expected that the change control process will yield the list of affected files from the development or quality assurance stage.
The feature set provided by Samhain is rather generic and not tied to any particular change control software.
Immediately before a change is implemented, the affected files can be tested for their integrity. I.e. it can be verified that the system is in a »known-good« state before the change is put into effect.
After a change is implemented, the baseline, i.e. the database of »known-good« file signatures, can be updated.
The approval of the change(s) performed can be securely communicated to the running Samhain daemon on the affected machine(s), such that a subsequent file integrity check will raise no alerts.

Use cases

The feature set provided by Samhain has been drafted as the result of a workshop with the participation of iSAX, among others. The use cases that were discussed are as follows:

Case I: Machine taken offline for a large patch

This case is best handled by a comple re-initialisation of the baseline database. The running Samhain daemon would perform an on-demand file system scan immediately before the machine is taken offline to ensure a valid state, the database would be initialized after the patch has completed, and the Samhain daemon would re-start when the machine goes online again.

Show me the details

Case II: Installation of a new package

In this case, because the package is not installed yet a "pre-flight" scan may be deemed unneccessary. After installation of the package, a »DeltaDB« (delta database) containing the added files would be generated and transferred to the server. There, it would be merged into the existing baseline database.

The approval of the file system changes would then be done by the server asking the client to download the DeltaDB. For security, there is no client-side mechanism to trigger an approval of the file system changes.

Show me the details

Case III: Configuration change / Package upgrade

This case differs from Case II insofar as there are already installed files, and therefore it is desirable to verify the integrity of those files before the change is put into effect. To perform this check, one would generate on the server a »PartialDB« (partial database, containing only data for affected files) from the full baseline database, transfer it to the client and perform a verification scan.

If the affected files are found to be in a consistent state, one would proceed as in Case II then.

Show me the details

Standalone Configuration

Samhain (standalone)

Change Control Process Integration

Production machines have scheduled changes ...and those should not raise alerts

Samhain can integrate into your change control process

Use cases

Production machines have scheduled changes
...and those should not raise alerts