WittCode💻

Monitor Server Authentication Activity (Grafana, Loki, Promtail, Docker Compose)

By

Monitor a server's authentication activity using the /var/log/auth.log file. Everything will be done with Grafana provisioning.

Table of Contents 📖

Authentication Monitoring

To monitor a Linux server's authentication activity, we can observe the /var/log/auth.log file. This file records authentication related events (such as login attempts) on Linux systems. This includes logins, privilege escalations, failed SSH attempts, etc. Therefore, we want to set up a real-time monitoring solution for this file.

The Stack

In this demonstration, we will monitor a Linux server's authentication activity using Promtail, Loki, and Grafana. All of these instances will be deployed using Docker Compose. Specifically, Promtail will watch the /var/log/auth.log file and send them to Loki, Loki will receive and store the logs, and then we will visualize them with Grafana dashboards and visualizations.

Environment Variables

To start, lets set up our environment variables to define the location of our containers in the Docker network. The variables starting with GF are reserved by the Grafana image for configuration. Specifically, they disable the login page and set out standard output log level.

PROJECT_NAME=GRAFANA_LOKI_PROMTAIL

LOKI_CONTAINER_NAME=l-c
LOKI_PORT=3100

PROMTAIL_CONTAINER_NAME=p-c
PROMTAIL_PORT=9080

GRAFANA_CONTAINER_NAME=g-c
GRAFANA_PORT=3000
GF_AUTH_ANONYMOUS_ENABLED=true
GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
GF_LOG_LEVEL=warn

Loki Configuration

This configuration file sets up Loki to store log data on the local filesystem. Authentication is disabled, allowing open access, and an HTTP server is configured to listen on the specified address and port. Log data is saved in the /loki directory. The file storage uses an in-memory key-value store for temporary storage, and schema settings define how logs are indexed and stored, rotating index files every 24 hours. This setup is designed for local, single-instance usage with basic storage and limited clustering or high-availability features.

INFO: Replication Factor of 1: There is only one copy of the data. If that node or storage location fails, the data could be lost. For single-node setups, a replication factor of 1 is often sufficient since no redundancy across nodes is possible.

# Enables or disables authentication for Loki, where 'X-Scope-OrgID' is required if set to true.
auth_enabled: false

# Configures Loki's HTTP server, defining the listening address and port for incoming requests.
server:
  http_listen_address: ${LOKI_CONTAINER_NAME}
  http_listen_port: ${LOKI_PORT}

# Sets limits on data handling, such as whether structured metadata is allowed within log entries.
limits_config:
  allow_structured_metadata: false

# Shared configuration across modules, including storage settings.
common:
  # Defines the base directory for Loki's files.
  path_prefix: /loki
  # Configures storage details, setting up directories for data chunks and rule files.
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  # Defines the number of replicas for redundancy; 1 indicates no additional replicas.
  replication_factor: 1
  # Configuration for distributed storage or clustering, here using an in-memory key-value store.
  ring:
    kvstore:
      store: inmemory

# Defines the schema and indexing configurations for stored log data.
schema_config:
  configs:
    - from: 2020-10-24  # Start date for this configuration.
      store: boltdb-shipper  # Storage method used for index data.
      object_store: filesystem  # Object storage type for log chunks.
      schema: v13  # Schema version to use.
      index:
        prefix: index_  # Prefix for index files.
        period: 24h  # Frequency of index file rotation.

Promtail Configuration

This Promtail configuration sets up a log collection service to monitor and read authentication logs from the /var/log/auth.log file on the local machine. Promtail saves its read positions in a temporary file (/tmp/positions.yaml) so that it resumes where it left off after restarts. It pushes these logs to a Loki instance specified by the clients section, allowing for centralized log storage and querying through Loki. Each scrape job includes labels, making it easier to filter logs by type (e.g., auth) in Loki or Grafana.

INFO: Promtail continuously reads logs from specified files (like /var/log/auth.log). As it reads, it records the last read position (or "offset") in the positions file. This prevents duplicate log entries in Loki by ensuring that when Promtail restarts, it resumes from where it left off.

# Configures Promtail's web server for health checks and status. 
server:
  http_listen_address: ${PROMTAIL_CONTAINER_NAME}   # Sets the server's listening address.
  http_listen_port: ${PROMTAIL_PORT}                # Sets the server's listening port for HTTP requests.

# Specifies where Promtail will store its reading position for each log file. 
positions:
  filename: /tmp/positions.yaml                     # File storing the last read position for each log file.

# Defines the Loki client configuration, specifying where Promtail will push logs.
clients:
  - url: http://${LOKI_CONTAINER_NAME}:${LOKI_PORT}/loki/api/v1/push # Loki endpoint for receiving logs.

# Configures which log files Promtail should scrape and any relevant metadata.
scrape_configs:
  - job_name: auth                    # The name for this scrape job, usually descriptive of the log type.
    static_configs:
      - targets:                      # Lists the scrape targets; typically 'localhost' for local log files.
          - localhost
        labels:
          job: auth                   # Assigns a label identifying the job for easier querying in Loki.
          __path__: /var/log/auth.log # Specifies the file path of the logs to be scraped.

Grafana Provisioning

This Grafana provisioning configuration file defines a datasource to connect Grafana to Loki. It sets Loki as the log data source, configures it to proxy connections through the Grafana server, and points to the Loki instance using a URL with environment variables. Additional settings in jsonData specify a 60-second query timeout and a maximum retrieval limit of 1000 log lines per query.

INFO: The maxLines field in the Grafana provisioning file limits the maximum number of log lines returned by a single query from Loki to Grafana. This setting helps prevent excessive data retrieval in cases where queries return large amounts of logs, which can slow down the Grafana dashboard, consume unnecessary resources, and increase load times.

# Specifies the API version for Grafana's provisioning configuration.
apiVersion: 1

datasources:
  # Defines a datasource with its configuration settings.
  - name: Loki                    # The display name for this datasource in Grafana.
    type: loki                    # Specifies the datasource type as Loki.
    access: proxy                 # Sets the access mode to 'proxy', routing requests through the Grafana server.
    url: http://${LOKI_CONTAINER_NAME}:${LOKI_PORT}   # URL for the Loki instance, with dynamic environment variables.

    # Additional JSON configuration for this Loki datasource.
    jsonData:
      timeout: 60                 # Timeout setting in seconds for queries sent to Loki.
      maxLines: 1000              # Limits the maximum number of log lines to retrieve in a single query.

Docker Compose Services

Here we define our multi-container setup for monitoring and analysis of authentication logs, making it easier to track and visualize server activity.

  • Loki stores the authentication logs and is configured through an external YAML file.
  • Promtail collects logs from the system (in this case, specifically from the /var/log/auth.log file) and forwards them to Loki. It is configured using its own YAML file.
  • Grafana provides a user interface for visualizing the logs stored in Loki, with its data sources defined in an external configuration file.

WARNING: The expose key is for communication between services in the same network while ports makes the service accessible to the outside world.

# Project name placeholder that can be replaced by the actual project name.
name: ${PROJECT_NAME}

services:

  # Loki service for log storage and retrieval.
  loki:
    pull_policy: always  # Always pull the latest image for the Loki service.
    image: grafana/loki  # Docker image for Loki.
    container_name: ${LOKI_CONTAINER_NAME}  # Name of the container for easier reference.
    env_file: .env  # Loads environment variables from the .env file.
    volumes:
      - ./monitoring/local-config.yaml:/etc/loki/local-config.yaml  # Mounts the local Loki config file.
    expose:
      - ${LOKI_PORT}  # Exposes the specified port for communication with other services.
    command:
      - -config.expand-env=true  # Expands environment variables in the configuration file.
      - -config.file=/etc/loki/local-config.yaml  # Specifies the configuration file for Loki.

  # Promtail service for log collection and forwarding to Loki.
  promtail:
    pull_policy: always  # Always pull the latest image for the Promtail service.
    image: grafana/promtail  # Docker image for Promtail.
    container_name: ${PROMTAIL_CONTAINER_NAME}  # Name of the container for Promtail.
    env_file: .env  # Loads environment variables from the .env file.
    volumes:
      - ./monitoring/promtail.yaml:/etc/promtail/config.yml  # Mounts the local Promtail config file.
      - /var/log/auth.log:/var/log/auth.log  # Mounts the auth.log file for log scraping.
    expose:
      - ${PROMTAIL_PORT}  # Exposes the specified port for communication with other services.
    depends_on:
      - loki  # Ensures Loki is started before Promtail.
    command:
      - -config.expand-env=true  # Expands environment variables in the configuration file.
      - -config.file=/etc/promtail/config.yml  # Specifies the configuration file for Promtail.

  # Grafana service for visualizing logs stored in Loki.
  grafana:
    pull_policy: always  # Always pull the latest image for the Grafana service.
    image: grafana/grafana  # Docker image for Grafana.
    container_name: ${GRAFANA_CONTAINER_NAME}  # Name of the container for Grafana.
    env_file: .env  # Loads environment variables from the .env file.
    volumes:
      - ./monitoring/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yml  # Mounts the data sources configuration.
    ports:
      - ${GRAFANA_PORT}:${GRAFANA_PORT}  # Maps the specified port for access to the Grafana UI.
    depends_on:
      - loki  # Ensures Loki is started before Grafana.

Running the Application

To run the application, simply navigate to the top level and run docker compose up.

docker compose up

Now navigate to localhost:${GRAFANA_PORT} in your browser, go to datasources, click on Loki, and explore the authentication data.

Monitor Server Authentication Activity (Grafana, Loki, Promtail, Docker Compose)