WittCode💻

Get System Logs with Promtail Loki Grafana

By

Learn how to get system logs with Promtail, Loki, and Grafana using Docker Compose. Each instance will be in a Docker container and will receive a custom configuration.

Table of Contents 📖

The Stack

  • Promtail - Collects logs using its scraping settings. Transports logs to Loki for storage and analysis.
  • Loki - A data store for logs. Makes log analysis and retrieval a breeze.
  • Grafana - Grafana visualizes the logs stored in Loki, presenting the data on a user friendly dashboard.

Environment Variables

The environment variables starting with GF are reserved by the Grafana image for configuration. Here we are using them to disable to login page and set the log level to warning and above. The other variables set server locations.

PROJECT_NAME=GRAFANA_LOKI_PROMTAIL

LOKI_CONTAINER_NAME=l-c
LOKI_PORT=3100

PROMTAIL_CONTAINER_NAME=p-c
PROMTAIL_PORT=9080

GRAFANA_CONTAINER_NAME=g-c
GRAFANA_PORT=3000
GF_AUTH_ANONYMOUS_ENABLED=true
GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
GF_LOG_LEVEL=warn

Loki

Here, lets configure the Loki server location and set the storage location for logs to be the local file system.

# Enables authentication through the X-Scope-OrgID header, which must be present if true.
auth_enabled: false

# Configures the server.
server:
  http_listen_address: ${LOKI_CONTAINER_NAME}
  http_listen_port: ${LOKI_PORT}

# Common configuration to be shared between multiple modules.
common:
  path_prefix: /loki
  # Where to store the logs. Can store logs in S3, GCS, filesystem, etc.
  storage:
    # Locations to store chunks and rules in.
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

INFO: Note that the ${VAR_NAME} are environment variables. They will be replaced by Docker Compose.

Promtail

Now lets set up Promtail to scrape for logs in the /var/log directory on the local machine. We will then push these logs to Loki.

server:
  http_listen_address: ${PROMTAIL_CONTAINER_NAME}
  http_listen_port: ${PROMTAIL_PORT}

# Configures where Promtail will save a file indicating how far it has read into a file
positions:
  # Location of positions file
  filename: /tmp/positions.yaml

# The clients block configures how Promtail connects to instances of Loki
clients:
  # The URL where Loki is listening, denoted in Loki as http_listen_address and http_listen_port.
  - url: ${LOKI_CONTAINER_NAME}:${LOKI_PORT}/loki/api/v1/push

# Configures how Promtail can scrape logs from a series of targets using a specified discovery method. 
# Promtail uses the same Prometheus scrape_configs.
scrape_configs:
- job_name: system
  # Static targets to scrape.
  static_configs:
  - targets:
      # Configures the discovery to look on the current machine.
      # localhost is also the default value.
      - localhost
    # Defines a file to scrape and an optional set of additional labels to apply to
    # all streams defined by the files from __path__.
    labels:
      job: varlogs
      # The path to load logs from.
      __path__: /var/log/*log

Grafana

Now lets set up Loki as a datasource for Grafana using Grafana's provisioning system. Provisioning is when you push configurations into a Grafana instance with conifugration files.

apiVersion: 1

datasources:
  # The datasource name. How it is refered to in panels and queries.
  - name: Loki
    # Sets the datasource type
    type: loki
    # Sets the access mode, either proxy or direct (Server or Browser in the UI).
    access: proxy
    # URL to the datasource, including the port
    url: http://${LOKI_CONTAINER_NAME}:${LOKI_PORT}
    # Fields to convert to JSON and store in jsonData.
    jsonData:
      timeout: 60
      maxLines: 1000

Docker Compose Services

Now lets spin up separate Docker containers for Grafana, Loki, and Promtail using Docker Compose. We will also use Docker Compose to load the configurations into the containers.

name: ${PROJECT_NAME}

#  Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

services:

  loki:
    pull_policy: always
    image: grafana/loki
    container_name: ${LOKI_CONTAINER_NAME}
    env_file: .env
    volumes:
      - ./configs/local-config.yaml:/etc/loki/local-config.yaml
    ports:
      - ${LOKI_PORT}:${LOKI_PORT}
    command:
      - -config.expand-env=true
      - -config.file=/etc/loki/local-config.yaml

  grafana:
    pull_policy: always
    image: grafana/grafana
    container_name: ${GRAFANA_CONTAINER_NAME}
    env_file: .env
    volumes:
      - ./configs/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    ports:
      - ${GRAFANA_PORT}:${GRAFANA_PORT}
    depends_on:
      - loki

  promtail:
    pull_policy: always
    image: grafana/promtail
    container_name: {PROMTAIL_CONTAINER_NAME}
    env_file: .env
    volumes_from:
      - nginx
    volumes:
      - ./configs/promtail.yaml:/etc/promtail/config.yml
    ports:
      - ${PROMTAIL_PORT}:${PROMTAIL_PORT}
    depends_on:
      - loki
    command:
      - -config.expand-env=true
      - -config.file=/etc/promtail/config.yml

ERROR: Note that we have no data persistence in these containers. To add this, we would need to create Docker volumes mapped to specific directories in each container.

Running the Application

Now we can simply spin up the containers with Docker Compose.

docker compose up

Navigating to http://localhost:3000/connections/datasources will display the Grafana Loki datasource and navigating to http://localhost:9080/targets will display our Promtail targets. To better demonstrate what is going on, lets go inside the Promtail container and create another log file.

docker exec -it p-c bash
echo "hi how are you" > /var/log/access.log

Now navigate to http://localhost:3000/connections/datasources, click on "Explore" in the Loki datasource, and create a label filter for where the filename is equal to access.log. Click "Run query" and look at the data output.

Get System Logs with Promtail Loki Grafana