Comprehensive Guide to Logging and Monitoring with Grafana, Loki, and OpenTelemetry in Docker

Setting Up Robust Logging and Monitoring with Grafana, Loki, and OpenTelemetry in Docker

Modern applications demand robust solutions for logging and monitoring to ensure reliability and performance. In this guide, we’ll explore how to set up a complete local monitoring stack using Grafana, Loki and OpenTelemetry, all running in Docker. This includes advanced logging configurations in .NET with Serilog and NLog and distributed tracing and real-time metrics monitoring.

Why Loki + Grafana for Logging?

Loki is a log aggregation system designed to work seamlessly with Grafana. Unlike traditional logging solutions like Elasticsearch, Loki focuses on efficiently indexing metadata, making it lightweight and cost-effective.

Prerequisites

1. Docker and Docker Compose installed on your system.
2. Basic understanding of Docker, .NET, and logging frameworks.

Setting Up the Monitoring Stack with Docker

We’ll deploy Loki and Grafana using Docker Compose.

Step 1: Create a docker-compose.yaml
Create a file named docker-compose.yaml

version: '3.8'

services:
  loki:
    image: grafana/loki:2.9.1
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-data:/loki
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml

  grafana:
    image: grafana/grafana:9.5.0
    container_name: grafana
    ports:
      - "3000:3000"
    depends_on:
      - loki
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Note: Please create a directory namedloki-data in the same directory where docker-compose.yaml will create.

Step 2: Loki Configuration
Create a loki-config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100
  
ingester:
  wal:
    enabled: true
    dir: /loki/wal
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  max_transfer_retries: 0

storage_config:
  boltdb:
    directory: /loki/index
  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Step 3: Start the Stack
Run the following command to start all services:

docker-compose up -d

Note: please use docker-compose down to stop.

• Grafana: Accessible at http://localhost:3000, default login: admin/admin
• Loki: Runs on port 3100

Logging in .NET with Serilog and Loki

We will use Serilog for structured logging and send logs to Loki.

Step 1: Install Dependencies
Run the following commands in your .NET project:

dotnet add package Serilog
dotnet add package Serilog.Sinks.Grafana.Loki

Step 2: Configure Serilog
Update your Program.cs:

using Serilog;
using Serilog.Sinks.Grafana.Loki;

var logger = new LoggerConfiguration()
    .WriteTo.GrafanaLoki("http://localhost:3100", new List<LokiLabel>
    {
        new() { Key = "app", Value = "web_app" }
    }, ["app"])
    .CreateLogger();

Log.Logger = logger;

try
{
    for (var i = 0; i < 100; i++)
    {
        Log.Information("Information log {i}", i);
        await Task.Delay(100);
        Log.Error("Error Log {i}", i);
        await Task.Delay(100);
        Log.Debug("Debug log {i}", i);
        await Task.Delay(100);
        Console.WriteLine($"Log send {i}");
    }
}
catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

Adding Grafana Loki Datasource

1. Open Grafana at http://localhost:3000.
2. Login using admin/admin.
3. Go to Configuration > Data Sources > Add Data Source.
4. Choose Loki and set the URL to http://loki:3100.
5. Save & Test.

Visualizing Logs and Metrics in Grafana

1. Logs:
   * Create a dashboard and add a Loki panel.
   * Query: {job=”varlogs”} to view logs.
2. Metrics:
   * Add a Prometheus data source pointing to http://localhost:9090
   * Create panels for custom metrics (e.g., HTTP requests).

Conclusion

With Loki, Grafana and Serilog, you have a powerful stack for local logging and monitoring. Adding Prometheus and OpenTelemetry further enhances your observability. This setup not only simplifies debugging but also ensures your applications remain performant and reliable.