PrometheusのHTTP Service Discoveryを試す

PrometheusのHTTP Service Discoveryを試したときのメモ。

準備

Prometheus

DockerのPrometheusとNode exporter２台を使う。

prometheus.yml

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['host.docker.internal:9090']
  - job_name: http-sd
    http_sd_configs:
      - url: 'http://host.docker.internal:8080/http-sd'

compose.yml

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  node-exporter1:
    image: quay.io/prometheus/node-exporter:latest
    ports:
      - "9101:9100"
  node-exporter2:
    image: quay.io/prometheus/node-exporter:latest
    ports:
      - "9102:9100"

PromethusとNod exporterを起動する。

% docker compose up

HTTP SD endpoint

HTTP SD endpointはspring bootで実装する。

build.gradle

plugins {
    id 'java'
    id 'org.springframework.boot' version '3.0.2'
    id 'io.spring.dependency-management' version '1.1.0'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '17'

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'io.micrometer:micrometer-registry-prometheus'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
}

tasks.named('test') {
    useJUnitPlatform()
}

application.yml

management:
  endpoints:
    web:
      exposure:
        include: prometheus

HttpSdController.java

ドキュメントに記載されているフォーマットで、Node exporter２台とこのspring bootアプリの情報を返す。

import java.util.List;
import java.util.Map;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HttpSdController {
    private static final Logger logger = LoggerFactory.getLogger(HttpSdController.class);
    public record HttpSd(List<String> targets, Map<String, String> labels) {}

    @GetMapping("/http-sd")
    public List<HttpSd> httpSd(@RequestHeader("X-Prometheus-Refresh-Interval-Seconds") String refreshInterval) {
        logger.info("Refresh-Interval-Seconds: {}", refreshInterval);
        return List.of(new HttpSd(List.of("host.docker.internal:9101",
                                          "host.docker.internal:9102"),
                                  Map.of("job", "node-exporter")),
                       new HttpSd(List.of("host.docker.internal:8080"),
                                  Map.of("job", "spring-boot",
                                         "__metrics_path__", "/actuator/prometheus")));
    }
}

具体的にはこんなJSONを返す。

[
  {
    "targets": [
      "host.docker.internal:9101",
      "host.docker.internal:9102"
    ],
    "labels": {
      "job": "node-exporter"
    }
  },
  {
    "targets": [
      "host.docker.internal:8080"
    ],
    "labels": {
      "job": "spring-boot",
      "__metrics_path__": "/actuator/prometheus"
    }
  }
]