kolo docsbeta

Monitoring a pinning peer

Read the metrics of a kolo-pin pinning peer with Prometheus, keep the endpoint private, and get an alert before its storage fills up.

On this page

The kolo-pin command serves metrics in the Prometheus text format. They tell you how full the pinning peer is, how busy it is and why it refuses requests. This page starts from the peer of Run a pinning peer on a VPS.

The metrics endpoint

kolo-pin run answers GET /metrics on 127.0.0.1:7452 by default. The flag is -metrics and its environment variable is KOLO_PIN_METRICS; a flag wins over its variable. Set it to off to turn the endpoint off. It is a separate listener from the health endpoint, GET /healthz on 127.0.0.1:7451.

The address is a TCP address, IPv4 only. An IPv6 address is refused and kolo-pin exits with code 2. At start, kolo-pin logs where it listens:

Text
time=2026-11-12T08:16:33.990+07:00 level=INFO msg=metrics addr=127.0.0.1:7452

On the server, ask it for the main numbers:

bash
curl -s http://127.0.0.1:7452/metrics | grep -E '^kolo_pin_(bytes|disk|owners|logs|entries|requests)'

This is what it printed right after one owner pinned a log of four entries:

Text
kolo_pin_bytes_max 1e+11
kolo_pin_bytes_received_total 2040
kolo_pin_bytes_served_total 0
kolo_pin_bytes_used 3081
kolo_pin_disk_free_bytes 1.73485133824e+11
kolo_pin_entries 4
kolo_pin_logs 1
kolo_pin_owners 1
kolo_pin_requests_total{op="add"} 1
kolo_pin_requests_total{op="remove"} 0
kolo_pin_requests_total{op="revoke"} 0
kolo_pin_requests_total{op="status"} 1
kolo_pin_requests_total{op="sync_pull"} 0

The storage and request metrics appear once the store is open, around the serving log line. Counters start again at 0 each time kolo-pin starts.

A Go program that runs the library itself gets the same kolo_pin_* metrics by passing a prometheus.Registerer in Options.Metrics, and serves them as it likes.

The metrics

  • kolo_pin_bytes_used (gauge): bytes stored for all owners, keys and values.
  • kolo_pin_bytes_max (gauge): the -max-bytes limit.
  • kolo_pin_disk_free_bytes (gauge): free bytes on the filesystem of the data folder, on Linux, macOS and Windows only.
  • kolo_pin_owners (gauge): owners with a capability in use or a hosted log.
  • kolo_pin_logs (gauge): hosted logs.
  • kolo_pin_entries (gauge): entries of all hosted logs.
  • kolo_pin_requests_total (counter, label op: add, remove, status, revoke, sync_pull): requests answered; sync_pull counts the kolo-sync streams other peers open on a hosted log: a new reader opens two (header, then pull), and FetchHead and Fetch open one each.
  • kolo_pin_refused_total (counter, label reason: bad_token, expired_token, replayed_token, revoked, quota_exceeded, disk_full, oversized_entry, bad_signature, bad_proof, other): requests refused.
  • kolo_pin_delete_markers_applied_total (counter): owner delete markers applied, each dropping one ciphertext the peer held. An entry that arrives already deleted is not counted.
  • kolo_pin_bytes_received_total (counter): bytes of new entries (records and ciphertexts) pulled from owners and stored.
  • kolo_pin_bytes_served_total (counter): bytes kolo-sync wrote to peers pulling hosted logs, every frame included.
  • kolo_pin_active_connections (gauge): open kolo-secure connections.
  • kolo_pin_active_sync_streams (gauge): kolo-sync streams running now, those served to pulling peers plus the pulls from owners.
  • kolo_pin_active_pulls (gauge): of those, the pulls from owners running now.
  • kolo_pin_sync_pull_duration_seconds (histogram): how long each pull from an owner takes.
  • kolo_pin_request_duration_seconds (histogram, label op: add, remove, status, revoke): how long each pin request takes. kolo-sync streams (sync_pull) are not timed, so that label is not here.
  • kolo_pin_build_info (gauge, labels version, go_version): always 1; the labels name the build.
  • kolo_pin_uptime_seconds (gauge): seconds since the program started.

Both histograms use buckets of 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60 and 300 seconds. Every op and reason value is there from the start, at 0, so an alert sees the first refusal. The command also exports the standard go_* runtime metrics, and process_* metrics on Linux and Windows.

What the metrics reveal

They show totals for the whole peer: bytes, owners, logs, entries, request and refusal counts by kind, durations, and the build.

They never hold an owner key, a log ID, a peer key, an IP address or any number for one owner. Labels take only the fixed values listed above, so nothing a peer sends ever becomes a label.

Totals still say something. Anyone who reads the endpoint learns how many owners and logs you host, how much you store, and how often and why requests are refused. Keep it private. The logs of kolo-pin hold more (public keys, log IDs, addresses), as Run a pinning peer on a VPS explains.

Scrape it with Prometheus

If Prometheus runs on the VPS, point it at the loopback address. This prometheus.yml scrapes kolo-pin every 30 seconds and loads the alert rules further down:

prometheus.yml
rule_files:
  - kolo-pin-rules.yml

scrape_configs:
  - job_name: kolo-pin
    scrape_interval: 30s
    static_configs:
      - targets: ["127.0.0.1:7452"]

If Prometheus runs on your own computer, open an SSH tunnel to the VPS instead of opening a port:

Terminal
ssh -L 7452:127.0.0.1:7452 user@203.0.113.10

While the tunnel is open, the same config works on your computer, since 127.0.0.1:7452 there now leads to the VPS. When the tunnel closes, the scrapes fail and the last alert below fires.

Set it in systemd

The default already listens on 127.0.0.1:7452. To make it explicit, or to change it, add this line to the [Service] section of the unit from the VPS guide, next to the other Environment= lines:

/etc/systemd/system/kolo-pin.service
[Service]
Environment=KOLO_PIN_METRICS=127.0.0.1:7452

Use off as the value to turn the endpoint off. Then reload and restart:

bash
sudo systemctl daemon-reload
sudo systemctl restart kolo-pin

journalctl -u kolo-pin shows the metrics line with the address.

Alert rules

Three rules in Prometheus format, for the file kolo-pin-rules.yml that the config above loads:

kolo-pin-rules.yml
groups:
  - name: kolo-pin
    rules:
      - alert: KoloPinStorageAlmostFull
        expr: kolo_pin_bytes_used / kolo_pin_bytes_max > 0.9
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "kolo-pin on {{ $labels.instance }} uses over 90% of -max-bytes"
      - alert: KoloPinRefusingDiskFull
        expr: increase(kolo_pin_refused_total{reason="disk_full"}[10m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "kolo-pin on {{ $labels.instance }} refuses new data: disk full"
      - alert: KoloPinNoMetrics
        expr: up{job="kolo-pin"} == 0 or absent(kolo_pin_bytes_used{job="kolo-pin"})
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "No kolo-pin metrics for 5 minutes"
  • The first fires when the peer has used 90% of -max-bytes for 15 minutes. Raise -max-bytes (if the disk has room), or list your owners.
  • The second fires when Add returned pin: pinning peer disk full (ErrDiskFull) at least once in the last 10 minutes: owners can no longer add new data.
  • The third fires when no scrape has worked for 5 minutes. up == 0 covers a stopped peer or a closed tunnel, and absent(...) covers a target that is missing or answers without the kolo-pin metrics.

-max-bytes counts stored keys and values, not disk blocks, so the real disk can fill first. Watch kolo_pin_disk_free_bytes as well.

Keep it off the public address

kolo-pin never listens for metrics on a public address unless you give it one. The endpoint has no authentication and no TLS.

What was not tested

Every run behind this page was on one computer, on 127.0.0.1: the build, kolo-pin run with the default addresses, curl of /healthz and /metrics before and after the owner program of the VPS guide pinned a log, -metrics off and the IPv6 refusal. Prometheus, the scrape config and the alert rules (only checked to be valid YAML), the SSH tunnel, the systemd unit, a reverse proxy and a Linux server were not tested.

Next steps