XMRig monitoring, part 3: logs, alerts and troubleshooting

XMRig monitoring series

1. Collect and store metrics
2. Read the dashboards
3. Logs, alerts and troubleshooting

Part 3 adds logs and alerting to the metrics pipeline built in Part 1. It shows how to distinguish a miner failure from stale or missing telemetry, test notification delivery, and investigate the symptoms shown in the dashboard walkthrough in Part 2.

Use the same miner identity, data-source UIDs and authenticated forwarding path established in Part 1. The configuration bundle includes the service examples, alert rules and validation fixtures used across the series. These examples have not been deployed end to end on a fresh host.

Collect XMRig logs that answer a specific question

Metrics narrow the time range. Logs help explain what happened inside it. For the 16:00 hashrate dip shown in Part 2, useful next checks include scheduled jobs, package activity, mining state changes, pool reconnects and messages about the memory or CPU backend. The charts do not choose between those explanations.

Our deployed Alloy configuration reads host journals and applies credential-pattern redaction. The tutorial takes a narrower approach and keeps only the XMRig unit. Add these components to the Alloy configuration from Part 1:

loki.relabel "mining" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    regex = "xmrig.service"
    action = "keep"
  }
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label = "unit"
  }
}

loki.source.journal "mining" {
  max_age = "1h"
  labels = {instance = "miner-a", job = "mining-journal"}
  relabel_rules = loki.relabel.mining.rules
  forward_to = [loki.process.redact.receiver]
}

loki.process "redact" {
  stage.replace {
    expression = `(?i)bearer\s+([A-Za-z0-9._~+/-]+)`
    replace = "[REDACTED]"
  }
  stage.replace {
    expression = `(?i)(?:password|token|secret|authorization|api_key)["']?\s*[=:]\s*["']?([^"'\s,]+)`
    replace = "[REDACTED]"
  }
  forward_to = [loki.write.central.receiver]
}

loki.write "central" {
  endpoint {
    url = "https://metrics.example.net/loki/api/v1/push"
    tls_config {
      ca_file = "/etc/alloy/telemetry/ca.crt"
      cert_file = "/etc/alloy/telemetry/client.crt"
      key_file = "/etc/alloy/telemetry/client.key"
    }
  }
}

Replace xmrig.service with the actual unit name. Give the Alloy account journal access. On the Debian and Ubuntu arrangement documented by Grafana, that means membership in adm and systemd-journal, followed by an Alloy restart. Without the necessary access, the journal component can start successfully and still collect no entries. See the journal source documentation.

The replacement rules cover only the patterns shown. Other secrets or identifying addresses can pass through. Keep logs private, avoid logging credentials at the source, and inspect representative entries after changes. A successful bearer-token test verifies that pattern, not the safety of every log entry.

In Grafana Explore, select Loki and start with:

{instance="miner-a",unit="xmrig.service"}

Then narrow the time window and, when useful, filter messages:

{instance="miner-a",unit="xmrig.service"} |~ "(?i)error|reject|timeout|connect"

The filter is a convenience for investigation. It will not find messages that use other wording, and the word “connect” also matches successful connections. Read neighbouring entries before assigning a cause.

Choose a log-retention period against actual ingestion volume. For a Loki installation using supported TSDB storage, retention needs the compactor and its retention settings configured. Setting a duration without enabling the relevant retention machinery is insufficient. The Loki retention documentation describes the storage-specific requirements. Monitor free disk space even when time retention is enabled.

Alert on distinct failure states

Our inspected rules distinguish unavailable mining APIs, zero reported hashrate after startup, excessive rejected work, stale collector output and missing agent samples. Those conditions call for different responses. A single “miner down” notification loses that distinction.

The following example adds an independent expected-miner record. Save it as /etc/prometheus/mining-rules.yml:

groups:
  - name: mining-inventory
    rules:
      - record: mining:expected
        expr: vector(1)
        labels:
          instance: miner-a
  - name: mining-health
    rules:
      - alert: MiningTelemetryMissing
        expr: mining:expected unless on(instance) (count_over_time(mining_collector_timestamp_seconds[5m]) > 0)
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'No recent collector samples for {{ $labels.instance }}'
      - alert: MiningCollectorStale
        expr: time() - mining_collector_timestamp_seconds > 120
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'Collector file is stale on {{ $labels.instance }}'
      - alert: MiningExporterDown
        expr: up{job="mining-hosts"} == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'Local exporter scrape failed on {{ $labels.instance }}'
      - alert: XMRigAPIUnavailable
        expr: (xmrig_api_up == 0) and on(instance) (time() - mining_collector_timestamp_seconds < 120)
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'XMRig API collection failed on {{ $labels.instance }}'
      - alert: XMRigNotHashing
        expr: (xmrig_hashrate_hps{window="60s"} == 0) and on(instance) (xmrig_uptime_seconds > 300) and on(instance) (xmrig_api_up == 1) and on(instance) (time() - mining_collector_timestamp_seconds < 120)
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'XMRig reports zero hashrate on {{ $labels.instance }}'
      - alert: XMRigHashrateUnavailable
        expr: ((xmrig_api_up == 1) and on(instance) (xmrig_uptime_seconds > 300) and on(instance) (time() - mining_collector_timestamp_seconds < 120)) unless on(instance) xmrig_hashrate_hps{window="60s"}
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'XMRig has no usable hashrate window on {{ $labels.instance }}'
      - alert: XMRigRejectRateHigh
        expr: ((rate(xmrig_submitted_shares_total[15m]) - rate(xmrig_accepted_shares_total[15m])) / clamp_min(rate(xmrig_submitted_shares_total[15m]), 0.000001) > 0.05) and (increase(xmrig_submitted_shares_total[15m]) > 10) and on(instance) (xmrig_api_up == 1) and on(instance) (time() - mining_collector_timestamp_seconds < 120)
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Sustained rejected work on {{ $labels.instance }}'

Add an expected record for each miner you intend to monitor. Keep this inventory on the monitoring side. If the only record of a miner's existence comes from that miner's agent, losing the agent also removes the evidence that something is missing.

The missing-telemetry rule waits for a five-minute window without collector samples, then remains pending for two minutes. A scrape failure can be detected sooner because Alloy can still report up=0. An entirely disconnected agent cannot send that failure value, which is why the inventory-based missing rule is also needed.

The stale rule catches a different case: a dead collector whose old file continues to be scraped. The API rule catches a collector that is running but cannot obtain a usable summary. The zero-hashrate rule requires a fresh successful API observation and enough process uptime to get past startup. A separate rule catches an API that keeps responding without a usable 60-second hashrate window after startup.

The for durations mean a condition must remain active across evaluations before firing. Short interruptions can resolve while pending. These timings should match the consequences of a failure and the ordinary behaviour of your miners. Prometheus explains pending and firing states in its alerting-rule documentation.

Route the resulting alerts through Alertmanager to a receiver you use. Group related failures, include the affected miner and condition, and suppress dependent noise when the telemetry path itself is unavailable. Confirm receipt at the destination instead of relying on Grafana's alert list.

The configuration bundle does not select a notification destination or include delivery credentials. Add your receiver, send a test notification, and verify its arrival and resolved message. If the monitoring server shares power and networking with the miners, use a separate external check for a site-wide outage. Otherwise, the component expected to report the outage can disappear with it.

Keep P2Pool and payout observations separate

XMRig acceptance counters describe results submitted to its configured pool connection. P2Pool sidechain shares and payouts involve additional conditions. Do not label the accepted-share panel “XMR earned” or infer a payout from a positive hashrate graph. P2Pool's upstream documentation explains how its payout process relates to sidechain shares and Monero blocks.

When we investigate mining through a local P2Pool instance, we also inspect its process, peer connectivity, local Stratum activity and relationship with the Monero daemon. Monero synchronization and fresh P2Pool state belong in a separate view so they do not get confused with the miner's own API health. Our P2Pool monitoring guide shows those panels and explains the local JSON collector.

Prefer numeric aggregates for a general dashboard. Avoid exposing worker connection addresses or payout addresses just to populate a legend. If you estimate hashrate from accepted work, label it as an estimate and state the time window and difficulty assumptions. It will not necessarily match a miner's short internal hashrate window.

Our earlier P2Pool rejection investigation shows why the connection path matters. A collection of miners can report a related symptom because they share a network dependency. Repeated failures across miners deserve a shared-path investigation before separate hardware changes on each machine.

Validate the pipeline before relying on it

Validation should follow the same route as the data. Start at the API, then move outward. These checks belong on your installation, not on a production miner you cannot safely interrupt.

Check the server configuration and rules with Prometheus's validation tool:

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/mining-rules.yml

Syntax checks are only the first pass. Use promtool rule tests with synthetic series for an API failure, a stale file, a missing agent, a counter reset and a short hashrate dip. Verify that an ordinary dip stays pending or resolves, while a sustained failure fires. These tests can exercise the logic without stopping a real miner.

For collector tests, return a valid summary, a startup summary with null windows, an authentication failure, malformed JSON and an unexpected field shape. Confirm that failure does not preserve old success values in a newly written file. Confirm that a publication failure leaves a timestamp that can age instead of a partially written exposition file.

For a staging end-to-end test, temporarily use an invalid token and watch the API alert path. Separately stop the collector timer while leaving the exporter running to test stale-file detection. Stop forwarding to test missing telemetry. Restore each component and confirm recovery before moving to the next test. Those failures are intentionally different.

Troubleshoot in the order the data travels

Avoid changing mining settings while the observation path is still suspect. A stale exporter can make an effective change look ineffective. A restarted miner can reset counters and make raw totals look like a loss. A mislabeled unit can create a scale error without any underlying change in performance.

Operate and share it without publishing the infrastructure

Our screenshots expose a selected observation, not a connection directory. The setup examples use generic paths, a fictional DNS endpoint and a neutral miner name. Real credentials, wallet addresses, internal addresses, complete host inventories, client certificates and raw logs do not belong in the public article or its configuration bundle.

Use stable, bounded labels. instance, an optional miner, a small set of hashrate windows, and a known backend name are usually enough for the main charts. Avoid labels that change per job, share, connection or timestamp. Those create new time series and can make a small mining deployment expensive to store and query. Prometheus discusses this trade-off in its instrumentation guidance.

Preserve the state that makes the history useful: Prometheus storage, Loki storage, Grafana provisioning and saved dashboards, Alloy's persistent data, and alert configuration. Back up private keys separately under restricted, encrypted storage. Test a restore before treating the backup as a recovery plan.

Track certificate expiration, monitoring-server disk growth and collector changes as operational work. After a miner or exporter update, compare the API fields and a few representative queries. After a dashboard change, inspect both the plotted values and their units. After a routing change, test notification delivery again.

Continue with P2Pool monitoring

For miners connected to P2Pool, the next guide adds peer connections, sidechain shares, effort and local-file freshness. It uses the same monitoring pipeline and shows which pool-side observations need a separate interpretation.

Previous: Part 2, read the dashboards · Start the series: Part 1.