Prometheus Interview Questions: Counter Resets and Rule Tests

Run a deterministic rule exercise

The files below were checked with promtool 3.15.0 on macOS arm64. Save them as rules.yml and test.yml in the same directory. The rule check validates syntax; the unit test validates expected values and alert behavior. Both are needed to reproduce this exercise.

Instance a resets between minutes 2 and 3. Instance b keeps increasing, which can hide that reset in an aggregate. The queue rises above the practice threshold, later returns to a measured zero, then becomes explicitly stale.

groups:
  - name: practice
    rules:
      - record: practice:aggregate_counter
        expr: sum by (service) (practice_requests_total)
      - record: practice:wrong_rate
        expr: rate(practice:aggregate_counter[3m])
      - record: practice:requests_per_second
        expr: sum by (service) (rate(practice_requests_total[3m]))
      - alert: PracticeQueueHigh
        expr: practice_queue_depth > 4
        for: 2m
        labels:
          severity: practice

1. Why apply rate before aggregation?

At minute 4, the tested sum of per-instance rates is 2.625 requests per second. The rate of the previously aggregated counter is 1.625. The aggregate increases even when instance a resets, so its history loses the reset information available in the individual series. The original series must retain their identity while rate handles their counter behavior; the result can then be grouped by service.

Follow-up practice: Use the input values at minutes 2 and 3 to show how one instance’s reset is hidden by the other instance’s increase. Do not describe a counter reset as negative request traffic.

2. Which labels remain in the correct recording rule?

The expression sums by service, so instance is removed from the result while service="booking" remains. The recording rule gives the output its own metric name. The test checks the complete expected label set as well as the number, which catches an accidental grouping change that a value-only check could miss.

Follow-up practice: What would change if you removed by(service)? Predict the output labels, then change the exercise deliberately and inspect the test failure.

Use the complete synthetic test file

This test includes the deliberately wrong result as a comparison, not an endorsed rule for production use. The correctly named practice:requests_per_second rule is the one whose behavior we want to preserve.

rule_files:
  - rules.yml
evaluation_interval: 1m
tests:
  - name: counter reset, labels, and alert timing
    interval: 1m
    input_series:
      - series: 'practice_requests_total{service="booking",instance="a"}'
        values: '0 60 120 15 75 135'
      - series: 'practice_requests_total{service="booking",instance="b"}'
        values: '100 220 340 460 580 700'
      - series: 'practice_queue_depth{service="booking"}'
        values: '0 6 6 6 0 stale'
      - series: 'practice_queue_depth{service="missing"}'
        values: '_ _ _ _ _ _'
    promql_expr_test:
      - expr: practice:wrong_rate
        eval_time: 4m
        exp_samples:
          - labels: 'practice:wrong_rate{service="booking"}'
            value: 1.625
      - expr: practice:requests_per_second
        eval_time: 4m
        exp_samples:
          - labels: 'practice:requests_per_second{service="booking"}'
            value: 2.625
      - expr: practice:requests_per_second
        eval_time: 5m
        exp_samples:
          - labels: 'practice:requests_per_second{service="booking"}'
            value: 2.75
      - expr: 'resets(practice_requests_total{instance="a"}[5m])'
        eval_time: 5m
        exp_samples:
          - labels: '{service="booking",instance="a"}'
            value: 1
      - expr: 'practice_queue_depth{service="booking"}'
        eval_time: 4m
        exp_samples:
          - labels: 'practice_queue_depth{service="booking"}'
            value: 0
      - expr: 'practice_queue_depth{service="booking"}'
        eval_time: 5m
        exp_samples: []
      - expr: 'absent(practice_queue_depth{service="missing"})'
        eval_time: 5m
        exp_samples:
          - labels: '{service="missing"}'
            value: 1
    alert_rule_test:
      - eval_time: 2m
        alertname: PracticeQueueHigh
        exp_alerts: []
      - eval_time: 3m
        alertname: PracticeQueueHigh
        exp_alerts:
          - exp_labels:
              service: booking
              severity: practice
      - eval_time: 4m
        alertname: PracticeQueueHigh
        exp_alerts: []

Execute syntax and behavior checks

The reproduced result was a successful syntax check with four rules and a successful unit test. This is a local promtool run; it does not start a Prometheus server or evaluate a real service.

promtool --version
promtool check rules rules.yml
promtool test rules test.yml

3. Why is the correct rate 2.75 rather than 3 at minute 5?

The range contains the minute-3, minute-4 and minute-5 samples. For instance a, those counter values are 15, 75 and 135. Rate extrapolation near the range boundary is limited by the estimated zero point, rather than extending that counter below zero. The tested contribution from a is 0.75 per second; b contributes 2. The total is 2.75. A raw difference divided by the distance between the first and last samples misses that boundary behavior.

Follow-up practice: Why should you run a test before asserting that an observed one-per-second slope guarantees a one-per-second range rate after a recent reset? State the samples and range you are discussing.

4. When does the practice alert start firing?

The queue first exceeds 4 at minute 1. With a two-minute for duration and one-minute rule evaluations, the alert is not firing at minute 2 and is firing at minute 3 in this test. At minute 4 the queue is zero, so the expression is no longer true and the firing alert disappears. These boundaries are tested rather than inferred from the threshold alone.

Follow-up practice: If the queue fell below the threshold before the for duration elapsed, what would you expect? Add a separate input series and expected alert test to check your prediction.

5. Are a measured zero, a missing sample and a stale series equivalent?

No. At minute 4, booking has a real zero sample; at minute 5 it has an explicit stale marker and the instant-vector query is empty. The missing service has no samples at all, and absent returns a value indicating that absence. A single missing scrape does not necessarily make an instant query empty immediately, because earlier samples may still be considered under the query’s lookback behavior. The test intentionally distinguishes these cases instead of replacing all of them with zero.

Follow-up practice: Why would drawing a missing series as zero change the meaning of a queue chart? Identify which facts you actually have in each of the three cases.

6. How can you prove the test would catch the aggregation mistake?

Change only the correct recording rule’s expression to rate(practice:aggregate_counter[3m]) and rerun the same tests. In the verified mutation, the minute-4 result became 1.625 instead of 2.625 and the minute-5 result became 3 instead of 2.75, so the test failed. Restore the original expression and the test passes. The failure demonstrates that the expected values detect this specific wrong implementation.

Follow-up practice: What would a syntax-only check miss here? Explain why a valid expression can still answer the wrong operational question.

Practice explaining the result without inventing an incident

Use the local experiment as a technical example. Describe the counter histories, expected labels and tested alert times; do not recast the exercise as a production outage you resolved.