!5069odin/tc-metrics-skill → develop8f53933f0c

tc-metrics: rank Midgard's metric anomaly scores, plus Go 1.26.10

AI-generatedEverything on this page was written by an AI agent from the diff: the summaries, the narration, the diagrams, the flags and the glossary. It can be wrong or incomplete, so check it against the code before you rely on it.

11commits
25files
412lines changed
205lines to read
736series ranked
1.26.10Go toolchain

Where the diff is

5-minute explainer

What changed

What Midgard scores

Each series' newest hour against its own hourly history.

newestcommonnessshare ≥ this extrememodifiedZsize and sign

Empty /anomalies

Floor 2/n = 0.18 at n=11, above the 0.05 cut.

0.1820.0830.05n=11n=24n=40n=3360.05 cut

Ranking order

STALE first, commonness ascending, then |z| descending.

samples = 0commonness ↑|modifiedZ| ↓every row kept

try it ↓

Rows that mislead

Notes flag thin history, MAD collapse, floor hits.

  • n < 8 · score is noise
  • |z| > 100 · flat, then moved
  • commonness ≤ 2/n · at floor
  • 3.5 ≤ |z| ≤ 100 · outlier

Dead collectors

Carried-forward ties look healthier; flagged STALE and sorted first.

collector diesSTALEcommonness of newest hour

Read by value

Invariants and solvency always printed, whatever their rank.

thornode_invariant_broken = 1thornode_solvency_excess_e8 < 0

a permanently broken invariant scores as common

Whose metric is it

Node-local, seen by one node, or network-wide.

go_* process_*node-local*_consensus_*corroboratethornode_* pool_*network

health + tc-debug

Optional step 1b sweep; tc-debug points here for vague symptoms.

metrics.py --grep … --top 20--seriesLoki windowreport row

Go 1.26.10

Toolchain bump across go.mod, CI images, Dockerfiles, scripts.

1.26.91.26.10CI_TOOLS_VERSION 2 → 3 · image repin in a follow-up MR

Ranking playground · same note() and sort key as metrics.py

Newest hour per series: edit any cell

State + upgrade

Versions and defaults

  • changedGo toolchain (go.mod, .go-version, images)1.26.9 → 1.26.10
  • changedtest/simulation go directive1.26.9 → 1.26.10
  • changedCI_TOOLS_VERSION2 → 3
  • newmetrics.py --window14d
  • newmetrics.py --top40
  • newMAD_FLAT (|z| artifact cut)100

Line by line · every changed logic line, narrated

9:36runtime
10code chapters
100%logic lines narrated (163/163)
6flagged spots
speed

Chapters

What to check

Ask an AI about this change

Chat assistantClaude, ChatGPT, … (reads a link)

Coding agentClaude Code, Cursor, … (reads the repo)

The context file is context.md. Use Copy with context for an AI that can't open links; it pastes the whole file in.

Glossary

/scores, /anomalies, /data
Midgard's /v2/metrics routes: every series' newest hour, those cut at a commonness threshold, and one series' full history. tc-metrics SKILL.md ↗
bucketCount (n)
Hours of history a series has in the window, 1 to 336. Sets its floor 2/n, so commonness isn't comparable across rows. tc-metrics SKILL.md ↗
CI tools image
The dind jobs' tools image, versioned by CI_TOOLS_VERSION and pinned by digest in .gitlab-ci.yml after a second MR. ci/Dockerfile.ci-tools ↗
Claude Code skill
A folder under .claude/skills with a SKILL.md an AI agent follows, plus helper scripts. health and tc-debug are existing ones. health SKILL.md ↗
CometBFT / tendermint_*
THORNode's consensus engine. Its metrics carry a tendermint_ or cometbft_ prefix depending on version. CometBFT docs ↗
commonness
Share of hours in the window at least as extreme as the newest, two-tailed. 1 is ordinary; 2/n is the most extreme possible. tc-metrics SKILL.md ↗
GOTOOLCHAIN
Go setting that picks the compiler; auto downloads the toolchain go.mod pins when the installed Go is older. Go toolchains ↗
Loki
Grafana's log store; the health skill queries THORNode logs there with logcli. Grafana Loki docs ↗
Midgard
THORChain's indexer and API. Here it also scrapes one THORNode's Prometheus metrics and scores each series hourly. midgard!733 ↗
modifiedZ / MAD
0.6745·(value − median)/MAD, where MAD is the median absolute deviation. A flat series has MAD ≈ 0, so |z| explodes. metrics.py ↗
STALE
A row with samples = 0: no reading this hour, value carried forward. Its source failed; the score can't show it. metrics.py ↗