Defining a tracing span schema

Context

EDRs 007 (Observability) and 015 (nView metrics) define our general approach to observability and some guidance regarding metrics to enable infrastructure reuse between node implementations. This EDR defines the guaranteed tracing spans exported for Cardano network health monitoring as well as the approach towards adding spans specific to Amaru maintenance and development and their initial setup.

Decision

This section is structured by the kind of traces under consideration:

• all data handling pertaining to blockchain handling, namely headers, blocks, and everything related to these
• handling transactions in the mempool
• peer management activities

Each of these areas contains parts that are intended for node operators as well as implementation details relevant for Amaru developers. It is important that the node admin can enable any desired trace to aid in debugging issues that may be observed in production.

A span is classified by its target and its span id. The target gives a rough categorization and allows selecting traces pertaining to one of the major Amaru components (precise granularity to be decided case by case).

The span id is the dotted suffix stored as the span’s metadata name. name: / NAME is the event name only, not this id. The span id identifies the source location within a given target and is composed from multiple components separated by dots. For example

1. a function, logical unit, data item, or similar
2. an operation (if applicable) described as a verb

IMPORTANT NOTICE ON tracing LEVELS

The tracing infrastructure we use to emit OpenTelemetry spans is built around log levels TRACE, DEBUG, INFO, WARN, ERROR. Log messages (including spans) from INFO upwards are printed to stderr by default and are part of the CLI / TUI design; these must not occur with high frequency and be relevant to a human observer. TRACE level is only intended for debug builds when actually debugging a specific code module, it is not considered viable to switch on all TRACE logging, and it is not considered viable to switch on ANY TRACE logging during normal operation; TRACE level is removed from release binaries by compile-time options.

For these reasons all nominally emitted trace spans intended for production use MUST be emitted at DEBUG or the INFO levels!

Since OpenTelemetry treats logs (i.e. tracing events a.k.a. log messages) as individually important, all debug logging that may occur at high rate MUST be emitted at TRACE level.

Impact on EDR 007 (Observability)

EDR-007 on observability defines the general approach to observability and the use of OpenTelemetry. It prescribes the use of a schema to specify spans attributes and their types with namespacing to avoid collisions.

In the current EDR the first two levels define the target. The next levels are the event name when the schema is an event, and the span id when it is a span. They are not NAME. NAME is only the event name (name:).

Here are two examples. During consensus, we receive an upstream message informing the node of a new block header and we need to evolve the nonce associated with the current header. In that case the fully qualified path in the schema is:

amaru::consensus::header::evolve_nonce

In this example, the target is amaru::consensus and the span id is header.evolve_nonce. Similarly for the ledger, we can find a schema entry like:

amaru::ledger::validation_context::create

In this case, the target is amaru::ledger and the span id is validation_context.create.

We also mandate that these suffixes are unique, even across targets, to avoid having two spans with span id block.validate both in amaru::consensus and in amaru::ledger.

In addition to the target and span id, the schema can define a number of tags, that translate to boolean span attributes. This can be used to classify spans across target and span id. For example:

- `setup` for starting up a component or subsystem
- `check` for validating inputs, peer behaviour, authorization, integrity, etc.
- `cpu` for computing or updating internal state
- `db` for structured storage operations
- `io` for all other file / storage / network operations
- `cli` for all operations related to the command-line interface or terminal user interface

Blockchain

The unit of processing for the chain is one block, which is logically also the unit for tracing. We therefore generate a trace ID for each header received from upstream and associate the header hash with that trace ID as a span attribute (header_hash). The trace ID can already be generated by the per-connection multiplexer when seeing inbound chainsync messages: traces that pertain to messages other than chainsync’s RollForward message will not be tagged with the header hash property.

Each span pertaining to header or block processing shall be associated with the trace ID for that header hash; this implies that this trace ID is transported through the consensus stages and via the external effects into stores and ledger.

Cardano Health Monitoring

TARGET: amaru::network

Even though the below spans are constructed in different consensus stages, this purpose justifies the overarching target selection.

One result of the Node Diversity Workshop in Porto (June 2–3, 2026) was a reaffirmation of the statement that core network health monitoring relies on recording four processing points:

1. first reception of a header from some upstream peer
2. first request to fetch the block sent to an upstream peer
3. first reception of the block from some upstream peer
4. local adoption of the block

The duration of that path is the span id perf.header.forward on target amaru::network, opened upon successful decoding of the header in the track_peers stage and closed in the select_chain stage, either upon seeing that the header is not on the best chain candidate or upon receiving the block validation result (which is slightly after adopting the block but typically before communicating the new tip to downstream peers). This records points 1 and 4.

Points 2 and 3 are recorded by the span id perf.blocks.fetch on target amaru::network, opened in fetch_blocks when requesting a range containing that block and closed when the block has been received.

These duration spans, together with perf.header.block_fetch_wait and perf.fork.switch, are span schemas on target amaru::network. They are not a substitute for the blockperf log lines below, and those log lines do not replace the spans. OpenTelemetry exports the spans as spans and the blockperf events as logs. Each schema is either a span or an event; that kind is what emission uses.

Operators also need these points as individual log lines they can turn on with EnvFilter. EnvFilter matches a bracketed name against the current span, so an event whose metadata name is perf.header.lifecycle is not selected by amaru::consensus[perf.header.lifecycle{peer}]. The four points are also emitted as events on target amaru::blockperf, in addition to the duration spans:

• header.announced — the first three distinct peers to announce a header hash, with rank 1, 2, or 3. A header already in the store does not open a new rank-1 line; later peers are logged only while that header’s first three slots are still open and the header has not been adopted
• block.requested — the peers asked for that block body
• block.received — each distinct peer that delivers the body, with rank in arrival order
• block.adopted — local adoption; peer is the first peer that delivered the body when one did

Each of these lines carries slot_latency_ms, milliseconds since the onset of the block’s slot. block.received also carries fetch_latency_ms, milliseconds since the request was sent to that peer, so a later staggered ask is timed from its own request.

AMARU_LOG=info,amaru::blockperf=debug adds them to the usual info log. AMARU_LOG=off,amaru::blockperf=debug prints only them. The terminal perf.header.lifecycle event still carries the intervals between the points for the TUI.

adopt_chain stores a consensus mode from the adopted tip’s slot onset and the wall clock. A lag strictly under 60 seconds is live. A change of mode is logged at info as tip.mode. While syncing, the four events are debug and tip.adopt is limited to one info line per second. While live, the four events and every adoption are info. track_peers logs chainsync.chain_lagging at most once a minute when near-now headers have been arriving for a minute and the adopted tip is not getting closer to the wall clock. A sync that is still adopting faster than 10 blocks per second, and the first minute of that condition, stay quiet.

Switching to a different fork opens the span id perf.fork.switch on target amaru::network for all blocks on the target fork. That span is separate from the consensus event of the same id, which records the outcome when a fork switch ends.

Consensus

TARGET: amaru::consensus

The span ids for this target don’t have to follow the names of the stages in the consensus pipeline, but rather reflect the logical processing steps that are performed on a header or block.

External pure-stage effect handling is extended such that the calling stage’s span is set as the context when executing the effect logic. This will tie for example all store or ledger spans to their parent span from the consensus span that triggered the effect.

Messages sent to the peer_selection stage stemming from adversarial peer behaviour must be associated with the related header’s trace ID.

The waiting time of a header in select_chain before being eligible for fetching blocks during catch-up (due to the back-pressure decoupling twixt select_chain and fetch_blocks) must be made explicitly visible in the traces by opening and closing the span id perf.header.block_fetch_wait on target amaru::network.

Ledger

TARGET: amaru::ledger

All ledger operations (validating headers and blocks, performing reward calculations, etc.) are traced in this category. Those activities that are related to a specific block will automatically be associated with the right tracing context via the parent propagation added to pure-stage effect handlers. Activities spawned from such invocations that are not related to that block or header (like asynchronous rewards calculations) are NOT associated with the calling trace context and instead create new trace IDs.

Mempool

TARGET: amaru::mempool

Each received transaction gives rise to a trace ID associated with the transaction hash as a property. The spans recorded are:

• span id state.transaction.submit: is opened upon local reception of the transaction and closed when inserted in the mempool or rejected
• span id state.transaction.receive: is opened upon N2N reception of the transaction and closed when inserted in the mempool or rejected
• span id state.transaction.forward: is opened when an upstream peer is requesting transactions and this transaction’s ID is sent; it is closed when the transaction has been sent to the peer

Protocols

TARGET: amaru::protocols

This target groups spans that are related to the implementation of specific protocols, like chainsync, txsubmission, keepalive etc…

Network

TARGET: amaru::network

This target is reserved for spans that are related to the network layer, regardless of the protocol being used. It is typically used for tracking the lifecycle of network connections.

References