Defining a tracing span schema

Context

EDRs 007 (Observability) and 015 (nView metrics) define our general approach to observability and some guidance regarding metrics to enable infrastructure reuse between node implementations. This EDR defines the guaranteed tracing spans exported for Cardano network health monitoring as well as the approach towards adding spans specific to Amaru maintenance and development and their initial setup.

Decision

This section is structured by the kind of traces under consideration:

• all data handling pertaining to blockchain handling, namely headers, blocks, and everything related to these
• handling transactions in the mempool
• peer management activities

Each of these areas contains parts that are intended for node operators as well as implementation details relevant for Amaru developers. It is important that the node admin can enable any desired trace to aid in debugging issues that may be observed in production.

All spans are classified first by a TARGET and within that target by a NAME. The target gives a rough categorization and allows selecting traces pertaining to one of the major Amaru components (precise granularity to be decided case by case).

The name uniquely identifies the source location within a given target (see discussion below) and is composed from multiple components separated by dots. For example

1. a function, logical unit, data item, or similar
2. an operation (if applicable) described as a verb

IMPORTANT NOTICE ON tracing LEVELS

The tracing infrastructure we use to emit OpenTelemetry spans is built around log levels TRACE, DEBUG, INFO, WARN, ERROR. Log messages (including spans) from INFO upwards are printed to stderr by default and are part of the CLI / TUI design; these must not occur with high frequency and be relevant to a human observer. TRACE level is only intended for debug builds when actually debugging a specific code module, it is not considered viable to switch on all TRACE logging, and it is not considered viable to switch on ANY TRACE logging during normal operation; TRACE level is removed from release binaries by compile-time options.

For these reasons all nominally emitted trace spans intended for production use MUST be emitted at DEBUG or the INFO levels!

Since OpenTelemetry treats logs (i.e. tracing events a.k.a. log messages) as individually important, all debug logging that may occur at high rate MUST be emitted at TRACE level.

Impact on EDR 007 (Observability)

EDR-007 on observability defines the general approach to observability and the use of OpenTelemetry. It prescribes the use of a schema to specify spans attributes and their types with namespacing to avoid collisions.

In the current EDR the first two levels are used to define the target of a span and the next levels to define the name of the span.

Here are two examples. During consensus, we receive an upstream message informing the node of a new block header and we need to evolve the nonce associated with the current header. In that case the fully qualified name of the span in the schema is:

amaru::consensus::header::evolve_nonce

In this example, the target is amaru::consensus and the name is header.evolve_nonce. Similarly for the ledger, we can find a schema entry like:

amaru::ledger::validation_context::create

In this case, the target is amaru::ledger and the name is validation_context.create.

We also mandate that the span names are unique, even across targets, to avoid having two spans called block.validate both in amaru::consensus and in amaru::ledger.

In addition to the target and name of a span, the schema can define a number of tags, that translate to boolean span attributes. This can be used to classify spans across target and span names. For example:

- `setup` for starting up a component or subsystem
- `check` for validating inputs, peer behaviour, authorization, integrity, etc.
- `cpu` for computing or updating internal state
- `db` for structured storage operations
- `io` for all other file / storage / network operations
- `cli` for all operations related to the command-line interface or terminal user interface

Blockchain

The unit of processing for the chain is one block, which is logically also the unit for tracing. We therefore generate a trace ID for each header received from upstream and associate the header hash with that trace ID as a span attribute (header_hash). The trace ID can already be generated by the per-connection multiplexer when seeing inbound chainsync messages: traces that pertain to messages other than chainsync’s RollForward message will not be tagged with the header hash property.

Each span pertaining to header or block processing shall be associated with the trace ID for that header hash; this implies that this trace ID is transported through the consensus stages and via the external effects into stores and ledger.

Cardano Health Monitoring

TARGET: amaru::network

Even though the below spans are constructed in different consensus stages, this purpose justifies the overarching target selection.

One result of the Node Diversity Workshop in Porto (June 2–3, 2026) was a reaffirmation of the statement that core network health monitoring relies on recording four processing points:

1. first reception of a header from some upstream peer
2. first request to fetch the block sent to an upstream peer
3. first reception of the block from some upstream peer
4. local adoption of the block

We support this by opening a span with NAME perf.header.forward upon successful decoding of the header in the track_peers stage and closing that span in the select_chain stage, either upon seeing that the header is not on the best chain candidate or upon receiving the block validation result (which is slightly after adopting the block but typically before communicating the new tip to downstream peers). This records points 1 and 4.

Points 2 and 3 are recorded by opening a span with NAME perf.blocks.fetch in fetch_blocks when requesting a range containing that block and closing that range when the block has been received.

Switching to a different fork will then open a span with NAME perf.fork.switch for all blocks on the target fork.

Consensus

TARGET: amaru::consensus

The span names for this target don’t have to follow the names of the stages in the consensus pipeline, but rather reflect the logical processing steps that are performed on a header or block.

External pure-stage effect handling is extended such that the calling stage’s span is set as the context when executing the effect logic. This will tie for example all store or ledger spans to their parent span from the consensus span that triggered the effect.

Messages sent to the peer_selection stage stemming from adversarial peer behaviour must be associated with the related header’s trace ID.

The waiting time of a header in select_chain before being eligible for fetching blocks during catch-up (due to the back-pressure decoupling twixt select_chain and fetch_blocks) must be made explicitly visible in the traces by opening and closing a span with NAME perf.header.block_fetch_wait accordingly.

Ledger

TARGET: amaru::ledger

All ledger operations (validating headers and blocks, performing reward calculations, etc.) are traced in this category. Those activities that are related to a specific block will automatically be associated with the right tracing context via the parent propagation added to pure-stage effect handlers. Activities spawned from such invocations that are not related to that block or header (like asynchronous rewards calculations) are NOT associated with the calling trace context and instead create new trace IDs.

Mempool

TARGET: amaru::mempool

Each received transaction gives rise to a trace ID associated with the transaction hash as a property. The spans recorded are:

• NAME state.transaction.submit: is opened upon local reception of the transaction and closed when inserted in the mempool or rejected
• NAME state.transaction.receive: is opened upon N2N reception of the transaction and closed when inserted in the mempool or rejected
• NAME state.transaction.forward: is opened when an upstream peer is requesting transactions and this transaction’s ID is sent; it is closed when the transaction has been sent to the peer

Protocols

TARGET: amaru::protocols

This target groups spans that are related to the implementation of specific protocols, like chainsync, txsubmission, keepalive etc…

Network

TARGET: amaru::network

This target is reserved for spans that are related to the network layer, regardless of the protocol being used. It is typically used for tracking the lifecycle of network connections.

References