Roadmap
Milestones
Typed observer hooks for embedders
Embedders have to reconstruct node health from telemetry. This is a proposition to extend `LedgerObservers` to facilitate and make more robust amaru integration.
feat(stores): add resumable immutable block ranges
Add resumable immutable block-range reads to the chain store and expose them through `NodeRunning::block_reader()`, allowing embedders to process blocks in bounded, cancellable batches from a saved slot/hash checkpoint.
feat(amaru): add `amaru peer probe` for N2N peer diagnostics
## Summary - Add `amaru peer probe <TARGETS> [OPTIONS]` for one-shot N2N checks: TCP connect, handshake (default max version 15), tip, peer-share, keep-alive ping, and optional `--peer-check` of shared peers - Targets may be `IP:PORT`, `HOSTNAME:PORT`, or a CIP-0155 SRV DNS name (all usable `_cardano._tcp.<name>` records); comma-separated lists are probed in order - Operator controls: wall-clock `--timeout` (default 5s, fresh budget per `--retry`), step timeouts, live ping output in human mode, `--json`, and docs at `docs/peer-probe.md` (linked from README Monitoring)
Remove `Address::Stake` Variant
This PR addresses a longstanding `TODO`, removing the `Address:Stake` variant which isn't needed. the `RewardAccount` type should be used instead, which removes some `unreachable` branches in the code. It also prevents illegal state, such as specifying a reward account in a transaction output.
Conditionally enable malachite backend for big integers.
`dashu` remains the default, because it has a more convenient license scheme (Apache-2.0). Malachite is LGPL-3.0, so it is fine to distribute through the binary since the sources and build can be tweaked according to the license terms, but it forces library consumer to be LGPL-compliant when shipped as part of the library. So we selectively use the malachite backend for the release workflow, but leave dashu as a library default.
Fix credential validation
This PR ensures we check credentials exist when delegating, with test fixtures to prove it.
detect stage messaging deadlocks
Closes #1480 the Tokio runtime now detects a cycle of stages deadlocked sending to each other and kills the node with a clear diagnostic that names the stages, marks the cycle, and shows each blocked message and mailbox occupancy, while the bookkeeping stays off the uncontended send path.
Uplc memory leak fix and further optimisations
Some background work while busy doing other stuff. The main intent was to investigate some of the memory usage in the uplc VM identified by the benchmark; found one memory leak in some of the builtins. Then pursued some "low hanging" optimisations that minimize code changes and maintenance overhead.
chore: do not fail sync when an error is shown
Sync should fail only in case of amaru crashing. Still, display error and warnings so that run can be investigated.
chore: lib fixes
Improve lib usages, do not assume all crates always are enabled.
UPLC and ledger conformance (polish, fixes, rework)
Supersedes #1522
TUI Docs Update
This PR updates the TUI docs to reflect the latest state of the TUI, with updated images.
NTP Check on Unix Systems
Closes #1369
chore: remove dead code
Removed some dead code.
test: update the cbor dataset and fix conformance gaps
This PR selects a CBOR conformance corpus with a better coverage for the `HuddleSpec` specification and fixes a few gaps revealed by this new corpus:
Zero-denominator Conway governance parameter update is admitted to the Amaru mempool
## What revision are you using?
CIP-203 Amaru Payload and Usage
Closes #1360
stage messaging deadlock detection
A deadlocked cycle of stages waiting to send a message to the respective next stage should be recognised by the runtime and kill the node with a clear diagnostic including the involved stages and the types of messages in their mailboxes.
feat: improve kes signing support
Hold KES signing keys in a separate `amaru-kes-signer` executable for block signing.
Support mempool submission and monitoring for library consumers
Applications embedding Amaru need to submit transactions and track mempool contents without reconstructing state from traces.
chore: wait for proper node shutdown
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix(node): retain shutdown ownership and allow cleanup retries
Improve shutdown API for embedders: * allow to customize shutdown timeout * allow to retry a failed shutdown * share details about why shutdown failed
chore: general bootstrap/mithril cleanup
Cleanup / leftover removals from `amaru-bootstrap` / `amaru-mithril` dureing recent refactorings.
make TUI snappy
Currently, the TUI takes hundreds of milliseconds to react to clicks or key presses for me.
TUI: copy text and fix on large screen
- **allow copying text from the TUI outside copy mode** - **restrict TUI to less than 65536 cells**
chore: allow to see fetch error details
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
tone down ERROR and WARN that shouldn’t be
There’s an ERROR when a peer disconnects (which due to the impolite disconnection process might remain a WARN), and there’s a WARN when we get an invalid transaction (which can happen for entirely legitimate reasons).
Check the conformance of cbor encoding with the Haskell node
### Abstract
Report the coverage of the cbor dataset against the specification + edge cases
### Abstract
Amaru Watermark in Protocol Minor Version
### Abstract
KES key number check
Before accepting a new KES key from the chain database, the node must check that the opcert sequence number is exactly one greater than that of the previous key.
check quality of peers shared by peer_sharing
Markus found the following when looking at the peers shared by Amaru:
Check the conformance of flat decoding
### Abstract
Extend the CBOR conformance coverage
### Abstract
Removes the long-lived in-memory cache of two full stake distributions; only one is needed.
Supersedes #1339 (cc @bomanaps)
add block forging CLI commands
Adds three new subcommands under amaru keys to help operators manage block forging key materials: opcert inspect, kes period, and pool-id closes #1368.
fix(ledger): skip storage removal for transient DRep registrations
Reg/unreg of dreps during a single blocks shouldn't lead to persistency modification.
chore(amaru-ledger): rename option operator-cert
Rename `--operator-operational-certificate`/ `AMARU_OPERATOR_OPERATIONAL_CERTIFICATE` to `--operator-cert`/`AMARU_OPERATOR_CERT`
chore: better mithril sync
Improve both UI (CLI) and performance during mithril sync.
blockfetch initiator terminates on Done
Currently, demoted peers remain in Diffusion because the blockfetch initiator doesn’t terminate, then the group transition timeout kills the bearer after 5min. This fixes it and adds an integration test for this behaviour.
test: allow competing blocks with different parents
Fix https://github.com/pragma-org/amaru/actions/runs/38631677282/job/116084601598
fix: use ephemeral outbound ports in transaction submission E2E
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
Correct is_valid=false validation
In the Haskell node the "UTXOW" checks (validating witnesses) run regardless of the phase 2 validity. In Amaru, currently, we only "require witnesses" when `is_valid=true`, which means there are blocks we accept that Haskell rejects, and vice versa.
make sure that downstream promotion is done responsibly
If we just always prefer existing connections then there may be a heightened risk of eclipse attacks. We should audit this behaviour
make sure (and document) that TraceBuffer is usable in production
It should basically work already, but has been disabled due to high overhead observed initially.
make sure that stopping the node gracefully stops all mini-protoocols
Not sure what the current state is: we should try to cleanly shut down all bearers in consensus before terminating the process (unless of course a second kill signal comes in).
fix: make sure amaru has proper time to cleanly shutdown
Skip-changelog
monitor remote_use and improve sharing our socket address
This change records which initiator mini-protocols the remote has opened on an established bearer, derives a temperature of none, maintenance, or diffusion, emits a remote_use trace only when that temperature changes, and shows it in the TUI. Share replies append this node’s connected-socket IP and listen port, dropping the last sampled address when the reply is already full, and staying within the requested amount and a cap of 10. Maintenance now starts keep-alive and, when the peer is advertisable, the peer-sharing initiator on an inbound duplex bearer as well as an outbound one. Peer select…
TUI improvements: 1s/3s/5s percentiles, peer scores
The score is not yet used in peer selection, that part is still awaiting a proper overhaul.
don’t log cardano-cli ping as “WARN mux.failed …”
Currently, such a ping looks like this: ``` 14:42:22 INFO amaru::protocols manager.message.process manager.peer.handshake_completed advertisable=false conn_id=154 full_duplex=false message_type=HandshakeComplete peer=5.135.141.64:59754 role=responder 14:42:22 INFO amaru::protocols manager.peer.local_use_applied conn_id=154 local_use=none peer=5.135.141.64:59754 14:42:22 INFO amaru::protocols peer_selection.peer.remote_use conn_id=154 peer=5.135.141.64:59754 remote_use=diffusion
feat: tastyer keys
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
proper linking of events and spans for header and block perf telemetry
including world loop test to assert proper OTel structure
parent span requirements can be formulated in trace schema
second step of #1422
trace schema distinguishes spans from events
first step of #1422
add facilities for logging effect timings and using them
- also adds --log-output command line argument for adding separate output streams
rework peer sharing resource to remove pure-stage messaging cycles
This change makes the consensus performance worker the single owner of peer-population state. Protocols record handshakes, closes, failed dials, applied local use, keep-alive round trips, shared addresses, and answered share requests on that worker through a new PeerTracking trait. track_peers records an intersection miss the same way and clears that
use connection reset to remove adversarial peers
Currently, ending a peer connection uses orderly TCP shutdown, flushing all data. A misbehaving peer can use this to hold the Manager ransom.
Synthetic bench for roll forward: simple transaction (1 input -> 1 output)
Exercise `State::roll_forward` against a real RocksDB-backed ledger with 10,000 seeded UTxOs and a saturated volatile window. Benchmark empty blocks and blocks with increasing counts of signed ADA-only transactions so the delta isolates per-transaction application cost from fixed block overhead. ``` roll_forward fastest │ slowest │ median │ mean │ samples │ iters ╰─ simple_transactions │ │ │ │ │
clean cli and align options with EDR-019 - guidelines for cli
This is a bit painful, but waiting longer for this is not going to help. With adoption growing and more and more docs being produced, we need to anchor the "final" api surface for users real soon. The best moment to do this was 6 months ago. The second best moment is now.
feat: add KES key generation CLI
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
chore: logs
Some more log cleanup
figure out sync slowness with many peers
Currently, sync speed is much faster with one peer than with 100 on my Hetzner box. This is quite likely not limited by network bandwidth, although I haven’t yet done a proper analysis.
chore: lower internal log level
Skip-changelog
More UPLC Improvements
This PR introduces more UPLC fixes, mostly conformance issues. All introduced test cases were run against `plutus 1.70`.
Puzzling error message
I frequently see this error message:
feat: allow to run longer nightly tests
Allow to run nightly syncs from initial bootstrap snapshot available. Introduced `amaru snapshot epochs` command to list all snapshots available.
allow multiple listeners to be opened
Currently, only one `--listen-address` is allowed, which may prevent listening on IPv4 and IPv6 at the same time.
fix: make sure that in a multi assets bundle the policies and assets are sorted
This PR fixes a `MultiAssets` regression introduced by #1400.
chore: stop sync when error happens
Skip-changelog
fix: wrong koios host for mainnet
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
stop TUI logs from moving when not at most recent position
Currently, scrolling to the desired place in the logs does not stabilize that view, especially DEBUG logging is unusable that way because it keeps scrolling. The first line on display should be fixed unless changed by explicit user interaction (moving the slider, using cursor keys, etc). This applies both to normal and copy modes.
make pure-stage mailbox size configurable
- enlarge for Manager and PeerSelection - remove the 10'000 override for simulation tests
deploy typestate API across amaru-consensus and protocols
Changing the communication behaviour of stages in the graph usually has an influence on the minimum required mailbox size of recipient stages, and since those need careful design, we need to both control changes as well as make the actual communication behaviour accessible to static analysis. Capturing it as typestate allows the construction of typegraphs in code (usually in tests, but possibly even when starting the node with a particular configuration — number of upstreams/downstreams will have an influence on the analysis) to ensure that we don’t make mistakes too easily. It can also guide …
move peer tracking into the PeerPerformance resource
Today, there are lots of messages flowing back and forth between the `manager` (and the network stack) and the `peer_selection` stage. This led to the ad hoc bug fix of grossly enlarging the mailbox sizes of both stages, while the proper fix is to perform the management and querying of shared state without sending messages:
write proper consensus design documentation
We have bits and pieces everywhere, the design is basically working and will need a few tweaks here and there, so it is time now for an honest review and a write-up of the final shape of the consensus and protocols implementation architecture. This includes choices on how to
make pure-stage internals concise and understandable
Currently, `pure-stage` still has almost all features from the beginning with v1 internal structure. Usage has matured enough to pare down the API to what is actually being used outside the crate, then come up with a nice and simple internal implementation that humans can review and work on with confidence, keep the tests (of the retained API) working as is, and in general make it fit for release.
chore: also test mainnet sync nightly
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix: lifecycle improvements
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
should remove unproductive peers
We measure some things about peers (like RTT, announcements). We are not yet quantifying the badness of a peer, like using our bandwidth and CPU without producing value.
add back-pressure on accepting inbound connections
Currently, one connection is accepted every 100ms and sent to the PeerSelection without flow control. With the PeerSelection mailbox being used by many other purposes, a dedicated back-pressure mechanism should be implemented, e.g. by PeerSelection giving permission for receiving a new connection.
ensure equal semantics between Tokio and Simulation runtime
- require a timeout when the callee terminates during a call - deliver a queued call when its mailbox slot frees - time out calls that cannot enqueue
fix: make decoders conform to the cbor dataset
This PR fixes the decoding gaps exposed by the cbor dataset generated in the [`cardano-cbor-dataset` repository](https://github.com/r2rationality/cardano-cbor-dataset/pull/1).
chore: improve store errors
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
feat: make sure progress always ends with a summary
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
Prevent some panics in UPLC
This PR fixes 5 explicit panics that were reachable on chain, and one typo that resulted in a panic. This is not every explicit panic (`unreachable!`, `panic!`, etc.), but it is a bundle of ones that I easily reasoned about, and moves us in the right direction.
Harden some UPLC builtins and switch some dependencies in the UPLC virtual machine
Why?
properly fix blockfetch typestate tracking
Once `eff.call(...)` returns a result that may state “the message was not actually sent”, this will require more thought on how to match the resulting type graph to the session spec: there’s a new edge from the “request sent to muxer” state back to the initial state, and the “pull” needs to be injected not when leaving the switch state `Idle` but when leaving the “Idle or Request Sent” conglomerate state. Not sure yet how to work this out.
Purge Mempool Transactions on Block Adoption
### Abstract
Mempool: For Some Parent and Slot, Provide a Block Body
### Abstract
Trace Schemas for Forging
### Abstract
check and improve inbound connection limit handling
Currently, the handshake concludes successfully and then the bearer is dropped if the inbound connection limit is exceeded. We should investigate how the Haskell implementation signals this to the peer and possibly improve the behaviour accordingly.
Persist findings of consensus behaviour in Cardano blueprints
As discussed in Porto last week, any gaps found in the Ouroboros network spec shall eventually be filled with information on https://github.com/cardano-scaling/cardano-blueprint. This ticket serves as a reminder to myself that I should start with the notes taken in Porto from very fruitful discussions with Marcin M.
Ledger predicate: VRFKeyHashAlreadyRegistered
### Abstract
Use a separate column family for `BEST_CHAIN_PREFIX`
### Abstract
Support chained transactions in the mempool
### Abstract
chore: update hickory
So that amaru works on ios
Merge Test Suites
We currently have (at least) two different test suites for the ledger; the `evaluate_ledger_state` tests which partially run CBOR encoded test vectors used by the Haskell implementation, and the JSON fixtures we produce and use ourselves.
remove generic_const_exprs feature
fixes #1406
Cleanup Property Generators
Closes #1216
chore: do not use fixed peer
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
feat: allow 3rd party dev to have full PR run
Make sure external contributors PR can run fully so that anyone can get good DX.
WARN on Local Clock Drift
### Abstract
fix(protocols): give each mini-protocol its ingress limit
Chain-sync was capped at the 5760-byte handshake and peer-sharing window, so a pipelined header burst closed the connection. Look the limit up from the mini-protocol number with the responder bit masked, using the ouroboros-network ingress sizes for both directions.
Optimize reward payouts using RocksDB merge operators
### Abstract
Discrepancy with Haskell's node on pools fallback drep
### What revision are you using?
fix(ouroboros): accept cardano-cli VRF signing keys
The Conway text envelope stores a 64-byte Praos secret: the 32-byte seed followed by the verification key. Loading that file failed because the parser required a 32-byte seed.
perf(amaru-uplc): defer builtin cost argument sizing
~15-20% performance improvements; but more importantly, avoids potential heavy resource usage for some types when evaluating costs.
Minor CI Improvements
This PR introduces 2 minor changes to our CI workflow: 1) Write rust dependency cache once nightly (0:00 UTC) - Should cut pushes to main CI workflow time in half - Each run will NOT start from scratch, which was always the case before - Blacksmith (paid) does not rebuild the cache on every push, saving us maybe a couple bucks.
Generate .env from clap's definitions (for node run + node bootstrap & globals)
See also #1404.
fix: hold early mini-protocol segments until handlers register
The previous fix missed the case where the responder gets the handshake early.
Forge Header Effect
This PR introduces the effect and the trait used to forge headers, as well as new world tests by @rkuhn. It also includes the changes introduced in, and therefore supersedes, #1375.
Expose typed mempool services for embedders
### Abstract
Remove "Nightly Synchronization" job and simplify e2e test setup
### Abstract
Amaru for epoch-end ledger audits
First, let me thank you for Amaru. I'm currently working on [Tessera](https://tessera-preview.matthieu-pizenberg.workers.dev/), a tool implementing [CIP-179 on-chain surveys](https://github.com/cardano-foundation/CIPs/blob/master/CIP-0179/README.md). We want survey results to be reproducible and independently verifiable, so Amaru is one of the options I've explored. This is an experience report about what I did, and where I struggled.
Compilation warning
When compiling locally
Block Forging CLI Commands
### Abstract
TUI: Forging View
### Abstract
add EDR on KES key handling
fixes #1414
consolidate reconnect and resolv timers
fixes #1428
Stop compiling test-utils into production builds
This PR removes the `test-utils` feature from the default list in `amaru-consensus`, which meant the test logic was included in the compilation process of any production build of `amaru`. This mostly has no tangible benefit (the compiler would drop the unreferenced test logic), but it does remove 3 crates from the binary and may save a few seconds in compilation. It does mean that it is semantically correct to use and enable `test-utils`, and a future accidental reference will error during compilation.
fix: verify both slot and hash
`BlockFetch` only succeeds if both asked slot and hash match.
chore: added missing peer staging
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
feat: add API for bootstrap and mithril progress report
This PR exposes bootstrap and Mithril synchronization as reusable library workflows, with cancellation, crash recovery, and canonical progress reporting shared by the CLI and embedding applications.
decode each VersionData with its own version, keep bytes is unknown
fixes #1425
add staggered block fetching
fixes #1423
make blockperf logs selectable via AMARU_LOG and inculde peer
- **log blockperf at DEBUG** - **blockperf logging, live vs sync mode** - **clean up logs at transition from sync to live**
Update packages metadata for v10.11.20260925
Updates packages metadata generated from the published release [`v10.11.20260925`](https://github.com/pragma-org/amaru/releases/tag/v10.11.20260925).
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4){kind=small}
ci: some fixes
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix: explicitely set network when starting oura
add forge_block stage
fixes #1361 fixes #1362
fix: workaround github api rate limits
Make sure CI doesn't fail due to GH API rate limit (unauthenticated calls). Relies on git clone instead.
add session type verification of typestate protocol handlers
This is the last preparatory step before moving all mini-protocol handlers to the new typestate API. It gives us full binary session type verification of the wire protocol in mermaid syntax, including timeouts (*) and `WantNext` and with a generic (i.e. independently auditable) implementation of pipelining.
chore: improve PR safety
**Currently blocked due to API rate limits while accessing peer snapshots. We might need to reconsider current approach**
fix(amaru): add missing systemd units definitions to debian archive.
fix(amaru-ledger): prune descendants of expired proposals
Fix governance ratification so that:
Reorganize Leadership Logic
In an attempt to familiarize myself with the new `forge_block` stage, I pulled the code locally and decided to isolate the leadership calculation logic. My goal is to decouple the "business logic" (Cardano-specific logic) from the pure stage wiring as much as possible, while also using the type system to restrict illegal states as much as possible. This is my proposed organization, and we can replicate this design for other pieces of the business logic. This means the computation logic lives with the storage logic, and is easier to read, review, and understand since it's not tangled into pure …
fix(tui): restore terminal before reporting panics
Make sure panic message is properly shown when tui is enabled.
chore: proper node shutdown
This PR does 2 main thing:
Monitor RocksDB metrics
### Abstract
get rid of “analyzing changelog ...” notifications
no-changelog
test: run decoding tests from a node-agnostic test suite
This PR adds a test suite that checks the decoding expectations of the conformance test suite created in the [`cbor-dataset` repository](https://github.com/r2rationality/cardano-cbor-dataset):
fix(progress): leave completed terminal progress bars visible
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
fix: decode uplc with smaller stack
Playing on iOS, I got some issues with stack sizes and uplc decoding. Building on top of recent related changes, use similar technics to solve that.
feat: test full sync nightly
Make sure full sync is tested every day. This will help catch new issues that might arise from new blocks.
chore: reduce trace levels
Traces that can happen w/o breaking amaru should be `warn`ings.
fix: more resilient pipe
Skip-Changelog
bench epoch transition at scale
This pr adds a divan benchmark that seeds a RocksDB with realistic pools, UTxOs, and accounts, drives a State to the epoch boundary, and measures the full transition including the background rewards thread, pool updates, and stable flush.
TUI: clarify when tui mode shall exit and how
Currently, `amaru node run` may open the TUI and immediately close it, leaving nothing on the console.
Automatic release process for crates
In preparation for a release of the `amaru` crate on crates.io we need to install the following mechanisms:
Bootstrap should fill db with blocks
### Abstract
Optimize startup time edge case
### Abstract
Block multi-era decoding
### Abstract
Witness Set v12 Decoding
### Abstract
Flat-decoding optimizations
## Abstract
VRF Key Uniqueness Validation - Take 2
Local transaction is not being diffused, despite peers asking?
### What revision are you using?
Add rust docs to TraceContext
As a way to solidify my understanding and softening the learning curve of future devs, the interplay between `tracing`, `opentelemetry` and serialization/deserialization (esp. with the `TraceBuffer`) should be spelt out.
Disentangle ledger store
### Abstract
amaru snapshot create fails with "Permission denied (os error 13)" when no --dist-dir and --snapshot-dir is given
### What revision are you using?
Correct behavior around invalid phase two transactions
Currently, Amaru runs all validation rules when a transaction has `is_valid: false`. This is a divergence from the Haskell implementation, where only the `UTXOW` state transition rule runs (which includes the `UTXO and UTXOS` rules). The `UTXOS` rule branches on `isValid`. The state update is also different, only the UTxO state changes are persisted (and here, it is the collateral spend and return only).
Cardano Node Diversity platform
### Abstract
Optimize the download of transactions in the txsubmission protocol across several peers
### Abstract
Do not revalidate transactions that are already in the Mempool
### Abstract
Consider memory + cpu execution units when computing the available mempool capacity
### Abstract
Perform integrity/sanity check of the ledger & chain db
### Abstract
Update the animations for the simulation
### Abstract
Add ledger checks to the simulation
### Abstract
Add the txsubmission protocol to the simulation
### Abstract
Epoch Transition Benchmark
### Abstract
Granular Ledger Benchmarks
### Abstract
Global Ledger Benchmarks
### Abstract
Enable the TraceBuffer in production
### Abstract
Promote downstream connections to initiator mode on incoming connections (warm -> hot peer transition)
### Abstract
fix amaru-sim test termination
### Abstract