doin-node

Status: ACTIVE — the unified participant runtime of the DOIN family. It supersedes the retired standalone doin-optimizer and doin-evaluator clients: optimizer, evaluator/inference-worker, and network-node responsibilities all run as configured roles inside this single process.

doin-node is the runnable node of DOIN, the Decentralized Optimization and Inference Network (older code docstrings abbreviate it DON). One process per machine loads a JSON config that selects, per optimization domain, whether this machine optimizes, evaluates, or only relays; which plugins implement the domain; and how it joins the peer network. Consensus is proof-of-optimization: blocks are generated when verified optimization improvements cross a dynamic threshold, so a chain of blocks doubles as an auditable record of optimization progress, mirrored into an embedded OLAP star schema for analysis.

Role and non-responsibilities

Role: everything needed to participate: the event loops for the optimizer/evaluator roles, HTTP transport and peer protocols, blockchain storage and sync, shared-population coordination, the monitoring dashboard, and the OLAP-on-blockchain analytics layer (src/doin_node/stats/).

Not in this repository:

• Protocol primitives (consensus rules, models, wire schema, crypto, plugin ABCs and entry-point groups) are defined in doin-core and imported here.
• Plugin implementations live in doin-plugins or any external package registering the doin.* entry-point groups; doin-node declares no plugins of its own.
• Domain optimizers/models remain external installable packages (for example predictor and agent-multi) that work locally without DOIN. DOIN extends local optimization collaboratively — it does not absorb domain concerns.

Architecture

Module	Responsibility
src/doin_node/unified.py	UnifiedNode: role dispatch (optimizer loop, shared-population optimizer loop, evaluator loop, gossip/discovery/maintenance loops), champion migration, candidate leasing, dedup
src/doin_node/cli.py	doin-node console script: config parsing/validation, plugin loading, lifecycle
src/doin_node/blockchain/	Chain state, block application, sync
src/doin_node/network/	aiohttp transport, flooding and gossipsub protocols, peer discovery
src/doin_node/storage/	Chain persistence backends (sqlite via aiosqlite, or json)
src/doin_node/stats/	OLAP-on-blockchain: star schema (v3: dim_domain, dim_experiment, fact_round, fact_experiment_summary, fact_chain_optimae), experiment tracker (CSV + SQLite dual-write), chain metrics, SQLite→PostgreSQL sync
src/doin_node/dashboard/	Web monitoring UI served at /dashboard
src/doin_node/scheduling/, validation/, benchmarks/	GPU/job scheduling, input validation, benchmark harnesses

src/doin_node/node.py is an earlier orchestrator kept for reference; the CLI runs UnifiedNode from unified.py.

Requirements

From pyproject.toml:

• Python >=3.10
• doin-core>=0.1.0, aiohttp>=3.9, aiosqlite>=0.20, psutil>=5.9
• Dev extras: pytest, pytest-cov, pytest-asyncio, mypy, ruff
• Domain plugins bring their own requirements (the quadratic reference domain needs only doin-plugins; predictor/trading domains need their external ML stacks)

Installation

Install the three packages in dependency order (no PyPI releases; source installs):

git clone https://github.com/harveybc/doin-core.git
git clone https://github.com/harveybc/doin-plugins.git
git clone https://github.com/harveybc/doin-node.git
pip install -e doin-core -e doin-plugins -e doin-node

Verified 2026-08-10 in the maintainer's Python 3.12 environment: importing doin_node succeeds (version 0.1.0) and the smoke run below works.

Quickstart: single-node quadratic run

Runs one process that optimizes and evaluates the self-contained quadratic reference domain, using the repository-owned config examples/quadratic_single_node.json:

cd doin-node
doin-node --config examples/quadratic_single_node.json
# equivalently: python -m doin_node.cli --config examples/quadratic_single_node.json

Executed 2026-08-10 (30-second smoke): the node created OLAP schema v3 at doin-data-single/olap.db, registered the quadratic domain (optimize=evaluate=true), loaded the simple_quadratic optimizer/evaluator/synthetic plugins, initialized a chain with a genesis block, passed its port self-check on :8470, served the dashboard at http://localhost:8470/dashboard, and shut down cleanly on Ctrl+C.

Useful CLI flags (see doin-node --help): --port, --data-dir, --peers, --identity, --stats-file, --olap-db, --reset-chain, --log-level.

Unified role configuration (per-machine JSON)

Each machine runs one doin-node process with one JSON config. Top-level keys map to UnifiedNodeConfig in src/doin_node/unified.py; every key is optional and defaults are defined there.

Area	Keys
Identity / network	node_label, host, port, data_dir, identity_file, bootstrap_peers, network_protocol (gossipsub or flooding), gossip_heartbeat_interval, discovery_enabled, discovery_interval
Consensus	target_block_time, initial_threshold, acceptance_tolerance, quorum_min_evaluators, quorum_fraction, quorum_tolerance, commit_reveal_max_age, finality_confirmation_depth, external_anchor_interval, require_deterministic_seed
Role loops	optimizer_loop_interval, eval_poll_interval, eval_max_concurrent
Shared population	shared_min_peers, shared_claim_timeout, shared_claim_result_patience, shared_claim_settle_seconds, shared_claim_confirmation_rounds, shared_initialize_before_peers, shared_peer_wait_timeout
Storage / analytics	storage_backend (sqlite or json), db_path, snapshot_interval, prune_keep_blocks, experiment_stats_file, olap_db_path, dashboard_enabled, reset_chain
Economics (optional)	fee_market_enabled, fee_config

The domains list assigns roles and plugins per domain (parsed into DomainRole by src/doin_node/cli.py):

{
  "domain_id": "quadratic",
  "optimize": true,
  "evaluate": true,
  "optimization_plugin": "simple_quadratic",
  "inference_plugin": "simple_quadratic",
  "synthetic_data_plugin": "simple_quadratic",
  "has_synthetic_data": true,
  "optimization_config": { "n_params": 10, "step_size": 0.5 },
  "param_bounds": { "x": [-20.0, 20.0] }
}

Additional per-domain keys: inference_config, synthetic_data_config, synthetic_data_validation, higher_is_better, metric_type, resource_limits, incentive_config, target_performance (convergence stop). Plugin names are resolved through the doin.optimization / doin.inference / doin.synthetic_data entry-point groups defined by doin-core; implementations come from doin-plugins or any external package. A domain without a synthetic-data plugin gets zero consensus weight.

Distributed usage

Two machines / two processes, quadratic domain: examples/quadratic_node_a.json and examples/quadratic_node_b.json (node B bootstraps to node A). The helper scripts/run_two_node_test.sh installs the three packages, launches both nodes locally, and polls /status.

Predictor and trading campaigns: examples/ contains per-machine configs for timeseries-predictor domains, and examples/trading/ holds the per-machine configs (93 files, grouped per campaign phase) used for shared-population trading experiments driven by the external agent-multi package. In a shared-population campaign every participating machine's JSON points at the same domain_id, deterministic-seed/genome contract, and shared-population settings, and each machine differs only in its identity, port, data directory, and role booleans.

Shared-population semantics (champion migration, candidate claim leasing, duplicate-evaluation avoidance, fork-choice tie-breaks, restart recovery) are specified normatively in docs/shared_population_semantics.md and implemented in unified.py. In brief: candidates are claimed through lease-based coordination over HTTP (/api/shared/candidates, .../claim, .../release, .../result) where only the claim owner's heartbeat renews a lease; results deduplicate by transaction id with first-result-wins conflict handling; champions propagate optimistically and at stage boundaries; a restarting node recovers the canonical population from the chain and rejoins quorum before claiming again.

Tests

pip install -e .[dev]
pytest -q

Observed 2026-08-10: pytest -q --collect-only | tail -1 reports 409 tests collected in tests/. (Collection count only; the full suite includes multi-node network tests and takes correspondingly longer.)

Artifacts, outputs, and reproducibility

Per run, under the configured data_dir:

• identity.json — the node's private key (see security notes)
• chain.json (json backend) or chain.db (sqlite backend) — the blockchain
• olap.db — OLAP star schema (schema v3), optionally synced to PostgreSQL via src/doin_node/stats/olap_sync.py

Plus the experiment stats CSV and a .summary.json next to it (path set by experiment_stats_file / --stats-file). The dashboard serves live state at /dashboard.

Reproducibility: with require_deterministic_seed enabled, evaluation seeds are derived from commitment hashes (doin-core deterministic_seed), so verifications are reproducible from on-chain data. The genesis block is fixed, so a fresh experiment means a new domain_id and a new data_dir — never reuse a data directory across incompatible configs. Sample sealed results from a completed campaign are kept under examples/results/.

Security and safety notes

• identity.json holds the node's private key: keep it out of version control and restrict file permissions.
• Consensus hardening (from doin-core, wired in here): commit-reveal for optimae, quorum verification with tolerance, asymmetric reputation, resource-limit validation, finality checkpoints, external anchoring, deterministic per-evaluator seeds.
• The HTTP transport is plain HTTP intended for trusted/private networks; do not expose node ports to untrusted networks.
• No exchange, broker, or API credentials are required or read by this repository. Trading domains operate purely on historical/synthetic data through simulation and backtesting; no live orders are placed. Nothing here is financial advice.

Limitations and legacy notes

• Version 0.1.0 (alpha); no PyPI releases; wire compatibility between versions not guaranteed — run matching versions on all peers.
• The standalone doin-optimizer and doin-evaluator clients are retired and are not required for any current deployment; their roles are the optimize / evaluate booleans of this package's domain config.
• Both gossipsub and flooding protocols are implemented; the in-repo example configs (including all shared-population trading examples) use flooding.
• src/doin_node/node.py is a pre-unified orchestrator kept for reference only.

Related repositories and docs

• doin-core — protocol primitives this node implements; its docs/ folder holds the network/security/ scalability papers
• doin-plugins — plugin implementations loaded by entry-point name
• predictor, agent-multi — external domain packages used by the predictor/trading domains
• docs/shared_population_semantics.md — normative shared-population specification

License

Declared MIT in pyproject.toml; the repository does not currently ship a standalone LICENSE file.