LCRO — Logical Consistency Regularized Optimization

LCRO augments Direct Preference Optimization (DPO) with a step-level logical consistency reward, reducing contradiction rates in LLM reasoning chains without sacrificing task accuracy.

Paper: LCRO: Logical Consistency Regularized Optimization for Coherent Multi-Step Reasoning in Large Language Models

---

Key Idea

Standard DPO labels preference pairs using only final-answer correctness. This ignores whether the reasoning chain is internally coherent — and in practice, DPO training can inflate the rate of logical contradictions between steps. LCRO fixes this by enriching the reward:

r_logic(τ) = r_task(τ) + β_logic · (C_τ − λ · V_τ)

• r_task — binary correctness reward (1 if answer is correct, 0 otherwise)
• C_τ — causal consistency score: how much each reasoning step causally drives the correct answer
• V_τ — verifier contradiction score: DeBERTa-v3 NLI classifier detecting contradictions between adjacent steps
• β_logic = 0.5, λ = 0.3 (set in configs/lcro_config.json)

Preference pairs are labeled with this richer reward; the DPO loss is unchanged. LCRO is a drop-in replacement for the pair-labeling step in any DPO pipeline.

---

Results Summary

Model	Dataset	Vanilla DPO CR	LCRO CR	Δ CR
Qwen2-7B	GSM8K	0.0316	0.0285	−9.8%
Qwen2-7B	MATH	0.0334	0.0348	—
Mistral-7B	GSM8K	0.0429	0.0413	−3.7%
Mistral-7B	MATH	0.0462	0.0388	−16.1%
Falcon3-3B	BBH	0.0137	0.0121	−11.7%

CR = Contradiction Rate (lower is better). See the paper for full accuracy results and ablation tables.

---

Repository Structure

lcro/
├── configs/
│   ├── lcro_config.json           # single source of truth for all hyperparameters
│   ├── eval_config.json
│   └── verifier_config.json
├── scripts/
│   ├── run_preprocess.py          # Step 1: preprocess datasets
│   ├── run_train_verifier.py      # Step 2: train DeBERTa verifier
│   ├── run_generate_trajectories.py  # Step 3: sample K trajectories per problem
│   ├── run_score_trajectories.py     # Step 4: compute C_τ, V_τ, r_logic
│   ├── run_build_pairs.py            # Step 5: construct preference pairs
│   ├── run_ref_logprobs.py           # Step 6: cache reference log-probs
│   ├── run_dpo_training.py           # Step 7: DPO fine-tuning (LCRO / vanilla / ablations)
│   ├── run_sft_training.py           # Step 7b: SFT baseline
│   ├── run_evaluation.py             # Step 8: evaluate on test sets
│   └── run_ablations.py              # Step 9: ablation sweeps
├── src/
│   ├── dpo/
│   │   ├── lcro_trainer.py    # LCROTrainer (subclasses TRL DPOTrainer)
│   │   └── ref_logprobs.py    # reference log-prob caching
│   ├── verifier/
│   │   ├── model.py           # DeBERTa NLI classifier
│   │   ├── scorer.py          # C_τ, V_τ, r_logic computation
│   │   └── trainer.py         # verifier fine-tuning loop
│   ├── trajectory/
│   │   ├── generator.py       # trajectory sampling + answer extraction
│   │   └── deduplicator.py    # remove duplicate trajectories
│   ├── evaluation/
│   │   ├── evaluator.py       # accuracy, CR, causal delta
│   │   └── ablation.py        # ablation runner
│   ├── visualization/         # plotting utilities
│   ├── checkpoint_utils.py    # CheckpointManager + TimeoutGuard
│   ├── logging_utils.py       # JSONL structured logger
│   └── reproducibility.py     # seed management
├── setup.sh                   # one-time setup (conda env + datasets)
├── download_models.sh         # download model weights from HuggingFace
├── test_suite.py              # 10-check verification suite
└── environment.yml            # conda environment spec

---

Quick Start

1. Clone and set up

git clone <repo-url> && cd lcro
bash setup.sh            # creates conda env 'lcro', downloads datasets, runs tests
bash download_models.sh  # downloads ~30 GB of model weights

2. Run the pipeline

Activate the environment once:

conda activate lcro
export TOKENIZERS_PARALLELISM=false

Then run each step in order:

# Step 1 — preprocess datasets
python scripts/run_preprocess.py \
    --config configs/lcro_config.json \
    --output_dir data/processed

# Step 2 — train DeBERTa verifier
python scripts/run_train_verifier.py \
    --config configs/lcro_config.json \
    --data_dir data/processed \
    --output_dir checkpoints/verifier

# Steps 3–7: per model x dataset
MODEL=qwen
DATASET=gsm8k

python scripts/run_generate_trajectories.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json \
    --data_dir data/processed --output_dir data/trajectories

python scripts/run_score_trajectories.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json \
    --traj_dir data/trajectories \
    --verifier_ckpt checkpoints/verifier/best_model.pt \
    --output_dir data/scored

python scripts/run_build_pairs.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json \
    --scored_dir data/scored --output_dir data/pairs

python scripts/run_ref_logprobs.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json \
    --pairs_dir data/pairs --output_dir data/ref_logprobs

# Step 7a — LCRO DPO training
python scripts/run_dpo_training.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json \
    --pairs_dir data/pairs \
    --ref_cache_dir data/ref_logprobs \
    --verifier_ckpt checkpoints/verifier/best_model.pt \
    --output_dir checkpoints/dpo

# Step 7b — SFT baseline (optional)
python scripts/run_sft_training.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json

# Step 8 — Evaluate
python scripts/run_evaluation.py \
    --model $MODEL --dataset $DATASET \
    --config configs/lcro_config.json \
    --checkpoint_dir checkpoints/dpo/lcro \
    --verifier_ckpt checkpoints/verifier/best_model.pt \
    --data_dir data/processed --log_dir logs/app

3. Vanilla DPO and ablations

Pass --variant to run_dpo_training.py:

# Vanilla DPO (correctness reward only)
python scripts/run_dpo_training.py --model qwen --dataset gsm8k \
    --variant vanilla_dpo ...

# Ablation: remove consistency term
python scripts/run_dpo_training.py --model qwen --dataset gsm8k \
    --variant no_consistency ...

---

Configuration

All hyperparameters live in configs/lcro_config.json. Key values:

Parameter	Default	Description
reward.beta_logic	0.5	Weight of the logic regularizer
reward.lambda_contradiction	0.3	Down-weight of contradiction penalty vs consistency score
training.beta	0.1	DPO KL penalty
training.learning_rate	5e-7	Learning rate
generation.K	4	Number of trajectories sampled per problem
generation.temperature	0.7	Sampling temperature
lora.r	16	LoRA rank
lora.alpha	32	LoRA alpha

To use a locally cached model, add a local_path key to any entry in the models array:

{"slug": "qwen", "hf_name": "Qwen/Qwen2-7B-Instruct", "local_path": "/path/to/local/model"}

---

Environment

Python 3.10
torch 2.0.1 + CUDA 11.8
transformers 4.45.0
peft 0.10.0
trl 0.8.6
datasets 2.18.0

Full spec: environment.yml. For CPU-only testing:

pip install torch==2.0.1  # CPU only

---

Verification

Run the test suite (no GPU required) after installation:

conda activate lcro
python test_suite.py

All 10 checks should pass. The suite covers reproducibility, logging, checkpoint management, reward computation, answer extraction, and evaluation metrics.