BERTScore and ROUGE-L Implementation Summary

✅ What Was Added

1. New Metrics in `evaluation_metrics.py`

calculate_bertscore() - Semantic similarity using BERT embeddings
calculate_rouge_l() - Longest Common Subsequence matching
calculate_bleu_score() - N-gram overlap scoring
print_advanced_metrics_table() - Display advanced metrics

2. New Test File: `test_advanced_metrics.py`

TestBERTScore class - 4 test cases
TestROUGEL class - 4 test cases
TestBLEU class - 3 test cases
TestAdvancedMetricsIntegration - Integration tests
TestMetricsOnRealQueries - Tests with real RAG responses

3. Updated Dependencies in `test_requirements.txt`

bert-score>=0.3.13
rouge-score>=0.1.2
nltk>=3.8.1

4. Documentation

ADVANCED_METRICS_GUIDE.md - Comprehensive guide to all metrics

🚀 How to Use

Install New Dependencies

pip install bert-score rouge-score nltk

Run BERTScore and ROUGE-L Tests

# Run all advanced metric tests
pytest tests/test_advanced_metrics.py -v

# Run only BERTScore tests
pytest tests/test_advanced_metrics.py::TestBERTScore -v

# Run only ROUGE-L tests
pytest tests/test_advanced_metrics.py::TestROUGEL -v

# Run only BLEU tests
pytest tests/test_advanced_metrics.py::TestBLEU -v

Use in Python Code

from tests.evaluation_metrics import GenerationMetrics

# Example: Evaluate a physics answer
reference = "Newton's first law states that objects remain at rest unless acted upon by force."
response = "The first law of Newton says objects stay still without external force."

# Calculate BERTScore
bert_scores = GenerationMetrics.calculate_bertscore(response, reference)
print(f"BERTScore F1: {bert_scores['bertscore_f1']:.4f}")

# Calculate ROUGE-L
rouge_scores = GenerationMetrics.calculate_rouge_l(response, reference)
print(f"ROUGE-L F1: {rouge_scores['rouge_l_f1']:.4f}")

# Calculate BLEU
bleu_score = GenerationMetrics.calculate_bleu_score(response, reference)
print(f"BLEU: {bleu_score:.4f}")

📊 What These Metrics Measure

Metric	What It Measures	When to Use	Score Range
BERTScore	Semantic similarity (meaning)	Paraphrasing, semantic correctness	0-1 (>0.85 excellent)
ROUGE-L	Word sequence overlap (structure)	Summarization, factual accuracy	0-1 (>0.70 excellent)
BLEU	N-gram precision (exact words)	Translation, exact matching	0-1 (>0.40 excellent)

🎯 Example Scores

Query: "What is Newton's first law of motion?"

Reference: "Newton's first law states that an object at rest stays at rest and an object in motion stays in motion unless acted upon by an external force."

Response A (Good): "The first law of Newton states objects remain at rest or in motion unless an external force acts on them."

BERTScore: 0.88 ✅ (Same meaning)
ROUGE-L: 0.65 ✅ (Good overlap)
BLEU: 0.35 ✅ (Moderate n-gram match)

Response B (Poor): "Energy cannot be created or destroyed in a system."

BERTScore: 0.45 ❌ (Different meaning)
ROUGE-L: 0.05 ❌ (No overlap)
BLEU: 0.01 ❌ (No matching n-grams)

📈 Integration with Existing Tests

The advanced metrics work alongside your existing metrics:

from tests.evaluation_metrics import EvaluationReport

evaluation = EvaluationReport.evaluate_single_query(
    query="What is Newton's first law?",
    response=generated_response,
    context=retrieved_context,
    retrieved_docs=docs_with_relevance,
    reference=reference_answer,      # ← Add this
    include_advanced=True             # ← Enable advanced metrics
)

# Now you get ALL metrics:
print(evaluation)
# {
#   'mrr': 1.0,
#   'hit@10': 1.0,
#   'faithfulness': 0.85,
#   'relevancy': 0.78,
#   'bertscore_f1': 0.88,     ← NEW
#   'rouge_l_f1': 0.65,       ← NEW
#   'bleu': 0.35              ← NEW
# }

📋 Complete Test Output Example

When you run pytest tests/test_advanced_metrics.py -v -s:

tests/test_advanced_metrics.py::TestBERTScore::test_bertscore_perfect_match PASSED
tests/test_advanced_metrics.py::TestBERTScore::test_bertscore_semantic_similarity PASSED
tests/test_advanced_metrics.py::TestROUGEL::test_rouge_l_perfect_match PASSED
tests/test_advanced_metrics.py::TestROUGEL::test_rouge_l_partial_match PASSED
tests/test_advanced_metrics.py::TestBLEU::test_bleu_perfect_match PASSED

=== Advanced Metrics Evaluation ===
BERTScore F1: 0.8756
ROUGE-L F1: 0.6543
BLEU Score: 0.3521

🔍 Where Are the Tests?

tests/
├── test_advanced_metrics.py        ← NEW: BERTScore, ROUGE-L, BLEU tests
├── evaluation_metrics.py           ← UPDATED: Added 3 new metric functions
├── test_retrieval.py               ← Existing: MRR, Hit@k
├── test_generation.py              ← Existing: Faithfulness, Relevancy
└── test_system_integration.py      ← Existing: End-to-end tests

💡 Key Differences from Original Metrics

Original Metrics	Advanced Metrics
Faithfulness - Checks if response uses context	BERTScore - Checks semantic similarity to reference
Relevancy - Checks if response addresses query	ROUGE-L - Checks word sequence overlap
MRR - Ranking of retrieved docs	BLEU - N-gram precision matching

Both sets complement each other:

Original metrics = How well RAG system works
Advanced metrics = How good the generated text is

📚 Quick Reference

Good Scores for NCERT Physics Q&A:

BERTScore F1: > 0.80
ROUGE-L F1: > 0.65
BLEU: > 0.30
Faithfulness: > 0.75
Relevancy: > 0.70
MRR: > 0.80

Installation Check:

python -c "import bert_score; import rouge_score; import nltk; print('All packages installed!')"

Run Single Test:

pytest tests/test_advanced_metrics.py::TestBERTScore::test_bertscore_perfect_match -v

🎓 Read More

For detailed explanation of each metric, see:

ADVANCED_METRICS_GUIDE.md - Full guide
README_TESTING.md - General testing guide
TESTING_QUICKSTART.md - Quick start

✅ Summary

You now have complete evaluation coverage for your RAG system:

✅ Retrieval Metrics (MRR, Hit@k, Precision@k) ✅ Generation Quality (Faithfulness, Relevancy)
✅ Semantic Similarity (BERTScore) ← NEW ✅ Text Overlap (ROUGE-L) ← NEW ✅ N-gram Matching (BLEU) ← NEW ✅ Full test suite with 15+ test cases ✅ Integration with existing evaluation pipeline ✅ Comprehensive documentation

Run pytest tests/test_advanced_metrics.py -v to get started!

FilesExpand file tree

BERTSCORE_ROUGEL_README.md

Latest commit

History