EduRAG/TESTING_SUMMARY.py at main · super-sg/EduRAG · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
"""
Summary of Testing Framework for EduRAG
========================================

## What Was Created

### 1. Test Structure (tests/ directory)
   ├── __init__.py                    # Package initialization
   ├── conftest.py                    # Pytest configuration & fixtures
   ├── test_queries.py                # Test dataset (15 NCERT Physics queries)
   ├── evaluation_metrics.py          # All metrics (MRR, Faithfulness, BERTScore, ROUGE-L, BLEU)
   ├── test_retrieval.py              # Retrieval component tests
   ├── test_generation.py             # Generation component tests
   ├── test_advanced_metrics.py       # BERTScore, ROUGE-L, BLEU tests (NEW!)
   └── test_system_integration.py     # End-to-end system tests

### 2. Evaluation Script
   run_evaluation.py                  # Complete evaluation runner (generates paper-style reports)

### 3. Documentation
   ├── README_TESTING.md              # Comprehensive testing guide
   └── TESTING_QUICKSTART.md          # Quick start guide

### 4. Configuration
   ├── pytest.ini                     # Pytest configuration (updated)
   └── test_requirements.txt          # Test dependencies (already exists)


## How to Use

### Step 1: Install Dependencies
```powershell
pip install pytest pytest-cov numpy
```

### Step 2: Choose Your Testing Approach

#### A) Quick Unit Tests (Fast)
```powershell
pytest tests/test_retrieval.py -v
pytest tests/test_generation.py -v
```

#### B) Complete Evaluation (Like the Paper)
```powershell
python run_evaluation.py
```
This generates tables similar to Tables III & IV in the research paper.

#### C) All Tests
```powershell
pytest
```

#### D) Skip Slow Tests
```powershell
pytest -m "not slow"
```


## What Gets Tested

### Retrieval Metrics (Table II style)
- MRR (Mean Reciprocal Rank)
- Hit@k (Hit@1, Hit@5, Hit@10)
- Precision@k

### Generation Metrics (Table III & IV style)
- Faithfulness Score (0-1)
- Relevancy Score (0-1)
- Response Length (word count)

### Test Queries (Table I style)
10 predefined educational queries covering:
- Historical analysis
- Technical concepts
- Examples and applications
- Process analysis
- System architecture


## Example Output

When you run `python run_evaluation.py`:

╔════════════════════════════════════════════════════════════════════════════════════════════════╗
║                           EVALUATION RESULTS - RAG SYSTEM PERFORMANCE                          ║
╠════════════╦══════════╦══════════╦═══════════════╦═══════════════╦══════════╣
║ Query ID   ║   MRR    ║  Hit@10  ║  Faithfulness ║   Relevancy   ║  Length  ║
╠════════════╬══════════╬══════════╬═══════════════╬═══════════════╬══════════╣
║ Q1         ║  1.0000  ║  1.0000  ║    0.8500     ║    0.7200     ║   145    ║
║ Q2         ║  0.5000  ║  1.0000  ║    0.7800     ║    0.8100     ║   132    ║
║ Q3         ║  0.3333  ║  1.0000  ║    0.6900     ║    0.7500     ║   118    ║
║ ...        ║   ...    ║   ...    ║      ...      ║      ...      ║   ...    ║
╠════════════╬══════════╬══════════╬═══════════════╬═══════════════╬══════════╣
║ AVERAGE    ║  0.6667  ║  1.0000  ║    0.7733     ║    0.7600     ║   131    ║
╚════════════╩══════════╩══════════╩═══════════════╩═══════════════╩══════════╝


## Key Features

✓ Paper-style evaluation metrics
✓ Automated testing with pytest
✓ Comprehensive metric calculations
✓ Visual evaluation reports
✓ Configurable test queries
✓ Coverage reporting
✓ Fast and slow test separation
✓ Integration and unit test markers


## Metrics Interpretation

Good Scores:
- MRR: > 0.7
- Hit@10: > 0.9
- Faithfulness: > 0.7
- Relevancy: > 0.7

Your System Performance:
→ Run `python run_evaluation.py` to see your scores!


## Next Steps

1. Run evaluation: `python run_evaluation.py`
2. Review results and identify weak areas
3. Adjust parameters (chunk size, prompt, model) to improve
4. Add more test queries to `tests/test_queries.py`
5. Compare different configurations

For detailed information, see:
- TESTING_QUICKSTART.md (quick guide)
- README_TESTING.md (comprehensive guide)
"""

if __name__ == "__main__":
    print(__doc__)