HuggingFace Papers
7/10 signal
EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration
evalmulti-agentresearch
Summary
This paper introduces EduPanel, a three-agent LLM judge for evaluating the pedagogical quality of teaching videos. The system is multimodal, rubric-grounded, and conditioned on a specific learner persona to assess pedagogical fitness rather than universal quality. In an expert study, EduPanel achieves reliability comparable to a median human expert and improves expert scoring accuracy (MAE 0.87 to 0.73). A planted-error study shows that experts can detect unreliable outputs (AUC = 0.77), suggesting EduPanel can serve as an effective assistant rather than a replacement for human evaluators.
Problem
Evaluating the pedagogical quality of teaching videos is difficult to scale, as it requires trained human raters. Existing automatic judges are inadequate because they often fail to incorporate multimodal evidence (what is shown vs. said) and do not account for the fact that teaching quality is inherently learner-dependent. A video effective for an advanced student may be inappropriate for a beginner, a distinction that learner-independent judges cannot capture.
Method
The authors developed EduPanel, a three-agent judge using a gemini-3-flash backbone. Agent 1 (Content Analyst) processes the video to create a content map. Agent 2 (Objective Scorer) scores objective rubric dimensions based on the text report from Agent 1. Agent 3 (Persona Simulator) scores learner-dependent dimensions by role-playing a specified student persona while watching the video. The system was evaluated against 12 expert raters on a benchmark of 12 videos using a two-round protocol (blind, then AI-assisted), which included planted errors to test expert calibration.
Details
Reliability and Error Profile:
- Against an AI-free trimmed human consensus, EduPanel achieved a Mean Absolute Error (MAE) of 0.85, comparable to the median blind human expert's MAE of 0.87. Against the adjudicated ground truth, its MAE was 0.55.
- Performance varied by information modality: EduPanel was most accurate on transcript-grounded dimensions (e.g., A2, C1, E2) and least accurate on visually grounded ones (A1, C4, D2).
- The deployed Gemini backbone exhibited a systematic positive bias, which was larger for visually grounded dimensions (+0.67 average signed bias) than for transcript-only dimensions (+0.24).
- EduPanel identified valid issues missed by all 12 experts in some cases, such as a factual error about DNA replication in a biology video.
Cross-Backbone Comparison:
| Backbone | MAE vs. Consensus | Bias (Visual) | Bias (Transcript) |
|---|---|---|---|
| gemini-3-flash (deployed) | 0.85 | +0.64 | +0.34 |
| GPT (sampled frames) | 0.87 | +0.06 | +0.02 |
Architecture Ablations:
| Configuration | MAE | Δ MAE | Within-1 | Score SD | Correlation (r) |
|---|---|---|---|---|---|
| Full 3-agent | 0.55 | — | 92% | 1.44 | 0.84 |
| Monolithic | 0.55 | -0.05 | 95% | 0.77 | 0.78 |
| No content-analyst | 0.61 | +0.01 | 93% | 1.32 | 0.81 |
| Persona-ablated | 0.75 | +0.15 | 87% | 1.34 | 0.74 |
| Transcript-only | 1.07 | +0.47 | 72% | 0.66 | 0.23 |
- Removing video input (Transcript-only) caused the largest performance drop (MAE +0.47).
- Ablating the learner persona (Persona-ablated) selectively degraded performance on learner-adaptability dimensions (F-family MAE increased from 0.42 to 0.90).
- The Monolithic judge achieved similar MAE but compressed score variance (SD 0.77 vs. 1.44), using a narrower portion of the rating scale.
Persona Sensitivity Analysis:
On a separate set of 32 videos, changing the target learner from school-grade to university-level produced the following mean score changes (Δ):
| Dimension | Judge Δ | Human Δ | Judge Correct Direction |
|---|---|---|---|
| F1 Vocabulary | +1.43 | +0.57 | 25/32 |
| F2 Prerequisite | +1.47 | +0.57 | 23/32 |
| F3 Pacing | +0.36 | +0.20 | 15/32 |
Expert Workflow and Error Detection:
- When used as an assistant, EduPanel improved expert accuracy on 86 non-planted items, reducing MAE from 0.87 to 0.73.
- Inter-rater agreement also increased, with Krippendorff’s α rising from 0.38 to 0.50.
- In a planted-error study, experts could distinguish incorrect AI outputs from correct ones with a pooled ROC AUC of 0.77.
- Experts were not easily misled; the misdirection rate (revising toward a planted error) was 17%.
What's new
This work introduces a novel framework for evaluating teaching videos that is at once multimodal, rubric-grounded, and learner-conditioned, assessing 'pedagogical fitness' for a specific learner rather than absolute quality. It also provides a novel system-level analysis of such a judge within a human expert workflow, using a planted-error study to measure experts' ability to critically verify the AI's outputs.
Limitations
The authors state several limitations. The ground truth was established by a single researcher who had access to the AI's commentary. The expert study was a within-subject design without a no-AI control group, so improvements in expert accuracy cannot be definitively attributed to the AI assistant. The evaluation benchmark is small (12 videos), and the persona sensitivity analysis used only two learner profiles with crowd raters. Finally, the observed modality-dependent bias appears specific to the deployed Gemini backbone and not a general property of LLM judges.
Conclusion
The paper concludes that EduPanel, a multimodal and learner-conditioned judge, can achieve human-level consistency on some evaluation dimensions while showing limitations on others, especially those requiring visual reasoning. The workflow study suggests such judges can be effective decision-support tools, improving expert accuracy and agreement while allowing experts to detect many incorrect AI judgments. The authors propose that as AI-generated educational content scales, evaluating its fitness for the intended learner will become as critical as the generation process itself.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.