This is a compact, curated transcript generated with the prompt-native Co-STORM preview contract. The participants below are visibly labeled simulated roles, not independently running agents or the upstream CoStormRunner.
Use the prompt-native Co-STORM preview to explore how an engineering team should evaluate a RAG system. Start with a roundtable, maintain a cited mind map, and let me steer.
Basic fact writer
The original RAG architecture combines a parametric generator with retrieved non-parametric memory. Evaluation therefore needs to observe both the retrieval path and the generated answer, rather than treating the system as one opaque model.[1]
Evaluation scientist
RAGAs separates relevant retrieval, faithful use of context, and generation quality, and proposes metrics that do not require human ground truth for every sample. That makes it useful for frequent evaluation loops, but the paper’s automation goal does not remove the need to validate metric behavior in the target domain.[2]
Systems diagnostician
RAGChecker explicitly evaluates retrieval and generation modules with fine-grained diagnostic metrics. Its purpose is not only to rank systems, but to reveal architecture trade-offs and likely failure locations.[4]
Moderator
The first decision is whether the team needs fast regression feedback, calibrated comparison, or failure diagnosis. ARES adds a distinct path: trained lightweight judges combined with a small human-annotated set through prediction-powered inference.[3]
RAG evaluation
├── System boundary
│ ├── Retrieval
│ └── Generation
├── Quality dimensions
│ ├── Context relevance [2][3]
│ ├── Answer faithfulness [2][3]
│ └── Answer relevance [3]
└── Evaluation intent
├── Fast multidimensional feedback → RAGAs [2]
├── Calibrated system comparison → ARES [3]
└── Fine-grained diagnosis → RAGChecker [4]
These options are suggestions, not a closed menu; any free-form question, constraint, redirection, or request to conclude is valid steering.
Diagnose failures. Show me how the roles disagree about what to measure first.
Systems diagnostician — Primary speaker
Start with component evidence. RAGChecker’s retrieval and generation metrics are designed to reveal patterns that an end-to-end answer score can hide.[4] If retrieved context is irrelevant, generation faithfulness alone cannot repair the pipeline; if retrieval is adequate, the investigation moves to context use and answer construction.
Evaluation scientist — Response
Component detail is useful, but a diagnostic metric is still a measurement model. ARES addresses judge error by pairing trained lightweight judges with a small human-annotated set and prediction-powered inference.[3] The disagreement is therefore not “diagnostics or calibration”; it is how much calibration a diagnostic decision requires.
Moderator
The mind map now contains two linked questions: where did the failure occur, and how trustworthy is the metric that says so? The next move should connect them in a layered evaluation stack rather than select one framework as universally best.
Evaluation intent
└── Fine-grained diagnosis
├── Locate retrieval failure [4]
├── Locate generation failure [4]
└── Validate diagnostic judgments
├── Human-annotated calibration set [3]
└── Domain-transfer check [3]
The same free-form steering rule applies after every round.
record-turn
path, and no upstream Co-STORM parity is claimed.For a longer run that reaches explicit file output and a final report, see the RAG technology research example.