常青:面向语义聚合的高效声明验证系统 / Evergreen: Efficient Claim Verification for Semantic Aggregates
1️⃣ 一句话总结
针对大语言模型生成的语义聚合结果可能包含不准确陈述的问题,本文提出Evergreen系统,通过将验证过程转化为可优化的语义查询任务,结合早期停止、相关性排序等高效策略,在保证验证质量的同时大幅降低计算成本与延迟。
With recent semantic query processing engines, semantic aggregation has become a primitive operator, enabling the reduction of a relation into a natural language aggregate using an LLM. However, the resulting semantic aggregate may contain claims that are not grounded in the underlying relation. Verifying such claims is challenging: they often involve quantifiers, groupings, and comparisons over relations that far exceed LLM context windows and require a costly combination of semantic and symbolic processing. We present Evergreen, a system that recasts claim verification as a semantic query processing task with tailored optimizations and provenance capture. Evergreen compiles each claim into a declarative semantic verification query and executes it on the same engine that produced the aggregate. To reduce cost and latency, Evergreen avoids unnecessary LLM calls through verification-aware optimizations (early stopping, relevance sorting, and estimation with confidence sequences) and general-purpose optimizations for semantic queries (operator fusion, similarity filtering, and prompt caching). Each verdict is accompanied by citations that identify a minimal set of tuples justifying the result, with semantics based on semiring provenance for first-order logic. On a benchmark of real-world restaurant review datasets reflecting production-inspired workloads, Evergreen achieves excellent verification quality (F1 = 1.00) with a strong LLM while reducing cost by 3.2x and latency by 4.0x compared to unoptimized verification. Even with a significantly weaker LLM, Evergreen outperforms a strong LLM-as-a-judge baseline in F1 at 48x lower cost and 2.3x lower latency. Relative to a retrieval-augmented agent, Evergreen compares favorably in F1 and latency with similar cost when both use a strong LLM; yet, with a much weaker LLM, it achieves the same F1 at 63x lower cost and 4.2x lower latency.
常青:面向语义聚合的高效声明验证系统 / Evergreen: Efficient Claim Verification for Semantic Aggregates
针对大语言模型生成的语义聚合结果可能包含不准确陈述的问题,本文提出Evergreen系统,通过将验证过程转化为可优化的语义查询任务,结合早期停止、相关性排序等高效策略,在保证验证质量的同时大幅降低计算成本与延迟。
源自 arXiv: 2604.26180