向内扩展,而非向上?使用GPT-5和脆弱提示测试深度引文语境分析 / Scaling In, Not Up? Testing Thick Citation Context Analysis with GPT-5 and Fragile Prompts
1️⃣ 一句话总结
这篇论文通过实验发现,使用GPT-5等大语言模型进行深度引文分析时,提示词的细微变化会系统性地影响模型生成的解读方向和用词,揭示了将AI用作可审查的分析助手时既有机遇也存在风险。
This paper tests whether large language models (LLMs) can support interpretative citation context analysis (CCA) by scaling in thick, text-grounded readings of a single hard case rather than scaling up typological labels. It foregrounds prompt-sensitivity analysis as a methodological issue by varying prompt scaffolding and framing in a balanced 2x3 design. Using footnote 6 in Chubin and Moitra (1975) and Gilbert's (1977) reconstruction as a probe, I implement a two-stage GPT-5 pipeline: a citation-text-only surface classification and expectation pass, followed by cross-document interpretative reconstruction using the citing and cited full texts. Across 90 reconstructions, the model produces 450 distinct hypotheses. Close reading and inductive coding identify 21 recurring interpretative moves, and linear probability models estimate how prompt choices shift their frequencies and lexical repertoire. GPT-5's surface pass is highly stable, consistently classifying the citation as "supplementary". In reconstruction, the model generates a structured space of plausible alternatives, but scaffolding and examples redistribute attention and vocabulary, sometimes toward strained readings. Relative to Gilbert, GPT-5 detects the same textual hinges yet more often resolves them as lineage and positioning than as admonishment. The study outlines opportunities and risks of using LLMs as guided co-analysts for inspectable, contestable interpretative CCA, and it shows that prompt scaffolding and framing systematically tilt which plausible readings and vocabularies the model foregrounds.
向内扩展,而非向上?使用GPT-5和脆弱提示测试深度引文语境分析 / Scaling In, Not Up? Testing Thick Citation Context Analysis with GPT-5 and Fragile Prompts
这篇论文通过实验发现,使用GPT-5等大语言模型进行深度引文分析时,提示词的细微变化会系统性地影响模型生成的解读方向和用词,揭示了将AI用作可审查的分析助手时既有机遇也存在风险。
源自 arXiv: 2602.22359