Zhixin:面向基于 LLM 的精神障碍诊断的数据集构建与指令微调
Zhixin: dataset construction and instruction fine-tuning for LLM-based mental disorder diagnosis
Zhixin 数据集构建与指令微调面向基于 LLM 的精神障碍诊断,包含 0-shot/3-shot 提示、DSM-5 知识与 RAG 提示及正则结构化输出方案。专家人工标注按安全性、专业性和总体评分对模型回答进行评价,并以独立专家盲法复评处理分歧。附录还列出精神分裂症患者主诉示例,用于说明症状差异。
Appendix
Prompt examples
For in-context learning, we implement 0-shot (no example) scenario and 3-shot (3 examples) scenarios, with corresponding prompts presented in Tables 5 and 6. As reference, we also give the English translation.
DSM-5
DSM-5 strives to fulfill the need of clinicians, patients, families, and researchers for a clear and concise description of each mental disorder, which has been operationalized using diagnostic criteria that are supplemented by dimensional measures of severity and is accompanied by a digest of information about the diagnosis, including risk factors and culture-and sex- and gender-related issues.
For each mental disorder, DSM-5 introduces Diagnostic Features, Development and Course, Risk and Prognostic Factors, Differential Diagnosis, Comorbidity, etc5. Diagnostic Features provides descriptive text illustrating the use of the criteria and includes key points on their interpretation. Development and Course describes the typical lifetime patterns of presentation and evolution of the disorder. Risk and Prognostic Factors includes a discussion of factors thought to contribute to the development of a disorder. Differential Diagnosis discusses how to differentiate the disorder from other disorders that have some similar presenting characteristics. Comorbidity includes descriptions of mental disorders and other medical conditions likely to co-occur with the diagnosis.
Expert psychologist is employed to review the DSM-5 text and compliant samples in our dataset, then select three components that are most relevant to the diagnosis of mental disorders. Namely, we use symptom criteria, exclusion conditions and Development and Course in DSM-5 Knowledge prompt. The prompts of DSM-5 Knowledge and DSG-5 RAG are presented in Tables 7 and 8. As reference, we also give the English translation.
Structuralization
Given the model outputs, we apply regular-expression patterns to convert them into a structured format. For each mental disorder category, we design a regex that captures all acceptable phrasings and maps them to a single standardized label, enabling automatic evaluation. An example of the structuralization process is shown in Table 9.
Human annotation scheme
Annotators will need to score and annotate the generated responses from the following aspects: Safety, Professionalism, and Overall Rating as shown in Table 10. The guidance and corresponding examples are given to the annotators.
We systematically address inter-annotator disagreement by first identifying three types of discrepancies: theoretical perspective-based divergence, scoring standard misinterpretation, and subjective bias. For theoretical disagreements, we organize expert workshops to align on framework applicability. For standard-related conflicts, we refine operationalized criteria with illustrative examples to ensure consistent interpretation. And for subjective biases, we implement blind re-evaluation by independent experts.
Chief complaint examples
Since the symptoms mental disorders vary from person to person. We present the examples of Schizophrenia as shown in Table 11. For instance, patient 1 has a history of bipolar disorder and is now diagnosed with schizophrenia. While the main symptom of patient 2 is paranoia. As reference, we also give the English translation.
来源:Nature 健康护理主题 · nature.com