检索增强生成模型在临床症状与ICD-10-CM编码匹配中的应用
A retrieval-augmented local large language model for ICD-10-CM terminology mapping of clinical signs and symptoms
该研究提出了一种检索增强生成模型,用于提高临床症状与ICD-10-CM编码的匹配准确性。通过多嵌入模型检索候选代码,并结合生成模型进行上下文决策,结果显示该方法在匹配准确性和减少无效代码生成方面优于基线模型。
Introduction
Early identification of signs and symptoms (S&S) is a cornerstone of disease prevention and cancer control. Subtle or nonspecific manifestations—such as hematuria, persistent cough, unexplained weight loss, or chronic fatigue—often precede diagnostic confirmation and thus offer a critical window for timely intervention. In oncology, early symptom recognition directly influences diagnostic interval length, disease stage at presentation, and treatment outcomes. Studies have shown that earlier interpretation of S&S substantially improves survival across multiple cancer types, including colorectal, lung, and bladder cancers1,2,3. However, these valuable indicators are typically buried within unstructured clinical narratives in the electronic health record (EHR), making them inaccessible for computational use in large-scale analyses or predictive modeling4,5. This gap underscores the need for automated methods to extract, standardize, and encode S&S data.
The International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM), provides a globally adopted framework for representing diseases, signs, and symptoms in a hierarchical structure6. Developed by the Centers for Medicare & Medicaid Services (CMS) and the National Center for Health Statistics (NCHS), it replaced ICD-9-CM to provide more detailed and clinically accurate information for patient care, research, and reimbursement. ICD-10-CM enables consistent communication across healthcare settings, supports quality measurement, and facilitates public health reporting and epidemiological studies. Converting free-text S&S into ICD-10-CM terminology may support structured symptom representation for research, phenotyping, and future predictive-modeling applications.
Structurally, each ICD-10-CM code contains three to seven alphanumeric characters (Table 1). The first character is always a letter representing the general chapter or body system; the second character is numeric, and the third through seventh characters may be either letters or numbers. A decimal point follows the third character. The fourth to sixth characters provide additional clinical details such as etiology, anatomical site, and severity, while a seventh character, when required, indicates the type of encounter (e.g., initial, subsequent, or sequela)7.
ICD-10-based structured data have proven useful in building diagnostic and prognostic models across a range of diseases8. Machine-learning algorithms trained on longitudinal ICD-coded EHRs have been used to predict disease onset, progression, and comorbidity networks in chronic and oncologic conditions. In cancer informatics, ICD-10-coded symptom trajectories can reveal precursor patterns of malignancy9,10,11,12. For example, in a Danish registry-based study of over 6.9 million individuals (including 23,592 with Pancreatic cancer), sequential ICD-10 coded symptom and disease trajectories (e.g., “cough → jaundice → intestinal obstruction”) were identified among patients who developed pancreatic cancer12. Therefore, accurate and consistent mapping of S&S into ICD-10-CM terminology may provide useful, structured inputs for future symptom-trajectory and prediction-modeling research.
Despite its clinical utility, manual ICD-10-CM assignment remains labor-intensive, requiring specialized coders with extensive medical and terminological knowledge. The growing volume of clinical documentation has prompted research into automatic and semi-automatic coding approaches. Early machine-learning methods, such as support vector machines and rule-based classifiers, were developed to map diagnoses extracted from discharge summaries to ICD-10 codes13. Although these systems achieved moderate success, they struggled with linguistic variability, context dependence, and polysemous terminology typical of medical text. More recent deep-learning approaches, including convolutional and recurrent neural architectures, improved semantic representation of medical text14,15,16. However, they are highly dependent on the quality of training data and struggle with unseen or imbalanced labels, lack the symbolic reasoning coders use to integrate contextual and contradictory information, and face limits processing long, redundant clinical notes that exceed Transformer input capacities17,18.
Recent large language models (LLMs), trained on trillions of tokens of general and biomedical text, demonstrate powerful zero-shot reasoning and contextual understanding. When prompted appropriately, LLMs can identify S&S in narrative text and generate corresponding ICD-10-CM codes19. However, their generative flexibility also introduces unreliability: models may produce plausible but invalid codes, misinterpret context, or invent code descriptions absent from official records20,21. In clinical NLP tasks, these hallucinations not only reduce accuracy but also erode user trust, highlighting the need for mechanisms that constrain generation to factual, evidence-grounded information.
Retrieval-Augmented Generation (RAG) offers a compelling framework to mitigate hallucination by coupling generative models with external knowledge retrieval. Rather than relying solely on static, pretrained parameters, RAG encodes both user queries and documents into semantic embeddings—dense vector representations generated by models such as Bert—and stores them in vector databases for efficient similarity search. During inference, the system retrieves the most relevant passages based on cosine similarity and integrates them into the prompt, enabling the language model to generate grounded and verifiable responses. This embedding-driven retrieval mechanism significantly reduces hallucination and improves adaptability across specialized domains such as biomedicine and legal text analysis22,23,24.
Recent studies have begun applying retrieval-augmented and retrieval-supported LLM methods to automated ICD coding. Klang et al. evaluated RAG-enhanced LLMs for ICD-10-CM coding from emergency department clinical records, while MedCodER framed medical coding as an extraction, retrieval, and re-ranking task using disease diagnoses, supporting evidence, and candidate ICD codes25. Puts et al. developed an ICD-10 coding assistant that combines lead-term extraction with description-based code selection using retrieved ICD code information, and RAG-Coding uses structured ICD-10-CM external knowledge, including the tabular list and coding guidelines, to support automated medical coding26. Although these studies demonstrate the promise of retrieval-grounded ICD coding, they primarily focus on encounter-level diagnosis coding, lead-term coding, or broad medical coding. In contrast, the present study addresses the narrower task of mapping explicitly documented clinical signs and symptoms to ICD-10-CM codes. This distinction is important because S&S mapping requires preserving documented symptom evidence without inferring diagnoses, and the relevant output is often an analytic symptom/sign code rather than a complete billing code set for an encounter.
Study Objective
Building upon these advances, the present study proposes a retrieval-augmented large language model (RAG-LLM) designed to map extracted S&S from clinical notes to ICD-10-CM codes with higher precision and reliability. The system integrates multiple domain-specific embedding models for candidate retrieval and employs a large-scale generative model for contextual decision-making. By combining retrieval accuracy with generative flexibility, this hybrid approach aims to.
-
1.
Improve ICD-10-CM terminology-mapping accuracy for clinical S&S;
-
2.
Reduce invalid ICD-10-CM code generation and improve raw description consistency; and.
-
3.
Support future use of structured symptom data in symptom-based phenotyping and prediction-modeling research.
This work thus contributes to both methodological innovation and translational informatics by demonstrating that grounding LLMs through retrieval can enable more trustworthy, reproducible, and proof-of-concept ICD-10-CM terminology mapping for symptom-based phenotyping.
Method
Human-labeled reference standard
We annotated 1,000 clinical notes from MTsamples with explicitly documented signs and symptoms (S&S) and their corresponding ICD-10-CM codes27. Each clinical note was independently annotated by two annotators: a nurse and a clinical informatician with experience in clinical documentation review and ICD-10-CM coding. Annotators were instructed to identify only signs and symptoms explicitly documented in the note and to avoid inferring diagnoses, undocumented clinical conditions, historical findings, or negated findings. Each extracted S&S was assigned the best corresponding ICD-10-CM code based on the symptom wording and surrounding note context. Disagreements between the two annotators were reviewed and resolved through discussion to create the final reference standard.
Cohen’s kappa was calculated at the extracted S&S/code-label level within each annotation batch. The mean Cohen’s kappa was 0.75 ± 0.06 across five 200-note annotation batches, indicating substantial agreement. For the present proof-of-concept evaluation, we randomly selected 20% of the annotated notes as the test set. The finalized human-annotated S&S and ICD-10-CM code labels in this test set served as the reference standard for evaluating model performance.
Large language model and embedding models
We applied GPT-OSS-120B as the base large language model for the ICD-10-CM mapping task. This open-weight model was deployed locally, allowing the mapping pipeline to be evaluated without transmitting clinical text to an external commercial API. The same base model was used for both the LLM-only baseline and the RAG-LLM model so that the effect of retrieved ICD-10-CM candidate information could be evaluated under a controlled setting.
The retrieval component used eight embedding models from HuggingFace, including five medical or clinical embedding models and three general-purpose embedding models. The medical or clinical embedding models were MedBERT, ClinicalBERT, SapBERT, Bge Large Medical, and Medical Embedded v5. The general-purpose embedding models were All-MiniLM-L6-v2, Sentence-t5-xxl, and Gtr-t5-xxl28,29,30,31,32,33,34,35. These models were used to convert ICD-10-CM code descriptions and extracted S&S terms into dense vector representations for similarity-based retrieval (Table 2). The eight-model retrieval configuration was selected as a design choice for this proof-of-concept study; alternative numbers or combinations of embedding models were not evaluated through ensemble ablation.
ICD-10-CM retrieval database and candidate generation
We downloaded the complete ICD-10-CM code set from the official Centers for Disease Control and Prevention (CDC) website. The reference catalogue contained 98,186 ICD-10-CM codes. Each ICD-10-CM code title was embedded separately using each of the eight embedding models, and the resulting vectors were stored with the corresponding ICD-10-CM code and code title.
For each extracted S&S, the same embedding model was used to convert the S&S term into a query vector. Cosine similarity was then calculated between the S&S vector and all ICD-10-CM code-title vectors generated by that embedding model. The top five ICD-10-CM codes with the highest cosine similarity scores were retrieved from each embedding model. Duplicate ICD-10-CM codes retrieved by multiple embedding models were then removed before the candidate list was provided to the RAG-LLM, resulting in a deduplicated list of up to 40 unique candidate codes (Fig. 1).
How the Retrieval-Augmented Generation (RAG) component retrieves candidate ICD-10-CM codes for input S&S.
Compared systems
We evaluated three systems: a retrieval-only voting baseline, an LLM-only baseline, and the final RAG-LLM model.
The retrieval-only voting baseline used only the retrieval results and did not use LLM-based final selection. For each S&S, the top-ranked ICD-10-CM candidate from each of the eight embedding models was collected, and the final retrieval-only code was selected by majority vote across the eight top-1 candidates. The top-1 candidate was used because it represented each embedding model’s highest-ranked retrieval; including all top-5 candidates in an unweighted voting scheme would give equal voting influence to lower-ranked candidates and could dilute agreement across models. If multiple codes were tied for the highest vote count, the first code returned by the Pandas mode function was selected, corresponding to the lexicographically first tied ICD-10-CM code. This configuration was used as a simple retrieval-only comparator rather than as an empirically optimized retrieval strategy. The LLM-only baseline used GPT-OSS-120B without retrieved ICD-10-CM candidates. The model received the extracted S&S and the corresponding clinical note context and generated the best matching ICD-10-CM code based only on its pretrained knowledge and the provided note context.
The RAG-LLM model used GPT-OSS-120B with retrieved ICD-10-CM candidate codes (Fig. 2). The model received the extracted S&S, the corresponding clinical note context, and the retrieved ICD-10-CM candidate codes and titles. The prompt instructed the model to select the best matching ICD-10-CM code from the retrieved candidates using the extracted S&S and note context. If no retrieved candidate adequately described the extracted S&S, the prompt allowed the model to generate the best matching ICD-10-CM code using its own knowledge. Therefore, the RAG-LLM model was retrieval-guided but not strictly candidate-constrained. The baseline and RAG-LLM prompt templates are shown in Table 3.
Workflow of the LLM-only baseline, retrieval-only voting baseline, and RAG-LLM model.
Retrieval diagnostic analysis
To characterize the retrieval component before LLM selection, we conducted retrieval-stage diagnostic analyses. First, we calculated individual embedding-model retrieval performance by measuring how often the human-labeled ICD-10-CM code appeared within the top 1 through top 5 retrieved candidates for each embedding model.
Second, we calculated retrieval recall for the combined multi-embedding candidate set. Exact-code Recall@k was defined as the proportion of S&S cases in which the full human-labeled ICD-10-CM code appeared in the retrieved candidate set. Three-character Recall@k was defined as the proportion of cases in which at least one retrieved candidate shared the same first three ICD-10-CM characters as the human reference code. Recall was calculated for increasingly large candidate sets by taking the top 1, 2, 3, 4, or 5 candidates from each of the eight embedding models, corresponding to maximum candidate sets of 8, 16, 24, 32, and 40 codes.
Third, we examined whether the final RAG-LLM output appeared in the retrieved candidate set provided in the prompt. Outputs whose normalized ICD-10-CM code matched one of the retrieved candidates were classified as candidate-selected outputs. Outputs whose normalized code did not appear in the retrieved candidate set were classified as outside-candidate outputs. Because the prompt did not require the model to explicitly state whether it used the fallback instruction, outside-candidate output was used as an operational proxy for fallback-pathway behavior.
Evaluation metrics
The human-annotated ICD-10-CM codes were treated as the reference standard for evaluating the retrieval-only voting baseline, LLM-only baseline, and RAG-LLM model. The proposed system should be interpreted as an ICD-10-CM terminology-mapping approach for symptom-based phenotyping and prediction modeling, not as a validated billing-code assignment system.
The primary performance metric was three-character ICD-10-CM category-level match. A model output was considered a three-character match if the first three characters of the normalized model-generated ICD-10-CM code matched the first three characters of the normalized human reference code. This metric was selected as the primary outcome because the intended downstream use was broad symptom/sign phenotyping and disease-prediction modeling.
Full-code exact match was reported as a secondary metric. A model output was considered a full-code exact match if the complete normalized model-generated ICD-10-CM code exactly matched the complete normalized human reference code. This secondary analysis was included to clarify the specificity of the generated ICD-10-CM mappings, although the study was not designed to evaluate billing-grade code assignment.
Adjudication of non-exact matches
In addition to exact three-character ICD-10-CM category matching, as an exploratory analysis, we adjudicated non-exact model outputs to determine whether they remained clinically acceptable representations of the documented S&S. This secondary outcome was used because more than one ICD-10-CM code may reasonably represent the same documented S&S, particularly when codes differ in specificity, wording, or subcategory granularity.
For each model output that did not exactly match the human reference label at the three-character level, two researchers independently reviewed the extracted S&S, the original clinical note context, the human reference code, and the model-generated code. A non-exact output was considered clinically acceptable if it met all of the following criteria: (1) the generated ICD-10-CM code existed in the official ICD-10-CM code set; (2) the generated code accurately represented the explicitly documented S&S in the note; and (3) the generated code preserved the clinically relevant symptom/sign concept for phenotyping or disease-prediction modeling. Codes were considered not acceptable if they represented a different clinical concept, contradicted the note context, or mapped the S&S to an unrelated ICD-10-CM category. Disagreements between the two reviewers were resolved through discussion with a third researcher. Examples of adjudication decisions are shown in Table 4.
Output validity and description consistency
The outputs of the LLM-only baseline and RAG-LLM model were validated against the official ICD-10-CM reference data frame for code validity and description consistency. Code validity was evaluated using the invalid code rate, defined as the proportion of model-generated ICD-10-CM codes that did not exist in the official ICD-10-CM code set. This metric was treated as the primary hallucination-related output-validity measure because it directly reflects generation of nonexistent ICD-10-CM codes.
Description consistency was evaluated only among valid ICD-10-CM codes. We calculated raw description exact-match mismatch by comparing the model-generated description with the official ICD-10-CM description for the generated code. Because exact string matching may classify semantically equivalent wording, capitalization, punctuation, or word-order differences as mismatches, description mismatches were further categorized as minor lexical deviations or wrong descriptions. Minor lexical deviations referred to descriptions that preserved the same clinical meaning despite differences in wording or ordering. Wrong descriptions referred to descriptions that changed the clinical meaning of the ICD-10-CM code or described a different clinical concept. Outputs without an extractable valid ICD-10-CM code after normalization were classified as invalid-code outputs.
Statistical analysis
Because all systems were evaluated on the same S&S instances, paired comparisons were performed using exact McNemar tests. The primary RAG-LLM model was compared with the LLM-only baseline and the retrieval-only voting baseline for both three-character ICD-10-CM category matching and full-code exact matching.
Because multiple S&S could be extracted from the same clinical note, 95% confidence intervals were estimated using note-level cluster bootstrap resampling with 10,000 iterations. In each bootstrap iteration, clinical notes were sampled with replacement, and all S&S associated with each selected note were retained. Percentile-based 95% confidence intervals were reported for model performance and paired differences.
Result
Among the 200 randomly selected clinical notes, 174 contained at least one S&S. A total of 545 S&S were identified across these 174 notes, corresponding to an average of 3.13 ± 1.97 S&S per note. The maximum number of S&S identified in a single note was 11.
Retrieval results
Retrieval-stage performance varied across embedding models. At top 5, the highest exact-code retrieval counts were observed for SapBERT, which retrieved the human-labeled ICD-10-CM code in 239/545 cases (43.85%), Bge Large Medical in 234/545 cases (42.94%), and Gtr-t5-xxl in 204/545 cases (37.43%). General-purpose embeddings also contributed meaningful retrieval coverage, including All-MiniLM-L6-v2 with 195/545 cases (35.78%) at top 5. These results suggest that no single embedding model fully captured the reference code set and support the use of a multi-embedding retrieval strategy to improve candidate diversity (Tables 5 and 6).
For the combined multi-embedding candidate set, exact-code Recall@k increased from 221/545 cases (40.55%) at top 1/model to 314/545 cases (57.61%) at top 5/model. Three-character Recall@k increased from 402/545 cases (73.76%) at top 1/model to 494/545 cases (90.64%) at top 5/model. Because the main evaluation used three-character ICD-10-CM category matching, the 90.64% three-character Recall@40 result indicates that the retrieval stage captured the correct ICD-10-CM category for most S&S cases before LLM selection. Recall gains were largest between top 1/model and top 3/model and then plateaued, suggesting diminishing returns as the candidate set expanded beyond top 3/model.
To examine whether the final LLM-selected RAG output came from the retrieved candidate set, we compared each normalized final ICD-10-CM code with the retrieved candidate codes provided in the prompt. Among the 545 S&S cases, 522 final RAG-LLM outputs (95.78%) matched a code in the retrieved candidate set, whereas 23 outputs (4.22%) were outside the retrieved candidate set. Among the 522 within-candidate outputs, 457/522 (87.55%) achieved three-character agreement with the human reference. Among the 23 outside-candidate outputs, 11/23 (47.83%) achieved three-character agreement. In 5 of the 23 outside-candidate cases (21.74%), the human-labeled ICD-10-CM code was present in the retrieved candidate set but the LLM did not choose it.
Model performance comparison
We compared three systems: a retrieval-only voting baseline, an LLM-only baseline, and the final RAG-LLM model. The retrieval-only voting baseline selected the final ICD-10-CM code by majority vote among the top-ranked candidates retrieved by the eight embedding models, without LLM-based selection. The LLM-only baseline generated ICD-10-CM codes without retrieved candidates. The RAG-LLM model used retrieved ICD-10-CM candidates and clinical note context to guide final code selection. Because the retrieval-only voting baseline used eight top-1 candidates whereas the RAG-LLM model received up to 40 top-5 candidates and applied LLM-based final selection, this comparison represents the performance of the complete pipelines rather than an ablation isolating the independent contribution of LLM selection.
For three-character ICD-10-CM category matching, the retrieval-only voting baseline achieved 55.60% accuracy (95% CI: 51.31–59.86), the LLM-only baseline achieved 72.11% accuracy (95% CI: 68.37–75.88), and the RAG-LLM model achieved 85.87% accuracy (95% CI: 82.58–89.02). For full-code exact matching, the retrieval-only voting baseline achieved 27.16% accuracy (95% CI: 23.28–31.34), the LLM-only baseline achieved 32.66% accuracy (95% CI: 28.73–36.59), and the RAG-LLM model achieved 50.28% accuracy (95% CI: 45.76–54.76) (Table 7).
Paired exact McNemar testing showed that the RAG-LLM model significantly outperformed both baseline systems. For three-character matching, the RAG-LLM model was correct in 97 cases where the LLM-only baseline was incorrect, whereas the LLM-only baseline was correct in 22 cases where the RAG-LLM model was incorrect (p = 1.99 × 10⁻¹²). Compared with the retrieval-only voting baseline, the RAG-LLM model was correct in 188 cases where the retrieval-only baseline was incorrect, whereas the retrieval-only baseline was correct in 23 cases where the RAG-LLM model was incorrect (p = 2.21 × 10⁻³³). Similar paired improvements were observed for full-code exact matching (Table 8).
As an exploratory analysis, we evaluated whether non-exact three-character outputs remained clinically acceptable representations of the documented S&S. Among the 152 non-exact outputs from the LLM-only baseline, 72 (47.37%) were judged clinically acceptable, resulting in 465/545 total clinically acceptable mappings (85.32%). Among the 77 non-exact outputs from the RAG-LLM model, 71 (92.21%) were judged clinically acceptable, resulting in 539/545 total clinically acceptable mappings (98.90%) (Table 9).
Output validity comparison
For output validity, the RAG-LLM model generated 6 nonexistent ICD-10-CM codes out of 545 outputs, corresponding to a nonexistent code rate of 1.10%. In contrast, the baseline model generated 54 nonexistent codes, corresponding to a nonexistent code rate of 9.91%. Among the 539 valid RAG-LLM outputs, 21 (3.90%) had minor lexical deviations that preserved the same clinical meaning, and 9 (1.67%) had wrong descriptions. Among the 491 valid LLM-only baseline outputs, 89 (18.13%) had minor lexical deviations and 184 (37.47%) had wrong descriptions.
For the LLM-only baseline, 54 outputs were invalid codes, 89 outputs were valid codes with minor lexical deviations, and 184 outputs were valid codes with wrong descriptions; however, this comparison should be interpreted cautiously because the RAG prompt supplied candidate ICD-10-CM codes with official titles, whereas the baseline model generated descriptions without access to retrieved official titles. No separate parsing-only error category was identified after ICD-10-CM code normalization; outputs without a valid extractable ICD-10-CM code were counted under the invalid code category (Tables 10 and 11).
Discussion
The present study demonstrates that grounding a large language model in retrieved ICD-10-CM knowledge substantially improves the accuracy and reliability of mapping clinical signs and symptoms to ICD-10-CM codes. Compared with a baseline in which GPT-OSS-120B generated codes from its pretrained knowledge without retrieval, the RAG-LLM model achieved higher exact match performance at the three-character ICD-10-CM block level and a higher proportion of non-exact outputs judged to be clinically acceptable S&S mappings in the exploratory adjudication. The most clinically meaningful hallucination-related improvement was the reduction in nonexistent ICD-10-CM code generation, indicating that retrieval grounding helped guide the model to valid codes in the official ICD-10-CM vocabulary. These findings are consistent with emerging work showing that retrieval-augmented generation improves factual consistency and reduces hallucinations in medical and healthcare settings by grounding outputs in trusted sources36,37.
Description-level findings should be interpreted more cautiously. Our original exact-match description metric captured whether the raw model output reproduced the official ICD-10-CM title exactly, but exact string mismatch does not necessarily indicate a clinically meaningful hallucination. Moreover, because the RAG prompt supplied candidate codes with official titles, the RAG model could copy canonical descriptions directly from the prompt, whereas the baseline model generated descriptions without access to retrieved official titles. Therefore, we interpret the description-level result as improved canonical-title consistency rather than as independent evidence of reduced clinically meaningful description error.
The observed performance of the RAG-LLM pipeline illustrates the potential of combining dense retrieval with generative reasoning for ICD-10-CM terminology mapping. Prior automated ICD coding methods have primarily relied on supervised deep learning models—such as CNNs, RNNs, transformer encoders, and probabilistic label models—trained on large corpora like MIMIC-III and MIMIC-IV15,16,38,39. While these approaches achieve strong performance on frequent codes, they often struggle with rare labels, long documents, and evolving code sets. Recent systematic reviews on LLM-based ICD coding similarly emphasize data requirements, brittleness to distribution shift, and hallucination risk when models are used in an unconstrained generative mode17,18.
By embedding the full ICD-10-CM code set using multiple complementary embedding models and guiding the model with a small retrieved candidate set per symptom, our RAG framework effectively narrows the search space while still allowing the LLM to leverage note-level context. The observation that many residual “mismatches” shared the same three-character stem as the human label or were judged clinically acceptable alternatives suggests that the system often selects codes within a semantically coherent neighborhood of the reference label, rather than producing arbitrary or spurious codes.
The candidate-selection pathway analysis further supports the interpretation that the final model output was primarily retrieval-guided. The RAG-LLM model selected codes from the retrieved ICD-10-CM candidate set in most cases, while outside-candidate generation occurred in only 23/545 cases. However, these outside-candidate cases also show that the system was not strictly candidate-constrained. In 11 of these 23 cases, the outside-candidate output matched the human reference label, suggesting that the fallback option sometimes corrected incomplete retrieval. In 5 of the 23 cases, however, the human-labeled code was already present in the retrieved candidate set, indicating that selection errors can still occur even when retrieval is successful. Because the prompt did not require the model to explicitly label whether the fallback instruction was used, candidate-selected and fallback-generated outputs could not be directly separated; therefore, outside-candidate generation was used as an operational proxy for fallback behavior, and the invalid-code rate should be interpreted as the result of the complete retrieval-guided pipeline with fallback enabled.
Our work also connects to broader literature showing that longitudinal ICD-coded EHR data can support powerful diagnostic and prognostic models. Deep learning architectures such as transformer-based sequence models and temporal neural networks have been used to predict a range of outcomes, including future disease onset, progression, and multimorbidity patterns, by modeling trajectories of diagnosis codes and visit histories40,41,42. In this context, improving the fidelity of S&S-to-ICD-10-CM terminology mapping may provide more consistent structured inputs for future symptom-trajectory, phenotyping, and prediction-modeling studies. The exploratory adjudication indicates that many non-exact mappings remained clinically plausible representations of the documented S&S. Whether such mappings preserve useful signal for phenotyping or prediction requires direct evaluation in future downstream studies.
From a translational perspective, the ability to perform high-quality mapping in a locally deployed, open-weight LLM setting is particularly important. Many current automated coding pipelines rely on proprietary models or require transmitting PHI to external services, which may be unacceptable in sensitive domains. Our findings complement recent proposals for institution-specific medical coding frameworks and generative AI pipelines, which combine terminologies, coding guidelines, and local claims data to tailor models for operational use15,29. By showing that an open-weight model guided by ICD-10-CM retrieval can improve terminology-mapping performance with low invalid-code generation in this MTsamples evaluation, this study supports further investigation of privacy-preserving, on-premises approaches for symptom-based informatics research.
Several limitations warrant consideration. First, our evaluation was conducted on a relatively small set of notes and S&S drawn from a single corpus, which may not fully capture the heterogeneity of documentation styles and coding practices seen in real-world EHRs. This concern mirrors findings from large-scale automated coding benchmarks, where performance varies substantially across datasets, specialties, and institutions16,38,39. Second, we evaluated correctness at the three-character block level, which is appropriate for many analytic and prediction tasks but does not fully reflect the granularity required for reimbursement or fine-grained clinical documentation. Third, our gold standard assigned only one ICD-10-CM code per symptom, even though multiple codes may sometimes be clinically reasonable, potentially underestimating the system’s effective accuracy. Finally, we focused on the mapping step using human-extracted S&S; in end-to-end pipelines, overall performance will also depend on upstream extraction accuracy and error propagation.
This study was designed as a proof-of-concept evaluation of a retrieval-grounded S&S-to-ICD-10-CM mapping pipeline. MTsamples was appropriate for this initial evaluation because it provides publicly available clinical note exemplars that allow controlled testing of the mapping step using human-labeled S&S and ICD-10-CM codes. However, MTsamples consists of transcription exemplars that may be cleaner, more complete, and more templated than real-world EHR notes. Therefore, the reported performance should be interpreted as proof-of-concept evidence rather than a definitive estimate of performance in routine clinical documentation. Future work should validate the proposed system on multi-institutional EHR datasets with more heterogeneous note types, documentation styles, abbreviations, missing context, and ambiguous clinical language.
A direct head-to-head comparison with previously published RAG-based ICD coding systems was not performed because no directly comparable RAG benchmark was identified for the symptom/sign-level mapping task evaluated in this study. Prior RAG-ICD studies generally operate at the encounter or document level and aim to assign diagnosis or billing codes from complete clinical records. By contrast, our study evaluates a more retrieval-guided mapping task in which human-extracted S&S are mapped to ICD-10-CM codes using note-level context. This difference affects the input unit, target label space, evaluation metric, and acceptable clinical output. Therefore, reusing prior systems without task adaptation would not provide a fair estimate of comparative performance. Instead, we used an internally controlled comparison between the same locally deployed LLM with and without retrieval, allowing us to isolate the effect of retrieval augmentation for S&S-to-ICD-10-CM mapping. Future work should evaluate multiple RAG-based ICD coding architectures on shared symptom/sign-level datasets once such benchmarks become available.
This proof-of-concept study evaluated ICD-10-CM mapping primarily at the three-character category level because the intended downstream use was phenotyping and disease-prediction modeling rather than billing-grade code assignment. Three-character matching captures the major symptom/sign category but does not reflect the full specificity required for reimbursement or formal clinical coding workflows. Although this study reports full-code exact-match performance, paired McNemar tests, and note-level cluster-bootstrap confidence intervals, it remains a proof-of-concept evaluation on MTsamples and should not be interpreted as billing-grade validation or real-world EHR deployment evidence.
Future work will extend this framework along several directions. Validating the system on multi-institutional EHR datasets, broader cancer types, and non-oncologic chronic conditions will be essential for establishing generalizability. Integrating the RAG-LLM module into an end-to-end pipeline that performs both S&S extraction and ICD-10-CM mapping will allow more realistic evaluation against provider-assigned codes.
Conclusion
This proof-of-concept study shows that retrieval-guided LLM candidate selection can improve symptom/sign-level ICD-10-CM terminology mapping on MTsamples. Compared with both an LLM-only baseline and a retrieval-only voting baseline, the RAG-LLM model achieved higher three-character and full-code exact-match performance, with significant paired improvements and lower invalid-code generation. These findings suggest that grounding a local LLM with retrieved ICD-10-CM candidate codes can improve the consistency and validity of mapping explicitly documented S&S into structured ICD-10-CM terminology.
The proposed system should not be interpreted as a validated billing-code assignment or downstream prediction tool. The primary evaluation focused on three-character ICD-10-CM category-level matching as a measure of broad symptom/sign category agreement. Whether this level of mapping is sufficient for phenotyping or prediction modeling requires direct evaluation in future downstream studies. Full-code exact matching was therefore reported as a secondary measure of mapping specificity.
References
Astin, M., Griffin, T., Neal, R. D., Rose, P. & Hamilton, W. The diagnostic value of symptoms for colorectal cancer in primary care: a systematic review. Br. J. Gen. Pract. 61 (586), e231–e243. https://doi.org/10.3399/bjgp11X572427 (2011).
Neal, R. D. et al. Comparison of cancer diagnostic intervals before and after implementation of NICE guidelines: analysis of data from the UK General Practice Research Database. Br. J. Cancer. 110 (3), 584–592. https://doi.org/10.1038/bjc.2013.791 (2014).
Li, R., Agor, J. K. & Özaltın, O. Y. Temporal pattern mining for knowledge discovery in the early prediction of septic shock. Pattern Recognit. 151, 110436. https://doi.org/10.1016/j.patcog.2024.110436 (2024).
Wang, Y. et al. Clinical information extraction applications: A literature review. J. Biomed. Inf. 77, 34–49. https://doi.org/10.1016/j.jbi.2017.11.011 (2018).
Thyvalikakath, T. P. et al. Leveraging Electronic Dental Record Data for Clinical Research in the National Dental PBRN Practices. Appl. Clin. Inf. 11 (2), 305–314. https://doi.org/10.1055/s-0040-1709506 (2020). Epub 2020 Apr 29. PMID: 32349142; PMCID: PMC7190390.
Centers for Medicare & Medicaid Services; National Center for Health Statistics. ICD-10-CM Official Guidelines for Coding and Reporting. (2023).
Centers for Medicare & Medicaid Services. ICD-10 overview slides. (2010). http://www.cms.gov/medicare/coding/icd10/downloads/032310_icd10_slides.pdf. Accessed 15 Dec 2025.
Mukherjee, P. et al. SCOPE: predicting future diagnoses in office visits using electronic health records. Sci. Rep. 13, 11005. https://doi.org/10.1038/s41598-023-38257-9 (2023).
Wang, X. et al. Prediction of the 1-year risk of incident lung cancer: prospective study using electronic health records from the State of Maine. J. Med. Internet Res. 21 (5), e13260. https://doi.org/10.2196/13260 (2019).
Wang, L. et al. Discovering signature disease trajectories in pancreatic cancer and soft-tissue sarcoma from longitudinal patient records. J. Biomed. Inf. 171, 104935. https://doi.org/10.1016/j.jbi.2025.104935 (2025).
Placido, D. et al. A deep learning algorithm to predict risk of pancreatic cancer from disease trajectories. Nat. Med. 29 (5), 1113–1122. https://doi.org/10.1038/s41591-023-02332-5 (2023).
Hjaltelin, J. X. et al. Pancreatic cancer symptom trajectories from Danish registry data and free text in electronic health records. Elife 12, e84919. https://doi.org/10.7554/eLife.84919 (2023).
Boytcheva, S. Automatic matching of ICD-10 codes to diagnoses in discharge letters. In: Proceedings of the Second Workshop on Biomedical Natural Language Processing. Association for Computational Linguistics, Hissar, pp 11–18 (2011).
Xie, P. & Xing, E. A neural architecture for automated ICD coding. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Melbourne, pp 1066–1076 (2018).
Wang, Z. et al. ICDXML: enhancing ICD coding with probabilistic label trees and dynamic semantic representations. Sci. Rep. 14, 18319. https://doi.org/10.1038/s41598-024-69214-9 (2024).
Merchant, A. M., Shenoy, N., Lanka, S. & Kamath, S. Ensemble neural models for ICD code prediction using unstructured and structured healthcare data. Heliyon e36569. (2024). https://doi.org/10.1016/j.heliyon.2024.e36569
Dong, H. et al. Automated clinical coding: what, why, and where we are? npj Digit. Med. 5, 159. https://doi.org/10.1038/s41746-022-00705-7 (2022).
Gershon, A., Soffer, S., Nadkarni, G. N. & Klang, E. Automatic ICD coding using LLMs: a systematic review. medRxiv https://doi.org/10.1101/2025.07.30.25330916 (2025).
Bai, Y., Cui, W. & Finkelstein, J. Utilizing open-source large language models to extract genitourinary symptoms from clinical notes. Stud. Health Technol. Inf. 328, 16–20. https://doi.org/10.3233/SHTI250664 (2025).
Bai, Y., Cui, W. & Finkelstein, J. Performance of open-source large language models to extract symptoms from clinical notes. Stud. Health Technol. Inf. 329, 663–667. https://doi.org/10.3233/SHTI250923 (2025).
Soroush, A. et al. Large language models are poor medical coders — benchmarking of medical code querying. NEJM AI. 1 (5). https://doi.org/10.1056/AIdbp2300040 (2024).
Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates, Red Hook, pp 9459–9474 (2020).
Izacard, G. & Grave, E. Leveraging passage retrieval with generative models for open-domain question answering. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Kiev, pp 874–880. (2021). https://doi.org/10.18653/v1/2021.eacl-main.74
Klang, E. et al. Assessing retrieval-augmented large language model performance in emergency department ICD-10-CM coding compared to human coders. medRxiv https://doi.org/10.1101/2024.10.15.24315526 (2024).
Klang et al. Assessing Retrieval-Augmented Large Language Models for Medical Coding. NEJM AI. https://doi.org/10.1056/AIcs2401161 (2025).
Puts et al. Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection. JMIR Formative Res. https://doi.org/10.2196/60095 (2025).
Medical Transcription Samples. http://www.mtsamples.com. Accessed 15 Dec 2025.
Vasantharajan, C. et al. MedBERT: a pre-trained language model for biomedical named entity recognition. In: 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, pp 1482–1488. (2022). https://doi.org/10.23919/APSIPAASC55919.2022.9980157
Hugging Face. ClinicalBERT model repository. (2024). https://huggingface.co/medicalai/ClinicalBERT. Accessed 15 Dec 2025.
Liu, F., Shareghi, E., Meng, Z., Basaldella, M. & Collier, N. Self-alignment pretraining for biomedical entity representations. In: Proceedings of NAACL 2021. Association for Computational Linguistics, Online, pp 4228–4238. (2021). https://www.aclweb.org/anthology/2021.naacl-main.334
Hugging Face. bge_large_medical embedding model. (2024). https://huggingface.co/ls-da3m0ns/bge_large_medical. Accessed 15 Dec 2025.
Reimers, N. & Gurevych, I. Sentence-BERT: sentence embeddings using Siamese BERT networks. In: Proceedings of EMNLP 2019. Association for Computational Linguistics. (2019). http://arxiv.org/abs/1908.10084
Hugging Face. all-MiniLM-L6-v2 sentence transformer. (2024). https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2. Accessed 15 Dec 2025.
Ni, J. et al. Large dual encoders are generalizable retrievers. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, pp 9844–9855. (2022). https://doi.org/10.18653/v1/2022.emnlp-main.669
Ni, J. et al. Sentence-T5: scalable sentence encoders from pre-trained text-to-text models. In: Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dublin, 1864–1874. (2022). https://doi.org/10.18653/v1/2022.findings-acl.146
Neha, F., Bhati, D. & Shukla, D. K. Retrieval-augmented generation (RAG) in healthcare: a comprehensive review. AI 6 (9), 226. https://doi.org/10.3390/ai6090226 (2025).
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S. & Seidel, J. Retrieval augmented generation for large language models in healthcare: a systematic review. PLOS Digit. Health. 4 (6), e0000877. https://doi.org/10.1371/journal.pdig.0000877 (2025).
Edin, J. et al. Automated medical coding on MIMIC-III and MIMIC-IV: a critical review and replicability study. In: Proceedings of the 46th Intl ACM SIGIR Conf on Research and Development in Information Retrieval. ACM, New York, pp 2572–2582. (2023). https://doi.org/10.1145/3539618.3591918
Aden, I., Child, C. H. T. & Reyes-Aldasoro, C. C. International classification of diseases prediction from MIMIC-III clinical text using pre-trained ClinicalBERT and NLP deep learning models achieving state of the art. Big Data Cogn. Comput. 8 (5), 47. https://doi.org/10.3390/bdcc8050047 (2024).
Yang, Z., Mitra, A., Liu, W., Berlowitz, D. & Yu, H. TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nat. Commun. 14, 7857. https://doi.org/10.1038/s41467-023-43715-z (2023).
Grout, R. et al. Predicting disease onset from electronic health records for population health management: a scalable and explainable deep learning approach. Front. Artif. Intell. 6, 1287541. https://doi.org/10.3389/frai.2023.1287541 (2024).
Hama, T. et al. Enhancing patient outcome prediction through deep learning with sequential diagnosis codes from structured electronic health record data: systematic review. J. Med. Internet Res. 27, e57358. https://doi.org/10.2196/57358 (2025).
来源:Nature 健康护理主题 · nature.com