{Reference Type}: Journal Article
{Title}: Distilling large language models for matching patients to clinical trials.
{Author}: Nievas M;Basu A;Wang Y;Singh H;
{Journal}: J Am Med Inform Assoc
{Volume}: 0
{Issue}: 0
{Year}: 2024 Apr 19
{Factor}: 7.942
{DOI}: 10.1093/jamia/ocae073
{Abstract}: <B>OBJECTIVE: </B>The objective of this study is to systematically examine the efficacy of both proprietary (GPT-3.5, GPT-4) and open-source large language models (LLMs) (LLAMA 7B, 13B, 70B) in the context of matching patients to clinical trials in healthcare.<BR><B>METHODS: </B>The study employs a multifaceted evaluation framework, incorporating extensive automated and human-centric assessments along with a detailed error analysis for each model, and assesses LLMs' capabilities in analyzing patient eligibility against clinical trial's inclusion and exclusion criteria. To improve the adaptability of open-source LLMs, a specialized synthetic dataset was created using GPT-4, facilitating effective fine-tuning under constrained data conditions.<BR><B>RESULTS: </B>The findings indicate that open-source LLMs, when fine-tuned on this limited and synthetic dataset, achieve performance parity with their proprietary counterparts, such as GPT-3.5.<BR><B>CONCLUSIONS: </B>This study highlights the recent success of LLMs in the high-stakes domain of healthcare, specifically in patient-trial matching. The research demonstrates the potential of open-source models to match the performance of proprietary models when fine-tuned appropriately, addressing challenges like cost, privacy, and reproducibility concerns associated with closed-source proprietary LLMs.<BR><B>CONCLUSIONS: </B>The study underscores the opportunity for open-source LLMs in patient-trial matching. To encourage further research and applications in this field, the annotated evaluation dataset and the fine-tuned LLM, Trial-LLAMA, are released for public use.