The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR

Quick Overview

The researchers propose a novel training paradigm for longitudinal Electronic Health Record (EHR) prediction by transforming a language model into a world model using a Joint Embedding Predictive Architecture (JEPA), which successfully predicts future patient states, like disease progression or mortality, more effectively than standard forecasting models by focusing on underlying physical dynamics rather than just word sequences.

Key Points: The paper introduces a new training paradigm where a language model is converted into a world model using the Joint Embedding Predictive Architecture (JEPA) for longitudinal EHR prediction. JEPA-trained models significantly outperformed standard forecasting models (like GPT-4) in predicting patient outcomes, such as acute kidney injury and mortality, across datasets like MIMIC-III and eICU. The core innovation is forcing the model to learn the underlying physics/dynamics of the disease trajectory rather than simply predicting the next word in a sequence. The MSK oncology dataset (23,000 cancer patients) showed the JEPA model was significantly ahead of the standard SFT model in predicting mortality. The predictive task involves masking future tokens (like predicting mortality or organ failure) and using a simple linear probe over the frozen internal representation (embedding) to make the prediction. The research suggests that models trained this way learn universal dynamics that apply across different diseases, moving beyond specific textual patterns.

Context: The discussion centers on advancing Artificial Intelligence applications in healthcare, specifically for predicting patient outcomes from Electronic Health Records (EHRs). The researchers address the limitations of standard language models (LLMs) in handling the complex, dynamic nature of patient health, proposing a paradigm shift using a world model architecture called Joint Embedding Predictive Architecture (JEPA) to better capture the underlying temporal evolution of diseases.