NOAH: Learning the Full Patient Journey

A holistic patient state encoder and prediction engine. Built from all data.

NOAH can consume all of a patient’s lifetime data as one sequence of timestamped events: textual reports, X-ray images, ultrasound videos, ECG waveforms, vital measurements, lab results, medication, procedures, and more.

All pieces of information are considered singular events and put on one timeline per patient, in arbitrary order and count. They can span multiple emergency department (ED) visits, hospital admissions, and ICU stays, as well as online medical records collected outside clinical care.

NOAH’s design can deal with unseen continuous value embeddings and free-text event types without retraining, overcoming the limitations of a closed vocabulary.

For the first time in its field, NOAH is fully multimodal, yet generative. It actually consumes the contents of events, e.g. pixel data of an X-ray instead of just the information that an image was taken. NOAH is a transformer decoder pretrained on these multimodal sequences from the merged MIMIC dataset family (Johnson et al., 2023).

A landscape of patient states and transitions.

NOAH is pretrained to predict the next token in a variable latent space for each timestep, i.e. for every clinical event in the input sequence, given the history. By this, NOAH learns a landscape of patient states and transitions. It produces an embedding for every timestep, forming a trajectory over the entire patient journey. It serves as a versatile patient state encoder.

As a characteristic of NOAH, the model allows quantifying its surprise, i.e. the difference between its prior expectation and the posterior informed by the actual next event, for each step in a patient’s timeline. This surprise corresponds to clinical novelty. It rises before formally charted stay onsets, peaks for first responses in the emergency department, and decays as routines emerge.

Patient states by care level

Patient states by event type

NOAH’s learned trajectories for all test set patients form a landscape that is mainly organized by contextualized events, e.g. by event type under a certain level of care.

A prediction engine with optional time control.

Due to its generative nature, NOAH can also predict future states autoregressively. Thus, NOAH also serves as a task-agnostic prediction engine.

Prompted with a patient’s record up to a cutoff, NOAH rolls out future events and also allows for controlling the timing of each next event. It reaches AUROC 0.83-0.95 with Brier scores 0.03-0.11 for type and modality of the next events within horizons of up to 72 h.

NOAH’s built-in variability allows for sampling multiple such rollouts under one prompt. This enables Monte Carlo estimation of clinical outcome probabilities such as in-hospital mortality, prolonged stay, and 30-day readmission, zero-shot without extra training. We achieve an AUROC of 0.97 for 72-hour mortality, 0.74 for prolonged stay, and 0.61 for readmission.

A simulator for counterfactual interventions.

Using the mechanism described above, NOAH allows swapping a factual treatment in the prompt for a counterfactual and observing the change in the probabilities of the simulated outcomes.

For 1,085 patients, we substitute the first intravenous fluid given within six hours of a sepsis marker (0.9% saline vs. lactated Ringer’s), simulate both arms, and estimate mortality, length of stay, and MAKE-30 (Semler et al., 2018). The simulated treatment effect, +8.92 pp MAKE-30 and +8.86 pp 30-day mortality under Ringer’s against saline, matches the sign of the effect in the sepsis subgroup of the SMART randomized trial (Brown et al., 2019), +4.7 pp, at roughly twice its magnitude.

A powerful feature extractor for downstream tasks.

NOAH produces a patient state embedding at every timestep, aggregating information from a long complex history of diverse multimodal events. These representations can serve as powerful features to facilitate downstream tasks.

Due to NOAH’s novel architecture, those features are highly predictive and prognostic. DeepSurv (Katzman et al., 2018) survival models reach a C-index (Antolini et al., 2005) of up to 0.85 on NOAH’s patient state embeddings while staying calibrated (Haider et al., 2020).

Linear probes for retrieval reach AUROC 0.72-0.94 across 15 ICD chapters and 29 comorbidities (Quan et al., 2005). An MLP recovers the NEWS2 early-warning score (Smith et al., 2019) with MAE 1.34 and is applicable to any patient state at any point in time.

A novel variability module and bidirectional time integration.

NOAH is a causally masked transformer decoder pretrained on sequences of multimodal patient data. Every sequence item is considered a timestamped event consisting of high-level type category, free-text type specifics, and continuous value embedding. NOAH learns end-to-end to project those composite events into a shared space, following the token scheme of TAMME (Susetzky et al., 2025).

Variational autoregression. Given a multimodal input sequence, for timestep $t-1$ NOAH predicts the next event $\mathbf{x}_t$ through transformer output $\mathbf{h}_{t-1}$ and a latent patient state $\mathbf{z}_t$. This patient state is shared across the output heads decoding the multiple event components. The overall setup adopts the concepts of the Variational Autoencoder (VAE) (Kingma & Welling, 2013) and of Variational Recurrent Neural Networks (VRNN) (Chung et al., 2015):

The prior $p_\theta(\mathbf{z}_t \mid \mathbf{h}_{t-1})$ is a learned, context-conditional Gaussian over the continuation that is expected given the history. During training, an informed posterior $q_\phi(\mathbf{z}_t \mid \mathbf{h}_{t-1}, \mathbf{e}_t)$ additionally sees the embedding $\mathbf{e}_t$ of the realized event $\mathbf{x}_t$. The model’s objective is to reconstruct $\mathbf{x}_t$ from the posterior while simultaneously minimizing the divergence between prior and posterior. This explicitly encourages the model to learn a predictive representation from history and encode only the truly unpredictable residual via the posterior, the surprise. At inference time, sampling $\mathbf{z}_t$ from the prior yields stochastic trajectories.

Bidirectional temporal attention. The gap between two adjacent events in our multimodal event sequences is the forward gap of the earlier one and, at the same time, the backward gap of the later one. NOAH explicitly separates these two functionally distinct views in the self-attention mechanism: queries $\mathbf{q}$ carry the forward gap $\widetilde{\delta}^{\mathrm{next}}_t$ to the next event while keys $\mathbf{k}$ are associated with the backward gap $\widetilde{\delta}^{\mathrm{prev}}_t$ to the previous event. Both include the event’s index position $p_t$ within the sequence as well as the patient’s age $a_t$ at that event. All features are encoded through a sinusoidal basis $\Psi$:

By this, attention explicitly conditions on the horizon it predicts for.

Citation

@misc{susetzky2026noah,
  title         = {NOAH: Learning the Full Patient Journey. A Longitudinal
                   Multimodal Time-Aware Model for Representation and Forecasting},
  author        = {Tobias Susetzky and Raphael Rehms and Dmitrii Seletkov and
                   Özgün Turgut and Michelle Espranita Liman and Lisa Steinhelfer and
                   Rickmer Braren and Daniel Rueckert},
  year          = {2026},
  eprint        = {2609.09140},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2609.09140}
}

References