End-to-end pipeline
Data acquisition via light-sheet calcium imaging, followed by segmentation and ΔF/F trace extraction — the full ZAPBench workflow at a glance.
ICLR 2025 · Google Research × Janelia × Harvard
ZAPBench: forecasting an entire vertebrate brain.
The Zebrafish Activity Prediction Benchmark is the first dataset that pairs whole-brain activity recordings with the structural connectome of the very same animal — a vertebrate, at single-cell resolution.
Pipeline · Live
00:08
Fig. 1 · The end-to-end pipeline Animated overview of the ZAPBench workflow: a larval zebrafish under light-sheet calcium imaging (raw volume 2048 × 1328 × 72 × 7,879 voxels), followed by neuron-soma segmentation and ΔF/F trace extraction.
The preparation
A transparent vertebrate that thinks.
Six-day-old larval zebrafish are almost perfectly transparent and small enough that a specialised laser-scanning microscope can image the entire brain slice-by-slice in 3D. The fish used in ZAPBench was genetically modified to express GCaMP — a fluorescent calcium indicator that lights up when calcium ions enter active neurons.
While immobilised in a jelly-like medium, the fish was shown nine different computer-generated stimuli modelling the conditions it would meet in shallow rivers: shifting water currents, alternating light and darkness, powerful sweeps that would normally carry it away. Electrodes on the tail recorded "fictive" swimming — motor intent without movement.
Two hours of whole-brain activity were captured. Collaborators Alex Chen and Misha Ahrens at HHMI Janelia performed the recording; the Engert and Lichtman labs at Harvard contribute the synapse-resolution EM reconstruction that is still underway on the same specimen.
FFN · Segmentation
00:12
Fig. 2 · Flood-Filling Network segmentation Automated soma segmentation of the zebrafish brain. Individual colours represent distinct agglomerated cells — 71,721 in total. Credit: Google Research.
Processing
From 4D video to 1D traces.
The raw recording is a four-dimensional volumetric video — 2048 × 1328 × 72 voxels per frame, 7,879 frames in total, XY resolution 406 nm. Motion artefacts were first removed with phase cross-correlation for translational alignment, followed by two rounds of elastic alignment using Google's open-source SOFIMA library to correct non-rigid deformations.
A customised Flood-Filling Network, trained on manually annotated ground-truth from VAST, then segmented the neuronal somas. The segmentation was downsampled 4× to match the activity-data resolution; averaging the intensity of voxels inside each mask yielded an activity trace — a 1D time series — for every one of 71,721 neurons.
Fig. 3 · The activity matrix Each row is a neuron, each column a timepoint. Brightness encodes firing. Vertical lines mark stimulus transitions; rows are sorted so neurons with similar response profiles cluster together. Credit: Google Research.
The forecasting task
Given the past, predict the next 30 seconds.
ZAPBench poses the prediction problem in the same form as weather or language: given a clip of recorded activity, how accurately can a model forecast the subsequent horizon? Two data regimes are evaluated — time-series models that consume the 1D traces, and volumetric-video models that operate directly on the 3D+time voxel data.
The short-context setting uses C = 4 past frames; long-context uses C = 256. All predictions are evaluated against ground truth over a horizon of H = 32 frames using mean absolute error, plus Corr_H and Corr_W correlation metrics at both whole-trace and horizon granularity.
Models benchmarked
Video beats traces when you have enough context.
Initial baselines include univariate time-series predictors (TSMixer, TiDE) and a 4D U-Net that treats time as extra channels and learns directly from volumetric video. Strikingly, trace-based multivariate models that could in principle learn functional coupling between cells do not significantly outperform univariate ones — but the volumetric U-Net does, by exploiting spatial context that never exists in a 1D trace.
Dataset size
7,879frames
Approximately two hours of continuous whole-brain recording at cellular resolution.
Raw volume
2048× 1328 × 72
Per-frame voxel grid. Each frame is a full 3D image of the brain.
XY voxel size
406nm
Sub-micrometre lateral resolution sufficient to resolve individual neuronal somas.
Stimuli conditions
9VR scenes
Including optomotor gain, darkness, dots, turning, looming — each a simplified ethological context.
Prediction horizon
H = 32frames
Roughly 30 seconds of forecast activity evaluated against held-out ground truth.
Training steps
500K
AdamW, cosine learning-rate schedule from 10⁻⁴ → 10⁻⁷, MAE loss.
Results · Compare
00:12
Fig. 4 · Trace-based vs volumetric models Side-by-side visualisation of prediction loss. Trace-only models treat each cell as an independent 1D time series; volumetric models operate directly on the 4D light-sheet data and exploit spatial context between cells. Credit: Google Research.
Findings
Some brain regions are harder to predict than others.
Three results emerged from the baseline models. First, more context helps — significantly. Second, volumetric-video models outperform trace-based models in many conditions, thanks to spatial information that is simply absent from 1D traces. Third, when models fail, they fail in geographically concentrated regions of the brain — meaning some territories are intrinsically harder to forecast than others.
Video models also work well at lower resolution, hinting that much of the signal lives at spatial scales coarser than single cells. Google is now completing a whole-brain connectome for the same specimen at HHMI Janelia — the first time a vertebrate activity recording will be paired with a full structural reconstruction of the same brain.
