Radiology Report Generation from 3D Chest CT

Overview

This project was my MSc thesis research in the AI for Science programme at the African Institute for Mathematical Sciences (AIMS) South Africa, carried out from January to November 2024 in collaboration with Mehrdad Ghaziasgar at the University of the Western Cape. It asks whether a compact vision-language system can produce expert-level radiology reports directly from a full 3D chest CT volume, and whether that can be done under a strict compute budget. The paper was presented orally at PAAISS 2025 in Kumasi, Ghana, received the conference’s 2nd Best Paper Award, and was published by Springer Nature in LNICST volume 697.

Research problem

A single chest CT examination contains hundreds of slices, and radiologists synthesise findings across three anatomical planes before writing a report. Automating this is harder than report generation for 2D chest X-rays for three reasons:

• Volumetric context. Findings such as nodules, effusions or consolidation extend across slices and planes; a model that only sees isolated slices loses that continuity.
• Compute and memory. Full 3D encoders are expensive to train, and pre-trained 3D weights for medical imaging are scarce.
• Paired data. Until the release of CT-RATE in 2024 there was no large public corpus of 3D chest CT volumes with paired radiologist-written reports.

Methodology

1. Data and preprocessing. CT-RATE pairs non-contrast chest CT volumes with reports. Because of storage and compute limits, about 20% of the dataset was used: 11,597 volumes (9,597 for training and 2,000 held out for testing), roughly 3 TB of data. Volumes were converted to Hounsfield units, clipped to the diagnostic range, resampled to 0.75 mm × 0.75 mm × 1.5 mm and cropped or padded to 240 × 480 × 480. Only the findings section of each report was used as the generation target.
2. Volumetric feature extraction with ACS convolutions. A ResNet-50 pre-trained on RadImageNet, a medical imaging dataset, was adapted to 3D input using Axial-Coronal-Sagittal (ACS) convolutions, which split each 2D kernel across the three anatomical planes so that pre-trained 2D weights can be reused on volumes. Only the first ten layers of the network were used because of GPU memory, producing a 10 × 20 × 20 × 256 feature map per volume.
3. Multimodal language model. The visual features were projected into GPT-2’s 768-d embedding space, and cross-attention layers were added to GPT-2 so that report generation attends to the volumetric features, with early layers weighted towards the visual input.
4. Training under constraint. The model was trained for 40 epochs with batch size 4 and AdamW (learning rate 5 × 10⁻⁵) on a single NVIDIA L4 GPU with 24 GB of memory, taking about 40 hours on a Google Cloud virtual machine.

Results (as reported in the paper)

Evaluation on the 2,000 unseen test volumes:

Compared with CT2Rep, the state of the art trained on the full dataset with substantially more compute:

BLEU-1 comes within 0.06 of the state of the art despite a fifth of the data, a truncated encoder and a single GPU; higher-order n-grams lag further behind, as expected when context and capacity are constrained. Qualitatively, generated reports for unseen volumes reproduce many reference sentences exactly; in the example analysed in the paper, ten sentences of the generated report matched the radiologist’s report word for word.

Why it matters

Volumetric imaging is where much of the diagnostic information in medicine lives, yet most multimodal models are built around 2D images. This project showed that a carefully designed pipeline can approach state-of-the-art report generation under constraints typical of resource-limited settings, and it is the origin of my interest in multimodal systems that reason over complex, high-dimensional data and in the structured, interpretable modelling such systems need to be trusted in clinical use.