MedGPT
Knowledge-guided, explainable medical visual question answering
A model that gives a confident wrong answer about a chest X-ray is worse than no model. So the architecture here is organised around a different question than accuracy: can you see why it said that?
Python · PyTorch · Qwen2-VL · LoRA / QLoRA · BioBERT · CLIP ViT · FastAPI · DeepSpeed · Docker
MedGPT is built on Qwen2-VL-7B, fine-tuned with LoRA under 4-bit quantisation so an 8B-class vision-language model can be adapted on a single GPU. Four public medical VQA corpora — PathVQA, SLAKE, MedVQA and VQA-RAD — are harmonised into one schema covering radiology, pathology and multi-modal imaging.
The knowledge path
What separates this from a general VLM pointed at medical images is a second encoder. Alongside the CLIP ViT vision path, a BioBERT/PubMedBERT encoder embeds domain knowledge retrieved through UMLS and SciSpacy, and a cross-attention fusion module combines the two before the answer head. The intent is that the model reasons with medical vocabulary rather than around it, and that its answers are anchored to retrievable statements rather than to correlations in the training images.
Three explanations, not one
Every answer carries Grad-CAM, attention rollout and integrated gradients. Three explainers rather than one because they can disagree, and disagreement is itself diagnostic — a Grad-CAM that highlights the lesion while integrated gradients points at the scanner annotation is a model that got the right answer for the wrong reason, and no single method would have caught it.
The pipeline handles DICOM conversion, medical-specific image augmentation and text preprocessing, trains through a custom multi-objective trainer with optional DeepSpeed, and serves through a FastAPI backend, containerised with GPU support.