MedGPT
Medical visual question answering, fine-tuned on an 8-billion-parameter model
A model that gives a confident wrong answer about a chest X-ray is worse than no model. So every answer here comes with a heatmap, and the results table includes the open-ended questions the model finds hardest.
- Overall accuracy
- 78.5%
- yes/no and open-ended
- Yes/no
- 87.1%
- Open-ended
- 73.9%
- Token F1
- 81.2%
- 5 imaging modalities
MedGPT answers natural-language questions about medical images across five modalities: X-ray, CT, MRI, ultrasound and pathology slides. It is built on Qwen3-VL-8B-Instruct and adapted with LoRA, so only 210M of the model's 9B parameters, about 2.3%, are trained.
Training in two stages
Training runs in two stages. Pre-training uses PMC-VQA, at roughly 140,000 samples, to learn the medical domain. Fine-tuning then uses VQA-RAD, SLAKE and PathVQA, at roughly 3,500, 14,000 and 32,000 samples. The LoRA adapter uses rank 64 and alpha 128, trained for three epochs in bfloat16 on an NVIDIA H200 in about 5.6 hours.
Showing where it looked
Every prediction can return a Grad-CAM heatmap showing the regions the model focused on. A heatmap cannot prove an answer is right, but it can show when the model reached the right answer by looking at the wrong thing, such as a scanner annotation and not the lesion.
Shipping it
A FastAPI backend serves GPU inference through a REST API, alongside a React dashboard for analysing images and viewing training curves and metrics. The whole application is containerised with Docker and NVIDIA CUDA.
All figures from the MedGPT model card on Hugging Face (March 2026). An earlier GitHub version used a different base model and a knowledge-retrieval path, and published no evaluation.