The first version of AquaSafe had five models behind one API, trained on 500,000 synthetic records in 36 minutes. Its training report opens with a water quality classifier at 99.95% accuracy. It would be easy to put that on a slide and stop. This post is about the row underneath it.
What the row actually says
Accuracy of 42.46% over eight classes sounds like a number you could defend, since uniform guessing would get 12.5%. The F1 scores are where that defence runs out. Weighted F1 is 0.3388 and macro F1 is 0.2020. Weighted F1 counts each class by its size and macro F1 counts every class equally, so a gap that wide means the rare diseases are doing much worse than the common ones.
Uniform guessing is also the wrong comparison when classes are imbalanced. The right floor is always predicting the most common disease, and the report does not compute it. That is a gap I should close, because without it I cannot say how much of the 42% is the model and how much is the class distribution.
The training time was a clue
The disease model took 1,806 seconds. Severity took 17 and outbreak risk took 31. By that arithmetic it used about 83% of the whole 36-minute run, for the worst result in the set. More compute did not find a signal, and the likeliest explanation is that the signal is not in those features: cholera, typhoid and giardiasis overlap clinically, and water chemistry alone does not separate them.
Being suspicious of the good number too
The same honesty has to apply upward. The water quality model reports an AUC-ROC of 1.0000. On synthetic data that is not a triumph. It mostly tells you the model recovered the rules used to generate the labels. Its top features are pH, coliform count, turbidity, TDS and E. coli, which is reassuring, because a sanitary engineer would name the same ones. It shows the pipeline is wired correctly. It does not show how the model would do on samples from a real well.
Three places the report misleads
Reading the report line by line turned up things I would not repeat, even though they are in my own repository:
- Its recommendations section says top-3 disease accuracy is 80%+ and outbreak R² is 96%+. The measured values a few lines above are 73.79% and 0.9496. The summary text is more generous than the data, so the measurements are the ones to quote.
- Outbreak risk logs a mean absolute percentage error of 31,956,231. Percentage error explodes when true values sit near zero, so the figure says nothing useful about the model. R² and RMSE (0.0746) are the ones to read.
- The summary table lists five models, but the fifth, patient outcome, has no metrics at all.
What changes when you take it seriously
A model at 42% should not name a diagnosis. At 73.79% top-3 it can reasonably hand a health worker a shortlist to narrow down by symptom. So that version presented it as a shortlist, which is a product decision driven by an evaluation number. That only happens if the number stays in the room.
Then the number changed
The current version, trained on 5.25 million synthetic records with 36 input features and published at IEEE IC-SIT 2026, reports 95.19% on the same eight-way question. It would be convenient to retire this post. But the dataset, the features and the generator all changed, so that jump cannot be credited to the model alone. The questions above still apply: what is the macro F1, what does always guessing the commonest disease score, and how does it do on a real sample?
The useful output of an evaluation is knowing which questions your data can answer.