← ALL CASE FILES
CASE FILE · KM-05

Combination as Compromise.

Three models tried to catch COVID-19 from a cough recording. The one built to combine them looked strongest on paper, and weakest at the one job that mattered: catching real cases.

Machine LearningPythonModel EvaluationEnsemble MethodsHealthcare AITeam Collaboration
Case File Snapshot
StakeholderITAO 40250: Unstructured Data Analytics, Notre Dame (Course Project) Framing: Fictional Health-Tech Client “AccessaHealth”
Role4-Person Team: Led the Model 1 (Clinician Baseline) Workflow Supported the Model 2 (Audio) Workflow, Helped Delegate Team Roles
DatasetCoughVID: 25,000+ Crowdsourced Recordings Cleaned & Balanced to 420 Observations (140 per Class)
MethodXGBoost Metadata Model, Wav2Vec2 Audio Embeddings + SVM, MLP Stacking Ensemble
OutputsFinal Presentation, 3 Trained Classifiers Model Comparison Across Accuracy & Per-Class Recall
Applied ValueShows why the highest-accuracy model isn’t always the safest one to deploy
The Story
The Question

Traditional COVID-19 testing is slow, costly, and hardest to reach for under-resourced communities. Could a cough recording alone stand in for a clinical read, or does it need one?

The Tension

The obvious hypothesis was that more signal wins: pair what a doctor hears in a symptom history with what a model hears in the audio, and get a better diagnosis than either alone. That combination scored close to the baseline model overall, and worse than the audio model at the one job that mattered.

The Insight

The best model on paper wasn’t the best model at catching real COVID cases.

The System

The pipeline held together on four decisions:

  • Consensus Labeling — Physician ratings were combined by median (numeric) or mode (categorical) across up to 4 raters per cough.
  • Strict Balancing — Healthy and symptomatic cases were undersampled to match the smaller COVID-19 count, so no class could win on volume alone.
  • Paired Modalities — Every case kept its audio and metadata rows aligned 1-to-1, so the same patients fed both models.
  • Stacking, Not Averaging — The combined model learned from each model’s full probability vector, not just a simple vote or average.
The Approach
01Problem FramingScoped the project around COVID-19 detection access gaps for under-resourced and remote communities.
02Dataset Acquisition & CleaningFiltered the 25,000+ recording CoughVID dataset down to a clean, label-ready set by cough clarity and metadata completeness.
03Feature ConsolidationConsolidated up to 4 physicians’ ratings per cough into 13 consensus features per patient.
04Balanced SplitUndersampled to 420 observations (140 per class) and split 20% test, then 85/15 train/val.
05Clinician Baseline ModelLed the XGBoost metadata classifier, tuned via 5-fold cross-validation, as the telehealth-triage baseline.
06Audio Deep-Feature ModelSupported the Wav2Vec2 + SVM audio classifier, trained on 768-dimension deep audio embeddings with no metadata.
07Stacked Ensemble & EvaluationHelped coordinate the team’s work on the MLP meta-learner and the final cross-model comparison on accuracy and per-class recall.
Tools (Secondary)
XGBoostWav2Vec2 (Hugging Face)scikit-learn
Key Findings
InsightEvidenceStrategic Meaning
The clinician-style model defaulted to “healthy.”EvidenceThe metadata-only model (XGBoost) hit 44% overall accuracy but only 26% recall on COVID-19, with most classes over-predicted as healthy.Strategic MeaningThe highest-accuracy model was also the most dangerous one: it under-caught the class it existed to catch.
Audio alone learned all three classes, unevenly.EvidenceThe Wav2Vec2 + SVM audio model reached 37% accuracy but the best COVID-19 recall of the three models, at 42%.Strategic MeaningAudio carried real diagnostic signal on its own, even without a doctor’s input.
Combining the models didn’t combine their strengths.EvidenceThe MLP stacking ensemble reached 40% accuracy, between the other two, but only 35% recall on COVID-19, below the audio model alone.Strategic MeaningA model can look better on the headline metric and still be worse at the one class that carries the most risk to miss.
Self-reported symptoms outweighed physician audio ratings.EvidenceFever/muscle pain and pre-existing respiratory condition were the top two features in the clinician model, ranked above any physician-rated cough characteristic.Strategic MeaningPatient history pulled more weight than the clinical audio scoring it was paired with.
Getting to a clean, balanced dataset cost more than 95% of the raw data.EvidenceOf 25,000+ crowdsourced recordings, only 420 (140 per class) survived clarity filtering, consensus labeling, and class balancing.Strategic MeaningThe real bottleneck wasn’t modeling. It was finding data trustworthy enough to model on.
Output Preview
Three-model architecture diagram: metadata classifier, audio classifier, and MLP meta-learner combining both
Model Architecture · 1/3
Learnings From This Case
Optimize for the Metric That Matters
The combined model’s accuracy looked competitive, but its COVID-19 recall was the worst of the three at the one job the whole project existed to do. Pick the metric tied to the real-world cost of an error before comparing models.
Simpler Can Beat Combined
Audio alone caught more real COVID-19 cases than the model built to combine every signal. Adding a modality doesn’t guarantee it improves the result, so test the simpler model before assuming the combination wins.
Small, Clean Data Beats Large, Messy Data
Only 420 of 25,000+ recordings survived clarity and completeness filtering, and that smaller, trustworthy set was what actually made the models comparable. Budget real time for filtering before modeling, not after.

Selected slides from the final CoughDx presentation.

Ask Me About This Project →