Viola Kempa Say hello

All work Case 03 · Machine learning · [Year]

Flare Days

Scatter plot of predicted against actual severity on the test set. The points form horizontal bands, one per step of the 0 to 4 scale.
A model that predicts tomorrow’s symptom severity for people with autoimmune conditions.

Result

12% less error than “tomorrow is like today”: 0.538 to 0.471.

My part
Everything, alone: the question, data pipeline, models, evaluation and write-up.
Time
[How long, and when.]
Constraint
One public export, 7.98 million rows, with weather missing on about 2 in 5.

Read the code

Flaredown is a symptom tracker for autoimmune conditions: a 686 MB export of 7.98 million rows. The target was the next calendar day’s mean severity, on a 0 to 4 scale. The bar to clear is “tomorrow is like today”, and that bar is hard to beat.

[Why you picked this question, in one sentence.]

Test on the future, never on shuffled days.

The usual way to score a model is a random 80 / 20 split. Here it would let the model learn from days after the ones it predicts, and from the same person on both sides. The score would look good and mean little.

[The evidence: what a shuffled split scored against the split in time, if you ran both.]

  1. Rejected

    Shuffle the days

    The model would learn from days after the ones it predicts.

  2. [Sketch of the option]

    Rejected

    [Another option you weighed]

    [Why it lost, in one sentence.]

  3. Chosen

    Split in time, group by person

    The only kind of score that says something about next week.

I chose

Everything after 1 January 2019 held out, with folds grouped by person.

It cost

[What it gave up.]

The limits are in the report, not hidden from it.

The limit What it does What it means
2 in 5 rows have no weather data The environment model sees less than it could. The weather finding is real, and weaker than with full coverage.
A 0 to 4 scale, treated as continuous Simpler to model and explain. The cost shows as bands in the plot.
[A third limit from project_overview.md] [What it does.] [What it means.]
Scatter plot of predicted against actual severity on the test set. The points form horizontal bands, one per step of the 0 to 4 scale.
Fig. 1 · Predicted against actual severity, test set [confirm axes]

Test RMSE · lower is better

  • Naive baseline: tomorrow = today 0.538
  • Linear regression 0.471 · −12%
  • HistGradientBoosting 0.468

Grouped-CV RMSE · what predicts tomorrow

  • Her own symptom history 0.48
  • Environment only, weather included 0.83
  1. Why the plot comes in bands Severity is a 0 to 4 scale, modelled as continuous. A documented trade-off, visible right here.
  2. The bar to clear “Tomorrow is like today” is hard to beat. Gradient boosting barely beat a straight line.
  3. The weather lost Environment alone: 0.83. The body’s own momentum: 0.48.
  1. Kaggle export Flaredown, 686 MB, 7.98M rows
  2. Target Next day’s mean severity
  3. Models Baseline, linear, gradient boosting
  4. Evaluation Hold-out 2019, GroupKFold by user

What to notice

Two hypotheses. H1: a simple model beats “tomorrow is like today”. Supported. H2: a person’s own history predicts better than their environment. Supported, by a wide margin. [Confirm: were they fixed before the models ran?]

regression_health

0.48 / 0.83

Grouped-CV RMSE: a person’s own symptom history against their environment alone. Recent symptoms predict tomorrow far better than the weather does. A study, so there are no users to count.

[What you would do differently, concrete enough to be wrong.]