Fitness Tracker Analysis.
A 358,497-row PySpark batch pipeline and three scikit-learn models behind a Streamlit dashboard, audited by actually running the shipped pipeline: the dashboard's headline accuracy numbers traced to a model that was never deployed, and one live prediction tool returned the same answer no matter what a user selected. Both are now fixed, live-computed, and covered by regression tests.
$ pytest -q
22 passed in 8.82s
$ python -m src.train_dashboard_models
wrote dashboard/models/metrics.json
$ dashboard/1_Overview.py reads it live:
89.3% accuracy, R2 = 0.918
Not a screenshot. Fixed directly in the source project: the training script now scores itself and the dashboard reads the result live.
Daily records processed
1,959 users, 183 days, Apr to Sep 2023
Held-out accuracy, now live
Computed by the training script itself and shown on the dashboard, not a carried-over notebook number
Bugs found and fixed
A misattributed accuracy claim and a dead dropdown, both closed with regression tests
Live Inference dropdown bug
Used to return one identical prediction for all 6 options, now differentiates correctly
Tests passing
6 new regression tests added for these fixes
Extraction speedup
0.441s -> 0.055s, full 358K rows
Refactor phases
Broken tests to tested package, one day
ETL smoke test
124MB peak RSS
A prototype that looked further along than it was.
A raw fitness export arriving as 183 files nested three directories deep is not yet analyzable. Getting from that to a chart or a prediction means discovering every valid parquet file regardless of nesting depth, computing derived features once rather than once per consumer, and training models whose preprocessing exactly matches what inference applies, all while working within a real constraint: the trained regressor is 326MB and cannot be committed to a public GitHub repo without Git LFS.
The project's own retrospective document, project_refactor.md, is unusually candid about the second problem: by the time of this audit, the repo had accumulated three overlapping training scripts, hardcoded Windows paths, and a test suite that could not even import. Below is what that starting state actually looked like, verified directly rather than taken on its word.
Raw source files
Nested parquet, sibling CSVs ignored
Activity types
walking, running, cycling, swimming, yoga, gym_workout, hiking
What the repo actually looked like.
Not a generic list of tech debt. Six specific, verifiable defects, straight from the project's own retrospective document.
Tests did not even import
pytest -q failed with ModuleNotFoundError: No module named 'src' before a single test could run, per the project's own project_refactor.md.
Three scripts trained the same models, differently
train_dashboard_models.py, train_cloud_models.py and train_segmentation_only.py each built a different Random Forest configuration against the same processed data.
A hardcoded Windows path in a live data generator
generate_streaming_data.py wrote to C:/Project/Fitness Tracker Analysis/data_lake/streaming_input, unrunnable outside one specific machine.
A Docker-only path assumed by application code
etl_pipeline.py assumed /app/..., baking a container-only path into code meant to also run locally.
Documentation ahead of the implementation
The README and system_design.svg describe real-time telemetry, a user feedback loop and CI/CD automation. None of these exist in the code.
No package initialization, ad hoc path bootstrapping everywhere
Every entrypoint needed its own sys.path insertion just to import from src, rather than one shared, tested resolution.
Five phases, one day, one commit each.
Verified directly in git history: cleanup, then bug fixes, then pipeline repair, then modularization, then performance, in that order, each phase a single dated commit.
Phase 1: cleanup
3 -> 1 training scriptsTwo duplicate training scripts (train_cloud_models.py, train_segmentation_only.py) moved to archive/legacy/, leaving one canonical training path.
Phase 2: bug fixes & paths
0 -> 16 tests passingFixed the failing test import (ModuleNotFoundError: No module named 'src') and centralized every path in src/config.py, overridable via FITNESS_TRACKER_ROOT.
Phase 3: pipeline repair
reconnected end to endReconnected extract, transform, load, train and serve into one traceable path a developer can actually run end to end.
Phase 4: modularization
1 -> 5 focused modulesSplit monolithic scripts into src/etl/{extract,transform,load,run}.py and src/models/training.py, each independently importable and testable.
Phase 5: performance
88% faster extractionReplaced a per-file pd.read_parquet + pd.concat loop with a single PyArrow dataset scan across all 183 files.
Left unresolved
2 live entrypointsTwo ETL entrypoints still coexist: the Dockerfile runs the pure-PySpark path, the README's Quickstart runs the PyArrow+PySpark path. The refactor's own plan called for choosing one.
Five stages, a dashboard that never blocks.
Extract, transform, load, train, serve. The serving layer is designed so a missing model file degrades gracefully rather than failing.
$ python -m src.run_pipeline
Extract
One PyArrow dataset scan across all 183 nested parquet files, CSV siblings ignored.
Transform
PySpark: day_of_week, calories_to_steps_ratio, a zero-guard for the divide-by-steps case.
Load
Parquet, partitioned by year and month, overwrite semantics for a simple, reasonable rerun.
Train
3 scikit-learn Pipelines, fit on 100% of the processed data, no held-out split in the script.
Serve
Streamlit reads the processed parquet and, if present, the pickled pipelines.
The pickled sklearn Pipeline is loaded and predicts directly, imputation and scaling already baked in.
A deterministic heuristic (threshold rule, closed-form estimate, or step-ranked band) answers instantly. Nothing blocks, nothing errors.
src/etl_pipeline.py -> src/etl/run.py
What the Dockerfile runs. Pure PySpark end to end: spark.read.parquet straight through to spark.write.parquet.
src/run_pipeline.py
What the README tells a developer to run. PyArrow extraction, PySpark transform, pandas/PyArrow write, then trains the dashboard models in the same process.
The refactor's own plan called for choosing one canonical path. It was not fully closed out, flagged here rather than described as resolved.
Overlapping activities, a generator with no memory.
358,497 synthetic daily records, 1,959 users, six months. Verified directly: pd.read_parquet on the processed dataset in this environment.
Adjacent activities overlap by thousands of steps, cycling and gym_workout, walking and running, running and hiking. A plain step-count threshold cannot cleanly separate them; a Random Forest, evaluating steps, calories and heart rate together, earns its keep here in a way the dashboard's own baseline heuristic cannot.
walking / running / cycling / hiking
calories = steps x factor + random noise
steps and calories correlated by construction
swimming / yoga / gym_workout
calories = random range, independent of steps
no relationship to steps at all
Preprocessing rides inside the model, not beside it.
Every scikit-learn Pipeline bundles imputation, scaling and encoding with the estimator, fit and pickled as one object.
dashboard/utils.py calls pipeline.predict(raw_features) directly. There is no second, hand-maintained feature-transformation function on the serving side that could silently drift from what training used, a bug class closed by construction rather than by discipline.
The same file caps K-Means at min(5, len(user_df)) rather than a hardcoded 5, specifically so a 2-user test fixture does not crash a request for 5 clusters. It is a small line, and it is exactly the kind of detail that separates code written to pass a demo from code written to survive its own test suite.
1# Every pipeline bundles its own preprocessing,2# trained and pickled together with the estimator.3class_pipeline = Pipeline([4 ("imputer", SimpleImputer(strategy="median")),5 ("scaler", StandardScaler()),6 ("classifier", RandomForestClassifier(7 n_estimators=MODEL_TREES, max_depth=20,8 n_jobs=-1, random_state=42,9 )),10])11 12# Capped by data size, not hardcoded: a 2-user13# test fixture cannot crash this call.14n_clusters=min(5, len(user_df))The dashboard cited a model it never shipped.
"84% accuracy" and "R-squared = 0.91" on the Architecture tab matched, to 4 significant figures, a Spark MLlib model from an exploratory notebook, not the Random Forest that dashboard/models/ actually contains. Fixed below: the card now reads its numbers from a metrics.json the training script writes itself.
Held-out accuracy, measured
vs. 84% previously claimed on the dashboard, for a different model
Held-out R-squared, measured
vs. 0.91 previously claimed. RMSE 126.15
Evaluations, before this fix
Now: every training run writes dashboard/models/metrics.json
Per-user step variation
Coefficient of variation across 1,959 users
Activity classifier accuracy
Calorie regressor R-squared
The first two bars in each pair were, to 4 significant figures, the same number, the dashboard's claim was never re-measured after the model behind it changed. The shipped Random Forests turn out to beat that borrowed number, not lag it, which made the missing evaluation more notable, not less. The third bar is no longer a one-off measurement for this case study, it's what dashboard/1_Overview.py reads live from metrics.json today.
Six options, one identical answer, once.
Used to return the same prediction no matter which activity was picked (see Failures & Iterations below). Try it now: every option differentiates correctly against the actual fitted regressor.
Pick any activity from the dropdown a real visitor sees today. Inputs held constant: 8,000 steps, 130bpm, 7.5h sleep.
201.3kcal
Correctly differentiated, 201 to 1,076 kcal depending on activity, for the exact same steps, heart rate and sleep inputs.
dashboard/utils.py exposes get_activity_categories(), a single source of truth read from the processed dataset. The dropdown, the baseline heuristic (the path most visitors actually hit, since models aren't downloaded by default) and the classifier's accuracy page all read from it, so this can't drift apart again the way it used to.
Five personas, and very little signal to find.
Not a bug. A structural consequence of how the synthetic data was generated, worth stating in the same breath as the segmentation results, not after.
Per-user average daily steps, all 1,959 users
Generated for this case study directly from the processed dataset. Real matplotlib output, not a mockup.
The synthetic data generator assigns each user's daily activity independently and uniformly at random, no persona, no per-user bias, nothing that would make a habitually sedentary user stay sedentary across days. Over 183 days, per-user averages regress hard toward the population mean.
Measured directly: the coefficient of variation of avg_steps across all 1,959 users is 7.5%, mean 6,348, min 4,861, max 8,084. K-Means still returns 5 confidently labeled clusters, "Sedentary" through "High Performer", but with this little inter-user variance, it is separating small, largely noise-driven differences, not genuinely distinct behavioral archetypes.
Seven real decisions, not a generic checklist.
What was chosen, the constraint behind it, what it cost, and what actually happened, each verified against the running system.
Preprocessing lives inside the model, not beside it
Every scikit-learn pipeline bundles its own SimpleImputer, StandardScaler and (for the regressor) OneHotEncoder together with the estimator, fit and pickled as one object.
Verified directly: no second feature-engineering implementation exists anywhere in dashboard/.
Two fixed by the refactor. Three fixed here today.
The first two were closed by the project's own same-day rebuild, re-verified here. Three more, found during this case study's research pass, were fixed and covered by new regression tests in this same pass. The last is a data-generation limitation, not a bug, and has no code fix.
The test suite could not even import the project
pytest -q tests/test_etl.py failed with ModuleNotFoundError: No module named 'src', per the project's own project_refactor.md.
Fixed by the project's Phase 2 refactor: path resolution centralized in src/config.py, sys.path bootstrapping added to entrypoints. Re-verified here: 16/16 tests pass.
Three overlapping scripts trained three different models
train_cloud_models.py used n_estimators=30, max_depth=12, min_samples_leaf=10 aimed at a stated "<50MB" target, a different, never-reconciled answer to the same artifact-size problem the canonical script represents.
Fixed by Phase 1: the two non-canonical scripts moved to archive/legacy/, leaving src/models/training.py as the one path exercised by tests and by src/run_pipeline.py.
Every Live Inference calorie prediction returned the same number
The dropdown offered Walking, Running, Cycling, Yoga, HIIT, Strength Training. The regressor's OneHotEncoder was fit on the real category values: cycling, gym_workout, hiking, running, swimming, walking, yoga, lowercase, snake_case, two of which (HIIT, Strength Training) never existed in training data at all. handle_unknown="ignore" silently zeroed every one of the six UI options, and the baseline heuristic (the actual default path most visitors hit) had the identical bug in its own hand-typed activity_factor dict.
Fixed: dashboard/utils.py now exposes get_activity_categories(), a single source of truth read from the processed dataset itself. The Live Inference dropdown and the baseline heuristic both consume it, closing both instances of the same bug. Covered by a new regression test that fits a fixture and asserts the dropdown options equal the fixture's real categories.
The dashboard's headline accuracy figures described a model that was never shipped
"84% accuracy" and "R-squared = 0.91" on the Architecture tab matched, to 4 significant figures, the notebook's genuine Spark MLlib LogisticRegression/LinearRegression results. No code anywhere evaluated the Random Forest models that actually ship.
Fixed: src/models/training.py now runs an 80/20 held-out evaluation (stratified for the classifier) alongside fitting the shipped models on 100% of the data, and writes the result to dashboard/models/metrics.json. dashboard/1_Overview.py reads that file live: 89.3% held-out accuracy, R-squared 0.918, for the models actually in dashboard/models/, with real on-disk file sizes computed the same way rather than hardcoded.
A casing fix on one page never propagated to the page with the live bug
dashboard/pages/3_Advanced_Modelling.py was updated, same day, to lowercase and strip both true and predicted labels before comparing accuracy. dashboard/pages/4_Live_Inference.py, which fed a capitalized string straight into the regressor's encoder, was never touched by that fix.
Fixed as part of F-03: both pages, and the baseline heuristic, now read activity labels from the same get_activity_categories() function, so the two pages can no longer drift apart the way they did before.
The "5 user personas" have very little real signal to separate
The synthetic data generator assigns each user's daily activity independently and uniformly at random, no persona, no per-user bias. Over 183 days, per-user averages regress hard toward the population mean: coefficient of variation on avg_steps across 1,959 users is 7.5%.
Not a code bug, a data-generation limitation. K-Means still produces 5 labeled clusters, but with this little inter-user variance they mostly separate small, largely noise-driven differences rather than genuinely distinct behavioral archetypes.
What's still left open.
- .The two live ETL entrypoints (Dockerfile vs. README Quickstart) still coexist; consolidating them was outside this pass's scope.
- .The "5 user personas" segmentation limitation (below) is a data-generation property, not a code bug, and was left as-is: no amount of dropdown or metrics fixing changes how little inter-user variance the synthetic generator produces.
- .Held-out accuracy/R-squared for the shipped Random Forests are now computed by the training script itself and written to dashboard/models/metrics.json on every run, rather than being a one-off number computed only for this case study.
A dashboard is production surface area.
- A dashboard's "Architecture" tab is production surface area, not decoration. It made falsifiable claims that needed the same verification as any other output, and once verified, the fix was to compute and display the real number, not just to relabel the claim as unverified.
- A handle_unknown="ignore" encoder is safe against crashes and silent about drift at the same time. The actual fix wasn't retyping the dropdown's labels to match, it was collapsing two independent hand-typed label lists (a dropdown and a baseline heuristic dict) into one function so the same drift cannot reopen in only one of them.
- A refactor's own audit document is a primary source. project_refactor.md gave a precise, dated account of the pre-refactor failure modes that would otherwise have needed git archaeology to reconstruct.
- A synthetic-data generator encodes assumptions that downstream models inherit invisibly. A uniform-random, no-persona activity generator quietly sets the ceiling on what any later behavioral-segmentation model can find, and unlike the other two bugs, that one has no code fix.
The dashboard is live, and the fix above shipped with it.