Fitness Tracker Analysis.
A batch pipeline that cleans 358,497 rows of synthetic fitness data, trains three scikit-learn models on it, and renders the results as a static site with no backend. Every training run scores itself on a held-out split, so the accuracy and R-squared shown here are numbers the pipeline produces itself, not figures computed once and pasted in.
$ pytest -q
22 passed in 7.56s
$ python -m src.run_pipeline
wrote artifacts/metrics.json
$ python scripts/export_frontend_artifacts.py
89.3% accuracy, R2 = 0.918, now in frontend/public/data/
Not a screenshot. Real output from running the pipeline end to end, the export step is what the static site actually reads.
Daily records processed
1,959 users, 183 days, Apr to Sep 2023
Held-out accuracy
RandomForestClassifier, scored on an untouched 20% split
Held-out R-squared
RandomForestRegressor, RMSE 126.15
Tests passing
Unit tests plus one full pipeline integration test
Tests passing
pytest -q, 7.56s
Extraction speedup
0.441s -> 0.055s, single PyArrow scan across all 358K rows
Held-out eval floor
Below this, or with a class under 2 examples, metrics report null
Pipeline stages
Extract, transform, load, train, export
Raw exports, not yet anything you can use.
A fitness tracker export arriving as 183 files nested three directories deep is not yet analyzable. Getting from that to a chart or a prediction means discovering every valid parquet file regardless of nesting depth, computing derived features once rather than once per consumer, and training models whose preprocessing exactly matches what inference applies later, all while working within a real constraint: the trained regressor is large enough that it has no business being committed to a public repository.
None of that is exotic on its own. What makes it worth doing carefully is that every one of those steps has an easy, wrong shortcut, reading files one at a time instead of scanning the dataset as a whole, letting a division by zero on a rest day silently poison a ratio, training a model without ever scoring it against data it hasn't seen. The pipeline below is built so those shortcuts fail loudly, in a test, rather than quietly, in a number someone eventually has to trust.
Raw source files
Nested parquet, sibling CSVs ignored
Activity types
walking, running, cycling, swimming, yoga, gym_workout, hiking
Five stages, a frontend that never blocks.
Extract, transform, load, train, export. Everything past the export step is a static file the frontend reads; nothing in production depends on a live process.
$ python -m src.run_pipeline
Extract
One PyArrow dataset scan across all 183 nested parquet files, CSV siblings ignored.
Transform
PySpark: day_of_week, calories_to_steps_ratio, a zero-guard for the divide-by-steps case.
Load
Parquet, partitioned by year and month, overwrite semantics for a simple, reasonable rerun.
Train
3 scikit-learn Pipelines, each scored on a held-out split before being refit on all the data.
Export
Deterministic JSON aggregates and a compressed parquet copy, written for the static frontend to read.
The export step loads the pickled sklearn Pipeline and scores it directly, imputation and scaling already baked in.
A deterministic heuristic (threshold rule, closed-form estimate, or step-ranked band) fills in instantly. Nothing blocks, nothing errors.
src/etl_pipeline.py -> src/etl/run.py
What the Dockerfile runs. Pure PySpark end to end: spark.read.parquet straight through to spark.write.parquet.
src/run_pipeline.py
What local development runs. PyArrow extraction, PySpark transform, PyArrow write, then trains the models in the same process.
The refactor's own plan called for choosing one canonical path. It was not fully closed out, flagged here rather than described as resolved.
Overlapping activities, a generator with no memory.
358,497 synthetic daily records, 1,959 users, six months. Verified directly against the processed dataset in this environment.
Adjacent activities overlap by thousands of steps, cycling and gym_workout, walking and running, running and hiking. A plain step-count threshold cannot cleanly separate them; a Random Forest, evaluating steps, calories and heart rate together, earns its keep here in a way a simple rule of thumb cannot.
walking / running / cycling / hiking
calories = steps x factor + random noise
steps and calories correlated by construction
swimming / yoga / gym_workout
calories = random range, independent of steps
no relationship to steps at all
Preprocessing rides inside the model, not beside it.
Every scikit-learn Pipeline bundles imputation, scaling and encoding with the estimator, fit and pickled as one object.
src/serving.py calls pipeline.predict(raw_features) directly. There is no second, hand-maintained feature-transformation function that could silently drift from what training used, a bug class closed by construction rather than by discipline.
The same module caps K-Means at min(5, len(user_df)) rather than a hardcoded 5, specifically so a 2-user test fixture does not crash a request for 5 clusters. It is a small line, and it is exactly the kind of detail that separates code written to pass a demo from code written to survive its own test suite.
1# Every pipeline bundles its own preprocessing,2# trained and pickled together with the estimator.3class_pipeline = Pipeline([4 ("imputer", SimpleImputer(strategy="median")),5 ("scaler", StandardScaler()),6 ("classifier", RandomForestClassifier(7 n_estimators=MODEL_TREES, max_depth=20,8 n_jobs=-1, random_state=42,9 )),10])11 12# Capped by data size, not hardcoded: a 2-user13# test fixture cannot crash this call.14n_clusters=min(5, len(user_df))Numbers the pipeline produces itself.
Every training run clones the shipped pipeline, fits it on an 80/20 split, and scores the untouched 20%. The model that actually ships is still fit on all the data; the split only exists to produce an honest number.
Held-out accuracy
Activity classifier, stratified 80/20 split
Held-out R-squared
Calorie regressor
Held-out RMSE
Calorie regressor, same split
Per-user step variation
Coefficient of variation across 1,959 users
Five clusters, and very little signal to find.
Not a bug. A structural consequence of how the synthetic data was generated, worth stating in the same breath as the segmentation results, not after.
Per-user average daily steps, all 1,959 users
Generated directly from the processed dataset. Real matplotlib output, not a mockup.
The synthetic data generator assigns each user's daily activity independently and uniformly at random, no persona, no per-user bias, nothing that would make a habitually sedentary user stay sedentary across days. Over 183 days, per-user averages regress hard toward the population mean.
Measured directly: the coefficient of variation of avg_steps across all 1,959 users is 7.5%, mean 6,348, min 4,861, max 8,084. K-Means still returns 5 confidently labeled clusters, "Sedentary" through "High Performer", but with this little inter-user variance, it is separating small, largely noise-driven differences, not genuinely distinct behavioral archetypes.
Six real decisions, not a generic checklist.
What was chosen, the constraint behind it, what it cost, and what actually happened, each verified against the running system.
Preprocessing lives inside the model, not beside it
Every scikit-learn pipeline bundles its own SimpleImputer, StandardScaler and (for the regressor) OneHotEncoder together with the estimator, fit and pickled as one object.
Verified directly: no second feature-engineering implementation exists anywhere in src/serving.py.
Two real gaps, closed and regression-tested.
Found while building the evaluation and inference paths, fixed in the same pass, and covered by new tests so neither can reopen silently.
Model accuracy had never actually been measured
Every model was fit on 100% of the processed data and never scored against anything held out, so there was no real accuracy or R-squared number anywhere in the repository, only what the full-data fit happened to produce.
src/models/training.py now clones each pipeline, fits it on an 80/20 split (stratified for the classifier), and scores the untouched 20%. Below 20 rows, or a class with fewer than 2 examples, it reports null instead of a number computed from noise. Covered by test_metrics_are_null_rather_than_misleading_on_too_little_data and test_metrics_report_real_held_out_performance_on_enough_data.
Two hand-typed activity lists could silently disagree
The prediction tool's options and the calorie baseline's activity_factor dictionary were each their own list of activity labels, with no shared source. A OneHotEncoder configured with handle_unknown="ignore" would zero out any label it didn't recognize rather than error, so a mismatch had no visible failure mode.
get_activity_categories() in src/serving.py derives the category list once, from the processed dataset itself, and every consumer, the classifier's encoder, the frontend's prediction form, reads from it. Regression-tested: 22/22 passing.
What's still left open.
- .Two ETL entrypoints coexist: the Docker image runs a pure-PySpark path (src/etl_pipeline.py), local development runs a PyArrow-plus-Spark path that also trains the models (src/run_pipeline.py). Both produce the same processed schema through different code.
- .The synthetic data generator assigns each user's daily activity independently and uniformly at random, no per-user persona. Per-user step averages regress hard toward the population mean (coefficient of variation 7.5% across 1,959 users), so the segmentation model separates real but small differences rather than distinct behavioral archetypes.
- .Tests run manually with pytest. There is no CI configured yet.
An unscored model is just a guess.
- An unscored model does not have an accuracy number, it has a guess. Cloning the pipeline for an honest held-out split, and reporting null below a data-size floor rather than a misleading figure, is what turned "89.3%" into a number worth putting on a page.
- A single source of truth beats a second fix in a second place. Deriving activity categories once and having every consumer read from it was less code than two hand-typed lists, and it closes the whole bug class rather than one instance of it.
- A synthetic-data generator encodes assumptions that downstream models inherit invisibly. A uniform-random, no-persona activity generator quietly sets the ceiling on what any later segmentation model can find, worth stating next to the results, not after them.
The pipeline runs, the tests pass, and the site reads what it writes.