A cosmetics retailer has 109M behavioral events but lacks the infrastructure to act on them. No existing solution fits the constraints: commodity hardware, zero cloud budget, production-quality analytics.
Raw CSV to live analytics dashboard: each stage produces a versioned artifact that feeds the next. Click any stage to expand the implementation detail.
Full pipeline runtime: ~15 minutes on a single 16 GB machine. Market basket: 10M+ product pairs in 90 seconds via DuckDB self-join. Dashboard queries: under 1 second via pre-aggregated dimensional tables. Total infrastructure cost: $0.
Step through the four optimization layers that turned a guaranteed MemoryError into sub-second interactive analytics. Each step is quantified with before/after numbers.
LightGBM trained on October behavior to predict November purchases. Temporal split prevents leakage. Top-5% threshold driven by campaign budget constraints, not accuracy optimization.
Can't Lose Them: R=1 (long inactive) but F>=4 (historically high-frequency buyers, avg spend over $900). Without behavioral analytics, these users are indistinguishable from churned users. RFM surfaces them explicitly for high-priority reactivation campaigns.
Each decision is explicitly justified in the codebase. Engineers who explain tradeoffs are more hireable than engineers who only explain what they built.
PySpark adds 30-second JVM startup and cluster overhead for zero performance gain at 100M rows. DuckDB matches Spark on single-node workloads via vectorized SIMD execution, costs $0, and ships as a single pip install. The decision is documented inside the dashboard itself.
Polars lazy API pushes query predicates down to the scan layer. scan_parquet reads only the columns needed. sink_parquet writes row groups without holding the full output in RAM. The 109M-row transformation fits in a single-pass stream on a 16 GB machine.
UUID strings as Python objects cost ~80 bytes each. Cast to pl.Categorical, they become 4-byte integer indices into a shared dictionary. With 15 million unique sessions across 109M rows, this single transformation saves 1.14 GB and is the difference between crashing and running.
A random split would leak November behavior (future data) into October training features, inflating AUC artificially. Temporal split mirrors real deployment: train on historical behavior, predict future actions. Harder evaluation, but the resulting 4.5x lift is genuine and production-realistic.
A 0.5 probability threshold optimizes accuracy, but ignores marketing budget reality. Targeting the top 5% of users by propensity score is a business constraint: limited campaign budget. At this threshold, conversion jumps to 36.6% vs 8% baseline, and every user contacted has high expected return.
Explore the full dashboard or read the source code. All analytics queries run in under one second.