Academic / research
Predicting Peak Task Periods in Distributed Clusters
An applied machine-learning / forecasting research project: predict when a distributed cluster's load will peak so compute capacity could potentially be planned more intelligently. A forecasting and capacity-planning study, not a production autoscaling or scheduling system. I built the full pipeline in Python — data preparation, feature engineering, model comparison, time-aware evaluation, and honest limitation reporting.
Problem
Why predict peak demand?
Distributed clusters do not load evenly. The operationally important signal is when demand will peak — that is what drives capacity planning, autoscaling headroom, and the placement of time-sensitive work — and forecasting peak demand is both harder and more useful than predicting average utilization. This project asks whether historical workload behavior can be used to anticipate those peaks, as a concrete application of time-series methods to an infrastructure problem. It studies forecasting; it did not control or optimize a live production cluster.
Research question
The question
Can historical workload behavior be used to predict peak task demand accurately enough to inform capacity-planning decisions? The study evaluates that question on historical data; the answer is framed as forecasting quality, not as proof of production readiness.
Approach
Two complementary models
Two models are compared because they attack the problem from different angles:
- Random Forest. A tree ensemble that can capture nonlinear relationships in engineered workload and time features — strong when the signal is in lagged / rolling features rather than a clean trend.
- Prophet. A forecasting-oriented model built around additive trend and seasonality — a useful baseline for the temporal structure of load, and a clean point of comparison.
Comparing them is the point: one leans on engineered nonlinearity, one on explicit temporal decomposition, so differences in behavior are interpretable. The headline metrics below are the results for the configuration that performed best on the evaluation split; the study did not publish a full per-model comparison table, so a model-versus-model delta is not reported here.
Data / Features
What the models were built from
- Historical workload / task data from a distributed computing environment, resampled onto a regular time grid.
- Temporal features — day-of-week and hour-of-day encodings that make the cyclical structure of load explicit.
- Lag / rolling / history-based features — recent-window statistics derived from prior observations.
- Preprocessing and train / test separation so the evaluation is meaningful. No dataset size, source names, sampling interval, or feature count is claimed beyond what the study documents.
Time-aware evaluation
Chronology must not leak
Forecasting evaluation has to respect chronology: the train / test split is time-ordered, so future observations never leak into the training data used to predict earlier periods. Random shuffling would violate that and overstate performance. The evaluation uses this chronological split; walk-forward validation and cross-validated time-series methods are not claimed, since they were not part of the study.
Results
Reported metrics, interpreted
R²
0.86
The model explains a large portion of the observed variation in load on the evaluated dataset. R² is a measure of explained variance, not a percentage of correct predictions.
MAE
96.21
Mean absolute error — the average magnitude of prediction error, in the target's task-count units. It says how far off a typical prediction is, on average.
MSE
38,487.24
Mean squared error — each error is squared, so larger misses are penalized more heavily. Its large absolute value reflects the unit scale of task counts, not necessarily model weakness.
Read together: the model explains a large portion of the observed variation in load (R² = 0.86), the average prediction error is on the order of ~96 task-count units (MAE = 96.21), and squared errors (MSE = 38,487.24) weight the larger misses more heavily. These are reported with their unit context rather than as a single cherry-picked “accuracy” number — and R² is not a percentage of correct predictions.
Limitations
What the study does and does not prove
- Historical workload patterns can shift (new workload classes, rescheduling policy changes), and the forecast degrades as behavior drifts away from what was observed.
- Unusual or out-of-distribution spikes are harder to predict than the recurring patterns the model learned.
- Evaluation on historical data does not establish production performance — it measures forecasting quality on the evaluated dataset.
- Forecasting quality does not by itself determine the optimal capacity policy; the prediction is an input to a planning decision, not the decision.
Takeaway
Transferable skills, production boundary
What this exercise demonstrates is the method, not the hardware: translating an infrastructure problem into an ML problem, feature engineering, comparing complementary models, time-aware evaluation, interpreting error metrics against their unit scale, and stating limitations honestly.
The production boundary is deliberate: this was a forecasting / capacity-planning study, not a production autoscaling or scheduling system. It was not deployed, used for live inference, wired into automatic scaling, or adopted operationally — and none of that is claimed here.
Academic / research coursework: metrics are the reported study results, not production benchmarks.