Data Analysis of Correlation and Regression Notebooks
Notebook 1: Covariance, Pearson Correlation and Regression
Google Colab notebook: Open in Google Colab · Raw Notebook: Download .ipynb
Purpose of the notebook
I expanded a basic correlation and regression example into a set of controlled experiments exploring how dataset properties affect statistical results. I varied sample size, noise, outliers, non-linearity, and data range, calculating and visualising correlations, regressions, and R² for each.
Changes and experiments
I first made the notebook reusable by adding helper functions instead of writing all the calculations in one code block.
For the sample-size experiment, I used 10, 30, 100 and 1,000 points. Small datasets produced unstable estimates because individual points heavily influenced the slope and correlation, while larger samples stabilised the relationship.
Increasing the noise spread out the scatter plot, reduced the correlation, and lowered R², showing how noisy data obscures genuine relationships.
When I added outliers, extreme values shifted the regression line noticeably, especially when they were far from the main cluster.
The non-linear experiment produced a clear pattern in the plot, but Pearson correlation and linear regression did not describe it well. This demonstrated that low correlation may reflect an unsuitable method rather than the absence of a relationship.
Finally, restricting the input range made the relationship appear weaker. This demonstrated that models trained on narrow data do not generalise well to real-world variation.
What the results showed
Correlation and regression results depend heavily on dataset conditions. More data points made estimates more reliable; more noise obscured relationships; outliers distorted the fitted line; non-linearity limited linear methods; restricted ranges masked real patterns.
| Experiment | n | Pearson r | R² | Notes |
|---|---|---|---|---|
| Baseline (linear, σ=10) | 1,000 | 0.89 | 0.79 | Clear positive trend |
| Small sample | 10 | 0.95 | 0.91 | Looks strong but unstable across reseeds |
| Large sample | 1,000 | 0.89 | 0.79 | Stable estimate |
| Low noise (σ=2) | 1,000 | 0.99 | 0.99 | Almost deterministic |
| High noise (σ=50) | 1,000 | 0.36 | 0.13 | Relationship largely obscured |
| Clean data | 100 | 0.89 | 0.79 | Reference for outlier test |
| With 3 high-leverage outliers | 103 | 0.21 | 0.04 | Outliers collapse the fit |
| Curved (y ≈ x² + noise) | 300 | -0.05 | 0.00 | Linear methods miss the pattern |
| Full x range | 500 | 0.78 | 0.61 | Reference for range test |
| Restricted x range | 500 | 0.29 | 0.09 | Narrow range hides the trend |
The results demonstrated why statistics should not be interpreted in isolation. Scatter plots mattered just as much as the numbers because they revealed the structure of the data. In practice, good analysis combines statistical measures with visual inspection and careful documentation.
Analysis and conclusions
The results show that statistical outputs cannot be interpreted in isolation from the data that produced them. Sample size, noise, outliers, non-linearity, and the range of observed values each changed correlation strength and regression fit, even though the underlying generative relationship was held constant.
The implications also extend beyond the numbers. Models used in sensitive areas (finance, healthcare, recruitment) need reliable data to produce defensible predictions. Restricted datasets can create systems that work for some groups but not others. Outlier handling is a judgement call: unusual points may be errors, but they may also represent important rare cases. In any setting, assumptions and limitations should be documented.
Critical reflection
What stood out from the notebook was that data analysis is not purely a technical task. A high correlation value or a neat regression line can be misleading if the dataset is too small, too narrow or affected by unexamined bias. Likewise, a low correlation value can be misleading if the relationship is non-linear. Machine learning professionals therefore need to understand the context of the dataset before choosing or evaluating an algorithm.
The notebook also shows how basic statistical methods can raise important professional questions, and why dataset quality, representativeness and transparency are central to responsible machine learning.
Notebook 2: Linear Regression
Google Colab notebook: Open in Google Colab · Raw Notebook: Download .ipynb
Purpose of the notebook
Starting with a 13-point baseline model, I explored how changes to the dataset affected linear regression. I varied the sample size, added consistent points, introduced noise and outliers, and restricted the range. For each variation, I recalculated the correlation, refitted the regression line, and observed how the predictions changed.
Changes and experiments
I tested five variations of the original 13-point dataset:
Sample size: Repeated random sampling from a simulated population showed the slope's standard deviation falling from ~1.13 (n=5) to ~0.16 (n=100). With only 5 points, the fitted line looked plausible but varied widely across resamples; with 100, it stabilised.
Trend-consistent points: Adding 10 points that followed the existing negative pattern strengthened the correlation (-0.76 → -0.88) and raised R² (0.58 → 0.78). Prediction at x=10 stayed essentially unchanged (~85.5).
Noisy data: Adding inconsistent points effectively destroyed the relationship: Pearson r flipped to +0.04 and R² collapsed to 0.001. The algorithm still produced a line and prediction, but both were meaningless.
Outlier: A single contradictory point (x=18, y=120) almost completely wiped out the negative trend (r: -0.76 → -0.10, R²: 0.58 → 0.009). Outliers deserve investigation; they may be errors or represent important rare cases.
Restricted range: Using only x ≤ 8 produced stronger local statistics (r=-0.90, R²=0.82) but the predicted value at x=10 drifted to 76.4 (from 85.6 baseline). High correlation did not guarantee reliable extrapolation.
What the results showed
The results changed sharply with the quality of the data. Consistent data strengthened relationships, noise obscured them, outliers distorted them, and narrow ranges masked extrapolation problems. The algorithm could not detect when the data was unsuitable; it generated predictions regardless.
| Experiment | n | Pearson r | R² | Predict at x=10 |
|---|---|---|---|---|
| Original | 13 | -0.76 | 0.58 | 85.6 |
| + Trend-consistent points | 23 | -0.88 | 0.78 | 85.5 |
| + Noisy points | 23 | +0.04 | 0.001 | 89.6 |
| + 1 Outlier (18, 120) | 14 | -0.10 | 0.009 | 91.5 |
| Restricted range (x ≤ 8) | 8 | -0.90 | 0.82 | 76.4 |
Analysis and conclusions
The clearest finding is that data quality matters as much as algorithm choice. The same LinearRegression call produced a reasonable model on the original data, a stronger model when trend-consistent points were added, and an essentially meaningless model once noise or a single contradictory outlier was introduced. Restricting the range produced the most subtle failure mode: the local fit improved (r=−0.90) while the prediction at x=10 drifted by nearly 10 units, illustrating that a high in-sample R² does not guarantee reliable extrapolation.
In practical settings such as loan scoring, pricing and risk assessment, regression models trained on biased, incomplete or unrepresentative data will still produce confident predictions. Those predictions will still be acted on, even though the algorithm cannot tell the operator when its inputs are unsuitable.
Critical reflection
Regression is fundamentally about fitting a line to a specific dataset. A high R² might seem to validate the model, but statistics cannot guarantee practical utility. Professional responsibility lies in understanding data limitations, validating assumptions, investigating anomalies, and acknowledging the conditions that predictions depend on.
Notebook 3: Multiple Linear Regression
Google Colab notebook: Open in Google Colab · Raw Notebook: Download .ipynb
Purpose of the notebook
Using a small car dataset, I explored how changes to the data affected multiple linear regression. I predicted CO2 emissions from Weight and Volume, then tested different sample sizes, trend-consistent points, noisy data, outliers, and restricted ranges to see how each change affected correlations, coefficients, R², and predictions.
Changes and experiments
I added helper functions and expanded the original single-prediction example into a series of controlled experiments:
Baseline: Fitted a model with Weight and Volume predicting CO2, establishing baseline correlations (Weight–CO2: 0.55, Volume–CO2: 0.59, R²: 0.38). Predicted CO2 at Weight=2300, Volume=1300 ≈ 107.2 g.
Fewer data points: Reduced to 12 samples; estimates became unstable, correlations weakened toward zero (Weight–CO2 even flipped sign), and R² dropped to ~0.18.
Trend-consistent points: Added points following the expected pattern, strengthening correlations (~0.62 / 0.64) and lifting R² to ~0.44.
Noisy data: Added conflicting points that reversed relationships, collapsing R² to ~0.02, showing how poor data quality can mask real patterns.
Outlier: One unusual point pulled coefficients sharply, dropping R² to ~0.04 and weakening both correlations.
Restricted range: Using only lower weight values produced moderate local statistics (R² ~0.22) but unreliable extrapolation to heavier cars.
What the results showed
The model's apparent reliability shifted across the experiments as data quality, sample size, and range coverage changed. R² ranged from ~0.02 to ~0.44, yet the algorithm still produced confident predictions in every case and could not signal when the data was unsuitable.
| Experiment | n | Corr W–CO2 | Corr V–CO2 | R² | Pred CO2 (2300/1300) |
|---|---|---|---|---|---|
| Baseline | 36 | 0.55 | 0.59 | 0.38 | 107.2 |
| Fewer points | 12 | -0.17 | 0.08 | 0.18 | 68.6 |
| + Trend-consistent | 41 | 0.62 | 0.64 | 0.44 | 109.3 |
| + Noisy points | 41 | 0.06 | 0.13 | 0.02 | 95.6 |
| + Outlier | 37 | 0.17 | 0.20 | 0.04 | 105.5 |
| Restricted (low weight) | 18 | 0.25 | 0.43 | 0.22 | 85.1 |
Analysis and conclusions
Multiple linear regression appears deceptively definitive: two coefficients, an intercept, an R². Yet every one of those numbers shifted substantially across the experiments. Even the baseline R² of 0.38 already signals that Weight and Volume alone are incomplete predictors of CO2; fuel type, engine efficiency, vehicle age and driving conditions are all absent from the dataset. This omission of relevant predictors is what limits the model's usefulness.
Models like this are routinely used in emissions reporting, insurance pricing and regulatory compliance. A practitioner deploying one would need to justify the choice of predictors, document how the dataset was assembled, and be transparent about the proportion of variation the model leaves unexplained.
Critical reflection
The multiple-regression notebook showed that regression is fundamentally estimation; it is curve-fitting adapted to a specific dataset. The models were too small and simplified for real decision-making. Their value lies in showing why professional judgement matters as much as technical execution: understanding data limitations, investigating anomalies, documenting assumptions, and refusing to overstate what statistics can prove.
Notebook 4: Polynomial Regression
Google Colab notebook: Open in Google Colab · Raw Notebook: Download .ipynb
Purpose of the notebook
Using tollbooth data with a curved relationship between time of day and vehicle speed, I explored how changes to the dataset affected polynomial regression. I fitted polynomial models, then varied the sample size, added trend-consistent and noisy data, introduced outliers, restricted the range, and compared polynomial degrees to see how each change affected reliability.
Changes and experiments
I added helper functions and expanded the original third-degree polynomial example into controlled experiments:
Baseline: Fitted a degree-3 polynomial to predict speed at 17:00 (≈88.9). R² was 0.94 but Pearson r was only 0.43, because the relationship is curved, not linear.
Fewer data points: Used every other observation. R² stayed high (~0.96) and the prediction barely moved (~88.5), but the model was based on much less evidence and would be more vulnerable to sampling bias.
Trend-consistent points: Added points following the curved pattern; R² stayed at ~0.95 and predictions remained stable.
Noisy data: Added conflicting points that distorted the curve, dropping R² to ~0.65 and pulling the 17:00 prediction down to ~81.
Outlier: One extreme value (x=11, y=130) pulled the curve sharply, collapsing R² to ~0.39 and shifting predictions to ~92.
Restricted range: Using only 05:00–16:00 data produced strong local fit (R² ~0.94) but the 17:00 prediction dropped to ~77, showing how extrapolation outside the observed range becomes unreliable.
Polynomial degrees: Compared degrees 1 through 6. Degree 1 underfitted (R²=0.18); degree 3 captured the pattern well (R²=0.94); degree 6 fitted training data even more closely (R²=0.98) but risked overfitting noise rather than learning generalisable structure.
What the results showed
Polynomial regression handled curved patterns effectively, but the results were highly sensitive to data quality and model complexity. Predictions changed across the experiments, while Pearson correlation alone proved insufficient for evaluating curved relationships. Outliers and noise distorted the curves, restricted ranges masked extrapolation problems, and higher-degree polynomials improved the training fit at the risk of overfitting.
| Experiment | n | Pearson r | R² | RMSE | Predict at x=17 |
|---|---|---|---|---|---|
| Baseline (degree 3) | 18 | 0.43 | 0.94 | 3.5 | 88.9 |
| Fewer points (every other) | 9 | 0.26 | 0.96 | 2.9 | 88.5 |
| + Trend-consistent | 22 | 0.49 | 0.95 | 3.2 | 88.9 |
| + Noisy points | 22 | 0.22 | 0.65 | 9.2 | 81.1 |
| + 1 Outlier (11, 130) | 19 | 0.33 | 0.39 | 14.5 | 92.3 |
| Restricted (05:00–16:00) | 11 | 0.95 | 0.94 | 2.0 | 77.1 |
Polynomial degree comparison (original data):
| Degree | R² | RMSE | Predict at x=17 |
|---|---|---|---|
| 1 | 0.18 | 13.4 | 83.9 |
| 2 | 0.76 | 7.3 | 81.2 |
| 3 | 0.94 | 3.5 | 88.9 |
| 4 | 0.95 | 3.2 | 87.2 |
| 6 | 0.98 | 2.1 | 84.3 |
Analysis and conclusions
The degree comparison is the clearest illustration of the bias and variance trade-off in this set of notebooks. A degree-1 model underfits the curved tollbooth pattern (R²=0.18) and a degree-6 model achieves the best training R² (0.98), but the degree-6 fit is shaped as much by noise as by the underlying pattern. This is visible in how its 17:00 prediction drifts to 84.3 while the more conservative degree-3 model predicts 88.9. The right choice is the simplest model that captures the structure, not the one with the highest R².
The other experiments echo points seen in the linear cases (outliers and noisy points distort the curve, restricted ranges break extrapolation), but they land harder here because polynomial models are more flexible and therefore more sensitive to each of these problems. In a real traffic-prediction setting, an overfitted model deployed against limited or biased data could produce confidently wrong outputs that feed into planning or safety decisions.
Critical reflection
The polynomial experiment showed that more flexible models are not automatically better. Polynomial regression handles curves that linear regression cannot, but it is also more sensitive to outliers and prone to overfitting. Model selection requires balancing expressiveness with reliability while accounting for the limits of the data. High R² does not guarantee practical utility, especially when data is limited, noisy, or used beyond its observed range.