Predicting Oat Crops from Satellite Data: A Classifier Comparison

This notebook reformats a report I completed for a coursework assignment in FIT3152 - Data Analytics at Monash University. I did all the code, judgement and analysis myself. I used AI to reorder the sections and rewrite the write-up to match the new structure. See the original submission here.

Overview

Farmers, insurers and agricultural agencies need a cheap way to know what’s growing in a field. Walking every field costs time and money, but satellite radar and optical sensors already pass over farmland every few days. I used that satellite data to build a model that tells whether a given parcel grows Oats or something else.

I turned this into a broader exercise in applied machine learning. I explored a messy, imbalanced dataset, built and compared eight classification techniques, and worked out which measurements matter. Throughout, I engineered around the fact that Oats are the rare crop in this data, tracking precision, recall, F1-score, balanced accuracy and ROC/AUC instead of relying on accuracy alone. Accuracy alone is misleading here, as the Baseline section shows.

At a glance:

I analysed ~1.6 million data points: a 5,000-farm sample, drawn from 100,000 available, each described by 20 satellite features
Only 14.3% of farms grow Oats, a roughly 1-in-6 class imbalance and the central challenge of the project
I built and benchmarked 8 classifier families head-to-head: Decision Tree, Naive Bayes, Bagging, Boosting (AdaBoost), Random Forest, Balanced Random Forest, a Neural Network, and XGBoost
A plain decision tree hits 84.7% accuracy by predicting every farm as “not Oats”; it never once detects the crop it’s supposed to find. By resampling for the imbalance, without touching a single input feature, I lifted the F1-score for detecting Oats from 0.00 to 0.44 and balanced accuracy from 50% to 69.8%
I trained 20 XGBoost models, one per feature count, and found that 9 of the 20 satellite features carried nearly all of the predictive signal

The Data

The dataset is a modified version of the Winnipeg crop mapping dataset (UCI Machine Learning Repository): real farmland parcels near Winnipeg, Canada, described by features from fused optical and radar satellite imagery. Each of the 30 original features (A01-A30) measures a different satellite band or radar polarisation, a spectral and radar fingerprint of what’s growing on that patch of land, captured without setting foot on it. The target, Class, flags whether the parcel grows Oats (1) or another crop (0).

To keep the analysis at an individual, manageable scale, I used a fixed random seed (my student ID) to draw a working sample of 5,000 farms and a random subset of 20 of the 30 features:

set.seed(SEED)
WD = read.csv("WinnData.csv", stringsAsFactors = TRUE)
WD = WD[sample(nrow(WD), 5000, replace = FALSE), ]
WD = WD[, c(sort(sample(1:30, 20, replace = FALSE)), 31)]
dim(WD)

## [1] 5000   21

names(WD)

##  [1] "A01"   "A02"   "A03"   "A04"   "A05"   "A10"   "A11"   "A13"   "A14"  
## [10] "A18"   "A20"   "A21"   "A22"   "A23"   "A24"   "A25"   "A27"   "A28"  
## [19] "A29"   "A30"   "Class"

Class balance

oats_count = sum(WD$Class == 1)
other_count = sum(WD$Class == 0)
oats_to_other_ratio = oats_count / other_count

data.frame(Class = c("Oats (1)", "Other (0)"),
           Count = c(oats_count, other_count),
           Proportion = round(c(oats_count, other_count) / nrow(WD), 4))

##       Class Count Proportion
## 1  Oats (1)   716     0.1432
## 2 Other (0)  4284     0.8568

cat("Oats-to-Other ratio:", round(oats_to_other_ratio, 4))

## Oats-to-Other ratio: 0.1671

The proportion of Oats to Other comes to 0.1671335, so the dataset is imbalanced: the majority class is Other, the minority class is Oats. Only 716 of the 5,000 farms (14.3%) grow Oats. That imbalance shapes almost every modelling decision from here on: plain accuracy rewards a model that ignores the minority class.

Distributions and data quality

round(sapply(WD[, 1:20], summary), 3)

##           A01     A02   A03   A04     A05     A10   A11   A13   A14   A18   A20
## Min.    0.000 -25.810 0.048 0.006 -22.423 -16.346 0.000 0.000 0.007 0.079 0.000
## 1st Qu. 0.000 -20.135 0.103 0.015 -16.214  -8.681 0.000 0.000 0.035 0.829 0.000
## Median  0.000 -16.288 0.214 0.077   0.000   0.000 0.006 0.074 0.149 0.943 0.530
## Mean    0.106 -15.974 0.366 0.274  -7.677  -4.258 0.010 0.106 0.313 1.077 0.583
## 3rd Qu. 0.197 -12.588 0.604 0.522   0.000   0.000 0.010 0.196 0.572 1.324 1.061
## Max.    0.575   0.431 1.289 1.183   0.000   0.000 0.155 0.568 1.169 1.944 2.197
##             A21   A22   A23    A24   A25   A27   A28    A29   A30
## Min.      1.083 0.154 0.000  1.015 0.041 0.001 0.000 -4.042 0.000
## 1st Qu.   3.577 0.412 0.000  3.123 0.493 0.031 0.000  0.076 0.000
## Median    6.300 0.557 0.000  4.632 0.632 0.082 0.000  0.195 0.018
## Mean      7.192 0.669 0.253  5.843 0.583 0.290 0.310  0.319 0.026
## 3rd Qu.   9.888 0.917 0.519  8.191 0.679 0.542 0.681  0.582 0.044
## Max.    107.165 1.664 0.728 15.109 0.836 1.072 1.131  1.224 0.131

sort(round(sapply(WD[, 1:20], sd), 4))

##    A11    A30    A13    A01    A25    A23    A04    A03    A27    A14    A22 
## 0.0182 0.0315 0.1212 0.1244 0.1288 0.2719 0.3210 0.3211 0.3276 0.3298 0.3425 
##    A18    A28    A29    A20    A24    A10    A21    A02    A05 
## 0.3432 0.3452 0.3643 0.5291 3.3230 4.5456 4.9517 4.9832 8.0644

boxplot(WD[, 1:20], main = "Distribution of satellite features", las = 2, cex.axis = 0.7)

Looking at the table and the boxplot, a few things stood out to me:

Several variables sit at 0 for Min., 1st Qu., and even Median, such as A01, A11 and A13, meaning a large share of zero values. These are candidates for omission, since they might not carry much information and could hurt model performance.
Some features take negative values, namely A02, A05, A10 and A29.
Two variables have exceptionally large Max. values, A21 and A24, suggesting outliers. A21’s maximum (107.2) runs past ten times its 75th percentile (9.9), and A24 shows the same pattern. I kept both features in; the Feature Importance section below shows they matter.
A few variables have very low SD, such as A30 and A11. These are also omission candidates, since such low-variance features likely carry little information.
No missing values anywhere in the sample, confirmed with any(is.na(WD)), so I skipped imputation.

corr = melt(round(cor(WD[, 1:20]), 2))
ggplot(corr, aes(x = Var1, y = Var2, fill = value)) +
  geom_tile(color = "white") +
  scale_fill_gradient2(low = "blue", mid = "white", high = "red", midpoint = 0) +
  labs(title = "Feature correlation heat map", x = NULL, y = NULL) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1))

The heat map adds a few more useful points. The variables in the dataset are mostly weakly correlated: the strongest pairwise correlation anywhere in the data is 0.37 (between A11 and A02), and the average absolute correlation across all pairs is 0.05. Some pairs are correlated moderately, shown as brighter red, but none too strongly. A handful of features have negligible negative correlations.

Preprocessing

To make the dataset suitable for model fitting, I performed one step:

WD$Class = as.factor(WD$Class)

Convert the data type of the label column Class from int to factor.

No other steps were needed:

No missing values, as stated on the UCI Machine Learning Archive’s website and confirmed with any(is.na(WD)).
No need for feature scaling, since none of the five modelling techniques below depend on distance-based calculations; they handle raw feature magnitudes as they are. I do introduce scaling later, right before the Neural Network, where it matters.

Setting Up a Fair Evaluation

I split the dataset into two: a training set and a test set, 70/30, using my student ID as the random seed again. I also defined the metrics used to judge every model once, so I apply them identically everywhere.

set.seed(SEED)
train.row = sample(1:nrow(WD), 0.7 * nrow(WD))
WD.train = WD[train.row, ]
WD.test = WD[-train.row, ]
cat("Training rows:", nrow(WD.train), " | Test rows:", nrow(WD.test))

## Training rows: 3500  | Test rows: 1500

Given the class imbalance, I track five complementary metrics for every model, using the performance_metrics() function defined in the setup chunk unchanged throughout the notebook:

Precision: of the farms flagged “Oats”, how many are Oats? (cost of false alarms)
Recall: of the farms that are Oats, how many did the model catch? (cost of missed detections)
Accuracy: overall fraction correct. Intuitive, but misleading on an imbalanced dataset: a model that always predicts “Other” already scores 85.7%.
F1-score: the harmonic mean of precision and recall; punishes models that only do well on one of the two.
Balanced accuracy (BAcc): the average of recall and specificity; treats both classes as equally important regardless of how many examples of each exist. I introduce it from the imbalance-focused models onward, since it answers whether a model is better at finding the class we care about.

Baseline Model Bake-off

I implemented five classic classification techniques at their default settings, so I could compare them on equal footing before any tuning or imbalance-handling:

Decision Tree: implemented with tree() from the package tree.
Naive Bayes: implemented with naiveBayes() from the package e1071.
Bagging: implemented with bagging() from the package adabag. As the parameter mfinal = 100 by default, a comprehensive plot isn’t practical, so I skip a visualisation for it.
Boosting: implemented with boosting() from the package adabag, same mfinal = 100 default and the same reason for skipping a plot.
Random Forest: implemented with randomForest() from the package randomForest. ntree = 500 by default rules out a comprehensive plot here too.

set.seed(SEED); WD.tree = tree(Class ~ ., data = WD.train)
WD.predtree = predict(WD.tree, WD.test, type = "class")
t1 = table(Actual_Class = WD.test$Class, Predicted_Class = WD.predtree)

set.seed(SEED); WD.bayes = naiveBayes(Class ~ ., data = WD.train)
WD.predbayes = predict(WD.bayes, WD.test)
t2 = table(Actual_Class = WD.test$Class, Predicted_Class = WD.predbayes)

set.seed(SEED); WD.bag = bagging(Class ~ ., data = WD.train)
WD.predbag = predict.bagging(WD.bag, WD.test)
t3 = table(Actual_Class = WD.test$Class, Predicted_Class = WD.predbag$class)

set.seed(SEED); WD.boost = boosting(Class ~ ., data = WD.train)
WD.predboost = predict.boosting(WD.boost, WD.test)
t4 = table(Actual_Class = WD.test$Class, Predicted_Class = WD.predboost$class)

set.seed(SEED); WD.rf = randomForest(Class ~ ., data = WD.train)
WD.predrf = predict(WD.rf, WD.test)
t5 = table(Actual_Class = WD.test$Class, Predicted_Class = WD.predrf)

I place a set.seed(SEED) immediately before every stochastic model call, which makes every result in this notebook reproducible on a re-run (see the Reproducibility Notes at the end).

plot(WD.tree); text(WD.tree, pretty = 0)

Confusion matrices and metrics

t1; t2; t3; t4; t5

##             Predicted_Class
## Actual_Class    0    1
##            0 1270    0
##            1  230    0

##             Predicted_Class
## Actual_Class    0    1
##            0 1141  129
##            1  178   52

##             Predicted_Class
## Actual_Class    0    1
##            0 1270    0
##            1  228    2

##             Predicted_Class
## Actual_Class    0    1
##            0 1225   45
##            1  185   45

##             Predicted_Class
## Actual_Class    0    1
##            0 1264    6
##            1  221    9

WD.pred.tree = predict(WD.tree, WD.test, type = "vector")
WDD.pred = prediction(WD.pred.tree[, 2], WD.test$Class)
WD.pred.bayes = predict(WD.bayes, WD.test, type = "raw")
WDN.pred = prediction(WD.pred.bayes[, 2], WD.test$Class)
WDBag.pred = prediction(WD.predbag$prob[, 2], WD.test$Class)
WDBoost.pred = prediction(WD.predboost$prob[, 2], WD.test$Class)
WD.pred.rf = predict(WD.rf, WD.test, type = "prob")
WDR.pred = prediction(WD.pred.rf[, 2], WD.test$Class)

plot(performance(WDD.pred, "tpr", "fpr"), col = "steelblue", lwd = 2,
     main = "ROC curves: baseline classifiers")
abline(0, 1, col = "grey60", lty = 2)
plot(performance(WDN.pred, "tpr", "fpr"), add = TRUE, col = "orange", lwd = 2)
plot(performance(WDBag.pred, "tpr", "fpr"), add = TRUE, col = "forestgreen", lwd = 2)
plot(performance(WDBoost.pred, "tpr", "fpr"), add = TRUE, col = "brown", lwd = 2)
plot(performance(WDR.pred, "tpr", "fpr"), add = TRUE, col = "darkgreen", lwd = 2)
legend("bottomright",
       legend = c("Decision Tree", "Naive Bayes", "Bagging", "Boosting", "Random Forest"),
       col = c("steelblue", "orange", "forestgreen", "brown", "darkgreen"), lwd = 2)

auc.tree = as.numeric(performance(WDD.pred, "auc")@y.values)
auc.bayes = as.numeric(performance(WDN.pred, "auc")@y.values)
auc.bag = as.numeric(performance(WDBag.pred, "auc")@y.values)
auc.boost = as.numeric(performance(WDBoost.pred, "auc")@y.values)
auc.rf = as.numeric(performance(WDR.pred, "auc")@y.values)

baseline_metrics = rbind(
  "Decision Tree" = c(performance_metrics(t1["1","1"], t1["1","0"], t1["0","1"], t1["0","0"]), AUC = round(auc.tree, 4)),
  "Naive Bayes"   = c(performance_metrics(t2["1","1"], t2["1","0"], t2["0","1"], t2["0","0"]), AUC = round(auc.bayes, 4)),
  "Bagging"       = c(performance_metrics(t3["1","1"], t3["1","0"], t3["0","1"], t3["0","0"]), AUC = round(auc.bag, 4)),
  "Boosting"      = c(performance_metrics(t4["1","1"], t4["1","0"], t4["0","1"], t4["0","0"]), AUC = round(auc.boost, 4)),
  "Random Forest" = c(performance_metrics(t5["1","1"], t5["1","0"], t5["0","1"], t5["0","0"]), AUC = round(auc.rf, 4))
)
knitr::kable(baseline_metrics, caption = "Baseline classifier comparison")

Baseline classifier comparison
	recall	precision	accuracy	f1_score	bacc	AUC
Decision Tree	0.0000	NaN	0.8467	0.0000	0.5000	0.6544
Naive Bayes	0.2261	0.2873	0.7953	0.2530	0.5623	0.6779
Bagging	0.0087	1.0000	0.8480	0.0172	0.5043	0.7136
Boosting	0.1957	0.5000	0.8467	0.2812	0.5801	0.7021
Random Forest	0.0391	0.6000	0.8487	0.0735	0.5172	0.7392

bp = as.data.frame(baseline_metrics)
bp$model = rownames(bp)
bp_long = melt(bp, id.vars = "model", variable.name = "metric", value.name = "value")
ggplot(bp_long, aes(x = metric, y = value, fill = model)) +
  geom_bar(position = "dodge", stat = "identity") +
  labs(title = "Baseline model performance", y = NULL, x = NULL)

Is there a single “best” baseline model?

As the table and chart show, no model outperforms the others across every metric. Random Forest posts the highest accuracy (0.8487) but one of the lowest recalls (0.0391) at the same time. Bagging has the best precision (1.0) and one of the highest accuracies (0.8480), but its recall (0.0087) means it almost never predicts Oats at all. There is no single best classifier here; different models perform better under different evaluation criteria.

On the one hand, a model with higher precision, making fewer false-positive errors, is more valuable when false positives are costly, such as marking important emails as spam in a spam filter. On the other hand, a model with higher recall, catching more actual positives, is more valuable when missing a positive is the serious mistake, such as fraud detection. Model performance depends heavily on the context: the specific goals and constraints of the task.

Given the imbalance, two of the five stand out as relatively stronger than the rest: Naive Bayes and Boosting. Naive Bayes balances precision and recall (0.29 and 0.23), giving it one of the higher F1-scores here, though both figures sit well below what a balanced dataset would typically allow; Kim and Lee (2023) note that Naive Bayes, like other traditional classifiers, tends to perform poorly on minority classes because of “its sensitivity to class distribution”. Boosting has the best F1-score of the five (0.28) here, with a higher precision than Naive Bayes at a similar recall. AdaBoost has a built-in way to address this: it increases the weight of misclassified instances, usually from the minority class, so the algorithm is “trained to give more attention to the underrepresented class” as training goes on (Reed, 2024); when the imbalance isn’t too severe, this lets Boosting outperform plain AdaBoost’s baseline case (Shahri et al., 2021). The other three models, despite high accuracy, are biased toward the majority class and weak at catching the crop I care about.

Some model families handle imbalance better than others because of how they’re structured (Olamendy, 2024); Naive Bayes also has imbalance-aware variants, such as Complement Naive Bayes (2020), that outperform the plain version on skewed data. Understanding each algorithm’s mechanism, strengths and weaknesses, and the modifications built for imbalanced data, matters for getting good performance out of any of them.

Feature Importance & Selection

Examining each of the models, I determine the most important attributes in predicting Oats versus Other. Three of the models above produce feature-importance scores, so instead of guessing, I pulled the splits and importance rankings straight from the fitted models.

summary(WD.tree)$used

## [1] A21 A24 A25 A02
## 21 Levels: <leaf> A01 A02 A03 A04 A05 A10 A11 A13 A14 A18 A20 A21 A22 ... A30

To determine importance for the Decision Tree, I look at which features it used for splitting. It only ever splits on A21, A24, A25 and A02: these are the most critical attributes for this model, and with only those four it separates the classes well (AUC 0.65). Naive Bayes doesn’t produce a variable importance score at all: it assumes feature independence and uses every attribute equally by construction.

bag_imp = sort(WD.bag$importance, decreasing = TRUE)
boost_imp = sort(WD.boost$importance, decreasing = TRUE)
rf_imp = data.frame(feature = names(WD.train)[1:20], importance = WD.rf$importance[, 1])
rf_imp = rf_imp[order(-rf_imp$importance), ]

par(mfrow = c(1, 3))
barplot(bag_imp, las = 2, main = "Bagging", ylab = "Importance", cex.names = 0.7)
barplot(boost_imp, las = 2, main = "Boosting", ylab = "Importance", cex.names = 0.7)
barplot(setNames(rf_imp$importance, rf_imp$feature), las = 2, main = "Random Forest", ylab = "Importance", cex.names = 0.7)

par(mfrow = c(1, 1))

head(rf_imp, 8)

##     feature importance
## A25     A25   61.69747
## A24     A24   55.16716
## A02     A02   54.60062
## A21     A21   53.57598
## A18     A18   49.91023
## A14     A14   49.57383
## A03     A03   49.14839
## A04     A04   48.39138

From the table, the most critical variables in the Bagging classifier are A25, A23, A04, A02 and A03. For Boosting, they’re A21, A25, A14, A03 and A24. For Random Forest, they’re A25, A24, A02, A21 and A18. A25, A24, A02 and A21 sit near the top across all three ensembles, reassuring since three independently-trained models agree on what matters. At the other end:

tail(sort(WD.bag$importance), 5)

##       A03       A02       A04       A23       A25 
##  6.304343  6.726871  7.287413 14.702196 14.711849

tail(sort(WD.boost$importance), 5)

##      A24      A03      A14      A25      A21 
## 6.523821 6.665209 6.731088 7.377754 8.544807

tail(rf_imp[order(rf_imp$importance), ], 5)

##     feature importance
## A18     A18   49.91023
## A21     A21   53.57598
## A02     A02   54.60062
## A24     A24   55.16716
## A25     A25   61.69747

A20 stands at or near the bottom of the important-variable lists for Bagging, Boosting and Random Forest alike, and it’s also absent from the Decision Tree’s split list above, so it’s safe to omit from the data. A10 and A11 join it in the bottom five for all three ensembles, the strongest evidence that these three add little discriminative value; referring back to the EDA, A11 was one of the lowest-SD columns and A10/A20 were among the most zero-heavy. A05 and A13 turn up in the bottom five for two of the three models, and were flagged in the EDA for the same reason (large numbers of zeros or low SD), so I’d treat them as secondary candidates too. All five look safe to drop from the data with little effect on performance.

A Model Simple Enough to Use by Hand

Starting from the Decision Tree in the Baseline section, I build a classifier simple enough for a person to classify Oats (1) versus Other (0) by hand. Looking at that existing model, it classifies every case as 0. The class balance from the EDA explains why: with Oats such a small share of the data, decision trees tend to favor the majority class, leading to high misclassification for the minority class (GeeksforGeeks, 2024d). So to make this new model classify some cases as class 1, I only include 2,000 of the 4,284 class-0 rows in the training data, together with every class-1 row.

I also simplify the tree further by keeping only the top 5 Random Forest features from the section above. Random Forest is “extremely useful and efficient in selecting the important features” (Chen et al., 2020): it provides a built-in method for evaluating attribute importance, and the ranking is reliable, since this type of model handles high dimensionality and captures complex relationships between variables well (GeeksforGeeks, 2024c).

top5 = rf_imp$feature[1:5]
top5

## [1] "A25" "A24" "A02" "A21" "A18"

set.seed(SEED)
WD.fs.zero = WD[WD$Class == 0, ]
WD.fs = rbind(WD.fs.zero[sample(nrow(WD.fs.zero), 2000, replace = FALSE), ],
              WD[WD$Class == 1, ])

set.seed(SEED)
train.row.hand = sample(1:nrow(WD.fs), 0.7 * nrow(WD.fs))
WD.fs.train = WD.fs[train.row.hand, ]
WD.fs.test = WD.fs[-train.row.hand, ]

hand_formula = as.formula(paste("Class ~", paste(top5, collapse = " + ")))
set.seed(SEED)
WD.tree.hand = tree(hand_formula, data = WD.fs.train)
WD.tree.hand

## node), split, n, deviance, yval, (yprob)
##       * denotes terminal node
## 
##  1) root 1901 2166.00 0 ( 0.74329 0.25671 )  
##    2) A24 < 3.2555 465  300.10 0 ( 0.90108 0.09892 ) *
##    3) A24 > 3.2555 1436 1773.00 0 ( 0.69220 0.30780 )  
##      6) A18 < 0.8565 250  187.40 0 ( 0.87600 0.12400 ) *
##      7) A18 > 0.8565 1186 1531.00 0 ( 0.65346 0.34654 )  
##       14) A25 < 0.5545 155  137.00 0 ( 0.83871 0.16129 ) *
##       15) A25 > 0.5545 1031 1364.00 0 ( 0.62561 0.37439 )  
##         30) A02 < -9.876 927 1246.00 0 ( 0.60194 0.39806 ) *
##         31) A02 > -9.876 104   92.64 0 ( 0.83654 0.16346 ) *

plot(WD.tree.hand); text(WD.tree.hand, pretty = 0)

WD.predtree.hand = predict(WD.tree.hand, WD.fs.test, type = "class")
t6 = table(Actual_Class = WD.fs.test$Class, Predicted_Class = WD.predtree.hand)
t6

##             Predicted_Class
## Actual_Class   0   1
##            0 587   0
##            1 228   0

hand_metrics = performance_metrics(t6["1","1"], t6["1","0"], t6["0","1"], t6["0","0"])
hand_pred_vec = predict(WD.tree.hand, WD.fs.test, type = "vector")
hand_auc = as.numeric(performance(prediction(hand_pred_vec[, 2], WD.fs.test$Class), "auc")@y.values)
c(hand_metrics, AUC = round(hand_auc, 4))

##    recall precision  accuracy  f1_score      bacc       AUC 
##    0.0000       NaN    0.7202    0.0000    0.5000    0.6674

A person using this model to classify Oats and Other crops by hand would follow the tree’s actual splits: check A24, then A18, then A25, then A02, thresholds simple enough to write on an index card. But every single leaf in this fitted tree predicts “Other”, so it never flags a single Oats farm on the held-out test set (recall = 0), even after downsampling the majority class and hand-picking the strongest features. Its AUC (0.667) edges past the full-feature baseline Decision Tree (0.654), so the ranking underneath is sound; no split is confident enough to flip a prediction to the minority class. That gap between ranking well and classifying well is the imbalance problem the next two sections take on.

Tackling Class Imbalance Properly

One weakness of the models in the Baseline section is that they tend to be biased toward the majority class (“Other”), visible as low recall paired with high precision, and so a low F1-score. From here, I want to focus on lifting F1-score and recall, and reducing that bias.

Given the baseline metrics, I consider Random Forest and Boosting the strongest candidates to build on. I choose Random Forest for this section: both models offer a path to the improvement I want, but Random Forest looks like the safer classifier to tune, thanks to the overfitting resistance it inherits from bagging, simpler hyperparameter tuning, and lower sensitivity to noisy data (GeeksforGeeks, 2024a).

As the EDA showed, the dataset is highly imbalanced, leading to high misclassification rates for the minority class (Oats). According to Chen and Liaw (2004), there are two effective ways to handle this with Random Forest: Balanced Random Forest (BRF) and Weighted Random Forest (WRF). There’s no clear evidence that one is superior, but WRF looks more vulnerable to noise than BRF, since it weights the minority class rather than resampling it, and it’s less computationally efficient on large datasets. I chose to work on BRF here.

In building this model, I decided to keep every one of the 20 available features rather than removing any. BRF performs downsampling on its own, reducing the number of observations in the training set; removing features on top of that would cut the amount of information available to the model further and raise the risk of underfitting.

According to Thölke et al. (2023), in an imbalanced dataset the Accuracy metric shows “misleadingly high performances”, and Balanced Accuracy (BAcc) gives “more reliable performance evaluations”. A high F1-score, meanwhile, shows a model achieving high precision and high recall at the same time. So I focus mainly on BAcc and F1-score when judging this model.

The idea behind BRF is that I sample the same number of observations from class 0 and class 1 for each tree; this should improve the model’s ability to learn from, and correctly classify, the minority class (GeeksforGeeks, 2024b). According to Fox et al. (2017), a BRF can be built using the sampsize argument of randomForest().

set.seed(SEED)
WD.brf = randomForest(
  Class ~ ., data = WD.train,
  sampsize = c("0" = nrow(WD.train[WD.train$Class == 1, ]),
               "1" = nrow(WD.train[WD.train$Class == 1, ]))
)
WD.predbrf = predict(WD.brf, WD.test)
t7 = table(Actual_Class = WD.test$Class, Predicted_Class = WD.predbrf)
t7

##             Predicted_Class
## Actual_Class    0    1
##            0 1028  242
##            1   95  135

WD.pred.brf = predict(WD.brf, WD.test, type = "prob")
brf.pred = prediction(WD.pred.brf[, 2], WD.test$Class)
brf_auc = as.numeric(performance(brf.pred, "auc")@y.values)
brf_metrics = performance_metrics(t7["1","1"], t7["1","0"], t7["0","1"], t7["0","0"])
c(brf_metrics, AUC = round(brf_auc, 4))

##    recall precision  accuracy  f1_score      bacc       AUC 
##    0.5870    0.3581    0.7753    0.4448    0.6982    0.7414

Looking at BAcc, F1-score, Recall and AUC, this Balanced Random Forest has a superior performance compared to every classifier in the Baseline section. It ranks first on BAcc (0.6982), F1-score (0.4448) and Recall (0.5870), each well ahead of the rest, up from Random Forest’s own 0.5172, 0.0735 and 0.0391 with no feature changed at all. It also edges out plain Random Forest on AUC, 0.7414 versus 0.7392, the best of any model built so far. Precision and Accuracy are worse than some of the baseline models, but as noted above, Accuracy is not a reliable metric under this kind of imbalance, and a lower precision paired with a higher recall is the expected outcome, and the target, of a model built to find the minority class. Accuracy is well known to be a misleadingly optimistic metric under imbalance, and that’s the trade this model makes on purpose. Therefore, this Balanced Random Forest is the strongest model built so far.

A Deep Learning Approach

Setting up the Artificial Neural Network classifier (Keras/TensorFlow) calls for a bit more preprocessing than the models above: recoding the output Class as numeric, and normalising the inputs with min-max scaling to [0, 1]. There’s no missing-value handling needed, since the dataset has none. For the class_weight parameter, I set the value for class 0 to 1 and class 1 to 6, so the model pays six times more attention to Oats errors than Other errors, rather than addressing the imbalance by resampling.

min_max_scaling = function(col) (col - min(col)) / (max(col) - min(col))

run_ann = function(features, input_dim, seed = SEED) {
  x.train = as.matrix(sapply(WD.train[, features], min_max_scaling))
  x.test  = as.matrix(sapply(WD.test[, features], min_max_scaling))
  y.train = ifelse(WD.train$Class == "0", 0, 1)
  y.test  = ifelse(WD.test$Class == "0", 0, 1)

  tensorflow::set_random_seed(seed)
  model = keras_model_sequential()
  model %>%
    layer_dense(units = 256, activation = "relu", input_shape = c(input_dim)) %>%
    layer_dropout(rate = 0.4) %>%
    layer_dense(units = 128, activation = "relu") %>%
    layer_dropout(rate = 0.3) %>%
    layer_dense(units = 1, activation = "sigmoid")
  model %>% compile(loss = "binary_crossentropy", optimizer = "adam", metrics = c("accuracy"))
  invisible(model %>% fit(x.train, y.train, epochs = 100, batch_size = 16,
                           validation_split = 0.3, verbose = 0,
                           class_weight = list("0" = 1, "1" = 6)))

  pred = model %>% predict(x.test, verbose = 0)
  tab = table(Actual = y.test, Predicted = ifelse(pred >= 0.5, 1, 0))
  auc = as.numeric(performance(prediction(pred, y.test), "auc")@y.values)
  list(table = tab, auc = auc)
}

Regarding feature selection for this model, I decided to compare two feature sets: the top 10 most important variables ranked by the Random Forest model above, and the full set of 20 features. After training both, I compare their performance and settle on a final choice.

top10 = rf_imp$feature[1:10]

ann_top10 = run_ann(top10, input_dim = 10)
ann_top10$table

##       Predicted
## Actual   0   1
##      0 884 386
##      1 109 121

ann_top10_metrics = performance_metrics(ann_top10$table["1","1"], ann_top10$table["1","0"],
                                         ann_top10$table["0","1"], ann_top10$table["0","0"])
c(ann_top10_metrics, AUC = round(ann_top10$auc, 4))

##    recall precision  accuracy  f1_score      bacc       AUC 
##    0.5261    0.2387    0.6700    0.3284    0.6111    0.6633

ann_full = run_ann(names(WD.train)[1:20], input_dim = 20)
ann_full$table

##       Predicted
## Actual   0   1
##      0 961 309
##      1 109 121

ann_full_metrics = performance_metrics(ann_full$table["1","1"], ann_full$table["1","0"],
                                        ann_full$table["0","1"], ann_full$table["0","0"])
c(ann_full_metrics, AUC = round(ann_full$auc, 4))

##    recall precision  accuracy  f1_score      bacc       AUC 
##    0.5261    0.2814    0.7213    0.3667    0.6414    0.6714

ann_df = data.frame(rbind(
  cbind(model = "Top 10 features", t(ann_top10_metrics), AUC = ann_top10$auc),
  cbind(model = "Full 20 features", t(ann_full_metrics), AUC = ann_full$auc)
))
ann_long = melt(ann_df, id.vars = "model", variable.name = "metric", value.name = "value")
ann_long$value = as.numeric(ann_long$value)
ggplot(ann_long, aes(x = metric, y = value, fill = model)) +
  geom_bar(position = "dodge", stat = "identity") +
  labs(title = "ANN: top-10 vs. full feature set", y = NULL, x = NULL)

As inferred from the tables and chart above, the full-feature model matches the top-10 model on recall and leads it on every other metric here: accuracy 0.72 versus 0.67, precision 0.28 versus 0.24, F1-score 0.37 versus 0.33, BAcc 0.64 versus 0.61, and AUC 0.67 versus 0.66. With 256 and 128-unit dense layers plus dropout, this network has enough capacity to pull a useful signal out of the extra features rather than getting confused by them, so I take the full-feature model as the final one here.

This is the reverse of what I found the first time I worked through this analysis, when the top-10 version came out ahead (see the Reproducibility Notes at the end for why). It’s a useful finding on its own: feature-selection wins from tree ensembles don’t guarantee anything for a neural network with enough capacity to use the extra inputs, so each model family is worth testing on its own terms rather than assumed.

Pushing Further with Gradient Boosting

For this last model, I selected XGBoost. Extreme Gradient Boosting, or XGBoost, is an ensemble learning algorithm: like AdaBoost, the idea is that it uses decision trees as weak learners and builds them in a sequence, each one correcting the errors of the one before it. The main difference between AdaBoost, used in the Baseline section, and XGBoost is in how each new tree learns from those errors: AdaBoost focuses on reweighting samples, whereas XGBoost uses gradient descent to iteratively fit new weak learners that correct the residual errors of the previous ones. It also has a scale_pos_weight parameter, a lever built for imbalanced classification.

I implemented this model using the package xgboost (https://cran.r-project.org/web/packages/xgboost/index.html). Initially, I trained it with the full set of features, since I want to investigate the importance ranking the package provides.

WD.xgboost = WD
WD.xgboost$Class = ifelse(WD.xgboost$Class == "0", 0, 1)
WD.xgboost.train = WD.xgboost[train.row, ]
WD.xgboost.test = WD.xgboost[-train.row, ]
pos_weight = nrow(WD.xgboost.train[WD.xgboost.train$Class == 0, ]) /
             nrow(WD.xgboost.train[WD.xgboost.train$Class == 1, ])

set.seed(SEED)
WD.xgb.full = xgboost(data = data.matrix(WD.xgboost.train[, 1:20]),
                       objective = "binary:logistic", eval_metric = "auc", nrounds = 25,
                       label = WD.xgboost.train$Class, max_depth = 15,
                       scale_pos_weight = pos_weight, verbose = 0)

importance_matrix = xgb.importance(colnames(WD.xgboost.train[1:20]), model = WD.xgb.full)
xgb.plot.importance(importance_matrix, main = "XGBoost feature importance (Gain)")

To find the best set of features, I built 20 versions of XGBoost, each using an incremental set of top-ranked features, starting from the top 1, up to the top 20, based on the importance ranking above.

xgb.recall = c(); xgb.auc = c(); xgb.accuracy = c(); xgb.bacc = c(); xgb.f1 = c()
for (i in 1:20) {
  set.seed(SEED)
  m = xgboost(data = data.matrix(WD.xgboost.train[, importance_matrix[1:i, ]$Feature]),
              objective = "binary:logistic", eval_metric = "auc", nrounds = 25,
              label = WD.xgboost.train$Class, max_depth = 15,
              scale_pos_weight = pos_weight, verbose = 0)
  p = predict(m, data.matrix(WD.xgboost.test[, importance_matrix[1:i, ]$Feature]))
  tt = table(Actual = WD.xgboost.test$Class, Predicted = ifelse(p >= 0.5, 1, 0))
  mm = performance_metrics(tt["1","1"], tt["1","0"], tt["0","1"], tt["0","0"])
  a = as.numeric(performance(prediction(p, WD.xgboost.test$Class), "auc")@y.values)
  xgb.recall = c(xgb.recall, mm["recall"]); xgb.auc = c(xgb.auc, a)
  xgb.accuracy = c(xgb.accuracy, mm["accuracy"]); xgb.bacc = c(xgb.bacc, mm["bacc"])
  xgb.f1 = c(xgb.f1, mm["f1_score"])
}

plot(1:20, xgb.recall, type = "l", col = "steelblue", ylim = c(0, 1), lwd = 2,
     xlab = "Top k features used", ylab = "Metric",
     main = "XGBoost performance vs. number of features")
lines(1:20, xgb.auc, col = "orange", lwd = 2)
lines(1:20, xgb.accuracy, col = "brown", lwd = 2)
lines(1:20, xgb.bacc, col = "forestgreen", lwd = 2)
lines(1:20, xgb.f1, col = "purple", lwd = 2)
abline(v = which.max(xgb.f1), lty = 2, col = "grey40")
legend("bottomright", legend = c("Recall", "AUC", "Accuracy", "BAcc", "F1-score"),
       col = c("steelblue", "orange", "brown", "forestgreen", "purple"), lwd = 2)

best_k = which.max(xgb.f1)
best_k

## f1_score 
##        9

set.seed(SEED)
WD.xgb.best = xgboost(data = data.matrix(WD.xgboost.train[, importance_matrix[1:best_k, ]$Feature]),
                       objective = "binary:logistic", eval_metric = "auc", nrounds = 25,
                       label = WD.xgboost.train$Class, max_depth = 15,
                       scale_pos_weight = pos_weight, verbose = 0)
p.best = predict(WD.xgb.best, data.matrix(WD.xgboost.test[, importance_matrix[1:best_k, ]$Feature]))
t11 = table(Actual_Class = WD.xgboost.test$Class, Predicted_Class = ifelse(p.best >= 0.5, 1, 0))
t11

##             Predicted_Class
## Actual_Class    0    1
##            0 1191   79
##            1  161   69

xgb_best_metrics = performance_metrics(t11["1","1"], t11["1","0"], t11["0","1"], t11["0","0"])
xgb_best_auc = as.numeric(performance(prediction(p.best, WD.xgboost.test$Class), "auc")@y.values)
c(xgb_best_metrics, AUC = round(xgb_best_auc, 4))

##    recall precision  accuracy  f1_score      bacc       AUC 
##    0.3000    0.4662    0.8400    0.3651    0.6189    0.6931

As inferred from the graph, the model with the top 9 features has one of the best performances: it ranks first in F1-score and BAcc, and is competitive on Accuracy and Recall too. I chose this as the final model, cutting the feature set by more than half with no loss in the model’s ability to find Oats farms, and getting a model that trains faster and is easier to explain as a bonus.

This XGBoost model doesn’t top any single metric on the Final Comparison table below, but it never bottoms out either: third-to-fifth place across recall, precision, accuracy, F1-score, BAcc and AUC, ahead of Naive Bayes, the Balanced Random Forest and the Neural Network on precision. It’s the strongest choice when I want to limit false positives without giving up too much on the metrics that matter most for an imbalanced problem like this one.

Final Comparison & Takeaways

The table below puts every technique side by side, using each model’s best configuration from above: the full-feature Neural Network and the 9-feature XGBoost.

final_metrics = rbind(
  "Decision Tree" = c(performance_metrics(t1["1","1"], t1["1","0"], t1["0","1"], t1["0","0"]), AUC = round(auc.tree, 4)),
  "Naive Bayes"   = c(performance_metrics(t2["1","1"], t2["1","0"], t2["0","1"], t2["0","0"]), AUC = round(auc.bayes, 4)),
  "Bagging"       = c(performance_metrics(t3["1","1"], t3["1","0"], t3["0","1"], t3["0","0"]), AUC = round(auc.bag, 4)),
  "Boosting"      = c(performance_metrics(t4["1","1"], t4["1","0"], t4["0","1"], t4["0","0"]), AUC = round(auc.boost, 4)),
  "Random Forest" = c(performance_metrics(t5["1","1"], t5["1","0"], t5["0","1"], t5["0","0"]), AUC = round(auc.rf, 4)),
  "Balanced RF"   = c(brf_metrics, AUC = round(brf_auc, 4)),
  "Neural Net"    = c(ann_full_metrics, AUC = round(ann_full$auc, 4)),
  "XGBoost"       = c(xgb_best_metrics, AUC = round(xgb_best_auc, 4))
)
knitr::kable(final_metrics, caption = "All eight classifiers, side by side")

All eight classifiers, side by side
	recall	precision	accuracy	f1_score	bacc	AUC
Decision Tree	0.0000	NaN	0.8467	0.0000	0.5000	0.6544
Naive Bayes	0.2261	0.2873	0.7953	0.2530	0.5623	0.6779
Bagging	0.0087	1.0000	0.8480	0.0172	0.5043	0.7136
Boosting	0.1957	0.5000	0.8467	0.2812	0.5801	0.7021
Random Forest	0.0391	0.6000	0.8487	0.0735	0.5172	0.7392
Balanced RF	0.5870	0.3581	0.7753	0.4448	0.6982	0.7414
Neural Net	0.5261	0.2814	0.7213	0.3667	0.6414	0.6714
XGBoost	0.3000	0.4662	0.8400	0.3651	0.6189	0.6931

fm = as.data.frame(final_metrics); fm$model = rownames(fm)
fm_long = melt(fm, id.vars = "model", variable.name = "metric", value.name = "value")
ggplot(fm_long, aes(x = metric, y = value, fill = model)) +
  geom_bar(position = "dodge", stat = "identity") +
  labs(title = "Every classifier, every metric", y = NULL, x = NULL)

Three findings stand out:

Accuracy is not a reliable scoreboard here. Every model scores between 72% and 85% accuracy, yet recall, the share of Oats farms caught, ranges from 0% to 59%. Pick evaluation metrics that match the real cost of each error type.
Engineering for the imbalance wins outright. The Balanced Random Forest tops every one of the metrics that matter most for this problem: best F1-score (0.44), best recall (0.59), best balanced accuracy (0.70), and even the best AUC (0.74) of all eight models, ahead of its own non-balanced Random Forest counterpart on every one of those measures, at the cost of some accuracy and precision.
No single model wins across the board. Bagging still holds the best pure precision (1.0, on the rare occasions it predicts Oats at all), and XGBoost is the steadiest all-rounder, never finishing best or worst on any metric. The right choice depends on whether false positives or missed detections cost more in the deployment context.

Skills demonstrated

Exploratory data analysis on real, high-dimensional sensor data: identifying near-zero-variance and zero-heavy features, quantifying class imbalance, and checking multicollinearity
Implementing and correctly evaluating 8 classifier families: Decision Tree, Naive Bayes, Bagging, AdaBoost, Random Forest, Balanced Random Forest, a Keras/TensorFlow Neural Network, and XGBoost
Rigorous, imbalance-aware evaluation methodology: confusion matrices, precision/recall/F1/balanced accuracy, and ROC/AUC, applied across every model instead of relying on accuracy alone
Data-driven feature selection: cross-referencing importance scores from three independent ensemble methods, and a systematic 20-model sweep to find the optimal feature count for XGBoost
Class-imbalance techniques across three mechanisms: stratified downsampling (the by-hand tree), balanced bootstrap sampling (Balanced Random Forest), and class-weighted loss (Neural Network)
Communicating trade-offs: translating precision/recall/F1 differences into concrete guidance on when each model is the right choice

Reproducibility notes

A few things changed relative to how I first carried out this analysis for coursework:

I added an explicit set.seed(SEED) immediately before every stochastic model call: Bagging, Boosting, Random Forest, the by-hand tree, Balanced Random Forest, the Neural Network, and every XGBoost fit. This makes the notebook reproducible end-to-end, but individual figures can differ from the original coursework write-up, where I fit these models in an interactive session without a seed reset before each call. Decision Tree and Naive Bayes are fully deterministic given the fixed data split, and match the original figures. XGBoost and Balanced Random Forest, run as the first random draw after their own seed reset, also matched. The largest shifts landed in Bagging, Boosting and Random Forest, which depend on the exact prior sequence of random draws, and in the Neural Network, which is also sensitive to weight initialisation and dropout.
In this run, the full-feature Neural Network outperformed the top-10-feature version on every metric, the reverse of the original coursework conclusion. I’m disclosing this rather than smoothing it over: it shows that feature-selection benefits from tree ensembles don’t transfer to a dense neural network with enough capacity.
I now build chart data, including the multi-model comparison plots, from the metrics computed in-session, instead of round-tripping through hand-edited intermediate CSV files as in the original coursework workflow.
I left a few exploratory dead ends from the original working script out of this notebook: an SVM variant, a plain neuralnet-package model, and an early-stopping Keras variant. None of them made it into the final reported comparison.

References

Chen, C., & Liaw, A. (2004). Using Random Forest to Learn Imbalanced Data. https://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf

Chen, R.-C., Dewi, C., Huang, S.-W., & Caraka, R. E. (2020). Selecting critical features for data classification based on machine learning methods. Journal of Big Data, 7(1). https://doi.org/10.1186/s40537-020-00327-4

Complement Naive Bayes (CNB) Algorithm. (2020, July 23). GeeksforGeeks. https://www.geeksforgeeks.org/complement-naive-bayes-cnb-algorithm/

Fox, E. W., Hill, R. A., Leibowitz, S. G., Olsen, A. R., Thornbrugh, D. J., & Weber, M. H. (2017). Assessing the accuracy and stability of variable selection methods for random forest modeling in ecology. Environmental Monitoring and Assessment, 189(7). https://doi.org/10.1007/s10661-017-6025-0

GeeksforGeeks. (2024a, March 6). Gradient Boosting vs Random Forest. GeeksforGeeks. https://www.geeksforgeeks.org/gradient-boosting-vs-random-forest/

GeeksforGeeks. (2024b, March 11). Bagging and Random Forest for Imbalanced Classification. GeeksforGeeks. https://www.geeksforgeeks.org/bagging-and-random-forest-for-imbalanced-classification/

GeeksforGeeks. (2024c, May 28). Feature Selection Using Random Forest. GeeksforGeeks. https://www.geeksforgeeks.org/feature-selection-using-random-forest/

GeeksforGeeks. (2024d, November 25). Training a decision tree against unbalanced data. GeeksforGeeks. https://www.geeksforgeeks.org/training-a-decision-tree-against-unbalanced-data/

Kim, T., & Lee, J.-S. (2023). Maximizing AUC to learn weighted naive Bayes for imbalanced data classification. Expert Systems with Applications, 217, 119564. https://doi.org/10.1016/j.eswa.2023.119564

Olamendy, J. C. (2024, March 8). Tackling the Challenge of Imbalanced Datasets: A Comprehensive Guide. Medium. https://medium.com/@juanc.olamendy/tackling-the-challenge-of-imbalanced-datasets-a-comprehensive-guide-2feb11ca2fa0

Reed, V. (2024, October 8). AdaBoost: Solving Class Imbalance In Datasets Effectively. AICompetence. https://aicompetence.org/adaboost-solving-class-imbalance-in-datasets/

Shahri, N. H. N. B. M., Lai, S. B. S., Mohamad, M. B., Rahman, H. A. B. A., & Rambli, A. B. (2021). Comparing the Performance of AdaBoost, XGBoost, and Logistic Regression for Imbalanced Data. Mathematics and Statistics, 9(3), 379-385. https://doi.org/10.13189/ms.2021.090320

Thölke, P., Mantilla-Ramos, Y.-J., Abdelhedi, H., Maschke, C., Dehgan, A., Harel, Y., Kemtur, A., Mekki Berrada, L., Sahraoui, M., Young, T., Bellemare Pépin, A., El Khantour, C., Landry, M., Pascarella, A., Hadid, V., Combrisson, E., O’Byrne, J., & Jerbi, K. (2023). Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data. NeuroImage, 277, 120253. https://doi.org/10.1016/j.neuroimage.2023.120253