Building a Robust Ensemble Model for Spam Detection

Ensemble methods are powerful techniques in machine learning that combine multiple models to achieve better predictive performance than any single model alone. This challenge will guide you through building and evaluating a simple ensemble for a common classification task: spam detection. You will learn to implement two fundamental ensemble techniques: Bagging and Boosting.

Problem Description

Your task is to implement two distinct ensemble methods, Bagging and Boosting, for a binary classification problem. You will then combine the predictions of these ensembles using a voting mechanism. The ultimate goal is to build a more accurate and robust spam detection system by leveraging the collective intelligence of multiple base models.

Key Requirements:

Implement Bagging: Create a Bagging ensemble classifier.
- Bagging involves training multiple instances of a base estimator (e.g., a Decision Tree) on different bootstrap samples of the training data.
- Predictions from individual base estimators should be aggregated using majority voting.
Implement Boosting: Create a Boosting ensemble classifier.
- Boosting sequentially trains base estimators, where each new estimator focuses on correcting the errors of the previous ones.
- You will implement AdaBoost (Adaptive Boosting) as your boosting algorithm.
- Weights of misclassified samples should be updated, and base estimators should be weighted based on their accuracy.
Combine Ensembles: Create a final ensemble that combines the predictions from your Bagging and Boosting classifiers.
- This final ensemble should also use majority voting to determine the final prediction.
Evaluate Performance: Measure the accuracy of each implemented classifier (Bagging, Boosting, and the combined ensemble) on a given test set.

Expected Behavior:

Your code should accept training and testing data (features and labels) as input.
The implemented classifiers should return predicted labels for the test data.
The accuracy metric should be calculated correctly.
The final output should clearly show the accuracy of each ensemble and the combined ensemble.

Edge Cases to Consider:

Imbalanced Datasets: While not explicitly tested in the examples, real-world spam detection can have imbalanced classes. Consider how your algorithms might handle this (though explicit handling isn't required for this challenge).
Small Datasets: How might the performance of bagging and boosting change with very few training samples?

Examples

Example 1: Simple Classification Scenario

Let's imagine a simplified scenario with a small dataset.

Input Data (Conceptual):

Training Data (X_train, y_train):
- X_train: [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8], [0.9, 1.0], [0.2, 0.3], [0.4, 0.5], [0.6, 0.7], [0.8, 0.9], [1.0, 1.1]]
- y_train: [0, 0, 0, 1, 1, 0, 0, 1, 1, 1] (0 for not spam, 1 for spam)
Test Data (X_test):
- X_test: [[0.15, 0.25], [0.85, 0.95]]

Expected Output (Illustrative - actual values depend on implementation details like random seeds):

Bagging Accuracy: 0.75
Boosting Accuracy: 0.80
Combined Ensemble Accuracy: 0.85

Explanation:

The Bagging classifier, trained on bootstrap samples, will make predictions. The Boosting classifier will sequentially learn. The Combined Ensemble will take the majority vote of the Bagging and Boosting predictions. The accuracies reflect how well each method performs on the test data.

Example 2: Using Scikit-learn for Inspiration (Conceptual)

This example demonstrates how you might conceptually approach this using scikit-learn components, though you are expected to implement the core logic yourself.

Input Data (Conceptual - assume load_iris dataset split into train/test):

Training Data: X_train_iris, y_train_iris
Test Data: X_test_iris, y_test_iris

Conceptual Code Snippet (Not the solution, but shows intent):

# From sklearn.ensemble import BaggingClassifier, AdaBoostClassifier
# From sklearn.tree import DecisionTreeClassifier
# From sklearn.model_selection import train_test_split
# From sklearn.metrics import accuracy_score

# --- Bagging ---
# base_estimator = DecisionTreeClassifier(max_depth=3)
# bagging_model = BaggingClassifier(base_estimator=base_estimator, n_estimators=10, random_state=42)
# bagging_model.fit(X_train_iris, y_train_iris)
# y_pred_bagging = bagging_model.predict(X_test_iris)
# accuracy_bagging = accuracy_score(y_test_iris, y_pred_bagging)

# --- Boosting (AdaBoost) ---
# base_estimator_boost = DecisionTreeClassifier(max_depth=1) # Often weak learners for AdaBoost
# boosting_model = AdaBoostClassifier(base_estimator=base_estimator_boost, n_estimators=10, learning_rate=1.0, random_state=42)
# boosting_model.fit(X_train_iris, y_train_iris)
# y_pred_boosting = boosting_model.predict(X_test_iris)
# accuracy_boosting = accuracy_score(y_test_iris, y_pred_boosting)

# --- Combined Ensemble (Voting) ---
# from sklearn.ensemble import VotingClassifier
# final_ensemble = VotingClassifier(estimators=[('bag', bagging_model), ('boost', boosting_model)], voting='soft') # or 'hard'
# final_ensemble.fit(X_train_iris, y_train_iris)
# y_pred_combined = final_ensemble.predict(X_test_iris)
# accuracy_combined = accuracy_score(y_test_iris, y_pred_combined)

# print(f"Bagging Accuracy: {accuracy_bagging}")
# print(f"Boosting Accuracy: {accuracy_boosting}")
# print(f"Combined Ensemble Accuracy: {accuracy_combined}")

Constraints

The base estimator for both Bagging and Boosting should be a Decision Tree.
For Bagging, you should create 10 bootstrap samples and train 10 Decision Trees.
For Boosting (AdaBoost), you should train 10 Decision Trees sequentially.
The maximum depth of individual Decision Trees used as base estimators should be limited (e.g., max_depth=3 for Bagging, max_depth=1 for Boosting).
Your implementation should not rely on pre-built BaggingClassifier or AdaBoostClassifier from libraries like scikit-learn. You need to implement the core logic of these algorithms.
The final combined ensemble should use majority voting (hard voting).
The input data will be provided as NumPy arrays.
The accuracy calculation should be standard (number of correct predictions / total predictions).
Performance: Your code should execute within a reasonable time for typical input sizes (e.g., up to a few thousand samples, a few dozen features).

Notes

Bagging: For each base estimator, you will need to create a bootstrap sample. A bootstrap sample is a sample with replacement from the original training data, of the same size as the original dataset.
Boosting (AdaBoost):
- Initialize sample weights equally.
- In each iteration:
  - Train a weak learner on the weighted data.
  - Calculate the error rate of the weak learner.
  - Calculate the weight for this weak learner based on its error rate.
  - Update the sample weights: increase weights for misclassified samples and decrease for correctly classified ones.
- When making predictions, use the weighted sum of the base learner predictions.
Decision Tree Implementation: You can use a simplified Decision Tree implementation or assume one is available that can be trained on weighted data for Boosting. For this challenge, you can focus on the ensemble logic. If you need to implement a Decision Tree from scratch, that's a significant undertaking; assume you have a functional DecisionTree class that can be trained and predict.
Voting: For hard voting, the final prediction is the class that receives the most votes from the individual classifiers.
Randomness: When creating bootstrap samples or training Decision Trees, random choices are made. For reproducibility, you might consider setting random seeds if your underlying Decision Tree implementation allows it. However, the core focus is on the ensemble logic itself.
Final Output: Ensure your printed output clearly labels the accuracy for each of the three components (Bagging, Boosting, Combined Ensemble).