Back to projects

Spam Email Detector

A machine learning system built from scratch to detect spam emails using Support Vector Machines (SVM) and a robust data processing pipeline.

PythonScikit-learnJoblibPandasFlaskMySQLReactRegex
Source Code

# Spam Email Detector


An end-to-end machine learning solution designed to accurately classify emails as spam or legitimate (ham) using Support Vector Machines.


Project Overview


This project focuses on building a high-performance spam detection system from the ground up. Unlike simple tutorials, this system handles the entire lifecycle of a machine learning project: from aggregating multiple raw datasets (totaling over 94,000 emails) into a MySQL database to deploying a Flask API for real-time predictions.


Technologies Used


  • Language: Python
  • ML Libraries: Scikit-learn, Joblib, Pandas
  • Backend API: Flask
  • Database: MySQL
  • Frontend: React (Client Interface)
  • Tools: Regex, PyCharm, DataGrip

    • Key Features


    1. **Massive Dataset Integration**: Merged and balanced four distinct datasets to create a corpus of 94,000 emails.

    2. **Custom Feature Engineering**: Extracted specific metadata features such as symbol ratios, link counts, and keyword presence (e.g., "Win", "Buy Now").

    3. **Database-Driven Pipeline**: Utilized MySQL tables for structured storage of raw data, cleaned features, and model specifications instead of flat files.

    4. **Real-time API**: A Flask backend that serves the trained model, allowing for instant classification of user input.


    Project Details


    The core of this project is its rigorous data pipeline. It implements a custom preprocessing engine that sanitizes raw text, removing HTML tags and noise, before feeding it into a Support Vector Machine (SVM) model tuned with various kernels (Linear, RBF, Poly).


    ### Challenges


  • Data Quality: Cleaning varied formats of raw text data from multiple sources to ensure consistent feature extraction.
  • Class Imbalance: Ensuring the model doesn't become biased towards the more frequent "Ham" emails by carefully balancing the training sets.
  • Model Selection: Determining the optimal kernel and hyperparameters for the SVM to maximize precision and recall.

    • ### Solutions


  • Unified SQL Storage: Created a central `finalDatasetTables` in MySQL to manage the merged data, ensuring integrity and easy splitting for training/validation.
  • Regex Preprocessing: Developed a robust set of regular expressions to identify and quantify suspicious patterns like excessive capitalization or currency symbols.
  • Iterative Tuning: Implemented an automated testing loop to evaluate different model configurations and store performance metrics in the database.

    • Conclusion


    The Spam Email Detector showcases the application of classical machine learning algorithms to solve real-world cybersecurity problems, emphasizing the importance of clean data and structured engineering pipelines.