Case Study · SENAI SP Training (Unit 390)

Data Science & Machine Learning

Hands-on 6-week journey through Data Science, Feature Engineering, Predictive Modeling, and Web Scraping. Progressing from descriptive statistics and data visualization using Pandas, Seaborn, and Matplotlib to automated web crawling with Scrapy and Linear Regression model training using Scikit-Learn.

Python 3.x Pandas & NumPy Scikit-Learn Seaborn & Matplotlib Scrapy Framework Streamlit

⚡ The Challenge: From Raw Data to Predictive Insight

In real-world data engineering scenarios, raw datasets arrive unstructured, containing missing values, truncated fields, or complex JSON strings nested inside spreadsheet cells.

The core objective of this project was to establish an end-to-end data pipeline: from web crawling with Scrapy, data sanitization and relational merging in Pandas, statistical exploration using Seaborn, to training and evaluating Machine Learning regression models ($MSE, RMSE, R^2$).

🗺️ Data Pipeline Architecture

🕸️ 1. Collection Scrapy & APIs (TMDB, Quotes, E-commerce)

🧹 2. Ingestion & Cleaning Pandas DataFrames, Imputation & JSON Parsing

📈 3. EDA & Statistics Boxplots, Histograms, Correlation Matrix

🤖 4. ML Modeling Scikit-Learn (LinearRegression, Train/Test Split)

🚀 5. Delivery Streamlit Dashboards & Validation Reports

🔬 Developed Projects & Modules

Final Capstone Challenge

MovieScope — Predictive Analysis on TMDB 5000 Dataset

Code on GitHub

End-to-end data intelligence application analyzing 5,000 movies and their credits. Merges tmdb_5000_movies.csv and tmdb_5000_credits.csv via Pandas relational merge. Deserializes embedded JSON strings to extract main genres and directors, handles skewness, and trains a Linear Regression model with Scikit-Learn.

Integrated Dataset 5,000 Movies & Credits

Scikit-Learn Model LinearRegression

Explained Variance ($R^2$) R² = 0.33

Pandas Merge JSON Deserialization Scikit-Learn Train/Test Outlier Analysis Seaborn Charts

Physicochemical Analysis: Wine Quality

Chemical properties analysis of red and white wines. Generated correlation heatmaps comparing volatile acidity, pH, sulfates, and alcohol content against sensory quality scores.

Seaborn Heatmaps Pandas Correlation

Distributed Web Scraping (Scrapy)

Asynchronous Scrapy spiders built to extract structured web data, handling automated pagination, selectors, and exporting structured JSON/JSONL datasets.

Scrapy Spiders CSS/XPath Selectors