Chicago Vanguard
A large-scale crime pattern analysis project using Apache Spark, distributed analytics, Spark SQL, and KMeans clustering.
Project overview
Chicago Vanguard analyzes large-scale public crime data to reveal spatial, temporal, and behavioral patterns across Chicago. The project transforms millions of raw records into structured insights using Apache Spark.
The workflow combines scalable preprocessing, RDD transformations, Spark SQL queries, feature engineering, and unsupervised machine learning to identify hotspots and discover hidden crime groupings.
The challenge
The original dataset contains more than eight million incidents collected across many years. Traditional single-machine processing becomes slow and difficult to scale at this volume.
Missing coordinates, invalid timestamps, duplicate IDs, inconsistent text, and geographic outliers also needed to be resolved before the data could support reliable analytics or clustering.
Analysis pipeline
The complete workflow was designed as a clear sequence from raw CSV ingestion to clustering evaluation. Every stage prepares the dataset for the next while preserving reproducibility.
Understanding the data
The selected schema combines categorical, numerical, geographic, temporal, and Boolean attributes. Together, these fields support both hotspot detection and time-based crime analysis.
| Feature | Type | Analytical purpose |
|---|---|---|
| Primary Type | Categorical | Crime category analysis |
| District | Numeric | District-level hotspot detection |
| Community Area | Numeric | Local spatial aggregation |
| Date | Timestamp | Hour, weekday, and month extraction |
| Latitude / Longitude | Numeric | Geographic clustering |
| Arrest / Domestic | Boolean | Contextual crime indicators |
Preprocessing at scale
Critical null records were removed, descriptive fields were filled with UNKNOWN, duplicate incidents were eliminated by ID, and dates were converted into valid timestamps.
Coordinates outside Chicago boundaries were filtered, text values were trimmed and standardized, and extreme geographic values were capped before a reproducible 5% sample was created.
Clean
Null handling, deduplication, timestamp validation
Reduce
5% seeded sample and focused feature selection
Transform
Type casting, indexing, encoding, and scaling
Validate
Coordinate boundaries and outlier treatment
Feature engineering
New features converted raw timestamps and locations into more useful indicators. These additions allowed the analysis to compare periods, recognize hotspot behavior, and give the clustering model richer context.
Distributed RDD analysis
The transformed data was converted into Resilient Distributed Datasets. Operations such as map, filter, reduceByKey, and sortBy produced frequency-based and temporal summaries across partitions.
val crimeCounts = rdd
.map(row => (row.primaryType, 1))
.reduceByKey(_ + _)
.sortBy(_._2, ascending = false)Hotspot districts
District-level aggregation exposed areas with consistently higher incident activity.
Common crime types
Primary Type grouping ranked the most frequently reported crime categories.
Hour and weekday patterns
Time-based features revealed meaningful variation across hours and days.
Spark SQL analytics
A temporary SQL view made large-scale filtering, grouping, ranking, and aggregation easier to interpret. Advanced queries also used CTEs, subqueries, and window functions.
SELECT Primary_Type, COUNT(*) AS Crime_Count
FROM crime_data
GROUP BY Primary_Type
ORDER BY Crime_Count DESC
LIMIT 10;KMeans model selection
Five numerical features were assembled and standardized before an 80/20 train-test split. Models from k = 2 through k = 6 were compared using the silhouette score.
The final model used k = 5 because it produced the strongest score, improving separation over the k = 2 baseline while maintaining balanced, non-empty clusters.
Technology stack
Apache Spark provided distributed processing through DataFrames, RDDs, Spark SQL, and MLlib. Scala supported Spark transformations and analytical logic, while CSV-based outputs preserved the processed data.
Project outcome
Chicago Vanguard demonstrates an end-to-end big data workflow that converts a noisy public dataset into scalable, interpretable crime analytics and behavioral clusters.
The result supports hotspot exploration and pattern discovery without claiming to predict individual incidents. Future work could add social context and compare KMeans with DBSCAN or Gaussian Mixture Models.