Autonomous Car Vision System

Overview

This project implements the vision and decision layer for a miniature autonomous car. The car has two tasks: follow a lane marked on the floor, and correctly respond to road signs (stop sign, speed limit, turn markers). The entire processing pipeline runs on-board on a Raspberry Pi 4, taking camera frames from a PiCamera2 and outputting motor commands via a serial ASCII protocol.

A key design choice was the three-tier sign detection architecture: three backends of increasing complexity and accuracy (template matching → custom CNN → YOLOv5) are available and can be switched at runtime. This allows the system to operate in compute-constrained modes (template matching runs at 30+ FPS on Pi 4) or accuracy-critical modes (YOLOv5 at ~8 FPS). The lane following always runs separately, unaffected by which sign detection backend is active.

Processing Pipeline

Lane Following Algorithm

Lane following uses a classic computer vision approach:

1. Region of Interest (ROI): The bottom 40% of the frame is cropped — the lane line is always in this region during normal operation.
2. Binary thresholding: Convert to grayscale, apply adaptive Gaussian threshold to produce a binary mask where lane line pixels are white.
3. Centroid analysis: Find the centroid (x-coordinate of the centre of mass) of the white pixels. If the centroid is left of the frame midpoint, steer left; if right, steer right; if approximately centred, go straight.

Sign Detection Backends

Tier 1: Template Matching

OpenCV matchTemplate with TM_CCOEFF_NORMED. Pre-cropped sign templates at multiple scales are slid across the frame; matches above a threshold trigger detection. Fast (>30 FPS) but sensitive to lighting changes and rotation.

Tier 2: Custom CNN

A compact convolutional network trained on a dataset of road sign images collected specifically for this course competition. Architecture: 3× Conv+MaxPool blocks → GlobalAvgPool → Dense(128) → Softmax. Data augmentation: horizontal flip, brightness jitter ±30%, rotation ±15°, perspective warp. Runs at ~15 FPS on Raspberry Pi 4.

Tier 3: YOLOv5

YOLOv5 (ultralytics) fine-tuned on the competition sign dataset using transfer learning from the COCO-pretrained weights. YOLOv5n (nano) model chosen for Pi 4 performance. Detects bounding box + class in a single forward pass. ~8 FPS on Pi 4 — sufficient for the competition speed.

Serial Motor Protocol

Commands are sent over UART at 115200 baud to the motor controller MCU. The motor controller ACKs each command with OK\n; if no ACK arrives within 50 ms, the Pi resends.