Real-Time Multi-Person Action Detection & Exam Proctoring
Edge-Optimized Spatial-Temporal Deep Learning Architecture on Consumer Hardware
Executive Overview
Developed as a Bachelor’s Thesis project at Damascus University for the Syrian Computer Society (SCS), this system automates behavioral monitoring and cheating detection across International Computer Driving Licence (ICDL) examination centers in Syria.
The Real-World Engineering Problem
Large-scale certification facilities in resource-constrained environments face three critical engineering bottlenecks:
- Hardware Inaccessibility: Traditional video action recognition relies on heavy 3D Convolutional Neural Networks (e.g., I3D, SlowFast, Video Swin Transformers) that require dedicated, high-power GPU clusters. In regional examination centers, only standard consumer-grade office PCs and laptops (such as Intel Core i5 or AMD Ryzen mobile CPUs) are available.
- Camera Heterogeneity: Exam rooms utilize diverse CCTV setups, varying between webcams, IP RTSP streams, and pre-recorded surveillance clips, demanding a hardware-agnostic ingestion pipeline.
- The Temporal Nuance of Misconduct: Cheating is fundamentally a temporal process rather than a static visual event. An isolated glance towards a neighbor’s desk may be completely benign, whereas sustained head turning, leaning, or note passing spanning multiple seconds indicates cheating. A single-frame classifier produces unacceptable false-alarm rates.
To solve this, the project introduces an edge-optimized two-stage architecture: it decouples spatial feature extraction (via Google MoveNet MultiPose Lightning) from temporal sequence classification (via an engineered LSTM / 1D-CNN network), maintaining 15 FPS real-time multi-person action detection on standard laptop CPUs with zero dedicated GPU requirements.
Figure 1: Modular system architecture detailing configuration management, preprocessing utilities, feature generators, vectorized tracking, and real-time inference execution.
Architectural Comparison: Heavy 3D-CNNs vs. Decoupled Spatial-Temporal Pipeline
To illustrate the technical advantages of this design, the table below contrasts the decoupled pipeline against traditional deep learning video architectures:
| Architectural Metric | Heavy 3D-CNNs (I3D / SlowFast) | Naive Frame-by-Frame 2D-CNN | Decoupled Spatial-Temporal (This System) |
|---|---|---|---|
| Input Representation | Raw RGB Video Tensor (B x T x C x H x W) | Individual Video Frames (B x C x H x W) | 17-Keypoint Vector Time Series (T x 34) |
| Compute Overhead | Massive (~50–100 GFLOPs per clip) | High (Full image convolutions every frame) | Ultra-Low (~0.5 GFLOPs for temporal model) |
| Hardware Requirement | High-end workstation GPU (RTX 2080Ti+) | Dedicated mid-range GPU | Consumer Laptop CPU / Integrated Graphics |
| Temporal Context | Captures pixel-level motion blur | Zero (Blind to previous frame actions) | Full Sequence State Memory (LSTM Cell State) |
| Multi-Person Tracking | Heavy region proposals (Slow R-CNN) | Separate bounding box detector required | Vectorized Spatial Centroid Association |
| Inference Latency | > 350ms (Non-real-time on CPU) | ~80ms (High false-positive rate) | ~125ms (~15 FPS real-time edge budget) |
End-to-End System Pipeline Architecture
The surveillance pipeline executes across six continuous, modular stages designed to minimize CPU memory overhead and avoid redundant computations.
Figure 2: End-to-end action detection pipeline showing spatial pose estimation, candidate deduplication, centroid tracking, temporal rolling buffers, and LSTM action prediction.
+----------------------------------------------------------------------------------------------------+
| Real-Time Action Detection Pipeline |
| |
| [ RTSP Camera Stream / Video Feed ] |
| │ |
| ▼ |
| [ Frame Decoupling & Preprocessing (OpenCV) ] |
| │ |
| ▼ |
| [ Stage 1: MoveNet Multi-Pose Lightning ] ──(Extracts 17 Body Keypoints per candidate) |
| │ |
| ▼ |
| [ Stage 2: Quality Filtering & De-duplication ] ──(Min 4 keypoints + Euclidean Deduplication) |
| │ |
| ▼ |
| [ Stage 3: Vectorized Centroid Tracker ] ──(Mutual nearest-neighbor Euclidean distance matching)|
| │ |
| ▼ |
| [ Stage 4: Person Keypoint Rolling Buffer ] ──(FIFO window of N historical timesteps) |
| │ |
| ▼ |
| [ Stage 5: LSTM / 1D-CNN Sequence Classifier ] ──(Evaluates temporal progression of joints) |
| │ |
| ▼ |
| [ Stage 6: HUD Annotation & Alert Overlay ] ──(Real-time bounding box & status display) |
+----------------------------------------------------------------------------------------------------+Stage 1: Spatial Pose Estimation (MoveNet Multi-Pose)
Incoming video frames are ingested via OpenCV and passed to a pre-trained MoveNet Multi-Pose Lightning convolutional model via TensorFlow Hub.
Unlike single-person pose estimators that require a separate person detector (Top-Down approach, multiplying inference time by N individuals), MoveNet Multi-Pose utilizes a Bottom-Up architecture:
- Runs in a single forward pass (~40ms on consumer integrated graphics).
- Concurrently locates and tracks up to 6 individuals in the scene.
- Emits 17 anatomical keypoints per person: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles.
- Each keypoint is represented as a normalized tuple
(y, x, score)wherescorein[0, 1]represents model confidence.
Stage 2: Heuristic Filtering & Euclidean De-duplication
Raw model outputs in real-world exam halls suffer from camera noise, lens distortion, and partial occlusions. The system applies two strict heuristic filters before downstream tracking:
-
Minimum Articulation Thresholding (
num_of_keypoints >= 4): Cheating detection in seated students focuses heavily on upper-body gestures (head turns, shoulder tilts, arm movements). Detections with fewer than 4 high-confidence keypoints (score > 0.4) represent phantom detections or irrelevant background artifacts and are pruned immediately. -
Spatial Euclidean De-duplication: When students sit close together, MoveNet occasionally predicts two candidate skeletons for the same physical individual. The algorithm computes the pairwise normalized Euclidean distance across all corresponding joints between candidate A and candidate B. If the distance falls below
dist_thresh = 0.07, the lower-confidence candidate is suppressed.
Stage 3: Vectorized Centroid Tracker (Tracker Class)
MoveNet outputs arbitrary lists of skeletons with no temporal memory. To classify gestures over time, the system must assign and preserve consistent person identities (ID: 0, 1, 2, ...).
The custom Tracker class implements a mutual nearest-neighbor spatial association algorithm:
- Matching Condition: For skeleton S1[i] at frame t-1 and skeleton S2[j] at frame t, a match is confirmed if and only if:
- S2[j] is the nearest skeleton in frame t to S1[i].
- S1[i] is the nearest skeleton in frame t-1 to S2[j].
- The joint distance is strictly below
dist_thresh.
- Identity Allocation & Pruning: Unmatched skeletons in frame t are registered as newly arrived candidates up to
max_humans = 6. When an individual leaves the frame or remains occluded beyond the threshold, their ID is cleared and returned to the allocation pool, and their temporal buffer is flushed.
class Tracker(object):
"""
A lightweight spatial tracker:
Matches previous skeletons (S1) and current skeletons (S2) using
mutual nearest-neighbor Euclidean distance.
"""
def __init__(self, dist_thresh=0.07, score_thresh=0.4, num_of_keypoints=4, max_humans=6):
self.dist_thresh = dist_thresh
self.score_thresh = score_thresh
self.num_of_keypoints = num_of_keypoints
self.max_humans = max_humans
self.id2skeleton = {}
self.people = [False] * max_humans
def track(self, curr_skels, curr_boxes):
# 1. Prune skeletons with insufficient high-confidence keypoints
valid_skels = [s for s in curr_skels if self._count_valid(s) >= self.num_of_keypoints]
# 2. Compute mutual nearest neighbors across frame t-1 and frame t
matched_pairs = self._mutual_nn(self.id2skeleton, valid_skels, self.dist_thresh)
# 3. Update active track dictionary and recycle dropped IDs
return self._update_tracks(matched_pairs, valid_skels)Stage 4: Rolling Temporal Sequence Buffer
For each active individual, the pipeline maintains a FIFO rolling sequence buffer of length sequence_length = 30 frames. At each frame step:
- The 17 normalized keypoints are flattened into a 34-dimensional feature vector:
v_t = [x_1, y_1, x_2, y_2, ..., x_17, y_17] - A stride subsampling parameter (
frame_distance = 4) reduces redundant consecutive frames, allowing a 30-frame buffer to capture a representative temporal window of ~4–5 seconds of student activity. - The resulting input tensor passed to the model has shape
(batch_size, 30, 34).
Stage 5: Temporal Sequence Classification (LSTM / 1D-CNN)
Once the buffer reaches sequence_length, the 2D matrix is fed into the classifier to predict the action probability.
The architecture supports two interchangeable model backends through a unified ModelFactory:
- Long Short-Term Memory (LSTM) Backend: Retains cell state memory across timesteps, ideal for tracking continuous gradual trajectories (e.g., turning the torso and head toward a peer).
- 1D Convolutional Neural Network (CNN1D) Backend: Uses 1D temporal convolutions with pooling layers (
Conv1D(64, kernel_size=3) -> MaxPooling1D(2) -> Dense(32) -> Dropout(0.5) -> Softmax), optimizing forward inference speed on resource-constrained CPUs.
Stage 6: Heads-Up Display (HUD) Annotation
Predictions are rendered directly onto the video stream using OpenCV:
- Green bounding boxes indicate normal exam behaviors (typing, reading, desk focus).
- Red bounding boxes with high-confidence alert labels indicate cheating behaviors (looking at neighbor’s screen, exchanging notes, unpermitted movement).
- Alerts are persisted across consecutive frames to notify human proctors.
Figure 3: Production codebase structure showing modular separation across dataset preprocessing, feature generators, models, and real-time detection scripts.
Model Evolution & Empirical Training Journey
Training a robust classifier in an examination setting presents severe scientific challenges: data scarcity, privacy constraints in exam halls, and high class imbalance. The dataset was collected across controlled ICDL examination sessions, yielding 100 cheating sequences and 100 normal behavior sequences (~6,000 total annotated frames).
Reaching an enterprise-grade model required three rigorous experimental iterations:
+----------------------------------------------------------------------------------------------------+
| Model Evolution Timeline |
| |
| [ Iteration 1: Shallow Baseline ] ───► Overfitting Collapse (100% false-positive rate on test) |
| │ |
| ▼ |
| [ Iteration 2: Dropout Regularization ] ──► Loss curves converge, but high false alarms (16/19) |
| │ |
| ▼ |
| [ Iteration 3: Stacked Topology + Noise Augmentation ] ──► 66.6% Unseen Real-World Test Accuracy |
+----------------------------------------------------------------------------------------------------+Iteration 1: The Shallow Baseline & Overfitting Collapse
The initial network architecture comprised:
- Topology: 1x LSTM Layer (16 units) -> 1x Dense Layer (16 units, ReLU) -> 1x Output Dense (1 unit, Sigmoid)
- Optimizer: Adam (alpha = 0.001) | Loss: Binary Cross-Entropy
Empirical Outcome & Failure Diagnosis:
- Training Metrics: Training loss dropped to near-zero; training accuracy reached ~0.99 within 40 epochs.
- Validation Metrics: Validation loss diverged sharply, climbing above 2.4.
- Confusion Matrix Analysis: The model predicted 100% of validation and test instances as “Cheating”.
- Root Cause: With 34 continuous spatial dimensions across 30 time steps (1,020 input values per sequence), the shallow network possessed sufficient degrees of freedom to memorize student desk coordinates rather than learning generic temporal gesture dynamics.
Iteration 2: Dropout Regularization
To break feature co-adaptation, two Dropout layers (20% rate) were integrated into the feed-forward stages.
Empirical Outcome & Limitations:
- Convergence: Training loss and validation loss stabilized and converged closely.
- Confusion Matrix Analysis: When tested on a 19-sample non-cheating validation set, the network misclassified 16 non-cheating sequences as cheating.
- Root Cause: While dropout prevented catastrophic node memorization, the model suffered from high variance due to small sample size. Body proportion differences and minor seat shifts were misinterpreted as anomalous movements. More training variance was required.
Iteration 3: Stacked Topology & Domain-Specific Coordinate Augmentation
To achieve generalized feature learning, the training pipeline was fundamentally overhauled:
1. Synthetic Keypoint Gaussian Noise Augmentation
To multiply dataset diversity without recording intrusive exam footage, synthetic Gaussian noise was injected into joint coordinates during batch generation:
x_perturbed = x + Gaussian_Noise(mean=0, std=0.02)
This mathematically simulated camera jitter, seating distance variations, and individual anthropometric differences, expanding the effective dataset four-fold.
2. Deep Stacked Topology
The classifier was redesigned into a deep hierarchical network:
- Layer 1: LSTM (16 units,
return_sequences=True) - Layer 2: LSTM (8 units,
return_sequences=False) - Layer 3: Dropout (Rate = 0.3)
- Layer 4: Fully Connected Dense (8 units, ReLU activation)
- Layer 5: Output Dense (1 unit, Sigmoid activation)
Final Quantitative Results:
- Training Stability: Training loss and validation loss descended smoothly with zero divergence.
- Validation Performance: Misclassified only 18 out of 208 validation sequences (91.3% validation accuracy).
- Unseen Subject Generalization: Evaluated against completely novel student subjects in blind testing, the model achieved 66.6% accuracy (10 correct classifications out of 15 trials), successfully proving that the network learned genuine invariant gesture dynamics.
Edge Latency Benchmarking & Performance Budget
In automated exam proctoring, latency must operate within an instantaneous human reaction window. The extraction engine was benchmarked across diverse consumer and workstation hardware profiles:
Spatial Keypoint Extraction Latency (MoveNet MultiPose)
| Hardware Profile | GPU / Compute Unit | Extraction Latency per Frame | Max Theoretical FPS |
|---|---|---|---|
| Lenovo ThinkPad v7 (Linux) | Intel UHD 620 Integrated Graphics | 40ms | 25 FPS |
| Apple MacBook Pro 13” (2019) | Intel Iris Plus Graphics 655 (1.5 GB) | 37ms | 27 FPS |
| Apple MacBook Pro 15” (2019) | AMD Radeon Pro 555X (4 GB Dedicated) | 24ms | 41 FPS |
| Lenovo P520 Workstation | NVIDIA GeForce RTX 2080Ti | 19ms | 52 FPS |
Turnaround Latency Budget (AMD Ryzen 5 4500U CPU)
On a standard Lenovo ThinkBook 15 Gen 2 (AMD Ryzen 5 4500U CPU with integrated Radeon graphics), the pipeline operates within a strict latency budget:
+-------------------------------------------------------------------------------+
| End-to-End Frame Latency Budget |
| |
| [ MoveNet MultiPose Pose Estimation: ~40ms ] |
| │ |
| ▼ |
| [ Mutual Nearest-Neighbor Centroid Tracking: ~15ms ] |
| │ |
| ▼ |
| [ LSTM / CNN1D Sequence Evaluation: ~70ms ] |
| │ |
| ▼ |
| Total Turnaround Latency: ~125ms (~15 FPS Real-Time Throughput on CPU) |
+-------------------------------------------------------------------------------+Multi-Camera Concurrency
In production exam facilities, proctors must monitor multiple camera angles simultaneously. Because the temporal sequence classifier only executes every N frames per person, inference time is staggered across streams:
- A single quad-core laptop CPU handles 4 concurrent RTSP camera feeds at ~5 FPS.
- Because deliberate cheating gestures (e.g., whispering or looking sideways) typically persist for 2–4 seconds, a 5 FPS sampling rate captures 10–20 temporal observations per action, providing reliable detection without dropping frames.
Technical Configuration & CLI Usage
The system is configured via a centralized config.yaml file parsed into strongly typed schemas:
# config.yaml
data_directory: "DATA_SET"
test_set_path: "TEST_SET"
classes: ["Normal", "Looking_Neighbor", "Passing_Notes", "Standing"]
threshold: 0.4
sequence_length: 30
frame_distance: 4
batch_size: 64
epochs: 100
optimizer: "adam"
learning_rate: 0.0001
loss: "binary_crossentropy"
architecture: "LSTM"Reproduction & CLI Execution
# 1. Clone repository & install dependencies
git clone https://github.com/AliSaleemHasan/real-time-action-detection
cd real-time-action-detection
pip install -r requirements.txt
# 2. Extract skeleton dataset from pre-recorded exam footage
python src/preprocessing.py --input /path/to/exam_hall_videos --del_nonUsed True
# 3. Train the stacked LSTM network with keypoint noise augmentation
python src/train.py
# 4. Launch real-time detection on exam center RTSP stream
python src/detect.py --input rtsp://192.168.1.100:554/exam_room_1