Real-time multi-person action detection

What it is

Bachelor thesis: cheating is a sequence (a glance that lasts, a lean, a note pass), so a single-frame classifier is the wrong tool, and a 3D CNN is the wrong budget for the machines in those rooms.

The repo is a single-process Python pipeline. cv2.VideoCapture takes a webcam, a video path, or one RTSP URL. It is not a multi-room product and there is no deploy config for ICDL centers in the repository.

Source: AliSaleemHasan/real-time-action-detection (GPL-3.0).

Figure 1. Config, preprocessing, pose, tracker, models, detect loop — from the repo (filename keeps the original typo).

---

Pipeline

Figure 2. Pose → tracker → per-person buffer → classifier → overlay.

1. Frame in. OpenCV capture.
2. Pose. Up to 6 people, 17 keypoints as (y, x, score), plus a box. Two backends implement the same interface:
   • MoveNet MultiPose Lightning (TF Hub, 160×256 pad) — what the README and thesis describe.
   • YOLOv11-nano pose (yolo11n-pose.pt) — what detect.py / preprocessing.py currently instantiate.
3. Filter. Keep skeletons with at least 4 keypoints above the score threshold (tracker default 0.5; config.yaml also has threshold: 0.4 for boxes). Near-duplicate candidates are suppressed in actual_skels().
4. Track. utils/tracker.py — mutual nearest-neighbor on joints 1–11. New IDs up to max_humans = 6; lost IDs go back into the pool.
5. Buffer. Per-ID FIFO of sequence_length (default 30). Features are keypoints.flatten() → 51 numbers (17 × 3), not 34. A stride (frame_distance, default 4 in config) can thin the buffer; detect.py currently forces frame_distance=1 unless a DATA_SET/sequence_rate.txt override exists.
6. Classify. When the buffer is full, ModelFactory runs LSTM or CNN1D from config.yaml.
7. Draw. OpenCV overlay: green keypoints, blue boxes, class label as putText. There is no green=normal / red=cheating HUD in the current drawer.

VideoCapture
    → FeatureGenerator.extractFeatures     # MoveNet or YOLO
    → Tracker.track                        # mutual NN, max 6 IDs
    → per-person list[-sequence_length:]
    → model.predict                        # LSTM or CNN1D
    → draw_output

---

Models

src/models/ is a small factory. Layers come from YAML, not hardcoded thesis diagrams.

Checked-in config.yaml (exercise demo):

classes: [JumpingJack, Lunges, PullUps, PushUps]
no_classes: 4
sequence_length: 30
no_sequences: 30
epochs: 2000
batch_size: 64
optimizer: adam
learning_rate: 0.0001
loss: categorical_crossentropy
architecture: CNN1D
frame_distance: 4
model:
  - { layer: Conv1D, filters: 64, kernel_size: 3, activation: relu }
  - { layer: MaxPooling1D, pool_size: 2 }
  - { layer: Flatten }
  - { layer: Dense, units: 32, activation: relu }
  - { layer: Dropout, drop_perc: 0.5 }
  - { layer: Dense, units: 4, activation: softmax }

The LSTM wrapper is still there (return_sequences stacked cells). Dataset folders are gitignored. README collection: short clips from disk (e.g. UCF50) or webcam; preprocessing.py --del_nonUsed True drops videos where pose found too few joints.

---

Thesis experiments (binary cheating)

The PDF thesis (not the README) is the source for the ICDL training story. Classes there are cheating vs not — 100 sequences each, about 6 000 frames. That is a different head from the 4-class softmax in git.

Iteration	Network	What happened
1	LSTM 16 → Dense 16 → sigmoid	Train accuracy near 1; validation collapsed to “always cheating”
2	+ Dropout 20%	Curves closer; 16 / 19 non-cheating clips still called cheating
3	LSTM 16 → LSTM 8 → Dropout 0.3 → Dense 8 → sigmoid, plus coordinate noise	18 errors / 208 val (≈ 91.3%); 5 errors / 15 unseen subjects (≈ 66.6%)

Augmentation in code (FeatureGenerator.augmentSkels) is additive noise, default 4 copies, intensity 0.1. The thesis run is the 15-clip number; the repo does not re-report it.

Thesis pose-only times (not the full loop): ThinkPad UHD 620 40 ms, MacBook 13” Iris 37 ms, MacBook 15” 555X 24 ms, P520 2080Ti 19 ms. Thesis LSTM step on CPU: 60 ms.

---

README timings (full loop, one laptop)

Lenovo ThinkBook 15 Gen 2, AMD Ryzen 5 4500U + integrated GPU:

• ~15 frames on one video
• ~5 frames with four videos in the same process
• Breakdown written in the README: pose ~40 ms, track ~15 ms, classifier ~70 ms (125 ms if you add them)

Those are the measurements I published. They are not a p99 and not a four-camera RTSP product.

---

Code layout

src/
  detect.py  preprocessing.py  train.py  config_schema.py
  models/          LSTM, CNN1D, factory
  pose_estimation/ interface, MoveNet, YOLO, FeatureGenerator
utils/             tracker, draw, data_processing
tests/             config, tracker, models, features, detect factory
scripts/setup_cuda.py

Python 3.9, TensorFlow 2.10 in pyproject.toml. README CUDA 12 path mentions newer TF; that is the setup script, not the lockfile default. Tests are unittest/pytest around the factory and tracker — no accuracy or FPS suite, no CI file.

Figure 3. 2022 snapshot of the tree. Current layout splits src/models and src/pose_estimation.

---

Run it

git clone https://github.com/AliSaleemHasan/real-time-action-detection
cd real-time-action-detection
uv run scripts/setup_cuda.py
uv lock && uv sync

# dataset from a folder of clips, or webcam with no --input
python src/preprocessing.py --input /path/to/videos --del_nonUsed True
python src/train.py
python src/detect.py --input None          # webcam
python src/detect.py --input /path/to.mp4
python src/detect.py --input rtsp://...

Weights and the MoveNet/YOLO files are not in git. config.yaml is the switch for classes, architecture, and sequence length.