A two-stage computer vision pipeline that combines a fine-tuned **YOLOv8n** detector with a **EfficientNet** classifier to detect and identify products on retail shelves.
├── 3_inference.py ← end-to-end detect + classify on new images
├── api_server.py ← FastAPI polling server
└── train_classifier_kaggle.ipynb
```
---
## 2. Installation
The project uses a **Conda environment** for reproducible dependency management.
### Create the environment
```bash
conda env create -f environment.yml
conda activate product-detection
```
### `environment.yml`
```yaml
name: product-detection
channels:
- conda-forge
dependencies:
- python=3.10
- numpy<2
- pip
- pip:
- torch
- torchvision
- opencv-python
- ultralytics
- pillow==10.0.0
- tqdm
- pyyaml
- simsimd
- albumentations
- faiss-cpu
- fastapi
- uvicorn[standard]
- flask
```
### GPU users
`torch` and `torchvision` are installed as CPU builds by default via pip inside conda. To use a GPU, replace the `torch` and `torchvision` lines in the yml with the CUDA-enabled builds before creating the environment:
Once you have a working classifier, always use **Path B**. It is faster, more accurate, and requires far less manual labelling work.
---
## 4. Step-by-step Guide
### Step 1 — Generate crops
> Use this step if you are starting from scratch with **no classifier yet**.
> If you already have `runs/classify/best.pt`, skip to Step 2.
Runs YOLOv8n on your shelf images and saves every detection as a cropped JPEG into `data/unknown/`. No classification happens here — you label manually afterwards.
| `--output_dir` | `data` | Where confirmed crops are written |
| `--det_conf` | `0.25` | Detection threshold |
| `--cls_conf` | `0.50` | Min classifier confidence to auto-label |
| `--port` | `5000` | Browser UI port |
The browser opens automatically at `http://localhost:5000`.
#### Review UI controls
| Action | How |
|---|---|
| Open crop detail | Click a card |
| Confirm prediction | `C` key or **Confirm** button |
| Reject crop | `R` key or **Reject** button |
| Override the class | Pick from dropdown → **Apply override** |
| Navigate crops | `←``→` arrow keys |
| Close modal | `Esc` |
| Multi-select | `Ctrl+click` or `Shift+click` |
| Bulk confirm/reject/reclassify | Select multiple → use bulk bar |
| Confirm all visible | **✓ Confirm all visible** button |
| Filter by class | Click a class in the left sidebar |
| Filter by status | Toggle buttons in toolbar |
| Filter by confidence | Drag the min-conf slider |
#### Status meanings
| Status | Meaning |
|---|---|
| `auto` | Classifier's prediction, not yet reviewed |
| `confirmed` | You agreed with the prediction |
| `reclassified` | You changed it to a different class |
| `rejected` | Not a useful crop (bad detection, not a product of interest) |
#### Committing
Click **Commit dataset →** to write all crops to `data/`:
```
data/
cola_can/ ← confirmed + reclassified
pepsi_can/
_rejected_/ ← rejected crops (kept for reference)
_unreviewed_/ ← anything still "auto" when you committed
```
> **Important:** Commit is always additive. It never deletes existing files in `data/`.
> Running `auto_label.py` on new shelf images simply grows the dataset.
> The only exception: if you run it twice on the **exact same source image**, crop files
> are overwritten (same content, so this is harmless).
---
### Step 3 — Split the dataset
Splits `data/` into `train/`, `val/`, and `test/` using a **stratified** strategy — every class gets the same ratio across all splits, so rare classes are not accidentally lost.
> **Recommended minimums:** ≥ 50 training images and ≥ 10 val images per class.
> Classes below these thresholds should get more data before training.
---
### Step 4 — Train the classifier
Trains an **EfficientNet-B0** (ImageNet pre-trained) on `crops_dataset/`. A `WeightedRandomSampler` is used automatically so class imbalance does not bias training.
```bash
python 2_train_classifier.py \
--data_dir crops_dataset \
--model efficientnet_b0 \
--epochs 50 \
--batch_size 64 \
--amp
```
| Argument | Default | Description |
|---|---|---|
| `--data_dir` | required | Root with `train/` and `val/` |
| `--output_dir` | `runs/classify` | Where checkpoints are saved |
| `--freeze_backbone` | off | Only train the head — recommended when you have fewer than ~50 images per class |
**Outputs saved to `runs/classify/`:**
| File | Description |
|---|---|
| `best.pt` | Best checkpoint by val accuracy |
| `last.pt` | Last epoch checkpoint |
| `class_names.json` | `{index: class_name}` used by inference and API |
| `training_curves.png` | Loss & accuracy plot |
#### Option B — Kaggle notebook (recommended for free GPU access)
`train_classifier_kaggle.ipynb` is a self-contained notebook version of the training script, designed to run on Kaggle's free T4 GPU with no local hardware required.
**Setup steps:**
1. Zip your `crops_dataset/` folder and upload it as a Kaggle Dataset:
- Go to [kaggle.com/datasets](https://www.kaggle.com/datasets) → **New Dataset**
- Upload the zip, give it a name (e.g. `my-product-crops`)
- Wait for processing to complete
2. Create a new Kaggle Notebook:
- Go to [kaggle.com/code](https://www.kaggle.com/code) → **New Notebook**
- Upload `train_classifier_kaggle.ipynb` via **File → Import Notebook**
3. Attach your dataset:
- In the right panel click **+ Add Data**
- Search for your dataset name and attach it
- It will appear at `/kaggle/input/my-product-crops/`
| `--cls_conf` | `0.50` | Min classifier confidence to draw a label |
| `--det_iou` | `0.45` | NMS IoU threshold |
| `--padding` | `8` | Extra pixels added around each crop before classification |
| `--output_dir` | `inference_results` | Where annotated images/video are saved |
| `--show` | off | Display live with `cv2.imshow` |
| `--save_crops` | off | Also save individual crop images |
| `--no_save` | off | Skip saving annotated output |
#### Dynamic batch sizing
The classifier automatically probes available VRAM at startup and picks the largest safe batch size (with a 20% headroom margin). If an OOM occurs at runtime (e.g. an unusually dense frame with 260+ products), the batch is halved automatically and persisted for future frames. No manual tuning needed.
**`FileNotFoundError: No images found in: 01.jpg`**
You passed a file path that does not exist relative to your working directory. Use the full path or `cd` into the correct folder first.
**`OverflowError: Python integer 276 out of bounds for uint8`**
This was a known bug (HSV hue > 179) — already fixed in the current `3_inference.py`.
**Val loss stuck at ~5 during training**
See [Diagnosing a stuck val loss](#diagnosing-a-stuck-val-loss) above.
**`CUDA out of memory` during inference**
The dynamic batch sizer handles this automatically by halving the batch and retrying. If it keeps happening, your crops may be very large — reduce `--padding` or `--img_size`.
**`CUDA out of memory` during training**
Reduce `--batch_size`. Start at 32 and halve until it fits.
**Browser does not open automatically after `auto_label.py`**
Open `http://localhost:5000` manually.
**Classifier only predicts one class**
Almost always class imbalance. Check the imbalance report printed during `build_loaders`. The `WeightedRandomSampler` should compensate, but if one class has 10× more images, collect more data for the minority classes.
---
## 8. Tips & Best Practices
**Data collection**
- Aim for **≥ 50 training images per class** before expecting good results.
- Vary lighting, angles, partial occlusion, and zoom levels in your source images.
- Run `auto_label.py` on new batches of images regularly — the dataset grows incrementally and the model improves each iteration.
**Labelling**
- Use the **confidence slider** in the review UI to start reviewing high-confidence crops first (`> 0.8`) — these are almost always correct and can be bulk-confirmed quickly.
- Filter the sidebar to one class at a time and use **Confirm all visible** — scanning one class at a time is much faster than random-order review.
- Never put unrecognised products into a catch-all class. Leave them in `_unreviewed_/` or reject them. Use `--cls_conf` to suppress uncertain predictions at inference time.
**Training**
- Start with `--freeze_backbone` if you have fewer than 50 images per class. Once you have more data, retrain without it.
- Use the Kaggle notebook for free T4 GPU access — it includes a diagnostic cell that checks class counts and imbalance before training starts.
-`efficientnet_b0` is the best default — fast, accurate, small. Only upgrade to `efficientnet_b2` if B0 accuracy plateaus and you have plenty of data.
**Iterative improvement loop**
```
Shoot new shelf photos
↓
auto_label.py (detect + auto-classify)
↓
Review UI (confirm / fix / reject)
↓
1b_split_dataset.py (re-split the grown dataset)
↓
2_train_classifier.py (retrain from scratch or from last.pt)
