Peterande/D-FINE

We conduct object detection using D-FINE and YOLO11 on a complex street scene video from YouTube. Despite challenging conditions such as backlighting, motion blur, and dense crowds, D-FINE-X successfully detects nearly all targets, including subtle small objects like backpacks, bicycles, and traffic lights. Its confidence scores and the localization precision for blurred edges are significantly higher than those of YOLO11.

https://github.com/user-attachments/assets/e5933d8e-3c8a-400e-870b-4e452f5321d9

• E25: Re-trained and extended the pretraining to 25 epochs.
• AP^val is evaluated on Objects365 full validation set.
• AP⁵⁰⁰⁰ is evaluated on the first 5000 samples of the Objects365 validation set.

1. Download COCO2017 from OpenDataLab or COCO.

2. Modify paths in coco_detection.yml

   1 2 3 4 5 6

   train_dataloader: img_folder: /data/COCO2017/train2017/ ann_file: /data/COCO2017/annotations/instances_train2017.json val_dataloader: img_folder: /data/COCO2017/val2017/ ann_file: /data/COCO2017/annotations/instances_val2017.json

   Copy

1. Download Objects365 from OpenDataLab.

2. Set the Base Directory:

1

export BASE_DIR=/data/Objects365/data

3. Extract and organize the downloaded files, resulting directory structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

${BASE_DIR}/train
├── images
│   ├── v1
│   │   ├── patch0
│   │   │   ├── 000000000.jpg
│   │   │   ├── 000000001.jpg
│   │   │   └── ... (more images)
│   ├── v2
│   │   ├── patchx
│   │   │   ├── 000000000.jpg
│   │   │   ├── 000000001.jpg
│   │   │   └── ... (more images)
├── zhiyuan_objv2_train.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

${BASE_DIR}/val
├── images
│   ├── v1
│   │   ├── patch0
│   │   │   ├── 000000000.jpg
│   │   │   └── ... (more images)
│   ├── v2
│   │   ├── patchx
│   │   │   ├── 000000000.jpg
│   │   │   └── ... (more images)
├── zhiyuan_objv2_val.json

4. Create a New Directory to Store Images from the Validation Set:

1

mkdir -p ${BASE_DIR}/train/images_from_val

5. Copy the v1 and v2 folders from the val directory into the train/images_from_val directory

1
2

cp -r ${BASE_DIR}/val/images/v1 ${BASE_DIR}/train/images_from_val/
cp -r ${BASE_DIR}/val/images/v2 ${BASE_DIR}/train/images_from_val/

6. Run remap_obj365.py to merge a subset of the validation set into the training set. Specifically, this script moves samples with indices between 5000 and 800000 from the validation set to the training set.

1

python tools/remap_obj365.py --base_dir ${BASE_DIR}

7. Run the resize_obj365.py script to resize any images in the dataset where the maximum edge length exceeds 640 pixels. Use the updated JSON file generated in Step 5 to process the sample data. Ensure that you resize images in both the train and val datasets to maintain consistency.

1

python tools/resize_obj365.py --base_dir ${BASE_DIR}

8. Modify paths in obj365_detection.yml

   1 2 3 4 5 6

   train_dataloader: img_folder: /data/Objects365/data/train ann_file: /data/Objects365/data/train/new_zhiyuan_objv2_train_resized.json val_dataloader: img_folder: /data/Objects365/data/val/ ann_file: /data/Objects365/data/val/new_zhiyuan_objv2_val_resized.json

   Copy

Download COCO format dataset here: url

To train on your custom dataset, you need to organize it in the COCO format. Follow the steps below to prepare your dataset:

1. Set remap_mscoco_category to False:

   This prevents the automatic remapping of category IDs to match the MSCOCO categories.

   1	remap_mscoco_category: False

   Copy

2. Organize Images:

   Structure your dataset directories as follows:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14

   dataset/ ├── images/ │ ├── train/ │ │ ├── image1.jpg │ │ ├── image2.jpg │ │ └── ... │ ├── val/ │ │ ├── image1.jpg │ │ ├── image2.jpg │ │ └── ... └── annotations/ ├── instances_train.json ├── instances_val.json └── ...

   Copy
   • images/train/: Contains all training images.
   • images/val/: Contains all validation images.
   • annotations/: Contains COCO-formatted annotation files.

3. Convert Annotations to COCO Format:

   If your annotations are not already in COCO format, you’ll need to convert them. You can use the following Python script as a reference or utilize existing tools:

   1 2 3 4 5 6 7 8

   import json def convert_to_coco(input_annotations, output_annotations): # Implement conversion logic here pass if __name__ == "__main__": convert_to_coco('path/to/your_annotations.json', 'dataset/annotations/instances_train.json')

   Copy

4. Update Configuration Files:

   Modify your custom_detection.yml.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

   task: detection evaluator: type: CocoEvaluator iou_types: ['bbox', ] num_classes: 777 # your dataset classes remap_mscoco_category: False train_dataloader: type: DataLoader dataset: type: CocoDetection img_folder: /data/yourdataset/train ann_file: /data/yourdataset/train/train.json return_masks: False transforms: type: Compose ops: ~ shuffle: True num_workers: 4 drop_last: True collate_fn: type: BatchImageCollateFunction val_dataloader: type: DataLoader dataset: type: CocoDetection img_folder: /data/yourdataset/val ann_file: /data/yourdataset/val/ann.json return_masks: False transforms: type: Compose ops: ~ shuffle: False num_workers: 4 drop_last: False collate_fn: type: BatchImageCollateFunction

   Copy

1. Set Model

1

export model=l  # n s m l x

2. Training

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --use-amp --seed=0

3. Testing

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --test-only -r model.pth

4. Tuning

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --use-amp --seed=0 -t model.pth

1. Set Model

1

export model=l  # n s m l x

2. Training on Objects365

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/objects365/dfine_hgnetv2_${model}_obj365.yml --use-amp --seed=0

3. Tuning on COCO2017

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/objects365/dfine_hgnetv2_${model}_obj2coco.yml --use-amp --seed=0 -t model.pth

4. Testing

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --test-only -r model.pth

1. Set Model

1

export model=l  # n s m l x

2. Training on Custom Dataset

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/dfine_hgnetv2_${model}_custom.yml --use-amp --seed=0

3. Testing

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/dfine_hgnetv2_${model}_custom.yml --test-only -r model.pth

4. Tuning on Custom Dataset

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/objects365/dfine_hgnetv2_${model}_obj2custom.yml --use-amp --seed=0 -t model.pth

5. [Optional] Modify Class Mappings:

When using the Objects365 pre-trained weights to train on your custom dataset, the example assumes that your dataset only contains the classes 'Person' and 'Car'. For faster convergence, you can modify self.obj365_ids in src/solver/_solver.py as follows:

1

self.obj365_ids = [0, 5]  # Person, Cars

You can replace these with any corresponding classes from your dataset. The list of Objects365 classes with their corresponding IDs: https://github.com/Peterande/D-FINE/blob/352a94ece291e26e1957df81277bef00fe88a8e3/src/solver/_solver.py#L330

New training command:

1

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/dfine_hgnetv2_${model}_custom.yml --use-amp --seed=0 -t model.pth

However, if you don’t wish to modify the class mappings, the pre-trained Objects365 weights will still work without any changes. Modifying the class mappings is optional and can potentially accelerate convergence for specific tasks.

For example, if you want to double the total batch size when training D-FINE-L on COCO2017, here are the steps you should follow:

1. Modify your dataloader.yml to increase the total_batch_size:

   1 2	train_dataloader: total_batch_size: 64 # Previously it was 32, now doubled

   Copy

2. Modify your dfine_hgnetv2_l_coco.yml. Here’s how the key parameters should be adjusted:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

   optimizer: type: AdamW params: - params: '^(?=.*backbone)(?!.*norm|bn).*$' lr: 0.000025 # doubled, linear scaling law - params: '^(?=.*(?:encoder|decoder))(?=.*(?:norm|bn)).*$' weight_decay: 0. lr: 0.0005 # doubled, linear scaling law betas: [0.9, 0.999] weight_decay: 0.0001 # need a grid search ema: # added EMA settings decay: 0.9998 # adjusted by 1 - (1 - decay) * 2 warmups: 500 # halved lr_warmup_scheduler: warmup_duration: 250 # halved

   Copy

If you’d like to train D-FINE-L on COCO2017 with an input size of 320x320, follow these steps:

1. Modify your dataloader.yml:

   1 2 3 4 5 6 7 8 9 10 11 12

   train_dataloader: dataset: transforms: ops: - {type: Resize, size: [320, 320], } collate_fn: base_size: 320 dataset: transforms: ops: - {type: Resize, size: [320, 320], }

   Copy

2. Modify your dfine_hgnetv2.yml:

   1	eval_spatial_size: [320, 320]

   Copy

1. Setup

1
2

pip install onnx onnxsim
export model=l  # n s m l x

2. Export onnx

1

python tools/deployment/export_onnx.py --check -c configs/dfine/dfine_hgnetv2_${model}_coco.yml -r model.pth

3. Export tensorrt

1

trtexec --onnx="model.onnx" --saveEngine="model.engine" --fp16

1. Setup

1
2

pip install -r tools/inference/requirements.txt
export model=l  # n s m l x

2. Inference (onnxruntime / tensorrt / torch)

Inference on images and videos is now supported.

1
2
3

python tools/inference/onnx_inf.py --onnx model.onnx --input image.jpg  # video.mp4
python tools/inference/trt_inf.py --trt model.engine --input image.jpg
python tools/inference/torch_inf.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml -r model.pth --input image.jpg --device cuda:0

1. Setup

1
2

pip install -r tools/benchmark/requirements.txt
export model=l  # n s m l x

2. Model FLOPs, MACs, and Params

1

python tools/benchmark/get_info.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml

2. TensorRT Latency

1

python tools/benchmark/trt_benchmark.py --COCO_dir path/to/COCO2017 --engine_dir model.engine

1. Setup

1
2

pip install fiftyone
export model=l  # n s m l x

4. Voxel51 Fiftyone Visualization (fiftyone)

1

python tools/visualization/fiftyone_vis.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml -r model.pth

1. Auto Resume Training

1

bash reference/safe_training.sh

2. Converting Model Weights

1

python reference/convert_weight.py model.pth

1. Overview of D-FINE with FDR. The probability distributions that act as a more fine- grained intermediate representation are iteratively refined by the decoder layers in a residual manner. Non-uniform weighting functions are applied to allow for finer localization.

2. Overview of GO-LSD process. Localization knowledge from the final layer’s refined distributions is distilled into earlier layers through DDF loss with decoupled weighting strategies.

Visualizations of FDR across detection scenarios with initial and refined bounding boxes, along with unweighted and weighted distributions.

The following visualization demonstrates D-FINE’s predictions in various complex detection scenarios. These include cases with occlusion, low-light conditions, motion blur, depth of field effects, and densely populated scenes. Despite these challenges, D-FINE consistently produces accurate localization results.