Huge time taken to finetune model on 10 samples #15
Description
HI,
I am trying to finetune the "lp://TableBank/faster_rcnn_R_50_FPN_3x/config" model on my custom data. I tried with 10 samples on g4dn GPU machine and it is showing ETA as 14:29.
[01/18 14:48:02 d2.utils.events]: eta: 14:29:49 iter: 1399 total_loss: 0.1085 loss_cls: 0.02311 loss_box_reg: 0.07594 loss_rpn_cls: 0.0008741 loss_rpn_loc: 0.004918 time: 0.6547 data_time: 0.0045 lr: 0.0005 max_mem: 4069M
Command used: python tools/train_net.py --dataset_name tables --json_annotation_train tables/train.json --image_path_train tables/images --json_annotation_val tables/test.json --image_path_val tables/images --config-file /tmp/.X11-unix/config.yaml OUTPUT_DIR /tmp/.X11-unix/outputs/prima/fast_rcnn_R_50_FPN_3x/ SOLVER.IMS_PER_BATCH 2 MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE 256 SOLVER.CHECKPOINT_PERIOD 20000 SOLVER.MAX_ITER 80000
Is this expected?
Regards,
Karthik