Cannot run algo.train()

[GowthamE7]

I tried the example on Auto3Dseg model and i couldn't able to start the training in my colab with GPU enabled.

[KumoLiu]

Hi @GowthamE7, could you please refer to this discussion and try to use the command shared there? Thanks!

[GowthamE7]

I am getting error in folds:

[KumoLiu]

Hi @GowthamE7, did you use the data json under this folder which contains "fold"?

[GowthamE7]

Hi @KumoLiu, I tried using that file but I got ended in this error:

So I tried base json file by downloading the dataset:

[KumoLiu]

Hi @GowthamE7, from the screenshot you shared I can't see the cause of the problem. But you should at least use the JSON file in the tutorial repo. And also share the whole error message that you get when you use that JSON file. Maybe I could take a deep look into it.

[GowthamE7]

@KumoLiu, I tried the json file inside the tutorials but it throws error.

JSON snapshot from tutorials:

But if I use the json which is downloaded while downloading the dataset I can run the code without Error.

JSON snapshot from dataset:

[KumoLiu]

Hi @GowthamE7, I noticed that here has a different format of the file path, maybe it's the reason.

[GowthamE7]

Hi @KumoLiu, Thanks for the answer I can now run the tutorial. I tried the same procedure to my custom dataset but I cannot able to run the training.

DataLoader

Error:

[KumoLiu]

Hi @GowthamE7, could you please share the whole error message? Thanks!

[GowthamE7]

@KumoLiu , This is the error message

/usr/local/lib/python3.8/dist-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.init:image_only: Current default value of argument image_only=False has been deprecated since version 1.1. It will be changed to image_only=True in version 1.3.
warn_deprecated(argname, msg, warning_category)
[info] number of GPUs: 1
[info] world_size: 1
train_files_w: 103
train_files_a: 104
val_files: 53
/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 6 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
2023-02-20 15:27:51.872644 - Length of input patch is recommended to be a multiple of 32.
num_epochs 1000
num_epochs_warmup 500
num_epochs_per_validation 20
[info] amp enabled
2023-02-20 15:27:55.253999: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-20 15:27:56.182970: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-20 15:27:56.183134: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-20 15:27:56.183157: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

epoch 1/1000
learning rate is set to 0.025
[2023-02-20 15:28:02] 1/52, train_loss: 1.0072
Process Process-4:
Process Process-6:
Process Process-2:
Process Process-5:
Process Process-3:
Exception ignored in: <function _MultiProcessingDataLoaderIter.del at 0x7f2bb0a24d30>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1466, in del
self._shutdown_workers()
File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1430, in _shutdown_workers
w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
File "/usr/lib/python3.8/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 44, in wait
if not wait([self.sentinel], timeout):
File "/usr/lib/python3.8/multiprocessing/connection.py", line 931, in wait
ready = selector.select(timeout)
File "/usr/lib/python3.8/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/content/ref_api_work_dir/dints_0/scripts/search.py", line 653, in
fire.Fire()
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/content/ref_api_work_dir/dints_0/scripts/search.py", line 331, in run
outputs = model(inputs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/networks/nets/dints.py", line 504, in forward
outputs = self.dints_space(inputs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/networks/nets/dints.py", line 1046, in forward
self.cell_tree[str((blk_idx, res_idx))](inputs[self.arch_code2in[res_idx]], weight=_w)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/networks/nets/dints.py", line 315, in forward
x = self.op(x, weight)
^C

[KumoLiu]

Hi @GowthamE7, I see KeyboardInterrupt in your error message, I would like to say you may interrupt the process unintentionally. Thanks!

[GowthamE7]

@KumoLiu, I didn't press ctrl+c. The code is getting stopped by itself. I checked it by running the code again and not touching my laptop and I am getting the same error.
/usr/local/lib/python3.8/dist-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.init:image_only: Current default value of argument image_only=False has been deprecated since version 1.1. It will be changed to image_only=True in version 1.3.
warn_deprecated(argname, msg, warning_category)
[info] number of GPUs: 1
[info] world_size: 1
train_files_w: 3
train_files_a: 4
val_files: 7
^C

[dongyang0122]

hi @GowthamE7, in your latest run, the log did not show the first iteration (even the smaller amount of data). But your previous run showed. can you confirm that the error messages are the same?

[KumoLiu]

Hi @GowthamE7

/usr/local/lib/python3.8/dist-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.init:image_only: Current default value of argument image_only=False has been deprecated since version 1.1. It will be changed to image_only=True in version 1.3. warn_deprecated(argname, msg, warning_category)

The message here, it's just a warning here, you can filter this warning message by adding image_only=True in LoadImage, but it's not the issue and won't stop your process.

[info] number of GPUs: 1 [info] world_size: 1 train_files_w: 3 train_files_a: 4 val_files: 7

Here is just some logging information.

^C

And here is the 'ctrl+c' I think.

[GowthamE7]

Hi @dongyang0122, I tried to run many times but the code is stopping because of "^C", but I din't press ctrl+c.Thanks!