PyTorch多卡训练
2021-11-15 本文已影响0人
Valar_Morghulis
https://blog.csdn.net/kejizuiqianfang/article/details/102454278
https://blog.csdn.net/hu378910532/article/details/102860618
参照上面第一条链接：出现：
Single-Process Multi-GPU is not the recommended mode for DDP.
In this mode, each DDP instance operates on multiple devices and creates multiple module replicas within one process.
The overhead of scatter/gather and GIL contention in every forward pass can slow down training.
Please consider using one DDP instance per device or per module replica by explicitly setting device_ids or CUDA_VISIBLE_DEVICES.
"Single-Process Multi-GPU is not the recommended mode for "
另外，还有就是batch_size， lr， 梯度累积的间隔数的问题，有待进一步确认
