【问题记录】Tensorflow-GPU下训练出现 CUDA_E
太长不看版
解决问题的思路：
- 从头到尾看看自己安装配置的环节是否齐全，包括C++编译库、CUDA安装、CuDNN环境配置、tensorflow-gpu的下载安装。
- 检查版本是否对应，Python, CUDA, CuDNN, Tensorflow版本是否对应以及兼容。
- 是否只是运行特定代码时出错
- 否：继续尝试下一步
- 是： 检查代码是不是太过复杂，你的机器承受不了
你可以运行tensorflow官网给出的简单示例代码：>>> import tensorflow as tf >>> hello = tf.constant('Hello, TensorFlow!') >>> sess = tf.Session() >>> print(sess.run(hello))
- 目前来说最万全之策：从源码编译安装Tensorflow。
源码编译安装是为了最大程度上使得Tensorflow的运行更适配你的计算机配置，发挥出最大效用，也能支出AVX等更进一步加速计算，也能在一种程度上解决运算效率的问题。
参考 Build from source on Windows
Speed up TensorFlow inference by compiling it from source
问题记录
> python .\0042_demo.py
...
Extracting MNIST_data\t10k-images-idx3-ubyte.gz
Extracting MNIST_data\t10k-labels-idx1-ubyte.gz
...
2018-09-28 14:56:04.341923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
1409 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0)
Iter 0, Test Accuracy 0.9493 Training Accuracy 0.9581636
2018-09-28 14:56:20.996376: E tensorflow/stream_executor/cuda/cuda_driver.cc:1000] could not wait stream on event: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2018-09-28 14:56:20.996373: E tensorflow/stream_executor/cuda/cuda_driver.cc:1130] failed to enqueue async memcpy from host to device: CUDA_ERROR_LAUNCH_TIMEOUT:
the launch timed out and was terminated; GPU dst: 0000000402DD1100; host src: 000001196EC8CB80; size: 313600=0x4c900
2018-09-28 14:56:20.996423: E tensorflow/stream_executor/cuda/cuda_driver.cc:1000] could not wait stream on event: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2018-09-28 14:56:21.015902: I tensorflow/stream_executor/stream.cc:4986] [stream=0000011977EA22B0,impl=00000119008317F0] did not memcpy host-to-device; source: 0000011966E1FC00
2018-09-28 14:56:21.093012: E tensorflow/stream_executor/stream.cc:325] Error recording event in stream: error recording CUDA event on stream 000001197FBFD2C0: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2018-09-28 14:56:21.103354: I tensorflow/stream_executor/stream.cc:4986] [stream=0000011977EA22B0,impl=00000119008317F0] did not memcpy host-to-device; source: 000001196EBA6180
2018-09-28 14:56:21.128940: E tensorflow/stream_executor/cuda/cuda_event.cc:48] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_TIMEOUT:
the launch timed out and was terminated
2018-09-28 14:56:21.179315: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:274] Unexpected Event status: 1
环境
- Win 10
- GeForce 940MX
- CUDA 9.0
- CuDNN 7.3 for CUDA9.0
- Tensorflow 1.11.0
- Visual Studio 2010
遇到同样的问题出现
#1060
https://github.com/tensorflow/tensorflow/issues/1060
推荐的解决方案是：
From a different issue #2810, we've found some problems with 940M cuda driver. The problem was solved by:
#2810 (comment)
- Build from source while explicitly setting 5.0 build target in "configure".
- Or install the latest graphics driver 367.27.
Not sure whether it is related. But it is worth trying.
#8517
https://github.com/tensorflow/tensorflow/issues/8517
Than you poxvoculi, it occurs every time I run the program.
Actually, this issue does not occur on the TensorFlow built from source. But it does occur on pip version.
BTW, I think it only happens on multi-gpu system.
cudaErrorLaunchTimeout
This indicates that the device kernel took too long to execute. This can only occur if timeouts are enabled - see the device property kernelExecTimeoutEnabled for more information. The device cannot be used until cudaThreadExit() is called. All existing device memory allocations are invalid and must be reconstructed if the program is to continue using CUDA.
------本文来自 todayq 的CSDN 博客 ，全文地址请点击：https://blog.csdn.net/dan1900/article/details/17411203?utm_source=copy
目前的处理策略 关闭显卡TDR（没用）
百度一番之后发现原来是windows系统的显卡超时检测和恢复（TDR）功能惹的祸。关闭TDR的方法是在HKLM\System\CurrentControlSet\Control\GraphicsDrivers下创建Dword值TdrLevel，并赋值为0
（https://answers.microsoft.com/zh-hans/windows/forum/windows_7-hardware/win7%E4%B8%AD%E5%A6%82%E4%BD%95%E9%85%8D%E7%BD%AE/69384e71-5075-4afe-a437-372425c0a3bb?auth=1）
---------------------本文来自 qq_32464407 的CSDN 博客 ，全文地址请点击：https://blog.csdn.net/qq_32464407/article/details/79164305?utm_source=copy
所以，我调这么久的错，原因只是，我的电脑，配置不够高。
- 网上的解决方案，包括源码构建，升级显卡驱动，都是为了尽可能提升性能，提升瓶颈
- 我把隐藏层神经元个数从2000调整成200，完美运行
运行设别相关的代码
- 指定CPU设备运行 tf.device() 指定本地或者远程的设备
with tf.device('/cpu:0'):
#各种operation
- 查看运行每一个运算的设备: Session()中指定参数
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess :
