Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR using GPU on PyAgender












1















Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow







share|improve this question

























  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14
















1















Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow







share|improve this question

























  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14














1












1








1








Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow







share|improve this question
















Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow




python tensorflow






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 3 at 2:17









talonmies

59.8k17132199




59.8k17132199










asked Jan 3 at 0:54









SpaceX RulesSpaceX Rules

122




122













  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14



















  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14

















This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

– SMX
Jan 15 at 5:14





This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

– SMX
Jan 15 at 5:14












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54015059%2fcould-not-create-cudnn-handle-cudnn-status-internal-error-using-gpu-on-pyagende%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54015059%2fcould-not-create-cudnn-handle-cudnn-status-internal-error-using-gpu-on-pyagende%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Can a sorcerer learn a 5th-level spell early by creating spell slots using the Font of Magic feature?

Image
28 6 $begingroup$ Per the Font of Magic feature, sorcerer can use Flexible Casting to create 5th-level spell slots at level 7, even though they are not typically available until level 9. You can transform unexpended sorcery points into one spell slot as a bonus action on your turn. The Creating Spell Slots table shows the cost of creating a spell slot of [5th level is 7] If such a sorcerer levels up while still having this 5th-level spell slot, can they choose a 5th-level spell as the spell they gain upon levelling up? The Spellcasting feature states: Additionally, when you gain a level in this class, you can choose one of the sorcerer spells you know and replace it with another spell from the sorcerer spell list, which also must be of a level for which you have spell slots.
Read more

Does disintegrating a polymorphed enemy still kill it after the 2018 errata?

Image
up vote 13 down vote favorite 1 After the 2018 errata, the disintegrate spell description now reads: A creature targeted by this spell must make a Dexterity saving throw. On a failed save, the target takes 10d6 + 40 force damage. The target is disintegrated if this damage leaves it with 0 hit points. If you were to polymorph an enemy into a rat and then disintegrate it, would the enemy be disintegrated or would it just return to its original form? I know that for Druids, it’s not an instant kill anymore, but is this the case for polymorph as well? dnd-5e spells polymorph errata share | improve this question edited 18 hours ago
Read more

A Topological Invariant for $pi_3(U(n))$

Image
3 3 $begingroup$ Recently, I saw a construction of topological invariant for $pi_3(U(n))$ with $ngeq 2$ : $$ N=frac{1}{24pi^2}int_{S^3} d^3x epsilon^{ijk} Tr[(U^{-1}partial_{x_i}U)(U^{-1}partial_{x_j}U)(U^{-1}partial_{x_k}U)] , $$ where $Uin U(n)$ depends on $boldsymbol{x}=(x_1,x_2,x_3)in S^3$ , $epsilon^{ijk}$ is the Levi-Civita symbol, $i,j,k=1,2,3$ , and the duplicated indexes are summed over. It is claimed that $N$ is an integer, but why? Update 02/02/2019 I think I got an argument for $n=2$ . In this case, $U=e^{i varphi} q$ with $qin SU(2)$ . Due to the trace and the Levi-Civita symbol in $N$ , $varphi$ does not contribute to $N$ . As $Tr[q^{dagger}partial_i q]=0$ and $(q^{dagger}partial_i q)^{dagger}=-q^{dagger}partial_i q$ , $q^{dagger}partial_i q$ in geneeral has the form $q^{dagger}partia
Read more