Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR using GPU on PyAgender












1















Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow







share|improve this question

























  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14
















1















Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow







share|improve this question

























  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14














1












1








1








Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow







share|improve this question
















Sample code runs fine with CUDNN but using code installed with pip returns Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR.



I am using Tensorflow 1.12, CUDA 10.0 and CUDNN 7.4.1 compiled with GPU 2080 (x2)



This is Ubuntu 16.04 LTS. I also use venv. Everything is done in venv using Python 3.5.2.



I am trying to use Py-Agender from https://pypi.org/project/py-agender/



After compiling CUDA with GPU flags and successfully verifying that it works, I tried:
enter code herepip3 install py-agender.



Running it then causes an internal CUDNN error.



I have removed ~/.nv as suggested in other replies and it did not help



I have also confirmed that CuDNN does in fact see and can use GPUs:



print(device_lib.list_local_devices()) returns:



2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0


... so we at least something works!



b. mNIST sample runs and returns values as expected.



The part of code is as follows:



agender = PyAgender() 



def get_gender(value) :

    if value > 0.5:

        return 'female'

    else :

        return 'male'



input_dir = sys.argv[1]



directories = os.listdir(input_dir)



if len(directories):

    for video in directories:

        files = glob.glob(input_dir + video + '/*')



        if len(files):

            for file_path in files:



                filename, file_extension = os.path.splitext(os.path.basename(file_path))

                faces = agender.detect_genders_ages(cv2.imread(file_path))


The code worked fine with Tensorflow compiled with CPU only. With GPU, the full error is:



$ python age-gender.py ./frames/part1/



Using TensorFlow backend.

- Began to analyze the age and gender 

2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:

name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8

pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su

ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found

 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB

2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin

g visible gpu devices: 0, 1

2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device

 interconnect StreamExecutor with strength 1 edge matrix:

2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1

2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y

2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N

2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->

 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)

2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->

 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)

2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 

2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 

2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Traceback (most recent call last):

  File "age-gender.py", line 50, in <module>

    faces = agender.detect_genders_ages(cv2.imread(file_path))

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages

    height=face['height'])

  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age

    result = self.resnet.predict(np.array([test_img]))

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict

    steps=steps)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop

    batch_outs = f(ins_batch)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

    return self._call(inputs)

  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

    fetched = self._callable_fn(*array_vals)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

    run_metadata_ptr)

  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

    c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]





python tensorflow




python tensorflow






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 3 at 2:17









talonmies

59.8k17132199




59.8k17132199










asked Jan 3 at 0:54









SpaceX RulesSpaceX Rules

122




122













  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14



















  • This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

    – SMX
    Jan 15 at 5:14

















This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

– SMX
Jan 15 at 5:14





This might be similar to the issue raised here: github.com/tensorflow/tensorflow/issues/24496

– SMX
Jan 15 at 5:14












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54015059%2fcould-not-create-cudnn-handle-cudnn-status-internal-error-using-gpu-on-pyagende%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54015059%2fcould-not-create-cudnn-handle-cudnn-status-internal-error-using-gpu-on-pyagende%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

'app-layout' is not a known element: how to share Component with different Modules

Image
0 My application was working fine until I decided to go with lazy loading: So, my shared component looks like this: import { Component, Renderer2 } from '@angular/core'; export interface FormModel { captcha?: string; } @Component({ selector: 'app-layout', templateUrl: './app-layout.component.html', styleUrls: ['./app-layout.css'] }) export class FullLayoutComponent { } app-layout.component.ts I am using this component in my Another component say school-home.component.html like this: <app-layout> </app-layout> and it was working fine until I created a Module school-home.module.ts like this: import { NgModule } from '@angular/core'; import { CommonModule } from '@angular/common'; import { SchoolhomeRoutingModule ...
Read more

android studio warns about leanback feature tag usage required on manifest while using Unity exported app?

Image
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 This is a simple project showing cubes on Unity, exported to android project and imported on Android studio. On maifest file it showed following warning: Expecting <uses-feature android:name="android.software.leanback" android:required="false" /> tag I have no information about it. And also on default emptyview android project does not use/mention this feature. I know it is required by TV's and such but why its not added by default? Why is this? And Should I concern about it? android unity3d ...
Read more

WPF add header to Image with URL pettitions [duplicate]

Image
up vote 0 down vote favorite This question already has an answer here: Image source URI requiring authorization 3 answers Add header to all* HTTP requests on a machine 1 answer first of all sorry for my english. Recently I added some security in the backend of my project, and now I need to send a validation code(apiKey) inside the headers to retrieve the images. I have a problem with ALL images in my project that I had attached in my UI. For example, before I had: <Image Height="90" Source="{Binding Video.Thumbnail}"/> Now I need to change my code to create a CUSTOM petition add the apiKey in the header:...
Read more